The evaluation of the system developed in the final study was based upon both qualitative and quantitative methods. The first stage of the study involved the isolation of the characteristics, features and variables involved in the phenomenon under study, the use of a student model and individual configuration in a multimedia course. This is often referred to as the open and axial coding stages of the method. By the use of this method, it was possible to establish relationships between the variables and to understand better the phenomenon. In the following sections, the stages involved in the process of identifying these characteristics and variables are described.
Establishment of the main categories
The theme of the study related to how individual configuration of the multimedia course was important in the teaching and learning process. In order to understand this phenomenon it was necessary to identify the issues that were important. This was achieved by the process of open coding as described by Strauss and Corbin (1990) and as outlined in the main paper. Variables were identified in an initial pre -study at 7 FE Colleges. Early prototypes of the learning materials were demonstrated to groups of tutors, learning centre managers and college middle/senior managers. Presentations were accompanied by discussion and question sessions. After the presentations, participants were asked to list features that they considered to be important in multimedia learning. A total of 82 lists were examined and summarised in order to prepare a questionnaire to be administered to workers in the area of multimedia and education. This questionnaire rated how important they considered the listed features to be in educational multimedia on a scale of 1 to 5. The instrument also recorded the position of the responder within their organisation. Questionnaires were distributed to 21 colleges of FE in the UK at meetings, training sessions, presentations and demonstrations for a range of college staff involved in multimedia and learning. Staff were also asked to list any features they considered important and not covered by the questionnaire. The results of this study are presented here.
The identification of the main categories involved the production of a hierarchical structure, which showed tentative relationships between factors identified from the work described above, that were consistently rated as being important to the phenomenon. This hierarchy is presented here. The production, refinement and modification of this hierarchy of categories and sub-categories is described in the following sections.
Related variables
Once the main categories relating to the individual configuration of multimedia had been identified as described above, it was important to understand how these categories were related. Relationships between the categories were identified as follows:
Key staff (i.e. those involved in the day to day delivery of open and flexible learning) at 8 colleges of FE were provided with final prototypes of the application. These contained full instructions for their installation and use. They were asked to comment on features identified as being important in the previous stage from the questionnaire and lists. Each of the key staff provided a semi-structured written report on the use of the prototypes at their colleges. They were encouraged to show the prototype to others and elicit their responses. Key staff were also asked to provide additional comments on any aspect of the application they themselves considered to be important.
After the initial analysis of these data, interviews were conducted with volunteer experts from four of the colleges involved. These interviews were unstructured or semi-structured with the intention of exploring issues identified in written reports. Interviews took place between four and eight weeks after receipt of the written report. The objective of this was to identify how those variables identified in the report might be related one to another. For example, where pedagogy was identified as an important theme in the delivery of multimedia, issues related to pedagogy were discussed in the interview with the intention of identifying related variables, and issues and understanding features of the relationship between them.
Interviews were recorded on audio tape and were transcribed in full. An initial tentative logical structure for the phenomenon under study was generated. This structure attempted to clarify relationships between the categories or variables involved in the phenomenon. A main category, sub-categories and variables/features were identified. The hierarchical structure developed earlier (and presented here) was modified based upon evidence from the interviews. The relationships identified in this exercise were a tentative representation of one view of the many possible ways these relationships could be presented. This particular view, however, had the advantage that it was derived from the phenomenon as it was understood by workers and experts in the area. The next stage involved refining the relationships described.
7.3.3 Refining relationships
In this section, the further elucidation and refinement of relationships between categories presented here is described. This was accomplished using the standard techniques of Grounded Theory as were described in the previously.
An important feature in establishing relationships between categories and sub-categories is the identification of the dimensions of a category. For example the category 'student model' had several associated sub-categories, including 'language' that had the dimension 'level' that could take values of 'low level' or 'high level'. Once dimensions had been identified in this way, then the range of possible values were described for each category and sub-category. The result of this exercise was the production of a modified, detailed and extended structure to which could be attached code notes or memos pertaining to each of the categories and sub-categories, their dimensions and ranges of values, shown here. At the same time, a detailed and richer understanding of the research phenomenon was being developed.
Memos were obtained in several ways. For example, comments made by experts were recorded and attached to relevant nodes in the structure. Transcripts of interviews and the results of text searches were also related to categories and sub-categories. In this way an understanding of the phenomenon began to emerge in terms of the relationships between categories, sub-categories and their dimensions. Memos were also used in the early stages of the analysis, to record the context of relationships between categories. For example if a category was considered to be important, consistently, in several interviews, then this fact was recorded along with possible reasons for this. Where phenomena were consistently related, details were recorded in memos and attached to related nodes. It was hoped that ideas would emerge from this phase of the study that would guide the main data acquisition stage of the project and the development of an understanding of the phenomenon. New categories were identified and added to the structure during this process and a phase of reorganisation of the structure was undertaken.
The aim of the stage of the work described above was to gain an understanding of the phenomenon in order that a core category could be identified. The core category was identified as 'the quality of learning'. All other categories were then organised around this central theme, which was intended to allow an understanding or statement of the phenomenon in a form that could be tested against data later in the study..
The work described above was undertaken as a preliminary stage of the final study. It is presented here as a linear exercise in time, for the sake of clarity. The process of relating main categories, sub-categories and variables in a structure that could be tested against data was an iterative process. Indeed modifications to the categories and their structure were made throughout the whole study, including the data collection and analysis phase, As ideas became clearer about relationships, and as more data became available, the structure of categories and sub-categories was modified accordingly. The structure of ideas produced in the open and axial coding stages of the study was instrumental in organising the implementation of the final study, which is described in the next section.