Data collection methods used in the study
This section describes the main methods of data collection used in the final study. Throughout the duration of the study, the software development team met regularly to discuss progress and to ensure that correct data collection procedures were being followed. In this way, data collection was systematic and its interpretation was as objective as could be made possible.
Video recordings
It was intended to make video recordings of one sample session at each location. However, only four complete video recordings were available at the end of the trial. One recording was incomplete and ignored. One other recording was not submitted. Video recordings were made by groups according to a script that was sent to each group prior to the beginning of the trial. The scripts were very general in nature and left opportunity for exploration by tutors, students and video makers. Additional information was provided in the script to assist in this exploration activity. The most useful video recordings, it is argued, are those made in consideration of the local environment and circumstances. For example the script required the video to provide some idea of the environment in the learning centre or study room. Lighting, noise levels, seating arrangements and other facilities were thought to be important. The relative importance of any aspect of the environment, however, was best left to those on site at the time.
Tutors were asked to provide video of students working on tasks together in groups and students working alone. Students were asked to log in to the system, log out and navigate, locate and undertake various activities either alone or in groups. Scripts were provided for short tasks to be recorded during the sessions. The script required recordings to be made of how tutors interacted with learners in these sessions, if indeed they did.
When videos were returned, they were viewed by an experienced rater who compared the content of the video with the script provided. A written summary report of each video was made by the rater, under headings related to the requirements of the script and Grounded Theory categories. For example, the levels of the various interactions within the session were rated as high, medium or low. Other features such as the environment, ease of task completion were rated in a similar way.
Interviews
At the end of the trial, 24 students following the multimedia course, 4 selected from each of the 6 participating colleges, volunteered to undergo a short interview. These interviews were semi-structured in that a script was provided for the interviewer, yet exploration was encouraged within the questions. If, for example, an interviewee reported that one section of the course was difficult, then the interviewer could explore this, asking questions intended to find out more information about reasons for the situation. Interview sessions were kept as short as possible and were recorded onto audio tape. Interviewers were also asked to keep a note of any points they considered important, separately. Tapes and interviewers’ notes were handed in at the conclusion of the trial for analysis.
Analysis of the notes involved making a partial transcription of comments and coding these against the categories identified previously in the Grounded Theory process of selective coding.. The process employed here had the following features
1) Causal relationships were identified where possible.
2) Reasons for differences were recorded.
3) The magnitude of effects was estimated and recorded.
4) The effect of change in circumstances and how these related to the magnitude of variables was noted.
The objective of this coding exercise was to attempt to relate the impressions of learners to the categories identified earlier as important in the phenomenon.
Pre-tests and post-tests
All participants on the multimedia course and the mirror course participated in a pre-test and post-test study. A test containing twenty calculation type questions, directly related to the material in assignment one of the course was prepared by a subject expert for use in a pre-test. The pre-test was administered immediately prior to the start of the course. For the post-test, twenty different equivalent questions were prepared. Post-tests were administered at different times after the course, depending on when the course had been completed. The timing of the post-test was controlled by the course tutor. Although there were some differences between locations, all post-tests were completed by learners within one week of finishing the course. Sample questions from the pre-test and post-test are shown here, and results are presented in the paper.
Data logging
The multimedia application was designed to record all navigation and interactions that users made whilst following the course. This was saved to a user log file located on a network file server. Data was appended to these files during subsequent sessions, providing a complete log of all interactions that took place. In addition to navigation and interaction data, the users' files held information on login and logout times, inline questions and tasks attempted and scores and time spent in sections of the course. Use of the help system was also recorded, including level of help requested as well as frequency.
Data was stored at a sufficiently high level to allow analysis of user's intention. This contrasts with other systems reported where low-level data in the form of key strokes and mouse movements precluded any attempt to understand high-level user intentions (Laws and Barber, 1993). It was possible to examine some user's files in detail and all files provided useful summary information, though the sheer volume of data made it impossible to examine all interactions in depth. Results relating to user log files are presented in the paper.
Questionnaire
All learners following the multimedia course were asked to complete a questionnaire at the end of the course. The questionnaire was intended to measure features of the learning experience in following the course. The questionnaire was delivered on-line in a multimedia format. Results were saved directly to the network and results were analysed automatically using computer software specially designed for the purpose, at the end of the study.
Tasks and questions
All learners completed questions and tasks during their progress through the course. The level of questions and tasks users were asked to perform were set by the tutor in response to diagnostic information and pre-test scores as previously described. This information was held in a file that the system used in order to determine for each user, the level of task required (user configuration file). Tutors and students were asked to keep a record of performance on tasks and questions set by the system. It was possible to compare how performance on these tasks and questions was related to the level at which they were configured for the individual.
Tasks required users to undertake an activity at the computer or away from the computer singly or in groups. Off-computer tasks rarely had an absolute right or wrong answer, though computer-based tasks were usually assigned scores.
Questions were delivered on-line and mostly had right or wrong answers. Question levels and scores for questions and tasks were recorded for all learners in the data log file. Changes in the task and question level throughout the course were also recorded..
Focus groups
A focus group is an important way of finding out information about how learners used the application and what learners and tutors felt about the application. It was hoped to perform one focus group session at each location for the study. However due to time factors, focus groups were held at only two locations involving staff, students and a facilitator. The same group facilitator was employed at both sessions performed in this study. The richness and directness of information obtained at these sessions made them extremely useful methods for collecting high quality information.A focus group tool was developed for the sessions based upon a subset of the categories developed earlier in the project. The objectives of the session were shared with the facilitator who was instructed in the use of the tool. The facilitator was encouraged to explore issues and questions raised by the groups more fully. Tutors were asked to join in discussions any place where they felt it was appropriate, but not to lead, encourage or discourage discussion. Learners who followed the multimedia delivered course were mixed with learners who followed mirror courses. A few questions were addressed specifically to multimedia course followers and some to followers of both groups. Most questions however were open to participants from both multimedia and mirror groups and to the tutors. With the agreement of participants, group sessions were recorded on video tape and analysed at the end of the study..
Staff evaluation
The views and attitudes of staff using the application with students were considered to be central to the study. Several measures were taken to elucidate staff opinion regarding the application and its use. These are described in the following paragraphs. The staff evaluation of the application centred very much on how effective the student model was in configuring learning for the individual, and also upon features that facilitated tutor involvement in using the application.
Staff diary
Staff were asked to keep a diary or record of their use of the materials. This log was used to record times and dates of sessions, number of learners present and also to record anecdotes and incidents that occurred during computer and discussion sessions, along with their interpretation. No guidance was given relating to keeping the diaries. They were a useful source of anecdotal information about the sessions and also rich sources of insights into tutors' attitudes to the applications. Any movement in tutors’ opinion during the project was also evidenced in the diaries. Three such diaries were collected at the end of the study for analysis of the content. The content of the diaries was coded to the categories and sub-categories used to describe the structure of the phenomenon earlier.
Staff interviews
Six tutors involved in the project agreed to undergo an interview at the end of the study. An interview tool was created that focused on the use of the multimedia course. It was also concerned with comparisons between the traditional delivery of the course and the multimedia method. The interview was structured as follows:
Interviews were semi-structured and exploratory in nature. The average duration of an interview was approximately one hour. Interviews were recorded on audio tape and were transcribed at the end of the study.
Staff report
Tutors provided a summary report at the end of their work. Part one was a structured part that focused on the use of multimedia, the use of the student model, learning and the management of the learning process in the course, including issues relating to the learning environment provided. Structured questions were based on the categories and variables identified as being important, earlier in the study. There was also a free or open section in the report, where tutors were able to comment in any way they liked on any part the course or overall process. Tutors were asked to relate their reports directly to their experience of delivering the course. At the end of the study, information contained in reports was collated and coded against relevant categories and sub-categories.
Expert review
Expert review is one of the primary evaluation strategies used in both formative and summative evaluation of computer applications. An expert review checklist based on Reeves and Harmon's expert review tool was provided for use by four experts in instructional design (Reeves and Harmon 1994). These were experienced teachers who had been involved in multimedia learning materials production and delivery in an FE context for several years. They each understood the objectives of the material, but none of them had been involved in the material design, or in any earlier stages of the project. Each was given an introductory briefing immediately prior to the evaluation.
Final Report
After all data had been collected, it was subjected to analysis by the project manager and coded into the categories and sub-categories determined at the preliminary stage of the project, described here. The organisation of these data was managed using QSR Nudist software. Analysis of the data was undertaken using methods and techniques described by Strauss and Corbin (1990). During this phase, a major restructuring of the categories and sub-categories was undertaken in the light of the evidence.
References
Laws, J. V. and Barber, P. J. (1995). Video Analysis in Cognitive Ergonomics: a methodological perspective. Ergonomics 32(11), 1303-1318.
Reeves, T. C. and Harmon, S. W. (1994). Systematic Evaluation Procedures for interactive Multimedia for Education and Training, Chapter 15 in: Sorel Reisman, Multimedia Computing - preparing for the 21st Century. Harrisburg (USA) and London (UK): Idea Group Publishing.
Strauss, A. and Corbin, J. (1990). Basics of qualitative research: Grounded theory procedures and techniques. Sage Publications.