3. Method and results

This chapter describes which method was used during the research. First, the participants are described. Next, it describes the research questions, the research design and a justification for this.  Thirdly, the materials and apparatus (the workshops and the moblog systems) are described. Finally, it describes the procedure that was used.

Participants

The research consisted of two classes of full-time students, taking the ‘Training & Human Development’ bachelor degree course at HAN University, Nijmegen, the Netherlands. Participants were all at the same level in their studies and had met the same pre-requisites. Group A consisted of 12 students (6 male and 6 female) and group B consisted of 13 students (5 male and 8 female). The mean age of the students was 20.

Research question

• Does the use of moblogging have any effect on the quality of reflection with first-year students of the Training & Human Development study programme?

Research design

The design of this research can be qualified as pre-experimental research, design # 3: comparison of two existing groups, with post-test only. ( Baarda & de Goede, 2001, p. 115). In this design, there is a experimental group (the group that receives the intervention) and a control group (the group which does not receive any intervention). Using a control group allows the researcher to compare the results with a group who did not receive the intervention (in this case, the workshop Moblogging), making sure any differences can be attributed to that intervention.

According to Harinck (2007), an experiment is especially suitable for measuring whether a certain method or approach has proven results. In order to find out whether the use of moblogging has any effect on the quality of reflection, an experiment seems very suitable.

The reason for chosing a pre-experimental research design as opposed to a ‘true’ experimental design are the following:

Randomization. In a true experimental design, particpants of the two groups should be appointed to the groups randomly. The group that receives the intervention should also be selected at random (Baarde & de Goede, 2001).
This proved to be difficult to achieve in the Training & Human Development study programme. Courses and their pedagogy have been designed for classes to stay together as a group, e.g. because some learning activities required the students to know eachother. Composing two groups based on randomization would mean the two classes would have to be mixed. The results of this are unpredictable; students may for example refuse to partcipate in the research, or classes would have been different to schedule. According to Cohen, Manion & Morrison (2003), bias may be introduced as a result of differences in the selection of subjects for the comparison groups or when intact classes are employed as experimental or control groups.  In this case, the choice was to keep the two groups intact, because the two groups are very similar with regard to age and history, and proved to have a similar level of quality in reflection (shown by previous assessments).

There are some additional risks to be considered when choosing a pre-experimental research design:

• Baarde & de Goede (2001) argue that there is a risk of grow- effects in this research design. This means that there is a possibility that students become better reflectors over time (not because of the moblogging intervention). The hazard of grow-effects are limited, because the experiment takes a relatively short time (8 weeks). There is however a danger: the 8 weeks are at the end of the academic year, a period of time when students traditionally show some weariness and fatigue with their studies.
• Cohen, Manion & Morrison (2003) argue that there may be the risk of experimental mortality, .i.e the loss of subjects through drop-out. This happened in both class 1 A and 1B during the experiment; see also results.

The design of the experiment can be visualized as follows (click on the image to enlarge):

 

Class B was chosen as the experimental group. This group of students was asked to log any reflective experiences with the use of moblogging. They were not allowed to use standard means like paper notes. Every reflection that could be useful for the reflective analysis report was to be recorded with the use of their mobile phone and their moblog systems.

Class A was chosen as the control group. This group would continue their way of recording reflections, i.e. with the use of pen and paper, and a notebook.

An educational course unit (‘Begeleiden van Lerenden 1’) was chosen that required the students to write a reflective analysis report. During this educational course unit, group A was to write their reflective logs in the way they did in previous course units (i.e. by typing logs on their desktop PCs or by using a notebook) and group B would make notes of their experiences by using their mobile phones and several moblog services. Students of class 1B  would post their reflective experiences to their moblogs by using text messaging and / or picture messaging.

At the end of the course, both groups would use their notes to process them into a reflective analysis report, used for assessment. 

After all students had submitted their reflective analysis reports, The reports of both groups would then be compared using the assessment criteria which are used normally. The criteria for the evaluation form have been developed by the study programme, and are discussed in the theoretical framework. The evaluation form can be found here.

Materials and apparatus

Prior to the course, both groups received a training in reflection, to make sure both groups would know what was expected of them in terms of writing a reflective analysis report. The workshop built further upon what they had already learned about reflecting in previous courses.

This workshop lasted three hours and contained the following items:

- what is ‘good’ reflection? This item also discusses the indicators that would be used in the evaluation form at the end of the course.

- Models for reflection: A look at Korthagen’s model for reflection.

Click here for the PowerPoint used in this workshop, and here for the handout provided to the students.

Workshop moblogging for class 1b

In addition, class 1B received a three-hour workshop on moblogging. The workshop had three aims:

• To discuss the use of weblogs and moblogs for reflection.
  If you want your students to use weblogs and moblogs during a course, it is best to show them a number of good practices (Ferdig & Kaye, 2004). That was done during this section of the workshop.
• To make sure students knew how to use their cellphones in order to upload videos and images to the web.
• To make sure their devices had been enabled technically for moblogging.

Click here for the handout used in this workshop.

Course ‘Begeleiden van Lerenden 1’

During the course which was chosen for the experiment, students used their moblogs to gather reflections. During the course, three additional sessions were held to discuss how the experiment was going,  to find out if the students were encountering any problems and to keep the students motivated.

Results

After the course, the students used their moblogs to write the reflective analysis reports for their portfolios.  Of class 1A, 5 out of 12 students managed to submit their reflective analysis report before the deadline (three students dropped out during the experiment; the others simply failed to complete their portfolios before the deadline) . Of class 1b, 10 out of 13 managed to submit their reflective analysis report before the deadline.

The reflective analysis reports were then assessed by their teachers. To reduce intersubjectivity, the criteria (which are abstract and difficult to operationalize), the criteria were discussed by the teachers beforehand and assessments were compared and discussed afterwards. The researcher gathered both the reflective analysis reports and the assessments. The researcher assessed the reflective analysis reports, and then compared his assessment with the assessments of other teachers. The conclusion were mostly the same.

A comparison of the assessment criteria for both classes can be found in the table below:

Assessment criteria:	Class 1A (n=5):	Class 1B (n=10):
Describes reflection according to a cyclic model (E.g. Korthagen or Kolb)	60%	50%
Criticizes their own thinking and action when neccesary	60%	70%
Uses peer feedback proactively	80%	70%
Recognizes their own learning needs and can name these	80%	80%
Arranges suitable learning interventions for themselves.	60%	60%

These numbers do not show any significant differences, even though it can be argued that only a relatively small number of students of 1A managed to submit their reflective analysis report before the deadline. Based on the theories discussed in the theoretical framework, one would expect the percentages for class 1 B to be much higher.

A group interview with class 1 B was then scheduled to find out why the numbers did not differ very much from class 1 B. The interview and its outcomes is discussed in the following chapter ‘Qualitative research’.