FINDINGS
The project generated a mix of quantitative data, taken from Blackboard statistics and a general University-wide survey, and qualitative data from a series of small focus groups and short videoed feedback sessions. On each module all students (approx between 20 – 60) were given access to the Blackboard materials. Approximately 10 students on each module were invited to focus group sessions and video feedback sessions were completed with all students on the Media module, though only a handful on the music programme (due to low turn out).
Given the richness of the information, numerous lines of enquiry could be established (the focus groups transcipts have been made available here should they provide supplementary data for other projects - see appendix materials in the Bibliography). However, the headline findings are relatively straightforward to explain. Put simply, there is seemingly no correlation between the online study materials and student performance. In isolation this may seem a very disappointing finding. However, when set against what students themselves have to say about the materials and how they approach their studies, it begins to make greater sense.
For example, a series of excerpts from videoed feedback with the BA Media students, who were asked direct questions about how they worked with the study pack, quickly shows a wide range of responses. Whilst the majority are very positive about the materials, students clearly have very different feelings about the value of them, how they themselves worked with them and what they felt they have achieved because of them. Some students, for example, suggest the language of the study is ‘too academic’ and they feel pressured to respond in kind. Yet, others say the opposite, that the study pack presents the topics in accessible language and help to unpack otherwise difficult concepts. Some students suggest the study pack has a negligible effect on their work (except perhaps less trips to the library), whilst others feel it has a huge impact on their work.
The following video clips demonstrate the various thoughts students have about using the Blackboard materials:
Data Analysis
Looking at the actual recorded usage of Blackboard by BA Media students, two key trends can be shown from the Blackboard statistical reports. In terms of the total access in minutes across the duration of the module (shown in Figure 1 below), which – including the Easter Break – ran for a series of 17 weeks, there is a very definite surge in usage in the week leading up to the submission of the first assessment (Week 11). Following which use drops markedly (not entirely surprisingly, since the materials on Blackboard relate specifically to the first assignment, whilst the development of the second assignment required only face to face tutorials and workshops). Usage is also at a higher point (though not at high as in Week 11) at the very start of the module and in Week 4. The latter would seem to suggest that, by Week 4, students have secure orientation of the module and have come to expect the release of reading materials, which occurred over the first 5 weeks. Week 6 is the lowest use of Blackboard prior to submission of assignment one, but this correlates with a class visit away from campus to the National Media Museum (in Bradford).
Figure 1
The use of Blackboard on the Performance (Music) module is not so straightforward to analyse. The learning materials created specifically for this testbed project on the composer Messiaen related to just three weeks of teaching (within a 14 week course). They were designed as an aid for only one point of assessment (out of three assessment points) and it was not a requirement to write about the composer covered in the materials. In fact, during focus group sessions, students noted how they felt it was better not to write about Messiaen as that might be perceived as 'copying', or not doing enough work. Also the module involved team-teaching, which adds to problems of tracking the relationship between teaching materials, modes and student work. Subsequently, it has not be possible to compare Blackboard use and module grades in the way it is possible with the Media module materials (see below). Anecdotally, there is some suggestion the materials helped. One lecturer on the module commented: 'My perception was that the Messiaen essay I marked showed a fairly good grasp of the modes and methods of composition, [more so] than the Jazz essays I marked'.
It should be noted, use of Blackboard on the module (and indeed across the Performance degree as a whole) is limited, both in terms of staff and student use. A fifth of the students enrolled on the Music-Text module (12 out of 50) show not to have accessed Blackboard throughout its entire duration. Amongst the other students, use of Blackboard is limited, with students accessing on average no more than 3 or 4 times during the term. In the teaching week prior to the lectures on Messiaen, there is was a total of just 5 users, with the average time spent online below 10 seconds. During the three weeks in which lectures were given on Messiaen, the numbers increase slightly - 23 users / 9'46" average time spent in the first week of the lectures; followed by 3 users / 3'27" average time spent; and 8 users / 1'08" average time spent. The week after the lectures were given there are 9 users showing, with an average time spent of 3'54". Figure 2 (below) shows the recorded file usage, which reveals that with the exception of the item 'Music Text Lecture 1' (lecture notes related to Week 1 of the module), students are purely accessing the specific materials related to Messiaen.
Figure 2
However the actual number of times files are accessed needs to be put into perspective, against the more general use of Blackboard on this module. The first week of lectures given on Messiaen shows the highest number of users for any week during the entire module, which as noted above was 23 users (with an average time spent of 9'46"). However, Figure 3 (below) records how the module handbook (the first item in the list) is overwhelmingly the primary interest for those accessing files, far out-stripping the number of times files are accessed of the Messiaen materials. Furthermore the lecture notes from Week 1 on the module (Music Text Lecture 1) show the next highest number of downloads. Nevetheless, given the fact that Blackboard is relatively under-used for the Performance (Music) programme, it is clear the Messiaen materials generate a great deal more traffic than would have been the case otherwise. Students are evidently willing to make use of such materials, though during the time this study was conducted there was obviously less precedent for the use of Blackboard and subsequently there is not such an immediate sense of take-up.
Figure 3
For those who teach on a regular basis, it is perhaps no real surprise to see that Blackboard usage often peaks very close to the submission deadline, ‘at the last minute’ (this was certainly evident with the Media module) . But how do students perform in their work, set against the time they actually spend using Blackboard? Looking at the statistic related to the Medid module, the answer would seem to be that there is no definite correlation. Table 1 (below) demonstrates how very high usage results in a mix of results. Looking at the 41 students enrolled on the module, the average time spent across the duration of the module is 1 hour 28 minutes, with the average grade being 56.8%. Students who spent more than 2 hours on Blackboard average slightly, but arguably insignificantly higher than average [with a grade of 59.3%]. Students who spent more than 3 hours on Blackboard actually show a lower than average grade, at 55.8%. Yet, equally, to further compound the lack of a coherent picture, those students who spent less than 30 minutes on Blackboard across the duration of the entire module average similarly to those spending 2 hours or more, with an average grade of 58.4% - again just slightly higher than the overall average. No students used Blackboard for more than 4 hours, except one whose time spent was 7 hours and 5 minutes. In this case, the student shows regular use of Blackboard in the first 5 weeks (between 3.5 Hours to 30 minutes), followed by no usage at all until week 10 and 11 (and then again no usage for the remainder of the module). The student’s grade for assignment 1 was a reasonable 62%. Yet, amongst those using Blackboard for less than 30 minutes for the entire duration of the module, there are grades recorded of 64, 66, and 68%. So, again, very little can be ascertained from these figures.
There are four students who performed extremely well, with grades over 70% (average 75.8%). Interestingly the average time these students spent using Blackboard is 54 minutes, which is significantly lower than the overall average. In fact three of the four students use Blackboard for no more than 42 minutes. The other student actually shows usage of 1 hour 38 minutes. Yet, looking deeper into the figures, this student makes no use of Blackboard until Week 10, when it is used for less than 3 minutes and then Week 11 (the week of submission for assignment 1), for 1 hour 35 minutes.
Student |
Time |
Grade |
Average Student |
1 hour 28 min |
56.8% |
Under 30 min
Student 1
Student 2
Student 3
Student 4
Student 5
Student 6
Student 7
Student 8
Student 9
Student 10
Average
Average (excluding n/s) |
29 min
5 min
18 min
2 min
14 min
4 min
24 min
28 min
23 min
15 min
-----
----- |
47.0%
48.0%
50.0%
64.0%
55.0%
Non-submission
68.0%
66.0%
68.0%
60.0%
52.6%
58.4% |
Over 2 hours
Student 1
Student 2
Student 3
Student 4
Student 5
Student 6
Average |
2 hours 09 min
2 hours 01 min
2 hours 33 min
2 hours 30 min
2 hours 12 min
2 hours 09 min
----- |
54.0%
56.0%
56.0%
65.0%
67.0%
58.0%
59.3% |
Over 3 hours
Student 1
Student 2
Student 3
Student 4
Average |
3 hours 35 min
3 hours 18 min
3 hours 37 min
3 hours 32 min
----- |
50.0%
64.0%
43.0%
66.0%
55.8% |
Over 7 hours
Student 1 |
7 hours 05 min |
62.0% |
Over 70% Grade
Student 1
Student 2
Student 3
Student 4
Average |
42 min
39 min
37 min
1 hour 38 min
54 min |
78.0%
72.0%
78.0%
75.0%
75.8% |
TABLE 1: Blackboard Use by Login time and Overall Grade (Sample: 41 BA Media Students)
The variation and extremes all point towards that ‘alchemy of learning’ that never seems to get pinned down; at least not through e-learning technologies. Further data analysis can track more closely the actual materials students are commonly accessing. Yet, short of somehow ‘tagging’ students throughout the learning process there seems inevitably a great deal we cannot quite ascertain about the learning process.
Oliver and Trigwell (2005) – in attempting to ‘redeem’ the notion of Blended Learning – adopt the variation theory of learning, which ‘is based on the idea that for learning to occur, variation must be experienced by the learner’ (22). ‘Actual blended learning,’ they suggest, ‘would involve students learning through experiencing variation in aspects of what it is that they are studying’ (22). This, they point out might be very different to the blended learning environment as intended by the instructional designer. Nevertheless they leave open the possibility that blended teaching methods may give rise to students experiencing variation in learning. As they put it: ‘Good teaching seems … to have to do with the nature of the space of learning constituted. This position is strongly supported by some results suggesting that learners are sensitive to the pattern of variation constituted in what it is they are asked to learn’ (24). Arguably, this brings Oliver and Trigwell back full circle to the fact that Blended Learning is more about teaching delivery than actual learning. They argue for ‘a closer analysis of the critical aspects of the subject matter that are in variation’, so suggesting micro-design and analysis of the elements of blended learning materials. Yet, focusing on the importance of the ‘nature of the space of learning’, a crucial point would seem to be some of the more basic concerns over of its construction and maintenance.
Sensitivities over the design and accessibly of learning materials come out very strongly in the focus group sessions, which were conducted both before and after the students used the new, dedicated learning materials (see appendix materials in the Bibliography). In the focus group sessions before the use of the materials (in Semester 1, 2007/08), the idea of Blackboard as a ‘security blanket’ is articulated by more than one student and another rather poetically refers to it as a ‘digital reception’. The overriding sense is of Blackboard as an information centre. They log in to check they have the most up-to-date information regarding lecture times and announcements. They even check to see nothing new has been added, so relieving anxieties. They get frustrated when access is slow, if the server is slow or even just if there is ‘lots of clicking’ to work through menus to get to the one thing they want or need. A recurring theme is the need for consistency across all modules. Quite at odds with many lecturers’ desires to customise the presentation of a module, to give it an identity of its own, students in the focus groups before and after using the new materials returned again and again to the idea of consistency in layout and even in part regarding content of the modules on Blackboard. One exchange with the interviewer ran as follows:
Student 1:
|
…I don’t know it it is lecturers doing it or the layout of Blackboard, but sometimes it can get confusing to know where the work is, because you don’t always know that it’s there, whether the lecturer has put it up or not
|
| Student 2: |
Same with announcements, they put it somewhere and someone else puts it somewhere else.
|
| Student 1: |
Yeah, maybe instead of a blank page for the lecturer to do something with, they could put it in sections so you know where to look. It should be structured so it’s clear …lecture notes, announcments etc.
|
| Interviewer: |
So consistency across?
|
| Student 1: |
[laughs] consistency with lecturers altogether! |
In another exchange a student explains how even a very common term such as ‘module handbook’ (which are always uploaded to Blackboard) does not always mean a great deal to some students. The student remarks: ‘the module handbook has to have more of an advertisement. Lecturers maybe have to say what a module handbook is. It needs to be shown more, and talked about more’. This kind of remark highlights again the students overriding interest in the more informational aspects of Blackboard, but also highlights the need to bridge the contents of Blackboard with actual face to face, contact time.
The focus groups conducted after the use of the new materials yield similar observations. Whilst there are interesting insights into the habits of studying, it proves very difficult to discover anything specific about when and how learning happens. Students focus predominately upon practicalities. The Media students note, for example, how they tend to prefer the study pack materials in hard-copy, yet explain the value of the online version for when they go away at weekends, so that they do not have to carry lots of paperwork, or in case they lose handouts etc. Convenience is a common theme. They like the fact that the online version of the materials enables them to quickly link to online materials and generally organises the main materials they need. Again, a key point raised is about the need of consistency of information across all modules and for layout to be simple and aesthetically pleasing to navigate.
