OldGlobeMOOC is about to start it’s 4th week (following a week’s break for July 4th celebrations in the US), and the Week 3 assignment peer reviews are in. For me this assessment process is one of the most interesting aspects of this xMOOC. I have thought since the first MOOC in 2008 (CCK08 Connectivism and Connective Knowledge), designed and run by Stephen Downes and George Siemens, that assessment may be the sticking point for MOOCs.
In my last post , I outlined some of the difficulties that OldGlobeMOOC is experiencing with the assessment and peer review process. It seems to me, once again, but this time for an xMOOC, that if MOOCs are going to be sustainable and successful, then the assessment process has to be ‘cracked’ and meaningful.
Some MOOCs have taken the approach of restricting the number of participants who can be assessed. CCK08 did this. I think the number was 25, and FSLT12 and 13 have done this with a similar number – the idea being that a small number of participants can be assessed by a tutor. FSLT13 offers credit for this:
The course has been recently accredited (10 transferrable academic credits at level 7, postgraduate). FSLT is recognised towards the Oxford Brookes Postgraduate Certificate in Teaching in Higher Education (PCTHE) and Associate Teachers (AT) courses. (http://openbrookes.net/firststeps13/)
But these are cMOOCs.
OldGlobeMOOC has taken a different approach as I described in my last post and I understand from other participants that this is similar to a number of other Coursera MOOCs. For me this my first xMOOC, but it is not for quite a few OldGlobeMOOC participants, who have taken numerous Coursera courses and in the forums have shared their experience of the peer review process.
I will add my experience to the mix, and just so you know what we are talking about here are links to my assignments with their peer reviews.
If you read these, you will see that the assignments are not very different in their style and level to my blog posts, i.e. they are not academic pieces of work – rather discussion pieces or personal reflection. And judging by the assignments I have reviewed, other participants’ assignments are of a similar level.
Which brings me to the review process, which I reflected on in my last post, but will add a few things here.
- The idea is that each participant submits an assignment and peer reviews five assignments for each week, which I have done. If this is not done, i.e. the peer review, then a 20% penalty is incurred.
All students wishing to obtain a Statement of Accomplishment must achieve 7 out of 12 points and submit 5 peer reviews each week. If a student fails to complete the 5 peer reviews, that week’s assignment will incur a 20% penalty.
Despite the fact that I definitely submitted five peer reviews for Assignment 3, I received a 20% penalty and therefore scored 1.6 instead of 2. It’s very easy to know that you have completed the 5 peer reviews, by the way the Coursera system takes you through the 5 assignments allocated for review; and the system confirms for you at the end of the process that you have submitted 5 – so I know that I did. So there’s been a blip in the system somewhere. It’s not a big deal for me, as I’m only doing this to experience the process and because I like the assignments and find the discussions interesting. I am not doing the course for the Certificate – but I do wonder how a blip in the system affects people who are really keen to receive a Statement of Accomplishment.
- There is no guarantee that you will receive 5 peer reviews. I received five in Week 1, three in Week 2 and four in Week 3. There has been some discussion in the forums about how this might affect the overall system and whether or not you have to review more than 5 assignments to receive 5 reviews.
- I have no complaints about the quality of most of the peer reviews and so far no one has given me a score of less than 2 – but this peer review for Assignment 3 is indicative of how the game can be played to ensure that you get a Certificate. It made me smile 🙂
peer 2 → I’m headed for an airplane so don’t have time to review, and I won’t be back until after evaluation time ends so I’m just giving everyone a 2.
Aside from this here are some further reflections. The OldGlobeMOOC is a great experience in terms of the diversity of participants. Unfortunately the younger participants, in their teens, who signed up, seem to have fallen out of the discussion forums. This does not mean that they are no longer participating through observation and reading – it’s difficult to know. But I have wondered how an 11 year old might review the assignment of an academic Professor, or how an academic Professor might respond to a learner with special needs, or a very young participant, or someone whose first language is not English, and so on. The assignment submission is anonymous. Do these differences have implications for the equity of the peer review process?
Despite all this I am finding OldGlobeMOOC a fascinating and enjoyable experience and am looking forward to the start of Week 4.
Jenny Connected 
Hi Jenny, do think you might be attracted to things that are too difficult for mere humans to resolve? Handicapping the ponies might be easier:-)
The major problem here seems to be eliminating the variability of human bias in order to come up with a “true” assessment. Problem two (rule one: number everything) MOOCs don’t per-sort for experience, native ability or culturally predictable responses so we have no initial numbers to test against. I won’t go on…
In his book “13 things that don’t make sense” Michael Brooks (who wisely decided to not discuss grading in MOOCs quotes Isaac Asimov:
‘The most exciting phrase to hear in science, the one that heralds the most discoveries, is not “Eureka!,” but “That’s funny…”‘
I have to read about the FSLT decision. Within a close group of known people with known qualifications already to their credit there may be almost no variables to measure as they already have proven and acceptable skills–and it still seems difficult.
Back to the ponies. Or maybe not. Predictions of performance are not guarantees of actual performance so maybe this is a human sport question?
Thanks for the post.
Hi Jenny, Thanks for this rich sharing. Do you think there are any gender differences in this perception on aging? Would daughters more likely to have the strong affection to their mum than their dad? Would story telling/sharing be more interesting than academic telling when one ages?
Hi Scott – what interesting comments. Thanks! If I understand you correctly you are saying that assessment will always involve human bias and therefore will never be equitable. Yes – I suppose this is why I have spent so much time in my working life writing and continually refining marking criteria in the light of experience and then moderating assignment marks with others. My experience is that it can take a year or two to get the writing of the assignment and the criteria right. By right, I mean that it would be a fair test of student performance and that a different set of markers would each interpret the criteria and mark it similarly. So OldGlobe is a new experience for me in terms of the diversity.
Hi John, thanks for your visit. I don’t know if there are gender differences in perceptions of ageing. I don’t see why there should be, although there are – as we have been told in OldGlobe – biological differences between men and women in relation to ageing, which you would expect. And I don’t think there’s any evidence that daughters have a stronger affection for their mothers than their fathers, is there? In terms of story/telling – I think it’s important in OldGlobe because the course has been deliberately designed to attract a wide diversity of people. It is not an academic course, which it could be if, for example, it dealt heavily with the science of ageing. I don’t think it’s necessarily more interesting than an academic piece of work – at any age – it just depends on what the purpose is. But stories are often easier to read than an academic piece of work, which can be full of jargon!
Thanks Jenny for your insightful responses. It is interesting to have peer assessment on such topics. I am wondering if the peer assessment by peers are assessing based on whether the stories are resonating to their personal perceptions. Assessment may also relate to the degree of relatedness to ageing. After reading through those peer reviews, I have a few questions in mind. 1. Since the experience relates to the writer’s story and opinions, would the assessment relate more on how the story and opinions on ageing (emotional response) or the story structure and written expression (clarity and coherence)? 2. How to ensure a fair assessment under such blind assessment arrangement? As you mentioned, it may likely that a novice (11 old teenager) assessing an academic professor, or a veteran or geek. 3. How would one appeal to any “inappropriate assessment or review”? Are there any appeal mechanism or procedures in place? Should there be one, as you mentioned that you have been penalized though you have submitted all five reviews on time. I ask these for the sake of discourse, not as a challenge to the authority, or a “complaint”. I think it is important to ensure any assessment be viewed as open, transparent, and are based on the honest feedback, rather than hasty response. Though there might be good reasons why some people just give others a “2”, I just wonder if this would be “acceptable” in a peer assessment system. What do you think would be a better way to ensure a fairer assessment with xMOOC? John
Hi John – thanks for all these questions. I don’t think OldGlobe is typical of xMOOCs. In fact Sarah Kagan has called it a cMOOC and it does have a lot of the characteristics of a cMOOC – it is being run mostly on the Coursera platform, but there is also a Facebook group and Twitter, but I’m not aware of other bloggers. It is open in the sense that access is open, the assessment is open and there is very little course content, e.g. there are weekly videos, but no readings, and discussion can follow any path the participants would like. The course has great diversity of participation and discussion, and participants do have a lot of autonomy if they are not bothered about assessment. So a lot more like a cMOOC than an xMOOC. And judging from comments in the forums, the assessment in OldGlobe has been different.
The nature of the assessment questions does lead to storytelling and for me this is a strength because it means that anyone from any background can engage with the assessment – but as I have described it does lead to some difficulties with the peer review system. I think it’s possible for reviewers to take a number of approaches to the peer assessment – but this is how peer reviewers are asked to respond:
Please type your 100-250 word peer assessment below.
What do you think about this participant’s portfolio item choice to answer this question of the week?
How does this participant’s perspective differ from your point of view?
How is your point of view similar?
So in OldGlobe the criteria for peer assessment are quite loose and non-academic. So I don’t think it’s possible to think in terms of fairness, as we normally understand it in academic assessment, but rather in terms of empathy, kindness and respect. Reviewers are told:
An assignment only receives a zero if it is incomplete or did not follow the guidelines set for this week. Don’t be afraid to be generous!
And yes it is possible to appeal, by putting a question in the Help Forum – but participants have to have a genuine cause for appeal. It’s no good saying ‘I didn’t have time to do the peer reviews this week’ – hence the peer review I got which said
peer 2 → I’m headed for an airplane so don’t have time to review, and I won’t be back until after evaluation time ends so I’m just giving everyone a 2.
🙂
Thanks Jenny for your wonderful response. I have written a long response post http://suifaijohnmak.wordpress.com/2013/07/11/what-is-most-important-when-conversing-online/ and if you don’t mind I could post it here with details. John