Feeds:
Posts
Comments

Posts Tagged ‘peer assessment’

I am now, 3 weeks into ModPo, very aware of the differences between the original cMOOCs (e.g. CCK08 – the very first MOOC run by Stephen Downes and George Siemens) and xMOOCs – and I think it relates to this slide that Stephen Downes recently talked us through at the ALT-C Conference

SD ALT-C slideshareWhat are cultures of Learning – http://www.slideshare.net/Downes/2013-09-12-altc

xMOOCs might be either A) Centralised or B) Decentralised but they are not C) Distributed, i.e. not in the same sense that CCK08 and subsequent MOOCs such as Change 11, run by Downes and Siemens, were.  Although xMOOCs such as ModPo do have a Twitter stream and a Facebook group, they do not encourage people to find and create their own discussion groups in locations of their choice, as the original cMOOCs did.

ModPo for me is very centralized – with the centre being Al Filreis and to a certain extent his TAs. No Al Filreis – no ModPo. He is the ‘sage on the stage’. And it seems to be working well for most people. Al is charismatic. There are hundreds of discussion threads and Al Filreis and his team of TAs are very visible in there. They must be exhausted.

I am loving the poetry in ModPo – all new to me – and the video discussions which model and demonstrate how to close read these poems are very engaging. Even within one week I felt I had learned a lot, not least that some poets resonate and others do not.

But, despite this, there are elements of ModPo that I find disturbing and they are mostly to do with the assessment process, which on a professional level (as an educator) have concerned me.

I have already mentioned in a previous post  that I can’t see any value in having to post to discussion forums as an assessment requirement. Now there are three other points related to assessment that I find troubling.

1. The assessment criteria (peer review instructions) were not posted before people submitted their assignments and this does make a difference – because, for example, the reviewers were asked to judge whether assignment writers had understood Emily Dickinson’s use of dashes in her poetry. Whilst dashes were discussed at length in the videos, they were not mentioned in the assignment writing guidance. Participants/students should always know the criteria they are being assessed against.

2. The fact that all the assignments, once they have received one peer review, are automatically posted to one of the Coursera forums, i.e. all the 30 000 participants can see the submitted assignment if they have the time and energy to wade through the 75 (at the last count) that have automatically been posted.

Assignment writers were not asked whether they would be willing for this to happen.  In an Announcement to the class they were told that “This enables everyone to participate, at least a little bit, in the reading and reviewing of essays” – but frankly all it does it load even more discussion threads to the forums, which are already overloaded and – more significantly – takes the control and ownership of the assignment and learning process out of the hands of the learner more than is necessary.

For me a successful adult learning process relies on learners having as much autonomy as possible (another principle from the early cMOOCs, but also one backed up by research into adult learning). All it needed was consent from the assignment writer.

3. The third point is the worst. A participant has been publicly named and shamed for plagiarism in the assignment submission forum mentioned above. Her assignment was automatically posted as explained – so she had no choice over the matter. The reviewer had not noticed the plagiarism (a section copied from Wikipedia) – but to the title of her post has been added (Note from Al: this essay has been plagiarized).  At the beginning of the course there was a stern warning in the initial announcement about plagiarism – although I can’t find it now – and participants submitting assignments are asked to tick a box saying that the work is their own.

It could be argued that public naming and shaming of a participant serves as a warning to all other participants – but I think it is cruel and ultimately destructive. I know from experience that foreign students often have difficulty understanding what plagiarism means and as far as I can see there is no advice on the site about citing sources. However you look at it, I don’t believe a student should ever be publicly named and shamed. She should have been contacted privately by email. That would have been enough – especially since she may not get the certificate anyhow, since she hasn’t made any discussion forum posts. Did anyone bother to check?

These exhibitions of power, control and centralization are a long way off the original conception of MOOCs.

Read Full Post »

OldGlobeMOOC is about to start it’s 4th week (following a week’s break for July 4th celebrations in the US), and the Week 3 assignment peer reviews are in. For me this assessment process is one of the most interesting aspects of this xMOOC. I have thought since the first MOOC in 2008 (CCK08 Connectivism and Connective Knowledge), designed and run by Stephen Downes and George Siemens, that assessment may be the sticking point for MOOCs.

In my last post , I outlined some of the difficulties that OldGlobeMOOC is experiencing with the assessment and peer review process. It seems to me, once again, but this time for an xMOOC, that if MOOCs are going to be sustainable and successful, then the assessment process has to be ‘cracked’ and meaningful.

Some MOOCs have taken the approach of restricting the number of participants who can be assessed. CCK08 did this. I think the number was 25, and FSLT12 and 13 have done this with a similar number – the idea being that  a small number of participants can be assessed by a tutor. FSLT13 offers credit for this:

The course has been recently accredited (10 transferrable academic credits at level 7, postgraduate). FSLT is recognised towards the Oxford Brookes Postgraduate Certificate in Teaching in Higher Education (PCTHE) and Associate Teachers (AT) courses. (http://openbrookes.net/firststeps13/)

But these are cMOOCs.

OldGlobeMOOC has taken a different approach as I described in my last post and I understand from other participants that this is similar to a number of other Coursera MOOCs.  For me this my first xMOOC, but it is not for quite a few OldGlobeMOOC participants, who have taken numerous Coursera courses and in the forums have shared their experience of the peer review process.

I will add my experience to the mix, and just so you know what we are talking about here are links to my assignments with their peer reviews.

Assignment 1 with peer review

Assignment 2 with peer review

Assignment 3 with peer review

If you read these, you will see that the assignments are not very different in their style and level to my blog posts, i.e. they are not academic pieces of work  – rather discussion pieces or personal reflection. And judging by the assignments I have reviewed, other participants’ assignments are of a similar level.

Which brings me to the review process, which I reflected on in my last post, but will add a few things here.

  • The idea is that each participant submits an assignment and peer reviews five assignments for each week, which I have done. If this is not done, i.e. the peer review, then a 20% penalty is incurred.

All students wishing to obtain a Statement of Accomplishment must achieve 7 out of 12 points and submit 5 peer reviews each week. If a student fails to complete the 5 peer reviews, that week’s assignment will incur a 20% penalty.

Despite the fact that I definitely submitted five peer reviews for Assignment 3, I received a 20% penalty and therefore scored 1.6 instead of 2. It’s very easy to know that you have completed the 5 peer reviews, by the way the Coursera system takes you through the 5 assignments allocated for review; and the system confirms for you at the end of the process that you have submitted 5 – so I know that I did. So there’s been a blip in the system somewhere. It’s not a big deal for me, as I’m only doing this to experience the process and because I like the assignments and find the discussions interesting. I am not doing the course for the Certificate – but I do wonder how a blip in the system affects people who are really keen to receive a Statement of Accomplishment.

  • There is no guarantee that you will receive 5 peer reviews. I received five in Week 1, three in Week 2 and four in Week 3. There has been some discussion in the forums about how this might affect the overall system and whether or not you have to review more than 5 assignments to receive 5 reviews.
  • I have no complaints about the quality of most of the peer reviews and so far no one has given me a score of less than 2 – but this peer review for Assignment 3 is indicative of how the game can be played to ensure that you get a Certificate. It made me smile 🙂

peer 2 I’m headed for an airplane so don’t have time to review, and I won’t be back until after evaluation time ends so I’m just giving everyone a 2. 

Aside from this here are some further reflections. The OldGlobeMOOC is a great experience in terms of the diversity of participants. Unfortunately the younger participants, in their teens, who signed up, seem to have fallen out of the discussion forums. This does not mean that they are no longer participating through observation and reading – it’s difficult to know. But I have wondered how an 11 year old might review the assignment of an academic Professor, or how an academic Professor might respond to a learner with special needs, or a very young participant, or someone whose first language is not English, and so on. The assignment submission is anonymous. Do these differences have implications for the equity of the peer review process?

Despite all this I am finding OldGlobeMOOC a fascinating and enjoyable experience and am looking forward to the start of Week 4.

 

Read Full Post »

I am finding OldGlobeMOOC a fascinating experience – quite unlike any other MOOC I have participated in, and my first xMOOC. For me one of the best things about OldGlobe is the diversity of the participant group. All the other MOOCs I have participated in have attracted groups in which similarities can easily be seen, i.e. mostly from academic backgrounds or interested in e-learning technologies.

But OldGlobeMOOC is truly diverse. It has attracted a huge age range from 11 to upper 80s, and people from all continents apart from Antarctica, although as you would expect the American participant presence is, I think, dominant – it would be interesting to see the analytics. But more importantly, it has attracted people from all walks of life and from very different education backgrounds. We have people very knowledgeable about the discipline of gerontology and related health and medical issues, but also very many people who have no subject related background other than we are all ageing and therefore all have a point of view.

This diversity is great. The stories being told in OldGlobe are richly diverse and a privilege to read.

But this diversity brings its own problems in relation to peer assessment of the OldGlobe assignments.

I mentioned in a previous post how the assignments are open to a range of approaches and creativity. Originally I hadn’t intended to complete the assignments, but I was drawn in by the energy and enthusiasm of the OldGlobe community and I’m glad I was.  I have seen examples of participants posting videos of themselves speaking about the assignment question, writing about their own personal experiences, drawing on academic literature, posting links to videos, websites and photos, and drawing on literature and poetry to illustrate their response. A requirement of the assignment process is peer review of 5 assignments and I have reviewed some wonderful submissions. The first assignment I reviewed used this song in response to the question ‘What is ageing?”

Fantastic!

But the peer review process is where the wonderful diversity in OldGlobeMOOC creates problems. We have the whole continuum of people from those who have no experience of peer review to people who have worked in Higher Education for years and are very experienced in assessing student assignments and reviewing research articles.

This is a dilemma for OldGlobe, because some participants are getting a bit of a rough deal in terms of their feedback, despite the OldGlobe team urging participants to be generous with their feedback and scoring. For example, one participant has been accused of plagiarism for his/her original submission, another has been accused of plagiarism for using an essay site, even though this was cited as a source, another has received the feedback ‘I don’t get it’ and a mark of zero for an academic piece of work, others have received one line or less in their feedback. It all seems a bit of a lottery. So diversity brings both advantages and disadvantages. How might this dilemma be overcome?

I applaud the OldGlobe team for designing the MOOC to attract such a diversity of participants and for designing the assignment tasks such as they can be completed by anyone from any background. We are all getting older. We all live in a society where we can see people getting older. We all know old people. We all have some thoughts and perspectives about the ageing society. Even an 11 year old can say something about their grandparents and an 89 year old has a wealth of experience to share. We can all read other people’s discussion forum posts and assignments and have a response.

OldGlobe has asked us to answer the following questions when responding

Please type your 100-250 word peer assessment below.

What do you think about this participant’s portfolio item choice to answer this question of the week?
How does this participant’s perspective differ from your point of view?
How is your point of view similar?

I think the whole age range could do this, provided they could understand the submission.

The problem comes with the scoring. Here are the instructions:

Here is the rubric for the assignment. You’ll use this as a guide to complete your own work in the Submission Phase, and as a guide for grading your peers in the Evaluation Phase:

2 points

Assignment is completed with a clear commentary of 250 – 500 words that pertains to the question of the week

1 point

Assignment is completed with some commentary that may or may not pertain to the question of the week

0 points

Assignment is missing an item, a commentary, or both

It seems straightforward, but given the diversity of course participants is so open to misinterpretation or overly subjective interpretation, which some participants seem to have experienced.

For me, the peer review process on the first assignment has been positive. I have really enjoyed reading the submissions I was sent to review. They were not all academic pieces of work, but they had all been thought about and I appreciated the open sharing of experience however articulate or inarticulate that might be.

This MOOC is not for credit. Participants will simply get at Statement of Achievement.

All students wishing to obtain a Statement of Accomplishment must achieve 7 out of 12 points and submit 5 peer reviews each week. If a student fails to complete the 5 peer reviews, that week’s assignment will incur a 20% penalty.

This makes me wonder if we need points at all. I think the feedback is very valuable and I would prefer to call it feedback than peer review, which I think puts the emphasis in a different place.

But perhaps we don’t need the points. Perhaps it’s enough for participants to complete the assignments and 5 peer reviews to receive the Statement of Accomplishment. Of course, using this system, some people will receive the Statement of Accomplishment for exceptional work and some for simply submitting ‘any old thing’. But does that really matter, given that this course is not for Higher Education credit?

If it came to a choice between diversity and peer review – I would go for diversity, and trust that people are participating in the learning environment just as much as they want to and need to for their own purposes.

There is so much of interest in OldGlobeMOOC. As an educator myself I find this tension beween diversity and peer assessment very interesting, quite apart from the fascinating discussions about ageing.

Read Full Post »