#PLENK2010 More thoughts about evaluation and assessment

I have to admit to being confused about exactly what the focus of this week of the PLENK course has been about. The title of the week has been – ‘Evaluating Learning in PLE/Ns’. But the language used has been very confusing, starting with the words assessment and evaluation being used interchangeably – which I was relieved to see from Heli’s comment – I am not alone in being concerned about. And then in today’s Elluminate session there was a lot of ‘talk’ about learning outcomes, when sometimes it seemed to me that what was being talked about was learning objectives.

I have also not been clear about whether the focus is on evaluating PLEs/PLNs and therefore this PLENK as a learning environment, or on individual participants’ learning within this environment, which are two different things and would require different processes despite being linked.

Then there is the confusion about whether we are talking about people assessing their own learning within the PLENK course or whether we are trying to decide whether it is possible to assess people in these environments – again two different processes.

The question was asked ‘How do I know that I am making progress/have learned something’? It would be interesting to collect people’s thoughts about that. My own quick response to this would be that I often don’t know until some time after the event and recognising that I made progress/learned something is very context dependent and therefore for me there is no one answer to this question.

Also for me an important question from this week is whether or not personal learning environments enhance learning – its interesting to consider how this might be measured.

Another interesting question (not related directly to the assessment issue) which was raised in today’s Elluminate session is ‘How do you get the balance right between providing course structure and allowing students the type of freedom that is characteristic of a MOOC.’ I loved the ‘Roots and Wings’ metaphor that someone posted – sorry not to have noted the name to be able to attribute this correctly.

And then Heli has thrown down the gauntlet in her comment in response to my last blog post

Almost all questions are open .. are we afraid of measuring because we want to be up-to-date and postmodern or ..?

Now there’s an interesting question. My own feeling is that there does seem to be a tendency to move away from measurement (e.g. in the UK there has been resistance by teachers of young children to using standardised tests)  – mainly because it’s so difficult to get the measures correct. This was implicit in the You Tube video link that was posted in the Elluminate session tonight – RSA Animate – Changing Education Paradigms – but I don’t think this is because we are afraid – more because traditional modes of assessment just don’t seem to fit with learning that takes place in distributed personal learning networks – particularly if the learning is taking place in a MOOC.

Finally, my own thinking about assessment of learners in traditional settings has been influenced by Gibbs and Simpson’s article – Gibbs, G. & Simpson, C. (2004) ‘Conditions under which assessment supports students’ learning’, Learning and Teaching in Higher Education, 1, 3- 31 http://resources.glos.ac.uk/shareddata/dms/2B70988BBCD42A03949CB4F3CB78A516.pdf

… but if learning is to take place in distributed networks and we want this learning to be accredited then how do we apply Gibbs and Simpson’s advice? Do we need to – or should we be trying to think outside the box and swim against the tide?

#PLENK2010 Evaluation and assessment

I am skating around the edges of this MOOC. I am not unduly worried about this. I have been involved in MOOCs before and know that you need a lot of time to be involved in and make sense of the chaotic mess that is nearer the ‘heart’ of it – and currently I don’t have that time – or more likely, my priorities are elsewhere.

However, ever since CCK08 I have been thinking about the problematic issue of assessment when learning in a MOOC – (see for example the final paragraph of this post – A Pause for Thought – in October 2008)  – so Helene Fournier’s presentation on Elluminate tonight attracted my attention (recording not yet posted – but eventually it will be posted here – http://ple.elg.ca/course/moodle/mod/wiki/view.php?id=60&page=Recordings )

I will have to listen to the recording again – because I know there is lots of thought provoking stuff in there – but I was distracted through a lot of it by the insistent thought in my head that there is a distinction between assessment and evaluation which Helene said she used interchangeably. I’m not usually pedantic – but assessment has such an impact on so many people’s lives that I think it is important to ensure that we are all talking about the same thing ( as much as is possible).

Many thanks to Viplav Baxi (who I ‘know’ from CCK08 :-))  for posting this link – http://www.adprima.com/measurement.htm in the chat room and which I think is really helpful in making the distinctions between measurement, assessment and evaluation – and the further link within this link – http://www.edpsycinteractive.org/topics/intro/sciknow.html – is also helpful.

In my job as an Education Consultant – I don’t use measurement (according to the definitions in these links), but I do assess – in that I assess students’ work against given learning objectives and criteria – and even though I do this I am very aware of how very difficult this is and the many associated contradictions. For example – as a tutor – what do you do when you know that the students’ work is better/ more creative and innovative than the learning objectives set by the course?   A dilemma for the tutor!

I also evaluate – but usually I don’t do this myself but ask students to – for example – evaluate the course, or my teaching – or I evaluate someone else’s course/teaching.  This usually takes the form of a questionnaire or interviews. The questionnaire is more likely to tie me down to specific criteria but the interviews less likely. For evaluations I am not judging individual responses but looking at responses as a whole. For assessment I am thinking about individuals. I am also aware of the problems of evaluation. How do you know that the right questions have been asked or that the respondents have interpreted your questions as you intended? Not at all straightforward.

I am still not sure that I have these distinctions or my understanding of these terms completely clear in my head, but like the author of the first link – Dr. Bob Kizlik – I think they are important. I was not even clear about exactly what it was that people are trying to assess/evaluate in a MOOC/PLENK.  According to Dave Cormier in a MOOC we don’t know what the learning is supposed to be. If this is the case, then what are we supposed to be assessing? Is assessment even relevant in a MOOC, PLE/PLN?

Stephen ‘said’ in the chat room –

everybody wants me to be focused (and especially focused on outcomes). But I am the antithesis of focus

If this is what MOOCs are about – and one or two people in the chat room said that they thrive on chaos – then is it worth thinking about assessment at all in these circumstances?

A fascinating subject and I am still thinking/pondering/questioning 🙂

#PLENK2010 – Open courses and the ‘Granny Cloud’ phenomenon

Thanks to Alec Couros for further information about his open course – EC&I 831: Social Media & Open Education – and for the link to his call for mentors for this course which made for very interesting reading – and has had me reflecting on the question of how to scaffold open courses further – or whether we need to scaffold them at all.

Voluntary mentoring of online courses is not a new idea. John Smith, Bron Stuckey and Etienne Wenger always use mentors on their Foundations of Communities of Practice online course in CPsquare. This is not an open course, but mentors work voluntarily, having first participated in the course themselves. I was privileged to be a mentor myself for one the courses. As in Alec’s course, the mentor plays a different role to the course convenor. In CPsquare this is to support participants in finding their way in the course, to support them in their learning and interaction, to promote and encourage discussion and to support the course facilitators in their management of the course. This sounds similar to what happens in Alec’s course – the difference being that in the CPsquare course all mentors are already known to the course convenors and have been participants on the course for which they are a mentor.

Alec’s idea of a ‘call for mentors’ also struck me as very similar to Sugata Mitra’s ‘granny cloud’. Mitra is renowned for his ‘Hole -in- the -Wall’ experiments in India, which resulted in evidence that children can organise their own learning and teach each other – see http://www.ted.com/talks/sugata_mitra_the_child_driven_education.html for details. However, there was also evidence that the experiments did not always work – see Arora’s work and this blog post for an introduction. Following this critique by Arora, Mitra decided that children’s interest and motivation to learn in the absence of a teacher would be more likely if they were supported by what he has called a ‘granny cloud’. So he recruited hundreds of British grandmothers who are willing to voluntarily connect with the children online and answer their questions – a very similar idea to Alec Couros’ call for mentors.

These ‘experiments’ in learning with minimum intervention from a teacher raise all sorts of complex questions about the role of teaching, both in traditional settings and in open settings. The one that strikes me as being important is how the quality of mentoring is controlled. What sorts of checks do we need to have in place to ensure the safety of learners and that they get a ‘fair’ deal. Under what circumstances would it be worse to be ‘mentored’ than to be left to manage your learning on your own?

I need to read around a bit more (Mitra, Arora and Couros) and see whether these questions have already been addressed.

#PLENK2010 Scaffolding Open Courses

I have just attended the Friday Elluminate session of the Plenk2010 course (will post recording as soon as it is available).

I have been out of touch for more than a week trying to meet research and work deadlines and so it was great to be able to attend this session and also that the session focussed on a topic which is of great interest to me. The question that I honed in on was around the role of educators in ‘massive/large’ open courses. I have missed more than one week’s content of the course, so I am unsure of the context in which this question arose, but since I have participated in at least one other large open course – notably CCK08 – I do have some thoughts about this.

It hit me today that in a MOOC, the massiveness is not a given in that for an open course the facilitators/moderators/tutors (whatever you wish to call them) can have no idea of how many people the course will attract. CCK08 was massive – more than 2000 people signed up. The critical Literacies course was less ‘massive’ in terms of numbers and definitely fairly small by the end. PLENK2010 is massive – more than 1000 participants – many of whom are very active.

But the ‘openness’ is a given. We can attend for ‘free’ – but – we are expected to work autonomously and openly ‘share’ our resources and thinking in a very diverse group. The expectation is that thinking and learning processes will be transparent – but despite these expectations, we can still choose not to – making the whole experience very flexible. This flexibility can be experienced as a double-edged sword.

We found in our research following CCK08 that the more massive the open course the more difficult it becomes to function effectively as autonomous independent learners and the more difficult it is to adhere to the expectation of openness – Also, the more likely it is that participants will congregate in small groups and therefore be liable to ‘group think’ – another problem that was mentioned today – although my personal experience has been that small group work does not necessarily lead to group think, but can instead lead to significant learning as has been my experience with Matthias, John Mak, Roy and other f2f colleagues.

But if we stick with learning in a ‘massive’ network – as a number of people have already noted in PLENK2010 it is easy to feel lost, to find the open course lacking in terms of ‘tutor’ support and scaffolding, to experience the ‘dark side of networking’ as was mentioned in the chat room today.

It seems to me that it’s not possible to have it all ways. Evidently Alec Couros has managed to provide scaffolding in his open course (which I admit I know nothing about so this is second hand information) by ‘recruiting’ mentors to support his online learners. I would have to see this for myself to be able to judge it in action.

My feeling – during the session this evening (UK time:-)) – was that it’s a question of knowing what you have signed up for and what you can expect – and given that these open courses are free, then, as learners, we have a responsibility to check on what we have signed up for and what we can expect.

My expectations would be:

– for a small open course, there would be recognisable structure and ‘tutor’ input (small I would regard as anything under 30 – or possibly 50)
– for a medium sized open course, I would expect less tutor interaction and more peer-to-peer support (not sure about the numbers here, but anything between 50 and 200)
-for a large open course – I would be thinking in terms of ‘networked learning’ rather than course and not expect any personal interaction with the ‘tutor’ at all and to have to rely totally on my peers for support.

The numbers I have put in here are arbitrary – and just to give and idea of what I mean.

However, if I was paying for the course I would expect significant tutor interaction and support, but not to ‘have my hand held’ by the tutor. I would hope that even on a paying course a tutor would be encouraging independent autonomous learning.

I think it’s rather a shame that convenors of a ‘MOOC’ have to justify their approach when they are giving freely of their time and effort. That’s not to say that we and they don’t have a lot to learn about the management of open courses – but it is something we can do together rather than being an ‘us’ and ‘them’ situation.

#PLENK2010 Research, technology and networks

The guest speaker for the second week of the PLENK course has been Martin Weller – what a treat!  The link to the recording of his Elluminate presentation is here

Basically – his talk was about how depressed he is that research is not ‘keeping up with the times’ in terms of advances in technology and networked learning. I am a new researcher – but I can so completely relate to this.

So what did he say? These are the key points as I interpreted him –

  • researchers are not making full use of the new technologies available to them
  • they are risk averse and work in ‘traditional’ mode
  • they work in small personal contexts, often with the same groups of people and do not make use of network possibilities
  • they don’t like the spontaneity of blogs
  • they are conservative and cautious

Why are they like this? Because they may not have tenure and therefore have to ‘fit in’ with University requirements. If you want tenure you are encouraged to be traditional and are therefore less likely to be innovative or take risks. Research is about ‘control’ – particularly for scientists seeking predictive models, whereas the very nature of working in Web 2 is the exact opposite. We don’t know what will happen in Web 2.0. It is unpredictable.

Martin then went on to discuss the changing nature of research as evidenced by the number of people who are publishing in blogs , experimenting as they go along, turning to people in the network for peer review.  However, there are difficulties with this as I have already posted here following a discussion with Matthias.

What I found particularly interesting in Martin’s presentation is that it was directly relevant to how I have recently been working. I am a new researcher. My first two papers were both published following the traditional pattern.

1. K. Guldberg, J. Mackness (2009) Foundations of communities of practice: enablers and barriers to participation. Journal of Computer Assisted Learning

2. Rhona Sharpe, Jenny Mackness (2010)Evaluating the development of a community of e-learning researchers: from short-term funding to sustainability International Journal of Web Based Communities 6 (2) p. 148

These two papers are in closed journals. The second has received one expression of interest via email. The first has received about 15 expressions of interest via email.

Following participation in CCK08 John Mak, Roy Williams and I published two papers in the open environment of the Networked Learning Conference. We had also published drafts of the papers in the CCK09 Moodle site before submitting them to the Networked Learning Conference. This felt much more like an ‘open’ process.

Blogs and Forums as Communication and Learning Tools in a MOOC John Mak, Sui, Fai, Roy Williams, Jenny Mackness (2010) Networked Learning Conference, Aarlborg p. 275

The Ideals and Reality of Participating in a MOOC Jenny Mackness, John Mak, Sui, Fai, Roy Williams (2010)Networked Learning Conference, Aarlborg p. 266-274

Presenting these papers at the Networked Learning conference was disappointing from my perspective, but George and Stephen invited us to present the Ideals and Reality paper again in an Elluminate session and that was much more rewarding. We were able to ‘talk’ to more people and this made the research seem more worthwhile.

Recently, I have gone one step further with Matthias Melcher, in working for many weeks/months on a paper on e-resonance and simply publishing it here on this blog – but the process we worked through was not open to all.

The points that have arisen for me in all this are:

1. If we want research to be open there are two stages to consider – the actual researching and then the publishing.
2. Being ‘open’ at the researching stage might invalidate the research. There are difficulties associated with confidentiality and ‘ownership’ of ideas and writing.
3. Open publishing also brings its difficulties. How can the work be measured/peer reviewed?

I suppose all this brings in to question what we understand by research. There was discussion in the Elluminate ‘chat’ about the boundaries between learning and research becoming blurred and Stephen posted that ‘Learning = Research’. I thought at the time that this depends on how you define research. Who is it for? What is it for? How will the ‘network’ influence research and will research be able to influence the network?

I really enjoyed Martin’s talk, but I was left wondering whether we had really got to grips with how research will be influenced by networked learning and Web 2.0.  My experience is that there are still an awful lot of people in HE (where a lot of research currently happens) who are working in institutions with high research ratings, with outstanding publications records, but who are not connected on the Web in the sense that we have been talking about. This indicates that good research has been and continues to be published without the Web or being networked. I think we need to think more/be more explicit about what might be lost by giving up this ‘traditional’ system and more explicit about what we can gain by being more ‘innovative’.

I thought Martin could perhaps have been more explicit about why his really good presentation was relevant to a PLENK course.

#PLENK2010 Curation and Balance

End of Week 1 Notes (based on listening to the recordings of the two live sessions, having read a couple of references from the list and commented here – having kept an eye on the Daily and Twitter stream – and popped in on the Moodle forums – but not read much)

There has been lots of discussion this week about whether Personal Learning Environment (PLE) and/or Personal Learning Network (PLN) are the right terms to describe what this is all about and some recognition that this a semantics issue. According to Rita Kop PLE is a UK term and PLN an American term. Dave Cormier questions whether the term personal should be used at all. Stephen Downes points out that personal is an OK term if you think about [Personal Learning] Network as opposed to [Personal] Learning Network – and similarly for PLE. I like that – but for me, the words are not as important as the process – although I can see that the process needs nominalising for ease of reference. If I am going to think about introducing the idea of PLEs/PLNs to my colleagues or students then I will be talking about the process and the implications of this process for learning rather than what we should call it, i.e. why it might be preferable for students to learn in environments/spaces of their own choice rather than be confined to an institutions VLE/LMS.

There was a brief discussion in one of the live sessions about whether we choose networks or whether networks choose us – an interesting thought.  I would have thought both. In terms of PLEs/PLNs, Stephen’s comment was that its not a question of choosing so much as not having choices imposed on us – to me this is a good way of thinking about what the ‘personal’ might mean in PLE/PLN. It reminds me of my exhortations to my children to study harder at school because one of the values of education is that it affords increased choice.

A big discussion has been around Curation and Curatorial Teaching in the context of the role of supporting PLEs/PLNs. What does it mean? At first I thought it is just the American term for what Gilly Salmon calls ‘summarising’ in her work on the role of the e-moderator. But I can see that in a massive open online course (MOOC) it needs to be a little different to this and also that George’s definition of curator/curation has a different emphasis – http://www.connectivism.ca/?p=93

Here is a quote from George’s blog post from as long ago as 2007 (and don’t miss the political correctness in terms of gender references! Would this be necessary in 2010?)

The joint model of network administrator and curator form the foundation of what education should be. An expert (the curator) exists in the artifacts displayed, resources reviewed in class, concepts being discussed. But she’s behind the scenes providing interpretation, direction, provocation, and yes, even guiding. A curatorial teacher acknowledges the autonomy of learners, yet understands the frustration of exploring unknown territories without a map. A curator is an expert learner. Instead of dispensing knowledge, he creates spaces in which knowledge can be created, explored, and connected. While curators understand their field very well, they don’t adhere to traditional in-class teacher-centric power structures. A curator balances the freedom of individual learners with the thoughtful interpretation of the subject being explored. While learners are free to explore, they encounter displays, concepts, and artifacts representative of the discipline. Their freedom to explore is unbounded. But when they engage with subject matter, the key concepts of a discipline are transparently reflected through the curatorial actions of the teacher.

So from this post and from the discussion in this week’s live session we can see that a curator is intended to be more of an interpreter than a summarizer, possibly less objective than a summarizer, but with no less or more of a teaching role, which according to Stephen should be to aggregate, assimilate, analyse and advise (my interpretation of Stephen’s comments!). This is what Stephen does with The Daily – but he doesn’t think that curation is an adequate word for what this is all about, which he sees as journalism, reportage, descriptive activity and selection. On reflection and looking back over my experience of working with Gilly Salmon’s ideas, I think that summarising is probably very similar to curating when  describing the moderator’s role. In my experience, a summary inevitably involves selection and interpretation.

Finally – this week – I have noted how often the word ‘balance’ has cropped up in discussions. So we have had:

We need to find a balance between

  • over-guiding and not providing enough structure
  • niche and diversity
  • creation of and curation of artefacts and resources
  • facilitator and participant voice
  • power relationships in networks

This for me is getting at what we need to be thinking about, if we are considering introducing this approach to learning in more traditional settings. What else do we need to balance as this sort of thinking seems to me to be key to learning in and/or facilitating learning in open networks?

# PLENK2010 – Breadth versus depth – an illusion?

This is a response to Dave Ferguson’s and Stephen Downes’ comment’ on my blog post . Well actually more of a reflection than a response.

I understand Dave’s response. I can see that he is coming from the same place as I am in concerns about balancing depth and breadth in online courses, i.e. the practicalities of knowing how to manage the breadth of information we are exposed to on an open online course and knowing where to focus. This is a common concern. For example in her blog post Linn – talks about trying to avoid being ‘overfed’. Others have talked about feeling overwhelmed as George has noted – in his Moodle forum post (Making sense of (in?) abundance – in the General Discussion Forum). These feelings are very common, so is the depth versus breadth problem an illusion as Stephen claims?

To quote Stephen in full – he writes:

It occurs to me that the depth versus breadth problem is an illusion. One person’s breadth is another person’s depth. It’s an artifact of how we divide the world. If we divide it by discipline – computer science, physics, art – depth looks like one thing. But if you divide it by function – saving lives, educating children, building bridges – depth looks like something very different.

My experience with thinking about depth versus breadth has always been in terms of the ‘overload’ problem discussed above and ‘recognised’ by Dave. This is a real problem so in that sense is not an illusion – it is something experienced by many learners and something that many teachers think about in trying to select a curriculum for their learners – and it has become more of a problem now that we have so much ready information and networks at our finger tips. How do we know where to focus? This ability to filter, select and focus is a critical literacy skill that will be important to develop. This was discussed in the Critical Literacies course and Matthias and I have also discussed it in relation to e-resonance.

But Stephen’s point gives us a different perspective which is also very relevant to the networked world. I think my own experience of looking for depth has been in going deeper into a given discipline – but I can now see that the links/connections that you can make as a result of being part of a large learning network can also enable a depth of knowledge and understanding that might not be achievable through a single discipline.

This reminds me that Etienne Wenger often talks of the value of learning that takes place at the boundaries of communities of practice – i.e. where there is overlap between different communities. Paul Lowe has also made reference to Etienne Wenger in his blog post – The PLE as a roadmap of the landscape of practice. I can see links between these ideas and those related to balancing breadth and depth in learning.

I can also see that in this age of PLEs, PLNs and networked learning it will be important to be able to gain depth of knowledge and understanding both through digging deeper into a given discipline and through being able to exploit the diversity and breadth of our networks. But the question still remains of how best to keep this breadth and depth in balance and avoid losing out on both counts through an inability to manage information overload.

#PLENK2010 Immediate thoughts

It is interesting that this course has attracted so many people (over 1000?), but the Critical Literacies course attracted far fewer – and I’m wondering why, since a critical literacy must surely be to be able to manage a personal learning environment/network. Is it because the management of a personal learning environment/network is more practically focussed, but consideration of critical literacies is more conceptual/academic?

I have had a quick look at all the readings for this week. I was intrigued by Scott Leslie’s Mother of All PLE Diagram Compilation and thought I had better try and construct my own diagram – which I started to do and even considered using Prezi, until I realised that all this is terribly time consuming and I didn’t see that I would gain a lot. In my head I know which tools I use, why, when and with whom – I use most of them every day. I also know who I am networked with, which communities I follow and which tools I use to meet up with different groups/individuals. Having said that, looking at the diagrams was a spur to activating my Twitter account which has lain dormant since I created it ages ago. Now seems like a good time to test out whether it should be part of my PLE/PLN.

But more interesting for me is Dave Cormier’s blog post – Five points about PLEs and PLNs – Dave Cormier (Blog post) because he is talking about the related issues and why we should think about this at all. Like him I have always been concerned about the confusion between e-portfolios and PLEs (he didn’t express it like this – but this is the issue that his post raised for me). A lot of universities in the UK have introduced e-portfolio systems which are tied into the University’s platform. (Is this because of assessment requirements or am I just being cynical?). When the students graduate and leave the University they have to buy their own portfolio. It all seems very inflexible to me and ties the students to a system which ultimately may not suit their needs, when they move out into the world of work.

But an alternative perspective on e-portfolios is that at least everything is in one place in what is presumably a secure environment.  The disadvantages of open source distributed environments are not too difficult to identify; for example, you may lose your environment, as when Ning suddenly decided that users would have to pay for their previously free site.

There is also a concern lurking in the back of my mind about the effect of distributed environments on the quality of learning – i.e. the old breadth versus depth concerns. I personally find it very difficult to balance these. I have been very fortunate that my experience with distributed networks such as those promoted by the open courses I have attended, CCK08 and Critical Literacies (I only attended part of this one) has enabled me to experience more depth than breadth, in that I have ‘met’ research partners in these courses and have been able to collaborate in research projects which, as an independent consultant, not affiliated to any institution, would have been difficult to organise without these networks.

For me the  personal/conceptual interactions between small groups are more stimulating/interesting/fulfilling than a wide network of connections, but paradoxically I need a distributed network in order to find the resonating connections to lead to the conceptual and personal connections that I value. Resonating connections is very much at the forefront of my mind at the moment since Matthias Melcher and I have just completed writing a paper on this very topic after months of discussion. See The Riddle of Online Resonance – and yes – now that I have realised that there obviously is a link between the issues surrounding PLE/Ns and e-resonance – this is a shameless plug of our paper 🙂