Evaluating open learning scenarios – keynote presentation

Here are the slides and notes for the keynote presentation that Roy Williams and I gave at the E Learning Conference, FH Joanneum, in Graz, Austria, on Wednesday 17 September 2014. This keynote included two interactive activities – see slides below, but  Slideshare does not enable powerpoint animations, so the post below also includes links to the original powerpoint.

Many thanks to the conference delegates for being active, friendly and supportive participants, especially given that we were not speaking in their native language, German.

The conference instigator, Jutta Pauschenwein, has written about the conference on her blog –http://zmldidaktik.wordpress.com/2014/09/21/elt14-rund-um-offene-lernszenarien-und-ihre-reflexion/ and there are further posts on this blog about our preparations for this conference.


Notes to accompany each slide:

Keynote presentation notes:

 1. A brief introduction to who we are

Jenny: I am an independent education consultant and researcher and have been working with HE institutions in this capacity since 2005. I have been publishing research since 2008. My whole career has been in education, starting with teaching in schools, and before going independent, working in HE as a teacher trainer. This involved developing and running a distance learning teacher training programme, which in turn led to an ongoing interest in how learners learn in open learning environments.

Roy: I am interested in exploring ways for people to explore, create, reflect on, and share their learning – individually and collaboratively.  I work and publish across several fields: semiotics, critical discourse analysis, epistemology, ecological psychology, politics, narrative, e-learning, e-assessment, knowledge management, synaesthesia, complexity theory, international development, art design & media, and most of all, on open learning.

2. How did we meet and start working together?

This is a cMap, created by a colleague we have worked with in the past, Matthias Melcher, to explain the open learning environment of the CCK08 MOOC. This was the first MOOC on Connectivism and Connective Knowledge run by Stephen Downes and George Siemens in 2008.

We did not know each other at this point and you can see that we did not even meet in the MOOC. Roy for the most part occupied the forums and Jenny worked on the MOOC from her blog.

You can see from the rest of the map that there were many locations to work in on this MOOC and participants could choose where to work from, which paths to follow and who to connect with. This truly was an open learning environment, with participants working autonomously across distributed platforms. The environment encouraged autonomy, diversity, openness and interaction.

We finally ‘met’ (virtually) at the end of this MOOC, when we began to collaborate on two research papers in which we investigated learners’ experiences in this MOOC. These papers were ultimately published in the International Review of Research in Open and Distance Learning, an open journal. We have listed the full references at the end of this presentation.

3. Why are we interested in emergent learning?

Following our research into our first MOOC experience, we realized that it was very difficult to ‘pin down’ and explain how and what learners learned in the MOOC and began to understand that this was because much of the learning was unpredictable, unexpected and emergent. In open learning environments we (as teachers) cannot know what learners are doing – what they are learning.

An example of this in nature is the way in which flocks of starlings form murmuration patterns just before roosting at the end of the day. In this behaviour, the patterns the starlings make and the directions they follow are always unpredictable. In addition, there is no leader. They are self-organising, constantly changing and adapting their direction as information is passed between the birds through continual interaction. This is a social behaviour.

Another example of emergent learning that we wrote of in one of our first papers, which is nearer to our work as teachers, was the example of April, a mature, part-time student of Early Years Education, who worked as a manager of a pre-school. She visited a pre-school centre of excellence and was impressed by how quiet and well-behaved the children were and how there were glass bottles and vases of flowers on the tables. She took this unexpected learning back to her own pre-school where she instigated changes. This learning journey was unexpected and beyond the prescriptive learning programme of her course. This was transformative learning for April.

We have recently submitted a book chapter in which we discuss the relationship between transformative and emergent learning. We believe that open learning environments offer potential for transformative learning.

4. What is an open learning environment?

This image shows some of the characteristics of an open space reaching into the distance, where we can’t see what is over the horizon. But the image only shows one path. In open learning environments there are many paths and the learner is free to choose which path to follow. It might be that the learner chooses to follow a more prescribed path as shown in this image, but equally the learner may choose to go off the prescribed path and out of the sight of the ‘teacher’.

So open learning environments can be experienced as quite safe – by following the prescribed path – or much less safe, e.g. a learner may get lost or fall off the edge of a cliff. On the other hand the learner may really enjoy freely wandering about in the environment. If learners leave the prescribed path they don’t know who they will meet or what will happen. The learning outcomes are unpredictable.

A key thing about open learning environments is that they offer learners the choice of which paths to follow.

These were our early thoughts about open learning environments which we then began to develop.

5. We started to think about all the factors which might influence learning in an open learning environment and quickly realized that these characteristics could be grouped into four clusters.

Two of these clusters relate to the learning environment and the other two to the individual learner. So we started to think about what the structure of an open learning environment might look like and how that environment might promote interaction.

And then we thought about how the learner can take control of their learning (agency) and what kinds of activities the learner might be involved in, to establish a presence in an open learning environment. So the questions we are interested in, in relation to the four clusters are

  • What is the balance between Openness and Structure? (Open/Structure)
  • How is the learning design implemented? (Interactive environment)
  • Do learners develop their own capacity for action, or just compliance with given roles? (Agency)
  • What traces do you make and leave behind you? (Presence/writing)

6. Ultimately we ended up with 25 characteristics or factors organized into the 4 clusters, which we think enable a learner to reflect on their learning experience in any given learning environment.

Why 25? Well – there could have been more, but we consider these to be the most important factors to consider. But this is not a definitive list. There could be alternative factors and some factors might not be useful to some learners, teachers or designers.

This was not the first list we came up. It took us a considerable amount of time to refine this list, but testing it out on various audiences and groups of learners.

The presence/writing was the most difficult set of characteristics to determine, but we knew that we had to consider the trace that learners leave behind them as they move through an open learning environment, or even a more prescribed course.

7. In thinking about how to describe the factors in each cluster, we have recently added an image to each description, thinking that perhaps this might make the factors easier to relate to, for some users of the footprints of emergence framework. These are the images we have selected for the Openness/Structure cluster.

8. These are the images we have selected for the Interactive Environment Cluster.

9. These are the images we have selected for the Agency cluster.

10. These are the images we selected for the Presence/Writing cluster. This was the last cluster of factors that we worked on, when we realised that learners need to be aware of the traces that they leave when they interact in open learning environments.

11. So now we had 25 factors organized in 4 clusters to use for reflecting on any given learning experience, but particularly learning in open learning environments. Whilst we could score these, or write about them in a list, we realized that some sort of visualization would have far more impact on learners and would also help to explain the environment as we understand it.

The animated gif created by our colleague Matthias Melcher, gives a sense of how precarious and unsafe an open learning environment might be and how easy it might be to fall off the edge as many MOOC participants do, i.e. they ‘fall out’ of the course. But the gif gives an impression that you can also fall through the middle which doesn’t happen in our framework. (Slideshare does not enable animations. Please access the link to see the animated gif).

In our framework – you can see a cross section of it on the slide – the central zone (the dark blue centre on the footprint template) is the safe prescribed zone. It is safe and comfortable but also quite restrictive and a learner has to make quite an effort to climb out of the valley and up onto the open plateau, which we have depicted in white in the template (where learning is likely to be sweetly emergent). Here the learner has many more choices about which paths to follow, but the as the learner moves further away from the centre and towards the darker blue edge, the learning becomes more challenging. We have described this as the sharply emergent zone. The learner may or may not enjoy this challenge. If the learner gets too close to the edge, the learning will be experienced as chaotic and the learner is in danger of falling off the edge (the dark blue zone).

We think it is possible to describe any learning environment as being on a spectrum between prescribed and chaotic and have thought about our 25 factors in those terms.

12. Here is a slide of how we have described the first two factors in the Open/structure cluster.

We have named the factor and given it an abbreviation for ease of reference, provided a graphic image which might help more visual learners, raised a question to prompt reflection and described the spectrum from prescribed to chaotic learning,

This sheet is used when drawing footprints of emergence and users are encouraged to add comments which explain how they have interpreted each factor in relation to their own learning experience.

13. Where have the factors come from?

Well they have not been ‘plucked out of thin air’. In thinking about how learning emerges in open learning environments we immediately drew on complexity theory and ideas of adaption and self-correction.

The influence of Etienne Wenger’s work and his emphasis on communities of practice, social learning, interaction, negotiated learning and identity development can be seen in the list of 25 factors.

Our experience of MOOCs and knowledge of connectivism, from the work of Stephen Downes and George Siemens, has been influential in the choice of factors for all the clusters, factors such as risk, disruption, multipath, co-evolution, self-organization, autonomy, diversity etc.

We think it would be fair to say that most teachers have been influenced by social constructivism; in their creation of experiential learning environments, recognition of ambiguity and liminal space and their emphasis on trust and support.

Finally Gibson’s work on affordances can be recognized in many of the factors. Social media offer many affordances and possibilities for emergent learning. Twitter, for example, allows for many casual, serendipitous encounters, informal writing and networking.

14. Let’s consider one factor in more depth and how it might have impacted on your own learning, by working on a short activity, which will involve discussion in groups of 4

15. This is an outline of the activity in English. We’ll read through it here first, but the next slide has the same information in German. You will have 10 minutes to discuss the questions in groups of four

  • How have you experienced risk?
  • How does risk affect the way you learn?

And then for 5 minutes we will take feedback in English from some of the groups. So one person in each group should be prepared to feedback one statement about how risk affects learning or the design for learning.

16. Activity instructions in German

17. Our experience is that emergent learning will occur when there is frequent interaction between many people and resources, where no-one is able to follow everything, as happens in most MOOCs. In these circumstances people need to be self-organising and independent and the environment needs to be adaptive. Learning will be unpredictable and emergent.

So how do we know what our learners are doing in these environments.

We have created the drawing footprints tool to help to visualize these emergent learning experiences.

18. Here is an example of a drawn footprint.

In the Table is a list of the factors we have already discussed and their associated abbreviations. You can see these abbreviations beside the points on the footprint line. Each point has been placed on the spectrum between prescribed and chaotic learning. So, for example we can see that the factor Experiential in the Interactive Environment cluster was thought to be very challenging and near the edge of chaos.

The image of the palette is there because we view the list of factors as a palette that you can choose from, just as an artist chooses colours to paint with. Not all the factors need to be used.

The footprints can either be drawn by hand or electronically.

19. Let’s have a quick look at some examples of drawn footprints.

A footprint of a standards driven course (e.g. teaching, nursing) is likely to look like this, i.e. very much in the prescribed zone – but this footprint would help designers to reflect on the possibility of making some changes.

20. This footprint shows that it’s possible to superimpose one footprint on another. In this case the footprint show the learners interpretation of the course design intentions (the yellow line) and the actual learning experience (the red line)

Through this different footprints can be compared – in this case a perspective of the design and one of the actual experience. i.e. the experience was far more chaotic than intended.

21. Footprints can also be used to show how the experience of learning changes over time. Here on the top left the learner has drawn her perspective of the design intentions for the course. The footprint top right was drawn at the end of Week 1, bottom left at the end of Week 2 and bottom right at the end of Week 4. Whilst at the end of Week 1 the learner is experiencing what she expected, the experience in Week 2 and Week 4 is far less comfortable.

22. Now we would like you to think about a course you have recently taught or taken. (The animation in this slide and the next one does not show in Slideshare. To see how it works see the powerpoint presentation – Surfacing, Sharing and Valuing Tacit Knowledge 17-09-2014.

Overall would you describe that course as being in the prescribed learning zone, the sweetly emergent learning zone, the challenging emergent learning zone, or was it chaotic, or did you fall out of the course, off the edge.

You can see that when we click this red button it moves along the line.

When we do this again. We would like you to stand up when the red dot reaches the zone which describes your experience and sit down when it leaves the zone.

23. Stand up and sit down now.

24. We see the footprints as a tool for reflective practice. We have deliberately made it a flexible tool. Factors can be used or discarded. They could also be changed. For example if we were working with children, we would need to adapt them. Maybe the children themselves could suggest how to adapt them.

Currently the research into learning experience in MOOCs and open learning environments is focusing on gathering Big Data. We don’t believe that this can capture the learner experience in open learning environments, because much of that learning will be invisible and unpredictable, but nevertheless valuable and possibly transformative for the learner.

For understanding learning in MOOCs and open learning environments we will need tools that can encourage learners to deeply reflect on their experience. Our experience is that drawing footprints can do this, but they are nor a quick fix. They require time, thought and discussion.

25. We believe that the value of drawing footprints lies in providing a tool for eliciting tacit knowledge and understanding with minimal or ‘light touch’ facilitation/disturbance. Standard evaluation tools such as questionnaires, tend to be for the benefit of organisations and teachers, rather than for the learners and do not encourage depth of reflection. Drawing footprints encourages learners to dig deeper.

26. Here are some final thoughts, which we have only summarised here but which we discuss in more depth in the paper we have submitted for this conference.

Williams, R. & Mackness, J. (2014). Surfacing, sharing and valuing tacit knowledge in open learning. 13. ELearning Tag FH JOANNEUM am 17. September 2014

One thought on “Evaluating open learning scenarios – keynote presentation

  1. jupidu September 24, 2014 / 5:05 pm

    One comment to: 22. “Now we would like you to think about a course you have recently taught or taken”. A red button moved along the line at your slide from the prescribed learning zone, the sweetly emergent learning zone, the challenging emergent learning zone to the chaotic zone.

    And then you invited us participants to stand up when the red button passed the region we related to the course we reflected about.

    It was similar to a wave – in the very prescribed zone nobody stood up, then people started to stand up, others joined them, whereas the some sat already down again. I hope that this wave will be visible at the video 🙂
    It was a nice experience being member of the wave 🙂

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s