Problems with MOOC research

Like Frances Bell and Roy Williams, I too have listened to Stephen Downes’ recent presentation to a German audience in Tubingen, Germany. Thanks to Matthias Melcher for sharing the link.

Digital Research Methodologies Redux 

May 26, 2014: Keynote presentation delivered to, Tübingen, Germany

Screen Shot 2014-05-28 at 15.02.37

The image above is a screenshot. You can access the presentation on OLDaily  – or at 

Also like Frances and Roy, I found the presentation very thought-provoking and relevant to the research I am doing with Frances and Mariana Funes on rhizomatic learning – and the research I am doing with Roy on emergent learning. Both these areas of research are trying to discover more about how people learn in open learning environments, such as MOOCs.

But Stephen is skeptical about the possibility of doing any worthwhile research into MOOCs if we continue to take a traditional approach to research, which he describes in this slide:

Screen Shot 2014-05-24 at 11.01.22

(This is a screenshot – not a video!)

I would expect most people who work in education to recognize this model (the hypothetico-deductive model), particularly those with a science background and those who read research papers, many of which follow this model.

Stephen’s argument is that much of the research that has been done into MOOCs or is being done into MOOCs, using this model, presupposes its own conclusions, i.e. you find what you are looking for because that is what you expect to see. In addition there can be, in this type of research, the implication or assumption that it is possible to find some sort of ‘truth’ or a ‘universal theory’, but Stephen believes that observation and experience are the foundation of knowledge and that there is no truth that can be known before experience just by thinking about it; we can also question whether there is any ‘truth’.

So a key question for this presentation was ‘Which methods and theoretical conceptions are appropriate for MOOC research?’

The suggestion was that traditional approaches to research do not account for the horribly messy, complex, always changing world in which we are now living and conducting research. There are no universal theories. Whilst we may walk daily on the earth assuming and believing that the ground will not open up and swallow us, this can in fact happen (see YouTube videos on ‘sink holes’). Our generalizations about the world come about as a result of our experience of the world and are not based on any underlying principle.

Whilst I come from a science background and have published research which would be recognized as using the traditional approach Stephen describes, what he says resonates with me, in particular the discussion about research being like learning a language.

Having just spent three days speaking only Portuguese with some Brazilian friends, and having lived in Brazil in the past, I know from experience that learning a language requires immersion – that this is extremely messy, that at the start we only understand a fraction of what is being said, that there are all sorts of cultural nuances that take years to assimilate, that there are words and even ways of thinking that simply do not translate, that understandings are context-dependent and often out of our control, and that communication can be an illusion. It is only through continuous and/or continual long-term immersion that recognizable patterns of understanding eventually emerge.

Research conducted from this perspective, i.e. a perspective of immersion, communication and collaboration is a process of exploration and discovery. It does not set out to ‘prove’ anything or necessarily to draw conclusions and reify knowledge. It recognizes that there is no one right way of describing the world. If there is a theory, it will emerge from the totality of the work. Developments around MOOCs have happened serendipitously through design-led research, rather than through research-led design. (See Liz Sander’s paper – An evolving map of design practice and design research).

All this makes sense to me, but that doesn’t mean to say that I have done or do research like this. Nor does it mean to say that I can easily put these ideas into practice. It is difficult not to presuppose some conclusions, particularly if I have been involved in the context which I am researching. It is difficult to remain objective if I am fully immersed in the context I am researching. It is difficult to prevent bias and subjectivity creeping in. And it is difficult to be credible if I appear to be ‘tinkering to see what happens?’

But I do agree that MOOC-related research necessitates the description of emergent phenomena rather than the identification of something that is true, and hence my interest in emergent and rhizomatic learning and in how Stephen’s ideas and presentation can inform this.

What is Academic Rigour?

This is a question that was raised near the end of Tom Reeves  very interesting presentation to ChangeMooc this week.

Recording for this session

The focus of Tom’s presentation was educational research and the lack of impact of educational technology research on educational policy and practice. To address this problem he has worked with colleagues to develop a model – Design-Based Research. in which the focus is on researching a problem.

Tom emphasised the importance of academic rigour, but this led to the question – what is academic rigour? There was no ready answer or consensus in the session.  Some answers to this question from Tom and participants were:

  • not for the faint-hearted; takes effort and commitment (Tom Reeves)
  • unchanging, in the sense that ‘rigorous’ means performing the same (type of) study every time, conforming to the same (set of) principles etc. (Stephen Downes)
  • more likely to lead to the truth (but what is truth?) (Stephen Downes)
  • disciplined, measurable, stands up to scrutiny by others (brainysmurf)
  • can replicate the methods (Tom Reeves)

None of these answers quite satisfies me and this dissatisfaction has led to some further thinking and discussion.

It seems to me now that there is not an absolute concept of academic rigour – but rather there are degrees of it depending of the closeness of research to known theory, whether or not the research is supported by known theory and the credibility of the data from which inferences are drawn.  This can be thought of in terms of the following diagram:

What is Academic Rigour?

Following a given model or a systematic process won’t necessarily lead to academic rigour or even reliable success.  I was interested in Stephen Downes’ comment that he is more a follower of Feyerabend in being “Against Method” and that ‘there is great liberation in understanding that ‘method’ is based on sociological desire for conformity rather than scientific desire for truth’.

Finally, there was the question of whether or not research needs to start with a problem – the Design-Based Research Model states that it does …..

… but this wouldn’t account for ‘Eureka’, ‘Ah Ha’ moments or accidental findings . This would suggest that ‘academic rigour’ is not always needed for good research.