MOOCs, Educational Technology, and an Oxford thesis

Last year I completed an MSc at the University of Oxford, where I wrote my thesis on MOOCs. This post talks a little bit about my research and why I think it’s important.  The full version of my thesis can be found here, and an abridged version here.

What a difference a year makes.

The New York Times proclaimed 2012 the ‘Year of the MOOC.’  Massive open online courses, with their hard-to-take-seriously acronym ‘MOOCs’, had become the fascination of New York, Silicon Valley, and thousands of universities in between.  As university officials scrambled to hop on board, a debate was brewing over the efficacy of MOOCs.  While some technologists and educators were predicting a revolution in higher education, others warned of displaced professors and for-profit education run amok.

Fast forward a year.  As the first studies and analyses about MOOCs have since been published, much of the hype has abated.  As it turns out, retention rates have been low, usually between 10-15%, and MOOC students are more likely to be well-educated than disadvantaged, despite the rhetoric of accessibility and openness.  Even Sebastian Thrun, CEO of Udacity, one of the leading MOOC platforms, has acknowledged the shortcomings of current MOOCs.  This dampened rhetoric is a welcome change—these technologies are young and need improvements—but MOOCs remain, in my view, an incredible opportunity for improving education.

I happened to be searching for a dissertation topic while this debate was unfolding last year.  I arrived at Oxford intending to study the effects of personalization and algorithmic filtering on political discourse but became more and more interested in education and the growing debate around MOOCs.  Faced with an impending deadline to declare a topic, I chose the education route because I believed there was more to contribute in this new and emerging area of study, one that remains critically important as we face stagnant educational outcomes across the US.

My challenge was finding an appropriately scaled research question within the context of MOOCs, which was particularly difficult given the lack of existing research.  Retention rates caught my attention.  Why were so many students dropping out of MOOCs?  Was this an indication that MOOCs weren’t working, or were the numbers missing something?

What are MOOCs?

Massive open online courses attempt to deliver high quality educational content in a digital, and therefore highly scalable, environment.  MOOCs use two primary tools to accomplish this goal: 1) modular lectures and assessments, optimized for ‘mastery learning’; 2) online communities for peer-support and peer-grading.  Let me unpack those a little bit.

MOOCs are designed to make learning a more personalized experience than traditional in-classroom experiences.  The learner may proceed at their own pace, progressing to the next lecture or assessment only when they are ready.  Lectures are ‘modular,’  meaning that they are broken down into small, self-contained segments usually between 8-12 minutes.  Each lecture is accompanied by a short assessment to check for understanding.  This course design aims to provide flexibility for the learner and achieve ‘mastery learning,’ a pedagogical method that stresses understanding over progress.  In this context, learners are encouraged to understand each learning unit completely before progressing to the next.  In a famous study, Benjamin Bloom demonstrated that mastery learning can result in a one standard deviation improvement in learning outcomes compared with traditional instruction.

Although computers may excel in providing a personalized learning experience, they struggle to provide human interaction.  To an extent, there may be no remedy to this shortcoming, but MOOCs are borrowing features from social networking sites and using them to provide an element of peer interaction while maintaining their massive scale.  MOOCs rely largely on online communities that interact through message boards and forums.  Students interact with each other and answer questions, provide feedback on assignments, and in some courses, grade each other’s work.

Taken together, these two characteristics provide a foundation for most MOOCs.  Having said that, there are many variations in the specific structure and content of MOOCs, and each of the major platforms has taken a different approach.  If you really want to learn more about MOOCs, you should try one!  Coursera, edX, and Udacity are all good places to start, and they’re free.

Where are MOOCs headed?

The more I learned about MOOCs, the more excited I became.  But I kept returning to the question of sustainability.  If MOOCs are typically free to students, how can they earn enough money to be financially self-sufficient?

The first answer is that they don’t have to, not yet.  Coursera and Udacity are both generously funded by (mostly) Silicon Valley venture capitalists.  Their funding model allows a great deal of flexibility in the early years of the investment, where the platforms will be able to operate at a loss while they scale their platform and user base.  edX, meanwhile, is a non-profit funded by MIT and Harvard.  Although both schools have indicated their preference for a financially sustainable model, their endowments are more than sufficient to fund the platform as long as it aligns with their educational missions.

The major MOOC platforms in the US, therefore, won’t need to become financially sustainable overnight—but the clock is ticking.  In response, each of these platforms has experimented with various revenue models.  Udacity has developed a partnership with Georgia Tech to offer a Masters in Computer Science through their platform.  Students will pay roughly $7000 in tuition (a fraction of the cost of the in-person degree), of which Udacity will earn 40%.  edX charges universities a fee to offer certain types of courses on its platform.  Coursera is experimenting with a program called Signature Track, which provides verification services for a small fee charged to students.

There is a strong economic argument to be made that MOOC business models will converge on one in which their technology is embedded into existing educational models (I make the argument in an earlier paper, available here).  If that is the case, then students will be paying to access content either directly, to the MOOC platform, or indirectly, through their tuition to the traditional educational institution.  And this brings us back to retention rates.  Will people really pay for a service that successfully graduates less than 10% of its enrollees?

The trouble is, those retention rates are based on courses where students can enroll for free, with just a couple of clicks.  There are no costs to enroll and no costs to drop out.  Is this a fair comparison?  Can the value of MOOCs be judged based on their free enrollment and in order to evaluate their efficacy within existing educational models?  I didn’t think so.  I wanted to find a better way to evaluate MOOCs, one that evaluated student behavior in a context more analogous to the environments I suspected MOOCs might inhabit in the future.

The Research Plan

In early 2013, the University of Pennsylvania offered a course called “Calculus: Single Variable” (CSV).  CSV was one of the first Coursera courses to offer Signature Track, an optional addition to the standard course that gives students the opportunity to earn a “Verified Certificate”.  For a fee, $49 in this case, students have their identity verified throughout the course and receive a “verifiable electronic certificate” upon completion.  This verification makes the certificate more valuable to students, but since the course content itself remains unchanged, it also provides an opportunity to study MOOCs in an environment where students are paying for access.  Moreover, since Signature Track students take the course at the same time as free students, it provides an opportunity to study the difference in student behavior between the two groups.

There were two main components to the research.  In the first, I developed a set of metrics for evaluating student behavior in MOOCs, metrics that were better calibrated to the specific characteristics of MOOCs than existing metrics, which had been borrowed from other disciplines.  In the second, I used a series of quantitative analysis methods to examine the course data from CSV for differences in student behavior between the Signature Track and non-Signature Track cohorts.

The Results

My analysis demonstrated significant differences in student behavior between the paying students (Signature Track) and those taking the course for free.  Across the board, paying students were more likely to complete the course and score well in exams than their non-paying classmates.  Signature Track students engaged with four times as much content, exhibited ten times higher persistence, and achieved seven times higher grades on average than Non-Signature track students.  Most importantly, Signature Track students were 25 times more likely to pass the course.

(Please see my thesis for details on results)

Why is this important?

As my research progressed, it became clear that the pendulum of public opinion was moving away from its previously exuberant perception of MOOCs.  The reasons for this shift are complex and varied.  There are valid reasons for a retreat from the over-hyped predictions of 2012, but the debate is also embedded in wider conversations about education reform, education technology, and politics.  These larger debates involve constituents either deeply invested in the status quo or committed to its dissolution.  They have, on both sides, co-opted the conversation about MOOCs to fit their interests.

Perhaps the most vivid example of this conflict is the debate about education technology.  For some, education technology offers an opportunity to revolutionize the classroom and improve the educational outcomes for millions of students who desperately need it.  For others, education technology distracts from the real needs of students: quality teachers, safe classrooms, and better systems of social support.  To the first group, the other seems deeply invested in a failing status quo.  The second group, meanwhile, sees its opponents as profiteers of a crumbling system.

The MOOC conversation has become ground zero for this conflict and attracted powerful supporters to each side.  On one hand, proponents of MOOCs and educational technology are widely aligned with the technology industry, it’s venture capital backers, and certain elements of the educational reform movement.  On the other, opponents (or at least vociferous critics) of MOOCs and educational technology often align with teachers unions, as well as those critical of education reform.  There’s far more nuance to this debate than I can sufficiently explain in this post—it’s a complicated environment—but suffice it to say that each side brings substantial resources to the table.

The debate about MOOCs has split largely along these lines.  As in any debate, each side has adopted its own set of statistics and numbers that best support its position.  Proponents cite enrollment numbers in the millions without acknowledging how many of those students never show up to class.  Opponents cite low retention rates without acknowledging how many students do pass these courses (even a small percentage of a really big number is a big number).  My research tried to develop a more neutral understanding of MOOC outcomes.

In the course I studied, over 66,000 students enrolled to take the course.  This is that big number that technologists love, but only about 29,000 students ever watched a video or took a quiz.  Of these students, only 611 finished, an especially low retention rate even for MOOCs.  This is that low number that opponents of educational technology love, but it fails to include any indication of student intent.  Of those students that indicated a high degree of intent by paying for the course, 57% (76/133) passed.  My research also shows that an additional 3,600 students watched most of the videos for the course but didn’t take any quizzes.  These students didn’t pass, but are we to say that they didn’t gain anything by watching the lectures?

The MOOC debate is just beginning.  We may ultimately stop calling them MOOCs (I’d be ok with that), and the environment in which students encounter them may change, but their core function and the technology that enables it isn’t going anywhere.  We need to ground this important debate in meaningful numbers, numbers that reflect what’s actually happening in MOOCs.  I think my thesis took a few humble steps toward that goal.