Duolingo is an unlikely candidate to be a household name, but the language learning app company is not far off from being one. Although best known for its free courses in nearly 40 languages, Duolingo also offers an English language proficiency test similar to the well-established IELTS and TOEFL examinations. Cheaper, quicker and more accessible than its competitors, what exactly is the catch?
The pandemic has encouraged more universities to recognise the Duolingo assessment, at least in the short-term. Universities require international students to prove their English proficiency before accepting them onto courses, but physically going to an examination centre over the past year and a half was out of the question in many countries. Duolingo offered an alternative channel, and while many universities appear poised to cut ties with the assessment post-pandemic, others will likely keep the new channel open.
Importance of English proficiency exams
English proficiency exams play a significant role in assessing the ability of international students to study at English speaking institutions. Simply understanding the content of a course is a fundamental requirement to taking part in university, let alone for succeeding in studies. Standardised tests allow this process to be efficient and – in theory – effective. Given that it is impractical for universities to individually assess the language ability of all international candidates, trusted proficiency tests enable university student recruiters to assess the language proficiency of a student at least with a relatively high degree of accuracy.
Problems with existing assessments
The established assessments are not without problems. They can be very expensive, especially when the cost of courses dedicated to assessment preparation is taken into account. IELTS, the main assessment for entry to UK institutions, put candidates back around USD 200 just to take the exam. Many students will pay hundreds more to tutors in order to master not just their language proficiency but also the structure and expectations of the exam.
The traditional tests are also costly in time. Even highly proficient students find themselves investing time into learning the format of the assessment. As is the case with most of the world’s challenging language assessments, in ensuring they are challenging, often the test structure becomes the challenge rather than the content. Students must familiarise themselves with the idiosyncrasies of the test and devise strategies for dealing with them. Test day itself is also a black hole for time, the IELTS assessment taking the greater part of a day to complete.
Aside from the cost in money and time, access to the established assessments can be challenging in many countries at the best of times, let alone during a global pandemic. It is not possible to have a test centre in every town, and candidates have to get to the exam before they can sit it.
The Duolingo assessment goes a long way in addressing some of the largest pitfalls of the more established tests, so much so that at first glance it could appear game-changing. It costs a fraction of the price, namely USD 50 or a quarter of the price of IELTS. The whole assessment takes only one hour to complete and results are back within a few days. Perhaps the most significant point however as the pandemic continues to hinder travel both internationally and within local communities is the entirely online format. There is no need to travel to an assessment centre as candidates take the test from their own computers at home.
Superficially, the Duolingo assessment appears to threaten putting the other assessments out of business with its model focussed on speed and accessibility. There are however still some key factors missing from the comparison, those that deal with the actual assessment content rather than its delivery – quality and reliability of results.
Quality
If an examination fails to accurately measure the ability of its taker, then all the efficiency of its administration in the world is meaningless. For an alternative language proficiency assessment to establish, it must accurately represent proficiency in speaking, listening, reading and writing. Measures against cheating must be in place and need to work. Ideally, the content of the test should also be relevant to the needs of both the candidate and universities.
An independent study in 2014 into the reliability of the Duolingo assessment concluded that scores correlated moderately with TOEFL scores, which would at first suggest that Duolingo passes quality requirements.
Looking at the data in the study however suggests the correlation to be moderate verging on slight. Of candidates that scored 100 (out of 120) on TOEFL, their Duolingo results roughly covered 60 per cent of the test score scale (from 4.0 to 10.0), arguably too large a spread to be fully comparable. The spread could be attributed partially to factors such as better preparedness for one assessment over the other. Correlation between the two assessments in speaking and writing ability were in particular lower.
It is important to state that the scoring system has changed since the study released its results and could well have improved.
Background to growth of the Duolingo assessment
Before digging deeper into whether Duolingo provides the quality assessment it needs to in order to be widely valuable to universities and prospective international students, it serves to consider the background of the Duolingo assessment and the remarkable growth of the company that runs it.
Duolingo began its life as a computer science project in Pittsburgh. It was first opened to the public in 2012 after accumulating a waiting list of 300,000 users through its half-year long beta test. In its original form, Duolingo was conceptualised as a platform where users would learn a language while translating the web. Each user would receive a short clip of real text, and the collated translations could then be merged to produce an increasingly refined translation over time. Once complete, these translations were sold to businesses, monetising the free translation services of the website’s language learners.
The assessment first appeared in 2014 as part of a business model shift away from B2B translation. While recognition of the test was muted for some time, the pandemic has made an increasing number of universities consider accepting it for university entry. In 2021, Duolingo reported that over 1,700 universities globally accepted its assessment for undergraduate enrolments, and 930 for postgraduate courses. In the UK, 66 institutions recognised Duolingo for undergraduate entry and 55 for postgraduate. According to Duolingo’s CEO, Luis von Ahn, 17 of the top 20 US colleges now accept the Duolingo test for admissions.
While the assessment only accounts for 9% of Duolingo’s revenue, the company as a whole is now a giant of the edtech world. Their language learning platform has grown from 120 million users in 2016 to 500 million in 2020. The platform offering has also multiplied, from 19 languages to 39 in the same time period.
Tech giant Google clearly agrees with the hype. Alphabet, the parent company of Google, invested USD 30 million into Duolingo in 2019, kicking up the company’s valuation to over USD 1 billion. Only two years on, Duolingo went public in July this year and reached sky heights immediately. The IPO offering price of USD 102 per share grew by 40 per cent to USD 141.4 on its Nasdaq debut. Duolingo’s market cap is now over USD 6 billion.
Excited private sector, tentative universities
With such hype surrounding the company, one may have expected universities to be jumping at the opportunity to collaborate with Duolingo. However, of those institutions that have experimented with recognising the Duolingo assessment over the pandemic, some explicitly state on their websites that it will be accepted only for a specific year (read, until the pandemic eases). While researching this article it proved challenging to persuade any institution to comment on their motivations for only temporarily accepting the Duolingo assessment, but it is telling that some universities on Duolingo’s own list no longer list the assessment as a valid proof of English proficiency on their websites.
Structure of Duolingo exam
Online, done and dusted in less than an hour. That is quite the offer from Duolingo, but does it fit everything it should into the test within that timescale?
A marketing video from Duolingo presents the content of its test very transparently, but in doing so it raises questions. The video follows a candidate as they run through the exam setup process and a few example questions from each section.
All the questions shown in the video are extremely short, even those considered as essay questions. Following a similar format to Duolingo’s language learning app, most questions are multiple choice and close-ended in nature.
There are questions demanding either written or spoken descriptions of a given prompt, but the quick-fire format removes possibility of much depth. One high scoring former candidate of the assessment recommends in a video that exam-sitters should simply fire off synonyms when answering these questions given the lack of time to produce a proper answer.
The speaking assessment appears to be much in the same bent. Strikingly the interview section appears to be lacking an interviewer. Instead the candidate receives a prompt like in previous questions, and must deliver a spoken monologue on the topic. While this certainly demands a high degree of speaking proficiency it does not count as conversational ability. Speaking is a social action and comprises much more than just words. Interaction, response, listening ability, understanding of body language: these all are neglected in a monologue.
Comparison with IELTS
How does this stand up when compared to the UK standard, IELTS? Duolingo certainly beats it on price and convenience but ultimately an assessment must accurately measure proficiency. There are a number of points where IELTS holds a significant advantage over the newer test.
Past the pandemic, we should accept that in-person speaking assessments are invaluable. A recording of speech is not a substitute for a real conversation. Where Duolingo has shunned this, IELTS maintains a conversational speaking section. That is not to discount the possibility of online speaking assessments. Although recording a monologue is not equal to a conversation, video call speaking assessments could strike a viable balance.
IELTS is indeed now available in some countries online. The British Council offers its online version of IELTS in over 40 countries. The list is however missing some of the largest markets for international students including the very largest, China. Duolingo meanwhile is accessible from anywhere in the world, so long as you have an internet connection.
The short form questions that constitute the entire Duolingo test also pale compared to the older, more established assessment. Candidates applying for university need to be able to write cohesive text in English. A 5 minute essay question – as Duolingo provides – does not cut it for proving this ability. IELTS meanwhile requires a much longer piece of writing.
What do former English assessments candidates think?
While writing this article, some former candidates of English assessments were interviewed for their thoughts. A former candidate and teacher of IELTS was familiar with the Duolingo assessment but voiced concerns. They thought some students opted for Duolingo due to a belief that it was both easier to score highly, and easier to cheat in. Although Duolingo has added more security over time by requiring those sitting the exam to show an ID during the assessment, this candidate believed many still managed to cheat. The speed of the assessment process including receiving results, and the significantly lower cost of the assessment were however noted as major advantages over competitors.
Another former candidate believed the established IELTS assessment to be an accurate indicator of English proficiency but highlighted it was a far from a perfect test. They argued that the expectations for reading and listening were far lower than for speaking and writing. They also reported having come across (but not used) professional exam sitters taking IELTS in place of actual examinees – Cheating appears to be prevalent enough to make a business out of it, though it is hard to verify. This candidate was not familiar with the Duolingo assessment but found its price and accessibility highly appealing.
Conclusions
Language assessments have a difficult balancing act to maintain. Their quality and reliability is ultimately key, but they must also be accessible to a diverse potential student base. The pandemic accentuated the accessibility side of the balance as students faced extra challenges to reach assessment centres and sit tests in-person, while universities were compelled to expand recruitment channels. As the Coronavirus shifts from a pandemic to becoming an endemic disease, the balance needs to shift back to quality.
At this current moment, the traditional language proficiency assessments hold the higher benchmark for quality over Duolingo. That does not mean that they can’t learn from the newer model, or that Duolingo won’t develop into a better assessment method. Duolingo has innovated in accessibility and the older assessments should learn form that. It would however be naïve for universities to rely on the language learning company’s short-question, highly automated format in the long term, unless it were to develop more in-depth questions alongside the existing content.