A version of this post was first published in Research Intelligence, the British Educational Research Association’s termly magazine.
When the PISA 2015 results were released in December last year, Vietnam was one of the countries that stood out as doing remarkably well. (PISA is the Organisation for Economic Co-operation and Development’s triennial assessment of 15-year-olds around the world.)
In particular, Vietnam was ranked 8th out of all participating countries in science, with an average score of 525 test points.
This was significantly higher than the average score for the United Kingdom – 509 – which came 15th in the PISA science rankings.
Should we copy Vietnam?
This is not the first time that Vietnam has apparently excelled in PISA, with a strong performance from this country in the last round, conducted in 2012. Indeed, Andreas Schleicher, the OECD education director who runs PISA, wrote an article for the BBC in 2015, discussing reasons for this developing country’s stunning success.
But does Vietnam’s amazing performance in PISA, given that it is still a low-income, developing country, mean we should rush to copy what they are doing in their schools – much like what the Department for Education has naively done in its attempts to copy Shanghai? No!
Vietnam’s PISA results, and the league tables so beloved by policymakers, are giving us an inflated perspective on how well this country is doing in educating its young people.
To understand why, we first need to recall what the PISA study is trying to do. It is attempting to measure the reading, science and mathematics skills of the in-school population of 15-year-olds across the world once every three years.
The key words in the sentence above are ‘in-school population’.
What about young people who are not in school, or have already left the education system completely? Simple – they are excluded from the group PISA is attempting to measure.
To put this problem into context, let’s say PISA was a study of 17-year-olds rather than 15-year-olds. How would the UK do? My guess is pretty well, because many of our lowest-achieving pupils leave the education system at age 16 – and hence would be excluded from the study. Suddenly, the UK would have higher average scores and less educational inequality than many other countries across the world.
This is exactly what happens with the PISA results for Vietnam. According to the OECD’s own figures, only 48.5% of Vietnam’s 15-year-olds are actually included in the PISA study (see table A2.1).
Those who don’t feature in the study (e.g. children who have left school early) are likely to be academically weaker than those who have actually been tested. Thus Vietnam’s PISA scores are artificially inflated, making its education system appear to be much stronger than it really is.
Quantifying the impact
Can we get a handle upon how much impact this is likely to have had? Digging through the many hundreds of OECD PISA tables, it’s possible to find an alternative set of PISA results (see table I.2.4d) where it is assumed that, in each country, 15-year-olds who are not included in the study all perform below the national average.
Using this information, we can compare these alternative results to the headline PISA findings, and consider the difference.
This is exactly what I do in the chart below, with results referring to the 75th percentile of PISA science scores[1]. The horizontal axis presents the headline figures from PISA, while the vertical axis provides the alternative results after all 15-year-olds in the country have been included, with the assumption that those not in school perform below the national average.
As we can see, Vietnam is a major outlier. Specifically the ‘real’ performance of Vietnam is probably between 50 to 60 points lower than that reported in the headline PISA rankings.
In fact, in these alternative results, Vietnam is now well behind the UK, with a score of 519 versus 566, with Vietnam ranked a lowly 47th in these revised rankings. (The UK would be in 17th position.)
Lessons from this
It is critical to note that this result is not about an issue with sampling for PISA in Vietnam. Rather it is a limitation of the way in which the study population has been defined.
Either way, Vietnam serves as an important case study of just how hypnotic international rankings like PISA can be – and how easily they can lead us astray.
Much deeper and more considered interpretation of the results, and analysis of the data, is needed for PISA and other international studies to really be useful for education policy-making. At the moment, there continues to be far too much hysteria surrounding what often turns out to be some quite flaky results.
Want to stay up-to-date with the latest research from Education Datalab? Sign up to our mailing list to get notifications about new blogposts, or to receive our half-termly newsletter.
1. I focus on the 75th percentile as, for Vietnam, results for the median do not exist – given that more than half of 15-year-olds do not take part.
So, what is the percentage of the in-school population (out of the whole) that the PISA tests cover in Shanghai?
Dont know for shanghai specifically. But can pull out the figure for “china” in pisa 2015 (four provinces that took part).
If you look at the graph in the blog, the other outlier is china. So it has a reasonable impact there as well (around 20 points or so) but not as much as viet nam…
Are Canada’s good results contaminated with any known sampling or other biases?
Thank you.
The UK is just making excuses for doing not so well.
From a Vietnamese:
Not totally agree with this. Vietnamese students have a tradition of learning, and are learning pretty hard at math and science.
I have contact with students in VN an US and see that on the same scale, VNese students can actually outdo US student in a test.
China and Vietnam are essentially “cheating” themselves in this PISA scores because they’re essentially have high percentage of missing students.
It’s not like other Asian don’t have a tradition of learning.Indian,Chinese,Japanese.The US missed 16% of its expected coverage, which is kind of lousy. Not as bad as Vietnam (51% missing), Mexico (38%), or China (36%).This mean the government deliberately don’t allow average performing schools to participate and only hand selected the best one.
Countries in Europe and East Asia(Singapore,HK) are the most honest.
In the test PISA and Olympiads Vietnam students have always result better or the same like US students.
I don’t agree with the author’s analysis either. I am sure that if you put all the Vietnamese kids not going to school now in class room, they would excel as the kids currently inside the class room. Kids outside the class room are not stupid, they just don’t have opportunity to go to school because their families situation. The real issue is deeper than that. It involves with the education system, the parents and the kids themselves. I offer my feedback from my own experience as a Vietnamese-America growing up in Vietnam as well as the US.
I came to the US when I was a teenager. As a 10th grader I was able to take math classes at 12th grade level, and still got A easily. I just think the cirriculum in the US education system is not challenging enough. You can’t expect the kids do well while the school doesn’t offer the right materials.
I honestly think children in US (or UK) are as smart as the Vietnam students. However, in the Western world, people don’t value education as much as the Asian world. Many parents and especially the kids don’t know that these fundamental skills (math and science) will greatly impact their futures. There is no sense of urgency when the kids don’t do well in those subjects; in Asia families, that would be a serious matter. Perhaps in the Western world, everyone is rich, their future don’t depend much on education. But in Vietnam, a poor country, one would be a blue collar forever unless he/she could obtain these skills. For them, it’s an important and urgent matter. Should it be an important matter for Westerners?
Lastly, I blame it on Bill Gates and Sony. Their inventions of the Xbox and PS systems really corrupt our younger generations. After school, kids spent most of their times on games instead of doing home works. This is an issue on both sides, but in Vietnam, it has less impact because not too many families could afford to buy Xbox. To minimize the damage of gaming, teachers must constantly monitor and be tougher on the kids if they are failing on these subjects.
All those are issues that could be corrected, but required lots of efforts. But if we aim nothing to resolve this conundrum, we will hit nothing.
Look at Vietnamese students perfomence in Germany. May be useful for you !
Do you know?
https://www.theregister.co.uk/2013/03/22/vietnam_kids_google_interview_pass/
I don’t agree with you. You know what? In Asia, especially in Vietnam and China, students have to learn so hard to get good job in the future, although it is not their favourite subjects. Parents force their children to learn so hard. When I was eighteen, I have to learn 16 hours per day, play soccer and games in 2 hours and I just sleep in 6 hours per day. Why do I have to learn so hard? Because I want to get in top university in Vietnam. In that time, my friends and I often learned together to solve many difficult Maths Excercises, Physical Exercises or Chemistry Exercises. We used to gather in the local library to study together. I guess the reason why you say PISA was wrong is that Vietnam is still poor. Vietnam is communist country and the economy is not free to business, so many Vietnamese students come to Australia, New Zealand, the US, Germany, Russia to work.
I think the author do not know much about Vietnam education today, I believe that PISA result for vietnamese students is not surprised and very dependable. I bet if the test executes with 17 year old students, Vietnamese students is definitely outstanding, maybe, the result can be better than for 15 year old studdents, because 17 year old students are in font of university entering examination, they study much harder.
A notice I want to point out that, nowadays, Vietnamese young people is more than 99% amount finished general education at grade 12 of high school and 40% will enter high education at university.
If you add the conditions for the ranking of PISA: it must be inversely proportional to the weight, the position of the American and Australian students will be raised even higher. If factual points are calculated, the ranking as everyone has seen.Look at the results IPhO 2019 and write a comment!
It’s not really surprising that Vietnamese kids did so well, considering how well the Vietnamese descent kids in developed countries outperform their peers there. On top of that, their scores in international math and science competition are among the elite for years. Maybe it’s the culture, genetic, or perhaps the combination of the two, but they are generally smarter. Same goes for many kids in that part of Asia. Some are physically stronger, and some are smarter, not by much, but just enough to tell.
I personally don’t have much experience, but I’m a Vietnamese still in the learning process thus I believe I can put in a somewhat reasonable explanation to why we perform so well in tests. Vietnam has its own national education program and the information that’s taught to students is complicatedly-worded. People tend to develop the knowledge to students plainly without any accompanying practicals, causing them to rote in order to prepare for examinations. It’s not about limitation in people either; what we’re forced to consume is all that matters since the majority of children are granted the opportunity to go to school. In fact, I heavily disapprove this teaching method; even if we can score highly in PISA, facing brand new questions should be challenging and the majority of us wouldn’t be able to improvise.
Author missed out a more important issue of Social Integrity. PISA organised by the OECD, with the E for Economic NOT Education. The VNmese girls out-performed the boys, especially in science. The Resilient, of the the poorest, was among the best. So many kids were deprived of education because their families/country couldn’t afford their schooling.
Growing up in VN, I had a poor friend of mine who had to help his mum washing dishes in their street food stall after school & before school. We were 10 then. (don’t campaign against child labour, plz). Would you reckon that this might effect his school performance. Oops, my due apology, this’s an Education platform, not the Social Study.
Digression: In case you’re curious, that friend of mine is a moped mechanic now. He left school at yr. 8 to learn a trade. He’d bought his small & humble house cum workshop with his own saving (no mortgage at all) He’s quite content with life in a small town in VN. He respects the erudite people like your guys. So do I.
low achieving 15 y/o’s dropping out of school and being excluded from the sample is not unique to Vietnam. this is moronic article.
Such a bad shortsighted article. Here is a presentation of a more nuanced, more sophisticated statistical analysis. It even features regressions! (of which you do not)
https://www.youtube.com/watch?v=5u3Z1RP8D_k
This is a very wrong interpretation of the result and just outrageous. How can someone inflate the score when it comes to math, science or reading? This is not a political competition. With these subjects, the answers are very clear. If anyone who took standardized exams, they know there is usually only 1 answer. How can you inflate the score? This is an education website, but you discredit others’ academic efforts. Such a shame! This kind of culture breeds injustice; and no-one will try anymore.
dont hate the player, hate the game. And the game being intelligence. This is article is a clear example of jealousy of western countries losing to a much more brighter youth in Asia…