Why does Vietnam do so well in PISA? An example of why naive interpretation of international rankings is such a bad idea

By

A version of this post was first published in Research Intelligence, the British Educational Research Association’s termly magazine.

When the PISA 2015 results were released in December last year, Vietnam was one of the countries that stood out as doing remarkably well. (PISA is the Organisation for Economic Co-operation and Development’s triennial assessment of 15-year-olds around the world.)

In particular, Vietnam was ranked 8th out of all participating countries in science, with an average score of 525 test points.

This was significantly higher than the average score for the United Kingdom – 509 – which came 15th in the PISA science rankings.

Should we copy Vietnam?

This is not the first time that Vietnam has apparently excelled in PISA, with a strong performance from this country in the last round, conducted in 2012. Indeed, Andreas Schleicher, the OECD education director who runs PISA, wrote an article for the BBC in 2015, discussing reasons for this developing country’s stunning success.

But does Vietnam’s amazing performance in PISA, given that it is still a low-income, developing country, mean we should rush to copy what they are doing in their schools – much like what the Department for Education has naively done in its attempts to copy Shanghai? No!

Vietnam’s PISA results, and the league tables so beloved by policymakers, are giving us an inflated perspective on how well this country is doing in educating its young people.

To understand why, we first need to recall what the PISA study is trying to do. It is attempting to measure the reading, science and mathematics skills of the in-school population of 15-year-olds across the world once every three years.

The key words in the sentence above are ‘in-school population’.

What about young people who are not in school, or have already left the education system completely? Simple – they are excluded from the group PISA is attempting to measure.

To put this problem into context, let’s say PISA was a study of 17-year-olds rather than 15-year-olds. How would the UK do? My guess is pretty well, because many of our lowest-achieving pupils leave the education system at age 16 – and hence would be excluded from the study. Suddenly, the UK would have higher average scores and less educational inequality than many other countries across the world.

This is exactly what happens with the PISA results for Vietnam. According to the OECD’s own figures, only 48.5% of Vietnam’s 15-year-olds are actually included in the PISA study (see table A2.1).

Those who don’t feature in the study (e.g. children who have left school early) are likely to be academically weaker than those who have actually been tested. Thus Vietnam’s PISA scores are artificially inflated, making its education system appear to be much stronger than it really is.

Quantifying the impact

Can we get a handle upon how much impact this is likely to have had? Digging through the many hundreds of OECD PISA tables, it’s possible to find an alternative set of PISA results (see table I.2.4d) where it is assumed that, in each country, 15-year-olds who are not included in the study all perform below the national average.

Using this information, we can compare these alternative results to the headline PISA findings, and consider the difference.

This is exactly what I do in the chart below, with results referring to the 75th percentile of PISA science scores[1]. The horizontal axis presents the headline figures from PISA, while the vertical axis provides the alternative results after all 15-year-olds in the country have been included, with the assumption that those not in school perform below the national average.

As we can see, Vietnam is a major outlier. Specifically the ‘real’ performance of Vietnam is probably between 50 to 60 points lower than that reported in the headline PISA rankings.

In fact, in these alternative results, Vietnam is now well behind the UK, with a score of 519 versus 566, with Vietnam ranked a lowly 47th in these revised rankings. (The UK would be in 17th position.)

Lessons from this

It is critical to note that this result is not about an issue with sampling for PISA in Vietnam. Rather it is a limitation of the way in which the study population has been defined.

Either way, Vietnam serves as an important case study of just how hypnotic international rankings like PISA can be – and how easily they can lead us astray.

Much deeper and more considered interpretation of the results, and analysis of the data, is needed for PISA and other international studies to really be useful for education policy-making. At the moment, there continues to be far too much hysteria surrounding what often turns out to be some quite flaky results.

Want to stay up-to-date with the latest research from Education Datalab? Sign up to our mailing list to get notifications about new blogposts, or to receive our half-termly newsletter.

Notes
[1] I focus on the 75th percentile as, for Vietnam, results for the median do not exist – given that more than half of 15-year-olds do not take part.
By | 2017-10-23T12:57:58+00:00 July 19th, 2017|Exams and assessment, International studies|

About the Author:

John Jerrim is a research associate at Education Datalab and a professor of education and social statistics at UCL Institute of Education. John’s research interests include the economics of education, access to higher education, intergenerational mobility, cross-national comparisons and educational inequalities.

7 Comments

  1. Bill R July 19, 2017 at 8:54 am - Reply

    So, what is the percentage of the in-school population (out of the whole) that the PISA tests cover in Shanghai?

    • John J July 19, 2017 at 9:04 am - Reply

      Dont know for shanghai specifically. But can pull out the figure for “china” in pisa 2015 (four provinces that took part).

      If you look at the graph in the blog, the other outlier is china. So it has a reasonable impact there as well (around 20 points or so) but not as much as viet nam…

  2. Robert Flynn July 19, 2017 at 8:37 pm - Reply

    Are Canada’s good results contaminated with any known sampling or other biases?

    Thank you.

  3. Bob November 27, 2017 at 8:25 pm - Reply

    The UK is just making excuses for doing not so well.

  4. Quang January 27, 2018 at 4:01 am - Reply

    From a Vietnamese:
    Not totally agree with this. Vietnamese students have a tradition of learning, and are learning pretty hard at math and science.
    I have contact with students in VN an US and see that on the same scale, VNese students can actually outdo US student in a test.

    • Absolution April 13, 2018 at 1:23 pm - Reply

      China and Vietnam are essentially “cheating” themselves in this PISA scores because they’re essentially have high percentage of missing students.
      It’s not like other Asian don’t have a tradition of learning.Indian,Chinese,Japanese.The US missed 16% of its expected coverage, which is kind of lousy. Not as bad as Vietnam (51% missing), Mexico (38%), or China (36%).This mean the government deliberately don’t allow average performing schools to participate and only hand selected the best one.
      Countries in Europe and East Asia(Singapore,HK) are the most honest.

  5. Qdang April 1, 2018 at 10:31 am - Reply

    I don’t agree with the author’s analysis either. I am sure that if you put all the Vietnamese kids not going to school now in class room, they would excel as the kids currently inside the class room. Kids outside the class room are not stupid, they just don’t have opportunity to go to school because their families situation. The real issue is deeper than that. It involves with the education system, the parents and the kids themselves. I offer my feedback from my own experience as a Vietnamese-America growing up in Vietnam as well as the US.
    I came to the US when I was a teenager. As a 10th grader I was able to take math classes at 12th grade level, and still got A easily. I just think the cirriculum in the US education system is not challenging enough. You can’t expect the kids do well while the school doesn’t offer the right materials.
    I honestly think children in US (or UK) are as smart as the Vietnam students. However, in the Western world, people don’t value education as much as the Asian world. Many parents and especially the kids don’t know that these fundamental skills (math and science) will greatly impact their futures. There is no sense of urgency when the kids don’t do well in those subjects; in Asia families, that would be a serious matter. Perhaps in the Western world, everyone is rich, their future don’t depend much on education. But in Vietnam, a poor country, one would be a blue collar forever unless he/she could obtain these skills. For them, it’s an important and urgent matter. Should it be an important matter for Westerners?
    Lastly, I blame it on Bill Gates and Sony. Their inventions of the Xbox and PS systems really corrupt our younger generations. After school, kids spent most of their times on games instead of doing home works. This is an issue on both sides, but in Vietnam, it has less impact because not too many families could afford to buy Xbox. To minimize the damage of gaming, teachers must constantly monitor and be tougher on the kids if they are failing on these subjects.
    All those are issues that could be corrected, but required lots of efforts. But if we aim nothing to resolve this conundrum, we will hit nothing.

Leave A Comment