Why does Vietnam do so well in PISA? An example of why naive interpretation of international rankings is such a bad idea

By

A version of this post was first published in Research Intelligence, the British Educational Research Association’s termly magazine.

When the PISA 2015 results were released in December last year, Vietnam was one of the countries that stood out as doing remarkably well. (PISA is the Organisation for Economic Co-operation and Development’s triennial assessment of 15-year-olds around the world.)

In particular, Vietnam was ranked 8th out of all participating countries in science, with an average score of 525 test points.

This was significantly higher than the average score for the United Kingdom – 509 – which came 15th in the PISA science rankings.

Should we copy Vietnam?

This is not the first time that Vietnam has apparently excelled in PISA, with a strong performance from this country in the last round, conducted in 2012. Indeed, Andreas Schleicher, the OECD education director who runs PISA, wrote an article for the BBC in 2015, discussing reasons for this developing country’s stunning success.

But does Vietnam’s amazing performance in PISA, given that it is still a low-income, developing country, mean we should rush to copy what they are doing in their schools – much like what the Department for Education has naively done in its attempts to copy Shanghai? No!

Vietnam’s PISA results, and the league tables so beloved by policymakers, are giving us an inflated perspective on how well this country is doing in educating its young people.

To understand why, we first need to recall what the PISA study is trying to do. It is attempting to measure the reading, science and mathematics skills of the in-school population of 15-year-olds across the world once every three years.

The key words in the sentence above are ‘in-school population’.

What about young people who are not in school, or have already left the education system completely? Simple – they are excluded from the group PISA is attempting to measure.

To put this problem into context, let’s say PISA was a study of 17-year-olds rather than 15-year-olds. How would the UK do? My guess is pretty well, because many of our lowest-achieving pupils leave the education system at age 16 – and hence would be excluded from the study. Suddenly, the UK would have higher average scores and less educational inequality than many other countries across the world.

This is exactly what happens with the PISA results for Vietnam. According to the OECD’s own figures, only 48.5% of Vietnam’s 15-year-olds are actually included in the PISA study (see table A2.1).

Those who don’t feature in the study (e.g. children who have left school early) are likely to be academically weaker than those who have actually been tested. Thus Vietnam’s PISA scores are artificially inflated, making its education system appear to be much stronger than it really is.

Quantifying the impact

Can we get a handle upon how much impact this is likely to have had? Digging through the many hundreds of OECD PISA tables, it’s possible to find an alternative set of PISA results (see table I.2.4d) where it is assumed that, in each country, 15-year-olds who are not included in the study all perform below the national average.

Using this information, we can compare these alternative results to the headline PISA findings, and consider the difference.

This is exactly what I do in the chart below, with results referring to the 75th percentile of PISA science scores[1]. The horizontal axis presents the headline figures from PISA, while the vertical axis provides the alternative results after all 15-year-olds in the country have been included, with the assumption that those not in school perform below the national average.

As we can see, Vietnam is a major outlier. Specifically the ‘real’ performance of Vietnam is probably between 50 to 60 points lower than that reported in the headline PISA rankings.

In fact, in these alternative results, Vietnam is now well behind the UK, with a score of 519 versus 566, with Vietnam ranked a lowly 47th in these revised rankings. (The UK would be in 17th position.)

Lessons from this

It is critical to note that this result is not about an issue with sampling for PISA in Vietnam. Rather it is a limitation of the way in which the study population has been defined.

Either way, Vietnam serves as an important case study of just how hypnotic international rankings like PISA can be – and how easily they can lead us astray.

Much deeper and more considered interpretation of the results, and analysis of the data, is needed for PISA and other international studies to really be useful for education policy-making. At the moment, there continues to be far too much hysteria surrounding what often turns out to be some quite flaky results.

Want to stay up-to-date with the latest research from Education Datalab? Sign up to our mailing list to get notifications about new blogposts, or to receive our half-termly newsletter.

Notes
[1] I focus on the 75th percentile as, for Vietnam, results for the median do not exist – given that more than half of 15-year-olds do not take part.
By | 2017-10-23T12:57:58+00:00 July 19th, 2017|Exams and assessment, International studies|

About the Author:

John Jerrim is a Research Associate at Education Datalab and a Reader in Educational and Social Statistics at UCL Institute of Education. John’s research interests include the economics of education, access to higher education, intergenerational mobility, cross-national comparisons and educational inequalities.

3 Comments

  1. Bill R July 19, 2017 at 8:54 am - Reply

    So, what is the percentage of the in-school population (out of the whole) that the PISA tests cover in Shanghai?

    • John J July 19, 2017 at 9:04 am - Reply

      Dont know for shanghai specifically. But can pull out the figure for “china” in pisa 2015 (four provinces that took part).

      If you look at the graph in the blog, the other outlier is china. So it has a reasonable impact there as well (around 20 points or so) but not as much as viet nam…

  2. Robert Flynn July 19, 2017 at 8:37 pm - Reply

    Are Canada’s good results contaminated with any known sampling or other biases?

    Thank you.

Leave A Comment