Progress 8 is too favourable to grammar schools and understates secondary modern achievement

By

Progress 8 is the new measure by which secondary schools will be judged. It works by comparing each child’s achievement in eight subjects at GCSE with the average GCSE results for other children who got the same results in exams taken at age 11. The Department for Education designed it to incentivise schools to provide a broad academic curriculum and focus on helping all pupils make good progress. By comparing pupils with similar levels of academic attainment, it is supposed to provide a fair test of a school’s performance and trigger intervention where an individual school falls below the ‘floor standard’.

In 2015 the average Progress 8 score (using FFT’s Progress 8 estimate) across the 13 fully-selective local authorities was about zero, meaning their secondary schools were no better or worse than in non-selective areas. But while the grammar schools in selective areas had a Progress 8 score of +0.34 (i.e. each pupil scored about a third of a grade higher across eight GCSEs), the secondary moderns had a negative Progress 8 score of -0.11. Progress 8 measures therefore appear to show that selection has no effect on overall attainment, but does boost the attainment of grammar pupils at the expense of their neighbours attending secondary moderns.

But there’s reason to be sceptical. Progress 8 almost certainly overstates grammar school performance while understating secondary modern performance. This is because a Key Stage 2 test score is quite a noisy measure of a child’s educational attainment at age 11. It is not very highly correlated with 11-plus results, for example. While a KS2 score of 5.2 (indicated by the green line in the chart below) indicates the point where the majority attend a grammar, there are many with lower KS2 scores who pass the 11-plus and vice versa.
gram_p8_1

Why does this matter? Well, a student who scores as low as 4.5 on their KS2 and yet has managed to pass the 11-plus is probably more able than students scoring 4.5 who do not pass. They may have just had a bad day when they took their KS2 exams. We would therefore expect them to achieve much better GCSEs, regardless of school attended. Conversely, the student who scores as high as 5.5 on their KS2 and yet fails their 11-plus is probably less able than other students scoring 5.5. Failing the 11-plus suggests they may have been lucky with their KS2 exams: perhaps the questions they wanted all came up. This is the noise – things other than ability – picked up by the exam.

The chart below shows how Progress 8 scores relate to KS2 scores for both grammars and secondary moderns. Those who have very low KS2 scores (on the left of the graph) and yet pass the 11-plus (the pink grammar line) achieve incredibly high Progress 8 scores simply because their KS2 score almost certainly understates their academic ability at age 11. Some of the progress made during primary school is then unfairly attributed to their grammar school. By contrast, those who have very high KS2 scores (on the right of the graph) and yet have failed the 11-plus (the blue secondary modern line) have very low Progress 8 scores. Again, this is because they are likely to be less capable, on average, than others with the same KS2 score. Some of the progress at their secondary modern is therefore unfairly attributed to their primary school.

gram_p8_2

What can we do to fairly judge the performance of grammars and secondary moderns? The chart below shows the relationship between KS2 score and Progress 8 in fully selective areas. Progress 8 scores above zero indicate that pupils are outperforming their peers, and vice versa. The grey line on the chart below shows that in selective areas pupils with low prior attainment tend to achieve below what is expected, while high KS2 achievers tend to outperform their peers in the rest of the country. One option for judging performance is to judge schools in these areas collectively, recognising the sorting of pupils that is taking place. However this does not allow a judgment of whether individual schools have fallen below the floor standard and would therefore undermine the point of the Progress 8 measure.

gram_p8_3

Second, we could recalculate Progress 8 for grammar schools and secondary moderns by excluding students who pass (or are close to passing) the 11-plus. But KS2 is such a poor predictor of passing the 11-plus that this would mean excluding half the children in these areas from our estimates. This would probably make school’s Progress 8 scores very unstable over time. It would also undermine the government’s aim for Progress 8 to incentivise equal focus on all children.

A third, more sensible approach, would be to change how we calculate average progress. In selective areas, instead of comparing pupil’s progress to that of all other pupils with similar prior attainment, we should instead compare pupil’s progress with other pupils with similar prior attainment in selective schools (or non-selective schools). That would achieve all the main aims of the Progress 8 measure, including assessing whether a particular school meets the floor standard. But it would still not allow us to compare grammars and secondary moderns because, for any given level of KS2, we would always expect a pupil who passes the 11-plus to make more progress than a pupil who did not.

By | 2017-10-23T13:18:21+00:00 September 16th, 2016|Admissions|

About the Author:

Rebecca Allen is Director of Education Datalab and an expert in the analysis of large scale administrative and survey datasets, including the National Pupil Database and School Workforce Census. Her research explores the impact of government reforms on school behaviour, with a particular focus on accountability and teacher labour markets. She is currently on leave from her academic post as Reader in Economics of Education at UCL Institute of Education.

5 Comments

  1. Chris Ryall September 16, 2016 at 6:34 pm - Reply

    Firstly, I really appreciate the focus that your team puts on illuminating these questions.

    A question we’ve been considering within our schools is the connection between prior attainment and Progress 8 scores. As a companion to your graph for selective LAs Progress 8 by KS2 prior attainment, could you share a similar graph for non-selective and all LAs. Or point me to where i could make my own?

  2. Kim Price September 23, 2016 at 4:21 pm - Reply

    Hi Rebecca, interesting research. You state that a child’s KS2 score is not highly correlated with whether or not they pass the 11+ and that “a student who scores as low as 4.5 on their KS2 and yet has managed to pass the 11-plus is probably more able than students scoring 4.5 who do not pass”. What do you think the impact of tutoring to the test is on this? Not something you can collect data on unfortunately but I’d be fascinated to know if the children who score lower in the KS2 test but then went on to achieve the 11+ pass mark are largely those who have received tutoring towards the grammar school test. Do you have any thoughts on this?

  3. John Fletcher September 26, 2016 at 7:23 pm - Reply

    Hello Rebecca, As a head of a secondary modern I was wondering if you have the any information on 2016 figures in light of changes to Progress scores that were amended today. and the impact it has had on the secondary modern score of -0.11. Has the ‘average’ improved or declined?

    Thank you

  4. Zoe James January 19, 2017 at 3:43 pm - Reply

    You say: ” Well, a student who scores as low as 4.5 on their KS2 and yet has managed to pass the 11-plus is probably more able than students scoring 4.5 who do not pass.”

    – more able, I wonder, or more coached?

  5. Alexandra A. March 10, 2017 at 7:03 am - Reply

    Living in a London borough which retained the 11-plus. I can point to many children who did not pass the 11-plus, perhaps they had an off morning? However they went on to gain 8 or more A and A* grades at GCSE. The reality being these children were not aggressively tutored and in most cases had average SAT results. The 11 -plus test itself has also not been constant, over the years it’s has changedone in terms of formalities and content. It really does not test things like capacity to learn, resilience and application.

Leave A Comment