Do you really need to wait to calculate Progress 8?

By

There has been some discussion on blogs and twitter in recent days about whether schools should try calculating Progress 8 on results day. Given that it is the measure used to define the floor standard, it is quite understandable that schools will want to know as soon as possible where they lie in relation to the floor of -0.5 points per subject below expectation. Not only that, internal school analysis of each year’s results informs improvement planning for the next year. Yet the general consensus is that so much is changing you would be best to wait and, indeed, this is the advice we gave earlier this year. Is this really the case?

Yeah. Why not use last year’s estimates to calculate Progress 8?

Surely schools can just grab the 2015 attainment 8 estimates and compare each pupil’s 2016 Attainment 8 score given their Key Stage 2 prior attainment? You could. It might tell you something. But how close will it be to the figure a school will see during the Autumn term when the Performance Tables checking exercise starts?

Let’s go back a year. Let’s imagine schools have just received their 2015 results. What if we grabbed the 2014 attainment 8 estimates and calculated a “results day” P8 measure? How different would it be to the actual P8 measure that would arrive some time later? We show the difference in the chart below.

P8estimates

For schools with lower attaining intakes there is very little difference although in some cases the “results day” measure would have under-stated the actual score but only by a small amount. We know that entries (and as a result Attainment 8) increased for pupils with lower prior attainment.

But the big differences are among schools with higher attaining intakes. Actual scores tended to be better than those that might have been calculated on results day, sometimes by a large margin. One Grammar school changed from a “results day” measure of +0.37 to an actual measure of +0.68.

This is a legacy of the Key Stage 2 boycott of 2010. All pupils affected who were assessed at level 5 by teachers were allocated a Key Stage 2 fine level of 5.5 for the purposes of Progress 8. If they had been tested, they would (mostly) have ended up in lower prior attainment bands. The 2015 estimate for this group certainly cannot be trusted. In fact, it is lower than the estimate for Key Stage 2 fine level 5.4. If it was to be used in the calculation of P8 for 2016, pupils with an average fine level of 5.5 would have estimates that were too low and so the “results day” P8 score would be too high compared to the actual.

Can we really not do anything?

So this year we won’t have the problem of the boycott. However, its legacy blights the published Attainment 8 estimates for 2015.

We could recalculate the 2015 Attainment 8 estimates stripping out the pupils affected by the boycott. Such a table is presented below if you really want to use it. Just bear in mind that it ignores the increase entries this year in subjects which count in the Ebacc buckets of Attainment 8, which will affect pupils with lower prior attainment more than others. As a result, you might calculate a P8 score that is slightly too high. But it might still be a useful starting point, albeit for internal consumption only.

The danger is that schools put a lot of wasted effort into calculating numbers which a) will change and b) may not be particularly helpful anyway. If, for example, you calculate a low P8 score that leaves you close to the floor standard, it won’t provide you with much insight about where to target improvement efforts. Would it not be better to cut out the middleman and just examine departmental performance as you would have done in previous years?

Perhaps the best thing to do is follow the sage advice of headteacher Stephen TierneyDon’t rush to calculate Progress 8!

Thanks to @DataEducator for his comments on a draft of this post.

P8lookuptable

By | 2016-12-07T12:55:12+00:00 August 25th, 2016|Exams and assessment, School accountability|

About the Author:

Dave Thomson is Chief Statistician at FFT with over fifteen years’ experience working with educational attainment data to raise attainment in local government, higher education and the commercial sector. His current research interests include linking education and workplace datasets to improve estimates of adult attainment and study the impact of education on employment and benefits outcomes.

4 Comments

  1. Simon Carson September 10, 2016 at 6:09 pm - Reply

    One approach to calculating at least a ‘worst case’ P8 score I have suggested here before. The DfE statistics show how many of the EBacc and Open ‘buckets’ were filled at each KS2 starting points, so it is possible to scale up the EBacc and Open national figures to produce ‘filled bucket’ estimates for the EBacc and Open groups and then for the full A8 estimates. This enables schools to estimate a ‘worst case’ P8; using 2015 figures probably then gives a ceiling, with the real P8 score most likely lying somewhere in between.

    A further problem faces us in 2017 when the point scores for the old A*-G grades change, with the awarded points flattening out for the low grades and becoming steeper for the high grades. How will this affect P8 in 2017? As an experiment, I converted old point scores (1-8) to new point scores (1 to 8.5) by using linear interpolation between the ‘fixed points’ at whole grades, and then recalculated what a filled bucket P8 score would look like with our 2016 data. The overall P8 score changed by over 0.1, for the better – a significant change. The amount of change will depend on the distribution of P8 scores, and will be greater for higher ability students, of course.

    Has EducationDataLab done any modelling of the forthcoming change? It makes calculating P8 (estimating, I mean, or perhaps, simply guessing!) in 2017 even more fraught with problems.

  2. Dave Thomson September 11, 2016 at 8:45 am - Reply

    Hello Simon. Thanks for commenting. Your 2015 method sounds sensible. Number of entries is likely to change more than the average grade per bucket.

    We did a bit of work on the 2017 points when they were first announced. We concluded the same thing- it would benefit even further schools with more able intakes, including Grammar schools https://educationdatalab.org.uk/2015/03/getting-from-b-to-a-or-g-to-f/.

    We’ll be doing more on the change when the 2016 data becomes available.

  3. Simon Carson September 13, 2016 at 8:50 am - Reply

    I too have been thinking about the 2017 changes. I have drafted a paper for our SLT on the problem of making any sensible predictions in 2017, even when we have the 2016 figures. I would be more than happy to share it, and would be very interested in any feedback or comments – or, even better, help and advice!

Leave A Comment