Difficult questions about some schools’ Key Stage 2 results

By

This blogpost supports a new report available for secondary schools in FFT Aspire but may be of interest to a wider audience. The school performance data in FFT Aspire and this blogpost is provided by the Department for Education under FFT’s accredited access to Analyse School Performance data.

Let’s imagine you’re a member of the senior leadership team of a school and you’re reviewing your school’s 2017 performance data. You decide to examine how pupils from each of your eight main primary feeder schools (those from which at least 10 pupils were recruited) performed using Progress 8.

To do this, you produce a boxplot of P8 scores for each of the eight schools (labelled A to H) such as that shown below, including a final box for pupils recruited from other schools, labelled O.

Your eye is immediately drawn to School H. On average, the pupils from this school went on to achieve a Progress 8 score of -1.9.

Assuming a notional 9-1 grading system for all qualifications, this equates to almost two grades per subject lower performance than that of pupils with similar prior attainment nationally. Just four pupils achieved a P8 score above 0.

Your curiosity is piqued.

How to interpret a box plot

The charts used here are box plots, a type of chart which most people are likely to recognise – but the details of which we may not all remember.

The line across the middle of the box is the median – the value which exactly half of results fall below, and the other half fall above. The box itself shows the range between the lower quartile – the value below which exactly a quarter of results fall – and the upper quartile – the value above which exactly a quarter of results fall. This definition means that exactly 50% of results are within the range of values covered by the box.

The difference between the lower quartile and the upper quartile is the inter-quartile range. The whiskers stretching out from the box reach to the maximum and minimum values, excluding outliers, which are shown as dots. Outliers are defined as results more than 1.5 times the inter-quartile range away from either the lower or upper quartile.

What were pupils’ KS2 results like?

Apart from a couple of outliers, pupils from School H achieved very high KS2 results, as the below chart shows.

In total 18 of the 23 pupils from School H achieved an average fine grade of 5.5 or higher in reading and maths.

Was the 2017 Progress 8 score a one-off?

You decide to examine data from 2015 and 2016 as well. As Progress 8 is affected by pupil demographics such as disadvantage, you decide you’ll use a contextual value added (CVA) version of Progress 8, which we’ll call Attainment 8 CVA. (This is available to FFT Aspire users).

The next three charts show the outcome of this, for 2015, 2016 and 2017.

So, no, the 2017 data was not a one-off. Pupils from School H have consistently achieved results two grades lower per subject at Key Stage 4 on average than pupils with similar prior attainment nationally.

Difficult questions

This is a real example. The secondary school is in an Ofsted category, criticised in its latest inspection for the progress of its most able pupils.

School H is rated outstanding.

We need to ask difficult questions about the Key Stage 2 results of School H.

Was it really the case that pupils performed at such a level at School H that they simply could not maintain their progress at secondary school?

Could they have received additional assistance when taking their KS2 tests?

Does test security need to be improved?

What can be done?

FFT Aspire users

If you are a secondary school using FFT Aspire, log in to the site to access your report – see the ‘Downloads’ section.

Not an FFT Aspire user?

Learn more here.

We have produced a report for FFT Aspire users to enable them to see a breakdown of Progress 8 by primary school (see the sidebar for details of how to access this).

Are there any primary schools from which pupils perform particularly, and consistently, poorly? Or even consistently well, perhaps indicating some under-performance at Key Stage 2? How does your P8 score look with these schools removed?

Looking at Attainment 8 CVA for the last three years and taking account of secondary school attended[1], we found around 30 feeder primary schools with a score of -1 or lower, indicating that pupils achieved at least one grade below expectation per subject on average.

There were none with a score of +1 or higher.

The Department for Education plans to cap the P8 scores of pupils with anomalously large scores. Ofsted’s Inspection Data Summary Report (IDSR) shows the effect of removing outliers on schools’ value added scores  [PDF, p.5].

We would also recommend to both DfE and Ofsted that they also consider making adjustments for pupils who attended primary schools with anomalously high KS2 results.

Better still, don’t rely on a single metric to decide which (if any) schools are performing poorly.

Want to stay up-to-date with the latest research from Education Datalab? Sign up to our mailing list to get notifications about new blogposts, or to receive our half-termly newsletter.

Notes
[1] As secondary schools vary in Attainment 8 CVA scores, with some performing poorly and some very well, pupils’ CVA scores are centred around the secondary school average to enable a fairer comparison of primary schools.
By | 2018-04-17T09:12:25+00:00 April 17th, 2018|Exams and assessment, School accountability|

About the Author:

Dave Thomson is chief statistician at FFT with over fifteen years’ experience working with educational attainment data to raise attainment in local government, higher education and the commercial sector. His current research interests include linking education and workplace datasets to improve estimates of adult attainment and study the impact of education on employment and benefits outcomes.

8 Comments

  1. Deb April 17, 2018 at 3:00 pm - Reply

    Interesting article, brilliant that we can look at the report in FFT Aspire too.

  2. Nick April 18, 2018 at 11:36 am - Reply

    As the data manager at a school with one of ‘the 30’ feeder primary schools mentioned, this report is so helpful to me.
    Every time we have an OFSTED data meeting I bring up this issue. Every time I get the ‘all secondary schools say this’ response and it then takes me several hours to ‘persuade’ them that this is a real issue and that our overall progress figure (now P8 but previously 3 and 4 levels progress) would be so much better. It obviously impacts hugely on our P8 figure for ‘High PA’ pupils as all of these pupils are being judged as level 5. Ofsted come into the school with the preconceived idea that the High ability students are performing very badly.

  3. Steffan Woodhead April 18, 2018 at 11:46 am - Reply

    Is this report available in Wales? I can’t seem to find it.

    • Dave Thomson April 19, 2018 at 9:58 am - Reply

      Hi Steffan. No, it’s currently only available for England (as Progress 8 doesn’t apply in Wales).

  4. Stephan April 19, 2018 at 9:30 am - Reply

    It’s interesting that the immediate reaction to high end of KS2 test scores is that the primary school is somehow fiddling results. I would ask how the test scores correlate with the end of KS2 teacher assessment, what the secondary schools baseline assessment on entry for these pupils looks like etc before deciding that these results can be deemed inaccurate (with the possible consequence that these pupils are not given the opportunities they need to fulfil their potential).

    • Clive April 20, 2018 at 10:50 am - Reply

      I have been beating this drum for a while
      The issue has always been the difference between the KS2 score and their CAT score.
      Plus when we internally test the skills of the pupils with our own version of a SAT paper we notice that the same pupils underperform against the expected result you would have by looking at the score they achieved at KS2.
      But always came up against a brick wall when you tried to discuss this with OfSted or County.

  5. Chris April 19, 2018 at 5:53 pm - Reply

    Is this a pdf report that is already generated ie is available in the ‘downloads’ from self evaluation > see all downloads area? That one is black and white only; I am just wondering if I am meant to be able to generate this myself somewhere within aspire.

    • Dave Thomson April 20, 2018 at 3:27 pm - Reply

      Hi Chris. Yes you go to “see all” in the downloads section. On the reports for schools the funnel plots are in grey, so if you’ve managed to find a 3 page pdf with the first page titled “Progress 8 by Feeder School (2017)” then you’ve found it.

Leave A Comment