andallthat.co.uk
  • Blog
  • GCSE / A Level Topics
    • America 1789-1900
    • Antisemitism
    • British Radicalism 1789-1900
    • Crusades
    • Elizabeth I
    • Germany 1919-45
    • Historical Interpretations
    • Historic Environment
    • International Relations 1900-2000
    • Italian Renaissance
    • Medicine
    • Medieval Kings
    • Russia 1855-1921
    • Soviet Russia
    • Politics Files
  • KS3 Topics
    • Interpretations
    • Stand-Alone Lessons
    • NEW KS3 File Store
    • OLD KS3 File Store
    • Student Resources
  • Advice
    • GCSE Options
    • A Level Options
    • Personal Studies
    • University Applications >
      • Choosing Courses
      • General Advice
      • Personal Statements
      • Predicted Grades
  • Teachers
    • YHEP Teach Meet
    • Stand-Alone Lessons >
      • Ancient World
      • Medieval World
      • Early Modern
      • Industrial Revolution
      • Modern World
      • Post-Modern World
    • KS3 Teaching Resources
  • MeetTheHistorians
  • Trips
  • About
    • SubBlog
  • Contact

The Progress 8 Fallacy - Why P8 Results Don't Prove Grammars Work!

10/17/2016

0 Comments

 
Picture
I'm afraid today's blog, a bit like yesterday's is a bit of a rant. More to the point, it is a response to yet another partially researched claim, namely that Grammar Schools (Schools for Everyone TM?) are more effective than their comprehensive counterparts. Indeed, the Telegraph ran an article a few days ago stating that this was a ringing endorsement for May's flagship education policy. Here I wanted to unpick some of the claims being made about Grammar Schools, and ask that we take a moment to be cautious before endorsing a systemic change based on limited evidence. I am fairly sure I have my calculations right here, however please let me know if you think I have made a mistake.

The Progress 8 Issue
The claim that Grammar Schools outperform state Comprehensives does have some basis in evidence. This can be seen in the latest GCSE statistics published by the DfE. The Telegraph explains that...
Under the new Progress 8 measure, the national average for progress is 0 for mainstream schools. Today’s figures show that the average Progress 8 score for all grammar school pupils is 0.33, compared with -0.01 for comprehensive schools.
Here they imply that the Grammar School score of 0.33 shows that Grammars significantly outperform Comprehensives. On the surface this is true, on average, Grammar Schools did outperform state Comprehensives. However there are a number of very good reasons to be cautious here.

Before I go any further, I think it would be worth explaining a little more about Progress 8. Essentially the measure aims to see how far pupils have progressed between Key Stage 2 and Key Stage 4. Each pupil is ranked by their KS2 attainment using a "fine level"; which might be 4.2, or 5.6 for instance. In the example from the DfE below, Gillian has a KS2 fine level of 5.1. Once GCSEs have been taken and marked, the mean average Attainment 8 score of all pupils with a fine level of 5.1 is found (in this case 59.8). This number will change every year depending how children perform. The mean average now becomes the benchmark against which all pupils are measured. Their score against this mean average is their Progress 8 score. In Gillian's case, she scored significantly above the average for pupils with a fine level of 5.1, so her Progress 8 score is +0.72. This means she scored nearly 3/4 of a grade better than other children with a 5.1 fine level at KS2 nationally. A child who scored -0.5 would have made half a grade less progress than children with the same fine level as a staring point. Effectively children are norm referenced by their KS2 fine level and are in direct competition with other children on the same fine level starting point - not everybody can succeed! In fact, the average Progress 8 for each fine level should come out around 0, forming a lovely bell curve.
Picture
Picture
A school's Progress 8 score is calculated by adding all the individual students' Progress 8 scores together and dividing by the number of students. Nationally speaking, Progress 8 for all state schools comes out at zero due to this norm referenced approach.

Are Grammars Performing Better?
With all this taken into account, Grammar Schools scoring 0.33 does appear to show that they are more effective than state Comprehensives when moving students from their starting points. Certainly it means that there are Comprehensives in which higher ability children are under performing on average compared to Grammars. However, the following should be taken into account:
  1. Because Comprehensive Schools account for 91% of all the schools compared by the government, and the system is norm referenced, it is entirely natural that they should come out with a Progress 8 score on or around 0. This is because the whole system is set to default to 0. The 163 Grammar Schools represent a tiny sample of this much larger set. The fact they score 0.33 suggests they are having an impact, but comparing this to Comprehensives is almost meaningless using Progress 8 as the measure. Education DataLab go into more detail on why Grammars have things weighted in their favour HERE
  2. Grammar Schools take substantially fewer children who have free school meals status. The Sutton Trust found that 66% of children with Level 5 in English and Maths went to Grammars in selective areas compared to 40% of similarly high achieving pupils with FSM. As FSM has a known impact on outcomes, this may go some way to explaining part of the success of Grammars.
  3. Grammar School coaching is a thriving industry. The fact that many students who attend Grammar Schools have already shown an aptitude for testing and are likely to have been supported by parents in passing their entrance exam also suggests some possible impact. Parents who spend money to coach their children for the 11+ may also spend money on tuition for GCSE examinations. More research needs to be done into this, but it is certainly a salient issue.
  4. I would contend that Grammar Schools are more likely to enter their students for Ebacc subjects. This is vital as without a suite of Ebacc subjects, many students effectively throw away 3 qualifications which could count towards their overall grades. Pupils in a Comprehensive which allows free choice of subjects are therefore likely to score lower than those in schools which force Ebacc choices. This puts the first group of students at a significant disadvantage when comparing Progress 8 scores.
  5. Grammar Schools are not directly comparable to the outcomes for other Comprehensives as they do not cater for the same range of pupils - there is much hidden in the numbers (more on this later). 
  6. High achieving Comprehensives are weighed out by low achieving ones. There are many factors which might make a school low achieving which are not captured in this debate.
  7. When I compared local Grammar Schools to Comprehensives with high SES and high ability intakes, the differences were much less clear cut. In the example below you can see that the highest performing school in North Yorkshire is Harrogate Grammar (a Comprehensive), which fits the high prior attainment and high SES profile discussed. Another 4 schools come before the first Grammars: Ripon and Skipton Girls'. Ermysted's Grammar gets a P8 of just 0.9 - notably it enters only 46% of its students for Ebacc compared to Harrogate Grammar's 73%, potentially illustrating the Ebacc effect. 
Picture
Profile is everything
Now here is the really big issue with Progress 8. It boils everything about a school down to a single measure. Ripon Grammar might for example argue that it is doing better than Selby High (P8 -0.2) and  this is true on average, but the important comparison would be a profile one. Because the two schools have very different intakes, it is impossible to directly compare their effectiveness at teaching high end students. A simplistic view suggests that this is what is happening. The chart below shows the view portrayed by the Progress 8 figure. It suggests that students of all abilities are scoring the same in terms of P8 (see below).
Picture
If this were the case, then the Grammar School really would be outperforming the Comp on comparable measures. However, the profiles to get these results might look very different. Take the example below.
Picture
In this example, Grammar 2 averages out with a P8 of around +0.38 (having some high ability students and a good deal of v high). Comp 2 however has chosen not to push for Ebacc choices and has therefore under performed with its middle ability students (the bulk of the school). Despite the fact they have done very well with their weakest and exceptionally well with their higher ability students, these are outweighed by the significant proportion of middle ability students. As a result, such a school might score -0.2 on their Progress 8, despite doing a better job of teaching the higher end than the Grammar. 

By the same token, a Comprehensive might boost its middle ability students at the expense of its top end, and yet appear to be very effective overall. Here is an imaginary comparison of Grammar 2 above (0.38 average) with an imaginary school scoring +0.67 (and thereby appearing better on paper) thanks to its work with middle ability students who form the bulk of its intake. As you can see, the profile is everything! 
Picture
So there we have it. I could go on at length here on what kinds of things schools might do to get a more favorable Progress 8 outcome, but I shall leave it there. I just ask that we think carefully before using a low-resolution Progress 8 statistic to prove any kind of educational point.
0 Comments



Leave a Reply.

    Tweets by @AndAllThatWeb

    Archives

    March 2019
    February 2019
    January 2019
    August 2018
    July 2018
    January 2018
    December 2017
    June 2017
    May 2017
    February 2017
    January 2017
    November 2016
    October 2016
    September 2016
    July 2016
    June 2016
    May 2016
    April 2016
    March 2016
    February 2016
    January 2016
    November 2015
    October 2015
    August 2015
    July 2015
    June 2015
    May 2015
    April 2015
    March 2015
    February 2015
    January 2015
    November 2014
    October 2014
    June 2014
    May 2014
    March 2014
    February 2014
    January 2014
    December 2013
    November 2013
    October 2013
    July 2013
    June 2013
    April 2013
    February 2013
    December 2012
    October 2012
    September 2012
    July 2012
    June 2012
    April 2012
    March 2012
    March 2011
    February 2011
    June 2000

    Categories

    All
    2014 KS3 Curriculum
    Book Review
    BUSK Reading
    Christmas
    Comment
    History Ancient
    History C19th
    History C20th
    History Early Modern
    History Medieval
    History Thematic
    PGCE
    Rant
    Teachers Assessment
    Teachers Case Study
    Teachers Classroom Management
    Teachers Concepts
    Teachers Conference Notes
    Teachers Curriculum
    Teachers Exams
    Teachers Frideas
    Teachers Government
    Teachers Leadership
    Teachers Misc
    Teachers NQT
    Teachers Pedagogy
    Teachers Planning
    Teachers Progression
    Teachers Purpose
    Teachers Stand Alone Lessons
    Teachers Technology
    Teachers Training
    Teachers Trips
    Teaching Personnel
    Tvfilm Reviews79f7bb4075

    RSS Feed

Powered by Create your own unique website with customizable templates.