Angles on the Algorithm

The Algorithm according to the Government, Ofqual, Exam Boards, Schools and Me.

Government

On 25th February Gavin Williams’ speech to the House of Commons included:

“I can confirm that no algorithm will be used for this process. Grades will be awarded on the basis of teachers’ judgment and will only ever be changed by human intervention.” 

The DfE Statement followed with this on 25th February:

As well as the checks of schools’ and colleges’ quality assurance processes, exam boards will complete checks of the evidence for a sample of student grades in a sample of subjects, in a sample of schools and colleges over June and July.

But crucially please also note the statement included:

A check will also be triggered:

if a school or college’s results are out of line with expectations based on past performance.

So, is there an algorithm in 2021? 

The government is saying: not really but we encourage you to base outcomes on past performance.

Ofqual

Recent Ofqual guidance states:

In addition, exam boards will carry out more detailed checks of a sample of centres, reviewing the evidence for one or more subjects. Some centres will be selected from a random sample that is representative of different centre types and some will be selected based on risk.

In a normal year once exam boards have awarded provisional grades, the grade distribution is sent to Ofqual with a comparison to the predicted grades. Ofqual then allows that grade distribution or asks the boards to reconsider their awards.  This keeps grades broadly in line year on year. 

21st July has been the date scheduled for the data exchange to take place this year but this may be brought forward.

All I have read at this stage suggests that Ofqual will not intervene in 2021.  So 21st July will be when the DfE will have a good picture of what has happened. The boards will work hard to keep grades in line to protect the validity of grades and will also need to remember that the 2023 qualification and discount codes list is yet to be published.   

So, is there an algorithm in 2021? 

Ofqual is saying: not really but what does ‘selected based on risk’ mean?

Exam Boards 

On 26th February AQA published its response to the 2021 proposals:

Quality assurance

Schools' or colleges' internal quality assurance arrangements will need to include consideration of their profile of results in previous years as a guide to help check that judgements aren’t unduly harsh or lenient.

This seems an explicit instruction for schools to include the use of an algorithm in their internal QA process.

So, is there an algorithm in 2021? 

AQA is saying: yes but giving vague guidance about what it is.

Schools

The light touch approach from DfE puts a huge amount of pressure on Head Teachers.  They are to be judge, jury and executioner.  The whole reason we have no external exams is because the disruption to learning varies greatly between pupils so that standardisation is impossible. 

Mr Freedman tweeted:

I'm disappointed that a significant number of educationalists don't seem to realise that it is not possible to make impartial judgements under the proposed system. Nothing to do with teachers' integrity or ability. They're being asked to do something that can't be done.

Schools though as ever rise to the challenge and I urge you to take a few minutes to listen to Mr Benson outline with clarity and reassurance how they will sort the 2021 exams situation.

Conclusion

Schools are advised to use algorithms – but, as Gavin stated, the grades will only be changed with human intervention and as part of their extensive QA process. Last year most schools and colleges used broadly the same algorithm; this year there is flexibility to make up your own. 

What do I suggest: I would use the same calculations used in 2020, using the 2017, 2018 and 2019 prior results plus cohort prior attainment – there’s no need to re-invent the wheel. Collect the TAs and as part of the QA process, compare them to last year’s model.  Where there are significant differences, use this to have a closer check, alongside the rest of your QA process.  

So, is there an algorithm in 2021? 

I am saying yes but with a sensible local human touch.

 

 

 

 

Exams 2021: the circle that cannot be squared

3 imperfect solutions

I’m well aware of the rigour with which exam marking is undertaken every year. It begins with a meticulous standardisation process to establish ‘the standard’ which all examiners must maintain throughout. ‘Seeded’ responses hidden within the mass of live marking must be marked accurately if an examiner is permitted to continue. Next there is the precise and detailed annotation of each individual response, justifying the final mark given by the examiner. Outcomes are then benchmarked against a well established statistical model based on expected progress of the national cohort and then, finally, they are given the ok by Ofqual, acting on behalf of the Department for Education. 

The unprecedented situation (ie no exams) in which we find ourselves for the second year means that when deciding final grades for this unfortunate cohort that rigour is inevitably compromised. The pandemic has led to lots of pupils losing hours and weeks of teaching and learning. Pupils have accessed the curriculum in lots of different ways.  Even asking the exam boards to supply mini tasks doesn’t really help.  Inconsistency, unfairness and injustice prevails.  

It’s an impossible situation with no perfect solution. How do we ask the exam system for a consistent and fair approach when rewarding this cohort with the grades that impact the rest of their lives? 

These are my imperfect solutions to that dilemma:

● Key Stage 4 students: focus on destinations; local schools and colleges work collaboratively to get students on to the right courses, using the year 9 GCSE options process as a model, no grades;

● Key Stage 5 students: undertake tests of some sort which is moderated nationally with the possibility of exam resits remaining possible a year later;

● Make the third year of Key Stage 5 fully funded. It is currently funded but at a reduced rate of £3455 in band 4a.  Change this to £4188 and allow it to be in band 5.  This allows for some year 13s to study for an extra year and for some year 11s to study level 2 for an extra year before they move to level 3 courses in year 13 and 14.

And I’m going to say it: let’s part finance the above by not giving the exam boards their annual £200 million payment.  They will just have to live off last year's profits.

Progress is Dead! Long live Progress!

3 Ways to Report Outcomes in January 2021

Ten years ago Ofsted said a school had to show progress.  National Curriculum levels made it a straightforward process.  Now Ofsted no longer requests it and the pandemic means national baselines are disappearing.  

So what to do?  Here are my top 3 ways to show progress currently in a helpful and fair way:

1. Continue with mini low impact assessments to inform your overall age-related attainment.

2. Make it age-related attainment e.g. Pupil is Working Towards, Working At or Working Beyond at the end of the year or working at grade 3, 5, 7 standard.

3. Benchmark it contextually to national outcomes (2019).

GCSE Maths 2018

Those that have read my posts know I tend to concentrate on GCSE English but after Ofqual's recent report I have to turn my attention to Mathematics in 2018.

Ofqual 2018 GCSE report


Ever since August 2012 I have followed GCSE outcomes.  That summer was the year that Ofqual and the Dfe changed GCSE outcomes and really put an end to years of grade inflation. It was a summer that changed my view on the "system".  A system that at times appears to be unfair.


My 2015 Blog

Ofqual then set about setting up a system of comparable outcomes.  How it now works is that every year the exam boards in July send there outcomes to Ofqual with the expected outcomes based on Progress from Key Stage 2.  The exam boards are under extreme pressure to make sure the outcomes match expected.  Ofqual now has a strangle hold over the boards.  This is achieved by the importance of "the list".

The List

To gain points that count towards Progress 8 schools have to choose only subjects that are approved by Ofqual and added to the list.  If a course is not on the list it will cost the exam boards a great deal of money.  (I will not on this blog talk about how bad it is that the list is produced so late - that is so obviously wrong and shows how Ofqual have no idea how long it takes to design and implement a curriculum, you will have noticed that 2021 has not yet been published and options evening will start after Christmas).

So we move to 2018 Maths.  Ofqual spotted the problem of comparable outcomes based on assessments that are 5 years old and some state are unreliable.  They then introduced at additional costs the National Reference Tests to be sat by year 11 Pupils that could be used to "allow" exam boards to increase pass rates if pupils performed at a higher level.  In 2018 Ofqual stated "In maths, NFER report a statistically significant change, which suggests that student performance has improved slightly."

National Reference Tests 2018

In the same set of Reports Ofqual published the comparability graded for Maths.  This showed that for the two boards with most entries for Maths they exactly equalled the expected outcomes from Key Stage 2.

The progress remained the same and the outcomes only went up by 0.3% not the 2.4% that could have been expected,  this equates to 10,329 pupils receiving a grade 3 rather than a grade 4.

The impact on the lives of these pupils is significant.  I understand the "second year" effect but at Key Stage 2 in the second year outcomes were increased to reflect this.

The question is whether the millions of pounds being paid to exams boards and our exam board regulators Ofqual is value for money when it does not appear to be delivering a fair system.

Assessment Manager Set Up and Training

If you would like help setting up or using Assessment Manager more effectively do get in touch.  A large part of my work is creating Excel ad-dons that automatically create the dashboards that many schools find useful.

SIMS Assessment Manager has the flexibility to support almost any assessment system.  It's power comes from its direct link to the timetable and other pupil details held in SIMS.

Set up can take some time but once done is a good solution to both whole school and departmental assessments.

Do contact schooldatamanaged@gmail.com for a very competitive rate.

Progress 8 in September 2017

Progress 8 in 2017

We use the Department for Educations Progress 8 to measure progress.  This measures the actual attainment 8 score against an estimated attainment 8 score.  The estimate is based on the pupils’ Key Stage 2 reading and maths scores combined. 

Attainment 8

Attainment 8 is made up from Maths and English (Highest from English Language or Literature), the three highest Ebacc exam grades and the three highest non Ebacc subjects count. The non Ebacc bucket can contain English Language or Literature or remaining Ebacc qualifications.  If a bucket/qualification is not filled it counts as zero. English and Maths are double weighted. It effectively becomes 10 buckets/qualifications.

All qualifications have to be approved by Ofqual. English and Maths have to be the new reformed qualifications.

In 2017 new points have been introduced.  The system has been designed to accommodate the transitional two years when GCSEs are awarded in both grades and numbers.

There is a significant difference in points that can be awarded.

In 2016 the highest attainment 8 score was 80

In 2017 the highest attainment 8 score can be 87.5

In 2018 it will go up to 90 for most pupils  but be 89 (approx.) for pupils who have unreformed subjects.

A pupil achieving E grades will have a lower attainment 8 outcome in 2017 form 2016.

In 2016 the score would be 30 points

In 2017 the score could be 25 points.  (The pupil needs to achieve a 3 in reformed GCSEs to achieve this.)

The outcomes for 2017 for reformed subjects were in line with 2016. 

Ofqual ensured that at G+/1+, C+/4+ and A+/7+ benchmarks the proportions achieving each based on prior attainment remained in line with 2016.

Progress 8

Progress 8 is the difference between a pupils estimated Attainment 8 and actual Attainment 8 divided by 10 (English and Maths is double weighted). The Attainment 8 estimates are based on the Key Stage 2 fine level.  In 2017 pupils took the Key Stage 2 in 2012 and the fine level is based on the average from their Reading and Maths Key Stage 2 Scores. This score is used to give attainment 8 estimates.

In the above example Gillian’s Key Stage 2 score gave her an estimated attainment 8 of 59.92.  She actually scored 67 points.  The difference is then divided by 10.

A progress 8 score of 0 is in line with national progress of pupils from a similar starting point. A score of +1 in 2016 represented progress on average for all the 8 subjects at one grade higher than expected.  As shown on page 1 the scoring has changed and 1 point does not now represent a grade for non-reformed subjects.  A pupil who achieved a B rather than a C scores 1.5 points, however a pupil who achieved an E rather than an F scores 0.5 points. This means that the impact of gaining a grade higher or lower at the top end is greater than at the lower end. 

Additional Impacts on Progress 8 2017

1.    2017 was the first of the Reformed GCSEs.  This has made predictions difficult as it is a new syllabus with a new points system.  In English the IGCSE was not approved by Ofqual,   therefor over 200,000 leaners moved to the reformed GCSE from this qualification.  The vast majority moved to AQA who saw there entries double in 2017. They were busy.

2.    To fit in line with Attainment 8 the number of entries for Science and Geography increased in 2017.  This will means the number of Ebacc buckets populated has increased and will impact the estimates.

3.    2017 is the last year that Ofqual have approved the ECDL qualification.  Ofsted have raised concerns about entering whole cohorts. Schools with Lower Prior attaining cohorts that have done this can add up to 0.2 to the progress 8 score.  The issue is if you have not done this then your scores will go down. The Attainment 8 estimates are based on comparisons to others. We do not have the number of entries but in 2016 it was 90,000.

4.    It is worth noting that in 2012 Key stage 2 scores were half a sub level higher than in 2011. This will also negatively impact any estimates calculated.

5.    SISRA and 4Matrix have provided schools with estimates based on the results that they received from schools.  These must be used with caution. Provisional attainment 8 estimates will be released at the end of September.

6.    The Department for Education’s Floor target is -0.5.  The Coasting score is -0.25, this is over a three year period.

IGCSE 2016

Last year the impact of IGCSE English results without any reporting tolerance added to them was significant on the overall outcomes for all learners.  In the summer action was taken by the great and good to ensure that IGCSE English outcomes were kept sensible though this had varying consequences for pupils and schools.  This year the problem has started early and our friends at the Dfe need to take a wee look at it.

Ofqual have helpfully as ever released this 
 https://www.gov.uk/government/statistics/november-gcse-and-level-1level-2-certificates-entries-in-England

I have asked Ofqual to provide the number of entries for the last 3 years for November IGCSE.  The Cambridge International Examinations Board refused to give me this information.  The claim at a PIXL conference was that schools were reporting an 80% pass rate from November IGCSEs.  It actually came in at 71.1%   These results were not compared to Keystage 2 results and were certainly not assessed against the new national reference tests.

Over 41,000 pupils from this Cohort have taken this exam in November.  This means almost 30,000 pupils have already passed the exam.  This will be approximately 8% of the cohort.  I do not know if the IGCSE will this year be subject to reporting tolerance in the summer of 2016 - it has to be if the system is to retain any hint of fairness.

You may be thinking that the November Series GCSE is for re-sits only but as Ofqual point out in the document.  "The re-sit only rule in November does not apply to Level 1/ Level 2 certificates, and students may still enter for a Level 1/ Level 2 certificate for the first time in November."

So what happens from here.  The pupils who sit GCSE English in the summer 2016 should be treated fairly.  They were not in 2015.  All exams should be made to go through the reported tolerance exercise.  This year the progress 8 calculation for schools has to be done twice -once including IGCSEs and once without.  It was so unfair that Raise on Line was published last year with absolutely no reference to the English GCSE farce of the Summer of 2015.

I await the response from the great and good of Education.

Chris Beeden