Angles on the Algorithm

The Algorithm according to the Government, Ofqual, Exam Boards, Schools and Me.

Government

On 25th February Gavin Williams’ speech to the House of Commons included:

“I can confirm that no algorithm will be used for this process. Grades will be awarded on the basis of teachers’ judgment and will only ever be changed by human intervention.” 

The DfE Statement followed with this on 25th February:

As well as the checks of schools’ and colleges’ quality assurance processes, exam boards will complete checks of the evidence for a sample of student grades in a sample of subjects, in a sample of schools and colleges over June and July.

But crucially please also note the statement included:

A check will also be triggered:

if a school or college’s results are out of line with expectations based on past performance.

So, is there an algorithm in 2021? 

The government is saying: not really but we encourage you to base outcomes on past performance.

Ofqual

Recent Ofqual guidance states:

In addition, exam boards will carry out more detailed checks of a sample of centres, reviewing the evidence for one or more subjects. Some centres will be selected from a random sample that is representative of different centre types and some will be selected based on risk.

In a normal year once exam boards have awarded provisional grades, the grade distribution is sent to Ofqual with a comparison to the predicted grades. Ofqual then allows that grade distribution or asks the boards to reconsider their awards.  This keeps grades broadly in line year on year. 

21st July has been the date scheduled for the data exchange to take place this year but this may be brought forward.

All I have read at this stage suggests that Ofqual will not intervene in 2021.  So 21st July will be when the DfE will have a good picture of what has happened. The boards will work hard to keep grades in line to protect the validity of grades and will also need to remember that the 2023 qualification and discount codes list is yet to be published.   

So, is there an algorithm in 2021? 

Ofqual is saying: not really but what does ‘selected based on risk’ mean?

Exam Boards 

On 26th February AQA published its response to the 2021 proposals:

Quality assurance

Schools' or colleges' internal quality assurance arrangements will need to include consideration of their profile of results in previous years as a guide to help check that judgements aren’t unduly harsh or lenient.

This seems an explicit instruction for schools to include the use of an algorithm in their internal QA process.

So, is there an algorithm in 2021? 

AQA is saying: yes but giving vague guidance about what it is.

Schools

The light touch approach from DfE puts a huge amount of pressure on Head Teachers.  They are to be judge, jury and executioner.  The whole reason we have no external exams is because the disruption to learning varies greatly between pupils so that standardisation is impossible. 

Mr Freedman tweeted:

I'm disappointed that a significant number of educationalists don't seem to realise that it is not possible to make impartial judgements under the proposed system. Nothing to do with teachers' integrity or ability. They're being asked to do something that can't be done.

Schools though as ever rise to the challenge and I urge you to take a few minutes to listen to Mr Benson outline with clarity and reassurance how they will sort the 2021 exams situation.

Conclusion

Schools are advised to use algorithms – but, as Gavin stated, the grades will only be changed with human intervention and as part of their extensive QA process. Last year most schools and colleges used broadly the same algorithm; this year there is flexibility to make up your own. 

What do I suggest: I would use the same calculations used in 2020, using the 2017, 2018 and 2019 prior results plus cohort prior attainment – there’s no need to re-invent the wheel. Collect the TAs and as part of the QA process, compare them to last year’s model.  Where there are significant differences, use this to have a closer check, alongside the rest of your QA process.  

So, is there an algorithm in 2021? 

I am saying yes but with a sensible local human touch.

 

 

 

 

Exams 2021: the circle that cannot be squared

3 imperfect solutions

I’m well aware of the rigour with which exam marking is undertaken every year. It begins with a meticulous standardisation process to establish ‘the standard’ which all examiners must maintain throughout. ‘Seeded’ responses hidden within the mass of live marking must be marked accurately if an examiner is permitted to continue. Next there is the precise and detailed annotation of each individual response, justifying the final mark given by the examiner. Outcomes are then benchmarked against a well established statistical model based on expected progress of the national cohort and then, finally, they are given the ok by Ofqual, acting on behalf of the Department for Education. 

The unprecedented situation (ie no exams) in which we find ourselves for the second year means that when deciding final grades for this unfortunate cohort that rigour is inevitably compromised. The pandemic has led to lots of pupils losing hours and weeks of teaching and learning. Pupils have accessed the curriculum in lots of different ways.  Even asking the exam boards to supply mini tasks doesn’t really help.  Inconsistency, unfairness and injustice prevails.  

It’s an impossible situation with no perfect solution. How do we ask the exam system for a consistent and fair approach when rewarding this cohort with the grades that impact the rest of their lives? 

These are my imperfect solutions to that dilemma:

● Key Stage 4 students: focus on destinations; local schools and colleges work collaboratively to get students on to the right courses, using the year 9 GCSE options process as a model, no grades;

● Key Stage 5 students: undertake tests of some sort which is moderated nationally with the possibility of exam resits remaining possible a year later;

● Make the third year of Key Stage 5 fully funded. It is currently funded but at a reduced rate of £3455 in band 4a.  Change this to £4188 and allow it to be in band 5.  This allows for some year 13s to study for an extra year and for some year 11s to study level 2 for an extra year before they move to level 3 courses in year 13 and 14.

And I’m going to say it: let’s part finance the above by not giving the exam boards their annual £200 million payment.  They will just have to live off last year's profits.