This week I released a video on the Examiner’s Report for the 2017 AQA English Language GCSE. The resulting Twitter spat amongst English teachers reveals some issues with our profession’s attitudes to the purpose and validity of exams.
There were three principal issues.
- If we show students how to answer questions on a specific paper, they will use these and get better marks when they sit that specific paper in a mock.
- Schools won’t be able to judge the grades of their students during mocks, because students might have watched my video on a particular question, or indeed all the questions. I am undermining the whole system by undermining the mock.
- The examiner’s report and mark scheme is published by AQA on their eAQA site, not available to the general public, so I am breaching copyright, and I am an objectionable human being, who does not understand there is no ‘I’ in team.
Although there is some validity to the third of these, I’ll beat myself up about it later.
The first two points are a huge concern, and reveal how far we have moved, as a profession of English teachers, and perhaps a profession of teachers, from our principal role: making our students great at our subject.
How Valid is a Mock?
English Marking is Not Very Accurate
To quote Dylan Wiliam in the September 2017 edition of Impact, “as Lee Cronbach pointed out almost 50 year ago, an assessment is simply a procedure for drawing inferences (Cronbach, 1971)”.
If we want to draw inferences about the validity of a mock paper, there is a real problem, which is the huge margin for error. “A typical school test would have a margin of error of 10 marks or more” says Wiliam.
An English paper would have a much higher margin for error. This is a blog by Dr Chris Wheadon it delivers a devastating factual account of the problems of marking the English papers:
“When a 24 mark item is marked it is normal for one marker to give 9 marks, and another to give 15 marks which is the difference between ‘Simple, limited’ and ‘Some success’.
To be clear, Ofqual’s research is based on the differences that are normal for established specifications, after marker standardisation, and with the use by exam boards of sophisticated statistical rules which stop poor marking as soon as it is detected.”
Because not all our teachers in a department will have been trained as markers, and because we won’t have the same level of standardisation, our own marking of mocks will have a far greater margin of error. With examiners themselves, the gap is potentially 25% of the available marks. How many grade boundaries do you think that covers? How confident are you that your department is better than a team of trained examiners?
In other words, when we mark a mock, we cannot be certain of a mark, and we can only arrive at a best guess at a grade. We certainly can’t rely on these with any certainty.
Do Your Mock Results Act as Feedback for Your Students, or for the SLT?
The grades we report to students and more importantly, in many teachers’ eyes, to SLT are to some extent made up.
However, if we were teaching with learning in mind, this would not matter. We would acknowledge that the grade is just not that reliable. We would simply tell our students, “these are the skills which I don’t think you are reaching in each question. Practise them to get better at them.” We would always be right in the diagnosis of these missing skills. Where we go wrong is saying where each skill would place the student in the mark scheme. As to your grade, it is 5, 6 or 7.
To take just one example of the mark scheme, level 4 in the 8 reading questions always includes the key words ‘perceptive’ and ‘detailed’ as two of the three key words that examiners use to define the level.
You have a long answer, with PEE paragraphs, which take a long time to make an argument, but the student has lots of them. Is this detailed, or does it fail the perceptive test? Maybe it makes the band, maybe it doesn’t.
But if the answer is concise, integrating quotations into an argument, the student doesn’t have to write many paragraph to be detailed – being concise means each sentence will be detailed, integrating more than one quotation and often signposting more than one explanation. This will also meet the perceptive criterion. Boom, top band.
The objection to my video suggests that teachers are not doing mocks primarily for this purpose, it suggests they are not doing it primarily for their students – they are doing it to get a grade to see if the class, cohort and school are on track. This is not information which is useful to the student. It is not feedback. It is arse covering.
Does Your Feedback From the Mock Make Much of a Difference Anyway?
Another problem of a student getting a low grade in a question is understanding the real problem:
Was it that they didn’t revise, didn’t read the question properly, didn’t know what quotations to use, didn’t know how much time to spend on the question, didn’t proof read, don’t know how to construct a coherent argument or simply didn’t think it was required, didn’t evaluate, or simply didn’t know they had to evaluate…etc.
This means the feedback we give the student is often the wrong feedback – we’ve diagnosed the wrong problem with an incorrect inference.
Converesly, it might be the right feedback, but the student dismisses it as not relevant – “If I’d revised, I wouldn’t have made that error…of course I can embed quotations, or evaluate, I just forgot to…”
From the student’s perspective, much of the mock can be a waste of time. This is compounded by not being able (or willing) to resit it two or three weeks later, to see what they have actually learned. The logical consequence of feedback to the mock would be to get your class to resit it. “I’ve given you feedback – I’ll know how much you’ve learned by how close you get to 100% when you do the same test again.”
I doubt very much that you do this. Instead we hope the feedback has had an impact. We just don’t take the time to find out.
Retaking the same test would actually be a much more valid test of your feedback. If students have learned from it, we would expect them to get 100%. Where they fail to get 100% on a question, we would then diagnose a problem that was real – students would know that they had seen the correct answers, that they thought they understood why the answers were correct. But now, given exactly the same questions, they have not got 100%.
Ok, now they will know that any feedback is directly relevant to them. It is totally real. The marks they lost are for skills and knowledge which they simply lack. They would know precisely what to revise through practice.
This is what my videos do. First, they feed forward, showing students exactly what the skills they need are, not just generically, but with this particular text. Then they try the test on that text to see how they do.
What if Some of Your Students Have Watched my Videos, Seeing the Answer First Before Your Mock?
Granted, they will also get higher marks than they would have without watching the video, but they will factor this in to their feedback – “I know I had some help, so I am not yet at this grade. I will still have to practise another question to see what I have actually mastered, and to retest myself on the skills I know I missed (because I didn’t get 100%).”
It is a legitimate complaint that the teacher won’t know who has watched my video, although at 250 views on the first day it is unlikely to unbalance a mock in a student population of over 700,000! If you are worried that it could be your kids, well ask them.
“How many of you watched Mr Salles? Ooh, all 30 of you! How unexpected. How many marks do you think you gained by seeing the video first? Let’s take 10 marks away then.”
But this misses the point. Watching my videos before doing the mock would actually create a better mock, because it would give the student and teacher better feedback about what they don’t yet know. This is what the mock should be for. The problem is, for many/most teachers the mock seems to be about what feedback they can give their SLT – “look, 79% of our cohort are already working at grades 9-4!”
A better solution would be to make all students watch my videos first (and tell them not to watch the adverts – that way I don’t make any money from this request!)
Mocks are not the Best way to Prepare for the GCSE Exam Anyway
Let’s be honest. Your students do a mock. It takes two weeks minimum to turn around the marking. In many schools, a marking team does it, so you don’t even mark all your own answers. Even marking your own, you probably spend only one lesson going through the feedback. How many questions do you get your students to actually redo? One, two, three? Out of 10 language questions!
So how much difference will the feedback you gave them on their mock make to your students’ marks in the real GCSE – will they even remember the improvement they made in February when the exam comes round in June?
It is very easy to lie to ourselves about this, but the answer is very little. Best case scenario, the feedback does not arrive too late, the student does actually redo the question (maybe even two or three), your feedback (rather than just having another go anyway) did contribute to them getting a higher mark. What is the effect on their overall grade in the real thing?
Not so much. Even 3 questions out of total of 10 isn’t going to lead to rapid progress.
However, if I make them watch a 20 minute video on each of 8 of the 10 questions, they have spent at least 2 hours and 40 minutes revising for the reading part of the language exam before they even take the mock! Now you can set them a much higher pass mark – “you’ve seen the answers kids, you’re in a middle set, so the pass mark is 75%. You’re in a top set, so your pass mark is 90%.”
How many of your middle to bottom sets did more than 2 and a half hours revision for the whole of their language mock? Be honest. How many did that much revision for the whole of the final exam?
And if they still need feedback when they get less than that 75% or 90%, they can simply replay the video to find out where they went wrong.
What Does the Research Tell Us About Mocks?
Some of you will not find this argument compelling. It will be much more important to you to ascribe grades, even though we have already established that the grades have a high chance of being wrong, and that the way you mark and deal with the mock is too time consuming to be of wholescale benefit to your students.
Fine, be that way.
But here’s the kicker. It doesn’t matter what you think, and it doesn’t matter what I think.
Research is our Rock.The answer is already out there.
You are wasting your time doing a full mock anyway. Click on the following links to find out about spaced and massed practice, and why mocks are not the best way to prepare for the exam.
Spaced Learning, Massed Practice V Spaced Practice
So the whole premise of doing a mock is wrong. Students will learn far more if we give them the same style of question at three week intervals, or progressively greater intervals.
How would that work?
- Watch my video.
- Do the question three weeks later.
- 3 weeks after that, do the same question from Specimen paper 1.
- 3 weeks later do the same question from Specimen Paper 2.
- If you are serious about getting your students to learn from their practice questions, this is the logical solution. A mock simply isn’t.
A full mock in November or December is hardly any use to your students at all.
One final argument remains.
Surely students need to do a full mock before the real exam to see what the whole paper feels like to sit? This way they will learn about their stamina, the order in which to do the questions, what it feels like to keep going for so long in silence and not be distracted by those around you.
None of these are skills of English. So when should we teach these skills so that they are retained?
April. Even May.
Here students would find out what they are not yet good at. At this stage, it won’t be necessary to mark their mock – after all, if they can’t mark their own by comparing them to some model answers, they won’t be able to do it one month later anyway.
They can self and peer mark each question, comparing their answer to say a level 3, 5, 7 and 9 responses, and deciding where theirs fits best. For example, a grade 5 student will then look at the two better answers to see how they can improve.
Between April and the beginning of June they are unlikely to forget the exam specific skills this teaches them. I am happy for them to cram these, because they are transitory: they are just hoops to pass through for the exam. They have zero impact on their future study of English, they are just the rules of the game. Short term memory is a perfect place for them to lie.
If you want to see just how essential this, go to eAQA and the question analysis. Look at the marks for questions 5 on both papers. How many students chucked away their chances because they simply ran out of steam?
When you did your mocks, they could kid themselves that they would improve – there was still plenty of time, it wasn’t a real test anyway, yadda yadda yadda.
In April or May, this kind of self-deceit is impossible. “Crap, I failed this exam because I gave up after 90 minutes. Ok, the one thing I will make sure I will do is to keep writing for the full exam.”
Of course, if you are smart, you won’t rely on this. Someone senior in the English department is going to go into the exam hall next year and stay for the duration. A laser stare will let every student know that putting their pen down is not an option. A quiet word or hint of threat will let them know they can’t simply opt out. “You give up, and fail to try: you can’t go to the leavers ball, or you stay behind tonight till 5, or whatever.”
Check out your eAQA – my money says at least 5% of your students simply gave up, and at least 20% of your students stopped writing with 30 minutes of the exam still left. Want to make a difference? Make the buggers write.
(In my next blog we’ll look at the moral high ground. Copyright, AQA, my videos – do I have the moral right to make them? First I need to contact AQA to find out why an examiner’s report is kept hidden from the students who sat it and the 100,000 thousand who will need to retake it because they got grade 3 or below.)