These three blogs by Alan Rusbridger, principal of Lady Margaret Hall, University of Oxford, were originally published on the college's .
Part 1: Shortlisting:
No sooner is term over than the admissions process begins – selecting candidates to come up to Oxford next year.?
The spotlight turned on the process last week when a report criticised Oxford and Cambridge for their failure to increase the number of state school pupils studying at Oxbridge colleges. I sat in on three stages of the process to see how candidates are shortlisted, interviewed and pooled between colleges.
I begin with the shortlisting...
I’m sitting in the book-lined study of a fellow at Lady Margaret Hall. In one hand she has a list of numbers, in the other a sheaf of densely typed papers. At her feet there’s a carefully annotated notebook. There are two other colleagues, similarly equipped, on adjoining chairs.?
They are wrestling with a dilemma. Here’s a candidate who would like to come to Lady Margaret Hall to study – let’s call him Ash –? but his marks are not what they should be. There are more polished candidates in the pile with better Ucas applications and higher scores for written work. But something has caught the eye of one of the fellows about Ash and the hand hovers. Does he end up on the pile of discarded forms on the floor, or should there be a longer discussion??
As we near the end of full term – another week to go – there’s no sense of winding down among the fellows. Quite the opposite. They’re all preparing for a week that appears to be the most intense period of the entire year: admissions.?
The complexity arises from the college system at Oxford. In other universities there would be one central admissions funnel through which all applicants pass. At Oxford there are the faculties as well as 30 colleges (not to mention six permanent private halls) that admit undergraduates.??
Applicants may indicate a first choice of preference for a particular college.?But there’s inevitably a mismatch between the numbers of applicants and the places available. If they don’t choose a college they’re allocated one by computer and treated as if they had.?
Brasenose (I’m making this up) might have 50 would-be students wanting to read Serbo Croat, but only four places this year. So the Brasenose rejects – they may include outstanding candidates who will go on to achieve first-class degrees – drop into a pool for reallocation to other colleges.?
“It’s like horse trading,” says one of the tutors of this post-interview period. “We tried doing it digitally one year but it didn’t work. So we end up on three nights round a table.” Another tutor speaks of this period as a time of “ruthless hunting down” to find the best candidates who are floating around the system, yet to be allocated.?
That’s to come. People in college speak of it darkly as a period of immense pressure as candidates are summoned from college to college for interviews. There will be tears and long nights ahead, I’m warned. And that’s just the academics and staff running the process, never mind the candidates.?
But today, in the fellow’s room, there is a studied calm.?
By each candidate’s name are five numbers. The first is the score – from 1 to 10 – for the overall Ucas application. Each of the three tutors has marked this separately before reaching an average. On the sheet in front of me no one has scored less than five. Five have notched up 10 out of 10.?
Then there’s a mark for written work, again out of 10. Each candidate is assessed by two of the three fellows and an average score arrived at.??
The next column has marks out of 100 for the aptitude test that all candidates sit. Here the marks range from 27 (poor) to 57 (good). Those scores feed through into the next column, which places people in bands of 1 to 4. In the penultimate column there’s an overall mark out of 10 that comes from a computer munching its way through the figures in the previous four columns.?
There are three options for the tutors today: S, D or R. S is to summon a candidate for interviews in mid-December. D is to de-summon and R is to reserve a candidate: ie, keep them at this stage for LMH only, and not (yet) for the horse trading. “We get first dibs on Rs,” says a tutor.?
I wonder about the algorithm used to arrive at the final mark. The tutors believe that the final score is heavily (unduly?) influenced by the score in the aptitude test. Generally speaking, a candidate below six is going out at this stage unless (see Ash) there may be special considerations.?
At the heart of the discussion today is a search for fairness. All the LMH colleagues to whom I’ve spoken want to attract the best candidates, regardless of background. But what is “fair” when there is, in the background, such a disparity of income, schooling, coaching and grooming??
The tutors have some help in levelling out the playing field – a system of flags that show if a candidate has come from an area of social deprivation or from a school that does not routinely supply a production line of Oxbridge candidates.?The tutors take note of anyone who has attended a UNIQ summer school, an access programme that gives priority to students from low socioeconomic backgrounds.?
By the time we get to Ash we’ve sifted through a few that are more easily decided. X? is open and shut: A*s all the way in life so far and high scores in the written work and aptitude test.?And here’s one who – after a brief discussion – is rejected. Modest marks, nothing jumping off the page.?
Here’s one from a good public school. Look closer and see they started at a state school.?“You have to read the forms very closely to see what the story is,”? says a tutor.?“Sometimes they can move from one sector to another – maybe on a scholarship. Perhaps it’s improved their work, but affected their confidence.” A query for this candidate – decide later.?
Here’s one from a comprehensive with pretty good marks and an arresting writing style. She’s through to the interview stage.?
And then we get to Ash. The computer’s ranked him well below the average. His written work’s fine, but not spectacular. He has an A star at GCSE in the subject he wants to study, is widely read and comes with a glowing reference from his teacher. Plus he has two “access flags”. But, on the aggregate score alone, he’s for a D.?
What should they do? Are they raising his hopes by asking him up when, on paper, he’s not as strong as some others? But what of the signal sent when such aspiring candidates don’t even get an interview at Oxford? On the other hand, if you ask too many candidates up for interview, all agree, you can get punch-drunk. Sometimes it’s better to see fewer, but delve into them deeper.?
One tutor remembers another such case – someone who scored similarly poorly on the aggregate score but shone at interview and went on to achieve a very high 2:1.?
Time is ticking away. There’s a pile more candidates to consider. Ash is through to the next stage.?
And so it goes on. This one’s from a school that they know can produce outstanding candidates – but this isn’t one. Out. This one’s borderline but is from an Offa (Office of Fair Access) school. In. This one’s scored brilliantly and comes from an Offa?school. In. This next one’s out: “The teacher’s marking the work as superb. It isn’t,” says one tutor crisply.?
They’re done. Over the next week or so thousands of Oxford applicants will get emails inviting them for interview in December…or not.
Is the “system” fair? That’s doubtless the subject of many more blogs. But ?at LMH – on the basis of what I’ve seen so far – the existing process is done with care, thought and thoroughness.
Part 2: The interview...
...in which the Old Testament story of the Judgement of Solomon is used to test would-be students’ ability to think about economics and management.
I am sitting in on six interviews for students wanting to read economics and management. They take place in the study of Brian Bell, associate professor in economics – brown carpets and sofas matching the dark oak 1930s doors and shelves of LMH’s Deneke building. Brian is in jeans and a hoodie. His co-examiner is Pawel Dziewulski, a career development fellow in economics, in grey flannels and an orange jumper.
The two of them are seeing 12 candidates for four places. The process goes like this: they’ll choose a maximum of three at this stage, marking further candidates marginal rejects? or (in one case) a marginal accept. There’s then a period in which they see a few other candidates who haven’t quite made the initial cut at other colleges before settling on a final list. The interviews about to take place – 20 minutes each – will count for about a third of the overall consideration this week, the other two main factors being the scores in the essays and the TSA (Thinking Skills Assessment) test that the candidates have already? taken. And, of course, there are past and predicted exam results, personal statements and school references as crucial parts of the mix.
The first candidate appears, rather more formally dressed than either of his interlocutors. In his hand is a two-page passage he’s had some time to study, which quotes the First Book of Kings, Chapter 3, 16-28 – how King Solomon settled a claim between two women, both of whom claimed to be the mother of the same child. The paper suggests that Solomon’s approach was flawed – and invites candidates to identify how. It then suggests alternative ways in which Solomon could have decided which of the two competing women was the true mother of the disputed baby. One Bell/Dziewulski scenario involves the payment to the king by Mother B if she claims the child is hers.
Pawel leads the discussion, probing away with rapid fire questions about different scenarios.
Do the mothers have an incentive to tell the truth? What if A is telling the truth? How does it change if B is telling the truth? What if Solomon simply sold the child to the correct mother by asking a price high enough to deter the false mother? What price would he have to offer?
The candidate – whose first language is not English – seems to be enjoying the cut and thrust of it all, smiling at each twist in the game-playing. But he can’t afford to relax because Pawel now wants to know why anyone would buy an alarm clock with a snooze button. If you want to wake up, wake up, he says. How does this make sense?
The candidate wrinkles his nose, smiles to himself, changes gear and does his best to argue his case.
We’re halfway through and it’s Brian’s turn. From an economic perspective, he wants to know, what are the problems with a wealth tax? A series of four or five questions follow. How might the wealthy react to such a tax? In equilibrium, what are the implications for global tax rates if people move to avoid higher taxes? If the answer’s zero, how does France get away with taxing the rich? Does taxing property get round the mobility of capital? What policy? initiatives would help to stop the flight of capital in response to changes in the taxation rates of individual countries?
With some candidates? – the ones who dealt most efficiently with the taxation questions – Brian moves on to ask how George Osborne pulled off his recent Budget trick of seeking to reduce the deficit while taking advantage of the ?27bn the Office for Budget Responsibility found down the back of the sofa. With other candidates he goes further, demanding about the possible economic advantages to Germany of allowing a massive influx of immigrants.
There’s a break after the first three candidates. “Now what we do”, says Brian, “is magically produce a mark out of our heads.”
The two of them score the people they’ve seen so far out of 100: in each case there is no more than three marks difference between the two examiners.
Anyone scoring more than 70 is likely to be well-placed. Anyone getting less than 60 would have to have performed very strongly in the rest of their application to compensate. Most are clustered somewhere in the sixties.
What are they trying to prove with these questions?
“Well, there’s no ‘right’ answer,” says Pawel. “We’re trying to assess their ability to think as the interview progresses.”
“We try to find questions that they won’t have done at school,” says Brian. “So we tend not to ask the things they would know if they had been taught a certain kind of economics. A few years ago everyone asked about the prisoner’s dilemma. But then schools picked up on that and started teaching it. So we dropped it.”
“The aim is to find the keen minds, make sure we can find people who haven’t been coached for Oxford entrance. The TSA test is supposed to be unteachable, but I’m not so sure now.”
And so on to the next three candidates. Some are nervous, some ice cool. Most seem pretty adept at thinking under pressure – even with questions raining in on them (I counted 20 in 10 minutes from Pawel). There are moments when brains seem to freeze and the only human reaction is pity. Some visibly wilt after 15 minutes of intellectual cut and thrust. Others seem to draw energy from it.
The day’s over and the two examiners sit down to compare their scores.
“I know we’re going to disagree about one,” says Brian. They do. One has been marked well for the King Solomon test and badly for his grasp of the Osborne Budget.
“That really annoys me,” says Brian. “I mean, they want to do economics and they haven’t followed the Budget properly?”
There are six more interviews the following morning. By lunchtime Brian and Pawel have five names in order on his whiteboard. Three of the interviewees I witnessed are there. There’s one person from a flagged background ?– indicating socioeconomic circumstances that might be taken into consideration. The list is topped by someone whose work was so outstanding that (they say) she would have had to have remained silent for 20 minutes not to have got a place.
Later the two of them compare their rankings with the fellow responsible for the management part of the course. He had, separately, come to the same conclusion about the candidates. It has, they all agree, been an easy process this year with a remarkable degree of unanimity about whom to admit.
Part 3: Pooling
These three posts were written with thanks to colleagues who allowed me to sit in on their deliberations.
They jokingly call it horse-trading – the meeting when admissions tutors from different colleges get together to agree a pool of candidates who deserve a place at Oxford, but aren’t among the first choices of the individual colleges.
We’re meeting in a modern, light faculty room with representatives from the undergraduate colleges that teach this subject. “Horse-trading”? is slightly misleading: colleges no longer “swap” applicants in the way they once did. Today’s meeting is about making collective decisions about entry to a course. The task of herding all the representatives in the room falls to a senior colleague from one of the colleges. We’ll call them Hilary.
On the screen in front of the academics is a university database of candidates? – a giant spreadsheet with about 30 columns of information. There’s the candidate’s name, the school they attended and the two colleges that have interviewed them so far.
Most of the academics in the room have the same information open on laptops in front of them. They also can see applicants’ GCSE and AS results , along with their predicted A-level grades. They can read some contextualised information – including the GCSE school performance compared with the national average and how the individual candidate performed against the average at the school. Personal statements and references from schools are there at the click of a mouse.
By changing view on the screen an admissions tutor can also see if a candidate has been flagged for some form of socioeconomic disadvantage or has come from a school that may not regularly send students to apply to Oxbridge.
The information is certainly comprehensive. Whether or not it is easily digested and readable at speed in the cut and thrust of the next 80 minutes is a different matter.
This is the final stage of an exhaustive process in which a large number of would-be candidates has first been gradually whittled down to a manageable number for interview. Candidates are then each interviewed at two different colleges. By this stage the colleges have already decided on the vast majority of applicants. It is, says Hilary, a “drawn-out, detailed, and, we believe, fair and caring process.”
As the academics settle down Hilary explains that they have a number of candidates in front of them today – all of whom have scored A1 after individual colleges have seen their work and interviewed them. Today’s task is to reduce that number by about a third.
Hilary reminds the group that the pool is needed because – for whatever reason – a number of candidates fail each year to achieve the grade necessary to secure their place at Oxford. So the extra candidates selected today will create a large enough pool to allow for those who fall at the last fence.
“You have to be really, really sharp and critical today,” Hilary urges the colleagues. “If you really feel someone should have a place, then fine. If there’s doubt in your mind I’m afraid we have to say no to that person.”
The first candidate flashes up on the screen.? An admission tutor from one of the two colleges to have seen him speaks up. “I think he was pretty good, but he didn’t get on our long list.”
Hilary looks doubtful. “I’m not convinced by this chap. I’m going to put him as a question mark.” His name is chalked on a blackboard at the front.
The next candidate. One college’s verdict: “quite strong”. Someone looks at some of her grades and queries why they were weak. She’s a question mark at this stage.
Another applicant. “We thought she was quite good, a good breadth of knowledge.“ It feels lukewarm.“You wouldn’t say she was brilliant?” asks Hilary. “I don’t think she’s going to make the cut. I’m sorry.”
The next candidate has good contextualised GCSE scores but, says a tutor, “we thought she’d be an interesting candidate but we got very little out of her...a very good case why we shouldn’t do interviews. It’s about the substantially reduced chances of people from non-public schools.”
“We haven’t heard that for a few weeks,” says Hilary drily.
Another candidate illustrates the problem. He was better than the previous candidate in interview – “but he was really well prepped”, says a tutor. It’s illustrative of why interviews are problematic, says one tutor.
Now there’s an applicant who did well and was from a widening participation school: she’s through with little discussion.
Here’s one the two colleges who have seen her can’t agree on. One thinks she was “very disappointing, but clearly knows her subject”. The other college’s view: “She was very good.”
You begin to sense some frustration on Hilary’s behalf in trying to interpret the conflicting views. There are worries that they’re not moving fast enough to get through the business.
The next candidate doesn’t help? – at the bottom of one college’s list of people who didn’t quite make the cut but “absolutely at the top” of the other college’s list. No immediate decision.
We arrive at a candidate who is “flagged all over the place”.
“I think I terrified the life out of him in interview,” jokes a tutor. “He started off answering quite well and then petered off.” The other college partially disagrees: “He did the opposite with us. He clearly wasn’t prepared for an Oxford interview, but, given his track record it was greatly to his credit that he’s done well. It was difficult to get answers out of him, though.”
The applicant’s details flash up on screen and the academics peer at his predicted exam results.? His personal statement scrolls down and the group looks at the school he’s at.? “It’s not the best preparation…he seems to have done quite well.”
But suddenly one tutor registers his unhappiness at the drift of the discussion. “That’s my whole frustration about this process. You can’t take him on the basis of that,” he says animatedly. “Should we be putting people into the pool on the basis of all these flags? This person will be imposed on a college who hasn’t seen them.” Others disagree with this view.
It looks as though the candidate is heading for a “No.” Another tutor from a different college expresses disquiet at the discussion around access and flags in the context of this pooling meeting.
The unease doesn’t seem to be about encouraging “access students”: it’s obvious that many tutors in the room have a keen appetite for that. It’s over how much, at this stage of the process and on the information in front of them (and without having the benefit, most of them, of having met a particular applicant), a challenging background should be taken into consideration.
“This process is quite embarrassing,” says this tutor. “We’re not talking about access places. This process is for people who we’re pretty confident will make their grades.”
A tutor from another college appears unhappy at the likely rejection of this applicant, and declares that she, too, was an access candidate once. “I’m hearing this candidate did quite well in one interview.”
One of the tutors who interviewed the candidate contests this. “Someone is going to have to take this student for three years. The test is, would we be happy to have them?”
Hilary intervenes to cut the discussion short.
But a similar problem soon crops up with another candidate: poor exam grades, and/but from a failing school. You could argue she’s done very well to overcome the obvious hurdle of a challenged school. How should she be treated?? The candidate hangs in at this stage.
By this stage some tutors are becoming advocates on behalf of certain candidates. “Even now I’m not sure why we didn’t take this one,” says one.? Another speaks up for a student who was “fully flagged”.
A sceptical voice in the room says: “So can someone else have him.”
There are some easier cases who go on the yes list and, at the end of the first cut, the group has managed to sift out quite a few applicants. It’s taken nearly an hour so far. They still need to lose more to hit the target. It’s time to revisit some of the marginal calls.
But, again, there is some disagreement about what allowances the group should be making at this stage for the backgrounds of candidates.“This one comes from a state school and he’s weaker than X,” says a tutor. “On paper he’s a weak candidate who’s going to be imposed on someone. I’m declaring some alarm at the whole system. There are some colleges who are going to be getting some candidates who, on paper, are not good enough. I don’t think this process is working.”
Another colleague voices similar unease. “At this stage it’s not about Offa or social engineering. That’s a fine decision for an individual college to make, but not at this stage.”
One tutor expresses disagreement with the process: “I’m very happy to take people from the pool, but I’m not happy to take people who haven’t got the right GCSEs.”
He threatens to withdraw his college from the pool.? “It’s not OK at this stage to take people who aren’t cast iron on paper. I’m going to unilaterally withdraw from the pool. We can’t take this risk.”? [He actually stays in the room.]
Hilary looks even more frustrated at the turn of the conversation. “We can’t go on for ever,” Hilary says. “We can discuss the methodology later.”
They move onto another candidate – one, notes a tutor, with flags for social-economic disadvantage who performed well in interview.
“Why didn’t you take them then?” shoots back another tutor.
One college, keen to protect one of the candidates who’s looking doubtful, offers to take him out of the pool and find a college place for him.
The group is gradually getting to the last choices – peering at the contextualised data. Someone argues the case for a candidate from a flagged “low-participation” school. “She did really well at interview, she answered questions quickly. She’d be good to teach.”
In another part of the room two tutors who have both interviewed the same candidate are conferring among themselves and volunteer to take someone off the list. At the last knockings another candidate falls as a tutor concedes: “I can’t support her.”
The department has its list – agreed by all. About 20 per cent of the total list – chosen today and over the past week or so – are flagged up as “access” candidates. Hilary looks relieved to have arrived at the end. “It’s the hardest I can remember.”
Alan Rusbridger is principal of Lady Margaret Hall, University of Oxford.
