Implementation, not intent, is flaw with REF

Three scholars have argued that the research excellence framework¡¯s reliance on hasty peer review by generalists limited sample size and accuracy (¡°The REF is perverse to ignore journal hierarchies¡±, Opinion, 4 October).

When I was research dean at our institution, I did a simple exercise: to weight each publication by the five-year citation impact factor of the journal it appeared in and aggregate this to a total score for the institution. It basically replicates the REF rankings without much issue. I did the same exercise in Australia with similar results.

In reality, no single publication matters much when an institution¡¯s sample is 300 or 400 articles in a subject area, and the reading exercise does little more than replace random, flawed human evaluation with random, flawed citation data.

Academics dislike the metrics (they want their papers evaluated by peers, believing that their work is unique) but this is an institutional flaw introduced by university managers pushing the aggregate evaluation system down to individual level assessments.

³ÉÈËVRÊÓƵ

While every university denies doing this, every university does it, as part of its REF and staff evaluation processes. Hence, the system becomes not a measure of overall research quality but rather a process of enforcement of individual key performance indicators by managers. The new REF requirement that everyone takes part mitigates this a bit (other than then creating two classes of academics as the process shifts from deciding which articles are included to one that decides which individuals should be on a research-related contract), as does the minimum requirement of one article per person. However, it is not the intent of the process (to monitor as a means of improving research output) that creates issues, but its implementation.

Similarly, journal lists are skewed towards local journals. In business we have the Chartered Association of Business Schools list, which includes a number of dubious 4* journals, and gives faculty the illusion that they are publishing in a leading world-class journal when it is a leading journal only in the UK. The REF process could easily be simplified, but the reality is that it is not meant to determine the truly best institutions and departments (this is blindingly obvious to anyone working in key areas or by looking at the most highly cited scholars and who is publishing in the outlets that matter to scholars globally) but who is in what tier, particularly for institutions, faculties and schools wanting to advertise that they are in the single digits of the rankings or are highly rated in some niche area of scholarship.

³ÉÈËVRÊÓƵ

Timothy M. Devinney
Via timeshighereducation.com

Tutorials a true test

One answer to the rise of essay mills discussed by UCL¡¯s Anthony Smith (¡°As a matter of fairness and social justice, we must ban essay mills¡±, Opinion, 11 October) would be for students to know that they were at risk of being thoroughly grilled in a tutorial (we do still do tutorials, don¡¯t we?).

If a student fully understands the submitted material, they must have written it themselves or must have nevertheless acquired the requisite knowledge embodied in an essay or a thesis (other than one involving empirical studies).

If they cannot demonstrate that understanding, it must be the tough love of ¡°Room 101¡± ¨C expulsion or a suitable sanction.

Paul G. Ellis
Business school tutor
London

Exclusive English

The report on ¡°Unease over growth of English-language courses in Europe¡± (News, 11 October) gave a welcome overview of the challenges that institutions are facing.

³ÉÈËVRÊÓƵ

There is anxiety about the impact of English on the health of national languages for academic purposes. Linguistic debates are bound up with other resonant issues of our time, such as national identity and workforce migration. While the report encapsulated the views of vice-chancellors, it barely touched on the experience ¡°from below¡±. University leaders need to be aware that English-taught courses impact profoundly on staff and students; such policies can have negative consequences for them.

In a review in the journal Language Teaching , researchers at the University of Oxford, led by Ernesto Macaro, reached some sobering conclusions. Lecturers worldwide are ¡°deeply concerned¡± about their students¡¯ failure to thrive in English-taught courses, with lecturers in South Korea considering nearly a third of students ill-equipped to benefit. English marginalises those who lack sufficient language skills.

More lecturers also report language problems than do not, and as the THE article noted, a significant number of Swedish students are ¡°less than enthusiastic¡± about their teachers¡¯ English skills. The Oxford research found that ¡°nearly all studies¡­allude to¡­the extra work involved¡­and the laborious nature of [English-medium instruction]¡±.

My recent research at a private English-medium institution in Central Asia reinforces these findings. English-taught degrees should not be seen as a cash cow; they also have their price.

³ÉÈËVRÊÓƵ

Andrew Linn
Professor of language, history and society
Pro vice-chancellor (research)
University of Westminster

Letters should be sent to:

THE.Letters@timeshighereducation.com

Letters for publication in?³ÉÈËVRÊÓƵ?should arrive by 9am Monday. For terms and conditions, see www.timeshighereducation.com

³ÉÈËVRÊÓƵ