/

The Complete University Guide 10th Anniversary Conference

National league tables: the evolving relationship with the university sector – 27 September 2017

University league tables have been with us for a quarter of a century, exposing the sector to a level of scrutiny that was previously the preserve of soccer clubs and consumer magazines.

In 1993, The Times published its first league table of the UK universities in a small paperback. At that time there were 96 institutions and their rank was determined by 14 measures, including staff qualifications and student accommodation. The outcome was met with derision by the body representing university vice-chancellors. 

Its press release condemned the tables as a dubious exercise, wrong in principle and flawed in execution. While accepting the principle we shared some, but not all, of the misgivings and wrote to The Times in a more constructive vein. This resulted in a partnership which lasted for more than ten years whereby we, Mayfield University Consultants, developed and produced the rankings and the narrative surrounding them.

At the beginning of the league table era, the World Wide Web was in its infancy. The focus was on publication in newspapers and in book form.

By 2007, we were convinced that the web was the way forward for an annual university guide and set up The Complete University Guide in partnership with the digital publisher, Constable and Robinson.

Fast forward a decade and the guide is recognised as the leading independent and trusted source for university applicants and their mentors. Other guides are available, but both remain print-led and one is behind a pay wall.

The Complete University Guide is not associated with any newspaper publisher, and is, most importantly, free to access for its key audience of prospective university students, available in a form with which their generation is firmly familiar, and with a wealth of associated information that will guide their choice of course and institution.

So, with more than 20 years' experience of compiling university league tables and 10 years since the founding of the guide, we have been encouraged to call a conference to take stock and look forward to the future.

We have brought together experts in the field, mainly from the university sector, to discuss and debate all the major rankings in what promises to be a seminal gathering. 

Bernard Kingston (Dr)
Chairman, The Complete University Guide

#CUG10YEARS

Programme

Click on the speaker's name to download a PDF version of the presentation slides used on the day.

Ten years on from the launch of TheCompleteUniversityGuide.co.uk, and more than 20 years since Andrew Hindmarsh and Bernard Kingston first began compiling UK university rankings for a national newspaper, we are taking stock of the current scene and looking to the future at this one-day conference in London.

We have brought together senior colleagues from the university sector and beyond, leaders in their respective fields and institutions, to discuss the part league tables play in the strategies and performance of their organisations, including what they think of them, why they use them, how they use them and where they use them.

Individual sessions will be devoted to the rankings in the context of current practice within governance and management, student recruitment, marketing and graduate employment and a plenary lecture will examine the national tables vis à vis the international rankings. The final session will call on the experience and expertise of the compilers to take a look at how university league tables might develop in the future.

10.00 Registration and refreshments

10.30 Welcome and introduction
Simon Emmett, CEO, Hotcourses Group
Dr Bernard Kingston, Chairman, The Complete University Guide

10.45 Are national league tables a more valuable guide for target users than their international counterparts?
Professor Simon Marginson, Director, Centre for Global Higher Education, University College London

11.15 Uses and abuses of league tables – do universities use them or ignore them?
Panel discussion with: Professor Cara Aitchison, Vice Chancellor, Cardiff Metropolitan University
Dr John Cater, Vice Chancellor, Edge Hill University
Professor John Latham, Vice Chancellor, Coventry University

12.15 How effective are league table compilers in getting their message over to students and universities?
Open session with: Amy Baker, Managing Director, The PIE News

12.45 Lunch

13.45 Informed choices? Using league tables and public information in selecting university courses
Mike Nicholson, Director of Admissions and Recruitment, University of Bath

14.15 What does ‘good’ look like? How careers services and employers view and use league tables
Dr Bob Gilworth, Director, The Careers Group, University of London
Samuel Gordon, Research Analyst, Institute of Student Employers

14.45 Break and refreshments

15.00 Do league tables take themselves too seriously? Do universities take league tables too seriously? Does it matter?
Dr Paul Greatrix, Registrar, University of Nottingham

15.30 What about the future – where are league tables going?
Dr Andrew Hindmarsh, CUG League Tables Compiler and Data Adviser
James Seymour, CUG Content Contributor and Editorial Adviser

16.00 Closing remarks
Dr Bernard Kingston

Speakers' presentation notes

Are national league tables a more valuable guide for target users than their international counterparts?

SimonM
Simon Marginson

Simon Marginson
ESRC/HEFCE Centre for Global Higher Education

University College London (UCL IOE)

Let me begin with warm congratulations to the Complete University Guide, especially Bernard Kingston, on reaching its 10th birthday as an Internet-based guide. Of course the whole journey has been longer than ten years. To start with nothing but the felt need for the information and to build that to a totally professional product with 20 million users, a third outside the UK, is phenomenal. Well done!

Today I’ve been asked to tackle the question ‘Are national league tables a more valuable guide for target users than their international counterparts?’ There’s a short answer—‘it depends!’ Still, they don’t pay me to make two word speeches. And being a professor, I could go on all day without finishing. So I’ll compromise. I’ll attempt to get to the bottom of league tables and climb back in 25 minutes. Wish me luck!

Next 25 minutes

It’s a good time to reflect on the vast and growing industry that university ranking has become. Most of the issues will be better discussed by later speakers, I am sure. Perhaps I can open up some of them.

First, what I see as the big issues in university comparisons and league tables.

Big issues

The starting point is information. Everyone needs complex information about this complex social sector, higher education, simply expressed. This is the power of the Complete University Guide. Perhaps it was disappointing that government, and official university bodies, could not meet the need for public information, obvious though it is. The initiative had to come from outside. But one of our strengths of our community is that initiatives like CUG can emerge from civil society, free of political, institutional or commercial agendas.

Once established, comparisons and rankings created a new form of accountability. People have a right to comparative data—though they also have a right to know the limits, for it isn’t always done well. Institutions often wish they were not accountable in this way, especially when they score badly, or the ranking is inaccurate, or it is not appropriate to their mission. But they are locked in. In many universities, meeting one or more comparative performance indicators—the REF, national student survey, TEF, or national or global rankings, or some combination—is the driving strategic objective.

This underlines the fact that comparisons and rankings have great power to map activities and choices, determine the life decisions of users and shape university behaviour. It is all the more important to make the comparisons better, to ensure the sector itself improves, in the interests of all.

But not all those who conduct rankings have the best interests of all in mind. Most rankings develop with more limited purposes. We have our work cut out to nudge those rankings towards the general interest. The ARWU in Shanghai in 2003 was a benchmarking exercise to demonstrate the gap in science between China and America that China had to make up. So it focused just on research and its indicators represented key features of US research universities. The Times Higher in 2004 wanted a ranking that would differ from ARWU, service the global student market in education and position British universities well. The survey and the internationalisation indicators advanced those goals. The culmination was this year when Oxford and Cambridge were placed at the top, though outside UK almost everyone thinks Harvard is number one. The purpose of QS is more directly commercial— on the back of its loss-leader ranking the company runs a global business in consulting, conferences, marketing and ranking-related services in higher education. It’s a clever business model. Ranking puts the QS brand on university websites without cost to the company and opens university doorways. Universities need QS as much as it needs their business.

At the same time, a ranking must be credible. Some years ago a research team was working on a new ranking. The statistician crunched the numbers. ‘It’s no good’, he said, ‘it won’t survive the laugh test. The Chinese are second last’. The ranking team knew that research in China was climbing fast, and they knew that many others knew. What to do? ‘I know’ said the statistician. ‘I’ll increase one of the indicators, total number of publications, from 5% weighting to 20%, and adjust the other weights. That’ll bump the Chinese up to halfway’. And that’s what happened. No one laughed at the results. This is a true story. Rankings have to pass the laugh test.

Ease of use is also vital. This—as much as fascination with competition and hierarchy—has led to the dominance of the league table format, despite its many problems, such as the fact that all universities are compared against one mission template though their missions are not the same. More complex comparisons tend to struggle, such as Multirank.

By the same token, rankings must be transparent. The days are gone when a ranker could send out the league table without saying how it was derived. The Complete University Guide has been a model in that regard. I must say that I’ve always wanted to know more about the QS and Times Higher surveys.

Finally, most important, rankings must not only inform the user, they must improve higher education overall. If the comparison elevates a few universities but over time depresses the work of the rest it is doing more harm than good. 

Ranking: The virtuous circle

Competition can lift performance, but not all competition does. If university effort is centered on gaming the system, as with Cardiff in the last REF, rises in the table is decoupled from necessary improvement in real performance. The wrong incentives are operating. Universities put their efforts into smoke and mirrors not substance. Worse, if institutions really do improve learning, research or services and their ranking declines, say because of changes in methodology, why should they work on their real performance in future?

Between 2011 and 2012, the University of East Anglia rose 25 places in the QS ranking but dropped 31 places in the Times Higher. One suspects that neither move was based on changes in real performance, and it’s very unlikely that both were! Even though the measures differ in some respects. If the virtuous circle between effort, performance and ranking is broken, users are better off without the league table.

Ranking: The unvirtuous circle

Users are also better off when the unvirtuous circle is not in operation. That’s what happens when a ranking is derived from opinion surveys. Rightly, the Complete University Guide avoids reputational surveys. Survey respondents do not know which university is the ‘best’, especially in teaching, and go with the established big names. Real performance, expressed in a league table, should drive reputation, not reputation driving reputation in circular fashion. Survey-based league tables look solid, but are meaningless in terms of either competition or performance. This is league tables at their worst.

Validity

That brings me to the controversial terrain of validity, where not everyone is going to agree with me. But I look at it in terms of social science.

First, comparisons never compare the whole ‘best university’. At best they are grounded in parts of what universities do and are specific to the purposes of the ranking. Regrettably, many league tables are oversold as measures of the holistic ‘best university’.

As this suggests, rankings embody norms. Each league table is fashioned on the basis of an ideal institution, one that achieves peak performance on each component of the ranking. ARWU is 100 per cent grounded in science publishing and prizes; Webometrics largely in web presence and traffic, and so on. The CUG covers a more heterogeneous mix. Rankers should be clear about their ideal model and provide a reasoned justification for it.

In addition, comparison and ranking should not skate over the often careless assumptions made about causality. Consider graduate employment-related data, used by the TEF among others. Using these data without reflection feeds the myth that higher education alone determines employability and salary—rather than being part of a cluster of influences, and often not the most important. Employability is also governed by family background, and social networks that continue to play out after graduation, not to mention the local and national economies.

The next point I want to make about validity is that data sources affect league table outcomes. In some league tables data are shaped through lobbying by the affected universities. Here the CUG has been a leader in validity, avoiding self-generated data as much as possible and carefully reporting those instances where it has had to break its own rule.

Objective and subjective data

As a social scientist I don’t feel comfortable when tables combine objective qualities like money spent, or published papers, with subjective qualities like survey returns. For example, student satisfaction varies on the basis of certain factors that are not necessarily connected to the real, material quality of provision. For example, third generation students tend to be more critical than first generation students. Both objective and subjective data have a place, they each tell us something important, but they belong in separate tables.

A larger problem is the use of proxy indicators. The problem here is that the education function is the most important function but we don’t have credible comparative measures of student learning achievement or teaching quality. Everybody knows that student-staff ratios, student satisfaction scores and other proxies have no necessary relationship to student learning but they are widely used. This has slowed the development of valid comparative measures, as in the OECD AHELO fiasco.

But where I deviate most from current league tables, except for Leiden, Scimago and Multirank, is in relation to multi-indicators and weightings. Most national and global league tables are multi-indicator rankings that collect data in several areas of institutional performance, and then combine those data into a single indicator and rank. The problem is not the rich, varied data sets—a great source of comparative information—it lies in the way they are combined using weightings. This creates two problems. First, the link between performance in each area, and the ranked outcome, is no longer transparent—the specific data are buried in the single overall score. The drivers of improvement are blunted. Second, the weightings are arbitrary. There is no basis for choosing one set of weights over others, and changing the weightings changes both absolute and comparative university ‘performance’, often dramatically. This problem cannot be overcome. No set of weights is unquestionably correct in an objective sense. All multi-indicator rankings are the creation of the ranker. And who said the ranker can play god? But both of these problems can be avoided, if the multilevel ranking is disaggregated into its separate components, creating several league tables rather than one. The ranking then becomes a valid source of comparative data. But that means giving up the claim to a holistic ‘best university ranking’. I know it’s hard for a ranking organisation to do this!

World Cup single metric ranking—the winner is determined by

Proxies and multi-indicator rankings together lead to some absurd contortions. Let me illustrate the point with an analogy. The winner of the World Cup in football is determined by who scores the most goals in the allotted time on the field in the final. That’s a simple metric.

World Cup multi-indicator ranking—the winner is determined by

Now what if FIFA changes the rules. Instead of rewarding the final performance alone—who scores the most goals in the allotted time—it decides to give 50% to the most goals, 20% to the size of the team’s fan base, 10% to total player product endorsement income, and 20% to the volume of team media coverage.

Unfortunately, in universities we don’t have one neat and agreed indicator such as ‘scores most goals’. But in its absence, should we use doubtful proxies and guess-work weightings, in which the realities of the sector are obscured? It isn’t credible when applied to football. Is it more credible in higher education?

So … combining objective and subjective data in one table. Proxies. Multi-indicator ranking. These areas worry me. Is a ranking system that draws partly on proxies and uses multi-indicators better than nothing? I’m not sure it is. It’s not strong enough, and it is too easy to manipulate, not by CUG but by some other rankers. And it sets the bar low, it slows progress, it blocks us from coming up with something better. It is Alchemy rather than Chemistry.

We need to move beyond cooking up scores like this, to establish university comparison as a genuine social science. We need a more valid set of data for the millions of users. They deserve the best information.

But let’s touch on the question I was asked to address: national versus global league tables. Regardless of the flaws I’ve discussed, target users face that choice.

National vs. Global

Here there are three issues at play. First, who is the ‘target user’? If it is international students and their families, or some researchers and doctoral students, global league tables are primary. For most first degree students and their families it is national rankings.

Second, there is the EU subsidiarity principle. That means you don’t make decisions at an aggregated level when you can make them locally. Those with local knowledge tend to be the best judges when local needs are in question. This means that all else being equal, a CUG-type league table is a better source of data about higher education in the UK than any global league table can be.

Third, there is the prestige factor. This points the other way. The prestige of an institution or a degree is important, often shaping people’s preferences. At the top end of higher education—though not other levels—the most important factor that determines prestige is research. This is conducted and measured on a world basis. This guarantees that global rankings focused on research performance will continue to play a key role. The REF is important in terms of UK prestige, resources and where academics work, but because it is a national measure it has not replaced global rankings.

Global league tables

Before concluding, I’ll comment briefly on the main global league tables.

Times Higher and QS have similar strengths and weaknesses. Both are well presented in website form and produce a wide array of information. They are transparent, up to a point. Both are multi-indicator rankings that use arbitrary weights and freely mix objective and subjective data in incoherent fashion. In both the league table format exaggerates the meaning of small changes in relative position which may not be statistically significant. My academic opinion is that the overall outcome of each is junk, but the individual indicators can be very valuable. If the Times Higher and QS released several league tables with each based on one single indicator, in the manner of, say, Leiden, my verdict would be different.

The Shanghai ARWU is a multi-indicator ranking that uses weightings but avoids most of the drawbacks because the different indicators correlate well. It is easy to use and transparent. Unlike the Times Higher and QS there is little scope for negotiations between rankers and universities that affect the outcome. The down side is ARWU’s 30% reliance on Nobel Prize data. The Prize is not an accurate measure of merit. It is affected by lobbying, and in any given year it is often the case that several people or teams could be selected. The other limitation of ARWU is its sole focus on research.

This is also ARWU’s strength. Data on research are useful. It is fit for purpose. The Leiden ranking and Scimago produce clean measures of research publication and citation, without the complications engendered by combining different indicators into a single number. Leiden is also very easy to use.

U-Multirank has tried to overcome the problems of multi-indicator rankings and league table hierarchies, without losing breadth of coverage. It offers a large range of indicators, though it is overly dependent on surveys. The user can select preferred indicators, weightings and universities for comparison. It presents the results in three broad bands rather than a league table. On the down side, Multirank is complex, lacks premiership ladder excitement, and unfortunately, many leading research universities have boycotted it.

Final thoughts

Final thoughts. Comparisons and league tables have an immense responsibility. Some rankers have agendas. The Complete Universities Guide focuses on users. This is why it has built a tremendous platform, user community and credibility. This genuine orientation to users will drive better rankings in future.

I believe the way to go is disaggregation of multi-indicator league tables. To break down the single composite index into its components will establish genuine validity and provide much more information for users. We should respect the fact that higher education has several purposes, there is no ‘one best’ model, different users have different purposes, and they need different league tables. Comparison should reflect this plurality. Give users a research league table, a resources league table, a selectivity league table, an employability league table, a widening participation league table, and so on.

At the global level, we need to think more about world-class systems, not just world-class or leading universities. In some national systems, all choices are good choices. The floor is sufficiently high. That is not true of all countries.

Lastly, the justification for a system of comparison, whether league table or multi-rank, whether focused on research or taking in many elements, is whether it generates improvement, in many universities, in the interests of users. The Complete University Guide can be proud. It has done this. It has been a leader. It has developed technically and lifted the bar. I trust that it will continue to evolve in managing and communicating data. Congratulations again on the 10th anniversary, and may there be more decades!