I am intrigued by the difference in the administrative burden that I deal with in my privately funded research organisation the IZA Institute for the Study of Labor, in Bonn, compared with what I was used to in a university. OK, it is a small institute, with 40 in-house researchers and 20 administrators (and 1,000 research fellows). But nevertheless, the systems and processes are concise and unbureaucratic.

Its director, Klaus Zimmermann, who is a labour economist, offered me three reasons why the institute is efficiently run: first, he tries to employ the best he can find from the private or public sectors; second, he never allows the number of administrators to exceed or come close to the number of researchers; and finally, "the most important thing", he says, "is that both sides understand each other and share the same spirit".

You think this is obvious, right? Yet complaints in the UK and the US (see, for example, Benjamin Ginsberg's recent book, The Fall of the Faculty, the Rise of the All-Administrative University and Why it Matters) point to the increasing struggle between managers on the one hand and faculty on the other. At its simplest, the disagreements are about processes. Management, which in the US and UK is very influenced by accounting practices, would like to run organisations in a way that is seen as counter-productive and counter-cultural by faculty.

It is surprising that we are still debating this topic. For years, researchers have known that work environments are important to employees' creativity and to their performance. Psychologists have been examining creatives and experts since the 1900s, and since the 1960s attention has turned to academic researchers. In 1967, Frank Andrews, a distinguished psychologist at the University of Michigan, wrote an article entitled Creative Ability, the Laboratory Environment and Scientific Performance. And in the Web of Science database, in the fields of social science and the humanities alone, there are more than 13,000 articles with a focus on creativity. Of these, the fifth most-cited paper, published in 1996, is Assessing the Work Environment for Creativity, by the Harvard psychologist Teresa Amabile and colleagues. There are now more than 300 articles on this environment theme by psychologists including Michael Mumford, Christina Shalley and Dean Simonton, among many others.

³ÉÈËVRÊÓƵ

So don't tell me we don't know how academics work best.

I cannot understand why this excellent research has been ignored by governments and often our own institutions. I will summarise the main findings of these and other authors to try to clear things up (with particular help from a Mumford and colleagues review article in Leadership Quarterly, 2002).

³ÉÈËVRÊÓƵ

First, let's get one thing straight: everyone is creative in some way. Creativity is not confined to a small group of scientists, artists or writers. Nevertheless, the creative people who sit in the labs and research institutes in universities (who are the focus of this piece) receive the title of "creative" because, as the literature suggests, they have made a substantial investment in expertise and the ongoing development of expertise over many years. Academics are more often driven by intrinsically motivated curiosity rather than purely extrinsic factors, such as money.

Intrinsic motivation is defined in the literature as a drive to do something for the sheer enjoyment, interest and personal challenge of the task itself (rather than solely for some external goal). Extrinsic factors such as money tend to be viewed as less important.

Numerous academics became interested in one subject area early in their lives, and they continue to be interested in it at the end of their lives, often to the exclusion of other things. Interestingly, the evidence suggests that the most successful scientists were those who started thinking about their topic early.

Research shows that creative people have their identity heavily bound up in their work and, therefore, success and recognition in work is a powerful motivator. Also, creative people are evaluated by their profession (or discipline) rather than their employers, which might explain why obtaining institutional loyalty from faculty can be challenging. In one study referenced in Mumford's review article, two authors (A.M. Harrell and M.J. Stahl) assessed the motivations of scientists and managers. They used David McClelland's three measures of motivation: the achievement motive, where incentive comes from improving one's performance in a given task; the power motive, which is impact, control or influence over another person; and finally, the affiliative motive, which includes social factors such as being around other people, commitment and intimacy. Harrell and Stahl found that scientists obtained higher scores on measures of achievement motivation, while managers obtained higher scores on measures of power motives and affiliative needs.

This makes sense when you think about how disorganised academics can be as a group (herding cats etc). Before I became one of them, I used to scratch my head and wonder why academics allowed governments to tread roughshod over parts of their world. I used to think: "Why don't they get organised and do something!" They - we - acquiesced when funding bodies mutated into managerial mazes.

³ÉÈËVRÊÓƵ

But all is now clear. Mumford reports that because creative people strongly desire autonomy (they perform better under conditions of autonomy) and they tend to be less politically and socially motivated, they often appear to be the "odd man out". This is not to say that academics per se are not political or motivated by power; of course, there are many who are. Nevertheless, ask any dean or pro vice-chancellor why they cannot hire a department head and they will say, "because the faculty just want to be left on their own". Although scientists were found to share many similarities with artists (openness, flexibility, cognitive complexity, self-confidence, dominance and introversion), they differed in their view of power: scientists were found to be more accepting of authority whereas artists were more rebellious.

One of the most reported conditions found to enhance the performance of creatives and experts is autonomy or having a sense of control on the job, either in terms of how work is done or how time is allocated. Creativity is viewed as being on a continuum; correspondingly, the most creative individuals require the most autonomy.

As Andrews reported in 1967, creativity was found to be higher in scientists and research and development staff when they had freedom at work, received encouragement and had adequate resources.

³ÉÈËVRÊÓƵ

In contrast, conditions that have been found to inhibit creativity include: working in an overly controlled environment governed by rigid procedures, low levels of individual autonomy, the use of surveillance measures, reduced resources and supervision that was viewed as critical and excessively monitoring. In recent years there have been claims of over-bureaucratisation and managerialism in universities, witnessed through intensified auditing, excessive controls and the overuse of accounting practices when measuring performance. I know that some of our best researchers are refusing to apply for funding from the research councils because of the excessive box-ticking they have to go through. The same processes are required when reviewing others' applications. These practices are known to impede creativity and innovation - so why are they there?

My own research looks at the leaders of creatives and experts. Many people believe that specialists should be led by generalists, who can take care of things and let the experts get on with their job. But this is not borne out by the evidence. Mumford and colleagues report on a number of studies showing that a leader's technical expertise is the best predictor of creative performance among followers.

My research tells the same story - it shows that universities performed better when they were led by top scholars. Recently I have been looking at hospitals, asking the question: should they be led by managers or qualified doctors? Hospitals used to be led only by doctors; however, in the US today only 4 per cent of hospital chief executives are medically trained, most are professional administrators, and the same is approximately true in the UK. Yet my study shows, in a simple cross-section, that US hospitals ranked higher (in three specialisms) are more likely to be led by doctors not professional managers. I'm now researching the question in longitudinal data.

We know that the economy needs more creativity and innovation, and we also know that creative people do not work well when accounting-based management practices are used - so why are we constantly fighting them off? As Robert Locke and J.C. Spender point out in their interesting book Confronting Managerialism (2011), the start-up enterprises that exploded out of Stanford's computer science department, creating the Silicon Valley of today, would never have succeeded in a box-ticking environment. These innovative firms that were started by scientists are like our universities used to be.

³ÉÈËVRÊÓƵ