Tuesday, 8 April 2014

Thoughts from an ESRC Mock Panel

Last month we held an ESRC mock panel as part of the Grants Factory. This is a really useful exercise; it gives participants a flavour of the discussions and debates that take place in a real Grant Assessment Panel (GAP), but it also gives them an idea of the tough assessment their application will have to go through.

Whilst I've posted elsewhere on this blog about what makes a good application (eg here, here, and here), a couple of points were raised at the workshop that I thought that they were worth repeating:

  • Firstly, panellists rarely read your proposal in the strict order in which its presented. The two panellists who led the workshop said that they normally read the JeS form first, to get a sense of what the project is about, then skip to the reviewers' comments and PI's response, before returning to the Case for Support. Your response, then, is crucial. This is true of all the Councils. A panellist for one of the other Research Councils said the following after returning from a panel meeting:
'The PI's responses were key and a substantial number of these were badly done (serving simply to refute or to point out disagreement between reviewers rather than rebut with argument, to clarify or to accept reviewers' suggestions).  Not all PIs made use of the whole space allowed.  Spending time reminding the panel of the positive things that reviewers had said was a waste of space if there were substantial issues to be addressed...I'm certain that I saw applications that would have received a higher final grade (and possibly funding) if the PI response had been better done.'
  • Secondly, most panellists won't have a background in your area. The GAPs are quite broad (see their disciplinary configuration here), so you need to make sure of two things: first, that you explain your research in a way that an intelligent general reader can understand; and secondly, that your methodology is watertight. Why? Because although the panellists might not understand the specifics of your project, they will all understand (or think they understand) the underlying methodology. So that is where they're going to pick holes. In particular, you need to be strong on how you analyse the data. Try and preempt any problems they might see in your methodology, and head them off at the pass. 
The final mock panel of the year will focus on the EPSRC, and will take place on 4 June. Drop me a line if you want to take part.

ESRC Changes Grant Assessment Panels

The ESRC has decided to change the configuration of its Grant Assessment Panels. These changes will take effect from November 2014.

Previously, the three panels covered the ESRC's remit as follows:

From November they will be:

My understanding is that these changes came about partly to balance out disproportionate workloads, but partly because (to quote my source) 'panels were getting set in their ways'.

This makes sense. I wouldn't want the panels to ossify (to borrow the ESRC's own term). Nevertheless, this does seem like an odd mix, and there are some strange divisions, such as 'economics' and 'economic and social history' being in separate panels. Similarly, I would have thought that 'Social Policy' had more in common with 'Sociology' and 'Socio-Legal Studies' than, say, international relations.

But the ESRC's task was a thankless one. They were never going to please everyone. I wish them well with this, and I do hope - for all our sakes - that the new arrangement works. For those who might be nervous about how their application might be viewed under the new system, there's still time to get your proposal in to be viewed in July under the final meeting of the old panels.

Monday, 7 April 2014

'The Sower of Discord'

Somewhere in Turkmenistan
Last week David Willetts suggested that the Government was considering 'franchising' the REF to countries overseas. Now read on.

A terrorist arms bazaar, somewhere in Afghan/Turkmenistan border, a la Tomorrow Never Dies. Amidst all the hardware is a small stall piled high with A4 brochures. A bald man in rimless glasses stands behind the stall. He's wearing a Life of Brian style beard.

Willetts (for it is he): Pssst!
Terrorist (points to himself): Who, me?
Willetts: Yes! Are you interested in causing (looks to left and right) chaos and confusion?
Terrorist: Well...
Willetts: Do you want to blow apart civilisation? Destroy social conventions? Do you want to strike  at the heart of the intelligensia?
Terrorist (suspiciously): Maybe...
Willetts: Ah! Come! I have a wondrous weapon! More powerful than any nuclear warhead! More dangerous than a box of hand grenades without their pins!
Terrorist (looking at the stall, and picking up a brochure entitled 'Panel Criteria and Working Methods'): But this is just a...
Willetts: (waving his arms) Don't be fooled my friend! This has the potential to return society to a prehistoric struggle for survival! (Grabs the brochure and stabs his fingers at its cover) This has the power to make 'civilised' people claw each other's eyes out, and trample each other into the dirt! Not only that, but you can force the smartest people in your country to spend hours - weeks! - years! in futile bureaucracy!
Terrorist (shaking his head): That will never happen.
Willetts: It will! I have tried it and I know it works! You don't need guns, you don't need tanks. All you need is 'The Sower of Discord'!
Terrorist: 'Sower of Discord'? Is that its name?
Willetts: Yes! It is the Armageddon Machine! But we must move stealthily. In my country it has been codenamed (whispers): 'The Research Excellence Framework'.
Terrorist: So how does it work, this Sower of Discord?
Willetts: Ah! That is the clever part! This is the - mwhahahah! - genius of my plan! These intelligent infidels are the creators of their own destruction! See, I give them a pile of money and say that they have divide it up amongst themselves. But they mustn't do it equitably! Oh no! They must work out who is the 'cleverest' amongst them.
Terrorist: That doesn't sound too hard.
Willetts: Oh but it is! How do you measure 'clever', eh? Eh?
Terrorist: Hmm.
Willetts: Yes! They tear themselves apart with H Indexes, and impact factors, and all sorts of crazy bibliometrics! But my friend, I've not got to the best bit. Not only do they have to work out who is the cleverest, but also who is the most - heh! heh! -  'impactful'.
Terrorist: What does that mean?
Willetts (eyes gleaming): Ah - ha! That is the genius! No one knows! And we do not tell them! We just say, 'work it out for yourselves!' Hahahahahahahahahahahahaha! It creates institutional and personal madness! Madness!
Terrorist (with disgust): But that is...monstrous. What kind of sick individual are you? I may be a terrorist, but I do have some standards. You keep your, your...so-called 'Research Framework Excellence'. I'm off to buy some canisters of sarin gas.

Schizophrenia and the Funding of Regional Universities

Hull: one of 'Wave 2 Growth Hubs'
In February the Government announced that it was offering £32m to 20 regional cities in the UK to support economic growth and development. The cities, which included areas as diverse as Milton Keynes, Hull, Stoke and Sunderland, were offered the funding in return for a promise to 'improve a serious local economic problem'.

Ellie Hamilton, the director of the 'Wave 2 Growth Hubs' project at the University of Lancaster, saw universities as key to this. 'Universities are powerful forces in helping cities to develop, through employment, attracting large student populations from the UK and overseas year on year, creating demand for products and services, supporting businesses on their technology parks and linking to business and funding opportunities worldwide.'

Hamilton went on to challenge the binary distinction between London and the rest of the UK. 'It’s not just a case of London against the rest; smaller cities must somehow hold their own as thriving centres alongside the likes of Manchester, Birmingham and Leeds. Helping them to achieve this is one purpose of the second wave of City Deals funding.'

But hang on a minute. Am I the only one to see a slightly schizophrenic attitude to smaller, regional universities here? On the one hand they're being recognised as 'powerful forces in helping cities to develop'; on the other, for the past decade, they have been losing out to the larger, urban (and metropolitan) centres, such as those mentioned by Hamilton.

This has been the result of what's known euphemistically as 'concentration'. Concentration has been an inevitable result of policy moves, such as the development of Doctoral Training Centres, and the frenzied, apocalyptic madness of the RAE/REF. The Russell Group, somewhat inevitably, is all for it.

Kent has done well at both the DTCs and the last RAE, but in some cases has had to link with other regional centres (such as through the AHRC's CHASE) to compete with larger universities. It is now developing more long-term links with UEA and Essex through the Eastern ARC to further safeguard its place in this competitive, concentrated world.

Which makes it all the more galling when the Government makes the case for the economic importance of regional universities. Either you believe in them or you don't. Don't pay lip service to them; instead, make it possible, through policy change, for the smaller, regional universities to compete equitably with the urban centres on a level playing field.

Sunday, 23 March 2014

The Trouble with Citations

Gender and Citation: Nature breaks down the figures
For the past two years the Times Higher (THE) has published a league table of the top 100 universities under 50. This is a spin off of its more established World University Rankings. Whilst I accept that we live in a league table world, and have joked about this in the past on this blog ('Almost Everyone Can Score Well at Something'), I do have a niggling worry about this latest manifestation of the genre.

My worry comes from a concern about the methodology the table uses. THE understandably lays less store in the subjective 'reputation' of these young upstarts (given that they've had less time to develop laurels on which to rest), and concentrates instead on 'hard, objective ­performance indicators':
  • Research: volume, income and reputation (30 per cent) 
  • Citations: research influence (30 per cent) 
  • Teaching: the learning environment (30 per cent) 
  • International outlook: people and research (7.5 per cent) 
  • Industry income: innovation (2.5 per cent). 
The trouble is that 'hard, objective performance indicators' are never quite as hard and objective as you think. 

Take, for example, citations. These have increasingly become the gold standard for research reputation over the last few years. With the rise in the h-index and the g-index, and the ease with which we can access these through Scopus, Google Scholar and elsewhere, citations have become shorthand for 'excellent research.' Indeed, the science of citations, bibliometrics, came close to being used as a proxy for research excellence when HEFCE was deciding on the shape of REF2014

So it's understandable that THE has chosen to give almost a third of its weighting for its league table to citations. Its understandable, but I don't think its acceptable. 

See, if you dig just a little below the surface, you find that citations are riven with difficulties, not least of which is the gender gap that underlies them. It's only recently that some quantitative analysis has been undertaken to assess this. The first time I became aware of such an example was when Daniel Maliniak, Ryan Powers and Barbara F. Walter published their findings on The Gender Citation Gap in International Relations in International Organization in October last year.

Looking at over 3,000 articles published between 1980 and 2006, the authors showed that 'women are systematically cited less than men'. On average, men are cited one and a half times as often as women, getting a mean citation rate of 18.47 compared to 13.65. Maliniak et al offered two essential reasons for this:
  • that women tend to cite themselves less than men;
  • that men tend to cite men more than women.
Going deeper, they suggest that this could be due to the following:
  • that there are 'informal networks' underlying the citation culture. 'In a field numerically dominated by men, men tend to cite men, and women tend to cite women, resulting in disproportionately fewer citations for women'; 
  • second, that 'women may publish less in the early years of their career as a result of their need to take parental leave.' As citations are, to a degree, dependent on profile and recognition, it may just be that women have not had the same opportunities to climb as high, and are reliant on a smaller body of publications;
  • thirdly, in International Relations, there is a (broad) gender divide in topics: men tend to write on security, US foreign policy and methods; women on human rights, comparative foreign policy, health, international law and the environment. These topics may be less popular and well cited.
Sugimoto et al focussed on this issue in Nature in December:
'We find that in the most productive countries, all articles with women in dominant author positions receive fewer citations than those with men in the same positions. And this citation disadvantage is accentuated by the fact that women's publication portfolios are more domestic than their male colleagues — they profit less from the extra citations that international collaborations accrue.' 
They concurred with one of the conclusions of the earlier article, that 'it is likely that many of the trends we observed can be explained by the under-representation of women among the elders of science.'

But, as The Specials said, it doesn't make it alright. However, the situation might not be as gloomy as it appears. The IR Blog reported on an interesting session at the American Political Science Association Conference in 2013 at which Beth Simmons suggested that there may be a glimmer of hope:
  • the citation gender gap seems to be narrowing. This was also picked up in the Washington Post's Monkey Cage blog.  
  • the effect may be more extreme in the States (Maliniak et al were focused on there), and that it may be less pronounced elsewhere;
  • and - hell! - people now know about it, so we could and should act on it. 
Which brings me back to The Times Higher. Concentrating on a bald indicator like citations is retrogressive. It positively encourage universities to focus on the citation rates of its staff. By so doing it exacerbates inequality within academia, entrenching an unrepresentative and unhelpful system that may give a very broad sense of research excellence, but in essence just demonstrates what self belief and lucky breaks an academic's had.

Thursday, 13 March 2014

ESRC Seeks New CEO

The ESRC has started the process of recruiting its new Chief Executive. As is more and more the case, it has put its trust in the hands of headhunters. This is mistake number one. As everyone knows, headhunters almost never understand the specific requirements of academia, and assume it functions like any other business. Thus, Messrs Saxton Bamfylde (for it is they) are advertising the position as follows:

'The Secretary of State for Business Innovation and Skills (BIS) will appoint a new Chief Executive of the ESRC when the post becomes vacant on 1 September 2014. Candidates need to be able to command the respect of the academic and user communities across the domain of the economic and social sciences through their personal achievements in research and their standing in the field. They will also need to demonstrate the ability to provide sound management and strong leadership of a substantial and complex body with efficiency and probity, combined with excellent communication skills required to represent the Council and foster links with its numerous and diverse stakeholders across Government, academia, learned societies, industry/business as well as the public at large, both in the UK and internationally'. 
So let's see. Essential criteria are:

  • Ability to command respect through personal achievement and standing in the field;
  • Ability to provide sound management and strong leadership of a complex body with efficiency and probity;
  • Excellent communication skills;
  • Ability to foster links with numerous and diverse stakeholders.
Given these, I think that the headhunters will come up with a shortlist of six:

  • Candidate 1: Worzel Gummage. Strong track record in commanding respect (amongst crows) whilst standing in a field. Demonstrates sound management and strong leadership (of turnips), and control over complex body (that has the ability to shift between mannequin and human) is admirable. Communicates well, despite a slight speech impediment and a penchant for 'a cup of tea and a slice of cake', and has fostered excellent links with Aunt Sally.
  • Candidate 2: Adolf Hitler: Whilst somewhat brutal, his ability to command respect is undeniable. Strong leadership and excellent communication too: this guy sounds like the complete package. Fostering links is - ah - a bit of an Achille's heel. Tends to make friends one day and invade you the next.
  • Candidate 3: Joan of Arc. Dead white female. Which makes a change, and will do wonders for ESRC's Athena SWAN credentials. Commands respect, especially amongst Catholics. Strong leadership. Excellent communication skills (with God). Once again, fostering links is a bit dodgy. Excellent if you're French. Not so good for the English. Which could be a problem. 
  • Candidate 4: Cat in the Hat. Excellent communication skills, and commands absolute respect amongst under fives. His ability to do tricks and balance things is useful when securing the research budget and justifying the Social Sciences. Can make excellent links with Thing One and Thing Two. Somewhat chaotic, however, and could end very nastily. Nevertheless, if we stipulate that he must bring his all clearing machine, he could be a winner.
  • Candiate 5: Zeus. No problem commanding respect, and good leadership qualities. However, communication skills leave something to be desired; I mean, does he have to keep using thunderbolts? It's a bit rudimentary, isn't it? Hasn't he heard of email? Worse still, we could open ourselves for so many paternity suits and sex scandals. I mean, all those partners! All those children!
  • Candidate 6: Nick Clegg. Oh come on. Be serious.

Wednesday, 5 March 2014

Switzerland Exiled from Europe

EU to CH: none of your penknife/cuckoo clock research here,
thank you very much

These are nervous times for Swiss researchers. On 26 February the EU decided to categorise Switzerland as a 'Third Country' within the new Horizon2020 programme rather than an 'Associated Country'.

So what does that mean? Why should we care?

Well, it means that, whilst Switzerland can participate in (some of) H2020, it's researchers, companies and institutions will be unlikely to receive funding from it, and will not count as one of the minimum three partners needed for most collaborative projects. Moreover, they will be excluded from any 'single beneficiary' schemes, such as those run by the ERC, and the SME instruments.

This does seem slightly odd. I'm sure there's Borgian machinations underlying the decision, but that fact that Israel is an Associated Country but Switzerland is not suggests that maps - or, indeed, an understanding of the geographic limitations of Europe - are in short supply in Brussels.

But never fear: the modern answer to all of the world's ills has ridden to the rescue. Yes, you can now sign an epetition to get DG Research to reconsider. Whilst it's been signed by 7,486 unterstutzer, I wouldn't hold your breath. Epetitions make us feel better about stuff, and it's good to register your thoughts, they haven't got a great track record on actually getting governments (or Eurocrats) to change their minds. I mean, when oh when will the Department of Transport see sense and rename the A63 as the 'Highway to Hull'?

Still, I think both the Swiss and EU officials are doing a disservice to all across (geographical) Europe. European research can only benefit from the participation of the Swiss, particularly when it comes to penknife technology and cuckoo clock research.

Enough stereotypes. Do a little digging and you can see what we (Europeans) are losing. The Complete University Guide highlights the fact that 'Switzerland has the highest ranked university in Continental Europe in the 2013–14 QS World University Rankings: ETH Zürich (the Swiss Federal Institute of Technology), which is ranked 12th. The country also has three other institutions in the QS top 100: École Polytechnique Fédérale de Lausanne at 19th, University of Geneva at 71st, and University of Zurich at 78th.'

We - as a continent - cannot afford to turn our backs on such world-beating, quality research.