How we select the best research projects
At Epilepsy Research UK, we are determined to identify the best research for funding. We look for innovative research ideas, of the highest scientific merit, that have the greatest potential for making a real difference to patients’ lives. We also want value for money, and work completed on time.
In order to do this, we run an annual funding competition using a two-stage process to evaluate proposals. Our Scientific Advisory Committee (SAC), which consists of eight epilepsy researchers from a variety of scientific backgrounds and two lay people, make the final decision about which applications to fund. Using a committee of scientists to evaluate other scientists’ work is called peer review. As part of our peer review process we also borrow the expertise of independent specialist researchers around the world.
Our selection process
Projects and fellowship applications
In the first instance, applicants are required to submit a preliminary application. This is only two to three pages long and it include a lay description of the planned work and a short technical proposal.
At this stage the SAC are interested in the quality of the science to be undertaken, i.e.:
- Is the question being asked worth answering?
- Are the investigations proposed the best way to find an answer to the question?
Each member of the SAC independently assigns a score to every application and an average score for each is calculated. The committee then holds a meeting (usually in November) where the applications are discussed and the most promising are shortlisted. These candidates are then invited to submit a more detailed (‘full’) application. We normally ask for full applications for two to three times as much research as we’re able to fund, so that we capture a good range of research ideas.
CONFLICTS OF INTEREST
In order to avoid bias, SAC members do not score proposals for which they have a conflict of interest – i.e. those from their own institution, and those where one of the applicants is someone they have previously collaborated with or know personally. Members are required to declare their conflicts of interest at the beginning of the SAC meeting, and they leave the room when these applications are being discussed. This procedure is implemented, not only during the application process, but also when grant progress reports are being reviewed.
Full applications require much more detail about just how the question posed in the preliminary application will be answered, e.g. how many patients will be tested? What equipment will be used and why? How will the results be analysed? The application also requests a short CV from each applicant and a detailed costing.
For each of these proposals, the SAC selects two (in the case of project grant applications) or three (in the case of fellowship applications) independent reviewers. These are scientists working anywhere in the world in the same scientific field as that of the applicant. They must not have worked with the applicant before, but they are likely to be aware of them and their work. The external reviewers are sent the proposal confidentially, and return a critique of it to the SAC. In the 2013-2014 grant round, 41 experts in the UK, USA, Canada, Germany, France, Italy, Australia and New Zealand provided reviews.
An anonymised copy of each external review is sent to the applicant, and he/she then has the opportunity to respond to the comments raised.
Taking into account the expert reviews and the applicants’ responses, the SAC again independently assign scores to each application (which are then averaged). At this stage, the SAC has to consider factors other than pure science:
- Do the applicants have the necessary expertise to successfully complete the project?
- Is the proposal value for money?
- Which proposals have the greatest potential for patient benefit?
At their final meeting (in March), the committee discusses each shortlisted proposal and decides which will be funded.
Pilot grants in epilepsy
These smaller grants were introduced in the 2012-2013 grant round and require one-stage application process. Applicants are asked to submit a full application at the same time as the preliminary applications for project and fellowship grants. These are scored by the committee, the average score for each is calculated, and a shortlist of approximately six is drawn up at the November SAC meeting. The shortlisted proposals are sent to one external reviewer and the SAC takes their comments into consideration to re-score the applications. Please note that the candidates do not get the opportunity to respond to their reviewers’ comments. At the March SAC meeting, the committee discusses each pilot grant and decides which will be funded.
Why use peer review?
The peer review system is almost universally used in medical science. It has flaws but it is the best system available. Epilepsy Research UK’s procedures aim to ensure that our process is accountable, balanced, independent and impartial.
Using peer review to select research is a badge of quality for medical research charities. It is a criterion for membership of the
Association of Medical Research Charities. It’s also used by the Higher Education Funding Council of England to prioritise which charity-funded research at universities will be eligible for top-up funding.
We aim to make the process as constructive as possible to all applicants by feeding back comments from external reviewers. This means that even if a project is not funded, the applicant should have gained some ideas about how to improve their next proposal.
Problems with peer review
Peer review has obvious flaws. It can be difficult for people in one scientific discipline to appreciate the importance of another, leading to bias. Peer review relies at all stages on scientists’ behaving honourably, not borrowing ideas, being overly enthusiastic about friends’ projects or overly negative about rivals’. It can be very incestuous in small research communities, where everybody is commenting on each other’s work. It’s also quite a secretive process and it tends to focus on the negative points of studies rather than the positive ones. The quality of review and exact standards being applied can very hugely between reviewers.
However there is no ‘evidently better’ system. Bibliometrics – a method of analysing researchers’ publication records for quantity and quality – is not as widely accepted. Open review, where a proposal is published and anyone can comment, is also not favoured by researchers, who don’t want anyone to steal their ideas.
At Epilepsy Research UK, the members of our SAC are carefully selected to be representative of the wide range of scientific disciplines which contribute to epilepsy research, and also of the geographical distribution of research centres in the UK. We also have two lay members on our SAC who help to counteract any bias and ensure that patient benefit is at the forefront of people’s minds.
We don’t require members of our SAC to refrain from applying for funding themselves while they’re on the committee, as the UK epilepsy research community is too small to allow this. We also don’t want to be responsible for taking excellent researchers ‘off the market’ for five years whilst they sit on our committee. We do, however, restrict the number of active grants that SAC members can hold at any time to one project grant and one pilot grant.
We do feed back reviews to applicants at the full application stage, although anonymously (if reviewers know their name will be sent back to the applicant, their reviews tend to be less critical and we need the most honest opinions we can get).
We never use external reviews in isolation, we always ‘filter’ them through the eyes of the SAC. This way we maintain a quality standard as far as we can. It’s in everyone’s interests that we fund the best research projects we can. More information about peer review can be found on Sense About Science – a highly recommended read.