In November 2015, the independent High Level Expert Group has released its report containing an ex post evaluation of the EU’s Seventh Framework Programme for Research, as required by the European Parliament Decision and European Council decision setting up FP7. In early 2016 the Commission will publish its response. The objective of this Group was to evaluate the effectiveness of FP7 and its implementation, analysing the main achievements and shortcomings of this Programme, with a view to providing recommendations for Horizon 2020 and all future FPs.
We have analysed the 120 pages long report for you: here is a short summary of the Group’s key findings.
1. Main achievements and improvements with respect to the previous FPs
FP7 was an innovative programme in that it established new sub-programmes, such as the FP7-IDEAS (ERC), which supported individual researchers from all academic disciplines working on excellent frontier research, and the Joint Technology Initiatives (JTIs), which better addressed the needs of industry, allowing more effective collaboration.
Among the major achievements, FP7 had an estimated generation of 11€ of direct and indirect economic effects for each euro spent by the Commission. The macro-economic effects, including an approximate additional annual European GDP of 20 billion euro, were accompanied by micro-economic effects, with participating enterprises reporting innovative product developments, increased turnover, improved productivity and competitiveness.
Concerning the strengthening of European excellence in research, FP7 sub-programmes managed to set and maintain a very high quality standard, attracting excellent researchers both at individual and institutional level, particularly on frontier research. A cross-sector and cross-border cooperation culture was successfully encouraged, engaging large corporations and SMEs alike, through mechanisms such as public-private-partnerships. Mobility of researchers was intensified, as well as the number and extensiveness of investments in European research infrastructures.
2. Shortcomings of FP7
The first, major shortcoming of FP7 has been the presence of inconsistencies and even competing or overlapping elements within the Programme and between it and national programmes.
Secondly, although more than 120 independent evaluations of FP7 have been conducted in the Programme’s lifetime, there has not been a consolidation, validation, comparison or synthesis of this information. The existing data sets have been improved in 2007-2013, but their use has not been channelled towards a systematic generation of strategic intelligence. Moreover, advisory bodies suffered from high fluctuation rates and the outsourcing of Programme implementation activities to distinct agencies has weakened monitoring, governance and control aspects.
3. Recommendations to Horizon 2020 and national research programmes
The independent experts group highlights the need for Horizon 2020 and the future FPs to focus on a number of specific “key areas” in which Europe can play a leading role on a global scale, and to concentrate its efforts on ensuring its competitive advantage on them at a global level. In order to do so, it will be important to engage the private sector more properly, for example by establishing a formal, permanent mechanism of dialogue, as well as by further strengthening JTIs.
In order to further harmonise Horizon 2020 and the future FPs with national programmes, it will be necessary to combine specific policy objectives with decentralised, flexible implementation procedures. This can take place, for example, by a more effective use of Structural Funds and by the establishment of an EU-wide quality stamp for scientific and enterprise driven proposals. This point has perhaps been partially addressed by a recent initiative of the Commission on a new “Seal of Excellence” scheme, whereby regions will be able to recognise the most promising proposals submitted under Horizon 2020 via a quality label.
Concerning a better harmonisation within the Framework Programmes, the structure of Horizon 2020 and future FPs should be carefully analysed with a view to avoiding duplications and of reinforcing synergy potentials. Moreover, the programme structure should facilitate budget transfers between sub-programmes, and the agencies in charge of implementing the Programme should improve their coordination.
A strategic and professional monitoring and evaluation system is highly needed to increase transparency and provide a comprehensive and trusted source of evidence-based decision-making. New key data sets should be created, such as tracing of individual researchers, gender monitoring and proposal evaluation results.
Quantitative measures are also key to understanding the impact of FP7. Here is a selection of the most interesting discussions.
1. Participation of organisations
The A-Group includes the top 500 organisations (research institutions such as the Fraunhofer Gesellschaft, leading universities such as University College London, or industry organisations such as Thales) which were awarded the highest amounts of EC funding in FP7 (60.3%). On average, each A-Group organisation participated in 120 FP7 projects and received a share of about €460,000 per project. The B-Group contains about 4,000 organisations that received 29.1% of EC funding; in this group, Italy, Spain and Portugal are overrepresented. Finally, the C-Group was awarded 10.6% of contribution and includes about 80% of all organisations that participated in FP7.
2 . Participation of countries
At a first glimpse, this table may suggest a bias against EU-13 countries. However, the major difference between the two groups is that EU-15 countries usually host the most prestigious centres of excellence in Europe, which have made substantial investments in highly qualified human resources and support structures. On the other hand, Mediterranean countries have suffered most from the economic crisis, decreasing their public RTD expenditures. The lower shares of funding to EU-13 countries can be rethought when looking at the very low science and innovation funds they receive at a national level, thus proving the key role FP7 has played here.
3. Success rates
An interesting point here concerns FP7-COOPERATION, in which Socio-Economic Sciences and Humanities (SSH) showed a low adjusted success rate (18%), but a high average score of proposals above the threshold but not receiving funding (1,159), as well as a very little chance for resubmission of proposals (1%). This is in contrast with other sub-programmes, in which adjusted success rates are higher, despite a lower percentage of high quality proposals, such as FP7-IDEAS. The authors of the report therefore suggest that while an increase of budget would be justified for SSH, two improvements are necessary in FP7-IDEAS: expectation management and training for applicants, in order to avoid an overflow of weak proposals and to help proposers with high potential achieve better results.
4. Impact of FP7 on mobility
As may be expected, this Programme showed a significant flow from the Mediterranean area to countries with strong research and innovation systems. Given these figures, it is important to maintain this kind of programme as having as a main target the active rotation of scientists, rather than promoting a “brain drain”. This can be achieved, for example, by ensuring that national research systems are open for returning researchers.
Two more themes are mentioned throughout the Report, and deserve a specific discussion.
The first one concerns the financial and economic crisis that hit Europe in 2008. Throughout the report, many references are made to the (positive) ability of FP7 to adapt to the changing environment caused by this crisis; according to the authors, this was made possible by the fact that the Programme had very general aims, which allowed for flexibility. A doubt arises here: the fact that FP7 had such generic aims was also stated as one of its main weaknesses, since leaving ambiguous targets may lead to the risk of not attaining them. Therefore, the recommendation that Horizon 2020 and future FPs should focus on key, strategically important challenges seems to be somewhat in contrast with the need of remaining flexible in view of potential unexpected challenges. On the other hand, this contradiction can be resolved when leaving flexibility at the structural level, such as by allowing budget transfers from one sub-programme to another. Similar measures could mitigate the risk provided by specific, concrete objectives.
The second theme concerns the importance given by the report to communication aspects of research. Among the main conclusions, the report in fact lists “Bring science closer to the European people”. While acknowledging the improvements made by FP (e.g. by creating the Science in Society programme), the authors also point to the lack of civil society organisations in relevant decision-making bodies, such as evaluation boards or expert groups. Improved dissemination strategies, meaning more tailored and targeted activities, should therefore be encouraged. This has also the potential to increase trust in science and innovation by the citizens, which in Europe seems to be very low compared to other regions of the world.
Finally, in a brief overview of Horizon 2020, the authors report the three most discussed items in the scientific community at the moment:
- Competition is higher for Horizon 2020 than it was for FP7; be it for the decrease of national funding or else, the authors here suggest better investigating the root causes for this, but only taking into account those proposals which have passed the threshold.
- Administrative simplification has already been implemented, but further steps are necessary, especially taking into account trust issues.
- The decrease of the flat rate for the reimbursement of indirect costs from 60% in many cases of FP7 to 25% in Horizon 2020. Although this has been often criticised, it should not be forgotten that in Horizon 2020 the funding rate of direct costs has been increased to 70% and 100%, from 50% or 75%.
Although the recommendations of this ex-post evaluation have been reflected to some extent in Horizon 2020, there seem to be specific areas that still need further improvement:
- Ensure focus on critical challenges and opportunities in the global context;
- Align research and innovation instruments and agendas in Europe;
- Integrate the key components of the Framework Programmes more effectively;
- Bring science closer to the European people;
- Establish strategic programme monitoring and evaluation.
As Horizon 2020 soon celebrates its third year of business, we will continue collecting and analysing all available data and information, including those based on our own experiences.
Follow our blog to read about these in the coming months.
