Evaluation of projects – the FWF as an evaluation user
The evaluation of projects forms the backbone of the FWF’s work. It represents the basis for quality assurance in all the FWF’s funding programmes. Since the FWF’s establishment, international peer-review has been the instrument of choice. Proven scientists with relevant experience and qualifications (scientists whose work is closely related to that of the applicant, or their “peers”) are consulted and their opinions sought on
- applications for project support (exante–before the funding decision is taken) 1,
- the project’s progress (interim evaluation for quality assurance during the duration of the project, e.g. for large-scale projects in the Priority Research Areas or Graduate Schools programmes) or
- the project’s results (expost–after conclusion of the project).
The FWF does not have a fixed “pool” of referees. Instead, referees are selected for each individual project application, based on their expertise in the relevant subject area. The FWF’s peer-review procedure is highly international in nature: for the past decade and more all referees’ reports come from outside Austria. A third of them come from non-German speaking countries in Europe and a further third from further afield. Furthermore, the FWF attempts to ensure that at least 25% of reports come from female referees. Selection of referees is highly efficient: about 40% of those contacted agree to review applications for the FWF. The vast majority of reports are submitted in writing. However, panel hearings areused in the evaluation of complex large-scale research projects and of excellence programmes: in these, experts interact with applicants.
In all stages of the procedure the FWF relies on a finely tuned system to avoid conflicts of interest and to ensure that all operations are checked (using the “many eyes” principle).
The expost evaluation of projects is the FWF’s most recent procedure. Since 2003, all projects are evaluated after completion. The procedure is significant for several reasons:
- It represents an important element in association with the “global budget” the FWF provides to support projects. The introduction of the global budget was one of the most significant steps in the development of the FWF’s procedures and considerably reduced the administrative complexity and “bureaucracy” scientists faced in carrying out their projects. The term “global budget” is used to mean that project leaders may use the money granted for the project in the manner they think best for the project’s performance and likely to produce the best results in the most efficient way. In other words, deviations from the original planning are permitted without the necessity for consulting the FWF before every change. The FWF’s “philosophy” with regard to the controlling and quality assurance of the projects it funds is simple and clear:
- strict examination before funding, then maximum flexibility for the scientists in carrying out the project; check of the results after completion of the project by international experts.
- This ex post evaluation is taken into account when considering funding in the future. The ex post evaluation gives information on the project’s performance and its success as well as for the FWF’s PR work. Key data from the project are stored in the FWF’s database. Scientific publications and contributions to conferences, the career development of project co-workers (especially young scientists), awards and prizes resulting from the project as well as application-oriented results such as patents and licenses are all recorded, in addition to measures for science communication. The FWF is placing increasing importance on the use of “open access” to disseminate scientific results.
- All of these data form an essential basis for further evaluations and for illustrating the impact of the FWF’s research funding, both in the national and in an international context.
1 Details are given in “General Principles of the Decision-making Procedure”.