The FWF awards contracts for evaluations, studies, and other research policy-related services to independent experts with strong professional backgrounds in the relevant field. All contracts are awarded using a transparent selection procedure based on pre-defined criteria¹. The FWF supports the principle of open access to scholarly knowledge and provides open access not only to research results arising from FWF-funded projects, but also to evaluations, studies, and other research policy-related services commissioned by the FWF, and to information on the costs that arise from these contracts. Where legally possible, this applies to all results and data arising from these studies, evaluations, and other research policy-related services².

This document defines the FWF's approach to commissioned evaluations, studies, and other research policy-related services, and the technical criteria for open access to the resulting data and publications. The Appendix defines the metadata structure that evaluators and study authors need to provide.

For a funding agency like the FWF, it is crucial that any evaluation of the organization’s programs meets the same high quality standards imposed on the review and evaluation of FWF-funded research projects. For this purpose, it is necessary to define and publish the criteria applicable to the evaluations of the FWF’s funding programs.

One of the main objectives of these standards is to ensure independence and impartiality as key quality criteria for proper evaluation activities, and to address the issues raised by the often necessarily close relationship between evaluators and the FWF as the organization funding such program evaluations. To maintain this transparency, FWF follows the guidelines of the Austrian Platform for Research and Technology Policy Evaluation (fteval)³.

The FWF Executive Board makes all decisions on the awarding of evaluation contracts and their execution, and documents these decisions.

Quality

Evaluations of FWF programs:

• Are scheduled either at the start of the program or at appropriate, cyclical intervals after the start of and during the running time of programs
• Are generally only assigned to independent, professional evaluators (or institutions) with relevant experience and expertise. Depending on the specific context of the program, evaluators (institutions) from outside Austria are given preference wherever possible.
• With regard to design and execution, evaluations must generally fulfill the quality standards stipulated by the Fteval; these standards must also be included in the terms of reference (TORs).
• Contracted evaluations of FWF programs must be advertised in a public invitation to tender. However, the FWF reserves the right to evaluate very small-scale programs in-house, based on publicly accessible indicators.

Transparency and supervision

For future evaluations of its programs, the FWF will establish an impartial auditing system which ensures compliance with international quality standards.

In the interest of maximum transparency and impartiality, these audits will be carried out by at least one international, independent expert or with the involvement of international sister organizations or other relevant institutions (e.g. Science Europe).

These experts will review and comment on the following:

• Content/wording of the TORs
• Content/wording of the invitation to tender
• Draft of the final report

These experts will also participate and cooperate in the following activities:

• Project presentations by bidders (as necessary)
• Selecting a bidder for the contract award
• Presentations of interim findings (as necessary)
• Approval of the final report
• Presentations of the findings (as necessary)
• Public presentations of the final report (as necessary)

For each evaluation, the FWF has to prepare its own written comments, which, like the evaluation report, must be published. The supervising expert's final comments on the evaluation process must also be included in these comments.

The FWF Executive Board makes all decisions on the awarding and execution of studies and other research policy-related services and documents these decisions. If the budget is €10,000 or higher, the contract must be awarded through a public invitation to tender based on transparent criteria.

One special case is the award of contracts in cooperation with other organizations. In such situations, a joint procedure for the contract award is to be defined on a case-by-case basis; however, an FWF Executive Board decision and the relevant documentation are always required.

The results of research policy-related services (e.g. consultations and similar services) must be documented for the Executive Board, and at least a brief summary must be published on the FWF website.

Quality and supervision

FWF contracts for studies and other research policy-related services are awarded only to independent experts with a strong professional background in the relevant field. Depending on the topic in question, experts residing outside of Austria are to be given preference whenever possible. These studies may be carried out with the involvement of third parties (national partners, international sister organizations, or other relevant institutions) if necessary.

The findings of FWF-commissioned evaluations, studies, or other research policy-related services are generally published in accordance with the FWF's Open Access Policy, the Berlin Declaration on Open Access to Knowledge in the Sciences and Humanities and the Open Government approach by the European Commission.

Technical criteria under the FWF's Open Access Policy

The following technical criteria for open access to evaluations, studies, and data must be observed:

• A suitable repository which is registered in the Directory of Open Access Repositories is to be selected for the storage of publications and data arising from FWF-commissioned evaluations and studies
• Publications and data must be assigned a Digital Object Identifier (DOI)
• Publications and data are to be published under the Creative Commons Attribution CC BY or a similar open licence
• Publications are to be made available in PDF and in HTML or XML format (XML: http://www.biomedcentral.com/about/datamining); and data in the respective data-specific format
• All relevant output (metadata, publication text, underlying data, citations, licensing information) must be made available in a machine-readable form via “open standards.”⁴.

Metadata

The following metadata template applies for every published report:

¹ Since January 1, 2023, pursuant to Art. 20 para. 5 of the Public Procurement Act (Bundesvergabegesetz, B-VG), public contracting authorities are obliged to “…publish studies, expert opinions, and surveys commissioned by them, together with their costs, in a manner accessible to the general public, as long as and to the extent that their secrecy is not required pursuant to para. 3. (statutory obligation to maintain secrecy).”

² Potential conflicts of interest must be disclosed.
Offers for evaluations, studies, or other research policy services by former FWF employees can only be considered after a period of five years. A declaration of impartiality must be signed in any case.  Furthermore, former employees cannot be considered if they were directly involved with the subject matter at the FWF or held a management position during the period in question.

³ Paraphrased from the evaluation standards of the Austrian Platform for Research and Technology Policy Evaluation (fteval). The fteval is a nonpartisan association of all stakeholders involved in evaluation in Austria, www.fteval.at

⁴ In this context, the standards of the Open Archives Initiative Protocol for Metadata Harvesting (OAI-PMH) are to be applied.