Peer Review Guide
The Tri-Agency Interdisciplinary Peer Review Committee

• 1 Introduction
• 2 Objective
• 3 Review Process
  • 3.1 Important Dates
  • 3.2 Assigning Applications
    • 3.2.1 Committee Orientation and Calibration
  • 3.3 Review of Applications before the peer review meeting
  • 3.4 Review Process
    • 3.4.1 Evaluation Criteria and rating scale
    • 3.4.2 Peer Review Meetings
  • 3.5 After the peer review meeting
• 4. Roles and Responsibilities
  • 4.1 Chair
  • 4.2 Scientific Officers (SO)
  • 4.3 Peer Review Committee Members
  • 4.4 External Assessors
  • 4.5 Funding Agency Staff
• 5. Description of Evaluation Criteria
• 6. Resources

1. Introduction

The aim of this document is to serve as a resource for reviewers adjudicating applications submitted to the Tri-Agency Interdisciplinary Peer Review Committee. This shared peer review committee is accessed via the following gateway funding opportunities: SSHRC Insight Grants and Insight Development Grants, CIHR Fall and Spring Project Grant, and NSERC Discovery Horizons.

On behalf of SSHRC, CIHR, and NSERC, we would like to thank the peer review committee members. These dedicated experts who generously give their time and expertise are essential to ensuring the success of the peer review process.

If you need further guidance or would like to have more information about this review process, please do not hesitate to contact:

• SSHRC: research@sshrc-crsh.gc.ca
• CIHR: ProjectGrant-TIR-SubventionsProjet@cihr-irsc.gc.ca
• NSERC: horizons@nserc-crsng.gc.ca

2. Objective

Individuals applying for SSHRC Insight Grants and Insight Development Grants, CIHR Fall and Spring Project Grant, and NSERC Discovery Horizons will be eligible to be considered for evaluation by the Tri-Agency Interdisciplinary Peer Review Committee. This committee provides an option for researchers working in interdisciplinary research to direct their application to a committee with expertise from across the social sciences, humanities, natural sciences, engineering, and health sciences.

The agencies recognize that the terms "multidisciplinary," "transdisciplinary," and "interdisciplinary" have different connotations and that there may be a lack of consistency in their usage across and between research disciplines. For the purpose of this committee, "interdisciplinary" is used as an umbrella term to refer to all three concepts inclusively.

Applications considered relevant for review by this committee must represent research across disciplines and subject areas pertaining to two or more of the (1) social sciences and humanities, (2) natural sciences and engineering, and (3) health and wellness, and that clearly articulate interdisciplinary approaches.

Interdisciplinarity must be a key characteristic of proposed projects, where the project goals could not be achieved without an interdisciplinary approach.

3. Review Process

3.1 Important Dates

In addition to the above timelines, the following activities will take place for each Gateway Funding Opportunity according to its timelines:

3.2 Assigning Applications

Following the submission of full applications, committee members will be asked to complete the Ability to Review exercise. To do so, members are given access to the summaries of proposed research projects to declare any conflicts of interest and indicate their level of expertise (i.e., High/Medium/Low/Not enough expertise). Given the interdisciplinarity nature of the committee, members are encouraged to take a broad view of their own expertise as it relates to interdisciplinary applications and provide context regarding their expertise, where relevant. The workload of the reviewers will be balanced across the applications from the three agencies.

Each application will be assigned to 5 committee members: 3 as Reviewers and 2 as Readers (see section 4, below, for more details on the roles):

• The application will be assigned by agency staff to three committee members as Reviewers with different but relevant perspectives based on the subject matter of the application.
• The application will be assigned to two additional committee members as Readers (who will read but not provide preliminary ratings on the application, although they will vote on the applications to which they are assigned).
• The agencies will also seek to obtain, but cannot guarantee, at least two External Assessment Reports for every application from subject matter experts of the content of the application.

3.2.1 Committee Orientation and Calibration

Agency staff will convene preliminary meetings of the Tri-Agency Interdisciplinary Peer Review Committee in advance of the peer review meeting for the purposes of committee orientation and committee calibration.

As this committee’s processes and evaluation criteria have been harmonized and will include members from across the social sciences, humanities, natural sciences, engineering, and health sciences, the committee orientation provides an opportunity to explain processes and procedures that will apply to this committee, as well as address any questions from reviewers stemming from their previous peer review experiences. Agency staff will also introduce the peer review platform and other related information technologies being used to support the peer review meeting.

The committee calibration offers an opportunity for discussion of the consistent use of evaluation criteria and the scoring system, time-management strategies for the adjudication meeting, and other topics.

3.3 Review of Applications before the peer review meeting

All applications must be treated as protected information and all committee members will sign a tri-agency Conflict of Interest and Confidentiality Agreement.  Committee members must not divulge application contents to others or use any information herein for any other purpose than peer review. Committee members are also asked to ensure that all Peer Review materials are handled safely and disposed of according to the document “Guide on Handling Documents used in Peer Review”.

3.4 Review Process

Reviewers will be asked to submit a written review whereby they will provide strengths and weaknesses of the project in relation to the Tri-Agency Interdisciplinary Peer Review Committee evaluation criteria within the context of the specific funding opportunity to which the application was submitted. Reviewers and Readers should consider, but are not bound by, the External Assessment Reports.

Note: A template will be provided to the reviewers to facilitate their written report (which can then be copied and pasted into the online system).

After the reviewers have submitted their final reviews, they will be able to access the other finalized reviewers’ reports for that application. These reports can be used in preparation for the discussion at the committee meeting. Reviewers will have the opportunity to update their written reviews after the peer review meetings.

If applicable, reviewers may also comment on the budget requested and make a formal recommendation, including clear and detailed reasons for any recommended budget cuts.

3.4.1 Evaluation Criteria and rating scale

To support the strategic objectives of this funding opportunity, the following evaluation criteria will be used with the goal to support the specific strategic objectives of the funding opportunity the applications have been submitted to while taking into account particular context of that funding opportunity. Reviewers will enter scores for each of the three criteria, which will be given equal weight.

See section 5 below for a more detailed discussion of the evaluation criteria.

The following rating scale will be used:

Applications are submitted according to the guidelines, formatting, page limits, required attachments etc. of the relevant funding opportunity. Applicants should not be penalized as long as they are following the rules appropriate to their individual agencies.

3.4.2 Peer Review Meetings

The Tri-Agency Interdisciplinary Peer Review Committee will meet over multiple sessions to review the eligible and relevant applications submitted via the agencies’ funding opportunities. Applications submitted to each funding opportunity will be reviewed in separate sessions in accordance with the funding opportunity timelines. Members are expected to participate in at least two to three sessions per year. Please note that the agencies may streamline applications based on existing practices, and as a result the committee may not discuss all files.

One reviewer will be assigned as ‘primary’ reviewer and will open the discussion by presenting their assessment of the strengths and weaknesses of the proposal according to their perspective and specific to each of the evaluation criteria for the funding opportunity. The remaining reviewers and readers follow and supplement the discussion by concentrating on points of agreement or disagreement, and elaborating points not yet addressed, with attention given to their specific perspective. The open discussion will be structured and discussion will proceed according to the order of the evaluation criteria. Where necessary, agency-specific guidance may be provided, such as regarding the review of budgets.

Once the discussion is complete, the Chair will invite reviewers and readers to confidentially vote on the application.

Additional guidance from the committee may be sought on specific points such as budget recommendations, ethics, eligibility, etc, as appropriate to the funding opportunity.

Upon conclusion of the review and rating of all applications, reviewers will be asked to provide feedback to the funding agencies regarding the effectiveness and functioning of the committee and the peer review process in its entirety.

3.5 After the peer review meeting

Reviewers who wish to make changes to their reviews will have one week to do so after the meeting. Reviewer reports will be made available (without the reviewer’s name or score) to the applicants as part of the feedback package provided by their agency. As such, reviewers should refrain from divulging their identities in their reviews to ensure the confidentiality of the review process.

The review should be clear and concise, using objective and non-inflammatory language, and include justification (see, for example, the CIHR College of Reviewer’s Review Quality webpage). Constructive advice to the applicant will allow them to improve the quality and efficiency of the proposed research. Note: in circumstances where the Chair or staff deems that the review expresses comments that could be construed as sarcastic, flippant, arrogant, or inappropriate in any way or contain factually incorrect information regarding funding agency’s policies, the funding agencies reserve the right to remove such information from the review.

While applications from the three funding opportunities will be reviewed by the shared Tri-Agency Interdisciplinary Peer Review Committee, funding decisions will be made, funding will be provided, and grants will be administered by the agency to which the application was submitted, in accordance with that agency’s respective guidelines and decision making processes. Similarly, applicant feedback packages will follow the format of the agency to which the application was submitted.

4. Roles and Responsibilities

The Tri-Agency Interdisciplinary Peer Review Committee consists of a Chair, three Scientific Officers, and Peer Review Committee Members. In addition to the committee, External Assessors will also be engaged to support the committee through the provision of written External Assessor Reports. The committee is supported by program staff at each of the three agencies.

4.1 Chair

The committee Chair is responsible for ensuring that the peer review committee functions smoothly, effectively and objectively. They establish a positive, constructive, and fair-minded environment in which the applications are to be evaluated. The Chair (along with the Scientific Officers and program staff) fulfills an oversight role and does not rate applications. Their responsibilities include:

• Working with program staff to manage conflicts of interest;
• Ensuring the involvement of the committee members, as appropriate, with recommendations for each application;
• Ensuring that the perspectives of all reviewers assigned to an application are heard and that they participate actively to the discussion;
• Guiding the committee discussions that will inform the final voting score;
• Working with the Scientific Officers and/or program staff to summarize the discussion concerning each application; and,
• Abide by the agencies’ guidelines on confidentiality and conflict of interest.

4.2 Scientific Officers (SO)

The Scientific Officers (SO) are responsible for supporting the Chair during the peer review process as appropriate, including by acting as back-up chairs when the Chair has a conflict of interest. In addition, the SO will take notes on the committee proceedings which may form part of the applicant feedback packages. Each agency recruits an SO. In addition to the duties shared with the Chair, the Scientific Officer:

• Supports the Chair in their role throughout the peer review process, including by acting as a back-up chair in the cases where the chair has a conflict of interest;
• Takes notes of the discussion as it is proceeding, which may be sent to applicants as part of their feedback package, according to existing agency practice;
• Ensures that issues of ethics, budgetary adjustments and other concerns that have been flagged for the attention of the funding agencies are recorded for each application, following each agencies’ existing practices; and,
• Abides by the agencies’ guidelines on confidentiality and conflict of interest.

4.3 Peer Review Committee Members

Peer Review Committee Members are recruited from across the social sciences, humanities, natural sciences, engineering, and health sciences. They will be assigned as Reviewers and Readers to applications assessed by the committee, as supported by External Assessor reports. The primary responsibilities of the peer review committee members are to:

• Indicate their ‘ability to review’ (i.e., level of expertise and conflicts) for each application;
• Provide a reviewer report and rating for each assigned application as Reviewer.  Reviewers should consider, but not be bound by, the External Assessment Reports;
• Review all application material (including External Assessment Reports) and be prepared to discuss and vote on applications to which they are assigned as Reviewer and Reader;
• Attend the committee meeting and participate in the discussions of the applications; and,
• Abide by the agencies’ guidelines on confidentiality and conflict of interest.

4.4 External Assessors

The funding agencies will seek to obtain, but cannot guarantee, at least two External Assessment Reports from subject matter experts for each application. Committee members should take into consideration, but are not bound by, these reports while conducting their reviews. External Assessors are not part of the committee. The role of the External Assessors is to provide written expert reviews of the application(s) assigned to them.  External Assessors will also abide by the agencies’ guidelines on confidentiality and conflict of interest.

4.5 Funding Agency Staff

Funding agency staff are involved throughout the funding opportunity process, including supporting the Peer Review Committee in its work. The responsibilities of Funding Agency staff include:

• Assigning applications to committee members;
• Identifying and recruiting External Assessors based on the content of the applications;
• Providing guidance and advice to members on agency guidelines and policies;
• Keeping notes on procedural aspects of the committee’s functions;
• Recording recommendations and concerns raised by the committee on issues requiring later attention by staff (e.g., ethics, etc.); and,
• Recording feedback about the effectiveness and functioning of the committee as a whole and the meeting process.

5. Description of Evaluation Criteria

The Tri-Agency Interdisciplinary Peer Review Committee employs harmonized evaluation criteria. While the specific definition of terms may vary between agency-specific funding opportunities, applicants and committee members will consider the harmonized description of the evaluation criteria. Please note that not every participating funding opportunity has the same requirements in their applications and that different amounts of space may be provided between applications. Applicants and committee members are asked to communicate with their agency contacts if they have any questions.

Where relevant, committee members will also apply funding opportunity specific guidelines. Agency staff will provide these materials during funding opportunity-specific orientation sessions.

Merit of the Proposed Research

Quality of the Proposal

• The clarity, scope and originality of research questions and objectives

  Applicants will clearly describe the context and needs (issues and/or gaps) of the proposed project, including its importance, originality, and anticipated contribution to knowledge. The objectives of the project will clearly define the proposed lines of inquiry and/or activities required to meet the goal.

  The proposed project must be based on a logical integration of concepts including a proper framing of the research in the context of the relevant scholarly literature with appropriate reference(s) to other relevant work in the field. The theoretical approach or framework (if applicable) will be well-defined and clear.

  Committee members will consider the extent to which the proposal suggests and explores novel or potentially transformative concepts and lines of inquiry and the extent to which the proposal will lead to advances in the knowledge in the relevant field(s).

• Extent to which the proposed research advances disciplines across the mandate areas of at least two funding agencies.

  Applicants will clearly describe how their proposed research contributes to disciplines supported by at least two of the funding agencies. This contribution should be well integrated into the application.

Methods and Approaches

• Appropriateness of the proposed methods and approaches, including the integration of interdisciplinary approaches to achieve the project goals.

  Applicants will provide a clear and detailed description of the research methodologies (approaches/procedures) that will be used to achieve the stated objectives. The methodologies must be appropriate for achieving the stated research goals. Applicants will also clearly integrate interdisciplinary approaches into the proposed research.

  The timelines for the project should be appropriate in relation to the proposed project activities, and the applicant should demonstrate how the project’s goals and objectives will be met within the proposed timeline.

  Critical challenges should be identified (such as scientific, technical, or organizational), and the applicant should present a realistic plan and mitigating measures to tackle these potential risks. An exhaustive list is not expected.

• Appropriate incorporation of sex, gender and/or diversity in the research proposal, if applicable

  Applicants will describe their consideration of sex (biological), gender (socio-cultural) and/or diversity in their research design, if applicable. This may include clearly describing the rationale and methodology for including sex, gender and diversity in the research design as well as demonstrating how aspects of sex, gender and/or diversity are addressed in the research design, methods, analysis and interpretation, and/or dissemination of findings.

  If appropriate, committee members will be asked to provide recommendations to the applicants on how they might improve the strength of their applications with respect to the integration of sex, gender and/or diversity.

• Evidence of commitment to the development of active and meaningful partnerships with appropriate individuals and organizations, if applicable

Feasibility

• Extent to which the budget is appropriate and justified in relation to the proposed activities

  Applicants will demonstrate appropriate budget planning and clearly explain the suitability of the budget in relation to the proposed methodology(ies) and expected results in terms of scale and feasibility of research plans. They should also clearly describe any financial and in-kind contributions from other sources, where appropriate, including demonstrating that funds requested in the current application are not for expenses supported or submitted for support through other sources.

  The committee is asked to adhere to the principle of “Minimal Essential Funding”, which requires committee members to carefully examine budgets and cut expenses which are deemed to be inflated or if they judge that a particular request is inappropriate or inadequately justified, and if it is confident that savings can be achieved without jeopardizing the project objectives. Committee members will provide a justified recommendation as to whether the budget should be: Accepted, as described; or Reduced by a specific dollar amount or percentage (total), as appropriate.

  Note: The committee will not take the budget into account in determining its final consensus score for CIHR funding opportunities; however, it will consider the budget in determining scores for SSHRC and NSERC funding opportunities.

• Appropriateness of the environment (academic institution and/or other organization) to enable the conduct and success of the proposed activities

  Applicants should have access to the appropriate infrastructure, facilities, support personnel, equipment, resources, and/or supplies to carry out their respective roles. This might include evidence of commitment to the development of active and meaningful partnerships with appropriate individuals and organizations, if applicable.

  Committee members will assess project environments, according to their ability to support the proposed project activities. Institutions often function as “networked” environments or interdisciplinary networks, which means there may be multiple satellite environments contributing to the support environment.

  Committee members also should consider that for smaller institutions and/or affiliated research facilities that resources and/or services may be obtained through networks or may be contracted out.

Anticipated Outcomes

Impact

• Potential for the outcomes of the proposed activities to have influence and impact

  Applicants must clearly describe the potential for the project results to have influence and impact within and/or beyond the project’s research field(s), such as the potential for policy impacts or the achievement of results with importance to a broad range of applications or the potential to advance knowledge in the field and influence the direction of thought and activity. When applicable, knowledge translation and/or commercialization approaches and/or methodologies should be described, as well as opportunities to apply research findings nationally and internationally.

  Committee members will consider whether the anticipated impact(s) are realistic, i.e., directly stemming from the project outputs, as opposed to marginally related, as well as the significance of developed applications to general and/or limited end users (firms, institutions, etc.).

Knowledge mobilization and dissemination

• Quality and appropriateness of knowledge mobilization plans, including effective dissemination, exchange and engagement with stakeholders within and/or beyond the research community, where applicable

  Applicants will clearly describe the anticipated contribution(s) (e.g., publications in peer-reviewed journals and other outputs, as appropriate) of the project through a knowledge mobilization plan that includes a description of the proposed dissemination, exchange and engagement with stakeholders within and/or beyond the research community, as appropriate. These proposed project outputs should be aligned to the project’s objectives.

  Committee members will consider whether the anticipated contribution(s) are realistic, i.e., directly stemming from the project outputs, as opposed to marginally related. In addition, committee members will determine whether the research and/or knowledge translation and/or commercialization approaches, methods and/or strategies are well-defined and justified in terms of being appropriate for accomplishing the objectives of the project.

Training Plan

• SSHRC and NSERC Only: Extent to which the proposed activities incorporate plans for the training and development of highly qualified personnel

  Applicants must provide a training plan which describes the specific roles and responsibilities of students, research assistants, and other highly qualified personnel including how their duties will complement their academic training (if applicable). The plan will describe how the applicant will interact with research personnel as well as the approach taken to train and impart knowledge to future researchers, including the skillsets to be imparted to ensure their success.

  The research training and development opportunities provided for highly qualified personnel can include, but are not limited to: participation and involvement of highly qualified personnel in outreach activities; interdisciplinary research experiences; promoting equity, diversity, and inclusion in the research community; as well as collaborations, and/or interaction with the private and public sectors.

  Committee members will consider the appropriateness of the level and mix of highly qualified personnel for the proposed research project (e.g., is the proposed work suitable for an undergraduate student, a master’s student, PhD candidate, or postdoctoral fellow?). Committee members will also take into consideration, where appropriate, the quality of training opportunities to be provided to students, such as the level of intellectual involvement of highly qualified personnel in the proposed research project and how this training will complement the students’ own academic training (i.e., the proposed research should leave room for growth and development and highly qualified personnel should be more than simply extra hands for the researcher).

• NSERC-only: Appropriate inclusion of considerations of equity, diversity and inclusion in the training plan

  As appropriate, applicants will describe how they will increase the inclusion and advancement of under-represented and disadvantaged groups in the research enterprise as one way to enhance excellence in research and training. They should include a discussion of the challenges or barriers encountered in ensuring an inclusive research and training environment as well as a qualitative description of specific actions implemented to support equity and inclusion in recruitment practices, mentorship approaches, and initiatives aimed at ensuring an inclusive research and training environment and trainee growth.

Important: trainee demographic data is not requested, nor required to assess impacts related to equity, diversity and inclusion in the research and training environment.

Applicant(s) Record

Expertise and Experience

• Appropriate expertise to undertake the proposed research, including complementary and interdisciplinary knowledge, expertise and experience

  Applicants will clearly demonstrate the combined expertise and experience needed to execute the project. While the principal investigator’s achievements should form an important component of the evaluation of the team, as this person is responsible for the planning, co-ordination and intellectual direction of the entire research project, the team’s overall capability will be assessed as a collective in light of each member’s role in the project, placing more weight on the achievements of those with more central roles.

  The level of engagement (e.g., time and other commitments) of the applicant(s) and/or research team should be appropriate to the roles and responsibilities described as well as their capacity to undertake the planned program given their commitments to other research endeavours, as presented in the application. The rationale for the team composition with regards to diversity and equity must be explained, as applicable.

  Committee members will consider, as applicable, whether the research team contains the required expertise from complementary disciplines to address the defined objectives competently and complete the research successfully. The contributions of individuals to the research effort must be clear.

• Experience and track record of the investigator(s) – importance, originality, quality and impact of past research

  Committee members will consider contributions made by the applicant(s) over the past six years and based on their stage of career. Committee members are also required to take into consideration any special circumstances or career interruptions that may have delayed or interrupted their research career during that period. Applicants with eligible leaves of absence may include contributions from their most recent active research period prior to the last six years for a period equivalent to the duration of the leave.

  Assessment of this sub-criterion must be based on the quality and impact of all contributions, not only on the number of publications or conference presentations. Venues with the highest impact (as measured by readership or attendance) may not be the most appropriate for an applicant’s research results and it is the responsibility of the applicant to explain the choice of venues for dissemination. As a result, committee members will assess productivity broadly (i.e., not just based on publications) following the San Francisco Declaration on Research Assessment (DORA), which recognizes that scholarly outputs are not limited to published journal articles but can include a broader range of outputs.

  Possible outputs include, but are not limited to:

  • refereed contributions (e.g., books, book chapters, article in scholarly refereed journals, review articles, conference proceedings, papers presented at scholarly meetings or conferences, articles in professional or trade journals, etc.);
  • non-refereed contributions (e.g., book reviews, published reviews of the researcher’s work, research reports, policy papers, chairing conference sessions, public lectures and other knowledge mobilization activities, etc.);
  • creative outputs (e.g., exhibitions, performances, publications, presentations, film, video, audio recordings, etc);
  • involvement in public outreach activities (e.g., organizing research promotional events, taking on leadership positions in research outreach, etc.);
  • contributions to the research enterprise (e.g., membership on committees, editorial boards and/or advisory boards, and/or the promotion of equity, diversity and inclusion in the research enterprise); and,
  • research results from previous research grants.

Past Contribution to Training of Highly Qualified Personnel (HQP)

• Quality and extent of past contributions to the HQP

  Applicants must demonstrate their contributions to the training and mentoring of future researchers in a research training environment that generates new knowledge or insights, including the supervision of graduate theses and/or the involvement of students in research activities in the context of their current career stage (i.e., Early Career Researchers may not have had the opportunity to develop a large training record).

  Committee members will consider the depth and breadth of highly qualified personnel’s participation in the applicant’s research contributions (e.g., as co-authors, depending on the discipline), which can include but is not limited to conferences, presentations, publications, patents, and/or technical reports; and/or awards, scholarships and/or fellowships won by the highly qualified personnel. Other examples of training impact may include the progression of highly qualified personnel into further studies and/or careers that have impact, whether as professionals in the private, public, and/or academic sectors. While this impact can be either within or beyond the research enterprise, it needs to be clear how the skills gained in the applicant’s research training environment are being used by the highly qualified personnel.

6. Resources

Applicants should review the funding opportunity specific guidelines and documents available on their websites:

• SSHRC Insight Grants
• SSHRC Insight Development Grants
• NSERC Discovery Horizons
• CIHR Project Grant

In addition to the funding opportunity specific resources, applicants should also review the following tri-agency guidelines:

• Selecting the Appropriate Federal Granting Agency
• Research Ethics
• Responsible Conduct of Research
• Conflict of Interest and Confidentiality
• Tri-Agency Open Access Policy on Publications
• Tri-Agency Guide on Financial Administration

Date modified:

2024-07-05