Analyzing outcomes and impacts
We have a responsibility to demonstrate the impacts of our science and technology investments, to assess the efficiency of the public spending we oversee, and to assess our contribution to achieving social and economic objectives for Canada. But measuring and evaluating the outcomes of CFI investments is complex. Research and innovation is inherently risky and outcomes and impacts linked to research infrastructure are difficult to measure. For example, there’s the time lag between when the investment is made and when the outcomes are realized, and it is often challenging to attribute the results of research to any particular investment. That’s why we use a range of data and assessment approaches to evaluate progress at the organizational level through to the societal level.
A new approach to mapping investments to impacts
Our presentation at the French National Institute for Agriculture Research conference in Paris
A socioeconomic impact assessment is a systematic analysis of the economic, social and cultural impacts, outputs and outcomes related to a particular set of investments. Understanding and measuring the impacts of public research and development is necessary to evaluate the efficiency of public spending and assess its contribution to achieving social and economic goals.
Impact Analysis of funding for research and development in medical imaging
In partnership with the Canadian Institutes of Health Research, we undertook a pilot project to quantify the socioeconomic impacts of our collective investments in medical imaging.
The study highlights the importance of an open and competitive research system to move innovation along the continuum from new ideas, to working prototypes, to products which benefit Canadians.
The project benefitted from an advisory committee and expert peer review who contributed both their time and their expertise to this pilot study.
To request a complete copy of the technical report, O’Connor, Alan C. and Albert N. Link. 2013. Pilot socioeconomic impact analysis of CFI and CIHR Funding: Medical Imaging. Prepared for CFI and CIHR by RTI International, please contact evaluations [at] innovation.ca, Director, Evaluation and Outcome Assessment.
Mapping investments to impacts is an approach we use to explore our contribution to the national research landscape in a given thematic area with specific emphasis on the pathways to impact.
Agricultural research in Canada
The CFI, working in collaboration with its partners, has gathered evidence to identify benefits to Canadians stemming from five areas of agricultural research, and uncover common pathways from research to impact. The Agricultural Institute of Canada, a national organization that advocates for agricultural research and innovation, compiled an overview of the Canadian landscape as context for this study.
The five specific areas of agricultural research were selected for the study in consultation with an advisory group composed of representatives from 20 organizations including universities, federal and provincial government departments and agricultural associations.
- Greenhouse gases (GHG) — research on evaluating agricultural policies, techniques and tools to help mitigate Canada’s impact on the environment
- Resilient crops — research on seed and plant genetics to make them more valuable, less vulnerable and increase yield
- Dairy farming — research on all facets of the industry including dairy cattle welfare, milk production, dairy by-product processes and genetic evaluation
- Antimicrobial resistance (AMR) — research on mitigating the growing risks of antimicrobial resistance through novel prevention and therapeutic approaches in animal agriculture
- Grain storage — research on methods, technologies and structures used to house or prepare grain for storage
The CFI has committed over $200 million toward 510 research infrastructure projects at 61 institutions in the area of agriculture and food science between 1998 and 2016. This represents about four percent of total CFI investments made in research infrastructure during the same period. The CFI funds up to 40 percent of a project’s research infrastructure costs. Partner funding represents more than $550 million invested across Canada, for a total investment of more than $750 million in infrastructure to advance agricultural research.
Key Pathways to impact highlighted in the summary report
- It is through strong collaboration linkages that new ideas become innovations and yield benefits to Canadians.
- Trainees are central to creating and strengthening linkages across and within sectors.
- Engagement between researchers and farmers accelerates the uptake of new technologies
Feeding the country’s need for new knowledge in agriculture
Canadian agriculture has long relied on leading-edge research and technology development to keep pace with the rapidly changing ways we produce food. Read our collection of stories to learn more.
Platform outcome measurement studies allow us to examine large-scale, specialized, purpose-built research infrastructure that serves specific needs of the Canadian research community, while considering the uniqueness of each platform.
The Canadian Coast Guard Ship (CCGS) Amundsen contains a comprehensive pool of specialized scientific equipment and facilities for research in the Canadian Arctic. The research platform was made possible after a consortium of 15 Canadian universities and research centres, in partnership with the federal government, received funding from the CFI for the retrofit of the decommissioned Canadian Coast Guard icebreaker Sir John Franklin as an Arctic Ocean research vessel. This contribution of $27.5 million from the CFI International Joint Venture Fund included nearly $19.4 million for structural transformations to the ship for science, and for scientific equipment as well as $5.5 million in support for the operations of the platform.
We conducted a platform outcome measurement study that assessed the activities and achievements of the Canadian Research Icebreaker, Canadian Coast Guard Ship (CCGS) Amundsen.
Key findings of the Expert Panel:
- The Amundsen platform is enabling science of the highest international quality and is facilitating the translation and application of new knowledge to address societal issues of major consequence for the Arctic regions of Canada and for other Arctic settings.
- The Amundsen research program has had a major impact on the productivity, reach and influence of Canadian Arctic science as shown by the strong publication record and by the seminal papers produced on such topics as sea ice and ecological research in the Beaufort Sea.
- The Amundsen program has engaged a diverse set of end-users encompassing federal and provincial science-based government departments and agencies, industry, and communities.
Canadian Research Knowledge Network
The Canadian Research Knowledge Network (CRKN) is a partnership of Canadian universities dedicated to expanding access to digital content for the academic research enterprise in Canada. Between 2000 and 2012, CRKN has received two CFI contributions for a total of $97.7 million, including partner funding from provincial governments and Canadian universities. We assembled a panel of experts to assess the outcomes and impacts of CRKN in accordance with the POMS framework.
Key findings of the Expert Panel:
- CRKN is among the leading information-enabling organizations worldwide and is recognized as a “game changer” for the Canadian research community.
- The investment in CRKN by the CFI and its provincial government partners was essential, timely and catalytic, and has already been returned many times over.
- CRKN has well-developed and efficient operations to address its current mandate; however, recent changes to the governance structure were viewed as potentially detrimental to its future.
- CRKN may not have the resilience with its current level of resources to deal with the ongoing transformation in scholarly communication and the ways in which digital content is used by the research community.
Prior to 2011, we conducted a series of Outcome measurement studies designed to assess the degree to which our investment in research infrastructure is a critical contributing factor in the realization of five outcomes: strategic research planning, research capacity, highly qualified personnel, research productivity, and innovation. These early efforts to measure outcomes and impacts have helped to shape our current approaches to outcome and impact assessment.
Methodology for Outcome measurement studies
In December 2010, we published a paper about the outcome measurement studies methodology in Research Evaluation, an international, peer-reviewed journal.
The Canada Foundation for Innovation’s outcome measurement study: a pioneering approach to research evaluation (PDF)
Outcome measurement study summary report
In 2008, independent consultants examined the findings of the Outcome measurement studies conducted to date. Here is their report: