Definition of health research waste

The seminal commentary by Altman (1994)¹ is considered^2,3 to be among the first to declare the “scandal” of health research waste (HRW). Even though not using the term HRW, Altman discussed poor design, poor conduct, misconduct, misleading academic and industry governance, touched on measures of research impact, and raised ethical concerns – covering most aspects of today’s discourse about HRW.

The term HRW was introduced 15 years later when Chalmers & Glasziou (2009) published a commentary where they claimed that “85% of all health research is being avoidably ‘wasted’”⁴. This commentary generated considerable academic interest, with Google Scholar reporting over 2900 citations as of April 2026⁵, including the seminal Lancet series on HRW^6–8. It is instrumental to recapitulate the components underlying the estimate, which “commonly elicits disbelief”⁹:

1. Registered clinical trials “never published in full”¹⁰.
2. Published reports of research that are not “sufficiently clear, complete, and accurate for others to interpret, use, or replicate the research correctly.” This includes measured endpoints that are “often not reported,” methods and analysis that are “poorly explained,” and interventions “insufficiently described for others – researchers, health professionals and patients – to use”⁶.
3. New research studies not “designed to take systematic account of lessons and results from previous, related research”¹¹.

Drawing on evidence addressing these three components, Chalmers & Glasziou estimated “the percent of research that does NOT satisfy these stages” at 87.5%, rounding down to 85%.

The original definition of HRW was, therefore, explicitly deontic¹², meaning that it was based on a view of how health research ought to be done, as opposed, for example, to an axiological treatment, based on direct estimation of a study’s value and the loss of that value. While several variations on the definition of HRW have been published since (see^6,7,13–19 for a non-exhaustive list), the framing appears to remain consistently deontic. This includes considerations beyond methodology such as views on how research ought to be prioritized or disseminated. A notable recent example is the inclusion of HRW into the World Medical Association’s Ethical principles for medical research involving human participants (Declaration of Helsinki^20,21), paragraph 21: “Medical research involving human participants must have a scientifically sound and rigorous design and execution that are likely to produce reliable, valid, and valuable knowledge and avoid research waste” (emphases added).

I, too, avoid a primarily axiological framing of HRW because, in contrast to the economic evaluation of health interventions^22–24, utility values have not been established for the different states that a research study may occupy in its design and execution (e.g., ranging from “flawed beyond recovery” to “state of the art”). Buxton et al. produced a significant body of research dedicated to measuring the economic value of health research^25–27, but they found that existing evaluations focused almost exclusively on estimating direct cost savings to health care systems, with a few evaluations focusing on social value such as employment statistics. No evaluations focused on direct spending on research. The tool proposed in this work could help establish research utilities in the future (Section 1.5).

Therefore, I follow the established deontic tradition and define an instance of potential HRW as a documented failure of a study to follow conduct or reporting guidelines appropriate for the research question or study design, or a flag identified using a research assessment tool. I insist on the hedged wording (i.e., potential HRW) because the existing guidelines and tools vary widely^28,29 in terms of their measurement properties (e.g., validity and reliability³⁰), and the goal of this work is to establish a comprehensive array of HRW flags and make them available to researchers and knowledge users (KUs, i.e., those who can directly use research results³¹) at the point of use, rather than compare and select among them. As such, I refrain from definitive HRW labelling in this work. I further unpack the definition by specifying the eligibility criteria below.

Eligibility criteria

Guidelines are defined according to the U.S. National Library of Medicine as “statements of principles or procedures that assist professionals in ensuring quality in such areas as clinical practice, biomedical research, and health services”³². Tools refer to quantitative measurement instruments as outlined in COnsensus-based Standards for the selection of health Measurement INstruments (COSMIN)³⁰ or qualitative/mixed-methods approaches used for purposes of research improvement, evaluation, assessment, critical appraisal, peer review, funding review, or ethics review across health or other fields. Health research is defined according to the Canadian Institutes of Health Research (CIHR) as biomedical, clinical, health service, social, cultural, environmental, and population health research³³.

Failures generally fall into Methodological, Invisible, Negligible, Underreported, or Structural (MINUS) categories¹⁹ (Table 1).

Abbreviations: HRW, health research waste.

Some reports identify research failure modes that may be difficult to classify using MINUS, for example, sexism leading to reduced research productivity³⁹, the excessive production of environmental waste in research studies, contributing to climate change⁴⁰, and the proliferation of poorly reported studies using large language models (LLM)⁴¹. I include these under my definition of HRW if they specify the guideline or tool to be used to detect the failure.

HRW is documented in methodological studies, defined as “studies that evaluate the design, analysis or reporting of other research-related reports”⁴², including qualitative field studies such as interviews of researchers and KUs (e.g.,⁴³), or reports by investigative journalists or so-called “scientific sleuths” (e.g.,^44–46).

Across guidelines, tools, and methodological studies, I will only include publicly available documents to be able to verify the claim of non-adherence. This includes academic publications, preprints, study register records, as well as documents that are difficult to locate or “not published commercially and may be inaccessible via bibliographic databases” (i.e., grey literature⁴⁷). Private correspondence or other materials not intended for public disclosure will not be included.

Several concepts intersect with HRW; I clarify how they are treated in this work:

Impacts of health research, according to a systematic review by Cruz Rivera et al., encompass five groups of metrics⁴⁸: 1) primary research-related impact (e.g., number of citations), 2) influence on policy-making (e.g., number of policies enacted), 3) health and health systems impact (e.g., patient outcomes, cost savings), 4) health-related and societal impact (e.g., health literacy), and 5) broader economic impacts (e.g., return on investment). The term traditionally implies societal benefits, with few authors applying it to the harms⁴⁹. Establishing connections between HRW and research impacts lost to HRW is outside of the scope of this proposal. I therefore exclude research impacts from my definition of HRW.

Research integrity, as per the European Code of Conduct for Research Integrity, “encompasses the basic responsibility of the research community to formulate the principles of research, to define the criteria for proper research behaviour, to maximise the quality, reliability, and robustness of research and its results, and to respond adequately to threats to, or violations of, good research practices”⁵⁰. It is a component of the “trinity of good research,” proposed by Kolstoe & Pugh as an overarching term for research integrity, ethics, and governance⁵¹. Research integrity violations amount therefore to “bad research”^3,52 and include falsification, fabrication, and plagiarism⁵³, as well as so-called unacceptable (questionable) research practices. The latter category is poorly defined and includes dozens of specific manifestations^50,54–56, which I omit here because all of “misconduct,” “unacceptability,” and “questionability” appear to have a deontic connotation (i.e., as something one ought to avoid), suggesting therefore that the entire category is eligible under my definition of HRW above – provided that the methodological study under consideration identifies what specific guideline or tool is used to establish the violation. This suggestion is in tentative conflict with the point of view that misconduct is distinct from HRW due to its assumed intentional nature²⁹. However, feedback from my late patient partner, Dr. Lisa Ridgway, JD⁵⁷, and my dissertation committee raised reasonable doubt as to whether this distinction is clear. Furthermore, multiple methodological studies, guidelines, and tools cover retractions^46,58 and integrity investigations^45,59, and they are difficult to disentangle from guidelines and tools related to HRW. As such, I include research integrity violations in my definition of HRW.

Examples of eligible studies

85% of HRW amounts to 35.5 million publications (out of 41.7 million in the field Medicine; OpenAlex, March 2026^60,61). I highlight several illustrative examples of HRW evidence to be included in AWARE. Importantly, in all examples the claim that the findings described below constitute HRW was made explicitly by the authors of the included methodological study. As such, it does not necessarily represent scientific consensus or my own interpretation.

• Randomized trials are widely regarded as one of the highest levels of evidence⁶², yet 16–22% of clinical trials remain unpublished⁶³, and of those published, 92% are judged to be at a high or unclear risk of bias³, limiting their value for decision-making.
• Patients are randomized to placebo in new trials despite conclusive evidence of intervention effectiveness from meta-analyses, leading to preventable harms: Jia et al. (2021, 2023) identified 6775 additional major adverse cardiac events in non-treatment groups across 3352 redundant trials, including 1334 more deaths^64,65.
• 90% of systematic review search strategies are irreproducible⁶⁶, and an additional 67 “problems with systematic reviews” were identified^67–69; major overlaps are reported in evidence synthesis efforts, including but not limited to systematic reviews⁷⁰;
• 67% of clinical practice guideline developers do not involve patients in the development process, increasing the risk that recommendations are misaligned with patient priorities⁷¹.

While this evidence has traditionally focused on randomized controlled trials (RCTs) or systematic reviews and has largely originated from studies conducted in the United States⁴, these do not represent exclusion criteria in this proposal but rather demonstrate the inherent biases in the evidence.

No tool covers all HRW elements

There is no established tool to match a health research study to the guidelines it ought to follow, or the aspects of HRW it is particularly subject to and therefore ought to avoid – which is the gap this proposal addresses.

Initiatives address some of the issues, including but not limited to reporting⁷², risk of bias⁷³, patient and public engagement⁷⁴, Indigenous quality appraisal⁷⁵, and prioritizing research questions according to policy needs⁷⁶. A 2024 scoping review identified 93 methods for assessing HRW across 1037 relevant “reviews with a systematic search”²⁹. Using the eligibility criteria from Section 1.2, my ongoing searches show that between January 2024 and March 2026, at least 2–3 new HRW studies were added to PubMed each week^77,78. Existing tools are not holistic and typically cover only 1–2 out of 5 MINUS; no pre-existing tool covers all MINUS elements²⁹, making it difficult for researchers and KUs to assess a study’s value for decision-making.

Indeed, a 2026 survey of researchers serving as grant application peer reviewers or research-performing organizations’ hiring committees found that while 90% of them considered credibility assessment of research proposals very important, fewer than half felt confident assessing it⁷⁹. This gap was greatest for assessing research integrity and methodological completeness, with 57% relying on journal reputation and Journal Impact Factor as proxies of research excellence despite recognizing their limitations⁷⁹. Similar concerns were voiced by research ethics boards (REBs)⁸⁰ and patients^81–83. My own consultations with patient partners have likewise highlighted the need for a patient-centered tool to address HRW⁸⁴.

Major resources cataloging HRW tools have not been user-friendly. Among systematic review peer reviewers, 45% expressed interest in a consolidated resource integrating existing tools⁸⁵. The EQUATOR Network^72,86 catalogs hundreds of reporting guidelines but is not available at the “point-of-use” such as manuscript review, requiring users to visit the website and sift through which guideline to use (Table 2).

Abbreviations: HRW, health research waste; MINUS, MINUS framework¹⁹.

A 2021 initiative GoodReports.org addressed some usability concerns but as of March 2026 is no longer operational⁸⁸. LATITUDES⁸⁹ and riskofbias.info⁹⁰ are catalogues of quality appraisal tools with similar limitations: not available at point-of-use, each website serving a specific purpose, all covering limited HRW aspects. The “comprehensive framework for INcreasing QUality In patient-oriented academic clinical REsearch” (INQUIRE)⁹¹ has seen limited uptake, with only 24 citations from 2018–2025 – perhaps due to its limited dissemination beyond academic journals – unlike this proposal that aims to release a free online tool and features a multichannel knowledge mobilization strategy (Section 3.3).

Overall, existing HRW tools provide siloed sets of recommendations but do not provide the functionality of the proposed tool (Section 1.6), which capitalizes on these lessons: the focus on co-creation, availability at point-of-use (e.g., journal submission, ethics review, funding review), and a “one-stop-shop” approach to evaluating a study.

Aims and design overview

I propose to create a web-based tool called Avoiding WAste in health REsearch (AWARE) that allows researchers and KUs to upload a study report or protocol and holistically appraise it using guidelines and tools related to HRW, supporting the assessment of its value for decision-making. The proposal uses a multi-method co-design approach (Figure 1): scoping review to comprehensively map existing evidence (Aim 1), interviews and focus groups to elicit researcher and KU needs for a new tool (Aim 2), and co-development and pilot-testing of the tool with researchers and KUs (Aim 3).

Aim 1 – Scoping Review: Identify and characterize methodological studies, guidelines, and tools relevant to HRW assessment, and build an evidence and gap map linking identifiable study attributes to HRW-related problems and solutions, forming a prototype for the tool. Outputs: An online, interactive, and printable evidence and gap map; plain-language summary (example⁹³); protocol and report published as two peer-reviewed open access journal publications; and a presentation at a local conference.

Aim 2 – Interviews and Focus Groups: Collect and analyze insights on researcher and KU needs for decision-support in the evaluation of health research. Outputs: A peer-reviewed, open access journal publication; a presentation at the national level.

Aim 3 – Tool Co-development and Pilot-testing: Co-develop AWARE and pilot-test its usability, validity, and reliability. Outputs: An open source tool on the tool’s website⁹⁴ along with a printable PDF version; source software code for the tool; a user guide; free webinar and online educational module; peer-reviewed, open access journal publication; and a presentation at an international conference.

Planned tool overview and workflow example

Design overview. AWARE may be used by researchers, patient partners, and other KUs both prospectively, before committing to a project role (i.e., uploading a protocol), and retrospectively, after a study’s completion (i.e., uploading a study manuscript). While the final design will be shaped by participants of Section 3.2 and Section 3.3, I currently envision AWARE as a three-screen workflow (Figure 2).

Infrastructure. I plan to implement AWARE as a client-side application running in the web browser – a so-called local-first approach. This approach is beneficial for AWARE because it offers increased data ownership for users, enables fully featured offline work, and remains operational without central maintenance⁹⁵. Under this architecture, the AWARE website will display static webpages served from a Content Delivery Network (CDN), which is a distributed global network of computers where the geographic location is determined dynamically by the CDN provider and is often at the closest location available (e.g., in Canada or the US). Once the static page is loaded and cached in the web browser, it contains all the components necessary for AWARE to operate. I have experience with such implementations, including developing a plugin for the diagramming software Draw.io⁹⁶ and an in-browser Tampermonkey script⁹⁷.

Privacy policy. The AWARE application is executed locally inside the web browser using the cached static assets and code. In this fashion, individual data entered by the user remain on-device, substantially reducing privacy risks. If a data input field needs to be shared with a third party (e.g., a study identifier is used to pre-populate bibliographic information), this will be clearly indicated and the user may choose how they wish to proceed. The privacy policy will be adapted from a similar policy elsewhere (e.g.,⁹⁸) and co-developed with researchers and KUs in Aim 3 (Section 3.3).

Artificial intelligence (AI) policy. AWARE is not an AI tool. However, I anticipate that users may wish to use external AI tools (e.g., ChatGPT, Perplexity) to facilitate data entry regardless of whether this was intended. To support these users, AWARE will provide AI-friendly prompts that can be copied to the clipboard or exported into a local file, which may then be pasted or imported into an external AI tool; AI-produced artifacts may then be pasted or imported back into the checklist form. Before using the export feature, a notice will be shown that the user is responsible for the data privacy of any interactions outside of AWARE and will explain potential implications. Likewise, before using the import feature, a notice will be shown that explains common shortcomings of AI-generated data and offers “tips & tricks” on how to avoid them. The wording of these notices will be informed by similar policies elsewhere (e.g.,⁹⁹) and co-developed in Aim 3. This will help ensure that AWARE’s scope is clearly defined and presented in an accessible format while keeping the development feasible within this thesis project.

User interface. Screen 1 – Intake gathers basic information about the study to streamline the assessment process. Users can enter a study identifier such as a digital object identifier (DOI) to automatically retrieve data from public repositories (e.g., Crossref, DataCite), which can be edited. If no identifier is available, study details can be manually entered. Users then select which perspective they are assessing the study from: (1) peer reviewer, (2) research ethics reviewer, (3) funder reviewer, (4) point-of-care clinical decision-maker, (5) policy reviewer, and (6) general patient and public user. While there is no universally established taxonomy of situations in which researchers or KUs may need to assess health research, this list originates from the SPIRIT (Standard Protocol Items: Recommendations for Interventional Trials) 2025 statement¹⁰⁰. Findings of Aim 2 will help clarify the similarity, differences, and potential overlap between the perspectives (Section 3.2).

Screen 2 – Checklist presents the checklist based on the perspective that was selected. Relevant information from the intake screen is automatically populated. Fields can be manually completed.

Screen 3 – Flag report returns a final screen with potential problems (red and yellow flags), strengths (green flags), and suggested actions or solutions. Each flag is accompanied by supporting citations and relevant excerpts to ensure transparency and traceability.

Illustrative example:

Screen 1 – Intake. User perspective: Peer reviewer. Study: A randomized trial of an anticholesterol medication (statin) versus placebo for preventing heart disease.

Screen 2 – Checklist. Established reporting and appraisal checklists, e.g., CONSORT¹⁰¹, SPIRIT¹⁰⁰, RoB 2⁷³, Evaluating Public and Patient Involvement in Interventional Research (EPPIIC)⁷⁴.

Screen 3 – Flag report. The report might identify methodological concerns (e.g., high risk of bias), transparency gaps (e.g., incomplete reporting or protocol detail), engagement limitations (e.g., no patient engagement), redundancy signals (e.g., similarity to other trials on the same question), or integrity issues (e.g., citing a retracted paper).

Research standpoint

I identify with dialectical realism^105–110, which emphasizes that researchers believe in what they do. I believe that following relevant research guidelines and applying assessment tools can help reduce HRW; accordingly, I mention the ones I used or plan to use throughout this proposal.

Patient-oriented research is central to this proposal, which was conceived, in part, in response to concerns about the detrimental effects of HRW on patient outcomes^64,65 as well as scrutiny over public research spending^10,111,112. In Section 3.2 and Section 3.3, patient and public partners will specifically be recruited to tailor AWARE to their perspectives. Patient and public involvement throughout this project will be reported using the Guidance for Reporting Involvement of Patients and the Public (GRIPP 2 long form)¹¹³. I will measure the degree of meaningful patient engagement in Aims 1–3 using the Patient Engagement In Research Scale (PEIRS-22)¹¹⁴ to ensure accountability and improve my practices.

Aim 1 (DDD Discover) – Scoping review

Overview. I will conduct a scoping review to create an evidence and gap map of HRW studies, that will form a prototype for AWARE. As informed by the Right Review tool¹¹⁵, a scoping review is a type of knowledge synthesis that can identify/clarify available research and gaps in research and, as such, is the most appropriate for this aim¹¹⁵. I will conduct the scoping review using the JBI guide¹¹⁶ and report using the Preferred Reporting Items for Systematic Reviews and Meta-Analyses extension for Scoping Reviews (PRISMA-ScR 2018)¹¹⁷.

Registration. I will register the protocol with the Open Science Framework (OSF)⁷⁷. Registration will enhance transparency and reduce the risk of unplanned duplication of other research¹¹⁸. All screening and data abstraction forms and code will be provided. In preparation of this proposal, I sought and incorporated feedback from Drs. Louise Rosengaard^19,29, Lesley Uttley^67–69, and David Moher^6,8,86 (a member of my dissertation committee), who previously published on and developed guidelines/tools related to HRW, to prevent overlap and ensure value in this research.

Search strategy. I will import an existing dataset from Rosengaard’s MINUS scoping review^19,29 for screening, which comprises 1037 studies published in August 2023 or earlier. I will also import the Uttley’s Systematic Reviewlution living review dataset^67–69, which comprises 637 included articles published in May 2023 or earlier. I will include INQUIRE⁹¹ (plus 90 studies from the linked systematic review¹¹⁹) and SciScore^120,121 as the closest tools I am aware of that resemble AWARE’s content and workflow, respectively. Additionally, an information specialist will conduct literature searches across multiple sources: Databases: MEDLINE, Embase, OpenAlex⁶⁰, Web of Science, CNKI, PsycINFO, CINAHL, and others. Grey literature: OpenGrey, Grey Matters⁴⁷, ProQuest, Overton, and Google. Preprint servers: medRxiv, bioRxiv, arXiv, and OSF. Registries: PROSPERO, OSF, SWAR Repository. Websites: EQUATOR, JBI, Cochrane, LATITUDES, RoRI¹²², AIMOS¹²³, COS Metascience¹²⁴. No limits will be applied by date or language. The main search strategy will be peer-reviewed by a second librarian using the PRESS checklist¹²⁵. I will use citationchaser software¹²⁶ for backward and forward citation tracking of included articles to identify additional publications. I will search OpenAlex⁶⁰ and Retraction Watch¹²⁷ for errata and retractions for all included studies. Searches will be reported using PRISMA-Search¹²⁸.

Eligibility criteria. I will include HRW methodological studies that describe, analyze, or critique the development, validation, or application of HRW guidelines and tools as defined in Section 1.1. This will be relevant for quantitative, qualitative, and mixed-methods research study designs and will not be limited by the type of study design, including trials, reviews, and other designs. Each included study must link explicit study attributes or features to a “flag“ representing a “problem” or a “solution.” Editorials, commentaries, and opinion pieces not reporting original evidence will be excluded, as will tools that have not been used in the real-world. I will also exclude quantitative tools for which no clinimetric/psychometric properties (e.g., validity as defined in the COSMIN taxonomy³⁰) are reported.

Automation tools. I will use the Responsible use of AI in Evidence Synthesis (RAISE) recommendations^129,130 and report AI use according to the Transparent Reporting of AI in Comprehensive Evidence Synthesis (trAIce) checklist¹³¹. For eligibility assessment, I will machine-translate reports in languages other than English using a Generative Pre-trained Transformer (ChatGPT) model. Median agreement between data extracted from machine translations and from human translations by native speakers ranged from 50% to 85% across nine languages, versus 100% for English, in 2012^132,133, and from 85% to 97% in 2019¹³⁴. As of 2024, GPT-4 “performs comparably to junior translators in terms of total errors made but lags behind medium and senior translators” while “translation capability gradually weakens from resource-rich to resource-poor language pairs”¹³⁴. Contingent on funding, I will obtain professional translations to ensure inclusion.

Study selection. Duplicate records will be removed using DistillerSR screening software¹³⁵. Together with a second reviewer, we will independently assess titles and abstracts against the eligibility criteria, preceded by a pilot on a random sample of 73 records (calculated using the z-score, assuming the confidence level of 95% and the anticipated inclusion rate of 5.00±0.25% as previously estimated⁷⁷). I will calculate interobserver agreement using Randolph’s free-marginal kappa (target κ ≥0.70)^136,137. Full texts will be retrieved, uploaded to DistillerSR¹³⁵, and independently assessed for eligibility by me and a second reviewer. Discrepancies will be resolved by discussion. I will generate a PRISMA 2020 flow diagram^138,139 to document the study selection process.

Data abstraction. I will collate duplicated reports across conference abstracts, preprints, and publications related to the same study. A standardized data extraction form will be developed iteratively at the start of Aim 1 by me and reviewed by a second reviewer. The data abstraction form will capture study characteristics and the HRW flag (problem or solution) being reported, including examples of use; data items are specified below. Multiple flags or usage examples may be identified in a single study; these will be counted separately. I will identify additional examples of use from forward citation searches. If included in the MINUS²⁹ or Systematic Reviewlution⁶⁹ datasets, the data will be reused to pre-fill the abstraction form. Together with a second reviewer, we will independently pilot-test the form on a subset of included studies and refine it as needed to ensure clarity and consistency until complete agreement on the form content is achieved. Following piloting, I and a second reviewer will independently extract data from all included studies in DistillerSR. Discrepancies will be resolved by discussion.

Data items. Study characteristics: authors, affiliations, year and venue of publication, source of funding, major study design features, and theoretical background. HRW flags: study attributes or features linked to the flag, a concise description of the flag, supporting evidence, and known examples of use, with references. EDI considerations: according to the PROGRESS-Plus framework^140,141, Place of residence, Race/ ethnicity/ culture/ language, Occupation, Gender/ sex, sexuality, Religion, Education, Socioeconomic status, and Social capital, and other intersecting factors (e.g., age, disability)^142,143. I will abstract sex and gender considerations using the Sex and Gender Equity in Research (SAGER) flowchart¹⁴⁴. Addressing socioeconomic status, I will abstract the study’s country of origin and match it to the World Bank country classification (Fiscal Year 2026)¹⁴⁵ to establish high and low-middle income economy countries (LMICs).

Data analysis and synthesis. I will export extracted data into a spreadsheet and analyze it using descriptive statistics. Together with a second reviewer, we will independently synthesize data using qualitative mixed-coding content analysis, consistent with scoping review guidance¹⁴⁶. In this process, abstracted flags will be categorized by MINUS and study attributes converted into checklist questions. We will calibrate the coding procedure on a subset of studies until complete agreement on the process is reached. Conflicts will be resolved by discussion. Identified tools and their key characteristics and methods will be summarized and presented in online, interactive, and printable tables and diagrams to produce an evidence and gap map^93,147, forming a prototype for the tool.

Aim 2 (DDD Define) – Interviews and a focus group

Overview. I will conduct semi-structured interviews and a focus group to identify researcher and KU needs for a new HRW tool¹⁴⁸. Each participant will be offered a choice between a one-on-one interview and a focus group, and data collection will be tailored to their preferences, recognizing that some patient partners may be more comfortable participating in a small group^81,82. I aim to collect participant insights on the perceived need and appropriate ways for conceptualizing and evaluating HRW, solicit requested tool features, and identify barriers and facilitators to using such a tool. The findings from Aim 1 will inform interview guide development as described below. I follow Lincoln & Guba’s (1985) guidance to strengthen the trustworthiness of qualitative research¹⁴⁹ and report findings using the Consolidated Criteria for Reporting Qualitative Research (COREQ) checklist¹⁵⁰. The study will be pre-registered with OSF, and the interview guide will be provided.

Team. I received training in qualitative interviewing and worked with interview recordings and transcripts as part of my doctoral coursework and research experience. In addition, I have a strong track record of non-research “professional conversation”^151,152 with diverse individuals and in diverse settings, including clinical work, policy-making, and co-development of software with non-technical users. This has equipped me well to lead Aim 2 sessions; for additional support, I will engage an experienced facilitator during all sessions.

Study setting. The study activities will be hybrid (in person/phone and online) in affiliation with CIHR’s Strategy for Patient-Oriented Research (SPOR)¹⁵³ Evidence Alliance (SPOR EA)^154–157. The SPOR EA encompasses 465 members across Canada.

Participants. An optional self-identification questionnaire will be offered during onboarding. The participant characteristics that I primarily target are the anticipated key HRW tool user profiles (i.e., the proposed six AWARE perspectives – Section 1.6), aiming for at least 2–3 participants per each characteristic. However, I recognize that these, even if relevant, do not capture the full diversity of potential AWARE users. Reflecting on the intersectional¹⁴³ factors that might affect user experience, I aim to recruit at least two participants per each of the additional perspectives: low-bandwidth or no regular internet access; limited familiarity with research evaluation; limited research or digital literacy; limited access to academic literature; physical or cognitive accessibility needs; and diverse areas of professional expertise (e.g., various medical specialties, social science). I consider additional equity factors under the PROGRESS-Plus framework^140,141: For feasibility in this study setting, I focus on Canada’s demographics. 80.3% of Canada’s total population reside outside of the Greater Toronto and Hamilton Area (GTHA)¹⁵⁸; 17.8% of the total population reside in rural areas. Accordingly, I will recruit primarily from outside of the GTHA, aiming for about a fifth of participants to be represented by residents outside of any metropolitan area. I will aim to recruit an approximately equal number of women and men in accordance with the United Nations Sustainable Development Goal 5 “Gender Equality”¹⁵⁹ and CIHR’s Sex and Gender-Based Analysis Plus framework¹⁶⁰. Being mindful of Canada’s gender/sexual and racialized/ethnic disparities as well as the colonial legacy, it is important to be aware that sexual/gender diverse people represent 4.4% of the total population, Black, 4.3%, and Indigenous, 5.0% (Statistics Canada, 2021^158,161); given the sample size below, I anticipate including at least one participant per each of these characteristics.

Sampling and recruitment. I will recruit participants from the SPOR EA network and its connections. I will use purposive maximum-variation sampling to reflect the participant profiles described above. I aim to recruit 2–3 participants per each AWARE perspective and 12–18 participants overall, which is consistent with sample sizes used in previous research¹⁶². In keeping with the “information power” approach to qualitative sample size determination, I will recruit more participants as necessary to sufficiently reflect the study aim, specificity, theory, dialogue, and analysis^163,164. The recruitment will continue on a rolling basis until the purposive targets described above are adequately represented. If a participant needs to discontinue, they can suggest another within their organization or network to mitigate attrition per the snowballing approach. Participants will be offered the option to also commit to Aim 3 sessions at recruitment time.

Data collection. Sessions will be conducted in English via Zoom; or if preferred, in French or by phone/in person. I do not anticipate interviewing in another language because these two cover 98.2% of Canada’s population¹⁵⁸, but I will attempt to accommodate such a request. I will ask participants to set aside one hour for the session, but will accommodate 30-minute interviews if requested for feasibility. All sessions will be audio-recorded and transcribed verbatim. Transcripts in French will be professionally translated into English for analysis. To ensure that participants feel open sharing their views, I will use the SPOR EA reflective exercise¹⁶⁵.

Interview and focus group guides. The findings and the plain language summary from Aim 1 will be offered to participants for review. Before scheduled sessions, participants can review the AWARE checklist questions and advise how they should be tailored for each intended perspective. I will develop the guides based on the findings of the scoping review and refine them in a pilot interview. The guide will explore understanding of HRW, relevance of the MINUS framework, tailoring to KU perspectives, appropriate checklist composition, content and composition of the tool report, considerations around accessibility and the printable version, privacy concerns, and any other suggestions on the tool.

Data analysis plan. I adapt a three-step approach from the Select Tool¹⁶⁶: 1) Categorize barriers and facilitators to AWARE use according to the intersectionality-enhanced¹⁶⁷ Theoretical Domains Framework (TDF)^168,169. To this end, I will analyze interview and focus group transcripts using the Framework Method¹⁷⁰ as adapted by Gale et al.¹⁴⁸. The framework method uses a distinctive matrix to chart “cases” (individual participants) in rows and “codes” (identified themes) in columns, making it useful for inter- and intra-case analysis of multi-disciplinary participants such as patient partners and clinicians/researchers¹⁴⁸. In accordance with the framework method, a second coder will be involved. Coders will discuss the codes that have arisen after preliminary coding of the first few transcripts and reaching consensus on a set of codes to proceed with for the remaining analysis, which will be maintained in a central codebook. Further revisions will be made as needed as analysis proceeds. The analysis will be integrated across interview and focus group data using dialectic triangulation, which explores the tensions between themes as well as the systemic differences in participant profiles¹⁰⁹. Discrepancies will be resolved by discussion. 2) For each TDF domain, I will use the Select Tool to identify and prioritize the corresponding theoretical Intervention Functions (i.e., “activities aimed at changing behaviour” as defined in the Behaviour Change Wheel (BCW)¹⁷¹). 3) For each prioritized theoretical intervention function, I will name, define, and specify, as recommended by Proctor et al.¹⁷², the implementation strategies, i.e., the specific software features that I deem technically feasible to implement in Aim 3. These will be used to modify the Aim 1 prototype and form the design document that will guide co-development in Aim 3. The design document will include a draft of AWARE privacy and AI policies. Member checks will be performed through communicating aggregated study findings (i.e., the design document written in accessible language) to participants¹⁴⁹. Participants will be offered acknowledgment in or co-authorship of the manuscript depending on the nature of contributions according to the International Committee of Medical Journal Editors (ICMJE) guidelines¹⁷³. Patient and public partners will be compensated in accordance with CIHR guidelines¹⁷⁴ and the SPOR EA policy¹⁷⁵.

Aim 3 (DDD Develop) – Tool co-development and pilot-testing

Overview. Together with researchers and KUs, I will lead the co-development of AWARE, working from the findings of Section 3.1 and the KU-informed design document from Section 3.2. The study will be pre-registered with OSF and all code will be provided. The study will comprise three phases:

Phase 1: I will apply my programming skills⁹⁶ to implement the design document in software code. As discussed in Section 1.6, the application will “live” entirely in the client web browser and will consist of two components: 1) a backend, which uses Aim 1 findings and the Aim 2 design document to create a decision tree that connects study attributes to HRW-related problems, solutions, citations, and supporting documentation; 2) a frontend, which renders a form-based graphical user interface to interact with the backend.

Phase 2: I will deploy the resulting application to a pilot-test environment for researcher and KU co-developers to review and discuss during the co-development sessions described below. Prior to the sessions, I will discuss the planned session format with co-developers and invite suggestions. As described below, each session will conclude with a list of suggested amendments to the Aim 2 design document, categorized as either bug fixes or feature requests. I will apply bug fixes on a rolling basis.

Phase 3: Once pilot-testing is completed, I will consolidate feature requests and circulate the list among co-developers for review as well as brief them about the results of the preliminary evaluation, described below. Amendments to the design document will be finalized by group discussion via email or Zoom and implemented prior to public release.

Co-developers, sampling, and recruitment. Participants from Aim 2 will be invited to join as co-developers. I expect to engage at least 6 individuals, assuming a 30–50% response rate. This rate is reasonable based on previous literature^176–179 and is further incentivized by offering Aim 2 participants an opportunity to commit to Aim 3 sessions at the time of recruitment (Section 3.2). Recruitment will follow the approach described in Section 3.2. To diversify the co-development team, I will recruit an additional 6–12 individuals from the SPOR EA network who were not involved in Aim 2, using the same sampling frame as in Aim 2 and aiming for at least one previous and one new co-developer per each of the six AWARE perspectives. I will specifically encourage involvement of researchers and KUs who prefer different devices and/or require accessibility accommodations to diversify the tool testing scenarios. In total, I anticipate recruiting 12–18 individuals.

Data collection. Sessions: Co-developers will join virtually or in person to use the tool synchronously, with the opportunity to participate in English or French and use a device of their choice to interact with the tool (e.g., a desktop, handheld, tablet, screen reader, or printable PDF). The session guide from Aim 2 will be modified to support think-aloud protocols, in which “participants are asked to make spoken comments as they work” with the tool^180,181. Sessions will last for 60 to 90 minutes, allotting approximately 30 minutes to testing the tool on one study. Before the start of the session, the current AWARE privacy and AI policies will be presented and discussed. I will take field notes and illustrative (anonymized) screenshots. I will particularly attend to suggested amendments to the Aim 2 design document and categorize them as either bug fixes or feature requests. Before the end of the session, I will member-check the list of suggested amendments from my notes with the co-developer. Tasks: Co-developers are asked in advance to think of studies they would like to test, including “bad,” “good,” or “unclear” examples, implementing therefore a “bring your own study” policy. If the co-developer has no preference, studies will be offered to them from the test library of positive and negative cases (n = 154, see below). I will ask co-developers to complete at least two studies per session within their preferred AWARE perspective: one with known HRW and one with lower expected HRW, adding up to at least 24 cases assessed overall, which corresponds to 15.6% of the test library or less, depending on how many of the assessed cases will be brought by co-developers. At the end of each assessment, the co-developer will be asked to mark any flags they deem irrelevant among the flags shown in the AWARE report, as well as to record any flags that they deem relevant but missing in the report. Completion: I deem further increasing the sample size to be infeasible, for which reason the remaining cases from the test library will be completed by me and a researcher co-developer (outside of the think-aloud sessions). As such, 84.4% or more assessments will be completed from a peer reviewer perspective. At least two people will independently complete each case to enable inter-rater reliability assessment. Discrepancies will be resolved by discussion. I will record any suggested amendments to the tool design that arise in this process and add them to the consolidated list for group review.

Evaluation strategy. I will assess AWARE’s internal validity and reliability as defined in the COSMIN taxonomy of measurement properties³⁰ and report using the COSMIN reporting guideline¹⁸². Because AWARE is not designed to track changes in HRW over time, I will not assess the tool responsiveness. I will assess the tool usability to evaluate its perceived helpfulness beyond measurement properties.

Internal validity assessment will employ diagnostic test accuracy (DTA) statistics because AWARE is conceptualized as a tool that either detects or fails to detect the relevance of a particular HRW study, guideline, or tool included in Aim 1 to the study uploaded by the user. My DTA strategy prioritizes negative predictive value (NPV) as the primary metric because the high pre-test probability of HRW (e.g., 87.5%⁹) makes HRW likely in any given study even without using a tool. It is thus desirable to detect studies with a lower probability of HRW as a signal of potential value for decision-makers. A higher NPV or lower false negative rate is cultivated through acquiring a diverse sample of positive cases of HRW in Aim 1 (Section 3.1), which is theorized to increase diagnostic test sensitivity¹⁸³. I will use studies that have a lower pre-test probability of HRW to acquire negative cases for the assessment of specificity; these include studies conducted within initiatives established to avoid HRW, such as SPOR EA, U.S. Patient-Centered Outcomes Research Institute (PCORI)¹⁸⁴, and U.K. Research Excellence Framework (UKREF)¹⁸⁵, because their policies often include commitment to using research guidelines and tools.

Test cases include (n ≥ 154): Subset 1 – “bring your own study” examples optionally supplied by co-developers, which may include red, yellow, or green reference flags (n ≥ 0); Subset 2 – known examples of HRW or red/yellow reference flags, sampled randomly from Aim 1 (n = 25); and Subset 3 – lower-waste examples or green reference flags (n = 129), comprising 43 randomly selected studies from each of: SPOR EA (43 of 372, 11.6%)¹⁵⁴, PCORI (43 of 2649, 1.6%)¹⁸⁴, and UKREF (43 of 6361, 0.7%)¹⁸⁵. Subset 3 will be restricted to studies published from 2015 onward so they post-date the Lancet 2014 series on HRW by at least one year (see Section 1.1). My preliminary, keyword-based analysis of Subset 3 shows that the sample is expected to contain at least 50 reviews, at least 6 trials, including at least 2 RCTs, and at least 31 studies of other designs, suggesting substantial diversity.

Reference standard. For Subset 2, the description provided in the original methodological study reporting on the guideline or tool, as abstracted in Aim 1, will be used to establish the true relevance of the corresponding red/yellow/green flag. For Subsets 1 and 3, I will use researcher and KU judgments as to which red/yellow/green flags are relevant or irrelevant for the study being assessed; these will be determined by participants in the data collection as described above. All participant judgments will be treated equally, with no judgment preferred over another.

Index test. AWARE’s inclusion or failure to include the reference red/yellow/green flag in the flag report for the given study will be interpreted as the diagnostic test result. For example, if the study being assessed was reported as redundant by one of the methodological studies included in the Aim 1 scoping review, but AWARE failed to flag it, this would count as a false negative. As another example, if AWARE flagged a given RCT as non-compliant under CONSORT, but during a think-aloud session a co-developer expressed that the flag was not warranted, this would count as a false positive.

Data analysis. Primary analysis: I will calculate sensitivity, specificity, positive and negative predictive values, and area under the receiver operating characteristic curve¹⁸³. Secondary analyses: I will disaggregate these metrics by the type of design of the study being assessed, (e.g., reviews, RCTs, qualitative studies, and other designs).

Risk of bias. The reference standard is composed of researcher and KU judgments from three distinct subsets as specified above. These judgments will be independent from one another, but the independence from the index test will vary: For Subset 2 cases, reference judgments will be derived from published studies and will therefore be masked to the AWARE test result. For Subsets 1 and 3, the judgments will not be masked to the test result because they will be produced during think-aloud sessions (i.e., the evaluation of the tool usability). This is a limitation of this design, dictated by feasibility in this pilot study. I will adapt the Quality Assessment of Diagnostic Accuracy Studies (QUADAS)-2¹⁸⁶ checklist to self-assess the study before publication, aiming to minimize the risk of bias and increase the applicability.

Sample size calculation. For the primary analysis, I target a sensitivity of 0.95 (95% confidence interval ±0.05) and specificity of 0.80 (95% confidence interval ±0.10). Using the approach by Buderer (1996)¹⁸⁷, the estimated sample size is at least 73 HRW-positive cases (i.e., studies that require a reference red/yellow/green flag) and 62 HRW-negative cases (i.e., those not requiring the given reference flag), or 135 in total. Assuming that the prevalence of HRW is two times below the average (i.e., 43.75% vs. 87.50%⁹) in initiatives focused on reducing HRW (i.e., SPOR EA, PCORI, and UKREF), 110 studies from Subset 3 need to be assessed to reach 62 negative cases. The sample size is increased to 127 to account for 15% inconclusive cases. To distribute these approximately evenly across the Subset 3 initiatives, I will randomly sample 43 studies from each (129 in total). Under this assumption, Subset 3 is expected to contribute approximately 48 positive cases. The remaining 25 positive cases will be randomly sampled from Subset 2 to reach the estimated 73 positive cases. The total sample size is therefore 154 cases. At this sample size, this pilot study has only 56.1% power to show that NPV exceeds 50% and as such is unlikely to produce a conclusive statistical estimate of real-world AWARE performance. At the real-world prevalence of 87.5% HRW, this would yield an estimated NPV of 69.6% (95% CI, 48.2% to 90.9%, propagating uncertainty from the planned sensitivity and specificity). Therefore, after a negative AWARE result, there may still be an almost 50:50 chance that the study is HRW-positive. Nevertheless, this would constitute a considerable improvement over the baseline because without AWARE, the chance that a study is not HRW is 12.5% rather than 69.6% (95% CI, 48.2% to 90.9%). Secondary analyses by the type of study design are not factored in the sample sized calculation, which aligns with the scope of this pilot study; as such, they may not be sufficiently powered.

Reliability. Inter-rater reliability will be assessed using duplicate independent evaluations during data collection, with Randolph’s free-marginal kappa targeting κ ≥0.70¹³⁶. Intra-rater reliability will not be assessed due to the infeasibility within this pilot-test study.

Usability. All AWARE users will be asked to complete the System Usability Scale (SUS)^188,189 questionnaire on the last screen of the tool (Section 1.6). SUS is commonly used for KT tool usability^190,191. Optional free-text feedback will be collected through standard online feedback forms. I will target a mean SUS score of ≥70, indicating good usability¹⁸⁸. I will review the suboptimal SUS scores in an ongoing fashion as the study proceeds and triangulate them against qualitative feedback from co-development sessions and feedback forms to identify and resolve the observed usability issues. The SUS scores will be reported descriptively for the purposive sample of researchers and KUs, for which reason no formal sample size calculation is conducted. I will disaggregate the scores by test case Subset to examine the differences in usability when applying AWARE to studies with a lower and higher pre-test HRW probability. I will use web analytics to capture user metrics for time-to-completion, flag impressions, and checklist selection frequencies and report them descriptively.

Accessibility. I will design the website layout adaptive to mobile and desktop screens, screen readers, and printable PDF documents. I will follow Web Content Accessibility Guidelines (WCAG) 2.2¹⁹² when co-developing the tool.

Knowledge mobilization plan. I adopted an iterative, multi-channel approach to knowledge mobilization and dissemination¹⁹³. Web and social media: I created a website (ResearchWaste.info) for this project to obtain feedback and provide project updates⁸⁴. A draft of this research proposal was posted for public feedback. The website is also used to collect usage analytics. As of March 2026, the ResearchWaste.info website received 807 unique visits, including 296 visits from Canada, which included feedback from patient partners. In addition, a total of 11195 impressions and 150 reactions were generated with 25 project-related posts on social media (LinkedIn). Newsletters: This project was featured on the SPOR EA newsletter and social media account, leading to interactions with five patient partners, including five virtual meetings to discuss AWARE. Findings will be communicated to SPOR EA and KT Canada members, BC SUPPORT Unit networks, and other relevant newsletters.

End-of-project KT. Project milestones will be communicated via social media and organization websites, ≥4 peer-reviewed publications; ≥3 presentations at national and international events; data will be released (anonymized where applicable). I will test and deploy the tool on the ResearchWaste.info website⁸⁴, making it freely available. A detailed usage guide will be published as a supplement to the peer-reviewed article.

Out of scope – DDD Deliver

The Deliver phase of DDD supports scalability considerations to manage “risk before launching on a large scale”⁹². As such, this phase is out of the thesis scope. I highlight measures taken to preserve the data and code for any future endeavours.

Sustainability. I will prepare and release the data produced in this research according to Findability, Accessibility, Interoperability, and Reusability (FAIR) principles¹⁹⁴ (anonymized where necessary). I will maintain the ResearchWaste.info website⁸⁴ and linked social media for a minimum of 5 years post-graduation, with potential integration into existing research infrastructure. To ensure that all datasets and software code generated throughout the project remain available online publicly, freely, and indefinitely, I will host them redundantly using two centralized services (GitHub, a major software code repository maintained by Microsoft¹⁹⁵, and OSF¹⁹⁶) and one distributed network (InterPlanetary File System¹⁹⁷). Additionally, AWARE will be available for download and fully local, offline deployment.

Commercialization. If the decision is made in the future to scale up AWARE, the version released as a result of this proposal will remain public and free. I commit to conducting a POSI (Principles of Open Scholarly Infrastructure) self-assessment¹⁹⁸ ahead of any potential commercialization of future versions.

Potential challenges and mitigation strategies

Challenge 1. A large number of references identified: I possess advanced expertise in evidence synthesis and automation tools (e.g.,¹⁹⁹), and this work is supervised by Dr. Andrea Tricco, who is a recognized expert in the field.

Challenge 2. Poor study recruitment: I will utilize the extensive network of the SPOR EA. While the SPOR EA officially ended in April 2026²⁰⁰, the team’s connections remain available. I will engage patient partners and SPOR EA leadership in recruitment (i.e., Dr. Tricco). I will also utilize our connections with the broader SPOR, for example, the connection for AWARE with the British Columbia (BC) SUPPORT (SUpport for People and Patient-Oriented Research and Trials) Unit Island Centre. This connection was established thanks to my late patient partner Dr. Lisa Ridgway, who unfortunately passed away in the fall of 2025, and was later confirmed by the Unit. While not the primary channel for recruitment, I note that connections were established for AWARE with multiple individuals, including Drs. Sharon Straus (Unity Health Toronto), David Moher (Ottawa Hospital Research Institute and the EQUATOR Network), Vincent Larivière (Université de Montréal and the UNESCO Chair in Open Science), Evan Mayo-Wilson (University of North Carolina in Chapel Hill and the Patient-Centered Outcomes Research Institute – PCORI, USA), Lisa Puchalski Ritchie (Unity Health Toronto), Larissa Shamseer (Adelaide University, Australia), Lesley Uttley (University of Sheffield, UK), Louise Olsbro Rosengaard (Herlev Hospital, Denmark), and David Tovey (Emeritus Editor-in-Chief, Cochrane, UK). To incentivize recruitment, I will offer honoraria consistent with CIHR rates¹⁷⁴, emphasize the importance of the study and the value of participant contributions, and offer flexible scheduling and modalities.

Challenge 3. Lack of engagement with the tool after dissemination: I possess advanced web development skills (e.g.,⁹⁶), will offer training and support materials, provide user feedback mechanisms, and maintain ongoing activity on social media and the project website.

Challenge 4. Development and piloting in a predominantly English-speaking, high-income environment: EDI efforts are integrated throughout, alongside a free tool and FAIR data sharing.

Challenge 5. Duplication of effort with similar initiatives conducted in parallel: I regularly communicate with other scholars studying HRW and research integrity (e.g., Drs. Rosengaard, Uttley, Culina²⁰¹, Moher, Shamseer, among others) and conduct weekly search updates using PubMed and DataCite⁷⁷, increasing the chances of detecting similar initiatives and aligning efforts early.

Challenge 6. Uptake risk based on previous HRW tools: AWARE aims to capture a comprehensive range of terminology and data sources across health research and adjacent disciplines (Aim 1), and is co-developed with KUs (Aim 3). Its usability will be evaluated and the tool refined iteratively based on feedback (Aim 3). AWARE will be free, fully transparent, and open. I receive guidance from my supervisor and committee members who have extensive experience with developing and launching similar tools, including the Right Review tool¹¹⁵, accessed by 232,000 visitors across multiple countries in regions of North America, Europe, Asia, Oceania, South America, and Africa, and the EQUATOR website⁷².