Manufacturers looking to use artificial intelligence (AI) tools to generate real-world evidence (“RWE”) on the use, safety, effectiveness or cost of health technologies as part of their health technology assessment (“HTA”) should consider Canada's Drug Agency's (the “CDA”) recently released guidance, Use of Artificial Intelligence Methods to Generate Real-World Evidence Submitted to Canada's Drug Agency for HTA (the “Guidance”). The Guidance is directed both to manufacturers generating and submitting evidence to the CDA as well as to those responsible for reviewing and assessing that evidence. It builds on the CDA’s 2025 Position Statement on the Use of Artificial Intelligence and was adapted from guidance published by the United Kingdom’s National Institute for Health and Care Excellence (“NICE”). While the CDA does not prohibit the use of AI, its overall position is cautious: AI should generally supplement, not replace, established methods.
Key Takeaways from the CDA:
- AI Should Not Replace Reliable Methods: Where explainable non-AI methods are available and can deliver robust results, those methods should remain the primary approach.
- Human Oversight is Required: Organizations using AI must maintain active and appropriately qualified human involvement throughout the evidence-generation process.
- Transparency in AI Usage: Submissions should disclose the particular AI methods used, how they were applied, and the validation methods used to assess reliability.
- Reliability Through Validation: Manufacturers should be able to demonstrate the performance and reliability of AI-enabled data extraction.
- AI-Related Risks Must be Assessed: Manufacturers should consider risk from bias, generalizations, cybersecurity concerns, transparency, reproducibility and the possible effect of errors on analytical conclusions.
- Heightened Scrutiny for Causal Inference: The use of AI is considered to be a higher-risk application that requires additional validation and sensitivity analyses.
Scope of Guidance
RWE is largely generated using real-world patient and health data collected from various sources including electronic medical records, clinical and disease registries, and administrative databases. The Guidance is particularly focused on AI applications used to extract structured variables from unstructured data, which includes patient and health data that does not follow a consistent format and is therefore more difficult to sort, search, process, analyze and share using traditional electronic systems. Examples may include free-text narratives, text-based laboratory values, descriptions of imaging or diagnostic tests, and image-based documents in electronic health records.
Justification for Using AI
Manufacturers are expected to explain why AI is appropriate for a specific task and why it should be used instead of recognized non-AI methods. AI may be used in a supplementary role where its use is clearly justified but where established and explainable methods that do not use AI can produce robust results, those methods should remain the primary approach. Where AI is applied, manufacturers should document the rationale for its use, identify key assumptions, assess the plausibility of its outputs, and ensure that the associated risks have been identified and mitigated.
Transparency and Reporting
The CDA expects that sufficient information will be provided where AI-enabled methods are used in order to ensure that reviewers can properly assess the resulting evidence. Specifically, manufacturers will need to declare if AI has been used, explain the chosen method, and report how it was applied, including the nature and extent of human input.
Depending on the technology used, submissions should address matters including:
- data preprocessing and transformation steps applied to unstructured data;
- software and versions used;
- the type, source, ownership, and version of the AI model;
- for AI models, information regarding the type, source, version and ownership of the model and, where available, a model card;
- for LLMs, details on model architecture, configuration, parameter selection and prompting strategies;
- the extraction schema and rules used, including how the system addresses issues such as negation, temporal context in clinical notes, and ambiguous terminology; and
- overview of adjudication procedures and oversight, such as dispute-resolution methods, interreviewer agreement metrics, and the qualifications and expertise of those responsible for AI-enabled data extraction and adjudication.
Human Oversight and Validation
Manufacturers remain accountable for all content in their HTA submissions. AI-assisted extraction should remain subject to active human oversight throughout the evidence-generation process. AI-enabled processes must also be validated. CDA expects manufacturers to explain how the training and test datasets were developed, who reviewed or annotated the data, and whether reviewers reached consistent results. Manufacturers should also identify any differences between the validation data and the analytical dataset that could affect the reliability or generalizability of the results.
AI performance should be evaluated using metrics appropriate to the specific task . It should also be assessed across important subpopulations, together with the potential impact of errors on the intended analytical use.
Causal Inference as Higher Risk
Where AI is used for causal inference, CDA expects that additional safeguards, such as sensitivity analyses, validation against alternative methods and triangulation with available clinical evidence, be used. Ideally, machine learning methods should also be supported by prespecified, outcome-blind simulations conducted independently.
Legal, Regulatory and Ethical Considerations
Manufacturers using AI remain responsible for determining and documenting compliance with applicable requirements. These may include federal, provincial and territorial data-protection legislation, applicable AI regulations and guidance, technical and scientific standards, cybersecurity requirements, and ethical standards.
The Guidance also emphasizes licensing and intellectual property considerations. Manufacturers should determine whether they are authorized to use copyrighted or licensed materials within a given AI tool, if further licensing is necessary, who holds the rights in AI-generated outputs, and whether those outputs may be shared with the CDA.
The CDA also notes AI-specific cybersecurity risk should be assessed and addressed through security measures.
What Does this Mean for Organizations?
Manufacturers using AI to generate RWE for submission to the CDA should consider incorporating AI-specific governance, documentation and validation measures into their evidence-generation processes from the outset. Specifically, manufacturers should be prepared to justify the use of AI rather than proven non-AI methods, document the methodologies employed and the role of human reviewers, and demonstrate the reliability of AI-enabled data extraction through appropriate validation and performance reporting.
The Guidance is framed as an iterative document that may be updated over time as AI methods and validation practices continue to develop.
