Moving Towards Good Linguistic Practices in AI-Assisted Regulatory Communications

As artificial intelligence (AI) continues transforming regulatory workflows, its adoption in compliance-driven environments is becoming inevitable. The FDA Copilot, powered by large language models (LLMs) like GPT-4 and enhanced by Retrieval-Augmented Generation (RAG), may significantly change how regulatory professionals interact with data, draft reports, and conduct pre-submission reviews. While this evolution can improve efficiency, it raises concerns about reliability, bias, and consistency. These challenges stem mainly from the probabilistic nature of LLMs, which can lead to generative reply bias, misinformation, and out-of-context information.

To address these concerns, professionals, regulatory agencies, and the industry should implement a structured approach that ensures AI-generated insights align with verified prior knowledge, regulations, and current regulatory expectations. This requires standardized frameworks for language usage and processing, prompt engineering, and output validation in AI-assisted regulatory decision-making.

In earlier articles from Pharma Now, we examined the practical use of an AI/ML-driven FDA Copilot integrated with SmartSearch+ within a Microsoft ecosystem. Case studies showcased AI's capability to streamline regulatory tasks, such as responses to FDA Warning Letters and Quality Overall Summary (QOS) evaluations for new drug applications (NDAs). Nevertheless, crucial questions remain broadly and significantly in the regulatory context of the evidence required.:

• Can AI-generated content reliably mitigate the risks of inaccuracies, misinformation, disinformation, and false claims more effectively than traditional methods? 
• If so, can it be processed more efficiently while ensuring verifiability and compliance?

We seek to establish a structured validation mechanism to address these questions and effectively produce trustworthy, beneficial AI-generated content. This article outlines what we expect to evolve as Good Linguistic Practices (GLP) for AI-assisted regulatory communication, ensuring reliable prompt intent and “Text-Context Integrity” for factual accuracy and transparency. Specifically, we advocate for disciplined RAG prompting, structured verification using SmartSearch+, and an iterative AI-human validation approach known as the "Dueling Banjo" methodology. By developing and integrating these practices, we aim to enhance AI-generated communications' precision, clarity, and regulatory compliance, ensuring that AI is a trusted tool rather than a potential liability.

Background

This report examines how commercial LLMs such as ChatGPT—trained on vast but often inaccessible or non-transparent [to the user] datasets—can be optimized for regulatory use, improving automation, quality management, and compliance communication. We utilize AI/ML tools like the FDA Copilot, hosted within, for instance, a Microsoft OpenAI ChatGPT-4, and a platform designed to function within a broader ecosystem, as illustrated in Figure 1. Note that this figure is AI-generated and contains some misspellings and inaccuracies illustrating AI/ML inherent bias, yet we included it to emphasize the central point of this article. 

Figure 1. AI/ML-driven FDA Copilot and SmartSearch+ in the Microsoft ecosystem; an AI-generated image 

A previous report from Pharma Now analyzed the performance of AI-assisted extractive summarization in Section P2: Pharmaceutical Development from the A-MAB Case Study (2009). The evaluation focused on whether the FDA Copilot accurately extracted key regulatory insights from this Common Technical Document (CTD) section. We noted several advantages along with some challenges. 

We employed a structured process to assess AI-generated regulatory content's reliability and regulatory compliance. Figure 2 illustrates an internal corporate review process for making "ready to file" Go/No-Go decisions. This process employs the FDA AI/ML Copilot, hosted within the Microsoft OpenAI ChatGPT-4 platform. It outlines six color-coded steps for conducting an internal corporate review in the context of a Go/No-Go decision-making process regarding the readiness of an NDA, ANDA, or BLA for submission. 

Figure 2. An illustration of an internal (corporate) review process on “ready to file,” Go/No-go decision-making, utilizing AI/ML tools as the FDA Copilot hosted within a Microsoft OpenAI ChatGPT-4 platform generates an initial draft response, followed by RAG-Prompting grounded with SmartSearch+ to implement the “Dueling Banjo” methodology. The improvement feedback loop is to improve the review process and also the QbD product development.   

Framework for AI-Assisted Regulatory Review

In mature pharmaceutical organizations, well-defined policies, procedures, and cross-functional agreements form the foundation of Quality by Design (QbD) principles in regulatory applications. However, such structured and functioning maturity remains an exception rather than the norm. This underscores the necessity for a transparent, structured review strategy, especially when incorporating AI/ML tools such as FDA Copilot.

The FDA’s draft guidance on AI-assisted regulatory decision-making recommends a risk-based approach. It emphasizes defining the “question of interest” and the “context of use” for AI models, assessing inherent risks, and implementing structured validation plans to establish credibility. We align with this guidance by advocating for RAG-prompting based on the Question-Based Review (QBR) approach and an iterative verification process using the Dueling Banjo methodology. Advanced reasoning in LLMs, such as GPT-4.0, significantly improves output quality. Human reviewers can employ RAG-prompting and consider counterfactual prompting to refine the reasoning process. Table 1 summarizes the key steps and associated process:

Steps to Implement Dueling Banjo Methodology

Internal Review Strategy: Define RAG-prompting strategies and verification processes.

Initial AI-Generated Draft: Generate an initial draft without RAG; subsequent drafts use disciplined RAG prompting.

SmartSearch+ Verification: RAG-prompting is refined using SmartSearch+ to bring awareness and context to the text of regulatory sources.

Dueling Banjo Methodology: AI outputs undergo iterative human validation and enhancement.

SmartSearch+ Verification of RAG-Prompts

AI models produce the best results when prompts are clear, structured, and contextually relevant. Ambiguous prompts often lead to incomplete, inaccurate, or misleading responses, challenging regulatory compliance.

To ensure AI-generated content meets compliance standards, we propose Question-Based Review (QBR) as a structured prompting strategy:

• Define key regulatory questions to align AI outputs with compliance expectations.
• Ensure regulatory alignment by integrating domain-specific terminology.
• Clarify precision requirements to reduce misinterpretation risks.

While LLMs like ChatGPT-4 are optimal for rapid summarization of massive data and text generation, they are prone to hallucinations, context misinterpretations, and reliance on static knowledge bases. RAG addresses these limitations by dynamically retrieving verified, real-time regulatory data before generating responses. SmartSearch+ further enhances this process by providing:

• Full-text search across FDA guidance and global regulatory documents, 
• Precision retrieval for AI-assisted review, and 
• Confidential corporate knowledge indexing, ensuring compliance-driven AI accuracy.

The Dueling Banjo Methodology: Iterative AI-Human Validation

The Dueling Banjo Methodology ensures that AI-generated regulatory content undergoes iterative human validation, significantly improving accuracy, reliability, and compliance assurance. This approach consists of:

• Initial AI Draft Generation: Structured RAG-prompting guides AI in producing regulatory responses.
• Expert Review & Refinement: Human reviewers identify errors, inconsistencies, and gaps.
• Iterative AI Enhancement: AI refines outputs based on expert feedback and regulatory references.
• Final Compliance Verification: The cycle continues until the output meets quality, compliance, and traceability standards.

Effective oversight depends on qualified human reviewers with expertise in AI literacy, structured prompt engineering, and validation techniques. Training and continuous development are integral to generating and ensuring that human oversight is an active safeguard rather than a passive checkpoint. Organizations should develop these practices within their corporate policies and procedures to institutionalize them. The essential policy considerations are outlined in Table 2.

Essential Policy Considerations 

Objective Definition

Clearly define the goal of utilizing AI/ML tools to ensure that the outputs are reliable, reproducible, and linguistically accurate.

Identify Processes for AI Assistance

Determine which processes and documents would benefit from AI assistance, such as drafting warning letters, creating quality summaries, and preparing regulatory submissions.

Specify Regulatory Frameworks

Outline the relevant regulatory frameworks and guidelines, such as the FDA's draft guidance on the use of AI.

Define Key Terms

Provide clear definitions for essential terms such as "AI-assisted outputs," "hallucinations," "prompting," "chain-of-thought," and "Dueling Banjo methodology." Additionally, categorise risks based on the criticality of regulatory documents (e.g., high-risk decisions versus routine communications) and establish thresholds for acceptable error rates. Outline procedures for addressing hallucinations or inconsistencies when they occur.

Establish Audits and Evaluations

Implement routine audits and performance evaluations using predetermined metrics such as accuracy, consistency, and reproducibility. Regularly test the AI system with known case scenarios to identify performance drift or degradation.

Incident Reporting Mechanisms

Develop protocols for logging and reporting AI errors (e.g., hallucinations or misinterpretations), including a clear escalation path. Create feedback loops with internal teams and AI vendors to ensure that lessons learned are translated into system improvements.

Conclusion

Integrating SmartSearch+ with RAG prompting allows organizations to establish a centralized, validated repository of AI-generated regulatory insights, ensuring precision, consistency, and compliance in AI-assisted workflows. This framework ensures that AI is a trusted compliance tool rather than a liability by standardizing regulatory terminology, minimizing ambiguity, and enhancing traceability.

SmartSearch+ verification of RAG prompts will be crucial in maintaining regulatory integrity and trust as AI adoption expands. Implementing structured AI validation methodologies, supported by expert oversight and Good Linguistic Practices (GLP), will drive a more reliable, transparent, and future-ready regulatory ecosystem.

Key Takeaway: AI-assisted regulatory workflows must prioritize structured validation, human oversight, and regulatory compliance to ensure trust, accountability, and accuracy in pharmaceutical decision-making.