In drug development, trust in the models utilized is critical. Even when a model demonstrates acceptable predictive performance, certain recurring issues can significantly reduce reviewer confidence. In regulatory settings and as mentioned in previous posts (see the first and second installment of this series), trust is not built on fit alone—it depends on whether the entire modeling process is transparent, justified, and reproducible.

In this blog post, we’ll share six issues that are among the most common sources of regulatory skepticism.

1. Poor traceability or undocumented data transformations

When reviewers cannot clearly trace how the analysis dataset was derived from source data, confidence in the entire analysis is undermined. Missing documentation for data cleaning, filtering, or variable derivation creates ambiguity and prevents independent evaluation. Even minor inconsistencies, such as unexplained exclusions or time variable misalignment, can trigger extensive follow-up questions and delay review.

2. Lack of transparency in handling BLQ data, outliers, or missing values

Ad hoc or poorly justified approaches to data issues are a frequent source of regulatory concern. For example, excluding outliers without justification, applying inappropriate BLQ methods, or ignoring missing data mechanisms can introduce bias. Reviewers expect clear documentation of these decisions, along with sensitivity analyses demonstrating that conclusions are not driven by arbitrary choices.

Something to remember, most of these issues do not stem from what would appear to be “incorrect” modeling techniques, but from insufficient transparency and justification. In practice, reviewers may lose trust in models not because they may be wrong, but because the reviewer cannot confidently determine whether they are right.

3. Inadequate sensitivity analyses for key assumptions

High-impact assumptions, such as handling of BLQ data, residual error structures, extrapolation beyond the available data, or included mechanisms of action must be stress-tested. Failure to explore how alternative assumptions affect conclusions raises concerns about robustness. Sensitivity analyses are particularly important for models informing dosing or labeling decisions, where small changes can have meaningful clinical implications.

4. Unjustified model structure or overly complex models without added value

Models that are overly complex without clear improvement in predictive performance or interpretability raise concerns about overfitting and lack of parsimony. Similarly, structural assumptions (e.g., compartmental models, mechanistic components) must be justified based on prior knowledge or biological rationale. Without this, reviewers may question whether the model reflects true system behavior or is simply tailored to the dataset.

5. Inconsistent or incomplete diagnostics

Providing standard diagnostics (e.g., goodness-of-fit plots, Visual Predictive Checks, VPCs) is necessary but not sufficient. Trust is reduced when diagnostics are incomplete, poorly labeled, or inconsistent with reported conclusions. For example, claiming adequate model performance while visual predictive checks show systematic bias will immediately raise concerns. Diagnostics must not only be presented but also interpreted in a way that aligns with the overall narrative.

6. Over-reliance on statistical significance without clinical interpretation

Statistical criteria (e.g., p-values, likelihood ratio tests) should not be the sole basis for model decisions or interpretation of model results. As an example, covariates might be included based on statistical criteria. However, regulatory reviewers will expect covariate effects to be clinically meaningful and biologically plausible. Models that include statistically significant but clinically irrelevant covariates risk being viewed as unstable or non-generalizable, especially in extrapolation scenarios.

It is critical to understand regulatory perspectives and build your development strategy to avoid these and other potential issues around model trustworthiness. Schedule a 15-minute call with one of our regulatory experts to learn how we can support your program.

Schedule a Call