Diagnostics Gates

For the workflow interpretation of these checks, start with Stage 4: Computation and Sampler and Stage 5: Model Adequacy. This page is the threshold and policy reference.

Use this page when you need exact gate thresholds, status aggregation, or YAML policy semantics. It does not replace substantive model review: a model can clear threshold tables and still be a poor basis for interpretation.

Model selection for time-ordered data

PSIS-LOO and WAIC treat pointwise observations as conditionally exchangeable. That assumption is not generally appropriate for time-ordered MMM data, where nearby weeks can remain dependent after conditioning on the fitted model.

Use the runner’s expanding-window blocked CV or leave-future-out CV as the primary evidence when selecting among time-series MMM specifications. Treat PSIS-LOO, WAIC, and Pareto-k outputs as supplementary fit and influence diagnostics. They do not establish future-period predictive performance, causal validity, or a publish-gate pass on their own.

Purpose

DSAMbayes runs a deterministic diagnostics framework after model fitting. Each diagnostic check produces a pass, warn, or fail status. The policy mode controls how lenient or strict the thresholds are. This page defines the check taxonomy, threshold tables, policy modes, identifiability gate, and the overall status aggregation rule.

How to use this page

• Use Stage 4: Computation and Sampler to understand which checks are non-negotiable before trusting the posterior.
• Use this page to see the exact DSAMbayes thresholds and artifact semantics.
• Use Stage 5: Model Adequacy before treating a passing diagnostics table as permission for decomposition, model comparison, or optimisation.

Policy modes

The diagnostics framework supports three policy modes, configured via diagnostics.policy_mode in YAML:

The mode is resolved by diagnostics_policy_thresholds(mode) in R/diagnostics_report.R.

Check taxonomy

Checks are organised into phases:

Each check row includes:

Design checks

Condition number thresholds by mode

P0 sampler checks (MCMC only)

Mode adjustments for sampler checks

DSAMbayes treats any Rhat above 1.01 as a failure in publish and strict modes, following the rank-normalised, folded Rhat guidance in Vehtari et al. (2021) and the Stan warnings guide. In explore mode, the fail threshold is deliberately relaxed to 1.10, while the warning threshold remains 1.01.

P1 residual checks

Mode adjustments for residual checks

In publish mode, values beyond the raw residual fail boundary remain warn. The persisted threshold field states that failure is disabled and retains the raw boundary for triage. Non-residual P0 failures can still make the overall run fail.

P1 boundary hit check

In explore mode, boundary hits cannot fail. Publish mode retains the raw fail > 0.20 boundary for triage but records values beyond it as warn. Failure is disabled for this check in publish mode. In strict mode, thresholds tighten to warn > 0.02, fail > 0.10.

P1 within-group variation check

This check applies to hierarchical models and flags groups where within-group variation is extremely low relative to between-group variation. In explore mode, the fail threshold is zero (cannot fail). Publish mode retains the raw fail < 0.05 boundary for triage but records values below it as warn; failure is disabled for this check. Strict mode applies warn < 0.15 and fail < 0.10.

Identifiability gate

The P1 pre_identifiability_baseline_media_corr check measures the maximum absolute correlation between baseline terms and media terms in the design matrix. It is configured via diagnostics.identifiability in YAML:

diagnostics:
  identifiability:
    enabled: true
    media_terms: [m_tv, m_search, m_social]
    baseline_terms: [trend, seasonality]
    baseline_regex: ["^h_", "^sin", "^cos"]
    abs_corr_warn: 0.80
    abs_corr_fail: 0.95

Term detection

• Media terms: explicitly listed in media_terms.
• Baseline terms: union of baseline_terms, generated time-component terms, and matches from baseline_regex patterns.
• Both sets are intersected with actual design-matrix columns and filtered to remove constant columns.

Thresholds by mode

Skip conditions

The identifiability gate reports skipped when:

• identifiability.enabled: false
• No configured media terms found in the design matrix
• No baseline terms detected from configured terms/regex
• All resolved baseline or media terms are constant

Overall status aggregation

The overall diagnostics status is determined by diagnostics_overall_status():

1. If any check has status == "fail" → overall status is fail.
2. If any check has status == "warn" (and none fail) → overall status is warn.
3. Otherwise → overall status is pass.

Checks with status == "skipped" do not affect the overall status.

P2 rows use the emitted post_model_selection_method, post_psis_loo, post_psis_loo_options_ignored, post_psis_loo_time_dependence, post_psis_loo_elpd, post_psis_loo_looic and post_psis_loo_pareto_k identifiers. They record supplementary predictive evidence and do not override P0 or P1 failures.

The runner also emits post_sampler_params_available when sampler parameters can be inspected and post_sampler_unavailable_for_optimise for the explicit MAP not-applicable path. These identifiers are part of the persisted report; they are not substitute MCMC diagnostics for a MAP fit.

Runner artefact output

The diagnostics framework produces:

Interpretation guidance

• pass: no enabled check breached its configured threshold. This does not establish substantive adequacy, causal validity or suitability for the intended decision.
• warn: review recommended; the model may have quality concerns but does not block the configured policy.
• fail: at least one enabled check breached a fail threshold; resolve it before using the result for decisions.

Common remediation actions

Cross-references

• Model Classes, which diagnostics apply to each class
• Response Scale Semantics, residuals are computed on model scale
• Config Schema, diagnostics.* YAML keys
• Output Artefacts, diagnostics artefact paths
• Diagnostics Plots, visual diagnostic outputs