Last updated: 2026-11-04

Checks: 5 2

Knit directory: coderepo-local/

This reproducible R Markdown analysis was created with workflowr (version 1.7.2). The Checks tab describes the reproducibility checks that were applied when the results were created. The Past versions tab lists the development history.


The R Markdown is untracked by Git. To know which version of the R Markdown file created these results, you’ll want to first commit it to the Git repo. If you’re still working on the analysis, you can ignore this warning. When you’re finished, you can run wflow_publish to commit the R Markdown file and build the HTML.

The global environment had objects present when the code in the R Markdown file was run. These objects can affect the analysis in your R Markdown file in unknown ways. For reproduciblity it’s best to always run the code in an empty environment. Use wflow_publish or wflow_build to ensure that the code is always run in an empty environment.

The following objects were defined in the global environment when these results were created:

Name Class Size
alpha_summary data.frame 17.5 Kb
appendix_b_class_counts function 11.9 Kb
APPENDIX_B_FASHR_REMOTE_SHA character 152 bytes
APPENDIX_B_FASHR_VERSION character 112 bytes
assert_scalar_number function 29.8 Kb
at005 data.frame 2.9 Kb
bf_below integer 56 bytes
bf_grid data.frame 195.9 Kb
build_appendix_b_datasets function 31.5 Kb
c1_contract function 13.1 Kb
c1_fit_from_likelihood function 15.1 Kb
c1_fit_models function 10.7 Kb
c1_gaussian_likelihood function 25.8 Kb
c1_gaussian_setup function 15.7 Kb
c1_grid_long function 65.8 Kb
c1_load_display function 8.2 Kb
c1_load_report function 142.6 Kb
c1_plot_alpha function 33.5 Kb
c1_plot_grid function 21.4 Kb
c1_plot_theme function 11.4 Kb
c1_run_seed function 63.8 Kb
c1_sha function 4 Kb
c1_summary function 129 Kb
c1_validate function 163.8 Kb
c1_write function 3.5 Kb
c1j_contract function 3.8 Kb
c1j_load_display function 48.3 Kb
c1j_load_report function 145.2 Kb
c1j_select_examples function 82.4 Kb
c1j_source_paths function 4 Kb
c1j_summarize function 3.7 Kb
c1j_validate function 132.9 Kb
capture_warnings function 3.8 Kb
chosen integer 96 bytes
cumulative_lfdr_calls function 53.4 Kb
display list 41 Kb
evaluate_appendix_b_discoveries function 10.7 Kb
example_data data.frame 9.8 Kb
extract_fash_lfdr function 10 Kb
extract_fash_null_weight function 14.9 Kb
fashr_provenance function 7 Kb
fdr_curves data.frame 11.5 Kb
fit_appendix_b_models function 22.4 Kb
focused_alpha_curve function 22.3 Kb
grid_results data.frame 296.6 Kb
grid_summary data.frame 28 Kb
labels character 336 bytes
method_linetypes character 472 bytes
method_linewidths numeric 360 bytes
method_shapes integer 352 bytes
null_curves data.frame 29.2 Kb
observations data.frame 4.7 Kb
panel_ids character 680 bytes
power_curves data.frame 11.5 Kb
predictions data.frame 33 Kb
raw_zero integer 56 bytes
report list 11.6 Mb
require_appendix_b_fashr function 6.6 Kb
summarize_appendix_b_grid_fit function 11.7 Kb
tab data.frame 1.6 Kb
truth data.frame 83.4 Kb
value005 function 17.8 Kb

The command set.seed(20251109) was run prior to running the code in the R Markdown file. Setting a seed ensures that any results that rely on randomness, e.g. subsampling or permutations, are reproducible.

Great job! Recording the operating system, R version, and package versions is critical for reproducibility.

Nice! There were no cached chunks for this analysis, so you can be confident that you successfully produced the results during this run.

Great job! Using relative paths to the files within your workflowr project makes it easier to run your code on other machines.

Great! You are using Git for version control. Tracking code development and connecting the code version to the results is critical for reproducibility.

The results in this page were generated with repository version bfc8f00. See the Past versions tab to see a history of the changes made to the R Markdown and HTML files.

Note that you need to be careful to ensure that all relevant files for the analysis have been committed to Git prior to generating the results (you can use wflow_publish or wflow_git_commit). workflowr only checks the R Markdown file, but you know if there are other scripts or data files that it depends on. Below is the status of the Git repository when the results were generated:


Ignored files:
    Ignored:    .DS_Store
    Ignored:    .Rhistory
    Ignored:    .Rproj.user/
    Ignored:    Rplots.pdf
    Ignored:    analysis/.DS_Store
    Ignored:    analysis/.Rhistory
    Ignored:    code/.DS_Store
    Ignored:    code/.Rhistory
    Ignored:    code/revision_simulations/.DS_Store
    Ignored:    data/appendixB/
    Ignored:    data/dynamic_eQTL_real/
    Ignored:    data/revision_simulations/
    Ignored:    data/toy_example/
    Ignored:    output/dynamic_eQTL_real/
    Ignored:    output/pdf/
    Ignored:    output/playwright/
    Ignored:    output/revision_simulations/
    Ignored:    scripts/

Untracked files:
    Untracked:  analysis/balanced_variant_thinning.rmd
    Untracked:  analysis/enhancer_enrichment.rmd
    Untracked:  analysis/evaluation_grid_mc_sensitivity.rmd
    Untracked:  analysis/functional_testing_simulation.rmd
    Untracked:  analysis/matched_fash_linear_real_data.rmd
    Untracked:  code/revision_simulations/internal/appendix_c1_j1000/
    Untracked:  code/revision_simulations/internal/per_unit_enhancer_enrichment/
    Untracked:  code/revision_simulations/internal/per_unit_expression_correlation/compare_corshrink_versions.R
    Untracked:  code/revision_simulations/internal/per_unit_expression_correlation/diagnose_discordant_calls.R
    Untracked:  code/revision_simulations/internal/per_unit_expression_correlation/diagnose_smooth_mismatch.R
    Untracked:  code/revision_simulations/internal/per_unit_expression_correlation/extract_perunit_lfdr.R
    Untracked:  code/revision_simulations/internal/per_unit_expression_correlation/plot_call_venns.R
    Untracked:  code/revision_simulations/internal/per_unit_expression_correlation/plot_class_variograms.R
    Untracked:  code/revision_simulations/internal/per_unit_expression_correlation/plot_corshrink_result.R
    Untracked:  code/revision_simulations/internal/per_unit_expression_correlation/plot_corshrink_versions.R
    Untracked:  code/revision_simulations/internal/per_unit_expression_correlation/plot_cs_vs_diagonal.R
    Untracked:  code/revision_simulations/internal/per_unit_expression_correlation/plot_day0_day1_distribution.R
    Untracked:  code/revision_simulations/internal/per_unit_expression_correlation/plot_mechanism_summary.R
    Untracked:  code/revision_simulations/internal/per_unit_expression_correlation/plot_perunit_variogram.R
    Untracked:  code/revision_simulations/internal/per_unit_expression_correlation/plot_strober_concordance.R
    Untracked:  code/revision_simulations/internal/per_unit_expression_correlation/plot_three_models.R
    Untracked:  code/revision_simulations/internal/per_unit_expression_correlation/prepare_reference_sets.R
    Untracked:  code/revision_simulations/internal/per_unit_expression_correlation/prototype_corshrink.R
    Untracked:  code/revision_simulations/internal/per_unit_expression_correlation/recover_design_factor.R
    Untracked:  code/revision_simulations/internal/per_unit_expression_correlation/test_corshrink_reimplementation.R
    Untracked:  code/revision_simulations/internal/posterior_interval_coverage/
    Untracked:  code/revision_simulations/internal/r3_stage1_method_comparison/plot_pi0_before_after.py

Unstaged changes:
    Modified:   analysis/about.Rmd
    Modified:   analysis/appendixB.rmd
    Modified:   analysis/dynamic_eQTL_real.rmd
    Deleted:    analysis/dynamic_eQTL_real_cl.rmd
    Deleted:    analysis/dynamic_eQTL_real_full.rmd
    Modified:   analysis/index.Rmd
    Modified:   analysis/license.Rmd
    Modified:   analysis/nonlinear_dynamic_eQTL_real.Rmd
    Deleted:    analysis/nonlinear_dynamic_eQTL_real_cl.Rmd
    Deleted:    analysis/nonlinear_dynamic_eQTL_real_full.rmd
    Deleted:    analysis/revision_balanced_variant_thinning.rmd
    Deleted:    analysis/revision_combined_spiky_genotype_simulation.rmd
    Deleted:    analysis/revision_correlated_error_simulation.rmd
    Deleted:    analysis/revision_enhancer_enrichment.rmd
    Deleted:    analysis/revision_enhancer_enrichment_cl.rmd
    Deleted:    analysis/revision_functional_testing_simulation.rmd
    Deleted:    analysis/revision_genotype_level_simulation.rmd
    Deleted:    analysis/revision_internal_baseline_ld_variant_enrichment.rmd
    Deleted:    analysis/revision_internal_correlated_likelihood_sensitivity.rmd
    Deleted:    analysis/revision_internal_correlated_permutation_discoveries.rmd
    Deleted:    analysis/revision_internal_evaluation_grid_mc_sensitivity.rmd
    Deleted:    analysis/revision_internal_evaluation_grid_mc_sensitivity_middle_4_11.rmd
    Deleted:    analysis/revision_internal_fash_cl_manuscript_impact.rmd
    Deleted:    analysis/revision_internal_fash_cl_variant_enrichment_comparison.rmd
    Deleted:    analysis/revision_internal_fash_discovery_variant_count.rmd
    Deleted:    analysis/revision_internal_fash_knockoff.rmd
    Deleted:    analysis/revision_internal_fash_linear_real_data_ablation.rmd
    Deleted:    analysis/revision_internal_fash_temporal_class_enhancer_enrichment.rmd
    Deleted:    analysis/revision_internal_fashr_correlated_observation_model.rmd
    Deleted:    analysis/revision_internal_mashr_mean_z_null_correlation.rmd
    Deleted:    analysis/revision_internal_matched_fash_linear_real_data.rmd
    Deleted:    analysis/revision_internal_matched_fash_linear_real_data_cl.rmd
    Deleted:    analysis/revision_internal_observational_null_set_pilot.rmd
    Deleted:    analysis/revision_internal_one_variant_per_gene_refit.rmd
    Deleted:    analysis/revision_internal_r1_normality_assumption.rmd
    Deleted:    analysis/revision_internal_r3_full_calibration.rmd
    Deleted:    analysis/revision_internal_r3_ideal_gaussian_measurement.rmd
    Deleted:    analysis/revision_internal_r3_truth_specification_pilot.rmd
    Deleted:    analysis/revision_internal_real_genotype_simulation.rmd
    Deleted:    analysis/revision_internal_roadmap_enhancer_exploration.rmd
    Deleted:    analysis/revision_internal_small_m_clean_working_null_sensitivity.rmd
    Deleted:    analysis/revision_internal_small_m_matched_null_eb_sensitivity.rmd
    Deleted:    analysis/revision_internal_targeted_broad_calibration.rmd
    Deleted:    analysis/revision_internal_temporal_class_roadmap_enrichment.rmd
    Deleted:    analysis/revision_internal_three_likelihood_comparison.rmd
    Deleted:    analysis/revision_internal_twenty_variant_per_gene_refit.rmd
    Deleted:    analysis/revision_internal_variant_annotation_enrichment.rmd
    Deleted:    analysis/revision_internal_zero_intercept_correlation.rmd
    Deleted:    analysis/revision_ld_tagging_switch_example.rmd
    Deleted:    analysis/toy_example.rmd
    Modified:   code/figure_exports/pages/export_revision_balanced_variant_thinning.R
    Modified:   code/figure_exports/pages/export_revision_enhancer_enrichment.R
    Modified:   code/figure_exports/pages/export_revision_functional_testing_simulation.R
    Modified:   code/figure_exports/pages/export_revision_internal_evaluation_grid_mc_sensitivity.R
    Modified:   code/figure_exports/pages/export_revision_internal_matched_fash_linear_real_data.R
    Modified:   code/revision_simulations/r5_one_variant_per_gene/reporting.R

Note that any generated files, e.g. HTML, png, CSS, etc., are not included in this status report because it is ok for generated content to have uncommitted changes.


There are no past versions. Publish this analysis with wflow_publish() to start tracking its development.


Overview

This simulation evaluates dynamic eQTL screening followed by temporal classification. It uses ten Monte Carlo seeds and two truth mechanisms: broad random B-spline trajectories and compact raised-cosine trajectories. The analysis proceeds in two stages:

  1. Compare BF-adjusted FASH-IWP1, BF-adjusted FASH-linear, and direct linear and quadratic interaction tests for dynamic discovery across nominal FDR levels from 0.005 through 0.20, using all 6,362 units.
  2. At the fixed threshold \(q_{\mathrm{dyn}}=0.05\), classify the FASH-IWP1 dynamic discoveries as Early, Middle, Late, or Switch, separately for each functional, using \(q_{\mathrm{func}}=0.05\).

For donor \(i\), unit \(j\), and observed time \(t\),

\[ Y_{ijt}=\alpha_{jt}+G_{ij}\beta_j(t)+X_i^T\gamma_{jt}+\epsilon_{ijt}, \qquad \epsilon_{ijt}\sim N(0,1). \]

For each seed, one tested cis variant is sampled uniformly for each of the 6,362 tested genes, and its YRI dosage vector is retained for the same 19 donors. Each replicate has 16 time points and five standardized covariates. Per-time regressions include an intercept, genotype, and five covariates; their t-adjusted standard errors use 12 residual degrees of freedom. The dynamic, constant, and zero-effect counts are exactly 1,272/2,545/2,545, with truth classes balanced across empirical-MAF deciles.

The dynamic effects have Early/Middle/Late probabilities 0.29/0.42/0.29. Switch status is balanced within each location, giving six exact cell counts 185/184/267/267/185/184. The final trajectory, including its random genetic main effect, must satisfy the assigned location and Switch margins. Early is \(t\leq3\), Middle is \(3<t<12\), and Late is \(t\geq12\) on the dense 0.1-day grid; Switch uses threshold 0.25. The two mechanisms retain the same truth contract:

  • Broad effects are cubic B-splines with six independent standard-normal coefficients, centered over the observed times and scaled to maximum absolute deviation 2.
  • Compact effects contain one, two, or three raised-cosine peaks with half-width 1.5 and are scaled to centered RMS 0.9.

Examples of true effect curves

Figure 1 shows six examples from seed 12345: broad B-spline effects classified as Early, Middle, or Late (A-C), and transient raised-cosine effects with one, two, or three generating peaks (D-F). Each panel shows the first eligible unit in each category. Black points are estimated effects, gray bars span \(\pm 2\) reported standard errors, and orange curves show the true \(\beta_j(t)\), including the genetic main effect.

# Select one stored example per category.
example_members <- unique(truth_example_pool[, c("truth_mechanism", "target",
  "unit_index", "pair_key", "true_functional", "peak_count")])
example_members <- example_members[order(example_members$unit_index), ]
panel_labels <- c("A. Broad: Early", "B. Broad: Middle", "C. Broad: Late",
                  "D. Transient: one peak", "E. Transient: two peaks",
                  "F. Transient: three peaks")
selected_examples <- do.call(rbind, lapply(seq_along(panel_labels), function(i) {
  eligible <- if (i <= 3L) {
    example_members$truth_mechanism == "random_bspline" &
      example_members$target == c("early", "middle", "late")[[i]] &
      example_members$true_functional > 0
  } else {
    example_members$truth_mechanism == "raised_cosine" &
      !is.na(example_members$peak_count) & example_members$peak_count == i - 3L
  }
  member <- head(example_members[eligible, c("truth_mechanism", "unit_index",
    "pair_key", "peak_count")], 1L)
  stopifnot(nrow(member) == 1L)
  member$panel <- panel_labels[[i]]
  member
}))
# Match stored peak counts to the generating labels.
peak_labels <- readRDS(file.path(project_root,
  "output/revision_simulations/internal/r3_peak_stratified_power_20260918",
  "peak_labels.rds"))$records[["12345"]]$units
transient_examples <- selected_examples[4:6, ]
stopifnot(identical(transient_examples$peak_count, 1:3),
  identical(transient_examples$peak_count,
    peak_labels$peak_count[match(transient_examples$pair_key, peak_labels$pair_key)]))
example_curves <- merge(unique(truth_example_pool[, c("truth_mechanism",
  "unit_index", "pair_key", "time", "true_effect")]), selected_examples,
  by = c("truth_mechanism", "unit_index", "pair_key"), sort = FALSE)
example_observed <- merge(attr(truth_example_pool, "observed"), selected_examples,
  by = c("truth_mechanism", "unit_index", "pair_key"), sort = FALSE)
example_curves$panel <- factor(example_curves$panel, levels = panel_labels)
example_observed$panel <- factor(example_observed$panel, levels = panel_labels)
stopifnot(!anyNA(example_curves$panel), !anyNA(example_observed$panel),
  all(table(example_curves$panel) == 151L),
  all(table(example_observed$panel) == 16L))

# Plot estimated effects with +/-2 SE and orange truth.
ggplot2::ggplot(example_observed, ggplot2::aes(time, estimate)) +
  ggplot2::geom_errorbar(ggplot2::aes(ymin = estimate - 2 * se,
    ymax = estimate + 2 * se), width = 0, color = "grey55", linewidth = 0.35) +
  ggplot2::geom_point(size = 1.15, color = "black") +
  ggplot2::geom_line(data = example_curves, ggplot2::aes(time, true_effect),
    color = "#D55E00", linewidth = 0.8) +
  ggplot2::facet_wrap(~ panel, ncol = 3, scales = "free_y") +
  ggplot2::scale_x_continuous(breaks = c(0, 5, 10, 15)) +
  ggplot2::labs(x = "Time (days)", y = "Effect") +
  ggplot2::theme_bw(base_size = 11) +
  ggplot2::theme(panel.grid.minor = ggplot2::element_blank(),
    strip.text = ggplot2::element_text(size = 10),
    axis.text = ggplot2::element_text(size = 9))
Figure 1. Six representative simulated trajectories. A-C: broad B-spline effects with Early, Middle, and Late truth, respectively. D-F: transient raised-cosine effects with one, two, and three generating peaks, respectively. Each panel shows one unit: black points are estimated effects, gray bars span +/-2 reported standard errors, and the orange curve is the true effect. Panels use separate effect scales.

Figure 1. Six representative simulated trajectories. A-C: broad B-spline effects with Early, Middle, and Late truth, respectively. D-F: transient raised-cosine effects with one, two, and three generating peaks, respectively. Each panel shows one unit: black points are estimated effects, gray bars span +/-2 reported standard errors, and the orange curve is the true effect. Panels use separate effect scales.

Stage 1: dynamic discovery across alpha

FASH-IWP1 and FASH-linear share the unrestricted constant null, the one-day predictive-SD grid (zero plus 51 positive values from \(\exp(-5)\) to 1), and penalty = 10. FASH-linear uses a finite mixture of Gaussian linear slopes. Both fits use the pinned BF update, which retains the conditional alternative mixture and updates its total weight against the constant null.

The direct linear LRT tests the genotype-by-time term; the quadratic LRT jointly tests the linear and quadratic interaction terms. Their eFDR uses 100 donor-level permutations, with expression trajectories and covariates permuted together relative to genotype. The direct tests are supplied with the true \(\pi_0=5090/6362\); FASH estimates its null weight. All methods use the same simulated data. FASH receives per-time regression summaries, while the direct tests receive individual-level data.

Let \(D_m(\alpha)\) be the discoveries for method \(m\). FASH uses cumulative mean lfdr, and direct tests use monotone permutation eFDR q-values. For each replicate,

\[ \mathrm{Power}_m(\alpha)=\frac{|D_m(\alpha)\cap H_1|}{|H_1|}, \qquad \mathrm{FDP}_m(\alpha)= \frac{|D_m(\alpha)\cap H_0|}{\max\{|D_m(\alpha)|,1\}}. \]

The curves show the ten-seed means of power and FDP (empirical FDR). Ribbons are pointwise replicate minima and maxima. Each panel is one truth mechanism; the vertical line marks \(\alpha=0.05\), and the FDR diagonal marks nominal calibration.

# FASH: rank all-unit lfdr values and select by their cumulative mean.
lfdr_order <- order(lfdr, unit_index)
fdr <- cumsum(lfdr[lfdr_order]) / seq_along(lfdr_order)
fash_calls <- lfdr_order[fdr <= alpha]

# Direct LRTs: apply the same donor permutations to both interaction degrees.
permutation_null <- compute_direct_interaction_permutation_null(
  out = individual_level_data,
  n_permutations = 100L,
  interaction_degrees = c(1L, 2L),
  seed = seed + 10000L,
  permute_covariates_with_expression = TRUE,
  num_cores = num_cores
)
direct_qvalues <- empirical_fdr_qvalues(
  observed_pvalues = observed_lrt$pvalue,
  null_pvalues = as.vector(permutation_null$null_pvalues[[degree_label]]),
  pi0_method = "fixed",
  fixed_pi0 = 5090 / 6362
)$qvalue
direct_calls <- which(direct_qvalues <= alpha)

Power as a function of alpha

# Select the four main-comparison methods.
power_curves <- stage1_summary[
  stage1_summary$method %in% report$contract$display_methods,
  c("truth_mechanism", "method", "alpha", "mean_power", "power_min", "power_max")
]
power_curves <- power_curves[order(
  power_curves$truth_mechanism,
  match(power_curves$method, report$contract$display_methods),
  power_curves$alpha
), ]
r3c_plot_stage1(power_curves, metric = "power")
Figure 2. Dynamic discovery power for BF-adjusted FASH-IWP1, BF-adjusted FASH-linear, and direct linear and quadratic interaction tests. Panels show broad random B-spline and compact raised-cosine truth; curves are ten-seed means and ribbons are pointwise replicate minima and maxima. Direct-test eFDR uses the true null proportion and 100 donor permutations.

Figure 2. Dynamic discovery power for BF-adjusted FASH-IWP1, BF-adjusted FASH-linear, and direct linear and quadratic interaction tests. Panels show broad random B-spline and compact raised-cosine truth; curves are ten-seed means and ribbons are pointwise replicate minima and maxima. Direct-test eFDR uses the true null proportion and 100 donor permutations.

Compact-truth power by number of peaks

We split the compact-truth power comparison by the number of generating raised-cosine peaks. For each seed and group, power is the fraction of that group’s true dynamic units selected using the original global method scores and thresholds. The three strata exhaust the dynamic units; curves average these within-group powers across the same ten seeds. Each replicate contains 426, 424, and 422 true dynamic units with one, two, and three peaks, respectively.

# Each stratum's denominator is all true dynamic units with that peak count.
peak_curves <- peak_power$summary[
  peak_power$summary$method %in% report$contract$display_methods,
  c("peak_count", "method", "alpha", "mean_power", "power_min", "power_max")
]
method_labels <- setNames(c("FASH-IWP1 (BF)", "FASH-linear (BF)",
  "Direct linear", "Direct quadratic"), report$contract$display_methods)
peak_curves$method_label <- factor(method_labels[peak_curves$method],
  levels = unname(method_labels))
peak_curves$peak_group <- factor(peak_curves$peak_count, levels = 1:3,
  labels = c("1 peak\n426 units per seed", "2 peaks\n424 units per seed",
             "3 peaks\n422 units per seed"))
stopifnot(nrow(peak_curves) == 480L, !anyNA(peak_curves$method_label),
          !anyNA(peak_curves$peak_group))
method_colors <- setNames(c("#0072B2", "#D55E00", "#009E73", "#CC79A7"),
  unname(method_labels))
method_lines <- setNames(c("solid", "longdash", "dotted", "dotdash"),
  unname(method_labels))
ggplot2::ggplot(peak_curves, ggplot2::aes(alpha, mean_power,
    color = method_label, fill = method_label, linetype = method_label)) +
  ggplot2::geom_ribbon(ggplot2::aes(ymin = power_min, ymax = power_max),
    alpha = 0.09, color = NA, show.legend = FALSE) +
  ggplot2::geom_line(linewidth = 0.8) +
  ggplot2::geom_vline(xintercept = 0.05, color = "grey50",
    linetype = "dashed", linewidth = 0.4) +
  ggplot2::facet_wrap(~peak_group, nrow = 1) +
  ggplot2::scale_color_manual(values = method_colors, drop = FALSE) +
  ggplot2::scale_fill_manual(values = method_colors, drop = FALSE) +
  ggplot2::scale_linetype_manual(values = method_lines, drop = FALSE) +
  ggplot2::scale_x_continuous(breaks = c(0.005, 0.05, 0.10, 0.15, 0.20)) +
  ggplot2::coord_cartesian(ylim = c(0, 1)) +
  ggplot2::labs(x = "Nominal Stage-1 FDR level", y = "Mean within-group dynamic power",
    color = NULL, fill = NULL, linetype = NULL) +
  ggplot2::theme_bw(base_size = 12) +
  ggplot2::theme(legend.position = "bottom", panel.grid.minor = ggplot2::element_blank(),
    strip.background = ggplot2::element_rect(fill = "grey95")) +
  ggplot2::guides(color = ggplot2::guide_legend(nrow = 2, byrow = TRUE))
Figure 3. Dynamic discovery power under compact raised-cosine truth, stratified by one, two, or three generating peaks. All four methods retain their original global discovery thresholds. Each curve is the ten-seed mean of within-group power; ribbons show pointwise replicate minima and maxima. The vertical line marks alpha = 0.05.

Figure 3. Dynamic discovery power under compact raised-cosine truth, stratified by one, two, or three generating peaks. All four methods retain their original global discovery thresholds. Each curve is the ten-seed mean of within-group power; ribbons show pointwise replicate minima and maxima. The vertical line marks alpha = 0.05.

At \(\alpha=0.05\), direct quadratic minus BF-adjusted FASH-IWP1 mean power is +17.0, +6.3, and -13.6 percentage points for one, two, and three peaks, respectively. Figure 5 reports the corresponding overall empirical FDR comparison.

Power by temporal category under both truth mechanisms

We split both the broad and compact truth comparisons by true Early, Middle, and Late category, with 369, 534, and 369 dynamic units per seed under each mechanism. Each group includes both Switch and non-Switch units. Power uses all true dynamic units in that category as its denominator and the original global discovery thresholds. This summarizes Stage-1 discovery within true temporal categories; Stage-2 conditional classification is evaluated separately below.

# Each denominator is all true dynamic units with a positive temporal functional.
location_curves <- location_power$summary[
  location_power$summary$method %in% report$contract$display_methods,
  c("truth_mechanism", "target", "method", "alpha", "mean_power", "power_min", "power_max")
]
method_labels <- setNames(c("FASH-IWP1 (BF)", "FASH-linear (BF)",
  "Direct linear", "Direct quadratic"), report$contract$display_methods)
location_curves$method_label <- factor(method_labels[location_curves$method],
  levels = unname(method_labels))
location_curves$location_group <- factor(location_curves$target,
  levels = c("early", "middle", "late"),
  labels = c("Early\n369 units per seed", "Middle\n534 units per seed",
             "Late\n369 units per seed"))
location_curves$mechanism <- factor(location_curves$truth_mechanism,
  levels = c("random_bspline", "raised_cosine"),
  labels = c("Broad random B-spline", "Compact raised cosine"))
stopifnot(nrow(location_curves) == 960L, !anyNA(location_curves$mechanism), !anyNA(location_curves$method_label),
          !anyNA(location_curves$location_group))
method_colors <- setNames(c("#0072B2", "#D55E00", "#009E73", "#CC79A7"),
  unname(method_labels))
method_lines <- setNames(c("solid", "longdash", "dotted", "dotdash"),
  unname(method_labels))
ggplot2::ggplot(location_curves, ggplot2::aes(alpha, mean_power,
    color = method_label, fill = method_label, linetype = method_label)) +
  ggplot2::geom_ribbon(ggplot2::aes(ymin = power_min, ymax = power_max),
    alpha = 0.09, color = NA, show.legend = FALSE) +
  ggplot2::geom_line(linewidth = 0.8) +
  ggplot2::geom_vline(xintercept = 0.05, color = "grey50",
    linetype = "dashed", linewidth = 0.4) +
  ggplot2::facet_grid(mechanism ~ location_group) +
  ggplot2::scale_color_manual(values = method_colors, drop = FALSE) +
  ggplot2::scale_fill_manual(values = method_colors, drop = FALSE) +
  ggplot2::scale_linetype_manual(values = method_lines, drop = FALSE) +
  ggplot2::scale_x_continuous(breaks = c(0.005, 0.05, 0.10, 0.15, 0.20)) +
  ggplot2::coord_cartesian(ylim = c(0, 1)) +
  ggplot2::labs(x = "Nominal Stage-1 FDR level", y = "Mean within-group dynamic power",
    color = NULL, fill = NULL, linetype = NULL) +
  ggplot2::theme_bw(base_size = 12) +
  ggplot2::theme(legend.position = "bottom", panel.grid.minor = ggplot2::element_blank(),
    strip.background = ggplot2::element_rect(fill = "grey95")) +
  ggplot2::guides(color = ggplot2::guide_legend(nrow = 2, byrow = TRUE))
Figure 4. Dynamic discovery power stratified by true Early, Middle, and Late category. Rows show broad random B-spline (top) and compact raised-cosine (bottom) truth. Each stratum includes both Switch and non-Switch units. Curves are ten-seed mean within-category powers using the original global discovery thresholds; ribbons show pointwise replicate minima and maxima. The vertical line marks alpha = 0.05.

Figure 4. Dynamic discovery power stratified by true Early, Middle, and Late category. Rows show broad random B-spline (top) and compact raised-cosine (bottom) truth. Each stratum includes both Switch and non-Switch units. Curves are ten-seed mean within-category powers using the original global discovery thresholds; ribbons show pointwise replicate minima and maxima. The vertical line marks alpha = 0.05.

Empirical FDR as a function of alpha

# Use the same truth, method order, and alpha grid as the power comparison.
fdr_curves <- stage1_summary[
  stage1_summary$method %in% report$contract$display_methods,
  c("truth_mechanism", "method", "alpha", "mean_empirical_fdr",
    "empirical_fdr_min", "empirical_fdr_max")
]
fdr_curves <- fdr_curves[order(
  fdr_curves$truth_mechanism,
  match(fdr_curves$method, report$contract$display_methods),
  fdr_curves$alpha
), ]
r3c_plot_stage1(fdr_curves, metric = "fdr")
Figure 5. Empirical dynamic FDR for the same four methods and paired datasets. Curves average replicate-specific false-discovery proportions across ten seeds; ribbons are pointwise replicate minima and maxima. The diagonal marks nominal FDR, and the vertical line marks alpha = 0.05.

Figure 5. Empirical dynamic FDR for the same four methods and paired datasets. Curves average replicate-specific false-discovery proportions across ten seeds; ribbons are pointwise replicate minima and maxima. The diagonal marks nominal FDR, and the vertical line marks alpha = 0.05.

At \(\alpha=0.05\), mean power for BF-adjusted FASH-IWP1 and direct quadratic is 0.887 and 0.850 under broad truth, and 0.401 and 0.434 under compact truth. Under compact truth, their mean empirical FDR values are 0.005 and 0.053, respectively; power should therefore be read together with empirical FDR.

Stage 2: conditional functional classification at 0.05

For the remainder of the analysis, \(D(0.05)\) denotes the BF-adjusted FASH-IWP1 discoveries at 0.05. This candidate set remains fixed for Stage 2. The fitted prior remains the full-data prior. For each candidate and each functional, 3,000 posterior draws are evaluated on the dense time grid. The candidate lfsr values are then ranked separately for Early, Middle, Late, and Switch, and calls are made while their cumulative FSR is at most 0.05. The four classifications are evaluated separately and need not be mutually exclusive.

For target \(k\), let \(S_k\subseteq D(0.05)\) be its final calls and let \(T_k\{\beta_j\}>0\) denote true membership. We report

\[ \mathrm{FSR}_k= \frac{\sum_{j\in S_k}\mathbf{1}[T_k\{\beta_j\}\leq0]} {\max(|S_k|,1)}, \]

and conditional classification power

\[ \mathrm{Power}_{k\mid\mathrm{Stage\ 1}}= \frac{\sum_{j\in S_k}\mathbf{1}[T_k\{\beta_j\}>0]} {\sum_{j\in D(0.05)}\mathbf{1}[T_k\{\beta_j\}>0]}. \]

The tables average each replicate-specific rate and call count across the ten seeds.

dynamic_candidates <- r3ts_stage1(
  fdr_table,
  n_units = 6362L,
  q = 0.05
)
lfsr_by_target <- compute_functional_lfsr(
  fit = fit_iwp1_bf,
  functionals = temporal_functionals,
  indices = dynamic_candidates,
  smooth_var = seq(0, 15, by = 0.1),
  num_cores = num_cores,
  seed = functional_posterior_seed
)
stage2_calls <- r3ts_stage2(
  dynamic_candidates,
  lfsr_by_target,
  n_units = 6362L,
  q = 0.05
)
# Select the truth mechanism and retain only the three displayed estimands.
broad_table <- stage2_summary[
  stage2_summary$truth_mechanism == "random_bspline",
  c("truth_mechanism", "target", "mean_empirical_fsr",
    "mean_conditional_classification_power", "mean_calls")
]
broad_table <- broad_table[
  match(c("early", "middle", "late", "switch"), broad_table$target),
]
r3ts10_show_table(
  r3ts10_format_stage2_table(broad_table, "random_bspline"),
  paste(
    "Table 1. Ten-seed mean Stage-2 results under broad random B-spline truth.",
    "Classification power is conditional on true target members retained by",
    "Stage 1 at q = 0.05."
  )
)
Table 1. Ten-seed mean Stage-2 results under broad random B-spline truth. Classification power is conditional on true target members retained by Stage 1 at q = 0.05.
Metric Early Middle Late Switch
Empirical FSR 0.001 0.037 0.009 0.034
Conditional classification power 0.285 0.914 0.294 0.919
Mean number of calls 86.6 492.6 90.5 575.4
# Select the truth mechanism and retain only the three displayed estimands.
compact_table <- stage2_summary[
  stage2_summary$truth_mechanism == "raised_cosine",
  c("truth_mechanism", "target", "mean_empirical_fsr",
    "mean_conditional_classification_power", "mean_calls")
]
compact_table <- compact_table[
  match(c("early", "middle", "late", "switch"), compact_table$target),
]
r3ts10_show_table(
  r3ts10_format_stage2_table(compact_table, "raised_cosine"),
  paste(
    "Table 2. Ten-seed mean Stage-2 results under compact raised-cosine truth.",
    "Classification power is conditional on true target members retained by",
    "Stage 1 at q = 0.05."
  )
)
Table 2. Ten-seed mean Stage-2 results under compact raised-cosine truth. Classification power is conditional on true target members retained by Stage 1 at q = 0.05.
Metric Early Middle Late Switch
Empirical FSR 0.013 0.037 0.005 0.034
Conditional classification power 0.083 0.198 0.085 0.824
Mean number of calls 12.4 43.5 12.8 332.9

Stage-1 screening and Stage-2 classification answer distinct questions. The curves quantify which true dynamic effects reach the candidate set; the tables then quantify functional classification among true target members in that set.

Reproducibility

The simulation scripts are in code/revision_simulations/internal/r3_two_stage_functional_testing_pilot10/ (two-stage functional classification) and code/revision_simulations/internal/r3_stage1_method_comparison/ (Stage-1 method comparison), using fashr 0.1.43. To rebuild this page:

Rscript --vanilla -e \
  'workflowr::wflow_build("analysis/functional_testing_simulation.rmd")'

sessionInfo()
R version 4.5.1 (2025-06-13)
Platform: aarch64-apple-darwin20
Running under: macOS Sequoia 15.6.1

Matrix products: default
BLAS:   /System/Library/Frameworks/Accelerate.framework/Versions/A/Frameworks/vecLib.framework/Versions/A/libBLAS.dylib 
LAPACK: /Library/Frameworks/R.framework/Versions/4.5-arm64/Resources/lib/libRlapack.dylib;  LAPACK version 3.12.1

locale:
[1] en_CA/en_CA/en_CA/C/en_CA/en_CA

time zone: America/Chicago
tzcode source: internal

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base     

other attached packages:
[1] ggplot2_4.0.3

loaded via a namespace (and not attached):
 [1] gtable_0.3.6       jsonlite_2.0.0     dplyr_1.1.4        compiler_4.5.1    
 [5] promises_1.5.0     tidyselect_1.2.1   Rcpp_1.1.1-1.1     stringr_1.6.0     
 [9] git2r_0.36.2       dichromat_2.0-0.1  later_1.4.4        jquerylib_0.1.4   
[13] scales_1.4.0       yaml_2.3.12        fastmap_1.2.0      R6_2.6.1          
[17] labeling_0.4.3     generics_0.1.4     workflowr_1.7.2    knitr_1.50        
[21] tibble_3.3.0       rprojroot_2.1.1    bslib_0.9.0        pillar_1.11.1     
[25] RColorBrewer_1.1-3 rlang_1.2.0        cachem_1.1.0       stringi_1.8.9     
[29] httpuv_1.6.16      xfun_0.55          S7_0.2.2           fs_1.6.6          
[33] sass_0.4.10        otel_0.2.0         cli_3.6.6          withr_3.0.3       
[37] magrittr_2.0.5     digest_0.6.39      grid_4.5.1         rstudioapi_0.17.1 
[41] lifecycle_1.0.5    vctrs_0.7.3        evaluate_1.0.5     glue_1.8.1        
[45] farver_2.1.2       whisker_0.4.1      rmarkdown_2.30     tools_4.5.1       
[49] pkgconfig_2.0.3    htmltools_0.5.9