EDI v1.0.1 Exhaustive C++ Model Fit Benchmarks

Generated: 2026-09-24 09:32:23 IDT

R analog of benchmark_model_fits_python.html — same three tables, same table shape, same three-color row coding.

This report compares the performance of EDI's Rcpp-optimized model fitting paths against low-level canonical R implementations (e.g., glm.fit, lm.fit, coxph.fit) where possible.

Compilation Context

These rows are read from build metadata compiled into the loaded EDI shared object via edi_build_info_cpp().

Compilation warning: EDI model-fit timings are sensitive to the compiler flags used to build the loaded EDI.so. If EDI is compiled without the proper optimized flags, or with flags that are known to degrade these kernels such as problematic LTO builds, the benchmark can show substantial performance regressions that reflect the binary build rather than the modeling algorithms.

• EDI shared object: /home/kapelner/R/x86_64-pc-linux-gnu-library/4.7/EDI/libs/EDI.so
• EDI shared object mtime: 2026-09-24 08:09:47
• Capture method: configure-generated header compiled into EDI.so
• Build timestamp: 2026-09-20 06:41:28 IDT
• Build host: LAPTOP-J2T9TGGB
• R version at build: R Under development (unstable) (2026-04-23 r89955) -- "Unsuffered Consequences"
• R CXX20 at build: g++
• R CXX20STD at build: -std=gnu++20
• R CXX20FLAGS at build: -O3 -march=native -funroll-loops -fno-math-errno
• R SHLIB_OPENMP_CXXFLAGS at build: unavailable
• Build env at build: EDI_PORTABLE=0, EDI_DISABLE_VECTORIZATION=0, EDI_NATIVE_SPEED=1, EDI_NATIVE_LTO=0
• Package PKG_CPPFLAGS at build: ``
• Package PKG_CXXFLAGS at build: $(SHLIB_OPENMP_CXXFLAGS) -DNDEBUG -DEIGEN_NO_DEBUG -march=native -mtune=native -Wno-ignored-attributes -fno-lto; override CXXFLAGS += -O3 -g0
• Package PKG_LIBS at build: $(SHLIB_OPENMP_CXXFLAGS) $(LAPACK_LIBS) $(BLAS_LIBS) $(FLIBS) -fstack-protector
• Compiler reported by binary: 15.2.0
• Compiler optimization macro enabled: TRUE
• Compiler fast-math macro enabled: FALSE
• Eigen vectorization disabled macro enabled: FALSE

Benchmark Dataset Specification

All benchmarks were performed on a synthetic clinical-trial-scale dataset generated for each response type. The data generation process ensures numerical stability and fair solver comparison by using the following parameters:

• Sample Size ($N$): 1,000 subjects for most models; 500 subjects for survival models. Exact and trend tests may use smaller scaled samples (N=100-500) as noted in the results.
• Predictors ($p$): 5 total predictors, including a global intercept, a balanced binary treatment assignment from fixed iBCRD, and 4 continuous covariates ($X \sim \text{Normal}(0, 1)$).
• Effect Sizes: Covariate coefficients are sampled from $\text{Normal}(0, 0.5)$. The treatment coefficient is set to 0.5 in the linear predictor so the benchmarked treatment effect is meaningfully separated from zero.
• EDI Design Template: EDI benchmark objects are instantiated on a fixed iBCRD design.
• Response Generation:
  • Continuous: Linear model with additive $\text{Normal}(0, 0.5)$ noise.
  • Incidence: Binary outcomes via a Logistic link.
  • Count: Integer outcomes via Poisson or Negative Binomial distributions with an exponential link.
  • Proportion: Continuous outcomes in $(0, 1)$ via a Beta distribution with a logit link.
  • Survival: Exponentially distributed event times with approximately 20% random censoring.
  • Ordinal: 3-level categorical outcomes generated from the same ordinal construction used elsewhere in the benchmark suite.
• Stratified Cox Exception: For InferenceSurvivalStratCoxPHRegr, the benchmark injects low-cardinality covariates before outcome generation so the row exercises a genuinely stratified Cox fit rather than the unstratified fallback.

Methodology

• Bare Metal EDI Timing: EDI rows call the exported C++ functions directly (e.g., fast_logistic_regression_cpp, fast_ordinal_regression_cpp) with all design matrices and fixed inputs pre-built outside the timed region. There is no R6 object instantiation, no cached state management, no warm start storage, and no standard error computation in the timed region — only the raw numerical solver.
• Apples-to-Apples Canonical Timing: Canonical R timings likewise call the lowest-level publicly exposed interfaces (e.g., glm.fit, lm.fit, coxph.fit) with pre-built design matrices. If a canonical package exposes no low-level function, the formula-based API is used instead.
• Low-Level Comparison: Both EDI and canonical timings are measured on pre-built numeric matrices, removing formula parsing, model-frame construction, and R6/S3/S4 dispatch overhead from the timed region wherever the API permits.
• Limitation: Some canonical comparators only expose formula-based APIs. Those rows remain included but their canonical timings carry formula/model-frame overhead not present in the EDI bare-metal timing.
• Averaging: All timings are medians over 30 cold estimate-only timing samples measured with adaptive batched system.time; paths below 0.01 ms use microbenchmark(times = 5000) instead.
• Timing P-Value: Timing Pval reports a Welch two-sample t-test comparing the EDI and canonical timing replicate distributions for each row. The unlabeled final column marks thresholds with *** for p < 0.001, ** for p < 0.01, and * for p < 0.05.
• Row Highlighting: Light green rows indicate Speedup > 1 and Timing Pval < 0.05; light grey rows indicate NA timing comparisons from a failed fit; light blue rows are estimators with no canonical R implementation at all (only EDI is timed, Canonical Time/Speedup/Timing Pval are NA by design).
• Constraints: Most matched-pair/KK and highly custom paths are excluded as per user request; the exceptions are the four light-blue rows below, whose custom joint likelihood or estimator family (KK combined matched+reservoir, Weibull frailty, zero-one-inflated beta) has no canonical R implementation to compare against — see package_metadata/python_bindings_package_spec.md for the underlying baseline-gap analysis.

Results

EDI faster, significant   NA timing comparison   no canonical R implementation

Wald Test Performance (Full Inference)

This table compares the performance of Full Inference (Model Fit + Standard Error calculation + P-value derivation). Unlike the point-estimation table above, these results include the computational cost of the variance-covariance matrix (Hessian or Fisher Information) and the Wald test statistic calculation. All paths (EDI and Canonical) use a reduced sample size ($N=200$) for this full-inference benchmark to ensure iterative stability. Stratified Cox Exception: For InferenceSurvivalStratCoxPHRegr, the benchmark injects low-cardinality covariates before outcome generation so the row exercises a genuinely stratified Cox fit rather than the unstratified fallback. EDI regression models (Logistic, Poisson) are benchmarked using the IRLS optimizer for these Wald tests. Solver-Only Prebuilds: Benchmark setup prebuilds exposed observed-data design matrices, reduced design matrices, strata IDs, and other fixed working inputs outside the timed region when the implementation exposes those hooks. The timed region then measures the full-inference kernel on those fixed inputs. Limitation: Some canonical comparators only expose formula-based APIs rather than comparable low-level fit kernels. Those rows remain included, but their canonical timings may still contain formula/model-frame overhead beyond the numerical solver, variance, and p-value work itself. Timing Note: All timings are medians over 30 warmed runs measured with adaptive batched system.time; paths below 0.01 ms use microbenchmark(times = 5000) instead. Timing P-Value: Timing Pval reports a Welch two-sample t-test comparing the EDI and canonical timing replicate distributions for each row. The unlabeled final column marks thresholds with *** for p < 0.001, ** for p < 0.01, and * for p < 0.05. Row Highlighting: Light green rows indicate Speedup > 1 and Timing Pval < 0.05; light grey rows indicate NA timing comparisons from a failed fit; light blue rows are estimators with no canonical R implementation at all (only EDI is timed, Canonical Time/Speedup/Timing Pval are NA by design).

EDI faster, significant   NA timing comparison   no canonical R implementation

Garbage Collection and Cache Management

To ensure that the benchmark results are highly precise, reproducible, and represent the actual computation speed of the numerical solvers, the benchmarking harness uses the following garbage collection and cache management strategies:

1. Garbage Collection (GC) Filtering

Garbage collection cycles run automatically by the R interpreter and can introduce significant, arbitrary pauses that skew timing measurements. To isolate the execution time of the code from R's GC overhead:

• GC Disabling: We disable R's memory stress-testing mode using gctorture(FALSE) before running timing loops.
• Proactive Compaction: In the system.time() path, we invoke gc(verbose = FALSE) immediately before timing each replicate. This starts the timer on a clean, compacted heap, minimizing the likelihood of triggering an automatic garbage collection cycle mid-replicate.
• Automatic Filtering: In the microbenchmarking path, we utilize the bench::mark() engine with the filter_gc = TRUE parameter, which automatically tracks and discards timing iterations during which a garbage collection event occurred.

2. Cold-Start Guarantee for EDI and Symmetric Warm-Up for Both Sides

Both EDI and canonical timing expressions receive a single validation/warm-up call executed once before the calibration loop begins. This puts the machine code and working data into the instruction and data caches in the same warmed state for both sides, so the official timed replicates start on equal footing.

EDI timings call exported C++ functions directly — no R6 objects are instantiated during benchmarking. As a result, no R6 result caches exist to manage. Each call to the C++ solver (e.g. fast_logistic_regression_cpp, fast_ordinal_regression_cpp) starts from a freshly zero-initialized parameter vector (or a model-specific data-driven initialization when smart_cold_start = TRUE). No prior-fit results are carried across timing repetitions, so every replication is a genuine cold start for the numerical optimizer.

Utility / Math Kernel Performance

This table is an exhaustive inventory of every fast_* scalar math kernel in EDI/src — used inside the NegBin/Beta/ZINB/Hurdle likelihoods, KK21 negative-binomial fitting, probit cold-start heuristics, ordinal cloglog/cauchit link derivatives, and LRT/score-test p-values — against base R's vectorized equivalents. Unlike the model-fit tables above, these are not full estimators: each row is a single vectorized special-function evaluation (digamma, trigamma, log-gamma, log-beta, error function, normal CDF/PDF/quantile (both closed-form and log-scale), chi-squared upper-tail p-value, arctangent, softplus, and the mu-parameterized negative-binomial density) over a fixed-length vector, with no design matrix, optimizer, or R6 object involved on either side. Vector length: All rows evaluate the function over a vector of length $N=5000$; inputs are drawn from a domain realistic for each kernel's actual call sites in EDI (e.g. shape/rate-like values in $(0.5, 50)$ for digamma/trigamma/lgamma, probabilities in $(0, 1)$ for qnorm). Bare Metal EDI Timing: Eight kernels (digamma, trigamma, lgamma, lbeta, dnbinom_mu, qnorm, log_pnorm, log_dnorm) call the package's own exported fast_*_vec_cpp wrapper (EDI/src/fast_math_utils.cpp, documented and exported via roxygen/NAMESPACE like any other EDI function) directly from the installed package — no separate compile step. The remaining six (pchisq_upper, erfc, pnorm_fast, dnorm_fast, atan, log1pexp) aren't promoted to package exports yet, so those rows call a thin bare-metal wrapper compiled standalone via Rcpp::sourceCpp() against the same EDI/src leaf headers (benchmark/fast_math_utils_bench.cpp) — same convention as benchmark/fast_trigamma_speed_compare.cpp. Either way, the wrapper just loops the internal fast_* scalar kernel over the input vector — no R6, no caching, no warm starts. Canonical R Timing: The base R column calls the corresponding vectorized base/stats function directly on the same input vector (e.g. digamma(), lgamma(), stats::qnorm(), stats::dnbinom(..., mu = , log = TRUE)). Timing Note: All timings use the same adaptive batched system.time/microbenchmark harness as the tables above (medians over 30 cold samples; paths below 0.01 ms fall back to microbenchmark(times = 5000)). Timing P-Value: Timing Pval reports a Welch two-sample t-test comparing the EDI and base R timing replicate distributions for each row. The unlabeled final column marks thresholds with *** for p < 0.001, ** for p < 0.01, and * for p < 0.05. Row Highlighting: Light green rows indicate Speedup > 1 and Timing Pval < 0.05; light grey rows indicate NA timing comparisons from a failed evaluation.

EDI faster, significant   NA timing comparison   no canonical R implementation