Home

Flashiest

New features included in the flashier implementation of EBMF.

Investigation 21. Benchmarking flashier.

Investigation 27. Using low-rank approximations to the data as input to flashier.

Investigation 28. Performing backfitting updates in parallel.

Investigation 29. Accelerating backfits via extrapolation.

Under Construction

Investigation 18 and accompanying notes explore stochastic approaches to fitting FLASH objects to very large datasets.

• Notes on fitting FLASH models when \(n\) is manageable and \(p\) is very large.

• Investigation 18. I implement the idea described in the previous notes and I test it out on data from the GTEx project.

Investigation 22 was meant as an illustration of how to apply flashier to tensor data, but ended up highlighting some interesting issues that arise with missing data.

• Investigation 22. A flashier analysis of the GTEx brain subtensor.

Investigation 30. A comparison of different variance structures and initialization methods using a GWAS dataset collated by Jean Morrison.

Investigation 31. A simulation study examining the effect of non-Gaussian noise on the number of factors selected by flash.

Investigation 32. Approximating the NPMLE using ebnm.

Investigation 32. Approximating the NPMLE using ebnm.

Reference Material

Investigation 4 and accompanying notes describe a way to compute the FLASH objective directly (rather than using the indirect method implemented in flashr).

• Notes on computing the FLASH objective function. I derive an explicit expression for the KL divergence between prior and posterior.

• Notes towards an alternate algorithm for optimizing the FLASH objective, using the explicit expression derived in the previous notes.

• Investigation 4. The alternate algorithm agrees with flashr with respect to both the objective and fit obtained.

Abandoned, For Now

I’m no longer pursuing acceleration via SQUAREM/DAAREM. My reasons are detailed in these notes.

• Investigation 10. SQUAREM does poorly on FLASH backfits. DAAREM (a more recent algorithm by one of the authors of SQUAREM) does better, but offers smaller performance gains than parallelization.

flashier implemented the ability to specify the order in which factors are updated. The order makes very little difference when factor loadings are nearly orthogonal (as they usually are).

• Investigation 11. The order in which factor/loading pairs are updated (during backfitting) makes some difference, but not much.

We became particularly interested in fitting models with a known error covariance matrix when we were working with GTEx data. The approach is much less promising for the applications I’m currently interested in.

• Investigation 12. To fit a FLASH model with an arbitrary error covariance matrix, I follow up on a suggestion by Matthew Stephens.

Investigations 14 and 16-17 were some very early experiments with count data. While the first made the usual FLASH assumption that errors are Gaussian, the second and third explicitly attempted to model the entries as count or binary data. So far, the latter approaches haven’t worked terribly well, primarily due to the difficulty of choosing a good point to expand the log likelihood around.

• Investigation 14. An example of how to use nonnegative ASH priors to obtain a nonnegative matrix factorization.

• Investigation 16. Instead of directly fitting FLASH, I fit count data via a Gaussian approximation to the Poisson log likelihood…

• Investigation 17. … then I fit binary data via an approximation to the binomial log likelihood.

Historical Curiosities

An early set of notes identified key ways to reduce the memory footprint of flashr. The good ideas were implemented in flashier. Not all of the ideas were good.

The bug causing the problem described in Investigations 1-3 was fixed in version 0.1-13 of package ebnm.

• Investigation 1. The FLASH objective function can behave very erratically.

• Investigation 2. This problem only occurs when using ebnm_pn, not ebnm_ash.

• Investigation 3. The objective can continue to get worse as loadings are repeatedly updated. Nonetheless, convergence takes place (from above!).

Since flashier uses a home-grown initialization function, Investigations 5a-b and 13 are no longer relevant.

• Investigation 5a. An argument for changing the default init_fn to udv_si_svd when there is missing data and udv_svd otherwise. Based on an analysis of GTEx data.

• Investigation 5b. More evidence supporting the recommendations in Investigation 5a.

• Investigation 13. A counterargument. Results in Investigations 5a-b probably depend on the fact that \(n\) is small (\(n = 44\)). For large \(n\), setting init_fn to udv_si is best.

Investigations 6 and 7 dealt with warmstarts. ashr now uses mixsqp rather than Rmosek. ebnm now calls nlm rather than optim.

• Investigation 6. Poor optim results can produce large decreases in the objective function. We should use warmstarts when ebnm_fn = ebnm_pn.

• Investigation 7. The advantages of warmstarts are not nearly as compelling when ebnm_fn = ebnm_ash.

Investigations 8 and 9 were concerned with parallel backfitting updates, which are better covered by Investigation 28.

• Investigation 8. Parallelizing the backfitting algorithm shows promise.

• Investigation 9. An additional trick is needed to parallelize the backfitting updates performed in this MASH v FLASH GTEx analysis.

The changes tested here were implemented in version 0.6-2 of flashr.

• Investigation 15. Tests an implementation of changes to the way tau is stored, as discussed here.

The changes tested here were implemented in version 2.2-29 of ashr.

• Investigation 23. I benchmark the rewritten my_etruncnorm and my_vtruncnorm functions in package ashr against their counterparts in package truncnorm.

Investigations 19a-b, 20, and 24-26 were early experiments on large single-cell RNA datasets. I’ve since created a repository dedicated to studying applications of FLASH to scRNA data.

• Investigation 19a. An analysis of the smaller “droplet” dataset from Montoro et al.

• Investigation 19b. I redo my analysis of the “droplet” dataset, but this time I follow the authors’ preprocessing steps. Results are, I think, of much lower quality.

• Investigation 20. An analysis of the larger “pulse-seq” dataset from Montoro et al.

• Investigation 24. I propose a new approach to factorizing count data that uses adaptive shrinkage to estimate the rate matrix.

• Investigation 25. I compare three different data transformations, three approaches to handling the heteroskedacity of the log1p transformation, and two approaches to dealing with row- and column-specific scaling.

• Investigation 26. I compare FLASH fits of the “droplet” dataset in Montoro et al. using three different data transformations.