Evaluation of performances

Last updated: 2020-01-10

Checks: 7 0

Knit directory: 2019-feature-selection/

This reproducible R Markdown analysis was created with workflowr (version 1.6.0). The Checks tab describes the reproducibility checks that were applied when the results were created. The Past versions tab lists the development history.

R Markdown file: up-to-date

Great! Since the R Markdown file has been committed to the Git repository, you know the exact version of the code that produced these results.

Environment: empty

Great job! The global environment was empty. Objects defined in the global environment can affect the analysis in your R Markdown file in unknown ways. For reproduciblity it’s best to always run the code in an empty environment.

Seed: set.seed(20190522)

The command set.seed(20190522) was run prior to running the code in the R Markdown file. Setting a seed ensures that any results that rely on randomness, e.g. subsampling or permutations, are reproducible.

Session information: recorded

Great job! Recording the operating system, R version, and package versions is critical for reproducibility.

Cache: none

Nice! There were no cached chunks for this analysis, so you can be confident that you successfully produced the results during this run.

File paths: relative

Great job! Using relative paths to the files within your workflowr project makes it easier to run your code on other machines.

Repository version: db0baaa

Great! You are using Git for version control. Tracking code development and connecting the code version to the results is critical for reproducibility. The version displayed above was the version of the Git repository at the time these results were generated.

Note that you need to be careful to ensure that all relevant files for the analysis have been committed to Git prior to generating the results (you can use wflow_publish or wflow_git_commit). workflowr only checks the R Markdown file, but you know if there are other scripts or data files that it depends on. Below is the status of the Git repository when the results were generated:


Ignored files:
    Ignored:    .Rhistory
    Ignored:    .Rproj.user/
    Ignored:    .Ruserdata/
    Ignored:    .drake/
    Ignored:    .vscode/
    Ignored:    analysis/rosm.cache/
    Ignored:    data/
    Ignored:    inst/Benchmark for Filter Methods for Feature Selection in High-Dimensional  Classification Data.pdf
    Ignored:    inst/study-area-map/study-area.qgs~
    Ignored:    log/
    Ignored:    renv/library/
    Ignored:    renv/staging/
    Ignored:    reviews/
    Ignored:    rosm.cache/

Untracked files:
    Untracked:  code/05-modeling/paper/01-eda.R
    Untracked:  code/05-modeling/paper/02-param-sets.R
    Untracked:  code/05-modeling/paper/03-learner.R
    Untracked:  code/05-modeling/paper/04-filter-wrapper.R
    Untracked:  code/05-modeling/paper/05-pca-wrapper.R
    Untracked:  code/05-modeling/paper/06-resampling.R
    Untracked:  code/05-modeling/paper/07-tune-ctrl.R
    Untracked:  code/05-modeling/paper/08-tune-wrapper-mbo.R
    Untracked:  code/05-modeling/paper/09-tune-wrapper.R
    Untracked:  code/05-modeling/paper/10-train.R
    Untracked:  code/98-paper/ieee/._IEEEtran.cls
    Untracked:  code/98-paper/ieee/IEEEtran.cls
    Untracked:  code/98-paper/ieee/ieee-manuscript.dvi

Unstaged changes:
    Modified:   .Rprofile
    Modified:   _drake.R
    Modified:   analysis/report-defoliation.Rmd
    Deleted:    code/05-modeling/paper/eda.R
    Modified:   code/05-modeling/paper/feature-importance.R
    Deleted:    code/05-modeling/paper/filter-wrapper.R
    Deleted:    code/05-modeling/paper/learner.R
    Deleted:    code/05-modeling/paper/param-sets.R
    Deleted:    code/05-modeling/paper/pca-wrapper.R
    Deleted:    code/05-modeling/paper/resampling.R
    Deleted:    code/05-modeling/paper/tune-ctrl.R
    Deleted:    code/05-modeling/paper/tune-wrapper-mbo.R
    Deleted:    code/05-modeling/paper/tune-wrapper.R
    Modified:   code/98-paper/ieee/pdf/correlation-filter-nri-1.pdf
    Modified:   code/98-paper/ieee/pdf/correlation-nbins-1.pdf
    Modified:   code/98-paper/ieee/pdf/defoliation-distribution-plot-1.pdf
    Modified:   code/98-paper/ieee/pdf/spectral-signatures-1.pdf
    Modified:   code/98-paper/journal/defoliation-distribution-plot-1.pdf

Note that any generated files, e.g. HTML, png, CSS, etc., are not included in this status report because it is ok for generated content to have uncommitted changes.

These are the previous versions of the R Markdown and HTML files. If you’ve configured a remote Git repository (see ?wflow_git_remote), click on the hyperlinks in the table below to view them.

File	Version	Author	Date	Message
Rmd	db0baaa	pat-s	2020-01-10	wflow_publish(knitr_in(“analysis/eval-performance.Rmd”), view = FALSE,
html	7f9507f	pat-s	2019-12-10	Build site.
Rmd	951e98c	pat-s	2019-12-10	wflow_publish(knitr_in(“analysis/eval-performance.Rmd”), view = FALSE, verbose =
Rmd	b27c623	pat-s	2019-12-09	Standard Error -> SE
Rmd	559a59d	pat-s	2019-12-09	add scatterplots to vis BM perf
html	482a158	pat-s	2019-11-01	Build site.
html	becf5ea	pat-s	2019-11-01	Build site.
html	bd7c7f5	pat-s	2019-10-31	Build site.
html	62ff96f	pat-s	2019-10-07	Build site.
html	a947654	pat-s	2019-10-02	Build site.
html	49da171	pat-s	2019-09-22	Build site.
html	c6317a8	pat-s	2019-09-19	Build site.
Rmd	d7c72a8	pat-s	2019-09-19	wflow_publish(knitr_in(“analysis/eval-performance.Rmd”), view = FALSE,
html	7fd40ca	pat-s	2019-09-18	Build site.
Rmd	44ff84b	pat-s	2019-09-18	wflow_publish(knitr_in(“analysis/eval-performance.Rmd”), view = FALSE,
html	41aae14	pat-s	2019-09-12	Build site.
html	ff340b8	pat-s	2019-09-03	Build site.
Rmd	a524819	pat-s	2019-09-03	wflow_publish(knitr_in(“analysis/eval-performance.Rmd”), view = FALSE,
html	b181c52	pat-s	2019-09-02	Build site.
Rmd	cf6e820	pat-s	2019-09-02	wflow_publish(“analysis/eval-performance.Rmd”)
Rmd	1bec10d	pat-s	2019-09-01	no timestamps in latex tables
html	4e363ac	pat-s	2019-09-01	Build site.
Rmd	518d0cb	pat-s	2019-09-01	style files using tidyverse style
html	8e7e4fe	pat-s	2019-09-01	Build site.
Rmd	8941bca	pat-s	2019-09-01	wflow_publish(knitr_in(“analysis/eval-performance.Rmd”), view = FALSE,
Rmd	297ed93	pat-s	2019-08-31	add filter vs no filter comparison plot
html	7582c67	pat-s	2019-08-31	Build site.
html	abd531f	pat-s	2019-08-31	Build site.
Rmd	9117eee	pat-s	2019-08-31	wflow_publish(knitr_in(“analysis/eval-performance.Rmd”), view = FALSE,
Rmd	5b629cb	pat-s	2019-08-19	add new tasks to performance eval report
html	1ec8768	pat-s	2019-08-17	Build site.
html	df85aba	pat-s	2019-07-12	Build site.
html	3a44a95	pat-s	2019-07-10	Build site.
html	c238ce4	pat-s	2019-07-10	Build site.
Rmd	e98cb01	pat-s	2019-07-10	wflow_publish(knitr_in(“analysis/eval-performance.Rmd”), view = FALSE,
Rmd	24e318f	pat-s	2019-07-01	update reports
Rmd	ca5c5bc	pat-s	2019-06-28	add eval-performance report

Warning: Expected 2 pieces. Additional pieces discarded in 72 rows [21, 22,
23, 24, 57, 58, 59, 60, 93, 94, 95, 96, 145, 146, 147, 148, 181, 182, 183,
184, ...].

Warning: Expected 2 pieces. Missing pieces filled with `NA` in 168 rows [1,
2, 3, 4, 37, 38, 39, 40, 73, 74, 75, 76, 109, 110, 111, 112, 113, 114, 115,
116, ...].

Aggregate performances and add standard error column.

(Table) All leaner/filter/task combinations ordered by performance.

Overall leaderboard across all settings, sorted descending by performance.

Task	Model	Filter	RMSE	SE
HR	RF	Relief	30.658	13.569
HR	RF	MRMR	30.678	13.608
HR	XGBoost	Borda	30.752	13.749
HR	XGBoost	CMIM	31.042	14.310
HR	XGBoost	Relief	31.332	14.870
HR	XGBoost	MRMR	31.438	15.076
NRI	XGBoost	MRMR	31.472	15.767
HR-NRI	SVM	Borda	31.704	15.621
HR-NRI	XGBoost	Pearson	31.713	15.905
NRI	SVM	Info	31.779	15.390
VI	SVM	Info	31.817	15.398
VI	SVM	CMIM	31.817	15.496
HR-NRI	SVM	Pearson	31.819	14.766
HR-NRI-VI	SVM	Borda	31.833	15.269
VI	SVM	Pearson	31.860	15.301
HR	SVM	Pearson	31.866	15.312
HR	SVM	Info	31.866	15.312
HR-VI	SVM	MRMR	31.866	15.312
HR	SVM	Relief	31.876	15.310
HR	SVM	MRMR	31.876	15.310
HR	SVM	CMIM	31.876	15.310
HR-VI	SVM	CMIM	31.876	15.310
NRI	SVM	Borda	31.895	15.309
HR-VI	SVM	Car	31.896	15.372
HR-VI	SVM	Borda	31.905	15.367
HR-VI	SVM	Pearson	31.905	15.390
HR-VI	SVM	Info	31.913	15.382
HR-NRI-VI	SVM	Car	31.915	15.283
NRI	SVM	Pearson	31.964	15.323
NRI	SVM	Relief	31.973	15.229
HR	SVM	PCA	32.007	15.200
HR-NRI	SVM	CMIM	32.016	15.193
HR-NRI-VI	SVM	MRMR	32.020	15.187
VI	SVM	PCA	32.027	15.178
HR-NRI	SVM	PCA	32.027	15.179
HR-VI	SVM	PCA	32.027	15.181
HR-NRI-VI	SVM	PCA	32.027	15.179
HR	SVM	No Filter	32.030	15.178
VI	SVM	No Filter	32.030	15.178
HR-NRI	SVM	No Filter	32.030	15.178
HR-VI	SVM	No Filter	32.030	15.178
HR-NRI-VI	SVM	No Filter	32.030	15.178
HR-NRI	SVM	MRMR	32.030	15.178
NRI	SVM	PCA	32.032	15.178
VI	SVM	Borda	32.037	15.244
HR	SVM	Borda	32.040	15.175
VI	SVM	Car	32.049	15.164
HR	SVM	Car	32.062	15.239
HR-NRI-VI	SVM	Pearson	32.080	15.283
HR-VI	SVM	Relief	32.089	15.131
VI	SVM	MRMR	32.111	15.154
NRI	XGBoost	Car	32.264	13.565
HR-NRI-VI	SVM	Info	32.457	16.012
HR-NRI	XGBoost	Info	32.721	15.749
HR-NRI	SVM	Relief	32.744	14.658
NRI	XGBoost	Info	32.839	14.900
HR-NRI-VI	XGBoost	MRMR	32.847	13.396
HR-NRI-VI	XGBoost	Borda	32.852	14.629
HR-NRI	XGBoost	Relief	33.020	13.009
NRI	Ridge-CV	No Filter	33.307	8.168
HR-NRI	XGBoost	CMIM	33.314	12.784
HR-NRI-VI	XGBoost	No Filter	33.358	12.592
NRI	XGBoost	Borda	33.408	12.942
HR-NRI-VI	XGBoost	Relief	33.437	13.963
HR-NRI	XGBoost	MRMR	33.562	13.663
HR-NRI-VI	XGBoost	Car	33.664	13.301
NRI	XGBoost	Pearson	33.749	13.621
HR-NRI	SVM	Car	33.793	14.153
HR-NRI	XGBoost	Borda	33.861	13.192
HR-NRI-VI	XGBoost	Pearson	34.353	10.355
NRI	RF	Car	34.431	10.726
NRI	XGBoost	CMIM	34.509	12.991
HR-NRI	XGBoost	Car	34.529	10.303
HR-NRI	RF	Car	34.605	10.621
HR	XGBoost	Car	34.790	13.327
NRI	SVM	Car	34.929	13.575
NRI	XGBoost	No Filter	34.942	11.297
VI	RF	Info	35.051	13.476
NRI	XGBoost	Relief	35.088	10.167
HR-VI	RF	Borda	35.164	13.705
HR-NRI	RF	PCA	35.243	12.554
HR-NRI-VI	RF	PCA	35.258	13.089
NRI	RF	CMIM	35.352	12.648
HR-NRI-VI	RF	CMIM	35.671	12.484
HR-NRI	SVM	Info	35.770	13.568
HR-NRI	XGBoost	No Filter	35.771	10.446
HR-NRI-VI	XGBoost	CMIM	35.870	12.558
HR-NRI-VI	XGBoost	Info	36.016	11.549
HR	RF	Borda	36.018	12.671
HR-NRI	RF	MRMR	36.090	13.063
HR-NRI	RF	Pearson	36.138	12.723
HR-NRI	RF	CMIM	36.176	12.371
HR-NRI-VI	RF	No Filter	36.359	12.324
HR-NRI-VI	RF	Info	36.382	12.684
NRI	RF	Info	36.427	12.660
NRI	RF	No Filter	36.458	12.062
HR-NRI	RF	No Filter	36.467	12.385
HR	RF	Info	36.529	12.362
NRI	RF	MRMR	36.623	13.247
NRI	RF	Pearson	36.634	12.710
HR	XGBoost	Info	36.673	13.344
HR	RF	Pearson	36.695	13.139
NRI	RF	Borda	36.730	12.756
VI	RF	Car	36.731	13.243
HR-NRI-VI	RF	Pearson	36.733	12.798
HR-NRI	RF	Info	36.740	12.427
HR-NRI-VI	RF	Borda	36.789	12.546
VI	XGBoost	Borda	36.801	13.089
HR-NRI	RF	Borda	36.815	12.864
VI	XGBoost	MRMR	36.839	13.174
VI	XGBoost	Info	36.851	13.201
VI	XGBoost	Car	36.979	13.389
HR-NRI-VI	SVM	CMIM	37.035	14.466
VI	XGBoost	Relief	37.100	13.451
VI	XGBoost	Pearson	37.160	13.561
VI	XGBoost	CMIM	37.255	13.707
HR-NRI-VI	RF	Relief	37.266	8.763
HR	XGBoost	Pearson	37.373	13.912
HR	RF	CMIM	37.538	14.282
HR	RF	Car	37.623	14.344
HR-VI	XGBoost	PCA	37.794	10.142
HR-NRI-VI	RF	MRMR	37.828	14.074
NRI	RF	PCA	38.147	12.111
HR-NRI-VI	RF	Car	38.241	8.252
VI	RF	No Filter	38.432	11.170
HR-NRI	XGBoost	PCA	38.851	12.264
NRI	SVM	MRMR	38.854	15.516
NRI	SVM	No Filter	38.893	15.169
VI	SVM	Relief	38.935	16.191
VI	RF	Relief	39.020	8.772
NRI	SVM	CMIM	39.068	15.701
HR-VI	RF	MRMR	39.193	10.676
HR-NRI	RF	Relief	39.401	6.609
HR-NRI-VI	SVM	Relief	39.469	15.723
NRI	RF	Relief	39.470	10.648
HR-NRI	Ridge-CV	No Filter	39.701	11.270
NRI	XGBoost	PCA	39.716	13.821
HR-VI	RF	Info	39.741	9.577
HR-VI	RF	No Filter	39.904	9.508
VI	RF	MRMR	39.957	8.447
HR	Ridge-CV	No Filter	39.996	11.509
VI	RF	CMIM	39.999	8.595
VI	RF	Pearson	40.082	8.747
VI	RF	Borda	40.210	8.681
HR-VI	RF	CMIM	40.254	8.465
HR-VI	RF	Car	40.310	9.469
HR-VI	RF	Relief	40.389	8.557
HR-VI	RF	Pearson	40.442	8.545
HR-VI	XGBoost	MRMR	40.464	10.139
VI	RF	PCA	40.932	8.304
HR-VI	XGBoost	CMIM	41.443	8.947
HR	RF	No Filter	41.738	7.618
HR-VI	XGBoost	Car	42.406	13.494
HR-VI	XGBoost	Pearson	42.530	9.193
HR-VI	RF	PCA	42.784	6.164
HR	RF	PCA	43.533	4.596
HR-NRI-VI	XGBoost	PCA	43.731	9.297
VI	XGBoost	No Filter	44.670	7.928
VI	XGBoost	PCA	45.104	5.893
HR	XGBoost	PCA	45.343	7.067
HR-VI	XGBoost	Info	45.584	3.944
HR-VI	XGBoost	Borda	45.722	5.301
HR-VI	XGBoost	No Filter	46.547	3.413
HR-VI	XGBoost	Relief	47.051	2.893
HR	Lasso-CV	No Filter	47.220	20.520
HR-NRI	Lasso-CV	No Filter	47.533	20.878
NRI	Lasso-CV	No Filter	47.946	20.686
HR	XGBoost	No Filter	49.263	4.485
HR-NRI-VI	Lasso-CV	No Filter	54.259	24.509
VI	Lasso-CV	No Filter	54.465	24.107
HR-VI	Lasso-CV	No Filter	54.465	24.107
HR	Lasso-MBO	No Filter	55.113	22.839
VI	Lasso-MBO	No Filter	55.113	22.839
NRI	Lasso-MBO	No Filter	55.113	22.839
HR-NRI	Lasso-MBO	No Filter	55.113	22.839
HR-VI	Lasso-MBO	No Filter	55.113	22.839
HR-NRI-VI	Lasso-MBO	No Filter	55.113	22.839
HR	Ridge-MBO	No Filter	55.113	22.839
VI	Ridge-MBO	No Filter	55.113	22.839
NRI	Ridge-MBO	No Filter	55.113	22.839
HR-NRI	Ridge-MBO	No Filter	55.113	22.839
HR-VI	Ridge-MBO	No Filter	55.113	22.839
HR-NRI-VI	Ridge-MBO	No Filter	55.113	22.839
HR-NRI-VI	Ridge-CV	No Filter	25199281.276	50398430.025
HR-VI	Ridge-CV	No Filter	27460680.793	54921229.059
VI	Ridge-CV	No Filter	27777883.484	55555634.441

(Table) Best learner/filter/task combination

Learners: On which task and using which filter did every learner score their best result on?

*CV: L2 penalized regression using the internal 10-fold CV tuning of the glmnet package

*MBO: L2 penalized regression using using MBO for hyperparameter optimization.

Task	Model	Filter	RMSE	SE
HR	RF	Relief	30.658	13.569
HR	XGBoost	Borda	30.752	13.749
HR-NRI	SVM	Borda	31.704	15.621
NRI	Ridge-CV	No Filter	33.307	8.168
HR	Lasso-CV	No Filter	47.220	20.520
HR	Lasso-MBO	No Filter	55.113	22.839
HR	Ridge-MBO	No Filter	55.113	22.839

(Plot) Best learner/filter combs for all tasks

Version	Author	Date
7f9507f	pat-s	2019-12-10
482a158	pat-s	2019-11-01
becf5ea	pat-s	2019-11-01
bd7c7f5	pat-s	2019-10-31
62ff96f	pat-s	2019-10-07
a947654	pat-s	2019-10-02
49da171	pat-s	2019-09-22
41aae14	pat-s	2019-09-12
b181c52	pat-s	2019-09-02
8e7e4fe	pat-s	2019-09-01
7582c67	pat-s	2019-08-31
abd531f	pat-s	2019-08-31

(Plot) Best filter combination of each learner vs. no filter per task vs. Borda

Showing the final effect of applying feature selection to a learner for each task. The more left a certain filter appears for a given task compared to the purple dot (No Filter), the higher the effectivity of applying feature selection for that given learner on the given task.

Version	Author	Date
7f9507f	pat-s	2019-12-10

(Plot) Best filter combination of each learner vs. Borda filter

Showing the final effect of the ensemble Borda filter vs the best scoring simple filter.

Version	Author	Date
7f9507f	pat-s	2019-12-10
482a158	pat-s	2019-11-01
becf5ea	pat-s	2019-11-01
bd7c7f5	pat-s	2019-10-31
62ff96f	pat-s	2019-10-07
a947654	pat-s	2019-10-02
49da171	pat-s	2019-09-22
7fd40ca	pat-s	2019-09-18
41aae14	pat-s	2019-09-12
ff340b8	pat-s	2019-09-03
b181c52	pat-s	2019-09-02

(Plot) Performances of all filter methods

(Plot) Scatterplots of filter methods vs. no filter for each learner and task

Showing the final effect of applying feature selection to a learner for each task. All filters are colored in the same way whereas using “no filter” appears in a different color.

Version	Author	Date
7f9507f	pat-s	2019-12-10

(Plot) Scatterplots of filter methods vs. Borda for each learner and task

Showing the final effect of applying feature selection to a learner for each task. All filters are summarized into a a single color whereas the “Borda” filter appears in its own color.

Version	Author	Date
7f9507f	pat-s	2019-12-10

R version 3.6.1 (2019-07-05)
Platform: x86_64-pc-linux-gnu (64-bit)
Running under: CentOS Linux 7 (Core)

Matrix products: default
BLAS:   /opt/spack/opt/spack/linux-centos7-x86_64/gcc-9.2.0/r-3.6.1-j25wr6zcofibs2zfjwg37357rjj26lqb/rlib/R/lib/libRblas.so
LAPACK: /opt/spack/opt/spack/linux-centos7-x86_64/gcc-9.2.0/r-3.6.1-j25wr6zcofibs2zfjwg37357rjj26lqb/rlib/R/lib/libRlapack.so

locale:
 [1] LC_CTYPE=en_US.UTF-8       LC_NUMERIC=C              
 [3] LC_TIME=en_US.UTF-8        LC_COLLATE=en_US.UTF-8    
 [5] LC_MONETARY=en_US.UTF-8    LC_MESSAGES=en_US.UTF-8   
 [7] LC_PAPER=en_US.UTF-8       LC_NAME=C                 
 [9] LC_ADDRESS=C               LC_TELEPHONE=C            
[11] LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C       

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base     

other attached packages:
 [1] xtable_1.8-3      flextable_0.5.6   ggbeeswarm_0.7.0 
 [4] ggpubr_0.1.6      ggsci_2.9         ggrepel_0.8.0    
 [7] ggplot2_3.2.1     dplyr_0.8.0.1     magrittr_1.5     
[10] mlr_2.16.0.9000   ParamHelpers_1.12 tidyselect_0.2.5 

loaded via a namespace (and not attached):
 [1] httr_1.4.0         tidyr_0.8.2        jsonlite_1.6      
 [4] viridisLite_0.3.0  splines_3.6.1      R.utils_2.8.0     
 [7] assertthat_0.2.0   base64url_1.4      vipor_0.4.5       
[10] yaml_2.2.0         gdtools_0.2.1      mlrMBO_1.1.2      
[13] pillar_1.3.1       backports_1.1.5    lattice_0.20-38   
[16] glue_1.3.0         uuid_0.1-2         digest_0.6.22     
[19] RColorBrewer_1.1-2 promises_1.0.1     checkmate_1.9.1   
[22] colorspace_1.4-0   htmltools_0.3.6    httpuv_1.4.5.1    
[25] Matrix_1.2-15      R.oo_1.23.0        pkgconfig_2.0.3   
[28] lhs_1.0.1          misc3d_0.8-4       drake_7.8.0       
[31] purrr_0.3.0        scales_1.0.0       parallelMap_1.4   
[34] whisker_0.3-2      later_0.8.0        officer_0.3.5     
[37] mco_1.0-15.1       git2r_0.26.1       tibble_2.0.1      
[40] txtq_0.1.4         withr_2.1.2        lazyeval_0.2.1    
[43] survival_2.43-3    RJSONIO_1.3-1.1    crayon_1.3.4      
[46] evaluate_0.13      storr_1.2.1        R.methodsS3_1.7.1 
[49] fs_1.3.1           xml2_1.2.0         beeswarm_0.2.3    
[52] tools_3.6.1        data.table_1.12.6  BBmisc_1.11       
[55] stringr_1.4.0      plotly_4.8.0       munsell_0.5.0     
[58] zip_2.0.0          compiler_3.6.1     systemfonts_0.1.1 
[61] rlang_0.4.1        plot3D_1.1.1       grid_3.6.1        
[64] htmlwidgets_1.3    igraph_1.2.4.1     labeling_0.3      
[67] base64enc_0.1-3    rmarkdown_1.13     gtable_0.2.0      
[70] smoof_1.5.1        R6_2.4.1           knitr_1.23        
[73] fastmatch_1.1-0    filelock_1.0.2     workflowr_1.6.0   
[76] rprojroot_1.3-2    stringi_1.3.1      parallel_3.6.1    
[79] Rcpp_1.0.0         xfun_0.5