Evaluation of performances

(Table) T1 All leaner/filter/task combinations ordered by performance.
(Table) T2 Best learner/filter/task combination
(Plot) P1 Best learner/filter combs for all tasks
(Plot) P2 Scatterplots of filter methods vs. no filter for each learner and task
(Plot) P3 Scatterplots of filter methods vs. Borda for each learner and task

Last updated: 2020-03-05

Checks: 7 0

Knit directory: 2019-feature-selection/

This reproducible R Markdown analysis was created with workflowr (version 1.6.0). The Checks tab describes the reproducibility checks that were applied when the results were created. The Past versions tab lists the development history.

R Markdown file: up-to-date

Great! Since the R Markdown file has been committed to the Git repository, you know the exact version of the code that produced these results.

Environment: empty

Great job! The global environment was empty. Objects defined in the global environment can affect the analysis in your R Markdown file in unknown ways. For reproduciblity it’s best to always run the code in an empty environment.

Seed: set.seed(20190522)

The command set.seed(20190522) was run prior to running the code in the R Markdown file. Setting a seed ensures that any results that rely on randomness, e.g. subsampling or permutations, are reproducible.

Session information: recorded

Great job! Recording the operating system, R version, and package versions is critical for reproducibility.

Cache: none

Nice! There were no cached chunks for this analysis, so you can be confident that you successfully produced the results during this run.

File paths: relative

Great job! Using relative paths to the files within your workflowr project makes it easier to run your code on other machines.

Repository version: d487d51

Great! You are using Git for version control. Tracking code development and connecting the code version to the results is critical for reproducibility. The version displayed above was the version of the Git repository at the time these results were generated.

Note that you need to be careful to ensure that all relevant files for the analysis have been committed to Git prior to generating the results (you can use wflow_publish or wflow_git_commit). workflowr only checks the R Markdown file, but you know if there are other scripts or data files that it depends on. Below is the status of the Git repository when the results were generated:


Ignored files:
    Ignored:    .Rhistory
    Ignored:    .Rproj.user/
    Ignored:    .Ruserdata/
    Ignored:    .drake/
    Ignored:    .vscode/
    Ignored:    analysis/rosm.cache/
    Ignored:    data/
    Ignored:    inst/Benchmark for Filter Methods for Feature Selection in High-Dimensional  Classification Data.pdf
    Ignored:    inst/study-area-map/._study-area.qgs
    Ignored:    inst/study-area-map/study-area.qgs~
    Ignored:    log/
    Ignored:    renv/library/
    Ignored:    renv/staging/
    Ignored:    reviews/
    Ignored:    rosm.cache/

Untracked files:
    Untracked:  code/06-modeling/project/

Unstaged changes:
    Modified:   .Rprofile
    Modified:   _drake.R
    Modified:   analysis/report-defoliation.Rmd
    Modified:   code/02-hyperspectral-processing.R
    Modified:   code/03-sentinel-processing.R
    Modified:   code/06-modeling/paper/08-train.R

Note that any generated files, e.g. HTML, png, CSS, etc., are not included in this status report because it is ok for generated content to have uncommitted changes.

These are the previous versions of the R Markdown and HTML files. If you’ve configured a remote Git repository (see ?wflow_git_remote), click on the hyperlinks in the table below to view them.

File	Version	Author	Date	Message
Rmd	d487d51	pat-s	2020-03-05	wflow_publish(knitr_in(“analysis/eval-performance.Rmd”), view = FALSE,
html	776b35f	pat-s	2020-03-03	Build site.
Rmd	d0c645a	pat-s	2020-03-03	wflow_publish(knitr_in(“analysis/eval-performance.Rmd”), view = FALSE,
html	274a918	pat-s	2020-02-25	Build site.
Rmd	2e632e1	pat-s	2020-02-25	wflow_publish(knitr_in(“analysis/eval-performance.Rmd”), view = FALSE,
Rmd	a53ea68	pat-s	2020-02-24	add prefixes to plots and tables
Rmd	379bd5d	pat-s	2020-02-24	wflow_publish(knitr_in(“analysis/eval-performance.Rmd”), view = FALSE,
html	f680221	pat-s	2020-01-15	Build site.
html	b25e779	pat-s	2020-01-10	Build site.
Rmd	db0baaa	pat-s	2020-01-10	wflow_publish(knitr_in(“analysis/eval-performance.Rmd”), view = FALSE,
html	7f9507f	pat-s	2019-12-10	Build site.
Rmd	951e98c	pat-s	2019-12-10	wflow_publish(knitr_in(“analysis/eval-performance.Rmd”), view = FALSE, verbose =
Rmd	b27c623	pat-s	2019-12-09	Standard Error -> SE
Rmd	559a59d	pat-s	2019-12-09	add scatterplots to vis BM perf
html	482a158	pat-s	2019-11-01	Build site.
html	becf5ea	pat-s	2019-11-01	Build site.
html	bd7c7f5	pat-s	2019-10-31	Build site.
html	62ff96f	pat-s	2019-10-07	Build site.
html	a947654	pat-s	2019-10-02	Build site.
html	49da171	pat-s	2019-09-22	Build site.
html	c6317a8	pat-s	2019-09-19	Build site.
Rmd	d7c72a8	pat-s	2019-09-19	wflow_publish(knitr_in(“analysis/eval-performance.Rmd”), view = FALSE,
html	7fd40ca	pat-s	2019-09-18	Build site.
Rmd	44ff84b	pat-s	2019-09-18	wflow_publish(knitr_in(“analysis/eval-performance.Rmd”), view = FALSE,
html	41aae14	pat-s	2019-09-12	Build site.
html	ff340b8	pat-s	2019-09-03	Build site.
Rmd	a524819	pat-s	2019-09-03	wflow_publish(knitr_in(“analysis/eval-performance.Rmd”), view = FALSE,
html	b181c52	pat-s	2019-09-02	Build site.
Rmd	cf6e820	pat-s	2019-09-02	wflow_publish(“analysis/eval-performance.Rmd”)
Rmd	1bec10d	pat-s	2019-09-01	no timestamps in latex tables
html	4e363ac	pat-s	2019-09-01	Build site.
Rmd	518d0cb	pat-s	2019-09-01	style files using tidyverse style
html	8e7e4fe	pat-s	2019-09-01	Build site.
Rmd	8941bca	pat-s	2019-09-01	wflow_publish(knitr_in(“analysis/eval-performance.Rmd”), view = FALSE,
Rmd	297ed93	pat-s	2019-08-31	add filter vs no filter comparison plot
html	7582c67	pat-s	2019-08-31	Build site.
html	abd531f	pat-s	2019-08-31	Build site.
Rmd	9117eee	pat-s	2019-08-31	wflow_publish(knitr_in(“analysis/eval-performance.Rmd”), view = FALSE,
Rmd	5b629cb	pat-s	2019-08-19	add new tasks to performance eval report
html	1ec8768	pat-s	2019-08-17	Build site.
html	df85aba	pat-s	2019-07-12	Build site.
html	3a44a95	pat-s	2019-07-10	Build site.
html	c238ce4	pat-s	2019-07-10	Build site.
Rmd	e98cb01	pat-s	2019-07-10	wflow_publish(knitr_in(“analysis/eval-performance.Rmd”), view = FALSE,
Rmd	24e318f	pat-s	2019-07-01	update reports
Rmd	ca5c5bc	pat-s	2019-06-28	add eval-performance report

Last update:

[1] "Thu Mar  5 10:34:01 2020"

(Table) T1 All leaner/filter/task combinations ordered by performance.

Overall leaderboard across all settings, sorted descending by performance.

Task	Model	Filter	RMSE	SE
HR-NRI	SVM	Car	27.983	19.193
HR-NRI-VI	SVM	Relief	28.049	19.124
HR-NRI	SVM	Relief	28.117	19.125
HR	SVM	Car	28.122	19.123
HR	SVM	Info Gain	28.122	19.123
VI	SVM	Relief	28.122	19.113
HR	SVM	CMIM	28.122	19.123
NRI-VI	SVM	PCA	28.122	19.123
HR-NRI-VI	SVM	PCA	28.122	19.123
VI	SVM	No Filter	28.122	19.123
HR-NRI	SVM	No Filter	28.122	19.123
HR-NRI-VI	SVM	No Filter	28.122	19.123
NRI-VI	SVM	Car	28.123	19.123
HR	SVM	Relief	28.123	19.122
HR	SVM	Pearson	28.123	19.123
HR	SVM	Borda	28.123	19.123
NRI	SVM	PCA	28.123	19.123
HR-NRI	SVM	PCA	28.123	19.123
HR	SVM	No Filter	28.123	19.123
NRI	SVM	No Filter	28.123	19.123
NRI-VI	SVM	No Filter	28.123	19.123
HR	SVM	PCA	28.125	19.122
VI	SVM	Info Gain	28.126	19.123
VI	SVM	MRMR	28.130	19.114
HR-NRI-VI	SVM	CMIM	28.130	19.114
VI	SVM	Borda	28.135	19.146
NRI-VI	SVM	Borda	28.140	19.099
VI	SVM	PCA	28.143	19.093
NRI-VI	SVM	MRMR	28.144	19.142
HR	SVM	MRMR	28.153	19.123
HR-NRI-VI	SVM	Info Gain	28.162	19.077
HR-NRI-VI	SVM	MRMR	28.163	19.051
VI	SVM	Pearson	28.171	19.212
NRI	SVM	Pearson	28.186	19.121
NRI	SVM	Relief	28.188	19.022
VI	SVM	CMIM	28.188	19.200
NRI	SVM	Borda	28.198	19.024
NRI-VI	SVM	Pearson	28.209	19.193
VI	SVM	Car	28.212	19.109
HR-NRI-VI	SVM	Pearson	28.227	19.169
HR-NRI	SVM	Info Gain	28.238	18.987
HR-NRI-VI	SVM	Car	28.280	18.919
NRI	SVM	CMIM	28.300	18.889
HR-NRI	SVM	Borda	28.307	18.912
NRI-VI	SVM	CMIM	28.348	18.826
HR-NRI	SVM	MRMR	28.511	18.610
NRI	SVM	MRMR	28.517	18.607
NRI-VI	SVM	Relief	28.622	18.824
HR-NRI	SVM	CMIM	28.683	18.395
HR-NRI-VI	SVM	Borda	28.914	18.109
NRI	SVM	Car	28.947	18.602
NRI-VI	SVM	Info Gain	29.158	17.712
HR-NRI	SVM	Pearson	30.978	18.394
HR-NRI	RF	Car	31.077	17.093
NRI-VI	XGBOOST	Relief	32.103	17.730
HR-NRI-VI	XGBOOST	MRMR	32.313	17.381
NRI	SVM	Info Gain	32.354	18.629
HR-NRI-VI	RF	Car	32.451	15.894
NRI	XGBOOST	No Filter	32.556	16.516
HR-NRI	XGBOOST	Relief	32.566	15.251
NRI-VI	RF	PCA	32.708	16.423
NRI	Ridge-CV	No Filter	32.974	7.237
HR-NRI-VI	XGBOOST	CMIM	33.084	15.219
NRI	RF	PCA	33.311	15.957
HR-NRI-VI	RF	PCA	33.370	15.964
HR-NRI	RF	PCA	33.598	15.604
HR-NRI-VI	XGBOOST	Car	33.692	14.135
HR-NRI-VI	XGBOOST	Info Gain	33.772	15.459
NRI	XGBOOST	Borda	34.012	15.435
HR-NRI	RF	Borda	34.077	14.375
NRI-VI	XGBOOST	CMIM	34.423	13.880
HR	RF	Car	34.467	12.604
HR-NRI	XGBOOST	Car	34.537	15.019
NRI	XGBOOST	Car	34.546	13.779
NRI	XGBOOST	Pearson	34.612	14.619
NRI-VI	XGBOOST	Info Gain	34.690	15.672
NRI-VI	RF	Car	34.813	13.146
HR-NRI-VI	XGBOOST	Borda	35.024	14.277
NRI-VI	XGBOOST	Borda	35.085	15.110
HR	XGBOOST	Car	35.094	13.624
NRI	RF	MRMR	35.109	13.318
NRI	RF	Car	35.182	12.041
NRI	RF	Borda	35.182	12.301
HR-NRI-VI	RF	MRMR	35.230	13.808
NRI-VI	XGBOOST	Car	35.238	13.304
HR-NRI	XGBOOST	Pearson	35.262	13.043
HR	RF	Info Gain	35.271	12.755
HR-NRI	XGBOOST	CMIM	35.315	13.794
NRI-VI	RF	CMIM	35.330	14.913
HR-NRI	XGBOOST	Info Gain	35.367	14.339
HR-NRI-VI	XGBOOST	Pearson	35.370	14.406
NRI-VI	RF	MRMR	35.511	14.834
HR-NRI-VI	RF	Borda	35.527	13.643
NRI	XGBOOST	CMIM	35.535	13.654
HR-NRI	RF	Pearson	35.556	13.076
HR-NRI-VI	XGBOOST	Relief	35.556	14.832
NRI	RF	CMIM	35.629	13.644
NRI-VI	XGBOOST	Pearson	35.660	13.066
HR-NRI	XGBOOST	Borda	35.662	13.289
HR-NRI	RF	MRMR	35.697	14.425
NRI	XGBOOST	Info Gain	35.728	14.460
HR-NRI-VI	XGBOOST	No Filter	35.762	12.884
HR-NRI-VI	RF	No Filter	35.867	12.747
NRI	RF	Info Gain	35.980	13.862
HR-NRI-VI	RF	CMIM	36.087	14.219
NRI-VI	RF	Borda	36.183	13.091
HR-NRI	XGBOOST	No Filter	36.187	10.588
HR-NRI	RF	No Filter	36.211	12.830
NRI-VI	RF	No Filter	36.344	12.844
NRI	RF	No Filter	36.389	12.483
NRI-VI	RF	Info Gain	36.406	13.666
HR-NRI	RF	Info Gain	36.671	13.625
NRI-VI	XGBOOST	MRMR	36.700	13.207
HR-NRI	XGBOOST	MRMR	36.700	11.892
NRI	XGBOOST	MRMR	36.731	11.896
HR-NRI	RF	CMIM	36.741	13.790
NRI-VI	RF	Pearson	36.755	13.206
HR-NRI-VI	RF	Info Gain	36.813	13.486
HR-NRI-VI	RF	Pearson	36.837	13.846
NRI	RF	Pearson	36.842	13.567
HR	RF	PCA	37.096	12.302
NRI-VI	XGBOOST	No Filter	37.143	12.421
HR	XGBOOST	Pearson	37.239	14.416
HR	XGBOOST	Relief	37.307	10.286
HR	RF	Borda	37.342	13.099
HR	RF	Pearson	37.351	13.852
NRI	XGBOOST	Relief	37.539	10.630
HR	XGBOOST	MRMR	37.774	12.026
VI	RF	MRMR	38.320	11.654
HR-NRI-VI	RF	Relief	38.450	8.774
HR-NRI	RF	Relief	38.598	8.998
NRI-VI	RF	Relief	38.621	8.764
VI	RF	No Filter	38.698	10.946
VI	RF	Relief	38.879	11.204
VI	RF	Borda	38.974	11.901
NRI	RF	Relief	38.985	8.031
HR	XGBOOST	Info Gain	39.214	12.433
HR	RF	CMIM	39.437	8.018
VI	RF	Car	39.901	12.918
VI	XGBOOST	Relief	39.903	9.438
VI	RF	Info Gain	39.907	9.395
VI	RF	CMIM	40.046	9.784
VI	RF	Pearson	40.247	9.882
HR-NRI	Ridge-CV	No Filter	40.271	10.513
VI	XGBOOST	Info Gain	40.301	10.101
VI	XGBOOST	Pearson	40.361	9.898
HR	Ridge-CV	No Filter	40.520	11.207
HR-NRI-VI	XGBOOST	PCA	40.535	11.371
VI	RF	PCA	40.761	8.516
NRI	XGBOOST	PCA	40.797	11.785
HR	XGBOOST	CMIM	40.960	9.937
HR-NRI	XGBOOST	PCA	40.974	10.499
HR	RF	No Filter	41.067	8.444
HR	RF	Relief	41.302	10.593
VI	XGBOOST	CMIM	41.639	10.152
HR	XGBOOST	Borda	42.073	14.376
VI	XGBOOST	PCA	42.115	7.037
HR	RF	MRMR	42.423	7.454
HR	XGBOOST	PCA	43.129	7.629
NRI-VI	XGBOOST	PCA	43.312	8.997
VI	XGBOOST	No Filter	44.444	8.759
HR	Lasso-CV	No Filter	45.960	18.350
NRI	Lasso-CV	No Filter	46.331	20.036
VI	XGBOOST	Borda	46.502	4.109
VI	XGBOOST	MRMR	46.520	5.598
HR-NRI	Lasso-CV	No Filter	46.653	20.279
VI	XGBOOST	Car	46.682	7.759
HR	XGBOOST	No Filter	49.382	1.358
NRI-VI	Lasso-CV	No Filter	54.497	24.638
HR-NRI-VI	Lasso-CV	No Filter	54.497	24.638
VI	Lasso-CV	No Filter	55.325	23.009
HR-NRI-VI	Ridge-CV	No Filter	9230336.565	18460539.886
NRI-VI	Ridge-CV	No Filter	9409773.079	18819412.914
VI	Ridge-CV	No Filter	11337132.755	22674132.319

(Table) T2 Best learner/filter/task combination

Learners: On which task and using which filter did every learner score their best result on?

*CV: L2 penalized regression using the internal 10-fold CV tuning of the glmnet package

*MBO: L2 penalized regression using using MBO for hyperparameter optimization.

Task	Model	Filter	RMSE	SE
HR-NRI	SVM	Car	27.983	19.193
HR-NRI	RF	Car	31.077	17.093
NRI-VI	XGBOOST	Relief	32.103	17.730
NRI	Ridge-CV	No Filter	32.974	7.237
HR	Lasso-CV	No Filter	45.960	18.350

(Plot) P1 Best learner/filter combs for all tasks

Version	Author	Date
274a918	pat-s	2020-02-25
b25e779	pat-s	2020-01-10
7f9507f	pat-s	2019-12-10
482a158	pat-s	2019-11-01
becf5ea	pat-s	2019-11-01
bd7c7f5	pat-s	2019-10-31
62ff96f	pat-s	2019-10-07
a947654	pat-s	2019-10-02
49da171	pat-s	2019-09-22
41aae14	pat-s	2019-09-12
b181c52	pat-s	2019-09-02
8e7e4fe	pat-s	2019-09-01
7582c67	pat-s	2019-08-31
abd531f	pat-s	2019-08-31

(Plot) P2 Scatterplots of filter methods vs. no filter for each learner and task

Showing the final effect of applying feature selection to a learner for each task. All filters are colored in the same way whereas using “no filter” appears in a different color.

Version	Author	Date
274a918	pat-s	2020-02-25
b25e779	pat-s	2020-01-10
7f9507f	pat-s	2019-12-10

(Plot) P3 Scatterplots of filter methods vs. Borda for each learner and task

Showing the final effect of applying feature selection to a learner for each task. All filters are summarized into a a single color whereas the “Borda” filter appears in its own color.

Version	Author	Date
776b35f	pat-s	2020-03-03
274a918	pat-s	2020-02-25
b25e779	pat-s	2020-01-10
7f9507f	pat-s	2019-12-10

R version 3.6.1 (2019-07-05)
Platform: x86_64-pc-linux-gnu (64-bit)
Running under: CentOS Linux 7 (Core)

Matrix products: default
BLAS:   /opt/spack/opt/spack/linux-centos7-x86_64/gcc-9.2.0/r-3.6.1-j25wr6zcofibs2zfjwg37357rjj26lqb/rlib/R/lib/libRblas.so
LAPACK: /opt/spack/opt/spack/linux-centos7-x86_64/gcc-9.2.0/r-3.6.1-j25wr6zcofibs2zfjwg37357rjj26lqb/rlib/R/lib/libRlapack.so

locale:
 [1] LC_CTYPE=en_US.UTF-8       LC_NUMERIC=C              
 [3] LC_TIME=en_US.UTF-8        LC_COLLATE=en_US.UTF-8    
 [5] LC_MONETARY=en_US.UTF-8    LC_MESSAGES=en_US.UTF-8   
 [7] LC_PAPER=en_US.UTF-8       LC_NAME=C                 
 [9] LC_ADDRESS=C               LC_TELEPHONE=C            
[11] LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C       

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base     

other attached packages:
 [1] forcats_0.4.0     dplyr_0.8.3       mlr_2.17.0.9001  
 [4] ParamHelpers_1.12 here_0.1          ggpubr_0.1.6     
 [7] magrittr_1.5      ggrepel_0.8.0     ggsci_2.9        
[10] ggbeeswarm_0.7.0  ggplot2_3.2.1     flextable_0.5.8  
[13] xtable_1.8-3      tidyselect_0.2.5  drake_7.10.0     

loaded via a namespace (and not attached):
 [1] fs_1.3.1           filelock_1.0.2     RColorBrewer_1.1-2
 [4] httr_1.4.0         rprojroot_1.3-2    tools_3.6.1       
 [7] backports_1.1.5    R6_2.4.1           vipor_0.4.5       
[10] lazyeval_0.2.1     colorspace_1.4-0   withr_2.1.2       
[13] mco_1.0-15.1       compiler_3.6.1     git2r_0.26.1      
[16] parallelMap_1.4    xml2_1.2.2         plotly_4.8.0      
[19] officer_0.3.7      labeling_0.3       scales_1.0.0      
[22] checkmate_1.9.1    plot3D_1.1.1       systemfonts_0.1.1 
[25] stringr_1.4.0      digest_0.6.23      txtq_0.1.4        
[28] rmarkdown_1.13     R.utils_2.8.0      smoof_1.5.1       
[31] base64enc_0.1-3    pkgconfig_2.0.3    htmltools_0.3.6   
[34] lhs_1.0.1          htmlwidgets_1.3    rlang_0.4.4       
[37] BBmisc_1.11        mlrMBO_1.1.2       jsonlite_1.6      
[40] zip_2.0.4          R.oo_1.23.0        Matrix_1.2-15     
[43] Rcpp_1.0.3         munsell_0.5.0      gdtools_0.2.1     
[46] lifecycle_0.1.0    R.methodsS3_1.7.1  stringi_1.3.1     
[49] whisker_0.3-2      yaml_2.2.0         storr_1.2.1       
[52] RJSONIO_1.3-1.1    grid_3.6.1         misc3d_0.8-4      
[55] parallel_3.6.1     promises_1.0.1     crayon_1.3.4      
[58] lattice_0.20-38    splines_3.6.1      zeallot_0.1.0     
[61] knitr_1.23         pillar_1.4.3       igraph_1.2.4.1    
[64] uuid_0.1-2         base64url_1.4      fastmatch_1.1-0   
[67] glue_1.3.1         evaluate_0.13      data.table_1.12.6 
[70] vctrs_0.2.1        httpuv_1.4.5.1     gtable_0.2.0      
[73] purrr_0.3.3        tidyr_1.0.0        assertthat_0.2.1  
[76] xfun_0.5           later_1.0.0        survival_2.43-3   
[79] viridisLite_0.3.0  tibble_2.1.3       beeswarm_0.2.3    
[82] workflowr_1.6.0    ellipsis_0.3.0