Last updated: 2020-03-05

Checks: 7 0

Knit directory: 2019-feature-selection/

This reproducible R Markdown analysis was created with workflowr (version 1.6.0). The Checks tab describes the reproducibility checks that were applied when the results were created. The Past versions tab lists the development history.


Great! Since the R Markdown file has been committed to the Git repository, you know the exact version of the code that produced these results.

Great job! The global environment was empty. Objects defined in the global environment can affect the analysis in your R Markdown file in unknown ways. For reproduciblity it’s best to always run the code in an empty environment.

The command set.seed(20190522) was run prior to running the code in the R Markdown file. Setting a seed ensures that any results that rely on randomness, e.g. subsampling or permutations, are reproducible.

Great job! Recording the operating system, R version, and package versions is critical for reproducibility.

Nice! There were no cached chunks for this analysis, so you can be confident that you successfully produced the results during this run.

Great job! Using relative paths to the files within your workflowr project makes it easier to run your code on other machines.

Great! You are using Git for version control. Tracking code development and connecting the code version to the results is critical for reproducibility. The version displayed above was the version of the Git repository at the time these results were generated.

Note that you need to be careful to ensure that all relevant files for the analysis have been committed to Git prior to generating the results (you can use wflow_publish or wflow_git_commit). workflowr only checks the R Markdown file, but you know if there are other scripts or data files that it depends on. Below is the status of the Git repository when the results were generated:


Ignored files:
    Ignored:    .Rhistory
    Ignored:    .Rproj.user/
    Ignored:    .Ruserdata/
    Ignored:    .drake/
    Ignored:    .vscode/
    Ignored:    analysis/rosm.cache/
    Ignored:    data/
    Ignored:    inst/Benchmark for Filter Methods for Feature Selection in High-Dimensional  Classification Data.pdf
    Ignored:    inst/study-area-map/._study-area.qgs
    Ignored:    inst/study-area-map/study-area.qgs~
    Ignored:    log/
    Ignored:    renv/library/
    Ignored:    renv/staging/
    Ignored:    reviews/
    Ignored:    rosm.cache/

Untracked files:
    Untracked:  code/06-modeling/project/

Unstaged changes:
    Modified:   .Rprofile
    Modified:   _drake.R
    Modified:   analysis/report-defoliation.Rmd
    Modified:   code/02-hyperspectral-processing.R
    Modified:   code/03-sentinel-processing.R
    Modified:   code/06-modeling/paper/08-train.R

Note that any generated files, e.g. HTML, png, CSS, etc., are not included in this status report because it is ok for generated content to have uncommitted changes.


These are the previous versions of the R Markdown and HTML files. If you’ve configured a remote Git repository (see ?wflow_git_remote), click on the hyperlinks in the table below to view them.

File Version Author Date Message
Rmd d487d51 pat-s 2020-03-05 wflow_publish(knitr_in(“analysis/eval-performance.Rmd”), view = FALSE,
html 776b35f pat-s 2020-03-03 Build site.
Rmd d0c645a pat-s 2020-03-03 wflow_publish(knitr_in(“analysis/eval-performance.Rmd”), view = FALSE,
html 274a918 pat-s 2020-02-25 Build site.
Rmd 2e632e1 pat-s 2020-02-25 wflow_publish(knitr_in(“analysis/eval-performance.Rmd”), view = FALSE,
Rmd a53ea68 pat-s 2020-02-24 add prefixes to plots and tables
Rmd 379bd5d pat-s 2020-02-24 wflow_publish(knitr_in(“analysis/eval-performance.Rmd”), view = FALSE,
html f680221 pat-s 2020-01-15 Build site.
html b25e779 pat-s 2020-01-10 Build site.
Rmd db0baaa pat-s 2020-01-10 wflow_publish(knitr_in(“analysis/eval-performance.Rmd”), view = FALSE,
html 7f9507f pat-s 2019-12-10 Build site.
Rmd 951e98c pat-s 2019-12-10 wflow_publish(knitr_in(“analysis/eval-performance.Rmd”), view = FALSE, verbose =
Rmd b27c623 pat-s 2019-12-09 Standard Error -> SE
Rmd 559a59d pat-s 2019-12-09 add scatterplots to vis BM perf
html 482a158 pat-s 2019-11-01 Build site.
html becf5ea pat-s 2019-11-01 Build site.
html bd7c7f5 pat-s 2019-10-31 Build site.
html 62ff96f pat-s 2019-10-07 Build site.
html a947654 pat-s 2019-10-02 Build site.
html 49da171 pat-s 2019-09-22 Build site.
html c6317a8 pat-s 2019-09-19 Build site.
Rmd d7c72a8 pat-s 2019-09-19 wflow_publish(knitr_in(“analysis/eval-performance.Rmd”), view = FALSE,
html 7fd40ca pat-s 2019-09-18 Build site.
Rmd 44ff84b pat-s 2019-09-18 wflow_publish(knitr_in(“analysis/eval-performance.Rmd”), view = FALSE,
html 41aae14 pat-s 2019-09-12 Build site.
html ff340b8 pat-s 2019-09-03 Build site.
Rmd a524819 pat-s 2019-09-03 wflow_publish(knitr_in(“analysis/eval-performance.Rmd”), view = FALSE,
html b181c52 pat-s 2019-09-02 Build site.
Rmd cf6e820 pat-s 2019-09-02 wflow_publish(“analysis/eval-performance.Rmd”)
Rmd 1bec10d pat-s 2019-09-01 no timestamps in latex tables
html 4e363ac pat-s 2019-09-01 Build site.
Rmd 518d0cb pat-s 2019-09-01 style files using tidyverse style
html 8e7e4fe pat-s 2019-09-01 Build site.
Rmd 8941bca pat-s 2019-09-01 wflow_publish(knitr_in(“analysis/eval-performance.Rmd”), view = FALSE,
Rmd 297ed93 pat-s 2019-08-31 add filter vs no filter comparison plot
html 7582c67 pat-s 2019-08-31 Build site.
html abd531f pat-s 2019-08-31 Build site.
Rmd 9117eee pat-s 2019-08-31 wflow_publish(knitr_in(“analysis/eval-performance.Rmd”), view = FALSE,
Rmd 5b629cb pat-s 2019-08-19 add new tasks to performance eval report
html 1ec8768 pat-s 2019-08-17 Build site.
html df85aba pat-s 2019-07-12 Build site.
html 3a44a95 pat-s 2019-07-10 Build site.
html c238ce4 pat-s 2019-07-10 Build site.
Rmd e98cb01 pat-s 2019-07-10 wflow_publish(knitr_in(“analysis/eval-performance.Rmd”), view = FALSE,
Rmd 24e318f pat-s 2019-07-01 update reports
Rmd ca5c5bc pat-s 2019-06-28 add eval-performance report

Last update:

[1] "Thu Mar  5 10:34:01 2020"

(Table) T1 All leaner/filter/task combinations ordered by performance.

Overall leaderboard across all settings, sorted descending by performance.

Task

Model

Filter

RMSE

SE

HR-NRI

SVM

Car

27.983

19.193

HR-NRI-VI

SVM

Relief

28.049

19.124

HR-NRI

SVM

Relief

28.117

19.125

HR

SVM

Car

28.122

19.123

HR

SVM

Info Gain

28.122

19.123

VI

SVM

Relief

28.122

19.113

HR

SVM

CMIM

28.122

19.123

NRI-VI

SVM

PCA

28.122

19.123

HR-NRI-VI

SVM

PCA

28.122

19.123

VI

SVM

No Filter

28.122

19.123

HR-NRI

SVM

No Filter

28.122

19.123

HR-NRI-VI

SVM

No Filter

28.122

19.123

NRI-VI

SVM

Car

28.123

19.123

HR

SVM

Relief

28.123

19.122

HR

SVM

Pearson

28.123

19.123

HR

SVM

Borda

28.123

19.123

NRI

SVM

PCA

28.123

19.123

HR-NRI

SVM

PCA

28.123

19.123

HR

SVM

No Filter

28.123

19.123

NRI

SVM

No Filter

28.123

19.123

NRI-VI

SVM

No Filter

28.123

19.123

HR

SVM

PCA

28.125

19.122

VI

SVM

Info Gain

28.126

19.123

VI

SVM

MRMR

28.130

19.114

HR-NRI-VI

SVM

CMIM

28.130

19.114

VI

SVM

Borda

28.135

19.146

NRI-VI

SVM

Borda

28.140

19.099

VI

SVM

PCA

28.143

19.093

NRI-VI

SVM

MRMR

28.144

19.142

HR

SVM

MRMR

28.153

19.123

HR-NRI-VI

SVM

Info Gain

28.162

19.077

HR-NRI-VI

SVM

MRMR

28.163

19.051

VI

SVM

Pearson

28.171

19.212

NRI

SVM

Pearson

28.186

19.121

NRI

SVM

Relief

28.188

19.022

VI

SVM

CMIM

28.188

19.200

NRI

SVM

Borda

28.198

19.024

NRI-VI

SVM

Pearson

28.209

19.193

VI

SVM

Car

28.212

19.109

HR-NRI-VI

SVM

Pearson

28.227

19.169

HR-NRI

SVM

Info Gain

28.238

18.987

HR-NRI-VI

SVM

Car

28.280

18.919

NRI

SVM

CMIM

28.300

18.889

HR-NRI

SVM

Borda

28.307

18.912

NRI-VI

SVM

CMIM

28.348

18.826

HR-NRI

SVM

MRMR

28.511

18.610

NRI

SVM

MRMR

28.517

18.607

NRI-VI

SVM

Relief

28.622

18.824

HR-NRI

SVM

CMIM

28.683

18.395

HR-NRI-VI

SVM

Borda

28.914

18.109

NRI

SVM

Car

28.947

18.602

NRI-VI

SVM

Info Gain

29.158

17.712

HR-NRI

SVM

Pearson

30.978

18.394

HR-NRI

RF

Car

31.077

17.093

NRI-VI

XGBOOST

Relief

32.103

17.730

HR-NRI-VI

XGBOOST

MRMR

32.313

17.381

NRI

SVM

Info Gain

32.354

18.629

HR-NRI-VI

RF

Car

32.451

15.894

NRI

XGBOOST

No Filter

32.556

16.516

HR-NRI

XGBOOST

Relief

32.566

15.251

NRI-VI

RF

PCA

32.708

16.423

NRI

Ridge-CV

No Filter

32.974

7.237

HR-NRI-VI

XGBOOST

CMIM

33.084

15.219

NRI

RF

PCA

33.311

15.957

HR-NRI-VI

RF

PCA

33.370

15.964

HR-NRI

RF

PCA

33.598

15.604

HR-NRI-VI

XGBOOST

Car

33.692

14.135

HR-NRI-VI

XGBOOST

Info Gain

33.772

15.459

NRI

XGBOOST

Borda

34.012

15.435

HR-NRI

RF

Borda

34.077

14.375

NRI-VI

XGBOOST

CMIM

34.423

13.880

HR

RF

Car

34.467

12.604

HR-NRI

XGBOOST

Car

34.537

15.019

NRI

XGBOOST

Car

34.546

13.779

NRI

XGBOOST

Pearson

34.612

14.619

NRI-VI

XGBOOST

Info Gain

34.690

15.672

NRI-VI

RF

Car

34.813

13.146

HR-NRI-VI

XGBOOST

Borda

35.024

14.277

NRI-VI

XGBOOST

Borda

35.085

15.110

HR

XGBOOST

Car

35.094

13.624

NRI

RF

MRMR

35.109

13.318

NRI

RF

Car

35.182

12.041

NRI

RF

Borda

35.182

12.301

HR-NRI-VI

RF

MRMR

35.230

13.808

NRI-VI

XGBOOST

Car

35.238

13.304

HR-NRI

XGBOOST

Pearson

35.262

13.043

HR

RF

Info Gain

35.271

12.755

HR-NRI

XGBOOST

CMIM

35.315

13.794

NRI-VI

RF

CMIM

35.330

14.913

HR-NRI

XGBOOST

Info Gain

35.367

14.339

HR-NRI-VI

XGBOOST

Pearson

35.370

14.406

NRI-VI

RF

MRMR

35.511

14.834

HR-NRI-VI

RF

Borda

35.527

13.643

NRI

XGBOOST

CMIM

35.535

13.654

HR-NRI

RF

Pearson

35.556

13.076

HR-NRI-VI

XGBOOST

Relief

35.556

14.832

NRI

RF

CMIM

35.629

13.644

NRI-VI

XGBOOST

Pearson

35.660

13.066

HR-NRI

XGBOOST

Borda

35.662

13.289

HR-NRI

RF

MRMR

35.697

14.425

NRI

XGBOOST

Info Gain

35.728

14.460

HR-NRI-VI

XGBOOST

No Filter

35.762

12.884

HR-NRI-VI

RF

No Filter

35.867

12.747

NRI

RF

Info Gain

35.980

13.862

HR-NRI-VI

RF

CMIM

36.087

14.219

NRI-VI

RF

Borda

36.183

13.091

HR-NRI

XGBOOST

No Filter

36.187

10.588

HR-NRI

RF

No Filter

36.211

12.830

NRI-VI

RF

No Filter

36.344

12.844

NRI

RF

No Filter

36.389

12.483

NRI-VI

RF

Info Gain

36.406

13.666

HR-NRI

RF

Info Gain

36.671

13.625

NRI-VI

XGBOOST

MRMR

36.700

13.207

HR-NRI

XGBOOST

MRMR

36.700

11.892

NRI

XGBOOST

MRMR

36.731

11.896

HR-NRI

RF

CMIM

36.741

13.790

NRI-VI

RF

Pearson

36.755

13.206

HR-NRI-VI

RF

Info Gain

36.813

13.486

HR-NRI-VI

RF

Pearson

36.837

13.846

NRI

RF

Pearson

36.842

13.567

HR

RF

PCA

37.096

12.302

NRI-VI

XGBOOST

No Filter

37.143

12.421

HR

XGBOOST

Pearson

37.239

14.416

HR

XGBOOST

Relief

37.307

10.286

HR

RF

Borda

37.342

13.099

HR

RF

Pearson

37.351

13.852

NRI

XGBOOST

Relief

37.539

10.630

HR

XGBOOST

MRMR

37.774

12.026

VI

RF

MRMR

38.320

11.654

HR-NRI-VI

RF

Relief

38.450

8.774

HR-NRI

RF

Relief

38.598

8.998

NRI-VI

RF

Relief

38.621

8.764

VI

RF

No Filter

38.698

10.946

VI

RF

Relief

38.879

11.204

VI

RF

Borda

38.974

11.901

NRI

RF

Relief

38.985

8.031

HR

XGBOOST

Info Gain

39.214

12.433

HR

RF

CMIM

39.437

8.018

VI

RF

Car

39.901

12.918

VI

XGBOOST

Relief

39.903

9.438

VI

RF

Info Gain

39.907

9.395

VI

RF

CMIM

40.046

9.784

VI

RF

Pearson

40.247

9.882

HR-NRI

Ridge-CV

No Filter

40.271

10.513

VI

XGBOOST

Info Gain

40.301

10.101

VI

XGBOOST

Pearson

40.361

9.898

HR

Ridge-CV

No Filter

40.520

11.207

HR-NRI-VI

XGBOOST

PCA

40.535

11.371

VI

RF

PCA

40.761

8.516

NRI

XGBOOST

PCA

40.797

11.785

HR

XGBOOST

CMIM

40.960

9.937

HR-NRI

XGBOOST

PCA

40.974

10.499

HR

RF

No Filter

41.067

8.444

HR

RF

Relief

41.302

10.593

VI

XGBOOST

CMIM

41.639

10.152

HR

XGBOOST

Borda

42.073

14.376

VI

XGBOOST

PCA

42.115

7.037

HR

RF

MRMR

42.423

7.454

HR

XGBOOST

PCA

43.129

7.629

NRI-VI

XGBOOST

PCA

43.312

8.997

VI

XGBOOST

No Filter

44.444

8.759

HR

Lasso-CV

No Filter

45.960

18.350

NRI

Lasso-CV

No Filter

46.331

20.036

VI

XGBOOST

Borda

46.502

4.109

VI

XGBOOST

MRMR

46.520

5.598

HR-NRI

Lasso-CV

No Filter

46.653

20.279

VI

XGBOOST

Car

46.682

7.759

HR

XGBOOST

No Filter

49.382

1.358

NRI-VI

Lasso-CV

No Filter

54.497

24.638

HR-NRI-VI

Lasso-CV

No Filter

54.497

24.638

VI

Lasso-CV

No Filter

55.325

23.009

HR-NRI-VI

Ridge-CV

No Filter

9230336.565

18460539.886

NRI-VI

Ridge-CV

No Filter

9409773.079

18819412.914

VI

Ridge-CV

No Filter

11337132.755

22674132.319

(Table) T2 Best learner/filter/task combination

Learners: On which task and using which filter did every learner score their best result on?

*CV: L2 penalized regression using the internal 10-fold CV tuning of the glmnet package

*MBO: L2 penalized regression using using MBO for hyperparameter optimization.

Task

Model

Filter

RMSE

SE

HR-NRI

SVM

Car

27.983

19.193

HR-NRI

RF

Car

31.077

17.093

NRI-VI

XGBOOST

Relief

32.103

17.730

NRI

Ridge-CV

No Filter

32.974

7.237

HR

Lasso-CV

No Filter

45.960

18.350

(Plot) P1 Best learner/filter combs for all tasks

Version Author Date
274a918 pat-s 2020-02-25
b25e779 pat-s 2020-01-10
7f9507f pat-s 2019-12-10
482a158 pat-s 2019-11-01
becf5ea pat-s 2019-11-01
bd7c7f5 pat-s 2019-10-31
62ff96f pat-s 2019-10-07
a947654 pat-s 2019-10-02
49da171 pat-s 2019-09-22
41aae14 pat-s 2019-09-12
b181c52 pat-s 2019-09-02
8e7e4fe pat-s 2019-09-01
7582c67 pat-s 2019-08-31
abd531f pat-s 2019-08-31

(Plot) P2 Scatterplots of filter methods vs. no filter for each learner and task

Showing the final effect of applying feature selection to a learner for each task. All filters are colored in the same way whereas using “no filter” appears in a different color.

Version Author Date
274a918 pat-s 2020-02-25
b25e779 pat-s 2020-01-10
7f9507f pat-s 2019-12-10

(Plot) P3 Scatterplots of filter methods vs. Borda for each learner and task

Showing the final effect of applying feature selection to a learner for each task. All filters are summarized into a a single color whereas the “Borda” filter appears in its own color.

Version Author Date
776b35f pat-s 2020-03-03
274a918 pat-s 2020-02-25
b25e779 pat-s 2020-01-10
7f9507f pat-s 2019-12-10

R version 3.6.1 (2019-07-05)
Platform: x86_64-pc-linux-gnu (64-bit)
Running under: CentOS Linux 7 (Core)

Matrix products: default
BLAS:   /opt/spack/opt/spack/linux-centos7-x86_64/gcc-9.2.0/r-3.6.1-j25wr6zcofibs2zfjwg37357rjj26lqb/rlib/R/lib/libRblas.so
LAPACK: /opt/spack/opt/spack/linux-centos7-x86_64/gcc-9.2.0/r-3.6.1-j25wr6zcofibs2zfjwg37357rjj26lqb/rlib/R/lib/libRlapack.so

locale:
 [1] LC_CTYPE=en_US.UTF-8       LC_NUMERIC=C              
 [3] LC_TIME=en_US.UTF-8        LC_COLLATE=en_US.UTF-8    
 [5] LC_MONETARY=en_US.UTF-8    LC_MESSAGES=en_US.UTF-8   
 [7] LC_PAPER=en_US.UTF-8       LC_NAME=C                 
 [9] LC_ADDRESS=C               LC_TELEPHONE=C            
[11] LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C       

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base     

other attached packages:
 [1] forcats_0.4.0     dplyr_0.8.3       mlr_2.17.0.9001  
 [4] ParamHelpers_1.12 here_0.1          ggpubr_0.1.6     
 [7] magrittr_1.5      ggrepel_0.8.0     ggsci_2.9        
[10] ggbeeswarm_0.7.0  ggplot2_3.2.1     flextable_0.5.8  
[13] xtable_1.8-3      tidyselect_0.2.5  drake_7.10.0     

loaded via a namespace (and not attached):
 [1] fs_1.3.1           filelock_1.0.2     RColorBrewer_1.1-2
 [4] httr_1.4.0         rprojroot_1.3-2    tools_3.6.1       
 [7] backports_1.1.5    R6_2.4.1           vipor_0.4.5       
[10] lazyeval_0.2.1     colorspace_1.4-0   withr_2.1.2       
[13] mco_1.0-15.1       compiler_3.6.1     git2r_0.26.1      
[16] parallelMap_1.4    xml2_1.2.2         plotly_4.8.0      
[19] officer_0.3.7      labeling_0.3       scales_1.0.0      
[22] checkmate_1.9.1    plot3D_1.1.1       systemfonts_0.1.1 
[25] stringr_1.4.0      digest_0.6.23      txtq_0.1.4        
[28] rmarkdown_1.13     R.utils_2.8.0      smoof_1.5.1       
[31] base64enc_0.1-3    pkgconfig_2.0.3    htmltools_0.3.6   
[34] lhs_1.0.1          htmlwidgets_1.3    rlang_0.4.4       
[37] BBmisc_1.11        mlrMBO_1.1.2       jsonlite_1.6      
[40] zip_2.0.4          R.oo_1.23.0        Matrix_1.2-15     
[43] Rcpp_1.0.3         munsell_0.5.0      gdtools_0.2.1     
[46] lifecycle_0.1.0    R.methodsS3_1.7.1  stringi_1.3.1     
[49] whisker_0.3-2      yaml_2.2.0         storr_1.2.1       
[52] RJSONIO_1.3-1.1    grid_3.6.1         misc3d_0.8-4      
[55] parallel_3.6.1     promises_1.0.1     crayon_1.3.4      
[58] lattice_0.20-38    splines_3.6.1      zeallot_0.1.0     
[61] knitr_1.23         pillar_1.4.3       igraph_1.2.4.1    
[64] uuid_0.1-2         base64url_1.4      fastmatch_1.1-0   
[67] glue_1.3.1         evaluate_0.13      data.table_1.12.6 
[70] vctrs_0.2.1        httpuv_1.4.5.1     gtable_0.2.0      
[73] purrr_0.3.3        tidyr_1.0.0        assertthat_0.2.1  
[76] xfun_0.5           later_1.0.0        survival_2.43-3   
[79] viridisLite_0.3.0  tibble_2.1.3       beeswarm_0.2.3    
[82] workflowr_1.6.0    ellipsis_0.3.0