Evaluation of performances

(Table) T1 All leaner/filter/task combinations ordered by performance.
(Table) T2 Best learner/filter/task combination
(Table) T3 All leaner/filter/task combinations ordered descending by performance.
(Plot) P1 Best learner/filter combs for all tasks
(Plot) P2 Scatterplots of filter methods vs. no filter for each learner and task
(Plot) P3 Scatterplots of filter methods vs. Borda for each learner and task
(Table) T4 Number of features selected during tuning

Last updated: 2021-02-20

Checks: 7 0

Knit directory: 2019-feature-selection/

This reproducible R Markdown analysis was created with workflowr (version 1.6.2). The Checks tab describes the reproducibility checks that were applied when the results were created. The Past versions tab lists the development history.

R Markdown file: up-to-date

Great! Since the R Markdown file has been committed to the Git repository, you know the exact version of the code that produced these results.

Environment: empty

Great job! The global environment was empty. Objects defined in the global environment can affect the analysis in your R Markdown file in unknown ways. For reproduciblity it’s best to always run the code in an empty environment.

Seed: set.seed(20190522)

The command set.seed(20190522) was run prior to running the code in the R Markdown file. Setting a seed ensures that any results that rely on randomness, e.g. subsampling or permutations, are reproducible.

Session information: recorded

Great job! Recording the operating system, R version, and package versions is critical for reproducibility.

Cache: none

Nice! There were no cached chunks for this analysis, so you can be confident that you successfully produced the results during this run.

File paths: relative

Great job! Using relative paths to the files within your workflowr project makes it easier to run your code on other machines.

Repository version: 6b712c3

Great! You are using Git for version control. Tracking code development and connecting the code version to the results is critical for reproducibility.

The results in this page were generated with repository version 6b712c3. See the Past versions tab to see a history of the changes made to the R Markdown and HTML files.

Note that you need to be careful to ensure that all relevant files for the analysis have been committed to Git prior to generating the results (you can use wflow_publish or wflow_git_commit). workflowr only checks the R Markdown file, but you know if there are other scripts or data files that it depends on. Below is the status of the Git repository when the results were generated:


Ignored files:
    Ignored:    .Rhistory
    Ignored:    .Rproj.user/
    Ignored:    .Ruserdata/
    Ignored:    .drake/
    Ignored:    .vscode/
    Ignored:    analysis/rosm.cache/
    Ignored:    data/
    Ignored:    inst/Benchmark for Filter Methods for Feature Selection in High-Dimensional  Classification Data.pdf
    Ignored:    inst/study-area-map/._study-area.qgs
    Ignored:    inst/study-area-map/study-area.qgs~
    Ignored:    log/
    Ignored:    renv/library/
    Ignored:    renv/staging/
    Ignored:    reviews/
    Ignored:    rosm.cache/

Unstaged changes:
    Modified:   _drake.R
    Modified:   analysis/feature-importance.Rmd
    Modified:   code/060-modeling/paper/09-feature-importance.R
    Modified:   code/090-reports.R

Note that any generated files, e.g. HTML, png, CSS, etc., are not included in this status report because it is ok for generated content to have uncommitted changes.

These are the previous versions of the repository in which changes were made to the R Markdown (analysis/eval-performance.Rmd) and HTML (docs/eval-performance.html) files. If you’ve configured a remote Git repository (see ?wflow_git_remote), click on the hyperlinks in the table below to view the files as they were in that past version.

File	Version	Author	Date	Message
html	a953299	pat-s	2020-08-12	Build site.
html	8b5e422	pat-s	2020-08-05	Build site.
html	6f88553	pat-s	2020-06-21	Build site.
Rmd	c94e7fc	pat-s	2020-06-21	wflow_publish(knitr_in(“analysis/eval-performance.Rmd”), view = FALSE,
html	9a54378	pat-s	2020-05-29	Build site.
Rmd	679a87a	pat-s	2020-05-29	wflow_publish(knitr_in(“analysis/eval-performance.Rmd”), view = FALSE,
html	6eeb69b	pat-s	2020-05-06	Build site.
Rmd	fc28c0f	pat-s	2020-05-06	wflow_publish(knitr_in(“analysis/eval-performance.Rmd”), view = FALSE,
html	a2a6562	pat-s	2020-05-03	Build site.
Rmd	b9d07a0	pat-s	2020-05-03	wflow_publish(knitr_in(“analysis/eval-performance.Rmd”), view = FALSE,
html	f7737ce	pat-s	2020-04-30	Build site.
Rmd	576a62f	pat-s	2020-04-30	wflow_publish(knitr_in(“analysis/eval-performance.Rmd”), view = FALSE,
html	fcabe1a	pat-s	2020-04-30	Build site.
Rmd	3aa1860	pat-s	2020-04-30	wflow_publish(knitr_in(“analysis/eval-performance.Rmd”), view = FALSE,
html	bf3eee4	pat-s	2020-04-30	Build site.
Rmd	3a4a1ae	pat-s	2020-04-30	wflow_publish(knitr_in(“analysis/eval-performance.Rmd”), view = FALSE,
html	ebf4147	pat-s	2020-04-30	Build site.
Rmd	96f47c3	pat-s	2020-04-30	wflow_publish(knitr_in(“analysis/eval-performance.Rmd”), view = FALSE,
html	1c1c5f4	pat-s	2020-04-30	Build site.
Rmd	b1d64ea	pat-s	2020-04-30	wflow_publish(knitr_in(“analysis/eval-performance.Rmd”), view = FALSE,
html	9ea0dff	pat-s	2020-04-29	Build site.
Rmd	ba54b0d	pat-s	2020-04-29	wflow_publish(knitr_in(“analysis/eval-performance.Rmd”), view = FALSE,
html	3bdb92f	pat-s	2020-04-29	Build site.
Rmd	1b98cdd	pat-s	2020-04-29	wflow_publish(knitr_in(“analysis/eval-performance.Rmd”), view = FALSE,
html	01b8c74	pat-s	2020-04-29	Build site.
Rmd	c79c23d	pat-s	2020-04-29	wflow_publish(knitr_in(“analysis/eval-performance.Rmd”), view = FALSE,
html	e20d376	pat-s	2020-04-29	Build site.
Rmd	4c8c633	pat-s	2020-04-29	wflow_publish(knitr_in(“analysis/eval-performance.Rmd”), view = FALSE,
html	72c4b82	pat-s	2020-04-19	Build site.
html	89adb44	pat-s	2020-04-18	Build site.
Rmd	6b863d0	pat-s	2020-04-18	wflow_publish(knitr_in(“analysis/eval-performance.Rmd”), view = FALSE,
html	07fb043	pat-s	2020-04-18	Build site.
Rmd	abc2b51	pat-s	2020-04-18	wflow_publish(knitr_in(“analysis/eval-performance.Rmd”), view = FALSE,
html	e670acc	pat-s	2020-04-18	Build site.
Rmd	1fc0c78	pat-s	2020-04-18	wflow_publish(knitr_in(“analysis/eval-performance.Rmd”), view = FALSE,
html	1d1bee4	pat-s	2020-04-18	Build site.
Rmd	f3103fc	pat-s	2020-04-18	wflow_publish(knitr_in(“analysis/eval-performance.Rmd”), view = FALSE,
html	4982c76	pat-s	2020-04-18	Build site.
Rmd	b4fda43	pat-s	2020-04-18	wflow_publish(knitr_in(“analysis/eval-performance.Rmd”), view = FALSE,
html	6c42b7c	pat-s	2020-04-18	Build site.
Rmd	8c30483	pat-s	2020-04-18	wflow_publish(knitr_in(“analysis/eval-performance.Rmd”), view = FALSE,
html	544e288	pat-s	2020-04-12	Build site.
Rmd	ce72956	pat-s	2020-04-12	wflow_publish(knitr_in(“analysis/eval-performance.Rmd”), view = FALSE,
html	6bded83	pat-s	2020-03-05	Build site.
Rmd	21453cb	pat-s	2020-03-05	wflow_publish(knitr_in(“analysis/eval-performance.Rmd”), view = FALSE,
html	f59d02a	pat-s	2020-03-05	Build site.
Rmd	167fdbc	pat-s	2020-03-05	wflow_publish(knitr_in(“analysis/eval-performance.Rmd”), view = FALSE,
html	2ee982d	pat-s	2020-03-05	Build site.
Rmd	d487d51	pat-s	2020-03-05	wflow_publish(knitr_in(“analysis/eval-performance.Rmd”), view = FALSE,
html	776b35f	pat-s	2020-03-03	Build site.
Rmd	d0c645a	pat-s	2020-03-03	wflow_publish(knitr_in(“analysis/eval-performance.Rmd”), view = FALSE,
html	274a918	pat-s	2020-02-25	Build site.
Rmd	2e632e1	pat-s	2020-02-25	wflow_publish(knitr_in(“analysis/eval-performance.Rmd”), view = FALSE,
Rmd	a53ea68	pat-s	2020-02-24	add prefixes to plots and tables
Rmd	379bd5d	pat-s	2020-02-24	wflow_publish(knitr_in(“analysis/eval-performance.Rmd”), view = FALSE,
html	f680221	pat-s	2020-01-15	Build site.
html	b25e779	pat-s	2020-01-10	Build site.
Rmd	db0baaa	pat-s	2020-01-10	wflow_publish(knitr_in(“analysis/eval-performance.Rmd”), view = FALSE,
html	7f9507f	pat-s	2019-12-10	Build site.
Rmd	951e98c	pat-s	2019-12-10	wflow_publish(knitr_in(“analysis/eval-performance.Rmd”), view = FALSE, verbose =
Rmd	b27c623	pat-s	2019-12-09	Standard Error -> SE
Rmd	559a59d	pat-s	2019-12-09	add scatterplots to vis BM perf
html	482a158	pat-s	2019-11-01	Build site.
html	becf5ea	pat-s	2019-11-01	Build site.
html	bd7c7f5	pat-s	2019-10-31	Build site.
html	62ff96f	pat-s	2019-10-07	Build site.
html	a947654	pat-s	2019-10-02	Build site.
html	49da171	pat-s	2019-09-22	Build site.
html	c6317a8	pat-s	2019-09-19	Build site.
Rmd	d7c72a8	pat-s	2019-09-19	wflow_publish(knitr_in(“analysis/eval-performance.Rmd”), view = FALSE,
html	7fd40ca	pat-s	2019-09-18	Build site.
Rmd	44ff84b	pat-s	2019-09-18	wflow_publish(knitr_in(“analysis/eval-performance.Rmd”), view = FALSE,
html	41aae14	pat-s	2019-09-12	Build site.
html	ff340b8	pat-s	2019-09-03	Build site.
Rmd	a524819	pat-s	2019-09-03	wflow_publish(knitr_in(“analysis/eval-performance.Rmd”), view = FALSE,
html	b181c52	pat-s	2019-09-02	Build site.
Rmd	cf6e820	pat-s	2019-09-02	wflow_publish(“analysis/eval-performance.Rmd”)
Rmd	1bec10d	pat-s	2019-09-01	no timestamps in latex tables
html	4e363ac	pat-s	2019-09-01	Build site.
Rmd	518d0cb	pat-s	2019-09-01	style files using tidyverse style
html	8e7e4fe	pat-s	2019-09-01	Build site.
Rmd	8941bca	pat-s	2019-09-01	wflow_publish(knitr_in(“analysis/eval-performance.Rmd”), view = FALSE,
Rmd	297ed93	pat-s	2019-08-31	add filter vs no filter comparison plot
html	7582c67	pat-s	2019-08-31	Build site.
html	abd531f	pat-s	2019-08-31	Build site.
Rmd	9117eee	pat-s	2019-08-31	wflow_publish(knitr_in(“analysis/eval-performance.Rmd”), view = FALSE,
Rmd	5b629cb	pat-s	2019-08-19	add new tasks to performance eval report
html	1ec8768	pat-s	2019-08-17	Build site.
html	df85aba	pat-s	2019-07-12	Build site.
html	3a44a95	pat-s	2019-07-10	Build site.
html	c238ce4	pat-s	2019-07-10	Build site.
Rmd	e98cb01	pat-s	2019-07-10	wflow_publish(knitr_in(“analysis/eval-performance.Rmd”), view = FALSE,
Rmd	24e318f	pat-s	2019-07-01	update reports
Rmd	ca5c5bc	pat-s	2019-06-28	add eval-performance report

Last update:

[1] "Sat Feb 20 18:12:40 2021"

Fold performances of “SVM MBO No Filter” on the HR Task

Fold 1: Luiando
Fold 2: Laukiz1
Fold 3: Laukiz2
Fold 4: Oiartzun

Plot	RMSE	Test Plot
1	28.05	Laukiz1
2	21.17	Oiartzun
3	54.26	Luiando
4	9.00	Laukiz2

(Table) T1 All leaner/filter/task combinations ordered by performance.

Overall leaderboard across all settings, sorted ascending by performance.

Task	Model	Filter	rsq	RMSE	SE
NRI	SVM	Info Gain	-0.07425239	28.027	19.128
HR-NRI-VI	SVM	Car	-9.30552337	28.108	19.140
HR-NRI	SVM	MRMR	-9.28913635	28.117	19.113
HR-NRI-VI	SVM	MRMR	-9.28913635	28.117	19.112
HR-NRI	SVM	CMIM	-9.29805138	28.122	19.124
HR	SVM	Info Gain	-3.04212685	28.122	19.123
HR	SVM	No Filter	-3.04212680	28.122	19.123
NRI-VI	SVM	No Filter	-0.15114809	28.122	19.123
HR-NRI	SVM	No Filter	-0.15114809	28.122	19.123
HR-NRI-VI	SVM	No Filter	-3.04212680	28.122	19.123
NRI	SVM	PCA	-9.29805049	28.122	19.123
HR-NRI-VI	SVM	PCA	-3.04212680	28.122	19.123
HR	SVM	Borda	-0.17328117	28.123	19.123
HR	SVM	Car	-9.29808592	28.123	19.123
NRI-VI	SVM	Car	-9.29805138	28.123	19.123
HR	SVM	CMIM	-0.15114809	28.123	19.123
HR-NRI-VI	SVM	CMIM	-3.04212680	28.123	19.122
VI	SVM	No Filter	-0.15114809	28.123	19.123
NRI	SVM	No Filter	-9.29798156	28.123	19.123
NRI-VI	SVM	PCA	-0.17351412	28.123	19.123
HR-NRI	SVM	PCA	-9.29806571	28.123	19.123
HR	SVM	Pearson	-3.04333963	28.124	19.123
HR	SVM	PCA	-3.04424994	28.125	19.122
NRI-VI	SVM	CMIM	-0.15114809	28.128	19.117
VI	SVM	MRMR	-0.17854498	28.128	19.117
NRI	SVM	Borda	-9.29800198	28.133	19.115
HR-NRI-VI	SVM	Relief	-2.79608424	28.133	18.701
HR-NRI-VI	SVM	Info Gain	-0.15114809	28.134	19.130
NRI-VI	SVM	Borda	-9.30105597	28.138	19.109
NRI-VI	SVM	MRMR	-9.32088441	28.138	19.150
VI	SVM	PCA	-9.29806343	28.144	19.091
HR-NRI-VI	SVM	Borda	-3.04212680	28.146	19.076
VI	SVM	Info Gain	-0.15114808	28.149	19.166
HR-NRI-VI	SVM	Pearson	-9.27814852	28.152	19.042
HR	SVM	MRMR	-3.07724341	28.153	19.123
VI	SVM	Borda	-3.04212704	28.154	19.180
VI	SVM	CMIM	-0.15114809	28.167	19.216
HR-NRI	SVM	Info Gain	-0.21285247	28.180	19.110
VI	SVM	Pearson	-0.17328169	28.180	19.228
HR-NRI	SVM	Borda	-0.15114809	28.181	19.230
HR-NRI	SVM	Relief	-0.24786309	28.190	19.031
NRI	SVM	Pearson	-3.05318431	28.203	19.099
VI	SVM	Car	-9.42434769	28.264	19.197
NRI-VI	SVM	Relief	-3.04212680	28.278	19.336
NRI	SVM	Relief	-3.04245372	28.312	18.873
NRI-VI	SVM	Pearson	-0.15114809	28.336	19.038
HR	SVM	Relief	-3.34781578	28.383	19.129
NRI	SVM	MRMR	-0.15114808	28.517	18.607
NRI	SVM	CMIM	-3.04244840	28.657	18.428
HR-NRI	SVM	Pearson	-3.04212680	29.245	17.749
HR-NRI	SVM	Car	-1.10714340	30.000	18.561
NRI	SVM	Car	-3.04628885	30.160	18.151
HR-NRI	XGBoost	Car	-0.29505854	31.024	17.629
VI	Lasso-MBO	No Filter	-5.96803654	31.108	14.907
NRI	Lasso-MBO	No Filter	-5.96599316	31.165	15.025
NRI	Ridge-MBO	No Filter	-0.18019889	31.165	15.025
NRI-VI	Lasso-MBO	No Filter	-5.96802951	31.201	15.089
HR-NRI	Lasso-MBO	No Filter	-5.96802951	31.201	15.089
HR-NRI-VI	Lasso-MBO	No Filter	-0.18035181	31.201	15.089
NRI	RF	Car	-5.96599316	32.141	15.968
HR-NRI-VI	XGBoost	Car	-3.45834660	32.374	16.364
NRI-VI	RF	PCA	-3.81019111	32.562	16.700
VI	SVM	Relief	-9.29963762	32.581	19.038
NRI	XGBoost	Pearson	-1.96017124	32.948	14.845
NRI-VI	XGBoost	Borda	-4.29694185	33.134	14.724
NRI	RF	PCA	-4.04443059	33.144	15.999
HR-NRI	RF	PCA	-5.41645275	33.228	15.877
NRI-VI	XGBoost	CMIM	-2.24351526	33.282	15.662
NRI-VI	SVM	Info Gain	-3.00796288	33.422	17.840
HR-NRI-VI	XGBoost	No Filter	-4.15731221	33.542	14.849
HR-NRI-VI	RF	PCA	-7.58348469	33.612	15.692
NRI-VI	RF	CMIM	-6.91388576	34.015	14.714
HR-NRI-VI	XGBoost	MRMR	-3.32541730	34.110	14.903
HR-NRI	XGBoost	Pearson	-4.27170701	34.151	14.436
HR-NRI-VI	XGBoost	CMIM	-4.47556738	34.170	15.377
NRI	XGBoost	Car	-1.92350384	34.219	15.204
HR	XGBoost	CMIM	-7.93728786	34.286	12.404
HR	RF	Relief	-5.96599316	34.386	13.554
HR-NRI	XGBoost	No Filter	-3.31998872	34.471	15.244
HR	RF	Car	-2.49138719	34.483	12.655
NRI	XGBoost	Relief	-4.63750631	34.541	14.615
NRI-VI	XGBoost	Car	-4.38817981	34.571	13.901
HR-NRI	XGBoost	CMIM	-4.27578414	34.656	15.704
HR	RF	Borda	-3.03745683	34.707	13.568
NRI-VI	RF	MRMR	-4.39118458	34.789	13.735
NRI-VI	XGBoost	Relief	-6.23724860	34.798	13.080
NRI	XGBoost	MRMR	-2.36033619	34.809	14.099
HR-NRI	XGBoost	Borda	-4.24738762	34.823	13.514
HR-NRI	RF	Relief	-4.21171189	34.996	13.235
NRI	RF	CMIM	-0.66142161	35.015	14.939
HR	XGBoost	Pearson	-4.36410646	35.186	11.679
NRI	XGBoost	No Filter	-4.55835264	35.191	12.621
HR-NRI	XGBoost	Relief	-3.08693087	35.209	12.936
HR-NRI	XGBoost	Info Gain	-4.33579272	35.219	14.268
HR-NRI-VI	XGBoost	Pearson	-3.72944139	35.240	14.642
HR	Lasso-MBO	No Filter	-3.84736108	35.260	12.242
NRI	XGBoost	Borda	-3.65217438	35.273	14.826
NRI-VI	XGBoost	Pearson	-3.50838385	35.277	12.881
NRI	RF	MRMR	-6.34647720	35.339	13.877
HR-NRI	RF	MRMR	-4.43060828	35.396	15.055
HR	Ridge-MBO	No Filter	-4.22859256	35.453	12.655
NRI-VI	XGBoost	Info Gain	-3.83280129	35.470	14.168
NRI-VI	RF	Car	-5.96599316	35.479	12.371
HR-NRI-VI	RF	Car	-4.44986608	35.493	12.386
HR-NRI	RF	Car	-3.03745683	35.517	12.415
HR-NRI-VI	XGBoost	Relief	-3.40691504	35.517	12.091
NRI-VI	RF	Borda	-0.73731759	35.605	13.668
HR-NRI-VI	RF	Borda	-0.85840746	35.617	13.109
NRI	RF	No Filter	-6.43591869	35.759	12.744
HR-NRI	RF	CMIM	-7.58866273	35.808	14.496
HR-NRI	RF	Borda	-15.15576783	35.878	13.031
NRI-VI	XGBoost	No Filter	-6.03723195	35.963	12.265
HR-NRI	RF	No Filter	-0.82551581	35.976	12.794
NRI-VI	RF	No Filter	-0.79452345	36.037	12.854
HR-NRI	XGBoost	MRMR	-6.16927339	36.045	13.112
NRI-VI	XGBoost	MRMR	-3.46742569	36.168	12.025
NRI	RF	Pearson	-15.12369006	36.186	13.613
HR	RF	Pearson	-3.70102423	36.187	12.306
HR-NRI-VI	RF	MRMR	-12.19457141	36.226	14.983
HR	XGBoost	Car	-5.96599316	36.265	15.825
NRI	XGBoost	CMIM	-4.18963784	36.398	13.210
NRI	RF	Info Gain	-0.67464489	36.423	13.659
HR	RF	Info Gain	-5.96599316	36.444	13.775
HR-NRI-VI	RF	No Filter	-0.91454459	36.504	12.513
HR-NRI-VI	XGBoost	Info Gain	-4.39723715	36.544	15.371
HR-NRI-VI	RF	CMIM	-0.75911709	36.587	14.183
HR-NRI	RF	Pearson	-0.74827939	36.616	13.906
HR	XGBoost	Info Gain	-3.03745683	36.658	14.283
NRI-VI	RF	Pearson	-0.80956674	36.696	13.420
NRI-VI	RF	Info Gain	-16.36364585	36.707	13.812
HR-NRI	XGBoost	PCA	-7.21433253	36.714	14.938
HR	XGBoost	Borda	-3.03745683	36.725	14.325
HR-NRI-VI	RF	Info Gain	-0.60073088	36.742	13.931
HR-NRI-VI	XGBoost	Borda	-4.72460042	36.757	11.934
NRI	RF	Borda	-0.85623402	36.780	13.621
HR-NRI-VI	RF	Pearson	-4.34073637	36.917	13.781
HR	RF	PCA	-4.36445338	36.986	12.140
HR-NRI	RF	Info Gain	-4.32406255	37.118	13.670
HR-NRI-VI	RF	Relief	-3.03745683	37.152	9.393
NRI	RF	Relief	-6.47745457	37.801	9.480
NRI	XGBoost	Info Gain	-5.56677675	38.193	11.611
VI	RF	No Filter	-4.01139563	38.298	11.635
VI	RF	MRMR	-4.02921417	38.509	11.254
NRI-VI	RF	Relief	-4.10203693	38.570	8.196
VI	RF	Borda	-8.06100336	38.896	11.882
VI	RF	Relief	-2.25231040	39.168	10.545
VI	XGBoost	Info Gain	-3.03745683	39.604	9.087
HR	RF	CMIM	-21.79975783	39.744	8.158
VI	RF	Info Gain	-3.03745683	39.801	9.537
HR	XGBoost	MRMR	-25.71505676	39.834	9.488
VI	RF	Car	-8.64575022	40.010	12.038
VI	RF	CMIM	-4.03745588	40.184	9.649
VI	RF	Pearson	-19.63864622	40.383	9.914
VI	XGBoost	CMIM	-4.58908033	40.718	10.508
VI	RF	PCA	-22.40728839	40.869	8.531
VI	XGBoost	No Filter	-4.32896907	41.199	8.716
VI	XGBoost	MRMR	-4.61964196	41.238	9.549
VI	XGBoost	Relief	-3.03745683	41.364	9.700
HR	XGBoost	Relief	-3.03745683	41.461	10.930
VI	XGBoost	PCA	-1.83410416	41.673	9.515
HR	RF	No Filter	-8.08450094	41.850	7.899
NRI-VI	XGBoost	PCA	-5.30002378	41.951	9.792
NRI	XGBoost	PCA	-3.57302784	42.186	9.938
HR	RF	MRMR	-23.87496754	42.437	7.705
HR	XGBoost	PCA	-25.29241540	43.466	6.662
VI	XGBoost	Pearson	-4.44716910	43.938	9.291
VI	XGBoost	Borda	-4.85233625	45.538	5.268
HR-NRI-VI	XGBoost	PCA	-7.90909703	45.744	4.695
HR	XGBoost	No Filter	-8.35454535	45.887	6.773
VI	XGBoost	Car	-9.35016686	46.176	7.076
NRI-VI	Ridge-MBO	No Filter	-7,606,649,897,414,941,696.00000000	11,658,468,597.678	23,316,937,147.742
HR-NRI-VI	Ridge-MBO	No Filter	-8,929,972,505,062,526,976.00000000	12,631,934,180.910	25,263,868,314.206
HR-NRI	Ridge-MBO	No Filter	-3.03909205	12,650,121,073.664	25,300,242,099.713
VI	Ridge-MBO	No Filter	-0.18136489	49,359,394,487.663	98,718,788,927.708

(Table) T2 Best learner/filter/task combination

Learners: On which task and using which filter did every learner score their best result on?

*CV: L2 penalized regression using the internal 10-fold CV tuning of the glmnet package

*MBO: L2 penalized regression using using MBO for hyperparameter optimization.

Task	Model	Filter	rsq	RMSE	SE
NRI	SVM	Info Gain	-0.07425239	28.027	19.128
HR-NRI	XGBoost	Car	-0.29505854	31.024	17.629
VI	Lasso-MBO	No Filter	-5.96803654	31.108	14.907
NRI	Ridge-MBO	No Filter	-0.18019889	31.165	15.025
NRI	RF	Car	-5.96599316	32.141	15.968

(Table) T3 All leaner/filter/task combinations ordered descending by performance.

Overall leaderboard across all settings, sorted descending by performance.

Task	Model	Filter	rsq	RMSE
VI	Ridge-MBO	No Filter	-0.1813649	49,359,394,487.663
HR-NRI	Ridge-MBO	No Filter	-3.0390920	12,650,121,073.664
HR-NRI-VI	Ridge-MBO	No Filter	-8,929,972,505,062,526,976.0000000	12,631,934,180.910
NRI-VI	Ridge-MBO	No Filter	-7,606,649,897,414,941,696.0000000	11,658,468,597.678
VI	XGBoost	Car	-9.3501669	46.176
HR	XGBoost	No Filter	-8.3545453	45.887
HR-NRI-VI	XGBoost	PCA	-7.9090970	45.744
VI	XGBoost	Borda	-4.8523362	45.538
VI	XGBoost	Pearson	-4.4471691	43.938
HR	XGBoost	PCA	-25.2924154	43.466

(Plot) P1 Best learner/filter combs for all tasks

Version	Author	Date
e20d376	pat-s	2020-04-29
07fb043	pat-s	2020-04-18
1d1bee4	pat-s	2020-04-18
6c42b7c	pat-s	2020-04-18
544e288	pat-s	2020-04-12
f59d02a	pat-s	2020-03-05
2ee982d	pat-s	2020-03-05
274a918	pat-s	2020-02-25
b25e779	pat-s	2020-01-10
7f9507f	pat-s	2019-12-10
482a158	pat-s	2019-11-01
becf5ea	pat-s	2019-11-01
bd7c7f5	pat-s	2019-10-31
62ff96f	pat-s	2019-10-07
a947654	pat-s	2019-10-02
49da171	pat-s	2019-09-22
41aae14	pat-s	2019-09-12
b181c52	pat-s	2019-09-02
8e7e4fe	pat-s	2019-09-01
7582c67	pat-s	2019-08-31
abd531f	pat-s	2019-08-31

(Plot) P2 Scatterplots of filter methods vs. no filter for each learner and task

Showing the final effect of applying feature selection to a learner for each task. All filters are colored in the same way whereas using “no filter” appears in a different color.

`summarise()` has grouped output by 'learner_group', 'task.id'. You can override using the `.groups` argument.

Version	Author	Date
e20d376	pat-s	2020-04-29
1d1bee4	pat-s	2020-04-18
6c42b7c	pat-s	2020-04-18
544e288	pat-s	2020-04-12
f59d02a	pat-s	2020-03-05
2ee982d	pat-s	2020-03-05
274a918	pat-s	2020-02-25
b25e779	pat-s	2020-01-10
7f9507f	pat-s	2019-12-10

(Plot) P3 Scatterplots of filter methods vs. Borda for each learner and task

Showing the final effect of applying feature selection to a learner for each task. All filters are summarized into a a single color whereas the “Borda” filter appears in its own color.

`summarise()` has grouped output by 'learner_group', 'task.id'. You can override using the `.groups` argument.

Version	Author	Date
e20d376	pat-s	2020-04-29
1d1bee4	pat-s	2020-04-18
6c42b7c	pat-s	2020-04-18
544e288	pat-s	2020-04-12
f59d02a	pat-s	2020-03-05
2ee982d	pat-s	2020-03-05
776b35f	pat-s	2020-03-03
274a918	pat-s	2020-02-25
b25e779	pat-s	2020-01-10
7f9507f	pat-s	2019-12-10

(Table) T4 Number of features selected during tuning

The model/task combinations which were selected relate to the best performance of the respective algorithm on the HR-NRI-VI task in the overall benchmark.

Learner	Test Plot	Features (\%)	\#	RMSE
\multirow{4}{*}{\specialcell{RF \\ Car}}	Laukiz1	0.00484534067	3	27.76546
	Oiartzun	0.00002368463	1	37.20174
	Luiando	0.02875193581	9	15.57394
	Laukiz2	0.04572609270	21	38.62678
\midrule\multirow{4}{*}{\specialcell{XGB \\ Borda}}	Laukiz1	0.50710147051	284	38.19389
	Oiartzun	0.73141731393	364	29.85402
	Luiando	0.94799922723	286	36.91005
	Laukiz2	0.23877169248	108	15.32781
\midrule\multirow{4}{*}{\specialcell{SVM \\ Relief}}	Laukiz1	0.19391379909	109	14.84888
	Oiartzun	0.00024613753	1	35.30642
	Luiando	0.04490753097	14	36.24539
	Laukiz2	0.00015260726	1	37.48127

Aggregated mean and standard deviation:

Learner	Mean (Features (%))	SD (Features (%))
RF Car	0.01983676	0.02134665
SVM Relief	0.05980502	0.09185636
XGBoost Borda	0.60632243	0.30404501

R version 4.0.3 (2020-10-10)
Platform: x86_64-pc-linux-gnu (64-bit)
Running under: CentOS Linux 7 (Core)

Matrix products: default
BLAS:   /opt/spack/opt/spack/linux-centos7-x86_64/gcc-9.2.0/r-4.0.3-hkh5nywkwodhye5qvukisarvhbj264ob/rlib/R/lib/libRblas.so
LAPACK: /opt/spack/opt/spack/linux-centos7-x86_64/gcc-9.2.0/r-4.0.3-hkh5nywkwodhye5qvukisarvhbj264ob/rlib/R/lib/libRlapack.so

locale:
 [1] LC_CTYPE=en_US.UTF-8       LC_NUMERIC=C              
 [3] LC_TIME=en_US.UTF-8        LC_COLLATE=en_US.UTF-8    
 [5] LC_MONETARY=en_US.UTF-8    LC_MESSAGES=en_US.UTF-8   
 [7] LC_PAPER=en_US.UTF-8       LC_NAME=C                 
 [9] LC_ADDRESS=C               LC_TELEPHONE=C            
[11] LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C       

attached base packages:
[1] stats     graphics  grDevices datasets  utils     methods   base     

other attached packages:
 [1] forcats_0.5.1     dplyr_1.0.4       mlr_2.18.0.9002   ParamHelpers_1.14
 [5] here_1.0.1        ggpubr_0.4.0      ggrepel_0.9.1     ggsci_2.9        
 [9] ggbeeswarm_0.7.0  ggplot2_3.3.3     flextable_0.6.3   xtable_1.8-4     
[13] usethis_2.0.0     magrittr_2.0.1    drake_7.13.1     

loaded via a namespace (and not attached):
  [1] colorspace_2.0-0   ggsignif_0.6.0     smoof_1.6.0.2     
  [4] ellipsis_0.3.1     rio_0.5.16         rprojroot_2.0.2   
  [7] base64enc_0.1-3    fs_1.5.0           rstudioapi_0.13   
 [10] farver_2.0.3       xml2_1.3.2         splines_4.0.3     
 [13] R.methodsS3_1.8.1  knitr_1.31         mco_1.15.6        
 [16] jsonlite_1.7.2     workflowr_1.6.2    broom_0.7.4       
 [19] R.oo_1.24.0        mlrMBO_1.1.5       compiler_4.0.3    
 [22] httr_1.4.2         backports_1.2.1    assertthat_0.2.1  
 [25] Matrix_1.3-2       lazyeval_0.2.2     cli_2.3.0         
 [28] later_1.1.0.1      htmltools_0.5.1.1  prettyunits_1.1.1 
 [31] tools_4.0.3        igraph_1.2.6       misc3d_0.9-0      
 [34] gtable_0.3.0       glue_1.4.2         fastmatch_1.1-0   
 [37] Rcpp_1.0.6         parallelMap_1.5.0  carData_3.0-4     
 [40] cellranger_1.1.0   vctrs_0.3.6        RJSONIO_1.3-1.4   
 [43] xfun_0.20          stringr_1.4.0      openxlsx_4.2.3    
 [46] lifecycle_0.2.0    renv_0.12.5-39     rstatix_0.6.0     
 [49] scales_1.1.1       hms_1.0.0          promises_1.1.1    
 [52] parallel_4.0.3     plot3D_1.3         RColorBrewer_1.1-2
 [55] BBmisc_1.11        yaml_2.2.1         curl_4.3          
 [58] gdtools_0.2.3      DiceKriging_1.5.8  stringi_1.5.3     
 [61] highr_0.8          checkmate_2.0.0    lhs_1.1.1         
 [64] filelock_1.0.2     zip_2.1.1          storr_1.2.5       
 [67] rlang_0.4.10       pkgconfig_2.0.3    systemfonts_1.0.0 
 [70] evaluate_0.14      lattice_0.20-41    purrr_0.3.4       
 [73] labeling_0.4.2     htmlwidgets_1.5.3  tidyselect_1.1.0  
 [76] R6_2.5.0           generics_0.1.0     base64url_1.4     
 [79] txtq_0.2.3         pillar_1.4.7       haven_2.3.1       
 [82] whisker_0.4        foreign_0.8-81     withr_2.4.1       
 [85] abind_1.4-5        survival_3.2-7     tibble_3.0.6      
 [88] crayon_1.4.0       car_3.0-10         uuid_0.1-4        
 [91] plotly_4.9.3       rmarkdown_2.6      officer_0.3.16    
 [94] progress_1.2.2     readxl_1.3.1       grid_4.0.3        
 [97] data.table_1.13.6  git2r_0.28.0       digest_0.6.27     
[100] tidyr_1.1.2        httpuv_1.5.5       R.utils_2.10.1    
[103] munsell_0.5.0      beeswarm_0.2.3     viridisLite_0.3.0 
[106] vipor_0.4.5        tcltk_4.0.3