Evaluation of performances

(Table) T1 All leaner/filter/task combinations ordered by performance.
(Table) T2 Best learner/filter/task combination
(Table) T3 All leaner/filter/task combinations ordered descending by performance.
(Plot) P1 Best learner/filter combs for all tasks
(Plot) P2 Scatterplots of filter methods vs. no filter for each learner and task
(Plot) P3 Scatterplots of filter methods vs. Borda for each learner and task
(Table) T4 Number of features selected during tuning

Last updated: 2020-08-12

Checks: 7 0

Knit directory: 2019-feature-selection/

This reproducible R Markdown analysis was created with workflowr (version 1.6.1). The Checks tab describes the reproducibility checks that were applied when the results were created. The Past versions tab lists the development history.

R Markdown file: up-to-date

Great! Since the R Markdown file has been committed to the Git repository, you know the exact version of the code that produced these results.

Environment: empty

Great job! The global environment was empty. Objects defined in the global environment can affect the analysis in your R Markdown file in unknown ways. For reproduciblity it’s best to always run the code in an empty environment.

Seed: set.seed(20190522)

The command set.seed(20190522) was run prior to running the code in the R Markdown file. Setting a seed ensures that any results that rely on randomness, e.g. subsampling or permutations, are reproducible.

Session information: recorded

Great job! Recording the operating system, R version, and package versions is critical for reproducibility.

Cache: none

Nice! There were no cached chunks for this analysis, so you can be confident that you successfully produced the results during this run.

File paths: relative

Great job! Using relative paths to the files within your workflowr project makes it easier to run your code on other machines.

Repository version: 073e62a

Great! You are using Git for version control. Tracking code development and connecting the code version to the results is critical for reproducibility.

The results in this page were generated with repository version 073e62a. See the Past versions tab to see a history of the changes made to the R Markdown and HTML files.

Note that you need to be careful to ensure that all relevant files for the analysis have been committed to Git prior to generating the results (you can use wflow_publish or wflow_git_commit). workflowr only checks the R Markdown file, but you know if there are other scripts or data files that it depends on. Below is the status of the Git repository when the results were generated:


Ignored files:
    Ignored:    .Rhistory
    Ignored:    .Rproj.user/
    Ignored:    .Ruserdata/
    Ignored:    .drake/
    Ignored:    .vscode/
    Ignored:    analysis/rosm.cache/
    Ignored:    data/
    Ignored:    inst/Benchmark for Filter Methods for Feature Selection in High-Dimensional  Classification Data.pdf
    Ignored:    inst/study-area-map/._study-area.qgs
    Ignored:    inst/study-area-map/study-area.qgs~
    Ignored:    log/
    Ignored:    renv/library/
    Ignored:    renv/staging/
    Ignored:    reviews/
    Ignored:    rosm.cache/

Unstaged changes:
    Modified:   analysis/report-defoliation.Rmd

Note that any generated files, e.g. HTML, png, CSS, etc., are not included in this status report because it is ok for generated content to have uncommitted changes.

These are the previous versions of the repository in which changes were made to the R Markdown (analysis/eval-performance.Rmd) and HTML (docs/eval-performance.html) files. If you’ve configured a remote Git repository (see ?wflow_git_remote), click on the hyperlinks in the table below to view the files as they were in that past version.

File	Version	Author	Date	Message
html	8b5e422	pat-s	2020-08-05	Build site.
html	6f88553	pat-s	2020-06-21	Build site.
Rmd	c94e7fc	pat-s	2020-06-21	wflow_publish(knitr_in(“analysis/eval-performance.Rmd”), view = FALSE,
html	9a54378	pat-s	2020-05-29	Build site.
Rmd	679a87a	pat-s	2020-05-29	wflow_publish(knitr_in(“analysis/eval-performance.Rmd”), view = FALSE,
html	6eeb69b	pat-s	2020-05-06	Build site.
Rmd	fc28c0f	pat-s	2020-05-06	wflow_publish(knitr_in(“analysis/eval-performance.Rmd”), view = FALSE,
html	a2a6562	pat-s	2020-05-03	Build site.
Rmd	b9d07a0	pat-s	2020-05-03	wflow_publish(knitr_in(“analysis/eval-performance.Rmd”), view = FALSE,
html	f7737ce	pat-s	2020-04-30	Build site.
Rmd	576a62f	pat-s	2020-04-30	wflow_publish(knitr_in(“analysis/eval-performance.Rmd”), view = FALSE,
html	fcabe1a	pat-s	2020-04-30	Build site.
Rmd	3aa1860	pat-s	2020-04-30	wflow_publish(knitr_in(“analysis/eval-performance.Rmd”), view = FALSE,
html	bf3eee4	pat-s	2020-04-30	Build site.
Rmd	3a4a1ae	pat-s	2020-04-30	wflow_publish(knitr_in(“analysis/eval-performance.Rmd”), view = FALSE,
html	ebf4147	pat-s	2020-04-30	Build site.
Rmd	96f47c3	pat-s	2020-04-30	wflow_publish(knitr_in(“analysis/eval-performance.Rmd”), view = FALSE,
html	1c1c5f4	pat-s	2020-04-30	Build site.
Rmd	b1d64ea	pat-s	2020-04-30	wflow_publish(knitr_in(“analysis/eval-performance.Rmd”), view = FALSE,
html	9ea0dff	pat-s	2020-04-29	Build site.
Rmd	ba54b0d	pat-s	2020-04-29	wflow_publish(knitr_in(“analysis/eval-performance.Rmd”), view = FALSE,
html	3bdb92f	pat-s	2020-04-29	Build site.
Rmd	1b98cdd	pat-s	2020-04-29	wflow_publish(knitr_in(“analysis/eval-performance.Rmd”), view = FALSE,
html	01b8c74	pat-s	2020-04-29	Build site.
Rmd	c79c23d	pat-s	2020-04-29	wflow_publish(knitr_in(“analysis/eval-performance.Rmd”), view = FALSE,
html	e20d376	pat-s	2020-04-29	Build site.
Rmd	4c8c633	pat-s	2020-04-29	wflow_publish(knitr_in(“analysis/eval-performance.Rmd”), view = FALSE,
html	72c4b82	pat-s	2020-04-19	Build site.
html	89adb44	pat-s	2020-04-18	Build site.
Rmd	6b863d0	pat-s	2020-04-18	wflow_publish(knitr_in(“analysis/eval-performance.Rmd”), view = FALSE,
html	07fb043	pat-s	2020-04-18	Build site.
Rmd	abc2b51	pat-s	2020-04-18	wflow_publish(knitr_in(“analysis/eval-performance.Rmd”), view = FALSE,
html	e670acc	pat-s	2020-04-18	Build site.
Rmd	1fc0c78	pat-s	2020-04-18	wflow_publish(knitr_in(“analysis/eval-performance.Rmd”), view = FALSE,
html	1d1bee4	pat-s	2020-04-18	Build site.
Rmd	f3103fc	pat-s	2020-04-18	wflow_publish(knitr_in(“analysis/eval-performance.Rmd”), view = FALSE,
html	4982c76	pat-s	2020-04-18	Build site.
Rmd	b4fda43	pat-s	2020-04-18	wflow_publish(knitr_in(“analysis/eval-performance.Rmd”), view = FALSE,
html	6c42b7c	pat-s	2020-04-18	Build site.
Rmd	8c30483	pat-s	2020-04-18	wflow_publish(knitr_in(“analysis/eval-performance.Rmd”), view = FALSE,
html	544e288	pat-s	2020-04-12	Build site.
Rmd	ce72956	pat-s	2020-04-12	wflow_publish(knitr_in(“analysis/eval-performance.Rmd”), view = FALSE,
html	6bded83	pat-s	2020-03-05	Build site.
Rmd	21453cb	pat-s	2020-03-05	wflow_publish(knitr_in(“analysis/eval-performance.Rmd”), view = FALSE,
html	f59d02a	pat-s	2020-03-05	Build site.
Rmd	167fdbc	pat-s	2020-03-05	wflow_publish(knitr_in(“analysis/eval-performance.Rmd”), view = FALSE,
html	2ee982d	pat-s	2020-03-05	Build site.
Rmd	d487d51	pat-s	2020-03-05	wflow_publish(knitr_in(“analysis/eval-performance.Rmd”), view = FALSE,
html	776b35f	pat-s	2020-03-03	Build site.
Rmd	d0c645a	pat-s	2020-03-03	wflow_publish(knitr_in(“analysis/eval-performance.Rmd”), view = FALSE,
html	274a918	pat-s	2020-02-25	Build site.
Rmd	2e632e1	pat-s	2020-02-25	wflow_publish(knitr_in(“analysis/eval-performance.Rmd”), view = FALSE,
Rmd	a53ea68	pat-s	2020-02-24	add prefixes to plots and tables
Rmd	379bd5d	pat-s	2020-02-24	wflow_publish(knitr_in(“analysis/eval-performance.Rmd”), view = FALSE,
html	f680221	pat-s	2020-01-15	Build site.
html	b25e779	pat-s	2020-01-10	Build site.
Rmd	db0baaa	pat-s	2020-01-10	wflow_publish(knitr_in(“analysis/eval-performance.Rmd”), view = FALSE,
html	7f9507f	pat-s	2019-12-10	Build site.
Rmd	951e98c	pat-s	2019-12-10	wflow_publish(knitr_in(“analysis/eval-performance.Rmd”), view = FALSE, verbose =
Rmd	b27c623	pat-s	2019-12-09	Standard Error -> SE
Rmd	559a59d	pat-s	2019-12-09	add scatterplots to vis BM perf
html	482a158	pat-s	2019-11-01	Build site.
html	becf5ea	pat-s	2019-11-01	Build site.
html	bd7c7f5	pat-s	2019-10-31	Build site.
html	62ff96f	pat-s	2019-10-07	Build site.
html	a947654	pat-s	2019-10-02	Build site.
html	49da171	pat-s	2019-09-22	Build site.
html	c6317a8	pat-s	2019-09-19	Build site.
Rmd	d7c72a8	pat-s	2019-09-19	wflow_publish(knitr_in(“analysis/eval-performance.Rmd”), view = FALSE,
html	7fd40ca	pat-s	2019-09-18	Build site.
Rmd	44ff84b	pat-s	2019-09-18	wflow_publish(knitr_in(“analysis/eval-performance.Rmd”), view = FALSE,
html	41aae14	pat-s	2019-09-12	Build site.
html	ff340b8	pat-s	2019-09-03	Build site.
Rmd	a524819	pat-s	2019-09-03	wflow_publish(knitr_in(“analysis/eval-performance.Rmd”), view = FALSE,
html	b181c52	pat-s	2019-09-02	Build site.
Rmd	cf6e820	pat-s	2019-09-02	wflow_publish(“analysis/eval-performance.Rmd”)
Rmd	1bec10d	pat-s	2019-09-01	no timestamps in latex tables
html	4e363ac	pat-s	2019-09-01	Build site.
Rmd	518d0cb	pat-s	2019-09-01	style files using tidyverse style
html	8e7e4fe	pat-s	2019-09-01	Build site.
Rmd	8941bca	pat-s	2019-09-01	wflow_publish(knitr_in(“analysis/eval-performance.Rmd”), view = FALSE,
Rmd	297ed93	pat-s	2019-08-31	add filter vs no filter comparison plot
html	7582c67	pat-s	2019-08-31	Build site.
html	abd531f	pat-s	2019-08-31	Build site.
Rmd	9117eee	pat-s	2019-08-31	wflow_publish(knitr_in(“analysis/eval-performance.Rmd”), view = FALSE,
Rmd	5b629cb	pat-s	2019-08-19	add new tasks to performance eval report
html	1ec8768	pat-s	2019-08-17	Build site.
html	df85aba	pat-s	2019-07-12	Build site.
html	3a44a95	pat-s	2019-07-10	Build site.
html	c238ce4	pat-s	2019-07-10	Build site.
Rmd	e98cb01	pat-s	2019-07-10	wflow_publish(knitr_in(“analysis/eval-performance.Rmd”), view = FALSE,
Rmd	24e318f	pat-s	2019-07-01	update reports
Rmd	ca5c5bc	pat-s	2019-06-28	add eval-performance report

Last update:

[1] "Wed Aug 12 11:02:07 2020"

Fold performances of “SVM MBO No Filter” on the HR Task

Fold 1: Luiando
Fold 2: Laukiz1
Fold 3: Laukiz2
Fold 4: Oiartzun

Plot	RMSE	Test Plot
1	21.17	Laukiz1
2	28.05	Oiartzun
3	9.00	Luiando
4	54.26	Laukiz2

(Table) T1 All leaner/filter/task combinations ordered by performance.

Overall leaderboard across all settings, sorted ascending by performance.

Task	Model	Filter	RMSE	SE
NRI	SVM	Info Gain	27.991	19.152
HR-NRI-VI	SVM	Relief	28.067	19.143
VI	SVM	Relief	28.097	19.135
HR-NRI-VI	SVM	Car	28.107	19.126
HR-NRI	SVM	MRMR	28.117	19.113
VI	SVM	Pearson	28.119	19.099
HR-NRI	SVM	CMIM	28.119	19.090
HR	SVM	Info Gain	28.122	19.123
HR	SVM	CMIM	28.122	19.123
NRI-VI	SVM	PCA	28.122	19.123
HR-NRI	SVM	PCA	28.122	19.123
HR-NRI-VI	SVM	PCA	28.122	19.123
NRI-VI	SVM	No Filter	28.122	19.123
HR-NRI	SVM	No Filter	28.122	19.123
HR	SVM	Car	28.123	19.123
HR	SVM	Borda	28.123	19.123
NRI	SVM	PCA	28.123	19.123
HR	SVM	No Filter	28.123	19.123
VI	SVM	No Filter	28.123	19.123
NRI	SVM	No Filter	28.123	19.123
HR-NRI-VI	SVM	No Filter	28.123	19.123
HR	SVM	Pearson	28.124	19.123
NRI-VI	SVM	Car	28.125	19.129
VI	SVM	MRMR	28.125	19.119
HR	SVM	PCA	28.125	19.122
VI	SVM	Info Gain	28.126	19.123
HR-NRI-VI	SVM	MRMR	28.126	19.100
HR	SVM	Relief	28.130	19.114
HR-NRI-VI	SVM	Borda	28.133	19.362
VI	SVM	PCA	28.144	19.091
HR	SVM	MRMR	28.153	19.123
VI	SVM	Borda	28.155	19.183
NRI-VI	SVM	CMIM	28.157	19.077
HR-NRI-VI	SVM	CMIM	28.162	19.071
VI	SVM	CMIM	28.169	19.218
HR-NRI	SVM	Info Gain	28.175	19.131
NRI-VI	SVM	Borda	28.181	19.045
HR-NRI-VI	SVM	Info Gain	28.204	19.023
HR-NRI	SVM	Borda	28.205	19.007
HR-NRI	SVM	Pearson	28.213	19.151
VI	SVM	Car	28.263	19.277
NRI	SVM	Pearson	28.321	18.942
HR-NRI	SVM	Relief	28.337	18.840
HR-NRI	SVM	Car	28.384	18.762
NRI	SVM	Car	28.403	18.744
NRI-VI	SVM	MRMR	28.440	18.753
HR-NRI-VI	SVM	Pearson	28.469	18.640
NRI	SVM	MRMR	28.517	18.607
NRI-VI	SVM	Relief	28.575	18.710
NRI	SVM	CMIM	28.657	18.428
NRI-VI	SVM	Info Gain	28.664	18.299
NRI	SVM	Borda	28.679	18.400
NRI-VI	SVM	Pearson	28.738	18.534
NRI	RF	Car	30.774	16.861
VI	Lasso-MBO	No Filter	31.009	14.714
HR-NRI-VI	XGBoost	Borda	31.052	17.009
HR-NRI	XGBoost	CMIM	31.089	16.576
NRI	Lasso-MBO	No Filter	31.165	15.025
NRI	Ridge-MBO	No Filter	31.165	15.025
NRI-VI	Lasso-MBO	No Filter	31.201	15.089
HR-NRI	Lasso-MBO	No Filter	31.201	15.089
HR-NRI-VI	Lasso-MBO	No Filter	31.201	15.089
NRI-VI	XGBoost	Car	32.082	17.672
NRI-VI	XGBoost	MRMR	32.595	16.279
NRI-VI	XGBoost	CMIM	32.721	15.436
HR-NRI	XGBoost	Relief	32.743	14.094
NRI	RF	PCA	32.966	16.161
NRI	RF	Borda	33.050	14.858
NRI-VI	RF	PCA	33.096	16.001
NRI	XGBoost	Car	33.154	16.403
HR-NRI	RF	PCA	33.255	15.723
NRI-VI	XGBoost	No Filter	33.505	14.599
HR-NRI-VI	RF	PCA	33.588	15.710
HR-NRI-VI	RF	Car	33.779	14.297
HR-NRI	RF	Car	34.044	13.434
NRI	XGBoost	CMIM	34.168	15.571
HR-NRI-VI	XGBoost	MRMR	34.170	14.940
NRI	SVM	Relief	34.177	19.905
NRI	XGBoost	Pearson	34.209	12.246
HR-NRI	XGBoost	Pearson	34.234	13.734
NRI	XGBoost	MRMR	34.393	14.037
HR	RF	Car	34.467	12.604
HR-NRI-VI	XGBoost	Pearson	34.516	14.253
NRI-VI	XGBoost	Info Gain	34.518	15.005
HR-NRI	XGBoost	Borda	34.590	14.887
NRI-VI	XGBoost	Relief	34.709	14.513
NRI	RF	CMIM	34.734	13.733
HR	XGBoost	Borda	34.748	13.572
HR-NRI-VI	RF	CMIM	34.765	13.823
NRI-VI	XGBoost	Pearson	34.771	13.534
HR-NRI-VI	XGBoost	No Filter	34.774	13.804
HR	XGBoost	Relief	34.775	13.317
HR-NRI-VI	XGBoost	CMIM	34.797	14.758
HR-NRI-VI	XGBoost	Car	34.800	14.702
HR	XGBoost	Car	34.806	12.915
NRI	RF	MRMR	34.843	13.937
HR-NRI	RF	MRMR	34.895	13.781
HR-NRI-VI	XGBoost	Info Gain	34.900	13.618
NRI	XGBoost	Info Gain	34.904	15.799
HR-NRI	XGBoost	No Filter	34.963	13.769
NRI-VI	RF	Car	34.996	13.308
HR-NRI-VI	RF	MRMR	35.007	13.634
NRI-VI	RF	CMIM	35.189	14.172
NRI-VI	RF	MRMR	35.223	14.129
HR-NRI-VI	XGBoost	Relief	35.246	12.820
HR	Lasso-MBO	No Filter	35.261	12.243
HR	XGBoost	CMIM	35.276	13.665
HR	XGBoost	Pearson	35.325	13.678
HR	Ridge-MBO	No Filter	35.453	12.655
HR	RF	Borda	35.593	12.949
NRI-VI	XGBoost	Borda	35.608	11.823
NRI	XGBoost	Borda	35.640	12.364
NRI-VI	RF	Borda	35.691	14.051
HR	RF	Info Gain	35.825	13.848
HR-NRI	XGBoost	Info Gain	35.842	14.543
NRI	RF	No Filter	35.868	12.873
HR	XGBoost	MRMR	35.869	13.866
NRI-VI	RF	No Filter	35.999	12.895
NRI	XGBoost	No Filter	36.051	11.604
NRI	XGBoost	Relief	36.062	12.779
HR-NRI	RF	Borda	36.074	13.787
HR-NRI	RF	No Filter	36.222	12.833
HR-NRI	RF	CMIM	36.317	14.144
HR-NRI-VI	RF	No Filter	36.356	12.638
HR-NRI	RF	Info Gain	36.432	13.943
NRI	RF	Info Gain	36.452	13.827
HR-NRI-VI	RF	Info Gain	36.560	13.713
NRI-VI	RF	Info Gain	36.575	13.856
HR-NRI-VI	RF	Borda	36.654	13.050
HR-NRI	RF	Pearson	36.883	13.619
NRI-VI	RF	Pearson	36.903	13.495
HR	RF	Pearson	36.920	14.457
HR-NRI	XGBoost	MRMR	37.010	13.147
NRI	RF	Pearson	37.141	13.266
VI	XGBoost	Relief	37.147	14.262
HR	RF	PCA	37.153	12.329
HR-NRI-VI	RF	Pearson	37.300	13.443
NRI	RF	Relief	37.376	9.425
HR-NRI	XGBoost	Car	37.572	11.101
HR-NRI	RF	Relief	37.690	9.839
VI	RF	MRMR	38.181	12.255
NRI	XGBoost	PCA	38.429	14.982
NRI-VI	RF	Relief	38.498	8.671
HR-NRI-VI	RF	Relief	38.498	8.909
VI	RF	No Filter	38.754	11.271
VI	RF	Borda	38.840	11.237
HR	RF	CMIM	39.687	8.106
VI	XGBoost	Info Gain	39.872	10.880
VI	RF	Info Gain	39.920	9.458
HR	XGBoost	Info Gain	40.070	10.743
VI	RF	CMIM	40.099	9.758
VI	RF	Pearson	40.175	9.959
HR-NRI-VI	XGBoost	PCA	40.185	11.391
VI	RF	Relief	40.449	8.367
NRI-VI	XGBoost	PCA	40.567	10.839
VI	RF	Car	40.581	12.448
VI	RF	PCA	40.719	8.567
HR	RF	Relief	41.656	10.640
VI	XGBoost	CMIM	41.727	9.810
HR	RF	No Filter	41.884	7.806
HR	XGBoost	PCA	42.148	8.077
HR	RF	MRMR	42.312	7.705
HR-NRI	XGBoost	PCA	42.578	8.278
VI	XGBoost	PCA	43.667	8.067
VI	XGBoost	Pearson	44.610	5.783
VI	XGBoost	No Filter	45.689	5.304
VI	XGBoost	Borda	46.041	3.871
VI	XGBoost	MRMR	46.256	9.422
VI	XGBoost	Car	46.399	7.575
HR	XGBoost	No Filter	46.800	3.921
NRI-VI	Ridge-MBO	No Filter	11658468597.678	23316937147.742
HR-NRI-VI	Ridge-MBO	No Filter	12631934180.910	25263868314.206
HR-NRI	Ridge-MBO	No Filter	12650121073.664	25300242099.713
VI	Ridge-MBO	No Filter	49359394487.653	98718788927.687

(Table) T2 Best learner/filter/task combination

Learners: On which task and using which filter did every learner score their best result on?

*CV: L2 penalized regression using the internal 10-fold CV tuning of the glmnet package

*MBO: L2 penalized regression using using MBO for hyperparameter optimization.

Task	Model	Filter	RMSE	SE
NRI	SVM	Info Gain	27.991	19.152
NRI	RF	Car	30.774	16.861
VI	Lasso-MBO	No Filter	31.009	14.714
HR-NRI-VI	XGBoost	Borda	31.052	17.009
NRI	Ridge-MBO	No Filter	31.165	15.025

(Table) T3 All leaner/filter/task combinations ordered descending by performance.

Overall leaderboard across all settings, sorted descending by performance.

Task	Model	Filter	RMSE
VI	Ridge-MBO	No Filter	49359394487.653
HR-NRI	Ridge-MBO	No Filter	12650121073.664
HR-NRI-VI	Ridge-MBO	No Filter	12631934180.910
NRI-VI	Ridge-MBO	No Filter	11658468597.678
HR	XGBoost	No Filter	46.800
VI	XGBoost	Car	46.399
VI	XGBoost	MRMR	46.256
VI	XGBoost	Borda	46.041
VI	XGBoost	No Filter	45.689
VI	XGBoost	Pearson	44.610

(Plot) P1 Best learner/filter combs for all tasks

Version	Author	Date
e20d376	pat-s	2020-04-29
07fb043	pat-s	2020-04-18
1d1bee4	pat-s	2020-04-18
6c42b7c	pat-s	2020-04-18
544e288	pat-s	2020-04-12
f59d02a	pat-s	2020-03-05
2ee982d	pat-s	2020-03-05
274a918	pat-s	2020-02-25
b25e779	pat-s	2020-01-10
7f9507f	pat-s	2019-12-10
482a158	pat-s	2019-11-01
becf5ea	pat-s	2019-11-01
bd7c7f5	pat-s	2019-10-31
62ff96f	pat-s	2019-10-07
a947654	pat-s	2019-10-02
49da171	pat-s	2019-09-22
41aae14	pat-s	2019-09-12
b181c52	pat-s	2019-09-02
8e7e4fe	pat-s	2019-09-01
7582c67	pat-s	2019-08-31
abd531f	pat-s	2019-08-31

(Plot) P2 Scatterplots of filter methods vs. no filter for each learner and task

Showing the final effect of applying feature selection to a learner for each task. All filters are colored in the same way whereas using “no filter” appears in a different color.

Version	Author	Date
e20d376	pat-s	2020-04-29
1d1bee4	pat-s	2020-04-18
6c42b7c	pat-s	2020-04-18
544e288	pat-s	2020-04-12
f59d02a	pat-s	2020-03-05
2ee982d	pat-s	2020-03-05
274a918	pat-s	2020-02-25
b25e779	pat-s	2020-01-10
7f9507f	pat-s	2019-12-10

(Plot) P3 Scatterplots of filter methods vs. Borda for each learner and task

Showing the final effect of applying feature selection to a learner for each task. All filters are summarized into a a single color whereas the “Borda” filter appears in its own color.

Version	Author	Date
e20d376	pat-s	2020-04-29
1d1bee4	pat-s	2020-04-18
6c42b7c	pat-s	2020-04-18
544e288	pat-s	2020-04-12
f59d02a	pat-s	2020-03-05
2ee982d	pat-s	2020-03-05
776b35f	pat-s	2020-03-03
274a918	pat-s	2020-02-25
b25e779	pat-s	2020-01-10
7f9507f	pat-s	2019-12-10

(Table) T4 Number of features selected during tuning

The model/task combinations which were selected relate to the best performance of the respective algorithm on the HR-NRI-VI task in the overall benchmark.

Learner	Test Plot	Features (\%)	\#	RMSE
\multirow{4}{*}{\specialcell{RF \\ Car}}	Laukiz1	0.00442770793	3	27.47555
	Oiartzun	0.00004284398	1	37.20174
	Luiando	0.00001759137	1	36.97731
	Laukiz2	0.03037475841	14	15.38889
\midrule\multirow{4}{*}{\specialcell{XGB \\ Borda}}	Laukiz1	0.00604085450	4	14.99858
	Oiartzun	0.68226628614	340	36.14989
	Luiando	0.99512538539	300	38.18847
	Laukiz2	0.99975780141	451	29.62486
\midrule\multirow{4}{*}{\specialcell{SVM \\ Relief}}	Laukiz1	0.00017322853	1	35.50464
	Oiartzun	0.71146238948	354	37.48157
	Luiando	0.70671610551	213	14.89400
	Laukiz2	0.94695622557	428	37.18944

Aggregated mean and standard deviation:

Learner	Mean (Features (%))	SD (Features (%))
RF Car	0.008715725	0.01458741
SVM Relief	0.591326987	0.40974874
XGBoost Borda	0.670797582	0.46741715

R version 3.6.1 (2019-07-05)
Platform: x86_64-pc-linux-gnu (64-bit)
Running under: CentOS Linux 7 (Core)

Matrix products: default
BLAS:   /opt/spack/opt/spack/linux-centos7-x86_64/gcc-9.2.0/r-3.6.1-j25wr6zcofibs2zfjwg37357rjj26lqb/rlib/R/lib/libRblas.so
LAPACK: /opt/spack/opt/spack/linux-centos7-x86_64/gcc-9.2.0/r-3.6.1-j25wr6zcofibs2zfjwg37357rjj26lqb/rlib/R/lib/libRlapack.so

locale:
 [1] LC_CTYPE=en_US.UTF-8       LC_NUMERIC=C              
 [3] LC_TIME=en_US.UTF-8        LC_COLLATE=en_US.UTF-8    
 [5] LC_MONETARY=en_US.UTF-8    LC_MESSAGES=en_US.UTF-8   
 [7] LC_PAPER=en_US.UTF-8       LC_NAME=C                 
 [9] LC_ADDRESS=C               LC_TELEPHONE=C            
[11] LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C       

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base     

other attached packages:
 [1] forcats_0.4.0     dplyr_0.8.3       mlr_2.17.0.9001  
 [4] ParamHelpers_1.12 here_0.1          ggpubr_0.1.6     
 [7] magrittr_1.5      ggrepel_0.8.0     ggsci_2.9        
[10] ggbeeswarm_0.7.0  ggplot2_3.3.0     flextable_0.5.8  
[13] xtable_1.8-3      tidyselect_0.2.5  drake_7.12.0     

loaded via a namespace (and not attached):
 [1] fs_1.4.1           filelock_1.0.2     RColorBrewer_1.1-2
 [4] progress_1.2.0     httr_1.4.0         rprojroot_1.3-2   
 [7] tools_3.6.1        backports_1.1.5    R6_2.4.1          
[10] vipor_0.4.5        lazyeval_0.2.1     colorspace_1.4-0  
[13] withr_2.1.2        prettyunits_1.0.2  mco_1.0-15.1      
[16] compiler_3.6.1     git2r_0.26.1       parallelMap_1.4   
[19] cli_2.0.2          DiceKriging_1.5.6  xml2_1.2.2        
[22] plotly_4.8.0       officer_0.3.7      labeling_0.3      
[25] scales_1.1.0       checkmate_2.0.0    plot3D_1.1.1      
[28] systemfonts_0.1.1  stringr_1.4.0      digest_0.6.25     
[31] txtq_0.1.4         rmarkdown_1.13     R.utils_2.8.0     
[34] smoof_1.5.1        base64enc_0.1-3    pkgconfig_2.0.3   
[37] htmltools_0.3.6    lhs_1.0.1          htmlwidgets_1.3   
[40] rlang_0.4.5        BBmisc_1.11        farver_2.0.3      
[43] mlrMBO_1.1.2       jsonlite_1.6       zip_2.0.4         
[46] R.oo_1.23.0        Matrix_1.2-15      Rcpp_1.0.3        
[49] munsell_0.5.0      fansi_0.4.1        gdtools_0.2.1     
[52] lifecycle_0.1.0    R.methodsS3_1.7.1  stringi_1.3.1     
[55] whisker_0.3-2      yaml_2.2.0         storr_1.2.1       
[58] RJSONIO_1.3-1.1    grid_3.6.1         misc3d_0.8-4      
[61] parallel_3.6.1     promises_1.0.1     crayon_1.3.4      
[64] lattice_0.20-38    splines_3.6.1      hms_0.5.3         
[67] knitr_1.23         pillar_1.4.3       igraph_1.2.4.1    
[70] uuid_0.1-2         base64url_1.4      fastmatch_1.1-0   
[73] glue_1.4.0         evaluate_0.13      data.table_1.12.8 
[76] vctrs_0.2.4        httpuv_1.4.5.1     gtable_0.3.0      
[79] purrr_0.3.4        tidyr_1.0.0        assertthat_0.2.1  
[82] xfun_0.5           later_1.0.0        survival_2.43-3   
[85] viridisLite_0.3.0  tibble_2.1.3       beeswarm_0.2.3    
[88] workflowr_1.6.1    ellipsis_0.3.0