Last updated: 2019-05-27

Checks: 6 0

Knit directory: 2018-model-comparison/

This reproducible R Markdown analysis was created with workflowr (version 1.3.0). The Checks tab describes the reproducibility checks that were applied when the results were created. The Past versions tab lists the development history.

R Markdown file: up-to-date

Great! Since the R Markdown file has been committed to the Git repository, you know the exact version of the code that produced these results.

Environment: empty

Great job! The global environment was empty. Objects defined in the global environment can affect the analysis in your R Markdown file in unknown ways. For reproduciblity it’s best to always run the code in an empty environment.

Seed: set.seed(20190523)

The command set.seed(20190523) was run prior to running the code in the R Markdown file. Setting a seed ensures that any results that rely on randomness, e.g. subsampling or permutations, are reproducible.

Session information: recorded

Great job! Recording the operating system, R version, and package versions is critical for reproducibility.

Cache: none

Nice! There were no cached chunks for this analysis, so you can be confident that you successfully produced the results during this run.

Repository version: 6257f3b

Great! You are using Git for version control. Tracking code development and connecting the code version to the results is critical for reproducibility. The version displayed above was the version of the Git repository at the time these results were generated.

Note that you need to be careful to ensure that all relevant files for the analysis have been committed to Git prior to generating the results (you can use wflow_publish or wflow_git_commit). workflowr only checks the R Markdown file, but you know if there are other scripts or data files that it depends on. Below is the status of the Git repository when the results were generated:


Ignored files:
    Ignored:    .Rproj.user/
    Ignored:    .drake/
    Ignored:    analysis/figure/
    Ignored:    code/07-paper/._files
    Ignored:    data/
    Ignored:    log/
    Ignored:    packrat/lib-R/
    Ignored:    packrat/lib-ext/
    Ignored:    packrat/lib/
    Ignored:    rosm.cache/
    Ignored:    tests/testthat/

Unstaged changes:
    Modified:   _drake.R
    Modified:   analysis/index.Rmd
    Modified:   code/07-paper/submission/3/latex-source-files/cv_boxplots_final_brier-1.pdf
    Modified:   inst/rsync-jupiter.sh

Staged changes:
    Modified:   packrat/packrat.lock
    New:        packrat/snapshot.lock
    New:        packrat/src/here/here_0.1.tar.gz

Note that any generated files, e.g. HTML, png, CSS, etc., are not included in this status report because it is ok for generated content to have uncommitted changes.

These are the previous versions of the R Markdown and HTML files. If you’ve configured a remote Git repository (see ?wflow_git_remote), click on the hyperlinks in the table below to view them.

File	Version	Author	Date	Message
Rmd	6257f3b	pat-s	2019-05-27	wflow_publish(knitr_in(“analysis/benchmark-pathogens.Rmd”), view =
Rmd	dac5d6a	pat-s	2019-05-25	upd reports
Rmd	6af4181	pat-s	2019-05-23	update reports
Rmd	769718b	pat-s	2019-05-23	Start workflowr project.

Introduction

This document shows the predictive performances for the possible infection risk of trees in the Basque Country by the following pathogens:

Armillaria mellea
Diplodia sapinea
Fusarium circinatum
Heterobasidion annosum

The following algorithms were benchmarked:

Boosted Regression Trees (BRT)
Generalized Additive Model (GAM)
Generalized Linear Model (GLM)
k-Nearest Neighbor (KNN)
Random Forests (RF)
Support Vector Machine (SVM)
Extreme Gradient Boosting (XGBOOST)

Resampling Strategies

The abbreviations of the tabbed resampling strategies follow the scheme:

<outer resampling> / <inner resampling>

For example, setting “Spatial-Spatial” means that in both levels a “spatial cross-validation” (Brenning (2012)) has been applied.

The inner resampling refers to the hyperparameter tuning level of the nested cross-validation that was applied.

Results structure

The structure of the following results presentation is as follows:

Table view of all performances for each resampling setting
Boxplot comparison for each pathogen and algorithm
Aggregated performances for each pathogen and algorithm

Resampling strategies

Spatial-Spatial

          task.id            learner.id brier.test.mean
1      armillaria    classif.kknn.tuned       0.2929540
2      armillaria     classif.gam.tuned       0.2873762
3      armillaria     classif.svm.tuned       0.2674570
4      armillaria     classif.gbm.tuned       0.2499506
5      armillaria classif.xgboost.tuned       0.2485308
6      armillaria  classif.ranger.tuned       0.2472924
7        diplodia classif.xgboost.tuned       0.2178637
8        diplodia     classif.svm.tuned       0.2000721
9        diplodia     classif.gam.tuned       0.1899124
10       diplodia    classif.kknn.tuned       0.1877918
11       diplodia  classif.ranger.tuned       0.1594762
12       diplodia     classif.gbm.tuned       0.1590360
13       fusarium classif.xgboost.tuned       0.1922855
14       fusarium    classif.kknn.tuned       0.1515723
15       fusarium     classif.svm.tuned       0.1439149
16       fusarium     classif.gam.tuned       0.1283013
17       fusarium     classif.gbm.tuned       0.1277329
18       fusarium  classif.ranger.tuned       0.1225918
19 heterobasidion classif.xgboost.tuned       0.2129888
20 heterobasidion     classif.gam.tuned       0.1958724
21 heterobasidion     classif.svm.tuned       0.1943352
22 heterobasidion    classif.kknn.tuned       0.1928612
23 heterobasidion  classif.ranger.tuned       0.1685481
24 heterobasidion     classif.gbm.tuned       0.1664094
   timetrain.test.mean
1             115.6956
2             253.3654
3             303.2256
4            1538.8172
5             969.9928
6             353.9625
7             980.5603
8             194.6988
9             230.0244
10            109.7034
11            240.8807
12           1820.5990
13            814.4897
14            119.0948
15            202.8788
16            301.8513
17           2711.5960
18            265.6483
19            909.4810
20            264.7740
21            272.2715
22            117.6447
23            285.5910
24           1942.1879

Boxplot comparison

Aggregated performances

Spatial-Non-Spatial

          task.id            learner.id brier.test.mean
1      armillaria     classif.gam.tuned       0.3013001
2      armillaria    classif.kknn.tuned       0.2939335
3      armillaria     classif.gbm.tuned       0.2935869
4      armillaria classif.xgboost.tuned       0.2487059
5      armillaria     classif.svm.tuned       0.2412515
6      armillaria  classif.ranger.tuned       0.2396800
7        diplodia classif.xgboost.tuned       0.2144304
8        diplodia     classif.svm.tuned       0.2125015
9        diplodia     classif.gam.tuned       0.1922458
10       diplodia    classif.kknn.tuned       0.1882630
11       diplodia     classif.gbm.tuned       0.1637896
12       diplodia  classif.ranger.tuned       0.1528584
13       fusarium classif.xgboost.tuned       0.1903816
14       fusarium    classif.kknn.tuned       0.1517528
15       fusarium     classif.svm.tuned       0.1441707
16       fusarium     classif.gam.tuned       0.1286404
17       fusarium     classif.gbm.tuned       0.1150637
18       fusarium  classif.ranger.tuned       0.1033548
19 heterobasidion classif.xgboost.tuned       0.2094289
20 heterobasidion     classif.gam.tuned       0.1945875
21 heterobasidion    classif.kknn.tuned       0.1888104
22 heterobasidion     classif.gbm.tuned       0.1853278
23 heterobasidion     classif.svm.tuned       0.1810352
24 heterobasidion  classif.ranger.tuned       0.1620154
   timetrain.test.mean
1             251.5898
2             116.7439
3            2586.9143
4            1203.0043
5             307.8218
6             467.1088
7            1130.0532
8             389.3593
9             294.0095
10            121.3932
11           2095.5327
12            445.1206
13            840.9277
14            121.1683
15            221.6203
16            304.9121
17           2838.8523
18            309.1905
19           1013.3030
20            264.6495
21            117.8883
22           3010.3889
23            280.5194
24            383.2899

Boxplot comparison

Aggregated performances

Non-Spatial-Non-Spatial

          task.id            learner.id brier.test.mean
1      armillaria     classif.gam.tuned      0.22700637
2      armillaria classif.xgboost.tuned      0.20195496
3      armillaria    classif.kknn.tuned      0.13283583
4      armillaria     classif.svm.tuned      0.13054167
5      armillaria     classif.gbm.tuned      0.12190209
6      armillaria  classif.ranger.tuned      0.10428849
7        diplodia classif.xgboost.tuned      0.19924264
8        diplodia     classif.gam.tuned      0.13952038
9        diplodia     classif.svm.tuned      0.12296446
10       diplodia    classif.kknn.tuned      0.12013106
11       diplodia     classif.gbm.tuned      0.11186382
12       diplodia  classif.ranger.tuned      0.09952600
13       fusarium classif.xgboost.tuned      0.17346649
14       fusarium     classif.gam.tuned      0.06053425
15       fusarium     classif.svm.tuned      0.05512555
16       fusarium    classif.kknn.tuned      0.05227486
17       fusarium     classif.gbm.tuned      0.03002464
18       fusarium  classif.ranger.tuned      0.02947792
19 heterobasidion classif.xgboost.tuned      0.18180963
20 heterobasidion     classif.gam.tuned      0.14701701
21 heterobasidion    classif.kknn.tuned      0.06937628
22 heterobasidion     classif.svm.tuned      0.06687576
23 heterobasidion     classif.gbm.tuned      0.05104793
24 heterobasidion  classif.ranger.tuned      0.04547533
   timetrain.test.mean
1             250.7466
2            1218.4911
3             116.8461
4             306.5662
5            2569.8308
6             477.9465
7            1149.5982
8             290.8944
9             406.3464
10            121.3484
11           1999.3005
12            440.4030
13            868.2870
14            288.3264
15            202.4960
16            120.7944
17           2786.2014
18            319.6525
19           1027.6968
20            269.1685
21            116.9535
22            290.4438
23           3021.3505
24            391.0410

Boxplot comparison

Aggregated performances

Non-Spatial-No Tuning

          task.id       learner.id brier.test.mean timetrain.test.mean
1      armillaria classif.binomial      0.23430477            0.019098
2      armillaria  classif.xgboost      0.19308152            0.588122
3      armillaria      classif.gbm      0.17435081            0.047364
4      armillaria      classif.gam      0.16339620            0.234416
5      armillaria     classif.kknn      0.14319871            0.000534
6      armillaria      classif.svm      0.14008004            0.207078
7      armillaria   classif.ranger      0.10934871            0.357818
8        diplodia  classif.xgboost      0.18104208            1.016850
9        diplodia      classif.svm      0.16032300            0.638496
10       diplodia      classif.gam      0.14672641            0.615932
11       diplodia     classif.kknn      0.11956687            0.000772
12       diplodia classif.binomial      0.11861260            0.021954
13       diplodia      classif.gbm      0.11375122            0.060954
14       diplodia   classif.ranger      0.10170460            0.667556
15       fusarium  classif.xgboost      0.14483346            0.186976
16       fusarium      classif.svm      0.07603583            0.241892
17       fusarium      classif.gam      0.07261828            0.363594
18       fusarium     classif.kknn      0.05268736            0.000528
19       fusarium classif.binomial      0.04841708            0.027582
20       fusarium      classif.gbm      0.03622046            0.050656
21       fusarium   classif.ranger      0.03103918            0.195660
22 heterobasidion classif.binomial      0.16290012            0.022228
23 heterobasidion  classif.xgboost      0.16076201            0.210498
24 heterobasidion      classif.gbm      0.09434126            0.045940
25 heterobasidion      classif.gam      0.08696898            0.252826
26 heterobasidion     classif.kknn      0.07479138            0.000570
27 heterobasidion      classif.svm      0.06915348            0.181514
28 heterobasidion   classif.ranger      0.04976475            0.249832

Boxplot comparison

Aggregated performances

Spatial-No Tuning

          task.id       learner.id brier.test.mean timetrain.test.mean
1      armillaria      classif.gam      0.34203122            0.224132
2      armillaria     classif.kknn      0.28773607            0.000632
3      armillaria classif.binomial      0.28400164            0.018868
4      armillaria      classif.gbm      0.27648700            0.048196
5      armillaria  classif.xgboost      0.25095005            0.612810
6      armillaria      classif.svm      0.25076442            0.201892
7      armillaria   classif.ranger      0.23791242            0.325748
8        diplodia      classif.gam      0.25052534            0.569212
9        diplodia      classif.svm      0.20094838            0.627202
10       diplodia  classif.xgboost      0.19986164            1.025680
11       diplodia classif.binomial      0.17030072            0.022436
12       diplodia     classif.kknn      0.16892370            0.001134
13       diplodia      classif.gbm      0.15981173            0.062854
14       diplodia   classif.ranger      0.14990091            0.651948
15       fusarium  classif.xgboost      0.17056495            0.162042
16       fusarium      classif.svm      0.15712093            0.230380
17       fusarium      classif.gam      0.13216430            0.362944
18       fusarium     classif.kknn      0.13069975            0.000566
19       fusarium      classif.gbm      0.10680714            0.050994
20       fusarium   classif.ranger      0.10410518            0.184228
21       fusarium classif.binomial      0.09171266            0.026594
22 heterobasidion      classif.gam      0.25236362            0.287420
23 heterobasidion classif.binomial      0.19785015            0.022448
24 heterobasidion  classif.xgboost      0.19305571            0.165192
25 heterobasidion      classif.gbm      0.18311438            0.045782
26 heterobasidion      classif.svm      0.18252532            0.175006
27 heterobasidion     classif.kknn      0.17918672            0.000638
28 heterobasidion   classif.ranger      0.16098088            0.236158

Boxplot comparison

Aggregated performances

References

Brenning, A. 2012. “Spatial Cross-Validation and Bootstrap for the Assessment of Prediction Rules in Remote Sensing: The R Package Sperrorest.” In 2012 Ieee International Geoscience and Remote Sensing Symposium, 5372–5. https://doi.org/10.1109/IGARSS.2012.6352393.

R version 3.5.1 (2018-07-02)
Platform: x86_64-pc-linux-gnu (64-bit)
Running under: CentOS Linux 7 (Core)

Matrix products: default
BLAS/LAPACK: /opt/spack/opt/spack/linux-centos7-x86_64/gcc-7.3.0/openblas-0.3.5-zncvk4jccaqyfl4z3vszaboeps6hyzta/lib/libopenblas_zen-r0.3.5.so

locale:
[1] C

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base     

other attached packages:
 [1] tidyselect_0.2.5    workflowr_1.3.0     here_0.1           
 [4] kableExtra_1.1.0    ggExtra_0.8         ggrepel_0.8.0      
 [7] reporttools_1.1.2   xtable_1.8-3        cowplot_0.9.3      
[10] hrbrthemes_0.6.0    ggpubr_0.2          future.callr_0.4.0 
[13] furrr_0.1.0.9002    future_1.11.1.1     ggsci_2.9          
[16] clustermq_0.8.6     ggspatial_1.0.3     ggplot2_3.0.0      
[19] rgenoud_5.8-3.0     fs_1.2.6            curl_3.2           
[22] R.utils_2.7.0       R.oo_1.22.0         R.methodsS3_1.7.1  
[25] GSIF_0.5-5          stringr_1.3.1       RSAGA_1.3.0        
[28] plyr_1.8.4          shapefiles_0.7      foreign_0.8-71     
[31] gstat_1.1-6         glue_1.3.0          rasterVis_0.45     
[34] latticeExtra_0.6-28 RColorBrewer_1.1-2  lattice_0.20-35    
[37] raster_2.8-19       viridis_0.5.1       viridisLite_0.3.0  
[40] rgdal_1.4-3         sp_1.3-1            tibble_2.0.1       
[43] forcats_0.3.0       lwgeom_0.1-6        dplyr_0.8.0.1      
[46] sf_0.7-4            parallelMap_1.3     purrr_0.2.5        
[49] mlrMBO_1.1.2        smoof_1.5.1         checkmate_1.8.5    
[52] BBmisc_1.11         magrittr_1.5        mlr_2.13.9000      
[55] ParamHelpers_1.11   drake_7.2.0        

loaded via a namespace (and not attached):
  [1] backports_1.1.2   Hmisc_4.2-0       fastmatch_1.1-0  
  [4] igraph_1.2.2      lazyeval_0.2.1    splines_3.5.1    
  [7] storr_1.2.1       listenv_0.7.0     digest_0.6.15    
 [10] htmltools_0.3.6   base64url_1.4     cluster_2.0.7-1  
 [13] readr_1.3.1       globals_0.12.4    extrafont_0.17   
 [16] xts_0.11-0        extrafontdb_1.0   colorspace_1.3-2 
 [19] rvest_0.3.2       pixmap_0.4-11     xfun_0.7         
 [22] callr_3.1.0       crayon_1.3.4      jsonlite_1.5     
 [25] hexbin_1.27.2     survival_2.42-3   zoo_1.8-3        
 [28] gtable_0.2.0      webshot_0.5.1     Rttf2pt1_1.3.7   
 [31] scales_1.0.0      DBI_1.0.0         miniUI_0.1.1.1   
 [34] Rcpp_1.0.0        plotrix_3.7-4     spData_0.2.9.0   
 [37] htmlTable_1.12    units_0.6-2       Formula_1.2-3    
 [40] intervals_0.15.1  dismo_1.1-4       htmlwidgets_1.3  
 [43] httr_1.3.1        FNN_1.1           aqp_1.17         
 [46] acepack_1.4.1     pkgconfig_2.0.2   reshape_0.8.8    
 [49] XML_3.98-1.16     nnet_7.3-12       RJSONIO_1.3-1.1  
 [52] labeling_0.3      later_0.7.5       rlang_0.3.1      
 [55] munsell_0.5.0     tools_3.5.1       evaluate_0.13    
 [58] yaml_2.2.0        processx_3.2.1    knitr_1.23       
 [61] mime_0.5          whisker_0.3-2     xml2_1.2.0       
 [64] compiler_3.5.1    rstudioapi_0.10   plotly_4.8.0     
 [67] e1071_1.7-0       spacetime_1.2-2   lhs_0.16         
 [70] stringi_1.2.4     ps_1.2.1          gdtools_0.1.7    
 [73] plot3D_1.1.1      Matrix_1.2-14     classInt_0.2-3   
 [76] pillar_1.3.1      plotKML_0.5-9     data.table_1.11.8
 [79] httpuv_1.4.5      colorRamps_2.3    R6_2.2.2         
 [82] promises_1.0.1    gridExtra_2.3     codetools_0.2-15 
 [85] MASS_7.3-50       assertthat_0.2.0  rprojroot_1.3-2  
 [88] withr_2.1.2       hms_0.4.2         parallel_3.5.1   
 [91] grid_3.5.1        rpart_4.1-13      tidyr_0.8.2      
 [94] class_7.3-14      rmarkdown_1.12    misc3d_0.8-4     
 [97] mco_1.0-15.1      git2r_0.23.0      shiny_1.2.0      
[100] base64enc_0.1-3

Evaluating classifier performance on pathogens

Patrick Schratz, Friedrich-Schiller-University Jena

Introduction

Resampling Strategies

Results structure

Resampling strategies

Spatial-Spatial

Boxplot comparison

Aggregated performances

Spatial-Non-Spatial

Boxplot comparison

Aggregated performances

Non-Spatial-Non-Spatial

Boxplot comparison

Aggregated performances

Non-Spatial-No Tuning

Boxplot comparison

Aggregated performances

Spatial-No Tuning

Boxplot comparison

Aggregated performances

References