Last updated: 2019-08-28

workflowr checks: (Click a bullet for more information)

  • ✔ R Markdown file: up-to-date

    Great! Since the R Markdown file has been committed to the Git repository, you know the exact version of the code that produced these results.

  • ✔ Environment: empty

    Great job! The global environment was empty. Objects defined in the global environment can affect the analysis in your R Markdown file in unknown ways. For reproduciblity it’s best to always run the code in an empty environment.

  • ✔ Seed: set.seed(20190513)

    The command set.seed(20190513) was run prior to running the code in the R Markdown file. Setting a seed ensures that any results that rely on randomness, e.g. subsampling or permutations, are reproducible.

  • ✔ Session information: recorded

    Great job! Recording the operating system, R version, and package versions is critical for reproducibility.

  • ✔ Repository version: 454b53b

    Great! You are using Git for version control. Tracking code development and connecting the code version to the results is critical for reproducibility. The version displayed above was the version of the Git repository at the time these results were generated.

    Note that you need to be careful to ensure that all relevant files for the analysis have been committed to Git prior to generating the results (you can use wflow_publish or wflow_git_commit). workflowr only checks the R Markdown file, but you know if there are other scripts or data files that it depends on. Below is the status of the Git repository when the results were generated: 
    
Ignored files:
    Ignored:    .Rhistory
    Ignored:    .Rproj.user/
    Ignored:    LaTeX/SOM_node_summary.docx
    Ignored:    LaTeX/SOM_node_summary.pdf
    Ignored:    data/NAPA/NAPA_clim_U.Rda
    Ignored:    data/NAPA/NAPA_clim_V.Rda
    Ignored:    data/NAPA/NAPA_clim_W.Rda
    Ignored:    data/NAPA/NAPA_clim_emp_ice.Rda
    Ignored:    data/NAPA/NAPA_clim_emp_oce.Rda
    Ignored:    data/NAPA/NAPA_clim_fmmflx.Rda
    Ignored:    data/NAPA/NAPA_clim_mldkz5.Rda
    Ignored:    data/NAPA/NAPA_clim_mldr10_1.Rda
    Ignored:    data/NAPA/NAPA_clim_qemp_oce.Rda
    Ignored:    data/NAPA/NAPA_clim_qla_oce.Rda
    Ignored:    data/NAPA/NAPA_clim_qns.Rda
    Ignored:    data/NAPA/NAPA_clim_qsb_oce.Rda
    Ignored:    data/NAPA/NAPA_clim_qt.Rda
    Ignored:    data/NAPA/NAPA_clim_runoffs.Rda
    Ignored:    data/NAPA/NAPA_clim_ssh.Rda
    Ignored:    data/NAPA/NAPA_clim_sss.Rda
    Ignored:    data/NAPA/NAPA_clim_sst.Rda
    Ignored:    data/NAPA/NAPA_clim_taum.Rda
    Ignored:    data/NAPA/NAPA_clim_vars.Rda
    Ignored:    data/NAPA/NAPA_clim_vecs.Rda
    Ignored:    data/NAPA/synoptic_vec_states.Rda
    Ignored:    data/SOM/packet.Rda
    Ignored:    data/SOM/synoptic_states.Rda
    Ignored:    data/SOM/synoptic_states_other.Rda
    Ignored:    data/anom/ALL_anom.Rda
    Ignored:    data/anom/ALL_other.Rda
    Ignored:    data/anom/ERA5_mslp_anom.Rda
    Ignored:    data/anom/ERA5_qnet_anom.Rda
    Ignored:    data/anom/ERA5_t2m_anom.Rda
    Ignored:    data/anom/ERA5_u_anom.Rda
    Ignored:    data/anom/ERA5_v_anom.Rda
    Ignored:    data/anom/GLORYS_mld_anom.Rda
    Ignored:    data/anom/GLORYS_u_anom.Rda
    Ignored:    data/anom/GLORYS_v_anom.Rda
    Ignored:    data/anom/OISST_sst_anom.Rda
    Ignored:    data/base/ERA5_lhf.Rda
    Ignored:    data/base/ERA5_lwr.Rda
    Ignored:    data/base/ERA5_mslp.Rda
    Ignored:    data/base/ERA5_qnet.Rda
    Ignored:    data/base/ERA5_shf.Rda
    Ignored:    data/base/ERA5_swr.Rda
    Ignored:    data/base/ERA5_t2m.Rda
    Ignored:    data/base/ERA5_u.Rda
    Ignored:    data/base/ERA5_v.Rda
    Ignored:    data/base/GLORYS_mld.Rda
    Ignored:    data/base/GLORYS_u.Rda
    Ignored:    data/base/GLORYS_v.Rda
    Ignored:    data/base/OAFlux.Rda
    Ignored:    data/base/OISST_sst.Rda
    Ignored:    data/clim/ALL_clim.Rda
    Ignored:    data/clim/ERA5_mslp_clim.Rda
    Ignored:    data/clim/ERA5_qnet_clim.Rda
    Ignored:    data/clim/ERA5_t2m_clim.Rda
    Ignored:    data/clim/ERA5_u_clim.Rda
    Ignored:    data/clim/ERA5_v_clim.Rda
    Ignored:    data/clim/GLORYS_mld_clim.Rda
    Ignored:    data/clim/GLORYS_u_clim.Rda
    Ignored:    data/clim/GLORYS_v_clim.Rda
    Ignored:    data/clim/OISST_sst_clim.Rda
    Ignored:    data/comparison/node_mean_all_anom.Rda
    Ignored:    data/comparison/packet_all.Rda
    Ignored:    data/comparison/packet_all_anom.Rda
    Ignored:    data/comparison/packet_nolab.Rda
    Ignored:    data/comparison/packet_nolab14.Rda
    Ignored:    data/comparison/packet_nolabgsl.Rda
    Ignored:    data/comparison/packet_nolabmod.Rda
    Ignored:    data/comparison/som_all.Rda
    Ignored:    data/comparison/som_all_anom.Rda
    Ignored:    data/comparison/som_nolab.Rda
    Ignored:    data/comparison/som_nolab14.Rda
    Ignored:    data/comparison/som_nolab_16.Rda
    Ignored:    data/comparison/som_nolab_9.Rda
    Ignored:    data/comparison/som_nolabgsl.Rda
    Ignored:    data/comparison/som_nolabmod.Rda

Unstaged changes:
    Modified:   code/functions.R
    Note that any generated files, e.g. HTML, png, CSS, etc., are not included in this status report because it is ok for generated content to have uncommitted changes.
Expand here to see past versions:
    File Version Author Date Message
    Rmd d980be6 robwschlegel 2019-08-26 Working towards the node summary figure pipeline update
    Rmd 27b124b robwschlegel 2019-08-23 Working on code to smoothly introduce the other data needed for creating summary figures but that aren’t used in the SOM calculation
    Rmd 9c4a1d8 robwschlegel 2019-08-21 An additional thought
    Rmd ac54ed0 robwschlegel 2019-08-21 Another round of figure creation
    html 826c73d robwschlegel 2019-08-15 Build site.
    html 143b1a6 robwschlegel 2019-08-15 Build site.
    Rmd 78f4977 robwschlegel 2019-08-15 Re-publish entire site.
    Rmd 07fe2a2 robwschlegel 2019-08-14 Nearly through the node summaries for the three SOM experiments
    Rmd a61b420 robwschlegel 2019-08-13 Working on SOM write-up
    html 20ae166 robwschlegel 2019-08-11 Build site.
    html 19bea26 robwschlegel 2019-08-11 Build site.
    html 2652a3a robwschlegel 2019-08-11 Build site.
    Rmd adc762b robwschlegel 2019-08-08 Re-worked the GLORYS data and propogated update through to SOM analysis figures for all experiments
    html f0d2efb robwschlegel 2019-08-07 Build site.
    Rmd ed626bf robwschlegel 2019-08-07 Ran a bunch of figures and had a meeting with Eric. More changes coming to GLORYS data tomorrow before settling on one of the experimental SOMs
    html f66aa38 robwschlegel 2019-08-01 Build site.
    Rmd 5e12d9e robwschlegel 2019-08-01 Re-publish entire site.
    Rmd 9a9fa7d robwschlegel 2019-08-01 A more in depth dive into the potential criteria to meet for the SOM model
    Rmd 240a7a0 robwschlegel 2019-07-31 Ran the base SOM results
    html aa82e6e robwschlegel 2019-07-31 Build site.
    Rmd 498909b robwschlegel 2019-07-31 Re-publish entire site.
    html 35987b4 robwschlegel 2019-07-09 Build site.
    Rmd 34efa43 robwschlegel 2019-07-09 Added some thinking to the SOM vignette.
    html e2f6f42 robwschlegel 2019-07-09 Build site.
    Rmd 609cca8 robwschlegel 2019-07-09 Added some thinking to the SOM vignette.
    html 81e961d robwschlegel 2019-07-09 Build site.
    Rmd 7ff9b8b robwschlegel 2019-06-17 More work on the talk
    Rmd b25762e robwschlegel 2019-06-12 More work on figures
    Rmd 413bb8b robwschlegel 2019-06-12 Working on pixel interpolation
    html c23c50b robwschlegel 2019-06-10 Build site.
    html 028d3cc robwschlegel 2019-06-10 Build site.
    Rmd c6b3c7b robwschlegel 2019-06-10 Re-publish entire site.
    Rmd 1b53eeb robwschlegel 2019-06-10 SOM packet pipeline testing
    Rmd 4504e12 robwschlegel 2019-06-07 Working on joining in vector data
    html c61a15f robwschlegel 2019-06-06 Build site.
    Rmd 44ac335 robwschlegel 2019-06-06 Working on inclusion of vectors into SOM pipeline
    html 6dd6da8 robwschlegel 2019-06-06 Build site.
    Rmd 07137d9 robwschlegel 2019-06-06 Site wide update, including newly functioning SOM pipeline.
    Rmd 990693a robwschlegel 2019-06-05 First SOM result visuals
    Rmd 25e7e9a robwschlegel 2019-06-05 SOM pipeline nearly finished
    Rmd 4838cc8 robwschlegel 2019-06-04 Working on SOM functions
    Rmd 94ce8f6 robwschlegel 2019-06-04 Functions for creating data packets are up and running
    Rmd 65301ed robwschlegel 2019-05-30 Push before getting rid of some testing structure
    html c09b4f7 robwschlegel 2019-05-24 Build site.
    Rmd 5dc8bd9 robwschlegel 2019-05-24 Finished initial creation of SST prep vignette.
    html a29be6b robwschlegel 2019-05-13 Build site.
    html ea61999 robwschlegel 2019-05-13 Build site.
    Rmd f8f28b1 robwschlegel 2019-05-13 Skeleton files

Introduction

This vignette contains the code used to perform the self-organising map (SOM) analysis on the mean synoptic states created in the Variable preparation vignette. We’ll start by creating custom packets that meet certain experimental criteria before then feeding them into a SOM. We will finish up by creating some cursory visuals of the results. The full summary of the results may be seen in the Node summary vignette.

# Insatll from GitHub
# .libPaths(c("~/R-packages", .libPaths()))
# devtools::install_github("fabrice-rossi/yasomi")

# Packages used in this vignette
library(jsonlite, lib.loc = "../R-packages/")
library(tidyverse) # Base suite of functions
library(lubridate) # For convenient date manipulation
library(yasomi, lib.loc = "../R-packages/") # The SOM package of choice due to PCI compliance
library(data.table) # For working with massive dataframes

# Load functions and objects to be used below
source("code/functions.R")

Data packet

In this last stage before running our SOM analysis we will create a data packet that can be fed directly into the SOM algorithm. This means that it must be converted into a super-wide matrix format. In the first run of this analysis on the NAPA model data it was found that the inclusion of the Labrador Sea complicated the results quite a bit. It was also unclear whether or not the Gulf of St Lawrence region should be included in the analysis. So in the second run of this analysis multiple different SOM variations were employed and it was decided that the gsl region should be included.

Unnest synoptic state packets

Up first we must simply load and unnest the synoptic state packets made previously.

# Load the synoptic states data packet
system.time(
  synoptic_states <- readRDS("data/SOM/synoptic_states.Rda")
) # 3 seconds

# Unnest the synoptic data
system.time(
  synoptic_states_unnest <- synoptic_states %>% 
    select(region, event_no, synoptic) %>% 
    unnest()
) # 4 seconds

Create packet

With all of our data ready we may now prepare and save them for the SOM.

# Packet for entire study region
system.time(
  packet <- wide_packet_func(synoptic_states_unnest)
) # 179 seconds
saveRDS(packet, "data/SOM/packet.Rda")

Run SOM models

Now that we have our data packet to feed the SOM with a function that ingests them and produces results for us. The function below has been greatly expanded on from the previous version of this project and now performs all of the SOM related work in one go. This allowed me to remove a couple hundreds lines of code and text from this vignette.

# The SOM on the entire study area
packet <- readRDS("data/SOM/packet.Rda")
system.time(som <- som_model_PCI(packet)) # 69 seconds
# som$ANOSIM # p = 0.001
saveRDS(som, file = "data/SOM/som.Rda")

And there we have our SOM results. Up next in the Node summary vignette we will show the results with a range of visuals.

References

Session information

sessionInfo()
R version 3.6.1 (2019-07-05)
Platform: x86_64-pc-linux-gnu (64-bit)
Running under: Ubuntu 16.04.6 LTS

Matrix products: default
BLAS:   /usr/lib/openblas-base/libblas.so.3
LAPACK: /usr/lib/libopenblasp-r0.2.18.so

locale:
 [1] LC_CTYPE=en_CA.UTF-8       LC_NUMERIC=C              
 [3] LC_TIME=en_CA.UTF-8        LC_COLLATE=en_CA.UTF-8    
 [5] LC_MONETARY=en_CA.UTF-8    LC_MESSAGES=en_CA.UTF-8   
 [7] LC_PAPER=en_CA.UTF-8       LC_NAME=C                 
 [9] LC_ADDRESS=C               LC_TELEPHONE=C            
[11] LC_MEASUREMENT=en_CA.UTF-8 LC_IDENTIFICATION=C       

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base     

other attached packages:
 [1] bindrcpp_0.2.2    tidync_0.2.1      heatwaveR_0.4.0  
 [4] data.table_1.11.6 yasomi_0.3        proxy_0.4-22     
 [7] e1071_1.7-0       lubridate_1.7.4   forcats_0.3.0    
[10] stringr_1.3.1     dplyr_0.7.6       purrr_0.2.5      
[13] readr_1.1.1       tidyr_0.8.1       tibble_1.4.2     
[16] ggplot2_3.0.0     tidyverse_1.2.1   jsonlite_1.6     

loaded via a namespace (and not attached):
 [1] Rcpp_0.12.18      lattice_0.20-35   class_7.3-14     
 [4] assertthat_0.2.0  rprojroot_1.3-2   digest_0.6.16    
 [7] foreach_1.4.4     R6_2.2.2          cellranger_1.1.0 
[10] plyr_1.8.4        backports_1.1.2   evaluate_0.11    
[13] httr_1.3.1        pillar_1.3.0      rlang_0.2.2      
[16] lazyeval_0.2.1    readxl_1.1.0      ncmeta_0.0.4     
[19] rstudioapi_0.7    whisker_0.3-2     R.utils_2.7.0    
[22] R.oo_1.22.0       rmarkdown_1.10    htmlwidgets_1.3  
[25] munsell_0.5.0     broom_0.5.0       compiler_3.6.1   
[28] modelr_0.1.2      pkgconfig_2.0.2   htmltools_0.3.6  
[31] tidyselect_0.2.4  workflowr_1.1.1   codetools_0.2-15 
[34] doMC_1.3.5        viridisLite_0.3.0 crayon_1.3.4     
[37] withr_2.1.2       R.methodsS3_1.7.1 grid_3.6.1       
[40] nlme_3.1-137      gtable_0.2.0      git2r_0.23.0     
[43] magrittr_1.5      scales_1.0.0      ncdf4_1.16.1     
[46] cli_1.0.0         stringi_1.2.4     xml2_1.2.0       
[49] iterators_1.0.10  tools_3.6.1       glue_1.3.0       
[52] RNetCDF_1.9-1     hms_0.4.2         parallel_3.6.1   
[55] yaml_2.2.0        colorspace_1.3-2  rvest_0.3.2      
[58] plotly_4.9.0      knitr_1.20        bindr_0.1.1      
[61] haven_1.1.2

This reproducible R Markdown analysis was created with workflowr 1.1.1