Last updated: 2021-04-03

Checks: 6 1

Knit directory: fa_sim_cal/

This reproducible R Markdown analysis was created with workflowr (version 1.6.2). The Checks tab describes the reproducibility checks that were applied when the results were created. The Past versions tab lists the development history.

R Markdown file: uncommitted changes

The R Markdown is untracked by Git. To know which version of the R Markdown file created these results, you’ll want to first commit it to the Git repo. If you’re still working on the analysis, you can ignore this warning. When you’re finished, you can run wflow_publish to commit the R Markdown file and build the HTML.

Environment: empty

Great job! The global environment was empty. Objects defined in the global environment can affect the analysis in your R Markdown file in unknown ways. For reproduciblity it’s best to always run the code in an empty environment.

Seed: set.seed(20201104)

The command set.seed(20201104) was run prior to running the code in the R Markdown file. Setting a seed ensures that any results that rely on randomness, e.g. subsampling or permutations, are reproducible.

Session information: recorded

Great job! Recording the operating system, R version, and package versions is critical for reproducibility.

Cache: none

Nice! There were no cached chunks for this analysis, so you can be confident that you successfully produced the results during this run.

File paths: relative

Great job! Using relative paths to the files within your workflowr project makes it easier to run your code on other machines.

Repository version: 9b4272d

Great! You are using Git for version control. Tracking code development and connecting the code version to the results is critical for reproducibility.

The results in this page were generated with repository version 9b4272d. See the Past versions tab to see a history of the changes made to the R Markdown and HTML files.

Note that you need to be careful to ensure that all relevant files for the analysis have been committed to Git prior to generating the results (you can use wflow_publish or wflow_git_commit). workflowr only checks the R Markdown file, but you know if there are other scripts or data files that it depends on. Below is the status of the Git repository when the results were generated:


Ignored files:
    Ignored:    .Rhistory
    Ignored:    .Rproj.user/
    Ignored:    .tresorit/
    Ignored:    _targets/
    Ignored:    data/VR_20051125.txt.xz
    Ignored:    data/VR_Snapshot_20081104.txt.xz
    Ignored:    output/blk_char.fst
    Ignored:    output/ent_blk.fst
    Ignored:    output/ent_cln.fst
    Ignored:    output/ent_raw.fst
    Ignored:    renv/library/
    Ignored:    renv/local/
    Ignored:    renv/staging/

Untracked files:
    Untracked:  analysis/m_01_7_check_demog.Rmd

Unstaged changes:
    Modified:   _targets.R
    Deleted:    analysis/01-3_check_resid.Rmd
    Modified:   analysis/index.Rmd
    Modified:   analysis/m_01_5_check_admin.Rmd
    Modified:   analysis/m_01_6_check_resid.Rmd

Note that any generated files, e.g. HTML, png, CSS, etc., are not included in this status report because it is ok for generated content to have uncommitted changes.

There are no past versions. Publish this analysis with wflow_publish() to start tracking its development.

# NOTE this notebook can be run manually or automatically by {targets}
# So load the packages required by this notebook here
# rather than relying on _targets.R to load them.

# Set up the project environment, because {workflowr} knits each Rmd file 
# in a new R session, and doesn't execute the project .Rprofile

library(targets) # access data from the targets cache

library(tictoc) # capture execution time
library(here) # construct file paths relative to project root
library(fs) # file system operations
library(vroom) # fast reading of delimited text files
library(tibble) # enhanced data frames
library(stringr) # string matching
library(skimr) # compact summary of each variable
library(lubridate) # date parsing


Attaching package: 'lubridate'

The following objects are masked from 'package:base':

    date, intersect, setdiff, union

library(forcats) # manipulation of factors
library(ggplot2) # graphics
library(tidyr) # data tidying

# start the execution time clock
tictoc::tic("Computation time (excl. render)")

# Get the path to the raw entity data file
# This is a target managed by {targets}
f_entity_raw_tsv <- tar_read(c_raw_entity_data_file)

1 Introduction

The aim of this set of meta notebooks is to work out how to read the raw entity data. and get it sufficiently neatened so that we can construct standardised names and modelling features without needing any further neatening. To be clear, the target (c_raw_entity_data) corresponding to the objective of this set of notebooks is the neatened raw data, before constructing any modelling features.

This notebook documents the checking of the deemographic variables for any issues that need fixing. These are the non-name variables that are reasonably interpretable as properties of the person. The subsequent notebooks in this set will checking the other variables for any issues that need fixing.

We will probably use some of these variables as predictors in a compatibility model and/or as blocking variables.

Regardless of whether there are any issues that need to be fixed, the analyses here may inform our use of these variables in later analyses.

We have no intention of using the residence variables as predictors for entity resolution. However, they may be of use for manually checking the results of entity resolution. Consequently, the checking done here is minimal.

Define the demographic variables.

sex_code - Gender code
sex - Gender description
age - Age at snapshot date (years)
birth_place - Birth place

vars_resid <- c(
  "sex_code", "sex", "age", "birth_place" 
)

2 Read entity data

Read the raw entity data file using the previously defined functions raw_entity_data_read(), raw_entity_data_excl_status(), raw_entity_data_excl_test(), raw_entity_data_drop_novar(), raw_entity_data_parse_dates(), and raw_entity_data_drop_cancel_dt().

# Show the data file name
fs::path_file(f_entity_raw_tsv)

[1] "VR_20051125.txt.xz"

d <- raw_entity_data_read(f_entity_raw_tsv) %>% 
  raw_entity_data_excl_status() %>% 
  raw_entity_data_excl_test() %>% 
  raw_entity_data_drop_novar() %>% 
  raw_entity_data_parse_dates() %>% 
  raw_entity_data_drop_cancel_dt()
dim(d)

[1] 4099699      24

Take a quick look at the distributions.

d %>% 
  dplyr::select(sex_code, sex, age, birth_place) %>% 
  skimr::skim()

Table 2.1: Data summary
Name	Piped data
Number of rows	4099699
Number of columns	4
_______________________
Column type frequency:
character	4
________________________
Group variables	None

Variable type: character

skim_variable	n_missing	complete_rate	min	max	n_unique
sex_code	0	1.00	1	1	3
sex	0	1.00	3	6	3
age	0	1.00	1	3	135
birth_place	718647	0.82	2	2	56

sex_code 100% filled
sex 100% filled
age 100% filled
birth_place 82% filled

3 sex_code & sex

sex_code Gender code
sex Gender description

d %>% 
  with(table(sex_code, sex, useNA = "ifany"))

        sex
sex_code  FEMALE    MALE     UNK
       F 2239888       0       0
       M       0 1844220       0
       U       0       0   15591

sex_code and sex in 1-1 relationship

4 birth_place

birth_place Birth place

table(d$birth_place,  useNA = "ifany") %>% sort() %>% rev()


     NC    <NA>      NY      VA      SC      PA      OC      OH      FL      NJ 
1875088  718647  189726  131356   97827   93496   76209   75765   69032   67588 
     GA      CA      MI      IL      WV      TX      TN      MD      MA      IN 
  56176   50560   48081   47613   42143   37852   36797   36369   33519   26719 
     KY      AL      DC      CT      MO      WI      LA      CO      MS      IA 
  24276   23877   22563   22394   16066   15663   15362   12803   12047   10891 
     MN      OK      WA      KS      AR      ME      RI      NE      DE      AZ 
  10388    9609    9083    8656    6614    6284    6039    5592    5373    5043 
     NH      HI      VT      OR      NM      AK      UT      PR      ND      SD 
   4880    3870    3783    3764    3435    3201    3088    2591    2399    2240 
     ID      MT      NV      WY      VI      GU      AS 
   2003    1901    1542    1280     355     149      32

birth_place values appear to be 2-character US state abbreviations
OC might mean “other country”
VI, GU, AS are probably Virgin Islands, Guam, American Samoa

5 age

age Age (years)

I presume that the source documents actually record date of birth rather than age, and that age is reported in these files as a gesture to privacy.

Look at the distribution of age.

x <- d %>% 
  dplyr::mutate(age = as.integer(age))
                
x$age %>% summary()

   Min. 1st Qu.  Median    Mean 3rd Qu.    Max. 
   0.00   33.00   45.00   46.93   58.00  221.00

x$age %>% quantile(probs = c(0.003, 0.004, 0.995, 0.996, 0.997, 0.998, 0.999))

 0.3%  0.4% 99.5% 99.6% 99.7% 99.8% 99.9% 
    0    18    98   105   105   105   204

x %>% 
  # dplyr::filter(age >= 80) %>% 
  ggplot() +
  geom_vline(xintercept = c(17, 105, 125, 204), colour = "orange") +
  geom_histogram(aes(x = age), binwidth = 1) +
  scale_y_log10()

Warning: Transformation introduced infinite values in continuous y-axis

Warning: Removed 87 rows containing missing values (geom_bar).

Voters may be pre-registered at 17 years
The ages less than 17 years (especially zero) are effectively missing
The spikes at 105, 125, and 204 years must be from some special process which generates those specific ages.
Any age greater than or equal to 105 years, seems very implausible.

Timing

Computation time (excl. render): 175.757 sec elapsed

sessionInfo()

R version 4.0.3 (2020-10-10)
Platform: x86_64-pc-linux-gnu (64-bit)
Running under: Ubuntu 20.10

Matrix products: default
BLAS:   /usr/lib/x86_64-linux-gnu/blas/libblas.so.3.9.0
LAPACK: /usr/lib/x86_64-linux-gnu/lapack/liblapack.so.3.9.0

locale:
 [1] LC_CTYPE=en_AU.UTF-8       LC_NUMERIC=C              
 [3] LC_TIME=en_AU.UTF-8        LC_COLLATE=en_AU.UTF-8    
 [5] LC_MONETARY=en_AU.UTF-8    LC_MESSAGES=en_AU.UTF-8   
 [7] LC_PAPER=en_AU.UTF-8       LC_NAME=C                 
 [9] LC_ADDRESS=C               LC_TELEPHONE=C            
[11] LC_MEASUREMENT=en_AU.UTF-8 LC_IDENTIFICATION=C       

attached base packages:
[1] stats     graphics  grDevices datasets  utils     methods   base     

other attached packages:
 [1] tidyr_1.1.3      ggplot2_3.3.3    forcats_0.5.1    lubridate_1.7.10
 [5] skimr_2.1.3      stringr_1.4.0    tibble_3.1.0     vroom_1.4.0     
 [9] fs_1.5.0         tictoc_1.0       here_1.0.1       workflowr_1.6.2 
[13] targets_0.3.1   

loaded via a namespace (and not attached):
 [1] Rcpp_1.0.6        ps_1.6.0          assertthat_0.2.1  rprojroot_2.0.2  
 [5] digest_0.6.27     utf8_1.2.1        R6_2.5.0          repr_1.1.3       
 [9] evaluate_0.14     highr_0.8         pillar_1.5.1      rlang_0.4.10     
[13] data.table_1.14.0 callr_3.6.0       jquerylib_0.1.3   rmarkdown_2.7    
[17] labeling_0.4.2    igraph_1.2.6      bit_4.0.4         munsell_0.5.0    
[21] compiler_4.0.3    httpuv_1.5.5      xfun_0.22         pkgconfig_2.0.3  
[25] base64enc_0.1-3   htmltools_0.5.1.1 tidyselect_1.1.0  bookdown_0.21    
[29] codetools_0.2-18  fansi_0.4.2       crayon_1.4.1      dplyr_1.0.5      
[33] withr_2.4.1       later_1.1.0.1     grid_4.0.3        jsonlite_1.7.2   
[37] gtable_0.3.0      lifecycle_1.0.0   DBI_1.1.1         git2r_0.28.0     
[41] magrittr_2.0.1    scales_1.1.1      cli_2.3.1         stringi_1.5.3    
[45] farver_2.1.0      renv_0.13.2       promises_1.2.0.1  bslib_0.2.4      
[49] ellipsis_0.3.1    generics_0.1.0    vctrs_0.3.7       tools_4.0.3      
[53] bit64_4.0.5       glue_1.4.2        purrr_0.3.4       parallel_4.0.3   
[57] processx_3.5.0    yaml_2.2.1        colorspace_2.0-0  knitr_1.31       
[61] sass_0.3.1

[meta] Check demographic variables

m_01_7_check_demog

Ross Gayler

2021-04-03