Getting data into R

Thus far, we've already learned what R and RStudio are. There's one essential prerequisite:

We need data!

Content of this session

• What are R's internal data types?
• How to work with different data types?
• How to import data in different formats?
• How to export data in different formats

Data built in

data()Copy Code

It boils all down to...

https://www.stat.berkeley.edu/~nolan/stat133/Fall05/lectures/DataTypes4.pdf

Numeric data

1LCopy Code

## [1] 1Copy Code

1.1Copy Code

## [1] 1.1Copy Code

typeof(1L)Copy Code

## [1] "integer"Copy Code

typeof(1.1)Copy Code

## [1] "double"Copy Code

Character strings

At first glance, a character is a letter somewhere between a-z. String in this context might mean that we have a series of characters. However, numbers and other symbols can be part of a character string, which can then be, e.g., part of a text. In R, character strings are wrapped in quotation marks.

"Hi. I am a character string, the 1st of its kind!"Copy Code

## [1] "Hi. I am a character string, the 1st of its kind!"Copy Code

Note: There are no values associated with the content of character strings unless we change that, e.g., with factors.

Factors

Factors are data types that assume that their values are not continuous, e.g., as in ordinal or nominal data.

factor(1.1)Copy Code

## [1] 1.1
## Levels: 1.1Copy Code

factor("Hi. I am a character string, the 1st of its kind!")Copy Code

## [1] Hi. I am a character string, the 1st of its kind!
## Levels: Hi. I am a character string, the 1st of its kind!Copy Code

Factors take numeric data or character strings as input as they simply convert them into so-called levels.

Logical values

Logical values are basically either TRUE or FALSE values. These values are produced by making logical requests on your data.

2 > 1Copy Code

## [1] TRUECopy Code

2 < 1Copy Code

## [1] FALSECopy Code

Logical values are at the heart of creating loops. For this purpose, however, we need more logical operators to request TRUE or FALSE values.

Logical operators

There are quite a few logical operators in R:

https://www.statmethods.net/management/operators.html

Moreover, there are some more is.PROPERTY_ASKED_FOR() functions, such as is.numeric(), which also return TRUE or FALSE values.

R's data formats

R's different data types can be put into 'containers'.

https://devopedia.org/r-data-structures

Vectors

Vectors are built by enclosing your content with c() ("c" for "concatenate")

numeric_vector   <- c(1, 2, 3, 4)
character_vector <- c("a", "b", "c", "d")
numeric_vectorCopy Code

## [1] 1 2 3 4Copy Code

character_vectorCopy Code

## [1] "a" "b" "c" "d"Copy Code

Vectors are really like vectors in mathematics. Initially, it doesn't matter if you look at them as column or row vectors.

...but it matters when you combine vectors

Using the function cbind() or rbind() you can either combine vectors column-wise or row-wise. Thus, they become matrices.

cbind(numeric_vector, character_vector)Copy Code

##      numeric_vector character_vector
## [1,] "1"            "a"             
## [2,] "2"            "b"             
## [3,] "3"            "c"             
## [4,] "4"            "d"Copy Code

rbind(numeric_vector, character_vector)Copy Code

##                  [,1] [,2] [,3] [,4]
## numeric_vector   "1"  "2"  "3"  "4" 
## character_vector "a"  "b"  "c"  "d"Copy Code

Matrices

Matrices are the basic rectangular data format in R.

fancy_matrix <- matrix(1:16, nrow = 4)
fancy_matrixCopy Code

##      [,1] [,2] [,3] [,4]
## [1,]    1    5    9   13
## [2,]    2    6   10   14
## [3,]    3    7   11   15
## [4,]    4    8   12   16Copy Code

You cannot store multiple data types, such as strings and numeric values in the same matrix. Otherwise, your data will get coerced to a common type, as seen in the previous slide. This is something that happens already within vectors:

c(1, 2, "evil string")Copy Code

## [1] "1"           "2"           "evil string"Copy Code

Data frames

library(randomNames) # a name generator package
fancy_data <-
  data.frame( 
    who = 
      randomNames(n = 10, which.names = "first"),
    age = 
      sample(14:49, 10, replace = TRUE), # you see what we are doing here?   
    salary_2018 = 
      sample(15:100, 10, replace = TRUE),  
    salary_2019 = 
      sample(15:100, 10, replace = TRUE)
  )
fancy_dataCopy Code

↪️

Tibbles

Tibble conversion

library(tibble)
as_tibble(fancy_data)Copy Code

## # A tibble: 10 × 4
##    who        age salary_2018 salary_2019
##    <chr>    <int>       <int>       <int>
##  1 Joseph      27          30          93
##  2 Asha        37          40          99
##  3 Emily       23          86          18
##  4 Michaela    23          77          68
##  5 Jordan      38          52          66
##  6 Burhaan     40          92          48
##  7 Lasandra    36          45          41
##  8 Caleb       31          28          15
##  9 Angelica    39          91          80
## 10 Alfred      38          97          60Copy Code

One last type you should know: lists

Lists are perfect for storing numerous and potentially diverse pieces of information in one place.

fancy_list <- 
  list(
    numeric_vector,
    character_vector,
    fancy_matrix,
    fancy_data
  )
fancy_listCopy Code

↪️

Nested lists

fancy_nested_list <-
  list(
    fancy_vectors = list(numeric_vector, character_vector),
    data_stuff = list(fancy_matrix, fancy_data)
  )
fancy_nested_listCopy Code

↪️

Accessing elements by index

Generally, the logic of [index_number] is to access only a subset of information in an object, no matter if we have vectors or data frames.

Say, we want to extract the 2nd element of our character_vector object, we could do that like this:

character_vector[2]Copy Code

## [1] "b"Copy Code

More complicated cases: matrices

Matrices can have more dimensions, often you want information from a specific row and column.

a_wonderful_matrix[number_of_row, number_of_column]Copy Code

Note: You can do the same indexing with data.frames.

Matrices and subscripts (as in mathematical notation)

Identifying rows, columns, or elements using subscripts is similar to matrix notation:

fancy_matrix[, 4] # 4th column of matrix
fancy_matrix[3,] # 3rd row of matrix
fancy_matrix[2:4, 1:3] # rows 2,3,4 of columns 1,2,3Copy Code

The case of data frames

A nice feature of data.frames or tibbles is that their columns are names, just as variable names in ordinary data.

fancy_data$whoCopy Code

##  [1] "Joseph"   "Asha"     "Emily"    "Michaela" "Jordan"   "Burhaan" 
##  [7] "Lasandra" "Caleb"    "Angelica" "Alfred"Copy Code

Just place a $-sign between the data object and the variable name.

[] in data frames

Sometimes we also have to rely on character strings as input information, e.g., for iterating over data. We can also use [] to access variables by name.

fancy_data[1]Copy Code

##         who
## 1    Joseph
## 2      Asha
## 3     Emily
## 4  Michaela
## 5    Jordan
## 6   Burhaan
## 7  Lasandra
## 8     Caleb
## 9  Angelica
## 10   AlfredCopy Code

fancy_data["who"]Copy Code

##         who
## 1    Joseph
## 2      Asha
## 3     Emily
## 4  Michaela
## 5    Jordan
## 6   Burhaan
## 7  Lasandra
## 8     Caleb
## 9  Angelica
## 10   AlfredCopy Code

Difference between [] and [[]]

https://twitter.com/hadleywickham/status/643381054758363136

Data frame check 1, 2, 1, 2!

The most high-level information you can get is about the object type and its dimensions.

# object type
class(fancy_data)Copy Code

## [1] "data.frame"Copy Code

# number of rows and columns
dim(fancy_data)Copy Code

## [1] 10  4Copy Code

# number of rows
nrow(fancy_data)Copy Code

## [1] 10Copy Code

# number of columns
ncol(fancy_data)Copy Code

## [1] 4Copy Code

Data frame check 1, 2, 1, 2!

You can also print the first 6 lines of the data frame with head(). You can easily change the number of lines by providing the number as the second argument to the head() function.

head(fancy_data, 3)Copy Code

##      who age salary_2018 salary_2019
## 1 Joseph  27          30          93
## 2   Asha  37          40          99
## 3  Emily  23          86          18Copy Code

Data frame check 1, 2, 1, 2!

If we want some more (detailed) information about the data set or object, we can use the base R function str().

str(fancy_data)Copy Code

## 'data.frame':    10 obs. of  4 variables:
##  $ who        : chr  "Joseph" "Asha" "Emily" "Michaela" ...
##  $ age        : int  27 37 23 23 38 40 36 31 39 38
##  $ salary_2018: int  30 40 86 77 52 92 45 28 91 97
##  $ salary_2019: int  93 99 18 68 66 48 41 15 80 60Copy Code

Data frame check 1, 2, 1, 2!

If you want to have a look at your full data set, you can use the View() function. In RStudio, this will open a new tab in the source pane through which you can explore the data set (including a search function). You can also click on the small spreadsheet symbol on the right side of the object in the environment tab to open this view.

View(fancy_data)Copy Code

Viewing and changing names

We can print all names of an object using the names() function...

names(fancy_data)Copy Code

## [1] "who"         "age"         "salary_2018" "salary_2019"Copy Code

...and we can also change names with it.

names(fancy_data) <- c("name", "age", "salary_2018", "salary_2019")
names(fancy_data)Copy Code

## [1] "name"        "age"         "salary_2018" "salary_2019"Copy Code

TidyTuesday Dataset

# Option 1: tidytuesdayR package 
## install.packages("tidytuesdayR")
tuesdata <- tidytuesdayR::tt_load('2024-09-03')
qname_levels_single_response_crosswalk <- tuesdata$qname_levels_single_response_crosswalk
stackoverflow_survey_questions <- tuesdata$stackoverflow_survey_questions
stackoverflow_survey_single_response <- tuesdata$stackoverflow_survey_single_response
# Option 2: Read directly from GitHub
qname_levels_single_response_crosswalk <- readr::read_csv('https://raw.githubusercontent.com/rfordatascience/tidytuesday/master/data/2024/2024-09-03/qname_levels_single_response_crosswalk.csv')
stackoverflow_survey_questions <- readr::read_csv('https://raw.githubusercontent.com/rfordatascience/tidytuesday/master/data/2024/2024-09-03/stackoverflow_survey_questions.csv')
stackoverflow_survey_single_response <- readr::read_csv('https://raw.githubusercontent.com/rfordatascience/tidytuesday/master/data/2024/2024-09-03/stackoverflow_survey_single_response.csv')Copy Code

Gapminder Data

How to use the data in general

To code along and be able to do the exercises, you should store the data files for the tuesdata in a folder called ./data in the same folder as the other materials for this course.

R is data-agnostic

Data formats & packages

What you will learn

• Getting the most common data formats into R
  • e.g., CSV, Stata, SPSS, or Excel spreadsheets
• Using the different methods of doing that
• We will rely a lot on packages and functions from the tidyverse instead of using base R

Before writing any code: RStudio functionality for importing data

You can use the RStudio GUI for importing data via Environment - Import data set - Choose file type.

Where to find data

Browse Button in RStudio

Code preview in Rstudio

Simple vs. not so simple file formats

Basic file formats, such as CSV (comma-separated value file), can directly be imported into R

• they are 'flat'
• few metadata
• basically text files

Other file formats, particularly the proprietary ones, require the use of additional packages

• they are complex
• a lot of metadata (think of all the labels in an SPSS file)
• they are binary (1110101)

Disclaimer

In the following slides, we'll jump right into importing data. We use a lot of different packages for this purpose, and you don't have to remember everything. It's just for making a point of how agnostic R actually is regarding the file type. Later on, we will dive more into the specifics of importing.

Importing a CSV file using base R

titanic <- read.csv("./data/titanic.csv")
titanicCopy Code

##   PassengerId Survived Pclass
## 1           1        0      3
## 2           2        1      1
## 3           3        1      3
## 4           4        1      1
## 5           5        0      3
## 6           6        0      3
##                                                  Name    Sex Age SibSp
## 1                             Braund, Mr. Owen Harris   male  22     1
## 2 Cumings, Mrs. John Bradley (Florence Briggs Thayer) female  38     1
## 3                              Heikkinen, Miss. Laina female  26     0
## 4        Futrelle, Mrs. Jacques Heath (Lily May Peel) female  35     1
## 5                            Allen, Mr. William Henry   male  35     0
## 6                                    Moran, Mr. James   male  NA     0
##   Parch           Ticket    Fare Cabin Embarked
## 1     0        A/5 21171  7.2500              S
## 2     0         PC 17599 71.2833   C85        C
## 3     0 STON/O2. 3101282  7.9250              S
## 4     0           113803 53.1000  C123        S
## 5     0           373450  8.0500              S
## 6     0           330877  8.4583              QCopy Code

A readr example: CSV files

library(readr)
titanic <- read_csv("./data/titanic.csv")Copy Code

Importing Excel files with readxl

library(readxl)
unicorns <- read_xlsx("./data/observations.xlsx")Copy Code

No output ☹️

Other data import options

These were just some very first examples of applying functions for data import from the different packages. There are many more...

Data type specifications for tibbles

• characters
  • indicated by <chr>
  • specified by col_character()
• integers
  • indicated by <int>
  • specified by col_integer()
• doubles
  • indicated by <dbl>
  • specified by col_double()
• factors
  • indicated by <fct>
  • specified by col_factor()
• logical
  • indicated by <lgl>
  • specified by col_logical()

Changing variable types

As mentioned before, read_csv 'guesses' the variable types by scanning the first 1000 observations. NB: This can go wrong!

Luckily, we can change the variable type...

• before/while loading the data
• and after loading the data

While loading the data in read_csv

titanic <-
  read_csv(
    "./data/titanic.csv",
    col_types = cols(
      PassengerId = col_double(),
      Survived = col_double(),
      Pclass = col_double(),
      Name = col_character(),
      Sex = col_character(),
      Age = col_double(),
      SibSp = col_double(),
      Parch = col_double(),
      Ticket = col_character(),
      Fare = col_double(),
      Cabin = col_character(),
      Embarked = col_character()
    )
  )
titanicCopy Code

↪️

While loading the data in read_csv

titanic <-
  read_csv(
    "./data/titanic.csv",
    col_types = cols(
      PassengerId = col_double(),
      Survived = col_double(),
      Pclass = col_double(),
      Name = col_character(),
      Sex = col_factor(), # This one changed!
      Age = col_double(),
      SibSp = col_double(),
      Parch = col_double(),
      Ticket = col_character(),
      Fare = col_double(),
      Cabin = col_character(),
      Embarked = col_character()
    )
  )
titanicCopy Code

↪️

After loading the data

titanic <-
  type_convert(
    titanic,
    col_types = cols(
      PassengerId = col_double(),
      Survived = col_double(),
      Pclass = col_double(),
      Name = col_character(),
      Sex = col_factor(),
      Age = col_double(),
      SibSp = col_double(),
      Parch = col_double(),
      Ticket = col_character(),
      Fare = col_double(),
      Cabin = col_character(),
      Embarked = col_character()
    )
  )Copy Code

Exporting data

Sometimes our data have to leave R, for example, if we....

• share data with colleagues who do not use R
• want to continue where we left off
  • particularly if data wrangling took a long time

For such purposes, we also need a way to export our data.

All of the packages we have discussed in this session also have designated functions for that.

Examples: CSV and Stata files

write_csv(titanic, "titanic_own.csv")Copy Code

R's native file formats

There are 2 native 'file formats' to choose from. The advantage of using them is that they are compressed files, so that they don't occupy unnecessarily large disk space. These two formats are .Rdata/.rda and .rds. The key difference between them is that .rds can only hold one object, whereas .Rdata/.rda can also be used for storing several objects in one file.

.Rdata/.rda

Saving

save(mydata, file = "mydata.RData")Copy Code

Loading

load("mydata.RData")Copy Code

.rds

Saving

saveRDS(mydata, "mydata.rds")Copy Code

Loading

mydata <- readRDS("mydata.rds")Copy Code

Note: A nice property of saveRDS() is that just saves a representation of the object, which means you can name it whatever you want when loading.

Saving just everything

If you have not changed the General Global Options in RStudio as suggested in the Getting Started session, you may have noticed that, when closing Rstudio, by default, the programs asks you whether you want to save the workspace image.

You can also do that whenever you want using the save.image() function:

save.image()Copy Code

Other packages for data import

For data import (and export) in general, there are even more options, such as...

• data.table or fst for large data sets

• jsonlite for .json files

Reminder regarding file paths

In general, you should avoid using absolute file paths to maintain your code reproducible and future-proof. We already talked about this in the introduction, but this is particularly important for importing and exporting data.

As a reminder: Absolute file paths look like this (on different OS):

# Windows
load("C:/Users/cool_user/data/fancy_data.Rdata")
# Mac
load("/Users/cool_user/data/fancy_data.Rdata")
# GNU/Linux
load("/home/cool_user/data/fancy_data.Rdata")Copy Code

Use relative paths

Instead of using absolute paths, it is recommended to use relative file paths. The general principle here is to start from a directory where your current script currently exists and navigate to your target location. Say we are in the "C:/Users/cool_user/" location on a Windows machine. To load your data, we would use:

load("./data/fancy_data.Rdata")Copy Code

If we were in a different folder, e.g., "C:/Users/cool_user/cat_pics/mittens/", we would use:

load("../../data/fancy_data.Rdata")Copy Code