Lecture 5

Reading and Transforming Data with R

Byeong-Hak Choe

bchoe@geneseo.edu

SUNY Geneseo

September 29, 2026

📂 Reading a Data Table as a data.frame

📄 A CSV file is a plain-text table

• A CSV (comma-separated values) file is a plain text file where each value is separated by a comma.
  • The first line usually gives the column names.
  • Each later line supplies one row of values.
  • The commas separate columns.
  • CSV files are widely used for storing data from spreadsheets and databases.
• Example
  • https://bcdanl.github.io/data/tvshows.csv

📍 An absolute pathname gives a file’s full location

• An absolute pathname tells the computer the exact location of a file, starting from the very top folder of your computer.
  • It does not depend on R’s working directory, but it changes if the file is moved or renamed.

Computer	Example of an absolute pathname
Mac	/Users/USERNAME/Downloads/custdata_rev.csv
Windows	C:\Users\USERNAME\Downloads\custdata_rev.csv

Windows displays the pathname with backslashes (\). When entering it in R, use forward slashes (/) instead:

library(tidyverse)
df <- read_csv("/cloud/project/data/custdata_rev.csv")

• R treats a single backslash (\) as a special character inside quotation marks.

🧭 A relative pathname starts at R’s working directory

The working directory is the folder R uses as its starting point. Run getwd() in the Console to see it.

Role	Example
Given: working directory	/cloud/project
Given: CSV file’s full location	/cloud/project/data/custdata_rev.csv
Result: relative pathname	data/custdata_rev.csv

getwd()
custdata <- read_csv("data/custdata_rev.csv")

📥 Read a downloaded CSV into R

1. Download custdata_rev.csv from the Class Files module in Brightspace.
   • The file will usually be saved in your computer’s Downloads folder.
   • Do not open this file in a spreadsheet app (e.g., Excel or Numbers).
2. In Posit Cloud, create a subfolder named data (Files Pane → + Folder).
   
3. Click the data folder, then upload the custdata_rev.csv file into it.
   • At the top of the Files Pane, click Upload to choose and add your file.
4. Read the file using a relative pathname:

# Read the CSV file from the data subfolder
custdata <- read_csv("data/custdata_rev.csv")

🧪 Classwork 5: Absolute Pathnames

Classwork 5: Finding an Absolute Pathname →

Find the downloaded file Copy its absolute pathname Paste it into a text editor

🌐 A CSV can also be read from a web address

# Read the CSV file directly from the web (GitHub repo)
custdata_web <- read_csv(
        'https://bcdanl.github.io/data/custdata_rev.csv')

• We can load a CSV file directly from the web into R.

🔎 Getting to Know a data.frame

🔎 Inspect the size and contents of custdata

class(custdata)
dim(custdata)       # number of rows and columns
nrow(custdata)      # number of rows
ncol(custdata)      # number of columns
names(custdata)     # column names

summary(custdata)

# Install once
install.packages("skimr") 

library(skimr)
skim(custdata)
custdata_sum <- skim(custdata)

• dim() shows both the number of rows and columns.
  
• nrow() and ncol() give the row count and column count separately.
  
• summary() gives a quick overview, while skimr::skim() provides a more detailed, user-friendly summary of variables across all data types.

💲 operator extracts a column as a vector

The $ operator extracts a column from a data.frame as a vector.

custdata$age
skim(custdata$age)

🧹 Tidy data gives each value a clear place

In this example, one observation is one customer’s spending in one month.

Customer	Month	Spending
Ana	January	42
Ana	February	38
Bo	January	55

1. Each variable has its own column.
2. Each observation has its own row.
3. Each value has its own cell.

👥 Observations in data.frame

• Rows in a data.frame represent individual units or entities for which data is collected.
  
• Examples:
  • Student Information: Each row = one student
    
  • Employee Information: Each row = one employee
    
  • Daily S&P 500 Index Data: Each row = one trading day
    
  • Household Survey Data: Each row = one household

🏷️ Variables in data.frame

• Columns in a data.frame represent attributes or characteristics measured across multiple observations.
  
• Examples:
  • Student Data: Name, Age, Grade, Major
    
  • Employee Data: EmployeeID, Name, Age, Department
    
  • Customer Data: CustomerID, Name, Age, Income, HousingType

Note

• In a data.frame, a variable is a column of data.
  
• In general programming, a variable is the name of an object.
  

🔄 Data Transformation with dplyr

🛠️ dplyr changes a data.frame in useful ways

• dplyr is a core tidyverse package for data manipulation — tasks like filtering, sorting, selecting, and renaming.
  

Function	What changes?
filter(DF, LOGICAL_CONDITIONS)	Keep rows that meet a condition
arrange(DF, VARIABLES)	Sort rows
distinct(DF, VARIABLES)	Keep unique rows or combinations
select(DF, VARIABLES)	Select or remove columns
rename(DF, NEW_NAME = CURRENT_NAME)	Rename a column name

• dplyr functions take a data.frame as a first argument.
  • They return a data.frame as output.

🔗 The pipe connects steps from left to right

• The pipe operator (|> or %>%) passes the result on its left into the function on its right
  • It makes code easier to read by connecting steps in order.
• How it works:
  • f(x, y) is the same as x |> f(y).
    
  • Example with a data frame DF:
    • filter(DF, logical_condition)
      
    • is the same as
      DF |> filter(logical_condition)
• You can read the pipe as “then”:
  • Example: Start with DF, then keep rows where logical_condition is TRUE.

🔗 The pipe works well with dplyr verbs

• Common dplyr functions with the pipe operator:
  • DF |> filter(LOGICAL_CONDITIONS)
    
  • DF |> arrange(VARIABLES)
    
  • DF |> distinct(VARIABLES)
    
  • DF |> select(VARIABLES)
    
  • DF |> rename(NEW_NAME = CURRENT_NAME)
• Why this works so well:
  • dplyr functions usually take a data.frame as the first argument.
    
  • They return a data.frame as output.
• This allows you to chain together multiple steps naturally:
  • Example: Start with data, then filter, then arrange
    \(\qquad\quad\;\;\) DF |> filter(...) |> arrange(...)

☁️ Insert the pipe in RStudio or Posit Cloud

• To enable the native pipe operator (|>) in RStudio:
  • Go to Tools > Global Options > Code (side menu).
    
  • Under Pipe operator, choose Use native pipe operator (|>).

• Keyboard shortcut for inserting a pipe:
  • Windows: Ctrl + Shift + M
    
  • Mac: Cmd + Shift + M

🎯 Filter observations with filter()

🔢 One small table for our filtering examples

Row	num	chr
1	8	A
2	9	C
3	10	B
4	11	A

Create the same table in R:

df <- data.frame(
  num = c(8, 9, 10, 11),
  chr = c("A", "C", "B", "A")
)

We will use this df on the next slides.

⚖️ Mathematical comparisons produce TRUE or FALSE

Condition	Meaning
V1 == V2	Is V1 equal to V2?
V1 != V2	Is V1 not equal to V2?
V1 > V2	Is V1 greater than V2?
V1 >= V2	Is V1 greater than or equal to V2?
V1 < V2	Is V1 less than V2?
V1 <= V2	Is V1 less than or equal to V2?

• Here, both V1 and V2 are variables, and the comparisons are applied element-wise (vectorized).

• For logical conditions using inequalities, we focus on cases where V1 and V2 are integer or numeric.

filter() keeps only the rows where the condition is TRUE

df <- data.frame(
  num = c(8, 9, 10, 11),
  chr = c("A", "C", "B", "A")
  )

df |> 
  filter(num > 8)

df |> 
  filter(chr == "A")

🔣 Combine conditions with logical operators — AND, OR, NOT

• Here, both x and y are logical conditions/variables.

• What logical operations (&, |, !) do is combining logical variables/conditions, returning a logical variable when executed.

• & requires both conditions.

• | requires at least one condition.

• ! reverses a condition.

🧠 Venn diagram visualizes logical operations

• x and y are logical conditions.
  • If x is TRUE, it highlights the left circle.
    
  • If y is TRUE, it highlights the right circle.

• The shaded regions show which parts each logical operator selects.

& or a comma keeps rows that meet both conditions

df <- data.frame(
  num = c(8, 9, 10, 11),
  chr = c("A", "C", "B", "A")
  )

df |> 
  filter(num > 8 & 
         num < 11)
                
df |> 
  filter(num > 8,
         num < 11)

• In filter(), separating conditions with a comma is equivalent to combining them with the & operator.

& can combine conditions on different columns

df <- data.frame(
  num = c(8, 9, 10, 11),
  chr = c("A", "C", "B", "A")
  )

df |> 
  filter(num < 10 & 
         chr == "A")

df |> 
  filter(num < 10, 
         chr == "A")

| keeps rows that meet at least one condition

df <- data.frame(
  num = c(8, 9, 10, 11),
  chr = c("A", "C", "B", "A")
  )

df |> 
  filter(num < 10 | 
         chr == "A")

✈️ Apply the same conditions to flights

The nycflights13 package supplies the flights data frame.

# Run once if the package is not installed:
# install.packages("nycflights13")
library(nycflights13)
library(dplyr)

# Flights in January or December
flights |> filter(month == 1 | month == 12)

# Flights on January 1
flights |> filter(month == 1, day == 1)

Each condition checks the values in its column, one row at a time.

❓ NA means a value is missing or unknown

NA > 5
10 == NA
NA + 10
NA / 2
(1 + NA + 3) / 3

mean( c(1, NA, 3) )
sd( c(1, NA, 3) )

• NA (not available) represents a missing or unknown value in R.

• In most calculations, if one value is unknown, the result will also be unknown (NA).

🧮 na.rm = TRUE removes missing values for calculation

• Without na.rm = TRUE, both functions return NA for this vector.

mean(c(1, NA, 3))  # NA
sd(c(1, NA, 3))    # NA

• na.rm = TRUE removes missing values for the calculation.

mean(c(1, NA, 3), na.rm = TRUE)  # 2
sd(c(1, NA, 3), na.rm = TRUE)    # 1.414214

• Here, R uses only non-missing values, 1 and 3, for calculation. The original data stay unchanged.

The argument name na.rm stands for removing NA values. Its default is FALSE. Setting it to TRUE excludes missing values from this calculation. It does not replace NA with zero, fill in an unknown value, or edit the original vector. The mean of 1 and 3 is 2. R’s sd() returns the sample standard deviation, which is sqrt(2), approximately 1.414214. Before ignoring missing values in real data, consider why they are missing and how many observations remain.

🆚 Comparing Missing Values

v1 <- NA
v2 <- NA
v1 == v2

• Suppose v1 is Mary’s age (unknown) and v2 is John’s age (unknown).
• Can we say they are the same age?

• Since both values are missing, R cannot know — so the result is also NA.

🔎 Checking for Missing Values with is.na()

num_missing <- NA
is.na(num_missing)  # is num_missing NA?

text_missing <- "missing"
is.na(text_missing) # is text_missing NA?

V1 <- c(1, NA, 3)
is.na(V1)  # is V1 NA?
!is.na(V1)  # is V1 not NA?

df <- data.frame(
  v1 = c(1, NA, 3),
  v2 = c(1, 2, 3)
  )

df |> 
  filter( is.na(v1) )

df |> 
  filter( !is.na(v1) )

• Use is.na() to test whether a value is missing (NA).

• In filter(), you can:
  • Use is.na() to keep observations with missing values.
  • Use !is.na() to remove observations with missing values.

↕️ Arrange Observations with arrange()

⬆️ arrange() sorts whole observations

# Sort observations 
  # by dep_delay (ascending)
flights |> 
  arrange(dep_delay)

# Sort by dep_delay, 
  # then by sched_dep_time
flights |> 
  arrange(dep_delay, sched_dep_time)

• arrange() sorts out observations.
  • Missing values appear at the end.
• When you provide multiple variables, arrange() sorts by the first variable, and then uses the next variable(s) to break ties.

⬇️ desc() reverses the sorting direction

# Sort by departure delay 
  # in descending order
flights |> 
  arrange(desc(dep_delay))

# Equivalent shortcut 
  # for numeric variables
flights |> 
  arrange(-dep_delay)

• Use desc(VARIABLE) to sort in descending order.
  • For numeric variables, you can also use a leading minus sign.

🔢 Each extra variable in arrange() breaks ties in the one before it

df <- data.frame(
  year = c(2024, 2021, 2024, 2024),
  month = c(7, 10, 7, 4),
  day = c(20, 19, 15, 9)
)

df |> 
  arrange(year, month, day)

• If we provide more than one variable name, each additional variable will be used to break ties in the values of preceding variables.

🆔 Find All Unique Observations with distinct()

🧹 distinct() removes repeated rows

df <- data.frame(
  country = c("USA", "Korea", "USA"),
  city = c("D.C.", "Seoul", "D.C."),
  subregion = c("Georgetown", 
                "Gangnam", 
                "Georgetown") 
  )

# Remove duplicate observations
df |> 
  distinct()

• distinct() removes duplicate observations.
  • By default, it checks all variables to find unique observations.

🧩 distinct() can check selected columns

# Remove duplicate observations 
  # from the flights data.frame
flights |> 
  distinct()

# Find all unique 
  #  origin-destination pairs
flights |> 
  distinct(origin, dest)

• You can pass one or more variables to distinct().
  • This returns unique combinations of just those variables (while ignoring the rest).

🧪 Classwork 6: Data Transformation - Rows

Classwork 6: Data Transformation - Rows →

filter() arrange() distinct()

🎯 Select Variables with select()

📌 select() keeps or removes variables

• It’s not uncommon to get datasets with hundreds or thousands of variables.

• select() allows us to narrow in on the variables we’re actually interested in.

• We can select variables by their names.

flights |> 
  select(year, month, day)

• With select(-VARIABLES), we can remove variables.

flights |> 
  select(-year)

🏷️ rename() changes a variable name

flights |> 
  rename( tail_num = tailnum )

• rename() can be used to rename variables:

  • DF |> rename(NEW_NAME = CURRENT_NAME)

🔄 Put the transformation steps together

flights |>
  filter(month == 1, !is.na(dep_delay)) |>
  arrange(desc(dep_delay)) |>
  select(origin, dest, dep_delay) |>
  rename(departure_delay = dep_delay)

• Read the result:
  1. January flights with a known departure delay
  2. ordered from the largest delay down
  3. selecting only three variables
  4. renaming dep_delay to the clearer name departure_delay

• Check yourself: Which function changed the observations, which changed their order, and which changed the variables?

🧪 Classwork 7: Data Transformation - Columns

Classwork 7 Data Transformation - Columns →

select() rename()