Transform + organize · Week 07

Data Transformation + Big Data

Finish transforming rows and columns in R, practice with two classworks, then begin Big Data and the Modern Data Infrastructure on Wednesday, October 7.

Oct 5–9Lectures 5–6 + Classworks 6–7
ImportantFrom transformation to databases
  1. Continue Lecture 5 after filter(), starting with arrange(), then cover distinct(), select(), and rename().
  2. Complete Classwork 6 (transforming rows) and Classwork 7 (transforming columns).
  3. Begin Lecture 6: Big Data and the Modern Data Infrastructure on Wednesday, October 7.

Homework 2 is due October 9 at 11:59 PM. Data visualization will begin after the October 19 midterm.

Focus

  • Sorting rows and finding unique observations
  • Selecting and renaming columns
  • An introduction to big data and data infrastructure

Prepare

  • Reopen your Lecture 5 R script and review filter()
  • Have Classworks 6 and 7 ready in Posit Cloud
  • Open Lecture 6 for Wednesday and finish Homework 2 by Friday

Practice

  • Transform flight data with filter(), arrange(), and distinct()
  • Select and rename variables in NYC payroll data
  • Connect a single data frame to an organization’s broader data needs

Learning targets

  • Sort rows with arrange() and change the sorting direction with desc().
  • Keep unique rows or combinations of values with distinct().
  • Keep or remove columns with select() and change their names with rename().
  • Combine transformation steps in a pipe and check that the result answers the question.
  • Begin distinguishing data types and explaining how databases and modern infrastructure support analytics.
Tip

Check the result after each transformation: Did you keep the intended rows, sort them in the intended direction, and retain the columns needed for the question?

Core pattern

flights |>
  filter(month == 1, !is.na(dep_delay)) |>
  arrange(desc(dep_delay)) |>
  select(origin, dest, dep_delay)

Lecture slides

TipLecture slide shortcuts

Click inside the slide preview, then use:

  • the visible ←/→ controls to page through the slides;
  • your mouse wheel or trackpad to keep scrolling the weekly page;
  • M to open the menu;
  • F to enter fullscreen;
  • E to enter or exit PDF export mode;
  • Ctrl + Shift + F to search within the slides;
  • Esc to exit the menu or fullscreen; and
  • T to switch between dark and light mode.
Lecture 5 · Data Transformation · continue with arrange()View slides in new tab
Lecture 6 · Big Data and the Modern Data Infrastructure · begins WednesdayView slides in new tab

This week’s materials

Lecture 5 · Data Transformation — continue with arrange()Lecture 6 · Big Data and the Modern Data Infrastructure — begins WednesdayClasswork 6 · Data Transformation — filter(), arrange(), and distinct()Classwork 7 · Data Transformation — select() and rename()Homework 2 · Descriptive Statistics and Reading and Transforming Data with R — due Oct 9, 11:59 PM
Back to top