Lecture 6

Data Types, Databases, and Big Data

Byeong-Hak Choe

bchoe@geneseo.edu

SUNY Geneseo

October 7, 2026

🗂️ Understanding Different Types of Data

🧭 Start with the data and the question

• Suppose a retailer wants to decide which products to stock next month.
• Its evidence may include sales tables, customer reviews, and product photos.
• Before analysis, ask:
  • What does one row represent?
  • What does each variable mean?
  • How can information from different sources be connected?

Discuss: Which source would help you count purchases? Which could help explain why customers liked a product?

🗃️ Data can be organized in different ways

Structured

Defined rows and columns.

Example: an order table with an ID, date, and total.

·

Semi-structured

Labels or tags organize information, but records may differ.

Example: an app's labeled data response.

·

Unstructured

The main content does not follow a fixed table layout.

Example: review text, photos, or videos.

• A review can have a structured date and rating alongside unstructured text.
• To analyze text or images in a table, we first identify useful features, such as a topic or image category.

🏷️ A variable’s meaning determines useful comparisons

Kind of variable	What the values mean	Useful comparison
Nominal	Categories without a ranking	Same or different
Ordinal	Categories with a ranking	Higher or lower
Interval	Equal differences; no meaningful zero	How much higher or lower
Ratio	Equal differences and a meaningful zero	How much; how many times as much

• These describe measurement, not R’s storage types such as character, numeric, or integer.
• A column of numbers may contain quantities, ratings, or ID labels. Read its definition before doing arithmetic.

🏷️ Nominal data names categories

Example: preferred shopping channel

Customer	Channel
C01	Website
C02	Store
C03	App

• Nominal data puts observations into categories with no natural ranking.
• Website, store, and app are different channels; none is automatically “higher.”
• We can count customers in each category, but we cannot calculate an average channel.

Discuss: If we label these channels 1, 2, and 3, does that make “average channel = 2” meaningful?

🪜 Ordinal data has an order

Example: customer satisfaction

Response	Order
Dissatisfied	Lower
Neutral	Middle
Satisfied	Higher

• Ordinal data has categories we can rank.
• The order is meaningful, but equal gaps between categories are not guaranteed.
• Coding responses as 1, 2, and 3 preserves their order; it does not prove that each step represents the same change in satisfaction.

Discuss: On a 1–5 satisfaction scale, does 4 mean “twice as satisfied” as 2?

🌡️ Interval data supports differences

Example: temperature

Day	Temperature (°F)
Monday	70
Tuesday	80
Wednesday	90

• Interval data has equal-sized measurement steps.
• A change from 70°F to 80°F and from 80°F to 90°F is the same 10°F increase.
• But 0°F does not mean “no temperature.” So 80°F is not twice as hot as 40°F.

Check: Which comparison works here—“10°F warmer” or “twice as hot”?

Clock time is a less straightforward example: 23:00 to 01:00 crosses midnight. A numerical rating is not automatically interval; equal gaps require an assumption about what the scale measures.

⏱️ Ratio data has a meaningful zero

Example: time spent shopping

Customer	Minutes
C01	0
C02	15
C03	30

• Ratio data supports meaningful differences and ratios.
• Zero minutes means no time spent shopping during the measured period.
• Thirty minutes is 15 minutes longer than 15 minutes, and twice as long.
• Counts, durations, and distances are common examples.

🔢 Numbers do not always measure quantities

Variable	Example	How to interpret it
Customer ID	101, 102, 103	Labels identifying people
Satisfaction score	1–5	Ordered responses; gaps may differ
Number of orders	0, 1, 2, 3	A count with a meaningful zero

• Context matters: “101” can be an ID, a count, or a measurement.
• A clock time wraps around at midnight; a rating scale may not have equal gaps.

Discuss: Why would an average customer ID be unhelpful, even if R can calculate it?

🧪 Classwork 8: Taxonomy of Data

Classwork 8 · Taxonomy of Data →

Categories and rankings Differences and ratios Interpret values in context

• Use each variable’s description, not just its appearance, to choose a measurement scale.
• Be ready to explain why your comparison makes sense.

🗄️ Databases, Tables, and Keys

🗄️ A database stores data; software manages it

• A database (DB) is an organized collection of data stored electronically.
• A database management system (DBMS) is software that manages that data.
  • It supports storage, updates, access permissions, and retrieving information.
  • PostgreSQL and MySQL are examples.
• A query is a request for data—for example, “show orders from Rochester.”

Note

SQL (Structured Query Language) is commonly used to query database tables. In this course, we use R to practice working with tables.

Reference: IBM · Relational databases

📁 A file, a database, and a data frame have different roles

Item	Role	Example in our workflow
CSV file	Saves one table as plain text	A downloaded sales export
Database	Stores and organizes data for ongoing use	A retailer’s customer and order tables
R data frame	Holds a table in an R session	Data read with read_csv()

• We can import a file or retrieve database data into R for analysis.
• Changing a data frame in R does not automatically update its original file or database.
• To keep a result after the session ends, we must save it.

📐 A schema describes a table’s structure

• A schema specifies the columns, their data types, and rules for the stored data.
• For our retailer’s orders, we could define:

Column	Meaning	Example rule
order_id	Identifies an order	Required and unique
customer_id	Identifies the customer	Uses the agreed ID format
total	Order amount in dollars	Numeric; unit documented

Discuss: A total of $4,000 is numeric. What else would you check before trusting it?

A schema helps catch structural problems. It cannot prove that a correctly formatted value matches reality. Ask about the receipt, units, and possible entry errors.

🧩 Related tables store different kinds of records

• A relational database organizes data as tables with rows and columns. Shared identifiers can connect information across tables.
• First, define what one row represents.

orders: one row per order

order_id	customer_id	total
101	C01	25
102	C02	40
103	C01	15
104	C03	30

customers: one row per customer

customer_id	city
C01	Rochester
C02	Buffalo
C04	Albany

Discuss: Which column could connect an order to its customer’s city?

All retailer data in this lecture are fictional. Notice that customers and orders have different units of observation. Keeping customer details in their own table avoids repeatedly storing the same details for every order.

🔑 A key variable connects records between tables

• A key variable is a column used to identify records or match records between tables.
• In our example, the shared key variable is customer_id.

In orders

customer_id tells us which customer placed each order.

C01 appears in two orders because the same customer ordered twice.

In customers

customer_id identifies the customer whose city is recorded.

C01 matches the customer record for Rochester.

• To connect the tables, match the same customer ID, rather than matching row positions.

Key values must refer to the same kind of record in both tables. The customer ID can repeat in the orders table because one customer can place multiple orders. These practice data deliberately include C03 without a customer match to demonstrate the left join later.

🛠️ Create the practice tables in R

library(dplyr)

orders <- data.frame(
  order_id = c(101, 102,
               103, 104),
  customer_id = c("C01", "C02",
                  "C01", "C03"),
  total = c(25, 40, 15, 30)
)

customers <- data.frame(
  customer_id = c("C01", "C02",
                  "C04"),
  city = c("Rochester",
           "Buffalo", "Albany")
)

• orders is our main table; we want to keep its orders.
• customers supplies the city to attach to each matching order.
• The tables can have different row counts and row orders. We match ID values, not row positions.

🔗 A left join attaches matching information

orders_with_city <- orders |>
  left_join(customers, by = "customer_id")

• Start with orders, the table on the left.
• by = "customer_id" tells R which column’s values to match.
• left_join() keeps every order and adds city from matching customer records.
• If there is no match, the added value is NA.

Reference: dplyr · Mutating joins

🧾 Read the join result one order at a time

order_id	customer_id	total	city
101	C01	25	Rochester
102	C02	40	Buffalo
103	C01	15	Rochester
104	C03	30	NA

• Orders 101 and 103 both match C01 and receive Rochester.
• Order 104 remains, but its city is NA because C03 has no customer match.

Check: Why does C04’s city, Albany, not appear in this result?

C04 has no order in the main table. A left join adds matches to the left table; it does not add a new row just because the reference table has an unused key. NA is missing information, not zero and not proof that the customer has no city.

⚠️ Repeated reference keys can multiply rows

• Suppose customers accidentally lists two cities for C01.

Two matching reference rows

customer_id	city
C01	Rochester
C01	Syracuse

One order now has two matches

order_id	customer_id	city
101	C01	Rochester
101	C01	Syracuse

• The join returns both matches. Counting these rows as separate orders would overcount.
• For this lookup, confirm one customer record per customer ID before joining.

Discuss: Which city belongs to C01? What evidence would you check?

Repeated customer IDs in orders are expected because a customer can place several orders. Repeated customer IDs in customers are a problem for this intended one-customer-record lookup. Do not delete records arbitrarily; identify whether the records are errors, history, or a different unit of observation.

✅ Check a join before using its result

1. Meaning: Does the matching key identify the same thing in both tables?
2. Matches: Are IDs spelled and stored consistently? Which orders have no match?
3. Row count: Does each order match at most one customer record? Did the join add unexpected rows?

Predict: If we start with customers instead, which records would we be trying to keep?

The answer is customers, including C04. Reversing the tables changes the question and may produce a different result. Keep the main table aligned with the intended unit of analysis.

📦 Preparing Data with ETL

🔄 ETL moves and prepares data for use

• ETL stands for Extract, Transform, Load—one common data preparation workflow.

📥 Extract

Copy data from its sources.

Example: collect order exports and customer records.

→

🔧 Transform

Clean and connect the data.

Example: standardize IDs and attach customer cities.

→

💾 Load

Save prepared data to a destination.

Example: store a reporting table in a warehouse.

• Each step should preserve enough information to check where the result came from.

Reference: IBM · ETL

📥 Extract: collect data and inspect what arrived

• Data may come from a CSV, spreadsheet, database, or app.
• In R, read_csv() is one way to import a source table.
• Before changing the data, inspect:
  • Coverage: Which dates, stores, or customers are included?
  • Structure: Are the expected rows and columns present?
  • Meaning: What do the units and missing values represent?

Discuss: Your file has no orders for Sunday. How could you tell whether the store was closed or the export is incomplete?

🔧 Transform: make values consistent and connect tables

Issue in the source	What to check or change
" C01 " versus "C01"	Remove unintended spaces in IDs
10/07/2026	Confirm the date convention before converting it
Amounts from different countries	Check currencies before combining totals
Repeated or missing customer IDs	Investigate before joining

• In R, filter() keeps chosen rows, select() selects columns, and left_join() connects tables.
• Document the reason for a change. A missing value does not automatically mean zero, and an unusual value does not automatically mean an error.

💾 Load: save a result that can be used again

In an organization

Prepared data is saved to a destination, such as a database or data warehouse.

Other people can retrieve it for reporting and analysis.

In our classroom workflow

We hold a prepared table in an R data frame.

Saving it as a CSV is a simple way to retain a result outside the R session.

readr::write_csv(orders_with_city, "orders_with_city.csv")

• This relative pathname saves the file in R’s working directory.

The code is displayed, not executed while rendering this deck. A classroom CSV workflow illustrates the preparation steps; it is not a full database system. Some workflows use ELT: extract, load, then transform in the destination.

🌐 Big Data: More Than a Large File

🧠 Big data creates challenges for the available tools

• Big data refers to data whose size, speed, or complexity makes it difficult to manage with the tools and resources available.
• The five V’s help describe the challenge:

V	Question to ask
Volume	How much data must we store or process?
Velocity	How quickly does new data arrive or need to be used?
Variety	What different formats and sources must we combine?
Veracity	How reliable and complete is the data?
Value	What useful decision can this data support?

Reference: IBM · Big data

📚 Volume: how much data are we handling?

Common storage units

Unit	Decimal size in bytes
Kilobyte (kB)	1,000
Megabyte (MB)	1,000,000
Gigabyte (GB)	1,000,000,000
Terabyte (TB)	1,000,000,000,000
Petabyte (PB)	1,000,000,000,000,000

• In these decimal units, each step is 1,000 times the previous one.
• Illustration: one million files of 1 MB each require about 1 TB, before extra storage overhead.
• Size affects whether data fit in a laptop’s memory and how long copying or processing takes.

These are decimal units, not binary KiB, MiB, or GiB. File size and memory use need not be identical: R objects can use more memory than the source CSV. No global data-growth forecast is required to explain volume.

⚡ Velocity: how soon do we need the data?

Delivery dispatch

New orders and driver locations arrive continuously.

A dispatcher needs recent information to assign a nearby driver.

Monthly planning

Managers review sales across the completed month.

They need consistent totals, but not necessarily an update every second.

• Velocity concerns how fast data arrive and how quickly they must be processed.
• The needed update frequency depends on the decision.

Discuss: Could yesterday’s accurate driver locations help dispatch an order right now?

🧩 Variety: useful evidence comes in different formats

Sales table

Order IDs, dates, quantities, and prices.

+

Customer review

Written comments explaining a shopping experience.

+

Product photo

An image showing a product or possible defect.

• Combining sources requires agreement about IDs, dates, units, and definitions.
• A table can store an image’s filename or link, while the image itself contains different information.

🔎 Veracity: can we trust what the data represent?

• Veracity concerns the reliability of data.
• For an order report, check:
  • Accuracy: Does the amount match the receipt?
  • Completeness: Are any stores or days missing?
  • Consistency: Is “sales” defined the same way across stores?
  • Duplicates: Was the same order imported twice?

Discuss: If every value fits the expected data type, which of these problems could still remain?

🎯 Value: connect data to a useful decision

• Value comes from using data to answer a relevant question.
• For next month’s inventory, a retailer could compare:
  • Products customers viewed.
  • Products customers bought.
  • Products that sold out.
• These measure different things. Low sales may reflect low demand—or unavailable stock.

Discuss: What could you miss if you chose next month’s stock using sales totals alone?

🏗️ From Source Systems to Useful Analysis

💻 Infrastructure helps when one computer is not enough

• Data infrastructure is the storage, software, and connections that move and manage data.
• Different needs call for different arrangements:

Need	Possible response
More data than one computer can hold	Use larger storage or spread it across computers
A task takes too long	Divide suitable work across several computers
Many people need reliable access	Use a shared database with access rules

• Cloud services provide computing or storage through a network.
• More computing capacity still requires clear definitions and data-quality checks.

🏛️ A data warehouse brings sources together for analysis

• A data warehouse brings data from multiple sources into a shared store designed for querying, reporting, and analysis.

Store sales

Online orders

Inventory records

→

Data warehouse

Prepared, connected data with shared definitions.

→

Reports

Dashboards

Analysis in R

• ETL is one way to prepare the data flowing into it.

Reference: IBM · Data warehouses

⚖️ Running a business and analyzing it use data differently

	Operational database	Data warehouse
Main purpose	Support daily operations	Support analysis and reporting
Typical task	Record an order or update stock	Compare sales across stores and months
Data focus	Records needed by the application	Prepared data combined across sources

• A warehouse can retain history to support comparisons over time.
• Its usefulness depends on consistent definitions, quality checks, and appropriate access.
• A warehouse does not need a particular number of rows or years of history to count as a warehouse.

🏬 Walmart: A Pioneer in Data Warehousing

Illustrative image

1992 milestone: Walmart’s Teradata data warehouse became the first to reach 1 TB.

• 2025 scale: Walmart reported about 10 petabytes generated per day from its stores, website, and app.
• Checkout sales and Wally: Walmart’s internal analytics tool helps merchants compare product sales across stores and investigate poor performance.
• Inventory and deliveries: Compare sales with stock levels to identify shortages and redirect products, reducing missed sales and excess stock.
• In-store cameras: Walmart says cameras may capture checkout images for security and operations, including store design.

Possible analytics use: Combine entry/exit counts with checkout activity by store and hour to identify busy periods and plan checkout staffing.

Sources: Teradata · 1992 milestone, Walmart · 2025 data scale, Walmart · Wally, Walmart · Supply chain, Walmart · Privacy notice

The warehouse’s value comes from connecting records that would otherwise sit in separate systems. Checkout transactions describe purchases. Inventory records help explain whether low sales reflect weak demand or empty shelves. Comparing these records by product, store, and date can guide replenishment and reduce unnecessary stock.

Walmart Global Tech’s September 15, 2025 article, “Inside the network powering Walmart,” reports approximately 10 petabytes of data generated per day from visits, purchases, and interactions across its stores, website, and app. This is a data-generation figure across the business, not a measured throughput for one data warehouse. The article also says its networks process and manage those data in real time.

Walmart’s March 18, 2025 Wally announcement describes comparisons across stores, regions, sales channels, products, and brands, along with investigation of product underperformance. Its July 17, 2025 supply-chain article describes inventory systems that detect stock imbalances and redirect products where needed. These sources document current business uses without assuming that every source feeds one particular warehouse in real time.

The privacy notice, updated August 20, 2026, describes purchase histories, cameras inside and outside stores, and images that may be captured at checkout. It lists security and operational purposes, including store design. It does not establish that Walmart identifies every shopper or routinely links each person’s entrance, exit, and purchases at every store.

The entry/exit example is a possible analytics use, not a verified company-wide practice. A retailer could turn camera or sensor observations into aggregate visitor counts, then connect those summaries with checkout activity using store, date, and hour as key variables. More visitors and growing checkout queues could motivate additional staffing. Visitor counts alone cannot establish why someone leaves without buying. Summaries can support this comparison without matching individual shoppers, and collection and access still require privacy safeguards.

Teradata reports the 1992 one-terabyte milestone. The supplied data-center image is illustrative, not evidence of Walmart’s physical infrastructure.

🛒 Retail data supports stocking and marketing

Business decisions

• Stocking: Connect purchases with stock availability to choose what to reorder.
• Marketing: Use purchase history to estimate what a customer might buy and select relevant recommendations or offers.
• Illustration: Frequent pasta purchases could prompt a coupon for pasta sauce.

Wegmans: marketing vs. security

• Purchase history: Wegmans customizes digital coupons using customers’ previous purchases.
• Biometric data: Measurements of physical features, such as a face, that can help identify a person.
• Security news: Wegmans already uses facial recognition in a small fraction of stores. It says these data are used only for security.

Discuss: Which data would help suggest a product? Which collection raises greater privacy concerns?

Sources: Wegmans · Personalized coupons, Wegmans · Facial-recognition statement

Stocking decisions concern how much of a product to have available. Marketing decisions concern which products or offers may interest a customer. Prepared purchase records can support both, but each analysis needs the right unit of observation. To personalize an offer, connect previous purchases to the appropriate customer account and product records, check dates and returns, and respect the permitted use of the data.

The pasta-sauce coupon is an illustrative prediction example, not a claim about a particular Wegmans campaign or model. A purchase history suggests possible interests but cannot establish what a customer will buy next. Purchases may be gifts, preferences may change, and a discount does not guarantee a sale. Check whether recommendations are useful and whether an offer changes purchasing behavior rather than simply rewarding a purchase that would have happened anyway.

Wegmans’ official app FAQ describes digital coupons customized to purchase history. Its separate official facial-recognition statement says cameras with this technology have already been deployed in a small fraction of stores. The company describes their purpose as identifying people previously flagged for misconduct and keeping stores safe. It says facial-recognition data are not used for other purposes. Do not infer that these security data feed marketing recommendations or that every Wegmans store has the system.

Wegmans also says it does not collect retinal scans or voiceprints, uses facial recognition as one investigative lead rather than the sole basis for a decision, and disposes of the data after the security need ends. Ordinary video, aggregate visitor counts, and biometric identification are different types of data collection. Facial recognition involves identifying people from facial features and raises concerns about privacy and mistaken matches. Keep these concerns distinct from the documented purchase-history coupon example.