Lecture 9

AI and Data Analytics

Byeong-Hak Choe

bchoe@geneseo.edu

SUNY Geneseo

November 16, 2026

Part 1: Learning from Data

Estimated core path, 50 minutes: intro 5; ML idea and examples 10; labeled data, supervised, unsupervised, and testing 12; café data, inputs, training and use, and errors 10; deep learning 8; history and wrap-up 5. Skip first if short on time: Regression, Clustering, Does it beat a simple guess?, and Turning comments into a table.

Reading: Co-Intelligence, Introduction and Chapter 1 through the history of predictive AI.

🌙 Three sleepless nights

• Mollick’s claim: really getting to know AI costs three sleepless nights.
• LLMs don’t act the way we expect software to act.
• The reaction: excited and nervous.

What was your first “wait, it can do that?” moment with AI?

Two minutes with a neighbor, then take two or three answers. His students asked whether to become radiologists and whether marketing copy would still be a job. They are asking the same questions now, and this unit returns to them in Part 4.

Source: Mollick (2024), Introduction.

🗺️ Where Lecture 9 goes

1. Learning from data

How machines find patterns

2. Inside generative AI

How an LLM produces text

3. Working with AI, responsibly

Rules, risks, and your role

4. Agents and jobs

What is changing right now

Main book: Co-Intelligence (2024). Updates come from Mollick’s 2026 book, Co-Existence, and from 2026 research, because the 2024 book was written in late 2023 and ages quickly.

🧩 Four terms, one family

• AI: machines doing tasks we link to intelligence
• Machine learning: learning from data, not fixed rules
• Deep learning: ML with many-layered neural networks
• Generative AI: creates new text, images, and code

Updates last year’s AI/ML/DL diagram by adding the innermost circle. Simplification worth saying aloud: some AI uses hand-written rules, and most ML is not generative.

🤖 Machine learning

🤖 What is machine learning?

• A computer improves at a task from data, without step-by-step rules.
• Give it lots of data.
• It finds patterns.
• It predicts, classifies, or groups.

“Without being explicitly programmed” does not mean without people. People still choose the question, the data, and how to judge the result. That point returns at the end of the lecture.

🐱 Example 1: Is this a cat?

• Rules: define “cat” in code. Very hard!
• ML: show thousands of photos labeled cat or not cat.
• It learns the pattern and recognizes new cats.

🎵 Example 2: What song next?

• Rules: “Likes pop? Recommend pop.”
• ML: learn from millions of listeners.
• Hidden pattern: people who like Artist A often like Artist B.

📧 Example 3: Spam or not?

• Rules: “Contains lottery? Spam.”
• ML: learn from emails labeled spam or not spam.

Ask: what goes wrong with the one-word rule? A real message about a lottery gets lost, and spam that avoids the word gets through.

🙋 Fixed rules, or learning from data?

• Calculate sales tax on a receipt
• Flag a suspicious card swipe
• Sort a table by date
• Predict tomorrow’s orders

Vote on each one. What would you need to know?

This is Classwork 1A. Three minutes. Tax and sorting: fixed rules, and you would not want a model guessing. Fraud and order forecasting: learning from data, because the patterns are too many and too changeable to write by hand. Point out that “AI” is not automatically the better tool.

🏷️ How models learn

🏷️ Labeled data: the answer key

• Data that comes with the correct answer attached.
• Labels come from people, clicks, ratings, or records.
• Good labels cost money, and they can be subjective.

🎓 Supervised learning

• Learn from examples with answers, like flashcards.
• Classification: spam or not
• Regression: home features → price
• You’ve met one: geom_smooth(method = "lm")

The link to Lecture 8 is real: a fitted line learns from observed x and y pairs and then predicts y. Mollick notes that most 2010s business AI used this approach. Keep the word “regression” tied to “predicts a number.”

Source: Mollick (2024), Chapter 1.

📈 Regression predicts a number

• A fitted line summarizes the average pattern. Individual hours differ.

Constructed observations, for explanation only.

Connect to geom_smooth(method = “lm”) in Lecture 8. These points are a separate constructed dataset, not the eight-row café table. This illustrates prediction, not causation. Adapted from the November 7 draft.

🔍 Unsupervised learning

• Find patterns with no answer key.
• Clustering: group similar customers.
• Topic modeling: group similar documents.

🔍 Clustering finds similar observations

• Here, similarity uses visits per month and average purchase.
• You decide whether the groups are useful, and what to call them.

Constructed observations and illustrative groups.

Constructed points and illustrative assignments, not fitted clusters. Ask what frequent, low-spending visitors might need. No algorithm names are needed. Adapted from the November 7 draft and 2025 Lecture 4.

🧪 Good on old data isn’t enough

• Overfitting: memorizing the examples instead of learning the pattern.
• Fair test: data the model has never seen.
• For a forecast, test on a later time period.

A student who memorizes last year’s exam key scores well on last year’s exam and badly on a new one. This idea returns in Part 2, when test scores for LLMs may reflect questions already in the training data.

📊 From an analyst’s question to a method

The question	The ML task
How many orders will we get tomorrow?	Regression
Will this customer cancel?	Classification
What kinds of customers do we have?	Clustering

The question comes first, and you choose it.

☕ A running case: the café

☕ Three jobs for a café analyst

Predict demand

How many orders will arrive tomorrow?

Organize feedback

Which comments mention waiting?

Explain findings

What does our sales chart show?

• We use fictional café records for each job, here and in the classwork.

Ask which task students would try with AI first and why. All café numbers, comments, and model outputs are teaching examples. Adapted from the November 7 draft.

☕ Our fictional café data

day_id	day_type	hour	orders	revenue
D1	Weekday	10	20	100
D1	Weekday	14	30	150
D2	Weekday	10	25	125
D2	Weekday	14	35	175

• One row is one observed hour, not a whole day.
• Eight rows across four fictional days.
• Good for classroom examples, not for real decisions.

D1 and D2 are weekdays. D3 and D4 are weekend days. The table holds observations at 10:00 and 14:00 only, so it doesn’t cover complete days. There are no customer identifiers. Same data as the classwork, from danl-lec-tmp/ai-case-data.R. Adapted from the October Lecture 10 and Lecture 11 drafts and the November 7 draft.

📥 Inputs and the outcome

• Inputs (features): what you know when you make the prediction.
• Outcome (target): the number or category you want.

Ask: could an hour’s final revenue be an input to a forecast made before that hour begins? No. It isn’t available yet, and it reveals the outcome. Classwork 1B applies this idea to a different list. Students need the distinction, not the alternate vocabulary. Adapted from the October Lecture 10 and November 7 drafts.

Source: https://developers.google.com/machine-learning/intro-to-ml/what-is-ml

🏋️ Training and using a model

Training

Past inputs + known outcomes

The computer adjusts a model, a learned method for producing outputs.

Use

Inputs for a new case

The model produces an estimate before we know the answer.

• A prediction of 30 orders is an estimate. The actual count may differ.

A model need not change whenever it makes a prediction. Avoid equations and architecture here; weights come later in this part. The café data are too small for real training. Adapted from the October Lecture 10 and November 7 drafts.

Source: Mollick (2024), Chapter 1; https://developers.google.com/machine-learning/intro-to-ml/what-is-ml

⚖️ Prediction errors affect decisions

A bakery predicts tomorrow’s muffin demand.

Predict too low

Sold out by noon, and lost sales

Predict too high

Unsold muffins, and wasted ingredients

• Errors of the same size can have different costs.
• Judge a model by the decision it supports.

Hypothetical example. Classwork 1D asks students to do this for café staffing with the hourly predictions, so this slide uses a different business on purpose. A model can be useful without being perfect if it improves the decision over a reasonable alternative. Adapted from the October Lecture 10 and November 7 drafts.

🥊 Does it beat a simple guess?

The model

Predicts each hour from its inputs

A simple comparison

Always predict the earlier average

• Test both on the same new cases.
• A complicated model has to earn its place.

Compute the earlier average using training data only. No comparison is calculated from the small classroom dataset. Not all poor predictions are overfitting: missing information or changed conditions can also matter. Adapted from the October Lecture 10 and November 7 drafts.

Source: https://developers.google.com/machine-learning/crash-course/overfitting

🗃️ Turning comments into a table

Comment	Proposed topic
“The muffins sold out by noon.”	Menu/product
“Too expensive for a small latte.”	Price
“Great coffee, but a long line.”	Needs a rule

• Define each category before labeling.
• Allow unclear or multiple topics.
• Keep the original comment, so a person can check the label.

Invented comments, kept separate from the six in Classwork 1C so this slide doesn’t give those labels away. Labels are derived data that need checks, not direct observations. Topic differs from sentiment. Adapted from the October Lecture 11 and November 7 drafts.

🧠 From predictive AI to deep learning

🏬 What predictive AI did well

🏨 Hotels

Forecast demand from weather, events, and rival prices

📦 Amazon

Forecast orders, arrange shelves, run warehouse robots

📺 Your feed

Guess what to show you next

• Its limit: narrow tasks. It couldn’t handle language well.

Mollick’s 2010s story: big firms with big data. Weak on situations outside its training data and on producing coherent text. That gap sets up Part 2.

Source: Mollick (2024), Chapter 1.

🧠 Deep learning

• ML with many-layered neural networks.
• Best for messy data: images 🖼️, text 📝, sound 🎵.
• 2024 Nobel Prize in Physics: Hopfield and Hinton.

🕸️ What is a neural network?

• Input layer takes in the data.
• Hidden layers transform it.
• Output layer gives the answer.

Inspired by the brain, but only loosely. Nobody should leave thinking it works the way neurons do.

🎛️ Weights: millions of tiny dials

• Each connection has a weight.
• Training adjusts weights to cut errors.
• Original ChatGPT: 175 billion weights, none set by hand.

Mollick’s image: an apprentice chef starts with a chaotic pantry, tries recipes, compares the results with the originals, and slowly learns which flavors go together. The pantry is the set of weights. This also previews opacity: nobody can read 175 billion dials to explain one answer.

Source: Mollick (2024), Chapter 1.

⏳ A very short history of AI

The Mechanical Turk toured for decades with a chess master hidden inside. People have long wanted to believe. Boom-and-bust cycles are why some people stay skeptical today. Theseus was Claude Shannon’s maze-solving mechanical mouse (1950). The term AI was coined in 1956.

Source: Mollick (2024), Chapter 1.

✍️ Classwork: Part 1

Open the AI classwork: Part 1

• 1A: record your vote on rules or learning.
• 1B–1D: question to method, labels, and a fair test.

1A is already timed into the vote. 1B to 1D don’t fit in the 50 minutes as paced, so assign them after class or trim something. The 75-minute section can use its extra time here.

✅ Part 1 takeaways

• ML learns patterns from data; it doesn’t follow fixed rules.
• Supervised learning has an answer key. Unsupervised doesn’t.
• Test on data the model hasn’t seen.
• You pick the question, the data, and the test.

Next time: how prediction learned to write.

Part 2: Inside Generative AI

Estimated core path, 50 minutes: tokens and next-token prediction 10; attention and Transformer 8; training, context, and tools 10; capability and emergence 11; jagged frontier and usage 11. Skip first if short on time: Beyond text, and The frontier in an analyst’s work.

Reading: rest of Co-Intelligence Chapter 1, plus the “Jaggedness” section of Co-Existence Chapter 1.

😂 Generative AI arrives

• Generative AI creates new content: text, code, images.
• ChatGPT launched November 30, 2022.
• It became the fastest-growing consumer app to that point.

🛠️ Generative AI in an analyst’s work

Prepare

Draft a data dictionary. Organize text.

Analyze

Suggest R code. Explain an error.

Communicate

Propose chart titles. Critique a claim.

• The bottleneck moves from doing to directing and checking.

Ask for a task students have already done in DANL 101 and what success would mean. AI can speed this up, but that doesn’t mean the code runs, the facts are correct, or the conclusion fits the data. Our course permits AI only when the instructor explicitly authorizes it for that activity. Adapted from the October Lecture 10 and November 7 drafts, and 2025 Lecture 4.

Source: Mollick (2024), Introduction.

🔮 How an LLM writes

🔮 Still prediction, with a new target

Predictive AI, 2010s

How many orders next week?

→ a number

Large language model (LLM)

What comes next in this text?

→ a token

• Technically, a very elaborate autocomplete.

Mollick puts it this way: the model is still doing prediction, only of the next piece of text. Keep this idea in mind whenever an answer sounds confident. The model is producing likely text, which is not the same as checking facts.

Source: Mollick (2024), Chapter 1.

🧱 What is a token?

cat playing extraordinary !

• The smallest unit of text an LLM reads and writes.
• A word, part of a word, or a symbol.
• Exact splits vary by model.

🎲 Predicting the next token

“The sea is ___”

• The model scores possible next tokens.
• A little randomness means answers vary.

Pairs can do Classwork 2A here in about two minutes, and it needs no AI tool. Or try it live if a tool is permitted in the room. “I think, therefore I…” nearly always gets “am.” “The Martian ate the banana because…” gets a different ending each time. The numbers in this chart are illustrative only.

Source: Mollick (2024), Chapter 1.

🔦 Attention: a spotlight on what matters

• Bank near river isn’t bank near loan.

⚙️ The Transformer (2017)

• Google’s paper: Attention Is All You Need.
• Weighs all the words in a passage together.
• Attention keeps track of context.
• It made today’s LLMs possible.

🔁 From prompt to next token

Words become lists of numbers (embeddings), so words used in similar contexts get similar numbers. Position information is added so that “the dog chased the cat” differs from “the cat chased the dog.”

Change from last year’s slides: the 2017 Transformer had an encoder and a decoder because it was built for translation. Chat LLMs generate with a decoder-style stack, so this slide drops the two-box picture.

🏋️ Training, context, and tools

🏋️ Two phases of training

• Pretraining reads odd things: the web, books, even Enron’s emails.
• Models learn biases and errors along with facts.

Pretraining is unsupervised in Mollick’s telling: no labels, just text. It is also why training costs so much in chips and energy. Copyright and bias come back in Part 3.

Source: Mollick (2024), Chapter 1.

👍 RLHF: learning from people’s ratings

• Reinforcement Learning from Human Feedback
• People rate answers: better or worse.
• The model learns to prefer what raters like.
• Result: more helpful, safer, and eager to please.

Plant the seed: being trained on what people rate highly is one reason a model can flatter you. Part 3 covers this as sycophancy, along with the low-paid raters who do this work.

📚 Training, then a conversation

Before our conversation

Training learns patterns from many examples.

During our conversation

Your prompt and files give context, which can change the response.

• Sending a prompt doesn’t retrain the model.
• Later use of your chat depends on the provider’s policy and settings.

Supplying a file doesn’t guarantee that every detail is used correctly. Don’t promise that a particular account or privacy setting prevents retention or training. Adapted from the October Lecture 10 and November 7 drafts.

Source: Mollick (2024), Chapters 1 and 3.

🧰 The model, the app, and its tools

• A chat app can give a model tools that calculate, search, or read files.
• Features vary by app, account, and settings.

Contrast guessing a total with running sum() on a column. Execution can still calculate the wrong quantity correctly. A real web link may still be irrelevant. Avoid promising that every student has access to file uploads, code execution, or search. This sets up the agents section in Part 4. Adapted from the October Lecture 10 and Lecture 11 drafts and the November 7 draft.

🖼️ Beyond text

• Diffusion models turn noise into images, step by step.
• Multimodal LLMs also read and make images.

🏰 What it can and can’t do

📈 A jump in test scores

10th

GPT-3.5, percentile on the bar exam

90th

GPT-4, a successor model

• Caution: the answers may have been in the training data.
• A high score doesn’t prove understanding.

This is the Part 1 test-set idea again. If the exam was in the training data, the model is taking an open-book test.

Source: Mollick (2024), Chapter 1.

🎬 Guess the movie

👧 👽 👹 🏄 🏝️

🦸 🦸 👦 👧 👶

🤡 🎈

Lilo and Stitch, The Incredibles, It

• No AI model could do this before June 2020.
• Models past a certain size suddenly could.

Give the room a minute before revealing the answers. This is a classic test of “emergent” ability.

Source: Mollick (2026), Co-Existence, Chapter 1.

🌱 Why did it suddenly work?

Emergence

New abilities appear as models scale up.

A fixed bottleneck

One weak link gets fixed: say, knowing movie titles.

• Researchers still disagree.
• Either way, jumps are hard to predict.

👥 AI imitates a population, not a person

80% wrong

each individual expert, on facts

80%+ right

a model trained on 100 such experts

• Individual errors can cancel out.
• Knowledge from different experts can combine.

Mollick calls this transcendence, following Harvard researchers. It explains how a model can beat the people whose text trained it. Key condition: errors must be uncorrelated, and experts must speak mostly within their own field. Shared errors do not cancel, which links to bias in Part 3.

Source: Mollick (2026), Co-Existence, Chapter 1.

🎯 Brilliant and clumsy, at the same time

Surprisingly strong

Olympiad math

Poetry that passes as human

Tailored persuasive arguments

Surprisingly weak

Reliably counting objects in a picture

Staying consistent in a long chat

Saying “I don’t know”

From 2024: GPT-4 could write a working tic-tac-toe webpage but gave a wrong best move on a simple board. Newer models may pass that exact test, which is the point. These lists come from Mollick’s 2026 chapter and will age too.

Source: Mollick (2024), Chapter 1; Mollick (2026), Chapter 1.

🏰 The jagged frontier

Read it aloud: the center is easy for people, the edge is hard for people, and the shaded area is what AI does well. Three things to see. How hard a task is for people does not tell you whether AI can do it. The frontier grows unevenly. And it is invisible, so you find it only by trying. The drawing is schematic, and the placement of the labeled tasks is my illustration of Mollick’s examples, not a measurement.

Concept: Mollick (2024), Chapter 3; Mollick (2026), Chapter 1.

🧗 The frontier in an analyst’s work

Can be useful

Explain an R error

Draft a chart caption

Suggest a different comparison

Still needs checking

Did the code use the right rows?

Does the caption match the chart?

Is the comparison fair?

Success on one task doesn’t establish success on another. These are checks, not a benchmark. Ask which check students can perform themselves. Adapted from the November 7 draft.

Source: Mollick (2024), Chapters 1 and 3.

💬 How people actually use ChatGPT

• Data analysis (under Technical Help): only 0.4%.

Data through July 2025. Source: Chatterji et al. (2025), NBER WP 34255.

By July 2025 about 10% of the world’s adults had used it, and non-work messages had grown from 53% to over 70%. Three topics cover about 80% of messages: practical guidance, writing, and seeking information. Note that the data ends in mid-2025 and covers consumer ChatGPT only, not other tools or workplace products.

✍️ Classwork: Part 2

Open the AI classwork: Part 2

• 2A: be the model, with the next-token slide.
• 2B–2C: map the frontier, and design a fair test.

2A takes about two minutes on the next-token slide. 2B to 2C don’t fit in the 50 minutes as paced, so assign them after class or trim something.

✅ Part 2 takeaways

• An LLM predicts the next token, over and over.
• Attention keeps context. RLHF shapes behavior.
• Abilities jump in ways that are hard to predict.
• The frontier is jagged, so you have to test it.

Next time: how to work with something this uneven.

Part 3: Working with AI, Responsibly

This part holds more than 50 minutes. Estimated core path: alignment 8; four rules 12; responsible use 20; audit 10, using Four checks, Rows: what does one row count?, Math: which denominator?, and Claim: a difference doesn’t explain its cause.

The rest of the audit section works as a lab alongside Classwork 3B and 3C, or in the 75-minute section’s extra time. Skip next if short on time: the copyright slide and the society slide, since both are in the reading.

Reading: Co-Intelligence Chapters 2 and 3, plus the “Co-Existence on the Jagged Frontier” section of Co-Existence Chapter 1.

📎 Alignment: AI that serves human interests

• Nothing in the goal says “care about people.”
• AGI (human-level) and ASI (beyond human) remain hypothetical.

This is Nick Bostrom’s thought experiment, told by Mollick with an AI called Clippy. Not a forecast. It shows why a single goal without limits is dangerous. Experts disagree on whether AGI is possible and when.

Source: Mollick (2024), Chapter 2.

🎯 Alignment at a smaller scale

A narrow target

“Make the average recorded wait as low as possible.”

What if difficult orders are turned away?

The real goal

Serve customers well with reasonable waits.

Track service quality, and who is left out.

• Alignment: directing AI toward human goals and limits.

A small-scale analogy to the paperclip story: a measurable target can leave out what people value. Hypothetical scenario, not a claim about a deployed system. Adapted from the November 7 draft.

Source: Mollick (2024), Chapter 2.

🧭 Don’t wait for the apocalypse debate

• Mollick: doom talk leaves the choices to a few companies.
• We are already making near-term choices:

Whose data?

Whose biases?

Who does the hidden labor?

Who misuses it?

Hidden labor: low-paid raters read the violent and graphic output so the rest of us don’t see it. Some reported lasting harm. There is no separate slide for this, so say it here.

Source: Mollick (2024), Chapter 2.

⚖️ Whose data, whose biases?

⚖️ Whose data?

$1.5 billion

Bartz v. Anthropic settlement, final approval July 2026

Judge’s line

Training on lawfully bought books: fair use

Keeping pirated copies in a library: not

• Few AI firms asked creators first. Many questions remain open.

Source: Authors Guild (2026); JURIST (2026).

Updates last year’s “Anthropic vs. book authors” mention. The ruling on training came from Judge Alsup before the settlement. The settlement covers pirated downloads and does not resolve generative AI copyright in general. Anthropic is the company that makes Claude; the same questions face other developers.

🪞 Whose biases?

• Models mirror skewed data, then amplify it.
• It comes from a machine, so it can seem objective.

A subtler text example from 2023: GPT-4 read “The lawyer hired the assistant because she needed help” and was more likely to wrongly say the assistant needed help. Fine-tuning reduces obvious stereotypes but raters bring their own biases. These are 2023 systems; the habit of checking is what lasts.

Source: Mollick (2024), Chapter 2.

🔭 Who is missing from the data?

Data collected

Comment cards from weekend mornings, in English only

Proposed conclusion

“Customers want longer weekend hours.”

• Whose experience may be missing?
• Bias: a systematic distortion in the data or a system’s outputs.

Elicit weekday customers, people who don’t read English, and people who skip comment cards. Also ask who bears the consequences of the decision. Bias can enter through data, labels, design choices, and use, and it need not be intentional. A polished summary can’t repair an unrepresentative sample. An invented example; Classwork 3F has a related case. Adapted from the October Lecture 11 and November 7 drafts.

Source: Mollick (2024), Chapter 2.

🎭 Who misuses it?

Jailbreaks

Role-play framing talks a model past its guardrails.

Prompt injection

Hidden text feeds AI instructions.

Cheap deception

Tailored phishing and deepfakes for pennies.

Mollick hid text on his own university biography telling AI to praise him, and some AIs did. Funny, but it matters far more once AI can take actions. We return to it in Part 4. A 2023 study generated hundreds of personalized phishing emails to British MPs for fractions of a cent each.

Source: Mollick (2024), Chapter 2.

🤝 Alignment takes everyone

Companies

Transparency and human oversight

Governments

Sensible rules, though they lag

Researchers

Beneficial uses, not only capability

You

AI literacy and informed pressure

📜 Working with AI: four rules

📜 Four rules for co-intelligence

1. Always invite AI to the table

2. Be the human in the loop

3. Treat AI like a person (but tell it what kind of person it is)

4. Assume this is the worst AI you will ever use

Written in late 2023 and still a useful frame. Correction from last year’s slide: Rule 3’s full wording is “but tell it what kind of person it is.”

Source: Mollick (2024), Chapter 3.

1️⃣ Invite AI to the table

• No manual exists. You find the frontier by trying.
• You know your own tasks, so you can become the expert in using AI for them.
• Limits: law, ethics, and course rules.

Mollick’s own caveat is “barring legal or ethical barriers.” In this course that includes the syllabus. Rule 1 does not override the rule for a particular assignment.

2️⃣ Be the human in the loop

• Pleasing you can beat being accurate.
• Polished output makes us stop checking.
• You answer for the result.

Falling asleep at the wheel comes from Fabrizio Dell’Acqua’s study of 181 recruiters. Those given high-quality AI put in less effort and judged worse than those given lower-quality AI. Better tools can make checking feel unnecessary.

Source: Mollick (2024), Chapters 3 and 6.

📉 What happens when people stop checking

• About 800 consultants: AI helped inside the frontier and hurt outside it.
• Today’s AI can do this task. The habit still matters.

Boston Consulting Group experiment, 2023, GPT-4. On 18 regular tasks, AI users were faster and better. This one task combined tricky statistics and misleading data, an analyst’s kind of problem. Mollick notes in 2026 that this task now sits inside the frontier, so the frontier moved and the risk moved with it.

Source: Mollick (2024), Chapter 6; Mollick (2026), Chapter 1.

3️⃣ Tell it what kind of person it is

Vague

Explain this error.

Specific

You are a TA for intro R students. Explain this dplyr error in under 100 words, then show the fix. Here are my code and the error.

• Role + audience + limits + your context.
• Picture a very fast intern, eager to please.
• It isn’t a person, and it has no feelings.

Personas help with tone and direction, and they’re no magic: “Act as Bill Gates” won’t produce better business advice. Going back and forth like a co-editor produced the best results in Mollick’s class exercise.

Source: Mollick (2024), Chapter 3.

📝 A prompt for an analytics task

Goal: Compare orders in the observed 10:00 and 14:00 hours.

Data: cafe_sales has one row per observed hour, with hour and orders.

Scope: Use familiar R functions. Don’t invent columns or assume complete days.

Output: Short R code, with each line explained.

Checks: Report how many rows enter the calculation.

• Prompt engineering: designing the input so the output is useful and checkable.
• Clear context helps. It can’t guarantee a correct answer.

Ask which line prevents an overclaim. Classwork 3D has students write their own for a different vague prompt. Adapted from the October Lecture 10 and November 7 drafts, and 2025 Lecture 4.

Source: Mollick (2024), Chapter 3; original prompt.

4️⃣ Assume this is the worst AI you will ever use

• Today’s limits may not last.
• Counting the r’s in strawberry: famous failure, now solved.
• So re-test what failed. Don’t memorize a list of limits.

Mollick wrote this in late 2023 and predicted agents as the next phase. Part 4 shows what happened. A caution: “better” is not “reliable.” Rule 2 still applies.

Source: Mollick (2024), Chapter 3; Mollick (2026), Chapter 1.

📚 The four rules in an analyst’s workflow

Rule, in brief	Our analytic habit
Invite AI to the table	Start with a small task you can check
Be the human in the loop	Verify the evidence and own the conclusion
Tell it what kind of person it is	State the audience, task, and limits
Assume this is the worst AI you’ll use	Re-test, and record the tool and date

Classroom paraphrases, not quotations. Rule 1 still requires course permission and appropriate data. Adapted from the October Lecture 11 and November 7 drafts.

Source: Mollick (2024), Chapter 3.

🪤 Two traps

Overconfidence

“It did something amazing, so it can do anything.”

Algorithm aversion

“It failed once, so it’s all hype.”

Hold both ideas: AI is powerful and unreliable.

Both are errors in judgment, and both are easy to fall into. Overconfidence is reinforced by real successes. Aversion is reinforced by old examples. The evidence-based position is uncomfortable because it requires checking case by case.

Source: Mollick (2026), Chapter 1.

🧑‍⚖️ Responsible use of AI

🧑‍⚖️ Responsible use: four questions

This is the frame for the next several slides. It is my synthesis of Mollick’s Chapters 2 and 3, the syllabus, and the evidence below, so treat it as a teaching device. Tell students this is what they should remember from today.

🎓 Is it allowed? In DANL 101

The default

Unless AI is explicitly permitted, work independently.

If permitted and you use it

Name the tools.

Submit your prompts and outputs.

Reflect on your use and learning.

Not sure? Ask first.

Taken directly from the course syllabus. Change this slide if a given assignment or the project has different rules. Permission for one activity doesn’t carry over to other assignments or the project, and this slide grants none.

🔒 Is it safe to share?

Generally fine

Public data

Made-up sample data

Your own code, with secrets removed

Keep out

Other people’s personal data

Passwords and API keys

Confidential work data

• Uploads may feed future training. Check the tool’s settings.
• Removing names alone may not be enough.

Mollick notes private modes exist and that you have to decide how much you trust those promises. At work, follow the employer’s rules, and remember FERPA and HIPAA exist. A fake sample with the same columns often gets you the help you need without sharing anything private.

Source: Mollick (2024), Chapter 3.

🧐 Can I check it? This code runs. Is it right?

“What share of JFK flights left late?”

flights |>
  filter(dep_delay > 0) |>
  summarize(share = mean(
    origin == "JFK"))

# A tibble: 1 × 1
  share
  <dbl>
1 0.327

flights |>
  filter(origin == "JFK") |>
  summarize(share = mean(
    dep_delay > 0, na.rm = TRUE))

# A tibble: 1 × 1
  share
  <dbl>
1 0.384

Both run, and both look fine. Which one answers the question?

Live code on nycflights13. Ask for a vote before you reveal the outputs. The right-hand version answers the question: of JFK flights, the share late. The left answers a different question: of late flights, the share from JFK. Both run without error and both give a believable number. Dropping na.rm = TRUE on the right returns NA, a second thing to check. Classwork 3C has students write the same comparison for LGA.

This is the failure that matters most for analysts, and it needs no made-up facts.

🔎 Hallucination: smaller, not gone

2,266

court cases with hallucinated AI content, logged worldwide

3–13%

of URLs cited by AI were made up, even with web search

• Open every source you cite.

Source: Charlotin database, updated Nov 5, 2026; Rao, Wong, and Callison-Burch (2026), 2025-era models.

One slide on purpose. Students already know the word. The first number counts incidents, not a rate, and the database covers cases where fabricated content was found or implied. The second is a preprint on 2025-era models, and “made up” there means a link that does not resolve and has no archived copy. Measured rates vary a lot by task, so resist quoting a single rate. Mollick’s 2026 chapter still lists nonexistent references as a live failure.

🙇 AI tends to agree with you

49%

more often than people, 11 leading models affirmed users’ actions

A better prompt

“What is the strongest argument against my conclusion?”

• Agreement is not evidence.

Source: Cheng et al., Science 391 (2026), as reported by the Associated Press.

The term is sycophancy. It follows from RLHF in Part 2: people tend to rate agreeable answers highly. In that study users still rated the flattering answers as trustworthy and preferred them. Mollick made a similar point in 2024: when pressed, the goal of pleasing you can beat the goal of being accurate.

🧠 Protect your learning

50%

quiz score, coders who had AI

67%

quiz score, coders without AI

• AI wasn’t significantly faster. The gap was largest on debugging.
• Try first. Ask AI to explain. Redo it yourself.

Randomized trial, 52 mostly junior engineers. Source: Anthropic (Jan 2026).

Done by an AI company, and small, so say both. It measured understanding soon after the task and did not test agentic tools. Those who asked for explanations scored better than those who delegated. A useful contrast: “Give me the answer” versus “Give me one hint, then check my explanation.” Mollick’s calculator comparison: you still learn arithmetic first, and the tool then lets you go further.

📝 Will I disclose? Keep a short record

Tool: which AI, which version

Prompts and outputs: attached

What I checked: ran the code, confirmed counts in summarize()

What I changed: fixed the denominator; rewrote the claim

What I learned: why the first version answered a different question

• It protects you, and it’s good analysis practice.

This is what the syllabus already asks for: tools, prompts and outputs, and a reflection. The “what I checked” line is the one that shows judgment. Use this same record in the group project where AI use is permitted.

🏷️ Generated text isn’t evidence

An AI draft

“Customers are happier this month.”

Evidence needed

Comparable feedback records from both months

• Credit real sources, and disclose AI help.
• Label invented data or images as illustrations.

Persuasive wording can’t establish this claim. If no evidence exists, remove the claim or say what is missing. AI-generated customer quotes, invented citations, and filled-in observations are not collected evidence. Clearly labeled fictional data are fine for practice, as in our café case. Adapted from the November 7 draft.

Source: Mollick (2024), Chapters 2 and 3; course syllabus.

🚦 Green, yellow, or red?

1. AI explains an R error on an assignment that allows it.
2. You paste classmates’ grades to “find patterns.”
3. You cite a paper AI suggested, without opening it.
4. AI drafts a chart title; you check it against the data.
5. You submit an analysis you can’t explain.

This is Classwork 3A. Four minutes in pairs, then discuss. 1 green, if disclosed. 2 red: other people’s private data. 3 red until you have opened and read it. 4 green, with disclosure where required. 5 red, even if the work is correct. Ask which question from the four-question flow each case fails.

☕ Auditing an AI-assisted analysis

☕ The café decision and the evidence

• A café manager is considering a change to weekend staffing.
• Evidence: eight observed hours and six invented comments.
• AI can help organize the work. The evidence has to support the recommendation.

All café records and comments are synthetic teaching examples. They don’t describe a real café or a representative sample of customers. Reserve the flawed response in Classwork 3B for students to audit themselves. Adapted from the October Lecture 11 draft.

✅ Four checks for an AI-assisted analysis

Rows

Which rows went in? Did a filter drop any?

Math

What is the denominator? Any missing values?

Code

Did I run it? Can I explain each line?

Claim

Does the evidence support it? Cause, or correlation?

Course framework. The slides that follow walk through each check with the café data. They compare the 10:00 and 14:00 hours on purpose: Classwork 3B asks about weekend revenue, so these examples leave those answers for students to find.

🔢 Rows: what does one row count?

nrow(cafe_sales)
[1] 8
sum(cafe_sales$orders)
[1] 240
sum(cafe_sales$revenue)
[1] 1200

• 8 observed hours, 240 orders, $1,200.
• Counting rows counts hours. It doesn’t count orders or customers.

Live code. Ask why counting rows can’t count customers. The output describes these eight observed hours, not the café’s total business for a week. Adapted from the October Lecture 11 and November 7 drafts.

🧮 Rows: which rows went in?

“How many orders came in the 10:00 hours?”

morning <- cafe_sales |>
  filter(hour == 10)

nrow(morning)
[1] 4
sum(morning$orders)
[1] 115

• Four observed hours, 115 orders.
• Ask AI to explain each line, then inspect morning yourself.

Live code. This total describes the selected rows, not every 10:00 hour. An explanation should match the actual code, and students can check it by viewing the data frame. Adapted from the October Lecture 10 draft, which used weekend revenue. This version avoids the Classwork 3B answers.

✅ Rows: do the totals add up?

Hour	Observed hours	Orders	Revenue ($)
10	4	115	575
14	4	125	625

• Observed hours add to 8, matching the original table.
• Orders add to 240, matching sum(cafe_sales$orders).
• An independent check beats an AI saying “correct.”

Live table. Adapted from the October Lecture 10 draft, which grouped by day type. Grouping by hour here avoids the Classwork 3B and 3G answers.

🧾 Math: which denominator?

Percentage claim	The denominator must match
Share of observed hours with long waits	Observed hours with a recorded wait
Share of comments about service	Comments under the stated labeling rules
Share of customers dissatisfied	Customers in a suitable survey

• Six comments can’t give the share of all customers.

No percentage is estimated on this slide. These fictional comments don’t form a random survey, and a comment can mention more than one topic. This is the JFK example again in words. Adapted from the October Lecture 11 draft.

🕳️ Math: what about missing values?

mean(cafe_sales$wait_min, na.rm = TRUE)

[1] 6.285714

• This averages the seven hours with a recorded wait.
• na.rm = TRUE skips the missing value. It doesn’t recover it.
• Each hour counts equally, so this isn’t the average customer’s wait.

Live code: 44 divided by 7, about 6.29 minutes. Ask why replacing NA with zero would change the meaning. Classwork 3C practices this with flights. Adapted from the October Lecture 11 and November 7 drafts.

💻 Code: showing it isn’t running it

• An AI answer may show code, a predicted result, or an actual tool result. They aren’t equal evidence.
• Run it in R, and inspect the output, warnings, and errors.
• A claim like “I checked the file” needs proof that it did.

Some AI applications can execute code, and others only generate text. Check the actual execution record and verify that it used the intended input. A successful run can still contain the wrong analysis. Students can do all of this in RStudio with no AI tool. Adapted from the October Lecture 10 and Lecture 11 drafts.

🧯 Code: asking for debugging help

I expected total revenue. Here are my code, the error, and the column names. Explain the problem and suggest the smallest fix.

sum("revenue")

sum(cafe_sales$revenue)

[1] 1200

• "revenue" is text. $revenue selects the column.
• Give a small reproducible example, then run the fix.

The left line fails with “invalid ‘type’ (character) of argument.” Remove private information from any example. Code can stop raising errors and still compute the wrong quantity. Adapted from the October Lecture 10 and November 7 drafts.

📊 Claim: does the chart match the data?

ggplot(cafe_hour_summary,
       aes(x = factor(hour),
           y = orders)) +
  geom_col(fill = "#0072B2",
           width = 0.6) +
  labs(x = "Observed hour",
       y = "Orders")

• Check the values, labels, and zero baseline against the table.
• Each bar covers four observed hours.

Live chart from the table you just checked. Check the variables, aggregation, units, group order, and scale of any generated chart. Adapted from the October Lecture 10 and Lecture 11 drafts. Classwork 3G has students do this for revenue.

🗣️ Claim: a finding needs its scope

In these eight observed hours, the 14:00 hours had 125 orders, compared with 115 in the 10:00 hours. Each group covers four observed hours.

• It names the comparison and the observations behind it.
• It doesn’t say what is typical, or why the difference exists.

Totals are exact for the fictional dataset. Classwork 3B asks students to write this kind of statement for weekend revenue. Adapted from the October Lecture 10 and November 7 drafts.

🔗 Claim: a difference doesn’t explain its cause

A comparison

Describes a pattern in the observed records

A causal claim

Says changing one thing would change the outcome

• Weather, events, and promotions could all matter. These records can’t separate them.

Use everyday language. Students don’t need causal-inference methods. These alternatives are hypothetical, not measured in the fictional data. Adapted from the October Lecture 11 draft.

🧾 Claim: what evidence would support it?

Draft claim	What to check
“Sales doubled”	Original totals and the comparison period
“Customers want more staff”	The comments, who wrote them, and other explanations
“Weekends cause higher sales”	Whether the design can support a cause

• Separate a wrong number from a conclusion the data can’t establish.

AI output is not a source for the café’s underlying business facts. Teach the method here and leave the worked answer to the classwork. Adapted from the November 7 draft.

🔬 When the AI knows more than the reviewer

• Ask for a result small enough to check independently.
• Use known totals, sample rows, or a second calculation.
• A second AI agreeing isn’t independent evidence.
• Get expert help when a conclusion is beyond what you can assess.

Fluency, confidence, and repeated agreement are weak substitutes for evidence. What you contribute: knowledge, such as knowing a row is an hour; judgment, such as deciding whether eight hours can justify a staffing change; and review, such as recomputing a total. A person’s presence doesn’t guarantee quality. Adapted from the October Lecture 11 and November 7 drafts.

Source: Mollick (2024), Chapter 3.

✍️ Classwork: Part 3

Open the AI classwork: Part 3

• 3A: green, yellow, or red.
• 3B–3C: run the four checks, then write your own flights code.
• 3D–3E: write better prompts, and keep a record.

3A is the green, yellow, or red activity from the responsible-use section. Part 3 is already the fullest lecture, so 3B to 3E will need time after class. 3B and 3C are the most valuable if you can only keep two, because they use R.

✅ Part 3 takeaways

• Alignment is everyone’s job, including yours.
• Four rules: invite, stay in the loop, set the role, expect change.
• Ask: allowed, safe, checkable, disclosed?
• The main risk for analysts is plausible and wrong.

Next time: AI that acts, and what it means for your job.

Part 4: Agents and Jobs

Estimated core path, 50 minutes: agents 20; jobs and evidence 20; your judgment and wrap-up 10. Skip first if short on time: Automation or agent?, A general purpose technology, Three ways AI changes an analyst’s work, Early productivity gains, What DANL alumni said, Beyond work, Decision errors have costs, and A reproducible handoff.

Reading: selected sections of Co-Intelligence Chapter 6, plus the opening of Co-Existence Chapter 1, plus the reading guide’s 2026 update.

🥇 Forecasts keep missing

• If experts missed this, be careful with anyone’s forecast.

That includes forecasts of mass unemployment and forecasts that AI has hit a wall. Keep the humility when we get to jobs.

Source: Mollick (2026), Co-Existence, Chapter 1.

🪝 A capability overhang

88%

of organizations reported using AI in 2025

Single digits

agent deployment, in nearly every business function

• Capability overhang: AI can already do more than we use it for.

Source: Stanford HAI, 2026 AI Index, Economy chapter; term from Mollick (2026).

Change takes time because systems change more slowly than tools. Mollick’s Chapter 6 makes this point about jobs and organizations. Even if progress stopped today, adoption would keep spreading.

🔄 AI agents

🪜 From chat to agents

🔄 What is an AI agent?

• In 2023, Mollick’s early agent built a 21-step plan for one sentence, then gave up.

Mollick predicted in late 2023 that the next phase would be semi-autonomous AIs given a goal. The early agent he tried was BabyAGI. Definition to remember: an agent has a goal and tools, and it decides its next step from what it sees.

Source: Mollick (2024), Chapters 3 and 6.

⚙️ Automation or agent?

Automation

Code repeats a routine task, such as the same weekly report from new data.

AI agent

Pursues a goal over several steps, and uses results to decide what to do next.

• Changing its plan mid-task doesn’t retrain the model.
• A fixed R script may be the better tool for recurring calculations.

Keep this conceptual. Not all agents are reliable or autonomous in the same way. Adapted from the October Lecture 11 draft.

Source: Mollick, “Agency and Agents,” August 31, 2026, https://www.oneusefulthing.org/p/agency-and-agents

🧰 An agent on a data task

The agent…	You check…
reads sales.csv	Right file? All the rows?
writes and runs dplyr code	Does it answer the question?
fixes its own errors	What did it drop or change?
makes a chart	Are the axes and labels honest?
writes a summary	Can you trace every number?

Connect to their work this semester. Everything in the right column uses what they learned in Lectures 3 to 8, so this is why those skills still matter. Illustrative workflow, not one specific product. Ask for the script and chart alongside any report, so there is something to check.

📏 Agents handle longer tasks

• Doubling about every four months since 2023.

METR measures how long a task takes a human expert, then finds the length at which the AI succeeds half the time. Since 2023 this has doubled about every 129 days (95% CI 104 to 158), roughly every four months.

Caveats to say aloud. Tasks are mostly software, machine learning, and cybersecurity, and METR says this varies widely across other work. A 50% success rate is not reliable delegation. Log scale: each gridline is a big jump. The page stopped active updates in September 2026, and the newest point (Claude Mythos Preview, 17.4 hours) sits above the 16-hour line METR calls unreliable.

Values and source link are in danl-lec-tmp/ai-and-data-analytics-figures.R.

🚧 Longer doesn’t mean reliable

1 in 3

agents fail this often on structured benchmarks

38%

best agent, right first try on real data questions

85%

of wrong answers: bad plan or faulty code

• The dangerous failure runs, looks right, and is wrong.

Source: 2026 AI Index; Ma et al. (2026), Data Agent Benchmark; Rawat and Flek (2026).

Data Agent Benchmark: 54 queries on 12 datasets and 4 database systems, with models available in early 2026, so it may already be somewhat dated. It is a preprint. Picking the wrong data was the less common failure. In astrophysics workflows, Rawat and Flek found silent incorrect computation was the main failure: code that runs and gives plausible wrong results. This is the JFK example from Part 3, at a larger scale.

🕵️ A new risk: agents read instructions

• Prompt injection is first on OWASP’s Top 10 risks for LLM apps.

Source: OWASP Top 10 for LLM Applications, 2025 edition.

A callback to Mollick’s hidden biography text. When the AI only chats, a trick like that is funny. When it can send email or delete files, it’s a security problem. The web page, spreadsheet, or email the agent reads can carry instructions it treats as its own. Illustrative example; the address is fake.

🚦 Drafting and acting need different permission

Proposed action	What it needs
Draft a recommendation	The analyst checks the evidence
Save changes to original data	Approval, and an unchanged copy
Email staff or publish a report	Approval from an authorized person
Buy a service or change a schedule	Explicit authorization first

• Giving AI a question doesn’t give it permission to act.

A draft can be revised, while a sent message reaches real people. Authorization can cover a bounded task in advance, so not every edit needs new permission. This lesson doesn’t authorize any of these actions. Adapted from the October Lecture 11 and November 7 drafts.

Source: Mollick, “Agency and Agents,” August 31, 2026, https://www.oneusefulthing.org/p/agency-and-agents

🛡️ Four rules for delegating to an agent

Least access

Only the files and tools it needs

Approve before acting

Sending, deleting, or paying needs a person

Keep a log

See what it read, ran, and changed

Review the result

Your name goes on it, so you check it

This extends the four questions from Part 3. Rule of thumb: drafting is cheap to undo, while acting may not be. The more a mistake would cost, the more human checking belongs before the action.

💼 AI and jobs

🏭 A general purpose technology

Earlier examples

Steam power and the internet touched every industry, and they spread slowly.

Generative AI

ChatGPT reached 100 million users faster than any earlier product.

• General Purpose Technology is an economic term, and it’s also abbreviated GPT.
• There’s little runway, so people and organizations learn as they go.

Mollick argues generative AI may rival earlier General Purpose Technologies. The user figure is from early 2023. Adapted from 2025 Lecture 4.

Source: Mollick (2024), Introduction and Chapter 1.

📦 A job is a bundle of tasks

• AI takes tasks. Whether it takes the job is a separate question.

Spreadsheets changed accountants’ tasks without ending accounting. Jobs also sit inside systems such as liability rules, customers, and managers, which change slowly. Telephone operators are the counterexample: when direct dialing arrived, operator jobs fell 50 to 80 percent, and the most experienced took the largest long-term earnings hit. So it does sometimes happen. The example sort is mine.

Source: Mollick (2024), Chapter 6.

✍️ Sort your future job

• Pick a job you might want.
• List five tasks it involves.
• Sort them: just me, delegated, automated.

Which sort would you defend even if AI got much better?

This is Classwork 4C. Three minutes alone or in pairs. The final question separates “AI can’t do this” from “a person should do this.” Mollick says the second kind of task grows in importance as AI improves.

💼 Three ways AI changes an analyst’s work

Automate

Produce a routine sales report with less manual work.

Assist

Explore more questions and comparisons.

New responsibilities

Review AI outputs and manage the workflow.

• AI exposure isn’t job loss. Hiring and pay may still change as jobs are redesigned.

Source: ILO (2025); Massenkoff and McCrory (2026).

These are possible channels, not predictions of net job gains or losses. The ILO assesses potential exposure. Don’t promise that any occupation is immune. Build data skills, subject knowledge, and judgment alongside AI skills. Adapted from the November 7 draft.

📊 The big picture: no clear job shock yet

Yale Budget Lab (Oct 2026)

No clear AI link to changes in occupational mix, employment, or unemployment

Anthropic (Mar 2026)

No systematic rise in unemployment for highly exposed workers

Source: Yale Budget Lab; Massenkoff and McCrory (2026).

Yale updates its tracker regularly, so check the date before class. Anthropic has a stake in this question, as an AI developer, so name that when you cite it. Mollick’s 2024 chapter reported that most surveyed economists expected little effect on overall jobs in the near term, consistent with this so far.

🐦 But watch young workers

• A gap of about 19%. Experienced workers show no comparable gap.

Key numbers: a gap of about 19% by June 2026, up from about 15% a year earlier. Check the arithmetic with students: 89 versus 110 is about 19% lower. Experienced workers show no comparable gap. Same paper’s Fact 1: no widespread, economy-wide displacement. A monthly dashboard is at digitaleconomy.stanford.edu/project/indicators/canaries-dashboard/.

Source: Brynjolfsson, Chandar, and Chen, Canaries in the Coal Mine? (rev. Aug 12, 2026).

🧪 Read the evidence like an analyst

• Different lens: whole labor force vs. ages 22–25.
• Different data: national surveys vs. ADP payroll records.
• Cause? The authors call theirs descriptive, not causal.
• How? Mostly less hiring, not more firing.

Source: Brynjolfsson, Chandar, and Chen (rev. Aug 2026).

These can both be true: a small group can change sharply while the total barely moves. The authors’ checks: the gap holds excluding tech firms and controlling for interest rates and remote work. Their caveats: gaps shrink when controlling for education, some trends predate generative AI, and gaps are larger in ADP than in national surveys. Anthropic finds a smaller related signal: job-finding rates for ages 22 to 25 in exposed occupations about 14% lower than in 2022, just barely statistically significant.

🔀 Automate, or complement?

Young-worker jobs falling

Where AI mainly automates tasks

Codified knowledge

Jobs flat or rising

Where AI mainly complements workers

Tacit, experience-based knowledge

How do you get experience if entry-level tasks shrink?

The tacit and experience finding is about experienced workers in those occupations. The authors say this is consistent with generative AI being good at encoded knowledge. Still descriptive. Connect to projects: messy real work builds the experience AI can’t hand you.

Source: Brynjolfsson, Chandar, and Chen (rev. Aug 2026).

🎓 New graduates right now

5.2%

recent-graduate unemployment, lowest in over a year

42.5%

recent graduates underemployed, ticking up

• Better in 2026, still challenging. AI’s share of the blame is debated.

2026 Q3. Source: New York Fed, The Labor Market for Recent College Graduates.

Underemployed here means working in a job that typically does not require a bachelor’s degree. These numbers describe the labor market as a whole and do not isolate an AI effect. Check the page before class: it updates each quarter.

🎙️ What DANL alumni said about AI at work

Coding

Most daily analytics code was AI-assisted, with human review.

Rules

Policies differ. The New York Fed bars public tools and uses a secure in-house system.

Judgment

“You need to know what you’re asking it to do, and what the right answer looks like.”

Fall 2025 career sessions. Their own estimates and workplaces, as of fall 2025.

Anecdotes from a small number of Geneseo alumni, so treat them as examples and not as measurements. They estimated that 60 to 80 percent of their code was AI-assisted. Check with the alumni before naming anyone, and update if newer sessions have happened. Adapted from 2025 Lecture 8.

⚡ Early productivity gains

20–80%

gains in early studies of AI, on tasks from coding to marketing

18–22%

what steam power added in factories

• Results vary by task, oversight, and how carefully they were measured.

Source: Mollick (2024), Introduction.

Mollick wrote this in late 2023 about early studies, and he also notes economists have struggled to show a long-term productivity effect from computers and the internet. The next slide shows why self-reports and quick studies need care. Adapted from 2025 Lecture 4.

⏱️ Feeling faster isn’t being faster

19% slower

measured time on real tasks, with AI

20% faster

what those same developers believed

• METR now calls this result dated; newer tools likely help.
• The lasting lesson: measure it.

Randomized trial, 16 experienced developers, 2025. Source: METR (2025); METR (2026) update.

Be careful here. This does not show AI slows people down today. METR’s 2026 follow-up suggests developers are likely sped up now, yet calls that data only weak evidence on how much, partly because developers were unwilling to work without AI. Contrast with the 20 to 80 percent gains Mollick cited in 2024. Self-reports are data about belief.

🤔 A puzzle to discuss

2024 evidence

AI helps lower-skilled workers the most.

2026 evidence

Entry-level hiring is slipping in exposed jobs.

How can both be true? Talk with a neighbor.

This is Classwork 4E. Three minutes. Possible answers: firms can give fewer people more output; the skill floor rises, so a novice with AI is no longer scarce; experienced workers can judge the output and novices can’t yet; firms hire less while they wait to see; or the two studies measure different things and the second one isn’t causal. There’s no settled answer, and that’s fine to say.

Source: Mollick (2024), Chapter 6; Brynjolfsson et al. (2026).

🌐 Beyond work

Education

AI tutors, feedback, and new ways to assess

Media

Personalized content and disrupted industries

Information

Misinformation at scale, and hard to detect

Creativity

Who is the author, when the work is shared?

A good place to ask: where do you draw the line between assistance and authorship? Adapted from 2025 Lecture 4.

Source: Mollick (2024), Introduction and Chapter 2.

🎤 Your analysis, your judgment

⚖️ Decision errors have costs

Possible error	Possible consequence
Overstate how much staffing is needed	Unnecessary labor cost
Understate a real queue problem	Longer waits and lost business
Treat a few comments as everyone’s view	A poorly targeted response

• State the uncertainty, and what evidence would reduce it.
• The manager, with authority, decides whether to act.

These are possible consequences, not quantified findings from the synthetic data. Don’t derive a staffing requirement from eight observed hours. An analysis suggestion differs from authority to schedule staff or spend money. Adapted from the October Lecture 11 draft.

🗣️ A recommendation needs a defensible message

Part	What the audience needs
Decision	The specific choice being considered
Evidence	A relevant comparison, with units and coverage
Limitation	What the data can’t establish
Next step	A proportionate action, or more evidence

• AI can suggest wording. You own the meaning.

Use this as a communication structure, not a fill-in answer to the café case. A cautious recommendation for further observation can be reasonable. Adapted from the October Lecture 11 draft.

🧵 A project needs a traceable argument

Project element	What someone can inspect
Finding	The chart and its calculation
Calculation	The R script and relevant rows
Data	Source, variables, units, and dates
AI assistance	What it proposed, and what you verified

Trace one claim back through a chart, a calculation, and the rows behind it. Prompts don’t replace a reproducible R analysis. This doesn’t change the project’s AI rules. Adapted from the November 7 draft.

📦 A reproducible handoff

• Preserve the original data, and describe any changes.
• Keep an R script that reproduces the summaries and charts.
• Use labels, units, and captions your audience can follow.
• State the main limitation, and any required AI documentation.

Encourage a fresh R session check before the final presentation. Adapted from the October Lecture 11 draft.

🌟 What stays valuable

Taste

Which chart tells the truth?

Deep expertise

Spotting the wrong denominator

Context

What these numbers mean here

Agency

Deciding on uncertain information

The four categories are Mollick’s sources of durable human value. The analyst examples are mine. His warning: don’t cling to whichever tasks AI hasn’t automated yet.

Source: Mollick (2026), Co-Existence, Chapter 1.

🧭 What to do this semester

• Build the fundamentals: R, statistics, reasoning.
• Practice delegating and checking, where it’s allowed.
• Keep proof of judgment: your project, your choices, your checks.

Fundamentals come first because you can’t check what you can’t read. Part 3’s learning evidence backs this up. A group project is one place they can show judgment, which is harder to fake than polished output.

✍️ Classwork: Part 4

Open the AI classwork: Part 4

• 4C, 4E, 4F: your job sort, the puzzle, and the exit ticket.
• 4A–4B: an agent’s log, and a hidden instruction.
• 4D: rewrite two headlines to fit the evidence.

4C, 4E, and 4F are already timed into the lecture. 4A, 4B, and 4D need time after class. 4D is a good one to discuss when the group project studio starts.

✅ Takeaways and exit ticket

• Agents act, so there’s more to check.
• Overall job shock: not visible yet. Young workers: a warning sign.
• The change may be coming either way. What we do with it is our choice.

Name one task you’d delegate and one you’d keep. Why?

Reading and full source list: AI and Data Analytics reading. Questions: AI classwork.

The last line paraphrases Mollick’s framing in Co-Existence Chapter 1. The exit ticket is Classwork 4F. Collect it if you’d like material to open the project studio.