AI and Data Analytics

Reading for Lecture 9, Parts 1–4

Author

Byeong-Hak Choe

Published

November 16, 2026

How to use this reading

Lecture 9 has four parts, each about one 50-minute class meeting. Before each part, read the assigned book sections, then read the matching part of this guide. The guide explains the core ideas in plain language and adds what a book cannot: evidence from the last two years.

We use two books by Ethan Mollick. Co-Intelligence (2024) is our required textbook. He finished it in late 2023, so its model names and test results are history now, though its ideas have held up well. Co-Existence (2026) is his follow-up, and we use only its first chapter.

Open the lecture slides · Open the classwork

Reading plan

Part Assigned reading About
Part 1. Learning from data Co-Intelligence: Introduction, and Chapter 1 up to the section “Scary? Smart? Scary-Smart?” 25 min
Part 2. Inside generative AI Co-Intelligence: rest of Chapter 1. Co-Existence, Chapter 1: section “Jaggedness” 15 min
Part 3. Working with AI, responsibly Co-Intelligence: Chapters 2 and 3. Co-Existence, Chapter 1: section “Co-Existence on the Jagged Frontier” 40 min
Part 4. Agents and jobs Co-Intelligence, Chapter 6: the opening, “Tasks and the Jagged Frontier,” “Tasks for Me, Tasks for AI,” and “From Systems to Jobs.” Co-Existence, Chapter 1: the opening, before “Jaggedness” 25 min

Times are rough estimates and don’t include this guide. Part 3 is the longest, so start it early. See Brightspace for how to access the Co-Existence chapter.

Optional, if you want more: Co-Intelligence, Chapter 7, section “After the Homework Apocalypse” (Part 3), and Chapter 9, on four possible futures for AI (Part 4).

Part 1: Learning from data

What to look for

In the Introduction, notice both the excitement and the worry in Mollick’s classroom. His students asked whether their chosen careers would survive, and you may have asked the same thing. In Chapter 1, notice how AI meant something different in the 2010s than it does now.

Machine learning in plain language

Machine learning (ML) lets a computer improve at a task by finding patterns in data, rather than following step-by-step rules written by a person. Writing rules that define “cat” is nearly impossible. Showing a computer thousands of photos labeled cat or not cat works far better.

Type How it learns Example Predicts
Supervised: regression From examples with answers Home features → price A number
Supervised: classification From examples with answers Email → spam or not A category
Unsupervised: clustering With no answer key Customers → groups Groups you must interpret

You have already used a simple supervised model. When you wrote geom_smooth(method = "lm") in Lecture 8, R learned a line from observed x and y values and used it to predict y.

Labeled data is the answer key that makes supervised learning possible. Labels come from people, clicks, ratings, and existing records. They cost money and can be subjective, so a model is only as good as the labels behind it.

A fair test

A model can look excellent on the data it trained on simply because it has memorized that data. This is overfitting. The fair test is data the model has never seen, the same way a real exam uses new questions. Hold on to this idea. It will explain why a high exam score from an LLM doesn’t always impress researchers.

From ML to deep learning

Deep learning is ML using neural networks with many layers. It works especially well on messy data such as images, text, and sound. Each connection in a network has a weight, a number that sets how much that connection matters. Training adjusts the weights to reduce errors. The original ChatGPT had 175 billion of them, and no person set them by hand. That scale is why even the people who build these systems often can’t explain a specific output.

What ML doesn’t change

ML doesn’t choose your question. A person still decides what to predict, which data to use, and how to judge the result. Those are analyst decisions, and they’re what this course has been training you to make.

Questions to bring to class

  1. Name a task in your intended career that needs fixed rules, and one that would benefit from learning from data. What makes them different?
  2. Mollick says early predictive AI struggled with “unknown unknowns.” Give an example from a business you know.

Classwork: Part 1 questions

Part 2: Inside generative AI

How an LLM works

A large language model (LLM) is still doing prediction. It predicts the next token, which is a word or a piece of a word. It adds that token to the text and predicts again, one token at a time, until the answer is complete. Mollick calls this a very elaborate autocomplete, and that’s technically accurate, even if the results don’t feel like autocomplete.

The breakthrough behind this came in 2017, when Google researchers introduced the Transformer. Its key idea, attention, lets a model weigh which earlier words matter for the current prediction. In “the bank by the river flooded,” attention links bank to river, so the model doesn’t treat it as a place that holds money.

A prompt becomes tokens, then numbers, passes through attention, and yields next-token probabilities; the chosen token is added and the process repeats.

A prompt becomes tokens, then numbers, then passes through attention to produce next-token probabilities.

Two phases of training

In pretraining, the model reads an enormous amount of text and learns to predict what comes next. No one labels this text. It’s a strange mix of websites, books, and even corporate email, so a model absorbs biases and errors along with knowledge.

In fine-tuning, people step in. With Reinforcement Learning from Human Feedback (RLHF), they rate answers as better or worse, and the model learns to prefer what they rate highly. This makes a model more helpful and safer. It also teaches a model to please people, which will matter in Part 3.

The jagged frontier

Mollick’s most useful idea is the jagged frontier. Imagine a map of tasks, easy for people at the center and hard at the edge. Now draw a wall around everything AI does well. You might expect a neat circle. Instead the wall is wildly jagged and invisible. Olympiad math is inside it, while saying “I don’t know” can be outside it.

Two radial maps of AI capability. Newer models cover more area than older ones, but the boundary stays jagged and some tasks easy for people remain outside it.

Schematic only, not a measurement. Concept from Mollick (2024, 2026).

In Co-Existence, Mollick adds that the frontier keeps moving, and unevenly. Two ideas may help explain why:

  • Emergence or bottleneck? Sometimes a model suddenly gains an ability as it grows. Researchers disagree on whether that’s a truly new ability or just one weak link being fixed.
  • Transcendence. A model imitates a population, not a person. If individual errors don’t line up, they can cancel out, and knowledge from different specialists can combine. That’s how a model can outperform the people whose writing trained it.

Questions to bring to class

  1. Find one task that you expected AI to do well and that it did badly, or the reverse. What does that tell you about the frontier?
  2. How would you test whether a model’s high exam score reflects real ability?

Classwork: Part 2 questions

Part 3: Working with AI, responsibly

Alignment

Alignment means making sure AI serves human interests. Chapter 2 opens with the famous paper-clip thought experiment, then deliberately turns away from it. Mollick’s argument is that fixating on doom leaves the decisions to a few companies, while real choices are already being made: whose data trains these systems, whose biases they carry, who does the low-paid work of rating disturbing content, and who misuses the tools.

Some of this has moved since the book. In Bartz v. Anthropic, a judge found that training on lawfully acquired books was fair use, while keeping pirated copies in a permanent library was not. A $1.5 billion settlement with authors received final approval in July 2026. Many broader copyright questions remain open.

Four rules for co-intelligence

Rule What it means In this course
1. Always invite AI to the table Experiment to learn what it can do Only where AI is permitted
2. Be the human in the loop Stay engaged and responsible You check it, and you answer for it
3. Treat AI like a person (but tell it what kind of person it is) Give it a role, audience, limits, and context Include your code, your error, and your question
4. Assume this is the worst AI you will ever use Limits you see today may not last Re-test instead of memorizing limits

Two traps

Co-Existence names two errors. Overconfidence comes from watching AI do something amazing and assuming it can do anything. Algorithm aversion comes from watching it fail once and writing off the whole technology. The sensible position is harder to hold: AI is powerful and unreliable, so you judge case by case.

Responsible use: four questions

Ask these before relying on AI for anything that matters.

Four checks before using AI: is it allowed, is the data safe to share, can I check the result, and will I disclose it. Any no means stop or change the plan.

Is it allowed? Our syllabus is clear. Unless AI is explicitly permitted for an assignment or activity, you must work independently. When it’s permitted and you use it, you must name your tools, include your prompts and outputs, and reflect on your use and learning. If you’re unsure, ask first.

Is it safe to share? Data you upload may be used in future training, depending on the tool and its settings. Don’t paste other people’s personal data, passwords, API keys, or confidential work data. A made-up sample with the same columns will often get you the help you need.

Can I check it? This is the question that matters most for analysts. Here are two pieces of code for “What share of JFK flights left late?”

# Share of late flights that came from JFK: a different question (0.327)
flights |>
  filter(dep_delay > 0) |>
  summarize(share = mean(origin == "JFK"))

# Share of JFK flights that left late: the right question (0.384)
flights |>
  filter(origin == "JFK") |>
  summarize(share = mean(dep_delay > 0, na.rm = TRUE))

Both run without errors and both give a believable number, so running the code isn’t enough. Before you trust an AI-assisted analysis, check four things: the rows (which went in?), the math (what’s the denominator? any missing values?), the code (can you explain each line?), and the claim (does the evidence support it?).

Will I disclose it? Keep a short record of the tool, your prompts and outputs, what you checked, and what you changed. If you’d rather not disclose it, treat that as a warning sign.

A note on hallucination

You’ve probably heard this word. A hallucination is confident, made-up content, such as a fake citation. Newer models do it less often in many tests, and rates vary widely by task, so be wary of any single number. It hasn’t gone away. A public database lists 2,266 court cases involving hallucinated AI content as of November 2026, and one 2026 preprint found that 3–13% of URLs cited by search-enabled models from 2025 were fabricated. The rule is simple: open every source you cite. For data work, though, the JFK example above is the bigger worry, because it involves no made-up facts at all.

Two habits worth building

Don’t take agreement as evidence. LLMs tend to flatter. A 2026 study in Science found that 11 leading models affirmed users’ actions 49% more often than people did. This is called sycophancy. Try asking, “What’s the strongest argument against my conclusion?”

Protect your learning. In a randomized trial with 52 mostly junior engineers, those who had AI while learning a new coding library averaged 50% on a later quiz, against 67% for those who coded by hand, and they weren’t significantly faster. The gap was largest on debugging. The study is small and came from an AI company, but the lesson fits what we know about learning: struggle does part of the work. Try first, ask AI to explain, then redo it yourself.

Questions to bring to class

  1. Where would you draw the line between assistance and authorship on a class project?
  2. Pick one of Mollick’s four rules that you think is wrong or incomplete today. Defend your view.

Classwork: Part 3 questions

Part 4: Agents and jobs

What is an AI agent?

A chatbot answers. An agent is given a goal and tools, then works through a loop: it plans, acts, looks at what happened, and decides what to do next, repeating until it’s done. In late 2023, Mollick predicted that agents would come next, and he described an early one that made a 21-step plan to write a single sentence and then gave up.

An agent loop: from a goal, the agent plans, acts, observes, and checks repeatedly, then returns a result for a person to review.

Three years later, the picture looks different in three ways.

Agents handle much longer tasks. The research group METR measures the length of task, in human-expert time, that an AI completes half the time. That rose from about 4 minutes for GPT-4 in 2023 to several hours by 2025, doubling about every four months since 2023. These are mostly software and machine-learning tasks, and METR warns that it varies widely in other work, and that measurements above 16 hours are unreliable.

Scatter chart with log scale: the length of task AI completes half the time rose from 4 minutes for GPT-4 in 2023 to about 17 hours for the newest model tested in April 2026.

Source: METR Time Horizon 1.1.

They’re far from reliable. The 2026 AI Index reports that agents still fail roughly one in three attempts on structured benchmarks. On a benchmark of real data questions, the best agent answered 38% correctly on its first try, and 85% of wrong answers came from a bad plan or faulty implementation. The most worrying failure is the one you’d miss: code that runs and gives a plausible, wrong result.

Most organizations haven’t caught up. About 88% of organizations reported using AI in 2025, while agent deployment stayed in the single digits across nearly all business functions. Mollick calls the gap between what AI can do and what we use it for a capability overhang.

New responsibilities when AI acts

When AI can act, errors have consequences. An agent also reads files and web pages as text, so hidden instructions in a document can steer it. This prompt injection tops OWASP’s list of risks for LLM applications. Four habits help: give an agent only the access it needs, require approval before anything hard to undo, keep a log of what it did, and review the result yourself.

Jobs are bundles of tasks

Mollick’s key move in Chapter 6 is to stop asking whether AI will take “a job” and to look at tasks. He sorts them into Just Me tasks (AI stays out), Delegated tasks (AI does it, you check), and Automated tasks (AI handles it unchecked). Jobs also sit within systems of customers, rules, and managers that change slowly.

What the labor market evidence shows

Researchers are still debating this, so read closely.

Overall, there’s no clear shock so far. The Yale Budget Lab (October 2026) finds that the occupational mix isn’t yet changing in ways that clearly align with AI, and that AI usage measures show no connection to changes in employment or unemployment. Anthropic economists (March 2026) find no systematic rise in unemployment among highly exposed workers.

Young workers are the warning sign. Using ADP payroll data, Brynjolfsson, Chandar, and Chen (revised August 2026) find that from November 2022 to June 2026, employment of 22-to-25-year-olds fell about 11% in the most AI-exposed occupations and grew about 10% in less exposed ones. Experienced workers show no comparable gap.

Line chart from November 2022 to June 2026 for workers ages 22 to 25: employment rose 10 percent in less AI-exposed jobs and fell 11 percent in the most AI-exposed jobs.

Lines connect the two reported endpoints, not the monthly path.

Read this like the analyst you’re becoming:

  • These findings can both be true. A small group can change sharply while the total barely moves.
  • It’s descriptive, not causal, in the authors’ own words. They ruled out some alternatives, such as interest rates and remote work, but note that gaps shrink when controlling for education and that some trends began before generative AI.
  • It shows up in hiring, not firing. The adjustment comes mainly from fewer young workers being hired.
  • The pattern depends on how AI is used. Declines concentrate where AI automates tasks. Where it complements workers, employment is flat or rising.

For context, the New York Fed reports recent-graduate unemployment of 5.2% in 2026 Q3, its lowest in more than a year, and underemployment of 42.5%.

Feeling faster isn’t being faster

In a 2025 randomized trial, 16 experienced developers took 19% longer with AI, while believing they had been 20% faster. METR’s 2026 update says that result is out of date and that developers are likely sped up now, though its newer data is only weak evidence of how much. So this doesn’t show that AI slows you down. It shows that how fast you feel isn’t a measurement.

What the book said, and what 2026 adds

Co-Intelligence (2024) What has happened since
Agents are next, and early ones get lost Task length has grown fast, though agents still often fail
Hallucination is a serious problem Less common in many tests, still appearing in court filings and citations
Early studies show 20–80% productivity gains Results are more mixed and depend on task and measurement
Most surveyed economists expect little near-term effect on total jobs Largely true so far, with a warning sign for young workers
AI helps lower-skilled workers most Entry-level hiring is slipping in exposed jobs: a puzzle we discuss in class
The BCG “outside the frontier” task fooled consultants using AI Mollick reports that task is now inside the frontier
Copyright is unsettled A $1.5 billion settlement, with broader questions still open

What stays valuable

Mollick warns against clinging to whichever tasks AI hasn’t automated yet. He points instead to lasting sources of human value: taste, deep expertise, broad contextual knowledge, and the agency to act on uncertain information. For an analyst, that means choosing the chart that tells the truth, spotting the wrong denominator, knowing what a number means in this business, and making a call when the data doesn’t settle it.

You can’t check what you can’t read, so learning R and statistics still matters even when AI can write the code.

Questions to bring to class

  1. Choose a job you might want. List five of its tasks and sort them into Just Me, Delegated, and Automated.
  2. AI seems to help novices most, yet entry-level hiring is slipping in exposed jobs. How can both be true? What data would help you find out?

Classwork: Part 4 questions

Sources

Books

  • Mollick, Ethan. 2024. Co-Intelligence: Living and Working with AI. Portfolio.
  • Mollick, Ethan. 2026. Co-Existence: The Next Phase of AI. Portfolio.

2025–2026 evidence

A note on sources. Several of these are preprints that haven’t finished peer review, and this field changes fast, so check the dates. Two studies come from Anthropic, a company that sells AI. Keep an eye on who produced a number and what stake they have. That’s a good habit for any data, and it applies to this page too.

Back to top