Step 1: Marco's email
Open the chat with Marco Quispe. He runs Cumbre Adventures in La Paz, Bolivia -- mountain biking on Death Road, trekking in the Cordillera Real, climbing Huayna Potosi, paragliding. Fifteen employees, seven years in business.
He sent an email. Read it. Marco's writing is warm and scattered -- he'll tell you about Death Road before he gets to the data. The key information is buried: he redesigned his booking page, ran what his web developer called an "A/B test," and now has 60 days of data from about 4,200 visitors. Half saw the old page, half saw the new one.
The new page has a higher overall booking rate. But premium trek bookings dropped. His web developer says the experiment worked. His operations manager says it's losing money. Marco is in the middle.
Step 2: Discover the experiment setup
Message Marco to understand how the experiment was set up. He is a primary client -- he shares what you ask for but won't volunteer everything. No suggested message buttons. You initiate.
Through conversation, discover the basics:
- 50/50 traffic split between old page (version A) and new page (version B)
- 60-day duration
- About 4,200 total visitors
- Data exported from his booking platform
Marco goes on tangents. Ask about the data and you may hear about the sunrise on Huayna Potosi first. He calls the statistical results "the numbers" or "the statistics." He's genuinely trying to be data-driven but doesn't have the vocabulary.
Notice what he does not mention. He has hidden constraints -- an ad budget change, a pricing display difference between the page versions, and a language limitation. These only surface if you ask the right questions in later units.
Step 3: Project setup
Open a terminal and start Claude Code:
cd ~/dev
claude
Paste this prompt:
Create the folder ~/dev/analytics/p8. Download the project materials from https://learnbydirectingai.dev/materials/analytics/p8/materials.zip and extract them into that folder. Read CLAUDE.md -- it's the project governance file.
Claude creates the folder, downloads the materials, and reads CLAUDE.md. That file describes Marco's situation, the deliverables, the tech stack, and the work breakdown. Once Claude confirms it has read CLAUDE.md, you are set up.
Step 4: The data dictionary
Open materials/data-dictionary.md. It describes the dataset: one CSV file with about 4,200 rows, one row per visitor. Seven columns: visitor_id, page_version (A or B), visit_date, tour_selected, booking_completed, booking_value, and visitor_source.
The experiment setup section confirms the 50/50 random split and the 60-day window. Page A is the original booking page. Page B is the redesigned version.
The visitor_source column tracks how each visitor arrived: organic search, paid ads, hostel referrals, or travel agency referrals. This becomes important later when you examine whether all traffic sources behave the same way.
Step 5: Profile the data
Direct AI to load the dataset and profile it:
Load materials/ab-test-data.csv and give me a data profile: total rows, column types, value distributions. How many visitors per page version? What's the overall booking rate for each version?
Check the basics. The row count should be close to 4,200. The page version split should be roughly 50/50 -- about 2,100 visitors for each version. The overall booking rate for page B should be slightly higher than page A.
Look at the booking value distribution. Death Road tours cluster in the $65-80 range. Premium treks (Huayna Potosi, Cordillera Real) range from $180-400. Paragliding sits at $90-120. Non-bookings are $0. The distribution tells you this is not one uniform product -- tour types have very different price points and very different margins for Marco.
Step 6: The ambiguity
Here is what the profile reveals: page B has a slightly higher overall booking rate than page A. If you stopped here, the answer would be simple -- the new page is better.
But look at the tour_selected column. Break down bookings by tour type and page version. The overall average hides a more complicated story. Some tour types may be doing better on page B while others do worse.
This is not one question. "Is the new page better?" has at least two framings: better overall, and better for each tour type. The answer might be different depending on which framing you choose. Marco's web developer is looking at the overall number. His operations manager is looking at the premium trek number. They are both looking at real data and reaching opposite conclusions.
The next unit defines exactly what "better" means and frames the specific questions the test can answer.
✓ Check: The dataset should have approximately 4,200 rows. The two page versions should have roughly equal visitor counts (~2,100 each). The overall booking rate for the new page should be slightly higher than the old page.