Metric definitions and automated tests

Step 1: The retention rate definition

Retention is the metric Diego cares most about. "How many members stick around?" Sounds simple. It is not.

A member signs up for unlimited at 48,000 CRC per month. Three months later, they downgrade to 4-class at 21,000. They are still a member. They still attend classes. But they are paying less than half what they were.

Are they retained?

If you count downgrades as retained, the retention rate is one number. If you count downgrades as churned, the retention rate is a materially different number. The gap between the two definitions can be ten or more percentage points. Both definitions are defensible. Neither is wrong. The choice changes what Diego tells his investors.

Before you compute anything, write down your definition. Plain language first: what does "retained" mean for Movimiento Fitness? Then translate that into a SQL-level rule: how do you identify a retained member in the data?

Step 2: Senior colleague

Dr. Nkechi Obi, a senior analyst available in the colleague chat, has one question for you: "How are you defining retained? Write it down before you compute it."

She is right. The definition comes before the calculation. If you compute first and define later, the definition fits whatever number you got -- and that is not governance, that is rationalization.

Write the definition. Then open materials/membership-types.md to remind yourself what the tiers are and what transitions are possible.

Step 3: Revenue per member

Revenue per member is the second key metric. It looks straightforward until you remember the quality assessment findings from Unit 2.

The Escazu location charges 30% more than the other three locations. If you calculate raw revenue per member across all locations, Escazu looks like it has the highest-value members. It does not -- it has the highest prices. The metric needs to account for the pricing difference, or the location comparison will be misleading.

You also discovered members with zero revenue in their first month -- referral promotions. Including them in the average deflates the number. Decide whether to exclude them, flag them, or segment them separately.

Write the revenue per member definition. Plain language first, then SQL.

Step 4: Class attendance

Class attendance is simpler but still requires a decision. Attendance per class (how full are classes?), attendance per member (how often do members attend?), or attendance per location (how busy is each studio?)? Diego wants all three eventually, but start with the one that answers his retention question: do members who attend more classes stay longer?

Write the definition. Plain language, then SQL.

Step 5: Automated validation tests

Each metric definition now gets a test -- an assertion that runs against the data and checks whether the definition still holds.

A retention rate test: if month-over-month retention changes by more than 15 percentage points, something needs investigation. A revenue test: no member should have negative total revenue. A referral test: members with zero revenue in their first month should follow a predictable pattern.

Direct AI to write these tests. AI commonly generates tests that check the obvious -- non-null values, positive amounts, type constraints. These are fine but insufficient. The tests that matter are business-rule tests: constraints that would produce a wrong business decision if violated.

Write validation tests for the retention rate, revenue per member, and class attendance metrics. Include business-rule tests, not just type and null checks. Reference the metric definitions we wrote.

Review what AI produces. Are the tests checking what the definitions actually promise? Or are they checking generic data properties that any dataset would pass?

Step 6: Run tests and investigate

Run the tests against the current data. Some will pass. Some may flag.

A test that flags is not an error to fix. It is information. If the referral test flags members with zero revenue beyond their first month, that could mean: (a) the referral promotion was extended, (b) there is a data error, or (c) the business policy changed. Distinguishing between these requires understanding the definition, the data, and Diego's business.

Investigate each flag. Decide what it means. Document your reasoning.

Step 7: Cross-check

Direct a second AI to review the metric definitions and the tests:

Here are three metric definitions (retention rate, revenue per member, class attendance) and the automated tests for each. Review whether the tests actually enforce what the definitions promise. Flag any test that checks the wrong thing or any definition gap the tests don't cover.

Update Diego on the retention rate finding if both definitions are computed. The difference between the two numbers makes metric governance concrete -- this is what it means to define a metric carefully.