Product Management for Engineers · Page 4
Metrics, Experiments and Goals: OEC, Guardrails, HEART, OKRs and Trust
A team needs a number it can move in two weeks that still points at what it wants in two years, a list of things it will not break to move it, and a habit of distrusting results that look too good. This page covers the overall evaluation criterion and guardrails, the HEART and Goals-Signals-Metrics frameworks, what experiment base rates imply for a portfolio of work, how OKRs and their alternatives set goals, and what changes when a feature behaves probabilistically. Afterwards you can write a goal-signal-metric chain for a feature, choose an OEC and guardrails for it, grade an OKR set, and say what evidence would justify shipping with or without an experiment.
1. The overall evaluation criterion
An experiment needs one question: did the treatment make things better? Kohavi, Tang and Xu name the quantity that answers it the overall evaluation criterion and define it by two constraints. It "must be measurable in the short term (the duration of an experiment) yet believed to causally drive long-term strategic objectives" (Ch. 1, p. 6)1. The first constraint is operational, and the book puts a number on it: an organization "should define an OEC that can be easily measured over relatively short durations (e.g., one to two weeks)" (p. 12)1. The second is a belief, not a measurement, and the book says where the effort goes. The authors' companies keep teams "focused on metrics: choosing metrics, validating metrics, and evolving metrics over time," and those teams work out "which metrics measurable in the short term drive long-term objectives, since experiments usually run over a shorter time frame" (p. 22)1.
For an engineer the structure is familiar: the OEC is a surrogate objective, and the experiment optimizes the surrogate. How far the result can be trusted depends on the correlation between surrogate and true objective, which is the one quantity a two-week experiment cannot observe (my framing of the book's definition). The book states the difficulty as "finding metrics measurable in a short period, sensitive enough to show differences, and that are predictive of long-term goals" (p. 12)1. It prefers a single metric, "possibly as a weighted combination" of objectives (p. 7)1, while allowing that large organizations may have several OECs or "several key metrics that are shared with refinements for different areas" (p. 12)1.
1.1 Candidate OECs and how each fails
The fetched sources give no recipe for choosing an OEC, only examples, and the failures teach more than the successes. Table 1 lines up every candidate they discuss.
| Candidate | Window used | What the source says | Where |
|---|---|---|---|
| Profit | short | Profit is "not a good OEC," because "short-term theatrics (e.g., raising prices) can increase short-term profit, but may hurt it in the long run"1 | Ch. 1, p. 12 |
| Customer lifetime value | long | Called "a strategically powerful OEC"; the open problem is a short-period predictor of it1 | Ch. 1, p. 12 |
| Revenue alone | short | Revenue alone "is insufficient as an OEC": it "could lead to plastering the web site with ads." Bing weighs revenue against user-experience metrics including sessions per user1 | Ch. 1, p. 5 |
| Click-through rate | short | Named as "too gameable and needing to evolve"1; ads clicked is "a short term metric that will lead to short-term gains and long-term doom" while time on site and visit frequency are "better criteria"2 | Ch. 1, p. 22; MS 2009 §6.2 |
| Active days per user | experiment duration | The book's own example: "increasing this OEC implies that users are visiting your site more often"1 | Ch. 1, p. 6 |
| Six-week return rate | 2 weeks to pick a cohort, then 6 weeks | MSN video pre-roll test: a window longer than the one to two weeks the book recommends, chosen because the business question was loyalty2 | MS 2009 §4.5 |
| Clicks on the Buy button | short | Treatment had 64% fewer users click Buy; the authors say the experimenters "should have done the additional work to measure what should have been the OEC, actual purchases"2 | MS 2009 §4.6 |
The Buy-button case is the cleanest illustration of proxy failure. The Control had more than twice as many users click Buy. The authors believe the cause is that the Treatment showed the product price and the Control did not, because "when the price is not shown there are more clicks because some users are looking for this crucial bit of information," and they note it is "entirely possible that no more users purchased the product"2. The proxy rewarded hiding information. Every row of Table 1 is a claim about causation that someone must check; none of them is a property of the metric.
1.2 A worked derivation: the OEC as an exchange rate
A composite OEC adds quantities that do not share a unit. The weights are exchange rates, and the MSN video test shows what that means in numbers. The Control played an ad before the first video; the Treatment moved the ad after the content. The site owners' question was "Would the loyalty of users increase enough in the Treatment to make up for the loss of revenue from not showing the ad up front?" The OEC was the six-week return rate of a cohort. The return rate "increased just over 2% in the Treatment, not enough to make up for the loss of ad impressions, which dropped more than 50%"2.
The paper gives the two deltas and the verdict but not the valuations. They can be backed out. At baseline, let \( a_0 \) be the pre-roll ad revenue per user, \( r_0 \) the share of the cohort that returns, and \( V \) the value of one return. Assume ad revenue is proportional to impressions and read both reported changes as relative (the paper does not say whether the 2% is relative or absolute). The Treatment loses \( 0.5\,a_0 \) per user and gains \( 0.02\,r_0 V \). It wins only if
$$ 0.02\, r_0 V \;>\; 0.5\, a_0 \quad\Longleftrightarrow\quad r_0 V \;>\; 25\, a_0 . $$So the baseline value of return visits per user must exceed 25 times the pre-roll revenue per user before a 2% loyalty gain pays for a 50% impression loss. If the 2% is instead two percentage points on the return rate, the condition becomes \( V > 25\,a_0 \): same factor, different quantity. Both reported figures are rounded ("just over 2%," "more than 50%"), so 25 is an approximate threshold. The authors' conclusion, "not enough," therefore says something about their valuation: it implied returns worth less than roughly that multiple. This is my arithmetic on the paper's two reported figures, not a figure from the paper.
The same team did the explicit version in the MSN home-page ads test. It valued clicks two ways (what the destination property assigns to a click, and what Microsoft pays search engines to win that visit back), and "the numbers were close enough to get agreement on the monetization value to use." In a 12-day test on 5% of US home-page users, click-through rate and page views per user-day each fell 0.35% (relative); the lost clicks were worth more than the added ad revenue, an estimated loss of "millions of dollars per year" had it shipped2. The experiment supplied two small deltas. The valuation carried the argument, and it was settled by agreement.
1.3 Strategy picks the OEC; experiments climb within it
The book is direct about the dependency: "Having a strategy is critical for running experiments: the strategy is what drives the choice of OEC" (p. 21). Experiments then "help hill-climb to a local optimum based on your current strategy and product," and the failure the authors have seen is "when the OEC is not properly chosen"1. Their image for a bad OEC is "experimenting to improve the food or lighting on a sinking cruise ship" (p. 22)1. Page 6 treats strategy as a contested concept; the consequence here is narrower. Whoever chooses the OEC is making a strategic decision, and an engineer handed a metric to optimize has been handed a strategy.
The one fetched case where an OEC is used as a team goal with a number is Bing's relevance team: "several hundred people tasked with improving a single OEC metric by 2% every year." The 2% "is the sum of the Treatment effects (i.e., the delta of the OEC) in all controlled experiments that shipped to users over the year, assuming they are additive," and because "the team runs thousands of experiment Treatments, and some may appear positive by chance," credit is assigned from a separate certification experiment run with a single Treatment (pp. 14-15)1. Dilution matters when counting: "a 5% improvement for 10% of your users" is 0.5% overall, if the triggered users resemble the rest (p. 14)1. By that arithmetic the 2% target is four shipped wins of 0.5% each (my division, under the book's additivity assumption). Section 10 returns to this as one of the alignment mechanisms.
2. Guardrails and the ship decision
An OEC says what the organization wants more of. A guardrail says what it will not give up to get it. The book ties the two to strategy: "Defining guardrail metrics for experiments is important for identifying what the organization is not willing to change, since a strategy also 'requires you to make tradeoffs in competing – to choose what not to do'" (Ch. 1, p. 22, quoting Porter)1. The software example it offers is crashes: "if a feature is increasing crashes for the product, the experience is considered so bad, other factors pale in comparison." The Bing headline win passed this test by construction: revenue rose 12% "without significantly hurting key user-experience metrics" (p. 3)1.
2.1 A guardrail is a constraint, not a heavy weight
The book says safety can be handled "either via high weight in the OEC, or, equivalently, using passenger safety as a guardrail metric" (p. 22)1. The two are equivalent only in a limit. Write the composite as \( \text{OEC} = \sum_i w_i m_i \) and let \( g \) be the guarded metric with floor \( g^{*} \). A heavy weight corresponds to a penalty \( \lambda \min(0,\, g - g^{*}) \) with finite \( \lambda \). That penalty is an exchange rate in the sense of Section 1.2: a large enough OEC gain still buys a small loss in \( g \). A guardrail is the case \( \lambda \to \infty \), maximize the OEC subject to \( g \ge g^{*} \), where no gain buys the loss. The practical difference is who gets to argue. With a weight, every ship review reopens the price. With a gate, the price was settled in advance and the review only checks whether the gate held. This is my formalization of the book's one-line equivalence.
The fetched chapter gives three guardrail examples (crashes, latency, sessions per user) and no taxonomy. The book's dedicated treatment is a sidebar in Chapter 6 (p. 98 in its table of contents), which was not fetched1. Not every constraint is a metric either: some of Bing's monthly ad packages were known to be negative "as a result of space constraints or legal requirements" (p. 15)1.
2.2 The anti-metric: percent of plan delivered
The opposite of an OEC is a metric that cannot fail. The authors describe the temptation: "It is often easier to generate a plan, execute against it, and declare success, with the key metric being: 'percent of plan delivered,' ignoring whether the feature has any positive impact to key metrics" (p. 12)1. A team measured this way ships on schedule and learns nothing. Its failure is a case of a general one: a proxy becomes the target. The book gives Goodhart's law, Campbell's law and the Lucas critique a section in Chapter 7 (p. 109 in the table of contents), unfetched here1; the chapter's own instance is click-through rate, "too gameable and needing to evolve" (p. 22)1. Section 9 shows the same failure in goal-setting, where key results written as outputs make a team's grade a function of delivery.
2.3 The decision table
The book does not print a ship-decision matrix. Table 2 assembles one from its scattered statements, and says which row rests on which page.
| # | OEC result | Guardrails | Reading | Basis |
|---|---|---|---|---|
| 1 | Up, as expected | All hold | Ship | Writer's synthesis |
| 2 | Up | One fails | Do not ship as built; the gate was set in advance so the price is not renegotiated | Writer's synthesis, from p. 22 |
| 3 | Flat | All hold | No measured value; the remaining inputs are costs such as "estimates of maintenance costs for the new code" (p. 12) | Quoted, p. 12 |
| 4 | Down | Any | Stop this tactic, but one failure may mean "the specific tactic being poor, not necessarily indicating that the overall strategy is bad" (p. 23) | Quoted, p. 23 |
| 5 | Far above expectation | Any | Verify before any other row (Section 5) | Writer's synthesis, from Preface and p. 3 |
Row 4 hides a second decision the table cannot make. Repeated failures can mean the strategy is wrong, and the book's answer is "However beautiful the strategy, you should occasionally look at the results" (attributed to Churchill, p. 23)1. Section 4 gives the case where Bing acted on it.
3. What has to be true before you can experiment
The book is candid that "not every decision can be made with the scientific rigor of a controlled experiment": you cannot run one on a merger, "as we cannot have both the merger/acquisition and its counterfactual" (Ch. 1, p. 10)1. It lists four technical ingredients (pp. 10-11) and three organizational tenets (p. 11)1. The ingredients are the part an engineer can check against his own product. Table 3 sets them beside a reading for a runtime that ships inside customers' applications; that last column is my reading, flagged as such, because no fetched source covers infrastructure or on-device products.
| Ingredient | What the book requires | Reading for an embedded runtime (conditional) |
|---|---|---|
| 1. Units without interference | Units "can be assigned to different variants with no interference (or little interference)"; treated users must not affect control users | The unit has to be chosen: end-user device, app, or developer account. The choice drives row 2 |
| 2. Enough units | "thousands of experimental units: the larger the number, the smaller the effects that can be detected"; small startups can start "initially looking for big effects" | If the unit is a developer account and customers number in the hundreds, the unit count fails; if it is an end-user device, the count may pass but the data may belong to the customer's app |
| 3. An agreed, loggable OEC | "Key metrics, ideally an OEC, are agreed upon and can be practically evaluated"; where goals are hard to measure, agree on surrogates | Time from install to first successful inference is loggable inside the runtime only if telemetry is permitted |
| 4. Changes are easy to make | "Server-side software is much easier to change than client-side"; new hardware devices are rarely experimented on because lead times are long | A runtime is client-side code: a variant reaches users with an app release, not a server flag, so iteration is slower than the book's examples |
When ingredients fail, the book does not say "decide by opinion." It says controlled experiments are "the most reliable and sensitive mechanism to evaluate changes" when they can be run, and otherwise recommends "other, less trustworthy, methods, including observational studies" (pp. 10-11)1. Chapter 10 on complementary techniques lists the alternatives (logs-based analysis, human evaluation, user experience research, focus groups, surveys, external data; pp. 127-135 in the table of contents), and Chapter 11 covers observational causal studies; neither was fetched1. Two cautions come with the thousands-of-units rule. The book is written by people whose companies each run "more than 20,000 controlled experiments a year" (front cover)1, so its examples are drawn from the best-instrumented products in the industry. And the second tenet, willingness "to invest in the infrastructure and tests to run controlled experiments and ensure that the results are trustworthy" (p. 11), is the cost of entry; building that platform is Part IV of the book, "aimed toward engineers" (Preface, p. xvi), and outside this page1.
4. Experiment results as a portfolio
The book's third tenet is that "the organization recognizes that it is poor at assessing the value of ideas" (Ch. 1, p. 11)1. The evidence offered is a set of success rates, and they are four different quantities. Figure 2 draws them as separate bars and does not average them. Page 1 uses the same figures for a different argument, how ideas die before and after testing; here they matter for what a team should plan on.
4.1 Four base rates, kept apart
Only the Microsoft row has a primary source in hand: the 2009 paper reports that among "well-designed and executed experiments that were designed to improve a key metric, only about one-third were successful at improving the key metric"2. The other three reach this page through the book, as a citation of Manzi, a quotation of a Slack executive's post, and a secondhand reading of Netflix. The 2009 paper adds an offline base: QualPro, a consultancy running multivariable controlled experiments, tested 150,000 business-improvement ideas over 22 years and reported that 75 percent "either have no impact on performance or actually hurt performance"2. The same paper reports that "only three out of 21 people guessed the winner" among six widget designs, and the three "were from the ExP team" (§4.1)2. What survives is the weaker claim the authors make: "Not every domain has such poor statistics, but most who have run controlled experiments in customer-facing websites and applications have experienced this humbling reality" (p. 14)1.
4.2 The payoff tail
The same chapter opens with the opposite end of the distribution. A Bing ad-headline change was "prioritized low and languished in the backlog for more than six months until a software developer decided to try the change, given how easy it was to code"; the test raised revenue 12%, "over $100M annually in the US alone" (p. 3), a return on "a few days' work for one engineer" (p. 4)1. The authors then say how rare that is: "Bing runs over 10,000 experiments a year, but simple features resulting in such a big improvement happen only once every few years" (p. 5)1. Taking "a few" as two to five years, which is my reading, that is roughly one such result in 20,000 to 50,000 experiments. Typical gains are of a different order: "improvements to key metrics are achieved by many small changes: 0.1% to 2%" (p. 14)1. Success rates of 10 to 33%, a typical gain of 0.1 to 2%, and a tail event worth $100M a year describe a heavy-tailed payoff, and in such a game the cost per draw decides the strategy (my inference). The book points the same way when it lists "the overhead of running an experiment must be small" among the themes of the Bing case (p. 5)1. What the case implies for backlogs and for how ideas are ranked is the subject of page 3.
4.3 How many experiments before a win
If each experiment independently improves its target metric with probability \( p \), the chance of at least one win in \( n \) is \( 1-(1-p)^n \). To reach confidence \( c \), solve for \( n \):
$$ n \;\ge\; \frac{\ln(1-c)}{\ln(1-p)} . $$At \( c = 0.9 \) this gives 5.68 (so 6 experiments) for \( p = 1/3 \), 10.32 (so 11) for \( p = 0.2 \), and 21.85 (so 22) for \( p = 0.1 \), the edges of the reported range. The assumptions are strong: independence, a constant \( p \), and "win" meaning the metric moved, not that the change was worth shipping. The arithmetic is mine. Its use is to turn a base rate into a capacity requirement. A team that can finish four experiments a month and works where \( p \) is near 0.1 should expect about five and a half months before a 90% chance of one win, and the calculator below lets you move all three knobs.
Experiments, and months, until a win
Formula: \( n = \lceil \ln(1-c) / \ln(1-p) \rceil \). Independence and a constant success rate are assumptions; the default 15% sits inside the reported 10-33% span but is a free parameter, not a measurement.
4.4 Near bets, radical bets, and sunk costs
The book's recommendation follows from the numbers: "always having a portfolio of ideas: most should be investments in attempting to optimize 'near' the current location, but a few radical ideas should be tried," because "most big jumps fail (e.g., big site redesigns), yet there is a risk/reward tradeoff: the rare successes may lead to large rewards that compensate for many failures" (Scenario 2, p. 23)1. Redesigns fail for a measurable reason: they "subject users to primacy effects" and "commonly fail not only to achieve their goals, but even fail to achieve parity with the old site on key metrics" (p. 21)1. The same effect biases the test, since a short experiment on a redesign measures "primacy effects or change aversion" and "may not measure the true long-term effect" (p. 23)1.
The portfolio also has an exit rule. Bing pursued a social-media integration strategy for about two years; "after spending over $25 million on the strategy with no significant impact to key metrics, the strategy was abandoned," and the authors add that "failed bets are sunk costs, and we should make a forward-looking decision based on the available data" (pp. 23-24)1. A single failed experiment tests one tactic; many failed tactics are a signal about the strategy. That is the row-4 caveat of Table 2 and the reason the OEC and the strategy must be revisited together.
5. Trusting a number: Twyman's law
The Bing headline test did not announce itself as a triumph. "A few hours after starting the test, a revenue-too-high alert triggered, indicating that something was wrong with the experiment." The authors call these "too good to be true" alerts "very useful, as they usually indicate a serious bug, such as cases where revenue was logged twice (double billing) or where only ads displayed, and the rest of the web page was broken." In this case the increase "was valid," and "the experiment was replicated multiple times over a long period" (Ch. 1, p. 3)1. The Preface turns the habit into a rule: "We believe in the skepticism implied by Twyman's Law: Any figure that looks interesting or different is usually wrong; we encourage readers to double-check results and run validity tests, especially for breakthrough positive results" (p. xv)1.
5.1 Twyman's law as Bayes' rule with a heavy-tailed prior
The rule has a mechanism, and Section 4.2 supplied its input. Let \( q \) be the prior probability that an experiment contains a genuine effect as large as the one observed, and \( b \) the probability that it contains a defect (double logging, a broken page) that fakes a jump of the same size. If either cause produces the jump with certainty, then
$$ P(\text{real} \mid \text{jump}) \;=\; \frac{q}{q+b}. $$From the book's figures, Bing runs 10,000 or more experiments a year and a result of the $100M kind comes "once every few years" (p. 5)1, so with \( y \) years between such wins \( q = 1/(10{,}000\,y) \). At \( y = 3 \) that is 3.3 per 100,000 experiments. The odds turn even when \( b = q \): a defect that fakes a jump of that size in one experiment out of every 30,000 is enough. Nothing in the source gives \( b \); it is the free knob in Figure 4. The structural point does not depend on its value. The larger the claimed effect, the smaller \( q \), because the payoff tail is heavy, while defects do not become rarer as they get bigger (a metric logged twice doubles regardless). That argument is mine, not the book's. Skepticism should therefore grow with the size of the surprise, which is what "especially for breakthrough positive results" asks for. The book's definition of an interesting result points the same way: "Interesting experiments are ones where the absolute difference between the expected outcome and the actual result is large" (p. 16)1. The results most worth learning from are the ones most likely to be wrong.
Is the jump real?
Formula: \( q/(q+b) \). The defaults 10,000 and 3 are the book's order of magnitude and a reading of "every few years"; the faked-jump rate has no source and starts at an arbitrary 5.
5.2 Causation, and the ladder of evidence
The second trust problem is older than experiments. The book's example is Microsoft Office 365: users who see error messages and crashes churn less, "but that does not mean that Office 365 should show more error messages." All three facts have one cause, usage: "heavy users of the product see more error messages, experience more crashes, and have lower churn rates" (pp. 8-9)1. The authors rank evidence on a ladder adapted from medicine (Figure 1.3, p. 9): meta-analyses of randomized experiments at the top, then randomized controlled experiments, then other controlled experiments, then observational studies, and at the bottom "case studies ..., anecdotes, and personal (often expert) opinion, a.k.a. HiPPO"1. Section 8 uses this ladder to say what evidence a ship decision needs. The rest of the book's trust machinery, including Chapter 3 on threats to validity (p. 39) and Chapter 21 on sample ratio mismatch (p. 219), sits in chapters not fetched1.
6. PULSE and HEART
An experiment platform tells you whether a treatment moved a metric. It does not tell you which metrics deserve to exist. Google's UX research team addressed that gap in a 2010 CHI note by Rodden, Hutchinson and Fu3. They start from what teams already track, which they call PULSE: "Page views, Uptime, Latency, Seven-day active users (i.e. the number of unique users who used the product at least once in the last week), and Earnings." These are "all extremely important," and an engineer will recognize them as what an operations dashboard shows. The authors' complaint is that they are "either very low-level or indirect metrics of user experience, making them problematic when used to evaluate the impact of user interface changes," and may be ambiguous: a rise in page views may occur "because the feature is genuinely popular, or because a confusing interface leads users to get lost in it, clicking around to figure out how to escape" (section "Pulse Metrics")3.
6.1 HEART is a set of categories, not a set of metrics
HEART stands for "Happiness, Engagement, Adoption, Retention, and Task success." The authors stress that "these are categories, from which teams can then define the specific metrics that they will use," that "it is not always appropriate to employ metrics from every category," and that the framework's value is to make "an explicit decision about including or excluding a particular category." Their example of exclusion is Engagement in an enterprise product where use is required, with the note that "it may still be meaningful to consider Engagement at a feature level"3. Happiness and Task success generalize older usability measures; Engagement, Adoption and Retention are "new categories, made possible by large-scale behavioral data"3.
| Category | Kind of data | Paper's definition | Paper's example |
|---|---|---|---|
| Happiness | attitudinal | Subjective aspects: satisfaction, visual appeal, likelihood to recommend, perceived ease of use | iGoogle weekly in-product survey, 7-point bipolar scale; satisfaction fell after a major redesign, then recovered, "indicating that change aversion was probably the cause" |
| Engagement | behavioral | Behavioral proxies such as "the frequency, intensity, or depth of interaction over some time period"; report as an average per user, not a total | Gmail: "the percentage of active users who visited the product on five or more days during the last week" |
| Adoption | behavioral | How many new users start using the product in a period | Accounts created in the last seven days |
| Retention | behavioral | How many users from one period are still present in a later one | Share of a week's seven-day actives still active three months later |
| Task success | behavioral | Efficiency (time to complete), effectiveness (percent completed), error rate | Google Maps single vs dual search box, A/B test compared on error rates |
The paper supports two further uses. In September 2008 Google Finance saw a surge in page views and seven-day actives, and those totals "did not indicate whether the surge was driven by new users interested in the crisis, or existing users panic-checking their investments." Adoption and Retention metrics separated the two groups and showed the rate at which new users kept using the site3. And the Gmail metric was chosen with an explicit reason, "engaged users should check their email account regularly, as part of their daily routine," and the authors found it "strongly predictive of longer-term retention, and therefore could be used as a bellwether for that metric"3. That sentence is the only place in the note where the short-term-to-long-term link that an OEC depends on (Section 1) is reported as tested, and it gives no numbers.
6.2 Why totals mislead: the seven-day-active trap
The note makes a precise claim about its own PULSE metric: seven-day actives "measures the overall volume of the user base, but gives no insight into the users' level of commitment," does not "differentiate between new users and returning users," and "in a worst-case retention scenario of 100% turnover in the user base from week to week, the count of seven-day active users could still increase, in theory"3. The reason is arithmetic. If \( N_t \) new users arrive in week \( t \) and none returns, then \( \mathrm{7DA}_t = N_t \), which rises whenever acquisition rises. Figure 5 draws that case with an illustrative 10% weekly growth in new users. The cure the paper gives is normalization: "raw counts will go up as your user base grows, and need to be normalized; ratios, percentages, or averages per user are often more useful"3.
Three limits are worth stating plainly. The framework was "applied to more than 20 different products and projects from a wide variety of areas within Google," a company whose products are "almost all web-based, and have millions of users each." The note reports no evidence that using HEART improved a product's outcomes; its examples show decisions that metrics informed3. The metrics "are primarily useful for evaluation of launched products, and are not a substitute for early or formative user research," and "should be triangulated with findings from other sources, such as usability studies and field studies"3. And no HEART category has been checked for a product whose users are developers: whether an SDK's "Adoption" is a signup, a first successful call, or a production release is a definition the team has to make, which is the work the next section structures.
7. Goals, signals, metrics
HEART names categories; a second piece of the same paper says how to get from a category to a number. The process has three steps. In the paper's words it "steps teams through articulating the goals of a product or feature, then identifying signals that indicate success, and finally building specific metrics to track on a dashboard," because a metric "is unlikely to be useful in practice unless it explicitly relates to a goal"3. Each step has tips from Google's experience, and they read as engineering advice.
- Goals. "Different team members may disagree about what the project goals are. This process provides a great opportunity to collect all the different ideas and work towards consensus (and buy-in for the chosen metrics)." Goals for a feature "may be different from those for the product as a whole." Do not "get too distracted at this stage by worrying about whether or how it will be possible to find relevant signals or metrics"3.
- Signals. Ask what behavior or attitude would show the goal met. Choose signals "sensitive and specific to the goal," which "should move only when the user experience is better or worse, not for other, unrelated reasons." "Sometimes failure is easier to identify than success," such as task abandonment or "undo" events. Logs and surveys are the two sources used most often, and the check to make is whether "the relevant actions" are "currently being logged, or could they be?"3.
- Metrics. Normalize raw counts. Filter "traffic from automated sources (e.g. crawlers, spammers)." Confirm that "all of the important user actions are being logged (which may not happen by default, especially in the case of AJAX or Flash-based applications)"3.
The goal step carries more weight than its length suggests. If teammates "disagree about what the project goals are," the metrics that follow will each serve a different goal and the dashboard will not settle any argument. That is why the paper treats the process as a consensus tool before it is a measurement tool.
A hypothetical chain shows the template on a developer-facing feature. It is not from any source, and no source covers infrastructure products. Goal: a developer integrates an on-device runtime and gets a first successful inference without help. Signals: an install followed by a successful inference call; no support contact before it; an install that never reaches one, since "failure is easier to identify than success." Metrics: the share of each week's new installs that reach a first successful inference within a stated window, and the median time to it, both per install cohort rather than totals. By analogy with the paper's enterprise note, where Engagement may still be meaningful "at a feature level"3, the engagement category would belong at the feature level for such a product; that extension is mine. Latency, memory and crash rate are not in the chain; they are the guardrails of Section 2.
8. How much evidence before acting
Sections 1 to 5 describe an instrument. Whether to wait for its reading is a separate decision, and two fetched sources answer it in opposite tones. Kohavi et al. say a "data-driven or a data-informed organization gathers relevant data to drive a decision and inform the HiPPO ... rather than relying on intuition," and that "a decision should be made with many sources of data, including controlled experiments, surveys, estimates of maintenance costs for the new code" (Ch. 1, p. 12)1. Bezos, in Amazon's 2016 shareholder letter, writes that "most decisions should probably be made with somewhere around 70% of the information you wish you had. If you wait for 90%, in most cases, you're probably being slow," and that "being wrong may be less costly than you think, whereas being slow is going to be expensive for sure"4.
8.1 The decision as an inequality
The two positions are answers to the same cost question, which can be written down. Let \( p_f \) be the probability that the idea does not help (the base rates in Section 4 put it at roughly 0.67 to 0.9), \( C_w \) the cost of shipping a dud until it is found and reversed, \( C_e \) the cost of running the experiment, \( t \) the time it takes, and \( D \) the value a good idea loses per unit of delay. Assume the test is reliable. Shipping now is worth \( (1-p_f)V - p_f C_w \); testing first is worth \( (1-p_f)(V - Dt) - C_e \). Testing wins when
$$ p_f\, C_w \;>\; C_e + (1-p_f)\, D\, t \qquad\Longleftrightarrow\qquad \frac{C_w}{D\,t} \;>\; \frac{1-p_f}{p_f} + \frac{C_e}{p_f\, D\, t}. $$If the platform makes the experiment nearly free (\( C_e \to 0 \)), the right-hand side is \( (1-p_f)/p_f \): 0.5 at \( p_f = 2/3 \), 0.25 at 0.8, 0.11 at 0.9. A test is worth waiting for whenever a dud costs more than about a tenth to a half of what the delay costs a good idea. That is a low bar, and it explains why the book insists that "the overhead of running an experiment must be small" (p. 5)1: cutting \( C_e \) and \( t \) is what moves the decision. A two-way door is the case where \( C_w \) is small because reversal is cheap, and there the inequality flips toward shipping at 70% and watching, which is Bezos's rule. A one-way door makes \( C_w \) large and the rule inverts. This is my derivation. It omits imperfect tests, statistical power, and the information a decision generates for later ones (the book names the Expected Value of Information, p. 241).
8.2 What each source warns against
The warnings differ. Kohavi et al. fear the plan that cannot fail ("percent of plan delivered," Section 2.2) and note that "objective measurements typically show that progress is not as rosy as initially envisioned" (p. 12)1. They also record the exception that proves nothing: Apple under Steve Jobs, where Ken Segall "claimed that 'we didn't test a single ad. Not for print, TV, billboards, the web, retail, or anything'" (pp. 11-12)1. That is one sentence about advertising, reported secondhand, not a study of product decisions. Bezos fears stand-ins for the real thing and names two: process ("the process becomes the proxy for the result you want") and averages ("market research and customer surveys can become proxies for customers"). His example is "Fifty-five percent of beta testers report being satisfied with this feature. That is up from 47% in the first survey," which he calls "hard to interpret." His prescription is not to skip testing: "I'm not against beta testing or surveys. But you, the product or service owner, must understand the customer ... Then, beta testing and research can help you find your blind spots"4. Rodden et al. say metrics "should be triangulated" with qualitative findings and are no "substitute for early or formative user research"3.
| Situation | Cost of a dud | Evidence that justifies shipping | Premise from |
|---|---|---|---|
| Reversible, experiment available | low | A stated hypothesis, an OEC and guardrails; ship as an experiment and read the result (Table 2) | Kohavi p. 5; Bezos "two-way doors" |
| Reversible, experiment impossible (too few units, slow client releases) | low to medium | Decide near 70% of the information, then watch logs and use complementary techniques; say in advance what would trigger a revert | Kohavi pp. 10-11; Bezos on correcting quickly |
| Hard to reverse (a published interface, a persisted format; hypothetical examples) | high | More than 70%: prototype with representative users, then decide; the inequality in 8.1 inverts | Writer's derivation; Bezos on "two-way doors" |
| Not testable at all (a merger, a strategy) | varies | Judgment informed by "many sources of data," with the decision owner's reasoning written down | Kohavi pp. 10, 12 |
| Result far above expectation | high if false | Replicate and audit the logging before acting (Section 5) | Kohavi p. 3, Preface |
None of these sources has outcome data comparing evidence-heavy and judgment-heavy decision styles; Section "Where experts disagree" returns to that.
9. OKR mechanics and what they optimize
An OEC is a single scalar that a team moves by experiment. An OKR set is a different kind of object: a handful of ambitious statements, each with measurable key results, graded after a fixed period. Google's re:Work guide, "originally published 2016" and preserved "as a record," is the primary description5. Marty Cagan's 2015 essay on roadmaps names OKRs "my current favorite" and supplies the product-side rationale6. Table 6 lists the parameters.
| Parameter | What the guide says |
|---|---|
| Count | "three to five objectives with about three key results for each objective"; more "can lead to over-extended teams and a diffusion of effort" |
| Objective | Objectives "are ambitious and may feel somewhat uncomfortable" and use "tangible, objective, and unambiguous terms," so an observer can tell whether it was achieved |
| Key result | "Measurable and should be easy to grade with a number"; they "should describe outcomes, not activities" |
| Grade | 0.0 to 1.0; each key result is graded and "using a rough average" the objective is graded; sometimes weighted |
| Target | "Sweet spot" 60 to 70%; "the expectation is to get an average of 0.6 to 0.7 across all OKRs"; success "means achieving 70% of the objectives, while fully reaching them is considered extraordinary performance" |
| Cadence | Annual and quarterly; company-wide meetings quarterly to share and grade; an optional mid-quarter check-in |
| Visibility and use | Public; "not synonymous with employee evaluations"; "not a shared to-do list"; low grades are "data to help refine the next OKRs" |
9.1 The grading scale is a calibration target
Most goal systems treat a miss as a failure. This one budgets for it: if a team's average is 0.6 to 0.7 the system is working, and "if someone consistently fully attains their objectives, their OKRs aren't ambitious enough"5. A consequence the guide spells out is for dependents. "If they are using stretch goals you should expect them to deliver on about 70% of their stated OKRs," so a team that depends on another team's stretch objective should plan on 70%5. An OKR grade is therefore an estimate with a built-in haircut, not a commitment. Cagan separates the two: dates are handled by "high-integrity commitments," used "for those situations where we need to actually commit to a date or a specific deliverable"6, and page 3 covers their cost.
The figure makes a limit visible. Because the objective grade is a mean, a team that fully misses one key result and fully makes the others lands in the same place as a team that was partly short on all three. The guide's own grading examples show how binary scoring enters: a key result "to 'Launch a new widget marketing campaign'" is 0 or 1, while "Launch six new features" with three launched is "graded with a 0.5"5.
9.2 Failure modes the guide lists, and one it contains
The guide's own list of mistakes is a catalog of ways a number stops measuring what it was written for. Business-as-usual OKRs are "written based on what the team believes it can achieve without changing anything they're currently doing." Sandbagging: "teams who can meet all of their OKRs without needing all of their team's bandwidth may either be hoarding resources, not pushing their teams, or both." Low-value objectives: "could the OKR get a 1.0 under reasonable circumstances without providing direct organizational benefit?" Insufficient key results: if they "don't represent all that is needed to fully achieve that objective," a miss arrives late and unexpected5.
The guide also contains the failure it warns against. Its text says key results "should describe outcomes, not activities," yet five of the seven key results in its two sample sets name a deliverable or an activity and no measured change, among them "Launch xx feature to all users," "Secure tech support to build xx feature in Q1," and "Re-establish [product]'s leadership by speaking at three industry events"; only the revenue-per-user and error-response-time results carry a measured change5. That is the pattern Kohavi et al. call "percent of plan delivered" (Section 2.2)1. Cagan's version of the principle is that the system "is all about outcome rather than output," illustrated by an objective to cut onboarding from 30 days to under 3 hours; he introduces the figures with "suppose your product currently requires 30 days," so they are an illustration, not data6. The guide's evidence for OKRs is one sentence: Sears Holding Company "rolled out OKRs to their 20,000 employees and saw positive impacts on bottom-line sales and individual performance," with no figures5.
| # | Test | Fails when | Passes when (hypothetical) |
|---|---|---|---|
| 1 | Outcome, not output | "Ship the quantized backend" | "Median time from install to first inference falls from \( T_0 \) to \( T_1 \)" |
| 2 | Gradable by a number, without argument | "Improve developer experience" | The grade follows from a logged quantity |
| 3 | Not business as usual or sandbagged | The team could score 1.0 while changing nothing | Stack-ranked against current work, it is a top effort |
| 4 | Key results suffice for the objective | All three can score 1.0 and the objective still fails | The key results together imply the objective |
| 5 | Three to five objectives, about three key results each | Eight objectives, or one key result | Within the stated counts |
| 6 | Hard to game | A cheap action moves the number and no user would want it | A guardrail covers the cheapest bad path |
| 7 | Honest about its use | The grade also decides pay, with no statement of how | The use of the grade is stated in advance |
Test 7 is where sources diverge. The re:Work guide says OKRs are "not synonymous with employee evaluations"5; Section 10 shows a company that ties its goals to pay review and sets a different standard for what a miss means.
10. Alignment alternatives and where each fails
An organization has to say what its teams are for. Five fetched sources do it in five ways, and Table 8 sets them side by side.
| Mechanism | Unit of commitment | Measure | Who sets it | Failure mode |
|---|---|---|---|---|
| OKRs (Google re:Work) | Objective with about 3 key results, graded 0.0-1.0 each quarter | Key-result grade; 0.6-0.7 is the target | Top-down and bottom-up mix; team leaders set team OKRs | Sandbagging, business-as-usual, low-value objectives; sample key results that are outputs5 |
| OEC with guardrails (Kohavi et al.) | A delta in one composite metric, certified by a replication experiment | OEC change in an experiment, gated by guardrails | Strategy chooses the OEC; metrics teams validate it | A badly chosen OEC ("experimenting to improve the food or lighting on a sinking cruise ship"); needs experiment scale1 |
| Initiatives and a product timeline (Linear) | A named stream of work; projects with target dates | "Measure progress with actual work": the diff, the design file | Not stated on the fetched pages | Not stated; the measure is progress on work, not effect (writer's reading)9 |
| Appetite-bounded bets (Shape Up) | One shaped project for one six-week cycle | None named; "a specific payout" is defined in the pitch | The betting table: CEO, CTO, a senior programmer, a product strategist | A failed bet is read as a shaping error: "we did something wrong in the shaping"10 |
| Output goals plus a monthly growth review (PostHog) | A list of things to build, each with one named owner | "Use metrics only if they help you"; a PM-run monthly review of revenue, usage and feedback data | Executives give high-level goals; engineers choose the work | Autonomy "can become directionless" without the review78 |
Two warnings on reading the table. First, absence of a mechanism on a fetched page is silence, not opposition. The Shape Up chapters on bets and the betting table (7 and 8) contain none of the words "metric," "OKR," "goal" or "measure," which I confirmed by searching the pages10; Linear's practices and direction pages mention neither metrics nor OKRs9. Second, every row is a company describing its own practice, and none reports outcome data.
10.1 PostHog: why a company dropped metric goals
PostHog's James Hawkins and a colleague wrote in December 2024 that "we used to do OKRs and metric-based goals, but we ran into two problems with it": "teams wasted too much time coming up with the right metric to focus on," and "engineers focused on small quick wins to move those metrics, rather than building what our users actually wanted"7. Their replacement is a quarterly list of things to build and who owns them, chosen by engineers after executives share high-level company goals (for example, "we need to increase top of funnel adoption")7. The accountability moves to a monthly growth review run by the product manager, which compiles "revenue metrics" (MRR, churn), "product analytics" (active users, retention) and "user feedback" (NPS, interviews, tickets) and asks, for example, "Can we find leading indicators that predict long-term product usage?" The engineering team lead then decides whether to continue, reprioritize or change goals. The stated reason: "Without this accountability, autonomy can become directionless"7. The handbook's current planning page keeps the shape. Goals are "primarily output-based," with "metrics only if they help you," and the stated worry about metrics is that "all objectives are bad - they have many compromises, are fallible, easy to game"8. The same page also asks for "what we'll ship," a rationale, and "we'll know we're successful when: metric or outcome," and says "hitting an overarching objective is more important than shipping specific things"8. The handbook page carries no date and is a living document.
One difference from Google deserves a line. The re:Work guide treats a 0.6 to 0.7 average as the system working5. PostHog's page says "if your team is consistently missing objectives, they are too hard or possibly the wrong objectives," and "consistently hitting ambitious objectives over the long term is an important factor in the pay review process"8. The first uses misses as a calibration signal and keeps goals out of evaluation; the second treats misses as a goal-setting error and ties hits to pay. The phrase "goal" covers two different contracts.
PostHog's loop reads metrics after the fact instead of setting them as targets. That avoids the gaming problem of Section 2.2 at the price of the one Kohavi et al. describe, a plan that can be executed and declared a success. What separates PostHog from that failure is the review, not the goals, and the review is only as good as the PM's data and the willingness of the CEO or a senior leader to "challenge assumptions, ask hard questions, and ultimately hold the team accountable"7.
10.2 Linear and Shape Up: progress as work, bets as commitments
Linear's method asks teams to "measure progress with actual work," and says "the clearest way to see whether something is complete or not is to show the diff in the code or design file"9. Its direction page calls initiatives "a critical tool for shaping this direction" and wants a timeline "ideally a bit out of reach"9. Its "set useful goals" page asks for "a goal that propels you forward in some measurable way," with a startup example that runs from finding 10 users, to 100, to $1000 in monthly recurring revenue9. Linear's own account of how it ships says readiness "comes down to the engineers working on the project feeling that the project is ready," and the fetched post has no passage on judging a project's effect after release11. Shape Up's mechanism is the cap: "a smart bet has a cap on the downside. If we bet six weeks on something, the most we can lose is six weeks," and the betting table looks "at the bigger movement of progress on the whole product after the six weeks"10. The cap bounds the loss from a wrong bet but does not say how a team learns it was wrong. Neither source gives one.
11. Metrics for features that behave probabilistically
Everything above assumes a feature whose behavior the team can read off the code. A feature built on a model does not satisfy that. Amershi and colleagues at Microsoft Research open their CHI 2019 paper with the cause: "automated inferences are typically performed under uncertainty, often producing false positives and false negatives," so AI-infused systems "can violate established usability guidelines of traditional user interface design." Their example is consistency: "many AI components are inherently inconsistent due to poorly understood, probabilistic behaviors based on nuances of tasks and settings, and because they change via learning over time" (§1)12. Two properties follow for measurement. The error rate is part of the product, and the behavior can change between two visits by the same user.
11.1 The 18 guidelines
The paper consolidates "more than 150 design recommendations" into 18 guidelines, with a user study of 49 practitioners testing them on 20 AI-infused products (abstract, §3)12. Each is "a rule of action, containing about 3-10 words and starting with a verb," grouped by when in an interaction it applies (§3, Table 1). Figure 8 draws them in the paper's four phases. I have marked the four that govern a system that changes over time; the marking is mine.
The three that matter most for a metrics decision are G2, G11 and G14. G2: "Make clear how well the system can do what it can do. Help the user understand how often the AI system may make mistakes." G11: "Make clear why the system did what it did." G14: "Update and adapt cautiously," glossed as "limit disruptive changes when updating and adapting the AI system's behaviors"12. The paper reports 785 examples across the 20 products: 313 applications, 277 violations, 89 neutral and 106 judged "does not apply." G11 "had one of the highest number of violations," with among the fewest "does not apply" ratings, which the authors read as participants who could imagine an explanation but were often unable to obtain one. G5 and G6 were "noted as the least clear." G10 and G14 were hard to observe: "it's a bit difficult to assess this in a single session," one participant wrote about G1412. This study measured whether practitioners could apply and understand the guidelines. It did not measure retention, revenue or task success, so no sentence in it says that following a guideline raises any business metric.
G14 connects to the experiment machinery. The book warns that short tests of big changes measure "primacy effects or change aversion" (p. 23)1, and Google saw iGoogle satisfaction fall after a redesign and recover3. A model that adapts changes the product continuously, and the effects of adaptation on a user may accumulate more slowly than an OEC window of one to two weeks can show. G14 is therefore a design constraint to hold even when the OEC does not move.
11.2 Offline gain is not business gain
A team building a model has a second metric the web-experiment literature does not discuss, the offline score on held-out data. The fetched evidence on how it relates to business impact is one paper: Bernardi, Mavridis and Estevez at Booking.com, KDD 2019, an analysis of "about 150 successful customer facing applications of Machine Learning" validated through randomized controlled trials13. Its Section 3 reports that "increasing the performance of a model, does not necessarily translates to a gain in value." The data are 23 comparisons (46 models) between a baseline and a new model, classifiers scored by ROC AUC and rankers by mean reciprocal rank, each compared in an online trial against one business metric. The correlation between relative offline gain and relative business gain is -0.1 (Pearson, 90% confidence interval -0.45 to 0.27) and -0.18 (Spearman, interval -0.5 to 0.19)13.
The authors hedge the result themselves. They write that they "do not want to overstate the generality," because the models work in one context, "all target the same business metric," and each is "trying to improve it after a previous model already did"; they note a similar weak link for machine-translation scores and human judgment, and say a correlation shows up "only where the offline metric is almost exactly the business metric"13. They list four mechanisms. Value-performance saturation: the value-versus-performance curve flattens, and gains become "impossible to detect in an RCT in reasonable time." Segment saturation: triggered analysis counts only users for whom two models disagree, and as models converge that population shrinks, cutting statistical power. Uncanny valley: users find a model that predicts too well unsettling. Proxy over-optimization: a recommender trained on click-through might "end up 'just driving clicks'," recommending near-identical hotels, which risked "drowning them into the paradox of choice and hurting conversion"13. The last is Section 1's Buy-button failure with a model as the optimizer: a better optimizer finds the gap between proxy and goal faster. Their conclusion is operational: "Offline model performance metrics are only a health check."
11.3 Latency has a measured price, in web products
A model's cost is also a product metric. The same Booking.com paper ran a four-arm experiment with synthetic latency and found "an increase of about 30% in latency costs more than 0.5% in conversion rate"13. Its design for separating a model's benefit from its slowdown is a three-group experiment. Control sees no change. A first treatment calls the model and shows its output to the users it triggers. A second calls the model and shows nothing different. Comparing the second with control prices the slowdown alone; comparing the two treatments, which carry the same computational load, prices the functionality alone (§7.2-7.3)13. Table 9 collects the latency prices in the fetched sources.
| Company, year | Latency change | Reported effect | Source |
|---|---|---|---|
| Bing, 2012 experiment | 10 ms faster | Pays for the fully loaded annual cost of an engineer | Kohavi et al. p. 18 |
| Bing, 2015 follow-up (95th percentile under 1 s) | 4 ms faster | Funds an engineer for a year | Kohavi et al. p. 18 |
| Amazon (Linden 2006) | 100 ms slower | Sales down 1% | Kohavi et al. p. 18 |
| Booking.com, 2019 | about 30% slower | Conversion down by more than 0.5% | Bernardi et al. §5 |
These are the closest fetched evidence for an inference engineer's usual claim that latency is a business metric. They are all web or search products, measured in an experiment with user-visible latency, and the Bing and Amazon rows are second-order citations in the book. Whether a per-token latency in a text-generating or on-device feature has a comparable price is outside the fetched sources.
11.4 What the sources do not cover
Offline-versus-online evaluation is a stated gap. The one candidate source, the Booking.com paper, covers classifiers and rankers at one company in 2019. The fetched material says nothing on evaluating generative models, on offline evaluation sets as a specification, or on guardrails for output quality. The guidelines paper itself says its list "might not adequately address all types of AI-infused systems," and that it leaves out principles that "require decisions to be made at the modeling layer of a system" (§7)12. A team shipping a model-backed feature has the OEC and guardrail structure of Sections 1 and 2, the base rates of Section 4 and the cautions above. It does not have a sourced method for choosing between an offline score and an experiment.
Where experts disagree
Three disagreements run through the fetched sources: how much evidence to wait for, whether goals are set as measurable targets or reviewed as work, and how fast a product should change under its users. None of the positions below is backed by outcome data comparing it with the others; each is a company or author describing its own practice, so the comparison is of mechanisms and costs.
Evidence or judgment: how much data before acting?
Be data-informed; formalize the OEC
Kohavi, Tang and Xu, Trustworthy Online Controlled Experiments (2020)1
The organization "is poor at assessing the value of ideas," so run controlled experiments and let data "inform the HiPPO."
- Base rates of 10 to 33% success (Figure 2) show intuition fails often.
- A marketing senior vice-president's objection to cart recommendations at Amazon lost to an experiment (p. 17).
- Needs units, an agreed OEC and cheap changes (Table 3); many decisions cannot be tested.
- The authors' companies are the best-instrumented in the industry.
Decide near 70%; resist proxies
Jeff Bezos, 2016 shareholder letter4
Wait for 90% and "you're probably being slow." Averages from surveys and betas "can become proxies for customers"; the owner "must understand the customer."
- Prices delay, which a test-everything rule ignores (Section 8.1).
- Names a real failure: a satisfaction average that moves without anyone knowing why.
- Apple under Jobs did not test ads, reported secondhand in the book1.
- No evidence offered that deciding at 70% beats waiting; the letter's one extended example is a disagree-and-commit story about a TV show.
Tie metrics to goals; triangulate
Rodden, Hutchinson and Fu, CHI 20103
Metrics "should be triangulated" with usability and field studies and are "not a substitute for early or formative user research."
- Fits the two others: numbers evaluate, qualitative work generates.
- Google-internal examples; no outcome data.
The positions answer different cost regimes. Section 8.1's inequality says to test when a dud is expensive relative to a good idea's delay cost, and base rates make duds the common case, so for reversible changes in a product with cheap experiments Kohavi's rule wins. For a one-way door, a product with too few units, or an invention where no baseline exists, Bezos's 70% rule and Rodden's triangulation are what remains. The one genuine collision is the status of surveys: Kohavi lists them as one source among several; Bezos warns that an average can replace understanding. A study that assigned comparable decisions to evidence-heavy and judgment-heavy styles and tracked outcomes would settle it; none appears in the fetched sources.
Aligning teams to goals: targets, or bounded work and review?
Commit to measurable targets
Google re:Work (OKRs)5; Cagan6; Kohavi et al. (OEC)1
Teams are told the result and its measure, and choose the method: "tell the team what you need them to accomplish, and how the results will be measured." The OEC is the single-metric version, certified by replication.
- Moves argument from features to results; "percent of plan delivered" cannot pass.
- The grading scale budgets for misses.
- Google's own sample key results are mostly outputs (Section 9).
- PostHog reports that teams "wasted too much time" choosing metrics and chased "small quick wins"7.
Commit to bounded work, then review
Shape Up10; Linear9; PostHog78
Fix the time or the list of things to build, cap the downside, and judge the effect afterward: a six-week cap, "measure progress with actual work," a monthly growth review.
- Avoids metric gaming and metric debates before the work starts.
- Engineers choose the work.
- "Autonomy can become directionless" without the review7.
- Shape Up and Linear name no method for learning that a shipped bet was wrong.
The axes are not exclusive. PostHog's own handbook asks for "what we'll ship" and a success measure in the same template8, and Kohavi's Bing relevance team commits to an OEC delta yet works through shipped experiments. A target works when a measurable proxy is trusted and gaming is covered by a guardrail; bounded work works when the team is small, senior and has a reviewer who reads the data. For an inference-runtime team, a small one by assumption, I would take bounded work with a stated outcome and one or two guardrails. That is my judgment, not a finding. A comparison of teams under each regime on shipped-effect per quarter would settle it; the sources have none.
Speed or trust: how fast should a product change under its users?
Change quickly, because most changes fail cheaply
Kohavi et al.1
Bing runs over 10,000 experiments a year; most big redesigns fail, and continuous iteration is preferred to complete redesigns that "subject users to primacy effects."
- Cheap tests find the rare large win.
- Their own caution: short tests of big changes confound "change aversion."
Change cautiously, because users bear the cost
Amershi et al. (G14)12; Linear11; Cagan14; Rodden et al.3
"Update and adapt cautiously"; "if you release something and iterate too much, it can be difficult for your customers"; customers can feel "abused because of constant, erratic change."
- Names a cost the OEC window can miss; satisfaction dipped after an iGoogle redesign before recovering.
- No measured size for the trust cost in any fetched source.
Cagan resolves most of it: rapid test-and-learn runs "on select groups of users and customers, enabling us to protect our general users and customers from this experimentation," and some customers opt in14. That is a statement about who is exposed, not how fast the team learns. A runtime whose behavior changes between releases has a sharper form: a change shipped to everyone cannot be a two-way door in the sense of Section 8, because users' integrations depend on it (my reading). Longitudinal data on retention after changes to an adaptive feature would settle the size of the trust cost; the sources have none.
What is changing
Metric definitions are being rewritten for traffic that is not human. PostHog's handbook lists as a Q2 2026 product-management theme "make sure our products & tracking are set up for a agent-first world," including "tracking human vs agent actions, and defining success for each," and, separately, ensuring "metrics definitions are mature enough for our scale (e.g. inconsistencies in activation & product intent)"15. A developer product whose callers are agents inherits the same problem, in my reading: a seven-day-active count (Section 6.2) stops meaning a person came back. The handbook is a living page and likely to change within a year. The fetched evidence on offline-versus-online evaluation is a 2019 paper about classifiers and rankers13, and the re:Work OKR guide is an archived 2016 record5. Nothing in the source map tests either claim for generative-model features.
References
- textbookKohavi, R., Tang, D. and Xu, Y. Trustworthy Online Controlled Experiments: A Practical Guide to A/B Testing. Cambridge University Press, 2020 (DOI 10.1017/9781108653985). Front matter, Preface and Chapter 1, pp. i-xviii and 3-24, from the authors' companion-site PDF: https://experimentguide.com/wp-content/uploads/TrustworthyOnlineControlledExperiments_PracticalGuideToABTesting_Chapter1.pdf. Accessed 2026-10-26. Used for: OEC definition and the one-to-two-week window (pp. 6, 12), customer lifetime value and profit (p. 12), guardrails (p. 22), tenets and necessary ingredients (pp. 10-13), success rates (p. 13), Bing headline case and base rates (pp. 3-5), strategy and the portfolio (pp. 20-24), performance experiments (pp. 17-18), Twyman's law (Preface, p. xv). Later chapters are cited only through the table of contents.
- paperKohavi, R., Crook, T. and Longbotham, R. Online Experimentation at Microsoft. Third Workshop on Data Mining Case Studies and Practice Prize, 2009. https://exp-platform.com/Documents/ExP_DMCaseStudies.pdf. Accessed 2026-10-26. Used for: the one-third success rate (§5), QualPro 75% figure (§5), MSN pre-roll and home-page ads OECs (§§4.2, 4.5), Buy-button case (§4.6), necessary ingredients and OEC as long-term value (§6.2), widget guessing contest (§4.1).
- paperRodden, K., Hutchinson, H. and Fu, X. Measuring the User Experience on a Large Scale: User-Centered Metrics for Web Applications. CHI 2010 (ACM author's version, Google Research). https://research.google.com/pubs/archive/36299.pdf. Accessed 2026-10-26. Used for: PULSE and HEART definitions, the seven-day-active trap, normalization, the Gmail, iGoogle, Maps and Finance examples, the Goals-Signals-Metrics tips, scope of 20+ products and the triangulation caveat.
- officialBezos, J. 2016 Letter to Shareholders. Amazon (aboutamazon.com copy; the page carries no date). https://www.aboutamazon.com/news/company-news/2016-letter-to-shareholders. Accessed 2026-10-26. Used for: the 70% information rule, two-way doors, disagree and commit, and the "Resist Proxies" passage on process, surveys and beta averages.
- officialGoogle re:Work. Guide: Set goals with OKRs. "Originally published 2016," preserved "as a record of our findings." https://rework.withgoogle.com/intl/en/guides/set-goals-with-okrs/. Accessed 2026-10-26 (page text read directly). Used for: Table 6, the 0.6-0.7 grading band, the stretch-goal and dependency guidance, the OKR-writing mistakes list, the sample OKRs, the Sears sentence.
- blogCagan, M. The Alternative to Roadmaps. Silicon Valley Product Group, September 7, 2015. https://www.svpg.com/the-alternative-to-roadmaps/. Accessed 2026-10-26. Used for: OKRs as "my current favorite," outcome rather than output, the hypothetical 30-day-to-3-hour onboarding objective, high-integrity commitments.
- blogHawkins, J. and Neu-ner, L. Product management is broken. Engineers can fix it. PostHog newsletter, Dec 3, 2024. https://posthog.com/newsletter/product-management-is-broken. Accessed 2026-10-26. Used for: why PostHog dropped OKRs and metric-based goals, the quarterly planning flow, the monthly growth review and its data inputs, "autonomy can become directionless."
- officialPostHog. Setting quarterly goals, Handbook. Undated living page. https://posthog.com/handbook/company/goal-setting.md. Accessed 2026-10-26. Used for: output-based goals, "use metrics only if they help you," the goal template, named owners, the pay-review and missed-objectives wording.
- officialLinear. Linear Method: Principles & Practices; Set the product direction; Set useful goals. Undated. https://linear.app/method/introduction, https://linear.app/method/product-direction, https://linear.app/method/set-useful-goals. Accessed 2026-10-26. Used for: "measure progress with actual work," initiatives and the product timeline, the goals page's 10-users-to-MRR example, and the absence of metrics and OKRs on these pages (confirmed by searching the page text).
- officialSinger, R. Shape Up: Stop Running in Circles and Ship Work that Matters. 37signals / Basecamp, web edition (page states only "Copyright 1999-2026 37signals LLC"). Chapters 7 and 8: https://basecamp.com/shapeup/2.1-chapter-07, https://basecamp.com/shapeup/2.2-chapter-08. Accessed 2026-10-26. Used for: the betting table's membership, "a smart bet has a cap on the downside," the pitch's "specific payout," failure read as a shaping error, and the absence of the words metric, OKR, goal and measure from both chapters.
- blogYu, N., Roman, S. and Largman, R. How we run projects at Linear. Linear, October 5, 2023. https://linear.app/now/how-we-run-projects-at-linear. Accessed 2026-10-26. Used for: engineers deciding readiness, "iterate too much," and the absence of a post-release effect review in the post.
- paperAmershi, S., Weld, D., Vorvoreanu, M., Fourney, A., Nushi, B., Collisson, P., Suh, J., Iqbal, S., Bennett, P., Inkpen, K., Teevan, J., Kikin-Gil, R. and Horvitz, E. Guidelines for Human-AI Interaction. CHI 2019 (DOI 10.1145/3290605.3300233). https://www.microsoft.com/en-us/research/wp-content/uploads/2019/01/Guidelines-for-Human-AI-Interaction-camera-ready.pdf. Accessed 2026-10-26. Used for: §1 on probabilistic behavior, §3 phases, Table 1 (18 guidelines), §5 results (785 examples; G11, G5, G6, G10, G14 findings), §7 limits.
- paperBernardi, L., Mavridis, T. and Estevez, P. 150 Successful Machine Learning Models: 6 Lessons Learned at Booking.com. KDD 2019, Applied Data Science Track (DOI 10.1145/3292500.3330744). Listing: https://www.kdd.org/kdd2019/accepted-papers/view/150-successful-machine-learning-models-6-lessons-learned-at-booking.com; text read from a PDF copy of the paper hosted at https://blog.kevinhu.me/2021/04/25/25-Paper-Reading-Booking.com-Experiences/bernardi2019.pdf (the ACM page returned HTTP 403). Accessed 2026-10-26. Used for: §3 and Figure 4 (23 comparisons, correlations and intervals), the four mechanisms, §5 latency experiment, §7.2-7.3 three-group design.
- blogCagan, M. Build To Learn FAQ. Silicon Valley Product Group, April 27, 2026. https://www.svpg.com/build-to-learn-faq/. Accessed 2026-10-26. Used for: protecting general users from rapid experimentation, "constant, erratic change," opt-in test groups.
- officialPostHog. Product management at PostHog (product-team page), Handbook. Living page, references Q2 2026. https://posthog.com/handbook/product/product-team.md. Accessed 2026-10-26. Used for: the Q2 2026 product-management themes on metric definitions and human-versus-agent tracking.