Product Management for Engineers · Page 4

Metrics, Experiments and Goals: OEC, Guardrails, HEART, OKRs and Trust

A team needs a number it can move in two weeks that still points at what it wants in two years, a list of things it will not break to move it, and a habit of distrusting results that look too good. This page covers the overall evaluation criterion and guardrails, the HEART and Goals-Signals-Metrics frameworks, what experiment base rates imply for a portfolio of work, how OKRs and their alternatives set goals, and what changes when a feature behaves probabilistically. Afterwards you can write a goal-signal-metric chain for a feature, choose an OEC and guardrails for it, grade an OKR set, and say what evidence would justify shipping with or without an experiment.

Deep dive 65 min read 15 sources As of 2026-10-26
The measurement chain: strategy chooses the OEC, an experiment moves it, a trust check and guardrail gates sit between the result and the ship decision Strategy OEC Experiment Trust check picks targets result Chooses the OEC and what not to trade away Measurable within the experiment, believed to drive goals Control vs Treatment on randomized users; hill-climbs the OEC Too good to be true? Replicate, check the logging first Guardrail gates: what the organization will not trade Crash rate Latency Sessions per user pass Ship, iterate, or stop verified result A guardrail failure blocks the ship even when the OEC rises. Gate names are examples from the source, not a fixed set.
Figure 1. The measurement chain this page covers. Strategy drives the choice of OEC, which must be measurable within the experiment yet believed to drive long-term goals (Kohavi et al. pp. 6, 21)1; guardrails mark what the organization is unwilling to change (p. 22)1; Twyman's law is the reason a surprising result is checked before it is believed (Preface, p. xv)1. The three gates are the source's own examples: crashes (p. 22), latency (pp. 17-18) and sessions per user (p. 5).

1. The overall evaluation criterion

An experiment needs one question: did the treatment make things better? Kohavi, Tang and Xu name the quantity that answers it the overall evaluation criterion and define it by two constraints. It "must be measurable in the short term (the duration of an experiment) yet believed to causally drive long-term strategic objectives" (Ch. 1, p. 6)1. The first constraint is operational, and the book puts a number on it: an organization "should define an OEC that can be easily measured over relatively short durations (e.g., one to two weeks)" (p. 12)1. The second is a belief, not a measurement, and the book says where the effort goes. The authors' companies keep teams "focused on metrics: choosing metrics, validating metrics, and evolving metrics over time," and those teams work out "which metrics measurable in the short term drive long-term objectives, since experiments usually run over a shorter time frame" (p. 22)1.

For an engineer the structure is familiar: the OEC is a surrogate objective, and the experiment optimizes the surrogate. How far the result can be trusted depends on the correlation between surrogate and true objective, which is the one quantity a two-week experiment cannot observe (my framing of the book's definition). The book states the difficulty as "finding metrics measurable in a short period, sensitive enough to show differences, and that are predictive of long-term goals" (p. 12)1. It prefers a single metric, "possibly as a weighted combination" of objectives (p. 7)1, while allowing that large organizations may have several OECs or "several key metrics that are shared with refinements for different areas" (p. 12)1.

1.1 Candidate OECs and how each fails

The fetched sources give no recipe for choosing an OEC, only examples, and the failures teach more than the successes. Table 1 lines up every candidate they discuss.

Table 1. Candidate OECs and the failure or strength each source reports. Page numbers are the book's; "MS 2009" is Kohavi, Crook and Longbotham, section numbers as printed12.
CandidateWindow usedWhat the source saysWhere
ProfitshortProfit is "not a good OEC," because "short-term theatrics (e.g., raising prices) can increase short-term profit, but may hurt it in the long run"1Ch. 1, p. 12
Customer lifetime valuelongCalled "a strategically powerful OEC"; the open problem is a short-period predictor of it1Ch. 1, p. 12
Revenue aloneshortRevenue alone "is insufficient as an OEC": it "could lead to plastering the web site with ads." Bing weighs revenue against user-experience metrics including sessions per user1Ch. 1, p. 5
Click-through rateshortNamed as "too gameable and needing to evolve"1; ads clicked is "a short term metric that will lead to short-term gains and long-term doom" while time on site and visit frequency are "better criteria"2Ch. 1, p. 22; MS 2009 §6.2
Active days per userexperiment durationThe book's own example: "increasing this OEC implies that users are visiting your site more often"1Ch. 1, p. 6
Six-week return rate2 weeks to pick a cohort, then 6 weeksMSN video pre-roll test: a window longer than the one to two weeks the book recommends, chosen because the business question was loyalty2MS 2009 §4.5
Clicks on the Buy buttonshortTreatment had 64% fewer users click Buy; the authors say the experimenters "should have done the additional work to measure what should have been the OEC, actual purchases"2MS 2009 §4.6

The Buy-button case is the cleanest illustration of proxy failure. The Control had more than twice as many users click Buy. The authors believe the cause is that the Treatment showed the product price and the Control did not, because "when the price is not shown there are more clicks because some users are looking for this crucial bit of information," and they note it is "entirely possible that no more users purchased the product"2. The proxy rewarded hiding information. Every row of Table 1 is a claim about causation that someone must check; none of them is a property of the metric.

1.2 A worked derivation: the OEC as an exchange rate

A composite OEC adds quantities that do not share a unit. The weights are exchange rates, and the MSN video test shows what that means in numbers. The Control played an ad before the first video; the Treatment moved the ad after the content. The site owners' question was "Would the loyalty of users increase enough in the Treatment to make up for the loss of revenue from not showing the ad up front?" The OEC was the six-week return rate of a cohort. The return rate "increased just over 2% in the Treatment, not enough to make up for the loss of ad impressions, which dropped more than 50%"2.

The paper gives the two deltas and the verdict but not the valuations. They can be backed out. At baseline, let \( a_0 \) be the pre-roll ad revenue per user, \( r_0 \) the share of the cohort that returns, and \( V \) the value of one return. Assume ad revenue is proportional to impressions and read both reported changes as relative (the paper does not say whether the 2% is relative or absolute). The Treatment loses \( 0.5\,a_0 \) per user and gains \( 0.02\,r_0 V \). It wins only if

$$ 0.02\, r_0 V \;>\; 0.5\, a_0 \quad\Longleftrightarrow\quad r_0 V \;>\; 25\, a_0 . $$

So the baseline value of return visits per user must exceed 25 times the pre-roll revenue per user before a 2% loyalty gain pays for a 50% impression loss. If the 2% is instead two percentage points on the return rate, the condition becomes \( V > 25\,a_0 \): same factor, different quantity. Both reported figures are rounded ("just over 2%," "more than 50%"), so 25 is an approximate threshold. The authors' conclusion, "not enough," therefore says something about their valuation: it implied returns worth less than roughly that multiple. This is my arithmetic on the paper's two reported figures, not a figure from the paper.

The same team did the explicit version in the MSN home-page ads test. It valued clicks two ways (what the destination property assigns to a click, and what Microsoft pays search engines to win that visit back), and "the numbers were close enough to get agreement on the monetization value to use." In a 12-day test on 5% of US home-page users, click-through rate and page views per user-day each fell 0.35% (relative); the lost clicks were worth more than the added ad revenue, an estimated loss of "millions of dollars per year" had it shipped2. The experiment supplied two small deltas. The valuation carried the argument, and it was settled by agreement.

1.3 Strategy picks the OEC; experiments climb within it

The book is direct about the dependency: "Having a strategy is critical for running experiments: the strategy is what drives the choice of OEC" (p. 21). Experiments then "help hill-climb to a local optimum based on your current strategy and product," and the failure the authors have seen is "when the OEC is not properly chosen"1. Their image for a bad OEC is "experimenting to improve the food or lighting on a sinking cruise ship" (p. 22)1. Page 6 treats strategy as a contested concept; the consequence here is narrower. Whoever chooses the OEC is making a strategic decision, and an engineer handed a metric to optimize has been handed a strategy.

The one fetched case where an OEC is used as a team goal with a number is Bing's relevance team: "several hundred people tasked with improving a single OEC metric by 2% every year." The 2% "is the sum of the Treatment effects (i.e., the delta of the OEC) in all controlled experiments that shipped to users over the year, assuming they are additive," and because "the team runs thousands of experiment Treatments, and some may appear positive by chance," credit is assigned from a separate certification experiment run with a single Treatment (pp. 14-15)1. Dilution matters when counting: "a 5% improvement for 10% of your users" is 0.5% overall, if the triggered users resemble the rest (p. 14)1. By that arithmetic the 2% target is four shipped wins of 0.5% each (my division, under the book's additivity assumption). Section 10 returns to this as one of the alignment mechanisms.

2. Guardrails and the ship decision

An OEC says what the organization wants more of. A guardrail says what it will not give up to get it. The book ties the two to strategy: "Defining guardrail metrics for experiments is important for identifying what the organization is not willing to change, since a strategy also 'requires you to make tradeoffs in competing – to choose what not to do'" (Ch. 1, p. 22, quoting Porter)1. The software example it offers is crashes: "if a feature is increasing crashes for the product, the experience is considered so bad, other factors pale in comparison." The Bing headline win passed this test by construction: revenue rose 12% "without significantly hurting key user-experience metrics" (p. 3)1.

2.1 A guardrail is a constraint, not a heavy weight

The book says safety can be handled "either via high weight in the OEC, or, equivalently, using passenger safety as a guardrail metric" (p. 22)1. The two are equivalent only in a limit. Write the composite as \( \text{OEC} = \sum_i w_i m_i \) and let \( g \) be the guarded metric with floor \( g^{*} \). A heavy weight corresponds to a penalty \( \lambda \min(0,\, g - g^{*}) \) with finite \( \lambda \). That penalty is an exchange rate in the sense of Section 1.2: a large enough OEC gain still buys a small loss in \( g \). A guardrail is the case \( \lambda \to \infty \), maximize the OEC subject to \( g \ge g^{*} \), where no gain buys the loss. The practical difference is who gets to argue. With a weight, every ship review reopens the price. With a gate, the price was settled in advance and the review only checks whether the gate held. This is my formalization of the book's one-line equivalence.

The fetched chapter gives three guardrail examples (crashes, latency, sessions per user) and no taxonomy. The book's dedicated treatment is a sidebar in Chapter 6 (p. 98 in its table of contents), which was not fetched1. Not every constraint is a metric either: some of Bing's monthly ad packages were known to be negative "as a result of space constraints or legal requirements" (p. 15)1.

2.2 The anti-metric: percent of plan delivered

The opposite of an OEC is a metric that cannot fail. The authors describe the temptation: "It is often easier to generate a plan, execute against it, and declare success, with the key metric being: 'percent of plan delivered,' ignoring whether the feature has any positive impact to key metrics" (p. 12)1. A team measured this way ships on schedule and learns nothing. Its failure is a case of a general one: a proxy becomes the target. The book gives Goodhart's law, Campbell's law and the Lucas critique a section in Chapter 7 (p. 109 in the table of contents), unfetched here1; the chapter's own instance is click-through rate, "too gameable and needing to evolve" (p. 22)1. Section 9 shows the same failure in goal-setting, where key results written as outputs make a team's grade a function of delivery.

2.3 The decision table

The book does not print a ship-decision matrix. Table 2 assembles one from its scattered statements, and says which row rests on which page.

Table 2. A ship decision from OEC result and guardrail status. Rows 3 and 4 quote the book; rows 1, 2 and 5 are my synthesis of its logic1.
#OEC resultGuardrailsReadingBasis
1Up, as expectedAll holdShipWriter's synthesis
2UpOne failsDo not ship as built; the gate was set in advance so the price is not renegotiatedWriter's synthesis, from p. 22
3FlatAll holdNo measured value; the remaining inputs are costs such as "estimates of maintenance costs for the new code" (p. 12)Quoted, p. 12
4DownAnyStop this tactic, but one failure may mean "the specific tactic being poor, not necessarily indicating that the overall strategy is bad" (p. 23)Quoted, p. 23
5Far above expectationAnyVerify before any other row (Section 5)Writer's synthesis, from Preface and p. 3

Row 4 hides a second decision the table cannot make. Repeated failures can mean the strategy is wrong, and the book's answer is "However beautiful the strategy, you should occasionally look at the results" (attributed to Churchill, p. 23)1. Section 4 gives the case where Bing acted on it.

3. What has to be true before you can experiment

The book is candid that "not every decision can be made with the scientific rigor of a controlled experiment": you cannot run one on a merger, "as we cannot have both the merger/acquisition and its counterfactual" (Ch. 1, p. 10)1. It lists four technical ingredients (pp. 10-11) and three organizational tenets (p. 11)1. The ingredients are the part an engineer can check against his own product. Table 3 sets them beside a reading for a runtime that ships inside customers' applications; that last column is my reading, flagged as such, because no fetched source covers infrastructure or on-device products.

Table 3. Kohavi et al.'s four ingredients for useful controlled experiments (Ch. 1, pp. 10-11, 13)1, and a conditional reading for a runtime embedded in customers' apps (writer's reading, not from the source).
IngredientWhat the book requiresReading for an embedded runtime (conditional)
1. Units without interferenceUnits "can be assigned to different variants with no interference (or little interference)"; treated users must not affect control usersThe unit has to be chosen: end-user device, app, or developer account. The choice drives row 2
2. Enough units"thousands of experimental units: the larger the number, the smaller the effects that can be detected"; small startups can start "initially looking for big effects"If the unit is a developer account and customers number in the hundreds, the unit count fails; if it is an end-user device, the count may pass but the data may belong to the customer's app
3. An agreed, loggable OEC"Key metrics, ideally an OEC, are agreed upon and can be practically evaluated"; where goals are hard to measure, agree on surrogatesTime from install to first successful inference is loggable inside the runtime only if telemetry is permitted
4. Changes are easy to make"Server-side software is much easier to change than client-side"; new hardware devices are rarely experimented on because lead times are longA runtime is client-side code: a variant reaches users with an app release, not a server flag, so iteration is slower than the book's examples

When ingredients fail, the book does not say "decide by opinion." It says controlled experiments are "the most reliable and sensitive mechanism to evaluate changes" when they can be run, and otherwise recommends "other, less trustworthy, methods, including observational studies" (pp. 10-11)1. Chapter 10 on complementary techniques lists the alternatives (logs-based analysis, human evaluation, user experience research, focus groups, surveys, external data; pp. 127-135 in the table of contents), and Chapter 11 covers observational causal studies; neither was fetched1. Two cautions come with the thousands-of-units rule. The book is written by people whose companies each run "more than 20,000 controlled experiments a year" (front cover)1, so its examples are drawn from the best-instrumented products in the industry. And the second tenet, willingness "to invest in the infrastructure and tests to run controlled experiments and ensure that the results are trustworthy" (p. 11), is the cost of entry; building that platform is Part IV of the book, "aimed toward engineers" (Preface, p. xvi), and outside this page1.

4. Experiment results as a portfolio

The book's third tenet is that "the organization recognizes that it is poor at assessing the value of ideas" (Ch. 1, p. 11)1. The evidence offered is a set of success rates, and they are four different quantities. Figure 2 draws them as separate bars and does not average them. Page 1 uses the same figures for a different argument, how ideas die before and after testing; here they matter for what a team should plan on.

4.1 Four base rates, kept apart

Four reported experiment success rates drawn as separate bars with different bases: Microsoft about one third, Bing and Google 10 to 20 percent, Slack about 30 percent, Netflix implied 10 percent secondhand 0%10%20%30%40% share of ideas reported as successful (the basis differs in every row) Microsoft tests meant to improve a key metric about one third Bing and Google some measures, via Manzi 2012 10 to 20% Slack monetization experiments only about 30% positive Netflix, secondhand "considers 90% wrong," not a rate implied 10%
Figure 2. Reported experiment success rates, one bar per basis. Microsoft: "only one third of the ideas tested ... improved the metric(s) they were designed to improve"; Bing and Google: "some measures' success rate is about 10-20%" (Manzi 2012); Slack: "only about 30% of monetization experiments show positive results"; Netflix: Moran (2007) "wrote that Netflix considers 90% of what they try to be wrong," so the bar is an inference, drawn dashed (Kohavi et al., Ch. 1, p. 13)1. The Microsoft figure is also in the original 2009 paper, §52. Slack's director of product adds: "get used to, at best, 70% of your work being thrown away. Build your processes accordingly."

Only the Microsoft row has a primary source in hand: the 2009 paper reports that among "well-designed and executed experiments that were designed to improve a key metric, only about one-third were successful at improving the key metric"2. The other three reach this page through the book, as a citation of Manzi, a quotation of a Slack executive's post, and a secondhand reading of Netflix. The 2009 paper adds an offline base: QualPro, a consultancy running multivariable controlled experiments, tested 150,000 business-improvement ideas over 22 years and reported that 75 percent "either have no impact on performance or actually hurt performance"2. The same paper reports that "only three out of 21 people guessed the winner" among six widget designs, and the three "were from the ExP team" (§4.1)2. What survives is the weaker claim the authors make: "Not every domain has such poor statistics, but most who have run controlled experiments in customer-facing websites and applications have experienced this humbling reality" (p. 14)1.

4.2 The payoff tail

The same chapter opens with the opposite end of the distribution. A Bing ad-headline change was "prioritized low and languished in the backlog for more than six months until a software developer decided to try the change, given how easy it was to code"; the test raised revenue 12%, "over $100M annually in the US alone" (p. 3), a return on "a few days' work for one engineer" (p. 4)1. The authors then say how rare that is: "Bing runs over 10,000 experiments a year, but simple features resulting in such a big improvement happen only once every few years" (p. 5)1. Taking "a few" as two to five years, which is my reading, that is roughly one such result in 20,000 to 50,000 experiments. Typical gains are of a different order: "improvements to key metrics are achieved by many small changes: 0.1% to 2%" (p. 14)1. Success rates of 10 to 33%, a typical gain of 0.1 to 2%, and a tail event worth $100M a year describe a heavy-tailed payoff, and in such a game the cost per draw decides the strategy (my inference). The book points the same way when it lists "the overhead of running an experiment must be small" among the themes of the Bing case (p. 5)1. What the case implies for backlogs and for how ideas are ranked is the subject of page 3.

4.3 How many experiments before a win

If each experiment independently improves its target metric with probability \( p \), the chance of at least one win in \( n \) is \( 1-(1-p)^n \). To reach confidence \( c \), solve for \( n \):

$$ n \;\ge\; \frac{\ln(1-c)}{\ln(1-p)} . $$

At \( c = 0.9 \) this gives 5.68 (so 6 experiments) for \( p = 1/3 \), 10.32 (so 11) for \( p = 0.2 \), and 21.85 (so 22) for \( p = 0.1 \), the edges of the reported range. The assumptions are strong: independence, a constant \( p \), and "win" meaning the metric moved, not that the change was worth shipping. The arithmetic is mine. Its use is to turn a base rate into a capacity requirement. A team that can finish four experiments a month and works where \( p \) is near 0.1 should expect about five and a half months before a 90% chance of one win, and the calculator below lets you move all three knobs.

Probability of at least one win against number of experiments run, for success rates of one third, 20 percent and 10 percent, with the 90 percent crossing marked at 6, 11 and 22 experiments 0%25%50%75%100% probability of at least one win 0 5 10 15 20 25 30 experiments run, n (independent, same p) 90% chance of a win n=6 n=11 n=22 p = 1/3 (Microsoft) p = 20% (upper end) p = 10% (lower end)
Figure 3. Probability of at least one win, \( 1-(1-p)^n \), for the reported success rates of one third (Microsoft) and 20% and 10% (the Bing and Google range)12. The orange points mark where each curve first passes 90%: 6, 11 and 22 experiments. Derived by the writer under independence and constant \( p \); only the three rates are sourced.

Experiments, and months, until a win

Experiments needed
Months at that pace

Formula: \( n = \lceil \ln(1-c) / \ln(1-p) \rceil \). Independence and a constant success rate are assumptions; the default 15% sits inside the reported 10-33% span but is a free parameter, not a measurement.

4.4 Near bets, radical bets, and sunk costs

The book's recommendation follows from the numbers: "always having a portfolio of ideas: most should be investments in attempting to optimize 'near' the current location, but a few radical ideas should be tried," because "most big jumps fail (e.g., big site redesigns), yet there is a risk/reward tradeoff: the rare successes may lead to large rewards that compensate for many failures" (Scenario 2, p. 23)1. Redesigns fail for a measurable reason: they "subject users to primacy effects" and "commonly fail not only to achieve their goals, but even fail to achieve parity with the old site on key metrics" (p. 21)1. The same effect biases the test, since a short experiment on a redesign measures "primacy effects or change aversion" and "may not measure the true long-term effect" (p. 23)1.

The portfolio also has an exit rule. Bing pursued a social-media integration strategy for about two years; "after spending over $25 million on the strategy with no significant impact to key metrics, the strategy was abandoned," and the authors add that "failed bets are sunk costs, and we should make a forward-looking decision based on the available data" (pp. 23-24)1. A single failed experiment tests one tactic; many failed tactics are a signal about the strategy. That is the row-4 caveat of Table 2 and the reason the OEC and the strategy must be revisited together.

5. Trusting a number: Twyman's law

The Bing headline test did not announce itself as a triumph. "A few hours after starting the test, a revenue-too-high alert triggered, indicating that something was wrong with the experiment." The authors call these "too good to be true" alerts "very useful, as they usually indicate a serious bug, such as cases where revenue was logged twice (double billing) or where only ads displayed, and the rest of the web page was broken." In this case the increase "was valid," and "the experiment was replicated multiple times over a long period" (Ch. 1, p. 3)1. The Preface turns the habit into a rule: "We believe in the skepticism implied by Twyman's Law: Any figure that looks interesting or different is usually wrong; we encourage readers to double-check results and run validity tests, especially for breakthrough positive results" (p. xv)1.

5.1 Twyman's law as Bayes' rule with a heavy-tailed prior

The rule has a mechanism, and Section 4.2 supplied its input. Let \( q \) be the prior probability that an experiment contains a genuine effect as large as the one observed, and \( b \) the probability that it contains a defect (double logging, a broken page) that fakes a jump of the same size. If either cause produces the jump with certainty, then

$$ P(\text{real} \mid \text{jump}) \;=\; \frac{q}{q+b}. $$

From the book's figures, Bing runs 10,000 or more experiments a year and a result of the $100M kind comes "once every few years" (p. 5)1, so with \( y \) years between such wins \( q = 1/(10{,}000\,y) \). At \( y = 3 \) that is 3.3 per 100,000 experiments. The odds turn even when \( b = q \): a defect that fakes a jump of that size in one experiment out of every 30,000 is enough. Nothing in the source gives \( b \); it is the free knob in Figure 4. The structural point does not depend on its value. The larger the claimed effect, the smaller \( q \), because the payoff tail is heavy, while defects do not become rarer as they get bigger (a metric logged twice doubles regardless). That argument is mine, not the book's. Skepticism should therefore grow with the size of the surprise, which is what "especially for breakthrough positive results" asks for. The book's definition of an interesting result points the same way: "Interesting experiments are ones where the absolute difference between the expected outcome and the actual result is large" (p. 16)1. The results most worth learning from are the ones most likely to be wrong.

Probability that a huge observed jump is real against the rate of bugs that fake a jump, for a genuine huge win every 1, 3 or 10 years, on a log axis 0%25%50%75%100% probability the jump is real 0.1 1 10 100 defects that fake an equally large jump, per 100,000 experiments (log scale) b=10 b=3.3 b=1 a genuine huge win comes once per year 3 years 10 years
Figure 4. Probability that a huge observed jump is real, \( q/(q+b) \), with \( q = 1/(10{,}000\,y) \) taken from Bing's reported "over 10,000 experiments a year" and a result of that size "once every few years" (Kohavi et al., Ch. 1, p. 5)1. The horizontal axis is the unsourced knob \( b \). Orange points mark \( b = q \), where the odds are even. Derived by the writer.

Is the jump real?

Prior of a genuine huge win, per 100,000
Probability the jump is real

Formula: \( q/(q+b) \). The defaults 10,000 and 3 are the book's order of magnitude and a reading of "every few years"; the faked-jump rate has no source and starts at an arbitrary 5.

5.2 Causation, and the ladder of evidence

The second trust problem is older than experiments. The book's example is Microsoft Office 365: users who see error messages and crashes churn less, "but that does not mean that Office 365 should show more error messages." All three facts have one cause, usage: "heavy users of the product see more error messages, experience more crashes, and have lower churn rates" (pp. 8-9)1. The authors rank evidence on a ladder adapted from medicine (Figure 1.3, p. 9): meta-analyses of randomized experiments at the top, then randomized controlled experiments, then other controlled experiments, then observational studies, and at the bottom "case studies ..., anecdotes, and personal (often expert) opinion, a.k.a. HiPPO"1. Section 8 uses this ladder to say what evidence a ship decision needs. The rest of the book's trust machinery, including Chapter 3 on threats to validity (p. 39) and Chapter 21 on sample ratio mismatch (p. 219), sits in chapters not fetched1.

6. PULSE and HEART

An experiment platform tells you whether a treatment moved a metric. It does not tell you which metrics deserve to exist. Google's UX research team addressed that gap in a 2010 CHI note by Rodden, Hutchinson and Fu3. They start from what teams already track, which they call PULSE: "Page views, Uptime, Latency, Seven-day active users (i.e. the number of unique users who used the product at least once in the last week), and Earnings." These are "all extremely important," and an engineer will recognize them as what an operations dashboard shows. The authors' complaint is that they are "either very low-level or indirect metrics of user experience, making them problematic when used to evaluate the impact of user interface changes," and may be ambiguous: a rise in page views may occur "because the feature is genuinely popular, or because a confusing interface leads users to get lost in it, clicking around to figure out how to escape" (section "Pulse Metrics")3.

6.1 HEART is a set of categories, not a set of metrics

HEART stands for "Happiness, Engagement, Adoption, Retention, and Task success." The authors stress that "these are categories, from which teams can then define the specific metrics that they will use," that "it is not always appropriate to employ metrics from every category," and that the framework's value is to make "an explicit decision about including or excluding a particular category." Their example of exclusion is Engagement in an enterprise product where use is required, with the note that "it may still be meaningful to consider Engagement at a feature level"3. Happiness and Task success generalize older usability measures; Engagement, Adoption and Retention are "new categories, made possible by large-scale behavioral data"3.

Table 4. The five HEART categories with the definitions and examples in Rodden, Hutchinson and Fu (CHI 2010). Examples are Google-internal; the note reports no outcome data for any of them3.
CategoryKind of dataPaper's definitionPaper's example
HappinessattitudinalSubjective aspects: satisfaction, visual appeal, likelihood to recommend, perceived ease of useiGoogle weekly in-product survey, 7-point bipolar scale; satisfaction fell after a major redesign, then recovered, "indicating that change aversion was probably the cause"
EngagementbehavioralBehavioral proxies such as "the frequency, intensity, or depth of interaction over some time period"; report as an average per user, not a totalGmail: "the percentage of active users who visited the product on five or more days during the last week"
AdoptionbehavioralHow many new users start using the product in a periodAccounts created in the last seven days
RetentionbehavioralHow many users from one period are still present in a later oneShare of a week's seven-day actives still active three months later
Task successbehavioralEfficiency (time to complete), effectiveness (percent completed), error rateGoogle Maps single vs dual search box, A/B test compared on error rates

The paper supports two further uses. In September 2008 Google Finance saw a surge in page views and seven-day actives, and those totals "did not indicate whether the surge was driven by new users interested in the crisis, or existing users panic-checking their investments." Adoption and Retention metrics separated the two groups and showed the rate at which new users kept using the site3. And the Gmail metric was chosen with an explicit reason, "engaged users should check their email account regularly, as part of their daily routine," and the authors found it "strongly predictive of longer-term retention, and therefore could be used as a bellwether for that metric"3. That sentence is the only place in the note where the short-term-to-long-term link that an OEC depends on (Section 1) is reported as tested, and it gives no numbers.

6.2 Why totals mislead: the seven-day-active trap

The note makes a precise claim about its own PULSE metric: seven-day actives "measures the overall volume of the user base, but gives no insight into the users' level of commitment," does not "differentiate between new users and returning users," and "in a worst-case retention scenario of 100% turnover in the user base from week to week, the count of seven-day active users could still increase, in theory"3. The reason is arithmetic. If \( N_t \) new users arrive in week \( t \) and none returns, then \( \mathrm{7DA}_t = N_t \), which rises whenever acquisition rises. Figure 5 draws that case with an illustrative 10% weekly growth in new users. The cure the paper gives is normalization: "raw counts will go up as your user base grows, and need to be normalized; ratios, percentages, or averages per user are often more useful"3.

Seven-day active users rise 10 percent a week from 100 to 195 over eight weeks while retention and the share of users active five or more days stay at zero, under 100 percent weekly turnover Seven-day active users 100wk 1 110wk 2 121wk 3 133wk 4 146wk 5 161wk 6 177wk 7 195wk 8 Per-user and retention metrics 100%0% Week-over-week retention and share of users active 5+ days: 0% every week
Figure 5. The worst case in the HEART paper, drawn: 100% weekly turnover with new users growing 10% a week (100, 110, 121, ...), each user visiting once. Seven-day actives rise every week while retention and the "five or more days" share (the Gmail metric) are zero. The growth rate is an illustrative input chosen by the writer; the claim that the count "could still increase" is the paper's3.

Three limits are worth stating plainly. The framework was "applied to more than 20 different products and projects from a wide variety of areas within Google," a company whose products are "almost all web-based, and have millions of users each." The note reports no evidence that using HEART improved a product's outcomes; its examples show decisions that metrics informed3. The metrics "are primarily useful for evaluation of launched products, and are not a substitute for early or formative user research," and "should be triangulated with findings from other sources, such as usability studies and field studies"3. And no HEART category has been checked for a product whose users are developers: whether an SDK's "Adoption" is a signup, a first successful call, or a production release is a definition the team has to make, which is the work the next section structures.

7. Goals, signals, metrics

HEART names categories; a second piece of the same paper says how to get from a category to a number. The process has three steps. In the paper's words it "steps teams through articulating the goals of a product or feature, then identifying signals that indicate success, and finally building specific metrics to track on a dashboard," because a metric "is unlikely to be useful in practice unless it explicitly relates to a goal"3. Each step has tips from Google's experience, and they read as engineering advice.

  • Goals. "Different team members may disagree about what the project goals are. This process provides a great opportunity to collect all the different ideas and work towards consensus (and buy-in for the chosen metrics)." Goals for a feature "may be different from those for the product as a whole." Do not "get too distracted at this stage by worrying about whether or how it will be possible to find relevant signals or metrics"3.
  • Signals. Ask what behavior or attitude would show the goal met. Choose signals "sensitive and specific to the goal," which "should move only when the user experience is better or worse, not for other, unrelated reasons." "Sometimes failure is easier to identify than success," such as task abandonment or "undo" events. Logs and surveys are the two sources used most often, and the check to make is whether "the relevant actions" are "currently being logged, or could they be?"3.
  • Metrics. Normalize raw counts. Filter "traffic from automated sources (e.g. crawlers, spammers)." Confirm that "all of the important user actions are being logged (which may not happen by default, especially in the case of AJAX or Flash-based applications)"3.
Four goal-signal-metric chains from the HEART paper: Gmail engagement, iGoogle happiness, Google Maps task success, and Google Finance adoption and retention GoalSignalMetricHEART Gmail Understand how engagedusers are Engaged users checkmail on most days % of active users on5+ days in last week Engagement iGoogle Judge the impact ofchanges and features Satisfaction aftera major redesign Weekly survey score,7-point bipolar scale Happiness Maps Is a single searchbox simplest? Errors made whilesearching Error rate, single vsdual box (A/B test) Task success Finance Understand a trafficsurge, Sept 2008 New users vsexisting users New-user continuationrate Adoption,Retention Cell text is the writer's paraphrase of what each team wanted to learn; the paper does not print the chains in this form.
Figure 6. Goal, signal and metric chains reconstructed from the four examples in Rodden, Hutchinson and Fu (CHI 2010): Gmail's daily-routine engagement metric, iGoogle's survey after a redesign, the Google Maps single-box A/B test measured by error rates, and the September 2008 Google Finance surge separated by Adoption and Retention3.

The goal step carries more weight than its length suggests. If teammates "disagree about what the project goals are," the metrics that follow will each serve a different goal and the dashboard will not settle any argument. That is why the paper treats the process as a consensus tool before it is a measurement tool.

A hypothetical chain shows the template on a developer-facing feature. It is not from any source, and no source covers infrastructure products. Goal: a developer integrates an on-device runtime and gets a first successful inference without help. Signals: an install followed by a successful inference call; no support contact before it; an install that never reaches one, since "failure is easier to identify than success." Metrics: the share of each week's new installs that reach a first successful inference within a stated window, and the median time to it, both per install cohort rather than totals. By analogy with the paper's enterprise note, where Engagement may still be meaningful "at a feature level"3, the engagement category would belong at the feature level for such a product; that extension is mine. Latency, memory and crash rate are not in the chain; they are the guardrails of Section 2.

8. How much evidence before acting

Sections 1 to 5 describe an instrument. Whether to wait for its reading is a separate decision, and two fetched sources answer it in opposite tones. Kohavi et al. say a "data-driven or a data-informed organization gathers relevant data to drive a decision and inform the HiPPO ... rather than relying on intuition," and that "a decision should be made with many sources of data, including controlled experiments, surveys, estimates of maintenance costs for the new code" (Ch. 1, p. 12)1. Bezos, in Amazon's 2016 shareholder letter, writes that "most decisions should probably be made with somewhere around 70% of the information you wish you had. If you wait for 90%, in most cases, you're probably being slow," and that "being wrong may be less costly than you think, whereas being slow is going to be expensive for sure"4.

8.1 The decision as an inequality

The two positions are answers to the same cost question, which can be written down. Let \( p_f \) be the probability that the idea does not help (the base rates in Section 4 put it at roughly 0.67 to 0.9), \( C_w \) the cost of shipping a dud until it is found and reversed, \( C_e \) the cost of running the experiment, \( t \) the time it takes, and \( D \) the value a good idea loses per unit of delay. Assume the test is reliable. Shipping now is worth \( (1-p_f)V - p_f C_w \); testing first is worth \( (1-p_f)(V - Dt) - C_e \). Testing wins when

$$ p_f\, C_w \;>\; C_e + (1-p_f)\, D\, t \qquad\Longleftrightarrow\qquad \frac{C_w}{D\,t} \;>\; \frac{1-p_f}{p_f} + \frac{C_e}{p_f\, D\, t}. $$

If the platform makes the experiment nearly free (\( C_e \to 0 \)), the right-hand side is \( (1-p_f)/p_f \): 0.5 at \( p_f = 2/3 \), 0.25 at 0.8, 0.11 at 0.9. A test is worth waiting for whenever a dud costs more than about a tenth to a half of what the delay costs a good idea. That is a low bar, and it explains why the book insists that "the overhead of running an experiment must be small" (p. 5)1: cutting \( C_e \) and \( t \) is what moves the decision. A two-way door is the case where \( C_w \) is small because reversal is cheap, and there the inequality flips toward shipping at 70% and watching, which is Bezos's rule. A one-way door makes \( C_w \) large and the rule inverts. This is my derivation. It omits imperfect tests, statistical power, and the information a decision generates for later ones (the book names the Expected Value of Information, p. 241).

8.2 What each source warns against

The warnings differ. Kohavi et al. fear the plan that cannot fail ("percent of plan delivered," Section 2.2) and note that "objective measurements typically show that progress is not as rosy as initially envisioned" (p. 12)1. They also record the exception that proves nothing: Apple under Steve Jobs, where Ken Segall "claimed that 'we didn't test a single ad. Not for print, TV, billboards, the web, retail, or anything'" (pp. 11-12)1. That is one sentence about advertising, reported secondhand, not a study of product decisions. Bezos fears stand-ins for the real thing and names two: process ("the process becomes the proxy for the result you want") and averages ("market research and customer surveys can become proxies for customers"). His example is "Fifty-five percent of beta testers report being satisfied with this feature. That is up from 47% in the first survey," which he calls "hard to interpret." His prescription is not to skip testing: "I'm not against beta testing or surveys. But you, the product or service owner, must understand the customer ... Then, beta testing and research can help you find your blind spots"4. Rodden et al. say metrics "should be triangulated" with qualitative findings and are no "substitute for early or formative user research"3.

Table 5. What evidence justifies shipping, by situation. Cost-of-dud reasoning is the writer's (Section 8.1); each row names the source for its premise134.
SituationCost of a dudEvidence that justifies shippingPremise from
Reversible, experiment availablelowA stated hypothesis, an OEC and guardrails; ship as an experiment and read the result (Table 2)Kohavi p. 5; Bezos "two-way doors"
Reversible, experiment impossible (too few units, slow client releases)low to mediumDecide near 70% of the information, then watch logs and use complementary techniques; say in advance what would trigger a revertKohavi pp. 10-11; Bezos on correcting quickly
Hard to reverse (a published interface, a persisted format; hypothetical examples)highMore than 70%: prototype with representative users, then decide; the inequality in 8.1 invertsWriter's derivation; Bezos on "two-way doors"
Not testable at all (a merger, a strategy)variesJudgment informed by "many sources of data," with the decision owner's reasoning written downKohavi pp. 10, 12
Result far above expectationhigh if falseReplicate and audit the logging before acting (Section 5)Kohavi p. 3, Preface

None of these sources has outcome data comparing evidence-heavy and judgment-heavy decision styles; Section "Where experts disagree" returns to that.

9. OKR mechanics and what they optimize

An OEC is a single scalar that a team moves by experiment. An OKR set is a different kind of object: a handful of ambitious statements, each with measurable key results, graded after a fixed period. Google's re:Work guide, "originally published 2016" and preserved "as a record," is the primary description5. Marty Cagan's 2015 essay on roadmaps names OKRs "my current favorite" and supplies the product-side rationale6. Table 6 lists the parameters.

Table 6. OKR parameters as stated in Google's re:Work guide (2016 record)5.
ParameterWhat the guide says
Count"three to five objectives with about three key results for each objective"; more "can lead to over-extended teams and a diffusion of effort"
ObjectiveObjectives "are ambitious and may feel somewhat uncomfortable" and use "tangible, objective, and unambiguous terms," so an observer can tell whether it was achieved
Key result"Measurable and should be easy to grade with a number"; they "should describe outcomes, not activities"
Grade0.0 to 1.0; each key result is graded and "using a rough average" the objective is graded; sometimes weighted
Target"Sweet spot" 60 to 70%; "the expectation is to get an average of 0.6 to 0.7 across all OKRs"; success "means achieving 70% of the objectives, while fully reaching them is considered extraordinary performance"
CadenceAnnual and quarterly; company-wide meetings quarterly to share and grade; an optional mid-quarter check-in
Visibility and usePublic; "not synonymous with employee evaluations"; "not a shared to-do list"; low grades are "data to help refine the next OKRs"

9.1 The grading scale is a calibration target

Most goal systems treat a miss as a failure. This one budgets for it: if a team's average is 0.6 to 0.7 the system is working, and "if someone consistently fully attains their objectives, their OKRs aren't ambitious enough"5. A consequence the guide spells out is for dependents. "If they are using stretch goals you should expect them to deliver on about 70% of their stated OKRs," so a team that depends on another team's stretch objective should plan on 70%5. An OKR grade is therefore an estimate with a built-in haircut, not a commitment. Cagan separates the two: dates are handled by "high-integrity commitments," used "for those situations where we need to actually commit to a date or a specific deliverable"6, and page 3 covers their cost.

The OKR grading scale from 0.0 to 1.0 with the 0.6 to 0.7 sweet spot highlighted, and two objectives with different key-result grades that both average 0.67 sweet spot 0.6 to 0.7 1.0 = fully achieved 0.0 0.0 0.2 0.4 0.6 0.8 1.0 Objective AKRs: 1.0, 0.7, 0.3 mean 0.67 Objective BKRs: 1.0, 1.0, 0.0 mean 0.67
Figure 7. The re:Work grading scale: 0.0 to 1.0, with a "sweet spot" of 60 to 70% and the rule that consistently reaching 1.0 means the objectives are not ambitious enough5. Objectives A and B are the writer's arithmetic: key results graded 1.0, 0.7, 0.3 and 1.0, 1.0, 0.0 both average 0.67, inside the band, though B contains a key result that scored zero. The guide itself calls averaging "rough."

The figure makes a limit visible. Because the objective grade is a mean, a team that fully misses one key result and fully makes the others lands in the same place as a team that was partly short on all three. The guide's own grading examples show how binary scoring enters: a key result "to 'Launch a new widget marketing campaign'" is 0 or 1, while "Launch six new features" with three launched is "graded with a 0.5"5.

9.2 Failure modes the guide lists, and one it contains

The guide's own list of mistakes is a catalog of ways a number stops measuring what it was written for. Business-as-usual OKRs are "written based on what the team believes it can achieve without changing anything they're currently doing." Sandbagging: "teams who can meet all of their OKRs without needing all of their team's bandwidth may either be hoarding resources, not pushing their teams, or both." Low-value objectives: "could the OKR get a 1.0 under reasonable circumstances without providing direct organizational benefit?" Insufficient key results: if they "don't represent all that is needed to fully achieve that objective," a miss arrives late and unexpected5.

The guide also contains the failure it warns against. Its text says key results "should describe outcomes, not activities," yet five of the seven key results in its two sample sets name a deliverable or an activity and no measured change, among them "Launch xx feature to all users," "Secure tech support to build xx feature in Q1," and "Re-establish [product]'s leadership by speaking at three industry events"; only the revenue-per-user and error-response-time results carry a measured change5. That is the pattern Kohavi et al. call "percent of plan delivered" (Section 2.2)1. Cagan's version of the principle is that the system "is all about outcome rather than output," illustrated by an objective to cut onboarding from 30 days to under 3 hours; he introduces the figures with "suppose your product currently requires 30 days," so they are an illustration, not data6. The guide's evidence for OKRs is one sentence: Sears Holding Company "rolled out OKRs to their 20,000 employees and saw positive impacts on bottom-line sales and individual performance," with no figures5.

Table 7. Seven tests for grading an OKR set. Tests 1 to 5 restate the re:Work guide5; test 1 also draws on Cagan6; test 6 draws on Kohavi et al. on gameable metrics1; test 7 is the writer's. Examples are hypothetical, with symbolic targets.
#TestFails whenPasses when (hypothetical)
1Outcome, not output"Ship the quantized backend""Median time from install to first inference falls from \( T_0 \) to \( T_1 \)"
2Gradable by a number, without argument"Improve developer experience"The grade follows from a logged quantity
3Not business as usual or sandbaggedThe team could score 1.0 while changing nothingStack-ranked against current work, it is a top effort
4Key results suffice for the objectiveAll three can score 1.0 and the objective still failsThe key results together imply the objective
5Three to five objectives, about three key results eachEight objectives, or one key resultWithin the stated counts
6Hard to gameA cheap action moves the number and no user would want itA guardrail covers the cheapest bad path
7Honest about its useThe grade also decides pay, with no statement of howThe use of the grade is stated in advance

Test 7 is where sources diverge. The re:Work guide says OKRs are "not synonymous with employee evaluations"5; Section 10 shows a company that ties its goals to pay review and sets a different standard for what a miss means.

10. Alignment alternatives and where each fails

An organization has to say what its teams are for. Five fetched sources do it in five ways, and Table 8 sets them side by side.

Table 8. Five ways to align a team to goals. "Unit" is what the team commits to; "Failure mode" is what the source itself says or what follows from its mechanism (marked where it is the writer's reading)1578910.
MechanismUnit of commitmentMeasureWho sets itFailure mode
OKRs (Google re:Work)Objective with about 3 key results, graded 0.0-1.0 each quarterKey-result grade; 0.6-0.7 is the targetTop-down and bottom-up mix; team leaders set team OKRsSandbagging, business-as-usual, low-value objectives; sample key results that are outputs5
OEC with guardrails (Kohavi et al.)A delta in one composite metric, certified by a replication experimentOEC change in an experiment, gated by guardrailsStrategy chooses the OEC; metrics teams validate itA badly chosen OEC ("experimenting to improve the food or lighting on a sinking cruise ship"); needs experiment scale1
Initiatives and a product timeline (Linear)A named stream of work; projects with target dates"Measure progress with actual work": the diff, the design fileNot stated on the fetched pagesNot stated; the measure is progress on work, not effect (writer's reading)9
Appetite-bounded bets (Shape Up)One shaped project for one six-week cycleNone named; "a specific payout" is defined in the pitchThe betting table: CEO, CTO, a senior programmer, a product strategistA failed bet is read as a shaping error: "we did something wrong in the shaping"10
Output goals plus a monthly growth review (PostHog)A list of things to build, each with one named owner"Use metrics only if they help you"; a PM-run monthly review of revenue, usage and feedback dataExecutives give high-level goals; engineers choose the workAutonomy "can become directionless" without the review78

Two warnings on reading the table. First, absence of a mechanism on a fetched page is silence, not opposition. The Shape Up chapters on bets and the betting table (7 and 8) contain none of the words "metric," "OKR," "goal" or "measure," which I confirmed by searching the pages10; Linear's practices and direction pages mention neither metrics nor OKRs9. Second, every row is a company describing its own practice, and none reports outcome data.

10.1 PostHog: why a company dropped metric goals

PostHog's James Hawkins and a colleague wrote in December 2024 that "we used to do OKRs and metric-based goals, but we ran into two problems with it": "teams wasted too much time coming up with the right metric to focus on," and "engineers focused on small quick wins to move those metrics, rather than building what our users actually wanted"7. Their replacement is a quarterly list of things to build and who owns them, chosen by engineers after executives share high-level company goals (for example, "we need to increase top of funnel adoption")7. The accountability moves to a monthly growth review run by the product manager, which compiles "revenue metrics" (MRR, churn), "product analytics" (active users, retention) and "user feedback" (NPS, interviews, tickets) and asks, for example, "Can we find leading indicators that predict long-term product usage?" The engineering team lead then decides whether to continue, reprioritize or change goals. The stated reason: "Without this accountability, autonomy can become directionless"7. The handbook's current planning page keeps the shape. Goals are "primarily output-based," with "metrics only if they help you," and the stated worry about metrics is that "all objectives are bad - they have many compromises, are fallible, easy to game"8. The same page also asks for "what we'll ship," a rationale, and "we'll know we're successful when: metric or outcome," and says "hitting an overarching objective is more important than shipping specific things"8. The handbook page carries no date and is a living document.

One difference from Google deserves a line. The re:Work guide treats a 0.6 to 0.7 average as the system working5. PostHog's page says "if your team is consistently missing objectives, they are too hard or possibly the wrong objectives," and "consistently hitting ambitious objectives over the long term is an important factor in the pay review process"8. The first uses misses as a calibration signal and keeps goals out of evaluation; the second treats misses as a goal-setting error and ties hits to pay. The phrase "goal" covers two different contracts.

PostHog's loop reads metrics after the fact instead of setting them as targets. That avoids the gaming problem of Section 2.2 at the price of the one Kohavi et al. describe, a plan that can be executed and declared a success. What separates PostHog from that failure is the review, not the goals, and the review is only as good as the PM's data and the willingness of the CEO or a senior leader to "challenge assumptions, ask hard questions, and ultimately hold the team accountable"7.

10.2 Linear and Shape Up: progress as work, bets as commitments

Linear's method asks teams to "measure progress with actual work," and says "the clearest way to see whether something is complete or not is to show the diff in the code or design file"9. Its direction page calls initiatives "a critical tool for shaping this direction" and wants a timeline "ideally a bit out of reach"9. Its "set useful goals" page asks for "a goal that propels you forward in some measurable way," with a startup example that runs from finding 10 users, to 100, to $1000 in monthly recurring revenue9. Linear's own account of how it ships says readiness "comes down to the engineers working on the project feeling that the project is ready," and the fetched post has no passage on judging a project's effect after release11. Shape Up's mechanism is the cap: "a smart bet has a cap on the downside. If we bet six weeks on something, the most we can lose is six weeks," and the betting table looks "at the bigger movement of progress on the whole product after the six weeks"10. The cap bounds the loss from a wrong bet but does not say how a team learns it was wrong. Neither source gives one.

11. Metrics for features that behave probabilistically

Everything above assumes a feature whose behavior the team can read off the code. A feature built on a model does not satisfy that. Amershi and colleagues at Microsoft Research open their CHI 2019 paper with the cause: "automated inferences are typically performed under uncertainty, often producing false positives and false negatives," so AI-infused systems "can violate established usability guidelines of traditional user interface design." Their example is consistency: "many AI components are inherently inconsistent due to poorly understood, probabilistic behaviors based on nuances of tasks and settings, and because they change via learning over time" (§1)12. Two properties follow for measurement. The error rate is part of the product, and the behavior can change between two visits by the same user.

11.1 The 18 guidelines

The paper consolidates "more than 150 design recommendations" into 18 guidelines, with a user study of 49 practitioners testing them on 20 AI-infused products (abstract, §3)12. Each is "a rule of action, containing about 3-10 words and starting with a verb," grouped by when in an interaction it applies (§3, Table 1). Figure 8 draws them in the paper's four phases. I have marked the four that govern a system that changes over time; the marking is mine.

The 18 human-AI interaction guidelines in four phases: initially G1 to G2, during interaction G3 to G6, when wrong G7 to G11, over time G12 to G18, with G13, G14, G16 and G18 marked as concerning adaptation Initially G1Say what it can do G2Say how well it does During interaction G3Time by context G4Show relevant info G5Match social norms G6Mitigate biases When wrong G7Easy invocation G8Easy dismissal G9Easy correction G10Scope when in doubt G11Explain why Over time G12Remember recent use G13Learn from behavior G14Adapt cautiously G15Granular feedback G16Show consequences G17Global controls G18Notify of changes
Figure 8. The 18 guidelines in the paper's four phases: G1-G2 initially, G3-G6 during interaction, G7-G11 when wrong, G12-G18 over time (Amershi et al., §3 and Table 1)12. Labels are paraphrased and shortened from the paper's wording. Orange blocks (G13, G14, G16, G18) are the guidelines I read as governing change over time; the paper does not mark them.

The three that matter most for a metrics decision are G2, G11 and G14. G2: "Make clear how well the system can do what it can do. Help the user understand how often the AI system may make mistakes." G11: "Make clear why the system did what it did." G14: "Update and adapt cautiously," glossed as "limit disruptive changes when updating and adapting the AI system's behaviors"12. The paper reports 785 examples across the 20 products: 313 applications, 277 violations, 89 neutral and 106 judged "does not apply." G11 "had one of the highest number of violations," with among the fewest "does not apply" ratings, which the authors read as participants who could imagine an explanation but were often unable to obtain one. G5 and G6 were "noted as the least clear." G10 and G14 were hard to observe: "it's a bit difficult to assess this in a single session," one participant wrote about G1412. This study measured whether practitioners could apply and understand the guidelines. It did not measure retention, revenue or task success, so no sentence in it says that following a guideline raises any business metric.

G14 connects to the experiment machinery. The book warns that short tests of big changes measure "primacy effects or change aversion" (p. 23)1, and Google saw iGoogle satisfaction fall after a redesign and recover3. A model that adapts changes the product continuously, and the effects of adaptation on a user may accumulate more slowly than an OEC window of one to two weeks can show. G14 is therefore a design constraint to hold even when the OEC does not move.

11.2 Offline gain is not business gain

A team building a model has a second metric the web-experiment literature does not discuss, the offline score on held-out data. The fetched evidence on how it relates to business impact is one paper: Bernardi, Mavridis and Estevez at Booking.com, KDD 2019, an analysis of "about 150 successful customer facing applications of Machine Learning" validated through randomized controlled trials13. Its Section 3 reports that "increasing the performance of a model, does not necessarily translates to a gain in value." The data are 23 comparisons (46 models) between a baseline and a new model, classifiers scored by ROC AUC and rankers by mean reciprocal rank, each compared in an online trial against one business metric. The correlation between relative offline gain and relative business gain is -0.1 (Pearson, 90% confidence interval -0.45 to 0.27) and -0.18 (Spearman, interval -0.5 to 0.19)13.

Pearson correlation -0.1 with 90 percent interval from -0.45 to 0.27 and Spearman -0.18 with interval from -0.5 to 0.19, between offline performance gain and business value gain, both intervals spanning zero -1.0 -0.5 0 0.5 1.0 correlation between offline gain and business gain, 23 comparisons (46 models) Pearson -0.10 [-0.45, 0.27] Spearman -0.18 [-0.50, 0.19]
Figure 9. Correlation between the relative offline performance gain and the relative business-metric gain of a new model over its baseline, with 90% confidence intervals (Bernardi et al., KDD 2019, §3, Figure 4)13. Both intervals contain zero and are wide, so the data fit anything from a moderate negative to a moderate positive relationship.

The authors hedge the result themselves. They write that they "do not want to overstate the generality," because the models work in one context, "all target the same business metric," and each is "trying to improve it after a previous model already did"; they note a similar weak link for machine-translation scores and human judgment, and say a correlation shows up "only where the offline metric is almost exactly the business metric"13. They list four mechanisms. Value-performance saturation: the value-versus-performance curve flattens, and gains become "impossible to detect in an RCT in reasonable time." Segment saturation: triggered analysis counts only users for whom two models disagree, and as models converge that population shrinks, cutting statistical power. Uncanny valley: users find a model that predicts too well unsettling. Proxy over-optimization: a recommender trained on click-through might "end up 'just driving clicks'," recommending near-identical hotels, which risked "drowning them into the paradox of choice and hurting conversion"13. The last is Section 1's Buy-button failure with a model as the optimizer: a better optimizer finds the gap between proxy and goal faster. Their conclusion is operational: "Offline model performance metrics are only a health check."

11.3 Latency has a measured price, in web products

A model's cost is also a product metric. The same Booking.com paper ran a four-arm experiment with synthetic latency and found "an increase of about 30% in latency costs more than 0.5% in conversion rate"13. Its design for separating a model's benefit from its slowdown is a three-group experiment. Control sees no change. A first treatment calls the model and shows its output to the users it triggers. A second calls the model and shows nothing different. Comparing the second with control prices the slowdown alone; comparing the two treatments, which carry the same computational load, prices the functionality alone (§7.2-7.3)13. Table 9 collects the latency prices in the fetched sources.

Table 9. Measured prices of latency in web and search products, as reported. None of these is an on-device or inference-runtime measurement113.
Company, yearLatency changeReported effectSource
Bing, 2012 experiment10 ms fasterPays for the fully loaded annual cost of an engineerKohavi et al. p. 18
Bing, 2015 follow-up (95th percentile under 1 s)4 ms fasterFunds an engineer for a yearKohavi et al. p. 18
Amazon (Linden 2006)100 ms slowerSales down 1%Kohavi et al. p. 18
Booking.com, 2019about 30% slowerConversion down by more than 0.5%Bernardi et al. §5

These are the closest fetched evidence for an inference engineer's usual claim that latency is a business metric. They are all web or search products, measured in an experiment with user-visible latency, and the Bing and Amazon rows are second-order citations in the book. Whether a per-token latency in a text-generating or on-device feature has a comparable price is outside the fetched sources.

11.4 What the sources do not cover

Offline-versus-online evaluation is a stated gap. The one candidate source, the Booking.com paper, covers classifiers and rankers at one company in 2019. The fetched material says nothing on evaluating generative models, on offline evaluation sets as a specification, or on guardrails for output quality. The guidelines paper itself says its list "might not adequately address all types of AI-infused systems," and that it leaves out principles that "require decisions to be made at the modeling layer of a system" (§7)12. A team shipping a model-backed feature has the OEC and guardrail structure of Sections 1 and 2, the base rates of Section 4 and the cautions above. It does not have a sourced method for choosing between an offline score and an experiment.

Where experts disagree

Three disagreements run through the fetched sources: how much evidence to wait for, whether goals are set as measurable targets or reviewed as work, and how fast a product should change under its users. None of the positions below is backed by outcome data comparing it with the others; each is a company or author describing its own practice, so the comparison is of mechanisms and costs.

Evidence or judgment: how much data before acting?

Be data-informed; formalize the OEC

Kohavi, Tang and Xu, Trustworthy Online Controlled Experiments (2020)1

The organization "is poor at assessing the value of ideas," so run controlled experiments and let data "inform the HiPPO."

  • Base rates of 10 to 33% success (Figure 2) show intuition fails often.
  • A marketing senior vice-president's objection to cart recommendations at Amazon lost to an experiment (p. 17).
  • Needs units, an agreed OEC and cheap changes (Table 3); many decisions cannot be tested.
  • The authors' companies are the best-instrumented in the industry.

Decide near 70%; resist proxies

Jeff Bezos, 2016 shareholder letter4

Wait for 90% and "you're probably being slow." Averages from surveys and betas "can become proxies for customers"; the owner "must understand the customer."

  • Prices delay, which a test-everything rule ignores (Section 8.1).
  • Names a real failure: a satisfaction average that moves without anyone knowing why.
  • Apple under Jobs did not test ads, reported secondhand in the book1.
  • No evidence offered that deciding at 70% beats waiting; the letter's one extended example is a disagree-and-commit story about a TV show.

Tie metrics to goals; triangulate

Rodden, Hutchinson and Fu, CHI 20103

Metrics "should be triangulated" with usability and field studies and are "not a substitute for early or formative user research."

  • Fits the two others: numbers evaluate, qualitative work generates.
  • Google-internal examples; no outcome data.
Assessment

The positions answer different cost regimes. Section 8.1's inequality says to test when a dud is expensive relative to a good idea's delay cost, and base rates make duds the common case, so for reversible changes in a product with cheap experiments Kohavi's rule wins. For a one-way door, a product with too few units, or an invention where no baseline exists, Bezos's 70% rule and Rodden's triangulation are what remains. The one genuine collision is the status of surveys: Kohavi lists them as one source among several; Bezos warns that an average can replace understanding. A study that assigned comparable decisions to evidence-heavy and judgment-heavy styles and tracked outcomes would settle it; none appears in the fetched sources.

Aligning teams to goals: targets, or bounded work and review?

Commit to measurable targets

Google re:Work (OKRs)5; Cagan6; Kohavi et al. (OEC)1

Teams are told the result and its measure, and choose the method: "tell the team what you need them to accomplish, and how the results will be measured." The OEC is the single-metric version, certified by replication.

  • Moves argument from features to results; "percent of plan delivered" cannot pass.
  • The grading scale budgets for misses.
  • Google's own sample key results are mostly outputs (Section 9).
  • PostHog reports that teams "wasted too much time" choosing metrics and chased "small quick wins"7.

Commit to bounded work, then review

Shape Up10; Linear9; PostHog78

Fix the time or the list of things to build, cap the downside, and judge the effect afterward: a six-week cap, "measure progress with actual work," a monthly growth review.

  • Avoids metric gaming and metric debates before the work starts.
  • Engineers choose the work.
  • "Autonomy can become directionless" without the review7.
  • Shape Up and Linear name no method for learning that a shipped bet was wrong.
Assessment

The axes are not exclusive. PostHog's own handbook asks for "what we'll ship" and a success measure in the same template8, and Kohavi's Bing relevance team commits to an OEC delta yet works through shipped experiments. A target works when a measurable proxy is trusted and gaming is covered by a guardrail; bounded work works when the team is small, senior and has a reviewer who reads the data. For an inference-runtime team, a small one by assumption, I would take bounded work with a stated outcome and one or two guardrails. That is my judgment, not a finding. A comparison of teams under each regime on shipped-effect per quarter would settle it; the sources have none.

Speed or trust: how fast should a product change under its users?

Change quickly, because most changes fail cheaply

Kohavi et al.1

Bing runs over 10,000 experiments a year; most big redesigns fail, and continuous iteration is preferred to complete redesigns that "subject users to primacy effects."

  • Cheap tests find the rare large win.
  • Their own caution: short tests of big changes confound "change aversion."

Change cautiously, because users bear the cost

Amershi et al. (G14)12; Linear11; Cagan14; Rodden et al.3

"Update and adapt cautiously"; "if you release something and iterate too much, it can be difficult for your customers"; customers can feel "abused because of constant, erratic change."

  • Names a cost the OEC window can miss; satisfaction dipped after an iGoogle redesign before recovering.
  • No measured size for the trust cost in any fetched source.
Assessment

Cagan resolves most of it: rapid test-and-learn runs "on select groups of users and customers, enabling us to protect our general users and customers from this experimentation," and some customers opt in14. That is a statement about who is exposed, not how fast the team learns. A runtime whose behavior changes between releases has a sharper form: a change shipped to everyone cannot be a two-way door in the sense of Section 8, because users' integrations depend on it (my reading). Longitudinal data on retention after changes to an adaptive feature would settle the size of the trust cost; the sources have none.

What is changing

Metric definitions are being rewritten for traffic that is not human. PostHog's handbook lists as a Q2 2026 product-management theme "make sure our products & tracking are set up for a agent-first world," including "tracking human vs agent actions, and defining success for each," and, separately, ensuring "metrics definitions are mature enough for our scale (e.g. inconsistencies in activation & product intent)"15. A developer product whose callers are agents inherits the same problem, in my reading: a seven-day-active count (Section 6.2) stops meaning a person came back. The handbook is a living page and likely to change within a year. The fetched evidence on offline-versus-online evaluation is a 2019 paper about classifiers and rankers13, and the re:Work OKR guide is an archived 2016 record5. Nothing in the source map tests either claim for generative-model features.

References

  1. textbookKohavi, R., Tang, D. and Xu, Y. Trustworthy Online Controlled Experiments: A Practical Guide to A/B Testing. Cambridge University Press, 2020 (DOI 10.1017/9781108653985). Front matter, Preface and Chapter 1, pp. i-xviii and 3-24, from the authors' companion-site PDF: https://experimentguide.com/wp-content/uploads/TrustworthyOnlineControlledExperiments_PracticalGuideToABTesting_Chapter1.pdf. Accessed 2026-10-26. Used for: OEC definition and the one-to-two-week window (pp. 6, 12), customer lifetime value and profit (p. 12), guardrails (p. 22), tenets and necessary ingredients (pp. 10-13), success rates (p. 13), Bing headline case and base rates (pp. 3-5), strategy and the portfolio (pp. 20-24), performance experiments (pp. 17-18), Twyman's law (Preface, p. xv). Later chapters are cited only through the table of contents.
  2. paperKohavi, R., Crook, T. and Longbotham, R. Online Experimentation at Microsoft. Third Workshop on Data Mining Case Studies and Practice Prize, 2009. https://exp-platform.com/Documents/ExP_DMCaseStudies.pdf. Accessed 2026-10-26. Used for: the one-third success rate (§5), QualPro 75% figure (§5), MSN pre-roll and home-page ads OECs (§§4.2, 4.5), Buy-button case (§4.6), necessary ingredients and OEC as long-term value (§6.2), widget guessing contest (§4.1).
  3. paperRodden, K., Hutchinson, H. and Fu, X. Measuring the User Experience on a Large Scale: User-Centered Metrics for Web Applications. CHI 2010 (ACM author's version, Google Research). https://research.google.com/pubs/archive/36299.pdf. Accessed 2026-10-26. Used for: PULSE and HEART definitions, the seven-day-active trap, normalization, the Gmail, iGoogle, Maps and Finance examples, the Goals-Signals-Metrics tips, scope of 20+ products and the triangulation caveat.
  4. officialBezos, J. 2016 Letter to Shareholders. Amazon (aboutamazon.com copy; the page carries no date). https://www.aboutamazon.com/news/company-news/2016-letter-to-shareholders. Accessed 2026-10-26. Used for: the 70% information rule, two-way doors, disagree and commit, and the "Resist Proxies" passage on process, surveys and beta averages.
  5. officialGoogle re:Work. Guide: Set goals with OKRs. "Originally published 2016," preserved "as a record of our findings." https://rework.withgoogle.com/intl/en/guides/set-goals-with-okrs/. Accessed 2026-10-26 (page text read directly). Used for: Table 6, the 0.6-0.7 grading band, the stretch-goal and dependency guidance, the OKR-writing mistakes list, the sample OKRs, the Sears sentence.
  6. blogCagan, M. The Alternative to Roadmaps. Silicon Valley Product Group, September 7, 2015. https://www.svpg.com/the-alternative-to-roadmaps/. Accessed 2026-10-26. Used for: OKRs as "my current favorite," outcome rather than output, the hypothetical 30-day-to-3-hour onboarding objective, high-integrity commitments.
  7. blogHawkins, J. and Neu-ner, L. Product management is broken. Engineers can fix it. PostHog newsletter, Dec 3, 2024. https://posthog.com/newsletter/product-management-is-broken. Accessed 2026-10-26. Used for: why PostHog dropped OKRs and metric-based goals, the quarterly planning flow, the monthly growth review and its data inputs, "autonomy can become directionless."
  8. officialPostHog. Setting quarterly goals, Handbook. Undated living page. https://posthog.com/handbook/company/goal-setting.md. Accessed 2026-10-26. Used for: output-based goals, "use metrics only if they help you," the goal template, named owners, the pay-review and missed-objectives wording.
  9. officialLinear. Linear Method: Principles & Practices; Set the product direction; Set useful goals. Undated. https://linear.app/method/introduction, https://linear.app/method/product-direction, https://linear.app/method/set-useful-goals. Accessed 2026-10-26. Used for: "measure progress with actual work," initiatives and the product timeline, the goals page's 10-users-to-MRR example, and the absence of metrics and OKRs on these pages (confirmed by searching the page text).
  10. officialSinger, R. Shape Up: Stop Running in Circles and Ship Work that Matters. 37signals / Basecamp, web edition (page states only "Copyright 1999-2026 37signals LLC"). Chapters 7 and 8: https://basecamp.com/shapeup/2.1-chapter-07, https://basecamp.com/shapeup/2.2-chapter-08. Accessed 2026-10-26. Used for: the betting table's membership, "a smart bet has a cap on the downside," the pitch's "specific payout," failure read as a shaping error, and the absence of the words metric, OKR, goal and measure from both chapters.
  11. blogYu, N., Roman, S. and Largman, R. How we run projects at Linear. Linear, October 5, 2023. https://linear.app/now/how-we-run-projects-at-linear. Accessed 2026-10-26. Used for: engineers deciding readiness, "iterate too much," and the absence of a post-release effect review in the post.
  12. paperAmershi, S., Weld, D., Vorvoreanu, M., Fourney, A., Nushi, B., Collisson, P., Suh, J., Iqbal, S., Bennett, P., Inkpen, K., Teevan, J., Kikin-Gil, R. and Horvitz, E. Guidelines for Human-AI Interaction. CHI 2019 (DOI 10.1145/3290605.3300233). https://www.microsoft.com/en-us/research/wp-content/uploads/2019/01/Guidelines-for-Human-AI-Interaction-camera-ready.pdf. Accessed 2026-10-26. Used for: §1 on probabilistic behavior, §3 phases, Table 1 (18 guidelines), §5 results (785 examples; G11, G5, G6, G10, G14 findings), §7 limits.
  13. paperBernardi, L., Mavridis, T. and Estevez, P. 150 Successful Machine Learning Models: 6 Lessons Learned at Booking.com. KDD 2019, Applied Data Science Track (DOI 10.1145/3292500.3330744). Listing: https://www.kdd.org/kdd2019/accepted-papers/view/150-successful-machine-learning-models-6-lessons-learned-at-booking.com; text read from a PDF copy of the paper hosted at https://blog.kevinhu.me/2021/04/25/25-Paper-Reading-Booking.com-Experiences/bernardi2019.pdf (the ACM page returned HTTP 403). Accessed 2026-10-26. Used for: §3 and Figure 4 (23 comparisons, correlations and intervals), the four mechanisms, §5 latency experiment, §7.2-7.3 three-group design.
  14. blogCagan, M. Build To Learn FAQ. Silicon Valley Product Group, April 27, 2026. https://www.svpg.com/build-to-learn-faq/. Accessed 2026-10-26. Used for: protecting general users from rapid experimentation, "constant, erratic change," opt-in test groups.
  15. officialPostHog. Product management at PostHog (product-team page), Handbook. Living page, references Q2 2026. https://posthog.com/handbook/product/product-team.md. Accessed 2026-10-26. Used for: the Q2 2026 product-management themes on metric definitions and human-versus-agent tracking.