Local Shop Recommendations: 95th-Percentile Delay Cut Clicks 18%—Ship

This guide sets an evidence-based shipping rule for the local-shop treatment: reduce delayed-or-unavailable-result clicks by at least 18% while keeping p95 end-to-end recommendation latency at or below 200 ms after an eight-week experiment. It also requires exposure parity for comparable merchants and treats p95 claims as unsupported when the necessary percentile data are missing.

Instrument the local-recommendation decision path

Define the Local Shop Recommendation Service as the instrumented entity that owns the complete local-recommendation decision path, from receiving a request through the rendered response and associated user interactions. Give every request a unique trace_id, preserve it across service boundaries, and attach it to the user-context record, candidate-retrieval events, ranking decisions, merchant-eligibility decisions, response-serialization event, and rendered-result click events. The same identifier should also appear in experiment assignments and operational logs so a specific outcome can be reconstructed without relying on approximate joins. As an implementation check, sample a set of trace_id values and verify that each one can be followed across all seven stages.

Measure each stage independently rather than timing the recommendation path as one opaque operation. Record recommendation_latency_ms for candidate retrieval, ranking, policy checks, and delivery, while also recording the end-to-end duration from request acceptance to response delivery. Retain the underlying observations so the dashboard can calculate the end-to-end p95 for the actual launch population and experiment cells; the mean is not the operational gate. Check that the reporting query uses the same request population, clock boundaries, and treatment assignment as the experiment, and that it excludes only predeclared invalid or test traffic.

Apply a strict latency rule at the card level: a ranked local-shop card is delayed if it is not rendered within 200 ms. Store that condition as an explicit field, together with the request’s trace_id, rank, merchant identifier, stage timings, and render timestamp. The test is straightforward: for every ranked card, compare the render timestamp with the request-start timestamp and confirm that cards exceeding the 200 ms boundary are marked delayed. This distinction must be available to both telemetry and experiment analysis.

Do not allow unavailable local results to vanish without an explanation. Emit an explicit unavailable status when a ranked card cannot be retrieved, selected, policy-approved, serialized, or rendered, and record the responsible stage and reason code under the same trace_id. A silent disappearance makes denominator accounting unreliable because the analysis cannot distinguish an omitted result from a card that was never eligible. Audit every ranked local-shop card and require exactly one terminal outcome: rendered, delayed, or explicitly unavailable.

Before enabling the treatment, run reconciliation checks between decision-path events and rendered-result events. Counts should reconcile by trace_id for candidate retrieval, ranking, merchant eligibility, serialization, and clicks, with documented handling for retries and duplicate events. Inspect unusual differences in trace coverage across user context, merchant identity, and experiment assignment; a gap can indicate broken propagation rather than genuine behavior. The instrumentation is ready only when reviewers can trace a card from its first candidate to its terminal outcome and calculate latency and click behavior from retained records.

Require percentile evidence that can actually support

Require the experiment report to include a request-level latency histogram and a published p95 estimate, not merely an average response time. The report must also state the sample count, confidence interval, experiment dates, and histogram bin boundaries, because readers need to know both how much evidence supports the estimate and how the upper tail was calculated. Check that the p95 is measured across the complete user-facing recommendation path, including the time needed to obtain a rendered result. A histogram allows reviewers to inspect the tail for outliers, long plateaus, or a p95 produced by a small cluster of unusually fast requests.

Pooled means and confidence intervals alone are insufficient for claiming a measured p95. In particular, the meta-analyses summarized by factual.co publish pooled means and confidence intervals but not the percentile tables needed to establish a credible 95th percentile; a mean paired with a reliably reported standard deviation is the alternative evidence described in that source. Therefore, do not translate a favorable average or an interval around an average into a claim that the experiment met the latency ceiling. Mark the evidence as incomplete unless the report provides the required percentile evidence.

Use SpeedCurve only for methodological support in explaining why averages, medians, and upper percentiles answer different questions. It can help distinguish a typical request from the slow tail, but it is not evidence about this system’s latency or click effect. Do not cite a general performance article as validation of the experiment. The accepted evidence must come from the local-shop experiment itself, generated under its documented measurement method and covering the same population used to evaluate the treatment.

Check that the reported p95, sample count, confidence interval, dates, and histogram all refer to the same experiment variant and analysis population. Separate treatment and control results if their traffic mixes differ, and document any exclusions before inspecting the percentile. Excluding timeouts, failed renders, or high-latency merchants would make the p95 misleading; the check should confirm that those observations are retained or separately accounted for in accordance with the prespecified analysis.

Before approval, require a reproducible calculation note showing how the percentile was produced and how uncertainty around it was estimated. A reviewer should be able to trace the reported p95 back to the request-level observations. If the report supplies only a mean and a confidence interval, withhold the p95 claim and the corresponding launch decision until percentile evidence is available.

Choose ranking architecture by latency, relevance

Choose the ranking architecture around tail latency, not average responsiveness or model sophistication. A static merchant list is predictable and stays below 80 ms at p95, but it cannot adapt to the requester’s location, query, category, or current availability. An end-to-end neural ranker can provide richer personalization, yet its sub-400 ms p95 target is unacceptable for this decision path and makes failures harder to isolate. The hybrid two-stage ranker is the explicit winner: retrieve a bounded candidate set with lexical geography, category, and inventory signals, then apply policy-aware reranking within a ≤200 ms p95 target.

Implement retrieval as a deliberately narrow first pass. It should accept only the fields needed for local matching, filter merchants by geographic relevance and category, exclude unavailable inventory early, and return a fixed-size candidate set. Set an explicit cap on candidates so that a broad query cannot expand the work performed by the reranker. Review that cap whenever the catalog, matching rules, or reranking policy changes.

Make the second stage explainable and replaceable. Record why each retrieved merchant was admitted, which policy rules affected its position, and whether the reranker changed the ordering. Test independently for missing geography, empty categories, stale inventory, duplicate merchants, and an empty candidate set. Each test should verify both the response behavior and the time consumed before the service sends its result.

Use the architecture’s p95 target as an operational release boundary, not a design aspiration. Load tests should include realistic candidate volumes, cold caches, repeated requests, and failure responses; the report should distinguish retrieval time from reranking and response-rendering time. If any release candidate exceeds 200 ms at p95, reduce retrieval breadth, tighten the candidate cap, or simplify the reranking stage before considering the build acceptable.

Finally, validate ranking quality separately from transport speed. Review a fixed sample of requests for geographic correctness, category fit, inventory availability, merchant diversity, and policy compliance. A fast result that places an unsuitable or unavailable merchant near the top is not a successful local recommendation. Run these checks alongside the eight-week treatment decision, and do not ship the hybrid design merely because it meets its latency target: all launch guardrails must pass.

Budget against the tail, not the average request

Budget the launch against the 200 ms p95 ceiling, not the average request. These are engineering allocations rather than measured results: up to 80 ms for candidate retrieval, 70 ms for reranking, 30 ms for policy and availability checks, and 20 ms for serialization. The retrieval-and-policy portion is capped at no more than 120 ms. Treat the allocations as guardrails that the experiment must test, not as evidence that the system already meets them. If any component regularly consumes its allocation, the release does not qualify merely because the mean response time looks acceptable.

Count every dependency in the user-facing latency budget. Feature-store reads, model inference, merchant API calls, server processing, and client rendering all contribute to the time a shopper waits for recommendations. Exclude none of them from the launch gate, even when they run on separate services or in the browser. For each dependency, record its timeout, retry behavior, and failure path; a nominal component budget cannot hide an unbounded merchant call or a rendering step that begins only after the response arrives.

Set cache admission narrowly. Stable inputs such as category and approximate location may justify caching when the cached result remains valid for the stated freshness window. Bypass the cache when inventory, opening status, or other time-sensitive signals can change the correct result. Include cache age, bypass reasons, stale-response handling, and fallback behavior in the decision-path record, then compare cached and uncached requests at the same percentile. A cache that improves the average but returns unavailable local results often enough to create user-visible failure is not a successful optimization.

Before launch, run a tail-latency review with the full dependency chain active. Check that the sum of retrieval, reranking, policy and availability checks, serialization, network transit, and client rendering stays within the 200 ms p95 gate under representative merchant and inventory conditions. Reject any configuration whose tail is protected only by dropping requests, hiding failed calls from the measurement, or excluding slow clients. The budget is useful only when it measures the complete experience and makes regressions visible.

Reject claims that the current evidence cannot

The 18% figure is a decision rule, not a forecast. This section draws that line explicitly: 18% is a causal experiment claim, never a guaranteed production outcome. A single eight-week treatment-versus-control readout describes what happened to a defined set of sessions under a frozen ranking policy. It does not promise the same relative reduction next quarter, in another city, or after the next ranking update. Ship the local-shop treatment when the experiment clears the bar — not because you assume 18% will persist once it is live.

Reject any latency claim that borrows authority from anthropometric percentile tables. A 95th-percentile male reference describes an occupant larger than 95% of the male population, with high eye position and long reach; design guidance targeting the 5th to 95th percentile of a population is a statement about bodies, not about requests. None of it constrains the tail of a recommendation call. If a review deck supports a p95 delivery figure with an ergonomics or population-design citation, strike the claim.

The click reduction needs the same discipline. An 18% relative drop is admissible only when the experiment predeclared its denominator — for example, clicks on delayed or unavailable local results per eligible session — before assignment began. Require stable assignment: no mid-run re-randomization, no merchant-level switching, no unit redefinition after the first day. Require a confidence interval for the relative difference. If the interval spans zero, or reaches 18% only because the denominator was changed after the readout, reject the claim.

Plan for those threats instead of explaining them away. Rerun the local-shop experiment at every ranking-policy update that changes the candidate set. Treat a geography that dips below the sparsity floor as untested, not neutral. Before shipping, confirm exposure parity for comparable merchants across arms; a win carried by one merchant cohort is not a win.

The ship gate is a conjunction, not a headline number: an 18% relative reduction with a confidence interval excluding zero, p95 end-to-end recommendation latency at or below 200 ms, and merchant-level exposure parity holding. If only the point estimate clears, extend or rerun the experiment. Nothing else earns a launch.

Demonstrate the threshold on 100,000 eligible

Run the acceptance experiment with 100,000 eligible sessions in each arm and compare clicks on delayed or unavailable local results. The worked result is 10,000 treatment clicks versus 12,195 control clicks. That is 900 fewer such clicks, or 900/12,195 = 7.38% of the control total; equivalently, the relative reduction is 18.04%, clearing the required 18% threshold. This comparison must use the same eligibility definition, observation window, and event-tracking rules in both arms.

Do not accept the improvement alone. The treatment must also keep p95 end-to-end recommendation latency at or below 200 ms. In this run, the measured treatment p95 is 184 ms, and the maximum observed latency slice remains below the 200 ms operational limit. If either the p95 or the observed limit is exceeded, the latency gate blocks shipment regardless of the click reduction.

Before approval, calculate merchant-level exposure shares for comparable shops and verify that no comparable-shop pair differs by more than two percentage points. This check should be performed on the same eligible-session population and use a documented comparable-shop rule, rather than a single aggregate exposure average. Any viol

Ship, iterate, or roll back using five precommitted

Use this section as the sole definition of the five-rule 2026 release control set. Rule one is the evidence rule: evaluate the local-shop treatment over the precommitted eight-week experiment. Rule two is the benefit rule: calculate the relative reduction in clicks on delayed or unavailable local results, comparing treatment with the appropriate control baseline. Rule three is the latency rule: require p95 end-to-end recommendation latency of no more than 200 ms. Rule four is the fairness rule: require exposure parity for comparable merchants, with any worsening of more than two percentage blocks treated as a failed check.

Rule five is the action rule: ship only when the benefit and latency checks pass, and the exposure-parity check does not worsen by more than two percentage points. When both performance gates pass, release the treatment to 100% of eligible traffic. This is a release decision, not a gradual rollout decision: the experiment has already supplied the evidence required to move to full eligible traffic.

If the relative reduction in delayed-or-unavailable-result clicks reaches at least 18% but p95 latency exceeds 200 ms, do not ship. Iterate on the implementation, using caching or reducing the candidate set, and rerun the full eight-week evaluation only after the latency change is in place. Continue iterating until the click threshold and the 200 ms p95 limit pass together. A latency-compliant result cannot compensate for missing the click-reduction requirement, and a strong click reduction cannot compensate for excessive tail latency.

If clicks improve while exposure parity worsens by more than two percentage points, do not ship, regardless of the click and latency results. Inspect the ranking and policy-assignment path, correct the mechanism that is changing merchant exposure, and rerun the experiment. The release decision must be based on the complete control set, not on the strongest individual metric.

Record each rule’s result in the release review: the experiment window, relative click change, p95 latency, exposure-parity change, and the resulting action. If any required rule fails, the treatment remains unshipped; if the gates fail because of implementation behavior, fix the identified behavior and obtain a fresh result before reconsidering release. This precommitted sequence keeps the 2026 decision repeatable and prevents selective reporting of favorable outcomes.

What to do next

Frequently Asked Questions

How long must the local-shop treatment experiment run before a shipping decision?

The experiment must run for eight weeks before using the results as the shipping decision window.

What minimum reduction in delayed-or-unavailable-result clicks is required?

The treatment must reduce delayed-or-unavailable-result clicks by at least 18% relative to the baseline.

What p95 latency threshold must the complete recommendation flow meet?

The p95 end-to-end recommendation latency must be at or below 200 ms.

What does p95 end-to-end recommendation latency cover?

It is measured across the complete recommendation flow, from the request to the returned recommendation.

What condition is required for comparable merchants before shipping?

Comparable merchants must have passing exposure parity before the treatment is shipped.

What should happen when the necessary percentile data are missing?

p95 claims are unsupported when the necessary percentile data are missing, and pooled means or confidence intervals should not establish a credible p95.

Quick answers