Designing a Virtual Try-On pilot that supports a decision

A virtual try-on can be approved for the accuracy of its frames, prove usable on the intended devices and still leave its effect on sales uncertain. These outcomes do not contradict each other, because they concern different questions. A pilot project creates value when it makes explicit which of these questions it can answer, with what evidence, and for which subsequent investment decision.

In eyewear, the test involves at least three things: the 3D model of the product, the service that makes it available and the buying behaviour of the audience. Folding them into a single success indicator makes it hard to tell what to extend and what to fix.

Designing the pilot therefore starts with the decision: approving a first batch of assets, making the service available in a market, investing in digitising more collections or integrating new channels. The evidence required must be proportionate to that decision.

Separate the checks that need different evidence

Product validation concerns the match between the physical frame and its digital representation. It covers proportions, geometry, materials, lenses and variants. It needs approved references and people qualified to make the judgement. A high number of try-ons does not fix an inaccurate asset.

Operational validation concerns the experience under the intended conditions: devices, browsers, network, camera permissions and integration into the shopping flow. It needs observation of loading, errors, timings and the handling of updates. A successful demo on a recent phone covers only part of this question.

Commercial evaluation concerns an outcome such as purchases, bookings or qualified enquiries. It needs a measure of the result and a comparison able to separate the effect of the intervention from initial differences between audiences.

These checks can proceed in stages. Asset quality and problems that prevent use need to be addressed before interpreting buying behaviour. A successful operational test can then be extended while data on the economic impact continue to be collected, provided the decision acknowledges what remains uncertain.

Define the population the result must apply to

A sample of frames made up only of products that are easy to render can be useful for initial fine-tuning. To assess a whole collection, you need to include the differences that affect rendering: translucent acetates, thin metals, oversized shapes, clip-ons and lenses with different tints or mirrored coatings, if the range includes them.

It helps to keep the set of frames chosen to test the difficult cases separate from the one intended for commercial measurement. The first must bring out the limitations. The second must have the availability, traffic and relevance the business question calls for. The two sets can overlap, but they are chosen on different criteria.

The same applies to devices. A technical exclusion narrows the population the result applies to. If the service is assessed only on supported, higher-performing browsers, the conclusion must stay within that scope; deciding to extend it to all traffic also means considering how much of the audience is left out.

Before the test starts, eligibility also needs defining: which pages, products, markets and visitors can enter the test. The rule should be observable in the same way for both groups, so that access to the feature does not on its own determine who is included in the analysis.

Agree on the measure before looking at the result

The pilot protocol can be short, but it must set out the primary goal, its denominator and the window within which it is observed. ‘Conversion’ could mean an order in the same session or a purchase within a later period: the two measures can produce different results.

Alongside the primary goal, you need measures to interpret it. For Virtual Try-On (VTO), these can include loading reliability, time to the first usable try-on, variant coverage and issues by device. Quality thresholds on the standard shopping flow are also useful: the shop must remain usable for people who do not open the feature.

Microsoft Research suggests defining, before the test, a testable hypothesis, success metrics and guardrail metrics, together with the sample size needed to detect the expected change.¹ In an eyewear project, this also means stating the size of improvement that would justify a wider rollout.

The primary metric must be kept separate from exploratory analysis. Searching after the fact for a combination of device, market and model that shows a favourable result can generate a hypothesis for a new test; treating it as confirmation requires a design suited to the comparisons made.

Compare groups that can answer the causal question

People who choose of their own accord to open the try-on may be more interested in buying, more willing to use the camera or more careful about comparing frames. Their conversion describes those people and the steps they took; comparing them with people who do not open the try-on does not isolate the effect of the feature.

Where the context allows, random assignment to an experience with or without VTO makes it possible to build a more suitable comparison. The unit of assignment must stay consistent with the behaviour being observed. If someone returns several times and sees both experiences, the test must be interpreted in the light of that contamination; persistent assignment has to be designed within the limits of the data and tools available.

In the main analysis, it is useful to keep subjects in the group they were assigned to, including those who do not use the try-on. The service is, after all, offered to an audience, part of which decides not to open it. An analysis limited to users would once again select people on the basis of behaviour that occurred after assignment. It is the same principle that the field’s standard handbook applies to triggered analyses (exposed populations) and to guardrail metrics.²

Narrower populations can be studied, for example people who reach an eligible product page, when the condition is defined correctly for both groups. Microsoft describes this precaution for triggered analyses: the rule must also recognise those who would have encountered the feature in the control group, and must not exclude people affected by the change.³

A before-and-after comparison is still useful for monitoring operations, but prices, promotions, seasonality and traffic mix can change at the same time as the feature. In those cases, attributing the effect requires further assumptions, which the decision must take into account. Observational designs with control groups exist and have been used to estimate the effect of AR on a retailer’s sales, with an increase concentrated on less popular products and on new customers;⁴ with these designs, a pilot without random assignment can still be informative, but the weight shifts onto the assumptions, which must be stated.

Check that the measurement system does not distort the comparison

Before reading any uplift, you need to check how the data arrived. If a 50/50 split was planned, an unexpected and statistically significant imbalance between the observed groups can signal problems with assignment, eligibility or data collection.

This check, known as Sample Ratio Mismatch, is part of the method Microsoft uses to verify the validity of its tests.⁵ A small random deviation is normal; a significant anomaly requires tracing the cause before attributing any movement to the feature.

With VTO, a real risk is starting to log the try-on group only after the engine has loaded successfully. Anyone who hits an error disappears from the analysis, while the others remain. The report ends up describing a subset selected by the technical success of the experience.

Definitions need checking too. An event called ‘try-on started’ can mean the click on the button, the camera being available or the frame already rendered. If the meaning changes during the test, the series is no longer consistent. Likewise, numerators and denominators from different platforms need their respective windows and counting units checked.

Interpret the uncertainty, not just the percentage uplift

Consider a hypothetical fixed-duration test, with random assignment and 10,000 independent visitors per group. Each visitor counts only once in the measure and has the same window in which to buy.

The observed uplift is 10% in relative terms, an absolute difference of 0.20 percentage points. Using a normal approximation for the difference between two independent proportions, the 95% confidence interval runs from about −0.20 to +0.60 percentage points.⁶

The interval includes both negative differences and positive improvements. The data are compatible with a range of effects wide enough to leave the decision on the commercial uplift open. The central value of 10% does not, on its own, express how precise the estimate is.

A relative +10% on a small number of orders is still compatible with a null effect.

The example shows why the length of the pilot cannot depend only on the project schedule. What matters is eligible traffic, how often the outcome occurs and the size of the effect you want to detect. In the example, telling a 10% relative uplift apart from a null effect with reasonable confidence would take about 80,000 eligible visitors per group, not 10,000.⁷ For a shop with few orders, the test can give a clear picture of operational reliability without reaching the precision needed for a conclusion about sales.

The stopping rule must be agreed too. If the results are checked every day and the test may stop at the first favourable outcome, the analysis requires methods that account for repeated looks at the data: classical p-values and intervals are not reliable when the sample size is decided by watching the data as they come in, and there are sequential procedures designed for this.⁸ Microsoft flags this problem explicitly and, in the absence of appropriate methods for interim analyses, recommends sticking to the predetermined duration.⁹

Finally, for returns, the cohort needs time to mature: orders must be delivered and pass through the period in which a return can occur. Comparing recent purchases with others already observed for weeks introduces a difference in the opportunity to return.

Link the result to the next investment

The decision to extend the service takes in the cost of integration, the production and maintenance of assets, usage, approval work and the financial contribution of any additional orders. Incremental revenue and the margin available to cover these costs are different quantities.

The effect observed on the first range should not be transferred automatically to new products or markets. A collection with different materials may need further checks; a channel with lower-intent traffic may have a different usage rate. Extending the service is also a decision about the transferability of the test conditions.

The final result should therefore specify what has been approved, the scope it applies to, the remaining uncertainty and the work needed for the next step. One outcome may support going live in a market while still requiring observation of the financial return. Another may suggest correcting the 3D model before widening the catalogue.

On the ARShades Enterprise path, Studio lets you track catalogues, validation and the use of experiences; commercial outcomes require a connection to the systems that record orders or enquiries. The design of the comparison and the decision remain the responsibility of a project shared by the product, e-commerce and analytics teams. The quality of a pilot lies in the match between the question asked, the evidence gathered and the investment that evidence is meant to support.

Sources

1. Machmouchi, W., Gupta, S., Zhang, R. and Fabijan, A. (2020), Patterns of Trustworthy Experimentation — Pre-Experiment Stage, Microsoft Research, 31 July 2020. ↩

2. Kohavi, R., Tang, D. and Xu, Y. (2020), Trustworthy Online Controlled Experiments — A Practical Guide to A/B Testing, Cambridge University Press; in particular the chapters on triggered analyses and on Sample Ratio Mismatch. ↩

3. Microsoft Research, Experimentation Platform (2020), Patterns of Trustworthy Experimentation — Post-Experiment Stage, triggered analyses and inclusion criteria. ↩

4. Tan, Y.-C., Chandukala, S. R. and Reddy, S. K. (2022), Augmented Reality in Retail and Its Impact on Sales, Journal of Marketing, 86(1), 48–66. ↩

5. Fabijan, A. et al. (2020), Diagnosing Sample Ratio Mismatch in A/B Testing, Microsoft Research, 14 September 2020. ↩

6. Example constructed for illustration. Difference: p₁ − p₀. Approximate standard error: √[p₀(1 − p₀)/n₀ + p₁(1 − p₁)/n₁]. Interval: difference ± 1.96 × standard error. The calculation assumes independent observations and a fixed-horizon comparison; it does not include adjustments for repeated looks or multiple comparisons. ↩

7. Illustrative calculation for comparing two proportions (2.00% against 2.20%), two-sided test at the 5% level with 80% power: about 80,700 visitors per group. The same caveats about the example apply as in the previous note. ↩

8. Johari, R., Pekelis, L. and Walsh, D. J., Always Valid Inference — Bringing Sequential Analysis to A/B Testing, arXiv:1512.04922. ↩

9. Microsoft Research, Experimentation Platform (2020), Patterns of Trustworthy Experimentation — During-Experiment Stage. ↩