Summary & Learning Objective

<aside> 💡

What do we intend to do, and what are we trying to learn by running this experiment?

Concisely describe the experiment in plain language.

</aside>

<aside> 💡

A great format to use is:

We want to [Change X] for [User Group U]. We hope to improve [Metric Y], and do no harm to [Metric Z]. When this experiment concludes, we hope to have learnt [Learning objective]

</aside>

**We want to [ ] for [ ]. We hope to improve [ ], and do no harm to [ ]. When this experiment concludes, we hope to have learnt [ ]

Hypothesis

<aside> 💡

What is our hypothesis? This should be grounded in some prior data (qualitative and/or quantitative) to maintain experiment quality and avoid reliance on opinion. State both the null (H₀) and alternative (H₁) hypotheses.

</aside>

<aside> 💡

A great format to use is:

Null Hypothesis (H₀):

[Change X] will have no effect on [primary metric Y].

Alternative Hypothesis (H₁):

Because [evidence/observation], We believe that [change X], Will result in [expected direction and magnitude of change] to [primary metric Y].

</aside>

<aside> 💡

Remember that we design experiments to gather evidence against the null hypothesis (H₀), not to directly prove the alternative hypothesis (H₁). Our statistical analysis will determine whether we can reject H₀, not whether we can prove H₁ true.

</aside>

Null Hypothesis (H₀):

[Change X] will have no effect on [primary metric Y].

Alternative Hypothesis (H₁):

Because [evidence/observation], We believe that [change X], Will result in [expected direction and magnitude of change] to [primary metric Y].

<aside> 💡 Remember that great hypotheses are grounded in evidence, not plucked from thin air. Be specific with the evidence to help people understand the why behind the hypothesis. It’s good practice to link to any research collections, analytics charts or other evidence supporting the hypothesis.

</aside>

<aside> 💡 *Think about evidence in 2 parts:

The observation of the situation (e.g. 98.2 % of users visiting this screen do not click the signup CTA), and
Why we think the test experience will change the observed situation (e.g. user research suggests that the signup CTA is not noticed, and a competing CTA that is more prominent is clicked by 16.4% of users)*

</aside>

<aside> ☑️

Example:

Null Hypothesis (H₀):

Changing the location of the signup CTA will have no effect on signup conversion rate.

Alternative Hypothesis (H₁):

Because 98.2% of visitors to the homepage do not click on the CTA to sign up, and a competing CTA to book a demo is clicked by 16.4% of users, and recent observational studies suggest the signup CTA is rarely noticed,

We believe that changing the location of the signup CTA to make it more prominently visible,

Will result in more visitors seeing and interacting with the CTA, increasing signup conversion rate from 1.8% to 10%.

</aside>