Analytics is the Rule

This week I had to slow down in my studies with the CXL institute because the demand for work in my company increased in a significant way. First, the COVID-19 brought more uncertainty for the decision-makers and that made at first that it will be necessary to increase the pace of implementation of campaigns, as well as the amount of work that was done in each “sprint”.

As the days went by, everyone in the company (working in an IT company), realized that our company was one of the lucky ones in keeping its operations in remote, from home, without errors in the processes or infrastructure, so the continuity was maintained. Of course, we saw a decrease in the purchase of our services, since our main market is the United States of America, a country that is currently experiencing a hard situation because of the virus and therefore companies have suffered this setback during the month of April.

After this hard week, here we are again to resume my degree in Growth Marketing with the seventh publication of my learning process. This week I continued with the A/B test program, in which emphasis was placed on the user experience and the need to have a process by which a complete analysis can be achieved from start to finish.

I finished this very interesting course and I am very excited to start with the google analytics course. And I say very excitedly because it is a tool that is used most of the day for a person who works in digital marketing, and it is necessary to be an expert in this platform if you really want to understand what you are doing. Here, some points of the week;

Which KPI to Pick

From a mature perspective, you might select a KPI in importance from top to bottom if you are a mature company.

• Potential Lifetime Value
• Revenue per user
• Transactions (at least this, if you want to focus on a more business approach)
• Behavior
• Clicks

What can be optimized?

• Customer behavior study: Start looking at what your customers want, frictions, etc..
• Get the most important insights into your customer journey

Track your website changes with several tools. Also, we can track the changes of any competitor page to see if there are major changes in the site, so we can test also. If the population is shared with them.

Behavioral metrics for website

1. % Light interactions in a website
2. % High interactions in a website
3. % Low intention to purchase
4. % High intention to purchase

What to report when we have these numbers?

1. Amount of users in every cluster
2. Time for users to move from a cluster to another.

Also, it is important to talk with customers' service or hear a call in order to understand what customers want, and need about our product.

Create modules asking for feedback online. Use at much as possible your current users. It could be the ones who interact with your service or product already.

What type of test can we do to evaluate our assumptions

1. Five seconds test (measure users first impression)
2. Question test (get users feedback)
3. Click test (visualize where users click)
4. Preference test (find out what users prefer)
5. Navigation test (find how your users navigate in your site)

Google Optimize

It’s important to run an A/B test in Google Optimize as similar as possible to every possibility. So, it’s recommended to create a set of pages: “Original”, “Default” and “Variant”. The original one is going to receive 0% of the total traffic, and the default and variant 50% each. So, in this way, we make sure that the original and test version is as similar as possible, so results are more accurate.

How to calculate A/B test length?

• Why do we have to take a complete week for a test?

Weekdays' behavior affects results compared to weekends. Also, evening effects compare to business hours.

• Why 1, 2, 3 to 4 weeks?

Sample dilution or not

Pace/velocity versus business cycles.

You have to take into account the amount of time that a visitor converts on your website, so you can recognize the complete effect of the experiment on a business cycle of a customer.

Difference between an SRM-Sample Ratio Mismatch, when we design a test with the same amount of visitors (50% 50% split test), and you have a 50.2% for any variation instead of 50%, there is a bug in the test. So there is a formula that enables you to find the mismatch.

Statistics Fundamentals of testing

Statistics is the way marketers can tell if an A/B test is true or false, according to data and more importantly, validates statistically any hypothesis.

Population: all potential users from a group that we want to measure.

Parameters: variable of interest that can be measured.

Sample parameter: it’s a sample of a representative group.

Population parameter: it’s the interest parameter we want to measure, from all the population.

Population Parameters

Mean: U

Standard deviation: O

Sample Parameters

Mean: x

Variance: S

Mean: is the average of all points of the data (central tendency measure)

Variance: is the shape of the data and how the spread is the data.

Standard deviation: shows how much is the variability of the data.

Confidence Intervals Are the range of values that is a specific probability of the value of that parameter is confidence interval.

• Mean
• Sample Size
• Variability
• Confidence Level

Statistical Significance and P-Value

Quantify if a result is real.

• P-Value: the probability of obtaining the difference between a sample if it isn’t a difference.

Statistical Power

The probability that any test of significance will reject a false null hypothesis.

Is determined by:

• Size of the effect you want to detect
• Size of the sample used

Sample size and how to calculated

Sample size variables

• Control group expected conversion rate
• Minimum relative change in conversion you want to detect (Lift)
• Confidence Level

Levels held constant for the sample size: Confidence level and Power

• Statistics Trap #1

Regression to the mean & sampling error

Sampling error: A sample has an error so there are a lot of outliers in the data points. So, there is more than one variation and the media is not accurate.

• Statistics Trap #2

Too many variants

Optimization: Hypothesis-driven! You need a process behind it.

• Low error probability: accept a low error probability (for example, 5%), increase when there are many variables tested.

Correct for yourself:

• Analysis of variance
• Unifactorial analysis

Limit the number of test variants

• No more than 3 variables adding the control variable, should be the max. (expert advice)
• Statistics Trap #3

Click Rates & Conversion Rate

Select and prioritize the main KPI before starting any test. Any metric that takes into consideration an effect on the main KPI or north metric is the ones called Macro-Conversions. There are two kinds of conversions, Macro and Micro, the one who tells you “How Much”, and the other that tells you “Why”.

HOW MUCH — Macro conversions:

• Conversions
• Orders
• Revenue
• Profit
• Returns

WHY — Micro conversions:

• Clicks
• Visits
• Views
• Scrolls
• Bounces
• Statistics Trap #4

Frequentists Vs Bayesian test procedures

In the Bayesian procedure, we assigned a probability to a test before running an experiment. In the Frequentist, we do not assign a probability before the test.

Data & Analytics Course

What does GA answer for a company

1. Who are my users
2. Where are my users coming from
3. What actions are my users taking
4. What are the results of those actions

See you next time.