Analytics is the Rule
This week I had to slow down in my studies with the CXL institute because the demand for work in my company increased in a significant way. First, the COVID-19 brought more uncertainty for the decision-makers and that made at first that it will be necessary to increase the pace of implementation of campaigns, as well as the amount of work that was done in each “sprint”.
As the days went by, everyone in the company (working in an IT company), realized that our company was one of the lucky ones in keeping its operations in remote, from home, without errors in the processes or infrastructure, so the continuity was maintained. Of course, we saw a decrease in the purchase of our services, since our main market is the United States of America, a country that is currently experiencing a hard situation because of the virus and therefore companies have suffered this setback during the month of April.
After this hard week, here we are again to resume my degree in Growth Marketing with the seventh publication of my learning process. This week I continued with the A/B test program, in which emphasis was placed on the user experience and the need to have a process by which a complete analysis can be achieved from start to finish.
I finished this very interesting course and I am very excited to start with the google analytics course. And I say very excitedly because it is a tool that is used most of the day for a person who works in digital marketing, and it is necessary to be an expert in this platform if you really want to understand what you are doing. Here, some points of the week;
Which KPI to Pick
From a mature perspective, you might select a KPI in importance from top to bottom if you are a mature company.
- Potential Lifetime Value
- Revenue per user
- Transactions (at least this, if you want to focus on a more business approach)
What can be optimized?
- Customer behavior study: Start looking at what your customers want, frictions, etc..
- Get the most important insights into your customer journey
Track your website changes with several tools. Also, we can track the changes of any competitor page to see if there are major changes in the site, so we can test also. If the population is shared with them.
Behavioral metrics for website
- % Light interactions in a website
- % High interactions in a website
- % Low intention to purchase
- % High intention to purchase
What to report when we have these numbers?
- Amount of users in every cluster
- Time for users to move from a cluster to another.
Also, it is important to talk with customers' service or hear a call in order to understand what customers want, and need about our product.
Create modules asking for feedback online. Use at much as possible your current users. It could be the ones who interact with your service or product already.
What type of test can we do to evaluate our assumptions
- Five seconds test (measure users first impression)
- Question test (get users feedback)
- Click test (visualize where users click)
- Preference test (find out what users prefer)
- Navigation test (find how your users navigate in your site)
It’s important to run an A/B test in Google Optimize as similar as possible to every possibility. So, it’s recommended to create a set of pages: “Original”, “Default” and “Variant”. The original one is going to receive 0% of the total traffic, and the default and variant 50% each. So, in this way, we make sure that the original and test version is as similar as possible, so results are more accurate.
How to calculate A/B test length?
- Why do we have to take a complete week for a test?
Weekdays' behavior affects results compared to weekends. Also, evening effects compare to business hours.
- Why 1, 2, 3 to 4 weeks?
Sample dilution or not
Pace/velocity versus business cycles.
You have to take into account the amount of time that a visitor converts on your website, so you can recognize the complete effect of the experiment on a business cycle of a customer.
Difference between an SRM-Sample Ratio Mismatch, when we design a test with the same amount of visitors (50% 50% split test), and you have a 50.2% for any variation instead of 50%, there is a bug in the test. So there is a formula that enables you to find the mismatch.
Statistics Fundamentals of testing
Statistics is the way marketers can tell if an A/B test is true or false, according to data and more importantly, validates statistically any hypothesis.
Population: all potential users from a group that we want to measure.
Parameters: variable of interest that can be measured.
Sample parameter: it’s a sample of a representative group.
Population parameter: it’s the interest parameter we want to measure, from all the population.
Standard deviation: O
Mean: is the average of all points of the data (central tendency measure)
Variance: is the shape of the data and how the spread is the data.
Standard deviation: shows how much is the variability of the data.
Confidence Intervals Are the range of values that is a specific probability of the value of that parameter is confidence interval.
- Sample Size
- Confidence Level
Statistical Significance and P-Value
Quantify if a result is real.
- P-Value: the probability of obtaining the difference between a sample if it isn’t a difference.
The probability that any test of significance will reject a false null hypothesis.
Is determined by:
- Size of the effect you want to detect
- Size of the sample used
Sample size and how to calculated
Sample size variables
- Control group expected conversion rate
- Minimum relative change in conversion you want to detect (Lift)
- Confidence Level
Levels held constant for the sample size: Confidence level and Power
- Statistics Trap #1
Regression to the mean & sampling error
Sampling error: A sample has an error so there are a lot of outliers in the data points. So, there is more than one variation and the media is not accurate.
- Statistics Trap #2
Too many variants
Optimization: Hypothesis-driven! You need a process behind it.
- Low error probability: accept a low error probability (for example, 5%), increase when there are many variables tested.
Correct for yourself:
- Analysis of variance
- Unifactorial analysis
Limit the number of test variants
- No more than 3 variables adding the control variable, should be the max. (expert advice)
- Statistics Trap #3
Click Rates & Conversion Rate
Select and prioritize the main KPI before starting any test. Any metric that takes into consideration an effect on the main KPI or north metric is the ones called Macro-Conversions. There are two kinds of conversions, Macro and Micro, the one who tells you “How Much”, and the other that tells you “Why”.
HOW MUCH — Macro conversions:
WHY — Micro conversions:
- Statistics Trap #4
Frequentists Vs Bayesian test procedures
In the Bayesian procedure, we assigned a probability to a test before running an experiment. In the Frequentist, we do not assign a probability before the test.
Data & Analytics Course
What does GA answer for a company
- Who are my users
- Where are my users coming from
- What actions are my users taking
- What are the results of those actions
See you next time.