If you’re like most PPC managers, you’re somewhat … neurotic about account monitoring. While this is a great thing for overall performance, it can wreak havoc on your peace of mind. Every little fluctuation, every minor deviation from the running average can be a source of stress.
Even more disturbing to those that watch their data like a hawk, is the emergence of trends that seem to signal an imminent collapse in conversions. This is made all the worse by the tendency of the human brain to seek patterns in data – where none may even exist. It’s critical to temper these observations with a healthy dose of statistical knowledge.
In this article I’m going to show you exactly how to deal with this problem. It won’t be a rigorous statistical treatment, but I can give you some hard rules of thumb to know when it’s time to panic, and when it’s time to let it ride. So you know what you’re getting into, it’s about 1,200 words with a few pictures thrown in for good measure.
Consider the plot below. This is from one of my accounts under management, and as a neurotic watcher of data, I must admit I became concerned. At the start of the boxed in time period, the conversions for the day are in line with the historical moving average. Business as usual.
As the next few days unfold, a worrisome trend develops. Day after day, the conversions inch lower. I start to sweat.
Aware of my own short comings as a byproduct of evolution, I contain my panic and hold on for the ride. As the days go by and the trend continues, my heart starts to sink. I contact the client to let him know of the trend and that I’m watching it.
It finally bottoms out over the weekend, at 15 conversions. By this time, I’ve checked everything I can think of. As far as I can tell, there has been no unusual downtime for the site (though I can’t be sure, I’m not a developer) and no ads point to 404 pages. In other words, nothing systematically wrong with the account to explain the trend.
Nevertheless, I start formulating a plan of action and decide to give it another day.
To my elation, the next dawn brings light. The conversions pop back in line with the historical average, and business returns to normal. Yet, I’m not quite satisfied. Why had my innate tendency to see patterns in data lead me astray? What could I do to prevent needless worry in the future?
Fortunately, I’ve settled on a solution. As I said, it’s not statistically rigorous, but it’s a strong rule that can be implemented in your own account watching.
You see, buyer behavior, from our point of view, is inherently random. Obviously, it’s not truly random, but in the absence of systematic drivers of conversions (site downtime, holiday sales, addition of new campaigns), it is random enough to make some statistical generalizations. To account for the fact that people structure their lives around their work week, I look at conversions on by-day basis.
In statistics, there is the bell curve. Many random systems have variables whose distribution makes a bell curve. This is powerful because it allows us to make some intelligent statements about more or less random data. Of course, the conversions in our account are not strictly random, and won’t exactly follow a bell curve, rather it will be skewed (appear to lean) to one side or another (more or less conversions). Not a huge deal, just know that there are some limitations to my solution.
In the bell curve, the width of the distribution is characterized by the standard deviation. This is just a measure of how much variance there is in the data, and it’s the key to helping you sleep better at night. The larger the standard deviation, the wider the distribution. More importantly, the larger the standard deviation, the more likely it is that a drop (or increase) in conversions is due to chance.
In general, 66% of the data will fall within plus or minus 1 standard deviation from the average.
95% will fall within plus or minus 2 standard deviations of the average.
99.5% will fall within plus or minus 3 standard deviations from the average.
The rule of thumb I have employed is that if a data point falls within 1 standard deviation of the average, I don’t worry. If they start to fall outside of that range, then I start looking for issues that can be the root cause of the trend.
Moral of the story: ignore the trends and pay attention to average and standard deviation for that day of the week. If sales are within 1 standard deviation of the average for that day of the week, let it ride. If you have a day that is 2 or more standard deviations from the average, it’s time to worry.
Now that you know the what, here’s the how:
1) Download 6 months of daily data from the AdWords dashboard. You’ll need it blocked by day. Make sure you include conversions.
2) Open the data in an Excel spreadsheet. Manually insert the day of the week into a new column. Just start with the first day and double click on the bottom right of the cell to auto fill the whole column.
3) Highlight all the data and insert a Pivot Chart into a new worksheet.
4) Select day of the week as the Axes category and sum / standard deviation of the conversions as the values
You should end up with a plot like this:
Looking at the plot we can see right away that even the low sales on Thursday from our first plot (15 conversions that day) are within 1 standard deviation of the average (27.4 plus or minus 13). This lends credence to the idea that this “trend” was nothing more than an anomaly of day to day variation.
One last technical point: the issue of the bell curve. For this particular account, if you create a pivot chart with the conversions as the axes category, and the count of conversions as the value, you get something that is not at all a bell curve. In fact it is bi-modal with a second distribution around the lower conversions. If you squint, you can even convince yourself there are three modes. This gives even more reason to not worry so much about a dip in conversions, as low conversion days are not uncommon. They are more common than high conversion days, in fact.
Here’s the plot:
In conclusion, when you see a trend in your data don’t panic. Compute the averages and standard deviations for the previous 6 months by day of the week. Check to make sure the days in question are within 1 standard deviation of your average. If they are, you’re good to go. If not, it’s time to panic.
I hope this has been helpful. If you have any questions, please feel free to contact me.
If you’d like to see how this is done in practice, you can check out the link below. I have a YouTube video where I walk you through the process.