Some time ago, I was working with an SEO professional who was helping my client with some A/B testing of a website (testing two different versions to see which performs better).
After just a week, they came back with some good news. They tested a new version they had developed and indeed, there was a small, positive bump in sales. I asked what I thought was a logical question, “How do we know with statistical certainty that your change made the difference and that this didn’t happen by chance alone, or some confounding factor like advertising or seasonality?”
They looked at me with blank stares.
“You don’t understand,” the SEO professional said. “We’ve been doing this a long time. We KNOW there was a change.”
Real marketing data analysis
I hired a statistician to put the data through the appropriate test and learned that there was an 80 percent chance they were wrong and that the small increase was accounted for through normal variability in the data. I told the SEO I wouldn’t pay them until they started providing statistically valid analysis.
The sad thing is, this is the way it is in most places.
