One day, while working at an agency I'd rather not name, an account executive showed me a direct mail test grid for a consumer client. I wasn't on that particular team but liked seeing how everyone's tests were structured.
As I studied it my jaw dropped. I quickly asked, "Has it run?" He said nothing had moved forward. I breathed a sigh of relief and filled him in on the fatal flaw.
They were trying three offers, but instead of running a controlled test, they went with this approach:
Offer A: Cross-Section of Lists 1-3
Offer B: Cross-Section of Lists 4-6
Offer C: Cross-Section of Lists 7-9
See the problem? By running each offer test on a different set of lists, there was absolutely no way of knowing if differences in outcomes were due to offers or lists. You could drive a truck through the hole in their scheme.
The account executive didn't seem to care, so I went to the guy in charge of the account. He smiled when I shared my observations and said I forgot one important consideration. When I asked "What's that?" he claimed different collections of lists pretty much perform identically. I asked where he picked that up, and realized he was making it up out of thin air.
We went back and forth for a while, and eventually he conceded it was possible for different sets of lists to perform differently if all other factors stayed the same. This meant we both knew we were looking at the makings of a blown test -- a waste of the client's money that could cause an inferior offer to run ... and run.
Instead of picking up the phone, admitting the problem, and offering to shoot over a new test grid, he said something I would never forget: "We don't have to tell the client."
For this and other reasons, I resigned weeks later.
What do you think drives smart people to "manage" client relationships in such stupid ways?