Why Most Marketers Will Fail In The Era Of Big Data

Galileo once said that "All truths are easy to understand once they are discovered; the point is to discover them."  Great marketers uncover those obvious, but unexpected truths to win consumers hearts and minds.
Marketing is, after all, the art and science of discovering what people want and offering it to them.  The right product, combined with the right message, can achieve wonders.  It can transform even a mediocre enterprise into a dominant market leader.
Historically, marketers operated by instinct. They would see an opening and pounce.  Over the years, numbers crept in—quantitative research, focus groups and analysis became basic tools of the trade. The next great shift, however, will be automation and it will require not just new skills, but a new perspective and that will be a much harder bridge to cross.
» The Magic Of Vision
Very few brands have enjoyed the success of Marlboro, which has been a marketing icon for generations.  But it didn't start out that way.  In fact, at one time it was struggling brand.  As a filtered cigarette, it was considered to be a mild tasting product for women and sales were sluggish.
Yet where others saw failure, advertising genius Leo Burnett saw an opportunity.  Health concerns were making filtered cigarettes popular with men and, because of its small consumer base, Marlboro was ripe for a radical repositioning.  With that insight, the Marlboro Man was born.
Leo Burnett came up with the most masculine image that anyone could think of—a cowboy.  Featuring real ranch hands in his ads, he portrayed Marlboro as a product that rough hewn men would smoke, perched atop a horse, for a bit of respite from a long day tending to their herds.  "Welcome to Marlboro Country" the headline read.
That blend of insight and image became Leo Burnett's hallmark.  By 1972, Marlboro was the #1 selling cigarette.  Campaigns for other brands, like the Pillsbury Dough Boy, the Jolly Green Giant and Tony the Tiger found similar success.  In advertising's golden age, a marketer with good instincts could move a brand from worst to first.
» The Road To New Coke
Alas, the world never stays so simple.  As the consumer culture matured, markets fragmented and marketers soon learned that they didn't need to make everybody happy, but could win sales by targeting a particular segment of the market.  In 1984, Pepsi launched its enormously successful "Choice of a New Generation" campaign.
Coke, as the market leader, felt it had to do something.  Pepsi had already made inroads with its long-running "Take The Pepsi Challenge" campaign and this new generational attack was putting it within striking distance.  No longer dominant in the category, Coke felt it had to do something radical.
So it began to develop a new version of its legendary formula, one that was sweeter and would appeal to the younger demographic that favored Pepsi.  After extensive testing which involved over 200,000 consumers, it looked like they had a winner.  All the data lined up: The taste tests, the marketing research and the sales numbers.
Of course, we all know what happened next.  New Coke was probably the biggest disaster in the history of marketing.  Many might have liked the taste of the samples in tests, but nobody wanted their icon taken from them.  In just a few short months, the company beat a hasty retreat and the old formula was reintroduced.
It's hard to imagine Leo Burnett making the same mistake.  He might not have had reams of data or armies of statisticians, but he had what's known as "marketing guts."  He was in tune with the consumer zeitgeist.  He looked beyond their wants and needs and communed with their spirits.
» The Rise Of The Planners
Marketers continued to adapt.  They would no longer be beholden to doing things by the numbers, nor would they operate by instinct, but would form a hybrid of highly trained planners that could do both.  They would still rigorously research the consumer, but to enhance creativity rather than stifle it.
Demographic groups, such as "women 18-34" or "professional men", gave way to psychographic groups like "mainstreamers" and "young aspirers."  They would no longer look at statistics, but try to understand behaviors.  After all, how somebody feels about herself and her life has a lot more to do with what she will buy than her age or profession.
Planners soon became immensely powerful within marketing organizations, but before long it became clear that many were falling prey to confirmation bias.  In effect, they were using research much like a drunk uses a lamppost, for support rather than for illumination.
In a sense, they can hardly be blamed.  Humans are miserably bad at information processing.  So, we naturally learn to filter and focus on the information that seems important.  And what could be more important than data that confirms what we already know to be true in our heart?
» Simulation Marketing
There is a new approach evolving that I call simulation marketing in which machines, rather humans, analyze data.  After all, computers can process far more information than we can and new machine learning techniques allow them to recognize patterns in the data that can alert us to valuable opportunities.
One company, Fractal Analytics, provides a solution they call consumer genomics, which can combine a variety of different sources, including store level sales data, social media and competitive actions to come up with new segments that marketers would have never thought of.
Concentric uses an altogether different approach called agent based modeling in which they simulate not consumers, but entire markets.  This allows marketers to come up with "what if" scenarios and test them virtually rather than in-market.  Its models have proven to be over 90% accurate.
So the key performance attribute for marketers of the future will not be insight, but awareness, adaptation and a willingness to experiment.  If Leo Burnett were alive today, he would use algorithms to plow through terabytes of data, create a series of Marlboro Men, and A/B test them to determine the most effective version for each customer segment.
» Time To Welcome Our New Computer Overlords
Up till now the story seems like a happy one.  A continuous progression from folk wisdom to technologically driven insight.  The old, disheveled, merchants of the past have given way to black-clad, data-driven, socially-networked growth hackers.  The past is gone, the dinosaurs are dead and tribal superstitions no longer need to be abided.
Yet, the story is not so happy.  As MIT's Andrew McAfee points out in a recent HBR post, the challenge of big data is that we need to learn to trust our judgment less, to relinquish the "marketing guts" that we so admire in marketing legends like Leo Burnett—and like to think we see in ourselves—and learn to collaborate with machines.
In fact, he points to research which shows that rather than trying to use data to inform our judgment, we would do much better by putting our energy into building better models, but taking their answers at face value.  Applying our own subjective judgment after the fact is much more likely to worsen results than it is to improve them.
And that's what many marketers will be unable to accept.  We like thinking of ourselves as experts with an intuitive feel for the marketplace.  We like being respected for our judgments and our special gift of insight.  In the end, we will have to decide what's more important: our intrinsic feeling of self-worth or winning in the marketplace.
I'm afraid that for many that will be an easy choice.  It is far more glamorous to be a failed romantic than to be a successful practitioner.