Measuring the ROI of Marketing: A/B Tests vs. Market-Mix Models vs. Multi-Touch Attribution

Some marketers have despaired of ever being able to connect a sales impact to a specific and unique element of complex and multi-faced marketing plans – preferring instead to use their “gut” to guide marketing decisions and to rely on anecdote or vanity metrics to justify the results. However, in our era of data-driven marketing, this “know-nothing” approach has become increasingly untenable as CFOs and CEOs demand more exacting demonstrations of marketing ROI.

Some marketers lean heavily on A/B tests that iteratively compare a “test” cell to a “control” cell. This allows the marketer to try different messages, offers, color schemes, creative elements, mailing lists or marketing targets. Marketers appreciate the simplicity and clarity of an A/B. test. However, this approach has some important limitations:

•     If you are not a direct marketer, it can be difficult to identify the right response variable to monitor—and thus to read the results correctly.
•     A/B tests give you lots of tactical information, but not the bigger strategic questions—how much to spend, on which channels, toward what kind of brand strategy?

Market-Mix Models (MMM) and Multi-Touch Attribution (MTA). The two approaches share many characteristics. Both rely on large volumes of market data covering both exposure to marketing messages and subsequent sales or other impacts. Both employ advanced statistical techniques to assess the relationship between marketing inputs and outputs. Both depend largely on correlational analyses though, occasionally, experimentally-derived factors or results from in-market tests are taken into account in the models.

For all their similarities, these two techniques have important differences:

•     MMM historically have relied on market-level (rather than individual-level) data analyzed after a campaign was already finished. The “inputs” might be a given price promotion, an estimate of the number of TV and print ads delivered into a market, measures of economic vitality in that market, and measures of competitive activity; the sales response “outputs” might be sales volume or net revenue in that market. The models generate strategic-level findings that take account of a variety of market inputs and media channels, as well as exogenous variables (like the economy) that affect consumer demand.
•     MTA models only examine digital touchpoints. But they do that at the individual level so that the resulting analyses can benefit from some of the findings from A/B tests discussed above. That allows the analyses to happen very quickly to make changes in real time. MTA’s confinement to digital comes with the liability of exposure to the high levels of fraud that have bedeviled the digital advertising supply chain.

Unfortunately, firms that specialize in these analytic techniques tend to treat what they do as very secret intellectual property; as such the field is blighted by an opacity that is particularly galling for marketers.

The bottom line for marketers is that all three of these techniques can provide value, but none should be treated as “the answer”. CMOs who want to get better control of their measurement metrics and ROI diagnostics should:

•     Acquire statistical literacy—sufficient at least to understand terminology and fundamental principles.
•     Think not just in terms of a campaign’s impact, but also in terms of what you are learning about the consumer’s evolving relationship to your brand, product and category. Try to convert the results of your ROI studies into a narrative about the customer’s motivations and drivers, and use that to frame further variables to test going forward. Your internal insights team can be very helpful in this process, just as they can tackle the complex problems of data management, model development, and insight discovery.
•     Be careful that the pursuit of answers to doesn’t lead you to an exclusive focus on short-term results and to bottom-of-funnel metrics. Even if you figure out how to spike sales in the short term, are you building brand equity and customer lifetime value?

McDonald, S. (2018, January 23). Measuring the ROI of Marketing: A/B Tests vs. Market-Mix Models vs. Multi-Touch Attribution. Forbes.

To read more CLICK HERE.