Marketing Science: The Curse and the Promise of Dimensionality

Marketing Science

Data science, the “sexiest job of the 21^st century,”¹ is, at its core, the application of mathematical methods to problems inherent to the information age. Because marketing data is plentiful and critical to making business decisions, marketing has become one of the most fertile areas for data science innovation. While there tends to be a focus in most of the general data science community on specific tools, there are deeper (and, it could be argued, more important and interesting) questions that arise well before the consideration of specific solutions is appropriate.

Dimensionality

What makes marketing science uniquely challenging is a complex and multifaceted question. Dimensionality, or the number of dimensions needed to describe the “marketing space,” is a good place to begin to analyze the problem. For example, three dimensions are needed to describe a spacecraft traveling through space, with time being a situation-dependent fourth possible dimension.

Various concepts surrounding dimensionality are directly or indirectly responsible for many marketing science-specific challenges. Revisiting the spaceship example, forward movement is not dependent on lateral or vertical movement, but all are dependent on time. Interdependencies between dimensions can pose immense problems, especially in systems that, unlike our hypothetical spacecraft, do not have well-defined mathematical relationships between dimensions. For example, there is no obvious quantitative way to predict how a consumer’s brand consideration will evolve over time, a two-dimensional interaction. What impact will targeting and creative decisions have? We now have a problem with 3 dimensions, all of which are dependent on the others. What happens when we consider millions of consumers, over long periods of time, with a large number of potential conversion pathways, knowing that the individual consumers interact with each other, with each creative, and with time in probabilistic, but ultimately unpredictable ways? In an abstract sense, each consumer, each creative, each channel, and every point in time is a separate dimension. To optimize an individual consumer’s conversion pathway, the holy grail of digital marketing, is to know, with a high degree of certainty, the interactions between each of those millions of dimensions. This so-called “curse of dimensionality”² is not totally unique to marketing, but the combination of and relationships between the dimensions is.

Sources of Dimensionality in Marketing

The roots of dimensionality in marketing are interesting both in the problems they pose and the applications of the data they generate. Each source of dimensionality is a deep topic that deserves further elaboration and will be addressed in future posts:

• Time: time series analysis and forecasting is common in marketing, and business more generally. While time can usually be thought of as one dimension, there are optimization problems wherein specific points in time must be considered as separate dimensions. Many traditional forecasting and analysis techniques are very widely known and adopted but can be insufficient under certain circumstances.
• Consumer psychology and behavior: for a psychologist to examine and truly understand one patient under very controlled conditions is a notoriously difficult task. Marketers have millions of consumers to understand, with very little control over conditions and limited ways to observe their behavior. Segmentation and psychographic analyses are attempts to alleviate this, while providing both predictive and descriptive power, but introduce more assumptions and discard granularity and personalization opportunities in the process.
• Content and creative: how many options are there for ad creative for one placement? From a technical standpoint, this can be calculated from character or pixel combinations. From a practical standpoint, the answer is “an infinite number.” Which one will be most effective? Which one will be most effective, given a particular date and time? Which one will be most effective for a particular consumer at a particular date and time? In practice, a limited number of creatives are available and are typically served under very naïve assumptions about consumers. Moving towards a more sophisticated, futuristic approach to marketing, the implications of the above questions are staggering in scale and opportunity.

Developing a Comprehensive Model of Marketing

A comprehensive model for the interplay between time, consumer behavior and marketing campaigns presents tremendous opportunities for improving marketing efficacy and ultimately increasing revenue. The challenges are immense, but so are the opportunities. A number of methods for approaching such high-dimensional problems have been implemented or, at a minimum, theorized.

While there are inherent challenges in dealing with high-dimensional data, probably the most frustrating feature is the lack of knowledge about interplay between dimensions. Various statistical methods exist for inferring such mathematical relations, from simple curve-fitting to particle filters³. A clever and carefully designed testing regimen can elucidate these relationships and go a long way towards building a comprehensive user engagement model, with the aim of not just predicting consumer behavior but driving it.

Beyond the theoretical ability to define inter-dimensional interactions, there are many tried-and-true methods in the marketing data scientist’s toolbox. Multi-touch attribution (Markov Chains⁴), forecasting⁵ (ARIMA, linear regression), optimizations⁶, and audience segmentation (clustering⁷) all have mathematical foundations hundreds of years old and originally developed for applications that predate the information age and, more generally, the modern profession of marketing.

Although current approaches for time series analysis, segmentation and attribution are generally effective, innovation will be required to tie them together. This is the current direction of marketing science: a unified, comprehensive model that ties in all sources and types of data available that can maximize return on marketing spend against an arbitrary KPI.

Concluding Thoughts

The aspirational goal of science is (and to an extent, has always been) to derive a comprehensive “theory of everything” that can, in a system of equations, boil the behavior of the physical universe down to a finite set of coherent, self-consistent principles. The Data Science practice at Softcrylic aspires to develop a “theory of marketing”; in essence, to make marketing decisions largely calculable. We can never hope to remove the human decision-making and creative elements from marketing, but we believe that we can improve, automate and simplify many of the time-consuming and tedious aspects of modern marketing to increase returns and free up marketers’ time to do what they do best: create and innovate.

We are creating an integrated toolset, one tool at a time, to do exactly that. Our tools leverage Softcrylic’s unique products and core competencies as a digital solutions provider to enable informed, data-driven decision making.

1. Davenport, T. H. & Patil, D. J. Data scientist: The sexiest job of the 21st century. Harv. Bus. Rev. (2012). doi:10.1109/MITP.2016.41
2. Bellman, R. The Theory of Dynamic Programming. Bull. Am. Math. Soc. (1954). doi:10.1090/S0002-9904-1954-09848-8
3. Fallis, A. . The Unscented Particle Filter. J. Chem. Inf. Model. (2013). doi:10.1152/jn.00126.2006
4. Kakalejčík, L. & Bucko, J. MULTICHANNEL MARKETING ATTRIBUTION USING MARKOV CHAINS. J. Appl. Manag. Investments (2018)
5. Box, G. E. P., Jenkins, G. M., Reinsel, G. C. & Ljung, G. M. Time Series Analysis: Forecasting & Control. Prentice Hall New Jersey 1994 (2015). doi:10.1016/j.ijforecast.2004.02.001
6. Snyman, J. A. Practical Mathematical Optimization. Africa (2005). doi:10.1007/b105200
7. Tan, P.-N., Steinbach, M. & Kumar, V. Data Clustering: Algorithms and Applications. Data Mining and Knowledge Discovery Series (2014). doi:10.1016/0022-4405(81)90007-8