Small Data and AI: Accelerating Innovation

English mathematician Clive Humb coined the phrase "Data is the new oil" in 2006. Since then, it's become a common metaphor to illustrate the value of data in today's economy.
 
However, the expression also hides a dark side: Data is not valuable unless it's "refined." Unfortunately, many companies seem to accept the phrase at face value, collecting and storing massive amounts of data. Still, few can generate business impact through its use. A Boston Consulting Group study evidenced this contrast, showing that, in 2020, around 60% of companies had an Artificial Intelligence (AI) strategy. Yet, only 10% of those companies achieved significant financial returns. Data refinement has proven to be more difficult than refining oil.
 
According to the same study, companies benefiting from big data and AI projects are well beyond the basics. They have the correct data, technology, and talent organized around a cohesive corporate strategy. 
 
To achieve significant financial returns, companies must make extensive changes to multiple processes, integrating big data and AI as strategic components of their business. Moreover, AI and the company must have a feedback loop so AI can learn from human input and humans can learn from AI. In short, it is not enough to embrace big data and AI projects. It's also necessary to develop a data-driven culture. Companies would be 5x more likely to benefit from such a culture than those that make little or no changes to their processes.

The alternative and progressive path to success: Small Data

The path to success with big data and artificial intelligence projects involves extensively changing business processes. However, these changes do not need to be abrupt. One way to evolve organically and progressively is to embrace “small data” opportunities.
 
Many projects do not require extensive data. According to an article from the Harvard Business Review, they may be completed in a few months by part-timers, resulting in annual financial benefits of up to $250,000.00. These projects may incorporate machine learning/AI methods or focus on using basic Analytics/Statistics methods, accessible to everyone. The benefits go beyond the financial ones, covering the data literacy of employees, the democratization of the use of data for decision-making, and the building of trust by the organization in the management of data projects.
 
In other words, small data projects help companies create the culture that big data and AI require to obtain significant benefits from their deployments.

Innovation via small data and AI

Choosing small data projects first does not mean abandoning or postponing the search for innovation — quite the opposite.
 
In the Corporate Venture model, the disruptive innovation process involves short learning cycles through quick experimentation, aiming to reduce uncertainty through small bets that systematically find a path to scale a product. These experiments vary widely in nature and execution, ranging from qualitative research to simulated sales. However, they are generally associated with collecting attitudinal or behavioral data from consumers regarding the product under development with hundreds/thousands of observations, sorting them as small data projects. In fact, a posteriori data analysis usually involves only basic statistics. 
 
Despite being small data, the application of AI can leverage the findings and improve the product/market fit process. We'll explain that further, starting with how to obtain data.
 
Dan Olsen's book The LEAN Product Playbook outlines that it's common to start with a qualitative survey to understand the relevant issues to consumers in market surveys. Afterward, a quantitative survey finds the number of consumers providing each answer.
 
This approach says "what" to do but not "how" to do it. Anthony Ulwick's Outcome-Driven Innovation (ODI) process can fill the gap, making innovation predictable. Specifically, the ODI process has a (quantitative) research instrument, referred to as Opportunity Scoring, that discovers opportunities and prioritizes efforts in product development. Consumers respond to the degree of importance and satisfaction for each desired result of their current solutions (obtained from qualitative research). Once complete, the scores of all respondents from both processes are combined to calculate an opportunity score for each feature, mapping points that consumers are dissatisfied with but consider essential.

AI enhancing data analysis

Would aggregating all respondents' answers be the best approach, using an average score from a statistically representative sample of consumers to make inferences regarding population preferences? Statistically, yes, but in product development, no, as products are generally made to serve a specific target audience rather than the general public. This is even more true in terms of customization. Moreover, there may be different entry angles for a product depending on the observed market niche. The choice of that entry angle may result in different strategies in product development. One way to check for these biases is to perform a stratified data analysis for each customer type and each persona identified for your product. On average, the population may be well-served with current solutions. On the other hand, a market niche, potential consumers, or a person, is not.
 
How is it possible to reach such different conclusions just by changing the data analysis method? To understand why, let's focus our discussion on the media first. In his book The Black Swan, Nassim Taleb writes that the average is dumb. Taleb's argument is more detailed, explaining that there are classes of magnitudes to which the use of the average is adequate. But, in most real-life cases, it is not. It is beyond the scope of this article to look into such details, but let's use an example to illustrate the concept. Imagine that a shoe retailer's annual sales were 40% of women's size 36 shoes and 60% of men's size 41 shoes. When summarizing this information for an executive report, we could say that, on average, the size of shoes sold was 39, a size that customers did not even buy. Therefore, the average does not represent the size distribution of the shoes sold. Consequently, someone in possession of such limited information could make an erroneous estimate/prediction.
 
On the other hand, note that the average is adequate to summarize the size of shoes sold by gender. In this case, a single attribute, gender, is sufficient to define profiles in which the average represents the group. 
 
In other cases, the average may not represent the group, or more granular categorization may be desired by incorporating other relevant attributes. These could include customer consumption habits, evaluating the shopping experience, the purpose or use of a product, or social and demographic characteristics, among other data points. For those cases, creating personas via basic statistics or combinatorial analysis can be very complex. Moreover, complexity grows exponentially with an increasing number of attributes. Therefore, it is necessary to use more sophisticated techniques. That is where AI or, more specifically, machine learning comes in.

Creating personas through AI

Creating AI-based personas is composed of two macro steps: Grouping by similarity and profile identification. Grouping by similarity is a machine learning task, usually known as clustering, which automatically finds groups of individuals with similar characteristics. There is homogeneity between individuals of the same group and heterogeneity between individuals of different groups. This clustering step would be enough for some applications, such as product selection for cross-selling in recommendation systems.
 
However, characterizing those groups is necessary in product development, where understanding the market niche that the product can reach is required. Then comes the step of identifying profiles, also known as customer profiling, which involves identifying the segmentation rules for each group. For example, group 2 are consumers between 30 and 40 years of age who have consumed at least one premium product and live in places with low commercial activity. This type of segmentation can use several methods. The most common is the decision tree, which uses a machine-learning algorithm to solve classification tasks, providing groups with well-defined segmentation rules.

Final considerations

Creating personas is just one example of how AI can accelerate innovation projects. Still, there is a broader range of opportunities. Even in a small data scenario, those opportunities expand as the organization's maturity with innovation in AI projects grows along with the quantity (and quality) of stored data. It's not necessary to collect years of data from countless processes, build a vast data lake, and create a scalable data processing environment to start benefiting from model-based (IA) decision-making. Of course, some steps are essential for implementing big data projects and should be part of the company's long-term destiny. Until then, take the opportunity to mature progressively and systematically through small data and AI projects. The future has started!

---