'Bundle entropy' as an optimised measure of consumers’ systematic product choice combinations in mass transactional data

In this work, a novel measure is developed based on entropy to directly measure the predictability of basket composition: 'bundle entropy', where zero denotes a bundle’s total predictability and one – the total unpredictability.

Duration: October 2020 - October 2023

Funder:

Engineering and Physical Sciences Research Council (EPSRC)

Partners:

Tesco

Key people:

Research summary

Understanding and measuring the predictability of consumer purchasing (basket) behaviour is of significant value. While predictability measures such as entropy have been well studied and leveraged in other sectors, their development and application to very large multi-dimensional data sets present in the retailing sector are less common.

While a small number of methods exist, we demonstrate they fail to accord with intuition, leading to the potential for misunderstandings between those who conduct the analysis and those who act on the insights. We delineate the requirements for such a measure in this domain to demonstrate these issues in context. A novel measure is then developed based on entropy to directly measure the predictability of basket composition. The measure is designated as bundle entropy (zero denotes a bundle’s total predictability, one the total unpredictability). We empirically compare the proposed bundle entropy against existing measures using two large-scale real-world transactional data sets, each including more than 2,000 households (frequent shoppers) over two years.

First, we demonstrate how the proposed measure is the only measure that behaves according to the desired properties. Second, we show empirically that bundle entropy differs noticeably from the other measures. Finally, we consider some use case analyses and discuss the utility of the proposed measure in practice.

Research findings

The empirical analysis reveals that the proposed metric, bundle entropy, is the sole measure aligning with the desired properties for such a measure, and it significantly differs from other existing measures. Additionally, practical use case scenarios are considered, discussing the potential utility of the proposed measure in real-world applications.