Rotterdam school of Management, Erasmus University compact logo

Article: Tuesday, 26 November 2024

Imagine it’s a rainy afternoon and you’re considering a ride home from your office. Opening your Lyft app, you see a ride offer for $14.00. Would you accept it?

Perhaps you might, but what if the price was $13.99? Despite the minimal difference, analysts at Lyft discovered through an analysis of over 600 million price offers during a seven-month period that travelers were disproportionately drawn to fares just below a whole dollar. They estimated that if Lyft had adopted a 99-cent price strategy in 2019, profits would have increased by about $160M.

Now, switch the scenario to subscribing to a Substack newsletter. Would a price of $13.99 make you more likely to subscribe than $14.00? Interestingly, Substack’s recent experiment found the opposite effect as Lyft: round-priced newsletters were more attractive than those priced at 99 cents.

These cases highlight two key points. First, categorical boundaries like pricing thresholds play significant psychological and financial roles for consumers and businesses. Second, these boundaries can have opposing effects in different contexts. While 99-ending prices attracted more customers for Lyft, round prices proved more appealing for Substack subscribers.

In the social sciences, this issue is known as “external validity”—the extent to which findings from one context reliably generalize to another. A critical factor determining external validity is whether the observed effect is influenced by specific background characteristics that vary between the original and new contexts. If the effect does not interact with background differences, previous findings and models can help predict outcomes in new contexts. However, if background characteristics matter and vary across contexts, relying solely on past data will fail to predict future outcomes.

dr. G. (Gabriele) Paolacci
Associate Professor of Marketing Management
Rotterdam School of Management (RSM)
Erasmus University Rotterdam
Photo
Gabriele Paolacci

Dr. Bart De Langhe

Professor of Marketing

KU Leuven
Vlerick Business School

An important term in social sciences is "external validity” - the extent to which findings from one context reliably generalise to another.

Italy is learning this the hard way. In 2020, the Italian government launched the “superbonus” program, offering an unprecedented 110% reimbursement for homeowners undertaking energy-efficient renovations. This decision was based on extrapolations from previous schemes. The State General Accounting Office expected the costs to merely double compared to previous schemes that offered 50% to 65% tax credits, predicting manageable expenditures of around €35 billion over a 15-year period. However, it turned out this generalization was unwarranted. Four years after its introduction, the “superbonus” has already turned into a €160 billion financial debacle. A dramatic miscalculation with potentially dire fiscal consequences for Italy.

To avoid the trap of overgeneralisation, we offer decision-makers and data analysts three recommendations:

1. Rigorously assess contextual similarity

Before extrapolating from a previous context, it is crucial to thoroughly map out the similarities and differences between the contexts. Anchoring on the current decision context is essential for this assessment. 

For example, the State General Accounting Office in Italy failed to evaluate whether the assumptions of the models built out of data from previous schemes were met in the current situation. This oversight ignored key differences crucial to homeowners’ decisions. A 110% tax credit is not just double of 55%. It is disproportionately appealing because it exceeds the total cost of renovations. This effectively eliminated any financial risk for homeowners, aligning their interests entirely with builders and inadvertently encouraging them to inflate costs, sometimes fraudulently. Additionally, the program allowed homeowners to transfer their tax credits to third parties, enabling renovations at no direct cost regardless of the homeowners’ liquidity or taxable income levels. These crucial differences rendered past data, though seemingly useful, largely irrelevant to the current context.

2. Build a theory for past data

When considering whether and how to extrapolate from previous data, it is essential to understand why past results have been observed. Generalisability is fundamentally a theoretical question, involving the identification of necessary conditions for a result to be observed and the disabling conditions that might eliminate or reverse it. Historical data, whether correlational or from randomised trials, rarely provides complete information on these aspects. By theorising about the reasons behind observed results, decision-makers can identify the necessary conditions for those results.

For instance, consider the extensive dataset Walmart obtained in 2020. In collaboration with a team of behavioural scientists, Walmart pharmacies across the United States tested and ranked the effectiveness of 22 alternative text messages to encourage people to get vaccinated. To what extent can Walmart trust that the most effective messages for getting people to the pharmacy would work in different contexts, such as getting people to shop? After the study, the team theorised about—and to some extent empirically validated—the critical factors explaining the differential effectiveness of the alternative messages. They found that the most effective messages framed the target product as individual items waiting for customers to pick them up. This analysis positions Walmart better for generalising (or not) from this highly specific context (flu vaccination) to new ones.

3. Collect new data

Theorising is challenging. Key background variables may not be observable in the available data, and theories are inherently incomplete, potentially overlooking relevant variables that differ across contexts. Moreover, theories could be incorrect if they are based on unreliable data. For these reasons, decision-makers should often consider collecting new data.

In a recent study with a national wholesaler of construction supplies, researchers investigated the effectiveness of a predictive sales analytics tool that estimates churn probabilities for individual construction businesses. The team tested an intervention aimed at fostering realistic expectations about the tool’s performance to increase its adoption among salespeople. Contrary to past evidence suggesting this intervention would boost adoption, the team found that salespeople used the tool less effectively when informed via a disclaimer that it might overestimate the risk of churn for some customers. Past evidence on these disclaimers was not predictive of this specific situation, and new data was invaluable in uncovering this inconsistency.

Sometimes, data is cheaper to collect than one might think. A simple survey of how Italian homeowners’ intentions to renovate vary across different repayment rates could have indicated that past results were not generalisable and provided more contextualised knowledge to predict the expenses of the superbonus. Randomised controlled trials are also often easy and fast for digital companies, as demonstrated by Substack. Unfortunately, available data can often obscure the need for more contextualised understanding, more theorising, and more data collection.

Related articles

RSM Discovery

Want to elevate your business to the next level using the latest research? RSM Discovery is your online research platform. Read the latest insights from the best researchers in the field of business. You can also subscribe to the newsletter to receive a bimonthly highlight with the most popular articles.

Do you want to learn more about this subject?

Check out these RSM education programmes

Leading with Decision-Driven Analytics
Leading with Decision-Driven Analytics
  • 1 Apr 2025
  • 3 days
  • 3,500
Digital Leadership and Change
Digital Leadership and Change
  • 2 Jun 2025
  • 3 days
  • 3,600
Digital Innovation
Digital Innovation
  • 7 May 2025
  • 3 days
  • 3,600
Strategic Account Management
Strategic Account Management
  • 12 Mar 2025
  • 3 days
  • 4,000
Executive MBA
Executive MBA
  • 18 months
  • 68,000
MSc Business Information Management
MSc Business Information Management
  • 12 months
    • €2,530 (EEA)*|
    • €22,500 (Non-EEA)
    • *see details on page
Diploma Programme in Digital Transformation
Diploma Programme in Digital Transformation
  • 10 Mar 2025
  • 10 March
  • 11,000
Digital Analytics and Customer Insights
Digital Analytics and Customer Insights
  • 7 Apr 2025
  • 3 days
  • 3,600
Digital and AI Strategy
Digital and AI Strategy
  • 10 Mar 2025
  • 3 days
  • 3,500
Global Executive MBA
Global Executive MBA
  • 21 months
  • 72,000
MSc Marketing Management
MSc Marketing Management
  • 12 months
    • €2,530 (EEA)*|
    • €22,500 (Non-EEA)
    • *see details on page
MScBA Business Analytics & Management
MScBA Business Analytics & Management
  • 12 months
    • €2,530 (EEA)*|
    • €22,500 (Non-EEA)
    • *see details on page
International Full-time MBA
International Full-time MBA
  • 12 months
  • 65,000
Diploma Programme in General Management
Diploma Programme in General Management
  • 17 Mar 2025
  • 1 year
    • €13,100 -
    • €15,500 depending on modules chosen *
    • * see details on the page
Your contact for more information:
Danielle Baan

Science Communication and Media Officer

Portrait of Erika Harriford-McLaren
Erika Harriford-McLaren

Corporate Communications & PR Manager

Erasmus University campus in autumn, showcasing its iconic red trees, viewed from across the campus pool.