• Skip to primary navigation
  • Skip to main content

The One Percent Almanack

Wit and Wisdom on Investing, Business, and Life

  • Home
  • Members
  • Log In|Log Out
  • Show Search
Hide Search

Behaviouronomics: A Few Data Biases

July 19, 2020

Paras Chopra, an Indian tech entrepreneur, runs a successful (bootstrapped and profitable) SaaS company with annual revenue of 20+ million dollars. Recently he shared his experience related to one of his products. He wrote —

In June, we ran a test on our homepage and while I was looking at conversion rate by segments, I noticed that users from Windows had a 400% higher signup rate for VWO free trial as compared to users using Mac OS X.

Now, that’s baffling and our team spent a good deal of time trying to understand why was that happening. Someone in marketing hypothesized that perhaps Mac OS X users have a better design aesthetic and our homepage wasn’t appealing to them. Was it true?

However, the first conclusions are often wrong, especially when you’re dealing with complex systems.

Consider the case of a stock market investor who goes to a newly opened Mall and sees a long queue in front of a fast-food chain. Being curious, he finds that the fast-food chain is owned by a quick-service restaurant brand that has issued new stocks in a recent IPO.
[show_to accesslevel=’almanack-all’]

In a jiffy, the smart investor calls his broker and asks him to load up on the stock.

And then the stock price starts falling.

Our smart(ish) investor goes to the mall again and verifies that the long queue still exists in front of the restaurant. This confused investor then digs further and realizes that the long queue wasn’t because of the high demand but because of slow service.

Drawing wrong (and hasty) conclusions and calling it supported by data is not a new habit. It has plagued the human decision making since the time we discovered numbers.

In the 18th century, when the medical community was still struggling to wrap its head around the disease Diabetes, they identified one symptom that was consistent among all diabetes patients. Both their blood and urine tasted sweet. In fact, tasting the urine of the patient for sweetness was the classic diagnostic test for Diabetes.

In 1776, the English physician Matthew Dobson identified sugar as the substance causing this characteristic sweet taste.

French physician Pierre Piorry (1794-1879) advised people with diabetes to eat large quantities of sugar to replace what they lost in their ruin. While the logic seemed reasonable at the time, it was not a successful strategy.

Correlation isn’t always the causation. Paras also figured something similar to his issue. He explains —

When we dug into data, we realized that our recently installed automated QA service creates signups on the homepage every hour or so (to ensure the form doesn’t stop working) and guess what, that automated service used Windows. After removing such QA signups from data, Mac OS X and Windows conversion rate became comparable. This is a perfect example of Twyman’s law. Remember, if the data is too good to be true, it’s probably wrong.

Many extreme results are more likely to be the result of an error in instrumentation (e.g., logging), loss of data (or duplication of data), or a computational error.

This mental model — Twyman’s law was certainly new to me.

Twyman’s law has been described as one of the most important laws of data analysis. It states that any figure that looks interesting or different is usually wrong.

Another aspect of the case described by Paras is related to the change in systems’ behaviour from an attempt to measure or observe the system.

In the book Universal Principles of Design, authors explain —

The act of measuring certain sensitive variables in a system can alter them, and confound the accuracy of the measurement. For example, a common method of measuring computer performance is event logging: each event that is performed by the computer is recorded. Event logging increases the visibility of what the computer is doing and how it is performing, but it also consumes computing resources, which interferes with the performance being measured.

In cases where highly invasive measures are used over long periods of time, it is common for systems to become permanently altered in order to adapt to the disruption of the measure. For example, the goal of standardized testing is to measure student knowledge and predict achievement. However, the high stakes associated with these tests change the system being measured: high stress levels cause many students to perform poorly; schools focus on teaching the test to give their students an advantage; students seek training on how to become test wise and answer questions correctly without really knowing the answers; and so on. The validity of the testing is thus compromised, and the invasiveness of the measure fundamentally changes the focus of the system from learning to test-preparation.

This uncertainty bias has special importance for investors. Those who are into speculative trading require constantly looking at the stock market data and the portfolio. The more you stare at your portfolio, the more it interferes with your reasoning and perception.

It’s devilishly hard not to get impacted by a lot of red colours on your portfolio tracker. Or not get overconfident because of a few lucky wins.

Another way data causes bias is when investors spend too much time digging out obscure data about a company without questioning its relevance. When presented with new data, our brain usually retains the information that agrees to its pre-formed notions and discards any conflicting evidence. As a result, more data simply makes our hypothesis more plausible (to us) and not more probable.

The biggest and the most dangerous data bias is where we rely on a data point not appearing in the future because it hasn’t been observed in the past. In other words, we believe that the absence of evidence is evidence of absence.

Shane Parrish writes in his blog –

…our documented history can blind us. All we know is what we have been able to record.

We think because we have sophisticated data collecting techniques that we can capture all the data necessary to make decisions. We think we can use our current statistical techniques to draw historical trends using historical data without acknowledging the fact that past data recorders had fewer tools to capture the dark figure of unreported data. We also overestimate the validity of what has been recorded before and thus the trends we draw might tell a different story if we had the dark figure of unreported data.

The thing that makes data i.e., numbers appealing, is the preciseness it offers and humans are suckers for anything that can take away their uncertainty-anxiety.

Max Gunther, in his book Zurich Axioms, writes —

…the stock market has no patterns. It almost never repeats itself and never does so in a reliably predictable way. Making charts of stock prices is like making charts of ocean froth. You’ll see each pattern once, and then it will be gone. Only by blind chance will you ever see it again. If you do see it again it will have no significance, for it predicts nothing.

The biggest risk of big-data, Machine Learning, and Artificial Intelligence is precisely this. The illusion of certainty that it will eventually bring. The illusion that crunching petabytes of data using sophisticated computers and AI-powered algorithms can solve every problem.

It has been more than ten years since a major crash has hit the stock market. As a result, a significant portion of the investor crowd in the stock market is made up of people who opened their Demat accounts after 2008. These are the people who have only heard the stories about the crash but never really experienced one. And even those who witnessed the last financial crisis have a fading memory of it now.

[/show_to] [hide_from accesslevel=’almanack-all’]

Want to Read More? This content is exclusive for members of Value Investing Almanack. Login to read if you are a member. Else, click here to subscribe.

Benefits to VIA Members
 
  • Spotlight: Big ideas from Value Investing and why applying them in your investment decision making will be a great deal
  • InvestorInsights: Interviews with experienced value investors, learners, and deep thinkers
  • StockTalk: Thorough analysis of business models of companies (without any recommendations)
  • Behaviouronomics: Deep analysis of human behaviour and how it impacts investment decision making
  • BookWorm: Reviews of the best books on Value Investing and related subjects
  • Free Course – Financial Statement Analysis for Smart People (otherwise priced at Rs 5,900)
  • Archives: Instant access to our huge archive from the past three years
Become a VIA Member. Click to Subscribe
[/hide_from]

Filed Under: Behaviouronomics

Handcrafted with in India