A question from Steve in the USA:
“Assuming that it is important during elicitation to get accurate information, how does a business analyst overcome confirmation bias in him or herself, and in the person providing the information?”
What is Confirmation Bias?
Confirmation bias describes the behaviour where you look for, analyse and remember facts in a way that matches what you already believe.
It is very common and we easily spot it in other people (but rarely ourselves, of course!)
A familiar example is when people believe that Friday the 13th is an unlucky day. They list off dreadful things that have happened to them on that day but manage to ignore or disregard any wonderful events on Friday the 13th!
That’s a very day-to-day example, but it happens in projects as well. People pick problems or solutions that fit with how they view the world.
Business cases are frequently subject to confirmation bias. You might hear things like “We need to implement this new system immediately to replace the current one as it is always going down” or “We need 3 more people to do analysis as we spend so much time helping other people we can’t get our own work done”. As it is easy enough to find examples of the bad experience we assume that the person making the case has the evidence to back their case, but they may be exhibiting confirmation bias.
Overcoming Confirmation Bias
To overcome confirmation bias you need techniques that support decision making based on real data.
Confirmation Bias in ‘As-Is’ situations
One method I find really useful for exploring problem or process issues is the DMAIC Process which is a cornerstone of the Six Sigma approach.
DMAIC stands for Define-Measure-Analyse-Improve-Control.
Define – Clearly articulate the problem you want to solve and identify the data you need to gather.
Measure – Measure that data
Analyse – Analyse the data
Improve – Based on the analysis, do something that helps solve the problem
Control – Monitor the problem, using the same data, to make sure the improvement sticks
The focus on measurement and analysis is where the real value comes when trying to reduce confirmation bias.
Once you have measurements relating to a problem you can sensibly ask questions such as
“How often does this happen in our situation?”
“How often does this happen compared to other situations?”
Then you can ask the question
“Is this actually worth fixing, or should we be focussing on something else??”
The analysis makes it clear whether somebody is highlighting a problem because they think they see it (confirmation bias) or because it’s a real problem (supported by data).
Gathering data is frequently as simple as getting a group of people to keep a tally of how many times they come across a problem in a given time-period. Alternatively, your organisation may have work or problem logging systems that you can analyse. Or you may have to be a bit more inventive!
Confirmation Bias in ‘To-Be’ situations
Another area where confirmation bias can creep in is when you are assessing different vendors for a new product or service. Actually, it doesn’t creep in this situation, it tends to storm in with hobnailed boots on!
People can only judge something new and unknown against past experiences and those are different for everyone in the team. So, in advance, you agree how to evaluate a vendor, the criteria to use and how important each criterion is. Then you can generate what is called a Weighted Scorecard.
As you assess each vendor you score them against each of the agreed criteria. At the end of the process, there is a set of data that you can use to decide which product suits your needs best. You can be confident that key information hasn’t been forgotten or glossed over.
There may be push back
Sometimes you get resistance to using these techniques. Some people resist hearing that they are not completely rational and logical. Others see them as unnecessary and time wasting. Disparaging cries of “Let’s not get into analysis paralysis” may be heard from those who just want to DO stuff. So just make sure that the amount of measurement or assessment you do is proportionate with the size of the problem you are trying to solve, it doesn’t have to be statistically significant or have identified every possible option but it does have to be able to answer the questions. Is this problem really a problem and is this the solution that we want?
Let me know what you do to try and overcome confirmation bias. What problems has it caused for you?
I was a guest on Penny Pullan’s BA Virtual Summit early in 2018 talking about cognitive biases and the Business Analyst. As part of this attendees were invited to submit questions to speakers. As Penny was kind enough to provide me with a transcript of my discussion with her, I am sharing the questions and my answers as a series of blog posts. Edited, of course, to make me sound like I can string a coherent sentence together!