Where The Bullets Aren’t (or – how Survivorship Bias and WWII Mathematicians can inform your L&D strategy)
Posted on June 4th, 2015

 by Randy Sabourin

In order to continually improve our Learning & Development designs and deployments we frequently look to successful participants, members of our team, or the industry to try to understand what learning strategies we can glean from their success. This is a strategy I am often asked to execute when creating coaching or sales training programs for clients. Certainly there is a good deal to be learned from the most successful in any organization, however isolating them as the only source of ‘best practice’ is a very limited and potentially dangerous perspective. There are critical lessons to be learned from those who fail and those who may have moved on. It is sometimes difficult to recognize situations where Survivorship Bias affects our decision making negatively.


I find the stories about very smart people using math, technology, and brainpower to defeat the enemy during WWII fascinating. One of the lesser known stories that has offered valuable business insights for me is that of Abraham Wald.

Abraham Wald was a brilliant Hungarian mathematician, grandson of a Rabbi, son of a kosher baker, who was forced to immigrate to the United States in the late 1930s as Nazi Germany’s influence made life very difficult. Wald was invited to Columbia University and fought his war as part of the Statistical Research Group (SRG) think tank. SRG was the mathematical equivalent of the Manhattan project, but using statistics as opposed to explosions. The group of mathematicians solved some of the most important problems facing the Allied war effort: where guns should be placed on planes, fuel mixtures for optimal distances, torpedo barrages, bombing patterns, and numerous other problems that require mathematical expertise to be applied to real life situations.

One of the most interesting dilemmas was where to place armor in order to protect bombers and at the same time reduce the weight of the plane in order to gain distance. The optimal amount of armor was relatively simple to work out but the placement of that armor was the dilemma. The Air Force studied all the returning bombers to determine where to place the armor. The results were less than effective. The military turned over the data and the responsibility to the SRG and Wald. It seems that the damage wasn’t evenly distributed throughout the planes; there were more bullet holes in the fuselage compared to the engine area. The military had incorrectly assumed that the armor (and its weight) was more effectively used on the fuselage. Wald’s approach was much different: put the armour where the bullets aren’t. When he considered the bombers that did not make it back from bombing runs, Wald decided the  solution was to armour the engines. The military never considered the bullet holes in the engine blocks they did not observe.

We may not have Abraham Wald and the big brains of SRG to help solve our learning and development challenges but we can certainly learn from Survivorship Bias. Modeling learning programs after your best employees only is a perfect example of how Survivorship Bias can be an issue.

If the planes that didn’t return are key to solving the problem for Abraham Wald, who are those planes that should be considered in a Learning and Development strategy?

When we give feedback during a roleplay or coaching session we often start with what the participant did right. It’s not because we want to sugar coat constructive feedback with something nice first. The feedback is important because very often the participant does not always know they’ve done something right. Success can also be a result of luck or good timing. A stupid decision in the moment that turns out well is often a brilliant strategy in hindsight. The more valuable insight from success is what not to do and that requires a failure to work from. When we analyze why participants do not utilize the training they’ve received and you’ve invested in (the planes that did not return), we find that there are three main reasons:

Category One: participants are successful using their own methods and do not use the training.  

Category Two: participants try to use the training, experience failure, give up and go back to previous methods.

Category Three: participants use the training persistently, change and are successful.

Research and industry experts tell us that 10% to 15% of learning content delivered in formal learning (classroom, eLearning, etc) is retained successfully. This becomes a very low Return on Investment and makes Category Three the least effective of the three. The percentage between Category One and Two may not be relevant and varies from program to program, in my experience. Consider the participants in these Categories (the planes that did not return) as the key strategic ingredient in how to create and deliver Learning & Development successfully.

Before we explore how to extract this critical data let’s look at why 85% of participants fall into categories One and Two.

The Knowledge-Skill Gap  

There are effectively two parts to learning a new skill. The first step is acquiring knowledge. Reading a book, attending a workshop, or participating in an eLearning program are common knowledge transfer vehicles for Learning & Development organizations. The second step is converting that knowledge into a skill. Most learning organizations subscribe to the Center for Creative Leadership’s (CCL) 70:20:10 ratio. 70% of learning takes place on the job through practice or experience. 20% takes place in informal learning situations like observing others, and 10% takes place in formal learning (workshops, reading, eLearning), the knowledge transfer step. Interestingly enough according to the Association for Training Development’s 2014 State of the Industry report, 55% of L&D budgets are targeted at formal learning programs, the 10%.


Corporate Learning & Development departments are very good at delivering knowledge (the 10%) to a large number of participants in a variety of ways. The low retention and success rates in Category One and Two are a direct result of a lack of investment in the 70%. The 70% is left to unfunded Managers to ‘make the training stick’. Most Managers are far too busy to place practice and coaching as their highest priority. Most Managers are not trained L&D facilitators, and do not understand 100% of the content. In fact, many sat through the same workshop everyone else did and, like everyone else, only retained at best 15%. Also consider that it is very uncomfortable and unprofitable to practice new skills on clients.

Creating Safer Skies

Participants from Category One and Two are the evidence (the missing planes) we are not considering during the learning and development strategy and design process. Where to place the armor, i.e. where to apply your budget for the greatest return, is CCL’s 70%.

The time to address retention is during design, the place to address it is in the field, the method to address it is practice. Creating safe and deliberate practice in the field is the solution to preventing Survivorship Bias from working against you objective.

  1. Create practice based scenarios that break down the process into small steps.

  2. Repeat the process and increase skill and intensity

  3. Use professional roleplayers to simulate reality and increase emotional intensity

  4. Create trust by providing feedback and coaching throughout the practice

  5. Measure a defined set of skills

Bring more learners back from bombing runs

Step 5 of implementing a practice holds the key to extracting the data to where to place the armor. The professional roleplayer measures each skill during the roleplay so that you end up with relevant cohort data on each skill. Data from the lowest two or three performing skills are then fed back into the knowledge transfer stage (workshop, eLearning) to be reinforced. This feedback loop helps to avoid Survivorship Bias, and tells you where to put your armor.

The advantage to 1:1 scenario-based practice is that you have the opportunity to talk to every participant.

Abraham Wald and the big brains of SRG helped to win WWII in their own way; they fought with math and logic. Although it is not ‘life or death’, the battle for budget and resources in corporate learning and development is ours to win. We have a responsibility to our participants to arm them with the best training and practice so they can achieve their objectives in the field.  


5 responses to “Where The Bullets Aren’t (or – how Survivorship Bias and WWII Mathematicians can inform your L&D strategy)”

  1. Adam G says:

    Really enjoyed this posting, why do you think corporations invest in what research shows is ineffective training programs? I look forward to your response.

    • Randy Sabourin says:

      Great question Adam, I think it’s the combination of need for control and the tradition of teaching in a class room environment.

  2. SImon says:

    Very interesting way to look the idea of luck and revised history.

  3. Byron Behnke says:

    Brilliant perspective, and so completely ignored by those who would rather beat their own drum than learn from the casualties.

  4. JDH says:

    Another fun article, I love how you combine and twist everything. Can’t wait to read the next one.

Leave a Reply

To learn more about us and how we can help your employees become more effective communicators through deliberate, experiential practice, please contact us.
Twitter Facebook
Toronto 416-366-6296
Greensboro 336-663-0637
Toll-Free 1-866-945-0648

550 Queen Street East, Suite 300
Toronto, Ontario, Canada
M5A 1V2