Randomness plays a larger role in our lives than we would ever like to give it credit for. As humans we want to create causal connections between events. We want to extrapolate from the past, into the future. We want to feel that we can cause things to happen. We tell ourselves, “If I work hard and am honest I can make a good living.” Or “my stockbroker had a good year last year so he’ll have a good one this year.”

Yet in many situations, randomness plays a huge role in the outcome of the event. The past is a poor predictor of the future unless we account for a lot of randomness and luck.

One industry that seeks to fulfill this need for causal understanding is the business book industry. We are constantly told that successful leaders do X,Y, and Z and that if you do X,Y, and Z, you too can lead a successful business. Do business practices and leadership affect the outcomes of a business? Of course they do, just a lot less than the business literature would suggest.

For example, let’s say that the success of a business and the quality of the CEO is correlated generously at 0.30. All this means is that what makes a business successful and what makes a high-quality CEO overlaps 30% of the time. What we then must ask ourselves is this; from Daniel Kahneman‘s book Thinking Fast and Slow.

Suppose you consider many pairs of firms. The two firms in each pair are generally similar, but the CEO of one of them is better than the other. How often will you find that the firm with the stronger CEO is the more successful of the two?

In a well-ordered and predictable world, the correlation would be perfect (1), and the stronger CEO would be found to lead the more successful firm in 100% of the pairs. If the relative success of similar firms was determined entirely by factors that the CEO does not control (call them luck, if you wish), you would find the more successful firm led by the weaker CEO 50% of the time. A correlation of .30 implies that you would find the stronger CEO leading the stronger firm in about 60% of the pairs—an improvement of a mere 10 percentage points over random guessing, hardly grist for the hero worship of CEOs we so often witness.

Now this does not mean that management practices do not matter, it just tells us that identifying those best management practices is extremely difficult! When doing comparison work, such as that made famous by Jim Collins (whom I love but now question), we have to look at it much more skeptically. If the business with the better CEO performs outperforms randomness only 10% of the time, it will be very difficult to even identify those businesses that are really better by skill rather than luck.

Luck plays important roles in our lives. This can be a scary reality, or one that is somewhat freeing. It is certainly scary to realize how little we can control the outcome of our lives, but it can be freeing to have some of that weight lifted from our shoulders, and just maybe it will give us more empathy for those that are struggling.