Here’s How Technology Can Help Solve The Black Swan Problem

In The One Percent Doctrine, veteran journalist Ron Suskind described how former Vice President Dick Cheney, when told that a unlikely but dire threat had merely a one percent chance of happening, argued that, in terms of response, it should be treated as a certainty.

The idea has a certain logic to it. Clearly, if the potential impact of an event is consequential enough, then it needs to be addressed whether it is likely to happen or not. After all, unlikely things happen all the time. However, nobody’s resources are unlimited. So you need to balance the need to prepare for low probability events with the need to address more likely ones.

That, in essence, is the dilemma every business finds itself in. We spend most of our time working on the regular stuff, even though, as Nassim Taleb explained in The Black Swan, even very low probability events can have an enormous impact. Fortunately, today technology is transforming our ability unlock opportunity even in the most unlikely places.

The Vioxx Problem

When Vioxx was approved by the FDA in 1999, it gave hope to millions of people who suffer from chronic pain. While conventional pain relievers, such as aspirin, ibuprofen and naproxen, can degrade the lining of the stomach, Vioxx promised to relieve pain and inflammation without risk of gastrointestinal problems.

Yet it was soon found that Vioxx increased the risk of a heart attack in less than 1% of patients and it was pulled from the market in 2004. While nobody disputes that was the right decision to make, it is a bit strange that more than 99% of those who suffer from chronic pain can’t get an effective medicine due to an issue that very few people actually have.

The problem, of course, is that it’s very hard to identify the small proportion of people who are at risk, but even an infinitesimal percentage can add up after awhile. Multiplied by the millions of people who were taking Vioxx, it was estimated that between 88,000 and 140,000 patients experienced cardiac problems. That’s the size of a fairly large town.

The Opportunity Of 1% Markets

The Vioxx problem is bigger than you think, because we’re all one percent of something. Just like the body chemistry of that small percentage of Vioxx patients refused to conform, we all have our own idiosyncrasies and peccadillos that defy any conventional means of classifying us. Yet these small pockets represent not only risk, but opportunity as well.

As Mark Penn and Kinney Zalesne pointed out in Microtrends, 1% of the US translates to over 3 million people, more than enough to support a thriving business. Homeschooling parents, adventure seeking retirees, vegan children and millennials who knit are all relatively small markets, but add them up and there is enormous value to be unlocked.

Some companies, like Amazon and Netflix, have created great businesses catering to the weirdness in all of us by building platforms that access ecosystems of tastes, needs and desires. These platforms help aggregate those small pockets easily and cheaply enough to make them economically viable.

Still we’ve merely scratched the surface of what’s possible. Sometimes, what’s locked in those obscure pockets is more than just weird, it’s truly extraordinary. Yet it’s so small, we can’t identify it through conventional means, only through aggregation.

Unlocking 1% Of Scientific Data

Problems like Vioxx are so hard to detect because they fly under our radar screen. When you are working to develop a pain reliever, checking for the minuscule possibility that it may result in heart problems isn’t exactly top of mind. That’s why the problem only became noticeable when thousands of people began using the drug.

There’s a similar problem with cancer patients. Each one has its own unique genome with millions of base pairs. Scientists know that one small mutation can result in a specific type of cancer, but figuring out exactly which mutation causes a particular tumor is like looking for a very small needle in the largest of haystacks.

That’s why scientists at the National Cancer Institute and the National Human Genome Research Institute created The Cancer Genome Atlas. Rather than studying specific tumors, they aggregated thousands of them and released the data for anybody who wants to study it. The program has moved cancer research light years ahead.

More recently the Materials Genome Initiative began working on a similar solution for manufacturing. Rather than the typical trial and error process for identifying advanced materials, it aggregates information taken from thousands of different studies and uses machine learning to identify patterns that can lead to breakthroughs.

Notice how different this is from the traditional research process in which you start with a particular theory and then go out and try to prove it. Initiatives like The Cancer Genome Atlas and the Materials Genome Initiative collect seemingly meaningless data and find patterns in it. Those patterns can then be tested further to see if they can actually solve a problem.

Exploring The New Horizons Of The Highly Unlikely

Clearly we can’t, as Vice President Cheney suggested, treat every small probability as if it was a near certainty. Today, however, emerging technologies like big data and machine learning allow us to explore myriad possibilities, identify patterns and verify their significance.

That’s what makes companies like Amazon and Netflix so successful. Anybody can sell us a book or show us a movie, but they have unlocked tremendous value in helping us identify what we would be unlikely to find ourselves. Even better, we can provide feedback to them — either explicitly or through our behavior — that will help them identify even more great stuff.

Even more importantly is how these technologies are helping to create a new scientific revolution in which platforms like The Cancer Genome Atlas and the Materials Genome Initiative comb through far more data than a person ever could, detect patterns and open up new possibilities.

Yet what’s truly amazing is that very little of this was viable even a decade ago and we’ve done little more than scratch the surface up till this point. In the future, it seems clear that our technology will advance to where even the highly unlikely becomes a sure bet.

An earlier version of this article first appeared in Inc.com