Steve: Jay, have the investigations into the January 6 assault on the US Capital revealed any lessons for private sector organizations?
Jay: While we lack access to the most sensitive information, based on open sources it’s clear that intelligence and federal and state law enforcement agencies tragically underestimated the scope, size, and intensity of the protestors gathered just a few blocks from the US Capitol. And from these and my own experiences, Steve, I want to pass on to our private sector audience the single most important takeaway from what happened. It was, by definition, an intelligence failure, which occurs when analysts and their managers for multiple reasons fail to anticipate an event or development that impacts clearly-understood US interests, despite information that could have mitigated that risk. How bad can this get? By all official accounts, the fallout from the January 6 assault has been catastrophic in terms of loss of life, property damage and the undermining of our political institutions. The effects will be long lasting.
Steve: This notion of intelligence failures clearly applies to our audience. Organizations often have all the data they need to make sound decisions, but emotions, biases, and time pressure can derail the process when stakes are highest. I know from my own experience that such failures can happen to any organization at any time, regardless of its success rate in making good decisions. No organization one is immune, and the more organizations are convinced that their track record inoculates them from such intelligence failures, the more vulnerable they are to bad decisions. I also strongly believe that there is no single point of failure. These are systemic failures that prevent organizations from objectively assessing risk, especially under the pressure of high-stakes decisions.
Jay: Good point. In the case of the January 6 assault, multiple post-mortems made clear that even with the vast analytic and data-collection resources available to law enforcement agencies, that signaled the extent and intent of dozens of extremist groups, they mis-read the evidence. In fact, a sole DHS senior analyst had gathered reporting from all over the country that should have been a flashing red light to the decision-makers, but he was ignored and not even invited to planning meetings. In the CHP’s last “Special Event Assessment” on January 3, in their bottom-line-up-front (their label), they focused only on the expected efforts by senators and member of congress to stop the certification process. CHP analysts raised the potential for violence only in the last paragraph—labeled their “Overall Analysis”-- and then only assessed that it “cannot be ruled out.” The FBI’s analysis was no more accurate, assuming that the language in the reporting was “aspirational” and, therefore, protected by the First Amendment.
Steve: These were critical errors in judgment and make the crucial point of Intelligent Analysis, that immunity from bad high-stakes decisions is an illusion. It would be easy for our private sector audiences to claim (incorrectly) that the intelligence services were simply incompetent and that their own organizations don’t operate like that, that their decision making processes are airtight. But I have seen these same failures in decisions across business areas and within organizations that could have been avoided.
Jay: Right. Decision making is a minefield, but the analytic tools and processes set out in our book can help break down this sense of invulnerability that gets in the way of objective, evidence-based assessments. Here are three areas where public and private sector challenges overlap.
Steve: Applying Intelligent Analysis to a high-stakes decision means slowing down the decision-making process long enough to a) objectively examine the supporting data and analytics, b) counter cognitive biases and organizational influence, and c) flush out expert opinions which would otherwise be unheard. A good analogy for this slow-down is the widely-used traffic management device of speedbumps. Speedbumps don’t change a car trip’s ultimate destination, or its driver, or the vehicle being driven. They merely reduce its speed through an apparently clear stretch of road where serious accidents have been known to occur or are liable to occur.
Jay: This analogy is a good one. When drivers come across a speedbump, not only do they have to slow down in order to avoid damage to their vehicle and discomfort to the passengers, they’re also prompted to look around for the non-obvious danger conditions which prompted installation of the speedbump, such as children playing, cyclists, or elderly residents. Each of the steps in Intelligent Analysis is like a speedbump and, if skipped or ignored, there can be serious consequences. Our method doesn’t affect how an organization reaches its decision. What it does do is stress-test the decision, by examining the data supporting the assumptions which have to hold true in order for the decision to result in success.
The January 6 US Capital Insurrection Select Committee: Asking the Right Question Drives Everything Else
Steve: Jay, do you think the January 6 Select Committee asked the right question?
Jay: Albert Einstein famously said “If I had an hour to solve a problem and my life depended on the solution, I would spend the first 55 minutes determining the proper question to ask. For once I know the proper question, I could solve the problem in 5 minutes.” Intelligent Analysis agrees with Einstein, that a well formulated question — in our book we call it the key intelligence question (KIQ) - will lay the groundwork for an objective, evidence based final report. Indeed, getting the question right should be the committee’s first, and arguably the most important, step.
Steve: That sounds easy.
Jay: It’s not at all. Why it matters so much, for example, is reflected in the difference between these two questions:
Steve: Those two questions sound quite alike.
Jay: Let me explain the difference. What’s wrong with the first one is that it answers the question in the question. It assumes that white supremacists were involved. From that point on, the focus of gathering information and inquiries will be to make that argument. Conversely, the second question is an open question which leaves room for a broadly-based assessment that begins with no preconceived outcome, regardless of personal opinion. An open question shifts the focus to the data and can blunt the impact of deeply held biases and opinions going in. A question that answers the question leads to cherry-picking reporting and leaving aside the possibility of more nuanced analysis.
Steve: So, if the committee had selected the second question, how could they have answered it?
Jay: The committee could have broken down the KIQ into what we call sub-questions, that must be answered in order to answer the broader KIQ. These can be wide-ranging and probe more deeply into different aspects of what happened, for example:
Steve: It seems like a structured process is needed, to avoid falling into the trap of formulating a KIQ that answers the question in the question.
Jay: Exactly. Our book includes an extensive chapter on formulating the “right question,” including a tool to help bring discipline to the process that would please Einstein.
If you’ve landed on this page, you’ll probably appreciate that Intelligent Analysis isn’t just A book for Jay and myself, it’s THE book that we’ve felt compelled to write ever since our collaboration began in 2016. The methodology at the heart of our book is so fundamental to the practice of sound decision-making that we’re constantly looking for opportunities to discuss, teach and practice it, as well as learn new applications and examine high-stakes decisions profiled in the news through our distinctive lens. We hope that your contributions to this discussion thread will help us do that, and simultaneously stimulate your own thinking and decision-making practices.