The U.S intelligence community has a long history of blowing big calls -- the fall of the Berlin Wall, Saddam's WMD, 9/11. But in each collective fail, there were individual analysts who got it right. Now, the spy agencies want a better way to sort the accurate from the unsound, by applying principles of mathematics to weigh and rank the input of different experts.
Iarpa, the intelligence community's way-out research arm, will host a one-day workshop on a new program, called Aggregative Contingent Estimation (ACE). The initiative follows Iarpa's recent announcement of plans to create a computational model that can enhance human hypotheses and predictions, by catching inevitable biases and accounting for selective memory and stress.
ACE won't replace flesh-and-blood experts -- it'll just let 'em know what they're worth. The intelligence community often relies on small teams of experts to evaluate situations, and then make forecasts and recommendations. But a team is only as strong as its weakest link, and Iarpa wants to fortify team-based outputs, by using mathematical aggregation to "elicit, weigh, and combine the judgments of many intelligence analysts."
The system Iarpa's after should be able to collect and evaluate expert opinion based on each expert's specific expertise, learning style, prior performance and "other attributes predictive of accuracy." It'll then parse out the different predictions offered by analysts, and assign them degrees of probability based on where a particular expert sits in the rankings.
If Iarpa is able to master the mathematical art of aggregated probability, the agency's program would likely be in hot demand. Using probabilistic expert aggregation to make decisions has been toyed with in circles as diverse as big business, climatology and even criminal court. But until Iarpa's also mastered their plan to nip biases and memory lapses, they'll still be forced to contend with the inevitability of human imperfection. Notes risk communications expert Professor Morgan Granger in a decades-old paper, "One can only proceed with care, simultaneously remembering that elicited expert judgments may be seriously flawed, but are often the only game in town.”
[Photo: Wikimedia.org]
