The Calibration Curve
To understand what your calibration curve means, it helps to know how calibration curves are calculated. We start by counting all the times you assigned a likelihood of 0 per cent to a statement, and then count how many of those statements were actually true. Then we plot a point on the graph accordingly. If you are well calibrated, none of these statements should be true. After all, these are the statements that you were absolutely convinced were false. If any of them are actually true, it means that you were overestimating the extent of your knowledge.
We proceed in the same way for each of the other likelihoods. Let’s say you assigned five statements a likelihood of 20 per cent. If you are well calibrated, one of those statements (that is, 20 per cent of them) will be true. And so on.
The line along the bottom of the graph (the x-axis) represents the probability estimates you assigned to the various statements in the test (0%, 10%, 20%, etc.).
The vertical line on the left side of the
graph (the y-axis) represents the proportion of statements in each category that were in fact true. Look at the annotated calibration curve below. The point marked a indicates that, of all the statements
to which this person assigned a probability estimate of 10 per cent, around 30 per cent were in fact true. Even worse, the point marked b indicates that, of all the statements to which this person assigned a
probability estimate of 70 per cent, still only around 30 per cent were in fact true!
By now I hope it is clear why a perfect calibration curve lies on the diagonal line where x = y. The further away from that diagonal line the curve lies, the lower your risk intelligence is. If we shade the
area between the curve and the diagonal line, as in the figure to the right, then the size of this area is inversely proportional to your risk intelligence. In other words, as your risk intelligence increases,
this area shrinks, and vice versa. With a perfect calibration curve, the shaded area shrinks to nothing, and your risk intelligence is at a maximum.
The calibration curve in these two graphs (it’s the same curve actually) is typical of many I have seen. It starts out above the diagonal line of perfect risk intelligence, crosses this line at around the 50 per cent region, and then remains below the diagonal. If your calibration curve looks something like this, what does this say about you?
For one thing, it means that you are overconfident. You tend to say that you are absolutely sure of something when in fact you should be expressing slightly more doubt. When you say you are a hundred percent sure something is true (or false), a wise listener should take what you say with a pinch of salt.
This test is rather unusual in that you can score very highly even if you don’t know much. That’s because this test measures self-knowledge rather than factual knowledge. It rewards you for gauging your own level of uncertainty accurately, rather than for knowing a bunch of facts. This may be why some very intelligent people seem find the test more difficult than others. The brightest people are used to being rewarded for knowing facts, and as a result tend to be overconfident – and this test punishes both overconfidence and underconfidence. They also tend to be uncomfortable with uncertainty; they lack what Keats called “negative capability” – “when man is capable of being in uncertainties, Mysteries, doubts without any irritable reaching after fact and reason.”
The graph below illustrates the relationship between your level of
uncertainty and the likelihood estimates that the test asks you to
assign to various statements. If you are completely uncertain as to
whether a statement is true or false, this is indicated by stating
that the statement is 50% likely to be true. If you are completely
certain that the statement is false, this is indicated by giving a
likelihood estimate of 0%. A likelihood estimate of 100% also
indicates complete certainty – but in this case, you are certain that
the statement is true.
Risk intelligence really comes into its own when you are neither completely certain nor completely uncertain – in other words, when you give estimates from 10% to 40% or between 60% and 90%. This is the twilight zone between the stuff you really know and the stuff about which you don’t have a clue.
In our culture, doubt is often perceived as a sign of weakness. In the financial sector in particular, macho attitudes played no small part in stoking the bubble that burst in late 2007. What if the bankers who were making all those dodgy loans in the preceding decade had undergone regular calibration testing? It’s an interesting thought.