![[personal profile]](https://www.dreamwidth.org/img/silk/identity/user.png){kind=small}
Wikipedia
I've heard of this for years, but How To Measure Anything goes into it in some depth. And provides some samples tests here (MS Word doc).
Two kinds of questions: ones like "when was X born" or "how big is Y", where you give a range of numbers such that you think the answer is 90% likely to be in between them; if you're right, then out of 20 questions you should get 18 right, out of 100 questions you should get 90 right. Note this isn't a straight test of knowledge: if you're less certain, just give a wider range, right? But most people tend to get like 60% right, so are overconfident in their assessments.
The other kind of question is "Statement Z", and you pick whether it's true or false, and your confidence from 50% to 100%. (If you're less than 50% confident, you should have picked the opposite answer...) You add up your expected score -- if Q1 is 100% and Q2 is 70%, you expect 1.7, etc. -- then grade the test. If you score a lot lower than expected, you're overconfident!
The book has a short test in chapter 5 (10 questions of each type), then 2 longer (20 questions) tests in the back. On the first type I've done 80%, 75%, 95%; on the second I've always scored higher than expected, only missing the questions where I give 50% ("I have no clue") confidence. So underconfident there, which doesn't surprise me. On the first type, I think I added some error from believing I was underconfident and trying to correct for that.
There's a subtlety with the second type: Hubbard says that if you get a single 100% answer wrong, that's a sign of overconfidence, but his test just has you circle probabilities by 10% -- 50, 60... 90, 100%. So what should you do if you're like 96% confident? It's natural to round up, but that leaves you being wrong on "100%" occasionally; if you take the floor, then you'll get "90%" right more often than you should.
Anyway, so after having taken one such test and probably found that you're overconfident, the next step is to try to calibrate yourself. Tricks suggested include "try to imagine you're wrong, and what could have gone wrong"; "instead of picking a value and applying error bars, start with an absurdly wide range and narrow it"; "think that the true value should be 95% likely to be under your upper bound, and 95% likely to be above your lower bound", and equivalent bets.
That one is imagining that you could get money by being right (especially for the first type of question), or get money by spinning a wheel with 90% payout chance. If you'd rather go with the wheel, then you're not actually 90% confident in your answer; if you'd rather go with your answer, then your range is too wide. This is supposedly the main silver bullet for calibration, though I've found it hard to apply.
The chapter also talks about evidence that risk estimation is a learnable skill (for 95% of people) and that getting better on trivial questions does generalize to more useful applications.
Also I think you could make your own tests easily enough after seeing one: come up with your own questions where you don't know the answer but could easily look it up, trying to answer the questions, then score yourself. 'Fun', plus you'll learn things!
I've heard of this for years, but How To Measure Anything goes into it in some depth. And provides some samples tests here (MS Word doc).
Two kinds of questions: ones like "when was X born" or "how big is Y", where you give a range of numbers such that you think the answer is 90% likely to be in between them; if you're right, then out of 20 questions you should get 18 right, out of 100 questions you should get 90 right. Note this isn't a straight test of knowledge: if you're less certain, just give a wider range, right? But most people tend to get like 60% right, so are overconfident in their assessments.
The other kind of question is "Statement Z", and you pick whether it's true or false, and your confidence from 50% to 100%. (If you're less than 50% confident, you should have picked the opposite answer...) You add up your expected score -- if Q1 is 100% and Q2 is 70%, you expect 1.7, etc. -- then grade the test. If you score a lot lower than expected, you're overconfident!
The book has a short test in chapter 5 (10 questions of each type), then 2 longer (20 questions) tests in the back. On the first type I've done 80%, 75%, 95%; on the second I've always scored higher than expected, only missing the questions where I give 50% ("I have no clue") confidence. So underconfident there, which doesn't surprise me. On the first type, I think I added some error from believing I was underconfident and trying to correct for that.
There's a subtlety with the second type: Hubbard says that if you get a single 100% answer wrong, that's a sign of overconfidence, but his test just has you circle probabilities by 10% -- 50, 60... 90, 100%. So what should you do if you're like 96% confident? It's natural to round up, but that leaves you being wrong on "100%" occasionally; if you take the floor, then you'll get "90%" right more often than you should.
Anyway, so after having taken one such test and probably found that you're overconfident, the next step is to try to calibrate yourself. Tricks suggested include "try to imagine you're wrong, and what could have gone wrong"; "instead of picking a value and applying error bars, start with an absurdly wide range and narrow it"; "think that the true value should be 95% likely to be under your upper bound, and 95% likely to be above your lower bound", and equivalent bets.
That one is imagining that you could get money by being right (especially for the first type of question), or get money by spinning a wheel with 90% payout chance. If you'd rather go with the wheel, then you're not actually 90% confident in your answer; if you'd rather go with your answer, then your range is too wide. This is supposedly the main silver bullet for calibration, though I've found it hard to apply.
The chapter also talks about evidence that risk estimation is a learnable skill (for 95% of people) and that getting better on trivial questions does generalize to more useful applications.
Also I think you could make your own tests easily enough after seeing one: come up with your own questions where you don't know the answer but could easily look it up, trying to answer the questions, then score yourself. 'Fun', plus you'll learn things!