Are Confident People Prone to Overconfidence in Probability Assessments?
Alexandrides Daniel, Priscan Daniel, Sivaanandah, Vasan
Engineering Honours Group Project 2015
University of Adelaide
Probability assessments are a useful tool in decision making applications where there is uncertainty in variables. An interval estimate is a range of values which is suspected to contain the true value with an associated percentage of certainty and to make appropriate probability assessments correct interval estimates must be set. Calibration is a spectrum which refers to one’s ability to make accurate interval estimates. The level of calibration is described as underconfident, well calibrated or overconfident. To be well calibrated one must provide an interval which reflects the desired certainty. For example: When asked for an 80% confidence interval the upper and lower bounds would be P10 and P90 for a well calibrated individual. Overconfident and underconfident people provide narrower and wider ranges, respectively. Research has shown that there is a high prevalence of overconfident professionals within the oil and gas industry which is known to be a factor in poor decision making.
Identifying a predictor of calibration is one solution to this problem. Identifying where overconfidence will likely allow corrective action to take place, such as incorporating more training where it is needed or recruiting individuals best suited to decision making roles. Trait confidence was considered to be a possible predictor of calibration and so relationships between individuals’ level of calibration and personal trait confidence were investigated. It was expected that confident people would be prone to overconfident probability assessment based on literature and the similar definition of the two variables. In order to compare these calibration and confidence levels, 95 individuals undertook an online self-administered survey. To measure level of calibration participants were asked to supply 80% confidence
intervals to 20 general knowledge questions. The Personal Evaluation Inventory (PEI) was given to students to identify their confidence in terms of Academic, Appearance, Athletics, Romantic, Social, and Speaking which were then summed into an overall score reflecting their overall, trait confidence. The target demographic targeted was students who would be likely to enter the petroleum industry, including the study-fields of: Engineering, Science and
Overall confidence scores, as well as each of the subscale scores, were compared against calibration levels to determine whether or not any relationships exist. It was found that calibration is not correlated to confidence measured via the PEI as the results produced were not statistically significant. This means that confidence is not a predictor for calibration.
Interestingly, the results also showed a statistically significant weak correlation, suggesting ATAR is negatively correlated with calibration. This means that a higher ATAR predicts a higher overconfidence and the chance of such an extreme result occurring assuming given there is no relation between these two variables is 5% or less.
It is recommended that other predictors of calibration are investigated; however, there still is value in repeating this experiment with alternate measures of confidence. Having multiple methods to elicit confidence would greatly improve the validity of the study but was outside the scope of this investigation given time constraints.