Skip to content

Order Effects On Elicitation: Comparing The Superiority Of Expert Estimations Over Non-Experts Using Various Guided Elicitation Methods

Caluya, Alfred

Manash, Felix

Engineering Honours Degree, 2011

University of Adelaide


Elicitation methods have been focused in capturing the 80% confidence interval, when used in estimation of parameters for building models that were used in the petroleum industry, such as predicting oil in place, recovery factor and fluctuations in oil price. Moreover, the methods used to obtain these parameters from the experts relied heavily on subjective expert judgments and their personal beliefs. These were subject to numerous forms of cognitive biases such as anchoring, adjustment and overconfidence. Many psychological studies have identified the cognitive biases related to the order in which elicitation questions were presented to participants. However, the extent to which order effects influenced the outcomes of elicitation processes has not yet been clearly established. In recent years, a lot of research has been focused particularly in trying to establish ways in which these cognitive biases can be reduced.

This project aimed to confirm the results of the investigations of the order effects on elicitation that has already been started by many scholars. Three elicitation methods were investigated in this study. In the first method, participants were asked to provide (P10-P90) estimates. The second method asked for a best guess first value (P50-P10-P90) from the participants, followed by the (P10-P90). In the last method, participants were given additional guided false feedback information that either confirmed or contested the participant’s previous estimates. They were then asked if they would change their previous estimates from some of the questions in both section (1) and section (2). Survey (2) was similar to survey (1), except that participants were presented with section (2) questions first, followed by section (1) question. In survey (1), the section (1) questions were followed by section (2) questions. The survey was successfully conducted and the raw data values were downloaded and transformed to a scale that was easily used for statistical analysis. The participants were differentiated into experts and non-experts based on information obtained from the survey demographic. An expert was defined as an individual who has lived in Australia for over ten years, and had completed three years or more in Australian primary schooling. A t-test analysis of the results revealed that, performance of experts against non-experts in the three elicitation methods was indistinguishable.  The results obtained from the (p10-p90) method were superior to the best guess and guided feedback methods.   There was a greater likelihood of ranges being changed as a result of guided feedback. However, there was no statistical difference between negative and positive feedback. Amongst the trends observed, the positive feedback information narrowed the resulting ranges for most cases. In comparison, there was an equal likelihood of ranges being narrowed or widened, due to negative feedback information. 

Australian School of Petroleum



T: +61 8 8313 8000
F: +61 8 8313 8030