I’ve had a microsite lurking around here for some time and I thought I’d acknowledge it as a legitimate child. The site is a proof-of-concept for questions that elicit the subjective beliefs of respondents. For example, let’s say I wanted to ask someone how much an average Uber driver makes per average hour. I could just ask you to give me an estimate like, “$15.” I’ve come across hugely-varying estimates. A paper by authors that included Uber employees put mean earnings at $21.07 (Cook et al., 2018, Table 1). An independent survey by Ridester put the figure at $13.70 (pre-tip). I haven’t tried to ferret out the source of the disparity. It may have to do with differing normalizations of the cost of living across cities. It could also just be a survey issue. But think about what happens in your head when someone asks you that question. You probably weighed a lot of factors then answered with something like the mean of your beliefs. If I just ask you for a point estimate (i.e. one number) then I’m potentially leaving a lot of information concerning the extent of your uncertainty on the table. Are you almost certain it is $15 or do you think it could pretty much be anywhere between $10 and $20? Do you maybe think it is either $15 or $17? Each of these cases generates a whole new interpretation of question response. The site demonstrates a method of capturing all of this uncertainty.

I encourage you to quit reading right now and check out the question so that you’ll have a better idea of what I’m talking about below:

Go to the demo site.

The tool allows users to draw their beliefs. From the user’s point of view, they are just drawing a line. If the user thinks that values in a certain region are relatively likely, then they give the line a higher value in that area. In regions that have probability zero (i.e. are “impossible”) the user draws the line so that it is at the bottom/zero. Each time the user updates the line they are provided with new summary statistics (e.g. mean, median, standard deviation). Under the hood we’re using mouse drag events to construct a piecewise-linear probability distribution, which has analytically-tractable moments. My hope is that users with any statistical background will be able to use this tool. They don’t need to know what’s going on under the hood. They just need to understand the basic idea that “high lines → likely; low lines → unlikely.”

Business Use Cases

There’s a lot that you can do with this information in a business context.

One approach is to hook responses up to an automatic decision rule. This is what the (somewhat stupid) toy example does in the demo app. In short, it compares expected profits under two different prices and tells you the conditions under which it is optimal to make a price change.

A second use case involves communication of beliefs across a team. Suppose that we’re planning a major project. Everyone on the team has a set of tasks and we ask them how long we expect each task to take. Once the results are collected the team pow-wows and looks at the results. Are there any surprises? If so, we can use the data to narrow in on where the team members have different beliefs and, more importantly, the different pieces of information that underly those beliefs.

Concerns Raised by Others

Let me address to points/concerns that have popped up among testers of the demo site:

The first issue is that the tester didn’t understand the analysis that was spat back to them after they completed the question. This may have a bit to do with my failing to adequetely explain the analysis in the text. However, in general, question respondents need not understand the analysis that their responses are used to create. In many important use cases the respondent will never see the subsquent analysis, which will be carried out by an outside analyst.

The second issue goes like this: “I could see how this might be useful for company X because they have a pretty good idea of what they’re going to sell this year. For my company it is just too hard to tell.” I would suggest that companies in both situations could use something like this but that a company with highly uncertain sales/profits/input costs/whatever would especially benefit from it. A public company with uncertain profits can’t opt out of earnings forecasts because it is “just too hard.” They have to report something. This tool can aid in that process by making the extent and form of uncertainty more concrete and amenable to formal analysis.

Conclusion

I won’t say much more about the project here. I’ve written much more about use cases and implementation on the site itself. This is very much a work in progress. Any comments or questions are appreciated.