Better Project Planning With Prediction Markets

As I understand it, much of Coase’s work focuses on why companies’ internal structures don’t resemble markets all that much. Instead, most firms have feudal-ish structures. The CEO has, at the discretion of the board, final decision making power on nearly all decisions, and then their power trickles down recursively to their subordinates for decisions in which they’re not involved.

While I’m willing to swallow the bitter pill that feudal-ish structures are actually more efficient for many company functions than democratic ones would be, in my own experience, I’ve observed a few places where I think market-based decision making systems could improve both company efficiency and employee satisfaction. In this post, I’ll discuss two problems that seems common in large corporations and how I think prediction markets can help solve them.

Note that much, if not all, of what I’m about to discuss has likely been written about or at least thought about by Robin Hanson, but I still want to have this post as a concise proposal and reference to come back to. Later on in the post, I link to a bunch of Hanson’s writing on the topic, but I also wanted to call out the derivative nature of my proposals explicitly.

Why are projects always late? #

While people agree on little in software, there’s a rough consensus that software projects rarely finish on time. Traditional proposals for solving this problem focus on inside view driven solutions. Agile methodologies, for example, recommend a combination of faster iteration cycles, better estimation systems, and practices that encourage continuous improvement and reflection. Agile true believers argue that, if only more teams truly stuck to these practices, our industry would have escaped its late project reputation long ago. As far as I can tell though, these true believers don’t address the obvious follow up question: then why haven’t they?

Coming from my software practitioner, AI/econ dilettante perspective, I see the problem through a different lens. Employees and teams will only adopt better practices if they’re incentivized to do so. From what I can tell, most employers don’t actually incentivize accurate project estimation. People pay lip service to it but actual rewards don’t consistently flow to the best project estimators.

Instead of focusing on how to improve estimation, I propose we first figure out how to incentivize it. With the right incentives, employees will figure out how to make better estimates or allocate resources towards doing so.

Internal project prediction markets are a promising approach for incentivizing better estimation. As Robin Hanson’s discussed extensively (more), prediction markets work better than polls and other well known opinion aggregation methods for assigning probabilities to future binary outcomes. (Hanson even implemented a basic version of the idea I’m about to discuss a long time ago!)

A project prediction market would work as follows. A company could have an internal web application—see PredictIt for an example from a different domain—which would track predicted project timelines and give participants the opportunity to make predictions about them by buying “shares”. For example, if I’m working on a project to rewrite my company’s image processing system, a hypothetical project prediction market would link to my project proposal and summarize important details about it such as its predicted launch date, the number of people working on it, and any other relevant metadata. It would then have “shares” issued for different predictions about its actual launch date. Each participant would aim to beat the market by buying and selling shares of predictions that they think are under-/over-valued. If the market functioned correctly, market prices would converge to accurate estimates of probabilities of different predictions coming true given the available information at the time of prediction.

As long as prediction market success resulted in some tangible reward for employees, I predict that the average large software company could get enough employees to participate to make the market’s predictions of project timelines much more reliable than teams’ predictions of their own project timelines. In this world, technical and non-technical stakeholders would have much stronger incentives to produce accurate timeline estimates or otherwise to defer to the markets. Anecdotally, estimates currently serve an additional signaling role whereby individuals make aggressive estimates in order to show how hard they’re willing to work (I certainly feel this tension when making estimates myself, and I’m not under substantial pressure to compress estimates). In a company with internal project prediction markets, this tendency would be counter-balanced by the knowledge that the market will presumably factor an aggressive estimation track record into its prices. I suspect that the knowledge that a prediction market’s there to keep people honest would influence many employees’ behavior on its own, but companies could further incentivize estimation accuracy by highlighting teams whose estimations consistently get priced as accurate by the market. On the surface, this may seem very similar to rewarding teams for good estimates directly, which could be but isn’t done by any company I know of currently. However, incentivizing accurate market prices would provide a faster feedback loop since it wouldn’t require waiting for projects to finish to assign rewards.

Project prediction markets would improve company efficiency by providing more accurate information for employees and managers about project timelines and realistic resource needs, while reducing employee stress by counter-acting the tendency to compress estimates and then race to “catch up” later on in the project life cycle.

How to decide which projects are valuable? #

Much has been written about how forcing decision makers to achieve consensus before acting can be detrimental for a company’s effectiveness. Much has also been written in praise of consensus building. So what should we do? How do we decide both inside of and between teams what to prioritize?

One way to think about this is to ask why we struggle to achieve consensus in the first place. Theoretically, as employees of the same company, we all share the same long term goals. In practice, uncertainty and mis-aligned organizational priorities make achieving consensus hard. Solving the mis-aligned incentives problem is out of the scope of this post, but reducing uncertainty is something we can tackle.

The most common solution to deciding when and from whom to seek consensus seems to be relying on common knowledge and internal processes to make a contextually-driven decision about who to get buy in from. I fear that this status quo rewards employees for understanding implicit political structures and doesn’t converge towards optimal decisions over time.

To make this more concrete, imagine your team is asking another team for resources for a project your planning for the next quarter and this other team is arguing they have another project to work on that’s more important. If you had perfect information, you would simply look into the future and see which project turns out to have benefited the company more, perhaps by increasing profits, attracting users, preventing problems, etc. In lieu of that, we’d like some mechanism to estimate likelihoods of different outcomes and pick the one that looks the best. This would help us avoid having the team that yells and politics hardest win the debate.

At our hypothetical company where we’ve already adopted prediction markets for timeline prediction, we could easily expand our internal prediction market to also forecast likelihoods of different product outcomes. Going back to my x-team resourcing example, if our team and this other team disagree on which project will attract more users, we can feed something like, “in 6 months, project A will be responsible for attracting more users than project B, as measured by the conversion numbers from a month-long A/B experiment.” Then, we can use the market price as a proxy for how likely the different outcomes we were previously just arguing hypothetically over are and make a more informed decision.

In addition to leading to better decisions, this would hopefully reduce cross-org and cross-team tensions by reducing politics’ influence on prioritization decision outcomes.

Conclusions #

To summarize, I’m proposing that large software companies could incentivize better project estimation and internal prioritization by experimenting with internal prediction markets for project timelines and outcomes. I believe adoption of prediction markets for these purposes would improve both employee satisfaction and company efficiency in most medium to large companies.