Indeed.

If you are responsible for a project that has a spreadsheet as a deliverable, go and suspend it now. Yes, now. Go on. When you return I’ll explain why you had to do it.

Done? OK, I’ll continue.

First, let me change tack for a moment. I’m actually a big fan of spreadsheets. I use them regularly. Excel is a great application. However…

spreadsheets should not be used for “line of business” applications.

In fact, I’d go further:

spreadsheets should only be used by the person who created them.

And, sometimes, not even then. We’ve all seen spreadsheets like the one below.

This isn’t a spreadsheet. It’s a database. And there are applications designed specifically to handle this sort of data. They’re called “databases”.

The “spreadsheet as database” is a pet peeve of mine as I receive data in this format all the time—and it’s always a major pain. Consistency is hard to maintain when storing data in spreadsheets. You end up with customer IDs in the “Orders” sheet that can’t be found in the “Customers” sheet. Or a sales tax of “Smith”. A particularly pernicious problem with using spreadsheets in this manner is their tendency to autoformat fields. If I had a dime for every time a numeric customer ID with leading zeros had been formatted as 6.72512E+11…or an order had been fulfilled during the reign of Queen Victoria…

Also, there’s a significant risk of messing up your data when it’s held in a spreadsheet. Miss the fact that a sort hasn’t included all of your columns and your records could be rendered worthless.

Even if you’re maintaining a small database for your own use there are more appropriate solutions than using a spreadsheet. You don’t have to become an Oracle database administrator. Personal database products like Bento are as easy to use as a spreadsheet. If you have a spreadsheet “database” in a corporate setting, and it’s doing more than managing your lottery syndicate, then someone needs to be fired.

So, OK, it’s not a good idea to use a spreadsheet in place of a database. But, spreadsheets are perfect for managing your accounts, right? Columns of numbers, balances, etc. Ideal.

Well, not really. The problem is that there are all sorts of rules and structure that can be used to help you manage your accounting—and spreadsheets don’t utilize any of them out of the box. A specialized accounting application will check that accounts balance, report outstanding invoices/bills, calculate taxes, etc. You can build some of these features into a spreadsheet, but why do that when the work has already been done by experts? Also, no-one is going to be au fait with your custom spreadsheet, so when it comes time to share the accounts with a colleague, or your accountant, there’s scope for confusion and mistakes. If you use a professional accounting package, however, it will have a structure that is familiar to anyone who has experience of business accounting.

Another problem with the use of spreadsheets in accounting is accuracy—a problem that will pop up again and again in this article. It is all too easy for a newly added row not to be included in a summation formula. Or for a tax rate to be changed inconsistently.

Unless your spreadsheet is being used by your lemonade stand business to see if you can afford that hamster, buy Quickbooks.

This seems like an appropriate point to address, in more detail, the lack of accuracy in spreadsheets. Sarbanes-Oxley and similar laws have resulted in more scrutiny of financial planning systems—and consequently, more scrutiny of corporate spreadsheets. Research, such as that reported by Raymond Panko in “What We Know About Spreadsheet Errors”, has found that most of the spreadsheets used by organizations contain errors—and that a considerable number of those errors are serious. In one case reported in Panko’s research, the error would have caused a discrepancy of more than a billion dollars! Similarly, an article by Mark Ward in New Scientist reported that Coopers & Lybrand found errors in 90% of spreadsheets they audited. Finally, the European Spreadsheet Risks Interest Group maintains a list of horror stories about real-world losses due to spreadsheet errors. Don’t read it with the lights off.

Why are there so many errors in spreadsheets? To be fair, spreadsheets aren’t the only models that contain errors. We all know that software has its fair share of bugs. But the sheer number of spreadsheets, coupled with their “homespun” development, and the difficult of reviewing their logic, makes spreadsheet development the Wild West of the modeling community.

It’s extremely difficult to develop accurate spreadsheet models. Professional software developers liken it to programming in ancient languages that rely on the notorious “goto” statement. And then there’s the challenge of reviewing formulae that contain “variables” such as “JG271”. Paradoxically, familiarity with spreadsheets encourages inexperienced developers to “have a go”. While I applaud this adventurous spirit, it doesn’t result in well-crafted models.

The software development community has invested extensively in languages, frameworks and tools to improve the quality of software. In comparison, relatively little work has been done to improve the quality of spreadsheet modeling—and the work that has been done has had minimal impact on spreadsheet users. People find it difficult to maintain an understanding of complex logic, so they need to break large models down into simpler components. This isn’t as easy to do in a spreadsheet as in, say, a modern object-oriented programming language.

Spreadsheets become increasing complex when models contain conditional (if X then Y else Z) and looping (do X Y times) logic. All business analysts will have encountered Monte Carlo spreadsheets that had thousands of rows of automatically generated results—and involved lookup logic for using those results. Macros can be written to address some of the challenges of using spreadsheets, but extensive use of macros is a clear indication that a spreadsheet may not be the best platform for your modeling. You end up dealing with all the complexity of software development, but suffering the weaknesses of spreadsheets for your efforts.

It’s also all too easy to enter bad data into spreadsheets. A wrong keystroke and a formula is replaced with a static value, rendering the calculations meaningless. Freeform data entry allows you to destroy a model in an instant. Protected cells can only do so much. And, once again, if you protect entire sheets and have users interact with the data exclusively through macros, you’re not exploiting the native strengths of the spreadsheet platform in the first place.

Another (ab)use of spreadsheets that I’m compelled to address is their use in business models—strategic planning models, forecasts, simulations, what-if analyses, etc. These models tend to have an extended lifespan, over which they are “tweaked” extensively—by different analysts—and used as the basis for significant corporate decisions. These are probably the most challenge models of all to develop—and, therefore, the most prone to errors.

Consider this: how many business problems are actually a natural fit for a two– or three-dimensional grid of cells? The gap between the real-world and the modeling environment is known as the “impedance mismatch”—and, in spreadsheet modeling, it’s generally huge. The gap is generally smaller when a modern programming language is employed, as the software designer can map objects to real-world entities. Higher impedance mismatches place higher demands on the analyst, in terms of understanding the model. So, when dealing with an unfamiliar model—either due to its being a while since the model was used, or because it was originally developed by someone else—the chances of mistakes being made are correlated with the impedance mismatch.

If you are building a business model, use a specialist tool (e.g. a simulation application) or a general purpose programming language. That “little spreadsheet model someone threw together in a few days” will eventually cost you dearly.

Hold on? If spreadsheets are so bad at everything, should we be using them at all?!

Who said they were bad at everything? Certainly not me. I’d even go so far as to paraphrase the NRA and say that spreadsheets don’t create bad models—people create bad models. And spreadsheets are actually excellent for one very important business activity—prototyping. Prototyping is when you build a model for the purposes of exploring an idea. Or for checking a theory. It basically building a model that you’ll discard once you have your answer. Prototypes have a short lifespan and are only used by (or in collaboration with) the original author. They are never used in production.

Spreadsheets excel (no pun intended) in this activity. They are very inclusive, as everyone knows how to use them. And because a lot of thinking about the process they are being used to study is going on, mistakes are less likely to have grave consequences. They can also form the design specification for a more formal model, as the subsequent development process should place more rigorous checks on the logic.

In fact, one wag suggested that spreadsheets would be used more effectively if the save function were removed. I have some sympathy…

OK. I’m guilty of having been a little provocative in this article. But, only a little. If you are using spreadsheets for anything more than individual prototyping in your organization I urge you to seriously consider replacing them with more suitable models.

Porting your spreadsheets to a modern programming language, for example, will:

• result in a design that is a more natural fit to the real-world problem, and, therefore, produce a more accurate model
• allow the use of standard testing tools to improve the quality of the implementation
• allow the use of source code control systems to manage on-going development

There’s a stack of evidence that suggests your spreadsheet models are ticking time-bombs.

Please, don’t “solve” one problem by making it into two.

Budget Playground allows school district planning staff to forecast budgets up to ten years into the future and interactively modify different scenarios to create robust budgets that are acceptable to the school district’s stakeholders.

The iPad app is fed by a REST web service that exposes historical trend data collected and managed by the Center for Rural Schools.

The Center for Rural Schools currently has a web portal that presents the historical data in a form that can be consumed by school administrators. One of the center’s objectives is to decouple the data service from the tools that present that data. This would leverage the existing investment in data by allowing the development of an array of highly focused “client” apps that help school district administrators overcome their most urgent challenges.

Budget Playground represents a trial of this concept.

“The passage of a property tax cap in New York State creates an urgent need for school districts to have new tools to manage the budgeting process,” said Dr John Sipple, Director for the New York State Center for Rural Schools. “The Center for Rural Schools can now offer an intuitive, easy-to-use budget forecasting tool to all school districts in New York State. Budget Playground solves a pressing problem, without being burdened by unnecessary features or complexity.“
For more information contact Dr John Sipple, New York Center for Rural Schools (inquires@nyruralschools.org) or Sean Brady, Prism Decision Systems (Sean.Brady@prismdecision.com).

Potential future developments of budget playground include:

• collaborative planning features to allow groups to work directly on scenarios together
• benchmarking capabilities to allow school districts to compare their budgets to those of similar schools
• expansion of the data service to include, for example, national price indices
• user customization of the reported metrics

Andrew Tait presented at the International Executive Seminar in Political Management and Governance hosted by George Washington University’s Graduate School of Political Management. The theme of the conference was “governing in turbulent times”. Andrew’s talk covered stakeholder management (including Confrontation Analysis) and the development of political strategy.

The conference was attended by senior politicians, civil servants and representatives from NGOs.

Oh, and did we mention that districts where residents have the lowest average educational achievement tend to be the smaller ones. Yep. That’s right.

Eh? How does that work? Well, it’s a consequence of the higher variability of averages in smaller groups.

In a city like London, the average IQ will be close to the UK national average (probably about 100). Granted, for London, it may be a fraction higher, due to the likelihood that an international city is a draw for talent—but it won’t be much above the average.

However, imagine a picturesque hamlet where all nine residents have average IQs. A successful entrepreneur with a genuis-level IQ (say 160) decides to build his dream house there and move in. Suddenly the average IQ of the hamlet is now 106. If the entrepreneur had moved to London the average IQ of the city would have changed imperceptibly.

Relying on averages alone is misleading. We need also to consider sample size.

Let’s take another example. Imagine there’s a software development project that has three components—a database, a server application and an iPhone application. All three components are essential parts of the overall system.

Each component is assigned to a separate team and all are asked for estimates of how long their projects will take to complete. For the sake of simplicity, we’ll assume that they all say eight weeks—which we’ll interpret as being a 50% chance that the component will be completed within eight weeks. So, there’s a 50% chance that the entire project will be completed within eight weeks, right?

Well, that’s how it would probably be reported by many project managers, but it’s wrong. In fact, there’s only a 12.5% chance that the project will be completed with eight weeks. All three sub-projects have to go well for the project to deliver within eight weeks. So, it’s highly likely that the project will miss its deadline. It’s impossible to say by how much, as the sub-project estimates are single-point estimates, as opposed to (more realistic) distrubtions—but that’s a topic for another time.

Clearly, these are fairly simple examples. But this kind of “average” thinking is going on every day. And, as the importance of data as a decision-making tool grows, sloppy analysis is going to increasingly undermine the value of good data.

Organizations everywhere want to make similar use of their own data. Articles and conferences on “predictive analytics” and “data science” are popping up everywhere. Even the New York Times has been promoting careers in statistics as “cool”.

But, are we learning the right lessons from the successes of Amazon et al? Should decision scientists be focusing their efforts on helping organizations make sense of the data they have?

Massive datasets are a byproduct of something the showcase “big data” companies do that is, arguably, more important—they crowdsource data in real-time. Both crowdsourcing and real-time data collection are valuable. Together they are dynamite.

Making one small team within the organization responsible for collecting and “cleaning” the “official” data limits the volume of data that can be collected and increases the possibility of bias. Crowdsourcing mitigates those problems—and is cheaper.

The quality of data decays over time. Different industries experience different decay rates, but basing decisions on old data risks missing fundamental changes. Obviously, lagging data is almost useless when responding to a crisis. Real-time data collection means decisions can take the immediate situation into account.

Before you ask how you can draw insights from your existing databases, it may be advantageous to ask how you can build higher quality databases in the first place.

Tools to assist decision-makers are increasingly drawing on existing data and then combining it with the decision-makers’ assumptions and beliefs to predict outcomes and suggest action. These assumptions and beliefs are then often discarded once the decision has been made. However, these are real-time insights from the front-line. Capturing and storing them would allow decision-makers to tap into the current views of their peers—and monitor shifts in these views over time.

In addition to designing decision-making tools to produce insights, we also need to design them to collect insights. The latter activity may be the real innovation.

Privacy

Of course, there are potential privacy implications to be considered in crowdsourcing data. However, collecting data about an organization (as opposed to individuals), in the course of paid employment, and with full disclosure, raises few privacy issues. It is similar to writing and publishing a business report.

In the “knockout” stages of a football competition each match must produce a winner. If the teams have scored equal number of goals (after additional playing time), the game will be decided on “penalties”. A penalty is a free kick taken from directly in front of the goal, with only the defending goalkeeper between the penalty-taker and the goal.

A professional football player will put the ball in the back of the net in about a quarter of a second. This doesn’t give the goalkeeper enough time to respond, so he must anticipate the likely trajectory of the ball—for example, by diving to the side that the penalty-taker tends to prefer.

Of course, the penalty-taker wants to mislead the goalkeeper. While the penalty-taker may be stronger shooting, say, to the right, doing this every time would make his actions too predictable.

So, the goalkeeper and penalty-taker must optimize their own strategies without knowing the strategy that will be adopted by the other party. This is the type of decision-making activity that can be effectively studied using game theory. The players are attempting to optimize their “payoffs”.

Using game theory we can see that the optimal strategy for both players is to behave unpredictably. They must balance their strengths (e.g. the penalty-takers’ strong side) with the value of potentially misleading the other player. But how does this help your team lift the FIFA World Cup Trophy?

Well…it turns out that most professional football players are also expert economists. A few years ago Ignacio Palacios-Huerta demonstated that top players act in accordance with the theory—i.e. they optimize their penalty-taking strategies. While this may come as a shock to some fans, one can’t argue with the science…

Maybe, once the World Cup is over, we can convince some of the players to pop into their respective finance ministries and offer a bit of advice…

“Odds Are, It’s Wrong”, by Tom Siegfried, highlights some of the problems associated with testing hypotheses using statistical methods. These problems are well known within the statistics community, with “hundreds” of papers having been written on the subject. As Siegfried pithily observes, “if you believe what you read in the scientific literature, you shouldn’t believe what you read in the scientific literature”.

Statistical analysis is often presented in support of decisions. It’s rarely challenged. Yes, it’s sometimes ignored if it contradicts the views of key stakeholders. But it’s rarely challenged. This, we would suggest, is a consequence of:

• statistical analyses being considered to be objective; and
• decision-makers not being confident enough to dig around in the details.

Combine the reluctance to challenge statistical analyses with their widespread abuse and you clearly have cause for concern.

The article points out that, even among scientists, statistical literacy leaves much to be desired. For example:

• Statisical significance (e.g. p < 0.05) is often presented as a “black or white” binary concept. However, the use of 0.05 (…or 0.01…or 0.001) is completely arbitrary. In fact, with a p value of 0.05, there’s a 1 in 20 chance that the observed result is a fluke. Is that acceptable given the decision you have to make?
• Statistical significance at the 0.05 level is commonly equated to 95% certainty that the result could not have occurred by chance. This isn’t the case. You can’t draw conclusions about the likelihood of the hypothesis being correct based on its statistical significance. The correct interpretation, given a p value of 0.05, is that there is only a 5% chance of getting the observed result if no real effect is present.
• Studies also tend to equate statistical significance with practical significance. An example given in the article is that an expensive new drug may be statistically significantly better than an old one, but, if it only provides one new cure for every 1000 patients, that’s not of much practical use.

A final interesting observation made by Siegfried concerns random trials. Selecting groups at random provides no guarantee that they exhibit random traits with respect to the phenomenon of interest. Let’s say we walk out onto the street right now and select two groups of five professional strangers. How likely is it that both groups would express similar political views, for example? When discussing (as Siegfried does) drug trials, there are countless dimensions on which patients can vary.

Humorist Evan Esar was clearly onto something when he defined statistics as “the science of producing unreliable facts from reliable figures”…

Information is increasingly accessed via smartphones. That’s in no doubt. But does this mean that you need to rush out and build a smartphone version of your corporate accounting system?

There are clearly some situtations where the arguments for developing a smartphone app are compelling.

• If your customers are consuming your content in their leisure time, then it probably makes sense for you to provide them with a smartphone application. Sports scores, articles, event listings, products reviews, etc. are all likely to be increasingly accessed via mobile devices.
• If you’re selling lower priced, “impulse-purchase” items then you may benefit from developing a smartphone app. It’s unlikely that you’d need to allow customers to purchase a car or house over a smartphone (although you’d probably want to let them research such purchases via this medium — e.g. list cars on the lot and book a test-drive).
• If you are providing real-time services to customers who aren’t desk-bound, then it’s worth considering the development of a smartphone app. Providing medical professionals with access to patient data may fall under this category.

There’s no doubt that a wide range of organizations could (and will) benefit from the introduction of a smartphone app. However, this doesn’t mean that every application fits this mold.

Some tasks—like writing a lengthy report, buying a car, doing your company accounts, etc.—would seem unlikely to be well-supported via a “mobile experience”. These are better undertaken using a “traditional” desktop or web-based application.

In fact, the value of developing a smartphone app for a task may be inversely proportional to the importance, value or complexity of that task.

If your customers are likely to set blocks of time aside specifically to perform the tasks that require your products, then the chances are they will be able to arrange access to a more traditional computer (e.g. netbook, desktop) when performing those tasks. And, there’s no doubt that, with current technology, the smartphone experience still lags the traditional computer experience for most tasks.

The smartphone revolution is upon us. And, for many organizations, not having a smartphone app in place is nothing short of corporate suicide. However, that does not mean that all organizations have to dive in…yet.

One obstacle to effective decision-making is the inability of decision-makers to carve out the time required to give the decision due consideration. It is far too easy to get caught up in the day-to-day minutiae of running your organization (and life).

If you are constantly distracted by your “to do” list, you will never find the time for the analysis and reflection your major decisions deserve. A clear head is an essential prerequisite for effective decision-making. If you don’t have your basic workload under control, the quality of your decisions will suffer.

If you find that you need some help in getting your day-to-day responsibilities under control, there are numerous “personal productivity” solutions in the marketplace. At Decision Mechanics we’ve adopted David Allen’s “Getting Things Done”. It’s comprehensive and pragmatic.

The basic premise of “Getting Things Done” is that you need “de-clutter” your mind. If you’re relying on your memory to manage your “to do” list you’re keeping it occupied with information you’re not ready to act on. Our minds have a limited capacity to keep information in the foreground, so you’re wasting valuable mental resources that could be used on your major decisions.

To allow your mind to let go of things you must convince it that they will be dealt with, appropriately, at some point in the future. And to convince your mind of this you need repositories (e.g. folders, lists) that you store all your stuff in and review on a regular basis.

As stuff (e.g. letters, emails, ideas, appointments, tasks) enters your field of attention, it is placed in an “in basket”. This may actually be an in basket on your desk…or it may be a box file containing scraps of paper…or the inbox of your email client… In fact, you are likely to have more than one of these “in baskets”. For example, you may have an in tray at the office, a box file at home and your email inbox. The idea is to collect all your stuff in as few places as possible.

The next stage of the process is periodically (e.g. once a day) reviewing the stuff in your “in baskets”. You go through the items in strict order of when they were placed in the repository (no cherry-picking) and decide whether something is actionable. If not, you file it (as a reference, or as something to be prompted about in the future).

If an item is actionable—i.e. you need to do something about it (beyond throwing it away or filing it)—and that action can be completed in less than two minutes, complete it and discard the item. Completing one action may result in another task. In this case, add the new task to one of your “in baskets” (e.g. as a scribbled note).

An actionable item that takes longer than two minutes to complete will either:

• need to be done by a set date (implied by the action), in which case you put it on your calendar;
• need to be done as soon as possible, in which case you put it on your “next actions” list; or
• need to be delegated, in which case you make a note on your “waiting for” list.

Some tasks, such as projects, may require multiple actions to complete. In which case, you place the overall task on a “projects” list and add the next thing to be done on that project to your “next actions” list.

Your calendar, “next actions” list and “waiting for” list then contain your main tasks. When you find yourself with time available, you can return to one of these lists and choose an item to work on. The item you choose will depend on the time available, your frame of mind, etc.

The key to making the “Getting Things Done” process work for you is being disciplined in your review process. You must review all your repositories/lists religiously. If not, your mind will start holding onto things again. The one person you can’t fool is yourself.

This has been a necessarily brief introduction to Allen’s ideas, but we unreservedly recommend his book and website to those looking to organize their professional (or personal) lives so that they can place more focus on the decisions that really matter.

First, it is important to recognize that assisting decision-makers does not equate to replacing them. Have you closed your finance department since you purchased Excel? No? I thought not.

Software can aid decision-makers (primarily) by:

1. managing data
2. providing structure
3. encoding best practices

Managing data is the most obvious benefit that software can provide. Data analysis tools can summarize large volumes of data, allowing decision-makers to separate the wood from the trees. However, software can also assist decision-makers by managing their assumptions. We’ve all had the experience of constantly revisiting the same issues when involved in making complex decisions. If we use software to model each stage of the process, we can return to previously considered issues without having to recreate everything for the umpteenth time.

The provision of structure is something that is often overlooked. Without a formal decision-making model, it’s too easy to avoid facing important facts. Models ensure that we don’t get caught up in the excitement of a major decision and become blind to inconvenient data that doesn’t support the “sexy” option. Models don’t have to be encapsulated in software, but doing so helps promote disciplined behavior—you can’t “hand-wave” at a computer program.

Finally, good decision-aiding software supports the spread of best practices throughout your organization. Many powerful decision-making techniques take time to learn—and we all know that time is becoming increasingly precious. Decision-aiding software can guide decision-makers through the mechanics of a process, providing all decision-makers with access to the latest decision science techniques.

So, the next time you have to make a decision, consider letting software onto the team. It’s there to help…