Regardless of the stated advantages around quality, functionality and stability, most technology innovations never see widespread adoption for one reason: end users’ resistance to change. A recent study of Fortune 500 companies showed that nearly two-thirds of all major changes in organisations fail due to resistance1. Despite an industry shift that has seen data become a valuable asset to most organisations, data analytics and business insight still occur on the desktop with Microsoft Excel. Excel is the workhorse of data analytics and we are faithful to it, no matter what compromises on speed, ease of use or scalability it forces us to make.
Today, Excel continues to reign over enterprise class business intelligence tools. According to Gartner, 70 to 80 percent of business intelligence projects fail2. Excel should have been superseded long ago, if one considers its pure features and functionality, especially if assessed in today’s data rich enterprise environment. We have all been there, in a fog of pivot tables, pie charts and the dreaded ‘hourglass of doom’ as formulas are calculated and data is churned. Why are we all so reliant on Excel as our everyday data repository, analysis tool and engine?
The Limitations of Excel
Excel wasn’t designed with today’s working practices in mind; it can’t analyse at breakneck speed, it can’t even slice and dice in line with users’ needs, yet salespeople, scientists, marketers and CEOs can be seen pouring over their laptops as they shoehorn data into a programme that was first designed in 1982. Trawl through the many Excel forums and you will find posts such as this:
“Currently working on 552MB file… and it only contains the first 3 months worth of data for the Financial Year!!”
As a database application, Excel fails on tests for quality and accuracy of data for one simple reason: it is subject to errors of human input. It is not the most superior collaboration tool – editing is difficult in shared mode. As it is mainly an office solution, capacity for interaction with web-based solutions is limited. Reporting capabilities, as well as safety and performance, are also lacking.
Excel is based on Office files which, as widely known, duplicate, copy themselves, stick and multiply to ensure that there are several different versions of the truth. But because Excel deals in data, there should only ever be a single instance and output.
With Big Data, “good enough” is no longer satisfactory
I am sure some of you reading this will say that Excel does the job. Indeed, ’good enough’ has become the reason for Excel’s longevity. However satisfactory will not provide any business with competitive advantage. Data is power. Today, it is collected and harvested in enormous volumes. Tomorrow, we will not just store data, but study it in real time to provide immediate value. Once the value has been extracted, we will be shortening its life to preserve storage space. Solutions that are able to respond to the needs of today’s businesses are already a reality. Actian’s Vectorwise, for example, can simplify data preparation and data management; the solution shortens the development delivery cycle while enabling business managers to run queries faster and glean immediate insight.
Thanks to improvements in terms of machine to machine (M2M) technology, there will soon be automated decision making generated by alerts in real time databases. Humans will become supervisors whose role is to moderate (from a mobile application or web interface) the automated decision making process. Technology evolves at breakneck pace and we can soon expect to see advanced mechanisms being put in place to simplify processes. The time has come to break Excel’s monopoly and herald a move towards powerful, fast databases that are able to free up the business ‘oil’ that data has become in terms of its value.
Of course, from a policy maker’s point of view, this is a form of risk-taking as management will need to combat users’ resistance to change. Certainly, those companies for which Excel really is ’good enough‘ and ‘cheap enough’ have no reason to change. It is for the ones which depend on data analysis to create value that the change is needed. For the latter, it is time to leverage the power of analysis of large data volumes. It is also time that these analyses became available on existing commodity hardware, on a laptop or mobile devices.
All this power is now available and is waiting for companies to wake up to the limitations of Excel and explore the alternatives to the famous spreadsheet.
1Beyond the Wall of Resistance, Rick Mauer (http://www.rickmaurer.com/wp/articles-and-white-papers/resistance-to-change-why-it-matters-and-what-to-do-about-it)
2http://www.enterpriseappstoday.com/business-intelligence/why-most-business-intelligence-projects-fail-1.html