In the earliest days of information technology (IT), professional programmers used crude tools like COBOL and early relational databases (if you had big bucks: most of us used flat files) to write periodic (daily, weekly) historical reports. As a user you liked it or lumped it; IT was too busy doing “important” transactional work to get to your piddly departmental analysis.
Enter the spreadsheet, a magic tool that empowered users to automate calculations, deal with massive datasets (for the time), and present beautifully formatted reports that looked so . . . correct. Users flocked to VisiCalc, then Lotus 1-2-3 then Excel because they freed them from waiting for IT.
Then the unintended consequences happened:
- Models grew so large (I once saw a spreadsheet with several million rows and a couple of hundred columns) that they became databases, without the data quality or integrity.
- Models grew in complexity as powerful formulas were added and as spreadsheets linked to other files that linked to other files, so debugging them became hit-or-miss, and auditing them — forget it!
- Excel models were shared via email which meant multiple versions floated around, creating “model drift” and multiple “right” answers.
- Companies realized that Excel security ranges from nil to awful: corporate data can wind up anywhere a sloppy or malicious user sends it or saves it.
Addressing the Need to Be Data-Driven
Individual users or departments that used Excel may have solved their problems, but they caused bigger problems around the organization. Multiple sources of truth and differing calculations meant that top executives were inadvertently shown faulty data, leading to bad decisions. Audit processes had to be created for complex, undocumented spreadsheet processes, which raised costs (even worse, these clunky processes might fail an audit).
In short, the local solutions introduced by spreadsheets caused global confusion. Today, spreadsheets aren’t robust enough to help leaders build or support a data-driven organization that has strong governance, visibility, and the insights to act based on data in real time.
Fortunately, businesses have a range of modern tech platforms — from the cloud to data lakehouses, from hyperautomation to business intelligence (BI) — that now overcome many of the data management and reliability deficiencies inherent in spreadsheets. In fact, I predict these new platforms will ultimately convince most customers to ditch yesterday’s spreadsheet.
Which companies are the most important vendors in data? Check out the Acceleration Economy Data Modernization Top 10 Shortlist.
The Spread of Shadow IT
As “Shadow IT” (or as I prefer to call it, “Rogue IT”) spread throughout organizations, IT itself was changing. Data was growing larger and more complex (in terms of the “Three Vs”: volume, velocity, variety); IT data analytics tools were growing as fast as the data was; data was being used to drive more important decisions rather than just report history; and hyperautomation tools like low-code, robotic process automation (RPA), workflow, and artificial intelligence (AI) were empowering end users without sacrificing governance.
Unfortunately, it is human nature to stick with the familiar rather than take a chance on change. Who has the time — and the budget — to send key team members for training in new tools when they already know Excel? And executives argued that Excel solved their problem, so why invest in new tools when they could just keep adding on to their existing Excel models? So we find ourselves with increasingly powerful and manageable tools being the last resort at many organizations.
Meanwhile, the Excel models just get bigger, less comprehensible and riskier.
First Step in Filling a Hole: Stop Digging
Folks, it’s time to change! Redesigning your Excel models using modern cloud-based tools will improve your own local processes: improving data quality; reducing time and manual work needed to access data from corporate files; simplifying the testing and debugging of models; and reducing your audit costs.
In addition, you’ll benefit the organization as a whole by quickly and securely sharing your models, automating updates to reflect changes in data, and creating standardized building blocks that others can integrate into their models, improving data consistency and reducing confusion.
Here are some steps to start on your path to modernization and becoming a truly data-driven organization — and how your company will benefit along the way:
- Begin with a modern cloud data lakehouse; two good candidates are on our Top 10 Shortlist of Data Modernization vendors: Databricks or SingleStore. These platforms combine transactional power with analytic sophistication. You get one source of truth to drive processes as well as decisions. Combine that with data security/quality from a vendor like Rubrik, another Data Modernization top 10 vendor.
Result: You can stop using Excel as a database and work from one single, and secure, source of truth. - Also, mix in analytics tools such as those from other Data Modernization Top 10 vendors — Qlik and Informatica are two strong candidates — to build powerful, sharable, secure reports and dashboards that serve up single versions of the truth across the entire organization.
Result: No more model drift, no more inconsistent data or dueling reports, and vastly reduced audit costs. - Add hyperautomation in the form of a low-code tool like Automation Anywhere; a process mining tool like Celonis; or an AI tool like C3 AI (all Hyperautomation Top 10 vendors).
Result: Users (citizen developers) can manipulate and act on data using powerful tools that obey security and governance rules, and can collaborate with professional developers (IT) when more complex solutions are needed.
Final Thoughts
Are the tools I reference above as familiar as Excel? No.
Are they as easy to sit down with and throw together a quick and dirty model, similar to what you can do with Excel? Also, no. (Of course, Excel is probably fine if your need is only to build a quick model for personal use).
But these industrial-strength tools are cloud-based, visually oriented, and easy to get started with. And, speaking as a CIO who has to follow in the path of, and clean up after, departed Excel model builders: these tools are safer, better documented, more secure, and auditable.
And in today’s Acceleration Economy a data model that’s wrong, or insecure, or untrustworthy risks considerable damage to your organization. And that’s what happens when Excel is used for serious business decisions. So, join me in ditching spreadsheets!
Want more insights into all things data? Visit the Data Modernization channel:
