Over the years transactional systems have served as the backbone of information processing. The need for robust records management often influenced what underlying database management systems businesses used. Row-based stores were thought to be ideal for being write-optimized—so data could be written reliably. This reliability made them good candidates for running transactional systems like ERP and CRM.
Unfortunately, the price paid in this setup is the I/O latency; system performance suffers when there are many requests to the row-based systems for data and analysis of that data. Demand for data analysis called for a technique that would mitigate this weakness, so that information could be read fast and insights delivered quickly. To accomplish this, a number of techniques were employed. Data marts, and the curating of data before being subjected to the analytical task, sped up the process by analyzing only a subset of the data stored. Another technique was the use of columnar stores which are better optimized for reading data.
Such efforts show that the need for separate handling of transactional and analytical workloads has been more of a technology-driven matter than a business decision. It is inconceivable that an executive woke up one morning and said, “I need to have different approaches to handling my transactional and analytical workloads.” The fact that we have often talked about OLAP and OLTP as separate topics is because we have traditionally been saddled with the limitations of disk-based database management systems.
Now we are in a different era, however. It is a time when data and events are more likely to be streamed than recorded, and business models depend on real-time input and massive quantities of data. The question many CIOs are asking is, “What data—and how much data—are relevant for insights that can lead to actions resulting in business benefit?”
Answering this question gets to the heart of an important business-technology issue. Irrespective of the nature of your data store, you need to address the fact that the need for relevant data may limit how much you can prepare data before it is analyzed—how much you can prefabricate data—and still deliver a complete response. This need only grows when companies are working with live raw data in a real-time environment.
A real-time data platform encompassing true in-memory database management can answer this need. Designed right, this platform allows an enterprise to house all data in-memory in a column store, obviating the need to create a data mart or partial dataset and enabling access to data in real time. If the in-memory database management system can combine the capabilities of row- and column-based stores, it will be able to handle both transactional and analytical workloads. That means supporting both ERP and operational analytics needs using one comprehensive dataset.
Such a system can also power analytics that take advantage of a range of data types and sources such as social media data or unstructured and text data. In fact, a whole new paradigm for designing and building new apps opens up if you take advantage of in-memory computing that leverages complex processing within the database layer. These are the qualities that a real-time data platform should possess.
What does this mean in terms of business benefit? It dramatically simplifies the IT landscape. Additional data marts, efforts to curate the data, and duplication of stores can all be eliminated, freeing up enterprise resources to focus on other needs. And if the in-memory system is engineered with openness in mind, it has the potential to work across a range of hardware vendors.
The ability to handle both transactional and analytical workloads—for the same system to support multiple backbone applications like ERP, CRM, and SCM and an analytics suite at the same time, is significant. This is a real choice that enterprises didn’t have before. A real-time data platform, with the qualities described here, offers an opportunity to improve an enterprise’s agility, to reinvent or redeploy business models that could positively impact their value chain. This would include not only gaining new performance improvements in existing processes but also the ability to dream, design and deploy new processes previously not possible because of database and processing constraints.