For all the effort technology businesses put into rapid scaling, success can sometimes be double-edged for organizations that aren’t prepared when things go right. In other words, intended growth and early achievements can be an especially difficult IT challenge in their own right.
From the early start-up phase of our company, a provider of a cross-network digital advertising platform, we adopted a strategy that relies on external providers so that we could concentrate all possible internal resources on the crucial work of product development. The approach has worked well, enabling us to build the differentiated solution that we wanted without focusing too much on back-end IT. A particular case in point throughout this growth is that our extremely data-dependent platform has utilized the services of external database providers, with one notable shift worth discussing.
Database functionality is absolutely critical to the reliability and performance of our digital ad analytics and automation solution. Our customers are advertisers seeking insights that must be culled out of vast amounts of data, and our platform must continuously collect and analyze ad performance information from five major advertising networks. A single new customer may quickly collect terabytes of data for this purpose, requiring a database capable of seamless scalability to do the job right. At the same time, the availability of the database is a make-or-break factor for our solution, as downtime or high latency render it ineffective. These needs were intensified throughout our early growth phase by our rapid addition of new customers.
As a start-up, we vetted Apache Cassandra and determined that the NoSQL database could deliver the performance, scalability, and high availability we needed. Initially, we relied on the DataStax Startup Program, which provided the proprietary DataStax Enterprise version of Apache Cassandra. Use of this service was free for companies that could be defined as start-ups, and according to the terms of the program, remaining beneath a certain threshold of growth was necessary to avoid substantial licensing fees from kicking in. At this time, we also relied on Instaclustr as a service provider to manage our Apache Cassandra clusters.
As you might now see coming, it wasn’t too long before growth got in the way of our database status quo. The escalating data needs of our solutions led our production deployment to scale up to almost one hundred nodes. We were quickly approaching a size and level of use where we wouldn’t be able to use DataStax for free anymore. And while we would no longer technically be a start-up, paying the high fees associated with being a full-grown company wasn’t a sustainable option for us. At this point, we’d also had our eyes opened about the vendor lock-in we would face if we proceeded with this proprietary version of the database, which only added to concerns over future costs and capability issues. This reality – coupled with the fact that DataStax Enterprise offered no persuasive competitive differentiators to justify its costs and limitations versus the open source alternative – led us to begin planning a full transition over to open source Apache Cassandra.
We turned to Instaclustr to execute this transition, and the cloud database provider successfully moved our data into its new open source home with zero downtime, sparing our customers from any performance interruptions. Our platform continues to benefit from the many advantageous features of an Apache Cassandra deployment fully under our control, even as it’s now grown to include eight clusters and more than 100 nodes. As managed today, Apache Cassandra provides our platform with the low-latency and high-throughput data access that our clients require. Apache Cassandra’s architecture naturally promotes high availability by featuring no single point of failure, a fact that has enabled our platform to avoid downtime incidents. And with regard to scaling to serve new clients, Instaclustr’s dynamic scaling capabilities allow us to simply add new nodes to meet new demand.
Ultimately, our managed open source Apache Cassandra database deployment enables our business to continue to concentrate internal resources on our product and our customers’ needs – a recipe we hope will allow us to continue to grow.
Jason Wu is the co-founder and CTO of AdStage, a platform that connects digital marketers to the data and tools they need to succeed. Besides leading the engineering effort at AdStage, he also loves running and is constantly learning about new technical topics ranging from security to artificial intelligence.