A Lesson in Database Uptime Learned the Hard Way – and How It Changed Us

by   |   January 4, 2018 11:21 am   |   0 Comments

Vik Chaudhary. CEO, Dossier

Commanding the ever-growing volume of data required to successfully manage customer relationships and communications is an ongoing challenge for enterprises. Built to help companies organize this information, our company faced this issue during the development of our application – until an incident expedited a strategy overhaul.

Enterprises collect data through customer interactions conducted over many channels, from email, texts, website chats, in-app conversations, support channels, messengers and beyond. At the same time, enterprise teams use numerous workflow and customer relationship management (CRM) solutions, such as Slack, Asana, Salesforce and Zendesk, to keep track of tasks executed in response to customer requests. With Dossier, we set out to provide an application capable of centralizing the data from all these sources into a single database and interface, allowing enterprise teams to collaborate in real time and satisfy customer needs more efficiently.

Our core team approached the challenge of harnessing this data with a great deal of confidence. We had previously worked at enterprise software-as-a-service (SaaS) provider Dynatrace, where we deployed an Oracle database able to handle new records on the magnitude of 1 billion each day. As it turned out, our team had a bit too much confidence. Two years ago, at a moment early in Dossier’s development, I happened to be presenting the solution to investors. I claimed the application was so scalable that a large enterprise could have all its employees connect at one time and benefit from Dossier’s advantages within a half-hour.

Of course, I’d jinxed us – emergency calls began coming in immediately. After concluding the presentation as stoically as I could, I found out that a large enterprise with 20,000 employees did, in fact, just begin using our solution. Unfortunately, counter to my claim, adding thousands of users in 15 minutes resulted in our systems crashing. What came next was a 36-hour period where our team worked without rest until those systems were restored.

This episode came with a silver lining: It proved that Dossier was in demand as a solution and that it could be deployed across huge enterprises quickly. We just needed a database able to deliver the scalability and high availability required, given the massive amounts of data we were dealing with.

We had selected MongoDB, an open-source database program, for our database due to its NoSQL, non-relational nature. Doing so enabled our team to engage in faster product development, implementing new features without concern for schemas. We managed a single-node database cluster internally using Amazon Web Services (AWS), relying on the past database administration experience of our team members.

However, this experience was mainly in relational databases – we didn’t possess extensive knowledge when it came to managing multinode database clusters or scaling the throughput of our database without downtime. As Dossier continued adding new capabilities to interface with popular CRM and workflow platforms, the quantity of real-time messages being synced between users increased sharply. At the same time, our team’s level of MongoDB expertise produced an environment where database indexing and overall management weren’t getting the care they needed. In our hubris, we proceeded under these circumstances until the 36-hour downtime incident. In its aftermath – and following a bit of rest – we resolved to enlist outside MongoDB technology and expertise to ensure the proper management and monitoring of our database.

After comparing reviews (and even asking the creators of MongoDB themselves for advice), we found mLab to be a MongoDB database-as-a-service (DBaaS) provider recommended for its domain knowledge, platform capabilities and responsive support. Now with a hosted MongoDB solution in place, we soon tasked our new partner with preparing our database to handle tens of thousands of new user signups in just minutes, as we knew would be required if Dossier could succeed as an enterprise solution. This challenge was met with flying colors, and subsequently, the database has continued to scale and endure surges in traffic without any availability issues whatsoever.

Ultimately, our goal in seeking a hosted MongoDB provider was to establish total confidence in the reliability of our database so that our internal team could focus on developing an optimal product without distraction or concern for limitations. Our database now serves thousands of companies of various sizes, helping them provide a level of customer service that would be impossible if not for the ease with which the database handles the huge amount of data involved. As our solution and our database continue to grow, we look back at the moment we turned to DBaaS as the point that really made success possible, and we look forward to reliably achieving ever-greater scale because of it.

Vik Chaudhary is CEO of Dossier, app for organizing customer conversations.


Post a Comment

Your email is never published nor shared. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <s> <strike> <strong>