At the moment, it often feels like big data is one great networking party where the cool kids show off all of the great tech they are working on. For many young IT professionals, it’s almost like the game of buzzword bingo that was played back in college lecture theatres. “How many cool sounding technologies can I get my company to invest in and add to my résumé?” Hadoop? A must have, of course. But too common. Yarn? Mahout? CEPH? Spark? Check, check, check, check …
I’m not bashing the technologies behind big data, neither established nor brand new. I think they are amazing. These technologies have been built to tackle the unique challenges that big data poses and can help organizations get more out of their data stores today than ever before.
What I have been hearing from customers and prospects is that they need help, not only to understand what technologies are available today (they change and are added to so frequently that it’s a struggle to keep up), but also how they should be integrated and deployed. (How can the “new” stuff play with the “old”?) These companies struggle to move projects from their test and development environment to day-to-day operations. That’s due in part to a lack of professional direction, but also to a lack of experienced data management and technology practitioners.
The Party’s Over
If too much of the big data community is like a bunch of college kids on their summer holidays while the growing needs of businesses go unmet, the time has come, as my old headmaster would say, to knuckle down and get some work done. For everyone’s benefit, now is the right moment for these big data practitioners to have some professional adult supervision so that they can build on their experiences and experiments of the summer.
Let’s get back to work and add new lessons to the syllabus, like understanding the value of money, finishing what you start, building proven solutions, security matters, and how to work together. It’s time to ensure that the experienced data management professor is guiding and working alongside the bright young kids so that companies’ investments in big data see a return, ensuring that the future of these companies is secure.
Let’s not stop there. In fact, let’s go even bigger: Let’s figure out how companies can use big data to become the next “Uber” – the innovator, the disrupting force – of their industry. To do that, companies need to be able to look at all of their data, all of the time, so that they can understand the relationships and dependencies between complex processes, products, people, and price, in minutiae. And they need to be able to do all that in a timely manner so that they can make their move before the competition does.
This means that companies have to think about the bigger picture. They need to integrate data and technologies on a massive scale, at speed and with agility. That often means choosing proven, simplified, scalable, and secure technology stacks that have been integrated and optimized for the business processes and data of each specific enterprise.
That also means data professionals to make these efforts successful and deliver real results to the bottom line.
Just Make it Work
Ultimately, big data tech talk is of no interest to business users and managers. They want to be able to take volumes of detailed sensor data that is constantly refreshed and be able to distinguish what is significant and what is just noise. They want to understand how they can be more productive and competitive by building a holistic picture of their business, not only with big data but also with more traditional sources of information, like product specifications, maintenance records, and cost and profit statements.
They want to be able to predict failures, eliminate downtime, lower the cost of maintenance, and deliver better customer service, for which they can charge a premium. They don’t care what the technology is. They just want it to work.
This is why smart businesses are now demanding that the best of the new technologies be deployed alongside the best of what they already have certified and proven to work (all the time) at an enterprise level.
Despite all the “knuckling down” that is needed, big data is still a hugely exciting proposition, both for companies and for technology professionals. I am certainly not saying that there shouldn’t be any more partying. It’s just that it will need to be done during the weekends!
Niall O’Doherty is Business Development Director for Manufacturing and Energy at Teradata International. He began his career in the wine-making business and moved through supply chain management and management consulting before settling in Teradata at the beginning of 2002. In his current role, Niall is responsible for growing Teradata’s presence, solutions, and strategies in the manufacturing, oil and gas, government, and utilities industries. Bringing insight, innovation, new ideas, and best practices to customers are what drives Niall and his team.
Subscribe to Data Informed for the latest information and news on big data and analytics for the enterprise, plus get instant access to more than 20 eBooks.