Live Q & A Transcript: Capture Real-Time Operational Intelligence from Big Data

by   |   September 30, 2015 11:58 am   |   0 Comments

The following is a transcript of the Live Q and A discussion that Data Informed hosted with Albert Mavashev, Chief Technology Officer at jKool, on September 29, 2015.

Scott Etkin: Hello and welcome to the Live Q and A with Albert Mavashev, CTO at jKool! Feel free to post questions at any time.

Albert Mavashev: Hi, my name is Albert Mavashev. I am here to answer your questions about jKool. jKool is a platform for analyzing machine data in real-time, time series data like logs, metrics, transactions. Thank you for having me.

Albert Mavashev, CTO, jKool

Albert Mavashev, CTO, jKool

Comment From Charley: Can you explain what you mean by real-time operational intelligence and what that has to do with big data?

Albert Mavashev: When we say real-time operational intelligence, we mean being able to derive actionable information from seemingly disorganized, structured and unstructured data sources and gain insight that can be valuable to the business. These data source are logs, metrics, transactions. Why big data? Because there is a lot of this data in IT. You need to be able to store everything and analyze everything, and that requires big data solutions like jKool.

Scott Etkin: Just a reminder about the tablet computer drawing: Everyone who registered is entered in the drawing. Additionally, for each question you ask, you will receive an additional entry. We will draw the winner at the conclusion of the Q and A.

Comment From Dan: Can you give an example of analyzing transactions?

Albert Mavashev: An example would be analyzing performance, topology of a payment, claim, HTTP request, buy a book, a trade. When we analyze transactions, we discover the topology of transaction execution, performance, payload, exceptions for individual, as well as aggregate. This allows you to know what your end users are doing, how your business is behaving, and what you need to do to improve the user experience.

Comment From Brad: What kind of technical expertise does my staff need to take advantage of real-time analytics and visualization?

Albert Mavashev: jKool is a service (but can also be deployed on-prem). In general, all you need is an IT resource that can connect (configure) syslog, log4j, or other data sources that you may have. If you feed the data to jKool Cloud SaaS platform, that is all you need. No hardware, software, or databases to deploy. It can be done in less than 30 minutes.

Drew Harris: In your whitepaper, you talk about how real-time operational data can save in personnel costs. Are there other cost savings that real-time operational analytics can provide that might not be immediately apparent?

Related Stories

Capturing the Business Value of Big Data in Real Time.
Read the story »

Real Time, Location, and the Internet of Things.
Read the story »

Keep Your IT Systems Up and Running in the IoT Era.
Read the story »

How to Rescue IT from Big Data.
Read the story »

Albert Mavashev: There are several ways to realize benefits: reduced personnel costs or reduced costs of running your application. You can cut costs associated with reduced quality of your end-user experience. jKool allows you to find problems before they impact your business, such as security violations and performance issues. These can translate to huge cost savings, depending on what your application does and how mission-critical it is. In some organizations, it is in millions (tracking payments, trades, orders).

Comment From Charley: You mentioned metrics as something in addition to logs. Can you explain what these metrics are and what you do with them? How about a real-world use case?

Albert Mavashev: Metrics are typically name=value pairs such as CPU usage, memory usage. Logs are typically message such as exceptions, errors, and traces. You need both to properly gauge the performance of your application. You need to analyze logs, but you also need to measure and monitor metrics. Example: measuring response time of a transaction. jKool deals with logs, metrics, and transactions so that you can get one view of whole application.

Comment From Lynn: How do you maintain a consistent format for filters on cloud operations when users pull metrics? Such as one user pulls ABC, another ACD categories. Reporting these up through dept., site, plant.

Albert Mavashev: In jKool, we go with the model of store everything and analyze everything, since you never know what you might need to troubleshoot a security breach or performance issue. In our case, filtering is performed at the data source: syslog, log4j, etc. That is where the filters are maintained.

Comment From Josh: My team’s developers waste a lot of time diagnosing application problems. What can be done to limit this problem?

Albert Mavashev: In our experience, there can be several problems:

1) Not enough or inadequate instrumentation

2) Too much of it where you can’t make sense of it

3) Tough issues with concurrency, resources, deadlocks, architecture

jKool can help with 1 and 2, so that you developers are focused on developing features and less on looking for a needle in a haystack. We can take all the info from log, application metrics, transactions, and turn it into an actionable visualization. Most developers I know hunt through logs, and we can reduce that significantly. Others ask, “Why timeout?” “Where is my message?” “Why so slow?” We can help that with transaction tracing.

Comment From Duke: Can real-time analysis of streaming machine data improve security compliance?

Albert Mavashev: We consume and analyze security events and metrics the same as we do any other events. We can store them to help with forensics, aggregate, show you what is happening now (right now), as well as what happened before. You can observe DoS attacks (as an example) in real time.

Comment From Linda: Can real-time analysis do a better job of diagnosing performance issues and determining the causes?

Albert Mavashev: Real-time analysis of transactions is very valuable for this because you can track actual execution morphing, merging of requests, and replies in real time. We measure each leg of transaction execution, total transaction, exceptions. Imagine running a QA test of your application and something is slow. Where is it? Hard to answer without this visibility. Plus, there are always hidden error message, log entries, and metrics that you may not know are impacting your app, like memory, swap space, or IO rates. jKool can help here since we can consolidate all these sources: log, metrics, and transaction to light up (so to speak) your application.

Comment From Charley: Why would I use jKool if I already have Splunk?

Albert Mavashev: There several reasons: easier to use, more cost effective. We also have built-in transaction analytics (automated discovery of transactions). Those are just a few.

Comment From Alex: What are some ways to distinguish anomalies in your data that indicate a serious performance issue from an insignificant outlier?

Albert Mavashev: We automatically aggregate your metrics so we can answer normal vs. abnormal. One way is to look at the number of standard deviations from the mean. We do this automatically. Another is flow discovery. We can show you the flow of your transaction: A slowdown in one part of application can spill over to another. jKool can help here since we do automatic flow discovery.

Comment From Charley: What sort of analytics are available?

Albert Mavashev: Aggregations sum, avg, max, min, best, worst, top, bottom, Bollinger Bands, just to name a few. More are always being added. Transaction analytics, topology discovery, flow.

Comment From Raghu: What kind of insight into app performance can you provide?

Albert Mavashev: Insight into resources such as: GC, CPU, response time, memory, thread wait/block, count, application business metrics, transaction metrics, logs, exceptions, errors. Feature usage: best vs. worst.

Comment From Ben: Is real-time analytics something that is realistically within the reach of a smaller business? With the smaller data sets compared with an enterprise organization, is real-time worthwhile for a smaller business, or do the smaller data sets make batch processing and analysis an equally effective approach?

Albert Mavashev: jKool makes real-time analytics attainable for any business size. Smaller data sets do not mean the business is equipped to harvest its value. While you can do things in batch, today’s businesses want information now. Now is what we can bring, in addition to past. In many situations, the value of now is more critical: for example, preventable losses. What good is it to know that 1,000 users abandoned a shopping cart in one week? Knowing now gives you the capability to act now.

Comment From Charley: What sorts of data can jKool analyze and how do I get it in?

Albert Mavashev: We deal with time-series data streams: log, metrics, transactions, but not limited to that. We can consume sales force, patient info, payment data, or others. You can download our open-source collectors from www.jkoolcloud.com. You can also develop your own using our open-source API TNT4J available @ GitHub.

Scott Etkin: Thank you everyone for attending the Live Q and A with Albert Mavashev, CTO at jKool. Just a reminder that a transcript of the event will be emailed to all registrants. If you have additional questions for Albert, you can email him at albertm@jkoolcloud.com.

We will now draw the winner of the Samsung Galaxy Tab 4 tablet computer.

Albert Mavashev: Thank you for having me on.

Scott Etkin: Thanks again to everyone for great questions and a lively discussion. The winner of the Samsung Galaxy Tab 4 tablet computer is Lynn! Congratulations Lynn!

Thank you Albert, and thanks again to everyone who registered and attended today’s Live Q and A!


Subscribe to Data Informed
for the latest information and news on big data and analytics for the enterprise.

 


The Critical Moment: Getting Operational Intelligence from Logs, Metrics, and Transactions



Tags: , , , , , , ,

Post a Comment

Your email is never published nor shared. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <s> <strike> <strong>