Behold, the Humble Log File

Behold, the Humble Log File

Log files. Log files? Gosh. Mention this topic in almost any context and you can expect either one big collective sigh, or blank stares and a group “meh.” One of the reasons for this type of reaction is that log files remain largely unseen and they are easy to ignore. And yet, today’s technology landscape places a lot of detailed requirements on log file security and maintenance procedures. All of this throws shade on log files and log maintenance, relegating them to that special category of “necessary evil.”

But wait… It turns out log files are really special after all. As long as they don’t grow too large and clog up a hard drive, log files are basically well-behaved, humble creatures. They sit by themselves all day long, dutifully storing exactly the data that we have asked them to store. They willingly recycle themselves when their retention window passes. If nobody bothers to review them, they are never offended – they just keep on doing what they have been asked to do. If log files were people, they would be very stable and grounded individuals who are destined for greatness! So let’s dig a little deeper into the general concept of log files and logging, shall we?

Beyond logging system events and other things on our data center servers, the general concept of logging can be used in real-life with amazing results. Logging introduces a rational and pragmatic approach to problem solving that can be applied to just about anything. Doctors review food and fitness logs to find possible causes of a specific health problem. Traffic engineers review traffic logs to solve traffic congestion problems. Insurance companies analyze weather and seismic activity logs to properly spread risk assessment among all of their customers. We could go on and on, but what these applications have in common is this – general events and measurements are logged, and then someone analyzes those logs to look for something specific. In most cases the reason for analyzing the log was not specifically envisioned when the log parameters were determined. And that’s the point – the concept of logging is not just rational and pragmatic, it is an open-ended tool for solving lots of different problems in life. A good life hacker can appreciate the power of logging!

But let’s get back to technology, shall we? Consider the PCI Data Security Standard and its detailed requirements for logging. These requirements are quite burdensome and it would be easy to cry foul and only begrudgingly comply with them. The latest version (PCI DSS 3.0) brings further logging requirements that can get really tough. One requirement that was particularly interesting to us is requirement 10.2.6, which involves the initialization, pausing, and stopping of the logging process. While we were considering this along with other basic requirements, such as unified storage across all sources of log data, it became clear to us that a do-it-yourself approach was no longer wise. We reviewed the marketplace and found one vendor that outshined them all – Sumo Logic. Our implementation of the Sumo Logic system was easy – we found the log analytics platform to be powerful, and when all is said and done, we find that Sumo Logic enhances our DevOps practices in many ways.

I asked Sumo Logic’s CTO and co-founder, Christian Beedgen, about his insights on logging. It turns out my analysis of log files being extremely powerful and useful was spot-on. In fact, as CTO of Sumo Logic, Christian lives in this world on a daily basis. He shared with me the following very interesting remarks:

“Logs have morphed from those little poems that developers used to write to themselves, into the core of two exciting and fast growing trends - Big Data, first of all, and of course, today, machine data. As we are continuously building larger and larger scale distributed systems, the data generated by those systems as they go about their (your!) business has become mandatory for monitoring not just how well those systems are working, but has also emerged as the cornerstone of a new mode of business analytics: Agile BI. A truly interesting journey, I'd say.” – Christian Beedgen, CTO, Sumo Logic

And that’s my point exactly – logging can be a rational and pragmatic approach not just to server maintenance and security, but to business strategy itself. Here at Certify, we have found that log files can be an interesting part of our business strategy. You see, we use an integrated DevOps approach for delivering our SaaS cloud solution. While performing the PCI requirement of daily server log review, we continue this integrated DevOps approach by involving not only security experts, but also engineering (coding) experts. We find that just about every day there are a handful of product bugs that are brought to our attention by only this one source – the daily log files. Solving these problems in a proactive manner is a great way to continually increase product quality. Once in a while we even inform our Support Experts about one specific user whose problem has now been solved, even though they never contacted our Support department. They usually tell us that user was blown away when told that their problem was seen by our system administrators and has now been resolved. Log files may not be very sexy, but they sure can be a powerful tool. Behold, the humble log file!

Alan Neveu

Certify, CTO