True fact: unstructured data not only represents the average enterprise’s largest data set, but it’s also growing at a mind-boggling rate, which presents significant problems. Unstructured data, almost by definition, is not readily available to be analyzed.
Log management addresses a significant subset of this expanding pile of unstructured data: diagnostic and run-time log information produced by applications, servers, and devices. Think of these logs like IT’s exhaust. Since these data sets are massive and unwieldy, organizations often opt to avoid them altogether; those who do use them are typically forced to implement and support a costly legacy log management solution.
Those who choose not to leverage them are missing out on a significant opportunity to drive their IT and security operations to excellence. After all, it is hard to know what is really going on in an organization without looking at all the evidence, which is precisely the purpose logs serve. No other enterprise data set can record the realities across all infrastructure components with the same level of precision and detail.
Over the past ten years, a number of vendors have emerged, each claiming to have finally solved the cumbersome, costly log management conundrum, however, customers continue to face increasingly high annual licensing fees and hidden costs to manage and configure these systems, along with mounting hardware and storage requirements. An honest TCO calculation should include not just the cost of the annual software license and a realistic assessment of the unavoidable associated maintenance and support charges, but also, they should account for the inevitable internal staffing resources required to install and configure these systems, since maintaining a log management system is a complex and personnel-intensive proposition.
As we speak with companies, we’re discovering that in most cases, IT is spending innumerable hours to keep their commercial systems running in the first place. Today’s log management solutions are complex deployments mixing software with hardware appliances. They require hands-on tuning of RDBMS systems that don’t scale past one server. And finally, they exhibit an extreme hunger for expensive enterprise-class storage. Even so, they often fail at scaling to the entire log dataset, requiring complex arrangements for splitting the data over multiple systems that leads to additional management cost.
In the end, the system that is meant to monitor the actual IT systems is becoming an un-monitorable behemoth itself. “Hooray for recursion,” we say, biting our tails. For the IT practitioner, however, this is not a pretty picture.