If you’ve spent any time digging for insights in logs, you know the shape of the problem. A single log line might contain an IP address, a status code, a response time, and a user ID, but it’s all buried in one long, unstructured string. You know the information is there. Getting it into a field you can filter, group, or chart on is a different matter.
This is one of the most common bottlenecks in log analysis, and it’s worth understanding the actual mechanics of why it happens and what your options are for fixing it, whether you’re troubleshooting a single incident or building a permanent pipeline.
Why logs end up unstructured in the first place
Most applications write logs in whatever way a developer finds convenient in the moment, not the way an analyst will want to query them six months later. A single log line often mixes free text, timestamps, key-value pairs, and nested identifiers with no consistent delimiter. Multiply that across dozens of services, each with its own logging library and format, and you end up with a firehose of raw text that technically has everything you need and practically hides it from you.
Structured fields solve this by giving each piece of information a name. Instead of searching for a substring inside a wall of text, you search on status_code=500 or response_time>2000. The data doesn’t change. What changes is your ability to act on it.
Approach one: Parse fields at query time
Manually writing parse expressions
The traditional and most direct way to extract a field is to write a parse expression as part of your search. Most log platforms support some version of an anchor-based parse operator: you identify a piece of text in a sample log line, tell the system what surrounds it, and give the extracted value a field name.
This works well for one-off investigations. You’re looking at a specific error, you need one or two fields out of it right now, and you don’t need that parsing logic to persist beyond this search. The tradeoff is that it takes real syntax knowledge. Building an accurate parse expression by hand usually means identifying each field boundary, escaping the right characters, and testing against a few sample lines before you trust the result. For a team member who parses logs daily, this becomes second nature. For everyone else, it’s a skill gap that turns a two-minute investigation into a twenty-minute one.
Let AI do it
The above approach creates a bottleneck: you have to know how to write a parse expression. That’s a real barrier, especially if you’re newer to the team or only parse logs occasionally rather than daily.
AI-assisted parsing tools remove that step without removing your control over the outcome. In Sumo Logic’s Log Search, this looks like AI Parse Assist: you select a representative log line, choose “Parse with AI,” and let AI identify potential fields and generate the corresponding parse expression. Review the suggested fields and expression, make changes if needed, and apply them to your search. Nothing runs automatically. It’s an accelerator for the same skill, not a black box that hides what’s happening to your query.
Approach two: Extract fields at ingest time
If you find yourself writing (or using AI to write) the same parse expression over and over, query-time parsing is the wrong tool. Field extraction rules (FERs) solve this by parsing fields once, at ingest, rather than every time someone runs a search.
The advantage compounds. Extracting fields at ingest makes them available for ad hoc searches, scheduled searches, real-time alerts, and dashboards, so no one has to re-derive the parse logic. This also standardizes field names across your organization, so “IP address” doesn’t show up as ip, ip_addr, and client_ip depending on who wrote the search. FERs only apply going forward from when you create them, so if a field matters to your team, it’s worth setting up the rule early rather than waiting until you need historical data parsed retroactively.
The tradeoff here is upfront investment. You have to identify which fields are worth extracting platform-wide and configure the FER correctly, which again usually means knowing parse syntax or leaning on a teammate who does.
Choosing the right approach
- Investigating a single incident? Query-time parsing, whether by hand or AI-assisted, gets you the field you need without committing to anything permanent.
- Extracting the same field across every search? Set up a field extraction rule (FER) so the parsing happens once at ingest for everyone.
- New to parse syntax, or want to skip the trial and error? AI-assisted suggestions give you a starting point to accept, adjust, or learn from.
None of these are mutually exclusive. The best approach to parsing uses all three: extraction rules for the fields you always need, manual or AI-assisted parsing for the fields you need occasionally, and enough consistency across your team that nobody’s stuck relearning parse syntax mid-investigation. Build that habit, and every future search, dashboard, and alert built on those fields gets faster and more reliable, instead of everyone re-solving the same parsing problem from scratch.
See how it works. Schedule a demo.
