syslog-ng-users October 2010 archive
Main Archive Page > Month Archives  > syslog-ng-users archives
syslog-ng-users: Re: [syslog-ng] Thoughts on patterndb syntax

Re: [syslog-ng] Thoughts on patterndb syntax

From: Balazs Scheidler <bazsi_at_nospam>
Date: Thu Oct 21 2010 - 15:23:57 GMT
To: Syslog-ng users' and developers' mailing list <>

Hi Lars,

First of all thanks for your message. Any kind of feedback is very much
appreciated, basically these make me want to work on the code further :)

So I'd like to urge everyone to post their opinions, they really make my

Comments on your points are below, inserted into your message.

On Wed, 2010-10-20 at 21:57 -0400, Lars Kellogg-Stedman wrote:
> I've been playing with 3.2beta1 recently and getting my feet wet with
> the patterndb support, which I haven't had a chance to work with
> before. I have a few thoughts regarding the patterndb rule syntax,
> mostly targeted at making things a little bit easier to work with.
> - Rule IDs
> Is there any particular reason why unique IDs were selected as rule
> identifiers? They're not particularly meaningful to people, and
> they're hard to talk about. It's much easier to say, "we're suddently
> seeing lots of matches for ssh-accept-connection" than it is to say,
> "we're suddenly seeing lots of matches for
> 4dd5a329-da83-4876-a431-ddcb59c2858c". With the current version of
> syslog-ng it looks like it's possible to use arbitrary identifiers in
> place of UUIDs, and that's what I'm doing for my local rulesets.
> This even makes classification metadata more useful, because
> .classifier.rule_id=ssh-accept-connection is immediately meaningful,
> while a UUID is useless unless I go grepping around the database.

Well, I don't really made up my mind how rule_id's should be used. They
were proposed by Marci (who implemented the patterndb syntax in the
first place). syslog-ng doesn't really care, but they must be unique.

These are useful as we can attach a lot of information to the patterndb
rule. For example, if you use a "<description>" tag, then you can
retrieve this said description when you browse the log, based on the
unique ID.

This is what SSB (our syslog-ng appliance product) does for example.

> - Whitespace
> Log messages tend to be long, which makes them unwieldy in a number of
> situations. It would be nice if instead of this:
> <pattern>...some very long log message that makes my life difficult
> and my patterns hard to read ...</pattern>
> I could do this:
> <pattern collapse_whitespace="yes">...some very long log message
> that has been conveniently wrapped to
> that it's easier to edit, email, and otherwise
> work with.</pattern>
> Specifically, enabling "collapse_whitespace" would transform any
> sequence of whitespace to a single " ".
> As proposed here this would be a completely backwords-compatible
> change because the default behavior would remain the same.

Interesting idea and of course doable, but then if there's indeed
multiple spaces in the message, you get in trouble.

> - Reusable patterns
> We use Cisco firewall modules on our network. When these devices log
> a connection-related message, the source and destination address look
> something like this:
> ircs:3610:
> That's:
> interface:vlan:ip/port
> Which becomes:
> @ESTRING:fwsm.src_if::@@NUMBER:fwsm.src_vlan@:@IPv4:fwsm.src_ip@/@NUMBER:fwsm.src_port@
> When this occurs throughout the ruleset, and multiple times within a
> single message, it really lowers the readability of the rules. I wish
> there was a way to modularize this so that I could create custom
> types, something like this:
> <type name="FWSM_ADDRSPEC">
> @ESTRING:iface::@@NUMBER:vlan@:@IPv4:ip@/@NUMBER:fwsm.port@
> </type>
> And then do this:
> <pattern>Accepted src @FWSM_ADDRSPEC:fwsm.src:@ dst
> @FWSM_ADDRSPEC:fwsm.dst:@</pattern>
> And get this:
> fwsm.src.iface
> fwsm.src.ip
> fwsm.src.port

This is a good idea. Although I'm a bit fiddling with the idea to extend
the patterndb syntax a little bit. Nothing concrete yet, but reusable
components will necessarily be a part of the picture.

-- Bazsi ______________________________________________________________________________ Member info: Documentation: FAQ: