View your logfiles in visidata automagically!

[haydenflinner]

Based upon this paper (maybe free link?), this tool (save it to drain.py and try running it) should do a serviceable job with default config of turning your logfiles into processable tables. There are knobs to tweak to add 'domain knowledge' as the authors of the paper put it.

Example with a ZooKeeper log with 10 seconds of domain knowledge applied (I've never seen a zookeeper log before now)

# Minimum run gets serviceable but not great result:
# python drain.py ./Zookeeper_2k.log
python drain.py ./Zookeeper_2k.log --log-format '<date> <time> - <level> \[<something>\] - <content>' --st 0.7 \
   | vd -f csv
,LineId,date,time,level,something,content,EventId,EventTemplate,ParameterList
0,1,2015-07-29,"17:41:44,747",INFO,QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:2181:FastLeaderElection@774,Notification time out: 3200,0ce98313,<*> <*> <*> <*>,"['Notification', 'time out: 3200']"
1,2,2015-07-29,"19:04:12,394",INFO,/10.10.34.11:3888:QuorumCnxManager$Listener@493,Received connection request /10.10.34.11:45307,0ce98313,<*> <*> <*> <*>,"['Received', 'connection request /10.10.34.11:45307']"

[frosencrantz]

Hi @haydenflinner thank you for sharing this. Recently I have been examining log files, and have wondered how to extract templates with parameters. This looks like it could help.

I tried out your demo script, and it was able to parse a local Linux syslog file with the parameters in your script. It generated templates that looked like <*> <*> <*> <*>. There were a small number of templates generated, which is helpful. However, I was expecting something like the example from the README on https://github.com/logpai/logparser, where the templates look like Received block <*> of size <*> from /<*>.

Do you know if there is a way to generate a template that includes the constant part of the message? Even if there are more templates generated? It looked like they had generated some templates like that in https://github.com/logpai/logparser/blob/master/logs/Linux/Linux_2k.log_templates.csv or is that intended as input?

[haydenflinner]

I bet the problem is my default regex for "values".
Or maybe try with similarity threshold 70+. Maybe changing value regex to only accept numbers, like r"(\d+)". See the examples configs at bottom of file for various configs of different file types. My default config with this tool is really a misuse of the algorithm design to be sure it sees some patterns, and then I do interactive filtering in visidata as "post processing".

Edit: I also set the max tree depth to something ridiculous like 100, all of the defaults are like 4-8, and that probably changes things too. On my logs that made it too slow.

[frosencrantz]

Yes, it looks like you are right about the default regexes. Thanks!

I was using the values from: https://gist.github.com/haydenflinner/7062619e4638df061490e8b5275e115a#file-drain-py-L447-L453

However it was also including the DEFAULT_REGEX from https://gist.github.com/haydenflinner/7062619e4638df061490e8b5275e115a#file-drain-py-L522 If I take out that second regex, it starts to include words in the default templates. I think the ones without words were actually more useful, albeit too generic. It will require some playing.

It looks like EventId is an alternate name for the templates. My log files include filename and line number on each log line. Can drain.py use that to help it to identify common templates? That would be very useful to me, and maybe generate better templates.

[haydenflinner]

Hmm, that's a good point about the algo that I had missed the full implications of. It assumes that if your token has digits that it is a part of a template, replaces it with '<*>'. It would be nice if it could recognize that line numbers are an exception. I think no harm in hacking that in yourself, should be pretty easy to find where it does the substitution, think it's "has digits". Maybe just if has digits but it's prefixed by a : then let it go thru as a normal token.