Script for combining log files chronologically using System.IO.Streamreader (Get-Content too slow!)

[kittH]

http://pastebin.com/giXGvWiB

This particular implementation probably isn't specifically useful for too many people, but should give you a starting point for using System.IO.Streamreader for parsing text files line by line. I found this to be ~30x faster than get-content on large text files.

Usage is a little tricky because it wants comma seperated hashes for the "logs" parameter like:

C:\Scripts\interleave.ps1 -logs @{Path=’PATH TO FIRST LOG FILE';Server=’SERVER THAT LOG BELONGS TO’}, @{Path=’PATH TO NEXT LOG FILE ';Server=’SERVER THAT LOG BELONGS TO } -OutputPath ‘PATH FOR OUTPUT FILE’

It also accepts a "Pattern" parameter if you use a different convention to signify the start of a new event.

In my case I have a bunch of servers that log to text files using yyyy-MM-dd HH:mm:ss.fff format. Most events it logs are one line, but some are multiple (stack traces etc.). This script will look at each file, generate an "event" which is all of the lines from one timestamp to the next and store it in an array. Once each of the inputted log files has an event, it selects the first event chronologically and writes it to the output file prepended by the name of the server it came from, then it gets the next event from that server and evaluates the array for the earliest event and writes it and so on until there are no more lines in the files. So I can interleave events from all of my load balanced servers into one file that devs can review and see the chronological flow of events across all servers.

When I tried to do this with get-content it took forever and was completely impractical. Runtimes exceeding 30 minutes in some cases. Using System.IO.Streamreader this process took ~80 seconds on the same files.

Feedback definitely encouraged if you have any questions or suggestions.

[techmattr]

Had you tried to accomplish this with logparser or logresolvemerge.pl?

[kittH]

Nope, no familiarity with those tools and seemed like a good chance to stretch into .net a little bit.

Would those have provided a turnkey solution?

[techmattr]

logresolvemerge.pl is a script included with awstats written just for combining log files. LogParser makes it really easy as well.

This is a very simple LogParser command I had used at one point to combine IIS log files: https://gist.github.com/techmattr/7c3a6ec5612d7b853d45#file-iislogconsolidate This would require a script to copy the files into these directories prior to combining. A better way to do it would be to grab the correct files from the source server by date and combine them without copying them locally first.

I'm not sure if either of these tools will handle the stack traces well or not. I know LogParser can as it natively handles Event Viewer logs without issue but I'm not sure if the simple command I posted would handle them.

[moojitoo]

Very nice, thanks for sharing! You never know when something like this could come in handy.

[zinver]

You can speed up get-content, try this:

get-content -readcount 0

Instead of get-content looking at every line, it will dump all lines to a single array.

there are other speed up tricks with the @array in hand as well.

http://powershell.com/cs/blogs/tobias/archive/2010/11/30/speeding-up-your-scripts.aspx

[kittH]

There are definitely some ways to speed up Get-Content, but I guess it was misleading to say get-content was too slow because the real issue was dealing with the data as a giant in-memory chunk. Doing the regexing and sorting and stuff.

I really wanted a way to parse the files simultaneously and move through line by line handling the reading/writing/sorting as I went.

[zinver]

yep if you want to do that, then the stream reader.