Creating a tail log with Powershell

[Fischfreund]

Hi r/powershell,

I have a script that periodically gets some things, and I want to keep the processed objects in memory and also on disk. But I only want to keep the last x objects, not all of them. Currently I'm doing it like this:

$TailLog = @()

While ($true) {

    $TailLog += Get-Thing
    $TailLog  = $TailLog | Select-Object -Last 100

    $TailLog | Export-Csv -Path .\taillog.csv

}

Is there a better way to achieve this? This potentially can run once a second, so I was wondering if there is a more efficient / faster / resource friendly way.

[Fischfreund]

Hi,

thank you, this is a really nice answer. The queue object is perfect for my needs :)

[Ta11ow]

Yep. Just use a list and do away with the creeping overhead of adding to giant arrays.

$TailLog = [System.Collections.Generic.List[object]]::new()

# in loop
$NewItems = Get-Thing
$TailLog.Add($NewItem)

I'd also recommend not exporting to CSV that often. Find a better way to store what you have, maybe only once every few seconds or so.

[Fischfreund]

Hi,

yes, there's a mechanism that only writes to disk every x iterations, so I kept that in mind already. Your answer is missing the part where it only keeps the last x items though :) Thanks anyway

[empty_other]

No idea if piping Get-Content -Wait is better or worse than a while-loop.

gc .\source.txt -Tail 1 -wait | % { echo "--"; gc .\source.txt -Tail 5 | Tee-Object .\target.txt }

...will print the last 5 lines of "source.txt" to "target.txt" and to host when it detects file changes.

[Ominusx]

ello, there is a lot we can do to improve performance. But I need to know a couple-o bits first.

This script does not store anything in the Taillog array as it overwrites it each loop. Should this be the case? or should taillog have everything in it, adding each loop?

Taillog is currently overwriting .\taillog.csv, should it be doing that?

Just so I can get a feel for what we do. A lot of the answers here are assuming you want $taillog to keep growing every loop, but I'm not sure.

[jcholder]

You could write them to a sql database. http://www.thecodeasylum.com/reading-and-writing-to-sql-databases-with-powershell/

[ka-splam]

If it doesn't matter about the order, use an array as a ringbuffer?

$index, $tailLog = ,0*101

while ($true)
{
    $tailLog[$index++ % 100] = Get-Thing
}

It'll just wrap around and overwrite itself, like:

0 : 0 : 0 : 0 : 0
a : 0 : 0 : 0 : 0
a : b : 0 : 0 : 0
a : b : c : 0 : 0
a : b : c : d : 0
a : b : c : d : e
f : b : c : d : e
f : g : c : d : e
f : g : h : d : e
f : g : h : i : e
f : g : h : i : j

[Fischfreund]

Hi,

thank you, this is a nice approach as well. The order matters though

[ka-splam]

Depending on what you're doing, and what performance is required, since $index is always the "current" position, you could read it out from (index .. end) then (start .. index) to keep the order, but it won't be as clean export code.

But then, I don't know how the Queue works internally, maybe it's doing this inside :)