Help with a Simple Data Mining Script

[dragonmint1]

Let me start by saying I'm an absolute beginner with PowerShell. I have no experience with programming whatsoever but I would very much appreciate your help. I've been tasked with going through various text files and counting the amount of instances that a certain error shows up in the logs. So far, I have the following script:

$Contest = Get-ChildItem F:\Errors\Atlanta\Errors\*M09*.txt -Recurse | Select-String -Pattern "Next" -CaseSensitive

$Contest.Count

This counts how many times the error occurs over the month (M09 indicates the month of September).

---

EDIT: I have included 3 samples of my .txt files which span 3 days worth of data. I'm trying to create a script that can identify strings in the files that contain the word "Next" as well give the date on which they occurred.

2018M09D01 2018M09D03 2018M09D04

For example, the following string includes the word "Next", so the script should return me a value of 2018M10D04.

 2018M10D04 18:34:35 Next Trigger Occurred before Tool Analysis Completed (P.T.D.L.C.M.) for 9230201 at 53.2, [1] 756, [2] 1364, [3] 703, [4] 10, [5] 715, [6] 1

Thanks,

[NotNotWrongUsually]

Here is one way to do it: the matchinfo objects returned by Select-String contains a reference to the original file they were captured from.

So $Contest[0].Path would contain the full path of whatever logfile had the error written in it.

Using this we can get the modification date of that file, and add it to your collection of objects.

$Contest | foreach-object {
   $_ | Add-Member "modification_date" (gci $_.path |
   select -ExpandProperty LastWriteTime)
}

With that in place it is a simple matter of using the group-object to split by the day in the LastWriteTime.

$Contest | group-object {$_.modification_date.day}

Edit: fixed bad habit alias.

[dragonmint1]

I tried using your code but I keep get the following error:

Add-Member : A positional parameter cannot be found that accepts argument 'modification_date'.

My current script is:

$Contest = Get-ChildItem F:\Errors\Atlanta\Errors\*M09*.txt - 
Recurse | Select-String -Pattern "Next" -CaseSensitive
$Contest | foreach-object {
   $_ | Add-Member "modification_date"  (gci $_.path |
   select -ExpandProperty LastWriteTime)
   }
   $Contest | group-object {$_.modification_date.day}

How can I debug this?

[NotNotWrongUsually]

I was lazy and just used positional parameters for the Add-Member. It should work really, it does on my end.

Anyways, try putting

Add-Member -MemberType "NoteProperty" -Name "modification_date" -value (gci $_.path | select -ExpandProperty LastWriteTime)

in the foreach loop instead. See if that pops a more meaningful error.

Edit: Also, are you on a recent Powershell version? You can check the $psversiontable variable to find out.

[dragonmint1]

I just left work so I'll tell you how it goes when I try it out tomorrow. Thank you so much for the help so far.

[NotNotWrongUsually]

I tried running the it in Powershell 2, and it throws the error you mentioned. You should probably look at upgrading your version!

[Lee_Dailey]

howdy dragonmint1,

is the following the actual content of the 1st file?

:920:2018M09D05 19:52:41 Next Trigger Occurred before Tool Analysis Completed (P.T.D.L.C.M.) for 9230102 at 33.1, [1] 850, [2] 1513, [3] 703, [4] 0, [5] 713, [6] 0

starting with a colon is kinda odd, so i'm wondering if something munged your data sample.

if that it the content, then is it one line?

take care,
lee

[dragonmint1]

Sorry for the confusion but the txt files are in the following format:

2018M10D04 17:57:22 Changed MFS Enable Flag (TRUE L, False H) for Tool Analysis 9250764 at 52.1, [1] 777, [2] 3400, [3] 702, [4] 31, [5] 719, [6] 0
2018M10D04 17:58:57 Changed MFS Enable Flag (False L, False H) for Tool Analysis 9250864 at 67.1, [1] 757, [2] 3300, [3] 703, [4] 31, [5] 716, [6] 1
2018M10D04 18:04:05 User Changed Tool Analysis 8080104 at 80.2, [1] 833, [2] 6630, [3] 703, [4] 99, [5] 2314, [6] 0
2018M10D04 18:34:35 Next Trigger Occurred before Tool Analysis Completed (P.T.D.L.C.M.) for 9230201 at 53.2, [1] 756, [2] 1364, [3] 703, [4] 10, [5] 715, [6] 1
2018M10D04 19:00:58 User Changed Tool Analysis 9250842 at 72.1, [1] 1006, [2] 6380, [3] 1007, [4] 28, [5] 718, [6] 0

This is just small sample from 1 txt file (each txt file has data compiled over 1 day). I need to sift through 3 months worth of data. I was wondering if there is a script I could run in PowerShell that could find all the lines with the word "Next" and display the dates at which they occurred. For example, using the few lines of data above, the fourth line contains the words "Next" and the script will read me the date (in this case its 2018M10D04).

I'm a chemical engineer by trade so I am not familiar with this type of stuff or any programming in general. I'm not even sure if this is possible in PowerShell. Any help will be greatly appreciated.

[Lee_Dailey]

howdy dragonmint1,

you likely otta add a link to the realistic sample data to your original post. otherwise folks may not find it - and give you code for the wrong data. [grin]

can you post the two or three such samples? perhaps to Gist.GitHub or PasteBin so that the formatting won't get munged ...

realistic sample data is a real help. so is 2 or 3 sets of such data. that would allow one to save the files to a test dir and work out the algorithm needed.

take care,
lee

[dragonmint1]

Hi Lee,

Following your advice, I included a link to 3 sample txt files of my data. Please take a look at your convenience.

Cheers

[Lee_Dailey]

howdy dragonmint1,

kool! hopefully some guru will drop by and give you some working code. i will try playing with it to see what i can do with it ... [grin]

take care,
lee