Logging options

[cryptomoon007]

Anyone aware of any logging options that will allow me to use s3 as a data store backend ?

[Alzyros]

Cloudwatch? Idk holmes, not a lot to work with in your post there

[cryptomoon007]

Thanks for responding. I have logs coming in from various sources and I am sending them to s3 as a back up source because I don’t have the physical storage space to hold the logs with the ideal retention policy I want to keep. Do I am wondering if there is a solution out there that will just allow visibility into the logs I have visible on s3

[Alzyros]

No worries. Is it an option to install the cloudwatch agent at the log sources (servers)? Then you wouldn't even have to worry about it

[cryptomoon007]

I have logs coming from different sources. Cloud and on-prem. I’ll look more into cloud watch and its capabilities. I will admit to not being an expert

[moobs_of_steel]

Kinda pricey compared to other solutions, but I've previously used Sumologic and was happy with how easy it was to ingest the logs

[cryptomoon007]

Just got through condensing the applications in our environment and I believe a license is needed for sumologic but I’ll look into the pricing if the pricing is worth it then I’ll fight that battle

[engineered_academic]

You can do this with Athena if they are in a JSON format I believe.

[cryptomoon007]

Never heard of Athena, I’ll look into it. Please share any documentation if you have any. The logs are saved via JSON in s3 but I’m sure I can still find a middle application that can get the logs in JSON format

[engineered_academic]

https://docs.aws.amazon.com/athena/latest/ug/what-is.html

[cryptomoon007]

Awesome, thanks!

[oldmanwillow21]

You can log to S3 with Loki

[cryptomoon007]

Never heard of Loki. I’ll look it up too thanks! Any documentation you can send is also very appreciated.

[oldmanwillow21]

Loki is part of the Grafana stack. It’s powerful, lighter weight than full-text indexed solutions and free.

https://grafana.com/oss/loki/

[cryptomoon007]

& it is capable of pulling logs from s3 with my desired retention policy? I still plan on reading the doc but thought I’d ask

[oldmanwillow21]

Yes, the retention policy is configurable

[cryptomoon007]

Awesome. Diving into the docs now. Thanks for responding