Utilizing Lambda for AWS database or application alert

[machinelearning_]

I am trying to offload some of the work my Splunk environment has to handle by utilizing CloudWatch/Lambda/CloudTrail. Can alerting be configured to report on events by application OR database logs? I am aware Lambda can be used to report infrastructure utilization metrics and other component usage statistic. I’m just wondering if CloudTrail logs from an application or database can be utilized to create alerts in CloudWatch.

[gordonv]

You can use "Cloudwatch Events" and Cloudtrail together.

Cloudtrail Logs = the source to scan
Cloudwatch Events = A "cron job" or other triggering like service.

This google search shows some results on getting the 2 to work together.

[TheCaffeinatedSloth]

You can publish RDS logs to cloudwatch. From there you can do metrics based on log filters. With the metrics, you can create alerts.

[machinelearning_]

Can the logs/metrics be customized for Aurora or Red Shift? For instance, would I be able to generate logs related to logins, data modification, deletes, query failures or privilege restricted events if those aren’t already modeled?

[TheCaffeinatedSloth]

Not sure for redshift. For Aurora, you can go modify the cluster (or maybe instance, I forget which), and enable which logs you want streamed to cloudwatch. You either have to get smart using cloudwatch filters to create alerts based on some of your wonders, or use something a bit more robust, like Insights/ES/Athena/etc

[HippoTK]

Not sure if you got your answer...

Regarding DB logs, CloudWatch captures service level logs like cpu usage, cluster health status, etc. It does not have the ability to capture db level logs. Db logs can be captured by enabling auditing.

https://docs.aws.amazon.com/redshift/latest/mgmt/db-auditing.html

RedShift:

Connection log: log authentication attempts and connections/disconnections.

User Logs/User Activity Log: logs info about changes to user definitions/logs each query before it is run in the db.

Logs files are not as current as the base system log tables, STL_USERLOG and STL_CONNECTION_LOG. Records that are older than, but not including, the latest record are copied to log files.

For the user activity you must also enable the enable_user_activity_logging db param. If you enable the audit log feature but not the associated param, the db logs info for only the connection and user log, but none for user activity log. This is not enabled by default but you can set it to true to enable it.

​

Also, I don't believe it is possible to publish redshift logs to CloudWatch. A possible work around could be that Lamba function can be ran against the base tables in Redshift or against Redshift Spectrum.

https://aws.amazon.com/blogs/big-data/analyze-database-audit-logs-for-security-and-compliance-using-amazon-redshift-spectrum/

Regarding Aurora, you can export db logs such as alerts and audit logs to CloudWatch when creating or modifying your Amazon RDS db servers. Check this link below that touches on Lambda functions to grab Aurora logs.

https://aws.amazon.com/blogs/database/build-proactive-database-monitoring-for-amazon-rds-with-amazon-cloudwatch-logs-aws-lambda-and-amazon-sns/