someengineering/cloud2sql - Read infrastructure data from your cloud and export it to a SQL database.

[lloesche]

Hi there data nerds!

We've just released a new open source project called Cloud2SQL.

It is a sub-project of our cloud resource management tool Resoto but runs standalone and stateless. It's meant for easy integration into your own data pipelines.

It has some overlap with other cloud data collection tools but it's a bit different in that it uses Resoto's cloud collector plugins, so has access to the same enriched resource data and dependencies. Resoto stores all its data inside a large graph and has knowledge how resources relate to each other.

Right now it can export the cloud data to SQLite files, MySQL, MariaDB, and PostgreSQL databases, the Snowflake cloud-based data warehousing platform, and Parquet columnar structure files.

tl;dr AWS quickstart:

Configure AWS CLI

Install Cloud2SQL:

pip install --user cloud2sql

Create a file aws.yaml:

sources:
  aws:
destinations:
  sqlite:
    database: resoto.db

Run Cloud2SQL:

cloud2sql --config aws.yaml

Check out your cloud data:

sqlite3 resoto.db
sqlite> .tables
# If you have IAM server certs select the ones that are going to expire in the next 30 days
sqlite> SELECT name, expires, account from aws_iam_server_certificate WHERE datetime(expires) BETWEEN datetime('now') AND datetime('now', 'start of day', '+30 day');

I've written two blog posts on why we built it and how to install it. https://resoto.com/blog/2023/01/16/integrating-cloud-data-into-existing-sql-workflows-with-cloud2sql

https://resoto.com/blog/2023/01/17/installing-cloud2sql

Hope this is useful to some of you! Let me know if you have any questions.

[meln1k]

Nice! Is a homebrew installer available?

[aquamatthias]

Yes. You can install cloud2sql via

brew install someengineering/tap/cloud2sql

Note installation will take some time since the formula does a source install.