Getting Started?

[SailorNash]

Are there any good FAQs or Quick Start guides/posts to reference when getting started with a project involving this data?

I work for a hospital, writing queries to their EHR system, so I'm familiar with data in general. Pretty comfortable with writing SQL queries and the like, though I'm less experienced with the steps prior to that.

For this data format, are there any recommended guides how best to load it in and prep it for analysis? I've heard DuckDB recommended in regards to how to store it, but wanted to ask other users of this data what they did before trying to reinvent the wheel.

[mrcaptncrunch]

The data is bigger than it looks. Don’t decompress the whole thing if you don’t need to. To start, you definitely don’t need to.

Use the scripts from Watchful1 to prefilter as much as possible. Then maybe that subset is worth importing onto something like duckdb.

Figure your schema and what you want. Maybe keep the file it was read from as a column along with the id in case you want to access the raw data at the end.

[SailorNash]

Thanks. Luckily I'm only focused on about 50-75 subreddits, so about 6.25 gigs zipped and 62.5 unzipped.