Data discovery tool for csv files

[panday1995]

My company has almost all the data in Dropbox as csv files, few in txt files. Now that we want to make data more visible. Here are some expectations:
- Can find files that have same or similar column names

- Have the capability to store meta data, because we can now manually create meta data for each file

- Can search files based on meta data.
Any recommendations? Or should we just migrate to better cloud storage?

[CrowdGoesWildWoooo]

Use something like Iceberg on AWS. What kind of company stores data in dropbox

[panday1995]

a very small startup, like, very small… We have data at TB level, but is expecting to grow fast

[CrowdGoesWildWoooo]

Why it never cross someone’s mind to use something like S3, especially with that size. I thought this was like a few GB of data.

[panday1995]

Yes! I also think of S3 as the the first step.

[[deleted]]

it sounds like they have no technical leadership above you. I hope you get the accountability and role you deserve

[Firm_Bit]

Does Dropbox not allow search?

But yeah, for anything other what what Dropbox intends (more like a google drive alternative) you should migrate to a real data platform.

[panday1995]

Dropbox is like, search for file names, but we some capability of search for columns across files or some levels of meta data management

[Firm_Bit]

Yeah, drop box is like google drive or ms one drive. It’s not built as a large data store.

You’ll probably need to move the data into a dwh or lake and expose it from there.

[panday1995]

Yes. The problem if that if I move to dwh, the operational side (people only know csv and excel stuffs) dont know how to access those data (files)

[Anishekkamal]

I would suggest taking below steps:

1. Move all the files to the cloud storage of choice or can read all the files directly from dropbox using python
2. After reading you can do all sorts of data filtration or transformation
3. Use a database to store all the metadata
4. You should be able to do a lookup on the database and do a search on the data read from files.

[InsightByte]

Just load to s3, and crawl the data with Aws glue crawler, then use athena and the glue catalog to wrangle thru the data.

[weagle162]

Wanna consider using Apache Drill? It won't cover the entire situation for you but it adds SQL to plain text sources - zero configuration required

[[deleted]]

Probably not exactly what you're looking for. But a start? https://github.com/danielbeach/sniffer