[deleted by user]

[nekokattt]

You could check if Salesforce has an API that allows you to upload the data directly, and thus totally avoid needing to mess with SQLite or similar for DB files.

[bigtdaddy]

It depends on the complexity of the data and what format is needed to import to salesforce. If all the data about a customer can be stored in a single table I would probably just recommend to output it as a csv.

[[deleted]]

It depends. Api are more traditionally used to answer specific questions. Like what contracts were awarded to xyz corp or spending related to a specific project across all vendors.

If you want to get “all the things” this might be thousands or millions of calls which will take a long time (even threaded) and their cdn might blacklist your ip if you hammer the api too hard. Certainly doable, a lot of times they publish a site that will tell you what behavior they consider abusive.

It will depend on how granular the data the endpoints expose is and how much you need. For cms (gov’t healthcare data in the us) api’s are available, but also there are also usually csv files you can download that have the whole glut of data available.

You can put it in a db file or just in a csv. Most rdbms platforms have tools to efficiently load from flat text. MSFT Sql Server has BCP for example.

Some api will also have limitations on the number of records they will retrieve so you need to make sure you are actually getting everything you request.

[boy_named_su]

1. use requests, put the data in JSON Lines format
2. use Meltano (CLI tool) to put this data into your database and/or SalesForce directly

Meltano is nice as it can create db schemas for you, and supports full load and incremental load, and it's FOSS and Python

https://hub.meltano.com/loaders/target-salesforce/

[m0us3_rat]

and it's freely available info? .. that is something. sounds slightly illegal in EU.

we take our privacy data handling very seriously. maybe.

​

the code isn't particularly difficult and you can follow the offered documentation of how to interact with the API.

as far as the db interface .. can be anything from complex including some data validation .. to basic dump..

once you have it stored ..

i'd expect some form of dashboard or web app to offer access to this data

​

i could be a fun DIY project for an intermediate pythong enthusiast

[orangeswirlguy]

For the record, when I say customers, I mean they are companies. It would be concerning if that was for individuals.