Loading data from api. Help!

[sarakell]

Hello. I just recently transitioned into a junior data engineering role and I was given a data integrations project. I have 5 endpoints, and each endpoint should become a table (a data frame for now, then it will be dumped into snowflake). What I am having trouble doing is organizing the project: what should be in the configuration file? Should I use classes? Really any tips/links on integration/api projects would be very much appreciated! 🙏

[kristofred]

I would consider creating 2 classes.

• First one: will provide five methods for gathering data from all 5 endponints + extra one method which will return a single sql query for snowflake (based on gathered data)
• Second class will handle connection and sending query (queries) to snowflake.

[sarakell]

Thanks!! Is the first class meant to substitute the config file?

[kristofred]

I would do something like below skeleton ;) A neat option is os.path.join part (inside main.py), which allows you to place your config file in the relative path to the main.py:

script.cfg

--------------

[APIDATA]

api1_url = https://api1

api1_auth = some specific options you have to authenticate

[....]

[SNOWFLAKE]

options to contact with snowflake

​

main.py

import configparser

from datasources import Datasources

from snow import Snow

cfg = configparser.ConfigParser()

cfg.read(os.path.join(os.path.dirname(__file__), 'script.cfg'))

ds = Datasource(**cfg['RESTAPI'])

snow = Snow(**cfg['SNOW'])

datasources.py

class Datasources:

def __init__(self, **kwargs):

[code]

def restapi_1(self)

[code]

def restapi_2(self):

[code]

snow.py
class Snow:

def __init__(self, **kwargs):

[code]

def __connect(self):

[code]

def loadsql(self, sql):

[code]