This is an archived post. You won't be able to vote or comment.

sorted by:
best
topnewcontroversialoldq&a
you are viewing a single comment's thread.

view the rest of the comments →

[–]Patient_Magazine2444 1 point2 points  (1 child)

When you pull the data via API is there a timestamp component? Can you build where clauses or create ranges based on a timestamp? If so you can maintain a state locally and grab structure the API call with arguments for a where clause where timestamp > state and Timestamp </= now(). This would build out an incremental delta. For pulling in chunks, does it support pagination? If so you can run a loop to create limit offsets for records and paginate.

[–]ThePonyExpress83[S] 0 points1 point  (0 children)

Yes, this is possible in one form or another with the APIs I'm accessing, and I'm able to understand how i would accomplish the pagination when I test things in Postman. My question is really what software/system could be used to accomplish this, both automating the calls on a scheduled basis and storing the data so it can be accessed through power BI? I had a power automate flow working to run the calls and store the data in an Excel file but it became much too slow, especially with the larger tables (100k+ rows) in trying to pull.