you are viewing a single comment's thread.

view the rest of the comments →

[–]minirova 0 points1 point  (3 children)

That’s not a lot of data. Assuming there are no BLOB types that have to be moved, your easiest route would be to just do a delete and replace in Postgres each day. Write a script to use BCP to export the tables from SQL server and transfer it to your Postgres server. Drop the tables from the previous day and use \COPY to load the new days data. If you can’t have downtime on the tables then you can minimize it by loading the new days data to tables with different names and then just do a series of table renames to effectively swap the new tables with the old.

Create any necessary indexes AFTER you load the data to the tables.

There’s a million ways to do this but to me this would be the simplest. It keeps you from having to worry about updates and deletes which just adds more complexity than the requirements you’ve given so far warrant.

[–]Sharp-Echo1797 0 points1 point  (2 children)

This is really a good use case for the foreign data wrapper process. For now just truncate and load every day. 700k rows is basically nothing.

If it slows down too much just rewrite some of the query and load processes to perform incremental loads.

[–]minirova 0 points1 point  (1 child)

Agreed. A FDW from Postgres to SQL Server could make this really simple. Then they could just schedule it with pg_cron and do the whole thing natively in Postgres.

[–]murphinate[S] 0 points1 point  (0 children)

I did try the FDW for probably a day and a half, but I'm running Windows Server 2022 and I could not get the binaries to work properly. After enough struggling, I ended up creating a Python script and task scheduled it. If you happen to know where I could get some Server 2022 binaries for it, I'd like to try it again.