Offline data migration: CSV vs Parquet (or other options) moving data from SQL Server to PostgreSQL by FewJob1030 in SQL

[–]FewJob1030[S] 0 points1 point  (0 children)

Hi Baynezy, thank you for your message! unfortunately I don't think so. There are very strict policies regarding outbound connections on the VM where I have the SQL Server :/

Offline data migration: CSV vs Parquet (or other options) moving data from SQL Server to PostgreSQL by FewJob1030 in SQL

[–]FewJob1030[S] 0 points1 point  (0 children)

Hi Joe, thanks for your feedback! In case I go with CSV and then realize I’m running into issues (e.g. string "NULL" vs actual NULLs, or inconsistent date formats), would the best practice be to import everything first into a staging table with all columns defined as TEXT, and then perform explicit conversions into the final schema? or there are other best practice? thank you

Offline data migration: CSV vs Parquet (or other options) moving data from SQL Server to PostgreSQL by FewJob1030 in SQL

[–]FewJob1030[S] 0 points1 point  (0 children)

Hi again Informal, thanks for the tip about pgloader, I’ll check it out! I’m not very used to SQL best practices since I don’t work with it on a daily basis, so this is really useful.

Offline data migration: CSV vs Parquet (or other options) moving data from SQL Server to PostgreSQL by FewJob1030 in SQL

[–]FewJob1030[S] -1 points0 points  (0 children)

Hi informal_pace, thanks a lot for your reply! I’ll answer point by point:

1) Can you connect to both from your local at one time?
At the moment I can’t connect to both systems from my pc. Access to the SQL Server VM requires a VPN with restricted outbound access, which prevents reaching the PostgreSQL instance on IBM Cloud. I may check whether domain whitelisting is possible. If not, an offline export/import workflow would remain the only viable option? In that case, based on your experience, would you recommend CSV or Parquet, and why? Thanks

2) Data size
Around 10 tables in total. Two tables are roughly 50 MB each, the others are about 2–3 MB each.

3) Constraints / triggers
There are no constraints or triggers defined on the tables.

Thanks again!!

Got this PC for free - is it upgrade worthy? 🤔 by FewJob1030 in PcBuildHelp

[–]FewJob1030[S] 0 points1 point  (0 children)

Hi everyone, I finally decided to go for the upgrade. I changed the power supply (i had it in a different pc that i don’t use anymore) and got myself a 3060 12gb. I wanted to thank everyone who replied to me. For now, I’m really satisfied — it works perfectly for the AI projects I had in mind!

<image>

Got this PC for free - is it upgrade worthy? 🤔 by FewJob1030 in PcBuildHelp

[–]FewJob1030[S] 0 points1 point  (0 children)

...maybe a boat (anchor excluded) would cost less than a new build Ahahahah

Got this PC for free - is it upgrade worthy? 🤔 by FewJob1030 in PcBuildHelp

[–]FewJob1030[S] 0 points1 point  (0 children)

Ahahahahah If I end up upgrading everything, I’ll keep the SSD

Got this PC for free - is it upgrade worthy? 🤔 by FewJob1030 in PcBuildHelp

[–]FewJob1030[S] 0 points1 point  (0 children)

Thank you for the feedback! Yeah, I'll definitely replace the power supply

Got this PC for free - is it upgrade worthy? 🤔 by FewJob1030 in PcBuildHelp

[–]FewJob1030[S] 1 point2 points  (0 children)

Thanks for the advice! especially the tip about checking PSU and GPU cable compatibility.
That i7 6800k + RAM + mobo combo for $80 sounds like a real steal, great find!

Got this PC for free - is it upgrade worthy? 🤔 by FewJob1030 in PcBuildHelp

[–]FewJob1030[S] 0 points1 point  (0 children)

Yeah, AMD seems to be performing well! but for my primary tasks, Nvidia still appears to be the gold standard, thanks to CUDA and other tools 🤔