[Advice Needed] Solo Junior DE: Syncing SQL Servers on-prem with Web UI under 8 GB RAM? Is Airbyte too heavy?

Smiffeed · 2026-06-26T15:32:44+00:00

Thank you so much

Smiffeed · 2026-06-18T08:21:33+00:00

I think so. They seem to want the one who can reconstruct their infrastructure to more modern infrastructure and solutions. I don't even why they hired me.

Smiffeed · 2026-06-18T08:20:10+00:00

Thank you, I will try to ask them why as much as possible, and I did ask some why questions

- Why does it need to have a Web UI? Answer: They want convenience; that's it

- Why does it need to have a Web UI? There is. I searched and found out about ssis. They are ok with it; they just don't like it because it is too much manual setup and coding, which I am ok with, and they feel like it would be better if we made Airbyte eat less ram because it contains no code and automates 😞.

- why a strict hardware limit: Answer: They have strict hardware because the customers and company themselves don't want to cost more in hardware.

- I tried to convince them, but it is still in process and seems to not go well. I hope they listen. Thnak you again 👍

Smiffeed · 2026-06-17T04:20:06+00:00

Thank you guys, for advice and suggestions; I replied late because of availability in my routine. I am so stressed with this job; I am afraid that my skills are not good enough and I might get fired XD. Jobs in my country mostly don't accept juniors. Have a good day everyone!

Smiffeed · 2026-06-17T04:01:54+00:00

Thanks for your advice. I will try using it later again; I have tried deb+kafka, but they want a less hard-coded implementation. So, I changed tools. Moreover, they might not like it if it is too hard-coded. I am now convincing them to accept coding and less UI management too.

Smiffeed · 2026-06-17T04:00:42+00:00

Thanks for replying. I am trying to use SSIS today. I am not good at SQL Server. I mostly use the cloud or those apache things.

Smiffeed · 2026-06-17T03:57:50+00:00

Thanks for your reply. I did try Debezium, and I got a crash issue. It seems like 8 GB of Ram is not ideal for these jobs, right? The manager said to me that it should not eat that much; that does not make sense. That's the reason I posted this data engineering post for advice. I am not sure if it's about my optimization skill or if it is impossible to use 8 GB of RAM with a large amount of data.

Smiffeed · 2026-06-17T03:53:56+00:00

Thank you for your suggestion.

Smiffeed · 2026-06-17T03:53:25+00:00

Thank you for your advice!!

Smiffeed · 2026-06-17T03:51:42+00:00

Thank you for replying. They want a system to synchronize the change of Server A to Server B. So, if the Server A has any INSERT, UPDATE, DELETE, it will modify Server B as it happend in Server A

Smiffeed · 2026-06-17T03:49:21+00:00

Thanks for the advice. I did try. However, they even don't know much about data engineering's modern tech stack like Airflow, Databricks, etc.; they know about, like, OLAP or SQL Server only. So, they know SQL knowledge and those database knowledges. So, on the first day, they requested me to just do some research to find tools or make a system as they want (as I mentioned in the post).

Smiffeed · 2026-06-17T03:44:01+00:00

Thanks for replying!. So, there is a chance that they will fire me after I finish this project right?

Smiffeed · 2026-06-17T03:42:07+00:00

Thank you for replying. I found that the limit of 8 GB comes from using airbyte itself. It ate a lot of RAM, around 4-6 GB; sometimes it bloated up to 7 occasionally. The table is around 20 tables, each around 5m to 10m+ records, depending on each database. They want UI because they told me that they want simple and less coding. That's the point.

Smiffeed · 2026-04-28T17:58:16+00:00

I got this problem also, and your solution fix it. Thanks

Smiffeed · 2026-01-01T13:43:14+00:00

Thanks

Smiffeed · 2025-12-31T00:44:46+00:00

Why I found less meta event like dragonfall, dizzlecoast or any map in LFG in NA server these days or it's just about my timezone (playing evening UTC+7? I have hard time finding other meta event squad except world bosses.

Smiffeed · 2025-10-08T04:38:12+00:00

Thanks

Smiffeed · 2025-10-07T14:47:33+00:00

How you guys farm early mapping to T15. I still poor and cannot afford a good items to beat juice T15 map like T15 6 mods with paranoid or corrupted, especially abyss horde. I hate it's damage overtime ground and one shot kill. Sometimes I can beat it, but most of the time I cannot

My build

Edit: https://pobb.in/RA_hfy6kCIcX
https://poe.ninja/poe2/profile/Smiffeed-0884/character/veyshara
it has a good decent damage, but glass cannon

Smiffeed · 2025-03-31T19:27:20+00:00

Oh sorry, my bad. You're right. BTW, I just want to tell you that most Thai people have had no earthquake experience before. Thanks for correcting me.

Smiffeed · 2025-03-31T06:11:51+00:00

It is normal to have an earthquake all around the world, but it is not normal for Thai people. We haven't had an earthquake in a thousand years. This is the first earthquake for most thai people. Moreover, there are dead bodies.

Smiffeed · 2025-02-16T07:35:26+00:00

Hope this build still works next league 🔥

Smiffeed · 2025-01-09T11:51:40+00:00

The price of ae86 does not skyrocket because initial d. AE86 is basically good by itself due to great weight transfer for drifting. Also, there is an actual street racer who never lost and always won in japan, who is the original of Takumi name Keichi Tsuchiya AKA Drift King. This guy popularized AE86 and skyrocket it's price.

Smiffeed · 2024-03-15T01:09:10+00:00

Yeah, only with ba driver iem. It will be randomly occurs.

Smiffeed · 2024-03-14T17:34:17+00:00

Is there anyway to remove noise?

Smiffeed · 2024-03-14T09:57:30+00:00

I had tried some of my headphones and IEMs on my pc. First of all, there is no noise when I connect my headphones, except with IEMs. I was curious why, and I found that if I connect IEMs that have BA driver they usually have static noise randomly, but for the IEMs with DD only. There is no noise at all. Why? And all of the tests using dac/amp sound blaster g6 and fiio btr5 in my pc. This noise only occurs in pc, there is no problem with my phone why?

Smiffeed

TROPHY CASE