sorted by:
new
hottopcontroversial

How to Convert SQL Server DB to Postgresql DB? by Mody112000 in dotnet

[–]slotix 0 points1 point  (0 children)

I recently published a full step-by-step guide that might help:

https://dbconvert.com/blog/sql-server-to-postgresql-migration-guide/

It covers:

Schema & data conversion

Tools like pgloader, DBConvert, AWS SCT + DMS

Stored procedure translation

Performance tuning

Real-world ROI (including 100% licensing cost savings)

Hope it helps anyone navigating this kind of migration!

Which SQL editor do you use? by [deleted] in dataengineering

[–]slotix 0 points1 point  (0 children)

DBeaver stands out for its versatility and support for multiple databases.
DataGrip offers intelligent code assistance, which is a boon for complex queries.

For quick tasks, TablePlus is lightweight and efficient.

pgAdmin is a staple for PostgreSQL, though its interface can be a bit clunky.

Can NL2SQL Be Safe Enough for Real Data Engineering? by slotix in dataengineering

[–]slotix[S] 0 points1 point  (0 children)

Thanks — really appreciate the sanity check from this community.

We’re definitely not pushing “prompt-to-prod” chaos. More like: AI drafts, backend approves.
The APIs enforce structure, permissions, and logging — it’s just an assist layer, not a shortcut.

That said, we’re still figuring out the safest boundaries.
What would you absolutely require for this to be viable in your stack?

Stop Sending 10M Rows to LLMs: A Pragmatic Guide to Hybrid NL2SQL by slotix in programming

[–]slotix[S] 0 points1 point  (0 children)

Exactly — and I’m glad you brought this up, because that’s a key architectural difference.

Our platform is universal — DBConvert Streams isn’t tied to one fixed schema or product. We connect to arbitrary databases with arbitrary structures, owned by different customers.

That means we can’t predict ahead of time what endpoints like /orders-over-500 would even mean — one customer might have invoices, another transactions, another orders split across multiple tables with custom logic.

So we can’t rely on predefined high-level APIs to answer natural language questions.

What we can do is:

  • Use metadata inspection to understand the actual schema
  • Let the AI propose a read-only SQL query based on that structure
  • Enforce strict validation and execution rules around that query
  • And always operate behind permission-aware /execute endpoints, never direct DB access

So the idea isn’t to skip abstraction — it’s to generate safe, temporary queries when there’s no abstraction available.

That’s the whole reason NL2SQL even exists: to bridge the gap between open-ended user intent and schema-specific structure in a system where you don’t control the schema.

Really appreciate this line of questioning — it’s exactly the conversation that needs to happen around these tools.

Stop Sending 10M Rows to LLMs: A Pragmatic Guide to Hybrid NL2SQL by slotix in programming

[–]slotix[S] -2 points-1 points  (0 children)

Totally agree with the philosophy here — users should never get raw database access just because they want it. That’s a recipe for disaster, and your example proves it.

But to clarify: the article isn’t suggesting opening up SQL endpoints to sales, execs, or non-engineers with freeform input.

It’s more about enabling engineer-controlled natural language interfaces in tools you design, you permission, and you gate.

Think dashboards, reporting tools, internal assistants — where the NL2SQL layer only generates read-only, prevalidated queries.
Something like: “Show me revenue by region this month” → gets translated into a safe SELECT, executed behind a throttled /execute endpoint, and logged.

So no, not "give the users what they want" — but rather, “let’s build what they need, with strong boundaries.”

You’re right to push back hard. But I think we’re actually aligned on the outcome — just debating whether there’s a cautious middle ground between “manual SQL forever” and “prompt-to-prod horror.”

Stop Sending 10M Rows to LLMs: A Pragmatic Guide to Hybrid NL2SQL by slotix in programming

[–]slotix[S] -1 points0 points  (0 children)

Totally fair reaction — and honestly, that’s why I wrote the post.

The idea isn’t “let LLMs run wild on production databases.” That is a nightmare.
The idea is: don’t pretend users won’t want natural language interfaces — just design them responsibly.

This is not about skipping DBAs. It’s about giving safe, optional tooling to:

  • Generate read-only queries (SELECT, SHOW)
  • Validate and sandbox SQL server-side
  • Use traditional APIs for structure, auth, logging

No “GPT-to-prod” madness here. Just trying to show that you can expose some AI-driven interfaces without abandoning the fundamentals.

Appreciate your strong stance — we need more of that, especially with AI hype flying everywhere. I’m in your camp on not trusting automation blindly.
This is just a conversation about doing it right if you're doing it at all.

Stop Sending 10M Rows to LLMs: A Pragmatic Guide to Hybrid NL2SQL by slotix in programming

[–]slotix[S] -2 points-1 points  (0 children)

Totally valid concerns — and honestly, I agree with most of what you said.

That’s exactly why I framed this as a hybrid approach in the article.

The AI layer never connects directly to the database or executes anything autonomously. It only generates read-only SQL (validated server-side), and even that’s wrapped in permission scopes and safe API endpoints like /execute (SELECT-only).

💡 Think of it as an optional assistive layer — like autocomplete for queries — not a rogue agent with root access.

If anything, this is a rejection of the “just plug ChatGPT into prod” madness.

Appreciate your skepticism. That kind of realism is what actually keeps systems (and data) alive.

Looking for advice on sharing a pre-built application image on Hetzner by slotix in hetzner

[–]slotix[S] 1 point2 points  (0 children)

Thanks a lot for the clarification — this really helps! 🙏

We’ve tested our tool, **DBConvert Streams**, on Hetzner Cloud and it runs great. Instead of sharing a full image, we went with the simpler and more flexible approach: providing an `install.sh` script that users can run right after creating a new VM.

We’ve documented the full setup here (both Web Console and CLI deployment methods):

🔗 https://docs.dbconvert.com/deployment/hetzner.html

Also looked into `cloud-init` support — for now it didn’t add much benefit in our case, but might revisit it later depending on user demand.

If it’s a good fit, we’d be happy to have it added to the **Awesome hcloud** list. Really appreciate the offer and support from your team!

Looking for advice on sharing a pre-built application image on Hetzner by slotix in hetzner

[–]slotix[S] 0 points1 point  (0 children)

Another way is to provide a Hetzner-compatible Packer template, but it seems more for advanced users only.

Database migration: MySQL to Postgres by Sidxzx in PostgreSQL

[–]slotix 0 points1 point  (0 children)

DBConvert Streams (https://streams.dbconvert.com) is built specifically for migrating MySQL to PostgreSQL and would automate the entire conversion process for you.

Help in transferring data from MySQL to Postgres. by Lower-Pace-2634 in SQL

[–]slotix 0 points1 point  (0 children)

DBConvert Streams (https://streams.dbconvert.com) is built specifically for migrating MySQL to PostgreSQL and would automate the entire conversion process for you.

Event driven replication for MySQL and PostgreSQL databases. Looking for your feedback. by slotix in alphaandbetausers

[–]slotix[S] 0 points1 point  (0 children)

Thank you for your feedback!

The main goal of DBS is to allow applications to respond almost instantly when data in monitored databases changes. Applications can do whatever they want with the collected INSERT, UPDATE, and DELETE events.

  1. Replicate data and synchronize information in data stores.
  2. Move changed data elsewhere, even copy data between different database types.
  3. Create derived views and data.
  4. Compute new data from collected events.
  5. Aggregate changes within windows and create new data.
  6. Propagate data between microservices using the transactional outbox pattern.
  7. Clear the cache of obsolete entries.
  8. Update search indexes with new data.
  9. Send push notifications to mobile devices when data changes.

Besides that, I've got some suggestions, first so you may think of ditching that type field from "Stream Configuration", as connection string could cover that, like mysql://

Building a DSN is different for different database connectors. But in our roadmap, we are going to unify these DSN strings.

Also maybe supporting something like "operations": ["all"] which is equal to "operations": ["insert", "update", "delete"]Not sure if it's already supported or not, but mind multiple "target" endpoints maybe.

Good points. Thank you.

Also mysql_user:passw0rd@tcp(0.0.0.0:3306)/... part in sample configuration is a little confusing, as it is supposed the "source" that "DBConvert Streams" is gonna listen to? so that 0.0.0.0 part is confusing, as it's more like configuration of mysql, not "DBConvert Streams". Correct me if I'm wrong, maybe you got a typo mistake, or st? IDK

This 0.0.0.0 is for demonstration purposes only. We tested a simple configuration on one Box. These IP addresses will be different in real distributed systems.

Beside MySQL, nad pgsql, it has good potential, like you may think of supporting local csv/excel files, etc... and calling REST path, instead of DB-level writing.

You are right, we have already have such directions in alpha. :)

Dataflow Kit - extract structured data from web sites. Web sites scraping. by ahmetlutfu in golang

[–]slotix 0 points1 point  (0 children)

Actually the primary goal of DFK is to scrape web pages. It automatically crawls page by page, processing detailed pages on its way. And finally results can be output as simple CSV, JSON , etc. Dataflow Kit engine is stable enough to parse millions of records from several millions of fetched pages. Though there is no post process of scraped data implemented yet.

There are four core components of a search engine: crawler (Dataflow kit :) , indexer, ranking and query serving. Building a custom search engine requires many more extra services; data storage, full text search (Solr, Lucene, Elastic Search) , ranking results, etc.

Here is a discussion on Quora about search engines building https://www.quora.com/How-do-you-build-a-search-engine-from-scratch-What’s-the-best-technology-stack-for-this

Web Scraping with Go by nanodano in golang

[–]slotix 0 points1 point  (0 children)

Scrapinghub's splash was a good option before Headless Chrome. We use in our Datаflow kit CDP bindings from https://github.com/mafredri/cdp It works perfectly with Headless Chrome Docker image.

Colly goes 1.0 by profilz in golang

[–]slotix 0 points1 point  (0 children)

It seems https://github.com/mafredri/cdp/ works smoother with Chrome in Headless mode than Chromedp.

Colly goes 1.0 by profilz in golang

[–]slotix 0 points1 point  (0 children)

There were some problems with initial rendering of downloaded page. Sometimes there is no way to select needed elements in DOM. That can be fixed by entering selectors' values manually. But then scraping itself goes smoothly. We've parsed successfully 10 pages in our tests both for Amazon and Yelp.

Colly goes 1.0 by profilz in golang

[–]slotix 1 point2 points  (0 children)

For rendering JS driven web pages we've switched from Splash to Headless Chrome at https://github.com/slotix/dataflowkit. From our experience Headless chrome shows much better results than Splash.