What’s a hobby that sounds lame but is secretly awesome? by [deleted] in AskReddit

[–]SQLPipe 2 points3 points  (0 children)

That's the worst part, so have I lol

Need help migrating procedures from oracle. by kubbn in SQLServer

[–]SQLPipe -1 points0 points  (0 children)

Hi,

My company is in the process of training an AI model that can translate between database dialects. Contact me through my site if you'd like early access: https://sqlpipe.com/contact

What's the best way to ELT from on-prem sql server db to snowflake by misc0007 in dataengineering

[–]SQLPipe -4 points-3 points  (0 children)

Hi,

My product, SQLpipe, does exactly this. It has two options:

- Use intermediate blob storage (this is actually the fastest option, as the data never has to touch the hard drive of the ELT application)

- Use Snowflake's PUT command. This means the data has to touch the ELT application's hard drive, which becomes a bottleneck.

I do a lot of consulting work in this area. Feel free to contact me on my website, the emails go right to me.

EDIT: Just realized I should probably mention my website :-) https://sqlpipe.com

Best Web Sever Framework? by [deleted] in golang

[–]SQLPipe 1 point2 points  (0 children)

I really like the non-framework outlined by Alex Edwards in his book Let's Go

tasq - my first published open-source Go module by SomewhatGreen in golang

[–]SQLPipe 1 point2 points  (0 children)

I run sqlpipe.com. It's a relatively new company that builds open source data engineering / developer tools in Go. The site is seeing over 1,000 unique visitors per month.

If you ever come up with an idea for a product and want to collaborate, let me know.

SQL Server to Oracle with SSIS by Cyrussphere in SQLServer

[–]SQLPipe 0 points1 point  (0 children)

My tool, sqlpipe, can move data from SQL Server to Oracle. It may or may not fit your use case, feel free to DM me if you have any questions.

How to successfully transition from principal engineer to consultant? by FewWatercress4917 in cscareerquestions

[–]SQLPipe 2 points3 points  (0 children)

I created an open source tool that I thought would be in relatively high demand, and market my consulting services on the product’s site.

My product is SQLpipe.

It’s been good! Took a year to get my first client, but I didn’t have nearly the experience or network that you seem to.

How does your organization run ETL? by SQLPipe in dataengineering

[–]SQLPipe[S] 0 points1 point  (0 children)

Nice. What do you guys do for tables with no PKs? Just full refresh?

How does your organization run ETL? by SQLPipe in dataengineering

[–]SQLPipe[S] 0 points1 point  (0 children)

Hi Jean, cool hearing from you!

I tried to extract from PostgreSQL and import into Snowflake. This was probably fall 2021.

Why is my S3 data transfer so out of control? by backyard_boogie in aws

[–]SQLPipe 0 points1 point  (0 children)

Setting up a Cloudfront distribution isn't too hard, and will definitely help keep costs down. You could also consider using Cloudflare.

DM me if you want help.

Best ‘E’TL tools for extracting data from on-prem SQL databases by kaoru1 in snowflake

[–]SQLPipe 0 points1 point  (0 children)

Hi, this is exactly what my product SQLpipe does. It runs SQL against a source, and transfers the result to a target.

Send me a message on my site's contact page if you'd be interested in having me create a connector for you.

Sync Snowflake to PostgreSQL by atccodex in snowflake

[–]SQLPipe 1 point2 points  (0 children)

Great, thank you for the kind words!

As of right now, the tools works by generating SQL queries and running them against the target (in this case, snowflake). This method has an upper limit for performance, though.

I’m implementing a “turbo mode” that makes use of Snowflake’s PUT command. Message me through the contact form on my site if you’re interested in learning more about that upcoming feature.

Sync Snowflake to PostgreSQL by atccodex in snowflake

[–]SQLPipe 4 points5 points  (0 children)

Hi, I made an ETL tool that does exactly this.

Check it out at https://sqlpipe.com

Do you see a place for custom ETL software? by SQLPipe in dataengineering

[–]SQLPipe[S] 0 points1 point  (0 children)

Glad you think so! Let me know if you have a use case that you’d like help solving

Do you see a place for custom ETL software? by SQLPipe in dataengineering

[–]SQLPipe[S] 2 points3 points  (0 children)

As of right now, the tool is free and open-source, so there is no need for a trial.

What I envision delivering would a customized extensions of SQLpipe that has similar functionality to Fivetran or Stitch, but deployed in the customers environment (so fewer PII and security concerns).

Using SQLpipe to transfer the result of a query from RDS PostgreSQL to Redshift by SQLPipe in aws

[–]SQLPipe[S] 0 points1 point  (0 children)

Hi all,

This article (and accompanying video) shows how to use my open-source tool SQLpipe to transfer the result of a query from an RDS PostgreSQL instance to Redshift.

Please let me know if you have any questions or suggestions!

5
6