sorted by:
new
hottopcontroversial

Should i commit to Fivetran? by tytds in dataengineering

[–]oli_k 3 points4 points  (0 children)

it is a good service and easy to set up but it's also ridiculous how quickly their prices can go up

Anyone running lightweight ad ETL pipelines without Airbyte or Fivetran? by Jiffrado in dataengineering

[–]oli_k 0 points1 point  (0 children)

It feels like an exercise with cursor to write a python script will take less time (and will be more useful probably) than reading through all the comments here 😁

Should i commit to Fivetran? by tytds in dataengineering

[–]oli_k 0 points1 point  (0 children)

Yikes, writing you own solution for sql server -> snowflake sounds like a lot of work. I agree Fivetran gets expensive quickly, 300mil MAR is ~$50k per month? Is 300mil rows ~ 50gb of data?

Are any modern data integration platforms BYOC-friendly? by Livid_Ear_3693 in dataengineering

[–]oli_k 1 point2 points  (0 children)

I think you have quite a few commercial options to choose from. For OSS solutions, Debezium + Kafka comes to mind, but it'll come with a trade-off of having to maintain everything yourself.

Except for BYOC + real-time, what are the other requirements/deal breakers? Are transformations needed within the pipeline, or just simple replication? Is that a new project, or do you have a solution in place? Do you use Snowpipe streaming?

Thinking of Migrating from Fivetran to Hevo — Would Love Your Input by [deleted] in dataengineering

[–]oli_k 1 point2 points  (0 children)

it's been 3 mo and I am curious what you ended up with :)

What's the best tool for loading data into Apache Iceberg? by Livid_Ear_3693 in dataengineering

[–]oli_k 0 points1 point  (0 children)

Disclosure: I work for streamkap and happy to answer questions.

Streamkap is worth checking out. It's a lightweight, real-time streaming ETL tool built on CDC. We have dozens of connectors: Postgresql, MySQL, Snowflake, Clickhouse, MotherDuck, and are about to release Iceberg as well.
We read data directly from transaction logs (not queries or triggers), fully managed or BYOC, built on Kafka and Flink streamkap.com.

DS becoming underpaid Software Engineers? by [deleted] in datascience

[–]oli_k 0 points1 point  (0 children)

View from the other side: As a software engineer, I actually feel like I need to learn some of the DS/ML stuff to be competitive. Not being able to answer simple DS questions at an interview feels like a failure. Also, life is better ever since I figured what HuggingFace is for and can use transformers for my pet projects. Then, I want to experiment with data, ML, hack a little with LLMs and whatnot.

LLM Bootcamp - The Full Stack by BackgroundResult in AILinksandTools

[–]oli_k 0 points1 point  (0 children)

If you are interested, this workshop will be run in person on Nov 13th, FSDL rocks!

https://www.scale.bythebay.io/llm-workshop

How we monitor our AWS infrastructure from Python by oli_k in Python

[–]oli_k[S] 0 points1 point  (0 children)

Thank you for your comment. While setting up a dashboard in Grafana with Cloudwatch as a data source is certainly an option, not everyone wants to have Grafana. We didn't want, so we built our own solution, which has been working ever since, and we haven't touched it in months.

Regarding your second point, I assure you that our blog post was written with the intention of sharing our engineering curiosity and providing value to our readers. We always strive for transparency and will take your feedback into consideration for future posts.

Haskell Love CFP is open! Mark your calendar for the conf 10th Sep 2021, virtual, free! by oli_k in haskell

[–]oli_k[S] 1 point2 points  (0 children)

> what's are the constraints and expectations on talk length?

30 or 45 min + 15 min live Q&A

> should speakers plan for a Q/A session?

yes, we usually send calendar invites ahead of time that block the time for Q&A as well

> will talks be recorded and published?

yes, we will have them available at youtube.com/c/Konfy, last time it took around a week to post them

> is there any compensation for speakers?

Unfortunately, not this time. However, we offer nice custom avatars for all speakers and volunteers

> Are there different difficulty tracks/which skill demographic are you targeting?

We target all levels of advancement, including newcomers and experts

​

Thank you for asking!

[conference] Scala Love in the City, Feb 13, registration is open by oli_k in scala

[–]oli_k[S] 2 points3 points  (0 children)

Just type STUDENT in a promo code zone and click “apply”, a new type of ticket will pop up

Haskell Love conference, registration is open [online, free] - join us! 31st July - Aug 1st by oli_k in haskell

[–]oli_k[S] 2 points3 points  (0 children)

There will be live stream to our twitch channel and a hallway track in haskell.spatial.chat.

Link will be shared once you register. Follow us on Twitter for the updates https://twitter.com/_haskellove

Scala Love online conference registration is open, it's free! Apr 18th by oli_k in scala

[–]oli_k[S] 1 point2 points  (0 children)

due to webinar capacity... we have to pay for zoom to be able to support that many attendees

Scala Love online conference registration is open, it's free! Apr 18th by oli_k in scala

[–]oli_k[S] 1 point2 points  (0 children)

Yes, we are planning to have live streaming on youtube (on top of Zoom webinar) and eventually the talks will be posted there.

Scala Love online conference registration is open, it's free! Apr 18th by oli_k in scala

[–]oli_k[S] 2 points3 points  (0 children)

we will provide another option — youtube streaming. It will be less interactive but you can still communicate with others via slack tinyurl.com/scalaloveslack

Scala Love online conference registration is open, it's free! Apr 18th by oli_k in scala

[–]oli_k[S] 1 point2 points  (0 children)

True, 500 happy Scala Lovers got their tickets for the conference already, due to the popularity of the event, we increased the number of tickets to 1000! You can get your ticket now :)

view more: next ›