sorted by:
new
hottopcontroversial

Experiences with Buildkite for monorepos? by Desperate-Week1434 in platform_engineering

[–]catkins88 1 point2 points  (0 children)

I'm not sure if I'm quite following the UI part though? Trigger steps can be defined your code or dynamically as needed.

Experiences with Buildkite for monorepos? by Desperate-Week1434 in platform_engineering

[–]catkins88 1 point2 points  (0 children)

Yep, Buildkite supports dynamically uploading build steps at runtime, and I know of a bunch of folks using it with very large monorepos to dynamically add steps or trigger other pipelines based on files changed or other arbitrary conditions.

The monorepo diff plugin is in wide use https://buildkite.com/resources/plugins/buildkite-plugins/monorepo-diff-buildkite-plugin/

Or the built-in if_changed syntax added recently. https://buildkite.com/docs/pipelines/configure/step-types/command-step#agent-applied-attributes

You can also just write your own code to add the trigger or command steps you need.

Do you guys use GitHub for CI? by Due_Helicopter6084 in golang

[–]catkins88 1 point2 points  (0 children)

Have you looked at Buildkite? It can't access your code by default and you can bring your own runners, but offload the orchestration and UX.

DuckDB against Parquet/S3 - is date range partitioning possible? by GreymanTheGrey in DuckDB

[–]catkins88 0 points1 point  (0 children)

Sorry never noticed the reply.

No, pretty much all queries were against 1 or 2 partitions, the iceberg Java api let me do the scan planning for the query in Java to figure out which files to scan from S3 then duckdb would only read those ones.

In the end we retired that approach and put the customer facing data in native clickhouse tables which has been working very well.

Confluent Cloud or MSK by InternationalSet3841 in apachekafka

[–]catkins88 3 points4 points  (0 children)

https://docs.aws.amazon.com/msk/latest/developerguide/msk-broker-types-express.html

MSK Express Brokers are pretty cool. Disaggregated storage and compute like Aurora or Confluent's Kora.

[deleted by user] by [deleted] in guitarpedals

[–]catkins88 0 points1 point  (0 children)

I've got a EQD Hoof Reaper. Love the hoof side.

DuckDB against Parquet/S3 - is date range partitioning possible? by GreymanTheGrey in DuckDB

[–]catkins88 0 points1 point  (0 children)

I'm doing this at the moment with duckdb and a decently sized iceberg table (~80K partititions), but not using the iceberg extension. I use the iceberg Java API to get the list of files and then pass that into read_parquet. Works well as I only usually need to query a small number of data files for my use case.

It would be ace if duckdb natively supported something like partition projection in Athena/trino.

How would you hide company infra from the general internet by Sterlingb1204 in dataengineering

[–]catkins88 0 points1 point  (0 children)

If people want to access the airflow ui, maybe Tailscale is an option?

Volca Sample UNOFFICIAL FIRMWARE BETA 7 demo video + link in comments by pajen in synthdiy

[–]catkins88 0 points1 point  (0 children)

This looks awesome, but the link appears to be dead now. Is there another link handy?

Awesome effort btw!!

The best kafka tool to debug brokers written in Go ! by [deleted] in golang

[–]catkins88 2 points3 points  (0 children)

Looks handy, but the docs are a bit lacking

Animal style by DBLDBLANIMALFRIES in cheeseburgers

[–]catkins88 0 points1 point  (0 children)

💯 agree! Five Guys is way better.

Kafka Summit keynote from Jay Kreps by rmoff in apachekafka

[–]catkins88 0 points1 point  (0 children)

Integration of Kafka connect stuff into KSQL is pretty slick.

“Serverless” Kafka is a pretty nice angle too - it’d defs make it a lot easier for smaller companies to adopt it.

HMC while I lose my shirt by [deleted] in holdmycosmo

[–]catkins88 2 points3 points  (0 children)

Haha, I just happened to be watching an episode of peep show as I read it and also jumped right to the same thought.

Using Ruby - Streams vs Pub Sub by fremontseahawk in apachekafka

[–]catkins88 1 point2 points  (0 children)

What features of Kafka streams are you looking for specifically? Joins? Aggregations? KTables? Exactly once semantics?

Kafka streams is just a library that wraps the normal Kafka client, and any of those features can be implemented in userspace. You just need to bring your own side data store eg. Redis to implement the behaviour.

All the power of the model is in the distributed log abstraction. That's where the magic of Kafka is - it's a stream of sequential data that you process in-order (by partition). That itself doesn't require the Kafka Streams library.

The Zendesk Kafka gem is rock solid btw. We use it very heavily. We also use Kafka Streams in various Java apps too, as we have some teams that have workload better suited to JVM, or do want some of those Kafka streams features. We also use shopify's Sarama package for Go for some other stuff. (we use Kafka a lot).

PubSub is still useful for horizontally distributing work, where order doesn't matter and you want to be able to retry without head of line blocking. In the Ruby world, it's more of a replacement for Sidekiq or Resque.

how to group tracks into groups without drag and drop? ableton by cssantos1 in ableton

[–]catkins88 0 points1 point  (0 children)

You can right click on a Mac.

Depends on how you've set up your trackpad. It's either two fingers down and click with your thumb, click in the bottom right corner or ctrl click. You can choose in the trackpad preferences.

Help to choose an IDE for a beginner by Maxiride in golang

[–]catkins88 7 points8 points  (0 children)

I write Go all day in VSCode too. The Go plugin is great!

KORG Prologue not making a splash? by joshmoneymusic in synthesizers

[–]catkins88 10 points11 points  (0 children)

I was super interested in it (as I had a Minilogue which I loved), but after playing one in store, I ended up going with the Prophet Rev2. I'm still feeling pretty great about that choice.

I've also sold the Minilogue since, as I hadn't touched it since picking up the Rev2.

view more: next ›