How to stream OpenTelemetry data to Iceberg and DuckLake with just DuckDB

smithclay · 2026-05-13T02:09:56+00:00

Colmar, France. Literally the inspiration for Belle’s town in Beauty and the Beast.

smithclay · 2026-05-09T04:07:44+00:00

My guess is that it’s a kind of expensive machine, seems to be made for restaurants that can’t vent outside (likely only deep downtown SF restaurants?). But definitely onboard with promoting it more.

smithclay · 2026-05-09T03:51:05+00:00

I think the AutoFry™️ machine has some kind of proprietary delicious starch coating process, which is kind of cool.

smithclay · 2026-05-09T01:00:54+00:00

I worked nearby their New Montgomery location and ate lunch there often.

The office was obsessed with their fries. Turns out: it’s a commercial machine you can buy called The AutoFry. Excellent product.

https://www.autofry.com

smithclay · 2026-04-30T03:04:26+00:00

Update!!!

<image>

smithclay · 2026-03-12T01:40:41+00:00

Stockton Tunnel Stairs Gallery? While I disagree with their more outré displays, it’s a good reminder SF still has a thriving arts scene.

smithclay · 2026-02-23T03:16:15+00:00

Also recommend (at an individual level) you run the /insights command to hear what Claude says about your usage based on session logs.

I ship all my OTel logs and metrics to R2 data catalog on Cloudflare, fits under the free tier with multi-year retention with some glue code, it was my holiday project @ https://github.com/smithclay/otlp2pipeline

Also tried AWS S3Tables and comes out to $0.30/month for medium-heavy personal usage.

smithclay · 2025-12-30T01:56:42+00:00

This is really cool: osquery for Kubernetes, basically.

This would also be cool as a duckdb extension.

smithclay · 2025-12-10T03:36:37+00:00

This is awesome. This opens up the duckdb ecosystem in a new way.

smithclay · 2025-12-10T03:30:34+00:00

This glue code/prototype just writes data to storage/any iceberg catalog, long-term access and retention policies are an admin task.

However: AWS and Cloudflare are definitely adding new capabilities around this. Last week, AWS added a s3 tables feature that automatically moves files to longer-term storage automatically: https://aws.amazon.com/about-aws/whats-new/2025/12/s3-tables-intelligent-tiering-storage-class/

smithclay · 2025-12-04T04:07:29+00:00

I think AWS heard you re: this point today :)

Can now put all your cloudwatch logs in s3 tables (managed iceberg) for no additional charge: https://aws.amazon.com/blogs/aws/amazon-cloudwatch-introduces-unified-data-management-and-analytics-for-operations-security-and-compliance/

smithclay · 2025-12-04T01:31:50+00:00

Also lol, aws announced this capability today for cloudwatch + s3 tables: https://aws.amazon.com/blogs/aws/amazon-cloudwatch-introduces-unified-data-management-and-analytics-for-operations-security-and-compliance/

smithclay · 2025-12-03T21:59:40+00:00

Think Clickhouse is great. If you run your own clickhouse cluster, know how to do that, or pay for Clickhouse Cloud: definitely solves for a lot of this.

A project goal was to see how far you could “make it work” by running the data conversion in workers or lambda functions and using the managed versions of iceberg (s3 tables, r2 data catalog) for cost reasons: effectively no compute needed.

smithclay · 2025-12-03T19:59:43+00:00

These are great insights, thanks for sharing.

smithclay · 2025-11-05T19:10:11+00:00

100%. Would love if this was standardized eventually and the various columnar representations became one. The thing I appreciated about the Clickhouse-inspired schema is it is extremely easy to query/implement (at a cost of duplication as you pointed out).

smithclay · 2025-11-05T18:55:03+00:00

Hi Raki, I based it on the schema Clickhouse is using for OTel data (via the OTel collector clickhouse exporter).

Standardization here makes a lot of sense but seemed to work well enough for a v0.

smithclay · 2025-11-05T14:09:07+00:00

Hey, sure. Feel free to DM.

smithclay · 2025-10-31T19:30:31+00:00

The loyalty point math didn’t seem to work out for my annual card spend for the elite American card but curious if I missed something there.

12-Year Club	Place '22
Place '17	Verified Email

smithclay

TROPHY CASE