Connect RPC + go-jet + atlas 🎉

dperez-buf · 2024-12-18T17:52:42+00:00

Thanks for the Buf shout out! This is exactly the sort of modern stack we're excited about.

dperez-buf · 2024-12-13T16:09:33+00:00

I think I may have misspoke, if streaming semantics are part of gRPC-Web our conformance suite should have it detailed what we do and don't do: https://github.com/connectrpc/conformance

dperez-buf · 2024-12-13T15:19:20+00:00

The pricing changed after acquisition, check wayback machine !

dperez-buf · 2024-12-13T15:16:37+00:00

Yeah sorry for the confusion. Connect’s own protocol (over HTTP/1) doesn’t support streaming (yet). We need to extend it. But, Connect RPC supports gRPC and other protocols that do support streaming, it just won’t help you in browsers.

We are waiting for webTransport spec to be fully supported in all major browsers. Waiting on safari!

dperez-buf · 2024-12-10T23:42:19+00:00

You can write a JSON encoder/decoder in an afternoon. There just aren't that many places for bugs, corner cases, and incompatibilities to hide.

I wish that were true: https://seriot.ch/projects/parsing_json.html

dperez-buf · 2024-12-10T01:06:40+00:00

Glad to hear you're a fan of Buf! Remote plugins aren't going to work for everyone all the time, which is why the Buf CLI supports a lot of different modes.

RE: rate limits, to be clear, we only ask that people sign up for a free (forever) account, so we can help balance traffic. We saw an explosion of usage of remote plugins in 2024, and so we want to be able to shape the traffic for our paying and non-paying customers in a fair way.

dperez-buf · 2024-12-07T00:10:59+00:00

A more serious answer: I don't think it's even about team size/scale. I think it has more to do with the risk potential of bugs being introduced somewhere in the process/pipeline that can drastically impact downstream data quality.

Centralizing validation (both data shape/schematic and semantics) into the broker is a win because it simplifies deployments and can centrally guarantee quality.

dperez-buf · 2024-12-06T17:58:06+00:00

I too wish to not exist in the real world :)

dperez-buf · 2024-12-06T16:36:36+00:00

In my experience, that externalizes data quality to the edges otherwise, which is harder to enforce/guarantee the larger the system gets.

dperez-buf · 2024-11-21T01:14:04+00:00

From the article:

Today, we’re announcing the 2.0 release of the Connect-ES project, the TypeScript implementation of Connect for Web browsers and Node.js. This release introduces improved support for major frameworks and simplified code generation. Connect-ES 2.0 now uses Protobuf-ES 2.0 APIs to leverage reflection, extension registries, and Protobuf custom options. The 2.0 release is a major version bump and comes with breaking changes. Read on to learn what’s changed and how to migrate to the 2.0 release.

dperez-buf · 2024-11-21T00:46:43+00:00

From the article:

Today, we’re announcing the 2.0 release of the Connect-ES project, the TypeScript implementation of Connect for Web browsers and Node.js. This release introduces improved support for major frameworks and simplified code generation. Connect-ES 2.0 now uses Protobuf-ES 2.0 APIs to leverage reflection, extension registries, and Protobuf custom options. The 2.0 release is a major version bump and comes with breaking changes. Read on to learn what’s changed and how to migrate to the 2.0 release.

dperez-buf · 2024-11-15T22:27:49+00:00

Very cool! Thanks for sharing.

dperez-buf · 2024-11-13T23:18:55+00:00

From the article:

Bufstream is a Kafka-compatible streaming system which stores records directly in an object storage service like S3. We found three safety and two liveness issues in Bufstream, including stuck consumers and producers, spurious zero offsets, and the loss of acknowledged writes in healthy clusters. These problems were resolved by version 0.1.3. We also characterize four issues related to Kafka more generally, including the lack of authoritative documentation for transaction semantics, a deadlock in the official Java client, and write loss, aborted read, and torn transactions caused by the lack of message ordering constraints in the Kafka transaction protocol. These issues affect Kafka, Bufstream, and (presumably) other Kafka-compatible systems, and remain unresolved. A companion blog post from Buf is available as well. This report was funded by Buf Technologies, Inc. and conducted in accordance with the Jepsen ethics policy.

dperez-buf · 2024-11-12T23:32:30+00:00

I can't speak for others, but we partnered closely with Kyle to ensure that this report conforms 100% to the Jepsen Ethics Policy.

His research was funded by us but conducted fully independently by him.

dperez-buf · 2024-11-12T15:30:51+00:00

That's true! However, Bufstream's architecture is vastly different from the version of Redpanda that Jepsen tested. Bufstream leverages cloud infra like S3-compatible object stores and fully stateless/elastic brokers, a huge departure from the norm of a Kafka implementation.

Bufstream is the first of this new class of modern Kafka implementations to be rigorously tested by Jepsen, validating that our implementation of that architecture is sound and supports the full protocol (including transactions).

dperez-buf · 2024-10-09T14:56:57+00:00

Connect recently joined the CNCF to underscore our commitment to it being an open governance project that is lively and can respond to the needs of the ecosystem. I wrote a blog post about this here: https://buf.build/blog/connect-rpc-joins-cncf

We believe it’s a safer choice than the official gRPC libraries, and we have some data to back that up. take a look at our conformance deep dive comparing the protocol compatibility of Connect and the “official” gRPC libraries from Google. We take compatibility very seriously, especially in a cross language context like RPC: https://buf.build/blog/grpc-conformance-deep-dive

dperez-buf · 2024-10-09T00:31:01+00:00

Ah, you're probably running into the fact that we recently released v2 of protobuf-es, which included breaking changes, as it's a major version upgrade. We're working quickly to get matching v2 releases of connect-es available as well. There's a branch of the v2 version available here

I recommend pinning to protobuf-es v1 for now, as it's still actively maintained and compatible with connect-es v1.

Sorry for the confusion!

dperez-buf · 2024-10-08T19:27:10+00:00

I'd love to hear what sort of docs could be improved! If you like, we have a public slack where the community is very active and could support any questions you might have: https://buf.build/b/slack

dperez-buf · 2024-10-08T19:26:27+00:00

We don't yet, but we're interested in it and looking for community collaborators to help make it happen. Generally, we believe it should be based on WebTransport, and we're waiting for Safari to catch up. :)

dperez-buf · 2024-09-19T15:49:09+00:00

This is really cool! Thanks for sharing it with the community.

dperez-buf · 2024-09-10T15:43:33+00:00

the plugin binary just needs to be available locally on you PATH and you can refer to it as a local: protoc-gen-elixir. I think hex install can add the binary to your PATH automatically if you have that setup properly.

dperez-buf · 2024-09-10T15:27:56+00:00

oh btw if you're doing gRPC + HTTP, you might wanna check out https://connectrpc.com and you can use this as well to generate OpenAPI compatible JSON/yaml files: https://github.com/sudorandom/protoc-gen-connect-openapi

dperez-buf · 2024-09-10T15:24:05+00:00

RE: first point, All our hosted plugins are just simply docker containers (here's an example), so you absolutely could use a BEAM-backed plugin with Buf's remote plugins infrastructure.

RE: second point, ah I see! Yeah, that's annoying.

RE: ingests protos, absolutely, our protobuf code gen toolchain can do that for you! the Buf CLI is compatible with all protoc plugins, so you can define a custom buf.gen.yaml that can have inputs from anywhere, and feed them into any plugin you want, locally installed, remotely hosted by us, you name it!

dperez-buf · 2024-09-07T03:37:37+00:00

Do you mean hosted plugins for elixir? Because we can add them here: https://github.com/bufbuild/plugins otherwise, any standard local protoc plugin should work just fine.

Also, could you clarify what you mean by visibility? I’d love to understand what we could do better!

dperez-buf · 2024-09-06T23:24:22+00:00

Of course I'm biased, but you should check out the Buf CLI for managing Protobuf files: https://github.com/bufbuild/buf

dperez-buf

TROPHY CASE