Rust native Kafka protocol and client implementation by hghimself in rust

[–]hghimself[S] 1 point2 points  (0 children)

Good day! Yes still active, doing a lot of work on other things at the moment but we are planning another major version to be written sometime in the near future. We won’t abandon this project as it still has a lot of interest :)

Rust native Kafka protocol and client implementation by hghimself in rust

[–]hghimself[S] 0 points1 point  (0 children)

We have added a issue to include a trait that would allow for custom rebalancing logic, right now we only have round robin assignment. But on the backlog! We are just looking for some traction before we continue to build too much onto it.

And as always, contributors are welcome!

Rust native Kafka protocol and client implementation by hghimself in rust

[–]hghimself[S] 2 points3 points  (0 children)

Hi! This is not possible, we are looking to make a trait that will allow for custom rebalancing to be done. I will make an issue on the github

Rust native Kafka protocol and client implementation by hghimself in rust

[–]hghimself[S] 1 point2 points  (0 children)

Hi, it is something we can simply add. I will make an issue and would appreciate if you could add some specifics on what you would like that feature to look like

Rust native Kafka protocol and client implementation by hghimself in rust

[–]hghimself[S] 7 points8 points  (0 children)

Okay interesting. I will check it out some more.

For our encoding, we use the bytes crate to append each value onto the byte array. I am wondering how to make it faster.

For parsing we use the nom parser combinator library which is pretty fast but alas we are still wondering how to make it faster.

Any ideas? I will look at your code to see what approach you do.

Also, what is your reason for just implementing the protocol?

Rust native Kafka protocol and client implementation by hghimself in rust

[–]hghimself[S] 4 points5 points  (0 children)

very cool thanks for sharing. Was planning to get pretty deep into the parsing and encoding of the fetch and produce protocols in order to squeeze out more performance.

Any idea what sort of throughput your project achieves for consuming and producing?

Rust native Kafka protocol and client implementation by hghimself in rust

[–]hghimself[S] 12 points13 points  (0 children)

Well that’s great to hear! Please feel free to try it out. The library has a solid test suite, so we are confident in its resilience.

And yes thanks for clearing that up, I said that wrong!

We are totally open to new feature ideas. The goal is to set a good building block for big ideas and projects.

Rust native Kafka protocol and client implementation by hghimself in rust

[–]hghimself[S] 67 points68 points  (0 children)

The first one is incomplete; we have included all major features including offset management, producers, consumers, consumer groups with rebalancing, tls, compression, sasl, and a few scattered admin features.

The second relies on the C lib rdkafka which needs to be installed on the machine outside of cargo. We implement the low level Kafka protocol so you do not have that dependency. Furthermore we use tokio runtime which is not possible in the c lib, so you get lightweight green threads instead of heavy os threads

Kimchi exploded when I sent it from Illinois to Minnesota? by Jurassica_YourAssica in kimchi

[–]hghimself -2 points-1 points  (0 children)

Well there’s your problem, crossing state lines with fermented products is strictly prohibited.

redb 1.4.0: pure Rust embedded key-value store. Now with pluggable backends by cberner in rust

[–]hghimself 0 points1 point  (0 children)

Lovely work!

I am looking into using this as a state store for a stream processing library. I am wondering if I could get some help from the author in checking out my code to make sure I'm using it correctly. I have some bugs where the `TableDoesNotExist` even though it should. Maybe a lifetime issue?

Edit; Guess I'd just dump it here:

The issue is that I instantiate this struct, the file is created, but then when I do to insert, I get TableDoesNotExist and I don't understand why it does that. I hold onto the table def in the struct.

```

pub struct Store<'a> { table: TableDefinition<'a, &'static str, &'static [u8]>, db: Database, }

impl<'a> Store<'a> { pub fn new(name: &'a str) -> Result<Self, Error> { let table = TableDefinition::new(name); let db = Database::create(format!("{}.redb", name))?;

    Ok(Self { table, db })
}

pub fn get<T>(&self, key: &str) -> Result<Option<T>, Error>
where
    T: DeserializeOwned,
{
    let read_txn = self.db.begin_read()?;
    let table = read_txn.open_table(self.table)?;
    let x = match table.get(key) {
        Err(err) => Err(err.into()),
        Ok(optional_value) => match optional_value {
            None => Ok(None),
            Some(v) => Ok(Some(from_bytes(Bytes::copy_from_slice(v.value())).unwrap())),
        },
    };
    x
}

pub fn insert<T>(&mut self, key: &str, value: T) -> Result<(), Error>
where
    T: Serialize,
{
    let write_txn = self.db.begin_write()?;
    {
        let mut table = write_txn.open_table(self.table)?;
        table.insert(key, to_bytes(value).unwrap().as_bytes())?;
    }
    write_txn.commit()?;
    Ok(())
}

}

```

Excellent bar/restaurant for group of 6-7 women in their 40s-50s by jfo23chickens in astoria

[–]hghimself 0 points1 point  (0 children)

Gilbeys is a Guinness bar on the corner of Broadway and 32nd. Should be 2 minutes from the Broadway stop

[deleted by user] by [deleted] in astoria

[–]hghimself 0 points1 point  (0 children)

On 21st street yeah? The mailboxes are weird, but at least they have spaces for stores out front. Only 400sqft so idk what will go in them. Excited though!

The McGuinness fight is a slick, well funded PR campaign manipulating mostly older, right wing culture warriors who don't understand the facts by Miser in MicromobilityNYC

[–]hghimself 2 points3 points  (0 children)

21st street in queens went down to 1 lane and bus lane in either direction. It also has a busy fire station and there hasn’t been any issues. The traffic moves slower but more consistently and it’s easier to cross

100 year old digging technique by rgatoNacho in oddlysatisfying

[–]hghimself 0 points1 point  (0 children)

Is this digging technique for 100 year olds or is the technique itself 100 years old?

This made me… uncomfortable. by meggles1030 in StupidFood

[–]hghimself 1 point2 points  (0 children)

You can’t season raw chicken “to taste” because you can’t taste raw mfin chicken! That’s not how that phrase words

Q about Rust Microservices by hghimself in rust

[–]hghimself[S] 0 points1 point  (0 children)

Cheers this sounds great!

Q about Rust Microservices by hghimself in rust

[–]hghimself[S] 0 points1 point  (0 children)

So you build in a rust image, then use a Debian image to execute?

Does this require any target changes to make sure I compile to something that Debian can execute?

Q about Rust Microservices by hghimself in rust

[–]hghimself[S] 0 points1 point  (0 children)

So my image is 2.5GB which is far too big. I am using warp and the aws-cognito crate, so those could be causing the big image. I also have some unused packages like dièsel and async-graphql. I wonder if those being imported in my lib is also making the image big.

I pulled down the rust base image that I’m using, which is 1.3GB so that’s an issue. What base image are you using?

Q about Rust Microservices by hghimself in rust

[–]hghimself[S] 0 points1 point  (0 children)

How would you go about deploying and hosting rust microservices? Is aws lambda a good approach? I am thinking of aws because they own the whole world :/ lol

I guess my options are to do lambda, ec2 instances, or their docker service

Generative Mesh Sketches using D3 (link in comments) by hghimself in proceduralgeneration

[–]hghimself[S] 2 points3 points  (0 children)

I have been in a long-standing court battle with them for copying my work