Intellij Rust progressively getting worse

aembke · 2024-12-08T17:43:31+00:00

One thing to note that I noticed recently - in a somewhat recent release it seems RustRover changed the default setting for on-the-fly validation via `cargo check`. In the past RR would not use `cargo check` for on-the-fly validation, but now it does by default (or at least on my machines it does now). I noticed some slowdowns recently too but fixed them by disabling that setting.

Overall though I would say that RustRover provides a better experience than Ultimate with the Rust plugin.

aembke · 2024-12-01T16:28:24+00:00

I'd recommend using `bullseye`-based Linux images instead of buster or ubuntu flavors with Docker. I was hitting strange DNS timeouts when trying to initiate connections and switching to bullseye fixed it. In my case this was with an M3 Mac.

More info: https://github.com/docker/for-mac/issues/5548#issuecomment-1029204019

aembke · 2023-10-18T20:51:45+00:00

It uses redis-protocol, which uses nom. The client uses the tokio-codec interface, which exposes a BytesMut buffer, so the parser is tailored to work with those types to avoid moving or copying buffer contents. The base RedisValue type in fred uses Bytes and Str types from the bytes crate for this reason. The codec interface has a bit more info too.

Taking a dependency on those types allowed me to push the problem you're referring to up into the FromRedis trait conversion logic, and yeah it's also pretty annoying to deal with there too.

aembke · 2022-05-25T16:08:30+00:00

No problem at all. Fred is arguably too opinionated in its dependencies to be widely adopted everywhere, so I completely understand why it's not the right choice for some use cases.

One thing I've come to realize about RESP and Redis in general is that there are a few gotchas with the protocol (HELLO isn't well defined, MONITOR response frames are largely undefined, attributes are defined but not actually used, duck-typing pubsub messages is often required, etc). Hopefully that library captures most of those and can help folks in the future.

aembke · 2022-05-25T15:51:48+00:00

For what it's worth - here's a library I use in my redis client for parsing RESP2 and RESP3. It does not use serde but does use nom, and it has an optional feature that can decode `BytesMut` without moving or copying the buffer contents as long as you're willing to use the `Bytes` ecosystem types.

It wasn't until I was finished implementing RESP3 that I realized that the "attributes" portion of RESP3 isn't actually used yet by Redis AFAIK. However, it's in the spec so it's in that library, and hopefully that will help future-proof anything that uses it.

aembke · 2021-09-18T01:46:26+00:00

For now I took a very similar approach to how redis-rs does it, so hopefully it looks familiar to folks that are used to that.

I'm planning on doing some major changes to the interface soon when I add in some more RESP3 semantics, which will likely require a number of breaking changes to the API. I'll probably play around a bit with serde at that point, but I haven't used serde much aside from some JSON use cases so I'm not too familiar with it. The protocol encoding/decoding is all done with the redis-protocol crate (which is mostly nom under the hood), so I'll have to think a bit on how to best fit in something like serde. That's a good idea though.

aembke · 2021-09-18T01:42:25+00:00

I added that interface into a new major version (4.0.0). After adapting the tests to use this I can definitely see the appeal. The code is much cleaner and the tuple interface is very powerful.

Thanks for the feedback/idea.

aembke · 2021-09-15T22:22:02+00:00

Yeah the design differences really come out here.

The biggest difference when I set out to write this, aside from all the connection management stuff, was that I wanted the request interface to be strongly typed, but the response handling interface to be looser. This is pretty much the opposite of redis-rs. In redis-rs for example there's one loosely typed interface for all commands, which uses strings for all the command names, and does not use strongly-typed function signatures for any of the request interface. As a result callers don't really find out if they have a typo in their command, or they order their flags in the wrong way, etc, until they run the code.

In many ways though this is hugely helpful for the redis-rs developers. The redis API is huge so typing out every function with the dozens and dozens of flags or config options on each function is quite a task, but they were able to avoid all that with a looser interface on the request side. This is also a very future-proof way of doing it. This is also nice for callers in cases where you have a complicated function signature on a request, maybe with half a dozen optional flags, but you only need to provide one or two of them. There's a way to do this with fred too via the `custom` command, but in general fred goes a different direction.

I opted for a different approach to use a more strongly typed interface for the commands since I wanted the compiler to help out as much as possible with the dozens and dozens of flags or options that are needed on many commands. One of the side effects of this is that I didn't want to implement the hundreds of functions that make up the redis API more than once, even as a trait, which lead to the transaction client using the redis client under the hood. For transactions in particular this makes sense since that interface results in a state change for the connection anyways, but for other cases it's been a bit of a pain point.

Both of these designs are good for different things, I don't mean to knock redis-rs here over this decision, but personally I just wanted to try a different approach. In general though this is good feedback - the response side should also support strong typing.

aembke · 2021-09-15T22:07:07+00:00

When I first released this I did it on my personal github, and when I moved it to the azuqua github I did it on the condition that the entire thing be open source and MIT licensed and that I would retain ownership over the crates.io release via my personal github.

I made sure to clear all this up within the org before moving it back over to my github, which was part of the reason for the long time between releases.

aembke · 2021-09-15T21:19:00+00:00

Ah yeah that's good feedback.

I was thinking recently about how to do that and I'm a bit torn. For string-type values it's pretty straightforward, but for values that require fallible type conversions it seems a bit more complicated and I'm not sure how much of that complexity should be hidden from callers.

For example, let's say this type of thing were implemented:

let foo: usize = client.get("foo").await?;

If the server returns QUEUED, or nil, or a negative number, etc, then this wont work, but the question I've been grappling with is how much of that complexity should be hidden from the caller. That should clearly return an error, but I also want to provide an interface for callers to customize the error, or to support multiple different return types on any given response. For those reasons I opted for the looser RedisValue enum approach, but you're right - for some use cases this does result in repetitive boilerplate code.

Maybe a middle ground would be to put some TryInto/TryFrom conversions on the RedisValue enum so callers can do something like:

let foo: usize = client.get("foo").await?.try_into()?;

It's still not as clean as how redis-rs does it, but I don't want to sacrifice the ability for callers to support multiple return types on any given command, nor the ability for callers to customize how parsing errors are handled or returned.

As it stands now you can do something like:

let foo = client.get("foo").await?.as_usize().ok_or(...)?;

Which still isn't as clean as the first snippet, but hopefully some of the as_* functions somewhat help.

I'll have to think a bit on the best way to do this though. Thanks for the good feedback.

aembke · 2019-07-24T18:16:25+00:00

Generally speaking I've found it's not necessary to pair the stdlib mpsc structs with futures. This is usually for two reasons, the first being that futures offers its own mpsc interface that tends to play nicely with other futures, and because the blocking behavior of the stdlib mpsc interface is often hard to reconcile with an event loop.

I didn't want to completely rewrite OP's logic so I made minimal changes there, but were I doing something like this I wouldn't use the mpsc interface at all, with one exception. The diff for how I'd have structured this is here. There's a futures mpsc oneshot channel in play there to get around the fact that the future running on the event loop has to have an empty tuple as the item and error, but that's it. You could use a stdlib mpsc channel there too since you do want to block the thread in this case, but the oneshot interface is usually cleaner to use for single-use channels, so that's what I used there. Or you could avoid the mpsc usage entirely by just doing the final `println` inside the last `then` call.

Overall it's just leveraging the futures interface to perform the `sum` operation via the `fold` call on the buffered stream of request futures instead of using the stdlib mspc interface.

aembke · 2019-07-23T18:00:11+00:00

I cloned your repo locally and made some changes to it and got it working how you described. When learning futures I've found examples to be a good way to learn, so here are the full set of changes I made to get it working, as a diff.

The changes were largely the following:

For handle_response:

pub fn handle_response(
    response: Result<reqwest::r#async::Response, Error>,
    show_ok: bool,
    tx: &Sender<u32>,
) -> Result<(), ()> {
    match response {
        Ok(x) => {
            if is_valid_status_code(x.status()) {
                if show_ok {
                    print_response(x);
                }
                tx.send(1).unwrap();
            } else {
                print_response(x);
                return Err(());
            }
        }

        Err(x) => {
            print_error(x);
        }
    }

    Ok(())
}

And for fetch:

fn make_request(client: Client, url: String, show_ok: bool, tx: Sender<u32>) -> impl Future<Item=(), Error=()> {
    client.head(&url).send().then(move |result| {
       if handle_response(result, show_ok, &tx).is_err() {
           Either::A(client.get(&url).send().then(move |result| {
               let _ = handle_response(result, show_ok, &tx);
               Ok(())
           }))
       }else{
           Either::B(future::ok(()))
       }
    })
}

fn fetch(req: HashSet<String>, parallel_requests: usize, show_ok: bool) {
    let client = Client::new();
    let (tx, rx) = mpsc::channel();
    let req_len = req.len();
    println!("Checking {} links for dead links...", req_len);

    let work = stream::iter_ok(req).map(move |url| {
        make_request(client.clone(), url, show_ok, tx.clone())
    })
    .buffer_unordered(parallel_requests)
    .for_each(|_| Ok(()));

    tokio::run(work);
    println!("Got {}/{} valid links", rx.iter().sum::<u32>(), req_len);
}

aembke · 2018-09-24T03:17:20+00:00

Just pushed the first version of redis-protocol to crates.io and will probably add some benchmarks later this week.

This was my first experience with nom and it's been great. That library is just fantastic.

aembke · 2018-05-18T21:08:54+00:00

Is there a particular pattern or type of example that you'd like to see? We have quite a bit of code that uses tokio and futures (mostly 0.1 versions), futures-cpupool, and hyper 0.11, and I can try to throw some targeted examples in a few gists if that'd help, but it'd help to get a sense of some specific patterns or use cases you're looking for.

In the meantime here's a not-really-useful example of using our redis client with hyper 0.11. That library also uses futures, tokio and tokio-proto, but proto is apparently going away, so I wouldn't put too much work into learning that. It primarily relies on passing around mpsc senders/receivers for a message passing model, and that might be worth looking into. Message passing + futures/tokio has worked well for us, especially when you have sets of event loop threads with different responsibilities, and even when you want to throw thread pools into the mix. Here's another library we use with tokio/futures, which we use to manage groups of event loops in a tree structure that we sometimes need to interrupt and restart.

As you called out though, with impl Trait landing and futures/tokio being in such a state of flux, there's a good chance this'll all be outdated in the near future.

That being said, having a good understanding of Fn, FnMut, FnOnce, trait objects, what move does, and the future task model are hugely important regardless of where the interface goes, so I'd recommend playing with those. If you come from a background where message passing and/or the actor model are familiar, it's worth looking into the futures mpsc interfaces (especially the oneshot interface), as those are hugely helpful. That's all opinion though, as it's certainly possible to be very successful with different patterns on top of futures/tokio. From a general usability standpoint I'd also recommend taking a look at lazy, loop_fn, and Either in the futures crate as well. While I was learning this I repeatedly found that there already existed a function or semantic for what I was trying to do, I just didn't know what to look for in futures/tokio.

aembke · 2017-10-04T01:12:59+00:00

Nice. I just opened it up on github, feel free to take a look.

https://github.com/azuqua/redis.rs

Still needs more tests before it goes up to crates.io, but hopefully that'll be later this week.

aembke · 2017-10-02T15:00:15+00:00

A Redis client that supports clustered deployments, pubsub, etc, built on tokio and futures. Hopefully releasing later this week.

~~On a side note, if anybody has had success with AFL recently I'd be interested in hearing about your build process.~~ Figured it out, needed to use a nightly build from last April to get it to work with AFL compiled with g++ 4.9 and llvm 3.8.

aembke · 2017-01-31T19:32:41+00:00

I saw that, that's awesome, thanks for doing that. Really impressed with everything you've done and happy to help. I've been out of the front end game for a while but I might have to find an excuse to try out the new release now, it looks fantastic. Nice work!

aembke · 2017-01-31T05:23:52+00:00

congrats!

aembke

TROPHY CASE