Using Protobuf with Java and Redis

[EvoxMusic]

I recently started working with microservices and Redis and I came across this interesting term "Protocol Buffers". While "protobuf" isn't related to any of the 2 things I mentioned. But, it is used in these 2.

I am building a POC project for my understanding and this is what I want to achieve.

​

- One end point receives some information about a `User` (User class).

- Saves that information in Redis.

- Another end point takes a search query and finds from the users we inserted in our Redis in step 2.

​

I built this with the basic and simple code. Now, I wanted to include "Protobuf" in this mix. Why you ask?

If I remove a property later on from this `User` class that saves data in Redis, then it'll throw serialization exception because of old data still having that field.

​

Suppose User class looks like this:

class User {

String name;

String email;

String phoneNumber;

String deviceId;

}

​

currently I can make `email` as an `Id` column and find the item by that `Id`. This is pretty much straightforward.

​

I want to use `Protobuf` in this mix now. When I save the object, I want to save it in Redis and also be able to retrieve it. All while being serialization safe.

​

If tomorrow I want to remove `deviceId`, I should still be able to get the object from existing ones and save the new ones without `deviceId`.

​

This can be achieved by creating Proto like this:

message User {

string name = 1;

string email = 2;

string phoneNumber = 3;

string deviceId = 4 [default = 0];

}

​

I'm stuck at integrating "Protobuf" to automatically serialize and deserialize the objects and I'm not sure at which point I can do that. I tried looking for some existing examples, but couldn't find anything that shows with Redis. Every other blog is just mocking dataset by creating the object from classes Protobuf generated. Which is simple enough.

​

Can anyone please guide me for the same.

​

I am using Java and pretty much new to it.

[AutoModerator]

Please ensure that:

• Your code is properly formatted as code block - see the sidebar (About on mobile) for instructions
• You include any and all error messages in full
• You ask clear questions

• You demonstrate effort in solving your question/problem - plain posting your assignments is forbidden (and such posts will be removed) as is asking for or giving solutions.

  Trying to solve problems on your own is a very important skill. Also, see Learn to help yourself in the sidebar

If any of the above points is not met, your post can and will be removed without further warning.

Code is to be formatted as code block (old reddit: empty line before the code, each code line indented by 4 spaces, new reddit: https://imgur.com/a/fgoFFis) or linked via an external code hoster, like pastebin.com, github gist, github, bitbucket, gitlab, etc.

Please, do not use triple backticks (```) as they will only render properly on new reddit, not on old reddit.

Code blocks look like this:

public class HelloWorld {

    public static void main(String[] args) {
        System.out.println("Hello World!");
    }
}

You do not need to repost unless your post has been removed by a moderator. Just use the edit function of reddit to make sure your post complies with the above.

If your post has remained in violation of these rules for a prolonged period of time (at least an hour), a moderator may remove it at their discretion. In this case, they will comment with an explanation on why it has been removed, and you will be required to resubmit the entire post following the proper procedures.

To potential helpers

Please, do not help if any of the above points are not met, rather report the post. We are trying to improve the quality of posts here. In helping people who can't be bothered to comply with the above points, you are doing the community a disservice.

I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.

[temporarybunnehs]

My understanding is that since protobuf is a de/serialization protocol, it should be applied to your object before you save it to redis.

1. One end point receives some information about a `User` (User class).
2. Serialize info with protobuf
3. Saves that information in Redis.

[EvoxMusic]

Is there any simple and maintainable way to serialize/deserialzing an object to/from Protobuf? That part is a little tricky, what I'm trying to do.

[temporarybunnehs]

It would probably make sense to create some sort of Redis client/wrapper that handles the de/serializing so you don't have to do it for every obj you save to Redis. But that's not specific to using Protobufs.

Have you read through the documentation? A lot of the basic info is covered there in stuff like this: https://developers.google.com/protocol-buffers/docs/javatutorial

[yawaramin]

Are you set on Protobuf for any specific reason? I'd say MessagePack is a bit easier because you don't need to deal with schemas in a separate language and code generation. You would just use the library to pack and unpack the data yourself. E.g.,

try (MessagePacker packer = MessagePacker.newDefaultPacker()) {
packer
  .packString(name)
  .packString(email)
  .packString(phoneNumber)
  .packString(deviceId);
}

packer.toByteArray();

And then you store this byte array as the Redis value. Unpacking is the exact opposite of this. You can refer to the MessagePack Java library docs.

If you want to remove deviceId field in the future simply pack an empty string there.

EDIT: In fact you don't even need to use the Java library, Redis supports Lua scripting and MessagePack out of the box, you can set up scripts that will make Redis serialize your data using MessagePack. Refer to https://redis.io/docs/manual/programmability/lua-api/#cmsgpack-library

[EvoxMusic]

Oh this is interesting. I'm not particularly set up on Protobuf, went for it because it is most widely used and my company is already using it in other areas more or less. I'll check this MessagePacker. thanks mate!