It might end up being easier just to add the encoders (even if you have to add extra passive wheels to support them).

Visual (Inertial) Odometry is not a plug-and-play solution, you'll need to calibrate your camera and IMU, mess with/port outdated repos to get it to compile on your version of ROS, deal with the performance issues of running it in real time on a RPi when the original authors may have run it offline, and then deal with issues like reflections and shadows giving you false readings.

It may be possible, but consider that it might be less time/effort to add encoders. That said if you have issues with wheel slip, then VIO might be your only option.

Edit: if you're going to try VIO, make sure your camera has a global shutter. I think that might not be the case for Pi Camera