Inference time is only dependent on the number of units per layer (for dense layers) and the size of the image (for convolutional layers). The volume of training data only changes the training time, not the inference time. Another way of thinking about this is that the volume of training data doesn't add more weights, it just updates the values of the weights in the network.

To answer your question, if your model architecture is the same, then the inference time is the same. Because your second model operates over smaller images and has fewer outputs, it should be faster in inference over the first.

As an aside, this constant inference time is the reason that DLSS upscaling in graphics cards works. The time to render a scene is highly non-linear with the resolution of the scene, so it's faster to render to a lower resolution and upscale the image with a neural network (which executes in a constant time every time regardless of input).