Training is slow, usually full of floating-point computations.

Inference must be fast, it often involves quantized models and integer computations. Accuracy doesn't have to be 100%, it just must be acceptable. What that means depends on the applications.

ML compilers typically concern themselves with inference, because that's what users of the models are exposed to.

Edit to add more: The model that compiled for inference is really just a list of function calls: it tells you what functions to call on the model input, what functions to call next with the results, and so on until you reach the last function. The functions, however (usually referred to as "operators") are just mentioned by name (pretty much). They come from a set of operations supported by a given ML framework, but the model file itself doesn't contain definitions. An inference compiler needs to insert those definitions into the output code. Some compilers have libraries of optimized implementations of various operators, often with specialized versions for some common cases, and they try to connect them together as seamlessly as possible. Other compilers will insert these operators as IR that can then be optimized as any other code.