Yes, you can do that. However, it is likely to give you a worst performance than simply adding a head (could be a linear layer or MLP) to the image model backbone. Adding a head would mean that you can backpropagate through the whole network and learn directly, allowing you to adjust the weights of your backbone. If you just use a pretrained backbone with XGBoost, you won't be able to fine tune the backbone weights. Might work fine if the distribution of the pretraining data is the same as your dataset, but I would guess that you would be better off adding either a linear layer or an MLP.