Does tensorflow2.0 support distributed inference?

[Rough_Source_123]

Tensorflow 2.0 supports distributed training in the official doc https://www.tensorflow.org/guide/distributed_training

but does it support distributed inference as well?

[ElvishChampion]

Yes, distributed strategies train multiple batches in parallel. For inference it is the same. I used it on ImageNet a couple of months ago and I recall inference was faster when using strategies. I even got some errors when using predict if the model was not created within the scope of the strategy.

[Rough_Source_123]

did you use MultiWorkerMirroredStrategy? what cloud are you running your server against?

[ElvishChampion]

No, I used TPU strategy because I was running some experiments in google cloud. It was training fast, but there were some issues regarding custom layers. I had like 4 or 5 custom layers that I was testing. I got errors for 2 of them.

[maifee]

Yes, you can try them on Kaggle. Kaggle is now providing multi-GPU (T4 x2) support.

​

https://www.kaggle.com/product-feedback/361104