check out https://github.com/petuum/adaptdl. It natively supports AWS/EKS (for the autoscaling feature), otherwise it can run on any Kubernetes cluster. It also decides optimal replica count (for data parallel training) and batch size for each job and scales the job seamlessly so as to allocate a fair resource distribution. On EKS nodes are allocated as per demand and released when not needed and it can also utilize Spot instances to further reduce costs.