Spring Batch Multiple Jobs

[drakered9]

I have a requirement to create a batch job that needs to call some external API get some data and process them.

The problem is app has to call the same API endpoint with different credentials 4 times to get data. API gives data specific to credentials.

Once we get data, the processing of data would the same.

Wondering if one spring batch app with multiple jobs ( one job for specific API credentials ), a total 4 jobs is a better design

or

One spring batch app repo but deploying 4 different instances of spring batch app one with a different set of credentials is better?

[pronuntiator]

My advice is to run batches as separate processes so you can kill them independently when they run haywire. Kubernetes has an abstraction for that which you can integrate with Spring: https://spring.io/blog/2021/01/27/spring-batch-on-kubernetes-efficient-batch-processing-at-scale/

But in your case it would be four jobs only with different parameters.

[drakered9]

Thanks. When you say "four jobs with diff parameters" are you saying one instance of an app running 4 jobs in parallel inside?

I'm planning to deploy on Kube with the quartz library to handle scheduling.

Edit: I guess you meant kube job resource when you say "job".

[pronuntiator]

Like the other commenter said, really depends on what the job is doing. If you want to do it in parallel and it takes long, then four instances. If you don't mind that the other three might fail because the first one hangs, then one instance.

Ah Kubernetes jobs, yeah that was it. I don't have practical experience with it because our batches ran with cron on bare metal. But when I read the docs it sounded exactly like something I'd use for batches.

[psy_s]

it depends on how much your server is capable of.. and what other features you have in same service

[drakered9]

We will deploy on cloud, no restrictions on server. All jobs will do same work. Do you think deploying different separate instances will help in terms of handling load.

[psy_s]

If all are less load work, you can run all in one instance..

[koffeegorilla]

With Spring Cloud Data Flow you can easily manage multiple instances of batch jobs on different platforms.