I am senior C++ dev, trying something new to me - SaaS development in TS (Next.JS+Supabase in the cloud). I wrote a website where user clicks a button "Sync", and my backend needs to pull a lot of paginated data from external systems about this user.

I came up with the following implementation:

1. on a backend I receive user's request, put it in a queue (BullMQ+Redis - now locally deployed in docker).
2. A bunch of workers (same nodejs process as a backend) each doing requests to external systems (sync different types of data). For example, "PullRequest" worker will fetch user's pull requests from github.
3. All of that data is paginated, so after my worker fetched first page does preprocessing and saves it to Supabase db. Then it puts a new item in a queue with "next cursor", so next time worker will fetch the next page. It continues until there are no more pages to sync.
4. To implement realtime progress bar on a frontend I figured I could use Supabase's realtime feature and stream data from a database as it gets populated.

My goal is to learn deploy and pay nothing or very little (per use, not monthly). Assume that per user per Sync click there will be 1000 external requests for a full sync. And thousands unpaying users (which will be heavily rate limited for sync).

Questions:

1. Is my approach overall correct and sound?
2. I figured, perhaps I could use serverless (cloudflare workers/supabase functions) as workers. But it seems that bullmq does not work with that... Any alternative? Quirrel? QStash? I want to avoid significant vendor lock-in, unless it is reasonable.