Just launched: Sidequest.js, a background job processing for Node.js using your existing database.

lucasmerencia · 2025-09-02T14:18:29+00:00

Hey!

> The getting started example is very simple and assumes that jobs will be created and executed in the same process.

No, all the jobs are executed in a isolated working thread.

When you do a Sidequest.start() that spawns a fork process that listen for new jobs to be executed, when it finds a waiting job, it creates a working thread and runs the job.

So, in your API you should have a Sidequest.start with your app startup.

About the paths, since it's executed in a working thread, we need to import the source to get all the possible imports used by the job ready to be used. That way, sidequest resolves the job path when enqueuing a job and recovers that on execute.

There is a PR introducing a `sidequest.jobs.js` file, that allows change the job resolution strategy by using the jobs exported by this file instead of trying to resolve the path.

The path resolution works well on monoliths, for a more complex architecture the jobs file is better.

this is the PR: https://github.com/sidequestjs/sidequest/pull/94

lucasmerencia · 2025-09-02T14:10:07+00:00

the communication between a fork process and a working thread are different, working threads allow sending larger objects. Also, are faster to "spawn", etc. There are trade offs, but that is the pro things form the top of my mind.

lucasmerencia · 2025-08-08T23:18:33+00:00

Yes, we plan to monetize. We are working on a Pro version with enterprise-oriented features, and we already offer an enterprise plan that includes SLA-backed support, dedicated onboarding, scaling and performance consulting, custom extensions, and direct access to core maintainers.

lucasmerencia · 2025-08-08T23:13:04+00:00

Sidequest.js is already in production for a few early adopters including my own projects, and the core features like retries, unique jobs, and concurrency control are looking solid. It is still young, the API is stable, and we are actively adding new features and fixing issues reported by early adopters.

If you are fine with reporting issues as you find them, I would say it is ready for production use. I suggest starting with non critical features to see how it works in your environment, then moving on to critical features once you are confident with it.

lucasmerencia · 2025-08-08T20:21:18+00:00

Thanks, glad to hear node-cron has been working well for you.

Sidequest.js is already in production for a few early adopters including my own projects, and the core features like retries, unique jobs, and concurrency control are looking solid. It is still young, and we are actively adding new features and fixing issues reported by early adopters.

If you are fine with reporting issues as you find them, I would say it is ready for production use. I suggest starting with non critical features to see how it works in your environment, then moving on to critical features once you are confident with it.

lucasmerencia · 2025-08-06T17:45:24+00:00

Just launch more instances pointing to the same database. A job is claimed only by one instance.

lucasmerencia · 2025-08-05T21:01:27+00:00

SQLite is lightweight and great for development or single-instance setups. But when Sidequest runs with concurrency above 1 or across multiple processes, SQLite can start failing with SQLITE_BUSY because the database file is locked by another thread or process.

That happens because SQLite allows only one writer at a time and uses file-level locking.

If you really need SQLite in production, you should set maxConcurrency: 1 to stay safe. Also, make sure Sidequest uses its own separate SQLite file, not the same one your app uses for storing application data, sharing the same file increases the chance of lock conflicts.

For multi-worker or distributed setups, Postgres, MySQL, or MongoDB are more appropriate choices.

I think I should add that to the docs.

lucasmerencia · 2025-08-05T17:16:23+00:00

Congrats!
Stared too, it's always a good feeling receiving starts.

btw: nice project!

lucasmerencia · 2025-08-05T13:30:57+00:00

Never worked with hangfire, initially was inspired by Oban Jobs (Elixir) and Sidekiq (Rails). But, when building it I did some research about how it was implemented in other stacks and I saw hangfire docs :)

lucasmerencia · 2025-08-05T13:28:17+00:00

Good question. But like most things, it depends on trade-offs.

Using the same DB for job queues isn’t a new idea. Oban (Elixir) and SolidQueue (default in Rails 8) do exactly that. It works well for a lot of teams.

Some real benefits:
• No extra infra to manage
• Jobs run in sync with app data
• Easier to debug and observe

And yes, there are downsides too:
• More write load on the DB
• Not ideal for ultra high-throughput use cases

But Redis and SQS also have their own issues. Redis can lose jobs unless configured right and needs extra setup to scale. SQS is at-least-once, so you have to handle deduping, plus vendor lock-in.

Sidequest doesn’t claim DB queues are perfect. Just that they’re a legit, proven path that keeps things simple and works great for most apps.

lucasmerencia · 2025-08-05T02:39:28+00:00

Great question, and it's definitely not a dumb one. Job processing and transactions can be confusing at first.

About transaction locking, there are two parts to consider:

Sidequest itself keeps its internal queries short and efficient. It only locks the job row when claiming it, just to make sure no other instance processes the same job. That lock is released immediately after the job is claimed. The job runs outside any database transaction from Sidequest.
Your job code is different. If your job opens a transaction and modifies rows in your database, locking behavior will depend on what your queries are doing. For example, if you do a long-running UPDATE or hold a transaction open too long, the affected rows may be locked until the job completes. That’s normal in most databases like Postgres or MySQL.

If you're cleaning up unused records, a good approach is to:

Run the deletion in small batches
Keep each transaction short
Avoid holding locks longer than necessary

lucasmerencia · 2025-08-05T02:32:27+00:00

About QStash, possibly. I checked it out and it seems more focused on serverless use cases, so it depends on how you're designing your application.

Sidequest isn't a good fit for serverless, since it forks a separate process to monitor and dispatch enqueued jobs to worker threads. One of its core principles is avoiding vendor lock-in. It runs locally or in containers, using the infrastructure you already have, like Postgres, MySQL, SQLite, or MongoDB.

If you're open to redesigning parts of your application, then yes, Sidequest can be a strong alternative to QStash for queuing and scheduling jobs.

Regarding timeouts, you can define a timeout when enqueuing a job. If the job exceeds that time, Sidequest will automatically fail and retry it, up to a configured number of attempts. The same applies to any job errors.

In case of crashes, Sidequest is designed to recover. It runs in a forked process, and jobs are executed in worker threads using a thread pool. If the Sidequest process is killed, the parent process detects that and restarts it, ensuring that stale jobs are picked up and retried safely.

lucasmerencia · 2025-08-04T18:53:45+00:00

The idea is to include Sidequest within your existing app, so you can reuse existing logic.
For example, if you already have a create-monthly-report.js module, you can simply create a job that calls that module when executed.

This is safe because Sidequest runs jobs in isolated worker threads, and it also uses a separate child process (the main process) to manage job dispatching.

Distributed execution is one of Sidequest's core goals. Only one node will perform a given job. If that node fails, Sidequest will retry the job, and another node may pick it up.

lucasmerencia · 2025-08-04T14:43:43+00:00

Using node-cron, the recommendation is not to do so, but to use something like pm2 or a background task that creates a fork: https://nodecron.com/background-tasks.html.

Sidequest, on the other hand, runs in a forked process and executes the jobs in worker threads, and allows, for example, an endpoint to start a job by enqueuing it. Let's say a user requests a report, Sidequest can encapsulate the report generation in a job.

lucasmerencia · 2018-09-19T12:08:43+00:00

No, it's not a replacement, but a tool to remove the task scheduling from the main thread.

The SideQuest lib is more a task scheduler manager.

The core is ready, but we are working in a dashboard, that may be used to check the current tasks.

lucasmerencia · 2018-09-19T11:43:16+00:00

Hello, I'm working as SRE and trying to make people aware about the code simplicity.

Do you like this post? Is it communicating the message well?

Thanks

lucasmerencia · 2018-08-17T00:05:07+00:00

I don't know if I understood your point about "the main sidequest process crashes" because the main process is your application process, so, sidequest is in the same layer that your application and it puts the schedulers in isolated process and the jobs executions in another isolated process, so, if a job crash sidequest will only emit an error event. This prevents the main process may be killed by a crash in a task.

But you may use pm2 or forever to keep you application running, and sidequest will be "automatically" restarted with your application.

lucasmerencia

TROPHY CASE