Two jobs finish successfully. One account update disappears.
Suppose an account has 100 credits. You enqueue two jobs to add 10 and 20. Both jobs finish without an exception, but the account ends with 120 credits.
Each job runs this code:
class AddCreditsJob < ApplicationJob
def perform(account_id, amount)
account = Account.find(account_id)
account.update!(credits: account.credits + amount)
end
endAssume credits is an integer column, with no optimistic locking or custom database rule protecting the update. These are application credits; the example doesn't model a payment system.
Two workers can execute the job in this sequence:
Job A (+10) Job B (+20)
read 100
read 100
calculate 110
calculate 120
save 110
save 120
Expected: 130 Stored: 120The second write replaces the first job's result with a total calculated from an older value. The database accepts both updates. Neither job has asked it to check whether the starting value changed.
The writes themselves can run one after another. The overlap that matters begins when each job reads the record and ends when it writes the result.
The reproduction imposes that read/read/write/write sequence using separate Active Record objects. It runs on Rails 8.1.3 and SQLite; it doesn't run concurrent workers. The queue behaviour below is traced through Solid Queue 1.4.0 source.
Subscribe free to the next RailsRevelry article and Which Rails Layer Owns This?, a six-page debugging field guide. The welcome email includes a link to the guide.
Being claimed first doesn't mean finishing first
Putting both jobs in one queue doesn't make their executions sequential. A worker can have several threads, and several worker processes can consume that queue.
Solid Queue's ReadyExecution.claim selects and locks ready executions. The worker claims work according to its available thread capacity, then passes those executions to its pool.1 Each execution can take a different path through application code and database waits.
There are ordering rules here. Within a queue, Solid Queue selects work by priority, then job ID. When a worker polls an explicit list of queues, the queue-list order takes precedence over job priority.2 These rules determine which work gets claimed. They cannot establish the order in which separate executions finish their writes.
Even if job A is claimed first, job B can read the account before A commits. Both then calculate from 100.
Reducing one worker to a single thread can hide that interleaving. It only prevents overlap among executions on that worker. Another worker, a web request, or a maintenance script can still change the same account.
What a concurrency limit coordinates
Solid Queue provides more targeted control. Add this declaration to AddCreditsJob:
limits_concurrency to: 1,
key: ->(account_id, amount) { account_id }With the default blocking behaviour, jobs in the same concurrency group with the same key wait for permission to proceed. Different accounts can proceed independently.
For an immediately due job, Solid Queue acquires a semaphore before making the job ready. If permission is unavailable, it creates a blocked execution. Releasing a blocked execution makes it ready for a worker to claim; it doesn't start perform immediately.3
A permit can therefore be held while a job is still waiting for a worker. It doesn't prove the job has already read the account.
The computed key also includes a group, which defaults to the job class name.4 AddCreditsJob and RemoveCreditsJob do not share a limit merely because their key procs return the same account ID. Coordinating them requires an explicit shared group and consistently computed keys. A request that updates Account doesn't acquire either job's semaphore.
For participating jobs, this control can prevent ordinary overlap and reduce conflicts. I would use it when competing work is expensive. I would still inspect every path that writes the account before treating the record as protected.
The permit can expire while work remains
Concurrency controls also need a recovery path. A worker can disappear without releasing its permit, blocking other jobs.
Solid Queue stores an expiry on the semaphore. In version 1.4.0, duration defaults to three minutes unless configured otherwise.5 Its maintenance task deletes expired semaphores and attempts to release eligible blocked executions. A released job must acquire a permit before becoming ready.6
That recovery doesn't require the original job to have finished. A job that takes too long can still be running when another job gains permission to proceed. The configured duration is not a timeout that terminates the original execution. Maintenance timing and worker availability also mean that another execution need not start at the exact expiry instant.
Solid Queue's concurrency controls do not promise execution order.7 They coordinate admission within a configured scope and duration. They do not lock the account row for as long as any application code might use it.
Put the addition in the database
For this particular job, I would remove the Ruby calculation. The operation is an addition, so the database can apply it to the value stored when the update executes:
affected = Account.increment_counter(
:credits, account_id, by: amount
)
raise ActiveRecord::RecordNotFound unless affected == 1The affected-row check preserves the original job's failure when the account no longer exists. The counter API doesn't load the record first.
Rails builds an SQL increment through update_counters.8 With both jobs using this path, the second update adds to the first's result. It doesn't submit a replacement total calculated from 100. The reproduction produces:
unprotected: both saves succeeded; credits=120; intended=130
database increments: credits=130This changes the persistence path. Counter updates bypass model validations and callbacks; they do not automatically update timestamps unless requested. That is the same trade-off discussed in the persistence APIs that skip the record lifecycle.
An atomic increment solves this addition race when the competing updates use compatible operations. A different writer can still replace the total with a stale value. And executing the same increase twice still adds it twice. Duplicate execution remains a separate problem.
When the calculation depends on what was read
An operation may need to inspect the account before deciding what to write. Moving arithmetic into SQL may not express the whole rule.
A transaction alone doesn't necessarily close the gap. At PostgreSQL's default Read Committed isolation, both jobs can read 100 inside their own transactions, then submit replacement totals calculated from it. A locking read can protect the calculation:
account = Account.find(account_id)
account.with_lock do
account.update!(credits: account.credits + amount)
endRails' with_lock wraps the block in a transaction and reloads the record with a lock before yielding. On PostgreSQL, the default locking clause is FOR UPDATE. If A holds that lock, B waits at its locking read. After A commits 110, B's locking read returns 110, so B calculates from the updated value.9
The lock lasts until the transaction ends, which can be later than the block if it joins an outer transaction. Keep the protected work short and external calls out of it. An unprotected writer can still read a stale value, wait to update, then overwrite the result; all read-calculate-write paths need compatible protection. Higher isolation levels can reject conflicting transactions instead of following this wait-and-reload sequence. This explanation is backed by source code and documentation; the SQLite reproduction doesn't test row-lock waiting.
Optimistic locking instead checks the read assumption when saving. With an integer lock_version column, non-null and defaulting to zero, ordinary Active Record saves include the previously read version in their update condition. Rails advances the version on success and raises ActiveRecord::StaleObjectError if the update doesn't affect exactly one row.10
In the versioned reproduction, both objects read version 0. The first saves 110 and advances the version to 1. The second tries to save using version 0:
versioned: second save rejected; credits=110; version=1The second increase is still outstanding. The exception detected a conflict; it did not complete the operation.
For this database-only addition, explicitly reloading and recalculating applies the remaining increase and produces 130. For a workflow decision, the application must reconsider whether the transition is still allowed. Retrying an entire job also repeats any earlier work in that job, including external effects.
Version checking depends on writers participating in the protocol. A raw SQL update that leaves lock_version unchanged can undermine it. A queue setting cannot make that writer participate.
There is a further limit worth keeping separate. An old queued command can load a fresh record and save with the current lock_version. Optimistic locking then has no stale object to reject. If an earlier workflow command must never run after a later one, the application needs an expected state or business version that identifies that ordering requirement.
Reconstruct the reads and writes
When both jobs appear successful, start with one affected record and build a timeline for the attempts that touched it.
Use perform_start.active_job and perform.active_job notifications to locate the execution intervals.11 Correlate the Active Job ID with the backend job ID and worker process/thread. Then capture the application evidence that those events cannot supply:
The account ID, value or version read, and attempted change.
Whether the transaction committed, rolled back, or rejected a stale write.
The full concurrency key, including its group, and the queues and workers involved.
A SQL update notification alone doesn't prove a later transaction committed. On different machines, nearby timestamps do not prove causal order either. Record versions and committed outcomes can establish what each write was allowed to replace.
If the jobs were meant to share a concurrency limit, compare their stored keys. Check for a different job class, another writer, or expiry before assuming the control was ignored.
For the lost increase above, the decisive evidence is that both calculations used 100 and both writes were accepted. Before reducing worker counts, inspect the update that let the second calculation overwrite the first result. That is where this defect can be prevented across the writers that share the account.
If you have traced a similar conflict, reply with the rule that needed protecting and the write path that enforced it.
Next Sunday: Reconstructing a Background Job in Production - connecting enqueuing, worker attempts, and the business outcome.
Solid Queue 1.4.0: ReadyExecution.claim and Worker#poll.
Solid Queue 1.4.0: Execution.ordered, and queue order and priorities.
Solid Queue 1.4.0: Job::Executable#dispatch, Job::ConcurrencyControls, and BlockedExecution#release.
Solid Queue 1.4.0: ActiveJob::ConcurrencyControls.
Solid Queue 1.4.0: default_concurrency_control_period defaults to 3.minutes; limits_concurrency uses that setting when duration is omitted.
Solid Queue 1.4.0: Semaphore and Dispatcher::ConcurrencyMaintenance. BlockedExecution.unblock selects candidate keys from blocked executions whose own expires_at has elapsed, then filters for missing or available semaphores. It selects the next blocked job for each eligible key by priority and job ID; that selected job need not itself be expired.
Solid Queue 1.4.0: concurrency-control guarantees and duration.
Rails 8.1.3: CounterCache.increment_counter and Relation#update_counters. The latter uses update_all, bypassing model validations and callbacks.
Rails 8.1.3: with_lock and lock!. PostgreSQL 18: Read Committed isolation and row-level locking. The latter documents waiting, transaction-end release, and errors on changed rows under Repeatable Read or Serializable. This example assumes the row remains present and both transactions complete without a lock timeout or other error.
Rails 8.1.3: ActiveRecord::Locking::Optimistic, especially _update_row and _query_constraints_hash. With optimistic locking enabled, the earlier increment_counter path also advances lock_version through the update_counters override. It doesn't check a previously read version, but advancing the version lets a later save from an already-loaded object detect the change.
Rails 8.1.3: ActiveJob::Instrumentation.
