Module: Cybertrain::Job
- Defined in:
- cybertrain/job.rb,
cybertrain/job/cli.rb,
cybertrain/job/base.rb,
cybertrain/job/store.rb,
cybertrain/job/worker.rb,
cybertrain/job/executor.rb,
cybertrain/job/retry_policy.rb
Overview
Cybertrain::Job – a small database-backed job queue: “do this later,
and retry it if it fails”. Rows live in the application’s own SQLite
database, in the cybertrain_jobs table, which the framework creates
itself (CREATE TABLE IF NOT EXISTS, on the first enqueue or when the
worker starts) and leaves out of db/schema.rb.
An application writes job classes in app/jobs (Base: perform,
retry_on, discard_on, limits_concurrency); spin run gen writes
their perform_later and the Job.register call into gen/jobs.rb. A
block registered under a String kind is the short form:
# bin/
# in an action, after saving the user: Cybertrain::Job.enqueue(“welcome_mail”, JSON.generate({ “email” => user.email })) WelcomeJob.perform_later(“user_id” => user.id) # a job class
A row is queued, running, done, failed or discarded. A kind
registered with a concurrency_key: stores Job.concurrency_key_for
in the row’s concurrency_key column, and its limit counts per key.
The framework registers one kind itself, “cybertrain mail”
(Mailer.deliver_later, cybertrain/mailer/later.rb), so every
application that requires “cybertrain” has a handler.
A process that runs jobs (./<app> server unless Config#worker is
off, and ./<app> worker) starts Config#job_threads job threads
(Job.start!). Each claims the oldest due row, runs its handler and
marks it done, then looks again, waiting Config#job_poll_interval
seconds when nothing is due; a handler that raises is retried after 1,
5, 15 and then 60 minutes, until Config#job_max_attempts attempts have
failed, when the row is left failed with the error in last_error.
A heartbeat thread marks the process’s running rows alive every
Config#job_heartbeat_interval seconds, and Job.recover! queues again
the rows of a process that stopped beating (Config#job_heartbeat_stale),
so several worker processes can share one database. A block given to
Job.register may write its own outcome instead (Job.retry_later,
or Job.discard, which leaves the row discarded).
This module is the queue’s public surface and its registry of handlers
(Job.handle, Job.register, HANDLERS). The work behind the other
methods is in four parts they delegate to (docs/design.md D68):
Job::Store holds every SQL statement, Job::RetryPolicy the arithmetic
of waits and limits, Job::Executor one claimed attempt from handler to
outcome, and Job::Worker the threads.
Defined Under Namespace
Constant Summary collapse
- OUTCOME_REQUEUED =
What retry_later and discard answer: the row was queued again.
:requeued- OUTCOME_FAILED =
The row became
failed(or, for discard,discarded). :failed- OUTCOME_STALE =
Nothing was written: another claim owns the row, or the write failed every try.
:stale
Class Method Summary collapse
-
.backfill_keys(now) ⇒ Integer
Keys the
queuedrows of each keyed kind that still carry “” (rows enqueued before the kind had a key, or before the column existed), with Job.concurrency_key_for, a batch of 1000 rows at a time, each UPDATE only while the row is still unkeyed. -
.concurrency_key_for(kind, payload) ⇒ String
The concurrency key a job of
kindwithpayloadis counted under: the payload parsed as a JSON object, the field the kind’sconcurrency_key:names (Job.register), as a String (trueis “true”,7is “7”). -
.discard(run, error) ⇒ Symbol
Writes the outcome of an attempt a discard policy stopped (the code generated for
discard_on): the row becomesdiscarded, with the error’s class and message inlast_error, and is never retried. -
.enqueue(kind, payload, run_at: Time.current) ⇒ Integer
Adds a job: a
queuedrow that the worker runs at or afterrun_at. -
.handle(kind, concurrency: 0) {|payload| ... } ⇒ nil
Registers the code that runs jobs of
kind. -
.handlers? ⇒ Boolean
True once some handler is registered (Job.start! starts the job threads only then).
-
.heartbeat!(now) ⇒ Integer
Marks the jobs this process is running now as alive at
now(the heartbeat thread does it every Config#job_heartbeat_interval seconds), so no other process’s Job.recover! takes them. -
.install! ⇒ nil
Creates the table and its indexes if they are missing, and adds the columns a table made by an earlier release lacks (
claimed_by,heartbeat_at; a row such a release leftrunningkeeps a NULL heartbeat, which Job.recover! requeues at once;concurrency_key, “” until Job.backfill_keys or Job.recover! keys the row). -
.process_label ⇒ String
Who this process is in
claimed_by: “: : :<6 random hex digits>", computed once, so neither two processes on one host nor a container restarted within the same second with the same pid share a label. -
.prune_done(before) ⇒ Integer
Deletes
donerows last updated beforebefore(epoch seconds), for a scheduled task that keeps the table small. -
.prune_finished(before) ⇒ Integer
Deletes
failedanddiscardedrows whose outcome was written beforebefore(epoch seconds), once a person no longer needs them. -
.recover!(now, stale = Job.heartbeat_stale) ⇒ Integer
Puts
runningrows whose process stopped saying they run back in the queue: those whoseheartbeat_atis older thannow - stale(the process died mid-job, or was stopped before the job finished), and those with no heartbeat at all (leftrunningby a release before heartbeats, when install! added the column: every claim since writes one, so NULL is never a live job). -
.recovery_limit(kind) ⇒ Integer
The attempt count at which stale-heartbeat recovery (Job.recover!) fails a row of
kindinstead of queueing it again: the larger of Config#job_max_attempts and the kind’smax_attempts:(Job.register), so a policy that allows more attempts is honoured and one that allows fewer does not fail a row the default policy may still retry. -
.register(kind, concurrency: 0, concurrency_key: "", max_attempts: 0) {|run| ... } ⇒ nil
Registers the code that runs jobs of
kind, as Job.handle does, with a block that receives the attempt as a Run: its parsed args, its attempt number, and what Job.retry_later and Job.discard need to write its outcome themselves (the code generated forapp/jobsclasses uses it). -
.request_stop ⇒ nil
Tells the job threads to claim nothing more, and nothing else: safe in a signal handler (Application#serve’s TERM trap calls it next to the server’s request_stop, so the jobs drain during the requests’ drain).
-
.retry_later(run, error, wait_base, polynomial, attempts_limit) ⇒ Symbol
Writes the outcome of a failed attempt for a retry policy (the code generated for
retry_on):queuedagain withrun_atmoved by the wait whilerun.run_attempts < attempts_limit, elsefailed;last_erroris the error’s class and message either way. -
.running? ⇒ Boolean
True between Job.start! and Job.stop! (or Job.request_stop).
-
.running_count(kind) ⇒ Integer
How many jobs of
kindarerunningnow, in every process sharing the database. -
.start!(config = Cybertrain.config) ⇒ Boolean
Starts Config#job_threads job threads and the heartbeat thread, unless no handler is registered or they already run.
-
.state_count(state) ⇒ Integer
How many rows are in
state(“queued”, “running”, “done”, “failed” or “discarded”), e.g. -
.stop! ⇒ nil
Asks the job threads to end and waits, at most Config#job_drain_timeout seconds from when they first saw the stop (a Job.request_stop earlier, during the server’s drain, counts), for those running a job to finish it.
-
.work_due(now, limit = JOBS_PER_POLL) ⇒ Integer
Works through due jobs, at most
limitof them, one after another on the calling thread (a Main.command that drains the queue, or a test). -
.work_one(now) ⇒ Boolean
Claims and runs the oldest due job, if there is one, and records the outcome:
done;queuedagain withrun_atpushed back by BACKOFF after a failure;failedonce Config#job_max_attempts attempts have failed, or at once when no handler is registered for its kind. -
.work_one_after(floor, kind = "") ⇒ Boolean
Claims and runs the queued job with the lowest id above
floorand records the outcome as Job.work_one does: whatperform_enqueued_jobs { }andassert_emails { }call to run the jobs queued during their block (ids above the largest one before it), and the jobs those queue, on the test’s thread, in enqueue order.
Class Method Details
.backfill_keys(now) ⇒ Integer
Keys the queued rows of each keyed kind that still carry “” (rows
enqueued before the kind had a key, or before the column existed),
with concurrency_key_for, a batch of 1000 rows at a
time, each UPDATE only while the row is still unkeyed. A row the
derivation leaves “” (no such field, or not a JSON object) stays in
the “” bucket, which has the kind’s limit of its own, and is listed
once at warn with its id. start! calls it after recover!;
a second call finds nothing to key.
482 483 484 |
# File 'cybertrain/job.rb', line 482 def self.backfill_keys(now) Store.backfill_keys(now) end |
.concurrency_key_for(kind, payload) ⇒ String
The concurrency key a job of kind with payload is counted under:
the payload parsed as a JSON object, the field the kind’s
concurrency_key: names (register), as a String (true is
“true”, 7 is “7”). “” when the kind has no key (or no handler),
when the payload is not a JSON object, and when the field is missing
or null. enqueue, backfill_keys and recover! all
derive the key here, so they cannot disagree.
448 449 450 451 452 453 |
# File 'cybertrain/job.rb', line 448 def self.concurrency_key_for(kind, payload) name = job_key_name(kind) return "" if name.empty? Cast.str(Cast.json_hash(payload)[name]) end |
.discard(run, error) ⇒ Symbol
Writes the outcome of an attempt a discard policy stopped (the code
generated for discard_on): the row becomes discarded, with the
error’s class and message in last_error, and is never retried.
Fenced and marked written as retry_later is. It answers
OUTCOME_FAILED when the row became discarded (it is terminal, as a
failed row is) and OUTCOME_STALE when nothing was written.
728 729 730 |
# File 'cybertrain/job.rb', line 728 def self.discard(run, error) Executor.discard(run, error) end |
.enqueue(kind, payload, run_at: Time.current) ⇒ Integer
Adds a job: a queued row that the worker runs at or after run_at.
Callable from an action, a Main.command block or a handler, in any
process connected to the database, whether or not it runs the worker.
Inside a DB.transaction block the row is part of the transaction, so
a job is only enqueued if the work that asked for it commits.
A kind registered with a concurrency_key: stores the payload’s
concurrency_key_for with the row; its limit then counts per key.
Enqueued inside a request, the row records the request’s id (this
thread’s Logger.current_scope), and the job’s log lines carry it;
enqueued by a job, it records the request id that job carries, so a
chain of jobs keeps the id of the request that started it.
425 426 427 428 429 430 431 432 433 434 435 |
# File 'cybertrain/job.rb', line 425 def self.enqueue(kind, payload, run_at: Time.current) raise ArgumentError, "Job.enqueue: no handler registered for #{kind.inspect}" if handler_index(kind) < 0 key = concurrency_key_for(kind, payload) name = job_key_name(kind) if key.empty? && !name.empty? = "Job.enqueue #{kind.inspect}: the payload has no #{name.inspect} field for limits_concurrency" raise ArgumentError, end Store.insert(kind, payload, run_at.to_i, key, Logger.current_scope.scope_request_id) end |
.handle(kind, concurrency: 0) {|payload| ... } ⇒ nil
Registers the code that runs jobs of kind. The block receives the
payload String given to enqueue (encode structured data as JSON
yourself) and should return normally when the work is done; raising
marks the attempt failed and schedules a retry. Register every handler
in every process (bin/
A handler runs on one of the job threads (start!), outside any
request and outside any transaction. A SQLite call holds the OS
worker, so with SPINEL_WORKERS=1 a long query stalls every request
(batch big work over several jobs).
concurrency: n caps how many jobs of this kind run at once, counted
over every process sharing the database (a paid API that must not be
called twice at a time: concurrency: 1); 0, the default, is no cap.
The count is of running rows, so a job of a process that died keeps
its slot until it is recovered (Config#job_heartbeat_stale plus up
to a minute: 5 to 6 minutes by default).
Delivery is at least once: a job whose process stopped beating (it died, or was stopped mid-job) runs again elsewhere, so write handlers that do no harm when they run twice. It is register with a block that passes Cybertrain::Job::Run#run_payload, and the default max_attempts.
245 246 247 248 249 |
# File 'cybertrain/job.rb', line 245 def self.handle(kind, concurrency: 0, &block) check_registration("Job.handle", kind, concurrency, 0) HANDLERS << Handler.new(kind, proc { |run| block.call(run.run_payload) }, concurrency, "", 0) nil end |
.handlers? ⇒ Boolean
True once some handler is registered (start! starts the job threads only then).
333 334 335 |
# File 'cybertrain/job.rb', line 333 def self.handlers? !HANDLERS.empty? end |
.heartbeat!(now) ⇒ Integer
Marks the jobs this process is running now as alive at now (the
heartbeat thread does it every Config#job_heartbeat_interval
seconds), so no other process’s recover! takes them. Only rows
in flight here: a row whose outcome could not be written (the
database stayed locked) is no longer in flight, so its heartbeat stops
and staleness recovery runs it again, rather than it staying
running (and holding a concurrency slot) for as long as the
process lives. And only the attempts in flight here: a row recovered
and claimed again by this process is kept alive by its new attempt,
not by an older one that is still running (the heartbeat is fenced
by the attempt number, as the outcome is). Judgement call: the spec’s
statement was WHERE state = 'running' AND claimed_by = ?; this adds
the (id, attempts) of each claim in flight.
569 570 571 |
# File 'cybertrain/job.rb', line 569 def self.heartbeat!(now) Store.touch_heartbeats(now, process_label, Executor.in_flight_claims) end |
.install! ⇒ nil
Creates the table and its indexes if they are missing, and adds the
columns a table made by an earlier release lacks (claimed_by,
heartbeat_at; a row such a release left running keeps a NULL
heartbeat, which recover! requeues at once; concurrency_key,
“” until backfill_keys or recover! keys the row). The
job threads and enqueue call it once per connected database;
call it yourself only to create the table ahead of time.
495 496 497 |
# File 'cybertrain/job.rb', line 495 def self.install! Store.install! end |
.process_label ⇒ String
Who this process is in claimed_by: “
308 309 310 311 312 313 314 315 |
# File 'cybertrain/job.rb', line 308 def self.process_label label = @process_label if label.empty? label = "#{Socket.gethostname}:#{Process.pid}:#{Clock.real_now.to_i}:#{SecureRandom.hex(3)}" @process_label = label end label end |
.prune_done(before) ⇒ Integer
Deletes done rows last updated before before (epoch seconds), for
a scheduled task that keeps the table small. failed and discarded
rows are kept for a person to look at (prune_finished removes
them).
675 676 677 |
# File 'cybertrain/job.rb', line 675 def self.prune_done(before) Store.prune_done(before) end |
.prune_finished(before) ⇒ Integer
Deletes failed and discarded rows whose outcome was written
before before (epoch seconds), once a person no longer needs them.
684 685 686 |
# File 'cybertrain/job.rb', line 684 def self.prune_finished(before) Store.prune_finished(before) end |
.recover!(now, stale = Job.heartbeat_stale) ⇒ Integer
Puts running rows whose process stopped saying they run back in the
queue: those whose heartbeat_at is older than now - stale (the
process died mid-job, or was stopped before the job finished), and
those with no heartbeat at all (left running by a release before
heartbeats, when install! added the column: every claim since writes
one, so NULL is never a live job). A row
whose process still sends heartbeats (heartbeat!) is never
touched, so every process that runs jobs may call it; the scheduler
does at start and once a minute. The attempt was already counted
when the row was claimed, so a row whose stopped attempt was its
last (its kind’s recovery_limit) is left failed instead
(Job::Store.fail_exhausted): a payload that kills its process is not
requeued for ever. A row of a kind with a concurrency key is
requeued with its concurrency_key_for written in the same
UPDATE, so no claim sees it queued under the wrong key (a row left
by a release before the key, still running when the new workers
started, is keyed here).
527 528 529 |
# File 'cybertrain/job.rb', line 527 def self.recover!(now, stale = Job.heartbeat_stale) Store.requeue_stale(now, stale) end |
.recovery_limit(kind) ⇒ Integer
The attempt count at which stale-heartbeat recovery (recover!)
fails a row of kind instead of queueing it again: the larger of
Config#job_max_attempts and the kind’s max_attempts:
(register), so a policy that allows more attempts is honoured
and one that allows fewer does not fail a row the default policy may
still retry. A kind no longer registered gets the config value.
540 541 542 543 544 545 546 |
# File 'cybertrain/job.rb', line 540 def self.recovery_limit(kind) default = Cybertrain.config.job_max_attempts index = handler_index(kind) return default if index < 0 RetryPolicy.recovery_limit(HANDLERS[index].handler_max_attempts, default) end |
.register(kind, concurrency: 0, concurrency_key: "", max_attempts: 0) {|run| ... } ⇒ nil
Registers the code that runs jobs of kind, as handle does,
with a block that receives the attempt as a Run: its parsed
args, its attempt number, and what retry_later and
discard need to write its outcome themselves (the code
generated for app/jobs classes uses it). A block that returns
without writing an outcome ends the job done; one that raises
without writing one follows the default retry policy.
272 273 274 275 276 |
# File 'cybertrain/job.rb', line 272 def self.register(kind, concurrency: 0, concurrency_key: "", max_attempts: 0, &block) check_registration("Job.register", kind, concurrency, max_attempts) HANDLERS << Handler.new(kind, block, concurrency, concurrency_key, max_attempts) nil end |
.request_stop ⇒ nil
Tells the job threads to claim nothing more, and nothing else: safe in a signal handler (Application#serve’s TERM trap calls it next to the server’s request_stop, so the jobs drain during the requests’ drain). stop! still has to be called to wait for them.
372 373 374 |
# File 'cybertrain/job.rb', line 372 def self.request_stop Worker.request_stop end |
.retry_later(run, error, wait_base, polynomial, attempts_limit) ⇒ Symbol
Writes the outcome of a failed attempt for a retry policy (the code
generated for retry_on): queued again with run_at moved by the
wait while run.run_attempts < attempts_limit, else failed;
last_error is the error’s class and message either way. The wait
is wait_base seconds, or with polynomial attempts ** 4 + 2
(3, 18, 83, …). The write is fenced by the claim, as the worker’s
own: when another worker reclaimed the row, nothing is written
(logged at warn) and the answer is OUTCOME_STALE, as it is when the
write failed every try (the row is left to heartbeat recovery). The
Run is marked written in every case, so the worker writes nothing
more for it. Answering OUTCOME_FAILED only once the row really
became failed lets the caller run an exhaustion hook safely.
714 715 716 |
# File 'cybertrain/job.rb', line 714 def self.retry_later(run, error, wait_base, polynomial, attempts_limit) Executor.retry_later(run, error, wait_base, polynomial, attempts_limit) end |
.running? ⇒ Boolean
Returns true between start! and stop! (or request_stop).
379 380 381 |
# File 'cybertrain/job.rb', line 379 def self.running? Worker.running? end |
.running_count(kind) ⇒ Integer
How many jobs of kind are running now, in every process sharing
the database.
591 592 593 |
# File 'cybertrain/job.rb', line 591 def self.running_count(kind) Store.running_count(kind) end |
.start!(config = Cybertrain.config) ⇒ Boolean
Starts Config#job_threads job threads and the heartbeat thread,
unless no handler is registered or they already run. It first
creates the table if needed, queues again the rows whose heartbeat
is stale (recover!), and then keys the queued rows enqueued
before their kind had a concurrency key (backfill_keys). The
database must be connected.
Application#serve calls it (when Config#worker is on) after
Scheduler.start!, and so does ./<app> worker; the caller decides
whether this process runs jobs.
349 350 351 |
# File 'cybertrain/job.rb', line 349 def self.start!(config = Cybertrain.config) Worker.start!(config) end |
.state_count(state) ⇒ Integer
How many rows are in state (“queued”, “running”, “done”, “failed”
or “discarded”), e.g. for a health or admin page.
664 665 666 |
# File 'cybertrain/job.rb', line 664 def self.state_count(state) Store.state_count(state) end |
.stop! ⇒ nil
Asks the job threads to end and waits, at most
Config#job_drain_timeout seconds from when they first saw the stop
(a request_stop earlier, during the server’s drain, counts), for
those running a job to finish it. A job still running then keeps its
row running; its heartbeat has stopped, so another process’s
recover! queues it again once it is stale. Does nothing when
the threads do not run.
362 363 364 |
# File 'cybertrain/job.rb', line 362 def self.stop! Worker.stop! end |
.work_due(now, limit = JOBS_PER_POLL) ⇒ Integer
Works through due jobs, at most limit of them, one after another on
the calling thread (a Main.command that drains the queue, or a
test). Returns how many it claimed.
655 656 657 |
# File 'cybertrain/job.rb', line 655 def self.work_due(now, limit = JOBS_PER_POLL) Executor.work_due(now, limit) end |
.work_one(now) ⇒ Boolean
Claims and runs the oldest due job, if there is one, and records the
outcome: done; queued again with run_at pushed back by BACKOFF
after a failure; failed once Config#job_max_attempts attempts have
failed, or at once when no handler is registered for its kind. The
worker calls it; tests call it with a fixed now instead of waiting.
614 615 616 |
# File 'cybertrain/job.rb', line 614 def self.work_one(now) Executor.work_one(now) end |
.work_one_after(floor, kind = "") ⇒ Boolean
Claims and runs the queued job with the lowest id above floor and
records the outcome as work_one does: what perform_enqueued_jobs
{ } and assert_emails { } call to run the jobs queued during their
block (ids above the largest one before it), and the jobs those
queue, on the test’s thread, in enqueue order. A job not yet
attempted runs whatever its run_at; a retry (a failed attempt
rescheduled) only once its run_at has come. The attempt is timed at
the clock’s now (Clock.now: travel_to moves it, and can bring a
retry due).
634 635 636 |
# File 'cybertrain/job.rb', line 634 def self.work_one_after(floor, kind = "") work_one_after_id(floor, kind) >= 0 end |