Module: Cybertrain::Job

Defined in:
cybertrain/job.rb,
cybertrain/job/cli.rb,
cybertrain/job/base.rb,
cybertrain/job/store.rb,
cybertrain/job/worker.rb,
cybertrain/job/executor.rb,
cybertrain/job/retry_policy.rb

Overview

Cybertrain::Job – a small database-backed job queue: “do this later, and retry it if it fails”. Rows live in the application’s own SQLite database, in the cybertrain_jobs table, which the framework creates itself (CREATE TABLE IF NOT EXISTS, on the first enqueue or when the worker starts) and leaves out of db/schema.rb.

An application writes job classes in app/jobs (Base: perform, retry_on, discard_on, limits_concurrency); spin run gen writes their perform_later and the Job.register call into gen/jobs.rb. A block registered under a String kind is the short form:

# bin/.rb (or a file it requires), before Cybertrain::Main.run: Cybertrain::Job.handle("welcome_mail") do |payload| data = JSON.parse(payload) Mailer.deliver_welcome(data["email"].to_s) end

# in an action, after saving the user: Cybertrain::Job.enqueue(“welcome_mail”, JSON.generate({ “email” => user.email })) WelcomeJob.perform_later(“user_id” => user.id) # a job class

A row is queued, running, done, failed or discarded. A kind registered with a concurrency_key: stores Job.concurrency_key_for in the row’s concurrency_key column, and its limit counts per key. The framework registers one kind itself, “cybertrain mail” (Mailer.deliver_later, cybertrain/mailer/later.rb), so every application that requires “cybertrain” has a handler.

A process that runs jobs (./<app> server unless Config#worker is off, and ./<app> worker) starts Config#job_threads job threads (Job.start!). Each claims the oldest due row, runs its handler and marks it done, then looks again, waiting Config#job_poll_interval seconds when nothing is due; a handler that raises is retried after 1, 5, 15 and then 60 minutes, until Config#job_max_attempts attempts have failed, when the row is left failed with the error in last_error. A heartbeat thread marks the process’s running rows alive every Config#job_heartbeat_interval seconds, and Job.recover! queues again the rows of a process that stopped beating (Config#job_heartbeat_stale), so several worker processes can share one database. A block given to Job.register may write its own outcome instead (Job.retry_later, or Job.discard, which leaves the row discarded).

This module is the queue’s public surface and its registry of handlers (Job.handle, Job.register, HANDLERS). The work behind the other methods is in four parts they delegate to (docs/design.md D68): Job::Store holds every SQL statement, Job::RetryPolicy the arithmetic of waits and limits, Job::Executor one claimed attempt from handler to outcome, and Job::Worker the threads.

Defined Under Namespace

Classes: Base, Run

Constant Summary collapse

OUTCOME_REQUEUED =

What retry_later and discard answer: the row was queued again.

:requeued
OUTCOME_FAILED =

The row became failed (or, for discard, discarded).

:failed
OUTCOME_STALE =

Nothing was written: another claim owns the row, or the write failed every try.

:stale

Class Method Summary collapse

Class Method Details

.backfill_keys(now) ⇒ Integer

Keys the queued rows of each keyed kind that still carry “” (rows enqueued before the kind had a key, or before the column existed), with concurrency_key_for, a batch of 1000 rows at a time, each UPDATE only while the row is still unkeyed. A row the derivation leaves “” (no such field, or not a JSON object) stays in the “” bucket, which has the kind’s limit of its own, and is listed once at warn with its id. start! calls it after recover!; a second call finds nothing to key.

Parameters:

  • now (Integer) —

    epoch seconds (the start’s clock; the rows’ updated_at is left as it is, as the spec’s UPDATE writes only the key)

Returns:

  • (Integer) —

    how many rows were keyed



482
483
484
# File 'cybertrain/job.rb', line 482

def self.backfill_keys(now)
  Store.backfill_keys(now)
end

.concurrency_key_for(kind, payload) ⇒ String

The concurrency key a job of kind with payload is counted under: the payload parsed as a JSON object, the field the kind’s concurrency_key: names (register), as a String (true is “true”, 7 is “7”). “” when the kind has no key (or no handler), when the payload is not a JSON object, and when the field is missing or null. enqueue, backfill_keys and recover! all derive the key here, so they cannot disagree.

Parameters:

  • kind (String)
  • payload (String)

Returns:

  • (String)


448
449
450
451
452
453
# File 'cybertrain/job.rb', line 448

def self.concurrency_key_for(kind, payload)
  name = job_key_name(kind)
  return "" if name.empty?

  Cast.str(Cast.json_hash(payload)[name])
end

.discard(run, error) ⇒ Symbol

Writes the outcome of an attempt a discard policy stopped (the code generated for discard_on): the row becomes discarded, with the error’s class and message in last_error, and is never retried. Fenced and marked written as retry_later is. It answers OUTCOME_FAILED when the row became discarded (it is terminal, as a failed row is) and OUTCOME_STALE when nothing was written.

Parameters:

  • run (Job::Run) —

    the attempt

  • error (StandardError) —

    what it raised

Returns:

  • (Symbol) —

    OUTCOME_FAILED or OUTCOME_STALE



728
729
730
# File 'cybertrain/job.rb', line 728

def self.discard(run, error)
  Executor.discard(run, error)
end

.enqueue(kind, payload, run_at: Time.current) ⇒ Integer

Adds a job: a queued row that the worker runs at or after run_at. Callable from an action, a Main.command block or a handler, in any process connected to the database, whether or not it runs the worker. Inside a DB.transaction block the row is part of the transaction, so a job is only enqueued if the work that asked for it commits. A kind registered with a concurrency_key: stores the payload’s concurrency_key_for with the row; its limit then counts per key. Enqueued inside a request, the row records the request’s id (this thread’s Logger.current_scope), and the job’s log lines carry it; enqueued by a job, it records the request id that job carries, so a chain of jobs keeps the id of the request that started it.

Examples:

run in ten minutes

Cybertrain::Job.enqueue("reminder", order.id.to_s, run_at: Time.current + 600)

Parameters:

  • kind (String) —

    a kind registered with handle

  • payload (String) —

    handed to the handler as it is

  • run_at (Time) (defaults to: Time.current) —

    the earliest time to run it (now by default)

Returns:

Raises:

  • (ArgumentError) —

    when no handler is registered for kind, or the kind has a concurrency key and the payload cannot be keyed: the field is missing, its value is an empty String or null, or the payload is not a JSON object

  • (Cybertrain::DB::Error) —

    when the database is not connected or the insert fails



425
426
427
428
429
430
431
432
433
434
435
# File 'cybertrain/job.rb', line 425

def self.enqueue(kind, payload, run_at: Time.current)
  raise ArgumentError, "Job.enqueue: no handler registered for #{kind.inspect}" if handler_index(kind) < 0

  key = concurrency_key_for(kind, payload)
  name = job_key_name(kind)
  if key.empty? && !name.empty?
    message = "Job.enqueue #{kind.inspect}: the payload has no #{name.inspect} field for limits_concurrency"
    raise ArgumentError, message
  end
  Store.insert(kind, payload, run_at.to_i, key, Logger.current_scope.scope_request_id)
end

.handle(kind, concurrency: 0) {|payload| ... } ⇒ nil

Registers the code that runs jobs of kind. The block receives the payload String given to enqueue (encode structured data as JSON yourself) and should return normally when the work is done; raising marks the attempt failed and schedules a retry. Register every handler in every process (bin/.rb runs before Main.run in all of them): enqueue refuses a kind with no handler.

A handler runs on one of the job threads (start!), outside any request and outside any transaction. A SQLite call holds the OS worker, so with SPINEL_WORKERS=1 a long query stalls every request (batch big work over several jobs).

concurrency: n caps how many jobs of this kind run at once, counted over every process sharing the database (a paid API that must not be called twice at a time: concurrency: 1); 0, the default, is no cap. The count is of running rows, so a job of a process that died keeps its slot until it is recovered (Config#job_heartbeat_stale plus up to a minute: 5 to 6 minutes by default).

Delivery is at least once: a job whose process stopped beating (it died, or was stopped mid-job) runs again elsewhere, so write handlers that do no harm when they run twice. It is register with a block that passes Cybertrain::Job::Run#run_payload, and the default max_attempts.

Examples:

Cybertrain::Job.handle("notify") { |payload| Notifier.post(payload) }
Cybertrain::Job.handle("quote", concurrency: 1) { |payload| PaidApi.fetch(payload) }

Parameters:

  • kind (String) —

    the name jobs are enqueued under

  • concurrency (Integer) (defaults to: 0) —

    the most running at once; 0 for no limit

Yield Parameters:

  • payload (String)

Returns:

  • (nil)

Raises:

  • (ArgumentError) —

    for an empty kind, one already registered, or a negative concurrency



245
246
247
248
249
# File 'cybertrain/job.rb', line 245

def self.handle(kind, concurrency: 0, &block)
  check_registration("Job.handle", kind, concurrency, 0)
  HANDLERS << Handler.new(kind, proc { |run| block.call(run.run_payload) }, concurrency, "", 0)
  nil
end

.handlers? ⇒ Boolean

True once some handler is registered (start! starts the job threads only then).

Returns:

  • (Boolean)


333
334
335
# File 'cybertrain/job.rb', line 333

def self.handlers?
  !HANDLERS.empty?
end

.heartbeat!(now) ⇒ Integer

Marks the jobs this process is running now as alive at now (the heartbeat thread does it every Config#job_heartbeat_interval seconds), so no other process’s recover! takes them. Only rows in flight here: a row whose outcome could not be written (the database stayed locked) is no longer in flight, so its heartbeat stops and staleness recovery runs it again, rather than it staying running (and holding a concurrency slot) for as long as the process lives. And only the attempts in flight here: a row recovered and claimed again by this process is kept alive by its new attempt, not by an older one that is still running (the heartbeat is fenced by the attempt number, as the outcome is). Judgement call: the spec’s statement was WHERE state = 'running' AND claimed_by = ?; this adds the (id, attempts) of each claim in flight.

Parameters:

Returns:

  • (Integer) —

    how many rows it touched



569
570
571
# File 'cybertrain/job.rb', line 569

def self.heartbeat!(now)
  Store.touch_heartbeats(now, process_label, Executor.in_flight_claims)
end

.install! ⇒ nil

Creates the table and its indexes if they are missing, and adds the columns a table made by an earlier release lacks (claimed_by, heartbeat_at; a row such a release left running keeps a NULL heartbeat, which recover! requeues at once; concurrency_key, “” until backfill_keys or recover! keys the row). The job threads and enqueue call it once per connected database; call it yourself only to create the table ahead of time.

Returns:

  • (nil)


495
496
497
# File 'cybertrain/job.rb', line 495

def self.install!
  Store.install!
end

.process_label ⇒ String

Who this process is in claimed_by: “:::<6 random hex digits>", computed once, so neither two processes on one host nor a container restarted within the same second with the same pid share a label. The epoch is real time (Clock.real_now): an identity, not an instant a test travels.

Returns:

  • (String)


308
309
310
311
312
313
314
315
# File 'cybertrain/job.rb', line 308

def self.process_label
  label = @process_label
  if label.empty?
    label = "#{Socket.gethostname}:#{Process.pid}:#{Clock.real_now.to_i}:#{SecureRandom.hex(3)}"
    @process_label = label
  end
  label
end

.prune_done(before) ⇒ Integer

Deletes done rows last updated before before (epoch seconds), for a scheduled task that keeps the table small. failed and discarded rows are kept for a person to look at (prune_finished removes them).

Parameters:

  • before (Integer) —

    epoch seconds

Returns:

  • (Integer) —

    how many rows were deleted



675
676
677
# File 'cybertrain/job.rb', line 675

def self.prune_done(before)
  Store.prune_done(before)
end

.prune_finished(before) ⇒ Integer

Deletes failed and discarded rows whose outcome was written before before (epoch seconds), once a person no longer needs them.

Parameters:

  • before (Integer) —

    epoch seconds

Returns:

  • (Integer) —

    how many rows were deleted



684
685
686
# File 'cybertrain/job.rb', line 684

def self.prune_finished(before)
  Store.prune_finished(before)
end

.recover!(now, stale = Job.heartbeat_stale) ⇒ Integer

Puts running rows whose process stopped saying they run back in the queue: those whose heartbeat_at is older than now - stale (the process died mid-job, or was stopped before the job finished), and those with no heartbeat at all (left running by a release before heartbeats, when install! added the column: every claim since writes one, so NULL is never a live job). A row whose process still sends heartbeats (heartbeat!) is never touched, so every process that runs jobs may call it; the scheduler does at start and once a minute. The attempt was already counted when the row was claimed, so a row whose stopped attempt was its last (its kind’s recovery_limit) is left failed instead (Job::Store.fail_exhausted): a payload that kills its process is not requeued for ever. A row of a kind with a concurrency key is requeued with its concurrency_key_for written in the same UPDATE, so no claim sees it queued under the wrong key (a row left by a release before the key, still running when the new workers started, is keyed here).

Parameters:

  • now (Integer) —

    epoch seconds

  • stale (Integer) (defaults to: Job.heartbeat_stale) —

    seconds without a heartbeat that make a row abandoned (by default the Config#job_heartbeat_stale of the Config start! was given, or of Cybertrain.config before it)

Returns:

  • (Integer) —

    how many rows were requeued



527
528
529
# File 'cybertrain/job.rb', line 527

def self.recover!(now, stale = Job.heartbeat_stale)
  Store.requeue_stale(now, stale)
end

.recovery_limit(kind) ⇒ Integer

The attempt count at which stale-heartbeat recovery (recover!) fails a row of kind instead of queueing it again: the larger of Config#job_max_attempts and the kind’s max_attempts: (register), so a policy that allows more attempts is honoured and one that allows fewer does not fail a row the default policy may still retry. A kind no longer registered gets the config value.

Parameters:

  • kind (String)

Returns:



540
541
542
543
544
545
546
# File 'cybertrain/job.rb', line 540

def self.recovery_limit(kind)
  default = Cybertrain.config.job_max_attempts
  index = handler_index(kind)
  return default if index < 0

  RetryPolicy.recovery_limit(HANDLERS[index].handler_max_attempts, default)
end

.register(kind, concurrency: 0, concurrency_key: "", max_attempts: 0) {|run| ... } ⇒ nil

Registers the code that runs jobs of kind, as handle does, with a block that receives the attempt as a Run: its parsed args, its attempt number, and what retry_later and discard need to write its outcome themselves (the code generated for app/jobs classes uses it). A block that returns without writing an outcome ends the job done; one that raises without writing one follows the default retry policy.

Examples:

Cybertrain::Job.register("ReportJob", max_attempts: 10) { |run| ReportJob.run_job(run) }

Parameters:

  • kind (String) —

    the name jobs are enqueued under

  • concurrency (Integer) (defaults to: 0) —

    the most running at once; 0 for no limit

  • concurrency_key (String) (defaults to: "") —

    the args key the limit is counted per; “” for the whole kind

  • max_attempts (Integer) (defaults to: 0) —

    the most attempts any of the kind’s policies allows; 0 for Config#job_max_attempts. Stale-heartbeat recovery fails a row only once it reached recovery_limit.

Yield Parameters:

Returns:

  • (nil)

Raises:

  • (ArgumentError) —

    for an empty kind, one already registered, or a negative concurrency or max_attempts



272
273
274
275
276
# File 'cybertrain/job.rb', line 272

def self.register(kind, concurrency: 0, concurrency_key: "", max_attempts: 0, &block)
  check_registration("Job.register", kind, concurrency, max_attempts)
  HANDLERS << Handler.new(kind, block, concurrency, concurrency_key, max_attempts)
  nil
end

.request_stop ⇒ nil

Tells the job threads to claim nothing more, and nothing else: safe in a signal handler (Application#serve’s TERM trap calls it next to the server’s request_stop, so the jobs drain during the requests’ drain). stop! still has to be called to wait for them.

Returns:

  • (nil)


372
373
374
# File 'cybertrain/job.rb', line 372

def self.request_stop
  Worker.request_stop
end

.retry_later(run, error, wait_base, polynomial, attempts_limit) ⇒ Symbol

Writes the outcome of a failed attempt for a retry policy (the code generated for retry_on): queued again with run_at moved by the wait while run.run_attempts < attempts_limit, else failed; last_error is the error’s class and message either way. The wait is wait_base seconds, or with polynomial attempts ** 4 + 2 (3, 18, 83, …). The write is fenced by the claim, as the worker’s own: when another worker reclaimed the row, nothing is written (logged at warn) and the answer is OUTCOME_STALE, as it is when the write failed every try (the row is left to heartbeat recovery). The Run is marked written in every case, so the worker writes nothing more for it. Answering OUTCOME_FAILED only once the row really became failed lets the caller run an exhaustion hook safely.

Parameters:

  • run (Job::Run) —

    the attempt

  • error (StandardError) —

    what it raised

  • wait_base (Integer) —

    seconds before the retry

  • polynomial (Boolean) —

    wait attempts ** 4 + 2 seconds instead

  • attempts_limit (Integer) —

    the most attempts, the first included

Returns:

  • (Symbol) —

    OUTCOME_REQUEUED, OUTCOME_FAILED or OUTCOME_STALE



714
715
716
# File 'cybertrain/job.rb', line 714

def self.retry_later(run, error, wait_base, polynomial, attempts_limit)
  Executor.retry_later(run, error, wait_base, polynomial, attempts_limit)
end

.running? ⇒ Boolean

Returns true between start! and stop! (or request_stop).

Returns:



379
380
381
# File 'cybertrain/job.rb', line 379

def self.running?
  Worker.running?
end

.running_count(kind) ⇒ Integer

How many jobs of kind are running now, in every process sharing the database.

Parameters:

  • kind (String)

Returns:



591
592
593
# File 'cybertrain/job.rb', line 591

def self.running_count(kind)
  Store.running_count(kind)
end

.start!(config = Cybertrain.config) ⇒ Boolean

Starts Config#job_threads job threads and the heartbeat thread, unless no handler is registered or they already run. It first creates the table if needed, queues again the rows whose heartbeat is stale (recover!), and then keys the queued rows enqueued before their kind had a concurrency key (backfill_keys). The database must be connected. Application#serve calls it (when Config#worker is on) after Scheduler.start!, and so does ./<app> worker; the caller decides whether this process runs jobs.

Parameters:

  • config (Config) (defaults to: Cybertrain.config)

Returns:

  • (Boolean) —

    true when it started the threads



349
350
351
# File 'cybertrain/job.rb', line 349

def self.start!(config = Cybertrain.config)
  Worker.start!(config)
end

.state_count(state) ⇒ Integer

How many rows are in state (“queued”, “running”, “done”, “failed” or “discarded”), e.g. for a health or admin page.

Parameters:

  • state (String)

Returns:



664
665
666
# File 'cybertrain/job.rb', line 664

def self.state_count(state)
  Store.state_count(state)
end

.stop! ⇒ nil

Asks the job threads to end and waits, at most Config#job_drain_timeout seconds from when they first saw the stop (a request_stop earlier, during the server’s drain, counts), for those running a job to finish it. A job still running then keeps its row running; its heartbeat has stopped, so another process’s recover! queues it again once it is stale. Does nothing when the threads do not run.

Returns:

  • (nil)


362
363
364
# File 'cybertrain/job.rb', line 362

def self.stop!
  Worker.stop!
end

.work_due(now, limit = JOBS_PER_POLL) ⇒ Integer

Works through due jobs, at most limit of them, one after another on the calling thread (a Main.command that drains the queue, or a test). Returns how many it claimed.

Parameters:

  • now (Integer) —

    epoch seconds

  • limit (Integer) (defaults to: JOBS_PER_POLL)

Returns:



655
656
657
# File 'cybertrain/job.rb', line 655

def self.work_due(now, limit = JOBS_PER_POLL)
  Executor.work_due(now, limit)
end

.work_one(now) ⇒ Boolean

Claims and runs the oldest due job, if there is one, and records the outcome: done; queued again with run_at pushed back by BACKOFF after a failure; failed once Config#job_max_attempts attempts have failed, or at once when no handler is registered for its kind. The worker calls it; tests call it with a fixed now instead of waiting.

Parameters:

  • now (Integer) —

    epoch seconds: jobs with run_at <= now are due

Returns:

  • (Boolean) —

    true when a job was claimed



614
615
616
# File 'cybertrain/job.rb', line 614

def self.work_one(now)
  Executor.work_one(now)
end

.work_one_after(floor, kind = "") ⇒ Boolean

Claims and runs the queued job with the lowest id above floor and records the outcome as work_one does: what perform_enqueued_jobs { } and assert_emails { } call to run the jobs queued during their block (ids above the largest one before it), and the jobs those queue, on the test’s thread, in enqueue order. A job not yet attempted runs whatever its run_at; a retry (a failed attempt rescheduled) only once its run_at has come. The attempt is timed at the clock’s now (Clock.now: travel_to moves it, and can bring a retry due).

Examples:

floor = id_before_the_block
nil while Cybertrain::Job.work_one_after(floor)

Parameters:

  • floor (Integer) —

    only rows whose id is greater

  • kind (String) (defaults to: "") —

    only rows of this kind; “” for any

Returns:

  • (Boolean) —

    true when a job was claimed



634
635
636
# File 'cybertrain/job.rb', line 634

def self.work_one_after(floor, kind = "")
  work_one_after_id(floor, kind) >= 0
end