Module: Cybertrain::Scheduler

Defined in:
cybertrain/scheduler.rb,
cybertrain/scheduler/timetable.rb

Overview

Cybertrain::Scheduler – “run this every N seconds” or “at these minutes” (a cron expression) for a served application, on one background thread the framework starts.

# config/app.rb, or a file bin/.rb requires before Main.run: Cybertrain::Scheduler.every(3600, "prune sessions") do Cybertrain::DB.with { |c| c.execute("DELETE FROM sessions WHERE updated_at < datetime('now', '-30 days')") } end Cybertrain::Scheduler.cron("0 9 * * 1-5", "morning report", tz: "Asia/Tokyo", job: "morning_report", args: { "scope" => "all" })

The timetable is kept in the database (Timetable, cybertrain_schedules): when each task last ran and runs next. A restart or a new leader continues from it, so neither resets an every interval nor runs every task at once.

Application#serve starts the thread (Scheduler.start!) once the database is connected, the port is bound and SPINEL_WORKERS is set (spikes/NOTES.md rule 22), and only in a process that serves or runs as ./<app> worker: ./<app> migrate, db ..., jobs ... and Main.command commands never start it. Each run is wrapped in a rescue that logs an ERROR line, so a failing task does not end the loop. Jobs do not run here: Job.start! runs them on threads of their own, so a long job never delays a task (a long task still delays the other tasks: a task with real work to do should Job.enqueue it). Config#worker (CYBERTRAIN_WORKER=0) turns the thread off in a server process.

Several processes may run the scheduler against one database (./<app> worker twice, or a server and a worker): every tick takes or renews the lease “scheduler” (Lease.acquire, Config#scheduler_lease_ttl seconds), and only the process holding it runs the due tasks; the others keep ticking and take over once the leader’s lease expires. Every tick of a process that runs jobs, leader or not, also calls Job.recover!, at most once a minute (recovery is idempotent, so it needs no lease).

Defined Under Namespace

Modules: Timetable

Class Method Summary collapse

Class Method Details

.cron(expr, name, tz: "", catch_up: false, job: "", args: {}, &block) ⇒ nil

Registers a task that runs at the minutes a five-field cron expression matches in the zone tz (Cron.parse lists the syntax; across a DST gap a skipped wall time does not run, across a fold a fixed-hour task runs once). It takes a block, run on the scheduler thread as an every task’s is, or job:, a kind registered with Job.handle: the task then only enqueues that job with args as its JSON payload, which is what a task with real work should do. The kind is checked when the scheduler starts (start!).

A run missed while no process ran (a slot that passed during a restart) is skipped, as cron and Solid Queue do; with catch_up: true the task runs once at the first tick and then continues on its schedule.

Examples:

Cybertrain::Scheduler.cron("12 * * * *", "collect prices") { Prices.collect }
Cybertrain::Scheduler.cron("0 9 * * 1-5", "morning report", tz: "Asia/Tokyo",
                           job: "morning_report", args: { "scope" => "all" })

Parameters:

  • expr (String) —

    such as "*/5 * * * *"

  • name (String) —

    for the log; unique

  • tz (String) (defaults to: "") —

    an IANA zone name; “” (the default) for Config#scheduler_tz as it is when the scheduler starts

  • catch_up (Boolean) (defaults to: false) —

    whether a missed run happens once at start

  • job (String) (defaults to: "") —

    a job kind to enqueue instead of running a block; “” for a block task

  • args (Hash) (defaults to: {}) —

    the job’s payload, as JSON

Returns:

  • (nil)

Raises:

  • (ArgumentError) —

    for an expression Cron.parse refuses, a tz: given and unknown, an empty name or one taken, or a block and job: together or neither



184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
# File 'cybertrain/scheduler.rb', line 184

def self.cron(expr, name, tz: "", catch_up: false, job: "", args: {}, &block)
  raise ArgumentError, "Scheduler.cron: the name is empty" if name.empty?
  raise ArgumentError, "Scheduler.cron #{name.inspect}: already registered" if task_index(name) >= 0

  if block.nil? && job.empty?
    message = "Scheduler.cron #{name.inspect}: give a block or job:, not neither"
    raise ArgumentError, message
  end
  if !block.nil? && !job.empty?
    message = "Scheduler.cron #{name.inspect}: give a block or job:, not both"
    raise ArgumentError, message
  end
  parsed = schedule_parse(expr, name)
  # The default zone is resolved by schedule_all: Config#scheduler_tz
  # may still change, and boot reports a bad one (scheduler_error).
  zone = TimeZone::UTC
  unless tz.empty?
    zone = TimeZone.find(tz)
    unless zone.found?
      message = "Scheduler.cron #{name.inspect}: tz #{tz.inspect} is not a known time zone"
      raise ArgumentError, message
    end
  end
  payload = JSON.generate(args)
  TASKS << Task.new(name, 0, block, tz, zone, parsed, catch_up, job, payload)
  nil
end

.every(seconds, name, &block) ⇒ nil

Registers a task that runs every seconds seconds while the server runs: first seconds after the scheduler starts, then seconds after the start of each run (a run that takes longer is followed at once by the next). The block takes no arguments (Spinel has no instance_eval; reach what it needs through constants and module methods). Register before the server starts; a task added later is ignored until the next start.

Tasks run one after another on one thread. A SQLite call holds its OS worker until it returns, so with SPINEL_WORKERS=1 a task’s long query stalls every request for as long: keep tasks short and batch big work (delete a thousand rows per run, not a million).

With several processes, the one holding the scheduler lease runs the tasks. The lease is renewed before each task, so one task must finish within Config#scheduler_lease_ttl seconds, or another process may take the lease and run tasks beside it. A restart or a new leader continues from the persisted timetable: a run that came due while no process ran happens once, at the first tick. A task the old leader was running when it died runs again: write tasks that do no harm when they run twice.

Examples:

Cybertrain::Scheduler.every(60, "expire carts") { Cart.where_sql("updated_at < ?", [cutoff]).delete_all }

Parameters:

  • seconds (Integer) —

    at least 1

  • name (String) —

    for the log; unique

Returns:

  • (nil)

Raises:

  • (ArgumentError) —

    for seconds below 1, an empty name or one taken



145
146
147
148
149
150
151
152
# File 'cybertrain/scheduler.rb', line 145

def self.every(seconds, name, &block)
  raise ArgumentError, "Scheduler.every #{name.inspect}: seconds must be at least 1" if seconds < 1
  raise ArgumentError, "Scheduler.every: the name is empty" if name.empty?
  raise ArgumentError, "Scheduler.every #{name.inspect}: already registered" if task_index(name) >= 0

  TASKS << Task.new(name, seconds, block, "", TimeZone::UTC, nil, false, "", "{}")
  nil
end

.holder ⇒ String

Who this process is in the lease: Job.process_label.

Returns:

  • (String)


343
344
345
# File 'cybertrain/scheduler.rb', line 343

def self.holder
  Job.process_label
end

.job_kinds_error ⇒ String

The error start! raises (and Application#boot reports) when a cron task’s job: names a kind no Job.handle registered: the first such task, its kind and the registered kinds; “” when every kind is registered.

Returns:

  • (String)


227
228
229
230
231
232
233
234
235
236
237
238
# File 'cybertrain/scheduler.rb', line 227

def self.job_kinds_error
  i = 0
  while i < TASKS.size
    task = TASKS[i]
    kind = task.task_job_kind
    if !kind.empty? && Job.handler_index(kind) < 0
      return "Scheduler.cron #{task.task_name.inspect}: job #{kind.inspect} is not a registered job kind (#{registered_kinds_text})"
    end
    i += 1
  end
  ""
end

.last_tick ⇒ Integer

Epoch seconds of the scheduler’s last tick, 0 before the first. A health endpoint can compare it with Clock.now.to_i: a value older than a few Config#scheduler_intervals means the loop is stuck in a task (or not running).

Returns:



359
360
361
362
363
# File 'cybertrain/scheduler.rb', line 359

def self.last_tick
  tick = 0
  LOCK.synchronize { tick = @last_tick }
  tick
end

.leader? ⇒ Boolean

Whether this process held the scheduler lease at its last tick, and so ran the due tasks. A process that is not the leader still ticks (last_tick).

Returns:

  • (Boolean)


336
337
338
# File 'cybertrain/scheduler.rb', line 336

def self.leader?
  @leader
end

.prune_schedules ⇒ Integer

Deletes the timetable rows of task names no longer registered (a task renamed or removed keeps its row until then). start! calls it.

Returns:

  • (Integer) —

    how many rows were deleted

Raises:

  • (Cybertrain::DB::Error) —

    when the database is not reachable



260
261
262
263
264
265
266
267
268
# File 'cybertrain/scheduler.rb', line 260

def self.prune_schedules
  names = Array.new(0) { "" }
  i = 0
  while i < TASKS.size
    names << TASKS[i].task_name
    i += 1
  end
  Timetable.prune(names)
end

.run_due(now) ⇒ Integer

One tick: takes or renews the scheduler lease, and, holding it, runs every task whose time has come (in registration order) and records the run in the timetable; then, when Job threads run in this process and a minute has passed since the last one, Job.recover!; and records now as last_tick. A task the scheduler sees for the first time is only scheduled: an every task seconds after now, a cron task at the expression’s next match. After a run the next one is computed from it: the run plus the interval, or the expression’s next match after the run in the task’s zone. A cron task whose next match cannot be computed (none within five years) is logged once as a failed task: a due one still runs this once, and it is then skipped on every later tick until restart; the others run. A tick that takes the lease after one that did not (a change of leader, or the first leading tick after start!) first re-reads the timetable: every task with a row is scheduled from it, and a cron slot the row records as due (passed, not run) that came due since this process started runs once at that tick. A slot that passed before the start (while no process ran) keeps schedule_all’s rule however many ticks failed to take the lease: it is skipped unless catch_up. A tick that cannot reach the database logs the error, runs nothing and counts as not leading; the next one tries again. The thread calls it with the current time; tests call it with chosen times instead of sleeping.

Parameters:

Returns:

  • (Integer) —

    how many tasks ran (including ones that raised)



392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
# File 'cybertrain/scheduler.rb', line 392

def self.run_due(now)
  started = Process.clock_gettime(Process::CLOCK_MONOTONIC)
  was_leader = @leader
  leader = acquire_safely(now)
  @leader = leader
  # Taking over (this tick leads, the last one did not): the table the
  # leader wrote is the timetable, and a cron slot it records as due
  # runs once if it came due since the start (one that passed before
  # the start, while no process ran, is skipped unless catch_up).
  reload_timetable(now, @started_at) if leader && !was_leader
  ran = 0
  i = 0
  while i < TASKS.size
    task = TASKS[i]
    due = task.task_next_run
    if due < 0
      # Disabled: no next run could be computed (logged once).
      nil
    elsif due == 0
      first = schedule_next(task, now)
      task.task_next_run = first > 0 ? first : -1
    elsif due <= now
      if leader
        # Renewed before each task (the tick's own `now` plus the time
        # the tick has taken): a leader that lost the lease while a
        # task ran stops running tasks in this tick.
        elapsed = (Process.clock_gettime(Process::CLOCK_MONOTONIC) - started).to_i
        leader = acquire_safely(now + elapsed)
        @leader = leader
      end
      if leader
        # The next run is computed from this one (its last run), before
        # the task runs, so a raising task keeps its schedule. A due
        # task with no next run still runs this once; it is then
        # disabled (-1) and its row keeps the slot it ran as next.
        following = schedule_next(task, now)
        task.task_last_run = now
        task.task_next_run = following > 0 ? following : -1
        run_task(task)
        record_safely(task, following > 0 ? following : due, now)
        ran += 1
      else
        # A process that does not lead keeps a timetable of its own, as
        # if the leader had run the task when it was due (a `cron` task)
        # or now (an `every` task). It re-reads the table when it takes
        # over (reload_timetable), so this guess only stands for a task
        # the table has no row of.
        presumed = task.task_cron.nil? ? now : due
        following = schedule_next(task, presumed)
        task.task_last_run = presumed
        task.task_next_run = following > 0 ? following : -1
      end
    end
    i += 1
  end
  recover_jobs(now) if Job.running? && (@last_recovery == 0 || now - @last_recovery >= RECOVERY_INTERVAL)
  LOCK.synchronize { @last_tick = now }
  ran
end

.running? ⇒ Boolean

Returns true between start! and stop!.

Returns:



349
350
351
# File 'cybertrain/scheduler.rb', line 349

def self.running?
  @running
end

.schedule_all(now, default_tz = Cybertrain.config.scheduler_tz) ⇒ nil

Schedules every task from the timetable, as start! does at a start. A cron task given no tz: first takes the zone default_tz. An every task continues from its last run (an overdue one runs at the first tick); a cron task takes its next match after its last run, and when that has passed while no process ran, the next match after now – or, with catch_up: true, runs once at the first tick. A task with no row is scheduled from now (an interval after it, or the next match) and gets its row, unless another process wrote one meanwhile.

Parameters:

  • now (Integer) —

    epoch seconds

  • default_tz (String) (defaults to: Cybertrain.config.scheduler_tz) —

    the zone of cron tasks registered without tz: (Config#scheduler_tz); one TimeZone cannot load behaves as UTC (boot reports it)

Returns:

  • (nil)


611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
# File 'cybertrain/scheduler.rb', line 611

def self.schedule_all(now, default_tz = Cybertrain.config.scheduler_tz)
  default_zone = TimeZone.find(default_tz)
  rows = timetable_safely
  i = 0
  while i < TASKS.size
    task = TASKS[i]
    task.task_zone = default_zone if task.task_tz_name.empty?
    name = task.task_name
    if rows.key?(name)
      pair = rows[name]
      schedule_from_row(task, pair[0], pair[1], now, now)
    else
      task.task_last_run = 0
      first = schedule_next(task, now)
      task.task_next_run = first > 0 ? first : -1
      record_missing_safely(task, now) if first > 0
    end
    i += 1
  end
  @started_at = now
  nil
end

.start!(config = Cybertrain.config) ⇒ Boolean

Starts the scheduler thread, unless Config#worker is off, nothing is registered (no task and no Job handler, whose rows the ticks recover), or it already runs. Before it starts it removes the timetable rows of names no longer registered (prune_schedules) and schedules every task from the timetable (schedule_all). Application#serve calls it; the database must be connected.

Parameters:

  • config (Config) (defaults to: Cybertrain.config)

Returns:

  • (Boolean) —

    true when it started the thread

Raises:

  • (ArgumentError) —

    when a cron task’s job: is not a registered kind (job_kinds_error; Application#boot reports it first)



292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
# File 'cybertrain/scheduler.rb', line 292

def self.start!(config = Cybertrain.config)
  return false if @running

  kinds_error = job_kinds_error
  raise ArgumentError, kinds_error unless kinds_error.empty?

  unless config.worker
    Cybertrain.logger.info("Background worker off (CYBERTRAIN_WORKER=0): scheduled tasks and jobs run elsewhere")
    return false
  end

  return false if TASKS.empty? && !Job.handlers?

  prune_safely
  schedule_all(Clock.now.to_i, config.scheduler_tz)
  # The thread's first tick that leads re-reads the table (run_due).
  @leader = false
  @lease_ttl = config.scheduler_lease_ttl
  @running = true
  @generation += 1
  Cybertrain.logger.info("Scheduler started: #{task_summary}")
  spawn_loop(config.scheduler_interval, @generation)
  true
end

.stop! ⇒ nil

Asks the thread to end; it does after its current tick (within Config#scheduler_interval seconds). Tests and Application#serve’s shutdown call it.

Returns:

  • (nil)


322
323
324
325
326
327
328
329
# File 'cybertrain/scheduler.rb', line 322

def self.stop!
  was_running = @running
  @running = false
  # Hand the lease over at once rather than after its ttl. (A tick in
  # progress on the thread may still renew it; it then expires.)
  release_safely if was_running && @leader
  nil
end

.task_summary ⇒ String

“name every Ns, name (cron “expr” zone) next 2026-11-09T10:12Z, …” for the start line; “no scheduled tasks (job recovery only)” when none is registered.

Returns:

  • (String)


706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
# File 'cybertrain/scheduler.rb', line 706

def self.task_summary
  return "no scheduled tasks (job recovery only)" if TASKS.empty?

  out = +""
  i = 0
  while i < TASKS.size
    task = TASKS[i]
    out << ", " unless out.empty?
    out << task.task_name
    expr = task.task_cron
    if expr.nil?
      out << " every "
      out << task.task_interval.to_s
      out << "s"
    else
      out << " (cron \""
      out << expr.expression_text
      out << "\" "
      out << task.task_zone.zone_name
      out << ") next "
      out << summary_stamp(task.task_next_run)
    end
    i += 1
  end
  out
end