Module: Cybertrain::Scheduler
- Defined in:
- cybertrain/scheduler.rb,
cybertrain/scheduler/timetable.rb
Overview
Cybertrain::Scheduler – “run this every N seconds” or “at these minutes” (a cron expression) for a served application, on one background thread the framework starts.
# config/app.rb, or a file bin/
The timetable is kept in the database (Timetable,
cybertrain_schedules): when each task last ran and runs next. A
restart or a new leader continues from it, so neither resets an
every interval nor runs every task at once.
Application#serve starts the thread (Scheduler.start!) once the
database is connected, the port is bound and SPINEL_WORKERS is set
(spikes/NOTES.md rule 22), and only in a process that serves or runs
as ./<app> worker: ./<app> migrate, db ..., jobs ... and
Main.command commands never start it. Each run
is wrapped in a rescue that logs an ERROR line, so a failing task does not
end the loop. Jobs do not run here: Job.start! runs them on threads of
their own, so a long job never delays a task (a long task still delays
the other tasks: a task with real work to do should Job.enqueue it).
Config#worker (CYBERTRAIN_WORKER=0) turns the thread off in a
server process.
Several processes may run the scheduler against one database (./<app>
worker twice, or a server and a worker): every tick takes or renews
the lease “scheduler” (Lease.acquire, Config#scheduler_lease_ttl
seconds), and only the process holding it runs the due tasks; the
others keep ticking and take over once the leader’s lease expires.
Every tick of a process that runs jobs, leader or not, also calls
Job.recover!, at most once a minute (recovery is idempotent, so it
needs no lease).
Defined Under Namespace
Modules: Timetable
Class Method Summary collapse
-
.cron(expr, name, tz: "", catch_up: false, job: "", args: {}, &block) ⇒ nil
Registers a task that runs at the minutes a five-field cron expression matches in the zone
tz(Cron.parse lists the syntax; across a DST gap a skipped wall time does not run, across a fold a fixed-hour task runs once). -
.every(seconds, name, &block) ⇒ nil
Registers a task that runs every
secondsseconds while the server runs: firstsecondsafter the scheduler starts, thensecondsafter the start of each run (a run that takes longer is followed at once by the next). -
.holder ⇒ String
Who this process is in the lease: Job.process_label.
-
.job_kinds_error ⇒ String
The error Scheduler.start! raises (and Application#boot reports) when a
crontask’sjob:names a kind no Job.handle registered: the first such task, its kind and the registered kinds; “” when every kind is registered. -
.last_tick ⇒ Integer
Epoch seconds of the scheduler’s last tick, 0 before the first.
-
.leader? ⇒ Boolean
Whether this process held the scheduler lease at its last tick, and so ran the due tasks.
-
.prune_schedules ⇒ Integer
Deletes the timetable rows of task names no longer registered (a task renamed or removed keeps its row until then).
-
.run_due(now) ⇒ Integer
One tick: takes or renews the scheduler lease, and, holding it, runs every task whose time has come (in registration order) and records the run in the timetable; then, when Job threads run in this process and a minute has passed since the last one, Job.recover!; and records
nowas Scheduler.last_tick. -
.running? ⇒ Boolean
True between Scheduler.start! and Scheduler.stop!.
-
.schedule_all(now, default_tz = Cybertrain.config.scheduler_tz) ⇒ nil
Schedules every task from the timetable, as start! does at a start.
-
.start!(config = Cybertrain.config) ⇒ Boolean
Starts the scheduler thread, unless Config#worker is off, nothing is registered (no task and no Job handler, whose rows the ticks recover), or it already runs.
-
.stop! ⇒ nil
Asks the thread to end; it does after its current tick (within Config#scheduler_interval seconds).
-
.task_summary ⇒ String
“name every Ns, name (cron “expr” zone) next 2026-11-09T10:12Z, …” for the start line; “no scheduled tasks (job recovery only)” when none is registered.
Class Method Details
.cron(expr, name, tz: "", catch_up: false, job: "", args: {}, &block) ⇒ nil
Registers a task that runs at the minutes a five-field cron
expression matches in the zone tz (Cron.parse lists the syntax;
across a DST gap a skipped wall time does not run, across a fold a
fixed-hour task runs once). It takes a block, run on the scheduler
thread as an every task’s is, or job:, a kind registered
with Job.handle: the task then only enqueues that job with args
as its JSON payload, which is what a task with real work should do.
The kind is checked when the scheduler starts (start!).
A run missed while no process ran (a slot that passed during a
restart) is skipped, as cron and Solid Queue do; with catch_up:
true the task runs once at the first tick and then continues on its
schedule.
184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 |
# File 'cybertrain/scheduler.rb', line 184 def self.cron(expr, name, tz: "", catch_up: false, job: "", args: {}, &block) raise ArgumentError, "Scheduler.cron: the name is empty" if name.empty? raise ArgumentError, "Scheduler.cron #{name.inspect}: already registered" if task_index(name) >= 0 if block.nil? && job.empty? = "Scheduler.cron #{name.inspect}: give a block or job:, not neither" raise ArgumentError, end if !block.nil? && !job.empty? = "Scheduler.cron #{name.inspect}: give a block or job:, not both" raise ArgumentError, end parsed = schedule_parse(expr, name) # The default zone is resolved by schedule_all: Config#scheduler_tz # may still change, and boot reports a bad one (scheduler_error). zone = TimeZone::UTC unless tz.empty? zone = TimeZone.find(tz) unless zone.found? = "Scheduler.cron #{name.inspect}: tz #{tz.inspect} is not a known time zone" raise ArgumentError, end end payload = JSON.generate(args) TASKS << Task.new(name, 0, block, tz, zone, parsed, catch_up, job, payload) nil end |
.every(seconds, name, &block) ⇒ nil
Registers a task that runs every seconds seconds while the server
runs: first seconds after the scheduler starts, then seconds after
the start of each run (a run that takes longer is followed at once by
the next). The block takes no arguments (Spinel has no instance_eval;
reach what it needs through constants and module methods). Register
before the server starts; a task added later is ignored until the
next start.
Tasks run one after another on one thread. A SQLite call holds its OS
worker until it returns, so with SPINEL_WORKERS=1 a task’s long
query stalls every request for as long: keep tasks short and batch
big work (delete a thousand rows per run, not a million).
With several processes, the one holding the scheduler lease runs the tasks. The lease is renewed before each task, so one task must finish within Config#scheduler_lease_ttl seconds, or another process may take the lease and run tasks beside it. A restart or a new leader continues from the persisted timetable: a run that came due while no process ran happens once, at the first tick. A task the old leader was running when it died runs again: write tasks that do no harm when they run twice.
145 146 147 148 149 150 151 152 |
# File 'cybertrain/scheduler.rb', line 145 def self.every(seconds, name, &block) raise ArgumentError, "Scheduler.every #{name.inspect}: seconds must be at least 1" if seconds < 1 raise ArgumentError, "Scheduler.every: the name is empty" if name.empty? raise ArgumentError, "Scheduler.every #{name.inspect}: already registered" if task_index(name) >= 0 TASKS << Task.new(name, seconds, block, "", TimeZone::UTC, nil, false, "", "{}") nil end |
.holder ⇒ String
Who this process is in the lease: Job.process_label.
343 344 345 |
# File 'cybertrain/scheduler.rb', line 343 def self.holder Job.process_label end |
.job_kinds_error ⇒ String
The error start! raises (and Application#boot reports)
when a cron task’s job: names a kind no Job.handle registered:
the first such task, its kind and the registered kinds; “” when
every kind is registered.
227 228 229 230 231 232 233 234 235 236 237 238 |
# File 'cybertrain/scheduler.rb', line 227 def self.job_kinds_error i = 0 while i < TASKS.size task = TASKS[i] kind = task.task_job_kind if !kind.empty? && Job.handler_index(kind) < 0 return "Scheduler.cron #{task.task_name.inspect}: job #{kind.inspect} is not a registered job kind (#{registered_kinds_text})" end i += 1 end "" end |
.last_tick ⇒ Integer
Epoch seconds of the scheduler’s last tick, 0 before the first. A
health endpoint can compare it with Clock.now.to_i: a value older
than a few Config#scheduler_intervals means the loop is stuck in a
task (or not running).
359 360 361 362 363 |
# File 'cybertrain/scheduler.rb', line 359 def self.last_tick tick = 0 LOCK.synchronize { tick = @last_tick } tick end |
.leader? ⇒ Boolean
Whether this process held the scheduler lease at its last tick, and so ran the due tasks. A process that is not the leader still ticks (last_tick).
336 337 338 |
# File 'cybertrain/scheduler.rb', line 336 def self.leader? @leader end |
.prune_schedules ⇒ Integer
Deletes the timetable rows of task names no longer registered (a task renamed or removed keeps its row until then). start! calls it.
260 261 262 263 264 265 266 267 268 |
# File 'cybertrain/scheduler.rb', line 260 def self.prune_schedules names = Array.new(0) { "" } i = 0 while i < TASKS.size names << TASKS[i].task_name i += 1 end Timetable.prune(names) end |
.run_due(now) ⇒ Integer
One tick: takes or renews the scheduler lease, and, holding it, runs
every task whose time has come (in registration order) and records
the run in the timetable; then, when Job threads run in this
process and a minute has passed since the last one, Job.recover!;
and records now as last_tick. A task the scheduler sees
for the first time is only scheduled: an every task seconds after
now, a cron task at the expression’s next match. After a run the
next one is computed from it: the run plus the interval, or the
expression’s next match after the run in the task’s zone. A cron
task whose next match cannot be computed (none within five years) is
logged once as a failed task: a due one still runs this once, and it
is then skipped on every later tick until restart; the others run. A
tick that takes the lease after one that did not (a change of
leader, or the first leading tick after start!) first re-reads the
timetable: every task with a row is scheduled from it, and a cron
slot the row records as due (passed, not run) that came due since
this process started runs once at that tick. A slot that passed
before the start (while no process ran) keeps
schedule_all’s rule however many ticks failed to take
the lease: it is skipped unless catch_up. A tick that cannot reach
the database logs the error, runs nothing and counts as not leading;
the next one tries again. The thread calls it
with the current time; tests call it with chosen times instead of
sleeping.
392 393 394 395 396 397 398 399 400 401 402 403 404 405 406 407 408 409 410 411 412 413 414 415 416 417 418 419 420 421 422 423 424 425 426 427 428 429 430 431 432 433 434 435 436 437 438 439 440 441 442 443 444 445 446 447 448 449 450 |
# File 'cybertrain/scheduler.rb', line 392 def self.run_due(now) started = Process.clock_gettime(Process::CLOCK_MONOTONIC) was_leader = @leader leader = acquire_safely(now) @leader = leader # Taking over (this tick leads, the last one did not): the table the # leader wrote is the timetable, and a cron slot it records as due # runs once if it came due since the start (one that passed before # the start, while no process ran, is skipped unless catch_up). (now, @started_at) if leader && !was_leader ran = 0 i = 0 while i < TASKS.size task = TASKS[i] due = task.task_next_run if due < 0 # Disabled: no next run could be computed (logged once). nil elsif due == 0 first = schedule_next(task, now) task.task_next_run = first > 0 ? first : -1 elsif due <= now if leader # Renewed before each task (the tick's own `now` plus the time # the tick has taken): a leader that lost the lease while a # task ran stops running tasks in this tick. elapsed = (Process.clock_gettime(Process::CLOCK_MONOTONIC) - started).to_i leader = acquire_safely(now + elapsed) @leader = leader end if leader # The next run is computed from this one (its last run), before # the task runs, so a raising task keeps its schedule. A due # task with no next run still runs this once; it is then # disabled (-1) and its row keeps the slot it ran as next. following = schedule_next(task, now) task.task_last_run = now task.task_next_run = following > 0 ? following : -1 run_task(task) record_safely(task, following > 0 ? following : due, now) ran += 1 else # A process that does not lead keeps a timetable of its own, as # if the leader had run the task when it was due (a `cron` task) # or now (an `every` task). It re-reads the table when it takes # over (reload_timetable), so this guess only stands for a task # the table has no row of. presumed = task.task_cron.nil? ? now : due following = schedule_next(task, presumed) task.task_last_run = presumed task.task_next_run = following > 0 ? following : -1 end end i += 1 end recover_jobs(now) if Job.running? && (@last_recovery == 0 || now - @last_recovery >= RECOVERY_INTERVAL) LOCK.synchronize { @last_tick = now } ran end |
.running? ⇒ Boolean
349 350 351 |
# File 'cybertrain/scheduler.rb', line 349 def self.running? @running end |
.schedule_all(now, default_tz = Cybertrain.config.scheduler_tz) ⇒ nil
Schedules every task from the timetable, as start! does at a start.
A cron task given no tz: first takes the zone default_tz. An
every task continues from its last run (an overdue one runs at the
first tick); a cron task takes its next match after its last run,
and when that has passed while no process ran, the next match after
now – or, with catch_up: true, runs once at the first tick. A
task with no row is scheduled from now (an interval after it, or
the next match) and gets its row, unless another process wrote one
meanwhile.
611 612 613 614 615 616 617 618 619 620 621 622 623 624 625 626 627 628 629 630 631 632 |
# File 'cybertrain/scheduler.rb', line 611 def self.schedule_all(now, default_tz = Cybertrain.config.scheduler_tz) default_zone = TimeZone.find(default_tz) rows = i = 0 while i < TASKS.size task = TASKS[i] task.task_zone = default_zone if task.task_tz_name.empty? name = task.task_name if rows.key?(name) pair = rows[name] schedule_from_row(task, pair[0], pair[1], now, now) else task.task_last_run = 0 first = schedule_next(task, now) task.task_next_run = first > 0 ? first : -1 record_missing_safely(task, now) if first > 0 end i += 1 end @started_at = now nil end |
.start!(config = Cybertrain.config) ⇒ Boolean
Starts the scheduler thread, unless Config#worker is off, nothing is registered (no task and no Job handler, whose rows the ticks recover), or it already runs. Before it starts it removes the timetable rows of names no longer registered (prune_schedules) and schedules every task from the timetable (schedule_all). Application#serve calls it; the database must be connected.
292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 309 310 311 312 313 314 315 |
# File 'cybertrain/scheduler.rb', line 292 def self.start!(config = Cybertrain.config) return false if @running kinds_error = job_kinds_error raise ArgumentError, kinds_error unless kinds_error.empty? unless config.worker Cybertrain.logger.info("Background worker off (CYBERTRAIN_WORKER=0): scheduled tasks and jobs run elsewhere") return false end return false if TASKS.empty? && !Job.handlers? prune_safely schedule_all(Clock.now.to_i, config.scheduler_tz) # The thread's first tick that leads re-reads the table (run_due). @leader = false @lease_ttl = config.scheduler_lease_ttl @running = true @generation += 1 Cybertrain.logger.info("Scheduler started: #{task_summary}") spawn_loop(config.scheduler_interval, @generation) true end |
.stop! ⇒ nil
Asks the thread to end; it does after its current tick (within Config#scheduler_interval seconds). Tests and Application#serve’s shutdown call it.
322 323 324 325 326 327 328 329 |
# File 'cybertrain/scheduler.rb', line 322 def self.stop! was_running = @running @running = false # Hand the lease over at once rather than after its ttl. (A tick in # progress on the thread may still renew it; it then expires.) release_safely if was_running && @leader nil end |
.task_summary ⇒ String
“name every Ns, name (cron “expr” zone) next 2026-11-09T10:12Z, …” for the start line; “no scheduled tasks (job recovery only)” when none is registered.
706 707 708 709 710 711 712 713 714 715 716 717 718 719 720 721 722 723 724 725 726 727 728 729 730 731 |
# File 'cybertrain/scheduler.rb', line 706 def self.task_summary return "no scheduled tasks (job recovery only)" if TASKS.empty? out = +"" i = 0 while i < TASKS.size task = TASKS[i] out << ", " unless out.empty? out << task.task_name expr = task.task_cron if expr.nil? out << " every " out << task.task_interval.to_s out << "s" else out << " (cron \"" out << expr.expression_text out << "\" " out << task.task_zone.zone_name out << ") next " out << summary_stamp(task.task_next_run) end i += 1 end out end |