Background work
An application that has to do something outside a request (send a
notification after the response went out, sweep old rows every night, retry
a call that failed) hands it to the framework, which runs it on background
threads of the served process, or of a separate worker process. A job is a
class in app/jobs/, as in Rails. Jobs and scheduled tasks need
require "cybertrain/jobs" in bin/<app>.rb and bin/db.rb, which
cybertrain generate job adds (Subsystems):
# app/jobs/welcome_job.rb
class WelcomeJob < Cybertrain::Job::Base
retry_on Cybertrain::Mailer::TimeoutError, wait: 60, attempts: 5
discard_on ArgumentError
def perform(args)
user = User.find(Cybertrain::Cast.int(args["user_id"]))
Cybertrain::Mailer.deliver(to: user.email, subject: "Welcome", body: "Hello!\n")
end
end
# in an action, after saving the user
WelcomeJob.perform_later("user_id" => @user.id)
WelcomeJob.perform_in(600, "user_id" => @user.id) # or perform_at(time, args)
# config/app.rb (or a file bin/<app>.rb requires before Cybertrain::Main.run)
Cybertrain::Scheduler.cron("0 4 * * *", "nightly digest", job: "DigestJob")
Cybertrain::Scheduler.every(300, "refresh rates") { Rates.refresh }
- Where it runs. Jobs run on
job_threadsjob threads (3,CYBERTRAIN_JOB_THREADS); scheduled tasks run on one scheduler thread of their own, so a long job never holds up a task. Both start when the binary serves (./<app>,./<app> server) or runs as a worker (./<app> worker, see Process roles), after boot has connected the database and setSPINEL_WORKERS;migrate,db ...,jobs ...,consoleandMain.commandcommands never start them, and neither doesCybertrain::Test::Client. With jobs required, the job threads start in every such process, even in an application with no job class: the framework registers its own mail job (Mailer.deliver_later, Sending e-mail). - The table. Jobs are rows of
cybertrain_jobs, which the framework creates itself (CREATE TABLE IF NOT EXISTS, on first use) and leaves out ofdb/schema.rb. Inside aDB.transactionthe row commits or rolls back with the rest. Each job thread claims the oldest due row in a shortBEGIN IMMEDIATEtransaction, runs it outside the transaction and looks for the next; with nothing due it waitsjob_poll_intervalseconds (1), so a job enqueued by a request starts within about a second. A row isqueued,running,done,failed(its retries ran out; it stays for you to look at) ordiscarded(adiscard_onstopped it).
Job classes
spin run gen reads every file under app/jobs/ (nested directories too:
Admin::ReportJob in app/jobs/admin/report_job.rb) and writes the other
half of each class Cybertrain::Job::Base is an ancestor of into
gen/jobs.rb: perform_later, perform_at, perform_in, perform_now,
job_kind and the code that runs it, and registers the class. cybertrain
generate job send_digest --retry Cybertrain::Mailer::TimeoutError:60:5
writes a class file and a test for it.
argsis the Hash given toperform_later, stored as JSON and read back as aHash<String, Object>: narrow its values withCybertrain::Cast.int(args["user_id"]),Cast.str,Cast.hash,Cast.array. A job takes this one Hash; pass ids, not records (Rails' advice too).perform_laterraisesArgumentErrorfor anything that is not a Hash, and with no argument sends{}.- Enqueueing.
perform_later(args),perform_at(time, args)andperform_in(seconds, args)return the row's id; the row's kind is the class name ("WelcomeJob","Admin::ReportJob").perform_now(args)runsperformon the calling thread, with no queue, no retry policy and no rescue, and returnsnil: a test's shape. - The class body. Every job class defines
perform(args)(the generator refuses one that neither defines nor inherits it).retry_on,discard_onandlimits_concurrencyare read as text, one per line with literal values (an Integer or:polynomially_longerforwait:, an Integer forattempts:andto:, a quoted String forkey:); a block, a lambda,5.minutes, a hashrocket or a trailing comment stopsspin run genwith the file, the line and what to write instead. -
Which exception classes. A
retry_onordiscard_onnames a class the program defines before the job: a core class (ArgumentError,KeyError, ...), any exception class of the framework, whatever its name (Cybertrain::RecordNotFound,Cybertrain::RecordInvalid,Cybertrain::DB::Error, ...; the list isGen::JobScan::FRAMEWORK_ERRORS, and the generator adds the require one outside the core needs), or an application class defined inapp/models/, inapp/jobs/application_job.rbor above the line in the job's own file. Anything else, a class of the test harness included, stopsspin run gen, as inPATH:LINE: Cybertrain::Test::AssertionFailed is not a class this program defines (spikes/NOTES.md rule 89): under Spinel arescuenaming a class the program does not define can swallow every exception. examples/blog'sNotifyCommentJobdiscards a job whose comment was deleted and retries anything else:ruby class NotifyCommentJob < Cybertrain::Job::Base discard_on Cybertrain::RecordNotFound retry_on StandardError, wait: 30, attempts: 3 def perform(args) comment = Comment.find(Cybertrain::Cast.int(args["comment_id"])) # ... end endThe arms are tried in the order written and the first that matches wins, so the narrower class comes first; Rails checks the handler declared last first, so in a Rails application these two lines go the other way round. The discard is logged as
job 12 (NotifyCommentJob) discarded: Cybertrain::RecordNotFound: Couldn't find Comment with id=7. ApplicationJob. A classApplicationJob < Cybertrain::Job::Baseinapp/jobs/application_job.rbis not a job of its own: itsretry_onanddiscard_onlines apply to every job inheriting it, after the job's own, and itslimits_concurrencyto a job that sets none. Any other job class can be a parent; it is then a job in its own right.-
Job.handle, the short form. A block registered under a String kind needs no class and no generator; it gets the payload String and the default policy below:ruby # config/app.rb Cybertrain::Job.handle("notify", concurrency: 1) { |payload| Notifier.post(payload) } # anywhere, later Cybertrain::Job.enqueue("notify", JSON.generate({ "text" => "Deployed" }), run_at: Time.now + 60)Job classes and
Job.handlekinds share one namespace, soJob.enqueue("WelcomeJob", json)adds the same row asperform_later.Job.enqueuerefuses an unregistered kind.
Retries and discards
- The default policy. An exception that no line of the class matches
(and any exception of a
Job.handleblock) is retried 1, 5, 15 and then 60 minutes later, withClass: message(up to 1000 characters) inlast_error, untiljob_max_attempts(5) attempts have failed; the row is thenfailed. ANotImplementedError(a helper not written yet), aSystemStackErroror anotherScriptErrorfrom the job's code takes this policy too, although none is aStandardError; any other exception outsideStandardErroris logged (ERROR Job thread N: ...) and reported, the job thread carries on, and the row staysrunninguntil recovery (below) takes it. retry_on Klass, wait: 60, attempts: 5.wait:is seconds (3 when left out, Rails' default) or:polynomially_longer(attempts ** 4 + 2seconds: 3, 18, 83, 258, ...);attempts:(5 when left out) counts every attempt, the first included. Several lines may be given: the first whose class matches the exception (a subclass matches) wins, so write the narrower class first.retry_on StandardErrorbeforediscard_on ArgumentErrorretries theArgumentError.discard_on Klassstops at once: the row isdiscarded, with the error inlast_error, and is never retried. The log saysERROR [<Kind> <id>] [<request id>] job <id> (<Kind>) discarded: <Class>: <message>(the request id when a request enqueued the job), and the discard is reported once, withjob.max_attemptsequal tojob.attempt.def retries_exhausted(error)runs when aretry_online's attempts ran out, once the row is reallyfailed(never for an attempt another worker took over, whose own run may still succeed). It is Rails'retry_on ... do |job, error|block; an exception it raises is logged.- Recovery (below) follows the class too: a row of a class with
attempts: 20whose process died on attempt 5 is queued again, not failed.
Concurrency per kind and per key
limits_concurrency to: 1, key: "account_id" lets at most to: jobs of the
class run at once across every process sharing the database, counted per
value of args["account_id"] (to: alone counts for the whole class;
Job.handle(kind, concurrency: n) is the same for a block). Other kinds,
and other keys, go on running meanwhile. Enqueueing a keyed job whose args
lack the key raises ArgumentError. The limit counts running rows, so it
holds as long as heartbeats do: a job still running after its heartbeat went
stale is overlapped by the recovered row's next run (below); work that must
never overlap needs a guard of its own.
Scheduled tasks
Scheduler.every(seconds, name) { ... } runs a block every seconds;
Scheduler.cron(expr, name, ...) runs at the minutes a five-field cron
expression matches:
Cybertrain::Scheduler.cron("12 * * * *", "collect prices") { Prices.collect }
Cybertrain::Scheduler.cron("0 9 * * 1-5", "morning report", tz: "Asia/Tokyo",
job: "MorningReportJob", args: { "scope" => "all" })
Cybertrain::Scheduler.cron("*/5 * * * *", "poll feeds", catch_up: true) { Feeds.poll }
- The expression. Minute, hour, day of month, month, day of week, with
*,*/n,a-b,a,b,a-b/nand the namesjan-decandsun-sat(day of week0and7are Sunday);@hourlyand its kin are not accepted. An expression that cannot be parsed, or never matches, raisesArgumentErrornaming the task whenconfig/app.rbloads, so the boot stops. - Time zones.
tz:names an IANA zone (Time zones); without it the task readsscheduler_tz(CYBERTRAIN_SCHEDULER_TZ, which follows the application'stime_zonewhen unset). A wall time that a daylight-saving jump skips does not run that day. In the repeated hour of the autumn change, an expression whose hour field starts with*runs at both occurrences and one with a fixed hour runs once, as Vixie cron does. job:names a job kind (a class name as a String, or aJob.handlekind) andargs:its args: the task then only enqueues it, which is what a task with real work should do. An unknown kind stops the boot with the list of registered ones.- The timetable is kept in the database (
cybertrain_schedules, out ofdb/schema.rb): when each task last ran and runs next. A restart, or another process taking the lead, continues from it. Aneverytask whose time passed while nothing ran runs once at start; acronslot missed while nothing ran is skipped (cron's rule) unless the task sayscatch_up: true, which runs it once at start. A slot that came due while processes were up but no leader ran it (a lease changing hands) runs once on the takeover. A task you rename or remove keeps its row until the next start. A task that has not run yet follows its current schedule when that comes sooner than the next run an earlier release stored (a weekly task made hourly before its first run runs within the hour). - How a task runs. A block runs on the scheduler thread, one task at a
time; each run is wrapped in a rescue that logs
ERROR [schedule:name] Scheduled task "name" failed: ..., reports it, and the loop goes on. The block takes no arguments (Spinel has noinstance_eval). A SQLite call holds its OS worker until it returns: withSPINEL_WORKERS=1(the default) a long query stalls every request, and every other thread, for as long. Give a task with real work a job (job:, orJob.enqueuefrom the block), and batch big work (a thousand rows per run, one job per item).Cybertrain::Scheduler.last_tickis the epoch second of the last tick, for a health check of your own.
The jobs page and the jobs command
# config/routes.rb
Cybertrain::Routes.draw do
jobs_page "/admin/jobs", before_action: :require_admin, skip_before_action: :require_authentication
end
jobs_page mounts a page that lists the count of rows per state, the queued
count and the next due time, the failed and discarded rows (kind,
arguments, error, attempts, times) and the running ones (who claimed them,
how old their heartbeat is), jobs_page_per_page (50,
CYBERTRAIN_JOBS_PAGE_PER_PAGE) rows per list. A failed row can be retried
(queued again at once, attempts reset), discarded or deleted, a discarded
one retried or deleted, and every failed row retried or discarded at once;
each action touches only failed and discarded rows, so a row a worker
has just claimed is left alone.
- The generated controller inherits
ApplicationController, so the application's layout, helpers and CSRF protection apply, andbefore_action:names one of its methods: it must render or redirect to refuse.cybertrain generate authenticationdraws the line above, with its concern'srequire_admin(404 unless an admin is signed in; Admin users). Withoutbefore_action:the page is open to every visitor, andspin run genprints a warning saying so. skip_before_action:names one inherited before callback the page does not run. With:require_authentication,require_adminalone answers a visitor, with a 404 rather than a redirect to sign in. gen refuses it without abefore_action:, and when it names thebefore_action:itself.- Its buttons are forms with the CSRF token. Keep the page's path out of
csrf_exempt_prefixes: under an exempt prefix those forms would be accepted from any site.
From a shell, ./<app> jobs does the same on the configured database
(it starts no thread):
./blog jobs list # failed rows, newest first (the newest 1000); or: list queued|running|done|discarded
./blog jobs retry 42 # or: retry --all (every failed row)
./blog jobs discard 42
./blog jobs delete 42
./blog jobs prune --before 30 # delete failed and discarded rows older than 30 days (7 by default)
done rows stay until something deletes them: Job.prune_done(before_epoch)
from a task (Cybertrain::Scheduler.cron("0 3 * * *", "prune done jobs") {
Cybertrain::Job.prune_done(Time.now.to_i - 7 * 86_400) }).
Job.state_count("failed") serves a health page of your own.
Several workers, at least once, shutting down
- Several workers per database. Every process that runs jobs writes a
heartbeat for the jobs it is running every
job_heartbeat_interval(30 s). Arunningrow whose heartbeat is older thanjob_heartbeat_stale(a process that crashed or was stopped mid-job) is queued again by whichever process looks next, at its start and then once a minute, unless that attempt was its last (the larger ofjob_max_attemptsand the class's largestattempts:): then the row is leftfailed, withprocess stopped (heartbeat stale)inlast_error, so a job that kills its process is not retried for ever. Scheduled tasks run in one process at a time: the processes share a lease in the database (cybertrain_leases, renewed every tick,scheduler_lease_ttl30 s), and only its holder runs due tasks; another takes over when it stops renewing. The lease is renewed before each task, not while one runs, so a task longer than the ttl can overlap with the next holder's: keep tasks short and safe to repeat. So twoworkerprocesses, or a server and a worker, can share one SQLite file;CYBERTRAIN_WORKER=0turns jobs, tasks and feeds off in aserverprocess that should only serve HTTP. - At least once. A job can run twice: a process that is killed, or that
stalls for longer than
job_heartbeat_stale(300 s), may still be running a job when another process takes the row back. Only the outcome of the claim that holds the row counts (an earlier claimer's result, finishing late, is discarded), but the job's side effects happened twice, so write jobs that can safely run again (check whether the mail was already sent, upsert rather than insert), as with any job queue. A scheduled task can run twice around a change of leader for the same reason. - Shutting down. On
SIGTERMthe job threads stop claiming new jobs, and the process waits up tojob_drain_timeout(10 s) for the jobs in flight. A job still running then keeps its rowrunning; its heartbeat has stopped, so another process runs it again once it is stale. - Testing.
WelcomeJob.perform_now(args)runs a job,Cybertrain::Scheduler.run_due(now)runs one tick andCybertrain::Job.work_one(now)one queued job, with an epoch time you choose, so a test never sleeps; Testing jobs has the assertions.
Upgrading from 0.4
cybertrain_jobsgains the columnconcurrency_keyand the statediscarded; the first process that touches the table adds the column. Code that lists states (a dashboard overJob.state_count) addsdiscarded.Job.prune_donestill prunesdonerows only.everytasks continue from the timetable after a restart instead of starting afresh: an interval that passed while the process was down runs once at start.- Giving an existing kind a key (
limits_concurrency ... key:): stop every old process before the first new one starts, as for any upgrade (Process roles). The first new process keys thequeuedrows enqueued before (it logs the ids of rows whose payload has no such field), and arunningrow left by an old process gets its key when it is recovered. jobsis now a built-in command: an application command namedjobsmust be renamed.
Not built: job priority, queue names and uniqueness, an inline job mode for
tests (perform_now and perform_enqueued_jobs cover tests). The design:
docs/design.md D28 and D51 to D53.
Process roles
One binary runs in three roles. By default ./<app> server does everything,
as rails server with Solid Queue inside Puma does; the other two let a
deployment split the work across processes or containers, all from the same
image and the same database:
| Command | Runs | Listens |
|---|---|---|
./<app> server [PORT] |
HTTP, the stream endpoint, and unless CYBERTRAIN_WORKER=0 the scheduled tasks, the job threads and the WebSocket feeds |
yes |
./<app> worker |
the scheduled tasks (one process at a time, through the lease), the job threads, heartbeats and recovery | no |
./<app> feeds [NAME...] |
the WebSocket feeds, all of them or the named ones | no |
workerandfeedsprint a banner (=> worker: 3 job threads, 2 scheduled tasks), run untilSIGTERMorSIGINT(Ctrl-C) and then stop the way a server does. They always run their work:CYBERTRAIN_WORKER=0, set for the web role, does not turn them off.feedswith names runs only those subscriptions (one container per feed), and an unknown name stops the boot with the list of registered ones. With feeds in their own process, setCYBERTRAIN_WORKER=0on the servers, or every process would receive every feed.- A
feedsprocess restarts a feed whose thread died, and exits with status 1 when one feed keeps dying (feed_restart_limitrestarts withinfeed_restart_window), so run it under a restart policy (Docker'srestart: unless-stopped, systemd'sRestart=always); Consuming WebSocket feeds has the details../<app> feeds statusprints each feed's last reported status. - A typical split:
serverwithCYBERTRAIN_WORKER=0behind the proxy, one or moreworkers, andfeedsif there are subscriptions.cybertrain newwrites aweband aworkerrole intoconfig/deploy.yml(and a commentedfeeds); docs/deploy.en.md part 7, "Splitting the roles", has the systemd units. - Upgrading from a release without heartbeats: stop every old process before starting the first new one. A new worker would take the jobs an old one is still running for abandoned, since an old process writes no heartbeat.
Cybertrain::Main.worker?(ARGV)andMain.feeds?(ARGV), likeMain.server?(ARGV), tellbin/<app>.rbwhich role it is starting.workerneedsrequire "cybertrain/jobs"inbin/<app>.rb, andfeedsrequire "cybertrain/feeds"(whichcybertrain/websocketloads); without it the binary says which line is missing and exits with 1.
The cache store
Cybertrain::Cache keeps answers for a while in the application's own
database, so the next request does not ask the slow API again.
require "cybertrain" loads it; an application on cybertrain/core adds
require "cybertrain/cache" to bin/<app>.rb and bin/db.rb, or where it
is used:
require "cybertrain/cache"
text = Cybertrain::Cache.fetch("defillama/tvl", expires_in: 21_600, race_condition_ttl: 30) do
client.get("https://api.llama.fi/tvl").body
end
data = Cybertrain::Cache.fetch_hash("prices/#{sym}", expires_in: 60) { { "usd" => price } }
count = Cybertrain::Cache.fetch_int("articles/count", expires_in: 60) { Article.count }
Cybertrain::Cache.write("import-lock", "1", expires_in: 300, unless_exist: true) # true when written
- Values are Strings.
fetch(key, expires_in:, race_condition_ttl:) { String }answers the live value or runs the block and stores what it returns;fetch_hash,fetch_arrayandfetch_intstore JSON or Integer text and parse it on the way out, so a corrupt cell reads as{},[]or0until it expires and the block runs again (Spinel has noMarshal, and one method answers one type).read(key)(a String, or nil),write(key, value, expires_in:, unless_exist:),delete,exist?,increment(key, by: 1, expires_in:)(one statement; an expired counter starts again fromby, andexpires_in:sets the expiry only when the counter starts, so a counter for a minute counts that minute) andclearare the rest. - Expiry.
expires_in:is seconds;0, the default, never expires (so does a negative value).race_condition_ttl: n: when a row has expired, the first caller extends it bynseconds and computes the new value while the others keep reading the old one, as in Rails; without it every caller of an expired key computes. A block that raises writes nothing. - Storage. The rows live in
cybertrain_cache, which the framework creates on first use and leaves out ofdb/schema.rb. Nothing is kept in memory: every read is a SELECT, which SQLite answers from its page cache. Every write takes the database's writer lock, which requests writing their own rows wait for: cache what jobs produce, or values read far more often than they change, rather than writing in a busy request path. - Pruning. With jobs loaded, requiring the file registers the scheduled
task
"cybertrain cache prune", which deletes expired rows everycache_prune_intervalseconds (300,CYBERTRAIN_CACHE_PRUNE_INTERVAL), a thousand per statement. Without jobs the cache prunes itself: the first write of a process, and the first after eachcache_prune_interval, deletes up to a thousand expired rows. The interval is read when the file is required: set the variable, orc.cache_prune_intervalin aCybertrain.configureblock that runs before the require.Cache.prune(now)is the test entry point.
Not built: a separate cache database file, a size cap, an in-memory store.
The design: docs/design.md D54.
The console
./<app> console is a prompt over the application's models. Spinel has no
eval, so it does not run Ruby: it evaluates the expressions a template's
<%= %> evaluates (the template language,
without blocks), plus model names. In development, run it with
cybertrain spin run <app> -- console.
./blog console
Cybertrain console (production). Type help for commands, exit to quit.
>> Article.where(title: "Hello").count
1
>> a = Article.find(1)
#<Article id: 1, title: "Hello", body: "First post", created_at: 2026-11-05 10:00:00 UTC, updated_at: 2026-11-05 10:00:00 UTC>
>> a.update(title: "Hello again")
true
- A model name is its relation (
Article), withwhere,where_like,order,limit,offset,page,first,last,find,find_by,to_a,count,exists?,pluck_str/pluck_int/pluck_float,sum_int,count_by,delete_all,to_sqlandcreate. A record has its attributes and methods,update,save,destroy,reload,valid?,errorsandattributes.helplists them all. x = exprkeeps a value for the following lines, and_is the last value.sql "..."runs SQL and prints the rows as a table, with a cell's control characters escaped (\e,\n), so data can neither send the terminal an escape sequence nor break the table (a TEXT value holding a NUL byte is cut at the NUL when it is read, as in every raw-SQL read; a BLOB keeps every byte);tableslists the tables;exit,quitor Ctrl-D leaves.- A line that does not parse prints its error, a line that raises prints
Class: message, and the prompt comes back. A binary value prints as<N bytes>. - Writes are real. In production the console says so in a banner first.
./<app> dbconsole opens the sqlite3 shell on the configured database
(CYBERTRAIN_DATABASE), as Rails' dbconsole does; it needs the sqlite3
command installed, and passes its exit status through. Neither command
starts the scheduler, jobs or feeds.
Operations: backups, vacuum, health check
The data of a deployed application is one SQLite file, so the binary carries
the three operational tools such an application needs. An application with
file uploads also keeps their bytes under its upload root
(upload_root, storage/uploads/ by default): back that directory up and
restore it together with the database, or restored records point at missing
files (Add a file upload).
Backups. ./<app> db backup PATH writes a consistent copy of the
configured database (CYBERTRAIN_DATABASE) while the server keeps running:
SQLite's VACUUM INTO (3.27 or newer; the binary checks) reads one snapshot,
and under WAL the server's writes go on meanwhile. The copy is written to a
temporary file next to PATH, created exclusively with mode 0600 (the file
holds password digests and session tokens), fsynced, and only then renamed
to PATH, so a failed or interrupted copy never leaves a partial file under
the real name (the temporary one is deleted). An existing PATH is refused
unless you pass --force; the database itself never is a target, nor is a
:memory: database a source.
./blog db backup /srv/blog/backups/before-upgrade.sqlite3
./blog db backup /srv/blog/backups/ --keep 14 # blog-20261025T031500Z.sqlite3, newest 14 kept
A target ending in / (or an existing directory) gets a dated name,
<app>-<YYYYmmddTHHMMSSZ>.sqlite3 in UTC, in a directory created with mode
0700 if it does not exist; --keep N then deletes the oldest files of
exactly that shape beyond N and touches nothing else there. Restoring is
copying a backup over the database file with the server stopped (and
deleting that file's -wal and -shm beside it).
Vacuum. Deleting rows leaves free pages in the file. ./<app> db vacuum
runs VACUUM and then PRAGMA wal_checkpoint(TRUNCATE), so the rebuilt
pages reach the database file, the file is cut to its new size and the WAL
is emptied; it prints the size on disk (database plus WAL) before and after.
VACUUM rewrites the whole file and holds the write lock while it does, so
run it when the application is quiet. If a reader keeps the checkpoint from
finishing it says so, and the WAL shrinks at a later checkpoint.
Run backups outside the server. A SQLite call holds its OS worker until
it returns, so a large VACUUM INTO on a Scheduler task would stall
requests (with SPINEL_WORKERS=1, all of them) for as long as the copy
takes. Run the command from cron or a systemd timer instead, as a separate
process:
# crontab of the app's user: every night at 03:15, keep two weeks
15 3 * * * cd /srv/blog/current && ./blog db backup /srv/blog/backups/ --keep 14
An application that wants to take backups itself can run its own binary as
a child process from a scheduled task, rather than run the SQL on its own
thread: system waits for the child on the scheduler's thread only, and
requests go on being served meanwhile (use system with an argument list,
never a string with request data in it). A copy that takes longer than
health_worker_stale makes the health check answer worker stale until it
is done.
# config/app.rb, in a served binary started from its own directory (dist/)
Cybertrain::Scheduler.every(86_400, "nightly backup") do
system("./blog", "db", "backup", "storage/backups/", "--keep", "14")
end
Health check. While the health check is enabled (it is by default;
an empty config.health_path turns it off), the served application
answers GET /up (and HEAD) before the static files, the session, CSRF and the router: no controller
is involved and no login before_action can redirect it, and it sets no
cookie. It runs SELECT count(*) FROM sqlite_master, which has to read the
schema page from the file (a bare SELECT 1 never touches it), and answers
in text/plain with Cache-Control: no-store:
| Status | Body | When |
|---|---|---|
| 200 | ok |
the query succeeded |
| 503 | database unavailable |
it raised Cybertrain::DB::Error (the file is missing or unreadable, the pool is closed) |
| 503 | worker stale |
the background Scheduler runs and its last tick is older than health_worker_stale seconds (300): a task is stuck, or runs far too long |
| 503 | feed stale |
a WebSocket feed, in this process or another, has not reported its status, or has received nothing (not even a pong), for health_feed_stale seconds (180); see Feed health |
A poll is logged like any request (two lines). The path is
config.health_path (CYBERTRAIN_HEALTH_PATH, default /up; an empty
value turns the check off, and the path then goes on to public/ and the
router as before). A load balancer, kamal-proxy or a deploy script can poll
it; public/up.html, which cybertrain new writes and its Kamal
configuration still polls, only shows that the process answers HTTP.
Logs
The framework and the application log through Cybertrain.logger, one line
per message on standard output, flushed after each line, as a twelve-factor
application does (systemd's journal, Kamal and Docker collect it):
2026-11-14T09:30:00.120Z INFO [3f2a9c1e-6c1d-4f0b-9a52-0d1e2f3a4b5c] Started GET "/articles" for 127.0.0.1
2026-11-14T09:30:00.132Z INFO [3f2a9c1e-6c1d-4f0b-9a52-0d1e2f3a4b5c] Completed 200 in 12ms
2026-11-14T09:30:01.002Z ERROR [WelcomeJob 42] [3f2a9c1e-6c1d-4f0b-9a52-0d1e2f3a4b5c] job 42 (WelcomeJob) attempt 1 of 5 failed: RuntimeError: smtp down; retrying in 60 s
2026-11-14T09:30:02.410Z WARN [feed:prices] WebSocket "prices" silent for 60s; reconnecting
A line is the time in UTC with milliseconds (in a test it follows
travel_to), the level in capitals, the tags in brackets and the message.
Cybertrain.logger has debug, info, warn, error and fatal, each
taking one String. Lines from several threads never mix: each line is built
whole and written with one write under a lock that is held for that write
only. A write that fails (a closed pipe) drops the line; logging never
raises into its caller. A reader that stops reading standard output (a full
pipe) blocks every thread that logs until it reads again, so keep the
collector running.
One event, one line. A line feed in a message or in a tag is written
as the two characters \n, a carriage return as \r, so a message of
several lines, or a forged one, cannot start a line of its own:
Cybertrain.logger.info("ok\n2026-11-11T12:00:00.000Z ERROR fake") writes
one line:
2026-11-14T09:30:00.120Z INFO ok\n2026-11-11T12:00:00.000Z ERROR fake
A message's one line feed at its very end is the line's own end and is not
escaped (info("done\n") writes done); a second one is. Nothing else is
escaped: a backslash stays a backslash, and tabs, other control bytes and
invalid UTF-8 are written as they are. JSON lines escape as they always
did.
The level. c.log_level in config/app.rb, or CYBERTRAIN_LOG_LEVEL
(debug, info, warn, error, fatal or none, in any case), is the
least important level written; :info by default. :none also turns off
the request log, and still writes error lines. An assignment in
config/app.rb wins over the variable. Anything else stops the boot:
error: log_level must be :debug, :info, :warn, :error, :fatal or :none (got :verbose).
JSON lines. c.log_format = :json (or CYBERTRAIN_LOG_FORMAT=json)
writes one JSON object per line instead, for a log collector:
{"time":"2026-11-14T09:30:00.132Z","level":"INFO","request_id":"3f2a9c1e-6c1d-4f0b-9a52-0d1e2f3a4b5c","tags":[],"message":"Completed 200 in 12ms"}
request_id is there only when the line has one (inside a request, or in a
job a request enqueued) and log_tags shows it; tags holds the other
tags in order. When a message (or a tag) is not valid UTF-8, every byte of
it from 0x80 up, the bytes of valid characters included, is written as the
escape \ufffd in JSON, so one bad byte turns all of that message's
non-ASCII text into replacement characters; a valid message is written as
it is, and a text line keeps the bytes as they were logged. A format other than
:text or :json stops the boot:
error: log_format must be :text or :json (got :xml). Boot's own first
line (the warning about mail_delivery_method :test outside tests)
follows the format; the warnings about an application setting whose value
is invalid (setting_int, setting_bool falling back to the default) are
logged while config/app.rb runs,
before the format is known, and are always text lines.
The level and the format apply to ./<app> server, worker and feeds,
and to registered commands, jobs and console, which refuse an unknown
one with the same message and exit status 1 before they open the database.
migrate, db and dbconsole do not apply them: their lines are text at
the info level.
Request ids
Every request gets an id, a version 4 UUID, which the action reads as
request.request_id and the response sends back as X-Request-Id, also on
the host check's 403, the https redirect of force_ssl, the error pages,
the development diagnostics page and the server's own 500. The server's 400
and 413 for a request it cannot read, answered before the application sees
it, carry none. Every line logged while the request runs carries the id as
a tag (the first, with the default log_tags), the application's own Cybertrain.logger.info lines
included, and so does a streamed body (stream do ... end, Server-Sent
Events) or a WebSocket connection the request opened, for as long as it
stays open.
Cybertrain.logger.info("export #{report.id} for #{request.request_id}")
A proxy's id is used only from a proxy the application trusts: when the
connection comes from a trusted proxy (CYBERTRAIN_TRUSTED_PROXY or
CYBERTRAIN_TRUSTED_PROXIES, the check request.client_ip makes;
The client address behind a proxy)
and its X-Request-Id is 1 to 255 ASCII letters, digits and hyphens, the
request takes that id, so the proxy's log and the application's share it.
Any other X-Request-Id, a header sent twice included, is discarded and a
new id made; it is never altered.
c.log_tags chooses the request's tags, in order: [:request_id] by
default, [:request_id, :remote_ip] to add request.client_ip (off by
default: an address is personal data), [] for none (the header is still
sent). Anything else stops the boot:
error: log_tags may hold :request_id and :remote_ip (got :user_agent).
Tags
Cybertrain.logger.tagged adds tags to every line logged on this thread
during the block, after the request's own, and takes them off when the
block ends, also when it raises; it answers the block's value. Blocks nest.
There is no form without a block.
Cybertrain.logger.tagged("billing", "order #{order.id}") do
Cybertrain.logger.info("charged 300")
# 2026-11-14T09:30:00.140Z INFO [3f2a9c1e-...] [billing] [order 7] charged 300
end
Tags belong to the thread: a thread you start begins with none, whatever the thread that started it had. The framework tags its own work the same way:
- Jobs. Each attempt's lines carry
[<JobClass> <id>](for a block job fromJob.handle, its kind), then the id of the request that enqueued it:INFO [WelcomeJob 7] [3f2a9c1e-...] welcome 1. The id is recorded in the job's row (cybertrain_jobs.request_id) whateverlog_tagssays, and shown whenlog_tagsincludes:request_id. A job enqueued by a job records its parent's id; one enqueued by a scheduled task, a feed or a command records none. - Scheduled tasks.
[schedule:<name>]while a task runs, its failure line included. - Feeds.
[feed:<name>]on every line a feed's thread logs, its callbacks' included. The supervisor's restart and give-up lines, logged on its own thread, carry no tag.
Error reporting
Cybertrain.error collects the exceptions that end a request, a job's
attempt, a scheduled task or a feed, and the ones the application reports
itself, and hands each to the subscribers the application adds and to
Sentry (or GlitchTip) when a DSN is set. It is in the core; sending to
Sentry is its own line, which require "cybertrain" loads and an
application on cybertrain/core adds to bin/<app>.rb (not to
bin/db.rb; Subsystems). cybertrain new
writes the lines of this section commented out: the variables in
.env.example, a subscriber in config/app.rb, the DSN in
config/deploy.yml and .kamal/secrets, and this line in the comment of
bin/<app>.rb:
require "cybertrain/error_reporter/sentry" # bin/<app>.rb, after require "cybertrain/core"
# Something the application rescued, with what it knows:
begin
charge(order)
rescue PaymentError => e
Cybertrain.error.report(e, context: { "order_id" => order.id })
end
# Run a block, report what it raises, go on with nil:
price = Cybertrain.error.handle(context: { "feed" => "prices" }) do
parse_price(text)
end
report(e, handled: true, context: {}) answers nil at once: the report is
delivered on the reporter's own thread. The context's keys and values become
Strings, over the context the framework adds (a job's id and attempt).
handle(context: {}) { } answers the block's value, or nil after reporting
a StandardError (or JSON::ParserError, JSON::NestingError) the block
raised; break v in its block makes handle answer v, and return
returns from the method around it. An exception outside those classes raised
in handle's block goes on to the caller. Reporting itself never raises
into the caller: if the report fails, the log says
error report failed: <Class>: <message> at error.
Subscribers. A subscriber is a class of the application's, a subclass of
Cybertrain::ErrorSubscriber that defines report(event):
# config/app.rb, above the line that subscribes it (bin/<app>.rb loads
# config/app before app/, so a class under app/ is not loaded yet here)
class AuditSubscriber < Cybertrain::ErrorSubscriber
def report(event)
Cybertrain.logger.warn("reported #{event.error_class}: #{event.error_message} at #{event.error_where}")
end
end
Cybertrain.error.subscribe(AuditSubscriber.new)
Each subscriber receives every report, on the reporter's thread, not in the
request or the job that failed. One that raises is logged at error,
error subscriber AuditSubscriber failed: <Class>: <message>, and is never
reported itself; the others still receive the event. The event is a value
of Strings:
| Reader | What |
|---|---|
error_class |
the exception's class name, namespaced ("Cybertrain::DB::Error") |
error_message |
its message |
error_where |
"ArticlesController#create", "request" (no action ran), a job's class or kind, "cybertrain mail", "job thread", "schedule:<name>", "feed:<name>", or "" |
error_request_id, error_request_method, error_request_path |
the request's, the path without its query and filtered as the request log filters it; in a job, the id of the request that enqueued it; "" outside a request |
error_environment, error_release |
Cybertrain.config.env and c.release |
error_handled |
true for report and handle, false for what the framework reported |
error_context |
a Hash{String => String}: the application's context, and job.id, job.attempt, job.max_attempts in a job |
error_backtrace |
an Array<String>, empty in a binary from cybertrain build (below) |
error_occurred_at |
a Time, from the clock |
What is reported without a line of yours:
- A request that ends in a 500, once: an exception
Cybertrain::ClientError.status_formaps to 500 or more. Never a 4xx (a bad parameter, a CSRF failure, a 404), never whatrescue_fromhandled. Should serving the error page then raise a new exception (an unreadablepublic/500.html), that one is reported too: two reports, each a real failure. - A job's failed attempts (
whereits class or kind;job.attemptandjob.max_attemptsin the context), under the default policy,retry_onordiscard_on. Inside a job the dedupe key holds the job's id as well (Delivery, below): two jobs of one kind failing alike are two reports, and an attempt of one job that fails with the class and message of its own attempt reported less than a minute before is logged but not reported again (under the default policy the first retry waits a minute, so each attempt is reported; withretry_on ..., wait: 30the attempt 30 seconds after a reported one is not). A discard is reported once.job.max_attemptsisjob_max_attemptsunder the default policy, the policy'sattempts:underretry_on, and the attempt it ended at underdiscard_on. Adeliver_latermail that fails is thecybertrain mailjob's. An exception that escapes a job thread's loop, or a database error there, iswherejob thread. - A scheduled task whose block raises, or a
crontask with no next run within five years (schedule:<name>). - A feed's
on_open,on_messageoron_tickthat raises, a feed thread that dies, and the supervisor giving up on a feed (Cybertrain::Background::FeedFailed, a name with no class behind it, with the messagefeed <name> restarted <n> times in <w> s; exiting), allfeed:<name>.
An exception raised while a streamed body (stream do ... end,
Server-Sent Events) or an upgraded WebSocket connection is written is
reported once, after the server's stream error: <Class>: <message> line,
with the request's id, method and path and the where
ArticlesController#index (stream) (request (stream) when no action
ran). A client that went away (EOFError, IOError, Errno::ECONNRESET,
Errno::EPIPE out of the writer) is logged with that line and not
reported. Not reported: a job whose kind has no handler, and, in
development, an exception the diagnostics page itself raises while it
shows one that was already reported. Every log line the framework wrote
before stays as it was; the report comes in addition.
What is never sent. The parameters, headers, cookies, the session, the
client's address and the query string are not in an event; a request's path
is filtered as the request log filters it. Only what the application puts
in context: is added. Exception messages are sent as they are, unless a
filter changes them: a message that holds personal data (an SMTP server's
rejection naming the recipient) needs one.
Filters. A filter is a subclass of Cybertrain::ErrorReporter::Filter
that defines filter_event(event), answering true to keep the event and
false to drop it. It may change the message (event.error_message =) and
the context (event.set_error_context(key, value),
event.delete_error_context(key)). This one masks simple unquoted ASCII e-mail addresses and the
other drops a class the application does not want reported:
# config/app.rb
class MaskAddresses < Cybertrain::ErrorReporter::Filter
def filter_event(event)
event.error_message = event.error_message.gsub(/[\w.+-]+@[\w-]+(\.[\w-]+)+/, "[email]")
event.delete_error_context("email")
true
end
end
class DropDeclinedCards < Cybertrain::ErrorReporter::Filter
def filter_event(event)
event.error_class != "PaymentGateway::Declined"
end
end
Cybertrain.error.add_filter(MaskAddresses.new)
Cybertrain.error.add_filter(DropDeclinedCards.new)
The pattern in MaskAddresses misses two kinds of address. One with
characters outside ASCII, such as alice@例え.テスト: Ruby's \w matches
ASCII letters, digits and _ unless a Unicode property is asked for. And
one with a quoted local part, such as "alice smith"@example.com: the pattern needs a
letter, digit, ., +, - or _ right before the @, finds none after
the closing quote, and matches nothing in it, so the whole address stays in
the message. Write a pattern for the addresses you expect, or drop the
event instead.
Filters run on the reporter's thread, in the order added, before the
subscribers and the sender; the next filter, the subscribers and Sentry see
what a filter changed. Once one drops the event, the later ones do not run
and nothing receives it (it is not counted among the queue's drops). A
filter that raises drops the event, since a filter exists to keep
something from leaving, and logs
error filter failed: <Filter class>: <Class>: <message> at error; it is
never reported. The dedupe is decided before the filters, on the
unfiltered class, message and where: a filter's rewrite does not change
which reports collapse, and an event a filter drops still holds its slot
for the minute. The framework's own ERROR log line is written before the
event exists and is not filtered. A filter alone does not make reports
queue (a subscriber or a DSN does). Filters added in config/app.rb stay
for every test; one added in a test is removed before the next.
Delivery. Reports go on a queue of 100 and one thread delivers them.
The same class, message and where (and, inside a job, the job's id) is
queued at most once a minute, so a
page that fails for every visitor with the same message makes one report a
minute (a message that carries an id, Couldn't find Article with id=5,
makes one per distinct message); when the queue
is full, a report is dropped and counted, and error reports dropped: <n>
is logged at warn at most once a minute. While there is no subscriber and
no DSN, nothing is queued and no thread starts. A delivery that fails is
logged, error delivery failed: <Class>: <message>, and the next report
starts a new thread if the old one ended. When the server, a worker or a
feeds process stops, it waits up to 2 seconds for the queued reports after
its services stopped; what is left is lost. A development rebuild waits
too, and so do a registered command, ./<app> jobs and ./<app> console
before they close the database and exit.
Sending to Sentry or GlitchTip. With the line in bin/<app>.rb, setting
CYBERTRAIN_SENTRY_DSN (or c.sentry_dsn) to the project's DSN sends each
report as a Sentry event:
export CYBERTRAIN_SENTRY_DSN=https://PUBLIC_KEY@o1.ingest.sentry.io/42
export CYBERTRAIN_RELEASE=2026.11.14-3f2a9c1 # optional: sent as the event's release
export CYBERTRAIN_ERROR_SAMPLE_RATE=0.25 # optional: send a quarter of them
- The DSN reads
http[s]://<public key>[:<secret>]@<host>[:<port>][/<path>]/<project id>(the secret, Sentry's old second half, is ignored); events go to<scheme>://<host>[:<port>][/<path>]/api/<project id>/envelope/, which Sentry and GlitchTip both accept. It holds a key: keep it in the deploy's secrets, not in a committed file. One that does not read stops the boot, naming the part and never the key:error: sentry_dsn must look like https://<public key>@<host>/<project id>: the project id must be digits. - A DSN set in a program that does not link the sender stops the boot,
and refuses a registered command,
jobsandconsolebefore they open the database (exit status 1):error: CYBERTRAIN_SENTRY_DSN is set, but this program does not link the Sentry sender: add require "cybertrain/error_reporter/sentry" to bin/<app>.rb. A sender class of your own (a subclass ofCybertrain::ErrorReporter::Sender) definessender_nameto answer a name that is not empty, or it meets the same refusal. CYBERTRAIN_ERROR_SAMPLE_RATE(c.error_sample_rate, 1.0) is the share of reports sent, 0.0 to 1.0; subscribers still see every report. The variable must be digits with an optional.digits, or the boot stops:error: error_sample_rate must be a number from 0.0 to 1.0 (got "x").CYBERTRAIN_RELEASE(c.release, empty by default) names the version or commit; it is not checked.- Each send waits at most 5 seconds once the host's name is resolved (name
resolution blocks the delivery thread and is not cut short by the
deadline, so a stalled resolver can hold a send longer), and is never
retried. A refusal is
logged at warn,
error report not sent: Sentry answered 429, and so is a failure to connect,error report not sent: <Class>: <message>. - The event carries the class and message,
whereas the transaction, the environment, the release, the request id as a tag, the request's method and path, the context asextra, and the backtrace as frames when there is one. - The call goes through
Cybertrain::HttpClientand its guard against private addresses (Outbound HTTP): a GlitchTip on a private network or on the same machine needsCYBERTRAIN_HTTP_ALLOW_PRIVATE=1, which lifts the guard for every outbound call of the application.
Backtraces. A binary from cybertrain build is a Spinel release build,
whose exceptions carry no backtrace: its reports have the class, the message
and where. In a --debug build, and under CRuby, the frames are attached.
cybertrain test sends nothing to Sentry: it empties an exported
CYBERTRAIN_SENTRY_DSN unless config/app.rb assigns one, and a test points
the DSN at a Test::SentryReceiver instead
(Testing logs and error reports).
The design: docs/design.md D86 and D88.
Upgrading logs and error reports from 0.4
- Every log line changes shape: the level in brackets at the start
(
[INFO]) gives way to the time, the level and the tags,2026-11-14T09:30:00.132Z INFO [<request id>] Completed 200 in 3ms. Anything that reads the log (a grep for[ERROR]in a script, an alert) changes with it; a test that looked for a whole line looks for the level and the message separately (" ERROR "and the message; Upgrading from 0.4). - Every response the application answers carries
X-Request-Id(not the server's own 400 and 413 for a request it cannot read). A proxy that set one has it used only when the proxy is trusted. cybertrain_jobsgains the columnrequest_id; the first process that touches the table adds it.- A text log line no longer continues on the next lines: a line feed or
carriage return in a message or a tag is written as
\nor\r. Anything that read a multi-line message from several lines reads the escapes instead. - A model column named
handlenow reads ashandle_column: an application with ausers.handlecolumn changes itsuser.handlecalls touser.handle_column(queries keep the SQL name). Columns namedrequest_idorreleasethat are not text are refused by gen (Column names). cybertrain testignores an exportedCYBERTRAIN_LOG_FORMATandCYBERTRAIN_SENTRY_DSN(tests read text lines and send nothing) unlessconfig/app.rbsets them. The password testsgenerate authenticationwrites turn JSON lines off inside theirwith_config; an application generated earlier whose tests run withlog_format :jsonruns the generator again or copies that change.require "cybertrain"now linksHttpClientand the TLS layer, through the Sentry sender.