Outbound HTTP
An application that calls another server (a webhook, a Slack incoming
webhook, a third-party API) uses Cybertrain::HttpClient. require
"cybertrain" loads it and the TLS layer, through the
Sentry sender; an
application on cybertrain/core requires it where it is used, and the TLS
layer too when any URL is https:
require "json"
require "cybertrain/http_client"
require "cybertrain/net/tls" # only for https://; links the system libssl
client = Cybertrain::HttpClient.new(timeout: 5, deadline: 15, max_body: 1 << 20)
begin
res = client.post(ENV["SLACK_WEBHOOK_URL"], JSON.generate({ "text" => "Deployed" }),
content_type: "application/json")
Cybertrain.logger.warn("slack answered #{res.status}") unless res.success?
rescue Cybertrain::HttpClient::Error => e
Cybertrain.logger.warn("slack: #{e.message}") # names host:port, never the path
end
res = client.get("https://api.example.com/v1/status", headers: { "Authorization" => "Bearer #{token}" })
res.status # 200
res.header("ETag") # names in any case; res.headers has them lowercase
res.body # the bytes, chunked coding removed
Why it exists: Spinel's Net::HTTP does the TLS handshake and every read
after the first byte as blocking C calls, and each one holds an OS worker,
which with the default SPINEL_WORKERS=1 is the whole web server, for as
long as a stalled endpoint likes. HttpClient never blocks a worker on the
network:
- Every wait is bounded. Connecting (a non-blocking connect; Spinel's
TCPSocket.newignoresconnect_timeout:), each TLS handshake step, each read and each write waits at mosttimeoutseconds (default 5), and the whole request at mostdeadlineseconds (default 15). Past either,HttpClient::TimeoutError. Reads go through a byte buffer fed byread_nonblock, so bytes already received are never waited for. - Name resolution blocks.
getaddrinfohas no non-blocking form in the runtime: the thread making the request waits for DNS (usually milliseconds, but a dead resolver can take the system's timeout, about 5 s per attempt). The deadline counts that time but cannot interrupt it. - Non-public destinations are refused before anything is sent:
loopback (
127.0.0.0/8,::1), private (10/8,172.16/12,192.168/16,fc00::/7), link-local (169.254/16, which includes cloud metadata services,fe80::/10), unspecified (0.0.0.0/8,::), multicast, broadcast, CGNAT (100.64/10), the documentation and benchmarking ranges, and IPv4-mapped, NAT64 and 6to4 forms of those (Cybertrain::Net::Address.blocked_reason). The host name is resolved once and the addresses it resolved to are the ones checked and dialled (nothing is looked up again), so a name that resolves to a private address (an/etc/hostsentry, DNS rebinding) is refused like the literal; the TLS certificate is still checked against the name. The check is per address: a refused address is skipped, and only when every address is refused does the request fail.HttpClient::BlockedAddressErrorsays why. To call a service on your own network, or a local server in a test, setc.http_allow_private = trueinconfig/app.rb(orCYBERTRAIN_HTTP_ALLOW_PRIVATE=1), or passallow_private: truetoHttpClient.new; leave it off whenever a URL can come from a user. - Every resolved address is tried, in the order the resolver gives
them, IPv6 and IPv4 alike: when one refuses, is unreachable, resets or
does not answer within
timeout, the next is dialled, all of them withindeadline, and when every one fails the last one's error is raised. This is what makeslocalhostwork on macOS, where it resolves to::1first and a server listening on127.0.0.1refuses that.Cybertrain::Mailerdials the same way. - TLS is the openssl package's:
VERIFY_PEERagainst the operating system's trust store (SSL_CERT_FILE/SSL_CERT_DIRpoint it elsewhere; the package has noX509::Storefor a custom one), host-name checking, SNI, TLS 1.2 or newer. An https URL needs a host name: an IP address does not match a certificate's name. Withoutrequire "cybertrain/net/tls", an https request raisesHttpClient::TLSErrorsaying so;cybertrain/http_clientalone never links libssl. - A notification and API client, not a browser: HTTP/1.1 with
Connection: close, one connection per request, no keep-alive, redirects only when asked for (below), no proxy, no cookies. Responses may beContent-Length, chunked or read to the close; the headers are capped at 64 KiB and 100 lines, the body atmax_body(default 1 MiB), past whichHttpClient::TooLargeError. A response that is not HTTP/1.x, or ends early, isHttpClient::BadResponseError; a refused or reset connection, or a name that does not resolve,HttpClient::RefusedError. A connection that is reset or closed during the TLS handshake (a server that does not speak TLS) isHttpClient::TLSError("TLS handshake failed") on every platform, like a handshake OpenSSL rejects. All of them areHttpClient::Errors. - Errors never contain the path or the query, where a webhook URL
carries its secret: messages name the host and port only. A malformed URL
or a header with CR/LF is an
ArgumentErrorthat does not repeat it either.Host,Connection,Content-LengthandTransfer-Encodingare the client's;User-Agentdefaults tocybertrain/<version>.
The request runs on the calling thread, a green thread like the request
being served, so other connections keep being served while it waits. The
socket layer (cybertrain/net/socket.rb: bounded connect, buffered reads,
bounded writes) is separate from the HTTP layer so that other protocols can
use it. The design: docs/design.md D24 and D55.
Verbs, JSON and redirects
client = Cybertrain::HttpClient.new(timeout: 5, deadline: 15, redirects: 3)
res = client.request("PATCH", "https://api.example.com/v1/items/7",
body: JSON.generate({ "done" => true }), content_type: "application/json")
res = client.request("DELETE", "https://api.example.com/v1/items/7")
item = client.get("https://api.example.com/v1/items/7").json_hash # {} when the body is not a JSON object
list = client.get("https://api.example.com/v1/items").json_array # [] when it is not a JSON array
request(verb, url, body:, content_type:, headers:)sends GET, POST, PUT, PATCH, DELETE or HEAD;getandpoststay as the common forms. POST, PUT and PATCH always carry their body, DELETE one when it is not empty, GET and HEAD none (a body there is anArgumentError); a HEAD response has no body.Response#json_hashand#json_arrayparse the body, answering{}or[]for anything else.- Redirects are followed only when asked for:
redirects: 3follows up to three301,302,303,307or308with aLocation(a relative one resolves against the current URL), and the next one raisesHttpClient::TooManyRedirectsError. The default,0, returns a 3xx as it is. Every hop is checked like the first (non-public addresses refused), a hop from https to http is aBlockedAddressError, a303(and a301or302answering a POST) turns into a GET without a body, and thedeadlinecovers all the hops together.Authorization,CookieandProxy-Authorizationheaders are not sent to another origin (scheme, host or port), as browsers and curl do. ALocationthat is not a plain-ASCII http(s) URL is aBadResponseError.
Streaming and large responses
With a block, the body is not kept: each piece is yielded as it arrives (a
chunked frame, up to 1 MiB at a time, or what a read delivered) and the
Response comes back with an empty body. max_body does not apply then;
timeout and deadline do, so give a long stream a long deadline:.
get_to_file streams into a file:
client = Cybertrain::HttpClient.new(deadline: 600, redirects: 3)
client.get("https://example.com/export.ndjson") { |chunk| Importer.feed(chunk) }
res = client.get_to_file("https://example.com/dump.csv", "tmp/dump.csv")
Cybertrain.logger.info("#{res.bytes_written} bytes") if res.success?
- A 2xx body is written to
path.partas it arrives and renamed ontopathwhen it is complete, so a 78 MB download never sits in memory and a reader never sees half a file. Any other final status (a 404 page, a 503) never reaches the disk: checkres.success?,pathis left as it was. On an error the.partis removed. An empty 2xx body (a204) does replace the file with an empty one. - An exception the block raises reaches the caller unchanged.
gzip
The client asks for and inflates gzip bodies when the gzip support is
loaded: require "cybertrain" loads it, so in an application every request
sends Accept-Encoding: gzip (unless you send an Accept-Encoding header of
your own) and a Content-Encoding: gzip body comes back inflated. A program
that requires only cybertrain/http_client adds require
"cybertrain/http/gzip_ffi" (it links zlib); without it a gzip response is a
BadResponseError naming that file. A kept body is limited by max_body on
the wire and once inflated (TooLargeError past it); a streamed one is
inflated piece by piece, 64 KiB at most each, with no total limit, as a
streamed plain body. The Response's headers are the ones received:
Content-Encoding: gzip and the wire Content-Length stay while body is
the inflated text. Any other Content-Encoding (br, deflate) is a
BadResponseError; a truncated or corrupt gzip body is one too.
Server-Sent Events
Cybertrain::Sse::Parser turns the pieces of a streamed text/event-stream
response into events (it is not in require "cybertrain"):
require "cybertrain/sse"
parser = Cybertrain::Sse::Parser.new
client = Cybertrain::HttpClient.new(deadline: 180) # the whole stream must fit in the deadline
client.request("POST", "https://api.example.com/v1/chat", body: payload, content_type: "application/json") do |chunk|
parser.feed(chunk) { |event| Answer.append(event.event_data) unless event.event_data == "[DONE]" }
end
parser.finish { |event| Answer.append(event.event_data) } # a last event the stream did not end with a blank line
- Each
Sse::Eventhasevent_name("message"when the stream sends none),event_data(thedata:lines joined with a newline),event_id(the lastid:the stream sent, which carries over to later events) andevent_retry_ms(aretry:given since the previous event, 0 if none). - The parser follows the WHATWG rules: lines end at CR, LF or CRLF, cut
anywhere between two chunks; a line starting with
:is a comment; a blank line dispatches the event when it had data.finishyields an event left unterminated at the end of the stream. - It sets no limit on a line or an event: the request's
deadlinebounds the stream in time, not in memory. Binary chunks give binaryevent_data; set the encoding yourself.
Consuming WebSocket feeds
An application that listens to an outside feed (market ticks, device
telemetry, an event stream) while the server runs subscribes to it with
Cybertrain::WebSocket.subscribe. Like the HTTP client it is not in
require "cybertrain"; require it where the subscription is written, and the
TLS layer too for wss://. It loads the feeds subsystem (the feeds role,
the feed health check), so an application on cybertrain/core needs no
other line:
# config/app.rb (or a file bin/<app>.rb requires before Cybertrain::Main.run)
require "json"
require "cybertrain/websocket"
require "cybertrain/net/tls" # only for wss://; links the system libssl
Cybertrain::WebSocket.subscribe("trades", ENV["FEED_URL"],
headers: { "Authorization" => "Bearer #{ENV["FEED_TOKEN"]}" },
stale_after: 30)
.on_open { |conn| conn.send_text(JSON.generate({ "op" => "subscribe", "channel" => "trades" })) }
.on_message { |conn, text| Cybertrain::DB.with { |c| c.execute("INSERT INTO trades(raw) VALUES (?)", [text]) } }
Requiring cybertrain/websocket registers the feeds with
Cybertrain::Background, which Application#serve starts after the
scheduler and stops at shutdown, so there is nothing to call. Each
subscription gets one thread. It connects, runs on_open (send your
subscribe message there: it runs again after every reconnect), then hands
each message to on_message with the Connection (send_text,
send_binary, close, and last_opcode, :text or :binary). The
blocks take their arguments and cannot use self (Spinel has no
instance_eval).
- When it runs. The feeds start when the binary serves, like the
scheduler (Background work);
migrate,db ...,Main.commandcommands andCybertrain::Test::Clientnever start them../<app> feeds [NAME...]runs them in a process of their own, all of them or the named ones (Process roles). WithCYBERTRAIN_WORKER=0(config.worker = false) they do not run in that process (they simply do not start; the only notice is the scheduler's "run elsewhere" line, andWebSocket.start!called directly logs its own): turn it off in all but one when several processes serve, or each one receives the whole feed.subscribemust be called before the server starts; there is no adding or removing a feed while it runs. A name is unique (ArgumentError), and a feed withouton_messagefails the boot before the port is bound (error: WebSocket subscription "trades" has no on_message, exit 1). - Options.
headers:(extra handshake headers),timeout:(5 seconds per wait while connecting and per write),max_message:(1 MiB; a larger message closes the connection with 1009),ping_interval:(15) andstale_after:(45), andallow_private:(asHttpClient, below). - Waiting and reconnecting. After any failure the thread waits
min(60, 2 ** (attempt - 1))seconds, with 20 % of jitter either way: 1, 2, 4, ... 60. A connection that lasted 60 seconds starts again from 1. Every wait is bounded.WebSocket.stop!returns within about 2 seconds whatever the servers do: it waits at most 2 seconds for all the threads together, a connected feed sends a close (1000) and waits at most 1 second for the server's answer, and every feed'sconnected?is false once it returns (a thread still closing finishes on its own). - Silence. A feed can go quiet without ever closing the connection, so
the thread pings after
ping_intervalseconds without any frame (and then once per interval), and when no frame at all, a pong included, has arrived forstale_afterseconds it closes the connection and reconnects.stale_aftermust be more thanping_interval, which must be at least 1 (ArgumentErrorfromsubscribe). - Errors in your blocks. An exception in
on_messageis logged and the connection stays. An exception inon_openis logged and the connection is closed and retried, since a feed that did not subscribe is of no use. -
Timers.
.on_tick(seconds) { |conn| ... }runs a block everyseconds(an Integer, at least 1) on the feed's own thread, between its one-second waits, whether or not the feed is connected:connis theConnectionwhile connected andnilwhile not, so test it. The first call comessecondsafter the feed starts; an exception is logged and the feed goes on. Work that belongs to the feed (writing whaton_messagecollected in memory once a second, closing a minute candle) goes here, with no lock and no scheduler task. Oneon_tickper subscription.ruby BUFFER = Array.new(0) { "" } # a constant: the blocks cannot see locals Cybertrain::WebSocket.subscribe("trades", ENV["FEED_URL"]) .on_message { |conn, text| BUFFER << text } .on_tick(1) { |conn| Trades.flush(BUFFER) } # the app's code: one INSERT of the batch, then BUFFER.clear - Restarts. A supervisor thread checks every second that each feed's
thread is alive. A thread that ended anyway (an exception that is not a
StandardError, or the runtime killing it) is logged (ERROR [feed:trades] WebSocket "trades" thread ended: ...) and started again aftermin(30, 2 ** restarts)seconds, with aWARNline; its open connection is closed first. The restart count never resets, so after five restarts the wait stays at 30 seconds. In afeedsprocess, a feed that dies again afterfeed_restart_limitrestarts (10,CYBERTRAIN_FEED_RESTART_LIMIT) withinfeed_restart_windowseconds (600,CYBERTRAIN_FEED_RESTART_WINDOW) ends the process:FATAL feed trades restarted 10 times in 600 s; exiting, then every feed is stopped and the process exits with status 1, for Docker'srestart: unless-stoppedor systemd'sRestart=alwaysto start a fresh one. Aserverthat runs feeds never exits for them: it keeps restarting, and/upreportsfeed stale. An error in the supervisor itself (a database that went away while it wrote the status rows) is logged and supervision goes on. -
What is logged.
When Level Line start INFO WebSocket started: trades (wss://feed.example.com:443)connected INFO WebSocket "trades" connected to feed.example.com:443disconnected WARN WebSocket "trades" disconnected: Cybertrain::WebSocket::ClosedError: feed.example.com:443: closed by the server (1001) (retry in 2s)silent WARN WebSocket "trades" silent for 45s; reconnectinga block raised ERROR WebSocket "trades" on_message failed: KeyError: ...(oron_open)The path and the query are never logged or put in an error message: they often carry a token. Every line but the first is logged on the feed's thread and carries the tag
[feed:trades]; a block that raises, the thread's end and thefeedsprocess giving up are also reported. - Errors. All are
Cybertrain::WebSocket::Errors, with host and port only in the message. Once the handshake is done, a lost connection or a close by the server is aClosedError(close_code,close_reason; a reset or an end without a close frame is 1006); a write that times out or fails in TLS is still aTimeoutErrororTLSError. While connecting it isRefusedError,TimeoutError,TLSError,BlockedAddressError(a non-public address, refused by default exactly as in Outbound HTTP, with the sameallow_private:andhttp_allow_privatesetting) orHandshakeError(not a 101, a wrongSec-WebSocket-Accept, or the server answering with an extension or subprotocol).ProtocolError(the server broke the framing; the client sends 1002) andTooLargeErrorcan come at any time. - Health.
Cybertrain::WebSocket.feed_status("trades")answers aStatus:connected?,last_message_at(epoch seconds, 0 before the first message),reconnects,restarts(by the supervisor) andthread_alive?, read one after the other, not as one snapshot;connected?turns true when the connection is made, beforeon_openhas run. The supervisor also writes each feed's status into the database everyfeed_status_intervalseconds (10), whichGET /upand./<app> feeds statusread, in this process or any other: Feed health. - Keep
on_messageshort. The thread waits withwait_readableorsleep, so a stalled feed does not hold an OS worker while it waits. Two things do hold one: each connect attempt resolves the host name with a blockinggetaddrinfo(see "Name resolution blocks" in Outbound HTTP; withSPINEL_WORKERS=1a dead resolver stalls every request for the resolver's timeout, once per attempt per feed, and an IP address in the URL avoids the lookup), and a SQLite call insideon_messageuntil it returns, as in a scheduled task (Background work): withSPINEL_WORKERS=1a long write stalls every request. For a busy feed insert one small row, or collect messages in memory and write them in batches from aScheduler.everytask. - Not built. Extensions (
permessage-deflate) and subprotocols are not negotiated. There is no proxy, redirect or cookie support, no automatic saving of messages (browsers have an endpoint of their own: Realtime). Text frames are not checked for valid UTF-8 (a departure from RFC 6455 section 8.1): the bytes are passed on as they came. TheConnectionis valid only during the callback; sending from a request or a task is not supported.
For one connection without the reconnect loop (a script, a one-shot read) use the synchronous client:
require "cybertrain/websocket"
require "cybertrain/net/tls" # only for wss://
client = Cybertrain::WebSocket::Client.connect("wss://feed.example.com/v1", timeout: 5)
client.send_text("hello")
msg = client.receive(10) # waits at most 10 s; nil when nothing came
msg.ws_opcode # :text or :binary
msg.ws_data # the bytes
client.close
The design: docs/design.md D33 and D56.
Sending e-mail
Cybertrain::Mailer sends e-mail, plain text or with an HTML part, through
an SMTP server (your provider's submission server, or a local relay), now
(deliver) or from a job (deliver_later, below). require "cybertrain"
loads it; an application on cybertrain/core, or a program without the
framework, requires cybertrain/mailer (which needs no database:
deliver_later is in cybertrain/mailer/later, which loads jobs too).
TLS is a separate require:
require "cybertrain/mailer"
require "cybertrain/net/tls" # for starttls (the default) and tls; links the system libssl
# Settings from CYBERTRAIN_SMTP_HOST, _PORT, _USER, _PASSWORD, _FROM, _TLS:
begin
Cybertrain::Mailer.deliver(to: user.email, subject: "Welcome", body: "Hello #{user.name},\n...")
rescue Cybertrain::Mailer::Error => e
Cybertrain.logger.warn("mail: #{e.message}") # never contains the password
end
# Or explicit settings, bypassing Config:
mailer = Cybertrain::Mailer.new(host: "smtp.example.com", port: 587, tls: "starttls",
user: "app", password: ENV["SMTP_PASSWORD"].to_s,
from: "noreply@example.com", timeout: 10, deadline: 60)
mailer.deliver(to: "a@example.com, b@example.com", subject: "Report", body: report)
| Variable | Attribute | Default |
|---|---|---|
CYBERTRAIN_SMTP_HOST |
smtp_host |
"" (Mailer.new raises ArgumentError until it is set) |
CYBERTRAIN_SMTP_PORT |
smtp_port |
0: 587 for starttls, 465 for tls, 25 for none |
CYBERTRAIN_SMTP_USER |
smtp_user |
"": no AUTH |
CYBERTRAIN_SMTP_PASSWORD |
smtp_password |
"" (secret) |
CYBERTRAIN_SMTP_FROM |
smtp_from |
"" |
CYBERTRAIN_SMTP_TLS |
smtp_tls |
"starttls"; tls or none |
- TLS modes.
starttlsconnects in plain text, requires the server to offer STARTTLS (elseMailer::TLSError) and upgrades before anything else is said; bytes the server sends after its STARTTLS reply are refused.tlsspeaks TLS from the first byte (port 465). Both check the certificate againsthostas the HTTP client does (give a host name, not an IP address), and both needrequire "cybertrain/net/tls"(whichrequire "cybertrain"loads): without it,deliverraisesMailer::TLSErrorsaying so, before connecting.noneis plain text, for a relay on the same machine or a test server: with a user configured it refuses (Mailer::InsecureError, before connecting) unless the host resolves to a loopback address, and then dials only its loopback addresses. - The password is never logged or shown. It is not in the boot banner,
help, the log or any error message; it is sent only inside the AUTH PLAIN command, over TLS (or to loopback). Do notporinspectaMaileror theConfig, which hold it. - Every wait is bounded, as in Outbound HTTP: connecting, each TLS
step and each reply waits at most
timeoutseconds (default 10), the whole delivery at mostdeadlineseconds (default 60), elseMailer::TimeoutError. Name resolution blocks the calling thread. - The dialogue: the 220 greeting,
EHLO(the name ishelo:, defaultlocalhost),STARTTLSandEHLOagain,AUTH PLAINwhen a user is set (the server must offer PLAIN),MAIL FROM, oneRCPT TOper recipient,DATA,QUIT. A reply with another code is aMailer::RejectedErrorwhosesmtp_codeis the server's (421, 535, 550, ...) and whose message has the step and the server's text. The return value is the server's reply to the message ("250 2.0.0 Ok: queued as ..."). - The message:
Date,From,To,Subject,MIME-Version: 1.0,Content-Type: text/plain; charset=UTF-8(for an HTML message, see below). A non-ASCII subject is RFC 2047 encoded (=?UTF-8?B?...?=, split between characters). The body's line endings become CRLF and lines starting with.are dot-stuffed. ItsContent-Transfer-Encodingis7bitfor ASCII,8bit(withMAIL FROM ... BODY=8BITMIME) for UTF-8 when the server advertises 8BITMIME, andbase64otherwise or when a line is longer than 998 bytes. NoMessage-IDis added (submission servers add one). - What is refused with an
ArgumentError, before connecting: an address that is not a bare ASCIIlocal@domain(no display name, no<>, spaces, commas, quotes, CR or LF; at most 254 bytes), a subject with CR, LF or NUL (header injection) or over 998 bytes, a body that is not valid UTF-8 or holds a NUL, more than 100 recipients.to:takes one address or several separated by commas. - Mailers: a module in
app/mailers(gen loads them) whose method returns aCybertrain::MailMessage.new(to:, subject:, body:, html:, from:)reads as Rails does:PasswordsMailer.reset(user, url).deliver_laterqueues it,deliver_nowsends it at once.cybertrain/mailerloadsMailMessagewithdeliver_now;deliver_latercomes withcybertrain/mailer/later.cybertrain generate authenticationwrites two (Password reset) and adds both lines. - Not yet: attachments and inline images, mailer classes with
default from:and views found by name, display names,Cc/Bccheaders (send to several recipients instead), AUTH LOGIN/CRAM-MD5, SMTPUTF8 addresses, pipelining.deliverruns on the calling thread, a green thread like a request.
HTML mail
html = Cybertrain::Views.render_mail("mail/welcome", user: user) # app/views/mail/_welcome.html.erb
Cybertrain::Mailer.deliver(to: user.email, subject: "Welcome", body: "Welcome, #{user.name}!\n", html: html)
- With
html:, the message ismultipart/alternative: the plain-text part (body:) first, then the HTML part, eachcharset=UTF-8in base64 (so no line-length or dot-stuffing rule applies), between a random boundary. Mail programs show the HTML and fall back to the text. Withouthtml:the message is the plain text above, unchanged. Views.render_mail(name, locals)renders the partial (app/views/mail/_welcome.html.erb) outside a request, insideapp/views/layouts/mail.html.erbwhen that layout exists and alone otherwise;Views.render_partialrenders it without a layout. Values are HTML-escaped as on a page. A plain-text part can come from a partial too: render it yourself and pass it asbody:(it prints its values with<%== %>, since<%= %>escapes).
Sending later
Cybertrain::Mailer.deliver_later(to: user.email, subject: "Welcome", body: text, html: html)
deliver_later needs require "cybertrain/mailer/later" (or
require "cybertrain") in bin/<app>.rb and bin/db.rb. It takes
deliver's arguments, checks them at once (a refused
address or subject raises here and nothing is enqueued) and enqueues the
built-in job "cybertrain mail" with the finished message; a job thread
(Background work) sends it
with the SMTP settings in force then. Render the bodies before the call:
the job has no view context. It returns the job's id.
- Retries. A timeout, a refused connection or a
4xxreply (greylisting,421,452) is retried 60 seconds later, up to 5 attempts, then the row isfailed. A5xxreply (an address that does not exist, a refused login) discards the row at once: retrying would not change the answer. - Mistakes are discarded too. An
ArgumentErrorwhen the job runs is discarded with nothing sent: that covers a configuration mistake (noCYBERTRAIN_SMTP_HOST, no sender) as well as a payload edited in the database. Such mail is lost until someone reads the jobs page or./<app> jobs list discarded; fix the configuration, then./<app> jobs retry IDqueues it again (the same goes for a discarded535login failure). - One message, every recipient. A message to several recipients is one
SMTP transaction: when the server refuses one of them with a
5xx, the whole job is discarded and the others do not get it either. Enqueue one message per recipient when they should not depend on each other. - The message sits in the database (the job row's payload, shown on the jobs page) until the job has run, and for as long as the finished row is kept.
- Registering the job means every process that runs jobs starts its job threads, also in an application with no job class of its own.
The design: docs/design.md D27 and D57.