Class: Cybertrain::HttpClient

Inherits:
Object
  • Object
show all
Defined in:
cybertrain/http_client.rb

Overview

Sends HTTP requests (GET, POST, PUT, PATCH, DELETE, HEAD) to other servers, with bounded waits.

Examples:

A Slack incoming webhook

require "cybertrain/http_client"
require "cybertrain/net/tls"

client = Cybertrain::HttpClient.new(timeout: 5, deadline: 15)
begin
  res = client.post(ENV["SLACK_WEBHOOK_URL"], JSON.generate({ "text" => "Deployed" }),
                    content_type: "application/json")
  Cybertrain.logger.warn("slack answered #{res.status}") unless res.success?
rescue Cybertrain::HttpClient::Error => e
  Cybertrain.logger.warn("slack: #{e.message}") # host and port only
end

A large download, streamed into a file

client = Cybertrain::HttpClient.new(deadline: 600, redirects: 3)
res = client.get_to_file("https://example.com/dump.csv", "tmp/dump.csv")
Cybertrain.logger.info("#{res.bytes_written} bytes") if res.success?

Defined Under Namespace

Classes: BadResponseError, BlockedAddressError, Error, RefusedError, Response, TLSError, TimeoutError, TooLargeError, TooManyRedirectsError

Instance Attribute Summary collapse

Class Method Summary collapse

Instance Method Summary collapse

Constructor Details

#initialize(timeout: 5, deadline: 15, max_body: 1_048_576, allow_private: Cybertrain.config.http_allow_private, redirects: 0) ⇒ HttpClient

Returns a new instance of HttpClient.

Parameters:

  • timeout (Numeric) (defaults to: 5) —

    seconds for each wait: connecting, each TLS step, each read, each write. Default 5.

  • deadline (Numeric) (defaults to: 15) —

    seconds for the whole request, name resolution included. Default 15.

  • max_body (Integer) (defaults to: 1_048_576) —

    the largest response body accepted, in bytes (larger raises TooLargeError). Default 1 MiB.

  • allow_private (Boolean) (defaults to: Cybertrain.config.http_allow_private) —

    allow loopback, private and other non-public destinations. Default Config#http_allow_private.

  • redirects (Integer) (defaults to: 0) —

    how many 301 / 302 / 303 / 307 / 308 responses with a Location a request follows; past that it raises TooManyRedirectsError. Default 0: a 3xx is returned as it is. A Location that is not a plain-ASCII http(s) URL is a BadResponseError.

Raises:

  • (ArgumentError)


396
397
398
399
400
401
402
403
404
405
406
407
# File 'cybertrain/http_client.rb', line 396

def initialize(timeout: 5, deadline: 15, max_body: 1_048_576, allow_private: Cybertrain.config.http_allow_private,
               redirects: 0)
  @timeout = timeout.to_f
  @deadline = deadline.to_f
  @max_body = max_body.to_i
  @allow_private = allow_private == true
  @redirects = redirects.to_i
  raise ArgumentError, "timeout must be positive" unless @timeout > 0.0
  raise ArgumentError, "deadline must be positive" unless @deadline > 0.0
  raise ArgumentError, "max_body must be at least 0" if @max_body < 0
  raise ArgumentError, "redirects must be at least 0" if @redirects < 0
end

Instance Attribute Details

#allow_private ⇒ Boolean (readonly)

Returns whether non-public destinations are allowed.

Returns:

  • (Boolean) —

    whether non-public destinations are allowed



378
379
380
# File 'cybertrain/http_client.rb', line 378

def allow_private
  @allow_private
end

#deadline ⇒ Float (readonly)

Returns seconds the whole request may take.

Returns:

  • (Float) —

    seconds the whole request may take



372
373
374
# File 'cybertrain/http_client.rb', line 372

def deadline
  @deadline
end

#max_body ⇒ Integer (readonly)

Returns the largest body accepted, in bytes.

Returns:

  • (Integer) —

    the largest body accepted, in bytes



375
376
377
# File 'cybertrain/http_client.rb', line 375

def max_body
  @max_body
end

#redirects ⇒ Integer (readonly)

Returns how many redirects a request follows (0: none).

Returns:

  • (Integer) —

    how many redirects a request follows (0: none)



381
382
383
# File 'cybertrain/http_client.rb', line 381

def redirects
  @redirects
end

#timeout ⇒ Float (readonly)

Returns seconds any one wait may take.

Returns:

  • (Float) —

    seconds any one wait may take



369
370
371
# File 'cybertrain/http_client.rb', line 369

def timeout
  @timeout
end

Class Method Details

.gzip=(decoder) ⇒ GzipDecoder

Replaces the gzip support (cybertrain/http/gzip_ffi installs its decoder when it is required). The reader is gzip_decoder.

Parameters:

Returns:



363
364
365
# File 'cybertrain/http_client.rb', line 363

def self.gzip=(decoder)
  GzipDecoder.installed = decoder
end

.gzip_decoder ⇒ GzipDecoder

The gzip support requests use: GzipDecoder’s base (no gzip) until require "cybertrain/http/gzip_ffi" installs the zlib one. With it, every request says Accept-Encoding: gzip (unless the caller sent an Accept-Encoding of its own) and a Content-Encoding: gzip (or x-gzip) body is inflated: a kept body under max_body (the bytes on the wire and the inflated bytes alike), a streamed one piece by piece, at most 64 KiB at a time. Without it, a gzip response is a BadResponseError naming the file to require; a Content-Encoding other than gzip or identity is one either way. The Response’s headers are the ones received (Content-Encoding included).

Examples:

require "cybertrain/http_client"
require "cybertrain/http/gzip_ffi"

Cybertrain::HttpClient.gzip_decoder.gzip_ready? # => true

Returns:



354
355
356
# File 'cybertrain/http_client.rb', line 354

def self.gzip_decoder
  GzipDecoder.installed
end

.resolve_location(current, location) ⇒ String

Resolves a redirect’s Location against the URL that answered it (RFC 3986 section 5.2): an absolute URL as it is, //host/path with the current scheme, /path on the current origin, ?query on the current path, a relative path against the current path’s directory, dot segments removed. The fragment is dropped.

Examples:

Cybertrain::HttpClient.resolve_location("https://a.example/x/y?q=1", "../z")
# => "https://a.example/z"

Parameters:

  • current (String) —

    an absolute http:// or https:// URL

  • location (String) —

    the Location header’s value

Returns:

  • (String) —

    an absolute URL (not checked: a scheme the client does not take is returned as it is)

Raises:

  • (ArgumentError)


725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
# File 'cybertrain/http_client.rb', line 725

def self.resolve_location(current, location)
  base = current.to_s # NOTES rule 62
  loc = HttpClient.without_fragment(location.to_s)
  return loc if HttpClient.scheme_prefix?(loc)

  base = HttpClient.without_fragment(base)
  n = base.bytesize
  mark = base.index("://")
  raise ArgumentError, "the current URL has no scheme" if mark.nil?

  if loc.start_with?("//")
    out = String.new
    out << base.byteslice(0, mark) << ":" << loc
    return out
  end
  stop = mark + 3
  while stop < n
    b = base.getbyte(stop)
    break if b == 47 || b == 63 # / ?
    stop += 1
  end
  origin = base.byteslice(0, stop)
  rest = base.byteslice(stop, n - stop)
  query_at = rest.index("?")
  path = query_at.nil? ? rest : rest.byteslice(0, query_at)
  path = "/" if path.empty?
  out = String.new
  out << origin
  if loc.empty?
    out << path
    out << rest.byteslice(query_at, rest.bytesize - query_at) unless query_at.nil?
    return out
  end
  if loc.start_with?("?")
    out << path << loc
    return out
  end
  loc_query_at = loc.index("?")
  loc_path = loc_query_at.nil? ? loc : loc.byteslice(0, loc_query_at)
  merged = String.new
  if loc.start_with?("/")
    merged << loc_path
  else
    slash = path.rindex("/")
    merged << path.byteslice(0, slash + 1) unless slash.nil?
    merged << loc_path
  end
  out << HttpClient.remove_dot_segments(merged)
  out << loc.byteslice(loc_query_at, loc.bytesize - loc_query_at) unless loc_query_at.nil?
  out
end

Instance Method Details

#get(url, headers: NO_HEADERS) {|chunk| ... } ⇒ Response

GET url; with a block, the body is streamed to it (see #request).

Parameters:

  • url (String) —

    http:// or https://

  • headers (Hash{String => String}) (defaults to: NO_HEADERS) —

    extra request headers

Yields:

  • (chunk) —

    each piece of the body as it arrives, when a block is given

Yield Parameters:

  • chunk (String) —

    bytes of the body

Returns:

Raises:

  • (Error) —

    a subclass; ArgumentError for a URL or header the client refuses



417
418
419
# File 'cybertrain/http_client.rb', line 417

def get(url, headers: NO_HEADERS, &block)
  request("GET", url, headers: headers, &block)
end

#get_to_file(url, path, headers: NO_HEADERS) ⇒ Response

GETs url into the file path: a 2xx body is written to path + ".part" as it arrives (never held in memory, max_body does not apply) and renamed onto path when the response is complete. The body of any other final status (a 404 page, a 503) never reaches the disk: it is read and counted, the .part is removed and path is left as it was; check Cybertrain::HttpClient::Response#success?. On any error (or an interrupt) the .part file is removed and path is left as it was.

Parameters:

  • url (String) —

    http:// or https://

  • path (String) —

    the file to write; its directory must exist

  • headers (Hash{String => String}) (defaults to: NO_HEADERS) —

    extra request headers

Returns:

Raises:

  • (Error) —

    a subclass; SystemCallError when the file cannot be written; ArgumentError for a URL or header the client refuses



497
498
499
500
501
502
503
504
# File 'cybertrain/http_client.rb', line 497

def get_to_file(url, path, headers: NO_HEADERS)
  final = path.to_s
  part = "#{final}.part"
  file = File.open(part, "wb")
  sink = FileChunkSink.new(file)
  res = fetch_into(url.to_s, final, part, file, sink, headers)
  Response.new(res.status, res.headers, "", sink.sink_bytes)
end

#post(url, body, content_type: "application/json", headers: NO_HEADERS) {|chunk| ... } ⇒ Response

POST body to url; with a block, the response body is streamed to it (see #request).

Parameters:

  • url (String) —

    http:// or https://

  • body (String)
  • content_type (String) (defaults to: "application/json") —

    the Content-Type header

  • headers (Hash{String => String}) (defaults to: NO_HEADERS) —

    extra request headers

Yields:

  • (chunk) —

    each piece of the body as it arrives, when a block is given

Yield Parameters:

  • chunk (String) —

    bytes of the body

Returns:

Raises:

  • (Error) —

    a subclass; ArgumentError for a URL or header the client refuses



432
433
434
# File 'cybertrain/http_client.rb', line 432

def post(url, body, content_type: "application/json", headers: NO_HEADERS, &block)
  request("POST", url, body: body, content_type: content_type, headers: headers, &block)
end

#request(verb, url, body: "", content_type: "", headers: NO_HEADERS) {|chunk| ... } ⇒ Response

Sends verb to url. POST, PUT and PATCH always carry body (and its Content-Length, and content_type unless empty); DELETE carries one when it is not empty; GET and HEAD carry none. Content-Type accompanies a body only: a content_type: given to a request that sends none (GET, HEAD, a DELETE with an empty body) is not sent. A HEAD response has no body.

With a block, the response body is not kept: each piece is yielded as it arrives (a chunked frame, up to 1 MiB at a time, or what each read delivered) and the Response’s body is "". max_body does not apply then; timeout and deadline do, so give a long stream a long deadline:. The body of a redirect that is followed is never yielded.

Examples:

Server-Sent Events from an API

client = Cybertrain::HttpClient.new(deadline: 180)
client.request("POST", url, body: payload, content_type: "application/json") do |chunk|
  parser.feed(chunk) { |event| handle(event) }
end

Parameters:

  • verb (String) —

    “GET”, “POST”, “PUT”, “PATCH”, “DELETE” or “HEAD”

  • url (String) —

    http:// or https://

  • body (String) (defaults to: "")
  • content_type (String) (defaults to: "") —

    the Content-Type header of the body

  • headers (Hash{String => String}) (defaults to: NO_HEADERS) —

    extra request headers

Yields:

  • (chunk) —

    each piece of the body as it arrives, when a block is given

Yield Parameters:

  • chunk (String) —

    bytes of the body

Returns:

Raises:

  • (Error) —

    a subclass; ArgumentError for an unknown verb, a body on GET or HEAD, or a URL or header the client refuses



464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
# File 'cybertrain/http_client.rb', line 464

def request(verb, url, body: "", content_type: "", headers: NO_HEADERS, &block)
  name = verb.to_s
  unless VERBS.include?(name)
    message = "unknown HTTP verb #{name.inspect}: GET, POST, PUT, PATCH, DELETE or HEAD"
    raise ArgumentError, message
  end
  data = body.to_s
  if !data.empty? && (name == "GET" || name == "HEAD")
    message = "a #{name} request carries no body"
    raise ArgumentError, message
  end
  if block.nil?
    perform(name, url.to_s, data, content_type.to_s, headers, ChunkSink.new)
  else
    perform(name, url.to_s, data, content_type.to_s, headers, BlockChunkSink.new(block))
  end
end