tornado.httpclient — Asynchronous HTTP client

tornado.httpclient — Asynchronous HTTP client

`tornado.httpclient` — Asynchronous HTTP client

Blocking and non-blocking HTTP client interfaces.

This module defines a common interface shared by two implementations, simple_httpclient and curl_httpclient. Applications may either instantiate their chosen implementation class directly or use the AsyncHTTPClient class from this module, which selects an implementation that can be overridden with the AsyncHTTPClient.configure method.

The default implementation is simple_httpclient, and this is expected to be suitable for most users’ needs. However, some applications may wish to switch to curl_httpclient for reasons such as the following:

curl_httpclient has some features not found in simple_httpclient, including support for HTTP proxies and the ability to use a specified network interface.
curl_httpclient is more likely to be compatible with sites that are not-quite-compliant with the HTTP spec, or sites that use little-exercised features of HTTP.
curl_httpclient is faster.

Note that if you are using curl_httpclient, it is highly recommended that you use a recent version of libcurl and pycurl. Currently the minimum supported version of libcurl is 7.22.0, and the minimum version of pycurl is 7.18.2. It is highly recommended that your libcurl installation is built with asynchronous DNS resolver (threaded or c-ares), otherwise you may encounter various problems with request timeouts (for more information, see <http://curl.haxx.se/libcurl/c/curl_easy_setopt.html#CURLOPTCONNECTTIMEOUTMS> and comments in curl_httpclient.py).

To select curl_httpclient, call AsyncHTTPClient.configure at startup:

AsyncHTTPClient.configure("tornado.curl_httpclient.CurlAsyncHTTPClient")

HTTP client interfaces

class tornado.httpclient.HTTPClient(async_client_class:Type[AsyncHTTPClient] | None = None, __ kwargs: Any)[source]

A blocking HTTP client.

This interface is provided to make it easier to share code between synchronous and asynchronous applications. Applications that are running an IOLoop must use AsyncHTTPClient instead.

Typical usage looks like this:

http_client = httpclient.HTTPClient()
try:
    response = http_client.fetch("http://www.google.com/")
    print(response.body)
except httpclient.HTTPError as e:
    # HTTPError is raised for non-200 responses; the response
    # can be found in e.response.
    print("Error: " + str(e))
except Exception as e:
    # Other errors are possible, such as IOError.
    print("Error: " + str(e))
http_client.close()

Changed in version 5.0: Due to limitations in asyncio, it is no longer possible to use the synchronous HTTPClient while an IOLoop is running. Use AsyncHTTPClient instead.

close() → None [source]

Closes the HTTPClient, freeing any resources used.

fetch(request:HTTPRequest | str, __ kwargs: Any) → HTTPResponse[source]

Executes a request, returning an HTTPResponse.

The request may be either a string URL or an HTTPRequest object. If it is a string, we construct an HTTPRequest using any additional kwargs: HTTPRequest(request, **kwargs)

If an error occurs during the fetch, we raise an HTTPError unless the raise_error keyword argument is set to False.

class tornado.httpclient.AsyncHTTPClient(force_instance:bool = False, __ kwargs: Any)[source]

An non-blocking HTTP client.

Example usage:

async def f():
    http_client = AsyncHTTPClient()
    try:
        response = await http_client.fetch("http://www.google.com")
    except Exception as e:
        print("Error: %s" % e)
    else:
        print(response.body)

The constructor for this class is magic in several respects: It actually creates an instance of an implementation-specific subclass, and instances are reused as a kind of pseudo-singleton (one per IOLoop). The keyword argument force_instance=True can be used to suppress this singleton behavior. Unless force_instance=True is used, no arguments should be passed to the AsyncHTTPClient constructor. The implementation subclass as well as arguments to its constructor can be set with the static method configure()

All AsyncHTTPClient implementations support a defaults keyword argument, which can be used to set default values for HTTPRequest attributes. For example:

AsyncHTTPClient.configure(
    None, defaults=dict(user_agent="MyUserAgent"))
## or with force_instance:
client = AsyncHTTPClient(force_instance=True,
    defaults=dict(user_agent="MyUserAgent"))

Changed in version 5.0: The io_loop argument (deprecated since version 4.1) has been removed.

close() → None [source]

Destroys this HTTP client, freeing any file descriptors used.

This method is not needed in normal use due to the way that AsyncHTTPClient objects are transparently reused. close() is generally only necessary when either the IOLoop is also being closed, or the force_instance=True argument was used when creating the AsyncHTTPClient.

No other methods may be called on the AsyncHTTPClient after close().

fetch(request:str | HTTPRequest, raise_error:bool = True, __ kwargs: Any) → Future[HTTPResponse][source]

Executes a request, asynchronously returning an HTTPResponse.

The request may be either a string URL or an HTTPRequest object. If it is a string, we construct an HTTPRequest using any additional kwargs: HTTPRequest(request, **kwargs)

This method returns a Future whose result is an HTTPResponse. By default, the Future will raise an HTTPError if the request returned a non-200 response code (other errors may also be raised if the server could not be contacted). Instead, if raise_error is set to False, the response will always be returned regardless of the response code.

If a callback is given, it will be invoked with the HTTPResponse. In the callback interface, HTTPError is not automatically raised. Instead, you must check the response’s error attribute or call its rethrow method.

Changed in version 6.0: The callback argument was removed. Use the returned Future instead.

The raise_error=False argument only affects the HTTPError raised when a non-200 response code is used, instead of suppressing all errors.

classmethod configure(impl:None | str | Type[Configurable], __ kwargs: Any) → None [source]

Configures the AsyncHTTPClient subclass to use.

AsyncHTTPClient() actually creates an instance of a subclass. This method may be called with either a class object or the fully-qualified name of such a class (or None to use the default, SimpleAsyncHTTPClient)

If additional keyword arguments are given, they will be passed to the constructor of each subclass instance created. The keyword argument max_clients determines the maximum number of simultaneous fetch() operations that can execute in parallel on each IOLoop. Additional arguments may be supported depending on the implementation class in use.

Example:

AsyncHTTPClient.configure("tornado.curl_httpclient.CurlAsyncHTTPClient")

Request objects

HTTP client request object.

All parameters except url are optional.

Parameters:
- url (str) – URL to fetch
- method (str) – HTTP method, e.g. “GET” or “POST”
- headers (HTTPHeaders or dict) – Additional HTTP headers to pass on the request
- body (str or bytes) – HTTP request body as a string (byte or unicode; if unicode the utf-8 encoding will be used)
- body_producer (collections.abc.Callable) – Callable used for lazy/asynchronous request bodies. It is called with one argument, a write function, and should return a Future. It should call the write function with new data as it becomes available. The write function returns a Future which can be used for flow control. Only one of body and body_producer may be specified. body_producer is not supported on curl_httpclient. When using body_producer it is recommended to pass a Content-Length in the headers as otherwise chunked encoding will be used, and many servers do not support chunked encoding on requests. New in Tornado 4.0
- auth_username (str) – Username for HTTP authentication
- auth_password (str) – Password for HTTP authentication
- auth_mode (str) – Authentication mode; default is “basic”. Allowed values are implementation-defined; curl_httpclient supports “basic” and “digest”; simple_httpclient only supports “basic”
- connect_timeout (float) – Timeout for initial connection in seconds, default 20 seconds (0 means no timeout)
- request_timeout (float) – Timeout for entire request in seconds, default 20 seconds (0 means no timeout)
- if_modified_since (datetime or float) – Timestamp for If-Modified-Since header
- follow_redirects (bool) – Should redirects be followed automatically or return the 3xx response? Default True.
- max_redirects (int) – Limit for follow_redirects, default 5.
- user_agent (str) – String to send as User-Agent header
- decompress_response (bool) – Request a compressed response from the server and decompress it after downloading. Default is True. New in Tornado 4.0.
- use_gzip (bool) – Deprecated alias for decompress_response since Tornado 4.0.
- network_interface (str) – Network interface or source IP to use for request. See curl_httpclient note below.
- streaming_callback (collections.abc.Callable) – If set, streaming_callback will be run with each chunk of data as it is received, and HTTPResponse.body and HTTPResponse.buffer will be empty in the final response.
- header_callback (collections.abc.Callable) – If set, header_callback will be run with each header line as it is received (including the first line, e.g. HTTP/1.0 200 OK\r\n, and a final line containing only \r\n. All lines include the trailing newline characters). HTTPResponse.headers will be empty in the final response. This is most useful in conjunction with streaming_callback, because it’s the only way to get access to header data while the request is in progress.
- prepare_curl_callback (collections.abc.Callable) – If set, will be called with a pycurl.Curl object to allow the application to make additional setopt calls.
- proxy_host (str) – HTTP proxy hostname. To use proxies, proxy_host and proxy_port must be set; proxy_username, proxy_pass and proxy_auth_mode are optional. Proxies are currently only supported with curl_httpclient.
- proxy_port (int) – HTTP proxy port
- proxy_username (str) – HTTP proxy username
- proxy_password (str) – HTTP proxy password
- proxy_auth_mode (str) – HTTP proxy Authentication mode; default is “basic”. supports “basic” and “digest”
- allow_nonstandard_methods (bool) – Allow unknown values for method argument? Default is False.
- validate_cert (bool) – For HTTPS requests, validate the server’s certificate? Default is True.
- ca_certs (str) – filename of CA certificates in PEM format, or None to use defaults. See note below when used with curl_httpclient.
- client_key (str) – Filename for client SSL key, if any. See note below when used with curl_httpclient.
- client_cert (str) – Filename for client SSL certificate, if any. See note below when used with curl_httpclient.
- ssl_options (ssl.SSLContext) – ssl.SSLContext object for use in simple_httpclient (unsupported by curl_httpclient). Overrides validate_cert, ca_certs, client_key, and client_cert.
- allow_ipv6 (bool) – Use IPv6 when available? Default is True.
- expect_100_continue (bool) – If true, send the Expect: 100-continue header and wait for a continue response before sending the request body. Only supported with simple_httpclient.

Note

When using curl_httpclient certain options may be inherited by subsequent fetches because pycurl does not allow them to be cleanly reset. This applies to the ca_certs, client_key, client_cert, and network_interface arguments. If you use these options, you should pass them on every request (you don’t have to always use the same values, but it’s not possible to mix requests that specify these options with ones that use the defaults).

Added in version 3.1: The auth_mode argument.

Added in version 4.0: The body_producer and expect_100_continue arguments.

Added in version 4.2: The ssl_options argument.

Added in version 4.5: The proxy_auth_mode argument.

Response objects

HTTP Response object.

Attributes:

request: HTTPRequest object
code: numeric HTTP status code, e.g. 200 or 404
reason: human-readable reason phrase describing the status code
headers: tornado.httputil.HTTPHeaders object
effective_url: final location of the resource after following any redirects
buffer: cStringIO object for response body
body: response body as bytes (created on demand from self.buffer)
error: Exception object, if any
request_time: seconds from request start to finish. Includes all network operations from DNS resolution to receiving the last byte of data. Does not include time spent in the queue (due to the max_clients option). If redirects were followed, only includes the final request.
start_time: Time at which the HTTP operation started, based on time.time (not the monotonic clock used by IOLoop.time). May be None if the request timed out while in the queue.
time_info: dictionary of diagnostic timing information from the request. Available data are subject to change, but currently uses timings available from <http://curl.haxx.se/libcurl/c/curl_easy_getinfo.html>, plus queue, which is the delay (if any) introduced by waiting for a slot under AsyncHTTPClient’s max_clients setting.

Added in version 5.1: Added the start_time attribute.

Changed in version 5.1: The request_time attribute previously included time spent in the queue for simple_httpclient, but not in curl_httpclient. Now queueing time is excluded in both implementations. request_time is now more accurate for curl_httpclient because it uses a monotonic clock when available.

rethrow() → None [source]

If there was an error on the request, raise an HTTPError.

Exceptions

exception tornado.httpclient.HTTPClientError(code:int, message:str | None = None, response:HTTPResponse | None = None)[source]

Exception thrown for an unsuccessful HTTP request.

Attributes:

code - HTTP error integer error code, e.g. 404. Error code 599 is used when no HTTP response was received, e.g. for a timeout.
response - HTTPResponse object, if any.

Note that if follow_redirects is False, redirects become HTTPErrors, and you can look at error.response.headers['Location'] to see the destination of the redirect.

Changed in version 5.1: Renamed from HTTPError to HTTPClientError to avoid collisions with tornado.web.HTTPError. The name tornado.httpclient.HTTPError remains as an alias.

exception tornado.httpclient.HTTPError

Alias for HTTPClientError.

Command-line interface

This module provides a simple command-line interface to fetch a url using Tornado’s HTTP client. Example usage:

## Fetch the url and print its body
python -m tornado.httpclient http://www.google.com

## Just print the headers
python -m tornado.httpclient --print_headers --print_body=false http://www.google.com

Implementations

class tornado.simple_httpclient.SimpleAsyncHTTPClient(force_instance:bool = False, __ kwargs: Any)[source]

Non-blocking HTTP client with no external dependencies.

This class implements an HTTP 1.1 client on top of Tornado’s IOStreams. Some features found in the curl-based AsyncHTTPClient are not yet supported. In particular, proxies are not supported, connections are not reused, and callers cannot select the network interface to be used.

This implementation supports the following arguments, which can be passed to configure() to control the global singleton, or to the constructor when force_instance=True.

max_clients is the number of concurrent requests that can be in progress; when this limit is reached additional requests will be queued. Note that time spent waiting in this queue still counts against the request_timeout.

defaults is a dict of parameters that will be used as defaults on all HTTPRequest objects submitted to this client.

hostname_mapping is a dictionary mapping hostnames to IP addresses. It can be used to make local DNS changes when modifying system-wide settings like /etc/hosts is not possible or desirable (e.g. in unittests). resolver is similar, but using the Resolver interface instead of a simple mapping.

max_buffer_size (default 100MB) is the number of bytes that can be read into memory at once. max_body_size (defaults to max_buffer_size) is the largest response body that the client will accept. Without a streaming_callback, the smaller of these two limits applies; with a streaming_callback only max_body_size does.

Changed in version 4.2: Added the max_body_size argument.

class tornado.curl_httpclient.CurlAsyncHTTPClient(max_clients=10 , defaults=None)

libcurl-based HTTP client.

This implementation supports the following arguments, which can be passed to configure() to control the global singleton, or to the constructor when force_instance=True.

max_clients is the number of concurrent requests that can be in progress; when this limit is reached additional requests will be queued.

defaults is a dict of parameters that will be used as defaults on all HTTPRequest objects submitted to this client.

Example Code

A simple webspider shows how to fetch URLs concurrently.
The file uploader demo uses either HTTP POST or HTTP PUT to upload files to a server.