NOTE: SOME FUNCTIONALITY EMPLOYS JAVASCRIPT WASD Features and Facilities – HTTP/2

WASD Features and Facilities

5.HTTP/2

5.1WASD HTTP/2
5.2HTTP/2 and Performance
5.3HTTP/2 Configuration
5.3.1Global Configuration
5.3.2Service Configuration
5.3.3HTTP/2 Set Rules
5.4HTTP/2 Detection
5.5HTTP/2 References

HTTP/2 is the most recent standard (RFC 7540, 2015) for implementing how HTTP is represented by, and transported between, client and server. It is not a ground-up rewrite of the established standard, HTTP/1.1 (RFC 2616, 1999). Those elements and semantics remain substantially the same. Instead HTTP/2 modifies how the data is encapsulated (framed) and transferred between agents, abstracting the complexity of this within the new protocol layer, leaving the application level largely insulated from change. As a result all existing HTTP/1.1 web-based environments should be able to continue without modification.

The focus of the protocol is on performance, in particular end-user perceived page rendering and web application responsiveness. With the original web use case being a relatively simple, single resource request-response, and early markup involving text with a few illustrative images, the single network connection, back-to-back request-response paradigm was simple to implement and worked well enough. In short time this moved to multiple network connections, each loading elements in parallel as the complexity and density of the individual elements on the pages increased, and to the introduction of HTTP/1.1 pipelining (back-to-back requests over a single connection) in an attempt to avoid request-response-request latency. Modern web documents and applications tend to have dozens of fine-grained elements that dynamically load resources based on the content of the page and/or user interaction. The single, then multiple network connections, each with its round-trip TCP connection establishment overhead and request-response blocking of resources, did not scale effectively. HTTP/2 replaces it with a single TCP connection on which multiple resources concurrently can be requested, pushed, and transferred. A more rigorous and effective implementation of the pipeline concept.

While multiplexing communication over a single network connection is a core performance technology there are other contributing elements. The framing layer uses binary tokens and parameters. The plain-text request and response headers of HTTP/1.n are replaced with tokenised, encoded and dynamically cached equivalents, commonly providing compression in excess of eighty percent. The relationship and priority of resources can be established allowing inferior resources to be delivered after or dependent on superior ones. The HTTP/2 server can send multiple responses to a single request. Known as server push it can be used to pre-load the browser (cache) with resources it has not encountered yet.

HTTP/2 has the potential to place additional load on the client and server in comparison to HTTP/1.n. One particular consideration for WASD sites is the stream concurrency setting of the HTTP/2 connection. The server specifies to the client the maximum number of concurent request-response (and server push) streams it will accept. RFC 7540 contains, "This limit is directional: it applies to the number of streams that the sender permits the receiver to create. Initially, there is no limit to this value. It is recommended that this value be no smaller than 100, so as to not unnecessarily limit parallelism." This translates to a hypothetical ten browsers connected to the site each with up to one hundred concurrent streams, or potentially one thousand active requests! Time to check those server configuration and SYSGEN parameters…

Note that HTTP/1.1 has recently been revisited with RFC 7230 family of specifications (2014) providing some clarifications and refinements on the original.

5.1WASD HTTP/2

WASD HTTP/2 implements all of the essential requirements of RFC 7540 (naturally enough). This includes the framing protocol, datagram (message) and stream management, header compression (RFC 7541), connection settings and flow control, along with HTTP/2 connection establishment and termination (TLS ALPN and HTTP upgrade). It does not ((perhaps) currently) provide server-push or stream prioritisation and dependency.

Prior to the introduction of HTTP/2, WASD's fundamental abstraction was the request, with each request interfacing directly with the network stack. With an HTTP/2 protocol connection somewhat supplanting the role of a Transmission Control Protocol (TCP) connection in HTTP/1.n, a new level of communication abstraction was required between the request processing and the network processing. It should be noted that HTTP/2 itself is transported on TCP.

Another new layer of abstraction required interfacing each protocol's request/response header formats with the underlying server processing (avoiding excessive duplication of code). HTTP/1.n has a plain-text, carriage-control separated format, while HTTP/2 has a binary, compressed, lookup-table oriented format (RFC 7541). The layer was implemented using a key-value dictionary.

The accomodations for handling both HTTP/2 and HTTP/1.1, along with related and ancilliary design and code changes, have not measurably impacted overall WASD performance, although as noted below there is a server process CPU impost associated with HTTP/2.

It's fair to say…

Reimplementing the complexities and subtleties of TCP — and adding a few of its own — up in the application layer has made HTTP/2 a significantly more complicated and less transparent protocol of HTTP/1.1 and while solving some minor annoyances with that has sacrificed the usefulness and elegance of a once readable byte-stream. Certainly added layers and associated processing to WASD, breaking the original I/O event driven design for possibly minor performance improvements.
HTTP/2 and WATCH

WATCH reports have the network item: [x]HTTP/2. This provides a detailed overview of the underlying framing and connection management exchanges between client and server. WATCH reports are available to HTTP/2 connected clients with one consideration. Due to multiplexed requests over the single network connection, WATCHing the [x]HTTP/2 item of another request in the same browser (using the same HTTP/2 connection - and there can be multiple from a single browser) is not possible (or at least more code than it's worth). The HTTP/2 activity of the WATCHing generates more report items which generate … a descent into reporting oblivion.

WASD detects when a request is initiated on the same HTTP/2 connection as an [x]HTTP/2 WATCHing client and if this sort of reporting cascade is possible (any networking group item) advises

|Time_______|Module__|Line|Item|Category__|Event...| |22:00:55.22 WATCH 1823 0004 CONNECT HTTP/2 with 192.168.1.2,62446 on https://klaatu.private,443 (0.0.0.0)| |22:00:55.22 WATCH 1454 0004 CONNECT HTTP/2 rabbit hole|
Such a request is not reported on further.

Workarounds?

5.2HTTP/2 and Performance

With HTTP/2 not modifying the fundamentals of HTTP/1.1 semantics the commonly touted payoff for all the additional complexity (in implementation) is performance. While this is often stated in terms of page rendering speeds or web application responsiveness there is another significant measure of performance - efficiency. HTTP/2 much more efficiently utilises each network (TCP) connection, as well as reducing the (time and processing) overhead of setting-up and tearing-down of each of these required for parallelism under HTTP/1.1.

Is it all worth it?  As might be expected – that depends.

There are a number of sufficiently good analyses of both the factors that affect HTTP/2 performance and the actual performance relative to HTTP/1.1. See the references section and search the Web. This section contains some observations made during WASD HTTP/2 development. All of these seem to correspond with others' observations, as well as what might reasonably be expected considering the strategies employed by the protocol.

YMMV!

After some months (and now years) accessing WASD HTTP/2 over various LANs and WANs the developer, FWIW, can't shake the perception that it seems generally more responsive in the real world. Yet interestingly …
Performance Assessment

As described in Server and Site Testing in WASD Configuration the OWASP ZAP application is integral to WASD test and exercise. It can generate an intense stream of traffic via cleartext (port 80) or TLS (port 443).

┌───────────┐                    ┌────────────┐
│           │──HTTP/1.1 clear──│            │
│ OWASP ZAP │                    │    WASD    │
│           │───HTTP/1.1 TLS───│            │
└───────────┘                    └────────────┘

Using the nghttpx proxy utility (see reference below) it is also used to exercise WASD's HTTP/2.

┌───────────┐                    ┌────────────┐                ┌────────────┐ 
│           │──HTTP/1.1 clear──│            │                │            │
│ OWASP ZAP │                    │  nghttpx   │──HTTP/2 TLS──│    WASD    │
│           │───HTTP/1.1 TLS───│            │                │            │
└───────────┘                    └────────────┘                └────────────┘

On the development bench Alpha PWS500 formal performance assessment using this is disappointing

See ‘HTTP/2 (encrypted)’ in 11.1 Simple File Request Turn-Around in section 11. Server Performance.

This may just reflect the CPU capacity of the benchmark system and that all requests are being transported through a single encrypted connection.

HTTP Report

WASD keeps track of HTTP family statistics.

After 3.8 million requests via OWASP ZAP using the above configuration over a number of spider-generated scans, one third of which were HTTP/2, one third over TLS HTTP/1.1, and another third cleartext HTTP/1.1, the following image suggests requests using HTTP/2 take approximately 50% of HTTP/1.1.

Other Assessment

The simplest tool for getting a feel for, and elementary measurement of HTTP/2 may be found in the WASD_ROOT:[EXERCISE] directory. The document DOTTY.HTML and its companion files provide a page that loads a selectable number of resources (images) in a consistent and reproducible manner. This DOTTY.HTML can be accessed via unencrypted HTTP (http://), encrypted HTTP (https://) and services configured to provide HTTP/2 or HTTP/1.1. Using these combinations with the selectable volume of resources, elementary comparisons may be made in target environments.

The Server Admin, HTTP Report (9. Server Administration) contains comparative duration and bytes-per-second minimum/maximum/average for total server HTTP/2 and HTTP/1.n requests. These cannot simply be taken at face value without some consideration of the respective load profile but under controlled conditions can provide useful metrics.

Other development and load/performance tools were employed from a Linux platform. For someone educated in computing during the (19)70s, the availability of VM technology for such purposes is just brilliant!   But you know, we were happy in those days, though we were poor.

Indispensible were

Many thanks to the developer(s) of this package.

5.3HTTP/2 Configuration

While effectively transparent to the end-user, HTTP/2 has some aspects that need to be carefully considered by the server administrator.

5.3.1Global Configuration

HTTP/2 and its features are globally enabled and configured using directives contained in the WASD_CONFIG_GLOBAL configuration file.

HTTP/2 Global Configuration
Directive Description Default
[Http2Protocol] enabled or disabled on a whole-of-server basis disabled
[Http2FrameSizeMax] maximum frame size in octets (bytes) the server is prepared to receive 16384
[Http2HeaderListMax] maximum number of octets (bytes) permitted in a received header once uncompressed 65535
[Http2HeaderTableMax] maximum number of bytes permitted in the server-end header cache 4096
[Http2PingSeconds] number of seconds between connection RTT pings 300
[Http2StreamsMax] maximum number of concurrent streams (requests) the server permits on the connection 32
[Http2InitWindowSize] initial window size (number of octets in transit) for flow-control purposes 6291456

These largely reflect settings and defaults from RFC 7540 6.5.1

5.3.2Service Configuration

Using the WASD_CONFIG_SERVICE directive [ServiceHttp2Protocol] HTTP/2 may be disabled on a per-service basis. The default is enabled if HTTP/2 is enabled globally.

5.3.3HTTP/2 Set Rules

WASD request processing rules may be used on a per-path basis to modify (some) global configuration settings and provide other WevDAV configuation. See Request Processing Configuration of WASD Configuration).

Rule Description
HTTP2=PROTOCOL=1.1 send a "HTTP_1_1_REQUIRED" error causing the client to use HTTP/1.1 (RFC 7540 section 7)
HTTP2=SEND=GOAWAY send a "GOAWAY" frame to the client resulting in it dropping the HTTP/2 connection
HTTP2=SEND=PING send a "PING" frame to the client calculating the Round Trip Time (RTT) of the connection
HTTP2=SEND=RESET send a "RST_STREAM" frame to the client causing it to drop the HTTP/2 stream (request in progress)
HTTP2=STREAMS=MAX=integer set the maximum concurrent streams on a per-path basis
HTTP2=WRITE=low|normal|high When request data is written it is queued at the specified priority, where high priority are written before normal (default) and low priority, and normal priority before low. This is only for associated stream (request) and is not a connection or whole-of-server prioritisation.

Use path SETings to prioritise some resources (e.g. CSS and JavaScript) over others (e.g. images) and potentially improve page rendering speed. Where multiple concurrent requests are being serviced on the one HTTP/2 connection this will deliver the higher priority content before others.

# WASD_CONFIG_MAP SET **.css http2=write=high SET **.js http2=write=high

5.4HTTP/2 Detection

A request using HTTP/2 may be detected during processing with the http2: conditional.

if (http2:) do this endif

See Conditional Configuration of WASD Configuration).

A script may detect HTTP/2 using the REQUEST_PROTOCOL CGI variable with the value "HTTP/2". Other protocol versions are similarly represented.

A Server-Side Includes (SSI) document can use variations on the following construct (and similar to the script suggestion immediately above) to detect and process the request protocol.

<!--#if var={request_protocol} eqs="HTTP/2" --> HTTP/2 <!--#else--> HTTP/1.n <!--#endif-->
This is demonstrated in the example SSI document:

WASD_ROOT:[EXERCISE]SHTML.SHTML

At the time of writing there is no browser-supported mechanism for a dynamic document (i.e. JavaScript) determining the underlying HTTP protocol used to access a resource. To access this information the server must be used. The suggested method, and the one employed by the DOTTY.HTML tool described above, is to provide one JavaScript source for HTTP/2 and another for everything else.

The document would contain

<script type="text/javascript" src="/example-path/http.js"></script>
and the server configuration
# WASD_CONFIG_MAP if (http2:) map /example-path/http.js /example-path/http2.js else map /example-path/http.js /example-path/http1.js endif
where each contains a minimum variable setting or similar flag detectable by the document.

5.5HTTP/2 References

The following provide a starting-point for investigating HTTP/2 (verified available at time of publication).