|1Utilities and Facilities| |^ Foreign commands for external utilities (and the HTTPD control functionality) will need to be assigned from the adminstration users' LOGIN.COM either explicitly or by calling the |link%|/wasd_root/example/wasdverbs.com|WASD_ROOT:[EXAMPLE]WASDVERBS.COM| procedure. |code| $ AB == "$WASD_EXE:AB" $ HTTPD == "$WASD_EXE:HTTPD" $ HTTPDMON == "$WASD_EXE:HTTPDMON" $ MD5DIGEST == "$WASD_EXE:MD5DIGEST" $ QDLOGSTATS == "$WASD_EXE:QDLOGSTATS" $ SECHAN == "$WASD_EXE:SECHAN" $ STREAMLF == "@WASD_EXE:STREAMLF" $ WB == "$WASD_EXE:WB" |!code| |2Echo Facility| |^ Ever had to go to extraordinary lengths to find out exactly what your browser is sending to the server? The server provides a request echo facility. This merely returns the complete request as a plain-text document. This can be used for for checking the request header lines being provided by the browser, and can be valuable in the diagnosis of POSTed forms, etc. |^ This facility must be enabled through a mapping rule entry. |code| script /echo/* /echo/* |!code| |^ It may then be used with any request merely by inserting "/echo" at the start of the path, as in the following example. |code| http://www.example.com/echo/wasd_root/ |!code| |2Hiss Facility| |^ The |/hiss| facility provides a response stream made up of random alpha-numeric characters (a sort of alpha-numeric white-noise). No response header is generated and the stream will continue (by default) up to one megabyte of output, or until the client closes the connection. This maximum may be controlled my appending an integer representing the number of kilobytes maximum to the mapping. This facility must be enabled through a mapping rule entry and may then be used for specific requests. |code| map /**.dll* /hiss/64/*.dll* map /**/system32/* /hiss/64/*/system32/* map /**default.ida* /hiss/64/*default.ida* script /hiss/* /hiss/* |!code| |^ Usage details are described in |link%|../config/##Security Considerations++of++WASD Configuration||). |2Stream Facility| |^ The |/stream| facility provides a quantified or unlimited response stream of printable or binary octets. It is intended as a light-weight data source delivering content at the maximum throughput capable by the server and platform. This can be used as a test source or for end-to-end metrics. This facility must be enabled through a mapping rule. |code| script /stream/* /stream/* |!code| |^ It may then be used to generate streams of data with various characteristics and sizes by including parameters in the URL. |bullet| |item| Without parameters it produces a text/plain response header with unlimited stream of random 8 bit printable and newline characters. The stream ceases at client disconnection. |code| http://www.example.com/stream/ |!code| |item| With an integer parameter the stream ceases when the response has delivered that many kilobytes (1024) of characters. |code| http://www.example.com/stream/50/ |!code| |item| A 100 kilobyte stream of repeated 80 column, newline terminated characters in the range "+" (0x2b) to "z" (ox7a). Intended to provide an entirely predictable sequence for testing purposes. |code| http://www.example.com/stream/text:100/ |!code| |item| The following produces an application/binary response header with unlimited stream of random octets. |code| http://www.example.com/stream/binary/ |!code| |item| One megabyte of random octets. |code| http://www.example.com/stream/binary:1024/ |!code| |item| An unlimited stream of octets cycling from 0x00 to 0xff. Intended to provide an entirely predictable sequence for testing purposes. |code| http://www.example.com/stream/octets/ |!code| |!bullet| |2Where Facility| |^ Need to locate where VMS has the HTTPd files? This simple facility maps the supplied path then parses it to obtain a resulting VMS file specification. |*This does not demonstrate whether the path actually exists!| |^ This facility must be enabled through a mapping rule entry. |code| script /where/* /where/* |!code| |^ It may then be used with any request merely by inserting "/where" at the start of the path, as in the following example. |code| http://www.example.com/where/wasd_root/ |!code| |2Xray Facility| |^ The Xray facility returns a request's complete response, |*both header and body||, as a plain text document. Being able to |/see| the internals of the response header as well as the contents of the body rendered in plain text can often be valuable when developing scripts, etc. |^ This facility must be enabled through a mapping rule entry. |code| script /Xray/* /Xray/* |!code| |^ It may then be used with any request merely by inserting "/xray" at the start of the path, as in the following example. |code| http://www.example.com/xray/wasd_root/ |!code| |2CALogs| |^ The Consolidate Access LOGS utility (pronounced similar to the breakfast cereal brand :-) merges multiple HTTP server common and combined format access logs into a single log file with records in time-order. Due to the granularity of HTTP server entry timestamps (one second) the records are sorted to the one second but not within the one second. |^ It uses RMS and the VMS sort-merge routines to provide the basic consolidation functionality. An RMS search uses the supplied wildcard log file specification. Matching files are opened and each record read. The date/time field is parsed and a binary timestamp generated. Records with formats or date/time fields that do not make sense to the utility are discarded. When all files have been processed the sort-merge is performed using the timestamp as the key. The sorted records are then written to the specified output file. |^ |*.$ calogs [] []| |^ |0Parameters and Qualifiers| |table| |~_ |: Parameter|: Description |~ |~#* |. /HELP |. basic usage information |~ |. /NOPROXY |. discard proxy service records |~ |. /NOWASD |. discard WASD server status/timestamp entries |~ |. /OUTPUT= |. alternate method of specifying merged file name |~ |. /PROXY |. discard non-proxy service records |~ |. /QUIET |. no messages apart from errors |~ |. /VERBOSE |. per-file progress messages |~ |. /VERSION |. display the utility version and copyright message |!table| |0Usage Examples| |code| $ CALOGS == "$WASD_EXE:CALOGS" $ CALOGS WASD_LOGS:*200205*.LOG 2002_MAY.LOG $ CALOGS /VERBOSE WASD_LOGS: $ CALOGS /NOWASD WASD_LOGS:*200206*.LOG_* /OUTPUT=2002_JUNE.LOG $ CALOGS /PROXY /NOWASD WASD_LOGS:*2002*.LOG 2002_PROXY.LOG |!code| |2CSPreport[er]| |^ Content Security Policy (CSP) is an added layer of security that helps to detect and mitigate certain types of attacks, including Cross Site Scripting (XSS) and data injection attacks. |^+ |%https://en.wikipedia.org/wiki/Content_Security_Policy| |^- |%https://developer.mozilla.org/en-US/docs/Web/HTTP/CSP| |^ WASD provides CSP support using mapping rules. See |link%|../config/##Content Security Policy (CSP)++of++WASD Configuration||. |^ When POSTed to, this utility appends a timestamp and CSP report JSON to the file specified by the CSPREPORT_FILE logical name. This file must be located somewhere the scripting account has read+write access to. When accessed using a GET the utility accesses the stored CSP reports and returns a formatted HTML report listing each. GET requests (reporting) must be subject to authentication and authorisation. |^ For further infomation check the descriptive prologue in the |link%|/wasd_root/src/utils/cspreport.c|WASD_ROOT:[SRC.UTILS]CSPREPORT.C| source code. |2HTAdmin| |^ The HTAdmin utility assists in with the command-line maintenance of $HTA authorization databases. See |link%|../config/##Authorization Configuration (Basics)++of++WASD Configuration|| and |link|Authentication and Authorization||. |^ |*$ htadmin [] []| |0Parameters and Qualifiers| |table| |~_ |: Parameter|: Description |~ |~#* |. /ADD |. add a new record |~ |. /CONFIRM |. confirm deletion of database |~ |. /CONTACT="" |. contact information for record |~ |. /CREATE |. create a new database |~ |. /CSV[=TAB\|char] |. comma-separated listing (optional character) |~ |. /DATABASE= |. database name (or as command-line parameter) |~ |. /DELETE |. delete a database or username record from a database |~ |. /DISABLED |. username record is disabled (cannot be used) |~ |. /EMAIL="" |. email address for record |~ |. /ENABLED |. username record is enabled (can be used) |~ |. /FULL |. listing showing full details |~ |. /GENERATE |. generate a six character password |~ |. /HELP |. basic usage information |~ |. /[NO]HTTPS |. synonym for /SSL |~ |. /LIST |. listing (brief by default, see /FULL and /CSV) |~ |. /MODIFY |. synonym for /UPDATE |~ |. /NAME="" |. full name for username record |~ |. /OUTPUT= |. alternate output for database listing |~ |. /PASSWORD[=] |. username record password (prompts if not supplied) |~ |. /PIN |. generate four-digit "PIN number" for password |~ |. /[NO]READ |. username can/can't read |~ |. /SORT[=] |. sort the records into a new/another database |~ |. /[NO]SSL |. user can only authenticate via SSL ("https:") |~ |. /[NO]WRITE |. username can/can't write |~ |. /UPDATE |. update an existing username record |~ |. /USER= |. username |~ |. /VERSION |. display version of HTADMIN |!table| |0Usage Examples| |bullet| |item| To create a new database named EXAMPLE.$HTA (in the current directory) |code| $ HTADMIN EXAMPLE /CREATE |!code| |item| Delete an existing database |code| $ HTADMIN EXAMPLE /DELETE /CONFIRM |!code| |item| List (briefly) the records |code| $ HTADMIN EXAMPLE |!code| |item| List (briefly) the specific user record DANIEL |code| $ HTADMIN EXAMPLE DANIEL |!code| |item| List all detail (132 colums) of the specified user record |code| $ HTADMIN EXAMPLE DANIEL /FULL |!code| |item| To add the new record DANIEL with default read access |code| $ HTADMIN EXAMPLE DANIEL /ADD /NAME="Mark Daniel" |!code| |item| Add the new record DANIEL with contact details and read+write access |code| $ HTADMIN EXAMPLE DANIEL /ADD /WRITE /CONTACT="Postal Address" |!code| |item| Add the new record DANIEL and be prompted for a password, or to specify the password on the command-line, or have the utility generate a password or four-digit PIN style password (which is displayed after the record is sucessfully added) |code| $ HTADMIN EXAMPLE DANIEL /ADD /NAME="Mark Daniel" /PASSWORD $ HTADMIN EXAMPLE DANIEL /ADD /NAME="Mark Daniel" /PASSWORD=cher10s $ HTADMIN EXAMPLE DANIEL /ADD /NAME="Mark Daniel" /GENERATE $ HTADMIN EXAMPLE DANIEL /ADD /NAME="Mark Daniel" /PIN |!code| |item| To update an existing record |code| $ HTADMIN EXAMPLE DANIEL /UPDATE /EMAIL="Mark.Daniel@wasd.vsm.com.au" |!code| |item| Update the specified record's password (interactively) then to generate a four digit PIN for a password (which is then displayed) |code| $ HTADMIN EXAMPLE DANIEL /UPDATE /PASSWORD $ HTADMIN EXAMPLE DANIEL /UPDATE /GENERATE $ HTADMIN EXAMPLE DANIEL /UPDATE /PIN |!code| |item| Disable then enable an existing user record without changing anything else |code| $ HTADMIN EXAMPLE DANIEL /UPDATE /DISABLE $ HTADMIN EXAMPLE DANIEL /UPDATE /ENABLE |!code| |item| To list the entire database, first briefly, then in 132 column mode (with all detail), then finally as a comma-separated listing |code| $ HTADMIN EXAMPLE $ HTADMIN EXAMPLE /FULL $ HTADMIN EXAMPLE /CSV |!code| |!bullet| |0Sort Details| |^ The /SORT qualifier sorts the current database records according to the /SORT= parameters. It can be used with the /LIST qualifier to produce ordered reports or will output the records into another authentication file. By default it sorts ascending by username. Qualifier parameters allow a sort by DATE or COUNT. Each of these allows the further specification of which date or count; ACCESS, CHANGE or FAILURE. |bullet| |item| Generating a listing with specified order |code| $ HTADMIN EXAMPLE /LIST /SORT=DATE=ACCESS $ HTADMIN EXAMPLE /LIST /SORT=COUNT=FAILURE /OUTPUT=EXAMPLE.LIS |!code| |item| Sort descending by username into a higher version of EXAMPLE.$HTA |code| $ HTADMIN EXAMPLE /SORT |!code| |item| To sort by username into another .$HTA file |code| $ HTADMIN EXAMPLE /SORT /OUTPUT=ANOTHER |!code| |item| List by most-recently accessed |code| $ HTADMIN EXAMPLE /LIST /SORT=DATE |!code| |item| List by most-recently failed to authenticate |code| $ HTADMIN EXAMPLE /LIST /SORT=DATE=FAILURE |!code| |item| Sort file into order by most frequently authenticated (accessed) |code| $ HTADMIN EXAMPLE /SORT=COUNT |!code| |!bullet| |2HTTPd Monitor| |^ The HTTP server may be monitored in real-time using the HTTPDMON utility. |image%%|./httpdmon.png| |^ This utility continuously displays a screen of information comprising four or five of the following sections: |number| |item| |*System Information| |^- The nodename, instance number(s), monitor version and current date/time. |item| |*Process Information| |^- HTTPd process information includes its up-time, CPU-time consumed (excluding any subprocesses), I/O counts, and memory utilization. The "Servers:" item shows how many servers are currently running on the node/cluster. Changes in this count are indicated by the second, parenthesized number. |item| |*General Server Counters| |^- The server counters keep track of the total connections received, accepted, rejected, etc., totals for each request type (file transfer, directory listing, image mapping, etc.). |^- \**\ The request count of 3.8M is a real value, as are the others, with the screenshot taken during x86-64 (V9.1-A) testing using OWASP ZAP. |item| |*Proxy Serving Counters| |^- The server counters keep track of proxy serving connections, network and cache traffic, cache status, etc. |item| |*Latest Request| |^- This section provides the response status code, and some transaction statistics, the service being accessed, originating host and HTTP request. Note that long request strings may be truncated (indicated by a bolded ellipsis). |item| |*Status Message| |^- If the server is in an exceptional condition, for example exited after a fatal error, starting up, etc., a textual message may be displayed in place of the the request information. This may be used to initiate remedial actions, etc. |!number| |^ The "/HELP" qualifier provides a brief usage summary. |^ The server counter values are carried over when a server (re)starts (provided the system has stayed up). To reset the counters use the online Server Administration facility (|link|Server Administration||). |^ If [DNSlookup] is disabled for the HTTP server the HTTPDMON utility attempts to resolve the literal address into a host name. This may be disabled using the /NORESOLVE qualifier. |2MD5digest| |^ From RFC1321 |...| |^ " The [MD5] algorithm takes as input a message of arbitrary length and produces as output a 128-bit "fingerprint" or "message digest" of the input. It is conjectured that it is computationally infeasible to produce two messages having the same message digest, or to produce any message having a given prespecified target message digest. " |^ The MD5DIGEST utility is primarily provided with WASD for verifying kits as unchanged from the originals released. With the proliferation of mirror sites and other distribution resources it has become good practice to ensure kits remain unchanged from release, to distribution, to installation site (changes due to to data corruption or malicious intent - as remote a possibility as that may seem). Of course it may also be used for any other purpose where the MD5 hash is useful. |^ For verifying the contents of a WASD release connect to the |*original| WASD distribution site, refer to the download page, and make a comparison between the release MD5 hash found against the list of all archive hashes and the MD5 hash of your archive. That can be done as follows |code| $ MD5DIGEST == "$WASD_EXE:MD5DIGEST" $ MD5DIGEST device:[dir]archive.ZIP |!code| The result will look similar to |code| MD5 (kits:[000000]htroot710.zip;1) = 404bbdfe0f847c597b034feef2d13d2d |!code| |^ Of course, if you have not yet installed your first WASD distribution using the MD5DIGEST utility that is part of it is not feasable. The original site can provide kits and pre-built executables for this purpose. |2QDLogStats| |^ |*Quick-and-Dirty LOG STATisticS| is a utility to extract very elementary statistics from Web server common/combined format log files. It is intended for those moments when we think "I wonder how many times that new archive has been downloaded?", "How much data was transfered during November?", "How often is |/such-and-such| a client using the authenticated |/so-and-so| service?", "How much has the mail service been used?" |...| and want the results in a matter of seconds (or at least a few tens of seconds ;-) It is available at the command-line and as a CGI script. |image%%|./qdlogstats.png| |^ For QDLOGSTATS to be available as a CGI script it |*must| have authorization enabled against it (to prevent potential ad hoc browsing of a site's logs). The following provides some indication of this configuration, although of course it requires tailoring for any given site. |code| [VMS] /cgi-bin/qdlogstats ~webadmin,131.185.250.*,r+w ; |!code| |^ It could then be accessed using |code| http://the.host.name/cgi-bin/qdlogstats |!code| |^ The initial access provides a form allowing the various filters and other behaviours to be selected. The CGI form basically parallels the command-line behaviour described below. |0Filters| |^ A number of filters allow subsets of the log contents to be selected. These filters support the same string matching expressions as the server (see |link%|../config/##String Matching++of++WASD Configuration||). |^ A knowlege of the format and contents of the |/common| and |/combined| log formats will assist in deciding which and to what purpose filters should be used. Record filtering is done in the same order as is finally displayed, so |/method| would be processed before |/user-agent| for instance. Normally a record match terminates on the first non-matched filter (to expedite processing). To compare and report each filter for every record apply the /ALL qualifier. To view records as they are processed use the /VIEW qualifier. This by default displays all matched records, but the optional =ALL or =NOMATCH parameters will display all records, or all those but the matches. |^ |*$ QDLOGSTATS log-file-spec [pattern qualifiers] [other qualifiers]| |0Parameters and Qualifiers| |table| |~_ |: Parameter|: Description |~ |~#* |. /ALL |. compare and report on all supplied filters |~ |. /AUTHUSER= |. pattern (any authenticated username) |~ |. /BEFORE= |. log files before this VMS date/time |~ |. /CLIENT= |. pattern (client host name or IP address) |~ |. /DATETIME= |. pattern ("11/Jun/1999:14:08:49 +0930") |~ |. /DECODE[=|/keyword||] |. URL-decode PATH, QUERY, REFERER before match |~ |. /METHOD= |. pattern (HTTP "GET", "POST", etc.) |~ |. /OUTPUT= |. file specification |~ |. /PATH= |. pattern (URL path component only) |~ |. /PROGRESS |. show progress during processing; a "+" for each file started, a "." for each 1000 records processed |~ |. /QUERY= |. pattern (URL query component only) |~ |. /REFERER= |. pattern (HTTP "Referer:" field, COMBINED only) |~ |. /REMOTEID= |. pattern (RFC819 file) |~ |. /RESPONSE= |. pattern (HTTP response code) |~ |. /SINCE= |. log files after this VMS date/time |~ |. /SIZE[=|/keyword||] |. response size (in bytes) MIN=|/integer| MAX=|/integer| |~ |. /USERAGENT= |. pattern (HTTP "User-Agent:" field, COMBINED only) |~ |. /VIEW[=type] |. display matching log records (ALL, NOMATCH, MATCH) |!table| |0Usage Examples| |bullet| |item| Records from September 1999. |code| $ QDLOGSTATS WASD_LOGS:*1999*.LOG /DATE="*/SEP/1999*" |!code| |item| Records where the browser was an X-based Netscape Navigator |code| $ QDLOGSTATS WASD_LOGS:*.LOG /USERAGENT=*MOZILLA*X11* |!code| |item| Records of POST method requests |code| $ QDLOGSTATS WASD_LOGS:*.LOG /METHOD=POST |!code| |item| Records requesting a particular path |code| $ QDLOGSTATS WASD_LOGS:*.LOG /PATH="/cgi-bin/*" |!code| |item| Select proxy records requesting (a) particular site(s) |code| $ QDLOGSTATS WASD_LOGS:*8080*.LOG /PATH="http://*.compaq.com*" $ QDLOGSTATS WASD_LOGS:*8080*.LOG /METHOD=POST /PATH="http://*sex*.*/*" /VIEW |!code| |item| Records where the request was authenticated |code| $ QDLOGSTATS WASD_LOGS:*.LOG /AUTHUSER=DANIEL |!code| |!bullet| |2SECHAN Utility| |^ The SECHAN utility (pronounced "session") is used by [INSTALL]SECURE.COM and associated procedures to make file system security settings. It is also available for direct use by the site administrator. See |link%|../config/##Security Considerations++of++WASD Configuration||). |2StreamLF Utility| |^ This simple procedure used the FDL facility to convert files to STREAM_LF format. The WASD HTTPd server access STREAM_LF files in block/IO-mode, far more efficiently that the record-mode required by variable-record format files. |^ |*NOTE: | The server can also be configured to automatically convert any VARIABLE record format files it encounters to STREAM_LF. |2WAStee Utility| |^ WAStee is a utility to generate time-stamped log files containing intervals of a long-lived WASD server process, and/or to consolidate all process log files generated during the defined period. It is the tee in a PIPE sequence. |^ This utility is UNSUITABLE for sites using multiple instances and/or environments on a node. Only the first of multiple server processes will have the log teed. |^ For further information check the descriptive prologue in the |link%|/wasd_root/src/utils/wastee.c|WASD_ROOT:[SRC.UTILS]WASTEE.C| source code. |2WOTSUP Utility| |^ The "WASD Over-The-Shoulder Uptime Picket" is designed to monitor WASD in a production environment for the purpose of alerting operations staff to conditions which might cause that production to be adversely impacted. |^ Alert triggers include: |bullet#| |& server image exit and/or startup (default) |& server process non-existent or suspended (default) |& percentage thresholds on process quotas (optional) |& rates of HTTP status counter change (optional) |& maximum period without request processing (optional) |!bullet| |^ Alert reports can be delivered via any combination of: |bullet#| |& OPCOM message |& MAIL |& site-specific DCL command executed in a spawned subprocess |& log file entry |!bullet| |^ The utility runs in a detached process and monitors the server environment by periodically polling various server data at a default interval is 15 seconds. As the utility requires access to global memory accounting a per-system WOTSUP is required for each node to be monitored. |^ The following (somewhat contrived) example illustrates the format and content of a WOTSUP report delivered via OPCOM. Reports delivered via other mechanisms have the same content and similar format. |code| %%%%%%%%%% WOTSUP 24-OCT-2006 13:32:56.44 %%%%%%%%%%% Message from user SYSTEM on KLAATU Over-The-Shoulder (WASD_WOTSUP) reports: 1. server PID 001C0950 exit %X00000001 (%SYSTEM-S-NORMAL) 2. server STARTUP (10) 3. server PIDs are 0018C14F (HTTPd:80), 001C0950 (HTTPe:80) 4. pagfilcnt:395432 pgflquota:500000 79% <= 80% |!code| |^ For further information check the descriptive prologue in the |link%|/wasd_root/src/utils/wotsup.c|WASD_ROOT:[SRC.UTILS]WOTSUP.C| source code.