/*****************************************************************************/ /* GZIP.c Provides ZLIB-enabled GZIP compression (gzip, deflate) for WASD. Dynamically maps required functions from a ZLIB shareable image. If the image is found and all required functions present the ZLIB compression/decompression of network streams in enabled for suitable requests/responses. If not it is just left disabled. Based on ZLIB v1.2.1 port by Jean-François Piéronne (jf.pieronne@laposte.net). Requires this package to be installed and started on the runtime system for dynamic activation. The sharable image must be INSTALLed (without any particular privileges) before it can be activated by the privileged WASD HTTPd image. This module will compile on a system that does not have the LIBZ (ZLIB) header files. The WASD GZIP.H header contains all of the ZLIB structure and macro defintions required. Of course with WASD dynamically activating the LIBZ sharable image no static linking is required. RESPONSE CONTENT-ENCODING ------------------------- The WASD_CONFIG_GLOBAL directive [GzipResponse] controls whether this feature is enabled for the gzip content-encoding of suitable response bodies. This directive requires at least one parameter, the compression level in the range 1..9. Smaller values provide faster but poorer compression ratios while larger values better compression at the cost of more CPU cycles and latency. This corresponds to the GZIP utility's -1..-9 CLI switches. Two optional parameters could allow ZLIB's 'memLevel' and 'windowBits' to be adjusted by ZLIB afficiendos (level[,memory,window]). A small amount of experimentation by this author indicates minor changes in memory usage and compression ratio by fiddling with these. Be aware that GZIP encoding is MEMORY INTENSIVE. From 132kB to 265kB has been observed per compressing request (WATCH provides this in a summary line). These values apply across a wide range of transfer sizes (from kilobytes to tens of megabytes). It also is very CPU INTENSIVE and adds response latency (on the author's 333MHz system anyway), though that might be well be offset by significant reductions in transfer time on the Internet or other slower, non-intranet infrastructures. Text content compression has been observed from 30% to 10% of the original file size (even down to 1% in the case of the extremely redundant content of [EXAMPLE]64K.TXT). VMS executables (for want of another binary test case) at around 40%. In other words, GZIP encoding may not be suitable or efficient for every site or every request! Once enabled WASD will GZIP the responses for all suitable contents provided the client accepts the encoding and the response is not one of the following (see GzipShouldDeflate()): o less than 1400 bytes (no point in the overhead) o already content-encoded script or proxy response o script requested that the response not be GZIPed o a compressed image (e.g. GIF, JPEG, etc) o a video stream (presumably already compressed, e.g. MPEG) o a compressed audio stream o an obviously compressed application stream (e.g. GZIP, ZIP, JAR) o a PDF file, which seem to experience plug-in problems when GZIPed (this can be overridden using a 'set **.pdf response=gzip=all') o a Shockwave Flash file, which are chocka with compressed objects o SSI source file ('text/x-shtml' content type) Additional control may be exercised with the following path SETings: o response=GZIP=all matching paths will always have GZIP encoding performed (the above constraints still apply) o response=GZIP=none matching paths will never have GZIP encoding o response=GZIP= responses with content-lengths greater than the specified number of kilobytes will be GZIP content-encoded (if the content-length cannot be determined it will NOT not encoded and the above constraints still apply) REQUEST CONTENT-ENCODING ------------------------ Decoding of GZIP content-encoded request bodies is enabled using the WASD_CONFIG_GLOBAL directive [GzipAccept]. Enabling this using a value 15 (or 1) results in the server advertising it's acceptance of GZIPed requests using the "Accept-Encoding: gzip, deflate" response header. Requests containing bodies GZIP compressed will have these decoded as they are read from the client and before further processing, such as the upload of files into server accessable file-system space. THIS DECODING IS OPTIONAL and not the default with DCL and DECnet script processing. That is, a request body will be passed to the script still encoded unless specific mapping directs otherwise. Decoding by the server into the original data prior to transfering to the script can be enabled for all or selected scripts using the following path settings. o script=body=decode script gets the decoded stream o script=body=NOdecode script gets the raw, encoded stream (default) Note that scripts need to be specially aware of both GZIP encoded bodies and those already decoded by the server. In the first case the stream must be read to the specified content-length and then decoded. In the second case, a content-length cannot be provided by the server (without unencoding the entire stream ahead of time it cannot predict the final size). Where the server is to decode the request body before transfering it to the script it changes the CGI variable CONTENT_LENGTH to a single question-mark ("?"). Scripts may use this to detect the server's intention and then must ignore any transfer-encoding and/or content-encoding header information and read the request body until end-of-file is received. GZIP decoding (decompression) is understandably much less memory and CPU intensive. Experimentation indicates it does not contribute significantly to latency either. TESTING GZIP ------------ Most modern browsers accept a GZIP deflated response so that's straight-forward. Either the document displays correctly or not. Server processing of GZIP deflated request bodies is a little more involved. See the decription in the BODY.C module. VERSION HISTORY --------------- 22-APR-2020 MGD add WATCH reporting points for NETWORK 24-AUG-2013 MGD GzipInit() ZLIB shareable image via logical names WASD_LIBZ_SHR32, then GNV$LIBZSHR32, finally LIBZ_SHR32 18-AUG-2009 MGD bugfix; compress PDF if path set all 25-APR-2007 MGD GzipShouldDeflate() do not compress Shockwave Flash increase minimum size before compression to 1400 bytes 28-FEB-2007 MGD bugfix; GzipDeflateCache() allow for cached CGI header 26-OCT-2006 JPP bugfix; GzipShouldDefault() uninitialized 'cptr' when no content-type would cause WatchThis() "!AZ" to barf if 'cptr' was non-NULL but pointed into an invalid page 04-JUL-2006 MGD use PercentOf32() for more accurate percentages 04-JUL-2005 MGD bugfix; adjust CacheMemoryInUse/CachePermMemoryInUse bugfix; GzipDeflateCache() ambit buffer size calculation too small for small content lengths (just allow heaps!) 28-MAY-2005 MGD bugfix; GzipDeflateCache() ambit buffer size calculation (captr->ContentLength >> 9) now (.. >> 7) (jpp@esme.fr) 26-MAR-2005 MGD GzipDeflateCache() to deflate cache entry content, GzipShouldDeflate() modified so 'rqptr' is optional 04-FEB-2005 MGD GzipShouldDeflate() disable PDF deflation by default 22-JAN-2005 MGD bugfix; GzipDeflate() delay avail input until output non-zero 08-JAN-2005 MGD timer-based flush mechanism 15-NOV-2004 MGD bugfix; GzipDeflate() (with NET.C) 16-OCT-2004 MGD initial */ /*****************************************************************************/ #ifdef WASD_VMS_V7 #undef _VMS__V6__SOURCE #define _VMS__V6__SOURCE #undef __VMS_VER #define __VMS_VER 70000000 #undef __CRTL_VER #define __CRTL_VER 70000000 #endif #include #include #include "wasd.h" #include "gzip.h" #define WASD_MODULE "GZIP" /******************/ /* global storage */ /******************/ BOOL GzipAccept, GzipAvailable, GzipResponse; int GzipDeflateCompLevel, GzipDeflateMemLevel, GzipDeflateWindowBits, GzipFindImageStatus, GzipFlushSeconds, GzipFlushInitialSeconds, GzipInflateWindowBits; char *GzipZlibNamePtr, *GzipZlibVersionPtr; char *GzipZlibError [] = { "Unknown ZLIB error!", "ZLIB errno error.", "ZLIB stream error.", "ZLIB data error.", "ZLIB memory error.", "ZLIB buffer error." "ZLIB version error." }; int (*ZlibDeflateFn)(z_stream*, int); int (*ZlibDeflateInit2Fn)(z_stream*, int, int, int, int, int, char*, int); int (*ZlibDeflateEndFn)(z_stream*); int (*ZlibInflateFn)(z_stream*, int); int (*ZlibInflateInit2Fn)(z_stream*, int, char*, int); int (*ZlibInflateEndFn)(z_stream*); char* (*ZlibVersionFn)(void); /********************/ /* external storage */ /********************/ extern int CacheChunkInBytes, HttpdTickSecond; extern unsigned long CacheGzipDeflateCount, CacheMemoryInUse, CachePermMemoryInUse; extern uint64 CacheGzipDeflateBytesIn64, CacheGzipDeflateBytesOut64; extern char ErrorSanityCheck[]; extern ACCOUNTING_STRUCT *AccountingPtr; extern CONFIG_STRUCT Config; extern WATCH_STRUCT Watch; /*****************************************************************************/ /* Attempt to get the required functions from the ZLIB shareable image. Any errors, including the obvious one of image not found, just disables any gzip encoding. */ GzipInit () { static $DESCRIPTOR (LibzImageDsc, "LIBZ_SHR32"); static $DESCRIPTOR (GnvLibzImageDsc, "GNV$LIBZSHR32"); static $DESCRIPTOR (WasdLibzImageDsc, "WASD_LIBZ_SHR32"); static $DESCRIPTOR (DeflateDsc, "deflate"); static $DESCRIPTOR (DeflateInit2Dsc, "deflateInit2_"); static $DESCRIPTOR (DeflateEndDsc, "deflateEnd"); static $DESCRIPTOR (InflateDsc, "inflate"); static $DESCRIPTOR (InflateInit2Dsc, "inflateInit2_"); static $DESCRIPTOR (InflateEndDsc, "inflateEnd"); static $DESCRIPTOR (ZlibVersionDsc, "zlibVersion"); int status; struct dsc$descriptor_s *dscptr; /*********/ /* begin */ /*********/ if (WATCH_MODULE(WATCH_MOD_NET)) WatchThis (WATCHALL, WATCH_MOD_NET, "GzipInit()"); /* if it's not configured then don't initialize */ if (!(Config.cfMisc.GzipAcceptWindowBits || Config.cfMisc.GzipResponseCompLevel)) return; lib$establish (GzipInitHandler); status = lib$find_image_symbol (dscptr = &WasdLibzImageDsc, &ZlibVersionDsc, &ZlibVersionFn, 0, 0); if (status == RMS$_FNF) status = lib$find_image_symbol (dscptr = &GnvLibzImageDsc, &ZlibVersionDsc, &ZlibVersionFn, 0, 0); if (status == RMS$_FNF) status = lib$find_image_symbol (dscptr = &LibzImageDsc, &ZlibVersionDsc, &ZlibVersionFn, 0, 0); if (status == RMS$_FNF) { GzipFindImageStatus = status; GzipZlibNamePtr = dscptr->dsc$a_pointer; FaoToStdout ("%HTTPD-W-GZIP, shareable image not found\n"); return; } if (VMSok (status)) status = lib$find_image_symbol (dscptr, &DeflateDsc, &ZlibDeflateFn, 0, 0); if (VMSok (status)) status = lib$find_image_symbol (dscptr, &DeflateInit2Dsc, &ZlibDeflateInit2Fn, 0, 0); if (VMSok (status)) status = lib$find_image_symbol (dscptr, &DeflateEndDsc, &ZlibDeflateEndFn, 0, 0); if (VMSok (status)) status = lib$find_image_symbol (dscptr, &InflateDsc, &ZlibInflateFn, 0, 0); if (VMSok (status)) status = lib$find_image_symbol (dscptr, &InflateInit2Dsc, &ZlibInflateInit2Fn, 0, 0); if (VMSok (status)) status = lib$find_image_symbol (dscptr, &InflateEndDsc, &ZlibInflateEndFn, 0, 0); lib$revert (); GzipFindImageStatus = status; GzipZlibNamePtr = dscptr->dsc$a_pointer; if (VMSok (status)) { GzipAvailable = true; if (Config.cfMisc.GzipAcceptWindowBits) GzipAccept = true; if (Config.cfMisc.GzipResponseCompLevel) GzipResponse = true; GzipInflateWindowBits = Config.cfMisc.GzipResponseWindowBits; if (GzipInflateWindowBits < 8) GzipInflateWindowBits = 15; /* assume the default */ else if (GzipInflateWindowBits > 15) GzipInflateWindowBits = 15; GzipDeflateCompLevel = Config.cfMisc.GzipResponseCompLevel; if (GzipDeflateCompLevel > 9) GzipDeflateCompLevel = 9; GzipDeflateMemLevel = Config.cfMisc.GzipResponseMemLevel; /* if not specified default to ZLIB default */ if (!GzipDeflateMemLevel) GzipDeflateMemLevel = 8; if (GzipDeflateMemLevel > 9) GzipDeflateMemLevel = 9; GzipDeflateWindowBits = Config.cfMisc.GzipResponseWindowBits; /* if not specified default to ZLIB default */ if (!GzipDeflateWindowBits) GzipDeflateWindowBits = 15; if (GzipDeflateWindowBits < 8) GzipDeflateWindowBits = 15; /* assume the default */ else if (GzipDeflateWindowBits > 15) GzipDeflateWindowBits = 15; GzipFlushSeconds = Config.cfMisc.GzipFlushSeconds; if (GzipFlushSeconds < 0) GzipFlushSeconds = 0; GzipFlushInitialSeconds = Config.cfMisc.GzipFlushInitialSeconds; if (GzipFlushInitialSeconds < 0) GzipFlushSeconds = 0; GzipZlibVersionPtr = ZlibVersionFn(); FaoToStdout ("%HTTPD-I-GZIP, using !AZ V!AZ\n", GzipZlibNamePtr, GzipZlibVersionPtr); } else FaoToStdout ("%HTTPD-W-GZIP, !AZ %X!8XL\r\n-!-!&M\n", GzipZlibNamePtr, status); } /*****************************************************************************/ /* Just continue, to report an error if the image couldn't be activated or the required symbol not found. */ int GzipInitHandler () { /*********/ /* begin */ /*********/ if (WATCH_MODULE(WATCH_MOD_NET)) WatchThis (WATCHALL, WATCH_MOD_NET, "GzipInitHandler()"); return (SS$_CONTINUE); } /*****************************************************************************/ /* Return a pointer to the appropriate ZLIB message string. */ char* GzipZlibMsg ( z_stream *zsptr, int zstatus ) { char *cptr; /*********/ /* begin */ /*********/ if (WATCH_MODULE(WATCH_MOD_NET)) WatchThis (WATCHALL, WATCH_MOD_NET, "GzipZlibMsg() !UL", zstatus); if (!(cptr = zsptr->msg)) { zstatus = -zstatus; if (zstatus < 0 || zstatus > 5) zstatus = 0; cptr = GzipZlibError[zstatus]; } return (cptr); } /*****************************************************************************/ /* A function to deflate a cache entry's content (see CACHE.C) to provide an already GZIPed equivalent for serving, obviously ameliorating the expense of doing it dynamically for each request. Performs the full initialization, deflate and end ZLIB compression life-cycle in the one call. */ BOOL GzipDeflateCache ( REQUEST_STRUCT *rqptr, FILE_CENTRY *captr ) { int zstatus, BufferCount, BufferSize, SizeInBytes; char *BufferPtr; z_stream zStream; z_stream *zsptr; /*********/ /* begin */ /*********/ if (WATCHMOD (rqptr, WATCH_MOD_NET)) WatchThis (WATCHITM(rqptr), WATCH_MOD_NET, "GzipDeflateCache()"); zsptr = &zStream; zsptr->opaque = rqptr; zsptr->zalloc = &GzipAlloc; zsptr->zfree = &GzipFree; /* plus 16 is the magic number to get a gzip header and trailer!! */ zstatus = ZlibDeflateInit2Fn (zsptr, GzipDeflateCompLevel, Z_DEFLATED, GzipDeflateWindowBits + 16, GzipDeflateMemLevel, Z_DEFAULT_STRATEGY, ZLIB_VERSION, sizeof(z_stream)); if (WATCHMOD (rqptr, WATCH_MOD_NET)) WatchThis (WATCHITM(rqptr), WATCH_MOD_NET, "ZlibDeflateInit2Fn(!UL,!UL+16,!UL) !SL !&Z", GzipDeflateCompLevel, GzipDeflateWindowBits, GzipDeflateMemLevel, zstatus, zsptr->msg); if (zstatus != Z_OK) { ErrorNoticed (rqptr, SS$_BUGCHECK, GzipZlibMsg(zsptr,zstatus), FI_LI); return (false); } /* ambit buffer size will be length times two - heaps */ BufferSize = captr->ContentLength + captr->ContentLength + 12; if (captr->CgiHeaderLength) { zsptr->next_in = captr->ContentPtr + captr->CgiHeaderLength + 1; zsptr->avail_in = captr->ContentLength - captr->CgiHeaderLength - 1; } else { zsptr->next_in = captr->ContentPtr; zsptr->avail_in = captr->ContentLength; } zsptr->next_out = BufferPtr = VmGetCache (BufferSize); zsptr->avail_out = BufferSize; zstatus = ZlibDeflateFn (zsptr, Z_FINISH); if (zstatus != Z_STREAM_END) { ZlibDeflateEndFn (zsptr); VmFreeCache (BufferPtr, FI_LI); ErrorNoticed (rqptr, SS$_BUGCHECK, GzipZlibMsg(zsptr,zstatus), FI_LI); return (false); } /* whatever's in the buffer */ BufferCount = BufferSize - zsptr->avail_out; SizeInBytes = BufferCount; if (SizeInBytes % CacheChunkInBytes) SizeInBytes += CacheChunkInBytes; SizeInBytes = SizeInBytes / CacheChunkInBytes; SizeInBytes *= CacheChunkInBytes; if (!SizeInBytes) SizeInBytes = CacheChunkInBytes; if (rqptr->rqPathSet.CachePermanent) { captr->GzipContentPtr = VmGetPermCache (SizeInBytes); CachePermMemoryInUse += SizeInBytes; } else { captr->GzipContentPtr = VmGetCache (SizeInBytes); CacheMemoryInUse += SizeInBytes; } captr->GzipEntrySize = SizeInBytes; captr->GzipContentLength = BufferCount; memcpy (captr->GzipContentPtr, BufferPtr, BufferCount); CacheGzipDeflateCount++; CacheGzipDeflateBytesIn64 += captr->ContentLength; CacheGzipDeflateBytesOut64 += captr->GzipContentLength; zstatus = ZlibDeflateEndFn (zsptr); if (zstatus != Z_OK) ErrorNoticed (rqptr, SS$_BUGCHECK, GzipZlibMsg(zsptr,zstatus), FI_LI); VmFreeCache (BufferPtr, FI_LI); if (WATCHING (rqptr, WATCH_RESPONSE)) WatchThis (WATCHITM(rqptr), WATCH_RESPONSE, "CACHE deflate !UL->!UL bytes, !UL%", captr->ContentLength, captr->GzipContentLength, PercentOf32(captr->GzipContentLength,captr->ContentLength)); return (true); } /*****************************************************************************/ /* Initialize the ZLIB structure. Allocate memory from the ZLIB output buffer. Returns true and set the pointer to the ZLIB structure if successful. Returns false and leaves the pointer NULL if not. Reports any error via the usual WASD mechanism. */ BOOL GzipDeflateBegin ( REQUEST_STRUCT *rqptr, GZIP_COMPRESS *gzptr, int BufferSize ) { int zstatus; z_stream *zsptr; /*********/ /* begin */ /*********/ if (WATCHMOD (rqptr, WATCH_MOD_NET)) WatchThis (WATCHITM(rqptr), WATCH_MOD_NET, "GzipDeflateBegin() !UL", BufferSize); gzptr->DeflateStartStream = true; gzptr->DeflateEndOfStream = false; gzptr->DeflateMemoryAllocated = rqptr->GzipMemoryAllocated = 0; zsptr = (z_stream*)VmGetHeap (rqptr, sizeof(z_stream)); zsptr->opaque = rqptr; zsptr->zalloc = &GzipAlloc; zsptr->zfree = &GzipFree; /* plus 16 is the magic number to get a gzip header and trailer!! */ zstatus = ZlibDeflateInit2Fn (zsptr, GzipDeflateCompLevel, Z_DEFLATED, GzipDeflateWindowBits + 16, GzipDeflateMemLevel, Z_DEFAULT_STRATEGY, ZLIB_VERSION, sizeof(z_stream)); if (WATCHMOD (rqptr, WATCH_MOD_NET)) WatchThis (WATCHITM(rqptr), WATCH_MOD_NET, "ZlibDeflateInit2Fn(!UL,!UL+16,!UL) !SL !UL !&Z", GzipDeflateCompLevel, GzipDeflateWindowBits, GzipDeflateMemLevel, zstatus, rqptr->GzipMemoryAllocated, zsptr->msg); if (zstatus == Z_OK) { gzptr->DeflateZstreamPtr = zsptr; gzptr->DeflateBufferSize = BufferSize; gzptr->DeflateBufferPtr = VmGetHeap (rqptr, BufferSize); if (GzipFlushSeconds) { gzptr->FlushTickSeconds = HttpdTickSecond; if (GzipFlushSeconds > GzipFlushInitialSeconds) gzptr->FlushTickSeconds += GzipFlushInitialSeconds; else gzptr->FlushTickSeconds += GzipFlushSeconds; } return (true); } rqptr->rqResponse.HttpStatus = 500; ErrorGeneral (rqptr, GzipZlibMsg(zsptr,zstatus), FI_LI); return (false); } /*****************************************************************************/ /* Deallocate ZLIB structures. Returns true if there was no last-minute problem reported by ZLIB, false and reports the error if there was. */ BOOL GzipDeflateEnd ( REQUEST_STRUCT *rqptr, GZIP_COMPRESS *gzptr ) { int zstatus; z_stream *zsptr; /*********/ /* begin */ /*********/ if (gzptr->DeflateZstreamPtr) { if (WATCHING (rqptr, WATCH_RESPONSE)) if (!WATCHING (rqptr, WATCH_NETWORK)) WatchThis (WATCHITM(rqptr), WATCH_RESPONSE, "DEFLATE !UL->!UL bytes, !UL% (!ULkB)", gzptr->DeflateBytesIn, gzptr->DeflateBytesOut, PercentOf32(gzptr->DeflateBytesOut, gzptr->DeflateBytesIn), rqptr->GzipMemoryAllocated >> 10); } gzptr->DeflateEndOfStream = true; InstanceMutexLock (INSTANCE_MUTEX_HTTPD); AccountingPtr->GzipDeflateCount++; AccountingPtr->GzipDeflateBytesIn64 += gzptr->DeflateBytesIn; AccountingPtr->GzipDeflateBytesOut64 += gzptr->DeflateBytesOut; InstanceMutexUnLock (INSTANCE_MUTEX_HTTPD); if (zsptr = gzptr->DeflateZstreamPtr) { gzptr->DeflateMemoryAllocated = rqptr->GzipMemoryAllocated; zstatus = ZlibDeflateEndFn (zsptr); /* this as NULL indicates that deflate is no longer in use */ gzptr->DeflateZstreamPtr = NULL; } else zstatus = Z_OK; if (WATCHMOD (rqptr, WATCH_MOD_NET)) WatchThis (WATCHITM(rqptr), WATCH_MOD_NET, "ZlibDeflateEnd() !SL", zstatus); if (zstatus == Z_OK) return (true); rqptr->rqResponse.HttpStatus = 500; ErrorGeneral (rqptr, GzipZlibMsg(zsptr,zstatus), FI_LI); return (false); } /*****************************************************************************/ /* Data to be written to the network is intercepted and passed to this function. It uses the ZLIB deflate function to compress the data into the WASD buffer allocated during initialization. The '*InDataPtrPtr' and '*InDataLengthPtr' point to the input buffer. If '*InDataPtrPtr' is NULL then it's end-of-stream. When input space is available and the input buffer consumed '*InDataLengthPtr' is modified to be zero. This detected by the calling routine to indicate more input data (actually server output data, but input to GZIP) should be obtained. The '*OutDataPtrPtr' and '*DataLengthPtr' are then modified to reflect any compressed data available for writing (zero indicates none and more should be fed to the input). Return true to indicate success, false if the write (and request) should be terminated due to error (which is also reported by the usual WASD mechanism). */ BOOL GzipDeflate ( REQUEST_STRUCT *rqptr, GZIP_COMPRESS *gzptr, char **InDataPtrPtr, int *InDataLengthPtr, char **OutDataPtrPtr, int *OutDataLengthPtr ) { int zflush, zstatus, InDataLength, OriginalAvailIn; char *InDataPtr; z_stream *zsptr; /*********/ /* begin */ /*********/ zsptr = gzptr->DeflateZstreamPtr; if (WATCHMOD (rqptr, WATCH_MOD_NET)) WatchThis (WATCHITM(rqptr), WATCH_MOD_NET, "GzipDeflate() !&B !UL !UL !UL !&B", !*InDataPtrPtr, *InDataLengthPtr, zsptr->avail_in, zsptr->avail_out, gzptr->FlushTickSeconds < HttpdTickSecond); /* a NULL pointer indicates end-of-stream */ if (*InDataPtrPtr) { if (GzipFlushSeconds) { if (gzptr->DeflateBytesOut > GZIP_SIZE_OF_HEADER && gzptr->FlushBytesOut != gzptr->DeflateBytesOut) { /* bytes have been output by ZLIB, reset the 'flush' timer */ gzptr->FlushTickSeconds = HttpdTickSecond + GzipFlushSeconds; gzptr->FlushBytesOut = gzptr->DeflateBytesOut; } else if (gzptr->FlushTickSeconds < HttpdTickSecond) { /* no bytes output from ZLIB for the 'flush' number of seconds */ gzptr->DeflateFlushStream = true; gzptr->FlushTickSeconds = HttpdTickSecond + GzipFlushSeconds; gzptr->FlushBytesOut = gzptr->DeflateBytesOut; } } if (gzptr->DeflateFlushStream) zflush = Z_SYNC_FLUSH; else zflush = Z_NO_FLUSH; } else zflush = Z_FINISH; *OutDataLengthPtr = 0; *OutDataPtrPtr = NULL; /* if have data to process or have previously processed some */ if (*InDataLengthPtr || zsptr->total_in) { if (!zsptr->avail_in) { /* requires more input */ zsptr->next_in = InDataPtr = *InDataPtrPtr; zsptr->avail_in = InDataLength = *InDataLengthPtr; *InDataLengthPtr = 0; } if (!zsptr->avail_out) { /* output buffer needs (re)setting */ zsptr->next_out = gzptr->DeflateBufferPtr; zsptr->avail_out = gzptr->DeflateBufferSize; } zstatus = ZlibDeflateFn (zsptr, zflush); gzptr->DeflateBytesIn = zsptr->total_in; gzptr->DeflateBytesOut = zsptr->total_out; if (zstatus == Z_OK) { if (gzptr->DeflateFlushStream) { /* this many bytes are available for flushing */ gzptr->DeflateBufferCount = gzptr->DeflateBufferSize - zsptr->avail_out; if (zsptr->avail_out) { /* not a buffer-full though, go back to not flushing */ gzptr->DeflateFlushStream = false; /* reset the output buffer */ zsptr->next_out = gzptr->DeflateBufferPtr; zsptr->avail_out = gzptr->DeflateBufferSize; } } else if (!zsptr->avail_out) { /* the output buffer has filled, flush the contents */ gzptr->DeflateBufferCount = gzptr->DeflateBufferSize; } else gzptr->DeflateBufferCount = 0; } else if (zstatus == Z_STREAM_END) { /* whatever's in the buffer */ gzptr->DeflateBufferCount = gzptr->DeflateBufferSize - zsptr->avail_out; } else { /* yes JPP, you're right, kludgy as it looks - it seems necessary! */ if (zstatus == Z_BUF_ERROR) zstatus = Z_OK; gzptr->DeflateBufferCount = 0; } *OutDataPtrPtr = gzptr->DeflateBufferPtr; *OutDataLengthPtr = gzptr->DeflateBufferCount; if (WATCHMOD (rqptr, WATCH_MOD_NET)) WatchThis (WATCHITM(rqptr), WATCH_MOD_NET, "!SL !SL !UL/!UL !&X:!UL !&X:!UL !UL->!UL !&Z", zflush, zstatus, zsptr->avail_in, zsptr->avail_out, InDataPtr, InDataLength, *OutDataPtrPtr, *OutDataLengthPtr, gzptr->DeflateBytesIn, gzptr->DeflateBytesOut, zsptr->msg); if (gzptr->DeflateBufferCount) if (WATCHING (rqptr, WATCH_NETWORK)) WatchThis (WATCHITM(rqptr), WATCH_NETWORK, "DEFLATE !UL->!UL bytes, !UL% (!ULkB)", gzptr->DeflateBytesIn, gzptr->DeflateBytesOut, PercentOf32(gzptr->DeflateBytesOut, gzptr->DeflateBytesIn), rqptr->GzipMemoryAllocated >> 10); } else { /* no data has ever been processed, just clean up the ZLIB stuff */ zstatus = Z_STREAM_END; } rqptr->BytesTxGzipPercent = PercentOf32 (gzptr->DeflateBytesOut, gzptr->DeflateBytesIn); if (zstatus == Z_OK) return (true); if (zstatus == Z_STREAM_END) if (GzipDeflateEnd (rqptr, gzptr)) return (true); rqptr->rqResponse.HttpStatus = 500; ErrorGeneral (rqptr, GzipZlibMsg(zsptr,zstatus), FI_LI); GzipDeflateEnd (rqptr, gzptr); return (false); } /*****************************************************************************/ /* Can and should the content body be deflated? Return true to indicate deflate, false to leave as-is. */ BOOL GzipShouldDeflate ( REQUEST_STRUCT *rqptr, char *ContentTypePtr, int ContentLength ) { BOOL EncodeGzip; char *cptr; char ResponseKbytes [32]; /*********/ /* begin */ /*********/ if (WATCHMOD (rqptr, WATCH_MOD_NET)) WatchThis (WATCHITM(rqptr), WATCH_MOD_NET, "GzipShouldDeflate() !&B !&Z !SL", GzipAvailable, ContentTypePtr, ContentLength); if (!GzipAvailable) return (false); if (ContentTypePtr) EncodeGzip = true; else { cptr = "no content-type"; ContentTypePtr = ""; EncodeGzip = false; } if (rqptr) { /* request related considerations */ if (!rqptr->rqHeader.AcceptEncodingGzip) { EncodeGzip = false; cptr = "client does not accept"; } else if (rqptr->rqResponse.ContentIsEncodedGzip || rqptr->rqResponse.ContentIsEncodedUnknown) { EncodeGzip = false; cptr = "response already encoded"; } else if (MATCH0 (ContentTypePtr, "application/pdf", 15) && rqptr->rqPathSet.ResponseGzip != MAPURL_RESPONSE_GZIP_ALL && rqptr->rqCgi.ContentEncodingGzip != CGI_OUTPUT_GZIP) { EncodeGzip = false; cptr = "PDF files are not compressed unless specified by SET rule"; } else if (rqptr->rqCgi.ContentEncodingGzip == CGI_OUTPUT_NO_GZIP) { EncodeGzip = false; cptr = "script requested response NOT to be GZIPed"; } else if (rqptr->rqPathSet.ResponseGzip == MAPURL_RESPONSE_GZIP_NONE && rqptr->rqCgi.ContentEncodingGzip != CGI_OUTPUT_GZIP) { /* note that the script requesting GZIP overrides the mapping rule */ EncodeGzip = false; cptr = "response=GZIP=none"; } else if (!rqptr->rqPathSet.ResponseGzip && ContentLength >= 0 && ContentLength < GZIP_MIN_CONTENT_LENGTH) { /* when fairly small avoid the compression overhead */ EncodeGzip = false; cptr = "minimal content-length"; } else if (rqptr->rqPathSet.ResponseGzip && rqptr->rqPathSet.ResponseGzip != MAPURL_RESPONSE_GZIP_ALL && rqptr->rqPathSet.ResponseGzip != MAPURL_RESPONSE_GZIP_NONE && (ContentLength < 0 || rqptr->rqPathSet.ResponseGzip >= ContentLength >> 10)) { EncodeGzip = false; if (WATCHING (rqptr, WATCH_RESPONSE)) { if (ContentLength >= 0) FaoToBuffer (cptr = ResponseKbytes, sizeof(ResponseKbytes), NULL, "response=GZIP=!UL (!ULkB)", rqptr->rqPathSet.ResponseGzip, ContentLength >> 10); else FaoToBuffer (cptr = ResponseKbytes, sizeof(ResponseKbytes), NULL, "response=GZIP=!UL (?kB)", rqptr->rqPathSet.ResponseGzip); } } else if (rqptr->rqResponse.NoGzip) { EncodeGzip = false; cptr = "response NOT to be GZIPed"; } } if (EncodeGzip) { /* content-type considerations */ if (MATCH6 (ContentTypePtr, "image/") && !MATCH3 (ContentTypePtr+6, "bmp") && !MATCH4 (ContentTypePtr+6, "tiff") && !MATCH8 (ContentTypePtr+6, "x-bitmap") && !MATCH8 (ContentTypePtr+6, "x-pixmap")) { /* most image content is already compressed (except the above) */ EncodeGzip = false; cptr = "compressed image"; } else if (MATCH6 (ContentTypePtr, "audio/") && !MATCH3 (ContentTypePtr+6, "wav") && !MATCH5 (ContentTypePtr+6, "x-wav")) { /* generally audio is already compressed (except WAVs AFAIK) */ EncodeGzip = false; cptr = "compressed audio"; } else if (MATCH6 (ContentTypePtr, "video/")) { /* video is (hopefully!) already compressed */ EncodeGzip = false; cptr = "compressed video"; } else if (MATCH0 (ContentTypePtr, "application/", 12) && (strstr (ContentTypePtr+12, "zip") || strstr (ContentTypePtr+12, "gtar") || strstr (ContentTypePtr+12, "archive") || strstr (ContentTypePtr+12, "stuffit") || strstr (ContentTypePtr+12, "x-compressed"))) { /* compressed application stream */ EncodeGzip = false; cptr = "compressed archive"; } else if (MATCH0 (ContentTypePtr, "multipart/", 10) && strstr (ContentTypePtr+10, "zip")) { /* same, same */ EncodeGzip = false; cptr = "compressed archive"; } else if (MATCH0 (ContentTypePtr, "application/pdf", 15) && rqptr->rqPathSet.ResponseGzip != MAPURL_RESPONSE_GZIP_ALL && rqptr->rqCgi.ContentEncodingGzip != CGI_OUTPUT_GZIP) { EncodeGzip = false; cptr = "PDF files are not compressed"; } else if (MATCH0 (ContentTypePtr, "application/x-shockwave-flash", 29)) { EncodeGzip = false; cptr = "Shockwave Flash files are not compressed"; } else if (MATCH0 (ContentTypePtr, "text/x-shtml", 12)) { EncodeGzip = false; cptr = "SSI source are not compressed"; } else if (ContentLength >= 0 && ContentLength < GZIP_MIN_CONTENT_LENGTH) { /* when fairly small avoid the compression overhead */ EncodeGzip = false; cptr = "minimal content-length"; } } if (WATCHING (rqptr, WATCH_RESPONSE)) if (!EncodeGzip) WatchThis (WATCHITM(rqptr), WATCH_RESPONSE, "DEFLATE no, !AZ", cptr); return (EncodeGzip); } /*****************************************************************************/ /* Initialize the ZLIB structure. Allocate memory from the ZLIB output buffer. Returns true and set the pointer to the ZLIB structure if successful. Returns false and leaves the pointer NULL if not. Reports any error via the usual WASD mechanism. */ BOOL GzipInflateBegin ( REQUEST_STRUCT *rqptr, GZIP_COMPRESS *gzptr, int BufferSize ) { int zstatus; z_stream *zsptr; /*********/ /* begin */ /*********/ if (WATCHMOD (rqptr, WATCH_MOD_NET)) WatchThis (WATCHITM(rqptr), WATCH_MOD_NET, "GzipInflateBegin()"); gzptr->InflateStartStream = true; gzptr->InflateEndOfStream = false; gzptr->InflateMemoryAllocated = rqptr->GzipMemoryAllocated = 0; zsptr = (z_stream*)VmGetHeap (rqptr, sizeof(z_stream)); zsptr->opaque = rqptr; zsptr->zalloc = &GzipAlloc; zsptr->zfree = &GzipFree; /* plus 16 is the magic number to remove a gzip header and trailer!! */ zstatus = ZlibInflateInit2Fn (zsptr, GzipInflateWindowBits + 16, ZLIB_VERSION, sizeof(z_stream)); if (WATCHMOD (rqptr, WATCH_MOD_NET)) WatchThis (WATCHITM(rqptr), WATCH_MOD_NET, "ZlibInflateInit2Fn(!UL+16) !SL !UL !&Z", GzipDeflateWindowBits, zstatus, rqptr->GzipMemoryAllocated, zsptr->msg); if (zstatus == Z_OK) { gzptr->InflateZstreamPtr = zsptr; gzptr->InflateBufferSize = BufferSize; gzptr->InflateBufferPtr = VmGetHeap (rqptr, BufferSize); return (true); } rqptr->rqResponse.HttpStatus = 500; ErrorGeneral (rqptr, GzipZlibMsg(zsptr,zstatus), FI_LI); return (false); } /*****************************************************************************/ /* Deallocate ZLIB structures. Returns true if there was no last-minute problem reported by ZLIB, false and reports the error if there was. */ BOOL GzipInflateEnd ( REQUEST_STRUCT *rqptr, GZIP_COMPRESS *gzptr ) { int zstatus; z_stream *zsptr; /*********/ /* begin */ /*********/ if (gzptr->InflateZstreamPtr) { if (WATCHING (rqptr, WATCH_RESPONSE)) if (!WATCHING (rqptr, WATCH_NETWORK)) WatchThis (WATCHITM(rqptr), WATCH_RESPONSE, "INFLATE !UL->!UL bytes, !UL% (!ULkB)", gzptr->InflateBytesIn, gzptr->InflateBytesOut, PercentOf32(gzptr->InflateBytesOut,gzptr->InflateBytesIn), rqptr->GzipMemoryAllocated >> 10); } gzptr->InflateEndOfStream = true; InstanceMutexLock (INSTANCE_MUTEX_HTTPD); AccountingPtr->GzipInflateCount++; AccountingPtr->GzipInflateBytesIn64 += gzptr->InflateBytesIn; AccountingPtr->GzipInflateBytesOut64 += gzptr->InflateBytesOut; InstanceMutexUnLock (INSTANCE_MUTEX_HTTPD); if (zsptr = gzptr->InflateZstreamPtr) { gzptr->InflateMemoryAllocated = rqptr->GzipMemoryAllocated; zstatus = ZlibInflateEndFn (zsptr); /* this as NULL indicates that inflate is no longer in use */ gzptr->InflateZstreamPtr = NULL; } else zstatus = Z_OK; if (WATCHMOD (rqptr, WATCH_MOD_NET)) WatchThis (WATCHITM(rqptr), WATCH_MOD_NET, "ZlibInflateEnd() !SL", zstatus); if (zstatus == Z_OK) return (true); if (zstatus == Z_BUF_ERROR) return (true); rqptr->rqResponse.HttpStatus = 500; ErrorGeneral (rqptr, GzipZlibMsg(zsptr,zstatus), FI_LI); return (false); } /*****************************************************************************/ /* Data read from the network is intercepted and passed to this function. It uses the ZLIB inflate function to decompress the data into the WASD buffer allocated during initialization. The 'DataPtr' and 'DataLength' are then modified to reflect the decompressed data and it can then be further processed. Return true to indicate success, false if the write (and request) should be terminated due to error (which is also reported by the usual WASD mechanism). */ BOOL GzipInflate ( REQUEST_STRUCT *rqptr, GZIP_COMPRESS *gzptr, char **DataPtrPtr, int *DataLengthPtr ) { int zstatus; z_stream *zsptr; /*********/ /* begin */ /*********/ if (WATCHMOD (rqptr, WATCH_MOD_NET)) WatchThis (WATCHITM(rqptr), WATCH_MOD_NET, "GzipInflate() !&X !UL", *DataPtrPtr, *DataLengthPtr); zsptr = gzptr->InflateZstreamPtr; if (!zsptr->avail_in) { zsptr->next_in = *DataPtrPtr; zsptr->avail_in = *DataLengthPtr; } if (zsptr->avail_in || zsptr->total_in) { /* if we have data to process or have previously processed some */ if (!zsptr->avail_out) { /* output buffer needs (re)setting */ zsptr->next_out = gzptr->InflateBufferPtr; zsptr->avail_out = gzptr->InflateBufferSize; } if (WATCHMOD (rqptr, WATCH_MOD_NET)) WatchThis (WATCHITM(rqptr), WATCH_MOD_NET, "!UL(!UL)/!UL", zsptr->avail_in, gzptr->InflateAvailIn, zsptr->avail_out); zstatus = ZlibInflateFn (zsptr, Z_NO_FLUSH); gzptr->InflateBytesIn = zsptr->total_in; gzptr->InflateBytesOut = zsptr->total_out; if (zstatus == Z_OK && !zsptr->avail_out) gzptr->InflateBufferCount = gzptr->InflateBufferSize; else if (zstatus == Z_STREAM_END) gzptr->InflateBufferCount = gzptr->InflateBufferSize - zsptr->avail_out; else gzptr->InflateBufferCount = 0; if (WATCHMOD (rqptr, WATCH_MOD_NET)) WatchThis (WATCHITM(rqptr), WATCH_MOD_NET, "!SL !UL/!UL !UL !UL->!UL !&Z", zstatus, zsptr->avail_in, zsptr->avail_out, gzptr->InflateBufferCount, gzptr->InflateBytesIn, gzptr->InflateBytesOut, zsptr->msg); if (gzptr->InflateBufferCount) if (WATCHING (rqptr, WATCH_NETWORK)) WatchThis (WATCHITM(rqptr), WATCH_NETWORK, "INFLATE !UL->!UL bytes, !UL% (!ULkB)", gzptr->InflateBytesIn, gzptr->InflateBytesOut, PercentOf32(gzptr->InflateBytesOut,gzptr->InflateBytesIn), rqptr->GzipMemoryAllocated >> 10); } else { /* no data has ever been processed, just clean up the ZLIB stuff */ zstatus = Z_STREAM_END; gzptr->InflateBufferCount = 0; } if (zstatus == Z_OK || zstatus == Z_STREAM_END) { if (zstatus == Z_STREAM_END) if (!GzipInflateEnd (rqptr, gzptr)) return (false); gzptr->InflateAvailIn = zsptr->avail_in; *DataPtrPtr = gzptr->InflateBufferPtr; *DataLengthPtr = gzptr->InflateBufferCount; return (true); } rqptr->rqResponse.HttpStatus = 500; ErrorGeneral (rqptr, GzipZlibMsg(zsptr,zstatus), FI_LI); GzipInflateEnd (rqptr, gzptr); return (false); } /*****************************************************************************/ /* Allocate request heap memory on behalf of ZLIB demands. Shouldn't leak any memory this way (not that ZLIB is likely to). */ char* GzipAlloc ( REQUEST_STRUCT *rqptr, int items, int size ) { /*********/ /* begin */ /*********/ if (WATCH_MODULE(WATCH_MOD_NET)) WatchThis (WATCHALL, WATCH_MOD_NET, "GzipAlloc() !UL !UL", items, size); rqptr->GzipMemoryAllocated += size*items; return (VmGetHeap (rqptr, size*items)); } /*****************************************************************************/ /* Free GzipAlloc()ed request heap memory. */ GzipFree ( REQUEST_STRUCT *rqptr, char *mptr ) { /*********/ /* begin */ /*********/ if (WATCH_MODULE(WATCH_MOD_NET)) WatchThis (WATCHALL, WATCH_MOD_NET, "GzipFree()"); VmFreeFromHeap (rqptr, mptr, FI_LI); } /*****************************************************************************/