[0001]
[0002]
[0003]
[0004]
[0005]
[0006]
[0007]
[0008]
[0009]
[0010]
[0011]
[0012]
[0013]
[0014]
[0015]
[0016]
[0017]
[0018]
[0019]
[0020]
[0021]
[0022]
[0023]
[0024]
[0025]
[0026]
[0027]
[0028]
[0029]
[0030]
[0031]
[0032]
[0033]
[0034]
[0035]
[0036]
[0037]
[0038]
[0039]
[0040]
[0041]
[0042]
[0043]
[0044]
[0045]
[0046]
[0047]
[0048]
[0049]
[0050]
[0051]
[0052]
[0053]
[0054]
[0055]
[0056]
[0057]
[0058]
[0059]
[0060]
[0061]
[0062]
[0063]
[0064]
[0065]
[0066]
[0067]
[0068]
[0069]
[0070]
[0071]
[0072]
[0073]
[0074]
[0075]
[0076]
[0077]
[0078]
[0079]
[0080]
[0081]
[0082]
[0083]
[0084]
[0085]
[0086]
[0087]
[0088]
[0089]
[0090]
[0091]
[0092]
[0093]
[0094]
[0095]
[0096]
[0097]
[0098]
[0099]
[0100]
[0101]
[0102]
[0103]
[0104]
[0105]
[0106]
[0107]
[0108]
[0109]
[0110]
[0111]
[0112]
[0113]
[0114]
[0115]
[0116]
[0117]
[0118]
[0119]
[0120]
[0121]
[0122]
[0123]
[0124]
[0125]
[0126]
[0127]
[0128]
[0129]
[0130]
[0131]
[0132]
[0133]
[0134]
[0135]
[0136]
[0137]
[0138]
[0139]
[0140]
[0141]
[0142]
[0143]
[0144]
[0145]
[0146]
[0147]
[0148]
[0149]
[0150]
[0151]
[0152]
[0153]
[0154]
[0155]
[0156]
[0157]
[0158]
[0159]
[0160]
[0161]
[0162]
[0163]
[0164]
[0165]
[0166]
[0167]
[0168]
[0169]
[0170]
[0171]
[0172]
[0173]
[0174]
[0175]
[0176]
[0177]
[0178]
[0179]
[0180]
[0181]
[0182]
[0183]
[0184]
[0185]
[0186]
[0187]
[0188]
[0189]
[0190]
[0191]
[0192]
[0193]
[0194]
[0195]
[0196]
[0197]
[0198]
[0199]
[0200]
[0201]
[0202]
[0203]
[0204]
[0205]
[0206]
[0207]
[0208]
[0209]
[0210]
[0211]
[0212]
[0213]
[0214]
[0215]
[0216]
[0217]
[0218]
[0219]
[0220]
[0221]
[0222]
[0223]
[0224]
[0225]
[0226]
[0227]
[0228]
[0229]
[0230]
[0231]
[0232]
[0233]
[0234]
[0235]
[0236]
[0237]
[0238]
[0239]
[0240]
[0241]
[0242]
[0243]
[0244]
[0245]
[0246]
[0247]
[0248]
[0249]
[0250]
[0251]
[0252]
[0253]
[0254]
[0255]
[0256]
[0257]
[0258]
[0259]
[0260]
[0261]
[0262]
[0263]
[0264]
[0265]
[0266]
[0267]
[0268]
[0269]
[0270]
[0271]
[0272]
[0273]
[0274]
[0275]
[0276]
[0277]
[0278]
[0279]
[0280]
[0281]
[0282]
[0283]
[0284]
[0285]
[0286]
[0287]
[0288]
[0289]
[0290]
[0291]
[0292]
[0293]
[0294]
[0295]
[0296]
[0297]
[0298]
[0299]
[0300]
[0301]
[0302]
[0303]
[0304]
[0305]
[0306]
[0307]
[0308]
[0309]
[0310]
[0311]
[0312]
[0313]
[0314]
[0315]
[0316]
[0317]
[0318]
[0319]
[0320]
[0321]
[0322]
[0323]
[0324]
[0325]
[0326]
[0327]
[0328]
[0329]
[0330]
[0331]
[0332]
[0333]
[0334]
[0335]
[0336]
[0337]
[0338]
[0339]
[0340]
[0341]
[0342]
[0343]
[0344]
[0345]
[0346]
[0347]
[0348]
[0349]
[0350]
[0351]
[0352]
[0353]
[0354]
[0355]
[0356]
[0357]
[0358]
[0359]
[0360]
[0361]
[0362]
[0363]
[0364]
[0365]
[0366]
[0367]
[0368]
[0369]
[0370]
[0371]
[0372]
[0373]
[0374]
[0375]
[0376]
[0377]
[0378]
[0379]
[0380]
[0381]
[0382]
[0383]
[0384]
[0385]
[0386]
[0387]
[0388]
[0389]
[0390]
[0391]
[0392]
[0393]
[0394]
[0395]
[0396]
[0397]
[0398]
[0399]
[0400]
[0401]
[0402]
[0403]
[0404]
[0405]
[0406]
[0407]
[0408]
[0409]
[0410]
[0411]
[0412]
[0413]
[0414]
[0415]
[0416]
[0417]
[0418]
[0419]
[0420]
[0421]
[0422]
[0423]
[0424]
[0425]
[0426]
[0427]
[0428]
[0429]
[0430]
[0431]
[0432]
[0433]
[0434]
[0435]
[0436]
[0437]
[0438]
[0439]
[0440]
[0441]
[0442]
[0443]
[0444]
[0445]
[0446]
[0447]
[0448]
[0449]
[0450]
[0451]
[0452]
[0453]
[0454]
[0455]
[0456]
[0457]
[0458]
[0459]
[0460]
[0461]
[0462]
[0463]
[0464]
[0465]
[0466]
[0467]
[0468]
[0469]
[0470]
[0471]
[0472]
[0473]
[0474]
[0475]
[0476]
[0477]
[0478]
[0479]
[0480]
[0481]
[0482]
[0483]
[0484]
[0485]
[0486]
[0487]
[0488]
[0489]
[0490]
[0491]
[0492]
[0493]
[0494]
[0495]
[0496]
[0497]
[0498]
[0499]
[0500]
[0501]
[0502]
[0503]
[0504]
[0505]
[0506]
[0507]
[0508]
[0509]
[0510]
[0511]
[0512]
[0513]
[0514]
[0515]
[0516]
[0517]
[0518]
[0519]
[0520]
[0521]
[0522]
[0523]
[0524]
[0525]
[0526]
[0527]
[0528]
[0529]
[0530]
[0531]
[0532]
[0533]
[0534]
[0535]
[0536]
[0537]
[0538]
[0539]
[0540]
[0541]
[0542]
[0543]
[0544]
[0545]
[0546]
[0547]
[0548]
[0549]
[0550]
[0551]
[0552]
[0553]
[0554]
[0555]
[0556]
[0557]
[0558]
[0559]
[0560]
[0561]
[0562]
[0563]
[0564]
[0565]
[0566]
[0567]
[0568]
[0569]
[0570]
[0571]
[0572]
[0573]
[0574]
[0575]
[0576]
[0577]
[0578]
[0579]
[0580]
[0581]
[0582]
[0583]
[0584]
[0585]
[0586]
[0587]
[0588]
[0589]
[0590]
[0591]
[0592]
[0593]
[0594]
[0595]
[0596]
[0597]
[0598]
[0599]
[0600]
[0601]
[0602]
[0603]
[0604]
[0605]
[0606]
[0607]
[0608]
[0609]
[0610]
[0611]
[0612]
[0613]
[0614]
[0615]
[0616]
[0617]
[0618]
[0619]
[0620]
[0621]
[0622]
[0623]
[0624]
[0625]
[0626]
[0627]
[0628]
[0629]
[0630]
[0631]
[0632]
[0633]
[0634]
[0635]
[0636]
[0637]
[0638]
[0639]
[0640]
[0641]
[0642]
[0643]
[0644]
[0645]
[0646]
[0647]
[0648]
[0649]
[0650]
[0651]
[0652]
[0653]
[0654]
[0655]
[0656]
[0657]
[0658]
[0659]
[0660]
[0661]
[0662]
[0663]
[0664]
[0665]
[0666]
[0667]
[0668]
[0669]
[0670]
[0671]
[0672]
[0673]
[0674]
[0675]
[0676]
[0677]
[0678]
[0679]
[0680]
[0681]
[0682]
[0683]
[0684]
[0685]
[0686]
[0687]
[0688]
[0689]
[0690]
[0691]
[0692]
[0693]
[0694]
[0695]
[0696]
[0697]
[0698]
[0699]
[0700]
[0701]
[0702]
[0703]
[0704]
[0705]
[0706]
[0707]
[0708]
[0709]
[0710]
[0711]
[0712]
[0713]
[0714]
[0715]
[0716]
[0717]
[0718]
[0719]
[0720]
[0721]
[0722]
[0723]
[0724]
[0725]
[0726]
[0727]
[0728]
[0729]
[0730]
[0731]
[0732]
[0733]
[0734]
[0735]
[0736]
[0737]
[0738]
[0739]
[0740]
[0741]
[0742]
[0743]
[0744]
[0745]
[0746]
[0747]
[0748]
[0749]
[0750]
[0751]
[0752]
[0753]
[0754]
[0755]
[0756]
[0757]
[0758]
[0759]
[0760]
[0761]
[0762]
[0763]
[0764]
[0765]
[0766]
[0767]
[0768]
[0769]
[0770]
[0771]
[0772]
[0773]
[0774]
[0775]
[0776]
[0777]
[0778]
[0779]
[0780]
[0781]
[0782]
[0783]
[0784]
[0785]
[0786]
[0787]
[0788]
[0789]
[0790]
[0791]
[0792]
[0793]
[0794]
[0795]
[0796]
[0797]
[0798]
[0799]
[0800]
[0801]
[0802]
[0803]
[0804]
[0805]
[0806]
[0807]
[0808]
[0809]
[0810]
[0811]
[0812]
[0813]
[0814]
[0815]
[0816]
[0817]
[0818]
[0819]
[0820]
[0821]
[0822]
[0823]
[0824]
[0825]
[0826]
[0827]
[0828]
[0829]
[0830]
[0831]
[0832]
[0833]
[0834]
[0835]
[0836]
[0837]
[0838]
[0839]
[0840]
[0841]
[0842]
[0843]
[0844]
[0845]
[0846]
[0847]
[0848]
[0849]
[0850]
[0851]
[0852]
[0853]
[0854]
[0855]
[0856]
[0857]
[0858]
[0859]
[0860]
[0861]
[0862]
[0863]
[0864]
[0865]
[0866]
[0867]
[0868]
[0869]
[0870]
[0871]
[0872]
[0873]
[0874]
[0875]
[0876]
[0877]
[0878]
[0879]
[0880]
[0881]
[0882]
[0883]
[0884]
[0885]
[0886]
[0887]
[0888]
[0889]
[0890]
[0891]
[0892]
[0893]
[0894]
[0895]
[0896]
[0897]
[0898]
[0899]
[0900]
[0901]
[0902]
[0903]
[0904]
[0905]
[0906]
[0907]
[0908]
[0909]
[0910]
[0911]
[0912]
[0913]
[0914]
[0915]
[0916]
[0917]
[0918]
[0919]
[0920]
[0921]
[0922]
[0923]
[0924]
[0925]
[0926]
[0927]
[0928]
[0929]
[0930]
[0931]
[0932]
[0933]
[0934]
[0935]
[0936]
[0937]
[0938]
[0939]
[0940]
[0941]
[0942]
[0943]
[0944]
[0945]
[0946]
[0947]
[0948]
[0949]
[0950]
[0951]
[0952]
[0953]
[0954]
[0955]
[0956]
[0957]
[0958]
[0959]
[0960]
[0961]
[0962]
[0963]
[0964]
[0965]
[0966]
[0967]
[0968]
[0969]
[0970]
[0971]
[0972]
[0973]
[0974]
[0975]
[0976]
[0977]
[0978]
[0979]
[0980]
[0981]
[0982]
[0983]
[0984]
[0985]
[0986]
[0987]
[0988]
[0989]
[0990]
[0991]
[0992]
[0993]
[0994]
[0995]
[0996]
[0997]
[0998]
[0999]
[1000]
[1001]
[1002]
[1003]
[1004]
[1005]
[1006]
[1007]
[1008]
[1009]
[1010]
[1011]
[1012]
[1013]
[1014]
[1015]
[1016]
[1017]
[1018]
[1019]
[1020]
[1021]
[1022]
[1023]
[1024]
[1025]
[1026]
[1027]
[1028]
[1029]
[1030]
[1031]
[1032]
[1033]
[1034]
[1035]
[1036]
[1037]
[1038]
[1039]
[1040]
[1041]
[1042]
[1043]
[1044]
[1045]
[1046]
[1047]
[1048]
[1049]
[1050]
[1051]
[1052]
[1053]
[1054]
[1055]
[1056]
[1057]
[1058]
[1059]
[1060]
[1061]
[1062]
[1063]
[1064]
[1065]
[1066]
[1067]
[1068]
[1069]
[1070]
[1071]
[1072]
[1073]
[1074]
[1075]
[1076]
[1077]
[1078]
[1079]
[1080]
[1081]
[1082]
[1083]
[1084]
[1085]
[1086]
[1087]
[1088]
[1089]
[1090]
[1091]
[1092]
[1093]
[1094]
[1095]
[1096]
[1097]
[1098]
[1099]
[1100]
[1101]
[1102]
[1103]
[1104]
[1105]
[1106]
[1107]
[1108]
[1109]
[1110]
[1111]
[1112]
[1113]
[1114]
[1115]
[1116]
[1117]
[1118]
[1119]
[1120]
[1121]
[1122]
[1123]
[1124]
[1125]
[1126]
[1127]
[1128]
[1129]
[1130]
[1131]
[1132]
[1133]
[1134]
[1135]
[1136]
[1137]
[1138]
[1139]
[1140]
[1141]
[1142]
[1143]
[1144]
[1145]
[1146]
[1147]
[1148]
[1149]
[1150]
[1151]
[1152]
[1153]
[1154]
[1155]
[1156]
[1157]
[1158]
[1159]
[1160]
[1161]
[1162]
[1163]
[1164]
[1165]
[1166]
[1167]
[1168]
[1169]
[1170]
[1171]
[1172]
[1173]
[1174]
[1175]
[1176]
[1177]
[1178]
[1179]
[1180]
[1181]
[1182]
[1183]
[1184]
[1185]
[1186]
[1187]
[1188]
[1189]
[1190]
[1191]
[1192]
[1193]
[1194]
[1195]
[1196]
[1197]
[1198]
[1199]
[1200]
[1201]
[1202]
[1203]
[1204]
[1205]
[1206]
[1207]
[1208]
[1209]
[1210]
[1211]
[1212]
[1213]
[1214]
[1215]
[1216]
[1217]
[1218]
[1219]
[1220]
[1221]
[1222]
[1223]
[1224]
[1225]
[1226]
[1227]
[1228]
[1229]
[1230]
[1231]
[1232]
[1233]
[1234]
[1235]
[1236]
[1237]
[1238]
[1239]
[1240]
[1241]
[1242]
[1243]
[1244]
[1245]
[1246]
[1247]
[1248]
[1249]
[1250]
[1251]
[1252]
[1253]
[1254]
[1255]
[1256]
[1257]
[1258]
[1259]
[1260]
[1261]
[1262]
[1263]
[1264]
[1265]
[1266]
[1267]
[1268]
[1269]
[1270]
[1271]
[1272]
[1273]
[1274]
[1275]
[1276]
[1277]
[1278]
[1279]
[1280]
[1281]
[1282]
[1283]
[1284]
[1285]
[1286]
[1287]
[1288]
[1289]
[1290]
[1291]
[1292]
[1293]
[1294]
[1295]
[1296]
[1297]
[1298]
[1299]
[1300]
[1301]
[1302]
[1303]
[1304]
[1305]
[1306]
[1307]
[1308]
[1309]
[1310]
[1311]
[1312]
[1313]
[1314]
[1315]
[1316]
[1317]
[1318]
[1319]
[1320]
[1321]
[1322]
[1323]
[1324]
[1325]
[1326]
[1327]
[1328]
[1329]
[1330]
[1331]
[1332]
[1333]
[1334]
[1335]
[1336]
[1337]
[1338]
[1339]
[1340]
[1341]
[1342]
[1343]
[1344]
[1345]
[1346]
[1347]
[1348]
[1349]
[1350]
[1351]
[1352]
[1353]
[1354]
[1355]
[1356]
[1357]
[1358]
[1359]
[1360]
[1361]
[1362]
[1363]
[1364]
[1365]
[1366]
[1367]
[1368]
[1369]
[1370]
[1371]
[1372]
[1373]
[1374]
[1375]
[1376]
[1377]
[1378]
[1379]
[1380]
[1381]
[1382]
[1383]
[1384]
[1385]
[1386]
[1387]
[1388]
[1389]
[1390]
[1391]
[1392]
[1393]
[1394]
[1395]
[1396]
[1397]
[1398]
[1399]
[1400]
[1401]
[1402]
[1403]
[1404]
[1405]
[1406]
[1407]
[1408]
[1409]
[1410]
[1411]
[1412]
[1413]
[1414]
[1415]
[1416]
[1417]
[1418]
[1419]
[1420]
[1421]
[1422]
[1423]
[1424]
[1425]
[1426]
[1427]
[1428]
[1429]
[1430]
[1431]
[1432]
[1433]
[1434]
[1435]
[1436]
[1437]
[1438]
[1439]
[1440]
[1441]
[1442]
[1443]
[1444]
[1445]
[1446]
[1447]
[1448]
[1449]
[1450]
[1451]
[1452]
[1453]
[1454]
[1455]
[1456]
[1457]
[1458]
[1459]
[1460]
[1461]
[1462]
[1463]
[1464]
[1465]
[1466]
[1467]
[1468]
[1469]
[1470]
[1471]
[1472]
[1473]
[1474]
[1475]
[1476]
[1477]
[1478]
[1479]
[1480]
[1481]
[1482]
[1483]
[1484]
[1485]
[1486]
[1487]
[1488]
[1489]
[1490]
[1491]
[1492]
[1493]
[1494]
[1495]
[1496]
[1497]
[1498]
[1499]
[1500]
[1501]
[1502]
[1503]
[1504]
[1505]
[1506]
[1507]
[1508]
[1509]
[1510]
[1511]
[1512]
[1513]
[1514]
[1515]
[1516]
[1517]
[1518]
[1519]
[1520]
[1521]
[1522]
[1523]
[1524]
[1525]
[1526]
[1527]
[1528]
[1529]
[1530]
[1531]
[1532]
[1533]
[1534]
[1535]
[1536]
[1537]
[1538]
[1539]
[1540]
[1541]
[1542]
[1543]
[1544]
[1545]
[1546]
[1547]
[1548]
[1549]
[1550]
[1551]
[1552]
[1553]
[1554]
[1555]
[1556]
[1557]
[1558]
[1559]
[1560]
[1561]
[1562]
[1563]
[1564]
[1565]
[1566]
[1567]
[1568]
[1569]
[1570]
[1571]
[1572]
[1573]
[1574]
[1575]
[1576]
[1577]
[1578]
[1579]
[1580]
[1581]
[1582]
[1583]
[1584]
[1585]
[1586]
[1587]
[1588]
[1589]
[1590]
[1591]
[1592]
[1593]
[1594]
[1595]
[1596]
[1597]
[1598]
[1599]
[1600]
[1601]
[1602]
[1603]
[1604]
[1605]
[1606]
[1607]
[1608]
[1609]
[1610]
[1611]
[1612]
[1613]
[1614]
[1615]
[1616]
[1617]
[1618]
[1619]
[1620]
[1621]
[1622]
[1623]
[1624]
[1625]
[1626]
[1627]
[1628]
[1629]
[1630]
[1631]
[1632]
[1633]
[1634]
[1635]
[1636]
[1637]
[1638]
[1639]
[1640]
[1641]
[1642]
[1643]
[1644]
[1645]
[1646]
[1647]
[1648]
[1649]
[1650]
[1651]
[1652]
[1653]
[1654]
[1655]
[1656]
[1657]
[1658]
[1659]
[1660]
[1661]
[1662]
[1663]
[1664]
[1665]
[1666]
[1667]
[1668]
[1669]
[1670]
[1671]
[1672]
[1673]
[1674]
[1675]
[1676]
[1677]
[1678]
[1679]
[1680]
[1681]
[1682]
[1683]
[1684]
[1685]
[1686]
[1687]
[1688]
[1689]
[1690]
[1691]
[1692]
[1693]
[1694]
[1695]
[1696]
[1697]
[1698]
[1699]
[1700]
[1701]
[1702]
[1703]
[1704]
[1705]
[1706]
[1707]
[1708]
[1709]
[1710]
[1711]
[1712]
[1713]
[1714]
[1715]
[1716]
+-------------------------------------+
| WASD HTTP SERVER - "NUTS AND BOLTS" |
+-------------------------------------+
WASD VMS Web Services
Copyright (C) 1996-2021 Mark G. Daniel.
Revision: v12.0.0 (September 2021)
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this software except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
Versions prior to v12.0.0 were distributed under a GPL licence.
http://www.gnu.org/licenses/gpl.txt
Thanks to BETA Testing Sites
----------------------------
WASD has become a large package and requests for additional functionality
frequent enough that the relatively long development-testing-refinement-
release cycle of the first few years is no longer possible. It is no longer
feasable for the one author to code and exhaustively test all the functionality
of each new release. The role of BETA testing has become indispensable.
Thanks to all who participate in this part of the cycle, in particular (and in
no significant order);
Jeremy Begg VSM Software Services, Adelaide, South Australia
Alex Daniels themail, United Kingdom
DECUServe eisner.decus.org, MA, USA
John Dite Compinia GmbH & Co.
Victoriano Giralt University of Malaga, Spain
Dr Klaus-Werner Gurgle Hamburg University, Germany
Alexander Ivanov S&B Joint Stock Company, St Petersburg, Russia
Ruslan Laishev Delta Telecom, St Petersburg, Russia
Tom Linden Kednos Enterprises, California, USA
Jean-Pierre Petit ESME-Sudria, Paris, France
Jean-François Piéronne European WASD Advocate :^), Paris, France
Geoff Roberts St Mark's College, Pt Pirie, South Australia
There may be others but these are the ones who pester me with email. As the
BETA test sites are also often the ones requesting additional functionality
they should also be thanked for pushing along WASD's capability set.
2017 Confession Time: actually formal BETA testing now, and for some time, is
seldom undertaken. The VMS community has shrunk to a point where there is an
insufficient pool for a meaningful BETA test cycle. So, I do the best I can,
and then release a <major>.<minor>.0. The moral - beware any point-point-zero
release as effectively it has become the field-test phase of the life-cycle!
Brief Introduction to HTTPd Code
--------------------------------
This document is designed to be only a broad overview of the basic workings of
the HTTP server. It is derived from an original layed-up document, "Nut and
Bolts", which was abandoned in favour of something requiring a little less
up-keep (therefore increasing it's chances of being up-to-date :^) and that
could be kept with the server code itself. This description only covers the
server image itself, not the full suite of WASD VMS Web Services software.
General Design
--------------
(Almost) without apology the server is a large, monolithic, shall we be kind
and say old-fashioned, piece of software. Lots of things would be done
differently if it was being started over. Other things wouldn't be. The code
is always attempting to do things faster or more efficiently (especially
because it's VMS) and so it's a bit clunky in places. Other clunkiness is due
entirely to the author ;^)
Server Behaviour
----------------
The HTTPd executes permanently on the server host, listening for client
connection requests on TCP/IP port 80 (by default). It provides concurrent
services for a (technically) unlimitted number of clients (constrained only by
the server resources). When a client connects the server performs the following
tasks:
1. creates a thread for this request (this term does not denote the use of
DECthreads or other specific thread library, just a thread of execution)
2. reads and analyzes the HTTP request sent ...
o initiates transfer of the requested file, either from the
file system or from the file cache
o initiates processing of an SSI file
o initiates directory listing
o initiates processing of a clickable-image mapping file
o initiates file/directory create/update
o initiates server administration
o initiates web file-system update
o creates a (detached) process to execute a CGI(plus) script with:
- SYS$COMMAND and SYS$OUTPUT assigned to intermediate mailboxes
(essentially pipes)
- SYS$INPUT logical name providing a mailbox allowing the script to read
the raw request body (if any)
- CGIPLUSIN logical name providing a mailbox allowing a CGIplus
(persistent) script to read CGI variables
- (for non-CGIplus) DCL symbols containing CGI-compilant variables
- for the life of the script process HTTPd:
* controls the essential behaviour via its SYS$COMMAND
* receives data written to its SYS$OUTPUT, writing this to the client
* supplies the body of a (POSTed) request via SYS$INPUT
o creates a (detached) process to execute a WebSocket script with:
- SYS$COMMAND assigned to intermediate mailbox (controls process)
- SYS$OUTPUT assigned to mailbox until WebSocket established
- CGIPLUSIN logical name providing a mailbox allowing the WebSocket
(CGIplus, persistent) script to read CGI variables
- WEBSOCKET_INPUT logical name providing a mailbox to read client
data for the life of the request
- WEBSOCKET_OUTPUT logical name providing a mailbox to write data
to the client for the life of the request
- for the life of the script process:
* controls the essential behaviour via its SYS$COMMAND
* disconnects the SYS$OUTPUT and CGIPLUSIN mailboxes once WebSocket
is established and allows another WebSocket request to be accepted
* manages multiple WEBSOCKET_INPUT and WEBSOCKET_OUTPUT mailboxes
(i.e. multiple clients) connected to the process
o creates a (detached) process to execute a "Raw"Socket script with:
- RAWSOCKET_INPUT logical name providing a mailbox to read client
data for the life of the request
- RAWSOCKET_OUTPUT logical name providing a mailbox to write data
to the client for the life of the request
- other aspects substantially as per WebSocket scripting
o creates a network process to execute a CGI or OSU script
* controls the essential behaviour of a CGI script, providing the
CGI-compilant variable and receiving CGI-compilant output
* provides an emulation of the native OSU scripting environment
3. closes the connection to the client and disposes of the thread
For I/O intensive activities like file transfer and directory listing, the
AST-driven code provides an efficient, multi- threaded environment for the
concurrent serving of multiple clients.
Multi-Threaded
--------------
The WASD HTTPd is written to exploit VMS operating system characteristics
allowing the straight-forward implementation of event-driven, multi-threaded
code. Asynchronous System Traps (ASTs), or software interrupts, at the
conclusion of an I/O (or other) event allow functions to be activated to post-
process the event. The event traps are automatically queued on a FIFO basis,
allowing a series of events to be sequentially processed. When not responding
to an event the process is quiescent, or otherwise occupied, effectively
interleaving I/O and processing, and allowing a sophisticated client multi-
threading.
Multi-threaded code is inherently more complex than single-threaded code, and
there are issues involved in the synchronization of some activities in such
an environment. Fortunately VMS handles many of these issues internally.
After connection acceptance, all of the processing done within the server is at
USER mode AST delivery level, and for all intents and purposes the processing
done therein is atomic, implicitly handling its own synchronization issues.
The HTTPd is written to make longer duration activities, such as the transfer
of a file's contents, event-driven. Other, shorter duration activites, such as
accepting a client connection request, are handled synchronously.
It is worth noting that with asynchronous, and AST-driven output, the data
being written must be guaranteed to exist without modification for the duration
of the write (indicated by completion AST delivery). This means data written
must be static or in buffers that persist with the thread. Function-local
(automatic) storage cannot be used. The server allocates dynamic storage for
general (e.g. output buffering) or specific (e.g. response headers) uses.
AST Behaviour
-------------
With server functions having AST capability, in particular $QIO, the server is
designed to rely on the AST routine to report any error, including both those
that occur during the IO operation and any that occur when initiating the IO
(which would normally prevent it being queued) even if that requires directly
setting the IO status block with the offending status and explicitly declaring
the AST. This eliminates any ambiguity about under what conditions ASTs are
delivered ... ASTs are always delivered.
If a call to a server function with AST capability does not supply an AST
routine then it must check the return status to determine whether it can
continue processing. If it supplies an AST routine address then it must not
act on any error status returned, it must allow the AST routine to process
according to the IO status block status.
Server "Tasks"
--------------
Each request can have one or more tasks executed sequentially to fullfil the
request. This occurs most obviously with Server-Side Includes (SSI, the HTML
pre-processor) but also, to a more limited extent, with directory listing and
its read-me file inclusion. A task is more-or-less defined as one of:
o transfer file from file-system or cache
o directory listing
o SSI interpretation
o DCL execution or script processing
o DECnet script processing
o POST/PUT processing
o WebDAV processing
o WebSocket processing
o proxy processing
o update facility processing
Each one of the associated modules executes relatively independently. Before
commencing a task, a next-task pointer can be set to the function required to
execute at the conclusion of that task. At that conclusion, the next- task
functionality checks for a specified task to start or continue. If it has been
specified control is passed to that next-task function via an AST.
Some tasks can only be called once per request.
Other tasks have the possibility of being called within other tasks or multiple
times serially during a request. An example is the transfer file task
(non-cache), which can be used within directory listings to insert read-me
files, and when "<!-#include"ing multiple files within an SSI document.
Two tasks, the directory listing and SSI interpretation tasks, can be called
multiple times and can also have concurrent instances running. For example, an
SSI file can <!-#include another SSI file, nesting the SSI execution. The same
SSI document can have an embedded directory listing that contains an SSI
read-me file with another directory listing. Can get quite convoluted! The
tasks are inplemented using a linked-list FILO stack allowing this nesting.
SSI documents have a maximum depth for nesting, preventing recursive document
inclusion.
Memory Management
-----------------
Memory management is (almost) exclusively done using VMS system library virtual
memory routines. Using these rather that generic C library routines is a
deliberate design decision, and done with the following considerations.
o The library routines allow a more precise integrity checking and error
reporting for both the allocation and freeing of dynamic memory chunks.
o Separate zones provide some measure of isolation between threads of usage
and in this way assist in isolating any errors in memory usage.
o Separate zones may be created with characteristics tailored to specific
memory request profiles, reducing overhead and improving performance.
o A separate zone may be used for each request thread improving
deallocation performance at request disposal.
o Memory behaviour for the various aspects of server usage is more easily
monitored where separate zones represent distinct usages.
Per-request memory is managed in three distinct portions.
1. A fixed-size structure of dynamic memory is used to contain the core
request thread data. This is released at thread disposal. This is
allocated from a specific virtual memory zone tailored for fixed-size
management.
2. A heap of dynamically allocated memory is maintained during the life of a
thread structure. When a dynamic structure is required during request
processing it is allocated from a request-thread-specific zone of virtual
memory. This list is released in one operation at thread disposal, making
it quite efficient. Maintaining a thread-specific heap of vritual memory
also makes it easier to avoid memory leakage.
3. Per-task data structures are allocated using the above heap. These
structures are used to store task-specific data. If a task is used
multiple times within the one request (see above) the previous allocated
and now finished-with (but not deallocated) task structures can be
reused, reducing overhead.
Per-HTTP/2-connection memory is managed in two portions.
1. A fixed-size structure of dynamic memory is used to contain the HTTP/2
connection data. This is allocated from a specific virtual memory zone
tailored for fixed-size management and is released at disconnection.
2. A heap of dynamically allocated memory is maintained during the life of
an HTTP/2 structure. When a per-stream dynamic structure is required
to service a new stream (request) processing it is allocated from this
virtual memory zone. Of course, the actual in-progress request memory
is allocated as described in the per-request section above. Any memory
required in servicing the HTTP/2 networking of this connection is
allocated from the associated HTTP/2 connection's heap. This list is
released in one operation at HTTP/2 connection disposal.
Output Formatting
-----------------
The increasing complexity of the formatting of output (particularly with the
introduction of extended file specifications with ODS-5) prompted the
devlopment of a $FAO-like set of functions for writing formatted output. This
can write directly into the request dynamic network buffers or into static
character storage. The directives run parallel to those supported by $FAO,
although it is not a complete implementation and contains a number of variants
and extensions to that service's behaviour.
Output Buffering
----------------
To reduce the number of individual network writes, and thus provide significant
improvements in efficiency, generated output can be buffered into larger
packets before sending to the client. Not all modules use this (e.g. File.c)
and not all modules use it all of the time, but all modules work to implement a
seamless integration of output via this mechanism (best seen in the SSI.c
module).
The output buffer functionality underwent a complete redesign for v5.0. It is
now based on a list of one or more buffers that can be used in two modes.
1. When both an AST address and data to be buffered is supplied the
buffering function operates to fill one entire buffer, overflowing into a
second linked into the list. When that overflow occurs the first is
written to the network asynchronously (calling the supplied AST when
complete) and the second moved to the head of the list, effectively to
the front of the buffer, and so on.
2. When no AST address is supplied with the data to be buffered, it keeps on
filling buffers and adding others to the tail of the list as required,
creating a virtual buffer with no fixed length.
The first mode is used for general buffering (e.g. SSI and directory listings),
streaming data to the client in a sequence of larger aggregates. The second
mode is useful for functions that must block (e.g. those reporting on data
structures such as the file cache), write a lot of output for a report, and not
want to block general server activity for a long-ish period due to network
throughput (e.g. again the caching reports). In these cases the entire report
can be written to buffer, then simply asynchronously output, unblocking any
resource it may have held.
String Matching
---------------
Matching of strings is a pervasive and important function within the server.
Two types are supported; wildcard and regular expression. Wildcard matching is
generally much less expensive (in CPU cycles and time) than regular expression
matching and so should always be used unless the match explicitly requires it.
The StringMatchAndRgex() function attempts to improve the efficiency of both by
performing a preliminary pass to eliminate obvious mismatches using a
light-weight match. This either matches or doesn't or encounters a pattern
matching meta-character and abort to drop through for a full pattern matching.
Wildcard matching uses the '*' and '%' to match any zero or more, or any one
character respectively. The '*' wildcard can either be greedy or non-greedy
depending on the context (and for horrible historical reasons). It can also be
forced to be greedy by using two consecutive ('**'). By default it is not
greedy when matching request paths for mapping or authentication, and is greedy
at other times (matching strings within conditional testing, etc.)
Regular expression matching uses the essentials of the GNU RX 1.5 package.
Matching is case insensitive (in line with other WASD behaviour) and uses the
Posix EGREP pattern syntax and capabilities. Regular expressions are
differentiated from wildcard patterns by a leading '^' (non-significant)
character. Regular expression matching offers significant but fairly expensive
functionality. One of those expenses is expression compilation. WASD attempts
to eliminate this by pre-compiling expressions whereever feasable.
Both wildcard (implemented by the WASD matching function) and regular
expressions (integral) use the posix-style registers for noting the offsets of
the matched portions of the strings. These are then used for wildcard and
specific wildcard (i.e. "*'1") substitution where result strings provide this
(e.g. mapping 'pass' and 'redirect' rules). A maximum of nine such wildcard
substitutions are supported (one other, the zeroeth, is the full match).
Auto-Scripting
--------------
The WASD VMS HTTP server has the facility to automatically invoke a script to
process a non-HTML document (file). This facility is based on detecting the
MIME content data type (via the file's extension) and causing a transparent,
local redirection, invoking the script as if it was specified in the original
request.
Internal Directives and "Scripts"
---------------------------------
The HTTPd server detects certain paths and query strings as directives about
its behaviour. Certain paths are interpreted as pseudo, or internal scripts,
handled internal to the server. Other directives are passed in the query string
component of the request, and as reserved sequences cannot occur in normal
requests (an unlikely combination of characters has been selected).
Server Security and Privileges
------------------------------
As a major security design criterion the WASD environment has specified the use
of a non-privileged, non-SYSTEM, non-system-group server account. In this way
it begins with a fairly restricted and safe base, resources limited to those
world- accessable or explicitly allowed to the server account. For access to
selected, essential resources (such a privileged IP ports, for example 80)
selected privileges are enabled only on an as-required basis, then as soon as
the need for that privilege has passed disabled. Hence, the executable is
installed with the minimum required extended privileges which are operating and
used only as required during the course of processing. The server program is
almost always executing with only NETMBX and TMPMBX enabled . . . in other
words as a completely average VMS user!
Extended privileges are required for the purposes listed below:
o ALTPRI - allows the server account to raise it's priority above 4 if
enabled by the /PRIORITY= qualifier.
o CMKRNL - required for single use of $GRANTID system service in DCL.C
module. This rights identifier is used to mark detached WASD script
processes, so that DclCleanupScriptProcesses() can identify them.
Is also used to allow the PERSONA_MACRO kludge to do it's thing.
o DETACH - allows use of the VMS V6.2 and later $PERSONA services for
non-server-account scripting.
o OPER - required for the $BRKTHRU service
o PRMGBL - used when creating a permanent system-wide global section.
Shared memory in a permananet global section is used to store accounting
data (in between server incarnations) and the directive and response
buffer used for command-line and server admininstration menu control
purposes (amongst other things).
o PRMMBX - used by the subprocess scripting module to create permanent
mailboxes (much more efficient that creating a new set with each script
subprocess).
o PSWAPM - allows the server process to prevent itself from being
swapped out if specified by the /[NO]SWAP qualifier.
o SECURITY - required by ACME for some SYSUAF authentication
o SHMEM - used when creating the permanent system global section (VAX
only, see PRMGBL).
o SYSGBL - used when creating the permanent system global section
(see PRMGBL).
o SYSLCK - the VMS Distributed Lock Manager is explicitly used to
coordinate some activities on systems and clusters where multiple servers
are executing. This privilege is required to enqueue system-wide locks.
o SYSNAM - is actually not required with version 8.n and later.
o SYSPRV - used for various purposes, including creating sockets within
the privileged port range (1-1023) which includes port 80 of course.
Accessing configuration files (which can be protected from world access).
To ensure the server can stream-LF convert a file. It is also extensively
used to enable AUTHORIZED write access to the file system. If the
authorization configuration is set up to allow write access to selected
portions of the Web-space (by default it's not and up to the local site
to configure) SYSPRV is enabled just before a file is $CREATEed and then
immediately disabled. If SYSUAF authentication is enabled (by default it
is not) then SYSPRV is enabled just before $GETUAI is used to check a
user's password then immediately disabled.
o WORLD - when control functions are used from the command line (e.g.
HTTPD/DO=RESTART) allows the server to retrieve process details (name,
user, etc.) of the issuer of that command for inclusion in server logs.
Not that the author doesn't have at least some confidence in his code ;^) but
has also placed a sanity checker which when the server becomes quiescent
establishes that only the NETMBX and TMPMBX privileges are enabled. The server
will exit with an error message if any extended privileges are enabled at the
time of the check. (During development in 1997 this check discovered an
instance where an EnableSysPrv() call inadvertantly had been coded instead of
a DisableSysPrv() call :^( so it does work in real-life :^)
The capacity for the server to write into the file system is a major concern,
and a lot of care has been taken to make it as secure as possible. Of course
there is always the chance of a problem :^( The main defence against a system
design or programming problem allowing write access to the file system is
having the server account as a separate user and group (and definitely
non-SYSTEM). In this way a part of the file system must explicitly have write
access granted to the server account for it to be able write into the file
system (or for it to have world write access ... but then what is the problem
with server access if the world has access?) This is recommended to be done
using an ACE (see the Technical Overview).
Server Process Instances
------------------------
The term 'instance' is used to describe an autonomous server process.
WASD will support multiple servers running on a single system, alone or in
combination with multiple servers running across a cluster. When multiple
instances are configured on a single system they cooperate to distribute the
request load between themselves and share certain essential resources such as
accounting and authorization information.
This sharing introduces several concurrency issues for multiple, per-node
instances (processes). Data is shared between such processes using global
sections and shared memory. Access to these is mediated by resource locking.
The VMS Distrubuted Lock Manager (DLM) is used extensively in three roles.
1. coordinating access to shared resources
2. storing and distributing data related to resources (e.g. socket use)
3. initiating actions on a per-node or per-cluster basis (e.g. /DO=)
Mutexes are used to control access to per-system shared memory (e.g. global
accounting data, activity statistics, authentication and SSL session caches).
IPv4 and IPv6
-------------
The server supports both IPv4 and IPv6 addressing. Two of the design
objectives were a source that could be compiled on system that did not support
or have the header files for IPv6, and an image that could be executed
regardless of whether IPv6 was suported by the underlying TCP/IP kernel. The
TCPIP.H module header file contains all the requires IPv4 and IPv6 defintions
and structures to remove dependency on system build environment. The server
runtime uses a WASD address structure that contains both IPv4 and IPv6 IP
address structures and the server adopts the appropriate behaviour for the
underlying address type being processed.
Server configuration handles the standard dotted-decimal addresses of IPv4, as
well as 'normal' and 'compressed' forms of standard IPv6 literal addresses, and
a (somewhat) standard variation of these that substitutes hyphens for the
colons in these addresses to allow the colon-delimited port component of a
'URL' to be resolved. The TCPIP.C module describes these.
The TCP/IP Services implementation (at least) has no asynchronous DNS host name
resolution interface for IPv6 as it does $QIO ACPCONTROL for IPv4 (it does have
a POSIX threads compliant C-RTL call but WASD does not use POSIX threads).
Hence AAAA record lookup under IPv6 is blocking (see TCPIP6.C module).
As a general observation; IPv6 obviously received no further development in
TCP/IP Services since the early twenty-first century. Completely neglected.
WASD IPv6, based on the BG-driver services, consequently also stalled. Let's
see what might happen with the VSI stack.
HTTP/2
------
HTTP/2 (RFC7540) is a replacement for how HTTP (1.n) is expressed "on the
wire". It is not a ground-up rewrite of the protocol. HTTP/1.1 methods,
status codes and semantics remain the basis, with a suprisingly small number of
wrinkles (most of which relate to network interaction/behaviour). In effect,
HTTP/2 undertakes a role analagous to TCP, establishing and managing the
connection, as well as multiplexing datagrams over it. Of course it does more
than *just* that. WASD implements core, mandatory HTTP/2 elements, some
non-mandatory, but few of the more esoteric/non-applicable elements such as
priority/dependency, server-push, and the like.
The introduction of HTTP/2 required significant changes to abstractions used by
WASD. Prior the HTTP/2 the major abstraction was the request, and network I/O
largely performed directly against the network stack. With HTTP/2 a layer of
abstraction needed to be introduced to appropriately hand-off direct network
I/O for HTTP/1.n and for HTTP/2 to its multiplexed streams. There is the
NETIO_STRUCT, with NETIO.C and SESOLANETIO.C handling the direct network I/O,
and HTTP2NET.C the multiplexed aspects.
Another, additional layer of abstraction required interfacing each protocol's
request/response header formats with the underlying server processing, without
excessive duplication of code. HTTP/1.1 has a plain-text, carriage-control
separated format, while HTTP/2 has a binary, compressed, lookup-table oriented
format (RFC7541). The WASD "dictionary" module was introduced, with the server
populating request and response elements, which are then formatted appropriate
to the particular protocol requirement. (The "dictionary" function assumes
other, more general purpose roles as well.)
HTTP/1.1 and /1.0
-----------------
HTTP/1.0 behaviour is based on descriptions in RFC1945 (May, 1996). Along with
a swag of de facto HTTP/1.0 extensions (e.g. keep-alives) and HTTP/1.1 style
functionality that crept into HTTP/1.0 behaviour.
HTTP/1.1 behaviour is based on descriptions in RFC2616 (June, 1999), with
reference to the more recent RFC 7230 (June, 2014) family of documents.
HTTP/1.1 semantics and non-network behaviours are (for the most part)
compatible with, and form the basis of, request processing for the HTTP/2
protocol. WASD supports a subset of the extensive capabilities described in
the above RFC but provides an acceptable and functional compliance. The 'core'
subset is largely based on RFC2616 section "Compatibility with Previous
Versions".
HTTP/1.1 'core compliance' offered by WASD:
o persistent connections
Where a response includes a content-length and the client has not
requested the connection closed it is maintained.
Where a content-length is not available or the connection will be
closed for some other reason the (final) response has a
"Connection: close" field provided.
Where a script provides a CGI response header it is checked for
content-length and the appropriate header fields generated.
The only response WASD does not enforce this field is with NPH scripts.
o absolute URI's in a request
WASD interprets an absolute URI as a proxy request. It ignores the
"Host:" field. This is the also the HTTP/1.0 behaviour.
o requests containing a chunked body
The body reading module accepts 'chunked' requests
Includes post-processing any trailing header fields.
o "Host:" request header
WASD returns a 400 (bad request) response if an HTTP/1.1 request
does not contain this header field. The field is supplied as a CGI
variable.
o "If-Modified-Since:" or "If-Unmodified-Since:" requests
WASD has historically provided HTTP/1.0 "If-Modified-Since:" support
(including the de facto 'length=nnn') when sending files. For HTTP/1.1
the 'length=nnn' is unsupported. WASD supports "If-Unmodified-Since:"
for file access (static and SSI, etc). Both request fields are provided
as CGI variables.
o "Expect: 100-continue" request
WASD will return a 417 (expectation failed) if it is not a
'100-continue' expectation. This field is supplied as a CGI variable.
o "100 Continue" response
If the request contained an "Expect: 100-continue" field, is an
HTTP/1.1 POST or PUT request, then WASD responds with a "100 Continue"
response header after evaluating such factors as authorization and
before beginning to read/process the body.
o "Date:" header in each response
The only response WASD does not enforce this field is with NPH scripts.
This sort of script needs to be modified for compliance.
The WASD 'REQUEST_TIME_GMT' CGI variable contains the current date/time.
o support of HTTP/1.0
Continued support for requests using the HTTP/1.0 protocol.
Other HTTP/1.1 functionality implemented:
o request pipelining
WASD supports request pipelining. With body-less requests any
octets in excess of the request header are considered possible
pipelining. Requests with bodies (if pipelined at all) are
implicitly handled as such because the body network reads are
precisely sized and so do not encroach on data belonging to any
following request.
o "Accept-Ranges: bytes" response
This advises clients of the byte-range support.
o "Range: ..." requests
WASD provides byte-range responses for non-variable record format
files from disk and all files stored in cache. The "Range:" field
is supplied as a CGI variable.
o "If-Range:" request header
This changes a "Range:" request from a 206 into a full file 200 if
the file has been modified since the date/time provided in the header.
o "Cache-Control:" request header
The cache-control parameters "no-cache", "no-store" and "max-age=0"
all disable WASD's various caching mechanisms, particular the file
cache (where at the very least the file is revalidated).
o "ETag:" response header
For responses generated from static files (also cached) an entity
tag is generated using the file ID (6 bytes) and last modification
time (8 bytes) as a 28 character hexadecimal string. This provides
a unique (enough) identifier in file-system space and time.
o "ETag: request header
WASD uses this field in conjunction with "If-match:" and/or
"If-none-match" during file response processing. This field is
supplied as a CGI variable.
o "If-Match: request header
Used in conjunction with the "ETag: field during file response
processing. This field is supplied as a CGI variable.
o "If-None-Match: request header
Used in conjunction with the "ETag: field during file response
processing. This field is supplied as a CGI variable.
o TRACE method
Returns a 200 response with the entire request, header and body,
as a content-type "message/http". It honours the "Max-forwards:"
request header field, attempting to proxy any non-zero values.
o OPTIONS method
Returns a standard 200 response header (only). It honours the
"Max-forwards:" request header field, attempting to proxy any
non-zero values.
o extension-methods
These are HTTP methods not defined in the RFC (i.e. not GET, HEAD,
POST, etc.) and not supported natively by the server. WASD allows
requests using such methods to be mapped to an external agent (RTE,
script, etc.) for handling.
if (request-method:EXAMPLE) \
exec /* (CGI_EXE:EXAMPLE.EXE)/cgi-bin/* map=method
The above example would allow any request using the EXAMPLE method
to be mapped to the EXAMPLE.EXE RTE for processing.
Transport Layer Security (TLS)
Secure Sockets Layer (SSL)
--------------------------
The basic WASD package supports only HTTP. TLS/SSL support can be provided by
linking against and using the Compaq/HP/VSI SSL for OpenVMS Alpha/Itanium
product, a separate WASD-specific OpenSSL based package, or an existing OpenSSL
installation.
The build procedure allows the conditional compilation of non-SSL or SSL object
code. The SSL image is built by linking the SSL object modules with the
OpenSSL toolkit. This approach was adopted to allow the export of WASD from
countries that prohibit such with cryptographic software. It does contain the
"hooks" for such functionality though which may prohibit export in themselves.
OpenSSL transitioned though the 0.9.n versions, with API incompatibility into
the 1.0.n versions, and another API incompaibility as 1.1.n was introduced. As
of 2017, the released WASD v11.n packages are built against the 1.0.n version,
with a local build against OpenSSL 1.1.n possible. Conditional compilation
allows builds against either 1.0.n or 1.1.n. It is imagined that as OpenSSL
1.1.n gains VSI traction that WASD packages will move to that API. 'Til then...
WebDAV
------
WebDAV 1,2 required a significant rework of many aspects of data handling in
the server. Before general release my feeling is that WebDAV was not worth the
effort. I guess time will tell. (2017 note; at least one significant
commercial implmentation has leverage WASD WebDAV.) The DAVWEB.C module
contains comprehensive commentary on WASD WebDAV functionality.
WebDAV makes extensive use of XML in its request bodies, response bodies and
meta-data maintained against resources (files and directories). EXPAT by Clark
Cooper was chosen. It has a reputation for being fast, efficient, and document
event-driven structure. The XML managed by WASD has a relatively simple
structure and EXPAT suits it fine.
There are several WebDAV modules providing WebDAV-specific processing. Some
other methods have WebDAV-specific behaviours. PUT is an obvious one. OPTIONS
also reports WebDAV methods when it is enabled.
WebSocket
---------
WebSocket is an HTML 5 capability providing an asynchronous, bidirectional
full-duplex connection over which messages can be sent between agents.
WebSocket applications (scripts) run in CGIplus (persistent) processes setup
and controlled by the DCL module, very similar to other CGIplus scripting.
However, WebSocket scripts can handle multiple, concurrent clients!
Managing persistent, long-running, asynchronous connections between clients and
scripting processes is generally counter to the HTTP request-response,
client-server paradigm, and is implemented using multiple additional
WebSocket-aware hooks throughout WASD request, DCL and response processing
code.
WASD "Raw"Socket is functionality heavily based on the WebSocket module code
and general infrastructure. It too provides the persistent, long-running,
asynchronous connections between clients and scripting proceesses but unlike
WebSocket is absolutely protocol-agnostic. The clients are not by default
web-oriented (i.e. browsers) but depend on the protocol the script implements.
Request Processing Flowchart
----------------------------
One of the more difficult aspects of event-driven (AST) programming is the
constant lack of a clear code path. This section attempts to provide an
overview of the major functional points in processing a request. Needless to
say there are are lots of detours and some dead-ends when processing but what
follows is the essence of a standard request. Processing is from top-to-bottom
except where the crude arrows flow otherwise. HTTP/2 and HTTP/1.1 have
independent request acceptance paths.
HTTP/2 (entry point)
Http2ClientRead() <--+ <--+ !receive frame from client
| | |
| {control frames} | !HTTP/2 internal processing
| | :
+--{not header frames}--+ : !frame processed by type
| .
HpackHeadersFrame() !request headers frame
|
Http2RequestBegin() !initialise request processing
|
Http2RequestProcess() !begin processing the request
|
RequestParseDictionary() !dictionary contains request
|
:
.
HTTP/1.n (entry point)
NetAcceptAst() !received socket connection
|
+- NetAccept() !queue the next socket accept
|
RequestBegin() <--+ !initialize request processing
| | .
+- SesolaNetBegin() --+ : !if SSL then SSL-accept connection
| :
+- "Raw"Socket service ----------~:~-+
| : |
RequestGet() <--+ | | !get request header
| | | |
+- Http2Preface() --> HTTP/2 -~|~-+ | !detect HTTP/2 connection
| | |
+- NetRead()/SesolaNetRead() --+ | !read request header
| |
RequestParseHeader() | !request header to dictionary
| |
: :
. . !both /2 and /1.n then ...
: :
| |
RequestFields() | !tease out header field data
| |
RequestParseAndExecute() | !parse request line
| |
RequestExecute() | !begin to process request content
| |
MapUrl_Map() <--+ !apply mapping rules
|
+- RequestEnd() !if redirect from mapping
| |
| RequestRedirect() --> !redirect (possibly restart request)
| |
| RequestEnd() !301/302 response redirect
|
+- ProxyRequestBegin() !proxy request (HTTP)
| :
| ProxyRequestEnd() !after proxy processing
| |
| RequestEnd() !end of request
|
Authorize() --+ !authorize access to request path
| |
RequestExecutePostAuth1() !either direct or after AST
|
+- AdminBegin() !server administration
| :
| RequestEnd() !once complete
|
+- HtAdminBegin() !HTA database administration
| :
| RequestEnd() !once complete
|
+- other-internal !other internally handled
| :
| RequestEnd() !once complete
|
+- Authorize() ------+ !authorize access to script path
| | |
| RequestEnd() | !not authorized
| |
RequestExecutePostAuth2() !either direct of after AST
| |
| RequestEnd() !not authorized
|
+- ThrottleBegin() --+ !request subject to queuing
| |
RequestExecutePostThrottle() !none, or post queuing after AST
|
+- RequestScript() !script to process (incl. WebSocket)
| |
| +- internal !internally handled look-alikes
| | :
| | RequestEnd() !once complete
| |
| +- DclBegin() !sub/detached process CGI script
| | :
| | RequestEnd() !once complete
| |
| +- DECnetBegin() !network CGI or OSU script
| :
| RequestEnd() !once complete
|
+- FileBegin() !search cache, continue if not found
| |
| CacheSearch() ---------+ !if not found in cache
| | |
| CacheBegin() | !cache entry require revalidation?
| | |
| CacheAcpInfoAst() --+ !if cache entry is stale
| | |
| CacheNext() | !transfer from cache
| : |
| RequestEnd() | !once cache transfer complete
| |
RequestExecutePostCache() <--+ !cannot be supplied from cache
|
DavWebRequest() !if WebDAV-specific method
| : !(some methods, e.g. PUT, can also
| DavWebEnd() ! have WebDAV-specific behaviour)
|
PutBegin() !if a PUT, POST, or DELETE method
| :
| RequestEnd() !once PUT or POSTed
|
+- RequestHomePage() <--+ !search for a home page
| | |
| + ----------------+ !possible multiple names
| |
| RequestFile() !home page found, transfer
| | :
| | RequestEnd() !once transfered
| |
| DirBegin() --> !no home page, directory listing
| :
| RequestEnd() !once listed
|
+- RequestEnd() !auto-script
| |
| RequestRedirect() --> !redirect (possibly restart request)
| |
| RequestEnd() !301/302 response redirect
|
+- RequestEnd() !search keyword
| |
| RequestRedirect() --> !redirect (possibly restart request)
| |
| RequestEnd() !301/302 response redirect
|
RequestFile() !transfer file
:
RequestEnd() !once transfered
|
RequestRedirect() --> !redirect (possibly restart request)
|
RequestEnd() !301/302 response redirect
HTTPd Modules
-------------
The HTTPd server comprises several main modules, implementing the obvious
functionality of the server, and other, smaller, support modules. Modules
contain descriptive prologues. As these are usually up-to-date (usually
more-so than discrete documentation such as this text anyway), it is strongly
recommended that the source code modules be consulted for specific information
on how each operates. This section is provided only to give a quick overview.
All files are located in WASD_ROOT:[SRC.HTTPD]
WASD.H
This C header file contains many of the general data structures and macros
required for the HTTPd.
ENAMEL.H
This header is used to work-around the issue of older versions of VMS and DECC
not having a long NAM structure or definitions used to support ODS-5. For the
same reason an extended FIB structure definition must be provided. For such
environments this header file provides the necessary infrastructure, allowing
extended file specification compliant code to be compiled and linked on
pre-v7.2 systems.
ADMIN.C
Server administration is based in this module, although other modules provide
their own functions for implementing menus, reports and actions.
AUTH.C
This module provides path-based authorization and authentica- tion. Uses the
HTTPD$AUTH file as the source of configuration information. This module uses
the other AUTHxxxxxx.C modules to provide particular required functionalities.
AUTHACME.C
Allows authentication and SYSUAF password changes using the ACME server via the
$ACM system service. Provides both SYSUAF authentication and from other
sources using other and third-party ACME agents.
AUTHAGENT.C
Provide authentication and authorization information from an external,
CGIplus-based script.
AUTHCACHE.C
Provides the caching of authentication/authorization information so that the
sources do not need to be accessed for every request.
AUTHCONFIG.C
Loads authorization configuration information from the HTTPD$CONFIG file into
a data structure that the HTTPd then uses during authorization.
AUTHHTA.C
Accesses information stored in binary authentication databases (files).
AUTHHTL.C
Accesses information stored in plain-text authentication files.
AUTHIDENT.C
Provide authentication and authorization from an RFC1413 source ("ident"
protocol).
AUTHTOKEN.C
Accesses authentication/authorization in one context and reflects that
authorization to another using a short-lived token delivered as a cookie.
AUTHVMS.C
Accesses authentication and authorization information from the system's SYSUAF
and RIGHTSLIST databases.
BASE64.C
Base-64 encoding/decoding (well, golllee!)
BASIC.C
Implmentation of the "BASIC" HTTP authentication scheme are implemented in
this module.
BODY.C
Controls the transfer of the request body (for POST and PUT methods) from the
client to the server and then on to a script, proxied server or internal HTTP
module for processing. A script or a proxied HTTP server receives the "raw"
data provided by the client. Internal modules or a proxied FTP server get data
that has been processed by one of three functions that "filter" the data from
a request body. The first just puts the entire body into a single buffer and
is used internally. The others extract data from
"application/x-www-form-urlencoded" or "multipart/form-data" content-types.
CACHE.C
This module implements a file data and revision time cache, designed to
provided an efficient static document request (file transfer) mechanism.
CGI.C
The CGI module provides the CGI scripting support functions used by the DCL.C
and DECNET.C modules.
CLI.C
Module to process (interpret) the HTTPD command-line.
CONTROL.C
This module implements the HTTPd command-line control functionality. At the
command-line an administrator may enter HTTP/DO=command[/ALL]. This is written
to the HTTPd via shared memory in a global sector, interpreted and the
requested action taken or an error message sent back to the administrator. The
Distributed Lock Manager (DLM) is used to alert a detached server process that
a command is available for actions, and to distribute a command to all server
cluster-wide when using the /ALL qualifier.
CONFIG.C
This module provides basic server and service configuration. Uses the
HTTPD$CONFIG file as the source of configuration information.
DICT.C
A hash dictionary (associative array) of key-value pairs. In particular, used
to store request and response header fields while interfacing between client
and server. The implementation is a standard hash array with collision
linked-list. An associated list allows iteration in order of insertion.
DAVCOPY.C
WebDAV copy a file or directory tree.
DAVDELETE.C
WebDAV delete a file or directory tree.
DAVLOCK.C
Manage a WebDAV lock on a resource (file or tree). Although WASD WebDAV uses
VMS locks and the DLM for managing access within its processing this module
deals exclusively with WebDAV locking. WASD WebDAV locks are stores in
meta-data files managed by DAVMETA.C.
DAVMETA.C
WebDAV meta-data uses XML for its representation. WASD WebDAV meta-data is
stored in a separate file along with the file it is the meta-data of (or
directory file if a directory). This module manages the WASD-specific data
stored in that (WebDAV locks) and independent client-supplied entities.
DAVMOVE.C
WebDAV move a file or directory tree. For efficiency renames the file or
directory file if on the same volume, or if on another volume copies and
deletes (much more expensive).
DAVPROP.C
Generates WebDAV "live" properties for files and directories. These are XML
representations of such data as creation and modification timer, size, etc.
DAVWEB.C
This is the primary WebDAV processing module. All WebDAV-specific processing
begins and ends with this. It also pr4ovides some common WebDAV processing
functionality and the WebDAV report code. The prologue contains significant
WASD Web-DAV comments.
DCL.C
The DCL execution functionality must interface and coordinate with an external
subprocess. Supports CGI and CGIplus scripting and SSI DCL-processed
directives.
DCLMEMBUF.C
Bulk data transfer from script to server using a (global section) memory
buffer. Intended for transfers of multiple megabytes.
DECNET.C
The DECnet module provides scripting based on process management using DECnet.
Both standard WASD CGI scripting and an emulation of OSU (DECthreads)
scripting are supported.
DESCR.C
The Descr.c module generates a file description by searching HTML files for
the first occurance of <TITLE>...</TITLE> or <Hn>...</Hn> tags, using the
description provided there-in. It is primarily used by the directory listing
module, but can also be used by the menu module.
DIGEST.C
This module provides authentication functionality for the DIGEST method. I do
not know of any browser or user group that employs the mechanism.
DIR.C
This module implements the HTTPd directory listing (Index-of) functionality.
ERROR.C
Error generating and reporting functions.
FAO.C
Provides formatted write capability to storage and network buffer space.
Functionality based on the VMS system service $FAO but contains all sorts of
extensions to facilitate WASD objectives. It's versatility makes it an
indispensable code module.
FILE.C
This module implements the static file transfer functionality.
GRAPH.C
This module generates the server activity graph.
GZIP.C
Provides ZLIB-enabled GZIP compression (gzip, deflate) for WASD. Dynamically
maps required functions from a ZLIB shareable image. If the image is found and
all required functions present the ZLIB compression/decompression of network
streams in enabled for suitable requests/responses. If not it is just left
disabled. Based on ZLIB port by Jean-François Piéronne. Requires this package
to be installed and started on the runtime system for dynamic activation.
HPACK.C
HPACK is a compressor that reduces bandwidth requirements for HTTP/2 (RFC7540)
request and response header fields. Defined by RFC7541.
HTADMIN.C
This module allows on-line administration of the HTTPd- specific
authentication (.HTA) databases.
HTTP2.C
Initialises HTTP/2 functionality for the server. Handles some core HTTP/2
connection behaviours such as instantiation (initial connection), stream reset,
go-away, ping, connection settings.
HTTP2NET.C
HTTP/2 network reads and writes. Reads are multiplexed onto the connection by
the client and similarly multiplexed off at the server end. The frames read
are then handed off to the appropriate processing function by frame type.
Writes are serialised onto the connection using FIFO queuing.
HTTP2REQUEST.C
Conjure up a request structure, populate the dictionary with its request
headers, supply it with a request body if required, execute the request,
populate the (HTTP/2) response header with response header fields and then
run-down the request. There are obviously life-cycle parallels to HTTP/1.1
processing.
HTTP2WATCH.C
Provide WATCHing of HTTP/2 processing.
HTTPD.C
This is the main() module of the server. It performs server startup and
shutdown, along with other miscellaneous functionality.
INSTANCE.C
Contains functions used to setup, maintain and coordinate action between,
multiple servers running on a single system, alone or in combination with
multiple servers running across a cluster. An "instance" in this context
refers to the (almost) completely autonomous server process.
LOGGING.C
The logging module provides an access log (server logs, including error
messages are generated by the detached HTTPd process. The access log format
can be that of the Web-standard, "common"-format, "common+server"-format or
"combined"-format, along with user-definable formats, allowing processing by
most log-analysis tools.
MAPCON.C
Used by the MAPURL.C module to process (the somewhat obsolesecent) mapping rule
conditionals.
MAPODS.C
Used by the MAPURL.C module to support mapping of URL-style specifications to
VMS file system specifications and back. Can process ODS-2, ODS-5 (EFS), SRI
(MultiNet NFS), PATHWORKS (v4/5) and Advanced Server (PATHWORKS V6) / Samba
encodings.
MAPURL.C
Main module supporting mapping of URLs to VMS file specifications and VMS
specifications to URLs, setting specified characteristics agaionst paths,
etc. Uses the HTTPD$MAP file as the source of configuration information.
MAPUSER.C
Used by the MAPURL.C module to support mapping /~ style requests (user home
directory) using SYSUAF account data to map the username's home file
specification.
MD5.C
Module providing MD5 digest code from RFC1321. This is used to generated
"digests" (or unique fingerprints of sequences of bytes and strings) for use
in authorization and the generation of file names in proxy caching.
MSG.C
The message database for the server is maintained by this module. Uses the
HTTPD$MSG file as the source of configuration information.
NET.C
This module handles all non-SSL TCP/IP network activites, from creating the
server socket and listening on the port, to reading and writing network I/O.
It manages request initiation and rundown, and controls connection
persistence. It is developed using, and based upon the behaviour of, both the
Digital TCP/IP Services (UCX) BG driver. Any other package that supports this
through emulation will support WASD.
NETIO.C
This module specialises in the low-level network data handling.
ODS.C
This module supports file system access for both ODS-2 and where appropriate
(Alpha VMS v7.2ff) ODS-5.
It does this by abstracting the NAM block so that either the standard or long
formats may be used without other modules needing to be aware of the
underlying structure. The VAX version does not need to know about long NAMs,
or extended file specifications in general (excluded for code compactness and
minor efficiency reasons using the ODS_EXTENDED macro). Alpha versions prior
to 7.2 do not know about NAMLs and so for compilation in these environments a
compatible NAML is provided (ENAMEL.H header file, see ENAMEL.H), allowing
extended file specification compliant code to be compiled and linked on
pre-v7.2 systems.
Runtime decision structures based on the VMS version, device ACP type, etc.,
must be used if pre-v7.2 systems are to avoid using the NAML structure with
RMS (which would of course result in runtime errors). In this way a single set
of base- level Alpha object modules, built in either environment, will support
both pre and post 7.2 environments.
PERSONA.C
For VMS V6.2 and later provides access to the $PERSONA services allowing the
server to create scripting processes executing under user accounts other than
itself (see DCL.C). For VMS versions not supporting the $PERSONA services
(earlier than V6.2) it creates stubs allowing the module to be linked but just
returning a status indicating it is not available.
PROXY.C
All data structures and general macros, etc., for proxy processing are located
in the following header file. Provides the request and network functionality
for proxy HTTP and HTTPS serving.
PROXYCACHE.C
Obsolete as of v12.0.0.
PROXYFTP.C
Provides FTP proxying for both GET (RETR) and POST/PUT (STOR) methods.
Interprets directory listings (LIST) for DOS, Unix and VMS FTP servers (makes a
feeble attempt for those it doesn't recognise).
PROXYMAINT.C
Since PROXYCACHE.C obsoleted above, a shadow of its former glory, performing a
few reporting functions.
PROXYNET.C
This module provides the essential networking functionality for proxy
processing. It also maintains the pool of persistent proxy->origin server
connections.
PROXYREWORK.C
The proxy rework facility will modify a target string to a replacement string
in the request header (e.g. Host:), the response header (e.g. set-cookie:), and
in the response body. Rework will be applied to HTML and CSS responses.
Probably needs a lot more (re)work(ing).
PROXYSOCKS.C
RFC 1928 SOCKS Version 5 proxy for TCP/IP.
Supports only CONNECT TCP/IP and not BIND or UDP-associate.
PROXYTUNNEL.C
WASD supports the CONNECT method which effectively allows tunnelling of RAW
octets through the proxy server. This facility is most commonly used to allow
secure SSL connections to be established with hosts on the 'other side' of the
proxy server. This basic mechanism is also used by WASD to provide an extended
range of tunnelling services.
PROXYVERIFY.C
Implements a relatively simple, pragmatic mechanism that allows a proxy server
to authorize a request locally, then convey that authorized username to a
reverse-proxied server using a standard HTTP "Authorization: basic ..." request
field, while keeping the original password private. It then provides an
HTTP-based mechanism for the proxied-to server to verify that the request is
indeed originating from the proxy server.
PUT.C
The PUT module allows files to be uploaded to, and stored by, the server. It
also allows the deletion of files, and the creation and deletion of
directories. This same module handles PUT, POST and DELETE methods
appropriately. It requires authorization to be enabled on the server.
REDIRECT.C
Handle internal (an important and fundamental aspect of some WASD request
processing) and external redirection.
REGEX.C
The GNU RX regular expression package REGEX.C and REGEX.H files, unmodified
except for a small WASD macro introduced at the beginning of the former.
REQUEST.C
This module reads the request header from the client, parses this, and then
calls the appropriate task function to execute the request (i.e. send a file,
SSI an HTML file, generate a directory listing, execute a script, etc.)
RESPONSE.C
This module provides support for generating appropriate HTTP responses and
response-related processing. It also deals specifically with National
Character Set (NCS) configuration and conversions.
SESOLA.C
The SESOLA.. modules provide support for SEcure SOcket LAyer processing. This
module provides the optional Transport Layer Security (TLS) (aka. Secure
Sockets Layer (SSL)) encrypted communication link functionality for WASD and
is named "SeSoLa" to avoid any confusion and/or conflict with OpenSSL package
library routines. These modules are conditionally compiled into two sets of
object modules, one for non-SSL servers, the other for those built against the
optional OpenSSL toolkit. Also see the more general comments on TLS/SSL above.
SESOLACACHE.C
Implements the instance-shared SSL session cache. This allows multiple
per-node instances to share session contexts.
SESOLACGI.C
Generates the SSL-specific CGI variables. Supports Purveyor and Apache
environment sets of variables.
SESOLACERT.C
Parse X509 certificates, issuer and subject, plus extensions.
SESOLACLIENT.C
Handles processing specifically for the negotiation for and verification of
peer (client) certificates. It also provide support for X.509 certificate
authentication and authorization.
SESOLAMKCERT.C
For a TLS/SSL service that does not use the default or have a unique X509
server certificate, this module will dynamically create one during startup.
The generation of a private key for the certificate can add significant time to
overall server startup.
SESOLANET.C
This module provides the TLS/SSL network processing. It has functions
supporting the encryption of plain data streams and the subsequent transmission
to the remote SSL service, as well as the decryption back to original data of
recei8ved SSL streams. It also supports the session acceptance (for SSL
requests) and session establishment (connections, for SSL gatewaying).
SESOLANETIO.C
This module specialises in the low-level TLS/SSL network data handling.
SHA1.C
Provides SHA-1 digest hashing used during the WebSocket request handshake.
SSI.C
The Server Side Includes (HTML pre-processor) module provides this
functionality as an integrated part of the server.
STMLF.C
The stmLF.c module converts VARIABLE format records to STREAM- LF. It is only
called from the File.c module and only then when STREAM-LF conversion is
enabled within the server configuration.
STRDSC.C
Particularly with the introduction of WebDAV and the associated XML processing
required some more versatile method of string handling was required. This
module uses a data structure allowing 32 bit sized strings (i.e. > 65365),
linked-lists of buffers of these, and automatically uses dynamic memory
associated with requests or server as appropriate. FAO.C and NET.C can use
these structures. Data contained in them does not necessarily have to be
ASCII/null-terminated of course.
STRNG.C
This code module contains functions related to string processing. In
particular string wildcard and regular expression matching (see above).
SUPPORT.C
The support module provides a number of miscellaneous support functions for
the HTTPd (well go-o-o-lee!).
SYSPLUS.C
Show essential system data without using a scripting process as does
WatchShowSystem().
TCPIP.C
With the introduction of IPv6 support some generic IP and TCP/IP functionality
was move or placed into this module. It's header contains some structure
definitions and macros used extensively in the support of QIO-based network I/O
in WASD. The code module contains several support functions that tmake IPv4
and IPv6 supportable without recompilation.
TCPIP6.C
IPv6 address resolution functions only available with VMS V7.0 and later.
IPv6 name/address resolution is a little problematic for WASD because there is
(currently) no native asynchronous interface to it in the same way as there is
with $QIO ACPCONTROL for IPv4. Fortunately IPv6 is somewhat of a niche
environment!
TCPIPALT.C
Currently experimental. Alternate TCP/IP functions, lookup by name and by
address. Both inline (synchronous) and by agent (DCL script, asynchronous).
The agent can take on the role of a network agent and drop connections. This
means that if using a lookup agent there is no need also to use a network
agent. A drop connection is cached.
THROTTLE.C
Provides request throttling functions. This controls the number of concurrent
processing requests against a specified path. Requests in excess of specified
limits are FIFO queued.
UPD.C
Implements the on-line web directory administration and file editing and
upload facility. It requires authorization to be enabled on the server. File
and directory modification are still performed by the Put.c module. The Upd.c
is an overly large body of code generating all of the dialogues and editing
pages. It also provides additional functionality for the server
administration, adding admin-specific portions of dialogues as required.
VERSION.C
The VERSION module merely provides a small, convenient place to generate some
build information.
VM.C
The virtual memory management module provides dynamic memory allocation and
deallocation functions. These functions use the VMS system library virtual
memory routines. Also see general comments in section "Memory Management".
WATCH.C
The WATCH facility provides an online, real-time, in-browser-window view of
request processing in the running server. It support functionality to allow a
site administrator to intergate the processing server for resolving
configuration and other run-time issues (the WATCH_CAT, or category
functionality). It also contains extensive in-detail request processing
functionality intended for server development and BETA debugging purposes (the
WATCH_MOD, or module funtionality).
WEBSOCK.C
Essentially this module handles the setup and tear-down of mailbox "pipes" to
persistent (CGIplus) processes managed by the DCL.C module. Each of these
processes can handle multiple such WebSocket clients. Once established the
mailbox "pipes" just transfer the raw network data to and from the client and
scripting process. The WebSocket protocol must be implemented by the script
(commonly using a library). Also provides WASD "Raw"Socket scripting using the
WebSocket infrastructure and module but with no imposed protocol.
+-----+
+ END +
+-----+