Log file Definitions and examples

• Introduction

  Below is an explanation of the typical contents of a webserver log. Also included are examples of the most often seen webserver log formats. If you have one or more of these webservers, check the examples against one of your own logs (open one in a text editor like Wordpad) to make sure you're getting all the information you can from your webserver, and if you're missing a few "fields" of information, contact your webmaster/ISP to see how you can get this information.
  

• Log Files Generally

  In most Internet situations, you have a webserver that hosts your site(s). This webserver typically produces ASCII text format logfiles to record the activity on your site(s). Then, you use Hit List to build a database from these logs to analyze the traffic by running reports against this database (usually your Hitlist.mwd).
  

  Each format of webserver logs has similar information, they are just arranged differently according to a few common standards. The "big four" standards are NCSA, W3SVC, Microsoft IIS3.0 format and O'Reilly Website format. Most other types you will see are simply variations of the above.
  

  Internet "Proxy" servers can produce text-based logfiles also, and Hit List can analyze them. Proxy servers serve a different role than "regular" webservers, discussion of which is beyond the scope of this document. Examples of the "big two" proxy servers (Netscape and Microsoft MS Proxy) are seen below.

• Logfile "fields" content explanation:

  For this discussion, an NCSA format log (similar to that you'd see an Apache or Netscape webserver generate) is used, but the "fields"are generally similar amongst other log formats, just in different order.

  Here is an example of such a logfile entry:

  205.218.110.166 - - [08/Dec/1996:15:02:10 -0800] "GET /info/index.html HTTP/1.0" 200 14912 "http://www.yourcompany.com/index.html " "Mozilla/3.0Gold (Win95; I)" "35bebd61b31211cfbdcd00c04fd611cf"

  The content of this entry explained, from left to right:

  "205.218.110.166" - - This is the IP address of the machine making a request of your web server - its domain name can be determined in HitList by enabling Reverse DNS lookups, assuming your server hasn't put this information in already - many so, some don't. (if the domain name was in there, you'd see its URL instead of the raw IP).

  "-" - this first dash is typically the server's IP address, which most NCSA format servers don't insert by default.

  "-" - this second dash is typically authenticated usernames, which again many NCSA format servers don't insert by default.

  "[08/Dec/1996:15:02:10 -0800]" - This is the date and time of the access, including the offset from Greenwich Mean Time - the latter is the "-800", meaning the web server being accessed is 8 hours ahead of GMT.

  "GET /info/index.html HTTP/1.0" - This is the actual request the visitor's browser made when at your page or server.

  GET is the "method" command to retrieve the HTML document. HEAD can be used here to retrieve the header portion, while POST can put up a POST-style application form.

  "/info/index.html" refers to the path of the requested document relative to its root directory location on your server, "index.html" being the ultimate HTML document requested.

  "HTTP/1.0" refers to the protocol and its version, here being version 1.0 of the http protocol.

  "200" - this is the server response code - a "successful" request (meaning the visitor's browser loaded the entire HTML/GIF/JPEG, etc.) generates a response code of 200. Others include:

  206 - Partial request successful (not complete)
  302 - URL has been redirected to another document
  400 - Bad request was made by the client
  401 - Authorization is required for this document
  403 - Access to this document is forbidden
  404 - Document not found
  500 - Server internal error
  501 - Application method (either GET or POST) is not implemented
  503 - Server is out of resources
  

  "14912" - This is the number of bytes transferred to the client during the visit. Since every request has some response, even erroneous requests will have a non-zero value for this field.

  "http://www.yourcompany.com/index.html" - This is the referrer field, or the site the visitor was on immediately prior to making this entry's request - in this case, the person was looking at the index.html (probably the home page) page before going to the /info/index.html page in this entry.

  "Mozilla/3.0Gold (Win95; I)" - this is the user-agent field, meaning the actual browser and OS used by the visitor - in this case, Mozilla is Netscape, the next value is the version (here, 3.0Gold), and the final value is the OS it was using (Windows 95).

  Finally, the "35bebd61b31211cfbdcd00c04fd611cf" is the cookie information, which may or may not be there, depending on whether the webserver used has cookies enabled and whether one was passed from webserver to the visitor's computer.

• Required Fields in the logs:
  

  These fields are REQUIRED:
  

  RequestType, URL, RequestDate, VisitorIP.
  

  for the following Hit List Plugins:
  

  IIS plugin (IIS2.0 and IIS3.0 format logs);
  NCSA plugin (Apache, Netscape, Lotus Domino);
  W3C plugin (IIS4.0 logs, WebSTAR logs);
  Winlog plugin (O'Reilly Website logs);
  NetscapeProxy plugin (Proxy Plugin, Commerce users and above);
  NovellProxy plugin (Proxy Plugin, Commerce users and above);
  

  These fields are REQUIRED:
  

  URL, RequestDate, VisitorIP
  

  for the following plugins:
  

  NetShow plugin (Microsoft Windows Media Technologies/Netshow server);
  MSProxy plugin (Microsoft Proxy Servers);
  

• Specific Logfile Format Examples:
  
  Microsoft IIS 3.0 and 2.0:

  157.55.69.103, -, 12/6/96, 7:08:22, W3SVC, WEBSERVER, 206.129.192.10, 10, 286, 14167, 200, 0, GET, /info/default.asp, Mozilla/2.0 (compatible; MSIE 3.0; Windows 95), http://www.yourcompany.com/default.htm, 35bebd61b31211cfbdcd00c04fd611cf, -,
  

  Microsoft IIS4.0 (W3SVC format):

  #Software: Microsoft Internet Information Server 4.0
  #Version: 1.0
  #Date: 1999-01-24 00:00:06
  #Fields: date time c-ip cs-username s-sitename s-computername s-ip cs-method cs-uri-stem cs-uri-query sc-status sc-win32-status sc-bytes cs-bytes time-taken s-port cs-version cs(User-Agent) cs(Cookie) cs(Referer)
  1999-01-24 00:00:05 208.208.7.34 - W3SVC1 WEBSERVER 206.129.192.10 GET /hitlist/newreports/mwhlcol.gif - 200 0 1119 366 32507 80 HTTP/1.0 Mozilla/4.08+[en]+(Win95;+U) - http://www.marketwave.com/hitlist/newreports/complete_navbar.htm
  1999-01-24 00:00:05 208.208.7.34 - W3SVC1 WEBSERVER 206.129.192.10 GET /hitlist/newreports/MWHLGraph24183.GIF - 200 0 9729 373 32967 80 HTTP/1.0 Mozilla/4.08+[en]+(Win95;+U) - http://www.marketwave.com/hitlist/newreports/complete_report.htm
  

  Netscape (NCSA format with unique format header):

  format=%Ses->client.ip% - %Req->vars.auth-user% [%SYSDATE%] "%Req->reqpb.clf-request%" %Req->srvhdrs.clf-status% %Req->srvhdrs.content-length% "%Req->headers.referer%" "%Req->headers.user-agent%" "%Req->headers.cookie%"
  207.225.167.48 - - [08/Dec/1997:02:00:39 -0700] "GET /BannersMsg.class HTTP/1.0" 200 1326 "-" "Mozilla/4.03 [en] (Win95; U)"
  207.225.167.134 - - [08/Dec/1997:02:02:20 -0700] "GET / HTTP/1.0" 200 9989 "-" "Mozilla/2.02E (Win95; U)" "35bebd61b31211cfbdcd00c04fd611cf"
  207.225.167.134 - - [08/Dec/1997:02:02:22 -0700] "GET /gfx/navbar/bg.gif HTTP/1.0" 304 0 "http://www.azfamily.com/" "Mozilla/2.02E (Win95; U)" "35bebd61b31211cfbdcd00c04fd611cf"
  207.225.167.134 - - [08/Dec/1997:02:02:22 -0700] "GET /gfx/index/hdr/monday.gif HTTP/1.0" 200 3469 "http://www.azfamily.com/" "Mozilla/2.02E (Win95; U)" "35bebd61b31211cfbdcd00c04fd611cf"
  

  Apache (NCSA format with NO format header):

  168.191.4.27 - - [28/Dec/1997:01:04:56 -0600] "GET / HTTP/1.0" 200 12077 "http://www.yahoo.com/Business_and_Economy/Companies/Packaging/" "Mozilla/4.02 [en]C-DIAL (Win95; U)"
  168.191.4.27 - - [28/Dec/1997:01:06:01 -0600] "GET /whatsnew14.html HTTP/1.0" 200 4791 "http://www.castlerockcontainer.com/" "Mozilla/4.02 [en]C-DIAL (Win95; U)"
  168.191.4.27 - - [28/Dec/1997:01:06:23 -0600] "GET /castlerock2.html HTTP/1.0" 200 9836 "http://www.castlerockcontainer.com/whatsnew14.html" "Mozilla/4.02 [en]C-DIAL (Win95; U)"
  

  Lotus Domino format:

  Domino servers log in NCSA format, similar to Apache and Netscape and thus if a Domino 4.6 or later server, you can produce ONE combined "access" log that will have ALL the information (access, user-agent and referrer) in one logfile. However, older Domino servers (4.5 and older) typically log in "three log" format, producing a separate access, agent and referrer log. All this means is that if you were to split the Apache log sample above into those three components, you'd have three separate logfiles recording three parts of "one" log entry.
  

  Thus, the combined Domino access log should look identical to the Apache example immediately above, while older Domino servers would have three logs for Hit List to concatenate together and then load into its database. For proper configuration of Domino & Hit List, check out our Domino Configuration FAQ document.
  

  O'Reilly WebSite format:

  04/03/98 05:23:43 scan.palacenet.net www2.palacenet.net HEAD / Ipswitch_WhatsUp/3.0 200 0 15
  04/03/98 05:23:44 pm2-11.palacenet.net www.microt.com GET /PRODUCTS/Images/USR/usr_modem_ani.gif http://www.palacenet.net/ Mozilla/3.01C-KIT (Win95; U) 304 0 0
  04/03/98 05:24:24 pm2-18.palacenet.net www.microt.com GET /PRODUCTS/Images/USR/usr_modem_ani.gif http://www.palacenet.net/ Mozilla/3.01C-KIT (Win16; U) 304 0 0
  04/03/98 05:24:41 pm2-18.palacenet.net www.microt.com GET /PRODUCTS/Images/USR/usr_modem_ani.gif http://www.palacenet.net/ Mozilla/3.01C-KIT (Win16; U) 206 5356 828
  04/03/98 05:24:45 scan.palacenet.net www2.palacenet.net HEAD / Ipswitch_WhatsUp/3.0 200 0 0
  04/03/98 05:25:03 pm1-17.palacenet.net www.microt.com GET /PRODUCTS/Images/USR/usr_modem_ani.gif http://www.palacenet.net/ Mozilla/3.01C-KIT (Win16; U) 304 0 172
  

  WebSTAR format (MacOS webserver, variation on W3SVC format)

  !!WebSTAR STARTUP 05/17/99:12:08
  #Version: 1.0
  #Software: WebSTAR/3.0.2
  #Start-Date: 05/17/99:12:08
  #Fields: BYTES C-DNS C-IP CS(COOKIE) CS(HOST) CS(REFERER) CS(USER-AGENT) CS-HOST CS-IP CS-METHOD CS-STATUS CS-URI CS-URI-QUERY CS-URI-STEM DATE SC-STATUS TIME TIME_TAKEN
  0 - 128.171.118.183:80 "" "www.pdadash.com" "" "Mozilla/4.5 [en]
  (WinNT; I)" - 128.171.118.183:80 CONDITIONAL_GET 304 /pdadash/advertise/images/goamerica/an-minstrel2.gif -
  /pdadash/advertise/images/goamerica/an-minstrel2.gif 1999-05-22 304 07:00:15 0
  0 - 128.171.118.183:80 "" "www.pdadash.com" "" "Mozilla/4.5 [en] (WinNT; I)" - 128.171.118.183:80 CONDITIONAL_GET 304 /pdadash/advertise/images/goamerica/an-minstrel2.gif - /pdadash/advertise/images/goamerica/an-minstrel2.gif
  1999-05-22 304 07:00:17 0
  0 - 169.229.52.69:80 "" "www.pdadash.com" "" "Mozilla/4.5 [en] (Win95; I)" -
  169.229.52.69:80 CONDITIONAL_GET 304 /pdadash/advertise/images/paragraph/po_and_cg_1.gif - /pdadash/advertise/images/paragraph/po_and_cg_1.gif 1999-05-22 304 07:00:18 3 0 -
  

  Proxy server logfile format examples:
  

  MS Proxy 1.0 format:

  -, -, MSProxy/1.0, N, 7/10/97, 0:16:08, W3Proxy, TERMINATOR, -, cnn.com, -, 80, 672, 9184, 172, http, tcp, GET, http://www.cnn.com/images/9706/nav/nav_bar_main.gif, -, VFInet, 200
  -, -, MSProxy/1.0, N, 7/10/97, 0:18:07, W3Proxy, TERMINATOR, -, webserver.marketwave.com, -, 80, -, 705, 146, http, tcp, GET, http://www.marketwave.com/products.htm, -, VCache, 304
  -, -, MSProxy/1.0, N, 7/10/97, 0:20:07, W3Proxy, TERMINATOR, -, www.microsoft.com, -, 80, 235, 14939, 202, http, tcp, GET, http://www.microsoft.com/library/images/gifs/rotateads/IISSide.gif, -, VCache, 304
  

  MS Proxy 2.0 "regular" format:
  
  206.129.192.39, anonymous, -, N, 8/5/98, 18:12:58, 1, -, -, www.hawaii.rr.com, -, 80, 906, 10626, 11016, http, tcp, -, http://www.hawaii.rr.com/cgi-bin/RoadRunner/News/news.cgi, -, Inet, 200, 0
  206.129.192.39, anonymous, -, N, 8/5/98, 18:12:58, 1, -, -, www.hawaii.rr.com, -, 80, 516, 3265, 3597, http, -, -, http://www.hawaii.rr.com/Around_Town/images/rrh_left.gif, -, Inet, 200, 0
  206.129.192.39, anonymous, -, N, 8/5/98, 18:12:58, 1, -, -, www.hawaii.rr.com, -, 80, 1000, 17791, 18127, http, tcp, -, http://www.hawaii.rr.com/Around_Town/images/hi-bkgrd_mat.gif, -, Inet, 200, 0
  

  MS Proxy 2.0 "verbose" format:
  
  206.129.192.32, anonymous, Mozilla/4.05 [en] (WinNT; I ;Nav), N, 8/5/98, 18:09:16, W3Proxy, LIVE1, -, www.digitaria.com, 192.215.146.37, 80, 141, 1217, 1564, http, tcp, GET, http://www.digitaria.com/pages/workf.html, text/html, Inet, 200, 0
  206.129.192.32, anonymous, Mozilla/4.05 [en] (WinNT; I ;Nav), N, 8/5/98, 18:09:17, W3Proxy, LIVE1, -, www.digitaria.com, 192.215.146.37, 80, 375, 2997, 3344, http, tcp, GET, http://www.digitaria.com/pages/workm.html, text/html, Inet, 200, 0
  206.129.192.32, anonymous, Mozilla/4.05 [en] (WinNT; I ;Nav), N, 8/5/98, 18:09:17, W3Proxy, LIVE1, -, www.digitaria.com, 192.215.146.37, 80, 468, 5471, 5818, http, tcp, GET, http://www.digitaria.com/pages/workt.html, text/html, Inet, 200, 0
  

  Netscape Proxy format:

  format=%Ses->client.ip% 146.127.62.22 %Req->vars.pauth-user% [%SYSDATE%] "%Req->reqpb.proxy-request%" %Req->srvhdrs.clf-status% %Req->vars.p2c-cl% %Req->vars.remote-status% %Req->vars.r2p-cl% %Req->headers.content-length% %Req->vars.p2r-cl% %Req->vars.c2p-hl% %Req->vars.p2c-hl% %Req->vars.p2r-hl% %Req->vars.r2p-hl% %Req->vars.xfer-time% %Req->vars.actual-route% %Req->vars.cli-status% %Req->vars.svr-status% %Req->vars.cch-status%
  146.127.123.16 146.127.62.22 - [10/Dec/1997:00:30:09 -0500] "GET http://www.nba.com/bulls/ HTTP/1.0" 200 8816 200 8816 - - 321 164 359 164 1 SOCKS(146.127.11.3:1080) FIN FIN NON-CACHEABLE
  146.127.253.84 146.127.62.22 - [10/Dec/1997:00:30:12 -0500] "GET http://www.pathfinder.com/NY1/bug.html HTTP/1.0" 200 377 200 377 - - 392 203 418 203 1 SOCKS(146.127.11.3:1080) FIN FIN REFRESHED
  146.127.253.84 146.127.62.22 - [10/Dec/1997:00:30:12 -0500] "GET http://www.pathfinder.com/NY1/images/steel.gif HTTP/1.0" 304 - 304 - - - 443 142 468 142 0 SOCKS(146.127.11.3:1080) FIN FIN UP-TO-DATE