How do I get extended logging (referrer and user agent) for Microsoft IIS, Netscape, NCSA, O'Reilly and Apache webservers?

Microsoft Internet Information Server

• IIS 2.0 and 3.0

Microsoft IIS 2.0 and 3.0 do not log referrer and user-agent information in the logs by default. Thus, if you are using either IIS2.0 or IIS3.0, to get this information in your logs, you can run Hit List setup on the server. It will install a very slightly modified Microsoft logging .dll file to "force" IIS to log this information. You will likely want to stop/restart IIS after this procedure (or just reboot) to make the change take effect.

Or, if installing Hit List on the webserver directly isn't an option, you can use the following information to "manually" install the dll on your webserver to get this information put into your logs by IIS.

Manually Installing MW's extended logging dll for IIS 2.0/3.0 (unnecessary for IIS4.0 webservers)

Do a windows Find for mwnewlog.dll on the machine where you've already installed Hit List.

Copy this file and put it on a floppy for transfer (or use a network connection) to the webserver NT machine.

Copy the MWNewLog.DLL to your \winnt\system32 directory on the webserver to be modified.

NOTE - please only have NT-certified techs complete the next step - we can't assume liability for improperly edited registries!

Run RegEdit

Go to the key:

\hkey_local_machine\System\CurrentControlSet\Service\W3SVC\Parameters

and highlight it. Now on the right, you should see the Filter DLLs "String" which you can highlight and right-click to "modify".
Now add the full path to the MWNewLog.DLL to this line. Separate it from existing entries with a comma
(e.g. --- add ,c:\winnt\system32\MWNewLog.DLL)

Exit Regedit and reboot.

• IIS4.0

Here are the instructions to set up extended logging in IIS4.0. The IIS3.0 filter, mwnewlog.dll, does not apply to IIS4.0 and is therefore unnecessary.

1) Open the Management Console on the IIS 4.0 webserver, usually under Start/Programs/WindowsNT4 Option Pack/MS Internet Information Server/Internet Service Manager

2) Click the plus next to Internet Information Server on the right, click the plus next to the resulting computer icon (your webserver), and then right-click the "Default Web Site" icon that appears, and go to Properties.

3) On the "Web Site" tab, down below where it says "Enable Logging" (likely checked already or you're simply not producing logs). Click the Properties button after making sure this is set to W3C Extended Logging.

4) For maximum data with Hit List, go to the "Extended Properties" tab and check the following boxes:

5) Click "OK" until you are out of the Management Console. Go to Control Panel/Services. Stop and Start the IIS related Services (specifically, World Wide Web publishing service) to make the change take effect (or reboot, but usually most people can't reboot their webserver very often as it's online).

• O'Reilly Website

If you run a O'Reilly WebSite, simply select the extended logging option.

All of the O'Reilly logging choices provide virtual site information because even the O'Reilly Common and Combined formats actually use a modified log format that Hit List understands. While Hit List can understand all three formats, we very strongly suggest that you use the Combined rather than the Common or Windows format because the Common format does not provide User-Agent and Referrer information, making it impossible for Hit List to show which browsers your visitors used and perform complex path analysis. The O'Reilly Windows format is proprietary, so it cannot be read by most other applications.

• Netscape

Customizing Netscape servers depends on the particular server involved. If youÍre using the older Commerce or Communication servers, you will have to get an NSAPI to add the additional information to your logs. If youÍre using Enterprise or Fast Track, you can add User_Agent and Referrer information simply by changing a setting. You can also insert server identification information (typically not included in the logs by default).

NOTE - As of this writing, these Netscape setup instructions were accurate. However, due to the ever-changing world of the Internet and software in general, the ultimate authority is your Netscape server documentation and Netscape technical support if these instructions lack some specifics or you need more help.

Adding User-Agent/Referrer to Enterprise and Fast Track

1. Find all copies of your OBJ.CONF files for your server. There will probably be two of these, one in a directory like c:\netscape\\httpd-\config and one in c:\netscape\\admserv

2. Copy them to backup files.

3. Open the original files in Notepad. Be certain that Word Wrap is off.

4 Find the line, towards the top, that contains:

Init fn=flex-init access=

and replace whatever is currently there (up to but not including any additional Init fn strings) with

Init fn=flex-init access="PutCompletePathAndFileNameForYourLogsHere" format.access="%Ses->client.ip% - %Req->vars.auth-user% [%SYSDATE%] \"%Req->reqpb.clf-request%\" %Req->srvhdrs.clf-status% %Req->srvhdrs.content-length% \"%Req->headers.referer%\" \"%Req->headers.user-agent%\""

Be sure to not add any line breaks and to enter this exactly as shown except replacing the PutCompleteƒ text with an actual file path and name to your log file. Netscape servers also demand that you use the UNIX/Internet standard forward-slash (/) character to separate directories, not the DOS standard back-slash (\) character. Therefore, a log file name of c:\Netscape\MyServer\Logs\access would be entered into the file as c:/Netscape/MyServer/Logs/access even on Windows 95 and NT systems.

After youÍve restarted your server, your logs should produce entries that look as follows -- except that there should be no line breaks:

205.218.110.166 - - [17/Mar/1997:15:02:10 -0800] "GET /info/index.html HTTP/1.0" 200 14912 "http://www.yourcompany.com/index.html " "Mozilla/3.0Gold (Win95; I)"

Subtopic: Working with virtual servers in Netscape

If every URL in your logs are prefixed with information about virtual servers (e.g. /client1/index.html, /client2/index.htm), you can use the Virtual Server Manager to teach Hit List how to differentiate between your virtual servers without making any changes to your logs. This also makes it easier to run reports for each of your virtual domains automatically.

Alternately, you can hard-code server identification information into your logs by replacing the - after the client.ip% phrase with the IP of the virtual server in question. As Netscape demands a new instance of the server (and, therefore, a new instance of obj.conf) for each virtual server, so long as you vary the identification information in each obj.conf file, Hit List will be able to differentiate between servers. For example, if your first virtual server is at IP 123.456.789.1 and your second is at 123.456.789.2, your obj.conf files would look like:

For the first server:

Init fn=flex-init access="PutCompletePathAndFileNameForServer1LogHere" format.access="%Ses->client.ip% 123.456.789.1 %Req->vars.auth-user% [%SYSDATE%] \"%Req->reqpb.clf-request%\" %Req->srvhdrs.clf-status% %Req->srvhdrs.content-length% \"%Req->headers.referer%\" \"%Req->headers.user-agent%\""

For the second server:

Init fn=flex-init access="PutCompletePathAndFileNameForServer2LogHere" format.access="%Ses->client.ip% 123.456.789.2 %Req->vars.auth-user% [%SYSDATE%] \"%Req->reqpb.clf-request%\" %Req->srvhdrs.clf-status% %Req->srvhdrs.content-length% \"%Req->headers.referer%\" \"%Req->headers.user-agent%\""

Finally, if you donÍt want to add this information to your logs or use the Virtual Server Manager, you should not load all your virtual sites into the same database. Instead, you should use report-level overrides for both the log file and the database specification to ensure that each virtual site is treated separately. Since each virtual server will be stored in its own database, Hit ListÍs automatic Run this report for every virtual server feature cannot work. Rather, to replicate a report for each virtual server, you should create the report you want for one server and use the Copy command to clone it for the other servers. Modify the log file and database specifications for each virtual server.

Note that if you are using Host Headers for Virtual Servers (instead of IP), you can have Netscape log this information. Simply put in the field "%Req->headers.host%" instead of the IP as discussed above. If you have further questions on this, ultimately the best source would be Netscape support.

• Apache

Working with Apache is similar to Netscape except that the log format customization method is different.

NOTE: - As of this writing, these Apache setup instructions were accurate. However, due to the ever-changing world of the Internet and software in general, the ultimate authority is your Apache server documentation and/or the Apache web site if these instructions lack some specifics or you need more help.

Adding User-Agent/Referrer to Apache

1. Find your copy of the mod_log_config.c file.
2. Modify the LogFormat directive to match the following string:
%h %l %u %t \î%r\î %s %b \î%{Referer }i\î \î%{User-agent}i\î
3. Compile and link this module into Apache.

Subtopic: Working with virtual servers in Apache

See "Working with virtual servers in Netscape" above as to logging virtual server information in Apache - all the same issues apply except that you would replace the %l field with the virtual server IP. It may also be possible to replace the %l field with %v to make Apache load the server IP in itself (rather than having to "hard-code" it) but again, check the Apache website for more specific instructions.

• NCSA webservers

Again, you can view using NCSA as the same as working with Netscape except that the log format customization method is different.

Adding User-Agent/Referrer to NCSA

1. Find your copy of httpd.conf
2. Set the LogOptions directive to Combined with the following line:
LogOptions Combined
3. Save the file and restart your server.

Subtopic: Working with virtual servers in NCSA

It does not seem possible to add virtual server identification to NCSA servers. Therefore you should not load all your virtual sites into the same database unless you use the Virtual Server Manager first. Instead, you should use report-level overrides for both the log file and the database specification to ensure that each virtual site is treated separately. Since each virtual server will be stored in its own database, Hit ListÍs automatic Run this report for every virtual server feature cannot work. Rather, to replicate a report for each virtual server, you should create the report you want for one server and use the Copy command to clone it for the other servers. Modify the log file and database specifications for each virtual server.