Home > Products & Services > Pilot HitList architecture

Architecture

Click to see a larger picture

Pilot HitList is an integrated, flexible and accurate web tracking and analysis application which offers powerful data collection, warehousing and custom reporting capabilities. Pilot HitList is a proven solution for virtually any web environment addressing the needs of a new web initiative as well as advanced functionality for web sites that receive millions of hits per day.

Pilot HitList captures information across multiple web and application servers at the network or at the web server level. Network level data provides a rich depiction of each user's visit to a site by recording events such as resets and visit duration. The network data is collected, summarised, and stored in the Pilot HitList database in near real time. At report time, Pilot HitList's powerful custom reporting capabilities combine both web traffic data and other enterprise data, such as transactions, into a single, comprehensive analysis. With Pilot HitList's ability to distribute the report instantly in a variety of formats, the analyst can understand and respond to changing conditions in "web time" rather than calendar time.

How it Works

Pilot HitList is a flexible and accurate web tracking and analysis application offering powerful data collection and reporting capabilities. Proven technology provides reliable monitoring of millions of hits per day.

Capture information across multiple web and application servers at the network level. This level of data provides a rich depiction of each user's visit to a site by recording events such as resets and time spent per page. This data is then transformed into valuable information to allow an enterprise to assess its web initiatives and to see how changes to content, services, graphics or site layout affect customers and ultimately impact site effectiveness. An enterprise can choose from a suite of flexible reporting options which are tailored to meet the needs of professionals throughout the organisation.

Pilot HitList provides an open and extensible architecture that can be extensively customised and flexibly integrated into an enterprise's environment. At its foundation, Pilot HitList is built on key architectural components as depicted below.

Web Data Collection

Pilot HitList tracks and collects data from the entire user's visit, across multiple servers and applications, and throughout the entire visit—including before, during and after a transaction. In order to accomplish this comprehensive data capture, Pilot HitList provides the widest range of flexible collection options. Together we can decide which method (or combination of methods) is most suitable for your site and your anlysis needs:

Network Collector
Unlike most vendors, Pilot HitList offers network data collection enabling vast quantities of data to be captured from multi server environments in near real time. The Network Collector monitors packets of information as they are transmitted over the network, capturing user events and network activity. Through the Network Collector, Pilot HitList records events and timing information not logged by a web server, such as form data or when a user presses the browser "Stop" button. With this information, companies can better evaluate the user's experience and understand if download times are too long, if files too large for practical use, or if visitor demand is overwhelming web resources. The Network Collector collects web traffic data from the network and streams information in near real-time to the Pilot HitList Database Writer.

Rich Data Captured
The Pilot HitList Network Collector captures a richer set of data than that logged by the web server:

• Resets (when the user presses the browser "Stop" button).
• HTTP POST parameters
• Time spent per page (web servers will log with an accuracy of plus or minus a second while the Network Collector is correct to the millisecond.)

Web site tagging collector
HitList's page tagging executes a Javascript code through the browser passing logging information to an empty gif file via query parameters. These tags can be customised to reflect the needs of your organisation to gather the information you need in the most effective way for example URLs can be converted to a value that provides business value to simplify reporting.
 
The very small size of the tag ensures that there is a minimal impact on site performance.
 
HitList supports 1st party tagging so minimising the impact of spyware programmes.
 
The flexibility of HitList's tagging and cookie generation methodology allows you maximum flexibility in campaign management without the need for site changes including the application of campaign life thresholds so that you can assign value on the basis of individual campaigns. 
 
HitList supports the use of tagging in conjuction with other data collection methods to ensure that your business needs are met.

Server Collectors and web server log files
While the Network Collector provides the richest data, sometimes sessions are encrypted or take place on a network segment that won't accommodate a Network Collector. For these cases, Pilot HitList can utilise any combination of the following approaches to ensure every part of every customer session is tracked:

Server Collector
A Server Collector is a plug-in to the web server's API (such as NSAPI) which collects web traffic data. The collected data is streamed in near real-time to the Pilot HitList Network Collector.

Web server log files
Pilot HitList can utilise the web server log files generated by any web server on any platform. The administrator specifies whether the entire log file is transferred to the Database Writer (via the FTP protocol) or just the records created since the last transfer.

Database Writer

The Database Writer accepts data streams in real-time from any combination of Network Collectors and Web server Data Collectors. This data is combined with information processed by a Database Writer plug-in which understands that particular web server log file format. (ISSEL consulting can create plug-ins or pre-processors to process new web server log file formats.)

The combined data is analysed according to the settings created by the Administrator, including mirroring and MIME type definitions for served content. In addition, the Database Writer can transform cryptic web addresses generated by dynamic content or application servers into a more analysable format.

Session Stitching
The Database Writer processes raw data from Network Collectors, web server Collectors, and web server log files. The Database Writer's ability to aggregate data from multiple sources provides customers with an integrated view of a visitor's sessions, across a mix of non-secure and secure servers, and mirrored and non-mirrored sites. In addition, this ensures that any web implementation, including pure content, mixed content / transaction, and 100% pure encrypted, can be fully analysed using data from a variety of sources.

Summary and Detail Information
To provide the perfect balance between minimising report response times and enabling detailed analysis, the Database Writer's QuickList X-TremeTM optimisation feature tabulates and stores summary information in near real-time for storage in the database along with the detailed information. The summary information is used to minimise response times for simple queries while the detail information is available for complex queries.

MIME types
The Database Writer specifies which file types or specific files should be categorised as a specific MIME type. For example, a company could specify that Adobe PDF files be counted as downloads or as page views.

Transformer
Using pattern-matching criteria, the Database Writer's Transformer feature transforms cryptic web addresses generated by dynamic content engines or application servers into a format that is intuitive and easy to understand. Transformation allows companies to focus on the results of the analysis and web site effectiveness, rather than decoding URLs and managing complex data.

Database

Pilot HitList's high-speed database stores all captured information in a variety of database platforms. The database holds both summary and detailed data for a customer determined time period, enabling both quick reporting and in-depth drill-down analysis.

Admin Utilities
Pilot HitList's administration utilities allow companies to focus on the information, not the complexities of data management and storage. The powerful and flexible utilities manage security, storage, manual loading and diagnostics.

Report Scheduling
Pilot HitList reports can be scheduled in off hours to improve response times during peak business hours.

Extensible and Flexible
Pilot HitList is fully Windows NT standards compliant and supports extensions coded in Microsoft ASP pages, Visual Basic, or Visual C++. Every data element, such as advertising impressions, is accessible to these extensions. In addition, via an integrated VBScript programming language and editor, enterprise applications can interact with Pilot HitList using a powerful yet easy-to-program script.

Analysis & Reporting

Pilot HitList's analysis and reporting provides the ability to better understand web site traffic and visitor behavior. Pilot HitList offers multiple interfaces to allow users to easily access the most appropriate level of data and analysis for their needs.

E-mail: With email reporting, users receive a reports delivered, via email, on a user-defined schedule. Users can select from a range of report options including text, Excel-ready or graphical formats. This feature enables decision makers to receive important information as soon as it's ready or just when needed.

HTML Reports: View pre-defined and cached reports in your web browser. Alternatively via an intuitive interface, modify the date range of existing reports or create custom reports as needed.

Pilot HitList Administration Interfaces: Pilot HitList includes Windows clients which allow both local and remote Administration of Pilot HitList. Pilot HitList Administration functions are provided via an easy-to-use user interface. The Administrator can also modify existing or design new reports.

Report Design client: Pilot HitList's Report Design client is a Windows client that presents an easy-to-use user interface. The end user can either modify existing or design new reports. The client provides over 375 elements to use when constructing a report as well as numerous templates for the report structure. In addition, over 80 standard reports are provided, grouped conveniently into "catalog" folders.