In this day and age, I'm sure everyone is convinced that they need up-to-date, accurate traffic measurement for their Web sites. Good stats tell you a lot of useful things that you can use to improve your site. If you sell ads, you'll find that clients and ad agencies like to see some data, and the nicer it looks, the better. People's traffic reporting needs vary, however. Webmasters for big companies opt for industrial-strength tools, while mom-and-pop sites may get by with one of the cheap shareware items, or perhaps a home-brewed CGI script. For those of us in the vast middle territory between these extremes, Hit List, from Marketwave, Inc., is just what we need: a powerful, customizable log analysis and traffic reporting tool with a manageable price tag.
July 21, 1997
Hit List analyses your log files and uses the data to create a variety of reports. A large library of report elements is available. A report element can be a figure such as total requests, monthly requests, daily visits, etc, or it can be a date or text. Any number of report elements can be included in a report, in any order. Some elements can be displayed as graphs. You can use one of the predefined reports or create your own. The output can be a text file, an HTML page, or both. The HTML reports can have any color scheme and fonts you want. By using a template, you can even insert Hit List's report elements into your own pages, and design your reports any way you like.
Hit List Pro has all the features that a medium-to-large Web site needs, and we have decided to use it as our main stats program here at the Web Developer's Journal. Don't worry, we still call them as we see them here at the WDJ, and I won't be glossing over the bad points. Just because we like the program, and use it, doesn't mean it's free of problems. It just has fewer problems than other packages in its price class.
So, first the caveats:
This is a powerful number-crunching application, so you had better be ready to dedicate some time and some hardware, at least at first. Hit List does not work directly on the log files. It loads your log files into a database (a standard MS Access database), then performs the calculations on the database. This is a much better and faster arrangement than analyzing the raw log files, but guess what? It still takes a long time. The length and complexity of each report varies, of course, but on my 166, a medium-size report takes twenty minutes or so to run. Loading the raw log files into the database is another coffee break. So how do you get anything done? Use the Event Scheduler.
The Event Scheduler allows you to run reports automatically, as well as performing housekeeping functions like log cycling, database compression, etc. If you're running Hit List on an NT server, you just schedule your reports, and Bob's your uncle. Here at the WDJ, our server runs Unix, so we run Hit List on a Windows 95 machine at the Florida office. At 5 am every morning, Hit List Pro can retrieve the log files from the server (using FTP), update the database, run a report (or 2 or more different reports), upload the HTML output to the server, and Bob is, hopefully, still your uncle. The only problem is, you have to start Hit List every night before you go home. You also have to be very careful that Hit List's schedule is synchronized with the server's log cycling schedule, and that nothing crashes during the whole scene.
By now it should be apparent that Hit List takes awhile to set up. It's not because it's hard to understand. In fact, it's all nicely laid out, with plenty of documentation, including context-sensitive help. It's pretty easy to figure out, but in order to get it to work on your site, you'll have to tweak a lot of little things, like where to get the log files, what kind of output you want, etc. If you change something and want to see the results, you'll have to run a report, which means lengthy processing time (try creating a really short test report for troubleshooting purposes). With Hit List, reading the manual and the readme files first will save you a lot of time and cussing!
Of course, once you have your reports laid out and your schedule in place, there's nothing more to do. Every morning (or whenever) your own customized report will be emailed to you, uploaded to an Inernet/intranet site, or both.
Hit List Pro can do reverse DNS lookups, so you can see the names of your visitors instead of numbers. However, there is apparently a bug in Windows 95 that makes it crash about half the time when it does this. There is a downloadable kernel update that supposedly fixes the problem, but the reverse DNS lookup is still a crashy proposition. The latest info on this and other stuff is available at Marketwave's site.
Once you get Hit List installed, you must configure it to work with your setup. First you enter your server name, email server and so forth, and tell it where to find your log files. You can set it up to run on your server or on a remote machine. If you run it on a remote machine, you can use local copies of your log files, or you can set up Hit List to download the log files from your server, using FTP. The program tells you to enter an address like:
Hit List is pretty intelligent about loading log files. You don't have to tell it the exact filename. If you place the wildcard *.* in the path, it will examine all log files in the directory. Only those that are new or have changed will be loaded into the database. Hit List can read both standard types of log files, and also works with ODBC databases such as MS SQL Server.
You can save space and time by excluding certain information from the database. For example, you can exclude requests for graphic files, scripts and so forth if you don't plan to use the data. It's a good idea to get all your options set the way you want them before you update your database and start running reports.
Loading the log files into the database takes some time, but it can be automated with the Event Scheduler. You can also set individual reports to update the database before they run. They will load only the log files that are new or have changed. You can even use the Event Scheduler to cycle your logs.
Hit List Pro 3.0 Main Screen
Report Design Window: Toolbox on the right contains over 250 report elements for the draggin' and droppin'.
Once you have Hit List configured the way you want it, and have updated the database, you can run one of the many predesigned reports, or create your own. The list of statistics that Hit List can calculate is very long, and they can be combined in several ways. A report can cover any time period you choose. Stats over time can be shown as a graph. 10 different types of colorful graphs, including bar graphs, 3D graphs and pie charts, are available.
Hitlist provides 22 predesigned reports, any of which can be copied and then customized as you like. A couple of examples show the range of information available.
The Complete Analysis does not actually include all of Hit List's report elements (such a report would be pages long and take a lifetime to run), but it includes:
Total Requests
Total Visitors
Total Visits
Average Visits per Day
Average Requests per Visit
Average Time Per
Total Requests Per Day of Week
Total Number of Requests per Domain (Virtual Server)
Most Common Visitors (Site Names)
Most Common Source Sites
Most Popular Pages
Most Popular Directories
Requests by Object
Total Number of HTML Page Requests (Page Views)
Average Number of HTML Pages Per Visit
Most Popular Entry Pages
Most Common Exit Pages
Number of Single Page Visits
Most Common Single Page Visits
(Graph) Average Visits Per Day of Week
(Graph) Average Visits Per Hour
Average Monthly Requests and Visits
Average Weekly Requests and Visits
Average Daily Requests and Visits
(Graph) Average Requests Per Hour
(Graph) Average Requests Per Day of Week
The Technical Analysis is a shorter report that includes:
Total Number of Requests
Total Bytes
Average Response Delay (Seconds)
Average Time Per Visit (Seconds)
Requests by Object
Average Requests Per Directory
Average Bytes by Hour
Average Requests by Hour
(Graph) Number of Requests in Each of the Last 7 Days
Most Popular URLs
HTTP Request Types per Week
HTTP Reponses by Week
Browser vs HTTP Codes
(Graph) Requests by Week
(Graph) Bytes Per Week
(Graph) Average Delay by Week
Here at the WDJ, we have designed an "editor's report" that shows things like typical paths through the site, most popular pages, most popular directories, least popular pages, and most common single-page views. This last one shows pages that failed to entice visitors to go to other parts of the site, perhaps due to weak content or poorly implemented navigation bars.
Other reports can measure impressions and clicks for your ads, or tell you how people reached your site. If your logs are complete enough, you can even see what keywords people typed into the major search engines to reach your site.
Although the predesigned reports are great starting points, most WDJ readers are probably do-it-yourself types who will want to customize their own reports, and perhaps even let different people in the organization create reports for different purposes. For example, everyone may want to see a brief report with all the major stats, the propheads can have their own more technically-oriented report, while yet another report might be specially designed for the eyes of prospective advertisers. With Hit List, it is easy not only to specify what information will be contained in a report, but how it is laid out and formatted.
To customize a report, select it and click on the Design button. You will see the report in Design View, with the Toolbox off to the side. All the possible report elements are contained in the Toolbox, including generated values, section headings, date fields, and definitions of terms. Simply drag and drop into your report. For each element, you can define various properties, such as the amount of data to display, whether to display it as a table, graph, or both, etc. You can also set properties for the report as a whole. The Format tab lets you choose foreground and background colors, and font face and sizes. If you want to do more elaborate formatting, or make the report fit your site design, you can do so by using a template, as discussed below. The output tab lets you select whether the report will be output as an HTML page, a text file (RTF or ASCII), a series of CSV text files, or all three. Text files may be emailed to one or more recipients. If you select a template to use for the HTML report, Hit List can insert its output into any HTML file you choose. All or any part of the report can be incorporated into your HTML pages in any order. This very powerful feature means that you can design your own report format from scratch, inserting Hit List-generated data however you like. Each report can also use different log files or a different database than the global settings.
There are a lot of features here, folks. We recommend Hit List Pro for any medium-to-large Web operation. Like any package of this kind, it takes awhile to set up, but once it's running, it's basically a set-it and forget-it deal. To recap a bit:
Read the documentation thoroughly, THEN
Install Hit List and configure it for your system.
Select and customize the reports you want to use.
Use the Event Scheduler to make them run while you sleep. Get your log cycling procedure figured out, so that all your old logs get archived and deleted.
Every morning with your coffee, you'll have the latest figures. If you're like me, every day you'll see some tiny thing you can tweak to make the reports better. Go ahead!
