Logfile options

One logfile Usually, httpd server are logging requests in one log file.
On NCSA or Apache server, log name is access_log
On CERN server, log name is httpd-log
In the config.pl file, you have to edit : $zip = 0

With traffic increase, most provider now support compressed logfile to save disk space. Some use daily compressed logile, others use monthly compressed logfiles. W3Perl is able to cope with both of them.


Several logfiles Introduced in v2.30, current version can cope with various filename string.
Select in the configuration file the filename log format you are using.

Log filename string can be whatever you want :

  • filename is the constant string in your logfile (ex : access_log),
  • day is a digits number (from 01 to 31),
  • month is a 2 digits number (from 01 to 12),
  • lettermonth is a 3 letters string with first upper (from Jan to Dec),
  • smallyear is 2 digits number (ex : 98)
  • year is 4 digits number (ex : 1998)
  • and suffix is the compression extension used by on your system (ex : gz or zip).

Example of supported string filename :

  • access_log.1998Mar.gz
  • 1998.03.10.raw.zip
  • log.03-12-1998.gz
  • in.9904


Daily gzip files A number of ISP are providing logfile in a daily compressed format.
Use $zip = 1 and $zipcut = 2 in the configuration file to select the use of daily compressed logfiles.


Day to day logfiles The package require that the logfile have more than one day's data.
But if your site is pretty busy and you decide to clear the log files each day, it's still possible to use w3perl.
You'll need to move your logfile at midnight to another filename, w3perl will update stats by reading the moved logfile. Follow the next instructions.
  1. At midnight, move your current logfile (access_log) to logfile.old (access_log_old)
    In your W3Perl configuration file, you should use this moved file (prefixlog should be access_log_old).
  2. Run your stats and then delete your moved logfile (access_log_old).
  3. First day, run cron-pages.pl.
    Next day use cron-inc.pl....
    Cron-inc.pl will scan only the previous day so you need only to keep the previous day log and then destroy it.
  4. Don't forget to run also cron-hour every hour, cron-day once a day, cron-week once a week. You can't use cron-month.pl, sorry.
Example :

    Day 1
    log file log_current is running from 00 h to 23 h 59

    Day 2
    move your log_current to log_previous at 00 h 00
    A new file log_current is running from 00 h to 23 h 59
    run cron-pages.pl (scanning the log_previous file) and then rm log_previous

    Day 3
    move log_current to log_previous at 00 h 00
    A new file log_current is running from 00 h to 23 h 59
    run cron-inc.pl (scanning the log_previous file) and then rm log_previous

    Day 4
    same as Day 3
    ....

You will have incremental stats with only a one day logfile. Of course, choose carefully the options in your config file as logfile are destroyed so stats can't be compute back.


Monthly gzip files Log files are growing very fast (especially for NCSA log file which can't disable local requests logging). In the config.pl file, you have to edit : $zip = 1 and $zipcut = 1 to select the use of monthly compressed logfiles.


Do it yourself W3Perl have been using monthly compressed logfile for many years. If you want to cut your logfile each month and use the compressed log file mode, follow the next lines :

To alter your old log files in the new format, use :

  • grep "/<month>/<year>" <logfile> > <logfile>.<monthdigits>-<year>
    where month is only 3 letters long (first letter is upper),
    year 4 digits long
    and monthdigits is the month number in two digits format.
    (eg : grep "/Dec/1994" access_log > access_log.12-1994)

  • gzip <logfile>.<monthdigits>-<year>
    (eg : gzip access_log.12-1995)
    At the end, you should have in your log directory, files like for example :

      access_log.10-1994.gz
      access_log.11-1994.gz
      access_log.12-1994.gz
      access_log.01-1995.gz

The gziped files should be in the same directory as the current logging file.
It will save you a lots of space disks as log file are growing very fast !

NT users can use zip instead of gzip. It will give you for example access_log.01-1995.zip intead of access_log.01-1995.gz.


Use squeezelog to cut and compress your logfile each month :

I'm using a small script squeezelog to cut and compress the log file each month. Compression rate is around 90% of the initial file saving a lots of disk space.
These replace the rotate-log utility provided with httpd server. If you are running IIS, using a monthly logfile would be my best choice.
W3Perl will also run faster as it will have only to scan the current month logfile.

You need to edit squeezelog file and configure it with your own system. The prog should be added in your crontab and executed once every first day of each month.

Example :
01 00 1 * * /usr/local/bin/perl /norfolk/www-data/w3perl/squeezelog

(if the logs files are root, you should ask your administration system to install squeezelog).

back