
Monitoring Interface for Apache 1.3 & MRTG
Daily Average Graph
www.snert.com
- apache2mrtg.pl
- mod_watch
- mod_watch.pl
-
- Welcome
- Software
-
Description
This module will watch and collect the bytes, requests, and documents in & out per virtual
host, file owner, remote-ip address, directory or location, and the web server as a whole. This
module was designed for use with MRTG,
which will make nice graphical representations of the data, but is general
enough that it can be applied to other purposes, as the raw data is accessed by a URL.
This module supports mod_vhost_alias and mod_gzip.
Configuration
The directives below can be added to the general Apache configuration file,
httpd.conf .
- SetHandler watch-info
- Context: global, <Directory>, <Location>, <VirtualHost>
This handler is used to display a line of plain text containing the
following space separated fields:
ifName ifUptime ifInOctets ifOutOctets ifRequests ifDocuments
The ifName is the target name of the request made in a URL.
The remaining fields are all numerical. ifUptime is the
web server's uptime in seconds. The following is required
in the global context of the httpd.conf file in order for mod_watch.pl
to function (without change):
This will permit the following URL forms to get the virtual-host, file-owner, server,
and spool-file, data respectively:
The token between the tilde (~) and the following slash (/) can be the name of
any key that appears in the watch-table list and is the
only way to obtain an entry when watching by-port.
- SetHandler watch-copyright
- Context: global, <Directory>, <Location>, <VirtualHost>
This handler permits me to verify the version and copyright of an installation
(see License part 2.c.). This is only required to meet
the License requirements for commercial redistribution:
- SetHandler watch-table
- Context: global, <Directory>, <Location>, <VirtualHost>
This handler is intended for debugging and analysis of the shared memory hash table
performance and weenie files:
- WatchDocuments mime-type ...
- Context: global
A list of MIME types considered to be documents, for the purpose of counting.
For example "text/html" would be considered a document, which loads page elements
such as images, style sheets, audio files, etc. which are not documents, but
elements of one. An asterisk in the mime-type string will act as a wildcard
for zero or more of anything. This directive will override the default list:
application/*word |
|
application/*excel |
|
application/pdf |
application/postscript |
application/rtf |
text/enriched |
text/html |
text/plain |
text/richtext |
text/xml |
- WatchHashTableSize size
- Context: global
A positive number specifying the size of the shared memory hash table used for the
WatchStorage hash policy. The default is 991 (a prime
number, but not necessary to be so) .
- WatchNotLogged handler ...
- Context: global
The combination of MRTG and mod_watch.pl can result in a significant number of
uninteresting entries concerning watch-info to
appear in the access log. This directive provides a means by which to
suppress those entries. By default everything is logged.
Specify one or more of the following handler names
for which logging should be suppressed:
watch-copyright, watch-info, watch-table, all
NOTE this directive could affect the correct operation of other modules
that follow after mod_watch in the execution sequence, and should be
used with some care.
A better way to achive the same behaviour would be to alter the
httpd.conf file with something similar to the following:
- WatchNetwork exclude ip/mask ...
- WatchNetwork include ip/mask ...
- Context: global
Add one or more network IP/mask pairs to either the exclusion or inclusion list.
Requests coming from a network or host that is a member of the exclusion list
and is not a member of the inclusion list are not counted. Otherwise the requests
will be counted as outlined in WatchStorage.
The ip/mask can be specified in one of several ways
and is the same as for Apache's Allow or Deny directives (domain or host
name strings are not supported):
Form |
|
Example |
|
Equivalent |
a | 10 | 10.0.0.0/8 |
a.b | 169.254 | 169.254.0.0/16 |
a.b.c | 192.168.1 | 192.168.1.0/24 |
a.b.c.d | 127.0.0.1 | 127.0.0.1/32 |
a.b.c.d/n | 192.168.1.0/24 | 192.168.1 |
a.b.c.d/A.B.C.D | 192.168.1.0/255.255.255.0 | 192.168.1.0/24 |
Please note that the IP address parser and lookup make an effort to support both IP4
and IP6 (and beyond), however it has NOT yet been tested on a machine supporting IP6.
- WatchStorage files
- WatchStorage hash
- Context: global
The WatchStorage directive specifies the perfered
storage policy for maintaining the counters.
The files and hash
storage policies both use the WatchSpoolDirectory
to store "weenie" text files. The filename of each text file is either the
virtual host name, file owner (login ID), the remote-IP address,
or the string "SERVER". The contents of each text file is a
string of ASCII text containing four space separated numerical fields given below.
The hash storage policy is the default storage
policy.
Name |
|
Type |
|
Description |
ifInOctets | Counter32 | bytes received from POST requests that specified a Content-Length |
ifOutOctets | Counter32 | bytes sent in response to requests |
ifRequests | Counter32 | number of requests (raw hits) received |
ifDocuments | Counter32 | number of documents sent |
When updating a set of counters, they are first read from the specified
storage medium. The number of octets in and out for a request are added
to the current counters, and the number of requests is incremented by one.
The number of documents is incremented by one, if the response MIME type
matches one of the MIME types specified for the
WatchDocuments directive. The counters are then returned from whence
they came. The counters are allowed to roll over.
The difference between files and
hash storage policies is that the
files policy will read/write to disk several
sets of counters (see WatchWhat) for each request.
The hash storage policy attempts to improve
performance by reducing disk access through the use of a shared memory
hash table, occasionally swapping one or more sets of counters out to disk.
- WatchSpoolFile filename
- Context: <Directory>, <Location>
Used to watch by directory or location. Each filename
must be unique amongst all the virtual hosts, file owners, IP addresses, and other
WatchSpoolFile directives, otherwise the behaviour is unspecified.
The filename is used to store runtime data about
a location within the WatchSpoolDirectory and is used
to extract this data using a URL of the form:
http://www.snert.com/~filename/watch-info
The filename will also appear as an entry in the
watch-table list.
- WatchSpoolDirectory directory
- Context: global
An absolute or server-root relative path to a directory where any support and
runtime files will be kept. This directory must be readable and writeable by the Apache child
processes. The default is /var/spool/mod_watch .
- WatchWhat item ...
- Context: global
mod_watch, by default, will collect information for the virtual host, file owner, and
the server as a whole for each request made. For performance reasons,
it might not be necessary to watch all these things. Specify one or more of the
following to select which items to watch:
virtual-host |
|
virtual-host-by-port |
file-owner |
file-owner-by-port |
remote-ip |
remote-ip-by-port |
server |
server-by-port |
headers-in |
headers-out |
all |
nothing |
The by-port variants collect and separate the information
according to the port number, instead of grouping all requests regardless of port.
For example, virtual-host-by-port would consider requests for
http://www.snert.com/ different from https://www.snert.com/ and so would
separate the information into "www.snert.com:80" and "www.snert.com:443".
This also holds true if Apache is listening on other ports. Specifying both variants
is allowed and would provide both a break down by port and the sum on all ports.
Specifying headers-in or headers-out
will add the length of the headers of the request and response to the
ifInOctets and ifOutOctets counters repectively. These are not on by default,
because Apache does not count them in the acceess log and may skew comparisions
made between web log analysis tools and the MRTG data.
Specifying all, remote-ip, or
remote-ip-by-port should
be used with care as monitoring remote-IP addresses can generate large volumes of data
and impact web server performance. Note that the values of ifInOctets and ifOutOctets
reflect data received from and sent to the remote-IP address.
Download
Installation
Unpack and install MRTG prior to installing this module.
Unpack this module into its distribution directory.
Before building the mod_watch binary, please review and alter the
configuration settings found at the top of the Makefile. Note that
Apache's "apxs" utility is required to have already been installed prior
to building either static or dynamic versions of mod_watch.
The fastest and simplest way to install mod_watch, is as an Apache
DSO library. This presumes that Apache has been compiled and built
with mod_so (see Apache documentation):
cd (path to)/mod_watch-3.13
make install-dynamic
Some system administrators prefer to statically link their Apache
modules into Apache itself. This requires more effort on the part
of the system administrator. To start the process do the following
and follow any additional instructions supplied by the Makefile:
cd (path to)/mod_watch-3.13
make install-static
In both dynamic and static installations, the Makefile will install the
following scripts into /usr/local/sbin (see SCRIPTDIR in the
Makefile):
apache2mrtg.pl,
mod_watch.pl
Also the directory /var/spool/mod_watch (see SPOOLDIR in the
Makefile) will be created to hold the weenie files managed by mod_watch.
The following must be added to the global server configuration
within httpd.conf in order for mod_watch.pl
to work (without modification):
<IfModule mod_watch.c>
# Allows the URL used to query virtual host data:
#
# http://www.snert.com/watch-info
#
<Location /watch-info>
SetHandler watch-info
</Location>
# Allows the URL used to query file owner and web
# server data:
#
# http://www.snert.com/~achowe/watch-info
# http://www.snert.com/~SERVER/watch-info
#
<Location /~*/watch-info>
SetHandler watch-info
</Location>
# Intended for debugging and analysis of shared memory
# hash table and weenie files:
#
# http://www.snert.com/watch-table
#
<Location /watch-table>
SetHandler watch-table
</Location>
# The following allows me to verify the version
# and copyright for the purposes of support:
#
<Location /watch-copyright>
SetHandler watch-copyright
Order allow,deny
Allow from snert.biz snert.com snert.info snert.net snert.org
</Location>
</IfModule>
Use apache2mrtg.pl to parse the
httpd.conf file to create an MRTG configuration script for all the
defined virtual hosts. The output can be appended to or included in an
existing mrtg.cfg file. There currently is no support to
generate MRTG configurations for virtual hosts derived from
mod_vhosts_alias or local user names.
perl apache2mrtg.pl (path to)/httpd.conf >web.cfg
Use this generated configuration with MRTG for a cron job that executes regularly
to generate the graphs (see MRTG installation documentation). This configuration
will periodically make calls to mod_watch.pl to obtain
the MRTG target name, uptime, bytes in & out, requests in, and documents out.
Stop/start the web server:
apachectl stop; apachectl start
Please note that "apachectl restart" does not always appear to load correctly
a DSO module that uses shared memory and semaphores. For this reason it is recommended
that you stop/start the web server instead.
Once the Apache web server starts receiving requests, all the bytes
received in requests (excluding headers), where a Content-Length header
is defined and all the bytes sent (excluding headers) in responses are
added to the current 32-bit counters and stored.
If the MRTG output is properly
configured, you should be able to use a URL to see assorted graphs concerning data
transfer rates of virtual hosts, file owners, and the web server.
Notes
Sample MRTG output & configurations for:
- Octets In & Out
- The follow MRTG configuration is produced by
apache2mrtg.pl:
Title[www.snert.com]: www.snert.com Data Traffic
Target[www.snert.com]: `/usr/local/sbin/mod_watch.pl http://www.snert.com/watch-info`
MaxBytes[www.snert.com]: 1250000
PageTop[www.snert.com]: <h2>www.snert.com Data Traffic</h2>
- Requests & Documents
- The following MRTG configuration was done by hand to produce Request &
Document traffic:
Options[www.snert.com,req]: growright, perhour
Title[www.snert.com,req]: www.snert.com Request & Document Traffic
Target[www.snert.com,req]: `/usr/local/sbin/mod_watch.pl -f ifRequests,ifDocuments http://www.snert.com/watch-info`
MaxBytes[www.snert.com,req]: 1250000
PageTop[www.snert.com,req]: <h2>www.snert.com Request & Document Traffic</h2>
YLegend[www.snert.com,req]: Hits per Hour
ShortLegend[www.snert.com,req]: H/h
Legend1[www.snert.com,req]: Requests (raw hits) received per Hour
Legend2[www.snert.com,req]: Documents sent per Hour
Legend3[www.snert.com,req]: Maximal 5 Minute Incoming Request Traffic
Legend4[www.snert.com,req]: Maximal 5 Minute Outgoing Document Traffic
LegendI[www.snert.com,req]: Hits:
LegendO[www.snert.com,req]: Docs:
mod_watch has been built and/or tested on the following platforms
(please email me your platform if you've
made it work on something different):
- Cobalt Qube 1, RedHat Linux 5.1, Apache 1.3.22
-
Use the default (System V) configuration.
- FreeBSD 4.2, 4.4, Apache 1.3.22
-
Use the default (System V) configuration. Not yet tested with POSIX API.
From <benoitdm@citeglobe.com>: "If Apache is running in a jail on
FreeBSD 4.4, you must setup jail.sysvipc_allowed=1 to allow use of shared
memory in the restricted environment. This OID is a kernel configuration
option, through you can set it up in /etc/rc.conf or on the fly with sysctl."
- HP-UX, Apache 1.3.22
-
Use the default (System V) configuration.
- Max OS X 10.1, Apache 1.3.24
-
Specify the following macros in the Makefile:
DEF=-DSERIALIZATION_API=FCNTL_API -DSHARED_MEMORY_API=MMAP_ANON
RANLIB=ranlib
- SunOS 5.7 (or better), Apache 1.3.22
-
mod_watch/3.0:
To build as a DSO module, Apache must be built with --enable-rule=SHARED_CORE.
Modify the top of the Makefile to enable the following C defines and link
with the correct libraries:
DEF=-DSERIALIZATION_API=POSIX_API -DSHARED_MEMORY_API=POSIX_API
LIB=-lrt
- Solaris 5.6, Apache 1.3.22
-
mod_watch/3.0:
To build as a DSO module, Apache must be built with --enable-rule=SHARED_CORE.
Modify the top of the Makefile to enable the following C defines and link
with the correct libraries:
DEF=-DSERIALIZATION_API=POSIX_API -DSHARED_MEMORY_API=POSIX_API
LIB=-lposix4
Help in French & English for MRTG and mod_watch.
Thank you to the folks at
Security Space for including mod_watch in their
monthly Apache Module Report.
License
Non-commercial redistribution and use of the original source
distribution, without modification, is permitted provided you retain
this notice, the author's copyright notice, and the disclaimer.
Commercial use, in the role of "end-user" for internal business
operations, of the original source distribution, without modification,
is permitted provided you retain this notice, the author's copyright
notice above, and the disclaimer below.
Commercial redistribution (including but not limited to bundling,
collections, resale) of the original source distribution and/or derived
binaries, with or without modification, is permitted with a) the prior
written permission from the author, b) financial remuneration to
the author, c) the above copyright notice is retained and reproduced by
the binary, and d) the disclaimer below appears in the documentation
and/or other materials provided with the distribution.
Disclaimer
THIS SOFTWARE IS PROVIDE "AS IS" AND ANY EXPRESSED OR IMPLIED WARRANTIES,
INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY
AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO WAY SHALL THE
AUTHOR OR LICENSEE BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY,
OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED
OF THE POSSIBILITY OF SUCH DAMAGE.
Support
Support is only provided for the author's original source distribution.
Free support is no longer provided, through enhancements requests and
problem reports are welcome.
The is now a mailing list for announcements, discussion, and community
support of my Apache modules:
To unsubscribe,
send an email with the subject "unsubscribe" to <modules-request@snert.com>
To subscribe,
send an email with the subject "subscribe" to <modules-request@snert.com>
To submit a message to the mailing list,
just send an email to <modules@snert.com>
- apache2mrtg.pl
- mod_watch
- mod_watch.pl
-
- Welcome
- Software
-
Copyright 2001, 2002 by
Anthony C Howe
All rights reserved.
light bulbs since 6 December 2001
|