%META:TOPICINFO{author="RussellAdams" date="1107379131" format="1.0" version="1.1"}%

Setting up XMTrend

• Setting up XMTrend
  • XMTrend
  • Finding metrics to record
    • Example output from xmpeek
    • Example metric specifications for XMTrend
    • Generating a list of metrics
  • Creating an XMTrend configuration file
    • Sample /etc/perf/xmtrend.cf
  • Running XMTrend
    • Typical paramters
    • Running at boot
  • Interpreting the recordings
  • Long Term Recording
  • Feedback

XMTrend

XMTrend is a background data collection daemon for performance monitoring in AIX.

Data is collected from the xmservd daemon, and saved to disk. Later, recordings can be viewed and manipulated by the azizo/jazizo reporting tools.

The difficulty in using XMTrend is in its configuration. IBM's documentation is lacking, and its nonobvious how it should be setup.

Finding metrics to record

XMTrend has a configuration file which must contain line by line each metric you wish to record.

I prefer to generate this list with xmpeek -l which displays every metric available on the current system.

The output isn't fit for XMTrend by default, the descriptions must be removed, as well as the hostname.

Example output from xmpeek

# xmpeek -l localhost
...
/localhost/CPU/cpu0/wait                    Time waiting for IO (percent)
/localhost/CPU/cpu0/idle                    Time CPU is idle (percent)
/localhost/CPU/cpu0/uticks                  CPU ticks executing in user mode
/localhost/CPU/cpu0/kticks                  CPU ticks executing in kernel mode
/localhost/CPU/cpu0/wticks                  CPU ticks waiting for IO
/localhost/CPU/cpu0/iticks                  CPU ticks while CPU is idle
...

Example metric specifications for XMTrend

...
CPU/gluser
CPU/glkern
CPU/glwait
CPU/glidle
CPU/cpu0/user
CPU/cpu0/kern
CPU/cpu0/wait
CPU/cpu0/idle
...

Generating a list of metrics

A quick way to generate a list of statistics formatted properly for XMTrend from the xmpeek output is to use the following script:

# xmpeek -l localhost | awk '{print $1}' | grep -v Proc | egrep -v '/$' | sed 's|/localhost/||g' > /tmp/localhost_stats.xmtrend

The resulting /tmp/localhost_stats.xmtrend file will contain a list of every metric available on the localhost, in a format you can later use to configure XMTrend.

Creating an XMTrend configuration file

Now that we have a list of metrics to record, we have to setup a configuration file for XMTrend instructing it to record those metrics for a specific duration at a specified interval.

XMTrend doesn't seem to have a default location for its configuration file, so for our purposes we will use /etc/perf/xmtrend.cf.

Sample /etc/perf/xmtrend.cf

# retain  
retain 7 1

# frequency 
frequency 2

# metrics to record
# optional wildcards (ie: CPU/cpu*/kern)
# optional per metric frequency (ie: CPU/cpu0/readch 15)
-- insert generated list of metrics here --

# start 
# the start below will record 24x7
start 00 00 00 00 00 00 00

The XMTrend configuration file begins with a retain line.

• The first parameter specifying how many days to keep a recording.
• The second parameter specifies how many days per recording file.

The next line is the recording frequency, or the interval at which samples will be taken for all metrics, except those metrics with individual frequencies.

Following the frequency, each line contains a metric to record, with wildcards allowed in the metric names. Also, an optional frequency can be specified for each metric, instead of using the default.

Last, the start line specifies the data collection period. This is supposed to indicate the day of week and hours during which to collect data, however I have insufficient samples to explain the format. The example above should however perform collection all the time, 24x7.

Running XMTrend

XMTrend can be started in your init scripts, or by hand at the command line.

The command line parameters are as follows:

• -f filename - Specifies the configuration file, including the metrics.
• -d directory - The directory to place recording files in.
• -n prefix - Prefix of each recording filename (typically xmtrend)
• -t log level - Log detail (1-9) in the XMTrend log file (not the recording file)

Typical paramters

# nohup xmtrend -f /etc/perf/xmtrend.cf -d /var/spool/xmtrend -n xmtrend -t 1 &

The command line above would run xmtrend in the background, using the configuration file located in /etc/perf/xmtrend.cf.

Recordings would be placed in /var/spool/xmtrend/ with each file named xmtrend.yymmdd (by date).

Logging would be at detail level 1 (very basic), to /var/spool/xmtrend/xmtrend.log#, where # is incremented each time it is run.

Running at boot

Normally performance data collection ought to start as soon as the system is booted.

A quick way to accomplish this is to add an inittab entry for xmtrend, ensuring it is always running.

mkitab "xmtrend:2:respawn:/usr/bin/xmtrend -f /etc/perf/xmtrend.cf -d /var/spool/xmtrend -n xmtrend -t 1"

Interpreting the recordings

Recordings can be loaded at a later date into the jazizo tool, for interpretation.

How to use jazizo is beyond the scope of this tutorial.

Long Term Recording

Typical disk usage for a recording file on a small machine with low activity was on the order of 7.5k/sample.

At a sampling rate of one every two minutes, approximately 2 Gigabytes of storage are required for a year of recordings.

There is supposed to be a tool to compress/consolidate the recordings, but that is not documented here.

Feedback

If you found this document helpful, or have suggestions regarding the content, please email me at either rladams@kelsey-seybold.com, or rladams@adamsinfoserv.com.

I've recently discovered that I'm the first hit on Google when searching for "xmtrend".

I was unaware this page was so popular.

Given that, I'm keenly interested in feedback from readers!

-- RussellAdams - 2 Feb 2005