SarCheck(TM): Automated Analysis of AIX sar and ps data

(English text version 4.00)

This is an analysis of the data contained in the file daxsar0831. The data was collected on 08/31/2000, from 08:00:01 to 17:00:01, from the RS/6000 IBM Model 7042/7043 (ED) system 'localhost'. There were 9 data records used to produce this analysis. The operating system used to produce the sar report was Release 4.3 of AIX. 1 processor is configured and 1 processor is online. 64 megabytes of memory are present.

NOTE: Sar -d statistics were not seen. In AIX version 4.3, undocumented sar -d statistics are sometimes seen when sar -A is run. Since these statistics are undocumented, we can not tell you how to make them appear. This data is not required by SarCheck but we will use the data when it is available and appears to be reliable.

SUMMARY

When the data was collected, no CPU bottleneck could be detected. A moderate I/O bottleneck was seen, but there was insufficient information available to determine its cause. A change to at least one tunable parameter has been recommended. No impending capacity limits were noted by SarCheck's capacity planning feature.

Some of the defaults used by SarCheck's rules have been overridden using the sarcheck_parms file. See the Custom Settings section of the report for more information.

RECOMMENDATIONS SECTION

NOTE: The following recommendations are being made in an environment where a large amount of spare capacity existed. If the sar data used to produce this report comes from a time when activity was uncharacteristically light, these recommendations may not help performance when the system is busy.

All recommendations contained in this report are based solely on the conditions which were present when the performance data was collected. It is possible that conditions which were not present at that time may cause some of these recommendations to result in worse performance. To minimize this risk, analyze data from several different days, implement only regularly occurring recommendations, and implement them one at a time.

Change the value of the thrashing threshold from 6 to 0 with the command '/usr/samples/kernel/schedtune -h 0'. This change disables the virtual memory manager's ability to suspend processes. If it causes performance degradation, add memory.

Changes made with schedtune are immediate and will last until the next reboot. If these changes improve performance, put the command in /etc/inittab or one of the /etc/rc scripts that are run when the system is booted.

Change the value of maxpgahead from 8 to 16 with the command '/usr/samples/kernel/vmtune -R 16'. This change has been recommended because an I/O bottleneck was detected. More information is available in the Resource Analysis Section of this report.

Change the value of maxfree from 127 to 135 with the command '/usr/samples/kernel/vmtune -F 135'. This change is recommended because maxfree should be at least as large as minfree plus maxpgahead. The value of minfree is 119 and the recommended value of maxpgahead is 16.

Changes made with vmtune are immediate and will last until the next reboot. If these changes improve performance, put the command in /etc/inittab or one of the /etc/rc scripts that are run when the system is booted.

Use the iostat command to look for disk bottlenecks. The filemon command may also be useful if it is installed on your system. The system appeared to spend siginificant time waiting for I/O but sar -d statistics were not seen, so no details are available to SarCheck.

A CPU upgrade is not recommended because the current CPU had significant unused capacity.

RESOURCE ANALYSIS SECTION

Average CPU utilization was only 27.2 percent. This indicates that spare CPU capacity exists. If any performance problems were seen during the entire monitoring period, they were not caused by a lack of CPU power. CPU utilization peaked at 33 percent from 08:00:01 to 09:00:03.

The run queue had an average length of 2.0 which indicates that processes were generally not bound by latent demand for CPU resources. Average run queue length (when occupied) peaked at 2.3 during multiple time intervals. The run queue length is the average number of processes which were ready to run. Consistently high values indicate a CPU-bound environment.

The CPU was idle (neither busy nor waiting for I/O) and had nothing to do an average of 43.0 percent of the time. If overall performance was good, this means that on average, the CPU was lightly loaded. If performance was generally unacceptable, the bottleneck may have been caused by remote file I/O which cannot be directly measured with sar and therefore cannot be considered by SarCheck.

The CPU was waiting for I/O an average of 29.8 percent of the time. This infers that the system may have been somewhat I/O bound. The time that the system was waiting for I/O peaked at 34 percent from 11:00:02 to 12:00:02.

Modest buffer cache activity was seen in the sar -b data. This indicates that some process is using raw block or raw character devices and a small amount of activity is not unusual.

The average context switch rate (cswch/s) was 462.67 per second. The context switch rate (cswch/s) peaked at 1029.0 per second from 08:00:01 to 09:00:03. Peak resource utilization statistics can be used to help understand performance problems. If performance was worst during the period of peak context switching, then a problem may be that too many processes were blocked for I/O or IPC.

No evidence of an overall memory shortage was seen in the following statistics: The average number of kernel threads waiting to be paged in (swpq-sz) was 1.66. The average number of kernel threads waiting to be paged in peaked at 1.9 during multiple time intervals. Peak resource utilization statistics can be used to help understand performance problems. If performance was worst when the number of kernel threads waiting to be paged in was at its peak, then a shortage of physical memory may be performance bottleneck.

The average number of page replacement cycles per second (cycle/s) was 0.00. If values greater than zero had been seen, a memory shortage might exist. Data from the cycle/s column does not indicate a lack of physical memory.

A recommendation has been made to disable thrashing control with schedtune -h. More information about this recommendation can be found on page 74 of Rudy Chukran's "Accelerating AIX". If this change causes performance to get worse, more memory should be added to the system because the current 64 megabytes is not enough.

The current setting for maxpin (vmtune -M) leaves 12.88 megabytes of memory unpinnable. No recommendation made because no problem was seen.

A change to the value of maxfree (vmtune -F) has been recommended in order to keep its value at least as large as minfree + maxpgahead.

An I/O bottleneck was seen in the sar statistics. A change to maxpgahead has been recommended and this change is most likely to help if a significant percentage of the disk I/O is sequential. You may be able to characterize the I/O with the filemon utility but the best way to see if a maxpgahead change will help is to try it.

The value of numclust (vmtune -c) is 1. If fast disk devices, disk arrays, or striped logical volumes are in use, the performance of disk writes could be improved by increasing this value. There is insufficient information to make any specific recommendation for tuning numclust.

The average rate of System V semaphore calls (sema/s) was 0.6 per second. No problems have been seen, and no changes have been recommended for System V semaphore parameters. Note that SarCheck only checks these parameter's relationships to each other since semaphore usage data is not available.

The average rate of System V (msg/s) message calls was 1242.5 per second. System V message activity peaked at a rate of 1847.64 per second from 15:00:01 to 16:00:01. Peak resource utilization statistics can be used to help understand performance problems. If performance was worst during the period of peak message activity, then that activity may be a performance bottleneck and application or database activity related to message usage should be looked at more closely. No problems have been seen, and no changes have been recommended for System V message parameters. Note that SarCheck only checks these parameter's relationships to each other since message usage data is not available.

There were no times when enforcement of the process threshold limit (kproc-ov) prevented the creation of kernel processes. This indicates that no problems were seen in this area.

The ratio of exec to fork system calls was 0.98. This indicates that PATH variables are efficient.

CAPACITY PLANNING SECTION

This section is designed to provide the user with a rudimentary linear capacity planning model and should be used for rough approximations only. These estimates assume that an increase in workload will affect the usage of all resources equally. These estimates should be used on days when the load is heaviest to determine approximately how much spare capacity remains at peak times.

Based on the limited data available in this single sar report, the system should be able to support a substantial increase in workload before impending CPU, memory, disk, or system table bottlenecks are seen. Run SarCheck regularly to detect bottlenecks before they impact performance.

CUSTOM SETTINGS SECTION

The default MLRATE threshold was changed in the sarcheck_parms file from 200.0 to 500.0 kb per hour but there is no ps -elf data to analyze.

Please note: In no event can Aurora Software Inc. be held responsible for any damages, including incidental or consequent damages, in connection with or arising out of the use or inability to use this software. All trademarks belong to their respective owners. Evaluation copy for: Your Company. This software expires on 01/20/2001 (mm/dd/yyyy). Code version: 4.00. Serial number: 00012345.

Thank you for trying this evaluation copy of SarCheck. To order a licensed version of this software, just type 'analyze -o' at the prompt to produce the order form, and follow the instructions.

(c) copyright 1995-2000 by Aurora Software Inc., Plaistow NH, USA, All Rights Reserved. http://www.sarcheck.com