SarCheck®: Automated Analysis of Linux data

(English text version 6.01.13)

This is an analysis of the data contained in the file 20070420. The data was collected on 04/20/2007, from 00:00:00 to 12:30:01, from system 'web-prod1'. Operating system was 2.6.13.3. 2 processors are present. The ht (hyperthreading) flag was seen in /proc/cpuinfo. The number of CPUs may not be reported correctly because it is impossible to tell if hyperthreading has been enabled. 883 megabytes of memory are present.

Data collected by the ps -elf command on 04/20/2007 from 03:40:00 to 12:20:00, and stored in the file ps0420, will also be analyzed.

The starting time was set to 03:30:00 by the -st switch. The ending time was set to 12:30:00 by the -en switch.

Table of Contents

• Recommendations Section
• Resource Analysis Section
• Capacity Planning Section
• Summary of Statistics

SUMMARY

When the data was collected, occasional CPU bottlenecks may have existed. A moderate memory bottleneck was seen. I/O bottlenecks were seen at peak times. A change has been recommended to at least one tunable parameter. Recommendations can be found in the Recommendations Section of this report.

RECOMMENDATIONS SECTION

All recommendations apply to 'web-prod1' operating system 2.6.13.3. These recommendations contained in this report are based solely on the conditions which were present when the performance data was collected. It is possible that conditions which were not present at that time may cause some of these recommendations to result in worse performance. To minimize this risk, analyze data from several different days and implement only regularly occurring recommendations.

Consider adjusting process priorities with the nice(C) command, optimizing applications, or improved job scheduling. Occasionally heavy CPU utilization was seen and these recommendations may help to avoid the need for a hardware upgrade.

Add memory. Additional memory may improve performance. If possible, borrow some memory for test purposes, and monitor system performance and resource utilization before and after installation.

Change the virtual memory parameter 'swappiness' from 60 to 70. This controls the systems likelihood of swapping out memory pages which have not been recently used. Larger values will swap out unused pages of memory, speeding the time required to load new pages into memory.

To change the value of swappiness immediately as described, use the following command:

    sysctl -w vm.swappiness="70"

If you are able to make this change and it improves performance, you can make the change permanent by adding the command to the /etc/sysctl.conf file.

Try to balance the disk I/O load across time or among other disk devices. The load on at least one disk was clearly excessive at peak times. By distributing the load over a greater time period or across other disk devices, the load will not be as likely to cause performance degradation.

RESOURCE ANALYSIS SECTION

While average CPU utilization was only 16.5 percent, occasional bursts in excess of 70 percent were seen. Depending on their frequency, periods of performance degradation due to CPU loading may have been felt by end users. CPU utilization peaked at 100.00 percent from 08:38:35 to 08:40:14.

Using the nice(C) command, optimizing applications, or improved job scheduling may help to prevent occasional performance degradation.

CPU number 0 was busy for an average of 26.17 percent of the time. During the peak interval from 08:50:05 to 08:52:11, this CPU was 100.00 percent busy. The CPU was busy with user work 24.04 percent of the time and was busy with system work 2.12 percent of the time. The sys/usr ratio on this CPU was 0.09 This is below the threshold of 2.50:1.

CPU number 1 was busy for an average of 8.64 percent of the time. During the peak interval from 08:38:35 to 08:40:14, this CPU was 100.00 percent busy. The CPU was busy with user work 7.78 percent of the time and was busy with system work 0.85 percent of the time. The sys/usr ratio on this CPU was 0.11:1.

The average amount of free memory was 6301.1 pages or 24.6 megabytes. The minimum amount of free memory was 1828 pages or 7.14 megabytes at 08:14:01.

The average amount of free memory including cached memory and memory used for buffers was 19124.4 pages or 74.7 megabytes. This is 8.5 percent of all the memory seen on the system. The minimum amount of free memory including cached memory and memory used for buffers was 6714 pages or 26.23 megabytes at 04:06:08.

In the following graph, the 'free' utility showed that the system could not support a large cache or buffers due to memory pressure.

A moderate memory bottleneck was seen. Non-zero swap out activity was seen at least one quarter of the time.

The value of the page_cluster parameter was 3. This means that 8 pages are read at once. Values of 2 or 3 are typically better for systems will small memory sizes, systems where response time is important, and systems where most I/O is not sequential. There may be an advantage in raising this value if you want to speed up the performance of programs which do mostly sequential I/O.

The average page in rate was 232.15 per second. Page ins peaked at 2680.39 per second from 11:48:08 to 11:50:02. An unusually high page in rate was detected. This may be normal for your environment, but it is still worth noting. The average page out rate was 248.56 per second. Page outs peaked at 1373.41 per second from 08:38:35 to 08:40:14. An unusually high page out rate was detected. This may be normal for your environment, but it is still worth noting.

The average swap in rate was 46.66 per second. Swap ins peaked at 653.72 per second from 11:48:08 to 11:50:02. An unusually high swap in rate was detected. This may be normal for your environment, but it is still worth noting. The average swap out rate was 30.15 per second. Swap outs peaked at 333.86 per second from 08:38:35 to 08:40:14. An unusually high swap out rate was detected. This may be normal for your environment, but it is still worth noting.

The /proc/sys/vm/swappiness file was seen. The swappiness value was 60. A higher value has been recommended for swappiness. This recommendation works best in a server environment where it makes sense to write unused pages of memory to swap in order to free the memory for some other process. It may cause response time problems for someone trying to use the system as a desktop.

The amount of swap space in use peaked at 1059.74 megabytes at 11:44:09. The average amount of swap space in use was 1005.85 megabytes. The size of swap space was 2047.98 megabytes. The peak amount of swap space in use was 51.75 percent of the total.

There was one swap partition seen in /proc/swaps.

The value of the logging_level parameter was 0. A value of zero disables logging. This has the least overhead but provides no information about SCSI activity. This program does not use data collected by SCSI logging and this information is provided in case the user wishes to get deeply involved in the SCSI activity of their system.

According to data collected from /proc/partitions, the system-wide disk I/O rate averaged 68.95 per second and peaked at 479.57 per second from 08:18:01 to 08:20:08. The read rate averaged 36.46 per second and peaked at 437.59 per second from 11:48:08 to 11:50:02. The write rate averaged 32.49 per second and peaked at 289.72 per second from 08:38:35 to 08:40:14.

The -dtoo switch has been used to format disk statistics into the following table.

The I/O rate on disk device sda averaged 69.21 per second and peaked at 479.57 per second from 08:18:01 to 08:20:08. The read rate averaged 36.60 per second. The write rate averaged 32.61 per second. This disk was busy for an average of 21.62 percent of the time and was 100.83 percent busy at peak times. According to the information collected from the system sda device used the anticipatory scheduler.

The noatime option was not specified on any of the mounted filesystems. This option was checked because non-trivial levels of disk activity were seen. Disk activity on the busiest disk device, sda, peaked at 100.83 percent busy from 04:42:05 to 04:44:03.

The value of the ctrl-alt-del parameter was 0. The value of 0 is better in almost all cases because it prevents an immediate reboot if the ctrl, alt, and delete keys are pressed simultaneously.

There were an average of 500.06 interrupts per second and the peak interrupt rate seen was 1222.60 per second from 08:42:01 to 08:44:02. The following graph shows the total interrupt rate during the monitoring period.

At 07:00:01 ps -elf data indicated that there were a peak of 89 processes present. This was the largest number of processes seen with ps -elf but it is not likely to be the absolute peak because the operating system does not store the true "high-water mark" for this statistic. There were an average of 83.8 processes present.

No runaway processes, memory leaks, or suspiciously large processes were detected in the data contained in file ps0420. No table was generated because no unusual resource utilization was seen in the ps data.

CAPACITY PLANNING SECTION

The section is designed to provide the user with a rudimentary linear capacity planning model and should be used for rough approximations only. These estimates assume that an increase in workload will affect the usage of all resources equally. These estimates should be used on days when the load is heaviest to determine approximately how much spare capacity remains at peak times.

Based on the limited data available in this single day of data, the system cannot support an increase in workload at peak times without some level of performance degradation. See the following paragraphs for additional information on the capacity of individual system resources. Implementation of some of the suggestions in the recommendations section may help to increase the system's capacity.

The CPU can support an increase in workload of approximately 0 percent at peak times. For more information on peak CPU and disk utilization, refer to the Resource Analysis section of this report. The system was actively trying to meet the need for free memory. A significant increase in workload may have to be accompanied by an increase in memory. The busiest disk can support a workload increase of approximately 0 percent at peak times. For more information on peak CPU and disk utilization, refer to the Resource Analysis section of this report.

Please note: In no event can Aptitune Corporation be held responsible for any damages, including incidental or consequent damages, in connection with or arising out of the use or inability to use this software. All trademarks belong to their respective owners. This software licensed exclusively for use on a single system by: Your Company. This software expires on 09/30/2007 (mm/dd/yyyy). Code version: SarCheck for Linux 6.01.13. Serial number: 00069665.

This software is updated frequently. For information on the latest version, contact the party from whom SarCheck was originally purchased, or visit our web site.

(c) Copyright 2003-2007 by Aptitune Corporation, Plaistow NH 03865, USA, All Rights Reserved. http://www.sarcheck.com/