Introduction
Features
Restrictions
Known limitations
How to install SarCheck (internal diskette drive)
How to install SarCheck (no internal diskette drive)
How to set up sar
How to set up ps -elf data collection
How to set up gnuplot
How to activate the software
How to deinstall SarCheck
How to run SarCheck, menu driven
How to run SarCheck, command driven
How to run SarCheck, crontab entry
How to analyze multiple sar reports
How to analyze real time data
How to change the menu defaults
How to change SarCheck's algorithms
How to create graphs from sar reports
Examples of how to use the switches
How to move SarCheck to another directory
How to produce the most accurate analysis from the menu
How to produce the most accurate analysis from the command line
How to access the online instructions/help text
How to get the most from SarCheck
How to interpret the analysis
The summary section
The recommendations section
The resource analysis section
The capacity planning section
The custom settings section
Disclaimers, trademarks, etc.
How to order SarCheck
How to get technical support for SarCheck
How to get technical support for gnuplot
Files included in this release
An example of a SarCheck report
The summary section
The recommendations section
The resource analysis section
The capacity planning section
The custom settings section
Frequently Asked Questions (The FAQ)
Bibliography
Special thanks
Appendix A: SarCheck_parms file keywords
Appendix B: Options available when running 'analyze'
The SarCheck utility analyzes your system for possible performance bottlenecks such as memory shortages and 'leaks', disk load imbalances, CPU bottlenecks, runaway processes, and improperly set tunable parameters. It also tells you approximately how much of an additional workload the system can support at peak times of the day. It does this by analyzing a user-specified sar report, the output of ps -elf, scanning various kernel structures, and producing a plain English report. The report explains the resource bottlenecks seen and makes recommendations which can improve system performance.
SarCheck will convert sar reports into a CSV-formatted form which is used by many popular graphing tools. It will also produce graphs if you have a version of gnuplot installed that supports PNG or JPEG output. If you ask for HTML output and the production of graphs, SarCheck will insert the graphs into the HTML document. The graphs will be inserted by using <img> tags complete with a description in the tag's alt attribute in order to meet accessibility requirements.
Important: SarCheck has not been designed to analyze sar data from one system on another. This is because not all of the data needed is in the sar and ps reports, and SarCheck has to look for additional data in the kernel. If SarCheck uses sar data from one system and kernel data from another system, it will issue a warning.
SarCheck's recommendations are designed to produce incremental improvements, so SarCheck should be run regularly. No attempt is made to guess the ultimately correct value for any parameter based on a single day's sar data. Instead, SarCheck will recommend that you increase or decrease values based on the data available, and will continue to recommend changes until there is no more room for improvement. Performance tuning is, by definition, a process of trial and error. SarCheck will not only help you to make those changes, but will also explain the reasons for each recommendation.
SarCheck is different from other performance tools because it does not monitor system activity. In much the same way that a UNIX performance expert would approach the problem, SarCheck uses the sar utility along with ps and several other tools, and analyzes the data available. Since sar is included with the operating system, we didn't see a need to create yet another monitor for you to buy.
SarCheck can be run from the command line, from a menu-driven front end script, or by cron. You must be logged in as root or superuser because SarCheck needs to interrogate the kernel in order to find the values of various tunable parameters, memory size, etc. For reasons of safety and security, SarCheck will not attempt to change tunable parameters or anything else in the kernel.
Based on its analysis of the resources and statistics described above, SarCheck may recommend a variety of steps which can be taken to improve system performance.
SarCheck for Solaris SPARC is designed to work with Solaris SPARC 2.5+ and Solaris x86 10+. Other versions of SarCheck are available for most versions of AIX, HP-UX, and Linux x86.
Note: This is a PC-formatted diskette and the software is a compressed tar archive named either scsol32.taz, scsol64.taz, or scsolx86.taz, depending on the platform it was built for. The volcheck command can be used to mount the floppy, and the file should be in the /floppy/floppy0 directory.
To install the software, log in as root, mount the floppy disk using volcheck, and detar the software. This only takes a few seconds.
volcheck
In some cases, it may be necessary to mount the floppy with the following command:
/etc/mount -F pcfs /dev/diskette0 /floppy
zcat < /floppy/floppy0/scsol64.taz | tar xvf - (64-bit SPARC)
zcat < /floppy/floppy0/scsol32.taz | tar xvf - (32-bit SPARC)
zcat < /floppy/floppy0/scsolx86.taz | tar xvf - (32-bit x86)
This will install the files on your system, described in the section "Files included in this release". The installation of SarCheck does not require rebuilding the kernel. This is important because it means that SarCheck will not increase the size of your kernel, and you won't have to reboot your system. Setting up sar may require a reboot.
To try SarCheck, type
cd /opt/sarcheck
./bin/analyze -ps ./etc/solsar22 | more
Warning: Do not implement any recommendations produced by analyzing the test file solsar22! This file has been included for test purposes only.
To reduce typing, you may want to add /opt/sarcheck/bin to root's PATH.
Note: This is a PC-formatted diskette and the software is a compressed tar archive. Mount the floppy in a system with a diskette drive and FTP the file named either scsol32.taz, scsol64.taz, or scsolx86.taz to the system you want to install SarCheck on.
Then just log in as root and detar the file. This only takes a few seconds.
zcat < /floppy/floppy0/scsol64.taz | tar xvf - (64-bit SPARC)
zcat < /floppy/floppy0/scsol32.taz | tar xvf - (32-bit SPARC)
zcat < /floppy/floppy0/scsolx86.taz | tar xvf - (32-bit x86)
This will install the files on your system, described in the section "Files included in this release". The installation of SarCheck does not require rebuilding the kernel. This is important because it means that SarCheck will not increase the size of your kernel and you won't have to reboot your system. Setting up sar may require a reboot.
To try SarCheck, type
cd /opt/sarcheck
./bin/analyze -ps ./etc/solsar22 | more
Warning: Do not implement any recommendations produced by analyzing the test file solsar22! This file has been included for test purposes only.
To reduce typing, you may want to add /opt/sarcheck/bin to root's PATH.
First, make sure the /var/adm/sa directory exists, and make it if it doesn't.
Next, make sure the SUNWaccu and SUNWaccr packages are installed.
Finally, uncomment the 3 lines in sys' crontab which run the /usr/lib/sa/sa1 and /usr/lib/sa/sa2 scripts. If these lines aren't there, add them:
0 * * * 0-6 /usr/lib/sa/sa1 20,40 8-17 * * 1-5 /usr/lib/sa/sa1 5 18 * * 1-5 /usr/lib/sa/sa2 -s 8:00 -e 18:01 -i 1200 -A
As an alternative, the following cron entries are oriented towards the 24x7 monitoring that many administrators prefer:
0,20,40 * * * * /usr/lib/sa/sa1 45 23 * * * /usr/lib/sa/sa2 -i 1200 -A
One of the most powerful features in SarCheck is its ability to analyze data collected by ps -elf and add this information to its analysis of sar data. A few simple steps are required to take advantage of this powerful feature:
mkdir /opt/sarcheck/ps
chmod g+w /opt/sarcheck/ps
chgrp sys /opt/sarcheck/ps
0,20,40 8-17 * * 1-5 /opt/sarcheck/bin/ps1
5 17 * * 1-5 /opt/sarcheck/bin/ps2
As an alternative, the following cron entries are oriented towards the 24x7 monitoring that many administrators prefer:
0,20,40 * * * * /opt/sarcheck/bin/ps1
45 23 * * * /opt/sarcheck/bin/ps2
It's best if the scheduling used for the ps1 script is the same as the scheduling for the /usr/lib/sa/sa1
We recommend using crontab -e to modify the crontab file.
The ps -elf data collected on large systems can take up a considerable amount of space. If you want to store this data somewhere other than /opt/sarcheck/ps, you can specify a different directory with the PSELFDIR keyword in the sarcheck_parms file.
WARNING: If you choose to specify a different directory, be sure to pick a directory that is not used for anything else. The purpose of the ps2 script is to remove any file in the ps -elf directory which is more than 14 days old and you don't want to accidentally remove files which contain something other than ps -elf data.
If you ask SarCheck to produce PNG or JPEG graphs, it will look for gnuplot and will try to use it to generate graphs. There are a number of different places on the World Wide Web where you can find gnuplot and it is generally available as source code or source code with a precompiled binary.
For the sake of consistency and convenience, we use the precompiled binary built by Ready to Run Software, Inc. It is available on our web site at http://www.sarcheck.com/gnuplot/
To set up gnuplot, please follow the instructions on the web site.
Evaluation and licensed software may require an activation key. If you need an activation key, run analyze -s and forward the output to sales@sarcheck.com. Feel free to install eval software on as many Solaris SPARC systems as you want and use it until it expires.
SarCheck was not designed to be regularly moved from one system to another, in an effort to provide "quick fixes" to a number of systems. Quick fixes will not allow you to take advantage of the long term iterative tuning that SarCheck makes possible.
Remove the files which are described in the section entitled "Files included in this release". If you are loading a new version of SarCheck over an old one, deinstallation is not necessary.
First, log in as root. To analyze sar statistics from a menu, type:
/opt/sarcheck/bin/sarcheck
To reduce typing, you may want to add /opt/sarcheck/bin to root's PATH.
A series of choices will appear on the screen. The easiest way to get started is to accept all the defaults by repeatedly pressing the Enter key, or entering an asterisk and pressing Enter. For security reasons, your account must have permission to access the sar data or report files that you wish to analyze.
The first question will ask you whether you want to analyze sar data, a sar report, or all of the sar reports at once by concatenating them. Sar data is usually found in /var/adm/sa/sann, where nn is the day of the month. Sar reports are already reduced into a readable form and are usually found in /var/adm/sa/sarnn.
Analyzing the reports will be marginally faster than analyzing the data, but an advantage to analyzing the data is that you can choose the start and end times for analysis by editing the sarcheck script.
Analyze what? d sar data (sa files)
r A sar report (sar files)
c Concatenate existing sar reports
* Accept all defaults
x Exit SarCheck
(keyword = DR, default = d): _
After you pick the d or r options, you will be prompted to enter the name of the data or report file. In either case the default will be the statistics from the previous business day. The c option will concatenate all of the sar reports present and will not ask you for the name of a file. The sarcheck script will change your working directory, so you do not have to use the absolute address of the file. To accept all defaults, enter an asterisk; to exit sarcheck, enter an x. To change any of the defaults, see the section "How to change the SarCheck menu defaults".
Sar data is usually found in /var/adm/sa/sann. Based on user-definable defaults, data from 08:00 to 17:00 will be analyzed. Enter the name of the sar data file that you wish to analyze.
Available data files in /var/adm/sa: sa20 sa21 sa25 sa26 sa27 sa28 sa29 sa30 (default = sa29):
Note that if you run sarcheck on a Saturday, Sunday or Monday, Friday's statistics will be analyzed by default. This is because weekend statistics are usually not representative of a "loaded" system, and there is a possibility that misleading recommendations would be generated.
The next option allows you to pick formatting. The default will produce a report with page numbers and page breaks (ctrl-L) included. For users that prefer to paginate the report with another tool, such as pg, the p option will suppress these page breaks. You can also choose to produce an HTML document at this point, and can decide whether to format the disk analysis in table form. HTML documents are best viewed with a web browser. If you wish to exit sarcheck, enter an x.
Pick formatting: n Normal, with page breaks
p Page breaks suppressed
h Create HTML document, no disk
table
t Create HTML document, with disk
table
o Create HTML document, disk
table only
* Accept remaining defaults
x Exit sarcheck
(keyword = OPT, default = n): _
The verbosity option controls how verbose the SarCheck report is. The default Verbose mode may produce a report 5 pages long, while Superquiet mode may only contain 5 lines of text. Please note that instructions for implementing recommendations, explanations, and alternate tuning strategies may be suppressed by the quiet modes. When you're first using SarCheck, we recommend using the verbose mode so that you don't miss anything. The superquiet mode will automatically suppress page breaks.
Verbosity level: v Verbose mode
q Quiet, most verbiage
suppressed
Q Superquiet, all verbiage
suppressed
* Accept all defaults
x Exit SarCheck
(keyword = VERBOSE, default = v): _
Analysis of ps -elf data will provide you with a closer look at memory bottlenecks and the ability to detect runaway processes and memory leaks. The enhanced sensitivity option increases the probability of generating "false alarms". See the section entitled "Setup of ps -elf data collection" for more information. If you wish to exit sarcheck, enter an x.
Analyze ps -elf n No, analyze sar data only
data? y Yes, analyze sar and ps
data
e Enhanced sensitivity of
ps data analysis
* Accept all defaults
x Exit sarcheck
(keyword = PSELFOPT, default = n): _
The disk filtration option is useful for large systems and has no effect on smaller systems. If more than 12 disks are present, sarcheck will only print a paragraph on each disk which has significant activity. SarCheck will try to determine what "significant" means on your system. If you find that SarCheck's filtration is too aggressive, use "z" to show all disks with some activity. We have discovered that on systems with several thousand disk devices, it is common for most of them to be completely idle. The "z" option will filter out only disks with no activity at all.
Disk filtration: y Filter disk analysis more than 12
disks.
n Analyze all disk activity if more
than 12 seen.
z Analyze all disks with some
activity.
* Accept all defaults
x Exit sarcheck
(keyword = DISKFLTR, default = y):
The tabular summary will print a summary of statistics in table form at the end of the report. An example is included later in the manual. If HTML output has been selected, an HTML table is created.
Tabular Summary? y Print a tabular summary at the
end of the report
i Print a tabular summary instead of
the report
n Print the report without a summary
* Accept remaining defaults
x Exit sarcheck
(keyword = TABULAR, default = n):
The "send output to:" option is used to decide where to send the analysis. Note that some of these choices will be different based on the pager you use and any modifications made to the sarcheck script.
If you choose to send the output to a file, you'll be prompted for the name of the file. The default file name is /tmp/yyyymmddhhmmss, which is a date/time stamp.
Send output to: 1 more (the screen)
2 lp -s (a printer)
3 A file
x Exit sarcheck
(keyword = OUTOPT, default = 1): _
You must be root / superuser to run SarCheck. To analyze a sar report file called sar12, type:
/opt/sarcheck/bin/analyze sar12
For best results, pipe the output to more so that you can read it, or redirect it to a file if you want to save it. A report will be produced which contains information about your system, a brief summary, a recommendations section (if applicable), a resource analysis section, and a capacity planning section. For more information, see the section entitled "How to Interpret the Analysis".
For users that prefer to paginate the report with another utility, such as pg, the -p option will suppress page numbers and page breaks. To take advantage of this option, type:
/opt/sarcheck/bin/analyze -p sar12 | pg
To reduce typing, you may want to add /opt/sarcheck/bin to root's PATH.
SarCheck can be run automatically by adding an entry to root's crontab file, ideally using crontab -e. Determine the time that /usr/lib/sa/sa2 is run (you'll have to set this up), and use cron to run /opt/sarcheck/bin/analyze after that time. Here are two examples which assume sa2 is run at 18:00, the analysis will be done at 18:05:
In order to print a SarCheck analysis every weeknight, use the following entry:
5 18 * * 1-5 /opt/sarcheck/bin/analyze /var/adm/sa/sar`date +\%d` | lp -s
(this should all be on one line)
To keep all of SarCheck's recommendations in the /usr/ops directory, use the following entry:
5 18 * * 1-5 /opt/sarcheck/bin/analyze /var/adm/sa/sar`date +\%d` > /usr/ops/`date +\%y\%m\%d`
(this should all be on one line)
Because the output of the analyze program is stdout, you can pipe or redirect it in lots of ways. It can be printed, mailed, stored... whatever works best in your environment.
SarCheck has the ability to analyze multiple days of data at once. SarCheck will analyze any number of concatenated sar reports. Here is an example of how to analyze all sar reports from the first seven days of the month.
cat /var/adm/sa/sar0[1-7] > /tmp/multisar
/opt/sarcheck/bin/analyze /tmp/multisar | more
Please note that the analyze program does not work if wildcard characters are used as a filename. Wildcard characters should be used with the cat command in order to produce a single file for the analyze program. Once you've become used to working with concatenated sar reports, you'll probably discover that the find command in /usr/lib/sa/sa2 removes the sar reports too quickly, and the naming convention used in /usr/lib/sa/sa1 is too restrictive. Make copies of the original sa1 and sa2 scripts, and then modify them to meet your needs.
The script /opt/sarcheck/bin/ondemand can be used to analyze resource utilization and it works much like sar or vmstat. To collect 10 sar and ps -elf samples of 30 seconds each and analyze them, use the following command:
/opt/sarcheck/bin/ondemand 30 10 | more
Because the output of the ondemand script can be lengthy and is stdout, you'll probably want to pipe it to more or redirect it to a file. It's best to use a sampling interval of at least 15 seconds because this script runs sar and ps with enough frequency to create a noticeable load on the system when the sampling interval is short.
The SarCheck script looks for the file named sarcheck_parms (by default in /opt/sarcheck/etc) and will use any values found there instead of the normal defaults. This file is not included as part of the SarCheck distribution and you'll need to create it if you want to use it. The syntax for the sarcheck_parms file is very simple. Create a line in the file with the keyword and its new default value, separated by a space.
The keywords for SarCheck's menu options can be found when running the SarCheck script, and the value should be one of the choices on the menu. For example, if you want the output of SarCheck to include a tabular summary at the end of the report, here is the menu selection that you will see:
Tabular Summary? y Print a tabular summary at the
end of the report
i Print a tabular summary instead of
the report
n Print the report without a summary
* Accept remaining defaults
x exit sarcheck
(keyword = TABULAR, default = n):
You can see the name of the keyword and the options available. To change the default from 'n' to 'y' for this menu item, add the following line to the sarcheck_parms file:
TABULAR y
Now when you run the SarCheck script, the default behavior will be to print a tabular summary at the end of the report.
After those two fields are parsed by the SarCheck script, the rest of the line is ignored and is available as a comment. Any line that starts with something other than a valid keyword is also treated as a comment and is ignored.
Once you have decided to change the defaults, create or edit the sarcheck_parms file. Here is an example of a sarcheck_parms file where the starting and ending times used for analysis have been changed, page numbering is suppressed, and a tabular summary is printed at the end of the report. Note that since the SarCheck script only looks at the first two fields on each line, the rest of the line is treated as a comment and lines that don't start with valid keywords are also treated as comments:
# file to customize sarcheck created by # Jess the sys admin on March 23, 2002 # ST 06:00 starting time is 6AM EN 15:00 ending time is 3PM OPT p suppress page numbering TABULAR y add a tabular summary GRAPHDIR /diskfarm/sarcheck/images
A complete list of keywords supported in the sarcheck_parms file can be found in Appendix A.
The sarcheck_parms file can also be used to change the thresholds used by SarCheck's tuning algorithms. Default values of SarCheck's thresholds have been established based on feedback from hundreds or thousands of systems and these values should not be overridden without good reason. A complete list of keywords supported in the sarcheck_parms file can be found in Appendix A.
There are two ways to make graphs from the sar data used by SarCheck.
The first way is to let SarCheck build graphs with the gnuplot utility. By adding the -png, -jpg, or -jpeg switches, SarCheck can use gnuplot to produce PNG or JPEG graphs and can insert those graphs in the HTML output of SarCheck. This will enable you to post some really interesting SarCheck reports on your corporate intranet. To produce an HTML report with PNG graphs, use the -html and -png switches when running analyze. For example, the command
analyze -html -png sar12 > rpt12.html
will produce an HTML report which can be read by your favorite browser. The most important parts of the report will be printed in bold type and headings are used to clarify what you're looking at. Graphs are inserted in appropriate places in the body of the report and some additional text is added to help explain the significance of the graphs. For more information, see the section entitled "Options available when running 'analyze'".
If you want SarCheck to produce graphs without the accompanying SarCheck report, use the -gonly switch to produce "graphs only".
The second way to produce graphs is by exporting CSV (Comma Separated Value) formatted data to a graphing program. The -gr switch in the analyze program will turn a sar report into output in CSV format. This format is easily understood by most spreadsheets. It's more work, but it is the best way to go if you don't have gnuplot or want to produce custom graphs.
The number of switches and options available in SarCheck continues to grow, and this section is designed to help you decide how to do what you want. In order to maintain some level of clarity, all examples will analyze the sar report file /var/adm/sa/sar23. The output of the analyze program is stdout, so you'll probably want to pipe it to more or redirect it to a file.
Example 1: Analyzing a sar report. We're going to start with the simplest possible example. The command below will run the analyze program and tell it to analyze the sar report file /var/adm/sa/sar23.
analyze /var/adm/sa/sar23
Example 2: Removing the page breaks. The -p switch removes the page breaks. This is especially useful when piping the report to pg instead of more. A number of other switches, the -html switch for example, will automatically invoke -p where it makes sense.
analyze -p /var/adm/sa/sar23
Example 3: Analyzing ps -elf output in conjunction with the sar report. The /opt/sarcheck/bin/ps1 script is used to collect ps -elf data which can be used in conjunction with the sar report. The -ps switch tells the analyze program to search for the ps -elf data and include it in the report if possible.
analyze -ps /var/adm/sa/sar23
Example 4: Creating an HTML-formatted report. Let's combine a few switches this time. The -html switch makes the -p switch unnecessary, but we want to incorporate ps -elf data into the analysis and we want to see a quick table of some statistics at the end of report.
analyze -html -ps -t /var/adm/sa/sar23
Example 5: Creating an HTML-formatted report with embedded graphs. This is the same as the previous example, except that graphs are now embedded in the HTML output and they will be visible when the output is viewed with a browser. For this to work properly, you must have a copy of gnuplot installed, SarCheck must be able to find it, and the output must be redirected to a file. The file is then opened with a browser that can display PNG graphs. You can also use the -jpeg or -jpg switches if your version of gnuplot supports jpeg output.
analyze -html -ps -t -png /var/adm/sa/sar23 > sar23.html
Example 6: Creating an HTML-formatted report, but this time on a big system. This time we're analyzing sar data from an E10000 system, and there are 250 disks on the system. By default, SarCheck will filter out information on 'uninteresting' disks, but it will still produce a paragraph on each disk. This can get a little ridiculous and hard to read, so we'll use the -dtbl switch to format the disk information into an HTML table, the -dbusy switch to sort the disk information by "%busy", the -d switch to show all disks (this will be a big table), and the -ctd switch to show the device names in the more useful c0t0d0 format. As with the -t switch, cells will be colored if SarCheck wants to draw your attention to specific data.
analyze -html -ps -t -dtbl -dbusy -d -ctd /var/adm/sa/sar23
Example 7: Emailing the output. If you manage a network of 200 systems, you may want to email interesting SarCheck reports to yourself, but you probably don't want to be spammed with 200 messages a day saying that everything's okay. The -r switch prevents SarCheck from producing any output at all if there are no recommendations. The -Q switch reduces verbiage to a minimum.
analyze -ps -r -Q /var/adm/sa/sar23 | mail root@wherever.com
Example 8: Suppressing the detection of memory leaks. False alarms are not uncommon when SarCheck attempts to detect memory leaks. Some programs, such as those found on systems running Oracle or other database products, will grow over time. This is apparently a deliberate memory leak, and depending on the behavior of programs running on your system, you may want to suppress the reporting of memory leaks, runaway processes, or unusually large processes. The -pml switch can be used to suppress memory leaks as follows:
analyze -ps -pml /var/adm/sa/sar23
Example 9: Specifying a different ps -elf file. If you move the SarCheck files to a directory other than /usr/local and you want to analyze ps -elf data, you have to tell the analyze program where to find the data. This example looks at the sar report sar23 and ps -elf file /tmp/pselffile:
analyze -ps -pf /tmp/pselffile /var/adm/sa/sar23
Example 10: Producing output which is easily parsed. If you want to use the analyze program to filter data from a sar report and produce output which is easily parsed, use the -csv switch.
analyze -csv /var/adm/sa/sar23
Example 11: Generating graphs from a sar report. A sar report can be turned into a .csv file which can be graphed with a spreadsheet using the -gr switch:
analyze -gr /var/adm/sa/sar23 > graph.csv
Example 12: Sending just the summary to your boss. Naturally the email address is ficticious in this example, and it's a way to not bury your boss in the detail of a SarCheck report and its specific recommendations:
analyze -summ /var/adm/sa/sar23 | mail your.boss@work.com
SarCheck by default is located in the /opt/sarcheck directory. There are two ways that you can move SarCheck to another directory. We will give you an example of each way and will use /tmp/sarcheck as the new location of SarCheck.
Example A will use an environment variable called SARCHECKDIR.
SARCHECKDIR=/tmp/sarcheck
export SARCHECKDIR
mv /opt/sarcheck/bin/* /tmp/sarcheck/bin
mv /opt/sarcheck/etc/* /tmp/sarcheck/etc
mv /opt/sarcheck/ps/* /tmp/sarcheck/ps
mv /opt/sarcheck/doc/* /tmp/sarcheck/doc
PATH=$PATH:/tmp/sarcheck/bin
Example B will use the sarcheck_parms file.
mv /opt/sarcheck/bin/* /tmp/sarcheck/bin
mv /opt/sarcheck/etc/* /tmp/sarcheck/etc
mv /opt/sarcheck/ps/* /tmp/sarcheck/ps
mv /opt/sarcheck/doc/* /tmp/sarcheck/doc
SARCHECKDIR /tmp/sarcheck
Important: Understand that you can inadvertently cause SarCheck to produce misleading or incorrect recommendations. SarCheck looks at an individual day of data reported by sar. That data should reflect the most active time of the day when performance is most important and the most active days of the week or month.
Analyze the data or report files that represent the busiest days. Determine the busiest times of the day, and if necessary, modify the menu defaults to analyze data from only that time period. To change any of the defaults, see the section "How to change the SarCheck menu defaults".
Important: Understand that you can inadvertently cause SarCheck to produce misleading or incorrect recommendations. SarCheck looks at an individual day of data reported by sar. That data should reflect the most active time of the day when performance is most important and the most active days of the week or month.
For example, if your system is in use from 8:00AM to 5:00PM, and runs batch jobs and backups at night with plenty of time to spare, performance is probably most important during the day. To help SarCheck produce the most useful analysis, use a command such as:
sar -A -s8 -e17 > reportfile
This command will use all options (-A), and will only include data collected between 8:00AM (-s8) and 5:00PM (-e17) in the report. If the busiest day in the last few weeks was the 29th of the month, and you want to produce a report of system activity between 8:00 and 5:00 on that day, use the following command:
sar -A -s8 -e17 -f /var/adm/sa/sa29 > report29
Of course, this will only work if /va/adm/sa/sa29 actually exists. Sar data should be regularly collected by a crontab entry.
It's important to analyze reports from days when the processing load was greatest. It may be that on those days, SarCheck will find resource bottlenecks which did not exist on days when the system did less work.
If you analyze data from the weekend, SarCheck may tell you how to optimize the system for weekend processing. Whether that makes sense or not in your environment (and it frequently won't) is up to you.
To display the online help text, type:
/opt/sarcheck/bin/analyze -h
This sends a subset of the instructions found in this manual to standard output (stdout), which defaults to the screen. More details are found in this manual.
For other help text, type:
/opt/sarcheck/bin/analyze -hp or
/opt/sarcheck/bin/analyze -hm or
/opt/sarcheck/bin/analyze -hg
A FAQ section can also be found at the end of this manual, and updated information can be found on the SarCheck web site:
To get the most benefit out of SarCheck, we recommend using it as follows:
If SarCheck makes multiple recommendations that should all be implemented at once, it will explain that. An example of this would be changes to the lotsfree, desfree, and minfree parameters.
At the beginning of the analysis, the name of the sar report file, the date, time, number of intervals, number of processors seen, amount of memory, and system name is printed for identification purposes.
Important: When data from one system is analyzed on another, it is likely to result in incorrect or misleading recommendations. A warning message will appear if the name of the system on the sar report is different than the name of the system running SarCheck, or if the operating system version recorded by sar is different than the version reported by the uname command. This is because the values of tunable parameters, memory size, etc., are likely to be different on different systems.
Warning messages will appear if impossible data is seen in the sar report. Examples would be CPU utilization of 313% or a run queue occupancy of -88%. The type of sar data which contains the problem will be identified. SarCheck will still produce a report, but you should realize that the analysis of anomalous data is, as always, likely to follow the rule of "garbage in, garbage out".
The Summary section will highlight any bottlenecks that were seen in the areas of CPU, memory, or I/O, and will indicate if any kernel parameters need to be changed. If no bottlenecks are seen, the summary will say so, and point out that no recommendations will be made.
If runaway processes, memory leaks, or suspiciously large processes have been detected, a message will appear at the end of the Summary section.
The Recommendations section is present only if SarCheck has recommendations to make. If SarCheck thinks that everything is fine, no recommendations will be made. This is a normal condition and once the system is properly tuned, you should not be surprised to see a lack of recommendations.
The recommendations are based solely on the data contained in the sar file and the values of various tunable parameters, and should be taken in that context. For example, if batch jobs are run on Saturdays, and SarCheck analyzes statistics from that day, it may decide that an I/O bottleneck existed and spare memory was present, and therefore, an increase in buffer size may be appropriate. Following these recommendations may improve performance on Saturdays, but could hurt performance during the week by reducing the amount of memory available to users.
The changes to tunable parameters recommended by SarCheck are designed to cause slow, gradual improvement in order to prevent surprises. These gradual changes are designed to prevent any unanticipated side effects of a major change in a tunable parameter.
Due to the interrelationships between tunable parameters and system resources, sarcheck goes beyond the basic rules of thumb whenever possible.
The Resource Analysis section translates the data contained in the sar report into English. Much of this data is provided for reference, and explanations are given where appropriate. The implications of various statistics regarding CPU utilization, buffer sizing, memory utilization, system table sizes, and disk I/O bandwidth are presented in this section.
The times when key resources are most heavily used appear in this section. If these times correlate well with the times that performance degradation was reported, it can be inferred that exhaustion of these resources may be a cause of performance problems. Peak usage statistics are also used by the capacity planning section.
If the number of disk devices on your system results in an unwieldy number of paragraphs, using the analyze program's -html and -dtbl switches can improve readability when viewing the report through a browser.
If you use the -html switch and the -png, -jpeg, or -jpg switches, and you have all the prerequisites (such as a working copy of gnuplot), SarCheck will embed graphs in the HTML text and explain their relevance.
The Capacity Planning section can be used to approximate the amount of capacity left on the system, based solely on the sar data being analyzed. CPU, memory, disk, and system table use statistics are examined in order to determine which resource is likely to become exhausted first.
This section is not meant to perform the same functions as the more expensive tools available for large systems. It is designed to help meet the needs of system administrators, many of which are managing growing systems and need to know how much "room" is left before various resources become exhausted.
The exhaustion of resources is defined as any single interval in which CPU usage exceeded 90 percent, and disk was busy more than 75 percent of the time, swapping was detected, or the process table was more than 80 percent full. Because the interval with the greatest resource usage is used, the capacity planning report will be less accurate if peak resource use occurred during an interval of less than 10 minutes.
The Custom Settings section is where both successful and unsuccessful changes to SarCheck's default thresholds are reported. See the sections "How to change the menu defaults" and "How to change SarCheck's algorithms" for more information.
Disclaimers, trademark information, etc. At the end of the report is a disclaimer, trademark and copyright information, your software serial number, code version, licensee, and if applicable, the software's expiration date.
Use the -o option of /opt/sarcheck/bin/analyze to produce an order form and call us, or ask your reseller to call us. The cost of shipping SarCheck via US Mail is included in the price of SarCheck. If you'd like the software shipped via Federal Express, DHL, etc., please provide your account number and we will be happy to accommodate you.
In some parts of the world, local resellers may charge prices which are higher than our list price because they pay for the currency conversions, international shipping, duties, support, etc. We urge our customers to support their resellers.
Please read the FAQ section of this manual and visit the FAQ section of our website first. This will always be the fastest way to get the answer to a frequently asked question. If that doesn't do it:
Call us at +1-603-382-4200,
fax us at +1-603-382-4247,
write to us at PO Box 1033, Plaistow NH 03865, USA,
use our email address: support@sarcheck.com,
visit our web site at http://www.sarcheck.com/
or contact the party from whom you purchased SarCheck.
See the FAQ at http://www.ucc.ie/gnuplot/gnuplot-faq.html
This release contains the following files:
/opt/sarcheck/bin/analyze:This program performs the analysis.
/opt/sarcheck/bin/sarcheck:This is the front end for 'analyze'. It's a simple Bourne shell script which allows you to analyze the previous business day's sar data by pressing the enter key a few times. Feel free to customize this script to meet your needs, but save an unmodified copy in order to be safe.
/opt/sarcheck/etc/analyze.txt:This file contains the text used to produce the analysis. In general, we recommend that you do not modify this file, because it may leave us unable to support the software. Users outside of the United States may modify the spelling of certain words in the file if they wish. For example, the word 'utilization' can be changed to 'utilisation'. If you would like a non-English version of SarCheck, please call us.
/opt/sarcheck/etc/analyze.key:This file contains the activation key. Tampering with this file may permanently disable SarCheck.
/opt/sarcheck/bin/ps1:This is a script that collects data from ps -elf and from the dnlcmon program, and is roughly analogous to the sa1 script used by sar.
/opt/sarcheck/bin/ps2:This is a script that cleans up ps -elf data, and is roughly analogous to a subset of the sa2 script used by sar.
/opt/sarcheck/bin/dnlcmon:This program collects the values of various kstat & sysconf data.
/opt/sarcheck/doc/solman602.html:This manual.
/opt/sarcheck/etc/solsar22:A sample sar report. Recommendations produced by this test file should not be implemented, since the problems which generate the recommendations do not exist on your system.
/opt/sarcheck/ps/20070422:A sample ps -elf report collected with the .opt/sarcheck/bin/ps1 script. Recommendations produced by this test file should not be implemented, since the problems which generate the recommendations do not exist on your system.
/opt/sarcheck/bin/ondemand:This is a script which can be used to get recommendations that are almost "real time".If your system is slow and you want to collect and analyze data while the system is slow, this script will enable you to do it.
The following examples were produced with the -w option, used to suppress page breaks and newlines. This option sounds pretty odd, but it's really useful when exporting SarCheck reports to a Word Processing program. Please note that the text of the SarCheck report is printed in Courier font, and the explanation follows.
SarCheck(TM): Automated Analysis of Solaris sar and ps data
(English text version s6.01.00)
This is an analysis of the data contained in the file may. The data was collected from 05/03/2004 to 05/05/2004, from system 'drew'. There were 135 data records collected over 3 days used to produce this analysis. Operating system is Solaris 2.7. One processor is configured. 64 megabytes of memory are present.
Data collected by the ps -elf command during 3 days between 05/03/2004 and 05/05/2004 will also be analyzed. This program will attempt to match the starting and ending times of the ps -elf data with those of the sar report file named may.
DIAGNOSTIC MESSAGE: The number of disks in SarCheck's disk table is 2 and the table is 0.044 percent full. The number of entries in SarCheck's path_to_inst table is 2 and the table is 0.007 percent full. The number of entries in SarCheck's /dev/rdsk table is 2 and the table is 0.057 percent full. The number of entries in SarCheck's ps table is 541 and the table is 8.323 percent full.
Command line used to produce this report: analyze -t -diag -w -p -ps may
When the data was collected, no CPU bottleneck could be detected. No significant memory bottleneck was seen. No significant I/O bottleneck was seen. A change has been recommended to at least one tunable parameter. Limits to future growth have been noted in the Capacity Planning section.
At least one possible memory leak has been detected. At least one suspiciously large process has been detected. See the Resource Analysis Section for details.
Some of the defaults used by SarCheck's rules have been overridden using the sarcheck_parms file. See the Custom Settings section of the report for more information.
All recommendations contained in this report are based solely on the conditions which were present when the performance data was collected. It is possible that conditions which were not present at that time may cause some of these recommendations to result in worse performance. To minimize this risk, analyze data from several different days, implement only regularly occurring recommendations, and implement them one at a time.
Change the value of maxpgio from 60 to 65536. The reason for this significant change can be found in the Resource Analysis Section. This parameter can be changed by adding the following line to the /etc/system file: 'set maxpgio = 65536'. NOTE: Don't forget to check /etc/system first to see if there's already a set command modifying this tunable parameter. If there is, modify that command instead of adding another one.
Change the value of priority paging from 0 to 1. This parameter can be changed by adding the following line to the /etc/system file: 'set priority_paging = 1'. NOTE: Don't forget to check /etc/system first to see if there's already a set command modifying this tunable parameter. If there is, modify that command instead of adding another one.
Change the value of ncsize from 4236 to 6354. This change is recommended because the DNLC hit ratio, as calculated from kernel statistics, is 83.47 percent. Kernel statistics have been used instead of sar -a data because the values seen in the sar data were too low to calculate the DNLC hit ratio accurately. The low values seen in the sar -a data mean that the degree of improvement realized by implementing this recommendation is likely to be small. This parameter can be changed by adding the following line to the /etc/system file: 'set ncsize = 6354'. NOTE: Don't forget to check /etc/system first to see if there's already a set command modifying this tunable parameter. If there is, modify that command instead of adding another one. Once this change has been implemented, changes to the maxusers parameter will no longer affect ncsize.
Change the value of ufs_ninode from 4236 to 6354. This recommendation will increase the value of ufs_ninode to match the value of ncsize. Whenever a recommendation is made to increase ncsize, this program will also recommend increasing ufs_ninode to the same value. For more information, see page 309 of the second edition of Sun Performance and Tuning by Adrian Cockcroft. This parameter can be changed by adding the following line to the /etc/system file: 'set ufs_ninode = 6354'. NOTE: Don't forget to check /etc/system first to see if there's already a set command modifying this tunable parameter. If there is, modify that command instead of adding another one. Once this change has been implemented, changes to the maxusers parameter will no longer affect ufs_ninode.
More information on how to change tunable parameters is available in the System Administration Guide. We recommend making a copy of /etc/system before making changes, and understanding how to use boot -a in case your changes to /etc/system create an unbootable system.
Average CPU utilization was only 0.9 percent. This indicates that spare capacity exists within the CPU. If any performance problems were seen during the monitoring period, they were not caused by a lack of CPU power. User CPU as measured by the %usr column in the sar -u data averaged 0.75 percent and system CPU (%sys) averaged 0.17 percent. The sys/usr ratio averaged 0.23. CPU utilization peaked at 23 percent from 15:20:01 to 15:30:00 on 05/05/2004. A CPU upgrade is not recommended because the current CPU had significant unused capacity.
The CPU was waiting for I/O an average of 0.3 percent of the time. This confirms the lack of a regularly occurring I/O bottleneck.
The CPU was idle (neither busy nor waiting for I/O) and apparently had nothing to do an average of 98.8 percent of the time. If overall performance was good, this means that on average, the CPU was lightly loaded. If performance was generally unacceptable, the bottleneck may have been caused by remote file I/O which cannot be directly measured with sar and cannot be considered by SarCheck.
The run queue had an average depth of 1.4. This indicates that there was not likely to be a performance problem caused by processes waiting for the CPU. Average run queue depth (when occupied) peaked at 5.0 from 10:30:00 to 10:40:01 on 05/03/2004. During that interval, the queue was occupied 0 percent of the time.
The average cache hit ratio of logical reads was 98.3 percent, and the average cache hit ratio of logical writes was 80.6 percent. Despite the room for improvement seen in the hit ratios, no recommendation has been made because disk activity was light.
The pageout daemon did not use any CPU time.
The fsflush daemon used only 0.009 percent of the CPU. This indicates that it is probably not using enough of the CPU to cause a problem.
In the event of a system crash, an average of 32 seconds worth of data will be lost because it will not have been written to disk. This is controlled by the autoup and tune_t_fsflushr parameters. This statistic has been calculated using the formula: autoup + (tune_t_fsflushr / 2).
The ratio of exec to fork system calls was 1.02. This indicates that PATH variables are efficient.
DNLC statistics show that the average DNLC miss rate was 0.3 per second and the average hit rate was 6.5 per second. The percentage of hits during the monitoring period was 95.2. The peak DNLC miss rate was 2.8 per second from 08:10:00 to 08:20:01 on 05/05/2004.
SarCheck saw activity which resembled usage of the 'find' command. Data from those time periods has been ignored because it can result in misleading recommendations. The DNLC hit ratio was recalculated without those time periods and was approximately 95.23 percent. This hit ratio was high enough to indicate that the DNLC size should not be changed.
The value of maxuprc is 917 and the size of the process table as reported by sar was 922. There is no reason to change the value of maxuprc or max_nprocs based on this data.
The number of active inodes fit in the inode cache. This indicates that the inode cache was large enough to efficiently meet the needs of the system. Peak used/max statistics for the inode cache during the monitoring period were 2374/4236. The value of %ufs_ipf was always zero, indicating that there was no page flushing, and the size of the inode cache was large enough.
At least 1.9 percent of the system's memory, or 1.23 megabytes, was always unused during sar sampling. The value of cachefree was 228 pages, or 1.78 megabytes of memory and the value of lotsfree was 114 pages, or 0.89 megabytes of memory. This indicates that while the system is not in need of memory, there isn't an unusually large quantity of physical memory that remains unused. Please note that measurements of unused memory are not true high-water marks of memory usage and only reflect what was happening when sar sampled system activity.
The average page scanning rate was 2.79 per second. Page scanning peaked at 66.70 per second from 09:40:00 to 09:50:00 on 05/03/2004. The page daemon scanning rate does not show evidence of a memory bottleneck. Peak resource utilization statistics can be used to help understand performance problems. If performance was worse during the period of high scanning activity, then a performance bottleneck may be caused by the lack of memory available. The threshold at which page scanning is considered to be a problem has been calculated at 81.0 per second. This calculation is based on the values of handspreadpages and autoup, and is optimized for sar sampling rates of 10 - 60 minutes.
The values of slowscan and fastscan were 100 and 3649. No changes are recommended because no problems were seen.
The value of maxpgio, the parameter that limits the number of page-outs per second, is set to 60. Recent testing indicates that the value of maxpgio should be set so high that it is effectively eliminated. The recommendation to increase maxpgio to 65536 will prevent the page scanner from limiting the number of writes per second.
The average system-wide local I/O rate as measured by the r+w/s column in the sar -d data was 0.47 per second. This I/O rate peaked at 8 per second from 09:40:00 to 09:50:00 on 05/03/2004.
An entry in the sarcheck_parms file of CTD has been found and SarCheck will attempt to map the physical device name of disk devices (cxtxdx) from the driver binding name and instance number (sdx).
The device dad0 (c0t0d0) was busy an average of 0.5 percent of the time and had an average queue depth of 0.2 (when occupied). The average service time reported for this device and its accompanying disk subsystem was 14.2 milliseconds. This is relatively fast considering that queuing time is included in this statistic. Service time is the delay between the time a request was sent to a device and the time that the device signaled completion of the request.
During multiple time intervals on 05/05/2004 ps -elf data indicated that there were a peak of 62 processes present. This was the largest number of processes seen with ps -elf but it is not likely to be the absolute peak because the operating system does not store the true "high-water mark" for this statistic. There were an average of 58.6 processes present.
Unusually large process size seen in /usr/openwin/bin/Xsun, owned by root, pid 246. The size of this process was 3552 pages, or 27.750 megabytes of memory. On this system the page size is 8 kilobytes.
Unusually large process size seen in /opt/NSCPcom/.netscape.bin, owned by drw, pid 481. The size of this process was 2445 pages, or 19.102 megabytes of memory.
Unusually large process size seen in /opt/NSCPcom/.netscape.bin, owned by drw, pid 340. The size of this process was 2442 pages, or 19.078 megabytes of memory.
Unusually large process size seen in /opt/NSCPcom/.netscape.bin, owned by drw, pid 363. The size of this process was 3654 pages, or 28.547 megabytes of memory. Between 08:20:01 and 08:40:00, this process grew from 2416 to 3654 pages. Memory usage grew at an average rate of 3717.1 pages/hr during that interval.
No runaway processes, memory leaks, or suspiciously large processes were detected in the ps -elf data.
More information on performance analysis and tuning can be found in The System Administration Guide Volumes 1 & 2, and in Adrian Cockcroft's Sun Performance and Tuning.
The section is designed to provide the user with a rudimentary linear capacity planning model and should be used for rough approximations only. These estimates assume that an increase in workload will affect the usage of all resources equally. These estimates should be used on days when the load is heaviest to determine approximately how much spare capacity remains at peak times.
Based on the data available in this single sar report, the system should be able to support a moderate increase in workload at peak times, and memory is likely to be the first resource bottleneck. See the following paragraphs for additional information.
The CPU can support an increase in workload of at least 100 percent at peak times. Because some swap space was used and significant page scanning or swapping statistics were not seen, the amount of memory present can probably handle a moderate increase in workload. The busiest disk can support a workload increase of at least 100 percent at peak times. For more information on peak CPU and disk utilization, refer to the Resource Analysis section of this report.
The process table, measured by sar -v, can hold at least twice as many entries as were seen.
The default AVGWIO threshold was changed in the sarcheck_parms file from 15.0 to 4.0 percent. This value is likely to compromise the accuracy of the analysis.
The default BGCOLOR threshold was changed in the sarcheck_parms file from #FFFF99 to #000066.
The default TEXTCOLOR threshold was changed in the sarcheck_parms file from black to yellow.
An entry in the sarcheck_parms file of CTD has been found and SarCheck will attempt to map the physical device name of disk devices (cxtxdx) from the driver binding name and instance number (sdx).
Please note: In no event can Aptitune Corporation be held responsible for any damages, including incidental or consequent damages, in connection with or arising out of the use or inability to use this software. All trademarks belong to their respective owners.
This is beta quality software and is to be used only in conjunction with a beta test program. This software is likely to contain defects and its recommendations should be regarded skeptically. This software is provided for the exclusive use of: Your Company. This software expires on 07/10/2004 (mm/dd/yyyy). Code version: 6.01.00 for Solaris SPARC 64-bit. Serial number: 00001234.
(c) copyright 1994-2004 by Aptitune Corporation, Plaistow NH 03865, USA, All Rights Reserved. http://www.sarcheck.com/
Statistics for system: drew
Statistics collected from: 05/03/2004
Statistics collected until: 05/05/2004
Average CPU utilization: 0.9%
Peak CPU utilization: 23%
Average user CPU utilization: 0.7%
Average sys CPU utilization: 0.2%
Average waiting for I/O: 0.3%
Peak waiting for I/O: 7.0%
Average run queue depth: 1.4
Peak run queue depth: 5.0
Calculated DNLC hit ratio: 95.23%
Pct of phys memory unused: 1.9%
Average page scanning rate: 2.8 / sec
Peak page scanning rate: 66.7 / sec
Page scanning threshold: 81.0 / sec
Average cache read hit ratio: 98.3%
Average cache write hit ratio: 80.6%
Average systemwide I/O rate: 0.47
Peak systemwide I/O rate: 8.00
Disk device w/highest peak: dad0 (c0t0d0)
Avg pct busy for that disk: 0.5%
Peak pct busy for that disk: 9.0%
Avg number of processes seen by ps: 58.6
Max number of processes seen by ps: 62
Approx CPU capacity remaining: 100%+
Approx I/O bandwidth remaining: 100%+
Remaining process tbl capacity: 100%+
Can memory support add'l load: Moderate
Thanks for your interest and support!
Q. What versions of Solaris are supported by SarCheck?
A. When this paragraph was last updated, versions 2.5 through 10 were supported on SPARC systems and version 10 was supported on x86 systems.
Q. Should I implement recommendations which only show up occasionally?
A. Feel free to try, but implement the regularly occurring recommendations first, since those will address the most frequently occurring problems. If SarCheck occasionally recommends increasing the amount of memory, you should certainly try it. On systems with some extra memory, SarCheck will be able to make additional recommendations that it would not make on systems where memory is "tight".
Q. Every time I make changes based on SarCheck's recommendations, it makes more recommendations. Why doesn't it just figure out the correct values for all the parameters?
A. That's not how real performance tuning works. There are no "correct" values because tuning is a series of compromises between various system resources. Performance tuning involves a certain degree of trial and error, and gradual change is the only way to do it.
Q. When I try to run sarcheck, I get the message "sarcheck: not found". What's wrong?
A. Check the following:
Q. How long does it take SarCheck to perform the analysis?
A. About one second of CPU time is required to analyze a small sar report on an 85 MHz SPARC 5. Some additional time is required if the -ctd, -png, -jpg, or -jpeg switches are used, and the amount of extra time required depends mainly on the number of disks and the size of the /etc/path_to_inst file. For more information on path_to_inst, refer to the man page.
Even though SarCheck uses very little CPU time in most cases, you should realize that 1 second of CPU time can equal at least 30 seconds of elapsed time on a system which is very busy. Analysis can be scheduled in cron if you prefer to analyze sar reports during off-peak hours.
Q. Why do I get the message "WARNING: Bad sar -b data has been detected"?
A. This is apparently caused by a bug in sar, and is present in release 2.5.1 and possibly others. Check the sar -b statistics for negative numbers. Their presence is an indication of the sar bug.
Q. I added the SUNWaccu and SUNWaccr packages, but I can't find the sa1 and sa2 scripts in the crontab entries for sys. Now what?
A. Add these lines to the crontab entries for user sys:
0 * * * 0-6 /usr/lib/sa/sa1
20,40 8-17 * * 1-5 /usr/lib/sa/sa1
5 18 * * 1-5 /usr/lib/sa/sa2 -s 8:00 -e 18:01 -i 1200 -A.i
(this should be all on one line)
Q. Why do I get the message "sa[0-3][0-9]: No such file or directory"?
A. This message indicates that there are no sar data files for sarcheck to analyze. The most common cause of this message is that there's a problem with the way sar is set up, or possibly the kernel was built at least 7 days ago and the system has not been rebooted since then.
Q. Why do I get the message "sar[0-3][0-9]: No such file or directory"?
A. This message indicates that there are no sar report files for sarcheck to analyze. The most common cause of this message is that there's a problem with the way sar is set up, or possibly the system is shut down at night before the sa2 script can be run by cron.
Q. Why did SarCheck stop producing reports?
A. Run 'analyze' and look for the expiration date at the bottom of the usage text. If you've licensed SarCheck and the expiration date doesn't make sense to you, run 'analyze -s' and send us the output.
Q. Why do I get the message "/dev/ksyms is not a 32-bit kernel namelist"?
A. This message indicates that a 32-bit version of the /opt/sarcheck/bin/analyze program is trying to read a 64-bit kernel. Contact us for a 64-bit version of SarCheck.
Q. Why do I get the message "/opt/sarcheck/bin/analyze syntax error at line 1..."?
A. This message indicates that you are trying to run the 64-bit version of SarCheck on a 32-bit system. Contact us for a 32-bit version of SarCheck.
Q. Why do I get the message "sar: malloc failed Not enough space"?
A. This is a sar problem, not a SarCheck problem. The message usually means that cron is running the sa1 script twice instead of once. For example, if your crontab entries look like this:
0 * * * 0-6 /usr/lib/sa/sa1
0,20,40 8-17 * * 1-5 /usr/lib/sa/sa1
The sa1 script gets run twice at 8:00 on weekday mornings, and gets run twice every hour after that until 17:00. This example was taken from an actual support call, and the problem disappeared once the second line was changed to this:
20,40 8-17 * * 1-5 /usr/lib/sa/sa1
Q. How do I concatenate the following files in order to produce a multiday analysis? sar18 sar19 sar20
A. There are several ways. The easiest is:
cat /var/adm/sa/sar18 /var/adm/sa/sar19 /var/adm/sa/sar20 > /tmp/multisar
(all on one line)
Other ways are
cat `ls /var/adm/sa/sar?? | egrep '1(8|9)|20'` > /tmp/multisar
(all on one line)
or the somewhat easier to read:
cat `ls /var/adm/sa/sar?? | egrep '18|19|20'` > /tmp/multisar
(all on one line)
The question you'll probably ask is how to concatenate any sar report which is less than 10 (or some other number) days old. To do that:
cat `/usr/bin/find /var/adm/sa \( -name 'sar*' \) -mtime -10` > /tmp/multisar
(all on one line)
Q. How do I change the starting and ending times of the period being analyzed?
A. When running the sarcheck script and analyzing data, you can change the starting and ending times by overriding the defaults with the /opt/sarcheck/bin/sarcheck_parms file. For more information see "How to change the menu defaults"
When analyzing a sar report, you will have to change the starting and ending times used to produce the report.
Q. How do I collect data every 10 minutes from 08:00 to 18:00?
A. The crontab entries should look like this:
0 * * * 0-6 /usr/lib/sa/sa1
10,20,30,40,50 8-17 * * 1-5 /usr/lib/sa/sa1
5 18 * * 1-5 /usr/lib/sa/sa2 -s 8:00 -e 18:01 -i 600 -A.(this should be all on one line)
Q. Why do I get the gnuplot message "unknown or ambiguous terminal type" when I try to produce graphs?
A. The version of gnuplot that is installed does not recognize the PNG or JPEG terminal type. To find out what terminal types are supported by your version of gnuplot, run gnuplot and type set terminal at the prompt.
UNIX System V Performance Management. 1994. Englewood Cliffs, NJ.: PTR Prentice Hall. ISBN 0-13-106429-1.
Alomari, A. Oracle & UNIX Performance Tuning. 1997. Upper Saddle River, NJ.: PTR Prentice Hall. ISBN 0-13-849167-4.
Cervone, H. F., Solaris Performance Administration. 1998. New York, NY.: McGraw-Hill. ISBN 0-07-011768-3.
Cervone, H. F., Solaris 7 Performance Administration Tools. 2000. New York, NY.: McGraw-Hill. ISBN 0-07-212211-0.
Cockcroft, A. Sun Performance and Tuning: Java and the Internet. 1998. Englewood Cliffs, NJ.: SunSoft Press, PTR Prentice Hall. ISBN 0-13-095249-4.
Loukides, M. System Performance Tuning. 1991. Sebastopol, CA.: O'Reilly & Associates, Inc. ISBN 0-937175-60-9.
Majidimehr, A. Optimizing UNIX for Performance. 1996. Englewood Cliffs, NJ.: PTR Prentice Hall. ISBN 0-13-111551-0.
We'd like to thank the following people for their suggestions, ideas, support, and yes, even a few bug reports:
D. J. Blackwood, Bill Campbell, Adrian Cockcroft, William Drescher, Robert P. Fries, Steve Gardiner, Jeff Hyman, Berni Jubb, Peter Kettle, Bob Long, Nancy Lorenz, Bela Lubkin, Rich Marotta, Gene Martin, Tom Melvin, Jeff Moskow, Lee Penn, Tom Podnar, Jean-Pierre Radley, Charlie Russel, David Simons, the folks at Ready to Run Software, Inc., and a number of others.
Many of the keywords and the defaults can be found by looking at the questions that the sarcheck script asks. Here is a complete list:
| PAGER | The pager to be used to display the analysis on the screen. The default is more, but pg or less are common alternatives. |
| LPS | The command for printing the analysis. The default is lp -s. |
| PSELFDIR | The directory where SarCheck will look for the ps -elf data. The analyze program and the ps1 and ps2 scripts will use this new directory. WARNING! Please pick a directory that contains nothing but ps -elf data! The ps2 script will use the find command to remove any file in the specificed directory which is more than 14 days old. We have tried to limit the potential damage by adding the -name switch to the find command but you should still be very careful with this. |
| SCDIR | The directory where the analyze program resides. This can be changed from the default of /opt/sarcheck/bin. |
| ETCDIR | The location of the file analyze.dlr. This file will be used if we ever use resellers who want to offer their own support. There is no point in changing this parameter at this time. |
| ST | The starting time for the analysis. The default is 08:00 and this should be entered in 24 hour format. |
| EN | The ending time for the analysis. The default is 17:00 and this should be entered in 24 hour format. |
| DR | Whether to analyze sar data or a sar report. The default is 'd'. For a list of options, run the sarcheck script and see what options are on the screen when the keyword is DR. |
| OPT | How to format the report. The default is 'n'. |
| VERBOSE | Whether the output should be verbose or quiet. The default is 'v'. |
| PSELFOPT | How verbose the ps -elf output should be. This option is used primarily to increase SarCheck's sensitivity to problems in the ps -elf data. The default is 'n'. |
| DISKFLTR | Whether or not to filter the disk analysis. Filtering sar's disk data is useful on large system with many disks when you're not using the HTML disk table option. The default is 'y'. |
| TABULAR | Whether or not to print a tabular summary at the end of the report or print a tabular summary instead of the report. The default is 'n'. |
| OUTOPT | This option controls where the output of the sarcheck script should go. The default is '1'. |
| GNUPLOT | The version of gnuplot present on your system. The default value is 3.7. |
| GNUPLOTDIR | The directory in which you've installed gnuplot. |
| GRAPHDIR | The directory in which the graphs will be stored. |
| CTD | The presence of the CTD keyword will perform the same function as the -ctd switch. No value is needed after the CTD keyword. |
| HTMLGRAPHDIR | The directory referenced in the HTML <img> tag. |
| SARCHECKDIR | The directory where the SarCheck programs reside. This can be changed from the default directory of /opt/sarcheck/bin. If this is used, all other entries in the file named /opt/sarcheck/etc/sarcheck_parms will be ignored and the sarcheck_parms file in the specified directory will be used instead. Refer to the section 'How to move SarCheck to another directory'. |
| DMY | Change the default date format to dd/mm/yyyy |
| YMD | Change the default date format to yyyy/mm/dd |
| HSIZE | Change the default width of the graphs generated by gnuplot. If you want to see graphs that are wider than the ones produced by the default width of 0.7, this keyword can be used to produce wider graphs. |
SarCheck will allow you to change the thresholds used by SarCheck's tuning algorithms. These changes can be implemented using the /opt/sarcheck/etc/sarcheck_parms file.
Please note that the default values of SarCheck's thresholds have been established based on feedback from thousands of systems. These values should not be overridden without good reason. Here is a list of thresholds which can be overridden, and the meaning of each is described below. Values outside the allowed range will be ignored and values outside the expected range will generate warnings.
| Keyword | Allowed Range | Expected Range | Default |
| AVGCPU | 50 - 100 | 60 - 100 | 70 |
| MAXCPU | 50 - 100 | 60 - 100 | 70 |
| AVGWIO | 1 - 50 | 5 - 25 | 15 |
| AVGRQ | 1 - 50 | 2.5 - 50 | 3.0 |
| MAXRQ | 1 - 500 | 2.5 - 50 | 5.0 |
| AVGRC1 | 10 - 99 | 60 - 99 | 90 |
| AVGRC2 | 10 - 99 | 65 - 99 | 96 |
| AVGWC1 | 10 - 99 | 50 - 90 | 65 |
| AVGWC2 | 10 - 99 | 50 - 90 | 80 |
| AVGDSK | 10 - 100 | 25 - 75 | 50 |
| CAPCPU | 25 - 100 | 60 - 100 | 80 - 90* |
| CAPDSK | 10 - 100 | 25 - 90 | 75 |
| CAPTBL | 25 - 100 | 40 - 90 | 80 |
| CPULIM | 0.05 - 100 | 10 - 100 | 20 |
| MLRATE | 1+ | 100+ | 200 |
| LGPROC | 32+ | 4096+ | formula |
| DCALL | any | any | 10 |
| DCLP | any | any | 10 |
| DCML | any | any | 10 |
| DCRP | any | any | 10 |
| SYSUSR | 0 - 999 | 0 - 999 | 2.5 |
| * dependent on number of processors | |||
AVGCPU: When average CPU utilization exceeds this value, SarCheck considers the system to be busy enough to cause concern.
MAXCPU: When Peak CPU Utilization exceeds this value, SarCheck assumes that performance degradation is likely.
AVGWIO: When the average value of the sar -u %wio column exceeds this value, SarCheck looks for evidence to corroborate an I/O bottleneck.
AVGRQ: When the average length of the run queue exceeds this value, SarCheck considers it to be an indication of a CPU bottleneck.
MAXRQ: When the maximum length of the run queue exceeds this value, SarCheck assumes that performance degradation is likely.
AVGRC1: The lowest acceptable value for the buffer cache read hit ratio. Note that several other factors are used to evaluate buffer cache effectiveness.
AVGRC2: The lowest acceptable value for the buffer cache read hit ratio when I/O is heavy. Note that several other factors are used to evaluate buffer cache effectiveness.
AVGWC1: The lowest acceptable value for the buffer cache write hit ratio. Note that several other factors are used to evaluate buffer cache effectiveness.
AVGWC2: The lowest acceptable value for the buffer cache write hit ratio when I/O is heavy. Note that several other factors are used to evaluate buffer cache effectiveness.
AVGDSK: When the average value of the sar -d %busy column exceeds this value, SarCheck will consider the disk's activity to be sufficient to cause a performance bottleneck..
CAPCPU: The value used to calculate the increase in CPU load that the system can support at peak times.
CAPDSK: The value used to calculate the increase in I/O load on the busiest disk that the system can support at peak times.
CAPTBL: The value used to calculate how much additional load can be supported before the process and open file tables become full.
CPULIM: The threshold in computed CPU utilization SarCheck uses to decide if a runaway process has been detected in ps -elf data.
MLRATE: The threshold in pages of memory per hour used by SarCheck to decide if a memory leak has been detected in ps -elf data.
LGPROC: The minimum size of a process which SarCheck will report as being suspiciously large.
DCALL: Disable the feature which limits the number of suspiciously large processes, memory leaks, and runaway processes.
DCLP: Disable the feature which limits the number of suspiciously large processes that are reported or change the number being reported. Using the keyword DCLP without a second field will disable the limit. Using a second field (for example: DCLP 25) will change the limit to the value in the second field.
DCML: Disable the feature which limits the number of processes with memory leaks that are reported or change the number being reported. Using the keyword DCML without a second field will disable the limit. Using a second field (for example: DCML 25) will change the limit to the value in the second field.
DCRP: Disable the feature which limits the number of runaway processes that are reported or change the number being reported. Using the keyword DCRP without a second field will disable the limit. Using a second field (for example: DCRP 25) will change the limit to the value in the second field.
SYSUSR: The threshold used to decide if it's worth mentioning if there is an unusual amount of %sys activity relative to %usr activity. The default of 2.5 means that %sys activity needs to be at least 2.5 times greater than %usr activity for this to be reported.
It is possible to set these parameters to values which can make SarCheck's recommendations meaningless or incorrect. Please override the default values with care.
The sarcheck_parms file can also be used to change the defaults used to generate HTML output.
| Keyword | Allowed range | Default |
| BGCOLOR | Any valid color | #FFEE88 |
| TEXTCOLOR | Any valid color | black |
| REDCOLOR | Any valid color | #FF9999 |
| PINKCOLOR | Any valid color | #FFCC99 |
BGCOLOR: The background color specified in the bgcolor attribute of the HTML <BODY> tag.
TEXTCOLOR: The text color specified in the text attribute of the HTML <BODY> tag.
REDCOLOR: The background color specified in the bgcolor attribute of certain <td> tags. The color used to highlight the cells of an HTML table when the values exceed certain thresholds. The default color is a shade of red and this keyword exists to give you an option if you want to use the color red as the text or background color.
PINKCOLOR: The background color specified in the bgcolor attribute of certain <td> tags. The color used to highlight the cells of an HTML table when the values exceed certain thresholds. The default color is a shade of pink and this keyword exists to give you an option if you want to use the color pink as the text or background color.
| -c | Turn off the capacity planning section. |
| -csv | Produce output in comma separated value (CSV) format. If the -html switch is used in conjunction with the -csv switch, these statistics will be printed as two HTML tables. If the -html switch is not used, the -csv switch will cause a SarCheck report to be generated with CSV output of statistics only. Disk statistics will be generated as if the -dtbl switch was used, and a tabular summary will be generated as if the -t switch was used. Please note that the -csv switch puts parts of the SarCheck analysis into CSV format. The -gr switch is used to put the sar report and the tabular summary (see the -t switch) into CSV format. |
| -ctd | Display disk devices using the c0t0d0 naming convention as well as the sd334 convention used in the sar report. To turn the option on permanently, add the CTD keyword to the sarcheck_parms file. See the sections entitled "How to change the menu defaults" and "How to change SarCheck's algorithms" for more information. On systems with several thousand disk devices, the use of the -ctd switch may increase the CPU utilization required by SarCheck. In tests we have conducted, as much as 30 seconds of a single processor's time may be required for systems with several thousand disk devices. Print info on all disks if more than 12 disk drives were seen in sar. Because the SarCheck report will produce a paragraph on each disk, reports may get too verbose on systems with 30 or more disk devices. Without this option, SarCheck will "filter out" information on disks which are lightly used. |
| -dblp | Suppress warnings about suspiciously large database processes. |
| -dbml | Suppress warnings about possible memory leaks in database processes. |
| -dbrp | Suppress warnings about possible runaway database processes. |
| -dcall | Disable limiting the number of warnings about suspiciously large processes, possible memory leaks, and possible runaway processes. |
| -dclp | Disable limiting the number of warnings about suspiciously large processes. |
| -dcml | Disable limiting the number of warnings about possible memory leaks in processes. |
| -dcrp | Disable limiting the number of warnings about possible runaway processes. |
| -dnz | Suppress the reporting of disks with no activity. This option is most likely to be useful when SarCheck is used on systems with thousands of disk devices. In one case where data on all 2,505 disks were reported in an HTML report using both tables and text, the size of report approached one megabyte, The size of the report was reduced by 90 percent with this switch. |
| -dbusy | If the -dtbl switch is used, -dbusy will sort the disk information by average percent busy. |
| -dserv | If the -dtbl switch is used, -dserv will sort the disk information by average service time. |
| -dtbl | If the -html switch is used, -dtbl will produce a table of disk statistics instead of generating a paragraph on each disk. Cells in the table will be color coded to highlight interesting disk statistics. This option is recommended for large systems where 50 or more individual paragraphs on disk activity would be hard to comprehend. If the -html switch is not used, -dtbl will cause disk statistics to be output in a comma separated value (CSV) format. CSV output should generally be produced with the -csv switch, but it can be done by using -dtbl too. |
| -dtoo | If the -html switch is used, -dtoo will produce a table of disk statistics in addition to generating a paragraph on each disk. Cells in the table will be color coded to highlight interesting disk statistics and will link to the appropriate paragraph. |
| -diag | This option will add a paragraph to the report showing how full SarCheck's internal tables have become. If a table comes too close to becoming full, a message should appear in the SarCheck report asking you to send a copy of the report to support@sarcheck.com |
| -dmy | This switch causes the date format used in the SarCheck report to appear in the format dd/mm/yyyy. |
| -en | The latest time in the sar report when data will be analyzed. The time can take the form of 16, 16:00, or 16:00:00. |
| -g24 | This switch will change the appearance of multiday graphs. It changes the graph to be displayed with an X-axis of up to 24 hours and data from different days will be superimposed. This can help to spot activity that occurs at the same time each day. |
| -gonly | Produce graphs only. This switch should be used together with the -jpeg, -jpg, or -png switches. The names of the graphs produced will be sent to stdout and no report will be produced. |
| -gd | Change the directory in which SarCheck puts graphs created with gnuplot. SarCheck will still determine the filenames of the graphs and the purpose of this switch is to allow you to store them wherever you want. The graphs can take up a considerable amount of space, especially JPEG graphs. |
| -gr | Produce output which can be used by graphing tools. While the output is in comma separated value (CSV) format, this option is different from the -csv switch because it reformats the sar report instead of reformatting SarCheck's analysis. |
| -h | Displays brief instructions and shows all of the possible switches. |
| -hg | How to produce graphs using the -jpg, -jpeg, and -png switches. |
| -hgd | Change the directory where the graphs appear to be in the HTML output's <img> tags. |
| -hm | How to analyze multiple days of sar data. |
| -hp | How to analyze supplemental ps -elf data. |
| -html | Insert HTML tags in text for use by a browser. |
| -jpeg or -jpg | These switches will cause SarCheck to look for gnuplot and use it to produce graphs in JPEG format. The naming convention used by SarCheck will append either ".jpeg" or ".jpg" to the file name of the graph, depending on the switch you use. JPEG formatted graphs are larger and do not look as crisp as PNG graphs, but they are much more likely to display correctly with older browsers. |
| -k | Allows you to change the activation key and software expiration date. |
| -mdy | Force the default mm/dd/yyyy date format to be used if it's overridden by the use of a non-English text file. Non-English text files are under development and were not available at the time of this printing. |
| -nfs | This switch causes disks which are NFS mounted to be treated as local disks by the analysis algorithms. This is probably not a useful thing to do, and is provided for backward compatibility with older versions of SarCheck. Those versions did not ignore the statistics associated with NFS disks. |
| -noparms | Ignore the contents of the sarcheck_parms file when generating the report. |
| -o | Prints an order/registration form for those wishing to purchase a software license, or register their licensed software. |
| -p | Suppress page numbering & page breaks. This is especially useful when the output is piped to pg. |
| -ps | Incorporate the analysis of a single ps -elf file called /opt/sarcheck/ps/yyyymmdd where the date is extracted from the sar data. |
| -pf | Include analysis of a specified file containing ps -elf data. |
| -pd | Change the directory in which SarCheck expects to find ps -elf data. SarCheck will still determine the name of the ps -elf data file and the purpose of this switch is to allow you to store ps -elf data wherever you want. This data can take up a considerable amount of space. |
| -pv | Verbose analysis of ps -elf data, overridden by -Q and -q. |
| -plp | Suppress warnings about suspiciously large processes. |
| -pml | Suppress warnings about possible memory leaks. |
| -prp | Suppress warnings about possible runaway processes. |
| -png | This switch will cause SarCheck to look for gnuplot and use it to produce graphs in PNG format. The naming convention used by SarCheck will append ".png" to the file name of the graph. PNG formatted graphs are smaller and look cleaner than JPEG graphs, but may not display correctly with older browsers. |
| -ptbl | If the -html switch is used, -ptbl will produce a table of ps -elf statistics instead of generating a paragraph on each process whose resource utilization exceeds the threshold. Cells in the table will be color coded to highlight the interesting statistics. This option is recommended for systems where a large number of individual paragraphs would be hard to comprehend. If the -html switch is not used, -ptbl will cause ps -elf statistics to be output in a comma separated value (CSV) format. CSV output should generally be produced with the -csv switch, but it can be done by using -ptbl too. |
| -ptoo | If the -html switch is used, -ptoo will produce a table of ps -elf statistics in addition to generating a paragraph on each process whose resource utilization exceeds the threshold. Cells in the table will be color coded to highlight interesting statistics. If the -html switch is not used, -ptoo will cause ps -elf statistics to be output in a comma separated value (CSV) format. In addition, the -csv switch generates a paragraph on each process whose resource utilization exceeds the threshold. CSV output should generally be produced with the -csv switch, but it can also be done by using -ptoo. |
| -Q | Print a non-verbose (super-Quiet) analysis. This option automatically sets the -p option. |
| -q | Print a less verbose analysis. |
| -r | Print an analysis only if recommendations are made. |
| -ret0 | Force a return code of zero. The analyze program normally returns zero if no recommendations are made and one if it makes recommendations. This option exists because some scheduling tools report non-zero return codes as errors or exceptional conditions. |
| -s | Display all the information needed to activate SarCheck. |
| -st | The earliest time in the sar report when data will be analyzed. The time can take the form of 16, 16:00, or 16:00:00. |
| -summ | This option causes the report to stop printing after the summary has been produced. |
| -t | This option will produce a summary of interesting statistics in a tabular format. This output can be parsed with relative ease. If the -html switch is used, the statistics will be presented in an HTML table, and cells in the table will be color coded to highlight noteworthy statistics. This option works well with -dtbl. |
| -tonly | This option will produce nothing but a summary of interesting statistics in a tabular format. All recommendations, analysis, and other hopefully interesting text will vanish. If the -html switch is used, the statistics will be presented in an HTML table, and cells in the table will be color coded to highlight noteworthy statistics. |
| -w | Suppress newline characters, primarily for export to PC-based word processing programs. |
| -wide | Change the width of graphs generated by gnuplot from 0.7 to 1.3. To pick a different value or change the default, use the HSIZE keyword in the sarcheck_parms file. |
| -ymd | This switch causes the date format used in the SarCheck report to appear in the format yyyy/mm/dd. |