One of the easiest ways to stay ahead of the curve on hard drive failures is to take advantage of SMART monitoring.
SMART, or Self-Monitoring, Analysis, and Reporting Technology, is a is a monitoring system for computer hard disks to detect and report on various indicators of reliability in the hope of anticipating failures.
Note that this guide will focus on Linux servers; for Windows users please download the SeaTools package from Seagate which will be good for 99 percent of the hard drives used at NanoServers.
You will need to have the smartmontools package installed on your server. Note that this is included in the base installation for most Linux distributions.
To begin, give the command smartctl -a /dev/hda, using the correct path to your disk, as root. If SMART is not enabled on the disk, you first must enable it with the -s on option.
The first part of the output (Listing 1) lists model/firmware information about the disk—this one is an IBM/Hitachi GXP-180 example. Smartmontools has a database of disk types. If your disk is in the database, it may be able to interpret the raw Attribute values correctly.
Listing 1. Output of smartctl -i /dev/hda
Device Model: IC35L120AVV207-0 Serial Number: VNVD02G4G3R72G Firmware Version: V24OA63A Device is: In smartctl database [for details use: -P show] ATA Version is: 6 ATA Standard is: ATA/ATAPI-6 T13 1410D revision 3a SMART support is: Available - device has SMART capability. SMART support is: Enabled
The second part of the output (Listing 2) shows the results of the health status inquiry. This is the one-line Executive Summary Report of disk health; the disk shown here has passed. If your disk health status is FAILING, back up your data immediately. The remainder of this section of the output provides information about the disks capabilities and the estimated time to perform short and long disk self-tests.
Listing 2. Output of smartctl -Hc /dev/hda
SMART overall-health self-assessment test result: PASSED
General SMART Values:
Off-line data collection status: (0x82) Offline data collection activity
was completed without error.
Auto Off-line Data Collection:
Enabled.
Self-test execution status: ( 0) The previous self-test routine
completed without error or no
self-test has ever been run.
Total time to complete off-line
data collection: (2855) seconds.
Offline data collection
capabilities: (0x1b) SMART execute Offline immediate.
Automatic timer ON/OFF support.
Suspend Offline collection upon new
command.
Offline surface scan supported.
Self-test supported.
No Conveyance Self-test supported.
No Selective Self-test supported.
SMART capabilities: (0x0003) Saves SMART data before entering
power-saving mode.
Supports SMART auto save timer.
Error logging capability: (0x01) Error logging supported.
General Purpose Logging supported.
Short self-test routine
recommended polling time: ( 1) minutes.
Extended self-test routine
recommended polling time: ( 48) minutes.
The third part of the output (Listing 3) lists the disks table of up to 30 Attributes (from a maximum set of 255). Remember that Attributes are no longer part of the ATA standard, but most manufacturers still support them. Although SFF-8035i doesnt define the meaning or interpretation of Attributes, many have a de facto standard interpretation. For example, this disks 13th Attribute (ID #194) tracks its internal temperature.
Listing 3. Output of smartctl -A /dev/hda
Vendor Specific SMART Attributes with Thresholds: ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE 1 Raw_Read_Error_Rate 0x000b 100 100 060 Pre-fail Always - 0 2 Throughput_Performance 0x0005 155 155 050 Pre-fail Offline - 225 3 Spin_Up_Time 0x0007 097 097 024 Pre-fail Always - 293 (Average 270) 4 Start_Stop_Count 0x0012 100 100 000 Old_age Always - 10 5 Reallocated_Sector_Ct 0x0033 100 100 005 Pre-fail Always - 0 7 Seek_Error_Rate 0x000b 100 100 067 Pre-fail Always - 0 8 Seek_Time_Performance 0x0005 125 125 020 Pre-fail Offline - 36 9 Power_On_Hours 0x0012 100 100 000 Old_age Always - 3548 10 Spin_Retry_Count 0x0013 100 100 060 Pre-fail Always - 0 12 Power_Cycle_Count 0x0032 100 100 000 Old_age Always - 10 192 Power-Off_Retract_Count 0x0032 100 100 050 Old_age Always - 158 193 Load_Cycle_Count 0x0012 100 100 050 Old_age Always - 158 194 Temperature_Celsius 0x0002 189 189 000 Old_age Always - 29 (Lifetime Min/Max 23/33) 196 Reallocated_Event_Count 0x0032 100 100 000 Old_age Always - 0 197 Current_Pending_Sector 0x0022 100 100 000 Old_age Always - 0 198 Offline_Uncorrectable 0x0008 100 100 000 Old_age Offline - 0 199 UDMA_CRC_Error_Count 0x000a 200 200 000 Old_age Always - 0
Studies have shown that lowering disk temperatures by as little as 5°C significantly reduces failure rates, though this is less of an issue for the latest generation of fluid-drive bearing drives. One of the simplest and least expensive steps you can take to ensure disk reliability is to add a cooling fan that blows cooling air directly onto or past the systems disks.
Each Attribute has a six-byte raw value (RAW_VALUE) and a one-byte normalized value (VALUE). In this case, the raw value stores three temperatures: the disks temperature in Celsius (29), plus its lifetime minimum (23) and maximum (33) values. The format of the raw data is vendor-specific and not specified by any standard. To track disk reliability, the disks firmware converts the raw value to a normalized value ranging from 1 to 253. If this normalized value is less than or equal to the threshold (THRESH), the Attribute is said to have failed, as indicated in the WHEN_FAILED column. The column is empty because none of these Attributes has failed. The lowest (WORST) normalized value also is shown; it is the smallest value attained since SMART was enabled on the disk. The TYPE of the Attribute indicates if Attribute failure means the device has reached the end of its design life (Old_age) or its an impending disk failure (Pre-fail). For example, disk spin-up time (ID #3) is a prefailure Attribute. If this (or any other prefail Attribute) fails, disk failure is predicted in less than 24 hours.
The names/meanings of Attributes and the interpretation of their raw values is not specified by any standard. Different manufacturers sometimes use the same Attribute ID for different purposes. For this reason, the interpretation of specific Attributes can be modified using the -v option to smartctl; please see the man page for details. For example, some disks use Attribute 9 to store the power-on time of the disk in minutes; the -v 9,minutes option to smartctl correctly modifies the Attributes interpretation. If your disk model is in the smartmontools database, these -v options are set automatically.
The next part of the smartctl -a output (Listing 4) is a log of the disk errors. This particular disk has been error-free, and the log is empty. Typically, one should worry only if disk errors start to appear in large numbers. An occasional transient error that does not recur usually is benign. The smartmontools Web page has a number of examples of smartctl -a output showing some illustrative error log entries. They are timestamped with the disks power-on lifetime in hours when the error occurred, and the individual ATA commands leading up to the error are timestamped with the time in milliseconds after the disk was powered on. This shows whether the errors are recent or old.
Listing 4. Output of smartctl -l error /dev/hda
SMART Error Log Version: 1 No Errors Logged
The final part of the smartctl output (Listing 5) is a report of the self-tests run on the disk. These show two types of self-tests, short and long. (ATA-6/7 disks also may have conveyance and selective self-tests.) These can be run with the commands smartctl -t short /dev/hda and smartctl -t long /dev/hda and do not corrupt data on the disk. Typically, short tests take only a minute or two to complete, and long tests take about an hour. These self-tests do not interfere with the normal functioning of the disk, so the commands may be used for mounted disks on a running system. On our computing cluster nodes, a long self-test is run with a cron job early every Sunday morning. The entries in Listing 5 all are self-tests that completed without errors; the LifeTime column shows the power-on age of the disk when the self-test was run. If a self-test finds an error, the Logical Block Address (LBA) shows where the error occurred on the disk. The Remaining column shows the percentage of the self-test remaining when the error was found. If you suspect that something is wrong with a disk, I strongly recommend running a long self-test to look for problems.