Firstly, createrepo in CentOS 5.5 does not work with either the Fedora 13 distribution image, or the Fedora 13 Updates repo. I got round that by rebuilding createrepo from Fedora 13 on CentOS 5.5 which was in itself a major undertaking as it also involved rebuilding several other packages which are dependencies of createrepo (yum, yum-utils, python-urlgrabber, python-pycurl, curl, libssh2).

Once I'd got the F13 distro imported into cobbler, and mirrored the F13 Updates repo, I created the cobbler system entry and ran koan to kick off the PXE install. All seemed to work fine and the installation completed. However, when I tried to start the domu it wouldn't start. In /var/log/xen/xend.log I found this was the last line:

[2010-09-01 16:43:05 9717] DEBUG (XendBootloader:113) Launching bootloader as ['/usr/bin/pygrub', '--output=/var/run/xend/boot/xenbl.11641', '/dev/mapper/vg_guests-mock--disk0']

On a hunch, I modified the kickstart to explicitly format the /boot partition as ext3 rather than the default ext4 and re-tried the install process. Bingo! pygrub in xen 3.4.2 on CentOS 5.5 doesn't work with ext4.

You have to remove the old version first, and then install the new version. Oh, and you also need to stop the Dell services, restart ipmi, then restart the Dell services.

Something like this:

yum -y remove srvadmin-* \
  && rm -Rf /opt/dell \
  && yum -y install srvadmin-all dell_ft_install \
  && srvadmin-services.sh stop \
  && service ipmi restart \
  && srvadmin-services.sh start

http://mywiki.wooledge.org/BashPitfalls

(I ran into #20, in case you're wondering)

Then, out of the blue, our monitoring started throwing alerts on seemingly random servers. Our queues were building up – basically, database performance had dropped dramatically and our processing scripts couldn't stuff data into the DBs fast enough.

What could be causing it?

I took a quick glance at our monitoring and various statistics confirmed the problems. You can clearly see in the following graphs that CPU Wait and Load Average increased at around 14:30, and at the same time MySQL throughput and command counters dropped dramatically:

So, I've found the symptoms; what could be the cause?

Digging around in the system logs, I found the following lines (emphasis is mine):

Apr 26 12:55:31 b008 Server Administrator: Storage Service EventID: 2176  The controller battery Learn cycle has started.:  Battery 0 Controller 0
Apr 26 12:56:36 b008 Server Administrator: Storage Service EventID: 2266  Controller log file entry: Battery is discharging:  Battery 0 Controller 0
Apr 26 12:56:36 b008 Server Administrator: Storage Service EventID: 2248  The controller battery is executing a Learn cycle.:  Battery 0 Controller 0
Apr 26 14:21:52 b008 Server Administrator: Storage Service EventID: 2278  The controller battery charge level is below a normal threshold.:  Battery 0 Controller 0
Apr 26 14:21:53 b008 Server Administrator: Storage Service EventID: 2188  The controller write policy has been changed to Write Through.:  Battery 0 Controller 0
Apr 26 14:21:53 b008 Server Administrator: Storage Service EventID: 2199  The virtual disk cache policy has changed.:  Virtual Disk 0 (Virtual Disk 0) Controller 0 (PERC 6/i Adapter)
Apr 26 14:55:06 b008 Server Administrator: Storage Service EventID: 2177  The controller battery Learn cycle has completed.:  Battery 0 Controller 0
Apr 26 14:55:21 b008 Server Administrator: Storage Service EventID: 2247  The controller battery is charging.:  Battery 0 Controller 0
Apr 26 14:55:21 b008 Server Administrator: Storage Service EventID: 2278  The controller battery charge level is below a normal threshold.:  Battery 0 Controller 0
Apr 26 15:25:41 b008 Server Administrator: Storage Service EventID: 2279  The controller battery charge level is operating within normal limits:  Battery 0 Controller 0
Apr 26 15:25:41 b008 Server Administrator: Storage Service EventID: 2189  The controller write policy has been changed to Write Back.:  Battery 0 Controller 0
Apr 26 15:25:42 b008 Server Administrator: Storage Service EventID: 2199  The virtual disk cache policy has changed.:  Virtual Disk 0 (Virtual Disk 0) Controller 0 (PERC 6/i Adapter)
Apr 26 18:50:26 b008 Server Administrator: Storage Service EventID: 2358  The battery charge cycle is complete.:  Battery 0 Controller 0

Bingo!

The RAID controller is running a battery learn cycle. When battery charge drops below a certain level, cache Write Back is disabled, and that kills disk performance. It seems this test is enabled by default and is configured to run every 90 days. Of course, Sod's law dictates that it has to trigger in our busy period!

The Dell tools are not able to turn off this feature, but the LSI MegaCli tool can (Dell PERC 6/i controllers are re-badged LSI cards). I've run the following script on all servers (thanks to burr86 on ##infra-talk @ Freenode):

#!/bin/sh
TMPFILE=$(mktemp -p /tmp bbu.relearn.off.XXXXXXXXXX) || exit 1
echo "autoLearnMode=1" > $TMPFILE # or =0 to enable the bbu relearn
MegaCli -AdpBbuCmd -SetBbuProperties -f$TMPFILE -a0
rm -f $TMPFILE

I wrote a puppet class to run this script on all nodes:

class megacli::check {
    exec{'perc_bbu_autolearn.sh':
        require => Class['megacli::install'],
        cwd    => '/tmp',
        path   => '/usr/local/bin:/bin',
        unless => 'MegaCli -AdpBbuCmd -GetBbuProperties -a0 | grep -q "Auto-Learn Mode: Disabled"'
    }
}

omconfig storage battery action=startlearn controller=0 battery=0

Previously, I have been doing this manually:

1. log in to the DRAC web interface
2. launch the java console
3. rebooting the server
4. go into the BIOS
5. navigate to Embedded Devices
6. manually record the MAC addresses

This takes quite a while, and is prone to error.

I recently had another 42 servers to deploy to I looked for a way to automate this process. I found one!

I got the inkling that this should be possible because I noticed that the System | Properties | System Details page in the DRAC web interface lists the MAC addresses in the Main System Chassis section:

Embedded NIC MAC Addresses
    NIC1  Ethernet  a4:ba:db:11:38:2d
          iSCSI     00:00:00:00:00:00
    NIC2  Ethernet  a4:ba:db:11:38:2e
          iSCSI     00:00:00:00:00:00

This dumps a whole load of information about the DRAC and the attached system. However, pass the output through a simple grep and … bingo!

$ racadm -r $DRAC -u root -p calvin racdump | egrep '^MAC Address|^NIC. Ethernet'
MAC Address             = a4:ba:db:11:38:2f
NIC1 Ethernet           = a4:ba:db:11:38:2d
NIC2 Ethernet           = a4:ba:db:11:38:2e
NIC3 Ethernet           = N/A
NIC4 Ethernet           = N/A

toggl make available desktop clients for Windows, Mac, & Linux, but the Linux packages are in .deb format for Ubuntu and, until recently, they did not provide x86_64 packages.

toggl recently released the desktop client as open source so I grabbed it and have built an RPM.

SRPM: TogglDesktop-2.5.1-1.fc12.src.rpm

RPM (Fedora 12, x86_64): TogglDesktop-2.5.1-1.fc12.x86_64.rpm

[root@b034 ~]# omreport storage pdisk controller=0
Invalid controller value. Read, controller=0
No controllers found.

After a little investigation, I found the source of the problem.

It turns out that the i386 package of compat-libstdc++-33 is required, although it's not pulled in as a dependency by the srvadmin-whatever RPM.

yum install compat-libstdc++-33.i386
service dataeng restart

Now, the same command works just fine:

[root@b034 ~]# omreport storage pdisk controller=0
List of Physical Disks on Controller PERC 6/i Adapter (Slot 1)

Controller PERC 6/i Adapter (Slot 1)
ID                        : 0:0:0
Status                    : Ok
Name                      : Physical Disk 0:0:0
etc.

Here's how to build RPMs from Ruby gems using gem2rpm.

I am assuming you have the necessary build tools installed (if not, yum install rpmdevtools) and have already created an RPM build environment, eg:

~/rpmbuild
|-- BUILD
|-- RPMS
|-- SOURCES
|-- SPECS
`-- SRPMS

First, make sure gem2rpm is installed:

yum install rubygem-gem2rpm

Then, grab the gem you want to convert to an RPM:

cd ~/rpmbuild/SOURCES
gem fetch capistrano

This will dump the gem file in the current directory, in this case: capistrano-2.5.14.gem.

Next, create a spec file:

gem2rpm capistrano-2.5.14.gem  > ../SPECS/rubygem-capistrano.spec

Finally, build the RPM(s):

rpmbuild -ba ../SPECS/rubygem-capistrano.spec

I use the date command to produce timestamps in the appropriate format, but I find the format specifier a bit long-winded and difficult to remember – is %m minutes or month?

There is a better way… date -I

To make the strings sort in a sane manner, I find it best to use a format with the timestamp elements in descending order of size, eg. YYYYMMDDhhmmss which is produced with a format specifier like this:

# date +%Y%m%d%H%M%S
20100202130818

But as I've mentioned, I find that to be a pain to remember. Wouldn't it be great if there was a single switch that produced a suitable timestamp format?

Well, it turns out that there is:  --iso-8601 or -I, with the seconds modifier:

# date -Iseconds
2010-02-02T13:07:40+0000

According to NEWS in the coreutils distribution:

  date accepts the new option --rfc-3339=TIMESPEC.  The old --iso-8602 (-I)
  option is deprecated; it still works, but new applications should avoid it.
  date, du, ls, and pr's time formats now support new %:z, %::z, %:::z
  specifiers for numeric time zone offsets like -07:00, -07:00:00, and -07.

The –rfc-3339 option is similar, but it includes a space in the output, which I prefer to avoid:

# date --rfc-3339 seconds
2010-02-02 13:16:10+00:00

So, I'll be continuing to use -Iseconds

When I first wrote my cron classes, I created a separate class for each cron job, eg.

class app::queue_processor::cron::cacheload {

    include
        user::base

    cron { cacheload:
        command => '/usr/bin/php /app/queue_processor/utils/cacheload.php account > /dev/null 2>&1',
        hour    => '0',
        minute  => '0',
        user    => 'app',
        require => User['app']
    }

}

I then created another class for each profile which includes all the required cron jobs, eg:

class profile::partition::cron {
    include
        app::queue_processor::cron::cacheload
        app::queue_processor::cron::send_move_backlog
}

This all works fine and the cron jobs are installed as required. However, I recently needed to remove the cacheload cron job from all machines.

This is simply a matter of specifying "ensure => absent" to the cron type. I therefore needed some way of passing this to the cron job classes.

I did this by creating a higher-level cron class, app::queue_processor::cron containing a definition app::queue_processor::cron::job which takes an ensure parameter that defaults to "present" if not specified. It looks like this:

class app::queue_processor::cron {}
define app::queue_processor::cron::job($ensure = present) {

    include user::base

    case $name {
        cacheload: {
            cron { $name:
                command => '/usr/bin/php /app/queue_processor/utils/cacheload.php account > /dev/null 2>&1',
                hour    => '0',
                minute  => '0',
                user    => 'app',
                require => User['app']
                ensure  => $ensure
            }
        }

        send_move_backlog: {
            cron { $name:
                # similar cron definition here
            }
        }

        default: { fail("Unknown queue_processor cron job type: $name") }
    }
}

I use this new define in the profile::partition::cron class like this:

class profile::partition::cron {
    include app::queue_processor::cron
    app::queue_processor::cron::job{ send_move_backlog: }
    app::queue_processor::cron::job{ cacheload: ensure => absent }
}