Last modified at 11/2/2013 11:44 AM by Koen Zomers
My dedicated low energy pfSense server is equipped with a SuperMicro X7SPA-HF-D525 motherboard. This board has the great IPMI functionality. This includes KVM over IP (iKVM) and Watchdog functionality. Since the current pfSense 2.0 RC1 software is not very stable, it comes in handy to make use of the watchdog functionality to let the hardware detect a crash or freeze and reboot automatically. Actually, strangely enough, I have experienced several times already that my highly instable pfSense system with at least 5 crashes or hangs a day has become totally stable with no crash or hang at all anymore after applying these steps. If you have this motherboard, I highly recommend applying these steps.

There are two methods to accomplish using the Watchdog functionality:
  1. Using the ICHWD driver (this is described in this article)
  2. Using the FreeIPMI package for FreeBSD (this is described below)
Both produce the same results. pfSense forum user _igor_ reported that the first method did not work with his hardware because he had AHCI enabled on his harddisks and thus created the second method. The second method also allows reading the sensors of your motherboard like the temperatures and voltage readings and will also reboot the pfSense machine in the rare case where it would freeze before the ICHWD driver would be loaded. Whichever method you choose is up to you, they both work fine on my hardware.

To enable this functionality within pfSense using the FreeIPMI package method, follow these steps:
  1. Connect to the console of your pfSense box via SSH, serial connection, physically on the machine, KVM over IP or any other way that suits you. Once connected, you'll be shown the pfSense menu. Use option 8 to go to the shell.
  2. Install the FreeIPMI FreeBSD package by running the following command:

    pkg_add -r ftp://ftp.freebsd.org/pub/FreeBSD/ports/amd64/packages-8-stable/Latest/freeipmi.tbz

    This will download the package and its dependencies.

    pfsense-ipmi-pkg_add-download.png
  3. Next, go to the /usr/local/etc/rc.d directory using the command:

    cd /usr/local/etc/rc.d
  4. Now create a new script to initiate the watchdog application at the startup of your pfSense machine. To do this, enter the following command:

    vi 10_watchdog.sh
  5. In the VI program press INSERT to start editing mode. Enter the following text (if you have applied method 1 already, make sure you comment out the /etc/rc.d/watchdogd forcestart command by prepending it with # and adding the following command on a new line):

    /usr/local/sbin/bmc-watchdog -d -i 20 -e 16

    Next press ESCAPE and type :wq followed by ENTER
  6. Your script should now have been created. Assign it execute rights by executing the following command:

    chmod 755 10_watchdog.sh
  7. Next, you must enable the IPMI functionality on the startup of pfSense. You can do this by typing:

    vi /boot/loader.conf.local
  8. In the VI program press INSERT to start editing mode. Enter the following text (or append it if there's already text in it):

    ipmi_load="YES"

    Press ESCAPE and type :wq followed by ENTER
  9. You're all set. Make sure you can physically reach the machine since you'll need to enter the BIOS during the next step (so a SSH connection will not do, physical console connection, COM port connection or KVM over IP connection will do). Reboot your pfSense machine by executing the command:

    shutdown -r now
  10. Make sure you enter the BIOS. This is mostly done by hitting the DEL key on your keyboard while its showing the BIOS load screen, for example:

    pfSense-IPMI-SuperMicroX7SPA-BootScreen.png
  11. Once in the BIOS, go to Advanced and make sure that the Watch Dog fuction setting is set to Disabled.

    pfSense-IPMI-BiosWatchdogSetting2.png


    Next go to Advanced -> IPMI Configuration. Set the BMC Watch Dog Timer Action to Reset system and set BMC WatchDog TimeOut [Min:Sec] to 5 Min.

    pfSense-IPMI-BiosWatchdogSetting.png
  12. Press F10 to save the new settings and exit the BIOS. Let pfSense boot.
  13. At the end of the boot procedure, you should notice the watchdog to be started automatically by means of the script you created at step 4 and 5:

    Watchdog-BootupScriptExecuted.png
  14. The watchdog is now active and providing the IPMI chipset with a heartbeat once every 20 seconds as specified in the startup parameters entered in the boot startup script at step 5. To verify this, connect to your console again like you did at step 1 and as stated in step 1, use menu option 8 to go to the console prompt.
  15. At the console prompt type the following command:

    bmc-watchdog -g

    This should produce an output similar to:

    pfSense-IPMI-BmcWatchdogGoutput.png
  16. The Initial Countdown field shows us at what interval it checks if the system is still responsive. This corresponds to the value set at step 5. The field Current Countdown shows us how many seconds there are left before it will try to get another heartbeat again, in this case in 15 seconds. If it does not receive a heartbeat after this interval, it will automatically reboot your system. You can verify this, if you want, by entering the following command:

    killall -9 bmc-watchdog

    If you keep checking the bmc-watchdog -g command from step 15 after providing the killall, you will see it countdown and when it reaches 0, it will reboot the machine.
  17. Another nice feature this method brings is that you can read the sensors on your board. You can do this by typing the following command on the console:

    /usr/local/sbin/ipmi-sensors

    This will produce an output similar to:

    pfSense-IPMI-IPMISensorsCommand.png
    pfSense forum user Cino even managed to create a PHP page which allows you to show these values right from the webinterface of pfSense. To load this on your system, on the pfSense console, type this command:

    cd /usr/local/www

    Next type:

    fetch http://www.zomers.eu/knowledge/pfSense/Downloads/Watchdog/IPMI/diag_ipmi-sensors.php

    To add a link to the IPMI Sensors page to the pfSense menu, first backup your current config. To do this, log on to the pfSense webinterface. Via the Diagnostics menu go to Backup/Restore.

    pfSense-IPMI-BackupConfig.png

    On the Diagnostics: Backup/restore page, uncheck all checkboxes in the Backup configuration section and click on Download configuration.

    pfSense-IPMI-BackupConfig2.png

    Open the downloaded XML file in an editor (i.e. notepad or any other editor will do). Search for <installedpackages>. Under it, add the following XML snippet:

    <menu>
    <name>IPMI Sensors</name>
    <tooltiptext>IPMI Sensors</tooltiptext>
    <section>Diagnostics</section>
    <url>/diag_ipmi-sensors.php</url>
    </menu>


    Save the modified config file and close your editor. Now on the same Diagnostics: Backup/restore page on the pfSense webinterface, click Browse in the Restore configuration section and select the XML file you have just modified. Click the Restore configuration button to proceed. This will automatically reboot your pfSense machine.

    pfSense-IPMI-RestoreConfig.png

    Once pfSense is done rebooting, open your browser and go to the pfSense webinterface. Check the Diagnostics menu. It should now display an option IPMI Sensors.

    pfSense-IPMI-DiagnosticsMenuLink.png

    If you click on it, you will be shown the readings from the sensors on your motherboard:

    pfSense-IPMI-IPMISensorsWebbrowser.png
More information on this topic can be found on the following pfSense forum topic:

http://forum.pfsense.org/index.php/topic,34056.0.html

Many thanks and all credits for this method go out to pfSense forum user _igor_ for discovering this method, Cino for creating the PHP page showing the sensor readings and jimp for providing the help on creating the menu item to the IPMI sensors page.