Last modified at 11/2/2013 11:44 AM by Koen Zomers
My dedicated low energy pfSense server is equipped with a SuperMicro X7SPA-HF-D525 motherboard. This board has the great IPMI functionality. This includes KVM over IP (iKVM) and Watchdog functionality. Since the current pfSense 2.0 RC1 software is not very stable, it comes in handy to make use of the watchdog functionality to let the hardware detect a crash or freeze and reboot automatically. Actually, strangely enough, I have experienced several times already that my highly instable pfSense system with at least 5 crashes or hangs a day has become totally stable with no crash or hang at all anymore after applying these steps. If you have this motherboard, I highly recommend applying these steps.

There are two methods to accomplish using the Watchdog functionality
  1. Using the ICHWD driver (this is described below)
  2. Using the FreeIPMI package for FreeBSD (this is described in this article)
Both produce the same results. pfSense forum user _igor_ reported that the first method did not work with his hardware because he had AHCI enabled on his harddisks and thus created the second method. The second method also allows reading the sensors of your motherboard like the temperatures and voltage readings and will also reboot the pfSense machine in the rare case where it would freeze before the ICHWD driver would be loaded. Whichever method you choose is up to you, they both work fine on my hardware.

To enable this functionality within pfSense using the ICHWD driver method, follow these steps:
  1. Specifically to the SuperMicro X7SPA-HF boards, make sure you DISABLE the watchdog in the BIOS to make this work correctly. You can find the watchdog setting under Advanced and another under Advanced\IPMI Configuration. Both need to be set to disabled as shown on the following screenprints:

    Watchdog-BIOS-Advanced.png

    Watchdog-BIOS-AdvancedIPMI.png
  2. Connect to the console of your pfSense box via SSH, serial connection, physically on the machine, KVM over IP or any other way that suits you. Once connected, you'll be shown the pfSense menu. Use option 8 to go to the shell.
  3. It is important to know if your pfSense installation is 32 bit / i386 or 64 bit / AMD64. To find out about this, you can run the command:

    uname -a

    It should output something similar to:

    FreeBSD <fqdn of your pfsense box> 8.1-RELEASE-p3 FreeBSD 8.1-RELEASE-p3 #1: Fri Apr 29 21:17:50 EDT 2011     root@FreeBSD_8.0_pfSense_2.0-AMD64.snaps.pfsense.org:/usr/obj.pfSense/usr/pfSensesrc/src/sys/pfSense_SMP.8  amd64

    From this output you can define that this would be the AMD64 thus 64 bit installation. If it shows i386 you're using the 32 bit installation.
  4. If you're using the 32 bit / i386 version of pfSense 2.0 RC1, download these two files:

    ichwd.ko
    ichwd.ko.symbols

    If you're using the 64 bit / AMD64 version of pfSense 2.0 RC1, download these two files instead:

    ichwd.ko
    ichwd.ko.symbols
  5. Next you should copy the two files downloaded at step 4 into /boot/kernel of your pfSense machine. You can do this either way you prefer. One of the options to accomplish this includes downloading them directly into your pfSense using fetch. To do this, first change the current working directory to /boot/kernel using the command:

    cd /boot/kernel

    Next, execute the following two fetch commands to download the files:

    For i386:
    fetch http://www.zomers.eu/knowledge/pfSense/Downloads/Watchdog/I386/ichwd.ko
    fetch http://www.zomers.eu/knowledge/pfSense/Downloads/Watchdog/I386/ichwd.ko.symbols

    For AMD64:
    fetch http://www.zomers.eu/knowledge/pfSense/Downloads/Watchdog/AMD64/ichwd.ko
    fetch http://www.zomers.eu/knowledge/pfSense/Downloads/Watchdog/AMD64/ichwd.ko.symbols
  6. Now execute the following command:

    kldload ichwd

    It should produce an output similar to the following:

    Watchdog-kldloadichwd.png
  7. Next, go to the /usr/local/etc/rc.d directory using the command:

    cd /usr/local/etc/rc.d
  8. Now create a new script to initiate the watchdog application at the startup of your pfSense machine. To do this, enter the following command:

    vi 10_watchdog.sh
  9. In the VI program press INSERT to start editing mode. Enter the following text:

    /etc/rc.d/watchdogd forcestart

    Next press ESCAPE and type :wq followed by ENTER
  10. Your script should now have been created. Assign it execute rights by executing the following command:

    chmod 755 10_watchdog.sh
  11. You're all set. Reboot your pfSense machine by executing the command:

    shutdown -r now
  12. At the end of the boot procedure, you should notice the watchdog to be started automatically by means of the script you created at step 8 and 9:

    Watchdog-BootupScriptExecuted.png
  13. The watchdog is now active and providing the IPMI chipset with a heartbeat once every 16 seconds as specified in the watchdog defaults since we did not override the defaults by providing startup parameters.
  14. To test if the watchdog is functioning correctly, connect to the console of your pfSense installation again like you did at step 2. Again, go to the shell by using option 8 in the pfSense menu. Once there, type the following command:

    killall -9 watchdogd

    This will stop the watchdog application and thus will stop providing the IPMI chipset with the heartbeats. Within 20 seconds (the 16 seconds default interval + some slack time), your system should will now automatically reboot.

More information on this topic can be found on the following pfSense forum topic:

http://forum.pfsense.org/index.php/topic,34056.0.html

Many thanks go out to the great community behind pfSense for being able to fix this issue. Specifically to stephenw10, wallabybob and Cino for providing me with the pointers to solve this.