One of these days I realised that one of the servers I am managing is no longer accessible. The machine was completely unresponsive on every protocol. I had no physical access to it. Fortunately I was able to connect to another machine in the same subnet in the datacenter. From there I was able to connect to the defective machine's BMC using ipmitool.
Having access to ipmi granted me access to serial console and ability to power cycle the machine.
So, I had access to the server but I couldn't use KVM's virtual media as the newer browsers have no support for Java. So the only option I had was to PXE boot the machine. I have followed the instructions from
http://www.system-rescue-cd.org/manual/PXE_network_booting/. Subsequently I configured an NginX webserver to serve content on the local subnet.
The first attempts to boot the system failed because IPMI sol is, as expected, a serial console. I adjusted the configuration file to have a serial console at boot time and I was able to boot the machine .
From there rebuilding the disk array and restoring the image from a backup was straightforward.
However one of the most important failures for me was the inability of using iLO's remote console due to Java's deprecation of NPAPI in browsers. Currently there is almost no possibility to substitute this functionality with other tools and vendors do not rush with alternatives even for more modern machines. The basic IPMI/RBMC tools lack many features as virtual media or frame buffer emulation. With Java no longer supported in the browsers I won't be able to access the iLO interface on my home server which is truly a major setback.
The files andmodifications are here:
https://github.com/dvoina/pxe_rescue
