Using DFIR Techniques To Recover From Infrastructure Outages

From isc.sans.edu

Recently, I was involved in a network outage caused by a defective pfSense firewall appliance. Due to storage issues (with the onboard flash), the firewall did not boot anymore. This can be quickly solved from a hardware point of view because this firewall model has a slot to install an M2-compatible flash device and boot from it. But, there was a problem with the configuration. The last backup they had was pretty old, and they made a lot of changes. No debate about the fact that a robust backup process should have been implemented. Let’s focus on the challenge of recovering the last configuration from the firewall. Challenge accepted!

First, I booted the firewall on an emergency USB stick and serial console access. First tip: always keep your console cables and emergency boot devices in a safe place. Once on the firewall, I tried to access the last configuration (stored as a big XML file) without luck. It was impossible to mount the corrupted filesystem. Because the file system was too big, it was impossible to take an image and store it on a USB key. Let’s dump it through the network! I manually configured a NIC to connect to a server and used our best friend: netcat!

Read more…