How to fix pfSense 2.7.2 to 2.8.1 Upgrade Bricks Booting VMs
TLDR
Updating pfsense 2.7.2 to 2.8.1 via normal in-place upgrade renders VMs unbootable, with the last message on the console being mostly Dual Console: Video Primary, Serial Secondary. Turns out upgrade installs a new kernel, reboots and fails going on with installing a new userland (init, sh, etc). Booting a rescue ISO, mounting the broken system and resuming the upgrade from the rescue systems working userland fixes the issue.
Symptoms
I recently had lots of problems upgrading pfSense VMs (both on VMware ESX & Nutanix AHV, which is KVM-based) from pfSense CE 2.7.2 to 2.8.1. After the first reboot, which is always part of a major version upgrade, the systems got stuck at boot: The last message either was Dual Console: Video Primary, Serial Secondary or output related to installed disks and their geometry (e.g. da0: 16384MB (33554432 512 byte sectors)).
The upgrade from 2.7.2 to 2.8.1 runs fine on hardware appliances/ bare metal (e.g. PC Engines apu4d4), and a direct installation of 2.8.1 on both hypervisors also leads to a bootable system that runs just fine. So it seemed neither a general problem with the update script, nor an incompatibility of the FreeBSD 15 base with the virtualisation platforms.
Diagnosis
My first assumption was that the primary output device just got switched from video to serial, but since the virtual firewalls didn't answer to ping even 2-3 minutes after "power on", when they really should have booted even without me seeing it on the virtual display (HTML5 consoles), it was obvious that the boot process got stuck somewhere and I could not see where.
I first attached a serial port to a broken VM to watch the boot process over this virtual serial port, hoping that the output goes further there - but no luck with this. Last message on the serial console was identical to what I could see via the virtual display: Dual Console: Video Primary, Serial Secondary.
Booting the system definitely got stuck here.
So I booted my VM from a pfSense ISO and started a rescue shell. I mounted the broken pfSense installation on the virtual disk & checked /cf/conf/upgrade_log.txt. The transcript stopped right after kernel/pkg installation falsely declaring that "packages are up to date" without ever installing the ~ 230-package installation that includes pfSense-base:
[1/1] Upgrading pfSense-kernel-pfSense from 2.7.2 to 2.8.1...
[1/1] Extracting pfSense-kernel-pfSense-2.8.1: .......... done
===> Keeping a copy of current kernel in /boot/kernel.old
>>> Removing unnecessary packages...
Checking integrity... done (0 conflicting)
Nothing to do.
>>> Unlocking package pkg...
...
[1/1] Upgrading pkg from 1.20.8_3 to 1.21.3_8...
[1/1] Extracting pkg-1.21.3_8: .......... done
...
Checking integrity... done (0 conflicting)
Your packages are up to date. <-- should not be true; pfSense-base and ~230 others never installed
>>> Upgrading Boot code...
...
Done.
System is going to be upgraded. Rebooting in 10 seconds.
I checked if upgrade installed a newer kernel, kernel modules and bootloader and it did. The date was plausible for 2.8.1 and the checksum matched kernel from a known-good installation:
root@pfSense-installer:~ # ls -la /tmp/root/boot/kernel/kernel
-r--r--r-- 1 root wheel 45455664 Aug 28 2025 /tmp/root/boot/kernel/kernel
root@pfSense-installer:~ # sha256 /tmp/root/boot/kernel/kernel
SHA256 (/tmp/root/boot/kernel/kernel) = 75e23110669820ea527fbffec337660fec1fdb284dc01028d41e98a6a3a5c9f7
However, "/sbin/init" predated the kernel significantly:
root@pfSense-installer:~ # ls -la /tmp/root/sbin/init
-r-xr-xr-x 1 root wheel 873280 Dec 6 2023 /tmp/root/sbin/init
root@pfSense-installer:~ # sha256 /tmp/root/sbin/init
SHA256 (/tmp/root/sbin/init) = b3400185e6a158ac3a54aed8026f6590a1af4ff8034c1268ac8fbe44535ca860
Compared to a working 2.8.1 installation:
[2.8.1-RELEASE][root@pfSense.home.arpa]/root: ls -la /sbin/init
-r-xr-xr-x 1 root wheel 889040 Aug 28 2025 /sbin/init
[2.8.1-RELEASE][root@pfSense.home.arpa]/root: sha256 /sbin/init
SHA256 (/sbin/init) = 84c6caa58a719a3d1f4f0e1e27200823c051ffda26d0b1eae45b1daaf00ae64d
Same for "/bin/sh" and other userland programs.
Cause
I don't know. pfSense CE upgrade runs in two stages: Stage 1 upgrades the kernel, boot loader and pkg - stage 2, on the next boot, is supposed to install the remaining packages, including pfSense-base (containing things like /sbin/init, /bin/sh and the rest of the userland.
On the broken VMs, stage 2 obiously never ran through. The upgrade_log shows the transaction correctly planned and fetched, then jumps directly to the boot-code update. No pfSense-base install.. which results in a 2.8.1 kernel trying to boot a 2.7.2 /sbin/init.
The wrapper script (pfSense-upgrade) sometimes prints libbe_init("") failed before skipping the installation. FreeBSD bug 257492 notes the message is misleading and just means "bectl couldn't resolve the active boot environment". Netgate has the same symptoms filed as Redmine #15495, but closed unfixed for a lack of reproducibility.
My best guess here is a race condition that is more likely on virtualised storage. I've never run into these problems on any bare metal installation.
Fix
Boot the broken VM from the installer ISO into rescue shell, mount the ZFS boot environment, chroot in and resume the upgrade manually:
# import the ZFS pool
mkdir -p /tmp/root
zpool import -f -R /tmp/root pfSense
# mount the boot environment
bectl -r pfSense/ROOT mount default /tmp/root
# networking in the rescue shell
ifconfig vtnet0 <ip> netmask <netmask> up
route add default <gateway>
echo "nameserver 8.8.8.8" > /tmp/root/etc/resolv.conf
# chroot in (menu pops up, pick 8 for a bare shell)
chroot /tmp/root /bin/sh
mkdir -p /var/run
# clear locks, check what's outstanding
pkg-static unlock -a
pkg-static version -vRL= # pfSense-base, php82-* etc. should show as outdated
# resume the upgrade
pkg set -v 0 php82
pkg-static upgrade -y
# exit chroot, unmount, halt, detach ISO, boot from disk
This worked reliably for me on a handful of VMs now, both testing and in production.
Hope that helps!