It's almost 2019, so using a SATA SSD as the boot drive for your main development OS is not cool anymore… and I was running out of space on this 128gb one, so I bought an NVMe drive to replace it. Yay.

Because I don't have anything with two or more M.2 slots and I was too lazy to find/make a bootable FreeBSD USB drive, moving the system involved inserting the new drive into another machine (server) and using ZFS replication to copy the data. (And forgetting to set bootfs on the pool, of course.)

But the fun part was that my 10G network card stopped working. Moving the card into the middle slot (from the bottom one) fixed it. Reported a FreeBSD bug.

The weirdest discovery of the day though was that MSI mainboards persist the "above 4G" PCIe setting across CMOS clears. What in the actual heck. This is the setting that breaks display output on most GPUs (funnily enough, mine did display non-EFI things such as the network card's boot prompt and the glitchy way FreeBSD displays the console when booting on UEFI with the EFI framebuffer disabled). It's a setting you very much need to clear.

LSISAS

When flashing an OEM LSI SAS/SATA HBA/RAID card (Dell PERC H200, IBM ServeRAID M1015) into an LSI 9211-8i (IT mode), megarec / any DOS stuff is NOT required! If you get a "Failed to Validate Mfg" error when cross-flashing from EFI Shell, just use an older version (P14 or older I think, P5 works fine) of sas2flash.efi! (And maybe flash vendor firmware first? Not sure if required, probably not.)