Saving My TrueNAS Boot From SSD Failure

Sometimes worry that my central TrueNAS server is going to die one day and take all my precious data with it. It’s a completely irrational fear because in reality all my data lives on a mirrored ZFS pool and is also backed up to a dedicated internal drive.

However, this week I discovered I wasn’t being totally irrational… My boot pool drive was single disk vdev meaning it had ZERO fault tolerance! I checked the tiny 128GB SSD health with a S.M.A.R.T scan and found that the Media_Wearout_Indicator was sitting at 1/100, all of my infrastructure sitting on a tiny disk that was at the end of its life with no redundancy.

I immediately purchased two relatively cheap 256GB SSDs (link) so that I could kill two birds with one stone.

While losing the boot pool wouldn’t destroy any of my data, it would still mean rebuilding the entire operating system and custom configurations set for all of my services. Mirroring this pool into two new disks would make this recovery process far less painful.

Assumptions & Current Setup

• TrueNAS Scale (Community) operating system
• Existing pools:
• One spare hot-swap bay available
• Minimal downtime.
• No command line involved, only using TrueNAS web GUI
• Finally, if all goes wrong we still have our data separately managed by the other pools which we can (annoyingly) restore if we lose the boot-pool somehow.

Unboxing and installing

I’m going to be installing each disk 1 at a time, first being the easiest sliding into the front disk enclosure. Amazing I was able to replace the disk drive on this PC with a 5.25in cage that holds 2×2.5in and 1×3.5in slot, see here.

Please ignore the front IO shield that I’ve converted into another disk bay… I’m trying to cram as much as possible in here!

Initially nothing showed in dmesg logs and the disk wasn’t visible in the web UI. Opening the back of the bay revealed the issue:

It wasn’t even plugged in… oh well…
Now we can see the disk on the disks screen and it gives us the option to add it to the Boot Pool now. Annoyingly it’s also changed the assigned disk names (sda, sdb, etc.) as they’re dynamically assigned based on the detection order.

Notably we’re going to select “Use all disk space” because we’re upgrading from a 128GB to 256GB one. If this isn’t selected, the new disk won’t partition the whole space leaving 128GB unusable even when the old disk is unplugged.
After selecting “Save” it’ll format the disk and begin resilvering (copying the existing boot pool data to the new disk so the mirror can be created).

Once it was replaced I did a test boot of the system on my desk while it’s guts were still everywhere and that’s where I hit another small snag. Boots fine past UEFI, however upon loading into the GRUB menu I was greeted with this error:

I wish I’d taken a proper screenshot of the error but the above is just recreation from memory (and search history).

A quick lookup of the error shows it was something to do with my Secure Boot settings. Rebooting into BIOS I found it had flipped my Secure Boot OS settings from “Other OS” to “Windows UEFI mode” (damn Microsoft). Reverting this resolved the issue and it booted normally

After booting I find TrueNAS complaining about it’s missing half it’s boot pool! (Thankfully this is by design). I select the missing disk and choose “Replace”, where the same pop-up window previously shown for adding the disk comes up.

A few minutes later and we have a functional boot pool!

A successful reboot after these new disks have fully replaced the old guard means everything is working smoothly and I can stop worrying.

I know there’s a lot of screenshots but I actually wish I had more in some sections because this was super nerve racking, especially when it didn’t boot properly at the replacement stage.