I have the station and website back up after an extensive power outage here in part of Sacramento.

I had a plethora of patches to update on my XCP-ng systems and I ended up having to restart the toolstacks on my systems. I started on my slave server restarting my “XCP-ng toolstack” which caused issues as it could not contact my master in the pool due to them talking to different versions of the software packages.

Took me a while to see this was the issue because problems started when I updated the Rocky Linux 9.x system and rebooted it and it would not restart. The Rocky Linux 9.x system was of course running on my slave XCP-ng server. After a while I recalled I needed to restart the XCP-ng hosts server’s toolstack which doesn’t interrupt the VM servers running on it as that was the likely issue . When I restarted the toolstack then suddenly the slave wouldn’t come out of maintenance mode. Tried several toolstack restarts and restarting the XAPI service without luck.

I looked at some documentation and saw I needed to disable HA (High Availability) before restarting toolstack and that didn’t help. I decided to reboot but move over my VMS to the master but couldn’t due to the master being a lower version after my tootstak restart. I finally rebooted and had the same issues and took down some other VM’s I couldn’t control but were still up while the server was in maintenance mode.

After rebooting the slave XCP-ng remained in maintenance mode but with my VMs down of course. I had my ShoutCast and IceCast servers for https://dhoytt.com/snake-ice-radio-blog/now-playing/ on my XCP-ng slave host so no one was able to listen while I was doing this however my web site was till up as that VM system was on the master.

After finally looking through logs I saw that the slave was trying to communicate with the master since they were in the same resource pool. The logs showed the reason the slave would not fully boot was it could not communicate with the master. The networking was fine so I figured it was the version causing them to no communicate.

This is when I decided to gracefully shutdown the web server and other VM serves on the XCP-ng master just in case the server needed to be rebooted and not the toolstack just restarted.

Well the simple XCP-ng master toolstack restart allowed the slave to communicate to it and then come back up. The lesson I need to remember is to do all updates and restarts on the master first or switch over the slave to the master if I do it backwards which is too much unnecessary work.

Back and running with a lot of maintenance and updates needed in my server environment as I have been working tirelessly on getting the rental back up to speed and about 4 other projects that consumed my time this year.