Updates of FreeNAS & XenServer to Current Versions Plus Disk Upgrades
Sounds like too much at one time but it was perfect since I moved all of my data to one NAS system. I have built duplicate NAS and Virtual systems using XenServer and FreeNAS. My main goal was to migrate out 2tb 3gbps drives for 3tb 6bps drives but since I had my environment in maintenance mode ready for updates I thought now would be a good time to also update FreeNAS and XenServer since I may not have this window again soon. Plus the release notes of the new versions of FreeNAS and XenServer pointed to items that would benefit my environment.
Introducing First New Drive into FreeNAS
I “Live Migrated” all of the VM’s in my XenServer environment to the XenServer SRs’ (Storage Repositories) pointing to my second NAS system so I could bring my first NAS system down and add my new drives into the first NAS. I then verified I could see the new drives in the bios of my Dell H310 SAS controller and made them non-RAID JBOD devices and rebooted back into FreeNAS.
At this point I just wanted to start the process of including the new drives into an existing Volume in FreeNAS and used information I found online in the FreeNAS guides to do that (http://doc.freenas.org/9.3/freenas_storage.html)
I started at section “8.1.11 Replacing Drives to Grow a ZFS Pool”.
I normally ssh into my FreeNAS system from my Fedora 20 workstation using a terminal or my Windows 8.1 workstation via putty.
Here are the steps I used:
#zpool status <pool-name>
#zpool get all <pool-name>
#zpool set autoexpand -on <pool-name>
In GUI-> Storage ->Volumes -> <Volume-Name> -> Volume Status (bottom Icon to far right) -> “Highlighted first drive I wanted replaced” -> Offline -> Replace <Select new drive)
– I then go and monitor resilvering time:
– #zpool status <pool-name>
After the resilvering of the first drive was completed and Volume was no longer in degraded state that comes with resilvering I decided to update FreeNAS from version FreeNAS-9.2.1.8-RELEASE-x64 (e625626 to FreeNAS-9.3-STABLE-201502162250.
Updating First FreeNAS System to 9.3
I simply went to http://www.freenas.org/download-freenas-release.html downloaded the GUI upgrade file onto my Fedora workstation. I then saved my FreeNAS config, then browsed to the downloaded GUI upgrade file and updated my FreeNAS system. Here are my steps all GUI.
System -> Settings -> Save Config > Saved to directory on Fedora Workstation
System -> Settings -> Advanced Firmware Update (follow steps)
A few reboots later and my first NAS has updated to FreeNAS 9.3.
Start Introduction of Second New Drive into FreeNAS
I did the second drive into FreeNAS the exact same way as the first:
#zpool status <pool-name>
#zpool get all <pool-name>
#zpool set autoexpand -on <pool-name>
In GUI-> Storage ->Volumes -> <Volume-Name> -> Volume Status (bottom Icon to far right) -> “Highlighted first drive I wanted replaced” -> Offline -> Replace <Select new drive)
– I then go and monitor resilvering time:
– #zpool status <pool-name>
This time however I decided to also update one of my XenServer hosts since I didn’t have any VM’s on it and shouldn’t affect the resilvering process since I had detached the iSCSI volume from Xenserver I was adding the new drives to already.
Start of First XenServer Host Upgrade
I downloaded the XenServer ISO @ http://xenserver.org/overview-xenserver-open-source-virtualization/download.html. There they have a link to the documentation I could check out to see if there was anything special to take into account: http://xenserver.org/overview-xenserver-open-source-virtualization/download/24-product/creedence/157-documentation.html.
I decided to go with the “Rolling Pool Upgrade” through the GUI Wizard and see if it would let me do this without a commercial license, I figured at the very least I could see what the “Prechecks” caught.
Here I will jump ahead and warn you to download XenCenter 6.5 if you will use the Rolling Pool Upgrade method whether CLI or GUI you will lose connectivity to the servers that migrate to 6.5 as soon as they are updated. Since the Master of the pool is migrated first you lose connectivity to the rest of the XenServer systems as well. The latest XenCenter is also on the XenServer.org download page http://xenserver.org/open-source-virtualization-download.html.
Resuming chronologically… I began my rolling upgrade which is fairly straightforward. I decided to offer my media over HTTP since I had that up and running and open via firewall to all the necessary systems from my Fedora workstation.
#cp /home/<my-home>/Downloads/XenServer/XenServer-6.5.0-xenserver.org-install-cd.iso /var/www/html
#mkdir xenserver-65
#mount -o loop -t iso9660 XenServer-6.5.0-xenserver.org-install-cd.iso xenserver-65/
In XenCenter GUI – > Tools -> Rolling Pool Upgrade -> Run Prechecks -> Http <Http://<ip.address-workstation>/xenserver-65 -> Checks OK -> Starts upgrade of Master Pool Server
The upgrade of XenServer master starts fine and after reboot of the successfully updated Master pool server I lost conductivity to the pool and my systems.
I quickly downloaded XenCenter by pointing my browser to my newly updated XenServer 6.5 hosts and downloaded XenCenter 6.5 for my Windows 8.1 workstation (one of the main reasons I keep a Windows workstation in the first place is so I can used XenCenter features) and tried to install and it would not install!
Updates Paused to Update XenCenter on Windows Workstation
Now you know one reason I said download XenCenter 6.5 first. I now have no visibility to XenServer except through OpenXenManager for Linux which I don’t often use and as far as I know will not help me upgrade through that GUI and if it does well I wasn’t going to see how it does at this moment. Had I installed XenCenter 6.5 I would have taken care of this ms-installer before trying the upgrades and been rolling right along.
After trying a few items on my Windows 8.1 workstation I could see something was wrong with the ms-installer and it was not installing anything including updates. The only other Windows system I have is 2012 R2 streaming my music using SAM Broadcaster software and this is where I installed XenCenter 6.5 until I could figure out what was wrong with my 8.1 workstation.
The issue with using my W2012 R2 Standard system is that it is on a KVM switch and the resolution for XenCenter was barely acceptable but it worked. The other issue is that my XenServer and FreeNAS servers are also on the same KVM switch so I would have to switch back and forth continuously to monitor reboots of the XenServer or FreeNAS system and then back to my 2012 R2 system that now has my XenCenter 6.5 installed.
Back to First XenServer Upgrade and to Second XenServer Upgrades
With XenCenter on my streaming media 2012 R2 server I can now see that my first XenServer hosts is upgraded but the second host is not. I confirm this by switching to the console of the system via my KVM as well.
I tested the newly updated 6.5 XenServer system and migrated some VM’s over started them and everything worked fine. I then decided it was time to start upgrade of the second XenServer system in my environment.
I started the second XenServer update with live test/development systems running so I could see if the Rolling Upgrade process would live migrate them. I live migrated my production VM’s on the already updated 6.5 XenServer system.
Restarting XenServer upgrade it started the same way and just picked the server that was still at 6.2 so process was the same.
In XenCenter GUI – > Tools -> Rolling Pool Upgrade -> Run Prechecks -> Http <Http://<ip.address-workstation>/xenserver-65 -> Checks OK -> Starts upgrade of Second Pool Server
This is where it shows XenServer running “host-evacuate” and migrated all the VM’s off of my Second pool server to the Master. Pretty cool stuff. Due to these live migrations though my Master Server began to run out of resources so I started shutting down unnecessary VM’s.
I noticed that the XenCenter GUI began to stall after it evacuated the disks and I switched over to my XenServer console via KVM and the system was hung so I did a reboot by holding down the power switch. Upon reboot it tried to restart the 6.5 install but couldn’t and came back up as a 6.2 system.
After validating that the system was up and undamaged at XenServer 6.2 I restarted the Rolling Upgrade through the 6.5 XenCenter GUI. This time I actively switched back and forth from the system with the XenCenter GUI to the XenServer host in time to see the XenServer host stall on a bios issue which I quickly remedied and rebooted. XenServer picked up the update on reboot an after a couple more reboots was upgraded successfully to XenServer 6.5 as verified on console of system and n the GUI.
I then started some systems up to Live Migrate them back to their home server to even out the VM distribution.
Back to Getting XenCenter Installed on Windows 8.1 Workstation
I finally was able to get XenCenter re-installed on Windows 8.1 after dozens of reboots and failed efforts in came down. to the following.
In Windows 8.1 – > File Manager -> Control Panel -> Troubleshooting -> System and Security -> Fix Problems with Windows Update, reboot. The trick is it’s Windows so you have to let it do it’s thing in the background and Windows will not let you in on what is going on. Once I was patient and let it do it’s thing it “Blue Screened” gave some useful information about “driver_power_state_failure” and that prompted me to unplug a USB SATA hard drive dock I had plugged into the system to look at some drives but was currently empty.
The Windows 8.1 workstation came back up I installed updates rebooted and then re-installed XenCenter updating it to 6.5 successfully!
Verify XenServer 6.5 Updated
Now with XenCenter properly on my Windows 8.1 workstation I was able to verify all the VM’s worked correctly. I then updated all the VM’s to XenTools 6.5 and balanced the VM’s on the respective XenServer hosts.
FreeNAS iSCSI to XenServer Storage Repository Issues
I was unable to get the former XenServer SR’s to repair and plug back into the XenServer environment. One of the reasons for updating to this version of FreeNAS was due to improvements and new features in iSCSI. I noticed that the NFS SR presented from FreeNAS upgraded NAS shows properly but the iSCSI targets will not connect in the XenServer environment.
I started looking over the iSCSI settings in FreeNAS and noticed that the “Initiators” group was gone! I compared with my other FreeNAS system I had not updated and added an initiators group.
I then had to point to the “Initiators” group in “Extents”. I also checked the “Xen initiator compat mode” option that is new in FreeNAS 9.3 on the “Extents” tab.
I still was unable to get XenServer to recognize the iSCSI targets from FreeNAS. I went to look on XenServer side of things and noticed something fantastic! The complete iSCSI target IQN was showing from FreeNAS in the General tab of the SR’s in XenServer from the updated FreeNAS 9.3 system! My non-updated FreeNAS SR’s just still had my description.
I had previously detached one SR in XenServer being presented from the FreeNAS system I updated to 9.3 so I verified the IQN on both sides and performed a reattach operation from XenServer and it worked!
I tried the repair operation with the other SR’s I did not detach prior to upgrading and had no success. I then started doing screen-shots and taking notes of the IQN’s from XenServer of the SR’s and from the targets on the FreeNAS side to prepare to “Detach” or “Forget”and reattach the SR’s within XenServer. The SR’s I am trying to get seen in XenServer again from the updated FreeNAS system have no detach options so I will have to use the “Forget” option and then try to bring these SR’s back.
Forget Bring Back FreeNAS Storage into XenServer
Using the “Forget” option and then using the “New SR” wizard in XenServer brought the iSCSI storage that formerly wouldn’t connect back into XenServer successfully!
This is what I did:
-
Took screen-shots of my SR settings in XenServer since they show IQN from FreeNAS
-
Verified the IQN and target information from FreeNAS
-
XenServer GUI-> Infrastructure > Right-Clicked -> SR -> Forget
-
XenServer GUI -> Storage -> New SR -> Software iSCSI -> Enter “Prior name from Notes/screen-shot” -> Ip.address of FreeNAS system (using default port 3260) -> Scan Target Host -> Target IQN -> Target Lun -> Reattach ( recognizes prior XenServer SR information and gives option format or reattach) – > Prior SR is back within XenServer.
That was my multiple upgrade adventure the past week which has culminated in more disk space an upgraded FreeNAS and XenServer environment. I still need to update my other FreeNAS system to the latest version after I migrate the data off that NAS on to the updated FreeNAS environment. I may also rename my updated FreeNAS volume.
