I have been having sporadic Internet connectivity issues lately and today noticed this happened several times Finally about 30 minutes ago I get texted from monitoring I have on my site notifying me my site is down, I discover I cannot contact my router and it’s literally powered off. I look over the router and the power switch is no longer depressed and will not stay depressed to keep the power on!
I had to use duct tape since electrical tape wasn’t quite adhesive enough and some folded paper to keep the power switch depressed with electrical tape keeping the edges down and smooth.
Then went on Internet and found this is a consistent issue with all ASUS routers. I have already had my share of issues from the “kernel: VFS: file-max Limit 22684 reached” error which causes me to have to occasionally reboot since that simple kernel parameter cannot be changed in ASUS’ firmware. Yes I know I can use “dd-wrt” or “tomatoe” open source router firmware but I have the VPN working nicely with this router.
Besides this recent power button issue and the kernel parameter issue I’m also not a huge fan of their interface, specifically the “Virtual Server/Port Forwarding” interface is clunky and a time sync! When I clone or switch web servers I like to just point to the new server in my environment. With ASUS I have to totally delete each prior entry and add a totally new entry for that port forwarding to the new system! Every other router interface you just change a few alphanumeric characters and apply and your done.
Well I moved my website over to a new OS Centos 7 from Fedora 21. Centos seems to be handing my enviornment better with the XFS filesystem than Fedora has been with the ext4 filesystem. I am also still making changes to the infrastructure and updated some features on the “House of Beats” (http://dhoytt.com/snake-ice-radio-blog/now-playing/) blog on this site.
Can’t wait to taste this home made concoction made from some mint plants running amok in the garden and some stevia plants I added for a bit of sweet. I didn’t place a lot of stevia in the mix but will add more later to individual cup of tea to sweeten it up a bit.
This is the tea brewing in the sun with some mint in the background I let grow a bit unchecked:
This is the tea done steeping in the sun all day (note tea basket with mint & stevia on bottom of container):
This shows a couple of patches of mint in my garden that is full of mint:
This shows a couple of patches of stevia in my garden:
As I was migrating data from one NAS to another this week my XenServer pool master had a memory failure while I was away at work. Well after isolating the DIMM with the issue I also had to go in and change the bios settings for the fans. I noticed that the IPMI log files had numerous references to the fans and noted not all internal fans were set at optimal server fan settings.
Once I had the XenServer back in the rack cabled up it I could not see it boot up and had to switch it to another port on the KVM to see it boot and make subsequent updates. I’ll have to test the KVM port later but I suspect all is okay since other ports on that KVM are fine and it’s connected with my other KVM that also works fine. The other issue that manifested itself was that the XenServer that had the memory issue could no longer see storage from the NAS system I was moving the data off of.
I ended up doing the following to get my storage seen on my XenServer host that had the memory error:
Switched XenServer pool masters
xe host-list <— Get UUID of =desired new XenServer pool master
Performed Full Copy of critical VM’s to other NAS storage through XenServer
Exported critical VM’s to workstation in OVF format for another bbackup
Brought up site on copy of VM on desired NAS storage
Updated FreeNAS NASto new version.
Detached storage from XenServer that had bad PBD plugins to the NAS I was moving data off for update.
Added detached storage successfully back into XenServer
I still want to reconfigure my NAS storage to make it more efficient and still have a lot of work to be done still configuring my applications that serve up my website. I just performed OS updates of the VM I have the site on and noticed Xen library updates which I hope helps in Fedora. I notice Centos behaves a little better in XenServer environment so I may switch this site to a Centos VM.
This week since Sunday evening I have been recovering my web site step by step and one of my NAS systems as well. This is why I have redundancy of systems and cloned systems. Still I really didn’t have my backups running the way I wanted or I would have been up right away.
I moved a few systems over each evening to my other NAS and made backups of databases and restored and copied files and made snapshots at every step so I would not have to start over. I was actually running my website on the cloned system so I revived the original system restored the MySQL backups from about a week ago and most everything was back. I still have a few plugins to install for the website and make a few tweaks. The music server being separate was up the whole time so music was still streaming.
As far as repairing the NAS the boot flash drive became corrupted. As long as I didn’t reboot it should have been fine and was since most of the executables were in memory at that point. I saved the configuration to another system, moved the virtual systems over to storage on my other NAS and reinstalled to another flash drive. I uploaded the saved configuration and everything looks great!
Well I finally updated my second NAS to FreeNAS 9.3 and it was not straightforward even though it has the exact same hardware build as the first NAS it crashed hard when I attempted the update and totally corrupted my FreeNAS install on that NAS! I performed the web update just like I did with the first NAS but when it rebooted and started applying the new update it said it failed and when I rebooted the USB drive I booted off had no bootstrap. Even installing back to 9.2.18 failed!
Well this was last Thursday on March 26th and so it took me until the next Wednesday, today April 1 to get it done since I had moved all my data carefully over to my other NAS during the days leading up to the upgrade attempt I was not rushed. I also had to take care of other things around the house like my yard. What I thought would be an hour update at the most turned into quite a bit more.
I started getting CPU errors plus some SCSI errors when my system tried to access my USB devices as well. The fact that I received USB errors was a very serious issue since I don’t have any internal DVD drives on these NAS systems but install off of USB thumb drives and USB DVD drives all gave errors when accessed. I could install FreeNAS from a SATA drive with the system covers open but I had to install to my USB drive for booting so that was still a no go. I figured I could update my motherboard bios and that may take care of the CPU errors but just in case I ordered a matched pair of Xeon E5345 Quad Core Processors so that when I was ready to work on the NAS they would be there in case the others were actually bad.
Well come today when I finally had some time to tackle this NAS issue I was able to get it going after installing the new Xeon E5345 CPU’s and updating the bios of my Intel S5000psl motherboard with “Intel’s Deployment Assistant” booting off of a temporarily connected internal SATA DVDRW drive I connected with the system chassis opened up. Turns out that I had to install the update since the old motherboard bios couldn’t handle the new CPU’s stepping code as well as taking care of the issue with the USB devices throwing errors. I had noted the stepping code of the new CPU’s and checked on the release notes and the new CPU stepping codes were included in the motherboard bios update.
With the new CPU’s and the motherboard updates the 9.3 FreeNAS update on my second NAS worked beautifully as I thought it would last week! I had good feelings right on initial boot off of the FreeNAS 9.3 image since when it had failed it would hang at the new FreeNAS grub screen and go no further. While following the error messages I had noticed the SCSI errors when the USB device “da0” was accessed and this no longer happened after the bios motherboard update.
I make a habit of saving my FreeNAS configuration even when doing minor updates so once I got my second NAS back into the rack and cabled up@ 9.3 FreeNAS version I logged into the web interface using the DHCP address it picked up and uploaded my old configuration and FreeNAS updated the old settings rebooting a couple of times and everything was back to normal. I verified from my XenServer shell I had the NIC’s cabled right by pinging the interfaces and then watching the storage become available in XenServer environment.
Now both NAS are @ FreeNAS 9.3 and I have my redundancy back! I also believe my old CPU’s are good but I had to test the new CPU’s to give EBay feedback plus didn’t want to switch back and forth so will repurpose the old CPU’s somehow.
Here are my babies at the bottom of my rack purring like normal. I will grab a couple of slim DVDRW devices for them for fall back devices for issues like my USB flaking on me though.
In the backyard I am pretty much finished with spring cleanup and initial planting and this past weekend was dedicated to cleanup in the backyard, the front yard I did some trimming and planted a few more flowers and put down some black bark.
In the backyard over the winter dirt and debris that catches in the pores of the aggregate concrete so I performed my annual pressure washing of the walkways. Every spring since 1997 (even at previous residence) I setup the garden haul in whatever dirt,mulch, compost then get the rototiller out and shape everything into furrows, sweep top dirt off walkways, pressure wash the rest of the dirt and then clean the pool of any dirt and debris that gets in there. It’s amazing how much dirt you track just on your feet going back and forth getting tools, going to the store getting beverages to drink etc..
I also planted some more vegetables and placed some liquid fertilizer from Miracle Grow on my drip system so I can get some auto fertilization action going for my backyard garden. In the front yard I sprinkled some rose 2-1 one fertilizer that takes care of the bugs that gather on the roses and feeds the roses as well. I planted some marigolds and petunias as well in the front and laid out some black bark that really offsets everything nicely. The black bark which I layered in generously should keep the moisture in longer as well during the drought conditions.
I also pulled out the hammocks and they are ready for spring, summer and fall duty! Took nice dip Sunday evening in the pool and ready to roll for spring and summer!
Front Yard Flower Beds Ready For Spring with Black Bark!
Backyard Vegetable Gardens Set to go! *Note Power Washed Concrete*
Hammocks on both sides of backyard out of garage and ready for relaxation!
The vegetable garden is officially kicked off now that I have turned over the soil with the rototiller which is always my official demarcation point for the start of my veggie garden! I was able to get all my vegetables that had maintained over the winter and replanted them this weekend except the mint plants. The mint plants grow so well and run so much I don’t think I will have any issues having a few replanted mint plants take off once back in the ground!
I replanted my collard greens, mustard greens, Swiss chard, and chives that had grown over the winter. I also ended up thinning out the chives and the Swiss chard which I should have done last year as well but didn’t.
I planted some new jalapeno peppers, green & red bell peppers, red leaf mustard greens, pear tomatoes, early bird tomatoes, beef eater tomatoes, Serrano peppers and that’s it so far. The grape vines are still hanging in there with the raspberry bushes ready to take over fertile ground if I don’t keep them in check. This year I may try to run the grape & raspberry vines over a trellis if I have time.
Below are this years before and after photos of the vegetable garden. I still have some clean up and more planting.
Look at all of the mint which overtook my garden.
These after pics were taken a bit after the sun had gone down which is when I finished. The replanted plants will perk up once watered and in the ground a few days.
Every year this time of the year I wage a battle against mint plants over running my garden. This year I have pledged to make better use of everything in my garden including these mint pants that will not go away. Before I would include them in some alcoholic beverages an that’s it Well this year I have started making tea with them and the tea is very tasty with a dash of sugar or better yet local honey. I even think its helping me with my allergies but we will wait for that definitive declaration!
Here is the runaway plants now that never die off over the winter they just don’t spread over night like they do in the warmer months like now You can also see below that my swiss chard, mustard and collard greens did very well over the winter. I will soon uproot and replant these plants after I turn the soil with a rototiller and start incorporating other plants into the garden this year
Above is the runaway mint plants with tea I brewed. The tea color is light but the flavor is very full and strong!
One bunch of mustard greens, swis chard, collards and mustard greens on the other side of the garden.
Updates of FreeNAS & XenServer to Current Versions Plus Disk Upgrades
Sounds like too much at one time but it was perfect since I moved all of my data to one NAS system. I have built duplicate NAS and Virtual systems using XenServer and FreeNAS. My main goal was to migrate out 2tb 3gbps drives for 3tb 6bps drives but since I had my environment in maintenance mode ready for updates I thought now would be a good time to also update FreeNAS and XenServer since I may not have this window again soon. Plus the release notes of the new versions of FreeNAS and XenServer pointed to items that would benefit my environment.
Introducing First New Drive into FreeNAS
I “Live Migrated” all of the VM’s in my XenServer environment to the XenServer SRs’ (Storage Repositories) pointing to my second NAS system so I could bring my first NAS system down and add my new drives into the first NAS. I then verified I could see the new drives in the bios of my Dell H310 SAS controller and made them non-RAID JBOD devices and rebooted back into FreeNAS.
At this point I just wanted to start the process of including the new drives into an existing Volume in FreeNAS and used information I found online in the FreeNAS guides to do that (http://doc.freenas.org/9.3/freenas_storage.html)
I started at section “8.1.11 Replacing Drives to Grow a ZFS Pool”.
I normally ssh into my FreeNAS system from my Fedora 20 workstation using a terminal or my Windows 8.1 workstation via putty.
Here are the steps I used:
#zpool status <pool-name>
#zpool get all <pool-name>
#zpool set autoexpand -on <pool-name>
In GUI-> Storage ->Volumes -> <Volume-Name> -> Volume Status (bottom Icon to far right) -> “Highlighted first drive I wanted replaced” -> Offline -> Replace <Select new drive)
– I then go and monitor resilvering time:
– #zpool status <pool-name>
After the resilvering of the first drive was completed and Volume was no longer in degraded state that comes with resilvering I decided to update FreeNAS from version FreeNAS-9.2.1.8-RELEASE-x64 (e625626 to FreeNAS-9.3-STABLE-201502162250.
Updating First FreeNAS System to 9.3
I simply went to http://www.freenas.org/download-freenas-release.html downloaded the GUI upgrade file onto my Fedora workstation. I then saved my FreeNAS config, then browsed to the downloaded GUI upgrade file and updated my FreeNAS system. Here are my steps all GUI.
System -> Settings -> Save Config > Saved to directory on Fedora Workstation
System -> Settings -> Advanced Firmware Update (follow steps)
A few reboots later and my first NAS has updated to FreeNAS 9.3.
Start Introduction of Second New Drive into FreeNAS
I did the second drive into FreeNAS the exact same way as the first:
#zpool status <pool-name>
#zpool get all <pool-name>
#zpool set autoexpand -on <pool-name>
In GUI-> Storage ->Volumes -> <Volume-Name> -> Volume Status (bottom Icon to far right) -> “Highlighted first drive I wanted replaced” -> Offline -> Replace <Select new drive)
– I then go and monitor resilvering time:
– #zpool status <pool-name>
This time however I decided to also update one of my XenServer hosts since I didn’t have any VM’s on it and shouldn’t affect the resilvering process since I had detached the iSCSI volume from Xenserver I was adding the new drives to already.
I decided to go with the “Rolling Pool Upgrade” through the GUI Wizard and see if it would let me do this without a commercial license, I figured at the very least I could see what the “Prechecks” caught.
Here I will jump ahead and warn you to download XenCenter 6.5 if you will use the Rolling Pool Upgrade method whether CLI or GUI you will lose connectivity to the servers that migrate to 6.5 as soon as they are updated. Since the Master of the pool is migrated first you lose connectivity to the rest of the XenServer systems as well. The latest XenCenter is also on the XenServer.org download page http://xenserver.org/open-source-virtualization-download.html.
Resuming chronologically… I began my rolling upgrade which is fairly straightforward. I decided to offer my media over HTTP since I had that up and running and open via firewall to all the necessary systems from my Fedora workstation.
In XenCenter GUI – > Tools -> Rolling Pool Upgrade -> Run Prechecks -> Http <Http://<ip.address-workstation>/xenserver-65 -> Checks OK -> Starts upgrade of Master Pool Server
The upgrade of XenServer master starts fine and after reboot of the successfully updated Master pool server I lost conductivity to the pool and my systems.
I quickly downloaded XenCenter by pointing my browser to my newly updated XenServer 6.5 hosts and downloaded XenCenter 6.5 for my Windows 8.1 workstation (one of the main reasons I keep a Windows workstation in the first place is so I can used XenCenter features) and tried to install and it would not install!
Updates Paused to Update XenCenter on Windows Workstation
Now you know one reason I said download XenCenter 6.5 first. I now have no visibility to XenServer except through OpenXenManager for Linux which I don’t often use and as far as I know will not help me upgrade through that GUI and if it does well I wasn’t going to see how it does at this moment. Had I installed XenCenter 6.5 I would have taken care of this ms-installer before trying the upgrades and been rolling right along.
After trying a few items on my Windows 8.1 workstation I could see something was wrong with the ms-installer and it was not installing anything including updates. The only other Windows system I have is 2012 R2 streaming my music using SAM Broadcaster software and this is where I installed XenCenter 6.5 until I could figure out what was wrong with my 8.1 workstation.
The issue with using my W2012 R2 Standard system is that it is on a KVM switch and the resolution for XenCenter was barely acceptable but it worked. The other issue is that my XenServer and FreeNAS servers are also on the same KVM switch so I would have to switch back and forth continuously to monitor reboots of the XenServer or FreeNAS system and then back to my 2012 R2 system that now has my XenCenter 6.5 installed.
Back to First XenServer Upgrade and to Second XenServer Upgrades
With XenCenter on my streaming media 2012 R2 server I can now see that my first XenServer hosts is upgraded but the second host is not. I confirm this by switching to the console of the system via my KVM as well.
I tested the newly updated 6.5 XenServer system and migrated some VM’s over started them and everything worked fine. I then decided it was time to start upgrade of the second XenServer system in my environment.
I started the second XenServer update with live test/development systems running so I could see if the Rolling Upgrade process would live migrate them. I live migrated my production VM’s on the already updated 6.5 XenServer system.
Restarting XenServer upgrade it started the same way and just picked the server that was still at 6.2 so process was the same.
In XenCenter GUI – > Tools -> Rolling Pool Upgrade -> Run Prechecks -> Http <Http://<ip.address-workstation>/xenserver-65 -> Checks OK -> Starts upgrade of Second Pool Server
This is where it shows XenServer running “host-evacuate” and migrated all the VM’s off of my Second pool server to the Master. Pretty cool stuff. Due to these live migrations though my Master Server began to run out of resources so I started shutting down unnecessary VM’s.
I noticed that the XenCenter GUI began to stall after it evacuated the disks and I switched over to my XenServer console via KVM and the system was hung so I did a reboot by holding down the power switch. Upon reboot it tried to restart the 6.5 install but couldn’t and came back up as a 6.2 system.
After validating that the system was up and undamaged at XenServer 6.2 I restarted the Rolling Upgrade through the 6.5 XenCenter GUI. This time I actively switched back and forth from the system with the XenCenter GUI to the XenServer host in time to see the XenServer host stall on a bios issue which I quickly remedied and rebooted. XenServer picked up the update on reboot an after a couple more reboots was upgraded successfully to XenServer 6.5 as verified on console of system and n the GUI.
I then started some systems up to Live Migrate them back to their home server to even out the VM distribution.
Back to Getting XenCenter Installed on Windows 8.1 Workstation
I finally was able to get XenCenter re-installed on Windows 8.1 after dozens of reboots and failed efforts in came down. to the following.
In Windows 8.1 – > File Manager -> Control Panel -> Troubleshooting -> System and Security -> Fix Problems with Windows Update, reboot. The trick is it’s Windows so you have to let it do it’s thing in the background and Windows will not let you in on what is going on. Once I was patient and let it do it’s thing it “Blue Screened” gave some useful information about “driver_power_state_failure” and that prompted me to unplug a USB SATA hard drive dock I had plugged into the system to look at some drives but was currently empty.
The Windows 8.1 workstation came back up I installed updates rebooted and then re-installed XenCenter updating it to 6.5 successfully!
Verify XenServer 6.5 Updated
Now with XenCenter properly on my Windows 8.1 workstation I was able to verify all the VM’s worked correctly. I then updated all the VM’s to XenTools 6.5 and balanced the VM’s on the respective XenServer hosts.
FreeNAS iSCSI to XenServer Storage Repository Issues
I was unable to get the former XenServer SR’s to repair and plug back into the XenServer environment. One of the reasons for updating to this version of FreeNAS was due to improvements and new features in iSCSI. I noticed that the NFS SR presented from FreeNAS upgraded NAS shows properly but the iSCSI targets will not connect in the XenServer environment.
I started looking over the iSCSI settings in FreeNAS and noticed that the “Initiators” group was gone! I compared with my other FreeNAS system I had not updated and added an initiators group.
I then had to point to the “Initiators” group in “Extents”. I also checked the “Xen initiator compat mode” option that is new in FreeNAS 9.3 on the “Extents” tab.
I still was unable to get XenServer to recognize the iSCSI targets from FreeNAS. I went to look on XenServer side of things and noticed something fantastic! The complete iSCSI target IQN was showing from FreeNAS in the General tab of the SR’s in XenServer from the updated FreeNAS 9.3 system! My non-updated FreeNAS SR’s just still had my description.
I had previously detached one SR in XenServer being presented from the FreeNAS system I updated to 9.3 so I verified the IQN on both sides and performed a reattach operation from XenServer and it worked!
I tried the repair operation with the other SR’s I did not detach prior to upgrading and had no success. I then started doing screen-shots and taking notes of the IQN’s from XenServer of the SR’s and from the targets on the FreeNAS side to prepare to “Detach” or “Forget”and reattach the SR’s within XenServer. The SR’s I am trying to get seen in XenServer again from the updated FreeNAS system have no detach options so I will have to use the “Forget” option and then try to bring these SR’s back.
Forget Bring Back FreeNAS Storage into XenServer
Using the “Forget” option and then using the “New SR” wizard in XenServer brought the iSCSI storage that formerly wouldn’t connect back into XenServer successfully!
This is what I did:
Took screen-shots of my SR settings in XenServer since they show IQN from FreeNAS
Verified the IQN and target information from FreeNAS
XenServer GUI-> Infrastructure > Right-Clicked -> SR -> Forget
XenServer GUI -> Storage -> New SR -> Software iSCSI -> Enter “Prior name from Notes/screen-shot” -> Ip.address of FreeNAS system (using default port 3260) -> Scan Target Host -> Target IQN -> Target Lun -> Reattach ( recognizes prior XenServer SR information and gives option format or reattach) – > Prior SR is back within XenServer.
That was my multiple upgrade adventure the past week which has culminated in more disk space an upgraded FreeNAS and XenServer environment. I still need to update my other FreeNAS system to the latest version after I migrate the data off that NAS on to the updated FreeNAS environment. I may also rename my updated FreeNAS volume.
