As I was migrating data from one NAS to another this week my XenServer pool master had a memory failure while I was away at work. Well after isolating the DIMM with the issue I also had to go in and change the bios settings for the fans. I noticed that the IPMI log files had numerous references to the fans and noted not all internal fans were set at optimal server fan settings.
Once I had the XenServer back in the rack cabled up it I could not see it boot up and had to switch it to another port on the KVM to see it boot and make subsequent updates. I’ll have to test the KVM port later but I suspect all is okay since other ports on that KVM are fine and it’s connected with my other KVM that also works fine. The other issue that manifested itself was that the XenServer that had the memory issue could no longer see storage from the NAS system I was moving the data off of.
I ended up doing the following to get my storage seen on my XenServer host that had the memory error:
Switched XenServer pool masters
xe host-list <— Get UUID of =desired new XenServer pool master
Performed Full Copy of critical VM’s to other NAS storage through XenServer
Exported critical VM’s to workstation in OVF format for another bbackup
Brought up site on copy of VM on desired NAS storage
Updated FreeNAS NASto new version.
Detached storage from XenServer that had bad PBD plugins to the NAS I was moving data off for update.
Added detached storage successfully back into XenServer
I still want to reconfigure my NAS storage to make it more efficient and still have a lot of work to be done still configuring my applications that serve up my website. I just performed OS updates of the VM I have the site on and noticed Xen library updates which I hope helps in Fedora. I notice Centos behaves a little better in XenServer environment so I may switch this site to a Centos VM.
This week since Sunday evening I have been recovering my web site step by step and one of my NAS systems as well. This is why I have redundancy of systems and cloned systems. Still I really didn’t have my backups running the way I wanted or I would have been up right away.
I moved a few systems over each evening to my other NAS and made backups of databases and restored and copied files and made snapshots at every step so I would not have to start over. I was actually running my website on the cloned system so I revived the original system restored the MySQL backups from about a week ago and most everything was back. I still have a few plugins to install for the website and make a few tweaks. The music server being separate was up the whole time so music was still streaming.
As far as repairing the NAS the boot flash drive became corrupted. As long as I didn’t reboot it should have been fine and was since most of the executables were in memory at that point. I saved the configuration to another system, moved the virtual systems over to storage on my other NAS and reinstalled to another flash drive. I uploaded the saved configuration and everything looks great!
Well I finally updated my second NAS to FreeNAS 9.3 and it was not straightforward even though it has the exact same hardware build as the first NAS it crashed hard when I attempted the update and totally corrupted my FreeNAS install on that NAS! I performed the web update just like I did with the first NAS but when it rebooted and started applying the new update it said it failed and when I rebooted the USB drive I booted off had no bootstrap. Even installing back to 9.2.18 failed!
Well this was last Thursday on March 26th and so it took me until the next Wednesday, today April 1 to get it done since I had moved all my data carefully over to my other NAS during the days leading up to the upgrade attempt I was not rushed. I also had to take care of other things around the house like my yard. What I thought would be an hour update at the most turned into quite a bit more.
I started getting CPU errors plus some SCSI errors when my system tried to access my USB devices as well. The fact that I received USB errors was a very serious issue since I don’t have any internal DVD drives on these NAS systems but install off of USB thumb drives and USB DVD drives all gave errors when accessed. I could install FreeNAS from a SATA drive with the system covers open but I had to install to my USB drive for booting so that was still a no go. I figured I could update my motherboard bios and that may take care of the CPU errors but just in case I ordered a matched pair of Xeon E5345 Quad Core Processors so that when I was ready to work on the NAS they would be there in case the others were actually bad.
Well come today when I finally had some time to tackle this NAS issue I was able to get it going after installing the new Xeon E5345 CPU’s and updating the bios of my Intel S5000psl motherboard with “Intel’s Deployment Assistant” booting off of a temporarily connected internal SATA DVDRW drive I connected with the system chassis opened up. Turns out that I had to install the update since the old motherboard bios couldn’t handle the new CPU’s stepping code as well as taking care of the issue with the USB devices throwing errors. I had noted the stepping code of the new CPU’s and checked on the release notes and the new CPU stepping codes were included in the motherboard bios update.
With the new CPU’s and the motherboard updates the 9.3 FreeNAS update on my second NAS worked beautifully as I thought it would last week! I had good feelings right on initial boot off of the FreeNAS 9.3 image since when it had failed it would hang at the new FreeNAS grub screen and go no further. While following the error messages I had noticed the SCSI errors when the USB device “da0” was accessed and this no longer happened after the bios motherboard update.
I make a habit of saving my FreeNAS configuration even when doing minor updates so once I got my second NAS back into the rack and cabled up@ 9.3 FreeNAS version I logged into the web interface using the DHCP address it picked up and uploaded my old configuration and FreeNAS updated the old settings rebooting a couple of times and everything was back to normal. I verified from my XenServer shell I had the NIC’s cabled right by pinging the interfaces and then watching the storage become available in XenServer environment.
Now both NAS are @ FreeNAS 9.3 and I have my redundancy back! I also believe my old CPU’s are good but I had to test the new CPU’s to give EBay feedback plus didn’t want to switch back and forth so will repurpose the old CPU’s somehow.
Here are my babies at the bottom of my rack purring like normal. I will grab a couple of slim DVDRW devices for them for fall back devices for issues like my USB flaking on me though.
In the backyard I am pretty much finished with spring cleanup and initial planting and this past weekend was dedicated to cleanup in the backyard, the front yard I did some trimming and planted a few more flowers and put down some black bark.
In the backyard over the winter dirt and debris that catches in the pores of the aggregate concrete so I performed my annual pressure washing of the walkways. Every spring since 1997 (even at previous residence) I setup the garden haul in whatever dirt,mulch, compost then get the rototiller out and shape everything into furrows, sweep top dirt off walkways, pressure wash the rest of the dirt and then clean the pool of any dirt and debris that gets in there. It’s amazing how much dirt you track just on your feet going back and forth getting tools, going to the store getting beverages to drink etc..
I also planted some more vegetables and placed some liquid fertilizer from Miracle Grow on my drip system so I can get some auto fertilization action going for my backyard garden. In the front yard I sprinkled some rose 2-1 one fertilizer that takes care of the bugs that gather on the roses and feeds the roses as well. I planted some marigolds and petunias as well in the front and laid out some black bark that really offsets everything nicely. The black bark which I layered in generously should keep the moisture in longer as well during the drought conditions.
I also pulled out the hammocks and they are ready for spring, summer and fall duty! Took nice dip Sunday evening in the pool and ready to roll for spring and summer!
Front Yard Flower Beds Ready For Spring with Black Bark!
Backyard Vegetable Gardens Set to go! *Note Power Washed Concrete*
Hammocks on both sides of backyard out of garage and ready for relaxation!
The vegetable garden is officially kicked off now that I have turned over the soil with the rototiller which is always my official demarcation point for the start of my veggie garden! I was able to get all my vegetables that had maintained over the winter and replanted them this weekend except the mint plants. The mint plants grow so well and run so much I don’t think I will have any issues having a few replanted mint plants take off once back in the ground!
I replanted my collard greens, mustard greens, Swiss chard, and chives that had grown over the winter. I also ended up thinning out the chives and the Swiss chard which I should have done last year as well but didn’t.
I planted some new jalapeno peppers, green & red bell peppers, red leaf mustard greens, pear tomatoes, early bird tomatoes, beef eater tomatoes, Serrano peppers and that’s it so far. The grape vines are still hanging in there with the raspberry bushes ready to take over fertile ground if I don’t keep them in check. This year I may try to run the grape & raspberry vines over a trellis if I have time.
Below are this years before and after photos of the vegetable garden. I still have some clean up and more planting.
Look at all of the mint which overtook my garden.
These after pics were taken a bit after the sun had gone down which is when I finished. The replanted plants will perk up once watered and in the ground a few days.
Every year this time of the year I wage a battle against mint plants over running my garden. This year I have pledged to make better use of everything in my garden including these mint pants that will not go away. Before I would include them in some alcoholic beverages an that’s it Well this year I have started making tea with them and the tea is very tasty with a dash of sugar or better yet local honey. I even think its helping me with my allergies but we will wait for that definitive declaration!
Here is the runaway plants now that never die off over the winter they just don’t spread over night like they do in the warmer months like now You can also see below that my swiss chard, mustard and collard greens did very well over the winter. I will soon uproot and replant these plants after I turn the soil with a rototiller and start incorporating other plants into the garden this year
Above is the runaway mint plants with tea I brewed. The tea color is light but the flavor is very full and strong!
One bunch of mustard greens, swis chard, collards and mustard greens on the other side of the garden.
Updates of FreeNAS & XenServer to Current Versions Plus Disk Upgrades
Sounds like too much at one time but it was perfect since I moved all of my data to one NAS system. I have built duplicate NAS and Virtual systems using XenServer and FreeNAS. My main goal was to migrate out 2tb 3gbps drives for 3tb 6bps drives but since I had my environment in maintenance mode ready for updates I thought now would be a good time to also update FreeNAS and XenServer since I may not have this window again soon. Plus the release notes of the new versions of FreeNAS and XenServer pointed to items that would benefit my environment.
Introducing First New Drive into FreeNAS
I “Live Migrated” all of the VM’s in my XenServer environment to the XenServer SRs’ (Storage Repositories) pointing to my second NAS system so I could bring my first NAS system down and add my new drives into the first NAS. I then verified I could see the new drives in the bios of my Dell H310 SAS controller and made them non-RAID JBOD devices and rebooted back into FreeNAS.
At this point I just wanted to start the process of including the new drives into an existing Volume in FreeNAS and used information I found online in the FreeNAS guides to do that (http://doc.freenas.org/9.3/freenas_storage.html)
I started at section “8.1.11 Replacing Drives to Grow a ZFS Pool”.
I normally ssh into my FreeNAS system from my Fedora 20 workstation using a terminal or my Windows 8.1 workstation via putty.
Here are the steps I used:
#zpool status <pool-name>
#zpool get all <pool-name>
#zpool set autoexpand -on <pool-name>
In GUI-> Storage ->Volumes -> <Volume-Name> -> Volume Status (bottom Icon to far right) -> “Highlighted first drive I wanted replaced” -> Offline -> Replace <Select new drive)
– I then go and monitor resilvering time:
– #zpool status <pool-name>
After the resilvering of the first drive was completed and Volume was no longer in degraded state that comes with resilvering I decided to update FreeNAS from version FreeNAS-9.2.1.8-RELEASE-x64 (e625626 to FreeNAS-9.3-STABLE-201502162250.
Updating First FreeNAS System to 9.3
I simply went to http://www.freenas.org/download-freenas-release.html downloaded the GUI upgrade file onto my Fedora workstation. I then saved my FreeNAS config, then browsed to the downloaded GUI upgrade file and updated my FreeNAS system. Here are my steps all GUI.
System -> Settings -> Save Config > Saved to directory on Fedora Workstation
System -> Settings -> Advanced Firmware Update (follow steps)
A few reboots later and my first NAS has updated to FreeNAS 9.3.
Start Introduction of Second New Drive into FreeNAS
I did the second drive into FreeNAS the exact same way as the first:
#zpool status <pool-name>
#zpool get all <pool-name>
#zpool set autoexpand -on <pool-name>
In GUI-> Storage ->Volumes -> <Volume-Name> -> Volume Status (bottom Icon to far right) -> “Highlighted first drive I wanted replaced” -> Offline -> Replace <Select new drive)
– I then go and monitor resilvering time:
– #zpool status <pool-name>
This time however I decided to also update one of my XenServer hosts since I didn’t have any VM’s on it and shouldn’t affect the resilvering process since I had detached the iSCSI volume from Xenserver I was adding the new drives to already.
I decided to go with the “Rolling Pool Upgrade” through the GUI Wizard and see if it would let me do this without a commercial license, I figured at the very least I could see what the “Prechecks” caught.
Here I will jump ahead and warn you to download XenCenter 6.5 if you will use the Rolling Pool Upgrade method whether CLI or GUI you will lose connectivity to the servers that migrate to 6.5 as soon as they are updated. Since the Master of the pool is migrated first you lose connectivity to the rest of the XenServer systems as well. The latest XenCenter is also on the XenServer.org download page http://xenserver.org/open-source-virtualization-download.html.
Resuming chronologically… I began my rolling upgrade which is fairly straightforward. I decided to offer my media over HTTP since I had that up and running and open via firewall to all the necessary systems from my Fedora workstation.
In XenCenter GUI – > Tools -> Rolling Pool Upgrade -> Run Prechecks -> Http <Http://<ip.address-workstation>/xenserver-65 -> Checks OK -> Starts upgrade of Master Pool Server
The upgrade of XenServer master starts fine and after reboot of the successfully updated Master pool server I lost conductivity to the pool and my systems.
I quickly downloaded XenCenter by pointing my browser to my newly updated XenServer 6.5 hosts and downloaded XenCenter 6.5 for my Windows 8.1 workstation (one of the main reasons I keep a Windows workstation in the first place is so I can used XenCenter features) and tried to install and it would not install!
Updates Paused to Update XenCenter on Windows Workstation
Now you know one reason I said download XenCenter 6.5 first. I now have no visibility to XenServer except through OpenXenManager for Linux which I don’t often use and as far as I know will not help me upgrade through that GUI and if it does well I wasn’t going to see how it does at this moment. Had I installed XenCenter 6.5 I would have taken care of this ms-installer before trying the upgrades and been rolling right along.
After trying a few items on my Windows 8.1 workstation I could see something was wrong with the ms-installer and it was not installing anything including updates. The only other Windows system I have is 2012 R2 streaming my music using SAM Broadcaster software and this is where I installed XenCenter 6.5 until I could figure out what was wrong with my 8.1 workstation.
The issue with using my W2012 R2 Standard system is that it is on a KVM switch and the resolution for XenCenter was barely acceptable but it worked. The other issue is that my XenServer and FreeNAS servers are also on the same KVM switch so I would have to switch back and forth continuously to monitor reboots of the XenServer or FreeNAS system and then back to my 2012 R2 system that now has my XenCenter 6.5 installed.
Back to First XenServer Upgrade and to Second XenServer Upgrades
With XenCenter on my streaming media 2012 R2 server I can now see that my first XenServer hosts is upgraded but the second host is not. I confirm this by switching to the console of the system via my KVM as well.
I tested the newly updated 6.5 XenServer system and migrated some VM’s over started them and everything worked fine. I then decided it was time to start upgrade of the second XenServer system in my environment.
I started the second XenServer update with live test/development systems running so I could see if the Rolling Upgrade process would live migrate them. I live migrated my production VM’s on the already updated 6.5 XenServer system.
Restarting XenServer upgrade it started the same way and just picked the server that was still at 6.2 so process was the same.
In XenCenter GUI – > Tools -> Rolling Pool Upgrade -> Run Prechecks -> Http <Http://<ip.address-workstation>/xenserver-65 -> Checks OK -> Starts upgrade of Second Pool Server
This is where it shows XenServer running “host-evacuate” and migrated all the VM’s off of my Second pool server to the Master. Pretty cool stuff. Due to these live migrations though my Master Server began to run out of resources so I started shutting down unnecessary VM’s.
I noticed that the XenCenter GUI began to stall after it evacuated the disks and I switched over to my XenServer console via KVM and the system was hung so I did a reboot by holding down the power switch. Upon reboot it tried to restart the 6.5 install but couldn’t and came back up as a 6.2 system.
After validating that the system was up and undamaged at XenServer 6.2 I restarted the Rolling Upgrade through the 6.5 XenCenter GUI. This time I actively switched back and forth from the system with the XenCenter GUI to the XenServer host in time to see the XenServer host stall on a bios issue which I quickly remedied and rebooted. XenServer picked up the update on reboot an after a couple more reboots was upgraded successfully to XenServer 6.5 as verified on console of system and n the GUI.
I then started some systems up to Live Migrate them back to their home server to even out the VM distribution.
Back to Getting XenCenter Installed on Windows 8.1 Workstation
I finally was able to get XenCenter re-installed on Windows 8.1 after dozens of reboots and failed efforts in came down. to the following.
In Windows 8.1 – > File Manager -> Control Panel -> Troubleshooting -> System and Security -> Fix Problems with Windows Update, reboot. The trick is it’s Windows so you have to let it do it’s thing in the background and Windows will not let you in on what is going on. Once I was patient and let it do it’s thing it “Blue Screened” gave some useful information about “driver_power_state_failure” and that prompted me to unplug a USB SATA hard drive dock I had plugged into the system to look at some drives but was currently empty.
The Windows 8.1 workstation came back up I installed updates rebooted and then re-installed XenCenter updating it to 6.5 successfully!
Verify XenServer 6.5 Updated
Now with XenCenter properly on my Windows 8.1 workstation I was able to verify all the VM’s worked correctly. I then updated all the VM’s to XenTools 6.5 and balanced the VM’s on the respective XenServer hosts.
FreeNAS iSCSI to XenServer Storage Repository Issues
I was unable to get the former XenServer SR’s to repair and plug back into the XenServer environment. One of the reasons for updating to this version of FreeNAS was due to improvements and new features in iSCSI. I noticed that the NFS SR presented from FreeNAS upgraded NAS shows properly but the iSCSI targets will not connect in the XenServer environment.
I started looking over the iSCSI settings in FreeNAS and noticed that the “Initiators” group was gone! I compared with my other FreeNAS system I had not updated and added an initiators group.
I then had to point to the “Initiators” group in “Extents”. I also checked the “Xen initiator compat mode” option that is new in FreeNAS 9.3 on the “Extents” tab.
I still was unable to get XenServer to recognize the iSCSI targets from FreeNAS. I went to look on XenServer side of things and noticed something fantastic! The complete iSCSI target IQN was showing from FreeNAS in the General tab of the SR’s in XenServer from the updated FreeNAS 9.3 system! My non-updated FreeNAS SR’s just still had my description.
I had previously detached one SR in XenServer being presented from the FreeNAS system I updated to 9.3 so I verified the IQN on both sides and performed a reattach operation from XenServer and it worked!
I tried the repair operation with the other SR’s I did not detach prior to upgrading and had no success. I then started doing screen-shots and taking notes of the IQN’s from XenServer of the SR’s and from the targets on the FreeNAS side to prepare to “Detach” or “Forget”and reattach the SR’s within XenServer. The SR’s I am trying to get seen in XenServer again from the updated FreeNAS system have no detach options so I will have to use the “Forget” option and then try to bring these SR’s back.
Forget Bring Back FreeNAS Storage into XenServer
Using the “Forget” option and then using the “New SR” wizard in XenServer brought the iSCSI storage that formerly wouldn’t connect back into XenServer successfully!
This is what I did:
Took screen-shots of my SR settings in XenServer since they show IQN from FreeNAS
Verified the IQN and target information from FreeNAS
XenServer GUI-> Infrastructure > Right-Clicked -> SR -> Forget
XenServer GUI -> Storage -> New SR -> Software iSCSI -> Enter “Prior name from Notes/screen-shot” -> Ip.address of FreeNAS system (using default port 3260) -> Scan Target Host -> Target IQN -> Target Lun -> Reattach ( recognizes prior XenServer SR information and gives option format or reattach) – > Prior SR is back within XenServer.
That was my multiple upgrade adventure the past week which has culminated in more disk space an upgraded FreeNAS and XenServer environment. I still need to update my other FreeNAS system to the latest version after I migrate the data off that NAS on to the updated FreeNAS environment. I may also rename my updated FreeNAS volume.
I corrected a hardware error which had one of my my CPU turbo fans directed the wrong way in one of my NAS chassis. The past week I have been doing incremental work on getting my NAS and Virtual infrastructure environment updated with more disk space and newer versions of FreeNAS and XenServer. The resilvering process of FreeNAS takes a very long time 24 hours and another 34 hours and I have other chores but that is for a much longer post coming soon.
Since I had migrated all of my data to my other NAS I was free to look into a few other issues that have been bugging me about this NAS. After plugging in and insuring I could recognize another new drive I pulled the NAS partially out of the rack to see if I could tell why the high velocity chassis fans were kicked on more than the other identical NAS. Initially I blew out and moved the fans to other spots but whatever fan was near my power supply kept blowing full speed regardless if the system needed cooling or not. I then spotted the issue and I had placed one of my turbo CPU fans the wrong way!
The turbo CPU fan was pointing straight into the chassis fan that is why it was always blowing and mt temps were off!
You can see the arrow going the wrong way even! What was I thinking when I did this I don’t know! This makes for horrible airflow turbulence in a system that should have a steady consistent flow out the back of the system away from the hard drives over the RAM straight out the back to some more waiting fans pointing out of the back of the system!
Well I should just have been able to loosen the screws on the offending CPU fan put down some more thermal compound on the CPU and fan assembly but that would not make me suffer enough. the standoffs decided to come loose from the chassis and not the fan assembly so I had to pull this heavy 70 pound system with drives out of the bottom of my rack late at night and take it to the hallway on my test bench after moving some other items I was working on from the bench. I had to take the motherboard and all the components completely out of the system to properly unscrew the chassis standoffs from the CPU cooling assembly.
Also if you look close enough you can see the second set of standoffs for the other CPU that had the fan assembly pointed in the proper direction had no issue staying in place. Well I put the standoffs tightly back in place then put the system back together and used my leveraging tricks to get my heavy system back into the rack. The system sounds so much better without two high powered fans blowing right into each other from about an inch away!
Tomorrow I resume my other upgrades!
Here is my NAS baby back in in place at the bottom of my 7 foot rack with a stack of my leverage pieces in the foreground.
Well it took me a while to move CMS platforms but the reasons I moved mainly had to do with Movable Type having so few development plugins\extensions and their almost total abandonment of the Open source version. I was a few years late moving from Movable Type to WordPress and there were several reasons including time and life’s normal flows.
One reason it took me so long to migrate off of Movable Type is that Movable Type was very flexible so any plugins or widgets as they call them in Movable Type you can create easily by embedding that code into custom widgets.
The main reason it took me so long to go to a new CMS is that Movable Type Pro has in it’s core a method for easily creating sub-domains and sites that other CMS’ didn’t have off of one code base and database. WordPress has had multi-sites for several years but it didn’t seem to be a core function and seemed to require additional plugins to use how I wanted to use it.
Movable Type is also very portable so moving from site to site even with simple names is very intuitive and easy. Worpress even in its recent development cycle it takes some tedious work and there is no native built in function as there is in Movable type. Movable Type makes it easy to use root relative paths while WordPress hard codes domains into the database. Unfortunately Movable Type could not parlay their strengths and involve more of the development community and still seem to be far behind other CMS in involving developers.
Another reason it took so long for me to switch CMS’ is I was also experimenting with other CMS’ like Drupal, Joomla, Plone and a few others. Easily having sub-domains off of one code and database was the reason I didn’t go with Drupal. The reason I didn’t go with Joomla was I was not comfortable with the blogging within Joomla it seemed adding content/posts was not a cohesive process and I didn’t want to be tied into any of the leading blogging extensions offered as a Joomla extension.
I wish WordPress was more portable is my biggest gripe. I tend to get my sites working on a local virtual system on my LAN and then just clone and forward the appropriate ports to the outside world. With WordPress I need to get into the database and get this done through searches or a script that has been supplied @ https://interconnectit.com/products/search-and-replace-for-wordpress-databases/. There is sufficient documentation on moving sites (http://codex.wordpress.org/Moving_WordPress) even for multi-site but I wish it wasn’t so tedious. I will also watch my naming conventions on my systems going forward as well so that in the future I will be able to use the script much more easily to change unique names without having to go through the scripts output to exclude items if I use common system names.
Wordpress has a wide range of themes, tons of plugins, a very active community, multi-site seems more part of the core now, and is more flexible than I thought. I can have sub-domains off of one code and database in WordPress making administration easier. Adding content, posts, pages, managing comments, putting custom code into widgets and plugins etc is very straightforward in WordPress which is what I was after so overall so far very happy with my WordPress move.
Well I finally switched off of Movabletype blogging\CMS for my site. I moved to WordPress mainly for the blogging and I have WordPress and Joomla for my Internet Radio site “Snakeice’s House of Beats”.
Importing the textual posts wasn’t that difficult but trying to massage the URLs for the same previous paths and getting images to show I had to do some behind the scene tweaks in my filesystem and with Apache rewrites and redirects.
I still have some tweaking to go but feel free to browse through the site there is quite a bit of items under the hood and I hope to get single-sign on going between WordPress and Joomla
