October 18, 2014
by Darryl Hoytt
Comments Off on Exciting Friday Night Updating NAS and Virtual Environment

Exciting Friday Night Updating NAS and Virtual Environment

It has been a while since I updated my NAS systems which run FreeNAS and my XenServer hosts which have my virtual hosts that I share to the world via my blogs and websites. My music server due to the high CPU utilization from running multiple audio streams with varied bit rates is a physical system that was unaffected except you couldn't see the website.

I decided to update the storage first but since I wanted to cut down on the time I decided not to migrate the storage and just shutdown all of the virtual systems. While the VM's shutdown I saved the FreeNAS configuration from both of my NAS systems to one of my local systems. Once the VM's shutdown I updated from the GUI my first NAS system I call "baja".

FreeNAS Update

Baja my first NAS system took the FreeNAS 9.2.1.8 update rebooted and looked ok until it hung with an "istgt_acceptor" error. I looked the error up didn't find much and decided to shut the power off of my NAS and then powered it back up performed a reboot on its own after coming all the way up almost then came up just fine. I tested by bringing up a couple of VM's that had used the iSCSI targets from that device and they came up nicely as well.

Mojave my second NAS system which I also use FreeNAS now that really caused some headaches! After updating to FreeNAS 9.2.1.8 from the GUI it started to come up and it had a CPU issue with some errors that had been logged. After I took care of the CPU issues on my NAS system "Mojave" I turned on a switch to stop on serious errors in the motherboards bios.

Well reboot my system stops in a bios error window with an "8525" memory error on DIMM B2! Well I located a system that I don't use as frequently and could stand a little less RAM and started to open it up and then stopped. I went ahead and opened up my "mojave" NAS system reseated DIMM B2 and confirmed the system would bypass that error and then slid it back into the rack. On reboot of second NAS system I now started getting a timeout error on my USB drive with my FreeNAS installation.

The error code I get now on my "mojave" NAS system is a timeout associated with the error "quirks no_6_byte" and it refuses to go any further. I powered the system down removed the USB blew on it and placed it back in and then the system hung on my iSCSI target just like my other NAS so I rebooted and then got hung up on the USB issue again. I went back and forth on this with reboots about 5 times with different results.

Wanting to take the USB out of the equation I downloaded the USB FreeNAS 9.2.1.8 image, placed the original USB I had the image on into another USB slot on my NAS server booted the system while I went to locate my travel bag of USB sticks. As I sat down to update my new USB stick that I had just removed some files from my second NAS system booted up just fine on the original USB stick on the different USB port. I tested with a few VM's and everything looks fine.

XenServer Update

I then installed about 5 patches to catch up on my XenServer patching I hadn't done since I updated to XS62ESP1 XenServer SP1 around December last year. The patching (patches I put on system: XS62E015, XS62ESP1003, XS62ESP1005, XS62ESP1009, XS62ESP1013, XS62ESP1014) was uneventful except for the multiple reboots. I updated via a putty shell ssh into my XenServer slave hosts that used to be my master. Everything looks good except I have two Centos 7 systems that have disks that are not activein XenServer. All my Fedora and Windows systems seem to be coming up fine.

I'm still working on trying to activate the disks on the Centos 7 VM's. I am taking two approaches on activating the disks. One VM system I had done some work but it's not a lot so that one I am in the process of exporting and then will import back to see how that works. A slow process so started writing this. The other centos 7 VM system I will try to activate by using the "vbd-plug" command and see how that flies.

My XenServer update to be continued later tonight, this morning or sometime this weekend. This post is just another test to see if my blog running on Fedora is working.

******************Update of XenServer Update****************

I took care of the two Centos 7VM's that I could not bring up after the updates of my FreeNAS and XenServer. I ended up going to the XenCenter console and placing the XenTools into the DVDCD and with the XenTools attached both VM's came up.

I then mounted XenTools ISO and installed XenTools without the supplied script:

#mount -t iso9660 /dev/sr0 /media

#cd /media/Linux/

#yum install ./xe-guest-utilities-6.2.0-1137.x86_64.rpm ./xe-guest-utilities-xenstore-6.2.0-1137.x86_64.rpm

Rebooted the Centos 7 VM's and everything looks great!

Yes I went down the wrong hole late but quickly corrected that and got it working! J

July 29, 2014
by Darryl Hoytt
Comments Off on Late Season Vegetable Garden Getting Better With Much Work

Late Season Vegetable Garden Getting Better With Much Work

Well the late season vegetable garden has had its first harvest a few weeks ago and has had a ton of weeds in between when I started the garden and now. The last of the weeds I mostly removed over the past weekend. The weeds sprang up very thick and plentiful due to me using compost from local horse stables last year which of course had seeds that had not been sterilized in the compost. The seedlings produce what looks like crab grass but I think it's called "quack grass". They spread further from my rototilling.

This quack grass spreads like a small carpet once it gets in the soil. The good thing about this quack grass is that once you pull it up by the roots it never grows in that area again. I tested that a few weeks ago when I pulled up about 4 x 33 gallon yard bags of the grass up and the places I pulled up the grass it never grew again. I then waited until the patches left grew to a size where I could pull them up as well by the roots. At first I was going to put down some weed block to kill them off but that would have ruined my flexibility with planting more seedlings and plants later in the season as I like.

I planted some more hot peppers, basil and some bell peppers. The plants I planted already are producing jalapeno peppers, tomatoes (early bird & yellow pear), egg plants and the cucumber vines are running and cucumbers are sprouting all over the place.

Here is my first small harvest a few weeks ago:

Here is the out of control weeds:

Here is what it looks like after the weed extraction process this past weekend:

Some veggies showing promise:

June 28, 2014
by Darryl Hoytt
Comments Off on Late Season Vegetable Garden in Place

Late Season Vegetable Garden in Place

My late start on a vegetable garden is finally done. Will probably add some more items here and there plus fine tune the sprinkler drip system as the season moves along but that's to be expected.

This is the final look with a bit of cement cleanup to do:

These are pictures after running the rototiller some more to really turn over the dirt and break it up with that good soil I used the wheelbarrow to get in the last few years. This year the soil was not sticking and I have pretty much diluted that hard packed clay soil that can be common around here in spots.

Oh well break is over time to hit the front yard a bit and then take the items I took out the garden to the dump along with a few other items. Plus I have a bunch of bugs crawling on me from the garden.

June 26, 2014
by Darryl Hoytt
Comments Off on A Hump Day Evening after Work Gardening Triumph

A Hump Day Evening after Work Gardening Triumph

I had the perfect storm this evening for getting my weeds chopped down in my garden this evening. First I happened to work from home the end of the day due to a family emergency that turned out ok. Then the skies were overcast and the work I had schedule for later that night for work I was really on top of already! I put on some shorts I didn't mind gardening in that already had some tears, a shirt that was well worn, some old running shoes, a water proof hat that provided shade and protection from vegetation, some gloves that were tough yet flexible and my knee pads that were built for flooring and I was ready!

Here is the disaster I was starting with:

This is the middle stage where I was going to leave things since I didn't think that my rototiller was going to work since I had left gas in there for a long period without using it:

This is what I ended up with by the end of the evening after realizing my rototiller was working ok:

So now I jus have to rake though the rototilled ground to get the remainder of the vegetation and then place my drips hoses back into place and start a late vegetable garden. I already have the plants scoped out at local stores that are already matured for a late start on a garden for this year!

April 14, 2014
by Darryl Hoytt
Comments Off on Hammock Rest Gone Wrong

Hammock Rest Gone Wrong

Well after a few hours cleaning up in the yard there is nothing like a nice stay in a hammock with a cold drink by your side unless the hammock feels you are not quite done cleaning up the yard! As you see in the pictures I'm not quite through with the yard, a trip to the dump and some pressure washing the concrete and a bit more lawn maintenance.

Thanks hammock message received I will commence to fixing up the rest of the yard!!

March 13, 2014
by Darryl Hoytt
Comments Off on Updated NAS Systems Again

Updated NAS Systems Again

Well added more functionality to both NAS systems. Yesterday I added the second Dell Perc H310 to one NAS system plus added two more 2 TB SAS 6 GB/s drives. Tonight after a very long session working on some clusters at work I finished up my second NAS adding another H310 plus identical drives as the first system.

The only thing I didn't do on the second NAS I did yesterday on the first NAS was to create new storage and another iSCSI target. Yesterday I moved virtual data to the first NAS that I had on the second NAS. Tonight I didn't much feel like creating additional storage groups; I'll do that on another date soon.

I now have the capabilities to talk to all sixteen SAS slots in my chassis. I could have went many routes like another expander, funky cables but adding the second set of SAS controllers was the most efficient and direct solution believe it or not. I would have liked to been able to daisy chain one section of the backplane to the other slots but I didn't see that option. Plus the price was right for the controllers from an EBay dealer so this closes this chapter, for now I can just add storage as needed and start to migrate off of my 3 GB/s drives.

Soon will update to the new FreeNAS point release I think there was some items in the release notes I could benefit from, I'll have to look again. I will also look into the latest updates of XenServer and then its diving back into looking into different CMS/Blog for my sites.

February 10, 2014
by Darryl Hoytt
Comments Off on Replaced Backplane on Second NAS

Replaced Backplane on Second NAS

Well I finally replaced the backplane on the second NAS system I had built. The backplane had come damaged and reseller had sent me a replacement a couple of weeks ago but today is the first time I had time to replace it.

I thought I could replace it while in the rack but the grounding screw for the backplane was blocked by the rail for the rack so I had to pull the whole system out of the rack. The big deal about pulling the system out of the rack is it is about 100 lb's plus filled out and at the bottom of the rack so it is not optimal for one person putting it back in. To assist me placing the system back into the rack and lining up the rails I stacked up some thick books that provided me a platform to line up the rails in the rack.

Here is a close up of the damaged SAS interfaces on the backplane. This is the reason I needed to replace the backplane.

I had to pull out the drives & empty drive bays so I could slide the backplane into place.

This is the inside front of the NAS system minus the backplane.

This shows damaged backplane next to replacement backplane.

Some backplane connections I had to temporarily disconnect to slide in the replacement backplane.

Everything tested fine and all drives are seen and the storage is presented to virtual enviornment and systems just fine again. Only issue is one of the LED connections that shows blue power on LED is out in upper right.

January 13, 2014
by Darryl Hoytt
Comments Off on Second NAS Rebuild Compete

Second NAS Rebuild Compete

Well finished rebuilding my second NAS into a new chassis identical to the other with the same specifications as the other NAS today with a few issues. Everything went pretty much the same as the other NAS rebuild except that I discovered damaged bays on the backplane on the upper right hand bay and then the bay just below that are completely smashed preventing the drives from sliding in.

So being that this chassis has 4 bays across and 4 bays down and the controller plugs into each row and I have 8 drives to place into these bays means I need two complete rows to make my current situation work in this chassis. Thankfully when I checked the two lower bays they had no damage. Still I could not get the drives into the end slots.

With the top of the case off I looked at the backplane and noticed that the right side was not seated properly and apparently someone had done the no-no for computers and tried to torque the drives into the end slots anyway without figuring out why they didn't go in smoothly!

So I made sure no drives were in the bays loosened the screws on the backplane and straightened out the backplane and was able to get the drives to slide in on the bottom rows all the way across. I then had to run the cables from my SAS card to the bottom rows of the backplane. Next step trying to get backplane replaced for future expansion and get the accompanying and missing SAS cable for the backplane as well.

So I was able to get FreeNAS up again and it's interacting with my virtual environment just fine plus sharing out other storage as well but need to get that new backplane in house for the future to have that warm fuzzy feeling that everything is okay.

I will have more pictures that will be fairly identical to the other later except placement of the drives due to the backplane damage.

January 2, 2014
by Darryl Hoytt
Comments Off on Rebuild of One NAS System Initial Complete and Moved FreeNAS Configuration

Rebuild of One NAS System Initial Complete and Moved FreeNAS Configuration

Well in an effort to save space and go up to 6gbs enviornment totally eventually I rebuilt my physical NAS system. I went from having a:

4U rack mount case that I booted FreeNAS off
an Adaptec 39160 SCSI card,
- with a 73gb 10k Segate ST373207LW SCSI drive

Also in the 4UCase:

a Dell PERC5e connected to
an External SGI 3U Omnistor SE301, 16 Bay JBOD Disk Array
- with 3x 2TB 3gbs SAS Drives
with a Dell PERC H310i 6gbs SAS card
- 2 x 2TB 6gbs SAS hard drives internally (in 4u Case)
- Plus 1 x 4tb 6gbs SAS hard drive (in 4u Case)
The motherboard is an Intel S5000PSL
2 x Intel(R) Xeon(R) CPU E5345 @ 2.33GHz
2 x Dynatron H185 Intel Xeon Socket 771
with 20gb FB DIMMS
with HP NC7170 PCI-X Dual Gigabit NIC

In my newly built enviornment I have it booting FreeNAS off of a thumb flash drive with the following:

3U Chenbro RM31616 16-bay Storage server Chassis with 6Gb/s mini SAS
Dell H310 6gbs SAS card
3 x 2TB 3gbs SAS drives
4 x 2tb 6gbs SAS drives
1 x 1 6gbs SAS drives
Intel S5000PSL Motherboard
2 x Intel(R) Xeon(R) CPU E5345 @ 2.33GHz
2 x Dynatron H185 Intel Xeon Socket 771
20GB FB DIMMS
HP NC7170 PCI-X Dual Gigabit NIC

I was able to move the FreeNAS configuration over by saving it to a file on one of my desktop systems and then after it picked up a DHCP address on boot up I uploaded the old configuration and it came up showing my old iSCSI targets to my Virtual environments as well as my NFS shares!

Well this was quite the overhaul! I had to customize and drill some additional holes into some old rails (get new cobalt bit from Home Depot) I have, got some smaller screw heads since this new case was a bit wider, jump over to Fry's get some cables to take internal fans from 4 pin to 3 pin then get that heavy beast up in the rack and cabled. I also had to take the chassis out to the garage and clean it since i got it used. I also added a couple of 50mm fans at the back of the case in addition to the 4 high volume fans that came with the case

I am happy with the functionality of the new unit and the hot swap capability with the 6gb SAS back-plane I can now make use of a source I have found for some decently priced 6gbs SAS hard drives. The only complaint I now have is that the noise level has gone up considerably mostly due to the 3 x 400W Power supplies.

December 14, 2013
by Darryl Hoytt
Comments Off on Rebuilt Nagios Server, Updated XenServer, Corrected HW Issue on FreeNAS Node, Reconfigured Router Tonight

Rebuilt Nagios Server, Updated XenServer, Corrected HW Issue on FreeNAS Node, Reconfigured Router Tonight

All in one night I took my Nagios server from running Fedora 16 to CENTOS 6.5, Updated XenCenter to the latest version 6.2.2 and updated XenServer to 6.2.0 SP1 and solved and issue where my iSCSI target was failing in FreeNAS. I had only planned to get Nagios up and going on the CENTOS server but upon completion of getting the Nagios server configured with the basic systems and monitoring I noticed this nice red X on one of my iSCSI targets in XenServer.

First the Nagios server I had copied my configuration files to my NFS drives prior to blowing away the Fedora 16 OS, so once I had CENTOS 6.5 installed I connected back to the NFS share and made the appropriate changes to my configuration files to point to my new servers. Now that my web server and gaming server plus a few other systems are now in my XenServer virtual environment with different names I just had to go and make the changes in the Nagios configuration files.

I was about to happily grab a book and retire to my bedroom when the iSCSI issue on my XenServer came to my attention. I looked and it said it was missing 2 of my multipathing connections. I checked the connectivity and that was fine as it should have been otherwise several other storage volumes should have shown the same issue.

Well logging into my second FreeNAS node I noticed that it was throwing out some errors and I could not login to do anything. I captured some of the errors with this really catching my eye: "istgt_iscsi.c:5551:worker:***ERROR***pthread_create() failed". I found a number of people having this issue in the FreeNAS forums with memory issues caused by ZFS volumes. Last time I checked this specific node having the issues was carrying less of a load and not using near as much memory as the other node not giving errors.

I finally decided to shutdown my environment and troubleshoot that FreeNAS node. What I found was that there were some CPU errors with the codes of "8121" (temp trip, a minor error) & "84ff (CPU system log full a major error)". I went into the bios of the system and cleared the error logs and the system then that FreeNAS node came up just fine. When I placed the loads back on both FreeNAS systems they both are once again working like troopers. I do need to tackle temperature issues in my computer room though.

While I was going through the logs of XenServer I had noticed the XenCenter update and the SP1 update and figured since I had created my own little maintenance windows I may as well do the updates now! Updating the XenCenter of course was no issues just run the ".msi" file and it took about 2 minutes. Installing XenServer XS62ESP1took a bit more time but went smoothly as well. The SP seemed to have some support for Windows 8.1 desktop and W2012 Server version plus some drivers so it should be a big plus and I will upgrade the VM XenTools later especially on my W2012 server evaluation server.

I also took the time to switch back to my original master for my XenServer pool cluster.

I also reconfigured one of my router's that I had already installed DD-WRT on for one of my static Ip's, what a night!

So tonight rebuilt Nagios server with new Linux version of Centos 6.5, updated my Virtual environment and corrected some issues on my FreeNAS. All I wanted to do was my Nagios server but a productive night with no sleep. I'm going to bed and will unplug the land line to the bedroom!