April 14, 2014
by dhoytt
0 comments
Well after a few hours cleaning up in the yard there is nothing like a nice stay in a hammock with a cold drink by your side unless the hammock feels you are not quite done cleaning up the yard! As you see in the pictures I’m not quite through with the yard, a trip to the dump and some pressure washing the concrete and a bit more lawn maintenance.
Thanks hammock message received I will commence to fixing up the rest of the yard!!
March 12, 2014
by dhoytt
0 comments
Well added more functionality to both NAS systems. Yesterday I added the second Dell Perc H310 to one NAS system plus added two more 2 TB SAS 6 GB/s drives. Tonight after a very long session working on some clusters at work I finished up my second NAS adding another H310 plus identical drives as the first system.
The only thing I didn’t do on the second NAS I did yesterday on the first NAS was to create new storage and another iSCSI target. Yesterday I moved virtual data to the first NAS that I had on the second NAS. Tonight I didn’t much feel like creating additional storage groups; I’ll do that on another date soon.
I now have the capabilities to talk to all sixteen SAS slots in my chassis. I could have went many routes like another expander, funky cables but adding the second set of SAS controllers was the most efficient and direct solution believe it or not. I would have liked to been able to daisy chain one section of the backplane to the other slots but I didn’t see that option. Plus the price was right for the controllers from an EBay dealer so this closes this chapter, for now I can just add storage as needed and start to migrate off of my 3 GB/s drives.
Soon will update to the new FreeNAS point release I think there was some items in the release notes I could benefit from, I’ll have to look again. I will also look into the latest updates of XenServer and then its diving back into looking into different CMS/Blog for my sites.
February 9, 2014
by dhoytt
0 comments
Well I finally replaced the back-plane on the second NAS system I had built. The back-plane had come damaged and reseller had sent me a replacement a couple of weeks ago but today is the first time I had time to replace it.
I thought I could replace it while in the rack but the grounding screw for the back-plane was blocked by the rail for the rack so I had to pull the whole system out of the rack. The big deal about pulling the system out of the rack is it is about 100 lbs plus filled out and at the bottom of the rack so it is not optimal for one person putting it back in. To assist me placing the system back into the rack and lining up the rails I stacked up some thick books that provided me a platform to line up the rails in the rack.
Here is a close up of the damaged SAS interfaces on the back-plane. This is the reason I needed to replace the back-plane.
I had to pull out the drives & empty drive bays so I could slide the backplane into place.
This is the inside front of the NAS system minus the back-plane.
This shows damaged backplane next to replacement backplane.
Some backplane connections I had to temporarily disconnect to slide in the replacement back-plane.
Everything tested fine and all drives are seen and the storage is presented to virtual environment and systems just fine again. Only issue is one of the LED connections that shows blue power on LED is out in upper right.
January 12, 2014
by dhoytt
0 comments
Well finished rebuilding my second NAS into a new chassis identical to the other with the same specifications as the other NAS today with a few issues. Everything went pretty much the same as the other NAS rebuild except that I discovered damaged bays on the backplane on the upper right hand bay and then the bay just below that are completely smashed preventing the drives from sliding in.
So being that this chassis has 4 bays across and 4 bays down and the controller plugs into each row and I have 8 drives to place into these bays means I need two complete rows to make my current situation work in this chassis. Thankfully when I checked the two lower bays they had no damage. Still I could not get the drives into the end slots.
With the top of the case off I looked at the backplane and noticed that the right side was not seated properly and apparently someone had done the no-no for computers and tried to torque the drives into the end slots anyway without figuring out why they didn’t go in smoothly!
So I made sure no drives were in the bays loosened the screws on the backplane and straightened out the backplane and was able to get the drives to slide in on the bottom rows all the way across. I then had to run the cables from my SAS card to the bottom rows of the backplane. Next step trying to get backplane replaced for future expansion and get the accompanying and missing SAS cable for the backplane as well.
So I was able to get FreeNAS up again and it’s interacting with my virtual environment just fine plus sharing out other storage as well but need to get that new backplane in house for the future to have that warm fuzzy feeling that everything is okay.
I will have more pictures that will be fairly identical to the other later except placement of the drives due to the backplane damage.
January 1, 2014
by dhoytt
0 comments
Well in an effort to save space and go up to 6gbs environment totally eventually I rebuilt my physical NAS system. I went from having a:
Also in the 4UCase:
a Dell PERC5e connected to
an External SGI 3U Omnistor SE301, 16 Bay JBOD Disk Array
with 3x 2TB 3gbs SAS Drives
with a Dell PERC H310i 6gbs SAS card
2 x 2TB 6gbs SAS hard drives internally (in 4u Case)
Plus 1 x 4tb 6gbs SAS hard drive (in 4u Case)
The motherboard is an Intel S5000PSL
2 x Intel(R) Xeon(R) CPU E5345 @ 2.33GHz
2 x Dynatron H185 Intel Xeon Socket 771
with 20gb FB DIMMS
with HP NC7170 PCI-X Dual Gigabit NIC
In my newly built enviornment I have it booting FreeNAS off of a thumb flash drive with the following:
3U Chenbro RM31616 16-bay Storage server Chassis with 6Gb/s mini SAS
Dell H310 6gbs SAS card
3 x 2TB 3gbs SAS drives
4 x 2tb 6gbs SAS drives
1 x 1 6gbs SAS drives
2 x Intel(R) Xeon(R) CPU E5345 @ 2.33GHz
20GB FB DIMMS
I was able to move the FreeNAS configuration over by saving it to a file on one of my desktop systems and then after it picked up a DHCP address on boot up I uploaded the old configuration and it came up showing my old iSCSI targets to my Virtual environments as well as my NFS shares!
Well this was quite the overhaul! I had to customize and drill some additional holes into some old rails (get new cobalt bit from Home Depot) I have, got some smaller screw heads since this new case was a bit wider, jump over to Fry’s get some cables to take internal fans from 4 pin to 3 pin then get that heavy beast up in the rack and cabled. I also had to take the chassis out to the garage and clean it since i got it used. I also added a couple of 50mm fans at the back of the case in addition to the 4 high volume fans that came with the case
I am happy with the functionality of the new unit and the hot swap capability with the 6gb SAS back-plane I can now make use of a source I have found for some decently priced 6gb\s SAS hard drives. The only complaint I now have is that the noise level has gone up considerably mostly due to the 3 x 400W Power supplies.
December 14, 2013
by dhoytt
0 comments
All in one night I took my Nagios server from running Fedora 16 to CENTOS 6.5, Updated XenCenter to the latest version 6.2.2 and updated XenServer to 6.2.0 SP1 and solved and issue where my iSCSI target was failing in FreeNAS. I had only planned to get Nagios up and going on the CENTOS server but upon completion of getting the Nagios server configured with the basic systems and monitoring I noticed this nice red X on one of my iSCSI targets in XenServer.
First the Nagios server I had copied my configuration files to my NFS drives prior to blowing away the Fedora 16 OS, so once I had CENTOS 6.5 installed I connected back to the NFS share and made the appropriate changes to my configuration files to point to my new servers. Now that my web server and gaming server plus a few other systems are now in my XenServer virtual environment with different names I just had to go and make the changes in the Nagios configuration files.
I was about to happily grab a book and retire to my bedroom when the iSCSI issue on my XenServer came to my attention. I looked and it said it was missing 2 of my multipathing connections. I checked the connectivity and that was fine as it should have been otherwise several other storage volumes should have shown the same issue.
Well logging into my second FreeNAS node I noticed that it was throwing out some errors and I could not login to do anything. I captured some of the errors with this really catching my eye: “istgt_iscsi.c:5551:worker:***ERROR***pthread_create() failed”. I found a number of people having this issue in the FreeNAS forums with memory issues caused by ZFS volumes. Last time I checked this specific node having the issues was carrying less of a load and not using near as much memory as the other node not giving errors.
I finally decided to shutdown my environment and troubleshoot that FreeNAS node. What I found was that there were some CPU errors with the codes of “8121” (temp trip, a minor error) & “84ff (CPU system log full a major error)”. I went into the bios of the system and cleared the error logs and the system then that FreeNAS node came up just fine. When I placed the loads back on both FreeNAS systems they both are once again working like troopers. I do need to tackle temperature issues in my computer room though.
December 2, 2013
by dhoytt
0 comments
Well this weekend I made great progress I built a test media server, virtualized a few old physical web servers, as well as my old but powerful XP desktop with 5 SCSI drives I just retired. I have been trying different methods with varied success depending on OS. I am still getting my preferred steps down but I have made progress in all of the various methods.
I have used Clonezilla & Xenserver combination:
Create an image to an NFS share on my NAS using Clonezilla Live CD to boot networked Linux webserver
Boot Clonezilla ISO using XenServer’s NFS ISO library
Restore Clonezilla image from NFS NAS
Boot Rescue Disk and rebuild initrd
Install Xenserver Tools
Used Vmware to Vmconvert Linux Webserver
Boot Rescue Disk and rebuild initrd
Install Vmware Tools
Then export template to desktop
Import template into XenServer
Use XenConvert to clone XP system (not much luck), created VHD disk for C: and quit.
Use Vmware 5.5 Converter to clone XP Desktop system with 5 local SCSI drives kept failing until I deselected last drive.
Crashed VMware ESXi hosts twice. I had upgraded to 5.5 this weekend on that ESXi server. I still need to troubleshoot what happened.
Also tried using Xenconvert on Linux systems using CD but I think with 6.2 it may not be possible.
I think in the future using “DD” and NFS with Xensever may also be an option, maybe even the best option for Linux systems but that’s for a bit later.
This whole exercise just got interrupted as the power on my 20 amp dedicated circuit got overwhelmed and took down my 2 custom built NAS devices running FreeNAS, along with my virtual environments of XenServer& Vmware, and internet radio staion. I will have to remedy that need for another dedicated 20 amp circuits and not wait any longer to take care of that issue! I had to run outside and reset the circuit breaker.
Anyway more details later and maybe I will have time to fully document the procedures along with some other items. I can now comfortably start allocating this hardware to other uses like media servers or pass them along to those who need systems with new desktop OS’ of course.
Looks like I still have a lot of work to do to get UPS and network wiring redone with new equipment along with redoing my websites. Looks like I will totally be moving to Drupal away from Movable Type. Stay tuned!
November 7, 2013
by dhoytt
0 comments
I was heading off to the bedroom to read a few pages of the latest I’m book reading while I performed a XenServer Storage motion on a couple of my virtual Linux servers after upgrading my NAS memory and storage capacity the last few nights and I could not find the book I was reading. I looked in my favorite eBook readers on my Motorola Atrix checking Moon Reader, Cool Reader, Aldiko and Kindle but none of the books on my continued reading were the fantasy book I had just started by reading a few pages of earlier this week.
Finally I clicked on an ad for an audible reader download while closing one of the eBook readers and looked over to my end table when I realized where my book was. I had been reading an actual physical book that you turn pages on made of paper! I had not read a paperback book in so long I failed to realize that what I was looking for was a book made out of actual tree materials! I found my book and I will be turning actual pages by hand, will not be able to automatically go to where I left off unless I leave a paper book mark, I will not be able to automatically look up the definition of words etc..
I feel this is like going back to nature or a trip to the past reading an actual novel not in electronic format. Looking over at my overstuffed bookshelf I am realizing that only a few years ago I was wondering where I was going to stuff anymore books I bought when I made bi-weekly trips to a used book barn to buy a handful of books at a time. Such nostalgia in a few years time of changed habits and expectations.
October 26, 2013
by dhoytt
0 comments
How have I done without FreeNAS for so long? This is like that fun gal that can go from the fine dining wine & tea, hang in the hood and then hit the outdoors without skipping a beat !
I built this NAS system earlier this year for my virtual environments mainly so they could be physically High Availability (HA) with shared storage networks that are redundant, multi-paths etc… I just needed one more piece to make it all work together and FreeNAS provides the interface, software know-how all together in a great Open Source package! Really the only thing holding me back has been I need to add another dedicated 20 amp circuit to go along with the one already in my computer room and I might add two more circuits because I like playing with stuff!
With FreeNAS I can share to any OS using all the usual suspects, NFS, iSCSI, CIFS, even Apple’s AFP which I have never used. I mainly use NFS & iSCSI and that let’s me create the storage needed for my XenServer & VMware virtual environment and present storage to my Linux and Windows servers as well as my Linux & Windows desktops.
So why do I say that FreeNAS is like that woman that can handle any environment without skipping a beat? Well FreeNAS has proven it can handle all the issues that come up without the use of a specific 3rd party app or additional handling of external devices one storage system does it all!
Here is what FreeNAS is doing that has me stoked :
Reliable storage making my Virtual environment HA where VM’s move from host to host seamlessly.
Used NFS to share out storage to save CloneZilla images of systems over LAN .
Outgrew my Linux desktop, built another Linux desktop system, shared NFS share to migrate the data in home directory that I needed.
Ran out of space on my Windows desktop, built another Windows desktop and needed to import Outlook mail to Thunderbird email but had no disk space on XP system (yes XP). Created 100gb iSCSI target on FreeNAS, downloaded Windows iSCSI initiator on XP system, format with NTFS, install Thunderbird email on iSCSI drive on windows XP start importing email to Thunderbird during day while I do other tasks. No need to install another hard drive on system or install Outlook again on my Windows 8.1 system.
Creating a media archive with all my music, movies, pictures, personal, financial, legal documents etc..
With Xenserver I can even use FreeNAS for my NFS ISO library so I can easily pick any OS or other ISO to boot off of to troubleshoot or build a VM.
I have one disk array acting as backup with FreeNAS.
Shares music I play on my Internet Radio station
I can keep the storage on specific LAN’s I now have 3 with one strictly storage but I may add another 1 or 2 since some of the iSCSI to my desktops are going over my traditional LAN’s.
FreeNAS adapts capably to whatever scenario I present and seamlessly just like the woman that can fit in anywhere, does it all and enjoys it! Haven’t met the gal like that yet but FreeNAS holds the blueprint!
July 13, 2013
by dhoytt
0 comments
Normal
0
false
false
false
EN-US
X-NONE
X-NONE
MicrosoftInternetExplorer4
/* Style Definitions */
table.MsoNormalTable
{mso-style-name:”Table Normal”;
mso-tstyle-rowband-size:0;
mso-tstyle-colband-size:0;
mso-style-noshow:yes;
mso-style-priority:99;
mso-style-qformat:yes;
mso-style-parent:””;
mso-padding-alt:0in 5.4pt 0in 5.4pt;
mso-para-margin-top:0in;
mso-para-margin-right:0in;
mso-para-margin-bottom:10.0pt;
mso-para-margin-left:0in;
line-height:115%;
mso-pagination:widow-orphan;
font-size:11.0pt;
font-family:”Calibri”,”sans-serif”;
mso-ascii-font-family:Calibri;
mso-ascii-theme-font:minor-latin;
mso-fareast-font-family:”Times New Roman”;
mso-fareast-theme-font:minor-fareast;
mso-hansi-font-family:Calibri;
mso-hansi-theme-font:minor-latin;}
Well I learned some valuable lessons about upgrading from XCP 1.6 to XenServer 6.2. The lessons were it was important to update to the latest version of XenCenter to match the version of XenServer you intend to update to first if using XenCenter manage systems and pay attention to your status output commands (I know I’ll explain).
For proper background information I updated my XCP1.6 installation with a local CDROM on my physical hosts system. I booted off the ISO DVD I created and it recognized my 1.6 XCP install asked if I wanted to update and followed the prompts from there. I was surprised that the XCP is no longer going to be developed but instead replaced by XenServer going forward since Citrix has now made XenServer fully Open Source.
Normal
0
false
false
false
EN-US
X-NONE
X-NONE
MicrosoftInternetExplorer4
/* Style Definitions */
table.MsoNormalTable
{mso-style-name:”Table Normal”;
mso-tstyle-rowband-size:0;
mso-tstyle-colband-size:0;
mso-style-noshow:yes;
mso-style-priority:99;
mso-style-qformat:yes;
mso-style-parent:””;
mso-padding-alt:0in 5.4pt 0in 5.4pt;
mso-para-margin-top:0in;
mso-para-margin-right:0in;
mso-para-margin-bottom:10.0pt;
mso-para-margin-left:0in;
line-height:115%;
mso-pagination:widow-orphan;
font-size:11.0pt;
font-family:”Calibri”,”sans-serif”;
mso-ascii-font-family:Calibri;
mso-ascii-theme-font:minor-latin;
mso-fareast-font-family:”Times New Roman”;
mso-fareast-theme-font:minor-fareast;
mso-hansi-font-family:Calibri;
mso-hansi-theme-font:minor-latin;}
The following link is to Xen XCP FAQ explaining XCP cloud functionality being rolled into XenServer 6.2 and all XenServer releases going forward: http://www.xenproject.org/component/content/article/100-misc/148-xcp-and-xenserver-faq.html
As I mentioned if you will be interacting with XenCenter at all to talk to your host servers upgrade to the XenCenter version that corresponds to your latest XenServer release which is in this case of course 6.2 I made some changes on my slave host, the first system I updated in my 2 node cluster that I should not have due to the older version of XenCenter not to being able to properly render certain properties of my newly updated XenServer and giving incorrect information.
Normal
0
false
false
false
EN-US
X-NONE
X-NONE
MicrosoftInternetExplorer4
/* Style Definitions */
table.MsoNormalTable
{mso-style-name:”Table Normal”;
mso-tstyle-rowband-size:0;
mso-tstyle-colband-size:0;
mso-style-noshow:yes;
mso-style-priority:99;
mso-style-qformat:yes;
mso-style-parent:””;
mso-padding-alt:0in 5.4pt 0in 5.4pt;
mso-para-margin-top:0in;
mso-para-margin-right:0in;
mso-para-margin-bottom:10.0pt;
mso-para-margin-left:0in;
line-height:115%;
mso-pagination:widow-orphan;
font-size:11.0pt;
font-family:”Calibri”,”sans-serif”;
mso-ascii-font-family:Calibri;
mso-ascii-theme-font:minor-latin;
mso-fareast-font-family:”Times New Roman”;
mso-fareast-theme-font:minor-fareast;
mso-hansi-font-family:Calibri;
mso-hansi-theme-font:minor-latin;}
The way I found out about the XenCenter compatibility issue was when I tried to place my slave hosts system back into the shared pool after removing the slave to correct network interfaces that had disappeared. XenCenter told me it could not place my slave back into the pool with the master since they were different versions. This is when I updated XenCenter and then the master. The master server in the pool updated without any of the issues I had with the slave.
You can get the new XenCenter version by pointing to the newly upgraded XenServer host in a browser window:
Normal
0
false
false
false
EN-US
X-NONE
X-NONE
MicrosoftInternetExplorer4
/* Style Definitions */
table.MsoNormalTable
{mso-style-name:”Table Normal”;
mso-tstyle-rowband-size:0;
mso-tstyle-colband-size:0;
mso-style-noshow:yes;
mso-style-priority:99;
mso-style-qformat:yes;
mso-style-parent:””;
mso-padding-alt:0in 5.4pt 0in 5.4pt;
mso-para-margin-top:0in;
mso-para-margin-right:0in;
mso-para-margin-bottom:10.0pt;
mso-para-margin-left:0in;
line-height:115%;
mso-pagination:widow-orphan;
font-size:11.0pt;
font-family:”Calibri”,”sans-serif”;
mso-ascii-font-family:Calibri;
mso-ascii-theme-font:minor-latin;
mso-fareast-font-family:”Times New Roman”;
mso-fareast-theme-font:minor-fareast;
mso-hansi-font-family:Calibri;
mso-hansi-theme-font:minor-latin;}
The next issue I had to deal with was reconfiguring the network information I had lost when trying to get the slave recognized in the older version of XenCenter while in the pool with the master who was still @ XCP 6.1 at this time. I placed the proper NIC configurations on the network cards and then placed the slave back in the pools with the master.
Normal
0
false
false
false
EN-US
X-NONE
X-NONE
MicrosoftInternetExplorer4
/* Style Definitions */
table.MsoNormalTable
{mso-style-name:”Table Normal”;
mso-tstyle-rowband-size:0;
mso-tstyle-colband-size:0;
mso-style-noshow:yes;
mso-style-priority:99;
mso-style-qformat:yes;
mso-style-parent:””;
mso-padding-alt:0in 5.4pt 0in 5.4pt;
mso-para-margin-top:0in;
mso-para-margin-right:0in;
mso-para-margin-bottom:10.0pt;
mso-para-margin-left:0in;
line-height:115%;
mso-pagination:widow-orphan;
font-size:11.0pt;
font-family:”Calibri”,”sans-serif”;
mso-ascii-font-family:Calibri;
mso-ascii-theme-font:minor-latin;
mso-hansi-font-family:Calibri;
mso-hansi-theme-font:minor-latin;}
Now I still could not get the slave to recognize my shared iSCSI storage once back in the pool while the master had no issues seeing the storage. Well to make a long discovery into a shorter story I had to destroy the previous shared pool plugin that existed previously on my system. I really thought that I had done that and tried to get the shared iSCSI storage plugged in several times. But there was a second plugin I had ignored but was pointing to the same shared storage that was causing me issues.
What I failed to realize was that while I had added the slave back into the pool with the same alphanumeric name I did not say “use the existing XenCenter name” therefore it added the slave XenServer host in with the same name but XenCenter and somehow the XenServer hosts still had the old information and treated it as a separate name. I actually had to delete the old name from the shared pool. There was also a leftover “Xen storage plugin configuration” I kept overlooking though it showed in the output I had ignored it since I knew it should have only one “Xen Storage Plugin configuration” I dealt with the most recent one.
To explain better in a more truncated way this is what I ended up doing to get the slave to see the iSCSI shared storage again on my newly upgraded XenServer 6.2 environment after I had corrupted the network configuration and storage repository:
– List shared iSCSI storage
o # xe sr-list type=lvmoiscsi
– Show the plugins if any associated with this iSCSI storage (this command showed output of two plugins one which I ignored and cause issues):
o # xe pbd-list sr-uuid=<SR UUID>
– Next unplug shared storage:
o xe pbd-unplug uuid=<PBD UUID>
– Next destroy plugin
o # xe pbd-destroy uuid=<PBD UUID>
o # xe pbd-destroy uuid=<PBD UUID>
– Then from XenCenter GUI I “detached” the shared storage and “forgot” the storage and then reattached and everything worked out just fine with both servers in the pool.
That is the story and its quite long but the actual work was much longer! The story to summarize when updating to a newer version fist work with the newer version tools since they are backwards compatible in most cases. Also never neglect your output it’s there for a reason!
