All in one night I took my Nagios server from running Fedora 16 to CENTOS 6.5, Updated XenCenter to the latest version 6.2.2 and updated XenServer to 6.2.0 SP1 and solved and issue where my iSCSI target was failing in FreeNAS. I had only planned to get Nagios up and going on the CENTOS server but upon completion of getting the Nagios server configured with the basic systems and monitoring I noticed this nice red X on one of my iSCSI targets in XenServer.
First the Nagios server I had copied my configuration files to my NFS drives prior to blowing away the Fedora 16 OS, so once I had CENTOS 6.5 installed I connected back to the NFS share and made the appropriate changes to my configuration files to point to my new servers. Now that my web server and gaming server plus a few other systems are now in my XenServer virtual environment with different names I just had to go and make the changes in the Nagios configuration files.
I was about to happily grab a book and retire to my bedroom when the iSCSI issue on my XenServer came to my attention. I looked and it said it was missing 2 of my multipathing connections. I checked the connectivity and that was fine as it should have been otherwise several other storage volumes should have shown the same issue.
Well logging into my second FreeNAS node I noticed that it was throwing out some errors and I could not login to do anything. I captured some of the errors with this really catching my eye: “istgt_iscsi.c:5551:worker:***ERROR***pthread_create() failed”. I found a number of people having this issue in the FreeNAS forums with memory issues caused by ZFS volumes. Last time I checked this specific node having the issues was carrying less of a load and not using near as much memory as the other node not giving errors.
I finally decided to shutdown my environment and troubleshoot that FreeNAS node. What I found was that there were some CPU errors with the codes of “8121” (temp trip, a minor error) & “84ff (CPU system log full a major error)”. I went into the bios of the system and cleared the error logs and the system then that FreeNAS node came up just fine. When I placed the loads back on both FreeNAS systems they both are once again working like troopers. I do need to tackle temperature issues in my computer room though.