{"id":888,"date":"2017-05-24T23:49:34","date_gmt":"2017-05-25T06:49:34","guid":{"rendered":"http:\/\/dhoytt.com\/mainblog\/?p=888"},"modified":"2017-06-10T08:49:28","modified_gmt":"2017-06-10T15:49:28","slug":"recovered-from-catastrophic-xenserver-failure-that-made-me-love-linux-even-more","status":"publish","type":"post","link":"http:\/\/dhoytt.com\/mainblog\/2017\/05\/recovered-from-catastrophic-xenserver-failure-that-made-me-love-linux-even-more\/","title":{"rendered":"Recovered from Catastrophic XenServer failure that made me Love Linux Even More"},"content":{"rendered":"<p>I now know why I really love <a class=\"zem_slink\" href=\"http:\/\/en.wikipedia.org\/wiki\/Linux\" title=\"Linux\" rel=\"wikipedia noopener noreferrer\" target=\"_blank\">Linux<\/a> right now after totally losing my XenServer configuration due to some overheating in my home data center today. I reinstalled my XenServer server reattached storage to my XenServer server and then reattached the luns I had created with unique names to my VM with the former boot drive in position 0 in the VM, and then my home VG drive in position 1 then my game drive as position 2 and the server booted <a class=\"zem_slink\" href=\"http:\/\/www.centos.org\/\" title=\"CentOS\" rel=\"homepage noopener noreferrer\" target=\"_blank\">Centos<\/a> 7 just fine! I had to reset the network interfaces on XenServer and VM plus sit through a Selinux re-label on boot and then voila I\u2019m up and running again<\/p>\n<p>&nbsp;<\/p>\n<p>This past week I have actually been dealing with overheating since it started dramatically warming up here in the Sacramento valley. I have been able to log in remotely and get my XenServer back up until today when even the <a class=\"zem_slink\" href=\"http:\/\/en.wikipedia.org\/wiki\/Intelligent_Platform_Management_Interface\" title=\"Intelligent Platform Management Interface\" rel=\"wikipedia noopener noreferrer\" target=\"_blank\">IPMI<\/a> connection went dead on me I knew the server was in trouble then!<\/p>\n<p>When I got home I still thought I would be able to clear some logs in the bios and crash dumps in XenServer and things would be back up in about 30 minutes. Not today, I had been neglecting these babies too long!<\/p>\n<p>I could not even boot off the safe mode kernel! For XenServer I could only get to the CLI in single-user and could not get past that. The system was complaining about not being able to mount a filesystem \u201c\/var\/xen\/xc-install\u201d. The complete error was: \u201cFailed to mount \/var\/xen\/xc-install\u201d.<\/p>\n<p>I looked in \/etc\/fstab and sure enough it had the line:<\/p>\n<ul>\n<li>\/opt\/xensource\/packages\/iso\/XenCenter.iso \/var\/xen\/xc-install\u00a0\u00a0 iso9660\u00a0\u00a0 loop\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 ,ro\u00a0\u00a0 0\u00a0 0<\/li>\n<\/ul>\n<p>Well I could see the file \/opt\/xensource\/packages\/iso\/XenCenter.iso\u00a0\u00a0 and the <a class=\"zem_slink\" href=\"http:\/\/en.wikipedia.org\/wiki\/Mount_%28computing%29\" title=\"Mount (computing)\" rel=\"wikipedia noopener noreferrer\" target=\"_blank\">mount point<\/a> \u201c\/var\/xen\/xc-install \u201cbut when I did \u201cmount \u2013a\u201d it said \u201cfilesystem type iso9660 unknown! I knew I had some module or kernel issues then or something else was royally corrupted. I played around with the basic XenServer commands then realized the cause was lost after I commented that line up got up and couldn&#8217;t get an IP address to stick and I didn\u2019t want to go forward with some kluge of a system.<\/p>\n<p>I did try and upgrade with a XenServer 7.0 DVD but that produced an error message I didn\u2019t bother to even notate something about a split.\u00a0 So I weighed my options and went for the clean install to my XenServer disk. I verified all my hardware was solid and in good shape even blew off some dust and reseated memory and storage controller cards in the XenServer.<\/p>\n<p>Then booted up my <a class=\"zem_slink\" href=\"http:\/\/www.freenas.org\/\" title=\"FreeNAS\" rel=\"homepage noopener noreferrer\" target=\"_blank\">FreeNAS<\/a> primary server reattached the <a class=\"zem_slink\" href=\"http:\/\/en.wikipedia.org\/wiki\/Network_File_System\" title=\"Network File System\" rel=\"wikipedia noopener noreferrer\" target=\"_blank\">NFS<\/a> share from there that has all my ISO\u2019s for installing OS\u2019 and was thinking I would boot off of the Centos 7 ISO and then \u201cdd\u201d all my boot drive then my home drive then may game drives to the new luns.<\/p>\n<p>As I reattached my prior drives to the new VM I decided that if I the Linux Rescue CD could detect my old environment and I received a proper path like \/dev\/xvdN \u00a0then I would try to position my VM&#8217;s\u00a0old luns properly\u00a0 and boot my VM up Well that&#8217;s what I did \u00a0and it worked as \u00a0I explained above!<\/p>\n<p>Maybe my Windows VM\u2019s will come up just fine with this method as well. This however means that I will now bring both of my XenServers up to run once again in an HA ( Highly Available) configuration with my FreeNAS as storage instead of using local storage I had been using so I could cut down on my energy costs. The use of local storage seems\u00a0dead now and it looks like I may have to bite the bullet on Solar since I haven\u2019t found a place I want to move to yet! Ahh well the complications of life at least now I will not have to login and bounce my systems with IPMI or \u201c<a class=\"zem_slink\" href=\"http:\/\/en.wikipedia.org\/wiki\/Wake-on-LAN\" title=\"Wake-on-LAN\" rel=\"wikipedia noopener noreferrer\" target=\"_blank\">wake on LAN<\/a>\u201d.<\/p>\n<p>Well I have a lot of work ahead of me with these issues later this week and beyond so excuse the typos and grammatical errors for now I\u2019m going to bed, I still get up pretty early!<\/p>\n<blockquote class=\"wp-embedded-content\" data-secret=\"WMYOvY96Rn\"><p><a href=\"https:\/\/www.cloudwards.net\/what-is-nas\/\">What is NAS? A Practical Guide<\/a><\/p><\/blockquote>\n<p><iframe loading=\"lazy\" class=\"wp-embedded-content\" sandbox=\"allow-scripts\" security=\"restricted\" style=\"position: absolute; visibility: hidden;\" title=\"&#8220;What is NAS? A Practical Guide&#8221; &#8212; Cloudwards\" src=\"https:\/\/www.cloudwards.net\/what-is-nas\/embed\/#?secret=jMiczvKtOG#?secret=WMYOvY96Rn\" data-secret=\"WMYOvY96Rn\" width=\"600\" height=\"338\" frameborder=\"0\" marginwidth=\"0\" marginheight=\"0\" scrolling=\"no\"><\/iframe><\/p>\n","protected":false},"excerpt":{"rendered":"<p>I now know why I really love Linux right now after totally losing my XenServer configuration due to some overheating in my home data center today. I reinstalled my XenServer server reattached storage to my XenServer server and then reattached &hellip; <a href=\"http:\/\/dhoytt.com\/mainblog\/2017\/05\/recovered-from-catastrophic-xenserver-failure-that-made-me-love-linux-even-more\/\">Continue reading <span class=\"meta-nav\">&rarr;<\/span><\/a><\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"wprm-recipe-roundup-name":"","wprm-recipe-roundup-description":"","_jetpack_memberships_contains_paid_content":false,"footnotes":""},"categories":[5,6],"tags":[501,571,389,573,499,572,569,574,370,393,575,576,411,413,570,386,384],"class_list":["post-888","post","type-post","status-publish","format-standard","hentry","category-home-projects","category-whole-site-development","tag-centos-7","tag-dvd","tag-freenas","tag-fstab","tag-ipmi","tag-iso9660","tag-linux-kernel","tag-linux-rescue-cd","tag-luns","tag-nfs","tag-selinux","tag-solar-energy","tag-virtual-server","tag-virtualization","tag-vm","tag-xencenter","tag-xenserver"],"jetpack_featured_media_url":"","jetpack_sharing_enabled":true,"jetpack_likes_enabled":true,"_links":{"self":[{"href":"http:\/\/dhoytt.com\/mainblog\/wp-json\/wp\/v2\/posts\/888","targetHints":{"allow":["GET"]}}],"collection":[{"href":"http:\/\/dhoytt.com\/mainblog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"http:\/\/dhoytt.com\/mainblog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"http:\/\/dhoytt.com\/mainblog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"http:\/\/dhoytt.com\/mainblog\/wp-json\/wp\/v2\/comments?post=888"}],"version-history":[{"count":5,"href":"http:\/\/dhoytt.com\/mainblog\/wp-json\/wp\/v2\/posts\/888\/revisions"}],"predecessor-version":[{"id":896,"href":"http:\/\/dhoytt.com\/mainblog\/wp-json\/wp\/v2\/posts\/888\/revisions\/896"}],"wp:attachment":[{"href":"http:\/\/dhoytt.com\/mainblog\/wp-json\/wp\/v2\/media?parent=888"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"http:\/\/dhoytt.com\/mainblog\/wp-json\/wp\/v2\/categories?post=888"},{"taxonomy":"post_tag","embeddable":true,"href":"http:\/\/dhoytt.com\/mainblog\/wp-json\/wp\/v2\/tags?post=888"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}