{"id":1883,"date":"2021-04-09T20:15:48","date_gmt":"2021-04-10T03:15:48","guid":{"rendered":"https:\/\/dhoytt.com\/mainblog\/?p=1883"},"modified":"2021-12-19T22:17:50","modified_gmt":"2021-12-20T06:17:50","slug":"lost-networking-xcp-ng-hosts-then-recovered","status":"publish","type":"post","link":"https:\/\/dhoytt.com\/mainblog\/2021\/04\/lost-networking-xcp-ng-hosts-then-recovered\/","title":{"rendered":"Lost Networking XCP-ng Hosts Then Recovered"},"content":{"rendered":"\n<p>Today I lost the XCP-ng networking to 3 XCP-ng hosts in a pool of 4 hosts. I logged in to check XCP-ng Center from a Windows workstation and couldn\u2019t connect to the pool. I logged into the master of the pool with putty and had no issues. Yet when I ran commands against the pool it would say \u201clost connection to the server\u201d. The logs on that workstation from XCP-ng Center said \u201cThe server that you are talking to is a slave\u201d during that time period.<\/p>\n\n\n\n<p>I thought maybe this was an issue with XCP-ng center and went to a laptop I had and it exhibited the same issues even after removing the configuration and trying to import the server. Then just as on the workstation it would note this server belongs to a pool would you like to import the pool master, I said yes and it wouldn\u2019t import the pool or the system thus not completing the operation.<\/p>\n\n\n\n<p>I performed a xe-toolstack-restart and still had the same issues. Finally logged into the physical console and all of the network information was gone from the console, yet I was still in the system via ssh with no issue, connected and able to perform commands.<\/p>\n\n\n\n<figure class=\"wp-block-image size-large\"><a href=\"https:\/\/i0.wp.com\/dhoytt.com\/mainblog\/wp-content\/uploads\/sites\/3\/2021\/04\/IMG_6553-scaled.gif?ssl=1\" rel=\"lightbox[1883]\"><img data-recalc-dims=\"1\" loading=\"lazy\" decoding=\"async\" width=\"620\" height=\"465\" data-attachment-id=\"1884\" data-permalink=\"https:\/\/dhoytt.com\/mainblog\/2021\/04\/lost-networking-xcp-ng-hosts-then-recovered\/img_6553\/\" data-orig-file=\"https:\/\/i0.wp.com\/dhoytt.com\/mainblog\/wp-content\/uploads\/sites\/3\/2021\/04\/IMG_6553-scaled.gif?fit=2560%2C1920&amp;ssl=1\" data-orig-size=\"2560,1920\" data-comments-opened=\"1\" data-image-meta=\"{&quot;aperture&quot;:&quot;0&quot;,&quot;credit&quot;:&quot;&quot;,&quot;camera&quot;:&quot;&quot;,&quot;caption&quot;:&quot;&quot;,&quot;created_timestamp&quot;:&quot;0&quot;,&quot;copyright&quot;:&quot;&quot;,&quot;focal_length&quot;:&quot;0&quot;,&quot;iso&quot;:&quot;0&quot;,&quot;shutter_speed&quot;:&quot;0&quot;,&quot;title&quot;:&quot;&quot;,&quot;orientation&quot;:&quot;0&quot;}\" data-image-title=\"IMG_6553\" data-image-description=\"\" data-image-caption=\"\" data-medium-file=\"https:\/\/i0.wp.com\/dhoytt.com\/mainblog\/wp-content\/uploads\/sites\/3\/2021\/04\/IMG_6553-scaled.gif?fit=300%2C225&amp;ssl=1\" data-large-file=\"https:\/\/i0.wp.com\/dhoytt.com\/mainblog\/wp-content\/uploads\/sites\/3\/2021\/04\/IMG_6553-scaled.gif?fit=620%2C465&amp;ssl=1\" src=\"https:\/\/i0.wp.com\/dhoytt.com\/mainblog\/wp-content\/uploads\/sites\/3\/2021\/04\/IMG_6553.gif?resize=620%2C465&#038;ssl=1\" alt=\"\" class=\"wp-image-1884\" srcset=\"https:\/\/i0.wp.com\/dhoytt.com\/mainblog\/wp-content\/uploads\/sites\/3\/2021\/04\/IMG_6553-scaled.gif?resize=1024%2C768&amp;ssl=1 1024w, https:\/\/i0.wp.com\/dhoytt.com\/mainblog\/wp-content\/uploads\/sites\/3\/2021\/04\/IMG_6553-scaled.gif?resize=300%2C225&amp;ssl=1 300w, https:\/\/i0.wp.com\/dhoytt.com\/mainblog\/wp-content\/uploads\/sites\/3\/2021\/04\/IMG_6553-scaled.gif?resize=768%2C576&amp;ssl=1 768w, https:\/\/i0.wp.com\/dhoytt.com\/mainblog\/wp-content\/uploads\/sites\/3\/2021\/04\/IMG_6553-scaled.gif?resize=1536%2C1152&amp;ssl=1 1536w, https:\/\/i0.wp.com\/dhoytt.com\/mainblog\/wp-content\/uploads\/sites\/3\/2021\/04\/IMG_6553-scaled.gif?resize=2048%2C1536&amp;ssl=1 2048w, https:\/\/i0.wp.com\/dhoytt.com\/mainblog\/wp-content\/uploads\/sites\/3\/2021\/04\/IMG_6553-scaled.gif?resize=640%2C480&amp;ssl=1 640w, https:\/\/i0.wp.com\/dhoytt.com\/mainblog\/wp-content\/uploads\/sites\/3\/2021\/04\/IMG_6553-scaled.gif?w=1240&amp;ssl=1 1240w, https:\/\/i0.wp.com\/dhoytt.com\/mainblog\/wp-content\/uploads\/sites\/3\/2021\/04\/IMG_6553-scaled.gif?w=1860&amp;ssl=1 1860w\" sizes=\"auto, (max-width: 620px) 100vw, 620px\" \/><\/a><\/figure>\n\n\n\n<p>This whole time all of my VM\u2019s on this system, the pool master was up and working just fine. The other two systems didn\u2019t have VM\u2019s on them at this time as I was transitioning away from them.<\/p>\n\n\n\n<p>On the physical console I went into the \u201cLocal Command Shell\u201d and ifconfig showed all my physical interfaces and I could ping out on any of them. After a reboot I could still ssh into the system but still no management interfaces. I tried several times performing xe-toolstack-restart that\u2019s when I started seeing physical interfaces in the local shell disappear. After trying xe-toolstack-restart a few times only my local loopback was left!<\/p>\n\n\n\n<p>Now of course my VM\u2019s on this pool master system are down now and not operating.<\/p>\n\n\n\n<p>Finally on the one XCP-ng host that was still up I decided to login via ssh and made that the pool master with \u201cxe -pool-emergency-transition-to-master\u201d and that worked. I could now connect to the pool with the one host in it with XCP-ng Center.<\/p>\n\n\n\n<p>Then I wanted to get the VM\u2019s up on the new pool master and did a \u201cxe host-list\u201d \u201cvm-list resident-on=&lt;UUID-pool-master&gt;\u201d then \u201cxe vm-reset-powerstate resident-on=&lt;UUID-pool-master&gt;\u201d and I could now see the VM\u2019s I wanted to start on the new pool master but couldn\u2019t because the one host system left operating with networking was in maintenance mode.<\/p>\n\n\n\n<p>I finally re-installed the original master with an XCp-ng 8.1 iso and then got the networking back and had it rejoin the pool as a slave and everything is starting to look good again and the new pool master is out of maintenance mode. I started my VM\u2019s and they are working. I am concerned that I lost the network configuration on 3 of my 4 XCP-ng hosts though.<\/p>\n\n\n\n<p>The only thing I can think of is that about a week and half ago I transitioned from an older pool master to a new pool master using this command \u201cxe pool-designate-new-master host-uuid =&lt;host-uuid>\u201d then since I was transitioning to new more updated hardware powered down 2 hosts leaving the 2 hosts I wanted to keep. I was planning to remove the old hosts from the pool this weekend. These were my only changes and there were no issues I observed until this morning. These servers have been the part of the same pool for months now.<\/p>\n\n\n\n<p>The good thing is that once I decided to just re-install the former XCP-ng pool master, recreate the networking (thank goodness I didn\u2019t have complex networking or VLAN scheme) everything worked well. Having the storage for my virtual systems on FreeNAS storage utilizing ZFS gives you great reliability.<\/p>\n\n\n\n<p>Earlier this week my dual 10GBE Intel 10 Gigabit XF SR Server Adapter went dead on my Windows workstation which has made it really a strange week in my computer lab. Thank goodness  I had a spare Intel 10 Gigabit XF SR Server Adapter and I was able to recover my virtual environment!<\/p>\n","protected":false},"excerpt":{"rendered":"<p>Today I lost the XCP-ng networking to 3 XCP-ng hosts in a pool of 4 hosts. I logged in to check XCP-ng Center from a Windows workstation and couldn\u2019t connect to the pool. I logged into the master of the &hellip; <a href=\"https:\/\/dhoytt.com\/mainblog\/2021\/04\/lost-networking-xcp-ng-hosts-then-recovered\/\">Continue reading <span class=\"meta-nav\">&rarr;<\/span><\/a><\/p>\n","protected":false},"author":1,"featured_media":1884,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"wprm-recipe-roundup-name":"","wprm-recipe-roundup-description":"","_jetpack_memberships_contains_paid_content":false,"footnotes":""},"categories":[5,6],"tags":[945,558,652,413,713,716,557],"class_list":["post-1883","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-home-projects","category-whole-site-development","tag-intel-10-gigabit-xf-sr-server-adapter","tag-network","tag-ssh","tag-virtualization","tag-xcp-ng","tag-xcp-ng-center","tag-xenserver-shared-pool"],"jetpack_featured_media_url":"https:\/\/i0.wp.com\/dhoytt.com\/mainblog\/wp-content\/uploads\/sites\/3\/2021\/04\/IMG_6553-scaled.gif?fit=2560%2C1920&ssl=1","jetpack_sharing_enabled":true,"jetpack_likes_enabled":true,"_links":{"self":[{"href":"https:\/\/dhoytt.com\/mainblog\/wp-json\/wp\/v2\/posts\/1883","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/dhoytt.com\/mainblog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/dhoytt.com\/mainblog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/dhoytt.com\/mainblog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/dhoytt.com\/mainblog\/wp-json\/wp\/v2\/comments?post=1883"}],"version-history":[{"count":3,"href":"https:\/\/dhoytt.com\/mainblog\/wp-json\/wp\/v2\/posts\/1883\/revisions"}],"predecessor-version":[{"id":1889,"href":"https:\/\/dhoytt.com\/mainblog\/wp-json\/wp\/v2\/posts\/1883\/revisions\/1889"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/dhoytt.com\/mainblog\/wp-json\/wp\/v2\/media\/1884"}],"wp:attachment":[{"href":"https:\/\/dhoytt.com\/mainblog\/wp-json\/wp\/v2\/media?parent=1883"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/dhoytt.com\/mainblog\/wp-json\/wp\/v2\/categories?post=1883"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/dhoytt.com\/mainblog\/wp-json\/wp\/v2\/tags?post=1883"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}