April 9, 2021
by dhoytt
Comments Off on Lost Networking XCP-ng Hosts Then Recovered

Lost Networking XCP-ng Hosts Then Recovered

Today I lost the XCP-ng networking to 3 XCP-ng hosts in a pool of 4 hosts. I logged in to check XCP-ng Center from a Windows workstation and couldn’t connect to the pool. I logged into the master of the pool with putty and had no issues. Yet when I ran commands against the pool it would say “lost connection to the server”. The logs on that workstation from XCP-ng Center said “The server that you are talking to is a slave” during that time period.

I thought maybe this was an issue with XCP-ng center and went to a laptop I had and it exhibited the same issues even after removing the configuration and trying to import the server. Then just as on the workstation it would note this server belongs to a pool would you like to import the pool master, I said yes and it wouldn’t import the pool or the system thus not completing the operation.

I performed a xe-toolstack-restart and still had the same issues. Finally logged into the physical console and all of the network information was gone from the console, yet I was still in the system via ssh with no issue, connected and able to perform commands.

This whole time all of my VM’s on this system, the pool master was up and working just fine. The other two systems didn’t have VM’s on them at this time as I was transitioning away from them.

On the physical console I went into the “Local Command Shell” and ifconfig showed all my physical interfaces and I could ping out on any of them. After a reboot I could still ssh into the system but still no management interfaces. I tried several times performing xe-toolstack-restart that’s when I started seeing physical interfaces in the local shell disappear. After trying xe-toolstack-restart a few times only my local loopback was left!

Now of course my VM’s on this pool master system are down now and not operating.

Finally on the one XCP-ng host that was still up I decided to login via ssh and made that the pool master with “xe -pool-emergency-transition-to-master” and that worked. I could now connect to the pool with the one host in it with XCP-ng Center.

Then I wanted to get the VM’s up on the new pool master and did a “xe host-list” “vm-list resident-on=<UUID-pool-master>” then “xe vm-reset-powerstate resident-on=<UUID-pool-master>” and I could now see the VM’s I wanted to start on the new pool master but couldn’t because the one host system left operating with networking was in maintenance mode.

I finally re-installed the original master with an XCp-ng 8.1 iso and then got the networking back and had it rejoin the pool as a slave and everything is starting to look good again and the new pool master is out of maintenance mode. I started my VM’s and they are working. I am concerned that I lost the network configuration on 3 of my 4 XCP-ng hosts though.

The only thing I can think of is that about a week and half ago I transitioned from an older pool master to a new pool master using this command “xe pool-designate-new-master host-uuid =<host-uuid>” then since I was transitioning to new more updated hardware powered down 2 hosts leaving the 2 hosts I wanted to keep. I was planning to remove the old hosts from the pool this weekend. These were my only changes and there were no issues I observed until this morning. These servers have been the part of the same pool for months now.

The good thing is that once I decided to just re-install the former XCP-ng pool master, recreate the networking (thank goodness I didn’t have complex networking or VLAN scheme) everything worked well. Having the storage for my virtual systems on FreeNAS storage utilizing ZFS gives you great reliability.

Earlier this week my dual 10GBE Intel 10 Gigabit XF SR Server Adapter went dead on my Windows workstation which has made it really a strange week in my computer lab. Thank goodness I had a spare Intel 10 Gigabit XF SR Server Adapter and I was able to recover my virtual environment!

March 30, 2021
by dhoytt
Comments Off on 2021 Vegetable Garden Comes out of Cocoon

2021 Vegetable Garden Comes out of Cocoon

Off to a later start than I would have liked for my vegetable garden but hey a lot’s going on right now but it has started! Friday I took stock of what was needed and formulated a plan, Friday night right after work checked my rototiller and it was pouring gas out after putting some gas in it out of the carburetor it so I called my small engine guy Paul, sent a short video clip and photo of the issue and he had me take it to him. The next morning he had the gaskets replaced, a new bulb and idle spring and the rototiller was ready to go.

I picked up the rototiller Saturday, got some top soil at my garden spot called Longers, they didn’t have mulch unfortunately. I then set out pulling the grass, weeds and mint that had overtaken the garden by hand, getting as much as I could by the root and filling my garden waste receptacle. I then took my drip system out of the garden and started my newly repaired rototiller and hit both sides of the garden.

I had a zoom meeting event for the memorial of Christs death and then went and did some on call work and then replace d the heatsinks on one of my Blade Servers.

Sunday I slept in a bit relaxed went got some mulch, pulled a bunch of weeds and rototilled some more to chop up and roots or vegetation left left and raked out what I could and dug down really deep to get some roots and pull roots along the edges of my pavers that contain my garden. I then took the wheelbarrow and moved the yard of top soil from my truck to my gardens and moved the mulch in place.

Monday after work I worked the top soil and mulch into the garden making sure it was thoroughly mixed down to about 1.5 feet at least and then did some rough leveling with the garden rake.

Tonight I did more leveling of the dirt put some row ins hooked up my drip system tested and then put in the handful of vegetables I had obtained. I put in mature beef eater and husky cherry tomatoes, pint sized Early girl tomato plant, small eggplant, purple and sweet basil, tarragon, habanero and jalapeno peppers, zucchini squash, Swiss chard. I left in place 2 really nice looking rosemary plants, my Thai basil bush pared down and some Swiss chard. Almost forgot I have my ever present the past 8 years or so since I have forgotten when I planted then my grape and raspberry vines in the back of the garden on the west side of the yard.

Here are the pictures with my progress until now the past few days.

Overgrown east side of vegetable garden start 2021.
Overgrown west side of vegetable garden start 2021.
After pulling weeds west side of vegetable garden start 2021.
After pulling weeds east side of garden start 2021.
Pepper plant I may try to replant later in 2021.
First hit of rototiller west side of 2021 vegetable garden as well as drip system in foreground plus Thai basil bush front of garden and grape vines and raspberry vines in back of the garden. You can barely see some rosemary to the right.
First hit of rototiller east side of 2021 vegetable garden. Anoter rosemary plant to the left and Swiss chard to the left.
Here’s a look after the top soil and mulch was mixed in with the rototiller on west side of the garden.
Here’s a look after the top soil and mulch was mixed in with the rototiller on east side of the garden.
Tonight after a bit more leveling of soil creating rows, putting in drip system and vegetable on the west side of the 2021 vegetable garden.
Tonight after a bit more leveling of soil creating rows, putting in drip system and vegetable on the east side of the 2021 vegetable garden

I still have some cleanup to do and then I will power wash my concrete and be set for a while as I incorporate more vegetables including some seeds.

March 27, 2021
by dhoytt
Comments Off on The Correct Heat Sink for Bl460c Gen8 Blade Server Finally

The Correct Heat Sink for Bl460c Gen8 Blade Server Finally

I have had this ongoing saga of not being able to find the correct heat sink for CPU#2 for my BL460c gen 8 blade server for a while. The original reseller used an integrator that came in at a lower price than I could build it myself for so I let them build it. I immediately had to make corrections to for NIC’s and memory. There was also some negotiation between the reseller and integrator who tried to up the price though the specs I provided them were clear and I was sticking to that price or would build it myself. We tried several heatsinks and none of them fit except the last they sent but it fit with the heat sink fins sideways which blocked the air flow but let the heat sink actually contact the CPU due to the t-shaped design of the bottom of the heat sink.

I’m usually pretty good a retrofitting and redesign but the little changes I did to make things fit didn’t work as well as I hoped and every time I upped the load on a VM that sat on the BL460c the whole chassis would rev up like jet engines taking off until I reduced the CPU load or moved the VM that had continually high CPU resource usage to my backup Dell r710 servers which didn’t have the same issues. I finally looked at a different set of part numbers for this heat sink and found it was for the Gen9 servers!

I tried a few resellers on Ebay and though they said their heatsinks were for CPU2 on the bl460c gen 8 blade servers they ended up being for the heat sink for the second CPU on a rack mount HPE gen8 which are physically too large by about1/8th an inch one each side. Now I recall the integrator passing a message that they would need to depopulate memory for the second heat sink. Turns out they were having the same issues I had locating the proper heat sink part numbers and didn’t realize their error.

Quickspecs manuals for the BL460c gen 8 blade server wasn’t as much help as normal for this part number as the heat sinks were not listed. Finally I went to Amazon found a vendor with the proper part number and description sent for it and it took forever to get here but it worked! I ordered another heat sink for my other bl460c gen8 blade server and it came within 2 days this time which happened to be today so I installed it around 150 minutes ago, then I started putting a load which would increase CPU resource usage that normally would have the entire chassis screaming at such a high decibel level of a jet plane and the chassis is not screaming for mercy with all of the fans on high! What a relief.

The time to go through the parts system they have for these systems if not listed in the systems Quicspecs which are definitive part manuals made this a nightmare for everyone as there are 4 or five part numbers that seem like they are correct, then with slow shipping with Covid-19 to get the parts and returning them it was a time wasting draining process. The correct part number ended up being 670032-001. The correct description was important too, I had this combination of correct part number and description with another vendor and still got the wrong part. Very important to get this right so the heat sink fins are oriented for proper air flow in this tight enclosure, the heat sink is properly connected and transferring heat for dissipation to the heat sink and height is correct as there is a small controller board that fits over the top of this heat sink in position number 2.

Every piece of hardware, firmware, application and OS usage matters in these tightly integrated systems and you have to keep searching grinding and matching to get a cohesive and redundant properly working solution.

Below are some earlier pictures that illustrate how the heat sinks air flow should go and the bottom picture shows the t-shaped bottom of the heat sink. Note these are also the wrong heat sinks. I was too excited to get this in to take pictures of the correct heat sinks being put in place.

March 25, 2021
by dhoytt
Comments Off on Scroll Through Multi-Genre and Era Song Drop In Workout Mix

Scroll Through Multi-Genre and Era Song Drop In Workout Mix

That’s right I did a quick scroll down through my songs that are not separated via era or genre dropped in some titles I would like to hear and threw them into my workout mix and I’m sharing with you as always!

I went with some medium to up tempo songs with various beats and of course multiple genres then went to more chilled to some slow jams.

I went far back like Aretha Franklin, Otis Redding, The Supremes, KC and the Sunshine Band, to current like Ariana Grande, Doja Cat, Roddy Ricch sneak in some Ini Kamoze, Culcha Candela, Bootsy, Lana Del Rey and its a true mix Snakice House of Beats Style.

March 21, 2021
by dhoytt
Comments Off on Boat Cleaned Going to New Home

Boat Cleaned Going to New Home

Reinell Blue Water 2004 and 1998 4×4 Dodge Ram 8 cylinder.

Got email notice my boat indoor storage place was closing in 30 days mid-afternoon yesterday , right before getting into some projects around the house so I went and found a new place for the boat within the next couple of hours. There’s a few boat places within a nice small radius from the house but also very close to where the American and Sacramento rivers come together at Discovery Park.

Since it has been a very long time since taking my boat out it was caked with dust and the tires of the boat trailer were low. I needed to clean the boat up and put air in the tires. The guys at the place I was leaving helped me fill the tires of my trailer using their air compressor plus used the hose to wash and rinse the boat down, I then used my bucket and some rags plus cleaner and portable vacuum to clean my boat out. Also check out my cool 1998 Dodge Ram I have had for 23 years and its still going strong!

This really made me realize how much I miss and want to be out on the water fishing.

March 14, 2021
by dhoytt
Comments Off on Big Cable Management Start

Big Cable Management Start

Did a major start on getting some cable management done in the home computer room. I have been wanting to start some cable management in the computer room for a very long time but wanted to get the hardware situation handled first.
Now that the C7000 blade chassis is pretty much set with my Virtual Connect networks to external LAN switches it was a good time to start the cable management process. There’s still much to do but I wanted to make sure I had a workable foundation.

I gathered some cable panduit and old speaker\cable mounts I have literally been packing around for almost 20 years. The speaker\cable mounts are in original boxes from a pallet I bought at an auction long ago. I am so glad didn’t sell all of them now!

I moved cables on one network then moved to the next network starting with my busiest network. I monitored and checked connectivity after each cable movement noting which systems would be affected each time before moving the cable, minimizing troubleshooting later.

I migrated my VM’s to my main servers on the bl460c Gen8 blades moved the cables connected to my backup hypervisors and then migrated the VM’s to my backup systems once I started moving the 10gbe fiber cables.
I moved the rj45 gigabit connections first and then my 10gbe connections. The gigabit connections are mainly my IPMI connections on my FreeNAS storage and backup XCP-NG hypervisors.

This was like an all day untangling fishing line type of deal that needed to be done. Not the fun part of computers but it will make tracing cables for troubleshooting, moving equipment around faster and easier. Cleaning up the back helps keep cables out of the way in case I need to replace modules in the HPE C7000 blade chassis.

I did a final inspection of my networking logging into the Virtual Connect, my managed switches and made sure I could access my applications like websites, music streaming site, virtual systems, CCTV, Plex media servers, XCP-NG systems, FreeNAS etc..

Here’s a before and after in the front. Still not pretty but much easier to manage. I also have a bit of multimedia cables in the mix.

The bottom of the rack is the HPE C7000, then Liebert UPS’, then my 2 FreeNAS storage servers using Supermicro 2u storage chassis’s, then backup Dell XCP-NG hypervisors, then Trendnet 16 port KVM, then my 3 Mikrotik switches, then Linksys and Netgear gigabit switches, finally the cable panduits and cable mounts.