May 24, 2017
by dhoytt
0 comments

Recovered from Catastrophic XenServer failure that made me Love Linux Even More

I now know why I really love Linux right now after totally losing my XenServer configuration due to some overheating in my home data center today. I reinstalled my XenServer server reattached storage to my XenServer server and then reattached the luns I had created with unique names to my VM with the former boot drive in position 0 in the VM, and then my home VG drive in position 1 then my game drive as position 2 and the server booted Centos 7 just fine! I had to reset the network interfaces on XenServer and VM plus sit through a Selinux re-label on boot and then voila I’m up and running again

 

This past week I have actually been dealing with overheating since it started dramatically warming up here in the Sacramento valley. I have been able to log in remotely and get my XenServer back up until today when even the IPMI connection went dead on me I knew the server was in trouble then!

When I got home I still thought I would be able to clear some logs in the bios and crash dumps in XenServer and things would be back up in about 30 minutes. Not today, I had been neglecting these babies too long!

I could not even boot off the safe mode kernel! For XenServer I could only get to the CLI in single-user and could not get past that. The system was complaining about not being able to mount a filesystem “/var/xen/xc-install”. The complete error was: “Failed to mount /var/xen/xc-install”.

I looked in /etc/fstab and sure enough it had the line:

  • /opt/xensource/packages/iso/XenCenter.iso /var/xen/xc-install   iso9660   loop                                             ,ro   0  0

Well I could see the file /opt/xensource/packages/iso/XenCenter.iso   and the mount point “/var/xen/xc-install “but when I did “mount –a” it said “filesystem type iso9660 unknown! I knew I had some module or kernel issues then or something else was royally corrupted. I played around with the basic XenServer commands then realized the cause was lost after I commented that line up got up and couldn’t get an IP address to stick and I didn’t want to go forward with some kluge of a system.

I did try and upgrade with a XenServer 7.0 DVD but that produced an error message I didn’t bother to even notate something about a split.  So I weighed my options and went for the clean install to my XenServer disk. I verified all my hardware was solid and in good shape even blew off some dust and reseated memory and storage controller cards in the XenServer.

Then booted up my FreeNAS primary server reattached the NFS share from there that has all my ISO’s for installing OS’ and was thinking I would boot off of the Centos 7 ISO and then “dd” all my boot drive then my home drive then may game drives to the new luns.

As I reattached my prior drives to the new VM I decided that if I the Linux Rescue CD could detect my old environment and I received a proper path like /dev/xvdN  then I would try to position my VM’s old luns properly  and boot my VM up Well that’s what I did  and it worked as  I explained above!

Maybe my Windows VM’s will come up just fine with this method as well. This however means that I will now bring both of my XenServers up to run once again in an HA ( Highly Available) configuration with my FreeNAS as storage instead of using local storage I had been using so I could cut down on my energy costs. The use of local storage seems dead now and it looks like I may have to bite the bullet on Solar since I haven’t found a place I want to move to yet! Ahh well the complications of life at least now I will not have to login and bounce my systems with IPMI or “wake on LAN”.

Well I have a lot of work ahead of me with these issues later this week and beyond so excuse the typos and grammatical errors for now I’m going to bed, I still get up pretty early!

March 19, 2017
by dhoytt
0 comments

2017 Garden Laid Out and Ready

Finished laying out my vegetable garden for 2017 I was rushed into due to mint plants threatening to overwhelm the garden even though I decimated the mints last year. I guess the mint battle will continue as long as I live here though this was not as bad as last year.

This year I had more replanting of plants from last years crop than ever before so I left some plants out for now. The plants from last year will go into seed at some point and I have already planted their replacements.

As mentioned before collard greens, Swiss chard, chives, sage, and rosemary were replanted. I added 3 different types of tomatoes, eggplants, squash, lettuce, peppers, cucumbers and basil.

I also got made some repairs of the drip system and tested that. I finished turning my soil with the rototiller adding mulch and fertilizer now will strategically add and remove vegetables plus as plants grow move parts of the drip system.

    March 18, 2017
    by dhoytt
    0 comments

    Garden Timeline Ready For Mulch 

    Ready for mulch after doing a little pick and weed work. After doing pick work went to local spot to get compost or mulch but they were out so went to big box hardware\outdoor supply store to get some bags of mulch. Got my gas for all my small engines, fed gas to rototiller churned the ground and ready for mulch but then have to work on some systems this evening for job. Just waiting for processing of on job and databases to go down verify all is good then finish up in the garden. Here is how things look presently.

    December 16, 2016
    by dhoytt
    0 comments

    Liquid Comet at Gym

    This evening at the gym while doing last reps of dumbbell presses laying flat on my back looking up to the ceiling working my gum in my mouth exhaling through my mouth, I blew a perfectly formed pearl of small liquid with a comet like trail of beads behind it. This perfectly clear beads of liquid I could see every drop in the comet like trail clearly as I was mid rep pushing the dumbbells the drops hit their apex gave way to gravity coming down in a perfect arch right into my left eye as I was fully extended on that rep. It looked like art in the that short span of time.

      June 27, 2016
      by dhoytt
      0 comments

      Rebuilt XenServer Slave

      I rebuilt my XenServer slave tonight after it didn’t come back up properly last week after a power failure where I decided to ride it out on my UPS and not properly shutdown a few of my systems. The only system that had issues was this XenServer slave and since it wasn’t running my important VM’s at the time I decided to perform some more important non-computer related projects until tonight. Really the only thing that prompted me tonight to do this was I wanted to install Fedora 24 Server on a VM and I recalled I needed to get this XenServer hosts up and running.

      Well the thing was it didn’t recognize and of my NIC’s built into my motherboard though they were enabled and showing properly in the bios. After several restarts and insuring cabling was correct I ended up having to remove power completely and power the system up, reset the bios and then reconfigure my bios settings to get the onboard NIC’s recognized finally.

      Things were so corrupted with XenServer that even after that the onboard NIC’s weren’t recognized so  I decided to reinstall since the VM’s are all on my NAS that I run FreeNAS on anyway and not local to the XenServer host. Had I needed I could have always run the VM’s on the master XenServer host.

      Anyhow I reinstalled,  XenServer 6.5 and tried to add it to the existing pool but it wouldn’t let me join the pool until I was patched up properly plus I still needed to destroy the old XnServer remnant name.

      I tried destroying the old XenServer name out of the shared pool but couldn’t since the VM’s that were running on that hosts were still in a quasi running state. I found this article “http://support.citrix.com/article/CTX126382”  and followed that to shutdown the VM’s:

      • xe host-list
      • xe vm-list resident-on=<UUID of XenServer host I wish to take out of pool>
      • xe vm-reset-powerstate uuid=<UUID of VM’s still on powered on state on above host>
      • Went to XenCenter -> Right-Clicked-> Destroy -> Remove failed XenServer host

      In XenCenter I then went to Tools -> Install Update -> Showed patch that was installed in Master but not on newly installed system and just followed the GUI instructions and it pushed the patch from the Master to the slave and I was then able to join the freshly installed XenServer into the shared pool. I then configured the storage networks, placed the XenServer slave into maintenance mode and turned on multipathing. Restarted my other VM’s and they are now running on the XenServer slave and I’m back up and fully running in redundant mode again!

       

      May 30, 2016
      by dhoytt
      0 comments

      Another Healthy Home Garden Based Meal

      The garden is starting to produce more than just the dominant swiss chard now, with some beans, chives, rosemary plus the swiss chard paired with medium rare trip tip that pulled apart with fingers. Sautéed the swiss char with the rosemary, basil, chives with a touch of brown sugar since I didn’t have any oyster sauce and some garlic in my wok then my friend she came up with the idea to grill the green beans on grill with garlic, salt and olive oil. All turned out excellent didn’t get finishing shots since the wine was flowing and started making plates and eating right away.

       

      IMG_0970 IMG_0969 IMG_0968 IMG_0967 IMG_0965

        May 14, 2016
        by dhoytt
        0 comments

        2016 Vegetable Garden Happenings

        Well the 2016 garden is off to a very nice start. Already I have been feasting on the swiss chards that are replants from last year. As the swiss chard replants from last year go to seed they will be replaced with already steadily growing new swiss chard I planted this year and I will replace the older swiss chard with some peas I have growing already in pots.

        This yeah I have swiss chard, collard greens, several type/colors bell peppers, different basil varieties, several tomato varieties, sweet red onions, yellows onions, chives, habanera peppers, jalapeño peppers, bush and gourmet beans, mesclun salad mix which consists different lettuce and green leafy vegetables for salads, mustard greens, stevia, cucumbers, squash, peas and beets. I’m sure I forgot some items but that’s most of what is planted.

        I was especially attentive this year to getting the weeds out of the garden early this year and properly fertilizing the garden and it seems to be paying off. My first squash to pick should even be ready this week already with cucumbers, tomatoes, basil, stevia and beans, not far behind.

        IMG_0908 IMG_0907 IMG_0906 IMG_0905 IMG_0904 IMG_0903 IMG_0902 IMG_0901 IMG_0900 IMG_0899

        These are pictures that show what it’s all about, meals and meal prep of items from garden:

        IMG_0875 IMG_0876 IMG_0877 IMG_0874 IMG_0873 IMG_0897 IMG_0895

         

        May 14, 2016
        by dhoytt
        0 comments

        Saturday Morning Updates Word Press Web Music Server Work Stations

        Updated the WordPress themes to the latest today in coordination with also updating my separate music streaming server and web server Operating Systems. I think some more tinkering is forthcoming but for now my Windows music streaming server, Linux Web Server and WordPress which resides on my Linux Web Server are updated along with my Windows 10 and Centos 7 workstations.