The past months I have been doing a complete planned upheaval of my entire infrastructure. I’m adding more capacity to my NAS plus implementing better RAID and updating my FreeNAS. I’m moving my entire network to 10gbe. I’m moving all servers from rack mounts to HPE C7000 chassis servers which I had to refamilarize myself with after being away from my surplus find for about 7 months and initially never really having the chance to dive into months ago. I have updated my Xcp-ng hypervisors as well and they will be decommissioned as part of the migration to the C7000 chassis eventually.
This infrastructure total rebuild along with both parents passing and dealing with the estate and family drama plus covid-19, my rental property, HOA board and Neighborhood Watch duties I have not had a boring moment in 2020. This doesn’t even include my possible plans to update my Linux and Windows workstations with hot swap cases as well as my gaming system that has been left totally neglected for a while.
All of this requires planning testing exhaustively before ordering even the smallest items poring over compatibility charts not just with HW but firmware all without support contracts to get to firmware for severe advisory or compatibility firmware fixes easily. Just getting the proper delivery platform to get the firmware to the various components is challenging. I think I have used about 10 different methods of updating different firmware updates just for the C7000 not to mention external switches and routers. With surplus parts getting them from all overt he country and locally you need to jump in and test right away to determine if they are working properly or you are doing something incorrectly because returning parts is time lost and can stall progress and with Covid-19 shipping has added days sometimes a whole week getting items.
I knocked out a couple of easy steps first like updating Xcp-ng hyper-visor to 8.1. Configured a few bl460c Gen7 chassis to play with Virtual Connect and get supremely comfortable with it and decide how to uplink to the network and that’s when the real fun began. I had to update and apply firmware just to be able to get initial control of the C7000 through the OA and Virtual Connect environment, through browsers.
Now with that lengthy summary here’s a bulletized version of what I have done minus things forgotten:
- Updated my C7000 OA version to latest 4.95
- Updated HP VC Flex-10 Enet Module HW to version that would work with OA and browsers.
- Ordered 2 x Gen8 bl460c with 10 cores CPU’s and 128gb or RAM for Hyper-visor
- Configured Gen7 bl460c to be new W2012-r2 music streaming server
- After above firmware updates this Gen7 bl460c and others hung and took over 20 minutes to boot into OS and when it did the W2012-r2 crashed.
- Updated to latest SPP for gen7 and individual ROM updates didn’t solve issue
- Discovered it was due to firmware of Flex NIC’s but firmware updates even from advisories about this issue never resolved the issue
- Turned my attention to 10gbe network and NAS as I was planning to virtualize this Streaming server anyway but not this soon.
- Came back to this issue lately while waiting to touch network again while data is replicating for NAS backup.
- Found another bl460c Gen7 blade that booted up fine and had what looked like a totally different fork of development train for it’s firmware. This server booted properly and I ended up installing w2019 evaluation on it and placed my streaming software and copied the database over from MySQL on my current sever to Mariadb on the new w2019 sever using HeidiSQL.
- The bl460c Gen7 working blade has “emulex bios v11.1.183.23” while the other bl460c Gen7 blades had versions like 4.0.360.15. Took me all a long time to find the firmware with the proper delivery method that matched the firmware files I had but I finally found a bootable ISO that updated my Oneconnect onboard NIC’s and resolved the bl460c Gen7 boot issues with Legacy_OneConnect-Flash-11.1.183.635.ISO.
- For my FreeNAS I got some 16TB SAS drives created a temporary striped volume and started replicating my NFS data and backups all 22 TB worth.
- I started with rsync but since they had no way I could pipe through parallel as the command isn’t on FreeNAS I switched to replication from FreeNAS and that ended up being the better faster option
- Initially the replication didn’t work until I updated to FreeNAS 11-2-UB from 11.2-u1 that had a bug.
- I started with rsync but since they had no way I could pipe through parallel as the command isn’t on FreeNAS I switched to replication from FreeNAS and that ended up being the better faster option
- For my 10GBE external switches I went with Mikrotik CRS 317 and made 3 different bridges for my 3 different networks.
- This ended up being problematic with a NAS and the huge learning curve involved so I ended up getting 2 Mikrotik CRS309 to keep the networks physically separated.
- More to this story later.
- I also scored 2 Procurve 6120XG switches for my C7000 which were a serious pain updating the firmware and wiping the config and finally one ended up being bad and I swapped that out with the vendor I bought it from who has been a pleasure to work with and is another former HP person that’s close by with much of what I need for HP supplies.
- On my FreeNAS I now have to place my original pools into a respectable RAID I should have earlier and told myself I would do but due to time I just kept adding to the current RAID.
- SFP’s have been crazy as well. To uplink from the HP VC Flex-10 Enet Modules you need proprietary HP SFP’s that work on Virtual Connects on each end of the fiber for 10GBE. Even the Rj45 SFP’s for Virtual Connect have to be HP’s.
- The Procurve 6120XG seem to have their own set of SFP’s though I found if you really can find some SFP’s with the “MSA” standard that cost considerably more they most likely would work.
- I found that Mikrotik has some SFP’s that will allow me to use CAT6/5 copper solution into their switch called a “Mikrotik S+RJ10 SFP”. Right now it will work with my 1gb from my routers to the appropriate switches but it will also work with 10GBE once I get to that point.
- I still need to move my stack of UPS into the rack once I phase out more systems.
- Plus may build another NAS to backup this one I currently have.
- Even my KVM decided to go haywire for some reason this past week so will be looking for a more solid version of that as well.
Well that is what I have been up to infrastructure wise the past several months along with the other craziness. I still have a ways to go and will catch up with all these tasks as I complete them hopefully! All this will end up being a scalable future proof environment I can hopefully start some serious programming and other testing and hosting with that will lower my power out put as well.
