That’s right a wild Saturday night in my computer room revolving around my secondary FreeNAS server\disk array. I have had so many issues updating this server to FreeNAS 11.1 I decided tonight I would make sure I came to a conclusion to alter my HW or see if this is a 11.1 FreeNAS issue so that I would know how to proceed with my main FreeNAS system I had to take back to FreeNAS 9.1 due to update issues and lack of time to properly spend on this.

I need to know how I want to invest my money going forward in my computer room and I build items redundantly meaning if I rebuild one FreeNAs server I would end up building two with the first rebuild being the model and test platform until I get things where I like which can mean multiple tries at various hardware and firmware configurations. I often times have controller cards, chassis, hard drives and network cards leftover from these tests and I don’t use consumer level gear so the costs can add up quickly even with planning and checking of posted requirements and hardware compatibility lists.

The first thing I did was boot my secondary FreeNAS and enter the bios and went through every screen to make sure I had configurations I wanted enable and anything unneeded disabled since I had replaced the dead battery on my motherboard with a new one and kept losing my settings every time I unplugged the system and there were several changes that needed to be made.

Secondly I attempted to install from a FreeNAS CDROM I had burned from an SATA connected DVD/RW to an internal flat SSD hard drive of the type created for laptops attached to SATA. This test ended like my prior attempts to install FreeNAS 11.1 and rebooted @  the 70% mark copying files to the SSD drive.

Next I tried to install to an external USB thumb drive and met with the same results system rebooted after reaching 70%.

I also want to note that prior to last night I had tested the memory extensively and pulled out every card in turn and even disconnected all drives and disabled IPMI on the mother board and numerous other isolation techniques going as bare metal as possible with the same results.

Well this time I found a version of FreeNAS 9.3 and attempted to install it with same hardware configuration but to the SATA SSD and that went fabulously proving that my configuration was valid!

While I was trying the install to the thumb drive and testing out the 9.3 install I downloaded the newest version of FreeNAS 11.1u2 which was released sometime in January. I then started browsing my thumb drives I have laying around looking for my FreeNAS thumb drive that was the previous boot drive for this system even though FreeNAS said it had issues that couldn’t be fixed via GUI or  manually clear with ZFS in case I hadn’t save off my configuration database of this FreeNAS system. The FreeNAS website had never stated they had an updated version of 11.1 or I missed it but it wasn’t apparent that it was different until I saw the file being downloaded listed as “FreeNAS-11.1-U2.iso”. I was downloading only thinking maybe I had a corrupted version of 11.1 not to get a new version but I felt a little hope seeing the new version “FreeNAS-11.1-U2.iso”.

Now starting the install to my SATA SSD from FreeNAS 11.1u2 and I’m at first disappointed to see it do its initial DD after I select my SATA SSD have “dd records in” then “ dd records out” be different as it had been in my previous 11.1 install failures while they were the same with 9.x installs. So I get up stretch start going through next steps but keeping an eye out for when this 11.1u2 install will fail when the install scoots past 70% then 80% and I’m thinking it will still fail because prior times I had made some configuration changes to get it to go from 50% to 70% but this version FreeNAS 11.1u2 installed then successfully came up!

Well my work is not over I changed my 3 NIC’s on the system to the appropriate network settings for the various networks from DHCP to static and started importing volumes. Still this did not share out my CIFS or NFS shares, iSCSI targets or any other configuration items I had made on this server and I didn’t feel like recreating something already so I started looking on my Linux workstation I was running the GUI from for my config files for this FreeNAS server and could not find them in the place I thought they should be. I jumped to the cli and did a find and found them on one of the path’s I had browsed but had not gone deep enough in and uploaded the configuration file to my server and after a couple of reboots it came up with a configuration minus a lot of my iSCSI targets. I looked and I had not uploaded the latest configuration file but once I did and a few mere reboots I had my FreeNAS up with proper configuration at 11.1u2!

One of My Routers WAN Configured Incorrectly

My next step was to see if there were any updates for FreeNAS 11.1u2. This system did not bring up “pending updates” but “no route to hosts” or a similar message to that. I go to my primary FreeNAS system which populates the “pending update” field properly. Check my network config and routes which all look fine on my secondary updated FreeNAs system.

Suspecting what I call my third physical network is the issue I ssh on one of my systems that has access to all my routers and networks start pinging from first the primary interface to the outside world using “yahoo.com” to see if there is an issue then I try my 3^rd interface. The primary interface card pings correctly over the WAN while I cannot get out over the WAN from my 3^rd interface (#ping –I <interface>).

I login into my GUI’s interface via a browser and my WAN information is all wrong to go over my third static WAN ip. I make the changes and try again but still get the issue. Looking in the setup of my router again I see 2 typo’s in WAN ip and my gateway is still showing my gateway of my old ISP I changed away from last month. I re-apply the changes and my gateway will not change! I keep trying and no matter what my gateway will not accept changes for the gateway!

Now I start an internet search to see if anyone else has seen this issue since I have DD-WRT installed on this old   and find that I have indeed ran into a bug with my older firmware per the DD-WRT forum string I found. I don’t feel like detaching this router from the network getting one of my laptops and installing new firmware tonight! Thank goodness on this forum string (https://www.dd-wrt.com/phpBB2/viewtopic.php?p=751544) they have a workaround that saved me some time and effort here is the workaround you can double-check with the link above:

OK, so I tried to replace the onclick event on the apply button to bypass the validation

with

After doing the above using a Firefox browser on my Centos 7 server that touches all my networks   I was able to save my gateway and other settings in my DD-WRT router.

I go to my cli on this system and still I cannot ping via my 3^rd network interface to any outside networks via the WAN. Now I look at the routing table on my system and note the only gateway is my primary NIC. I then try pinging out using my 2^nd NIC and cannot and I verify those WAN router settings are correct. I now go into the network config files for each interface (/etc/sysconfig/network-scripts/ifcfg-<interface-name>) change the parameter “DEFROUTE=” from “no” to “yes” bring the interface down with “ifdown” then back up with “ifup” and I have a gateway out to the WAN for each network now  and can ping out on any interface.

None of this of course helped with FreeNAS 11.1u2 not seeing update information but it did help me straighten out my 3^rd router and my Centos 7 routing though I may go back sometime and make other routing changes there.

Looking today my FreeNAS 11.1u2 is populating its “pending update” window properly so I think it was something ultimately at IX Systems end the company who creates FreeNAS.

Backing up Data from Primary FreeNAS to Secondary using Rsync

Now that I have the secondary system up I need to make sure that it has all of my NFS data that my primary system has. What I have done in the past is rsync using cli of the Centos system that has access to all of my networks with both systems mounted on that system but I wanted a FreeNAS system to FreeNAS system approach this time in case I  needed to do other things on that Centos 7 system unrelated to FreeNAS.

I decided to use rsync from one FreeNAS to another server and started to do this via cli and putty but then decided to use the FreeNAS GUI “rsync tasks” instead. I went to the guide they had which was pretty straightforward https://doc.freenas.org/11/services.html#rsync minus one detail I missed because of the way I like to browse the GUI using the top navigation at times. Where it gives you the breadcrumb trail (g Services ‣ Rsync ‣ Rsync Modules ‣ Add Rsync Module.) to create “Rsync Module” as the “rsync target” or “rsync server” , it doesn’t mention you can only access that via the left handed navigation not the top navigation menu.

Then from the “rsync target” FreeNAS server that is pushing the data you need to recall the basic “rsync” rules if you have existing data on the source and you don’t need a whole new directory structure as is my situation then you need to put in a trainilg “/” at the end of your source path so that “rsync” doesn’t create a whole new directory if that’s not what you want. Just make sure you consult the very robust “rsync man pages” from a Linux\Unix\BSD system as they seem to have the same switches.

There is no way to monitor the “rsync” progress from the FreeNAS GUI so you will have to ssh into the server or bring up the shell in the GUI and perform a “ps ax | grep rsync” to see the progress. I am also monitoring the directory growth via my Centos 7 system that accesses all my networks and I have the NFS filesystems of each FreeNAS mounted to. This also reminded me I needed to grow my second FreeNAS  system’s NFS volume and did this via the GUI striping in 2 spare 3TB SAS disks I had while still running rsync to this volume

That was my very wild Saturday night a bit different from some non-computer flavored wild and wordy but still not expressing all the hurdles as any of you in this field know.  This Saturday night ended with my FreeNAS systems syncing to each other and still doing so as I write this.

FreeNAS rsync trailing “\”:

FreeNAS Rsync Module Navigation Location:

Monitoring rsync proceess: