Adding Second NIC on Linux System points out Wiring Snafu

Wow!! Tonight really emphasized why I need to take some time and just straighten out my wiring situation in my computer room. I have had so much other real life drama I cannot even discuss I have taken a patch and get to it later mentality since the other issues are far more serious but I need to get this straight soon! I have also been on vacation so I relaxed and indulged in some other hobbies as well such as sight seeing and Scuba diving.
Okay what happened? Here’s the long and long of the story!
I have been testing out my new web server installing the base OS packages and performing updates. Until now I have been letting this new system I built running Fedora 6 run into whatever issues it can before I started installing and configuring my web applications.
Well last night before going to bed I decided to start with my web applications and installed Movabletype 3.33 on this FC6 system. I got everything up and configured up until I needed to configure my first blog and went to bed.
Today I explored the Sixapart Movabletype forums, manuals notes to see if I wanted to go to dynamic pages or PHP since a few of my other applications are PHP and I decide to stick with static pages for a few reasons.
I then started the process after watching some basketball and football games to start configuring my first blog and importing the data from my existing blogs over but first I wanted to test some functions. I ran into the typical permissions issues on plugins and directories. I then tried testing Stylecatcher and after running into more permission issues determined it couldn’t get out of my router on port 80.
I decided to fix this by opening up port 80 briefly on my router that is on the same network as my current web server briefly knowing the potential issues with two systems pointing to port 80 on the same router. Well Stylecatcher worked nicely and of course my current web server stated toggling with my web server I’m testing and this made my Web pages act little nuts for about 5 minutes until I discontinued pointing to the new web server with the router.
I guess I could have made Apache look for this system on another port other than 80 but I had a better idea that I wanted to do for other reasons. My other idea was to put another NIC in this system so it would connect to both my networks for data transfer and redundancy reasons. This way it would point to my other static ip.address and I could test at will and have minimal changes to make it my main web server when switching back to my other network.
I put in the additional NIC which wasn’t an issue. I took the system out of the rack put the card in plugged everything back in plus I also plugged in the new NIC to my other network and this is where the weirdness started!
I had it pick up an ip.address via DHCP just to prove it was working and connecting to the correct network. This seemed to work ok but I should have noticed the ip.address it picked up. I then set the ip.adress statically and then rebooted and walked away for a minute.
When I came back after the reboot I checked in /etc/sysconfig/devices/ and there was a device called “ifcfg-eth0.bak. I looked in “system-config-network” and it was listed there as well as “eth0.bak” along with “eth0” & “peth0”. “peth0” turned out to have the settings of my original NIC along with “eth0.bak” with “eth0” having the DHCP settings of the new NIC.
Well I tried to change “eth0” back to the original settings and that didn’t work. Then a new interface named “peth0.bak” appeared. I cannot remember all of the various different combinations of generated NIC names that I tried and were automatically generated but there were quite a few!
I then decide to look at my other multi-homed Linux system that is running Fedora5 and search in Google and other search engines for what some of the variables were in the/etc/sysconfig/devices/ files. I found I had to switch “PEERDNS=yes” to “PEERDNS=no” and fix my “resolv.conf” to the proper DNS nameservers for my ISP.
I switched the new NIC back to DHCP after all of this to prove it was connecting to the correct network. I checked the systems and they resolved the other systems in my network off both interfaces properly. I then checked the DHCP ip.adress assigned to my new NIC and it had a WAN ip.adress!!
The only way this could happen is if it was connecting straight to my WAN which I found out it did! I have my WAN come into a switch and then I have two routers connected to that switch and give both routers my static WAN ip.addresses and then each router has its own set of LAN addresses.
What happened during one of my testing periods when I was switching my LANs’ over to Gigabit was somehow both routers went to a Gigabit switch on the respective networks shared with multi-homed systems. All my systems have static ip.addresses so this issue never showed before!
I had to go through my spaghetti of LAN cables and make sure that from my ISP’s modem it comes to my 10/100mb switch, to my 4 port routers and then they uplink to the 1gb hubs and then to the correct systems!
This just shows what could happen when you test and don’t put things back. This has had to been like this for a few months. Now my network speed should be better for the WAN & LAN. Plus it heightens the awareness that I need to put into action my game plan with my patch panels and wire harnesses that I have already purchased to get things in order!