I’ve been looking forward to getting a new network card. Having multiple ports would give me some more flexibility with setting up my virtualized home lab for CCNA. I found a great deal on a quad port Intel PRO/1000 gigabit adapter for roughly $18 on eBay so I scooped one up and have been patiently waiting for it to arrive. When it did, I shutdown my Dell T5400, opened the side panel and was getting ready to line up the network card with a PCIe slot when I noticed my first problem:
there is no PCIe 4x slot!
Fortunately, larger slots like PCIe 8x and PCIe 16x will accept smaller cards and work just fine, as the number associated with the slot size is simply the number of PCIe lanes that are available. I have plenty of spare PCIe slots, so I found an empty PCIe 8x and tried to install the card when I ran in to my second issue:
I bought a low profile card!
As a low profile card with a half height bracket, it wouldn’t properly fit into the full height slot of my tower system. Not willing to accept defeat that easily, I decided to just unscrew the bracket and install it anyway. It’ll work just fine - I just have to make sure to be extra careful inserting or removing cables so I don’t move it too much. I’ll order a full height bracket later…
After closing everything up and turning the system on, I expected to be able to begin playing with my new toy. Of course, by now I should have known that things were not going to be that easy today, but I was still surprised when I couldn’t SSH or even ping the system anymore. At least I finally had an excuse to plug an extra monitor and keyboard into what is usually a headless system…
I logged in and checked my network interfaces by running ip a, which returned what I thought looked mostly correct: I now had 5 ethernet ports/interfaces, and my static IP address was still set correctly. However, since I wasn’t able to ping my router from the system I knew that there was something going on with how Debian and ProxmoxVE were handling the installation of the new device.
Figuring it would be easier to start over from the beginning I disassembled things again, removed the network card, and booted back up to find that my connectivity was restored! I could login through SSH and the web interface like normal. Issuing another ip a command showed me that the two interfaces I was working with were:
enp8s0: my onboard ethernet adaptervmbr0: the virtual bridge for ProxmoxVE (with a static IP)
A quick Google search lead me to the ProxmoxVE documentation wiki for network configuration - specifically to take a look at the naming conventions that ProxmoxVE uses for network interfaces. One of the examples given for the most common naming patterns looked familiar:
enp3s0f1 — is the NIC on pcibus 3 slot 0 and use the NIC function 1
That’s pretty handy! It tells me that my enp8s0 device is a network interface card on PCI bus 8, slot 0. Armed with this new information, I slapped the new network card in again to see what changes. Issuing another ip a command returned what I had seen before: 5 enpXsX devices that were all DOWN:
1 enp11s0
2 enp8s0f0
3 enp8s0f1
4 enp9s0f0
5 enp9s0f1
That looks a lot like my onboard NIC is still there on PCI bus 8 slot 0 (enp8s0) and I have two new devices on 9 and 11 (enp9s0 and enp11s0). Makes sense, but why were the devices down instead of automatically coming up when I plug an ethernet cable in? I had the onboard NIC and one of the 4 ports on the new card plugged right in to my router, so one of them should be working. This led me to editing my interface configuration file at /etc/network/interfaces where I changed all kinds of settings one at a time to see whether they would have any effect. I tried setting every device to auto, allow-hotplug, dhcp, manual, blah blah blah, then restarting the network with /etc/init.d/networking restart or even full reboots with shutdown -r now, all with no success until finally noticing one section of the config file for the vmbr0 interface:
auto vmbr0
iface vmbr0 inet static
bridge-ports enp8s0
This looked correct, because I was wanting this “virtual bridge” to point to my onboard network interface which was enp8s0. But my interface list was showing up with enp8s0f0 and enp8s0f1 now, so maybe changing the config to reflect the new names would work? Nope…
After several more changes to the /etc/network/interfaces config I finally stumbled upon the clue a setting that changed something for the positive! After setting the enp9s0f1 device to automatically initiate with DHCP, it was given an IP address on the next boot. I was able to ping my router, and several other devices on my LAN, so I finally had connectivity! But what the hell was enp9s0f1? In particular, what was this f1 - or “function 1” specification that was newly added to the enp8s0 and other interfaces?
That’s when the clues came together for me.
Initially the system had assigned the onboard network interface the name of enp8s0, without the “function” specification that was added after installing the new 4 port card. It’s kind of embarrassing to admit how long it took me to notice what the issue was. You may have already noticed it yourself, but take a look at my network interface list again:
1 enp11s0
2 enp8s0f0
3 enp8s0f1
4 enp9s0f0
5 enp9s0f1
It isn’t obvious to me at first, but it turns out that the “function” specification in this case refers to the port number of that particular device.
While the systems onboard network interface was assigned PCI bus 8 (enp8s0) initially, when the 4 port expansion card was plugged in to the PCI bus, it was given PCI bus 8 and 9 while the onboard interface was moved to PCI bus 11 and assigned the name enp11s0. That explains why the “function” specification was added to the two devices on bus 8 and 9: the card has 2 dual-port devices on it.
After finally realizing that setting DHCP on enp9s0f1 worked because it was a specific port on my new device, I was able to determine that enp11s0 was now my onboard network device! I made these changes in /etc/network/interfaces to reflect the new device with:
auto vmbr0
iface vmbr0 inet static
bridge-ports enp11s0
Another quick reboot and I was able to ping the router and other LAN devices from the correct interface! The Proxmox VE web panel is the “preferred way” to manage networking devices, so I opened it up and made the necessary changes to get things operational. All that was required was removing a few erroneous comments I’d left in my /etc/network/interfaces configuration efforts - which is handled automatically by ProxmoxVE - and setting all ports/interfaces to start automatically on bootup.
After a few hours of troubleshooting and putting my basic system and network administration skills to the test I not only have 5 high speed network interfaces to play with, but a better understanding of how Debian/ProxmoxVE handles network interfaces and their configuration. Now I can get on to the real fun with what will be my humble little CCNA lab.