Wednesday, July 27, 2011

Disable MSI on a bnx2 driver

Today I had a customer telling me that his network on one machine is slow sometimes. I took a look at it and and saw some Broadcom devices (I don't remember the model) and the bnx2 driver loaded. Also both devices were attached to a bonding device. First step was to figure out the active interface:

# cat /proc/net/bonding/bond0
Ethernet Channel Bonding Driver: v3.5.0 (November 4, 2008)

Bonding Mode: load balancing (round-robin)
Currently Active Slave: eth0
MII Status: up
MII Polling Interval (ms): 1000
Up Delay (ms): 0
Down Delay (ms): 0

Slave Interface: eth0
MII Status: up
Link Failure Count: 0
Permenent HW addr: XX:XX:XX:XX:XX:XX

Slave Interface: eth1
MII Status: up
Link Failure Count: 0
Permenent HW addr: XX:XX:XX:XX:XX:XX

The line "Currently Active Slave: eth0" shows clearly eth0. Checking with ethtool the active interface gave me the following:

# ethtool -S eth0 | errors
     rx_crc_errors: 1423

Doing some research in the internet gave me the hint to disable MSI on the bnx2 driver. So, next steps will be bringing down the interfaces and removing the network drivers:

# ifconfig eth0 down
# ifconfig eth1 down
# ifconfig bond0 down
# rmmod bonding
# rmmod bnx2

Load the bnx2 driver again with disabled MSI:

# modprobe bnx2 disable_msi=1

Check /var/log/messages for "MSI" messages, it should not show up after loading the bnx2 driver again:

# grep MSI /var/log/messages

Load the bonding driver again and set the bond0 interface up:

# modprobe bonding
# ifconfig bond0 up

Now attach the slave interfaces again:

# ifenslave bond0 eth0
# ifenslave bond0 eth1

Finally configure the bond0 device again:

# ifconfig bond0 netmask

To make the changes permanent (e.g. after reboot) add the following line to /etc/modprobe.conf:

# echo "option bnx2 disable_msi=1" >> /etc/modprobe.conf

Hopefully your network runs better now - mine did not...

Additional information: