Avoid Realtek network adapters for Type 1 Hypervisor platforms

bert64

Senior Member
Joined
Jan 20, 2020
Messages
1,027
Reaction score
539
I am pretty satisfied with the results now even with the "crappy" Realtek RTL8168 Gigabit Ethernet adapter. The virtual pfSense (FreeBSD based) router now runs pretty okay. The upload speed is pretty good. The download speed suffers a bit more but still prtty decent.

Rigtht now there is no package qemu-guest-agent available for the pfSense (but available for FreeBSD already). Not so sure if will make things even better.

pfSense router WAN -- Linux bridge vmbr0 using virtio driver, parent is the Realtek RTL8168 adapter
pfSense router LAN -- Linux bridge vmbr2 using virtio driver, parent is the Realtek USB 3 RTL8156 adapter running in USB 2.0 highspeed. I am not able to fix this as of now. But that only affects the link speed between the LAN VM and phsyical LAN machine (iperf3 shows around 320Mbps to 340Mbps). Internal LAN VMs have very fast speed to each other (iperf3 shows 6.6Gbps to 6.8Gbps).

Code:
Debian 11 VM behind virtual pfSense router
cuee@debian:~$ speedtest -s 25960

   Speedtest by Ookla

     Server: Singtel - Singapore (id = 25960)
        ISP: Singtel Fiber
    Latency:     1.54 ms   (0.29 ms jitter)
   Download:   652.86 Mbps (data used: 683.2 MB )                            
     Upload:   895.73 Mbps (data used: 1.3 GB )                            
Packet Loss:     0.0%
 Result URL: https://www.speedtest.net/result/c/6064d07a-a93d-4894-a2e2-9a39ae02c2c4

Physical Linux machine directly connected to the router (Singtel 1Gbps plan)
mcuee@ubuntu64:~$ speedtest -s 25960

   Speedtest by Ookla

     Server: Singtel - Singapore (id = 25960)
        ISP: Singtel Fiber
    Latency:     1.51 ms   (1.49 ms jitter)
   Download:   912.31 Mbps (data used: 469.4 MB)                            
     Upload:   945.73 Mbps (data used: 1.1 GB)                            
Packet Loss:     0.0%
 Result URL: https://www.speedtest.net/result/c/513f4227-c954-4f78-8403-1c4aacfa4c0e
Have you tried attaching the usb ethernet to different ports, or removing all other usb devices and see if that makes any difference?
 

xiaofan

High Supremacy Member
Joined
Sep 16, 2018
Messages
32,726
Reaction score
10,241
Have you tried attaching the usb ethernet to different ports, or removing all other usb devices and see if that makes any difference?

Yes I have tried that. I have three Ugreen 2.5Gbe adapters (one USB Type C version and two Type A version).

The mini PC has two USB 3.1 Type A ports, one Type C port and two USB 2.0 Type A ports.

It should work as per the various website reports. I will keep trying.

Maybe I should give the kernel built-in driver a try as it is supposed to be supported in the kernel 5.13.

The following is from the other J4105 mini PC running Ubuntu 21.10 and the adapter works fine there with kernel 5.13 without using vendor driver.

Code:
mcuee@ubuntu64:~$ uname -a
Linux ubuntu64 5.13.0-22-generic #22-Ubuntu SMP Fri Nov 5 13:21:36 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux

mcuee@ubuntu64:~$ lsusb -t
/:  Bus 02.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/7p, 5000M
    |__ Port 2: Dev 3, If 0, Class=Communications, Driver=cdc_ncm, 5000M
    |__ Port 2: Dev 3, If 1, Class=CDC Data, Driver=cdc_ncm, 5000M
/:  Bus 01.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/9p, 480M
 
Last edited:

xiaofan

High Supremacy Member
Joined
Sep 16, 2018
Messages
32,726
Reaction score
10,241
Interestingly after I plug-in a USB 3.0 stroage and then remove the USB 3.0 stroage, and then plug in the 2.5Gbe adapter, it is now recognized as USB 3.0 superspeed adapter.

Code:
root@pve:~# lsusb -t
/:  Bus 02.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/6p, 10000M
    |__ Port 1: Dev 3, If 0, Class=Vendor Specific Class, Driver=r8152, 5000M
    |__ Port 2: Dev 2, If 0, Class=Mass Storage, Driver=usb-storage, 5000M
/:  Bus 01.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/8p, 480M
    |__ Port 1: Dev 2, If 0, Class=Hub, Driver=hub/4p, 480M
        |__ Port 1: Dev 4, If 0, Class=Vendor Specific Class, Driver=asix, 480M
    |__ Port 8: Dev 3, If 1, Class=Wireless, Driver=btusb, 12M
    |__ Port 8: Dev 3, If 0, Class=Wireless, Driver=btusb, 12M

root@pve:~# ethtool enx2c16dba12018
Settings for enx2c16dba12018:
        Supported ports: [ MII ]
        Supported link modes:   10baseT/Half 10baseT/Full
                                100baseT/Half 100baseT/Full
                                1000baseT/Full
                                2500baseT/Full
        Supported pause frame use: No
        Supports auto-negotiation: Yes
        Supported FEC modes: Not reported
        Advertised link modes:  10baseT/Half 10baseT/Full
                                100baseT/Half 100baseT/Full
                                1000baseT/Full
                                2500baseT/Full
        Advertised pause frame use: No
        Advertised auto-negotiation: Yes
        Advertised FEC modes: Not reported
        Link partner advertised link modes:  10baseT/Half 10baseT/Full
                                             100baseT/Half 100baseT/Full
                                             1000baseT/Full
                                             2500baseT/Full
        Link partner advertised pause frame use: No
        Link partner advertised auto-negotiation: Yes
        Link partner advertised FEC modes: Not reported
        Speed: 2500Mb/s
        Duplex: Full
        Auto-negotiation: on
        Port: MII
        PHYAD: 32
        Transceiver: internal
        Supports Wake-on: pumbg
        Wake-on: g
        Current message level: 0x00007fff (32767)
                               drv probe link timer ifdown ifup rx_err tx_err tx_queued intr tx_done rx_status pktdata hw wol
        Link detected: yes
 
Last edited:

xiaofan

High Supremacy Member
Joined
Sep 16, 2018
Messages
32,726
Reaction score
10,241
However, it seems to be one direction, strange.

pve host N4500 mini PC -- 192.168.8.216
pfSense virtual router LAN -- 192.168.8.1
Ubuntu J4105 mini PC -- 192.168.8.203 (dhcp from pfSense)

Code:
mcuee@ubuntu64:~$ iperf3 -c 192.168.8.216
Connecting to host 192.168.8.216, port 5201
[  5] local 192.168.8.203 port 60620 connected to 192.168.8.216 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec   280 MBytes  2.35 Gbits/sec    0   1.65 MBytes      
[  5]   1.00-2.00   sec   280 MBytes  2.35 Gbits/sec    0   1.65 MBytes      
[  5]   2.00-3.00   sec   279 MBytes  2.34 Gbits/sec    0   1.73 MBytes      
[  5]   3.00-4.00   sec   280 MBytes  2.35 Gbits/sec    0   1.73 MBytes      
[  5]   4.00-5.00   sec   279 MBytes  2.34 Gbits/sec    0   1.94 MBytes      
[  5]   5.00-6.00   sec   280 MBytes  2.35 Gbits/sec    0   1.94 MBytes      
[  5]   6.00-7.00   sec   279 MBytes  2.34 Gbits/sec    0   2.05 MBytes      
[  5]   7.00-8.00   sec   280 MBytes  2.35 Gbits/sec    0   2.05 MBytes      
[  5]   8.00-9.00   sec   280 MBytes  2.35 Gbits/sec    0   2.15 MBytes      
[  5]   9.00-10.00  sec   279 MBytes  2.34 Gbits/sec    0   2.15 MBytes      
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  2.73 GBytes  2.34 Gbits/sec    0             sender
[  5]   0.00-10.00  sec  2.73 GBytes  2.34 Gbits/sec                  receiver

iperf Done.
mcuee@ubuntu64:~$ iperf3 -c 192.168.8.216 -R
Connecting to host 192.168.8.216, port 5201
Reverse mode, remote host 192.168.8.216 is sending
[  5] local 192.168.8.203 port 60624 connected to 192.168.8.216 port 5201
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.00   sec  45.0 MBytes   377 Mbits/sec                
[  5]   1.00-2.00   sec  38.8 MBytes   325 Mbits/sec                
[  5]   2.00-3.00   sec  47.8 MBytes   401 Mbits/sec                
[  5]   3.00-4.00   sec  51.3 MBytes   430 Mbits/sec                
[  5]   4.00-5.00   sec  37.0 MBytes   310 Mbits/sec                
[  5]   5.00-6.00   sec  53.7 MBytes   451 Mbits/sec                
[  5]   6.00-7.00   sec  40.8 MBytes   343 Mbits/sec                
[  5]   7.00-8.00   sec  42.4 MBytes   355 Mbits/sec                
[  5]   8.00-9.00   sec  41.2 MBytes   346 Mbits/sec                
[  5]   9.00-10.00  sec  33.8 MBytes   284 Mbits/sec                
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec   433 MBytes   363 Mbits/sec  106             sender
[  5]   0.00-10.00  sec   432 MBytes   362 Mbits/sec                  receiver

iperf Done.
 
Last edited:

xiaofan

High Supremacy Member
Joined
Sep 16, 2018
Messages
32,726
Reaction score
10,241
Looks like not related to pfSense as it is the same using OpenWRT virtual router.

Code:
cuee@ubuntu64:~$ iperf3 -s
-----------------------------------------------------------
Server listening on 5201
-----------------------------------------------------------
Accepted connection from 192.168.8.1, port 43354
[  5] local 192.168.8.239 port 5201 connected to 192.168.8.1 port 43356
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.00   sec  38.5 MBytes   323 Mbits/sec                 
[  5]   1.00-2.00   sec  30.6 MBytes   257 Mbits/sec                 
[  5]   2.00-3.00   sec  29.9 MBytes   251 Mbits/sec                 
[  5]   3.00-4.00   sec  42.9 MBytes   360 Mbits/sec                 
[  5]   4.00-5.00   sec  40.1 MBytes   336 Mbits/sec                 
[  5]   5.00-6.00   sec  35.4 MBytes   297 Mbits/sec                 
[  5]   6.00-7.00   sec  31.1 MBytes   261 Mbits/sec                 
[  5]   7.00-8.00   sec  37.5 MBytes   314 Mbits/sec                 
[  5]   8.00-9.00   sec  38.4 MBytes   322 Mbits/sec                 
[  5]   9.00-10.00  sec  31.6 MBytes   265 Mbits/sec                 
[  5]  10.00-10.00  sec  63.6 KBytes   268 Mbits/sec                 
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-10.00  sec   356 MBytes   299 Mbits/sec                  receiver
-----------------------------------------------------------
Server listening on 5201
-----------------------------------------------------------
Accepted connection from 192.168.8.1, port 43358
[  5] local 192.168.8.239 port 5201 connected to 192.168.8.1 port 43360
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec   273 MBytes  2.29 Gbits/sec    0   1.68 MBytes       
[  5]   1.00-2.00   sec   279 MBytes  2.34 Gbits/sec    0   2.13 MBytes       
[  5]   2.00-3.00   sec   280 MBytes  2.35 Gbits/sec    0   2.13 MBytes       
[  5]   3.00-4.00   sec   279 MBytes  2.34 Gbits/sec    0   2.31 MBytes       
[  5]   4.00-5.00   sec   279 MBytes  2.34 Gbits/sec    0   2.42 MBytes       
[  5]   5.00-6.00   sec   280 MBytes  2.35 Gbits/sec    0   2.85 MBytes       
[  5]   6.00-7.00   sec   279 MBytes  2.34 Gbits/sec    0   2.85 MBytes       
[  5]   7.00-8.00   sec   280 MBytes  2.35 Gbits/sec    0   2.85 MBytes       
[  5]   8.00-9.00   sec   279 MBytes  2.34 Gbits/sec    0   2.85 MBytes       
[  5]   9.00-10.00  sec   280 MBytes  2.35 Gbits/sec    0   2.85 MBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  2.72 GBytes  2.34 Gbits/sec    0             sender
-----------------------------------------------------------
Server listening on 5201
-----------------------------------------------------------
 

mrkiasu

Senior Member
Joined
Feb 20, 2001
Messages
2,001
Reaction score
59
Just avoid using "crab" whenever possible in any space (home [maybe] or commercial [never] or enterprise [never]).

Never like them since 2006, even worse when I was in network manufacturing firm.
 

xiaofan

High Supremacy Member
Joined
Sep 16, 2018
Messages
32,726
Reaction score
10,241
Finally got a good result from the pve host to a Windows laptop.

pve host -- Ugreen USB 3 Type A 2.5G adaptger, stock kernel cdc-ncm driver
Windows laptop -- Ugreen USB 3 Type C 2.5G adaptger

Code:
root@pve:~# lsusb -t
/:  Bus 02.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/6p, 10000M
    |__ Port 1: Dev 4, If 0, Class=Communications, Driver=cdc_ncm, 5000M
    |__ Port 1: Dev 4, If 1, Class=CDC Data, Driver=cdc_ncm, 5000M
/:  Bus 01.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/8p, 480M
    |__ Port 1: Dev 2, If 0, Class=Hub, Driver=hub/4p, 480M
        |__ Port 1: Dev 4, If 0, Class=Vendor Specific Class, Driver=asix, 480M
    |__ Port 8: Dev 3, If 1, Class=Wireless, Driver=btusb, 12M
    |__ Port 8: Dev 3, If 0, Class=Wireless, Driver=btusb, 12M

Code:
PS C:\work\speedtest\iperf3.10.1_64bit> .\iperf3 -c 192.168.8.216
Connecting to host 192.168.8.216, port 5201
[  5] local 192.168.8.200 port 57943 connected to 192.168.8.216 port 5201
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.00   sec   243 MBytes  2.04 Gbits/sec
[  5]   1.00-2.00   sec   242 MBytes  2.03 Gbits/sec
[  5]   2.00-3.00   sec   244 MBytes  2.04 Gbits/sec
[  5]   3.00-4.00   sec   242 MBytes  2.03 Gbits/sec
[  5]   4.00-5.00   sec   243 MBytes  2.04 Gbits/sec
[  5]   5.00-6.00   sec   242 MBytes  2.03 Gbits/sec
[  5]   6.00-7.00   sec   243 MBytes  2.04 Gbits/sec
[  5]   7.00-8.00   sec   242 MBytes  2.03 Gbits/sec
[  5]   8.00-9.00   sec   243 MBytes  2.04 Gbits/sec
[  5]   9.00-10.00  sec   243 MBytes  2.04 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-10.00  sec  2.37 GBytes  2.04 Gbits/sec                  sender
[  5]   0.00-10.05  sec  2.37 GBytes  2.02 Gbits/sec                  receiver

iperf Done.
PS C:\work\speedtest\iperf3.10.1_64bit> .\iperf3 -c 192.168.8.216 -R
Connecting to host 192.168.8.216, port 5201
Reverse mode, remote host 192.168.8.216 is sending
[  5] local 192.168.8.200 port 57951 connected to 192.168.8.216 port 5201
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.00   sec   283 MBytes  2.37 Gbits/sec
[  5]   1.00-2.00   sec   283 MBytes  2.37 Gbits/sec
[  5]   2.00-3.00   sec   283 MBytes  2.37 Gbits/sec
[  5]   3.00-4.00   sec   283 MBytes  2.37 Gbits/sec
[  5]   4.00-5.00   sec   283 MBytes  2.37 Gbits/sec
[  5]   5.00-6.00   sec   283 MBytes  2.37 Gbits/sec
[  5]   6.00-7.00   sec   283 MBytes  2.37 Gbits/sec
[  5]   7.00-8.00   sec   283 MBytes  2.37 Gbits/sec
[  5]   8.00-9.00   sec   283 MBytes  2.37 Gbits/sec
[  5]   9.00-10.00  sec   283 MBytes  2.37 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.05  sec  2.77 GBytes  2.36 Gbits/sec  266             sender
[  5]   0.00-10.00  sec  2.76 GBytes  2.37 Gbits/sec                  receiver

iperf Done.
 

xiaofan

High Supremacy Member
Joined
Sep 16, 2018
Messages
32,726
Reaction score
10,241
Finally got a good result from the pve host to a Windows laptop.

pve host -- Ugreen USB 3 Type A 2.5G adaptger, stock kernel cdc-ncm driver
Windows laptop -- Ugreen USB 3 Type C 2.5G adaptger

Good results as well with the OpenWRT virtual router.

PVE host -- mini PC integrated Realtek RTL8168 Gigabit Ethernet adapter, using vendor driver
Ugreen USB 3 Type A 2.5Gbe adapter (RTL8156B), stock kernel cdc-ncm driver

Vitual OpenWRT -- WAN bridge to the PVE RTL8168, virtio driver
LAN bridge to the PVE host Ugreen USB 3 Type A 2.5Gbe adapter, virtio driver

Windows laptop -- Ugreen USB 3 Type C 2.5Gbe adaptger, connected to the PVE host Ugreen adapter

Code:
PS C:\work\speedtest\iperf3.10.1_64bit> .\iperf3.exe -c 192.168.8.1
Connecting to host 192.168.8.1, port 5201
[  5] local 192.168.8.88 port 56803 connected to 192.168.8.1 port 5201
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.00   sec   242 MBytes  2.03 Gbits/sec
[  5]   1.00-2.00   sec   241 MBytes  2.03 Gbits/sec
[  5]   2.00-3.00   sec   241 MBytes  2.02 Gbits/sec
[  5]   3.00-4.00   sec   239 MBytes  2.01 Gbits/sec
[  5]   4.00-5.00   sec   239 MBytes  2.01 Gbits/sec
[  5]   5.00-6.00   sec   242 MBytes  2.03 Gbits/sec
[  5]   6.00-7.00   sec   240 MBytes  2.02 Gbits/sec
[  5]   7.00-8.00   sec   239 MBytes  2.00 Gbits/sec
[  5]   8.00-9.00   sec   239 MBytes  2.00 Gbits/sec
[  5]   9.00-10.00  sec   241 MBytes  2.02 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-10.00  sec  2.35 GBytes  2.02 Gbits/sec                  sender
[  5]   0.00-10.01  sec  2.34 GBytes  2.01 Gbits/sec                  receiver

iperf Done.
PS C:\work\speedtest\iperf3.10.1_64bit> .\iperf3.exe -c 192.168.8.1 -R
Connecting to host 192.168.8.1, port 5201
Reverse mode, remote host 192.168.8.1 is sending
[  5] local 192.168.8.88 port 56812 connected to 192.168.8.1 port 5201
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.00   sec   279 MBytes  2.34 Gbits/sec
[  5]   1.00-2.00   sec   282 MBytes  2.37 Gbits/sec
[  5]   2.00-3.00   sec   283 MBytes  2.37 Gbits/sec
[  5]   3.00-4.00   sec   282 MBytes  2.36 Gbits/sec
[  5]   4.00-5.00   sec   280 MBytes  2.35 Gbits/sec
[  5]   5.00-6.00   sec   272 MBytes  2.28 Gbits/sec
[  5]   6.00-7.00   sec   277 MBytes  2.32 Gbits/sec
[  5]   7.00-8.00   sec   276 MBytes  2.32 Gbits/sec
[  5]   8.00-9.00   sec   277 MBytes  2.32 Gbits/sec
[  5]   9.00-10.00  sec   279 MBytes  2.34 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  2.72 GBytes  2.34 Gbits/sec  262             sender
[  5]   0.00-10.00  sec  2.72 GBytes  2.34 Gbits/sec                  receiver

iperf Done.
 

xiaofan

High Supremacy Member
Joined
Sep 16, 2018
Messages
32,726
Reaction score
10,241
However, it seems to be one direction, strange.

pve host N4500 mini PC -- 192.168.8.216
pfSense virtual router LAN -- 192.168.8.1
Ubuntu J4105 mini PC -- 192.168.8.203 (dhcp from pfSense)

Looks like not related to pfSense as it is the same using OpenWRT virtual router.

I think this may not be related to the PVE host already, probably something to do with my J4105 mini PC running Ubuntu 21.10 (will upgrade to 22.04 next year).

It is using the stock cdc-ncm driver as well with kernel 5.13. I probably eed to troubleshoot from there. It may have different route to the PVE host.

I have just switched the J4105 mini PC netplan renderer from using NetworkManager to systemd-networkd as it runs more like a server now. And it has dual connections to the two home routers (gigabit Ethernet adapter). The PVE host also has dual connections to the two home routers (one gigabit Ethernet adapter and one with 100Mbps USB Ethernet adapter as a backup connection, will switch to USB gigabit adapter once I got them from Shopee). Not sure if that causes some issues.

Code:
mcuee@ubuntu64:~$ iperf3 -c 192.168.8.1
Connecting to host 192.168.8.1, port 5201
[  5] local 192.168.8.196 port 34408 connected to 192.168.8.1 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec   257 MBytes  2.16 Gbits/sec    0   2.60 MBytes     
[  5]   1.00-2.00   sec   275 MBytes  2.31 Gbits/sec  684   1.36 MBytes     
[  5]   2.00-3.00   sec   278 MBytes  2.33 Gbits/sec    0   1.45 MBytes     
[  5]   3.00-4.00   sec   276 MBytes  2.32 Gbits/sec    0   1.51 MBytes     
[  5]   4.00-5.00   sec   279 MBytes  2.34 Gbits/sec    0   1.55 MBytes     
[  5]   5.00-6.00   sec   279 MBytes  2.34 Gbits/sec    0   1.60 MBytes     
[  5]   6.00-7.00   sec   279 MBytes  2.34 Gbits/sec    0   1.67 MBytes     
[  5]   7.00-8.00   sec   279 MBytes  2.34 Gbits/sec    0   1.70 MBytes     
[  5]   8.00-9.00   sec   278 MBytes  2.33 Gbits/sec    0   1.72 MBytes     
[  5]   9.00-10.00  sec   274 MBytes  2.30 Gbits/sec  202   1.29 MBytes     
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  2.69 GBytes  2.31 Gbits/sec  886             sender
[  5]   0.00-10.00  sec  2.69 GBytes  2.31 Gbits/sec                  receiver

iperf Done.
mcuee@ubuntu64:~$ iperf3 -c 192.168.8.1 -R
Connecting to host 192.168.8.1, port 5201
Reverse mode, remote host 192.168.8.1 is sending
[  5] local 192.168.8.196 port 34412 connected to 192.168.8.1 port 5201
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.00   sec  36.3 MBytes   305 Mbits/sec               
[  5]   1.00-2.00   sec  32.2 MBytes   270 Mbits/sec               
[  5]   2.00-3.00   sec  42.1 MBytes   353 Mbits/sec               
[  5]   3.00-4.00   sec  34.5 MBytes   290 Mbits/sec               
[  5]   4.00-5.00   sec  38.2 MBytes   320 Mbits/sec               
[  5]   5.00-6.00   sec  38.8 MBytes   326 Mbits/sec               
[  5]   6.00-7.00   sec  41.6 MBytes   349 Mbits/sec               
[  5]   7.00-8.00   sec  43.5 MBytes   365 Mbits/sec               
[  5]   8.00-9.00   sec  52.4 MBytes   439 Mbits/sec               
[  5]   9.00-10.00  sec  39.8 MBytes   334 Mbits/sec               
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec   401 MBytes   336 Mbits/sec  142             sender
[  5]   0.00-10.00  sec   399 MBytes   335 Mbits/sec                  receiver

iperf Done.

cat /etc/netplan/01-network-manager-all.yaml
# Let systemd-networkd manage all devices on this system
network:
  version: 2
  renderer: networkd

  ethernets:
      enp1s0:
          dhcp4: yes
          dhcp4-overrides:
              route-metric: 100

      enp2s0:
          dhcp4: yes
          dhcp4-overrides:
              route-metric: 200

      eno1:
          dhcp4: no
          addresses: [192.168.8.218/24]
          #gateway4: 192.168.8.1
          nameservers:
              addresses: [1.1.1.3, 1.0.0.3]
          routes:
              - to: default
                via: 192.168.8.1
                metric: 300
 
      enp4s0:
          dhcp4: no
          addresses: [192.168.28.218/24]
          #gateway4: 192.168.28.1
          nameservers:
              addresses: [1.1.1.3, 1.0.0.3]
          routes:
              - to: default
                via: 192.168.28.1
                metric: 400

      enx2c16dba12018:
          dhcp4: yes
          dhcp4-overrides:
              route-metric: 500
 
Last edited:

xiaofan

High Supremacy Member
Joined
Sep 16, 2018
Messages
32,726
Reaction score
10,241
I think this may not be related to the PVE host already, probably something to do with my J4105 mini PC running Ubuntu 21.10 (will upgrade to 22.04 next year).

It is using the stock cdc-ncm driver as well with kernel 5.13. I probably eed to troubleshoot from there. It may have different route to the PVE host.

I have just switched the J4105 mini PC netplan renderer from using NetworkManager to systemd-networkd as it runs more like a server now. And it has dual connections to the two home routers (gigabit Ethernet adapter). The PVE host also has dual connections to the two home routers (one gigabit Ethernet adapter and one with 100Mbps USB Ethernet adapter as a backup connection, will switch to USB gigabit adapter once I got them from Shopee). Not sure if that causes some issues.

The issue is apparent not related to the PVE or with the connections. But rather it is because of the Ubuntu 21.10 Kernel cdc-ncm driver support for RTL8156B. I can reproduce the issue with my Acer Swift 3 laptop (dual boot Windows 11 and Ubuntu 20.04 as well) and the USB Type C RLT8156B 2.5Gbe adapter.

I just connect the laptop to the Intel J4105 mini PC (both have the RTL8156B 2.5Gbe USB adapter), if I use Windows on the laptop, I can get one direction 2.5Gbps and the other direction slower speed. If I run Ubuntu 20.04 Linux, then both direction will be slow. So the cdc-ncm driver in Ubuntu 20.04 also has the same issues.

Maybe it is because of some settings. Or maybe I need to use the vendor driver.
 

xiaofan

High Supremacy Member
Joined
Sep 16, 2018
Messages
32,726
Reaction score
10,241
Same thing happened between another Windows 11 laptop and Ubuntu 20.04 laptop. The Windows client has not issue to receive 2.5Gbps, but Ubuntu server can not receive 2.5Gbps, rather only around half. So it is pretty clear the issue on the Ubuntu side that Rx is slow.

Edit to add: note to self: iperf direction is from client to server. "-R" is from server to client.

Code:
mcuee@UbuntuSwift3:~$ lsusb -t
/:  Bus 04.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/4p, 10000M
/:  Bus 03.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/12p, 480M
    |__ Port 3: Dev 2, If 0, Class=Human Interface Device, Driver=usbhid, 12M
    |__ Port 3: Dev 2, If 1, Class=Human Interface Device, Driver=usbhid, 12M
    |__ Port 5: Dev 3, If 0, Class=Video, Driver=uvcvideo, 480M
    |__ Port 5: Dev 3, If 1, Class=Video, Driver=uvcvideo, 480M
    |__ Port 7: Dev 4, If 0, Class=Vendor Specific Class, Driver=, 480M
    |__ Port 10: Dev 5, If 0, Class=Wireless, Driver=btusb, 12M
    |__ Port 10: Dev 5, If 1, Class=Wireless, Driver=btusb, 12M
/:  Bus 02.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/4p, 10000M
    |__ Port 1: Dev 2, If 0, Class=Communications, Driver=cdc_ncm, 5000M
    |__ Port 1: Dev 2, If 1, Class=CDC Data, Driver=cdc_ncm, 5000M
/:  Bus 01.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/1p, 480M

mcuee@UbuntuSwift3:~$ uname -a
Linux UbuntuSwift3 5.11.0-43-generic #47~20.04.2-Ubuntu SMP Mon Dec 13 11:06:56 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux


mcuee@UbuntuSwift3:~$ iperf3 -s
-----------------------------------------------------------
Server listening on 5201
-----------------------------------------------------------
Accepted connection from 192.168.38.38, port 11682
[  5] local 192.168.38.88 port 5201 connected to 192.168.38.38 port 11683
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.00   sec   131 MBytes  1.10 Gbits/sec              
[  5]   1.00-2.00   sec   138 MBytes  1.16 Gbits/sec              
[  5]   2.00-3.00   sec   137 MBytes  1.15 Gbits/sec              
[  5]   3.00-4.00   sec   138 MBytes  1.16 Gbits/sec              
[  5]   4.00-5.00   sec   138 MBytes  1.16 Gbits/sec              
[  5]   5.00-6.00   sec   138 MBytes  1.16 Gbits/sec              
[  5]   6.00-7.00   sec   138 MBytes  1.16 Gbits/sec              
[  5]   7.00-8.00   sec   138 MBytes  1.16 Gbits/sec              
[  5]   8.00-9.00   sec   138 MBytes  1.16 Gbits/sec              
[  5]   9.00-10.00  sec   138 MBytes  1.16 Gbits/sec              
[  5]  10.00-10.06  sec  8.86 MBytes  1.16 Gbits/sec              
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-10.06  sec  1.35 GBytes  1.15 Gbits/sec                  receiver
-----------------------------------------------------------
Server listening on 5201
-----------------------------------------------------------
Accepted connection from 192.168.38.38, port 11686
[  5] local 192.168.38.88 port 5201 connected to 192.168.38.38 port 11687
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec   271 MBytes  2.27 Gbits/sec    0   1009 KBytes    
[  5]   1.00-2.00   sec   281 MBytes  2.36 Gbits/sec    0   1009 KBytes    
[  5]   2.00-3.00   sec   282 MBytes  2.37 Gbits/sec    0   1009 KBytes    
[  5]   3.00-4.00   sec   281 MBytes  2.36 Gbits/sec    0   1009 KBytes    
[  5]   4.00-5.00   sec   282 MBytes  2.37 Gbits/sec    0   1009 KBytes    
[  5]   5.00-6.00   sec   282 MBytes  2.37 Gbits/sec    0   1009 KBytes    
[  5]   6.00-7.00   sec   281 MBytes  2.36 Gbits/sec    0   1.04 MBytes    
[  5]   7.00-8.00   sec   281 MBytes  2.36 Gbits/sec  593    643 KBytes    
[  5]   8.00-9.00   sec   281 MBytes  2.36 Gbits/sec    0    918 KBytes    
[  5]   9.00-10.00  sec   282 MBytes  2.37 Gbits/sec    0    975 KBytes    
[  5]  10.00-10.05  sec  15.0 MBytes  2.41 Gbits/sec    0    975 KBytes    
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.05  sec  2.76 GBytes  2.36 Gbits/sec  593             sender
-----------------------------------------------------------
Server listening on 5201
-----------------------------------------------------------
^Ciperf3: interrupt - the server has terminated
 
Last edited:

xiaofan

High Supremacy Member
Joined
Sep 16, 2018
Messages
32,726
Reaction score
10,241
Same thing happened between another Windows 11 laptop and Ubuntu 20.04 laptop. The Windows client has not issue to receive 2.5Gbps, but Ubuntu server can not receive 2.5Gbps, rather only around half. So it is pretty clear the issue on the Ubuntu side that Rx is slow.

Edit to add: note to self: iperf direction is from client to server. "-R" is from server to client.

I have tried to tune a few parameters as per the following post but it does not seem to help. Never mind, I will wait for another time to try to fix this.
https://wu.renjie.im/blog/network/ax88u-nbase-t/
Code:
mcuee@UbuntuSwift3:~$ cat /sys/class/net/enx2c16dba0e649/queues/rx-0/rps_cpus
0f
mcuee@UbuntuSwift3:~$ cat /sys/class/net/enx2c16dba0e649/queues/rx-0/rps_flow_cnt
0
mcuee@UbuntuSwift3:~$ cat /proc/sys/net/core/netdev_max_backlog
2500
mcuee@UbuntuSwift3:~$ cat /proc/sys/net/core/rps_sock_flow_entries
0

Apparently pve host side has no such issues and it is also using the stock cdc_ncm driver. Kind of strange. And it does not need any tuning. Looks like a kernel difference issue.
 

xiaofan

High Supremacy Member
Joined
Sep 16, 2018
Messages
32,726
Reaction score
10,241
I have tried to tune a few parameters as per the following post but it does not seem to help. Never mind, I will wait for another time to try to fix this.

Apparently pve host side has no such issues and it is also using the stock cdc_ncm driver. Kind of strange. And it does not need any tuning. Looks like a kernel difference issue.

As part of the learning exercise, side benefit is that I learned how to sign a kernel module or kernel to boot with Secure Boot enabled. That does not help to sort out the issue but at least I learn something new.

I have tried Ubuntu kernel version 5.13.19 and 5.15.10 and both do not help.
https://kernel.ubuntu.com/~kernel-ppa/mainline/
Ref: Debian Wiki
https://wiki.debian.org/SecureBoot#MOK_-_Machine_Owner_Key
Code:
mcuee@UbuntuSwift3:~$ uname -a
Linux UbuntuSwift3 5.15.10-051510-generic #202112171031 SMP Fri Dec 17 10:53:39 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
 

xiaofan

High Supremacy Member
Joined
Sep 16, 2018
Messages
32,726
Reaction score
10,241
I have tried to tune a few parameters as per the following post but it does not seem to help. Never mind, I will wait for another time to try to fix this.
https://wu.renjie.im/blog/network/ax88u-nbase-t/
Apparently pve host side has no such issues and it is also using the stock cdc_ncm driver. Kind of strange. And it does not need any tuning. Looks like a kernel difference issue.

So I have theee different Rx results with different PC (Tx is all good)

1) PVE (Debian based) with kernel 5.13 --> Rx is good at 2.32Gbps.

2) New Acer Swift 3 laptop dual boot with Ubuntu 20.04, 1.15 Gbps Rx speed.

3) The worst seems to be the Intel J4105 mini PC which runs Ubuntu 21.10, it can not even achieve 400Mbps.

Upgrading the kernel from 5.13.0 to 5.15.10 does not help either. Now I will really give up for today.

Code:
mcuee@ubuntu64:~$ iperf3 -c 192.168.38.216
Connecting to host 192.168.38.216, port 5201
[  5] local 192.168.38.218 port 51296 connected to 192.168.38.216 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec   257 MBytes  2.15 Gbits/sec    0   1.74 MBytes       
[  5]   1.00-2.00   sec   279 MBytes  2.34 Gbits/sec    0   1.74 MBytes       
[  5]   2.00-3.00   sec   279 MBytes  2.34 Gbits/sec    0   1.83 MBytes       
[  5]   3.00-4.00   sec   280 MBytes  2.35 Gbits/sec    0   1.83 MBytes       
[  5]   4.00-5.00   sec   280 MBytes  2.35 Gbits/sec    0   1.83 MBytes       
[  5]   5.00-6.00   sec   279 MBytes  2.34 Gbits/sec    2   1.32 MBytes       
[  5]   6.00-7.00   sec   279 MBytes  2.34 Gbits/sec    0   1.45 MBytes       
[  5]   7.00-8.00   sec   280 MBytes  2.35 Gbits/sec    0   1.45 MBytes       
[  5]   8.00-9.00   sec   280 MBytes  2.35 Gbits/sec    0   1.61 MBytes       
[  5]   9.00-10.00  sec   279 MBytes  2.34 Gbits/sec    0   1.61 MBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  2.71 GBytes  2.32 Gbits/sec    2             sender
[  5]   0.00-10.01  sec  2.70 GBytes  2.32 Gbits/sec                  receiver

iperf Done.

mcuee@ubuntu64:~$ iperf3 -c 192.168.38.216 -R
Connecting to host 192.168.38.216, port 5201
Reverse mode, remote host 192.168.38.216 is sending
[  5] local 192.168.38.218 port 51300 connected to 192.168.38.216 port 5201
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.00   sec  34.1 MBytes   286 Mbits/sec                 
[  5]   1.00-2.00   sec  36.2 MBytes   304 Mbits/sec                 
[  5]   2.00-3.00   sec  44.3 MBytes   372 Mbits/sec                 
[  5]   3.00-4.00   sec  39.0 MBytes   327 Mbits/sec                 
[  5]   4.00-5.00   sec  40.9 MBytes   343 Mbits/sec                 
[  5]   5.00-6.00   sec  49.0 MBytes   411 Mbits/sec                 
[  5]   6.00-7.00   sec  33.2 MBytes   278 Mbits/sec                 
[  5]   7.00-8.00   sec  43.6 MBytes   366 Mbits/sec                 
[  5]   8.00-9.00   sec  50.0 MBytes   419 Mbits/sec                 
[  5]   9.00-10.00  sec  34.6 MBytes   290 Mbits/sec                 
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec   406 MBytes   340 Mbits/sec  118             sender
[  5]   0.00-10.00  sec   405 MBytes   340 Mbits/sec                  receiver

iperf Done.
mcuee@ubuntu64:~$ uname -a
Linux ubuntu64 5.15.10-051510-generic #202112171031 SMP Fri Dec 17 10:53:39 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
 

xiaofan

High Supremacy Member
Joined
Sep 16, 2018
Messages
32,726
Reaction score
10,241
So I have theee different Rx results with different PC (Tx is all good)

1) PVE (Debian based) with kernel 5.13 --> Rx is good at 2.32Gbps.

2) New Acer Swift 3 laptop dual boot with Ubuntu 20.04, 1.15 Gbps Rx speed.

3) The worst seems to be the Intel J4105 mini PC which runs Ubuntu 21.10, it can not even achieve 400Mbps.

Upgrading the kernel from 5.13.0 to 5.15.10 does not help either. Now I will really give up for today.

Decided to give the vendor a try and now it seems to work for the J4105 Mini PC.

Code:
mcuee@ubuntu64:/sys/class/net/enx2c16dba12018/queues$ lsusb -t
/:  Bus 02.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/7p, 5000M
    |__ Port 1: Dev 3, If 0, Class=Vendor Specific Class, Driver=r8152, 5000M
/:  Bus 01.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/9p, 480M
mcuee@ubuntu64:/sys/class/net/enx2c16dba12018/queues$ iperf3 -c 192.168.38.216
Connecting to host 192.168.38.216, port 5201
[  5] local 192.168.38.218 port 51876 connected to 192.168.38.216 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec   261 MBytes  2.18 Gbits/sec    0   1.34 MBytes      
[  5]   1.00-2.00   sec   279 MBytes  2.34 Gbits/sec    1   1.01 MBytes      
[  5]   2.00-3.00   sec   280 MBytes  2.35 Gbits/sec    0   1.11 MBytes      
[  5]   3.00-4.00   sec   280 MBytes  2.35 Gbits/sec    0   1.19 MBytes      
[  5]   4.00-5.00   sec   281 MBytes  2.36 Gbits/sec    0   1.24 MBytes      
[  5]   5.00-6.00   sec   279 MBytes  2.34 Gbits/sec    1    959 KBytes      
[  5]   6.00-7.00   sec   281 MBytes  2.36 Gbits/sec    0   1.00 MBytes      
[  5]   7.00-8.00   sec   279 MBytes  2.34 Gbits/sec    1    863 KBytes      
[  5]   8.00-9.00   sec   281 MBytes  2.36 Gbits/sec    0    952 KBytes      
[  5]   9.00-10.00  sec   280 MBytes  2.35 Gbits/sec    0   1.01 MBytes      
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  2.72 GBytes  2.33 Gbits/sec    3             sender
[  5]   0.00-10.00  sec  2.71 GBytes  2.33 Gbits/sec                  receiver

iperf Done.
mcuee@ubuntu64:/sys/class/net/enx2c16dba12018/queues$ iperf3 -c 192.168.38.216 -R
Connecting to host 192.168.38.216, port 5201
Reverse mode, remote host 192.168.38.216 is sending
[  5] local 192.168.38.218 port 51880 connected to 192.168.38.216 port 5201
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.00   sec   276 MBytes  2.32 Gbits/sec                
[  5]   1.00-2.00   sec   277 MBytes  2.32 Gbits/sec                
[  5]   2.00-3.00   sec   279 MBytes  2.34 Gbits/sec                
[  5]   3.00-4.00   sec   279 MBytes  2.34 Gbits/sec                
[  5]   4.00-5.00   sec   280 MBytes  2.35 Gbits/sec                
[  5]   5.00-6.00   sec   279 MBytes  2.34 Gbits/sec                
[  5]   6.00-7.00   sec   280 MBytes  2.35 Gbits/sec                
[  5]   7.00-8.00   sec   280 MBytes  2.35 Gbits/sec                
[  5]   8.00-9.00   sec   280 MBytes  2.35 Gbits/sec                
[  5]   9.00-10.00  sec   279 MBytes  2.34 Gbits/sec                
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  2.73 GBytes  2.34 Gbits/sec    0             sender
[  5]   0.00-10.00  sec  2.73 GBytes  2.34 Gbits/sec                  receiver

iperf Done.
mcuee@ubuntu64:/sys/class/net/enx2c16dba12018/queues$ uname -a
Linux ubuntu64 5.13.0-22-generic #22-Ubuntu SMP Fri Nov 5 13:21:36 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
 
Last edited:

xiaofan

High Supremacy Member
Joined
Sep 16, 2018
Messages
32,726
Reaction score
10,241
Decided to give the vendor a try and now it seems to work for the J4105 Mini PC.

However it does not work on my new Swift 3 laptop running Ubuntu 20.04. So much for today.

Code:
mcuee@UbuntuSwift3:~$ lsusb -t
/:  Bus 04.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/4p, 10000M
    |__ Port 1: Dev 3, If 0, Class=Vendor Specific Class, Driver=r8152, 5000M
/:  Bus 03.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/12p, 480M
    |__ Port 3: Dev 2, If 0, Class=Human Interface Device, Driver=usbhid, 12M
    |__ Port 3: Dev 2, If 1, Class=Human Interface Device, Driver=usbhid, 12M
    |__ Port 5: Dev 3, If 0, Class=Video, Driver=uvcvideo, 480M
    |__ Port 5: Dev 3, If 1, Class=Video, Driver=uvcvideo, 480M
    |__ Port 7: Dev 4, If 0, Class=Vendor Specific Class, Driver=, 480M
    |__ Port 10: Dev 5, If 0, Class=Wireless, Driver=btusb, 12M
    |__ Port 10: Dev 5, If 1, Class=Wireless, Driver=btusb, 12M
/:  Bus 02.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/4p, 10000M
/:  Bus 01.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/1p, 480M
mcuee@UbuntuSwift3:~$ iperf3 -c 192.168.38.218
Connecting to host 192.168.38.218, port 5201
[  5] local 192.168.38.98 port 38332 connected to 192.168.38.218 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec   264 MBytes  2.21 Gbits/sec   16    530 KBytes       
[  5]   1.00-2.00   sec   270 MBytes  2.26 Gbits/sec    0    587 KBytes       
[  5]   2.00-3.00   sec   262 MBytes  2.20 Gbits/sec    0    614 KBytes       
[  5]   3.00-4.00   sec   237 MBytes  1.99 Gbits/sec    0    621 KBytes       
[  5]   4.00-5.00   sec   248 MBytes  2.08 Gbits/sec    0    638 KBytes       
[  5]   5.00-6.00   sec   262 MBytes  2.20 Gbits/sec    0    643 KBytes       
[  5]   6.00-7.00   sec   241 MBytes  2.02 Gbits/sec    0    646 KBytes       
[  5]   7.00-8.00   sec   247 MBytes  2.07 Gbits/sec    0    648 KBytes       
[  5]   8.00-9.00   sec   253 MBytes  2.12 Gbits/sec    0    655 KBytes       
[  5]   9.00-10.00  sec   266 MBytes  2.23 Gbits/sec    0    655 KBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  2.49 GBytes  2.14 Gbits/sec   16             sender
[  5]   0.00-10.00  sec  2.49 GBytes  2.14 Gbits/sec                  receiver

iperf Done.
mcuee@UbuntuSwift3:~$ iperf3 -c 192.168.38.218 -R
Connecting to host 192.168.38.218, port 5201
Reverse mode, remote host 192.168.38.218 is sending
[  5] local 192.168.38.98 port 38336 connected to 192.168.38.218 port 5201
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.00   sec  99.6 MBytes   836 Mbits/sec                 
[  5]   1.00-2.00   sec   112 MBytes   937 Mbits/sec                 
[  5]   2.00-3.00   sec   110 MBytes   925 Mbits/sec                 
[  5]   3.00-4.00   sec   109 MBytes   916 Mbits/sec                 
[  5]   4.00-5.00   sec   112 MBytes   943 Mbits/sec                 
[  5]   5.00-6.00   sec   107 MBytes   902 Mbits/sec                 
[  5]   6.00-7.00   sec   113 MBytes   950 Mbits/sec                 
[  5]   7.00-8.00   sec   114 MBytes   954 Mbits/sec                 
[  5]   8.00-9.00   sec   113 MBytes   944 Mbits/sec                 
[  5]   9.00-10.00  sec   115 MBytes   961 Mbits/sec                 
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  1.08 GBytes   929 Mbits/sec  1088             sender
[  5]   0.00-10.00  sec  1.08 GBytes   927 Mbits/sec                  receiver

iperf Done.
 

xiaofan

High Supremacy Member
Joined
Sep 16, 2018
Messages
32,726
Reaction score
10,241
Even though the RTL8156B based 2.5G adapters more or less work now with my PVE 7.1 host (Intel N4500 based Chuwi Herobox Pro) and the Intel J4105 mini PC, I feel the adpaters are hot, so looks like the cooling is not good enough.

Therefore I just got another Ugreen gigabit adapter (ASIX AX88179 based) for the PVE host. This one is known to work with my Raspberry Pi 400 OpenWRT router as WAN port running 24/7.

iperf3 results from the PVE 7.1 host and my J4105 mini PC running Ubuntu 21.10.

Code:
mcuee@ubuntu64:~$ iperf3 -c 192.168.18.216
Connecting to host 192.168.18.216, port 5201
[  5] local 192.168.18.169 port 54130 connected to 192.168.18.216 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec   111 MBytes   928 Mbits/sec    0    590 KBytes
[  5]   1.00-2.00   sec   113 MBytes   944 Mbits/sec   17    461 KBytes
[  5]   2.00-3.00   sec   112 MBytes   943 Mbits/sec    0    520 KBytes
[  5]   3.00-4.00   sec   111 MBytes   934 Mbits/sec    6    402 KBytes
[  5]   4.00-5.00   sec   112 MBytes   944 Mbits/sec    0    510 KBytes
[  5]   5.00-6.00   sec   112 MBytes   944 Mbits/sec    0    523 KBytes
[  5]   6.00-7.00   sec   112 MBytes   943 Mbits/sec    0    526 KBytes
[  5]   7.00-8.00   sec   111 MBytes   934 Mbits/sec    0    533 KBytes
[  5]   8.00-9.00   sec   112 MBytes   944 Mbits/sec    0    537 KBytes
[  5]   9.00-10.00  sec   112 MBytes   944 Mbits/sec    6    380 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  1.09 GBytes   940 Mbits/sec   29             sender
[  5]   0.00-10.00  sec  1.09 GBytes   937 Mbits/sec                  receiver

iperf Done.
mcuee@ubuntu64:~$ iperf3 -c 192.168.18.216 -R
Connecting to host 192.168.18.216, port 5201
Reverse mode, remote host 192.168.18.216 is sending
[  5] local 192.168.18.169 port 54138 connected to 192.168.18.216 port 5201
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.00   sec   112 MBytes   941 Mbits/sec
[  5]   1.00-2.00   sec   112 MBytes   941 Mbits/sec
[  5]   2.00-3.00   sec   112 MBytes   942 Mbits/sec
[  5]   3.00-4.00   sec   112 MBytes   941 Mbits/sec
[  5]   4.00-5.00   sec   112 MBytes   941 Mbits/sec
[  5]   5.00-6.00   sec   112 MBytes   941 Mbits/sec
[  5]   6.00-7.00   sec   112 MBytes   941 Mbits/sec
[  5]   7.00-8.00   sec   112 MBytes   941 Mbits/sec
[  5]   8.00-9.00   sec   112 MBytes   941 Mbits/sec
[  5]   9.00-10.00  sec   112 MBytes   941 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  1.10 GBytes   944 Mbits/sec    0             sender
[  5]   0.00-10.00  sec  1.10 GBytes   941 Mbits/sec                  receiver

iperf Done.
 

bert64

Senior Member
Joined
Jan 20, 2020
Messages
1,027
Reaction score
539
If you get poor performance in one direction, try changing the flow control setting either on the card or the switch?
Also try with a direct cable between two hosts instead of via a switch and see if the problem goes away.
I've had issues with broken flow control on various combinations of ethernet controller and switches over the years, and it usually manifests either in poor performance in one direction or sometimes in extreme cases frequent carrier drops under load.
Also try disabling the various offload functions (frequently buggy either in drivers or hardware) or trying IPv6 (no ip header checksums to worry about and simpler implementation) to see if that makes a difference.

Different drivers and operating systems sometimes perform differently not because of inherent differences in the drivers, but just different defaults. For instance Linux enabled TCP window scaling many years ago while Windows did not enable it by default until Vista (or more commonly 7 since Vista was widely derided). This manifested itself with some broken sites rejecting connections because they didn't recognise the window scaling flags.
 

xiaofan

High Supremacy Member
Joined
Sep 16, 2018
Messages
32,726
Reaction score
10,241
If you get poor performance in one direction, try changing the flow control setting either on the card or the switch?
Also try with a direct cable between two hosts instead of via a switch and see if the problem goes away.
I've had issues with broken flow control on various combinations of ethernet controller and switches over the years, and it usually manifests either in poor performance in one direction or sometimes in extreme cases frequent carrier drops under load.
Also try disabling the various offload functions (frequently buggy either in drivers or hardware) or trying IPv6 (no ip header checksums to worry about and simpler implementation) to see if that makes a difference.
Different drivers and operating systems sometimes perform differently not because of inherent differences in the drivers, but just different defaults. For instance Linux enabled TCP window scaling many years ago while Windows did not enable it by default until Vista (or more commonly 7 since Vista was widely derided). This manifested itself with some broken sites rejecting connections because they didn't recognise the window scaling flags.

For my later tests they are all between two computers with direct connection and no switches are involved.

For example, the remaining issue is with the Acer laptop (dual boot) when running Ubuntu 20.04 (it does not seem to work with either vendor driver or the kernel built-in cdc-ncm driver despite using later kernels which officially support the RTL8156B).

My test is between this one and the other Acer laptop running Windows 11.

I have the 2.5Gbe unmanaged switch but I have not used it yet in all these tests. And yes I have read that some switches (including 10G switches which are also capable of 2.5G and 5G) which may create the issues.

I will still use the 2.5G adapters as temporary connections but I will not use them as permanent connection as of now due to the thermal concerns.

I am reluctant to get desktop PCs now to use proper 2.5G or 10G PCIE adapters. That is why I am looking at USB based solutions. Unfortunately the cheaper ones are all based on RTL8156B.
 
Last edited:

bert64

Senior Member
Joined
Jan 20, 2020
Messages
1,027
Reaction score
539
For my later tests they are all between two computers with direct connection and no switches are involved.

For example, the remaining issue is with the Acer laptop (dual boot) when running Ubuntu 21.04 (it does not seem to work with either vendor driver or the kernel built-in cdc-ncm driver despite using later kernels which officially support the RTL8156B).

My test is between this one and the other Acer laptop running Windows 11.

I have the 2.5Gbe unmanaged switch but I have not used it yet in all these tests. And yes I have read that some switches (including 10G switches which are also capable of 2.5G and 5G) which may create the issues.

I will still use the 2.5G adapters as temporary connections but I will not use them as permanent connection as of now due to the thermal concerns.

I am reluctant to get desktop PCs now to use proper 2.5G or 10G PCIE adapters. That is why I am looking at USB based solutions. Unfortunately the cheaper ones are all based on RTL8156B.
Yeah, decent 10GB nics are in the 300+ range. I have absolutely no need for 10gb speeds and 99% of what i do would even work fine with 100mbps, but would love to upgrade anyway just because - were it not for the price premium.

I early adopted 100mbps and 1gbps back in the days before they became mainstream but once the price gap was reasonable. Way back when, you used to need a high end SCSI disk array and multiple clients to saturate a 100mbps file server.

These problems are nothing new, even in the 100mbps days you had rock solid chipsets like the dec tulip that cost a premium, cheap realtek chipsets and various flakey noname clone chipsets which had even worse problems.
 
Important Forum Advisory Note
This forum is moderated by volunteer moderators who will react only to members' feedback on posts. Moderators are not employees or representatives of HWZ Forums. Forum members and moderators are responsible for their own posts. Please refer to our Community Guidelines and Standards and Terms and Conditions for more information.
Top