Hyper-V Virtual Machine Very Slow Network – VMQ – Broadcom

Download PDF

So couple of days ago I was setting up our new Dell Server running Server 2008 R2 operating system and a Hyper-V role.

After the initial setup was done and Hyper-v role configured it was time to install it’s first VM running Server 2008 R2 as well. Installation went through properly, added all required modules, apps, configs etc.

All good, but when I tried to login to it via one of the workstations I got about 200kbps network speed. At first I thought it could be something wrong with the workstation so I jumped to another client, but the speed was the same. Then thought maybe there is something wrong with the switch that network is using, but on the other hand internet speed was well above 20MBps. Hmmm..

I did another lan speed test but this time between the client and the Hyper-V physical server, the speed was almost a full gigabit! Maybe the ethernet port used by the VM was faulty so i merged both Hyper-V and the VM through the same ethernet port, but still no improvement.

At this point I was really confused (pissed off). I have installed many many VMs using Hyper-V platform, but I never had a similar problem.

To cut the story short, after few hours of troubleshooting I found what was causing the issue. In the Broadcom Network Card configuration “Virtual Machine Queuing” was enabled, which is fine since I have used it before but never had any issues. For some reason I found that the Virtual Machine Queuing” option creates issues with the Broadcom cards using drivers from October. 2012 (mostly used by Dell Servers) After contacting Broadcom they confirmed the same! Well, thanks for wasting my time!

As soon as i disabled the option, VM network speed was great!

Virtual Machine Queue

Hope someone finds this useful!

 

“Virtual machine queue (VMQ) is a feature available to computers running Windows Server 2008 R2 with the Hyper-V server role installed, that have VMQ-capable network hardware. VMQ uses hardware packet filtering to deliver packet data from an external virtual machine network directly to virtual machines, which reduces the overhead of routing packets and copying them from the management operating system to the virtual machine.

 

Requirements for VMQ


  • The management operating system must be running Windows Server 2008 R2.
  • The physical network adapter handling the traffic on the management operating system must support VMQ.
  • The virtual machines must be running Windows 7or Windows Server 2008 R2 or running Windows Server 2008 or Windows Vista with the Integration Services Setup Disk installed. Virtual machines running earlier versions of Windows cannot use VMQ. “
96 replies
  1. Joan
    Joan says:

    Thanks so much. We had the same problem with a brand new Dell server with Windows 2012.
    Disabling VMQ was enough to work fine.

    Reply
  2. J Morton
    J Morton says:

    THANK YOU so much!! I just setup a new Dell server that was pretty beefy. I created some VMs just like I had on my older Dell server and the network performance was terrible. I changed the setting you mentioned and voila – everything is working so much better.

    I really appreicate it!

    Reply
  3. Mat Berube
    Mat Berube says:

    Yes!!!! Thank you! Brand new Dell Server with 2012 and 2 VM’s. Ran through all the same initial troubleshooting steps you did. Sure glad I found this post, client was getting pretty antsy that his new “state-of-the-art” system was slower than his old SBS 2003 box 🙂

    Reply
    • OSmicro Admin
      OSmicro Admin says:

      No worries mate glad you got it working! After this one, I have setup at least a dozen of different Dell servers and they all had the same issue, every single one of them.
      Disabling VMQ works, but the proper way would be to upgrade Network Card drivers/Firmware. I have not had a chance to look into it yet, but if I find patched driver i’ll put a link up here.

      Reply
  4. Nimbus Dave
    Nimbus Dave says:

    I’ll join the chorus of thanks here – you’ve saved me a lot of anguish! A new Dell server running Server 2012 Hyper-V with 2 VMs – network throughput very, very slow (< 100 KB/s). Tried various things before finding this post and it worked a treat. Such a serious flaw!

    Thanks again – very much appreciated.

    Reply
  5. Randy
    Randy says:

    Another thank you although mine is a Realtek chipset. Also the offending option is called “Priority & VLAN”. I changed the settings to “Priority & VLAN Disabled”. All is well now.

    Reply
  6. Geoff
    Geoff says:

    Thanks for the information. I changed the setting to Disabled on my Server 2012 host and the 2003 R2 client is still running super slow. Any other thoughts? When I ping that IP address, it ranges between 50-100. I have 4 GB memory allocated and it is running on a fixed VHDX. I’m still stumped! thanks for your post and for any tips you can offer.

    Reply
    • Ken Johnson
      Ken Johnson says:

      I’ve seen where one or more of the “Offload” settings for the Broadcom NIC caused slow speeds for a physical 2003 server. You might want to try disabling those one at a time to see if it helps the responsiveness. You may need to change the “Offload” setting for the Virtual NIC inside the VM rather than on the host. I’m not sure since I never installed a 2003 server in a 2012 VM.

      Reply
      • geoffroise
        geoffroise says:

        I do have SP2 installed and the latest integration services. I disabled the following:
        IPv4 Checksum Offload
        Large Send Offload Version 1 (IPv4)
        TCP Checksum Offload (IPv4)
        TCP Checksum Offload (IPv6)
        UDP Checksum Offload (IPv4)
        UDP Checksum Offload (IPv6)

        The ping time hasn’t changed between the VM host (server 2012) and Client (2003 R2) at 20ms. With other clients, it is <1ms. The change in settings hasn't seemed to help.

        Thanks for the help and comments. Any other ideas?

        Reply
  7. Jeffrey
    Jeffrey says:

    Thanks for sharing this information, this problem was irritating like hell! Problem after problem.. now finally fixed thanks of you!

    Reply
    • Andrej OSmicro
      Andrej OSmicro says:

      🙂 glad many people find it useful, it is crappy when such a small option can cause so much trouble. Hopefully someone from Dell reads this and actually do something about it.

      Reply
  8. Ken Johnson
    Ken Johnson says:

    Thank you so much for this post. It really saved us. Who would have thought that Virtual Machine Queue needed to be disabled for Virtual Machines to communicate properly.

    Reply
  9. Erana
    Erana says:

    Excellent post!
    Luckily enough it was the first post I tried and it worked.
    Keep up the good work mate!

    Reply
  10. Anderson
    Anderson says:

    Thanks! It really worked! In my case, i am using a Hyper-V 2012 Server Core. You can change VMQ with PowerShell:
    Get-NetAdapter
    Get-NetAdapterAdvancedProperty NIC1
    Set-NetAdapaterAdvancedProperty NIC1 -DisplayName “Virtual Machine Queues” -DisplayValue Disabled

    Reply
  11. CHRISTIAN
    CHRISTIAN says:

    Just for information, I had the same problem with HyperV3 on a Windows 2012 Server. I use a recent driver and this “VM Queue” property is still enabled by default, causing very slow connections. Disabling it resolve my problem. Thanks because I’ve wasted hours searching mistakes in my VLAN configuration, installing KB, …..

    Reply
  12. Keith Spencer
    Keith Spencer says:

    I hade the same problem with a Dell 2012 server and disabling VMQ on the Broadcom NICs fixed it. Thanks!!!!!

    Reply
  13. Ulrich
    Ulrich says:

    I had the same Problem with a HP Proliant DL 380 G8 with a QLogic NIC (HP NC523 SFP 2-Port 10 GB Adapter). Disabling the VMQ works fine. Thanks so lot.

    Reply
  14. Claudio
    Claudio says:

    Thanks, pal! I have been searching for this for the last four days, and the bad vm performance in our new server was getting me nuts. You saved my life. I love you.

    Reply
  15. thanks
    thanks says:

    Just the solution I was looking for! Thank you so much!

    Note that my Win 2012r2 Dell server has a dual onboard Broadcom NIC. I use one port for the Hyper-V management interface and the other for the virtual switch. This appears as two adapters in Device Manager. It is possible to disable VMQ on just the switch-bound adapter, but my VMs did not seem to speed up until I disabled it on the management adapter too.

    Reply
  16. Bryan
    Bryan says:

    OMG Thank you so much! I was very close to pulling the pin on Hyper-V Server 2012 and booting up the old hardware. Instant results with no reboot.
    Thank you thank you thank you!

    Can’t believe this isn’t on any TechNet articles.

    Reply
  17. Nathan Lanchbury
    Nathan Lanchbury says:

    Worth a +1 here. Thank you heaps for documenting this Andrej, It’s help me out in multiple of sites already.

    Reply
  18. R121
    R121 says:

    Hi

    Thanks for the post its a lifesaver, I spent around an hour re checking the Virtual switching and re creating.

    Reply
  19. Joshua Marshall
    Joshua Marshall says:

    Saved me last night! Oddly the 2013 Dell VMs had been running fine for months. However after Windows Updates and a reboot, performance was abysmal. Critical apps weren’t able to run. I thought my weekend was shot but your post was an instant fix. If you’re ever in NYC the beer’s are on me.

    Reply
  20. ITadmin
    ITadmin says:

    I have the same issue with 2012R2 but mine is different. I have a 10gig Intel X520-T2 and all of a sudden performance through it is terrible on the host, but better on the VM??? Would the broadcom setting affect the host performance?

    Reply
    • Andrej OSmicro
      Andrej OSmicro says:

      Yes, disable IPv4 offloading as well and see if it helps. I usually have to play around and tune it until it works how I want it. So I would start by disabling IPV4 Large Send Offload and Checksum Offload. Try it and see how it works. Also QOS and IPv6 unless you use it.

      Reply
  21. ECL
    ECL says:

    Hi Andrej,

    Nice post thanks

    I have a small setup of 2 x HyperV servers in a Workgroup connected to a single switch (1GB);

    Host 1 – Dell T320, Windows 2102, HyperV, on board Broadcom Dual NIC’s teamed

    Host 2 – Dell T420, Windows 2012 R2, HyperV, on board Broadcom Dual NIC’s teamed, Separate Dual Broadcom NIC’s teamed.

    *all of the above Teamed in windows 2012 and HyperV

    Since enabling HyperV Switch the network is really slow and having problems RDP and copying files directly from host1 to host2. or anything outside the 2 hosts.

    I have disabled as sugguested the VMQ on all NIC’s and in the Teams in both hosts but its still very slow copying files between hosts is like 3mbs.

    However, from host2 I can RDP fast to it’s own hosted VM’s and the VM’s have fast RDP to the other Hosts and copy files is faster.

    While troubleshooting I I found if If I break the teaming its fast copying files at 200MBS host to host but RDP still struggles a bit host to host (sort of sticks determining / securing connection and 50/50 chance of connecting). Also slow when using IP6 to RDP from one host to another.

    Any idea what I should try next?

    Thanks

    ECL

    Reply
    • Andrej OSmicro
      Andrej OSmicro says:

      Disable IPv4 offloading as well and see if it helps. I usually have to play around and tune it until it works how I want it. So I would start by disabling IPV4 Large Send Offload and Checksum Offload. Try it and see how it works. Also QOS and IPv6 unless you use it. Let me know if still slow we can try few other things as well.

      Reply
  22. ECL
    ECL says:

    Just to update you all, my RDP and Network is fast now….

    The cause was because I ticked the “All managemnet operating system to share this network adapter” ….no wounder like I said the issue happened after I setup the HyperV Switch….

    Happy days 🙂

    Reply
  23. Randy
    Randy says:

    I am having a similar problem with 2012 R2 and a T320 with dual Broadcom NICs. My problem is that if I push through a LOT of data (multiple GBs) through a VM to the LAN or to a pass through drive, it just stops working. The NIC stops passing data. If I log onto the host, I can disable the NIC then enable it and everything works again for a while. Fortunately, I can remote into the host on the other NIC and do the dis/enabling. I will be trying the WMQ disabling.

    Reply
  24. Max
    Max says:

    Amigo Muito obrigado, estava sofrendo com esse problema a mais de 1 mês, hoje que conseguimos deixar a empresa off pude realizar vários testes ate desmontando o Raid criado, quando achei seu post, realizei o procedimento e voilaa….

    Obrigado mais uma vez….

    Reply
  25. Chris Johnson
    Chris Johnson says:

    Hi

    Was going to disable VMQ this weekend but wanted to check a few things before I did so if you be so kind as to give some advice.

    Am I ok disabling it via the advanced tab within the NIC properties or do I need to use the command line

    Set-NetAdapterVmq -Name “NIC 1″ -Enabled $False

    If I am doing it via the Advanced tab do I just disable Virtual Machine Queues or do I also need to disable the two options below – Shared and VLAN?

    Final question do I also do this on the overall team as well as the individual NIC’s that make up the team?

    Appreciate any feedback and thanks for posting this information.

    Cheers

    Reply
    • Andrej OSmicro
      Andrej OSmicro says:

      Hi Chris,

      Is there a particular reason why you wish to disable VMQ? If you are not experiencing any issues, you don’t have to disable it. However if you do, then just disable VMQ through the GUI (NIC>configure>advanced) – on both physical and virtual adapter depending on the setup. You don’t have to disable anything else as yet, VMQ usually solves the problem. If the issue persist I would look into Large Send Offload and TCP Checksum Offload.
      But to be honest with you, since I have written this post, Broadcom has updated it’s drivers on most of the rack and tower Dell servers, so by doing the update you should fix the issue without disabling anything.

      Please let me know how everything works out and if any problems I am here to help.

      Cheers.

      Reply
  26. Chris Johnson
    Chris Johnson says:

    Thanks for responding Andrej

    We are not experiencing the issues described in this blog however we are experiencing the issue in these articles http://support2.microsoft.com/kb/2986895 & http://www.aidanfinn.com/?p=16876

    So as this article went into disabling I was just checking which options I needed to do.

    I have updated with the latest Broadcom drivers available from Dell’s website which are dated 19/12/2013 version 16.4.0.2 and with the version number and date being before the article on Aidan Finn was posted I thought it would be best to disable VMQ until Dell released some new Broadcom drivers.

    I would go straight to Broadcom however I don’t believe it is recommended?

    Again thanks for responding it is very helpful!

    Cheers

    Reply
    • Andrej OSmicro
      Andrej OSmicro says:

      Hi Chris,

      I have seen this issue before where VMs lose connectivity. Disabling VMQ (via GUI or Shell – same thing) should help it, but make sure you cover both virtual and physical adapter (one may not have this option, but check both anyway). Let me know if it still doesn’t work.

      cheers

      Reply
      • Chris Johnson
        Chris Johnson says:

        PowerEdge R720 with built in Broadcom 5720 NIC and an additional 5719 NIC, knew I should have paid the extra for Intel!

        I have checked the Advanced > Virtual Machine Queues and it is available on the individual NIC’s and the NIC team that the VMSwitch is associated with.

        I will make thins change over the weekend,

        Do you know if I will need to redo any network settings after applying this change? I was going to do the change whilst the VM’s were turned off.

        Cheers

        Reply
        • Andrej OSmicro
          Andrej OSmicro says:

          Hmm, I have several PowerEdge R720 boxes and so far none of them had this problem. I have to get off my laptop now, but I’ll have a look tomorrow what drivers I am using on my servers and let you know.
          In regards to VMQ, there is nothing you need to do after, just disable it and thats it, it shouldn’t even disconnect you (if running remotely). But start by disabling it on the virtual adapter first and try, then hit the physical if it doesn’t work.

          I’ll let you know tomorrow what drivers I run.

          cheers

          Reply
        • Andrej OSmicro
          Andrej OSmicro says:

          One piece of advice, depending on your circumstances, try and get a “test” server for your lab (work, home etc) and test everything you do before putting in production. I recommend getting a Dell 1950 ($200 on eBay), put some old drives in it and replicate anything you do before deciding.

          Reply
          • Chris Johnson
            Chris Johnson says:

            Will do, thanks for the advice Andrej really appreciated.

            Hope you have a great weekend

          • Chris Johnson
            Chris Johnson says:

            This is the current VMQ setup if that could be causing the issue, maybe configured incorrectly?

            NIC4 Enabled BaseVmqProcessor 0:12 MaxProcessors 2 NumbersofRecieve 12
            NIC3 Enabled BaseVmqProcessor 0:8 MaxProcessors 2 NumbersofReceive 12

          • Chris Johnson
            Chris Johnson says:

            Hi Andrej

            Just wanted to let you know I disabled VMQ on the team and NIC members and the VM’s didn’t lose connectivity.

            I will now monitor the situation to see if the VM’s stop responding like they did with VMQ enabled.

            Did you get chance to look at the version of driver you are running?

            Thanks for your help/advice really appreciated.

            🙂

          • Andrej OSmicro
            Andrej OSmicro says:

            Great, hopefully it resolves the connectivity issue.
            Btw, I haven’t had a chance to look into my server, will do tomorrow. But I had a quick look on Dell’s website and drivers for Broadcom NetXtreme cards date 09 Sept 2014. So I would recommend to update them and see how it performs.
            http://www.dell.com/support/home/us/en/04/product-support/product/poweredge-r720/drivers

            Obviously if this is already a production server that approach wouldn’t be recommended before reading driver’s release notes and comments around forums.
            There is a Dell download, it’s an .iso file, huge like 9GB, that contains all sorts of updates. Once you mount the .iso it scans your entire server and detects any updates. Great tool, do a little search and you should be able to find it. Run it on the system and see what comes up with.

            All good mate, happy to help any time.

          • Chris Johnson
            Chris Johnson says:

            Yea that is the version I downloaded unfortunately when installed the driver is dated 19/12/2013 version 16.4.0.2 so I assume Dell took an awful long time testing it!

          • Chris Johnson
            Chris Johnson says:

            Hard to say as it only seemed to happen every 3-4 weeks so going to see how it goes over the next month.

            Will keep you posted.

          • Andrej OSmicro
            Andrej OSmicro says:

            Have you looked inside VM’s events log? Maybe the issue comes from the network side or a particular NIC.
            If you wish, tell me more about the setup and maybe I can be of more help.

          • Chris Johnson
            Chris Johnson says:

            Can’t see anything in the VM event log or host around when the VM’s lose network connectivity, with the problem being exactly like the one explained in the Microsoft article I am going to give this a go for a while to see if the problem resurfaces or not before I mess about with any other settings.

            Ordered one of them Dell servers off eBay by the way so will be able to setup a test environment in the future to test any system changes I am planning.

            Thanks for the help you have given me really appreciated.

          • Andrej OSmicro
            Andrej OSmicro says:

            Great, Dell 1950 is great, one of the best Dell legacy boxes. If you run Server 2012 R2, you can setup a little failover cluster by creating Hyper-V replication. Purchase few cheap Dell boxes like 1950 or a bit higher R410, pack them up with HDD space, RAM and use them as backup servers. If R720 stuffs up (it won’t but you never know) you have a backup server running with up to date VMs.
            There are always great deals on eBay that won’t hurt your budget and will provide great redundancy.
            You can find a good tutorial for Hyper-v replication here:
            http://www.msserverpro.com/configuring-hyper-v-replica-in-windows-server-2012-r2/

            No worries mate it is my pleasure to help any time.
            Let me know how it all works out.

          • Chris Johnson
            Chris Johnson says:

            Sorry forgot to mention we have identical R720’s one is the live host replicating to a Hyper-V replica.

          • Chris Johnson
            Chris Johnson says:

            I had to put a very strong argument together for it!

            Hopefully will never need it but if it comes into action then it will be priceless!

  27. Chris Johnson
    Chris Johnson says:

    Hi Andrej

    Just to let you know still no loss of connectivity on the VM’s since disabling VMQ on the Broadcom NIC’s and NIC Team.

    Over 40 days now which is a record so fingers crossed.

    Thanks again for your help/advice.

    Reply
  28. setwin2012r2
    setwin2012r2 says:

    Hi Andrej,

    I have setup a new vm on a new win server 2012r2, and the network speed is very slow. It has Broadcom netextreme nic. The vm is allocated 4gb memory and 2 virtual cores. The vm is using the host virtual switch. I tried disabling large send offload but the speed is still slow. I’m unable to figure out what is triggering the slowness. Can you please help throw some light on this?

    Thanks in advance for your help.

    Reply
  29. Shawn Burrell
    Shawn Burrell says:

    I was having this problem on a DELL PowerEdge R420 (Broadcom adapter).

    I disabled Virtual Machine Queues and instantly the server was responding properly. Thank you so much!

    Reply
  30. setwin2012r2
    setwin2012r2 says:

    Thank you for your reply Andrej. I have disabled the VMQ on the VM. The host already has the VMQ disabled on both the network adapters. Still the network is slow.. I’m still trying to figure out the time eater on the VM.

    Reply
    • Andrej OSmicro
      Andrej OSmicro says:

      When you say the network is slow, is it between the host and it’s VMs or network in general (workstation>VM / workstation>HOST). Try and identify which path is slow and let me know.
      You can always use free Lan Speed Test to measure file transfer speed. Let me know and I’ll help you out. cheers

      Reply
  31. setwin2012r2
    setwin2012r2 says:

    Thanks Andrej, I will do the LAN speed test and let you know the results on each path. cheers

    Reply
  32. Peter
    Peter says:

    This was the perfect fix on our ML350 Gen9. NIC-teamed all 4 NIC’s in Hyper-V. Disabled this on all NICs and network speed was blazing fast again.

    Reply
  33. Scott Dubak
    Scott Dubak says:

    2012R2 PowerShell command that disable it on all NICS from a remote computer.

    ICM -computername ServerXYZ -Scriptblock {Get-NetAdapterAdvancedProperty -Displayname “Virtual Machine Queues”;Set-NetAdapterAdvancedProperty -Displayname “Virtual Machine Queues” -Registryvalue “0”;Get-NetAdapterAdvancedProperty -Displayname “Virtual Machine Queues”}

    Reply

Trackbacks & Pingbacks

  1. […] I had this issue with Broadcom cards. The problem was virtual machine queues. Hyper-V Virtual Machine Very Slow Network – VMQ […]

  2. […] VMQ disabled.  I did a little research and found this article that referenced the whole thing, Hyper-V Virtual Machine Very Slow Network – VMQ.  Also Checked Technet regarding VMQ and found this […]

Leave a Reply

Want to join the discussion?
Feel free to contribute!

Leave a Reply