Huge Packet Drops (Tx drops) Observed on NetScaler

Source https://support.citrix.com/article/CTX215843

Shell Command : nsconmsg101 -K newnslog -g nic_err -s disptime=1 -d current

Shell Command : nsconmsg101 -K newnslog -g nic_err -s disptime=1 -d statswt0

Article | NetworkingPerformance | Created: 15 Dec 2014 | Modified: 11 Aug 2016

Applicable Products

NetScaler

Symptoms or Error

Huge packet drops was observed on NetScaler. The drops are TX drops (software drops).
Customer attempted to ping NSIP and could see that most of the packet gets dropped. All configured services goes down and comes up frequently. Users could see that application gets disconnected very frequently (Applications have been launched through NetScaler Gateway).

We can see from newnslog that "nic_err_congestionlimit_pkts_dropped" and "nic_err_tx_dropped" counters" are incrementing:

/var/nslog]$ nsconmsg101 -K newnslog -g nic_err -s disptime=1 -d current | grep congest |more
   6195       0        6159352      48887     6944 nic_err_congestionlimit_pkts_dropped interface(0/1) Wed Dec  3 17:17:38 2014
   6201       0        6208229      48877     6962 nic_err_congestionlimit_pkts_dropped interface(0/1) Wed Dec  3 17:17:45 2014
   6402       0        6257108      48879     6943 nic_err_congestionlimit_pkts_dropped interface(0/1) Wed Dec  3 17:25:32 2014
   6462       0        6305991      48883     6953 nic_err_congestionlimit_pkts_dropped interface(0/1) Wed Dec  3 17:27:45 2014
   6468       0        6354868      48877     6932 nic_err_congestionlimit_pkts_dropped interface(0/1) Wed Dec  3 17:27:52 2014
   6805       0        6403744      48876     6942 nic_err_congestionlimit_pkts_dropped interface(0/1) Wed Dec  3 17:40:55 2014
/var/nslog]$ nsconmsg101 -K newnslog -g nic_err_tx_dropped -s time=04Dec2014:17:00 -s disptime=1 -d current | more
 
reltime:mili second between two records Thu Dec  4 17:06:08 2014
  Index   rtime totalcount-val      delta rate/sec symbol-name&device-no&time
      0 2697427        8897070      48877     6942 nic_err_tx_dropped interface(0/1) Thu Dec  4 17:06:08 2014
      1    7040        8945947      48877     6942 nic_err_tx_dropped interface(0/1) Thu Dec  4 17:06:15 2014
      2    7170        8994878      48931     6824 nic_err_tx_dropped interface(0/1) Thu Dec  4 17:06:22 2014
      3   78151        9092633      97755    13885 nic_err_tx_dropped interface(0/1) Thu Dec  4 17:07:40 2014

Solution

Ensure that you are not using Management port (interface 0/1) for production traffic.

The best practice is to use management card only for Management traffic and the Network card for the Production traffic.

Management interfaces uses software RSS instead of hardware RSS, so this can impact performance.

Enabled the interface 1/1 and then disabled the interface 0/1.

Problem Cause

Management port (interface 0/1) used for production/data traffic.

NetScaler VPX Loses Network Connectivity on VMware ESXi 5.1.0 2191751, VMware ESXi 5.5 2143827 and also on VMware ESXi 6.0

Source https://support.citrix.com/article/CTX200278

Article | ConnectivityNetworkingThird PartyFixed Issues | 163 found this helpful | Created: 05 Nov 2014 | Modified: 29 Sep 2016

Applicable Products

NetScaler

Symptoms or Error

The following are the issues encountered when NetScaler VPX is run on VMware ESXi 5.1.0 2191751, VMware ESXi 5.5 2143827 and also on VMware ESXi 6.0

Network disconnectivity.
NetScaler GUI access is affected.
When running show interfaces command from NetScaler CLI you might notice that one of the interfaces has stalls and errors.

Solution

To fix this issue upgrade to NetScaler 10.5 build 55.8. This firmware can be download from Citrix Downloads.

For more information, see Fixed Issues section from NetScaler 10.5 Build 55.8 Release Notes.

The following are the workarounds for this issue:

Workaround 1

Install NetScaler VPX appliance on a previously known good VMware version and build.

Workaround 2

SSH and log on to Citrix NetScaler VPX appliance as nsroot.
Type shell.
Change directory (cd) to /flash/boot.
Create file /flash/boot/loader.conf.local (if not present) with same permissions as/flash/boot/loader.conf. Add the following line and reboot:
hw.em.txd=512
Note: To create the file, use command touch loader.conf.local.

vi Commands

The following are the vi commands to edit the document:

From NetScaler shell type:
vi <filename>
Move the cursor to the last character of text in the file, type "a" and click Enter.
Type the line:
hw.em.txd=512
Press the ESC key and then ":" key. The cursor will move to the bottom of the page, then type wq!.

Problem Cause

NetScaler VPX network connectivity issue on VMware ESXi 5.1.0 build 2191751 and VMware ESXi 5.5 build 2143827 is caused by "tx_ring_length" mismatch, which causes TX stalls.

Additional Resources

For more information refer to VMware Knowledge Base - 2092809.

====================== End