How To Debug and Troubleshoot VOIP

http://www.voip-info.org/wiki/view/How+To+Debug+and+Troubleshoot+VOIP

(SIP, MGCP, H.323, RTP, Skinny etc.)

One of the primary techniques is to view what is actually getting sent and received by VOIP devices. There are several ways to do this:

Monitor Ethernet Traffic
Debugging displays from a VOIP program

It helps to understand whats supposed to be happening. Studying the relevant RFC s and other protocol documents and tutorials is helpful.

Ethernet Monitoring Tools

ClarifiedNetworks
- HowNetWorks - a free VMWare appliance
- Tia - sophisticated monitoring and flow analysis tool with visualisation, multiple data sources, ...
List of Monitoring and sniffing software
ngrep (Available for Linux, Windows, Apple, BSD, etc.)
- Dumps only the ASCII portion of packets, excellent for ASCII based protocols
sngrep (Available for Linux, Apple, BSD.) by Irontec
- Curses based SIP dialogs monitoring tool.
- Support UDP, TCP and TLS.
- Graphical(curses) dialog summary and dialog flow detail.
- Very useful for debugging and learning purposes.
- GPL License
Packetyzer: User-friendly packet sniffer for Windows, supports SIP
Rate which provides real time packet-per-second and data transfer rates
SIP Workbench Displays SIP ladder diagram from WireShark/pcap captures
- Displays STUN/TURN interactions
- Allow users to filter on particular call flows
Spirent Communications - Test Solutions for VoIP networks and devices
STINGA SS7 Protocol Analyzer and Monitoring System
- Call trace, mature SS7 protocol decodes
- SIP, SS7, SIP-T, SIGTRAN, ISDN, MGCP and Megaco/H.248
- Performance analysis
- Open MySQL database
TamoSoft CommView - network analyzer for Windows
- Real-time VoIP call monitoring
- SIP and H.323 analysis and decoding, call playback
- Jitter, QoS, Bandwidth charts
tcpdump (standard utility in most Linux distributions)
Touchstone
- WinEyeQ
  - 100% software-based
  - monitors/analyzes/records/replays SIP and H.323 traffic, audio/video media and QOS.
- TraceBuster (free version available)
  - records/replays SIP and H.323 traffic, audio/video media and QOS.
VoIPmonitor is open source live network packet sniffer which analyze SIP and RTP protocol.
- Predicts MOS-LQE score according to ITU-T G.107 E-model
- Detailed delay/loss/MOS statistics stored to MySQL
- Each call is saved as standalone pcap file
- Possibilities to convert pcap to WAV
WinDump - tcpdump for Windows
Wireshark (Ex Ethereal) (Open Source and available for Linux, Windows, Apple, BSD, etc.)
- Support for decoding many VOIP protocols is included (including IAX)
- VoIP call analysis and call flow diagram for (SIP, MGCP, H323, etc.)
- RTP Statistics and graph analysis (jitter, delay, packet lost, etc.)
- RTP playback (Wireshark only)
SIPDump - an open source c# tool for logging calls to pcap files. SIP and RTP. Very early development. Runs on windows or Linux with mono.
StarTrinity SIP Tester - Freeware SIP and RTP monitoring tool with G.107 MOS/R-factor, RFC3550 global max jitter, realtime charts and reports.

Built-in Debug Tools

Asterisk Use the sip debug command
linphone Outputs useful diagnostics to the console as it uses the oSIP library
pjsua A command line SIP user agent from pjsip.org, available for various platforms, and very useful to debug SIP functionalities (call, presence, instant messaging, etc.) as well as media quality via RTCP statistics.
xten The x-lite and x-pro SIP soft phones have a buit-in display and decoding of received and sent SIP packets (Hit F9 to activate)
miniSipServer has inner trace tools to capture all SIP messages.

Traffic generators

Candela Technologies LANForge FIRE VOIP/RTP/PESQ call generator
Codima autoVoIP Blaster - Ensure VoIP Readiness - Stress Test Network Limits
Empirix Signaling and Media load and feature testing
GL Communications
- PacketGen - generates SIP calls with or without RTP traffic
- PacketScan - monitor, collect, and analyze QoS statistics on VOIP traffic
Integrated Research Prognosis will simulate, record and analyze VOIP traffic in real time.
Iperf creates network traffic and measures performance
- Can be used to test a network to see how it might perform with increased VOIP traffic
Ixia VOIP traffic generators and Network assessment tools
SIP Inspector Pro combines the best when in it comes to functionality and ease of use. This tool is a must have when it comes to working with SIP.
- Can create custom SIP signaling scenarios, messages
- Can play RTP from saved pcap file
- Unconditionally streams RTP
- Can act as a client and a server
- Transport protocols: UDP, TCP, websocket
Nexus Telecom Load testing, modular system
- 380,000 calls per hour, per module
- 4Mbps RTP real media per module
- Up to 400,000 simulated subscribers
MyVoIPSpeed simulates VoIP traffic over your Internet connection, measures key diagnostics including Jitter and Packet Loss, and provides an analysis of the voice quality
PacketIsland 4"x4" in-line micro-appliances used in a distributed multi-site enterprise or SME to generate live VoIP traffic and measure loss, jitter, MOS, route performance, route flaps, etc. Also measures ongoing data traffic in network.
pjsip-perf Open source call generator from pjsip.org to measure SIP call/transaction performance.
Sipp SIP Performance Test Tool - Performance tester for SIP
Spirent Communications - Test Solutions for VoIP networks and devices
Touchstone 100% software-based VoIP and video verification tools.
- WinSIP - SIP signaling and Audio/Video media generator
- Win323 - H.323 signaling and Audio/Video media generator
Valid8.com Valid8.com is a leading provider of SIP, H.323, Megaco, SIGTRAN traffic generation solutions.
ivrworx - high level Lua interface to SIP/RTSP/MRCP, for testing distributed VoIP scenarios (windows, Vista+ clients).
StarTrinity SIP Tester - bulk SIP call generator with complex test scenarios specified as a script, up to 5400 simultaneous G.729 calls per single server. G.107 MOS, PESQ MOS measurement, email alerts and daily reports.
StarTrinity VoIP Network Tester - freeware, generates traffic with multiple UDP streams and measures VoIP network jitter and delay.
Testbook SIP Trunk Tester & VoIP Service Tester Test tools for technicians and engineers installing and maintaining SIP Trunks and VoIP Services.
- Multiple Call Generator
- Call Quality Statistics (MOS)
- Customer Equipment (PBX) and Network Emulation Modes
- T38 Fax Emulation

Monitoring and Test Tools

ACQUA: VoIP Speech Quality Analysis System
Agilent Technologies DNA MX and TNA software
Brix Networks real-time VoIP performance management and service assurance solutions
ClarifiedNetworks Monitoring and high level flow analysis and visualization
ClearSight Networks ClearSite Analyzer
Codima Toolbox VoIP Management from pre deployment to post deployment and network visualization in Visio
Teraquant Communications System Testing and Network Monitoring
Consultronics
Drive testing and Quality of Experience (QoE) testing of wireless networks
Empirix monitoring and analysis
Experts in Speech and Video Quality assessment; PESQ, POLQA, J.247
Fluke Networks OptiView VoIP, ProtocolExpert Plus and Link Analyzer
HOMER SIP Capture & Monitoring, Analysis, PCAP extraction, Call Flows, API, Multi-User, OSS
Inet GeoProbe IP
Intelica VoIP Intelligence Call Monitoring, Analysis and Reporting
IPTEGO PALLADION SIP/IMS Real-Time Monitoring / Customer Experience Analysis / SIP Network API / Mediation
Malden
Minacom QOS monitoring and testing
NetTest monitoring of QOS and network performance
NetIQ monitoring of QOS and network performance
Net.Hunter - Hand Held, Battery powered packet capture/monitoring device at full GigE line speeds, optical and electrical interfaces.
Packet Data Systems Clarinet Protocol Test System
PacketIsland micro-appliances and asterisk agent for QoS monitoring of SIP and Cisco SCCP VoIP calls
PALLADION monitoring and analysis
PathSolutions SwitchMonitor VoIP
- Simulation and analysis of SIP, SIP-T & H.323. QOS measurement, recording, replay etc.
Pivetal - QspeeQ for total VoIP QoE management - multi-protocol signalling and VQM generation
Psytechnics monitoring of QOS
Qovia monitoring of QOS
Rochelle Analog QOS products - can be used with an ATA
Sage Instruments VOIP test equipment and systems
Spirent Communications - Test Solutions for VoIP networks and devices
Sunrise Telecom VOIP and Telecom test equipment
TamoSoft CommView and CommView for WiFi network analyzers for wired and wireless (802.11 a/b/g/n) networks
- Real-time VoIP call monitoring
- SIP and H.323 analysis and decoding, call playback
- Jitter, QoS, Bandwidth charts
Telchemy monitoring of QOS
Touchstone
- WinEyeQ
  - 100% software-based
  - monitors/analyzes/records/replays SIP and H.323 traffic, audio/video media and QOS.
- TraceBuster
  - records/replays SIP and H.323 traffic, audio/video media and QOS.
Unsniff Network Analyzer RTP/SIP/IAX2 visual analysis, one-click playback, whitepapers
Valid8.com Valid8.com is the leading provider of SIP, H.323, Megaco, SIGTRAN conformance test solutions.
VOIPFUTURE Efficient VoIP SLA monitoring and troubleshooting tools.
VoIPmonitor is open source live network packet sniffer which analyze SIP and RTP protocol.
- Predicts MOS-LQE score according to ITU-T G.107 E-model
- Detailed delay/loss/MOS statistics stored to MySQL
- Each call is saved as standalone pcap file
VoIP; SIP, IMS, SS7 and Wireless protocols
VoIP Spear — Web site that monitors your Internet connection 24×7x365 and shows you what your VoIP QoS is.
VQManager web-based, 24 X 7 real-time QoS monitoring tool for VoIP networks.
WildPackets EtherPeek VX - monitor QOS, packet loss, voice quality, etc.

Asterisk Tools

Nocom A simple script for viewing Asterisk config files with comments and empty lines removed.

Network Impairment Simulators

Apposite Technologies Linktropy 4500 hardware appliance to emulate WAN bandwidth, delay, and loss up to 155 Mbps.
Candela Technologies LANForge ICE Network Emulator
IPWave simulates many types of network impairments
iTrinegy offers a full range of network application performance testing tools
NIST Net allows a single Linux PC set up as a router to emulate a wide variety of network conditions
Shunra Network Simulator Shunra Virtual Enterprise (Shunra VE) network simulator creates a model of any production environment, including the network, remote locations, and the number and distribution of local and remote end-users. With Shunra VE, users can test the functionality, performance, scalability and robustness of the VoIP infrastructure under current and future production conditions â€” before deployment over the network.
Simena Network Emulator hardware appliance can simulate just about any possible network condition including latency, bandwidth, congestion, packet loss, etc. Test where your VoIP will break!
Spirent Communications - Test Solutions for VoIP networks and devices
UDP Packet Reflector and Forwarder open source tool that can drop packets, duplicate packets, and add jitter on a per port basis.
Net.Storm - Hand Held, Battery powered network simulator at full GigE line speeds, optical and electrical interfaces.

Decoding VOIP audio streams

There are several approaches to converting an RTP stream of packets into a playable audio.
See: Converting RTP to audio

SIP Debug

Callflow - creates a diagram of SIP flows
Distributed SIPFlow - Distributed application for capturing and displaying SIP callflows.
Free VoIP TraceBuster Decode and analyze Ehtereal/WireShark capture files!
PacketIsland 4"x4" in-line micro-appliances can capture SIP traffic and display ladder graphs of SIP call flow. SIP data from thousands of calls can be stored for weeks or months.
SIPFlow Standard - Java tool for displaying SIP traffic captured in real-time, or imported from Ethereal or tcpdump
SIP Grapher ( sip_grapher.pl ) Builds SIP flow graphs from Asterisk log files with sip debug enabled - search by phone number or Call-ID. Written in Perl.
Sipsak Command line utility for testing SIP devices and Programs
SIP Scenario - creates a diagram of SIP flow
siptest a command line test tool for sending and receiving SIP messages
sipviewer a visual SIP message trace tool
Wireshark (Ex Ethereal) using the VoIP call analysis feature.

Cisco Troubleshooting Tools

SMART-Smart Message Architecture Reader & Translator The most advanced audio and video protocol log parser in the world, capable of parsing and analyzing output logs from enterprise platforms such as Cisco Unified Communications Manager, Cisco Telepresense Systems, Cisco Telepresence Video Communication Server, Cisco IOS, and others. SIP Troubleshooting, H323 Troubleshooting and Q931 Troubleshooting.

Protocol Debug

Protocol Verification and Testing
http://www.teraquant.com/utel Conformance; Functional & Interoperability Testing

Other Sites

http://www.broadbandreports.com/tools Diagnostic, testing, and monitoring tools
Spirent Communications - Test Solutions for VoIP networks and devices
Step by step guide to troubleshoot your VoIP connection
TechTraces.com Sample VOIP call flows, traces and network captures.
VoIP Spear — Web site that monitors your Internet connection 24×7x365 and shows you what your VoIP QoS is.
http://www.voiptroubleshooter.com/ Entire site focused on VOIP troubleshooting
http://kb.smartvox.co.uk/voip-sip/top-reasons-voip-calls-drop/ Troubleshooting and fixing problems with dropping VoIP calls