Comparison of Static Source Code Security Detection Tools
1 Overview
With the rapid development of the network, various network applications continue to mature, and various development technologies emerge in an endless stream. Internet access has become an important part of people's daily life. While enjoying the convenience brought by the Internet, security issues have become more and more important. Hackers, viruses, Trojan horses, etc. are constantly attacking various websites. How to ensure the security of websites has become a very hot topic.
According to statistics from IT research and consulting firm Gartner, 75% of hacking attacks occur at the application layer. According to NIST statistics, 92% of the vulnerabilities belong to the application layer rather than the network layer. Therefore, the security of the application software itself is the most concerned issue in the field of information security, and it is also a new field we are facing, which requires the joint efforts of all our members at all levels of application software development and management. More and more security product manufacturers have also considered paying attention to the entire software development process, integrating security detection and monitoring into requirements analysis, outline design, detailed design, coding, testing and other stages to comprehensively ensure application security.
The detection of application security is currently mostly implemented through testing. Testing is generally divided into black box testing and white box testing. Black-box testing generally uses the penetration method. This method still has obvious shortcomings of black-box testing itself. It requires a large number of test cases to cover, and it is still impossible to guarantee whether the software is still at risk after the test is completed. Source code scanning in white-box testing is becoming a more and more popular technology. Using source code scanning products to scan software code, on the one hand, can identify potential risks, detect the software from the inside, and improve the security of the code , on the other hand can also further improve the quality of the code. The combination of black-box penetration testing and white-box source code scanning can greatly improve software security.
Source code analysis technology has a long history. Lloyd D. Fosdick and Leon J. Osterweil of the University of Colorado published the famous Data Flow Analysis in Software Reliability on ACM Computing Surveys in September 1976, which mentioned data flow analysis, State machine systems, boundary detection, data type verification, control flow analysis and other technologies. With the continuous evolution of computer languages, the technology of source code analysis is also becoming more and more perfect. In different segments, there are many good source code analysis products, such as Klocwork Insight, Rational Software Analyzer and Coverity, Parasoft and other companies' products . In terms of static source code security analysis, the static code analyzers from Fortify and Ounce Labs are very good products. There are many suppliers in the field of source code security detection. Here we select three representative ones for comparison, namely Fortify SCA of Fortify, Checkmarx Suite of Security Innovation and CodeSecure of Armorize.
2. Tool introduction
2.1. Fortify SCA(Source Code Analysis)
Fortify Software is a company headquartered in Silicon Valley, USA, dedicated to providing application software security development tools and management solutions. Fortify provides tools and establishes best application security practices and policies for application development organizations, security auditors, and application security managers, helping them identify and fix software source code with minimal time and cost during the software development lifecycle security risks in . Fortify SCA, part of the Fortify360 product suite, uses Fortify's proprietary X-Tier Dataflow™ analysis technology to detect software security issues.
Advantages: Currently the world's largest static source code detection manufacturer, supports the most languages
Disadvantages: expensive, inconvenient to use
2.2. Checkmarx CxSuite
Checkmarx is a high-tech software company in Israel. Its product CheckmarxCxSuite is specifically designed to identify, track and fix technical and logical security risks on software source code. It pioneered the use of query language to locate code security issues. It uses unique lexical analysis technology and CxQL patented query technology to scan and analyze security vulnerabilities and weaknesses in source code.
Pros: Utilize CxQL query language to customize rules
Disadvantages: The output report is not beautiful, the language support is not comprehensive
2.3. Armorize CodeSecure
Ama Technology was established in 2006, headquartered in Santa Clara, California, USA, and its R&D center is located in Nangang Software Industrial Park, Taiwan. Ama Technologies provides comprehensive network security solutions to defend businesses from attacks by hackers exploiting vulnerabilities in web applications. CodeSecure can effectively assist enterprises and developers to find out the risks of web applications in the software development process and after the project is launched, and clearly explain the ins and outs of the risks (how to enter the program, how to cause problems). The built-in syntax analysis function of CodeSecure does not need to depend on the compilation environment. Anyone can use the dual interface of Web operation and integrated development environment to find out the source code with information security problems, and provide patch suggestions for adjustment. CodeSecure relies on the self-developed host to perform remote source code detection, which facilitates users to perform web remote operations while ensuring stable speed.
Advantages: Web combined with hardware, fast, unique in-depth analysis
Disadvantages: less supported languages, expensive
3. Contrast
Fortify SCA is abbreviated as SCA, Checkmarx CxSuite is abbreviated as CxSuite, and Armonize CodeSecure is abbreviated as CodeSecure.
| SCA | CxSuite | CodeSecure | |
| Manufacturer | Fortify Software | Checkmarx | Ama Technology |
| Supported languages | Java,JSP,ASP.NET,C#, VB.NET,C,C++,COBOL, ColdFusion,Transact-SQL, PL/SQL,JavaScript/Ajax, Classic,ASP,VBScript,VB6,PHP |
JAVA、ASP.NET（C#、VB.NET）、JavaScript、Jscript、C/C++、APEX | ASP.NET（C#、VB.NET）、ASP、JAVA、PHP |
| 风险种类 | 400种 | 300种 | 参考CWE |
| 风险类型参考来源 | CWE、OWASP | CWE、OWASP | CWE、OWASP |
| 漏报率 | 最低 | 低 | 低 |
| 误报率 | 稍高 | 低 | 低 |
| 是否支持SaaS | 否 | 否 | 是 |
| 软硬件类型 | 纯软件 | 纯软件 | Web结合硬件设备 |
| 运行平台 | 无限制 | WindowsNET Framework 2.0 | 无限制 |
| 运行速度 | 取决于电脑配置速度不定 | 取决于电脑配置速度不定 | 由主机配置决定速度恒定 |
| 报告格式 | PDF、XML、CSV、HTML | Web、PDF | |
| 报告内容 | 完整按照风险级别不同分为多个文件 | 核心内容完整扫描信息等缺失 | 非常完整但修改建议放于最后 |
| 报价 | 100万/软件 | 70万/软件 | 100万/软硬件 |
| 性价比 | 中 | 高 | 低 |
从软件支持的源代码语言上来说，Fortify SCA（下文简称SCA）支持多达17种语言，Checkmarx CxSuite（下文简称CxSuite）其次，而Armonize CodeSecure（下文简称CodeSecure）在三款软件中支持的最少，仅仅支持几种最常见语言，不过这几种基本涵盖了绝大多数应用中使用的编程语言，基本上可以支持现在大多数应用的源代码扫描。
从风险的分类来说，各个厂商都有其自己独特的分类方式和不同的种类数量，不过从实际应用中可以看出，总体上仍为OWASP公布的几类风险，如SQL注入、跨站脚本等，已经可以满足实际中开发人员和测试人员的需求，对于各个厂商不同的部分，一般来说主要的区别在于理解不同，看问题的角度不同，并无谁错谁对之原则性问题。
从运行平台 的角度，CodeSecure这个产品目前看来已经将SaaS的理念很好的融合进来，整个软件的操作界面为Web方式，用户可以通过网页进行操作，B/S 的方式可以将操作系统的影响降到最低，只要有一台可以上网的电脑和浏览器，无论什么操作系统都可以使用CodeSecure远程进行源代码扫描，CodeSecure依托的是一台Armonize自行研制的主机，使用硬件设备的好处在于可以适用于多种场合，不会因为测试人员或是开发人员的电脑配置影响扫描速度，扫描的速度完全取决于主机的性能。而SCA和CxSuite主要还是单机软件，但目前也在不断地向SaaS的方向进行过渡，并且提供了相当全面的贯彻整个软件开发流程（SDLC）的解决方案与服务给用户。其中CxSuite这个产品标明了使用该软件的硬件配置，为Windows操作系统 和.NET框架，这个产品目前应该为利用.NET框架进行开发，所以运行环境有一定的局限性。同时，SCA和CxSuite因为是单机软件，一方面在使用 前需要安装，另一方面其运行速度取决于运行软件的电脑性能，对于使用该软件的电脑配置有一定的要求。
三种产品都使用了各自的技术对于威胁进行检测，SCA使用的是已获得专利的X-Tier™数据流分析器，这三种产品中只有CxSuite声称可以达到零误报率，因为 其对于风险的理解是风险必须在外形上呈现出来才被考虑为实际的风险，这种理解方式可以说是别出心裁，从代码安全的角度来说，检测的目的是为了发现问题并及时改正，同时要针对于最关键的问题进行改正，这也是这三款软件都包含TOP X的统计的目的，从这一点上讲，CxSuite的风险报告是非常谨慎的。SCA在以前的使用中发现有一定的误报率，不过换个角度想，误报相比漏报是可以容 忍的，规则越严格，误报率就会相应的上升而漏报率就会相应的下降，源代码检测工具目前均为静态的进行代码的扫描，即所有的检测均是按照“规则”来进行，任 何一款产品都不可能达到真正的零误报、零漏报。所以可以说SCA的规则检查稍显简单，CxSuite和CodeSecure的检查比较谨慎。
而从漏报率上来看，谨慎的查找势必会导致漏报率的提升，这一点上SCA和CodeSecure只说明了低漏报率，而CxSuite内部包含了一种类似于C#称为 CxQL的查询语言，支持使用这种语言进行查询，方便用户进行特定的查找。另两款软件使用的都是规则的方式，其本质上应该是相类似的，这一点上规则似乎更 容易被用户接收，但是CxQL的方式确实增强了用户的操作性。
从结果输出 上来说，三款软件都支持多种输出方式，而作为报告PDF格式可以说是最书面的一种格式。在这一点上，三款软件输出格式略有不同。
SCA报告构成如下：扫描概述、按风险的分类进行详细描述，包括每个风险的发现位置，代码上下文，风险源和风险输出，以及改进方法，各类风险描述之后是按照风险类别 的所有风险的统计和按照风险等级的统计图表。SCA的每种类型的文件生成一个PDF文件，便于用户对于风险严重程度的不同采取不同的策略。
CxSuite 报告构成如下：风险按照不同分类方式的统计图、风险的数据统计情况、风险最高的文件TOP 10、按照类别进行风险详述，包括风险的名称、描述、常见危害、在软件开发各阶段的相应处理方式、详细示例，列举每一个风险的传输路径和相应位置代码。
CodeSecure 报告构成如下：目录、重点精华，包括检测信息、弱点密度规范分布趋势、弱点最多的文件TOP 5，弱点索引，弱点的详细信息，包括弱点的全程跟踪，最后是弱点信息及修改建议、所有的进入点。
三款软件的 报告中以SCA的最有特色，将不同级别的风险分文件显示对于程序员进行修改是极为方便的；CodeSecure的报告最为规范，整个文档包括目录，结构完 整，唯一的不足是将风险的修改建议放在了最后，查阅有些不便；CxSuite的内容可以说是最概要的，只包含了风险的最关键内容，对于程序员来说应该是最 简洁的。
4. 总结
这三款静态源代码扫描工具都有其各自特色，SCA支持的语言多达17种，基本上涵盖了绝大多数的应用，具有相 当广泛的适用性，但同时也使得其价格非常昂贵；CxSuite支持的语言包括常见Web应用的语言，适用范围基本上包括了大部分的应用，其使用独创的语言来自定义规则非常有特色，价格较之SCA有一定的优势；CodeSecure支持的语言较少，不过基本上可以适用于当前大多数的B/S结构应用，它是唯一 的软硬件结合的产品，在免除用户安装步骤的同时将扫描运行于特定设备之上，有助于提高运行速度，也因为包括硬件的缘故，其价格不菲。
SCA极广的适用性使其适用于横跨多种语言的开发和测试人员，CxSuite的较高性价比使其适于基于Web 的开发人员和测试人员，CodeSecure稳定的速度和B/S的独特结构使得Web开发或测试的多人同时使用变得极为方便。
随着应用的安全性越来越受到人们的重视，静态源代码扫描和动态扫描将逐渐融合，未来将会有更多更优秀的源代码扫描工具诞生，让我们拭目以待吧。
附录A 其他静态源代码检测产品
| 公司 | 产品 | 支持语言 |
| art of defence | Hypersource | JAVA |
| Coverity | Prevent | JAVA .NET C/C++ |
| 开源 | Flawfinder | C/C++ |
| Grammatech | CodeSonar | C/C++ |
| HP | DevInspect | JAVA |
| KlocWork | Insight | JAVA .NET C/C++，C# |
| Ounce Labs | Ounce 6 | JAVA .NET |
| Parasoft | JTEST等 | JAVA .NET C/C++ |
| SofCheck | Inspector for JAVA | JAVA |
| University of Maryland | FindBugs | JAVA |
| Veracode | SecurityReview | JAVA .NET |
| FindBug | PMD／Lint4 | |