Original: http://mp.weixin.qq.com/s?__biz=MzA3NDcyMTQyNQ==&mid=2649254558&idx=1&sn=1c9246574affa51edd2100e9505e0c43&scene=23&srcid=0624GgKMMcAp3315VuRgtiEU#rd
Ali has many open source projects, which is also inseparable from @ Taobao Zhengming's open source attitude. There are many heavyweight projects, such as LVS, Tengine, or middleware with practical value. Today, Xiaobian summarizes Alibaba's open source projects for everyone, come and have a look! Service Framework Dubbo Dubbo is a high-performance and excellent service framework open sourced by Alibaba, which enables applications to realize the output and input functions of services through high-performance RPC, and can be seamlessly integrated with the Spring framework. JDBC connection pool, monitoring component Druid Druid is a JDBC component, which includes three parts: Agent Driver, which can provide a plug-in system based on Filter-Chain mode. DruidDataSource Efficient and manageable database connection pool. SQLParser Java's JSON processor fastjson fastjson is a high-performance JSON parser and generator implemented in the Java language, developed by engineers from Alibaba. Distributed file system FastDFS FastDFS is an open source distributed file system. It manages files. Its functions include: file storage, file synchronization, file access (file upload, file download), etc., which solves the problem of mass storage and load balancing. question. It is especially suitable for online services with files as the carrier, such as photo album websites, video websites and so on. Taobao Hadoop job platform Zeus Zeus What is Zeus (zeus) Zeus is a complete Hadoop job platform from the debug operation of Hadoop tasks to the periodic scheduling of production tasks Zeus supports the entire life cycle of tasks. Taobao distributed database OceanBase OceanBase is a high-performance distributed database system that supports massive data. It realizes cross-row and cross-table transactions on hundreds of billions of records and hundreds of terabytes of data. Advertising, application R&D and other departments are jointly completed. When designing and implementing OceanBase, the functions of non-urgent DBMSs, such as temporary tables and views, were temporarily abandoned. The R&D team focused limited resources on key points. Currently, OceanBase mainly solves the problem of data update consistency and high performance. Cross-table read transactions, range queries, joins, full data and incremental dumps, and batch data import. At present, OceanBase has been applied to Taobao Favorites, which is used to store Taobao users' favorite items and specific product and store information, and supports 40 to 50 million update operations every day. The applications waiting to be launched also include CTU, SNS, etc., with more than 2 billion updates per day and more than 2.5TB of updated data, and will be gradually promoted within Taobao. Distributed key/value storage system Tair Tair is a key/value structure data storage system independently developed by Taobao.com, and has a large-scale application in Taobao.com. You are directly or indirectly interacting with Tair when you log in to Taobao, view product detail pages, or "make a paste" with friends on Taojianghu. System information collection and monitoring tool Tsar Tsar is a collection tool of Taobao, mainly used to collect server system information (such as cpu, io, mem, tcp, etc.) and application data (such as squid haproxy nginx, etc.), tsar supports real-time viewing and history View, easy to understand application and server information! Similar to sar, it can monitor and collect information on servers and key applications, support real-time terminal display and centralized database storage queries, and send alarm information to Nagios. The modular design of Tsar is very convenient to expand new functions by developing new modules. Learn NodeJS in Seven Days Learn NodeJS in Seven Days is an open source document written by the front-end technology department of Alibaba International Station for learning Node.js. Module Loading Framework SeaJS Sea.JS is a CommonJS-compliant module loading framework that can be used to load arbitrary JavaScript modules easily and happily. The standard modules supported by SeaJS follow the define form of the Modules/Wrappings specification and can run on web browsers and node. JS and other environments. Distributed message middleware Metamorphosis Metamorphosis (MetaQ) is a high-performance, high-availability, and scalable distributed message middleware, similar to LinkedIn's Kafka, with the features of message storage sequential writing, high throughput, and support for local and XA transactions. It is suitable for scenarios such as high throughput, sequential messages, broadcast and log data transmission. It has a wide range of applications in Taobao and Alipay, and is now open source. Alibaba's distributed database synchronization system, otter otter, is based on incremental log parsing of the database, quasi-real-time synchronization to the mysql/oracle database in the local computer room or remote computer room. A distributed database synchronization system. HTML5 interactive game engine Hilo Hilo is an HTML5 cross-terminal game solution developed by Alibaba Group, which can help developers quickly create HTML5 games. JavaScript template engine Velocity.js Velocity.js is a javascript implementation of velocity template syntax. Velocity is a Java-based template engine that is widely used in various subsidiaries of Alibaba Group. Velocity templates are suitable for scenarios where a large number of templates are used, and support complex logical operations, including functions such as basic data types, variable assignment, and functions. Velocity.js supports Node.js and browser environments. Oracle data migration and synchronization tool yugong yugong is a data migration and synchronization tool to Oracle launched by Alibaba (full + incremental, target supports MySQL/DRDS). Around 2008, Alibaba began to try MySQL-related research, and developed a database based on MySQL Cobar/TDDL (currently Alibaba Cloud DRDS product), a related product of sub-table technology, solved the scalability problem that single-machine Oracle could not satisfy. At that time, there was also a wave of de-IOE projects, and the Yugong project was born. The goal of the solution is to help users complete the migration from Oracle data to MySQL, and complete the first step to IOE. Automated Testing Solutions Macaca Official Website: Link Macaca is a complete automated testing solution. Features: Support mobile and PC; support Native, Hybrid, H5 and other application types; provides client tools and continuous integration services. Non-intrusive runtime AOP framework Dexposed Dexposed is the first heavyweight Andorid open source software of Alibaba Wireless Division. Based on the transformation of Xposed, a famous open source project in the ROOT community, the ROOT part has been stripped and evolved into an AOP framework that serves the application itself. It supports the online minute-level client bugfix and online debugging capabilities of most of Alibaba's apps. Dexposed's AOP is purely non-intrusive, without any annotation processors, weavers or bytecode rewriters. The integration of Dexposed is very simple, just like loading a JNI library, only need to insert a line of code during initialization. Android Application Hot Repair Tool AndFix AndFix is Alibaba's open source Android application hot repair tool, which helps Anroid developers to fix online problems of applications. Andfix is short for "Android hot-fix". AndFix supports Android 2.3 - 6.0, ARM and x86 architectures, dalvik runtime and art runtime. Forks of AndFix are .apatch files. Distributed SQL Engine Lealone Lealone provides a distributed SQL engine for HBase, a project that tries to combine BigTable (HBase) and RDBMS (H2 database). php framework WindFramework Wind Framework is a php framework developed by the phpwind team. The development started in October 2010, and so far four versions (0.5, 0.8, 0.9, 1. 0). It originated from a refactoring project called 'Magpie Bridge' of the phpwind community product. It is intended to redefine the architecture-level development mode of phpwind to make it look clearer and easier to extend and secondary development. Later in a discussion the refactoring plan was divided into two parts. Part of it is the underlying core framework. The other part is the phpwind community data service center, the plug-in component mechanism. Reverse proxy test suite Macaroon Macaroon is an efficient reverse proxy test suite. The test framework itself is developed based on python2.7, which is convenient for cross-platform porting; the test cases are described in a format, so that the programming language is independent. Using Macaroon can easily and quickly complete the construction of test scenarios, the reconstruction of online failure cases, and the function, module, and protocol conformance testing of the software under test (DUT). Based on Node. js automated continuous integration Reliable Reliable is a distributed architecture continuous integration system developed by members of the Macaca team. It is suitable for scenarios such as integrated construction and integrated construction. She is a typical master-slave structure, which is divided into two parts: reliable-master and reliable-slave. Taobao React Framework React Web The Taobao front-end team's open source project React Web builds web applications through APIs consistent with React Native. Weex, a cross-platform mobile development tool On April 21, 2016, Alibaba announced the open-source cross-platform mobile development tool Weex at the Qcon conference. Weex can perfectly balance performance and dynamics, allowing mobile developers to write Native level through a simple front-end syntax It provides excellent performance experience and supports multi-terminal deployment such as iOS, Android, YunOS and Web. Java APNS open source library apns4j apns4j is the Java implementation of Apple Push Notification Service! Taobao Web Server Tengine Tengine is a web server project initiated by Taobao.com. On the basis of Nginx, it adds many advanced functions and features for the needs of high-traffic websites. The performance and stability of Tengine have been well tested on large websites such as Taobao.com and Tmall.com. Its ultimate goal is to create an efficient, stable, secure, and easy-to-use web platform. The commonly used web UI library kissy kissy is an open source JavaScript library on Taobao. It contains components: calendar, picture magnifying glass, card switching, pop-up window, input suggestions, etc. Bug Management System BugFree BugFree - A free and open source, Web-based, simplified version of the Bug Management System based on Microsoft's software R&D philosophy. Nginx development from entry to mastery Due to its excellent performance, nginx has attracted more and more people's attention around the world. It is widely used in Taobao. Many developers and operation and maintenance students are eager to understand the development of the nginx module and its However, there is no book on this aspect in China, so we decided to write one by ourselves. The author of this book is a member of Taobao core system server platform group. The idea of writing this book is to gradually transition from module development to nginx principle analysis. The content of the book will be regularly updated here. You are welcome to give your valuable comments. Whether it is a problem with the content of the book or a word error, you are welcome to submit an issue (there is a comment button on the left side of the chapter title), and we will follow up in time. . Messaging middleware RocketMQ RocketMQ is a distributed, queue-model messaging middleware with the following features: Strict message ordering, rich message pulling mode, high-efficiency subscriber horizontal scalability, real-time message subscription mechanism, hundreds of millions of messages The stacking capability Metaq3.0 version was renamed, and the product name was changed to RocketMQ Distributed Data Layer TDDL Taobao developed TDDL according to its own business characteristics (Taobao Distributed Data Layer nickname: The head is big ©_Ob) framework, which mainly solves the transparency of sub-database and sub-table to applications and data replication between heterogeneous databases. It is a jdbc datasource implementation based on centralized configuration, with active and standby, read-write separation , dynamic database configuration and other functions. Domestic forum system PHPWind The PHPWind forum system is a brand-new and complete powerful system that runs in the php+mysql database mode and can generate html pages. Because of its extraordinary access speed and excellent load capacity, it is deeply loved by friends at home and abroad. General Web Framework Webx Webx is a hierarchical and modular Web framework widely used within Alibaba Group. Webx is a WEB framework based on the classic MVC design pattern, advocating the concept of page-driven and convention over configuration. Webx is a Spring-based component framework. A component is a software package that can be extended by other components, and can also extend other components. Using these features, Webx can not only be used to develop highly customizable Web applications (which is its main function), but also can be used to help you develop highly extensible non-WEB applications. Distributed file system TFS TFS (Taobao FileSystem) is a distributed file system with high scalability, high availability, high performance and Internet service. Its design goal is to support massive unstructured data. At present, domestic self-developed file systems are rare. Taobao has made effective exploration and practice in this field. Taobao File System (TFS), as a distributed file system used internally by Taobao, has been specially optimized for the random read and write access performance of massive small files. Data storage such as pictures and product descriptions. The distributed core technology framework Fourinone FourInOne (the Chinese name "four dislikes") is a four-in-one distributed computing framework. Asynchronous file uploading component Uploader Uploader is a very powerful asynchronous file uploading component. It supports three solutions of ajax, iframe, and flash to achieve full compatibility with browsers. The call is very simple. It has built-in multiple sets of theme support and common plug-ins, such as verification and image preview. , progress bar, etc., are widely used in Taobao, such as refund system, love shopping, second-hand, auction, my Taobao, seller center, shopping guide center, etc. Heterogeneous data source data exchange tool DataX DataX is an offline synchronization framework/tool that allows you to easily exchange data between heterogeneous data sources. It realizes data exchange between arbitrary data processing systems. Currently, DataX has about 5,000 synchronization tasks distributed in Taobao every day. At various times throughout the day, the average daily data synchronization volume is 2-3TB. Lightweight distributed data access layer CobarClient Cobar Client is a lightweight distributed data access (DAL) implementation based on iBatis (renamed MyBatis) and Spring framework. Multi-browser compatibility testing overall solution F2etest F2etest is a multi-browser compatibility testing overall solution for front-end, testing, product and other positions. http stress testing tool httpsender httpsender is a lightweight http stress testing tool developed by Taobao test engineers in Java language. It can specify the number of concurrent connections to send a specified number of requests, and can also customize the header of the request. The requested URL supports specifying a range via regular expressions, and also supports random or sequential reading from a file. It also has more advanced features such as verifying that the data is complete and messy. The default application environment is Linux. Taotadpoles open source platform taocode Taocode is an open source platform launched by Taobao.com. The platform itself is also open source. Ali Redis Cache Server AliRedis AliRedis is a cache server transformed by Alibaba's operation and maintenance team based on Redis. The project has not yet been released. Incremental subscription of binlog & consumption component canal canal is the incremental subscription of binlog of Alibaba mysql database & consumer components. The mobile automated test platform Athrun Storm is a system similar to Hadoop MapReduce. The user implements a task according to the specified interface, and then submits the task to the JStorm system. JStorm runs the task and runs it 7*24 hours. When a worker fails unexpectedly, the scheduler immediately assigns a new worker to replace the failed worker. Therefore, from an application point of view, a JStorm application is a distributed application that adheres to a certain programming specification. From a system perspective, JStorm is a scheduling system similar to MapReduce. From the data point of view, it is a set of message processing mechanism based on pipeline. Real-time computing is now the most popular direction in the field of big data, because people's requirements for data are getting higher and higher, and the requirements for real-time performance are getting faster and faster. The traditional Hadoop Map Reduce is gradually unable to meet the demand, so the demand in this field is continuously. The distributed computing system JStorm Storm is a system similar to Hadoop MapReduce. The user implements a task according to the specified interface, and then submits the task to the JStorm system. JStorm runs the task and runs it 7*24 hours. When a worker fails unexpectedly, the scheduler immediately assigns a new worker to replace the failed worker. Therefore, from an application point of view, a JStorm application is a distributed application that adheres to a certain programming specification. From a system perspective, JStorm is a scheduling system similar to MapReduce. From the data point of view, it is a set of message processing mechanism based on pipeline. Real-time computing is now the most popular direction in the field of big data, because people's requirements for data are getting higher and higher, and the requirements for real-time performance are getting faster and faster. The traditional Hadoop Map Reduce is gradually unable to meet the demand, so the demand in this field is continuously. Distributed processing system for relational data Cobar Cobar is a distributed processing system for relational data. It can provide you with massive data services like traditional databases in a distributed environment. Taobao's Linux system taolinux is based on Taobao's years of server optimization and accumulation, Kernel optimization experience. nodejs-kissy KISSY is a lightweight and flexible JS framework developed by Taobao.com. It is now version 1.1.7 and is widely used in Taobao. , we launched the nodejs-kissy project, your KISSY program can be seamlessly ported to the server side. Dubbo Documentation Mirror dubbo-doc-static dubbo-doc-static is Dubbo's documentation mirror. Automated cloud testing platform AutoMan The predecessor of AutoMan is tCommon and TAM (taobao-automan). After continuous optimization and expansion of the two, a complete set of page automation platform has been gradually formed, which is intended to improve the test efficiency and test quality of testers. , galloping through the entire testing process like stepping on a "hot wheel". Online analysis and query system mdrill mdrill is a set of data software open sourced by Alimama. For TB-level data volume, it can only use 10 machines, achieve second-level response, and data can be imported in real time. Arbitrary dimensions can be combined and filtered. As an online data analysis and processing software, mdrill can analyze tens of billions of data in any combination of dimensions in a few seconds to tens of seconds. Taobao custom JVM TaobaoJVM Taobao has tens of thousands of Java application servers, thousands of Java engineers, and hundreds of Java applications. To this end, one of the tasks of the dedicated computing group of the core system R&D department is to focus on the optimization and customization of OpenJDK, and provide a stable, efficient and deeply customized JVM version according to business, application characteristics and developer needs: Taobao JVM. Interface document management tool RAP RAP helps WEB engineers manage interface documents more efficiently through GUI tools, and at the same time automatically generates mock data and verifies the correctness of real interfaces by analyzing the interface structure, making interface documents a strong dependency in the development process. With structured API data, RAP can do more, and we can avoid more duplication of effort. The asynchronous task processing system TBSchedule was renamed TBSchedule from the original project taobao-pamirs-schedule Distributed Hash Table FastDHT FastDHT is a high-performance distributed hash system (DHT), using Berkeley DB for data storage and libevent for network IO Handle, provide a Java version of the client interface package. It is suitable for storing small amount of data such as user online and session information. The asynchronous file upload component KF/Uploader supports three schemes of ajax, flash and iframe, and is compatible with all browsers. Simple configuration, supports pseudo-attribute configuration (data-config='{}') and configuration attributes to configure ajax and flash upload methods, with upload progress display queue upload, batch upload waiting for files, support canceling upload midway, support upload verification, Integrate kissy's validation component with good expansibility, free custom button and queue template style automated test task scheduling platform TOAST TOAST (Toast Open Automation System for Test) is a set of automated test task scheduling platform developed by Yitao advertising technology testing team. java expression calculation QLExpress QLExpress It is a lightweight Java-like grammar rule engine that is used in business systems as an embedded rule engine. Make business rule definition simple and flexible. Let business people define business rules. It supports standard JAVA syntax, and can also support custom operation symbols, operation symbol overloading, function definitions, macro definitions, data lazy loading, etc. Taobao Open Platform JAVA SDK top4java Easy maintenance and extension (new API support can be added without modifying the main class) Injection interpreter (injecting the corresponding interpreter according to different return formats) Centralized management of request parameters and parameter mapping for runtime Exceptional way to manage error responses Use generics for strongly typed programming Multi-protocol extension support (REST, RPC, SOAP, etc) Java image processing library SimpleImage SimpleImage is a Java image processing library from Alibaba that can implement Image thumbnails, watermarks, etc. Performance analysis tool TProfiler TProfiler is a performance analysis tool that can be used in the production environment for a long time. It supports both profiling and sampling methods, recording the time and number of method executions, generating method hotspot objects, creating hotspot thread status analysis and other data, for searching System performance bottlenecks provide data support. Code review tool Tao-ReviewBoard ReviewBoard is an open source code review tool, including server-side program (reveiwboard) and client-side command line (RBTools). Users can submit requests on the client side through a series of commands of RBTools, But there are many inconveniences to use and install. Tao-ReviewBoard is an eclipse-based reviewboard plug-in developed by Taobao, which can solve the installation and use problems very well. This plug-in has been used in Taobao for nearly a year, and now the tool is open source and shared with everyone. Svg-based chart component library KCharts KCharts is an Svg-based chart component library, compatible with mainstream browsers such as IE6+. A chart component library based on Taobao js framework KISSY, including line charts, curve charts, bar charts, scatter charts, pie charts, maps and other common charts. Adopt kissy's module loader to realize on-demand loading and support cdn dynamic merging. KCharts is developed based on RaphelJs, a large number of html+css implements basic grid, smooth animation, rich demo, and practical online chart generator (Chart Generator), which is convenient for beginners and practical. Mock data generator Mock.js Mock.js is a mock data generator designed to help front-end siege developers develop independently of back-end and help write unit tests. Expression parsing engine SimpleEL Simple EL is an expression parsing engine. It divides expression parsing into four parts: preprocessing, compilation, caching, and execution. Any part of these four steps can be replaced, similar to Template Method in design mode. GUI WEB interface management tool Rigel Automation RAP Rigel Automation Platform is a GUI WEB interface management tool. In RAP, you can define the URL, request & Response details format, etc. By analyzing this data, RAP provides automated tools such as MOCK services, testing services, etc. RAP also provides a large number of enterprise-level functions to help enterprises and teams work efficiently. Real-time data transmission platform TimeTunnel TimeTunnel (TT for short) is a real-time data transmission platform based on the thrift communication framework. It has the characteristics of high performance, real-time, sequential, high reliability, high availability, and scalability. tair-client-java tair-client-java is a Java client development kit for Taobao Tair, a distributed Key/Value storage system. Tair's java client is developed in pure java, the bottom layer uses the mina communication framework, and the interface is simple and easy to use. Stable and reliable after stress testing in the production environment. Logic-rich template engine xtemplate.js xtemplate is an independent rich-logic template engine, an extensible template engine library based on browsers and Node.js. xtemplate supports asynchronous control and inheritance, including logical expressions, custom functions, and more. Nginx's TFS module nginx-tfs nginx-tfs is an extension module of Nginx used to access Taobao's TFS file system. This module implements the TFS client and provides a RESTful API for TFS. The full name of TFS is Taobao File System, which is a distributed file system open sourced by Taobao. Redis java client Tedis Tedis is another redis java client. Tedis's goal is to create a highly available Redis solution that can be used directly in production environments. Data middle layer nQuery nQuery engine is a data middle layer from Alibaba. It is mainly used to deal with the three major problems of online query in big data scenarios: long data, wide data, and miscellaneous data. PgSQL, ql. io, etc., combined with its own business characteristics, a set of SQL-like language (nSQL) and parsing and execution engines that have been carefully designed and implemented by themselves. It has a wide range of application scenarios and strong practical value. It can provide translation and translation for *SQL databases. Compatible, for K/V data such as HBase, OTS or other general data sources, it can be used as a concise and powerful SQL query engine, and supports data JOIN in an innovative way of extending SQL. Combined with the NodeJS framework, it can provide a unified The high-concurrency, high-performance online query service of the SQL interface decouples the front-end and back-end, which greatly reduces the complexity of product development in big data scenarios. The persistent configuration center diamond diamond is a persistent configuration center that supports the publication and subscription of various persistent information (such as various rules, database configurations, etc.). ThreadLocal Transmittable ThreadLocal(TTL) that supports cached thread pools The demand scenario of ThreadLocal is the potential demand scenario of TTL. If your business needs to "transmit ThreadLocal while using thread pool and other components that cache threads", it is the TTL target scenario . Proxy Server AnyProxy AnyProxy is a Node.js based proxy server. The proxy server is a middleman, standing between the client and the server, and every bit of communication between the two parties will pass through it without leakage. It controls the complete request header, request body, response header, and response body, and can intervene in all processes without the client and server being aware of it. As long as the proxy server can be flexibly configured, the strange needs of front-end engineers will come to the fore. DSL parser generator dropincc.java A simple and easy-to-use parser generator; specially designed for implementing DSL solutions in the java language environment; Features: Use pure java syntax (Fluent Interface) to formulate user's lexical and grammar Rules; jdk1.6 compiler API dynamically compiles into bytecode; automatically manages bytecode, and users do not need to care about the specific generated parser source code; specially built for DSL, compared with other common tools (javacc, antlr, etc.) for DSL parsing , the amount of code has dropped sharply; it is easier to get started, allowing users to pay more attention to their own business content requires jdk1.6 or above to run, No other dependencies Recognize LL(*) syntax mysql proxy myrelay mysql proxy, supports the following features: Adopt event-driven, asynchronous, non-blocking design mode, excellent performance Adopt mysql native protocol, transparent to the application layer to support mysql read-write separation Provided MySQL connection pool management supports MySQL redundancy and health check Provides MySQL query logs to facilitate analysis of the application layer Supports MySQL configuration and dynamically loads C language to implement nginx-http-sysguard nginx-http-sysguard is an Nginx module developed by Alibaba. To protect the system load and memory usage of running Nginx server from being too high. Article source: Open Source China Reprinted from: IT Boss
Author: Toutiaohao / 51CTO Media
Link: http://toutiao.com/i6290693494210036225/
Source: Toutiaohao (the creation platform of Toutiao today)
The copyright belongs to the author. For commercial reprints, please contact the author for authorization, and for non-commercial reprints, please indicate the source.