Open source does not mean free.
Probably everyone hears this phrase when they first start learning about open source.
However, there is such an open source startup that has spent nearly 2 years contributing to the open source software Apache Pulsar and Apache BookKeeper, and even helping community users develop customized needs for free without taking a cent...
This company is StreamNative, and its two founders Guo Sijie and Zhai Jia are also founding members and PMC members of the open source project they maintain. Recently, we chatted with Zhai Jia, founder of StreamNative, about their story.
In the story of StreamNative, we can see the open source helmsman's persistence and thinking on the open source community, and we can see the investors' recognition and help behind the open source cause. Of course, we can see the unique development path of open source startups more - two years of free support does not mean charity, but waiting for the outbreak after accumulating energy, so that the open source community can grow into the firm's strongest moat.
A catfish can at least make waves
Recalling that when he decided to invest in the research and development of Pulsar in early 2019, Zhai Jia mentioned a sentence given to him by a friend: The big data market has been formed, and you are like a catfish when you come in. You may not be able to swim very far, but the market will definitely fluctuate. .
As a technician, leaving a presence in the turbulent big data market at that time sounds very exciting.
The outbreak of products in the database field can be traced back to three product design papers published by Google from 2003 to 2006. From the computing method to the storage method, these papers pioneered the distributed computing architecture and laid the foundation of big data computing technology. Under the guidance of these three papers, Doug Cutting, a big bull in the open source community, used Java to implement open source implementations of Google's core cloud computing technologies, mainly GFS and MapReduce. Later, the Apache Foundation integrated the contributions of Doug Cutting and other IT companies to launch the Hadoop ecosystem.
The advent of Hadoop has created a new wave of open source in the big data space. Before that, the development of open source software was mainly represented by the subscription model pioneered by Red Hat, and its users were more users than secondary development. After the emergence of Hadoop, developers can modify their code from the bottom, so as to truly participate in open source development more deeply. The open source development of many domestic technology companies including Baidu and Tencent also started from mass customization of Hadoop.
In the next decade or so, various entrepreneurs and databases of various attributes spewed out, gradually forming the current situation of competition in the database field. Among them, Pulsar already belongs to a "latecomer", but precisely because of its late appearance, Pulsar is naturally adapted to the characteristics of the cloud-native environment, which will become the characteristics that distinguish its "predecessors" in the future.
Around 2012, Yahoo needed to build a Cloud Message Service cloud messaging platform. At that time, Guo Sijie was working at Yahoo, and his team wanted to take the architectural route of separating storage and computing. However, the hardware bandwidth at that time was difficult to support this architecture, so it was not recognized by the industry. In order to convince Yahoo's technical committee, Guo Sijie made Pulsar's prototype system Hadwig in Yahoo's Beijing R&D Center, targeting MQ scenarios, which is different from Kafka's scenarios for large concurrency and high bandwidth.
Hadwig's test results convinced Yahoo's technical committee. After completing basic development within Yahoo from 2013 to 2014, Hadwig ran stably within Yahoo for a few years, and in 2016, it was open sourced and became Pulsar.
At that time, Kafka , which is also the basic software for big data message processing, has become the "industry standard" in the message field. Pulsar also often needs to use Kafka to help users establish a more intuitive understanding of Pulsar.
The message field is actually divided into two directions, one is MQ for key business scenarios, which is what Pulsar is good at. The other is the data pipeline products provided for big data engines, such as Kafka, for large concurrency scenarios. Due to different usage scenarios, the messaging system is naturally divided into two technology stacks in many companies, while Pulsar can take into account both scenarios, and also has a good advantage in the storage layer, not only to ensure the quality of data service, to meet the general situation The demand for MQ products can also meet the high-bandwidth and high-concurrency scenario requirements of Kafka. Gradually, many companies with distributed messaging system requirements in cloud scenarios turned to Pulsar.
Apache Pulsar Big Data Ecosystem
For the technical details of Pulsar, BookKeeper and the comparison between Pulsar and Kafka, please refer to OSCHINA's previous article " Interview with PMC, how does the open source project Apache Pulsar challenge Kafka?" 》
There is a good community soil in the country
The perception that Pulsar's world should not be limited was around Pulsar 's graduation.
Pulsar was donated to the Apache Software Foundation by Yahoo in June 2017, and graduated as an ASF top-level project in September 2018. On the eve of Pulsar graduation, Zhai Jia met a technical director of Zhaopin at a technology summit in China. At that time, Zhaopin wanted to build a message bus, which happened to be the scenario that Pulsar was born with. They needed to ensure that messages were not lost, and they could flexibly expand to meet the needs of dead letter queues, and had particularly high requirements for consistency.
The Zhaopin technical team first discovered the BookKeeper project of the underlying storage. When they were preparing to develop an MQ system by themselves, they found Zhai Jia's sharing about Pulsar, so they chatted with Zhai Jia about their needs. After that, Zhaopin Recruitment became the first user of Pulsar in China. Zhai Jia and community partners helped Zhaopin replace their internal MQ products and Kafka in their scenarios.
On September 25, 2018, Pulsar graduated from the Apache Software Foundation incubator.
Taking this opportunity, Zhai Jia and Guo Sijie returned to China to do two offline Meetups for Pulsar. The first one was held in Beijing. On a weekend in early October, 200 people signed up, and 1670 people actually showed up. In addition to developers from the Internet, there are also many developers from companies in traditional industries such as banks. "I remember that five or six people came to a research and development team of one of the four major state-owned banks at that time, and it was quite touching to us at that time. The feedback from the event makes us feel that people are still paying more attention to Pulsar.”
The second Meetup in Shanghai also received a lot of attention. In this way, Zhai Jia realized that even if the domestic publicity was just starting, the first event could attract so many developers, which means that the story belonging to Pulsar will be very likely.
After the two events, Zhai Jia and Guo Sijie began to gradually establish contact with some domestic users and open source teams, and return to visit domestic users. Slowly, Zhai Jia and the domestic team reached a consensus that domestic companies actually have a larger amount of data than foreign companies. For example, any of the four major banks has a larger transaction volume than Bank of America. The amount of data that the leading Internet companies need to process inside is also far more than many similar companies abroad, and these companies may be more willing to accept new technologies that can solve difficult problems. In addition, many startups are generally more open to new technologies.
Soon, the two decided to go back to China to start a business. "We felt that the country is a good soil for building a community, so we resigned resolutely, and it only took one or two months before and after."
In early 2019, StreamNative, based on Pulsar and BookKeeper, was established.
The earliest batch of users in China also included companies such as Tencent's billing platform and Tuya Smart. "Everyone can solve their own problems from community cooperation very early and can benefit from it."
Two years in the community
If we say that the emergence of domestic users has given Zhai Jia and Guo Sijie the courage to return to China to start a business. On the contrary, in the early days of their business, the two also gave the community users the greatest support regardless of returns.
"We focused on the community in the first two years, hoping to lay a solid foundation for the development of the community." Zhai Jia introduced, "The construction of the Pulsar community is mainly focused on products and interaction with users, contributing and improving Pulsar's functions, and enriching Pulsar's surroundings. Ecology, helping Pulsar version release and maintenance, communicating with community users and helping to solve obstacles encountered in launching Pulsar, organizing and participating in related Pulsar meetup and other promotion activities, etc. These work are mainly to improve Pulsar's documentation and lower the entry threshold for Pulsar , enrich the integration and interaction of Pulsar and other systems, and do the basic work for users to start."
When users put forward requirements after starting, StreamNative will provide implementation solutions. If both parties feel that the solution is feasible, they will develop the required functions together in the community. Whether it is providing solutions or specific development work, it is completely free for a long time.
Behind every open source startup that chooses to invest in the community, there must be open source helmers with fans and investors who trust them.
Guo Sijie regards the community as the core moat of StreamNative, "The technology has been iterating, and the current technology may be eliminated after 5 years, but there are many contributors in the community, on the one hand, they can bring you rich application scenarios, on the other hand On the one hand, it can bring a good reputation, attract more people to use your product, and then continue to iterate on your product.”
Zhai Jia believes that open source is an excellent way for everyone to reach consensus and improve efficiency. Assuming that there are no open source projects in the messaging scenario, each company may develop a system based on existing programs and its own background. Then after Pulsar, first a small number of people think that Pulsar is good, use and contribute together, then Pulsar will become better and better and attract more people. On the whole, it can promote the improvement of social efficiency. "This is of greater significance. From the perspective of open source, it is helpful to everyone to promote the progress of social division of labor and improve the efficiency of development and use."
Zhai Jia introduced that among the investors they contacted, they have actually reached a consensus that open source is a good customer acquisition channel and an important foundation for the company's development.
As for commercialization, it is necessary to examine the space and possibilities of the industry. So, how likely is Pulsar?
Looking back at Pulsar, it is a message middleware infrastructure. For example, a message on our mobile phone, or any sensor, can be regarded as a message. The message is used as a carrier of information to record events that occur in the real world. down, and then go to the server to record.
Take Pulsar's client, Tencent's billing platform, as an example. The billing platform is a typical MQ usage environment, which supports all billing services within Tencent, such as red envelopes being one of the scenarios. At this time, Pulsar can record the sending and receiving information of the red envelope, the amount used, and save it as an event, and then the database or subsequent systems can make specific consumption according to the database bandwidth. What Pulsar has to do is to cut peaks and fill valleys with a layer of buffer. Assuming there is no MQ layer, it may be difficult for any database to support the huge amount of red envelopes on New Year's Eve.
When the CTO starts selling
In this way, with the increase of users, StreamNative finally started the process of commercialization. CTO Zhai Jia began to travel between different customers in different cities and started sales part-time.
In June this year, StreamNative released StreamNative Platform, a product that supports enterprise privatization deployment. In July, started to form a sales team and tried to cooperate with public cloud vendors. Before the sales team was fully formed, Zhai Jia took on some sales and customer visits.
He found that the overall trust of domestic customers in open source software is increasing. When customers choose open source software, they are more interested in the functions that can be achieved by the software itself, rather than "free" and "low price". At the same time, many customers at this stage come from community users, which allows Zhai Jia to have a general understanding of how Pulsar can be better applied in his business scenarios when he contacts customers.
In StreamNative's early commercialization attempts in China, the cases were concentrated in the financial industry, including banks, securities companies, etc.
Compared with the domestic market, StreamNative's internationalization is developing faster, and its customer base is basically overseas. About 40% of them are from Europe, about 50% are from North America, and some are in India. Customers are concentrated in the three major industries of finance, e-commerce and IoT. Typical examples include hedge funds and stock traders in Boston and Chicago, Flipkart in India, Toyota Europe in the autonomous driving industry, and Applied Materials in the chip field.
Corresponding to the development area of StreamNative, StreamNative's employees also come from all over the world, which naturally creates an asynchronous and free working atmosphere for StreamNative.
When it first started, StreamNative rented an office in Beijing. During the epidemic, the office was idle, StreamNative simply quit the lease, and all employees worked remotely. According to Zhai Jia, after full telecommuting, StreamNative's employees either returned to their hometown or settled in an ideal city, and their lives have also undergone earth-shaking changes.
In fact, not only StreamNative, because the cooperation of the open source community is naturally transnational, asynchronous, and multi-person collaboration. Many open source startups in China have implemented the remote office mechanism, which has become a major feature of open source startups. In this way, OSCHINA specially launched the [Anytime Anywhere] project, which is aimed at open source startups to understand the working status of their employees.
Guest introduction
Jia Zhai, co-founder of StreamNative. Before founding StreamNative, he was engaged in the design and development of distributed, file system, and streaming storage at EMC. Currently, he is a PMC member of the Apache BookKeeper and Apache Pulsar projects.
Anytime Anywhere
Feel free to build software.
Project Introduction
What do you think open source is?
A consensus, collaboration model, or organizational form...
If the answer is not clear, let’s take a look at the tangible changes that open source has brought us. The open source community is growing day by day. Open source is not only changing the world of technology through hardware and software, but also giving more freedom to its participants and changing their work and lifestyle.
We found that many open source native commercial companies, as well as individual developers in the open source community, are fully committed to asynchronous, offsite work. There is no need to check in at a fixed point every day, concentrate on working in the office building, and do not need to be on call at any time. You can flexibly arrange your work according to your goals... In the end, how much reshaping will such a change bring to team and personal life, we hope that through The [Anytime Anywhere] project presents the answer.
The [Anytime Anywhere] project collects the work and life schedules, as well as their status and feelings of employees in open source native commercial companies working remotely in the form of public questionnaires. The final result is presented as a public graph.