Press: Short homemade video with different back-end systems CURD, mainly to solve the many large and small video files traffic problems. This article from US teacher shot wheat Junsheng share from which you can see what.
A short video market development
In recent years, short video applications in the domestic market application detonated, the company introduced the United States beat Mito, related products as well as vibrato, GIF deft, shoot second, depending on the micro, funny shot, shot to play, a series of short video products but also enriched the emergence of short video application market.
Broke out a short video, with several factors:
1, bandwidth, with the development of basic network environment in China, more and more users are turning to mobile 2G 3G / 4G network, which experience better upload and download bandwidth and more stable network, the mobile 3G / 4G of the proportion of users accounting for about 85%; and with the further lowering of tariffs, the average household monthly traffic has reached 360M, and even many part-level or even tens of GB GB of traffic, so in terms of an ordinary 10s video probably less than 1 ~ 2M or even a few hundred K, will undoubtedly enhance the bandwidth traffic will gradually reduce the threshold used by the user; in addition, the home also increases the bandwidth of the current 10M or even 100M `has become a mainstream consumer bandwidth, so as to short video development provided the necessary conditions.
2, mobile phone hardware configuration greatly improved, with the pixels is increased, the phone hardware configuration upgrade CPU, GPU, memory, etc., can make the phone faster to process and optimize video effects, which can bring to handle mobile video more creative space.
3, ability to express the traditional text and image rich enough, it is not possible to meet the needs of users, while the short video for a large enough space for expression.
4, the product itself, offers a variety of ways to reduce the threshold of making a video of the user, such as the United States beat MV provides special effects to enhance the same time interesting video production, lower the threshold for use. At the same time the diversification of products, but also to meet the different needs of various users, user-inspired spread.
Second, the development of the United States beat
US shot in the 2014.05 release month, only one day on the line, that is free to sign App Store first overall standings, ranking first month of downloads. When released nine months, users exceeded one million. Currently the United States took a day playing video daily volume of 2.7 million or more, long reach 1.83 million hours of video playback time of day.
Faced with this outbreak of users grow, the United States took also encountered many applications start at the beginning sweet and bitter memories of that year of experience, more than a year after the evolution of architecture, beauty shot has accumulated some experience, to form a set of high availability and scalable architecture practice. Although not do very beautiful, but it will continue with the evolution of architecture and continuous improvement.
Compared to normal text social classes APP, to do so in a short video product technical architecture level, what issues will face it?
And versatile text social networking products, the United States took top page popular, dynamic friends (feed stream), commenting service, private telecommunication services and other basic functions; so when the user explosive growth, will also face the application layer, databases, caching, access challenges layer, etc., then how to do low-latency, high availability yet. At the same time because it is a short video itself will also face some specific areas related problems.
Third, the architectural issues facing short video
Short video compared to text data, there are some differences:
1, the data size of the difference
For example, a beauty shot, on balance and clarity of video compression, 10s more than 1MB of video size, while the United States to shoot a video or even five minutes to reach tens of M, compared to text with dozens or hundreds of bytes bytes is much greater. Because the amount of data is much larger, it will also face some problems: how to upload, how to store and how to play the question.
About uploading, so you want to upload a video in the mobile phone, especially in weak network environment to upload such a file, upload success rate is relatively low, the evening peak time congestion at inter-provincial network, to a much more obvious. So for uploading, you need to go based on CDN dynamic acceleration to optimize the network link (measured by the tone for enhancing the stability and speed had some help), while for large video need to be part uploading, reducing the cost of failure retransmitted and failure probability to improve usability. At the same time the link status of different CDN vendors may appear different in different regions of different operators, so it requires a combination of tone test, choose some of the more suitable for their own CDN vendor link.
At the same time because the data is relatively large, when the data reaches a certain size, storage capacity will face some challenges, the current US video shoot capacity level reached PB-level scale, it requires the storage itself can have a relatively strong linear scalability, and there adequate resources redundant, while the traditional MySQL and other databases more difficult to support this scenario, so often need the help of a dedicated distributed object store. It can be solved by self service or cloud storage service. In recent years, thanks to the development of cloud storage, currently the United States shot mainly using cloud storage services to solve. Own distributed object storage is mainly used to solve some internal scenarios, such as for data privacy and security requirements are relatively high scene.
Playback, because the file is large, it tends to be affected by the network, so in order to avoid Caton, some of the details need to be addressed. For example, for 60s, 300s of video, you need to take into account the files are large, while dragging needs, so the general use http range of ways, or on-demand playback mode HLS, the former is based on simple and crude, but based on the player mechanism , but also to meet the needs, but also to achieve drag on demand. HLS based on direct way would be much friendlier, especially the longer ones of the video, such as five minutes or even more video, but this requires a separate transcoding support. Before the United States shot a short video is mainly based, so more use http range of ways. The subsequent scene with the video of five minutes or more, are gradually try to do something. At the same time for the player, in a weak network environment, may also face some problems, such as playing Caton problem, this general through the network link optimization; optimization or adaptive multi-rate, such as multiplexer code, then the code rate is selected according to a specific algorithm quantization user network model, the network poor low bit rate manner.
2, the difference data format standard
Compared to text data, binary data itself is short video, a relatively fixed encoding standards, such as H.264, H.265 and the like, and has a relatively fixed number of common format standard.
3, data processing requirements
Video information itself can carry more, we will be faced with a large number of data processing needs, such as watermarks, thumbnail frames, transcoding. The operation of the video processing is very slow, it will bring a huge resource overhead.
For the United States to shoot video processing, it is divided into two:
Client processing, video processing to the client as much as possible by using existing powerful mobile processing performance to reduce the pressure on the server, but it will face some problems processing efficiency of low-end models, but especially low-end models for uploading US pat itself is relatively few, so the problem is not obvious. The client is the main effect for video overlay, face recognition and facial beautification of various processing algorithms, we have here the client lab team, specialize in the optimization algorithm of this effect. At the same time the client process will increase the number of necessary transcoding and watermark video processing. Currently the client's video codec, encoding will be soft and hard-coded, soft-coded mainly better compatibility, better coding effect, but the drawback is the high energy consumption wait a minute more. The graphics and other means of hard-coded, it is possible to obtain a relatively low energy consumption and faster, but less effective, and compatible with a number of, particularly for some low distribution models. So now we tend to use a combination of ways.
Processing services side, mainly because some audit work transcoding video, there are some shots of smoke generated frame work, at present, some processing using ffmpeg. At some point the server itself needs to be considered, because resource consumption is relatively high, so the machine will take a few more, so in the end to do the service of video processing operations, will try to control in a reasonable range. At the same time as the United States beat this scenario, the peak will encounter these mutations hot events, so transcoding service cluster itself needs to have elastically stretchable and asynchronous of the peak and mechanisms in order to adapt to the scene of this sudden increase in requests.
Fourth, to support one hundred million users, the United States made some improvements to shoot architecture
With the rapid growth of users and traffic, beat the United States encountered a lot of challenges
- Performance Challenge
- Availability challenges
- Unexpected challenges hotspots
- Frequent business challenges iteration
In the case of frequent business iteration, how to ensure the availability of high enough mass in the request, while a better user experience and relatively low-cost way to provide service to become the direction of our efforts.
This is the picture of the overall architecture of the United States shot
It is also a framework is constantly evolving and continuous process, in addition to the construction of some basic service components, we also focus to do some related work in the service of governance, to ensure that services are available and overall stability.
Divide and conquer, to simplify
Overall planning framework, a clear responsibility single service module, try to maintain sufficient cohesion, and do decoupling between service module, so it can be more refined optimization work for a single module, while being able to use the appropriate technology to solve suit scenes issues.
Service interaction and communication between, we mainly go two ways:
- HTTP-based way
- 基于 config service + RPC 的方式
前者使用的方式比较简单，目前我们主要在跨团队、跨语言（比如 PHP 和 golang 之类的）会使用，主要会在七层 nginx 层做一些工作，如负载均衡、节点探测、并发保护等。
而第二种方式，我们主要用于内部系统之间的一些交互。目前我们主要基于 etcd 来实现我们的动态服务发现和配置服务，在 client 层面扩展实现了包含负载均衡、心跳、节点健康状态探测、etcd 节点挂掉的灾备等基础功能，同时会通过一些metrics埋点，以便跟踪内部的状态，用统一的 trace_id 来跟踪服务的链路调用情况。
开放扩展
主要针对下面几个点：
- 代码功能的可扩展性
- 交互协议的扩展性
- 数据存储格式的可扩展性
- 应用的可扩展性
- 资源的可扩展性
交互协议，既针对交互接口，也针对 app 客户端和服务端的交互协议。特点是 app 客户端和服务端的交互协议，因为 app 的升级较之服务端升级的时间久得多，比如你发布了一个客户端版本 V0.1，如果用户后面一直不升级，这个时间可能是几个月、半年甚至一年，那么就会引入一些兼容问题，所以在协议层面设计的关键点需要考虑这种情况的存在，需要保证协议能够向前兼容，预留好扩展点。
而关于数据存储格式的可扩展性，美拍第一个版本每个属性在数据库中为一个字段，并且为了保持一定的扩展性也多加了几个扩展字段。在发展过程中演化为所有属性字段序列化为 protocol buffer 数据的方式，这样能更好满足快速发展的业务需求。但是大家往往也更多关注在服务端，其实有时候比较坑的是在客户端。之前就踩过坑客户端上有个 id 字段的数据类型使用 int32，因为客户端基本很难做强升，一个这样小的事情最终需要很长时间来消化解决，并且为此还需要做一些兼容工作。所以针对这类事情，建议大家在一开始时候也多留意，尽量少为将来埋坑。
分级隔离
目前我们主要通过这几个维度进行一些隔离：
- 核心和非核心的隔离
- 单一集群的内部隔离
- 不同集群的外部物理资源隔离
- 不同集群的外部依赖资源的隔离
美拍在发展早期，跟多数发展早期的系统一样，也是多数接口部署在同一个集群中，包括也共用了一些资源（比如 memcached ），这样的好处是早期部署上足够的简单。在发展的过程中，业务在快速发展，业务复杂度也在逐步提升，接口调用量也急剧增加，逐步就暴露出一些问题。美拍的发展过程也是实际的去验证了前面提到的分级隔离机制。
在发展早期，曾经有个调用量不小的非核心的业务，在对存储数据结构做了调整后的上线过程中出现性能问题，导致整个集群服务都受到一定的影响。虽然通过降级策略和运维配套设施快速的解决了问题，但是也引发了我们的进一步思考。在架构上我们会尽量保证在开发效率、系统架构、部署和运维成本等方面达到一定的平衡，以避免过度设计或者架构支撑不了业务。这到了需要做一些事情的时候，我们把核心业务和非核心业务在七层和应用层做了部署上的隔离。
做完上面的核心业务和非核心业务拆分之后，接口互相之间的依赖影响降低很多。但是还没有解决核心业务或者非核心业务内部接口之间的依赖影响问题。所以接下来也更进一步，针对部分场景也做了内部隔离，通过限定每个接口最多只能使用的固定处理线程数方式，来避免因为单个集群内某个接口的问题导致整个集群出问题的情况发生。
以上主要是在接口层面做隔离，而在依赖的资源及其外部服务方面，如果没有相应的隔离机制，也会有互相依赖影响的问题，比较典型的有 memcached slab calcification 问题等。所以我们也在 memcached、mysql 等核心资源层面做了拆分。
综合来看，分级隔离本质上也是在解决服务之间依赖影响问题。
资源冗余
因为短视频是一个比较耗带宽的服务，因此在通用的应用自身资源冗余的情况下，还需要考虑到服务所依赖的外部资源，比如 CDN 和云存储服务本身的情况。对于 CDN 层面，可能还要考虑不同厂商在不同区域不同运营商下的资源冗余情况。而依赖的云服务等，这些服务本身从对外角度看是一个可无限扩展的服务，理论上通过扩展就能够满足性能需求，但是在使用往往会受限于实现，因为内部不一定是一个完全隔离的场景，比如说和别的企业服务混跑，同时可能只会分配对应的资源池，但这个资源超过性能预期的时候，不是一个可自动动态伸缩调度的场景。
容灾
美拍的容灾主要分为自身服务容灾、CDN容灾、云存储容灾等。
自身服务容灾主要包含一些典型的容灾场景，比如 cache 容灾，通过多级 cache、cache 的分片 hash 的方式、以及本地 cache 的方式来解决。目前我们这边的容灾也借鉴了微博的多级 cache 机制的机制，针对核心的 cache 资源会有主备节点，避免单一节点挂掉后，穿透会压垮后端 DB，同时对于请求量特别大的场景，比如对于某个热点资源访问量很大的情况下，也会在之前增加一层 L1 的 LRU cache 来规避和缓解这一问题。
CDN 容灾主要通过接入多家供应商进行互备，然后通过一些基调检测不同服务厂商的链路和服务状态，当发现服务有问题的时候，通过 DNS 进行区域的切换。不过不同 CDN 厂商的服务表现不对等，所以在选型 CDN 厂商的话，需要侧重关注可用性、节点布点和链路状况、回源量、资源冗余量、晚高峰的链路状况、以及对于多媒体是否有单独优化等等来评估靠谱性。
云存储容灾，目前美拍也主要使用两家互备的方式，因为国内的网络链路状况容易发生问题容易导致个别上传服务失败，以及云服务厂商服务挂掉的情况我们需要保证我们的服务可用。目前的做法是上传优先走主的云服务，如果上传失败的话，那么就会启用备的云服务。然后服务端层面也可以控制整体降级的方式，可以直接从主云服务直接降级读写备云服务。 基于每天的统计来看，通过这个方式至少提升上传的 0.1% 以上的可用性，在某些极端情况下，可能达到 1% 的可用性，当然这一块通过网络链路优化可能使得可用性情况没有数据中那么差。不过他的主要作用是在当某个云服务厂商节点服务出现短暂不可用或者长时间不可用的时候，我们也不会受太大影响。
五、后续的一些发展
随着短视频的不断的发展，以及实时直播的崛起，带宽的压力会越来越大，所以能够结合着 P2P + CDN 的方式来缓解服务端的带宽压力，不过 P2P 主要会面临着防火墙的问题、以及节点网络质量的影响，同时也依赖与视频播放的热度，这种对于效果都会有一些影响，同时为了更好的播放流畅度，单一的 P2P 无法满足需求，需要基于 P2P 和 CDN 的辅助进行。
带宽的另外一个节省之道，就是通过更好的编码标准来进行优化，比如 H.265 的编码标准，通过这个能够节省一半的流量。不过目前 H.265 在硬编支持不是很好，只有个别手机机型支持，而软编码的方式相比与 H.264，编解码速度要慢个几倍，这种对于能耗消耗比较高，处理也比较慢。而在往 H.265 演化的过程中，解码的普及程度也将会比编码来得更早。因为在解码算法层面，现有开源的方案还有很大的优化空间，以现有的手机硬件配置，是存在可以通过算法优化达到可以支撑 H.265 的空间。所以随着解码算法的不断优化和硬件的不断升级，解码普及的时间点也应该会比大家预期的时间来得更早，晋时也将会有更大比例的端能支持 H.265 的解码，对于 H.265 的普及奠定了很好的基础。
Ideally, the popularity of H.265 is the need for a large proportion of the end device has support at the level of encoding and decoding, in the case of decoding earlier popularity, the fact that there is a middle way transition: H.264 upload data upload end , the server into H.265, play according to their own end machine condition choose to use H.264 or H.265 data. Such a scheme requires the server to do a need for additional transcoding, and storage costs will increase. After the support H.265, so although there is extra overhead costs, but compared to the use of H.265 bandwidth costs brought about in the end there is a greater proportion of savings may more and more be ignored. And also can be done to control access based on the heat situation, both get a better balance.
The other direction, the current US shot will be more and more the more the number of clients Images beautification algorithm cloud of exposure as a service to internal use in other services, in order to be able to support more focus on product ecological chain construction of the "beauty" system . The main difficulty of this architecture will be facing is the high consumption of resources. And this solution will be dependent on the two ways, one to optimize the performance of the GPU hardware, co-processors, CPU SIMD instructions, etc. also need to solve the elastic automatic scheduling video processing cluster architecture issues, while some scenarios, for example, similar to H5 promotion page, will gradually be solved through a combination of public cloud scheduling approach.
