Java spring realizes downloading hls (m3u8+ts) real-time stream and merges mp4 and compression

reference connection

Link: Java downloads m3u8 video, decrypts and merges ts (3)

Link: Java Download HLS (m3u8) Video

First you need to understand what is HLS

Link: HTTP Live Streaming (HLS) - Concepts

Link: What is M3U8

The simple understanding is that the m3u8 file stores TS clips that can be played by the client

To put it simply, the m3u8 encryption technology is to divide the original video into n .ts files, and encrypt each .ts file with a key file. The m3u8 file stores the address of the key file and all the .ts files, so we need Decrypting this video requires the above three files, and finally you can use ffmpeg to merge and decompress;

M3U8 encoding format

m3u8 can basically be regarded as .m3u format files, the difference is that m3u8 files use UTF-8 character encoding.

// m3u文件头，必须放在第一行
#EXTM3U  
// 定义当前m3u8文件中第一个文件的序列号，每个ts文件在m3u8文件中都有固定唯一的序列号
// 该序列号用于在MBR时切换码率进行对齐
#EXT-X-MEDIA-SEQUENCE 
// 每个分片TS的最大的时长
#EXT-X-TARGETDURATION 
// 是否允许cache
#EXT-X-ALLOW-CACHE
// m3u8文件结束符
#EXT-X-ENDLIST 
// 分片TS的信息，如时长，带宽等
#EXTINF 
// 定义加密方式和key文件的url，用于取得16bytes的key文件解码ts文件
#EXT-X-KEY
// 提供关于PlayList的可变性的信息，对整个PlayList文件有效，是可选项。
// 格式如下：#EXT-X-PLAYLIST-TYPE：VOD（或者EVENT）。VOD表示服务器不能改变PlayList 文件；
// EVENT则表示服务器不能改变或是删除PlayList文件中的任何部分，但是可以向该文件中增加新的一行内容。
#EXT-X-PLAYLIST-TYPE 

Download the hls(m3u8+ts) process

First download the m3u8 file

Determine whether to decrypt

If the content contains the #EXT-X-KEY tag, it means that this link needs to decrypt the ts file, and then use the following .m3u8 if statement to obtain the link containing the key and the ts fragment.

Download ts files in sequence

Note:
If you are downloading an existing video, you can download the ts file in a multi-threaded manner, but when the download is currently live streaming, you can use a single thread

Merge downloaded ts files into mp4 format

Note: When merging, ensure that the ts file is continuous
Use #EXT-X-MEDIA-SEQUENCE to judge the continuity of the downloaded ts file by the sequence number of the current first TS segment

the code

import java.io.*;
import java.net.URL;
import java.net.URLConnection;
import java.nio.charset.StandardCharsets;
import java.util.ArrayList;
import java.util.HashMap;
import java.util.List;
import java.util.UUID;
import java.util.regex.Matcher;
import java.util.regex.Pattern;

public class Test {
    
    


    /**
     * 下载索引文件信息
     *
     * @param m3u8UrlPath
     * @return 索引文件信息
     */
    public static String getM3u8FileIndexInfo(String m3u8UrlPath) {
    
    
        try (BufferedReader in = new BufferedReader(new InputStreamReader(new URL(m3u8UrlPath).openStream(), StandardCharsets.UTF_8))) {
    
    
            StringBuilder content = new StringBuilder();
            String line;
            while ((line = in.readLine()) != null) {
    
    
                content.append(line).append("\n");
            }
            return content.toString();
        } catch (IOException e) {
    
    
            e.printStackTrace();
        }
        return null;
    }


    /**
     * 解析索引文件中的ts列表信息
     */
    public static List<String> analysisTsList(String m3u8FileIndexInfo) {
    
    
        Pattern pattern = Pattern.compile(".*ts");
        Matcher ma = pattern.matcher(m3u8FileIndexInfo);
        List<String> list = new ArrayList<>();
        while (ma.find()) {
    
    
            list.add(ma.group());
        }
        return list;
    }


    public static void downLoadIndexFile(List<String> tsList, String folderPath, String preUrlPath) {
    
    
        for (int i = 0; i < tsList.size(); i++) {
    
    
            String ts = tsList.get(i);
            String fileOutPath = folderPath + File.separator + ts;
            try {
    
    
                downloadTs(preUrlPath + "/" + ts, fileOutPath);
                System.out.println("下载成功：" + (i + 1) + "/" + tsList.size());
            } catch (Exception e) {
    
    
                System.err.println("下载失败：" + (i + 1) + "/" + tsList.size());
            }
        }
    }

    /**
     * 下载ts文件
     *
     * @param fullUrlPath
     * @param fileOutPath
     */
    public static void downloadTs(String fullUrlPath, String fileOutPath) {
    
    
        try (InputStream inStream = new URL(fullUrlPath).openConnection().getInputStream();
             FileOutputStream fs = new FileOutputStream(fileOutPath)) {
    
    
            int byteread;
            byte[] buffer = new byte[1204];
            while ((byteread = inStream.read(buffer)) != -1) {
    
    
                fs.write(buffer, 0, byteread);
            }
        } catch (IOException e) {
    
    
            e.printStackTrace();
        }
    }


    public static String composeFile(List<String> tsList, String folderPath){
    
    
        String fileOutPath = folderPath + File.separator + UUID.randomUUID() + ".mp4";
        try (FileOutputStream fileOutputStream = new FileOutputStream(new File(fileOutPath))){
    
    
            byte[] bytes = new byte[1024];
            int length;
            for (String nodePath : tsList) {
    
    
                File file = new File(nodePath);
                if (!file.exists()) {
    
    
                    continue;
                }
                try (FileInputStream fis = new FileInputStream(file);) {
    
    
                    while ((length = fis.read(bytes)) != -1) {
    
    
                        fileOutputStream.write(bytes, 0, length);
                    }
                    // 删除该临时文件

                } catch (IOException e) {
    
    
                    e.printStackTrace();
                }
            }
        } catch (Exception e) {
    
    
            e.printStackTrace();
        }
        return fileOutPath;
    }

    public static void main(String[] args) {
    
    
        // m3u8下載地址
        String m3u8UrlPath = "https://dh5.cntv.myalicdn.com/asp/h5e/hls/1200/0303000a/3/default/cdd3da535c12447a8cdb7c8ca949b2f6/1200.m3u8";

        // 下载索引文件信息
        String m3u8FileIndexInfo = getM3u8FileIndexInfo(m3u8UrlPath);
        System.out.println("========================");
        System.out.println(m3u8FileIndexInfo);
        System.out.println("========================");

        // 解析索引文件中的ts列表信息
        List<String> tsList = analysisTsList(m3u8FileIndexInfo);

        // 这里为了测试就先下载10个吧
        tsList = tsList.subList(0, 3);

        System.out.println(tsList);

        // 依次下载ts文件

        // 下载到本地的磁盘位置
        String folderPath = "D:/file";
        // 请求ts文件的下载地址
        String preUrlPath = "https://dh5.cntv.myalicdn.com/asp/h5e/hls/1200/0303000a/3/default/cdd3da535c12447a8cdb7c8ca949b2f6";
        downLoadIndexFile(tsList, folderPath, preUrlPath);
        String mp4Path = composeFile(tsList, folderPath);
        System.out.println(mp4Path);
    }

}

test

You can go to CCTV http://tv.cctv.com/, find a video to play, open F12 to view the requested interface, and find the m3u8 request address

#EXTM3U
#EXT-X-VERSION:3
#EXT-X-TARGETDURATION:15
#EXT-X-MEDIA-SEQUENCE:0
#EXT-X-PLAYLIST-TYPE:VOD
#EXTINF:11.120000,
0.ts
#EXTINF:9.680000,
1.ts
#EXTINF:9.360000,
2.ts
#EXTINF:11.640000,
3.ts
#EXTINF:10.120000,
4.ts
#EXTINF:8.520000,
5.ts
#EXTINF:10.000000,
6.ts
#EXTINF:10.600000,
7.ts
#EXTINF:12.360000,
8.ts
#EXTINF:9.640000,
9.ts
#EXTINF:8.360000,
10.ts
#EXTINF:8.760000,
11.ts
#EXTINF:13.400000,
12.ts
#EXTINF:9.200000,
13.ts
#EXTINF:9.920000,
14.ts
#EXTINF:9.560000,
15.ts
#EXTINF:8.160000,
16.ts
#EXTINF:9.720000,
17.ts
#EXTINF:13.440000,
18.ts
#EXTINF:9.640000,
19.ts
#EXTINF:8.480000,
20.ts
#EXTINF:9.880000,
21.ts
#EXTINF:9.920000,
22.ts
#EXTINF:9.160000,
23.ts
#EXTINF:10.960000,
24.ts
#EXTINF:8.720000,
25.ts
#EXTINF:10.840000,
26.ts
#EXTINF:13.080000,
27.ts
#EXTINF:6.680000,
28.ts
#EXTINF:10.000000,
29.ts
#EXTINF:12.440000,
30.ts
#EXTINF:8.440000,
31.ts
#EXTINF:9.240000,
32.ts
#EXTINF:11.040000,
33.ts
#EXTINF:8.080000,
34.ts
#EXTINF:10.080000,
35.ts
#EXTINF:11.280000,
36.ts
#EXTINF:10.000000,
37.ts
#EXTINF:8.800000,
38.ts
#EXTINF:10.160000,
39.ts
#EXTINF:13.360000,
40.ts
#EXTINF:6.840000,
41.ts
#EXTINF:13.680000,
42.ts
#EXTINF:5.760000,
43.ts
#EXTINF:10.320000,
44.ts
#EXTINF:13.200000,
45.ts
#EXTINF:7.800000,
46.ts
#EXTINF:8.680000,
47.ts
#EXTINF:12.640000,
48.ts
#EXTINF:9.600000,
49.ts
#EXTINF:8.440000,
50.ts
#EXTINF:11.360000,
51.ts
#EXTINF:7.960000,
52.ts
#EXTINF:10.840000,
53.ts
#EXTINF:9.240000,
54.ts
#EXTINF:10.800000,
55.ts
#EXTINF:11.200000,
56.ts
#EXTINF:9.640000,
57.ts
#EXTINF:9.880000,
58.ts
#EXTINF:8.840000,
59.ts
#EXTINF:10.960000,
60.ts
#EXTINF:10.440000,
61.ts
#EXTINF:10.960000,
62.ts
#EXTINF:8.160000,
63.ts
#EXTINF:9.160000,
64.ts
#EXTINF:10.600000,
65.ts
#EXTINF:13.320000,
66.ts
#EXTINF:5.880000,
67.ts
#EXTINF:10.920000,
68.ts
#EXTINF:10.200000,
69.ts
#EXTINF:8.920000,
70.ts
#EXTINF:10.800000,
71.ts
#EXTINF:10.480000,
72.ts
#EXTINF:9.160000,
73.ts
#EXTINF:9.920000,
74.ts
#EXTINF:10.760000,
75.ts
#EXTINF:11.480000,
76.ts
#EXTINF:7.480000,
77.ts
#EXTINF:11.000000,
78.ts
#EXTINF:10.280000,
79.ts
#EXTINF:9.640000,
80.ts
#EXTINF:12.480000,
81.ts
#EXTINF:8.600000,
82.ts
#EXTINF:7.840000,
83.ts
#EXTINF:13.080000,
84.ts
#EXTINF:9.120000,
85.ts
#EXTINF:8.760000,
86.ts
#EXTINF:9.160000,
87.ts
#EXTINF:11.280000,
88.ts
#EXTINF:9.000000,
89.ts
#EXTINF:10.800000,
90.ts
#EXTINF:11.200000,
91.ts
#EXTINF:9.760000,
92.ts
#EXTINF:8.880000,
93.ts
#EXTINF:9.600000,
94.ts
#EXTINF:9.960000,
95.ts
#EXTINF:11.520000,
96.ts
#EXTINF:9.480000,
97.ts
#EXTINF:9.960000,
98.ts
#EXTINF:10.960000,
99.ts
#EXTINF:11.680000,
100.ts
#EXTINF:5.800000,
101.ts
#EXTINF:12.920000,
102.ts
#EXTINF:7.560000,
103.ts
#EXTINF:10.600000,
104.ts
#EXTINF:11.400000,
105.ts
#EXTINF:7.720000,
106.ts
#EXTINF:9.880000,
107.ts
#EXTINF:11.200000,
108.ts
#EXTINF:10.440000,
109.ts
#EXTINF:8.720000,
110.ts
#EXTINF:11.400000,
111.ts
#EXTINF:9.040000,
112.ts
#EXTINF:9.320000,
113.ts
#EXTINF:10.200000,
114.ts
#EXTINF:11.120000,
115.ts
#EXTINF:11.080000,
116.ts
#EXTINF:8.240000,
117.ts
#EXTINF:9.160000,
118.ts
#EXTINF:10.480000,
119.ts
#EXTINF:9.520000,
120.ts
#EXTINF:10.480000,
121.ts
#EXTINF:11.640000,
122.ts
#EXTINF:9.080000,
123.ts
#EXTINF:9.840000,
124.ts
#EXTINF:9.280000,
125.ts
#EXTINF:12.840000,
126.ts
#EXTINF:9.600000,
127.ts
#EXTINF:7.320000,
128.ts
#EXTINF:13.200000,
129.ts
#EXTINF:7.240000,
130.ts
#EXTINF:10.160000,
131.ts
#EXTINF:11.680000,
132.ts
#EXTINF:9.760000,
133.ts
#EXTINF:8.960000,
134.ts
#EXTINF:12.520000,
135.ts
#EXTINF:8.160000,
136.ts
#EXTINF:8.680000,
137.ts
#EXTINF:11.360000,
138.ts
#EXTINF:11.360000,
139.ts
#EXTINF:9.600000,
140.ts
#EXTINF:7.320000,
141.ts
#EXTINF:11.320000,
142.ts
#EXTINF:9.480000,
143.ts
#EXTINF:11.520000,
144.ts
#EXTINF:7.960000,
145.ts
#EXTINF:12.080000,
146.ts
#EXTINF:10.120000,
147.ts
#EXTINF:7.440000,
148.ts
#EXTINF:10.000000,
149.ts
#EXTINF:10.360000,
150.ts
#EXTINF:13.800000,
151.ts
#EXTINF:9.440000,
152.ts
#EXTINF:7.080000,
153.ts
#EXTINF:11.600000,
154.ts
#EXTINF:8.040000,
155.ts
#EXTINF:9.720000,
156.ts
#EXTINF:9.960000,
157.ts
#EXTINF:11.240000,
158.ts
#EXTINF:11.840000,
159.ts
#EXTINF:10.120000,
160.ts
#EXTINF:6.840000,
161.ts
#EXTINF:13.800000,
162.ts
#EXTINF:6.160000,
163.ts
#EXTINF:11.800000,
164.ts
#EXTINF:9.600000,
165.ts
#EXTINF:13.000000,
166.ts
#EXTINF:6.280000,
167.ts
#EXTINF:10.120000,
168.ts
#EXTINF:10.200000,
169.ts
#EXTINF:9.520000,
170.ts
#EXTINF:10.240000,
171.ts
#EXTINF:11.920000,
172.ts
#EXTINF:7.680000,
173.ts
#EXTINF:12.480000,
174.ts
#EXTINF:7.320000,
175.ts
#EXTINF:13.000000,
176.ts
#EXTINF:10.160000,
177.ts
#EXTINF:10.080000,
178.ts
#EXTINF:7.000000,
179.ts
#EXTINF:13.080000,
180.ts
#EXTINF:8.480000,
181.ts
#EXTINF:8.280000,
182.ts
#EXTINF:12.280000,
183.ts
#EXTINF:10.000000,
184.ts
#EXTINF:8.000000,
185.ts
#EXTINF:9.880000,
186.ts
#EXTINF:9.760000,
187.ts
#EXTINF:10.000000,
188.ts
#EXTINF:10.440000,
189.ts
#EXTINF:10.080000,
190.ts
#EXTINF:10.080000,
191.ts
#EXTINF:11.920000,
192.ts
#EXTINF:9.120000,
193.ts
#EXTINF:11.920000,
194.ts
#EXTINF:7.600000,
195.ts
#EXTINF:9.560000,
196.ts
#EXTINF:9.720000,
197.ts
#EXTINF:11.360000,
198.ts
#EXTINF:8.200000,
199.ts
#EXTINF:10.640000,
200.ts
#EXTINF:11.280000,
201.ts
#EXTINF:8.680000,
202.ts
#EXTINF:10.200000,
203.ts
#EXTINF:9.800000,
204.ts
#EXTINF:9.600000,
205.ts
#EXTINF:10.440000,
206.ts
#EXTINF:11.760000,
207.ts
#EXTINF:8.680000,
208.ts
#EXTINF:9.600000,
209.ts
#EXTINF:12.600000,
210.ts
#EXTINF:6.800000,
211.ts
#EXTINF:10.720000,
212.ts
#EXTINF:10.840000,
213.ts
#EXTINF:9.600000,
214.ts
#EXTINF:8.840000,
215.ts
#EXTINF:11.680000,
216.ts
#EXTINF:9.840000,
217.ts
#EXTINF:9.240000,
218.ts
#EXTINF:9.600000,
219.ts
#EXTINF:9.440000,
220.ts
#EXTINF:9.960000,
221.ts
#EXTINF:11.080000,
222.ts
#EXTINF:12.000000,
223.ts
#EXTINF:7.520000,
224.ts
#EXTINF:12.520000,
225.ts
#EXTINF:10.840000,
226.ts
#EXTINF:6.200000,
227.ts
#EXTINF:14.600000,
228.ts
#EXTINF:9.600000,
229.ts
#EXTINF:9.600000,
230.ts
#EXTINF:6.920000,
231.ts
#EXTINF:10.320000,
232.ts
#EXTINF:9.040000,
233.ts
#EXTINF:10.040000,
234.ts
#EXTINF:9.960000,
235.ts
#EXTINF:10.880000,
236.ts
#EXTINF:12.280000,
237.ts
#EXTINF:7.680000,
238.ts
#EXTINF:9.840000,
239.ts
#EXTINF:9.720000,
240.ts
#EXTINF:11.200000,
241.ts
#EXTINF:9.800000,
242.ts
#EXTINF:11.400000,
243.ts
#EXTINF:11.160000,
244.ts
#EXTINF:5.840000,
245.ts
#EXTINF:14.320000,
246.ts
#EXTINF:7.400000,
247.ts
#EXTINF:9.600000,
248.ts
#EXTINF:8.640000,
249.ts
#EXTINF:10.760000,
250.ts
#EXTINF:9.880000,
251.ts
#EXTINF:12.800000,
252.ts
#EXTINF:7.920000,
253.ts
#EXTINF:11.200000,
254.ts
#EXTINF:9.600000,
255.ts
#EXTINF:8.720000,
256.ts
#EXTINF:9.320000,
257.ts
#EXTINF:11.000000,
258.ts
#EXTINF:9.320000,
259.ts
#EXTINF:10.400000,
260.ts
#EXTINF:9.960000,
261.ts
#EXTINF:9.760000,
262.ts
#EXTINF:9.720000,
263.ts
#EXTINF:10.160000,
264.ts
#EXTINF:13.360000,
265.ts
#EXTINF:6.120000,
266.ts
#EXTINF:11.680000,
267.ts
#EXTINF:8.400000,
268.ts
#EXTINF:9.480000,
269.ts
#EXT-X-ENDLIST

After the ts is downloaded, it is a blurry screen. It should be that the audio and video are separated and encrypted, so let’s do it~ Since it is encrypted, we will not download it

Anti-leech instructions

Anti-leeching in the IPTV system is a lot of headaches. There are many ways to prevent hotlinking, such as dynamic key, video address as hotlinking, p2p private protocol, etc., each of which has its own advantages and disadvantages. In fact, in addition to these, there is also the encryption of video streams. This is used more in video on demand, but it can also be realized in live broadcasts, that is, to encrypt the video in ts or m3u8 format provided by the copyright party, and to encrypt the content. Frame encryption, after encryption, it can only be played in the allowed APP, even if it is downloaded to other APPs, it cannot be played.

The whole usage process is as follows:

1. Deploy the point-to-point streaming media system. When transferring the live video stream provided by the copyright party, the encryption function is directly turned on, and the video stream is automatically encrypted. After encryption, the channel address is directly given.

2. Put the encrypted channel address in the CMS management background, and perform technical docking on the APP player side, so that the encrypted video can only be used in a specific APP. Moreover, the content frame is encrypted, which is more secure, and the encrypted video will be deleted after playing without saving.

3. After the video is encrypted once, it can be used in the APP under the Android and iOS systems, which is convenient and fast. With the Dianquan CMS background management system, one background can be used for multi-platform APPs.

4. It can also be connected to the existing system and used directly in the form of SDK, which will not affect the current user experience.