If you don’t force yourself, you don’t know that you can still be so good-Little Goose Learning Records Large-scale Queue Synchronization Actual Combat Log

foreword
1. Introduction to programming languages
2. Show off code sharing
3. Case sharing

foreword

To non-programmers, a programmer's code can feel mysterious, because most people don't understand the programming language and technical details that programmers use, and they can't understand the various difficulties and problems that programmers face when writing code. thought process. And the way the code written by programmers runs on the computer is relatively mysterious, and most people cannot see the intermediate processes and results generated when the code is running. These factors can create a sense of mystery.

However, to programmers, code is not mysterious. For them, writing code is a routine job that requires constant analysis of problems, thinking about solutions, and converting them into code implementations. They understand the programming language and technical details, and can understand the various variables, data types and logical relationships involved in the code. When writing code, programmers will also debug and optimize between old and new code, further reducing the mystery of the code.

Although for programmers, code is not a mysterious thing, but a routine work product, but in the process of continuous breakthrough and growth, there is indeed a sense of harvest and accomplishment every time.

1. Introduction to programming languages

1. PHP is the best language in the world

"PHP is the best language in the world". PHP is widely used in web development. It has good stability and development efficiency, and it also has relatively complete development tools and community support. However, PHP also has some disadvantages, such as its relatively low performance, which may not be suitable for the development of large and complex projects.

However, for web development, considering factors such as project requirements, technical characteristics, and resource conditions, and pursuing low-cost and high-efficiency pragmatists, I have already boarded the boat and can't get off.

2. The distress of third-party API docking

The most troublesome thing in php development is to connect with third-party APIs. The density of hair volume grows almost inversely proportional to the length of the development time process. When connecting to third-party API interfaces, common problems include the following points:

Authentication issues: Third-party APIs require authentication before they can be used, which requires programmers to provide corresponding authentication information and perform authentication operations. During authentication, you need to pay attention to whether the provided authentication information is correct and meets the requirements of the API interface, otherwise the authentication may fail.
Parameter problem: When calling the API interface, you need to set request parameters, request headers, etc. When setting, you need to pay attention to whether the format and value of the parameters meet the requirements of the API interface, otherwise the request may fail.
Interface return format issues: Some third-party APIs may return data in various formats, such as XML, JSON, etc., which require programmers to perform corresponding data parsing and processing before they can be used. When processing, you need to pay attention to whether the data format is consistent or whether conversion is required.
Timeout problem: When calling the API interface, you may encounter request timeout or return data timeout, you need to set the corresponding timeout period or perform error handling to avoid program interruption due to timeout problems.
Security issues: When using third-party APIs, you need to pay attention to data security. Sensitive information such as passwords need to be encrypted or hidden according to the requirements of the API interface to protect data security.

3. Why the good project schedule is a picture

"I never developed according to the time of the project schedule, and it feels like every day is a deadline."

First, the complexity of programming work inevitably leads to uncertainty in the schedule. Programmers need to face various technical problems and complex business logic, and each problem takes a certain amount of time to solve. At the same time, programming work may also encounter various problems, such as code conflicts, code quality problems, etc. These problems will take time to solve, resulting in uncertainty in the schedule.

Second, changes and risks in the software development process are also one of the reasons for the constant adjustment of the schedule. In the process of software development, situations such as requirement changes and functional refactoring are often encountered, and these changes may affect the original schedule. In addition, there are various risks in the process of software development, such as the difficulty of technical realization, the change of partners, etc. These risks will also affect the stability of the project schedule.

To sum up, the uncertainty of the programmer's project schedule is caused by the complexity of the programming work and the changes and risks in the software development process. In order to avoid affecting the project schedule and development quality, programmers need to adjust the schedule in time to ensure that the project can be delivered on time and maintain the stability of project quality.

2. Show off code sharing

The more advanced programmers are, the more proficient they are in using ctrl+c and ctrl+v.

Personally, I basically agree but deny it a bit: because it’s not that you want ctrl+c and ctrl+v, you can ctrl+c and ctrl+v.

It is a misunderstanding to say that programmer development is all copy and paste. Although programmers often use existing code during the development process (also known as copying and pasting), this is only a small part of the development process.

In fact, programmers need to design algorithms, write codes, and test multiple stages in the development process. Designing algorithms requires programmers with solid knowledge of mathematics and computer science to understand and solve complex problems. Writing code requires programmers to master the programming language and development tools, be able to convert algorithms into actual code implementation, and also need to consider the quality, readability and maintainability of the code. Testing requires programmers to use various testing tools and techniques to ensure the correctness and stability of the program.

Although programmers will find existing code from the Internet or other code bases to improve efficiency, this does not mean that programmers will only copy and paste. When using existing code, programmers need to understand its logic and implementation methods, and make appropriate modifications and optimizations according to actual needs to meet individual needs. In addition, the copied and pasted code also needs to go through multiple steps such as testing and debugging to ensure the correctness and stability of the code.

In short, the work of programmers not only includes copying and pasting codes, but also requires rigorous thinking, solid knowledge and superb technology to be able to complete high-quality development work.

3. Case sharing

In mid-May, the third-party platform connected to Gooselink was launched ahead of schedule.

Testing, code adjustments, basically in the process of constant revision...
Party A's phone calls, WeChat and other means of bombing brought pressure...
Staying up late, eating out of order...
…
well, got over it. On DEADLINE, the large-scale queue synchronization of Xiaogetong learning records was finally completed, and the effect is as follows:

1. Project requirements

The number of single applicants exceeds 30,000, and the frequency of token acquisition and learning record update frequency of Xiaogetong is limited, so it cannot be processed directly in batches;
Due to the limitation of PHP scripting language, multi-threading cannot be performed: that is, the click event is automatically processed in the background;
After the training is over and the course is completed, the super administrator will complete the course progress and learning time at one time; individual members log in to their own background and update their own learning progress individually.

PHP has a sleep function, but it is very different from python's execution. PHP always executes the entire process before outputting on the front end, which is determined by the characteristics of PHP. How to let the administrator know the synchronization records in real time?

Doesn't javascript have setTimeoutfunctions? Time-sharing and batch execution should be able to solve this requirement.

2. Development practice

(1) Introduce external js library

    <!--layui核心框架-->
    <script src="static/layui/layui.js"></script>
    <link rel="stylesheet" href="static/layui/css/layui.css" media="all">

(2) Create an HTML container

<div class="layui-fluid">
    <!--提示说明-->
    <div class="layui-card">
        <div class="layui-card-body">
            <div class="layui-card">
                <div class="layui-card-body" style="padding: 15px;text-align: center;">
                    <button class="layui-btn" id="sync"><i class="layui-icon layui-icon-time"></i> 队列同步</button>
                    <button class="layui-btn layui-btn-danger" id="cancelSync"><i
                            class="layui-icon layui-icon-logout"></i>停止同步
                    </button>
                </div>
            </div>
            <div class="layui-card">
                <div class="layui-card-header"><strong>提示说明：</strong></div>
                <div class="layui-card-body">
                    <div class="layui-form-item">
                        <p>1.基于小鹅通token和学习记录API接口的限制，采用队列同步大数据量；</p>
                        <p class="x-red">2.同步前，请先同步用户小鹅通user_eid；</p>
                        <p>3.为减少小鹅通接口的使用频率，建议课程结束后，进展同步；</p>
                        <p>4.同步时，不能关闭窗口，请最小化后进行其他事件操作；</p>
                    </div>
                </div>
            </div>
        </div>
    </div>
    <!--同步信息记录-->
    <div class="layui-card">
        <div class="layui-card-body">
            <div class="layui-card">
                <div class="layui-card-header"><strong><span class="x-red" id="total"></span></strong></div>
                <div class="layui-card-body">
                    <div class="layui-form-item" id="content"></div>
                </div>
            </div>
        </div>
    </div>
</div>

(3) Queue synchronization delay execution

 //同步队列
    $("#sync").click(function () {
    
    
        $.getJSON('./api/api.php?act=getStudyUser&token=3cab7ce4142608c0f40c785b5ab5ca24', {
    
    course_id: course_id}, function (res) {
    
    
            //console.log(res.data);
            if (res.data.length == 0) {
    
    
                $("#total").html("学习记录全部同步，无需重复操作。")
            } else {
    
    
                //禁止操作
                $("#sync").attr("disabled", true).addClass("layui-btn layui-btn-primary");
                //同步数据
                outputArrayInBatch(res.data, 0, 90);
            }
        })
    })

    //停止执行
    $("#cancelSync").click(function () {
    
    
        location.reload();
    })

    //封装函数;
    function outputArrayInBatch(arr, batchNum, interval) {
    
    
        var startIndex = batchNum * interval; // 当前批次的起始索引
        var endIndex = (batchNum + 1) * interval; // 当前批次的结束索引
        if (startIndex > arr.length - 1) {
    
    
            return;
        }
        if (endIndex > arr.length) {
    
    
            endIndex = arr.length;
        }
        var batchArr = arr.slice(startIndex, endIndex);
        //console.log(batchArr);
        //计算剩余数量
        var total = arr.length - startIndex;
        //当结余数量不足完整批次时，自动清零
        if (total <= interval) {
    
    
            total = 0;
        }
        $("#total").html("总数量：" + total + "条记录，预计批次：" + Math.ceil(total / interval) + "，预计执行时间：" + parseFloat(total * 1 / interval / 60).toFixed(2) + "分钟");
        $("#content").append("<p>第" + (batchNum + 1) + "批次，完成同步...<p>");

        //同步记录并入库
        $.ajax({
    
    
            type: "post",
            url: "./api/api.php?act=updateStudyRecord&token=3cab7ce4142608c0f40c785b5ab5ca24",
            async: true,
            data: {
    
    
                course_id: course_id,
                resource_id: resource_id,
                batch_user_list: batchArr
            },
            dataType: "json",
            success: function (res) {
    
    
                console.log(res);
                if (res.code.code == 2054) {
    
    
                    layer.msg('小鹅通API的请求短时间突增，稍等片刻重新同步', {
    
    icon: 2, timeout: 2000}, function () {
    
    
                        location.reload();
                    })
                    return false;
                }
            },
            error: function (err) {
    
    
                console.log(err);
            }
        });
        //延迟执行
        setTimeout(function () {
    
    
            outputArrayInBatch(arr, batchNum + 1, interval);
        }, 1000); // 每隔1秒输出一批次
    }

(4) Obtain large amount of data

 public function getStudyUser()
    {
    
    
        checkAdminAuth($_COOKIE['admin_roles'], '0', 1);//超管权限
        global $db, $res;
        dbc();
        @$course_id = get_param('course_id');
        $sql = "select sign_id,user_eid FROM " . $db->table('sign_2023') . " WHERE sync_mark = 0 AND user_eid <> ''";
        $sql .= " AND course_id = " . $course_id;//必须指定课程ID
        $sql .= " ORDER BY sign_id ASC";//升序执行
        $row = $db->queryall($sql);
        $res["data"] = $row;
        die(json_encode_lockdata($res));
    }

(5) PHP batch update

Obtain learning records, update the learning duration and learning progress;
If there is no learning record, directly update the logo of whether it has been updated to avoid repeated updates;

 //学习记录-同步至数据表
    public function updateStudyRecord()
    {
    
    
        checkAdminAuth($_COOKIE['admin_roles'], '0', 1);//超管权限
        global $db, $res;
        dbc();
        //1.获取参数
        @$course_id = $_POST['course_id'];
        @$resource_id = $_POST['resource_id'];
        @$batch_user_list = $_POST['batch_user_list'];

        /*2.同步获取小鹅通学习记录
         * $res['xiaoE']['code'] 状态码
         * $res['xiaoE']['data']['list'],返回小鹅通学习记录数组
         * $res['xiaoE']['data']['list'][0]['learn_progress'],学习进度
         * $res['xiaoE']['data']['list'][0]['stay_time'],学习时间，单位为秒
         * $res['xiaoE']['data']['list'][0]['user_id'],小鹅通user_id
        */
        $user_list = array();
        for ($i = 0; $i < count($batch_user_list); $i++) {
    
    
            $user_list[] = $batch_user_list[$i]['user_eid'];
        }
        require_once '../libs/Client.php';
        $client = new Client();
        $url = "https://api.xiaoe-tech.com/xe.user.leaning_record_by_resource.get/1.0.0";
        $method = "post";
        $data = array("search_max_learn_progress" => 0,
            "stay_time" => 0,
            "list" => $user_list);
        $params = ["resource_id" => $resource_id, 'data' => $data];
        $result = $client->request($method, $url, $params);
        $dataList = $result['data']['list'];

        //3.同步数据
        $xiaoData_user = array();
        if ($dataList) {
    
    
            $xiaoData = array();
            $len = count($dataList);
            for ($j = 0; $j < $len; $j++) {
    
    
                $xiaoData_user[] = $dataList[$j]['user_id'];
                $xiaoData[] = array('user_eid' => $dataList[$j]['user_id'], 'learn_progress' => $dataList[$j]['learn_progress'], 'stay_time' => $dataList[$j]['stay_time'], 'sync_mark' => 1);
            }
            //4-1.有学习记录-同步数据
            if (!empty($xiaoData)) {
    
    
                $sql = batchUpdate($db->table('sign_2023'), $xiaoData, "user_eid", ["course_id" => $course_id, "sync_mark" => 0]);
                $db->query($sql);
            }
        }

        //4.无学习记录的-更改状态码
        $noXiaoData = array();
        $diffXiaoData = array_diff($user_list, $xiaoData_user);
        for ($n = 0; $n < count($diffXiaoData); $n++) {
    
    
            $noXiaoData[$n]['user_eid'] = $diffXiaoData[$n];
            $noXiaoData[$n]['sync_mark'] = 1;
        }
        if (!empty($noXiaoData)) {
    
    
            $sql = batchUpdate($db->table('sign_2023'), $noXiaoData, "user_eid", ["course_id" => $course_id, "sync_mark" => 0]);
            $db->query($sql);
        }
        //输出状态
        $res['code'] = $result;
        $res["data"] = "课程同步学习记录，Status:OK";
        die(json_encode_lockdata($res));
    }

(6) Little Goose Cloud API pays attention to the status in real time

insert image description here

(7) Front-end console printing test