springboot episode 44: Kafka cluster and Lua scripts

servers: The address of the Kafka server. This is the address of the Kafka cluster that the producer will use to send messages.
retries: The number of times the producer will try to resend the message if the message delivery fails. This property specifies the number of retries.
batchSize: Specifies the message size (in bytes) that the producer accumulates before sending a message. Sending multiple messages at once can improve performance.
linger: Specifies the time (in milliseconds) the producer waits before sending a message. This can help send multiple messages together to reduce network overhead.
bufferMemory: Specifies the memory size (in bytes) used by the producer to buffer messages waiting to be sent.

This is a custom annotation @Log, used to annotate methods. Here is a line-by-line detailed description of this annotation:

@Target({ElementType.PARAMETER, ElementType.METHOD})
@Retention(RetentionPolicy.RUNTIME)
@Documented
public @interface Log {
    /** 标题 */
    public String title() default "";

    /** 功能 */
    public BusinessType businessType() default BusinessType.OTHER;
}

The function of this custom annotation is as follows:

@Target({ElementType.PARAMETER, ElementType.METHOD}): Specifies that this annotation can be used on methods and method parameters.
@Retention(RetentionPolicy.RUNTIME): specifies that this annotation is retained at runtime so that it can be read reflectively at runtime.
@Documented: Indicates that this annotation should be included in JavaDoc.

Annotation elements:

public String title() default "": This is an annotation element used to specify the title of the log. The default value is an empty string, which can be specified via annotation.
public BusinessType businessType() default BusinessType.OTHER: This is an annotation element used to specify the business type of the log. The default value is BusinessType.OTHER, indicating other business types. Different business types can be specified through annotations.

This custom annotation can be used on methods to help describe the function and business type of the method, and is usually used for logging or other related operations. When using, you can add @Log annotations to the method and specify the title and business type. For example:

@Log(title = "用户登录", businessType = BusinessType.LOGIN)
public void userLogin() {
    // 方法实现
}

In this example, method userLogin is marked as a login operation and the associated title is specified. This helps with logging and monitoring the application's operations.

Spring Framework automatically assembles these dependencies through @Autowired annotations, thereby reducing the work of manual configuration and dependency management.

Redis Lua script, typically used to perform atomic operations in Redis. The main purpose of this script is to increment the value of a key stored in Redis and set the expiration time on the first increment. Here are the main steps of the script:

currentVariable: First, the script declares a local variablecurrent to store the current value of the key.
redis.call("incr", KEYS[1]): This line performs a Redis atomic operation, which increments the value of key KEYS[1] by 1 and returns the incremented value, which is stored in currentIn variables.
if tonumber(current) == 1 then: Next, the script checks whether the value of current is equal to 1. This is to determine whether it is the first increment. tonumber(current) is used to convert the value of current to an integer for comparison.
If the value of current is equal to 1, indicating the first increment, then the script will perform the following operations:

redis.call("expire", KEYS[1], KEYS[2]): The expiration time of the setting keyKEYS[1] is KEYS[2], that is, the setting key expires after a certain period of time.

Finally, the script returns the value of current, which is the incremented value.

Overall, this script is used to implement a counter that is incremented each time it is called, but only sets the expiration time on the first increment. This mode is often used to perform limits or timer functions on certain operations.

The main function of this Lua script is to implement a counter in Redis and set the expiration time of the key when it is first incremented. Usually, this function can be used to limit the number of times a user can perform an operation within a period of time, or to implement a short-term effective counting function.

Example 1: Limit the number of times a user can send text messages

Suppose you want to limit users to sending text messages only once per minute, you can use this Lua script. The script will increment the count. If the user has sent a text message within one minute, subsequent requests will return 0, indicating that no more sending is allowed.

local current
current = redis.call("incr", KEYS[1])
if tonumber(current) == 1 then
    redis.call("expire", KEYS[1], KEYS[2])
end
return current

Example 2: Timer

You can use this script to implement a simple timer that records the number of times a certain event occurs. For example, you can record the number of times an event occurs in an hour.

local current
current = redis.call("incr", KEYS[1])
if tonumber(current) == 1 then
    redis.call("expire", KEYS[1], 3600)  -- 设置过期时间为1小时
end
return current

In both examples, the script first attempts to increment the count and then sets the expiration time on the first increment. If the count exceeds a certain limit, subsequent requests will return a flag that the limit has been reached and no more increments are allowed. This can be used to implement many different types of counting and limiting functionality.

Use registry.addMapping("/**") to configure cross-domain settings, which means that the same CORS configuration is applied to all paths.
.allowCredentials(false)Specifies that sending of credential information, such as cookies, is not allowed. This indicates that the client's request does not contain sensitive credentials.
.allowedOrigins("*")Allow requests from any source, including different domain names or IP addresses. Use "*" to allow requests from any origin.
.allowedHeaders("*")All request headers are allowed, which means the client can send any request headers to the server.
.allowedMethods("GET", "PUT", "POST", "DELETE")Specify the allowed HTTP methods, the common GET, PUT, POST and DELETE methods are configured here.
.exposedHeaders("*")Expose all response headers to the client, allowing the client to access all response header information.

The message queue was born to solve the problem of busy communication between messages, embodying the implementation of decoupling and asynchronousness
In order to solve the problem of busy communication between messages, we can understand that the introduction of A middleware (message queue), when the sender sends information, it does not send it directly to the receiver, but sends the information to the middleware, and the receiver obtains the information it wants through the middleware.
In this process, we can understand the sender as the producer and the receiver as the consumer.
Producers publish information and consumers subscribe to information (through middleware)
This leads to a question, how do consumers get the data they want? The solution is topic. Producers publish information about different topics to middleware (kafka). Consumers consume the data they want by subscribing to different topics.
Under the topic There will be partitions, and the partitions can be distributed on different servers. The producer stores the data in different partitions under the topic.
Two modes: 1. Producer specifies partition 2. Partitioner ( An algorithm) that arranges the storage space of data by the key of the message (a tag)

We can now know that a message may contain the following data: 1. Topic 2. Partition 3. Key 4. Value (the data you want to transmit)

How do consumers read data? Export offset
Offset (offset): which number
In a partition, the offset of each message is unique
Consumers can only read sequentially

In this case, we have implemented a broker, which contains topics, partitions, and
The broker sets the offset according to the partition for the incoming message and stores it on the disk. The broker also provides services and responses to consumers.
Multiple Borker clusters are kafka clusters, which provides message security. There may also be a cluster responsible for the controller role in this cluster.

image.png

Step 1:vi ~/.zshrc
Step 2: Press i to enter
Add:< a i=4>Step 3: Press to enter: to save and exitsource ~/.bash_profile
esc:wq

MQTT (Message Queue Telemetry Transmission) is a network protocol (long connection, which means that in addition to the client can actively communicate with the server, the server can also actively initiate communication with the client). It is also based on TCP/IP and is suitable for people with low computing power. The hardware device uses a message protocol based on the publish\subscribe paradigm.

image.png

Arduino IDE (Integrated Development Environment) is an integrated development environment for developing Arduino, an open source hardware platform. Arduino is an electronics prototyping platform based on open source hardware and software designed to help electronics enthusiasts, students, and professional developers quickly and easily create a variety of interactive electronics projects. The Arduino IDE is used to write, upload, and run Arduino code and interact with the hardware on the Arduino board.

Arduino IDE provides the following main functions:

Code writing: You can use Arduino IDE to write Arduino program code, which is usually written in C/C++ language.
Code Editing: The IDE includes a code editor with syntax highlighting, code auto-completion, and debugging features to make writing code easier.
Uploading code: Once you've written your Arduino codes, you can upload them to the Arduino board to actually run the code and control the hardware.
Hardware Interaction: The Arduino IDE allows you to interact with a variety of sensors, actuators, and other peripherals on the Arduino board to create a variety of IoT, embedded systems, and electronic arts projects.
Library management: The Arduino community provides a large number of libraries for common hardware and sensors, which can be managed and imported through the IDE for easy use of these hardware components.
Debugging and monitoring: Arduino IDE provides some basic debugging and monitoring tools to help you check the execution of the code and the status of the hardware.
Project Management: The IDE allows you to manage multiple projects in order to organize and track different Arduino applications.

image.png

If you are an ECS cloud host, click the instance>Click your server name>Security Group>Configure Rules>Add manually

Just add this one:

image.png

If you are a lightweight server, just click Security>Firewall>Add Rules, which is very different from the ECS settings.

3 is for the front-end page, 2 is for the back-end, 1 is my personal super user, and wemos is for the device, which is the username and password entered when the device is connected.

image.png

1883 (mqtt default port) is opened, of course, in the same way as 18083.

1804 websockets default port
3306 mysql default port
1803

After you start it, try to use mqttx to connect to the broker again, and you will find that you can connect.

image.png

Add a subscription on the left, and messages about the topic will appear in the chat box on the right.

image.png

Send a device online prompt to home/status/ every second in the loop

The equipment, server, and emqx console are all running smoothly.

Needless to say about the front end, we use echarts to carry display data. Due to the small size, we do not use any framework and directly use jq and echarts to implement it.

<script src="https://cdn.bootcdn.net/ajax/libs/mqtt/4.1.0/mqtt.min.js"></script>

http {
    sendfile on;
    tcp_nopush on;
    tcp_nodelay on;
    keepalive_timeout 65;
    types_hash_max_size 2048;
    include /etc/nginx/mime.types;
    default_type application/octet-stream;

    ##
    # SSL Settings
    ##
    server {
    listen 80;
    server_name jshub.cn;
    #将请求转成https
    rewrite ^(.*)$ https://$host$1 permanent;
    }
    server {
        listen 443 ssl;
                server_name xxx.cn;
                location / {
                    root /larryzhu/web/release/toolbox;
                    index index.html index.htm;
                    try_files $uri $uri/ /index.html;
                }
     location /mqtt {
           proxy_pass http://localhost:8083;
           proxy_http_version 1.1;
           proxy_set_header Upgrade $http_upgrade;
           proxy_set_header Connection "upgrade";
         }
        # SSL 协议版本
        ssl_protocols TLSv1.2;
        # 证书
        ssl_certificate /larryzhu/web/keys/9263126_xxx.cn.pem;
        # 私钥
        ssl_certificate_key /larryzhu/web/keys/9263126_xxx.cn.key;
        # ssl_ciphers ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384;
        # ssl_ciphers AES128-GCM-SHA256:AES256-GCM-SHA384:AES128-SHA256;

        # 与False Start没关系，默认此项开启，此处减少抓包的干扰而关闭
        # ssl_session_tickets off;

        # return 200 "https ok \n";
  }

If the amount of data exceeds 43200 (one is inserted every two seconds, which is the amount of one day), call the stored procedure to delete the earliest piece of data

image.png

Open the console of EMQ cloud server "**http://127.0.0.1:18083^[1]", log in for the first time The default username is "admin" and the password is "public"**

./emqx start

The EMQX cloud server comes with a client debugging function panel. Simple message publishing and subscription testing can be performed in the tool -> WebSocket. The MQTT protocol is a lightweight transmission protocol based on the publish/subscribe mode.

The subscription of MQTT protocol messages needs to include the topic of the message and the quality of service. The topic of the message is "a label for sending the message". The quality of service includes 0, 1, and 2. See the following table for details:

Qos value	Bit2	Bit1	describe
0	0	0	Distribute at most once
1	0	1	distributed at least once
2	1	0	only distributed once

Join the group and contact the author vx: xiaoda0423

Warehouse address: https://github.com/webVueBlog/JavaGuideInterview

References

[1]

http://127.0.0.1:18083: https://link.juejin.cn/?target=http%3A%2F%2F127.0.0.1%3A18083

springboot episode 44: Kafka cluster and Lua scripts

References

Guess you like