Core Dump：调试 ESP32 开发板上 Zephyr 程序错误的强大工具

业界资讯 2023-08-26 02:12:06 阅读次数: 0

Zephyr 是一个开源、可扩展、适应性强的实时操作系统 (RTOS)，能够在多种硬件平台上运行，包括 乐鑫 的 ESP32、ESP32-S2、ESP32-C3 和 ESP32-S3 系列产品。Zephyr 为嵌入式系统开发提供了丰富的功能，包括针对无法恢复的软件错误生成可供分析的 core dump。 

Core dump 是在程序崩溃时保留的内存快照，可用于调试程序，查找崩溃原因、以及确定导致崩溃的具体任务、代码行和调用堆栈。如果需要，还可以为变量赋值并存储变量内容。 

Zephyr 操作系统支持多个 core dump 后端，其中包括日志后端。日志后端将 core dump 输出到 UART，然后保存到文件中，这样就可以使用自定义的 GDB 服务器和 SDK 提供的 GDB 进行分析。 

如何使用 

要使用 core dump 的日志后端，需要启用相应的 Kconfig 选项： 

CONFIG_DEBUG_COREDUMP=y  
CONFIG_DEBUG_COREDUMP_BACKEND_LOGGING=y 

一旦启用了日志后端，应用程序在遇到致命错误时将生成 core dump。CPU 寄存器和内存内容将被打印到控制台。 

将内容复制并粘贴到一个名为 coredump.log 的文件中。 

我们需要将这个文本文件转换为自定义 GDB 服务器可以解析的二进制文件。只需运行 coredump_serial_log_parser.py 脚本即可。 

./scripts/coredump/coredump_serial_log_parser.py coredump.log coredump.bin

该脚本会把二进制文件输出到 coredump.bin。

使用 zephyr 应用程序中的 .elf 文件，并将 coredump.bin 作为参数，启动自定义的 GDB 服务器：

./scripts/coredump/coredump_gdbserver.py build/zephyr/zephyr.elfcoredump.bin

打开另一个终端，从 Zephyr SDK 里运行 Xtensa 的 GDB，将 .elf 文件作为参数：

~/zephyr-sdk-0.16.0/xtensa-espressif_esp32_zephyr-elf/bin/xtensa-espressif_esp32_zephyr-elf-gdb build/zephyr/zephyr.elf

在 GDB 内部，使用以下命令连接到自定义服务器：

(gdb) target remote localhost:1234

现在，您可以检查程序崩溃时的状态，读取变量值、回溯信息和寄存器值了。

示例

下面是一个使用 core dump 进行调试的例子。 

我们在 tests/subsys/debug/coredump 里创建并烧录 core dump 示例。 

该示例定义了三个函数：func_1，func_2 和 func_3。func_1 调用 func_2，func_2 调用func_3。func_3 试图对空指针进行解引用，这将导致程序崩溃。 

程序崩溃会生成一个 core dump 文件，我们将用它来调试程序。 

1.创建应用程序： 

west build -p -b esp32 tests/subsys/debug/coredump

2.烧录并监控 ESP32 开发板： 

west flash && west espressif monitor

预期输出： 

*** Booting Zephyr OS build zephyr-v3.3.0-3986-gebf86941118f ***  
Coredump: esp32  
E:  ** FATAL EXCEPTION  
E:  ** CPU 0 EXCCAUSE 29 (store prohibited)  
E:  **  PC 0x400d0435 VADDR (nil)  
E:  **  PS 0x60620  
E:  **    (INTLEVEL:0 EXCM: 0 UM:1 RING:0 WOE:1 OWB:6 CALLINC:2)  
E:  **  A0 0x80081716  SP 0x3ffe65e0  A2 0x3f401cb8  A3 (nil)  
E:  **  A4 0x3f401cb8  A5 0xff  A6 0x3ffb1ad8  A7 0x3ffe5dfc  
E:  **  A8 (nil)  A9 0x3ffe6590 A10 0x3f400968 A11 0x3f4012d0  
E:  ** A12 0x3ffb1ad8 A13 0x60420 A14 0x3ffe5dd8 A15 0x3ffe5e30  
E:  ** LBEG (nil) LEND (nil) LCOUNT (nil)  
E:  ** SAR 0x1b  
E: #CD:BEGIN#  
E: #CD:5a4501000500050000000000  
E: #CD:4101006800  
E: #CD:0202000135040d401d000000000000001b000000200606000000000016170880  
E: #CD:e065fe3fb81c403f00000000b81c403fff000000d81afb3ffc5dfe3f00000000  
E: #CD:9065fe3f6809403fd012403f000000009065fe3f6809403fd012403f00000000  
E: #CD:0000000000000000  
E: #CD:4d0100d81afb3f581bfb3f  
E: #CD:d81cfb3fd81cfb3f000000000180000000000000000000000000000000000000  
E: #CD:000000000000000000000000000000000000000000000000101bfb3f101bfb3f  
E: #CD:6d61696e00000000000000000000000000000000000000000000000000000000  
E: #CD:00000000305efe3f0008000000000000f5ffffff0864fe3f080afb3fffffffff  
E: #CD:4d0100305efe3f3066fe3f  
E: #CD:f4b6dbfcd7903c37a5f85ffb73a1c5ad7da3432d420e03142b7673dc1f70a451  
E: #CD:e4165f6f2491c43b0aae041aa6bad45d0d019d7bd7061165478aa8e8e9dfb05d  
E: #CD:5e9ca79574bd8eee1a446b5919d39bbdc7a78c067dd9348e7dd99f3bc0ae0e59  
E: #CD:05a5d5e1459c6325d537dd4db75d0210f30e67e7e292f5af90ea8bab11d9090c  
E: #CD:14bacecb11dc47c19ae8d3922a290fcbe82ff95e23d63ad210459e8fc79dc78a  
E: #CD:e443c9dbf22b94b0ed95e23bba0f6006022225d48e85ba76d148caff2b2fd519  
E: #CD:324d6de3d17410eeae3d5a6513bfc8eed68141495be8c069e69dcea9748fd38e  
E: #CD:59fc1f6e57f36d52ba51d383d2df4ab9746d893b9bf276879d2182cfd03fe1fe  
E: #CD:fce68691b4df96c5ac234d34a8eb21f199191138061a17245042ab8ac94e14ac  
E: #CD:0e017619e4b42895aec67da7fdfb525168850547d0f2530247b1995dfaf9ddc5  
E: #CD:168588d974a379bcc3ddbead2edba5571dcbdd29c06ab11b41793d3ba2e15ab4  
E: #CD:da808c516438cd28215077c160bb5e0221b6c2d827db238850687d5cad9736f4  
E: #CD:60f9f777da01758a56b11a2fa9fc4bd9eaddb172f844223ea356ac0d5f77324f  
E: #CD:14417a13726f99d28c42bdedf33cc9a198dbe361b4974d152256806c0db076f8  
E: #CD:12b3e2c95e5dc84dd6d028065abf26054ff992b205ca641e249417dd7368bd92  
E: #CD:d8faf9e19687346bf31e315db64a05c049b23cf3a10eb60d6472211fdd43e8cc  
E: #CD:7b334dbb3b549c5b91bea3352ac68b3f6b199e2967344cc07da7d79c4bdc884c  
E: #CD:6ebcd3093df87d7633ccb583b036864cc48166283095024d592b16600f4ed38f  
E: #CD:bd1e1bdaf1538c6134c078d1f7b69f9467774e910b318854a0fcdd8841b12e0d  
E: #CD:74ca51cad7c6b020789c6affacc31d39a034d13c0ebaa3a07b53f81091d7b704  
E: #CD:8641f161cbaacd75c8aa1bd1b557470083e87921b4ae92960764aa2ad6eff203  
E: #CD:c32fa391df51c8c38aa88c5a54bf130009f26b64a8b2a15418b4286ab0edc3ec  
E: #CD:08d9c9c133fad8213c38e9ccb7505e2940c9612ceeb772440786012cddf5bbaf  
E: #CD:435c265247afa7e2cdb315fa23e47a45b9050d8cb61b4a72646f58febc81773f  
E: #CD:cbf027958c8a67478a2f9441bc94767ec90f446ba54ff82b893dddaddd42a004  
E: #CD:2a26cb016dfb2c1e0c691e319505c4167e83031ee83cd0843165c00b63fa4e96  
E: #CD:01f75bce679271064f56595a83ca5ae00ea79ba5c396f560a709dfa617e1dd14  
E: #CD:714ee1c87b95a422386d41a70f88df76019315875cc82b030dd005ba35af53bc  
E: #CD:029df20768fe4612c91133a5f38602715c05ef9b330ea662402b4779b8de6e8b  
E: #CD:6d8cad92e2fe7600ea3c4c83744186abc127d75f6f9334365e0c6b8c2ea6c82b  
E: #CD:7ea6b0095adbdbc82ba150a670db718a67d5fe5d8b9f04a48eafff8d2200e075  
E: #CD:d18277cc7bcbd8419bb21b15cfddc8340bd7721707bd54ff04fb72c71ec12ce7  
E: #CD:8ce9635cf260717cbe2cf8392f7796c24a2384dd9db2142dc00e7dc83c9b8119  
E: #CD:3e8a971d612a603e88062e7679ada4082e8fddc80cdd34bfd9df64f4a887d9b0  
E: #CD:920fff4ae8ee63e1e48e611e887334929a89ccce5ecb250c03ca59178b3eb5dc  
E: #CD:d88de936c368020f892357c7700d2b9ac237affbe9aad454736025fc37e7a4f0  
E: #CD:b60103116de8e418f17b1ffea7c3840ce9b5de74f2714451e7c774b0fc94d2d1  
E: #CD:8c3e9f17ac30161adf79c9cee52cfa79e62cc228781234124557f6463fe95228  
E: #CD:80d4d0b905bc05fb54e134ced7f84ac128ba82cda103f29f9a62e4c5f4105021  
E: #CD:80f085cb993e77c86f82ecf01dac257e25942f840eccd2e11385a946fd7b508c  
E: #CD:71e13192d9b4ceba06ae199b77127b529791be9b878657590a3c5a263d9c8319  
E: #CD:b9f9487e4086539f1e4758b87ca24b70887d185c7e617ff288bde4944e2c6cf3  
E: #CD:fb23feb7aadf90c80c6d00f66d287f80f52c7d593a108e9ff178fc30defc40d4  
E: #CD:3b59f4efbd708b0f0192d457450ad657d3b45fc4d254406d7ac94f4eafdc5be4  
E: #CD:ba1719090cbed14af915ca7991db7b87027ced573a0e6aff7078a83139df611f  
E: #CD:3fa23f95b5c39b7e15c92108a5322206fb74ee59e39327b66180c48dd9cbc1cb  
E: #CD:00c04eeb3550aa7e24de68fa4aa720be1ee74e8dc10a57a93c64fe3f00000000  
E: #CD:00c050eb0000000000000000703d088000c04eeb010000006d00000000000000  
E: #CD:3266fe3f0000000000000000db32008c00c050eb010000000000000000000000  
E: #CD:0a0000000000000020020600703d0840093d0880001008406d000000a01c403f  
E: #CD:7c330880a064fe3f6d00000000000000fedf99ececff33aec26b3658ed9c5be5  
E: #CD:283d0880c064fe3f6d00000006000000c01cfb3fd81afb3fd81afb3fd865fe3f  
E: #CD:4c3d08804065fe3f6809403fc065fe3fd012403f000000008000037300000000  
E: #CD:00000000000000001a2488bc891e7d358fc4ecfa81e597aa9df65bb7a6a74806  
E: #CD:c065fe3fa065fe3f04000000c065fe3fa065fe3ffc3c0840000000007509403f  
E: #CD:000000007209403fdb8a502716f5e7f50400000025000000a065fe3f04000000  
E: #CD:30040d809065fe3f6809403fd012403f000000003b4bed4f08035fadadc14494  
E: #CD:0a1751af3f39a63cf90f74b2820658d1cbb50403fb4800949c65fe3fd81afb3f  
E: #CD:20040600d85dfe3f305efe3f000000009065fe3f6809403fd012403fb81c403f  
E: #CD:ff000000d81afb3ffc5dfe3fdf2daf23000000001d0000000000000000000000  
E: #CD:000000001b0000002006060035040d401617088000100840b81c403f00000000  
E: #CD:56040d800066fe3f0000000000000000b81c403fff000000d81afb3ffc5dfe3f  
E: #CD:000000002066fe3ff81608400000000000000000000000007f00000000000000  
E: #CD:000000004066fe3f000000000000000000000000000000000000000000000000  
E: #CD:END#  
E: >>> ZEPHYR FATAL ERROR 0: CPU exception on CPU 0  
E: Current thread: 0x3ffb1ad8 (main)  
E: Halting system

Coredump内容以#CD:BEGIN#开始，以#CD:END#结束。我们需要将这两者之间的内容复制到一个名为 coredump.log 的新文件中。

3.将 core dump 复制到 coredump.log：

E: #CD:BEGIN# 
E: #CD:5a4501000500050000000000 
E: #CD:4101006800 
E: #CD:0202000135040d401d000000000000001b000000200606000000000016170880 
E: #CD:e065fe3fb81c403f00000000b81c403fff000000d81afb3ffc5dfe3f00000000 
E: #CD:9065fe3f6809403fd012403f000000009065fe3f6809403fd012403f00000000 
E: #CD:0000000000000000 
E: #CD:4d0100d81afb3f581bfb3f 
E: #CD:d81cfb3fd81cfb3f000000000180000000000000000000000000000000000000 
E: #CD:000000000000000000000000000000000000000000000000101bfb3f101bfb3f 
E: #CD:6d61696e00000000000000000000000000000000000000000000000000000000 
E: #CD:00000000305efe3f0008000000000000f5ffffff0864fe3f080afb3fffffffff 
E: #CD:4d0100305efe3f3066fe3f 
E: #CD:f4b6dbfcd7903c37a5f85ffb73a1c5ad7da3432d420e03142b7673dc1f70a451 
E: #CD:e4165f6f2491c43b0aae041aa6bad45d0d019d7bd7061165478aa8e8e9dfb05d 
E: #CD:5e9ca79574bd8eee1a446b5919d39bbdc7a78c067dd9348e7dd99f3bc0ae0e59 
E: #CD:05a5d5e1459c6325d537dd4db75d0210f30e67e7e292f5af90ea8bab11d9090c 
E: #CD:14bacecb11dc47c19ae8d3922a290fcbe82ff95e23d63ad210459e8fc79dc78a 
E: #CD:e443c9dbf22b94b0ed95e23bba0f6006022225d48e85ba76d148caff2b2fd519 
E: #CD:324d6de3d17410eeae3d5a6513bfc8eed68141495be8c069e69dcea9748fd38e 
E: #CD:59fc1f6e57f36d52ba51d383d2df4ab9746d893b9bf276879d2182cfd03fe1fe 
E: #CD:fce68691b4df96c5ac234d34a8eb21f199191138061a17245042ab8ac94e14ac 
E: #CD:0e017619e4b42895aec67da7fdfb525168850547d0f2530247b1995dfaf9ddc5 
E: #CD:168588d974a379bcc3ddbead2edba5571dcbdd29c06ab11b41793d3ba2e15ab4 
E: #CD:da808c516438cd28215077c160bb5e0221b6c2d827db238850687d5cad9736f4 
E: #CD:60f9f777da01758a56b11a2fa9fc4bd9eaddb172f844223ea356ac0d5f77324f 
E: #CD:14417a13726f99d28c42bdedf33cc9a198dbe361b4974d152256806c0db076f8 
E: #CD:12b3e2c95e5dc84dd6d028065abf26054ff992b205ca641e249417dd7368bd92 
E: #CD:d8faf9e19687346bf31e315db64a05c049b23cf3a10eb60d6472211fdd43e8cc 
E: #CD:7b334dbb3b549c5b91bea3352ac68b3f6b199e2967344cc07da7d79c4bdc884c 
E: #CD:6ebcd3093df87d7633ccb583b036864cc48166283095024d592b16600f4ed38f 
E: #CD:bd1e1bdaf1538c6134c078d1f7b69f9467774e910b318854a0fcdd8841b12e0d 
E: #CD:74ca51cad7c6b020789c6affacc31d39a034d13c0ebaa3a07b53f81091d7b704 
E: #CD:8641f161cbaacd75c8aa1bd1b557470083e87921b4ae92960764aa2ad6eff203 
E: #CD:c32fa391df51c8c38aa88c5a54bf130009f26b64a8b2a15418b4286ab0edc3ec 
E: #CD:08d9c9c133fad8213c38e9ccb7505e2940c9612ceeb772440786012cddf5bbaf 
E: #CD:435c265247afa7e2cdb315fa23e47a45b9050d8cb61b4a72646f58febc81773f 
E: #CD:cbf027958c8a67478a2f9441bc94767ec90f446ba54ff82b893dddaddd42a004 
E: #CD:2a26cb016dfb2c1e0c691e319505c4167e83031ee83cd0843165c00b63fa4e96 
E: #CD:01f75bce679271064f56595a83ca5ae00ea79ba5c396f560a709dfa617e1dd14 
E: #CD:714ee1c87b95a422386d41a70f88df76019315875cc82b030dd005ba35af53bc 
E: #CD:029df20768fe4612c91133a5f38602715c05ef9b330ea662402b4779b8de6e8b 
E: #CD:6d8cad92e2fe7600ea3c4c83744186abc127d75f6f9334365e0c6b8c2ea6c82b 
E: #CD:7ea6b0095adbdbc82ba150a670db718a67d5fe5d8b9f04a48eafff8d2200e075 
E: #CD:d18277cc7bcbd8419bb21b15cfddc8340bd7721707bd54ff04fb72c71ec12ce7 
E: #CD:8ce9635cf260717cbe2cf8392f7796c24a2384dd9db2142dc00e7dc83c9b8119 
E: #CD:3e8a971d612a603e88062e7679ada4082e8fddc80cdd34bfd9df64f4a887d9b0 
E: #CD:920fff4ae8ee63e1e48e611e887334929a89ccce5ecb250c03ca59178b3eb5dc 
E: #CD:d88de936c368020f892357c7700d2b9ac237affbe9aad454736025fc37e7a4f0 
E: #CD:b60103116de8e418f17b1ffea7c3840ce9b5de74f2714451e7c774b0fc94d2d1 
E: #CD:8c3e9f17ac30161adf79c9cee52cfa79e62cc228781234124557f6463fe95228 
E: #CD:80d4d0b905bc05fb54e134ced7f84ac128ba82cda103f29f9a62e4c5f4105021 
E: #CD:80f085cb993e77c86f82ecf01dac257e25942f840eccd2e11385a946fd7b508c 
E: #CD:71e13192d9b4ceba06ae199b77127b529791be9b878657590a3c5a263d9c8319 
E: #CD:b9f9487e4086539f1e4758b87ca24b70887d185c7e617ff288bde4944e2c6cf3 
E: #CD:fb23feb7aadf90c80c6d00f66d287f80f52c7d593a108e9ff178fc30defc40d4 
E: #CD:3b59f4efbd708b0f0192d457450ad657d3b45fc4d254406d7ac94f4eafdc5be4 
E: #CD:ba1719090cbed14af915ca7991db7b87027ced573a0e6aff7078a83139df611f 
E: #CD:3fa23f95b5c39b7e15c92108a5322206fb74ee59e39327b66180c48dd9cbc1cb 
E: #CD:00c04eeb3550aa7e24de68fa4aa720be1ee74e8dc10a57a93c64fe3f00000000 
E: #CD:00c050eb0000000000000000703d088000c04eeb010000006d00000000000000 
E: #CD:3266fe3f0000000000000000db32008c00c050eb010000000000000000000000 
E: #CD:0a0000000000000020020600703d0840093d0880001008406d000000a01c403f 
E: #CD:7c330880a064fe3f6d00000000000000fedf99ececff33aec26b3658ed9c5be5 
E: #CD:283d0880c064fe3f6d00000006000000c01cfb3fd81afb3fd81afb3fd865fe3f 
E: #CD:4c3d08804065fe3f6809403fc065fe3fd012403f000000008000037300000000 
E: #CD:00000000000000001a2488bc891e7d358fc4ecfa81e597aa9df65bb7a6a74806 
E: #CD:c065fe3fa065fe3f04000000c065fe3fa065fe3ffc3c0840000000007509403f 
E: #CD:000000007209403fdb8a502716f5e7f50400000025000000a065fe3f04000000 
E: #CD:30040d809065fe3f6809403fd012403f000000003b4bed4f08035fadadc14494 
E: #CD:0a1751af3f39a63cf90f74b2820658d1cbb50403fb4800949c65fe3fd81afb3f 
E: #CD:20040600d85dfe3f305efe3f000000009065fe3f6809403fd012403fb81c403f 
E: #CD:ff000000d81afb3ffc5dfe3fdf2daf23000000001d0000000000000000000000 
E: #CD:000000001b0000002006060035040d401617088000100840b81c403f00000000 
E: #CD:56040d800066fe3f0000000000000000b81c403fff000000d81afb3ffc5dfe3f 
E: #CD:000000002066fe3ff81608400000000000000000000000007f00000000000000 
E: #CD:000000004066fe3f000000000000000000000000000000000000000000000000 
E: #CD:END#

4.转换为二进制格式：

./scripts/coredump/coredump_serial_log_parser.py coredump.log coredump.bin

5.启动自定义的 GDB 服务器：

./scripts/coredump/coredump_serial_log_parser.py coredump.log coredump.bin

预期输出：

[INFO][gdbstub] Log file: coredump.bin  
[INFO][gdbstub] ELF file: build/zephyr/zephyr.elf  
[INFO][parser] Reason: K_ERR_CPU_EXCEPTION  
[INFO][parser] Pointer size 32  
[INFO][parser] Memory: 0x3ffb1ad8 to 0x3ffb1b58 of size 128  
[INFO][parser] Memory: 0x3ffe5e30 to 0x3ffe6630 of size 2048  
[INFO][parser] ELF Section: 0x0 to 0x1f of size 32 (read-only data)  
[INFO][parser] ELF Section: 0x20 to 0x3b of size 28 (read-only data)  
[INFO][parser] ELF Section: 0x40080000 to 0x400803ff of size 1024 (text)  
[INFO][parser] ELF Section: 0x40080400 to 0x40083e7b of size 14972 (text)  
[INFO][parser] ELF Section: 0x3f400040 to 0x3f401bff of size 7104 (read-only data)  
[INFO][parser] ELF Section: 0x3f401c00 to 0x3f401c3f of size 64 (read-only data)  
[INFO][parser] ELF Section: 0x3f401c40 to 0x3f401cb7 of size 120 (read-only data)  
[INFO][parser] ELF Section: 0x3f401cb8 to 0x3f401cd9 of size 34 (read-only data)  
[INFO][parser] ELF Section: 0x3ffb0a20 to 0x3ffb0a57 of size 56 (read-only data)  
[INFO][parser] ELF Section: 0x400d0020 to 0x400d3d53 of size 15668 (text)  
[INFO][gdbstub] Waiting GDB connection on port 1234...

6.打开一个新的终端，启动 Zephyr SDK 中的 Xtensa ESP32 GDB：

~/zephyr-sdk-0.16.0/xtensa-espressif_esp32_zephyr-elf/bin/xtensa-espressif_esp32_zephyr-elf-gdb build/zephyr/zephyr.elf

7.在 GDB 内部，连接到远程服务器：

(gdb) target remote localhost:1234 
Remote debugging using localhost:1234
0x400d0435 in func_3 (addr=0x0) at zephyr/tests/subsys/debug/coredump/src/main.c:27 
27              *addr = 0;

8.运行 bt 命令查看回溯信息：

(gdb) bt  
#0  0x400d0435 in func_3 (addr=0x0)  
    at zephyr/tests/subsys/debug/coredump/src/main.c:27  
#1  func_2 (addr=0x0) at zephyr/tests/subsys/debug/coredump/src/main.c:40  
#2  func_1 (addr=0x0) at zephyr/tests/subsys/debug/coredump/src/main.c:45  
#3  main () at zephyr/tests/subsys/debug/coredump/src/main.c:52

结论

Core dump 是一个强大的工具，用于调试 ESP32 开发板上的 Zephyr 程序错误。通过使用 core dump 的日志后端，您可以轻松生成 core dump 文件，并使用调试器进行分析。这将帮助您快速、方便地找出程序崩溃的原因，并修复 bug。

猜你喜欢

转载自blog.csdn.net/espressif/article/details/132302408
今日推荐
TIOBE 5 月榜单：Fortran “复活”进入 Top 10
GCC 14.1 发布
面壁智能发布 Eurux-8x22B 开源大模型 —— 堪称「理科状元」
开源日报 | 谷歌扶持鸿蒙上位；开源Rabbit R1；Docker加持的安卓手机；微软的焦虑和野心；海尔电器把开放平台关了
中国码农的“35岁魔咒”
蘭雅 CorelDRAW 插件 2024.5.1 国际劳动节版，免费下载
Arc Browser for Windows 1.0 正式 GA
90后程序员开发视频搬运软件、不到一年获利超 700 万，结局很刑！
周排行
Java自定义时间格式
同步整形电路
在开发中最最最常用的字符串的属性大集合
Linux 查看端口占用并杀掉
Java基础四：ArrayList
多线程之死锁就是这么简单
mysql 基础命令集
awk 命令详解
Centos6.3编译安装nginx+php步骤
OCR （Optical Character Recognition，光学字符识别）
每日归档
更多
2024-05-08(42)
2024-05-07(14)
2024-05-06(40)
2024-05-05(0)
2024-05-04(7)
2024-05-03(19)
2024-05-02(0)
2024-05-01(4)
2024-04-30(1)
2024-04-29(40)

代码天地 © 2018 codetd.com

境外服务器   最新电视剧2022