Talking about the development of medical AI software testing technology

I. Introduction

With the rapid development of artificial intelligence, AI has also played a huge role in the medical field. Many domestic medical companies or organizations have invested in the research and development of medical AI software, such as:

In 2020 , Lepu Medical developed the electrocardiographic analysis software (AI—ECGPlatform) , which is based on artificial intelligence deep learning technology and automatically analyzes electrocardiogram events such as arrhythmia, myocardial infarction, ventricular hypertrophy, and ST—T abnormalities, and is generally accurate. The rate has reached more than 95%.

Coronary CT angiography image vascular stenosis auxiliary triage software , an AI medical product of Shukun Technology Co., Ltd., is characterized by automatic and intelligent completion of cardiac image reconstruction, plaque display of official lumen stenosis detection machine, plaque nature judgment, report generation, etc. Worked and produced a structured report, reviewed on November 10, 2020.        

In 2022, NeuBrainCARE (cerebral ischemia image-aided evaluation software) of Neusoft Medical won the first NMPA Class III certificate in the field of stroke in China . It can provide a perfusion map including 5 parameters of CBV, CBF, MTT, TTP, and Tmax within 90 seconds, and the accuracy rate is as high as 95% in detecting the core infarct area and ischemic penumbra.

It can be seen that medical AI software can well assist medical staff in quickly diagnosing and treating diseases. However, medical AI software is different from traditional software, with complex functions and high clinical risks. In the testing process, the lack of standards and more reliance on clinical trials have brought many difficulties and challenges.

2. Development of medical AI software technology

Medical AI software is a software that uses artificial intelligence technology to assist medical work. It can improve the accuracy and efficiency of medical diagnosis and treatment through data analysis, image recognition, natural language processing and other technologies, and help medical workers better handle complex medical data and information.

Medical AI software can be applied in many ways, such as:

1. Medical image analysis: Use deep learning algorithms to analyze medical images, such as X-rays, MRIs, and CT scans, to diagnose diseases and evaluate treatment effects.

2. Personalized treatment: Provide patients with more accurate and personalized treatment plans based on individualized data of patients, such as genomic information, medical record data and vital signs.

3. Medical consultation: Use natural language processing technology to analyze a large amount of medical data, such as medical literature and patient history records, to help medical workers better understand patients' conditions and provide more professional medical consultation services.

The application of medical AI software has many benefits, which can improve medical efficiency and accuracy, reduce medical costs, and allow doctors to pay more attention to humanized medical services. The difference between medical AI software and traditional medical software is as follows: 

In recent years, my country's medical AI software technology has developed rapidly, mainly in the following aspects:

1. Medical image recognition technology: Through deep learning and other technologies, the intelligent recognition, classification and segmentation of medical images can be realized, which can be used for early diagnosis of tumors, lesion localization and surgical assistance. For example: CerebralDoc, the head and neck CT angiography image-assisted evaluation software (CerebralDoc) of Shukun Technology, has been approved as the first NMPA Class III certificate in the field of head and neck CTA in China, which can assist doctors to quickly complete head and neck CT blood vessel image processing, such as automatic detection of vascular abnormalities and high-risk lesions such as lumps and aneurysms, etc., the qualified rate of reconstructed image quality is about 92.1%.

2. Medical auxiliary diagnostic technology: including artificial intelligence-based diagnostic auxiliary systems, intelligent interrogation systems, etc., which can help doctors improve the efficiency of diagnosis and treatment, reduce the workload of doctors, and improve the accuracy of diagnosis and treatment. For example: Tencent Healthcare's Intelligent Clinical Assisted Decision Support System (AI-CDSS) can use artificial intelligence technology to predict diseases in real time while doctors are writing medical records, and give examination or treatment recommendations to assist doctors in disease diagnosis and reduce misdiagnosis and missed diagnosis Rate.

3. Medical data analysis technology: Through the mining and analysis of medical data, it is possible to discover the regularity of disease occurrence, the effect of diagnosis and treatment, etc., to help doctors make more scientific diagnosis and treatment decisions, and at the same time promote the improvement of the quality of medical services. For example: Sulin Technology's intelligent medical statistical analysis platform , a one-stop medical statistical analysis platform for clinical researchers-provides data governance, data statistics, machine learning analysis, and intelligent article generation services.

4. Medical robotics: Through the application of robotics, precise and efficient surgical operations can be achieved, while reducing surgical risks and complications, and improving surgical safety and success rate. For example: ROIDMI RM-50 is the latest neurosurgery robot developed by Bohui Weikang, which can assist doctors to complete various types of cerebral hemorrhage, brain biopsy, craniotomy navigation and related minimally invasive brain operations.

5. Medical health management technology: Through data collection and analysis, individual health status can be assessed and predicted, and personalized health management programs can be formulated to promote the development of a healthy lifestyle and disease prevention. For example: Tencent Healthcare's regional smart medical platform , combined with advanced technologies such as artificial intelligence, big data, and smart devices, integrates public health population health information management with family doctor contract service management, and performs health tracking, management, and services for key populations and timely intervention.

3. Development of medical AI software testing technology

As the application of medical AI software becomes more and more extensive, the requirements for the reliability and security of medical AI software are also getting higher and higher. In traditional software testing, testers only need to perform functional and interface tests against product manuals. The most significant difference between medical AI software testing and traditional medical software testing lies in the introduction of AI algorithms. Here are some differences:

1. Dataset testing: Medical AI software requires large datasets to train AI algorithms, so testers need to test the datasets to ensure the accuracy and completeness of the datasets. Traditional medical software does not require dataset testing because they usually only deal with small amounts of data.

2. Model testing: Medical AI software needs to test the accuracy and efficiency of AI algorithms, so algorithms need to be tested and validated. Traditional medical software does not require such testing, as they typically use proven algorithms.

3. Automated testing: Medical AI software requires automated testing to ensure that algorithms work properly on different datasets. Traditional medical software also requires automated testing, but the scope and purpose of testing are different.

4. Quality standards: Medical AI software needs to comply with the standards and regulations of the medical industry, such as the "Guidelines for the Registration and Review of Artificial Intelligence Medical Devices " to ensure the safety and effectiveness of the software.

5. Verification and Confirmation Testing: Medical AI software requires verification and confirmation testing to ensure that the software's specifications and requirements are met. Traditional medical software also requires this kind of testing, but the importance of this kind of testing is more obvious in medical AI software.

In short, medical AI software testing is more complex than traditional medical software testing, and requires more detailed testing methods and testing techniques to ensure the effectiveness and safety of the software. At present, medical AI companies that have been approved in my country basically use software such as interface testing tools (such as JMeter) and performance testing tools (such as Loadrunner) to test medical AI software functions, Performance and other aspects are tested, and there is a lack of dedicated AI testing tools and methods.

The main functions and software types of AI software for auxiliary diagnosis that have obtained NMPA Class III certificates are shown in the table below:

my country's medical AI software testing technology is gradually developing, mainly from the standard evaluation database, stress testing, adversarial attacks, data pollution and other aspects of research.

(1) Standard evaluation database

Medical AI algorithm evaluation database is an important algorithm performance evaluation method. AI algorithms require a large amount of data for training, and medical data involves personal information of patients, which is often difficult to obtain, so public medical databases are often used for model training. Since the existing databases have problems such as quality and inconsistent standards, the establishment of a standard test database is conducive to unifying the evaluation standards of medical AI algorithms and ensuring the adequacy and objectivity of medical AI algorithm testing. Therefore, in 2021, Peking Union Medical College established an AI standard database for diabetic retinopathy; in 2021, the First Hospital of Jilin University joined forces with several units to establish the first pneumonia CT image AI standard database, etc.

 (2) Stress test

Disease data samples often have a long-tail effect, that is, a small number of data show extreme high or low conditions, while the vast majority of data show a moderate level of trend. The existence of the long tail effect makes us need to pay more attention to diseases with a small number of samples, because they may have a serious impact on human health, and some may even be rare diseases. Therefore, using rare or special clinical data samples to carry out algorithm performance testing is an important method to evaluate the generalization limit ability of algorithms.

(3) Adversarial attack

Adversarial attack is an important method to evaluate the robustness of algorithms, and also an important means to evaluate the network security of medical AI software. Check the resistance of the model by making modifications to the input data to improve the reliability and safety of the model. In the medical field, data sources are highly sensitive images and texts, which are vulnerable to adversarial perturbations, which affect the accuracy of deep learning model detection.

Adversarial attacks can be divided into the following types:

1. Adversarial example attack: The attacker modifies some samples in the original dataset so that the model makes mistakes in classifying these samples.

2. Adversarial training attack: An attacker tries to disrupt the training process of a model in order to train a vulnerable model.

3. Model evasion attack: The attacker tries to trick the model on unknown data to make wrong predictions.

4. Backdoor attack: The attacker injects malicious code into the model to make the model behave abnormally under certain input conditions.

(4) Data pollution

Data pollution refers to the addition or modification of false, false, or intentionally misleading data in a dataset in order to affect the performance or output of a machine learning model. In medical AI software testing, data contamination can have a significant impact on a model's test results. Therefore, data contamination should be considered as a key testing factor.

Data pollution in medical AI software testing can take several forms:

1. Adding Noise: Add random noise to medical images or data to simulate real-world noise and uncertainty. This can help test the robustness and stability of the model.

2. Removal of data: Part of the data is removed from the medical dataset to test the robustness and accuracy of the model to missing data.

3. Modify labels: Modify labels or annotations in medical datasets to test the sensitivity and accuracy of models to mislabeling or labeling errors.

4. Image or data modification: Medical images or data are modified to test the model's sensitivity and accuracy to image or data contamination.

Additionally, various data augmentation techniques can be used to increase the diversity and robustness of datasets to help test the performance and reliability of medical AI software.

4. Summary and Outlook

Medical AI software is software that uses artificial intelligence technology to process, analyze and diagnose medical images, data and information. Medical AI software can help doctors and medical professionals diagnose and treat diseases more accurately, improving the quality and efficiency of healthcare, while also providing patients with better medical services and treatment experiences.

The prospect of medical AI software is very broad. With the continuous development of artificial intelligence technology and continuous innovation in the medical field, medical AI software will play a more important role in the future. It will become an important part of healthcare, helping doctors and patients to better manage and treat diseases, improving the efficiency and quality of healthcare. At the same time, medical AI software will also promote the continuous development and progress of the medical field, bringing more opportunities and challenges. The medical AI software testing technology will gradually mature with the development and application of medical AI!

references:

[1] Peng Runlin, Wu Kai, Chen Chaomin, Deng Zeya, Chen Mingxiang, Zhou Jing. Overview of the development of medical artificial intelligence software and testing technology in my country [J]. China Medical Equipment, 2023, 38(04): 155-160.

[2] Wang Chenxi, Wang Quan, Li Jiage. Requirements and methods for independent software testing of medical devices [J]. China Medical Equipment, 2020,35(11):66-69+76.