Enabling the future of colonoscopy with intelligent and autonomous magnetic manipulation

这篇文章是Leeds组关于磁控内窥镜的各种理论/模块的结合起来的集大成者，发表在顶级期刊nature子刊machine intelligence。将自动驾驶中的自主性分级引入了智能机器人磁控内窥镜领域，总结了level-0至level-3的自主化磁控内窥镜，文中共介绍了三种机器人磁控内窥镜控制策略，对应于level-0的“直接控制机器人”，对应于level-1的“智能远程控制机器人”，和对应于level-3的“半自主化机器人”。详细介绍了在人类结肠仿体中的闭环转向/指位实验，三种机器人控制策略下的导航实验，比较了实验持续时间和成功率。还详细介绍了在猪结肠中的三种机器人控制策略下的导航实验，比较了实验持续时间。还介绍并评估了操作者（非专家，新手操作者）操作感受。详细总结性地介绍了磁控机器人内窥镜的磁控部分，图像转向/引导部分的理论。

通过智能和自主化磁操纵技术实现结肠镜检查的未来
Enabling the future of colonoscopy with intelligent and autonomous magnetic manipulation [1]
Paper Link
Authors: Martin, James W., et al.
2020, Nature Machine Intelligence

0. Abstract 摘要

结肠癌症的早期诊断实质上提高生存率。最终，超过一半的病例被太晚地诊断由于结肠镜的需求超过了容量，这是用于检查的黄金标准。结肠镜受限于传统内窥镜的过时的设计，这与使用的复杂性，费用，疼痛有关。磁内窥镜是一种有潜力的替代，并且克服疼痛和费用的缺点，但是他们努力达到过渡阶段当磁操纵是复杂的且不直观的。在这个工作中，我们使用机器视觉来开发磁内窥镜的智能的和自动化的控制，使非专家的用户能够有效地在内腔使用磁结肠镜。进一步说，我们定义为获取机器人内窥镜的自动化所需的特点。所这里描述的范例能被采纳在各种应用中，这些应用中的在非结构环境中的导航是需要的，比如导管，胰内窥镜，支气管镜和胃镜。这工作带来接近过渡阶段的可替代的内窥镜技术，增加了早期癌症治疗的可行性。
Early diagnosis of colorectal cancer substantially improves survival. However, over half of cases are diagnosed late due to the demand for colonoscopy—the ‘gold standard’ for screening—exceeding capacity. Colonoscopy is limited by the outdated design of conventional endoscopes, which are associated with high complexity of use, cost and pain. Magnetic endoscopes are a promising alternative and overcome the drawbacks of pain and cost, but they struggle to reach the translational stage as magnetic manipulation is complex and unintuitive. In this work, we use machine vision to develop intelligent and autonomous control of a magnetic endoscope, enabling non-expert users to effectively perform magnetic colonoscopy in vivo. We combine the use of robotics, computer vision and advanced control to offer an intuitive and effective endoscopic system. Moreover, we define the characteristics required to achieve autonomy in robotic endoscopy. The paradigm described here can be adopted in a variety of applications where navigation in unstructured environments is required, such as catheters, pancreatic endoscopy, bronchoscopy and gastroscopy. This work brings alternative endoscopic technologies closer to the translational stage, increasing the availability of early-stage cancer treatments.

1. Introduction 介绍

FE的特定设计限制包括（1）设备的内在复杂度，这阻止一个单次使用的方法，并且需要清洁和消毒，（2）病人疼痛由于组织拉伸当内窥镜被推进穿过结肠，限制社会接受性，并带来风险，比如组织穿孔和麻醉相关的不良事件，（3）缺少直观性，需要高度受训的人员，因而需要一个长期且昂贵的训练过程，内镜医师相对于需求的短缺。
Specific design limitations of the FE include (1) the inherent complexity of the device, which prevents a single-use approach and thus necessitates cleaning and sterilization, (2) patient pain due to tissue stretching as the endoscope is pushed through the colon, limiting social acceptance and introducing risks such as tissue perforation and anaesthesia-related adverse events, and (3) lack of intuitiveness, requiring highly trained personnel and thus a long and expensive training process and a shortage of endoscopists with respect to demand.

在这个工作中，关于我们磁内窥镜的讨论和自动化控制的开发将支持对如何为机器人内窥镜指定通用定义以及达到每个自治级别所需的功能的分析。对于机器智能领域的我们的贡献是去探索计算机辅助的不同等级如何可能改进机器热结肠镜的流程和减少用户工作量的能力。
In this work, the discussion about our magnetic endoscope and the development of autonomous control will support an analysis of how general definitions can be specified for robotic endoscopy as well as the features required to reach each autonomy level. Our contribution to the field of machine intelligence is the ability to explore how different levels of computer assistance may improve the procedure and reduce user workload in robotic colonoscopy.

在这篇文章中我们探讨的主要的科学问题是，（1）智能控制策略如何克服控制磁体内内窥镜的内在复杂性？（2）需要什么级别的自动化来使能一个非专家的操作者来导航一个磁内窥镜在一个非结构的环境中，比如结肠，同时保持与FE有可比性的流程持续时间？（3）有效的，智能的控制策略能够减少操作者的身体和心理负担吗？
The main scientific questions we investigate in this work are as follows. (1) How can intelligent control strategies overcome the inherent complexities of controlling magnetic intracorporeal endoscopes? (2) What level of autonomy is required to enable a non-expert operator to navigate a magnetic endoscope in an unstructured environment such as the colon, while maintaining procedure duration comparable to an FE? (3) Can effective, intelligent control strategies reduce the physical and mental burden of the operator?

在这篇文章中，我们展示了一个用于内窥镜自动化导航的全面的方法。
In this Article, we present a comprehensive approach to autonomous navigation of the endoscope.

为了探究这些科学问题，我们开发了一个控制理论，允许简化的用户输入和基于图像的自动化导航，能够基于对环境的实时视觉分析来计算运动。这理论被比较测试在桌面仿体和腔内装置中，使用非专家使用者。做这些事时，我们提供下面这些贡献：
（1）智能和自动化控制的演示使得非专家使用者能够成功地使用磁结肠镜通过腔内一段很长的距离，并且有与标准FE可比性的持续时间。
（2）定义医疗机器人自动化等级的框架被应用到机器人FE中
（3）被需要用来克服在非结构管状空洞中的磁操纵复杂度的自动化特征的分析
（4）用于磁内窥镜的智能和自动化控制策略的开发，这已经使得降低用户的负担。
To investigate these scientific questions, we developed a control methodology that allows simplified user inputs and image-based, autonomous navigation, capable of computing motion based on a real-time visual analysis of the environment. This methodology was tested comparatively in benchtop and in vivo (porcine
model) settings with non-expert users. In doing so, we provide the following contributions:
(1) The demonstration of intelligent and autonomous control enabling non-expert users to successfully perform magnetic colonoscopy by travelling a considerable distance in vivo, and with a duration comparable to standard FE
(2) A framework to define the increasing levels of autonomy in medical robotics applied to robotic flexible endoscopy
(3) An analysis of the autonomous features required to overcome the complexities of magnetic manipulation in unstructured tubular cavities
(4) The development of intelligent and autonomous control strategies for magnetic endoscopy, which have enabled a reduction in exertion for the user

每个层提供一组特征通过增加自动化度表征，依赖于底层提供的功能。
Each layer provides a set of features characterized by increasing autonomy, relying on functionalities offered by the underlying layers.

第一层，最简单的一层，直接机器人操纵。

所提供的功能与level 0有关，因为操纵器更是一个移动的驱动器，移动由人工操作员授予。
The functionality offered can be associated to level 0, as the manipulator is a mere executor of the movements imparted by the human operator.

第二层，用户输入是直接聚焦于导航内窥镜穿过结肠，同时系统承受产生合适的磁控动作的负担来完成期望的内窥镜动作。
In the second layer, user inputs are directly focused on navigating the endoscope through the colon, while the system carries the burden of generating a suitable magnetic control action to accomplish the desired endoscope motion.

用户直观地指导他们期望内窥镜相机如何在结肠内移动。
the user intuitively instructs how they wish the endoscope camera to move inside the colon.

我们把这一层定义为智能内窥镜远程操纵，这与level 1有关。
We define this layer as ‘intelligent endoscope teleoperation’, which can be associated to level 1 or robotic assistance.

第三层，运动的方向由图像分析算法计算，能识别内腔的中心。内窥镜被自动导向和前进穿过结肠。

为了高亮自动化程度，我们定义这一层为“半自动化导航”，这与level 3有关。
To highlight the autonomous features, we define this layer as ‘semi-autonomous navigation’. This layer can be associated to level 3 or conditional autonomy, where the system generates task strategies and relies on the operator to approve or override the choice.

在自动化程度的讨论中，level 2被忽略了。这层，定义为“任务自动化”，描述系统能执行半自动化的运动，但是依赖于人在环中标出结束目标和运动的路途点。level 2的例子有沿预定义轨迹的运动，自动翻转，和内窥镜末端的稳定化。
In the discussion of levels of autonomy, level 2 has been omitted. This level, defined as ‘task autonomy’, describes a system that carries out semi-autonomous motion but is dependent on a human-in-the-loop to indicate the end target and waypoints of that motion. Examples of level 2 in the context of endoscopy are motion along predefined trajectories, autonomous retroflexion and stabilization of the endoscope’s tip during interventional tasks.

因为结肠的形状不固定并且频繁改变，预定义轨迹的路途点和结束目标在level 2控制下也需要持续地被用户更新。
As the shape of the colon is not fixed and changes frequently, waypoints and end targets of predefined trajectories under level-2 control would need to be constantly updated by the user.

2. Benchtop and in vivo results 桌面仿体和体内实验结果

2.1 Benchtop experimental results

直接控制法，花费的时间是三种控制策略里最多的，而且存在盘绕轨迹。智能远程控制和半自主控制花费的时间远远少于直接控制。而且这三种控制方法所花费的时间都远远低于一个非专业结肠镜操作员完成结肠镜检查所花费的时间。

盘绕轨迹通常因为用户指位MFE到一个期望的位置上，卡住了，然后不得不通过缆线拉回端头，重新调整MFE的位置并重试。

2.2 In vivo experimental results

虽然比标准结肠镜检查花的时间多，但是半自主控制极大解放了人员的负担，而且只需要非专业人员就可以操作了。

标准结肠镜，虽然难以操控，但是操纵接口和端头之间存在物理连接，操作者可以预估反应。
缺少智能控制，缺少物理连接，导致直接控制法失败。
而智能远程控制和半自动化导航最终对新手用户有更少的要求。

3. Intelligent control and autonomous navigation 智能控制和自主化导航

3.1 磁控

基于之前那些文章

3.2 自主化导航

为了自动在结肠内导航，我们利用磁操纵算法和图像处理的组合来自动检测结肠的方向。
（1）自动转MFE相机帧对着结肠内腔的中心。
（2）自动推进MFE前进穿过结肠，一旦对准内腔时。
To autonomously navigate through the colon, we leverage a combination of the magnetic manipulation algorithms defined in the previous sections and image processing to autonomously detect the direction of the colon.
With this directional information we can (1) autonomously steer the MFE camera frame towards the centre of the colon lumen and (2) autonomously advance the MFE forwards through the colon, once aligned to the lumen.

我们选择的用来指明结肠内方向的方法不是基于结肠的特定特征，比如腹褶皱。相反地，这基于一个能可能适应到许多管状空洞的方法。
Our chosen approach to inferring direction in the colon is not based on specific features of the colon such as haustral folds. Conversely, it is based on an approach that could possibly be adapted to a variety of tubular cavities.

图像首先被分割来移除所有除了最黑和最不同的区域，由于假设这些区域最可能包含远端内腔。这分割被执行用RGB图像的红色通道，因为这通道扩大了亮的和暗色区域的不同。
The image is first segmented to remove all but the darkest and most distinct region, with the assumption that this area most likely contains the distal lumen. This segmentation is performed using the red channel of the RGB image, as this channel amplifies the distinction between bright and dark regions in the predominantly red-shaded colon

为了推内窥镜前进，我们假设相机应该在任何前进运动前指向内腔。
To advance the endoscope in the colon, we assume that the camera should be directed towards the lumen before any forward motion is requested.

内腔中心与图像中心之间的像素差通过一个两个比例增益后作为内窥镜朝向旋转指令和线速度指令。

为了防止胶囊正好面对肠壁，从而得出错误的内腔方向，利用FAST算法一直监视内镜图片，如果特征点数量很少就可以判断为面对肠壁，而特征点很多就是面对内腔空洞方向。

[1]: Martin, James W., et al. “Enabling the future of colonoscopy with intelligent and autonomous magnetic manipulation.” Nature Machine Intelligence (2020): 1-12.