Intelligent magnetic manipulation for gastrointestinal ultrasound

本文来自机器人领域顶刊，Science Robotics。介绍了一种用于消化道的带尾绳的磁控超声机器人。在以前的磁控拖动/定位胶囊的基础上，增加了内超声功能，利用超声信号强弱判断成像质量好坏ESR，并调整探头姿态，获取最优图像质量，利用磁控对轨迹进行闭环追踪，并做了猪体内实验。

用于消化道超声的智能磁操纵
Intelligent magnetic manipulation for gastrointestinal ultrasound [1]
Paper Link
Authors: Norton, Joseph C., et al.
2019, Science Robotics

0. 摘要 Abstract

在消化道中的检查内镜在几十年中保持不变，并受限于组织表面的可视化，用于诊断的取药样本的收集，还有最小介入，比如剪切或组织移除。在这个工作中，我们展示一个磁胶囊机器人用于显微解剖下的原位的，表皮下诊断的自动伺服。我们调查和展示了使用数字化微超声反馈的闭环磁控制的可行性；这对于在一个未知的且无限制的环境中获取鲁棒的图像是至关重要的。我们使用微超声演示了自动伺服算法的功能，基于台式试验和猪模型的体内实验。我们已经证实了这个以自动线性探头运动为实例的磁微超声伺服，并能够以$1.0\pm 0.9mm$定位精度定位一个琼脂仿体中的标记，使用机器人定位和微超声图像信息的融合。这个工作演示了在小肠中闭环机器人微超声成像的可行性，而不需要在传感器和体外工具间的刚性物理连接或者复杂的手动操纵。
Diagnostic endoscopy in the gastrointestinal tract has remained largely unchanged for decades and is limited to the visualization of the tissue surface, the collection of biopsy samples for diagnoses, and minor interventions such as clipping or tissue removal. In this work, we present the autonomous servoing of a magnetic capsule robot for in situ, subsurface diagnostics of microanatomy. We investigated and showed the feasibility of closedloop magnetic control using digitized microultrasound ($\mu$US) feedback; this is crucial for obtaining robust imaging in an unknown and unconstrained environment. We demonstrated the functionality of an autonomous servoing algorithm that uses $\mu$US feedback, both on benchtop trials and in vivo in a porcine model. We have validated this magnetic $\mu$US servoing in instances of autonomous linear probe motion and were able to locate markers in an agar phantom with $1.0\pm 0.9mm$ position accuracy using a fusion of robot localization and $\mu$US image information. This work demonstrates the feasibility of closed-loop robotic $\mu$US imaging in the bowel without the need for either a rigid physical link between the transducer and extracorporeal tools or complex manual manipulation.

1. 介绍 Introduction

消化道诊断和管理通常使用光学柔性内窥镜实施。自推出以来，尽管存在许多缺点和很少的设计改进，但这仍是当前的黄金标准。
Diagnosis and management of GI diseases are typically performed using optical flexible endoscopy (FE). This is the current gold standard despite numerous drawbacks and very few design improvements since its introduction.

通过操纵近端来推进内窥镜（推，拉，拧）导致组织伸长，这引起病人不舒服，并且在极端情况下，可对肠子穿孔。柔性内窥镜的进一步限制是小肠的检查是有挑战性的和不定期进行的。这已经激发胶囊内窥镜的开发和使用。
Advancing (pushing, pulling, and twisting) the endoscope by manipulating the proximal end results in tissue stretching that causes patient discomfort and, in extreme cases, can perforate the bowel. A further limitation of FE is that inspection of the small bowel is challenging and is not routinely conducted. This has motivated the development and use of capsule endoscopy (CE).

在这些模块中，EUS是安全且低价的，并有能力产生高解析度多截面图像。
Of these modalities, EUS is safe and low cost and has the ability to generate higher-resolution cross-sectional images

在EUS技术方面最新的发展已经研究使用高频（大于20MHz）研究薄的肠层（小于3mm）。
Recent advancements in EUS technology have enabled the study of thin (< 3 mm) bowel layers by using high-frequency(> 20MHz).

直到最近，$\mu$US技术才被整合到可摄取的胶囊中，已显示可产生与GI组织组织学密切相关的US图像，这提供了在胃肠道所有区域进行EUS的潜力。
Only recently has $\mu$US technology been integrated into ingestible capsules that have been shown to generate US images that correlate closely with GI tissue histology, which offers the potential to perform EUS, and specifically $\mu$US, in all regions of the GI tract.

这从根本上限制了超声图像采集在部分消化道中，比如胃部和结肠，这些地方的腔道都极大于胶囊。在这些状况中，胶囊受到的环境限制较小，组织的声学耦合不能被保证。问题是由空气配置的，这对声学耦合是一种有效的屏障。一个机器人系统在消化道的所有位置使能鲁棒的有目标的成像，通过主动与环境互动来解决这个问题。更进一步的，知晓传感器的姿态能够允许病灶的尺寸和它们在消化道中的相对位置被识别。如果临床实现，这种提供针对性原位诊断的系统能减少检查时间，缩短病理周转时间并提高图像引导的诊断率，同时还可以减轻用户的身体和认知负担。
It also fundamentally limits US image acquisition in other areas of the GI tract, such as the stomach and colon, where the cavity is considerably larger than the capsule. In these situations, the capsule is less constrained by the environment, and acoustic coupling with the tissue cannot be guaranteed. The problem is compounded by the presence of air, which is an effective barrier to acoustic coupling. A robotic system that enables robust, targeted imaging in all locations in the GI tract by actively engaging with the environment could solve this. Furthermore, knowledge of the pose of the transducers could allow the sizes of lesions and their relative location in the GI tract to be identified. If implemented clinically, such a system offering targeted in situ diagnostics could reduce procedure time, improve pathology turnaround time, and enhance image-guided diagnostic yield, while also reducing physical and cognitive burden on the user.

1.机器人辅助的探头操纵，机器人运动相对于超声图像开环，机器人技术在远程操纵中应用。
2.使用超声图像视觉伺服用于工具操纵，超声用来作为追踪机器人工具。
3.利用超声图像反馈的视觉伺服用于超声传感器的操纵，超声场是主动的甚至是自动化地调整的。
所有的以前的在这个领域的工作关注外部协作机器人的驱动或者已经有一部分通过刚性连接被插入人体内的机器人。
robot-assisted probe manipulation, where robot motion is open loop with respect to US imaging (32–36) and robotics is used to aid in teleoperation;
visual servoing using US image feedback for the steering of a tool, where US is used as a method of tracking (visualizing) a robotic tool and aiding its navigation toward a target;
visual servoing using US image feedback for the steering of the US transducer, where the US field of view can be actively, and even autonomously, adjusted;
all previous work in this field has concerned the actuation of extracorporeal robots or robots that have been partially inserted into a human via rigid links

在这个工作中，我们调查了以带尾绳的机器人胶囊内窥镜形式的磁驱动超声探头的可行性。磁驱动允许作用在内协作设备上力和力矩的应用，并且不需要一个在外协作驱动单元和内协作机器人之间的刚性连接。磁操纵和超声互相补充，磁吸引本质上提供了与组织的亲密接触，这对于可依赖的声学耦合是至关重要的。超声传感器的信号也能被在机器人控制环中被使用来提供在内窥镜和环境之间的交互的实时反馈，补充现存的反馈模态。
In this work, we investigated the feasibility of magnetically actuating a $\mu$US probe in the form of a tethered robotic capsule endoscope (RCE). Magnetic actuation allows for the application of force and torque on the intracorporeal device and does not require a rigid-link connection between an extracorporeal actuation unit (i.e., permanent magnet or electromagnet) and an intracorporeal robot. Magnetic manipulation and $\mu$US complement each other, with magnetic attraction inherently providing intimate contact with the tissue, which is crucial for reliable acoustic coupling. The $\mu$US transducer signals can also be used in the robotic control loop to provide real-time feedback on the interaction between the RCE and its environment, supplementing the existing feedback modalities—vision, pose, and force—available for intelligent magnetic manipulation

这个技术引领了医疗机器人，超声，消化道内窥镜，有以下关键优点：1.有利于自动化控制的环境感知级别的提升；2.有能力实现精确的内窥镜，超声传感器，的运动，利用磁操纵；3.在手术中分析超声图像质量，根据确保有效的超声图像采集调整机器人控制；4.在整个消化道中使用这个技术的潜力。
This technology advances medical robotics, EUS, and GI endoscopy, with the following key benefits: (i) the addition of a level of environmental perception that can facilitate autonomous control (55), improving overall performance and usability; (ii) the ability to implement precise RCE, and thus $\mu$US transducer, motions with the use of magnetic manipulation; (iii) a method of analyzing $\mu$US image quality during operation and adjusting robotic control accordingly to ensure effective $\mu$US image acquisition; and (iv) the potential to use this technology in the entire GI tract

2. 内窥镜系统

声波被处理来开发一个回声信号等级，ESR。ESR是一种基于幅值的评估方式，能决定接收到声学型号的最大值。
The waveforms are processed to develop an echo signal rating, or ESR. The ESR is an amplitude-based metric that determines the maximum magnitude of the received acoustic signal.

3. 评估 Evaluation

我们进行了一系列的台式实验来调查这个想法的可行性。它包含了1.内窥镜控制改进，在硅树脂仿体中测试，来更好理解如何操纵内窥镜来获取超声图像和表征控制参数和图像质量之间的关系；2.用超声反馈伺服，这里的使用超声回声的闭环控制的可行性被探索；3.台式系统验证，其中，ESR方法在声学逼真的琼脂模型上进行了验证（在这些测试中，还演示了系统自动获取与空间相关的超声图像的能力的证明）；4.在活猪模型中的体内验证和RCE采集$\mu$US波形的演示。
We conducted a series of benchtop experiments to investigate the feasibility of this concept. It included (i) RCE control development, where tests were performed in a silicone phantom to better understand how to manipulate the RCE to acquire $\mu$US images and characterize the relationship between control parameters and image quality; (ii) servoing using US feedback, where the feasibility of closing the robotic control loop using US echoes was explored; (iii) benchtop system validation, where the ESR method was validated on an acoustically realistic agar phantom (the demonstration of the system’s ability to autonomously acquire spatially relevant US images was also demonstrated during these tests); and (iv) in vivo validation and demonstration of an RCE acquiring $\mu$US waveforms in a living porcine model.

4. 结果 Results

4.1 内窥镜控制开发 RCE control development

我们发现足够幅度的超声信号的成功采集对四个主要的指标敏感，都相对于基质：1.超声耦合介质；2.传感器-组织接触力；3.传感器倾斜；4.传感器翻滚。
我们使用硅树脂仿体进行了台式实验来表征敏感度，并验证了系统的能力来有效地操纵传感器。
We found that the successful acquisition of $\mu$US signals of sufficient amplitude by our RCE is sensitive to four main criteria—all with respect to the substrate: (i) the presence of an US coupling medium, (ii) transducer-tissue contact force, (iii) transducer tilt, and (iv) transducer roll.
We conducted benchtop experiments using a silicone phantom to characterize this sensitivity and validate the system’s ability to effectively manipulate the transducers (i.e., RCE).

更高的作用力增强超声信号
a higher force can strengthen US signals

强声学耦合存在于倾斜$\pm 3^{\circ }$范围内
strong acoustic coupling exists in a $\pm 3^{\circ }$ tilt range of the RCE

滚动在$\pm 10.5^{\circ }$范围内时，声学耦合是强的
acoustic coupling was strong in a roll range of about $\pm 10.5^{\circ }$

这结果表明接触力有利于声学耦合，倾斜和滚动决定了传感器对齐，这对于保证反射回声被成功接收是必要的。
These results show that contact force facilitates acoustic coupling (tight contact), and tilt and roll determine the transducer alignment, which is necessary to ensure that reflected echoes are successfully received.

由于连续分泌粘液，在体内实验的声学耦合比在硅树脂中更好。
acousticcouplingislikelyto be better in an in vivo setting compared with our silicone phantom, owing to the continuous secretion of mucus.

一个已知力不能一定引起一个标准的回声质量。
a known force does not necessarily induce a standard echo quality.

期望来自我们的内窥镜的最大压力是0.75bar。这是假设内窥镜倾斜45度时计算的，组织缩进1mm，我们当前系统的可获得的最大接触力是2.5N。
The maximum (worst-case) pressure expected from our RCE is 0.75 bar. This was calculated by assuming an RCE tilt angle of 45°, a tissue indentation of 1 mm, and a maximum contact force achievable by our current system of 2.5 N.

实验随后被进行来确定给定的基于标定的位姿命令和自动化逼近它们是否是足够的用于鲁棒的超声图像。我们使用表征结果来决定参数，这些参数应该被控制和有恰当的范围。测试也被进行来获取超声信号是否能被成功的获得当内窥镜在一个线性轨迹上运动时。
Experiments were subsequently performed to determine whether giving calibration-based pose commands and autonomously approaching them are sufficient for robust GI $\mu$US imaging. We used the characterization results to determine the parameters that should be controlled and appropriate ranges for them. Tests were also performed to assess whether $\mu$US signals could be successfully acquired while the RCE moves along a linear trajectory; these were conducted using both teleoperation and autonomous navigation.

即使内窥镜能到达期望配置和路径，可依赖的图像不是总是可获得。这主要归因于超声反馈的缺失，和声学耦合和传感器对齐的必要性。
Although the RCE was able to reach desired configurations and traverse paths, reliable imaging was not always achieved. This was attributed primarily to the lack of $\mu$US feedback and the necessity for both acoustic coupling and transducer alignment.

4.2 使用超声反馈伺服 servoing using US feedback

简单地操纵内窥镜到达一个配置或通过一个轨迹对于可信赖的超声图像来说是不够的。因此，ESR被使用来提供一个超声耦合的定量的实时测量，没有这个评估标准的话，控制环无法和超声闭环。
Simply manipulating the RCE to a configuration or through a trajectory is insufficient for reliable $\mu$US imaging. Therefore, the ESR was used to provide a quantitative real-time measurement of the acoustic coupling; without this metric, a control loop could not be closed with $\mu$US.

比较它和经验设置的阈值
Our approach focuses on the amplitude of the received signal and compares it with an empirically set threshold

我们开发了自动回声搜寻算法，它使用这个ESR来获得清晰的声学图像。
We developed an autonomous echo-finding algorithm that uses this ESR for attaining clear acoustic images.

回声展现了仿体厚度的不同
The change in the observed depth of the back wall is attributed to variability in the phantom thickness.

4.3 台式系统实验验证 Benchtop system validation

这些测试演示使用ESR理论自动找到回声的能力，并且结合RCE定位和操纵来完成一个自动线性轨迹，同时收集有清晰特征的图像。
These tests demonstrate the capability to find echoes autonomously using the ESR method and to combine RCE localization and manipulation to complete an autonomous linear trajectory while gathering images with clear features.

4.4 体内实验 In vivo validation

机器人系统在一个猪模型体内被评估，重要目的是1.成功在体内实验结合超声，机器人，磁技术；2.以超声图像采集表现比较远程控制和自动化探头操纵；3.评估实施有目的超声的可行性，用于活体组织中消化道检查。实验包含远程操纵和自动回声检测实验，目的是操纵内窥镜获得特别的体内的肠壁的超声图像。
The robotic system was evaluated in an in vivo porcine model, with the primary objectives of (i) successfully merging $\mu$US, robotic, and magnetic technologies in vivo; (ii) comparing teleoperated and autonomous probe manipulations in terms of $\mu$US image acquisition performance; and (iii) assessing the feasibility of performing targeted $\mu$US for GI screening in a living organism. The experiments included both teleoperated and autonomous echo detection experiments, where the goal was to manipulate the RCE to acquire distinct in vivo $\mu$US images of the bowel wall.

5. 讨论 Discussion

我们的主要贡献是：1.有消化道内目标超声成像能力的内窥镜的开发和体内评估；2.实时超声回声处理的实现来增强机器人环境意识；3.自动化控制策略的开发，使用磁场和超声反馈，用于更鲁棒的超声图像采集。
our main contributions are (i) the development and in vivo evaluation of an RCE capable of targeted $\mu US$ imaging of the GI tract, (ii) the implementation of real-time $\mu US$ echo processing to enhance the robot’s environmental awareness, and (iii) the development of an autonomous control strategy that uses both magnetic field and $\mu US$ feedback for more robust $\mu US$ image acquisition.

我们相信，我们理论的有效性可以被提高通过使用更高的超声数据采集率。
we believe that the efficacy of our methods can be improved with a higher rate of $\mu US$ data acquisition.

用更高的采集率，ESR的导数被监视，系统能够识别信号强度的连续变化，并允许合适的运动补偿。
With a higher acquisition rate, the gradient of the ESR could be monitored, and the system could detect the continuous change in signal strength and allow for appropriate motion compensation.

当前的磁驱动不能使内窥镜绕自己的中心轴旋转。作为一个简单的解决方案，我们选择从内窥镜几何中心，偏移内磁铁，然后介绍一个矫正的力矩；我们也扁平化内窥镜的顶部，来增加滚动阻力。这些特征导致内窥镜的一个特殊最小能量配置，在这个配置下，传感器与组织接触。最长久可行的，最完整的解决方案是将单个超声传感器置换为曲面超声阵列，并完整地避免了精确滚动控制的需求。
As a simple solution, we chose to offset the internal magnet from the geometric center of the RCE, thus introducing a corrective torque; we also flattened the top of the RCE to increase roll resistance. These features result in a unique minimum energy configuration of the RCE in which the transducers are in contact with tissue. The most viable long-term and complete solution to this is to replace the single-element $\mu US$ transducers with curved $\mu US$ arrays and completely avoid the need for precise roll control of the RCE body.

[1]: Norton, Joseph C., et al. “Intelligent magnetic manipulation for gastrointestinal ultrasound.” Science Robotics 4.31 (2019): eaav7725.