An Exploratory Method for Deep Reinforcement Learning Based on Attentional Neural Networks: ARiADNE

ARiADNE:A Reinforcement learning approach using Attention-based Deep Networks for Exploration

参考论文：Cao Y, Hou T, Wang Y, et al. Ariadne: A reinforcement learning approach using attention-based deep networks for exploration[J]. arXiv preprint arXiv:2301.11575, 2023.
2023 IEEE International Conference on Robotics and Automation (ICRA 2023)

Autonomous Robotic Exploration (ARE)

ARE's Traditional Boundary Law

Autonomous robot exploration (ARE)

Goal: The goal of the ARE is to plan the shortest path to complete exploration where small noise/errors in the final map are tolerable

Mission Points: The mission point of the ARE is to plan a non-myopic path, to balance the trade-offs between exploring the environment (i.e. optimizing the map in already explored areas) and exploring new areas (usually more distant areas), and most importantly Only partially aware of the environment. Such exploration paths are usually planned online incrementally, as the local map is updated with new measurements along the way

Traditional boundary-based methods: Traditional boundary-based methods generate multiple candidate paths, each covering a boundary (i.e., the boundary between explored free regions and unexplored regions), and greedily select the path with the greatest payoff , usually defined as a combination of utility (i.e., the number of boundaries observable along a path) and cost (i.e., path length).

Limitations: This short-sighted choice of boundaries does not guarantee long-term optimality. Since the environment is only partially known, the previously optimal path tends to become suboptimal as more environments are exposed, or worse, leads to redundant moves (e.g., missed unexplored shortcut, and the two rooms were not previously known to be connected).

non-myopic path

Spatial non-myopia requires planners to reason about the current local map to balance the trade-offs between exploration and exploitation

Time is not myopic: Planners are asked to estimate the impact of current decisions on the future (e.g., predicting changes in the local map that may result from a given path planning decision).

The Positive Significance of Deep Reinforcement Learning Approaches

Discover local map dependencies:

  The ARE method based on deep reinforcement learning (DRL), under the support of the attention mechanism, allows the agent to reason about the dependencies of different regions in the local map at different spatial scales, thus allowing the agent to optimize long paths without efficiently sequence spatial non-myopic decisions.

Explore potential areas:

  The critic network implicitly provides the robot with the ability to estimate potential areas that may be discovered by learning state values, which further helps to make decisions that are beneficial to long-term efficiency, thus solving the problem of temporary non-myopia.

Method example:

  Autonomous exploration is first formulated as a sequential decision problem on a collision-free graph covering known traversable regions, where a node is the robot's current position. Then, an attention-based neural network is used to select a neighboring node of the robot's current location as the next viewpoint of the robot.

problem definition

Environment $\mathcal{E}$，由$x\times$2D occupancy grid map representation of$y$

Grid map $P$ , the unknown area is$P_u$, the known area is $P_k$, the free area is $P_f$, obstacle area $P_o$,There is also a proposition:
$$\begin{gathered} P_f\in P_k \\ {P}_o\in P_k \\ {P}_u\cup P_k=P \\ {P}_f\cup P_o=P_k \end{gathered}$$

The initial state map is completely unknown $P=P_u$

The exploration process is measured by the sensor (measuring range is $d_s$) Classify unknown regions into free and occupied regions

  The robot looks for the shortest collision-free trajectory $p^{\ast }$
$$p^{\ast }=\underset{\psi \in \Psi }{\mathrm{argmin}}C\left(\psi \right),\text{ s.t. }{P}_k=P_g$$
  for $C$ function,$p\to {\mathbb{R}}^{+}$Map the trajectory into its length and groud-truth$P_g$。

  Although the ground truth is not available in real-world deployments, it is known and can be used to evaluate the performance of the planner under test. In practice, most works treat the closed set of occupied regions as $P_k=P_g$

method

Mapping Modeling for Reinforcement Learning Problems

Problem definition:

  Since the update view of the free area is based on the movement of the robot, the planning of the online ARE is actually a sequential decision-making problem. According to the previous work [20] for information path planning, we set the robot trajectory ψ $\psi$is regarded as the sequence$p=(p_0,p_1，...),p_i\in P_f$

Sequential decision problems:

  In each decision step t, first in the current free area $P_f$Evenly distributed candidate viewpoints V t = { v 0 , v 1 , … } , ∀ vi = ( xi , yi ) ∈ P f Vt=\{v_0,v_1,\ldots\},∀v_i=(x_i, y_i) ∈ P_fVt={ v0​,v1​,…},∀vi​=(xi​,yi​)∈Pf​. Then, to find collision-free paths between viewpoints, each viewpoint is connected to its k-nearest neighbors by a straight line, and edges colliding with occupied or unknown regions are removed. In the process, construct a collision-free graph $G_t=(V_t,E_t)$ , where$V_t$is a set of nodes uniformly distributed on the free area (ie viewpoint), but $E_t$A set of traversable edges. Finally let the robot choose an adjacent node of its current position as the next viewpoint. Since the decision will be made upon reaching the last selected viewpoint, the trajectory is a sequence of waypoints such that $p_i\in V$。

Observation:

  The observed value of the agent $o_t=(G_t^{\prime },p_t)$，$G_t^{\prime }=(V_t^{\prime },E_t)$ is the collision-free graph$G_t$Augmented graph of , where $p_t$is the current position of the robot, $G_t^{\prime }$with $G_t$Shared edge set $E_t$, augmented graph node $v_i^{\prime }$not including $v_i=(x_i,y_i)$ attributes, but also contains the binary$b_i$attribute, indicating whether the node has been visited by the agent, and then with the utility $u_i$ Since $v_i^{\prime }=(x_i,y_i,u_i,b_i)$。

  $b_i$Effect of : The authors demonstrate experimentally that adding a binary access bit leads to better perception of previous movement

  $u_i$role: utility $u_i$Represents the node $v_i$The number of observable boundaries, the observable boundary is the boundary between the node and the collision-free boundary within the range of the sensor, the $u_i$定义为
$$\begin{array}{c}{u}_i=\left|F_{o,}\right|,\forall f_j\in F_{o,},\Vert f_j-v_i\Vert \le d_s,L\left(v_i,f_j\right)\cap \left(P-P_f\right)=\varnothing \end{array}$$
  $F_{o,}$means $v_i$The observable boundary set of $d_s$Indicates the range of the sensor, $L(v_i,f_j)$ means node$v_i$to boundary $f_j$, normalize the node coordinate system and utility before inputting into the neural network

action:

  At each decision step $t$, the observation value ot o_tof the given agent$o_t$, the attention-based neural network outputs a stochastic policy to select a node from all neighboring nodes as the next viewpoint to visit. It is defined as:
$${Pi}_{}\left(a_t\mid o_t\right)={Pi}_{}\left(p_{t+1}=v_i,\left(p_t,v_i\right)\in E_t\mid o_t\right)$$
  $\theta$ is the weight of the neural network, and the robot moves in a straight line to the next viewpoint, updating parts of the map based on data collected along the way.

award:

$r_o=|F_{o,p_{t+1}}|$ , the number of boundaries observed at the new viewpoint

$r_c=-\mathrm{C}\left(p_t,p_{t+1}\right)$ , the distance penalty between the previous viewpoint and the new viewpoint

$r_f=\{\begin{array}{ll}20,& P_k=P_g\\ 0,& \text{ otherwise }\end{array}$, after an episoid, the exploration task is completed and rewarded

总奖励$r_t\left(o_t,a_t\right)=a\cdot r_o+b\cdot r_c+r_f$，$a$ and$b$is a scalar parameter (reference value$a=1/50$,$b=1/64$）

policy network

  Attention-based neural network output strategy composed of encoder and decoder $p_{}$, the encoder extracts salient features from part of the map, especially $G^{\text{'}}$ Dependency relationship between nodes; based on these salient features and the current position of the robot, the decoder outputs a strategy for determining the next access point on adjacent nodes.

Use of Pointer Network

  Policy-based RL agents usually have a fixed action space, and the decoder is inspired by pointer networks, allowing the action space to depend on the number of neighboring nodes input in the network. This allows the network to adapt naturally to collision-free graphs, where nodes have an arbitrary number of neighbors.

Attention Layer

  Using the attention module as the basic building block, the input of the attention layer includes a query vector $h^q$and key-value vector$h^{k,v}$ , the output of the attention layer is$h_i^{\prime }$, which is a weighted sum of value vectors, with weighting factors depending on the similarity between key and value

Encoder

Mask MM   for edge sets$M$的元素由$m_{ij}=\begin{array}{l}\{\begin{array}{l}0,\left(v_i,v_j\right)\in E_t\\ 1,\left(v_i,v_j\right)\notin E_t\end{array}\end{array}$Calculation, the characteristics of the node are then passed to the multi-layer attention layer (actually 6 layers are taken, among them) $h^q=h^{k,v}=h^n$ , the input of each layer is the output of the previous layer, by setting$w_{ij}=0,\forall (i,j),m_{ij}=1$ The edge set mask is used to allow each node to access the features of adjacent nodes. Although the attention of each layer is limited to adjacent nodes, through this stacked attention structure, nodes can still be aggregated through multiple Node features to obtain features of non-adjacent nodes. We empirically find that this structure is better than graph transformers for learning pathfinding in maps with cluttered obstacles. We refer to the output of the encoder as the enhanced node feature${\hat{h}}^e$ , because each updated node feature${\hat{h}}_i^n$both contain and $v_i^{\prime }$Dependencies on other nodes.

Decoder

  The decoder is used to output the enhanced node features ${\hat{h}}^e$ and the current position of the robot$p_t$。

  The current position of the robot is expressed as a node $v_c=p_t$, first select the current node feature $h^c$ and neighbor feature$h^{nb}$，其中$\forall {\hat{h}}_i^{nb},\left(v_c,v_i\right)\in E_t$Correspondingly, it is to enhance the node features, and pass the current node features and enhanced node features into the attention layer $h^q=h^{k,v}=h^n$ , output it and$h^c$ concatenate and project it into a d-dimensional vector, and name this vector the enhanced current node vector${\hat{h}}^c$, after that, the enhanced current node features and neighbor node features are passed to the pointer layer, and the attention layer directly outputs the attention weight$w$和$h^q={\hat{h}}^c,h^{k,v}=h^{nb}$as output. Finally, the output of the pointer layer is used as the strategy of the robot (ie${Pi}_{}(a_t|o_t)=w_i$）

Policy Network Training Method

  soft actor critic(SAC)

judging network

Training purpose: Predict the value of state-action

State-action value: Since the value of the state-action value is an approximation of the long-term return, it has an implicit prediction of the potential return (that is, the potential area may be discovered). This property can help the robot sequence to make non-myopic decision-making . Actually train a judging network to approximate the soft update state-action value $Q_{}\left(o_t,a_t\right)$ ϕ$\varphi$ represents the weight set of the evaluation network.

Network structure: The network structure of the judgment network is almost the same as that of the policy network, except that there is no pointer layer at the end. The last is to directly concatenate the enhanced current node features and adjacent features, and project them into state-action values

train

Soft Actor-critic

  The goal of SAC is to learn a policy that maximizes reward while making its entropy as high as possible:
$${Pi}^{\ast }=\mathrm{argmax}{\mathbb{E}}_{\left(o_t,a_t\right)}\left[\sum _{t=0}^T{c}^t\left(r_t+aH\left(p\left(.\mid o_t\right)\right)\right)\right]$$
  ${Pi}^{\ast }$ is an optimization strategy,$T$is the number of decision-making steps,$\gamma$is the discount coefficient,$\alpha$is an important parameter to adjust the cross-entropy for the return.

软状态价值
$$V(o_t)={\mathbb{E}}_{a_t}\left[Q\left(o_t,a_t\right)\right]-a\log \left(p\left(a_t\mid o_t\right)\right)$$
critic损失函数
$$J_Q\left(\varphi \right)=E_{o_t}[\frac{1}{2}(Q_{}\left(o_t,a_t\right)-{\left(r_t+c{E}_{o_{t+1}}\left[V\left(o_{t+1}\right)\right]\right))}^2]$$
policy损失函数
$$J_{}\left(i\right)={\mathbb{E}}_{\left(o_t,a_t\right)}\left[a\log \left(p_{}\left(a_t\mid o_t\right)\right)-Q_{}\left(o_t,a_t\right)\right]$$
Self-adjusting$\alpha$损失
$$J\left(a\right)={\mathbb{E}}_{a_t}\left[-a\left(\log {Pi}_t\left(a_t\mid o_t\right)+\overline{\mathcal{H}}\right)\right]$$
  $H$denotes the target entropy. In practice, use a dual-target critic network for training

training details

basic work

  Training was performed using the same environments provided in references, generated by a random dungeon generator. Each environment is a 640 × 480 grid map, and the sensor range $d_s=80$

F. Chen, S. Bai, T. Shan, and B. Englot, “Self-learning exploration
and mapping for mobile robots via deep reinforcement learning,” in
Aiaa scitech 2019 forum, 2019, p. 0396.

Collision-free graph construction process

  To construct the collision-free map, 900 points are uniformly distributed to cover the entire environment, and all points within the known free area are considered as candidate viewpoints $V.$ _ We check the k = 20 nearest neighbors of each viewpoint, and if such an edge is collision-free, connect them to form the edge set$E$。

Task Completion Judgment Criteria

  We consider the exploration task complete once more than 99% of the ground-truth is detected (|Pk|/|Pg| > 0.99).

parameter

  During training, the maximum set length is set to 128 decision steps, the discount factor is set to γ ​​= 1, the batch size is set to 256, and the episoid buffer size is set to 10,000.
  Training starts after the episoid buffer has collected more than 2000 steps of data. The target entropy is set to $0.01\cdot log\left(k\right)$ . Each training step consists of 1 iteration and is performed after 1 episoid is completed. For the policy network and the critic network, we use a learning rate of$10^{-5}$Adam optimizer. The target critic network is updated every 256 training steps.

training time

  Models are trained on workstations equipped with i9-10980XE CPU and NVIDIA GeForce RTX 3090 GPU. We train our models using Ray, a distributed framework for machine learning, to parallelize and speed up data collection (32 instances in practice). Training takes around 24 hours to complete.

Code reference https://github.com/marmotlab/ARiADNE