【Paper Notes】Knowledge-Driven Encode, Retrieve, Paraphrase for Medical Image Report Generation (AAAI 2019)

Original paper: https://arxiv.org/pdf/1903.10122.pdf

Abstract

• Knowledge-driven Encode, Retrieve, Paraphrase (KERP) approach Knowledge-driven Encode, Retrieve, Paraphrase (KERP) approach
• decomposes medical report generation into explicit medical abnormality graph learning and subsequent natural language modeling

visual features --(Encode Module)–> an abnormality graph --(Retireve module)–> sequences of templates --(Paraphrase module)–> sequences of words

• generates structured and robust reports supported with accurate abnormality prediction
• produces explainable attentive regions which is crucial for interpretative diagnosis

 

GTR

• core of KERP

• dynamically transforms high-level semantics between graph-structured data of multiple domains such as knowledge graphs, images and sequences

GTR as a module

1. concatenating intra-graph message passing and inter- graph message passing into one step
   1. conduct message passing within target graph message passing within the target graph
   2. conduct message passing from one / multiple source graph pass message from one / multiple source graph
2. stacking multiple such steps into one module stacking multiple such steps into one module
   • convert target graph features into high-level semantics Convert target graph features into high-level semantics

GTR as a multiple domains

• ${\text{GTR}}_{\text{i2g}}$: image features --> graph’s features
• ${\text{GTR}}_{\text{g2s}}$: input - graph; output - sequence
• ${\text{GTR}}_{\text{g2g}}$: a graph --> another graph
  • abnormality graph --> disease graph
• ${\text{GTR}}_{}$: input - graph&sequence; output - sequence

GTR for sequential input/output

positional encoding – relative and absolute position information positional encoding – relative and absolute position information

 

KERP

Encode module

• transforms visual features into a structured abnormality graph by incorporating prior medical knowledge
• each node represents a possible clinical abnormality

updated node features:
$$\begin{gathered} {\text{h}}_u=GT{R}_{i2g}(\text{X}) \\ u=sigmoid({\text{W}}_u{\text{h}}_u) \end{gathered}$$

• ${\text{W}}_u$: linear projection to transform latent feature u into 1-d probability linear projection transforms latent feature u into one-dimensional probability
• ${\text{h}}_u=({\text{h}}_{u_1};{\text{h}}_{u_2};...;{\text{h}}_{u_N})\in R^{N,d}$ : the set of latent features of nodeswheredisfeature dimension
• u = ( u 1 , u 2 , . . . , u N ) , y i ∈ { 0 , 1 } , i ∈ { 1 , . . . , N } \text{u}=(u_1,u_2,...,u_N),y_i\in \{0,1\}, i\in\{1,...,N\} u=(u1​,u2​,...,uN​),yi​∈{ 0,1},i∈{ 1,...,N } :binary label for abnormality nodes Binary label for abnormality nodes

Retrieve module Retrieve

• retrieves text templates based on the detected abnormalities retrieves text templates based on the detected abnormalities

obtain template sequence:
$$\begin{gathered} {\text{h}}_t=GT{R}_{g2s}({\text{h}}_u) \\ t=\text{argmax}Softmax({\text{W}}_t{\text{h}}_t) \end{gathered}$$

• ${\text{W}}_t$: linear projection to transform latent feature to template embedding linear projection transforms latent features into template embeddings

Paraphrase module

• refine templates with enriched details and possibly new case-specific findings
  • by modifying information in the templates that is not accurate for specific cases
• convert templates into more natural and dynamic expressions convert templates into more natural and dynamic expressions
  • by robust language modeling for the same content by robust language modeling for the same content

$$\begin{gathered} {\text{h}}_w=GT{R}_{gs2}({\text{h}}_u,t) \\ R=\text{argmax}Softmax({\text{W}}_{w}f({\text{h}}_w)) \end{gathered}$$

• $f$: the operation of reshaping ${\text{h}}_w$ from $R^{N_s,N_w,d}$ to $R^{N_s\ast N_w,d}$
• ${\text{W}}_w$: linear projection to transform latent feature into word embedding linear projection transforms latent features into word embedding

Disease classification

multi-label disease classification: 多标记疾病分类
$$\begin{gathered} {\text{h}}_z=GT{R}_{g2g}({\text{h}}_u) \\ z=sigmoid({\text{W}}_z{\text{h}}_z) \end{gathered}$$
${\text{W}}_z$: linear projection to transform disease nodes feature into 1-d probability linear projection transforms disease node features into one-dimensional probability

Learning

During paraphrasing, the retrieved templates t, instead of latent feature ${\text{h}}_t$, is used for rewriting. Sampling the templates of maximum predicted probability breaks the connectivity of differentiable back-propagation of the whole encode retrieve-paraphrase pipeline.

1. train the Paraphrase with ground truth templates
2. then with **sampled templates **sampling template generated by Retrieval module

 

Results

Conclusion

• accurate attributes prediction
• dynamic medical knowledge graph
• explainable location reference explainable location reference