python semantic text similarity 解决单词,词组,句子相关性问题

一.什么是文本相似度?

需要解决文本相似度必须确定两段文本的词汇相似(lexical similarity) 以及 语义相似(semantic similarity)问题
具体文献参考: Text Similarities : Estimate the degree of similarity between two texts

二.代码实现

# -*- coding: utf-8 -*-
# @Time    : 2021/04/15 09:01
# @Author  :

from semantic_text_similarity.models import WebBertSimilarity

# from semantic_text_similarity.models import ClinicalBertSimilarity

web_model = WebBertSimilarity(device='cpu', batch_size=10)  # defaults to GPU prediction

# clinical_model = ClinicalBertSimilarity(device='cuda', batch_size=10) #defaults to GPU prediction

# 单词,词组
web_model.predict([("furniture", "decoration design")])

# 句子
# print(web_model.predict([("She won an olympic gold medal","The women is an olympic champion")]))

参考:https://pypi.org/project/semantic-text-similarity/