pyrouge在Ubuntu下的使用

pip install pyrouge

如报错：

Cannot open exception db file for reading: data/WordNet-2.0.exc.db

则执行以下命令：

cd /home/yzhao/soft/ROUGE-1.5.5/RELEASE-1.5.5/data
rm WordNet-2.0.exc.db
./WordNet-2.0-Exceptions/buildExeptionDB.pl ./WordNet-2.0-Exceptions ./smart_common_words.txt ./WordNet-2.0.exc.db

from pyrouge import Rouge155

r = Rouge155('/home/yzhao/soft/ROUGE-1.5.5/RELEASE-1.5.5')
r.system_dir = '/mnt/hgfs/NLP/document summarization/MyProject/docs/system'
r.model_dir = '/mnt/hgfs/NLP/document summarization/MyProject/docs/gold'
r.system_filename_pattern = 'system.(\d+).txt'
r.model_filename_pattern = 'gold.[A-Z].#ID#.txt'

output = r.convert_and_evaluate()
print(output)
output_dict = r.output_to_dict(output)

在system文件夹下，放置自己生成的摘要，在gold文件夹下放置参考的摘要。

gold文件夹下，文件名为：gold.A.0.txt。数字0代表某篇文章的ID号，A代表某个模型生成的结果。

system文件夹下，文件名为：system.0.txt。数字0代表某篇文章的ID号。

文件中都是一行一个句子！。

如果是中文摘要，可能会报错：

Illegal division by zero at ./ROUGE-1.5.5.pl line 2450.

可以建立全局字典，用字典的索引号代替自己摘要中的中文字符，以空格间隔。

另外，TXT文件中好像不允许出现'<'符号，例如'<unk>'，如果有可能会报错！

rouge的参数目前可以这样设置：

output = r.convert_and_evaluate(
        rouge_args='-e {}/data -a -2 -1 -c 95 -U -n 2 -w 1.2 -b 75'.format(rouge_path))

具体参数解释参见：ROUGE-1.5.5\RELEASE-1.5.5\README.txt

一个完整的例子：

from pyrouge import Rouge155
import shutil
import os

ROUGE_PATH = '/home/yzhao/soft/RELEASE-1.5.5'


def clear_dir(dpath):
    if os.path.exists(dpath):
        shutil.rmtree(dpath)
    os.mkdir(dpath)


def eval_rouge(results, golds):
    assert len(golds) == len(results)

    clear_dir('gold_summaries')
    clear_dir('result_summaries')

    r = Rouge155(ROUGE_PATH)
    r.system_dir = 'result_summaries'
    r.model_dir = 'gold_summaries'
    r.system_filename_pattern = 'result.(\d+).txt'
    r.model_filename_pattern = 'gold.[A-Z].#ID#.txt'

    for i in range(len(golds)):
        output_gold = open('gold_summaries/gold.A.%d.txt' % i, 'w')
        output_result = open('result_summaries/result.%d.txt' % i, 'w')

        if isinstance(golds[i], list):
            output_gold.write('\n'.join(golds[i]))  # 文件中一行一个句子！
        else:
            output_gold.write(golds[i])
        if isinstance(results[i], list):
            output_result.write('\n'.join(results[i]))
        else:
            output_result.write(results[i])

        output_gold.close()
        output_result.close()

    output = r.convert_and_evaluate(ROUGE_PATH)
    print(output)
    return r.output_to_dict(output)

下面是CNN/Daily Mail 数据集，测试集部分前三句话（lead 3）作为摘要的Rouge输出：

---------------------------------------------
/home/yzhao/soft/RELEASE-1.5.5 ROUGE-1 Average_R: 0.55091 (95%-conf.int. 0.54797 - 0.55401)
/home/yzhao/soft/RELEASE-1.5.5 ROUGE-1 Average_P: 0.33577 (95%-conf.int. 0.33329 - 0.33816)
/home/yzhao/soft/RELEASE-1.5.5 ROUGE-1 Average_F: 0.40273 (95%-conf.int. 0.40049 - 0.40505)
---------------------------------------------
/home/yzhao/soft/RELEASE-1.5.5 ROUGE-2 Average_R: 0.24293 (95%-conf.int. 0.24001 - 0.24585)
/home/yzhao/soft/RELEASE-1.5.5 ROUGE-2 Average_P: 0.14795 (95%-conf.int. 0.14593 - 0.14998)
/home/yzhao/soft/RELEASE-1.5.5 ROUGE-2 Average_F: 0.17738 (95%-conf.int. 0.17524 - 0.17959)
---------------------------------------------
/home/yzhao/soft/RELEASE-1.5.5 ROUGE-3 Average_R: 0.13791 (95%-conf.int. 0.13537 - 0.14055)
/home/yzhao/soft/RELEASE-1.5.5 ROUGE-3 Average_P: 0.08511 (95%-conf.int. 0.08334 - 0.08682)
/home/yzhao/soft/RELEASE-1.5.5 ROUGE-3 Average_F: 0.10140 (95%-conf.int. 0.09945 - 0.10336)
---------------------------------------------
/home/yzhao/soft/RELEASE-1.5.5 ROUGE-4 Average_R: 0.09013 (95%-conf.int. 0.08790 - 0.09242)
/home/yzhao/soft/RELEASE-1.5.5 ROUGE-4 Average_P: 0.05633 (95%-conf.int. 0.05472 - 0.05792)
/home/yzhao/soft/RELEASE-1.5.5 ROUGE-4 Average_F: 0.06666 (95%-conf.int. 0.06489 - 0.06843)
---------------------------------------------
/home/yzhao/soft/RELEASE-1.5.5 ROUGE-L Average_R: 0.49833 (95%-conf.int. 0.49550 - 0.50132)
/home/yzhao/soft/RELEASE-1.5.5 ROUGE-L Average_P: 0.30438 (95%-conf.int. 0.30205 - 0.30670)
/home/yzhao/soft/RELEASE-1.5.5 ROUGE-L Average_F: 0.36483 (95%-conf.int. 0.36254 - 0.36709)
---------------------------------------------
/home/yzhao/soft/RELEASE-1.5.5 ROUGE-W-1.2 Average_R: 0.21864 (95%-conf.int. 0.21717 - 0.22007)
/home/yzhao/soft/RELEASE-1.5.5 ROUGE-W-1.2 Average_P: 0.22222 (95%-conf.int. 0.22045 - 0.22402)
/home/yzhao/soft/RELEASE-1.5.5 ROUGE-W-1.2 Average_F: 0.21139 (95%-conf.int. 0.21012 - 0.21273)
---------------------------------------------
/home/yzhao/soft/RELEASE-1.5.5 ROUGE-S* Average_R: 0.26414 (95%-conf.int. 0.26140 - 0.26721)
/home/yzhao/soft/RELEASE-1.5.5 ROUGE-S* Average_P: 0.10291 (95%-conf.int. 0.10133 - 0.10447)
/home/yzhao/soft/RELEASE-1.5.5 ROUGE-S* Average_F: 0.13238 (95%-conf.int. 0.13078 - 0.13402)
---------------------------------------------
/home/yzhao/soft/RELEASE-1.5.5 ROUGE-SU* Average_R: 0.27598 (95%-conf.int. 0.27325 - 0.27906)
/home/yzhao/soft/RELEASE-1.5.5 ROUGE-SU* Average_P: 0.10853 (95%-conf.int. 0.10692 - 0.11011)
/home/yzhao/soft/RELEASE-1.5.5 ROUGE-SU* Average_F: 0.13958 (95%-conf.int. 0.13798 - 0.14124)

pyrouge在Ubuntu下的使用

猜你喜欢