shadow
This article introduces the practical experience of the author Alessandro Alviani, which is of great reference value.
Instruction redundancy and multi-step guidance are currently one of the methods to improve the output quality of LLM.
Towards Accurate Quote-Aware Summarization of News using Generative AI
Alessandro
Alviani
generative-ai-newsroom.com/towards-accurate-quote-aware-summarization-of-news-using-generative-ai-b786493e0c3d
Attribution is a fundamental principle of journalism. Correctly citing news sources without distorting the meaning of what is being expressed, or worse, adding information the reporter infers, is an essential skill for any journalist.
Large language models (LLMs) pose new challenges: they may fabricate citations or misassign accurate citations to wrong sources. This is because they work by predicting the next most likely word in a sequence based on previous text. Mistakes like this can erode trust in the media and should be avoided.
Our goal
At IPPEN.MEDIA, we have been experimenting with a large number of use cases for large language models LLM. Some of these include suggesting title and lead variations, and summarizing or rewriting articles to target different audiences. When dealing with citations, it is easy to make mistakes when generating abstracts or text variants.
In our first round of testing, we found that ChatGPT tends to rewrite references even when explicitly instructed not to. When we tried to summarize an article by adding specific constraints in the prompt while keeping all citations unchanged, ChatGPT just ignored these constraints and rewrote the citations. Worse, while prompts sometimes work as expected and all citations are reproduced correctly, most of the time they don't.
Part of this inconsistency is inherent in LLMs, which are probabilistic rather than deterministic models.
Even if we double-check all texts edited with ChatGPT and other LLMs, fictional texts may still pass the scrutiny of human editors.
our way
It turns out that one of the fundamental concepts of prompt engineering: constructing prompts that are as specific and clear as possible to define the desired output - may not be enough. Our approach requires a multi-step approach combined with instruction redundancy.
The abstract and article directives we originally added to the ChatGPT prompt failed in two ways: the original quote was either rewritten and quoted, or it was interpreted.
We try to break the initial prompt into two steps. We also provide more context by using system prompts to assign the model the role of an experienced news editor . Also, the model usually ends up explaining the original citation.
Timely iteration is key
A little trick, we set the temperature parameter to 0 to reduce the variation of the output.
We used a step-by-step approach again, but this time we did it differently: we instructed the model to first extract all quotes using the format "" (i.e. find anything between quotes), and then generate summaries or New text version.
It turned out to be much better. However, even if it effectively extracts all quotes from step 1, the model may still stray from the hint and use/rewrite quotes incorrectly.
Worse, for longer articles with multiple citations, the model can make two common mistakes: either failing to extract all citations, or misidentifying non-citation sentences as citations that appear next to or in the middle of actual citations. In general, the longer the text, the lower the number of quotes that will be recognized. This is especially true for the GPT-3.5 model.
The real game-changers in this iteration are the next two adjustments. First, add a simple system prompt . The results improved significantly, supporting the notion that providing the LLM with more background knowledge could improve its performance.
The second major improvement comes from using GPT-4. Our tests show that OpenAI's latest model outperforms GPT-3.5. Almost all were correctly identified using our two-step approach.
In summary, GPT-4 is far superior to GPT-3.5. In 11 of the 12 articles, all citations were correctly included in the AI-generated summaries.
Instruction redundancy works well. In our two-step approach, we asked GPT-3.5 and GPT-4 to rewrite or summarize the text in the second step, not only pasting all the citations extracted in step 1, but again pasting the original article at the end of the prompt . Even if GPT-3.5 and GPT-4 fail to provide correct conclusions in the first step, they are usually able to output correctly in the second step.
Remarks: Reminder to the engineering community
opus
The prompt project shared by the author is for your reference. Also welcome to join Mix's prompt engineering community~~
Hint works:
[1] The text contains quotations; they are enclosed in quotation marks. Quotations must remain as in the original.
[1] The text contains quotations; they are enclosed in quotation marks. Citations must remain as they are.
[2] You are an editor with 30 years of experience. You need to rewrite the following article into a new text. Think step by step.
Step 1: Rewrite the following original title using vivid but neutral language; Step 2: Make sure that all quotations within quotation marks are reproduced in the new content in the same way. Nothing in quotation marks may be rewritten.
[2] You are an editor with 30 years of experience. You need to rewrite the following article into new text. Think step by step.
Step 1: Rewrite the original headline below in vivid but neutral language; Step 2: Make sure all quotes within quotes are reproduced in the same way in the new content. Anything in quotes cannot be overridden.
[3] 1st step:
Extract all quotes between quotation marks such as “” in the following text:
###Text##
2nd step (after the model has extracted the quotes):
Rewrite the article and make sure the following quotes remain unchanged:
“quote”
“quote”
“quote”
Article: ###Text###
[3] Step 1:
Extract all quotes between quotes in the following text, such as "":
###Text##
Step 2 (after the model extracts quotes):
Rewrite the article and make sure the following quotes stay the same:
"quote"
"quote"
"quote"
article: ###Text###
[4] You are a precise journalist and editor.
[4] You are a serious reporter and editor.