Using generative AI to enhance Amazon Cloud Technology’s intelligent document processing

Data classification, extraction, and analysis can be challenging for organizations dealing with large volumes of documents. Traditional document processing solutions are manual, expensive, error-prone, and difficult to scale. Leveraging  AI services like Amazon Textract  , Amazon Cloud Intelligent Document Processing (IDP) allows you to leverage industry-leading machine learning (ML) techniques to quickly and accurately process data from any scanned document or image. Generative artificial intelligence (Generative AI) complements Amazon Textract to further automate document processing workflows. Features such as standardizing key fields and summarizing input data support faster management of document process workflows while reducing the potential for errors.

The Amazon Cloud Technology Developer Community provides developers with global development technology resources. There are technical documents, development cases, technical columns, training videos, activities and competitions, etc. Help Chinese developers connect with the world's most cutting-edge technologies, ideas, and projects, and recommend outstanding Chinese developers or technologies to the global cloud community. If you haven't followed/collected it yet, please don't rush through it when you see it. Click here to make it your technical treasure trove!

Generative AI is driven by large ML models called base models (FMs). FMs are changing the way you can solve traditionally complex document processing workloads. In addition to existing capabilities, businesses need to aggregate specific categories of information from documents such as financial reports and bank statements. FMs make it easier to generate such insights from extracted data. To optimize the time spent on manual reviews and increase employee productivity, errors such as missing digits in a phone number, missing documents, or a missing house number in an address can be flagged in an automated way. In the current scenario, you need to invest resources to complete these tasks through manual review and complex scripts. This method is tedious and expensive. FMs can help complete these tasks faster, with fewer resources, and convert different input formats into standard templates that can be further processed. At Amazon Cloud Technologies, we offer services like  Amazon Bedrock  , the easiest way to build and scale generative AI applications using FMs. Amazon Bedrock is a fully managed service that delivers data from leading AI startups and Amazon FMs via API so you can find the model that best suits your requirements. We also offer  Amazon SageMaker JumpStart , which allows ML practitioners to choose from a wide range of open source FMs. ML practitioners can deploy FMs into dedicated  Amazon SageMaker  instances from a network-isolated environment and use SageMaker custom models for model training and deployment.

Ricoh  provides workplace solutions and digital transformation services designed to help clients manage and optimize the flow of business information. “We are adding generative AI to our IDP solution to help customers take advantage of new capabilities such as question answering, summarization and normalized output to get work done faster and more accurately,” said Ashok Shenoy, vice president of portfolio solutions development. Amazon Cloud Technologies allows us to leverage generative AI while keeping each customer's data separate and secure."

In this article, we will share how to use generative AI to enhance the IDP solution on Amazon Cloud Technology.

Improve IDP process

In this section, we review how traditional IDP processes can be enhanced by FMs and walk through example use cases of Amazon Textract with FMs.

Amazon IDP consists of three phases: classification, extraction, and enrichment. For more details on each stage, see Intelligent Document Processing with Amazon Cloud AI Services: Part 1 and Part 2 . During the classification phase, FMs can now classify documents without any additional training. This means that the model can classify documents even if it has not seen similar examples before. FMs in the extraction phase can standardize date fields and validate addresses and phone numbers while ensuring consistent formatting. FMs in the enrichment phase allow reasoning, logical reasoning, and summarization. When you use FMs at every IDP stage, your workflow will be smoother and performance will improve. The diagram below illustrates the IDP process with generative AI.

Extraction phase of the IDP process

When FMs cannot directly process documents in native formats (such as PDF, img, jpeg, and tiff) as input, a mechanism is needed to convert the document into text. In order to extract text from a document and then send it to FMs for further processing, you can use Amazon Textract. Using Amazon Textract, you can extract lines and words and pass them to downstream FMs. The following architecture uses Amazon Textract to accurately extract text from any type of document and then sends it to FMs for further processing.

Typically, documents contain structured and semi-structured information. Amazon Textract can be used to extract raw text and data from tables and forms. Tables and data relationships in forms play a vital role in automating business processes. Certain types of information may not be directly processed by FMs. Therefore, we can choose to store this information in downstream storage or send it to FMs. The image below is an example of how Amazon Textract extracts structured and semi-structured information from documents, as well as lines of text that need to be processed by FMs.

Summary using Amazon Cloud Technology serverless services to leverage FMs

The IDP process we demonstrated earlier can be seamlessly automated using Amazon Cloud Serverless Services. Large enterprises have highly unstructured documents. These documents may span industries ranging from Securities and Exchange Commission (SEC) filings in the banking industry to assurance filings in the health insurance industry. With the development of generative AI on Amazon's cloud technology, people in these industries are looking for ways to extract summaries from these documents in an automated and cost-effective manner. Serverless services provide a mechanism to quickly build IDP solutions.  Services such as  Amazon Lambda , Amazon Step Functions  , and  Amazon EventBridge can help build document processing workflows that integrate with FMs, as shown in the diagram below.

The sample application used in the above architecture is event driven . An event is defined as a recent change in state. For example, when an object is uploaded to  an Amazon Simple Storage Service  (Amazon S3) bucket, Amazon S3 emits an object creation event. This event notification from Amazon S3 can trigger a Lambda function or Step Functions workflow. This type of architecture is called event-driven architecture . In this article, our sample application uses an event-driven architecture to process a sample patient discharge summary and summarize the details of the document. The workflow is as follows:

1. Amazon S3 fires an object creation event when a document is uploaded to an S3 bucket.

2. The EventBridge default event bus propagates events to Step Functions according to EventBridge rules.

3. A state machine workflow starts processing documents, starting with Amazon Textract.

4. Lambda functions transform the analyzed data for use in the next step.

5. The state machine calls the SageMaker endpoint that hosts FM  , through direct Amazon SDK integration.

6. The digest S3 target bucket receives digest responses collected from FM.

We use the example application with  the flan-t5 Hugging face model to summarize the following example patient discharge summary using the Step Functions workflow.

The Step Functions workflow uses  the Amazon SDK integration to call the Amazon Textract  AnalyzeDocument  and the SageMaker runtime InvokeEndpoint  API, as shown in the figure below.

This workflow generates a summary JSON object that is stored in the target bucket. The JSON object looks like this:

{
  "summary": [
    "John Doe is a 35-year old male who has been experiencing stomach problems for two months. He has been taking antibiotics for the last two weeks, but has not been able to eat much. He has been experiencing a lot of abdominal pain, bloating, and fatigue. He has also noticed a change in his stool color, which is now darker. He has been taking antacids for the last two weeks, but they no longer help. He has been experiencing a lot of fatigue, and has been unable to work for the last two weeks. He has also been experiencing a lot of abdominal pain, bloating, and fatigue. He has been taking antacids for the last two weeks, but they no longer help. He has been experiencing a lot of abdominal pain, bloating, and fatigue. He has been taking antacids for the last two weeks, but they no longer help. He has been experiencing a lot of abdominal pain, bloating, and fatigue. He has been taking antacids for the last two weeks, but they no longer help. He has been experiencing a lot of abdominal pain, bloating, and fatigue. He has been taking antacids for the last two weeks, but they no longer help."
  ],
  "forms": [
    {
      "key": "Ph: ",
      "value": "(888)-(999)-(0000) "
    },
    {
      "key": "Fax: ",
      "value": "(888)-(999)-(1111) "
    },
    {
      "key": "Patient Name: ",
      "value": "John Doe "
    },
    {
      "key": "Patient ID: ",
      "value": "NARH-36640 "
    },
    {
      "key": "Gender: ",
      "value": "Male "
    },
    {
      "key": "Attending Physician: ",
      "value": "Mateo Jackson, PhD "
    },
    {
      "key": "Admit Date: ",
      "value": "07-Sep-2020 "
    },
    {
      "key": "Discharge Date: ",
      "value": "08-Sep-2020 "
    },
    {
      "key": "Discharge Disposition: ",
      "value": "Home with Support Services "
    },
    {
      "key": "Pre-existing / Developed Conditions Impacting Hospital Stay: ",
      "value": "35 yo M c/o stomach problems since 2 months. Patient reports epigastric abdominal pain non- radiating. Pain is described as gnawing and burning, intermittent lasting 1-2 hours, and gotten progressively worse. Antacids used to alleviate pain but not anymore; nothing exacerbates pain. Pain unrelated to daytime or to meals. Patient denies constipation or diarrhea. Patient denies blood in stool but have noticed them darker. Patient also reports nausea. Denies recent illness or fever. He also reports fatigue for 2 weeks and bloating after eating. ROS: Negative except for above findings Meds: Motrin once/week. Tums previously. PMHx: Back pain and muscle spasms. No Hx of surgery. NKDA. FHx: Uncle has a bleeding ulcer. Social Hx: Smokes since 15 yo, 1/2-1 PPD. No recent EtOH use. Denies illicit drug use. Works on high elevation construction. Fast food diet. Exercises 3-4 times/week but stopped 2 weeks ago. "
    },
    {
      "key": "Summary: ",
      "value": "some activity restrictions suggested, full course of antibiotics, check back with physican in case of relapse, strict diet "
    }
  ]
 }

Generating these summaries at scale using an IDP with a serverless implementation can provide organizations with meaningful, concise, and presentable data in an efficient manner. Step Functions is not limited to processing documents one at a time. Its distributed mapping feature can summarize large numbers of documents on a scheduled basis.

The sample application uses  the flan-t5 Hugging face model ; however, you can choose your own FM endpoint. Training and running the model is beyond the scope of the example application. Please follow the instructions in the GitHub repository to deploy the sample application. The above architecture provides guidance on how to use Step Functions to orchestrate IDP workflows. For detailed instructions on building applications using Amazon Cloud AI services and FMs, see the  IDP Generative AI Workshop .

Set up solution

Follow  the steps in the README  to set up the solution architecture (except SageMaker endpoint). After you have your own SageMaker endpoint, you can pass the endpoint name as a parameter to the template.

clean up

To save money, delete the resources deployed in the tutorial:

1. Follow   the steps in the Cleanup section of the README .

2. Delete anything in the S3 bucket from the Amazon S3 console, then delete the bucket.

3. Delete any SageMaker endpoints you may have created through the SageMaker console.

in conclusion

Generative AI is changing how you can use IDP to gain insight into documents. Amazon Cloud AI services such as Amazon Textract and Amazon FMs can help accurately process any type of document. For more information about using generative AI on Amazon Cloud, see New Tools Released for Building Generative AI on Amazon Cloud .

Article source: https://dev.amazoncloud.cn/column/article/64e5fcaa5cf856038494f188?sc_medium=regulartraffic&sc_campaign=crossplatform&sc_channel=CSDN