On March 23, OpenAI dropped another blockbuster: launching a plug-in system for ChatGPT!
This move means that ChatGPT will usher in the "APP Store" moment, that is, around its capabilities, a developer ecosystem will be formed to create an AI-based "operating system"!
The plug-in system will bring a qualitative leap to ChatGPT, because with the help of plug-in services, it can obtain real-time Internet information and call third-party applications (booking hotel flights, ordering food, shopping, querying stock prices, etc.).
ChatGPT is an extremely smart brain, and plug-ins will become its eyes, ears, hands, feet, and even wings, with amazing capabilities, and the future is unimaginable!
The official currently provides two plugins:
-
A web browser. Using the API of the new Bing browser, search Internet content in real time, and give answers and links
-
A code interpreter. Using the Python interpreter, you can solve mathematical problems, do data analysis and visualization, edit pictures, edit videos, etc., and also support downloading processed files
In addition, OpenAI has also open sourced a knowledge base retrieval plug-in chatgpt-retrieval-plugin, which retrieves information from various data sources (such as files, notes, emails, and public documents) through natural language. With the open source code available, developers can deploy their own versions of the plugin.
Imagine if I provided a "Python knowledge base plugin" with all the official documents as the data source, then any questions about Python usage in the future, I just need to ask ChatGPT, then it calls the plugin and parses the data, and finally returns give me exact answer. This will save a lot of time!
Not only that, you can also use books as data sources to create "Journey to the West Knowledge Base", "Dream of Red Mansions Knowledge Base", "Encyclopedia Knowledge Base", "Personal Library Knowledge Base", etc.; Create an expert assistant with academic journals as the data source, and it will be very easy to write papers and search materials from now on; use the data sources of Socrates, Jobs, Musk and other celebrities to create a personalized personal consultant...
As the first open source ChatGPT plug-in, the chatgpt-retrieval-plugin project ranked first on the Github trend list as soon as it was released, and got 11K stars in just one week after its release.
This project is written entirely in Python, whether it is for the purpose of learning programming, or for reference in developing other plug-ins in the future, it is worth our time to study it carefully.
Next, I will share some information I gained while reading the project documentation and source code.
First of all, the project contains about 3 K of Python code, which is not a large scale. The project structure is also very clear, the directory is as follows:
| Table of contents | describe |
|---|---|
datastore |
Contains the core logic for storing and querying document embeddings using various vector database providers |
examples |
Includes configuration examples, authentication methods, and program provider-oriented examples |
models |
Contains data models used by plugins, such as document and metadata models |
scripts |
Holds useful scripts for processing and uploading files from different data sources |
server |
Store the main FastAPI server implementation |
services |
Contains utility services for tasks such as chunking, metadata extraction, and PII detection |
tests |
Includes integration tests for various vector database providers |
.well-known |
Store plugin manifest files and OpenAPI formats, define plugin configuration and API specifications, etc. |
Except for examples, tests, configuration files, etc., the three main directories are as follows:
datastore data storage
The text data of the data source will be mapped to a low-dimensional vector space, and then stored in the vector database. Official examples of data storage solutions such as Pinecone, Weaviate, Zilliz, Milvus, Qdrant, and Redis have been provided. In addition, there are several pull requests that want to add PostgreSQL support, and there is a high probability that it will be integrated in the future.
Used here 抽象工厂设计模式 , DataStore is an abstract class, and each data storage library is a concrete implementation class, which needs to implement three abstract methods:
(1)
_upsert(chunks: Dict[str, List[DocumentChunk]]) -> List[str]The method receives a dictionary parameter, which contains a list of DocumentChunk objects, and inserts them into the database. The return value is a list of document IDs.
(2)
_query(queries: List[QueryWithEmbedding]) -> List[QueryResult]The method receives a list parameter containing the query text to be embedded. Returns a list of query results containing matching document chunks and scores.
(3)
delete(ids: Optional[List[str]] = None, filter: Optional[DocumentMetadataFilter] = None, delete_all: Optional[bool] = None, ) -> boolmethod, delete according to id and other filter conditions, or delete all. Returns whether the operation was successful.
It is worth noting that the modules in this directory factory.py use the match-case syntax newly introduced by Python 3.10, following the new trend of the Python community~
server server interface
This directory has only one main.py file, which is the startup entry for the entire project. It uses the current mainstream FastAPI framework, provides several APIs for adding, deleting, modifying and checking, and uses the uvicorn module to start the service.
-
/upsert-fileInterface, used to upload a single file, convert it into a Document object, and then add or update -
/upsertInterface to upload a series of document objects for adding or updating -
/queryInterface, pass in a series of text conditions, convert them into QueryWithEmbedding objects, and then query from the vector database -
/deleteInterface, delete or delete all data in the database according to conditions
Among these interfaces, the addition, modification, and deletion functions are mainly for developers/maintainers, and ChatGPT only needs to call the query interface of the plug-in. Therefore, a "/sub" sub-application is also created in the code, which only contains /query the interface and is provided for ChatGPT calls.
In addition, it uses the mount method of FastAPI to mount a "/.well-known" static file directory, which exposes basic information about the plugin, such as name, description, author, logo, email, interface documents provided to OpenAPI, etc. wait.
services task processing method
Under this directory are some common functions, such as the following:
(1) The chunks.py file contains functions that divide strings and Document objects into small chunks, and get embedding vectors for each chunk.
(2) The file.py file provides functions to extract text content and metadata from uploaded files. File types currently supported for parsing include PDF, plain text, Markdown, Word, CSV, and PPTX.
(3) The openai.py file contains two functions: The get_embeddings function uses the OpenAI text-embedding-ada-002 model to embed the given text. The get_chat_completion function generates a conversation using OpenAI's ChatCompletion API.
On the whole, several interface functions of this plug-in are very clear, and the code logic is not complicated. The core text embedding operation is with the help of openai's Embedding interface, and the storage and query operations of text block information depend on the functions of various vector databases.
There is a blogger on YouTube who drew a schematic diagram by hand. Although the font is scribbled, you can understand it:
His video is worth watching, because the up master not only briefly introduces the working principle of the plug-in, but also demonstrates how to deploy to Digital Ocean, how to modify the configuration, and how to debug, and he has the plug-in permission of ChatGPT, so he can deploy his own The plug-in is connected to ChatGPT, and the use of the knowledge base plug-in is demonstrated on the spot!
Video: https://www.youtube.com/watch?v=hpePPqKxNq8
At present, there is still relatively little information about the introduction, development and configuration of the ChatGPT plug-in, after all, it is newly launched. However, there are countless individuals and organizations applying for the waitlist. Once open for use, various plug-ins will definitely expand the ChatGPT ecology like the rich open source libraries of the Python community.
Finally, the official documentation of the plug-in chatgpt-retrieval-plugin is the most detailed first-hand information, and I recommend everyone to study it. https://github.com/openai/chatgpt-retrieval-plugin