card
I really like using card notes . The advantages are obvious, such as significantly reducing the stress of writing. While the stress of writing an essay can be overwhelming when you're faced with a blank screen, it's much easier to jot down flashcard notes at any time. Since the notes exist in the form of cards, a large amount of reorganization and reuse can be performed, which is very friendly to content output .
However, card notes also pose some problems. Information is scattered across many cards, making it difficult to find and use. In order to solve this problem, existing card note-taking tools generally adopt a double chain method .
Double links link between notes. When we need to find notes, we no longer just search by keywords, but find a certain note as a seed, and then follow the path pointed out by the link to associate with the cluster, find the relevant cards, and perform integrated output. Card-note-taking tools like Obsidian, Roam Research, and Logseq all offer such bidirectional links.
doubt
Recently, I have often received questions from readers in the background of Knowledge Planet and Official Accounts: Can AI (such as ChatGPT) interact with our local card note library? This makes it possible to take the content of many cards that are closely related to a specific topic and integrate them organically. Then use ChatGPT's natural language question-and-answer interaction capabilities to allow AI to answer our questions in a smooth, clear, accurate and comprehensive way, forming a unique knowledge output.
Answering this question is somewhat difficult. There are already many solutions for the question-and-answer dialog of a single document, such as ChatDoc, which I recommended before. However, to comprehensively extract information from multiple documents, ChatDoc can't handle it. Easy-to-use multi-document question answering tools are readily available, such as ChatBase.
Easy to use is easy to use, but the price cannot be ignored. The minimum monthly fee for ChatBase is $19.
At first, I thought that $19/month might not be cheap, but it might be worth it if it could improve user productivity and bring a competitive advantage. However, I changed my mind after reading this author's interview.
The author observed that most users chose the lowest gear of $19 at the beginning, but soon either increased the subscription level (paying more), or simply did not renew the subscription. This shows that the $19 monthly subscription fee is simply not enough for most people .
Fortunately, I recently discovered a new application that provides multi-document conversations to most users in an affordable way.
application
It's called Quivr, and it's available here.
Before using, you need to set your OpenAI API key, and then select the relevant model. I suggest you choose gpt-3.5-turbo-16k the model, because the maximum token length of the model updated on June 13 has been significantly improved, which can avoid truncating the answer for no reason with a high probability.
In the process of using Quivr, you still have to pay for calling the OpenAI API. But that cost pales in comparison to ChatBase's $19 monthly fee.
upload
The interface of Quivr is simple and easy to use. Users can drag and drop a series of documents to upload, or directly specify a certain URL, and it will crawl the website information for you by itself. In this way, you can easily and happily have a conversation with your own card notes.
In order to demonstrate convenience and protect privacy, I uploaded some articles previously published on the official account, knowledge planet, and newsboy. If the tool can handle long-form material like an article well, then short card notes are no problem.
Besides text and Markdown files, Quivr supports other types of files like PDF, PowerPoint, Excel, Word, and even audio and video. This means that users can upload various materials and types, and then ask questions uniformly. I think this design accurately captures the pain points of knowledge production users.
During the upload, the tool will indicate which files were uploaded successfully. All I upload are markdown plain text files, which are small in size and completed quickly.
Under Explore Options, we can check the uploaded files to make sure nothing is missing.
Users can also view the current usage of storage space through graphs. If all your materials are Markdown files, the 200 MB should last you a while.
Now that our information has been uploaded, we can start asking questions.
ask questions
I start by asking which applications of GPT are included in my knowledge base.
Quivr returned four results, the first three of which were fairly accurate, and obviously not from a single file, but a combination of information from multiple files. Except for the fourth answer which is too general, I am quite satisfied overall.
I then asked how GPT-4 can help in programming. It lists some specific applications based on my profile, such as code interpreter and Github Copilot Chat, etc.
It's just that I don't understand the second item of the answer, what is the automated essay scoring (AES), I haven't written it? As a result, I queried in Obsidian and found that it was "AI Writes Literature Review, Is it easy to use?" The answer given by GPT-4 cited in the article. No wonder, lol.
I further asked which GPT functions and GPT plugins can be used for scientific literature review, and asked it to give comprehensive results and list original information.
Quivr separates GPT features and GPT plugins (from two different articles), each with separate sources. However, I found that Quivr did not list specific document names, and some of the results were misleading, such as mistaking Wolfram for a literature review plugin.
I suspected that my prompt words might be wrong, so I corrected the prompt words and asked new questions.
This time I asked for comprehensive results, specific details, and also asked for raw information.
Here's Quivr's answer:
After reading this answer, I was quite excited. First of all, Quivr has indeed synthesized the content of about 4 documents, and has sorted out and refined them. Listing the names of these original files in the answer makes it easier for us to verify the answer and provides a basis for further knowledge mining.
feature
In addition, I found that Quivr also saves previous conversations, so that users can review and analyze historical information at any time. This feature can help you keep in-depth dialogue with your card notes library, find deep insights, and uncover problems that may have been overlooked.
Quivr is completely open source, and you can find its source code repository on GitHub.
Quivr provides a way for users to deploy services locally, using Docker. If you need it, you can follow the official tutorial.
LangChain
Let's briefly describe the technology used by Quivr. Its underlying framework is LangChain, a particularly popular library on GitHub with over 50,000 stars and over a million monthly downloads.
The advantage of the LangChain library is that it solves the problem of repetitive operations when people use large language models. Repetitive operations such as derivation and backpropagation in deep learning are annoying, so deep learning frameworks such as TensorFlow and PyTorch have been born. Similarly, LangChain has in fact become a common framework in the LLM application field, and you can see its shadow in most of the current massive LLM applications.
If you are interested in LangChain, I recommend this free course jointly launched by Deep Learning AI and LangChain.
Although this is only a basic course, you can modify and adjust some existing projects after learning to meet your own application needs.
Safety
I know that after reading this, some readers will talk to me about privacy data leakage. Some people insist that as soon as you call the OpenAI API, your data will be harvested by the technology giants. In the ChatGPT scientific research preview stage, such worries are indeed necessary. I mentioned it specifically when I first introduced ChatGPT last December . However, we must also pay attention to advancing with the times. From March 1, 2023, OpenAI has made major adjustments to its data policy.
Data uploaded via the API will not be used by OpenAI to train or improve models unless explicitly requested to do so by the user. Due to regulatory requirements, OpenAI retains the data you upload through the API for 30 days and then deletes it.
If you are too sensitive to data privacy to use OpenAI, Quivr can help you too. More recently it has supported the open source model GPT4All. This model I introduced to you before can run on a laptop . Although there is still a considerable gap between GPT4All and GPT-4 in dealing with complex cognitive problems, it is still competent enough to answer the key points of several document extractions.
With the addition of more localized models and open source models, personal and institutional knowledge bases can also use Quivr for knowledge retrieval and question answering locally without worrying about data privacy leaks.
summary
Today I introduce you to Quivr, a free and open source software that uses ChatGPT to interact with your card note library. Reduce manual query operations when extracting content from cards, and reduce the pressure of sorting when you enter cards. Compared to apps like ChatBase, Quivr is free and open source. It supports a variety of file formats, and can also use the local open source large language model GPT4All, etc. Hope it can help you in knowledge management.
information
There are two other activities recently, and we look forward to your participation.
First of all, on the evening of June 29 (Thursday), I will do "How to use ChatGPT to improve scientific research efficiency?" in the sixth course of the "AI Insight" column. " keynote speech.
"AI Insight" is an AI column course specially designed for professionals, including 30+ high-quality live broadcasts, 60+ senior industry experts sharing cutting-edge information on AIGC/LLM/ChatGPT, and 4 dimensions explaining the bottom layer of the era of large models logic. Whether you are an AI beginner, a professional practitioner, a researcher, or an AIGC entrepreneur, you can gain something in the "AI Insights" column.
I also specially fought for fan benefits. The original price of 698 column courses is now priced at 298. My fans can also receive an additional 50 yuan discount coupon. After the coupon, you can subscribe to the column for only 248 (single course as low as 9.9) . Scan the QR code below to get fan benefits:
Next , at 2 pm on July 2 (Sunday), I will attend a new book launch event jointly organized by People’s Posts and Telecommunications Publishing House and Minorities at Beijing Yuedong Chaoyang Bookstore, and give a keynote speech of about 30 minutes.
If you are free, welcome to scan the QR code in the picture above to sign up.
Like + "Looking", and forward it to your friends in need. Can't receive push? That's because you only subscribed but didn't star it .
Welcome to subscribe to my small newsboy paid column, and no less than 10 articles will be updated every quarter. Subscribe for a full year at a great price.
If you have any questions, or want to join the community to discuss with friends who love to study, subscribe to Knowledge Planet . It not only includes push content from small newsboys, but also allows you to post and ask questions freely. There are hundreds of posts and questions and answers that have been accumulated before. Enough for you to turn around for a while. Knowledge Planet supports an unconditional refund within 72 hours, so you can try it with confidence.
If some links in the article may not be displayed and redirected normally, it may be due to the external link restrictions of the WeChat public platform. If you need to visit, please click the "Read the original text" link at the end of the article to view the version with complete links.