Search for the most relevant papers on arXiv with one click. Thanks to ChatGPT, it can be completed in a few days.
It seems that the paper search tool has also begun to roll in!
For friends who search for papers every day, don’t be too happy if there is a useful search tool, and the efficiency will be greatly improved.
But the actual result is that either the search tool is not powerful, or the keywords you enter do not work. Anyway, the paper you want to find in your mind and the search results cannot be said to have nothing to do with each other. They are simply different.
The website we will introduce below can help you solve the problems encountered in paper search. The website is called arXiv Xplorer, which is specially used for semantic search of papers on arXiv. According to the project authors, the site's internal algorithm uses OpenAI's latest embedding model to perform search queries for users to find the most relevant papers.
arXiv Xplorer address: https://arxivxplorer.com/
The project author stated that he was impressed by OpenAI's new embedding API, so he wanted to see how the embedding was used in practice. So he spent a few days building the project and so far it's working really well. In addition, he also used ChatGPT to write 80% of the UI, used pinecone to store the vector database, and used googlecloud functions to embed queries and perform lookups.
If you want to know more about the embedded model, you can go to the website to view it.
Embedding model: https://openai.com/blog/new-and-improved-embedding-model/
With arXiv Xplorer, you can find the paper you need. Even if your description is very vague, or even just enter an uninformative description like "interesting ML paper", the engine can help you complete it. As shown below, the query results displayed by the website after entering several keywords.
In the process, you can also discover interesting papers you've never seen before with traditional search tools like Google or arXiv's own search, and arXiv Xplorer seems to be even better than that.
You can also directly search for similar papers by pasting the arxiv url. For example, the input in the figure below is the address of the paper "A Generalist Agent". The search results are displayed (red box). The similarity of A Generalist Agent is 100%, and the other Search results are expanded in order of score.
In addition, you can also click on the small triangle in the red box above, and then the interface will become as shown below, showing the paper participants and abstract. There are two functions below: "More Like This" will show more Similar papers; "View" will link to the paper's homepage on arXiv.
Seeing this fully functional website, netizens couldn't hold back their curiosity and asked, "You used OpenAI's embedding technology, but this technology is charged, so how much did you pay for it?". According to the project author: "It costs $40 to embed all papers in the CS category (about 500,000 papers)."
Some netizens raised a series of questions about the technology, such as: "Does this website embed all arXiv titles?" The project author said: "He embedded the titles and abstracts of all papers, and initially manually did the cosine similarity and Sorting, but pinecone makes it super easy!"
Some netizens suggested: "The search function of this tool is very good! It would be cool if you could view and sort by release date." Regarding this, the project author said that he will continue to optimize in the future and strive to achieve more perfect functions. .
Reference link: https://twitter.com/tomtumiel/status/1611729847700570118?s=20&t=sW31zy64CvhMH81ntcxzXw