The Alibaba cloud disk resource search engine developed by myself has started trial operation, come and experience it.
Search entry: https://www.so1st.com/
The problem of insufficient space on the Aliyun disk also gave me a headache. There are too many resources, but the free space is too small to save them! ! Since you are a programmer, you can write an Alibaba cloud disk search engine service by yourself. You can find any resources you need by searching, and you don’t need to save them all in your own space. This reason is the original reason for this project. However, I am not an all-rounder. I am good at the backstage, but not good at the frontend. Then let's find resources on Alibaba Cloud Disk to learn.
The front-end page of the resource search engine of this project took a long time to learn, and currently only provides a simple search engine entry, although simple, but the function is still sufficient.
Whether the interface is beautiful or not, I don't care about it for the time being. Let's figure it out first and then consider other things. Those who want to experience can try it.
When accessing, there will be corresponding usage prompts to help you accurately match the resources you need.
project instruction
If you plan to write a search engine completely by yourself, the workload is obviously very large, so you need to use open source products to complete this project.
Briefly describe what this project uses:
- Redis
- MongoDB
- Monstache
- Elasticsearch / Meilisearch
- Crawler module + high anonymous proxy pool
- Vue frontend + GoLang
After the whole thing is done, it is mainly these modules.
a detour
Careful friends may have noticed that there are two core search engines, ES and meili, and there is a detour here.
At first I only planned to use meili to play around, but the result was disappointing. The index file update is slow, the disk space takes up a lot of space, maintaining the index requires the development of a bunch of tools, and a series of problems. Although I tried my best to solve these problems, in the end Gave up on using meili. meili is a bit unbeautiful.
If you directly use MongoDB to provide search services, the ability to retrieve Chinese text is very poor, and Elasticsearch is the most suitable choice.
illustrate
The whole project is basically available initially, and it took a long time to get it done from the back end to the front end.
After that, the data sources of crawlers expanded, and it may be necessary to start looking for a rich resource sharing platform .