Write a search engine for Alibaba cloud disk by yourself
Are you still troubled by the small disk space of Alibaba Cloud? There are too many good things, but there is only so little space to store all the things, why?
The problem of insufficient space in the Alibaba Cloud disk also caused me a headache. Although there are some free search engines for the Alibaba Cloud disk, they are not mine!
Finally decided to write an Ali cloud disk search engine service by myself!
Take a look at the finished product picture:
project instruction
If you plan to write a search engine completely by yourself, the workload is obviously very large, so you need to use open source products to complete this project.
Briefly describe what this project uses:
- Redis
- MongoDB
- Monstache
- Elasticsearch / Meilisearch
- Crawler module + high anonymous proxy pool
- Vue frontend + GoLang
After the whole thing is done, it is mainly these modules.
a detour
Careful friends may have noticed that there are two core search engines, ES and meili, and there is a detour here.
At first I only planned to use meili to play around, but the result was disappointing. The index file update is slow, the disk space takes up a lot of space, maintaining the index requires the development of a bunch of tools, and a series of problems. Although I tried my best to solve these problems, in the end Gave up on using meili. meili is a bit unbeautiful.
If you directly use MongoDB to provide search services, the ability to retrieve Chinese text is very poor, and Elasticsearch is the most suitable choice.
at last
At present, the search service is still in the testing stage, and many hardware resources and security issues have not been resolved, and it will be open for use at an appropriate time.
The current plan is to share the development process of the project step by step in the form of articles, and friends who like it can follow me.