Foreword
This update includes several parts:
- Better support Scrapy reptiles
- Git repository synchronization support
- Support long task
- Better management of reptiles
Update Log
Function / Optimization
- Better support Scrapy . Reptile identification,
settings.pyconfiguration, log level selection, reptiles choice. # 435 - Git synchronization allows users to synchronize Git project to Crawlab.
- Long mission support . Users can add tasks long reptile, these reptiles can run long-running tasks. 425
- Optimization reptile list . Sub-state task number column statistics, task list details pop-up box, the legend. 425
- Upgrade detection . Detect the latest version, inform the user to upgrade.
- Reptile batch operation , allowing users to run batch / stop reptiles task, and bulk delete crawlers.
- Copy the reptile . Reptile allow users to copy already exists to create new crawlers.
- Micro-channel two-dimensional code group .
Bug fixes
- Timing task reptile choice . With the Reptile field does not change the response.
- Timing task conflict . Reptile set the timing of two different tasks set to the same time, it may have bug. # 515 # 565
- Task log problem . Different tasks at the same time trigger may be written with a log file. # 577
- Task list filtering options insufficiency .
Product Planning
- The results show
- Support for other databases
- Yes placement Reptilia
- Splash crawler can be configured to support
- Configurable crawler support CrawlSpider
- Crawler can be configured to support regular expressions field
- You may be configured to support crawler crawler into custom
- task
- Task retry mechanism
- Regular tasks
- Show calendar
- Overall situation
- Supported version update detection
- Supported version of the update log shows
- server
- Mirror support terminal operation Docker
- SDK
- More command support
- Support Golang, Java
- Plug-in system