a plurality of collectors ## colly 8

Use multiple collectors

If the task is complex or with different types of subtasks enough, it is recommended to use multiple collectors to grab a job. coursera course scraper is a good example, it uses a two collector - a list view and analytic processing details page, another collection of courses.

Note : Using the collector. ID to distinguish different collectors in debugging

 

Cloning collector

If the collector has a similar configuration may be used collector Clone () method. Clone () to copy having the same configuration but without additional callbacks collector.

c := colly.NewCollector(
	colly.UserAgent("myUserAgent"),
	colly.AllowedDomains("foo.com", "bar.com"),
)
// Custom User-Agent and allowed domains are cloned to c2
c2 := c.Clone()

　　

Custom Data transfer between the collector

The collector using the Request () function can be shared with other collectors context.

Examples of shared context:

c.OnResponse(func(r *colly.Response) {
	r.Ctx.Put(r.Headers.Get("Custom-Header"))
	c2.Request("GET", "https://foo.com/", nil, r.Ctx, nil)
})