Repository: gocolly/colly

Stars: 25238

README.md

Colly

Lightning Fast and Elegant Scraping Framework for Gophers

Colly provides a clean interface to write any kind of crawler/scraper/spider.

With Colly you can easily extract structured data from websites, which can be used for a wide range of applications, like data mining, data processing or archiving.

![GoDoc](https://pkg.go.dev/github.com/gocolly/colly/v2)
![Backers on Open Collective](#backers) ![Sponsors on Open Collective](#sponsors) ![build status](https://github.com/gocolly/colly/actions/workflows/ci.yml)
![report card](http://goreportcard.com/report/gocolly/colly)
![view examples](https://github.com/gocolly/colly/tree/master/_examples)
![Code Coverage](https://codecov.io/github/gocolly/colly?branch=master)
![FOSSA Status](https://app.fossa.io/projects/git%2Bgithub.com%2Fgocolly%2Fcolly?ref=badge_shield)
![Twitter URL](https://twitter.com/gocolly)

Swiftproxy – Enterprise Residential Proxy Infrastructure for Scalable Data Access

Swiftproxy is a high-performance residential proxy network designed for reliable large-scale data collection. Powered by an 80M+ IP pool across 195+ countries, it ensures stable, anonymous, and high-success-rate connections. It integrates seamlessly with tools like Colly to bypass IP blocks and anti-bot systems. Free trial available, plus use code PROXY90 for an exclusive discount.

![Swiftproxy](https://www.swiftproxy.net/?ref=colly)

Features

- Clean API
- Fast (>1k request/sec on a single core)
- Manages request delays and maximum concurrency per domain
- Automatic cookie and session handling
- Sync/async/parallel scraping
- Caching
- Automatic encoding of non-unicode responses
- Robots.txt support
- Distributed scraping
- Configuration via environment variables
- Extensions

Example

import (
	"fmt"
	"github.com/gocolly/colly/v2"
)
func main() {
	c := colly.NewCollector()
	// Find and visit all links
	c.OnHTML("a[href]", func(e *colly.HTMLElement) {
		e.Request.Visit(e.Attr("href"))
	})
	c.OnRequest(func(r *colly.Request) {
		fmt.Println("Visiting", r.URL)
	})	c.Visit("http://go-colly.org/")
}

See examples folder for more detailed examples.

Installation

go get github.com/gocolly/colly/v2

Bugs

Bugs or suggestions? Visit the issue tracker or join #colly on freenode

Other Projects Using Colly

Below is a list of public, open source projects that use Colly:

- greenpeace/check-my-pages Scraping script to test the Spanish Greenpeace web archive.
- altsab/gowap Wappalyzer implementation in Go.
- jesuiscamille/goquotes A quotes scraper, making your day a little better!
- jivesearch/jivesearch A search engine that doesn't track you.
- Leagify/colly-draft-prospects A scraper for future NFL Draft prospects.
- lucasepe/go-ps4 Search playstation store for your favorite PS4 games using the command line.
- yringler/inside-chassidus-scraper Scrapes Rabbi Paltiel's web site for lesson metadata.
- gamedb/gamedb A database of Steam games.
- lawzava/scrape CLI for email scraping from any website.
- eureka101v/WeiboSpiderGo A sina weibo(chinese twitter) scraper
- Go-phie/gophie Search, Download and Stream movies from your terminal
- imthaghost/goclone Clone websites to your computer within seconds.
- superiss/spidy Crawl the web and collect expired domains.
- docker-slim/docker-slim Optimize your Docker containers to make them smaller and better.
- seversky/gachifinder an agent for asynchronous scraping, parsing and writing to some storages(elasticsearch for now)
- eval-exec/goodreads crawl all tags and all pages of quotes from goodreads.

If you are using Colly in a project please send a pull request to add it to the list.

Contributors

This project exists thanks to all the people who contribute. [[Contribute]](CONTRIBUTING.md).
<a href="https://github.com/gocolly/colly/graphs/contributors"><img src="https://opencollective.com/colly/contributors.svg?width=890" /></a>

Backers

Thank you to all our backers! 🙏 [Become a backer]

Sponsors

Support this project by becoming a sponsor. Your logo will show up here with a link to your website. [Become a sponsor]

License

![FOSSA Status](https://app.fossa.io/projects/git%2Bgithub.com%2Fgocolly%2Fcolly?ref=badge_large)

Repository: gocolly/colly

Stars: 25238

README.md Copy Chapter

Colly

Sponsor Copy Chapter

Features Copy Chapter

Example Copy Chapter

Installation Copy Chapter

Bugs Copy Chapter

Other Projects Using Colly Copy Chapter

Contributors Copy Chapter

Backers Copy Chapter

Sponsors Copy Chapter

License Copy Chapter