Package Usage: go: github.com/temoto/robotstxt
Package robotstxt implements the robots.txt Exclusion Protocol
as specified in http://www.robotstxt.org/wc/robots.html
with various extensions.
3 versions
Latest release: over 4 years ago
1,275 dependent packages
View more package details: https://packages.ecosystem.code.gouv.fr/registries/proxy.golang.org/packages/github.com/temoto/robotstxt