crawler

VarsioBot

Varsio helps athletes find a fitting US college program. To do that, our crawler reads public roster and staff pages. This page says what it takes, how it behaves, and how to make it stop.

What it reads

VarsioBot visits the public athletics websites of US colleges and reads two kinds of pages: team rosters and coaching staff pages. Only pages anyone can open without an account. Nothing behind a login or a paywall.

From a roster it keeps facts: name, class year, position, height, hometown and previous school. From a staff page it keeps name, title, and the work email and phone number the school published. It does not copy photos, biographies or articles. Every record stores the page it came from and the date it was read.

How it behaves

It follows robots.txt, including Crawl-delay.
It waits a few seconds between two requests to the same site.
It slows down when a server answers 429 and respects Retry-After.
It identifies itself as VarsioBot in the User-Agent header, with a link to this page.
If a site answers with an automated browser check, the page may be loaded through a browser service instead. A robots.txt block is always respected.
Staff pages are read again once a night, most roster pages about once a week. That is one request per page.

For athletics departments: stop the crawler

Add this to your robots.txt and VarsioBot stays away from your site:

User-agent: VarsioBot
Disallow: /

If the crawler causes load or you want existing records of your program removed, write to privacy@varsio.io.

For athletes and coaches: remove your data

You are listed because your school published you on its roster or staff page. If you want your record gone, send your name and school to privacy@varsio.io. We delete it and block it, so the crawler does not collect it again.

More on how personal data is handled: privacy policy.