ShflBot

ShflBot collects public music recommendations and album information for Shfl, a music discovery site operated by Caleb Wright. Recommendations link back to their sources. The collected material is not used to train AI models.

Identifying ShflBot

Our crawler identifies itself with this User-Agent:

ShflBot/3.0 (+https://crawler.theshfl.com/bot)

The robots.txt product token is ShflBot. Older requests used ShflBot/2.1 or ShflBot/2.2 and linked to theshfl.com/bot or theshfl.com/bot.html. ShflBot has been used for more than five years, with an extended pause in broad crawling before the current crawler was prepared.

Crawling preferences

Public-page collection checks robots.txt and honors matching Allow, Disallow and Crawl-delay directives. Each public-page collector waits at least two seconds between requests to a host, and waits longer when its robots.txt policy requires it. Collection is initiated by the operator.

To exclude ShflBot from your site, add:

User-agent: ShflBot
Disallow: /

We respect access restrictions and back off on rate limits. Contact us if you need a lower request rate, want your site excluded, or want an existing recommendation removed.

Verifying requests

Our Web Bot Auth public key directory is available at https://crawler.theshfl.com/.well-known/http-message-signatures-directory. Public-page and robots.txt requests carry HTTP Message Signatures using Ed25519 and the Signature-Agent identity https://crawler.theshfl.com.

A User-Agent alone does not prove that a request comes from us. Verify a request's signature against the published key. Publishing this directory does not imply approval by a bot-verification provider.

Source IP addresses

ShflBot's current crawler and refresh requests use this dedicated outbound IPv4 address (shown in CIDR notation):

34.59.134.158/32

ShflBot runs on Google Cloud. We do not publish Google's shared address ranges as ShflBot-exclusive addresses.

Contact

Caleb Wright · caleb@theshfl.com

Please include the affected domain and, if available, a request timestamp and User-Agent so we can investigate.