Proxy RSS reads through a static IP proxy like QuotaGuard

[jmduke]

THE PROBLEM: Cloudflare sometimes blocks our RSS fetches because we're a bot (which is fair — we are!).

THE SOLUTION: We register with them as a cool, chill, friendly bot

THE PROBLEM: To verify our incoming traffic, they need an allowlist of IPs — and Heroku doesn't give us sacrosanct ones!

This will let us submit Buttondown as a registered crawler to Cloudflare, which should obviate some issues we have with CF blocking our requests.

I think we can just use Fixie or some similar option (the overall bandwidth is pretty low); the HTTP call is in retrieve_items.

(We will also get to use this for the Validity stuff, if we need to!)

Tasks

Beta Give feedback

1. use a static IP to fetch blocked cloudflare sites https://github.com/buttondown/monorepo/pull/2005
2. respect robots.txt and add our static IP to docs https://github.com/buttondown/monorepo/pull/2017
3. apply to cloudflare verified bots and get approved
4. check that it works

[catdevnull]

https://developers.cloudflare.com/bots/reference/verified-bots-policy/
A bot or proxy must have a minimum amount of traffic for Cloudflare to be able find it in the sampled data. The minimum traffic should have more than 1000 requests per day across multiple domains.

do we do this much traffic?

user-agent

ButtondownBot/1.0 or are we already using something else?

• also, we'll need to start checking robots.txt :/
• we need to document our bot, do we have something like that already?

[jmduke]

do we do this much traffic?

yup!

user-agent

we are! and lightweight docs here: https://docs.buttondown.com/rss-to-email#troubleshooting, but can be expanded.

[catdevnull]

okay, I submitted the bot. now we have to wait. I think they will email you @jmduke