Our friends at Cloudflare just shipped the cleanest way to programmatically read any page on the web with their new /crawl endpoint. This is both awesome AND inexpensive. There is zero reason for any given page on a B2B website to be “unoptimized” any longer.

Awesome ship Cloudflare thank you! I especially love how you can crawl with authentication. That opens up some interesting “internal” scrape use cases as well as getting at stuff that is behind authwalls to pass to agents.

How to use:

Tell your tool of choice (openai codex, claude code, claude cowork… etc) to:

-> research this feature
-> read all the documentation
-> create a new SKILL.md agent skill
-> ask for directions on how to get a paid account, api key, whatever you don’t already know or have

Then you can programmatically call the skill to:

-> read any page on your site and recommend optimizations for whatever keyword / ICP

-> read any competitor page to extract information or to compare

-> monitor pages for changes and DM you a changelog in slack

Basically anything you want.

Yes, LLMs could just visit the page before, but doing so tends to fill the context with miscellaneous code and other junk that you don’t need.

Cloudflare just gets you the copy and content. If you need visuals as well, there are a bunch of inexpensive screenshot tools out there that you can call as needed to get page screenshots.

Great stuff, can think of a few use cases and experiments already.

Go check it out!