Auditing a 40-site Cloudflare portfolio with an agent: what we found and what we fixed first

One Cloudflare account, 42 domains. Some are mature products, some are experiments, and a few are parked. The owner wanted every live site reviewed, the issues ranked across the whole portfolio, and pull requests for the fixes worth doing first. An agent did the work with RankPropel's MCP service and a second, independent check on every finding.

Status, October 14, 2026. Four pull requests are open. None is merged or deployed, so nothing below is a before-and-after result yet. We make no ranking, traffic or indexing claims.

Step 1: map every hostname to the code that serves it

You can't open a pull request for a site until you know which repository deploys it. The agent read the account's Worker custom domains and Pages projects through the Cloudflare API. It then matched each project name against the name field in wrangler config files, both in local checkouts and on GitHub.

• 37 domains serve a live site: 21 on Workers, 16 on Pages.
• One domain is an alias that redirects to another site.
• Four were skipped and listed: three whose HTTPS no longer completes, and one retired product.
• A handful of Pages projects have no linked repository, so their source is still unmapped. You can't safely fix what you can't trace.

Step 2: one key for the whole portfolio

RankPropel now registers websites in bulk and issues an organization-scoped agent key. One call added 35 sites and one key covered all of them. Each audit still takes pages from a single site, so findings and request IDs stay per site.

The agent sent up to ten public pages per site: 297 pages across 36 sites. RankPropel never received repository access, credentials or private pages. One site couldn't be audited at all: its homepage is a single line of plain text, not an HTML document. That is a finding in itself.

Step 3: check everything a second way

RankPropel returned 85 automated findings, mostly duplicate titles and descriptions and pages without exactly one H1. Our own crawler covered 348 pages and checked things a page snapshot can't show: robots.txt and sitemap responses, a deliberately missing URL, www against the bare domain, and canonical tags. Each finding we act on was confirmed by a second client, the Cloudflare zone audit, a local reproduction or the repository itself.

Across the portfolio:

• Missing canonicals were the most common real issue. At least one sampled page lacked a canonical on 19 of 37 sites.
• Both hosts answer. On 20 domains, www and the bare domain both serve the site with a 200. Of the 14 we checked by hand, canonical tags pick one host on 8; on 6, nothing tells search engines which host is primary.
• Discovery files served as web pages. On 4 sites, robots.txt comes back as an HTML page and the sitemap isn't XML. This is a single-page-app fallback answering every path.
• Soft 404s. On 7 sites, a URL that shouldn't exist returns 200 with the homepage.
• Broken www hosts. Three www hostnames return an error. One www homepage redirects to a URL containing a literal /:path*, which is a 404.

The fixes we shipped first, as pull requests

We ranked the work by impact × confidence ÷ effort and opened one pull request per project for the top four. Each pull request has live evidence, the findings we withdrew, a test where the repository has tests, the repository's own finish steps, and after-deploy checks.

A workplace-safety publication: the www homepage was a 404. The site redirects www to the bare domain with a Next.js rule: source: "/:path*" with a host condition. On Cloudflare Workers the root path produced an empty :path*, which wasn't substituted, so visitors landed on /:path*. Deeper pages redirected correctly, which is why nobody noticed. We reproduced it in the local Workers runtime, then confirmed the fix there: an explicit rule for / placed before the path rule. The second check mattered here. We didn't stop at "the redirect looks right in config"; we watched it fail and then pass.

A writing app: no robots.txt, no sitemap, no canonicals. Both discovery files returned 404 and every public page shared one title. The fix adds prerendered metadata routes, canonicals and distinct titles for three marketing pages. The repository's rules require a guard test that fails when the guard is removed, so we broke the sitemap and the robots rule on purpose and watched the test fail.

A network-engineering course site: 16 pages without canonicals, plus a duplicate host. Adding canonicals had a trap. In Next.js, a page's alternates metadata replaces the layout's alternates wholesale, so a plain canonical would silently drop the site's RSS link. Two existing pages had already lost it. The fix uses one small helper that sets the canonical and keeps the RSS link, and adds the www redirect with the root rule from the case above.

A world-audio listening app: no sitemap. The robots.txt was only the content-signals block that Cloudflare manages, with no Sitemap line. Section pages are reachable under language-filter URLs, and nothing declared a canonical. The fix adds a small sitemap of the section pages, a robots.txt that Cloudflare merges with its own block, and canonicals.

What we withdrew or left alone

• A "private key" in a public page. RankPropel's secret scanner rejected a batch because a tutorial showed sample scanner output with a PEM header line. There was no key material, so we excluded the page and reported the false positive.
• "Blocks all crawlers." Our own check misread a per-crawler Disallow: / rule (aimed at AI crawlers) as a site-wide block. Those rules are the owner's recorded policy, and we didn't touch them.
• Findings on third-party pages. On sites with empty sitemaps, our crawler followed homepage links into sign-in and checkout endpoints, which redirect to external providers. RankPropel then audited those providers' HTML. 46 of the 85 findings were on sign-in, checkout or app pages. We dropped them.
• A homepage without an H1. It's real, but that homepage is a full-screen embed of a designed page, a recorded design decision.
• Product and content choices. We didn't change pricing pages, AI-crawler policies, index gates or copy. Redirects between hosts and DNS repairs are Cloudflare decisions, so they went to the owner as a list, not changes.
• Speed. We didn't measure it: a slow local proxy makes timings from this network meaningless.

Working inside other people's rules

Several repositories are worked on from another machine, and some have strict operator rules. One deploys only from a Windows PC and has retired its CI. Another says a different agent owns merges. So nothing went to main: every change is a branch from a fresh clone, plus a pull request that says what the build machine must run. Where a repository's full build couldn't run on this machine, the pull request says so.

What changes in RankPropel

The run exposed gaps in RankPropel itself. The biggest: there's no check for a missing canonical, which was the most common real issue here. Snapshots also can't show robots.txt, sitemaps, soft 404s or redirect chains. Error messages should name the failing field. And sign-in and checkout pages should be ranked below content pages. These are now on the product list. The fixes in this article came from the evidence gate, not from the automated audit alone.

After deploy, we will check each pull request's acceptance commands on the live sites, re-run the same audits, and compare them with the before audits. Over the following weeks we'll look at which canonical Google selects for the affected pages. We'll update this article with what actually changed, including anything that didn't.