Skip to main content

Let Meta's Crawler Reach Your Shop: Cloudflare and Firewall Settings

Imagine you have set everything up: the pixel is connected, the Pixel Helper shows a green ViewContent on every product page, and yet your catalogue in Commerce Manager stays empty. Or Commerce Manager says your feed URL "does not point to a valid or accessible file", even though it opens perfectly in your browser.

In many of these cases nothing is wrong with the module or the data. The problem is the door: before Meta adds a product to a catalogue, one of its servers has to visit your shop and read the page or the file. If your security system mistakes that visit for an attack and stops it, Meta never sees your products — while everything looks fine from your side.

The good news: this is quick to check, and the fix is a small exception in Cloudflare or in your hosting firewall. You do not need to switch off your protection.

The short answer​

  • Your events reach Meta from your visitors' browsers (and from your server, with the Conversion API). The catalogue needs something different: Meta's own crawler has to download the product page or the feed file.
  • A successful ViewContent event does not prove that Meta could read the page. If the crawler is blocked, products can stay out of the catalogue while every event test passes.
  • Cloudflare's own documentation describes the typical causes: I'm Under Attack mode, a custom rule that challenges or blocks Facebook's IP addresses, or a country rule. Meta crawls from the United States and Ireland, so a rule that challenges other countries can catch it.
  • The fix Cloudflare recommends is a rule that matches Meta's networks — the ASNs AS32934 and AS63293 — and skips only the protection that blocks them, instead of a hand-made list of IP addresses.

How Meta reads your shop​

What you seeWhat it means
ViewContent fires and the product appears in the catalogueEvents and crawler both work.
ViewContent fires, but the catalogue stays empty for daysThe events arrive; check whether the crawler is blocked.
The feed URL opens in your browser, Commerce Manager says it is not accessibleYour browser is let through; Meta's crawler is not.
The Sharing Debugger shows an error page or a 403 response codeMeta's crawler is being blocked or challenged.

Meta identifies its crawler with the user agent facebookexternalhit/1.1. On its crawler documentation page, Meta suggests allowing either that user agent or the crawler's IP addresses. Meta does not publish a fixed list of IP ranges there. That is why lists copied from old forum posts are unreliable: they go stale without notice.

Step by step​

1. Test from Meta's side, not from your browser​

Your own browser always passes, so it cannot tell you whether Meta is blocked. Use a tool that fetches the page from Meta's servers:

  1. Open Meta's Sharing Debugger (https://developers.facebook.com/tools/debug/).
  2. Paste the URL of one of your product pages (or your feed URL) and click Debug.
  3. Click Scrape Again (or Fetch New Scrape Information) so Meta fetches the page now.
  4. Look at the Response Code and at the title and image Meta found.

A 200 response with your real product title and image means the crawler gets through. A 403, a 503, or a title such as "Attention Required!" or "Just a moment..." means a security system is stopping it.

tip

Pixel Plus users can also use the Microdata test tool in the module's Microdata tab: it opens a random product page in Meta's Microdata Debug Tool, which also fetches the page from Meta's servers.

2. Look for Meta in Cloudflare's security events​

If your site uses Cloudflare:

  1. Open your Cloudflare dashboard and select your website.
  2. Open the security events log (Security → Events; the menu name varies between dashboard versions).
  3. Add a filter for the ASN field with the values 32934 and 63293, or for the user agent facebookexternalhit.
  4. Write down which service or rule took the action (for example a custom rule, the security level, a country rule, or bot protection) and whether the action was a block or a challenge.

This tells you exactly which protection to relax. Fix only that one.

3. Create a skip rule for Meta's networks​

Cloudflare's troubleshooting page for Facebook sharing gives two options: remove the rule that challenges or blocks Facebook, or create a skip rule for the ASNs AS32934 and AS63293 that skips Security Level.

To create the skip rule:

  1. In your Cloudflare dashboard, open Security → WAF → Custom rules (in the newer dashboard, Security → Security rules) and create a new rule.

  2. Give it a clear name, for example Allow Meta crawler.

  3. Click Edit expression and enter:

    (ip.src.asnum in {32934 63293})

    This is our example of an expression that matches requests coming from those two Meta networks.

  4. As the action, choose Skip.

  5. Select Security Level, as Cloudflare recommends. If the events from step 2 showed another protection blocking Meta (for example another custom rule), select that one too — and nothing else.

  6. Save and deploy the rule, and place it before the rules that were blocking Meta.

warning

Do not skip every protection "just in case". Skip only what the security events showed as blocking Meta. A narrow exception keeps your shop protected and still lets the crawler in.

Cloudflare also recommends reviewing your configuration rules and Page Rules, and checking that I'm Under Attack mode is not enabled. That mode challenges every visitor, Meta included.

4. Check your hosting firewall too​

Cloudflare and your hosting server can have independent security systems. Even when Cloudflare lets Meta through, the server behind it can still block the request. Common ones are:

  • ModSecurity (frequent on cPanel and Plesk hosting)
  • Imunify360
  • BitNinja
  • CSF or Fail2Ban

If you do not manage the server yourself, send your hosting provider the URL that fails and ask them to check their firewall logs for requests with the user agent facebookexternalhit/1.1, and to allow them.

5. Ask Meta to read the page again​

After changing a rule, go back to the Sharing Debugger and click Scrape Again. When it shows a 200 response with your real title and image, Meta can read your shop. Catalogue items then start to appear or update as Meta processes them; this is not instant.

If you see X, do Y​

If you see…Do this
Sharing Debugger: 403, 503, "Attention Required!" or "Just a moment..."Cloudflare is challenging Meta. Go to steps 2 and 3.
Cloudflare events show nothing, but the debugger still failsThe block happens on your server. Go to step 4.
Debugger shows 200, but the title or price is wrongThe crawler gets in; the problem is your page's microdata. Review it in the Pixel Plus Microdata tab.
Debugger shows 200 with correct data, catalogue still emptyCheck that the pixel is connected to the catalogue as a data source and shows as Ready, and look at the catalogue's Diagnostics in Commerce Manager.
A country rule challenges visitors from outside your marketExclude Meta's ASNs from that rule: Meta crawls from the US and Ireland.

Frequently Asked Questions​

Q: My pixel events work. Why would Meta's crawler be blocked? A: Events are sent from your visitors' browsers and from your server, so your security system never stops them. The crawler is a separate request that comes from Meta's own servers, and firewalls often treat automated requests from abroad as suspicious.

Q: Should I add Meta's IP addresses to my firewall's allow list? A: Meta does not publish a fixed list of crawler IP addresses on its crawler page, and old lists go stale. In Cloudflare, a rule based on Meta's ASNs (AS32934 and AS63293), as Cloudflare documents, is easier to maintain.

Q: Does the skip rule turn off Cloudflare's protection for my shop? A: No. It only applies to requests coming from Meta's networks, and only skips the protections you select. Every other visitor goes through the same checks as before.

Q: I created the rule. How long until products appear in the catalogue? A: The access itself works immediately — confirm it with the Sharing Debugger. Meta then processes the products at its own pace, so items appear gradually rather than all at once.

Q: Does this also apply to product feeds? A: Yes. When your catalogue uses a feed URL, Meta downloads that file from its servers, and the same Cloudflare or server rules can block it.

My pixel events work. Why would Meta's crawler be blocked?

Events are sent from your visitors' browsers and from your server, so your security system never stops them. The crawler is a separate request that comes from Meta's own servers, and firewalls often treat automated requests from abroad as suspicious.

Should I add Meta's IP addresses to my firewall's allow list?

Meta does not publish a fixed list of crawler IP addresses on its crawler page, and old lists go stale. In Cloudflare, a rule based on Meta's ASNs (AS32934 and AS63293), as Cloudflare documents, is easier to maintain.

Does the skip rule turn off Cloudflare's protection for my shop?

No. It only applies to requests coming from Meta's networks, and only skips the protections you select. Every other visitor goes through the same checks as before.

I created the rule. How long until products appear in the catalogue?

The access itself works immediately — confirm it with the Sharing Debugger. Meta then processes the products at its own pace, so items appear gradually rather than all at once.

Does this also apply to product feeds?

Yes. When your catalogue uses a feed URL, Meta downloads that file from its servers, and the same Cloudflare or server rules can block it.

Official documentation​