bot protection

Bot protection is the commercial filtering tier - DataDome, Cloudflare challenges and similar systems - that decides which automated clients see content at all. Its defining feature in practice is selective depth: some of a site stays open while the valuable part hardens.

G2 is the clean example in this slice. Its 2,239 category pages across 43 top-level groups are openly readable, but per its catalog record "product and review pages sit behind DataDome bot protection". Product Hunt shows the same split differently: public launch pages sit behind Cloudflare while an official GraphQL API serves structured data. Capterra permits AI and search crawlers by name yet "generic agents face Cloudflare challenges" across its 86,267 product profiles. Similarweb closes the loop in writing, with terms section 6 banning "any robot, spider, scraper, or other automated means" outright.

It gates depth, not existence. The visible surface - category listings, leaderboards - is often deliberately open for search engines, so a pilot crawl succeeds and the production crawl fails at exactly the records you needed. That asymmetry is why scraping estimates built on sampling category pages misprice projects on software directories.

Legal exposure stacks on top. Where protection exists, terms usually ban automation explicitly (Similarweb section 6), so a blocked request is also a documented violation. The durable alternatives are official APIs where offered, sitemap-enumerated polite crawling where tolerated, or licensing.

1. Map which paths are hardened. At G2, categories read openly while product pages do not - test the specific record type you need, not the homepage.
2. Check named-agent rules. Capterra allows ClaudeBot, GPTBot and peers while challenging generic agents; your user agent determines your experience.
3. Prefer the sanctioned API. Product Hunt's GraphQL API exists precisely because its Cloudflare-fronted pages discourage scraping.
4. Read the terms clause numbers. Similarweb's section 6 ban is explicit enough to quote; vaguer sources deserve more caution, not less.
5. Assess refresh cost honestly. Daily-updated directories behind challenges mean daily maintenance, which is a salary line, not a one-off fee.

Where to see it in context: application-software data.

Entries adjacent to bot protection:

• robots-txt-crawl-rules — the declared permissions that sit above Capterra's named-crawler allowances
• web-catalog-scraping — the extraction practice challenging 86,267 product pages like G2's and Capterra's