Indexing and machine readability
This site is built to be read by people, search crawlers and AI answer engines from the same server-rendered HTML. This page documents what is indexable, what is not, and what structured data each page type emits — generated from the same code that serves robots.txt and sitemap.xml.
Indexing rules by surface
Filter permutations stay crawlable so the graph is discoverable, but only editorially useful pages are
indexable. Every directive below keeps follow:
a page that should not rank is still how a crawler reaches the profiles that should.
| Surface | Directive | Note |
|---|---|---|
| Entity detail pages |
index, follow, max-image-preview:large, max-snippet:-1
|
Companies, founders, insights and reports. Full entity JSON-LD plus a BreadcrumbList built from the trail the reader is shown. |
| Directory and hub indexes |
index, follow, max-image-preview:large, max-snippet:-1
|
Canonical. Pagination is cumulative, so page three is a complete page rather than a fragment. |
| Single-facet filters |
index, follow, max-image-preview:large, max-snippet:-1
|
One active choice — companies in fintech, founders in Bangladesh — is a page people search for, and it is allowed to rank. |
| Multi-facet filters |
noindex, follow
|
More than one active choice, counting a search term as one. Near-duplicates of the pages they were built from: followed, not indexed. |
| Search results |
noindex, follow
|
A reflection of what somebody typed. Also disallowed in robots.txt, and it links out to indexable pages. |
| Thin industry hubs |
noindex, follow
|
Fewer than 3 published companies. Held out of sitemap.xml until the hub has substance, then indexed automatically. |
| Claim, submission, correction and enquiry flows |
noindex, follow
|
Transactional. Disallowed in robots.txt and absent from sitemap.xml entirely. |
robots.txt
Not a sample — this is the file, byte for byte, as served at /robots.txt.
# BizMend is written to be read by people, search crawlers and # answer engines from the same server-rendered HTML. Everything editorial is open. # Only transactional surfaces are closed, and they are absent from sitemap.xml too. # # Filter and pagination permutations are intentionally crawlable: they are how the # long tail is reached. They are kept out of the index by "noindex, follow" on the # page itself, not by a rule here. User-agent: * Allow: / Disallow: /search Disallow: /submit-company Disallow: /submit-founder Disallow: /claim/ Disallow: /corrections/ Disallow: /enquiry/ Disallow: /newsletter/ User-agent: GPTBot Allow: / Disallow: /search Disallow: /submit-company Disallow: /submit-founder Disallow: /claim/ Disallow: /corrections/ Disallow: /enquiry/ Disallow: /newsletter/ User-agent: OAI-SearchBot Allow: / Disallow: /search Disallow: /submit-company Disallow: /submit-founder Disallow: /claim/ Disallow: /corrections/ Disallow: /enquiry/ Disallow: /newsletter/ User-agent: ClaudeBot Allow: / Disallow: /search Disallow: /submit-company Disallow: /submit-founder Disallow: /claim/ Disallow: /corrections/ Disallow: /enquiry/ Disallow: /newsletter/ User-agent: PerplexityBot Allow: / Disallow: /search Disallow: /submit-company Disallow: /submit-founder Disallow: /claim/ Disallow: /corrections/ Disallow: /enquiry/ Disallow: /newsletter/ Sitemap: https://beta.bizmend.com/sitemap.xml
Structured data by page type
Emitted as JSON-LD in the head of the document. Any page with a breadcrumb trail also carries a BreadcrumbList built from the trail the reader is shown.
- Company profile
- Organization + Offer + FAQPage
- Founder profile
- ProfilePage + Person
- Industry and topic hubs
- CollectionPage + DefinedTerm + ItemList + FAQPage
- Country hub
- CollectionPage + Place + ItemList
- Insight
- Article
- Report
- Report + Dataset
- Pricing
- Product + Offer
- Policies and this page
- WebPage + Organization publisher
- Home
- Organization + WebSite + SearchAction + CollectionPage
- Search results
- BreadcrumbList only — the page is noindex
sitemap.xml
Generated from published records — 88 URLS right now, at /sitemap.xml. A sample of each group is shown below; nothing that renders noindex is submitted.
Companies (17)
Founders (11)
Hubs (26)
Editorial (22)
Trust (8)
What is left out
- Disallowed in robots.txt
- /search /submit-company /submit-founder /claim/ /corrections/ /enquiry/ /newsletter/
- Held out of the sitemap
- Search, claim, submission, correction, enquiry and newsletter routes, and industry hubs below 3 published companies.
- Never crawlable at all
- Nothing. No page on this site sends nofollow.