# D009 - SEO and AI Crawlers

> D009 - SEO and AI Crawlers: a note from the Obsidian vault behind https://saifsiddiqui.in.

Canonical URL: https://saifsiddiqui.in/hi/brain/d009-seo-and-ai-crawlers/ (this is its markdown representation; the same URL serves HTML to browsers)
Other languages: en: https://saifsiddiqui.in/brain/d009-seo-and-ai-crawlers/ · kn: https://saifsiddiqui.in/kn/brain/d009-seo-and-ai-crawlers/ · ur: https://saifsiddiqui.in/ur/brain/d009-seo-and-ai-crawlers/ · te: https://saifsiddiqui.in/te/brain/d009-seo-and-ai-crawlers/ · ar: https://saifsiddiqui.in/ar/brain/d009-seo-and-ai-crawlers/ · hi-Latn: https://saifsiddiqui.in/hi-latn/brain/d009-seo-and-ai-crawlers/

**Type:** decision  
**Status:** accepted  
**Created:** 2026-07-17  
**Tags:** decision, seo, ai

## D009: World-class SEO + AI-crawler friendly

**Context:** Saif wants best-in-class SEO and to be *allowed* and surfaced by AI crawlers.

**Decision:**
- `robots.txt` that **explicitly allows** AI crawlers (GPTBot, ClaudeBot/anthropic-ai, PerplexityBot, Google-Extended, CCBot, Applebot-Extended, Bytespider, etc.) + permissive default.
- `/llms.txt` (+ optional `/llms-full.txt`), clean Markdown summary for LLM consumption.
- Localized `sitemap.xml` (incl. [Knowledge Graph](/brain/knowledge-graph/) notes), `hreflang` + `x-default`, per-page/locale meta + OG, JSON-LD (`Person`, `CreativeWork`, `WebSite`, `BreadcrumbList`).

**Why:** [Astro's](/brain/d002-tech-stack/) server-rendered HTML means crawlers get full content without JS. Allowing AI crawlers means Saif's work (and his documented *reasoning* via [Knowledge Graph](/brain/knowledge-graph/)) becomes citable in AI answers, a modern discoverability edge. `/llms.txt` is a subtle "ahead of the standards" signal, on-brand for [About Saif](/brain/about-saif/).

**Trade-offs:** allowing training crawlers means content may be used for training, accepted deliberately for visibility. Compounds with [Internationalization](/brain/internationalization/) (per-locale pages).

## से लिंक किया गया

- [D010 - Knowledge Graph Build in the Open](/hi/brain/d010-knowledge-graph-build-in-the-open/)
- [D013 - Live Demo Ask This Site](/hi/brain/d013-live-demo-ask-this-site/)
- [Internationalization](/hi/brain/internationalization/)
- [Knowledge Graph](/hi/brain/knowledge-graph/)
- [Open Questions](/hi/brain/open-questions/)
- [Phase 1 - Content MVP](/hi/brain/phase-1-content-mvp/)
- [Portfolio Brain](/hi/brain/portfolio-brain/)
- [Roadmap](/hi/brain/roadmap/)

सभी नोट्स: [/hi/brain/](/hi/brain/)

---

Mohd Saif, Product Engineer. I build solutions, not dead software.
Email: saifmd238@gmail.com · GitHub: https://github.com/Saif-09 · LinkedIn: https://www.linkedin.com/in/mohd-saif-134076141/ · Résumé (PDF): https://saifsiddiqui.in/resume

More for agents:
- [/llms.txt](/llms.txt): short summary of this site
- [/llms-full.txt](/llms-full.txt): the complete text of every page in one fetch
- [/agents.md](/agents.md): when to use this site, and how to query it
- [/sitemap-index.xml](/sitemap-index.xml): every URL, in seven languages