{
  "generatedAt": "2026-07-07T07:58:59.529Z",
  "project": "LLMs.txt Kit",
  "domain": "llmstxtkit.com",
  "origin": "https://llmstxtkit.com",
  "status": "stackoverflow_answer_ready_for_owner_review",
  "postAuthorized": false,
  "externalPostingPerformed": false,
  "sourceInput": "dist/stackoverflow-answer-review-input.json",
  "summary": {
    "questionId": 77648557,
    "answerCount": 0,
    "isAnswered": false,
    "sourceSnapshots": 4,
    "fetchedSourceOk": 3,
    "hasPromotionGuidanceUrl": true,
    "recommendedDraft": "noLinkAnswer",
    "optionalDisclosureLinkDraftAvailable": true,
    "searchVisibilityObserved": true,
    "organicProofObserved": true,
    "priorityDistributionReady": "priority_distribution_ready",
    "liveOpportunityScanStatus": "live_opportunity_scan_has_manual_review_candidates"
  },
  "question": {
    "source": "Stack Exchange API",
    "fetchedAt": "2026-06-29T02:30:00+08:00",
    "questionId": 77648557,
    "url": "https://stackoverflow.com/questions/77648557/do-llm-crawlers-respect-the-robots-meta-tag",
    "title": "Do LLM crawlers respect the robots meta tag?",
    "tags": [
      "html-meta",
      "robots.txt",
      "large-language-model"
    ],
    "isAnswered": false,
    "answerCount": 0,
    "score": 1,
    "viewCount": 457,
    "lastActivityDate": "2023-12-25T07:13:27.000Z",
    "bodySummary": "The asker wants to know whether robots meta noindex, or a new meta directive such as noteach or nolearn, can be used as a granular LLM crawler opt-out instead of broad robots.txt user-agent rules."
  },
  "answerDecision": {
    "recommendation": "Ready for owner review; do not post automatically.",
    "rationale": [
      "The question directly asks whether robots meta noindex can act as granular LLM crawler control.",
      "The answer can stand alone without a promotional link.",
      "Official docs support the distinction between robots.txt crawler access and page/response-level robots meta or X-Robots-Tag behavior.",
      "Because this is Stack Overflow, the safest default is a no-link answer; use the disclosure link draft only if the owner explicitly authorizes and the link adds value."
    ],
    "useNoLinkDraftByDefault": true,
    "requireOwnerAuthorizationAtPostTime": true,
    "requireAffiliationDisclosureIfLinkUsed": true
  },
  "sourceBackedClaims": [
    {
      "claim": "robots.txt is the right first layer for documented crawler user-agent access preferences.",
      "sources": [
        "https://developers.openai.com/api/docs/bots",
        "https://developers.google.com/search/docs/crawling-indexing/robots/intro"
      ]
    },
    {
      "claim": "robots meta tags and X-Robots-Tag headers are page or response-level directives that require a crawler to fetch the content or response first.",
      "sources": [
        "https://developers.google.com/search/docs/crawling-indexing/robots-meta-tag"
      ]
    },
    {
      "claim": "OpenAI crawler controls should separate GPTBot and OAI-SearchBot instead of assuming one policy covers all OpenAI uses.",
      "sources": [
        "https://developers.openai.com/api/docs/bots"
      ]
    },
    {
      "claim": "Private account/customer content should use authentication or permissions, not crawler directives.",
      "sources": [
        "https://developers.google.com/search/docs/crawling-indexing/robots/intro"
      ]
    }
  ],
  "platformPolicyReview": {
    "source": {
      "id": "stackoverflow-promotion",
      "label": "Stack Overflow promotion guidance",
      "url": "https://stackoverflow.com/help/promotion",
      "fetchedStatus": 403,
      "relevantFinding": "The official guidance URL is recorded, but direct fetch was blocked by Cloudflare. Apply a conservative rule: the answer must stand alone, any owned link must be clearly disclosed, and the link must be optional rather than the point of the answer."
    },
    "conservativeRulesApplied": [
      "The answer must solve the question without requiring the owned link.",
      "Use the no-link draft by default.",
      "If the owned link is used, disclose affiliation in the answer body.",
      "Do not post if an existing answer already covers the same points fully.",
      "Do not post if Stack Overflow UI, comments, or moderator guidance indicate the question is off-topic, closed, duplicate, or not suitable for a new answer."
    ]
  },
  "drafts": {
    "noLinkAnswer": "Short answer: do not treat robots meta tags or noindex as a universal LLM crawler opt-out.\n\nThe important distinction is timing:\n\n1. robots.txt is checked before fetching a URL, so it is the normal place to express crawler access preferences for crawlers that document and honor those user-agent rules.\n2. A robots meta tag or X-Robots-Tag header is only visible after the crawler fetches the page or response.\n3. noindex is primarily an indexing/serving directive for search engines that support it. It is not a universal AI training or model-use opt-out.\n4. If robots.txt blocks a URL, the crawler may never fetch the page and therefore may never see a page-level meta tag.\n5. For private, account-level, customer, or tenant-specific content, use authentication and permissions. Crawler directives are not access control.\n\nFor the OpenAI-specific example in the question, I would keep GPTBot and OAI-SearchBot as separate robots.txt decisions. They are different crawler controls, so a CMS-level policy should not assume one page-level noindex rule covers both training-use and search-use cases.\n\nFor a CMS, a practical design is:\n\n- site owner sets broad crawler policy in robots.txt\n- account or tenant owner controls whether private pages are public at all\n- public pages can still use noindex or X-Robots-Tag for search indexing behavior\n- anything sensitive stays behind login instead of relying on robots directives",
    "disclosureLinkAnswer": "Short answer: do not treat robots meta tags or noindex as a universal LLM crawler opt-out.\n\nThe important distinction is timing:\n\n1. robots.txt is checked before fetching a URL, so it is the normal place to express crawler access preferences for crawlers that document and honor those user-agent rules.\n2. A robots meta tag or X-Robots-Tag header is only visible after the crawler fetches the page or response.\n3. noindex is primarily an indexing/serving directive for search engines that support it. It is not a universal AI training or model-use opt-out.\n4. If robots.txt blocks a URL, the crawler may never fetch the page and therefore may never see a page-level meta tag.\n5. For private, account-level, customer, or tenant-specific content, use authentication and permissions. Crawler directives are not access control.\n\nFor the OpenAI-specific example in the question, I would keep GPTBot and OAI-SearchBot as separate robots.txt decisions. They are different crawler controls, so a CMS-level policy should not assume one page-level noindex rule covers both training-use and search-use cases.\n\nFor a CMS, a practical design is:\n\n- site owner sets broad crawler policy in robots.txt\n- account or tenant owner controls whether private pages are public at all\n- public pages can still use noindex or X-Robots-Tag for search indexing behavior\n- anything sensitive stays behind login instead of relying on robots directives\n\nDisclosure: I maintain a small free checklist on this robots.txt vs robots meta vs X-Robots-Tag decision here: https://llmstxtkit.com/blog/do-llm-crawlers-respect-robots-meta-tag.html?utm_source=stackoverflow&utm_medium=answer&utm_campaign=first-distribution&utm_content=stackoverflow-llm-robots-meta-tag",
    "linkOnlyNotAllowed": false,
    "trackedUrl": "https://llmstxtkit.com/blog/do-llm-crawlers-respect-robots-meta-tag.html?utm_source=stackoverflow&utm_medium=answer&utm_campaign=first-distribution&utm_content=stackoverflow-llm-robots-meta-tag"
  },
  "skipWhen": [
    "The question has a complete existing answer when checked in the browser.",
    "The question is closed, duplicate, locked, deleted, or not accepting answers.",
    "The answer would mainly promote LLMs.txt Kit rather than answer the technical question.",
    "Affiliation cannot be disclosed clearly.",
    "The owner has not explicitly authorized posting in Stack Overflow.",
    "The final browser check shows current platform rules or UI state that conflict with this review."
  ],
  "evidenceCommands": {
    "recordPosted": "npm run distribution:evidence -- --placement-id 'stackoverflow-llm-robots-meta-tag' --url PUBLIC_POST_URL --posted-at YYYY-MM-DD --status 'posted' --note 'Owner posted after manual review; affiliation disclosed if owned link used.'",
    "recordSkipped": "npm run distribution:evidence -- --placement-id 'stackoverflow-llm-robots-meta-tag' --url PUBLIC_POST_URL --posted-at YYYY-MM-DD --status 'skipped' --note 'Skipped after manual Stack Overflow review.'",
    "refreshReview": "npm run distribution:stackoverflow-review",
    "refreshLiveOpportunityScan": "npm run distribution:scan",
    "refreshMeasurement": "npm run traffic:measurement",
    "refreshGoalAudit": "npm run goal:audit"
  },
  "safety": {
    "noExternalPostingDone": true,
    "noAutomatedComments": true,
    "noVoteManipulation": true,
    "noFakeTraffic": true,
    "directHelpfulAnswerPreferred": true
  }
}