ملخص أساسي

في SEO لعام 2026، يشرح robots.txt قواعد الزحف وتعرض sitemap.xml الروابط القانونية المهمة. يجب ألا يتعارضا.

دليل إعداد robots.txt و sitemap.xml لتحسين SEO في 2026

robots.txt syntax: User-agent, Disallow, Allow, Sitemap

Use robots.txt at the root of the host, for example https://millionscode.com/robots.txt. The practical 2026 baseline is simple: open the parts that should be crawled, block low-value operational paths, and declare the sitemap with a full URL. A safe starting point is:

User-agent: *
Allow: /

Sitemap: https://millionscode.com/sitemap.xml

For a production site, treat Disallow carefully. A single Disallow: / can stop crawling of the whole host. That is useful on staging and dangerous on a live site. If admin pages, carts, search results, or temporary filters should not be crawled, block those paths only:

User-agent: *
Disallow: /admin/
Disallow: /search
Disallow: /cart
Allow: /blog/
Allow: /tools/

Sitemap: https://millionscode.com/sitemap.xml

robots.txt is not a security layer. It is a crawler instruction file. Sensitive data must be protected by authentication, and pages that must disappear from search need the right removal method rather than only a crawl block.

sitemap.xml structure

A sitemap lists canonical URLs that deserve discovery. Keep it clean:

<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
  <url>
    <loc>https://millionscode.com/tools/meta-checker</loc>
    <lastmod>2026-06-03</lastmod>
  </url>
  <url>
    <loc>https://millionscode.com/blog/post-mp90fbw1</loc>
    <lastmod>2026-06-03</lastmod>
  </url>
</urlset>

Do not mix canonical and non-canonical versions. Do not submit URLs blocked by robots.txt. Use /tools/meta-checker for metadata checks, /blog/post-mp90fbw1 for sitemap generation planning, /blog/api-mpj0hwex for indexing API workflow, and /blog/post-mpkfy95s for Search Console checks.

Submission: Search Console and Naver

In Google Search Console, verify the property and submit sitemap.xml or sitemap_index.xml in the Sitemaps report. Then check status, discovered URL count, fetch errors, and whether blocked URLs are accidentally present. In Naver Search Advisor, verify ownership, run robots.txt diagnosis, submit the sitemap, and watch collection requests separately. For Korean search traffic, Naver diagnostics should not be skipped.

Common mistakes

The first mistake is leaving a staging rule on production. The second is listing blocked URLs in the sitemap. The third is mixing www, non-www, http, and https versions. The fourth is changing lastmod every day without real content changes. The fifth is using robots.txt as an index removal tool. The reliable 2026 pattern is open crawl paths, submit only canonical URLs, and use console tools for diagnosis.

ملاحظة عملية

For small and mid-size websites, the most useful habit is consistency. The pages in sitemap.xml should also be reachable through internal links. New posts, key tools, category pages, and evergreen guides should reinforce one another instead of living as isolated URLs. بعد أي نشر يغير الروابط أو canonical أو مسارات اللغات أو XML التلقائي، افحص الملفين من جديد.

FAQ

هل يمنع robots.txt الفهرسة وحده؟

لا. هو يتحكم غالبا في الزحف. للحذف أو الخصوصية استخدم noindex أو الحماية أو أدوات الإزالة أو رموز الحالة المناسبة.

أين أضع سطر Sitemap؟

اكتب رابطا كاملا في نهاية robots.txt غالبا.

هل Allow يتفوق دائما؟

عادة يفوز المسار الأكثر تحديدا، لذلك اختبر الملف.

هل أضع كل الروابط؟

لا. ضع الروابط القانونية والقابلة للفهرسة فقط.

هل الإرسال يضمن الفهرسة؟

لا. يساعد الاكتشاف والتشخيص فقط.

هل أرسله إلى Naver؟

نعم إذا كانت حركة البحث الكورية مهمة.