Create robots.txt files to control search engine crawlers. Set allow/disallow rules and sitemap locations.

User-agent: *
Allow: /