User-agent: * Allow: /$ Disallow: / # Allow only root page to be crawled