I have this since I don't want members pages, search, help, misc, login, register pages indexed from Google, and only threads and forums.
Robots.txt and sitemaps should always be in synch. In other words: if the xenforo sitemap sends a URL to search engines, then it should not be disallowed in ...
Old Hard to Find TV Series on DVD
I am getting robots blocked and redirection error messages too. When I checked for robots.txt file (ex: https://yourdomain.com/robots.txt ) then ...
Hello, I've wanted ask, what nowadays is recommended to setup robots.txt for XF2.2 installation. Any recommendations are very appreciated, ...
- A primary reason to use the robot text file is to prevent unwanted pages from appearing in search results. If you have a forum discussing ...
There is no default robots.txt file included in XenForo, but it's very easy to create one. Just upload it to the root of your installation.
I am thinking maybe my robots.txt is not good and blocking search engine from crawling my site. I would appreciate if someone can confirm to me ...
The allow command is not necessary in the Xenforo robots.txt file, as search engines are allowed to crawl unless you specify otherwise. The ...
Are there changes to XenForo's URL structure in the 2.0 version that warrant taking another look at your old robots.txt file?
All Squarespace sites use the same robots.txt file and as a Squarespace user you cannot access or edit it. Squarespace have already specified ...