RapidBot creates a robots.txt file that should not be read from others then Web crawlers.
That's why pushing the "Preview" button you'll get this message:
Hide the RapidBot page from the menu. Go to
General and un-check "
A robots.txt file restricts access to your site by search engine robots that crawl the web. These bots are automated, and before they access pages of a site, they check to see if a robots.txt file exists that prevents them from accessing certain pages. Though all respectable robots will respect the directives in a robots.txt file, some may interpret them differently.
You need a robots.txt file if your site includes content that you don't want search engines to index.
See Wikipedia robots.txt sample file for reference.
RapidBot is the incredibly easy-to-use RapidWeaver plugin to generate the robots.txt file for your site.
A robot (also known as Spider or Web Crawler) is a program that automatically traverses the Web's hypertext structure by retrieving a document and recursively retrieving all documents that are referenced.
You decide what spiders will visit in your site. Choose a crawler among our presets or a custom one. Then type a folder name or select it using default link picker. RapidBot is perfectly integrated with RapidWeaver!
Provide specific indexing rules thanks to RapidBot. Enable or disable files and folders crawling in a natural way: RapidBot will translate your directives into something that is understood by search engine bots.
Don't forget that instructing search engines bots help you adding visibility to your site and let people reach you in a more efficient way excluding irrelevant contents.
RapidBot is a must-have SEO tool! Only if search engines know what to do with your pages, they can give you a good ranking.
Discover features included in RapidBot!
Previous:*What is RapidBot* ##RapidBot requirements * RapidBot requires RapidWeaver 6.x or higher and an Intel-based Mac. You can buy your RapidWeaver license from Realmac Software store (RapidWeaver 6.x requires Mac OS X 10.9 or greater, 50 MB disk space).
##RapidBot installation After downloading the RapidBot plugin from our website, follow the instructions on
how to install the plugin in the General FAQ section.
RapidBot is a shareware: please test it before buy it and check if it suits your needs. When you decide to purchase a RapidBot license on omnidea store, follow the instruction on how to register in the General FAQ section.
Open your existing RapidWeaver Project or create a new one(1) by clicking the plus button on the left-bottom corner of RapidWeaver panel. Select RapidBot icon (2) among third party plugins and push "Choose" button (3).
RapidBot panel will show.
Robots.txt file is made of simple rules, made of one or more directive(s). Each directive can enable or disable access to specific content.
Add a new rule clicking on
+ button on the bottom left corner.
From right pane, you'll be able to specify:
* means every bot. You may want to set a custom user-agent or choose from the pre-defined list. A comprehensive list of known bots is available here.
Add a new directive clicking on
+ button on the bottom right corner.
Choose directive type:
Disallow for content:
Choose a content filter:
All content means every file on site, or you can set more specify filters based on files (and folder) name and extensions.
Folder / Fileand enter the Folder or File name in the following field. Pushing the
...will prompt the RapidWeaver standard link picker.
All content with extensionand enter
phpin the following field. This will create a rule such
Disallow (or Allow) /* .php$.
Enter a text comment that will be included in robots.txt rule. Bots won't read this information.
- button will remove the selected rule or directive.
Enter the sitemap.xml URL of your site on this field if you want to include it in your Robots.txt file.
The sitemap URL should be an absolute path.
Online validation... opens an online tool to validate your robots.txt file.
This tab will let you know which version of RapidBot you're running and other license details (such as name and email address linked to the license).
If you like to stay up-to-date, push the "Check for updates" button. (Remember that RapidWeaver also checks for add-ons updates at start).
Clicking on RapidBot icon will land you to the official RapidBot page on omnidea site.