RapidBot creates a robots.txt file that should not be read from others then Web crawlers.
That's why pushing the "Preview" button you'll get this message:
Hide the RapidBot page from the menu. Go to Page
Inspector
> General
and un-check "Show
in menu
".
A robots.txt file restricts access to your site by search engine robots that crawl the web. These bots are automated, and before they access pages of a site, they check to see if a robots.txt file exists that prevents them from accessing certain pages. Though all respectable robots will respect the directives in a robots.txt file, some may interpret them differently.
You need a robots.txt file if your site includes content that you don't want search engines to index.
See Wikipedia robots.txt sample file for reference.
RapidBot is the incredibly easy-to-use RapidWeaver plugin to generate the robots.txt file for your site.
A robot (also known as Spider or Web Crawler) is a program that automatically traverses the Web's hypertext structure by retrieving a document and recursively retrieving all documents that are referenced.
You decide what spiders will visit in your site. Choose a crawler among our presets or a custom one. Then type a folder name or select it using default link picker. RapidBot is perfectly integrated with RapidWeaver!
Provide specific indexing rules thanks to RapidBot. Enable or disable files and folders crawling in a natural way: RapidBot will translate your directives into something that is understood by search engine bots.
Don't forget that instructing search engines bots help you adding visibility to your site and let people reach you in a more efficient way excluding irrelevant contents.
RapidBot is a must-have SEO tool! Only if search engines know what to do with your pages, they can give you a good ranking.
Discover features included in RapidBot!
Previous:*What is RapidBot* ##RapidBot requirements * RapidBot requires RapidWeaver 6.x or higher and an Intel-based Mac. You can buy your RapidWeaver license from Realmac Software store (RapidWeaver 6.x requires Mac OS X 10.9 or greater, 50
MB disk space).
##RapidBot installation After downloading the RapidBot plugin from our website, follow the instructions on
how to install the plugin in the General FAQ section.
RapidBot is a shareware: please test it before buy it and check if it suits your needs. When you decide to purchase a RapidBot license on omnidea
store, follow the instruction on
how to register in the General FAQ section.
Previous: RapidBot Requirement and Installation
Open your existing RapidWeaver Project or create a new one(1) by clicking the plus button on the left-bottom corner of RapidWeaver panel. Select RapidBot icon (2) among third party plugins and push "Choose" button (3).
RapidBot panel will show.
Previous: Adding a RapidBot Page
Robots.txt file is made of simple rules, made of one or more directive(s). Each directive can enable or disable access to specific content.
Add a new rule clicking on +
button on the bottom left corner.
From right pane, you'll be able to specify:
*
means every bot. You may want to set a custom user-agent or choose from the pre-defined list. A comprehensive list of known bots is available here.
Add a new directive clicking on +
button on the bottom right corner.
Choose directive type:
Choose Allow
or Disallow
for content:
Choose a content filter:
All content
means every file on site, or you can set more specify filters based on files (and folder) name and extensions.
Folder / File
and enter the Folder or File name in the following field. Pushing the
...
will prompt the RapidWeaver standard link picker.
All content
with extension
and enter php
in the following field. This will create a rule such Disallow (or Allow) /*
.php$
.Enter a text comment that will be included in robots.txt rule. Bots won't read this information.
Clicking on -
button will remove the selected rule or directive.
Previous: RapidBot GUI | Adding a rule
Enter the sitemap.xml URL of your site on this field if you want to include it in your Robots.txt file.
The sitemap URL should be an absolute path.
Pushing Online validation...
opens an online tool to validate your robots.txt file.
Previous: Page Options >General
This tab will let you know which version of RapidBot you're running and other license details (such as name and email address linked to the license).
If you like to stay up-to-date, push the "Check for updates" button. (Remember that RapidWeaver also checks for add-ons updates at start).
Clicking on RapidBot icon will land you to the official RapidBot page on omnidea site.