RapidBot Support

RapidBot creates a robots.txt file that should not be read from others then Web crawlers.
That's why pushing the "Preview" button you'll get this message:

• "There's nothing to see here. This page is a controller for RapidBot and should be hidden from menu in the Inspector. A robots.txt file will be created on export. Please refer to RapidBot documentation for further details."

Hide the RapidBot page from the menu. Go to Page Inspector > General and un-check "Show in menu".

A robots.txt file restricts access to your site by search engine robots that crawl the web. These bots are automated, and before they access pages of a site, they check to see if a robots.txt file exists that prevents them from accessing certain pages. Though all respectable robots will respect the directives in a robots.txt file, some may interpret them differently.

You need a robots.txt file if your site includes content that you don't want search engines to index.

See Wikipedia robots.txt sample file for reference.

RapidBot is the incredibly easy-to-use RapidWeaver plugin to generate the robots.txt file for your site.

A robot (also known as Spider or Web Crawler) is a program that automatically traverses the Web's hypertext structure by retrieving a document and recursively retrieving all documents that are referenced.

You decide what spiders will visit in your site. Choose a crawler among our presets or a custom one. Then type a folder name or select it using default link picker. RapidBot is perfectly integrated with RapidWeaver!

Don't Get lost

Provide specific indexing rules thanks to RapidBot. Enable or disable files and folders crawling in a natural way: RapidBot will translate your directives into something that is understood by search engine bots.

Teach Bots how you want to be reached

Don't forget that instructing search engines bots help you adding visibility to your site and let people reach you in a more efficient way excluding irrelevant contents.
RapidBot is a must-have SEO tool! Only if search engines know what to do with your pages, they can give you a good ranking.

Discover features included in RapidBot!

Next:Requirements and installation

Previous:*What is RapidBot* ##RapidBot requirements * RapidBot requires RapidWeaver 6.x or higher and an Intel-based Mac. You can buy your RapidWeaver license from Realmac Software store (RapidWeaver 6.x requires Mac OS X 10.9 or greater, 50 MB disk space).

##RapidBot installation After downloading the RapidBot plugin from our website, follow the instructions on how to install the plugin in the General FAQ section.
RapidBot is a shareware: please test it before buy it and check if it suits your needs. When you decide to purchase a RapidBot license on omnidea store, follow the instruction on how to register in the General FAQ section.

Next:* Adding a RapidBot page*

Previous: RapidBot Requirement and Installation

Open your existing RapidWeaver Project or create a new one(1) by clicking the plus button on the left-bottom corner of RapidWeaver panel. Select RapidBot icon (2) among third party plugins and push "Choose" button (3).

RapidBot panel will show.

Next: Adding a rule

Previous: Adding a RapidBot Page

Robots.txt file is made of simple rules, made of one or more directive(s). Each directive can enable or disable access to specific content.

Add a new rule clicking on + button on the bottom left corner.

From right pane, you'll be able to specify:

Which bot(s) rule apply to.

* means every bot. You may want to set a custom user-agent or choose from the pre-defined list. A comprehensive list of known bots is available here.

Adding a directive to the rule:

Add a new directive clicking on + button on the bottom right corner.

Choose directive type:

Choose Allow or Disallow for content:

Choose a content filter:

All content means every file on site, or you can set more specify filters based on files (and folder) name and extensions.

• e.g. : Choose Folder / File and enter the Folder or File name in the following field. Pushing the ... will prompt the RapidWeaver standard link picker.
  

• e.g. : If you want spiders to not crawl your php configuration files, you can choose All content with extension and enter php in the following field. This will create a rule such Disallow (or Allow) /* .php$.

Comment:

Enter a text comment that will be included in robots.txt rule. Bots won't read this information.

Clicking on - button will remove the selected rule or directive.

Next: RapidBot settings

Previous: RapidBot GUI | Adding a rule

Enter the sitemap.xml URL of your site on this field if you want to include it in your Robots.txt file.

The sitemap URL should be an absolute path.

Pushing Online validation... opens an online tool to validate your robots.txt file.

Next: RapidBot settings | About

Previous: Page Options >General

This tab will let you know which version of RapidBot you're running and other license details (such as name and email address linked to the license).

If you like to stay up-to-date, push the "Check for updates" button. (Remember that RapidWeaver also checks for add-ons updates at start).

Clicking on RapidBot icon will land you to the official RapidBot page on omnidea site.