FAQ: Discarding hits from spiders

How can I throw away all the spider hits, so I only see statistics on non-spider hits?

Short Answer

Use a Log Filter to reject all hits from spiders (and worms).

Long Answer

Create a new Filter to reject all hits from spiders. The easiest way to create log filters is in the Log Filter Editor, in the Log Filters section of the Config. To get to the Log Filters editor, click Show Config in the Profiles list (or click Config in the reports), then click Log Data down the left, then click Log Filters. To create the filter:

You can also use the Advanced Expression Syntax option from the Filter Type drop down list (on the Filter tab), and type in this filter expression into the value field:

if (spider ne "(not a spider)") then "reject";

For log format which use the User Agent Analysis snapon to compute spider values, the spider field is called uaa_spider, and the expression above must be changed to refer to that field name instead.

Then rebuild your database, and all hits from spiders will be discarded.

For more details on Filters see Using Log Filters.