http.agent.name WinistaBot Our HTTP 'User-Agent' request header. http.robots.agents WinistaBot,Winista,* The agent strings we'll look for in robots.txt files, comma-separated, in decreasing order of precedence. http.robots.403.allow true Some servers return HTTP status 403 (Forbidden) if /robots.txt doesn't exist. This should probably mean that we are allowed to crawl the site nonetheless. If this is set to false, then such sites will be treated as forbidden. http.robots.honor true This configuration setting can be used to bypass checks for robots.txt file checks. By default this property is set to true meaning that we always honor settings in robots.txt file for that site. http.agent.description Winista - Crwaler for HTTP content indexing Further description of our bot- this text is used in the User-Agent header. It appears in parenthesis after the agent name. http.agent.url http://www.netomatix.com/Winista/bot.html A URL to advertise in the User-Agent header. This will appear in parenthesis after the agent name. http.agent.email support@netomatix.com An email address to advertise in the HTTP 'From' request header and User-Agent header. http.agent.version 1.6.7-dev A version string to advertise in the User-Agent header. http.timeout 10000 The default network timeout, in milliseconds. http.max.delays 3 The number of times a thread will delay when trying to fetch a page. Each time it finds that a host is busy, it will wait fetcher.server.delay. After http.max.delays attepts, it will give up on the page for now. http.allow.https false This value can be set to allow or disallow crawling of secure sites. http.content.limit 128536 The length limit for downloaded content, in bytes. If this value is nonnegative (>=0), content longer than it will be truncated; otherwise, no truncation at all. http.proxy.host The proxy hostname. If empty, no proxy is used. http.proxy.port The proxy port. http.verbose false If true, HTTP will log more verbosely. http.redirect.max 3 The maximum number of redirects the fetcher will follow when trying to fetch a page. http.contenttype.check.strict false charset.user.defined.mapping ISO-8859-1 Code page to use when page has content-type tag set to x-user-defined db.default.fetch.interval 30 The default number of days between re-fetches of a page. db.ignore.internal.links true If true, when adding new links to a page, links from the same host are ignored. This is an effective way to limit the size of the link database, keeping the only the highest quality links. db.score.injected 1.0 The score of new pages added by the injector. db.score.link.external 1.0 The score factor for new pages added due to a link from another host relative to the referencing page's score. db.score.link.internal 1.0 The score factor for pages added due to a link from the same host, relative to the referencing page's score. db.max.outlinks.per.page 100 The maximum number of outlinks that we'll process for a page. db.max.anchor.length 100 The maximum number of characters permitted in an anchor. db.fetch.retry.max 3 The maximum number of times a url that has encountered recoverable errors is generated for fetch. fetchlist.score.by.link.count true If true, set page scores on fetchlist entries based on log(number of anchors), instead of using original page scores. This results in prioritization of pages with many incoming links. fetcher.server.delay 5.0 The number of seconds the fetcher will delay between successive requests to the same server. fetcher.threads.fetch 10 The number of FetcherThreads the fetcher should use. This is also determines the maximum number of requests that are made at once (each FetcherThread handles one connection). fetcher.threads.per.host 1 This number is the maximum number of threads that should be allowed to access a host at one time. fetcher.verbose false If true, fetcher will log more verbosely. parser.threads.parse 10 Number of ParserThreads ParseSegment should use. parser.character.encoding.default windows-1252 The character encoding to fall back to when no other information is available parser.html.impl Winista.HTMLParser HTML Parser implementation. query.url.boost 4.0 Used as a boost for url field in Lucene query. query.anchor.boost 2.0 Used as a boost for anchor field in Lucene query. query.title.boost 1.5 Used as a boost for title field in Lucene query. query.host.boost 2.0 Used as a boost for host field in Lucene query. query.phrase.boost 1.0 Used as a boost for phrase in Lucene query. Multiplied by boost for field phrase is matched in.