Package com.crawljax.core.configuration

Class CrawljaxConfiguration.CrawljaxConfigurationBuilder

java.lang.Object

com.crawljax.core.configuration.CrawljaxConfiguration.CrawljaxConfigurationBuilder

Enclosing class:

CrawljaxConfiguration

public static class CrawljaxConfiguration.CrawljaxConfigurationBuilder
extends Object

Field Summary

Fields

Modifier and Type	Field	Description
private CrawljaxConfiguration	config
private CrawlRules.CrawlRulesBuilder	crawlRules
private com.google.common.collect.ImmutableList.Builder<Plugin>	pluginBuilder

Constructor Summary

Constructors

Modifier	Constructor	Description
private	CrawljaxConfigurationBuilder(URI url)

Method Summary

Modifier and Type	Method	Description
CrawljaxConfiguration.CrawljaxConfigurationBuilder	addPlugin(Plugin... plugins)	Add plugins to Crawljax.
CrawljaxConfiguration	build()
private void	checkOutputDirWritable()
CrawlRules.CrawlRulesBuilder	crawlRules()
CrawljaxConfiguration.CrawljaxConfigurationBuilder	setBasicAuth(String username, String password)	If the website uses Basic auth you can set the username and password here.
CrawljaxConfiguration.CrawljaxConfigurationBuilder	setBrowserConfig(BrowserConfiguration configuration)
CrawljaxConfiguration.CrawljaxConfigurationBuilder	setBrowserConfig(BrowserConfiguration configuration, BrowserOptions options)
CrawljaxConfiguration.CrawljaxConfigurationBuilder	setCrawlScope(CrawlScope crawlScope)	Sets the crawl scope.
CrawljaxConfiguration.CrawljaxConfigurationBuilder	setMaximumDepth(int depth)
CrawljaxConfiguration.CrawljaxConfigurationBuilder	setMaximumRunTime(long time, TimeUnit unit)
CrawljaxConfiguration.CrawljaxConfigurationBuilder	setMaximumStates(int states)
CrawljaxConfiguration.CrawljaxConfigurationBuilder	setOutputDirectory(File output)	Set the output folder for any Plugin you might configure.
CrawljaxConfiguration.CrawljaxConfigurationBuilder	setProxyConfig(ProxyConfiguration configuration)
CrawljaxConfiguration.CrawljaxConfigurationBuilder	setStateVertexFactory(StateVertexFactory vertexFactory)	Set a custom StateVertexFactory to be able to use your own StateVertex objects.
CrawljaxConfiguration.CrawljaxConfigurationBuilder	setUnlimitedCrawlDepth()	Set the crawl depth to unlimited.
CrawljaxConfiguration.CrawljaxConfigurationBuilder	setUnlimitedRuntime()	Set the maximum runtime to unlimited.
CrawljaxConfiguration.CrawljaxConfigurationBuilder	setUnlimitedStates()	Crawl without a maximum state limit.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

pluginBuilder

private final com.google.common.collect.ImmutableList.Builder<Plugin> pluginBuilder

config

private final CrawljaxConfiguration config

crawlRules

private final CrawlRules.CrawlRulesBuilder crawlRules

Constructor Detail

CrawljaxConfigurationBuilder

private CrawljaxConfigurationBuilder(URI url)

Method Detail

setBasicAuth

public CrawljaxConfiguration.CrawljaxConfigurationBuilder setBasicAuth(String username,
                                                                       String password)

If the website uses Basic auth you can set the username and password here.

Parameters:

username - The username for the website.

password - The password for the website.

Returns:

CrawljaxConfiguration.CrawljaxConfigurationBuilder for method chaining.

setCrawlScope

public CrawljaxConfiguration.CrawljaxConfigurationBuilder setCrawlScope(CrawlScope crawlScope)

Sets the crawl scope.

If null, then a DefaultCrawlScope is used.

Parameters:

crawlScope - the crawl scope

Returns:

this CrawljaxConfigurationBuilder for method chaining.

Since:

5.0

setMaximumStates

public CrawljaxConfiguration.CrawljaxConfigurationBuilder setMaximumStates(int states)

Parameters:

states - The maximum number of states the Crawler should crawl. The default is unlimited.

setUnlimitedStates

public CrawljaxConfiguration.CrawljaxConfigurationBuilder setUnlimitedStates()

Crawl without a maximum state limit.

setMaximumRunTime

public CrawljaxConfiguration.CrawljaxConfigurationBuilder setMaximumRunTime(long time,
                                                                            TimeUnit unit)

Parameters:

time - The maximum time the crawler should run. Default is one hour.

setUnlimitedRuntime

public CrawljaxConfiguration.CrawljaxConfigurationBuilder setUnlimitedRuntime()

Set the maximum runtime to unlimited.

setMaximumDepth

public CrawljaxConfiguration.CrawljaxConfigurationBuilder setMaximumDepth(int depth)

Parameters:

depth - The maximum depth the crawler can reach. The default is 2.

setUnlimitedCrawlDepth

public CrawljaxConfiguration.CrawljaxConfigurationBuilder setUnlimitedCrawlDepth()

Set the crawl depth to unlimited. The default depth is 2.

addPlugin

public CrawljaxConfiguration.CrawljaxConfigurationBuilder addPlugin(Plugin... plugins)

Add plugins to Crawljax. Note that without plugins, Crawljax won't give any output. For basic output at least enable the CrawlOverviewPlugin.

You can call this method several times to add multiple plugins

Parameters:

plugins - the plugins you would like to enable.

setProxyConfig

public CrawljaxConfiguration.CrawljaxConfigurationBuilder setProxyConfig(ProxyConfiguration configuration)

Parameters:

configuration - The proxy configuration. Default is ProxyConfiguration.noProxy()

crawlRules

public CrawlRules.CrawlRulesBuilder crawlRules()

Returns:

The CrawlRules.CrawlRulesBuilder to define crawling rules.

setBrowserConfig

public CrawljaxConfiguration.CrawljaxConfigurationBuilder setBrowserConfig(BrowserConfiguration configuration)

Parameters:

configuration - a custom BrowserConfiguration. The default is a single EmbeddedBrowser.BrowserType.FIREFOX browser.

setBrowserConfig

public CrawljaxConfiguration.CrawljaxConfigurationBuilder setBrowserConfig(BrowserConfiguration configuration,
                                                                           BrowserOptions options)

Parameters:

configuration - a custom BrowserConfiguration. The default is a single EmbeddedBrowser.BrowserType.FIREFOX browser.

setStateVertexFactory

public CrawljaxConfiguration.CrawljaxConfigurationBuilder setStateVertexFactory(StateVertexFactory vertexFactory)

Set a custom StateVertexFactory to be able to use your own StateVertex objects. This is useful when you want to have a custom comparator in the state-flow graph which relies on the Object.hashCode() or Object.equals(Object) of the StateVertex.

Parameters:

vertexFactory - The factory you want to use.

Returns:

The builder for method chaining.

setOutputDirectory

public CrawljaxConfiguration.CrawljaxConfigurationBuilder setOutputDirectory(File output)

Set the output folder for any Plugin you might configure. Crawljax itself doesn't need an output folder but many plug-ins do.

Parameters:

output - The output folder. If it does not exist it will be created.

Throws:

IllegalStateException - if the specified file is not writable or exists but isn't a folder.

checkOutputDirWritable

private void checkOutputDirWritable()

build

public CrawljaxConfiguration build()