Package com.crawljax.core

Class CandidateElementExtractor

java.lang.Object

com.crawljax.core.CandidateElementExtractor

public class CandidateElementExtractor
extends Object

This class extracts candidate elements from the DOM tree, based on the tags provided by the user. Elements can also be excluded.

Field Summary

Fields

Modifier and Type	Field	Description
private EmbeddedBrowser	browser
private ExtractorManager	checkedElements
private boolean	clickOnce
private boolean	crawlFrames
private com.google.common.collect.ImmutableMultimap<String,CrawlElement>	excludeCrawlElements
private boolean	followExternalLinks
private FormHandler	formHandler
private com.google.common.collect.ImmutableSortedSet<String>	ignoredFrameIdentifiers
private com.google.common.collect.ImmutableList<CrawlElement>	includedCrawlElements
private static org.slf4j.Logger	LOG
private boolean	randomizeElementsOrder
private String	siteHostName

Constructor Summary

Constructors

Constructor	Description
CandidateElementExtractor(ExtractorManager checker, EmbeddedBrowser browser, FormHandler formHandler, CrawljaxConfiguration config)	Create a new CandidateElementExtractor.

Method Summary

Modifier and Type	Method	Description
private void	addElement(Element element, com.google.common.collect.ImmutableList.Builder<Element> builder, CrawlElement crawlElement)
private void	addFramesCandidates(Document dom, List<CandidateElement> results, String relatedFrame, NodeList frameNodes)
private com.google.common.collect.ImmutableMultimap<String,CrawlElement>	asMultiMap(com.google.common.collect.ImmutableList<CrawlElement> elements)
boolean	checkCrawlCondition()
private boolean	elementMatchesXpath(EventableConditionChecker eventableConditionChecker, EventableCondition eventableCondition, com.google.common.collect.ImmutableList<String> expressions, Element element)
private void	evaluateElement(List<CandidateElement> results, String relatedFrame, CrawlElement crawl, Element sourceElement)
private void	evaluateElements(Document dom, CrawlElement crawl, List<CandidateElement> results, String relatedFrame)
com.google.common.collect.ImmutableList<CandidateElement>	extract(StateVertex currentState)	This method extracts candidate elements from the current DOM tree in the browser, based on the crawl tags defined by the user.
(package private) void	extractElements(Document dom, List<CandidateElement> results, String relatedFrame)
private com.google.common.collect.ImmutableList<String>	getFullXpathForGivenXpath(Document dom, EventableCondition eventableCondition)
private com.google.common.collect.ImmutableList<Element>	getNodeListForTagElement(Document dom, CrawlElement crawlElement, EventableConditionChecker eventableConditionChecker)	Returns a list of Elements form the DOM tree, matching the tag element.
(package private) boolean	hrefShouldBeIgnored(Element element)
private boolean	isExcluded(Document dom, Element element, EventableConditionChecker eventableConditionChecker)
private boolean	isExternal(String href)
private boolean	isFileForDownloading(String href)
private boolean	isFrameIgnored(String string)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

LOG

private static final org.slf4j.Logger LOG

checkedElements

private final ExtractorManager checkedElements

browser

private final EmbeddedBrowser browser

formHandler

private final FormHandler formHandler

crawlFrames

private final boolean crawlFrames

excludeCrawlElements

private final com.google.common.collect.ImmutableMultimap<String,CrawlElement> excludeCrawlElements

includedCrawlElements

private final com.google.common.collect.ImmutableList<CrawlElement> includedCrawlElements

clickOnce

private final boolean clickOnce

randomizeElementsOrder

private final boolean randomizeElementsOrder

ignoredFrameIdentifiers

private final com.google.common.collect.ImmutableSortedSet<String> ignoredFrameIdentifiers

followExternalLinks

private final boolean followExternalLinks

siteHostName

private final String siteHostName

Constructor Detail

CandidateElementExtractor

@Inject
public CandidateElementExtractor(ExtractorManager checker,
                                 EmbeddedBrowser browser,
                                 FormHandler formHandler,
                                 CrawljaxConfiguration config)

Create a new CandidateElementExtractor.

Parameters:

checker - the ExtractorManager to use for marking handled elements and retrieve the EventableConditionChecker

browser - the current browser instance used in the Crawler

formHandler - the form handler.

config - the checker used to determine if a certain frame must be ignored.

Method Detail

asMultiMap

private com.google.common.collect.ImmutableMultimap<String,CrawlElement> asMultiMap(com.google.common.collect.ImmutableList<CrawlElement> elements)

extract

public com.google.common.collect.ImmutableList<CandidateElement> extract(StateVertex currentState)
                                                                  throws CrawljaxException

This method extracts candidate elements from the current DOM tree in the browser, based on the crawl tags defined by the user.

Parameters:

currentState - the state in which this extract method is requested.

Returns:

a list of candidate elements that are not excluded.

Throws:

CrawljaxException - if the method fails.

extractElements

void extractElements(Document dom,
                     List<CandidateElement> results,
                     String relatedFrame)

addFramesCandidates

private void addFramesCandidates(Document dom,
                                 List<CandidateElement> results,
                                 String relatedFrame,
                                 NodeList frameNodes)

isFrameIgnored

private boolean isFrameIgnored(String string)

evaluateElements

private void evaluateElements(Document dom,
                              CrawlElement crawl,
                              List<CandidateElement> results,
                              String relatedFrame)

getNodeListForTagElement

private com.google.common.collect.ImmutableList<Element> getNodeListForTagElement(Document dom,
                                                                                  CrawlElement crawlElement,
                                                                                  EventableConditionChecker eventableConditionChecker)

Returns a list of Elements form the DOM tree, matching the tag element.

elementMatchesXpath

private boolean elementMatchesXpath(EventableConditionChecker eventableConditionChecker,
                                    EventableCondition eventableCondition,
                                    com.google.common.collect.ImmutableList<String> expressions,
                                    Element element)

getFullXpathForGivenXpath

private com.google.common.collect.ImmutableList<String> getFullXpathForGivenXpath(Document dom,
                                                                                  EventableCondition eventableCondition)

addElement

private void addElement(Element element,
                        com.google.common.collect.ImmutableList.Builder<Element> builder,
                        CrawlElement crawlElement)

hrefShouldBeIgnored

boolean hrefShouldBeIgnored(Element element)

isExternal

private boolean isExternal(String href)

isFileForDownloading

private boolean isFileForDownloading(String href)

Parameters:

href - the string to check

Returns:

true if href has the pdf or ps pattern.

evaluateElement

private void evaluateElement(List<CandidateElement> results,
                             String relatedFrame,
                             CrawlElement crawl,
                             Element sourceElement)

isExcluded

private boolean isExcluded(Document dom,
                           Element element,
                           EventableConditionChecker eventableConditionChecker)

Returns:

true if element should be excluded. Also when an ancestor of the given element is marked for exclusion, which allows for recursive exclusion of elements from candidates.

checkCrawlCondition

public boolean checkCrawlCondition()