Tokenizer (singa-simulation 0.7.0 API)

java.lang.Object
- uk.co.cogitolearning.cogpar.Tokenizer

```
public class Tokenizer
extends Object
```
A class for reading an input string and separating it into tokens that can be fed into Parser. The user can add regular expressions that will be matched against the front of the string. Regular expressions should not contain beginning-of-string or end-of-string anchors or any capturing groups as these will be added by the tokenizer itself.

Constructor Summary

Constructors
Constructor Description

Tokenizer()
Default constructor

Method Summary

All Methods Static Methods Instance Methods Concrete Methods
Modifier and Type	Method	Description
`void`	`add(String regex, int token)`	Add a regular expression and a token id to the internal list of recognized tokens
`static Tokenizer`	`getExpressionTokenizer()`	A static method that returns a tokenizer for mathematical expressions
`LinkedList<Token>`	`getTokens()`	Get the tokens generated in the last call to tokenize.
`void`	`tokenize(String string)`	Tokenize an input string.

Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

- Constructor Detail
  - Tokenizer
```
public Tokenizer()
```
    Default constructor
- Method Detail
  - getExpressionTokenizer
```
public static Tokenizer getExpressionTokenizer()
```
    A static method that returns a tokenizer for mathematical expressions
    
    Returns:
    
    a tokenizer that can handle mathematical expressions
  - add
```
public void add(String regex,
                int token)
```
    Add a regular expression and a token id to the internal list of recognized tokens
    
    Parameters:
    
    regex - the regular expression to match against
    
    token - the token id that the regular expression is linked to
  - tokenize
```
public void tokenize(String string)
```
    Tokenize an input string. The result of tokenizing can be accessed via getTokens
    
    Parameters:
    
    string - the string to tokenize
  - getTokens
```
public LinkedList<Token> getTokens()
```
    Get the tokens generated in the last call to tokenize.
    
    Returns:
    
    a list of tokens to be fed to Parser