Class Lexer

java.lang.Object

Lexer

public class Lexer
extends java.lang.Object

The Lexer has several different versions of a parsing method for expressions.

The first method is called scannerLexer, and is based on using a Scanner.
The second method is called splitLexer, which uses String.split(String).
The third method is call tokenizerLexer and is based on an object called StringTokenizer.

created by Chris Wilcox Spring17
modified by rbecwar and garethhalladay Fall17

Constructor Summary

Constructors

Constructor and Description
Lexer()

Method Summary

Modifier and Type	Method and Description
static java.util.List<java.lang.String>	scannerLexer(java.lang.String expr) Lex a String using an instance of Scanner.
static java.util.List<java.lang.String>	splitLexer(java.lang.String expr) Parse a String using String.split(String).

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

Lexer

public Lexer()

Method Detail

scannerLexer

public static java.util.List<java.lang.String> scannerLexer(java.lang.String expr)

Lex a String using an instance of Scanner.

Initialize a Scanner object with a String argument containing the expression. Using the Scanner.hasNext() and Scanner.next() methods:

1. Parse the String (using whitespace as your delimiter)
2. Add each element to the ArrayList

After implementing this approach, examine the results.

This approach seems to have trouble unless there is white space between every token. Is this fixable or is the Scanner approach not a good idea?

Parameters:

expr - a String in the form of an expression

Returns:

a list of tokens

splitLexer

public static java.util.List<java.lang.String> splitLexer(java.lang.String expr)

Parse a String using String.split(String).

Invoke the String.split(String) method on the String containing the expression. Remember that split returns an array of strings that you must iterate. Note that the method takes a regular expression, which we have not studied yet. As an introduction, you should check out the provided links and try to understand what the following expressions are matching.

Try passing the regular expression "[-+*()/]" the split method to see what happens.
It seems to match everything, but it removes the operators and parentheses.
Is the String.split(String) approach fixable?

Hint: Try this regular expression: "(?<=[-+*()/])|(?=[-+*()/])". This uses the lookahead and lookbehind operators, an advanced regex feature. Use the String.trim() method to remove whitespace in each token.

Parameters:

expr - a string in the form of an expression

Returns:

a list of tokens

See Also:

String.split(String)