Tokenizer Class Reference

#include <tokenizer.h>

Inheritance diagram for Tokenizer:

Collaboration diagram for Tokenizer:

Classes
class	StreamGuard

Public Member Functions
	Tokenizer (std::istream &stream, const std::string blankspaces, const std::string separators)
virtual	~Tokenizer ()
virtual int	countTokens ()=0
virtual bool	hasMoreTokens () const =0
virtual std::string	nextToken ()=0
virtual double	progress () const =0
void	blankspaceChars (const std::string)
std::string	blankspaceChars () const
void	separatorChars (const std::string)
std::string	separatorChars () const
void	lowercaseMode (const bool)
bool	lowercaseMode () const
std::string	streamToString () const

Protected Member Functions
bool	isBlankspace (const int character) const
bool	isSeparator (const int character) const

Protected Attributes
std::istream &	stream
std::ios::iostate	sstate
std::streamoff	offbeg
std::streamoff	offend
std::streamoff	offset

Private Attributes
std::string	blankspaces
std::string	separators
bool	lowercase

Detailed Description

The Tokenizer class takes an input stream and parses it into "tokens", allowing the tokens to be read one at a time.

The parsing process is controlled by the character classification sets:

• blankspace characters: characters that mark a token boundary and are not part of the token.
• separator characters: characters that mark a token boundary and might be considered tokens, depending on the value of a flag (to be implemented).
• valid characters: any non blankspace and non separator character.

Each byte read from the input stream is regarded as a character in the range '\u0000' through '\u00FF'.

In addition, an instance has flags that control:

• whether the characters of tokens are converted to lowercase.
• whether separator characters constitute tokens. (TBD)

A typical application first constructs an instance of this class, supplying the input stream to be tokenized, the set of blankspaces, and the set of separators, and then repeatedly loops, while method hasMoreTokens returns true, calling the nextToken method.

Definition at line 64 of file tokenizer.h.

Constructor & Destructor Documentation

Tokenizer()

Tokenizer::Tokenizer	(	std::istream &	stream,
		const std::string	blankspaces,
		const std::string	separators
	)

Definition at line 27 of file tokenizer.cpp.

References blankspaceChars(), blankspaces, offbeg, offend, offset, separatorChars(), separators, sstate, and stream.

Here is the call graph for this function:

~Tokenizer()

Tokenizer::~Tokenizer ( )

virtual

Definition at line 53 of file tokenizer.cpp.

References sstate, and stream.

Member Function Documentation

blankspaceChars() [1/2]

void Tokenizer::blankspaceChars

(

const std::string

chars

)

Sets blankspace characters.

Definition at line 61 of file tokenizer.cpp.

References blankspaces.

blankspaceChars() [2/2]

std::string Tokenizer::blankspaceChars

(

)

const

Gets blankspace characters.

Definition at line 66 of file tokenizer.cpp.

References blankspaces.

Referenced by Tokenizer().

Here is the caller graph for this function:

countTokens()

virtual int Tokenizer::countTokens ( )

pure virtual

Returns the number of tokens left.

Implemented in ForwardTokenizer, and ReverseTokenizer.

hasMoreTokens()

virtual bool Tokenizer::hasMoreTokens ( ) const

pure virtual

Tests if there are more tokens.

Implemented in ForwardTokenizer, and ReverseTokenizer.

isBlankspace()

bool Tokenizer::isBlankspace ( const int  character ) const

protected

Definition at line 91 of file tokenizer.cpp.

References blankspaces.

Referenced by ForwardTokenizer::nextToken(), and ReverseTokenizer::nextToken().

Here is the caller graph for this function:

isSeparator()

bool Tokenizer::isSeparator ( const int  character ) const

protected

Definition at line 101 of file tokenizer.cpp.

References separators.

Referenced by ForwardTokenizer::nextToken(), and ReverseTokenizer::nextToken().

Here is the caller graph for this function:

lowercaseMode() [1/2]

void Tokenizer::lowercaseMode

(

const bool

value

)

Sets lowercase mode.

Definition at line 81 of file tokenizer.cpp.

References lowercase.

Referenced by ContextChangeDetector::change(), ContextTracker::learn(), and main().

Here is the caller graph for this function:

lowercaseMode() [2/2]

bool Tokenizer::lowercaseMode

(

)

const

Gets lowercase mode.

Definition at line 86 of file tokenizer.cpp.

References lowercase.

Referenced by ForwardTokenizer::nextToken(), and ReverseTokenizer::nextToken().

Here is the caller graph for this function:

nextToken()

virtual std::string Tokenizer::nextToken ( )

pure virtual

Returns the next token.

Implemented in ForwardTokenizer, and ReverseTokenizer.

progress()

virtual double Tokenizer::progress ( ) const

pure virtual

Returns progress percentage.

Implemented in ForwardTokenizer, and ReverseTokenizer.

separatorChars() [1/2]

void Tokenizer::separatorChars

(

const std::string

chars

)

Sets separator characters.

Definition at line 71 of file tokenizer.cpp.

References separators.

separatorChars() [2/2]

std::string Tokenizer::separatorChars

(

)

const

Gets separator characters.

Definition at line 76 of file tokenizer.cpp.

References separators.

Referenced by Tokenizer().

Here is the caller graph for this function:

streamToString()

std::string Tokenizer::streamToString ( ) const

inline

Definition at line 109 of file tokenizer.h.

References offbeg, offend, and stream.

Member Data Documentation

blankspaces

std::string Tokenizer::blankspaces

private

Definition at line 154 of file tokenizer.h.

Referenced by blankspaceChars(), isBlankspace(), and Tokenizer().

lowercase

bool Tokenizer::lowercase

private

Definition at line 157 of file tokenizer.h.

Referenced by lowercaseMode().

offbeg

std::streamoff Tokenizer::offbeg

protected

Definition at line 146 of file tokenizer.h.

Referenced by ForwardTokenizer::countTokens(), ForwardTokenizer::ForwardTokenizer(), ReverseTokenizer::hasMoreTokens(), ReverseTokenizer::nextToken(), ReverseTokenizer::progress(), streamToString(), and Tokenizer().

offend

std::streamoff Tokenizer::offend

protected

Definition at line 147 of file tokenizer.h.

Referenced by ReverseTokenizer::countTokens(), ForwardTokenizer::hasMoreTokens(), ForwardTokenizer::nextToken(), ReverseTokenizer::nextToken(), ForwardTokenizer::progress(), ReverseTokenizer::progress(), ReverseTokenizer::ReverseTokenizer(), streamToString(), and Tokenizer().

offset

std::streamoff Tokenizer::offset

protected

Definition at line 148 of file tokenizer.h.

Referenced by ForwardTokenizer::countTokens(), ReverseTokenizer::countTokens(), ForwardTokenizer::ForwardTokenizer(), ReverseTokenizer::hasMoreTokens(), ForwardTokenizer::hasMoreTokens(), ReverseTokenizer::nextToken(), ForwardTokenizer::nextToken(), ForwardTokenizer::progress(), ReverseTokenizer::progress(), ReverseTokenizer::ReverseTokenizer(), and Tokenizer().

separators

std::string Tokenizer::separators

private

Definition at line 155 of file tokenizer.h.

Referenced by isSeparator(), separatorChars(), and Tokenizer().

sstate

std::ios::iostate Tokenizer::sstate

protected

Definition at line 145 of file tokenizer.h.

Referenced by Tokenizer(), and ~Tokenizer().

stream

std::istream& Tokenizer::stream

protected

Definition at line 144 of file tokenizer.h.

Referenced by ForwardTokenizer::countTokens(), ReverseTokenizer::countTokens(), ForwardTokenizer::nextToken(), ReverseTokenizer::nextToken(), ReverseTokenizer::ReverseTokenizer(), streamToString(), Tokenizer(), and ~Tokenizer().

---

The documentation for this class was generated from the following files:

• F:/msys64/home/MatteoV/git/presage/src/lib/core/tokenizer/tokenizer.h
• F:/msys64/home/MatteoV/git/presage/src/lib/core/tokenizer/tokenizer.cpp