ForwardTokenizer Class Reference

#include <forwardTokenizer.h>

Inheritance diagram for ForwardTokenizer:

Collaboration diagram for ForwardTokenizer:

Public Member Functions
	ForwardTokenizer (std::istream &stream, const std::string blankspaces, const std::string separators)
virtual	~ForwardTokenizer ()
virtual int	countTokens ()
virtual bool	hasMoreTokens () const
virtual std::string	nextToken ()
virtual double	progress () const
Public Member Functions inherited from Tokenizer
	Tokenizer (std::istream &stream, const std::string blankspaces, const std::string separators)
virtual	~Tokenizer ()
void	blankspaceChars (const std::string)
std::string	blankspaceChars () const
void	separatorChars (const std::string)
std::string	separatorChars () const
void	lowercaseMode (const bool)
bool	lowercaseMode () const
std::string	streamToString () const

Additional Inherited Members
Protected Member Functions inherited from Tokenizer
bool	isBlankspace (const int character) const
bool	isSeparator (const int character) const
Protected Attributes inherited from Tokenizer
std::istream &	stream
std::ios::iostate	sstate
std::streamoff	offbeg
std::streamoff	offend
std::streamoff	offset

Detailed Description

Tokenizes a stream from the beginning to the end.

---

|_|_|_|_|_|_|_| | | | | | offend | offset offbeg

Definition at line 38 of file forwardTokenizer.h.

Constructor & Destructor Documentation

ForwardTokenizer()

ForwardTokenizer::ForwardTokenizer	(	std::istream &	stream,
		const std::string	blankspaces,
		const std::string	separators
	)

Definition at line 27 of file forwardTokenizer.cpp.

References Tokenizer::offbeg, and Tokenizer::offset.

~ForwardTokenizer()

ForwardTokenizer::~ForwardTokenizer ( )

virtual

Definition at line 36 of file forwardTokenizer.cpp.

Member Function Documentation

countTokens()

int ForwardTokenizer::countTokens ( )

virtual

Returns the number of tokens left.

Implements Tokenizer.

Definition at line 39 of file forwardTokenizer.cpp.

References hasMoreTokens(), nextToken(), Tokenizer::offbeg, Tokenizer::offset, and Tokenizer::stream.

Here is the call graph for this function:

hasMoreTokens()

bool ForwardTokenizer::hasMoreTokens ( ) const

virtual

Tests if there are more tokens.

Implements Tokenizer.

Definition at line 61 of file forwardTokenizer.cpp.

References Tokenizer::offend, and Tokenizer::offset.

Referenced by countTokens(), ContextTracker::learn(), and main().

Here is the caller graph for this function:

nextToken()

std::string ForwardTokenizer::nextToken ( )

virtual

Returns the next token.

Implements Tokenizer.

Definition at line 72 of file forwardTokenizer.cpp.

References endl(), Tokenizer::isBlankspace(), Tokenizer::isSeparator(), Tokenizer::lowercaseMode(), Tokenizer::offend, Tokenizer::offset, and Tokenizer::stream.

Referenced by countTokens(), ContextTracker::learn(), and main().

Here is the call graph for this function:

Here is the caller graph for this function:

progress()

double ForwardTokenizer::progress ( ) const

virtual

Returns progress percentage.

Implements Tokenizer.

Definition at line 151 of file forwardTokenizer.cpp.

References Tokenizer::offend, and Tokenizer::offset.

Referenced by main().

Here is the caller graph for this function:

---

The documentation for this class was generated from the following files:

• F:/msys64/home/MatteoV/git/presage/src/lib/core/tokenizer/forwardTokenizer.h
• F:/msys64/home/MatteoV/git/presage/src/lib/core/tokenizer/forwardTokenizer.cpp