The lexer, in charge of creating a list of tokens. More...

#include <Lexer.hpp>

Collaboration diagram for Ark::internal::Lexer:

Public Member Functions
	Lexer (unsigned debug) noexcept
	Construct a new Lexer object.

void	feed (const std::string &code)
	Give code to tokenize and create the list of tokens.

std::vector< Token > &	tokens () noexcept
	Return the list of tokens.

Private Member Functions
constexpr bool	isHexChar (char chr)

TokenType	guessType (const std::string &value) noexcept
	Helper function to determine the type of a token.

bool	isKeyword (const std::string &value) noexcept
	Check if the value is a keyword in ArkScript.

bool	isIdentifier (const std::string &value) noexcept
	Check if the value can be an identifier in ArkScript.

bool	isOperator (const std::string &value) noexcept
	Check if the value is an operator in ArkScript.

bool	endOfControlChar (const std::string &sequence, char next) noexcept
	Check if a control character / sequence is complete or not.

void	throwTokenizingError (const std::string &message, const std::string &match, std::size_t line, std::size_t col, const std::string &context)
	To throw nice lexer errors.

Private Attributes
unsigned	m_debug

std::vector< Token >	m_tokens

Detailed Description

The lexer, in charge of creating a list of tokens.

Definition at line 26 of file Lexer.hpp.

Constructor & Destructor Documentation

◆ Lexer()

Ark::internal::Lexer::Lexer ( unsigned debug )

explicitnoexcept

Construct a new Lexer object.

Parameters

debug the debug level

Definition at line 15 of file Lexer.cpp.

Member Function Documentation

◆ endOfControlChar()

bool Ark::internal::Lexer::endOfControlChar	(	const std::string &	sequence,
		char	next
	)

privatenoexcept

Check if a control character / sequence is complete or not.

Parameters

sequence	the sequence without the leading \
next	the next character to come, maybe, in the sequence

Returns: true; false

Definition at line 301 of file Lexer.cpp.

Referenced by feed().

◆ feed()

void Ark::internal::Lexer::feed ( const std::string & code )

Give code to tokenize and create the list of tokens.

Parameters

code	the ArkScript code

Definition at line 19 of file Lexer.cpp.

References Ark::internal::Capture, endOfControlChar(), Ark::internal::GetField, Ark::internal::Grouping, guessType(), m_debug, m_tokens, Ark::internal::Mismatch, Ark::internal::Shorthand, Ark::internal::String, throwTokenizingError(), and Ark::internal::tokentype_string.

Referenced by Ark::internal::Parser::feed().

◆ guessType()

TokenType Ark::internal::Lexer::guessType ( const std::string & value )

privatenoexcept

Helper function to determine the type of a token.

Parameters

value

Returns: TokenType

Definition at line 262 of file Lexer.cpp.

References Ark::internal::Capture, Ark::internal::GetField, Ark::internal::Identifier, Ark::Utils::isDouble(), Ark::internal::Keyword, Ark::internal::Mismatch, Ark::internal::Number, Ark::internal::Operator, and Ark::internal::Spread.

Referenced by feed().

◆ isHexChar()

constexpr bool Ark::internal::Lexer::isHexChar ( char chr )

inlineconstexprprivate

Definition at line 54 of file Lexer.hpp.

◆ isIdentifier()

bool Ark::internal::Lexer::isIdentifier ( const std::string & value )

privatenoexcept

Check if the value can be an identifier in ArkScript.

Parameters

value

Returns: true; false

Definition at line 291 of file Lexer.cpp.

◆ isKeyword()

bool Ark::internal::Lexer::isKeyword ( const std::string & value )

privatenoexcept

Check if the value is a keyword in ArkScript.

Parameters

value

Returns: true; false

Definition at line 286 of file Lexer.cpp.

References Ark::internal::keywords.

◆ isOperator()

bool Ark::internal::Lexer::isOperator ( const std::string & value )

privatenoexcept

Check if the value is an operator in ArkScript.

Parameters

value

Returns: true; false

Definition at line 296 of file Lexer.cpp.

References Ark::internal::operators.

◆ throwTokenizingError()

void Ark::internal::Lexer::throwTokenizingError	(	const std::string &	message,
		const std::string &	match,
		std::size_t	line,
		std::size_t	col,
		const std::string &	context
	)

private

To throw nice lexer errors.

Parameters

message
match
line
col
context

Definition at line 330 of file Lexer.cpp.

References Ark::internal::makeTokenBasedErrorCtx(), and Ark::Utils::splitString().

Referenced by feed().

◆ tokens()

std::vector< Token > & Ark::internal::Lexer::tokens ( )

noexcept

Return the list of tokens.

Returns: std::vector<Token>&

Definition at line 257 of file Lexer.cpp.

References m_tokens.

Referenced by Ark::internal::Parser::feed().

Member Data Documentation

◆ m_debug

unsigned Ark::internal::Lexer::m_debug

private

Definition at line 51 of file Lexer.hpp.

Referenced by feed().

◆ m_tokens

std::vector<Token> Ark::internal::Lexer::m_tokens

private

Definition at line 52 of file Lexer.hpp.

Referenced by feed(), and tokens().

The documentation for this class was generated from the following files:

/Users/fola/Documents/ArkScript/Ark/include/Ark/Compiler/AST/Lexer.hpp
/Users/fola/Documents/ArkScript/Ark/src/arkreactor/Compiler/AST/Lexer.cpp

Public Member Functions

Private Member Functions

Private Attributes

Detailed Description

Constructor & Destructor Documentation

◆ Lexer()

Member Function Documentation

◆ endOfControlChar()

◆ feed()

◆ guessType()

◆ isHexChar()

◆ isIdentifier()

◆ isKeyword()

◆ isOperator()

◆ throwTokenizingError()

◆ tokens()

Member Data Documentation

◆ m_debug

◆ m_tokens