Documentation

Lexer
in package

Table of Contents

$code  : mixed
$dropTokens  : mixed
$filePos  : mixed
$identifierTokens  : mixed
$line  : mixed
$pos  : mixed
$prevCloseTagHasNewline  : mixed
$tokenMap  : mixed
$tokens  : mixed
$attributeCommentsUsed  : mixed
$attributeEndFilePosUsed  : mixed
$attributeEndLineUsed  : mixed
$attributeEndTokenPosUsed  : mixed
$attributeStartFilePosUsed  : mixed
$attributeStartLineUsed  : mixed
$attributeStartTokenPosUsed  : mixed
__construct()  : mixed
Creates a Lexer.
getNextToken()  : int
Fetches the next token.
getTokens()  : array<string|int, mixed>
Returns the token array for current code.
handleHaltCompiler()  : string
Handles __halt_compiler() by returning the text after it.
startLexing()  : mixed
Initializes the lexer for lexing the provided source code.
createTokenMap()  : array<string|int, mixed>
Creates the token map.
postprocessTokens()  : mixed
createIdentifierTokenMap()  : array<string|int, mixed>
defineCompatibilityTokens()  : mixed
handleInvalidCharacterRange()  : mixed
isUnterminatedComment()  : bool
Check whether comment token is unterminated.

Properties

$dropTokens

protected mixed $dropTokens

$filePos

protected mixed $filePos

$identifierTokens

protected mixed $identifierTokens

$prevCloseTagHasNewline

protected mixed $prevCloseTagHasNewline

$tokenMap

protected mixed $tokenMap

$tokens

protected mixed $tokens

$attributeCommentsUsed

private mixed $attributeCommentsUsed

$attributeEndFilePosUsed

private mixed $attributeEndFilePosUsed

$attributeEndLineUsed

private mixed $attributeEndLineUsed

$attributeEndTokenPosUsed

private mixed $attributeEndTokenPosUsed

$attributeStartFilePosUsed

private mixed $attributeStartFilePosUsed

$attributeStartLineUsed

private mixed $attributeStartLineUsed

$attributeStartTokenPosUsed

private mixed $attributeStartTokenPosUsed

Methods

__construct()

Creates a Lexer.

public __construct([array<string|int, mixed> $options = [] ]) : mixed
Parameters
$options : array<string|int, mixed> = []

Options array. Currently only the 'usedAttributes' option is supported, which is an array of attributes to add to the AST nodes. Possible attributes are: 'comments', 'startLine', 'endLine', 'startTokenPos', 'endTokenPos', 'startFilePos', 'endFilePos'. The option defaults to the first three. For more info see getNextToken() docs.

Return values
mixed —

getNextToken()

Fetches the next token.

public getNextToken([mixed &$value = null ][, mixed &$startAttributes = null ][, mixed &$endAttributes = null ]) : int

The available attributes are determined by the 'usedAttributes' option, which can be specified in the constructor. The following attributes are supported:

  • 'comments' => Array of PhpParser\Comment or PhpParser\Comment\Doc instances, representing all comments that occurred between the previous non-discarded token and the current one.
  • 'startLine' => Line in which the node starts.
  • 'endLine' => Line in which the node ends.
  • 'startTokenPos' => Offset into the token array of the first token in the node.
  • 'endTokenPos' => Offset into the token array of the last token in the node.
  • 'startFilePos' => Offset into the code string of the first character that is part of the node.
  • 'endFilePos' => Offset into the code string of the last character that is part of the node.
Parameters
$value : mixed = null

Variable to store token content in

$startAttributes : mixed = null

Variable to store start attributes in

$endAttributes : mixed = null

Variable to store end attributes in

Return values
int —

Token id

getTokens()

Returns the token array for current code.

public getTokens() : array<string|int, mixed>

The token array is in the same format as provided by the token_get_all() function and does not discard tokens (i.e. whitespace and comments are included). The token position attributes are against this token array.

Return values
array<string|int, mixed> —

Array of tokens in token_get_all() format

handleHaltCompiler()

Handles __halt_compiler() by returning the text after it.

public handleHaltCompiler() : string
Return values
string —

Remaining text

startLexing()

Initializes the lexer for lexing the provided source code.

public startLexing(string $code[, ErrorHandler|null $errorHandler = null ]) : mixed

This function does not throw if lexing errors occur. Instead, errors may be retrieved using the getErrors() method.

Parameters
$code : string

The source code to lex

$errorHandler : ErrorHandler|null = null

Error handler to use for lexing errors. Defaults to ErrorHandler\Throwing

Return values
mixed —

createTokenMap()

Creates the token map.

protected createTokenMap() : array<string|int, mixed>

The token map maps the PHP internal token identifiers to the identifiers used by the Parser. Additionally it maps T_OPEN_TAG_WITH_ECHO to T_ECHO and T_CLOSE_TAG to ';'.

Return values
array<string|int, mixed> —

The token map

postprocessTokens()

protected postprocessTokens(ErrorHandler $errorHandler) : mixed
Parameters
$errorHandler : ErrorHandler
Return values
mixed —

createIdentifierTokenMap()

private createIdentifierTokenMap() : array<string|int, mixed>
Return values
array<string|int, mixed> —

defineCompatibilityTokens()

private defineCompatibilityTokens() : mixed
Return values
mixed —

handleInvalidCharacterRange()

private handleInvalidCharacterRange(mixed $start, mixed $end, mixed $line, ErrorHandler $errorHandler) : mixed
Parameters
$start : mixed
$end : mixed
$line : mixed
$errorHandler : ErrorHandler
Return values
mixed —

isUnterminatedComment()

Check whether comment token is unterminated.

private isUnterminatedComment(mixed $token) : bool
Parameters
$token : mixed
Return values
bool —

Search results