o f2@sddZddlmZddlZejeeeedddlmZddlm Z m Z m Z eZ Gdd d eZ dS) z3 Python Lexical Analyser Scanning an input stream )absolute_importN)BOLEOLEOF NOT_FOUND)Errors)rrrc@seZdZdZdddZddZdd Zd d Zd d ZddZ ddZ ddZ ddZ ddZ dddZddZeddZdS) Scannera A Scanner is used to read tokens from a stream of characters using the token set specified by a Plex.Lexicon. Constructor: Scanner(lexicon, stream, name = '') See the docstring of the __init__ method for details. Methods: See the docstrings of the individual methods for more information. read() --> (value, text) Reads the next lexical token from the stream. position() --> (name, line, col) Returns the position of the last token read using the read() method. begin(state_name) Causes scanner to change state. produce(value [, text]) Causes return of a token value to the caller of the Scanner. NcCsd|_d|_d|_d|_d|_d|_d|_d|_d|_d|_ d|_ ||_ ||_ ||_ g|_d|_|dd|_d|_d|_t|_d|_|durV|d|d |_|_dSdS)a Scanner(lexicon, stream, name = '') |lexicon| is a Plex.Lexicon instance specifying the lexical tokens to be recognised. |stream| can be a file object or anything which implements a compatible read() method. |name| is optional, and may be the name of the file being scanned or any other identifying string. rr r)r rrN)tracebuffer buf_start_posnext_poscur_poscur_line start_poscurrent_scanner_position_tuplelast_token_position_tupletext state_namelexiconstreamnamequeue initial_statebegincur_line_startrcur_char input_state)selfrrr initial_posr"P/var/www/html/mig_web/myenv/lib/python3.10/site-packages/Cython/Plex/Scanners.py__init__Ms2  zScanner.__init__cCsp|j}|s,|\|_}|dur|d|n|||j}|dur*|||r|d\}|_|d=|S)a" Read the next lexical token from the stream and return a tuple (value, text), where |value| is the value associated with the token as specified by the Lexicon, and |text| is the actual string read from the stream. Returns (None, '') on end of file. Nr)r scan_a_tokenrproduceeofperformr)r ractionvalueresultr"r"r#readus    z Scanner.readcCs|jd||f|fdS)Nr)rinsert)r tokenr*positionr"r"r#unreadszScanner.unreadcC|jSN)rr r"r"r#get_current_scan_posszScanner.get_current_scan_poscCs|j|_|j|j|j|jf|_|}|dur9|jr&td||j|jf|j |j|j |j|j }||fS|j|jkrT|j t urH| |j dusR|j turTdSt||j)z Read the next input sequence recognised by the machine and return (text, action). Returns ('', None) on end of file. Nz"Scanner: read: Performing %s %d:%dr N)rrrrrrrun_machine_inlinedr printr rrr next_charrrUnrecognizedInputr)r r)rr"r"r#r%s*     zScanner.scan_a_tokencCst|j}|j}|j}|j}|j}|j}|j}|j}|j} t |} d\} } } }}}}|j } |rnumberr)NelsezState %dir  r blockedzDoing %s)rrrrrrrr rlenr r7reprgetrrrr,rrr)r staterrrrrrr rbuf_lenb_action b_cur_pos b_cur_lineb_cur_line_start b_cur_char b_input_state b_next_posr r)c new_state buf_indexdiscarddatar"r"r#r6s       P  zScanner.run_machine_inlinedcCs|j}|jrtdd||jf|dkr4|j|_|}|dkr't|_d|_n?|s0t|_d|_n6||_n2|dkr?d|_d|_n'|dkrX|jd7_|j|_ |_t |_d|_n|dkrct |_d|_nd |_|jrvtd ||j|jfdSdS) NzScanner: next: %s [%d] %dz rr<r r=r>r?r z--> [%d] %d %r) rr r7rr read_charrrrrrr)r rrMr"r"r#r8s8zScanner.next_charcCr1)am Return a tuple (name, line, col) representing the location of the last token read using the read() method. |name| is the name that was provided to the Scanner constructor; |line| is the line number in the stream (1-based); |col| is the position within the line of the first character of the token (0-based). rr3r"r"r#r/8s zScanner.positioncCs|S)zX Python accessible wrapper around position(), only for error reporting. )r/r3r"r"r# get_positionCszScanner.get_positioncCs|j||_||_dS)z8Set the current state of the scanner to the named state.N)rget_initial_staterr)r rr"r"r#rIs  z Scanner.begincCs(|dur|j}|j||f|jfdS)a Called from an action procedure, causes |value| to be returned as the token value from read(). If |text| is supplied, it is returned in place of the scanned text. produce() can be called more than once during a single call to an action procedure, in which case the tokens are queued up and returned one at a time by subsequent calls to read(), until the queue is empty, whereupon scanning resumes. N)rrappendr)r r*rr"r"r#r&Os zScanner.producecCsdS)z_ Override this method if you want something to be done at end of file. Nr"r3r"r"r#r'^sz Scanner.eofcCs |jdS)NrrSr3r"r"r# start_linees zScanner.start_liner5r2)__name__ __module__ __qualname____doc__r$r,r0r4r%r6r8r/rTrr&r'propertyrWr"r"r"r#r s  8(m  r )r[ __future__rcythondeclareobjectr rRegexpsrrrrr r"r"r"r#s