Parsing & the Token Stream

A line of Smojo code or a Smojo script in a text file is represented as a token stream - a sequence of strings delimited by spaces. This token stream is then interpreted token by token.
	   Each token represents a word or number or string. Quite often, your word needs to access tokens ahead of the token stream. You can do this using the 
PARSE
word.

Word Action
NEXT-TOKEN
Used in compilation mode. It takes the following token of the enclosing word. The token is stored as a string and it replaces the
NEXT-TOKEN
PARSE
Takes one argument, a character telling the word when to stop.
[CHAR]
Converts the following token into a character. Some useful characters are \n (carriage return) \r (linefeed) and \t (tab).
BL
A character representing a space
+PARSER
Takes an XT as an argument. If the XT returns 0, this input will be ignored. If the XT returns an item, say r and 1, the parser will then process the item r. If -1 is returned, the parser will pass this input.
-PARSER
Takes an XT as an argument. This removes the parser which has been included previously.

Parsing is the process of analysing a string of symbols. A parser takes input data and checks for correct syntax in the process. 
	   Using a 
+PARSER
and
-PARSER
, we are able create some words so that we can read in inputs of special forms and process them into a useful and valid information. A simple example is shown in Example 3 where we now can read fractions.

Examples

Example 1
	: coming-up
    		"What comes after?" . cr
    		next-token .
	;

	coming-up Me!

	What comes after?
	Me!
	ok
	
Example 2

CHOMP
reads the entire line to the carriage return and put the result on the stack.

	: chomp [char] \n parse ;
	
	chomp Hello World!
	.
	Hello World!
	
Example 3

The line comment word 
\
.
\
reads the entire line to the carriage return. The result is dropped then from the stack.

	: \ [char] \n parse drop ;
	
	
Example 4
	: fraction? ( "s" -- f ) 
		[char] / indexofc 0 > ;

	: to-number ( "a/b" -- d )
		"/" split drop
		int swap int swap /. ;
	
	: fractions ( "s" -- 0 | r 1 | -1 ) 
		dup fraction? -> to-number 1 |
		otherwise	 drop -1 |.
	;
	
	1/2
	Error: Unknown word 1/2

	' fractions +parser
	
	1/2 1/2 +. . cr
	1.0 ok

	' fractions -parser
		
	4/5
	Error: Unkown word 4/5
	

Quiz

Question 1

What are other words in Smojo which uses parsing?



Question 2

Write the stack comment word 
(
using parsing.



Question 3

In Example 3, what happens if we do the following? Why does it work even though 5 and 2 are not fractions?

	' fractions +parser
	5 2 + .
	' fractions -parser


Next: Compiling Programmatically