Lexical Conventions

There are five classes of tokens: identifiers, keywords, constants, operators, and other separators. Blanks, tabs, newlines, and comments (collectively, ``white space'') as described below are ignored except as they serve to separate tokens. Some white space is required to separate otherwise adjacent identifiers, keywords, and constants. If the input stream has been parsed into tokens up to a given character, the next token is taken to include the longest string of characters which could possibly constitute a token.