Tokens are the basic lexical building blocks of source code: they are the “words” of the language:
characters are combined into tokens according to the rules of the programming language. There are
five classes of tokens:
-
reserved words
- These are words which have a fixed meaning in the language. They cannot
be changed or redefined.
-
identifiers
- These are names of symbols that the programmer defines. They can be changed
and re-used. They are subject to the scope rules of the language.
-
operators
- These are usually symbols for mathematical or other operations: +, -, * and so
on.
-
separators
- This is usually white-space.
-
constants
- Numerical or character constants are used to denote actual values in the source
code, such as 1 (integer constant) or 2.3 (float constant) or “String constant” (a string:
a piece of text).
In this chapter we describe all the Pascal reserved words, as well as the various ways to denote
strings, numbers, identifiers etc.