@c =================================================================== @c
@c    serrano/prgm/project/bigloo/manuals/rgc.texi                     @c
@c    ------------------------------------------------------------     @c
@c    Author      :  Manuel Serrano                                    @c
@c    Creation    :  Sat Jun 13 17:51:10 1998                          @c
@c    Last change :  Mon Nov 12 15:17:54 2001 (serrano)                @c
@c    ------------------------------------------------------------     @c
@c    The documentation for Rgc                                        @c
@c =================================================================== @c

@c ------------------------------------------------------------------- @c
@c    Rgc                                                              @c
@c ------------------------------------------------------------------- @c
@node Regular Parsing, Lalr Parsing, Threads, Top
@comment  node-name,  next,  previous,  up
@chapter Regular parsing
@cindex Regular parsing

Programming languages have poor reading libraries since the lexical
information that can be specified is directly tied to the structure of
the language. For example, in C it's hard to read a rational number
because there is no type rational.  Programs have been written to
circumvent this problem: Lex [Lesk75], for example, is one of them.  We
choose to incorporate in Bigloo a set of new functions to assist in such
parsing. The syntax for regular parser of Bigloo 2.0 (the one described 
in this document) is not compatible with former Bigloo versions.

@menu
* A New Way of Reading::        
* The Syntax of The Regular Parser::  
* The Semantics Actions::       
* Examples of Regular Grammar::  
@end menu

@c -- A new way of reading ------------------------------------------- @c
@node  A New Way of Reading, The Syntax of The Regular Parser, , Regular Parsing
@comment  node-name,  next,  previous,  up
@section A new way of reading
@cindex A new way of reading

There is only one way in Bigloo to read text, @emph{regular reading},
which is done by the new form:

@deffn {bigloo procedure} read/rp regular-parser input-port

The first argument is a regular parser and the second a Scheme port.
This way of reading is almost the same as the Lex's one. The reader
tries to match the longest input, from the stream pointed to by
@var{input-port}, with one of several regular expressions contained in
@var{regular-parser}. If many rules match, the reader takes the first one
defined in the grammar. When the regular rule has been found the corresponding
Scheme expression is evaluated.

@emph{remark: } The traditional @code{read} Scheme function is implemented as:
@smalllisp
(define-inline (read port)
   (read/rp scheme-grammar port))
@end smalllisp
@end deffn

@deffn {bigloo variable} *about-to-read*

When this variable is bound to a function, then each time the reader has
filled its buffer, this function is called with an @code{input-port} as its
argument.
@end deffn

@c -- The syntax of the regular parser ------------------------------- @c
@node  The Syntax of The Regular Parser, The Semantics Actions, A New Way of Reading, Regular Parsing
@comment  node-name,  next,  previous,  up
@section The syntax of the regular parser
@cindex The syntax of the regular parser

A regular parser is built by the means of the form @code{regular-grammar}:

@deffn {bigloo syntax} regular-grammar (binding @dots{}) rule @dots{}

The @var{binding} and @var{rule} are defined by 
the following grammar:

@smallexample
<binding>  @expansion{} (<variable> <re>)
<rule>     @expansion{} (<cre> <s-expression> <s-expression> @dots{})
             | (<context> <re> <s-expression> <s-expresion> @dots{})
             | (@code{else} <s-expression> <s-expression> @dots{})
<cre>      @expansion{} <re>
             | (@code{context} <symbol> <re>)
             | (@code{when} <s-expr> <re>)
             | (@code{bol} <re>)
             | (@code{eol} <re>)
             | (@code{bof} <re>)
             | (@code{eof} <re>)
<re>       @expansion{} <variable>
             | <char>
             | <string>
             | (@code{:} <re> @dots{})
             | (@code{or} <re> @dots{})
             | (@code{*} <re>)
             | (@code{+} <re>)
             | (@code{?} <re>)
             | (@code{=} <integer> <re>)
             | (@code{>=} <integer> <re>)
             | (@code{**} <integer> <integer> <re>)
             | (@code{...} <integer> <re>)
             | (@code{uncase} <re>)
             | (@code{in} <cset> @dots{})
             | (@code{out} <cset> @dots{})
             | (@code{and} <cset> <cset>)
             | (@code{but} <cset> <cset>)
             | (@code{posix} <string>)
@c               | (@code{submatch} <re>)                              @c
<variable> @expansion{} <symbol>
<cset>     @expansion{} <string>
             | <char>
             | (<string>)
             | (<char> <char>)
@end smallexample

Here is a description of each construction.

@table @code

@item (context <symbol> <re>)
This allows us to @emph{protect} an expression. A @emph{protected}
expression matches (or accepts) a word only if the parser has been set to
the corresponding context. @xref{The Semantics Actions}, for more details.

@item (when <s-expr> <re>)
This allows us to @emph{protect} an expression. A @emph{protected}
expression matches (or accepts) a word only if the evaluation of 
@code{<s-expr>} is @code{#t}. For instance,

@smalllisp
(define *g*
   (let ((armed #f))
      (regular-grammar ()
	 ((when (not armed) (: "#!" (+ (or #\/ alpha))))
	  (set! armed #t)
	  (print "start [" (the-string) "]")
	  (ignore))
	 ((+ (in #\Space #\Tab))
	  (ignore))
	 (else
	  (the-failure)))))
   
(define (main argv)
   (let ((port (open-input-string "#!/bin/sh #!/bin/zsh")))
      (print (read/rp *g* port))))
@end smalllisp

@item (bol <re>)
Matches @code{<re>} at the beginning of line.

@item (eol <re>)
Matches @code{<re>} at the end of line.

@item (bof <re>)
Matches @code{<re>} at the beginning of file.

@item (eof <re>)
Matches @code{<re>} at the end of file.

@item <variable>
This is the name of a variable bound by a <binding> construction. In addition 
to user defined variables, some already exist. These are:

@smallexample
all    @equiv{} (out #\Newline)
lower  @equiv{} (in ("az"))
upper  @equiv{} (in ("AZ"))
alpha  @equiv{} (or lower upper)
digit  @equiv{} (in ("09"))
xdigit @equiv{} (uncase (in ("af09")))
alnum  @equiv{} (uncase (in ("az09")))
punct  @equiv{} (in ".,;!?")
blank  @equiv{} (in #" \t\n")
space  @equiv{} #\Space
@end smallexample


It is a error to reference a variable that it is not bound by a <binding>.
Defining a variable that already exists is acceptable and causes the former
variable definition to be erased. Here is an example of a grammar that binds
two variables, one called @samp{ident} and one called @samp{number}. These
two variables are used within the grammar to match identifiers and numbers.

@smalllisp
(regular-grammar ((ident  (: alpha (* alnum)))
                  (number (+ digit)))
   (ident  (cons 'ident (the-string)))
   (number (cons 'number (the-string)))
   (else   (cons 'else (the-failure))))
@end smalllisp

@item <char>
The regular language described by one unique character. Here is an example of
a grammar that accepts either the character @code{#\a} or the character
@code{#\b}:

@smalllisp
(regular-grammar ()
   (#\a (cons 'a (the-string)))
   (#\b (cons 'b (the-string)))
   (else (cons 'else (the-failure))))
@end smalllisp

@item <string>
This simple form of regular expression denotes the language represented
by the string. For instance the regular expression @code{"Bigloo"} matches
only the string composed of @code{#\B #\i #\g #\l #\o #\o}. The regular 
expression @code{".*["} matches the string @code{#\. #\* #\[}.

@item (: <re> ...)
This form constructs sequence of regular expression. That is a form
@code{<re1> <re2> ... <ren>} matches the language construction by
concatenation of the language described by @code{<re1>}, @code{<re2>}, 
@code{<ren>}. Thus, @code{(: "x" all "y")} matches all words of three
letters, started by character the @code{#\x} and ended with the character
@code{#\y}.

@item (or <re> ...)
This construction denotes conditions. The language described by
@code{(or re1 re2)} accepts words accepted by either @code{re1} or @code{re2}.
 
@item (* <re>)
This is the Kleene operator, the language described by @code{(* <re>)} is
the language containing, 0 or more occurrences of @code{<re>}. Thus, 
the language described by @code{(* "abc")} accepts the empty word and
any word composed by a repetition of the @code{abc} (@code{abc},
@code{abcabc}, @code{abcabcabc}, ...).

@item (+ <re>)
This expression described non empty repetitions. The form @code{(+ re)} is
equivalent to @code{(: re (* re))}. Thus, @code{(+ "abc")} matches the
words @code{abc}, @code{abcabc}, etc.

@item (? <re>)
This expression described one or zero occurrence. Thus, 
@code{(? "abc")} matches the empty word or the words @code{abc}.

@item (= <integer> <re>)
This expression described a fix number of repetitions. The form
@code{(= num re)} is equivalent to @code{(: re re ... re)}. Thus,
the expression @code{(= 3 "abc")} matches the only word @code{abcabcabc}.
In order to avoid code size explosion when compiling, @code{<integer>} 
must be smaller than an arbitrary constant. In the current version that 
value is @code{81}.

@item (>= <integer> <re>)
The language described by the expression @code{(>= int re)} accepts word
that are, at least, @code{int} repetitions of @code{re}. For instance,
@code{(>= 10 #\a)}, accepts words compound of, at least, 10 times the
character @code{#\a}. In order to avoid code size explosion when compiling, 
@code{<integer>} must be smaller than an arbitrary constant. In the current 
version that value is @code{81}.
 
@item (** <integer> <integer> <re>)
The language described by the expression @code{(** min max re)} accepts
word that are repetitions of @code{re}; the number of repetition is in
the range @code{min}, @code{max}. For instance, @code{(** 10 20 #\a)}.
In order to avoid code size explosion when compiling, 
@code{<integer>} must be smaller than an arbitrary constant. In the current 
version that value is @code{81}.

@item (... <integer> <re>)
The subexpression @code{<re>} has to be a sequence
of characters. Sequences are build by the operator @code{:} or by string
literals. The language described by @code{(... int re)}, denotes, the
first letter of @code{re}, or the two first letters of @code{re}, or the
three first letters of @code{re} or the @code{int} first letters of
@code{re}. Thus, @code{(... 3 "begin")} is equivalent to 
@code{(or "b" "be" "beg")}.

@item (uncase <re>)
The subexpression @code{<re>} has to be a sequence
construction. The language described by @code{(uncase re)} is the
same as @code{re} where letters may be upper case or lower case. For
instance, @code{(uncase "begin")}, accepts the words @code{"begin"},
@code{"beGin"}, @code{"BEGIN"}, @code{"BegiN"}, etc.

@item (in <cset> ...)
Denotes union of characters. Characters may be described individually
such as in @code{(in #\a #\b #\c #\d)}. They may be described by
strings. The expression @code{(in "abcd")} is equivalent to @code{(in
#\a #\b #\c #\d)}.  Characters may also be described using a range
notation that is a list of two characters. The expression @code{(in (#\a
#\d))} is equivalent to @code{(in #\a #\b #\c #\d)}. The Ranges may be
expresses using lists of string. The expression @code{(in ("ad"))}
is equivalent to @code{(in #\a #\b #\c #\d)}.

@item (out <cset> ...)
The language described by @code{(out cset ...)} is opposite to
the one described by @code{(in cset ...)}. For instance, 
@code{(out ("azAZ") (#\0 #\9))} accepts all words of one character
that are neither letters nor digits. One should not that if the character
numbered zero may be used inside regular grammar, the @code{out} 
construction never matches it. Thus to write a rule that, for instances,
matches every character but @code{#\Newline} including the character
zero, one should write: 

@smalllisp
(or (out #\Newline) #a000)
@end smalllisp

@item (and <cset> <cset>)
The language described by @code{(and cset1 cset2)} accepts words 
made of characters that are in both @code{cset1} and @code{cset2}. 

@item (but <cset> <cset>)
The language described by @code{(but cset1 cset2)} accepts words 
made of characters of @code{cset1} that are not member of @code{cset2}. 
 
@item (posix <string>)
The expression @code{(posix string)} allows one to use Posix string
notation for regular expressions. So, for example, the following two
expressions are equivalent:

@smalllisp
(posix "[az]+|x*|y@{3,5@}")

(or (+ (in ("az"))) (* "x") (** 3 5 "y"))
@end smalllisp

@c  @item (submatch <re>)                                              @c
@c  This matches the empty string and prepares submatch for the runtime function  @c
@c  @code{the-submatch}.                                               @c
@end table
@end deffn

@deffn {bigloo syntax} string-case string rule ...
This form dispatches on strings. it opens an input on @code{string}
a read into it according to the regular grammar defined by the
@code{binding} and @code{rule}. Example:

@smalllisp
(define (suffix string)
   (string-case string
      ((: (* all) ".")
       (ignore))
      ((+ (out #\.))
       (the-string))
      (else
       "")))
@end smalllisp
@end deffn
   
@c -- The semantics actions ------------------------------------------ @c
@node  The Semantics Actions,  Examples of Regular Grammar, The Syntax of The Regular Parser, Regular Parsing
@comment  node-name,  next,  previous,  up
@section The semantics actions
@cindex the semantics actions

The semantics actions are regular Scheme expressions. These expressions
appear in an environment where some ``extra procedures'' are defined.
These procedures are:

@deffn {bigloo rgc procedure} the-port
Returns the input port currently in used.
@end deffn

@deffn {bigloo rgc procedure} the-length
Get the length of the biggest matching string.
@end deffn

@deffn {bigloo rgc procedure} the-string
Get a copy of the last matching string. The function @code{the-string}
returns a fresh copy of the matching each time it is called. In consequence,

@smalllisp
(let ((l1 (the-string)) (l2 (the-string)))
   (eq? l1 l2))
   @result{} #f
@end smalllisp
@end deffn

@deffn {bigloo rgc procedure} the-substring start length
Get a copy of a substring of the last matching string. Here is an example
of a rule extracting a part of a match

@smalllisp
(regular-grammar ()
   ((: #\" (* (out #\")) #\")
    (the-substring 1 (-fx (the-length) 1))))
@end smalllisp
@end deffn

@deffn {bigloo rgc procedure} the-character
Returns the first character of a match.
@end deffn

@deffn {bigloo rgc procedure} the-symbol
Convert the last matching string into a symbol.
@end deffn

@deffn {bigloo rgc procedure} the-keyword
Convert the last matching string into a keyword.
@end deffn

@deffn {bigloo rgc procedure} the-fixnum
The conversion of the last matching string to fixnum.
@end deffn

@deffn {bigloo rgc procedure} the-flonum
The conversion of the last matching string to flonum.
@end deffn
 
@deffn {bigloo rgc procedure} the-failure
Returns the first char that the grammar can't match or the end of file
object.
@end deffn

@deffn {bigloo rgc procedure} ignore
Ignore the parsing, keep reading. It's better to use @code{(ignore)}
rather than an expression like @code{(read/rp @var{grammar} @var{port})}
in semantics actions since the @code{(ignore)} call will be done in a
tail recursive way. For instance,
 
@smalllisp
(let ((g (regular-grammar ()
            (")" 
             '())
            ("(" 
             (let* ((car (ignore))
                    (cdr (ignore)))
                (cons car cdr)))
            ((+ (out "()"))
             (the-string))))
      (p (open-input-string "(foo(bar(gee)))")))
   (read/rp g p))
   @result{} ("foo" ("bar" ("gee")))
@end smalllisp
@end deffn

@deffn {bigloo rgc procedure} rgc-context [context]
If no @var{context} is provide, this procedure reset the reader context
state. That is the reader is in no context. With one argument, @code{context}
set the reader in the context @var{context}. For instance,

@smalllisp
(let ((g (regular-grammar ()
            ((context foo "foo") (print 'foo-bis))
            ("foo" (rgc-context 'foo) (print 'foo) (ignore))
            (else 'done)))
      (p (open-input-string "foofoo")))
   (read/rp g p))
   @print{} FOO
      FOO-BIS
@end smalllisp
@end deffn

@deffn {bigloo rgc procedure} the-context
Returns the value of the current Rgc context.
@end deffn

@c  @deffn {bigloo rgc procedure} the-submatch number                  @c
@c  Get the submatch numbered @var{number}. Submatch are computed by the  @c
@c  @code{submatch} construction. For instance,                        @c
@c                                                                     @c
@c  @smalllisp                                                         @c
@c  (let ((g (regular-grammar ()                                       @c
@c              ((: (submatch (* digit)) "." (submatch (* digit)))     @c
@c               (cons (the-submatch 1) (the-submatch 2)))))           @c
@c        (p (open-input-string "3.1415")))                            @c
@c     (print (read/rp g p)))                                          @c
@c     @print{} ("3" . "1415")                                         @c
@c  @end smalllisp                                                     @c
@c                                                                     @c
@c  @end deffn                                                         @c

@c -- Examples of regular grammar ------------------------------------ @c
@node Examples of Regular Grammar,  , The Semantics Actions, Regular Parsing
@comment  node-name,  next,  previous,  up
@section Examples of regular grammar
@cindex examples of regular grammar
The reader who wants to find a real example should read the code of
Bigloo's reader. But here are small examples

@subsection Word count
The first example presents a grammar that simulates the Unix program @code{wc}.
@smalllisp
(let ((*char* 0)
      (*word* 0)
      (*line* 0))
   (regular-grammar ()
      ((+ #\Newline)
       (set! *char* (+ *char* (the-length)))
       (set! *line* (+ *line* (the-length)))
       (ignore))
      ((+ (in #\space #\tab))
       (set! *char* (+ *char* (the-length)))
       (ignore))
      ((+ (out #\newline #\space #\tab))
       (set! *char* (+ *char* (the-length)))
       (set! *word* (+ 1 *word*))
       (ignore))))
@end smalllisp

@subsection Roman numbers
The second example presents a grammar that reads Arabic and Roman number.
@smalllisp
(let ((par-open 0))
   (regular-grammar ((arabic (in ("09")))
                     (roman  (uncase (in "ivxlcdm"))))
      ((+ (in #" \t\n"))
       (ignore))
      ((+ arabic)
       (string->integer (the-string)))
      ((+ roman)
       (roman->arabic (the-string)))
      (#\(
       (let ((open-key par-open))
          (set! par-open (+ 1 par-open))
          (context 'pair)
          (let loop-pair ((walk (ignore))) 
             (cond
                ((= open-key par-open)
                 '())
                (else
                 (cons walk (loop-pair (ignore))))))))
      (#\)
       (set! par-open (- par-open 1))
       (if (< par-open 0)
           (begin
              (set! par-open 0)
              (ignore))
           #f))
      ((in "+-*\\")
       (string->symbol (the-string)))
      (else
       (let ((char (the-failure)))
          (if (eof-object? char)
              char
              (error "grammar-roman" "Illegal char" char))))))
@end smalllisp
