Compilers - Reentrant flex & bison

Author

Reentrant flex & bison

cherico

2004-03-27, 12:29 am

I am trying to write a reentrant scanner and parser using
flex and bison.

I discovered that the reentrance of flex is accomplished
by generating a c++ scanner class, while reentrant bison
remains the old c-style function call to yylex().

Anybody knows what the way to "glue" them (scanner & parser)
together is. flex and bison doesn't seem to co-work well
in this way, right?!
[Right. The C++ reentrant lex scanner doesn't talk to the
reentrant C bison parser. -John]

Clint Olsen

2004-03-27, 12:29 am

On 2004-03-15, cherico <cherico@bonbon.net> wrote:
> Anybody knows what the way to "glue" them (scanner & parser)
> together is. flex and bison doesn't seem to co-work well
> in this way, right?!
> [Right. The C++ reentrant lex scanner doesn't talk to the
> reentrant C bison parser. -John]

There has been a reentrant version of flex for ANSI C for some time. See
http://lex.sf.net.

-Clint

Paolo Bonzini

2004-03-27, 12:29 am

> I discovered that the reentrance of flex is accomplished
> by generating a c++ scanner class, while reentrant bison
> remains the old c-style function call to yylex().
>
> Anybody knows what the way to "glue" them (scanner & parser)
> together is. flex and bison doesn't seem to co-work well
> in this way, right?!

Much better, newer Flexes have more options that will suit your needs.
Here are excerpts from the docs. Bracket text is mine.

-----------------
`%option reentrant'

* All functions take one additional argument: `yyscanner'

* All global variables are replaced by their macro equivalents. (We
tell you this because it may be important to you during debugging.)

* `yylex_init' and `yylex_destroy' must be called before and after
`yylex', respectively.

* Accessor methods (get/set functions) provide access to common `flex'
variables.

* User-specific data can be stored in `yyextra'.

The following Functions are available in a reentrant scanner:

char *yyget_text ( yyscan_t scanner );
int yyget_leng ( yyscan_t scanner );
FILE *yyget_in ( yyscan_t scanner );
FILE *yyget_out ( yyscan_t scanner );
int yyget_lineno ( yyscan_t scanner );
YY_EXTRA_TYPE yyget_extra ( yyscan_t scanner );
int yyget_debug ( yyscan_t scanner );

void yyset_debug ( int flag, yyscan_t scanner );
void yyset_in ( FILE * in_str , yyscan_t scanner );
void yyset_out ( FILE * out_str , yyscan_t scanner );
void yyset_lineno ( int line_number , yyscan_t scanner );
void yyset_extra ( YY_EXTRA_TYPE user_defined , yyscan_t scanner );

In a reentrant C scanner, support for yylineno is always present
(i.e., you may access yylineno), but the value is never modified by
`flex' unless `%option yylineno' is enabled. This is to allow the user
to maintain the line count independently of `flex'.

The following functions and macros are made available when `%option
bison-bridge' (`--bison-bridge') is specified [together with
--reentrant]:

YYSTYPE * yyget_lval ( yyscan_t scanner );
void yyset_lval ( YYSTYPE * yylvalp , yyscan_t scanner );
yylval

The following functions and macros are made available when `%option
bison-locations' (`--bison-locations') is specified [together with
--reentrant]:

YYLTYPE *yyget_lloc ( yyscan_t scanner );
void yyset_lloc ( YYLTYPE * yyllocp , yyscan_t scanner );
yylloc

[There is also yyget_column, yyset_column and yycolumn, but they are
undocumented as of Flex version 2.5.31.]
----------------

If you really want C++ or Flex 2.5.4, you can use YYPARSE_PARAM so
that a pointer to the FlexLexer object is passed to the bison scanner,
and from there to an yylex wrapper function which actually calls the
flex scanner. --bison-bridge is not supported in the C++ class.

That is, rename the FlexLexer.h that comes with Flex to
FlexLexer_base.h, and in FlexLexer.h put something like this
(untested, plus omit yylloc if you do not use locations):

--------------
/* Add a member to the Flex base class (!)
#define yyout yyout; YYSTYPE *yylval; YYLTYPE *yylloc
#include "FlexLexer_base.h"
#undef yyout

/* Define the wrapper that calls the C++ scanner. */
inline int yylex (YYSTYPE *yylval, YYLTYPE *yylloc,
yyFlexLexer *yy_flex_lexer)
{
yy_flex_lexer->yylval = yylval;
yy_flex_lexer->yylloc = yylloc;
return yy_flex_lexer->yylex ();
}

/* Tell bison how to call the lexer */
#ifdef YYPURE
#define YYPARSE_PARAM yyFlexLexer *yy_flex_lexer
#define YYLEX_PARAM yy_flex_lexer
#endif
--------------

Paolo

cherico

2004-03-27, 12:29 am

I've kind of

by the version numbers of flex.
I use flex from gnu (2.5.4) and I found that there is
only version 2.5.31 on http://lex.sourceforge.net/.

Apperantly, flex 2.5.4 doesn't support the
"% opetion reentrant", but it has the feature of
generating a c++ scanner class. While, flex 2.5.31
supports the reentrant option.

I suppose, to work with bison well, flex 2.5.31 is a
better choice.

I am curious why flex 2.5.4 removes the support of the
reentrant option.
[I don't know who's doing the version at sourceforge, but it's not the
mainstream one. -John]

Hans Aberg

2004-04-03, 10:36 am

cherico@bonbon.net (cherico) wrote:

>I've kind of

by the version numbers of flex.
>I use flex from gnu (2.5.4) and I found that there is
>only version 2.5.31 on http://lex.sourceforge.net/.
....
>[I don't know who's doing the version at sourceforge, but it's not the
>mainstream one. -John]

The mainstream (=original, offical) Flex is now actively developed, and
has a combined help and bugs mailing list: The Help-flex mailing list
help-flex@gnu.org
http://mail.gnu.org/mailman/listinfo/help-flex

>Apperantly, flex 2.5.4 doesn't support the
>"% opetion reentrant", but it has the feature of
>generating a c++ scanner class. While, flex 2.5.31
>supports the reentrant option.
>
>I suppose, to work with bison well, flex 2.5.31 is a
>better choice.
>
>I am curious why flex 2.5.4 removes the support of the
>reentrant option.

One can get answers to such questions there.

Hans Aberg

Paul Eggert

2004-04-03, 10:36 am

At 26 Mar 2004 22:08:01 -0500, cherico@bonbon.net (cherico) writes:

> I suppose, to work with bison well, flex 2.5.31 is a
> better choice.

Either one should work. However, 2.5.31 is not an official stable
version so you can expect more gotchas with it.

> I am curious why flex 2.5.4 removes the support of the
> reentrant option.

2.5.31 came after 2.5.4, just as 2.5.10 came after 2.5.9.

> [I don't know who's doing the version at sourceforge, but it's not
> the mainstream one. -John]

W. L. Estes of UNC Greensboro is doing that version. I can't check
its current status, as sourceforge.net is down right now.

As far as I know Vern Paxson, the previous maintainer of 'flex',
hasn't had time to work on it since 2.5.4 came out in 1996. If you
want just bug fixes for 2.5.4 I suggest using the Debian GNU/Linux
patches, which you can get from
<http://packages.debian.org/stable/devel/flex>.

Paolo Bonzini

2004-04-03, 10:36 am

> I suppose, to work with bison well, flex 2.5.31 is a
> better choice.

Yes, it is.

> I am curious why flex 2.5.4 removes the support of the
> reentrant option.

Because 31 is supposed to come after 4, 5, 6, ..., 29, 30 and not to
be a decimal.

> [I don't know who's doing the version at sourceforge, but it's not the
> mainstream one. -John]

Bruno Haible is maintaining this updated version of flex, as well as
being the maintainer of gperf (tool to create perfect hashes, very
widely used with manually written lexical analyzers).

Paolo

Indradeep

2004-04-14, 1:31 am

If I get yer question correctly yew would like to make flex and bison
work while writing yer code in c++...

flex with the -+ option generates a c++ class FlexLexer in which yylex
is defined.
but bison thru yyparse() calls yylex()...

to work around this, in yer parser file declare
FlexLexer *lexer = new FlexLexer;
and then define a function

int yylex()
{
return lexer->yylex();
}
this works perfectly well....
bye..

cherico@bonbon.net (cherico) wrote
> I am trying to write a reentrant scanner and parser using
> flex and bison.

Sponsored Links