1. Dev C++ Flex Bison

Flex and bison howto for C users, v1.0. This howto accumulates all tweaks, fixes and hacks required to implement a bison/flex command parser using C. It assumes some basic grammar, parsing and regular expression knowledge. Note, the approach described here concentrates on building a parser winthin windows environmet, however all ideas.

P: n/a
Hi,
I am using flex and bizon to write HTTP parser.
I am passing well flex and bison tools but can't compile their output.
Flex file (http_parser.lpp)
%{
#include <iostream>
#include 'http_parser.tab.hpp'
#define ECHO yyerror('Parse error.')
%}
%x REQUEST
%x AFTER_METHOD
%x RESP_VERSION
%x AFTER_RESP_VERSION
%x RESP_CODE
%x AFTER_RESP_CODE
%x RESP_MSG
%x AFTER_RESP_MSG
%x REQ_HOST
%x REQ_AFTER_HOST
%x REQ_PORT_DELIMITER
%x REQ_PORT
%x REQ_PATH
%x AFTER_REQ_PATH
%x REQ_BEFORE_VERSION
%x REQ_VERSION
%x AFTER_REQ_VERSION
%x HEADER_NAME
%x HEADER_DELIMITER
%x HEADER_VALUE
%x AFTER_HEADER_VALUE
WS [x20t]+
CRLF (rn r n)
TOKEN [^x80-xFFx7Fx00-x1Fx20x22rnt]+
LETTER [a-zA-Z]
HEADER_NAME [-_a-zA-Z]+
DIGIT [0-9]
PROTOCOL (http:// ftp://)
HOST [^x2Fx3Ax80-xFFx7Fx00-x1Fx20x22rnt]+
PORT {DIGIT}+
METHOD {LETTER}+
%%
{METHOD}/{WS} {
BEGIN AFTER_METHOD;
std::cout << 'REQUESTn';
std::cout << 'METHOD=' << YYText() << 'n';
return T_REQ_METHOD;
}
<AFTER_METHOD>{WS} {
BEGIN REQUEST;
}
'HTTP/' {
BEGIN RESP_VERSION;
std::cout << 'RESPONSEn';
}
<RESP_VERSION>{DIGIT}+.{DIGIT}+/{WS} {
BEGIN AFTER_RESP_VERSION;
std::cout << 'RESP_VERSION=' << YYText() << 'n';
return T_RESP_VERSION;
}
<AFTER_RESP_VERSION>{WS} {
BEGIN RESP_CODE;
}
<RESP_CODE>{DIGIT}+/{WS} {
BEGIN AFTER_RESP_CODE;
std::cout << 'RESP_CODE=' << YYText() << 'n';
return T_RESP_CODE;
}
<AFTER_RESP_CODE>{WS} {
BEGIN RESP_MSG;
}
<RESP_MSG>{TOKEN}/{CRLF} {
BEGIN AFTER_RESP_MSG;
std::cout << 'RESP_MSG=' << YYText() << 'n';
return T_RESP_MSG;
}
<AFTER_RESP_MSG>{CRLF} {
BEGIN HEADER_NAME;
}
<REQUEST>'/' {
BEGIN REQ_PATH;
std::cout << 'REQ_RELATIVE_1n';
}
<REQUEST>'/'{WS} {
BEGIN REQ_BEFORE_VERSION;
std::cout << 'REQ_RELATIVE_2n';
}
<REQUEST>{PROTOCOL} {
BEGIN REQ_HOST;
std::cout << 'REQ_ABSOLUTE_1n';
std::cout << 'PROTOCOL=' << YYText() << 'n';
return T_REQ_PROTOCOL;
}
<REQ_HOST>{HOST}/: {
BEGIN REQ_PORT_DELIMITER;
std::cout << 'REQ_HOST_1=' << YYText() << 'n';
return T_REQ_HOST;
}
<REQ_HOST>{HOST}// {
BEGIN REQ_PATH;
std::cout << 'REQ_HOST_2=' << YYText() << 'n';
return T_REQ_HOST;
}
<REQ_HOST>{HOST}/{WS} {
BEGIN REQ_AFTER_HOST;
std::cout << 'REQ_HOST_3=' << YYText() << 'n';
return T_REQ_HOST;
}
<REQ_AFTER_HOST>{WS} {
BEGIN REQ_BEFORE_VERSION;
}
<REQ_PORT_DELIMITER>':' {
BEGIN REQ_PORT;
}
<REQ_PORT>{PORT}// {
BEGIN REQ_PATH;
std::cout << 'REQ_PORT_1=' << YYText() << 'n';
return T_REQ_PORT;
}
<REQ_PORT>{PORT}/{WS} {
BEGIN REQ_BEFORE_VERSION;
std::cout << 'REQ_PORT_2=' << YYText() << 'n';
return T_REQ_PORT;
}
REQ_PATH>{TOKEN}/{WS} {
BEGIN AFTER_REQ_PATH;
std::cout << 'REQ_PATH=' << YYText() << 'n';
return T_REQ_PATH;
}
<AFTER_REQ_PATH>{WS} {
BEGIN REQ_BEFORE_VERSION;
}
<REQ_BEFORE_VERSION>'HTTP/' {
BEGIN REQ_VERSION;
}
<REQ_VERSION>{DIGIT}+.{DIGIT}+/{CRLF} {
BEGIN AFTER_REQ_VERSION;
std::cout << 'REQ_VERSION=' << YYText() << 'n';
return T_REQ_VERSION;
}
<AFTER_REQ_VERSION>{CRLF} {
BEGIN HEADER_NAME;
}
<HEADER_NAME>{HEADER_NAME}+/:{WS} {
BEGIN HEADER_DELIMITER;
std::cout << 'HEADER_NAME=' << YYText() << 'n';
return T_HEADER_NAME;
}
<HEADER_DELIMITER>:{WS} {
BEGIN HEADER_VALUE;
}
<HEADER_VALUE>[^rn]*/{CRLF} {
BEGIN AFTER_HEADER_VALUE;
std::cout << 'HEADER_VALUE=' << YYText() << 'n';
return T_HEADER_VALUE;
}
<AFTER_HEADER_VALUE>{CRLF} {
BEGIN HEADER_NAME;
}
<AFTER_HEADER_VALUE>{CRLF}{CRLF} { ; }
%%
Bizon file (http_parser.ypp)
%{
#include <iostream>
extern 'C'
{
void yyerror(char *);
int yyparse(void);
int yylex(void);
int yywrap();
}
%}
%token T_REQ_METHOD
%token T_REQ_PROTOCOL
%token T_REQ_HOST
%token T_REQ_PORT
%token T_REQ_PATH
%token T_REQ_VERSION
%token T_RESP_VERSION
%token T_RESP_CODE
%token T_RESP_MSG
%token T_HEADER_NAME
%token T_HEADER_VALUE
%%
program:
request
response
;
request:
/* Absolute requests */
T_REQ_METHOD T_REQ_PROTOCOL T_REQ_HOST T_REQ_PORT T_REQ_PATH
T_REQ_VERSION headers {
std::cout << 'ABS => T_REQ_METHOD T_REQ_PROTOCOL T_REQ_HOST
T_REQ_PORT T_REQ_PATH T_REQ_VERSION headersn';
}

T_REQ_METHOD T_REQ_PROTOCOL T_REQ_HOST T_REQ_PATH T_REQ_VERSION
headers {
std::cout << 'ABS => T_REQ_METHOD T_REQ_PROTOCOL T_REQ_HOST
T_REQ_PATH T_REQ_VERSION headersn';
}

T_REQ_METHOD T_REQ_PROTOCOL T_REQ_HOST T_REQ_VERSION headers {
std::cout << 'ABS => T_REQ_METHOD T_REQ_PROTOCOL T_REQ_HOST
T_REQ_VERSION headersn';
}

/* Relative requests */
T_REQ_METHOD T_REQ_PATH T_REQ_VERSION headers {
std::cout << 'REL => T_REQ_METHOD T_REQ_PATH T_REQ_VERSION
headersn';
}

T_REQ_METHOD T_REQ_VERSION headers {
std::cout << 'REL => T_REQ_METHOD T_REQ_VERSION headersn';
}
;
response:
T_RESP_VERSION T_RESP_CODE T_RESP_MSG headers {
std::cout << 'T_RESP_VERSION T_RESP_CODE T_RESP_MSG headersn';
}
;
headers:
headers header

;
header:
T_HEADER_NAME T_HEADER_VALUE {
std::cout << 'T_HEADER_NAME T_HEADER_VALUEn';
}
;
%%
int yywrap(void) {
return 1;
}
void yyerror(char * msg)
{
std::cout << 'ERROR: ' << msg << std::endl;
}
int main( void )
{
FlexLexer* lexer = new yyFlexLexer;
while(lexer->yylex() != 0)
;
return 0;
}
Compile
1. flex --c++ http_parser.lpp
OK
2. bison -d http_parser.ypp
OK
3. Files list
http_parser.lpp
http_parser.tab.cpp
http_parser.tab.hpp
http_parser.ypp
lex.yy.cc
4. g++ -c lex.yy.cc http_parser.tab.cpp
http_parser.lpp: In member function `virtual int yyFlexLexer::yylex()':
http_parser.lpp:222: error: `yyerror' undeclared (first use this
function)
http_parser.lpp:222: error: (Each undeclared identifier is reported
only once
for each function it appears in.)
lex.yy.cc:1174: error: `yywrap' undeclared (first use this function)
http_parser.ypp: In function `int main()':
http_parser.ypp:90: error: `FlexLexer' undeclared (first use this
function)
http_parser.ypp:90: error: (Each undeclared identifier is reported only
once
for each function it appears in.)
http_parser.ypp:90: error: `lexer' undeclared (first use this function)
http_parser.ypp:90: error: parse error before `;' token
What i am doing wrong ???
Thanks a lot.
Thread Index]

Flex and bison howto for C++ users

From: oleg smolsky
Subject: Flex and bison howto for C++ users
Date: Tue, 18 Feb 2003 09:57:13 +1300

This howto accumulates all tweaks, fixes and hacks required to implement a bison/flex command parser using C++. It assumes some basic grammar, parsing and regular _expression_ knowledge.

Note, the approach described here concentrates on building a parser winthin windows environmet, however all ideas and tools are still applicable to unix/g++. One just need to remove a few lines of code, such as #include <windows.h> :)

Dev C++ Flex Bison

Written by Oleg Smolsky <address@hidden>, February 2003

The task

Lets imagine that we are building a debugger similar to NuMega SoftICE. It contains multiple windows and has a command driven, textual interface. The most basic of commands might be load <filename> and display <address>. So, lets imlement a system in C++, so that all scanning and parsing is done by using flex and bison respectevely.

Requred software

cygwin
install the latest version of cygwin and make sure that PATH environment variable contains cygwin's bin directpory. E.g. c:somethingcygwinbin. This way it is possible to call bash, flex and bison directly from a command prompts.
bison
make sure that you install bison version 1.875 during the cygwin install.
flex
install default flex that comes with cygwin. My version is 2.5.4
C++ compiler
you can use g++ or Microsoft VC++ (cl) with projects or makefiles

Core files' definition

Once all required tools are setup, we can discuss core files required to solve our task:
parser.y
the specification for the parser. Process it with bison:
bash -c 'bison -d -S lalr1.cc -o parser.cpp parser.y'
The way it works is this: you call bash with a command from your makefile or VC++ project. Once bash is operational, it can execute the given unix command that within the cygwin environment. This is important for tools such as bison, because it needs to access it's skeleton via a unix path: usrsharebisonlalr1.cc
scanner.l
the specification of the scanner. Process it with flex:
bash -c 'flex -oscanner.cpp scanner.l'

scanner.l

Lets consider the content of scanner.l The first section specifies a block of code to go to almost the very top of the .cpp file. It includes some standard c++ headers and declears a few macros. The following section defines a couple of terminals: T_DISPLAY and T_LOAD as well as a few non-terminals: NT_DECNUMBER, NT_HEXNUMBER and NT_STRING.
The tricky part here is the fact that yylex() will be called with a pointer to an instance of a user defined class decleared in parser.y. This way we can store data in multiple formats within this object.

parser.y

The first section of the file includes appropriate C++ headers and defines the main fundamental class, that will be used for scanning and parsing. A pointer to an instance of this class will be passed into the scanning routine, so the members can be filled in appropriately to the token type.

Reloop digital jockey 2 interface edition traktor pro mapping. The second section defines the grammer required to parse our sophisticated commands and handlers that call appropriate engine routines.

Flex Bison Dev C++

Source files

Once you have managed to setup the builds, above commands would produce the following files: parser.cpp parser.hpp location.hh stack.hh and scanner.cpp These files need to be compiled as part of your project. If you are using a VC++ project or makefile, make sure that the compiler option called 'precompiled headers' is switched off for these files.

Additional code

We now have a working scanner and parser, but there are still several questions to be answered:
  • How do we feed data to the scanner?
  • How do we implement error handing?

Here is a block of code that declares a paser instance, and implements a simple error handler. error_() is called when a given string is not part of the specified language.

Now, this block defines a funtion that feeds a new command to the scanner. Imageine, that this routine is executed immediately after the user has typed a command.

Odds and ends

There is another block of code that one would need in order to compile the scanner. The hack described above depends on the structure called yy_buffer_state that is copied from the flex generated code. Note, it might vary from version to version.

Put this block of code into scanner.h and #include it:

[Prev in Thread]Current Thread[Next in Thread]
  • Flex and bison howto for C++ users,oleg smolsky<=
    • Re: Flex and bison howto for C++ users, Jibin Han, 2003/02/19
    • Re: Flex and bison howto for C++ users, Akim Demaille, 2003/02/20
    • Re: Flex and bison howto for C++ users, oleg smolsky, 2003/02/20
    • Re: Flex and bison howto for C++ users, oleg smolsky, 2003/02/20
    • RE: Flex and bison howto for C++ users, Xinan Tang, 2003/02/21
      • RE: Flex and bison howto for C++ users, Hans Aberg, 2003/02/22
  • Prev by Date:Problem
  • Next by Date:Read and write to files within Bison
  • Previous by thread:Problem
  • Next by thread:Re: Flex and bison howto for C++ users
  • Index(es):