Python Code Generator for ANTLR 2.7.7

However, there a script is provided that let's you easily add antlr.py as module to your Python installation. After installation just run

${prefix}/sbin/pyantlr.sh install

Note that usually you need to be superuser in order to succeed. Also note that you can run this command later at any time again, for example, if you have a second Python installation etc. Just make sure that python is in your $PATH when running pyantlr.sh.

Note further that you can also do this to install ANTLR Python runtime immediatly after having called ./configure:

scripts/pyantlr.sh install

Specifying Code Generation

You can instruct ANTLR to generate your Lexers, Parsers and TreeParsers using the Python code generator by adding the following entry to the global options section at the beginning of your grammar file.

{
    language="Python";
}

After that things are pretty much the same as in the default java code generation mode. See the examples in examples/python for some illustrations.

One particular issue that is worth mentioning is the handling of comments in ANTLR Python. Java, C++, and C# all use the same lexical structures to define comments: // for single-line comments, and /* ... */ for block comments. Unfortunately, Python does not handle comments this way. It only knows about single-line comments, and these start off with a # symbol.

Normally, all comments outside of actions are actually comments in the ANTLR input language. These comments, and that is both block comments and single-line comments are translated into Python single-line comments.

Secondly, all comments inside actions should be comments in the target language, Python in this case. Unfortunately, if the actions contain ANTLR actions, such as $getText, the code generator seems to choke on Python comments as the # sign is also used in tree construction. The solution is to use Java/C++-style comments in all actions; these will be translated into Python comments by the ANTLR as it checks these actions for the presence of predefined action symbols such as $getText.

So, as a general issue: all comments in an ANTLR grammar for the Python target should be in Java/C++ style, not in Python style.

Python-Specific ANTLR Sections

• header - specify additional import directives

  You can instruct the ANTLR Python code generator to import additional Python packages in your generated Lexer/Parser/TreeParser by adding code to the header section which must be the first section at the beginning of your ANTLR grammar file, apart from any other header sections.

  header {
     import os, sys
  }
  

• header "__init__" - specify additional code in the __init__ method

  You can instruct the ANTLR Python code generator to include additional Python code in your generated Lexer/Parser/TreeParser by adding code to the init header section which must be the first section at the beginning of your ANTLR grammar file, apart from any other header sections. The code in the header is appended to the end of the __init__ method.

  header "__init__" {
     self.message = "This is the default message"
  }
  

  If your grammar file contains both a Lexer and a Parser (or any other multiple of definitions), the code in the __init__ header will be reproduced in the __init__ methods of all of these definitions without change. If you really want to update only one of the definitions, for instance, the __init__ method of the Lexer class you are creating, use

  header "<LexerGrammar>.__init__" {
     self.message = "This is the default message"
  }
  

  where <LexerGrammar> is the name of the Lexer grammar. The same construction also works with the Parsers and TreeParsers, of course.

  In the case both a generic init header and a grammar-specific header are present, the grammar-specific one will override the generic one.

• header "__main__" - specify additional code after the class definition

  You can instruct the ANTLR Python code generator to add additional Python code at the end of your generated Lexer/Parser/TreeParser, so after the class definition itself by adding code to the __main__ header section which must be the first section at the beginning of your ANTLR grammar file, apart from any other header sections.

  header "__main__" {
      print "You cannot execute this file!"
  }
  

  If your grammar file contains both a Lexer and a Parser (or any other multiple of definitions), the code in the __main__ header will be reproduced at the end of all of the generated class definitions. If you really want to add code after only one of the definitions, for instance, after the Lexer class, use

  header "<LexerGrammar>.__main__" {
      print "You cannot execute this file!"
  }
  

  where <LexerGrammar> is the name of the Lexer grammar. The same construction also works with the Parsers and TreeParsers, of course.

  In the case both a generic init header and a grammar-specific header are present, the grammar-specific one will override the generic one. If no __main__ headers are present and the grammar is for a Lexer, automated test code for that lexer is automatically added at the end of the generated module. This can be prevented by providing an empty __main__ header. In the latter case it is good practise to provide a comment explaining why an empty header is present.

  header "<LexerGrammar>.__main__" {
      // Empty main header to prevent automatic test code from being added
      // to the generated lexer module.
  }
  

  This automated test code can be executed by running Python with the generated lexer file (<LexerGrammar>.pywhere <LexerGrammar> is the name of the Lexer grammar) and providing some test input on stdin:

  python <LexerGrammar>.py < test.in
  

Python-Specific ANTLR Options

• className - change the default name of the generated class

  options {
      className="Scanner";
  }
  

  If you are using the className option conjunction with the Python specific header options, there will be no collisions. The className option changes the class name, while the main headers require the use of the grammar name which will become the module name after code generation.

  header "ParrotSketch.init" {
      self.state = JohnCleese.select("dead", "pushing up daisies", \
                                     "no longer", "in Parrot Heaven")
      print "This parrot is", self.state
  }
  
  class ParrotSketch extends Lexer;
  
  options {
      className="Scanner";
  }
  

A Template Python ANTLR Grammar File

As the handling of modules &emdash; packages in Java speak &emdash; in Python differs from that in Java, the current approach in ANTLR to call both the file and the class they contain after the name of the