TextTransformer is a visual development environment to develop text parsers. It lets to develop text parsers and either to store them as project files, or to export them into C++ code. There is also a possibility to implement into parsers semantic actions – scripts in C++-like language. It allows to create with the help of Text Transformer functionally complete programs to analyze and process texts.

Analysis and processing of texts are among the most frequently fulfilled applied tasks. Some examples of such tasks are search for all links in an HTML document, verification of data in a specific format, or translation of a program code from one programming language into another. To solve the tasks there exist special programs called parsers.

Parser is a program carrying out syntax analysis of a text, i.e. recognizing different elements of a text with respect to a determined set of rules. For example, there is a text containing some phone numbers:

(800) 555-7890
(391) 255-5555
(321) 321-3211

We determine a phone number as a sequence of figures consisting of a three-digit area code, enclosed in parentheses, followed by a three-digit prefix, followed by a dash, followed by a four-digit suffix.

Or formally:

PhoneNumber := "(" AreaCode ")" Prefix "-" Suffix
AreaCode := Digit{3}
Prefix := Digit{3}
Suffix := Digit{4}

Let us suppose, there is a program analyzing a text, resuming if the text is a phone number, and, if so, returning its area code, prefix, and suffix. Such a program is called a parser of phone number, the analysis is syntax analysis, and the set of rules is grammar.

Of course, a parser itself can be preset when we define grammar. That is why it's possible to create special programs generating parsers. TextTransformer is one of such programs. It's a visual development environment to develop text parsers.

TextTransformer lets to develop text parsers and either to store them as project files, or to export them into C++ code. There is also a possibility to implement into parsers semantic actions – scripts in C++-like language. It allows to create with the help of Text Transformer functionally complete programs to analyze and process texts.

TextTransformer's interface is quite complicated, not intuitive and abounds in various shortcomings. Though, it in no way affects the application's stable performance.

The interface consists of some bookmarks. The first of them – "Tetra" includes 2 windows: one to enter an initial text, another – a standard output.

TextTransformer has an embedded wizard which is, from my point of view, non-trivial and doesn't help to simplify the task of a new project creation.

There also realized support for transformation of large amount of files. The Transformation Manager tool helps to solve the task.

The next 2 bookmarks: Tokens and Productions serve to determine grammar.

In the TextTransformer program, grammar is determined in the terms of Production rules and Tokens. A token is a meaningful part of an initial text which can't be divided into smaller parts. Examples of tokens can be keywords, punctuation symbols, or fragments of a static text. For instance, for the case of phone numbers we can determine 3 tokens: AreaCode, Prefix and Suffix.

A production rule determines that grammar elements are either tokens, or compound element composed of other grammar elements by means of different operations.

In the TextTransformer program, production rules are written in accordance with Extended Backus-Naur form, and tokens – with the help of regular expressions.

While setting rules in TextTransformer we can use specific key rules (SKIP, EOF, BREAK, EXIT) and control structions (IF … ELSE … END, WHILE … END).

To parse the list of phone numbers, we can set the following rules:

TextTransformer provides a possibility to customize a parser adding to it variables and methods. To solve the task there exists the third bookmark – Interpreter.

But, in my opinion, the option of adding new elements is very inconvenient in use. For example, a variable differs from a method only in a check in a checkbox. If, adding a new variable, you forget to check the box, it might cause some future troubles.

What is more, on the list of elements, variables and methods don't differ visually, and while working on a project we have to inspect every element and their checkbox. Though, of course, the problem can be solved by keeping to some convention for naming of variables and methods. For example, we can name all variables using the m_ prefix.

Determined here variable and methods are available in semantic actions – scripts which can be applied to tokens and production rules. For example, a parser of phone numbers can count the total amount of phone numbers, the amount of phone numbers for a certain area, etc.

A distinctive feature of the TextTransformer program is the possibility to create unreachable production rules that are the rules which can't be derived from the main rule of a text analysis.

This allows to create additional production rules able to process the data previously aggregated by a parser. Thus, this allows to use one parser to fulfill several different tasks.

The next bookmark – the Test bookmark lets to create a set of parser tests. This serves to develop production rules more effectively.

TextTransformer can export a parser into C++ code as a class. To set templates of created header and source files, use the Editor bookmark.

It should be mentioned that TextTransformer makes certain demands to processed grammars. For example, a production rule can't be determined recursively, and when analyzing alternatives, a parser's choice is bases on a single look-ahead symbol only. The latter feature is called LL(1) feature, and the parser itself is LL(1) parser.

Such restrictions make impossible processing a recursive grammar describing mathematical expressions of whole numbers with parenthesis and operations "+" and "*".

E := E "+" T
T := T "*" P
P := [0-9]+
P := "(" E ")"

In TextTransformer we'll have to describe such expressions with the help of another grammar, for example, like that:

E := T
E := "+" T
T := F
T := "*" F
F := [0-9]+
F := "(" E ")"

TextTransformer provides good possibilities to set semantic actions. Along with embedded functions, there exist classes to work with strings, to format output, as well as iterators, data containers, and classes allowing to work with parser's state dynamically.

One of the most interesting is the container for functions – function table. It serves to effectively process complicated data structures, such as trees.

The support of dynamic work with parser provides a number of useful possibilities. A fundamental one is the possibility to add new tokens what is necessary, for example, when parsing C-code, particularly when processing the appearance of new data types.

In the program there realized an interactive debugger with stepping and breakpoints. TextTransformer also provides tools to analyze the debugging process, such as Variable Inspector.

There exists a command line version of TextTransformer – tetra_cl.exe. But it is not presented on a default distribution, and you'll have to download it from the developers' website. Besides, the Help section doesn't mention the fact.

A weak point of the CLI version is the impossibility to set a starting rule. It becomes essential when a parser contains several additional rules (for different tasks) for processing previously aggregated information, such as statistical output or conversion of data into another format. As it's impossible to launch a parser with the help of a command line and set a starting rule, it's impossible to determine what task we are going to fulfill.

Unfortunately, we couldn't test a generated by TextTransformer C++ code, as the option is available in the registered version only.

A bad, from my point of view, idea is to mark with different colors parts of the Help section related to different product versions. As a result, we can observe whole chapters written in green on the white, what is very inconvenient indeed.

In general, TextTransformer, in my opinion, is worse than its paid analogs, but it's cheaper and that's why might be interesting for those dealing with parsers creation Though, it's not for specialists solving more profound tasks, for example, developing compilers.

TextTransformer is a shareware product. Restrictions on the trial version are impossibility to export parsers into C++ class, and deactivation of some features after 30 days of use.

TextTransformer keywords: parser generator, parsergenerator, interpreter, programming, text processing, parser, scanner, debugger, transformation, conversion, translation, regular expression, regular expressions, reengineering, refactoring, boost