9. Implementation Details

The string pattern matching in Mathematica is built on top of the PCRE (Perl Compatible Regular Expressions) library by Philip Hazel [1].

In some cases the pre-5.1 Mathematica algorithms are used (e.g., when the pattern is just a single, literal string).

Any symbolic string pattern is first translated to a regular expression. You can see this translation by using the internal StringPattern`PatternConvert function.

In[164]:=

Out[164]//InputForm=

The first element returned is the regular expression, while the rest of the elements have to do with conditions, replacement rules, and named patterns.

The regular expression is then compiled by PCRE, and the compiled version is cached for future use when the same pattern appears again. The translation from symbolic string pattern to regular expression only happens once.

Mathematica conditions in the pattern are handled by external call-outs from the PCRE library to the Mathematica evaluator, so this will slow down the matching.

Explicit RegularExpression objects embedded into a general string pattern will be spliced into the final regular expression (surrounded by noncapturing parentheses "(?:...)"), so the counting of named patterns can become skewed compared to what you might expect.

Because PCRE currently does not support preset character classes with characters beyond character code 255, the word and letter character classes (such as WordCharacter and LetterCharacter) only include character codes in the Unicode range 0-255. Thus LetterCharacter and _?LetterQ do not give equivalent results beyond character code 255.

Because of a similar PCRE restriction, case-insensitive matching (e.g., with IgnoreCase->True) will only apply to letters in the Unicode range 0-127 (i.e., the normal English letters "a"-"z" and "A"-"Z").