Any regular character can be a primitive. This includes letters (like E) and digits. For instance, in the following example the capital E is the primitive:

    E{4}

What is the primitive here:

    4{2}

The correct answer is 4.

Other characters can also be used as primitives in a Regular Expression. For example:

    ls -la | egrep ' {11}'

(the Regular Expression is a space followed by multiplier {11}). This will show lines containing at least 11 consecutive spaces.

Some characters have special meanings, such as the curly brace ({). These characters must be escaped (they are preceded by a backslash (\)) to remove the special meaning. This will be discussed further later on.

The dot has a special meaning. As a primitive, it can be a placeholder to allow any character in that position. For instance:

    a.c

allows for one (remember the one-time multiplier being added?) a, followed by any single character, followed by one c. So

    echo reading the abc | egrep 'a.c'

fits.

And what will this fit on?

    a.{3}

The answer: one a followed by three arbitrary characters.

A character class is a placeholder for a number of characters. It is more restrictive than the dot placeholder, since it allows you to specify the characters the placeholder should match.

Keep in mind that the character class is a primitive, it is a placeholder for one of the specified characters.

To set up a character class, start with a [ (opening square bracket) and end it with ] (closing square bracket). Any combination of sets and ranges can be specified between these square brackets.

    [aouiyA]

    [aouiyA]{2}

    echo reading the abc | egrep 'a[bB]c'

    echo reading the aBc | egrep 'a[bB]c'

    [0-9]

    [a-z]

    [a-z0-9]

    [ghiGHI]{2}

    [g-iG-I]{2}

    [a-z0-9]

    [aouiy0-9]

    [A-Z][aouiy0-9][A-Z]

    echo This is JaN | egrep '[A-Z][aouiy0-9][A-Z]'

    [0-9]

    [^0-9]

    [^aeouy0-9]

The POSIX character classes form an extension to the character classes that were discussed above. They come in the following format:

The keyword is a word describing the class, for example, alnum or blank. Current implementations of GNU grep and egrep, as well as GNU awk (gawk), come with manpages that describe the POSIX character classes. Let's look at some examples.

[[:alnum:]]

[a-zA-Z0-9]

[[:alpha:]]

[[:upper:]]

[A-Z]

[[:lower:]]

[a-z]

[[:digit:]]

[0-9]

[[:xdigit:]]

[0-9A-Fa-f]

[[:blank:]]

[[:space:]]

[[:punct:]]

Multiple POSIX-character-set definitions can be combined in one character class:

    [[:punct:][:space:]]

This one fits a character which is either a punctuation character (such as a semicolon or a plus sign) or a white-space character.

The POSIX character set specifiers can be combined with any other character-class element, as shown below:

    [0-9]{4}[[:space:],\.:][0-9]{2}

In words: four digits, then one character that can be either a white-space character, a comma, a dot or a colon, followed by two digits.

Since the POSIX character class extensions are extensions to the regular character class, it can be inverted the same way:

    [^[:digit:]]

This matches one non-digit. It is identical to:

    [^0-9]

The * is the oldest multiplier around. As far as I know, it is implemented in every Regular Expression language. It is the same in classic and extended variants.

The * means: zero or more times. So,

    a*

means zero or more a's. This may not be that meaningful, since it is always true. In larger combinations, however, its use can be more meaningful:

    ba*c

That is, one b, and one c, possibly with a's between them.

The ? multiplier means zero or one time, in other words, present or not present. The extended version is ?, the version to use in classic Regular Expressions is \?.

An example: in sgml/xml sections start with <section followed by a lot of things, and end with </section, also followed by other things. To fetch both lines, use

    egrep '</?section' regexp.xml

The classic variant is

    grep '</\?section' regexp.xml

The + multiplier means one or more times. The extended version is +, the version to use in classic Regular Expressions is \+.

This (extended) Regular Expression

    ba+c

means one b, followed by at least one a, followed by a c.

To find long, regardless of how may o's it contains, use the (extended) Regular Expression

    lo+ng

In the extended variant, a normal set of curly braces is used, as in {4}. In the classic variant, a matching pair of \{ and \} (each curly brace is prefixed by a backslash) is used, as in \{4\}.

The curly-brace multipliers come in four variants (only the extended Regular Expressions will be shown). The general form is

In this, both m and n are numbers indicating the lowest (m) and the highest (n) number. It matches at least m, but at most n times. For instance, to match at least two, but at most four digits:

    [0-9]{2,4}

To specify a minimum number of times, omit the n (but not the comma!):

For instance, to match at least five digits:

    [0-9]{5,}

And to show lines that are at least 78 characters long (show also line numbers), select

    egrep -n '.{78,}' file1 file2 ...

To specify a maximum number of times, omit the m (but not the comma!):

For instance, to match up to three digits:

    [0-9]{,3}

In this case, the minimum will be zero.

To match something no more and no less than n times, specify only n (do not use a comma):

To match exactly four capital alphabetic characters, for instance:

    [A-Z]{4}

Support and portability.  Curly-brace multipliers are supported by Perl. In GNU awk, support must be enabled explicitly with the --posix option; other awk implementations might not (yet) support these multipliers (called interval expressions in awk terminology). GNU egrep will support them. GNU grep supports the classic version. In other versions of (e)grep, support may be lacking. GNU sed supports the classic implementation. There are implementations of sed that do not support curly-brace multipliers at all. In yet other implementations of sed, extended Regular Expressions can be selected. POSIX standards are said to require curly-brace multipliers, so support is expected to be implemented elsewhere over time. Table 13.3, “Portable multipliers” provides an overview.

Table 13.2, “Multipliers” shows all multipliers and variants together.

Portability.  If your Regular Expressions should also be used on non-GNU platforms portability becomes an issue. Old grep's will probably not support the \+ and \? multipliers. Curly-brace multipliers (called interval expression in awk terminology) will probably not be supported in other awks (though they are reported to be part of the POSIX standard). Even GNU awk (gawk) does not enable interval expressions by default: they can only be enabled by invoking either the --posix or the --re-interval option. Table 13.3, “Portable multipliers” lists multipliers that you can use safely across platforms. But even here, exceptions do exist.

When an anchor is used in a Regular Expression, the Regular Expression will only match if the string is in the position specified by the anchor.

Generally, newlines are stripped before the match is attempted. This is true for at least awk, sed and all grep variants. But, in Perl, you must either remove the newline using the chomp operator or include it in the Regular Expression. Furthermore, if the so-called multi-line mode is turned on, the meaning of begin and end anchors changes altogether in Perl. More in the section called “Perl Regular Expressions”.

Portability.  Begin and end anchors are supported in all Regular Expression variants and use the same notation everywhere.

Let's look at the begin and end anchors in more detail.

    echo Once upon ... | grep '^Once'

    echo He said: Once ... | grep '^Once'

    ls -lF | grep '/$'

    pwd | grep '/$'

    ^[0-9]+$'

    egrep -v '^ *$' listofnames

Word-boundary anchors fit between a word and something else. In most Regular Expressions word is defined as a series of alphanumeric characters. So, a word boundary fits between an alphanumeric character and a non-alphanumeric character. Furthermore, a word-boundary anchor will also fit between an alphanumeric character and the beginning or end (where ^ and $ would match).

There are two variants of word-boundary anchors. The first is \b, which matches on either side of a word. The second variant consists of two anchors: \< and \>. The \< anchor fits either between a non-alphanumeric character and an alphanumeric character or before the beginning of a line if the line starts with an alphanumeric character (like ^). The \> anchor fits either between an alphanumeric character and a non-alphanumeric character or after an alphanumeric character at the end of a line (like $). The spelling of word-boundary anchors is the same everywhere.

Portability.  Word-boundary anchors are not supported widely, but can be found in all GNU implementations of the programs. The only implementation you can really be sure of is \b in Perl.

Let's look at word-boundary anchors in more detail.

    echo The one and only | grep '\bone'

    echo gone with the wind | grep '\bone'

    echo printed onesided | grep 'one\b'

    \b[0-9]{6-10}\b

    echo The onesided page is gone | grep '\<one\>'

Table 13.4, “Anchors” provides an overview of anchors and their support.

A multiplier can be applied to a group. Simply put the multiplier right after the closing parenthesis of a group. To find a representation of time in the typical format two digits - colon - two digits - colon - two digits (such as 14:10:17), for example, use the following (extended) Regular Expression:

    ([0-9]{2}:){2}[0-9]{2}

To find all words with “ing” twice (such as singing), use this extended Regular Expression:

    (ing){2}

If a group is not followed by a multiplier, then multiplication by one is chosen automatically.

The part of the input on which a group matches comes available for re-use. A special mechanism for backward referencing, called a backreference is available for this. The backreference corresponding to the first group in a Regular Expression is indicated with “\1” (a backslash and the digit one), the one corresponding to the second group \2, and so on. If, for example, you are looking for the occurrence of four identical digits, you might use:

    ([0-9])\1\1\1

Suppose this is tried on:

    The 1-2-3 of 4444

First, the RE finds 1 as the first digit. There is a group around it, so the value is saved. In the Regular Expression there is a \1 after the group. This tells the Regular Expression engine to use the same value (digit 1) again. This fails, since there is a dash after the 1. This sequence is repeated for the 2 and the 3, but both fail. However, when it finds the 4 it than finds the other three 4s, so the Regular Expression matches at last.

In an earlier paragraph I needed to find words that contain twice the same three characters, like singing. Here is how I found words like that:

    egrep '(...)\1' /usr/share/dict/american-english

(I know, I should have used [A-Za-z]{3} instead of the dots. Since there are only letters in the file, dots work as well).

Extracting the matching part (or even subparts) using grouping is very important in Perl. More in the section called “Perl Regular Expressions”.

Alternation can be used inside a group. Then, the alternation applies only to the part in the group.

Suppose the mobile phone numbers should be at the beginning of the input line. Then we'll we need to use an anchor. But if we use an anchor followed by the existing Regular Expression (^06|...), then the anchor applies only to the first number. The following is the correct solution:

    egrep '^(06|00316|\+31)' phonenumbers

Another example: Find numbers in the range 1 .. 12. Leading zeroes are not allowed. Solution (extended Regular Expression):

    \b([1-9]|1[0-2])\b

In words: one digit in the range 1-9 or two digits in the range 10-12.

In some classic Regular Expression variants (notably sed), there is no easy primitive for a tab character, let alone for white space. It is probably best to refrain from using classic Regular Expressions for tab handling. Two workarounds are available.

    ... | tr '\011' ' ' | sed ...

    grep -v '^[[:blank:]]*#' whatever.conf

The extended Regular Expressions have the \t notation for a tab character. With that, it is easy to build a white-space character class:

    [ \t]

This means: either a space or a tab character. Applied:

    [0-9]{1,2}[ \t]+[A-Z]

That is, one or two digits, followed by one or more white-space characters and an uppercase letter.

POSIX character classes are used exactly the same way as in classic Regular Expressions (above).

If you can use Perl for white-space handling, you are lucky: Perl has the \s primitive that matches a white-space character. Applied:

    [0-9]{1,2}\s+[A-Z]

Or, to see lines starting with a # possibly preceded by white space:

    ^\s*#

For more on Perl Regular Expressions, see the section called “Perl Regular Expressions”.

After a successful match, the three related pseudo variables $`, $& and $' will contain the following:

An example:

    $_ = 'This is a nice book';

    if (/\w+ic\w+/)
    {
        print 
            "Input:             \"$_\"\n",
            "Part before match: \"$`\"\n",
            "Matching part:     \"$&\"\n",
            "Part after match:  \"$'\"\n";
    }

The output of this program part is:

    Input:             "This is a nice book"
    Part before match: "This is a "
    Matching part:     "nice"
    Part after match:  " book"

Note that the contents of $` has an extra trailing space and that of $' an extra leading space.

In the Perl Regular Expression language, one can use grouping to extract matching parts or even subparts from the input.

Suppose the input is:

    Today is  7 January 2002, a monday.

The following Perl Regular Expression matches the complete date in the input:

    \b\d{1,2}\s+[A-Za-z][a-z]+\s+\d{4}\b

In Perl, you can also group the day, the month and the year parts:

    \b(\d{1,2})\s+([A-Za-z][a-z]+)\s+(\d{4})\b

When this Regular Expression is matched against the input, three pseudo variables, namely $1, $2 and $3 will be filled-in with the corresponding parts of the date. The information corresponding to the first group (seen from the left) is inserted in $1, the second in $2, and so on:

     $_ = "Today is 7 January 2002, a monday.";

     if (/\b(\d{1,2})\s+([A-Za-z][a-z]+)\s+(\d{4})\b/)
     {
         print "Day:   $1\n",
               "Month: $2\n",
               "Year:  $3\n";
     }

Nested groups.  Groups can be nested:

     \b((\d{1,2})\s+([A-Za-z][a-z]+)\s+(\d{4}))\b

Information corresponding to the utmost group (here: the whole date) gets into $1. The parts inside that group fall into $2, $3 and so on. To see this working, use something like this:

     $_ = "Today is 7 January 2002, a monday.";

     if (/\b((\d{1,2})\s+([A-Za-z][a-z]+)\s+(\d{4}))\b/)
     {
         print "Date:  $1\n",
               "Day:   $2\n",
               "Month: $3\n",
               "Year:  $4\n";
     }

There is an alternative to the use of the pseudo variables ($1 etc.). The parts of the input that correspond to the groups in the Regular Expression can be written in an array. In the program part below, these subparts of the input are stored in the array @results:

    my $input = "Today is 7 January 2002, a monday.";

    my @result = $input =~ /\b((\d{1,2})\s+([A-Za-z][a-z]+)\s+(\d{4}))\b/;

    if (@result)    # if RE fits
    # etc

It is important to realize that

    $input =~ /RE/

actually is a statement that returns an array. To capture this, put an array assignment before it:

    @result = $input =~ /RE/

This really is correct Perl! Remember that the =~ operator is called the binding operator. It is not an assignment.

This program part puts everything in perspective:

    my $input = "Today is 7 January 2002, a monday.";

    my @result = $input =~ /\b((\d{1,2})\s+([A-Za-z][a-z]+)\s+(\d{4}))\b/;

    if (@result)    # if RE fits
    {
        for (my $i = 0; $i < scalar @result; $i++)
        {
            print "Field $i: $result[$i]\n";
        }
    }

Elements of the array will be filled in the same order the pseudo variables were assigned. The first element, $results[0], will get the complete date, $results[1] will get the day, $results[2] the month and $results[3] the year.