### 8.4 Changing the lexical structure of words ¶

> The macro `changeword` and all associated functionality is experimental. It is only available if the `--enable-changeword` option was given to `configure`, at GNU `m4` installation time. The functionality will go away in the future, to be replaced by other new features that are more efficient at providing the same capabilities. Do not rely on it. Please direct your comments about it the same way you would do for bugs.

A file being processed by `m4` is split into quoted strings, words (potential macro names) and simple tokens (any other single character). Initially a word is defined by the following regular expression:

    [_a-zA-Z][_a-zA-Z0-9]*

Using `changeword`, you can change this regular expression:

Optional builtin: changeword (regex) ¶

  Changes the regular expression for recognizing macro names to be regex. If regex is empty, use ‘`[_a-zA-Z][_a-zA-Z0-9]*`’. regex must obey the constraint that every prefix of the desired final pattern is also accepted by the regular expression. If regex contains grouping parentheses, the macro invoked is the portion that matched the first group, rather than the entire matching string.

  The expansion of `changeword` is void. The macro `changeword` is recognized only with parameters.

Relaxing the lexical rules of `m4` might be useful (for example) if you wanted to apply translations to a file of numbers:

    ifdef(`changeword', `', `errprint(` skipping: no changeword support
    ')m4exit(`77')')dnl
    changeword(`[_a-zA-Z0-9]+')
    ⇒
    define(`1', `0')1
    ⇒0

Tightening the lexical rules is less useful, because it will generally make some of the builtins unavailable. You could use it to prevent accidental call of builtins, for example:

    ifdef(`changeword', `', `errprint(` skipping: no changeword support
    ')m4exit(`77')')dnl
    define(`_indir', defn(`indir'))
    ⇒
    changeword(`_[_a-zA-Z0-9]*')
    ⇒
    esyscmd(`foo')
    ⇒esyscmd(foo)
    _indir(`esyscmd', `echo hi')
    ⇒hi
    ⇒

Because `m4` constructs its words a character at a time, there is a restriction on the regular expressions that may be passed to `changeword`. This is that if your regular expression accepts ‘`foo`’, it must also accept ‘`f`’ and ‘`fo`’.

    ifdef(`changeword', `', `errprint(` skipping: no changeword support
    ')m4exit(`77')')dnl
    define(`foo
    ', `bar
    ')
    ⇒
    dnl This example wants to recognize changeword, dnl, and `foo\n'.
    dnl First, we check that our regexp will match.
    regexp(`changeword', `[cd][a-z]*\|foo[
    ]')
    ⇒0
    regexp(`foo
    ', `[cd][a-z]*\|foo[
    ]')
    ⇒0
    regexp(`f', `[cd][a-z]*\|foo[
    ]')
    ⇒-1
    foo
    ⇒foo
    changeword(`[cd][a-z]*\|foo[
    ]')
    ⇒
    dnl Even though `foo\n' matches, we forgot to allow `f'.
    foo
    ⇒foo
    changeword(`[cd][a-z]*\|fo*[
    ]?')
    ⇒
    dnl Now we can call `foo\n'.
    foo
    ⇒bar

`changeword` has another function. If the regular expression supplied contains any grouped subexpressions, then text outside the first of these is discarded before symbol lookup. So:

    ifdef(`changeword', `', `errprint(` skipping: no changeword support
    ')m4exit(`77')')dnl
    ifdef(`__unix__', ,
          `errprint(` skipping: syscmd does not have unix semantics
    ')m4exit(`77')')dnl
    changecom(`/*', `*/')dnl
    define(`foo', `bar')dnl
    changeword(`#\([_a-zA-Z0-9]*\)')
    ⇒
    #esyscmd(`echo foo \#foo')
    ⇒foo bar
    ⇒

`m4` now requires a ‘`#`’ mark at the beginning of every macro invocation, so one can use `m4` to preprocess plain text without losing various words like ‘`divert`’.

In `m4`, macro substitution is based on text, while in TeX, it is based on tokens. `changeword` can throw this difference into relief. For example, here is the same idea represented in TeX and `m4`. First, the TeX version:

    \def\a{\message{Hello}}
    \catcode`\@=0
    \catcode`\\=12
    @a
    @bye
    ⇒Hello

Then, the `m4` version:

    ifdef(`changeword', `', `errprint(` skipping: no changeword support
    ')m4exit(`77')')dnl
    define(`a', `errprint(`Hello')')dnl
    changeword(`@\([_a-zA-Z0-9]*\)')
    ⇒
    @a
    ⇒errprint(Hello)

In the TeX example, the first line defines a macro `a` to print the message ‘`Hello`’. The second line defines `@` to be usable instead of `\` as an escape character. The third line defines `\` to be a normal printing character, not an escape. The fourth line invokes the macro `a`. So, when TeX is run on this file, it displays the message ‘`Hello`’.

When the `m4` example is passed through `m4`, it outputs ‘`errprint(Hello)`’. The reason for this is that TeX does lexical analysis of macro definition when the macro is defined. `m4` just stores the text, postponing the lexical analysis until the macro is used.

You should note that using `changeword` will slow `m4` down by a factor of about seven, once it is changed to something other than the default regular expression. You can invoke `changeword` with the empty string to restore the default word definition, and regain the parsing speed.