Module regex_automata::util::alphabet

This module provides APIs for dealing with the alphabets of finite state machines.

There are two principal types in this module, ByteClasses and Unit. The former defines the alphabet of a finite state machine while the latter represents an element of that alphabet.

To a first approximation, the alphabet of all automata in this crate is just a u8. Namely, every distinct byte value. All 256 of them. In practice, this can be quite wasteful when building a transition table for a DFA, since it requires storing a state identifier for each element in the alphabet. Instead, we collapse the alphabet of an automaton down into equivalence classes, where every byte in the same equivalence class never discriminates between a match or a non-match from any other byte in the same class. For example, in the regex [a-z]+, then you could consider it having an alphabet consisting of two equivalence classes: a-z and everything else. In terms of the transitions on an automaton, it doesn’t actually require representing every distinct byte. Just the equivalence classes.

The downside of equivalence classes is that, of course, searching a haystack deals with individual byte values. Those byte values need to be mapped to their corresponding equivalence class. This is what ByteClasses does. In practice, doing this for every state transition has negligible impact on modern CPUs. Moreover, it helps make more efficient use of the CPU cache by (possibly considerably) shrinking the size of the transition table.

One last hiccup concerns Unit. Namely, because of look-around and how the DFAs in this crate work, we need to add a sentinel value to our alphabet of equivalence classes that represents the “end” of a search. We call that sentinel Unit::eoi or “end of input.” Thus, a Unit is either an equivalence class corresponding to a set of bytes, or it is a special “end of input” sentinel.

In general, you should not expect to need either of these types unless you’re doing lower level shenanigans with DFAs, or even building your own DFAs. (Although, you don’t have to use these types to build your own DFAs of course.) For example, if you’re walking a DFA’s state graph, it’s probably useful to make use of ByteClasses to visit each element in the DFA’s alphabet instead of just visiting every distinct u8 value. The latter isn’t necessarily wrong, but it could be potentially very wasteful.

Structs

• ByteClassElements
  An iterator over all elements in an equivalence class.
• ByteClassIter
  An iterator over each equivalence class.
• ByteClassRepresentatives
  An iterator over representative bytes from each equivalence class.
• ByteClasses
  A representation of byte oriented equivalence classes.
• Unit
  Unit represents a single unit of haystack for DFA based regex engines.