Supported References

Search Back References

Re and Regex out of the box have very different feature sets. Backrefs tailors the supported features for each regular expression engine. For instance, Regex already supports Unicode properties, so Backrefs does not attempt to provide such support.

Re

Regex

Replace Back References

The replace back references below apply to both Re and Regex and are essentially non-specific to the regular expression engine being used. Casing is applied to both the literal text and the replacement groups within the replace template. In most cases you'd only need to wrap the groups, but it may be useful to apply casing to the literal portions if you are dynamically assembling replacement patterns.

Unicode Properties

A number of various Unicode properties are supported in Backrefs, but only for Re as Regex already has its own implementation of Unicode properties. Some properties may not be available on certain Python versions due to the included Unicode build.

It is important to note that Backrefs handles Unicode properties by transforming them to character classes with all the associated characters: \p{Cs} → [\ud800\udb7f-\udb80\udbff-\udc00\udfff]. Because of this, Backrefs can create really large regular expressions that the underlying engine must walk through. In short, Re with Backrefs will never be as efficient or fast as using Regex's Unicode properties, but it is very useful when you need or want to use Re.

Also, keep in mind that there are most likely some differences between Regex's Unicode Properties and Backrefs' Unicode properties. One notable difference is Regex does not currently implement script_extensions while Backrefs' does and uses them as the default when specifying them in the form \p{IsScriptValue} or \p{ScriptValue} just like Perl does. See Property Short Names for more info.

Exhaustive documentation on all these properties and their values is not currently provided. In general, we'll cover the syntax rules, and special short name handling to those rules for specific properties. Though we will outline all the values for General Category, we will not outline all of the valid values for the other properties. You can look at Perl's Unicode property documentation to get an idea of what values are appropriate for a given property, but keep in mind, syntax may vary from Perl's syntax.

Unicode properties are specific to search patterns and can be used to specify a request to match all the characters in a specific Unicode property. Unicode properties can be used in byte strings, but the patterns will be restricted to the ASCII range.

Additionally, Unicode properties are sensitive to the ASCII flag and will have their range limited to ASCII if used. The LOCALE flag is treated as ASCII in relation to Unicode properties.

Syntax

Unicode properties can be specified with the format: \p{property=value}, \p{property:value}. You can also do inverse properties by using the ^ character (\p{^property=value}) or by using a capital P (\P{property=value} or \P{property:value}).

Unicode properties may only have one property specified between the curly braces. If you want to use multiple properties to capture a singe character, create a character class: [\p{UnicodeProperty}\p{OtherUnicodeProperty}].

When specifying a property, the property and value matching is case insensitive and characters like [ -_] will be stripped out. So the following are all equivalent: \p{Uppercase_Letter}, \p{Uppercase-letter}, \p{UPPERCASELETTER}, \p{upper case letter}.

There are a number of binary properties. In general, binary properties are specified by stating the binary property and a boolean value. True values can be Yes, Y, True, or T. False values can be No, N, False, or F. For example, to specify characters that are "alphabetic", we can use \p{Alphabetic: Y}. To specify characters that are not "alphabetic": \p{Alphabetic: N}.

Property Short Names

General Category, Script Extensions, Blocks, and Binary all can be specified in a short form using just their name or alias: \p{value}, but they will be evaluated in the following order to resolve name conflicts as some the same value that is used in Script may be used in Blocks etc.

1. General Category
2. Script Extensions
3. Binary
4. Blocks

Script Extensions and Binary properties can also be defined in the format IsValue. For instance, if we wanted to match characters in the Latin script, we could use the syntax \p{IsLatin}, which would be the same as \p{Latin} or \p{scx: Latin}. For Binary properties, \p{IsAlphabetic} is the same as \p{Alphabetic: Y} or \p{Alphabetic}.

Block properties have a similar short form as Script and Binary properties. For Blocks you can use InValue to specify a block. If we wanted to match characters in the Basic_Latin block, we could use the syntax \p{InBasic_Latin}. This would be the same as \p{Block: Basic_Latin} or \p{Basic_Latin}.

/// warn | Short Name Conflicts When it comes to short names, each new Unicode version, there is a risk that new properties could cause conflicts with existing names and/or aliases. Currently, most of the conflicts involve the Block properties. To reduce friction, they are evaluated last.

Generally, it is discouraged to use short names for Block properties. But the option is still supported, but Block properties will be evaluated last. There are currently no known conflicts with In* properties, but in future Unicode versions there could.

As for short names for scripts, Binary, or General Categories, there is always the possibility that these could break in the future as well. Generally, more explicit is better and probably safer. ///

Lastly, you can specify general category properties in the form \pX where X is the single letter terse property form. In this form, you can only use the single character values. So you could specify Letter, whose terse form is L with \pL, but cannot specify Cased_Letter which has a terse form of Lc.

See the table below to see all the Unicode General Category values and their terse forms.

POSIX Style Properties

Backrefs allows for POSIX style properties in the form [:name:]. These properties can only be used inside character classes and are another form for expressing Unicode properties. Any Unicode property that can be expressed via the \p{name} form can also be expressed in the [:name:] form. To illustrate, the following are all the same:

• [[:upper:]] == [\p{upper}]
• [[:^upper:]] == [\p{^upper}]
• [[:alpha=yes:]] == [\p{alpha=yes}]

A number of POSIX property names are available via compatibility properties as outlined in the Unicode specification for POSIX compatibility. These properties will operate in the Unicode range and the ASCII range depending on the regular expression mode. These patterns, like all Unicode properties, are sensitive to the ASCII flag (or LOCALE which will treat them as ASCII).

It is important to note that whether used in the form [[:name:]] or \p{name}, each POSIX name is available both with and without the posix prefix, but it is recommended to use the posix prefix to get the POSIX definition of the pattern as number of patterns have both a POSIX and Unicode definition that differ. The Unicode specification for POSIX compatibility outlines all the POSIX compatible properties and the ones which have dual definitions: punct, alnum, digit, and xdigit all have a Unicode standard and a POSIX compatibility variant and must be accessed with the posix prefix to get the POSIX form.

In the table below, patterns with -- mean [[in this] -- [but not this]].

Compatibility Properties

Unicode specification for POSIX compatibility defines a number of properties, many of which double as Posix properties. These properties can be accessed via \p{name} or [[:name:]].

In the table below, patterns with -- mean [[in this] -- [but not this]].