Diacritics are not considered part of words

[xmo-odoo]

I'm not sure whether it's a bug or expected behaviour, but it seems odd so I figure reporting it is a good idea: while a precomposed character is considered "a word" by the regex engine (specifically \w), its decomposed form is not, because a diacritic is not considered part of a word.

>>> import re, unicodedata
>>> s = "ö"
>>> list(s)
['ö']
>>> list(unicodedata.normalize('NFD', s))
['o', '̈']
>>> re.fullmatch(r'\w+', s)
<re.Match object; span=(0, 1), match='ö'>
>>> re.fullmatch(r'\w+', unicodedata.normalize('NFD', s))

This leads to odd effects when ingesting and filtering decomposed data.

Tested on 3.8.13, 3.10.6, and 3.11.1 (all installed via pyenv), on a Mint 21.1).

[picnixz]

According to the official documentation concerning \w for Unicode (str) patterns,

Matches Unicode word characters; this includes alphanumeric characters (as defined by str.isalnum()) as well as the underscore (_). If the ASCII flag is used, only [a-zA-Z0-9_] is matched.

However, when using the normal form D (NFD or NFKD), the output is no more considered an alphanumeric character:

>>> s = 'ö'
>>> s.isalnum()
True
>>> s == unicodedata.normalize('NFC', s)
True
>>> unicodedata.normalize('NFD', s).isalnum()
False

IMHO, this does not conflict with the Python specifications, if we assume that the sentence "this includes [...] as well as [...]" defines what \w stands for. Note that there has been a very long discussion about whether Python should actually follow the Unicode specifications or not (#56940).