qazwsx qazwsx - 5 months ago 14
Python Question

In Python, how to list all characters matched by POSIX extended regex `[:space:]`?

In Python, how to list all characters matched by POSIX extended regex

[:space:]
?

Is there a programmatic way of extracting the Unicode code points covered by
[:space:]
?

Answer

Using a generator instead of a list comprehension, and xrange instead of range:

>>> s = u''.join(unichr(c) for c in xrange(0x10ffff+1))
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "<stdin>", line 1, in <genexpr>
ValueError: unichr() arg not in range(0x10000) (narrow Python build)

Whoops: in general use sys.maxunicode.

>>> s = u''.join(unichr(c) for c in xrange(sys.maxunicode+1))
>>> import re
>>> re.findall(r'\s', s)
[u'\t', u'\n', u'\x0b', u'\x0c', u'\r', u' ']

Whoops: Ummm what about "no-break space" etc?

>>> re.findall(r'\s', chrs, re.UNICODE)
[u'\t', u'\n', u'\x0b', u'\x0c', u'\r', u'\x1c', u'\x1d', u'\x1e', u'\x1f', u' '
, u'\x85', u'\xa0', u'\u1680', u'\u180e', u'\u2000', u'\u2001', u'\u2002', u'\u2
003', u'\u2004', u'\u2005', u'\u2006', u'\u2007', u'\u2008', u'\u2009', u'\u200a
', u'\u2028', u'\u2029', u'\u202f', u'\u205f', u'\u3000']

What is all that stuff? unicodedata.name is your friend:

>>> from unicodedata import name
>>> for c in re.findall(r'\s', chrs, re.UNICODE):
...     print repr(c), name(c, '')
...
u'\t'
u'\n'
u'\x0b'
u'\x0c'
u'\r'
u'\x1c'
u'\x1d'
u'\x1e'
u'\x1f'
u' ' SPACE
u'\x85'
u'\xa0' NO-BREAK SPACE
u'\u1680' OGHAM SPACE MARK
u'\u180e' MONGOLIAN VOWEL SEPARATOR
u'\u2000' EN QUAD
u'\u2001' EM QUAD
u'\u2002' EN SPACE
u'\u2003' EM SPACE
u'\u2004' THREE-PER-EM SPACE
u'\u2005' FOUR-PER-EM SPACE
u'\u2006' SIX-PER-EM SPACE
u'\u2007' FIGURE SPACE
u'\u2008' PUNCTUATION SPACE
u'\u2009' THIN SPACE
u'\u200a' HAIR SPACE
u'\u2028' LINE SEPARATOR
u'\u2029' PARAGRAPH SEPARATOR
u'\u202f' NARROW NO-BREAK SPACE
u'\u205f' MEDIUM MATHEMATICAL SPACE
u'\u3000' IDEOGRAPHIC SPACE