Regular expressions

Similar to wildcard patterns on the command line, such as ls *.py, which means list all filenames ending in .py.

More complex pattern matching is regular expressions. These are provided in the standard module re, which we’ll import. You define a string pattern that you want to match, and the source string to match against.

Match object instances also have several methods and attributes; the most important ones are:

Method/Attribute Purpose
group() Return the string matched by the RE
start() Return the starting position of the match
end() Return the ending position of the match
span() Return a tuple containing the (start, end) positions of the match

 

Exact Match with match()

Match() works only if the pattern is at the beggining of the source.

For simple matches, usage looks like this:
result = re.match('You', 'Young Frankenstein')
Here, ‘You’ is the pattern and ‘Young Frankenstein’ is the  source—the string you want to check.

match() checks whether the source begins with the pattern.
For more complex matches, you can compile your pattern first to speed up the match later:
youpattern = re.compile('You')

Then, you can perform your match against the compiled pattern:

>>>result = youpattern.match('Young Frankenstein')
>>>if result:
 print(result.group())

You

If you want to search a pattern not at the beggining, we can use regular expression to overcome this:

• . means any single character.
• * means any number of the preceding thing.  Together, .* mean any number of characters (even zero).

>>> m = re.match('.*Frank', source)
 >>> if m:  # match returns an object
 ...     print(m.group())
 Young Frank

This time it returns all the characters until the “frank”.

Group() and groups()

When using match() or search(), all matches are returned from the result object m as m.group().  If you enclose a pattern in parentheses, the match will be saved to its own group, and a tuple of them will be available as m.groups(), as shown here:

 >>> source = '''I wish I may, I wish I might
... Have a dish of fish tonight.''' 
 >>>m = re.search(r'(. dish\b).*(\bfish)', source)
 >>> m.group()
 'a dish of fish'
 >>> m.groups()
 ('a dish', 'fish')
First match with Search()

Match() works only if the pattern is at the beginning of the source. Search() find the first match pattern if the pattern is anywhere.

 

All matches with findall()

Find all will show how many instances in the source, and return a list. You can use len(list_name) to know how many were found.

E.g. if you want to know how many instances of the single-letter string ‘n’ are in the string?

 >>> m = re.findall('n', source)
 >>> m   # findall returns a list
 ['n', 'n', 'n', 'n']
 >>> print('Found', len(m), 'matches')
 Found 4 matches

How about ‘n’ followed by any character( not include empty)?

 >>> m = re.findall('n.', source)
 >>> m
 ['ng', 'nk', 'ns']

Notice that it did not match that final ‘n’ because there is no character after the n. We need to say that the character after ‘n’ is optional with ?:

 >>> m = re.findall('n.?', source)
 >>> m
 ['ng', 'nk', 'ns', 'n']
Split at matches with split()

we can split a string into a list according to our need, such as by space, comma, semi-column or other symbol, letter.

string1='this is a test'

m=re.split(' ',string1)

print(m)

output

['this', ' is', 'a', 'test']
Replace at matches with sub()

 

This is like the string replace() method, but for patterns rather than literal strings:

 >>> m = re.sub('n', '?', source)
 >>> m # sub returns a string
 'You?g Fra?ke?stei?'
Special characters

 

The basics:

• Literal matches with any non-special characters
• Any single character except \n with .
• Any number (including zero) with *
• Optional (zero or one) with ?

Pattern Matches
\d a single digit
\D a single non-digit
\w an alphanumeric character 0-10,A-Z, and _
\W a non-alphanumeric character
\s a whitespace character
\S a non-whitespace character
\b a word boundary (between a \w and a \W, in either order)
\B a non-word boundary

 

 

string module

The python string module has predefined strings for testing.

e.g.

import string

printable_str=string.printable

print(len(printable_str))

print(printable_str[0:10],printable_str[10:36])

output

100

0123456789 abcdefghijklmnopqrstuvwxyz

To find all digits:

digits_all=re.findall('\d',printable_str)

print(digits_all)

To find all digits, letters, or an underscore

re.findall('\w', printable)
Pattern specifiers

The characters ^ and $ are called anchors:

  • ^ anchors the search to the beginning of the search string,
  • $ anchors it to the end. .$ matches any character at the end of the line, including a period,
Pattern: Matches
 abc:  literal abc
 ( expr ) :  expr
 expr1 | expr2 : expr1 or expr2
 . : any character except \n
 ^ : start of source string
 $ : end of source string
 prev ? : zero or one prev
 prev * : zero or more prev, as many as possible
 prev *? : zero or more prev, as few as possible
 prev + : one or more prev, as many as possible
 prev +? :one or more prev, as few as possible
 prev { m } : m consecutive prev
 prev { m, n } : m to n consecutive prev, as many as possible
 prev { m, n }? : m to n consecutive prev, as few as possible
 [ abc ] : a or b or c (same as a|b|c)
 [^ abc ] : not (a or b or c)
 prev (?= next ) : prev if followed by next
 prev (?! next ) : prev if not followed by next
 (?<= prev ) : next next if preceded by prev
 (?<! prev ) : next next if not preceded by prev

E.g.

>>> source = '''I wish I may, I wish I might
... Have a dish of fish tonight.'''

find wish or fish anywhere:

>>> re.findall('wish|fish', source)

['wish', 'wish', 'fish']

Or you can use another way:

>>> re.findall('[wf]ish', source)
['wish', 'wish', 'fish']

Find wish at the beginning:

>>> re.findall('^wish', source)
[]

Find fish at the end:

>>> re.findall('fish$', source)
[]

Find ght followed by a non-alphanumeric:

>>> re.findall('ght\W', source)
['ght\n', 'ght.']

Find I followed by wish:

>>> re.findall('I (?=wish)', source)
['I ', 'I ']
Deal with conflict between special escape characters and regular expression

In the example above, the following pattern should match any word that begins with fish:

>>> re.findall('\bfish', source)
[]

in special escape characters, \b means backspace in strings, but in the mini-language of regular expressions it means the beginning of a word.

To avoid this, . Always put an r character before your regular expression pattern string, and Python escape characters will be disabled.

>>> re.findall(r'\bfish', source)
['fish']

Binary data

Bytes: immutable, like a tuple of bytes.

Bytearray: mutable, like a list of bytes.

blist=[1,2,3,255]

the_bytes=bytes(blist)

print(the_bytes)

print(the_bytes[1])

print(bytearray(the_bytes))

Output:

b'\x01\x02\x03\xff'
2
bytearray(b'\x01\x02\x03\xff')

you can not change a bytes variable:

the_bytes[1]=124

Bytearray: 

the_byte_array=bytearray(blist)

print(the_byte_array)

output

bytearray(b'\x01\x02\x03\xff')

you can change a byte array value:

the_byte_array[1]=127

Convert Binary Data with struct

With the struct module, you can convert binary data to and from Python data structures.

In the following example, we will extracts the width and height of an image from some PNG data.

In this example, I downloaded a png from here: http://oldsite.polycode.org/img/polycode_logo.png to /Volumes/Data/colors.png.

import struct

file=open('/Volumes/Data/colors.png','rb') # this line and the next one will be covered in chapter 8

data=file.read()

valid_png=b'\x89PNG\r\n\x1a\n'

if data[:8]==valid_png:

    print("It's a valid png file")

    width,height=struct.unpack('>2L',data[16:24])

    print('the width is {} and the height is {}'.format(width,height))

else:

    print('invalid png file')

Output:

It's a valid png file

the width is 204 and the height is 204

• valid_png_header contains the 8-byte sequence that marks the start of a valid PNG file.
• width is extracted from bytes 16-20, and height from bytes 21-24.

The >LL is the format string that instructs unpack() how to interpret its input byte sequences and assemble them into Python data types. Here’s the breakdown:

• The > means that integers are stored in big-endian format.
• Each L specifies a 4-byte unsigned long integer.

The endian specifiers go first in the format string.
Endian specifiers:

Specifier Byte order
< little endian
> big endian

Big-endian integers have the most significant bytes to the left. For more about endian, check here http://frankfu.click/python/endian-big-and-little/.

You can examine each 4-byte value directly:

>>> data[16:20]
b'\x00\x00\x00\x9a'
>>> data[20:24]0x9a
b'\x00\x00\x00\x8d'

Because the width and height are each less than 255, they fit into the last byte of each sequence. You can verify that these hex values match the expected decimal values:

>>> 0x9a
154
>>> 0x8d
141

When you want to go in the other direction and convert Python data to bytes, use the struct pack() function:

>>> import struct
>>> struct.pack('>L', 154)
b'\x00\x00\x00\x9a'
>>> struct.pack('>L', 141)
b'\x00\x00\x00\x8d'
Specifier Description Bytes
x skip a byte 1
b signed byte 1
B unsigned byte 1
h signed short integer 2
H unsigned short integer 2
i signed integer 4
I unsigned integer 4
l signed long integer 4
L unsigned long integer 4
Q unsigned long long integer 8
f single precision float 4
d double precision float 8
p count and characters 1+ count
s characters count

 

The type specifiers follow the endian character. Any specifier may be preceded by a number that indicates the count; 5B is the same as BBBBB.

E.g. You can use a count prefix instead of >LL:

>>> struct.unpack('>2L', data[16:24])
(154, 141)

Or, we can use another way:

We could also use the x specifier to skip the uninteresting parts, We have 30 bytes data, but we only need from 17-24:
• Use big-endian integer format (>)
• Skip 16 bytes (16x)
• Read 8 bytes—two unsigned long integers (2L)
• Skip the final 6 bytes (6x)

>>> struct.unpack('>16x2L6x', data)
(154, 141)
Other Binary Data tools

• bitstring
• construct
• hachoir
• binio

E.g, we use construct to extract the png info:

>>> from construct import Struct, Magic, UBInt32, Const, String
>>> # adapted from code at https://github.com/construct
>>> fmt = Struct('png',
... Magic(b'\x89PNG\r\n\x1a\n'),
... UBInt32('length'),
... Const(String('type', 4), b'IHDR'),
... UBInt32('width'),
... UBInt32('height')
... )
>>> data = b'\x89PNG\r\n\x1a\n\x00\x00\x00\rIHDR' + \
... b'\x00\x00\x00\x9a\x00\x00\x00\x8d\x08\x02\x00\x00\x00\xc0'
>>> result = fmt.parse(data)
>>> print(result)

Output:

Container:
length=13
type = b'IHDR'
width = 154
height = 141
Convert Bytes /strings with binascii()

To convert between binary data and various string representations: hex(base 16), base 64, unencoded, and others.

use binascii.hexlify(binary_data) to convert from binary to hex.

use binascii.unhexlify(hex_data) to convert from hex to binary.

Bit operators

If the integer a=5 binary 0b0101, and b=1, binary 0b0001.

Operator Description Example Decimal result Binary result
& and a&b 1 0b0001
| or a|b 5 0b0101
ˆ exclusive or a^b 4 0b0100
˜ flip bits ~a -6 binary representation depends on init size
<< left shift a<<1 10 0b1010
>> right shift a>>1 2 0b0010

Explanation:

The & operator returns bits that are the same in both arguments, and | returns bits that are set in either of them. The ^ operator returns bits that are in one or the other, but not both. The ~ operator reverses all the bits in its single argument; this also reverses the sign because an integer’s highest bit indicates its sign (1 = negative) in two’s complement arithmetic, used in all modern computers. The << and >> operators just move bits to the left or right.

Reference

Format: http://www.python-course.eu/python3_formatted_output.php

Regular expression: https://docs.python.org/3/library/re.html