4.8 The copy_reg Module

The copy_reg module provides a registry that you can use to register your own extension types. The pickle and copy modules use this registry to figure out how to process non-standard types.

For example, the standard pickle implementation cannot deal with Python code objects, as shown in the following example:

File: copy-reg-example-1.py

import pickle

CODE = """
print 'good evening'
"""

code = compile(CODE, "<string>", "exec")

exec code
exec pickle.loads(pickle.dumps(code))

good evening
Traceback (innermost last):
...
pickle.PicklingError: can't pickle 'code' objects

We can work around this by registering a code object handler. Such a handler consists of two parts: a pickler, which takes the code object and returns a tuple that can only contain simple datatypes, and an unpickler, which takes the contents of such a tuple as its arguments. Example 4-14 demonstrates this.

Example 4-14. Using the copy_reg Module to Enable Pickling of Code Objects

File: copy-reg-example-2.py

import copy_reg
import pickle, marshal, types

#
# register a pickle handler for code objects

def code_unpickler(data):
    return marshal.loads(data)

def code_pickler(code):
    return code_unpickler, (marshal.dumps(code),)

copy_reg.pickle(types.CodeType, code_pickler, code_unpickler)

#
# try it out

CODE = """
print "suppose he's got a pointed stick"
"""

code = compile(CODE, "<string>", "exec")

exec code
exec pickle.loads(pickle.dumps(code))

suppose he's got a pointed stick
suppose he's got a pointed stick

If you're transferring the pickled data across a network or to another program, the custom unpickler must be available at the receiving end as well.

For the really adventurous, Example 4-15 shows a version that allows you to pickle open file objects.

Example 4-15. Using the copy_reg Module to Enable Pickling of File Objects

File: copy-reg-example-3.py

import copy_reg
import pickle, types
import StringIO

#
# register a pickle handler for file objects

def file_unpickler(position, data):
    file = StringIO.StringIO(data)
    file.seek(position)
    return file

def file_pickler(code):
    position = file.tell()
    file.seek(0)
    data = file.read()
    file.seek(position)
    return file_unpickler, (position, data)

copy_reg.pickle(types.FileType, file_pickler, file_unpickler)

#
# try it out

file = open("samples/sample.txt", "rb")

print file.read(120),
print "<here>",
print pickle.loads(pickle.dumps(file)).read()

We will perhaps eventually be writing only small
modules, which are identified by name as they are
used to build larger <here> ones, so that devices like
indentation, rather than delimiters, might become
feasible for expressing local structure in the
source language.
     -- Donald E. Knuth, December 1974