7.8. codecs — Codec registry and base classes

This module defines base classes for standard Python codecs (encoders and decoders) and provides access to the internal Python codec registry which manages the codec and error handling lookup process.

It defines the following functions:

codecs.encode(obj[, encoding[, errors]])

Encodes obj using the codec registered for encoding. The default encoding is 'ascii'.

Errors may be given to set the desired error handling scheme. The default error handler is 'strict' meaning that encoding errors raise ValueError (or a more codec specific subclass, such as UnicodeEncodeError). Refer to Codec Base Classes for more information on codec error handling.

New in version 2.4.

codecs.decode(obj[, encoding[, errors]])

Decodes obj using the codec registered for encoding. The default encoding is 'ascii'.

Errors may be given to set the desired error handling scheme. The default error handler is 'strict' meaning that decoding errors raise ValueError (or a more codec specific subclass, such as UnicodeDecodeError). Refer to Codec Base Classes for more information on codec error handling.

New in version 2.4.

codecs.register(search_function)

Register a codec search function. Search functions are expected to take one argument, the encoding name in all lower case letters, and return a CodecInfo object having the following attributes:

• name The name of the encoding;

• encode The stateless encoding function;

• decode The stateless decoding function;

• incrementalencoder An incremental encoder class or factory function;

• incrementaldecoder An incremental decoder class or factory function;

• streamwriter A stream writer class or factory function;

• streamreader A stream reader class or factory function.

The various functions or classes take the following arguments:

encode and decode: These must be functions or methods which have the same interface as the encode()/decode() methods of Codec instances (see Codec Interface). The functions/methods are expected to work in a stateless mode.

incrementalencoder and incrementaldecoder: These have to be factory functions providing the following interface:

factory(errors='strict')

The factory functions must return objects providing the interfaces defined by the base classes IncrementalEncoder and IncrementalDecoder, respectively. Incremental codecs can maintain state.

streamreader and streamwriter: These have to be factory functions providing the following interface:

factory(stream, errors='strict')

The factory functions must return objects providing the interfaces defined by the base classes StreamReader and StreamWriter, respectively. Stream codecs can maintain state.

Possible values for errors are

• 'strict': raise an exception in case of an encoding error

• 'replace': replace malformed data with a suitable replacement marker, such as '?' or '\ufffd'

• 'ignore': ignore malformed data and continue without further notice

• 'xmlcharrefreplace': replace with the appropriate XML character reference (for encoding only)

• 'backslashreplace': replace with backslashed escape sequences (for encoding only)

as well as any other error handling name defined via register_error().

In case a search function cannot find a given encoding, it should return None.

codecs.lookup(encoding)

Looks up the codec info in the Python codec registry and returns a CodecInfo object as defined above.

Encodings are first looked up in the registry’s cache. If not found, the list of registered search functions is scanned. If no CodecInfo object is found, a LookupError is raised. Otherwise, the CodecInfo object is stored in the cache and returned to the caller.

To simplify access to the various codecs, the module provides these additional functions which use lookup() for the codec lookup:

codecs.getencoder(encoding)

Look up the codec for the given encoding and return its encoder function.

Raises a LookupError in case the encoding cannot be found.

codecs.getdecoder(encoding)

Look up the codec for the given encoding and return its decoder function.

Raises a LookupError in case the encoding cannot be found.

codecs.getincrementalencoder(encoding)

Look up the codec for the given encoding and return its incremental encoder class or factory function.

Raises a LookupError in case the encoding cannot be found or the codec doesn’t support an incremental encoder.

New in version 2.5.

codecs.getincrementaldecoder(encoding)

Look up the codec for the given encoding and return its incremental decoder class or factory function.

Raises a LookupError in case the encoding cannot be found or the codec doesn’t support an incremental decoder.

New in version 2.5.

codecs.getreader(encoding)

Look up the codec for the given encoding and return its StreamReader class or factory function.

Raises a LookupError in case the encoding cannot be found.

codecs.getwriter(encoding)

Look up the codec for the given encoding and return its StreamWriter class or factory function.

Raises a LookupError in case the encoding cannot be found.

codecs.register_error(name, error_handler)

Register the error handling function error_handler under the name name. error_handler will be called during encoding and decoding in case of an error, when name is specified as the errors parameter.

For encoding error_handler will be called with a UnicodeEncodeError instance, which contains information about the location of the error. The error handler must either raise this or a different exception or return a tuple with a replacement for the unencodable part of the input and a position where encoding should continue. The encoder will encode the replacement and continue encoding the original input at the specified position. Negative position values will be treated as being relative to the end of the input string. If the resulting position is out of bound an IndexError will be raised.

Decoding and translating works similar, except UnicodeDecodeError or UnicodeTranslateError will be passed to the handler and that the replacement from the error handler will be put into the output directly.

codecs.lookup_error(name)

Return the error handler previously registered under the name name.

Raises a LookupError in case the handler cannot be found.

codecs.strict_errors(exception)

Implements the strict error handling: each encoding or decoding error raises a UnicodeError.

codecs.replace_errors(exception)

Implements the replace error handling: malformed data is replaced with a suitable replacement character such as '?' in bytestrings and '\ufffd' in Unicode strings.

codecs.ignore_errors(exception)

Implements the ignore error handling: malformed data is ignored and encoding or decoding is continued without further notice.

codecs.xmlcharrefreplace_errors(exception)

Implements the xmlcharrefreplace error handling (for encoding only): the unencodable character is replaced by an appropriate XML character reference.

codecs.backslashreplace_errors(exception)

Implements the backslashreplace error handling (for encoding only): the unencodable character is replaced by a backslashed escape sequence.

To simplify working with encoded files or stream, the module also defines these utility functions:

codecs.open(filename, mode[, encoding[, errors[, buffering]]])

Open an encoded file using the given mode and return a wrapped version providing transparent encoding/decoding. The default file mode is 'r' meaning to open the file in read mode.

Note

The wrapped version will only accept the object format defined by the codecs, i.e. Unicode objects for most built-in codecs. Output is also codec-dependent and will usually be Unicode as well.

Note

Files are always opened in binary mode, even if no binary mode was specified. This is done to avoid data loss due to encodings using 8-bit values. This means that no automatic conversion of '\n' is done on reading and writing.

encoding specifies the encoding which is to be used for the file.

errors may be given to define the error handling. It defaults to 'strict' which causes a ValueError to be raised in case an encoding error occurs.

buffering has the same meaning as for the built-in open() function. It defaults to line buffered.

codecs.EncodedFile(file, input[, output[, errors]])

Return a wrapped version of file which provides transparent encoding translation.

Strings written to the wrapped file are interpreted according to the given input encoding and then written to the original file as strings using the output encoding. The intermediate encoding will usually be Unicode but depends on the specified codecs.

If output is not given, it defaults to input.

errors may be given to define the error handling. It defaults to 'strict', which causes ValueError to be raised in case an encoding error occurs.

codecs.iterencode(iterable, encoding[, errors]