Reference frameworks and identification methods

For an information system to render the information contained in a binary data stream, it must know the encoding format

There are two main categories of representation information:

Those based on information external to the content (metadata), and those that use internal signatures, recognizing the format from the structure of part of the content.

The file extension

The file name extension is an example of an external signature. Microsoft’s Windows operating system, for example, uses it to determine — via an internal registry — the software to use by default to display the content. Sometimes the system provides this information by default; otherwise, it is up to third-party software, when installed, to register itself as capable of opening certain file types.

It goes without saying that this mechanism has several major drawbacks:

First, it is impossible to determine the format without the file name information being available. This is incompatible with data exchange over protocols such as HTTP, for example, unless additional file headers are transmitted.

Next, this information is partial, since the file extension is not universal — it can potentially be used by several different pieces of software, or by several versions of the same format. The PDF format, for example, currently exists in 30 versions, some standardized and others not, all of which carry the same « pdf » extension.

Finally, this information can be altered by simply renaming the file. Replacing the « docx » extension of a Microsoft Word file with « zip » is enough to open it with compression software and see the files it contains. Replace the « pdf » extension of an Adobe document with « txt » and you prevent users from easily using it.

The file name extension should therefore be reserved for operating systems, which have a file system and maintain a registry of the applications usable by default to display them.

The MIME type

This is a two-part format identifier, originally used by the Multipurpose Internet Mail Extensions (MIME) standard to define the encoding of content or headers in non-ASCII emails.

This system was later extended to other protocols such as HTTP. IANA maintains a registry of media types and character encodings. Their list is publicly accessible on the Internet at the following address:

http://www.iana.org/assignments/media-types/media-types.xhtml

A MIME type is made up of at least two parts:

A type and a subtype, accompanied by one or more additional fields as needed. For example, subtypes of the « text » type have an optional « charset » field indicating the character encoding.

The MIME types are: audio, image, message (email), model (3D), multipart (email content), text, video, application, and vnd for vendor-specific types. The subtype specifies the format and depends on the type.

Below are a few examples of MIME types:

application/pdf
application/vnd.openxmlformats-officedocument.wordprocessingml.document
application/xml
application/zip
application/octet-stream
audio/mpeg
audio/flac
image/gif
image/jpeg
image/png
text/html
text/csv
text/plain
video/mpeg
video/mp4

The MIME type is determined by a software component shipped with the operating system, the software, or the runtime environment of a programming language. It uses the « magic numbers » of the content to link it to types and subtypes via a « magic database ». Format detection is thus achieved through an internal signature that provides the magic number — already a positive point, since it requires no external information.

On the other hand, the MIME type remains fairly imprecise, as it often cannot distinguish between format versions or multimedia encoding, for example. Moreover, the magic number database is not a controlled reference; it exists in many versions produced by various organizations or communities and is not officially registered with any authority.

The MIME type is suited to representing content in emails and web interfaces, not to archiving.

Format registries for archiving

Several digital format registry projects intended for the preservation of digital data have been initiated by public bodies from different countries. Some have been discontinued, while others remain active.

The Global Digital Format Registry (GDFR) was a project of the Harvard University Library that was discontinued in the late 2000s.

See: http://library.harvard.edu/preservation/digital-preservation_gdfr.html

The Library of Congress in the United States offers methods for content classification, characterization, and conversion planning, based on a reference registry that describes more than 400 formats in considerable detail.

See: http://www.nationalarchives.gov.uk/PRONOM

The National Archives of the United Kingdom has published the PRONOM registry since 2002, which remains to this day the only public and operational registry of its kind in the world, providing the community with tools (DROID) and methods to identify and qualify formats. Each format notably has a unique, universal identifier called a PUID, which makes resource management easier.

See: http://www.digitalpreservation.gov/formats

The Unified Digital Format Registry (UDFR) is a project developed by the University of California that attempts to unify the resources of the PRONOM and GDFR registries in a community-supported, open-source project.

See: http://www.udfr.org