Lucidity

Filesystem templating and management.

Guide

Overview and examples of using the system in practice.

Introduction

Lucidity is a framework for templating filesystem structure.

It works using regular expressions, but hides much of the verbosity through the use of simple placeholders (such as you see in Python string formatting).

Consider the following paths:

/jobs/monty/assets/circus/model/high/circus_high_v001.abc
/jobs/monty/assets/circus/model/low/circus_low_v001.abc
/jobs/monty/assets/parrot/model/high/parrot_high_v002.abc

A regular expression to describe them might be:

'/jobs/(?P<job>[\w_]+?)/assets/(?P<asset_name>[\w_]+?)/model/(?P<lod>[\w_]+?)/(?P<asset_name>[\w_]+?)_(?P<lod>[\w_]+?)_v(?P<version>\d+?)\.(?P<filetype>\w+?)'

Meanwhile, the Lucidity pattern would be:

'/jobs/{job}/assets/{asset_name}/model/{lod}/{asset_name}_{lod}_v{version}.{filetype}'

With Lucidity you store this pattern as a template and can then use that template to generate paths from data as well as extract data from matching paths in a standard fashion.

Read the Tutorial to find out more.

Installation

Installing Lucidity is simple with pip:

$ pip install lucidity

If the Cheeseshop (a.k.a. PyPI) is down, you can also install Lucidity from one of the mirrors:

$ pip install --use-mirrors lucidity

Alternatively, you may wish to download manually from Github where Lucidity is actively developed.

You can clone the public repository:

$ git clone git://github.com/4degrees/lucidity.git

Or download an appropriate tarball or zipball

Once you have a copy of the source, you can embed it in your Python package, or install it into your site-packages:

$ python setup.py install

Dependencies

For testing:

Tutorial

This tutorial gives a good introduction to using Lucidity.

First make sure that you have Lucidity installed.

Patterns

Lucidity uses patterns to represent a path structure. A pattern is very much like the format you would use in a Python string format expression.

For example, a pattern to represent this filepath:

'/jobs/monty/assets/circus/model/high/circus_high_v001.abc'

Could be:

'/jobs/{job}/assets/{asset_name}/model/{lod}/{asset_name}_{lod}_v{version}.{filetype}'

Each {name} in braces is a variable that can either be extracted from a matching path, or substituted with a provided value when constructing a path. The variable is referred to as a placeholder.

Templates

A Template is a simple container for a pattern.

First, import the package:

>>> import lucidity

Now, construct a template with the pattern above:

>>> template = lucidity.Template('model', '/jobs/{job}/assets/{asset_name}/model/{lod}/{asset_name}_{lod}_v{version}.{filetype}')

Note

The template must be given a name to identify it. The name becomes useful when you have a bunch of templates to manage.

Parsing

With a template defined we can now parse a path and extract data from it:

>>> path = '/jobs/monty/assets/circus/model/high/circus_high_v001.abc'
>>> data = template.parse(path)
>>> print data
{
    'job': 'monty',
    'asset_name': 'circus',
    'lod': 'high',
    'version': '001',
    'filetype': 'abc'
}

Note

When a group name in a pattern occurs more than once (as with {asset_name} and {lod} above), the last matching value is used for the returned data.

If a template’s pattern does not match the full path exactly then parse() will raise a ParseError:

>>> print template.parse('/jobs/monty/assets')
ParseError: Input '/jobs/monty/assets' did not match template pattern.

Formatting

It is also possible to pass a dictionary of data to a template in order to produce a path:

>>> data = {
...     'job': 'monty',
...     'asset_name': 'circus',
...     'lod': 'high',
...     'version': '001',
...     'filetype': 'abc'
... }
>>> path = template.format(data)
>>> print path
/jobs/monty/assets/circus/model/high/circus_high_v001.abc

In the example above, we haven’t done more than could be achieved with standard Python string formatting. In the next sections, though, you will see the need for a dedicated format() method.

If the supplied data does not contain enough information to fill the template completely a FormatError will be raised:

>>> print template.format({})
FormatError: Could not format data {} due to missing key 'job'.

Nested Data Structures

Often the data structure you want to use will be more complex than a single level dictionary. Therefore, Lucidity also supports nested dictionaries when both parsing or formatting a path.

To indicate a nested structure, use a dotted notation in your placeholder name:

>>> template = lucidity.Template('job', '/jobs/{job.code}')
>>> print template.parse('/jobs/monty')
{'job': {'code': 'monty'}}
>>> print template.format({'job': {'code': 'monty'}})
/jobs/monty

Note

Unlike the standard Python format syntax, the dotted notation in Lucidity always refers to a nested item structure rather than attribute access.

Custom Regular Expressions

Lucidity works by constucting a regular expression from a pattern. It replaces all placeholders with a default regular expression that should suit most cases.

However, if you need to customise the regular expression you can do so either at a template level or per placeholder.

At The Template Level

To modify the default regular expression for a template, pass it is as an additional argument:

>>> template = lucidity.Template('name', 'pattern',
                                 default_placeholder_expression='[^/]+')
Per Placeholder

To alter the expression for a single placeholder, use a colon : after the placeholder name and follow with your custom expression:

>>> template = lucidity.Template('name', 'file_v{version:\d+}.ext')

Above, the version placeholder expression has been customised to only match one or more digits.

Note

If your custom expression requires the use of braces ({}) you must escape them to distinguish them from the placeholder braces. Use a preceding backslash for the escape (\{, \}).

And of course, any custom expression text is omitted when formatting data:

>>> print template.format({'version': '001'})
file_v001.ext

Managing Multiple Templates

Representing different path structures requires the use of multiple templates.

Lucidity provides a few helper functions for dealing with multiple templates.

Template Discovery

Templates can be discovered by searching a list of paths for mount points that register template instances. By default, the list of paths is retrieved from the environment variable LUCIDITY_TEMPLATE_PATH.

To search and load templates in this way:

>>> import lucidity
>>> templates = lucidity.discover_templates()

To specify a specific list of paths just pass them to the function:

>>> templates = lucidity.discover_templates(paths=['/templates'])

By default each path will be recursively searched. You can disable this behaviour by setting the recursive keyword argument:

>>> templates = lucidity.discover_templates(recursive=False)

Template Mount Points

To write a template mount point, define a Python file containing a register function. The function should return a list of instantiated Template instances:

# templates.py

from lucidity import Template

def register():
    '''Register templates.'''
    return [
        Template('job', '/jobs/{job.code}'),
        Template('shot', '/jobs/{job.code}/shots/{scene.code}_{shot.code}')
    ]

Place the file on one of the search paths for discover_templates() to have it take effect.

Operations Against Multiple Templates

Lucidity also provides two top level functions to run a parse or format operation against multiple candidate templates using the first correct result found.

Given the following templates:

>>> import lucidity
>>> templates = [
...     lucidity.Template('model', '/jobs/{job}/assets/model/{lod}'),
...     lucidity.Template('rig', '/jobs/{job}/assets/rig/{rig_type}')
... ]

To perform a parse:

>>> print lucidity.parse('/jobs/monty/assets/rig/anim', templates)
({'job': 'monty', 'rig_type': 'anim'},
 Template(name='rig', pattern='/jobs/{job}/assets/rig/{rig_type}'))

To format data:

>>> print lucidity.format({'job': 'monty', 'rig_type': 'anim'}, templates)
('/jobs/monty/assets/rig/anim',
 Template(name='rig', pattern='/jobs/{job}/assets/rig/{rig_type}'))

Note

The return value is a tuple of (result, template).

If no template could provide a result an appropriate error is raised ( ParseError or FormatError).

Reference

API reference providing details on the actual code.

lucidity

lucidity.discover_templates(paths=None, recursive=True)[source]

Search paths for mount points and load templates from them.

paths should be a list of filesystem paths to search for mount points. If not specified will try to use value from environment variable LUCIDITY_TEMPLATE_PATH.

A mount point is a Python file that defines a ‘register’ function. The function should return a list of instantiated Template objects.

If recursive is True (the default) then all directories under a path will also be searched.

lucidity.parse(path, templates)[source]

Parse path against templates.

path should be a string to parse.

templates should be a list of Template instances in the order that they should be tried.

Return (data, template) from first successful parse.

Raise ParseError if path is not parseable by any of the supplied templates.

lucidity.format(data, templates)[source]

Format data using templates.

data should be a dictionary of data to format into a path.

templates should be a list of Template instances in the order that they should be tried.

Return (path, template) from first successful format.

Raise FormatError if data is not formattable by any of the supplied templates.

template

class lucidity.template.Template(name, pattern, default_placeholder_expression='[\w_.\-]+?')[source]

Bases: object

A template.

__init__(name, pattern, default_placeholder_expression='[\w_.\-]+?')[source]

Initialise with name and pattern.

name[source]

Return name of template.

pattern[source]

Return template pattern.

parse(path)[source]

Return dictionary of data extracted from path using this template.

Raise ParseError if path is not parseable by this template.

format(data)[source]

Return a path formatted by applying data to this template.

Raise FormatError if data does not supply enough information to fill the template fields.

error

Custom error classes.

exception lucidity.error.ParseError[source]

Bases: exceptions.Exception

Raise when a template is unable to parse a path.

exception lucidity.error.FormatError[source]

Bases: exceptions.Exception

Raise when a template is unable to format data into a path.

Glossary

LUCIDITY_TEMPLATE_PATH

Environment variable defining paths to search for template mount points. Can be multiple paths separated by the appropriate path separator for your operating system.

Indices and tables