Extending the Asset Pipeline¶

Overview¶

The asset pipeline is designed to be flexible and extendable. To extend the asset pipeline, developers can create new converters and conversion rules for the back end, or new compilers for the front end.

Converters and conversion rules are standalone modules of code that can be registered with the asset pipeline. Conversion rules define what files can be converted through the asset pipeline, whereas converters contain the actual conversion logic. Converters are also required to inform the asset pipeline of any dependency a source asset may have, after which the asset pipeline will ensure that these dependencies are up to date and built.

Compilers are executable modules that provide a front-end interface to the asset pipeline. There are three compilers provided with BWT: the Batch Compiler, JIT Compiler, and Test Compiler. You can design additional compilers to suit your needs (for example, compilers providing graphical interfaces or progress feedback).

Modules¶

The asset pipeline itself is divided into four library modules—the compiler, conversion, dependency, and discovery modules. When writing your own extensions for the asset pipeline, you should not need to modify these libraries and should treat the entire asset pipeline as a black box. The following descriptions are intended purely as a summary of how the asset pipeline operates.

Compiler¶

The compiler module contains the front-end compiler interface and the default base implementation of the compiler (asset_compiler). All user-defined compilers must inherit from the asset_compiler base class and not the compiler interface. This has been enforced by making the constructor of the compiler interface private, and only available to the asset_compiler base class. The asset_compiler provides a number callback functions that can be overloaded to monitor the progress of the asset pipeline during discovery and conversion phases.

Conversion¶

The conversion module is responsible for processing conversion tasks, checking dependency hashes and invoking the appropriate converters. It contains the converter interface which all user-defined asset converters must derive from, the conversion_task (a data struct that is used to track the status of an asset during conversion), the task_processor (a class which processes a queue of tasks on multiple threads, resolving any interdependencies), and the content_addressable_cache (a static class that acts an extension to the task_processor to allow for faster task processing).

Dependency¶

The dependency module is a simple data library that contains the different dependency structures that the asset pipeline knows how to handle. It contains the dependency base, the converter_dependency, converter_params_dependency, intermediate_file_dependency, output_file_dependency, directory_dependency, and source_file_dependency implementations and the dependency_list. As simple data structures, the only functionality the dependency classes provide are serialising to and from disk and generating dependency hashes. The dependency_list class is a slightly more complicated data structure that maintains a list of primary and secondary dependencies associated with a source asset and a list of output files generated by the source asset. The dependency_list structure represents the .deps files that are generated by the asset pipeline during conversion.

Discovery¶

The discovery module is responsible for the task collection that occurs during the discovery phase of the asset pipeline. It contains the task_finder class, which iterates directories and files on disk, searching for potential source assets eligible for conversion, and the conversion_rule interface, the interface all user-defined conversion rules must implement. Conversion rules are used by the task_finder to determine which files on disk can be converted.

The Conversion Process¶

A conversion task is created for every asset that the asset pipeline attempts to convert. A conversion task contains:

the file name of the asset being converted,
the id and version of the converter to be used,
any parameters to be passed to the converter,
the status of the conversion’s progress, and
links to any dependent conversion tasks for the asset.

To ensure minimal effort is spent processing tasks, the asset pipeline first attempts to determine if an asset needs to be converted at all. To do this, it attempts to deduce if an asset, any of its dependencies, or any of its outputs have changed since the last time the asset was processed. To track this, the asset pipeline uses deps files.

When an asset is processed, its deps file is queried from disk. If no deps file exists, a primary deps file is created. A primary deps file contains all the primary dependencies of a source asset. The primary dependencies of an asset are the asset source file itself, and the converter id, converter version and converter parameters required to convert the asset. If a deps file is found on disk, the stored primary dependencies are checked to see if they are up to date. If the primary dependencies are up not up to date, any secondary dependencies stored in the deps file are considered invalid and need to be regenerated. To regenerate the secondary dependencies, the converter for the asset is invoked via the converter function createDependencies.

Once the secondary dependencies have been generated, the asset pipeline must then verify that these dependencies are also up to date. As secondary dependencies can include dependencies on compiled assets, the asset pipeline may be required to trigger subtasks at this point. The asset pipeline has two modes of operation for accomplishing this: recursive conversion and non-recursive conversion.

Recursive Conversion¶

This conversion mode blocks the current thread processing a task, and recursively triggers conversion tasks of any sub dependencies. This has the benefit of simplicity and can be easily debugged, as the entire hierarchy of dependencies that triggered the conversion of a task can be viewed in the debug callstack. Complications do still arise with this method when multiple tasks being processed on separate threads have dependencies on the same subtask. In this case, the first thread to request the common subtask will trigger the conversion of the subtask while any subsequent threads will be required to block and wait for the first thread to finish. The drawback to this method is that when using large root tasks which trigger multiple subtasks, all of these subtasks must be processed on the same thread as the root parent task. For example, if a task to convert a space triggers multiple subtasks for models, textures and shaders, all of those subtasks need to be processed on the thread used to convert the space. This reduces the benefit of the parallel conversion capabilities that the asset pipeline provides.

Non-Recursive Conversion¶

To maximise the parallel-conversion capabilities of the asset pipeline, a non-recursive conversion mode is provided. In this mode, whenever a parent task is determined to have a subtask that must be converted, the parent task is paused and pushed back into the asset pipeline task queue. The asset pipeline goes through a task management process to ensure that the next time the parent task is processed, all of its subtasks, and any further recursive subtasks have been processed to completion. The asset pipeline also detects and protects against cyclic dependencies between assets. This mode of operation is the more efficient of the two modes of operation, but in this mode dependencies between assets can be harder to debug.

After all the secondary dependencies of a task have been processed, the hashes of the dependencies are checked against the assets deps file. If the dependencies match, the deps file is then checked for the expected outputs of the asset. If the expected outputs match those on disk the task is considered up to date and conversion is skipped. If the secondary dependencies do not match, or the expected outputs of the asset do not match, the converter for the asset is invoked to convert the asset via the converter function convert.

This conversion process is repeated for every conversion task discovered by the asset pipeline.

Writing Converters and Conversion Rules¶

The asset pipeline operates in two phases: the discovery phase and the conversion phase. During the discovery phase, the asset pipeline searches for files on disk that it can convert and creates conversion tasks for them. During the conversion phase, the asset pipeline does the actual asset conversion. Conversion rules are primarily used during the discovery phase and converters during the conversion phase. In most cases, when designing a new converter, you will need to create a converter and a conversion rule; however, there isn’t a one-to-one link between converters and conversion rules and the two are independent of each other.

Converters¶

To create a new converter, you must derive from the Converter interface and implement the following:

Constructor¶

// Constructor
// \param params command line parameters for initialising the converter
Converter( const BW::string& params )

The constructor for your converter must take a params string parameter. Your constructor should defer to the base class constructor which will store the params string in a member variable params_. The params parameter can contain any optional parameters of your choosing and is passed to your constructor from the asset pipeline via the conversion rules (see Conversion Rules).

createDependencies¶

// builds the dependency list for a source file.
// \param sourcefile the name of the source file on disk.
// \param dependencies the dependency list to generate.
// \return true if the dependency list was successfully generated
virtual bool createDependencies( const BW::string& sourcefile,
                                 const Compiler & compiler,
                                 DependencyList & dependencies )

This function must be overloaded to inform the asset pipeline of the dependencies of the source file you wish to convert. The sourcefile parameter contains the absolute path of the asset to convert. The compiler parameter contains a reference to the front-end compiler that has triggered this conversion. The dependencies parameters contains the dependency list structure that must be filled out with the source asset’s dependencies.

convert¶

// convert a source file.
// \param sourcefile the name of the source file on disk.
// \param convertedFiles a list of filenames that were converted from the source file.
// \return true if the source file was successfully converted.
virtual bool convert( const BW::string& sourcefile,
                      const Compiler & compiler,
                      BW::vector< BW::string > & intermediateFiles,
                      BW::vector< BW::string > & outputFiles )

This function must be overloaded to perform the actual conversion of your source asset. The sourcefile parameter contains the absolute path of the asset to convert. The compiler parameter contains a reference to the front-end compiler that has triggered this conversion. The intermediateFiles parameter contains a vector of strings to which you must append the path of any intermediate file produced by your conversion. The outputFiles parameter contains a vector of strings to which you must append the path of any output file produced by your conversion.

Absolute vs Relative Paths¶

Internally the asset pipeline operates on absolute paths to avoid errors with files with the same file name existing in multiple resource directories. However, when adding files to the dependency list in the createDependencies function, file names must first be converted to relative paths. This is necessary as these file names get saved to disk and shared via a file cache between multiple users, who may have different resource path setups. Conversely, when pushing file names back into the intermediateFiles and outputFiles outputs of the convert function, these file names need to be absolute paths. The reason for this is that the asset pipeline is required to hash the contents of these files to store in the generated deps file, and by using absolute paths we can ensure that the correct file is hashed.

There is, however, an important consideration that needs to be made when converting between relative and absolute paths. The asset pipeline can be run with custom intermediate and output directories. When resolving an intermediate or output file to an absolute path, this path needs to be resolved to the appropriate directory. As the intermediate and output directories are unknown to the converters until run time, the compiler interface provides a number of convenience functions for converting between absolute and relative paths.

virtual bool resolveRelativePath( BW::string & path ) const

Converts the path to a relative path.
virtual bool resolveSourcePath( BW::string & path ) const Converts the path to an absolute source path.
virtual bool resolveIntermediatePath( BW::string & path ) const

Converts the path to an absolute intermediate path.
virtual bool resolveOutputPath( BW::string & path ) const

Converts the path to an absolute output path.

In summary, before adding a file to a dependency list, call resolveRelativePath. Before adding a file to the intermediate files list, call resolveIntermediatePath. Before adding a file to the output files list, call resolveOutputPath. Also note, it is the responsibility of the converter to save any files it outputs to the appropriate directories. It is good practice to work with absolute paths during conversion where possible.

Dependency List¶

When pushing back dependencies in the createDependencies function, there are a number of options. Assets are currently allowed to have the following types of dependencies: source file dependencies, intermediate file dependencies, output file dependencies, directory dependencies, converter dependencies, and converter parameter dependencies.

Source file dependencies are when an asset file depends on another raw source asset. This dependency indicates that the file it depends on does not need any sort of processing by the asset pipeline.
Intermediate file dependencies and output file dependencies occur when an asset file depends on another compiled asset. These types of dependencies tell the asset pipeline that a subtask has to be initiated to convert a source asset into a compiled format before the conversion of the current asset can take place. The only difference between intermediate and output file dependencies is the location where the compiled asset is expected to reside.
Directory dependencies allow an asset to recursively depend on a directory of assets. Directory dependencies can use a regex pattern to filter all files in a directory into a smaller subset of files.
Converter dependencies allow an asset to depend on a certain version of a converter, and converter parameter dependencies allow an asset to depend on the parameter string that is passed to the converter’s constructor. These last two dependency types are only used internally by the asset pipeline system.

All dependencies can be marked as critical or non critical. A critical dependency is a dependency that must exist for the parent asset to be converted. If an error is encountered in a critical dependency, it will automatically fail the parent asset. A converter should ideally be designed to have as few critical dependencies as possible.

The DependencyList class provides the following functions for adding dependencies:

void addPrimarySourceFileDependency( const BW::string & filename )

Adds a primary source file dependency. The filename parameter specifies the name of the file to depend on.

void addPrimaryConverterDependency( size_t converterId, size_t converterVersion )

Adds a primary converter dependency. The converterId parameter specifies the id of the converter to depend on, converterVersion specifies the version of the converter to depend on.

void addPrimaryConverterParamsDependency( const BW::string & converterParams )

Adds a primary converter params dependency. The convertParams parameter specifies the parameter string to depend on.

void addSecondarySourceFileDependency( const BW::string & filename, bool critical )

Adds a secondary source file dependency. The filename parameter specifies the name of the file to depend on, and critical specifies whether this is a critical dependency.

void addSecondaryIntermediateFileDependency( const BW::string & filename, bool critical )

Adds a secondary intermediate file dependency. The filename parameter specifies the name of the compiled file to depend on, and critical specifies whether this is a critical dependency.

void addSecondaryOutputFileDependency( const BW::string & filename, bool critical )

Adds a secondary output file dependency. The filename parameter specifies the name of the compiled file to depend on, and critical specifies whether this is a critical dependency.

void addSecondaryDirectoryDependency( const BW::string & directory, const BW::string & pattern, bool recursive, bool critical )

Adds a secondary directory dependency. The pattern parameter specifies a regex pattern to match files within a directory, specified by the directory parameter, to depend on. Recursive specifies whether the pattern should be applied recursively to the directory and critical specifies whether this is a critical dependency.

Errors and Exceptions¶

The asset pipeline is designed to be run as an unattended process. Because of this we need to prevent asserts from halting the conversion process. As such all calls into the converters are wrapped in try/catch blocks. When an assert is fired, the asset pipeline swallows the assert and throws an exception. The exception is then caught by the task processor and the currently processing task is failed. The assert information is added to the error log of the task and the asset pipeline is able to continue on. Additionally, the asset pipeline handles the ERROR_MSG macro. When this macro is triggered within a converter, the currently processing task is marked with an error. Processing is allowed to continue on the task, but on completion the task is set as failed and the message added to the error log of the task.

Whilst processing a task, if your converter encounters an error and wishes to fail the current task without using an assert or an error message, simply return false from the createDependencies or convert function of your converter. This will set the current task as failed. If a task fails during the createDependencies function, the asset pipeline will not attempt to call the convert function.

Conversion Rules¶

To create a conversion rule you must implement the ConversionRule interface.

A conversion rule takes a single boolean argument in its constructor:

ConversionRule( bool bRoot )

The bRoot argument indicates whether a conversion rule is a root rule or a non-root rule. Root rules are rules used during the discovery phase of the asset pipeline to match files on disk to conversion tasks. Non-root rules are used to resolve tasks for sub dependencies of other tasks.

For example, if we had a rule to match source texture files to texture conversion tasks and we flagged this rule as a root rule, the discovery phase of the asset pipeline would identify tasks for every source texture on disk. However, if we were to flag this rule as a non-root rule, the discovery phase of the asset pipeline would not find any texture conversion tasks. During the conversion phase of the asset pipeline, if another task is processed and found to have a dependency on a compiled texture, the non-root texture conversion rule is queried to create the texture conversion task. In this way, by carefully selecting which conversion rules are flagged as root rules, we can ensure that only assets referenced by root assets are compiled and packaged to the output directory.

For a conversion rule to match a source asset file name to a conversion task, the following function must be overloaded:

/* returns true and populates a conversion task if the rule can match the input filename. */
virtual bool createTask( const BW::StringRef& sourceFile, ConversionTask& task )

This function is invoked by the asset pipeline whenever it tries to determine how to build a source asset. If your conversion rule is able to handle the asset passed in by the parameter sourceFile, you must initialise the task structure and return true. The task structure requires that you set the id, version, and parameters of the converter that will handle the conversion of this asset.

If your rule is a non-root rule (it is intended to be invoked when processing the dependencies of other assets), you must also overload the following function:

/* returns true if the rule can match the output filename. */
virtual bool getSourceFile( const BW::StringRef& file, BW::string& sourcefile ) const

This function is invoked by the asset pipeline whenever it tries to determine what source file is used to create a compiled dependency file. In this case, the parameter file will contain the file name of the compiled file the asset pipeline is trying to compile. If your conversion rule knows what source asset file is used to convert into this file, it should fill in the sourcefile parameter and return true.

Writing Compilers¶

The asset pipeline supports the creation of custom front-end interfaces. These interfaces enable users to interact with the asset conversion process. In the asset pipeline framework, these front-end interfaces are known as compilers.

Asset Compiler¶

Any custom front-end compiler you write must inherit from the AssetCompiler base class. The AssetCompiler class provides a number of overloadable callback functions that are called by the asset pipeline during different stages of the discovery and conversion phases. In most cases where a compiler callback has been overloaded, it is important that the base class implementation of the overloaded function is called at some point during your own callback handling code. In some cases, not doing so can cause the asset pipeline to fail.

Below are some of the more useful callbacks that you can overload.

Discovery Callbacks¶

The following callbacks are invoked during the discovery phase:

virtual bool shouldIterateFile( const BW::StringRef & file )
virtual bool shouldIterateDirectory( const BW::StringRef & directory )

These callbacks are invoked whenever the task finder attempts to iterate a file or directory whilst searching for potential source assets. By returning false from either of these functions, you can tell the asset pipeline to ignore certain files or directories.

Task Callbacks¶

The following callbacks are invoked during the conversion phase:

virtual void onTaskStarted( ConversionTask & conversionTask )
virtual void onTaskResumed( ConversionTask & conversionTask )
virtual void onTaskSuspended( ConversionTask & conversionTask )
virtual void onTaskCompleted( ConversionTask & conversionTask )

These callbacks are invoked by the task processor whilst processing conversion tasks. They provide a mechanism for notifying the compiler of the tasks that are currently executing. If you overload these callbacks, it is extremely important to call the base implementation, as the AssetCompiler uses these callbacks to manage the processing of dependent tasks and catching cyclic dependencies.

Converter Callbacks¶

These callbacks are invoked by the task processor prior to and after the createDependencies function on a converter is invoked and prior to and after the convert function on a converter is invoked. Once again, if you overload these callbacks, make sure you call the base implementation.

virtual void onPreCreateDependencies( ConversionTask & conversionTask )
virtual void onPostCreateDependencies( ConversionTask & conversionTask )
virtual void onPreConvert( ConversionTask & conversionTask )
virtual void onPostConvert( ConversionTask & conversionTask )

Conversion Callback¶

This callback is invoked for every intermediate and output file that is generated by the asset pipeline. Note, if an output file is not generated during a run of the asset pipeline, due to files being up to date, this callback will not be invoked.

virtual void onOutputGenerated( const BW::string & filename )

Cache Callbacks¶

These callbacks are invoked whenever the asset pipeline attempts to read or write from the shared file cache. If a cache read or write is successful, the onCacheRead and onCacheWrite callbacks will be invoked. If a cache read or write is not successful, due to network or disk errors or a file not existing in the cache, the onCacheReadMiss and onCacheWriteMiss callbacks are invoked.

virtual void onCacheRead( const BW::string & filename )
virtual void onCacheReadMiss( const BW::string & filename )
virtual void onCacheWrite( const BW::string & filename )
virtual void onCacheWriteMiss( const BW::string & filename )

Message Callbacks¶

These callbacks are invoked whenever a BigWorld Technology message macro is used. The intention of these callbacks is to enable the compilers to filter message spam from BigWorld Technology systems and choose which messages to display, and how to display them to the user. These callbacks are also integral to the operation of the asset pipeline, so if you choose to overload these functions, make sure you call the base implementations.

virtual bool handleMessage( DebugMessagePriority componentPriority, DebugMessagePriority messagePriority, const BW::string & category, DebugMessageSource messageSource, const char * format, va_list argPtr )

virtual void handleCritical( const char * msg )

Registering Converters and Conversion Rules¶

One of the responsibilities of the compiler front end is to manage the converters and conversion rules that are used by the asset pipeline framework. The AssetCompiler base class provides two functions to this end:

virtual void registerConversionRule( ConversionRule& conversionRule );
virtual void registerConverter( ConverterInfo& converterInfo );

Registering conversion rules is a straight forward process. An instance of the conversion rule must be created and then passed to the registerConversionRule function. Registering converters is a little more complicated. To register a converter, an instance of the following structure must be passed to the registerConverter function.

/// struct containing information about a converter in the asset pipeline.
struct ConverterInfo
{
public:
    /// the display name of the converter.
    BW::string name_;
    /// the id of the converter. Must be unique to each type of converter.
    size_t typeId_;
    /// the current version of the converter.
    size_t version_;
    /// converter flags
    enum
    {
        THREAD_SAFE         = 1 << 0, // can the converter be run on multiple threads.
        CACHE_DEPENDENCIES  = 1 << 1, // should dependencies be read and written to the cache.
        CACHE_CONVERSION    = 1 << 2, // should conversion be read and written to the cache.
        DEFAULT_FLAGS       = THREAD_SAFE | CACHE_DEPENDENCIES | CACHE_CONVERSION
    }                   flags_;
    /// function pointer for creating an instance of the converter.
    ConverterCreator    creator_;
};

This structure contains all the necessary information for the asset pipeline to instantiate converters to process tasks. The reason we do not simply register an instance of the converter on the compiler, as we do the conversion rules, is because the asset pipeline can process tasks on multiple threads, meaning more than one converter of the same type may be required at any one time. Allowing the asset pipeline to create converters on the fly avoids the need for converters to manage thread local storage or thread locks around member variables.

A standard practice for managing converter info structures is to define the following on your converter class definition:

/// MyConverter.hpp
static size_t getTypeId() { return s_TypeId; }
static size_t getVersion() { return s_Version; }
static const char * getTypeName() { return "MyConverter"; }
static Converter * createConverter( const BW::string& params ) { return new MyConverter( params ); }
static const size_t s_TypeId;
static const size_t s_Version;
/// MyConverter.cpp
const size_t MyConverter::s_TypeId = hash_string( MyConverter::getTypeName(), strlen( MyConverter::getTypeName()));
const size_t MyConverter::s_Version = 1;

Defining your converter info then becomes:

ConverterInfo myConverterInfo;
myConverterInfo.name_ = MyConverter::getTypeName();
myConverterInfo.typeId_ = MyConverter::getTypeId();
myConverterInfo.version_ = MyConverter::getVersion();
myConverterInfo.flags_ = myConverterFlags;
myConverterInfo.creator_ = MyConverter::createConverter;

A convenience macro exists to allow you to then replace this with:

ConverterInfo myConverterInfo;
INIT_CONVERTER_INFO( myConverterInfo, ConverterInfo , myConverterFlags);

Initiating a Build¶

Once all converters and conversion rules have been registered, a compiler can initiate a build. Compilers have access to both the discovery phase of the asset pipeline, through the taskFinder_ member, and the conversion phase of the asset pipeline, through the taskProcessor_ member.

To initiate a search for all source assets that match any of the registered root rules, call the following:

taskFinder_.findTasks( path );

where path is the directory or file to search.

To get the specific task for a source asset that matched either a root rule or a non-root rule, call the following:

ConversionTask & task = taskFinder_.getTask( file );
if (task.converterId_ != ConversionTask::s_unknownId)
{
    queueTask( task );
}

In this case, it is necessary to manually queue the conversion task so that it will be processed during the conversion phase.

To initiate the conversion phase, simply call:

taskProcessor_.processTasks();

Writing Plugins¶

The asset pipeline was designed with a DLL plugin architecture in mind. Although a compiler can be implemented without using the plugin framework, there are a number of benefits to using plugins. By using a plugin framework, converter DLLs can be built once and distributed between different compiler front ends. Also, developing a new converter plugin does not require a rebuild of all the different compiler front ends that may exist. This prevents changes to converter plugins, introducing bugs into the compiler executable.

Plugin Loader¶

To allow a compiler to make use of the asset pipeline plugin system, a compiler must inherit from the PluginLoader class.

The PluginLoader class provides a number of functions for loading and unloading converter DLLs.

void initPlugins();
void finiPlugins();
HMODULE loadPlugin( LPCWSTR plugin );
bool unloadPlugin( HMODULE plugin );

initPlugins reads a file named “<executable>_plugins.txt” and loads the appropriate (debug or hybrid) plugin DLLs specified within.
finiPlugins unloads all currently loaded DLLs.
loadPlugin loads the plugin with file name plugin.
unloadPlugin unloads a specific loaded plugin by its HMODULE.

When using these functions to load converter plugins, it is the responsibility of the converter plugin code to register conversion rules and converters with your compiler.

Converter Plugin¶

To create a converter plugin there are two functions that your DLL must expose. When scanning for potential plugins to load, the plugin loader will search for these functions to determine whether your plugin can be loaded. These functions are defined differently on debug and hybrid builds. This allows debug and hybrid configurations of your DLLs to reside side by side in a file directory whilst still ensuring that the plugin loader can, and will, only load the appropriate configuration version of your DLL. To facilitate exposing the correct dll functions based on configuration, use the macros PLUGIN_INIT_FUNC and PLUGIN_FINI_FUNC.

The following is an example of how to write a converter plugin:

BW_BEGIN_NAMESPACE
DECLARE_APP_DATA( "MyConverter", true )
MyConversionRule myConversionRule;
ConverterInfo myConverterInfo;

PLUGIN_INIT_FUNC
{

    Compiler * compiler = dynamic_cast< Compiler * >( &pluginLoader );
    if (compiler == NULL)
    {
        return false;
    }

    // Init any systems required by your converter
    INIT_CONVERTER_INFO( myConverterInfo, MyConverter, DEFAULT_FLAGS );
    compiler->registerConversionRule( myConversionRule );
    compiler->registerConverter( myConverterInfo );
    return true;
}

PLUGIN_FINI_FUNC
{
    // Fini any systems that were started by your converter
    return true;
}

BW_END_NAMESPACE

The DECLARE_APP_DATA macro is a convenience macro to set up a number of global constant values that allow the core BigWorld Technology systems to be run in a plugin. The first parameter of this macro is a unique identifier for your plugin, and the second is a boolean to indicate that the current module is being compiled as a plugin.

The PLUGIN_INIT_FUNC macro automatically generates an exposed function that gets called when your plugin is loaded. This function takes one argument: pluginLoader. As shown in the example above, you can verify the module attempting to load your plugin is actually a compiler by dynamically casting the pluginLoader argument to a Compiler object. If the cast fails, the function should return false.

The PLUGIN_FINI_FUNC macro automatically generates an exposed function that gets called when your plugin is unloaded.

Writing Unit Tests¶

The final step in writing a converter plugin for the asset pipeline is to write unit tests. To make this as easy as possible, a custom front-end compiler is provided. The TestCompiler provides a number of convenience functions that can be used to easily test conversion rules and converters.

To test a conversion rule, use the following function:

// Test that a conversion rule can be found for a source file
bool testConversionRule( const StringRef & sourceFile,
                         const StringRef & outputFile,
                         const StringRef & converterName,
                         const StringRef & converterParams );

This function takes a source asset file name, the expected output file, the expected name of the converter that would convert this asset, and the expected conversion parameters for this asset. If a source asset is expected to compile to more than one output file, this function should be called for each of the individual output file names.

There are several steps involved to test a converter.

First, the test needs to be set up. To set up a test we must provide the TestCompiler with the source files to convert. Optionally, we can also provide copies of the output files we expect to be produced by the conversion process. The following functions are provided for setting up a converter test:

// Set the directory where the source files for this test can be found
bool setSourceFileDirectory( const StringRef & sourceFileDirectory );
// Set the directory where the output files for this test can be found
bool setOutputFileDirectory( const StringRef & outputFileDirectory );
// Add a file from the source file directory that should be used in this test

bool addSourceFile( const StringRef & sourceFile );
// Add a file from the output file directory that should be built in this test

bool addOutputFile( const StringRef & outputFile );

The setSourceFileDirectory function tells the TestCompiler where to find the source assets to use for the test. The setOutputFileDirectory functiion tells the TestCompiler where to find the output files to verify the conversion process against. The addSourceFile and addOutputFile functions tell the TestCompiler which files in the source directory and output directory to use for the test.

After the test has been set up, it can be run with one of the following functions:

// Trigger a test build for a single file
bool testBuildFile( const StringRef & file );

// Trigger a test build for a directory
bool testBuildDirectory( const StringRef & directory );

The testBuildFile function tests building a single file. The testBuildDirectory function tests building an entire directory of files. The input to these functions should be the source asset or directory to build relative to the source file directory. Normally you will want to call testBuildDirectory with an empty directory parameter to build every source asset in the source file directory.

When the TestCompiler finishes building your assets it will automatically verify that the outputs produced by the asset pipeline exactly match the copies of the expected outputs that you provided when setting up the test. If you did not provide any expected outputs to the test setup then the TestCompiler will assume the conversion process completed successfully. The TestCompiler then provides the following functions to further verify the results of the conversion process:

// Get the number of tasks that were processed by this test
long getTaskCount() const { return taskCount_; }

// Get the number of tasks that were failed by this test
long getTaskFailedCount() const { return taskFailedCount_; }

// Returns if this test encountered a cyclic error
bool hasCyclicError() const { return hasCyclicError_; }
// Returns if this test encountered a dependency error
bool hasDependencyError() const { return hasDependencyError_; }
// Returns if this test encountered a conversion error
bool hasConversionError() const { return hasConversionError_; }

getTaskCount allows you to verify how many tasks were actually processed.
getTaskFailedCount allows you to verify how many of these tasks failed, as you may wish to test certain situations in which your converter should fail.
hasCyclicError will tell you if a cyclic dependency error occured during conversion.
hasDependencyError will tell you if any task reported an error during the createDependencies function of your converter.
hasConversionError will tell you if any task reported an error during the convert function of your converter.

After the TestCompiler finishes its testing, it will automatically clean up any changes it may have made on disk and return.