MXNet C++ package improvements

MXNet Edge team

MXNet C++ Package

Introduction

For using MXNet for inference in a native production setting where speed and reliability is critical, a solid C++ Package offering is necessary.

A C++ package allows to integrate MXNet into native applications. Also avoids runtime errors which are common with dynamically typed languages such as Python. There are also cases where the extra loss in performance by using Python can't be afforded. In addition, using MXNet from C++ allows for more complex setups where multithreading might be needed or reduced data copying.

Background

The C++ Package in MXNet is found in the folder `cpp-package/`. This package wraps the MXNet C API and implements some classes on top of it, such as NDArray or Symbol.

Most of the classes implemented or enums declared are duplicates of the ones found in MxNet itself and are needed because there is no way to propagate those though the C API. A special case are the operators that need an additional python script to go through the compiled mxnet library and the corresponding sources to extract and generate the implementation in the C++ layer (more on this below).

Right now the example C++ applications are intermingled in the MXNet source folder and is nontrivial to have an external application compiled against MXNet.

Most of the implementation is in located in header files and for different frameworks and platforms separated with preprocessor defines. If an example would be compiled outside of the main library build, it could not know about all the defines the library was compiled with. Providing the binary with the headers (including headers of dependencies) is not enough to make a successful run of the application. In most cases we observed the application just crashes after start, since without the right defines it’s impossible to understand what implementation is compiled into the library.

The exposure of the operators need a special python script to be executed to run through the compiled MxNet binary file and the library sources to generate C++ API code. This causes significant maintenance effort to synchronize the classes and usages throughout the layers. It does not allow semantic and explicit versioning and is likely to break when implementation details will change. Though an attempt was made to hide the implementation details of the library and as such provide a proper public API of the library by duplicating classes, the package is still not standalone and decoupled from the MXNet source, as it requires additional includes that are "private" to the MXNet implementation such as dmlc and nnvm.

Our primary objective would be to provide a standalone package with a CMake build file, so that users can build a C++ application against an MXNet binary library. This includes a semantic, fixed, documented API with binary compatibility throughout minor version changes (see more in Make implementation private).

Documentation would provide examples that have features marked from which versions of the library they are supported.

Another important goal is to separate inference and training API's, since customers want only effective, as lightweight as possible inference on their production edge devices. Training should also be considered, although with a lower priority.

From a user perspective, ideally they would get a package compiled for their distribution / OS and a set of header files to program and link against the MXNet library. Within a minor version change the library can be updated/replaced without recompilation and risking compatibility issues.

Goals

The proposed work in the C++ package aims to achieve the following improvements, areas of focus for users of the C++ package:

Usability

• Type safety

• Thread safety in critical areas

A detailed design document is worked on separately (Thread-safety in MXNet).

• Informative errors

• Documentation and versioning of the API

Exposed functions as well as their usage examples should be documented with doxygen or similar and added to the public documentation on the website.

• Inference / training separation of the API

With inference prioritization for now.

• Examples

End-to-end examples of using the C++ API. Some of those can already be found in the cpp_package examples folder.

• Standalone Package

A dynamic or static library artifact with corresponding headers should be provided for download on the website. These artifacts can be published to package managers as well improving significantly the availability of the library for developers.

C++ Development experience

• API / ABI compatibility

• Portable binary library with standalone headers

• Build configuration

Work areas

Remove intermediate C API layer for C++

We propose hooking directly into the C++ API and not going through the C API. This helps for type safety and maintainability. Also can be beneficial to propagate errors from MXNet to the C++ API via normal C++ exceptions without additional machinery, overhead and conversion to error codes in the C API boundary.

Make implementation private

As mentioned previously the library can not be used without all the defines it was compiled with, i.e. without exposing the actual implementation (see Background).

We propose improving the API including all key core classes in MXNet such as NDArray, Symbols or Operators utilising the "private pointer implementation (PIMPL)" idiom. It will provide the following advantages:

• Allowing to hide implementation details in private classes without exposing the details to the interface. This becomes very important due to the fact that mxnet uses different version of computation frameworks on multiple platforms. And there can be a lot implementation specific details that overwhelm the interface. A well defined interface means better architecture with less coupling.

• Hiding the implementation details enables the possibility for a stable API that does not need information about the compilation of the library to be used in applications.

• Recompilation time while development.

• Binary compatibility between minor version changes, so that the applications using the library could update/replace it without recompilation. While the latter is not a strong requirement for our users right now due to the experimental nature of our framework, it's commonplace in the software industry for commercial and open source libraries, where binary compatibility is required or nice to have (see https://www.qt.io/ for example).

• + additionally: A very easy copy-on-write optimization for copy heavy classes as well as most move constructors are provided for free.

Read more at: "Pointer to implementation" or "pImpl" is a C++ programming technique

Currently most of the implementations reside in common header files, separated with preprocessor defines making it hard to develop and to maintain.

Here is an example of a possible implementation of a imaginative multiply operation:

// MultiplyOperation.h

class Tensor;

class MultiplyOperation {
public:
    void setA(const Tensor &a);
    void setB(const Tensor &b);

    Tensor getResult() const;

private:
    class MultiplyOperationPrivate;

    std::unique_ptr <MultiplyOperationPrivate> p;
};

// MultiplyOperation.cpp

#include <MultiplyOperation.h>

void MultiplyOperation::setA(const Tensor &a) {
    p->a = a;
}

void MultiplyOperation::setB(const Tensor &b) {
    p->b = b;
}

Tensor MultiplyOperation::getResult() const {
    return p->getResultImpl();
}

class MultiplyOperationPrivate {
public:
    Tensor a, b;

    Tensor getResultImpl() const;
};

// MultiplyOperation_CUDA.cpp

Tensor MultiplyOperationPrivate::getResultImpl() const {
    return CUDA_CALL(multiply(a, b));
}

// MultiplyOperation_MKL.cpp

Tensor MultiplyOperationPrivate::getResultImpl() const {
    return mkl_multiply(a, b);
}

Semantic versioned API with no additional dependencies

Ideally no additional headers such as nnvm or dmlc would need to be added. We observed that libdmlc needs to be linked as well. And the build system has to be configured to handle cases such as when compiling with omp in the 3rd party folder with creates an additional binary artifact that is needed for linking.

Future work and considerations

With the library being binary compatible with minor version changes language bindings can be moved to use C++ API only and the there will be no need for maintaining a separate C interface. The bindings could be separated from the main project and have its own versioning.

Additional care must be taken with other MXNet dependencies if there would be a need to expose their API's as part of the public interface (OpenCV, OpenBLAS, MKL etc.) or if they will be distributed as separate libraries to link against. This needs to be investigated further and additional steps to take in these cases must be defined and documented.