Attracted by virtual constructs

June 17, 2014

Managing internal dependencies in a build of Calligra

Filed under: Calligra,KDE — by frinring @ 1:17 am

During the move of KDE’s software projects from Subversion to Git most projects split their subprojects over multiple Git repositories. Calligra did not, but is keeping all code of all apps and extras in one single repository. That is all of the apps Author, Braindump, Flow, Karbon, Kexi, Krita, Plan, Sheets, Stage and Words as well as all of the extras like the file format converter, the Okular generators, file thumbnailers and other file manager integration.
One of the reasons is that many libraries and plugins are shared among the different programs, and the API of the libraries is still changing a lot between releases. By having API-changing commits to be atomic to all of the Calligra code, all developers have less problems to have consistent revisions of the libs and programs.

All can be too much

A downside is: people interested in only one of the Calligra programs still have to get all of the Calligra code and also are faced with possibly having to build all of Calligra. Such people could be developers working only on e.g. Kexi, users wanting to only build the bleeding edge of their favourite program, e.g. Krita, or packagers/integrators only interested in e.g. viewer components for office file formats.

To support the different building needs, by the time more and more all kind of if(SOMEFLAG) [...] endif(SOMEFLAG) were added to random CMakeLists.txt files. And with additionally the conditional building due to optional external dependencies, things got complex. And thus sometimes broken.

Products, Features, and Product Sets

To get things more structured again, the concepts of “product”, “feature” and “product set” have been introduced to describe the stuff that gets build and their internal dependencies:

A “product” is the smallest functional unit which can be created in the build and which is useful on its own when installed. Examples are e.g. libraries, plugins or executables. Products have external and internal required dependencies at build-time. Internal dependencies are noted in terms of other products or features (see below) and could be e.g. other libraries to link against or build tools needed to generate source files. A product gets defined by setting an identifier, a descriptive fullname and the needed internal build-time requirements. Any other product or feature listed as requirement must have been defined before.

Example:

calligra_define_product(BUILDTOOL_RNG2CPP "rng2cpp")
calligra_define_product(LIB_CALLIGRA "Calligra core libs"  REQUIRES BUILDTOOL_RNG2CPP)

A “feature” is not a standalone product, but adds abilities to one or multiple given products. One examples is e.g. scriptability. Features have external and internal required dependencies at build-time. Internal dependencies are noted in terms of other products or features and could be e.g. other libraries to link against or build tools needed to generate source files. A feature gets defined by setting an identifier, a descriptive fullname and the needed internal build-time requirements. Any other product or feature listed as requirement must have been defined before.

Example:

calligra_define_feature(FEATURE_SCRIPTING "Scripting feature")

A “product set” is a selection of products which should be build together. The products can be either essential or optional to the set. If essential (REQUIRED), the whole product set will not be build if a product is missing another internal or external dependency. If optional (OPTIONAL), the rest of the set will still be build in that case.
The products to include in a set can be listed directly or indirectly: they can be named themselves, or another product set can be included in a set, whose products will then be part of the first set as well.
Products and product sets can be listed as dependencies in multiple product sets. As with dependencies for products, they must have been defined before.

Example:

calligra_define_productset(STAGE "Full Stage (for Desktop)"
    REQUIRES
        APP_STAGE
    OPTIONAL
        # extras
        FILEMANAGER
        # plugins
        PLUGIN_DEFAULTTOOLS
        PLUGIN_ARTISTICTEXTSHAPE
        PLUGIN_DOCKERS
        PLUGIN_PATHSHAPES
        PLUGIN_VARIABLES
        PLUGIN_CHARTSHAPE
        PLUGIN_PICTURESHAPE
        PLUGIN_TEXTSHAPE
        PLUGIN_PLUGINSHAPE
        PLUGIN_FORMULASHAPE
        PLUGIN_VECTORSHAPE
        PLUGIN_VIDEOSHAPE
        # filters
        FILTERS_STAGE
)

There are a number of predefined product sets, but everyone can add their own custom product set by adding a file locally in the folder cmake/productsets, named with the name of the productset in lowercase and the extension “.cmake” and containing simply the definition as described above.
The ids of products and features (but not sets) are used to generate CMake variables SHOULD_BUILD_${ID}, which then are used to control what is build and how.

Deciding what to build

The product set(s) to build are passed via the CMake flag PRODUCTSET and are a whitespace separated list of product sets, products and features, though usually just a single product set, e.g. the predefined “ALL”, which is also the default.

Based on the dependency tree that is resulting from the definition of all products, features and product sets, then the internally required products and features are estimated for the requested set.
Following that it is checked whose of those have all needed external dependencies or must be disabled from the build. Finally then the internal dependencies are checked again, and the final set of products and features that will be really built is collected.

Seeing the dependencies

With the knowledge about the internal dependencies available, one is tempted to export this data in a format that can further processed, e.g. to visualize it. And thus now when running CMake, a file product_deps.dot in DOT notation is generated in the top-level build directory. This one can e.g. be transformed on the commandline into a PNG file, by

dot -Tpng product_deps.dot > product_deps.png

The following is generated for me currently when “ALL” products and features should be build (I am missing a few external dependencies for some filters):

Calligra product set "ALL"

Calligra product set “ALL”

If I would like to only build the KEXI and SHEETS product sets, by passing -DPRODUCTSET="KEXI SHEETS" to CMake, the graph changes to this, showing that only the products will be built which are required or optional in the dependency trees of both product sets:

Calligra product set "KEXI SHEETS"

Calligra product set “KEXI SHEETS”

More ideas

Besides creating pretty graphs to look at to get a (better) picture, other use cases might be possible:

  • packagers could get some template file created for the packages they would create from all of Calligra
  • Libs which are dependencies to other libs or app products could be automatically added to the target_link_libraries and their headers dirs to the include_directories
  • CI build servers could only build those products and features which would be affected by the new commits

Then a use-case seems to be that people would like to select a pre-defined product set, but blacklist a few of the products/features. Support for that still has to been developed and done.

I wonder how much of all this might make sense enough to be moved into CMake itself. Currently though this whole system still needs to prove its usefulness by being adapted in more detail by all of Calligra, not only most parts. There is also a chance of having over-engineered things :) And instead Calligra should be simply split over multiple repositories as well. Not sure.

Be enlightened and inspired

This blog post basically should explain a little what all this product sets stuff is about to both all Calligra contributors who have yet not looked into details as well as externals with perhaps similar problems as the Calligra project.

If this approach could be also a solution for you, have a look at the macros in cmake/modules/CalligraProductSetMacros.cmake, they should be reusable outside of Calligra as well, only the new macro calligra_product_deps_report has Calligra specific code inside and could be made generic as well, if there is interest.

kickstarter-29-front-ban

About these ads

2 Comments »

  1. […] Managing internal dependencies in a build of Calligra […]

    Pingback by Links 17/6/2014: More Games, Plasma 5 Demos | Techrights — June 17, 2014 @ 9:40 pm |Reply

  2. Love Calligra! Keep it up guys!

    Comment by Sam — June 24, 2014 @ 3:30 am |Reply


RSS feed for comments on this post. TrackBack URI

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

The Toni Theme. Create a free website or blog at WordPress.com.

Follow

Get every new post delivered to your Inbox.

%d bloggers like this: