Almost two years ago (uh, already?) I went to Prague for a developer sprint of the team of Marble, the virtual globe and world atlas. My goal was to work on a proper maps shape plugin for Calligra.
We are ready to render, or? … Or?
Just, it showed that Marble at that time was not properly supporting the needs such a plugin has. Biggest showstopper was that Marble did not expose the information if and when all external data needed for rendering was available. Think e.g. of map pieces (so called tiles) still having to be downloaded from a server like OpenStreetMap‘s one. Not knowing the state is not good e.g. if a document with a map is to be rendered to a real printer.
Telling the File Manager some more about Marble and Geo Data Files
At that sprint then I worked instead on making sure that there are proper mime-types registered for the types of geo data files that Marble can read and display (like OSM data files, ESRI shapefiles and GPX files). So that such files are nicely displayed in the file manager with matching icon & type description and can get Marble assigned as handler. And:
The thumbnailer was started in the train back from Prague, but Dresden was arrived before it was properly working. Then… the code went out of focus and mind, that step frozen in the middle of the air…
Getting ready to render!
These days, Calligra still has no maps shape plugin. Now and then I nag^Wbeg the Marble developers for the issue (how lame, indeed), as there are other interesting use cases for rendering of map stills, like:
Creating a jigsaw puzzle for Palapeli, the jigsaw puzzle gaming program
Creating an animation for a movie in Kdenlive, the video editor
Yes, even creating the thumbnails for geo data files, if not using the simple installed map theme
Last week luckily earthwings found some personal interest to draft code for renderer state. Time to add a bit of momentum, by reviving the thumbnailer code, to serve as a use case. Still found in its old place, after some bit-rot cleanup and finally solving the problems of last time (different mindset, and things seem so obvious) it works now good enough for serious usage:
Thumbnails for Geo Data files
Some things need some more thinking (which map theme to use when, how to properly detect the sky object the data is for, how to deal with thumbnail generation in offline-state), but it’s a good first complete step 🙂 It does not yet use the new render state feature, but should be soon.
So look forward to Marble 1.9, bringing previews for geo data files to your filemanager and in the file dialogs! (And do not forget to enable them in Dolphin > Settings > Configure Dolphin… > General > Previews, disabled for all types by default).
And perhaps, perhaps, one day we finally also will have a maps shape plugin for Calligra…
BTW, if you want to help development of Marble, other KDE Edu projects and more, consider to donate for enabling the Randa Meetings 2014. Every little amount will help, as it sums up. Best do it now! 🙂
During the move of KDE’s software projects from Subversion to Git most projects split their subprojects over multiple Git repositories. Calligra did not, but is keeping all code of all apps and extras in one single repository. That is all of the apps Author, Braindump, Flow, Karbon, Kexi, Krita, Plan, Sheets, Stage and Words as well as all of the extras like the file format converter, the Okular generators, file thumbnailers and other file manager integration.
One of the reasons is that many libraries and plugins are shared among the different programs, and the API of the libraries is still changing a lot between releases. By having API-changing commits to be atomic to all of the Calligra code, all developers have less problems to have consistent revisions of the libs and programs.
All can be too much
A downside is: people interested in only one of the Calligra programs still have to get all of the Calligra code and also are faced with possibly having to build all of Calligra. Such people could be developers working only on e.g. Kexi, users wanting to only build the bleeding edge of their favourite program, e.g. Krita, or packagers/integrators only interested in e.g. viewer components for office file formats.
To support the different building needs, by the time more and more all kind of if(SOMEFLAG) [...] endif(SOMEFLAG) were added to random CMakeLists.txt files. And with additionally the conditional building due to optional external dependencies, things got complex. And thus sometimes broken.
Products, Features, and Product Sets
To get things more structured again, the concepts of “product”, “feature” and “product set” have been introduced to describe the stuff that gets build and their internal dependencies:
A “product” is the smallest functional unit which can be created in the build and which is useful on its own when installed. Examples are e.g. libraries, plugins or executables. Products have external and internal required dependencies at build-time. Internal dependencies are noted in terms of other products or features (see below) and could be e.g. other libraries to link against or build tools needed to generate source files. A product gets defined by setting an identifier, a descriptive fullname and the needed internal build-time requirements. Any other product or feature listed as requirement must have been defined before.
A “feature” is not a standalone product, but adds abilities to one or multiple given products. One examples is e.g. scriptability. Features have external and internal required dependencies at build-time. Internal dependencies are noted in terms of other products or features and could be e.g. other libraries to link against or build tools needed to generate source files. A feature gets defined by setting an identifier, a descriptive fullname and the needed internal build-time requirements. Any other product or feature listed as requirement must have been defined before.
A “product set” is a selection of products which should be build together. The products can be either essential or optional to the set. If essential (REQUIRED), the whole product set will not be build if a product is missing another internal or external dependency. If optional (OPTIONAL), the rest of the set will still be build in that case.
The products to include in a set can be listed directly or indirectly: they can be named themselves, or another product set can be included in a set, whose products will then be part of the first set as well.
Products and product sets can be listed as dependencies in multiple product sets. As with dependencies for products, they must have been defined before.
There are a number of predefined product sets, but everyone can add their own custom product set by adding a file locally in the folder cmake/productsets, named with the name of the productset in lowercase and the extension “.cmake” and containing simply the definition as described above.
The ids of products and features (but not sets) are used to generate CMake variables SHOULD_BUILD_${ID}, which then are used to control what is build and how.
Deciding what to build
The product set(s) to build are passed via the CMake flag PRODUCTSET and are a whitespace separated list of product sets, products and features, though usually just a single product set, e.g. the predefined “ALL”, which is also the default.
Based on the dependency tree that is resulting from the definition of all products, features and product sets, then the internally required products and features are estimated for the requested set.
Following that it is checked whose of those have all needed external dependencies or must be disabled from the build. Finally then the internal dependencies are checked again, and the final set of products and features that will be really built is collected.
Seeing the dependencies
With the knowledge about the internal dependencies available, one is tempted to export this data in a format that can further processed, e.g. to visualize it. And thus now when running CMake, a file product_deps.dot in DOT notation is generated in the top-level build directory. This one can e.g. be transformed on the commandline into a PNG file, by
dot -Tpng product_deps.dot > product_deps.png
The following is generated for me currently when “ALL” products and features should be build (I am missing a few external dependencies for some filters):
Calligra product set “ALL”
If I would like to only build the KEXI and SHEETS product sets, by passing -DPRODUCTSET="KEXI SHEETS" to CMake, the graph changes to this, showing that only the products will be built which are required or optional in the dependency trees of both product sets: Calligra product set “KEXI SHEETS”
More ideas
Besides creating pretty graphs to look at to get a (better) picture, other use cases might be possible:
packagers could get some template file created for the packages they would create from all of Calligra
Libs which are dependencies to other libs or app products could be automatically added to the target_link_libraries and their headers dirs to the include_directories
CI build servers could only build those products and features which would be affected by the new commits
Then a use-case seems to be that people would like to select a pre-defined product set, but blacklist a few of the products/features. Support for that still has to been developed and done.
I wonder how much of all this might make sense enough to be moved into CMake itself. Currently though this whole system still needs to prove its usefulness by being adapted in more detail by all of Calligra, not only most parts. There is also a chance of having over-engineered things 🙂 And instead Calligra should be simply split over multiple repositories as well. Not sure.
Be enlightened and inspired
This blog post basically should explain a little what all this product sets stuff is about to both all Calligra contributors who have yet not looked into details as well as externals with perhaps similar problems as the Calligra project.
If this approach could be also a solution for you, have a look at the macros in cmake/modules/CalligraProductSetMacros.cmake, they should be reusable outside of Calligra as well, only the new macro calligra_product_deps_report has Calligra specific code inside and could be made generic as well, if there is interest.
You might know that Okular has a plugin system, for adding support for more document formats. And you might know that Calligra since years also provides a plugin to Okular, which adds support to view slides from files in the OpenDocument Presentation (ODP) format. And not only for the ODP format: by simply using the Calligra import filters for PPT and PPTX you can also view the slides locked away in those formats.
The different apps of Calligra used to be built on the KParts system, so any files in formats supported by them would be also viewable in KPart-embedding programs like Konqueror or KDevelop. But due to the currently on-going creation of a new MVC-oriented foundation for the Calligra programs this has changed, the Calligra modules are no more KParts.
Now I happen to now and then read ODT files directly in KDevelop. And Okular has some native support for ODT. But it is not as powerful as what I am used to from Calligra, so I wanted that back. The best thing of course would be to write a Calligra-plugin directly for KDevelop (like done for the Okteta integration). But I wanted something quicker, and with less work. Writing KPart-wrappers around the Calligra modules would have been the next option. But then I remembered that Sven, who did the Calligra ODP generator plugin for Okular, had also once started an ODT generator, but left it in a branch. And Okular also has an UI optimized for document reading. So a commit cherry-pick and some bit-rot fixes later I had ODT files nicely displayed in KDevelop again, thanks to the chain Calligra Words engine -> Okular KPart -> KDevelop 🙂
See here Okular displaying the ODT 1.2 spec, of course in ODT format:
And like the ODP generator plugin adds support for PPT and PPTX by simply using the existing filters, the very same is possible with the ODT generator plugin and the import filters for documents locked away in DOC and DOCX formats. A link on a webpage to a file in DOCX format? Click it and view the file directly in the Okular KPart, powered by Calligra’s ODT generator plugin and DOCX import filter:
Currently the generators are just simple rendering ones. Of course we want the generators to be proper extended ones, including TextPage support, so you get all the comfort as you have when reading PDF files in Okular. Come and join the coding fun: navigate your editor to the generators’ code in extras/ and your browser to the excellent Okular generator Howto.
You instead would like to extend the support to other formats that Calligra has import filters for (Stage, Words)? Then take a look at the commit which added support for DOC and DOCX: just adding desktop files, per format one for the Application, one for the KPart and one for the generator itself 🙂
Your preferred format is not yet there? Consider adding it, e.g. by joining the Document Liberation Project and adding also a import filter to Calligra.
So look forward to Calligra 2.9 later this year, bringing a better ODT viewer and some for DOC & DOCX to an Okular near your fingertips 🙂 And perhaps more, at your will!
Attracted by virtual constructs 