Adding API dox QCH files generation to KDE Frameworks builds

Or: Tying loose ends where some are slightly too short yet.

When

  • you favour offline documentation (not only due to nice integration with IDEs like KDevelop),
  • develop code using KDE Frameworks or other Qt-based libraries,
  • you know all the KF5 libraries have seen many people taking care for API documentation in the code over all the years,
  • and you had read about doxygen’s capability to create API dox in QCH format,
  • and you want your Linux distribution package management to automatically deliver the latest version of the documentation (resp. QCH files) together with the KDE Frameworks libraries and headers (and ideally same for other Qt-based libraries),

the idea is easy derived to just extend the libraries’ buildsystem to also spit out QCH files during the package builds.

It’s all prepared, can ship next week, latest!!1

Which would just be a simple additional target and command, invoking doxygen with some proper configuration file. Right? So simple, you wonder why no-one had done it yet 🙂

Some initial challenge seems quickly handled, which is even more encouraging:
for proper documentation one also wants cross-linking to documentation of things used in the API which are from other libraries, e.g. base classes and types. Which requires to pass to doxygen the list of those other documentations together with a set of parameters, to generate proper qthelp:// urls or to copy over documentation for things like inherited methods.
Such listing gets very long especially for KDE Frameworks libraries in tier 3. And with indirect dependencies pulled into the API, on changes the list might get incomplete. Same with any other changes of the parameters for those other documentations.
So basically a similar situation to linking code libraries, which proposes to also give it a similar handling: placing the needed information with CMake config files of the targeted library, so whoever cross-links to the QCH file of that library can fetch the up-to-date information from there.

Things seemed to work okay on first tests, so last September a pull request was made to add some respective macro module to Extra-CMake-Modules to get things going and a blog post “Adding API dox generation to the build by CMake macros” was written.

This… works. You just need to prepare this. And ignore that.

Just, looking closer, lots of glitches popped up on the scene. Worse, even show stoppers made their introduction, at both ends of the process pipe:
At generation side doxygen turned out to have bitrotted for QCH creation, possibly due to lack of use? Time to sacrifice to the Powers of FLOSS, and git clone the sources and poke around to see what is broken and how to fix it. Some time and an accepted pull request later the biggest issue (some content missed to be added to the QCH file) was initially handled, just yet needed to also get out as released version (which it now is since some months).
At consumption side Qt Assistant and Qt Creator turned to be no longer able to properly show QCH files with JavaScript and other HTML5 content, due to QWebKit having been deprecated/dropped and both apps in many distributions now only using QTextBrowser for rendering the documentation pages. And not everyone is using KDevelop and its documentation browser, which uses QWebKit or, in master branch, favours QWebEngine if present.
Which means, an investment into QCH files from doxygen would only be interesting to a small audience. Myself currently without resources and interest to mess around with Qt help engine sources, looks with hope on the resurrection of QWebKit as well as the patch for a QtWebEngine based help engine (if you are Qt-involved, please help and push that patch some more!)

Finally kicking off the production cycle

Not properly working tools, nothing trying to use the tools on bigger scale… classical self-blocking state. So time to break this up and get some momentum into, by tying first things together where possible and enabling the generation of QCH files during builds of the KDE Frameworks libraries.

And thus to current master branches (which will become v5.36 in July) there has been now added for one to Extra-CMake-Modules the new module ECMAddQch and then to all of the KDE Frameworks libraries with public C++ API the option to generate QCH files with the API documentation, on passing -DBUILD_QCH=ON to cmake. If also having passed -DKDE_INSTALL_USE_QT_SYS_PATHS=ON (or installing to same prefix as Qt), the generated QCH files will be installed to places where Qt Assistant and Qt Creator even automatically pick them up and include them as expected:

Qt Assistant with lots of KF5 API dox

KDevelop picks them up as well, but needs some manual reconfiguration to do so.

(And of course ECMAddQch is designed to be useful for non-KF5 libraries as well, give it a try once you got hold of it!)

You and getting rid of the remaining obstacles

So while for some setups the generated QCH file of the KDE Frameworks already are useful (I use them since some weeks for e.g. KDevelop development, in KDevelop), for many they still have to become that. Which will take some more time and ideally contributions also from others, including Doxygen and Qt Help engine maintainers.

Here a list of related reported Doxygen bugs:

  • 773693 – Generated QCH files are missing dynsections.js & jquery.js, result in broken display (fixed for v1.8.13 by patch)
  • 773715 – Enabling QCH files without any JavaScript, for viewers without such support
  • 783759 – PERL_PATH config option: when is this needed? Still used?
  • 783762 – QCH files: “Namespaces” or “Files” in the navigation tree get “The page could not be found” (proposed patch)
  • 783768 – QCH files: classes & their constructors get conflicting keyword handling< (proposed patch)
  • YETTOFILE – doxygen tag files contain origin paths for “file”, leaking info and perhaps is an issue with reproducible builds

And a related reported Qt issue:

There is also one related reported CMake issue:

  • 16990 – Wanted: Import support for custom targets (extra bonus: also export support)

And again, it would be also good to see the patch for a QtWebEngine based help engine getting more feedback and pushing by qualified people. And have distributions doing efforts to provide Qt Assistant and Qt Creator with *Web*-based documentation engines (see e.g. bug filed with openSUSE).

May the future be bright^Wdocumented

I am happy to see that Gentoo & FreeBSD packagers have already started to look into extending their KDE Frameworks packaging with generated API dox QCH files for the upcoming 5.36.0 release in July, with other packagers planning to do so soon as well.

So perhaps one not too distant day it will be just normal business to have QCH files with API documentation provided by your distribution not just for the Qt libraries itself, but also for every library based on them. After all documentation has been one of the things making Qt so attractive. As developer of Qt-based software, I very much look forward to that day 🙂

Next stop then: QML API documentation :/

Marble in your CMake or qmake based project, now more easy

Marble, the virtual globe and world atlas, at its core is a library intended for reuse, libmarblewidget. This library is done with minimal dependencies to ease usage (e.g. just needs a few Qt5 modules).

The new version of Marble, coming with libmarblewidget v0.26 and released as part of KDE Applications 16.12, is now making it more easy to integrate libmarblewidget in your own software project, when using CMake or qmake as buildsystem.

To link your project to libmarblewidget and find its headers, you can have that in 1-2 lines:

For qmake a qt_Marble.pri file is installed. So in the pro file of your project just add:

QT += Marble

For this to work you have to do one out of these:

  • install Marble into the same prefix as your Qt5 installation (e.g. /usr)
  • pass -DMARBLE_PRI_INSTALL_USE_QT_SYS_PATHS=ON to cmake when building Marble
  • use the environment variable QMAKEPATH to point to the installation prefix of Marble when calling qmake on your project

For CMake some CMake config files are installed. So in the CMakeLists.txt of your project just add:

find_package(Marble REQUIRED)
target_link_libraries(my_project  Marble)

No more need for some own FindMarble.cmake file.

The CMake variant is already possible since libmarblewidget v0.25 (part of KDE Applications 16.08).

Learn more about how to use Marble in your project, e.g. by simple examples, at marble.kde.org.

Kate, Gwenview & Co. unwantedly trading loss of file meta data against loss of content data, due to QSaveFile

tl;dr Code using QSaveFile (so Kate, KWrite, KDevelop, Gwenview & others) can result in loss of file meta data with extended file attributes, a change of file ownership and loss of hard links on existing files. Cmp. QTBUG-56366.

Wanted: Atomic changes to file content

When you are looking for a way to do transactional writes of files, QSaveFile looks like a simple and clear solution. But, this is 2016, so support for transactional updates of file content with mainstream file systems seems to be still rocket science? At least QSaveFile does a dirty hack as work-around: it creates a new temporary file to which all data is written into, and once all the content has arrived on the permanent storage, on the call of commit() it tries to replace the original file with that temporary file, by registering the new file with the name of the old in the file entries as replacement, which is an atomic system call.

Content data safely arrived, but meta data lost on the way

Just, and this is where things get dirty, this registration as replacement also will result in a replacement of all meta data for the old file with that of the new. And if the old file was registered as a hard link, that link will be dropped as well, the new file registered for the given name is now a file on its own decoupled from the formerly linked ones which also never will learn about the new content just written.
So the hack with the separate temporary file needs some more efforts to simulate a real transactional update of the content of that file identified by its name. Sadly some things are impossible and some things seem not yet implemented. Impossible is e.g. to set the ownership in most cases because for obvious reasons this needs special rights (cmp. API documentation of chown()). And e.g. not yet implemented seems to be copying of extended file attributes.

You who makes a difference?

There is no big warning sign yet on the API documentation of QSaveFile, so some developers might not be aware of these traps when using QSaveFile, like e.g. all over code in KDE projects. Or they are, but the users of their products are not (cmp. related bugs for Kate 333577, 354405, 358457).

So if you desire transactional updates of file contents in your code, consider if there are cases where the loss of the mentioned file meta data can be an issue, e.g. when those files are candidates to be shared on multi-user systems or when extended file attributes are being made use of with those files.

If you are a developer experienced with file system API, please consider writing a fix or improvement for QSaveFile and adopting QTBUG-56366: “QSaveFile results in loss of extended file attributes and change of file ownership”.

If you are a developer of file systems, please consider writing some system API which covers the use cases of transactional access to files. So there could be a proper implementation variant of QSaveFile and also any helpless hacks like .lock files would also no longer be needed.

And if you found wrong claims in this text, please shout out in the comments, as I only explored this issue quick and superficially for now, so chances are I missed something.