Look what you have done^W^Wdo!

You are using Kate or KDevelop and often editing directly the sources of Markdown files, Qt UI files, SVG files, Dot graph files and whatever else formats which are based on plain text files?

And you are having to use a workflow to check the current state which is saving the file and (re)loading it in a separate viewer application?

Could be better, right? Perhaps like this:

Nice mock-up. Or is it? Seems not! So, KParts-based preview plugin for Kate & KDevelop coming soon near you to a well-stocked package repository? Read the whole story where things are heading.

And a Markdown kpart? Never seen before. Because it is new. Actually seems the week of Markdown in the KDE community, there is also a patch on review to add Markdown support for Okular. Sadly Markdown support in Okular with the Okular kpart would not yet replace that separate Markdown kpart, because Okular does not support webview-style display (single “reactive” page), misses support for data streaming without touching the filesystem, and has too much chrome in the kpart for simple usage. Hope is on the future and somebody (you?) improving that. I am doing my share of improving things with this preview plugin, so I am out 🙂

Adding API dox QCH files generation to KDE Frameworks builds

Or: Tying loose ends where some are slightly too short yet.

When

  • you favour offline documentation (not only due to nice integration with IDEs like KDevelop),
  • develop code using KDE Frameworks or other Qt-based libraries,
  • you know all the KF5 libraries have seen many people taking care for API documentation in the code over all the years,
  • and you had read about doxygen’s capability to create API dox in QCH format,
  • and you want your Linux distribution package management to automatically deliver the latest version of the documentation (resp. QCH files) together with the KDE Frameworks libraries and headers (and ideally same for other Qt-based libraries),

the idea is easy derived to just extend the libraries’ buildsystem to also spit out QCH files during the package builds.

It’s all prepared, can ship next week, latest!!1

Which would just be a simple additional target and command, invoking doxygen with some proper configuration file. Right? So simple, you wonder why no-one had done it yet 🙂

Some initial challenge seems quickly handled, which is even more encouraging:
for proper documentation one also wants cross-linking to documentation of things used in the API which are from other libraries, e.g. base classes and types. Which requires to pass to doxygen the list of those other documentations together with a set of parameters, to generate proper qthelp:// urls or to copy over documentation for things like inherited methods.
Such listing gets very long especially for KDE Frameworks libraries in tier 3. And with indirect dependencies pulled into the API, on changes the list might get incomplete. Same with any other changes of the parameters for those other documentations.
So basically a similar situation to linking code libraries, which proposes to also give it a similar handling: placing the needed information with CMake config files of the targeted library, so whoever cross-links to the QCH file of that library can fetch the up-to-date information from there.

Things seemed to work okay on first tests, so last September a pull request was made to add some respective macro module to Extra-CMake-Modules to get things going and a blog post “Adding API dox generation to the build by CMake macros” was written.

This… works. You just need to prepare this. And ignore that.

Just, looking closer, lots of glitches popped up on the scene. Worse, even show stoppers made their introduction, at both ends of the process pipe:
At generation side doxygen turned out to have bitrotted for QCH creation, possibly due to lack of use? Time to sacrifice to the Powers of FLOSS, and git clone the sources and poke around to see what is broken and how to fix it. Some time and an accepted pull request later the biggest issue (some content missed to be added to the QCH file) was initially handled, just yet needed to also get out as released version (which it now is since some months).
At consumption side Qt Assistant and Qt Creator turned to be no longer able to properly show QCH files with JavaScript and other HTML5 content, due to QWebKit having been deprecated/dropped and both apps in many distributions now only using QTextBrowser for rendering the documentation pages. And not everyone is using KDevelop and its documentation browser, which uses QWebKit or, in master branch, favours QWebEngine if present.
Which means, an investment into QCH files from doxygen would only be interesting to a small audience. Myself currently without resources and interest to mess around with Qt help engine sources, looks with hope on the resurrection of QWebKit as well as the patch for a QtWebEngine based help engine (if you are Qt-involved, please help and push that patch some more!)

Finally kicking off the production cycle

Not properly working tools, nothing trying to use the tools on bigger scale… classical self-blocking state. So time to break this up and get some momentum into, by tying first things together where possible and enabling the generation of QCH files during builds of the KDE Frameworks libraries.

And thus to current master branches (which will become v5.36 in July) there has been now added for one to Extra-CMake-Modules the new module ECMAddQch and then to all of the KDE Frameworks libraries with public C++ API the option to generate QCH files with the API documentation, on passing -DBUILD_QCH=ON to cmake. If also having passed -DKDE_INSTALL_USE_QT_SYS_PATHS=ON (or installing to same prefix as Qt), the generated QCH files will be installed to places where Qt Assistant and Qt Creator even automatically pick them up and include them as expected:

Qt Assistant with lots of KF5 API dox

KDevelop picks them up as well, but needs some manual reconfiguration to do so.

(And of course ECMAddQch is designed to be useful for non-KF5 libraries as well, give it a try once you got hold of it!)

You and getting rid of the remaining obstacles

So while for some setups the generated QCH file of the KDE Frameworks already are useful (I use them since some weeks for e.g. KDevelop development, in KDevelop), for many they still have to become that. Which will take some more time and ideally contributions also from others, including Doxygen and Qt Help engine maintainers.

Here a list of related reported Doxygen bugs:

  • 773693 – Generated QCH files are missing dynsections.js & jquery.js, result in broken display (fixed for v1.8.13 by patch)
  • 773715 – Enabling QCH files without any JavaScript, for viewers without such support
  • 783759 – PERL_PATH config option: when is this needed? Still used?
  • 783762 – QCH files: “Namespaces” or “Files” in the navigation tree get “The page could not be found” (proposed patch)
  • 783768 – QCH files: classes & their constructors get conflicting keyword handling< (proposed patch)
  • YETTOFILE – doxygen tag files contain origin paths for “file”, leaking info and perhaps is an issue with reproducible builds

And some related reported Qt issues:

  • QTBUG-60933 – No qtdbus.tags file generated
  • QTBUG-61790 – Doxygen tag files have invalid data for enum types and values

There is also one related reported CMake issue:

  • 16990 – Wanted: Import support for custom targets (extra bonus: also export support)

And again, it would be also good to see the patch for a QtWebEngine based help engine getting more feedback and pushing by qualified people. And have distributions doing efforts to provide Qt Assistant and Qt Creator with *Web*-based documentation engines (see e.g. bug filed with openSUSE).

May the future be bright^Wdocumented

I am happy to see that Gentoo & FreeBSD packagers have already started to look into extending their KDE Frameworks packaging with generated API dox QCH files for the upcoming 5.36.0 release in July, with other packagers planning to do so soon as well.

So perhaps one not too distant day it will be just normal business to have QCH files with API documentation provided by your distribution not just for the Qt libraries itself, but also for every library based on them. After all documentation has been one of the things making Qt so attractive. As developer of Qt-based software, I very much look forward to that day 🙂

Next stop then: QML API documentation :/

Give us a proper mimetype name for OpenCL C files!

KDevelop, your cross-platform IDE, since version 5.1 has initial OpenCL language support.

If you are into OpenCL and interested, we need your help for some improvement to that:

What is the mimetype name to use for OpenCL C files?

Due to KDevelop needs some weeks ago a mimetype for OpenCL has already been added to the shared-mime-info database, with text/x-opencl-src as mimetype name.
Though if we reflect, that entry was quickly requested and made without consulting a lot of OpenCL users. It currently reads:

<mime-type type="text/x-opencl-src">
    <_comment>OpenCL source code</_comment>
    <acronym>OpenCL</acronym>
    <expanded-acronym>Open Computing Language</expanded-acronym>
    <sub-class-of type="text/x-csrc"/>
    <glob pattern="*.cl"/>
</mime-type>

For one, “OpenCL source code” should be rather “OpenCL C source code”. Because there are now both C and C++ variants (see below) which would need to be reflected in the name.

Next, text/x-opencl-src was made up and seems to be not used by anyone before. Asking your search engine, you will see that people in their desperation had already started to use all kind of other mimetype names before, e.g.:

  • text/x-opencl
  • text/x-clsrc
  • text/x-opencl_c
  • application/x-opencl_c

So text/x-opencl-src might not have been the best choice. Even more as again the C and C++ variants need to be reflected in the name.

Is there no-one at the Khronos Group who had time to look at deciding on a mimetype name and registering that with IANA as MIME type (or now Media type)?

Can you poke them, or hint us whom to talk to?

OpenCL C++

And then there is OpenCL C++ (since OpenCL 2.1), yet to reach the masses.
What mimetype name shall it have? What will the file extension(s) be?

Wanted: your wishes on managing author identity data in KDevelop

If you are a software developer who likes IDEs, please help to shape a new feature that is currently worked on for KDevelop, your attractive cross-platform IDE for currently C, C++, Python, JavaScript and PHP:

support for management of multiple author identities (name, email, …)

Please help by telling your user stories around data about your author identity in your development process. Add the things you would like to see supported (soon or one day) here:
https://community.kde.org/KDevelop/AuthorIdentities#User_stories

Other contributions to that page are welcome, too. It is a wiki 🙂
(If a wiki scares you, replying in the comments to this blog post is also fine, everything will be picked up.)

To give you an idea what kind of things are wanted, read on:

Some initial user stories

Do not get in the way:

As developer working alone on a private project, I do not want to have to deal with any identity stuff, any defaults are fine with me.

Multiple identities in same system:

As developer working with the same computer/system on different projects for different customers or different project groups, I want KDevelop to always inject the correct/matching copyright & author information on generating code from file & app templates for different projects/customers, instead of having to manually post-edit the generated code.

Multiple identities in same project:

As developer working on the same project with the same computer system both for job and privately, I want to quickly switch identity data to be used depending on whether I work for job or privately, to mark copyright of contributions accordingly.

Pick up identity on import:

As developer importing a project into KDevelop, I want to have the identity automatically derived from account/server/fetched project and have the option to create a new identity from the found data if there is none, or overwrite with an existing identity.

Highlight my contributions in VCS history:

As developer working on a project with VCS support, I want my easily see my own commits in the vcs history, e.g. by being highlighted.

Update existing identity data:

As developer working on a project, I want to have assistance when identity properties had been changed (like email address or name) and this should be reflected in all existing copyright/contributor notions of a project, where possible.

Following some more background info why work has been started on this feature:

Problem

When it comes to copyright notions in code/files and authorship/committer id in version control systems, the name and the contact information, like email address, can be a few different ones for oneself. E.g. when working in different FLOSS projects, with each email address dedicated to the project, or when working on a FLOSS project both in name of an employer, but also privately in free times.

Currently KDevelop is hard-coded to use whatever is set for the default profile of KEMailSettings when generating new code from file templates, the only place right now using authorship metadata. KMail, the well-known email client, seems to be feeding KEMailSettings via the KIdentityManagement module, but otherwise there might be no data in there. So the developer has to manually complete the fields in the generated code.

The integration of KDevelop with git also ignores the author and commiter metadata used by git for the repo. They need to be set manually and are nowhere linked to what KDevelop knows.

For a truely integrated development experience this needs to be improved.

Solution

KDevelop could know about the concept of author identity, could allow to have multiple identities available to select from for a project, and could allow plugins to make use of identity details e.g. on doing VCS commits or generating code.

Here a screenshot of some proof-of-concept code:
Draft of KDevelop settings: Configure Author Identitites

So, what do you also think in general? Anything related which is worth looking
into for this? Other comments?

Marble in your CMake or qmake based project, now more easy

Marble, the virtual globe and world atlas, at its core is a library intended for reuse, libmarblewidget. This library is done with minimal dependencies to ease usage (e.g. just needs a few Qt5 modules).

The new version of Marble, coming with libmarblewidget v0.26 and released as part of KDE Applications 16.12, is now making it more easy to integrate libmarblewidget in your own software project, when using CMake or qmake as buildsystem.

To link your project to libmarblewidget and find its headers, you can have that in 1-2 lines:

For qmake a qt_Marble.pri file is installed. So in the pro file of your project just add:

QT += Marble

For this to work you have to do one out of these:

  • install Marble into the same prefix as your Qt5 installation (e.g. /usr)
  • pass -DMARBLE_PRI_INSTALL_USE_QT_SYS_PATHS=ON to cmake when building Marble
  • use the environment variable QMAKEPATH to point to the installation prefix of Marble when calling qmake on your project

For CMake some CMake config files are installed. So in the CMakeLists.txt of your project just add:

find_package(Marble REQUIRED)
target_link_libraries(my_project  Marble)

No more need for some own FindMarble.cmake file.

The CMake variant is already possible since libmarblewidget v0.25 (part of KDE Applications 16.08).

Learn more about how to use Marble in your project, e.g. by simple examples, at marble.kde.org.

Kate, Gwenview & Co. unwantedly trading loss of file meta data against loss of content data, due to QSaveFile

tl;dr Code using QSaveFile (so Kate, KWrite, KDevelop, Gwenview & others) can result in loss of file meta data with extended file attributes, a change of file ownership and loss of hard links on existing files. Cmp. QTBUG-56366.

Wanted: Atomic changes to file content

When you are looking for a way to do transactional writes of files, QSaveFile looks like a simple and clear solution. But, this is 2016, so support for transactional updates of file content with mainstream file systems seems to be still rocket science? At least QSaveFile does a dirty hack as work-around: it creates a new temporary file to which all data is written into, and once all the content has arrived on the permanent storage, on the call of commit() it tries to replace the original file with that temporary file, by registering the new file with the name of the old in the file entries as replacement, which is an atomic system call.

Content data safely arrived, but meta data lost on the way

Just, and this is where things get dirty, this registration as replacement also will result in a replacement of all meta data for the old file with that of the new. And if the old file was registered as a hard link, that link will be dropped as well, the new file registered for the given name is now a file on its own decoupled from the formerly linked ones which also never will learn about the new content just written.
So the hack with the separate temporary file needs some more efforts to simulate a real transactional update of the content of that file identified by its name. Sadly some things are impossible and some things seem not yet implemented. Impossible is e.g. to set the ownership in most cases because for obvious reasons this needs special rights (cmp. API documentation of chown()). And e.g. not yet implemented seems to be copying of extended file attributes.

You who makes a difference?

There is no big warning sign yet on the API documentation of QSaveFile, so some developers might not be aware of these traps when using QSaveFile, like e.g. all over code in KDE projects. Or they are, but the users of their products are not (cmp. related bugs for Kate 333577, 354405, 358457).

So if you desire transactional updates of file contents in your code, consider if there are cases where the loss of the mentioned file meta data can be an issue, e.g. when those files are candidates to be shared on multi-user systems or when extended file attributes are being made use of with those files.

If you are a developer experienced with file system API, please consider writing a fix or improvement for QSaveFile and adopting QTBUG-56366: “QSaveFile results in loss of extended file attributes and change of file ownership”.

If you are a developer of file systems, please consider writing some system API which covers the use cases of transactional access to files. So there could be a proper implementation variant of QSaveFile and also any helpless hacks like .lock files would also no longer be needed.

And if you found wrong claims in this text, please shout out in the comments, as I only explored this issue quick and superficially for now, so chances are I missed something.

Adding API dox generation to the build by CMake macros

Online documentations of APIs are a fine thing: by their urls they provide some shared universal location to refer to and often also are kept up-to-date to latest improvements. And they serve as index trap for search engine robots, so more people have the chance to find out about some product.

But the world is sometimes not an online heaven:

  • “my internet is gone!”
  • documentation webserver is down
  • documentation webserver is slooo….oo…ow
  • documentation webserver no longer has documentation for that older version

So having also access to offline documentation as alternative can make some developers using your library happy. If you are using Qt in your projects, you yourself might even prefer to study the offline Qt API documentation installed on your local systems, either via the Qt Assistant program or as integrated in IDEs like Qt Creator or KDevelop. Even better, the offline documentation comes together with the library itself (e.g. via packages on Linux or *BSD distributions), so the documentation is always matching the version of the library.

Being a good developer you have already added lots of nice API documentation in your code. Now how to turn this into a separate offline documentation manual for the consumers of your library, without too much effort?
If your project uses CMake for the build system and your target audience is using documentation viewers like Qt Assistant, Qt Creator or KDevelop, read on for a possible solution.

Deploying doxygen & qhelpgenerator

Very possibly you are already using doxygen to generate e.g. an HTML version of the API dox for your library project. And you might know doxygen can output to more formats than HTML. One of them is the format “Qt Compressed Help” (QCH). Which is a single-file format the above mentioned documentation viewers support well. QCH support by doxygen even exists since 2009.

So, with doxygen & qhelpgenerator installed and a respective doxygen config file prepared, the build script of your library just needs to additionally call doxygen as part of another target, and there is the API documentation manual as part of your project build, ready to be deployed and packaged with your library.

A CMake macro for convenience

To not have everyone duplicate the needed build setup code, like checking for the needed tools being installed, and to not have to learn all the details of the doxygen config file, some CMake macro doing the dull work recommends itself to be written and to be distributed for reuse.

Before doing that though, let’s consider another aspect when it comes to documenting libraries: including documentation for code constructs used from external libraries.

Linking between offline API dox manuals

Especially for C++ libraries, where subclassing across classes from different libraries is common, one wants to have direct links into the documentation of external libraries, so the reader has transparent access to any info.

The Qt help system has an abstract addressing system which works independently from the actual location of the QCH files e.g. in the filesystem. These addresses use the scheme qthelp: and a domain namespace for identifying a given manual (e.g. org.qt-project.qtcore). Additionally there is the concept of a virtual folder which is also set as root directory within a manual (e.g. qtcore). So the address of the document for the class QString would be qthelp://org.qt-project.qtcore/qtcore/qstring.html, and the documentation viewer would resolve it to the manuals registered by the matching metadata.

Doxygen supports the creation of such links into external QCH files. For a given manual of an external library, the references are created based on the domain name, the virtual folder name and a so-called tag file for that library. Such a tag file contains all kind of metadata about classes, methods and more that is needed to document imported structures and to create the references.

The QCH files for Qt libraries are coming with such tag files. But what about your own library? What if somebody else should be able to create documentation for their code with links into the documentation of your own library?
Doxygen helps here as well, during the creation of the API dox manual it optionally also creates a respective tag file.

Extending CMake Config files with API dox info

So for generating the references into some external library documentation, the location of its tag file, its name of the domain and its name of the virtual folder need to be known and explicitly added to the doxygen config file. Which is extra work, error-prone due to typos and might miss changes.

But it can be automated. All the information is known to the builder of that external library. And with the builder storing the information in standardized CMake variables with the external library’s CMake config files, it could also be automatically processed into the needed doxygen config entries in the build of your own library.

So a simple find_package(Bar) would bring in all the needed data, and passing Bar as name of the external library should be enough to let the to-be-written CMake macro derive the names of the variables with that data.

So with these theoretic considerations finally some real code as draft:

Patch uploaded for your review and comments

As possible extension of the Extra-CMake-Modules (ECM) the review request https://phabricator.kde.org/D2854 proposes 2 macros for making it simple and easy to add to the build of own libraries the generation of API dox manuals with links into external API dox manuals.

The call of the macro for generating the tag and qch files would e.g. be like this:

ecm_generate_qch(
    Foo
    VERSION "2.1.0"
    ORG_DOMAIN org.kde
    SOURCE_DIRS
        ${CMAKE_SOURCE_DIR}/src
    EXTERN
        Qt5Core
        Bar
    QCH_INSTALL_DESTINATION ${KDE_INSTALL_FULL_DATADIR}/qch
    TAGS_INSTALL_DESTINATION ${KDE_INSTALL_FULL_DATADIR}/qch
)

This would create and install the files ${KDE_INSTALL_FULL_DATADIR}/qch/Foo.qch and ${KDE_INSTALL_FULL_DATADIR}/qch/Foo.tags.

The call for generating the CMake Config file would be:

ecm_generate_package_apidox_file(
    ${CMAKE_CURRENT_BINARY_DIR}/FooConfigApiDox.cmake
    NAMES Foo
)

This generates the file FooConfigApiDox.cmake, whose content is setting the respective variables with the API dox metadata:

set(Foo_APIDOX_TAGSFILE "/usr/share/docs/Foo.tags")
set(Foo_APIDOX_QHP_NAMESPACE "org.kde.Foo")
set(Foo_APIDOX_QHP_NAMESPACE_VERSIONED "org.kde.Foo.210")
set(Foo_APIDOX_QHP_VIRTUALFOLDER "Foo")

This file would then be included in the FooConfig.cmake file

# ...
include("${CMAKE_CURRENT_LIST_DIR}/FooApiDox.cmake")
# ...

and installed together in the same directory.

Tightening and extending integration

In the current draft, for defining what source files doxygen should include for generating the API documentation, the macro ecm_generate_qch() takes simply a list of directories via the SOURCE_DIRS argument list, which then together with some wildcard patterns for typical C and C++ files is written to the doxygen config file.
Though being part of the build system, there might be more specific info available what files should be used, e.g. by using the PUBLIC_HEADER property of a library target. Anyone experienced with the usage of that or similar CMake features are invited to propose how to integrate this into the API of the proposed new macro.

Currently there is no standard installation directory for QCH files. So documentation viewers supporting QCH files need to be manually pointed to any new installed QCH file (exception are the QCH files for Qt, whose location can be queried by qmake -query QT_INSTALL_DOCS).
Instead it would be nice those viewers pick up such files automatically. How could that be achieved?

It would be also good if this or some similar API dox metadata system could be upstreamed into CMake itself, so there is a reliable standard for this, helping everyone with the creation of cross-project integrated API manuals. At least the metadata variable naming patterns would need to get wide recognition to get really useful. Anyone interested in pushing this upstream?

For proper documentation of public and non-public API perhaps the doxygen development could also pick up this old thread about missing proper support for marking classes & methods as internal. Or is there meanwhile a good solution for this?

On the product side, what manual formats beside QCH should be supported?
On the tool side, what other API documentation tools should be supported (perhaps something clang-based)?

Comments welcome here or on the phabricator page.