Adding API dox QCH files generation to KDE Frameworks builds

Or: Tying loose ends where some are slightly too short yet.

When

  • you favour offline documentation (not only due to nice integration with IDEs like KDevelop),
  • develop code using KDE Frameworks or other Qt-based libraries,
  • you know all the KF5 libraries have seen many people taking care for API documentation in the code over all the years,
  • and you had read about doxygen’s capability to create API dox in QCH format,
  • and you want your Linux distribution package management to automatically deliver the latest version of the documentation (resp. QCH files) together with the KDE Frameworks libraries and headers (and ideally same for other Qt-based libraries),

the idea is easy derived to just extend the libraries’ buildsystem to also spit out QCH files during the package builds.

It’s all prepared, can ship next week, latest!!1

Which would just be a simple additional target and command, invoking doxygen with some proper configuration file. Right? So simple, you wonder why no-one had done it yet 🙂

Some initial challenge seems quickly handled, which is even more encouraging:
for proper documentation one also wants cross-linking to documentation of things used in the API which are from other libraries, e.g. base classes and types. Which requires to pass to doxygen the list of those other documentations together with a set of parameters, to generate proper qthelp:// urls or to copy over documentation for things like inherited methods.
Such listing gets very long especially for KDE Frameworks libraries in tier 3. And with indirect dependencies pulled into the API, on changes the list might get incomplete. Same with any other changes of the parameters for those other documentations.
So basically a similar situation to linking code libraries, which proposes to also give it a similar handling: placing the needed information with CMake config files of the targeted library, so whoever cross-links to the QCH file of that library can fetch the up-to-date information from there.

Things seemed to work okay on first tests, so last September a pull request was made to add some respective macro module to Extra-CMake-Modules to get things going and a blog post “Adding API dox generation to the build by CMake macros” was written.

This… works. You just need to prepare this. And ignore that.

Just, looking closer, lots of glitches popped up on the scene. Worse, even show stoppers made their introduction, at both ends of the process pipe:
At generation side doxygen turned out to have bitrotted for QCH creation, possibly due to lack of use? Time to sacrifice to the Powers of FLOSS, and git clone the sources and poke around to see what is broken and how to fix it. Some time and an accepted pull request later the biggest issue (some content missed to be added to the QCH file) was initially handled, just yet needed to also get out as released version (which it now is since some months).
At consumption side Qt Assistant and Qt Creator turned to be no longer able to properly show QCH files with JavaScript and other HTML5 content, due to QWebKit having been deprecated/dropped and both apps in many distributions now only using QTextBrowser for rendering the documentation pages. And not everyone is using KDevelop and its documentation browser, which uses QWebKit or, in master branch, favours QWebEngine if present.
Which means, an investment into QCH files from doxygen would only be interesting to a small audience. Myself currently without resources and interest to mess around with Qt help engine sources, looks with hope on the resurrection of QWebKit as well as the patch for a QtWebEngine based help engine (if you are Qt-involved, please help and push that patch some more!)

Finally kicking off the production cycle

Not properly working tools, nothing trying to use the tools on bigger scale… classical self-blocking state. So time to break this up and get some momentum into, by tying first things together where possible and enabling the generation of QCH files during builds of the KDE Frameworks libraries.

And thus to current master branches (which will become v5.36 in July) there has been now added for one to Extra-CMake-Modules the new module ECMAddQch and then to all of the KDE Frameworks libraries with public C++ API the option to generate QCH files with the API documentation, on passing -DBUILD_QCH=ON to cmake. If also having passed -DKDE_INSTALL_USE_QT_SYS_PATHS=ON (or installing to same prefix as Qt), the generated QCH files will be installed to places where Qt Assistant and Qt Creator even automatically pick them up and include them as expected:

Qt Assistant with lots of KF5 API dox

KDevelop picks them up as well, but needs some manual reconfiguration to do so.

(And of course ECMAddQch is designed to be useful for non-KF5 libraries as well, give it a try once you got hold of it!)

You and getting rid of the remaining obstacles

So while for some setups the generated QCH file of the KDE Frameworks already are useful (I use them since some weeks for e.g. KDevelop development, in KDevelop), for many they still have to become that. Which will take some more time and ideally contributions also from others, including Doxygen and Qt Help engine maintainers.

Here a list of related reported Doxygen bugs:

  • 773693 – Generated QCH files are missing dynsections.js & jquery.js, result in broken display (fixed for v1.8.13 by patch)
  • 773715 – Enabling QCH files without any JavaScript, for viewers without such support
  • 783759 – PERL_PATH config option: when is this needed? Still used?
  • 783762 – QCH files: “Namespaces” or “Files” in the navigation tree get “The page could not be found” (proposed patch)
  • 783768 – QCH files: classes & their constructors get conflicting keyword handling< (proposed patch)
  • YETTOFILE – doxygen tag files contain origin paths for “file”, leaking info and perhaps is an issue with reproducible builds

And a related reported Qt issue:

There is also one related reported CMake issue:

  • 16990 – Wanted: Import support for custom targets (extra bonus: also export support)

And again, it would be also good to see the patch for a QtWebEngine based help engine getting more feedback and pushing by qualified people. And have distributions doing efforts to provide Qt Assistant and Qt Creator with *Web*-based documentation engines (see e.g. bug filed with openSUSE).

May the future be bright^Wdocumented

I am happy to see that Gentoo & FreeBSD packagers have already started to look into extending their KDE Frameworks packaging with generated API dox QCH files for the upcoming 5.36.0 release in July, with other packagers planning to do so soon as well.

So perhaps one not too distant day it will be just normal business to have QCH files with API documentation provided by your distribution not just for the Qt libraries itself, but also for every library based on them. After all documentation has been one of the things making Qt so attractive. As developer of Qt-based software, I very much look forward to that day 🙂

Next stop then: QML API documentation :/

Give us a proper mimetype name for OpenCL C files!

KDevelop, your cross-platform IDE, since version 5.1 has initial OpenCL language support.

If you are into OpenCL and interested, we need your help for some improvement to that:

What is the mimetype name to use for OpenCL C files?

Due to KDevelop needs some weeks ago a mimetype for OpenCL has already been added to the shared-mime-info database, with text/x-opencl-src as mimetype name.
Though if we reflect, that entry was quickly requested and made without consulting a lot of OpenCL users. It currently reads:

<mime-type type="text/x-opencl-src">
    <_comment>OpenCL source code</_comment>
    <acronym>OpenCL</acronym>
    <expanded-acronym>Open Computing Language</expanded-acronym>
    <sub-class-of type="text/x-csrc"/>
    <glob pattern="*.cl"/>
</mime-type>

For one, “OpenCL source code” should be rather “OpenCL C source code”. Because there are now both C and C++ variants (see below) which would need to be reflected in the name.

Next, text/x-opencl-src was made up and seems to be not used by anyone before. Asking your search engine, you will see that people in their desperation had already started to use all kind of other mimetype names before, e.g.:

  • text/x-opencl
  • text/x-clsrc
  • text/x-opencl_c
  • application/x-opencl_c

So text/x-opencl-src might not have been the best choice. Even more as again the C and C++ variants need to be reflected in the name.

Is there no-one at the Khronos Group who had time to look at deciding on a mimetype name and registering that with IANA as MIME type (or now Media type)?

Can you poke them, or hint us whom to talk to?

OpenCL C++

And then there is OpenCL C++ (since OpenCL 2.1), yet to reach the masses.
What mimetype name shall it have? What will the file extension(s) be?

Wanted: your wishes on managing author identity data in KDevelop

If you are a software developer who likes IDEs, please help to shape a new feature that is currently worked on for KDevelop, your attractive cross-platform IDE for currently C, C++, Python, JavaScript and PHP:

support for management of multiple author identities (name, email, …)

Please help by telling your user stories around data about your author identity in your development process. Add the things you would like to see supported (soon or one day) here:
https://community.kde.org/KDevelop/AuthorIdentities#User_stories

Other contributions to that page are welcome, too. It is a wiki 🙂
(If a wiki scares you, replying in the comments to this blog post is also fine, everything will be picked up.)

To give you an idea what kind of things are wanted, read on:

Some initial user stories

Do not get in the way:

As developer working alone on a private project, I do not want to have to deal with any identity stuff, any defaults are fine with me.

Multiple identities in same system:

As developer working with the same computer/system on different projects for different customers or different project groups, I want KDevelop to always inject the correct/matching copyright & author information on generating code from file & app templates for different projects/customers, instead of having to manually post-edit the generated code.

Multiple identities in same project:

As developer working on the same project with the same computer system both for job and privately, I want to quickly switch identity data to be used depending on whether I work for job or privately, to mark copyright of contributions accordingly.

Pick up identity on import:

As developer importing a project into KDevelop, I want to have the identity automatically derived from account/server/fetched project and have the option to create a new identity from the found data if there is none, or overwrite with an existing identity.

Highlight my contributions in VCS history:

As developer working on a project with VCS support, I want my easily see my own commits in the vcs history, e.g. by being highlighted.

Update existing identity data:

As developer working on a project, I want to have assistance when identity properties had been changed (like email address or name) and this should be reflected in all existing copyright/contributor notions of a project, where possible.

Following some more background info why work has been started on this feature:

Problem

When it comes to copyright notions in code/files and authorship/committer id in version control systems, the name and the contact information, like email address, can be a few different ones for oneself. E.g. when working in different FLOSS projects, with each email address dedicated to the project, or when working on a FLOSS project both in name of an employer, but also privately in free times.

Currently KDevelop is hard-coded to use whatever is set for the default profile of KEMailSettings when generating new code from file templates, the only place right now using authorship metadata. KMail, the well-known email client, seems to be feeding KEMailSettings via the KIdentityManagement module, but otherwise there might be no data in there. So the developer has to manually complete the fields in the generated code.

The integration of KDevelop with git also ignores the author and commiter metadata used by git for the repo. They need to be set manually and are nowhere linked to what KDevelop knows.

For a truely integrated development experience this needs to be improved.

Solution

KDevelop could know about the concept of author identity, could allow to have multiple identities available to select from for a project, and could allow plugins to make use of identity details e.g. on doing VCS commits or generating code.

Here a screenshot of some proof-of-concept code:
Draft of KDevelop settings: Configure Author Identitites

So, what do you also think in general? Anything related which is worth looking
into for this? Other comments?

KDevelop 4.1 arrived, features Okteta plugin

KDevelop is out in version 4.1, and better than ever! Thank you, great hackers of KDevelop, I would need to learn Emacs/Vim without you! 🙂

Among all the great features there is now also what had been basically done at the Kate-KDevelop sprint in Berlin this February, when streets were covered with thick ice and full of snow, a first integration of most of the Okteta components in KDevelop.

Activate the Okteta plugin, open any file as raw data simply by selecting “Open as Byte Array” in the context menu of all file listing sidebar tools, and enjoy the hex editing support directly in KDevelop:

Provided by the plugin is also the Structures tool (see also screenshot above). As of Okteta version 0.5 (part of KDE Applications 4.5) structures can now additionally be defined dynamically, using JavaScript. So if the old XML format was too restrictive for you, give it another try now. And think about sharing your results on kde-files.org as well!

Okteta going for KDevelop (and Kate?)

While there already is a KPart from the hex editor Okteta, which can be embedded e.g. into Konqueror, but also KDevelop, it is only used Read-Only. More, the KPart system does not enable mulitple views on the same document, so in KDevelop having two views opened for a file, like with a splitted view area, also means two copies of the document in the working memory. Which also results in synchronisation problems, if you do (if you could) changes in each of the views. All this is part of my motivation for the Kasten framework. But until that is done, real life demands pragmatical solutions. Like manually writing plugins for the individual frameworks, e.g. for Sublime/KDevPlatform, if going for KDevelop integration.

And having good integration of Okteta in KDevelop, with all tools and whistles, has been something I often wanted to have. What better can I do than joining a bunch of KDeveloper developers for some days, for first-class coding support? Like at the current Kate-KDevelop sprint in Berlin? Organized by Milian Wolff, and with support from the KDE e.V. and the Physics faculty of the FU Berlin, I am now happy to sit between the Kate and the KDevelop code crunchers (“Let’s remove this!”, “Yeah”, “No”, …) since three days.

Being attached to a big network first shortly distracted me, because there are a lot of devices with a lot of unknown services announced in the service description systems like zeroconf/DNS-SD. Just look at the number of printers:

It also gave me an idea for a new heuristic to guess the type of device: One with a lot of unknown services must be an iPhone, as each iPhone app seems to invent a new custom network service, with a custom protocol. I do not think the network:/ kio-slave will ever be able to and should support all of them.

After being distracted I returned to my initial target, and with the help of Alexander (adymo) , Niko (nsams) and Aleix (apol) the Okteta plugin for KDevelop is already at the state of the Okteta KPart and better. Just the tools need some more work, as the Kasten framework does not match the KDevPlatform/Sublime framework here, so the wrapper for the tools are a little more complicated to be done. But look doable. Screenshot at last:

And yes, Okteta as a stand-alone program is going to stay.

Binspekt KPart for reuse in KDevelop

Some days ago I wrote a mini-tutorial how to use the Okteta KPart in Konqueror/KDevelop. With the mind turned to the Okteta KPart I after that have taken on a long waiting task and removed some code duplication by porting it to the Kasten classes, now that these are available in shared libs. Which means, with KDE 4.4 you should have the same readonly actions available with the Okteta KPart that you also have in the Okteta program (not all done yet). Even better, I have just written a first code line to turn it into a ReadWritePart, so that you could also do the editing and saving in KDevelop. More on this later hopefully.

After having done this first wrapping of Kasten classes in a KPart I did another one, to learn more about how a generic KPart Kasten module might be best designed, to get the Kasten plug&play experience I am aiming at. The other program I am writing using the Kasten framework is Binspekt, a viewer (and perhaps editor) for binary files, that is executables, libraries, object files etc. It was basically copy,paste&replace, so here see the small and dirty Binspekt KPart inside the (amazing and great BTW) KDevelop, fresh from the KDE repository:
Binspekt KPart in KDevelop

Binspekt is still prealpha (yet doesn’t crash, because it doesn’t do a lot) if you are interested. But it all has to start somewhere, no? 🙂

Minitutorial: Viewing raw data of files in Konqueror/KDevelop

Want to quickly see the raw data of files in Konqueror or KDevelop? This is possible with the KPart coming with Okteta since KDE 4.1:

Preparations

Open “System Settings”, then select “File Associations” (Found under “Advanced”). Select the file type (e.g. “audio/x-wav”) you want to see the raw data of, choose the tab “Embedding”.

Within “Services Preference Order” press button “Add”, select “Embedded Binary Viewer (oktetapart)” [0] from the list, press “OK”:
File Associations - Add Service

If there are already other services listed in the services list make sure the oktetapart is the first entry by selecting it and pressing “Move Up” as often as needed. Press “Apply” and close the System Settings:
File Associations

[0] Binary Viewer is a misnomer, changed to Hex Viewer for KDE 4.4

Usage

Now start Konqueror and click with your right mouse button on a wav file. In the context menu select “Preview in Embedded Binary Viewer” and you should see the raw data of that file:
Hex View in Konqueror
In KDevelop you do as you do with text files, just select a wav file and it will be shown as hex view:
Hex View in KDevelop

Beware: This viewer is only read-only. And while using the Okteta core libs it currently does not come with any of the tools included in Okteta. This hopefully will change for KDE 4.4, but don’t hold your breath (better sit down and help with coding 🙂 ).

Undo

If you want to undo this change for the file type, go to the settings as described above, select “Embedded Binary Viewer (oktetapart)” in the list under “Services Preference Order” and press “Remove”.

Update:

Since its version 4.1 KDevelop has built-in support for viewing and editing of files as byte arrays (of course using the Okteta libraries). To open a file as byte array, use the context menu on a file in the “Filesystem”, “Projects” or “Documents” toolview (by click with the right mouse button) and chose “Open as Byte Array”. So for KDevelop using the Okteta KPart is no longer needed.