Kate, Gwenview & Co. unwantedly trading loss of file meta data against loss of content data, due to QSaveFile

tl;dr Code using QSaveFile (so Kate, KWrite, KDevelop, Gwenview & others) can result in loss of file meta data with extended file attributes, a change of file ownership and loss of hard links on existing files. Cmp. QTBUG-56366.

Wanted: Atomic changes to file content

When you are looking for a way to do transactional writes of files, QSaveFile looks like a simple and clear solution. But, this is 2016, so support for transactional updates of file content with mainstream file systems seems to be still rocket science? At least QSaveFile does a dirty hack as work-around: it creates a new temporary file to which all data is written into, and once all the content has arrived on the permanent storage, on the call of commit() it tries to replace the original file with that temporary file, by registering the new file with the name of the old in the file entries as replacement, which is an atomic system call.

Content data safely arrived, but meta data lost on the way

Just, and this is where things get dirty, this registration as replacement also will result in a replacement of all meta data for the old file with that of the new. And if the old file was registered as a hard link, that link will be dropped as well, the new file registered for the given name is now a file on its own decoupled from the formerly linked ones which also never will learn about the new content just written.
So the hack with the separate temporary file needs some more efforts to simulate a real transactional update of the content of that file identified by its name. Sadly some things are impossible and some things seem not yet implemented. Impossible is e.g. to set the ownership in most cases because for obvious reasons this needs special rights (cmp. API documentation of chown()). And e.g. not yet implemented seems to be copying of extended file attributes.

You who makes a difference?

There is no big warning sign yet on the API documentation of QSaveFile, so some developers might not be aware of these traps when using QSaveFile, like e.g. all over code in KDE projects. Or they are, but the users of their products are not (cmp. related bugs for Kate 333577, 354405, 358457).

So if you desire transactional updates of file contents in your code, consider if there are cases where the loss of the mentioned file meta data can be an issue, e.g. when those files are candidates to be shared on multi-user systems or when extended file attributes are being made use of with those files.

If you are a developer experienced with file system API, please consider writing a fix or improvement for QSaveFile and adopting QTBUG-56366: “QSaveFile results in loss of extended file attributes and change of file ownership”.

If you are a developer of file systems, please consider writing some system API which covers the use cases of transactional access to files. So there could be a proper implementation variant of QSaveFile and also any helpless hacks like .lock files would also no longer be needed.

And if you found wrong claims in this text, please shout out in the comments, as I only explored this issue quick and superficially for now, so chances are I missed something.

Adding API dox generation to the build by CMake macros

Online documentations of APIs are a fine thing: by their urls they provide some shared universal location to refer to and often also are kept up-to-date to latest improvements. And they serve as index trap for search engine robots, so more people have the chance to find out about some product.

But the world is sometimes not an online heaven:

  • “my internet is gone!”
  • documentation webserver is down
  • documentation webserver is slooo….oo…ow
  • documentation webserver no longer has documentation for that older version

So having also access to offline documentation as alternative can make some developers using your library happy. If you are using Qt in your projects, you yourself might even prefer to study the offline Qt API documentation installed on your local systems, either via the Qt Assistant program or as integrated in IDEs like Qt Creator or KDevelop. Even better, the offline documentation comes together with the library itself (e.g. via packages on Linux or *BSD distributions), so the documentation is always matching the version of the library.

Being a good developer you have already added lots of nice API documentation in your code. Now how to turn this into a separate offline documentation manual for the consumers of your library, without too much effort?
If your project uses CMake for the build system and your target audience is using documentation viewers like Qt Assistant, Qt Creator or KDevelop, read on for a possible solution.

Deploying doxygen & qhelpgenerator

Very possibly you are already using doxygen to generate e.g. an HTML version of the API dox for your library project. And you might know doxygen can output to more formats than HTML. One of them is the format “Qt Compressed Help” (QCH). Which is a single-file format the above mentioned documentation viewers support well. QCH support by doxygen even exists since 2009.

So, with doxygen & qhelpgenerator installed and a respective doxygen config file prepared, the build script of your library just needs to additionally call doxygen as part of another target, and there is the API documentation manual as part of your project build, ready to be deployed and packaged with your library.

A CMake macro for convenience

To not have everyone duplicate the needed build setup code, like checking for the needed tools being installed, and to not have to learn all the details of the doxygen config file, some CMake macro doing the dull work recommends itself to be written and to be distributed for reuse.

Before doing that though, let’s consider another aspect when it comes to documenting libraries: including documentation for code constructs used from external libraries.

Linking between offline API dox manuals

Especially for C++ libraries, where subclassing across classes from different libraries is common, one wants to have direct links into the documentation of external libraries, so the reader has transparent access to any info.

The Qt help system has an abstract addressing system which works independently from the actual location of the QCH files e.g. in the filesystem. These addresses use the scheme qthelp: and a domain namespace for identifying a given manual (e.g. org.qt-project.qtcore). Additionally there is the concept of a virtual folder which is also set as root directory within a manual (e.g. qtcore). So the address of the document for the class QString would be qthelp://org.qt-project.qtcore/qtcore/qstring.html, and the documentation viewer would resolve it to the manuals registered by the matching metadata.

Doxygen supports the creation of such links into external QCH files. For a given manual of an external library, the references are created based on the domain name, the virtual folder name and a so-called tag file for that library. Such a tag file contains all kind of metadata about classes, methods and more that is needed to document imported structures and to create the references.

The QCH files for Qt libraries are coming with such tag files. But what about your own library? What if somebody else should be able to create documentation for their code with links into the documentation of your own library?
Doxygen helps here as well, during the creation of the API dox manual it optionally also creates a respective tag file.

Extending CMake Config files with API dox info

So for generating the references into some external library documentation, the location of its tag file, its name of the domain and its name of the virtual folder need to be known and explicitly added to the doxygen config file. Which is extra work, error-prone due to typos and might miss changes.

But it can be automated. All the information is known to the builder of that external library. And with the builder storing the information in standardized CMake variables with the external library’s CMake config files, it could also be automatically processed into the needed doxygen config entries in the build of your own library.

So a simple find_package(Bar) would bring in all the needed data, and passing Bar as name of the external library should be enough to let the to-be-written CMake macro derive the names of the variables with that data.

So with these theoretic considerations finally some real code as draft:

Patch uploaded for your review and comments

As possible extension of the Extra-CMake-Modules (ECM) the review request https://phabricator.kde.org/D2854 proposes 2 macros for making it simple and easy to add to the build of own libraries the generation of API dox manuals with links into external API dox manuals.

The call of the macro for generating the tag and qch files would e.g. be like this:

    VERSION "2.1.0"
    ORG_DOMAIN org.kde

This would create and install the files ${KDE_INSTALL_FULL_DATADIR}/qch/Foo.qch and ${KDE_INSTALL_FULL_DATADIR}/qch/Foo.tags.

The call for generating the CMake Config file would be:

    NAMES Foo

This generates the file FooConfigApiDox.cmake, whose content is setting the respective variables with the API dox metadata:

set(Foo_APIDOX_TAGSFILE "/usr/share/docs/Foo.tags")
set(Foo_APIDOX_QHP_NAMESPACE "org.kde.Foo")

This file would then be included in the FooConfig.cmake file

# ...
# ...

and installed together in the same directory.

Tightening and extending integration

In the current draft, for defining what source files doxygen should include for generating the API documentation, the macro ecm_generate_qch() takes simply a list of directories via the SOURCE_DIRS argument list, which then together with some wildcard patterns for typical C and C++ files is written to the doxygen config file.
Though being part of the build system, there might be more specific info available what files should be used, e.g. by using the PUBLIC_HEADER property of a library target. Anyone experienced with the usage of that or similar CMake features are invited to propose how to integrate this into the API of the proposed new macro.

Currently there is no standard installation directory for QCH files. So documentation viewers supporting QCH files need to be manually pointed to any new installed QCH file (exception are the QCH files for Qt, whose location can be queried by qmake -query QT_INSTALL_DOCS).
Instead it would be nice those viewers pick up such files automatically. How could that be achieved?

It would be also good if this or some similar API dox metadata system could be upstreamed into CMake itself, so there is a reliable standard for this, helping everyone with the creation of cross-project integrated API manuals. At least the metadata variable naming patterns would need to get wide recognition to get really useful. Anyone interested in pushing this upstream?

For proper documentation of public and non-public API perhaps the doxygen development could also pick up this old thread about missing proper support for marking classes & methods as internal. Or is there meanwhile a good solution for this?

On the product side, what manual formats beside QCH should be supported?
On the tool side, what other API documentation tools should be supported (perhaps something clang-based)?

Comments welcome here or on the phabricator page.

WIP: Plasma World Map Wallpaper & World Clock Applet, powered by Marble

The core of Marble, the virtual globe and world atlas, is a library, intended for reuse. Next to exposing its native C++ interface (see API dox of development version of the Marble library), there is also the option to use it in a QtQuick variant.

The Marble code repository itself holds a few application and tools based on the library. Additionally it also has extensions & plugins for other applications & workspaces, like the KIO thumbnailer plugins for previews of KML, KMZ, GPX & GEOJSON files in KIO-using file manager or file dialogs, a Plasma Runner plugin for looking up geo coordinates or a world clock Plasma applet.

Moving to Plasma5, Marble KRunner plugin arrives first

Having extensions for external applications & workspaces in your repo requires yourself to keep up-to-date with those externals. When Plasma5 appeared with the big switch to QML-only plugin architecture, that left the Marble-based plugins a little behind for now.

Good news is that the KRunner/Plasma Runner plugin was not affected by that, and with “KDE Applications 16.08” and the new Marble release will now also be out in a version working in all locales this month.

Reviving the Marble world clock plasmoid from 2009

The Marble world clock plasmoid, introduced already in 2009 (see bottom of page), though needs a full rewrite, from C++ to QML, which is a bigger undertaking. It also needs a port from KTimeZone & KTimeZoneWidget to QTimeZone, and in that course a cleanup of the mapping between time zones to geocoordinates.

Good news here is that some initial work has been done now and is up for review. A lot of the features are still missing, and there are some glitches (e.g. vertical offset, license info displayed). But it is a start, and the current state might be even fancy enough for some users already🙂 So it should see a first release with “KDE Applications 16.12”.

It is a small mutation from a Plasma Applet to a Plasma Wallpaper

With the applet initially working, the idea came to have the map also shown fullscreen as wallpaper. While I could not find documentation how to do a wallpaper plugin, the existing code and examples though gave enough hints, and one hour later there was the first incarnation of a Marble-powered Plasma Wallpaper plugin.

See here a screenshot with both the World Map Wallpaper and the World Clock Applet in their current state:
Plasma World Map wallpaper & world clock applet, powered by Marble

Tell what kind of world maps you want to see as wallpaper

Because the World Map wallpaper can make full use of all Marble plugins, a lot of different kind of setups are possible:

  • Plain political map view
  • Globe view with stars in the background
  • Map view of just a continent/country
  • Satellite-photos-based view with live cloud rendering

There could be also plugin variants for the Moon, Venus or Mars.

So start Marble, play around with the settings to learn what is possible. Then tell in the comments what other kind of maps (or globe views) you would like to use for a wallpaper, so we could consider them as predefined settings at least.
As with the rewrite of the World Clock applet, a first release of the World Map wallpaper(s?) should happen with “KDE Applications 16.12”.

Create your own Plasma Wallpaper quickly by a new template

BTW, if you have other fancy ideas for Plasma “Wallpapers”, as by-product I did a template for Plasma Wallpaper plugins. It is currently up for review, so hopefully will be part of future KF5 Plasma frameworks releases. With that template, your first own wallpaper plugin is just a few clicks away e.g. in KAppTemplate or KDevelop.
Update: Until it arrives in releases, for now you can download the template file plasma-wallpaper.tar.bz2 from here and manually install it by moving that file into the folder /usr/share/kdevappwizard/templates/.
When playing around with the wallpaper settings, be aware there might be at least the issue of the wallpaper config page not reading current/default settings when switching the wallpaper type. As workaround, after switching the wallpaper type, one first has to close and reopen the dialog to be able to adjust the settings of the new wallpaper type.

Randa Meetings 2016 Part II: Marble

User Interface Needs Work For Facing More Users

The Randa Meetings 2016 were centered on bringing KDE technology on every device.

With regard to applications created by the KDE community, this means two things:

  1. Getting the source code to build for the operating systems on the target device
  2. Adapting the application UI to the specifics of the target device

The first point is by good parts covered by using the cross-platform library Qt and the cross-platform build system tool CMake, which eases enabling the build on many mainstream and also non-mainstream operating system. And the add-ons to these developed in the KDE community as shared elements, the Qt5 extensions KDE Frameworks (KF5) and the Extra CMake Modules (ECM), see lots of good work to cover more platforms as well.

The second point though is the larger hurdle here, when it comes to non-workstation-like devices, like tablets, smartphones, smartclocks or also TVs, fridges, in-vehicle entertainment screens. There especially the input options are quite different, which is also reflected in the UX of those devices in general.

Most existing applications created in the KDE community though have a UI done and optimized for the classic workstation-like setting. A setting, where input is done with a keyboard and a mouse. Where the main application window follows the concept of a menubar with submenus at the top, one or more toolbars below the menubar, a statusbar at the bottom, and the content in the middle.
And the source code architecture of most applications is often strongly coupled to this one UI, here and there even using the UI classes to store data belonging to the content that is viewed or edited, so no clean separation between UI layer and content model.

So adapting the application UI to the specifics of the target device most often means a big refactoring of the source code or perhaps even a complete rewrite. Something which is not done in a few weeks, but needs a lot of work over quite some time. Something for which people need motivation e.g. by owning and using such target devices and thus having the desire to also use those applications on them.

This showed to be a big hurdle even with devices which came with Qt pre-installed as the native UI toolkit of the operating system. Like the Meego devices, the BlackBerry 10 devices or the Sailfish OS devices. Only a few applications from the KDE community ever made an appearance there.

New: Direct Shipping To Devices By Developers

Another, though smaller issue to tackle for application developers often is to have to become a packager and distributor as well. Because other than with Linux distributions or the BSD port system, where the application developer just releases a tarball with the current sources which then is picked up the distributions’ packagers or the port system maintainers to make the software available in the package system of the respective operating systems, on other (usually proprietary) operating systems the end-user is to get their software from the application developers themselves directly or indirectly in a ready-to-install format, either by download from their homepages or via the so-called “app stores”. Which means more work to do at least.

Marble, a globe app going globally

One of the applications created in the KDE community whose developers managed to get it ported to more than the classic desktop computer, with regard to both operating system and UI, is the virtual globe and world atlas Marble. The reasons for that possibly can be seen in the capable and hard-working developers, but also in these:

  • Structuring the application in a modular way (so ports could be composed specifically for the target device use case, also some module being unported might not block the others being ported)
  • Avoiding dependencies on external libraries & modules if possible (so less things that also need to be available for the target platform)
  • Keeping dependencies on external libraries & modules optional (so if missing they do not block the rest of the application to be made working)

Myself I am more a happy user of the Desktop app Marble, not a core-contributor. So far I only did work on integration of Marble with the Plasma workspace, by writing some KRunner plugins to start Marble for geo coordinates or thumbnailer plugins for previews of KML, GPX & Co. in the file manager or file dialog.

Marble Maps on SailfishOS

Now, currently the Marble developers are working especially on Marble Maps, as a dedicated version of Marble for navigation purposes, with a first release planned for autumn this year. The main target platform right now is Android. As owner of a Jolla phone, which uses SailfishOS and thus Qt as part of the platform, last autumn I gave it a try to write a SailfishOS native variant of Marble Maps, to some quick first success. The uncertain future of Jolla then and thus of SailfishOS though kept me from investing more time into it. With Jolla these days still not sub-surface luckily and still sailing, I picked up those efforts now again, with the goal to merge my Maps variant soon into the Marble main code repository and have it part of the upcoming release, with feature-parity where possible.

With Qt being part of the operating system (and even more, slightly tuned there), there is some challenge though: SailfishOS is based on Qt 5.2 (with some backports from Qt 5.3), whereas either the typical unixoid target platforms themselves are on a more recent Qt or, with self-contained packages for Android, Windows & Co., one for good reasons picks the latest released Qt to ship along. Luckily so far the Marble core code could be kept working for all Qt versions. And there is some hope for more ease in the future, as work on updating SailfishOS to Qt 5.6 has been spotted.

So during this Randa Meetings I joined the Marble developers in their room for many days. For talking about roadmaps, features and plans with Marble Maps, when they had their heads not deep in the code. And learning more about the technologies behind navigation in Marble Maps, e.g. about how the turn-to-turn navigation is done.

More, I got a few hours long introduction into Marble’s elaborated rendering architecture by Torsten. Given my interest in static rendering of maps/globes for printing purposes or video frames generation, after nagging for that all the years I possibly have to scratch my itch here myself finally🙂 Though that knowledge will also be helpful when perhaps starting a plugin for OpenTripPlanner, another thing I would like to use on my device with Marble Maps.

What excites me most about Marble currently is the on-going work on vector tile rendering, which will allow many new or improved features. See here for a sneak-peak some current WIP state when it comes navigating yourself around Trafalgar Square in London with Marble Maps for SailfishOS, partially rendered in-device from vectors already (you can spot the pixel-based tiles still used as base layer) :
Trafalgar Square in London rendered from Vector tiles in Marble Maps on SailfishOS

Marble, it is a wonderful toy not only for the kid in the developer🙂 Get involved, either by making use of the Marble lib and plugins in your own app (and follow e.g. Subsurface and Digikam) or contribute directly to Marble’s own apps, lib and plugins.
To experience the wide set of possibilities with Marble, install the Marble desktop app (which IMHO should be renamed Marble Gallery🙂 ) and walk through all the features, e.g. different sky objects as central globe, satellite display, time-lapses through sun or moon ellipses, interactive routing, or map editing. All waiting for you to compile your own special use-case app from them. Like happened with the Behaim globe app for Android.

Support us

The Randa Meetings and other sprints bring our software forward, and also to more people and more platforms. Please check out the fundraiser for the Randa Meetings, and consider to do your little contribution to get things going:

Randa Meetings 2016 Part I: Okteta

Last Sunday the full week of Randa Meetings 2016 had passed, and it was time to find a route home, e.g. using KDE’s currently developed Marble Maps (here in the SailfishOS variant):
Planning the route home from Randa with Marble Maps on SailfishOS

Home, that would be for a place somewhere on our planet, like Africa, Asia, Europe, North America, or South America, where the 40-50 people who had got together that week had come from. They came into the Swiss Alps, traveling through space (not sure if also time, but at least quite many for quite some time) to collaborate in a valley deep between mountains covered by glaciers. To collaborate on bringing more of the set of applications developed in the KDE community also to other, even non-libre operating systems.

Getting Dirty With Other Operating Systems

The KDE community these days creates a large set of applications, and many of these are not bound to the one workspace for unixoid operating systems created by KDE, which is Plasma. No, these applications also run more or less fine in other (unixoid) workspaces, thanks to all the shared specifications of the freedesktop.org movement and other shared stacks/technologies (like D-Bus, X11 & soon Wayland).

Unixoid operating systems, that does not only mean Linux, but also all the BSD derivats. Sadly the *BSD subcommunity in KDE had become quite inactive the last years. So it was good to see that with Adriaan some *BSD veteran reactivated himself and is now working hard to get current versions of KDE applications and also Plasma to first class positions in the official software supply system. Next to that he also showed his real-world server acting abilities, as kitchen demon for Saturday, to everybody’s pleasure.

While now the big majority of KDE developers develop their applications with FLOSS unixoid workspaces/operating systems in mind, they often also find themselves and their target groups being bound to devices controlled by non-libre operating systems, due to some key apps or infrastructure in work or other parts of life only available with those. Operating systems like Microsoft’s Windows or Apple’s OSX/macOS. Still, people needing to use these devices can liberate themselves a little by at least using FLOSS applications on them. Like Firefox as their Web browser, in place of the non-libre Internet Explorer or the non-libre Safari.

As almost all current KDE applications are based on the cross-platform library Qt and the cross-platform build system tool CMake, which include support for the platforms Windows, OSX, the extra development work needed to get KDE applications running also on those platforms should be relatively small. And indeed since many years some people have been doing that needed extra work, with more or less success (see wiki pages on Windows and Mac). And since using Qt5 work has started to also extend this for Android.

Still there are many things that need more work to create serious products on release day for the average enduser. From documentation for application developers what to care for, integration into the KDE CI with platform specific builds, organized pre-release testing of packages, to providing the software products to end-users via proper distribution channels. All these are areas which had been talked about and worked on during this year’s Randa Meetings (see also Kevin’s post with a KDE on Windows status update).

Okteta on Windows: built out of the source box

Already when Okteta was started many years ago, using CMake, Qt4 & kdelibs4 made it possible to have builds of Okteta for Windows (XP) and OSX done by mainly pointing some generic build and packaging scripts at its source code, as shown with Okteta 0.1 in 2008.

Today, with CMake, ECM, Qt5, KF5, it is still the same. When asking for a Windows build, just to see what the state is, it again was just a matter of pointing the generic scripts at the sources, and there was Okteta 0.19 running on Windows 10 (thanks Kevin for builds and screenshot):
Okteta 0.19 on Windows 10

One nice sideeffect of cross-testing on different platforms is that forgotten issues get into the spotlight again, like visible in the screenshot:

  • Accelerator syntax shown verbatim in docker widget title bars: KAcceleratorManager from the KWidgetAddons module needs to get an idea how to properly handle QDockWidgets
  • Bad initial size of window and set of initially shown dockers: the current hack to work-around improper API control in Qt needs adaption to Qt5 and perhaps also other platforms

Running the unit tests showed a few issues, half of them due to not using platform-neutral access to the filesystem (quickly fixed), the other half because of tricks with XDG env vars which need some platform-neutral solution (still to be solved). As a result Okteta’s source code will be more clean, which is also a win for the version for the libre operating systems.

So if somebody would want Okteta on Windows (I had a few people asking for that by the years) and/or OSX, you are welcome to help out in the packaging and testing area. I myself do not have any of those operating systems and also would not invest into that, given my own priorities. Still happy to work together.

Support us

The Randa Meetings and other sprints bring our software forward, and also to more people and more platforms. Please check out the fundraiser for the Randa Meetings, and consider to do your little contribution to get things going:

Notes on other activites of mine at Randa Meetings 2016 in a following post.

Plasma Weather widget: code template available to add your favorite weather data provider

Quick recap

Plasma 5.6 has finally seen the return of the Plasma Add-ons Weather widget, which had been missing the port from Plasma4.
Next, the Weather widget does not talk to any weather data providers directly, instead it talks to a weather dataengine (currently part of the Plasma Workspace module), to query for any weather stations matching the entered location search string when configuring the widget and to subscribe to the data feed for a given weather station from a given weather data provider.
That weather dataengine itself again also does not talk directly to any weather data providers. Instead it relies on an extendable set of sub-dataengines, one per weather data provider. So-called “ion”s.

Currently there are only 3 ions part of Plasma: wetter.com (private company running wetter.com), envcan (Environment Canada, by Government of Canada), noaa (USA’s National Oceanic and Atmospheric Administration). That is not enough, right.

Get quickly started with a working template

Time to add an ion for your favourite weather data provider. An ion can be independently developed by everyone, no need to do this in the plasma-workspace module (though it would be great if your ion ends up there once it is working, so it can be shared more easily).

To allow you a quick start with your own ion development, for the next version of the Plasma, 5.7, a code template (in the kapptemplate format, as supported by KAppTemplate or KDevelop) has been added. So when installing the development packages for Plasma Workspace (e.g. “plasma5-workspace-devel”), there should be also a “Plasma Weather Ion Dataengine” code template available, in the “KDE/Plasma Dataengine” folder.

Until Plasma 5.7 is released, download a snapshot of the ion code template. This snapshot also works with Plasma 5.6. You will need to have the development packages for Plasma Workspace installed, so the “plasma/weather/ion.h” header is available.
To use the template with KDevelop, select in the menu “Project”/”New From Template…” and click the button “Load Template From File” to add the template to the list. Then select the template and follow the dialog.
To use it with KAppTemplate, you need to manually install the file in “/usr/share/kdevappwizard/templates/” and (for KAppTemplate before Applications 16.04) also add a dummy PNG file to “/usr/share/kdevappwizard/template_previews/“, with the same base name as the template, “ion-dataengine.png“.
If the template does not work for you (please tell in the comments if), download this sample ion zip file, made from the template and the name “trueweather”, and start from that.

Follow the “README” in the sources and learn how to build, install and test your ion dataengine, e.g. how to query it via the Weather dataengine with the Plasma Engine Explorer:
trueweather ion dataengine in Plasma Engine Explorer

Learn more details about the Plasma Weather dataengine system in the last blog post.

Do not hesitate to ask for help on the #plasma irc channel and the plasma-devel mailinglist.

Plasma Weather widget: add your favorite weather data provider

So Plasma 5.6 has seen the revival of the Weather widget that is part of KDE Plasma Add-ons module (revival as in: ported from Plasma4 API to Plasma5 API).
(And even got the interesting honour to be mentioned as the selling item in the headline of a Plasma 5.6 release news article by a bigger German IT news site, oh well🙂 )

This revival was concentrating for 5.6 to restore the widget in its old state, without any bugs. But that’s not where things are to end now…

And you, yes, you, are invited and even called in to help with improving the widget, and especially for connecting to more weather data providers, incl. your favourite one.

For a start, let’s list the current plans for the Weather widget when looking at the next Plasma release, 5.7:

  • Overhaul of look (IN PROGRESS)
  • Breeze-style weather state icons (IN PROGRESS)
  • also temperature displayed in compact widget variant, like on panel (TODO)
  • support for more weather data providers (YOUR? TODO)
  • privacy control which weather data providers are queried on finding weather stations (YOUR? TODO)

Overhaul of widget look

The KDE meta sprint at CERN (of groups WikiToLearn, Plasma, VDG, techbase wiki cleanup) at begin of this March, where I mainly went for techbase wiki cleanup work, of course was a great occasion to also work face-to-face with Plasma developers and VDG members on issues around the Weather widget, and so we did. Marco helped me to learn more about Plasma5 technologies which resulted in some small bugs fixed in the widget still in time for Plasma 5.6 release.
Ken presented me the drafts for the look & design and the config of the weather widget that he had prepared before the sprint, which we then discussed. Some proposals for the config could already be applied in time for Plasma 5.6 release (those without need for changed UI texts, to not mess the work of the translators). The rest of it, especially the new look & layout of the widget, is soon going to be transferred from our minds, the notes and the photos taken from the sketches on the whiteboard at CERN into real code.

Breeze-style weather state icons

Ken also did a Breeze style set of the currently used weather state icons. While touching the icons, a few icon naming issues are going to be handled as well (e.g. resolving inconsistent naming pattern or deviation from the weather state icon names actually defined in the XDG icon naming spec (see Table 12. Standard Status Icons). Should soon be done as well.

Temperature display in compact widget variant

One thing that has been missing also from the old version of the Weather widget is the display of the current temperature in the compact widget variant, where now only the current weather condition is shown (e.g. when on the panel). While some weather data providers (like wetter.com) do not provide that detail, so there is nothing to display, others do, and often it is pretty nice to know (clear sky can happen with temperatures of any kind, so it’s good information about if and then what kind of jacket to put on before stepping outside). First code is drafted.

Now, finally the things were you can improve things for yourself and others:

Supporting more weather data providers

The Weather widget does not talk to any weather data providers directly. Instead it talks to a weather dataengine (currently part of the Plasma Workspace module), to query for any weather stations matching the entered location search string when configuring the widget and to subscribe to the data feed for a given weather station from a given weather data provider.
That weather dataengine itself again also does not talk directly to any weather data providers. Instead it relies on an extendable set of sub-dataengines, one per weather data provider.

The idea here is to have by the one weather dataengine an abstraction layer (he, after all this is KDE software😉 ) which maps all weather data services into a generic one, so any UI, like the Weather widget, only needs to talk one language to whoever delivers the data.
Which works, more or less. Because of course there are quite some weather data specifications out there, what else did we expect😛 And possibly the spec interpretations also vary as usual.

((You might think: “Well, screw that over, there is only one user of the weather dataengine, so integrate that directly into the Weather widget!”
While that might be true right now, it does not need to stay this way. There are ideas like showing the weather forecast also with the days in Plasma calendar widgets. Having a KRunner plugin to quickly request weather in some place. Or having a wallpaper updater reflecting the current weather by showing matching images, yes, not everyone has the joy to work close to a window, enjoy if you do. And also alternative weather widgets with another UI, remember also the WeatherStation widget in kdeplasma-addons (still waiting for someone to port it), which focussed on details of the current weather state. These kind of usages might prefer a normalized model of weather data as well, instead of needing custom code for each and any provider again. Actually long-term such a model would be interesting outside of Plasma spheres, e.g. for any calendaring apps, for a plugin for Marble-based apps showing weather states over a map or whatever other fancy uses there can be. Feel free to share ideas in the comments, to improve motivation for creating such a Plasma-independent lib!))

While I only took over kind of maintainership in the last weeks, so did not design the current system myself, I still think it’s a good approach, having in mind reusable UIs and relative independence from any given weather data providers. So for now I do not plan to dump that, simply lacking a more promising alternative.

So, given you are still reading this and thus showing me and yourself your interest🙂 let’s have a closer look:

The sub-dataengines for the different weather data providers have been named “ion”s. On the code level they are subclasses of a class IonInterface, which itself is a subclass of Plasma::DataEngine.
See here for the header defining IonInterface: https://quickgit.kde.org/?p=plasma-workspace.git&a=blob&f=dataengines%2Fweather%2Fions%2Fion.h

This header and the respective lib libweather_ion are public and should be installed with the devel packages of Plasma Workspace. Which means you should be able to develop your custom ion as 3rd-party developer without the need to checkout the plasma-workspace repository and develop it in there.

Plasma 5.6 itself installs three such ions:

  • wetter.com: adapter to service of the private company running wetter.com
  • envcan: adapter to service of Environment Canada, by Government of Canada
  • noaa: adapter to service of USA’s National Oceanic and Atmospheric Administration

Find their sources here: https://quickgit.kde.org/?p=plasma-workspace.git&a=tree&f=dataengines%2Fweather%2Fions

In that source folder you will also spot a bbcukmet ion, for the BBC weather service. While being ported to build and install at least with Plasma5, the service API of BBC as used by the code seems to have been either changed or even removed. So that ion is currently disabled from the build (uncomment the #add_subdirectory(bbcukmet) to readd it to the build). Investigations welcome.
Update: There are patches for the Plasma4 version of the bbcukmet ion to make it work again with current BBC service API. See below in the comments for detailed info. Should be easy work and a chance to contribute also for beginners.
Update2: Those patches have been used to restore the bbcukmet ion, it will be part of Plasma 5.6.2 and later. Still would be good to have someone look closer at it, there is room for improvements.

Another old ion which though already got removed during the revival was a more fun one: there used to be a Debian “weather” service (random related blog post), which reported the status of Debian-based distros by number of working packages as weather reports, and this ion connected to them. But the service died some years ago, so the ion was just dead code now (find the unported code in versions of “Plasma 5.5” and before)

Talking about funny weather reports: why not write an ion for the CI system Jenkins, e.g. with build.kde.org, which while perhaps not being able to give forecasts at least reports the current build state, with builds mapped to stations. After all the short report for a build uses the weather metaphor, see https://build.kde.org/🙂

For more serious weather data provider ions again, as you surely know or can guess, there are many more public & private weather data providers than those 3 currently supported. And they not only may have a better coverage of your local area, but might also provide more data or more suited ones.

Please also see the proposal for using “weather data from the open data initiatives” in a comment on the first blog post on the Weather widget.

It would be great to have a larger selection of weather data providers supported in Plasma 5.7. So while having your ion as 3rd-party plugin somewhere else is fine, consider maintaining it inside one of the Plasma repositories, either with the existing ions in the repo plasma-workspace or as part of addons in the repo kdeplasma-addons. This should ensure more users and also more co-developers.

Do a good check of the licenses for using the data services of the weather data providers. Especially public ones should be fine given their purpose, but if in doubt after reading the details, ask the providers.

Privacy control

One issue in the old and current Weather widget code is that when searching for a weather station suiting the users desire, as expressed by the location search string, all currently installed ions are queried. Which of course is a problem from a privacy point of view. And will be worse the more ions there will be.

So there needs to be a way to limit the scope of ions that would be queried. Given that dataengines by design should be used by all kind of dataengine consumers, a Weather widget-only solution might be only a short jump here. There are very possibly other Plasma modules talking to remote services as well, like geolocation services. And ideally one is able to control system-wide (so even beyond Plasma scope) which remote services are allowed and which not.

Until such a global solution exists a Weather widget-only solution is better than nothing surely. Still, it needs to be designed UI-wise, so a job to be picked up by someone🙂

Getting started with your own ion

So while I am currently impeded these days from hacking, among other things by waiting for new development-capable hardware being delivered (disadvantage of small towns: need to use an online shop for special hardware desires, and there latency unit is days, not minutes, especially bad when wrong stuff is delivered and then also holidays get into the game. Looking on the bright side of life, my old hardware only broke right after the CERN sprint, not before🙂 )…
… Do not hesitate to look into things already. I would have liked to provide a KAppTemplate-package with an ion sample in this post already, so you could experiment right away. But perhaps you are experienced enough to get a new ion working by looking at the existing ones. If not, in a few weeks hopefully there should be a template, so watch out.
Update: There is now such a template, see this blog post.

And do not hesitate to ask for support on #plasma on irc or on the Plasma mailinglist, I received lots of help there with the Weather widget porting, so you should when trying to write an ion🙂