Synchronisation with Kasten, just to filesystems now

Sometimes it seems with faster and faster JavaScript-JIT-compilers and Gears/HTML5 the current software stacks, written in other languages, like e.g. Qt/KDE are going to be legacy. Google Docs & Co. show what is possible, with the browser as the generic adapter to the platform.
But the current software stacks are fighting back. Things like the openSUSE Build Service make it possible to have your code compiled and ready to be delivered for several platforms, run natively by the hardware you actually bought to do so, instead of having some resources abused for virtual layers. It just needs code which does not (over-)use platform-specific constructs. The expansion of Qt/KDE to more and more architectures is a fine example here what is possible.
Curious what will be next, perhaps Javascript programs using alternatively binary blocks of code compiled for popular hardware, given the OS has built-in access control? Things are merging.

With this in mind I congratulate the people from AbiWord on their new version 2.8, which not only improves the support for the open spec’ed on-disk format OpenDocument (.odt), but especially improves on the collaboration features which integrate the Web 2.0. That looks really great. My only nitpick is that this seems to be a non-FLOSS webservice. I just hope they find a good business model so that the software for the server can be shared, too. Others, like Inkscape or hopefully soon KOffice, might be interested in such a server system, too.

I also congratulate the people from Γ‰toilΓ©, who just have started to send objects over XMPP. This all sounds so promising.

Realtime collaboration and such data model synchronisation is something I hope Kasten, my WIP framework for composing programs, one day supports out of the box, too. From the very beginning this has been considered in the design, even if it is not yet too visible. And it still will take some time to get there.

There is other stuff to do first, like making Okteta work with very large files. This isn’t done yet, I am still working on it, but have hopes this feature will make it into the next release. Working with very large files means only loading the parts which are currently interesting (e.g. solved by memmapping the file). Currently Kasten, the framework Okteta is built on, only has support for the concept of loading document objects completely from the remote storage (like the filesystem on disk) to the working memory, done by subclasses of AbstractModelSynchronizer. There is a specialized subclass AbstractModelFileSystemSynchronizer, which takes care for the common stuff on dealing with the filesystem, including loading/storing to remote filesystems with KIO::NetAccess. So the/your document-specific subclass has just to implement the de- and encoding of the data from a QFile to the working data model.

Now, AbstractModelFileSystemSynchronizer was not really finished, so before adding a new construction site in Kasten for the partial-loading of files I want to finish the old one for the complete loading. E.g. there was no support for a property State of the remote storage. Thanks to KDirWatcher there is now some easily done initial one. Investigations are still needed how to make sure there is no race-condition where another process might break into writing to the file, reading the file’s timestamp of the own modification and starting to watch for changes again.
Also, KDirWatcher does not work for Kio-slaves. So files loaded from them are just marked to be in Unknown state. Besides, if the network connection is down, they are set to Unreachable. Solid::Networking serves this with just a few lines again πŸ™‚

See in the screenshot for the two state symbols now in the Documents tool: the left icon shows the working data state (saved changes, unsaved changes), the right one the remote storage state. The yellow flag means Unknown, in case you wonder, and turns into the Network-disconnected icon if you disconnect from the network, like pulling the plug (looks impressive if you have several files loaded e.g. by sftp:// πŸ˜‰ ). Well, seems I need to apply for some more icons from our icon heros.
Okteta Local and Remote Sync Status

Other changes in this area:

  • The action Revert is renamed to Reload. Revert will not do what the name says if the storage copy was modified since the loading of the document. Revert would rather mean undoing all changes, so that action semantic might rather appear in the Edit menu, for some quicker navigation through the change history.
  • The actions Reload and Save are only available if they are useful and usable. E.g. they are disabled if there are no known changes in both the working copy of the document data and the storage copy or the connection to the remote storage is lost.
  • Also new documents which are generated from random data or user input (like parameters) are set to modified, so closing them without adding any further content will still trigger the question if you really want to discard this perhaps unique data.

Okteta: Fighting the crisis with productivity from generators

(And if it is just random data)

There has been some productivity-related productivity in Kasten and Okteta development the last days, as can be seen by commit logs like “added: factories to create generators” πŸ˜‰ :

Kasten, my WIP framework for composing programs, gained some AbstractModelDataGenerator class to serve as base class for data generating stuff. It also got a general Insert controller which plugs into the menu, offers all available data generators and has the chosen one generate its data (if parameterized after showing a config dialog) and then inserts it into the current document. And with the AbstractModelDataGenerator around, I extended the AbstractDocumentFactory to also create new documents from QMimeData and reworked the general New controller: it now not only offers to create an empty document, but also to create one prefilled with data from one of the data generators. Or by the data from the clipboard, which also serves QMimeData. And while I was at it, the shell now no longer rejects D’n’D data other than file urls to non-document areas (like next to tabs or the empty space on start) but passes it to the document factory to see if it can construct a new document from it, all by just a few lines of codes. Kasten may look bloated to some (including myself sometimes) but it has it’s value here and there πŸ™‚

To see if it is working Okteta, the KDE 4 hex editor, which is built on Kasten, has got some generators:
For one Okteta since the first release had some simple Insert pattern controller, which now got ported to a plain data generator plugin (+ config editor). Then two weeks ago I found myself for the x-th time in life in the need for data in the whole range of all bytes, from 0 to 255. So I had done a clone of the pattern controller and already added as a Insert sequence controller (which started the idea for these general data generators). Got ported to a plain generator plugin, too.

Another generator perhaps useful might be a random data generator. So now there is one. It is based on srand(time(0)) and rand(), which may be of different quality depending of the platform’s libc. If you instead want a certain algorithm, please contribute and send me some code for it filling a QByteArray, licensed like the rest of the file, I am happy to add some alternative RNG to select from πŸ™‚

Lots of TODOs and ideas coming with this, but works already. If you like screenshots, have a look:
Data generators for Okteta

And there has been even more productivity with regard to Okteta: First external contribution is coming with a new tool πŸ™‚ More on this soon…

Tutorial: Add a Checksum/Hashsum Algorithm to Okteta

Here comes the second in a row of little tutorials showing how you can yourself add the features you need to Okteta, the KDE 4 hex editor (the first one was about Adding a Binary Filter).

Since KDE 4.3 (resp. Okteta 0.3) Okteta features a tool to calculate a checksum/hashsum from the bytes in the current selection. It comes with some built-in checksum algorithms (Modular sum (8/16/32/64 bit), Adler-32, CRC-32), but for hashsums relies on the QCA lib (Qt Cryptographic Architecture) (can be SHA-0/1/224/256/384/512, MD2/4/5, RIPEMD-160, Whirlpool, depending on the installed QCA plugins).

Ideally you would write a new hashsum algorithm as a plugin for the QCA library, so it is available to more programs than just Okteta. But if you want to add another checksum to Okteta, here is your tutorial:

Setup of the Development Environment

See for the instructions in the first tutorial about Binary Filters.

Adding a Checksum Algorithm

In this tutorial you write a checksum algorithm which will calculate the parity of all bytes using only a given bit position. It will look like this:
Bit-Parity checksum
Continue reading

UPnP devices coming to the network:/ kioslave

Since the Coherence/KDE sprint (Spain, 26 Β°C, sun), where work was done for some UPnP bindings for KDE, a week has passed (Germany, 4 Β°C, rain/hail). Besides the weather some other things have changed that week, too, but for the good πŸ™‚

While on Day 1 of the sprint I already managed to have first UPnP devices show up in the network:/ kioslave, it was done with hacky prototype code, to have some things working to start from. Day 2 didn’t change much in that regard.

The days since then Frank Scholz, the main developer of Coherence, has been very supportive and has continued adding needed features to the D-Bus API, and I have hammered a little more onto the code for the UPnP backend to the network:/ kioslave. To much success! UPnP devices are now coming and going in the network:/ listing, matching the real devices’ behaviour, with proper name and type.

All that is missing to my full pleasure is the usage of each UPnP device’s custom icon (So if you know how to set custom icons (i.e. not by name, but delivered from remote, cmp. favicons) for UDSEntries in a kioslave, please tell me!). And forwarding to the presentation url of the UPnP device, if available, but that looks like it needs some discussions with the KIO developers, because of the “abuse” of KIO for the network:/ kioslave.

No screenshot today, if you are interested please try it yourself:
The UPnP backend for the network:/ kioslave is based on the utility libkupnp which cares for the wrapping of the D-Bus connection to Coherence. So fetch both
libkupnp at
http://websvn.kde.org/trunk/playground/network/kupnp/
and the UPnP backend at
http://websvn.kde.org/branches/work/network-kioslave/adding-upnp-backend/
and compile as usual.

You also need a current version (>=#1440) from trunk of Coherence:

svn co https://coherence.beebits.net/svn/trunk/Coherence
sudo python ./setup.py install

(No idea how to install Coherence to a user-only prefix, hints welcome!)

If you are interested in Bart's work on UPnP support for Amarok, do not miss his last blog entry (which seems to have missed planet KDE).

The coming weeks I will see if libkupnp can be extended to be useful for other code interacting with UPnP devices (at least from a UPnP control point's view). For a start I will try a InternetGatewayDevice proxy class, which could be used by programs wanting NAT traversal (Konversation and KTorrent already have some custom implementation, but this could help others). Next would be a MediaServer proxy class, perhaps that could simplify Bart's upnp:/ kioslave.

Coherence/KDE sprint Day 2

Exactly one week already has passed since I arrived in Barcelona for the Coherence/KDE sprint, and almost four days since I left again, so before it is out-of-date here a short summary of Day 2 (see also Day 1 for some introduction):

Everyone continued hacking on his stuff, e.g. Bart on his kioslave for UPnP MediaCollections, to be used especially by Amarok, and me on the UPnP backend for the network:/ kioslave. Which also involved discussions with Frank about shaping the D-Bus API of Coherence to our needs a little. And how to merge best the UPnP device/service model and objects into the network classes model of the base library of the network:/ kioslave, along with the ones from DNSSD and Co. This merging needs more thoughts definitely.

Lunch was cared for by our host Edward Hervey from Collabora Multimedia leading us to another non-tourist restaurant with good food and local atmosphere.

Other stuff that day was done was demoing the things now working, like the Coherence plugin for the Spykee robot delivering pictures from its built-in webcam via UPnP. Or things that have been working for a while, like each one playing his favourite songs via UPnP on Frank’s computer, in a kind of battle style. Including controlling the volume from remote.

Zaheer, Konqui and me looking what Frank does that makes him wear the K-scarf this way
Zaheer, Konqui and me looking what Frank does that makes him wear the K-scarf this way

In some way Coherence is to UPnP what Avahi is to DNS-SD, a demon wrapping the specific system and making it accessible via D-Bus. Just that Coherence does a lot more, as UPnP is not only about discovery of services, but also of types of devices and the interfaces/services they implement and how to access them.

As the developers of Coherence are using GNOME-centric desktops the current desktop integration is basically done for GNOME programs, like Totem, Rhythmbox, and Nautilus.

Let’s hope KDE-Libs based programs can catch up to these πŸ™‚ This was the very purpose of this sprint, and it gave a good foundation. It just left us with quite some work to do. Especially Bart might appreciate help with his kioslave, as he is short on time. So if you want Amarok to as well access other UPnP MediaCollections as well as export it’s own collection as such in the next time, get in contact with him.

I think I might be able to get the integration into the network:/ kioslave done in October (promises, what promises?). Surely there will be the need for feedback. So please stay tuned, more detailed info about approach and progress to follow…

Coherence/KDE sprint Day 1

Big splash for yesterday evening: “Breakthrough in the hotel!”

Barcelona, wonderful city. Crowded with a lot of people, enjoying the life style and atmosphere. This weekend there are six people more, but here for hard work, to do some collaboration on UPnP support in FLOSS software. So is there a better place to stay at then the Collabora Multimedia office here in Barcelona?

Together with Bart Cerneels, from the Amarok crowd and to most of us known as the great organizer of Akademy 2008, I am here to write code to connect to the UPnP world, by using Coherence. If you haven’t already heard of Coherence before, it is an well developed UPnP-enabling framework, written in Python and accessible via D-Bus by any other program, hiding the UPnP complexity both to the providers and the consumers of UPnP services.
So e.g. Amarok could easily offer access to its own media collections for any UPnP capable client on the network as well as access those of others, at the same time control any other player in the network (MediaRenderer in UPnP terms).
And it is also a comfortable way to add a UPnP backend to the network:/ kioslave, so the UPnP-only devices and their services are also listed there. I was hoping someone else was going to do this, but, well, scratch your itch…

Still it feels like boring work to do compared to what some others sitting at the table with us are doing: There is Philippe Normand who is working on tubing UPnP over the intertubes with the Mirabeau plugin to Coherence. And there is Zaheer Abbas Merali, who is wrapping standard UPnP services around little next-generation-mars^W indoor robots, so you can access the robot with any standard UPnP client (like the PS3, PSP or XBox). Then there is Sebastian PΓΆlsterl who works to build a UPnP supporting personal video recorder (PVR) based on his DVB-Daemon.
Last but not least with us are Frank Scholz, the founder and main developer of Coherence, and Benjamin Kampmann, official vice-president of marketing for Coherence.

I had a quick start with the code Adriaan de Groot had written at the first Coherence/KDE sprint in Paris this spring. Still I had nothing to show when we left the office in the evening. But when most of us went to spend much money to watch 22 men catch a single ball, just to kick it away again, Bart and I waited for them in the hotel, typing even more code and then, the breakthrough, at least for me:

And it comes with congrats to Ahmed Moustaf! His computer was the first to appear in the network:/ kioslave by being feeded from the Coherence-driven UPnP backend. Sitting in the hotel room I had no longer all the devices available like we have with us in the Collabora office, but another friendly, although unknown person had his computer broadcasting his UPnP services, so there was content. You won’t see from the documenting screenshot it is from the UPnP, as that is one of the targets of the network:/ kioslave, hiding all the unneeded technical details from the user. And his computer disappeared soon after when I wanted to screenshot some UPnP only listing, was it because we started to browse his media collection and he found out?* πŸ˜‰ So just believe it πŸ™‚
First device in network:/ kioslave delivered by the UPnP backend

*Because if there is one thing which seems to suck with UPnP it is the lack of any access control built into the system. Everybody can access everything. Well, hopefully soon KDE programs can, too πŸ™‚