Synchronisation with Kasten, just to filesystems now

Sometimes it seems with faster and faster JavaScript-JIT-compilers and Gears/HTML5 the current software stacks, written in other languages, like e.g. Qt/KDE are going to be legacy. Google Docs & Co. show what is possible, with the browser as the generic adapter to the platform.
But the current software stacks are fighting back. Things like the openSUSE Build Service make it possible to have your code compiled and ready to be delivered for several platforms, run natively by the hardware you actually bought to do so, instead of having some resources abused for virtual layers. It just needs code which does not (over-)use platform-specific constructs. The expansion of Qt/KDE to more and more architectures is a fine example here what is possible.
Curious what will be next, perhaps Javascript programs using alternatively binary blocks of code compiled for popular hardware, given the OS has built-in access control? Things are merging.

With this in mind I congratulate the people from AbiWord on their new version 2.8, which not only improves the support for the open spec’ed on-disk format OpenDocument (.odt), but especially improves on the collaboration features which integrate the Web 2.0. That looks really great. My only nitpick is that this seems to be a non-FLOSS webservice. I just hope they find a good business model so that the software for the server can be shared, too. Others, like Inkscape or hopefully soon KOffice, might be interested in such a server system, too.

I also congratulate the people from Étoilé, who just have started to send objects over XMPP. This all sounds so promising.

Realtime collaboration and such data model synchronisation is something I hope Kasten, my WIP framework for composing programs, one day supports out of the box, too. From the very beginning this has been considered in the design, even if it is not yet too visible. And it still will take some time to get there.

There is other stuff to do first, like making Okteta work with very large files. This isn’t done yet, I am still working on it, but have hopes this feature will make it into the next release. Working with very large files means only loading the parts which are currently interesting (e.g. solved by memmapping the file). Currently Kasten, the framework Okteta is built on, only has support for the concept of loading document objects completely from the remote storage (like the filesystem on disk) to the working memory, done by subclasses of AbstractModelSynchronizer. There is a specialized subclass AbstractModelFileSystemSynchronizer, which takes care for the common stuff on dealing with the filesystem, including loading/storing to remote filesystems with KIO::NetAccess. So the/your document-specific subclass has just to implement the de- and encoding of the data from a QFile to the working data model.

Now, AbstractModelFileSystemSynchronizer was not really finished, so before adding a new construction site in Kasten for the partial-loading of files I want to finish the old one for the complete loading. E.g. there was no support for a property State of the remote storage. Thanks to KDirWatcher there is now some easily done initial one. Investigations are still needed how to make sure there is no race-condition where another process might break into writing to the file, reading the file’s timestamp of the own modification and starting to watch for changes again.
Also, KDirWatcher does not work for Kio-slaves. So files loaded from them are just marked to be in Unknown state. Besides, if the network connection is down, they are set to Unreachable. Solid::Networking serves this with just a few lines again

See in the screenshot for the two state symbols now in the Documents tool: the left icon shows the working data state (saved changes, unsaved changes), the right one the remote storage state. The yellow flag means Unknown, in case you wonder, and turns into the Network-disconnected icon if you disconnect from the network, like pulling the plug (looks impressive if you have several files loaded e.g. by sftp:// ). Well, seems I need to apply for some more icons from our icon heros.

Other changes in this area:

• The action Revert is renamed to Reload. Revert will not do what the name says if the storage copy was modified since the loading of the document. Revert would rather mean undoing all changes, so that action semantic might rather appear in the Edit menu, for some quicker navigation through the change history.
• The actions Reload and Save are only available if they are useful and usable. E.g. they are disabled if there are no known changes in both the working copy of the document data and the storage copy or the connection to the remote storage is lost.
• Also new documents which are generated from random data or user input (like parameters) are set to modified, so closing them without adding any further content will still trigger the question if you really want to discard this perhaps unique data.

Okteta: Fighting the crisis with productivity from generators

(And if it is just random data)

There has been some productivity-related productivity in Kasten and Okteta development the last days, as can be seen by commit logs like “added: factories to create generators” :

Kasten, my WIP framework for composing programs, gained some AbstractModelDataGenerator class to serve as base class for data generating stuff. It also got a general Insert controller which plugs into the menu, offers all available data generators and has the chosen one generate its data (if parameterized after showing a config dialog) and then inserts it into the current document. And with the AbstractModelDataGenerator around, I extended the AbstractDocumentFactory to also create new documents from QMimeData and reworked the general New controller: it now not only offers to create an empty document, but also to create one prefilled with data from one of the data generators. Or by the data from the clipboard, which also serves QMimeData. And while I was at it, the shell now no longer rejects D’n'D data other than file urls to non-document areas (like next to tabs or the empty space on start) but passes it to the document factory to see if it can construct a new document from it, all by just a few lines of codes. Kasten may look bloated to some (including myself sometimes) but it has it’s value here and there

To see if it is working Okteta, the KDE 4 hex editor, which is built on Kasten, has got some generators:
For one Okteta since the first release had some simple Insert pattern controller, which now got ported to a plain data generator plugin (+ config editor). Then two weeks ago I found myself for the x-th time in life in the need for data in the whole range of all bytes, from 0 to 255. So I had done a clone of the pattern controller and already added as a Insert sequence controller (which started the idea for these general data generators). Got ported to a plain generator plugin, too.

Another generator perhaps useful might be a random data generator. So now there is one. It is based on srand(time(0)) and rand(), which may be of different quality depending of the platform’s libc. If you instead want a certain algorithm, please contribute and send me some code for it filling a QByteArray, licensed like the rest of the file, I am happy to add some alternative RNG to select from

Lots of TODOs and ideas coming with this, but works already. If you like screenshots, have a look:

And there has been even more productivity with regard to Okteta: First external contribution is coming with a new tool More on this soon…

Tutorial: Add a Checksum/Hashsum Algorithm to Okteta

Here comes the second in a row of little tutorials showing how you can yourself add the features you need to Okteta, the KDE 4 hex editor (the first one was about Adding a Binary Filter).

Since KDE 4.3 (resp. Okteta 0.3) Okteta features a tool to calculate a checksum/hashsum from the bytes in the current selection. It comes with some built-in checksum algorithms (Modular sum (8/16/32/64 bit), Adler-32, CRC-32), but for hashsums relies on the QCA lib (Qt Cryptographic Architecture) (can be SHA-0/1/224/256/384/512, MD2/4/5, RIPEMD-160, Whirlpool, depending on the installed QCA plugins).

Ideally you would write a new hashsum algorithm as a plugin for the QCA library, so it is available to more programs than just Okteta. But if you want to add another checksum to Okteta, here is your tutorial:

Setup of the Development Environment

See for the instructions in the first tutorial about Binary Filters.

Adding a Checksum Algorithm

In this tutorial you write a checksum algorithm which will calculate the parity of all bytes using only a given bit position. It will look like this:

UPnP devices coming to the network:/ kioslave

Since the Coherence/KDE sprint (Spain, 26 °C, sun), where work was done for some UPnP bindings for KDE, a week has passed (Germany, 4 °C, rain/hail). Besides the weather some other things have changed that week, too, but for the good

While on Day 1 of the sprint I already managed to have first UPnP devices show up in the network:/ kioslave, it was done with hacky prototype code, to have some things working to start from. Day 2 didn’t change much in that regard.

The days since then Frank Scholz, the main developer of Coherence, has been very supportive and has continued adding needed features to the D-Bus API, and I have hammered a little more onto the code for the UPnP backend to the network:/ kioslave. To much success! UPnP devices are now coming and going in the network:/ listing, matching the real devices’ behaviour, with proper name and type.

All that is missing to my full pleasure is the usage of each UPnP device’s custom icon (So if you know how to set custom icons (i.e. not by name, but delivered from remote, cmp. favicons) for UDSEntries in a kioslave, please tell me!). And forwarding to the presentation url of the UPnP device, if available, but that looks like it needs some discussions with the KIO developers, because of the “abuse” of KIO for the network:/ kioslave.

No screenshot today, if you are interested please try it yourself:
The UPnP backend for the network:/ kioslave is based on the utility libkupnp which cares for the wrapping of the D-Bus connection to Coherence. So fetch both
libkupnp at
http://websvn.kde.org/trunk/playground/network/kupnp/
and the UPnP backend at
http://websvn.kde.org/branches/work/network-kioslave/adding-upnp-backend/
and compile as usual.

You also need a current version (>=#1440) from trunk of Coherence:

svn co https://coherence.beebits.net/svn/trunk/Coherence
sudo python ./setup.py install

(No idea how to install Coherence to a user-only prefix, hints welcome!)

If you are interested in Bart's work on UPnP support for Amarok, do not miss his last blog entry (which seems to have missed planet KDE).

The coming weeks I will see if libkupnp can be extended to be useful for other code interacting with UPnP devices (at least from a UPnP control point's view). For a start I will try a InternetGatewayDevice proxy class, which could be used by programs wanting NAT traversal (Konversation and KTorrent already have some custom implementation, but this could help others). Next would be a MediaServer proxy class, perhaps that could simplify Bart's upnp:/ kioslave.

Coherence/KDE sprint Day 2

Exactly one week already has passed since I arrived in Barcelona for the Coherence/KDE sprint, and almost four days since I left again, so before it is out-of-date here a short summary of Day 2 (see also Day 1 for some introduction):

Everyone continued hacking on his stuff, e.g. Bart on his kioslave for UPnP MediaCollections, to be used especially by Amarok, and me on the UPnP backend for the network:/ kioslave. Which also involved discussions with Frank about shaping the D-Bus API of Coherence to our needs a little. And how to merge best the UPnP device/service model and objects into the network classes model of the base library of the network:/ kioslave, along with the ones from DNSSD and Co. This merging needs more thoughts definitely.

Lunch was cared for by our host Edward Hervey from Collabora Multimedia leading us to another non-tourist restaurant with good food and local atmosphere.

Other stuff that day was done was demoing the things now working, like the Coherence plugin for the Spykee robot delivering pictures from its built-in webcam via UPnP. Or things that have been working for a while, like each one playing his favourite songs via UPnP on Frank’s computer, in a kind of battle style. Including controlling the volume from remote.

Zaheer, Konqui and me looking what Frank does that makes him wear the K-scarf this way

In some way Coherence is to UPnP what Avahi is to DNS-SD, a demon wrapping the specific system and making it accessible via D-Bus. Just that Coherence does a lot more, as UPnP is not only about discovery of services, but also of types of devices and the interfaces/services they implement and how to access them.

As the developers of Coherence are using GNOME-centric desktops the current desktop integration is basically done for GNOME programs, like Totem, Rhythmbox, and Nautilus.

Let’s hope KDE-Libs based programs can catch up to these This was the very purpose of this sprint, and it gave a good foundation. It just left us with quite some work to do. Especially Bart might appreciate help with his kioslave, as he is short on time. So if you want Amarok to as well access other UPnP MediaCollections as well as export it’s own collection as such in the next time, get in contact with him.

I think I might be able to get the integration into the network:/ kioslave done in October (promises, what promises?). Surely there will be the need for feedback. So please stay tuned, more detailed info about approach and progress to follow…

Coherence/KDE sprint Day 1

Big splash for yesterday evening: “Breakthrough in the hotel!”

Barcelona, wonderful city. Crowded with a lot of people, enjoying the life style and atmosphere. This weekend there are six people more, but here for hard work, to do some collaboration on UPnP support in FLOSS software. So is there a better place to stay at then the Collabora Multimedia office here in Barcelona?

Together with Bart Cerneels, from the Amarok crowd and to most of us known as the great organizer of Akademy 2008, I am here to write code to connect to the UPnP world, by using Coherence. If you haven’t already heard of Coherence before, it is an well developed UPnP-enabling framework, written in Python and accessible via D-Bus by any other program, hiding the UPnP complexity both to the providers and the consumers of UPnP services.
So e.g. Amarok could easily offer access to its own media collections for any UPnP capable client on the network as well as access those of others, at the same time control any other player in the network (MediaRenderer in UPnP terms).
And it is also a comfortable way to add a UPnP backend to the network:/ kioslave, so the UPnP-only devices and their services are also listed there. I was hoping someone else was going to do this, but, well, scratch your itch…

Still it feels like boring work to do compared to what some others sitting at the table with us are doing: There is Philippe Normand who is working on tubing UPnP over the intertubes with the Mirabeau plugin to Coherence. And there is Zaheer Abbas Merali, who is wrapping standard UPnP services around little next-generation-mars^W indoor robots, so you can access the robot with any standard UPnP client (like the PS3, PSP or XBox). Then there is Sebastian Pölsterl who works to build a UPnP supporting personal video recorder (PVR) based on his DVB-Daemon.
Last but not least with us are Frank Scholz, the founder and main developer of Coherence, and Benjamin Kampmann, official vice-president of marketing for Coherence.

I had a quick start with the code Adriaan de Groot had written at the first Coherence/KDE sprint in Paris this spring. Still I had nothing to show when we left the office in the evening. But when most of us went to spend much money to watch 22 men catch a single ball, just to kick it away again, Bart and I waited for them in the hotel, typing even more code and then, the breakthrough, at least for me:

And it comes with congrats to Ahmed Moustaf! His computer was the first to appear in the network:/ kioslave by being feeded from the Coherence-driven UPnP backend. Sitting in the hotel room I had no longer all the devices available like we have with us in the Collabora office, but another friendly, although unknown person had his computer broadcasting his UPnP services, so there was content. You won’t see from the documenting screenshot it is from the UPnP, as that is one of the targets of the network:/ kioslave, hiding all the unneeded technical details from the user. And his computer disappeared soon after when I wanted to screenshot some UPnP only listing, was it because we started to browse his media collection and he found out?* So just believe it

*Because if there is one thing which seems to suck with UPnP it is the lack of any access control built into the system. Everybody can access everything. Well, hopefully soon KDE programs can, too

Being Wow’ed by FLOSS, again

Lydia Pintscher* has written a nice summary about the results of this year’s Google Summer of Code for KDE. I think it’s really amazing what has been done. KDE 3 seems so simple and old compared to this.

Or is it just the Oxygen look which makes it look so elaborated?

* I have heard some think there are too few women doing FLOSS stuff. Seems, at least by looking at this weekend’s blog posts on Planet KDE, KDE is doing well and seems a good place not only for small and large, young and old, but also female and male humans. Much better obviously than the BMX community at the BMX ground in my neighbourhood, where when I passed this evening the only two female humans were two mothers waiting while there small boys where doing their jumping. Perhaps they can learn from us?

Tutorial: Add a Binary Filter to Okteta

Welcome to the first in hopefully a row of little tutorials showing how you can yourself add the features you need to Okteta, the KDE 4 hex editor.
Although, after writing this one, which took way too much time, I will see how I can make this more economic for me.
For now, you simply have to make use of this tutorial, so my time was well spent!

Setup of the development environment

Unless you already have a development environment, including the sources of kdeutils, e.g. after following the instructions from TechBase, the following little setup should work for you, too:

You need to have installed a proper compiler, cmake, a subversion client and the development packages of qt4 and kdelibs4. You might not need the kdelibs from trunk, version 4.3, perhaps even 4.2 should do it, I do not remember to have used any newer kdelibs API in Okteta so far. (Please report if you experience problems.)

Now get the sources of Okteta:

svn co svn://anonsvn.kde.org/home/kde/trunk/KDE/kdeutils -N
svn up kdeutils/okteta

Build the program in a separate directory (replace “yourinstallpath” with a proper path as useful for personal development, e.g. the output of kde4-config --prefix):

mkdir build
cd build
cmake -DCMAKE_INSTALL_PREFIX=yourinstallpath ../kdeutils
cd okteta
make

The build will take a while as all those files you just saw when doing the checkout want to get compiled…

Do a make install to see if the installation works, too.
Then testrun Okteta with yourinstallpath/okteta.
(Or just okteta if you are sure your shell does not outsmart you and instead starts okteta from a path it has cached from a previous start when this version of Okteta was not yet installed)

Hoping that all went smooth we can now turn to…

Adding a Binary Filter

In this tutorial you write a filter which will turn all bytes either to 0×00 or 0xFF, depending on whether their old value is below or above a given level. It will look like this:

Call for action: Write your own plugins for Okteta

Up to now I have written Okteta, the KDE 4 hex editor, completely on my own. This has pros and cons:
The biggest pro is that I do not have to argue a lot about design decisions, especially as I have some big plans with the co-developed framework Kasten, which Okteta is built on (to serve as testing area) and which might not yet be understood/followed by others (could just be a crazy idea, after all) as I have implemented only parts of it and also added unmarked intermediate shortcuts to get things running already.

The biggest con is that Okteta doesn’t grow as fast as it could.* So there might be some features you, dear Okteta user, are still missing from it. Additionally, remember that I do not really use Okteta myself, it really is almost only a testing area of Kasten for me (I still am happy if Okteta is already a useful tool to many, as this indirectly results in more testers for Kasten). And so I concentrate on things which help to develop Kasten further. I now turned to think about support for very large files, meaning working with data not completely copied to the working memory (just do not hold your breath for this ).

So, what about the binary filters (e.g. a bit switcher), your export plugins (e.g. Base64) or even complete tools (I can not even imagine) you are missing from Okteta and hoping for in a future release?

For the coming weeks I am going to show you how simple it should be to help yourself, in a series of short tutorials.

Okteta/Kasten has been designed very modular, with a lot of interfaces and plugin-prepared structures. For now everything is hardcoded, but as we have readwrite access to the sources this is of no great problem here

There is almost one month left for you to decide if you want to make your hands and keyboard dirty with an addition to Okteta, then the soft features freeze will hit the development branch for KDE 4.4. And only November 11th would set the barrier for a basically working solution, so should be enough time to get your plugin done, given it doesn’t include rocket science. Or you could be still pleased if it only will be part of KDE 4.5 next summer (well, still not assumed to include rocket science). In any case I will be happy to help you where doable.

The first tutorial will appear on sunday (September 27th) and teach you how to write a binary filter (Update: online now). So stay tuned if you are in need of a hex editor like Okteta and interested to give the implementation of your needed features yourself a try.

* For sure, the pace of development for Okteta has been quite good, just have a look at the automatic analysis done for Okteta at ohloh.net. 48,042 lines of code have been gathered in the last two or so years. Which, using LOC while admitting it’s an evil metric for code work, is estimated to have, taking the given default average salary of $ 55,000 / year, a “Project Cost” of some $ 641,343. I just hope I never will write myself this bill. Because the moment I do I am for one in big debt to someone and also in trouble because someone doesn’t pay my bill.
Another interesting metric seems behind the assigned property “Well-commented source code”: Well, I have spread the code over quite some files, and each and every file has 20 lines comment from the license header. The other comment I often leave is done with this pattern:

// TODO: some improvement

currently around 560 times. Instead almost all API is missing proper apidox. So, rather not well-commented. Still, I use expressive, thus pretty long names for classes and variables, so comments are not often needed (it’s obvious what ByteArraySourceCodeStreamEncoderConfigEditorFactory does, or?). But I wonder if this metric takes this into account

Polishing the tools for Okteta 0.4

While still avoiding to tackle the next big task for Okteta, supporting very large files, thanks to a strain in my right leg I have found some time to polish the tools a little.

Last night I worked on the input widgets for addresses/offsets and byte strings. Those enable the user to select one from several formats in which to do the input (auto detection still a TODO). While I wanted to make a widget similar to the one of the searchbar for Konqueror, with the current format to be selectable like the current engine in the left of the searchbar widget, I found this not easy to implement and the searchbar implementation not pleasing my needs (e.g. not focusable with keyboard, no method QComboBox::setEditedIcon() or similar). So I stayed with my old implementation where the format is selected in an external combobox, but which is now placed to the left of the input field and only shows an abbr. of the format type (“Hex”, “Dec”, …). The input field itself is a combobox which can and is used to store previous inputs (the popup still lacks a delegate to also show the corresponding format). These widgets are now used all over Okteta, just cmp. the screenshot below.

Already in trunk for some time are support in Kasten for inlined dialogs, with Okteta’s Goto dialog and the (new for 0.4) Select-range dialog (see screenshot) ported to it. Additionally all tool views are ported to the new AbstractToolWidget in Kasten which has support for default buttons which are activated on the press of the Return key, like you know from the dialogs. So if any widget in a tool view has focus (and doesn’t eat the Return key itself) one now longer needs to switch the focus to the button but can simply press Return (just seems that the Oxygen style does not give an extra style to default buttons?).

Also switched to use QFormLayout where possible. And found the great KIntNumInput, so any units are shown inside the input field (including bytes, as in number of bytes )

Just the Find and Replace dialogs may stay popup dialogs for a while. For one I have no idea how to best place all the options into an inline-dialog (e.g. the solution in Kate does not follow my workflow) without getting a monster dialog. Even with the Go-to and the Select-range inline-dialogs the window size is sometimes changed, which is a no-no IMHO. And for another on this change I want to implement search-as-you type, which is still lacking a proper foundation for worker threads and locking of document objects. So, support for very large files has a chance to be done before that.