Synchronisation with Kasten, just to filesystems now

Sometimes it seems with faster and faster JavaScript-JIT-compilers and Gears/HTML5 the current software stacks, written in other languages, like e.g. Qt/KDE are going to be legacy. Google Docs & Co. show what is possible, with the browser as the generic adapter to the platform.
But the current software stacks are fighting back. Things like the openSUSE Build Service make it possible to have your code compiled and ready to be delivered for several platforms, run natively by the hardware you actually bought to do so, instead of having some resources abused for virtual layers. It just needs code which does not (over-)use platform-specific constructs. The expansion of Qt/KDE to more and more architectures is a fine example here what is possible.
Curious what will be next, perhaps Javascript programs using alternatively binary blocks of code compiled for popular hardware, given the OS has built-in access control? Things are merging.

With this in mind I congratulate the people from AbiWord on their new version 2.8, which not only improves the support for the open spec’ed on-disk format OpenDocument (.odt), but especially improves on the collaboration features which integrate the Web 2.0. That looks really great. My only nitpick is that this seems to be a non-FLOSS webservice. I just hope they find a good business model so that the software for the server can be shared, too. Others, like Inkscape or hopefully soon KOffice, might be interested in such a server system, too.

I also congratulate the people from Étoilé, who just have started to send objects over XMPP. This all sounds so promising.

Realtime collaboration and such data model synchronisation is something I hope Kasten, my WIP framework for composing programs, one day supports out of the box, too. From the very beginning this has been considered in the design, even if it is not yet too visible. And it still will take some time to get there.

There is other stuff to do first, like making Okteta work with very large files. This isn’t done yet, I am still working on it, but have hopes this feature will make it into the next release. Working with very large files means only loading the parts which are currently interesting (e.g. solved by memmapping the file). Currently Kasten, the framework Okteta is built on, only has support for the concept of loading document objects completely from the remote storage (like the filesystem on disk) to the working memory, done by subclasses of AbstractModelSynchronizer. There is a specialized subclass AbstractModelFileSystemSynchronizer, which takes care for the common stuff on dealing with the filesystem, including loading/storing to remote filesystems with KIO::NetAccess. So the/your document-specific subclass has just to implement the de- and encoding of the data from a QFile to the working data model.

Now, AbstractModelFileSystemSynchronizer was not really finished, so before adding a new construction site in Kasten for the partial-loading of files I want to finish the old one for the complete loading. E.g. there was no support for a property State of the remote storage. Thanks to KDirWatcher there is now some easily done initial one. Investigations are still needed how to make sure there is no race-condition where another process might break into writing to the file, reading the file’s timestamp of the own modification and starting to watch for changes again.
Also, KDirWatcher does not work for Kio-slaves. So files loaded from them are just marked to be in Unknown state. Besides, if the network connection is down, they are set to Unreachable. Solid::Networking serves this with just a few lines again

See in the screenshot for the two state symbols now in the Documents tool: the left icon shows the working data state (saved changes, unsaved changes), the right one the remote storage state. The yellow flag means Unknown, in case you wonder, and turns into the Network-disconnected icon if you disconnect from the network, like pulling the plug (looks impressive if you have several files loaded e.g. by sftp:// ). Well, seems I need to apply for some more icons from our icon heros.

Other changes in this area:

• The action Revert is renamed to Reload. Revert will not do what the name says if the storage copy was modified since the loading of the document. Revert would rather mean undoing all changes, so that action semantic might rather appear in the Edit menu, for some quicker navigation through the change history.
• The actions Reload and Save are only available if they are useful and usable. E.g. they are disabled if there are no known changes in both the working copy of the document data and the storage copy or the connection to the remote storage is lost.
• Also new documents which are generated from random data or user input (like parameters) are set to modified, so closing them without adding any further content will still trigger the question if you really want to discard this perhaps unique data.

Okteta: Fighting the crisis with productivity from generators

(And if it is just random data)

There has been some productivity-related productivity in Kasten and Okteta development the last days, as can be seen by commit logs like “added: factories to create generators” :

Kasten, my WIP framework for composing programs, gained some AbstractModelDataGenerator class to serve as base class for data generating stuff. It also got a general Insert controller which plugs into the menu, offers all available data generators and has the chosen one generate its data (if parameterized after showing a config dialog) and then inserts it into the current document. And with the AbstractModelDataGenerator around, I extended the AbstractDocumentFactory to also create new documents from QMimeData and reworked the general New controller: it now not only offers to create an empty document, but also to create one prefilled with data from one of the data generators. Or by the data from the clipboard, which also serves QMimeData. And while I was at it, the shell now no longer rejects D’n'D data other than file urls to non-document areas (like next to tabs or the empty space on start) but passes it to the document factory to see if it can construct a new document from it, all by just a few lines of codes. Kasten may look bloated to some (including myself sometimes) but it has it’s value here and there

To see if it is working Okteta, the KDE 4 hex editor, which is built on Kasten, has got some generators:
For one Okteta since the first release had some simple Insert pattern controller, which now got ported to a plain data generator plugin (+ config editor). Then two weeks ago I found myself for the x-th time in life in the need for data in the whole range of all bytes, from 0 to 255. So I had done a clone of the pattern controller and already added as a Insert sequence controller (which started the idea for these general data generators). Got ported to a plain generator plugin, too.

Another generator perhaps useful might be a random data generator. So now there is one. It is based on srand(time(0)) and rand(), which may be of different quality depending of the platform’s libc. If you instead want a certain algorithm, please contribute and send me some code for it filling a QByteArray, licensed like the rest of the file, I am happy to add some alternative RNG to select from

Lots of TODOs and ideas coming with this, but works already. If you like screenshots, have a look:

And there has been even more productivity with regard to Okteta: First external contribution is coming with a new tool More on this soon…

Tutorial: Add a Checksum/Hashsum Algorithm to Okteta

Here comes the second in a row of little tutorials showing how you can yourself add the features you need to Okteta, the KDE 4 hex editor (the first one was about Adding a Binary Filter).

Since KDE 4.3 (resp. Okteta 0.3) Okteta features a tool to calculate a checksum/hashsum from the bytes in the current selection. It comes with some built-in checksum algorithms (Modular sum (8/16/32/64 bit), Adler-32, CRC-32), but for hashsums relies on the QCA lib (Qt Cryptographic Architecture) (can be SHA-0/1/224/256/384/512, MD2/4/5, RIPEMD-160, Whirlpool, depending on the installed QCA plugins).

Ideally you would write a new hashsum algorithm as a plugin for the QCA library, so it is available to more programs than just Okteta. But if you want to add another checksum to Okteta, here is your tutorial:

Setup of the Development Environment

See for the instructions in the first tutorial about Binary Filters.

Adding a Checksum Algorithm

In this tutorial you write a checksum algorithm which will calculate the parity of all bytes using only a given bit position. It will look like this:

Tutorial: Add a Binary Filter to Okteta

Welcome to the first in hopefully a row of little tutorials showing how you can yourself add the features you need to Okteta, the KDE 4 hex editor.
Although, after writing this one, which took way too much time, I will see how I can make this more economic for me.
For now, you simply have to make use of this tutorial, so my time was well spent!

Setup of the development environment

Unless you already have a development environment, including the sources of kdeutils, e.g. after following the instructions from TechBase, the following little setup should work for you, too:

You need to have installed a proper compiler, cmake, a subversion client and the development packages of qt4 and kdelibs4. You might not need the kdelibs from trunk, version 4.3, perhaps even 4.2 should do it, I do not remember to have used any newer kdelibs API in Okteta so far. (Please report if you experience problems.)

Now get the sources of Okteta:

svn co svn://anonsvn.kde.org/home/kde/trunk/KDE/kdeutils -N
svn up kdeutils/okteta

Build the program in a separate directory (replace “yourinstallpath” with a proper path as useful for personal development, e.g. the output of kde4-config --prefix):

mkdir build
cd build
cmake -DCMAKE_INSTALL_PREFIX=yourinstallpath ../kdeutils
cd okteta
make

The build will take a while as all those files you just saw when doing the checkout want to get compiled…

Do a make install to see if the installation works, too.
Then testrun Okteta with yourinstallpath/okteta.
(Or just okteta if you are sure your shell does not outsmart you and instead starts okteta from a path it has cached from a previous start when this version of Okteta was not yet installed)

Hoping that all went smooth we can now turn to…

Adding a Binary Filter

In this tutorial you write a filter which will turn all bytes either to 0×00 or 0xFF, depending on whether their old value is below or above a given level. It will look like this:

Call for action: Write your own plugins for Okteta

Up to now I have written Okteta, the KDE 4 hex editor, completely on my own. This has pros and cons:
The biggest pro is that I do not have to argue a lot about design decisions, especially as I have some big plans with the co-developed framework Kasten, which Okteta is built on (to serve as testing area) and which might not yet be understood/followed by others (could just be a crazy idea, after all) as I have implemented only parts of it and also added unmarked intermediate shortcuts to get things running already.

The biggest con is that Okteta doesn’t grow as fast as it could.* So there might be some features you, dear Okteta user, are still missing from it. Additionally, remember that I do not really use Okteta myself, it really is almost only a testing area of Kasten for me (I still am happy if Okteta is already a useful tool to many, as this indirectly results in more testers for Kasten). And so I concentrate on things which help to develop Kasten further. I now turned to think about support for very large files, meaning working with data not completely copied to the working memory (just do not hold your breath for this ).

So, what about the binary filters (e.g. a bit switcher), your export plugins (e.g. Base64) or even complete tools (I can not even imagine) you are missing from Okteta and hoping for in a future release?

For the coming weeks I am going to show you how simple it should be to help yourself, in a series of short tutorials.

Okteta/Kasten has been designed very modular, with a lot of interfaces and plugin-prepared structures. For now everything is hardcoded, but as we have readwrite access to the sources this is of no great problem here

There is almost one month left for you to decide if you want to make your hands and keyboard dirty with an addition to Okteta, then the soft features freeze will hit the development branch for KDE 4.4. And only November 11th would set the barrier for a basically working solution, so should be enough time to get your plugin done, given it doesn’t include rocket science. Or you could be still pleased if it only will be part of KDE 4.5 next summer (well, still not assumed to include rocket science). In any case I will be happy to help you where doable.

The first tutorial will appear on sunday (September 27th) and teach you how to write a binary filter (Update: online now). So stay tuned if you are in need of a hex editor like Okteta and interested to give the implementation of your needed features yourself a try.

* For sure, the pace of development for Okteta has been quite good, just have a look at the automatic analysis done for Okteta at ohloh.net. 48,042 lines of code have been gathered in the last two or so years. Which, using LOC while admitting it’s an evil metric for code work, is estimated to have, taking the given default average salary of $ 55,000 / year, a “Project Cost” of some $ 641,343. I just hope I never will write myself this bill. Because the moment I do I am for one in big debt to someone and also in trouble because someone doesn’t pay my bill.
Another interesting metric seems behind the assigned property “Well-commented source code”: Well, I have spread the code over quite some files, and each and every file has 20 lines comment from the license header. The other comment I often leave is done with this pattern:

// TODO: some improvement

currently around 560 times. Instead almost all API is missing proper apidox. So, rather not well-commented. Still, I use expressive, thus pretty long names for classes and variables, so comments are not often needed (it’s obvious what ByteArraySourceCodeStreamEncoderConfigEditorFactory does, or?). But I wonder if this metric takes this into account

Polishing the tools for Okteta 0.4

While still avoiding to tackle the next big task for Okteta, supporting very large files, thanks to a strain in my right leg I have found some time to polish the tools a little.

Last night I worked on the input widgets for addresses/offsets and byte strings. Those enable the user to select one from several formats in which to do the input (auto detection still a TODO). While I wanted to make a widget similar to the one of the searchbar for Konqueror, with the current format to be selectable like the current engine in the left of the searchbar widget, I found this not easy to implement and the searchbar implementation not pleasing my needs (e.g. not focusable with keyboard, no method QComboBox::setEditedIcon() or similar). So I stayed with my old implementation where the format is selected in an external combobox, but which is now placed to the left of the input field and only shows an abbr. of the format type (“Hex”, “Dec”, …). The input field itself is a combobox which can and is used to store previous inputs (the popup still lacks a delegate to also show the corresponding format). These widgets are now used all over Okteta, just cmp. the screenshot below.

Already in trunk for some time are support in Kasten for inlined dialogs, with Okteta’s Goto dialog and the (new for 0.4) Select-range dialog (see screenshot) ported to it. Additionally all tool views are ported to the new AbstractToolWidget in Kasten which has support for default buttons which are activated on the press of the Return key, like you know from the dialogs. So if any widget in a tool view has focus (and doesn’t eat the Return key itself) one now longer needs to switch the focus to the button but can simply press Return (just seems that the Oxygen style does not give an extra style to default buttons?).

Also switched to use QFormLayout where possible. And found the great KIntNumInput, so any units are shown inside the input field (including bytes, as in number of bytes )

Just the Find and Replace dialogs may stay popup dialogs for a while. For one I have no idea how to best place all the options into an inline-dialog (e.g. the solution in Kate does not follow my workflow) without getting a monster dialog. Even with the Go-to and the Select-range inline-dialogs the window size is sometimes changed, which is a no-no IMHO. And for another on this change I want to implement search-as-you type, which is still lacking a proper foundation for worker threads and locking of document objects. So, support for very large files has a chance to be done before that.

Okteta widgets now plugged into Qt Designer

As blogged about before, one of the things that will be new in the next release of KDE, 4.4, is the publishing of the Okteta Libraries, a set of classes to enable the display and editing of byte arrays of all kinds, using hexadecimal, decimal, octal, binary or several character encodings.

While I personally prefer to do the construction of dialogs and forms directly in code, others like the possibilities that come with the UI files and the ability to edit them in a drag-and-drop interface like Qt Designer offers. To please those among them who might be interested in the Okteta Libraries, trunk now features a Qt Designer plugin for the Okteta widgets. It should also improve discoverability of these widgets as well as enable people to play around with them to see if they fit their needs. Screenshot as proof for the next step in “Okteta everywhere” following:

You can write Qt Designer plugins for your widgets, too, if you haven’t already. Just follow the new tutorial on TechBase (and please report any problems you might encounter).

New in Okteta: Splitted Views

Konqueror can do it. Kate can do it. KDevelop can do it.
And possibly others, too. So, Okteta should not stand back and also be able to split a view and have two or more of them at the same or multiple documents. Can be useful in times. And afterall, the Okteta library can do MVC since some years (back then had classes named KHexEdit, even if hasn’t been used by and never shared anything with the program named KHexEdit, besides living in a subdir of it).

Sadly, all programs have their own implementation, which are also to a certain degree coupled with the rest of the program or the framework it is built on. So there is now another (draft of an) implementation, again coupled to a framework but this time to one with ambitions for more: the MultiViewAreas (working title) in Kasten.

While there are some glitches left most of the basic stuff is working, one can split views vertically and horizontically, here with the same approach like Sublime/KDevelop has:

See how on splitting also the settings are copied to the new view. Even better, it is tried to set the visible part in the new view aligned to the one in the old one, so it appears as if just the splitter and the new tabbar cut into the old view, and the user must not research the parts she has seen before.
(Tried as in: Currently broken for the use in Okteta, as some ensureCursorVisible() in the bytearray view widget moves the visible section around on the resizing after the split, workaround pending…)

The Kasten::MultiViewAreas is just another implementation of Kasten::AbstractGroupedViews, though hard coded for use in Kasten::ShellWindow temporarily. An alternative implementation of a multi view areas setup for a design like Kate has (no tabs, statusbar per view) will be the next to be done, so a developer using Kasten has some choice.

One more step on a long way yet for Kasten… But one less to be done

Minitutorial: Viewing raw data of files in Konqueror/KDevelop

Want to quickly see the raw data of files in Konqueror or KDevelop? This is possible with the KPart coming with Okteta since KDE 4.1:

Preparations

Open “System Settings”, then select “File Associations” (Found under “Advanced”). Select the file type (e.g. “audio/x-wav”) you want to see the raw data of, choose the tab “Embedding”.

Within “Services Preference Order” press button “Add”, select “Embedded Binary Viewer (oktetapart)” [0] from the list, press “OK”:

If there are already other services listed in the services list make sure the oktetapart is the first entry by selecting it and pressing “Move Up” as often as needed. Press “Apply” and close the System Settings:

[0] Binary Viewer is a misnomer, changed to Hex Viewer for KDE 4.4

Usage

Now start Konqueror and click with your right mouse button on a wav file. In the context menu select “Preview in Embedded Binary Viewer” and you should see the raw data of that file:

In KDevelop you do as you do with text files, just select a wav file and it will be shown as hex view:

Beware: This viewer is only read-only. And while using the Okteta core libs it currently does not come with any of the tools included in Okteta. This hopefully will change for KDE 4.4, but don’t hold your breath (better sit down and help with coding ).

Undo

If you want to undo this change for the file type, go to the settings as described above, select “Embedded Binary Viewer (oktetapart)” in the list under “Services Preference Order” and press “Remove”.

Next to do: Copy this tutorial to UserBase.

Kasten: WIP framework for composing programs

Since a few days kdeutils (from trunk) also installs some libraries, including public headers:

• Okteta libraries
• Kasten libraries

The Okteta libraries oktetacore and oktetagui provide some model and view classes for hex views of bytearrays. If you have some special data provider you could also subclass your own Okteta::AbstractByteArrayModel. So using the libs directly is now an option, if the KHexEdit interfaces from kdelibs don’t do it for you.

The Kasten libraries are far more interesting, at least to me. Kasten (former project title was Kakao), that is a project of mine where I try to develop a framework for composing programs from modules and interfaces by the model-view-controller pattern, with the pattern recursively applied to the view and the controller. Think KPart 2.0. So far the development was driven by the needs I had with Okteta (that program serving as the testing ground, now you know my motivation for Okteta. Byte arrays are just the most simple data structure.). The Kasten framework is still far from what I have in mind, but things at least are moving I try to express my ideas directly in code, so the ideas meet reality as soon as possible. Might be some more years before I am done, oh well.

Both libraries are still far from elaborated, especially Kasten, given that there has been only one usecase, that is the program Okteta. But you are invited to see if you already can make use of them. Just be prepared for changes. Which could also be done for your needs

The reason I turned to install these libraries now is that I want to reuse them in another project, working title Binspekt: It is about viewing and editing of binary files, that is executables, libraries, object files etc. I always forget about the parameters of objdump, objcopy, nm & Co., so I thought a GUI variant might be nice, and here we go (of course using the very same bfd library from the GNU binutils).

Obligatory screenshot of the current state, which is pre-alpha:

Find the code at playground/devtools/binspekt in the KDE repository, imported there a few hours ago.