Tutorial: Create your own Okteta structure definitions

The best new feature in the soon to be released version 0.4 of Okteta, the KDE hex editor, coming with the KDE SC 4.4, is surely the Structures tool (see also the “All new Okteta features for KDE SC 4.4 in a picture”). Because this tool is just a few months old, his author Alex concentrated on the code and not on the documentation, so the Okteta docs in the 4.4 branch have been missing any note of it. But Burkhard Lück, one of the heros of KDE documentation, pushed Alex and me to update the documentation in trunk at least with some quickly written text, so he might be able to backport it to 4.4, with success. If all works out, the updated doc will be part of the 4.4.1 release. So spend Burkhard a big $favourite_drink if this makes you happy

(Update: The updated manual is now part of the releases starting with 4.4.1 and also online available.)

Until then, find following the excerpt regarding the creation of structure definition. It is lazily copy-pasted from the meinproc4 generated html, hopefully doesn’t break too much of webpages embedding this. Please give this tool and the docs a try (e.g. if testing the RC1, where Okteta is part of kdeutils), feedback is very much welcome:

---

General

The Structures tool enables analysing and editing of byte arrays
based on user-creatable structure definitions, which can be built from
arrays, unions, primitive types and enum values.

It has an own settings dialog, which can be reached by using the
Settings button. There are various options that can be
configured, like the style (decimal, hexadecimal or binary) in which the
values are displayed. Moreover it is possible to choose which structure
definitions get loaded and which structures are shown in the view.

Structures are defined in Okteta Structure Definition files (based
on XML, with the file extension .osd).
Additionally a .desktop file contains
metadata about that structure description file, such as author, homepage
and license.

Currently there is no built-in support for creating, editing or
installing structure definitions, so this must be done manually as
described in the next sections.

Tutorial: Add a Checksum/Hashsum Algorithm to Okteta

Here comes the second in a row of little tutorials showing how you can yourself add the features you need to Okteta, the KDE 4 hex editor (the first one was about Adding a Binary Filter).

Since KDE 4.3 (resp. Okteta 0.3) Okteta features a tool to calculate a checksum/hashsum from the bytes in the current selection. It comes with some built-in checksum algorithms (Modular sum (8/16/32/64 bit), Adler-32, CRC-32), but for hashsums relies on the QCA lib (Qt Cryptographic Architecture) (can be SHA-0/1/224/256/384/512, MD2/4/5, RIPEMD-160, Whirlpool, depending on the installed QCA plugins).

Ideally you would write a new hashsum algorithm as a plugin for the QCA library, so it is available to more programs than just Okteta. But if you want to add another checksum to Okteta, here is your tutorial:

Setup of the Development Environment

See for the instructions in the first tutorial about Binary Filters.

Adding a Checksum Algorithm

In this tutorial you write a checksum algorithm which will calculate the parity of all bytes using only a given bit position. It will look like this:

Tutorial: Add a Binary Filter to Okteta

Welcome to the first in hopefully a row of little tutorials showing how you can yourself add the features you need to Okteta, the KDE 4 hex editor.
Although, after writing this one, which took way too much time, I will see how I can make this more economic for me.
For now, you simply have to make use of this tutorial, so my time was well spent!

Setup of the development environment

Unless you already have a development environment, including the sources of kdeutils, e.g. after following the instructions from TechBase, the following little setup should work for you, too:

You need to have installed a proper compiler, cmake, a subversion client and the development packages of qt4 and kdelibs4. You might not need the kdelibs from trunk, version 4.3, perhaps even 4.2 should do it, I do not remember to have used any newer kdelibs API in Okteta so far. (Please report if you experience problems.)

Now get the sources of Okteta:

svn co svn://anonsvn.kde.org/home/kde/trunk/KDE/kdeutils -N
svn up kdeutils/okteta

Build the program in a separate directory (replace “yourinstallpath” with a proper path as useful for personal development, e.g. the output of kde4-config --prefix):

mkdir build
cd build
cmake -DCMAKE_INSTALL_PREFIX=yourinstallpath ../kdeutils
cd okteta
make

The build will take a while as all those files you just saw when doing the checkout want to get compiled…

Do a make install to see if the installation works, too.
Then testrun Okteta with yourinstallpath/okteta.
(Or just okteta if you are sure your shell does not outsmart you and instead starts okteta from a path it has cached from a previous start when this version of Okteta was not yet installed)

Hoping that all went smooth we can now turn to…

Adding a Binary Filter

In this tutorial you write a filter which will turn all bytes either to 0×00 or 0xFF, depending on whether their old value is below or above a given level. It will look like this:

Call for action: Write your own plugins for Okteta

Up to now I have written Okteta, the KDE 4 hex editor, completely on my own. This has pros and cons:
The biggest pro is that I do not have to argue a lot about design decisions, especially as I have some big plans with the co-developed framework Kasten, which Okteta is built on (to serve as testing area) and which might not yet be understood/followed by others (could just be a crazy idea, after all) as I have implemented only parts of it and also added unmarked intermediate shortcuts to get things running already.

The biggest con is that Okteta doesn’t grow as fast as it could.* So there might be some features you, dear Okteta user, are still missing from it. Additionally, remember that I do not really use Okteta myself, it really is almost only a testing area of Kasten for me (I still am happy if Okteta is already a useful tool to many, as this indirectly results in more testers for Kasten). And so I concentrate on things which help to develop Kasten further. I now turned to think about support for very large files, meaning working with data not completely copied to the working memory (just do not hold your breath for this ).

So, what about the binary filters (e.g. a bit switcher), your export plugins (e.g. Base64) or even complete tools (I can not even imagine) you are missing from Okteta and hoping for in a future release?

For the coming weeks I am going to show you how simple it should be to help yourself, in a series of short tutorials.

Okteta/Kasten has been designed very modular, with a lot of interfaces and plugin-prepared structures. For now everything is hardcoded, but as we have readwrite access to the sources this is of no great problem here

There is almost one month left for you to decide if you want to make your hands and keyboard dirty with an addition to Okteta, then the soft features freeze will hit the development branch for KDE 4.4. And only November 11th would set the barrier for a basically working solution, so should be enough time to get your plugin done, given it doesn’t include rocket science. Or you could be still pleased if it only will be part of KDE 4.5 next summer (well, still not assumed to include rocket science). In any case I will be happy to help you where doable.

The first tutorial will appear on sunday (September 27th) and teach you how to write a binary filter (Update: online now). So stay tuned if you are in need of a hex editor like Okteta and interested to give the implementation of your needed features yourself a try.

* For sure, the pace of development for Okteta has been quite good, just have a look at the automatic analysis done for Okteta at ohloh.net. 48,042 lines of code have been gathered in the last two or so years. Which, using LOC while admitting it’s an evil metric for code work, is estimated to have, taking the given default average salary of $ 55,000 / year, a “Project Cost” of some $ 641,343. I just hope I never will write myself this bill. Because the moment I do I am for one in big debt to someone and also in trouble because someone doesn’t pay my bill.
Another interesting metric seems behind the assigned property “Well-commented source code”: Well, I have spread the code over quite some files, and each and every file has 20 lines comment from the license header. The other comment I often leave is done with this pattern:

// TODO: some improvement

currently around 560 times. Instead almost all API is missing proper apidox. So, rather not well-commented. Still, I use expressive, thus pretty long names for classes and variables, so comments are not often needed (it’s obvious what ByteArraySourceCodeStreamEncoderConfigEditorFactory does, or?). But I wonder if this metric takes this into account

Minitutorial: Viewing raw data of files in Konqueror/KDevelop

Want to quickly see the raw data of files in Konqueror or KDevelop? This is possible with the KPart coming with Okteta since KDE 4.1:

Preparations

Open “System Settings”, then select “File Associations” (Found under “Advanced”). Select the file type (e.g. “audio/x-wav”) you want to see the raw data of, choose the tab “Embedding”.

Within “Services Preference Order” press button “Add”, select “Embedded Binary Viewer (oktetapart)” [0] from the list, press “OK”:

If there are already other services listed in the services list make sure the oktetapart is the first entry by selecting it and pressing “Move Up” as often as needed. Press “Apply” and close the System Settings:

[0] Binary Viewer is a misnomer, changed to Hex Viewer for KDE 4.4

Usage

Now start Konqueror and click with your right mouse button on a wav file. In the context menu select “Preview in Embedded Binary Viewer” and you should see the raw data of that file:

In KDevelop you do as you do with text files, just select a wav file and it will be shown as hex view:

Beware: This viewer is only read-only. And while using the Okteta core libs it currently does not come with any of the tools included in Okteta. This hopefully will change for KDE 4.4, but don’t hold your breath (better sit down and help with coding ).

Undo

If you want to undo this change for the file type, go to the settings as described above, select “Embedded Binary Viewer (oktetapart)” in the list under “Services Preference Order” and press “Remove”.

Update:

Since its version 4.1 KDevelop has built-in support for viewing and editing of files as byte arrays (of course using the Okteta libraries). To open a file as byte array, use the context menu on a file in the “Filesystem”, “Projects” or “Documents” toolview (by click with the right mouse button) and chose “Open as Byte Array”. So for KDevelop using the Okteta KPart is no longer needed.