Okteta@Akademy 2008

The seven days of Akademy 2008 have passed quickly. Everything felt so smooth, seemed well balanced, nothing was really annoying (besides a sometimes jittering net access, but that’s it). The organisators must have really done an excellent work. Even the weather was fine. And the local team was very responsive and helpful all the time (even not moaning if contacted when already gone to bed). So once again thanks to all who were responsible! Especially because it was unpaid for almost all!

Now that the dust settles I think the most interesting thing I learned was the power of dtrace, during the OpenSolaris BoF: just handle an object of your interest as a virtual machine, plug into the interface, and voilà, create data in the dtrace fomat which can then be handled by the dtrace tools. And those are really powerful, too. Just too much CLI for my taste I really need to spare some time to learn more about dtrace.

Another interesting thing for me, as author of Okteta, was trying Okteta for the first time myself on Windows and OS X. While I do not really care for these platforms, I am still impressed they are supported, too, without me writing any special code for them, besides staying with the abstraction layers of Qt and KDE and using the proper CMake macros in the buildsystem. So Patrick Spendrin, one of the culprits for the Windows platform support, let me play with Okteta on Windows:

Okteta 0.1 running on Windows XP

And Alexander Neundorf was so kind to have it built and run on his OS X-powered machine:

Okteta 0.1 running on OS X

A minor problem is that on OS X it dies often if opening a filedialog (also seen with e.g. KMail, so this should be a problem in kdelibs). Both platforms are still experimental as of now, but I expect at least for KDE 4.2 official support, given the interest and the work done already.

Okteta is part of the KDE module kdeutils, which I have become the coordinator of. So it was also nice to get to know some of the other kdeutils developers face-to-face. See Jonathan Riddell’s blog for a group photo.

Another group photo is not online yet. It shows those of the city with the most representatives, which is Dresden in Germany. Or is there any other city which had more than 5 people coming from it (besides Oslo)? Hm, this is a statistic question, should be sent directly to Paul Adams, I guess Then the usefulness of a number like committers per city is in question: Does that town lack cultural offerings? Or does it have too many, so one is being spoilt by the choice? Is there just a greater mismatch between the genders? Or is the sun more glaring and the birds more noisy? No idea. If the KDE Research team is out of tasks, I will approach them with this.

I am less satisfied with my own performance at Akademy. I did not manage to achieve what I wanted to. So in the end I should have done more talking and socializing instead of sitting all the time in front of my laptop until late in the night. I got somewhere with the work on the new feature, but till Friday afternoon, when the WLAN was cut off exactly at 17 h, I rather found some little flaws in my design again, which took all of the time (as to be expected). Following you can see what Okteta currently looks like on my system, with some helper environment added for the feature in development:

Okteta: Connecting models...

If it works one day, I think it will be not only cool, but a start to more. Until then just guess what it will be

So another lesson learned: Talk, don’t hack, at Akademy. Let’s rename Hackathon to something else, to not mislead dummies like me.

Open now: Web agency of Okteta

During the recent weeks the module kdeutils, containing the KDE Utilities, has got a website at http://utils.kde.org/.

Okteta, part of the KDE Utilities, has got its share there at http://utils.kde.org/projects/okteta/.

So if you are interested in Okteta and want to know about the
features, to look at the FAQ, to read the documentation, to find your way into development, or to get in contact with the community, you now have a place to go to.

Open 24/7/365. You are welcome

Final polishing of Okteta for it’s first release

If all goes well Okteta will have it’s first real release ever this month, together with KDE 4.1.0. Exciting times.

Exciting because…

… I managed to make it happen. Yeah. Success.
… Okteta is going to fill the gap KHexEdit left. Quite a big gap, as I can guess from the search terms leading to my blog. I hope Okteta can fill large parts of that gap and meet most people’s expectations about what a simple hex editor should work like. There sure will be some disappointed that they will not find certain special features of KHexEdit, I hope they understand how development of Free Software works.
… the code has not yet received bigger real life usage tests. Large parts are covered by extensive unit tests, but only parts. And in the last days I did find some new bugs triggering data loss, when I had been thinking for some time they had been all found, oh dear. Fixed they are again, but I now understand completely how you feel to word things like this in the GPL:

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

Other than that, I am really satisfied with the state of Okteta. I managed to do all features I wanted to do, also some I just hoped to do. And for a 0.1 version, Okteta is already quite feature-rich and should serve me and others in a lot of cases well.

The last days I concentrated on fixing the last big flaws in the UI:
All processes which can take some time, like I/O operations, printing or searching, no longer block the refreshing of the UI, additionally also a busy mouse cursor is shown, like expected.
Another flaw were four hardcoded colors for the four different types of bytes I currently use. They resulted in unreadable views especially with dark color schemes. No more now, using KColorScheme and assigning one of the certain roles to each type of bytes, also in selections, so the colors of the current color schemes are used:

There are still a number of minor glitches left, but I think now: Let the release come

Oh, and BTW:

Catch more with Dutchmans than with others

[[Usenet is dead, now Email and Mailinglist are dying, in Web 2.0 times Blog is the way to communicate/discuss. I am just curious how Akregator will learn to display these threads, given that there can be multiple ancestors :)]]

There are currently some classes in Okteta used for rendering which are named the same (*ColumnRenderer), one of each pair living in a namespace (KHEUI), the other in the standard one. Product of lazyness and temporary solutions (printing needs a renderer, the one for the screen is not yet usable, so until it is, let’s just copy,paste&adapt it).

Without any dutchman:
It worked for me, as the code depending on the renderer never sees both of them.

Here comes dutchman #1:
Unless you compile KDE with enable-final turned on, which puts all source files into a single big one. I never have done, but dutchman #1: So Tom Albers sent in a patch to fix this. E.g. it added the prefixed double-colons to remove ambiguity. With lazyness still turned on, I just had him apply the patch as is, because while not too happy with the look I know^Whope the time of the doubled classes is limited (KDE 4.2). Also my knowledge regarding namespace stuff is, well, improveable currently.

Now dutchman #2 comes in:
On his quest to have the traditional targets of KDE, namely unixoid platforms, not being left aside due to some hyped new targets like OSX or Windows, dutchman #2 puts the KDE code on the plate of quite some more compilers, some I have even never heard of. And a few of them seem to have a different taste about what to swallow and what not, compared to all the compilers that have had it one their plate before. Especially regarding one of the solutions of the other dutchman, namely the prefixed double-colons. So Adriaan de Groot, with his brain’s parser broken so he can not detect the maintainer’s email address or his bookmark to bugs.kde.org, stays with his opened blog editor window and throws his findings into the sea of the world wide web, hoping one of the bottle copies will find the person addressed. Lucky he is, it did
So lazyness is no more, will just rename half of the classes and hopefully have all the compilers now eat all from the plate. So the weather will be nice (central europe mothers tell you this relation). Because I fear dutchmans are more used to play soccer in rain, so they will have a little more advantage against the germans in the final of the EM (European championship about dealing with a single ball and two nets the European style). I do not want to be guilty of that, because I want the german team to have a little chance against them. If only for a goal of honor. As to be seen yesterday, they at least have chances to improve, too. A lot of chances… Or they will have to catch a lot with dutchmans, too.

Broken by broken design fixed by fixed design

Or: Eat your own dogfood, not just taste it.

I was so happy with my draft piece table code pretty much working as the backend for the undo/redo system of the basic bytearray model implementation in Okteta. Alright, it still lacks some heavy tests that it can be thrown against. But from my simple tests by just using it it seemed to work.

Then I proudly showed Okteta to someone else. To see that editing in the value column is broken by it. The way the intermediate states in the editing process of a value are synched with the model did not match the change model in my piecetable. Ouch. So I had to switch editing in the value column completely off, just in time for the KDE 4.1 Beta 1 release. Still, I could be a little bit proud as the switch-off was just a one-variable substitution: mNavigator instead of mValueEditor. Things like the latter are what then keeps me thinking again I can code a little

One obvious solution to the problem above was to add the concept of grouped changes to the piecetable code. I had not planned to do this for KDE 4.1, but it was the best fix I could imagine. So when I found/took some time the last days I went ahead and tried to implement something like that. Looks today like I succeeded. There are some little bugs left, but it works enough that I switched editing in the value column on again, in time for 4.1 Beta 2, hurray.

As a bonus I did another few-liner, so changes done by the filter tool are not just named “Replace” as before, but get the name of the filter. I consider this also a fix, as you can only now identify the filter operations in the history, which is to be suspected I guess:

BTW: The version view on the right in the screenshot is just for development, not included in Okteta release builds since 4.1 Beta 1, so do not wonder if you do not find it in the tool menu.

Bonus for you reading until this line: There is a hidden feature in the value column: In Overwrite mode you can increase and decrease the value of the byte below the cursor by pressing + and - (And now that I write it I think I should add support for operating this way on whole selections. But that should be another feature for Okteta in KDE 4.2 ).

Final Okteta features for KDE 4.1

By becoming module coordinator for kdeutils I had to shift my time from hacking on Okteta to some module coordination. And that with the date of the hard feature freeze coming closer, oh well.

But the big cleanup of the module was finished just in time to get two more features done for Okteta:

• a strings extract tool
• an export facility

The first is roughly done, simply surprises me with new problems over and over. The latter is rushed in, ignoring some concepts I have with the Kakao framework, but works for now. And still has some smartness, as it reuses the encoder classes which are used for “Copy as…” (thanks to the QIODevice abstraction).

In the process I moved a very old, but well hidden feature to a visible place: the ability to copy the current selection as the view rendered to plain text. I still remember how I did it in 2003 in Nové Hrady, when I was new to all the KDE development, yet sitting in the same computer room with all the famous KDE guys, how exciting for me. I showed it there to Josef (Spillner), next to Tobias (König) one of my two leaders from Dresden. Besides him nobody might have ever noticed it, because you only triggered it if you dragged a selection from the value column of the Okteta part, which itself only a few might have used (KPilot generic database editor, KDevelop debugger view, …). Touching personal history, oh well…

Selecting the shown menu entry adds this to the clipboard, using all the settings of the view (except for “Show unprintable chars”, for a good reason):

0002:D0E0 |                 2F 68 6F  6D 65 2F 6B  6F 64 65 72 |      /home/koder
0002:D0F0 | 2F 4B 6F 64  65 2F 6B 64  65 73 76 6E  2F 74 72 75 | /Kode/kdesvn/tru
0002:D100 | 6E 6B 2F 4B  44 45 2F 62  75 69 6C 64  2E 64 65 62 | nk/KDE/build.deb
0002:D110 | 75 67 2F 6B  64 65 75 74  69 6C 73 2F  6F 6B 74 65 | ug/kdeutils/okte
0002:D120 | 74 61 2F 63  6F 72 65 2F  6F 6B 74 65  74 61 63 6F | ta/core/oktetaco
0002:D130 | 72 65 5F 61  75 74 6F 6D  6F 63 2E 63  70 70       | re_automoc.cpp

There is one more feature I would have liked to get done for KDE 4.1, but which won’t make it now: support for very large files. Is rescheduled for KDE 4.2. Until then just do not load too large files! I should perhaps add a warning to the file selector.

Okteta on its way to kdeutils + WhatsThis

The longest journey starts with the first step. Or: The biggest program starts with the first line. (Just: These terms do not contain any mentioning of the category time. I definitely need other proverbs which complement these to help me with life.)

Now, that a lot of lines have been collected in the subdirectories of the Okteta project, a new milestone is approached: The stand-alone program Okteta is heading for inclusion in kdeutils. Since Monday its code resides in kdereview, awaiting your objections or change requests.

Christian Ehrlicher already did what looks like almost every KDE4 module is having to cope with: He adopted 5 lines (in words: five, really only) and now it compiles also under another platform, delivered from Redmond. Impressive. CMake, Qt and KDE, your friends for almost-instant-multiplatforming, obviously.

Instead of an image (for a screenshot just go and build the code yourself) this time some words (just not thousands) for those, like Thomas Z., who are still curious what Okteta is at all:

Okteta is a successor to KHexEdit for KDE 4.x. It is a simple editor for the raw data of files. This type of program is also called hex editor or binary editor.

The data is displayed in the traditional view with two columns: one with the numeric values and one with the assigned characters. Editing can be done both in the value column and the character column. Besides the usual editing capabilities Okteta also brings a small set of tools, like a table listing decodings into common simple data types, a table listing all possible bytes with its’ character and value equivalents, a info view with a statistic and a filter tool. All modifications to the data loaded can be endlessly undone or redone.

Due to its very modular design it should become soon extensible for plugins.

So yet another hexeditor. Besides, this one can use sexy Oxygen.

“Releasemode: On” for Okteta

Release often and early. But not too early, first impression stays. Besides the fact, that holding your code in a public repository is like releasing all the time, I feel now is the time where the mentioned two rules are getting well balanced for Okteta.

So I have set myself in release mode, removing and hiding all undone things, even if they look interesting and like some fun, and setting up a list of those things which need at least to be done for a good first release. There a quite a few, especially after looking at the requirements to move to kdereview. And that is target No. 2. Two targets, that might be one too much. But a release is simply a subtarget of moving to kdereview, so there shouldn’t be any conflicts, I think.

One thing to be done is making the licensing of Okteta complying to KDE’s licensing policy. Okteta was a GPL v2-only project until now. Trying to grasp the idea of GPL v3 I think I am comfortable with that license, too. So I am relicensing all code also to the version 3. But which one to use for KAboutData, it supports only the mentioning of one license? Oh well, there is the source code, here is my editor, this is the free hour.,. so a patch for support of multiple licenses is pending for review in the mailinglist kde-core-devel, please have a look.

But there is another problem with the “About…” dialog. It only mentions the top level program. What about all the libraries and frameworks used by it, which only make the program possible? In the case of Okteta, the code of the program itself are just a few lines, that wire things together. Instead all real functionality is delivered by libkakao, libokteta*, libkde*, libqt*, …, down to libc. Where do all those software modules and their creators get the credit?

Anyway, for now at least the new program-centric multiple-licenses-capable About-dialog for Okteta, like it is possible with the mentioned patch:

(Copyright since 2006? Yes, the first code fragments date back to that year. It was just this winter where I pushed things a little)

Okteta turned into Construction site for Undo/Redo

Almost done, almost done, just the missing final 5 %. Which are quite tough. And never decreasing, one thing done opens a new one, hey.

But still what has been done meanwhile is quite pleasing: After many years wanting to do it I finally simply started and tried a first primitive implementation of the data struture concept called “piece table” as the backend for the undo/redo mechanism of the bytearray model. Piece tables are also used by Abiword and QTextDocument. If you are interested for a start read e.g. the page Part 17 - Editing Text with Piece Chains by some James Brown, so I can omit any experiments in describing it myself.

The code is now with Okteta in KDE’s repository. It is pretty rough and has flaws. Like the cursor forgetting where it has been at the previous changes. That is one of the tough problems, because with multiple views there are multiple cursors, the bytearray document does not know about views, views can come and go, so who is in charge to care for the cursor positions and/or how to calculate them?

And dependencies. I tried to make the piecetable code unaware of the basic data structure (byte array for me), worked so far. Because others might want to make use of the code, too (interested? contact me). But now I see that there are data structures which travel more than one module boundary, like the list of changes (e.g. on multiple undo), which is emitted by the piecetable via the bytearay model to the bytearray view, so the later can do the cursor jumping calculations. Repacking lists is not what I like. And encapsulating by wrapping iterators looks like too much work. Sigh

But other than that, things are looking good. There are already two controllers for undo/redo (or versioning), one classical, which embeds into menu and toolbar, and a (readonly) monitor, which sits in the sidebar and shows all versions. Both operate on the Versionable interface of the libkakao framework, so other document classes implementing this interface can share them (one day when libkakao is ready). The KByteArrayDocument class at least does now, and wraps things to the experimental KPieceTableByteArrayModel. That one still loads files completely in working memory. But once everything works the loading will be switched to memory mapping, so working with GB-sized files should be possible

For now: Stay away from editing with Okteta, it is broken again. Parents are in care for their children.

Standard screenshot following, presenting the latest achievements as described:

Update: Nick’s question made me dig a little. And I found it was almost exactly 4 years ago that I decided to try the piecetable approach, see this thread on the mailinglist koffice-devel. But it stayed on my TODO list until a week ago. Oh, and the PDF document of the paper by Charles Crowley in now available directly at his homepage, too.

Personality, not underwritten

A lot of the helper variables in Okteta are named byte. No problem, as this is no keyword in C++. Instead the name for the type representing a single byte, that is eight bits today, is char (are there any current machines, where this isn’t true?). So all byte arrays are treated as arrays of char,
following QByteArray, which also operates on char.

When implementing a rotation filter I used byte >> width. And wondered why this didn’t yield the expected results. Looking at them more closely (of course one can switch to the display of the binary values in Okteta) it triggered that what I see is an arithmetic right shift (which happily adds a leading 1 again, if the unshifted byte had there one, to keep it a negative value. Sure, I knew about this before. But it’s nothing that I have present in my mind. Why should it be?

After all… what do names mean? What should I think of, if I hear “unsigned character”?

Oh, those legacy. I want ++++C, sometime.

Filter tool of course now also dockable, rejoice about another screenshot: