Initial release of libkoralle, a simple Qt-based RIFF parser

There are Qt-based solutions for parsing tree-structured container formats like XML and JSON, but when a few days ago I came across a format based on the Resource Interchange File Format (RIFF) a quick search for a Qt-based parser yielded nothing for me… but a sigh and also the result of a mkdir command.

There is some nice documentation about RIFF on the English Wikipedia. This container format is as old as from 1991, and its ancestors even older. WAV and AVI formats are based on it, but also younger formats like Google’s WebP.

So to have other developers searching for a Qt-based RIFF parser yield something and to follow the release-often-and-early mantra, please find on the KDE ftp servers now (thanks again to the KDE admins for their less-in-a-day quick support!) what came after that mkdir command and what serves me quite well already in the parsing code of the format I deal with:

release 0.1.0 of libkoralle, a lib for parsing (and hopefully soon also writing) data in Resource Interchange File Format (RIFF) based formats.

How to use libkoralle:
Given a format based on RIFF with a structure like this:

RIFF id='XMPL'
  'VRSN'
  LIST id='DATT'
    'DATA'
    'DATA'

your code would, assuming the stream is well-formatted, be like this with version 0.1:

#include <Koralle0/RiffStreamReader>
...
Koralle0::RiffStreamReader reader(device);
reader.readNextChunkHeader();
// reader.chunkId(): Koralle0::FourCharCode('X','M','P','L')
// reader.isFileChunk(): true
// reader.isListChunk(): true
reader.openList(); // needs matching closeList();
  reader.readNextChunkHeader();
  // reader.chunkId(): Koralle0::FourCharCode('V','R','S','N')
  // reader.isFileChunk()/isListChunk(): false
  // reader.chunkData()/chunkSize(): data of the content
  reader.readNextChunkHeader();
  // reader.chunkId(): Koralle0::FourCharCode('D','A','T','T')
  // reader.isFileChunk(): false
  // reader.isListChunk(): true
  reader.openList();
  while(reader.readNextChunkHeader())
  {
    // reader.chunkId(): Koralle0::FourCharCode('D','A','T','A')
    // reader.isFileChunk()/isListChunk(): false
    // reader.chunkData()/chunkSize(): data of the content
  }
  reader.closeList();
reader.closeList();

Future version might have support for related container formats (IFF, RIFX, …) and allow passing of custom parsers for the data chunk content, to avoid the temporary QByteArray copy, as well as default parsers for standard chunk types like “INFO”.

See also Snorkel, a simple RIFF structure viewer I wrote using also libkoralle.

Contributors and feedback of course welcome!

Thanks for your help on writing geographic coordinates in $YOUR language

A few days ago I asked for help how to design the parser used in Marble to try to turn a text string into some geographical coordinates in such a way that it also can parse localized writings of such coordinates.

For those curious what came out of this, read on:

While I then forgot to mention the use-cases I was thinking of, like copying&pasting some pretty-printed string from some text, or entering the coordinates somebody tells you on the phone, still the broad feedback I got, both by those commenting on the blog post as well as those replying on the mailinglist of KDE’s translators (thanks to all of you), helped to accumulate quite some testing data, on which based the parser could be designed.

Well, it was done in a short rush, just in time for the translation soft freeze for the upcoming release of KDE Apps 4.8, but non-the-less can parse all of the testing data Which means basically the parser can parse coordinates in these patterns, for all languages:

• with latitude or longitude given as first
• with directions given as prefix or postfix
• with values given just as pure degree, as degree and minutes, or as degree, minutes and seconds
• the last subvalue can optionally be in floating point precision
• the first subvalue can be negative or positive, optionally with the + prefix
• latitude and longitude can be separated by the “,” or the “;” character or simply white space
• symbols for degree, minutes and seconds can be given or left out, if directions are given

The parser first tries to parse using the localized direction terms, than falls back to try the English ones. For the degree, minutes and seconds symbols it is quite tolerant, tries to understand all kind of character (combinations) people might try to express the symbols, using what their keyboard layout offers. Basic rule: if a human would get what is meant, so should the parser (e.g. two consecutive LEFT SINGLE QUOTATION MARKs ‘‘ would be accepted as seconds symbol). Additionally accepted would the localized variant of the symbol, like obviously typically found in non-latin based languages.

Additionally, for a very short notation, two pure floating point values, separated as described above, are parsed as first latitude degree and second longitude degree.

So also the Shorthand DMS notation, as hinted to by Kjetil Kilhavn, should be parsed without a problem. (Support for UTM is only worked on in Marble, no promises on when this might emerge)

For parsing of localized strings to work, the translators are asked to provide in the translation catalogs the information about additional local symbols for the degree, minutes and seconds, as well as how the directions are written. Per direction and symbol there can be multiple variants noted, so languages with variants of the direction terms are also covered. There is a short guideline for the translators, also linked from the strings in the code.

But things are not perfect yet:
only today I learned that Thai has a pattern which is not matching any of those described above, in that it also uses prefixes “longitude” and resp. “langitude” (in Thai) before those parts. Which is also the same pattern as with a sample I got for Icelandic, but I somehow had ignored that, only now found that entry marked as TODO.

So there are things left to do for the Marble version after the upcoming one. But then we all know, software is never finished

Perhaps the geographic coordinates localisation should also be added/moved to the general regional system settings, like Date & Time are, as coordinates become more and more used, due to mobile devices. So all programs dealing with coordinates would benefit and also be consistent (think KStars or Digikam).

Help needed: how to write geographic coordinates in $YOUR language?

As blogged about before, in KDE Apps 4.8 you will be able to enter geographic coordinates in Plasma Runner and then have Marble show it to you.

But there is a problem: the current code is only detecting coordinates where more or less the symbols for degree, minutes and seconds are °, ′, and ″ and where the directions are given as a single word/letter (with letter the first letter of the word, like North, South, East and West in English).

But looking e.g. at the Wikipedia pages about London in some different languages, where the coordinates is given in the DMS format (degree-minutes-seconds), I can see that the parser will fail, perhaps:

Japanese: 北緯51度30分28秒 西経0度07分41秒
Persian: ‏۳۹″ ۷′ ۰°غربی ‏۲۶″ ۳۰′ ۵۱°شمالی
Russian: 51°30′26″ с. ш. 0°07′39″ з. д.
Czech: 51°30´25´´ s. š. 0°07´37´´ z. d.
Georgian: 51°30′00″ ჩ. გ. 0°07′00″ ა. გ.

(no idea why the center of London is different in each language ).

So, to support as many languages as possible and to know what is needed, could you please help me with yours?

Please post a comment with a sample how the geographic coordinates are constructed in your language, and which symbols for degrees, minutes and seconds are used and which terms to describe the directions?

Template to use:

Language (in English):
Degree symbol:
Minutes symbol:
Seconds symbol:
Term(s) for North:
Term(s) for South:
Term(s) for East:
Term(s) for West:
Some example(s) (e.g. London):

Examples could be also additionally in DM or D format.

Looking forward to many examples, best today/tomorrow

Enter geographic coordinates in Plasma Runner …

… and have them being shown in OpenStreetMap with Marble by just the press of the Enter key. You should be able to do so with the upcoming KDE Apps 4.8, due to the just merged commits.

Recently I was offered the geographic coordinates of a place used for a meeting, because none of the involved people was too familiar with the area. “Sure” I said, sitting in front of my computer, “I will just use Marble, pass them on”. And then failed in finding a way to have Marble take these coordinates and locate them on the map for me.

Given that all things looked to be in place and to just need some connecting (and slight fixing, as I found), for any possible next time I will now be able to just open the Plasma Runner, enter the coordinates…

… and then have the place shown to me in OpenStreetMap with Marble by the press of just one more key:

And so will you
Should work with degrees given as pure decimal values, as decimal and minutes, and as decimal, minutes and seconds (small bummer: needs fixing for non-western style languages, hopefully done in time for 4.8).

But I also start to have some geographic bookmarks in Marble, adding support for them was just a few more lines. So now just drop some part of the bookmark title or description to the Plasma Runner …

… and then have Marble show you the area again:

When I initially tried to tell Marble the coordinates I basically missed that I could have entered the coordinates in the “Navigation” panel on the right into the “Search” field. Maybe I even tried, but then the old coordinates string parser is quite restrictive in what he accepts, so maybe I had some wrong space char in my input. That is fixed now.

While looking at the Marble code I also learned that besides coordinates, places/building names, and (postal) addresses one can also enter IP addresses into the “Search” field, to have the assumed location of that IP address shown in the map. Surely not often needed, but handy if.

And if you have another idea what could be translated into a geographic position, you just need to write a dedicated Marble Runner plugin.

For KDE Apps 4.9 again I plan to have finished the support of those Marble Runners directly in the Plasma Runner, so one can search for places or addresses right there. Yes, including locations of IP addresses

Name needed for single-doc-per-window Okteta variant?

Wuuuush – and DesktopSummit is over. What, already? I would like some more weeks of this kind! Although being slightly ill, I still enjoyed it as much as possible. A big “Thank you!” also from me to all the organizers, helpers and supporters!

While hearing and seeing and talking about a lot of interesting things, I also managed to use the time to push that simple hex editor named Okteta, or rather the component system named Kasten it is based on, a little further, so the difference between a so-called “Multiple document interface” (MDI) and a “Single document interface” (SDI) is almost only by the set of used components in the code. I am actually in the camp of the people disliking the MDI approach, as it can be considered a non-integrated workspace in a workspace, so pretty much broken. The SDI variant of Okteta provides all the tools and features of the MDI variant, just the tools Filesystem and Documents and the related entries in the menu have been removed/are not added.

But this poses a question:
Should this be just a difference in the settings? Or should this variant instead be shown as a different program, like users see Kate and KWrite as different programs?
For some reasons I guess these should be two different programs, but then I need a new name for the SDI variant Or should the MDI version be renamed, e.g. to “Okteta Studio” or similar?
Please post/discuss proposals in the comments, or send me your idea by email (see About dialog of Okteta)!
Final code is not yet commited to the central repo, needs some cleanup/finishing and also depends on the decision above, so sorry, cannot yet be tried out (but then besides MDI vs. SDI there is also nothing else new to see yet).

During the Desktop Summit it was interesting to hear how many people actually already have made use of Okteta. Thank you, good to know Kasten code to work in real life
I even was told feature requests on the hall ways. The one late on the last day was even easy to do, it was having a sane default count of bytes per line, which would be 16, without an adaption to the current view width. That was fixed by a single line addition and quickly tested and commited, so at least an Italian user will be more happy with 0.7.1

Beware of QScriptValue and QT_NO_CAST_FROM_ASCII (or: structure descriptions in JS broken in Okteta 0.7.0)

Following the advice to use QT_NO_CAST_FROM_ASCII in the QString API dox I did so before the latest 0.7 release of the hex editor Okteta. Just to have users find out after the release that their structure descriptions done in Javascript do no longer work correctly. How that?

I relied on the compiler pointing me to all places where the implicit cast to QString no longer works/is existing if the macro QT_NO_CAST_FROM_ASCII is set. But now we found out that there are traps with this approach: at least QScriptValue has some overloads of its constructors which among string variants also includes the bool type:

    QScriptValue(bool value);
    QScriptValue(const QString &value);
    QScriptValue(const QLatin1String &value);
#ifndef QT_NO_CAST_FROM_ASCII
    QT_ASCII_CAST_WARN_CONSTRUCTOR QScriptValue(const char *value);
#endif

Now guess what the compiler prefers for code like

QScriptValue value;
// second argument to setProperty(...) is a QScriptValue
// all QScriptValue constructors are implicit
value.setProperty(name, "some string");

if QScriptValue(const char *value); no longer is available… indeed, the constructor with the bool parameter. Picks it silently and continues its work without pointing me to the issue.

This could have been prevented if I had used the macro QT_ASCII_CAST_WARNINGS before, which seems to be usable to find all places where the implicit conversion is used, by issuing warnings during compilation. But it seems undocumented, at least it is not mentioned in the QString API dox and $SEARCH_ENGINE did not show up with something as well (complain about that is up as QTBUG-20821).

The problem with the broken structure descriptions in JS has been fixed by Alex for the upcoming Okteta 0.7.1 (will be part of KDE Apps 4.7.1) by using the proper QLatin1String wrapper for any (implicit) calls to the QScriptValue constructor, so look out for that version.

All new Okteta features for KDE Apps 4.7 in a picture

KDE Apps 4.7 is out, and with it the hex editor Okteta 0.7!

For 0.6 and 0.5 I missed to give updates, like done for 0.4, 0.3, and 0.2.

So what is new since 0.4?

Not that much as in the early versions, since life has been busy for both Alex and me. So the biggest issues are yet to be solved: no smart handling of very large files (still completely loaded into memory), no storing of bookmarks. But besides that there is still nice progress:

(Eh, small glitch with the otherwise awesome new KWin in KDE Workspaces 4.7 is that shadows are not working for non-Oxygen styles, please ignore)

First thing to notice (besides the very first start of the program after updating to 0.7):

• much faster program start-up thanks to Alex’ analysis and improvements
  (what, you cannot see this in the picture? )

Other stuff:

• More support for char encoding/decoding:
  • New charset conversion tool: rewrites the bytes so the respective chars are the same as with the other charset
    (only 8-bit charsets supported, and unmatched chars are currently substituted with a value hardcoded to 0, setting is disabled)
  • New encodings: ISO-8859-14, ISO-8859-16, Codepage 874
  • Fixed: show bytes without a defined chars as such
    (missed before that some of the supported charsets are incomplete)
• Many improvements in the Structures tool by Alex:
  • Structure definitions can now also be “described” in JavaScript (see the manual)
  • Begin of a structure can be pinned to a specific offset
  • Structure definitions can be installed using KNewStuff directly e.g. from kde-files.org
  • Support for strings (various Unicode encodings) in definitions
• New formats available for “Export to file”/”Copy as”:
  • Base32 (classic, Base32Hex, z-base-32)
  • Base85 (Ascii85 for now)
  • Uuencoding (historical, Base64)
  • Xxencoding
  • S-Record (16,24,32-bit address size)
  • Intel Hex (8,16,32-bit address size)
• Added embedded terminal
  (might be useless to most, but only needed a few code lines to implement )
• Menu entries to set the count of bytes per line and per grouped bytes
• File info tool now estimates the mimetype also for the unstored/edited data in the working memory
• Bytearray views and tools are updated on change of default fixed font in System Settings

Oh, and headers for the Okteta libs and Kasten libs are finally installed since version 0.5, including the Qt Designer plugin. So if you are interested to make use of these libs in your own code, see for Kasten and OktetaKasten e.g. this blog entry (hm, Binspekt code needs updating to current API, will not compile) and the code of the plugin for KDevelop, for the Qt Designer plugin this blog entry and the example code.
But beware: the next version of the libs, coming with Okteta 0.8, will already have a different API/ABI, but that will be protected by proper namespacing.

For future versions there are still plenty of ideas for Okteta left, let’s see what gets realized

So for now “Happy bytes peeping and pushing!” also with the 0.7 version of Okteta!

Competition you would like your state’s ministries to host as well

Just came across this one while looking what is the state with bada these days, as there were no headlines recently regarding bada (or Enlightenment, about whose state I am even more curious) making it to any of the usual news sites:

“Open Source Software World Challenge 2011” is the annual competition hosted by Ministry of Knowledge and Economy of Korea. This competition is mainly intended to promote open source software and expand various exchanges among open source software developers all over the world.

Definitely something to copy, paste, adapt, compile and run by ministries around the world!

Guess they will be also happy about some programs using Qt/KDE frameworks
Seems you can register until August 15th, so maybe give it a thought.

BTW:

Though I will take the bike, not walk, should be 15 min then

Avoiding a trap with marshalling to array or map with QDBusArgument

If you are going to send custom data types over D-Bus and for this write your own marshalling and demarshalling functions (as explained in the QDBusArgument API docs),
QDBusArgument& operator<<(QDBusArgument& argument, const MyStructure& mystruct)
const QDBusArgument& operator>>(const QDBusArgument& argument, MyStructure& mystruct)
there is a nice (not documented) trap you can fall into. I did, bah…

QtDBus internally uses the first of these both methods to initially calculate the signature to which your type is marshalled. For this it passes an instance of the class, constructed with the default constructor, in our example MyStructure().

So that is why this piece of code, where I marshal a custom class with two string properties into a map with a key of type string and a value of type string, does not work with QtDBus (will fail similar for creating an array):

QDBusArgument& operator<<( QDBusArgument& argument, const MyStructure& mystruct )
{
    static QString typeKey = QLatin1String("type");
    static QString dataKey = QLatin1String("data");
    argument.beginMap( QVariant::String, QVariant::String );
        argument.beginMapEntry();
            argument << typeKey << mystruct.type();
        argument.endMapEntry();
        argument.beginMapEntry();
            argument << dataKey << mystruct.data();
        argument.endMapEntry();
    argument.endMap();

    return argument;
}

you will get an error like this in the log:
QDBusMarshaller: type `MyStructure' produces invalid D-BUS signature `a{ss}ssss' (Did you forget to call beginStructure() ?)
Strange I thought, as this code should be kind of an unrolled version what e.g. the template code for QMap looks like:

template
inline QDBusArgument &operator<<(QDBusArgument &arg, const QMap &map)
{
    int kid = qMetaTypeId();
    int vid = qMetaTypeId();
    arg.beginMap(kid, vid);
    typename QMap::ConstIterator it = map.constBegin();
    typename QMap::ConstIterator end = map.constEnd();
    for ( ; it != end; ++it) {
        arg.beginMapEntry();
        arg << it.key() << it.value();
        arg.endMapEntry();
    }
    arg.endMap();
    return arg;
}

Well, it works for QMap, as with an empty QMap the for-loop will not be reached, so arg << it.key() << it.value(); will not be executed and thus not result in additional bogus info appended to the signature string.

So the solution for my code was to add an empty/invalid state to MyStructure and to set any instance to that if constructed with the default constructor, along with wrapping the map entry insertion with a check for isValid():

QDBusArgument& operator<<( QDBusArgument& argument, const MyStructure& mystruct )
{
    static QString typeKey = QLatin1String("type");
    static QString dataKey = QLatin1String("data");
    argument.beginMap( QVariant::String, QVariant::String );
    if( mystruct.isValid() )
    {
        argument.beginMapEntry();
            argument << typeKey << mystruct.type();
        argument.endMapEntry();
        argument.beginMapEntry();
            argument << dataKey << mystruct.data();
        argument.endMapEntry();
    }
    argument.endMap();

    return argument;
}

Yes, need to file a bug with Qt about this missing in the QDBusArgument docs, yet to be done.

Introducing KDE Games to fullscreen and touch interfaces @ Blue Wonder 2011

Two weeks ago I experimented with packaging the KDE Games for MeeGo and tried the results on the give-away Lenovo Ideapad from the MeeGo conference last november in Dublin with the MeeGo Netbook 1.1 installation. To find out that most games are quite usable even with only the touch-screen as input device, while some are not, but almost all are in desperate need for proper fullscreen support, for a full fun experience.

Remembering about the then upcoming KDE Games sprint Blue Wonder 2011 last weekend I quickly decided I should join this and start to do something about that (especially as Dresden is only a 2 h train ride away from Berlin and this also meant visiting my former home town and most fellows from the KDE developer community in Dresden). Thanks to Josef Spillner for being a host (and a good one) for me on that short notice Thanks also to my fellow workers Jon and Mathias at Openismus, they lent their Ideapads to me for the sprint.

While my plan was to work, for having a template, on my favourite game Palapeli, the really nice jigsaw puzzle game from Stefan Majewsky, I quickly was occupied with trying all the games, to gather some general information what works and what does not work. Do you know how many KDE games there are, only in the module kdegames? I did not, and was surprised about the number! That was my little blue wonder, and so I followed the sprint’s topic and may have played^Winvestigated all the games more than half a day in total (had to leave out ksirk and granatier for packaging problems)

So from my notes with regard to touch-screen input I found…
… these 15 games working out-of-the-box: Bovo, kdiamond, kfourinline, kigo, kiriki, kjumpingcube, klickety, klines, kmahjongg, konquest, kpat, kreversi, kshisen, ksquares, lskat.
They all only take either a LMB click (press&release at once) events on some coordinate or a LMB press, move, LMB release sequence as input. Which a tap or a swipe on the touchscreen is mapped to 1:1, besides by covering slightly more of the screen with the finger and the rest of the hand than the mouse cursor does.
… these 4 games working basically, but could gain from more support of touchscreens: kbreakout, kblackbox, killbots, ksudoku. The last three miss the highlighting of the rows/columns as triggered by hovering with the mouse, while kbreakout works but feels like that is only by accident. Beware: kbreakout will steal you more time than you planned to play it. It really works too well in fact
… these other games to need work to be usable with a touchscreen:
Bomber, Granatier, Kapman, katomic, kbattleship, kgoldrunner, kmines, knetwalk, kolf, kollision, ksirk, ktron, ktuberling. What their (partially slight) problems are and how these could be approached will be hopefully topic of a future post.

With regard to making full use of the available screen, there was work to be done for almost all. Being developed along all the other work-oriented KDE programs it seemed obvious that for consistency they also get the usual menubar, the usual toolbar and the usual statusbar. Which all three steal much space in the vertical. Now are most screens more wide than high, while most games which are resembled by the KDE game programs have a rather quadratic form factor. The result: big unused spaces to the left and to the right, e.g. with KPatience:

Just see how much bigger the actual content gets if the window bar and the menu bar are removed:

The button to toggle between fullscreen and normal window you see in the toolbar in the screenshots is not part of the current program, I created an experimental patch to add this button to almost all games, which for now is only applied to the sources used to build the MeeGo packages of the KDE Games on the Public/Community MeeGo Build Service. I would like to improve this by having that button aligned to the right side of the toolbar, as it rather acts on a different level than the other buttons (I also wonder how much overlapping with the functionality of the windowmanager is here. What is the semantical difference between maximized window and fullscreen?).
Now imagine that toolbar also being disappeared, only sliding in on demand, e.g. by tapping on the upper boarder of the screen, like it is done to get the MeeGo Netbook 1.1 system bar. I think of something like Okular has when in presentation mode. The buttons of the toolbar could also be shown as overlays on a tap in some free space, if there is plenty of that in the game board. And then there are gestures, but those are yet to arrive in the Qt API IIRC, so not yet considered.

To have proper support for going fullscreen is especially needed for small-size form factors like netbooks and handhelds, with and without touch-input (but then even more, given your fingertip is quite large, unless you use a sharpener, well, don’t! ). But also for bigger screens it can be quite a pleasure to see just the actual content and not all the clutter around it, which is usually of zero interest when you work on the content. I still remember when something like two decades ago (was it Word Perfect? MS Word?) I really enjoyed the fullscreen option where everything but the virtual paper was removed from the screen, so the actual content could be rendered with as much pixels as the physical screen has, and the screen was as pure as could be. Similar to what these days programs like FocusWriter are also specialising on.

Update: See the blog posts from Peter Penz and Martin Gräßlin from the day before on a related discussion about what to do with the menu bar, i.e. how to fix the cluttering it creates and how to gain the screen estate it takes.

Kudos to the developers of KDE Games and the artists responsible for the pretty graphics, the KDE Games are really some nice shiny diamonds. And addictive, as my flat-mate just proved when I showed to her what I worked on last week-end

Thanks also to the organizers of the sprint, Josef and Stefan, as well as to the supporters, as represented by the KDE e.V.