Tag Archives: utils

How to restrict the length of a unicode string

Ha, not with me!
It’s a pretty common tripwire: Imagine you have a unicode string and for whatever reason (which should be a good reason, so make sure you really need this) you need to make sure that its UTF-8 representation has at most maxsize bytes.
The first and in this case worst attempt is probably unicodeStr[:maxsize], as its UTF-8 representation could be up to 6 times as long.
So the next worse attempt could be this unicode(unicodeStr.encode("utf-8")[:maxsize], "utf-8"): This could cut a multi-byte UTF-8 representation of a codepoint in half (example: unicode(u"jörn".encode("utf-8")[:2], "utf-8")). Luckily python will tell you by throwing a UnicodeDecodeError.

The last attempt actually wasn’t that wrong, as it only lacked the errors="ignore" flag:

unicode(myUnicodeStr.encode("utf-8")[:maxsize], "utf-8", errors="ignore")

One might think we’re done now, but this depends on your Unicode Normalization Form: Unicode allows Combined Characters, for example the precomposed u"ü" could be represented by the decomposed sequence u"u" and u"¨" (see Unicode Normalization).
In my case I know that my unicode strings are in Unicode Normalization Form C (NFC) (at least the RDF Literal Specs say so. This means that if there is a precomposed char it will be used. Nevertheless Unicode potentially allows for Combined characters which do not have a precomposed canonical equivalent. In this case not even normalizing would help, multiple unicode chars would remain, leading to multiple multi-byte UTF-8 chars. In this case I’m unsure what’s the universal solution… for such a u”ü” is it better to have a u”u” or nothing in case of a split? You have to decide.
I decided for having an “u” in the hopefully very rare case this occurs.
So use the following with care:

def truncateUTF8length(unicodeStr, maxsize):
    ur""" This method can be used to truncate the length of a given unicode
        string such that the corresponding utf-8 string won't exceed
        maxsize bytes. It will take care of multi-byte utf-8 chars intersecting
        with the maxsize limit: either the whole char fits or it will be
        truncated completely. Make sure that unicodeStr is in Unicode
        Normalization Form C (NFC), else strange things can happen as
        mentioned in the examples below.
        Returns a unicode string, so if you need it encoded as utf-8, call
        .decode("utf-8") after calling this method.
        >>> truncateUTF8lengthIfNecessary(u"ö", 2) == (u"ö", False)
        >>> truncateUTF8length(u"ö", 1) == u""
        >>> u'u1ebf'.encode('utf-8') == 'xe1xbaxbf'
        >>> truncateUTF8length(u'hiu1ebf', 2) == u"hi"
        >>> truncateUTF8lengthIfNecessary(u'hiu1ebf', 3) == (u"hi", True)
        >>> truncateUTF8length(u'hiu1ebf', 4) == u"hi"
        >>> truncateUTF8length(u'hiu1ebf', 5) == u"hiu1ebf"
        Make sure the unicodeStr is in NFC (see unicodedata.normalize("NFC", ...) ).
        The following would not be true, as e and u'u0301' would be seperate
        unicode chars. This could be handled with unicodedata.combining
        and a loop deleting chars from the end until after the first non
        combining char, but this is _not_ done here!
        #>>> u'eu0301'.encode('utf-8') == 'exccx81'
        #>>> truncateUTF8length(u'eu0301', 0) == u"" # not in NFC (u'xe9'), but in NFD
        #>>> truncateUTF8length(u'eu0301', 1) == u"" #decodes to utf-8:
        #>>> truncateUTF8length(u'eu0301', 2) == u""
        #>>> truncateUTF8length(u'eu0301', 3) == u"eu0301"

    return unicode(unicodeStr.encode("utf-8")[:maxsize], "utf-8", errors="ignore")

Unicode and UTF-8 is nice, but if you don’t pay attention it will cause your code to contain a lot of sleeping bugs. And yes, probably I’d care less if there was no “ö” in my name ;)

PS: Günther, this is SFW. :p

How to convert hex strings to binary ascii strings in python (incl. 8bit space)

As this comes across may way again and again:

How do you turn a hex string like "c3a4c3b6c3bc" into a nice binary string like this: "11000011 10100100 11000011 10110110 11000011 10111100"?

The solution is based on the Python 2.6 new string formatting:

>>> "{0:8b}".format(int("c3",16))

Which can be decomposed into 4 bit for each hex char like this: (notice the 04b, which means 0-padded 4chars long binary string):

>>> "{0:04b}".format(int("c",16)) + "{0:04b}".format(int("3",16))

OK, now we could easily do this for all hex chars "".join(["{0:04b}".format(int(c,16)) for c in "c3a4c3b6"]) and done, but usually we want a blank every 8 bits from the right to left… And looping from the right pairwise is a bit more complicated… Oh and what if the number of bits is uneven?
So the solution looks like this:

>>> binary = lambda x: " ".join(reversed( [i+j for i,j in zip( *[ ["{0:04b}".format(int(c,16)) for c in reversed("0"+x)][n::2] for n in [1,0] ] ) ] ))
>>> binary("c3a4c3b6c3bc")
'11000011 10100100 11000011 10110110 11000011 10111100'

It takes the hex string x, first of all concatenates a "0" to the left (for the uneven case), then reverses the string, converts every char into a 4-bit binary string, then collects all uneven indices of this list, zips them to all even indices, for each in the pairs-list concatenates them to 8-bit binary strings, reverses again and joins them together with a ” ” in between. In case of an even number the added 0 falls out, because there’s no one to zip with, if uneven it zips with the first hex-char.

Yupp, I like 1liners ;)

Bash prompt indicating return value

Lately I’ve fiddled a lot with installing virtuoso on some virtual machines and found myself repeatedly asking bash for the return value of the last command echo $?. I remembered this blog post by Gecko quite a while ago and tuned it a bit to my needs. My user prompt now looks like this (it’s a slightly modified version of the old gentoo defaults, that I prefer over the ubuntu defaults, which only remind you that you’re root with a # instead of a $):

The code for the user prompt:

 PS1='${debian_chroot:+($debian_chroot)}[\033[01;32m]u@h[\033[00m]:[\033[01;34m]w $(if [[ $? = 0 ]];then echo "[\033[01;34m]";else echo "[\033[01;31m]";fi)$ [\033[00m]'

And the root prompt:

 PS1='[\033[01;31m]h[\033[01;34m] W $(if [[ $? = 0 ]];then echo "[\033[01;34m]";else echo "[\033[01;31m]";fi)$[\033[00m] '

And a small doc snippet you might want to include with your PS1 in your .bashrc, so you can still understand that cryptic stuff in a few days:

# h hostname
# u user
# w working dir (home == ~)
# $ a $ if UID == 0 else #
# A current 24-time: HH:MM
# \ a
# [ begin, ] end of non-printing (control chars)
# colors: have to be surrounded by: '[\033[' and 'm]' (without the ''. These literally are the left and right bracket!)
#  color codes are as follows, preceeded by a '0;' (dark) (default) or a '1;' (light)
#  FG and BG NULL: 00 resets
#  FG: 30 + ... 0: black, 1: red, 2: green, 3: yellow, 4: blue, 5: violet, 6: cyan, 7: white
#  BG: 40 + ... same

(URL)Encoding in python

Well, encodings are a never ending story and whenever you don’t want to waste time on them, it’s for sure that you’ll stumble over yet another tripwire. This time it is the encoding of URLs (note: even though related I’m not talking about the urlencode function). Perhaps you have seen something like this before:
http://de.wikipedia.org/wiki/Gerhard_Schr%C3%B6der which actually is the URI pendant to this IRI: http://de.wikipedia.org/wiki/Gehard_Schröder

Now what’s the problem, you might ask. The problem is that two things can happen here:
Either your browser (or the library you use) thinks: “hmm, this 'ö' is strange, let’s convert it into a '%C3%B6'” or your browser (or lib) doesn’t care and asks the server with the 'ö' in the URL, introducing a bit of non-determinism into your expectations, right?

More details here:

$ curl -I http://de.wikipedia.org/wiki/Gerhard_Schröder
HTTP/1.0 200 OK
Date: Thu, 22 Jul 2010 09:41:56 GMT
Last-Modified: Wed, 21 Jul 2010 11:50:31 GMT
Content-Length: 144996
Connection: close
$ curl -I http://de.wikipedia.org/wiki/Gerhard_Schr%C3%B6der
HTTP/1.0 200 OK
Date: Sat, 31 Jul 2010 00:24:47 GMT
Last-Modified: Thu, 29 Jul 2010 10:04:31 GMT
Content-Length: 144962
Connection: close

Notice how the Date, Last-Modified and Content-Length differ.

OK, so how do we deal with this? I’d say: let’s always ask for the “percentified” version… but before try to understand this:

# notice that my locale is en.UTF-8
>>> print "jörn"
>>> "jörn" # implicitly calls: print repr("jörn")
>>> print repr("jörn")
>>> u"jörn"
>>> print u"jörn"
>>> print u"jörn".encode("utf8")
>>> u"jörn".encode("utf8")
>>> "jörn".encode("utf8")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 1: ordinal not in range(128)

So, what happened here?
As my locale is set to use UTF-8 encoding, all my inputs are utf-8 encoded already.
If until now you might have wondered, why 'ö' is translated into '%C3%B6', you might have spotted that 'ö' corresponds to the utf-8 "xc3xb6", which actually is python’s in string escape sequence for non-ASCII chars: it refers to 2 bytes with the hex-code: c3b6 (binary: '11000011 10110110') (quite useful: "{0:b} {1:b}".format(int("c3", 16), int("b6",16))).
So in URLs these "xhh" are simply replaced by "%HH", so a percent and two uppercase ASCII-Chars indicating a hex-code. The unicode 'ö' (1 char, 1byte, unicode "xf6" ('11110110')) hence is first transformed into utf-8 (1char, 2byte, utf8: '11000011 10110110') by my OS, before entering it into python, internally kept in this form unless I use the u"" strings, and then represented in the URL with "%C3%B6" (6chars, 6byte, ASCII).
What this example also shows is the implicit print repr(var) performed by the interactive python interpreter when you simply enter some var and hit return.
Print will try to convert strings to the current locale if they’re Unicode-Strings (u""). Else python will not assume that the string has any specific encoding, but just stick with the encoding your OS chose. It will simply treat the string as it was received and write the byte-sequence to your sys.stdout.

So back to the manual quoting of URLs:

>>> import urllib as ul
>>> ul.quote("jörn")
>>> print ul.quote("jörn")

>>> ul.unquote('j%C3%B6rn')
>>> ul.unquote("jörn")
>>> print ul.unquote("jörn")


Just recently came across the python itertools “tools for efficient looping” again. Generators have the advantage of not creating the whole list on definition, but on demand (in contrast to e.g., list comprehensions). Really worth a look:

import itertools as it
g = it.cycle("abc") # a generator
g.next() # a
g.next() # b
g.next() # c
g.next() # a
g.next() # b
# ... and so on

g = it.cycle("abcde")
h = it.cycle("1234")
gh = it.izip(g,h) # iterzips n iterables together
gh.next() # (a,1)
gh.next() # (b,2)
# ... think about what this means with primes
gh.next() # (e,4)
gh.next() # (a,1)
# ...

Also very nice are the combinatoric generators:

it.product('ABCD',  repeat=2) # AA AB AC AD BA BB BC BD
                              # CA CB CC CD DA DB DC DD
it.permutations('ABCD',  2)   # AB AC AD BA BC BD CA CB CD DA DB DC
it.combinations('ABCD',  2)   # AB AC AD BC BD CD
it.combinations_with_replacement('ABCD',  2) # AA AB AC AD BB BC BD CC CD DD

Precision-Recall diagrams including F-Measure height lines

Today I was asked how to generate Recall-Precision diagrams including the f-measure values as height-lines from within python. Actually Gunnar was the one who had this idea quite a while ago, but constantly writing things into files, then loading them with his R code to visualize them, made me create a “pythonified” version. Looks like this (click for large version):

Here’s my python code-snippet for it: recallPrecision.py.

Uses a bunch of pylab internally, but after simply importing this module, you can easily visualize a list of (precision, recall) points like this:

import scipy as sc
import pylab as pl
import recallPrecision as rp
prs = sc.rand(15,2) # precision recall point list
labels = ["item " + str(i) for i in range(15)] # labels for the points
rp.plotPrecisionRecallDiagram("footitle", prs, labels)

(updated on 2014-10-28: link to gist)

Sort python dictionaries by values

Perhaps you already encountered a problem like the following one yourself:
You have a large list of items (let’s say URIs for this example) and want to sum up how often they were viewed (or edited or… whatever). A small one-shot solution in python looks like the following and uses the often unknown operator.itemgetter:

import sys
import operator
uriViews = {}
for line in sys.stdin:
    uri, views = line.strip().split()
    views = int(views)
    uriViews[uri] = uriViews.get(uri, 0) - views
    # why minus? could do + and use reverse=True below?
    # right, but then items also get sorted in reverse, if views are
    # the same (so Z would come first, a last)
for uri, views in sorted(uriViews.items(),
    print -views, uri

This approach can be a lot faster than self written lambda functions called for every comparison or a list comprehension which turns around all tuples and then sorts it. Also in contrast to many other solutions you can find online this one uses operator.itemgetter(1,0), which means that if two items have the same amount of views, their URIs are sorted alphabetically.

Remember that this approach sorts the whole list in RAM, so you might want to chose other approaches in case your lists are getting huge.
For further information you might want to read PEP-0265, which also includes a hint what to do when you’re only interested in the Top 1000 for example (will only sort these top1000):

import heapq
top1000 = heapq.nlargest(1000, uriViews.iteritems(), itemgetter(1,0))
for uri,views in top1000:
   print -views, uri

Min-Heap in Python

I recently wanted to implement a small event system where events can have different priorities. So for example the event with highest priority (lowest value) should be handled first.
Python comes with a heapq module which can transform a list into a heap in a way that it stays a list, but fulfills all heap constraints. Nevertheless you might want to wrap the heap like this, so you can do nice stuff:

import heapq
class Heap(object):
    """ A neat min-heap wrapper which allows storing items by priority
        and get the lowest item out first (pop()).
        Also implements the iterator-methods, so can be used in a for
        loop, which will loop through all items in increasing priority order.
        Remember that accessing the items like this will iteratively call
        pop(), and hence empties the heap! """

    def __init__(self):
        """ create a new min-heap. """
        self._heap = []
    def push(self, priority, item):
        """ Push an item with priority into the heap.
            Priority 0 is the highest, which means that such an item will
            be popped first."""

        assert priority >= 0
        heapq.heappush(self._heap, (priority, item))
    def pop(self):
        """ Returns the item with lowest priority. """
        item = heapq.heappop(self._heap)[1] # (prio, item)[1] == item
        return item
    def __len__(self):
        return len(self._heap)
    def __iter__(self):
        """ Get all elements ordered by asc. priority. """
        return self
    def next(self):
        """ Get all elements ordered by their priority (lowest first). """
            return self.pop()
        except IndexError:
            raise StopIteration

With this one could do the following:

h = Heap()
# add some nonsense:
h.push(10, "I'm a large one")
h.push(20, "Actually I'm larger")
h.push(5, "size is not everything")
h.push(0, "which size?")

# get it out in a cool way:
for item in h:
   print item

# prints:
# which size?
# size is not everything
# I'm a large one
# Actually I'm larger