Measure your code

I found a cool tool today to measure code: metrics. It measures SLOC, comments and cyclomatic complexity. It’s easy to install: pip install pygments metrics

Run this in your project’s root:

metrics -v `find . -type f \( -iname "*.css" -or -iname "*.py" -or -iname "*.js" -or -iname "*.html" -or -iname "*.txt" \) \! -path "*/migrations/*.py" -print`

I have a django project so I added \! -path "*/migrations/*.py" to skip any files that are in a migrations dir (I’d skip the automatically generated south migrations).

You probably bundle other libraries or apps in your source tree (eg: jquery or that nice django app the author didn’t bother to make a setup.py script for) so you want to measure only some specific paths. Eg, to collect stats only for files in src/foobar, lib/tools and src/otherbar:

metrics -v `find src/foobar lib/tools src/otherbar -type f \( -iname "*.css" -or -iname "*.py" -or -iname "*.js" -or -iname "*.html" -or -iname "*.txt" \) \! -path "*/migrations/*.py" -print`

If you work on multiple projects you can make a script or alias for this:

metrics -v `find \`cat METRICS\` -type f \( -iname "*.css" -or -iname "*.py" -or -iname "*.js" -or -iname "*.html" -or -iname "*.txt" \) \! -path "*/migrations/*.py" -print`

And in each project just save a METRICS file with the list of paths.

I get something like this for one random project:

Metrics Summary:
Files                       Language        SLOC Comment McCabe
----- ------------------------------ ----------- ------- ------
  129                         Python        4831     289    261
    2                      Text only           0       0      0
   49              HTML+Django/Jinja        1381      19    166
    7                     JavaScript        2204     231    352
   21                            CSS        1839     111      0
----- ------------------------------ ----------- ------- ------
  208                          Total       10255     650    779

Do McCabe (aka cyclomatic complexity) ratios (McCabe/(SLOC-Comment)) look odd ? (0.17 for javascript and and 0.05 for python).

Do you know other measurement tools adequate for Django projects ? Do tell me and, if possible, how to run it for specific files like above (I’m lazy :)

Custom app names in the django admin

EDIT: This approach is flawed – it will never work in the app_index page (/admin/appname/) and can cause problems with contenttypes queries. You are better off overriding the admin templates (lots of them unfortunately). To avoid hardcoding the app_label in the templates wrap it in trans tags and use the internationalization framework to map the internal name to your desired display name.

EDIT 2: You can also use this.

Suppose you have a model like this:

class Stuff(models.Model):
    class Meta:
        verbose_name = u'The stuff'
        verbose_name_plural = u'The bunch of stuff'

    ...

You have verbose_name, however you want to customise app_label too for different display in admin. Unfortunatelly having some arbitrary string (with spaces) doesn’t work and it’s not for display anyway.

Turns out that the admin uses app_label.title() for display so we can make a little hack: str subclass with overriden title method:

class string_with_title(str):
    def __new__(cls, value, title):
        instance = str.__new__(cls, value)
        instance._title = title
        return instance

    def title(self):
        return self._title

    __copy__ = lambda self: self
    __deepcopy__ = lambda self, memodict: self

Now we can have the model like this:

class Stuff(models.Model):
    class Meta:
        app_label = string_with_title("stuffapp", "The stuff box")
        # 'stuffapp' is the name of the django app
        verbose_name = 'The stuff'
        verbose_name_plural = 'The bunch of stuff'

    ...

and the admin will show “The stuff box” as the app name.

Mercurial tip of the day: tortoisehg’s shelve

Shelved your local changes using tortoisehg’s shelve UI and now you feel silly because tortoisehg won’t let you apply them because the first hunk is rejected ? No need to feel silly any more, just add this in .hgrc (or mercurial.ini for windows users):

[extensions]
tortoisehg.util.hgshelve=

This is a shelve extension bundled with tortoisehg. Other shelve extensions don’t use the same store as tortoisehg.

Now just run:

hg unshelve --force

The rejected hunks will be ignored (you still get the .rej files).

Stale formsets and the back button

I have a problem on one of my projects:

I have a page with a form that depends on what is stored in the database. If your using django formsets or have some form that saves over multiple objects you have this problem too.

The user saves the form, data gets saved. However, if the user uses the back button he will get a page with the old form (that expects different data in the database). If the form gets resubmitted all kinds of problems may appear.

I had one case when you could get a ValueError: invalid literal for int() with base 10: '' if you resubmit a formset but instead of a existing object you have a new one. Easy to pull off by a regular user if he has multiple tabs opened.

The best solution, I think, is to reload the page when the users goes back in history. Turns out this is easy to pull off with some http headers:

Cache-Control: no-cache, must-revalidate, no-store
Pragma: no-cache

The “no-store” option actually makes browses re-request when using the back button. Also, I’ve seen people adding “post-check=0, pre-check=0″ to Cache-Control. Do NOT use those. They are Microsoft extensions to the http protocol, and if set they will actually make Internet Explorer request the page two times ! see this.

Here’s a simple view decorator if you’re using django:

from django.utils.decorators import wraps
def must_revalidate(func):
    @wraps(func)
    def wrapper(*args, **kwargs):
        response = func(*args, **kwargs)
        response["Cache-Control"] = "no-cache, must-revalidate, no-store"
        response["Pragma"] = "no-cache"
        return response
    return wrapper

Having no-store creates some additional load on the server and creates other user experience problems:

• Users won’t have the old form data when they go back
• Page scroll position isn’t kept
• If the page has other state it isn’t kept – but depending on the browser that state might not be cached anyway (eg: dom changes)

But I think it’s still better than surprising the user with different results for a submission with the same form data or having to deal consistently with missing or extra data in the database on the server-side. What do you think ?

Wondering about django orm caching frameworks

So briefly looking over the code reveals that:

• johnny-cache will cache the rows returned by the execution machinery in django’s sql compiler (monkey-patches the compilers). It looks like it has fancy-pants invalidation (it basically has bulk invalidation through 2-tiered cache key scheme, unlike cache-machine with relies on set_many) and even support for transactions. I’m using this and it’s awesome.
• django-cache-machine will cache the result of the QuerySet.iterator method. It seems that it has some limitations: it only (automatically) invalidates on forward relations (FKs) so you have to perform carefull invalidation through your code (eg: you use qs.update(), run queries through models without the custom CachingManager, use Model.create() and whatnot …). Also, cache-machine will be heavy on the memcached traffic (1 call for every invalidated object, using set_many though …)
• django-cachebot will cache the rows on the same level as cache-machine (at QuerySet.iterator call). Also, it has a very nice feature that will prefetch objects from reverse relations (like FK reverse descriptors and many to many relations – eg: Group.objects.select_reverse('user_set') and then group.user_set_cache will be equal to group.user_set.all()). Unfortunately the author only tested it on django 1.1 and it needs a django patch to work (the django manager patch is only for 1.1). I really like that select_reverse feature – unfortunately I can’t use this on django 1.2 :(

So I’m thinking what I need is johnny-cache for the low level stuff and then some cache-machine to cache some of that “select_reverse” feng-shui that I would have to do myself.

Well, I’m probably missing something here and some other people should have better comparisons for these frameworks. Any feedback ?

Edit: cachebot’s select_reverse is based on django-selectreverse, however it’s enhanced to support nested reverse relations (eg, from the docs: Article.objects.select_reverse('book_set','book_set__publisher_set'))

The final literals vs constructors argument

>>> def literal():
...     a = {}
...     b = []
... 
>>> def builtinconstructor():
...     a = dict()
...     b = list()
... 
>>> import dis
>>> dis.dis(literal)
  3           0 BUILD_MAP                0
              3 STORE_FAST               0 (a)

  4           6 BUILD_LIST               0
              9 STORE_FAST               1 (b)
             12 LOAD_CONST               0 (None)
             15 RETURN_VALUE        
>>> dis.dis(builtinconstructor)
  3           0 LOAD_GLOBAL              0 (dict)
              3 CALL_FUNCTION            0
              6 STORE_FAST               0 (a)

  4           9 LOAD_GLOBAL              1 (list)
             12 CALL_FUNCTION            0
             15 STORE_FAST               1 (b)
             18 LOAD_CONST               0 (None)
             21 RETURN_VALUE        
>>> 

Seriously now, is there any reason to use constructors instead of literals for empty dicts/lists ?

The “yield from” proposal

There’s a bit of buzz on the python-ideas mailing list about Greg Ewing’s proposal for a new keyword “yield from“. He has made a draft PEP and patch for python 2.6.1.

On short it adds syntactic sugar for generator decomposition and adds a “return value” statement in generators (the underlying mechanism for that is a StopIteration(value) exception).

Here’s a straightforward example:

>>> def g1():
...     print "Starting g1"
...     yield "g1 ham"
...     result = yield from g2()
...     print "Got result: %r from g2" % result
...     yield "g1 eggs"
...     print "Finishing g1"
...
>>> def g2():
...     print "Starting g2"
...     yield "g2 spam"
...     yield "g2 more spam"
...     print "Finishing g2"
...     return "foobar"
...
>>> for x in g1():
...     print "Yielded", x
...
Starting g1
Yielded g1 ham
Starting g2
Yielded g2 spam
Yielded g2 more spam
Finishing g2
Got result: 'foobar' from g2
Yielded g1 eggs
Finishing g1

There are a bunch of other examples and tests bundled with Greg’s patch.

I got the patch and it works :). Had to manually rebuild the parser – make didn’t do it by itself and I had to run “make Parser/pgen & make Python/graminit.c & make“. I wonder why.