Tmux scripting

September 25, 2011 5 comments

I usually need to run more than 1 command for some project and got tired of searching through those putty windows for the session I want. So I thought of using a terminal multiplexer like Tmux.

I’m using celery with two queues and I need to run this:

  • manage.py celeryd -Q queueA
  • manage.py celeryd -Q queueB
  • manage.py celerycam -E

I need celerycam because it will get those stats in djcelery up to date.

It’s also a good idea to tail the postgresql log – when you break your models or database in some cases Django isn’t very helpful so this helps a lot:

  • tail -f /var/log/postgresql/postgresql-8.4-main.log

I use a wide screen so I want a layout like this:

    +------------------------------------+-------------------+
    |                                    |                   |
    |              runserver             |                   |
    |                                    |     celerycam     |
    +------------------------------------+                   |
    |                                    |                   |
    |               celeryd              +-------------------+
    |                                    |                   |
    +------------------------------------+                   |
    |                                    |   postgresql log  |
    |               celeryd              |                   |
    |                                    |                   |
    +------------------------------------+-------------------+

I also want to start a new tmux session from the same command so I can close everything easily – those celeryd’s don’t reload automatically :(

You’d usually run something like:

tmux new-session "tmux splitw 'command1';  tmux splitw 'command3'; tmux splitw 'command3'; command4"

but that get’s rather long and you need to quote and escape, calculate the panel sizes manually (I want equal height) and for the layout above you also need to select the right panels before splitting.

The commands vary across projects (some have more and some have less) – so how about we make a script:

import subprocess

left_commands = [
    "python manage.py runserver",
    "python manage.py celeryd -Q queueA -c 2 -E -n worker1",
    "python manage.py celeryd -Q queueB -c 2 -E -n worker2",
]
right_commands = [
    "python manage.py celerycam",
    "tail -f /var/log/postgresql/postgresql-8.4-main.log",
]
session = ''

if right_commands:
    session += 'tmux selectp -t 0; tmux splitw -hd -p 35 \"%s\"; ' % right_commands[-1]
for index, command in enumerate(right_commands[:-1]):
    session += 'tmux selectp -t 1; tmux splitw -d -p %i \"%s\"; ' % (
        100 / (len(right_commands) - index),
        command
    )

for index, command in enumerate(left_commands[1:]):
    session += 'tmux selectp -t 0; tmux splitw -d -p %i \"%s\"; ' % (
        100 / (len(left_commands) - index),
        command
    )
if left_commands:
    session += left_commands[0]

args = [
    'tmux',
    'new-session',
    session,
]
print 'Running ', args
subprocess.call(args)

Measure your code

July 3, 2011 9 comments

I found a cool tool today to measure code: metrics. It measures SLOC, comments and cyclomatic complexity. It’s easy to install: pip install pygments metrics

Run this in your project’s root:

metrics -v `find . -type f \( -iname "*.css" -or -iname "*.py" -or -iname "*.js" -or -iname "*.html" -or -iname "*.txt" \) \! -path "*/migrations/*.py" -print`

I have a django project so I added \! -path "*/migrations/*.py" to skip any files that are in a migrations dir (I’d skip the automatically generated south migrations).

You probably bundle other libraries or apps in your source tree (eg: jquery or that nice django app the author didn’t bother to make a setup.py script for) so you want to measure only some specific paths. Eg, to collect stats only for files in src/foobar, lib/tools and src/otherbar:

metrics -v `find src/foobar lib/tools src/otherbar -type f \( -iname "*.css" -or -iname "*.py" -or -iname "*.js" -or -iname "*.html" -or -iname "*.txt" \) \! -path "*/migrations/*.py" -print`

If you work on multiple projects you can make a script or alias for this:

metrics -v `find \`cat METRICS\` -type f \( -iname "*.css" -or -iname "*.py" -or -iname "*.js" -or -iname "*.html" -or -iname "*.txt" \) \! -path "*/migrations/*.py" -print`

And in each project just save a METRICS file with the list of paths.

I get something like this for one random project:

Metrics Summary:
Files                       Language        SLOC Comment McCabe
----- ------------------------------ ----------- ------- ------
  129                         Python        4831     289    261
    2                      Text only           0       0      0
   49              HTML+Django/Jinja        1381      19    166
    7                     JavaScript        2204     231    352
   21                            CSS        1839     111      0
----- ------------------------------ ----------- ------- ------
  208                          Total       10255     650    779

Do McCabe (aka cyclomatic complexity) ratios (McCabe/(SLOC-Comment)) look odd ? (0.17 for javascript and and 0.05 for python).

Do you know other measurement tools adequate for Django projects ? Do tell me and, if possible, how to run it for specific files like above (I’m lazy :)

Custom app names in the django admin

June 24, 2011 19 comments

EDIT:┬áThis approach is flawed – it will never work in the app_index page (/admin/appname/) and can cause problems with contenttypes queries. You are better off overriding the admin templates (lots of them unfortunately). To avoid hardcoding the app_label in the templates wrap it in trans tags and use the internationalization framework to map the internal name to your desired display name.

EDIT 2: You can also use this.

Suppose you have a model like this:

class Stuff(models.Model):
    class Meta:
        verbose_name = u'The stuff'
        verbose_name_plural = u'The bunch of stuff'

    ...

You have verbose_name, however you want to customise app_label too for different display in admin. Unfortunatelly having some arbitrary string (with spaces) doesn’t work and it’s not for display anyway.

Turns out that the admin uses app_label.title() for display so we can make a little hack: str subclass with overriden title method:

class string_with_title(str):
    def __new__(cls, value, title):
        instance = str.__new__(cls, value)
        instance._title = title
        return instance

    def title(self):
        return self._title

    __copy__ = lambda self: self
    __deepcopy__ = lambda self, memodict: self

Now we can have the model like this:

class Stuff(models.Model):
    class Meta:
        app_label = string_with_title("stuffapp", "The stuff box")
        # 'stuffapp' is the name of the django app
        verbose_name = 'The stuff'
        verbose_name_plural = 'The bunch of stuff'

    ...

and the admin will show “The stuff box” as the app name.

Tags: ,

Mercurial tip of the day: tortoisehg’s shelve

June 22, 2011 Comments off

Shelved your local changes using tortoisehg’s shelve UI and now you feel silly because tortoisehg won’t let you apply them because the first hunk is rejected ? No need to feel silly any more, just add this in .hgrc (or mercurial.ini for windows users):

[extensions]
tortoisehg.util.hgshelve=

This is a shelve extension bundled with tortoisehg. Other shelve extensions don’t use the same store as tortoisehg.

Now just run:

hg unshelve --force

The rejected hunks will be ignored (you still get the .rej files).

Tags:

Stale formsets and the back button

March 17, 2011 2 comments

I have a problem on one of my projects:

I have a page with a form that depends on what is stored in the database. If your using django formsets or have some form that saves over multiple objects you have this problem too.

The user saves the form, data gets saved. However, if the user uses the back button he will get a page with the old form (that expects different data in the database). If the form gets resubmitted all kinds of problems may appear.

I had one case when you could get a ValueError: invalid literal for int() with base 10: '' if you resubmit a formset but instead of a existing object you have a new one. Easy to pull off by a regular user if he has multiple tabs opened.

The best solution, I think, is to reload the page when the users goes back in history. Turns out this is easy to pull off with some http headers:

Cache-Control: no-cache, must-revalidate, no-store
Pragma: no-cache

The “no-store” option actually makes browses re-request when using the back button. Also, I’ve seen people adding “post-check=0, pre-check=0″ to Cache-Control. Do NOT use those. They are Microsoft extensions to the http protocol, and if set they will actually make Internet Explorer request the page two times ! see this.

Here’s a simple view decorator if you’re using django:

from django.utils.decorators import wraps
def must_revalidate(func):
    @wraps(func)
    def wrapper(*args, **kwargs):
        response = func(*args, **kwargs)
        response["Cache-Control"] = "no-cache, must-revalidate, no-store"
        response["Pragma"] = "no-cache"
        return response
    return wrapper

Having no-store creates some additional load on the server and creates other user experience problems:

  • Users won’t have the old form data when they go back
  • Page scroll position isn’t kept
  • If the page has other state it isn’t kept – but depending on the browser that state might not be cached anyway (eg: dom changes)

But I think it’s still better than surprising the user with different results for a submission with the same form data or having to deal consistently with missing or extra data in the database on the server-side. What do you think ?

Tags: , ,

Wondering about django orm caching frameworks

August 25, 2010 2 comments

So briefly looking over the code reveals that:

  • johnny-cache will cache the rows returned by the execution machinery in django’s sql compiler (monkey-patches the compilers). It looks like it has fancy-pants invalidation (it basically has bulk invalidation through 2-tiered cache key scheme, unlike cache-machine with relies on set_many) and even support for transactions. I’m using this and it’s awesome.
  • django-cache-machine will cache the result of the QuerySet.iterator method. It seems that it has some limitations: it only (automatically) invalidates on forward relations (FKs) so you have to perform carefull invalidation through your code (eg: you use qs.update(), run queries through models without the custom CachingManager, use Model.create() and whatnot …). Also, cache-machine will be heavy on the memcached traffic (1 call for every invalidated object, using set_many though …)
  • django-cachebot will cache the rows on the same level as cache-machine (at QuerySet.iterator call). Also, it has a very nice feature that will prefetch objects from reverse relations (like FK reverse descriptors and many to many relations – eg: Group.objects.select_reverse('user_set') and then group.user_set_cache will be equal to group.user_set.all()). Unfortunately the author only tested it on django 1.1 and it needs a django patch to work (the django manager patch is only for 1.1). I really like that select_reverse feature – unfortunately I can’t use this on django 1.2 :(

So I’m thinking what I need is johnny-cache for the low level stuff and then some cache-machine to cache some of that “select_reverse” feng-shui that I would have to do myself.

Well, I’m probably missing something here and some other people should have better comparisons for these frameworks. Any feedback ?

Edit: cachebot’s select_reverse is based on django-selectreverse, however it’s enhanced to support nested reverse relations (eg, from the docs: Article.objects.select_reverse('book_set','book_set__publisher_set'))

Tags: ,

The final literals vs constructors argument

February 4, 2010 6 comments
>>> def literal():
...     a = {}
...     b = []
... 
>>> def builtinconstructor():
...     a = dict()
...     b = list()
... 
>>> import dis
>>> dis.dis(literal)
  3           0 BUILD_MAP                0
              3 STORE_FAST               0 (a)

  4           6 BUILD_LIST               0
              9 STORE_FAST               1 (b)
             12 LOAD_CONST               0 (None)
             15 RETURN_VALUE        
>>> dis.dis(builtinconstructor)
  3           0 LOAD_GLOBAL              0 (dict)
              3 CALL_FUNCTION            0
              6 STORE_FAST               0 (a)

  4           9 LOAD_GLOBAL              1 (list)
             12 CALL_FUNCTION            0
             15 STORE_FAST               1 (b)
             18 LOAD_CONST               0 (None)
             21 RETURN_VALUE        
>>> 

Seriously now, is there any reason to use constructors instead of literals for empty dicts/lists ?

Tags:
Follow

Get every new post delivered to your Inbox.

Join 172 other followers