The first thing to do is to always use the legacy network adapter. If you use the normal one and activate the hv_netvsc module the vm won’t boot anymore (some kernel failure at boot). It will look like this:

Now, the first thing to do is disable vga16fb in the install: in the install menu press F6 and then add “vga16fb.modeset=0″ to the boot options. This will speed up the display a lot. Eg:

After the install disable again: echo blacklist vga16fb | sudo tee -a /etc/modprobe.d/blacklist-framebuffer.conf

Ubuntu 10.04 has the IC modules but they are not active by default. They aren’t even supported but they seem to work. Do the standard kernel module dance:

in /etc/initramfs-tools/modules add:

hv_vmbus
hv_storvsc
hv_blkvsc
hv_netvsc

and run update-initramfs –u

Make sure you take a snapshot before rebooting (just in case) !

Bonus:

If you get this piix4_smbus 0000:00:07.0: SMBus base address uninitialized - upgrade BIOS or use force_addr=0xaddr

Then sudo nano /etc/modprobe.d/blacklist.conf, add: blacklist i2c_piix4 and run update-initramfs -u again.

Well anyway, the Network and Sharing Center in Windows 8 is quite weird … could not change the network type (to private) for one virtual Hyper-V adapter even tho I’ve enabled all user control with gpedit.msc. To fix it I had to use the Network List Manager API’s SetCategory it seems. There was this PowerShell script that seem to do that but, damn it, I don’t want to edit the script every time I have the problem. And that PowerShell thing looks awful compared to Python …

Here’s a python script that does the right thing: asks the user what network should be made private.

import win32com.client
NETWORK_CATEGORIES = {
    1: "PRIVATE",
    0: "PUBLIC",
    2: "DOMAIN"
}
m = win32com.client.Dispatch("{DCB00C01-570F-4A9B-8D69-199FDBA5723B}")
more = 1
pos = 1
connections = m.GetNetworkConnections()

while more:
    connection, more = connections.Next(pos)
    if connection:
        network = connection.GetNetwork()
        category = network.GetCategory()
        print '%s. "%s" is %s' % (pos, network.GetName(), NETWORK_CATEGORIES[category])
        if not category and raw_input("Make private [N]") in ['y', 'Y']:
            network.SetCategory(1)
    pos += 1

Now isn’t this pretty ? (except the shitty iterator-wanna-be api in GetNetworkConnections …)

On the server and client, download and extract somewhere jmeter:

wget http://www.apache.org/dist/jmeter/binaries/apache-jmeter-2.6.tgz
tar -xf apache-jmeter-2.6.tgz
cd apache-jmeter-2.6

On the client edit bin/jmeter.properties and add:

remote_hosts=127.0.0.1:55501
client.rmi.localport=55512
mode=Batch
num_sample_threshold=250

On the server edit bin/jmeter.properties and add:

server_port=55501
server.rmi.localhostname=127.0.0.1
server.rmi.localport=55511

Now connect to the server using this:

ssh -L 55501:127.0.0.1:55501 -L 55511:127.0.0.1:55511 -R 55512:127.0.0.1:55512 user@hostname

If you’re using putty it should look like this:

Now run:

bin/jmeter-server -Djava.rmi.server.hostname=127.0.0.1

And on the client run:

bin/jmeter.sh -Djava.rmi.server.hostname=127.0.0.1
# or
bin/jmeter.sh -Djava.rmi.server.hostname=127.0.0.1 -t /path/to/test-plan.jmx

Now you can use the remote node !

Editable foreign keys in the changelist

If you have foreign keys in list_editable django will make 1 database query for each item in the changelist. Quite a lot for a changelist of 100 items. The trick is to cache the choices for that formfield:

class MyAdmin(admin.ModelAdmin):
    list_editable = 'myfield',
    def formfield_for_dbfield(self, db_field, **kwargs):
        request = kwargs['request']
        formfield = super(MyAdmin, self).formfield_for_dbfield(db_field, **kwargs)
        if db_field.name == 'myfield':
            myfield_choices_cache = getattr(request, 'myfield_choices_cache', None)
            if myfield_choices_cache is not None:
                formfield.choices = myfield_choices_cache
            else:
                request.myfield_choices_cache = formfield.choices
        return formfield

Foreign keys or many to many fields in admin inlines

If you have fk of m2m fields on InlineModelAdmin for every object in the formset you’ll get a database hit. You can avoid this by having something like:

class MyAdmin(admin.TabularInline):
    fields = 'myfield',
    def formfield_for_dbfield(self, db_field, **kwargs):
        formfield = super(MyAdmin, self).formfield_for_dbfield(db_field, **kwargs)
        if db_field.name == 'myfield':
            # dirty trick so queryset is evaluated and cached in .choices
            formfield.choices = formfield.choices
        return formfield

Enable template caching

It’s amazing how easy it is to forget to add this in your settings:

TEMPLATE_LOADERS = (
    ('django.template.loaders.cached.Loader', (
        'django.template.loaders.filesystem.Loader',
        'django.template.loaders.app_directories.Loader',
    )),
)

Use select_related for the edit forms too

In case you have some readonly fields on the edit form and they need related data to display list_select_related doesn’t help. Eg:

class MyAdmin(admin.ModelAdmin):
    readonly_fields = 'myfield',
    def queryset(self, request):
        return super(MyAdmin, self).queryset(request).select_related('myfield')

Use annotations if possible for function entries list_display instead of making additional queries

Check the aggregation api to see if you can use this. Here’s the typical example:

class AuthorAdmin(admin.ModelAdmin):
    list_display = 'books_count',
    
    def books_count(self, obj):
        return obj.books_count
    
    def queryset(self, request):
        return super(AuthorAdmin, self).queryset(
            request).annotate(books_count=Count('books'))

The models would look like this:

class Author(models.Model):
    name = models.CharField(max_length=100)

class Book(models.Model):
    name = models.CharField(max_length=100)
    author = models.ForeignKey(Author, related_name="books")

Cache the filters from the admin changelist

This has the obvious tradeoff that you’ll have stale data in the list of filter but if they don’t change that often and those distinct queries are killing your database then this will help a lot. Just add a custom change_list.html template in your project (eg: templates/<myapp>/change_list.html):

{% extends "admin/change_list.html" %}
{% load admin_list i18n cache %}

{% block filters %}
    {% cache 300 admin_filters request.GET.items request.path request.user.username %}
        {% if cl.has_filters %}
          <div id="changelist-filter">
            <h2>{% trans 'Filter' %}</h2>
            {% for spec in cl.filter_specs %}{% admin_list_filter cl spec %}{% endfor %}
          </div>
        {% endif %}
    {% endcache %}
{% endblock %}

Bonus trick

frame = sys._getframe(1)
while frame:
    if frame.f_code.co_name == 'render_change_form':
        if 'request' in frame.f_locals:
            request = frame.f_locals['request']
            break
    frame = frame.f_back
else:
    raise RuntimeError("Could not find request object.")

# do stuff with request

This could be used in some specific cases (eg: you need the request in a widget’s render method), as a last resort ofcourse ;)

–

What did you do to make django admin faster ?

You can use revokes for aborting tasks but they don’t give you enough control and it’s not guaranteed that your tasks will stop gracefully (or stop at all). Revokes can raise SoftTimeLimitExceeded if enabled (via TERM signal) however it might be tricky to perform cleanup – if you call C extension the exception will get delayed till the call returns. See the signal module docs for what happens when you raise an exception from a signal handler (that’s what celery does).

Given this, an alternative is to use redis to store the aborted task ids in a redis set. If you use the redis broker you can use this drop-in replacement:

from contextlib import contextmanager
import celery
from celery.task.base import Task
from celery.result import AsyncResult

from django.conf import settings

assert settings.BROKER_TRANSPORT == 'redis', "AbortableTask can only work with a 'redis' BROKER_TRANSPORT"
REDIS_KEY = getattr(settings, 'ABORTABLE_REDIS_KEY', 'task-aborts')

@contextmanager
def client_from_pool():
    connection = celery.current_app.pool.acquire()
    try:
        yield connection.default_channel.client
    finally:
        connection.release()

class AbortableAsyncResult(AsyncResult):

    def is_aborted(self):
        with client_from_pool() as client:
            return client.sismember(REDIS_KEY, self.task_id)

    def abort(self):
        with client_from_pool() as client:
            client.sadd(REDIS_KEY, self.task_id)

class AbortableTask(Task):

    @classmethod
    def AsyncResult(cls, task_id):
        return AbortableAsyncResult(task_id, backend=cls.backend,
                                             task_name=cls.name)

    def is_aborted(self, **kwargs):
        task_id = kwargs.get('task_id', self.request.id)
        with client_from_pool() as client:
            return client.sismember(REDIS_KEY, task_id)

    def cleanup(self, **kwargs):
        task_id = kwargs.get('task_id', self.request.id)
        with client_from_pool() as client:
            client.srem(REDIS_KEY, task_id)

    def after_return(self, status, retval, task_id, args, kwargs, einfo):
        self.cleanup(task_id=task_id)

This will use the broker’s connection pool if enabled (you should enable it, just set BROKER_POOL_LIMIT).

Put this in your project’s urls.py:

from django.utils.functional import curry
from django.views.defaults import server_error, page_not_found

handler500 = curry(server_error, template_name='admin/500.html')
handler404 = curry(page_not_found, template_name='admin/404.html')

I wonder why django doesn’t mention those templates in the docs …

If you have other drop-in apps that need authentication (like rosetta or sentry) bare in mind that the admin doesn’t have a reusable login view so you must hook one. You should just reuse django admin’s login template. Put this in the urlpatterns (don’t forget to match it to LOGIN_URL in the settings):

    url(r'^accounts/login/$', 'django.contrib.auth.views.login', {'template_name': 'admin/login.html'}),

You might note that this is not very DRY but actually the LOGIN_URL might differ than the one in the urlpatterns (eg: you mount the django wsgi handler on a non-root path).

I’m using celery with two queues and I need to run this:

• manage.py celeryd -Q queueA
• manage.py celeryd -Q queueB
• manage.py celerycam -E

I need celerycam because it will get those stats in djcelery up to date.

It’s also a good idea to tail the postgresql log – when you break your models or database in some cases Django isn’t very helpful so this helps a lot:

• tail -f /var/log/postgresql/postgresql-8.4-main.log

I use a wide screen so I want a layout like this:

    +------------------------------------+-------------------+
    |                                    |                   |
    |              runserver             |                   |
    |                                    |     celerycam     |
    +------------------------------------+                   |
    |                                    |                   |
    |               celeryd              +-------------------+
    |                                    |                   |
    +------------------------------------+                   |
    |                                    |   postgresql log  |
    |               celeryd              |                   |
    |                                    |                   |
    +------------------------------------+-------------------+

I also want to start a new tmux session from the same command so I can close everything easily – those celeryd’s don’t reload automatically :(

You’d usually run something like:

tmux new-session "tmux splitw 'command1';  tmux splitw 'command3'; tmux splitw 'command3'; command4"

but that get’s rather long and you need to quote and escape, calculate the panel sizes manually (I want equal height) and for the layout above you also need to select the right panels before splitting.

The commands vary across projects (some have more and some have less) – so how about we make a script:

import subprocess

left_commands = [
    "python manage.py runserver",
    "python manage.py celeryd -Q queueA -c 2 -E -n worker1",
    "python manage.py celeryd -Q queueB -c 2 -E -n worker2",
]
right_commands = [
    "python manage.py celerycam",
    "tail -f /var/log/postgresql/postgresql-8.4-main.log",
]
session = ''

if right_commands:
    session += 'tmux selectp -t 0; tmux splitw -hd -p 35 \"%s\"; ' % right_commands[-1]
for index, command in enumerate(right_commands[:-1]):
    session += 'tmux selectp -t 1; tmux splitw -d -p %i \"%s\"; ' % (
        100 / (len(right_commands) - index),
        command
    )

for index, command in enumerate(left_commands[1:]):
    session += 'tmux selectp -t 0; tmux splitw -d -p %i \"%s\"; ' % (
        100 / (len(left_commands) - index),
        command
    )
if left_commands:
    session += left_commands[0]

args = [
    'tmux',
    'new-session',
    session,
]
print 'Running ', args
subprocess.call(args)

Run this in your project’s root:

metrics -v `find . -type f \( -iname "*.css" -or -iname "*.py" -or -iname "*.js" -or -iname "*.html" -or -iname "*.txt" \) \! -path "*/migrations/*.py" -print`

I have a django project so I added \! -path "*/migrations/*.py" to skip any files that are in a migrations dir (I’d skip the automatically generated south migrations).

You probably bundle other libraries or apps in your source tree (eg: jquery or that nice django app the author didn’t bother to make a setup.py script for) so you want to measure only some specific paths. Eg, to collect stats only for files in src/foobar, lib/tools and src/otherbar:

metrics -v `find src/foobar lib/tools src/otherbar -type f \( -iname "*.css" -or -iname "*.py" -or -iname "*.js" -or -iname "*.html" -or -iname "*.txt" \) \! -path "*/migrations/*.py" -print`

If you work on multiple projects you can make a script or alias for this:

metrics -v `find \`cat METRICS\` -type f \( -iname "*.css" -or -iname "*.py" -or -iname "*.js" -or -iname "*.html" -or -iname "*.txt" \) \! -path "*/migrations/*.py" -print`

And in each project just save a METRICS file with the list of paths.

I get something like this for one random project:

Metrics Summary:
Files                       Language        SLOC Comment McCabe
----- ------------------------------ ----------- ------- ------
  129                         Python        4831     289    261
    2                      Text only           0       0      0
   49              HTML+Django/Jinja        1381      19    166
    7                     JavaScript        2204     231    352
   21                            CSS        1839     111      0
----- ------------------------------ ----------- ------- ------
  208                          Total       10255     650    779

Do McCabe (aka cyclomatic complexity) ratios (McCabe/(SLOC-Comment)) look odd ? (0.17 for javascript and and 0.05 for python).

Do you know other measurement tools adequate for Django projects ? Do tell me and, if possible, how to run it for specific files like above (I’m lazy :)

EDIT: This approach is flawed – it will never work in the app_index page (/admin/appname/) and can cause problems with contenttypes queries. You are better off overriding the admin templates (lots of them unfortunately). To avoid hardcoding the app_label in the templates wrap it in trans tags and use the internationalization framework to map the internal name to your desired display name.

EDIT 2: You can also use this.

Suppose you have a model like this:

class Stuff(models.Model):
    class Meta:
        verbose_name = u'The stuff'
        verbose_name_plural = u'The bunch of stuff'

    ...

You have verbose_name, however you want to customise app_label too for different display in admin. Unfortunatelly having some arbitrary string (with spaces) doesn’t work and it’s not for display anyway.

Turns out that the admin uses app_label.title() for display so we can make a little hack: str subclass with overriden title method:

class string_with_title(str):
    def __new__(cls, value, title):
        instance = str.__new__(cls, value)
        instance._title = title
        return instance

    def title(self):
        return self._title

    __copy__ = lambda self: self
    __deepcopy__ = lambda self, memodict: self

Now we can have the model like this:

class Stuff(models.Model):
    class Meta:
        app_label = string_with_title("stuffapp", "The stuff box")
        # 'stuffapp' is the name of the django app
        verbose_name = 'The stuff'
        verbose_name_plural = 'The bunch of stuff'

    ...

and the admin will show “The stuff box” as the app name.

[extensions]
tortoisehg.util.hgshelve=

This is a shelve extension bundled with tortoisehg. Other shelve extensions don’t use the same store as tortoisehg.

Now just run:

hg unshelve --force

The rejected hunks will be ignored (you still get the .rej files).

I have a page with a form that depends on what is stored in the database. If your using django formsets or have some form that saves over multiple objects you have this problem too.

The user saves the form, data gets saved. However, if the user uses the back button he will get a page with the old form (that expects different data in the database). If the form gets resubmitted all kinds of problems may appear.

I had one case when you could get a ValueError: invalid literal for int() with base 10: '' if you resubmit a formset but instead of a existing object you have a new one. Easy to pull off by a regular user if he has multiple tabs opened.

The best solution, I think, is to reload the page when the users goes back in history. Turns out this is easy to pull off with some http headers:

Cache-Control: no-cache, must-revalidate, no-store
Pragma: no-cache

The “no-store” option actually makes browses re-request when using the back button. Also, I’ve seen people adding “post-check=0, pre-check=0″ to Cache-Control. Do NOT use those. They are Microsoft extensions to the http protocol, and if set they will actually make Internet Explorer request the page two times ! see this.

Here’s a simple view decorator if you’re using django:

from django.utils.decorators import wraps
def must_revalidate(func):
    @wraps(func)
    def wrapper(*args, **kwargs):
        response = func(*args, **kwargs)
        response["Cache-Control"] = "no-cache, must-revalidate, no-store"
        response["Pragma"] = "no-cache"
        return response
    return wrapper

Having no-store creates some additional load on the server and creates other user experience problems:

• Users won’t have the old form data when they go back
• Page scroll position isn’t kept
• If the page has other state it isn’t kept – but depending on the browser that state might not be cached anyway (eg: dom changes)

But I think it’s still better than surprising the user with different results for a submission with the same form data or having to deal consistently with missing or extra data in the database on the server-side. What do you think ?

• johnny-cache will cache the rows returned by the execution machinery in django’s sql compiler (monkey-patches the compilers). It looks like it has fancy-pants invalidation (it basically has bulk invalidation through 2-tiered cache key scheme, unlike cache-machine with relies on set_many) and even support for transactions. I’m using this and it’s awesome.
• django-cache-machine will cache the result of the QuerySet.iterator method. It seems that it has some limitations: it only (automatically) invalidates on forward relations (FKs) so you have to perform carefull invalidation through your code (eg: you use qs.update(), run queries through models without the custom CachingManager, use Model.create() and whatnot …). Also, cache-machine will be heavy on the memcached traffic (1 call for every invalidated object, using set_many though …)
• django-cachebot will cache the rows on the same level as cache-machine (at QuerySet.iterator call). Also, it has a very nice feature that will prefetch objects from reverse relations (like FK reverse descriptors and many to many relations – eg: Group.objects.select_reverse('user_set') and then group.user_set_cache will be equal to group.user_set.all()). Unfortunately the author only tested it on django 1.1 and it needs a django patch to work (the django manager patch is only for 1.1). I really like that select_reverse feature – unfortunately I can’t use this on django 1.2 :(

So I’m thinking what I need is johnny-cache for the low level stuff and then some cache-machine to cache some of that “select_reverse” feng-shui that I would have to do myself.

Well, I’m probably missing something here and some other people should have better comparisons for these frameworks. Any feedback ?

Edit: cachebot’s select_reverse is based on django-selectreverse, however it’s enhanced to support nested reverse relations (eg, from the docs: Article.objects.select_reverse('book_set','book_set__publisher_set'))

Seriously now, is there any reason to use constructors instead of literals for empty dicts/lists ?

On short it adds syntactic sugar for generator decomposition and adds a “return value” statement in generators (the underlying mechanism for that is a StopIteration(value) exception).

Here’s a straightforward example:

>>> def g1():
...     print "Starting g1"
...     yield "g1 ham"
...     result = yield from g2()
...     print "Got result: %r from g2" % result
...     yield "g1 eggs"
...     print "Finishing g1"
...
>>> def g2():
...     print "Starting g2"
...     yield "g2 spam"
...     yield "g2 more spam"
...     print "Finishing g2"
...     return "foobar"
...
>>> for x in g1():
...     print "Yielded", x
...
Starting g1
Yielded g1 ham
Starting g2
Yielded g2 spam
Yielded g2 more spam
Finishing g2
Got result: 'foobar' from g2
Yielded g1 eggs
Finishing g1

There are a bunch of other examples and tests bundled with Greg’s patch.

I got the patch and it works :). Had to manually rebuild the parser – make didn’t do it by itself and I had to run “make Parser/pgen & make Python/graminit.c & make“. I wonder why.

The best part about this is that cogen could support any sort of old code that uses the usual file or socket apis in the standard python library.

Unfortunately greenlets don’t work with python 2.6 right now; let me shamelessly quote this very nice reply I got from one of py.lib’s (greenlets are part of py.lib) maintainers:

“I mean come on – it’s open source, if it works for me on 2.5 why I would bother too much?”

For the love of god, is this the right attitude to run a project these days ?! I mean come on …

Anyhow, nevermind that.

So back to those coroutines using greenlets – I named them corolets – feels like a good name given the fact the the regular generator stuff is named with coroutine and there’s some need for distinction. I also added some socket wrappers in a module socketlets (not very inspired). It has a Socket class that provides regular blocking interface and makes use of the good old _fileobject class from the socket module in stdlib for the makefile stuff.

I got those 2 pieces in cogen.magic.corolets and cogen.magic.socketlets. For those who didn’t understood much here’s an example:

import sys

from cogen.core import schedulers
from cogen.magic.corolets import corolet, yield_
from cogen.magic import socketlets

@corolet
def server():
    srv = socketlets.Socket()
    adr = ('0.0.0.0', len(sys.argv)>1 and int(sys.argv[1]) or 1200)
    srv.bind(adr)
    srv.listen(64)
    while 1:
        print "Listening on", adr
        conn, addr = srv.accept()
        print "Connection from %s:%s" % addr
        m.add(handler, args=(conn, addr))

@corolet
def handler(sock, addr):
    fh = sock.makefile()
    fh.write("WELCOME TO ECHO SERVER !\r\n")
    fh.flush()

    while 1:
        line = fh.readline(1024)
        if line.strip() == 'exit':
            fh.write("GOOD BYE")
            fh.close()
            sock.close()
            return
        fh.write(line)
        fh.flush()

m = schedulers.Scheduler()
m.add(server)
m.run()

Compare that to the regular generator based example (don’t miss those yields):

import sys

from cogen.core import sockets
from cogen.core import schedulers
from cogen.core.coroutines import coroutine

@coroutine
def server():
    srv = sockets.Socket()
    adr = ('0.0.0.0', len(sys.argv)>1 and int(sys.argv[1]) or 1200)
    srv.bind(adr)
    srv.listen(64)
    while 1:
        print "Listening on", adr
        conn, addr = yield srv.accept()
        print "Connection from %s:%s" % addr
        m.add(handler, args=(conn, addr))

@coroutine
def handler(sock, addr):
    fh = sock.makefile()
    yield fh.write("WELCOME TO ECHO SERVER !\r\n")
    yield fh.flush()

    while 1:
        line = yield fh.readline(1024)
        if line.strip() == 'exit':
            yield fh.write("GOOD BYE")
            yield fh.close()
            sock.close()
            return
        yield fh.write(line)
        yield fh.flush()

m = schedulers.Scheduler()
m.add(server)
m.run()

It’s pretty rough right now – I still need to add tests and check the performance, but I wanted to give an idea of what’s going on for the braver ones :)

Writing the wrappers wasn’t that hard – but if you aren’t careful with ctypes you are in a world of gruesome crashes and at least 2 wasted evenings (read nights) debugging why the hell some random allocation fails. Well, for a fact, I still have no idea to why do I own those to wasted nights but thank god it doesn’t crash on my unittests anymore – I think I could have figured out why some missing object cleanup resulted in weird crashes if I knew some ctypes internals but eh, I don’t :(

Take a look at the evil code here and here, and punish me for doing such an evil thing with ctypes *malefic laugh* =]

Here’s a gem:

#~ BOOL = SOCKET hSocket, HANDLE hFile, DWORD nNumberOfBytesToWrite, DWORD nNumberOfBytesPerSend, LPOVERLAPPED lpOverlapped,LPTRANSMIT_FILE_BUFFERS lpTransmitBuffers, DWORD dwFlags
TransmitFileType = WINFUNCTYPE(BOOL, SOCKET, HANDLE, DWORD, DWORD, POINTER(OVERLAPPED), POINTER(TRANSMIT_FILE_BUFFERS), DWORD)

def _GetTransmitFilePtr(given_socket=None):
    from socket import socket
    bogus_sock = given_socket or socket()
    bogus_bytes = DWORD()
    TransmitFile = TransmitFileType(0)
    ret = WSAIoctl(
        bogus_sock.fileno(), SIO_GET_EXTENSION_FUNCTION_POINTER, byref(WSAID_TRANSMITFILE), sizeof(WSAID_TRANSMITFILE),
        byref(TransmitFile), sizeof(TransmitFile), byref(bogus_bytes), None, None
    )
    return TransmitFile

TransmitFile = _GetTransmitFilePtr()

So far the new proactor works pretty nice in cogen.

You can write the client code like this:

    var s = new FlashSocket({
        on_data: function(data) {
            document.getElementById('datapanel').value = data;
        },
        on_io_error: function(msg) {
            alert("IO ERROR: "+msg);
        },
        on_security_error: function(msg) {
            alert("SECURITY ERROR: "+msg);
        },
        on_close: function(msg) {
            alert("Connection closed.");
        },
        on_connect: function() {
            s.write("GET / HTTP/1.0\r\n\r\n");
        }
    });
    s.connect('example.com', 80);

Here‘s the whole example.

How does it work? You can execute arbitrary javascript code though the external interface, so why not execute some boilerplate code and setup a global FlashSocket class that handles all the nasty actionscript-javascript communication.  I’ve seen this arguably good idea in the SWFHttpRequest bridge. If you do something like this be careful with those damn commas/semicolons or else you’re in i-dont-know-why-it-breaks hell :)

Here’s how the guts look like (note that the language is haxe):

        ExternalInterface.addCallback("connect", connect);
        ExternalInterface.addCallback("close", close);
        ExternalInterface.addCallback("write", write);
        ExternalInterface.addCallback("CAN_I_HAS_SOCKET", CAN_I_HAS_SOCKET);
        ExternalInterface.call([
        "(function(){",
            "if (window.FlashSocket) return;",
            "var Class = function(properties){",
                "var klass = function(event_handlers){ ",
                    "for (var p in event_handlers) {",
                        "if (event_handlers.hasOwnProperty(p)) {",
                            "this[p] = event_handlers[p];",
                        "}",
                    "}",
                    "return this.init.apply(this);",
                "};",
                "klass.prototype = properties;",
                "klass.constructor = arguments.callee;",
                "return klass;",
            "};",
            "window.FlashSocket = new Class({",
                "init: function(){",
                    "this._instance = ''+window.FlashSocket._instances.length;",
                    "window.FlashSocket._instances.push(this);",
                "},",
                "close: function(){ ",
                    "window.FlashSocket._instances[this._instance] = null;",
                    "window.FlashSocket._bridge.close(this._instance );",
                "},",
                "write: function(data){ ",
                    "window.FlashSocket._bridge.write(this._instance, data);",
                "},",
                "connect: function(host, port) {",
                    "window.FlashSocket._bridge.connect(this._instance, host, port);",
                "}",
            "});",
            "window.FlashSocket._instances = [];",
            "var f = function(tag){",
                "var elems = document.getElementsByTagName(tag);",
                "for (var i=0; i<elems.length; i++) if (elems[i].CAN_I_HAS_SOCKET) return elems[i];",
            "};",
            "window.FlashSocket._bridge = f('embed') || f('object');",
        "})" ].join('') );
        if (flash.Lib.current.loaderInfo.parameters.onloadcallback != null)
            ExternalInterface.call(flash.Lib.current.loaderInfo.parameters.onloadcallback);

The trick is that we can setup a global javascript class-like object from the flash clip.

You can grab the code here. Also, if you test from a local disc you might want to adjust your flash security settings. To compile the .hx file run “haxe compile.hxml“.

Now, the only thing I miss is some buffering/readline logic.

Say for example, if you do:

from setuptools import setup
setup(
...
install_requires=["Pylons==dev"],
...
)

It won’t work:

Installed d:\python25\lib\site-packages\pylons-0.9.7rc3dev_20081103-py2.5.egg
Reading http://www.pylonshq.com/download/0.9.7
error: Could not find required distribution Pylons==dev

I imagine it would match if the package would literally have the version “dev”.

One solution is to change the install_requires to ["Pylons==dev,!=foo"] or some other bogus version to match whatever is downloaded for the dev version. “>=0.1.2.3” will match for packages that don’t have tagged versions (won’t work for example for Pylons “0.9.7beta3dev-20080422″).

EDIT: Looks like I was wrong. having “!=foo” will work if you already have the package installed but will fail to find the dev package when downloading. Having “>=0.9.7″ in the requirements will not work because “0.9.7beta3dev-20080422″ is considered a lower version (prerelease to the final 0.9.7). A solution would be having a requirement like “>=0.9.7a”. “a” will be lower than “beta3dev-20080422″.

On short, functions have the __get__ method that make them work as descriptors when they are in a class.

I can do this:

>>> class Bar:
...     pass
...
>>> bar = Bar()
>>>
>>> def foo():
...     pass
...
>>> foo
<function foo at 0x00B6F270>
>>>
>>> foo.__get__(None, Bar)
<unbound method Bar.foo>
>>>
>>> foo.__get__(bar, Bar)
<bound method Bar.foo of <__main__.Bar instance at 0x00BCB738>>

Neat !

The runtime is completely different. They do try however to patch the normal python runtime in the sdk to have the same characteristics as the runtime on their servers but differences fall through the cracks. The have some sort of qa process for the sdk (probably to compensate for these differences) – I bet it must be very cumbersome to maintain it.

In the sdk you have so little python libraries available and what makes matters worse is that they are outdated so you have to do additional evil monkey patching to use newer versions.

Ok, so I started with the idea of a django app – but I gave up when i found the one in the sdk is very old (0.96) and i still need to zip it (django slightly goes over the 1000 file limit). So if I start zipping stuff I might just as well do it the right way and use eggs and setuptools – well, I’ve moved to that idea and still didn’t work out due to the fact setuptools can barely run in the appengine runtime.

Also, thinking on the zips in retrospective – they aren’t such a great idea. Why the hell would i waste cpu cycles on zips ?

And speaking of the 1000 file limit, their filesystem must work real bad with many files.  Guido said on the file limit: “it exists because of the sheer number of apps. E.g. 1,000,000 apps == 1,000,000,000 files”. Is 1B files too much ? For Google !?

Come to think of it a bit, the only advantage of appengine is the fact is almost free.

LE: corrected the arithmetic.

The only things I can think of right now are the unittests for the socket stuff – they lack good coverage and they were actually made for the previous incompatible cogen socket api. They will be fixed eventually.

Another thing is that i’m trying to push some patches in the pywin32 module for win32file.TransmitFile and win32file.ConnectEx. But it’s going so slow that I’m thinking it isn’t going to work out and I’d better write a separate module for the GetQueueCompletionStatus, CreateIoCompletionPort, WSASend, WSARecv, ConnectEx, TransmitFile and avoid all that insane overlapped struct hacking that happens in the win32file module. Wadda ya think ?

Featuring a sandbox model for tabs (tough I hope that sandbox will not be a complete process – as some speculate) and a “V8″ javascript engine – according to Joshua “code is instantly interpreted into machine code”. Damn, where did he knew that ? Really.

Surely will feature gears and other google mumbo-jumbo (some say it will even have a privacy feature ala IE8 aka the “porn mode”). I imagine it will be neat and fast but very unpolished – as the other google products launched in the “launch early and iterate” spirit. Speaking of, meh, app engine it’s still just a Toy.

I still put my money on the Shiretoko. (I hope I’m wrong tough)

I’ve made a userscript that offers me a way to bind a key combination to a clickable element chosen by a xpath expression. It still misses a way to easy edit previous bindings but I’ll provide it later, I promise!

You can still edit the bindings (well, more likely remove them) via about:config (the ‘greasemonkey.scriptvals.ShortcutBinder/Shortcut Binder.bindings’ setting).

Grab the script from here.

For a quick overview (bunch of screenshots) check this: http://code.google.com/p/webmonkey-userscripts/wiki/ShortcutBinder

Ah heck, here’s one quick and dirty:

Was it worth it ? Well, for one thing, i get a diploma – we’ll see if that really helps me at anything at all. But did it learnt me anything? Little.

Set aside the maths this faculty didn’t learn me shit. You don’t learn advanced stuff – from functional to unix programming everything is introductory stuff – you can finish the faculty with your head empty and your time wasted, easy. Just for a damn diploma.

You’ll even have to write a project for the diploma thesis, you’ll have to present it and at the end you’ll find yourself very unsatisfied because your audience will be clueless (they didn’t even know what a coroutine is, damn it !) and uninterested (they won’t even bother to read your paper that took time and effort).

I doubt working though the pile of arrogant, bored teachers – some not even deserving their degree – that don’t give a shit about you was worth giving 4 years of my life. I’m not saying not to do it – I’m saying to have very low expectations.

So here is the problem: I want to add sounds to buttons and whatever from css in a similar fasion to skinning.  I have a mess of all sorts of buttons everywhere and I don’t want to patch all of them with a mouseOverEffect, mouseUpEffect or whatever.

I would think having a <mx:SoundEffect source="@Embed(source='blabla.mp3')" id="foo" /> would make this snippet work (but it doesn’t):

Button {
    mouseOverEffect: foo; /* neither does "foo" */
}

So what to do ? StyleManager and CSSStyleDeclaration perhaps.

var myDynStyle:CSSStyleDeclaration;
myDynStyle = new CSSStyleDeclaration('Button');
myDynStyle.setStyle("mouseUpEffect", Sounds.clickButtonEffect);
StyleManager.setStyleDeclaration("Button", myDynStyle, true);

This is very cool. Allows me to set behaviors (via Effects) on whatever object classes.
Except it doesn’t work.

I’ve discovered by accident that setting those effect props to some bogus values in the css beforehand (like Button { mouseUpEffect: none; }) makes that programmatic styling work.

I wonder if this is a flex bug, though by the looks of it seems that is a compile-time vs run-time issue – some stuff that makes the effect work on the buttons is setted only on compile time?

The thought of cleaning up the socket and reactor internals really nags me. I would move the platform dependant code from the sockets module in the reactors – so, say, the reactors would have the platform specific code for recv, send, connect, accept instead of each socket operation class. Also, the fact is that the iocp was added later and the code was patched to support a what was a design scheme based on a reactor pattern – so adding support for another proactor based api (say, linux’s aio) would really mudge the internals. Also, moving the readline stuff out of the socket operations in a coroutine would make things real fancy and easy to follow.

Meh, on second thought, i would not have real gain, besides making more easy to add support for another proactor api like aio (wich btw just sucks, it wasn’t made for sockets – no support for accept or connect) and priding myself for the fancier internals i would probably get lower performance – and I still have yet to fine tune the performance part.

Now that I have relieved myself I’d better concentrate myself on that diploma thesis. =]

ClearType uses a technique named subpixel rendering that absolutely sucks – at least for me. Say for example i have some black text on white background then the subpixel rendering will make some adjacent pixels have some colors. Take a example, Consolas, a font that works only with ClearType:

That just looks painful for my eyes, i see every colored pixel in something that should had been only black and white (or gray at least) – it’s just friggin painful. If you don’t notice it (well, I might see better than you) look here:

And I’m not just running on crap hardware – I think a Dell 2407WPF-HC is good enough !

I’ve also tried to tune the ClearType a bit and it still looks like crap.

I’ve been playing recently with pylons and i’ve made a example chat app – just a proof-of-concept comet application with long pooling in a pylons app using a custom wsgi server (cogen.wsgi).

Here’s the controller code that give me a feeling I will be burned to the stake for too much magic. =]

import logging

from pylons import request, response, session
from pylons import tmpl_context as c
from pylons.controllers.util import abort, redirect_to, url_for

from chatapp.lib.base import BaseController
# import chatapp.model as model

log = logging.getLogger(__name__)
from cogen.core import queue, events
from cogen.core.coroutines import coro
from cogen.core.pubsub import PublishSubscribeQueue
pubsub = PublishSubscribeQueue()

class Client:
    def __init__(self):
        self.messages = queue.Queue(10)
        self.dead = False
    @coro
    def watch(self):
        """This is a coroutine that runs permanently for each participant to the
        chat. If the participant has more than 10 unpulled messages this
        coroutine will die.

        `pubsub` is a queue that hosts the messages from all the
        participants.
          * subscribe registers this coro to the queue
          * fetch pulls the recent messages from the queue or waits if there
        are no new ones.

        self.messages is another queue for the frontend comet client (the
        pull action from the ChatController will pop messages from this queue)
        """
        yield pubsub.subscribe()
        while 1:
            messages = yield pubsub.fetch()
            try:
                yield self.messages.put_nowait(messages)
            except:
                print 'Client %s is dead.' % self
                self.dead = True
                break
class ChatController(BaseController):

    def push(self):
        """This action puts a message in the global queue that all the clients
        will get via the 'pull' action."""
        yield request.environ['cogen.call'](pubsub.publish)(
            "%X: %s" % (id(session['client']), request.body)
        )
        # the request.environ['cogen.*'] objects are the the asynchronous
        # wsgi extensions offered by cogen - basicaly they do some magic to
        # make the code here work as a coroutine and still work with any
        # middleware
        yield str(request.environ['cogen.wsgi'].result)

    def pull(self):
        """This action does some state checking (adds a object in the session
        that will identify this chat participant and adds a coroutine to manage
        it's state) and gets new messages or bail out in 10 seconds if there are
        no messages."""
        if not 'client' in session or session['client'].dead:
            client = Client()
            print 'Adding new client:', client
            session['client'] = client
            session.save()
            yield request.environ['cogen.core'].events.AddCoro(client.watch)
        else:
            client = session['client']

        yield request.environ['cogen.call'](client.messages.get)(timeout=10)

        if isinstance(request.environ['cogen.wsgi'].result, events.OperationTimeout):
            pass
        elif isinstance(request.environ['cogen.wsgi'].result, Exception):
            import traceback
            traceback.print_exception(*request.environ['cogen.wsgi'].exception)
        else:
            yield "%s\r\n"% '\r\n'.join(request.environ['cogen.wsgi'].result)

The frontend code is fairly simple:

    <h1>chat powered by <a href="http://code.google.com/p/cogen/">cogen</a></h1>
    <form name="chat">
    <textarea id="ouput" name="output" rows="10" cols="80" readonly="readonly"></textarea><br/>
    <input id="tosend" name="tosend" value="" size="80"/>
    </form>
<script language="javascript">
    function xhr(url, callback, data) {
        var req = window.XMLHttpRequest?new XMLHttpRequest():new ActiveXObject('Microsoft.XMLHTTP');
        req.onreadystatechange = function() {
            if (req.readyState == 4) {
                if (callback) callback(req);
            }
        }
        req.open(data?"POST":"GET", url, true);
        req.send(data);
    }
    function pull(req) {
        if(req.status == 200) {
            document.chat.output.value = req.responseText + document.chat.output.value;
            xhr('/chat/pull', pull)
        } else {
            alert(req.responseText);
        }
    }
    xhr('/chat/pull', pull);
    document.chat.tosend.onkeydown = function (event) {
        event = event || window.event;
        var key = event.which || event.keyCode;
        if (key == 13) {
            xhr('/chat/push', null, document.chat.tosend.value);
            document.chat.tosend.value = '';
        }
    }
    document.chat.onsubmit = function() { return false; }
</script>

I think the javascript part looks fairly obvious, but the controller uses a bunch of magic – here’s the workflow:

• a client load the page and calls the pull page via xhr
• the pull page waits 10 sencond for a new message to be posted in the messages queue
• another client sends a message via push page via xhr
• the push page adds the message in the pubsub queue
• the coroutines (the ‘watch’ methods) are waken up and get the new message from the fetch() calls
• each ‘watch’ coroutine posts that message in the client’s queue (via messages.put_nowait)

also, the messages queue is limited to 10 messages so if there are 10 updates and a client doesn’t get them it’s associated ‘watch’ coroutine will die

• each loading pull page gets that message (via messages.get(timeout=10)) and send it to the client

The fairly obscure cogen docs might also be helpfull :)

Almost forgot, to run this you need a trunk version of cogen (easy_install cogen==dev or check out http://cogen.googlecode.com/svn/trunk/)
and the ChatApp example (you can view all the code here and get it from here)

Once you have installed cogen (easy_install or setup.py develop), install the ChatApp (setup.py develop) and start the app with paster serve test.ini. And try finding some bugs (just joking) at http://127.0.0.1:5000/

I was using nose with coverage via –with-coverage option to get some cover reports. But setup.py test borked with “setup.py: error: no such option: –cover-package”. After some digging in the sources i’ve found that some plugins aren’t compatible with setuptools’s test runner – you can’t run post test hooks with that runner apparently (sadly, the coverage plugin is one of them). I wish setuptools had a option to ignore bogus options and show only a warning – well, on second thought, that’s not setuptools related at all, the error is raised from optparse.

Luckily, setuptools has aliases for commands and instead of

[nosetests]
with-coverage = 1
cover-package = cogen

I can do:

[aliases]
test = nosetests --cover-package=cogen --with-coverage

in setup.cfg.

You whould use it like this:

from PyQt4 import QtGui

app = QtGui.QApplication([])
hello = QtGui.QPushButton( "Hello world!" )
hello.resize(100, 30)
hello.show()

from cogen.core import reactors, schedulers
m = schedulers.Scheduler(reactor=reactors.QtReactor)

def lorem_ipsum_app(environ, start_response):
    start_response('200 OK', [('Content-type','text/plain'), ('Content-Length','19')])
    return ['Lorem ipsum dolor..']

server = wsgi.WSGIServer(
  ('0.0.0.0', 9001),
  lorem_ipsum_app,
  m,
)
m.add(server.serve)
m.run()
# voila ! this actualy runs the QApplication loop with the cogen hooks

Though the reactor isn’t tested, well, unittested – i have a bunch of tests to keep the code sane.
The problem is that Qt is a big nasty monolithic framework, well, it’s nice if you don’t need to do any weird stuff – like running the main application loop in a second thread.

I just don’t understand why they made this ugly limitation in qt: the application loop must run in the main thread. I’ll fix those tests somehow, but in the meanwhile i’ll just continue cursing Qt for making me feel miserable about my unittests.

• javascripts aren’t concatenated, not even compressed – a lot of unnecessary http requests.
• static content doesn’t have proper caching headers – more unnecessary http requests.
• slow servers – i mean really, amazon aws might be cool and neato, but really slow – i just want my page loaded in 300ms.

And those are barely a scratch on the surface – there’s a lot of stuff that can be done towards a faster page.

Yes, I’m the cranky user that doesn’t like pages that load in 5seconds (yes, 5 god damn seconds!).

Sorry twitter – you suck. I hope your developers will fix you though – don’t worry, I have other stuff to waste my time on.

# history|python -c'import sys;c=[i.split()[1] for i in sys.stdin]; [sys.stdout.write("%s %s\n"%x) for x in sorted(set((c.count(i),i) for i in c), key=lambda (a,b):a)[::-1][:10]]'
198 python
71 ab
37 cat
36 /root/python/bin/python
31 cd
21 nc
21 strace
10 echo
9 history
8 easy_install

Still not shorter than the bash version:

history|awk '{a[$2]++} END{for(i in a){print a[i]" "i}}'|sort -rn|head

Just so you know, cogen is a coroutine framework (based on the bidirectional generators from python 2.5) that has a wsgi server with some async extensions.

Straight to the point, i’m going to show the basics by building the backend of a web based (with comet-style ajax) irc client. This is a proof of concept and the javascript part of the app will come in a future blog post.

So a async app with cogen works like a streaming app in pylons. Everything is fine if you don’t use middleware that consumes the appiter (sadly, the ErrorHandler middleware does just that – and it can’t be fixed becouse you can’t report a error after you sent a part of the response to the client).

Grab the latest TIP from pylons (get Mercurial first):

hg clone https://www.knowledgetap.com/hg/pylons-dev Pylons

Grab the latest trunk from paste:

svn co http://svn.pythonpaste.org/Paste/trunk Paste

Grab the latest trunk from cogen:

svn co http://cogen.googlecode.com/svn/trunk/ cogen

Ok, so install them (usualy a python setup.py develop). Next thing you make a pylons app (paster create -t pylons). And you need to make just 2 changes to de middeware config:

In middleware.py comment out the error middleware like this:

# Handle Python exceptions
#~ app = ErrorHandler(app, global_conf, **config['pylons.errorware'])

and change the Registry to work with streaming apps:

# Establish the Registry for this application
app = RegistryManager(app, streaming=True)

You need to change the development.ini, just replace the server:main section with this:

[server:main]
use = egg:cogen#http
host = 0.0.0.0
port = 5000

Allright, now add a controller (paster controller irc, or something like that). So we’ll make something extremely simple: we’ll write a relay that just passes the messages from the server to the client in json format with a timeout of 30 seconds when there no messages arrive (that’s our comet long-poll update).

Just put this code in controllers/irc.py:

import logging

from pylons import request, response, session
from pylons import tmpl_context as c
from pylons.controllers.util import abort, redirect_to, url_for
from pylons.decorators import validate
from formencode import validators
from cogenircapp.lib.base import BaseController
import cogenircapp.model as model

log = logging.getLogger(__name__)

from cogen.core.coroutines import coro, debug_coroutine
from cogen.core import events, sockets
from cogen.core.util import priority
from cogen.core import queue
from cogen.web import async

import simplejson

def parsemsg(s): # stolen from twisted.words
    """Breaks a message from an IRC server into its prefix, command, and arguments.
    """
    prefix = ''
    trailing = []
    if not s:
        raise Exception("Empty line.")
    if s[0] == ':':
        prefix, s = s[1:].split(' ', 1)
    if s.find(' :') != -1:
        s, trailing = s.split(' :', 1)
        args = s.split()
        args.append(trailing)
    else:
        args = s.split()
    command = args.pop(0)
    return prefix, command, args

class Connection:
    def __init__(self, server, reconnect_interval=60, sock_timo=15):
        self.server = server
        self.reconnect_interval = reconnect_interval
        self.connected = False
        self.sock_timo = sock_timo
        self.events = queue.Queue(25) # Max 25 pending events, well, messages
                    # from the server. After that we'll lose the connection.

    @coro
    def pull(self):
        """This coroutine handles the server connection, does a basic parse on
        the received messages and put them in a queue named events.
        The controllers pull method will take the messages from that queue.
        """
        self.sock = sockets.Socket()
        while not self.connected:
            try:
                addr = self.server.split(':')
                if len(addr) < 2:
                    addr.append(6667)
                else:
                    addr[1] = int(addr[1])
                yield self.events.put_nowait(('', 'CONNECTING', ''))
                yield self.sock.connect(tuple(addr), timeout=self.sock_timo)
                self.connected = True
            except events.OperationTimeout, e:
                yield self.events.put_nowait(('', 'CONNECT_TIMEOUT', str(e)))
                yield events.Sleep(self.reconnect_interval)

        yield self.events.put_nowait(('', 'CONNECTED', ''))
        while 1:
            try:
                line = yield self.sock.readline(8192)
                prefix, command, params = parsemsg(line)
                yield self.events.put_nowait((prefix, command, params))
            except Exception, e:
                yield self.events.put_nowait(('', 'ERROR', str(e)))
                break

from pylons.templating import render_mako as render

class IrcController(BaseController):
    """
    This controller supports multiple server connections.
    """
    def index(self):
        if 'connections' not in session:
            session['connections'] = {}
            session.save()
        return render('index.mako')

    def push(self, id):
        "Sends a message to the specified connection (id)"
        conn = session['connections'].get(id, None)
        if conn:
            yield request.environ['cogen.core'].sockets.WriteAll(conn.sock,
                            request.environ['wsgi.input'].read() +'\r\n')
            if isinstance(request.environ['cogen.wsgi'].result, Exception):
                yield simplejson.dumps(('', 'ERROR', str(e)))
            else:
                yield simplejson.dumps(('', 'PUSH_OK', ''))
        else:
            yield simplejson.dumps(('', 'ERROR', 'Invalid connection id.'))

    def connect(self, server):
        "Connects to a server and return a connection id."
        conns = session['connections']
        id = str(len(conns))
        conn = Connection(server)
        conns[id] = conn
        yield request.environ['cogen.core'].events.AddCoro(conn.pull)
        yield id

    def pull(self, id):
        """Take the messages from the queue and if there are none wait 30
        seconds till returning an empty message.

        Also, cogen's wsgi async extensions are in the environ and prefixed with
        'cogen.'
        """
        conn = session['connections'].get(id, None)
        if conn:
            ev_list = []
            while 1:
                # ok, so this might look a bit ugly but the concept is very simple
                #  you yield a special object from the environ that does some magic
                #  and the wsgi server will resume the app when it has the result
                yield request.environ['cogen.call'](conn.events.get_nowait)()
                event = request.environ['cogen.wsgi'].result
                # also, we can't have better exception handling in this wsgi
                # contraption and we need to check the result for exceptions
                if isinstance(event, queue.Empty):
                    break
                elif isinstance(event, Exception):
                    ev_list.append(('', 'ERROR', str(event)))
                    break
                else:
                    ev_list.append(event)
            if ev_list:
                yield simplejson.dumps(ev_list)
            else:
                # if we don't have any updates atm, we'll wait 30 secs for one
                yield request.environ['cogen.call'](conn.events.get)(timeout=30)
                event = request.environ['cogen.wsgi'].result
                if isinstance(event, events.OperationTimeout):
                    yield simplejson.dumps([])
                elif isinstance(event, Exception):
                    yield simplejson.dumps([('', 'ERROR', str(event))])
                else:
                    yield simplejson.dumps([event])
        else:
            yield simplejson.dumps(('', 'ERROR', 'Invalid connection id.'))

For the frontend i’m thinking to use YUI. I’ll have it finished in a couple of days.

LE: sources are at: https://cogen.googlecode.com/svn/trunk/examples/cogen-irc though the javascript interface needs polishing (a lot of it to get blog-post worthy).

Sadly they don’t work the same with unconnected sockets – i wonder why.

select (win32):

>>> import select, socket
>>> s = socket.socket()
>>> select.select([s], [s], [s], 1)
([], [], [])

select (linux):

>>> import select, socket
>>> s = socket.socket()
>>> select.select([s], [s], [s], 1)
([<socket._socketobject>], [<socket._socketobject>], [])

select (freebsd):

>>> import select, socket
>>> s = socket.socket()
>>> select.select([s], [s], [s], 1)
([], [], [])

Pretty inconsistent eh?poll (linux):

>>> import select, socket
>>> poll = select.poll()
>>> s = socket.socket()
>>> poll.register(s.fileno(), select.POLLIN | select.POLLOUT)
>>> (fd, ev), = poll.poll(1000)
>>> ev & select.POLLIN
0
>>> ev & select.POLLOUT
4
>>> ev & select.POLLERR
0
>>> ev & select.POLLHUP
16

poll (freebsd):

>>> import select, socket
>>> poll = select.poll()
>>> s = socket.socket()
>>> poll.register(s.fileno(), select.POLLIN | select.POLLOUT)
>>> (fd, ev), = poll.poll(1000)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ValueErr1r: need more than 0 values to unpack

Oh NOES!!!!1 It’s not portable.epoll:

>>> import epoll, socket
>>> efd = epoll.epoll_create(16)
>>> s = socket.socket()
>>> epoll.epoll_ctl(efd, epoll.EPOLL_CTL_ADD, s.fileno(), epoll.EPOLLIN | epoll.EPOLLOUT)
0
>>> (ev, fd), = epoll.epoll_wait(efd, 16, 1000)
>>> ev & epoll.EPOLLHUP
16
>>> ev & epoll.EPOLLOUT
4

kqueue:

>>> import kqueue, socket
>>> kq = kqueue.kqueue()
>>> s = socket.socket()
>>> kq.kevent(kqueue.EV_SET(s.fileno(), kqueue.EVFILT_READ | kqueue.EVFILT_WRITE, kqueue.EV_ADD | kqueue.EV_ENABLE))
>>> kq.kevent(None, 16, 1000000000)
[]

I wonder why are these so inconsistent. One would expect come consistent results for multiplexing a simple thing such a unconnected brand new socket, no?If you think that is horrible – well, i do – check out the tricks to check for connection success on non-blocking sockets – and they don’t cover win32:

• http://cr.yp.to/docs/connect.html
• http://www.cyberconf.org/~cynbe/ref/nonblocking-connects.html

However, win32 has a much nicer api in the so called overlapped io wich is very different: you don’t need to check the sockets before making a operation – you just make the socket call with a overlapped and check for for completed calls (via io completion ports – gqcs). Still, you can use some selects variants (WSAAsyncSelect, WSAEventSelect, select) wich are just as bad.

Unfortunately python has a incomplete binding, pywin32 (not that bad – only ConnectEx and TransmitFile missing) for that and it’s actually fairly easy to crash your python interpreter with this.

>>> def somegenerator():
...     print (yield 1)
...     print (yield 2)
...
>>> g = somegenerator()
>>> print g.next()
1
>>> print g.send('a')
a
2
>>>

Coroutines are program components that generalize subroutines to allow multiple entry points and suspending and resuming of execution at certain locations, to quote wikipedia. So we could look at a generator as a coroutine and the yield expressions as suspend/resume points. Take this example:

def somecoroutine():
    data = (yield nonblocking_read(my_socket, nbytes))

Well, this example is actually from the PEP342 spec.Here’s an example that uses cogen:

from cogen.core.coroutine import coroutine
from cogen.core.schedulers import Scheduler
from cogen.core.sockets import Socket
from cogen.core.reactors import SelectReactor

@coroutine
def somecoroutine():
    mysocket = Socket() # cogen's socket wrapper
    yield mysocket.connect(('www.google.com',80))
    yield mysocket.writeall("GET / HTTP/1.1\r\nHost: www.google.com\r\n\r\n")
    result = yield mysocket.read(10240)
    print result

sched = Scheduler(reactor=SelectReactor)
sched.add(somecoroutine)
sched.run() # this is the main loop

• coroutine is a special decorator that wrapps generators and functions alike in a special Coroutine class. So for example, if you decorate a function that isn’t actualy a generator nothing bad will happen.
• Socket is a special wrapper that looks like the regular socket object (well, we actualy have a different flavour for usual recv and send, namely: read, readall, readline, write, writeall) but returns some special objects that we call operations. These operations instruct the cogen scheduler what to do with the coroutine.
• reactor=SelectReactor – there are actually 5 reactors: SelectReactor, PollReactor, KQueueReactor, EpollReactor, IOCPProactor. For KQueueReactor and EpollReactor you need C extension modules (that are provided in the lib directory) and for IOCPProactor you need the pywin32 extensions – also, IOCPProactor is in development atm. The reactor coupled with the socket operations handles all the nasty asynchronous networking code.

Also, there are a bunch of examples here.

Calls to other coroutines need to be made through the scheduler. To call another coroutine you yield a Call operation. Here’s an example:

>>> from cogen.core.coroutine import coroutine
>>> from cogen.core.schedulers import Scheduler
>>> from cogen.core import events
>>>
>>> @coroutine
... def foo():
...     print 'foo'
...     result = yield events.Call(bar, args=("ham",))
...     print result
...
>>> @coroutine
... def bar(what):
...     print 'bar'
...     raise StopIteration("spam, %s and eggs" % what)
...
>>> sched = Scheduler()
>>> sched.add(foo)
<foo Coroutine instance at 0x00D1E6B8 wrapping <function foo at 0x00C01B30>, state: NOTSTARTED>
>>> sched.run()
foo
bar
spam, ham and eggs
>>>

cogen has also other usefull operations. Take for example this using signals:

>>> from cogen.core.coroutine import coroutine
>>> from cogen.core.schedulers import Scheduler
>>> from cogen.core import events
>>>
>>> @coroutine
... def foo():
...     yield events.Signal("bar", 'spam')
...     yield events.Signal("bar", 'ham')
...     yield events.Signal("bar", 'eggs')
...     yield events.Sleep(3)
...
>>> @coroutine
... def bar():
...     print (yield events.WaitForSignal("bar"))
...     print (yield events.WaitForSignal("bar"))
...     print (yield events.WaitForSignal("bar"))
...     try:
...         print (yield events.WaitForSignal("bar", timeout=2))
...     except events.OperationTimeout:
...         print 'No more stuff !'
...
>>> sched = Scheduler()
>>> sched.add(bar)
<bar Coroutine instance at 0x00D1E6B8 wrapping <function bar at 0x00C551F0>, state: NOTSTARTED>
>>> sched.add(foo)
<foo Coroutine instance at 0x00D1EC90 wrapping <function foo at 0x00C01B30>, state: NOTSTARTED>
>>> sched.run()
spam
ham
eggs
No more stuff !
>>>

• events.Signal(name, value) – name is an object that needs to be immutable (strings, tuples, numbers and frozen sets are immutable for example); value is what is yield events.WaitForSignal going to return.
• cogen‘s scheduler main loop (run) exits when there is no more stuff to run (no active coroutines, no socket operations waiting etc) – events.WaitForSignal does not qualify, otherwise the scheduler would run endlessly and nothing would happen (no other coroutines to raise the signal), that is why there is a events.Sleep(3) in foo.

There’s a lot more to talk about, so check the docs on events and sockets, the examples.