The “yield from” proposal

There’s a bit of buzz on the python-ideas mailing list about Greg Ewing’s proposal for a new keyword “yield from“. He has made a draft PEP and patch for python 2.6.1.

On short it adds syntactic sugar for generator decomposition and adds a “return value” statement in generators (the underlying mechanism for that is a StopIteration(value) exception).

Here’s a straightforward example:

>>> def g1():
...     print "Starting g1"
...     yield "g1 ham"
...     result = yield from g2()
...     print "Got result: %r from g2" % result
...     yield "g1 eggs"
...     print "Finishing g1"
...
>>> def g2():
...     print "Starting g2"
...     yield "g2 spam"
...     yield "g2 more spam"
...     print "Finishing g2"
...     return "foobar"
...
>>> for x in g1():
...     print "Yielded", x
...
Starting g1
Yielded g1 ham
Starting g2
Yielded g2 spam
Yielded g2 more spam
Finishing g2
Got result: 'foobar' from g2
Yielded g1 eggs
Finishing g1

There are a bunch of other examples and tests bundled with Greg’s patch.

I got the patch and it works :). Had to manually rebuild the parser – make didn’t do it by itself and I had to run “make Parser/pgen & make Python/graminit.c & make“. I wonder why.

cogen and greenlets

I was playing these days with greenlets and I finally managed to find some time to make and coroutine implementation for cogen that uses the seamless stack switching of greenlets instead of the usual generator yield-based switching.

The best part about this is that cogen could support any sort of old code that uses the usual file or socket apis in the standard python library.

Unfortunately greenlets don’t work with python 2.6 right now; let me shamelessly quote this very nice reply I got from one of py.lib’s (greenlets are part of py.lib) maintainers:

“I mean come on – it’s open source, if it works for me on 2.5 why I would bother too much?”

For the love of god, is this the right attitude to run a project these days ?! I mean come on …

Anyhow, nevermind that.

So back to those coroutines using greenlets – I named them corolets – feels like a good name given the fact the the regular generator stuff is named with coroutine and there’s some need for distinction. I also added some socket wrappers in a module socketlets (not very inspired). It has a Socket class that provides regular blocking interface and makes use of the good old _fileobject class from the socket module in stdlib for the makefile stuff.

I got those 2 pieces in cogen.magic.corolets and cogen.magic.socketlets. For those who didn’t understood much here’s an example:

import sys

from cogen.core import schedulers
from cogen.magic.corolets import corolet, yield_
from cogen.magic import socketlets

@corolet
def server():
    srv = socketlets.Socket()
    adr = ('0.0.0.0', len(sys.argv)>1 and int(sys.argv[1]) or 1200)
    srv.bind(adr)
    srv.listen(64)
    while 1:
        print "Listening on", adr
        conn, addr = srv.accept()
        print "Connection from %s:%s" % addr
        m.add(handler, args=(conn, addr))

@corolet
def handler(sock, addr):
    fh = sock.makefile()
    fh.write("WELCOME TO ECHO SERVER !\r\n")
    fh.flush()

    while 1:
        line = fh.readline(1024)
        if line.strip() == 'exit':
            fh.write("GOOD BYE")
            fh.close()
            sock.close()
            return
        fh.write(line)
        fh.flush()

m = schedulers.Scheduler()
m.add(server)
m.run()

Compare that to the regular generator based example (don’t miss those yields):

import sys

from cogen.core import sockets
from cogen.core import schedulers
from cogen.core.coroutines import coroutine

@coroutine
def server():
    srv = sockets.Socket()
    adr = ('0.0.0.0', len(sys.argv)>1 and int(sys.argv[1]) or 1200)
    srv.bind(adr)
    srv.listen(64)
    while 1:
        print "Listening on", adr
        conn, addr = yield srv.accept()
        print "Connection from %s:%s" % addr
        m.add(handler, args=(conn, addr))

@coroutine
def handler(sock, addr):
    fh = sock.makefile()
    yield fh.write("WELCOME TO ECHO SERVER !\r\n")
    yield fh.flush()

    while 1:
        line = yield fh.readline(1024)
        if line.strip() == 'exit':
            yield fh.write("GOOD BYE")
            yield fh.close()
            sock.close()
            return
        yield fh.write(line)
        yield fh.flush()

m = schedulers.Scheduler()
m.add(server)
m.run()

It’s pretty rough right now – I still need to add tests and check the performance, but I wanted to give an idea of what’s going on for the braver ones :)

ctypes based iocp proactor for cogen

I’ve finally mannaged to pull off a set of wrappers for using the raw windows iocp api using ctypes, namely wrappers for GetQueuedCompletionStatus, WSARecv, WSASend, AcceptEx, ConnectEx, TransmitFile and the whatever structs and utility calls they need.

Writing the wrappers wasn’t that hard – but if you aren’t careful with ctypes you are in a world of gruesome crashes and at least 2 wasted evenings (read nights) debugging why the hell some random allocation fails. Well, for a fact, I still have no idea to why do I own those to wasted nights but thank god it doesn’t crash on my unittests anymore – I think I could have figured out why some missing object cleanup resulted in weird crashes if I knew some ctypes internals but eh, I don’t :(

Take a look at the evil code here and here, and punish me for doing such an evil thing with ctypes *malefic laugh* =]

Here’s a gem:

#~ BOOL = SOCKET hSocket, HANDLE hFile, DWORD nNumberOfBytesToWrite, DWORD nNumberOfBytesPerSend, LPOVERLAPPED lpOverlapped,LPTRANSMIT_FILE_BUFFERS lpTransmitBuffers, DWORD dwFlags
TransmitFileType = WINFUNCTYPE(BOOL, SOCKET, HANDLE, DWORD, DWORD, POINTER(OVERLAPPED), POINTER(TRANSMIT_FILE_BUFFERS), DWORD)

def _GetTransmitFilePtr(given_socket=None):
    from socket import socket
    bogus_sock = given_socket or socket()
    bogus_bytes = DWORD()
    TransmitFile = TransmitFileType(0)
    ret = WSAIoctl(
        bogus_sock.fileno(), SIO_GET_EXTENSION_FUNCTION_POINTER, byref(WSAID_TRANSMITFILE), sizeof(WSAID_TRANSMITFILE),
        byref(TransmitFile), sizeof(TransmitFile), byref(bogus_bytes), None, None
    )
    return TransmitFile

TransmitFile = _GetTransmitFilePtr()

So far the new proactor works pretty nice in cogen.

Javascript-Flash socket bridge with haxe

There are a bunch of solutions for making a bridge to use the flash sockets (flash.net.Socket) in javascript, notably socketBridge and jssockets. But I didn’t like the way they made you write your client code: because you can’t pass javascript function references through the ExternalInterface you usually pass your callbacks as strings representing the name of an global object. IMHO, that just sucks so I’ve made my own mix named HaxeSocketBridge.

You can write the client code like this:

    var s = new FlashSocket({
        on_data: function(data) {
            document.getElementById('datapanel').value = data;
        },
        on_io_error: function(msg) {
            alert("IO ERROR: "+msg);
        },
        on_security_error: function(msg) {
            alert("SECURITY ERROR: "+msg);
        },
        on_close: function(msg) {
            alert("Connection closed.");
        },
        on_connect: function() {
            s.write("GET / HTTP/1.0\r\n\r\n");
        }
    });
    s.connect('example.com', 80);

Here’s the whole example.

How does it work? You can execute arbitrary javascript code though the external interface, so why not execute some boilerplate code and setup a global FlashSocket class that handles all the nasty actionscript-javascript communication.  I’ve seen this arguably good idea in the SWFHttpRequest bridge. If you do something like this be careful with those damn commas/semicolons or else you’re in i-dont-know-why-it-breaks hell :)

Here’s how the guts look like (note that the language is haxe):

        ExternalInterface.addCallback("connect", connect);
        ExternalInterface.addCallback("close", close);
        ExternalInterface.addCallback("write", write);
        ExternalInterface.addCallback("CAN_I_HAS_SOCKET", CAN_I_HAS_SOCKET);
        ExternalInterface.call([
        "(function(){",
            "if (window.FlashSocket) return;",
            "var Class = function(properties){",
                "var klass = function(event_handlers){ ",
                    "for (var p in event_handlers) {",
                        "if (event_handlers.hasOwnProperty(p)) {",
                            "this[p] = event_handlers[p];",
                        "}",
                    "}",
                    "return this.init.apply(this);",
                "};",
                "klass.prototype = properties;",
                "klass.constructor = arguments.callee;",
                "return klass;",
            "};",
            "window.FlashSocket = new Class({",
                "init: function(){",
                    "this._instance = ''+window.FlashSocket._instances.length;",
                    "window.FlashSocket._instances.push(this);",
                "},",
                "close: function(){ ",
                    "window.FlashSocket._instances[this._instance] = null;",
                    "window.FlashSocket._bridge.close(this._instance );",
                "},",
                "write: function(data){ ",
                    "window.FlashSocket._bridge.write(this._instance, data);",
                "},",
                "connect: function(host, port) {",
                    "window.FlashSocket._bridge.connect(this._instance, host, port);",
                "}",
            "});",
            "window.FlashSocket._instances = [];",
            "var f = function(tag){",
                "var elems = document.getElementsByTagName(tag);",
                "for (var i=0; i<elems.length; i++) if (elems[i].CAN_I_HAS_SOCKET) return elems[i];",
            "};",
            "window.FlashSocket._bridge = f('embed') || f('object');",
        "})" ].join('') );
        if (flash.Lib.current.loaderInfo.parameters.onloadcallback != null)
            ExternalInterface.call(flash.Lib.current.loaderInfo.parameters.onloadcallback);

The trick is that we can setup a global javascript class-like object from the flash clip.

You can grab the code here. Also, if you test from a local disc you might want to adjust your flash security settings. To compile the .hx file run “haxe compile.hxml“.

Now, the only thing I miss is some buffering/readline logic.

setuptools install_requires quirk

It’s interesting that having “dev” version packages in the install_requires list will fail.

Say for example, if you do:

from setuptools import setup
setup(
...
install_requires=["Pylons==dev"],
...
)

It won’t work:

Installed d:\python25\lib\site-packages\pylons-0.9.7rc3dev_20081103-py2.5.egg
Reading http://www.pylonshq.com/download/0.9.7
error: Could not find required distribution Pylons==dev

I imagine it would match if the package would literally have the version “dev”.

One solution is to change the install_requires to ["Pylons==dev,!=foo"] or some other bogus version to match whatever is downloaded for the dev version. “>=0.1.2.3” will match for packages that don’t have tagged versions (won’t work for example for Pylons “0.9.7beta3dev-20080422″).

EDIT: Looks like I was wrong. having “!=foo” will work if you already have the package installed but will fail to find the dev package when downloading. Having “>=0.9.7″ in the requirements will not work because “0.9.7beta3dev-20080422″ is considered a lower version (prerelease to the final 0.9.7). A solution would be having a requirement like “>=0.9.7a”. “a” will be lower than “beta3dev-20080422″.

Functions are descriptors

I didn’t knew that till i’ve read Ian’s post on decora-descriptors (I guess I can call them that). Actually it’s explained in more details here.

On short, functions have the __get__ method that make them work as descriptors when they are in a class.

I can do this:

>>> class Bar:
...     pass
...
>>> bar = Bar()
>>>
>>> def foo():
...     pass
...
>>> foo
<function foo at 0x00B6F270>
>>>
>>> foo.__get__(None, Bar)
<unbound method Bar.foo>
>>>
>>> foo.__get__(bar, Bar)
<bound method Bar.foo of <__main__.Bar instance at 0x00BCB738>>

Neat !

Giving up on Google Appengine

Quitter’s journal: After trying for half of week of getting some stuff going on appengine here is what I observed.

The runtime is completely different. They do try however to patch the normal python runtime in the sdk to have the same characteristics as the runtime on their servers but differences fall through the cracks. The have some sort of qa process for the sdk (probably to compensate for these differences) – I bet it must be very cumbersome to maintain it.

In the sdk you have so little python libraries available and what makes matters worse is that they are outdated so you have to do additional evil monkey patching to use newer versions.

Ok, so I started with the idea of a django app – but I gave up when i found the one in the sdk is very old (0.96) and i still need to zip it (django slightly goes over the 1000 file limit). So if I start zipping stuff I might just as well do it the right way and use eggs and setuptools – well, I’ve moved to that idea and still didn’t work out due to the fact setuptools can barely run in the appengine runtime.

Also, thinking on the zips in retrospective – they aren’t such a great idea. Why the hell would i waste cpu cycles on zips ?

And speaking of the 1000 file limit, their filesystem must work real bad with many files.  Guido said on the file limit: “it exists because of the sheer number of apps. E.g. 1,000,000 apps == 1,000,000,000 files”. Is 1B files too much ? For Google !?

Come to think of it a bit, the only advantage of appengine is the fact is almost free.

LE: corrected the arithmetic.

On the future of cogen coroutine library

The major refactor of the socket stuff is finally in the trunk. The Socket is more in line with the standard socket module in python (there’s a makefile and a _fileobject for readline just like the standard library). And it looks like performance is still pretty good.

The only things I can think of right now are the unittests for the socket stuff – they lack good coverage and they were actually made for the previous incompatible cogen socket api. They will be fixed eventually.

Another thing is that i’m trying to push some patches in the pywin32 module for win32file.TransmitFile and win32file.ConnectEx. But it’s going so slow that I’m thinking it isn’t going to work out and I’d better write a separate module for the GetQueueCompletionStatus, CreateIoCompletionPort, WSASend, WSARecv, ConnectEx, TransmitFile and avoid all that insane overlapped struct hacking that happens in the win32file module. Wadda ya think ?

Place your bets, Google vs Other browsers

Google’s shiny new browser is almost here: Google Chrome based on WebKit.

Featuring a sandbox model for tabs (tough I hope that sandbox will not be a complete process – as some speculate) and a “V8″ javascript engine – according to Joshua “code is instantly interpreted into machine code”. Damn, where did he knew that ? Really.

Surely will feature gears and other google mumbo-jumbo (some say it will even have a privacy feature ala IE8 aka the “porn mode”). I imagine it will be neat and fast but very unpolished – as the other google products launched in the “launch early and iterate” spirit. Speaking of, meh, app engine it’s still just a Toy.

I still put my money on the Shiretoko. (I hope I’m wrong tough)

Shortcuts again

I’ve added a manage bindings dialog and other minor improvements. Grab the userscript from http://userscripts.org/users/62351/scripts