Python annoyances - generators

Another annoyance in Python is generators. Since Python 2.5, the generator concept was extended to include coroutine support. (See http://www.python.org/dev/peps/pep-0342/ )
I for one think it's an odd design of coroutines.
Note that this comes from someone who is used to (and spoiled by) Lua's coroutines.

I have two problems with coroutines/generators in Python:

First of all, "def" gets multiple meanings. It's not just a function definition anymore.
It could also be a generator definition! And the only way to notice this is to
scan the entire function definition for a yield statement.

I would not have a problem with this if it wasn't for the case that generator
definitions and function definitions behave completely differently.

def gen():
 return 123

I can call this with gen() and get 123 back. Simple!

def gen():
 yield 123

If I call gen() now, I don't get 123 back, I get a generator object.
So I have to do this instead:

g = gen()
value = g()

That by itself isn't much trickier, but it's hard to know by just looking at the code if it's a generator. It's completely invisible at the call site, and at the definition it could be anywhere in the function. This is probably why most generators I've seen in Python have been fairly short and simple (which is a good thing for function design over all anyway).

Let's compare it to Lua, which was the language that introduced me to coroutines:

-- Simple function call
function gen()
 return 123
end
value = gen()

-- Using a generator / coroutine
function gen()
 coroutine.yield(123)
end
c = coroutine.wrap(gen)
value = c()

Here it's quite clear at the call site that we're dealing with a coroutine. And if we were to call gen as a regular function, we'd get a runtime error unless we were actually running in a coroutine already.

That gets us into the second annoyance. Python only supports yield in the generator definition. If you were to build a more complex generator, you'd have to write it as one long function instead of splitting it up into many small functions.

As it happens, I have a good example to use. The following is actual code I have for a hobby project. It's an AI implemented with Lua coroutines. To keep it small, not all functions have been defined, but I've kept everything that deals with coroutines.

function resume(entity)
 if not entity.coroutine then
   entity.coroutine = coroutine.create(function()
     return entity:runAI()
   end)
 end
 coroutine.resume(entity.coroutine)
end

function Entity:moveTowards(x, y)
 local dx = sign(x - self.x)
 local dy = sign(y - self.y)
 local dir = getDirection(dx, dy)
 if dir ~= -1 then
   TryMove(self.entity, dir)
   coroutine.yield()
 end
end

function Entity:moveTo(x, y)
 while self.x ~= x or self.y ~= y do
   self:moveTowards(x, y)
 end
end

function Entity:attack(dx, dy)
 local dir = getDirection(dx, dy)
 if dir ~= -1 then
   TryAttack(self.entity, dir)
   coroutine.yield()
 end
end

function Entity:chasePlayer()
 while true do
   local distance = self:distanceToPlayer()
   if distance > 8 then
     break
   end
   if distance <= 2 then
     local dx = px - self.x
     local dy = py - self.y
     self:attack(dx, dy)
   else
     self:moveTowards(px, py)
   end
 end
end

function Entity:patrolTo(x, y)
 while self.x ~= x or self.y ~= y do
   local distance = self:distanceToPlayer()
   if distance <= 5 then
      self:chasePlayer()
   else
     self:moveTowards(x, y)
   end
 end
end

-- Populate world with deadly enemies
local entity = Entity:new(20, 10)
function entity:runAI()
 while true do
   self:moveTo(20, 8)
   self:moveTo(20, 12)
 end
end
addEntity(entity)

local entity = Entity:new(15, 15)
function entity:runAI()
 while true do
   self:patrolTo(15, 15)
   self:patrolTo(15, 10)
   self:patrolTo(10, 10)
   self:patrolTo(10, 15)
 end
end
addEntity(entity)

As you can see, each entity type can be written by using very high level
constructs, which themselves are defined by simple logic and calls to lower level
functions. This lets you write an AI as if it was running in its own thread,
but in reality, the game engine only calls the resume-function for each entity
on its main thread.

Without coroutines, you'd have to resort to writing a state machine instead,
which can be more painful to write and maintain.

Now, I've tried converting this to Python to see what it would look like.
This was my first iteration, which won't run because the generators itself
don't contain the actual yield-call.

def resume(entity):
 if entity.coroutine == None:
   entity.coroutine = entity.runAI()
 return entity.coroutine()

class Entity:
 def __init__(self, x, y):
   self.x = x
   self.y = y

 def moveTowards(self, x, y):
   dx = sign(x - self.x)
   dy = sign(y - self.y)
   dir = getDirection(dx, dy)
   if dir != -1:
     TryMove(self.entity, dir)
     yield

 def moveTo(self, x, y):
   while self.x != x or self.y != y:
     self.moveTowards(x, y)

 def attack(self, dx, dy):
   dir = getDirection(dx, dy)
   if dir != -1:
     TryAttack(self.entity, dir)
     yield

 def chasePlayer(self):
   while True:
     distance = self.distanceToPlayer()
     if distance > 8:
       break
     if distance <= 2:
       dx = px - self.x
       dy = py - self.y
       self.attack(dx, dy)
     else:
       self.moveTowards(px, py)

 def patrolTo(self, x, y):
   while self.x != x or self.y != y:
     distance = self.distanceToPlayer()
     if distance <= 5:
       self.chasePlayer()
     else:
       self.moveTowards(x, y)

# Populate world with deadly enemies
class Runner(Entity):
 def runAi(self):
   while True:
     self.moveTo(20, 8)
     self.moveTo(20, 12)
entity = Runner(20, 10)
addEntity(entity)

class Patroller(Entity):
 def runAi(self):
   while True:
     self.patrolTo(15, 15)
     self.patrolTo(15, 10)
     self.patrolTo(10, 10)
     self.patrolTo(10, 15)

entity = Patroller(15, 15)
addEntity(entity)

Now, let's try refactoring until all yields are in the actual generator.
To do this, we could either simply inline all the functions that would yield,
until they're all in the runAi-method. The problem with this is a lot of
duplicated code, so that's not a good option.

Another way would be to only have the yields in the runAi, and let the currently
yielding functions return some value instead. The problem with that is that we
lose the knowledge of where to resume from, and thus we have to build some sort
of state maching to remember it, which would remove the point of having
coroutines in the first place.

If anyone can see a good way of refactoring that Python code (I am fairly new
to Python after all) to work as a coroutine, please let me know!

Until then, I'll just assume that python coroutines are inferior to Lua's. ;-)

[Here's an interesting article about coroutines: http://www.inf.puc-rio.br/~roberto/docs/MCC15-04.pdf ]