Friday, August 6, 2010

Python annoyances - generators

Another annoyance in Python is generators. Since Python 2.5, the generator concept was extended to include coroutine support. (See http://www.python.org/dev/peps/pep-0342/ )
I for one think it's an odd design of coroutines.
Note that this comes from someone who is used to (and spoiled by) Lua's coroutines.

I have two problems with coroutines/generators in Python:

First of all, "def" gets multiple meanings. It's not just a function definition anymore.
It could also be a generator definition! And the only way to notice this is to
scan the entire function definition for a yield statement.

I would not have a problem with this if it wasn't for the case that generator
definitions and function definitions behave completely differently.
def gen():
return 123
I can call this with gen() and get 123 back. Simple!
def gen():
yield 123
If I call gen() now, I don't get 123 back, I get a generator object.
So I have to do this instead:
g = gen()
value = g()
That by itself isn't much trickier, but it's hard to know by just looking at the code if it's a generator. It's completely invisible at the call site, and at the definition it could be anywhere in the function. This is probably why most generators I've seen in Python have been fairly short and simple (which is a good thing for function design over all anyway).

Let's compare it to Lua, which was the language that introduced me to coroutines:
-- Simple function call
function gen()
return 123
end
value = gen()

-- Using a generator / coroutine
function gen()
coroutine.yield(123)
end
c = coroutine.wrap(gen)
value = c()
Here it's quite clear at the call site that we're dealing with a coroutine. And if we were to call gen as a regular function, we'd get a runtime error unless we were actually running in a coroutine already.

That gets us into the second annoyance. Python only supports yield in the generator definition. If you were to build a more complex generator, you'd have to write it as one long function instead of splitting it up into many small functions.

As it happens, I have a good example to use. The following is actual code I have for a hobby project. It's an AI implemented with Lua coroutines. To keep it small, not all functions have been defined, but I've kept everything that deals with coroutines.
function resume(entity)
if not entity.coroutine then
entity.coroutine = coroutine.create(function()
return entity:runAI()
end)
end
coroutine.resume(entity.coroutine)
end

function Entity:moveTowards(x, y)
local dx = sign(x - self.x)
local dy = sign(y - self.y)
local dir = getDirection(dx, dy)
if dir ~= -1 then
TryMove(self.entity, dir)
coroutine.yield()
end
end

function Entity:moveTo(x, y)
while self.x ~= x or self.y ~= y do
self:moveTowards(x, y)
end
end

function Entity:attack(dx, dy)
local dir = getDirection(dx, dy)
if dir ~= -1 then
TryAttack(self.entity, dir)
coroutine.yield()
end
end

function Entity:chasePlayer()
while true do
local distance = self:distanceToPlayer()
if distance > 8 then
break
end
if distance <= 2 then
local dx = px - self.x
local dy = py - self.y
self:attack(dx, dy)
else
self:moveTowards(px, py)
end
end
end

function Entity:patrolTo(x, y)
while self.x ~= x or self.y ~= y do
local distance = self:distanceToPlayer()
if distance <= 5 then
self:chasePlayer()
else
self:moveTowards(x, y)
end
end
end

-- Populate world with deadly enemies
local entity = Entity:new(20, 10)
function entity:runAI()
while true do
self:moveTo(20, 8)
self:moveTo(20, 12)
end
end
addEntity(entity)

local entity = Entity:new(15, 15)
function entity:runAI()
while true do
self:patrolTo(15, 15)
self:patrolTo(15, 10)
self:patrolTo(10, 10)
self:patrolTo(10, 15)
end
end
addEntity(entity)

As you can see, each entity type can be written by using very high level
constructs, which themselves are defined by simple logic and calls to lower level
functions. This lets you write an AI as if it was running in its own thread,
but in reality, the game engine only calls the resume-function for each entity
on its main thread.

Without coroutines, you'd have to resort to writing a state machine instead,
which can be more painful to write and maintain.

Now, I've tried converting this to Python to see what it would look like.
This was my first iteration, which won't run because the generators itself
don't contain the actual yield-call.
def resume(entity):
if entity.coroutine == None:
entity.coroutine = entity.runAI()
return entity.coroutine()

class Entity:
def __init__(self, x, y):
self.x = x
self.y = y

def moveTowards(self, x, y):
dx = sign(x - self.x)
dy = sign(y - self.y)
dir = getDirection(dx, dy)
if dir != -1:
TryMove(self.entity, dir)
yield

def moveTo(self, x, y):
while self.x != x or self.y != y:
self.moveTowards(x, y)

def attack(self, dx, dy):
dir = getDirection(dx, dy)
if dir != -1:
TryAttack(self.entity, dir)
yield

def chasePlayer(self):
while True:
distance = self.distanceToPlayer()
if distance > 8:
break
if distance <= 2:
dx = px - self.x
dy = py - self.y
self.attack(dx, dy)
else:
self.moveTowards(px, py)

def patrolTo(self, x, y):
while self.x != x or self.y != y:
distance = self.distanceToPlayer()
if distance <= 5:
self.chasePlayer()
else:
self.moveTowards(x, y)

# Populate world with deadly enemies
class Runner(Entity):
def runAi(self):
while True:
self.moveTo(20, 8)
self.moveTo(20, 12)
entity = Runner(20, 10)
addEntity(entity)

class Patroller(Entity):
def runAi(self):
while True:
self.patrolTo(15, 15)
self.patrolTo(15, 10)
self.patrolTo(10, 10)
self.patrolTo(10, 15)

entity = Patroller(15, 15)
addEntity(entity)
Now, let's try refactoring until all yields are in the actual generator.
To do this, we could either simply inline all the functions that would yield,
until they're all in the runAi-method. The problem with this is a lot of
duplicated code, so that's not a good option.

Another way would be to only have the yields in the runAi, and let the currently
yielding functions return some value instead. The problem with that is that we
lose the knowledge of where to resume from, and thus we have to build some sort
of state maching to remember it, which would remove the point of having
coroutines in the first place.

If anyone can see a good way of refactoring that Python code (I am fairly new
to Python after all) to work as a coroutine, please let me know!

Until then, I'll just assume that python coroutines are inferior to Lua's. ;-)

[Here's an interesting article about coroutines: http://www.inf.puc-rio.br/~roberto/docs/MCC15-04.pdf ]

35 comments:

vignesh m said...

Hi this is vignesh i am having 3 years of experience as a php developer and i am certified. i have knowledge on OOPS concepts in php but dont know indepth. After learning hadoop will be enough to get a good career in IT with good package? and i crossed hadoop training in chennai website where someone please help me to identity the syllabus covers everything or not??

Thanks,
vignesh

vignesh m said...

Hi this is vignesh i am having 3 years of experience as a android developer and i am certified. i have knowledge on OOPS concepts in android but dont know indepth. After learning hadoop will be enough to get a good career in IT with good package? and i crossed hadoop training in chennai website where someone please help me to identity the syllabus covers everything or not??

Thanks,
vignesh

Kathiresan Muthu said...

Hi this is Kathiresan i am having 3 years of experience as a dot net developer and i am certified. i have knowledge on OOPS concepts in .NET but dont know indepth. After learning android will be enough to get a good career in IT with good package? and i crossed Android Training in Chennai website where someone please help me to identity the syllabus covers everything or not??

thanks,
kathiresan

Kathiresan Muthu said...

Hi this is Kathiresan i am having 3 years of experience as a dot net developer and i am certified. i have knowledge on OOPS concepts in .NET but dont know indepth. After learning android will be enough to get a good career in IT with good package? and i crossed Android Training in Chennai website where someone please help me to identity the syllabus covers everything or not??

thanks,
kathiresan

dwara kesh said...


Hi this is dwarakesh i am having 3 years of experience as a php developer and i am certified. i have knowledge on OOPS concepts in php but dont know
indepth. After learning java will be enough to get a good career in IT with good package? and i crossed
java training in chennai website where someone please help me to identity the syllabus covers everything or not??

thanks, dwarakesh

dwara kesh said...

Hi this is dwarakesh i am having 3 years of experience as a php developer and i am certified. i have knowledge on OOPS concepts in php but dont know
indepth. After learning java will be enough to get a good career in IT with good package? and i crossed
java training in Chennai website where someone please help me to identity the syllabus covers everything or not??

thanks, dwarakesh

Abinaya V said...


Hi this is abinaya i am having 3 years of experience as a java developer and i am certified. i have knowledge on OOPS concepts in java but dont know indepth. After learning oracle

will be enough to get a good career in IT with good package? and i crossed oracle

training in chennai">oracle training in chennai
website where someone please help me to identity the syllabus covers everything or not??

Thanks, abinaya

Abinaya V said...

Hi this is abinaya i am having 3 years of experience as a java developer and i am certified. i have knowledge on OOPS concepts in java but dont know indepth. After learning oracle

will be enough to get a good career in IT with good package? and i crossed oracle training in chennai website where someone

please help me to identity the syllabus covers everything or not??

Thanks, abinaya

Ramys said...

Hi, this is Ganesh I am having 3 years of experience as a Dot Net developer and I am certified. I have Knowledge on OOPS Concepts in .NET indepth. After learning Salesforce will be enough to get a good career in IT with good Package? and i crossed Salesforce Training in Chennai website where someone please help me to identify the syllabus covers everything or not??

Thanks,
Ganesh

Ramys said...

Hi, this is Ganesh I am having 3 years of experience as a Dot Net developer and I am certified. I have Knowledge on OOPS Concepts in .NET indepth. After learning Salesforce will be enough to get a good career in IT with good Package? and i crossed Salesforce Training in Chennai website where someone please help me to identify the syllabus covers everything or not??

Thanks,
Ganesh

Rajesh Ravi said...

Hi this is Rajesh i am having 3 years of experience as a php developer and i am certified. i have knowledge on OOPS concepts in php but dont know indepth. After learning Dot Net will be enough to get a good career in IT with good package? and i crossed Dot Net training in chennai website where someone please help me to identity the syllabus covers everything or not??

thanks, Rajesh

Rajesh Ravi said...



Hi this is Rajesh i am having 3 years of experience as a php developer and i am certified. i have knowledge on OOPS concepts in php but dont know indepth. After learning Dot Net will be enough to get a good career in IT with good package? and i crossed Dot Net training in chennai website where someone please help me to identity the syllabus covers everything or not??

thanks, Rajesh

Ganesh M said...

Hi this is Ganesh i am having 3 years of experience as a Java developer and i am certified. i have knowledge on OOPS concepts in java but dont know indepth. After learning php will be enough to get a good career in IT with good package? and i crossed php training in chennai website where someone please help me to identity the syllabus covers everything or not??

thanks, Ganesh

Ganesh M said...

Hi this is Ganesh i am having 3 years of experience as a Java developer and i am certified. i have knowledge on OOPS concepts in java but dont know indepth. After learning php will be enough to get a good career in IT with good package? and i crossed php training in chennai website where someone please help me to identity the syllabus covers everything or not??

thanks, Ganesh.

kovalan Jayamurugan said...

Excellent post on iOS mobile apps development!!! The future of mobile application development is on positive note. You can make most it by having in-depth knowledge on mobile application development platform and other stunning features. IOS Course in Chennai

john son said...

Hi, I have read your blog and I got a useful information from this blog. Thanks for sharing, keep posting..Salesforce is a cloud based CRM software. Today's most of the IT industry use this software for customer relationship management.
Regards
Salesforce Training in Chennai

dhanamlakshmi palu said...

very nice blogs!!! i have to learning for lot of information for this sites...Sharing for wonderful information.Thanks for sharing this valuable information to our vision. You have posted a trust worthy blog keep sharing. PHP Training in chennai | PHP Training chennai | PHP course in chennai | PHP course chennai

Jamuna Pandy said...

Oracle is a relational database management system produced by oracle corporation. Nowadays most of the multinational companies used this oracle database for storing and managing their data's and programs. So learning Best Oracle Training in Chennai is one of the best idea to make a bright career.

kovalan Jayamurugan said...

Thanks for your informative article. Android SDK allows you to create stunning mobile application loaded with more features and enhanced priority. With basis on Java coding language, you can create stunning mobile application with ease. Best Android Training institute in Chennai

jack wilson said...

Excellent information. HTML5 is a markup language used for designing responsive website and it is also used for structuring and presenting the website content.
HTML5 Training | PHP Course in Chennai



Jamuna Pandy said...

Nice information. Android is one of the most popular mobile operating system developed by Google. Learning Android Training in Chennai is useful to make a wonderful future in mobile application development field.

Martina Christy said...

Really nice post. Unix is a multiuser and multi tasking operating system at the same time. Unix Training Chennai offering real time Unix course at reasonable cost.

murali karthik said...

Really nice post. SEO is one of the digital marketing techniques used for improve the website ranking in search engine result page. To know more details please call 9003623340.
Regards..
SEO Training in Chennai

Amirtha rao said...

I am reading your post from the beginning, it was so interesting to read & I feel thanks to you for posting such a good blog, keep updates regularly.
Regards,
Salesforce course in Chennai |Salesforce training chennai

Stephen said...

There are lots of information about latest technology and how to get trained in them, like Hadoop Training Chennai have spread around the web, but this is a unique one according to me. The strategy you have updated here will make me to get trained in future technologies(Hadoop Training in Chennai). By the way you are running a great blog. Thanks for sharing this. FITA chennai reviews

Shashaa Tirupati said...

Very informative post. If interested, one can take up AngularJS training and stay up to date in technology

Vinoth Kumar said...

Welcome to Wiztech Automation - Embedded System Training in Chennai. We have knowledgeable Team for Embedded Courses handling and we also are after Job Placements offer provide once your Successful Completion of Course. We are Providing on Microcontrollers such as 8051, PIC, AVR, ARM7, ARM9, ARM11 and RTOS. Free Accommodation, Individual Focus, Best Lab facilities, 100% Practical Training and Job opportunities.

Embedded System Training in chennai
Embedded System Training Institute in chennai
Embedded Training in chennai
Embedded Course in chennai
Best Embedded System Training in chennai
Best Embedded System Training Institute in chennai
Best Embedded System Training Institutes in chennai
Embedded Training Institute in chennai
Embedded System Course in chennai
Best Embedded System Training in chennai

Shashaa Tirupati said...

Wow, brilliant article that I was searching for. Helps me a lot in taking class for my students, so using it in my work. Thanks a ton. Keep writing, would love to follow your posts.
Shashaa
Dot Net training in Chennai

for IT the said...

Great Article
Java Online Training | Java EE course

Java Training in Chennai | J2EE Training in Chennai | java j2ee training institutes in chennai ~ Java Course in Chennai | Java Training Institutes in Chennai

Java 360 | IT Technical Articles |Java Training Institutes

for IT the said...

Great Article
C# Online Training | ASP.NET Training | ASP.NET MVC Training

Dot Net Interview Questions

.net training online | Dot Net Training in Chennai | .Net Training in Chennai | Dot Net Training Institutes in Chennai

geethu said...

Thanks for sharing this unique and informative content which provided me the required information.
PHP Training in Chennai | PHP Training Center in Chennai | FITA Velachery

geethu said...

Thanks for sharing this unique and informative content which provided me the required information.
PHP Training in Chennai | PHP Training Center in Chennai | FITA Velachery

Priya said...

Well Said, you have furnished the right information that will be useful to anyone at all time. Thanks for sharing your Ideas.
oracle training in chennai | oracle training institutes in chennai

Priya said...

Excellent post!!!. The strategy you have posted on this technology helped me to get into the next level and had lot of information in it.
salesforce training in chennai | salesforce training institute in chennai

soumya Reddy said...

Excellent article you are shared python online training

Post a Comment