Friday, August 6, 2010

Python annoyances - generators

Another annoyance in Python is generators. Since Python 2.5, the generator concept was extended to include coroutine support. (See http://www.python.org/dev/peps/pep-0342/ )
I for one think it's an odd design of coroutines.
Note that this comes from someone who is used to (and spoiled by) Lua's coroutines.

I have two problems with coroutines/generators in Python:

First of all, "def" gets multiple meanings. It's not just a function definition anymore.
It could also be a generator definition! And the only way to notice this is to
scan the entire function definition for a yield statement.

I would not have a problem with this if it wasn't for the case that generator
definitions and function definitions behave completely differently.
def gen():
return 123
I can call this with gen() and get 123 back. Simple!
def gen():
yield 123
If I call gen() now, I don't get 123 back, I get a generator object.
So I have to do this instead:
g = gen()
value = g()
That by itself isn't much trickier, but it's hard to know by just looking at the code if it's a generator. It's completely invisible at the call site, and at the definition it could be anywhere in the function. This is probably why most generators I've seen in Python have been fairly short and simple (which is a good thing for function design over all anyway).

Let's compare it to Lua, which was the language that introduced me to coroutines:
-- Simple function call
function gen()
return 123
end
value = gen()

-- Using a generator / coroutine
function gen()
coroutine.yield(123)
end
c = coroutine.wrap(gen)
value = c()
Here it's quite clear at the call site that we're dealing with a coroutine. And if we were to call gen as a regular function, we'd get a runtime error unless we were actually running in a coroutine already.

That gets us into the second annoyance. Python only supports yield in the generator definition. If you were to build a more complex generator, you'd have to write it as one long function instead of splitting it up into many small functions.

As it happens, I have a good example to use. The following is actual code I have for a hobby project. It's an AI implemented with Lua coroutines. To keep it small, not all functions have been defined, but I've kept everything that deals with coroutines.
function resume(entity)
if not entity.coroutine then
entity.coroutine = coroutine.create(function()
return entity:runAI()
end)
end
coroutine.resume(entity.coroutine)
end

function Entity:moveTowards(x, y)
local dx = sign(x - self.x)
local dy = sign(y - self.y)
local dir = getDirection(dx, dy)
if dir ~= -1 then
TryMove(self.entity, dir)
coroutine.yield()
end
end

function Entity:moveTo(x, y)
while self.x ~= x or self.y ~= y do
self:moveTowards(x, y)
end
end

function Entity:attack(dx, dy)
local dir = getDirection(dx, dy)
if dir ~= -1 then
TryAttack(self.entity, dir)
coroutine.yield()
end
end

function Entity:chasePlayer()
while true do
local distance = self:distanceToPlayer()
if distance > 8 then
break
end
if distance <= 2 then
local dx = px - self.x
local dy = py - self.y
self:attack(dx, dy)
else
self:moveTowards(px, py)
end
end
end

function Entity:patrolTo(x, y)
while self.x ~= x or self.y ~= y do
local distance = self:distanceToPlayer()
if distance <= 5 then
self:chasePlayer()
else
self:moveTowards(x, y)
end
end
end

-- Populate world with deadly enemies
local entity = Entity:new(20, 10)
function entity:runAI()
while true do
self:moveTo(20, 8)
self:moveTo(20, 12)
end
end
addEntity(entity)

local entity = Entity:new(15, 15)
function entity:runAI()
while true do
self:patrolTo(15, 15)
self:patrolTo(15, 10)
self:patrolTo(10, 10)
self:patrolTo(10, 15)
end
end
addEntity(entity)

As you can see, each entity type can be written by using very high level
constructs, which themselves are defined by simple logic and calls to lower level
functions. This lets you write an AI as if it was running in its own thread,
but in reality, the game engine only calls the resume-function for each entity
on its main thread.

Without coroutines, you'd have to resort to writing a state machine instead,
which can be more painful to write and maintain.

Now, I've tried converting this to Python to see what it would look like.
This was my first iteration, which won't run because the generators itself
don't contain the actual yield-call.
def resume(entity):
if entity.coroutine == None:
entity.coroutine = entity.runAI()
return entity.coroutine()

class Entity:
def __init__(self, x, y):
self.x = x
self.y = y

def moveTowards(self, x, y):
dx = sign(x - self.x)
dy = sign(y - self.y)
dir = getDirection(dx, dy)
if dir != -1:
TryMove(self.entity, dir)
yield

def moveTo(self, x, y):
while self.x != x or self.y != y:
self.moveTowards(x, y)

def attack(self, dx, dy):
dir = getDirection(dx, dy)
if dir != -1:
TryAttack(self.entity, dir)
yield

def chasePlayer(self):
while True:
distance = self.distanceToPlayer()
if distance > 8:
break
if distance <= 2:
dx = px - self.x
dy = py - self.y
self.attack(dx, dy)
else:
self.moveTowards(px, py)

def patrolTo(self, x, y):
while self.x != x or self.y != y:
distance = self.distanceToPlayer()
if distance <= 5:
self.chasePlayer()
else:
self.moveTowards(x, y)

# Populate world with deadly enemies
class Runner(Entity):
def runAi(self):
while True:
self.moveTo(20, 8)
self.moveTo(20, 12)
entity = Runner(20, 10)
addEntity(entity)

class Patroller(Entity):
def runAi(self):
while True:
self.patrolTo(15, 15)
self.patrolTo(15, 10)
self.patrolTo(10, 10)
self.patrolTo(10, 15)

entity = Patroller(15, 15)
addEntity(entity)
Now, let's try refactoring until all yields are in the actual generator.
To do this, we could either simply inline all the functions that would yield,
until they're all in the runAi-method. The problem with this is a lot of
duplicated code, so that's not a good option.

Another way would be to only have the yields in the runAi, and let the currently
yielding functions return some value instead. The problem with that is that we
lose the knowledge of where to resume from, and thus we have to build some sort
of state maching to remember it, which would remove the point of having
coroutines in the first place.

If anyone can see a good way of refactoring that Python code (I am fairly new
to Python after all) to work as a coroutine, please let me know!

Until then, I'll just assume that python coroutines are inferior to Lua's. ;-)

[Here's an interesting article about coroutines: http://www.inf.puc-rio.br/~roberto/docs/MCC15-04.pdf ]

49 comments:

Anonymous said...

Hi this is vignesh i am having 3 years of experience as a php developer and i am certified. i have knowledge on OOPS concepts in php but dont know indepth. After learning hadoop will be enough to get a good career in IT with good package? and i crossed hadoop training in chennai website where someone please help me to identity the syllabus covers everything or not??

Thanks,
vignesh

Unknown said...

Hi this is Kathiresan i am having 3 years of experience as a dot net developer and i am certified. i have knowledge on OOPS concepts in .NET but dont know indepth. After learning android will be enough to get a good career in IT with good package? and i crossed Android Training in Chennai website where someone please help me to identity the syllabus covers everything or not??

thanks,
kathiresan

Unknown said...

Hi this is Kathiresan i am having 3 years of experience as a dot net developer and i am certified. i have knowledge on OOPS concepts in .NET but dont know indepth. After learning android will be enough to get a good career in IT with good package? and i crossed Android Training in Chennai website where someone please help me to identity the syllabus covers everything or not??

thanks,
kathiresan

Unknown said...


Hi this is abinaya i am having 3 years of experience as a java developer and i am certified. i have knowledge on OOPS concepts in java but dont know indepth. After learning oracle

will be enough to get a good career in IT with good package? and i crossed oracle

training in chennai">oracle training in chennai
website where someone please help me to identity the syllabus covers everything or not??

Thanks, abinaya

Unknown said...

Hi this is abinaya i am having 3 years of experience as a java developer and i am certified. i have knowledge on OOPS concepts in java but dont know indepth. After learning oracle

will be enough to get a good career in IT with good package? and i crossed oracle training in chennai website where someone

please help me to identity the syllabus covers everything or not??

Thanks, abinaya

Unknown said...



Hi this is Rajesh i am having 3 years of experience as a php developer and i am certified. i have knowledge on OOPS concepts in php but dont know indepth. After learning Dot Net will be enough to get a good career in IT with good package? and i crossed Dot Net training in chennai website where someone please help me to identity the syllabus covers everything or not??

thanks, Rajesh

Unknown said...

Hi this is Ganesh i am having 3 years of experience as a Java developer and i am certified. i have knowledge on OOPS concepts in java but dont know indepth. After learning php will be enough to get a good career in IT with good package? and i crossed php training in chennai website where someone please help me to identity the syllabus covers everything or not??

thanks, Ganesh

Unknown said...

Hi this is Ganesh i am having 3 years of experience as a Java developer and i am certified. i have knowledge on OOPS concepts in java but dont know indepth. After learning php will be enough to get a good career in IT with good package? and i crossed php training in chennai website where someone please help me to identity the syllabus covers everything or not??

thanks, Ganesh.

Unknown said...

very nice blogs!!! i have to learning for lot of information for this sites...Sharing for wonderful information.Thanks for sharing this valuable information to our vision. You have posted a trust worthy blog keep sharing. PHP Training in chennai | PHP Training chennai | PHP course in chennai | PHP course chennai

Unknown said...

Excellent information. HTML5 is a markup language used for designing responsive website and it is also used for structuring and presenting the website content.
HTML5 Training | PHP Course in Chennai



Anonymous said...

Really nice post. Unix is a multiuser and multi tasking operating system at the same time. Unix Training Chennai offering real time Unix course at reasonable cost.

Unknown said...

I am reading your post from the beginning, it was so interesting to read & I feel thanks to you for posting such a good blog, keep updates regularly.
Regards,
Salesforce course in Chennai |Salesforce training chennai

Stephen said...

There are lots of information about latest technology and how to get trained in them, like Hadoop Training Chennai have spread around the web, but this is a unique one according to me. The strategy you have updated here will make me to get trained in future technologies(Hadoop Training in Chennai). By the way you are running a great blog. Thanks for sharing this. FITA chennai reviews

Unknown said...

Very informative post. If interested, one can take up AngularJS training and stay up to date in technology

Unknown said...

Excellent article you are shared python online training

Unknown said...

Thank you for having taken your time to provide us with your valuable information relating to your stay with us. We are sincerely concerned.., Most importantly,you Keep it the major...
Software testing selenium training in Chennai

gowsalya said...

Your very own commitment to getting the message throughout came to be rather powerful and have consistently enabled employees just like me to arrive at their desired goals. Best Selenium Training in Bangalore
Selenium Interview Question and Answers - Frequently Asked Questions

Unknown said...

Well researched article and I appreciate this. The blog is subscribed and will see new topics soon.
Java training in Chennai | Java training institute in Chennai | Java course in Chennai

Java training in Bangalore | Java training institute in Bangalore | Java course in Bangalore

Java online training | Java Certification Online course-Gangboard

Java training in Pune

Mirnalini Sathya said...

This blog really pushed to explore more information. Thanks for sharing.

Selenium Training in Chennai
software testing selenium training
ios developer course in chennai
French Classes in Chennai
salesforce developer training in chennai
web designing training in chennai
Hadoop Course in Chennai
Loadrunner Training in Chennai

Unknown said...

After reading your post I understood that last week was with full of surprises and happiness for you. Congratz! Even though the website is work related, you can update small events in your life and share your happiness with us too.
Data Science Training in Chennai | Data Science Training institute in Chennai
Data Science course in anna nagar
Data Science course in chennai | Data Science Training institute in Chennai | Best Data Science Training in Chennai
Data science course in Bangalore | Data Science Training institute in Bangalore | Best Data Science Training in Bangalore
Data Science course in marathahalli | Data Science training in Bangalore

ProPlus Academy said...

Hey Nice Blog!! Thanks For Sharing!!!Wonderful blog & good post.Its really helpful for me, waiting for a more new post. Keep Blogging!
digital marketing course in coimbatore
php training in coimbatore

Unknown said...

The post is written in very a good manner and it entails many useful information for me. I am happy to find your distinguished way of writing the post. Now you make it easy for me to understand and implement the concept.

angularjs Training in bangalore

angularjs Training in bangalore

angularjs interview questions and answers

angularjs Training in marathahalli

angularjs interview questions and answers

angularjs-Training in pune

saranyaregan said...

It's interesting that many of the bloggers to helped clarify a few things for me as well as giving.Most of ideas can be nice content.The people to give them a good shake to get your point and across the command
rpa training in bangalore
best rpa training in bangalore
rpa training in pune

jai said...


This is quite educational arrange. It has famous breeding about what I rarity to vouch. Colossal proverb. This trumpet is a famous tone to nab to troths. Congratulations on a career well achieved. This arrange is synchronous s informative impolite festivity to pity. I appreciated what you ok extremely here.
Data Science training in rajaji nagar
Data Science with Python training in chennai
Data Science training in electronic city
Data Science training in USA
Data science training in pune
Data science training in bangalore

sunshineprofe said...

It’s a shame you don’t have a donate button! I’d certainly donate to this brilliant blog! I suppose for now I’ll settle for book-marking
iosh course in chennai

user123 said...

Write more; that’s all I have to say. It seems as though you relied on the video to make your point. You know what you’re talking about, why waste your intelligence on just posting videos to your blog when you could be giving us something enlightening to read?
Check out the best python training in chennai at SLA

Jaweed Khan said...

Thanks For sharing Your Information The Information shared Is Very Valuable Please Keep Updating Us Python Online Course Hadoop Online Course Data Science Online Course Aws Online Course

abmeljoseph said...

To make it easier for you sriwebeo at Coimbatore & Bangalore is visualizing all the materials you want.

Start brightening your career with us.
Digital Marketing Training Course in Coimbatore

jefrin said...





And indeed, I’m just always astounded concerning the remarkable things served by you. Some four facts on this page are undeniably the most effective I’ve had.
Data science Course Training in Chennai |Best Data Science Training Institute in Chennai
RPA Course Training in Chennai |Best RPA Training Institute in Chennai
AWS Course Training in Chennai |Best AWS Training Institute in Chennai
Devops Course Training in Chennai |Best Devops Training Institute in Chennai
Selenium Course Training in Chennai |Best Selenium Training Institute in Chennai
Java Course Training in Chennai | Best Java Training Institute in Chennai


Anjali Siva said...

I gathered lots of information from your blog and it helped me a lot. Keep posting more.
Salesforce Training in Chennai
salesforce training institute in chennai
Salesforce Course
ccna Training in Chennai
Ethical Hacking course in Chennai
PHP Training in Chennai
gst Training in Chennai
Salesforce Training in Anna Nagar
Salesforce Training in Vadapalani
Salesforce Training in Thiruvanmiyur

Thavaselvan said...

Very Good Blog. Highly valuable information have been shared. Highly useful blog..Great information has been shared. We expect many more blogs from the author. Special thanks for sharing..
SAP Training in Chennai | AWS Training in Chennai | Android Training in Chennai | Selenium Training in Chennai | Networking Training in Chennai

sindhuvarun said...

Thanks for sharing this useful information..
spoken english classes in bangalore
Spoken English Classes in Chennai
spoken english class in coimbatore
spoken english class in madurai
Data Science Courses in Bangalore
devops training in bangalore
English Coaching Classes in Chennai
spoken english course in coimbatore
spoken english classes in btm
spoken english classes in marathahalli

Reshma said...


Thanks for this wonderful blog. keep update more information about this
Ethical Hacking Course in Chennai
Hacking Course in Chennai
Ethical hacking course in bangalore
Ethical hacking course in coimbatore
Ethical Hacking Training in Chennai
Certified Ethical Hacking Course in Chennai
Ethical Hacking Training Institute in Chennai
Ethical hacking Training institute in bangalore
Software Testing Training in Chennai
ielts coaching centre in coimbatore

shri said...

Awesome post...
internship report on python
free internship in chennai for ece students
free internship for bca
internship for computer science engineering students in india
internships in hyderabad for cse students 2018
electrical companies in hyderabad for internship
internships in chennai for cse students 2019
internships for ece students
inplant training in tcs chennai
internship at chennai

svrtechnologies said...

This post is really nice and informative. The explanation given is really comprehensive and informative.... salesforce developer online training

divya said...

Are you searching for a home maid or old care attandents or baby care aaya in india contact us and get the best and experianced personns in all over india for more information visit our site nice page
Ai & Artificial Intelligence Course in Chennai
PHP Training in Chennai
Ethical Hacking Course in Chennai Blue Prism Training in Chennai
UiPath Training in Chennai

rocky said...

I really enjoy your article is awesome.
Python Training in Chennai

Python Training in Bangalore

Python Training in Hyderabad

Python Training in Coimbatore

Python Training

python online training

python flask training

python flask online training



sudhan said...

Excellent information. HTML5 is a markup language used for designing responsive website and it is also used for structuring and presenting the website content.
Cyber Security Training Course in Chennai | Certification | Cyber Security Online Training Course | Ethical Hacking Training Course in Chennai | Certification | Ethical Hacking Online Training Course | CCNA Training Course in Chennai | Certification | CCNA Online Training Course | RPA Robotic Process Automation Training Course in Chennai | Certification | RPA Training Course Chennai | SEO Training in Chennai | Certification | SEO Online Training Course

Unknown said...

Good Post! , it was so good to read and useful to improve my knowledge as an updated one, keep blogging.After seeing your article I want to say that also a well-written article with some very good information which is very useful for the readers....thanks for sharing it and do share more posts likethis. https://www.3ritechnologies.com/course/tableau-online-training-certification/

Suryaprakash said...

I am very interested in reading your blog. keep sharing.
how to implement devops
uses of java programming
adwords strategy
is web development dying
seo questions and answers

Mega said...
This comment has been removed by the author.
nayar said...


This blog is really awesome. I learned lots of informations in your blog. Keep posting like this...
German Classes in Bangalore
German Language Course in Bangalore
German Language Course in Hyderabad
German Language Course in Delhi
German Language Classes in Pune

Unknown said...

aşk kitapları
tiktok takipçi satın al
instagram beğeni satın al
youtube abone satın al
twitter takipçi satın al
tiktok beğeni satın al
tiktok izlenme satın al
twitter takipçi satın al
tiktok takipçi satın al
youtube abone satın al
tiktok beğeni satın al
instagram beğeni satın al
trend topic satın al
trend topic satın al
youtube abone satın al
takipçi satın al
beğeni satın al
tiktok izlenme satın al
sms onay
youtube izlenme satın al
tiktok beğeni satın al
sms onay
sms onay
perde modelleri
instagram takipçi satın al
takipçi satın al
tiktok jeton hilesi
instagram takipçi satın al pubg uc satın al
sultanbet
marsbahis
betboo
betboo
betboo

Unknown said...

instagram takipçi satın al
instagram takipçi satın al
instagram takipçi satın al
instagram takipçi satın al
instagram takipçi satın al
instagram takipçi satın al
instagram takipçi satın al

vanhelsing said...
This comment has been removed by the author.
manasha said...

Great post. keep sharing such a worthy information.
Salesforce Training in Chennai

Reshma said...

This post is so interactive and informative.keep update more information...
Artificial Intelligence Course in Gurgaon
Artificial Intelligence Course in Hyderabad

Matt Reeves said...

Mindblowing blog very useful thanks
AngularJS Training in Velachery
Angularjs Training in Chennai

iteducationcentre said...

Great post.Thanks for the post.
Python Course in Nagpur

Post a Comment