partiallydisassembled.net

What slows down development...

Sat, 16 Aug 2008 17:00:22 +1000

... Vista bluescreening every time I terminate a hung Python process that has a 
lock ("Process has locked pages").  Yep, I can trigger it consistently 
and reliably.

Makes.. me.. angry!

Multithreaded goodness for pyglet

Tue, 12 Aug 2008 00:01:13 +1000

I've finally gotten some good results after spending the last few months (on and 
off) refactoring the pyglet.media module in pyglet to use multiple threads.  
There were two main reasons for attempting this: to make better use of multi-core 
machines (pyglet is currently single threaded), and to avoid having to schedule 
audio pump in the main event loop (in pyglet 1.0 this was explicit, in 1.1 it's 
done on the clock).  

The problem with having the audio pump in the main loop is that it requires 
application developers to manually tune their programs to keep the audio buffer 
full -- and this is obviously very dependent on the hardware and drivers 
available.  One motivating example was my last PyWeek entry, which had background 
music that stalled during long loading times (I managed to get this down to just 
the initial loading time, and smooth for the rest of the game, but this was a lot 
of effort, and removing the initial pause proved too difficult).

The redesigned media module is still residing in the experimental/mt_media 
directory of SVN trunk.  Only the OpenAL driver is anywhere near complete.  The 
DirectSound driver has yet to be ported to the multithreaded API, and the ALSA 
driver has been deprecated in favour of a new PulseAudio driver.  PulseAudio is 
proving far more reliable and capable than the ALSA user-space daemon, and has 
been the default audio server in both Ubuntu and Fedora Core for some time.  The 
PulseAudio driver is mostly complete, but has no 3D capability (Linux users can 
still choose to use OpenAL as an alternative).

The audio driver modules are required to pump themselves now.  The PulseAudio 
driver does this using the built-in PA thread runloop, whereas the OpenAL driver 
creates a Python thread that periodically wakes.

Much of the shared code between drivers regarding video frame synchronisation 
has been moved into the base classes.  This should greatly reduce the amount of 
effort required to develop a new audio driver.  Audio drivers dispatch events to 
the Player object when a new video frame should be displayed.  New methods on 
pyglet.app make it easy to post events from one thread into the main event loop 
safely (these also automatically wake up the main event loop if it's sleeping, 
allowing more opportunities for people to integrate multithreaded code into their 
pyglet applications).

The AVbin source driver also now spawns a worker thread for decoding video 
frames.  Since most of the work in decoding the frame is done outside of Python 
(in FFmpeg), this is a great chance for pyglet to do some real multiprocessing.  
Quite a lot of work went into ensuring the queue of video frames to be decoded is 
kept up-to-date with what's needed on-screen; without an excessive preroll period 
or memory usage.

I've done most development on my 900MHz iBook, which is incapable enough that it 
can't even play YouTube videos in real time (though pyglet can play them just 
fine once downloaded, thanks to FFmpeg).  This has forced me to make the 
implementation efficient enough to work on old computers; the overhead of the 
extra thread processing is significant and required several manual context 
switches to get everything playing (almost) as well as the old, single-threaded 
code.

On newer computers the performance is fantastic.  I did a quick benchmark test 
on my 1.8GHz Core 2 Duo Mac Mini.  It happily plays 7 SD videos simultaneously 
(DivX or Xvid encoding) without sweating, whereas the old code could only play 3 
videos.  Watching the activity monitor shows that both cores are now being 
utilised fully when needed (and run each at around 5% when playing a single 
video).

Expect to see this new code in pyglet 1.2.  I'll introduce some more of the new 
features over the next few weeks.  I'll also be talking about pyglet 1.2 at OSDC 
2008 in Sydney in December.

Good looking tick mark distribution

Mon, 11 Aug 2008 18:40:25 +1000

Given a min/max range of any magnitude, how do you generate a "nice 
looking" distribution of values to cover the range?  My need for this was to 
generate tick marks on a graph.  The function I came up with works quite nicely, 
I think:

    import math

    def distribute_nicely(min_val, max_val, min_ticks=3, max_ticks=8):
        if max_val - min_val == 0:
            return [min_val]
        mag = 10 ** int(math.log10(max_val - min_val) - .5)
        if mag == 0:
            mag = 0.1
        for multiplier in (10, 5, 2, 1, .5, .2, .1):
            interval = multiplier * mag
            if min_ticks <= (max_val - min_val) / interval <= max_ticks:
                min_val = (min_val // interval) * interval
                break
        else:
            interval = (max_val - min_val) / max_ticks or max_val - min_val
        if interval > 1:
            return range(min_val, max_val + interval, interval)
        else:
            return [i * interval  \
                    for i in range(int(min_val / interval), 
                                   int(max_val / interval) + 1)]

Sample results:

    >>> distribute_nicely(0, 10)
    [0, 2, 4, 6, 8, 10]
    >>> distribute_nicely(0, 100)
    [0, 20, 40, 60, 80, 100]
    >>> distribute_nicely(0, 5)
    [0, 1, 2, 3, 4, 5]
    >>> distribute_nicely(3, 8)
    [3, 4, 5, 6, 7, 8]
    >>> distribute_nicely(24, 786)
    [0, 200, 400, 600, 800]
    >>> distribute_nicely(0.21, 0.86)
    [0.2, 0.4, 0.6, 0.8]
    >>> distribute_nicely(-8, 8)
    [-10, -5, 0, 5, 10]
    >>> distribute_nicely(-105, 105)
    [-150, -100, -50, 0, 50, 100, 150]

I'd be interested to hear if anyone has an alternative function or improvements 
to this one.  Or if there's one in the standard library -- I always tend to miss 
those :-)

Clean termination of daemon Python threads

Sat, 09 Aug 2008 11:23:03 +1000

Daemonic threads in Python are threads that will not prevent the interpreter 
from exiting (put another way, the interpreter shuts down when all non-daemon 
threads exit).  When the interpreter shuts down, all variables are set to None, 
then each daemon thread is woken up to crash (unexpectedly finding all its 
variables, globals and builtins missing).

Here's a nifty pattern that suppresses the ugly exception trace resulting from 
these crashes being printed to the terminal:

    def worker_thread():
        try:
            .... do work ...
        except:
            if True:
                raise

This ensures ordinary exceptions are re-raised, but interpreter shutdown related 
exceptions are suppressed, as True is evaluated as None.  In the off-chance that 
some future or alternative Python interpreter doesn't have the same strange 
daemon thread shutdown behaviour, the thread will still behave correctly.

Annotate observed types in a Python program

Wed, 18 Jun 2008 12:22:02 +1000

Neat script I wrote for a friend to annotate a Python script with the types 
observed during a run of the program.  Usage:

    python -m annotype test.py

With this script:

    def foo(bar, baz):
        pass

    foo(1, 'a')
    foo(2, 3)

Outputs 'out_annotype/test.py' with:
    
    def foo(bar, baz):
        # [annotype] bar: int
        # [annotype] baz: str, int
        pass

    foo(1, 'a')
    foo(2, 3)

Hopefully useful for discovering usage of undocumented APIs.

Python, meet Vista/64

Fri, 09 May 2008 00:06:24 +1000

Finally bought a copy of Vista/64 today.  First Python experience:

    C:\Python25>python
    Python 2.5.2 (r252:60911, Feb 21 2008, 13:17:27) [MSC v.1400 64 bit (AMD64)] 
on win32
    Type "help", "copyright", "credits" or 
"license" for more information.
    >>> import sys
    >>> sys.platform
    'win32'
    >>>

Sigh.  And then,

    >>> import ctypes
    Traceback (most recent call last):
      File "<stdin>", line 1, in <module>
    ImportError: No module named ctypes
    >>>

Um, should Python even be distributed as version "2.5.2" on this 
platform if parts of the standard library are missing?

(Oh, and Vista is pretty, except when the "Press this button to keep doing 
what you're doing" dialog pops up, because it always seems to blank the 
screen first -- possibly disabling Aero temporarily?).

Compiling for G3 on OS X 10.5

Fri, 18 Apr 2008 00:03:46 +1000

It took me a long time to figure this out.  By default gcc on Leopard compiles 
for OS X 10.5 only.  When cross-compiling to PowerPC, the default target is 
actually certain G4 models.  The magic options for G3's and OS X 10.3.9 (it looks 
like you can't compile for earlier versions any more) and later:

    -arch ppc 
    -isysroot /Developer/SDKs/MacOSX10.4u.sdk 
    -mmacosx-version-min=10.3.9
    -force_cpusubtype_ALL

(both linker and compiler).  Use 

    otool -hv xyz.dylib

to check that a Mach-O file is for the target you expect ("ppc" not 
"ppc7740" etc), and

    lipo -info xyz.dylib

to check the contents of a universal binary.

pyglet download statistics

Sun, 23 Mar 2008 22:44:21 +1000

I felt charty this evening, and this is what happened:



That's a breakdown by release and distribution of pyglet, from 1.0 alpha 1 
through to 1.1 alpha 1 (alpha 2, released hours ago, doesn't yet rate on this 
chart).

Interesting to note: the eggs are popular for release, but not development.  The 
only reason they're featured in 1.0 alpha 1 is that they were actually linked 
from the download page then.  (Now people are only likely to grab an egg if 
they're using easy_install from the command line).

The Windows installer is quite the popular, which makes me glad I spent so much 
time on it.  (I'm not making installers available for 1.1 alpha releases).

Here are the download stats for the packages including documentation/examples.



Looks to me like the number of developers isn't really increasing, but perhaps 
those developers are testing on more machines, and perhaps there are some runtime 
installations being made on non-developer machines.  Or maybe there are more 
developers but they're happy to use old or online documentation.

There are currently 271 pyglet-users members (or 269, depending on which counter 
on the same page you read).

Play with your peas

Tue, 12 Feb 2008 23:40:29 +1000

Danc of Lost Garden has posted a new game challenge that includes a design and 
sprites.

Under the cover of "testing the latest pyglet features" (but really 
for the hell of it) I made a quick stab at it last night and today.



You can play with peas-1.zip, you'll need pyglet trunk r1760.  It's nothing like 
complete: you can place and remove blocks, and the peas will A* themselves up to 
the higest point and "ninja" themselves off it; but the scoring, flag 
raising and deathliness is missing.

The physics seems quite stable (using Verlet integration with a fixed timestep), 
but the contact constraints on the corners are doubled, which is why the peas 
seem to gain energy when bouncing off walls.

I made a botch of the AI movement: every specific combination of blocks that can 
be moved between has to be listed (e.g., left side of standard block to top side 
of left ramp, ...).  I didn't get around to listing them all, so the peas will 
refuse to move around certain objects that seem trivial.  If you just place 
standard blocks they'll do fine.  I would rewrite this to something a bit 
cleverer if I was going to continue any further...

...which I don't think I will.  These peas are cute, but I don't think the game 
is going to turn out all that great, relative to the effort required to fix up 
the AI and physics.

I was able to pick up a few pyglet bugs writing this.  The sprite module seems 
(now) pretty solid and absolutely useful for this kind of game -- I didn't need 
to use pyglet.gl at all.

7

Sat, 19 Jan 2008 23:50:53 +1000

I think it's great that channel 7 chooses to simulcast the tennis on all five of 
their digital channels, rather than running the scheduled movie.  Just like 
watching someone else play that Wii game.

delta-v update

Tue, 15 Jan 2008 01:36:10 +1000

I ported delta-v to pyglet 1.0 beta 3, and also took the opportunity to fix some 
gameplay issues:

  * Baddies now die when shot, rather than just being pushed away.
  * The field of view is reduced.
  * There is a more obvious visual indicator when the player is hurt.
  * Player's health is gradually restored continually.
  * Motion blur is greatly reduced.
  * Survival mode score resets to zero before each attempt.

This makes the game far less frustrating.

Son of follower of lines

Tue, 01 Jan 2008 23:44:41 +1000

Much improved, as predicted.



I ripped up the front of the tribot and replaced the jaws with an articulated 
light sensor, which can sweep +/- around 60 degrees from center.  It took quite a 
while to figure out how to gear it and get everything solid enough; all these 
new-fangled lego pieces I've not seen before.

This is what it does:



It's much faster than the last one -- when it gets it right, it can work out the 
curvature of the line and steer exactly along it.  When it loses the line, it 
backtracks a little to find it.  At the end of the video you can see the sensor 
can't pick up the sharp turn in the line and the robot drives off into oblivion 
(more work required).

On the programming side of things, I'm getting more used to the NXT environment. 
 I used two threads: one to control the sensor motor and one for the drive 
motors.  They communicate through two semaphores: the angle of the sensor motor, 
and a flag that gets raised when the sensor has lost the line.

It probably would be feasible to do it in a single run-loop, albeit with more 
state variables, and the NXT tool has fits with two many nested conditionals (it 
also has fits about most other things, to be fair).

I find I have to change the tolerances of the light/dark sensing very carefully 
to avoid incorrect readings.  I have to adjust these tolerances throughout the 
day, by trial and error, due to the change in ambient light.  I still haven't had 
any luck with the calibration.

Here's the source code; this time I've added some annotations that might give a 
feel for what it's all about.  Note that this is the only view of the program 
you're given, and it can't be zoomed out (nor are there scrollbars...).

Mon, 31 Dec 2007 19:35:50 +1000

 

First Mindstorms robot.  Here it is in action:



I started out with the tribot, which has building instructions but no real 
programs.  I had to rip up the front of it to move the light sensor further from 
the floor -- in its initial position it couldn't tell the difference between 
white and black.

You can show the current readings of the sensors on the device, which would be 
handy when programming thresholds, but they don't seem to match up at all... so 
it needed quite a bit of tweaking.

Here's the source code:



Lots of room for improvement...

Method caching in CPython

Mon, 17 Dec 2007 11:32:50 +1000

This patch by Armin Rigo has been getting a lot of attention on the python-dev 
list recently.  From the description:

  This is a port of PyPy's method cache optimization.  It gives a very good 
  speed-up to PyPy, and it is simple enough to be attempted on CPython.

The patch introduces a global method cache, currently set to hold 1024 elements. 
 Each element in the cache looks like this:

  struct method_cache_entry {
      int version;
      PyObject *name;
      PyObject *value;
  };

The struct is 12 bytes wide on 32-bit systems and 20 bytes wide on 64-bit 
systems, making the table 12kb and 20kb large, respectively.

Each Python type is given a "version" number which is treated the same 
as version numbers in Microsoft's Dynamic Language Runtime.  Version numbers are 
obtained from a global counter (so no two types share the same version number).  
When a type is modified, that type's version number is refreshed (updated from 
the global counter), as are the version numbers of all that type's subtypes.

When an attribute is looked up on a type (i.e., because it has failed lookup on 
the object), the cache is first checked for the corresponding type's version 
number, and the name of the method.  The method cache is a hash table, with the 
hash formed by the tuple (version, name).

If that attribute is not found in the method cache, lookup proceeds through the 
MRO as usual; however that method will subsequently be cached before returning.  
The entry in the method cache that occupies the same hash location will be 
tossed.

The version number is a 32-bit int, and wraparound is handled correctly (when 
the global counter reaches 0 again, the method cache is cleared).  The last time 
I checked, Microsoft's DLR did not gracefully handle wraparound.  Because type 
modification is relatively rare, I believe the version number could be held in a 
16-bit integer without performance penalty, and with a small memory footprint 
benefit.  I haven't tried this though.

Armin and the patch reviewers have been reporting a consistent 20% or better 
performance improvement on total runtime of all applications, which is nothing to 
be sneezed at.  The patch is still under review but looks like it's heading for 
inclusion in Python 2.6.

The technique is similar to method caching used in Smalltalk-80.  All 
implementations of Smalltalk use a global descriptor table which maps the (class, 
method) tuple to the method object.  The earlier implementations, up until 1983, 
typically used global method caches like this one to decrease the cost of looking 
in the global table.  Since then, inline caches have been the typical approach, 
such as in Smalltalk (1984), Self (1994) and the Microsoft DLR / IronPython 
(2007).

TEN HD beg for my forgiveness

Sun, 16 Dec 2007 09:54:45 +1000

No doubt in response to my post last month, TEN has stopped using the term 
"full HD" to describe their new channel, which launches today.

Disappointingly the new channel offers little new content (one late night movie 
most days, and repeats of Veronica Mars and Battlestar Galactica), and even less 
timeshifted content (only the news, delayed by half an hour).

EBroadcast, with their larger-than-screen-sized banner ads, have the TEN HD 
schedule; YourTV, with their sensibly-sized ads, have not yet added TEN HD or 7 
HD.