partiallydisassembled.net

Snow Leopard WiFi

Sun, 06 Sep 2009 13:23:19 +1000

That thing about deleting the VPN profile turned out to be a red herring, the 
problem returned after a few hours.  Finally solved it today, by setting the DNS 
server to my ISP, rather than my router/modem.  Doing the same on my iPhone also 
solved its WiFi issues, which have been annoying me since the 2.1 OS update.  I 
guess whatever DNS stack change was made on iPhone was backported to Snow 
Leopard, and it happens to interact poorly with my modem (Billion BiPAC 7300RA, 
for the record).  I still don't quite understand where in the OS X software stack 
the problem is; since DNS lookups against the router work fine using nslookup; 
I'm guessing it's something in the Cocoa layer.

More Snow Leopard thoughts

Wed, 02 Sep 2009 22:53:48 +1000

I seem to have fixed my earlier WiFi issues by deleting the (unused and 
disabled) university VPN profile.

Many people have written that there are virtually no new front-facing features 
in SL.  The only ones that affect me are:

 - Under 10 seconds from cold boot to ready-to-use.  That's really, really 
insane.
 - You can show the date in the menu bar next to the clock now.

But the new APIs are really interesting.  Read the Ars Technica review for a 
really long exposition.  Read the Apple developer notes for a faster, more 
detailed explanation.

Some highlights:

 - Block objects in C (i.e., closures).
 - Grand Central Dispatch: POSIX-level threading API that cooperates across 
processes to balance work units across an optimal number of threads.  The API is 
dead simple, using the aforementioned C block objects.
 - OpenCL: numeric for GPU, without fucking about with textures, render targets 
and full-screen quads.
 - Explicit cache control: mark pages as volatile, and the OS will flush them 
without paging them out if it needs to.
 - Opt-in application termination: shutdown doesn't wait for processes to exit 
if they've marked themselves as "safe for immediate termination".
 - File ID-based URLs replacing POSIX paths; unifies URLs and paths, and means 
applications can find files after they've been moved (if they're on the same 
volume).  Presumably also does away with having to worry about relative vs 
absolute paths (if you don't care about changing volumes).
 - QuickTime X: simpler APIs for things like playing back movies/audio that 
don't require you to read documentation that dates from the 80's.

Rampant speculation:
 
For the last few years we've been developing in an environment in which the 
three platforms (Windows, OS X and Linux) are approximately equivalent in 
features and APIs.  OS X and Linux to the point where you can often cross-compile 
the same source code with minor changes [I recently recompiled a Linux networking 
library for iPhone changing only a single #include directive].  The Win32 
equivalent functions for POSIX are well-known and easy to find.  Writing 
cross-platform software is a matter of factoring out the common code, and either 
rewriting the user interface for each platform, or using something like Qt to 
smooth over virtually all of the differences.  Using a cross-platform language 
like Java or Python makes it even more of a no-brainer.

But it looks to me like the platforms are diverging again.  OS X is gaining 
features that are genuinely useful (not just eye-candy or domain-specific) that 
have no equivalents in Windows or Linux.  Microsoft have had big plans for 
Windows, too.  None of these plans (besides Vista's transactional filesystem) 
seem to have made it into Vista or 7, but it's reasonable to expect that some of 
them will eventually make it into a product.  At the very least, it's clear that 
Microsoft's focus is on supporting .NET/CLR rather than the C API now and in the 
future.

Someone writing a Photoshop-like application for OS X would definitely want to 
use a whole lot of OS X specific features: CoreImage, CoreAnimation, OpenCL, the 
cache APIs, Grand Central, and so on.  These features are not just 
"nice-to-have" additions, they change the way you'd develop the 
application.  Similarly, the same application written for Windows nowadays would 
probably want to start with .NET, at the very least to get the most up-to-date 
GUI look.

The amount of code that could be shared between the two implementations gets 
smaller the more you improve it for any particular platform.  Cocoa/Objective-C 
code and .NET/C# code don't look very much alike at all, especially when you 
really start taking advantage of them.  That leaves choosing an in-between 
language (C++) and doing without a bunch of features, or a cross-platform 
language (Java/Python) and also doing without a bunch of features (and looking 
like poo, to boot).

I'd guess this is exactly where both Apple and Microsoft want us going, because 
it again allows each platform to have its own speciality apps and points of 
difference, where for a while they were more or less interchangeable.

While the split between OS X and Windows increases, the difference between the 
desktop and mobile editions is vanishing.  OS X and iPhone APIs were always 
similar, and now features from iPhone are being back-ported to OS X (CoreLocation 
and QuickTime X).  Rumour has it that the next Zune API will be XNA.  That leaves 
two fairly serious platforms to choose between: Windows/WindowsMobile/Zune/Xbox 
and OS X/iPhone.  Game developers also target PSP (conceptually similar to 
iPhone), PS3 and DS (like nothing else, from what I understand).

I don't think the cross-platform languages (Python, Java, etc) and libraries 
(Qt, etc) will be relevant for much longer.  When the Windows/Mac split increases 
more, cross platform development will be the domain of Javascript+HTML+Canvas, 
which is starting to look like a real platform since the announcement of Google 
Chrome OS, and has the benefit of actually being cross platform by design.

Snow Leopard WiFi

Wed, 02 Sep 2009 08:27:47 +1000

Something's very, very wrong.  Command-line programs like ping and curl work 
fine.  Safari, Camino, Firefox routinely think they have no internet connection 
and give up trying.  Some things that help:

 - Flushing the DNS cache (fixes problem for one web page fetch)
 - Restarting WiFi, or creating a new "location" in system preferences 
(an oft-quoted solution to this problem on the forums) -- fixes problem for 
nearly a minute
 - Restarting computer (which, incidentally, is an insanely fast 10 seconds) -- 
fixes problem for about a minute.

This is on a late 2008 model MacBook connecting to a NetGear router.  Things 
I've yet to try:

 - Put the kernel in 64-bit mode
 - Try another router (I have an old Apple one)
 - Downgrade to Leopard (clean install I imagine -- shudder).

Javascript types

Wed, 27 May 2009 23:21:08 +1000

Javascript has won the language wars, and look what we've got.  A single Number 
type (hooray), which is internally represented by a 64-bit IEEE float (erm, ok) 
unless you're doing bitwise operations, in which case it's:

 - violently coerced to a 32-bit signed integer in JS <= 1.1 (NaN if out of 
range)
 - silently coerced to a 32-bit signed integer in JS > 1.1 (truncated 
fractional and high bits)

What have we done to deserve this?

(Disclaimer:  I haven't been bitten by this design decision of doom, because I 
don't do any Javascript programming.  But I find myself strangely compelled not 
to even bother, given the risks apparent).

About Reader

Tue, 05 May 2009 09:18:27 +1000

I said that comments on Reader really suck in my last post, and I want to 
explain why.

Google Reader has changed my life (this is no big statement; lots of things 
change my life; including whichever bastard on one of the many overcrowded trains 
I catch coughed on me and gave me this cold; keeping me home from work and thus 
with too much time to blog inanities, like this one).

Reader started out as a feed aggregator.  Technologically, this means checking 
the timestamps of a predetermined list of remote files, and downloading any that 
have changed.  The point of which was to keep up-to-date with news (real news, 
from newspapers; tech news, from procrastination web sites; or friends' news, 
from blogs).

When Reader was released, there were hundreds of desktop applications that did 
exactly the same thing -- Reader's advantage was that it followed you between 
desktops, being online.

But it's so much more now.

Reader is how I keep in contact with friends.  People of my generation don't 
actually want to talk to their friends, or even SMS or email them.  That kind of 
contact is far too personal and touchy.  All we need is some way of indicating 
that we thought about each other, if only for a moment.  This can be achieved on 
Facebook with a nudge, on MySpace with a chain letter, on Twitter with a 
140-character brain broadcast, and on Reader by sharing something.  To each their 
own (or, in the case of some people I know, all of them).

When you share an article on Reader, you're telling them, "Hey, I saw this 
and think you should read it."  Which is how most conversations go anyway.  
Except for Austin, who mostly shares pictures of animals, but mostly talks about 
compilers.  

Originally you could only share articles from feeds; typically ones you'd 
subscribed to.  But some time recently Google added the ability to share any URL 
with a single click.  This has changed my friend's sharing habits from keeping 
each other up-to-date on the few blogs that we don't subscribe to in common, to 
keeping each other up-to-date with what we're doing all day.  For example, based 
on Sofie's shares from yesterday, I can surmise that she was passing the time by 
reading about penises and Star Trek cakes.  This tells me much more about Sofie 
than I could ever have learned by actually talking to her.

And then there's commenting.  In response to a friend's share, you can write 
something back.  Usually "hahaha", with the number of "ha"'s 
being proportional to the amount of enjoyment you received.  This comment is then 
viewable by any other friend looking at that shared item.

Occasionally one of us tries to start some discourse via the comments.  Say, if 
you didn't agree with the shared article, you might comment, "lolz no way 
ps3 is teh better than poniez".  But this is where it all breaks down, 
because anyone who's already read that item will not see your comment, unless 
they specifically go looking for it.  There isn't even a flag or a number to 
indicate that articles you've read have since been commented on.

At the moment Reader is like a party where everyone is telling jokes, and 
everyone is hearing the jokes, but no-one can hear all the laughter, which is 
isolated.  If comments were integrated into the main view -- say, given 
first-class status as another article, then we'd be able to continue the 
discussion past a single retort, and have a real argument.  It makes me think of 
Slashdot, except that the posting of articles and comments would be restricted to 
just the people on your friends list.  It's what I want!

(Side note: apparently whatever spell-check dictionary I have loaded on my Mac 
at the moment understands "MySpace" but not "Facebook" or 
"Google".  Figure that one out!)

Device removal on Windows

Mon, 04 May 2009 08:19:04 +1000

In response to:

  http://blogs.msdn.com/oldnewthing/archive/2009/05/01/9581563.aspx

which Sofie shared from a Digg post.  I'd reply with a Reader comment, but they 
really suck.

Raymond Chen (MS genius) writes that users inadvertently get presented with the 
"advanced" device removal menu when right-clicking on the device 
removal notification icon, when all they should've done is left-click, which 
brings up the simpler menu.  The simple menu presents one menu item for each 
device, alongside the drive letter, making it relatively easy to find the usb 
stick you're trying to remove.  The advanced menu shows a device/bus tree that 
makes it near-impossible to find the device you're after.

What's amazing is that first Raymond, then all the commentators, blame the user 
for choosing the wrong menu ("oh, silly me, I didn't realise there was a 
left-click menu").  No-one points out that nowhere else in Windows does a 
left-click activate a menu.  (There is also a half-second delay in bringing up 
the simple menu, compared to the advanced menu, causing many to try both and end 
up with just the advanced one after the simple one doesn't appear to do 
anything).

Secondly, no-one points out that the "advanced" menu is pointless.  It 
has no more functionality than the simple menu, because you are still removing 
the same devices, they're just listed in a different format (one that is 
opaque).

Finally, the whole interface is completely insane.  Internal hard drives are 
listed alongside USB sticks, it asks for confirmation before ejecting the disk 
(something which is easily undoable and has no consequence), and everything in 
the menu is a "Mass Storage Device".

But the real kicker -- why doesn't Windows flush data to USB devices as soon as 
there's an idle IO period (say, after 1 second); and show a warning icon only 
when the device is in use?  That way, it's always safe to remove the device, 
except for the few seconds (at most) after writing to it, which would be 
indicated by a red warning symbol or some such, telling you to hold off for a 
second.

Hell, they could've put that light on the stick itself.  "Don't pull the 
stick out while the light is on".  Sounds like floppy disk technology.  
Genius!

vectypes

Thu, 09 Apr 2009 23:19:51 +1000

I had some free time because my thumbs were sore from playing Wipeout HD, so I 
wrote another vector/matrix module for Python.

    http://vectypes.googlecode.com/

This one has a better design than euclid. 

  * The API is almost identical to the GLSL API, supporting all the same vector 
and matrix types and functions (but not the scalar or component-wise functions).
  * There are no methods on vectors or matrices, so we don't get confused by 
writing imperative code and thinking it's functional, or vice-versa.
  * There's a big test suite.

Because of the arrangement of matrix data (as a list of column vectors), it's 
probably less efficient in both time and space than euclid.  On the other hand, 
it's more likely to give the right answer, and doesn't have the Point/Vector 
distinction^H^H^H^H^H^Hconfusion that euclid does.

I wrote a small templating engine to generate the code.  I think it's quite 
neat.  It needs no explanation; just run the makefile and look at the generated 
files, you'll see how it works.

Watched constants, in Python

Sat, 28 Mar 2009 19:50:24 +1000

Games need lots of magic constant value to set creature spawn rates, movement 
speeds, acceleration, rates of fire, random chance, etc etc.  Finding values for 
all these numbers that work well is usually a lot of guesswork and good luck.  
And, unless you have some very sophisticated value-tweaking engine built into 
your game, a lot of recompilation and restarting as well.

I stumbled upon this neat trick on Molly Rocket for automatically updating 
hard-coded constants in source code as the game is running.

Here's my quick hacked-up version for Python:

    def TWEAK(initial_value):
        import traceback
        import re

        # Name of this function
        stack = traceback.extract_stack()
        func_name = stack[-1][2]

        # Pattern to locate the new value
        pattern = re.compile('.*[^a-zA-Z0-9]%s\(([^\)]+)\).*' % func_name)

        # Filename and line number where this function was called
        filename, lineno = stack[-2][:2]

        # See what that line looks like at the moment
        try:
            line = list(open(filename))[lineno - 1]
            match = pattern.match(line)
            return eval(match.group(1))
        except:
            return initial_value

Import this into your code, then whenever you have some constant you're not sure 
of, instead of writing:

    self.x += 0.5

use:

    self.x += TWEAK(0.5)

Then run your game.  The change is reflected in the game as it runs, as soon as 
you save the source file.  As is it won't work for values that are used to 
intialise, say, a particle emitter; it can probably be improved with some clever 
use of __getattribute__ and small changes to the way you write your code like 
that.  Another improvement that wouldn't be impossible is supporting multiple 
TWEAK values on one line.

The takeaway

Fri, 20 Feb 2009 10:26:21 +1000

When did this become the hip new word for "conclusion"? It irks me.

Anathem

Sat, 11 Oct 2008 20:36:51 +1000

"Our opponent is an alien starship packed with atomic bombs," I said.  
"We have a protractor."

Apple C header licenses

Wed, 08 Oct 2008 23:19:24 +1000

Seen in the prelude of the Kernel.framework/Versions/A/Headers/IOKit/hid/*.h 
files...

"[...] APPLE HEREBY DISCLAIMS ALL SUCH WARRANTIES, INCLUDING WITHOUT 
LIMITATION, ANY WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE, 
QUIET ENJOYMENT OR NON-INFRINGEMENT. [...]"

Not fit for quiet enjoyment?  Oh, my.

What slows down development...

Sat, 16 Aug 2008 17:00:22 +1000

... Vista bluescreening every time I terminate a hung Python process that has a 
lock ("Process has locked pages").  Yep, I can trigger it consistently 
and reliably.

Makes.. me.. angry!

Multithreaded goodness for pyglet

Tue, 12 Aug 2008 00:01:13 +1000

I've finally gotten some good results after spending the last few months (on and 
off) refactoring the pyglet.media module in pyglet to use multiple threads.  
There were two main reasons for attempting this: to make better use of multi-core 
machines (pyglet is currently single threaded), and to avoid having to schedule 
audio pump in the main event loop (in pyglet 1.0 this was explicit, in 1.1 it's 
done on the clock).  

The problem with having the audio pump in the main loop is that it requires 
application developers to manually tune their programs to keep the audio buffer 
full -- and this is obviously very dependent on the hardware and drivers 
available.  One motivating example was my last PyWeek entry, which had background 
music that stalled during long loading times (I managed to get this down to just 
the initial loading time, and smooth for the rest of the game, but this was a lot 
of effort, and removing the initial pause proved too difficult).

The redesigned media module is still residing in the experimental/mt_media 
directory of SVN trunk.  Only the OpenAL driver is anywhere near complete.  The 
DirectSound driver has yet to be ported to the multithreaded API, and the ALSA 
driver has been deprecated in favour of a new PulseAudio driver.  PulseAudio is 
proving far more reliable and capable than the ALSA user-space daemon, and has 
been the default audio server in both Ubuntu and Fedora Core for some time.  The 
PulseAudio driver is mostly complete, but has no 3D capability (Linux users can 
still choose to use OpenAL as an alternative).

The audio driver modules are required to pump themselves now.  The PulseAudio 
driver does this using the built-in PA thread runloop, whereas the OpenAL driver 
creates a Python thread that periodically wakes.

Much of the shared code between drivers regarding video frame synchronisation 
has been moved into the base classes.  This should greatly reduce the amount of 
effort required to develop a new audio driver.  Audio drivers dispatch events to 
the Player object when a new video frame should be displayed.  New methods on 
pyglet.app make it easy to post events from one thread into the main event loop 
safely (these also automatically wake up the main event loop if it's sleeping, 
allowing more opportunities for people to integrate multithreaded code into their 
pyglet applications).

The AVbin source driver also now spawns a worker thread for decoding video 
frames.  Since most of the work in decoding the frame is done outside of Python 
(in FFmpeg), this is a great chance for pyglet to do some real multiprocessing.  
Quite a lot of work went into ensuring the queue of video frames to be decoded is 
kept up-to-date with what's needed on-screen; without an excessive preroll period 
or memory usage.

I've done most development on my 900MHz iBook, which is incapable enough that it 
can't even play YouTube videos in real time (though pyglet can play them just 
fine once downloaded, thanks to FFmpeg).  This has forced me to make the 
implementation efficient enough to work on old computers; the overhead of the 
extra thread processing is significant and required several manual context 
switches to get everything playing (almost) as well as the old, single-threaded 
code.

On newer computers the performance is fantastic.  I did a quick benchmark test 
on my 1.8GHz Core 2 Duo Mac Mini.  It happily plays 7 SD videos simultaneously 
(DivX or Xvid encoding) without sweating, whereas the old code could only play 3 
videos.  Watching the activity monitor shows that both cores are now being 
utilised fully when needed (and run each at around 5% when playing a single 
video).

Expect to see this new code in pyglet 1.2.  I'll introduce some more of the new 
features over the next few weeks.  I'll also be talking about pyglet 1.2 at OSDC 
2008 in Sydney in December.

Good looking tick mark distribution

Mon, 11 Aug 2008 18:40:25 +1000

Given a min/max range of any magnitude, how do you generate a "nice 
looking" distribution of values to cover the range?  My need for this was to 
generate tick marks on a graph.  The function I came up with works quite nicely, 
I think:

    import math

    def distribute_nicely(min_val, max_val, min_ticks=3, max_ticks=8):
        if max_val - min_val == 0:
            return [min_val]
        mag = 10 ** int(math.log10(max_val - min_val) - .5)
        if mag == 0:
            mag = 0.1
        for multiplier in (10, 5, 2, 1, .5, .2, .1):
            interval = multiplier * mag
            if min_ticks <= (max_val - min_val) / interval <= max_ticks:
                min_val = (min_val // interval) * interval
                break
        else:
            interval = (max_val - min_val) / max_ticks or max_val - min_val
        if interval > 1:
            return range(min_val, max_val + interval, interval)
        else:
            return [i * interval  \
                    for i in range(int(min_val / interval), 
                                   int(max_val / interval) + 1)]

Sample results:

    >>> distribute_nicely(0, 10)
    [0, 2, 4, 6, 8, 10]
    >>> distribute_nicely(0, 100)
    [0, 20, 40, 60, 80, 100]
    >>> distribute_nicely(0, 5)
    [0, 1, 2, 3, 4, 5]
    >>> distribute_nicely(3, 8)
    [3, 4, 5, 6, 7, 8]
    >>> distribute_nicely(24, 786)
    [0, 200, 400, 600, 800]
    >>> distribute_nicely(0.21, 0.86)
    [0.2, 0.4, 0.6, 0.8]
    >>> distribute_nicely(-8, 8)
    [-10, -5, 0, 5, 10]
    >>> distribute_nicely(-105, 105)
    [-150, -100, -50, 0, 50, 100, 150]

I'd be interested to hear if anyone has an alternative function or improvements 
to this one.  Or if there's one in the standard library -- I always tend to miss 
those :-)

Clean termination of daemon Python threads

Sat, 09 Aug 2008 11:23:03 +1000

Daemonic threads in Python are threads that will not prevent the interpreter 
from exiting (put another way, the interpreter shuts down when all non-daemon 
threads exit).  When the interpreter shuts down, all variables are set to None, 
then each daemon thread is woken up to crash (unexpectedly finding all its 
variables, globals and builtins missing).

Here's a nifty pattern that suppresses the ugly exception trace resulting from 
these crashes being printed to the terminal:

    def worker_thread():
        try:
            .... do work ...
        except:
            if True:
                raise

This ensures ordinary exceptions are re-raised, but interpreter shutdown related 
exceptions are suppressed, as True is evaluated as None.  In the off-chance that 
some future or alternative Python interpreter doesn't have the same strange 
daemon thread shutdown behaviour, the thread will still behave correctly.