Nowaday multiprocessor machine memory levels are more and more hierarchical: Opteron systems are just NUMAs, bicore chips can have a shared cache level, HyperThreaded logical processors share almost everything, ... How to properly schedule non-regular scientific computing on such machines ?!
The basic idea I've developped is providing programmers a way to express how threads of their application relate together: bubbles. A bubble expresses for instance that some threads work on the same set of data, that they often communicate together, ... so that they should be schedule in the same "corner" of the machine ; in a hierarchical manner.
I have developped an API that permits to manipulate these bubbles with a high level of abstraction. That way, people can experiment different distribution schedulers without having to care about hardware details for instance. They can really focus on algorithmic issues.
I have developped "bubble schedulers" that manipulates such hierarchy of bubbles: spreading the computation load while keeping affinities into account, gang scheduling, work stealing. Trainees could experiment some other strategies: favoring affinities above all, taking into account the size of data, how it is shared and the access rate, ... All this in a way that can automatically adapt itself to any hierarchical machine!
This is being developped within Marcel, the efficient, portable and flexible thread library of the PM2 project.