This page is a collection of news feeds from friends of mine and myself. Its not, as the name might imply, a planet of Horms, or for the most part even stuff written by me. For that, please go here.

Nearby Planets: Planet SLUG, Planet Linux Australia, Planet FP-Syd
Subscriptions: Adrian Chadd Alan Robertson Alexander Reeder Amanda Reid Anand Kumria Andrew Cowie Andy Fitzsimon Benno Chizuko Horman Chris DiBona Chris Yeoh Craige McWhirter Dave Miller Dave Ruys David Luyer Erik de Castro Lopo Horms Hugh Blemmings James Morris Jan Schmidt Jaq Jeff Waugh Jeremy Kerr John Ferlito Joseph Arruda Kfish Mark Greenaway Martin Pool Mary Gardiner Matt Palmer Mikey Neuling Nick Jenkins Ozone Pete Ryland Peter Hardy Peter Nixon Pia Waugh Raster Raz Real World Haskell Robert Collins Roger Barnes Russell Coker Rusty Sam Johnston Silvia Pfeiffer Ted T'so Tom Davies Tong Master Tractorgen - Commits Tridge Wichert Akkerman fusion94 あや

-----

September 02, 2010

Russell Coker

Raw Satire Usually Fails on the Internet

Sarcasm and satire usually don’t work on the Internet. One cause of this is the lack of out of band signalling via facial expression or tone of voice. Another issue is the fact that in real life people usually know something about the person who they listen to while on the Internet it’s most common [...]


September 02, 2010 05:48 AM

-----

September 01, 2010

Roger Barnes

Photo of the day - 02 Sep 2010 - View from Red Point Hills, Tasmania

View from Red Point Hills, Tasmania
View from Red Point Hills, Tasmania


September 01, 2010 10:46 PM

Russell Coker

Optimising the How To Vote Process

I previously wrote about my experience handing out How To Vote (HTV) cards at the federal election a couple of weeks ago [1]. One comment noted that at one polling place “all the volunteers for different candidates had combined into a single team, handing out all the cards together“, which makes sense. There is some [...]


September 01, 2010 02:48 PM

Jeff Waugh

Best (Scott Pilgrim) meta film review ever

Here’s what I’m saying: I’m a woman, I’m in my late thirties, I can’t handle first-person shooters, I’m afraid of Comic-Con, and I really, really liked Scott Pilgrim vs. The World.

I hope I’m not, you know, blowing your mind.

– Linda Holmes in her incredibly cool meta-film-review, ‘Scott Pilgrim’ Versus The Unfortunate Tendency To Review The Audience


September 01, 2010 06:39 AM

-----

August 31, 2010

Roger Barnes

Photo of the day - 01 Sep 2010 - View from Red Point Hills, Tasmania

View from Red Point Hills, Tasmania
View from Red Point Hills, Tasmania

Looking ahead, the Ironbound Range loomed large. There's no way around it, but we were lucky to be delayed by a day and not taking it on in gloomy weather.


August 31, 2010 09:53 PM

Horms

Going Solo

[Skydiving in Picton] I'm going solo, working for myself at Horms Solutions, a boutique free and open source software consultancy.


August 31, 2010 08:09 AM

Jeff Waugh

The Naked and Famous

If your eclectic tastes include legendary modern music-as-art heroes such as Radiohead, MGMT, Florence and the Machine, Nine Inch Nails and How to Destroy Angels, you simply must check out upcoming New Zealand band The Naked and Famous.

Their debut album will be released in early September, but they’ve put out some mind-blowingly good singles already… here’s Young Blood:

Update: Okay, I can’t just link the one. Here are my other favourites so far…


August 31, 2010 05:46 AM

Roger Barnes

Photo of the day - 31 Aug 2010 - View from Red Point Hills, Tasmania

View from Red Point Hills, Tasmania
View from Red Point Hills, Tasmania

Day 2 of the South Coast Track. We made our way through scrubby plains after departing the Buoy Creek campsite. At the top of the first significant ascent, Red Point Hills, we could see all the way back to the beach we had left.


August 31, 2010 12:15 AM

-----

August 30, 2010

Dave Miller

How GRO works

All modern device drivers should be doing two things, first they should use NAPI for interrupt mitigation plus simpler mutual exclusion (all RX code paths run in software interrupt context just like TX), and use the GRO NAPI interfaces for feeding packets into the network stack.

Like just about anything else in the networking, GRO is all about increasing performance. The idea is that we can accumulate consequetive packets (based upon protocol specific sequence number checks etc.) into one huge packet. Then process the whole group as one packet object. (in Network Algorithmics this would be principle P2c, Shift computation in time, Share expenses, batch)

GRO help significantly on everyday systems, but it helps even more strongly on machines making use of virtualization since bridging streams of packets is very common and GRO batching decreases the number of switching operations.

Each NAPI instance maintains a list of GRO packets we are trying to accumulate to, called napi->gro_list. The GRO layer dispatches to the network layer protocol that the packet is for. Each network layer that supports GRO implements both a ptype->gro_receive and a ptype->gro_complete method.

->gro_receive attempts to match the incoming skb with ones that have already been queued onto the ->gro_list At this time, the IP and TCP headers are popped from the front of the packets (from GRO's perspective, that actual normal skb packet header pointers are left alone). Also, the GRO'ability state of all packets in the GRO list and the new incoming SKB are updated.

Once we've committed to receiving a GRO skb, we invoke the ->gro_complete method. It is at this point that we make the collection of individual packets look truly like one huge one. Checksums are updated, as are various private GSO state flags in the head 'skb' given to the network stack.

We do not try to accumulate GRO packets infinitely. At the end of a NAPI poll quantum, we force flush the GRO packet list.

For ipv4 TCP there are various criteria for GRO matching.

Certain events cause the current GRO bunch to get flushed out. For example:

The most important attribute of GRO is that it preserves the received packets in their entirety, such that if we don't actually receive the packets locally (for example we want to bridge or route them) they can be perfectly and accurately reconstituted to the transmit path. This is because none of the packet headers are modified (they are entirely preserved) and since GRO requires completely regular packet streams for merging, the packet boundary points are known precisely as well. The GRO merged packet can be completely unraveled and it will mimmick exactly the incoming packet sequence.

GRO mainly the work of Herbert Xu. Various driver authors and others helped him tune and optimize the implementation.


August 30, 2010 08:46 PM

Russell Coker

Interesting Developments in Islamic Culture

Shereen El Feki gave an inspiring TED talk about Islamic youth culture [1]. She shows some interesting exerpts from the 4SHBAB TV network which is known as “Islamic MTV“, the music video from the US was of particularly high quality – while I expect high quality videos to be made in the US I don’t [...]


August 30, 2010 08:48 AM

-----

August 29, 2010

Roger Barnes

Photo of the day - 30 Aug 2010 - Sun Rays over the New Harbour Range, South Coast Track, Tasmania

Sun Rays over the New Harbour Range, South Coast Track, Tasmania
Sun Rays over the New Harbour Range, South Coast Track, Tasmania

As seen from Buoy Creek


August 29, 2010 10:04 PM

-----

August 28, 2010

Roger Barnes

Photo of the day - 29 Aug 2010 - Sun Rays over the New Harbour Range, South Coast Track, Tasmania

Sun Rays over the New Harbour Range, South Coast Track, Tasmania
Sun Rays over the New Harbour Range, South Coast Track, Tasmania


August 28, 2010 09:50 PM

Chris Yeoh

W510 & Ubuntu Lucid 10.04.1

My work laptop was upgraded to a Lenovo W510 recently, replacing a 3 year old T60p. The setup and install of Ubuntu 10.04.1 was fairly straightforward, just a couple of problems:

So far seems like a pretty fast machine – 4 cores (8 with HT). Should be a big improvement on the T60p.

An of course a new laptop requires new stickers (thanks Sarah!)

Chrome sticker


August 28, 2010 09:11 AM

-----

August 27, 2010

Roger Barnes

Photo of the day - 28 Aug 2010 - Sun Rays over the New Harbour Range, South Coast Track, Tasmania

Sun Rays over the New Harbour Range, South Coast Track, Tasmania
Sun Rays over the New Harbour Range, South Coast Track, Tasmania

The wind whips up dry sand over the beach while the sun puts on a show through the clouds. As seen from Buoy Creek.


August 27, 2010 10:01 PM

Mary Gardiner

Chat script for Exetel 3G

While trying to work out what was up with Wii error 32022, I was seeing if using our Exetel 3G dongle (rather than DSL) would let us update. This means that I got reasonable working PPP chatscripts for Exetel 3G.

/etc/ppp/peers/exetel-3g:

/dev/ttyUSB0
ipparam exetel1
230400
noauth
defaultroute
connect "/usr/sbin/chat -v -f /etc/chatscripts/exetel-3g"

/etc/chatscripts/exetel-3g:

ABORT 'BUSY'
ABORT 'NO CARRIER'
ABORT 'ERROR'
"" AT
OK AT&F
OK ATD*99***1#
CONNECT ""

These are an unholy combination of ideas from Ubuntu Living and etbe, since I am about 5 years too young to have had to learn the Hayes command set as a requirement to get on the 'net. (Well, a year too young perhaps, Andrew knows it.)

Setting up network address translation is left as an exercise for the reader.


August 27, 2010 09:28 AM

Wii update error 32022

There are reports of errors in the Wii's latest firmware update (4.2 to 4.3) around: that the Shop channel will tell the user that they need to do a System Update, and then the update will stall at about three-quarters done, and error 32022 will be reported.

32022 is supposed to be the error for not being able to reach the Nintendo servers, and the usual solution is to wait an hour or so for either their servers to come back up, or your connection to become stable. But in June/July a lot of people started reporting complete inability to upgrade due to this error. It hit us last night.

There are all kinds of arcane solutions to this around (check out AUDISIOJUNIOR's solution for arcane) but reports are that Nintendo tells people it's your ISP's fault. As best Andrew and I can tell Nintendo is right, it is your ISP's fault, at least in a way, although they aren't being very specific. There is a problem with the update (or perhaps with the update if it failed the first time) when you are using a transparent HTTP proxy. Most likely this is something your ISP set up.

Since getting your ISP to turn a transparent proxy off for you is usually something of a pain, you will probably find it fastest (although still very annoying) to connect your Wii to the 'net using a different provider.


August 27, 2010 09:17 AM

Russell Coker

Links August 2010

Urban Honking has an insightful article about the Arduino and suggests that it is one of the most important factors for the development of the computer industry in the near future [1]. It compares the Arduino to the Altair. Wired has an interesting article about a company that provides a satellite kit and a launch [...]


August 27, 2010 02:48 AM

-----

August 26, 2010

Dave Miller

Converting sk_buff to list_head.

I've been trying to make this happen, off and on, for at least two years now. Most of the kernel is straightforward and uses the skb_*() interfaces we have for manipulating skb objects on a list.

So for those, simply tweaking the interfaces in skbuff.h will make them all "just work".

However there are a few other spots in the kernel which manipulate the SKB list pointers directly:

I'm taking another stab at this, and hopefully I can work out these wrinkles. It'd be a really nice change because of lot of uses of "struct sk_buff_head" which don't care about the spinlock or the packet count can be converted to simply "list_head" saving serious space in various datastructures.


August 26, 2010 11:40 PM

Roger Barnes

Photo of the day - 27 Aug 2010 - Zen Rock Stack, Micro-Cairn

Zen Rock Stack, Micro-Cairn
Zen Rock Stack, Micro-Cairn

An impromptu sculpture, inspired by a breathtaking location and a lack of emails, phones, TVs and other distractions


August 26, 2010 10:07 PM

-----

August 25, 2010

Roger Barnes

Photo of the day - 26 Aug 2010 - Buoy Creek, South Coast Track, Tasmania

Buoy Creek, South Coast Track, Tasmania
Buoy Creek, South Coast Track, Tasmania

Fishing boat debris and other materials are used as markers, makeshift furniture and clotheslines.


August 25, 2010 10:07 PM

Russell Coker

A Netbook for Aircraft Navigation

There is apparently some MS-Windows software for navigating light aircraft in Australia. It takes input from a GPS device and knows the rules for certain types of common tasks (such as which direction to use when approaching an airport). My first question when I heard of this was “so if the Windows laptop crashes does [...]


August 25, 2010 11:48 AM

あや

ベトナム生活にピリオド

3年間のベトナム生活にピリオドを打ち、この度日本に帰国決定。
おそらく引っ越しは9月末か10月頭。

一応仕事の引継ぎ関係もあるが、すでに終盤に差し掛かっている(のかいないのか…)感じなので、来月1カ月はほぼ休み。


August 25, 2010 08:08 AM

-----

August 24, 2010

Roger Barnes

Photo of the day - 25 Aug 2010 - Buoy Creek, South Coast Track, Tasmania

Buoy Creek, South Coast Track, Tasmania
Buoy Creek, South Coast Track, Tasmania

The tannin coloured water is a feature of every creek on the track. Despite the colour, it tastes fine.


August 24, 2010 10:05 PM

-----

August 23, 2010

Roger Barnes

Photo of the day - 24 Aug 2010 - Buoy Creek, South Coast Track, Tasmania

Buoy Creek, South Coast Track, Tasmania
Buoy Creek, South Coast Track, Tasmania

After cruising past Point Eric, a relatively busy camp site, we negotiated our way around the Black Cliffs and continued on to Buoy Creek, our first campsite for the trip. The site was also well inhabited, so we had to make ourselves at home in a small clearing adjacent to the main camping area.


August 23, 2010 09:58 PM

Matt Palmer

My Hero!

I just have to say that this bull is my hero. That is all.


August 23, 2010 11:47 AM

-----

August 22, 2010

Roger Barnes

Photo of the day - 23 Aug 2010 - Cox Bight, South Coast Track, Tasmania

Cox Bight, South Coast Track, Tasmania
Cox Bight, South Coast Track, Tasmania

Our first sand walk saw us leave the familiar 'Tassie style' track behind for a while.


August 22, 2010 09:47 PM

Russell Coker

Telling People How to Vote

Yesterday I handed out how to vote (HTV) cards for the Australian Greens. The experience was very different to the one I had when I handed out cards for the Greens in the Victorian state election in 2006 [1]. The Labor party (ALP) hadn’t spread any gross lies about the Greens and there were no [...]


August 22, 2010 02:48 PM

Erik de Castro Lopo

LLVM Backend for DDC : Milestone #2.

For a couple of weeks after AusHac 2010 I didn't manage to find any time to working on DDC at all, but I'm now back on it and late last week I reached the second milestone on the LLVM backend for DDC. The backend now has the ability to box and unbox 32 bit integers and perform simple arithmetic operations on valid combinations of them.

Disciple code that can currently be compiled correctly via LLVM includes basic stuff like:


  identInt :: Int -> Int
  identInt a = a

  plusOneInt :: Int -> Int
  plusOneInt x = x + 1

  addInt :: Int -> Int -> Int
  addInt a b = a + b

  addInt32U :: Int32# -> Int32# -> Int32#
  addInt32U a b = a + b

  addMixedInt :: Int32# -> Int -> Int
  addMixedInt a b = boxInt32 (a + unboxInt32 b)

  cafOneInt :: Int
  cafOneInt = 1

  plusOne :: Int -> Int
  plusOne x = x + cafOneInt

where Int32# specifies an unboxed 32 bit integer and Int32 specifies the boxed version.

While writing the Haskell code for DDC, I'm finding that its easiest to generate LLVM code for a specific narrow case first and then generalize it as more cases come to light. I also found that the way I had been doing the LLVM code generation was tedious and ugly, invloving lots of concatenation of small lists. To fix this I built myself an LlvmM monad on top of the StateT monad:


  type LlvmM = StateT [[LlvmStatement]] IO

Using this I can then generate a block of LLVM code as a list of LlvmStatements and add it to the monad using an addBlock function which basically pushes the blocks of code down onto a stack:


  addBlock :: [LlvmStatement] -> LlvmM ()
  addBlock code
   = do	  state	<- get
          put (code : state)

The addBlock function is then used as the base building block for a bunch of more specific functions like these:


  unboxInt32 :: LlvmVar -> LlvmM LlvmVar
  unboxInt32 objptr
   | getVarType objptr == pObj
   = do     int32    <- lift $ newUniqueReg i32
            iptr0    <- lift $ newUniqueNamedReg "iptr0" (pLift i32)
            iptr1    <- lift $ newUniqueNamedReg "iptr1" (pLift i32)
            addBlock
                    [ Comment [ show int32 ++ " = unboxInt32 (" ++ show objptr ++ ")" ]
                    , Assignment iptr0 (GetElemPtr True objptr [llvmWordLitVar 0, i32LitVar 0])
                    , Assignment iptr1 (GetElemPtr True iptr0 [llvmWordLitVar 1])
                    , Assignment int32 (Load iptr1) ]
            return  int32


  readSlot :: Int -> LlvmM LlvmVar
  readSlot 0
   = do   dstreg    <- lift $ newUniqueNamedReg "slot.0" pObj
          addBlock  [ Comment [ show dstreg ++ " = readSlot 0" ]
                    , Assignment dstreg (Load localSlotBase) ]
          return    dstreg

  readSlot n
   | n > 0
   = do   dstreg    <- lift $ newUniqueNamedReg ("slot." ++ show n) pObj
          r0        <- lift $ newUniqueReg pObj
          addBlock  [ Comment [ show dstreg ++ " = readSlot " ++ show n ]
                    , Assignment r0 (GetElemPtr True localSlotBase [llvmWordLitVar n])
                    , Assignment dstreg (Load (pVarLift r0)) ]
          return    dstreg

  readSlot n = panic stage $ "readSlot with slot == " ++ show n

which are finally hooked up to do things like:


  llvmVarOfExp (XUnbox ty@TCon{} (XSlot v _ i))
   = do   objptr    <- readSlot i
          unboxAny (toLlvmType ty) objptr

  llvmVarOfExp (XUnbox ty@TCon{} (XForce (XSlot _ _ i)))
   = do   orig      <- readSlot i
          forced    <- forceObj orig
          unboxAny (toLlvmType ty) forced

When the code generation of a single function is complete it the list of LlvmStatement blocks is then retrieved, reversed and concatenated to produce the list of LlvmStatements for the function.

With the LlvmM monad in place converting DDC's Sea AST into LLVM code is now pretty straight forward. Its just a matter of finding and implementing all the missing pieces.


August 22, 2010 03:43 AM

-----

August 21, 2010

Roger Barnes

Photo of the day - 22 Aug 2010 - Cox Bight, South Coast Track, Tasmania

Cox Bight, South Coast Track, Tasmania
Cox Bight, South Coast Track, Tasmania

Setting off soon after a surprising number of other hikers, we made our way across the plain, past Freney Lagoon to Cox Bight.


August 21, 2010 10:07 PM

Erik de Castro Lopo

FP-Syd #27.

On Thursday August 12th, we held the 27th meeting of the Sydney Functional Programming group. The meeting was held at Google's Sydney offices and we had about 20 people show up to hear our two presenters.

First up we had Ben Lippmeier presenting the Haskell library REPA for doing high performance operations on regular, shape polymorphic, parallel arrays. Ben showed us some code for written with the REPA library. The interesting thing about the code was that even though REPA allows parallel execution on multiple cores, this parallel code is not vastly different from how someone would write the code to execute on a single code. Ben also provided some benchmarking figures comparing the multicore Haskell/REPA code performing well against single core code written in C.

Our second presenter for the evening was Simon Winwood who presented on the subject of the Template Haskell, which allows type safe, compile time meta programming. The need for templating in a powerful and flexible language like Haskell is obviously much less than in languages like C++, but still useful for some tasks like quasi-quotation. The mechanics of TH as such that it allows conversion between Haskell's concrete syntax and abstract syntax trees which can be manipulated by Haskell code. One downside of TH is that code relying on regularly breaks when new versions of the GHC compiler are released.

A big thanks to Ben and Simon for presenting and Google for providing the meeting venue and refreshments.


August 21, 2010 01:05 PM

-----

August 20, 2010

Roger Barnes

Photo of the day - 21 Aug 2010 - Looking down on the New Harbour Range, South Coast Track, Tasmania

Looking down on the New Harbour Range, South Coast Track, Tasmania
Looking down on the New Harbour Range, South Coast Track, Tasmania

The South Coast Track is one where, at times, one can see the track ahead for many kilometres


August 20, 2010 09:55 PM

-----

August 19, 2010

Roger Barnes

Photo of the day - 20 Aug 2010 - Aerial view of Point Eric, Cox Bight, South Coast Track, Tasmania

Aerial view of Point Eric, Cox Bight, South Coast Track, Tasmania
Aerial view of Point Eric, Cox Bight, South Coast Track, Tasmania

Nearing the airstrip, we started to see locations that we would see up close that afternoon.


August 19, 2010 10:00 PM

Jeff Waugh

Franklin Street Statement PDF

The Franklin Street Statement can be found on the web in the form of a fairly ugly blog post, which does not befit a document of such importance… and for that matter, neither does the printed form of that page.

I wanted a nice paper copy of the statement to put in a prominent position on my desk, and figured it would make sense to share my rendering. It may be an interesting thing to hand out at developer conferences or Software Freedom Day events.

The document is set in Free typefaces, Liberation Serif and League Gothic — appropriate, given that League is a wonderful Free alternative to Franklin Gothic! I know, I know, you see what I did there. Now… Download the Franklin Street Statement PDF!

Note: I wish the source ODF had survived the wrath of OpenOffice.org’s “eat my document” crash recovery feature, but at least the resulting PDF is fairly easy to edit. :-)


August 19, 2010 05:00 PM

Russell Coker

It’s Election Time Again

Linux People and Voting Chris Samuel (a member of LUV who’s known for his work on high performance computers and the “vacation” program) has described why he’s voting for the Greens [1]. His main reasons are the Greens strong support of human rights and for science-based policy. Paul Dwerryhouse (a member of the Australian Linux [...]


August 19, 2010 02:48 AM

-----

August 18, 2010

Roger Barnes

Photo of the day - 19 Aug 2010 - Aerial view of the Ironbound Range, South Coast Track, Tasmania

Aerial view of the Ironbound Range, South Coast Track, Tasmania
Aerial view of the Ironbound Range, South Coast Track, Tasmania

The clouds hid from view the true size of the challenge that lay ahead. Hiking over the Ironbound Range is typically completed in an 900m ascent and then descent off the other side over 8 or more hours.


August 18, 2010 09:44 PM

Raz

Gathering samples for malware analysis

I installed Microsoft’s Security Essentials today. Their security analysts have a rather direct way of receiving unknown files to analyse: shortly after the first scan, a dialogue popped up: Neat. Presumably developers creating .exe files need to turn this off eventually…


August 18, 2010 06:24 AM

James Morris

Linux Security Summit 2010 – Wrapup

The first Linux Security Summit (LSS) was held last Monday, 9th August in Boston, in conjunction with LinuxCon 2010 North America.

This event has its roots in the Linux security development community which emerged in the early 2000s, following the development of LSM and with the incorporation of a wide range of new security features into Linux. We’d previously met, as a community, in OLS BoF sessions, various conference hallway tracks, and at project-specific events such as the SELinux Symposium. There have also been very successful security mini-summits at LCA in 2008 and 2009, and a double security track at the 2009 Plumbers Conference.

This year, we tried to broaden the scope of the event as far as possible — to situate it with a more general Linux conference (than Plumbers, for example), and bring in not only developers, but the wider end-user community as well. We had great attendance from the security developer community, with pretty much all major areas of development represented, although not as many end-users as we’d hoped for. We were, however, easily able to fill up a days worth of bleeding edge technical discussions, with around 70 developers in attendance throughout.

Presentations were limited to thirty minutes, including discussion, to help ensure an interesting and stimulating event, aimed at fostering ongoing discussion and engagement. In this sense, it seems we were generally successful, with several strong discussions arising during presentations. There were many follow-up meetings between developers, end users and vendors during the remainder of LinuxCon, which was very gratifying to see.

Z. Cliffe Schreuders sparking a lively debate about security usability
Z. Cliffe Schreuders sparking a lively debate about security usability

Mobile security was one of the core issues discussed at LSS (and during the rest of the week), with the year of the Linux desktop now apparently permanently canceled due to smartphones and similar devices. There are certainly many very difficult and exciting challenges to be met in this area over the coming years, and it was great to be able to have the MeeGo security folk present on their work.

Another important area (as always), is security usability, with new high-level policy language work presented by Josh Brindle (lolpolicy). Z. Cliffe Schreuders presented the results of a comparative usability vs. efficacy study from his FBAC-LSM project, sparking some very robust and productive discussion. (Certainly from an SELinux point of view, we are trying to learn as much as possible from this kind of research, which is otherwise very thin on the ground).

Stephen Hemminger presented on the topic of integrating security into a router (Vyatta). This kind of presentation is really very useful to have when there are so many security developers present — it helps us better understand the nature & scope of security requirements for a wider range of real-world users.

Brad Spengler’s presentation addressed the difficult area of protecting the kernel itself, arising from his experiences developing grsecurity. As most of our protection mechanisms operate within the kernel, attacks on the kernel can render these mechanisms useless, so it is important to try and harden the kernel as much as possible. Brad outlined some areas which we still need to address upstream (or in distros, at least), a topic which was further developed by Kees Cook in his talk on Out of Tree security features.

IMHO, we face a number of challenges in this area: 1) core kernel developers are not always receptive to enhanced security, 2) the solutions proposed often are technically not acceptable to upstream (and require a lot of persistent reworking) and 3) we don’t have a huge pool of available expertise upstream in these areas. Kees has taken on some of the challenges here, and any additional contributors here would certainly be welcome, although I would not anticipate any smooth sailing.

We also had project updates from Mimi Zohar on EVM, Karl MacMillan on security management, Dan Walsh on SELinux Sandbox, and Stephen Gallagher on SSSD.

The panel discussion kicked off with a session on the viability of a standard Linux security API. It was good to get a discussion going here, with well-considered input from key developers. It seems the consensus is that our various security models are too fundamentally different to develop the kinds of APIs you might see in proprietary OSes, although the issues are certainly recognized (e.g. hindered ISV and end user adoption of security) and people are thinking about solutions. There are many difficult, open issues in this area, although we really don’t have the option of not solving them — as a society we’re ever increasingly reliant on computing, and thus also on its security.

Casey Schaufler leading the security API panel discussion
Casey Schaufler leading the security API panel discussion

 

There’s already been quite a lot of feedback from attendees on the format and co-location of future events. There was some talk of aiming at a more purely technical conference (e.g. Plumbers), although it seems to me that there was a great benefit in being able to assemble a critical mass of security developers alongside the other LinuxCon developer mini-summits, as well as general end users, vendors etc. A couple of people also mentioned the Collab summit, although I wonder if being invite-only may limit the overall scope of participation. We may also look at a two-day event next year, to allow for keynotes, a few selected longer talks for major new projects, and break-out sessions.

If anyone has feedback or ideas, please join the LSS mailing list and post your thoughts.

Slides from the presentation are now linked from the schedule (where available), and I’ve posted a brief photo set on flickr. If you post any photos or blogs from the event, please tag them with #lss2010, and drop me an email, so I can link to them from the web site.

Overall, it seems that we had a very productive and collaborative event, bringing together key people to discuss ongoing and emerging challenges in Linux security. Indications thus far are that we should expect to see useful developments arise out of discussions begun at this summit, in some of the areas mentioned above.

The Linux Foundation organizers seamlessly provided us with everything we could need in terms of a venue and support — allowing us to concentrate on the program itself. Many folk worked behind the scenes, but I’d like to especially thank Angela Brown, C. Craig Ross and Amanda McPherson.

Also thanks to everyone who presented and attended, and to the program committee, who worked quickly to review and evaluate all the proposals.


August 18, 2010 01:06 AM

-----

August 17, 2010

Roger Barnes

Photo of the day - 18 Aug 2010 - Aerial view of Granite Beach, South Coast Track, Tasmania

Aerial view of Granite Beach, South Coast Track, Tasmania
Aerial view of Granite Beach, South Coast Track, Tasmania

It hardly registered as we flew over this beach that we'd be hiking back this way 5 days later.


August 17, 2010 09:44 PM

Jeff Waugh

Wired? Tired? Expired.

Glyn Moody pointed out an awful article written by Chris Anderson (of Long Tail infamy) and Michael Wolff, “The Web is Dead. Long Live the Internet”. It is almost Nick Carr-esque in its pursuit of overwrought claims with little to no basis in fact.

The lunacy of this article will resonate with anyone remotely connected with the technology industry, let alone those involved in software and web development.

But here are my favourite bits of abject cluelessness:

JavaScript then, Objective-C now? HTML then, XML now? What the fuck have you been smoking, Chris Anderson? … and don’t tell me it was some “junior staffer” subbing the living crap out of your stupid lack of perspective.

… and then this little gem to finish things off, ignoring years of evidence that we enjoy temporary dalliances with convenience before returning to (often quite) revolutionary openness:

The Internet is the real revolution, as important as electricity; what we do with it is still evolving. As it moved from your desktop to your pocket, the nature of the Net changed. The delirious chaos of the open Web was an adolescent phase subsidized by industrial giants groping their way in a new world. Now they’re doing what industrialists do best — finding choke points. And by the looks of it, we’re loving it.

I’ll make my own bold, unsubstantiated claim… Wired: Tired and expired.

Glad I have that off my chest. Thank you, ball boys.

* * *

Update: Check out this hilarious take-down from Boing Boing … with facts! … and analysis! … and pretty graphs!


August 17, 2010 08:57 PM

-----

August 16, 2010

Roger Barnes

Photo of the day - 17 Aug 2010 - Aerial View of Actaeon Island, Tasmania

Aerial View of Actaeon Island, Tasmania
Aerial View of Actaeon Island, Tasmania


August 16, 2010 10:03 PM

Russell Coker

The Gift of Fear

I have just read The Gift of Fear and Other Survival Signals that Protect Us From Violence by Gavin de Becker. Like many self-help books it has a concept that can be described in a paragraph and explained in a few pages. The rest of the book shares anecdotes that help the reader understand the [...]


August 16, 2010 02:48 PM

-----

August 15, 2010

Roger Barnes

Photo of the day - 16 Aug 2010 - Aerial View of North Bruny Island, Tasmania

Aerial View of North Bruny Island, Tasmania
Aerial View of North Bruny Island, Tasmania

The cloudy sky made for great reflections off the water


August 15, 2010 09:49 PM

Jeff Waugh

Packet and Denby

This is Packet.

This is Denby.

I mention in the video that it’ll end up being “like TweetDeck”. I breezed past that a tad too quickly — it’s definitely not going to be an HTML5 clone of TweetDeck! (Turns out they’re working on one of those already.)

Instead, it will take inspiration from the multi-column approach, but hopefully improve the user experience on a number of levels:

Please comment if you have any thoughts or suggestions… crimes committed by other Twitter clients, ideas for lovely web/desktop integration, and so on. Thanks! :-)

HELLO DENBY.

* * *

Note that I’ve already had questions about StatusNet and identi.ca support. Thus far, they don’t have user streams, which was one of the main reasons behind building Denby. That said, it is entirely possible for the Denby server to poll the REST API (given that it already talks to Twitter’s) and send the results down the WebSockets tube… so, we’ll see.

Hmm. Perhaps this is the best way to build a user streams API for StatusNet anyway? The web app could send JSON messages to node.js, which could relay them to the intended users… via multiple protocols! Long-running HTTP, WebSockets, whatever. If anyone is inspired to do this, StatusNet could have a bi-directional WebSockets API before Twitter does!

* * *

Update: I made another quick video, showing a few improvements (including update bleeps — no chicken noises as yet), Denby in Firefox’s Prism environment, and sharing some thoughts about the web as a development platform. Enjoy!


August 15, 2010 01:53 PM

Matt Palmer

Anticlimax

This sounds exciting:

We think developers who want to work on the best, most dynamic, and most interesting open-source web technology project on the planet have no better place to work than here -

Fwowr, dynamic and interesting open-source web technology project! Sign me up!

working on Drupal.

That wasn't where I thought that was going. Those words obviously have very different meanings to different people. Never mind.


August 15, 2010 05:48 AM

-----

August 14, 2010

Roger Barnes

Photo of the day - 15 Aug 2010 - Aerial View near Hobart, Tasmania

Aerial View near Hobart, Tasmania
Aerial View near Hobart, Tasmania

Looking towards Lauderdale and Tranmere with the shores of Sandy Bay in the foreground


August 14, 2010 09:43 PM

John Ferlito

Linux Australia – President’s Report August 2010

It’s been just over a month since my last presidents report, which according to past presidents means that I’m doing well. Apparently the first report is the easy one, maintaining momentum is the key!

LCA2011

We are now well and truly into the run up to linux,conf.au 2011. The LCA Call for Papers, Miniconfs, Posters and Tutorials (that is quite a mouthful!), has been open since the 13th of July. The CFP closes at midnight tonight, so it’s still not too late to get a proposal in.

Everyone should spend the next 5 minutes thinking of the one person or topic they would love to be able to listen to at LCA next year. Now, go and email that person and convince them to submit something. But hurry, you don’t have long.

The Paper Review Committee will be performing an online review of all the papers over the next 2 weeks. They will then meet in Sydney for a final one day review to decide what makes it into the conference. I’ve been involved in the process for the last few years, and I can tell you that it is not an easy process. The quality of the submissions we receive for LCA each year is extremely high and it is a very difficult task to whittle down 200-300 submissions into the 90 or so proposals we have space for.

LCA2012 Bid process

Submissions for LCA2012 close on the 15th of August, which is tomorrow night! So far we have had an expression of interest from Ballarat, and the odd rumour that other cities also have some teams thinking about it.

Once the bids come in, the council will take time to review them, and then we will begin visiting each team so that they can pitch their bid to us in person why they should earn the honour of hosting the next LCA.

This year we have changed the process slightly and asked all the teams to post their submission publicly. I’m looking forward to reading the proposals and having a healthy community discussion about which city should host LCA.

Software Freedom Day

Software Freedom Day is just around the corner, being held on the 18th of September. SFD is a worldwide celebration of FOSS and also serves to educate the general public about the benefits of FOSS.

According to the SFD website, it looks like we have about 6 teams registered in Australia. Noticeably missing are most of our capital cities. Please bring up SFD at your next LUG meeting or on your LUGs mailing list and try to organise an event in your area.

This year Linux Australia will be assisting SFD teams by providing schwag from past LCAs to give away at events. You should see an email to the list with more details about this shortly.

Also don’t forget that when you register your team on the official SFD website, Software Freedom International will send out SFD schwag for you to use on the day.

Australian Treasury Department, SBR and Auskey (Update)

As I mentioned last month, I’ve been doing some work in my capacity as President as well as my day job in regards to creating an Open Source project around the Australian Treasury’s Standard Business Reporting (SBR) project.

We recently held a meeting with some representatives from the Department of Treasury, where we were able to discuss our plans and what is required to make SBR and Auskey available for the Open Source community. SBR have shown a keen interest in the project and have been quite helpful in making resources and people available to help us with the project.

SBR have also recently announced that they will be supporting Linux on the AusKey website. This has not been possible up till this point as a browser plug-in is required to be able to interact with Auskey. SBR hopes to have a solution released by the end of the year and will be initially supporting Ubuntu. This means that Australian businesses using Open Source Software will soon be able to submit their BASs online again.


August 14, 2010 02:27 AM

-----

August 13, 2010

Roger Barnes

Photo of the day - 14 Aug 2010 - Aerial View near Hobart, Tasmania

Aerial View near Hobart, Tasmania
Aerial View near Hobart, Tasmania


August 13, 2010 09:44 PM

Rusty

fcntl lock starvation and TDB

The Trivial DataBase (ccan variant here) uses fcntl locks for consistency: records are chained off a fixed-size hash table (or the free list), and a 1-byte fcntl lock at the offset of the chain head protects all records in that chain.

There’s also a tdb_lockall() function which grabs a lock across all the hash chains at once to let you do arbitrary atomic munging.  This works because fcntl() locks have an offset and length: you can lock arbitrary byte ranges.

Unfortunately, tdb_lockall() is subject to starvation, at least under Linux.  This is because the kernel merely checks whether a lock is available and gets it if it can, rather than queuing behind someone else who wants a superset of the lock.  So single byte lockers come in and out while the whole-file locker waits for them all to go away.

I’m not sure this is wrong, and as it’s simpler than the alternative, I’m not prepared to change it just yet.  Fortunately, there’s a way of avoiding this starvation in userspace, which occurred independently to both Volker Lendecke and me.  I called this variant tdb_lockall_gradual(), in which we try to lock each chain one at a time so we compete with the single-chain lockers on fair terms.  My first naive thought was to try to lock all the chains one at a time in order, nonblocking, then go back and retry (blocking) any we failed to get.  This is slow, and can deadlock against another process doing the same thing.  Volker’s suggestion was much nicer: we do a non-blocking lock, and if that fails we divide and conquer.  If we get down to a single record, we do a blocking lock.

I wrote  a test program which fires off N children, each of which grabs a random chain lock for 50-150 milliseconds before sleeping for 1 second, then repeating. The parent waits for a second, then tries to do a tdb_lockall() or tdb_lockall_gradual() depending on the commandline.  Running it five times and showing the average wait time for the lockall gives this:

Now, regarding performance.  While there can be 10000 hash chains, this isn’t as bad as it sounds.  The fast case is the uncontended one, and that’s as fast as before, and the contended case is already slow.  I annotated the source to print out how many blocking/nonblocking locks it’s actually doing.  Inevitably, if there’s contention, it will end up dividing down to a blocking lock, so log(numchains) locks before doing a blocking lock.

Processes Blocking locks Nonblocking locks Seconds
5 0-2 1-27 0.03
50 8-12 93-111 0.20
500 13-21 130-170 0.29
5000 309-347 1660-1832 9.1

Sure, that’s a lot of locking when we’re competing with 5000 processes, but it’s less the naive one per chain, and it’s clear that it’s not the cause of the slowdown (we’re doing fewer locks per second than the 5 processes case).

And anyway, locking the entire database cannot be a speed-critical operation.  Indeed, after the evidence here, I followed Volker’s suggestion to simply replace tdb_lockall() with the new implementation.


August 13, 2010 04:09 AM

-----

August 12, 2010

Roger Barnes

Photo of the day - 13 Aug 2010 - Aerial View near Hobart, Tasmania

Aerial View near Hobart, Tasmania
Aerial View near Hobart, Tasmania


August 12, 2010 09:51 PM

Rusty

Bob Jenkins and lookup8.c

I pulled Bob Jenkins’ superlative lookup3.c into ccan some time ago, and with the work on TDB2 (aka 64-bit TDB) I wondered if there was a 64-bit variant available.  I didn’t look hard enough: I should have checked his top-level page and seen lookup8.c before wasting his time with an email :(

I did note in my mail that since lookup3.c can return a pair of 32 bit numbers, I could combine those to get a 64 bit hash “but I wondered if it was possible to do better…”

His one-line reply was that he was working on a 128-bit hash now.  Respectful and informative.

The gold standard test of online manners is how you respond to a dumb random email.  A truly gracious gentleman/lady will respond as if you are far smarter far than you appear.  Thanks Bob.  And 128-bit; I’m a little stunned…


August 12, 2010 07:15 AM

-----

August 11, 2010

Roger Barnes

Photo of the day - 12 Aug 2010 - Aerial View near Hobart, Tasmania

Aerial View near Hobart, Tasmania
Aerial View near Hobart, Tasmania

After some delayed luggage caught up to our party of 3, we set off in a light plane towards Southwest National Park.


August 11, 2010 10:05 PM

Russell Coker

Ethernet Interface Naming

As far as I recall the standard for naming Linux Ethernet devices has always been ethX where X is a number starting at 0. Until fairly recently the interface names were based on the order that device drivers were loaded or the order in which the PCI bus was scanned. This meant that after hardware [...]


August 11, 2010 02:48 PM

-----

August 10, 2010

Roger Barnes

Photo of the day - 11 Aug 2010 - Sullivans Cove, Hobart, Tasmania

Sullivans Cove, Hobart, Tasmania
Sullivans Cove, Hobart, Tasmania

The view from Customs House Waterfront Hotel


August 10, 2010 10:09 PM

Mark Greenaway

One for the stats nerds

At USyd, we did all our stats in R. Now I'm working at the Department of Health, and we do most of our stats in SAS. SAS is pretty different to R, and so I've needed to work hard to try to learn it.

This is a rite of passage that most trainee biostatisticians go through, and so people have shoved various books into my hands to help me get up to speed. I'll omit the names of many of the books to protect the guilty, but the most useful book someone pressed innto my hands was The Little SAS Book, which I read cover to cover in two sittings.

The Little SAS Book is more technical than the others, hence more suitable for programmers, and actually gives you an inkling of what the designers of the language were thinking. That's helped me begin to think in the language, which is something none of the other books have helped me to do.

The best comparison I can come up with for now is that SAS is like German, whereas R is like Japanese. SAS has lots of compound statements, each of which does a lot, while R has many small statements which each do a little bit. So would you like to be able to speak German or Japanese? The correct answer is, of course, both, each at the appropriate time :)


August 10, 2010 03:37 AM

Russell Coker

Cyborgs solving Protein Folding problems

Arstechnica has an interesting article about protein folding problems being solved by a combination of brute-force software and human pattern recognition in the form of a computer game [1]. Here is a link to the primary source which also mentions the fact that players can design their own proteins which could potentially cure some diseases [...]


August 10, 2010 02:48 AM

-----

August 09, 2010

Roger Barnes

Photo of the day - 10 Aug 2010 - Tasman Island, Tasmania

Tasman Island, Tasmania
Tasman Island, Tasmania

The circular lighthouse was built in 1906, of cast-iron plates on a concrete base


August 09, 2010 10:12 PM

Russell Coker

The Lord of the Fries

Today I bought a box of fries from The Lord of the Fries [1]. I bought it from their new stand at Flinders St station because I was going past and saw no queue. In the past I had considered buying from their store on Elizabeth St but the queues were too long. The fries [...]


August 09, 2010 02:48 PM

-----

August 08, 2010

Roger Barnes

Photo of the day - 09 Aug 2010 - Tasman Island, Tasmania

Tasman Island, Tasmania
Tasman Island, Tasmania

The sun decided to arrive just in time as the cruise reached Tasman Island. Here, the base of the flying fox used to carry supplies, lighthouse keepers and their families can be seen.


August 08, 2010 09:52 PM

-----

August 07, 2010

Roger Barnes

Photo of the day - 08 Aug 2010 - Dolerite columns, Tasman Peninsula, Tasmania

Dolerite columns, Tasman Peninsula, Tasmania
Dolerite columns, Tasman Peninsula, Tasmania

On the way to Tasman Island, near Cathedral Rock


August 07, 2010 09:54 PM

Silvia Pfeiffer

WebSRT and HTML5 media accessibility

On 23rd July, Ian Hickson, the HTML5 editor, posted an update to the WHATWG mailing list introducing the first draft of a platform for accessibility for the HTML5 <video> element. The platform provides for captions, subtitles, audio descriptions, chapter markers and similar time-synchronized text both in-band (inside the video resource) and out-of-band (as external text files). Right now, the proposal only regards <video>, but I personally believe the same can be applied to the <audio> element, except we have to be a bit more flexible with the rendering approach. Anyway…

What I want to do here is to summarize what was introduced, together with the improvements that I and some others have proposed in follow-up emails, and list some of the media accessibility needs that we are not yet dealing with.

For those wanting to only selectively read some sections, here is a clickable table of contents of this rather long blog post:

THE WebSRT TIMED TEXT FORMAT

The first and to everyone probably most surprising part is the new file format that is being proposed to contain out-of-band time-synchronized text for video. A new format was necessary after the analysis of all relevant existing formats determined that they were either insufficient or hard to use in a Web environment.

The new format is called WebSRT and is an extension to the existing SRT SubRip format. It is actually also the part of the new specification that I am personally most uncomfortable with. Not that WebSRT is a bad format. It’s just not sufficient yet to provide all the functionality that a good time-synchronized text format for Web media should. Let’s look at some examples.

WebSRT is composed of a sequence of timed text cues (that’s what we’ve decided to call the pieces of text that are active during a certain time interval). Because of its ancestry of SRT, the text cues can optionally be numbered through. The content of the text cues is currently allowed to contain three different types of text: plain text, minimal markup, and anything at all (also called “metadata”).

In its most simple form, a WebSRT file is just an ordinary old SRT file with optional cue numbers and only plain text in cues:

  1
  00:00:15.00 --> 00:00:17.95
  At the left we can see...

  2
  00:00:18.16 --> 00:00:20.08
  At the right we can see the...

  3
  00:00:20.11 --> 00:00:21.96
  ...the head-snarlers

A bit of a more complex example results if we introduce minimal markup:

  00:00:15.00 --> 00:00:17.95 A:start
  Auf der <i>linken</i> Seite sehen wir...

  00:00:18.16 --> 00:00:20.08 A:end
  Auf der <b>rechten</b> Seite sehen wir die....

  00:00:20.11 --> 00:00:21.96 A:end
  <1>...die Enthaupter.

  00:00:21.99 --> 00:00:24.36 A:start
  <2>Alles ist sicher.
  Vollkommen <b>sicher</b>.

and add to this a CSS to provide for some colors and special formatting:

    ::cue { background: rgba(0,0,0,0.5); }
    ::cue-part(1) { color: red; }
    ::cue-part(2, b) { font-style: normal; text-decoration: underline; }

Minimal markup accepts <i>, <b>, <ruby> and a timestamp in <>, providing for italics, bold, and ruby markup as well as karaoke timestamps. Any further styling can be done using the CSS pseudo-elements ::cue and ::cue-part, which accept the features ‘color’, ‘text-shadow’, ‘text-outline’, ‘background’, ‘outline’, and ‘font’.

Note that positioning requires some special notes at the end of the start/end timestamps which can provide for vertical text, line position, text position, size and alignment cue setting. Here is an example with vertically rendered Chinese text, right-aligned at 98% of the video frame:

  00:00:15.00 --> 00:00:17.95 A:start D:vertical L:98%
  在左边我们可以看到...

  00:00:18.16 --> 00:00:20.08 A:start D:vertical L:98%
  在右边我们可以看到...

  00:00:20.11 --> 00:00:21.96 A:start D:vertical L:98%
  ...捕蝇草械.

  00:00:21.99 --> 00:00:24.36 A:start D:vertical L:98%
  一切都安全.
  非常地安全.

Finally, WebSRT files can be authored with abstract metadata inside cues, which practically means anything at all. Here’s an example with HTML content:

  00:00:15.00 --> 00:00:17.95 A:start
  <img src="pic1.png"/>Auf der <i>linken</i> Seite sehen wir...

  00:00:18.16 --> 00:00:20.08 A:end
  <img src="pic2.png"/>Auf der <b>rechten</b> Seite sehen wir die....

  00:00:20.11 --> 00:00:21.96 A:end
  <img src="pic3.png"/>...die <a href="http://members.chello.nl/j.kassenaar/
elephantsdream/subtitles.html">Enthaupter</a>.

  00:00:21.99 --> 00:00:24.36 A:start
  <img src="pic4.png"/>Alles ist <mark>sicher</mark>.<br/>Vollkommen <b>sicher</b>.

Here is another example with JSON in the cues:

  00:00:00.00 --> 00:00:44.00
  {
    slide: intro.png,
    title: "Really Achieving Your Childhood Dreams" by Randy Pausch,
             Carnegie Mellon University, Sept 18, 2007
  }

  00:00:44.00 --> 00:01:18.00
  {
    slide: elephant.png,
    title: The elephant in the room...
  }

  00:01:18.00 --> 00:02:05.00
  {
    slide: denial.png,
    title: I'm not in denial...
  }

What I like about WebSRT:

  1. it allows for all sorts of different content in the text cues – plain text is useful for texted audio descriptions, minimal markup is useful for subtitles, captions, karaoke and chapters, and “metadata” is useful for, well, any data.
  2. it can be easily encapsulated into media resources and thus turned into in-band tracks by regarding each cue as a data packet with time stamps.
  3. it is not verbose

Where I think WebSRT still needs improvements:

  1. break with the SRT history: since WebSRT and SRT files are so different, WebSRT should get its own MIME type, e.g. text/websrt, and file extensions, e.g. .wsrt; this will free WebSRT for changes that wouldn’t be possible by trying to keep conformant with SRT
  2. introduce some header fields into WebSRT: the format needs
    • file-wide name-value metadata, such as author, date, copyright, etc
    • language specification for the file as a hint for font selection and speech synthesis
    • a possibility for style sheet association in the file header
    • a means to identify which parser is required for the cues
    • a magic identifier and a version string of the format
  3. allow innerHTML as an additional format in the cues with the CSS pseudo-elements applying to all HTML elements
  4. allow full use of CSS instead of just the restricted features and also use it for positioning instead of the hard to understand positioning hints
  5. on the minimum markup, provide a neutral structuring element such as <span @id @class @lang> to associate specific styles or specific languages with a subpart of the cue

Note that I undertook some experiments with an alternative format that is XML-based and called WMML to gain most of these insights and determine the advantages/disadvantages of a xml-based format. The foremost advantage is that there is no automatism with newlines and displayed new lines, which can make the source text file more readable. The foremost disadvantages are verbosity and that there needs to be a simple encoding step to remove all encapsulating header-type content from around the timed text cues before encoding it into a binary media resource.

ASSOCIATING EXTERNAL TIMED TEXT RESOURCES WITH A VIDEO

Now that we have a timed text format, we need to be able to associate it with a media resource in HTML5. This is what the <track> element was introduced for. It associates the timestamps in the timed text cues with the timeline of the video resource. The browser is then expected to render these during the time interval in which the cues are expected to be active.

Here is an example for how to associate multiple subtitle tracks with a video:

  <video src="california.webm" controls>
    <track label="English" kind="subtitles" src="calif_eng.wsrt" srclang="en">
    <track label="German" kind="subtitles" src="calif_de.wsrt" srclang="de">
    <track label="Chinese" kind="subtitles" src="calif_zh.wsrt" srclang="zh">
  </video>

In this case, the UA is expected to provide a text menu with a subtitle entry with these three tracks and their label as part of the video controls. Thus, the user can interactively activate one of the tracks.

Here is an example for multiple tracks of different kinds:

  <video src="california.webm" controls>
    <track label="English" kind="subtitles" src="calif_eng.wsrt" srclang="en">
    <track label="German" kind="captions" src="calif_de.wsrt" srclang="de">
    <track label="French" kind="chapter" src="calif_fr.wsrt" srclang="fr">
    <track label="English" kind="metadata" src="calif_meta.wsrt" srclang="en">
    <track label="Chinese" kind="descriptions" src="calif_zh.wsrt" srclang="zh">
  </video>

In this case, the UA is expected to provide a text menu with a list of track kinds with one entry each for subtitles, captions and descriptions through the controls. The chapter tracks are expected to provide some sort of visual subdivision on the timeline and the metadata tracks are not exposed visually, but are only available through the JavaScript API.

Here are several ideas for improving the <track> specification:

EXPOSING A LIST OF TimedTracks TO JAVASCRIPT

This is where we take an extra step and move to a uniform handling of both in-band and out-of-band timed text tracks. Futher, a third type of timed text track has been introduced in the form of a MutableTimedTrack – i.e. one that can be authored and added through JavaScript alone.

The JavaScript API that is exposed for any of these track type is identical. A media element now has this additional IDL interface:

interface HTMLMediaElement : HTMLElement {
...
  readonly attribute TimedTrack[] tracks;
  MutableTimedTrack addTrack(in DOMString label, in DOMString kind,
                                 in DOMString language);
};

A media element thus manages a list of TimedTracks and provides for adding TimedTracks through addTrack().

The timed tracks are associated with a media resource in the following order:

  1. The <track> element children of the media element, in tree order.
  2. Tracks created through the addTrack() method, in the order they were added, oldest first.
  3. In-band timed text tracks, in the order defined by the media resource’s format specification.

The IDL interface of a TimedTrack is as follows:

interface TimedTrack {
  readonly attribute DOMString kind;
  readonly attribute DOMString label;
  readonly attribute DOMString language;
  readonly attribute unsigned short readyState;
           attribute unsigned short mode;
  readonly attribute TimedTrackCueList cues;
  readonly attribute TimedTrackCueList activeCues;
  readonly attribute Function onload;
  readonly attribute Function onerror;
  readonly attribute Function oncuechange;
};

The first three capture the value of the @kind, @label and @srclang attributes and are provided by the addTrack() function for MutableTimedTracks and exposed from metadata in the binary resource for in-band tracks.

The readyState captures whether the data is available and is one of “not loaded”, “loading”, “loaded”, “failed to load”. Data is only availalbe in “loaded” state.

The mode attribute captures whether the data is activate to be displayed and is one of “disabled”, “hidden” and “showing”. In the “disabled” mode, the UA doesn’t have to download the resource, allowing for some bandwidth management.

The cues and activeCues attributes provide the list of parsed cues for the given track and the subpart thereof that is currently active.

The onload, onerror, and oncuechange functions are event handlers for the load, error and cuechange events of the TimedTrack.

Individual cues expose the following IDL interface:

interface TimedTrackCue {
  readonly attribute TimedTrack track;
  readonly attribute DOMString id;
  readonly attribute float startTime;
  readonly attribute float endTime;
  DOMString getCueAsSource();
  DocumentFragment getCueAsHTML();
  readonly attribute boolean pauseOnExit;
  readonly attribute Function onenter;
  readonly attribute Function onexit;
  readonly attribute DOMString direction;
  readonly attribute boolean snapToLines;
  readonly attribute long linePosition;
  readonly attribute long textPosition;
  readonly attribute long size;
  readonly attribute DOMString alignment;
  readonly attribute DOMString voice;
};

The @track attribute links the cue to its TimedTrack.

The @id, @startTime, @endTime attributes expose a cue identifier and its associated time interval. The getCueAsSource() and getCueAsHTML() functions provide either an unparsed cue text content or a text content parsed into a HTML DOM subtree.

The @pauseOnExit attribute can be set to true/false and indicates whether at the end of the cue’s time interval the media playback should be paused and wait for user interaction to continue. This is particularly important as we are trying to support extended audio descriptions and extended captions.

The onenter and onexit functions are event handlers for the enter and exit events of the TimedTrackCue.

The @direction, @snapToLines, @linePosition, @textPosition, @size, @alignment and @voice attributes expose WebSRT positioning and semantic markup of the cue.

My only concerns with this part of the specification are:

RENDERING TimedTracks

The third part of the timed track framework deals with how to render the timed text cues in a Web page. The rendering rules are explained in the HTML5 rendering section.

I’ve extracted the following rough steps from the rendering algorithm:

  1. All timed tracks of a media resource that are in “showing” mode are rendered together to avoid overlapping text from multiple tracks.
  2. The timed tracks cues that are to be rendered are collected from the active timed tracks and ordered by the timed track order first and by their start time second. Where there are identical start times, the cues are ordered by their end time, earliest first, or by their creation order if all else is identical.
  3. Each cue gets its own CSS box.
  4. The text in the CSS boxes is positioned and formated by interpreting the positioning and formatting instructions of WebSRT that are provided on the cues.
  5. An anonymous inline CSS box is created into which all the cue CSS boxes are wrapped.
  6. The wrapping CSS box gets the dimensions of the video viewport. The cue CSS boxes are positioned so they don’t overlap. The text inside the cue CSS boxes inside the wrapping CSS box is wrapped at the edges if necessary.

To overcome security concerns with this kind of direct rendering of a CSS box into the Web page where text comes potentially from a different and malicious Web site, it is required to have the cues come from the same origin as the Web page.

To allow application of a restricted set of CSS properties to the timed text cues, a set of pseudo-selectors was introduced. This is necessary since all the CSS boxes are anonymous and cannot be addressed from the Web page. The introduced pseudo-selectors are ::cue to address a complete cue CSS box, and ::cue-part to address a subpart of a cue CSS box based on a set of identifiers provided by WebSRT.

I have several issues with this approach:

SUMMARY AND FURTHER NEEDS

We’ve made huge progress on accessibility features for HTML5 media elements with the specifications that Ian proposed. I think we can move it to a flexible and feature-rich framework as the improvements that Henri, myself and others have proposed are included.

This will meet most of the requirements that the W3C HTML Accessibility Task Force has collected for media elements where the requirements relate to accessibility functionality provided through alternative text resources.

However, we are not solving any of the accessibility needs that relate to alternative audio-visual tracks and resources. In particular there is no solution yet to deal with multi-track audio or video files that have e.g. sign language or audio description tracks in them – not to speak of the issues that can be introduced through dealing with separate media resources from several sites that need to be played back in sync. This latter may be a challenge for future versions of HTML5, since needs for such synchoronisation of multiple resources have to be explored further.

In a first instance, we will require an API to expose in-band tracks, a means to control their activation interactively in a UI, and a description of how they should be rendered. E.g. should a sign language track be rendered as pciture-in-picture? Clear audio and Sign translation are the two key accessibility needs that can be satisfied with such a multi-track solution.

Finally, another key requirement area for media accessibility is described in a section called “Content Navigation by Content Structure”. This describes the need for vision-impaired users to be able to navigate through a media resource based on semantic markup – think of it as similar to a navigation through a book by book chapters and paragraphs. The introduction of chapter markers goes some way towards satisfying this need, but chapter markers tend to address only big time intervals in a video and don’t let you navigate on a different level to subchapters and paragraphs. It is possible to provide that navigation through providing several chapter tracks at different resolution levels, but then they are not linked together and navigation cannot easily swap between resolution levels.

An alternative might be to include different resolution levels inside a single chapter track and somehow control the UI to manage them as different resolutions. This would only require an additional attribute on text cues and could be useful to other types of text tracks, too. For example, captions could be navigated based on scenes, shots, coversations, or individual captions. Some experimentation will be required here before we can introduce a sensible extension to the given media accessibility framework.


August 07, 2010 01:09 AM

-----

August 06, 2010

Roger Barnes

Photo of the day - 07 Aug 2010 - Dolerite columns, Cape Huay, Tasman Peninsula, Tasmania

Dolerite columns, Cape Huay, Tasman Peninsula, Tasmania
Dolerite columns, Cape Huay, Tasman Peninsula, Tasmania

The spectacular dolerite columns are a feature of the Tasman Peninsula coastline.


August 06, 2010 10:01 PM

Horms

Boston

[Billboard somewhere in the Mojhave Desert]

I will be in Boston next week to give a presentation on Network Bandwidth Control in Virtualized Environments at LinuxCon and attend the related mini-summits. Its been a while since I've attended an event in North America, I'm looking forward to it.


August 06, 2010 02:54 AM

-----

August 05, 2010

Roger Barnes

Photo of the day - 06 Aug 2010 - Dolerite columns, Cape Huay, Tasman Peninsula, Tasmania

Dolerite columns, Cape Huay, Tasman Peninsula, Tasmania
Dolerite columns, Cape Huay, Tasman Peninsula, Tasmania

The spectacular dolerite columns are a feature of the Tasman Peninsula coastline. The Candlestick and The Lanterns are popular with thrill seeking climbers, sailors, sea kakakers and surfers


August 05, 2010 10:11 PM

-----

August 04, 2010

Roger Barnes

Photo of the day - 05 Aug 2010 - Rocky Outcrop, Tasman Peninsula, Tasmania

Rocky Outcrop, Tasman Peninsula, Tasmania
Rocky Outcrop, Tasman Peninsula, Tasmania

The rather bleak conditions at the start of the trip really highlighted the isolation of the area


August 04, 2010 09:57 PM

Erik de Castro Lopo

From Gedit to Geany.

After effectively giving up on Nedit, my text editor of choice for the last fifteen years, I gave Gedit a serious try.

For a full two weeks, I stuck with Gedit, including the intense 2½ day hacking session of AusHac2010. Unfortunately, switching from a very full featured editor like Nedit to Gedit was painful. There were a bunch of features that I had grown used to that were just absent or inconvienient in Gedit. The problem is that Gedit aims to be a relatively full featured programmer's editor while still being the default easy-to-use editor in GNOME. As far as I am concerned, these two aims are in conflict, making Gedit an adequate simple text editor and a poor editor for advanced coders.

After butting my head against basic usability issues with Gedit I was even considered either modifying it extensively using plugins or maybe even forking it and maintaining a forked version. Yes, that would be a huge pain in the neck, but fortunately that will not now be necessary.

In response to my blog post titled "R.I.P. Nedit" fellow Haskell hacker and Debian Haskell Group member Joachim Breitner suggested I have a look at the Geany text editor and IDE.

Geany is obviously a tool aimed squarely as an editor for full time, committed programmers. Its also much more than just an editor, in that it has many features of an IDE (Integrated Development Environment). In fact, when I first fired it up it looked like this (click for a larger view):


Geany default window

On seeing this I initially thought Geany was not for me. Fortunately I found that the extra IDE-like features can easily be hidden, providing me with a simple-to-use, highly configurable, advanced text editor. The features I really like are:

There are still a few little niggles, but nothing like the pain I experienced trying to use Gedit. For instance, when run from the command line, Geany will open new files in a tab of an existing Geany instance. With multiple desktop workspaces, this is sub optimal. It would be much nicer if Geany would start a new instance if there was not already an instance running on the current workspace. After a brief inspection of the Gedit sources (Gedit has the desired feature), I came up with a fix for this issue which I will be submitting to the Geany development mailing list after a couple of days of testing.

Another minor issue (shared with Gedit) is that of fonts. Nedit uses bitmap fonts while Geany (and Gedit) use TrueType fonts. When I choose light coloured fonts on a black background I find the fonts in Geany (and Gedit) a lot fuzzier than the same size fonts in Nedit. I've tried a number of different fonts including Inconsolata but I've currently settled on DejaVu Sans Mono although I'm not entirely satisfied.

Currently my Geany setup (editing some Haskell code) looks like this:


Geany modified config

Light text on a black background with highlighting using a small number of colours; red for types, green for literals, yellow for keywords etc.

Geany is a great text editor. For any committed coders currently using either Nedit or Gedit and not entirely happy, I strongly recommend that you give Geany a try.


August 04, 2010 11:17 AM

Mikey Neuling

Linux Next Graphing

A while back Rusty posted about graphing the size of the daily linux-next patches.

Since we are heading towards the merge window for 2.6.33 and hence sfr has been getting home later and later, I thought I'd take another look at it.

The dodgy script I've been using to create this is out here. This also creates the raw data file which is here.

You can see some periods where there was no linux-next release, like around the 2.6.27 release. You can also see that linux-next is never zero size. Either Linus doesn't take everything in linux-next, or new stuff for the following release is coming in before the last release is done with. sfr mentioned that there is some stuff in linux-next that's been in there for ages and hasn't been merged up to Linus.

There is a difference between how Rusty got his data and how I did. Rusty used the size of the bz2 patch out on kernel.org. These patches are against the release and release candidates (ie. against 2.6.30, 2.6.30-rc1, 2.6.30-rc2, etc). I'm using the linux-next git tree to determine how big linux-next is for that day. Since sfr bases linux-next off Linus' git origin each day, I take the difference between Linus' git origin and the linux-next release to determine the size. Since Linus' origin is at least as new as the RCs, my size is never larger than Rusty's. This is especially noticeable in the merge window (the ~2 weeks between the release and rc1). In the merge window, Rusty's size continues to grow until rc1 is released, but mine starts to go down almost immediately after the main release as Linus starts merging trees into his git origin and making life easier for sfr. Also, Rusty is using patch size (bz2 compressed) and I'm using the number of lines changed (insertions + deletions).

It seems that maintainers are working/merging new code constantly throughout the cycle. Ideally (yeah, coz I'm is the authority on this!), we wouldn't see a lot of new code hit linux-next just before the merge window opens as new code should hopefully be being tested at this point. If the rate of new code was slowing down before the merge window, we'd see the line flatten to horizontally before the release. I guess we're hacking until the last minute, who would have thought!?!? ;-)

The peaks of linux-next seem to be a reasonable predictor of the relative size of the following kernel release. ie. if linux-next is bigger, so is the following release, although it's not perfect (ie. 2.6.29 vs 32)
Release Actual line changes linux-next changes linux-next/Actual %
2.6.29 1879345 1222635 (peak at 2.6.28) 65%
2.6.30 1547035 1168031 (peak at 2.6.29) 76%
2.6.31 1419059 1118892 (peak at 2.6.30) 79%
2.6.32 (-rc8) 1618369 1247456 (peak at 2.6.31) 77%

These last two ideas are interesting to combine. When a release is delayed, it's resulting in more code for the following release, since code is being developed right up until the merge window opens. So delaying a release is double edged sword. It improves the current release (more testing/debugging), but makes the follow release bigger. If we were developing earlier in the cycle and then just testing as the merge window approached, we wouldn't have this phenomenon. I suspect this is already known, but hopefully this backs it up a bit.

I haven't attempted to confirm what Rusty noticed about hackers working more on weekends but if someone wants to analyse the raw data....

Since I've got this scripted up, so I'll endeavour to keep this graph updated out here.


August 04, 2010 08:48 AM

Welcoming Liam Geoffrey and Rhys Edward Neuling

Please welcome Rhys Edward and Liam Geoffrey Neuling born at 6:00am on the 22nd day of May 2009 with weights of 2.3 and 2.8kg respectively.

Both mother and babies are doing well.


August 04, 2010 08:48 AM

Little Cool Guy

Yep, that's what happens when the Big Cool Guy has a kid. Congrats to Mel and Martin.


August 04, 2010 08:48 AM

Let's pop down to Zurich next weekend

OK, that's a pretty wanky title, but it's what Jo and I can do for the next few months while living Tubingen, Germany.

The story goes like this... about 6 months ago Arnd Bergmann suggested doing an exchange with him after LCA 2008. After thinking about it for all of about half a second, Jo and I decided to take him up on the offer. So we are spending 2 months in Tubingen, then a month traveling around Europe, then another month in Austin, all while Arnd lives in our apartment in Canberra.

We arrived in Germany about two days ago and got settled pretty quickly. We met our lovely new house mates, Ines and Sebastian, did a quick tour of Tubingen and then headed into the Boblingen offices where I'll be working.

I've setup a photo gallery here and thankfully Jo has started a blog, so you won't have to put up with me murdering the English language, just to find out what we are up to.


August 04, 2010 08:48 AM

linux.conf.au 2008 Hackfest

Get your coding pants on because it's time for the linux.conf.au 2008 Hackfest.


August 04, 2010 08:48 AM

Source Control Rulage/Suckage Ratio

It's time to bring some real science into the source control debate. Google Rulage/Suckage ratio!


August 04, 2010 08:48 AM

Howard Politics

Kristy, I completely agree that something needs to be done to stop the abuse of children. My point was about politics and process, not the policy itself.

What I do object to is the way in which Mr Howard is using this issue for political purposes. If he really considers this a national emergency, relevant leaders would have been consulted - or at least notified - of the policy prior to any announcement. Instead, Mr Howard attempted to get agreement from leaders while they learnt of his plan under media scrutiny. Doing this serves only one purpose, to paint him as a strong leader and put other leaders on the back foot. That is not about helping the children, its about winning an election.

In response to a couple of criticisms:
  • After 'children overboard' and locking kids up in detention centres for years, I don't think its wise to *trust* that Mr Howard's policy will be implemented in a way that is far better than how it initially sounds.
  • Yes, as a society we pay for child abuse. We also pay for racism, governmental overreaching and political manipulation.
  • Questions that this policy could cause long term damage for indigneous Australians, and possibly perpetuate the abuse of children, deserve decent answers - not spin.

    Also, just how regularly do need to blog to avoid having to show taste and insight? :-P


    August 04, 2010 08:48 AM

    Howard on message.... I mean, Howard on Indigenous Child Abuse

    Interviewer: So isn't your new initiative over reaching?
    Howard: Well possibly, but won't someone think of the children?!?
    
    Interviewer: OK.  So is it possible that your new laws are 
                 unconstitutional?
    Howard: Well possibly, but won't someone think of the children?!?
    
    Interviewer: Isn't it possible that you'll alienate Aboriginal
                 communities?
    Howard: Well possibly, but won't someone think of the children?!?
    
    Interviewer: Is it possible that this will cause long term
                 damage to these communities?
    Howard: Well possibly, but won't someone think of the children?!?
    
    Interviewer: Aren't you just overriding the States and Territories?
    Howard: Well possibly, but won't someone think of the children?!?
    
    Interviewer: Aren't your new laws racist by imposing
                 restrictions only to Indigenous Communities?
    Howard: Well possibly, but won't someone think of the children?!?
    
    Interviewer: What evidence do you have that shows prohibiting
                 alcohol and pornography will prevent child abuse?
    Howard: Well possibly, but won't someone think of the children?!?
    
    Interviewer: OK.  So what about the children you locked up in
                 detention centers?
    Howard: Well possibly, but won't someone think of the children?!?
    
    Interviewer: So is it true they've been throwing their children
                 overboard?	
    Howard: Well possibly, but won't someone think of the children?!?
    
    Interviewer: Doesn't this enable you to escape any legitimate
                 criticism of your policy by using political spin to claim
                 this is only about the children?
    Howard: Well possibly, but won't someone think of the children?!? 
    


    August 04, 2010 08:48 AM

    FuckWits 1.0

    People! Stop adding 2.0 to the end of names... Please!


    August 04, 2010 08:48 AM

    EuroTrash

    Jo and I spend the last 4 weeks traveling through Singapore, Spain (Seville, Granada, Valencia and Barcelona) and Sweden (Stockholm, Kiruna, Lulea, Hoga Kusten and Lake Siljan). Load of pictures here.


    August 04, 2010 08:48 AM

    Russell Coker

    Why Clusters Usually Don’t Work

    It’s widely regarded that to solve reliability problems you can just install a cluster. It’s quite obvious that if instead of having one system of a particular type you have multiple systems of that type and a cluster configured such that broken systems aren’t used then reliability will increase. Also in the case of routine [...]


    August 04, 2010 08:48 AM

    WordPress Maintainability

    For a while I’ve been maintaining my own WordPress packages. I use quite a few plugins that weren’t included in Debian, some of them have unclear licenses so they can’t go in Debian while the rest would have to go in Volatile at best because they update regularly and often have little or no information [...]


    August 04, 2010 05:48 AM

    -----

    August 03, 2010

    Roger Barnes

    Photo of the day - 04 Aug 2010 - Rocky Outcrop, Tasman Peninsula, Tasmania

    Rocky Outcrop, Tasman Peninsula, Tasmania
    Rocky Outcrop, Tasman Peninsula, Tasmania

    Departing from Eagle Hawk Neck, the Tasman Island Cruises tour takes in breathtaking scenery along the coastline of the Tasman Peninsula.


    August 03, 2010 10:05 PM

    Russell Coker

    Pre-Meeting Lightning Talks

    This evening I arrived at the LUV [1] meeting half an hour before it started. I was one of about a dozen people sitting in the room waiting, some of us had laptops and were reading email but others just sat quietly – the venue is sometimes open as much as an hour before the [...]


    August 03, 2010 02:48 PM

    -----

    August 02, 2010

    Roger Barnes

    Photo of the day - 03 Aug 2010 - Black-faced Cormorants (Phalacrocorax fuscescens)

    Black-faced Cormorants (Phalacrocorax fuscescens)
    Black-faced Cormorants (Phalacrocorax fuscescens)


    August 02, 2010 10:04 PM

    -----

    August 01, 2010

    Roger Barnes

    Photo of the day - 02 Aug 2010 - Rock Face

    Rock Face
    Rock Face


    August 01, 2010 10:07 PM

    -----

    July 31, 2010

    Roger Barnes

    Photo of the day - 01 Aug 2010 - Sea Spray

    Sea Spray
    Sea Spray


    July 31, 2010 09:44 PM

    -----

    July 30, 2010

    Roger Barnes

    Photo of the day - 31 Jul 2010 - Tasman Arch in the Rain, Tasmania

    Tasman Arch in the Rain, Tasmania
    Tasman Arch in the Rain, Tasmania


    July 30, 2010 09:59 PM

    -----

    July 29, 2010

    Roger Barnes

    Photo of the day - 30 Jul 2010 - Tasman Island, Tasmania

    Tasman Island, Tasmania
    Tasman Island, Tasmania


    July 29, 2010 09:49 PM

    Horms

    Perdition 1.19-rc3 Released

    [Container Ship]

    I have released version 1.19-rc3 of Perdition. This release fixes several problems. Perhaps most significantly, Thunderbird 3.1 can now re-connect to perdition SSL/TLS.

    The code and related libraries are available as tarballs here. More information is available in the announcement email. More information about perdition here.


    July 29, 2010 08:16 AM

    kexec-tools 2.0.2 Released

    [Sumo]

    I have released version 2.0.2 of kexec-tools. the user-space portion of kexec a soft-reboot and crash-dump facility of Linux and Xen.

    There are changes since 2.0.2-rc1.

    The code is available as a tarball here and in git here. More information is available in the announcement email.

    kexec-tools recently moved, so the URLs above are new.


    July 29, 2010 05:01 AM

    Moved

    [Coral in Miyakojima] My home page and blog have moved. The new locations are:



    July 29, 2010 03:23 AM

    -----

    July 28, 2010

    Tridge

    debugging startup problems on Ubuntu

    I recently upgraded my home server from Ubuntu Karmic to Lucid. It did not go well.

    The actual apt-get dist-upgrade went fine, with only minor problems which were easy enough to fix. The problem came when I rebooted. The boot started fine, but then got
    stuck at the purple boot page, which showed “Ubuntu 10.4″ and 5 dots which cycled
    between white and red. It never got past that point.

    The usual thing to do at this point is to reboot in single user mode and start debugging startup scripts. Unfortunately I found that single user mode with Ubuntu Lucid was not useful as it doesn’t start a shell until after a huge pile of other things are started. In my case a ’single’ boot got stuck at the same point. Getting rid of the quiet and splash options, and adding nomodeset also didn’t help.

    I found that if I booted an older kernel (2.6.31-19) then the system came up OK. That pointed to a likely driver issue. I could have just settled for that older kernel, but part of the reason for going to Lucid was to get a newer ALSA with better support for HDMI audio, so I didn’t really want to stick to an older kernel. I also wanted to know why the problem was happening.

    I was also able to get a shell using the latest kernel by using the init=/bin/bash trick, but that doesn’t help to actually debug the problem. To debug startup problems you need to be able to watch the startup process in action, to see what is waiting. This is much harder these days with the new upstart init system now used in Ubuntu, as startup is much more parallel than it used to be. Adding some echo lines to init scripts used to be a useful technique, but it is much harder to get anything sensible out of that when using upstart.

    To try to debug the problem I initially had a look for any startup debugging options. I found some promising options in /etc/default/rcS, and tried setting VERBOSE=yes and SULOGIN=yes. I found that the VERBOSE=yes option was somewhat useful, as it gave me some information on what jobs were started/waiting, but it didn’t really allow me to pin down the problem. The parallelism in upstart again made interpreting the output hard. When it says that a job is waiting it doesn’t say what it is waiting on, so you have no idea what the underlying problem really is.

    Despite the promising name, and the nice description in the rcS(5) manpage, the SULOGIN=yes option didn’t seem to do anything at all. A grep for SULOGIN in the startup scripts didn’t find any hits, so I suspect it isn’t actually implemented.

    As usual, the real key to solving the problem was a hack. I added the following to /etc/default/rcS:

    (
    /bin/sleep 10
    /sbin/ifconfig eth0 192.168.2.10 up
    /usr/sbin/sshd
    ) > /dev/null 2>&1 &

    The idea behind this hack was to allow me to login with ssh from my laptop during the startup process and watch what was going on. This worked really well and meant that I was finally able to debug the startup process with the most recent Lucid kernel.

    I rebooted again, logged into the system with ssh from my laptop, and started poking around with ps and initctl to see what was going on. I had assumed that “initctl list” would give me the information I needed. It does show what jobs are waiting, but as with the VERBOSE=yes messages it doesn’t tell you what it is waiting on.

    Poking around some more I saw 3 things that were suspicious:

    1) cryptdisks-enable was shown as “waiting”. I don’t have any encrypted disks on this system, so why should it be waiting?

    2) dmesg showed a segfault in plymouth, which is the process that asks for user input during startup (it also does splash screens). This could be linked to why cryptdisks was waiting, as its possible that cryptdisks wanted a passphrase (for what disk though? I don’t have any encrypted disks)

    3) dmesg also showed a lot of warnings from the dvb-usb-cxusb driver

    As I was running low on time I decided to try the triple whammy of removing the cryptsetup package, removing the dvb-usb and dvb-usb-cxusb drivers (by moving them out of /lib/modules and running depmod) and removing the plymouth-theme-ubuntu-text package to try to simplify plymouth. This did the trick and my system now boots fine.

    I still have the puzzle as to what is really causing the problem (and thus which of the changes matter), but I can leave that for another day. I thought it would be worthwhile sharing the ssh debug hack in case other people are also trying to debug upstart startup problems.


    July 28, 2010 11:55 PM

    Roger Barnes

    Photo of the day - 29 Jul 2010 - Port Arthur, Tasmania

    Port Arthur, Tasmania
    Port Arthur, Tasmania


    July 28, 2010 09:45 PM

    Russell Coker

    Yubikeys Have Arrived

    In my previous post about the Yubikey I suggested that computer users’ groups should arrange bulk purchases to get the best prices [1]. I ran such a buying club for Linux users in Australia as well as members of SAGE-AU [2]. The keys have arrived and I now have to start posting them out. Above [...]


    July 28, 2010 05:48 AM

    -----

    July 27, 2010

    Roger Barnes

    Photo of the day - 28 Jul 2010 - Hospital Ruins, Port Arthur, Tasmania

    Hospital Ruins, Port Arthur, Tasmania
    Hospital Ruins, Port Arthur, Tasmania


    July 27, 2010 09:54 PM

    Erik de Castro Lopo

    R.I.P. Nedit

    For serious programmers, the text editor they user is an intensely personal thing. Try suggesting to an Emacs user that they should switch to Vim or vice-versa. Most would shudder at the thought.

    My choice of editor for the last 15 years has been Nedit, the Nirvana Editor. Nedit has been an outstanding editor; feature full yet easy to use. When I first started using it, Nedit was a closed source binary-only download but sometime in the late 1990s, it was released under the GNU GPL.

    Unfortunately Nedit has been suffering from bit rot and neglect for a number of years. The main problem is that it uses the Motif widget toolkit. For open source, there are basically two options for Motif; Lesstif, an LGPL reimplementation of Motif which has been basically unmaintained for a number of years, or OpenMotif released under a license which is in no way OSI approved. On top of that, Nedit still doesn't support UTF-8, mainly because Lesstif doesn't support it.

    I have, in the past, tried to fix bugs in Nedit, but the bugs are not really in Nedit itself, but in an interaction between Nedit whichever Motif library it is linked against and the underlying X libraries. Depending on whether Nedit is linked against Lesstif and OpenMotif, Nedit will display different sets of bugs. I have tried fixing bugs in Nedit linked against Lesstif, but got absolutely nowhere. Lesstif is one of the few code bases I have ever worked on that I was completely unable to make progress on.

    With Nedit getting flakier with each passing year I finally decided to switch to a new editor. I had already discounted Emacs and Vim; switching from Nedit to either of those two archaic beasts was going to be way too painful. Of all the FOSS editors available, Gedit seemed to be the closest in features to Nedit.

    Unfortunately, Gedit does not compare well with Nedit feature wise. To me it seems to try to be simultaneously as simple as possible and to have as many features as possible and the features don't seem to fit together all that well from a usability point of view. On top of that, it lacks the following:

    On top of that Gedit could also do with some improved key bindings and some improvements to its syntax highlighting patterns. The Ocaml syntax highlighting is particularly poor.

    I'm now going to try to use Gedit, by customising its setup and and using the plugin system to see if I can regain the features that made Nedit such a pleasure to use.


    July 27, 2010 12:18 PM

    -----

    July 26, 2010

    Roger Barnes

    Photo of the day - 27 Jul 2010 - Eaglehawk Neck, Tasmania

    Eaglehawk Neck, Tasmania
    Eaglehawk Neck, Tasmania


    July 26, 2010 09:59 PM

    Russell Coker

    Creating a SE Linux Chroot environment

    Why use a Chroot environment? A large part of the use of chroot environments is for the purpose of security, it used to be the only way of isolating a user from a section of the files on a server. In many of the cases where a chroot used to be used for security it [...]


    July 26, 2010 11:48 AM

    -----

    July 25, 2010

    Russell Coker

    Links July 2010

    David Byrne gave an interesting TED talk about how changes to architecture drove changes to musical styles [1]. I think he does stretch the point a little. To a certain extent people develop the most complex instruments and the largest music halls that can be supported by the level of technology in their society – [...]


    July 25, 2010 11:48 PM

    Roger Barnes

    Photo of the day - 26 Jul 2010 - Church, Port Arthur, Tasmania

    Church, Port Arthur, Tasmania
    Church, Port Arthur, Tasmania


    July 25, 2010 09:52 PM

    Russell Coker

    SE Linux status in Debian/Squeeze

    ffmpeg I’ve updated my SE Linux repository for Squeeze to include a modified version of the ffmpeg packages without MMX support for the i386 architecture. When MMX support is enabled it uses assembler code which requires text relocations (see Ulrich Drepper’s documentation for the explanation of this [1]). This makes it possible to run programs [...]


    July 25, 2010 02:48 PM

    Erik de Castro Lopo

    FP-Syd #26.

    On Thursday July 15th, we held the 26th meeting of the Sydney Functional Programming group. The meeting was held at Google's Sydney offices and we had 18 people show up to hear our two presenters.

    First up we had your correspondent (thats me) with a presentation titled An LLVM Backend for DDC. This presentation covered the problems with the current C backend, gave a description of LLVM, the options for using LLVM from Haskell, why the LLVM code from GHC was chosen and how if fits into the DDC compile pipeline. Finally I demoed the very wonderful LLVM CGI script which allows you to enter a small C program and view the LLVM output.

    Our second presenter for the evening was Eric Willigers who presented on the subject of the ATS programming langauge. ATS is interesting because it offers functional programming with an advanced type system with things like dependent types and linear types but has excellent performance as shown on the Computer Language Benchmarks Game. Eric was able to demonstrate dependent types on a couple of list operations which certainly showed some of the promise of dependent types. ATS certainly does seem interesting but also seems to lack quite a bit of polish.

    A big thanks to Eric for presenting and Google for providing the meeting venue and refreshments.


    July 25, 2010 12:11 PM

    あや

    落ちつかない

    久々の日記更新。約半年放置。

    たまりにたまったものが爆発。
    馬鹿にされるのにも程がある。

    異動と思いきやそうじゃなくなった。
    自分で道筋立てて決めた。これから動く下準備もした。
    一言であえなくおじゃん。
    ダメなのは分かっている。
    ただ安定が欲しいだけ。

    PCが言うことを聞いてくれない。
    まだ買って2年も経っていない。
    使い方が尋常じゃないから。1年半で2回も修理。
    使い方は尋常じゃないけどせめて3年はもってくれないと。

    知人が亡くなった。こっちに来てから逢ってなかったけど
    私が日本に帰っている最中にすでに息を引き取っていたらしい。
    でも、発見が遅れた。知らせを聞いたのはベトナムに戻ってから。
    誰も何も言ってくれなかった。

    友人が出産した。
    ママになった。素敵。
    私には到底出来ない事。尊敬する。

    やっとビザが発行された。
    これでベトナム人と一緒。
    バイクも買える。家も買える。

    異動になる。今度は本当。
    道は自分で決める。何事も自分で決める。

    1工場閉鎖?
    本決定ではない。
    模索するも移転する事が決定的か・・・。

    問題発生。
    部下が起こした行動が大問題。
    自分はその管理をするべく人間。だから自分の責任。
    相手とは希薄な関係ではない。
    事情説明。疑われるのは慣れている。誤解もある。相手に頭を下げる。
    納得してくれたのか。
    普段通りの会話。普通に出来る。
    前より会話がスムーズ。これからが重要。信頼回復のために動く。

    異動が少し先延ばし。

    早いとこ安定したい。
    落ちつかない。




    July 25, 2010 06:32 AM

    Russell Coker

    SE Linux audit2allow -R and Milter policy

    Since the earliest days there has been a command named audit2allow that takes audit messages of operations that SE Linux denied and produces policy that will permit those operations. A lesser known option for this program is the “-R” option to use the interfaces from the Reference Policy (the newer version of the policy that [...]


    July 25, 2010 02:48 AM