Simon Wright

Building a markup language: part one

User-generated content sites like Wikipedia and Whirlpool rely heavily on the ability for users to submit richly and contextually marked up content; everything from references to quotations, from links to lists. And this ability needs to be given to a large array of users who don't know HTML, or worse still, know too much. So we build syntax that gives users the ability to provide rich content within the parameters we set — without opening the doors to unwanted, nuisance or malicious mark-up.

Wikipedia's answer is wikitext, a comprehensive (though occasionally bewildering) syntax which makes it possible to have a generally consistent feel to most articles. It still requires authors to follow the guidelines, but a lot of heavy lifting and boilerplate is handled by the parser.

Whirlpool has a similar need. We've got thousands of new posts every day, hundreds of private messages, a job board and our own Wiki — all needing rich text handling of one sort or another. It's an important component to get right.

Up till now, Whirlpool's answer was Whirlcode. It's a fairly simple mark-up that is also difficult to trigger accidentally. Tags like [*bold*], [/italics/] and ["quotes"] aren't necessarily intuitive, but once observed are easily learned and retyped. However over the past couple of weeks I've been hard at work on Whirlcode 2, a major rewrite of the parser that handles this mark-up. And yes, I've written it in JavaScript.

What features could a parser support? Here's a list of the ones I've specifically dealt with in the development of Whirlcode 2:

• Manage white-space. Users should not be able to post large chunks of blank space. Whirlcode 2 does this by differentiating between line breaks and paragraph breaks, and not allowing consecutive usage.
• Permit a limited subset of HTML. This must be done in a whitelisting fashion, as there is practically no limit to the variety of ways unmanaged HTML could screw up your pages. Whirlcode achieves this by escaping all HTML tags then progressively un-escaping tags which are permitted.
• Implement a custom syntax. Sure, HTML can do bold and italics, but you'll need your own mark-up for special items like quotations, wiki links, and colouring. We've got some fun ones as well, such as a tag to turn a string of words into a linked google search.
• Bulleted and numbered lists are nice to have. You can either let the correct HTML through, or support your own short-hand such as that offered by Textile.
• Break long words. If someone posts a ridiculously long word, it can often stretch and sometime break the page layout. Identifying and safely handling this is a particularly difficult thing to do, as you need to make sure you're not going to ruin mark-up in the process, nor have mark-up bypassing the long word test. I'll have more to say on this in a future post.
• You can detect common ascii-isms and replace them with more suitable mark-up. For example, (c) could be replaced with ©
• Users will frequently paste URLs and email addresses straight into their content. Automatically linking up these addresses is a useful convenience.
• Censor certain words. No regular expression can match every insult, and context often plays a larger part than we realise, but you can pick out the most egregious for special treatment. Generally, you'd replace the identified words with stars or #^%@ing ascii. I like to replace them with different, harmless words.
• Strip out unwanted unicode. Yes, it's wonderful how text strings can now contain every character from every language. But there are some which are only used to irritate.

Check out this live-in-browser example:

= Features of Whirlcode 2 ----- * [/Traditional/] ["Whirl"]['code'] [#support#] * <b>Limited</b> <small>HTML</small> * http://automatic.link/parsing *# Nested lists *#* Up to three levels *# And it creates good HTML * (c) 2008 Entity Recognition(tm) -- not 1/2 bad! * Plus headings, lines, blockquotes and more

I will be writing more about the specific algorithms developed as part of Whirlcode 2 in a future post.