Simon Wright

Building a markup language: part two

Previously, I've talked in general terms about Whirlcode 2, which is a mark-up language tailored for use on whirlpool.net.au, and coded in JavaScript.

One of the most challenging part of the code was to ensure reliable handling of excessively long strings of characters, which — if allowed to pass unfiltered — can play havoc with HTML page layouts. Not difficult, you might think: just search for long words and litter them with breakable characters? Not so fast — you're dealing with text that contains mark-up.

• Mark-up shouldn't count towards the size of a word. If the word length limit is 25, then we don't want to penalise "un<b><i>flapping</i></b>believable". So we can't count them. And HTML entities (such as "é") should only count as a single character.
• Similarly, mark-up shouldn't act as an escape sequence for long string handling. A space within an <a href> tag shouldn't be treated as a space within the document.
• And when we get around to littering these long words with breakable characters, we need to be absolutely sure we're not inserting them inside mark-up, HTML, or entities.
• Finally, we need performance to be good. You can brute-force just about any rationally solvable problem, but we want this code to run frequently, both on the server and on client machines.

Here's what I came up with:

var inputString = txt;
var outputArray = [];
var maxColumns = 30;
var spaceMatchRegexp = /^((<[^>]+>|[^<\s]){0,30}\s+)+/;
var ptagMatchRegexp = /^<\/?[p|h1|h2|h3|ul|ol|li|hr|br]>/;
var entityMatchRegexp = /^&\w+?;/;
var charCount = 0;
var howMuchToGrab = 0;
while(inputString.length) {

    // Short-circuit where near/at end
    if(inputString.length < maxColumns) {
        outputArray.push(inputString);
        break;
    }

    var spaceMatches = inputString.match(spaceMatchRegexp);
    var ptagMatches = inputString.match(ptagMatchRegexp);

    // skip to next candidate area
    if(spaceMatches && spaceMatches[0]) {
        howMuchToGrab = spaceMatches[0].length;
        charCount = 0;
    // jump html tags
    } else if(inputString.charAt(0) == "<" && inputString.indexOf(">") > -1) {
        howMuchToGrab = inputString.indexOf(">");
        if (inputString.match(ptagMatchRegexp)) charCount = 0; // if it's a paragraph tag, reset the character count
    // jump entities
    } else if(inputString.match(entityMatchRegexp) ) { 
        howMuchToGrab = (inputString.indexOf(";") > -1) ? 1+inputString.indexOf(";") : inputString.length;
        charCount++;
    // normal character
    } else {
        howMuchToGrab = 1;
        charCount++;
    }

    // move chunk from input to output 
    outputArray.push(inputString.substring(0, howMuchToGrab));
    inputString = inputString.substring(howMuchToGrab);

    // if the limit is hit, add a word break
    if(charCount >= maxColumns) {
        outputArray.push("<wbr>");
        charCount = 2; // not sure why but testing bares this out
    }

}
txt = outputArray.join("");

The little gem in this code is the following regular expression: ^((<[^>]+>|[^<\s]){0,30}\s+)+. It knows how to skip across large tracts of acceptable content, while intelligently ignoring HTML tags. It's the optimisation that makes the rest of the code go fast.

When this regexp isn't matched, the code loops over each printable character, periodically dropping <wbr> tags as it goes.