A history of web development

See also A history of browsers and Richard McManus's The Evolution of Corporate Web Sites.

On this page I give a short, personal history of web development. It runs from 1998, when I started as a professional, until the present day. It's not really exhaustive or objective, but I hope it'll set you thinking about the past few years.

In 1996 the Internet went BOOM. This had several unexpected consequences.

Where the WWW had up until then been the preserve of a limited number of geeks and semi-geeks, it was now discovered by the general public, or rather, by people who expected it to be discovered by the general public any moment.

The Internet boom, being a hype, is difficult to explain in rational terms. It was a kind of huge expectation that everybody had to believe in it, or pretend to believe in, because everybody else expected them to. This led to the hype becoming even more hypey. The net practical result was that lots of companies with lots of money decided they needed websites.

Few actually knew why they needed a website, but everyone was convinced that not having a website would spell immediate disaster. Don't expect me to explain this bit, it has always baffled me, though I gratefully accepted the money.

The invention of web developers

So there were eager buyers for websites. Not surprisingly, sellers quickly appeared on the scene. Unfortunately neither buyers nor sellers could really describe what they were buying and selling. Describing websites in terms of websites wasn't yet possible because theory hadn't advanced far enough and very few people understood the new medium.

Instead, to make sense to clients (indeed, to any general audience) websites had to be described in terms of design or technology.

Designers and programmers started a loud clamour that they understood what websites were all about. And all of them were right, from their own perspective that was severely limited by lack of experience with the medium.

Although designers and programmers had all the ideas and received all the money, most of them didn't really want to work on the technical side of things. This was mostly a matter of job status: designers were not supposed to foul their hands on actually making the websites. Programmers by and large felt the same: HTML, CSS and JavaScript are 'easy', therefore low status. Serious programmers don't do that stuff.

The wonder is that some programmers and a very few designers did make the jump to web development, that is, to actually implementing the plans of the other designers and programmers and dealing with various and serious technical problems.

These few programmers and designers were strengthened by a contingent of newbies who loved the Web but didn't come from design or programming. Instead they came from all kinds of odd corners of society, for instance from ancient history and teaching, like myself.

The sandwich hold

In the early days designers and programmers squeezed us web developers in an unbreakable sandwich hold.

The designers thought they knew everything, even though most of them came from print and were unable to look beyond its rigidly defined borders. Designers were primarily interested in fonts and colours, and they were very good with them.

They understood web development better than the web developers because HTML and JavaScript are 'easy', and Adobe GoLive generated bloat code to their heart's content.

When they needed expert help they demanded that their sites looked exactly the same in all browsers no matter the cost, and they waxed eloquently about nested tables being the perfect vehicle for the subtle drop shadow effects of their beautiful conceptual structures.

The programmers also thought they knew everything, even though most of them came from applications and databases and were unable to look beyond their rigidly defined borders. Programmers were primarily interested in complicated applications, and they were very good with them.

They also understood web development better than the web developers because HTML and JavaScript are 'easy', and their applications generated bloat code to their heart's content.

When they needed expert help and you asked where their applications hid the HTML templates, the answer ran into the dozens of objects, and they waxed eloquently about nested tables being the perfect vehicle for the clever abstraction layers of their beautiful modular structures.

These two groups didn't really care a fig about web development, except as an intermediate layer to show their great online concepts, creative or technical. This intermediate layer was essentially irrelevant, and it didn't really matter how it was constructed.

Of course the two groups hated each other, too, for intruding on each other's monopoly of 'understanding the Web'.

And web development was supposed to create a link between their worlds. It was supposed to make the application show the design and make the design show the application data.

Hardly an enviable position.

Technical problems

Of course we were supposed to solve serious technical problems, too. To get any website working we needed to carefully feel our way through a marsh of browser incompatibilities.

This was the time of the Browser Wars when Netscape and Microsoft made a distinct point of being as incompatible as possible with each other while still supporting the same WWW. Netscape 3 and Explorer 3 were different, and Netscape 4 and Explorer 4 were a whole lot more different.

Each supported interesting extras. JavaScript was enhanced critically by the invention of intermediate DOMs, and CSS support became serious, though far from complete. Nonetheless these new technologies weren't much used, for three reasons worth pointing out:

  1. Obviously, incompatibility was the first reason. When your page worked in Netscape, it didn't yet work in Explorer, and vice versa. This was the cause of many heartaches, decisions to support only one browser, and the rise of incompatibility documentation sites like this one.
  2. Secondly, legacy browsers became a problem. Not all users upgraded to Netscape or Explorer 4 immediately after they appeared on the market. Therefore a site had to support the Version 3 browsers, too, for the moment.
  3. Finally, the new technologies couldn't deliver what they'd promised.

Because of points 2 and 3, web developers did not incorporate many newer technologies into the sites they created. Instead, they used HTML tag soup. This course of action neatly evaded point 1.

All these problems combined to give the source code of the average site the look of an exploded Web design book shelf. Silly tags were all over the place, CSS was hardly used, JavaScript ran into the hundreds of lines.

We were quick to blame the browsers. And true, as long as they supported wildly different versions of everything they were hard to work with. Nonetheless part of the trouble came from an approach to making web sites forced on us and our own inability to see that there was more in the world than the sandwich hold.

DHTML demasque

The most important example of idiotic expectations is DHTML.

In 1997 it was announced as the greatest thing since the invention of hairdressers, but on closer inspection the dozens of 'revolutionary' DHTML sites that popped up all over the Net turned out to consist of a bunch of layers moving across computer screens for no apparent purpose. So much for revolution.

The failure of DHTML in bringing on the 'next revolution' shows that people were getting tired of using new technologies just for the wow-factor. They are all fine and dandy, but you have to know what to do with them. As a result, the initial reception of CSS and the W3C DOM was not as warm as it could have been.

Nonetheless that wasn't a bad thing. The DHTML demasque taught web developers to ask 'Why should I use it?', and demanded clear answers before investing time in learning new technologies. This forced proponents of innovation to give good reasons for using new possibilities.

In 1999 I discoverd what a great tool DHTML could be if you use it correctly. I learned to show and hide elements by changing their display declaration from 'block' to 'none'. Especially in large navigations this made a lot of sense. The effect still forms the core of my navigation frame.

To me, this proved that DHTML is quite useful for small localized effects that enhance the usability of a page. I used it from then on.

Quiet

Surprisingly, the Internet turned out not to be a huge world-bestriding cash cow after all. Clients became more sensible in spending their money, and less balanced web development companies went broke or struggled hard to keep their heads above the water.

In fact this was a Good Thing. Not having to deal with ignorant clients and consultants who only cared about websites as creators of revenue and bonuses, or designers and programmers who thought they understood it all, gave us a quiet spell to start thinking.

Websites had become just another product that had to prove its worth in an increasingly tight market. Users have never gone to websites to see 'killer applications' or a wow-factor Flash movie, they want either information or entertainment. During the boom nobody cared, but when the dust settled down people started to recognize this; more than before, anyway.

The dot.crash removed many opportunists and rank amateurs from the scene, leaving the more serious, sensible and professional web developers to seriously think about what we want with websites.

What we needed first of all was a theory.

A theory

In 1998, web developers took action in their own name for the first time. The Web Standards Project started calling for full browser adherance to the W3C standards.

The immediate cause was the ridiculous proprietariness of JavaScript, especially, and CSS to a lesser degree. Although web standards had been created and proclaimed, few browsers cared. That had to change.

But browser vendors, even Microsoft, started paying serious attention to the standards. Of course 'paying serious attention to' is not the same as 'supporting perfectly', but it was a significant advance.

WaSP could never have convinced Microsoft entirely on its own, though. For reasons unknown Microsoft must have been receptive to criticism in those days, or it might already have decided to move closer to the standards. Unfortunately I lack further data on this interesting episode.

WaSP's real accomplishment, though, was convincing not browser vendors but web developers of the need for a break with the past, of the need for a new approach.

It was the first one to offer a theory, a concept for creating websites. This theory put the website first. Pure design and server side applications came distinctly second.

It broke the sandwich hold of designers and programmers on web developers, at least in our own minds. It told us what we'd known all along: that we're better in creating websites than designers and programmers.

Now THAT was an interesting development.

Keep it simple

WaSP, and independent web developers sympathizing with its goals, showed how to create websites with simpler, cleaner code than the eternal nested tables everyone had become used to. Some developers eagerly switched to the new style; others were more skeptical.

The new style had to surmount a considerable inertia; some web developers didn't see any reason to change their coding habits. It took two years before the new style made a significant impact on the web development community, but once that was done inertia began to work to its advantage: there were more web developers to spread the word, and they reached more and more others.

Nested tables and other aberrations were on the way down, CSS was on the way up. In addition to these W3C standards in a narrow sense, web developers also started to pay serious attention to usability and accessibility.

Although usability has existed almost as long as websites have existed, it had never been particularly important before and during the Browser Wars. A flashy design and high-tech 'solutions' nobody needed were what the users supposedly wanted.

As to accessibility, it was a wholly new concept and initially met with skepticism.

All these changes in coding practices can be summarized as 'keeping it simple'.

Standard zealots

Nonetheless one ugly feature of that period (and, in fact, of the standards debate in any period) is generally forgotten: the advent of standard zealots, silly boys (rarely girls) who loudly proclaimed their standards orthodoxy in annoying terms and with a snotty and superior tone of voice.

These boys (whom I often suspect of being 'hard' programmers with the full contempt for 'easy' stuff like HTML) are usually immature and wish to prove their maturity by shouting a lot. They show a surfeit of zeal and a total lack of social skills, and they consider it their sacred mission to come down hard on any non-standard behaviour.

They show more than a passing resemblence to the archetypical intolerant Christian missionary who believes that he is the sole repository of divine knowledge and understanding, and that everyone who disagrees with so much as a comma of Holy Writ is a heretic who really ought to be burned at the stake.

The zealots' unholy ardour caused the standards and the new coding style they stand for to be taken less seriously. I found interesting evidence for this theory in a review of Zeldman's "Designing with Web Standards" on Slashdot. The author says:

He [Zeldman] is obviously well-educated with regard to the subject [web standards], and his passion for the work really shows through. Still, he never comes across as a zealot -- his style is even-handed, thoughtful, and easy to comprehend.

The author expects his audience to associate web standards with zealotry. He moves quickly to defuse this expectation and to suggest that this book is different:
"This guy talks about web standards, and he isn't even a raving maniac zealot! In fact, he's sensible. That's unusual, and interesting."

I suspect that the zealots' narrow-minded nonsense has chased more than one serious web developer away from the standards, so they didn't have to hear the zealots' endless militant speechifying any more. Thus the boys did their avowed cause great harm, as do all zealots.

Back to the roots

The combined effect of the standardization effort and the lack of confidence in new technologies led to a movement which I'd like to describe as 'Back to the roots'. Websites should be simple, and should do exactly what they need to do, without new technology for the sake of new technology.

The purpose of a website is to offer its visitors information or entertainment. Any technology that helps to fullfill this purpose is welcome, any technology that doesn't help should go.

Web developers learned to ask good questions, too. They challenged the early CSS evangelists: "CSS is a standard? So what? Why should I use it?".Many leading web developers eagerly took up this challenge by publishing examples and writing books. They found good answers by the dozen.

CSS has become accepted as a full-fledged Web technology, but only after having proven its worth. That's a refreshing change of perspective when compared to the DHTML nonsense we had to go through back in '98.

In fact, it's the way things ought to work. Someone devises a new technology, and subsequently web developers have to be convinced that the new technology is a good idea. No paternalist "we know better" attitude either, just give us the facts and we'll do the judging.

A change of perspective

Back during the Browser Wars web developers were wholly led by the silly expectations of designers and programmers. The dropshadow doesn't work? Insert more tables! Explorer 4 doesn't support document.layers? Then we have to change our JavaScript, and quickly, too!

Contrarily, nowadays web developers want to create websites that pay attention to web standards, browser compatibility, accessibility and usability. They have to constant juggle these factors in their minds, and take appropriate decisions. The wishes and desires of designers and programmers have taken a second place, though when they're also our paymasters we're sometimes forced to compromise.

Web developers have become less slavish in following new technologies for the sake of the wow-factor (though I suspect the current RSS fever to contain more than a little bit of these old instincts).

Above I said that CSS had to prove its worth before being accepted by the web development community. It has succeeded beyond reasonable doubt. Now the W3C DOM will have to deliver a similar proof of its usefulness. I hope to personally create some startling evidence.