The Question

A friend of mine recently asked the Twitterverse the following question: “Can anyone recommend a good book on designing a good software UI? What works, what doesn’t, and in which situations.”

The Simple (but ultimately unhelpful) Answer

• About Face 3.0 by Alan Cooper – Especially Part 2 (Chapters 8 to 14).
• Designing the User Interface (5th Edition) by Ben Shneiderman – The whole thing.

Keep reading for the real answer.

The Difficult Answer

I love it when people ask me this question, because it means that they are actively thinking about the Interface and they want to improve. I applaud their willingness to change. Unfortunately, wanting to change and reading a book (or two) will not get them the results they desire.

User Interface Design is a huge field of study, a speciality of decades (even centuries if you talk about Information Design) of research and learning.  There are undergraduate and graduate programs around the world that teach only this subject.  Succeeding in one of these programs is only the beginning.  Experience is what really counts in this field. A person, my friend or any person, is not going to learn this subject by reading one book.

It’s akin to me going to the bookstore and buying “The Complete Idiot’s Guide to Starting Your Own Business”.  Sure, this book will tell you the basics and give you some insights, but it’s only the tip of the iceberg.  Starting and running a business is an extremely complicated process and to think one book is going to get you there is woefully short sighted.

My problem with the original question is that it is naive.   It is naive to think that simply having the rules will allow you to effectively apply those rules.  It is naive to under-value practical experience in this field.  It is naive to treat an entire field of study as an after-thought.

The Real Question

The real question being asked is “Are there a couple of things that I can do to make my interfaces better?”

The answer to that is No … and Yes.

No, because as I’ve just said, there is no way to summarize quickly an entire field of study or years of experience.  Do not under-estimate just how valuable these things are to someone practicing in the field.

Yes, because I believe there are a number of tips that everyone can use to make their interfaces better from day one.  I’ll go into these really briefly, but I want to be very clear there’s a whole lot more to it, a whole lifetime of learning if you are willing to do it.  It’s an amazing, wonderful field and I thoroughly encourage everyone to study it.  Just be warned that it’s big, complex, and sometimes very unrewarding.

Remember: Interfaces are Everywhere

Anything people interact with is an interface: A Dictionary (the physical book kind) is an interface, a web site is an interface, a form you fill out for your employer is an interface, and a software API is an interface.  Each interface needs to be designed for the user.  So the next time you design something that someone else is going to use, or even for your own use, consider: how the interface works, how easy it is to use, and whether or not it meets your needs.

The Real Answer

While I encourage you to go and read the above books, if you do nothing else, keep the following rules in mind when you are designing any sort of user interface.

Consistency

The number one thing any software engineer can do to make interfaces better is to make things consistent.  I cannot begin to tell you how many interfaces I see that are inconsistent. (I’ve even made this mistake myself a number of times.)  If you do something one way in one part of your interface, always do it that way throughout the entire interface.  For example, if the Okay button is on the left and the Cancel button is on the right, do not change the order of these things somewhere else in your interface.

This also means adhering to the consistency norms defined by your Operating System or Operating Environment (a Web Browser is an Operating Environment).  Yes, you might not like it and you might think you can do it better, but the interface is NEVER (underlined and bolded) about you.  NEVER.

Details, Details, Details!

Anyone who does Interface Design should be horribly detailed oriented, almost compulsively so. Every aspect of your interface needs to be examined to ensure that the details are, going back to my previous point, consistent.  Form fields should be the same size, buttons the same size, everything aligned correctly, everything positioned perfectly, etc.  The details of the User Interface are the critical difference between good work and sloppy work.  And sloppy interfaces are bad interfaces.

Nothing annoys me more than going to another company and filling out their poorly designed forms.  (My current company is especially bad at this.)  These, as I mentioned before, are interfaces and spending a little time to make them more clean and more clear is worth its weight in gold.  I have, in the past, walked out of companies that were interviewing me merely because their HR forms sucked.

Know your Users

When beginning your design the first thing you should be asking yourself is what do you know about your users.  Shniederman and Cooper (the books above) can tell you a lot more about Actors and User Stories and all that, but it really just comes down to understanding how your users like to work, and how your interface is going to make some aspect of that easier.  So, get to know your users.  Are your users computer illiterate? If so, it’s not likely that they will understand something like Drag and Drop right away. Once you understand your users motivations and needs, then you can begin to design a system that best reflects them.

This is often very tricky because none of us have millions of dollars to do user studies, and shadowing, and user testing.  A lot of times our customers are abstract visions of customers.  That’s okay.  Just take some time to try and imagine (acting or role-playing training can be really helpful here) what those customers motivations and needs would be.  It’s not perfect, but it will do in a pinch.

Ease of Use

So it goes without saying that the easier to use an interface is, the more people will like it.  Of course, this must be tempered against the motivation and goals of the actual users.  So the real goal is that it must be easier for the users to do what their motivation and needs require.  I once read that Ease of Use can be defined as the number of mouse click or keyboard interactions required to perform some task.  It’s not a perfect measurement but keep it solidly in mind when designing.  And this segues into my next point…

People are Lazy

Assume that people are lazy and you will never be disappointed.  They want to do the least amount of effort for the greatest amount of payoff.    I call this the commitment factor: how much of my effort do I have to commit to receive the greatest payoff.  This is why the Lottery is so effective.  It does not seem to matter that statistics are stacked against the players, they still play because it’s easy to play and the potential reward is huge.

In interface design this is equally true.  The users will like any interface that makes things the easiest.  The converse of this, however, plays a valuable role as well… the users will like any interface that makes things easier, so long as they can control the results.  This means that lottery users like playing the lottery, so long as they can pick the numbers.  The more complicated the system to automatically pick the numbers, the less the users will like the results.

Ideally, the best systems are predictive systems that let the users control just the right amount of variables.  What is the right amount, well, that where iteration comes into play.  Try a low amount, try a high amount, calculate the best amount, and then keep refining.

Understand Flow

Cooper defines Flow as “When people are able to concentrate wholeheartedly on an activity, they lose awareness of peripheral problems and distractions.” (Cooper, 4th Edition, p119). We’ve all had these Flow moments: where what we are doing is so focused, so in-depth that we don’t notice external things such as what time it is, coworkers leaving for the day, or even phone calls from our significant others wondering why we are not home yet.  This is Flow, and it’s a very, very good thing.  Flow allows users to work on their specific need at an optimum level.  It is the goal of every good interface.

Poor interfaces interrupt flow with things like unnecessary dialogs, errors, hard to use process, etc.  The interface that interrupts less and is easier to use helps to encourage Flow.

As part of Flow I generally include visual flow in the discussion.  Visual flow is the ability of the human eye to find what it needs.  To this end, interfaces that focus or showcase what is most important to the user are better.  In the western world we read top to bottom, left to right, so items on the top left receive more attention than what is in the bottom right.  Keep this in mind as you build your interface.

Also, be aware that animated things attempting to engage or grab the users focus on your interface ALWAYS disrupt flow.  Use animation to enhance, never to engage.  Assume users have become oblivious to animated, blinking things and largely screen them out these days.

Design for Accessibility

One of the big failings of modern day design is that they fail to account for differences in human beings.  Some human beings cannot see, some cannot manipulate a mouse, some cannot determine the difference between red and blue.  Build for accessibility.  Be aware that some people view your site in really low resolution and that some view it in really high resolution.  Account for the differences in your fundamental design and from the beginning.  Going back and having to engineer your sight to meet section 508 standards can be extremely painful.  Do it right from the start.

One of the big things here is when sites use color to indicate differences.  Estimates seem to place color blindness in the US as 10 to 20% of the population.  Therefore using a color to indicate that some change has happened is not an acceptable solution.  When in doubt use a color change AND some other indicator (selection count, underlining, etc) to indicate response.

Feedback

Feedback is the process of responding to user behavior.  The more feedback, the more the user knows that they are doing things.  A common flaw is to do something without providing feedback to the user that something is occurring.  We see this in lots of User Interfaces because almost all User Interfaces rely on a single thread model wherein the interface rendering and response happen on the same thread as most processing.  The developer who pushes processing off into other threads (or into WebWorkers in the Web space) can respond with appropriate feedback to the user without waiting for the process to resolve.

Feedback is a key factor in responsiveness of a site and responsiveness is a key factor in a sites usability.  The more response a site appear, the more users feel like they are in control of how the system is behaving.

You Cannot Please Anyone

Just assume that no matter how great your user interface, not everyone is going to like it.  Instead, your goal should be to hit the 80% of user whom will like it.  There will always be edge users whom have different motivations and needs.  So upfront, identify all the users and determine what the 80% is that you can achieve.

Sure, it is possible to build an interface that scales to every type of user.  However, you will spend a disproportionate amount of time on the last 20% than on the middle 80%.  Think of a bell curve,and try to get the middle of that curve.

I know two sections back I just said to design for differences, but there is a separation between accessibility difference and designing for the edge users.

Learn from your Mistakes

Finally, learn from your mistakes.  I always believe that next version of your interface will be superior to the previous version, largely because you learn from the problems your users had with the current one and build a tighter interface for the next one.

As a co-worker of mine often says: It’s an Iterative Process.

Conclusion

So that’s what I have for you.  My long answer to a friends very simple question.  I hope I did not insult my friend, but the reality is that things are much more complicated than his initial question assumes.  That said, maybe my last section really answers the question he wanted answered.  Remember, User Interface Design is extraordinarily complicated.

A Final Note:  This topic does not take into account the whole Graphic Design aspect of UI design.  For that is an entirely differently field of study.

I recently moved arei.net to a new hosting solution.  a Little more monthly cost for a whole lot more stability and experience.  So far I am pleased and the migration could not have been easier.  Plus, the new hosting company lets me do other domains without blinking.  So all of the other domains I own (arei.me, sosay.us, and more) are now hosted with their own pages. (mind you arei.me is a mirror of arei.net and sosay.us is merely parked, but the point is that I could use them if I wanted to use them.)

Anyway, hope you like the new host and notice the performance upgrades.

I recently celebrated a birthday and the other day it dawned upon me that I’ll be approaching a milestone in my life at my next birthday. No, I won’t be 50 or some other birthday milestone. Instead, it occurred to me that with next year’s birthday *I’ll have reached 30 years of experience programming computers*. I’ll let that sink in with some of the kids out there.

It all began one fateful day in the 7th grade. I was twelve years old and not a whole heck of a lot was going on in my life. I spent most of my free time either reading books (I had just graduated into the adult section of our town library), drawing maps and plotting dungeons for my grand D&D adventures or just tilting at lawn furniture. I was a kid with an over active imagination and I gave it its free reign.

Being an avid reader I had befriended my junior high school librarian pretty quickly and was working my through the stacks there. Well, in my school’s library there were also five Commodore Pet 4000 seriescomputers. In order to use one of these you had to sign up in advance. Well, I decided to give it a whirl, and signed up for the following Wednesday, a week away. During the seven days that followed I never once gave it a moments thought as to using the computer or what to do, I just signed up and went on reading, drawing maps, and having sword fights with trees.

The day arrive of my computer usage and I showed up at the library. I was signed up for terminal 2 and I set down at it and turned it on. A few minutes later I was staring at the word “READY” and underneath it a glowing green block. I tried to type a few words and got back some sort of error, undoubtably the famous “Syntax Error” we all known and love from our BASIC days. But that was the limit of what I could do. I knew no BASIC, I knew no Commands, I had no idea what-so-ever about what to next.

Now, ever the bashful kid, I didn’t want to look stupid in front of the other kids on the 4 other computers whom were typing away furiously. So I played it cool. I typed away furiously. Syntax Error. Syntax Error. Syntax Error. I did this for about 10 minutes. I nervously glanced around to see if anyone was watching me. I furiously generated a few more Syntax Errors. I knew there was more to these computers, but I had no idea how to do anything.

I was just on the verge of giving up entirely, when the kid next to me took pity. His name was Thom and he clearly had been watching my failures out of the corner of his idea. I’d say he was probably laughing cruelly at me under his breath, but Thom just wasn’t that type of kid. After watching my struggles, he leaned over and said, “Would you like some help?”

“Sure,” I answered.

“Type in POKE 52768,32″ he told me. (Actually, I remember the 52768 number hands down, but the 32, the value being assigned to 52768 may or may not be correct.)

Instantly all the text on the screen switched from pure upper case characters to all lower case characters. It was pure magic, so much so that I still remember that memory location (52768) three decades later.

“Type in 10 PRINT “Hello” followed by a return, then 20 GOTO 10 followed by a return. Now type run.”

I did this as he was telling me, and when I typed RUN and press return, my screen lit up with an endless stream of HELLOs. It was cool.

“My name is Thom,” he said and he stuck out his hand.

“I’m Glen. What else can I do?”

And that was it. That was the magic moment.

That night I went home and drew a keyboard on the back of a piece of cardboard so I could play that I was a computer whiz like Thom. The next day I signed up to use the computer every day after school for the next week. (Sign-ups only went a week out, so I would end up having to sign-up every week for the next week for the next 18 months until I graduated to the High School.)

And Thom was my willing guide through it all. He taught me about line numbers and GOTOs and FOR loops and DATA and READ statements. He taught me how to POKE any character onto the screen at any time (POKE 32768,42 put at asterisk in the upper left hand corner of the screen.) Within a week I was formulating my own very small programs, and within a month I was doing much larger ones. I was checking out books on how to write Commodore BASIC and scouring the libraries for everything I could find. I pestered my parents for a computer on a daily basis but they never gave in. (We didn’t end up getting a computer until I was 15, when I finally went out and bought my own Timex Sinclair TS1000 http://en.wikipedia.org/wiki/Timex_Sinclair_1000 for $30 at the neighborhood CVS.)

Two months later saw the advent of my first game for the Science Fair to test hand eye coordination but really it was just me showing off that I could write a computer program. I remember that the program was about 45 lines long, a lot of input from Thom was in it, and it colossally sucked. I can still almost remember the program structure and even see it in my head. A lot of the little command details slip my mind though.

Pretty much every day since that one day almost thirty years ago I have spent programming a computer. (There was a break for about 4 years in there where I was going to school for English Literature, but even then I still spent a fair amount of time programming for an English Literature student.)

In those thirty years I’ve written so many programs I couldn’t even begin to count them. I’ve written just about every type of program you could imagine from Games to Web Browsers, in all kinds of languages like BASIC, PASCAL, Assembly, Java, Fortran, COBOL, C, Lisp, etc etc etc etc.

But in the end it all came down to that one day, thirty years ago, and that one moment where I changed all the characters on my screen from upper case to lower case. I basically performed magic for the first time using a computer, and I was hooked instantly.

Why are we still using Outlook Web Access, an internet email client that is seven years old and terrible design even for seven years ago. It pains me that my office uses a program that was bad even by the standards of 2003 when it came out. But the real question here is… why the hell hasn’t microsoft updated this piece of crap at all? Seriously? The company has billions of dollars, couldn’t they spare a quick million to rewrite the damn thing? Hell, for a mere $2 million, right now, I’ll quit my job and go write it for them. It should take about eight weeks to produce something better. Eight weeks… tops. And i’ll even write it to support IE.

Lately I’ve been the subject of what I call developer drift.

Developer Drift is the process by which an unchallenged developer slowly moves from one project to the next. The project may be in house or external or something completely fabricated by the developer’s mind, but it basically means a developer is less interested in the current project than the project over the horizon. It’s the “Grass is always greener” truism made concrete in software engineering.

For me, this has taken the form of the fact that our customer wants really boring user interfaces which I can crank out like they are nothing. Problem is, I almost never crank them out, because they are meaningless and I never feel challenged/creative by them. (For me challenged=creative.) So I take forever to implement them and tend to make a lot of excuses on why this is taking so long. I feel justified in why it takes so long in the fact that when I am challenged, I really do crank out the code at an extraordinary rate which borders on the obscene when compared to average developers. I’m very prolific when I want to be.

The same thing was true when I was back in college studying English Literature. I could crank out a paper that I found interesting in no time flat, but assign me something that was pedantic and I’d sooner rip my own teeth out with a spoon (“because it will hurt more”).

So the real question I’m trying to find an answer to is “How do you deal with developer drift?” How do you stop people from losing interest when they are bored because the project has become boring? A project I used to work on is suffering from this very problem… they want to keep the team together, but the more boring stuff they do, the less likely they are to be able to keep the team together? Is there a way for this project to challenge it’s developers at the same time as doing boring things? Would contests or “feats of skill” help keep things from getting stale? Or should they just accept this as the life cycle of the developer, assume that people are going to drift away, and prepare for the next generation?

You tell me… how does your project deal with developer drift?

“The simpler an interface looks, the faster it will seem,” – Mike Beltzner (Mozilla)

The purpose of a User Interface is information. Whether it be delivery of that information, manipulation of that information, or discovery of that information is secondary to the information itself. It’s that simple and as soon as your recognize that you can begin to build compelling experiences with the least amount of crap.

Yet time and again people whom build UI’s are forced by people whom do not build UI’s to add more. They want more widgets or charts or colors or fonts, more stuff on their site. And the more stuff you put on their site, the harder it becomes for the user to see the information they are seeking. The harder it becomes to see the forest through the trees as the saying goes.

Every time I build a UI for someone I ask them one simple question: What’s the most important thing you need to convey? Whatever that is, that piece should be 75% of the interface. I would say that in the twenty or so interfaces, only one project has ever answered that succinctly. Usually you get this long diatribe about how you should put X here and Y here and the menu needs to go here in 24 point comic sans ms. It’s usually at that point I wish I could just walk away from the project… and live on a beach in Guam. But I love my internet access too much to give it up for the beach lifestyle. Maybe in another year or two.

So the point is, that the people in charge of projects are under some sort of assumption that more is better. If you ever are in doubt of this, go look at Amazon’s front page. Too much crap to wade through just to get to a search box or your wish list or whatever. But the reality is less is better. Less is more usable. Less is faster.

So my question to you is, how do you take a customer that wants to add more trees and show them the forest?

I’m back baby! After several months of not posting, I’ve finally managed to get thing setup so I can post from work. Good news for all you out there whom have missed my diatribes on the world in which I live. You know who you are, my adoring fans, the unwashed legions of nerddom. Rejoice, for I have returned.

I’m an ANT guy. I’ve been using ANT since 1999. It’s amazingly powerful, amazingly useful and very flexible… and I’ve never once written my own ANT task. Just using the ANT tasks available or out there in the community I have been able to build hundreds of software projects.

Now, of course, everyone says, “Go Maven”. So I went Maven… and it sucked.

See, I have very strong feelings about Frameworks. (Not to confuse you here, but Maven, like ANT, is a Build Framework.) The problem with 99% of all Frameworks out there is that they force you into a specific way of doing something. “But that’s the whole point,” you scream at me. And I agree, that’s the point… until the moment you need to do something else.

Now, I use frameworks all the time in my software development, we all do. There’s one listed over on the right side in my links section that I actually endorse. So am I not hypocritical for deriding frameworks in one breathe while using them in another? Of course I am, but here’s where i’ll caveat it… I use frameworks that provide the least limitation on me doing new things. Maven, as an example is a rigid framework. Doing something new with it is difficult challenge. ANT on the other hand, while limiting, isn’t nearly as limited.

So here’s AREI’s Framework Measurement Testing System. First, evaluate a framework for the project you are working one, say building a Java Swing application. Next, evaluate the framework for another, completely different project you might work on in the future, say building a Website. How does the framework stack up for both things? Finally, consider what’s going to happen when you need to take the framework beyond it’s scope into new places. How accepting of that path is it?

Here’s some examples:

I had an ANT script that would append together a bunch of .JS files and then minify the entire set. Worked like a champ. The project I was on decided let’s give Maven a try instead of ANT. So I had to come up with a way to do the same thing in Maven. Two weeks of work later I ended up just calling ANT from inside of Maven.

Anyway, this post was really meant as just a simple link to an awesome blog posting that unleashes some much needed fury on Maven. But I got a little carried away in the intro.

So in summation, Maven sucks. Pick a framework that you can change. Damn the man! Save Empire!

I laid down some funky techno grooves. Check it: http://inudge.net/inudge#/ft8r

I have had a Google Wave Developer account for a few weeks now, but haven’t had the time to start developing with it.  Soon though.  Anyway, the point I was going to make is that recently a coworker just also got his account and he’s the first person I’ve had a live conversation with and well… it’s just fun. Yes, there are bugs and some of them are down right annoying, but once you get past those, the system really is just fun to play with.

I just finished upgrading the site to the new version of WordPress (2.8) which was a bit painful because the automatic update feature in wordpress died halfway through.  So I had to recover and manual do the upgrade, which once I figured it out was fine.

On somewhat related news, in the process of upgrading I noticed some hacker has been dropping crap into my file space.  Hi Hacker!  All crap has since been removed along with the usual processes one undertakes when they discover a hacker.

Google announced yesterday a new offering called Google Wave.  There is an excellent article at TechChrunch that gives a complete overview.  All I can say is: THIS IS HUGE PEOPLE. The concept of Google Wave, the underlying idea is exactly the right step that the web and the internet needs to take. It’s a combination of Email, Instant Messaging, Twitter, Transparency, Collaboration, Wiki, Media sharing, and so much more.  Oh, and it’s an Open API and Open Source to boot.

If you’ve ever talked tech with me in the last three years and we’ve had the discussion about what is next in technology, then we’ve had a very similar discussion about the concepts behind Google Wave.  I’m not claiming Google has once again stolen my idea, but what I will claim is that there clearly is a need for this type of product that I saw and google saw and others saw as well.

Two things in particular I want to call your attention to that make Google Wave a huge idea:

1). Adhoc groups… What adhoc grouping really does for you is to let people create internet groups as easily as they create real world groups.  Think of it this way… when you walk into your work place break room and two other people walk in and the three of you start talking, you have created a group.  It’s fast in the real world, why can it not be like that in the internet world? For most of the internet when you want to form a group and start working together there’s a fairly large investment in creating the meta systems the group needs: setting up a mailing list, setting up forums, setting up a media store, setting up a user database, etc etc.  It’s a pain in the ass really, and it make setting up a group fairly non-trivial.  To some degree, Yahoo Groups and Google Groups automates a lot of that, but then you still have to go through the effort of getting people to join etc.

What Google Wave does is to make group creation simple and almost instantaneous.  Basically, you create a group by dragging a bunch of contacts together and off you go.  You can immediately begin discussion, expand the group, whatever.  Additionally, if you need other tools for that group like a map or a document or some other thing, you can add a widget or a robot to participate in that  group and boom, you have more functionality to fill the groups need.

2) The second feature that is huge for me is transparency.  Transparency is the notion behind Twitter in that you broadcast snippets of your activities out to the world and everyone can see what you are doing.  This is the beginning of transparency though.  To take it further you need to automate transparency such that as you do things, they automatically are published.  Of course, this brings up privacy concerns, but I think there are simple ways to solve this.

I know that Wave has some level of transparency, but it’s still to early to tell how much.  I suspect that even if this is an opt in version of transparency, like Twitter, that soon there will be robots and widgets (the extensions to Wave) that will automate a lot of this functionality.

The point is, though, that transparecny can be a huge social tool… but more importantly it could be a huge business tool.  And the company that sees this is going to get a huge edge over the company that does not.

Anyways, that’s my thoughts on Google Wave.  You should efinately check it out.

Why am I never amazed when a software engineer wants to add complexity?  By this I am talking about all these abstraction layers of tools and frameworks and engines.  These tools/frameworks/engines are supposedly designed to save developers time, but really just add complexity to the stack and in the end don’t end up saving time at all.   Now, instead of just knowing programming language X I need to know Programing Language X and Abstraction Layer Y. How is working in two technologies easier than working in one?

I know some will argue that “Well you only really need to work in Abstraction Layer Y.”  Except that is never just the case.  Abstraction Layer Y is great until you have to do something that is outside the basic cases of Abstraction Layer Y.  Then you have to go back to working in Programming Language X.  Suddenly you’re working in both.

Don’t believe me?  Consider the case of Google Web Toolkit (GWT)… GWT lets you write web pages in GWT’s Java Toolkit.  It removes the need to work in HTML and Javascript and CSS.  Except, well, CSS is still use to provide your sites look and feel.  Oh, and you still use HTML inside the GWT code.  Oh, and you want to do something a little more complex than what GWT offers? Looks like you’re going to have to embed Javascript into your GWT code as well. So, instead of just having to work with HTML, CSS, and Javascript, now I have to work with HTML, CSS, Javascript and GWT.

Seriously, it’s basic math.  X + Y > X where Y is greater than 0. Yet time and again engineers are all gung ho about Abstraction Layer Y and I suspect it’s for one reason: Fear.  They’re afraid that they will have to do work, they’re afraid that their incompetence will be visible, they’re afraid that their value to the company will be realized. Fear is an extremely strong emotion and it can lead to panic, chaos and distrust.  But underneath it all is pure cowardice.  Cowardice to get things done, cowardice to admit your shortcomings, and cowardice to realize your own mediocrity.

And the worst kind of cowards are the ones that want to do trade studies.  Let us spend all of the budget and time for the project on comparing Abstraction X to Abstraction Y to Abstraction Z before we choose one.  Don’t worry though, we’ll have plenty of time and money after we run out of time and money to build the software.  Shouldn’t take more than two weeks, right?  The engineers that want to do this are the ones with the most to fear.  I believe that they would rather do trade studies than write code.  If you don’t want to write code, if you don’t LOVE to write code, you’re not a software engineer… you’re middle management.  You need to realize your fate and go work for the federal government wasting tax payer resources.

So, is there ever a case where Abstraction Layer Y makes sense? Probably, but it’s far, far less frequently than one might imagine.  Sometimes an abstraction layer can actually achieve the nirvanic state that it set out to do.  It’s kind of this sweet spot between being overly simple and overly bloated.  And I suspect that of the thousands of Abstraction Layer Y’s out there, only 1 or 2 have actually succeeded in finding that spot.

And yes, this is a little bit of a rant against Abstraction Layers… but it’s not a rant against the people who write Abstraction Layers.  The people who write Abstraction Layers are freaking genius.  Basically they have identified the fears of software engineers and are exploiting it to make money or recognition or whatever their particular motivation is.  This is brilliant in my opinion and I seriously need to start working at one of these companies.

To conclude: Abstraction Layer Y sucks and is a waste of time.  Just write the code yourself.  It’s not that hard.  If you think it is that hard, you need to quit your job now and go manage the fast food restaurant you were always destined to manage.

There has been a great deal of discussion lately about the value of minification, yet very little concert value. To that end I have decided to set down my thoughts from these discussions and share them such that everyone can be on the same page. Below I will outline what minification is and what it gains you. This is followed by discussion of minification versus HTTP Compression. Next, I will look at concept of minification as obfuscation and the usefulness of obfuscating in general. Finally, I will share my recommendation on the subject of Minification and which tools I would recommend.

MINIFICATION

To start, lets define what minification is: Minify or Minification is the process of taking some source file and compressing it to a smaller size by removing whitespace and comments. A number of other small rules are sometimes applied as part of the minify process, but these are less significant. For example, Minify will shorten variable names to two or three characters and thus gain some space there.

Consider this simple function which is not minified:

for (var i = 0; i < 100 ; i++)
{
    var randomnumber = Math.floor(Math.random() * i);
    document.write("A random number between 0 and " + i +
                   " is " + randomnumber);
}

When minification is applied this function becomes thus:

for(var i=0;i<100;i++){var randomnumber =Math.floor(Math.random()*L);document.write("A
random number between 0 and "+i+" is "+randomnumber);}

(Please note that any line breaks you see in the minified code above are due to word wrapping in your mail reader. Minified code is generally always a single line.)

In essence, minification reduces overall code size, and thus reduces the clients download time. The downside of minification is that by removing whitespace and comments and renaming some variables, the code becomes unreadable, extremely hard to debug, and may introduce unforeseen and hard to find bugs.

When we are talking about Web applications both Javascript (JS) and CSS files can be minified.

In a small test case I put two files through the minification process to see what the net gain would be. The Javascript file is of small size with few comments. The CSS is a large CSS file with few comments.

FILE SIZE                                JS                    CSS             
Bytes                                    17954                 39020
Minified Bytes                           10413                 31178
Net Gain                                 42%                   20%

As can be seen minification is much more useful to Javascript where whitespace is used much more. CSS tends to have less whitespace and thus it’s gains are lesser. Also, gains will be much larger if your code is heavily commented. Since comments are removed, minification gains are directly related to the amount of comments.

So, a 42% gain seems like a lot when you are talking about really slow network connections. Yet the question is, are those gains really worth the sacrifice in the ability to debug your code? And if one does not want to sacrifice the ability to debug and read the code, what can be done instead of minification?

HTTP COMPRESSION

HTTP Compression is a standard part of the HTTP 1.1 protocol that is in use in almost all major browsers and web servers on the market today. In essence whenever an HTTP request is made by a browser, if that browser supports HTTP Compression a special parameter is added to the outgoing request that lists the different compression algorithms that the browser can uncompress. When a Web Server sees this special parameter, it checks to see if the requested file can be compressed and if the server has one of the compression algorithms that the browser has said it can use. If all this is true, the server will run the compression algorithm over the response before it is sent back to the browser. The browser receives the compressed response and make it uncompressed before making it available.

HTTP Compression has many advantages and few disadvantages. Foremost, HTTP Compression is an entirely automated process that is handled by the server and requires only a minor configuration change on most Web Servers to enable. The Browser requires no special functionality to use compression. The disadvantage of compression is that there is a minor performance hit to both the server side and the client side. The server side must spend time compressing the response. The client side must spend time uncompressing the response. Yet, in most cases the file sizes involved make the time spent in compression minimal. If files sizes were significantly larger, compression might begin to cause performance problems.

Let us consider our test files again. This time let us look at our net gain from using HTTP compression only.

FILE SIZE                                JS                    CSS             
Bytes                                    17954                 39020
Compressed Bytes                         3906                  6708
Net Gain                                 78%                   82%

Right away its apparent that HTTP compression offers significant gains. When compared to our previous results from minification, HTTP Compression is the clear favorite. The reason for these significant gains that the HTTP Compression works across all the bytes of the source files whereas Minification largely leaves the meat of the CSS and JS files alone and works mostly on the whitespace of those files.

Given the gains of HTTP Compression the next logical step is to ask, what if I did both minification and HTTP Compression. The result is even greater improvements, but not as much as one might imagine. Consider our test files again:

FILE SIZE                                JS                    CSS             
Bytes                                    17954                 39020
Compressed & Minified                    2514                  5907
Net Gain                                 86%                   85%

Overall when employing both techniques, the gain of minification is not as significant. For our JS file there is only an 8% gain over just plain HTTP Compression. There is even less gain when looking at the CSS file. However, please remember that both the JS and CSS files in our test are very light on comments. More comments will increase the gains somewhat, but less than one might think.

Given these results, the question to be asking is does the sacrifice of being able to debug and read my Javascript and CSS outweigh my need to gain an additional 8% over just using HTTP Compression.

So maybe Minification is not the way to go after all, but what about Minification as a means of protecting my code from people whom want to steal my hard work?

OBFUSCATION

In the world of interpretive computer languages such as Java, C#, Javascript, PHP, Groovy, etc there has long been the question of how one can prevent someone from reading or stealing their code. The more openly accessible the language, the easier it is to read and copy it.

Obfuscation, is the process of applying a series of heuristics to code that will make it harder to read and more confusing to understand. The intent is that by applying these rules, the code becomes such an confusing mess that nobody would bother to read the code.

Java Obfuscation is a fairly complex process. There are hundreds of heuristics that are applied to the incoming source code. The resulting end code is a nightmare of crazy classes, method and members names that make scanning the code painful.

Obfuscation for Javascript is much less powerful. Because Javascript is meant to be a widely open language, the ability to obfuscate the code is minimal. Variables, objects and methods in Javascript have no scope privacy and thus can be accessed by anyone and are often designed that way. This means that their names cannot be obfuscated because the names have relevance to the structure. You cannot change the name of a given method, because in many cases there is no way to tell whom has to call that method or where in the code that might happen.

Yet the biggest problem of all with Obfuscation is that it is almost completely worthless. Obfuscation will only stop the most basic of people from copying your code. Extracting obfuscated code is fairly simple in most interpreted languages. With many of today’s sophisticated IDEs reformatting and stepping through code is fairly simple. Additionally, many IDEs support variable name matching such that when an obfuscated variable name is selected all of the matching uses are highlighted. All of this means taking obfuscated Javascript back to readable code is far easier than one might suspect.

Another big strike against obfuscation is that Javascript can be employed in very diverse and powerful ways. Closures, functions as objects, objects as associative arrays, and the like, can all lead up to some fairly complex Javascript. Often times this kind of programming can confuse an obfuscator that is not designed to handle the diversity of the language. Even when writing this email I managed to break one obfuscator with just the sample for loop code from above.

Obfuscation, at its best, serves to help keep honest people honest. It is not going to stop someone whose intent is to copy your code, it’s not even going to put up a decent struggle.

Minification, while not technically an obfuscator, does provide some obfuscation like processes. The removal of whitespace and the collapsing of localized variables all makes the code harder to read. Yet, not that much is really changed otherwise. Consider our sample code block that we looked at earlier. With minification obfuscation, it is still fairly readable:

for(var L=0;L<100;L++){var dS0=Math.floor(Math.random()*L);document.write("A random number between 0 and "+L+" is "+dS0);}

Given this code and about 2 minutes on the internet or a few search and replace calls and you can turn it back into fairly readable code as shown here:

for (var L = 0; L < 100; L++) {
    var dS0 = Math.floor(Math.random() * L);
    document.write("A random number between 0 and" + L + "is" + dS0);
}

It’s not a perfect match of our original code, but it is fairly close and takes little real work to achieve.

Ultimately, Obfuscation can have some utility in giving you a very basic first line of defense, but the ability to debug and read your code is reduced and in some instances obfuscation can add to code size, albeit a very small amount. These limitations must be carefully considered before undertaking obfuscation. What are you really trying to do when you obfuscate your code?

Unfortunately, there is no way to really protect your Javascript code on the internet. The design of HTML, Javascript, and CSS is all meant to be plain text readable formats and this means that absolutely nothing stands between the site code and the end user. The only real protection you have from people reading and copying your code is in how much the worry about your legal recourses.

CONCLUSIONS

So where does this leave us with regards to Minification?

Well, we showed that from a size compression stand point, minification has some small value, but not nearly as much value as just turning on HTTP Compression. From an obfuscation standpoint we outlined the fundamental weakness of obfuscation in general. Given these factors my recommendation is for careful consideration of whether or not minification is actually needed in the particular case you are making.

I would ask myself these questions:

1). How important is the ability to read and debug my code?

2). How much time and resources do I wish to invest in tracking down bugs in minified code?

3). How relevant is the commenting in my code to the ability to debug and read my code?

4). With Obfuscation, whom am I trying to protect my code from and how competent are they?

Ideally, I think the best application is to do no Obfuscation, HTTP Compression, and a reduced form of Minification in which only comments are removed from the code. This provides you with high compression and still maintains the ability to read and debug your code. In my opinion obfuscation really is not worth the effort to employ it for the benefit you receive from it.

CONFIGURATION NOTES

HTTP Compression can be turned on in either Tomcat of Apache Web Server with just a little bit of effort. Tomcat requires a mere 3 lines of code to get HTTP Compression working. Apache HTTP Server requires a few more lines, but it is still relatively easy. For more information about configuring tomcat see http://viralpatel.net/blogs/2008/11/enable-gzip-compression-in-tomcat.html. For more information about configuring Apache Web Server see http://httpd.apache.org/docs/2.0/mod/mod_deflate.html.

TOOLS NOTES

Finally, I will plug the YUI Compressor. There are a lot of Minification programs out there, but YUI Compressor seems to allow for the most configurability. Also, I liked YUI Compressors ability to minify both CSS and JS files within one simple program. You can find out more about YUI Compressor at http://developer.yahoo.com/yui/compressor/.

There’s only so much I can do to stop you from making poor decisions.  I tried to dissuade you, I tried to use the weight of my 28 years of programming expertise, I tried pointing out the compelling reasons not to do it.  Yet, still you go on.  I’m only going to yell into the wind so much before I give up and let you proceed into the pit of folly into which you have decided to jump.  I hope it all works out for you down in there, but I think you are wasting you time.  That’s all I’m going to say on the subject…

At least until I get to say “I told you so.”

I upgraded WordPress to a new version.  Let me know if you see any problems.

I found this an interesting concept.  The question of the day, however, is will it work?

So I found a fun little bug in IE 6 just now.  Don’t know if it is in IE 7 or 8.

Essentially if you have an element as shown:

<input type="checkbox" id="test-checkbox" />

And do the call:

document.getElementById("test-checkbox").checked

The result will not always reflect the checked state of the element.  The reason for this is the lack of a “name” attribute in the input element.  IE 6 appears to reset the state of any checkbox form element that is missing a name attribute under certain conditions.  Adding the “name”  attribute solves this problem.

Spent 2 hours on that gem.

I closed down the LiveJournal site that I have been maintaining for the past 8 years or so as this site basically duplicates it.  That and I never posted to it.  So it was largely a waste to keep it open at all.  There are few friends over in the LJ world that will lose keeping track of me, but I’ll still keep track of them, so it’s all good.

I was thinking the other day that web development is way too complicated.  My current project uses Servlets, JSPs, HTML, CSS, and Javascript and moving between each of these can be woefully complex.  So the question that came to me is, why not just do everything in one language?  Web pages could be written entirely in Javascript instead of the HTML/CSS/JS mix we use now.  Add in Phobos and you can replace most of the Servlet/JSP layer with Javascript as well.  So why isn’t this done? Why do we force ourselves to change gears constantly as we develop our sites out.

I suspect part of the problem is the inadequacies of Javascript: it’s lack of true Object Oriented design, it’s classless structure, it’s lack of package organization, it’s inability to support importing of other packages, and other things that are limiting JavaScript.  Another part of the problem is the historical perspective; people have been doing it the way we do it for years, so why change now.  Finally using a single language would minimize all of the strong features of the other languages that we are not using such as CSS value being completely removed.

Yet, there are some real benefits to a single language website.  First, the development readability factor is obvious.  Also consider that by using a single language, better design patterns can be enforced across all of the code.  Another thought that if a single language is used, cross browser compatibility becomes less of an issues because the compatibility problems apply only to that single language, not a set of 3 or 5 languages.

I think there’s some value in a single language approach.  I think I’ll give it a try on a small project in the near future.