arei.net
Abstracted Fear in Software Engineering

Why am I never amazed when a software engineer wants to add complexity?  By this I am talking about all these abstraction layers of tools and frameworks and engines.  These tools/frameworks/engines are supposedly designed to save developers time, but really just add complexity to the stack and in the end don’t end up saving time at all.   Now, instead of just knowing programming language X I need to know Programing Language X and Abstraction Layer Y. How is working in two technologies easier than working in one?

I know some will argue that “Well you only really need to work in Abstraction Layer Y.”  Except that is never just the case.  Abstraction Layer Y is great until you have to do something that is outside the basic cases of Abstraction Layer Y.  Then you have to go back to working in Programming Language X.  Suddenly you’re working in both.

Don’t believe me?  Consider the case of Google Web Toolkit (GWT)… GWT lets you write web pages in GWT’s Java Toolkit.  It removes the need to work in HTML and Javascript and CSS.  Except, well, CSS is still use to provide your sites look and feel.  Oh, and you still use HTML inside the GWT code.  Oh, and you want to do something a little more complex than what GWT offers? Looks like you’re going to have to embed Javascript into your GWT code as well. So, instead of just having to work with HTML, CSS, and Javascript, now I have to work with HTML, CSS, Javascript and GWT.

Seriously, it’s basic math.  X + Y > X where Y is greater than 0. Yet time and again engineers are all gung ho about Abstraction Layer Y and I suspect it’s for one reason: Fear.  They’re afraid that they will have to do work, they’re afraid that their incompetence will be visible, they’re afraid that their value to the company will be realized. Fear is an extremely strong emotion and it can lead to panic, chaos and distrust.  But underneath it all is pure cowardice.  Cowardice to get things done, cowardice to admit your shortcomings, and cowardice to realize your own mediocrity.

And the worst kind of cowards are the ones that want to do trade studies.  Let us spend all of the budget and time for the project on comparing Abstraction X to Abstraction Y to Abstraction Z before we choose one.  Don’t worry though, we’ll have plenty of time and money after we run out of time and money to build the software.  Shouldn’t take more than two weeks, right?  The engineers that want to do this are the ones with the most to fear.  I believe that they would rather do trade studies than write code.  If you don’t want to write code, if you don’t LOVE to write code, you’re not a software engineer… you’re middle management.  You need to realize your fate and go work for the federal government wasting tax payer resources.

So, is there ever a case where Abstraction Layer Y makes sense? Probably, but it’s far, far less frequently than one might imagine.  Sometimes an abstraction layer can actually achieve the nirvanic state that it set out to do.  It’s kind of this sweet spot between being overly simple and overly bloated.  And I suspect that of the thousands of Abstraction Layer Y’s out there, only 1 or 2 have actually succeeded in finding that spot.

And yes, this is a little bit of a rant against Abstraction Layers… but it’s not a rant against the people who write Abstraction Layers.  The people who write Abstraction Layers are freaking genius.  Basically they have identified the fears of software engineers and are exploiting it to make money or recognition or whatever their particular motivation is.  This is brilliant in my opinion and I seriously need to start working at one of these companies.

To conclude: Abstraction Layer Y sucks and is a waste of time.  Just write the code yourself.  It’s not that hard.  If you think it is that hard, you need to quit your job now and go manage the fast food restaurant you were always destined to manage.

permalink: http://www.arei.net/archives/104
Minification

There has been a great deal of discussion lately about the value of minification, yet very little concrete value. To that end I have decided to set down my thoughts from these discussions and share them such that everyone can be on the same page. Below I will outline what minification is and what it gains you. This is followed by discussion of minification versus HTTP Compression. Next, I will look at concept of minification as obfuscation and the usefulness of obfuscating in general. Finally, I will share my recommendation on the subject of Minification and which tools I would recommend.

MINIFICATION

To start, lets define what minification is: Minify or Minification is the process of taking some source file and compressing it to a smaller size by removing whitespace and comments. A number of other small rules are sometimes applied as part of the minify process, but these are less significant. For example, Minify will shorten variable names to two or three characters and thus gain some space there.

Consider this simple function which is not minified:

for (var i = 0; i <; 100 ; i++)
{
    var randomnumber = Math.floor(Math.random() * i);
    document.write("A random number between 0 and " + i +
                   " is " + randomnumber);
}

When minification is applied this function becomes thus:

for(var i=0;i<;100;i++){var randomnumber =Math.floor(Math.random()*L);document.write("A
random number between 0 and "+i+" is "+randomnumber);}

(Please note that any line breaks you see in the minified code above are due to word wrapping in your mail reader. Minified code is generally always a single line.)

In essence, minification reduces overall code size, and thus reduces the clients download time. The downside of minification is that by removing whitespace and comments and renaming some variables, the code becomes unreadable, extremely hard to debug, and may introduce unforeseen and hard to find bugs.

When we are talking about Web applications both Javascript (JS) and CSS files can be minified.

In a small test case I put two files through the minification process to see what the net gain would be. The Javascript file is of small size with few comments. The CSS is a large CSS file with few comments.

FILE SIZE                                JS                    CSS             
Bytes                                    17954                 39020
Minified Bytes                           10413                 31178
Net Gain                                 42%                   20%

As can be seen minification is much more useful to Javascript where whitespace is used much more. CSS tends to have less whitespace and thus it's gains are lesser. Also, gains will be much larger if your code is heavily commented. Since comments are removed, minification gains are directly related to the amount of comments.

So, a 42% gain seems like a lot when you are talking about really slow network connections. Yet the question is, are those gains really worth the sacrifice in the ability to debug your code? And if one does not want to sacrifice the ability to debug and read the code, what can be done instead of minification?

HTTP COMPRESSION

HTTP Compression is a standard part of the HTTP 1.1 protocol that is in use in almost all major browsers and web servers on the market today. In essence whenever an HTTP request is made by a browser, if that browser supports HTTP Compression a special parameter is added to the outgoing request that lists the different compression algorithms that the browser can uncompress. When a Web Server sees this special parameter, it checks to see if the requested file can be compressed and if the server has one of the compression algorithms that the browser has said it can use. If all this is true, the server will run the compression algorithm over the response before it is sent back to the browser. The browser receives the compressed response and make it uncompressed before making it available.

HTTP Compression has many advantages and few disadvantages. Foremost, HTTP Compression is an entirely automated process that is handled by the server and requires only a minor configuration change on most Web Servers to enable. The Browser requires no special functionality to use compression. The disadvantage of compression is that there is a minor performance hit to both the server side and the client side. The server side must spend time compressing the response. The client side must spend time uncompressing the response. Yet, in most cases the file sizes involved make the time spent in compression minimal. If files sizes were significantly larger, compression might begin to cause performance problems.

Let us consider our test files again. This time let us look at our net gain from using HTTP compression only.

FILE SIZE                                JS                    CSS             
Bytes                                    17954                 39020
Compressed Bytes                         3906                  6708
Net Gain                                 78%                   82%

Right away its apparent that HTTP compression offers significant gains. When compared to our previous results from minification, HTTP Compression is the clear favorite. The reason for these significant gains that the HTTP Compression works across all the bytes of the source files whereas Minification largely leaves the meat of the CSS and JS files alone and works mostly on the whitespace of those files.

Given the gains of HTTP Compression the next logical step is to ask, what if I did both minification and HTTP Compression. The result is even greater improvements, but not as much as one might imagine. Consider our test files again:

FILE SIZE                                JS                    CSS             
Bytes                                    17954                 39020
Compressed & Minified                    2514                  5907
Net Gain                                 86%                   85%

Overall when employing both techniques, the gain of minification is not as significant. For our JS file there is only an 8% gain over just plain HTTP Compression. There is even less gain when looking at the CSS file. However, please remember that both the JS and CSS files in our test are very light on comments. More comments will increase the gains somewhat, but less than one might think.

Given these results, the question to be asking is does the sacrifice of being able to debug and read my Javascript and CSS outweigh my need to gain an additional 8% over just using HTTP Compression.

So maybe Minification is not the way to go after all, but what about Minification as a means of protecting my code from people whom want to steal my hard work?

OBFUSCATION

In the world of interpretive computer languages such as Java, C#, Javascript, PHP, Groovy, etc there has long been the question of how one can prevent someone from reading or stealing their code. The more openly accessible the language, the easier it is to read and copy it.

Obfuscation, is the process of applying a series of heuristics to code that will make it harder to read and more confusing to understand. The intent is that by applying these rules, the code becomes such an confusing mess that nobody would bother to read the code.

Java Obfuscation is a fairly complex process. There are hundreds of heuristics that are applied to the incoming source code. The resulting end code is a nightmare of crazy classes, method and members names that make scanning the code painful.

Obfuscation for Javascript is much less powerful. Because Javascript is meant to be a widely open language, the ability to obfuscate the code is minimal. Variables, objects and methods in Javascript have no scope privacy and thus can be accessed by anyone and are often designed that way. This means that their names cannot be obfuscated because the names have relevance to the structure. You cannot change the name of a given method, because in many cases there is no way to tell whom has to call that method or where in the code that might happen.

Yet the biggest problem of all with Obfuscation is that it is almost completely worthless. Obfuscation will only stop the most basic of people from copying your code. Extracting obfuscated code is fairly simple in most interpreted languages. With many of today's sophisticated IDEs reformatting and stepping through code is fairly simple. Additionally, many IDEs support variable name matching such that when an obfuscated variable name is selected all of the matching uses are highlighted. All of this means taking obfuscated Javascript back to readable code is far easier than one might suspect.

Another big strike against obfuscation is that Javascript can be employed in very diverse and powerful ways. Closures, functions as objects, objects as associative arrays, and the like, can all lead up to some fairly complex Javascript. Often times this kind of programming can confuse an obfuscator that is not designed to handle the diversity of the language. Even when writing this email I managed to break one obfuscator with just the sample for loop code from above.

Obfuscation, at its best, serves to help keep honest people honest. It is not going to stop someone whose intent is to copy your code, it's not even going to put up a decent struggle.

Minification, while not technically an obfuscator, does provide some obfuscation like processes. The removal of whitespace and the collapsing of localized variables all makes the code harder to read. Yet, not that much is really changed otherwise. Consider our sample code block that we looked at earlier. With minification obfuscation, it is still fairly readable:

for(var L=0;L<;100;L++){var dS0=Math.floor(Math.random()*L);document.write("A random number between 0 and "+L+" is "+dS0);}

Given this code and about 2 minutes on the internet or a few search and replace calls and you can turn it back into fairly readable code as shown here:

for (var L = 0; L <; 100; L++) {
    var dS0 = Math.floor(Math.random() * L);
    document.write("A random number between 0 and" + L + "is" + dS0);
}

It's not a perfect match of our original code, but it is fairly close and takes little real work to achieve.

Ultimately, Obfuscation can have some utility in giving you a very basic first line of defense, but the ability to debug and read your code is reduced and in some instances obfuscation can add to code size, albeit a very small amount. These limitations must be carefully considered before undertaking obfuscation. What are you really trying to do when you obfuscate your code?

Unfortunately, there is no way to really protect your Javascript code on the internet. The design of HTML, Javascript, and CSS is all meant to be plain text readable formats and this means that absolutely nothing stands between the site code and the end user. The only real protection you have from people reading and copying your code is in how much the worry about your legal recourses.

CONCLUSIONS

So where does this leave us with regards to Minification?

Well, we showed that from a size compression stand point, minification has some small value, but not nearly as much value as just turning on HTTP Compression. From an obfuscation standpoint we outlined the fundamental weakness of obfuscation in general. Given these factors my recommendation is for careful consideration of whether or not minification is actually needed in the particular case you are making.

I would ask myself these questions:

1). How important is the ability to read and debug my code?

2). How much time and resources do I wish to invest in tracking down bugs in minified code?

3). How relevant is the commenting in my code to the ability to debug and read my code?

4). With Obfuscation, whom am I trying to protect my code from and how competent are they?

Ideally, I think the best application is to do no Obfuscation, HTTP Compression, and a reduced form of Minification in which only comments are removed from the code. This provides you with high compression and still maintains the ability to read and debug your code. In my opinion obfuscation really is not worth the effort to employ it for the benefit you receive from it.

CONFIGURATION NOTES

HTTP Compression can be turned on in either Tomcat of Apache Web Server with just a little bit of effort. Tomcat requires a mere 3 lines of code to get HTTP Compression working. Apache HTTP Server requires a few more lines, but it is still relatively easy. For more information about configuring tomcat see http://viralpatel.net/blogs/2008/11/enable-gzip-compression-in-tomcat.html. For more information about configuring Apache Web Server see http://httpd.apache.org/docs/2.0/mod/mod_deflate.html.

TOOLS NOTES

Finally, I will plug the YUI Compressor. There are a lot of Minification programs out there, but YUI Compressor seems to allow for the most configurability. Also, I liked YUI Compressors ability to minify both CSS and JS files within one simple program. You can find out more about YUI Compressor at http://developer.yahoo.com/yui/compressor/

permalink: http://www.arei.net/archives/97
There's Only So Much I Can Do

There’s only so much I can do to stop you from making poor decisions.  I tried to dissuade you, I tried to use the weight of my 28 years of programming expertise, I tried pointing out the compelling reasons not to do it.  Yet, still you go on.  I’m only going to yell into the wind so much before I give up and let you proceed into the pit of folly into which you have decided to jump.  I hope it all works out for you down in there, but I think you are wasting you time.  That’s all I’m going to say on the subject…

At least until I get to say “I told you so.”

permalink: http://www.arei.net/archives/94
Free is the new Black

So, being a Java guy I’m fairly religious to Sun as a company.  I invested in their stock a while back (for good or bad) and thus I always kind of keep an eye on the company.  Recently I came across a series of video blogs by Jonathan Schwartz, the CEO of Sun.  The one I want to share with you is his second video blog.  The blog talks about the Sun view of the technological market place, and how giving brands away for free (like Java and MySQL and others) drives adoption, which in turn can drive revenue. Now, I’m no corporate financial officer, just a code monkey, but I found the blog well thought out and I thought I would share it with you.

My own company is currently working on a new project with the goal of productizing it.  We’re still in the early development of this product and haven’t had the discussion about how to sell it to consumers yet.  However, as I go about building the User Interface for this site I cannot help but keep thinking to myself (even prior to reading Jonathan’s blog) that the value of the site can only be realized by driving users to the site (or as Schwartz put it: building adoption).  In order to achieve this, the site would need to be largely free to the masses and make its revenue either through some ancillary stream or by charging only for some specific “higher role” usage.  As I designed the front end and go about coding it, I keep telling myself that the site is meant to be used by millions of people freely and must scale accordingly.

Like I said, I’m not the business guy, just a code monkey, but I really can see this notion of giving something away and finding revenue beyond just usage.  I like to think of it as the Field of Dreams model… In the movie Ray, the main character, is told to “Build it and they will come” which is much like the philosophy Schwartz is espousing.  The existence of the Field of Dreams brings the people, and there is surely revenue to be made after the fact: selling popcorn or something; the method is not as important as getting the people, driving adoption.  If you have the people, selling them something they want beyond what you are offering should be easy.

Anyway, if you’d like to read/watch the video blog I am talking about, you can find it here: http://blogs.sun.com/jonathan/date/20090306

If you’d like to read/watch the entire video blog series (there are four of them), you can  find it here: http://blogs.sun.com/jonathan/date/20090302

If you’d like to follow Jonathan Schwartz’s blog, you can find it here: http://blogs.sun.com/jonathan/

permalink: http://www.arei.net/archives/84
Upgraded WP

I upgraded WordPress to a new version.  Let me know if you see any problems.

permalink: http://www.arei.net/archives/82
Where the Weekend Went

This past weekend had the Dad in town which, despite icky weather, was a nice visit.  We wandered the inner harbor, checked out the cherry blossoms, hit some of the favorite restaurants, and did some grilling.  The inner harbor was fine, but in my opinion made better with a stop at Pratt Street Alehouse to get some of their very tasty hand pumped beers.  The cherry blossoms were mostly gone, but it still made for a nice walk around the tidal basement hunting for the remaining blossoms and allowed my dad lots of picture opportunities.  We followed that with a stop to Arlington’s Rock Bottom Brewery for some cask conditioned beer, which is the only way to serve beer.  Other various places we visited were Victoria Gastro Pub, Five Guys Burgers and Fries, and Indian Food in Columbia.  As I mentioned, we also fired up the grill for the first time this spring cooking some tasty fresh salmon for Jennifer, and some steaks for the old man and I.  I was especially proud of how the steaks came out.  Very tasty.

The dad’s visit gave me a chance to rest the broken finger by not being able to play any frisbee all week.  I think that worked out well, but with Tuesday night being to rainy for frisbee, I have yet to test my frisbee abilities with a broken finger.  The doctor took another x-ray yesterday and said that things were progressing very well and that so long as I wear a splint there is no reason not to play frisbee.  He also said that otherwise, no more splint, just buddy taping the finger to the other finger for a while should be fine.  Good news all around.

permalink: http://www.arei.net/archives/76
Broken? Yes

To follow up on my last posting, I had a specialist look at my finger yesterday.  There is definately a break, but it’s very minor.  A small fragment of bone has broken off.  However, because it’s so small and because it’s separated, it will never reconnect with the primary piece of bone and it’s not really a big deal if it does.  At best the small piece could be removed, but really it should be just fine leaving it in there.  I have to wear a splint for a week, then buddy tape for another week, with new xrays at the end of each segment, but other than that things are fine.

The good news for me is that he sees no reason why I can’t go back to playing ultimate immediately so long as I wear the splint while playing.  Since I wasn’t planning to play this week anyways (I have family in town and no league games on saturday), I’ll take it easy just in case.  On tuesday I’ll go back to playing and see how it feels.

permalink: http://www.arei.net/archives/74
Broken

It looks as if I have broken the ring (or 4th) finger of my left hand.  I did something while playing Ultimate this weekend and toward the end of my 1st game (of 3 games that I had saturday) I noticed my finger was sore.  Well, I iced it for a bit and taped it up to its sibling finger, and kept playing.  Then sunday I iced it some more.  It was still sore monday, so I went to the urgent care facility and got an x-ray.  Lo and behold there is a tiny piece of bone broken away and lose and causing all kinds of problems.

So later today I’m off to an orthopeadic specialist to get a more detailed analysis.

In the meantime I’m typing rather oddly with one of my fingers in a splint.  It isn’t slowing me down that much, just a little in terms of overall typing speed, which is good.  Occasionally it gets in the way though.

I’m pretty worried about not being able to play frisbee for the next n weeks while things heal.

permalink: http://www.arei.net/archives/69
Fearing Leadership

There’s nothing I hate more than a group or organization that is disorganized or poorly led.  When faced with these types of groups I am compelled by some internal desire to step in and whip the group into shape.  I have done this time and time again, filling gaps in leadership when the need arises.  In variably I jump into these roles, do them for a few years, find someone willing to take over, and then leave and watch the group mire itself back into oblivion.

Yet, despite my compulsion to lead, I really have little desire to do it in a work role.  Maybe I fear the responsibility of work leadership?  On some level I know that in order to take on such a role I’d be sacrificing the parts of the job I love most and I really have little desire to do that.  Also there are far too many people I know that have moved into leadership roles only to find themselves forever stuck in the land of middle management suck.

However, lately in my work I have begun to feel that I it is inevitable I move upward into leadership roles, despite my lack of interest in it.  Oh sure, I can probably fight to stay where I am and be a code jockey forever, and I think I will be perfectly happy to do that.  Yet, sooner or later, I am going to be faced with an absence of leadership (or more than likely quality leadership) that will overcome my blase work leadership feelings.

The question is then, what do I do then?  Do I give in and take leadership as I have always done outside of my work life?  Or do I fight it tooth and nail to maintain my little place of professional nirvana?

permalink: http://www.arei.net/archives/66