One of the most important things when learning a new programming language is to have a couple of small programs in mind that you can implement. You don’t learn a language by reading about it, though that helps significantly. Also, you don’t learn a language by reading code, though once you have a handle on the syntax and semantics, this is invaluable to improving. You learn a language by writing code in it, running the code and debugging the code.

You don’t jump into a new language and write a dependency injection framework or build VisagePamphlet, the new killer social media application. You need to write a few small programs that you can personally validate are more or less correct with a test suite or simple manual execution.

The choice of what to write is kind of tricky, so here is what you should do :

Implement a Brainfuck Interpreter

WHAT.

If you have never attempted this, it might seem kind of insane, but it’s actually far less difficult than you may imagine. The language is so tiny that a reasonable implementation in any current mainstream language is going to be under 100 lines of code. But more importantly, this little program you are going to write requires:

• variables
• loops
• if/else
• switches
• possibly recursion
• thinking about types (if applicable)
• string manipulation
• IO

But this has an additional side benefit as well, you have to write Brainfuck programs to exercise and test out your new interpreter. This in and of itself is a challenge due to the limited features available in the language. In addition, once your test programs are working, there are a host of programs available on the Internet which you can download and throw at your interpreter.

Finally, depending on how deep you wish to go, this program is now an excellent launchpad for future experimentations into the language you wish to learn. Write something so you can dump the memory of the Brainfuck program. Add another pass to verify the program is valid before running it.

Roughly 500 routes, 8700 stops, 126000 trips and 3.4 million timepoints at those stops. Not a whopping amount of data, but enough to start having some fun. My initial plan was just to be able to plot the stops for a given route onto google maps. That’s done in it’s ugly glory at my stopfinder. If you want to search for a 1 or 2 digit route, put in the leading 0′s. Sorry, haven’t done that yet.

My next steps are going to be to publish a number of primitive operations on the data with results in JSON format. Things like ‘closest stop to lat,lon’, ‘how to get from stop x to stop y’, and other similar sorts of things. The idea being that if I can build up a suitable library of common operations on the dataset, any future ideas that do come to mind should be relatively easy to implement.

That and if anyone does want to do some data mining, well, this is an option. I’ll post any updates, formats and that sort of thing on this site as I work through it. In general, the services will be pretty much simply URL based and will return raw JSON. Nothing special, but fairly easy to parse and work with. I have a relatively irrational dislike of XML which I will probably get over at some point, but it will take someone making a very good argument.

So it is, but actually stating that, or anything along those lines? Way to kill the team, boss! (See Peopleware)

That said, quality assurance, quality control, QA, call it what you want, but it’s one of the more misunderstood aspects to software development. Oh sure, everyone knows that they need to do more QA or better QA, but lip service is about all that is ever paid towards it. I am notably not including in my ‘everyone’ those who feel that QA can be completely automated. You guys are wrong and I’m going to leave it at that. You also may think you don’t need to do it, see this article for some classic arguments against that fallacy.

I’m not going to go into depth about QA, how to do it, best practices or anything along those lines as I’m fairly unqualified. That said, I’m not really qualified to talk about anything, but that doesn’t really stop me.

QA is a processes, not a task

This particular fail case is something I’ve seen in multiple organizations now. The most obvious symptom of this is when management has decreed that there is a block of a few hours set aside to ‘do QA’ on an application with a few hundred known use cases. Another obvious indicator is when other employees are volunteered to do a few hours of QA on top of their normal job. Think you’re going to get good results from that?

The root cause of this failure is simply not understanding how QA works, so let’s walk through it a bit. In a very broad sense, the general list of tasks for QA is something like this:

1. Go through the basic cases

2. Go through the corner cases

3. Go through obscure, known failure cases

4. Exploratory testing

5. Automating 1, 2 and 3.

So, how does this fit into a day of work? Let’s find out:

First off, we’re going to go through the basic use cases for the application. Then, there is a pile of corner cases that are pretty valid that need to be checked out. Then it’s time to check all the really obscure, but horribly embarrassing failures that have been seen before. From there we can finally…What? You changed the code? Okay, first off, we’re going to go through the basic use cases for the application…

Interruption here! “Silly tester,” says the savvy developer, “You only need to re-test the parts of the system that were changed.” Nice theory, but wrong in many, many ways. Simply put, if this was the case, testing outside of developers would never be needed. That generally goes well.

Back to the task at hand, do the basic cases, do the corner caWHAT? Changed again? Basic cases…

The real job of QA starts at step 4, which we haven’t even seen yet. Exploratory testing is finding the embarrassing defects before they get out into the wild. A good tester at this phase is going to break your application in ways you haven’t even dreamed of. In ways that only 0.1% of your users would ever try to do. Of course, if 0.1% of your users do it, and you get 10k uniques per day? That’s 10 people per day that are going to hit this embarrassing bug that how could you possible let into the wild and I’m taking my business elsewhere right now as I obviously cannot trust you with my data. And if one of those has a blog? Heh. Have fun with that.

So the epic fail with having 16 hours scheduled in to test your quarter million lines of code application? If you’ve got bug fixing going on at the same time, any of your competent testers will never get past step 1. Any testers that listen to the savvy developer, or worse, are the savvy developer will miss basic cases and you deploy with fundamental breaks.

The purpose of QA is not to have someone say, “Wonderful developer, your application is perfect!” If I hear that from a tester, I assume the person isn’t doing their job very well. QA should hurt your feelings. Assumptions you made should be laid bare and justified or thrown out if incorrect. This is often the last line of defense before your customers see your application, take it seriously.

So there was lots of refreshing of information, which was notably taking up my brain time which meant to less time learning wholly new things. This is okay, but one of the best things about software development is the constant learning. So when a company mentioned that they were using Hadoop and HBase, this was instantly exciting to me as it gave me a perfectly good reason to go and research some new technologies.

If you haven’t heard of these two projects, and unless you are specifically working with them, you probably haven’t, Hadoop and HBase are free software implementations of two systems designed by Google, MapReduce and BigTable respectively. MapReduce is a framework developed by Google to facilitate processing and working with large datasets across a distributed network. BigTable is essentially a way to store structured data across a distributed network, though it is important to note that this structured in terms of nested hashes, not in a traditional relational manner.

Google had a few motivations for building something like this. They regularly worked with gigantic datasets, their search index itself, search logs and Google Maps tilesets as a few examples. Analyzing these datasets took massive CPU resources and a distributed approach was more or less deemed the only practical way to actually compute solutions. MapReduce takes the classic divide and conquer approach to solving a problem. The problem space is split up and doled out to dozens, hundreds or thousands of computers. This is the map phase. Results from each piece of the calculation is then returned to the master computer which then reduces the results down into some sort of final output.

This approach has been used for solving all sorts of distributed problems, but MapReduce was unique in that it was a framework, usable for a variety of tasks. The hard parts of managing a distributed network for a single calculation are often just that, the management tasks. Deciding which servers get which chunks of data depending on where they are in the network. Deciding if a server has crashed, or if it’s just slow. How and what, if any, data do you duplicate across multiple nodes to guarantee that you get an answer. These are all concerns that must be addressed when creating a distributed application.

What Google did with MapReduce was abstract away the management tasks so their developers could focus on actually writing the algorithms to solve the problems. Hadoop is a Java version of MapReduce. But I digress.

What was really nice was that in the midst of this review and stress, as looking for meaningful employment is not without some concerns, was an opportunity to read a couple of new papers and learn some new things. So I promptly went and wrote a distributed program to calculate how many lines each character in the collected works of Shakespeare spoke. This was not something that required much computer horsepower, but it was pretty cool to run the thing on two computers and get chunks distributed to each of them.

So that went over well. I am going to post this now as I have been sitting on this for a while and not finishing it up. I need to lower my standards somewhat as, to paraphrase Joel and Jeff, I only have one reader anyway.

As my Ruby experiences are measured in months rather than years at this point in time, more information is always better than less. And I have always been a sucker for a nice heavy book to keep beside the keyboard. 

The Ruby Way.

Basically this book is a list of common and not so common tasks and code snippets to show you how to handle them in Ruby. Another way to look at it is this book is basically the logical reverse of the language API. Rather than look up a method and find out what it does, you look up a task and it tells you the method. This can be very, very handy.

Ruby is a concise language with many excellent one liners. In the core classes, however, there are often a large number of methods. String is one good example, there is something like 100 methods in the class and I have, more than once, accidentally re-implemented one of them while trying to solve a problem. This is mainly due to simple lack of experience with the language. Now that I have a copy of this book, if I get the feeling that maybe Ruby has something built in to solve problem A, I can look up problem A and find out if there is a one liner rather than writing some custom method.

Even in my relatively small projects, this book has already saved me a handful of hours. Currently I am reading through the book, random chapters at a time. The main goal I have now is to build an internal catalogue of problems that this book solves so I can remember to reference it if those problems crop up. Shortly after I use a method a couple of times, then it is part of the repotoire and I won’t need to look it up again.

So, in a nutshell, if you need a good book to learn to make your Ruby more concise and make better use of the language, go with this one.

It was directly due to the words of a couple of well respected bloggers that I started this one. There are multiple reasons. By writing, you exercise the communication part of the brain, which is always good. By putting up a public blog, it becomes easier to control your own Internet presence. As long as your writing is even passable, it is good marketing for yourself. These were just a few of the points that were made. I bought into it and am now going on a couple of months of roughly weekly posts. I believe I can keep this up rather easily.

So, one mission accomplished.

On more than one occasion, one of these people who are well respected in the software development field will mention something about a compiler. Specifically, that every serious programmer should write one at some point in time. This makes sense as a compiler tends to be one of those bits of software that does a good job of covering pretty much everything that a programmer needs to do somewhat regularly. Lots of string manipulation, lots of recursion, lots of knowledge about how a computer actually works, memory management, the list goes on.

I have not yet written a compiler. It is something that I have wanted to do more than once, but never actually sat down and started working on it. I’ve seen enough mentions in passing recently that I feel it is time for me to complete this particular rite of passage and actually write a compiler. I am capable of it, but I needed the push down the right path to do it.

So it is not going to be a huge project and I’m not going to sink huge amounts of time into it, but I do have goals for it that should be completed by the end of 2009. I need to define my own language, it should have ifs and whiles, it should evaluate mathmatical expressions correctly. I think I will compile to JVM compatible byte code, though I have not decided on that yet either. That would theoretically allow me to use Java libraries as well, which is appealing. It’s not going to be object oriented, mainly as this is a learning experience and I don’t want to bite off more than I can chew so to speak.

I’m leaning towards Ruby for the project, but am stronger with Java which will make some things significantly easier. I will likely make the final decision once I’ve read some of the Dragon book and know more about what I am getting into. My intention is to write it all at this point. I don’t think I will use a parser generator or anything like that, simply as the point of this is to learn how to write a parser/lexer/etc.

So that’s the plan for 2009, dedicate a couple of hours a week to this project and hopefully be able to run some of my own code in my own language by the end of the year. Who knows if that’s reasonable, but it should be… illuminating.

During one of the interviews a comment was made in passing about programmers not wanting to talk to clients. This is a popular topic for developers to joke about, but I had always treated that as a joke. When it came down to the business of doing business, well, talking to your clients is just to be expected. Ultimately, they are the reason you have a job, so there is something very serious to be said for treating your clients with respect.

This particular ramble is going to go down a couple of paths, I’m going for breadth more than depth here.

Generally a software developer is writing code for one of two different reasons. You are writing software for a new product that may or may not have clients yet, you may be trying to carve a niche into an existing market or creating a new one. The second and by far the most common reason is that you are contracted to write software that accomplishes some task for another person/organization/demographic/etc.

So what does that make our job as developers? Our clients or employers have problems that they have decided for whatever reason are best solved by computers. We are to analyze that problem and develop a solution. Hopefully that solution makes our clients more money or saves them money by making existing processes more efficient. If the post-mortem shows that the cost/benefit ratio is less than 1, then we have failed. It really is that simple.

Solving a client’s problem is our job. One of the aspects that makes software development so much fun is that many times our client does not know exactly what they want. This may be a source of entertainment for us, but is a very important thing to keep in mind. The world of programming is one of absolutes, the computer does exactly what we tell it to in a nice and ordered fashion. (Defects being the times when we tell the computer to do the wrong thing) The real world is much less black and white. What this means is that we have to listen to our clients and understand what they are trying to accomplish. If the request is vague, we have to figure this out by asking the proper questions to build a better picture of their business.

This is challenging.

Worse, they may know exactly what they want, but are wrong. I have personally been asked to implement things with very little more than a vague bit of hand waving and comments to the effect of “We think having something that does x would be nice.” In this particular case, x was fairly obviously, to a developer of the system who had detailed usage statistics, not what our users needed. They actually needed y. Through some discussion with our clients, our team was able to make the case that x would only help less than 1% of our users, and a particular subset of users who did not and would not spend any money. That probably helped our case somewhat. We ended up implementing a modified y and our customer service requests for a certain class of problem dropped significantly.

What was interesting to me from a personal standpoint was that the same people who were arguing to just implement x were the same ones who did not want to talk to the client. Now, I should clarify, I don’t mean, the grudging, “Oh no, small talk and meetings with people I don’t know”, the typical introvert response where you know that you will come out of the meeting drained, though hopefully with the problem solved. In this case I mean attempting to make the case that it was not our job to talk to the client to figure out what they needed.

This did not sit well with me. There is a reason that the principles of the Agile Manifesto have several points which specifically reference and talk about dealing with the customer. These points all touch on dealing with the customer with respect and understanding as they are the experts in their own business. We are experts in software development.

As software developers, our job is to learn from our clients to build systems that improve their business. To do that, we will always need to interact with our clients to determine their needs and meet them.

The difficulty is that there are only so many programming books that are truly classics. While I appreciate and enjoy learning from well written Hibernate or book on the Google Web Toolkit, ultimately, any software book that starts to discuss a specific framework will quickly become obsolete. As one of my personal definitions of classic is a book that does not grow old with time, this means that a huge percentage of mass market “Learn how to gidazzle with widget fooxily” types of books will never reach the classic status.

Unfortunately, it seems that the dominant type of programming book that is on the shelf in your local bookstore are exactly the ones that will be out of date in a couple of years. This makes me somewhat sad as if the market is for books that quickly go obsolete, the number of people that will write books that go on to be classics will be smaller simply on a basis of supply and demand. I’m not sure where this leaves us. I like buying technology of the month books as well, so I help this. All in all, it’s likely not a problem I suppose. I guess I just wish I could actually find Peopleware or the Mythical Man Month in print on a shelf somewhere.

So now I reach the real crux of the matter. If you are reading this, then you probably read other blogs by various guys in the software industry and they already have pretty much written about the books that you absolutely should read. More, most of you have probably read some of the books that Joel or Jeff have suggested in the past. So I’d simply be duplicating content listing the exact same classic books and suggesting that a serious programmer should read them.

So this has all been a debate as to whether or not to do that. You see, I actually enjoy reading reading lists, especially if there is a little blurb about why the book is good. Every once in a while a wholly new book turns up and I will add it to my ‘to read’ list and I am the wiser for it. Still, I think I can stand up to it and avoid actually creating a reading list of some form.

That about wraps it up for today, this was more of a random ramble than anything else. Back to actual geek topics soon enough.

Back in the good old days, you could build entire desktop applications with one programming language, C++ or whatever that may have been. Whether the code you were working on was for the presentation layer or deep in the business logic, you were dealing with the same syntax, the same debugger and the same paradigms. The advent of large web applications completely destroyed that.

Depending on the language and framework you are using, any reasonable web application typically requires detailed knowledge of no less than four languages. HTML for your content, CSS for presentation, Javascript for any dynamic effects and some other language that you are using to do your server side processing. Technically speaking, two of these are not programming languages (HTML and CSS) as they offer no control structures, but they are still a syntax that you must keep in mind when working with them.

The problem with this is that no matter how good you are at programming, there is some mental overhead associated with the requirement to switch syntaxes in your head as you move from working on one aspect of the project to another. Generally speaking, most languages have their own distinct set of advantages and selecting the correct language for a given project is an important. That said, the more languages a programmer needs to use to work with a given project, the less likely they are to be an expert in all of them. This is an important enough point to note that it is corporate policy at Google to work with one of something like four languages. (Steve Yegge talks a bit about this here)

So I’m going to postulate that keeping the number of distinct languages that are needed to keep a given production application working and maintained to a minimum is an important consideration. If your project is using the aforementioned three web presentation languages, then a mixture of PHP, Java and Perl on the server side, you have dropped a large brick wall in front of any developers new to the system and added additional mental overhead to working on the project even for your top developers.

The grass is greener…

Which brings me to the Google Web Toolkit (GWT). I have used this in a few projects now and am fairly comfortable with the framework. I can safely say that I know it very well from an applications development perspective. When I first started using it, it was like magic. I got to write and debug Java code, which I was very familiar with, but when I was done, I could compile the project and run it in a browser with the same functionality.

The more I used it, the more I began to feel that I was actually more productive with it than by writing straight HTML/CSS/Javascript and tying that to a servlet or JSP based back end. I will emphasize that this was a ‘feel’ more than any sort of hard numbers, but more on that when I get to the criticism part of this post.

I’ve rambled before on some of the problems that I have with Java and was looking around to try out a new web application framework. I settled on Ruby on Rails to build Gifti.ca and I do really enjoy working with this framework. It has often been said that there is nothing like moving to a new language to show you deficiencies in the one you usually use, but the opposite is true as well. You see, with Rails, I was back to how I originally worked with web applications, four or five languages. This was when I had my ‘ahah!’ moment about the GWT and what it was that I really enjoyed about it.

Java, simpler?

In this case, yes, with the GWT, you need to work with two languages, Java and CSS. You can build an entire web application with the GWT and the resulting HTML that you actually put into the page is absolutely minimal. For all intents and purposes, you link to the Javacript file that the GWT generated for you and the entire application merrily builds itself in the web browser. For the most part, the major browsers work fairly interchangeably as well. The generated code is quite fast, better than humans can generate according to the Google propaganda and I have no reason to doubt them.

You do need to keep a handle on the CSS, but even then, a number of the classes have methods which allow you to programatically set various alignment parameters and the like so those can be taken out of the CSS as well. The AJAX functionality they have baked into the framework plugs into a servlet container so beautifully I’d be hard pressed to come up with a simple reason not to use Java on the server side with a GWT front end.

Eliminating two of the languages that you need to keep in mind when working on a project is a huge benefit mentally speaking. Fewer major context switches when going from layout to client side logic to styling meant that I found it easier to stay in the zone. This is why I need to make intelligent arguments with myself when starting up a new project as to why I’m not immediately firing up Eclipse and coding up a GWT project.

But…

I have some concerns about the framework. I feel more productive when I am using it, but I know that I spend longer building the user interface than I would have with a plain old fashioned web application. On the other hand, the UIs I tend to build with the GWT are more sophisticated. It is a balancing act. While I never bothered to keep any statistics, I would be surprised if the GWT was any more productive than anything other web application framework.

There is one very serious problem with the GWT and I have no real idea how to address it at this point, which is why my latest project isn’t using it. The GWT makes it easy, very easy to build your entire application with it. If you deploy a website built entirely with the GWT, search engines will not be able to index you. At least, not yet. All of your content will be buried in the compiled Javascript which in turn means that unless the search robot is running a full Javascript engine, you are out of luck.

Worse, if you want to support a version of the site that actually works without Javascript, you are left with writing another front end for the application entirely from scratch. My experience has been that this is painful.

Which also really means that before you jump into using the GWT for a given project, it pays to know your audience. For a general purpose web application that is to be used by a large and diverse group, the GWT is probably not the best choice. This is double so if the web application is geared more towards searchable content such as a blog. If you are building an intranet application or know for a fact that the clients that you are interested in will be running a full featured browser, then the GWT is an excellent technology to choose to go with. It’s probably also going to work rather well in an environment where web searchable content isn’t terribly important, a mashup with Google Maps or something along those lines.

Anyhow, long and the short of it is that I will continue to use the GWT for projects for the foreseeable future, it’s just too powerful for me to ignore. Being able to effectively ignore HTML and Javascript for a certain class of application is highly appealing to me.

HTML had some limitations though, it was kind of verbose and you had to mix up your content, paragraphs and such, with code that was used to show how the thing should be laid out and look. At more or less the same time, there were two new pushes to make HTML better. Javascript (1995) was developed to allow a web site designer to make things change on a page. No longer would a web page be a simple static document like a magazine or newspaper article. Cascading Style Sheets (CSS – 1995-1996ish) was created to help a web designer keep the content, the article, online store, etc. distinct and in a different place from the stuff that was responsible for how that content actually looked on the page.

And this was Awesome, except, it wasn’t. At the time that these developments were happening, the First Browser War was underway. Netscape put some Javascript and CSS stuff into their browser. Microsoft put different things into theirs. Bit players in the browser wars did other things. The end result was that if you wanted to make a sufficiently sophisticated web page, you had to create two or three versions of the thing so that it actually worked in multiple browsers. Now, for a long while, this did not actually matter as Internet Explorer 6 held so much market share that you could target IE6 only and get away with it.

That is no longer the case. Arguably, it was never the case, but there were lots of so called ‘web developers’ who happily wrote sites that worked in IE6 and IE6 alone. There are several major browsers now. More importantly, there are several major devices now. Blackberries, cell phones and game consoles all have limited support for browsing the web in various forms. It’s probably a good idea to try to write your web code in such a way it works in as many platforms as possible. More specifically, at least the major platforms that your clients are using.

Approaches

Inevitably, however, some of the fancy dynamic content, crazy layouts or Javascript dancing puppies will not work on some platform that at least some of your customers are using. There are a couple of different strategies for dealing with this scenario: Progressive Enhancement and Graceful Degradation. Effectively, the end result of either of these strategies is the same thing. A website that functions properly under just about every browser platform. There is overlap between the two strategies, in that you can safely say a website built via a progressive enhancement strategy will degrade gracefully. That said, there is definitely a difference in the development process when building a web application in either of these manners.

Having worked on projects that focused on graceful degradation, I can offer my opinion that it is a route fraught with problems. My gut feeling is that if you are going this route, you’re almost better off writing off customers who don’t have your minimum supported browser.

Progressive Enhancement and HIJAX

I’m mostly going to talk about progressive enhancement now as my current project has gone that route. I mainly went that way as I was concerned with the smell of the code that I had written or worked on in the past that went the other way around. Taking a full featured, beautiful Web 2.0 application and trying to hack in static HTML support was difficult and led to some fragile functionality in places. Progressive enhancement seemed to solve that problem in that you started with a functional, solid HTML version and then began to add the flashy enhancements.

Google for HIJAX and read some of the blog posts you find, all are pretty short and give you an overview of the approach I am making with this particular application. There isn’t really a definitive definition of HIJAX, suffice it to say, the point of the thing is what I outlined above. Normal, old fashioned site first, then start writing the flash. The basic philosophy is “Plan for AJAX from the start. Implement AJAX at the end.”

Of course, there is no silver bullet.

While I truly do like the development process that follows with HIJAX, I cannot help but feel I have traded one type of code smell with another. Well, in this case, it isn’t really code smell, it’s more of a project management smell. The single biggest problem I have with the simple HIJAX philosophy is how simply it discounts the nature of web applications.

Implement AJAX at the end.

This is where it all falls apart. Anyone who has built web application that goes beyond the trivial knows that there is no end. Paul Graham talks about the advantages to web development in The Other Road Ahead. He has an entire section dedicated to releases and notes that in a web application, you really don’t have releases. You have features that you deploy one or two at a time.

So if there is no end, implementing AJAX at the end isn’t really possible. So the HIJAX philosophy migrates more into a “Plan for AJAX from the start. Implement AJAX in chunks as you go along.” This is notably still different from “Implement AJAX from the start”, but must be pointed out. The majority of articles and essays on this topic that I have read thus far do not take a high level perspective. They deal with specific implementation issues and provide an outline that works at the feature level. The naive philosophy will not work when applied to an entire project.

This is not to say that it is a lost cause and we should get back to writing rich applications that only work in specific versions of Firefox. Progressive enhancement is the way to go for a general purpose application on the Internet where you do not want to drive away any customers. HIJAX, on the other hand, is a nice buzzword for a simplified philosophy that I think will die out quickly as it becomes apparant that it is just a bit too simple.