This Blog Has Moved!

I finally got my own domain. Please update your bookmarks to point to mikeomatic.net.

Why Desktop Computers Aren't Going Anywhere

Over the past few years, the web interface for creating applications has come a long way. With AJAX and similar technologies, we now have the ability to have the browser interact directly with the server without page reloads in a more or less standard way, a critical step in perceived performance for end users. Applications such as Google Maps and Spreadsheets, Writely, AjaxWrite, and Meebo, among others, prove that the web applications can perform at a level comparable to disk-based counterparts. It would seem like the late 90's vision of the Network Computer is finally upon us. And yet it isn't.

This infuriates many people in the software development world. We deal in a world of absolutes, where there is one right answer and it is inevitable, at least to people who trouble themselves to think of such things. Procedural languages give way to object-oriented ones. Proprietary formats give way to open standards. And desktop applications move to the web. There is just something that feels right about this, like a ship emerging from a fog. When it is in the fog it does its best to muddle its way through, but when it sees an opening filled with sunlight it suddenly knows the right way to go.

There are several things which prevent this from happening on a wide scale, however, some of which are either miscalculated or underestimated by people on the front lines developing the software.

Below are some of the hurdles web applications have to overcome to graduate to the next level of mainstream acceptance.

Latency is Bad

This one we all know. The problem with all applications delivered over a network is that the network itself is unpredictable. While it is neat to play with Google Spreadsheets to see what the guys in Mountain View are up to, trying to do real work in it is another thing entirely. If you lose your connection to the net because your ISP trips over your wire at the central office, you lose what you are working on. More likely, you will deal with nagging little performance issues as your AJAX calls aren't returned in real-time, or the Flash game on another tab in your browser sucks up all your CPU cycles and makes your page slow to refresh, or any number of other things that impact your browser and make your AJAX experience less than optimal. The web applications just don't have the same feel as desktop applications, much like Java Swing interfaces don't feel like native ones. It's a fact of life right now.

Lack of a Consistant UI and API for Dynamic Web Applications

This is more subtle and requires a little more explanation. First, there is a difference between a library and an API. A library obscures some functionality to let someone program at a higher level of abstraction. An API defines a platform. There are of course similarities between the two, but the bottom line is that there is not a consistant dynamic web platform that is accepted as a standard by normal users. What we are left with instead are 299 different AJAX libraries with largely overlapping functionality and slightly different assumptions and design goals. Building your application around one of these may help you from dealing with obnoxious things like browser detection and XMLHttpRequest object instantiation and may even give you some cool fading effects to play with, but it will most likely not make your application look like other web applications to end users or give you higher-level building blocks with which to build your applications and interfaces.

A quick rant: Developers (particularly open-source developers) believe that diversity in libraries and interfaces is a strength rather than a weakness. I agree only in so far as the diversity eventually pushes standards into place as the stronger ideas are adopted and the weaker ones dropped. To typical end-users (the type you need to make any type of revolution mainstream) having 10 ways to do the same thing is confusing. Look at the window managers available for Linux as an example. Should mainsteam users even know what a window manager is? Are the subtle differences between Gnome and KDE worth the fact that every newby book for Linux needs to dedicate a chapter to describing X Windows and how it decouples the presentation of windows from the lower level parts, and how you can choose between a million different interfaces despite the fact that you want the one that looks like Microsoft Windows? This stuff is confusing to people who do it for a living, nevermind people just looking to surf the web in peace. This same problem plagues web development, where the large number of developers and the Not-Invented-Here syndrome that they usually have leads to everyone wanting to make their own libraries rather than adopt someone elses'. Our inability to establish de facto standards on our own is what lets certain monolithic corporations do it for us. End rant.

I believe that the YouOS project is one of the first (or at least most popular at this moment) to attempt to tackle this problem head on, and I wish them luck. I haven't seen their code to know how it works, but their site looks impressive. It remains to be seen whether or not higher-level development will take place on this platform beyond the widgets I have seen so far. If they get OpenOffice running, I am there. :)

The Status Quo is Comfortable

We are in a mature industry, and people often forget that the "Next Big Thing" doesn't occur nearly as quickly as the "Last Big Thing" in such an environment. At this point the players are pretty well established in the computer industry, and the costs associated with the current structure are both well-known and accounted for. While web applications are being pursued for most in-house development nowadays (replacing buggy VB applications with buggy ASP.net replacements) there is no push to also replace standbys like Word and Outlook with web-based alternatives. The current versions work and we already paid for them. Plus we have a help desk that knows them. Plus we paid for training in them. Plus we have Active Directory and group policy already set for them. Plus the CIO wants more money in his budget to spend, not less. And so on.

Powerful Computers are Cheap

While this isn't necessarily something that will hold back web application development, it certainly removes some of the exigency for going in that direction. Web application supporters often say that less sophisticated users "like my grandmother" only use a PC for limited tasks, such as email or balancing a checkbook. Naturally, these users would flock to a cheap thin-clientesque computer running web-based applications since they are cheaper and would allow them to do everything they need.

This argument is problematic because of a simple misunderstanding about human nature: People never want to feel that they are getting ripped off. There is a reason Starbucks charges $3.25 for a 16oz drink and only $4.25 for the 32oz. People will pay a little bit more just to make sure that they get better value for the money. This is exactly the situation the thin-client finds itself in. If they are even available, the cost is only a shade below that of a full-fledged PC that comes without any of the limitations. And this is a big if. Most thin-client computers are only sold to corporate customers that have the ability to centralize a huge number of functionally identical machines, and even this is relatively rare.

For the most part, people will keep buying the best PC they can afford when upgrade time comes around, and that will be more than capable of running mainstream desktop applications.

Conclusion

Web applications have come a long way and are often genuine works of art. A lot of ingenuity was required in taking a platform meant to merely view static documents online and turning it into a mostly fully-functioning operating system. That being said, we as developers must be careful not to get too far ahead of ourselves in extrapolating this growth curve. If we want this work to result in a real desktop replacement platform, we will need the discipline to overcome the issues listed above.

Til next time.

IT in DC

I've lived here in the nation's capital for about 4 years now. The city has many great qualities and several not-so-great ones, so in the spirit of comraderie I'm putting together this little primer for my fellow nerds who may be considering a move to DC.

Employment

The availability of jobs is one of the primary reasons people come to this area. The federal government and military are huge employers, as are the numerous non-profits and NGOs who are located here due to the proximity to the political seat of power. Even more substantial to the IT set are the numerous contracting companies that cater to the afformentioned organizations.

If you have a college degree, pulse, and 3+ years of job experience you can find a job here in virtually no time. The hot languages are the typical J2EE and .Net as well as either Oracle or SQL Server, depending on the shop. Most of the places you will be working for (if you go the contractor route) are risk averse and very well funded; saving money isn't as important as using the same technology as everyone else. Open Source is starting to show up a little, and is undoubtably incorporated in many IT systems, but it is not a primary skill that most employers demand you have.

Entry level workers may have a hard time in DC due to how contracting works. Generally all candidates are presented (read: marketed) to the organization they will be working for/at, and these organizations love things that they can quantify. Like years of experience. As a result, contractors aren't likely to present inexperienced candidates since they only get so many shots to fill a particular slot. This is of course highly (over)simplified, but it is my experience.

As for the work that you will be doing at your new job, it will most likely be on an internal or intranet application. DC has a culture that is obsessed with the billable hour, and these types of projects eat them up. The teams implementing these projects often include employees of the organization (in management/leadership/design roles) and contractors (mostly coding). This type of arrangement doesn't usually produce the best software since the people doing the design and choosing the technologies aren't always the most technically competent, but the guy with the money writes the rulebook.

There is a lot of entrepreneurial activity in DC, but most of it is around making new contracting companies rather than new product companies. This is just due to practical reasons; with the billable hour culture there is a ton of work to go around, and those kind of companies are cheaper to start. It is also common for people to have side gigs, although most employers are sensitive to anything that competes with them in the billable hour field and will make you sign away your life to protect themselves. Use discretion, and do what you can to only sign agreements you can live with. I've personally turned down offers with $30k raises attached just due to disagreements on the paperwork; stupid, maybe, but you know that when push comes to shove they will use it. (My particular case involved them assigning personal liability on me for anything that happened on the job as an employee of theirs. Not worth it)

Housing

Housing in DC is expensive. If you can afford to buy it will save you a lot on a monthly basis, but the legend it that housing prices are facing a downturn. Due to the great Metro system, it is easy to live in the suburbs and get anywhere you need to. If you choose to live downtown you do not need a car; I've never had one and it has been fine.

Apartments, especially good ones, don't last long; this makes it difficult to line up housing ahead of time before moving here and not get fleeced. If you can find a temporary arrangement with friends or family that will let you stay with them while you looked you will have a better chance to find a good, cheap place. Popular venue for searches include the City Paper, Craigslist and Apartment Finder, and coworkers can also help you find openings.

People differ in what they look for in an area to live. Generally speaking, the Northwest quadrant of the city is the best crime-wise, with it getting better (and more suburban) the further north and west you go. There are great neighborhoods in all quadrants, however, so you really need to see them up close to determine what works for you. Dupont, Adams Morgan, and parts of Columbia Heights are good mixtures of city life and safety. Georgetown is nice, but yuppie and expensive. If you look outside the city, Alexandria and Arlington a popular alternatives, but also often yuppieish and pricy. Many good values can be found in Maryland, particularly my College Park, Greenbelt, and New Carrolton. These are all more suburban, however.

Culture/Nightlife

DC has a very unique character. Unlike most cities, it has no industrial district to speak of, and is not a financial or commercial hub. It essentially emerged fully formed as the seat of government, and the evolution since then has sprung from that. There aren't really highrises here due to an ordinance, so there isn't the canyon effect you see in Manhatten; just a bunch of office buildings, brownstones, and appartments arranged nicely. The original city plan and grid encompasses the Downtown area, consisting of the Capitol, White House, federal offices, Mall, monuments, and museums. As you move away from downtown the grid gets less consistant since it was more sprawl than the original plan.

Naturally the city has tons of cultural activities and is a great place to live if you are a history geek. I think it is one of the top places to go in the country if you enjoy walking around aimlessly, although there are lots of cities aiming for that title. :)

DC has significant immigrant communities, particularly Latin American and African, which is great for people who like food with a unique cultural twist. There are also great European, Indian, and Asian restaurants if those better suit your taste. The dcfoodies.com blog is a handy resource for finding the best places to eat.

As far as nightlife goes, where you hang out is often determined by proximity. Adams Morgan is the largest night district but there are good lounges, bar, and clubs everywhere. If you are a geek you should have no problem finding beer. Free beer is still rare though.

Conclusion

I would recommend DC to anyone looking for a fun, low pressure lifestyle. There is a lot to do in your spare time, and the copious jobs give lots of options for career advancement at a rapid pace. If you are looking for Silicon Valley and a fast-paced startup culture you may be disappointed, but there are plenty of opportunities available if you work hard enough. Plus you can hang out with senators at bars. That's got to worth something. :)

Til next time.

Hi everyone,

I've been reading a lot of Paul Graham recently and I've become very interested in learning Lisp as a result. Having the ability to program at a higher level of abstraction sounds very helpful, but my concern here is in the "glue" that must be used to make Lisp work with other technologies. I have order a book on the topic from Amazon so I am sure all questions will be answered in dead-tree format shortly.

Being somewhat new to the blogosphere, I'm a still a little mystified by the power of the top handful of bloggers in a particular space. Take Mr. Graham, for instance. There is likely a significant number of younger programmers who would only see Lisp as a relic language if not for his advocacy. If it is the tool that he claims it is I will be very grateful.

I am curious to learn more about what it takes to break into the elite blogger fraternity. My suspicion is that it will take a mixture of writing/storytelling talent (thank you, Mr. Spolsky) and a degree of success in the brick and mortar world. This latter part will help both the quantity and quality of those stories, since narratives read better than thought exercises and everyone loves a happy ending.

Enough talking, time to get started. ;)

Til next time.

Hey everyone,

I just got a Blackberry 8700C and this is my first post from the road. How does it look? :)

Why All Web Programming Frameworks Are Broken

(and how we can fix it)

Anyone who has been around the web programming world (or any programming world, really) has no doubt realized that there isn't a whole lot to what we do; basically, we create pretty forms and screens that manipulate data in some sort of database and return formatted results to the end user. Input goes in, processing happens, and output comes out. There are many different languages used to express this, but the underlying formula remains the same.

Now, programming is a very individualist exercise. We are generally a pretty talented group (at least in our own minds), and understanding the above formula often leads us to believe that we can somehow streamline the process so that creating a new web application takes much less code than it currently does in whatever platform we are using. And in this we are right; everybody who does this professionally comes up with their own framework or CMS system that they think is the cat's meow and purrfectly (sorry) summarizes their view of how a web application should fit together. Very often these systems allow us, as individuals, to spit out web applications faster. Some of the better ones (or better marketed ones) can even help other people make applications faster, as long as they can live with the assumptions made by the original developer.

That said, we have myriad application frameworks out there which promise the great MVC experience, where content is perfectly separated from the presentation, designers can be designers, coders can be coders, and the world will live in perfect harmony. Despite 10 years of this, however, we are still stuck in a world of buggy and difficult-to-maintain code. Systems like Fusebox, Ruby on Rails, Struts, and even .Net can all make your job a little easier if you follow their rules closely, but none are able to live up to the productivity hype which surrounds them or they would quickly jump to the front of the pack and take over the space. Think about it. If RoR made you 10 times as productive as .Net, would anyone still code in .Net? These systems all provide incremental improvements at best. The reason for this is that the problem doesn't lie in the web layer, but in the persistence/database layer.

databases you say?

Yes, databases.

I'll get right to the point; databases are great devices for storing tabular data, but they lack the crucial ability to represent state/time information in a uniform way. This probably requires some explanation, so I will try to do that below.

Relational databases handle data as rows, a logical abstraction which is what makes them good for being able to store arbitrary objects (the classes of which are represented by tables) in a uniform way. The problem comes in how the database can only see 3 different types of transactions on a per-row basis, the CREATE (in which the row is created), the UPDATE (any update to the row), and the DELETE (where the row is removed from the table entirely). For the most part, decent application design rules out ever performing DELETES (except for in associational tables) because it will result in a loss of rows needed for foreign key constraints, so practically speaking there are only two DML statements ever run, and one of those is only done once.

Now the database engine itself is only geared to handle these two transactions. Columns can be declared as NOT NULL, but this is only useful for cases in which the row will ALWAYS contain data. There is no clean way of saying (without ridiculous check constraints or trigger logic) that a field must always have a value when the row it is in reaches a certain point in its workflow. Similarly, any attempt to validate data at the database level (always a good practice if your database supports it) will likely result in adding another ON UPDATE trigger or independent stored procedure, resulting in another complicated piece of application logic that is left to the whims of the developer coding the procedure. Five years down the road, will anyone currently working on the project understand that STPROC_ADD_TICKNUM_U is the procedure to call that will associate a user with a problem ticket? Maybe or maybe not. I would guess that 50% of the applications out there are coded by one programmer who does things his or her own way, making naming conventions entirely arbitrary. Most shops that specialize in development adopt systems to handle just this situation, but even those involve retraining yourself to think like the guy who first developed the system.

Let's do a quick thought exercise to illustrate these issues.

The scenario: a typical web shopping cart, with the requirement that all of the steps of the checkout save themselves to the database so that the customer can pick up the process mid-order in case they leave the site in the middle for some reason.

The schema is simple (and not totally normalized):

Table 1: orders
order_id: int
order_num: varchar(10)
order_date: datetime
ship_date: datetime
status: int
create_date: datetime
mod_date: datetime
bill_name: varchar(100)
bill_address: varchar(200)
ship_name: varchar(100)
ship_address: varchar(200)

Table 2: order_items
order_item_id: int
item_id: int
price: float

Now the workflow. For the orders, we need to make all of the fields except for create_date and order_id nullable, since when the user is first entering data they can enter it piecemeal. The customer said the order_num should be sequential based on the order in which the orders were completed, a silly restraint but one which we must deal with anyway. The billing address, order number, and at least one item must exist when the order reaches "ordered" status, so we need to account for that somewhere. The ship date is only entered if the order reaches the "shipped" status. The create_date is pretty trivial in this example since it only represents the point in time that the user pressed the checkout button the first time, and the mod_date only represents the last time somebody touched the record for any reason. We are forced to have separate fields to account for the progression dates through the various statuses, and to update those either through some kind of application logic.

Even in this trivial and everyday example, we are left with putting the bulk of our order logic within application code, not data definition code. This is a critical difference, because it means the database itself has no concept of an order except for a list of possible fields it might have and therefore we can't use its powerful built-in integrity features to make sure the data stays pristine. In place of that, we have a bunch of procedures or DML statements written by developers in whichever style they choose, which aside from being prone to error, will also be totally arbitrary and difficult for new developers to decode.

a rant

I'd like to take a moment to talk about the art of software development.

I must say art here, because software is unlike any other type of engineering or mathematical discipline. It is often lamented by IT companies, project managers, and customers alike that software development projects are impossible to accurately forecast in terms of time and resources needed.

This is due to the fact that regardless of language, technology stack, and even methodology, 90% of what makes a program do interesting work is arbitrary code written by a single developer. Even if this code follows some sort of framework, this framework is only useful in doing what the designer of the framework intended for it to do. It is doing stuff that it was not designed to do that gets the developer paid.

Taking Ruby on Rails as an example, it contains handy convenience methods and conventions for creating new rows, updating mod_date fields, and even handling row locking (provided Rails is the only thing updating the database). But virtually every single statement you will write will involve some arbitrary WHERE clause, will need to SELECT rows using a JOIN and needing to skip the built-in functions because of the "id" column convention, or do some other cockimamy thing that the designers couldn't include in a general purpose framework. That is before you even get to how you choose to design and allocate functions between your various model classes, what you decide to name your functions, how you choose to design your controllers, what brand of HTML you want to use in your view layer, how you use CSS, etc.

The net result to all of this arbitrary code is that programming doesn't scale very well as you add more developers to it. Every new team member must learn what was done by those who came before in terms of both functionality and coding style, and then they will usually end up adding their own twists on top of that, making the next developer even more confused. For a project lasting a substantial period of time you must also deal with attrition as people leave the team, and then somehow regenerating their knowledge amongst new or existing staff.

As a result of all this, programming productivity doesn't increase linearly as you throw more bodies at it. If you get productivity 1.5 from 2 programmers, it may take you 7 to get to productivity 3.0. And even that is assuming you hire good people. :)

what does this have to do with web programming frameworks?

Good question. Getting back to the task at hand, the issue is that most web programming frameworks base themselves on some sort of database abstraction layer. This is usually the source of their substantial productivity claims, since it lets you (alternately)

1. Do all your programming in one language!
2. Not know SQL!
3. Let the framework handle database/SQL optimization!
4. Let the framework build your database for you!
5. Update your model classes based on any schema changes you make!

While these sound nice on paper, they are often of dubious value (3, 4, 5), are a bad programming practice (1), or outright falsehoods (2). The bigger issue, though, is that when the database doesn't let you design your schema in a way that much metadata can be extracted from it, any database abstraction layer will always be a lowest-common-denominator abstraction. It's convenient to not have to manually update your classes when you add new columns to the database, but you still need to incorporate those changes into any web forms that use the new field and any functions that need to incorporate it into the workflow. This means still more arbitrary, single developer code.

the solution...

The way to get out of this conundrum is to come up with a new version of the SQL standard that can account for status/time as a database-level entity. We need to give rows the concept of status that is separate from a programmer-designated, arbitrary field and a batch of procedures or application language functions.

One idea would be to have an

ALTER TABLE tablename ADD STATUS statusname LEADS TO subsequent_statusname1, subsequent_statusname2, ...

statement which can show the order in which statuses occur. The status that has nothing leading to it will be the initial status. Then triggers could be created on a per-status basis, and stored procedures could be defined that are tied to moves between statuses, either to a new one or back to an old one. Also, special syntaxes can be added for NOT NULL fields to account for the status changes.

Such a system, with some natural extensions, would provide a richer set of DDL-level data for the various frameworks to take into account when creating their model libraries, and some could even go to the lengths of making web pages templates based one what fields are available for editing on which statuses. Over time we can evolve to the point where more logic is stored in the database/persistence layer where it belongs than in arbitrary language code.

conclusion

Web development has reached a point in it is receiving diminishing returns with attempts to refine it further at the application code level. We have long ago distilled websites into their basic elements and attempted to abstract out all that can be abstracted out, and we are left with a battery of competing standards and frameworks which differ very little from one another in anything more than syntax. At the same time, demands for increasing productivity and faster time to market are always ringing in our ears.

The only way to fix this in a fundamental way is to add metadata that is presently being defined in an ad-hoc basis by developers into the database/persistence layer. This will accomplish the dual objectives of giving the framework developers a richer set of data with which to abstract the database, and incorporate information presently stored in crude 3rd generation languages into an elegant 4th generation language like SQL. Database integrity will be assured, and the world will be as one.

Until next time.

Hi Everyone,

So we've been pretty busy lately. :)

JustConfirmed.com has been launched with the beautiful new interface and we are test marketing it to figure out what other features need to be added; so far we've gotten great feedback from several medical offices we've talked to, and Ian is doing a great job at looking for other niches we can fill. StevesLeads.com is in the process of getting a facelift from Martin (a great designer I met from craigslist) and we are hoping to get to the coding of that (the backend is done, we just need the forms and presentation layer) in the not-to-distant future. BlurbCurb is also coming along slowly, but it shouldn't take too long. Most of the backend database-update functions are there already and the interface goes quickly when I have a chance to actually work on it.

Other than that I have been doing some contract work for the past few months, and have recently made a career (day job) transition to a non-profit association downtown. It seems like a great place so far (only been there for a day) and it is my first experience in working not as a contractor for another company or organization. The whole attitude there is much different from what I am used to and they seem to really value their employees, something you don't see much in this day and age. I just need to be careful that the complacency and sense of entitlement don't settle in. :)

Anyway, just wanted to let you all know that I am still alive.

-Mike