Functional test automation through UI is hard

Over the years, I have seen many projects that tried to implement automated acceptance tests by driving application under test through a UI. Most of these efforts fail. Those that don't, are still very cumbersome and expensive to maintain.

I'll leave the whole "why test automation tools industry looks like a massive fraud scheme" rant for a future post. Even good developers, not succumbing to the lure of expensive tools that allow a monkey do test automation, and otherwise doing all the right things, have trouble maintaining UI-based acceptance test suites. Here is why:

An API can be seen as a language that allows a programmer to express something in writing. Now, let's take a mental trip through the architectural layers of some typical business application, starting with the database. Caveat: I'm assuming that the app was designed by object thinkers (as opposed to old school data thinkers), and isn't a VisualBasic-style "SQL in views" reporting app.

As you go up the stack, the layer APIs become progressively more abstract, and more closely related to how a business analyst would think about the actual problem being solved by this software.

• The underlying plumbing (most of which is usually delegated to various third-party frameworks and libraries) is completely unrelated to the business problem.

• Data access layer has to deal with data maping, querying strategies, caching and other boring (from business analyst's perspective) stuff.
• A model is rather close to analyst's thinking, but still has some implementation concerns.
• A well-designed service layer basically talks in terms of problem domain.
• A well designed view layer...oh, wait a second!
  

Suddenly, the picture changes. View layer (markup AND JavaScript, if we are talking about web applications) takes these beautiful abstractions and converts them into a pile of strings, fields, buttons, etc. In other words, low-level constructs almost completely unrelated to the problem at hand. Ugly plumbing, necessary to show the user what's going on. Wonder why many developers shy away from the UI work... until they read something like The Design of Everyday Things and discover a whole new dimension in their craft, that is :) But I digress.

So, abstraction level at UI layer goes waaay down, and then your UI-based acceptance tests have to deal with those low-level constructs to see if the software actually meets the high-level business requirements. In a way, this is similar to the object-relational impedance (otherwise known as Vietnam of software development). Save for the fact that people doing UI automation rarely recognize that the problem even exists.

Next time I'm on the plane, I'll try to write something about alleviating this problem.