Functional test automation through UI is hard
Over the years, I have seen many projects that tried to implement automated acceptance tests by driving application under test through a UI. Most of these efforts fail. Those that don't, are still very cumbersome and expensive to maintain.
I'll leave the whole "why test automation tools industry looks like a massive fraud scheme" rant for a future post. Even good developers, not succumbing to the lure of expensive tools that allow a monkey do test automation, and otherwise doing all the right things, have trouble maintaining UI-based acceptance test suites. Here is why:
An API can be seen as a language that allows a programmer to express something in writing. Now, let's take a mental trip through the architectural layers of some typical business application, starting with the database. Caveat: I'm assuming that the app was designed by object thinkers (as opposed to old school data thinkers), and isn't a VisualBasic-style "SQL in views" reporting app.
As you go up the stack, the layer APIs become progressively more abstract, and more closely related to how a business analyst would think about the actual problem being solved by this software.
- The underlying plumbing (most of which is usually delegated to various third-party frameworks and libraries) is completely unrelated to the business problem.
- Data access layer has to deal with data maping, querying strategies, caching and other boring (from business analyst's perspective) stuff.
- A model is rather close to analyst's thinking, but still has some implementation concerns.
- A well-designed service layer basically talks in terms of problem domain.
- A well designed view layer...oh, wait a second!
So, abstraction level at UI layer goes waaay down, and then your UI-based acceptance tests have to deal with those low-level constructs to see if the software actually meets the high-level business requirements. In a way, this is similar to the object-relational impedance (otherwise known as Vietnam of software development). Save for the fact that people doing UI automation rarely recognize that the problem even exists.
Next time I'm on the plane, I'll try to write something about alleviating this problem.