But, I do think a lot of the problems stem from communication problems. Not only is it difficult to communicate our desires to the computer, it’s also frustrating when we don’t understand why it has done something. I think that if we understood why, we would probably be willing to put up with problems more.

And you’re right, there are several examples of automation where the problems outweigh the benefits. There are other scenarios — web search for example — where things would be completely unmanageable without some amount of “smart” decision making by the computer.

At some point, researchers in AI saw that user interfaces was a nice domain for applying their intelligent techniques and algorithms. Modelling users and predicting their needs sounds as a very challenging problem. I felt this challenge when I started switching from a more theoretic background to HCI. People who like algorithms and modelling techniques will try to apply them everywhere even if this is not appropriate. Nevertheless, starting from this point is totally wrong. Many poor designs of intelligent UIs have been driven less by real user needs and more by the desire of AI people to apply their theories. Yet, such theories are subject to unrealistic assumptions about users. Unfortunately, the greatest amount of the work on intelligent and adaptive user interfaces has been conducted on such grounds. I believe that the correct approach to designing a new UI is to first study the user needs and apply automation only if this approach seems to satisfy these needs much better than a simply intelligently-designed UI. In my experience, automation can be useful in a few only situations and not for everyone. People tend to consider that if a UI fails for the majority of the people, then it is not useful. What if a 20% of users get a great benefit from it?

I think that the use of the conversation metaphor to demonstrate the limitations of intelligent systems is quite misleading (I admit that I have also used it in the past as it is very elegant). Automation should not been designed having in mind a companion who tries to understand what the user wants. This is a wrong metaphor, whose use has lead to very poor designs. Finally, let me remind you of two very simple examples of automation who have been very successful even if their decision-making mechanism is not completely transparent. My mailbox would be unusable without my junk filter as I receive 100 junk messages per day. It rarely misses really junk messages and almost never filters out useful messages. I also find extremely helpful the suggestion/correction mechanism provided by eclipse although it makes several mistakes. When benefits coming from automation outweigh the cost of errors, then automation is useful. The cost of an error when driving a car is huge, but this is an extreme scenario.