HomeMaestro is a prototype of a home automation system that I built as part of my research to address the numerous issues plaguing current home automation systems. It consists of two main features: 1) a tangible, appliance-scripting interface, and 2) an "app store" for quickly and easily downloading functionality to the home. Or, as I like to pitch it, HomeMaestro is a platform for intuitively defining home appliance behavior.


(In case you're confused as to what a "light shelf" is, watch this.)

What follows is a pseudo-academic write up of my research and design process leading to the prototype.

Background and Current Industry Landscape

As part of my research with the Changing Places group at the Media Lab, I spent a good portion of my first semester looking into home automation systems. Home automation, for those who aren't familiar with it (and many, surprisingly, aren't), can encapsulate many disparate technologies; it can refer to systems that allow home owners to remotely control appliances such as lights and locks, systems that allow the creation of scripted "scenes" for appliances, systems that can seamlessly move media from device to device in the home, or even systems that allow for remote access to video feeds from security cameras.

The concept of home automation has been around for a long time and products have been on the market for decades, though no one solution has broken through to the mainstream yet. [5] Home automation technology generally consists of two parts: the communication protocols to get data to and from home appliances, and the user interfaces for controlling them.

There are dozens of existing home automation standards and protocols, the oldest of which surprisingly date back to 1975. Some are based on in-powerline communication such as X10, INSTEON, and KNX, some are wireless such as Z-Wave and ZigBee, and some are internet-connected and utilize RESTful web-based API's like ioBridge. Many of the organizations behind these protocols have industry partners that at one point or another have pledged to build products using those standards.

There are over 50 companies in the home automation industry in North America, yet the entire industry earns less than $1 billion in revenue annually, with the high-end market accounting for 82.2% of that in 2010. 32.6% of the market share is controlled by three main home automation companies: Crestron Electronics, Inc., AMX Corporation, and Control4 [11].

Many of these companies tend to have niche focus areas. Some are focused on the power monitoring and energy efficiency angle, such as Zerofootprint's TalkingPlug, and some specialize in one specific appliance type, such as Lutron's very extensive lighting control systems. Others focus solely on the DIY market and instead of selling an integrated, productized solution, they sell computer chips and let customers hack away at their own control and monitoring projects. These include ioBridge and XBee.

Two somewhat more interesting companies to recently enter the home automation foray are Google and Microsoft. At its 2011 I/O conference, Google announced Android@Home, an ambitious project in which it aims to install the android operating system on third party appliances. Google hasn't released too much to the market yet, but a big announcement is expected soon. Microsoft is similarly working on a project called HomeOS, an operating system for the home.

Problems with Existing Home Automation Solutions

Despite over 30 years of activity in the industry, none of the solutions in existence today have gone mainstream, leading me to conclude that home automation has essentially failed as a technology. By way of reading papers, talking to home automation users, and playing around with home automation products myself, I've discovered that there are a few superficial reasons for this as well as a few deeper reasons. The superficial reasons seem to be the following:

• Current systems are extremely expensive and generally only cater towards upper-class families with large houses and estates. These solutions are often intended simply to help the family manage complex lighting or home entertainment systems. [2][5][11]
• Current systems are difficult to install. They require professionals to go to the customers' houses one or multiple times to install appliances and configure control systems. [2][5][9][11]
• Current systems are generally difficult to use. Companies seem to focus on either the technology or interface side of things, but not both. As a result, control interfaces tend to have small, poor quality, black and white LCD screens with lots of buttons that are difficult to navigate and program. [2][9][10][11]
• Current systems tend to provide solutions for only one appliance category, such as lights or blinds. Thus, if customers want to control multiple appliance types, they need to coordinate amongst multiple different companies and deal with multiple different systems in their homes. [2][5][6]
• Current systems tend to provide either monitoring or control functionality, but not both. Thus, if a customer wants to both monitor an appliance's power usage and have the ability to remotely turn it on and off, he must again implement two different systems from two different companies. [5][6]
• Current systems are generally proprietary and closed, meaning that they are not very customizable by the end user. This constrains what users can do with the system to what the designers originally intended, which doesn't take the individual needs and daily lives of the users into account. [5]

When taken as a whole, these superficial problems create a very poor user experience with home automation systems, while at the same time preventing users from truly understanding the full range of capabilities that home automation has to offer.

The deeper reasons for home automation's failure are even more troublesome:

• There is always an exception to the rule. No matter how carefully a user scripts up his home, because people's schedules and preferences are always changing, any home automation "rule" will only work as anticipated at most 80% of the time. The other 20% of the time, it will fire unexpectedly, annoying or perhaps embarrassing the user. [5][8]
• Especially with internet connected systems, there are always security perception problems. Even if the home automation system is technically 100% secure, home owners will be concerned about safety, fearing that their door locks and security cameras have the potential to be "hacked". [2][5]
• Artificially intelligent home automation solutions often get so "smart" that their actions, in effect, become unintelligible to the user. The house may very well have calculated the precisely optimal time to open the windows based on time of day, weather conditions, and optimal cross breeze opportunities, but the user might have simply not wanted the window open right then, and because the house is a black box off doing its own thing, the user has no line of action to correct the mistake. [7]
• The "Good Mom" problem also pops up occasionally in the literature and in interviews. This is the theory that, parents and other family members have specific jobs that make them feel like productive and valuable members of the household (cooking, cleaning, etc), and therefore do not want a home automation system to take those jobs away from them. [8]
• The multi-user problem is an issue that occurs in households with more than one occupant. When more than one person (and thus, more than one set of preferences) is defining how the home should behave, scripting conflicts and ambiguities in desired behavior will naturally arise. [1]
• Finally, some researchers in the field believe that, even if one could solve all of the above problems, there are still no compelling use cases for a home automation system. They claim that people just don't care enough to want to be able to take out their iPhones and turn off their lights. They have no problem just getting up and hitting the light switch.

The Whole HomeMaestro System

The system I prototyped is only a small subset of the entire home automation system that I envision as a solution to most of these problems. I will first describe how the whole system would function, and then explain how the prototype fits within that vision.

There are four main products/features that make up the proposed system:

• A computer chip for manufacturers to include in their appliances. It can be easily configured for any type of appliance.
• A small Boxee-like box which contains the hardware and software. It runs an embedded version of an operating system which manages the devices in the home. This is all that home owners need to set up the system in their houses. Compatible appliances can be added to the house gradually over time as the customer sees fit.
• Interfaces for all types of mobile devices. These allow customers to see their appliances, send them commands, as well as ask them status questions. It also allows creation of cause-and-effect-based "rules" that chain multiple appliance commands together in a logical way.
• An Apple-app-store-like "Rule Marketplace". When users create rules, they have the option of uploading them to the Rule Marketplace to share the rules others. The rules can either be offered for free or for a price.

The computer chips for appliance manufacturers are small, wireless, devices that get included into the internal structure of a given appliance. The chips are powered from the same power source that the appliances use (usually mains power), and are as simple for manufacturers to use as the popular Arduino micro-controller platform.

The chips work in the following way:

• When they are powered on, they automatically register themselves with the box. They pass the box a unique ID that the manufacturer receives for this particular appliance and configures into the chip. For example, when an alarm clock is powered on for the first time, a user of the interface might see an icon of an alarm clock automatically pop up in the list of available appliances for the household.
• They have the ability to fire appliance-based events off to the box. For example, an oven might fire a "preheat temperature reached" event when the oven has reached the desired temperature. A user interface would show this event automatically pop up, and it would also trigger any rules that are contingent upon the oven's preheat temperature being reached. Manufacturers configure the events that can be fired upon installation.
• They listen for and execute commands. For example, the box may send an "off" command to a light fixture. The chip in the light fixture would recognize the command, and send the appropriate signal to the light to turn it off. Manufacturers configure the commands upon installation.
• They listen for and respond to queries. For example, the box may ask a TV what channel it is currently tuned to, and the TV would respond with the appropriate number. Manufacturers again configure these upon installation.
• They have built in power-monitoring capability, which the manufacturer has to do nothing to configure. This acts as an additional, special query, and power usage values can be retrieved by the box at any time.

The box is a small, wireless hub that communicates with all of the chips in the home's appliances. It contains the software that serves up the user interface and also contains the storage space needed to remember user-defined settings and rules. It is the only thing home owners need to use the HomeMaestro system. Setting it up is cheap and easy: Step 1: Buy the box. Step 2: Plug it into an outlet and ethernet port. Step 3: That's it. Really. No further configuration needed.

While nothing besides the box is required upon installation, users can add more automation functionality to their homes by purchasing and plugging in HomeMaestro-compatible appliances, which automatically register themselves with the box when they power up for the first time. Appliances (and thus richer home automation functionality) can be added to the home gradually over time as budgets allow. In this way, users do not need to purchase a large home automation solution in one go. They can pick and choose what they want to add to the system and when they want to add it.

The box works with any appliance that has the chip. Manufacturing of these appliances is distributed so customers are not forced to buy all from the same company, or to only have one type of compatible appliance.

The box also has an ethernet connection so it has the capability to do interesting things over the network. For example, it could allow users to create rules that upload home activity data to an online database for long term storage and data analytics, as well as monitor, control, and create rules for appliances in a remote home.

The interface to the system has the following features:

• Users can choose from any mobile or web-based interface.
• The "My Appliances" section displays a list of appliances in the home.
• When a user taps the icon for a specific appliance, it shows a list of all commands that can be sent to the appliance (i.e. "turn to 375 degrees"), as well as all questions that can be asked of the appliance (i.e. "what temperature are you currently at?"). Users can simply tap an action to make the device perform that action or tap a question to find out the answer.
• Users can tap "Record" to record a new rule. As they then move about the house and interact with appliances, the appliances fire events, which are recorded by the box. Helpful tips, hints, and modifiers pop up as needed in just-in-time fashion. The rule is saved, added to the list of rules, and activated for future use in the house.
• Slightly more advanced users can tap "Create" to use an intuitive drag and drop interface (akin to MIT's Scratch or Lego's MindStorm interfaces) to build up a custom rule in a more detailed way.
• Very advanced users can view documentation for the API as well as use a tool to auto-generate code for API calls.
• The "My Rules" section shows a list of all rules that are currently running in the house. Users can tap any rule to see what it does (i.e. "When I hit off on my alarm clock, open the blinds and start the coffee maker."). They can tap "disable" to temporarily disable it, or "delete" to remove it completely.
• Users can upload rules they are particularly proud of to the marketplace with one click to share it with others. They can also browse and download any rules they find particularly useful that have been made by other customers or appliance manufacturers.

As one can see, this entire system aims to solve all of the superficial problems, and at least begins to address most of the deeper problems.

The Prototype

The prototype in the video above is an implementation of the "Record" and "Rule Marketplace" features previously mentioned. The reason I chose these features is because together they address the main usability problems with home automation systems today. I strongly believe that once home automation becomes accessible to everyone (not just the extremely rich or extremely nerdy), people will quickly find the value in it and even begin to discover and invent interesting new use cases for it that no one has yet envisioned. Just as the Apple app store revolutionized the concept of the phone, I believe HomeMaestro could revolutionize the concept of the home.

At the moment, the prototype has three main features:

1. Users can hit the "Record" button to record simple "When/Then" style rules. Any number of events can be added to the "When" section (not just one as in the demo video), and they are related to each other with an AND conjunction. That is, only when ALL of those events in the "When" section happen at about the same time will the rule fire. Also, any number of events can be added to the "Then" section (again, not just one), and they will all be fired at the same time when the rule is triggered.
2. Users can swipe over to the "My Rules" sections to see all the rules they currently have running. They can select an individual rule to view a human readable description of how it behaves, and they can also enable or disable any particular rule.
3. Users can swipe over to the "Market" section where they can view pre-made rules and install whichever ones they want.

Potential Use Cases

The most compelling potential use cases seem to occur in a few genres: convenience and time-saving [1][3][5], money-saving and energy-saving [3][7], healthcare [4], communication, peace of mind [1][5] and quality of life [5], and entertainment. The following are hypothetical scenarios depicting how a HomeMaestro user might use the system.

• Everyday a businessman man wakes up at 7am, turns off his alarm clock, opens the blinds, gets dressed, turns on the news, then makes some coffee. One morning, he hits the record button on his HomeMaestro mobile interface as he performs these actions, which adds a "rule" to his house's list of rules. The next morning at 7am, the house automatically turns off his alarm clock, opens the blinds, turns the TV to channel 2, and turns on the coffee maker. The next week he decides to take a personal day off from work, so he un-checks the morning routine rule from the list of rules on his mobile interface before he goes to bed.
• An energy conscious college student uses the drag and drop rule creation interface to create a rule that asks all of his devices for their power usage information every minute then uploads it to pachube.com so he can see a graph of his usage over time. His friends visit his dorm and love it, so he uploads his rule to the rule marketplace. His friends download it in their own dorms and immediately begin monitoring their own energy usage.
• An energy and money conscious young professional records a few rules to ensure she never uses too much energy or spends too much on it. One rule allows only 5 appliances in her home to ever be on at the same time; if a 6th appliance is turned on, it automatically turns off the appliance that was turned on the longest time ago. Another rule automatically turns on the washing machine and HVAC system when electricity prices in her area are at their lowest.
• A tech-savvy high school student programs his own first person shooter iPad game. Using HomeMaestro's well-documented API's, he writes some code that causes the player's house lights to flicker when shooting a gun. In addition to selling his game in the Apple app store, he sells these rules in the rule marketplace as an added feature to make his game more immersive.
• A security conscious father records a rule that says, "If the front door is opened between the hours of 12am and 8am, flash all the lights in the house and set off the alarm."
• A bored middle schooler uses the drag and drop interface to create a game in his house. It turns off all the lights in the house, then turns on the light in a random room. He has 20 seconds to find the room and turn off the light, at which point it turns on the light in another random room, and so on. If he doesn't make it in 20 seconds, all the lights turn off and the game starts again.
• The concerned son of an aging woman sets up HomeMaestro in his mother's house. He uses the drag and drop interface to create a rule that sends him a text message if his mother hasn't opened the refrigerator or turned on the oven in the past 24 hours. He keeps an eye on his phone to make sure his mother is eating enough throughout the day.
• Due to job requirements, a married couple must live thousands of miles away from each other for a little while. They use the record feature to create a rule that ties a few of the lights in their homes together so that when one goes on in one house, it turns on a light in the other house and vice versa. This way, even when they're not talking on the phone, they can have the comforting feel of the other's presence as they move about the house. Due to timezone differences, they modify the rule to make sure its only active between mutually convenient hours.
• A woman sets up a large amount of rules in her house and gets them exactly how she likes them. One day she is offered a promotion and must move to the other side of the country. She takes her rules with her so that her new house behaves exactly like she's used to.

Future Work

Eventually, I intend to do a user study that will evaluate the effectiveness of this sort of "role-playing" interface. I would also like to add the following features to prototype to make it more versatile and general purpose:

• Allow the "When" actions to be combined with not only an "AND", but also an "OR" and "NOT".
• When a "When" action is added while recording, show a popup menu in just-in-time fashion to allow more fine-grained control of the event. For example, if a user turns on a fan and "when the fan turns on" appears, some options would balloon out of that event which the user can choose, such as "...after X o' clock", "...before X o' clock", "...and its more than X degrees outside", etc.
• Allow users to upload their own rules to the rule market place.

• [1] Tiiu Koskela, Kaisa Vaananen-Vainio-Mattila (2004) Evolution towards smart home environments: empirical evaluation of three user interfaces. Pers Ubiquit Comput 8: 234–240
• [2] W. Keith Edwards, Rebecca E. Grinter, Ratul Mahajan, David Wetherall (2011) Advancing the State of Home Networking. Communications of the ACM Vol. 56, No. 6
• [3] Genevieve Bell, Joseph ‘Jofish’ Kaye. Designing technology for domestic spaces: A Kitchen Manifesto.
• [4] Julie A. Kientz, Shwetak N. Patel, Brian Jones, Ed Price, Elizabeth D. Mynatt, Gregory D. Abowd (2008) The Georgia Tech Aware Home. CHI 2008 Proceedings · Research Landscapes.
• [5] A.J. Bernheim Brush, Bongshin Lee, Ratul Mahajan, Sharad Agarwal, Stefan Saroiu, Colin Dixon (2011) Home Automation in the Wild: Challenges and Opportunities. CHI 2011.
• [6] Colin Dixon, Ratul Mahajan, Sharad Agarwal A.J. Brush, Bongshin Lee, Stefan Saroiu, Victor Bahl (2010) The Home Needs an Operating System (and an App Store)
• [7] Stephen S. Intille (2002) Designing a Home of the Future. Pervasive computing.
• [8] Scott Davidoff, Min Kyung Lee, Charles Yiu, John Zimmerman, and Anind K. Dey (2006) Principles of Smart Home Control. Ubicomp 2006, LNCS 4206, pp. 19 – 34, 2006
• [9] Rebecca E. Grinter, W. Keith Edwards, Mark W. Newman, Nicolas Ducheneaut. The Work to Make a Home Network Work.
• [10] Marshini Chetty, Ja-Young Sung, and Rebecca E. Grinter (2007) How Smart Homes Learn: The Evolution of the Networked Home and Household. UbiComp 2007, LNCS 4717, pp. 127–144, 2007.
• [11] Frost & Sullivan (2011), US Markets - Home Automation