Ph.D. Defense

Multi-Source Option-Based Policy Transfer

James MacGlashan

10:00am Friday, 25 January 2013, ITE 325B

Reinforcement learning algorithms are very effective at learning policies (mappings from states to actions) for specific well defined tasks, thereby allowing an agent to learn how to behave without extensive deliberation.  However, if an agent must complete a novel variant of a task that is similar to, but not exactly the same as, a previous version for which it has already learned a policy, learning must begin anew and there is no benefit to having previously learned anything. To address this challenge, I introduce novel approaches for policy transfer. Policy transfer allows the agent to follow the policy of a previously solved, but different, task (called a source task) while it is learning a new task (called a target task). Specifically, I introduce option-based policy transfer (OPT). OPT enables policy transfer by encapsulating the policy for a source task in an option (Sutton, Precup, & Singh 1999), which allows the agent to treat the policy of a source task as if it were a primitive action. A significant advantage of this approach is that if there are multiple source tasks, an option can be created for each of them, thereby enabling the agent to transfer knowledge from multiple sources and to combine their knowledge in useful ways. Moreover, this approach allows the agent to learn in which states of the world each source task is most applicable. OPT's approach to constructing and learning with options that represent source tasks allows OPT to greatly outperform existing policy transfer approaches. Additionally, OPT can utilize source tasks that other forms of transfer learning for reinforcement learning cannot.

Challenges for policy transfer include identifying sets of source tasks that would be useful for a target task and providing mappings between the state and action spaces of source and target tasks. That is, it may not be useful to transfer from all previously solved source tasks. If a source task has a different state or action space than the target task, then a mapping between these spaces must be provided. To address these challenges, I introduce object-oriented OPT (OO-OPT), which leverages object-oriented MDP (OO-MDP) (Diuk, Cohen, & Littman 2008) state representations to automatically detect related tasks and redundant source tasks, and to provide multiple useful state and action space mappings between tasks. I also introduce methods to adapt value function approximation techniques (which are useful when the state space of a task is very large or continuous) to the unique state representation of OO-MDPs.

Committee: Dr. Marie desJardins (Chair), Dr. Tim Finin, Dr. Michael Littman, Dr. Tim Oates, Dr. Yun Peng

Every year the website Edge.org poses an annual question to dozens of scholars, scientists, writers, artists and thinkers.  Among them is Robert Provine, professor of psychology.

NPR’s 13.7 blog wrote about this year’s question, “What should we be worried about?” in a January 14 post.

Themes that the blog detected in the answers included “the impact of technology on individual minds and human relationships,” “the growing disconnect between science and the rest of human knowledge and culture,” and “worry itself.”

Provine gave a positive spin on the question, urging people to “stop worrying about worry.” “Too much worry strands us in an agitated state of despair, anxiety and paranoia; too little leaves us without motivation and direction,” he wrote.

Based on Provine’s advice, readers are urged to “go ahead and read all 150-plus responses to this year’s challenge. But in the interest of worrying just the right amount, you might not want to do so in a single sitting.”

UMBC professor Dr. Tim Oates has been named an Oros Family Professor in Computer Science and Technology. The five-year endowed professorship will fund Dr. Oates’ newly proposed research project in the area of mobile healthcare.

Read more about endowed professorships at UMBC here.

A professor of computer science and electrical engineering, Dr. Oates is principal investigator in UMBC’s Cognition, Robotics and Learning (CoRaL) Lab, where he pursues research in the broad areas of artificial intelligence, machine learning, robotics, and natural language processing.

“I was delighted to recommend Dr. James T. Oates’ appointment as Oros Family Professor in Computer Science and Technology, an endowed professorship established to honor and support the work of faculty pursuing scholarly activity in computer science,” says Dr. Warren Devries, Dean of UMBC’s College of Engineering and Information Technology.

“Tim Oates is a first rate teacher and scholar, and I had a chance to work with him as a key part of the College and CSEE department’s accreditation activity. Taken together this is precisely the type of faculty member this professorship is intended to recognize. The CSEE department’s faculty, staff and students are fortunate to have Tim Oates as a mentor and colleague.”

The funds will support graduate students to work with Dr. Oates on a project to improve healthcare in developing countries. “The issue is that in developing countries, almost everything is still done on paper,” explains Dr. Oates. That means that every record—things like immunization records and medical histories–are taken by hand. It’s an inefficient process that makes it hard to keep track of medical data.

His proposed solution is to develop an algorithm that can extract data from a cell-phone photo of the paper record. Once digitized, the medical data can be more easily shared and analyzed.

Dr. Oates chose to develop this technology for cell phones because they are a common and reliable tool in these areas, he explains. In many cases, cell phones can be found where basic needs are absent. “There are people living in villages that don’t have access to clean water, but they have cell phones,” he says.

Dr. Oates joins last year’s Oros Family Professor appointee, Dr. Anupam Joshi, who is similarly using the funds for his research in mobile healthcare.

– by Anissa Elmerraji ’11

This story originally appeared on the CSEE website.

Portions of two NASA Terra ASTER swaths, one before and one a decade after the collapse of the Larsen B ice shelf (early 2002) illustrate dramatic changes in ice area and glacier thickness. The ice edge position in early 2012 is indicated by the thin blue line that now extends well into each tributary of what was the Larsen B ice shelf. In a decade, nearly 5000 km² (~1900 mi²) of ice has been lost along this part of the Antarctic Peninsula. Ice thickness changes totaling up to 220 m (~720 ft.) have been documented by DEM differencing (digital elevation models, e.g. Berthier et al., 2012, GRL) but can also be clearly seen by comparing bedrock outcrops in the lower Hektoria and other glaciers. Please click here for larger image.