MIT Researchers Develop an Efficient Way to Train more Reliable AI Agents

Fields varying from robotics to medicine to political science are attempting to train AI systems to make meaningful choices of all kinds. For example, using an AI system to intelligently manage traffic in a busy city might help drivers reach their locations faster, while improving safety or sustainability.

Unfortunately, teaching an AI system to make excellent decisions is no easy job.

Reinforcement learning designs, which underlie these AI decision-making systems, still typically fail when faced with even small variations in the jobs they are trained to perform. When it comes to traffic, a model may have a hard time to manage a set of crossways with various speed limits, varieties of lanes, or traffic patterns.

To increase the dependability of reinforcement learning designs for complex tasks with variability, MIT scientists have actually presented a more efficient algorithm for training them.

The algorithm tactically picks the best jobs for training an AI representative so it can efficiently carry out all tasks in a collection of related jobs. In the case of traffic signal control, each job might be one intersection in a job space that includes all intersections in the city.

By concentrating on a smaller number of crossways that contribute the most to the algorithm’s general efficiency, this approach makes the most of performance while keeping the training cost low.

The scientists discovered that their strategy was between five and 50 times more effective than standard techniques on a range of simulated tasks. This gain in efficiency helps the algorithm find out a better solution in a faster way, ultimately improving the efficiency of the AI representative.

“We had the ability to see unbelievable performance enhancements, with a very simple algorithm, by believing outside package. An algorithm that is not extremely complex stands a much better possibility of being adopted by the neighborhood since it is simpler to carry out and much easier for others to comprehend,” states senior author Cathy Wu, the Thomas D. and Virginia W. Cabot Career Development Associate Professor in Civil and Environmental Engineering (CEE) and the Institute for Data, Systems, and Society (IDSS), and a member of the Laboratory for Information and Decision Systems (LIDS).

She is joined on the paper by lead author Jung-Hoon Cho, a CEE graduate student; Vindula Jayawardana, a graduate student in the Department of Electrical Engineering and Computer Technology (EECS); and Sirui Li, an IDSS college student. The research study will exist at the Conference on Neural Information Processing Systems.

Finding a happy medium

To train an algorithm to manage traffic lights at numerous intersections in a city, an engineer would normally select between two main methods. She can train one algorithm for each crossway individually, utilizing only that crossway’s data, or train a larger algorithm utilizing information from all intersections and after that use it to each one.

But each method comes with its share of disadvantages. Training a different algorithm for each job (such as an offered intersection) is a time-consuming process that needs an enormous amount of information and calculation, while training one algorithm for all tasks frequently results in substandard performance.

Wu and her partners sought a sweet area in between these two techniques.

For their method, they pick a subset of jobs and train one algorithm for each task individually. Importantly, they strategically choose private jobs which are probably to improve the algorithm’s overall performance on all jobs.

They utilize a typical technique from the support learning field called zero-shot transfer learning, in which a currently trained model is applied to a brand-new job without being more trained. With transfer learning, the design often carries out extremely well on the brand-new neighbor task.

“We understand it would be ideal to train on all the tasks, however we wondered if we could get away with training on a subset of those jobs, apply the outcome to all the jobs, and still see a performance increase,” Wu states.

To determine which tasks they must choose to make the most of expected efficiency, the researchers established an algorithm called Model-Based Transfer Learning (MBTL).

The has two pieces. For one, it models how well each algorithm would perform if it were trained separately on one task. Then it designs how much each algorithm’s performance would deteriorate if it were moved to each other job, a concept understood as generalization efficiency.

Explicitly modeling generalization efficiency allows MBTL to approximate the value of training on a new task.

MBTL does this sequentially, selecting the job which causes the highest performance gain first, then selecting extra jobs that provide the most significant subsequent marginal improvements to total performance.

Since MBTL just concentrates on the most appealing jobs, it can drastically enhance the performance of the training procedure.

Reducing training expenses

When the researchers evaluated this technique on simulated jobs, consisting of controlling traffic signals, handling real-time speed advisories, and carrying out numerous traditional control tasks, it was five to 50 times more efficient than other methods.

This indicates they might get to the same option by training on far less data. For circumstances, with a 50x effectiveness increase, the MBTL algorithm might train on simply 2 jobs and achieve the same efficiency as a standard approach which utilizes information from 100 tasks.

“From the point of view of the two primary techniques, that indicates data from the other 98 tasks was not needed or that training on all 100 jobs is confusing to the algorithm, so the efficiency ends up worse than ours,” Wu says.

With MBTL, adding even a small amount of additional training time might cause better performance.

In the future, the researchers plan to create MBTL algorithms that can extend to more complex problems, such as high-dimensional task spaces. They are also thinking about applying their approach to real-world problems, specifically in next-generation mobility systems.