Eighteen months ago, Mr. Cooper launched an intelligent recommendation system for its customer service agents to suggest solutions to customer problems. The company, formerly known as Nationstar, is the largest non-bank mortgage provider in the U.S., with 3.8 million customers, so the project was viewed as a high-profile cost-saver for the company. It took nine months to figure out that the agents weren’t using it, says CIO Sridhar Sharma. And it took another six months to figure out why.
The recommendations the system was offering weren’t relevant, Sharma found, but the problem wasn’t in the machine learning algorithms. Instead, the company had relied on training data based on technical descriptions of customer problems rather than how customers would describe them in their own words.
“We didn’t do a good job of making sure that the root of the question that the customer was asking was captured in the terms the customer was using,” he says. “It was coded in the technical terms that we were using internally.”
In addition, the system’s feedback mechanism, in which agents recorded the results of the calls, had overlapping categories, which made the problem even worse, says Sharma, who declined to say how much the project cost the company.
Mr. Cooper’s troubled foray into AI is no anomaly. According to a recent IDC survey, only about 30 percent of companies reported a 90 percent success rate for AI projects. Most reported failure rates of 10 to 49 percent, while 3 percent said that more than half of their AI projects failed.
