Picwell Blog

Study: Picwell’s AI-Powered Decision Empowers Health Insurance Agents

Apr 30, 2020 1:40:14 PM / by Sam Kina

Millions of people must choose a health insurance plan each year.  From seniors enrolling in Medicare to the large share of the U.S. population who choose an employer provided offering or selecting among plans on a state exchange, the one constant across all of these settings is that picking the right health insurance plan is extremely difficult. 

Why is insurance so different from other markets where consumers seem to make reasonable choices?  Insurance inherently deals with uncertainty that is very difficult for people to accurately assess.  Picking the right health insurance plan requires complex calculations that people just aren’t good at, including skilled experts.  However, AI-based technologies are well suited for solving these types of problems, and a recent study conducted by a team of economist, including Professors Jonathan Kolstad and Ben Handel of UC-Berkeley, Professor Jon Gruber of MIT, and Samuel Kina of Picwell, showed how Picwell’s AI-based decision support improved enrollment agents’ performance, helping them enroll their customers in better plans in less time.

In this study the research team looked at the performance of approximately 800 enrollment agents who helped customers choose Medicare Advantage plans between 2015 and 2017.  These enrollment agents worked for a private health insurance exchange that implemented Picwell across the board in 2017, requiring all agents to use Picwell’s decision support during their enrollment consultations.  Picwells’ decision support takes basic customer information, including age, sex, zip code and a list of routine prescriptions, uses machine-learning algorithms to predict both the cost and risk of each plan available to a consumer, and then identifies the best match for each customer, balancing cost, risk and consumer preferences for risk protection and other plan attributes such as provider networks and CMS star ratings.

Prior to the widespread adoption of Picwell on the exchange, agents expended considerable time helping their customers choose Medicare plans, typically spending close to an hour with each customer.  Even after all of that effort, the team found that the average plan enrollment led to a $1,260 financial loss for consumers, relative to the best financial option available to them.  These foregone savings amount to 30% of total costs.  

The team studied how choices are made by carefully assessing the factors that enter enrollee-agent product decisions.  For example, after controlling for other factors, a rational, informed enrollee or agent should consider an additional $1 in premium to be the same as an additional $1 in out-of-pocket spending; they’re both worth a dollar. In practice, without AI, choices are between 6 and 7 times more sensitive to plan premiums than to expected out-of-pocket spending.  In short, skilled agents alone do not overcome the choice errors made by consumers. 

When the team looked at the choices made in 2017, it found that after Picwell was implemented, the sources of choice error that were present in 2015 were reduced substantially and, in some cases, eliminated.  With AI, decision-making placed equal weight on premiums and expected out-of-pocket costs and placed less emphasis on plan characteristics such as deductible and out-of-pocket maximums, instead focusing on what actually matters to consumers: total cost. 

The end result of this improved decision making process was better choices.  After AI was adopted, the number of people who choose the best Medicare advantage plan - which, in this study, was defined as the financially optimal choice - nearly doubled from 10% to 19%.  Among enrollees who did not pick the financially optimal plan, the magnitude of choice errors fell across the entire distribution.  The median choice error was cut in half and even at the 90th percentile, large improvements in choice error were observed.

Distribution of choice error

After controlling for differences between the two time periods, the study concluded that improved decision making reduced choice error by an average of $278 per enrollee, representing a 22% reduction in “choice error” or the financial loss associated with picking a suboptimal plan.  Even after the adoption of decision support, many people selected plans that were not financially optimal, but there are many rational reasons why someone would select a plan that is not financially optimal - they may place a high premium on having access to the broadest network possible, they may have a strong preference for selecting a certain type of plan, such as a PPO over an HMO or an integrated health plan over a traditional health plan. The important finding is that AI shifted decision making in the right direction, towards more financially optimal, or better value, health plans.

The research team also looked at the relationship between improved choices and customer satisfaction by asking whether those who enrolled in recommended plans were more likely to stick with those choices over time.  The answer was a resounding yes: people who enrolled in plans that Picwell did not recommend in 2017 were more than twice as likely to switch plans the following year, compared to people who enrolled in a recommended plan.  They then develop a strategy to causally assess the impact of Picwell’s recommendations on turnover in Medicare Advantage, demonstrating that plans the AI predicts to be a better fit have dramatically lower switching.

Switch rates in 2018

In addition to evaluating how Picwell’s AI-based decision support improved health plan selections, the team studied whether AI has a greater impact on higher or lower quality agents.  They find that decision support improves recommendations for the large share of agents who were not performing well prior to AI (approximately 80%).  In fact, with AI, the least skilled agents make recommendations that are better than the highest quality agents without AI.  Furthermore, the quality of recommendations was equivalent across the entire agent distribution, regardless of skill or effort. 

The team also analyzed the impacts of decision support on agent productivity, finding that the introduction of decision support lowered call times by more than 20%, while improving recommendations.  Productivity gains are even larger for the lowest skilled agents.  While all agents spent less time on the phone, the lowest skilled agents not only lowered call times they were much more effective at making good enrollment recommendations.

Overall, the study results show that incorporating sophisticated AI into expert advising greatly improves outcomes on the dimensions that are explicitly included in the decision support, but experts continue to incorporate dimensions not included in the decision support that are valued by consumers.  This suggests that artificial intelligence of a decision support tool is a complement to the human intelligence that enrollment agents provide.  The fact that the agents seem to not blindly follow the decision support, shows that agents are able to integrate it in a fairly nuanced way with information that is excluded from the algorithm.  At the same time, this study also showed that there was one aspect of agent performance - an agent’s ability to evaluate a plans’ financial value - for which sophisticated AI-based decision support was a substitute for quality.  By leveraging decision support, the exchange was able to help their lower performing agents provide higher quality guidance to their customers.

Topics: Open Enrollment, Benefits Decision Support, Health Insurance, Consumer Directed Health, HR Tech

Sam Kina

Written by Sam Kina

Samuel Kina, Ph.D. is the SVP of Economics & Data Science at Picwell where he has led the company’s work in economic and predictive modeling since 2014. He has a wide range of experiences in health policy and economics and has taught courses in statistics, economics, and health policy. Prior to joining Picwell, Sam held positions at the Analysis Group, Congressional Budget Office, and the Alliance for Health Reform. Sam has a B.A. in Public Policy Analysis and Economics from Pomona College and a Ph.D. in Health Policy and Economics from Harvard University.