Getting to Why with NCQA’s Qualitative Analysis

By Nancy Ross Bell, RN

Estimated time to read:  5 minutes

Have you ever presented a Quantitative & Qualitative (Q&Q) analysis at a QI committee but saw no improvement from your interventions? Did you “fall into the trap” of working on the wrong problem because the root cause was not found and non-specific or weak interventions were executed?   

MHR has found that many organizations fail to perform qualitative analysis effectively in the way that NCQA describes in the Glossary. When this happens, it’s likely that your goal is not met, resources are wasted, your team is disillusioned, and, importantly, member experience is not optimal. That could lead to what we call “box checking”: new data is entered in the previous year’s analysis, the dates are changed, and everything else remains the same. It is obvious to the consultant and the surveyor that this is occurring and leads to a loss of points. The work was not done.

In MHR’s training on Q&Q Analysis, our consultants lead a discussion on the measurement and Q&Q process. Each time the staff goes through this process, the more independent and confident they become.  All analysis standards are high-point standards that reflect the extra work a robust analysis entails.

In our May 2023 blog, NCQA Q&Q Analysis That Brings Results, we highlighted a 10-step process to follow and common pitfalls we have seen with Q&Q Analysis. In this blog, we drill down on qualitative analysis and provide a fictional example of how you can use the 5 Whys method to help get to the root cause of a problem, keeping in mind that this is only one method and does not represent all the required steps for  NCQA’s Q&Q Analysis.

Why does Qualitative Analysis Matter?

Requirements for Q&Q analysis are found throughout the NCQA Accreditation standards to measure and analyze the performance or effectiveness of interventions. This is the heart and soul of a true quality improvement continuum. Ultimately, you can lose precious points during your NCQA survey when a qualitative analysis is not performed or is incorrectly performed!

While analysis most often includes both quantitative and qualitative analysis, NCQA does not always require both. Read the Explanation for each factor of the NCQA standard.

Who is Responsible for Conducting a Qualitative Analysis?

While organizations differ, quantitative analysis commonly involves data analysts, statisticians, or quality analysts who gather data from multiple sources and produce the results or statistics. Qualitative analysis requires key stakeholders, decision-makers, and subject matter experts (SMEs) closest to the problem to analyze what went right and what went wrong.

Depending upon the standard or project, Leads and Management from multiple areas and departments must attend, not only Leads from a Quality or Accreditation Department who may be tasked with writing the report.  When completing a qualitative analysis, participant titles and departments are required to be listed within the analysis.  Departments only are not acceptable.

When is a Comprehensive Qualitative Analysis Conducted?

When the health plan has all member and practitioner data, the next step is asking what do you see?. Three cycles of analysis are typically trended, including a baseline measurement of data followed by two remeasurement periods, usually conducted annually. After each measurement, a drill-down of the data by the persons closest to the problem moves the organization to understand the actions or barriers that led to the outcome.

Draw a Conclusion

Concluding a qualitative analysis, a conclusion must be drawn and documented. This step is a synopsis of the problem, opportunities identified, actions implemented, key results, a description of what intervention(s) impacted the movement toward the goal, and the titles and departments of all persons involved in the qualitative analyses at each measurement cycle.

KEY POINTS FOR QUALITATIVE ANALYSIS:

• Be clear on NCQA’s requirements for each Factor.
• Be on time: Many requirements are annual, which means you cannot be earlier than 12 months nor after 14 months to be counted. *
• Ensure everyone agrees on the problem statement.
• Examine reasons for or cause of results, including deficiencies or processes that caused barriers to improvement or failure.
• Using standard methods of root cause analysis (RCA) such as the 5 Whys, Fishbone Diagrams, or Brainstorming are helpful.
• Involve persons closest to the problem, key stakeholders, and those who are qualified to make decisions. Track titles and departments of all who participate.
• Encourage insight from multi-disciplinary points of view.
• Prioritize opportunities.
• Create interventions that are meaningful and measurable.
• Understand actions leading to goals being met and not
• Decide if more drill-down data, such as member survey data, specific CAHPS questions, or practitioner-level information, would be helpful.
• Detect if member complaints, appeals, or utilization data are consistent with the trend towards or away from the goal.
• Calculate percent or percentage point changes. Statistical significance may be helpful (Assess if NCQA requires significance testing.)
• Draw insightful conclusions regarding the problem being solved and the effectiveness of interventions.

*NCQA defines annual as a 12-month period plus a 2-month grace period (12–14 months). (NCQA Glossary)

Looking for Some Guidance on Q&Q Analysis?

If you are new to Q&Q analysis or maybe need a refresher, MHR is your guide to getting you independent in this challenging skill, ultimately leading to improved quality for your members. 

Don’t wait!  Don’t miss your look-back period!

Call us now or speak with your MHR Consultant.

Let’s see how it’s done.

Example of Qualitative Analysis using NET 1 D

Network adequacy is vital to members’ experience and quality of healthcare. A comprehensive report at the product level is required for Health Plan Accreditation-NET 3 A-C.  NET 3 uses data and reports from network availability (NET 1 B-D), network access (NET 2 A-C), and related complaints and appeals (ME 7 C-F) to determine if the network is adequate or sufficient for their members and if actions must be taken to improve.   

In this fictional example, we use NET 1 D for behavioral health (BH) practitioners. Factors 2 and 3 require measurable standards, and factor 4 requires an analysis of performance. Further, under the explanation for factor 4, Q&Q analysis is described for the initial measurement and remeasurement.

The health plan has identified:

• psychiatrists as one type of high-volume BH practitioner
• a standard and goal of 1 psychiatrist for every 2,000 members
• a goal of 95% of the time that a psychiatrist is available to members within 15 minutes driving time of a member’s home

The health plan’s Provider Network department learned that one of the health plan’s large in-network BH practitioner groups that hold clinics at an in-network urban hospital has decided to move to a suburban setting to provide more space to see patients. This move has increased driving time for many members, resulting in complaints to the health plan and requests for out-of-network services from other closer practitioners.

The health plan measured its performance against goals.

Baseline measurement results:

1. 1 Psychiatrist to 2,000 members Goal was met. No action is needed.
2. 1 Psychiatrist within 15 minutes driving time of member’s home was met only 90% of the time, which is 5 percentage points from the goal. The goal was NOT met. Action is needed.

Because the goal was not met, an RCA team from the health plan met consisting of the Medical Director of BH, the VP of Provider Network/Relations, the Network Manager, the Manager of Case Management, the Manager of Utilization Management, the Supervisor of Member Experience, and the Manager of Quality Improvement and Accreditation.  

The focused problem was stated as follows:

The geographic distribution of network psychiatrists does not meet the goal of one psychiatrist within a 15-minute drive time for members living in an urban setting, resulting in member complaints and requests for out-of-network services.   

The team used the 5 Whys Method of RCA.

1-Why did the 15-minute drive time increase for its urban members when in the prior year, this goal had been met?  

A large BH practitioner group recently left the clinic setting at the hospital and moved to a suburban setting, increasing member drive time and difficulty with public transportation over the last three months.

2-Why did the BH group leave?

The BH group left because of the need for more office space and more available daytime hours than what the hospital offered.

3-Why was there no additional space at the hospital?

The hospital planned to use the BH offices for specialty procedures instead.

4-Why was the hospital prioritizing specialty procedures over BH office space?

New contracts between specialists and the health plan who practice at the in-network urban hospital will increase revenue for the hospital more than  BH office visits.  

5-Why did the health plan pay more for specialty procedures over the increasing needed BH care?

Contract negotiations were done at different times by different individuals from the health plan, hospital, and practitioner groups.

After not meeting the goal of a 15-minute driving time 95% of the time, the team convened and brainstormed opportunities.

• Additional office space during the daytime was not available at the hospital, but later afternoon and evening hours may be possible
• Incenting the BH practitioners with a higher reimbursement rate for afternoon and evening hours could maintain the drive-time goal
• Bringing customer service staff in the loop with changes would help improve communications and improve member experience
• Offering patients more options for alternative appointment times can maintain urban clinic visits and keep driving time within 15 minutes
• Pursuing contracts for BH telehealth services

The team then prioritized the following opportunities as having the greatest chance of improving availability.

1. All Managers and Supervisor will track related issues and escalate resolution (immediate)
2. The Supervisor of Member Experience will send an email-blast to all customer service representatives, informing them of changes. (Target 2 weeks)
3. The Provider Relations VP and Network Manager will work with the hospital and BH practitioners in securing additional exam rooms during the later afternoon and evening hours (Target 3 months)
4. The Provider Relations VP and Medical Director will increase reimbursement rates for the BH practitioners agreeing to later afternoon and evening hours (Target 3 months)
5. The Director of Provider Network will update the provider directory to coincide with office hour changes. (Target 2 months)
6. The Provider Relations VP and Network Manager will pursue contracts for BH telehealth services and update the Quality Improvement Committee next quarter to “go live” by the end of the year.

The plan was implemented according to the target dates, and the following year, performance results improved by five percentage points.

Remeasurement 1 results:

1. 1 Psychiatrist to 2,000 members.  The goal was met. No action is needed.
2. 1 Psychiatrist within 15 minutes driving time of member’s home was met at 95% of the time, a rise of 5 percentage points from the baseline of 90%. The goal was met. No further action is needed.

If the goal had not been met, the team would have proceeded with a second set of 5 Whys to determine RCA and what further actions were needed. 

The team convened and brainstormed on the outcomes of the initial interventions and planned for the second annual Remeasurement to ensure goals were being met.

The positive drivers were:

• Increasing office space during the later afternoon and evening hours allowed members to maintain a 15-minute drive time at the location to which they were accustomed and avoid the longer drive to the new suburban office.
• Additional reimbursement for the BH practitioners allowed them to see patients at these new times, which improved availability since patients did not have to drive to the suburban setting, which was further away.
• Keeping customer service engaged kept members informed when they called the health plan and enabled tracking of issues.
• Members without transportation and unable to access public transportation are utilizing BH telehealth services.

The negative drivers were:

• Personal communications to members were not done, resulting in dissatisfaction due to increased drive time.

Health Plan’s Conclusion:

Improving the geographic distribution of high-volume psychiatrists was identified as a problem through annual performance measurement against goals. The goal of a 15-minute drive time was not met when, in the prior year, it had been. A performance improvement team, including the Medical Director of BH, the VP of Provider Relations, the Network Manager, the Supervisor of Member Experience, and the Managers of Case Management, Utilization, and  Quality Improvement & Accreditation, convened to uncover barriers, brainstormed and prioritized opportunities and decided on interventions.

A system problem was detected in that the practitioners needed additional office space, which the hospital could not provide, resulting in their move from an urban setting to the suburbs. The health plan needed to maintain a 15-minute drive time availability.

Strong interventions included the VP of Provider Relations collaborating with the practitioner group and hospital, resulting in more available office space in the afternoon and evening hours with incentives for higher reimbursement for BH practitioners taking these office hours. Additionally, customer service was more knowledgeable of the network changes and could assist members promptly.

Call to Action:     

• Already an MHR client? Ask your MHR Consultant to schedule training on Q&Q analysis before your look-back period begins. This two-hour training incorporates hands-on exercises, which give you beneficial practice.
• Looking to improve your Q&Q Analyses? Ask us about our template, which follows NCQA’s requirements.

Managedhealthcareresources.com or email Susan K. Moore at

 Kimberly Carpenter Petit, an Independent Consultant with MHR, has provided expert insight to this blog on Qualitative Analysis. Read more about Kimberly and our other consultants on our website under About Our Consultants.