10:05 - 10:25 amSaturday, September 28
LK 120
Use of sentiment analysis for capturing patient experience from free-text comments posted online
LK 120
Use of sentiment analysis for capturing patient experience from free-text comments posted online
Imperial College of London
Background There are large amounts of unstructured, free-text information about the quality of healthcare available on the Internet in blogs, social networks and on physician rating websites that are... Read more

Description

Background

There are large amounts of unstructured, free-text information about the quality of healthcare available on the Internet in blogs, social networks and on physician rating websites that are not captured in a systematic way. In other industries, real-time natural language processing, such as sentiment analysis, of large data sets has provided a useful analytical tool for finding patterns and understanding data. Application of these techniques to healthcare may allow us to understand and use this information more effectively to improve the quality of care.

The large number of free text comments about patient care on the English NHS’s main website allows an opportunity to examine this hypothesis. Patient’s comments are matched with their own quantitative ratings of the service – allowing us to measure the accuracy of natural language processing tools against the patient’s own assessment. Simultaneously, the NHS has a developed programme of patient experience measurement via a national survey of hospital inpatients. Using these data sources, we have a natural opportunity to compare our sentiment analysis of comments to traditional patient surveys at an organisational level.

Methods

We used sentiment analysis techniques to categorise online free-text comments by patients as either positive or negative descriptions of their healthcare. We used a bag of words technique, with prior polarity and information gain, and 2 word N-grams. We tried to automatically predict whether a patient would recommend a hospital, whether the hospital was clean, and whether they were treated with dignity from their free-text description, compared to the patient’s own quantitative rating of their care. We applied machine learning techniques to all 6412 online comments about hospitals on the English National Health Service website from 2010 using WEKA data-mining software, using all comments from 2009 and 2011 as a learning set. We also compared the results obtained from sentiment analysis with the paper-based national inpatient survey results at the hospital level from 2010 using Spearman’s rank correlation for all (161) acute adult hospital trusts in England.

Results

There was 81%, 83% and 89% agreement between quantitative ratings of care and those derived from free text comments using sentiment analysis for cleanliness, being treated with dignity and overall recommendation of hospital respectively (Kappa scores: 0.40 – 0.75, p<0.001 for all), which compares favourably to many other machine learning classification tasks using natural language in other settings. We observed significant, moderate associations (Spearman rho 0.37- 0.5, P<0.001 for all) between our machine learning predictions and responses to the large patient survey for the three categories of patient experience examined.

Conclusions

The prediction accuracy that we have achieved using this machine learning process suggests we are able to predict, from free-text, an accurate assessment of patients’ opinion about different performance aspects of a hospital. We also see that these machine-learning predictions are associated with results of more conventional surveys. This work is on-going and an iterative process but suggests that it may be possible to monitor the online ‘cloud of patient experience’ in real-time and by doing so efficiently analyse that data and turn it into meaningful information that can be use by policy makers, clinicians, and administrators to learn from and improve the patient experience.

 

Contact Us

We're not around right now. But you can send us an email and we'll get back to you, asap.

Not readable? Change text. captcha txt

Start typing and press Enter to search