Journal of Allergy & Therapy

Allergic disorders are common in the developed world. Approximately one in three people in the UK and one in five people in the USA suffer from allergic disease [1,2]. The management of allergies is primarily dependent on the patient being aware of the triggers and then reducing exposure to the relevant allergens. This requires the patient to have a clear understanding about their condition and how to manage their symptoms effectively. Information given verbally during a consultation can be quickly forgotten and is frequently misinterpreted [3]. Hence, patient information leaflets (PILS) are used to augment the advice offered during the clinical encounter [4], empowering patients to become more involved in their treatment decisions and to take greater self-care [5]. Good information leaflets have been found to be helpful in reducing anxiety about patients’ illness or treatment [6], improving compliance or concordance with treatment [7], and contributing to improved illness outcomes.

The quality and usability of written materials for patients with atopic disorders are not always optimum [8,9]. Our previous review of 168 leaflets for people with asthma showed that 23% contained inaccurate or misleading information and 97% of leaflets were written above the recommended reading level for health literature [10]. In another study exploring the readability of PILs about hay fever, leaflets were also found to be of variable quality as well as having readability scores making the information inaccessible to a substantial proportion of the patient population [11]. Similar problems with readability were also identified when reviewing a series of allergy-related leaflets published by the American Academy of Allergy, Asthma and Immunology [12].

It is important that information provided to patients is written at a level which ensures that it can be understood by most of the population. Improved readability has been found to be associated with better recall and comprehension of information [13]. Readability formulae can be used to assess health literature, and these are often preferred to reader review because of their speed and ease of use. There are several formulae in common use, including Flesch-Kincaid, Flesch Reading Ease, SMOG, Fry, Fog, and Dale-Chall. Each of these formulae use different mathematical equations to estimate the reading grade level required for the reader to understand the written information [14]. The characteristics of the written text used to calculate reading grade level vary between formulae. Some use average number of syllables in each word, others average number of words in a sentence, proportion of common words used, proportion of words with three or more syllables, or proportion of words which are monosyllabic [15]. Whilst readability formulae are now widely used in evaluation of written materials, they are not a panacea. They have been criticised for failing to acknowledge that the writing style, presentation format together with the prior knowledge and interests of the reader (such as familiarity with the terms associated with their own medical conditions) also impact on the readability of the text. To address these limitations of readability scores it is advisable to use alongside them measures which can assess the different facets of presentation and quality.

Allergy UK (originally formed in 1991 as the British Allergy Foundation, and re-launched as Allergy UK in 2002) is a charitable organization established to support and empower people with allergy, intolerance or sensitivities. The organization provides education and advice, publishing a large range of leaflets on allergy, intolerance and sensitivities for the public (www.allergyuk.org/info-factsheets.aspx). Allergy UK has been very proactive, constantly expanding its range of leaflets and revising and amending existing ones. This study thus aimed to assess the readability, presentation and quality of the leaflets published by Allergy UK and also investigates the changes in readability over time as the organization has refined their leaflets. We recognize that many studies have assessed the readability of PILs relating to a range of specific diseases or conditions, but we are not aware of any previous studies that have conducted a longitudinal assessment [16].

Assessment of readability

We evaluated readability of the leaflets using two scores, the Simple Measure of Gobbledegook (SMOG) and the Flesch-Kincaid Formula.

SMOG

The SMOG score [17] is calculated by taking 10 consecutive sentences from the beginning, middle and end of the text. The numbers of polysyllabic words within the 30 sentences are counted and the square root of this total is calculated. A constant of three is then added to yield the reading grade level of the text. For this study SMOG scores were calculated using an online software program developed by McLaughlin [18]. A SMOG score predicts the reading grade level needed to understand 100% of the text: the lower the reading grade level, the easier the information is to understand. Completed primary school education is needed for reading material with a SMOG score of 3-8, secondary school education is needed for SMOG score of 9-12, and tertiary education is needed for SMOG score of 13 and above [11]. It is recommended that health information should be written with a SMOG score of ≤ 5 [19].

Flesch-Kincaid

The Flesch–Kincaid formula incorporates word and sentence length, and the average number of syllables. The formula utilized to calculate the score is (0.39 × Average sentence length)+(11.8 × Average syllables per word)-15.59 [20]. This formula result in a Flesch- Kincaid score which corresponds to a school grade in an American school, for example, a score of 8.2 would mean that a student in the 8th grade in an American school would understand 75% or more of the passage. By adding five to the score, the equivalent age in years of the reader is obtained, for example 13 years in this case. The lower the grade level, the easier the information is to understand. For patient health information, it is recommended that readability Flesch-Kincaid Grade scores should be between 4-6 [21]. The Flesch-Kincaid score for the Allergy UK leaflets was calculated using the readability statistics application in Microsoft Word 2010.

In January 2013, we reviewed all the Allergy UK leaflets available at that time. By using a methodology used in previous reviews of Allergy UK leaflets conducted in 2008, 2010 and 2012 we were able to investigate the changes in SMOG readability scores across five years for those 31 topics for which leaflets were available at each time point. As different researchers were involved in these different assessments we needed to confirm the inter and the intra reliability of the SMOG assessments. We tested for inter-rater reliability by randomly selecting 10 leaflets and asking three raters (HS, GC, and PP) to score them. To evaluate intra-rater reliability, one rater (GC), assessed the same leaflets on two separate occasions. Intraclass correlation coefficients (ICC), together with 95% confidence intervals, were calculated using SPSS (SPSS, Inc., Chicago, Ill) statistical software. ICC values of <0.4 were deemed as representing poor reliability, 0.4-0.75 as fair to good reliability, and >0.75 as excellent reliability [22].

Assessment of presentation and quality

The quality and presentation of the leaflets were evaluated based on the Clear Print Guidelines of the Royal National Institute of Blind People (RNIB) [23] and the patient information appraisal system developed by the British Medical Association (BMA) [24]. The RNIB guidelines were created to facilitate the design of documents that are easy to read, enabling essential information to be clearly conveyed to readers. The check list includes 15 items (e.g. “simple and clear typeface used”, “text is left aligned”) and each leaflet is scored against each criteria as “yes,” “no,” “don’t know” or “not applicable (n/a)”. The BMA patient information appraisal system includes 27 items worded as questions (e.g. “is the target readership clearly stated?”) Twenty six items focus on quality and content, and one item on leaflet presentation (Table 1). Each item is judged “yes” (scored as 1 point), “no” or “do not know” (scored as 0 points). The total score for each leaflet was calculated together with the proportion of leaflets meeting individual criteria.

Table 1: SMOG score values of Allergy UK leaflets from 2008, 2010, 2012 and 2013.

In January 2013, 108 leaflets for adults were available on the Allergy UK website. The leaflets addressed allergy-related conditions (e.g. anaphylaxis, asthma, urticaria and angioedema), specific allergens (e.g. latex, egg, and nickel), diagnostic procedures and treatment options (e.g. immunotherapy, complementary and alternative therapies). Other leaflets focused on specific situations when self-care may be challenging (e.g. holidays, food shopping). A list of all the leaflets can be found in Appendices A and B.

Readability analysis

The SMOG scores of the leaflets ranged from 11 to 18 (median 14, mean 13.9, SD 1.2) (Table 2). No leaflets met the recommended SMOG score for health literature of five or below, eleven leaflets (10%) were at a reading level comparable with successful completion of secondary education and the majority (90%) required reading skills equivalent to a university level education. Flesch-Kincaid scores ranged from 5 to 17 (median 11, mean 10.86, SD 2.1). Six leaflets met the recommended level for patient health information (i.e. between 4-6).

Table 2: Readability scores of allergy UK leaflets.

Assessment of presentation and quality

All leaflets met the RNIB ‘See it right’ clear print guidelines relating to clear design and ease of reading with the exception of font size, all were published using font size of 11.5 rather than the recommended type size of 12 point or ideally 14 point.

The leaflets scored on average 10 (median 10, range 7-15) out of a maximum possible score of 27 on the BMA appraisal checklist for quality, content and presentation (Figure 1). All the leaflets had professionals involved in their development and evaluation, and clearly stated the authors and agencies involved in production. The language and tone used in the leaflets was clear and appropriate, and information on the source organization was stated in all leaflets. The target readership was clearly stated in 88% of leaflets. As many of the leaflets followed a common design template there was considerable consistency in the design and quality features that were missing, for example none of the leaflets were specifically targeted at non-English speakers, or those with disabilities, specific cultures or religious beliefs. In the majority of leaflets (81%), the scope and aims were not clearly stated, and in only approximately one third (31%) were uncertainties and gaps in scientific knowledge addressed. In none was there a statement that patients or users had been involved in the development and evaluation of the leaflet. Less than half of leaflets (41%) provided the reader with suggestions for further reading or useful addresses, and only 9% of the leaflets listed other leaflets published by Allergy UK. Only 19% of the leaflets contained a key points section (Table 3).

Table 3: Percentage of leaflets meeting BMA appraisal criteria (n=108).

Figure 1: Distribution of BMA appraisal criteria scores for allergy UK leaflets (n=108).

Longitudinal analysis

We were confident about comparing readability data across time as we demonstrated excellent intra- and inter-rater reliability for the SMOG score assessment (ICC=0.997 95% CI (0.989-0.999) and 0.996 (0.989-0.999) respectively).

There were 31 leaflets eligible for inclusion in the comparative analysis. A full list of these leaflets and their SMOG scores in 2008, 2010, 2012 and 2013 can be found in Table 2. The average SMOG score varied slightly between years (range 13.3 to 14.3), with the highest average score in 2010. The process of leaflet revision between 2008 and 2013 resulted 1% change in average readability score (mean readability score 13.37 vs 13.51; p=0.58). However, when considering the readability changes within individual leaflets there was a predominantly adverse trend, with six leaflets increasing their SMOG readability score by more than 10% and only three decreasing by less than 10%.

Summary of key findings

Our assessment of the readability and presentation of 108 leaflets published by Allergy UK in 2013 found that the leaflets were generally well presented according to the RNIB and BMA appraisal guidelines. Further modifications, such as increase in font size and the addition of a key point section, would improve the accessibility of information. Whilst presentation characteristics were moderately good, the readability levels of these leaflets was very high, often requiring a reading ability comparable with the successful completion of secondary education and in some cases a university degree. All the leaflets except six were still above the recommended reading grade level for health information and there had been no improvement in readability over five years of leaflet refinement and modification.

Choosing between two readability scores

In this study, the mean readability scores estimated using Flesch- Kincaid were three grades lower than the SMOG score. Such variations in scores complicate the interpretation of reading grade data. In the absence of standardized ways for applying and interpreting readability formulas, some experts suggest choosing the highest calculated reading grade level estimate, others suggest calculating an average, whilst others recommend prevention of the dilemma by confining assessment to a single readability formula [14]. The Flesch-Kincaid formula expects lower comprehension compared to SMOG (75% vs. 100% comprehension). For healthcare information it may be argued that close to 100% comprehension is needed and therefore the SMOG formula score is preferable to Flesch-Kincaid [14].

Why does readability matter?

Being attentive to readability of PILS is important given the prevalence of functional illiteracy in westernised countries: for example the National Literacy Trust estimates that in the UK one in six people have a reading ability less that that expected of an eleven year old, and therefore they struggle with literacy [25]. Our longitudinal assessment of readability of Allergy UK leaflets over five years shows that despite the strong commitment of the organization to improve their patient information there is a significant and sustained mismatch between the published material and the range of reading skills in the general public. Given the plethora of materials that exist on the writing and testing of user-friendly materials, one might wonder why organizations continue to publish leaflets that require relatively high reading skills that may not exist in a large proportion of their target population. A qualitative study in Israel explored the reasons behind this and highlighted some unexpected complexities of PIL development, including organizational politics and conflicting agendas [26]. The researchers found that PILS often emerged from an iterative and dynamic process that required resolution of opposing demands (e.g. existing materials versus suggestions from experts). They cited situations where experts insist on including additional technical detail in order to improve medical accuracy, to augment the perceived sophistication of the PIL or perhaps to improve medicolegal defensibility. It might be helpful for Allergy UK to undertake a broad ecological review of their process for creating and refining PILS and, having identified the intellectual, technical, organizational and political factors in play, to establish design procedures to prevent readability gaps.

The role of patients in PIL development

None of the Allergy UK leaflets were explicit about whether patients had been involved in the development and evaluation process of these leaflets. The supply of useful information to patients is dependent on establishing what patients already know and what they want to know [26]. Clinicians often make erroneous judgements about what patients do or do not want to know or assume they already are in possession of the knowledge [27], thus leaving patients with an unfulfilled desire for knowledge. Working directly with patients to inform content may also generate patient anecdotes and case studies: this is beneficial as it is recognized that knowledge of others’ experiences can facilitate compliance [26].

Strengths and weaknesses

Strength of this study is that it included two unusual features; a unique longitudinal assessment of readability scores across five years, and a formal assessment of the repeatability of SMOG score measurement. However, this study does have some potential weaknesses. Due to time constraints we did not assess the repeatability of all the measures used. As the BMA Appraisal guidelines are relatively subjective there is the potential for observer bias, but this risk could have been minimised by the use of more than one reviewer. A further methodological refinement would have been to re-evaluate the readability of the leaflets omitting common polysyllabic words associated with the illness and that would be very familiar to the reader, for example words such as allergy, anaphylaxis or urticaria. Removal from the text before scoring of complex, but frequently used, medical terminology has been associated with significant improvements in readability scores [28].

This study demonstrates that even when an organization has an active program of PIL revision readability does not automatically improve. A global review of the PIL writing procedure is needed to identify the drivers for change and resolve these issues that conflict with a need to increase readability. Other tactics include patient involvement in design, pre-publication screening of readability and post-production evaluation beyond just monitoring the number of leaflets requested. No PIL will ever meet every individual’s needs but a better understanding of literacy in the target population can help maximize the benefit derived from the resource-intensive process of leaflet preparation.