Objective: Olfactory dysfunction (OD) impacts nearly 1-in-4 adults, many of whom turn to online resources-including social media-for expected prognoses and treatment recommendations. The primary objective of this study was to assess the quality and understandability of OD-related content on TikTok, a social media platform with 1-billion monthly users.
Methods: TikTok videos were searched using 30 OD-related terms. Duplicate, non-English, or non-smell loss related videos were excluded. Video content creators were categorized as otolaryngologist (MD), non-otolaryngologist physician (MD/DO), non-physician healthcare provider, non-healthcare licensed professional, and general users. User engagement was assessed by number of views, likes, comments, shares, duration, and days since upload. Educational quality was assessed using Global Quality Score (GQS) (range:1-5), modified DISCERN score (range:5-25), Patient Education Materials Assessment Tool-Understandability and Actionability (PEMAT-A/U) (range:1-100%), and JAMA Benchmark Criteria (range:0-4).
Results: Of 447 videos, 363 were included, amassing 131,509,151 views; 15,477,339 likes; 267,804 comments; and 575,760 shares. Most frequently posted content were user experiences (47%), humor/entertainment (23%), and education (20%). Only 1 (0.28%) was posted by an otolaryngologist [vs. 320 (80%) general users]. Overall, educational quality was poor: GQS (2.28 ± 0.73); modified DISCERN (8.67 ± 2.42); PEMAT-A/U (60.77 ± 33.95, 80.12 ± 19.53); JAMA Benchmark Criteria (0.32 ± 0.52). Videos with higher GQS and DISCERN scores had less likes (p < 0.05). Videos created by general users had lower GQS and DISCERN scores (vs. professionals, p < 0.001).
Conclusions: TikTok is used to seek and share information about OD and management strategies. Most educational content is low quality and not created by health professionals. TikTok represents an expansive growing social media platform with opportunities to improve high-quality health information resources to the community.
TikTok, Olfactory dysfunction, Patient education, Social media, Smell loss, Parosmia, Anosmia
With the advent of social media in recent decades, information sharing has become more accessible to individuals of all demographics and age groups, with over 70% of the public participating in at least one social media platform [1-3]. Social media represents a potential avenue for clinicians in the realms of patient education, public health awareness, and health policy promotion, but remains largely underutilized. TikTok has been recognized as one of the fastest growing social media platforms today, with over 1 billion users worldwide and novel engagement algorithms [2,4,5]. One of TikTok’s signature features is its short 60-seconds-or-less videos which provide a “rapid-fire” dissemination of information, leading to higher levels of engagement especially amongst younger users [2]. However, this swift distribution of information without editorial oversight or professional verification has led to the spread of misinformation during the COVID-19 pandemic, possibly hindering public health efforts and causing increased anxiety amongst patients [6,7].
At the height of the COVID-19 pandemic, mandated isolation and quarantine periods prompted a rapid reduction in in-person healthcare visits, and consequentially direct dissemination of health information from providers. As a result, many individuals globally resorted to the Internet not only to stay educated on the common symptoms of COVID-19 but also to determine if they had contracted the virus themselves [8]. Amongst the many symptoms and potential clinical findings of COVID-19, smell loss emerged as a unique search term shown to be associated with spikes in COVID-19 cases [8]. Olfactory dysfunction (OD), referring to a reduced or distorted sense of smell, has long impacted the lives of many individuals with an estimated 12.4% of Americans over the age of 40 reporting some degree of OD [9]. However, with the reported prevalence of OD in COVID-19 exceeding 80% in mild cases, many individuals began sharing their experience with OD on platforms such as TikTok regardless of whether it was COVID-related [10].
Despite the proposed benefit of TikTok in spreading helpful public health information, promoting patient engagement, and formation of patient support groups, the platform is also subject to the inherent drawbacks of social media [1,2,4,5,11,12]. With healthcare professionals being severely underrepresented amongst TikTok users [12], the risk of rampant spread of health-related misinformation can lead to confusion or potential harming of patients [1,2,4,5,11,12]. A myriad of proposed treatments for OD exists on the Internet with only a select few to be shown through randomized-control trials (RCT) to consistently benefit persistent OD in patients [13]. TikTok’s short video format does not lend itself well for users to cite reputable sources, disclose financial conflicts of interest, or verify their credentials [2]. Consequently, users will have difficulty distinguishing true evidence-based treatments from anecdotally based home remedies. For TikTok to be reliably utilized as a source of information clinicians can safely recommend to patients, the quality and accuracy of its content must be assessed.
The primary objective of this study was to critically assess the quality and understandability of OD related content on TikTok. Our secondary objective was to investigate predictors of the most popular content published on this social media platform. Through these analyses, we aimed to identify gaps in education and healthcare provider engagement surrounding OD.
Thirty OD-related hashtags were established and queried on TikTok between July 1, 2022 to July 13, 2022 (Table 1). A newly created account was used to avoid previous searches from influencing the results generated by TikTok’s video recommendation algorithm. The top 20 videos from each hashtag were included in the final analysis. Duplicate videos across all hashtags were removed from review. Non-English or non-OD related videos were excluded from final analysis.
Table 1: Overall Included Hashtags and Video Metrics. View Table 1
The number of views, likes, comments, shares, duration [seconds (sec)], and average days since upload were abstracted from each video by two reviewers (JL, RW). Each video was then categorized by video account-owner (i.e., content creator) and video content characteristics. Content creators were categorized as follows: board-certified otolaryngologist (MD/DO), non-otolaryngology doctor of allopathic or osteopathic medicine (MD/DO), non-physician healthcare provider (i.e., dentists, doctors of pharmacology, doctors of physical therapy, registered nurses, nurse practitioners, and physician assistants), licensed professional non-healthcare providers (i.e., acupuncturists), patients; and other (i.e., anonymous accounts). The content creator type was confirmed by utilizing a combination of search-engine queries and linked social-media accounts. Patient and other content creator categories were further classified as “non-professional/patient” creators, and all other creators were grouped as “professional.” Video content was characterized as follows: patient experience, educational, live procedure/surgery, advertisement/product promotion, self-promotional, humor/entertainment, and career. Video characteristics were not mutually exclusive. For example, a video could be categorized as humor/entertainment but also educational. Any video whose content characteristics were unclear initially were jointly categorized by two authors (JSL, RW). Discrepancies were resolved by consensus or via discussion with the senior author (LJM).
A sub-analysis of educational videos was conducted using five previously established standardized tools to analyze online video content: Global Quality Scale (GQS); modified DISCERN; JAMA Benchmark Criteria; Patient Education Materials Assessment Tool-Understandability and Actionability (PEMAT-A/U); and Audiovisual Quality Scale (AVQ). The GQS and the modified DISCERN scale are scoring systems that have been previously shown to be a valid and reliable assessment of video educational quality [14,15]. The GQS is scored on a five-point scale, with 1 being the lowest educational quality and 5 being the highest. The modified DISCERN tool ranks videos on a scale of 1 to 5 in five subcategories (clarity of aims, source reliability, balanced/unbiased information, additional patient resources, and areas of uncertainty), which sum up to a total scale between 5-25 points. Higher total DISCERN scores equate to higher educational value and quality for patients. The JAMA Benchmark Criteria is a scale used to consists of 4 subcategories: Authorship (authors and their credentials/affiliations), attribution (clear references and sources provided), currency (date of initial posting and updates), and disclosure (ownership of video, sponsorship, commercial funding, etc.) [16]. Each subcategory is graded 0-1 and sum to a total range of 0-4 points, with a higher score indicating higher reliability and credibility. The PEMAT-A/U is a scale created by the Agency for Healthcare Research and Quality (AHRQ) (a subdivision of the Department of Health and Human Services) that systematically evaluates the understandability and actionability of online educational materials [17,18]. According to the AHRQ, patient education materials are understandable and actionable when consumers of diverse backgrounds and varying levels of health literacy can process and explain key messages and can identify what they can do based on the information presented, respectively [17].
Two authors (JSL, RW) independently evaluated and scored all educational videos and a composite score per metric was calculated. Interrater reliability was assessed using the intraclass correlation coefficient (ICC).
SPSS, version 28.0.1.0, was employed for statistical analysis. Parametric assumptions testing showed violations of normality. Mean and median with interquartile ranges were used to illustrate data distribution. Nonparametric tests were used in our analysis: Kruskal-Wallis H test was utilized to analyze video metrics (views, likes, comments, and shares) as stratified by video characteristics, while Mann-Whitney U tests were used to analyze video quality and metrics when stratifying by content creator type (professional vs. non-professional). A simple linear regression model was applied to determine the association of video quality scores with video metrics. An a-priori p-value of ≤ 0.05 was used to determine statistical significance.
Of the 477 videos identified from the initial query, 46 duplicates were excluded. Of the remaining 431 videos, 68 were excluded and 363 met inclusion criteria and were included in the final analysis (Figure 1 and Table 1). The included videos amassed a total of 131,509,151 views, 15,477,339 likes, 267,804 comments, and 575,760 shares. The average metrics per video were as follows [mean (SD)]: views: 362,284 (SD 1,340,618); likes: 42,637 (SD 171,610); comments: 737 (SD 2,979); shares: 1,586 (SD 7,952); duration (sec): 46 (SD 45), and days since upload: 282 (SD 159). The most popular hashtags were “parosmia” with a total of 51,234,300 views; “no smell and taste” with 16,555,883 views; “anosmia” with 11,330,200 views; “loss of smell” with 10,790,182 views; and “loss of smell cure” with 9,396,672 views. Additionally, of the 363 videos, 251 (69.1%) were COVID-19 related.
Figure 1: Identification of TikTok videos via search terms. Flow diagram of video retrieval method, inclusion, and analysis.
View Figure 1
Of the 363 OD-related videos included, most content was created by non-professionals/patients (Figure 2). Overall, patients posted content most frequently (n = 320, 88.2%), followed by other/not-specified creators (n = 16, 4.4%) Among professionals, licensed professional non-healthcare providers created the most content (n = 13, 3.6%), followed by non-physician healthcare providers (n = 10, 2.8%) and non-otolaryngology physicians (MD/DO) (n = 3, 0.8%). Only 1 (0.3%) video was created by a board-certified otolaryngologist. The mean number of views for patient videos were the highest at 388,004 (standard deviation (SD), 1,418,038), whereas non-otolaryngology physicians received the lowest views with only 20,966 (SD 16,448) views per video. Videos made by non-professionals/patients received significantly more views and shares versus videos made by professionals (p = 0.012 and p < 0.01, respectively). The mean duration of professional videos (78 sec; SD 47) was significantly longer than the length of patient videos (45 sec; SD 45) (p < 0.001). The average days since upload for professional providers was 301 days (SD 166), and was similar to the average days since upload for patients was 445 (SD 159) (p = 0.29). The number of views, likes, comments, and shares of professional providers and non-professional/patients are shown in Figure 2.
Figure 2: Overall video metrics by content creator type. Professional provider: board-certified otolaryngologist MD/DO (n = 1), non-otolaryngology MD/DO (n = 3), non-physician healthcare provider (i.e dentists, PharmD, DPT, RN, NP, PA) (n = 10), licensed professional non-healthcare providers (i.e acupuncturists) (n = 13). Non-professional/patient: patients (n = 320) and other content creator (n = 16). p = p-value, * = significant (a = 0.05).
View Figure 2
The 363 videos included in the analysis yielded a total of 482 content types: 225 (46.7%) videos were categorized as patient experience; 110 (22.8%) as humor/entertainment; 99 (20.5%) as educational; 37 (7.7%) as ad/product promotion; 8 (1.7%) as self-promotional; and 3 (0.6%) as other (Figure 3). The mean number of views for patient experience videos was 483,697 (SD 1,556,635). Educational videos received a slightly lower number of views per video at 288,009 (SD 1,173,105). Humorous videos received the least number of views, with only 34,253 views (SD 114,430) per video. This trend continued for likes, comments, and shares, with patient experience videos receiving the most, educational videos receiving the second most, and humorous videos receiving the least of all categories. No videos received a categorization of live procedure/surgery or career. The mean number of views, likes, comments, and shares were significantly different for all content categories (p < 0.001, p = 0.005, p < 0.001, and p < 0.001, respectively). The number of views, likes, comments, and shares for all video characteristics are shown in Figure 3. The average duration (sec) for patient experience, education, ad/product promotion, self-promotional, and humorous videos was 52s (SD 50), 61 (SD 38), 50 (SD 21), 48 (SD 22), and 26 (SD 27), respectively, and this was statistically significant (p < 0.001). The average days since upload for patient experience, education, ad/product promotion, self-promotional, and humorous videos was 290 (SD 159), 288 (SD 141), 261 (SD 114), 300 (SD 159), and 293 (SD 179), respectively, and not statistically significant (p = 0.709).
Figure 3: Overall video metrics by video characteristic. Patient experience (n = 225), Education (n = 99), live procedure/surgery (n = 0), ad/product promotion (n = 37), self-promotional (n = 8), humorous (n = 110), career (n = 0). p = p-value, * = significant (a = 0.05).
View Figure 3
A subgroup analysis was performed for 99 educational videos to evaluate their quality, understandability, and actionability. Educational videos amassed a total of 52,513,207 views; 4,834,625 likes; 72,854 comments; and 281,303 shares. The average duration for these videos was 61 seconds (SD 37.5), and the average days since upload was 288 days (SD 140.8). Patients contributed 70 (70.1%) videos compared to 3 videos (3%) from non-otolaryngology physicians, and 1 (1%) by a board-certified otolaryngologist.
Overall, video educational quality and reliability were poor, whereas actionability was adequate, and understandability was excellent. Audiovisual quality was poor, with mean AVQ measuring 1.84 (SD 0.50). Video educational quality (as measured by modified DISCERN and GQS) was poor, with mean GQS measuring 2.28 (SD 0.73) and modified DISCERN measuring 8.67 (SD 2.42). Video credibility and reliability was also poor, with the average JAMA Benchmark Score measuring 0.32 (SD 0.52). Video actionability was adequate, with PEMAT-A measuring 60.77% (SD 3.95), whereas understandability was excellent, with PEMAT-U measuring 80.12% (SD 19.53) (Table 2). ICC was excellent with ICC of all educational measures > 0.80 [19].
Table 2: Mean scores, standard deviations, and score ranges for educational videos. View Table 2
Videos created by patients had lower AVQ, GQS, DISCERN, PEMAT-A, and JAMA scores than videos created by professionals (p < 0.001) (Table 3). Additionally, higher GQS and DISCERN scores were significantly correlated with lower number of likes (p < 0.05) (Table 4). There were no significant associations between educational scores and number of views, shares or comments.
Table 3: Association of educational and video quality scores by content creator type. Professional provider: board-certified otolaryngologist MD/DO (n = 1), non-otolaryngology MD/DO (n = 3), non-physician healthcare provider (i.e dentists, PharmD, DPT, RN, NP, PA) (n = 10), licensed professional non-healthcare providers (i.e acupuncturists) (n = 13). Non-professional/patient: patients (n = 320) and other content creator (n = 16). View Table 3
Table 4: Association of educational and video quality scores with video metrics for educational videos (n = 99). View Table 4
The rapidly expanding social media platform, TikTok, represents an extraordinary opportunity for patient outreach and education. Although there have been many prior studies in a variety of medical specialties investigating other popular social media platforms like YouTube and Facebook, few have analyzed TikTok and none have done so in otolaryngology [20-23]. Given that about 72% of Americans use the Internet for health information, alongside the increasing symptom burden and awareness of OD as a consequence of COVID-19, the content and quality of online sources available to prospective otolaryngology patients must be understood [24]. To the authors’ best knowledge, this is the first study to quantify and describe OD-related otolaryngology content on TikTok.
Approximately 131 million views and 15.5 million likes garnered from just 363 OD-related videos reflects the popularity of TikTok and demonstrates a large interest in the community for smell-disorders related content. Young adult patients are much more likely to use and believe healthcare-related information found on social media, with one study showing 90% of 18-24 year-old healthcare consumers utilizing social media as an information resource [25]. With evidence showing that anosmia secondary to COVID-19 is more common in younger individuals, this finding is opportune yet worrisome [26]. Healthcare misinformation has become a substantial issue on social media platforms, with some studies finding misinformation rates ranging anywhere from 40% to 87% [27]. Though our investigation did not aim to analyze and label information as misinformation or false information, we nevertheless found low rates of source credibility and reliability among educational TikToks with an average modified DISCERN score of 8.67 (on a scale of 5-25) and JAMA Benchmark Criteria score of 0.32 (on a scale of 0-4). Additionally, we found that while 70% of OD-related educational videos were posted by patients, only 4% were posted by either a board-certified otolaryngologist or non-otolaryngology MD or DO. While patients in these TikToks offered relatively benign treatment strategies or educational tips for smell loss-such as eating “burned oranges,” or “flicking” oneself in the forehead-these anecdote-based treatments may cause patients undue distress when self-learned remedies fail to work, or worse yet delay diagnosis and treatment of etiologies other than post-viral mediated OD.
Otolaryngologists must also consider the consequences of potentially more dangerous non-evidence-based treatments presented as acceptable management strategies on social media. One study of epistaxis-related YouTube videos in 2016 found that some alternative treatments were to rub black pepper onto the septum, filling the nose with tampons, and smelling a raw onion during an attack-none of which were supported by a Medline literature search [28]. There is still a paucity of information regarding the dangers of misinformation on social media within the otolaryngology field, and more research is needed.
Overall video educational quality was poor, with GQS measuring 2.28 on a scale of 0-5, modified DISCERN measuring 8.67 on a scale of 5-25. Our study showed that patient-created educational videos scored significantly lower than professional-created educational videos when using the GQS, DISCERN, and JAMA scales. A previous study evaluating TikTok had similar findings, with nonphysician videos scoring lower on the DISCERN scale [29]. However, patients were the dominant creators of OD-related videos on TikTok, representing 88% of all OD-related videos posted. Board-certified otolaryngologists, non-otolaryngology MDs and DOs, non-physician healthcare providers, and licensed professional non-healthcare providers only represented 7.4% of videos posted, leading to concerns of the reliability and quality of OD-related educational content on TikTok.
Each educational video received an average of 288,009 views and 48,800 likes, demonstrating that patients are interested in seeking out information on their OD-related disorders through TikTok. This disparity in the number of videos posted by otolaryngologists versus patients suggests TikTok offers a potentially captive audience with a tremendous opportunity for otolaryngologists to satisfy an unmet need of providing high-quality, evidence-based medical information. Interestingly, humorous videos received less likes than educational and patient-perspective videos with only 34,253 views per video, suggesting that patients on TikTok are less interested in comedy and more concerned with how their peers managed their disorders and recommended treatment strategies. This finding may guide future decisions in content creation, as video production efforts may not need to rely on comedy as a mechanism of increasing the popularity of educational content.
One limitation of this study is the complexity and unpredictability of trends due to TikTok’s proprietary algorithm. Trends observed in the videos abstracted in the two-week period may not necessarily remain the same in the future, especially given the rapidly evolving environment that COVID-19 presented. However, the average age of 282 days of all videos posted under queried hashtags demonstrates that there is not a rapid influx of new videos and thus the transient nature of TikTok trends may not be as applicable to OD-related TikTok videos. Another limitation of this study may be the exclusion of non-English speaking videos due to the geographic location from which the authors input the search terms and TikTok’s prioritization of English-speaking videos on the east coast of the United States of America. There may have been trends observed in videos narrated in other languages that were missed, and therefore limits the generalizability of this study. Additionally, while two authors (JSL, RW) jointly categorized video and creator types and a third author (LJM) resolved discrepancies to standardize the classifications, how the authors labeled these videos may differ from how the public and other otolaryngologists view these videos.
Patients are turning to the Internet, especially social media, for healthcare-related information. TikTok, with a burgeoning platform of 1-billion monthly users, represents an opportunity for otolaryngologists to expand their social media platform and educate more patients. However, our study revealed a concerningly poor educational quality and source reliability of OD-related otolaryngology videos on TikTok. This represents a unique opportunity for otolaryngologists (and medical organizations) to increase and leverage their social media presence to provide high quality, evidence-based medical information, which may positively impact patient care and diagnostic accuracy, management, and outcomes.
None.
None.
Joseph Lu: Conceptualization, data curation, formal analysis, writing-original draft, writing-review & editing. Richard Wu: Conceptualization, data curation, writing-original draft, writing-review & editing. Maria Armache: Data curation, formal analysis, writing-original draft, writing-review & editing. Nicholas R. Rowan: Investigation, project supervision, writing-review & editing. Leila J. Mady: Conceptualization, investigation, project supervision, formal analysis, writing-review & editing.