{"id":351,"date":"2022-10-31T04:21:27","date_gmt":"2022-10-31T04:21:27","guid":{"rendered":"https:\/\/ap.pstek.nl\/pstek_wp\/?p=351"},"modified":"2022-10-31T04:21:27","modified_gmt":"2022-10-31T04:21:27","slug":"basic-text-analysis-and-visualization-in-r","status":"publish","type":"post","link":"https:\/\/ap.pstek.nl\/pstek_wp\/2022\/basic-text-analysis-and-visualization-in-r\/","title":{"rendered":"Basic Text Analysis and Visualization in R"},"content":{"rendered":"\n<p>At its most basic level, text analysis is about counting words. If words are frequently used, we assume that they are important. If words occur together, we assume that they are related. Obviously, that is not always the case, but a discerning researcher like yourself will be able to filter this information, provide context and meaning, and draw the appropriate conclusions.<\/p>\n\n\n\n<p>Text analysis often provides you with an opportunity to gather more solid evidence of the importance of certain words or concepts, and the relationship between them, and sometimes can lead to the discovery of hidden patterns, that you as a human observer may have missed.<\/p>\n\n\n\n<p>In this short tutorial, using the <strong>quanteda<\/strong>, <strong>ggplot2<\/strong> and <strong>quanteda.textplot<\/strong> packages, the following analysis methods are covered:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Word frequency analysis and visualization in a frequency chart and word cloud.<\/li>\n\n\n\n<li>Co-word analysis and visualization in a co-word network.<\/li>\n<\/ul>\n\n\n\n<p>Before starting, it is assumed that you have cleaned and created a Document Feature Matrix (DFM) called <strong>mydfm<\/strong> (<a href=\"https:\/\/ap.pstek.nl\/pstek_wp\/2022\/corpus-loading-and-text-cleaning-with-quanteda-in-r\/\" data-type=\"post\" data-id=\"345\">see earlier tutorial<\/a>).<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">Word Frequency Analysis<\/h3>\n\n\n\n<p>Word frequency analysis is about counting words, and it is useful to look at, say, the 100 most frequently used words (<strong>topfeatures<\/strong>), which can be extracted as follows:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>library(quanteda)\nwfreq &lt;-data.frame(topfeatures(mydfm, 100))<\/code><\/pre>\n\n\n\n<p>Aside from looking at the words in the data frame, they can also be visualized.<\/p>\n\n\n\n<p>Below an example of the 10 most frequently occurring words in a bar chart (based on a news corpus about sustainable investment in the Philippines). Note that we manipulate the wfreq data frame for it to be read properly by <strong>ggplot<\/strong>.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>library(ggplot2)\nwfreq &lt;- as.data.frame(topfeatures(mydfm, 10))\nwfreq$n &lt;- as.numeric(wfreq$.)\nwfreq$word &lt;- row.names(wfreq)\nggplot(wfreq, aes(x = reorder(word, n, function(n) -n), y = n)) + geom_bar(stat = \"identity\") + xlab('')<\/code><\/pre>\n\n\n\n<figure class=\"wp-block-image size-full\"><img loading=\"lazy\" decoding=\"async\" width=\"622\" height=\"340\" src=\"https:\/\/ap.pstek.nl\/pstek_wp\/wp-content\/uploads\/2022\/10\/image-1.png\" alt=\"\" class=\"wp-image-353\" srcset=\"https:\/\/ap.pstek.nl\/pstek_wp\/wp-content\/uploads\/2022\/10\/image-1.png 622w, https:\/\/ap.pstek.nl\/pstek_wp\/wp-content\/uploads\/2022\/10\/image-1-300x164.png 300w\" sizes=\"(max-width: 622px) 100vw, 622px\" \/><\/figure>\n\n\n\n<p>And next, the 100 most frequently occurring words in a word cloud!<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>library(quanteda.textplots)\nset.seed(123); textplot_wordcloud(mydfm, max_words = 100)<\/code><\/pre>\n\n\n\n<figure class=\"wp-block-image size-full\"><img loading=\"lazy\" decoding=\"async\" width=\"622\" height=\"340\" src=\"https:\/\/ap.pstek.nl\/pstek_wp\/wp-content\/uploads\/2022\/10\/image-2.png\" alt=\"\" class=\"wp-image-354\" srcset=\"https:\/\/ap.pstek.nl\/pstek_wp\/wp-content\/uploads\/2022\/10\/image-2.png 622w, https:\/\/ap.pstek.nl\/pstek_wp\/wp-content\/uploads\/2022\/10\/image-2-300x164.png 300w\" sizes=\"(max-width: 622px) 100vw, 622px\" \/><\/figure>\n\n\n\n<p>Word frequency analysis is a useful first step to explore your corpus. Based on it, you may wish to remove some additional highly frequent words which &#8220;overshadow&#8221; the rest of the corpus. But, these are always decisions that require thought and justification.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">Co-Word Analysis<\/h3>\n\n\n\n<p>For co-word analysis, we convert the DFM into a Frequency Co-occurrence Matrix (FCM) using <strong>fcm<\/strong>. For the purpose of this tutorial, think of the FCM as a &#8220;box&#8221; filled with the connections between words.<\/p>\n\n\n\n<p>Because the FCM can be very large, thus making it dificult to visually analyze, we find the 50 most frequently occurring words (<strong>topfeatures<\/strong>), and then select them in a new FCM called myselect using the <strong>fcm_select<\/strong> feature.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>myfcm &lt;- fcm(mydfm)\nfeat &lt;- names(topfeatures(myfcm, 50))\nmyselect &lt;- fcm_select(myfcm, pattern = feat, selection = \"keep\")<\/code><\/pre>\n\n\n\n<p>Having cut the FCM to size, we can now generate a co-occurrence network. To visualize the network well, it can be useful to impose a minimum frequency (<strong>min_freq<\/strong>), removing very rarely occurring links. You may also want to adjust the size of the dots (<strong>vertex_size<\/strong>) to reflect their importance.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>size &lt;- log(colSums(dfm_select(mydfm, feat, selection = \"keep\")))\nset.seed(112)\ntextplot_network(myselect, min_freq = 0.8, vertex_size = size \/ max(size) * 3)<\/code><\/pre>\n\n\n\n<figure class=\"wp-block-image size-full\"><img loading=\"lazy\" decoding=\"async\" width=\"812\" height=\"428\" src=\"https:\/\/ap.pstek.nl\/pstek_wp\/wp-content\/uploads\/2022\/10\/image-3.png\" alt=\"\" class=\"wp-image-357\" srcset=\"https:\/\/ap.pstek.nl\/pstek_wp\/wp-content\/uploads\/2022\/10\/image-3.png 812w, https:\/\/ap.pstek.nl\/pstek_wp\/wp-content\/uploads\/2022\/10\/image-3-300x158.png 300w, https:\/\/ap.pstek.nl\/pstek_wp\/wp-content\/uploads\/2022\/10\/image-3-768x405.png 768w\" sizes=\"(max-width: 812px) 100vw, 812px\" \/><\/figure>\n\n\n\n<p>The present network can be analyzed further using network analysis techniques, its visualization can be further refined, etc. The present tutorial gives you a basis to start with.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>At its most basic level, text analysis is about counting words. If words are frequently used, we assume that they are important. If words occur together, we assume that they are related. Obviously, that is not always the case, but a discerning researcher like yourself will be able to filter this information, provide context and [&hellip;]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"closed","ping_status":"","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[85],"tags":[101,19,102,42,43,96,59],"class_list":["post-351","post","type-post","status-publish","format-standard","hentry","category-tutorial","tag-data-visualization","tag-ggplot2","tag-network","tag-quanteda","tag-r","tag-text-analysis","tag-word-cloud"],"_links":{"self":[{"href":"https:\/\/ap.pstek.nl\/pstek_wp\/wp-json\/wp\/v2\/posts\/351","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/ap.pstek.nl\/pstek_wp\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/ap.pstek.nl\/pstek_wp\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/ap.pstek.nl\/pstek_wp\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/ap.pstek.nl\/pstek_wp\/wp-json\/wp\/v2\/comments?post=351"}],"version-history":[{"count":4,"href":"https:\/\/ap.pstek.nl\/pstek_wp\/wp-json\/wp\/v2\/posts\/351\/revisions"}],"predecessor-version":[{"id":359,"href":"https:\/\/ap.pstek.nl\/pstek_wp\/wp-json\/wp\/v2\/posts\/351\/revisions\/359"}],"wp:attachment":[{"href":"https:\/\/ap.pstek.nl\/pstek_wp\/wp-json\/wp\/v2\/media?parent=351"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/ap.pstek.nl\/pstek_wp\/wp-json\/wp\/v2\/categories?post=351"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/ap.pstek.nl\/pstek_wp\/wp-json\/wp\/v2\/tags?post=351"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}