{"id":345,"date":"2022-10-31T03:10:57","date_gmt":"2022-10-31T03:10:57","guid":{"rendered":"https:\/\/ap.pstek.nl\/pstek_wp\/?p=345"},"modified":"2022-10-31T04:21:45","modified_gmt":"2022-10-31T04:21:45","slug":"corpus-loading-and-text-cleaning-with-quanteda-in-r","status":"publish","type":"post","link":"https:\/\/ap.pstek.nl\/pstek_wp\/2022\/corpus-loading-and-text-cleaning-with-quanteda-in-r\/","title":{"rendered":"Corpus Loading and Text Cleaning with Quanteda in R"},"content":{"rendered":"\n

Assuming you have a file with text data (perhaps a spreadsheet that you have exported as a CSV file, or data scraped from Google News<\/a>), you can now start to build and clean your corpus. Fortunately, this is made very easy by functions in the Quanteda<\/a> package.<\/p>\n\n\n\n

First, we load quanteda and the corpus (mycorpus.csv<\/strong> with a column with text called all_para<\/strong>), and we make sure it is readable using quanteda’s corpus<\/strong> function, by transforming the loaded data into characters using as.characters<\/strong>.<\/p>\n\n\n\n

library(quanteda)\nmycorpus <- read.csv('mycorpus.csv')\nmycorpus$all_para <- as.character(mycorpus$all_para)\nmycorpus <- corpus(mycorpus$all_para)<\/code><\/pre>\n\n\n\n
Second, we use quanteda’s built-in functions to convert the corpus into tokens<\/strong> (for simplicity sake, think of tokens as individual words). We then carry our several “cleaning” steps, namely:<\/p>\n\n\n\n
    \n
  1. Remove punctuation (remove_punct = TRUE<\/strong>) and remove numbers (remove_numbers = TRUE<\/strong>),<\/li>\n\n\n\n
  2. Remove stop words such as “the”, “a”, “them”, “he”, “she”, etc. There are several repositories of stopwords<\/a> for different languages, but do test them carefully as not all may be equally good (e.g. issues with Korean<\/a>). In this case we use English stopwords (en<\/strong>) and the stopwords-iso<\/strong> source, which seems pretty good.<\/li>\n\n\n\n
  3. Next, depending on the topic you are researching, it can be useful to remove some additional words. For instance, the words used in your search terms, which will likely be very frequent, and can “overshadow” other words used in your analysis, because almost everything is related to them. Should this be the case, you can create a vector, which is called samesame<\/strong> in the example. In your first round of analysis you may want to leave this out, hence the #.<\/li>\n\n\n\n
  4. Then there are two more functions, one is shortening words (tokens) to their wordstem<\/strong>, e.g. making singular and plural into the singular. While this is a useful function, it sometimes truncates words in an odd way.<\/li>\n\n\n\n
  5. It is usually a good idea to also convert all words to lowercase, so that “Happy” and “happy” are not seen as two different words.<\/li>\n<\/ol>\n\n\n\n
    In the final step, the tokens are converted into a “Document Feature Matrix” (DFM<\/strong>). For the purposes of this tutorial, just imagine that it’s your box of cleaned words.<\/p>\n\n\n\n
    mytokens <- tokens(mycorpus, remove_punct = TRUE, remove_numbers = TRUE)\nmytokens <- tokens_select(mytokens, stopwords('en', source='stopwords-iso'), selection='remove')\n#mytokens <- tokens_select(mytokens, samesame, selection='remove')\nmytokens <- tokens_wordstem(mytokens)\nmytokens <- tokens_tolower(mytokens)\n\nmydfm <- dfm(mytokens)<\/code><\/pre>\n\n\n\n
    With your bright, shiny and clean DFM loaded, it’s time to do some text analysis!<\/p>\n\n\n\n
    Next: Basic Text Analysis and Visualization in R<\/a><\/p>\n","protected":false},"excerpt":{"rendered":"
    Assuming you have a file with text data (perhaps a spreadsheet that you have exported as a CSV file, or data scraped from Google News), you can now start to build and clean your corpus. Fortunately, this is made very easy by functions in the Quanteda package. First, we load quanteda and the corpus (mycorpus.csv […]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"closed","ping_status":"","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[85],"tags":[99,42,43,100,96],"_links":{"self":[{"href":"https:\/\/ap.pstek.nl\/pstek_wp\/wp-json\/wp\/v2\/posts\/345"}],"collection":[{"href":"https:\/\/ap.pstek.nl\/pstek_wp\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/ap.pstek.nl\/pstek_wp\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/ap.pstek.nl\/pstek_wp\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/ap.pstek.nl\/pstek_wp\/wp-json\/wp\/v2\/comments?post=345"}],"version-history":[{"count":5,"href":"https:\/\/ap.pstek.nl\/pstek_wp\/wp-json\/wp\/v2\/posts\/345\/revisions"}],"predecessor-version":[{"id":360,"href":"https:\/\/ap.pstek.nl\/pstek_wp\/wp-json\/wp\/v2\/posts\/345\/revisions\/360"}],"wp:attachment":[{"href":"https:\/\/ap.pstek.nl\/pstek_wp\/wp-json\/wp\/v2\/media?parent=345"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/ap.pstek.nl\/pstek_wp\/wp-json\/wp\/v2\/categories?post=345"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/ap.pstek.nl\/pstek_wp\/wp-json\/wp\/v2\/tags?post=345"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}