Automated Data Collection with R

A hands on guide to web scraping and text mining for both beginners and experienced users of R
* Introduces fundamental concepts of the main architecture of the web and databases and covers HTTP, HTML, XML, JSON, SQL.
* Provides basic techniques to query web documents and data sets (XPath and regular expressions).
* An extensive set of exercises are presented to guide the reader through each technique.
* Explores both supervised and unsupervised techniques as well as advanced techniques such as data scraping and text management.
* Case studies are featured throughout along with examples for each technique presented.
* R code and solutions to exercises featured in the book are provided on a supporting website.

Preface xv
1 Introduction 1
1. 1 Case study: World Heritage Sites in Danger 1
1. 2 Some remarks on web data quality 7
1. 3 Technologies for disseminating, extracting, and storing web data 9
1. 4 Structure of the book 13
Part One A Primer on Web and Data Technologies 15
2 HTML 17
2. 1 Browser presentation and source code 18
2. 2 Syntax rules 19
2. 3 Tags and attributes 24
2. 4 Parsing 32
3 XML and JSON 41
3. 1 A short example XML document 42
3. 2 XML syntax rules 43
3. 3 When is an XML document well formed or valid? 51
3. 4 XML extensions and technologies 53
3. 5 XML and R in practice 60
3. 6 A short example JSON document 68
3. 7 JSON syntax rules 69
3. 8 JSON and R in practice 71
4 XPath 79
4. 1 XPath--a query language for web documents 80
4. 2 Identifying node sets with XPath 81
4. 3 Extracting node elements 93
5 HTTP 101
5. 1 HTTP fundamentals 102
5. 2 Advanced features of HTTP 116
5. 3 Protocols beyond HTTP 124
5. 4 HTTP in action 126
6 AJAX 149
6. 1 JavaScript 150
6. 2 XHR 154
6. 3 Exploring AJAX with Web Developer Tools 158
7 SQL and relational databases 164
7. 1 Overview and terminology 165
7. 2 Relational Databases 167
7. 3 SQL: a language to communicate with Databases 175
7. 4 Databases in action 188
8 Regular expressions and essential string functions 196
8. 1 Regular expressions 198
8. 2 String processing 207
8. 3 A word on character encodings 214
Part Two A Practical Toolbox forWeb Scraping and Text Mining 219
9 Scraping the Web 221
9. 1 Retrieval scenarios 222
9. 2 Extraction strategies 270
9. 3 Web scraping: Good practice 278
9. 4 Valuable sources of inspiration 290
10 Statistical text processing 295
10. 1 The running example: Classifying press releases of the British government 296
10. 2 Processing textual data 298
10. 3 Supervised learning techniques 307
10. 4 Unsupervised learning techniques 313
11 Managing data projects 322
11. 1 Interacting with the file system 322
11. 2 Processing multiple documents/links 323
11. 3 Organizing scraping procedures 328
11. 4 Executing R scripts on a regular basis 334
Part Three A Bag of Case Studies 341
12 Collaboration networks in the US Senate 343
12. 1 Information on the bills 344
12. 2 Information on the senators 350
12. 3 Analyzing the network structure 353
12. 4 Conclusion 358
13 Parsing information from semistructured documents 359
13. 1 Downloading data from the FTP server 360
13. 2 Parsing semistructured text data 361
13. 3 Visualizing station and temperature data 368
14 Predicting the 2014 Academy Awards using Twitter 371
15 Mapping the geographic distribution of names 380
15. 1 Developing a data collection strategy 381
15. 2 Website inspection 382
15. 3 Data retrieval and information extraction 384
15. 4 Mapping names 387
15. 5 Automating the process 389
16 Gathering data on mobile phones 396
16. 1 Page exploration 396
16. 2 Scraping procedure 404
16. 3 Graphical analysis 406
16. 4 Data storage 408
17 Analyzing sentiments of product reviews 416
17. 1 Introduction 416
17. 2 Collecting the data 417
17. 3 Analyzing the data 426
17. 4 Conclusion 434
References 435
General index 442
Package index 448
Function index 449