Web scraping

class: center, middle, title

# UE Visualisation

### 2020-2021

## Dr. Maxime Wack

### AHU Informatique médicale
#### Hôpital Européen Georges Pompidou, </br> Université de Paris

---

# Web scraping

### Utilisation de `httr` et `rvest`

## httr

Permet de faire des requêtes réseau

→ interroger et télécharger directement depuis R

## rvest

Extraction de données depuis des pages HTML

---

# httr

Télécharger une page wikipedia

```r
GET("https://en.wikipedia.org/wiki/Comparison_of_operating_systems") -> wiki
```

```
## Response [https://en.wikipedia.org/wiki/Comparison_of_operating_systems]
##   Date: 2020-11-17 03:43
##   Status: 200
##   Content-Type: text/html; charset=UTF-8
##   Size: 227 kB
## <!DOCTYPE html>
## <html class="client-nojs" lang="en" dir="ltr">
## <head>
## <meta charset="UTF-8"/>
## <title>Comparison of operating systems - Wikipedia</title>
## <script>document.documentElement.className="client-js";RLCONF={"wgBreakFrames...
## "Articles with unsourced statements from February 2007","Wikipedia articles n...
## "Q3345986"};RLSTATE={"ext.globalCssJs.user.styles":"ready","site.styles":"rea...
## "ext.wikimediaEvents","ext.navigationTiming","ext.uls.compactlinks","ext.uls....
## <script>(RLQ=window.RLQ||[]).push(function(){mw.loader.implement("user.option...
## ...
```

---

# Parsing HTML

```r
wiki %>%
  read_html -> wiki_html
```

```
## {html_document}
## <html class="client-nojs" lang="en" dir="ltr">
## [1] <head>\n<meta http-equiv="Content-Type" content="text/html; charset=UTF-8 ...
## [2] <body class="mediawiki ltr sitedir-ltr mw-hide-empty-elt ns-0 ns-subject  ...
```

---

# Sélecteurs CSS

[W3Schools](https://www.w3schools.com/cssref/css_selectors.asp)

### Selecteurs permettant d'identifier un **nœud** précis dans le **DOM** (Document Object Model) d'une page HTML

### Permet de sélectionner par identifiant, classe, position dans la hiérarchie, position entre élements d'un même niveau, ou relativement entre élements

### Utiliser l'**inspecteur** des outils de développement du navigateur pour identifier les éléments à capturer

---

# Sélecteurs CSS

```r
wiki_html %>%
  html_nodes(".wikitable")
```

```
## {xml_nodeset (4)}
## [1] <table class="wikitable sortable" style="font-size: smaller; text-align:  ...
## [2] <table class="wikitable sortable" style="font-size: smaller; text-align:  ...
## [3] <table class="wikitable" style="font-size: smaller; text-align: center; w ...
## [4] <table class="wikitable" style="font-size: smaller; text-align: center; w ...
```

```r
wiki_html %>%
  html_node("div + .wikitable")
```

```
## {html_node}
## <table class="wikitable sortable" style="font-size: smaller; text-align: center; width: auto; sortby: name;">
## [1] <tbody>\n<tr>\n<th>Name\n</th>\n<th>Creator\n</th>\n<th abbr="Initial rel ...
```

---

# Extraction d'une table

```r
wiki_html %>%
  html_node("div + .wikitable") %>%
  html_table -> wikitable
```

<div id="htmlwidget-bb3e3440d2054f364517" style="width:100%;height:auto;" class="datatables html-widget"></div>
<script type="application/json" data-for="htmlwidget-bb3e3440d2054f364517">{"x":{"filter":"none","data":[["AIX","Android","AmigaOS classic","AmigaOS 4","ArcaOS","Chrome OS","DragonFly BSD","FreeBSD","Genode","GhostBSD","eComStation","EPOC32","GNU/Linux","Haiku","HP-UX","IBM i","Inferno","iOS","IRIX","Classic Mac OS","MVS","macOS","macOS Server","MPE","MCP","MPE/XL","MINIX 3","NetBSD","NetWare","NeXTSTEP","OpenBSD","OpenIndiana","OpenVMS","OS/360","OS/390","OS 2200","OS/2","Plan 9","QNX","Solaris","Symbian","Symbian platform","Windows Server (NT family)","Windows (NT family)","Windows (classic 9x family)","RISC iX","RISC OS","RISC OS 4","RISC OS 5","RISC OS 6","ZETA","STOP 6, XTS-400","ReactOS","TrueOS","VxWorks","z/OS","z/VSE","z/VM","HP NonStop","Name"],["IBM","Android, Inc., Google","Commodore International, Haage & Partner, Hyperion Entertainment","Hyperion Entertainment","Arca Noae, LLC","Google","Matthew Dillon","The FreeBSD Project","Genode Labs","Eric Turgeon","Serenity Systems, Mensys BV","Psion PLC","Notable contributors include: Richard Stallman for GNU Project and Linus Torvalds for Linux and the Unixes they emulated; Red Hat, Debian Project See: Comparison of Linux distributions and Linux Kernel#Development","Haiku Inc.","HP (now Hewlett Packard Enterprise)","IBM","Bell Labs","Apple Inc.","SGI","Apple Inc.","IBM","Apple Inc.","Apple Inc.","HP","Unisys","HP","Andrew S. Tanenbaum","The NetBSD Project","Novell","NeXT","The OpenBSD Project","Many, based on software developed by Sun Microsystems and many others","DEC (now VSI)","IBM","IBM","Unisys","IBM and Microsoft","Bell Labs","QNX Software Systems","Sun (now Oracle Corporation)","Symbian Ltd.","Symbian Foundation","Microsoft","Microsoft","Microsoft","Acorn Computers","Acorn Computers","RISCOS Ltd, Pace plc","Castle Technology, RISC OS Open","RISCOS Ltd","yellowTAB","BAE Systems","ReactOS development team","PC-BSD Software","Wind River Systems","IBM","IBM","IBM","HP (now Hewlett Packard Enterprise)","Creator"],["1986","2008","1985","2004","2017","2009","2003","1993","2008","2009","2001","1996","1991 (kernel), See: Comparison of Linux distributions and History of Linux","2009","1983","1988","1997","2007","1988","1984","1972","2001","2001","1974","1961","1987","2005","1993","1985","1989","1995","2010","1977","1966","1995","1967 as Exec 8e","1987","1993","1982","1992","1998","2010 (initially 1998 as Symbian)","1993","1993","1995","1988","1987","1999","2002","2006","2005","2003","1996","2006","1987","2000","2007","2000","1974","Initial public release"],["UNIX System V Release 3","None","TRIPOS (as the disk operating component of AmigaOS)","AmigaOS classic","OS/2","Chromium OS","FreeBSD","386BSD","None","FreeBSD","OS/2","","None","BeOS R5","UNIX System V","OS/400","Plan 9","macOS","UNIX System V","None[g 2][g 3]","OS/360","NeXTSTEP, BSD","NeXTSTEP, BSD","None","None","MPE","Minix2","386BSD","S-Net","Unix","NetBSD 1.0","OpenSolaris","RSX-11M","None","MVS","Exec 8, OS 1100","MS-DOS","Unix","Unix, POSIX","SunOS","EPOC32","Symbian","OS/2, Windows 3.x and MS-DOS","OS/2, Windows 9x and MS-DOS","MS-DOS, Windows NT 3.5","BSD 4.3","Arthur, also the BBC Master OS","RISC OS","RISC OS 4","RISC OS 4","BeOS R5","STOP 5, XTS-300","Windows NT","FreeBSD[g 5]","VRTX","OS/390","VSE/ESA","VM","Guardian","Predecessor"],["7.2 TL4","11","3.9 BB2","4.1 Final Edition","5.0.6","86.0.4240.77","5.8.3","12.1","19.05[1]","19.10[2]","2.1","ER5","5.8.9 (kernel)","R1/Beta 2","11i v3 2020 Release","7.4","Fourth Edition","14.0.1","6.5.30","9.2.2","MVS/ESA SP - JES3 Version 5 R2.2","10.15.1 Catalina","10.12 / September 20, 2016","MPE-V","CP OS 19.0","7.5","3.3.0","9.1","6.5 SP8","3.3","6.8","2020.04","9.0","Operating System/360 R21.8","OS/390 Version 2 R10","CP OS 18 (Exec 49.2)","4.52","Fourth Edition","7.0.0","11.4","9.5","3.0.4","Windows Server 2019 (Version 10.0.17763)","Windows 10 (Version 2004)","Windows Me (Win 4.90.3000)","1.21c","3.71","4.39","5.24[5]","6.20","1.5","6.5","0.4.13","10.1[6]","7","Version 2.4 (V2R4)","6.2","7.1","H06.24/J06.13","Current stable version"],["2019, November","2020, September 8","2002-03-20","2014","2020, August 31","2020, October 13","2020, September 24","2019, November 4","2019, May 29","2019, October 26","2011, May 20","1999","2020, September 12 (kernel)","2020, June 9","May 2020","2019, April 23","2009, June 30","2020, September 24","2006","2000","1995, September 29","2019, October 29","2016, September 20","1988","2019, June","2002","2014","2020, October 18","2009, May 6","1995","2020, October 18","2020, May 5","2020, May 15","1972, August","2000, September 29","2018, July 18","2001","2003 (except for minor later updates)","2017, March","2018, August 28","2009","2010","2018, October 2","2020, May 27","2000","1993","1997","2004","2018","2009","2007","2008, August","2020, April 9","2014, November 16","2014 March","2019, September 30","2017, December 1","2018, September 21 [7]","2012","Release date"],["Bundled with hardware","Free","Discontinued; Bundled with hardware up to version 3.0 (Amiga International hardware came with 3.1); versions 2.1, 3.0, 3.1, 3.5, 3.9 also available as separate packages","4.0 bundled with hardware; 4.0 for classic and 4.1 available as standalone package at €29","Personal edition US$129.00Commercial edition US$229.00","Bundled with hardware, 32-bit edition dropped","Free","Free","Free, source code only","Free","Home-student edition (max. three per site) $145.00business edition $290.00","Discontinued; Commercial","Free","Free","US$400","Bundled with hardware","Free","Bundled with hardware and free updates given to most existing users, subject to hardware requirements","Discontinued; Bundled with hardware","Discontinued; Was bundled with 68k and PowerPC Macs;\nversions 7-9 sold as retail upgrades[g 4]","Bundled with hardware","Bundled with hardware; No-cost update via Mac App Store for users of Mac OS X 10.6 or later, assuming hardware requirements are met","Previously bundled with hardware; No longer a separate operating system, but a group of services installed atop any current version of Mac OS X; US$19.99 on the Mac App Store","Discontinued; Was bundled with HP-3000 CISC hardware \"Classic\"","Bundled with hardware","Discontinued; Was bundled with HP-3000 PA-RISC hardware","Free","Free","Superseded by Novell Open Enterprise Server; Was US$184 (equivalent to $219.28 in 2019) (one-user)","Discontinued; Was bundled with hardware, then sold separately","Free","Free","Commercial, free non-commercial use","Bundled with hardware","Bundled with hardware","Bundled with hardware","Discontinued (see ArcaOS successor); Was US$300 (equivalent to $433.17 in 2019)","Free","Bundled with BlackBerry 10 and PlayBook devices. Commercial; an academic version exists that needs authorization code before installing","Commercial; (but free/no-cost perpetual license when used \"for the purpose of developing, testing, prototyping and demonstrating your applications\"[3])","Discontinued; Commercial","Free","US$1050 5 CALs server; other editions dependent on number of CALs purchased","Windows 10 Home US$119, Windows 10 Pro US$199[4]","Discontinued","Discontinued; Was bundled with hardware","Discontinued; Was bundled with hardware","Bundled with hardware, then sold separately at £70 (US$127)","Free","Bundled with hardware, then sold separately at £70 (US$127)","Discontinued","US$60,000 (equivalent to $71,249 in 2019)+; bundled with XTS hardware and OEM licensed","Free","Free","Paid","Monthly license fee, about US$130 and up","Monthly license fee","Monthly license fee","Non-free","Cost, availability"],["Proprietary","Apache 2.0, GNU GPLv2","Proprietary, open source clone available under AROS Public License","Proprietary","Proprietary","Proprietary: Google OS Terms of Service","BSD","BSD","AGPL","BSD","Proprietary","Proprietary","GNU GPLv2 (kernel)","MIT","Proprietary","Proprietary","MIT, GNU GPL, GNU LGPL, LPL","Proprietary higher level API layers; open source core system (ARM versions): APSL, GNU GPL, others","Proprietary","Proprietary","Proprietary","Proprietary higher level API layers; open source core system (Intel-PowerPC versions): APSL, GNU GPL, others","Proprietary higher level API layers; open source core system (Intel-PowerPC versions): APSL, GNU GPL, others","Proprietary","Proprietary","Proprietary","BSD","BSD","Proprietary","Proprietary","ISC","CDDL","Proprietary","Proprietary","Proprietary","Proprietary","Proprietary","LPL","Proprietary","CDDL","Proprietary","EPL","Proprietary; Source-available","Proprietary; Source-available","Proprietary","Proprietary","Proprietary","Proprietary","Apache License","Proprietary","Proprietary","Proprietary","GNU GPL, GNU LGPL","BSD","Proprietary","Proprietary","Proprietary","Proprietary","Proprietary","Preferred license[g 1]"],["Server, NAS, workstation","Smartphone, tablet computer, education","Workstation, personal computer","Workstation, personal computer","Server, workstation, personal computer","Chromebook, Chromebox, Chromebase and tablets","Server, workstation, NAS, embedded","Server, workstation, NAS, embedded","Desktop, Embedded, Server","Desktop, workstation","Server, workstation, personal computer","PDA","See: Comparison of Linux distributions","Personal computer","Server","Server","NAS, server, embedded","Smartphone, music player, tablet computer","Server, workstation","Workstation, personal computer","IBM mainframe","Workstation, personal computer, embedded","Server","Server","Server","Server","Workstation","NAS, server, workstation, embedded","Server","Workstation","Server, NAS, workstation, embedded","Server, workstation","Server, workstation","IBM mainframe","IBM mainframe","Server","Personal computer, server","Workstation, server, embedded, HPC","Automotive, medical, smartphones, consumer, industrial, embedded, safety","Server, workstation","Phones","embedded","Server, NAS, embedded","Workstation, personal computer, media center, Tablet PC, embedded","Personal computer, media center","Workstation","Education, personal computer","Education, personal computer","Education, personal computer","Education, personal computer","Personal computer, media center, workstation","Server, workstation","Workstation, personal computer","Personal computer, workstation, server","Embedded Real-time systems","IBM mainframe","IBM mainframe","IBM mainframe","HP Nonstop Servers","Target system type"]],"container":"<table class=\"display\">\n  <thead>\n    <tr>\n      <th>Name<\/th>\n      <th>Creator<\/th>\n      <th>Initial public release<\/th>\n      <th>Predecessor<\/th>\n      <th>Current stable version<\/th>\n      <th>Release date<\/th>\n      <th>Cost, availability<\/th>\n      <th>Preferred license[g 1]<\/th>\n      <th>Target system type<\/th>\n    <\/tr>\n  <\/thead>\n<\/table>","options":{"paging":false,"info":false,"searching":false,"order":[],"autoWidth":false,"orderClasses":false}},"evals":[],"jsHooks":[]}</script>

---

# Exercices

### Transformer cette table en forme normale

### Extraire la table avec les informations techniques

### Identifier les OS libres fonctionnant avec un microkernel