Have a personal or library account? Click to login
SASSCAL WebSAPI: A Web Scraping Application Programming Interface to Support Access to SASSCAL’s Weather Data Cover

SASSCAL WebSAPI: A Web Scraping Application Programming Interface to Support Access to SASSCAL’s Weather Data

Open Access
|Jul 2021

Figures & Tables

dsj-20-1221-g1.png
Figure 1

Visualisation of AWS data via the SASSCAL Weathernet.

dsj-20-1221-g2.png
Figure 2

Manually extracting data from the SASSCAL Weathernet. This process is costly, time consuming and error-prone.

dsj-20-1221-g3.png
Figure 3

Workflow of the SASSCAL WebSAPI.

Table 1

R packages proposed in this work.

PACKAGEDESCRIPTION
rvest (Wickham & Wickham 2016)web scraping
Xml2 (Lang & Lang 2015)XML document processing
stringr (Wickham & Wickham 2019)data cleaning and preparation
ggplot (Wickham 2011)visualisation of graphics
shiny (Chang et al. 2015)dashboard design
leaflet (Graul & Graul 2016)reactive maps
dygraphs (Vanderkam et al. 2015)time-series data and interactivity
data.table (Dowle et al. 2019)tables and data munging
flexdashboard (Allaire 2017)shiny dashboard design
Algorithm 1

Visualise the AWSs of a given country.

1c←dataframe(w, x, y, z)
2leaflet(data = c) %>%
3addTiles() %>%
4setMaxBounds(x1, y2, x2, y2) %>%
5addMarkers(∼ long,∼ lat, label= ∼ name)
Algorithm 2

Data scraper.

1AWS_ID_Getter← function(AWS) {
2V = c(“x”, “value”); parent = emptyenv()
3assign_hash ← Vectorize(assign, vectorize.args = V)
4get_hash ← Vectorize(get, vectorize.args = “x”)
5exists_hash ← Vectorize(exists, vectorize.args = “x”)
6source(“All_AWS_ID.R”)
7hash ← new.env(hash = TRUE, parent, size = 100L)
8assign_hash(AWS_Name, AWS_ID, hash)
9ID_Getter←hash[[AWS]]
10return(ID_Getter) }
Algorithm 3

Data harvesting.

1μ ← TheHarvester(AWS_NAME,DATE,ρ)
2DOMreadHTMLTable(URL)
3μDataWrangler(as.data.frame(DOM[β]))
4datatable(μ, ϕ, ω)
dsj-20-1221-g4.png
Figure 4

Visualising Botswana AWS using Algorithm 1.

dsj-20-1221-g5.png
Figure 5

Screenshot of the SASSCAL WebSAPI for capturing user input when requesting weather data. The GUI allows end users to select the geographical location of interest (i.e., Botswana), temporal resolution, the AWS of interest and the downloading of data. The functionality of multi-input selection of AWSs provides end users with a feedback mechanisms to notify about the selected AWS as seen on the tab titled “Currently Selected AWS.” This is quite useful for a quick exploration of geographic locations before downloading data.

Algorithm 4

Dashboard design for dissemination.

Input: It requires Algorithm 4.
Result: SASSCAL WebSAPI GUI
1While (Interactive) do
2      gui ← fluidPage (F ← DataScraper())
3      TdashboardHeader(…),
4      SDB← dashboardSidebar(…),
5      B←dashboardBody(fluidRow(…)));
6      server ← function(I,O) { Communicator(F) };
7      shinyApp(gui, server);
dsj-20-1221-g6.png
Figure 6

Screenshot of the SASSCAL WebSAPI’s GUI for data request, visualisation and extraction of data. In addition to selecting the desired AWS, temporal resolution, and the date range, the SASSCAL WebSAPI’s GUI allows end users to select the desired variables.

Language: English
Submitted on: May 26, 2020
Accepted on: May 7, 2021
Published on: Jul 28, 2021
Published by: Ubiquity Press
In partnership with: Paradigm Publishing Services
Publication frequency: 1 issue per year

© 2021 Tsaone Swaabow Thapelo, Molaletsa Namoshe, Oduetse Matsebe, Tshiamo Motshegwa, Mary-Jane Morongwa Bopape, published by Ubiquity Press
This work is licensed under the Creative Commons Attribution 4.0 License.