Lecture 2

2025-05-12

Topic 2: Introduction to Data Science

Data Representation

Tidy Data seminal paper by Hadley Wickham

Where possible, we want to collect and store data in a tabular format

• Each row represents an observation

• Each column represents a variable

• Each observational unit is a table

Each observation should have a unique identifier where possible

Before performing any data science, we must first ask ourselves how we should format and store data to make it amenable for data analysis. Adapted from the Tidy Data seminal paper by Hadley Wickham, the following tabular format representation is encouraged wherever possible, such that:

• Each row represents an observation

• Each column represents a variable

• Each observational unit is a table

Each observation should have a unique identifier where appropriate, and use snake_case (all lowercase letters with underscores between words) for column names. The ISO 8601 Date format (YYYY-MM-DD) should be used at all times and conditional formatting should be removed and represented instead in a separate data dictionary/legend. Missing values should be clearly labelled as NA or appropriately described in metadata (eg -1 or 999). Finally, fields should be either numeric or character for maximum cross platform and programming language compatibility, and special data types should be avoided where possible.

Data Science Workflow

• Import: Identify sources and acquire data, utilizing automation whenever possible. 
• Tidy: Wrangle data into a format conducive for understanding its properties and store it in an accessible location. Performed autonomously whenever possible. 
• Transform: Extract features of interest from tidy data and perform extra calculations. 

The data science workflow, as described in R for Data Science (2e) by the aforementioned Hadley Wickham, consists of how information is transformed from raw data into actionable insights. A full data solutions workflow consists of the following steps:

1. Import: Identify sources of and methods to acquire data, utilizing automation wherever possible.
2. Tidy: Wrangle data into a tidy format conducive for understanding its properties and store it in an accessible and secure location. Should be performed autonomously wherever possible.
3. Transform: Extract features of interest from data and perform additional computations to augment the dataset.
4. Visualize: Generate informative illustrations of the transformed data in order to provide additional context to insights.
5. Model: Apply statistical methods, machine learning, and artificial intelligence to understand data via prediction, identifying trends, and detecting anomalies.
6. Communicate: Interpret results and produce reports, web applications, and other compelling summaries of finds in a way that is digestible and actionable for your stakeholders.

Data Science Workflow

• Visualize: Generate informative illustrations of the transformed data to provide context to new insights. 

• Model: Apply statistical methods, machine learning, and artificial intelligence, to understand the data via prediction, identifying trends, and detecting anomalies. 

• Communicate: Interpret results and produce reports, dashboards, and compelling summaries of findings in a way that is digestible and actionable.

Guiding Questions

Import

• What information is necessary to address my questions?
• Where can I acquire this information from?
• How can this information be collected?
  • Can this be performed programmatically via web scraping, an API, or an existing package/library?
  • Must this be performed manually, such as live data collection or a bulk download/export?
• How frequently does this information need to be collected?

Guiding Questions

Tidy

• How is the information that I am looking to acquire currently formatted?
• What pieces of information do I want to retain to convert into a tidy format?

Transform

• What metrics should be derived from the collected data?
• What subject matter (domain) knowledge is necessary to calculate auxiliary values?

Guiding Questions

Visualize

• How should the information be presented visually?
• How has the information been visualized in analogous or adjacent use cases?

Model

• Will this information be used to predict future outcomes, make inferences on past events, or describe the phenomena taking place?

Guiding Questions

Communicate

• Who are my stakeholders receiving the insights generated from this analysis?
• What is the most sustainable medium for these insights to be shared?
• How often will these insights be shared?

As described in DevOps for Data Science by Alex K Gold, consider whether the insights will be shared statically (static reports), or incorporate live data processing (use a web application if ‘yes’, or an i nteractive report if ‘no’)

Case Study: Track and Field Performance Rankings

Motivation

Suppose that you are the High-Performance Manager for Athletics Canada, the national sport governing body for track and field, para athletics, cross country, and road, mountain, and trail running. You have two questions:

1. “What is our most successful event discipline?”

2. “Is there a relationship between our athlete’s performances and their age?”

Step 1: Import

• What information is necessary to address my questions?
• Where can I acquire this information from?
• How can this information be collected?
• How frequently does this information need to be collected?

Step 1: Import

• What information is necessary to address my questions?
  • Performance results of track and field athletes
• Where can I acquire this information from?
  • World Athletics Website
  • If you are ever in doubt, paste your question into ChatGPT. It’s INCREDIBLY good at pointing you in the right direction

Step 1: Import

• How can this information be collected?
  • Since the webpage uses static HTML, it can be scraped with rvest::read_html or pandas.read_html
  • There are several GitHub repositories with APIs (1, 2, 3)
  • Always look for existing data acquisition solutions before trying to create your own
• How frequently does this information need to be collected?
  • World Rankings are updated every Wednesday

Example Implementation in R

sexes = c("men", "women")
all_rankings = list()

for (sex in sexes) {
  # initialize athlete_rankings and page number
  athlete_rankings = data.frame()
  page = 1
  
  # the website is paginated, so we need to loop through each page
  while(TRUE) {
    website_url <- paste0(
      "https://worldathletics.org/world-rankings/overall-ranking/", 
      sex, 
      "?regionType=countries&region=can&page=", page)
    
    webpage <- rvest::read_html(website_url)
    #... continued on next slide

From there we check out the url to see how generalizable its structure is:

https://worldathletics.org/world-rankings/overall-ranking/women?regionType=countries&region=can&page=1

Here, we see three sections of the query string: regionType, region, and page, alongside a url endpoint of women, which after some fiddling around we find can also be set to men. Therefore to make our scraping protocol as flexible as possible we initialize and set those variables dynamically instead of hardcoding them in the read_html call.

Note that this webpage is paginated, meaning that only a subset of the entire table is rendered on each page. Therefore, we run the web scraper in a while loop and request new pages by incrementing the page number at the end of each iteration until an empty page is observed, after which the loop is terminated. We run this for both men and women.

Example Implementation in R

    #... continued from previously
    new_table <- 
      webpage |> 
      rvest::html_table() |> # extract the table from the webpage html
      dplyr::bind_rows() # combine into a single dataframe
    
    if(nrow(new_table) == 0) break # this is the end of the table
    
    page <- page + 1
    athlete_rankings <- dplyr::bind_rows(athlete_rankings, new_table) 
    # add new rows to existing rows
    Sys.sleep(2.5)  # Insert pauses so as to not overwhelm a webpage
  }
  
  all_rankings <- append(all_rankings, list(athlete_rankings)) 
  # combine mens and women's results into list
}

From there we check out the url to see how generalizable its structure is:

https://worldathletics.org/world-rankings/overall-ranking/women?regionType=countries&region=can&page=1

Here, we see three sections of the query string: regionType, region, and page, alongside a url endpoint of women, which after some fiddling around we find can also be set to men. Therefore to make our scraping protocol as flexible as possible we initialize and set those variables dynamically instead of hardcoding them in the read_html call.

Note that this webpage is paginated, meaning that only a subset of the entire table is rendered on each page. Therefore, we run the web scraper in a while loop and request new pages by incrementing the page number at the end of each iteration until an empty page is observed, after which the loop is terminated. We run this for both men and women.

Tidy

• How is the information that I am looking to acquire currently formatted?
• What pieces of information do I want to retain to convert into a tidy format?

Tidy

• How is the information that I am looking to acquire currently formatted?
  • Use dplyr::glimpse
• What pieces of information do I want to retain to convert into a tidy format?
  • Athlete names, ages, scores, events, sex
  • Additional subject matter knowledge is necessary to understand why multiple events appear in Events List

We use this information to perform several cleaning steps:

• Use janitor::clean_names to format the column names in snake_case

• Combine both men’s and women’s tables by adding a column to differentiate between them using the id argument in dplyr::bind_rows

• Convert the unnamed x4 column to a more informative name

Additional subject matter knowledge is necessary to understand why multiple events appear in events_list. After further examination, we find that the score is calculated as average performance score of the athlete across the events that they compete in. Athletes are assigned “main events”, which are the first elements in the events_list, and “similar events”, which are the elements in the [ ].

Different practitioners will have different uses for this information. For this case study, we will only consider an athlete’s “main event” when performing event grouping, and as such use some string manipulation using regular expressions to remove the “similar events”.

Finally, we notice that in the date of birth (dob) column only the year element is included for some athletes, while the entire date is retained for others. Therefore for consistency, we retain only the year for all athletes using string manipulation and several functions lubridate package.

Example Implementation in R

combined_table <- dplyr::bind_rows(all_rankings, .id = "sex") |> 
  janitor::clean_names() |> # format the column names in "snake_case" 
  dplyr::mutate(
    dob = case_when(
      stringr::str_detect(dob, "^\\d{4}$") ~ as.integer(dob), 
      # if only the year is displayed, preserve it
      !is.na(lubridate::dmy(dob, quiet = T)) ~ 
      lubridate::year(lubridate::dmy(dob, quiet = T))
      # if the entire birthdate is diplayed, extract the year
    ),
    sex = if_else(sex == "1", "Men", "Women"), 
    # since the men's table was scraped first
    event_list = str_remove(event_list, "\\s\\[.+\\]") 
    # remove "similar events"
    ) |> 
  dplyr::rename(country = x4, yob = dob) 
# convert x4 to 'country' and date of birth to year of birth

Transform

• What metrics should be derived from the collected data?
• What subject matter (domain) knowledge is necessary to calculate auxiliary values?

Transform

• What metrics should be derived from the collected data?
  • Individual events must be aggregated into event groups.
• What subject matter (domain) knowledge is necessary to calculate auxiliary values?
  • An understanding of event group classifications, using either World Athletics regulations or a similar standard.

Example Implementation in R

combined_table <- 
  combined_table |> 
  mutate(event_list = str_replace(event_list, "10,000", "10000")) |> 
  # so that the comma isn't caught by `separate_longer_delim`
  separate_longer_delim(cols = event_list, delim = ",") |> 
  mutate(event_group = case_match(event_list,
    c("100m", "200m", "400m", "400mH", "100mH", "110mH") ~ "Sprints",
    c("800m", "1500m", "3000mSC") ~ "Middle Distance",
    c("5000m", "10000m") ~ "Long Distance",
    c("Heptathlon", "Decathlon") ~ "Combined Events",
    c("High Jump", "Long Jump", "Triple Jump", "Pole Vault") ~ "Jumps",
    c("10km Walk", "10km Road", "20km Walk", "35km Walk", 
    "50km Walk", "Half Marathon", "Marathon") ~ "Road Races",
    c("Cross Country", "XC Senior Race") ~ "Cross Country",
    .default = "Throws" # the rest of the events
    )
) |> add_count(competitor, event_group, name = "n_events") |> 
  slice_max(order_by = n_events, n=1, by = competitor, with_ties = F)

Before we aggregate the data, we must perform some remedial measures to make sure that athletes appear in exactly one event group. For this example, we count the number of main events that an athlete has from each event group, and then select the group that has the most main events participated in by that athlete. Ties are separated arbitrarily, which in a more rigorous analysis can be further specified.

Canada’s Best Event Groups

combined_table |> 
  dplyr::group_by(event_group, sex) |> 
  dplyr::summarise(average_performance = mean(score, na.rm = T), 
            athletes = dplyr::n(), .groups = "drop") |> 
  dplyr::mutate(rank = dplyr::min_rank(
  dplyr::desc(average_performance)), .before = 1) |> 
  dplyr::arrange(rank) |> 
  # make a nice table
  gt::gt() |> 
  gtExtras::gt_theme_538() |> 
  gt::fmt_number(decimals = 0) |> 
  gt::cols_label_with(
    fn = \(x) gt::md(stringr::str_replace_all(x, "_", "<br>"))
  ) |> 
  gt::cols_align(align = "center") |>
  gt::tab_header(
    title = "Canada's Best Event Groups",
    subtitle = paste("As of", format(Sys.Date(), "%B %d, %Y")))

Canada’s Best Event Groups

Visualize

• How should the information be presented visually?
• How has the information been visualized in analogous or adjacent use cases?

Visualize

• How should the information be presented visually?
  • We can use a density plot to explore the distribution of athlete ages in the dataset
• How has the information been visualized in analogous or adjacent use cases?
  • A scatterplot with a smoothed line of best fit are commnly used to visualize aging curves.

Visualization 1

ggplot2::ggplot(data = combined_table, mapping = ggplot2::aes(x = yob)) +
  ggplot2::geom_density() +
  ggplot2::labs(title = "Density Plot of Ages of Athletes on the World Athletics Canadian Overall Rankings", 
       subtitle = paste("As of", format(Sys.Date(), "%B %d, %Y")),
       caption = "Data taken from https://worldathletics.org",
       x = "Year of Birth",
       y = "Density") +
  ggplot2::facet_wrap(~sex) + 
  ggplot2::theme_bw() 

Visualization 1

We see that the most prominent year of birth for Canada’s ranked male athletes is 2005, while for women it is 2003. This could lead to further exploratory questions that are out of scope for this case study.

Visualization 2

ggplot2::ggplot(data = combined_table, mapping = ggplot2::aes(x = yob, y = score)) +
  ggplot2::geom_jitter() +
  ggplot2::geom_smooth() +
  ggplot2::labs(title = "World Athletics Canadian Overall Rankings", 
       subtitle = paste("As of", format(Sys.Date(), "%B %d, %Y")),
       caption = "Data taken from https://worldathletics.org",
       x = "Year of Birth",
       y = "Score") +
  ggplot2::facet_wrap(~sex, scales = "free") + 
  ggplot2::theme_bw()

Visualization 2

From these aging curves we notice that there is unsurprisingly a large proportion of athletes born after 2000. We can investigate this phenomena by looking at the distribution of years of birth among athletes, again separated by sex.

Model

• Will this information be used to predict future outcomes, make inferences on past events, or describe the phenomena taking place?

Model

• Will this information be used to predict future outcomes, make inferences on past events, or describe the phenomena taking place?
  • We are interested in describing the relationship between an athletes performance score and their age and sex, so we can build a generalized linear model and perform parameter inference on the coefficients.

Example Implementation in R

# Fit a linear regression model
model <- lm(score ~ yob + I(yob^2) + sex, data = combined_table)
# Print the summary of the model
summary(model) 

Example Implementation in R

# Fit a linear regression model
model <- lm(score ~ yob + I(yob^2) + sex, data = combined_table)
# Print the summary of the model
summary(model) 

Call:
lm(formula = score ~ yob + I(yob^2) + sex, data = combined_table)

Residuals:
    Min      1Q  Median      3Q     Max 
-165.78  -52.95  -11.81   44.54  422.25 

Coefficients:
              Estimate Std. Error t value Pr(>|t|)    
(Intercept) -1.749e+06  1.989e+05  -8.789   <2e-16 ***
yob          1.757e+03  1.991e+02   8.822   <2e-16 ***
I(yob^2)    -4.411e-01  4.984e-02  -8.850   <2e-16 ***
sexWomen    -3.471e+00  4.860e+00  -0.714    0.475    
---
Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1

Residual standard error: 79.65 on 1140 degrees of freedom
  (32 observations deleted due to missingness)
Multiple R-squared:  0.1694,    Adjusted R-squared:  0.1672 
F-statistic:  77.5 on 3 and 1140 DF,  p-value: < 2.2e-16

We can use the lm function to build a linear regression model to predict the performance score of an athlete as a function of their year of birth and sex. We add a quardratic term \(yob^2\) to the model to account for the non-linear relationship observed between year of birth and performance from the aging curve.

The model summary shows that the model is statistically significant, with an adjusted R-squared value of 0.167. The coefficients for the year of birth and its square term are both statistically significant, which is exactly what we observed visually in the visualization step. Interestingly enough, we see that sex, which is encoded as a dummy variable, is not a statistically significant predictor of performance score.

Communicate

• Who are the stakeholders receiving the insights generated from this analysis?
• What is the most sustainable medium for these insights to be shared?
• How often will these insights be shared?

Communicate

• Who are the stakeholders receiving the insights generated from this analysis?
  • Funding committees, coaches, athletes, and other senior leadership
• What is the most sustainable medium for these insights to be shared?
  • Since there are a wide variety of stakeholders, a web-based static report is a desirable option.
• How often will these insights be shared?
  • These insights will need to be shared weekly.

Possible Dissemination Avenues

For reporting documents, Quarto is highly recommended, as it can be used for many academic and technical report styles that involve results derived from data and has flexible publishing options.

For static reports in particular, publishing a Quarto document using GitHub Pages is particularly effective, since this can be done for free on public repositories (and on private repositories with a GitHub Pro account).

There are numerous avenues by which the insights gleaned from the data science workflow can be disseminated.

Since this analysis is for human report and is a static document, we go with a static report, and use Quarto. Quarto is particularly flexible when it comes to publishing options, and a full list of available publishing services and their corresponding use cases can be found HERE. This article in particular is published using GitHub Pages, which are static websites that are hosted directly from a GitHub repository for free.

Extensions

• Task Automation:

  • We noted earlier that rankings are updated weekly, so we want to leverage some sort of scheduler to automate the process of updates and report generation.

• Data Storage:

  • If the dataset that we are collecting is large, scraping all of it when the document is rendered will be quite time consuming, as opposed to saving it remotely and loading it in at runtime. Can we do this for cheap/free?

We noted earlier that rankings are updated on a weekly basis, and since this report is generated statically, the document will only contain results from the last time that it was rendered. Since this rendering task is something that we would like to do regularly, we want to leverage some sort of scheduler to automate this process.

Extensions

• Task Automation:

  • Example solution: GitHub Actions

• Data Storage:

  • Example solution: GitHub Releases (i.e. Piggyback)

Next Up

• Module 1: Introduction to Data Science
  • R4DS Chapters 9-14
  • P4DS: Chapter 11-17
• Assignment 2: Due May 26 @ 11 AM on Crowdmark