Assignment 3

Due: Monday, June 2 at 11 AM ET to be submitted via Crowdmark

Note: Your submitted solution must be unique and written in your own words; if you collaborate with other students, external resources, and/or use AI tools, this must be clearly acknowledged in your submission.

Part 1

This assignment covers material from Chapters 15-21 of R4DS / Chapters 18-24 of P4DS. For P4DS readers, please also read the merging documentation from pandas. Please complete the ALL of following exercises.

First, acquire the following data, which can be read directly from the url using read_csv.

Questions

  1. What word in words has the most vowels? Which word has the highest proportion of vowels? (Hint: what is the denominator?)

  2. From words create regular expressions that find all words that:

    1. Start with “y”.

    2. Don’t start with “y”.

    3. End with “x”.

    4. Are exactly three letters long.

    5. Have seven letters or more.

    6. Contain a vowel-consonant pair.

    7. Contain at least two vowel-consonant pairs in a row.

    8. Only consist of repeated vowel-consonant pairs.

  3. Switch the first and last letters of all of the words in words. Which of those strings are still found in words?

  4. Using the gss_cat dataset, explore the distribution of rincome (reported income). What makes the default bar chart hard to understand? How could you improve the plot?

  5. Categorize partyid into {democrat, republican, independent, other}. How have the proportions of people identifying as Democrat, Republican, and Independent changed over time?

  6. Parse the following date-times:

    d1 = "January 1, 2010"
d2 = "2015-Mar-07"
d3 = "06-Jun-2017"
d4 = "August 19 (2015)"
d5 = "12/30/14" # Dec 30, 2014
t1 = "1705"
t2 = "11:15:10.12 PM"

  7. Revisit the flights dataset.

    1. How does the distribution of flight times within a day change over the course of the year?
    2. On what day of the week should you leave if you want to minimise the chance of a delay?
    3. Use joins to find all flights to the 10 most popular destinations in flights.
    4. Find the 48 hours (over the course of the whole year) that have the worst delays. Cross-reference it with the weather data. Can you see any patterns?

  8. What is dplyr::distinct() / pd.drop_duplicates() translated to in SQL? What about head()?

Part 2

Using a sports dataset from a library/package of your choice, perform the following steps of the data science workflow:

  1. Tidy
    1. Is your data tidy? If so, please briefly describe why, and if not, please describe the steps that need to be taken to make it tidy, such as separating columns that combine multiple variables or modifying column names.
    2. Identify whether your data is in “long” format or “wide” format. Using a subset of the columns from your data, convert your dataset from one format to the other.
      • For example, if you have play-by-play data in a long format, you can aggregate the values in a column by quarter/period/half and then make it wider.
  2. Transform.
    1. Use regular expressions and string manipulation to convert the surname of each athlete to all capital letters.
    2. If applicable, perform appropriate date-time manipulation to have such variables in ISO 8601 format.
    3. If applicable, identify missing values in the dataset and interpret why these values may be missing.
    4. Use logical conditions (boolean variables) to filter rows based on a specific criterion.