Back to: Introduction to R
We will focus on working with our data more. At this point we will consider how we pick the rows of the data that we wish to work with. If you consider many modern data sets, we have so much information that we may not want to bring it all in at once.
We have not discussed exactly how R works at this point, however R brings data into the RAM of your computer. This means you can be limited for what size data you can bring in at once. Very rarely do you need the entire data set. We will focus on how to pick the rows or observations we want now.
Enter the filter() Function
The filter() function chooses rows that meet a specific criteria. We can do this with Base R functions or with dplyr`.
Let’s say that we want to look at the flights data but we are only interested in the data from the first day of the year. We could do this without learning a new command and use indexing which we learned in dataframes:
flights[flights$month==1 & flights$day==1, ]## # A tibble: 842 × 19
## year month day dep_time sched_dep_time dep_delay arr_time
## <int> <int> <int> <int> <int> <dbl> <int>
## 1 2013 1 1 517 515 2 830
## 2 2013 1 1 533 529 4 850
## 3 2013 1 1 542 540 2 923
## 4 2013 1 1 544 545 -1 1004
## 5 2013 1 1 554 600 -6 812
## 6 2013 1 1 554 558 -4 740
## 7 2013 1 1 555 600 -5 913
## 8 2013 1 1 557 600 -3 709
## 9 2013 1 1 557 600 -3 838
## 10 2013 1 1 558 600 -2 753
## # ... with 832 more rows, and 12 more variables: sched_arr_time <int>,
## # arr_delay <dbl>, carrier <chr>, flight <int>, tailnum <chr>,
## # origin <chr>, dest <chr>, air_time <dbl>, distance <dbl>, hour <dbl>,
## # minute <dbl>, time_hour <dttm>Now this is not very difficult to do, what we have is that we are working with flights and we only want to keep the rows of data there month==1 and day==1. However we could use the filter() function to do this in a much easier to read format:
filter(.data, ...)where
.data is a tibble.
... is a set of arguments the data you want returned needs to meet.
This means in our example we could perform the following:
filter(flights, month==1, day==1)## # A tibble: 842 × 19
## year month day dep_time sched_dep_time dep_delay arr_time
## <int> <int> <int> <int> <int> <dbl> <int>
## 1 2013 1 1 517 515 2 830
## 2 2013 1 1 533 529 4 850
## 3 2013 1 1 542 540 2 923
## 4 2013 1 1 544 545 -1 1004
## 5 2013 1 1 554 600 -6 812
## 6 2013 1 1 554 558 -4 740
## 7 2013 1 1 555 600 -5 913
## 8 2013 1 1 557 600 -3 709
## 9 2013 1 1 557 600 -3 838
## 10 2013 1 1 558 600 -2 753
## # ... with 832 more rows, and 12 more variables: sched_arr_time <int>,
## # arr_delay <dbl>, carrier <chr>, flight <int>, tailnum <chr>,
## # origin <chr>, dest <chr>, air_time <dbl>, distance <dbl>, hour <dbl>,
## # minute <dbl>, time_hour <dttm>We could instead do the same command with piping to make it even easier to read:
flights %>%
filter(month==1, day==1)Finally we could also only do one filtering at a time and chain it:
flights %>%
filter(month==1) %>%
filter(day==1)These will all lead to the same output. Make sure you verify this on your own screen.
Further Filtering
filter() supports the use of multiple conditions where we can use Boolean. For example if we wanted to consider only flights that depart between 0600 and 0605 we could do the following:
flights %>% filter(dep_time >= 600, dep_time <= 605)## # A tibble: 2,460 × 19
## year month day dep_time sched_dep_time dep_delay arr_time
## <int> <int> <int> <int> <int> <dbl> <int>
## 1 2013 1 1 600 600 0 851
## 2 2013 1 1 600 600 0 837
## 3 2013 1 1 601 600 1 844
## 4 2013 1 1 602 610 -8 812
## 5 2013 1 1 602 605 -3 821
## 6 2013 1 2 600 600 0 814
## 7 2013 1 2 600 605 -5 751
## 8 2013 1 2 600 600 0 819
## 9 2013 1 2 600 600 0 846
## 10 2013 1 2 600 600 0 737
## # ... with 2,450 more rows, and 12 more variables: sched_arr_time <int>,
## # arr_delay <dbl>, carrier <chr>, flight <int>, tailnum <chr>,
## # origin <chr>, dest <chr>, air_time <dbl>, distance <dbl>, hour <dbl>,
## # minute <dbl>, time_hour <dttm>We would then get 2,460 different flights. Another way to accomplish this would be to use what is called a helper function. We can use between() which lets us pick specific numeric ranges:
flights %>%
filter(between(dep_time,600,605))We can also use the filter() function to remove missing data for us. Previously we learned about a class of functions called is.foo() where foo represents a data type. We could choose to only use flights that have a departure time. That means we wish to not have missing data for departure time:
flights %>% filter(!is.na(dep_time))## # A tibble: 328,521 × 19
## year month day dep_time sched_dep_time dep_delay arr_time
## <int> <int> <int> <int> <int> <dbl> <int>
## 1 2013 1 1 517 515 2 830
## 2 2013 1 1 533 529 4 850
## 3 2013 1 1 542 540 2 923
## 4 2013 1 1 544 545 -1 1004
## 5 2013 1 1 554 600 -6 812
## 6 2013 1 1 554 558 -4 740
## 7 2013 1 1 555 600 -5 913
## 8 2013 1 1 557 600 -3 709
## 9 2013 1 1 557 600 -3 838
## 10 2013 1 1 558 600 -2 753
## # ... with 328,511 more rows, and 12 more variables: sched_arr_time <int>,
## # arr_delay <dbl>, carrier <chr>, flight <int>, tailnum <chr>,
## # origin <chr>, dest <chr>, air_time <dbl>, distance <dbl>, hour <dbl>,
## # minute <dbl>, time_hour <dttm>On Your Own: RStudio Practice
Using the filter() function and chaining:
Choose only rows associated with
United Airlines (UA)
American Airlines (AA)
Your end result should be:
## # A tibble: 91,394 × 19
## year month day dep_time sched_dep_time dep_delay arr_time
## <int> <int> <int> <int> <int> <dbl> <int>
## 1 2013 1 1 517 515 2 830
## 2 2013 1 1 533 529 4 850
## 3 2013 1 1 542 540 2 923
## 4 2013 1 1 554 558 -4 740
## 5 2013 1 1 558 600 -2 753
## 6 2013 1 1 558 600 -2 924
## 7 2013 1 1 558 600 -2 923
## 8 2013 1 1 559 600 -1 941
## 9 2013 1 1 559 600 -1 854
## 10 2013 1 1 606 610 -4 858
## # ... with 91,384 more rows, and 12 more variables: sched_arr_time <int>,
## # arr_delay <dbl>, carrier <chr>, flight <int>, tailnum <chr>,
## # origin <chr>, dest <chr>, air_time <dbl>, distance <dbl>, hour <dbl>,
## # minute <dbl>, time_hour <dttm>