Reverse Geocode using Google API and XML package in R - PART 2

As promised in my previous post this post will dive deeper into understanding how to create links in R and further execute them to generate a list of XML output. In the third and final post we will use this list to filter the data and extract the information we need.

Setup:

If you have RStudio then this is the right time to power it up. In case you are new to R RStudio is an IDE for R. Once we have the RStudio open we can install the package and load the library called XML in R. We need the package for the third part of this series.

Setting up the Link:

In our previous post we discussed the API link that we need to send to google along with the latitude and longitude data to get back the actual address/ cross street of that location. This was refereed as Reverse Geocoding.

The data file which consists of all the stops information along with latitude and longitude data is available on MTA website [1] as well as here.

I have filtered the file downloaded from MTA website as that file consisted of more information. I used the column labelled stop_code  to filter only the stops that had 1. Since these were the exact locations of each stops. You can find the filtered data, that we will use in this exercise,here.

Before we use the data lets just look into paste() function. The simplest way to learn about it is type ?paste in your R Console Window. The paste() function uses just one arguments a list of values to be connected together. Try typing paste paste("Hello","World"). In case you like to separate these two words with a special character we can add the sep argument in the paste function like paste("Hello","World", sep=","). In order to create an API link we need to add the lat and lon data as well as the API key.

paste("https://maps.googleapis.com/maps/api/geocode/xml?","latlng=40.88925,-73.89858","&key=YOUR API KEY",sep="")

the above link will generate the following:

"https://maps.googleapis.com/maps/api/geocode/xml?latlng=40.88925,-73.89858&key=YOUR API KEY"

Now, to extract the XML data from google api service and bring it in RStudio we will use the readLines() function as follows:

link=paste("https://maps.googleapis.com/maps/api/geocode/xml?","latlng=40.88925,-73.89858","&key=YOUR API KEY",sep="")
data = readLines(link)
head(data)

Looping:

One question that arises is that what if we have a csv file with different lat and lon data how could you grab data for all those lat and long. The answer is simple write a loop function in R to generate links and then use the readLines() function to get the XML data in R. Its hard for me to get into the details of writting a loop but a simple google search should help you.

Following is an example of a simple loop which will use the data file and generate a list consisting of 10 values (links in our case).

setwd("C:\\Users\\agohil\\Documents\\Book\\blogposts\\NYC")
sub= read.csv("subway.csv")
sub=na.omit
test=list()
for(i in 1:10){
  test[i]= paste("https://maps.googleapis.com/maps/api/geocode/xml?",
                   "latlng=",sub[i,3],"YOUR API KEY",sep="")
 }

In the above mentioned code we set the the working directory using the setwd() function in R. The data is read in R using the read.csv() function. I usually use the na.omit() function to remove any NA in the data file. In the current data file we do not have any NA. But i like to be sure.

I have created a third column in the data file to make my life a bit easier. The third column uses the values from the first 2 columns separated by a comma. This third column will be used to generate the link. Google requires the latlng argument to be separated by a comma as in
latlng=40.88925,-73.89858

Finally the Loop:

In the loop below we start with an empty list test=list() before we execute the list. This is needed because when we run the loop R needs to store these values some where. In our case it is test.

for(i in 1:10){
  test[i]= paste("https://maps.googleapis.com/maps/api/geocode/xml?",
                   "latlng=",sub[i,3],"YOUR API KEY",sep="")
 }

In the loop above R will substitute values 1 through 10 every time it comes across an i. The sub[i,3] is simply instructing R to go to the  row i and column 3( as in sub[1,3], [2,3]...,[10,3]) and substitute the value in the paste function.

The whole process can easily be extended to all the rows in any file by simply using the following 

for(i in 1:494){
  test[i]= paste("https://maps.googleapis.com/maps/api/geocode/xml?",
                   "latlng=",sub[i,3],"YOUR API KEY",sep="")
 }

Conclusion:

The main objective of writing this post was to introduce new users of R to Looping in R, help them understand and use the paste() and readLines() function. In the next post we will use the extracted XML package in R to filter the list and extract the data we need.

Writing a simple Function in R

Introduction: This is post aims at introducing new users of R on how to write a function in R and execute the same to get reasonable output. We would write a very small function that calculates a t statistics to test equality of means. Hence, I would first introduce the concept under section 1 and implement the R code in section 2 and finally, execute the .R function file in section 3.

Section 1: Equality of means: If users have taken basic statistics, it would be hard for me to believe that the instructor diod not teach hypothesis testing for equality of means. But in order to understand how the function in R can be coded it is very essential for the user to know the concept.

We are interested in testing a simple hypothesis of equality of means i.e. mean of population X equals mean of population Y. Often economists are interested in knowing the impact of certain elements on a population against the section of the population that was exposed to certain different element.Some of the examples bellow discuss this

1)Average test score of GMAT of students who graduated with a math degree against ones who graduated with a degree in literature. 2)Average yield of a crop due to use of a certain fertilizers against the yield of the crop of farms that used a different fertilizer.

For each population we would like to test if the mean of population 1(MU1) equals mean of the populaton2(MU2). Since it is hard to estimate the true population mean and standard deviation we would use the sample mean and standard deviation to estimate the same.

Step 1: Null hypothesis : H0 : Mu1 = Mu2 Alternate Hypothesis : H1 : Mu1 ??? Mu2

Step2 : calculate the T stat using the following set of formulae:

Step 3: Compare the calculated T with the t from the table using the above mentioned equation and reject the null hypothesis if the following equality holds true.

Section 2: In order to create a function , users need to open a new R script available under the File drop down, located under the menubar. The teqmu1 function is written in R. Teqmu1 is the name of the function. This function takes vector x and vector y as inputs and returns the T statistic.

The first line of the function should define the name of the function followed by the input variables. The R command length is used in the function to calculate the length of the x and y vector which will then be used in the formula for the T statistic. We have also made use of R commands SD , which calculates the standard deviation and Mean which calculates the mean of the vectors X and Y.

teqmu1 <- function(x, y) {
    m = length(x)
    n = length(y)
    sp = sqrt(((m - 1) * sd(x)^2 + (n - 1) * sd(y)^2)/(m + n - 2))
    t = (mean(x) - mean(y))/(sp * sqrt(1/m + 1/n))
    return(t)

}

Section 3:

In order to execute the command we have to save the file as an teqmu1.r file. Note that the name of the file should match the name of the function( in this case the filename would be teqmu.r)

in order to run this function users have to souce it first. In order to source the function click on the code drop down available on R menu bar -> source the file and select the function file.

Now to execute the function create two random vectors in R:

data1 = c(1, 4, 3, 6, 5)  # vector for X
data2 = c(5, 4, 7, 6, 10)  # vector for Y
teqmu1(data1, data2)  # executing the function

## [1] -1.938

The last step is to call the function, once you get the T statistics you can look up the t value from the table using the equation under Step 3. The researcher now can decide to reject the null or not based on the step 3 formula.