Analyzing data using Boxplots

<!-- Styles for R syntax highlighter

Box Plots

A lot of economic data, in my opinion, can be better understood using the Boxplots rather than bar charts. Box plots have a number of uses when the research is in its intial stages. A researcher after collecting the data can plot this data using boxplots to get an overview. An analyst can plot a number of different types of boxplots and infer a great deal of information. Boxplots are a quick and easy way to compare multiple data series. Usually , economist use scatter plots, qqplots and histogram to prove the Skewness or normality of data and also further to detect outliers in their data. Boxplot instead of using the mean and standard deviation, to show the center of the distribution  and its spread,uses median and interquantile range.
My main purpose in this post is to introduce users of R to boxplots and take a step further to analyse data using various boxplots commands.

Data

The data for the boxplot is borrowed from Jeffrey Wooldridge book. The data comprises of 526 observations and 24 variables.

Interpretation:

The most insightful way to understand a boxplot is to plot one. Following are some of the commands that reads the data and then plots the graph.

wg = read.csv("dataw.csv")  ## read in the raw data file
summary(wg$wage)  ## summary statistics of the wage column

##    Min. 1st Qu.  Median    Mean 3rd Qu.    Max. 
##    0.53    3.33    4.65    5.90    6.88   25.00

boxplot(wg$wage)  ## plot the wage data

As discussed above the median and not the mean is the center of the distribution for boxplots, hencethe thick black line in the box shows the median(4.650). Further the box itself is approximately an interquantile range.Lastly, all the points above and below the fence are outliers.This shows us that the data is skewed and not symmetric.
Multiple boxplots can be plotted using the follwing r command.

boxplot(boxplot(wg$wage ~ as.factor(wg$female)))  ## Multiple boxplots

## Error: non-numeric argument to binary operator

In the data set suppose we want to study the distribution of wage between males vs females. The variable female in the data set is a dummy variable which assumes the value 1 if female and 0 if male.One can infer that on an average males are paid a higher wage than females. We can beautify the graph by adding colors and labels to the plot.

boxplot(wg$wage ~ as.factor(wg$female), col = ("blue"), xlab = "sex", ylab = "wage (average hourly earnings in $)")

now we can add more information to the graph by differentiating between males and female by color.

boxplot(wg$wage ~ as.factor(wg$female), col = c("blue", "red"), names = c("Male", 
    "Femle"), xlab = "sex", ylab = "wage (average hourly earnings in $)")

In the above image you would observe that i have defined “col” as a vector of two colors and have also included “names” command also as a vector. In my next post i will try to include more information to this boxplot and also introduce users to plot multiple plots and combine plots using basic R package.

Conclusion

We have studied data using boxplots and come to a conclusion that on an average females are paid lower wages. Also, we can observe the outlier in the data sets.
-->

Writing a simple Function in R

Introduction: This is post aims at introducing new users of R on how to write a function in R and execute the same to get reasonable output. We would write a very small function that calculates a t statistics to test equality of means. Hence, I would first introduce the concept under section 1 and implement the R code in section 2 and finally, execute the .R function file in section 3.

Section 1: Equality of means: If users have taken basic statistics, it would be hard for me to believe that the instructor diod not teach hypothesis testing for equality of means. But in order to understand how the function in R can be coded it is very essential for the user to know the concept.

We are interested in testing a simple hypothesis of equality of means i.e. mean of population X equals mean of population Y. Often economists are interested in knowing the impact of certain elements on a population against the section of the population that was exposed to certain different element.Some of the examples bellow discuss this

1)Average test score of GMAT of students who graduated with a math degree against ones who graduated with a degree in literature. 2)Average yield of a crop due to use of a certain fertilizers against the yield of the crop of farms that used a different fertilizer.

For each population we would like to test if the mean of population 1(MU1) equals mean of the populaton2(MU2). Since it is hard to estimate the true population mean and standard deviation we would use the sample mean and standard deviation to estimate the same.

Step 1: Null hypothesis : H0 : Mu1 = Mu2 Alternate Hypothesis : H1 : Mu1 ??? Mu2

Step2 : calculate the T stat using the following set of formulae:

Step 3: Compare the calculated T with the t from the table using the above mentioned equation and reject the null hypothesis if the following equality holds true.

Section 2: In order to create a function , users need to open a new R script available under the File drop down, located under the menubar. The teqmu1 function is written in R. Teqmu1 is the name of the function. This function takes vector x and vector y as inputs and returns the T statistic.

The first line of the function should define the name of the function followed by the input variables. The R command length is used in the function to calculate the length of the x and y vector which will then be used in the formula for the T statistic. We have also made use of R commands SD , which calculates the standard deviation and Mean which calculates the mean of the vectors X and Y.

teqmu1 <- function(x, y) {
    m = length(x)
    n = length(y)
    sp = sqrt(((m - 1) * sd(x)^2 + (n - 1) * sd(y)^2)/(m + n - 2))
    t = (mean(x) - mean(y))/(sp * sqrt(1/m + 1/n))
    return(t)

}

Section 3:

In order to execute the command we have to save the file as an teqmu1.r file. Note that the name of the file should match the name of the function( in this case the filename would be teqmu.r)

in order to run this function users have to souce it first. In order to source the function click on the code drop down available on R menu bar -> source the file and select the function file.

Now to execute the function create two random vectors in R:

data1 = c(1, 4, 3, 6, 5)  # vector for X
data2 = c(5, 4, 7, 6, 10)  # vector for Y
teqmu1(data1, data2)  # executing the function

## [1] -1.938

The last step is to call the function, once you get the T statistics you can look up the t value from the table using the equation under Step 3. The researcher now can decide to reject the null or not based on the step 3 formula.

Scatterplot in R

<!-- Styles for R syntax highlighter

Scatter plots in R

Abstract: The main purpose of this page is to learn to plot in R. This document also explains plotting scatter plot in R and how it can be used to visualize and interpret the data.
Introduction: scatter plots are widely used in economics and finance to get a basic idea of the underlying datset.
Data:
The dataset used is house price dataset available alongwith an undergarduate textbook “Introductiory Econometrics” by Jeffrey M. Wooldridge. The data consists of 506 observations.The dataset consist of the following variables:

1. price- median housing price, $
2. crime- crimes committed per capita
3. nox - nitrous oxide, parts per 100 mill.
4. rooms - avg number of rooms per house
5. dist - weighted dist. to 5 employ centers
6. radial - accessibiliy index to radial hghwys
7. proptax - property tax per $1000
8. stratio - average student-teacher ratio
9. lowstat - % of people 'lower status'
   
   1. lprice- log(price)
   2. lnox - log(nox)
   3. lproptax - log(proptax)
   for the purpose of this analysis we would only utilize price and crime.
   R code: Since the data was available in the Raw text format i copy pasted the data in Excel and saved it as a CSV file under my R directory. Then i use the read.csv command to read in the csv file and save the dataset in hprice. I have also made use of the colnames command to get additional information of the colnames and the summary command to breifly look at the center and distribution of X and Y variables.

hprice <- read.csv("hprice.csv", header = TRUE, sep = ",")
colnames(hprice)

##  [1] "price"    "crime"    "nox"      "rooms"    "dist"     "radial"  
##  [7] "proptax"  "stratio"  "lowstat"  "lprice"   "lnox"     "lproptax"

summary(hprice$price)

##    Min. 1st Qu.  Median    Mean 3rd Qu.    Max. 
##    5000   16800   21200   22500   25000   50000

summary(hprice$crime)

##    Min. 1st Qu.  Median    Mean 3rd Qu.    Max. 
##    0.01    0.08    0.26    3.61    3.68   89.00

The following line of commands will plot a scatterplot in R.The scatter plot generated will have circles which are not filled up. However, to fill up the circles the user needs to use the “pch” command in R.

plot(hprice$price, hprice$crime, col = "BLUE")

plot - basic plot command in R hprice$price - X variable in Plot hprice$crime - Y variable in plot command col - color

plot(hprice$price, hprice$crime, pch = 19, col = "BLUE")

You would observe that the points are too close to each other and so an additional command “cex” can be added. cex command will help in reducing the size of the circles.

plot(hprice$price, hprice$crime, pch = 19, cex = 0.5, col = "BLUE")

The image above provides much more information about the relationship between x and Y variables. We can observe that as the home prices rise the crime rate significantly drop. It is not hard to reason with this, as people would not like to live in areas where the crime rates are high and hence the supply of houses are greater than the demand resulting in lower prices. On the other hand demand for houses will be much higher when the crime rates are low as people are willing to spend extra for their security and safety.
-->

Links to Dive into -> 02/27/2013

1. I read in reader digest an interesting note on how organisms are named after famous people. Wiki has a list of all such organisms found here . The beetle named after Hitler or the spider named after Bram Stoker (author of Dracula) or a moth from Tibet named after Dalailama are some of the few i found interesting.

2. This is an interesting article on how social  media can help evaluate chronic disease. The data collected by corporations are secured due to piracy laws but using social media a great deal of awareness can be created.

3. Page 40 and 41 of the document is a short but interesting article on long term and short term determinants of  sovereign yields. A lot of interesting article can be found in JSTOR but this document is a good place to start.

Normal VS Lognormal Distribution using R

Objective:

The main aim of this topic is to study and observe the difference between the normal distribution and lognormal distribution using R commands. In financial time series analysis it is more appropriate to use the lognormal distribution than the normal distribution.

Introduction:

The moments of a normal distribution and lognormal distribution are not exactly the same and hence we observe a shift in log normal from a normal distribution.

The above formulae and further characteristics are also available on wikipedia using the following link - http://en.wikipedia.org/wiki/Log-normal_distribution.The wiki article also elaborates on the PDF and CDF which are covered in brief in this article. The mean of the distribution shifts by 0.5*variance, hence you can observe that the graph would shift with shift in variance from .5 to 1. This is clear from the image below.

R functionality:

The above plot is created in R using the following commands. Also, in R the plot may get cut while plotting. The best way to plot all the images correctly is plot the image with the maximum y value first using the PLOT command and subsequent images using the lines command in R.

 x<- seq(0,3,length = 100)

 plot(x, dlnorm(x,0, 0.25), type= "l", col ="blue")

 lines(x,dlnorm(x,0,0.5), type ="l", col = "red")

 lines(x,dlnorm(x,0,1), type ="l", col = "green")

dlnorm(n, mean, standard deviation) = computes the density of a lognormal distribution

type ="l" = allows r to plot a line graph

col = is used to color the line

lwd = 2   = can also be used in the plot to increase the width of the plot

Cumulative distribution function of a log normal in R:

x<- seq(0,3,length = 100)

plot(x, plnorm(x,0, 0.25), type ="l", col ="blue", lwd = 2)

lines(x,plnorm(x,0,1), type ="l", col = "green")

lines(x,plnorm(x,0,0.5), type ="l", col = "red")

In order to calculate the CDF of a log normal simply use the command PLNORM in r.

PLNORM(n, mean, standard dev.) = computes CDF for a log normal distribution.

RLNORM(n, mean , standard dev) = generates a data set with a given mean and Standard deviation.

qlnorm(p, mean, standard deviation) = can be used to generate p quantiles of the log normal distribution 

Normal Distribution using simple R command

Introduction:

The main objective of this study is to learn to plot the cumulative distribution function, quantiles and probability mass function using simple R commands.

R Commands:

·         seq – is used to generate a sequence of values.

·         pnorm – is used to calculate the cumulative distribution function.

·         dnorm – is used to calculate the probability mass function

·         qnorm- is used to generate quantiles or inverse of the normal distribution.

R commands and images:

Using Pnorm:

·         x<- seq(-5,5, length = 250)

·         plot(x,pnorm(x), type ="l", lwd = 2, col ="blue", xlab =" x values", ylab = "CDF")

Using dnorm:

·         x<- seq(-5,5, length = 250)

·         plot(x,dnorm(x), type ="l", lwd = 2, col ="blue", xlab =" x values", ylab = "PDF")

The primary goal is to understand the commands in R that help in creating basic statistical images. Readers can use Wikipedia in order to learn more about normal distribution and its usefulness in statistics and economics.

b