Links to Dive into -> 02/27/2013

1. I read in reader digest an interesting note on how organisms are named after famous people. Wiki has a list of all such organisms found here . The beetle named after Hitler or the spider named after Bram Stoker (author of Dracula) or a moth from Tibet named after Dalailama are some of the few i found interesting.

2. This is an interesting article on how social  media can help evaluate chronic disease. The data collected by corporations are secured due to piracy laws but using social media a great deal of awareness can be created.

3. Page 40 and 41 of the document is a short but interesting article on long term and short term determinants of  sovereign yields. A lot of interesting article can be found in JSTOR but this document is a good place to start.

Normal VS Lognormal Distribution using R

Objective:

The main aim of this topic is to study and observe the difference between the normal distribution and lognormal distribution using R commands. In financial time series analysis it is more appropriate to use the lognormal distribution than the normal distribution.

Introduction:

The moments of a normal distribution and lognormal distribution are not exactly the same and hence we observe a shift in log normal from a normal distribution.

The above formulae and further characteristics are also available on wikipedia using the following link - http://en.wikipedia.org/wiki/Log-normal_distribution.The wiki article also elaborates on the PDF and CDF which are covered in brief in this article. The mean of the distribution shifts by 0.5*variance, hence you can observe that the graph would shift with shift in variance from .5 to 1. This is clear from the image below.

R functionality:

The above plot is created in R using the following commands. Also, in R the plot may get cut while plotting. The best way to plot all the images correctly is plot the image with the maximum y value first using the PLOT command and subsequent images using the lines command in R.

 x<- seq(0,3,length = 100)

 plot(x, dlnorm(x,0, 0.25), type= "l", col ="blue")

 lines(x,dlnorm(x,0,0.5), type ="l", col = "red")

 lines(x,dlnorm(x,0,1), type ="l", col = "green")

dlnorm(n, mean, standard deviation) = computes the density of a lognormal distribution

type ="l" = allows r to plot a line graph

col = is used to color the line

lwd = 2   = can also be used in the plot to increase the width of the plot

Cumulative distribution function of a log normal in R:

x<- seq(0,3,length = 100)

plot(x, plnorm(x,0, 0.25), type ="l", col ="blue", lwd = 2)

lines(x,plnorm(x,0,1), type ="l", col = "green")

lines(x,plnorm(x,0,0.5), type ="l", col = "red")

In order to calculate the CDF of a log normal simply use the command PLNORM in r.

PLNORM(n, mean, standard dev.) = computes CDF for a log normal distribution.

RLNORM(n, mean , standard dev) = generates a data set with a given mean and Standard deviation.

qlnorm(p, mean, standard deviation) = can be used to generate p quantiles of the log normal distribution 

Normal Distribution using simple R command

Introduction:

The main objective of this study is to learn to plot the cumulative distribution function, quantiles and probability mass function using simple R commands.

R Commands:

·         seq – is used to generate a sequence of values.

·         pnorm – is used to calculate the cumulative distribution function.

·         dnorm – is used to calculate the probability mass function

·         qnorm- is used to generate quantiles or inverse of the normal distribution.

R commands and images:

Using Pnorm:

·         x<- seq(-5,5, length = 250)

·         plot(x,pnorm(x), type ="l", lwd = 2, col ="blue", xlab =" x values", ylab = "CDF")

Using dnorm:

·         x<- seq(-5,5, length = 250)

·         plot(x,dnorm(x), type ="l", lwd = 2, col ="blue", xlab =" x values", ylab = "PDF")

The primary goal is to understand the commands in R that help in creating basic statistical images. Readers can use Wikipedia in order to learn more about normal distribution and its usefulness in statistics and economics.

b 

Data Visualization through QQ plots and Normal Distribution:

Abstract

This article is an introduction to techniques and usefulness of QQ plots in Excel, Matlab and R. The main objective is to highlight the importance of using qq plots to convey meaningful analysis.

Introduction:

 QQ plots in statistics refer to quantile plots. They have been widely used in data analysis for comparing probability distributions. Statisticians use qq plots for testing if the error terms in Ordinary least square estimation are normally distributed. Equity analysts have used qq plots for determining if the distribution of returns is normal and further to detect fat tails. Application of QQ plots is not just limited to the fields of finance, economics and statistics but it has also been widely applied in the fields of science and engineering. 

The following read is divided into 4 sections. The first section introduces the users to plotting a normal curve in excel as well as the qq plots. The second section introduces the users to code qq plot in R. The third section applies the data and performs the plotting function using Matlab. The primary objective is to learn on various methods to visualize data.

QQ plots in Excel: 

Step 1: Open a new excel sheet and enter 6 different headings – Rank, Probability, Z score(theory), Data, standardized , zscore exp. I will discuss these headings as we start filling up each column with either data or formula.

Step 2: Under the data column (D2) enter the raw data that you would like to test for normality. If you do not have a standard data you can always use the =rand() command in excel to generate a data set for testing purpose. Please note that once you generate random data set copy and paste special values in the data column, or else every time you hit ENTER the data will change.

 Step 3: under column standardized(cell E2) we standardize the raw data. In Excel use the following formula : =STANDARDIZE(D2,AVERAGE($D$2:$D$43),STDEV($D$2:$D$43))

Step 4: once you standardize the data , copy the data -> column titled zscore exp -> paste special the data with values. Then sort the data in ascending order.

Step 5: Go to the Rank column and use the following formula =RANK(F2,$F$2:$F$43,1) This would rank your data.

Step 6: Go to the column probability and enter the following formula:

 =(A2-0.5)/COUNT($F$2:$F$43)

Since we have standardized the data , area under the curve is 1.

Step 7: Finally , go to column titled zscore and enter the following: =NORMSINV (B2)

Now plot a scatter diagram using the zscores under step 6 as X axis values and zscore under step 4 on y axis. You should see a plot that looks like a S curve. This is nothing but what statisticians call a QQ plot. Q meaning Quantile plot. Individuals can also plot a regression line through the plot and this would visually tell you how good is the fit. An r^2 of 1 would mean data is normally distributed. I have plotted different data sets here to show the users varying degrees of convergence and divergence from normality.

In Excel the easiest way to plot a regression line is by clicking on the image and then going to the Design ribbon on the top and clicking on 9th chart design. Your plot should look like the image bellow:

I have used two data sets alongwith excel function to show how it is done in excel.Please use the link below:
https://docs.google.com/open?id=0Bw3VZ9z5dDlPQlJlT0FwNHRZdVk

In the sheet titles QQplot i have plotted a random data set using the rand function, since excel generates this data set with mean 0.5, the qqplot will have a high rsquare value and regression line has an amazingly great fit. This confirms that the data set has a normal distribution. Also, i have plotted another qq plot for a dataset generated by me, and this dataset is plotted in red. The qqplot of this data deviates from normal and has a low rsquare value.

users can also use the NORMDIST function in excel to generate a probability mass function of a normal distribution. I have made use of this function under the sheet titled Normal. If you observe the scatter plot, the random data has a bell shape (red) but the data  generated by me has a positive skew and deviates from normal(blue)

Women Participation in GDP

Over past week I came across an interesting report published by Booz & Company titled "Empowering the Third Billion Women".  The following graph shows the increase in GDP if women participation in the labor force increases.

 In the graph lower contribution would imply that women are  participating in labor force and contributing to the national income. These are basically countries which are developed (Sweden, German, USA, U.K.  etc). The curious case is that of  Tanzania as it places itself among these developed economies . The main reason behind this is that women participation in agricultural sector is higher, but they are underpaid . On the other hand India has very low women participation , pointed out by the report, as they do not reach their full potential due to discrimination at work place and cultural reasons.