An introduction to R statistics

Recently, I spent some time learning the R environment. It take me a little while to Get-It. So I would like to describe the system in my way and hope it will help those brain that are wired like mine. There is no intention in covering the detail of R, but the basics.

The R environment data are objects that can have properties. But these objects do not have method. So, we can say they are more like a C structure than full featured objects. Also, R does not support the Object.Property or Object.method() syntax. Instead, the dot (.) is an allowable character for identifiers. Properties and methods are accessed through functions. So, the bottom line is R objects are like C structures. With this approach in mind, we can better understand the limitation of R and how it is constructed.

With this approach to objects, functions can be made to operate on multiple type of objects by knowing the type of the object. In R, properties for basic types are documented. The intrinsic properties are: mode, length and class. These properties can only be accessed through special functions: mode(), length() and class(). Other properties/attributes are accessed through attr(). The list of attributes can be viewed with attributes().

R supports the syntax of vector operation. For example, A*B can mean the multiplication of two vectors. This approach makes R an idea tool for expressing matrix operations and carrying out operations related to tabulated data, like those in the linear algebra and statistic survey.

Basic Data Type: Vector
The simplest R object type is the vector, which is an ordered list of components of the same kind. Component can be numeric, complex, character, logical, NA and others. Vectors have the mode property, where mode can be numeric, complex, character, logical and others. You use the mode() function to obtain access to the mode property. The other property of vectors is the length, or the number of components, and it can be accessed through the length() function. The names property is also supported. names property is a vector itself and gives each component a name. Components can be referred to by either integer indexes or their names.

Basic Data Type: Factor
Factor is an vector object with the levels property. Property levels is a vector of unique values of the original vector. This give those values an order.

R objects also have a property called class. The class of a vector is simply its mode. The class of a Factor is 'factor'. The class property can be accessed via class() function.

Basic Data Type: Array
Array is an vector object with dim property. The dim is a positive integer vector. The component of the dim vector specifies the size of each dimension. Matrix is an array of two dimensions. The mode of an array is the same as the mode of its component. The class of an array is 'array'. Array can be created by combining vectors or by setting a vector's dim property.

Basic Data Type: List
By combing objects of different type in an ordered list, we created a list object. List object can have a names property that is a vector of mode character and it gives each list-component a name. List objects have the class property set to 'list'.

Basic Data Type: data.frame
data.frame object is considered as an extension of list object with restrictions placed on the size of the list-component so that the data.frame resemble a table like structure with each column has the same number of values. These list-components can be vectors, factors, matrix or lists. data.frame have the class of 'data.frame'.

Useful Function/Operators
Environmental
getwd(), setwd(), objects(), ls(), library()

Constructors
Bgn:End (colon), c(), vector(), factor(), list(), data.frame(), matrix(), cbind(), rbind(), matrix()

Casting
as.vector(), as.factor() ...

Indexing

• [] return the same type
• Vctr[ NdxVctr ], Vctr[ NmsVctr ], Vctr[ LgcVctr ] ...
• Mtrx[ NdxVctr1 ][ NdxVctr2 ] ...
• [[ Ndx ]] == $ return the component.
  

[], [[]], $

Other
assign(), <-, ->, grep(), function(), tapply(), is.na(), is.vector(), summary(), names()
& (element-wise), | (element-wise), &&, ||

Control Structure

• if (Exp1) Exp2 else Exp3
• ifelse( LgcVctr, TrVctr, FlsVctr) return a vector with components from TrVctr and FlsVctr based on LgcVctr
• for ( Ndx in Vctr) { Expr... }
  
• while (Cndtn) { Expr ... }
• break
• Vrbl <- function (Arg1, Arg2, ...) { Expr ... }

The above provided enough info for the basic understanding of the R. For detail, please visit the R-Intro and the R-Reference.pdf.

News feeds to China public

According to the news, Google could retreated from China as soon as the end of March, 2010. Recently, I read a news article that I feel it really worth reading for all the Chinese people including people lived in Taiwan and I begin to think how can I help feed news to Chinese lived inside China's Great Firewall.

I am sorry to say that "Bill Gates, I do not agree with you on that there are a lot of (technical) ways that Chinese people can reach the outside world". I believe they need help. One thing I hope can be done is to defeat the filtering of search result. Here are some of my idea.

1. Create sites with China-Communist Party permissible articles so that it can be indexed. We will then provide CAPTCHA and allow users to view the unfiltered result - possibly from Google.
2. Encourage everyone that care about this issue to setup various web sites so that it become extremely expensive for China-Communist to filter all these sites.

Comments welcome. I believe there are a lot of smart people that can provide even better ideas like dynamically register new sites from time to time.

Youtube Flash Video download with Firefox

Firefox add-on: Download Flash and Video is a very nice add-on.

Once you installed the add-on, you will see a downward arrow on the lower-right side of the browser window. If you are on a web page, clicking on the arrow will show a list of flash sources on the page. If you mouse over an item, the source on the web page will be highlighted. You can save the .swf file directly or you can start the video and re-click the arrow to download the .flv files.

Once you have the video saved, you will need a player to view the video.

Census DataFerrett Install on other drive and Vista

On Other Drive
Census' stand-alone desktop DataFerrett can be downloaded from here. The current beta version for Windows (as of Feb. 8, 2010), when install, will automatically installed to C:\Program Files\.

Since my hard drive C: is almost full, I try to run the program from other drive. What I found out is that if you kept just the image and the messages sub-directory on C: and move everything else to other drive, the program will work fine.

Vista
Problem:
When I download the program on Feb. 11, 2010 and installed on my Vista Home C: drive. I also run into problems. Every time I start the program, it prompt to upgrade to version 1.03.05_B 2009-05-15 from version 1.03.04_B 2006???. I follow the prompt and program reported completion of the upgrade and asked me to click OK and to restart the program. But when I start the program, it again prompt me to upgrade to the same version.

Finally, I rename the file version.txt and start the program. The program reports error but it does send me to the normal starting window. I run into some problem when I try to download data and create table. But I will have to check more carefully to see if it is caused by my other actions.

Solution:
Well. On Vista, you need to right click the DataFerrett.exe and chose 'run as administrator'. This will download the new files into the \new sub-folder. After you click OK to dismiss the dialog, you need to right click on the file again and chose the 'run as administrator' again. This time, the old version of files will be copied into the \old sub-folder while the new files will be installed. The program should run fine from now on.

If you did copy the directory to other drive, the installation of new version is a snap. Somehow, the Vista is quite protective against the 'program files' directory.

Now I am a happy camper.

* I am an old fashion guy and I like to control the program I am running. I hate those program that automatically update themselves without any warning. If programs are going to upgrade themselves, they should notify the user, backup the old version and let the user decide if they want to upgrade or not. At least, they need allow the user to roll-back to old version - I think
Microsoft learned this well - updates could have side effects even if you are careful.

Windstream DSL service problem - Lincoln, NE

This is to document Windstream/Alltel's service problem at Lincoln Nebraska around Dec. 25, 2009. For now, I am using this blog to document the problem. Later I will post these to other board to raise the awareness of the poor quality of the Windstream/Alltel DSL service.

On both Dec. 25 and 26, the Windstream DSL service become un-acceptable around 6:30PM. My DSL modem shows that the DSL is constantly dropped and reconnected in a interval of every 2 minutes. I can't watch any Youtube videos since the line will be dropped before any video is done. My daughter can't even log into the Webkinz.

This does not seem to be caused by service outage or the circuit problem since in the early morning and even early afternoon, the connection is just fine. It seems what Windstream is trying to do is multiplex the usages based on either the time or the amount of data sent or received. In either case, the service provided is un-bearable. I can't see this is a step up from the old 56K modem service.

I encourage every Windstream user to post comments to this blog and, together, we can make our voice heard and force Windsteam to take actions. For citizen of Lincoln, this may give us a chance to let other service company to enter the Lincoln market.

Update Dec. 29, 2009: Time 18:10 - Same problem: slowdown, dis-connected frequently.
Update Dec. 30, 2009: Time 20:25 - Same problem: slowdown, dis-connected frequently.
Update Dec. 31, 2009: Time 06:44 - I can watch Youtube - Evening is the problem.
Update Dec. 31, 2009: Time 19:32 - This morning I got phone call from Windstream that they are sending people out to tweak the circuit. Later in the day, I spot Windstream tech that went to the post at my backyard. At this moment, every thing works beautifully.

Tag, path and weighted tag

Tags have been slowly replacing the path/folder we used to use to organize articles/emails ... etc.

Tags, in mathematic terms, can be considered based on set theory. An information item can be considered an element while tags establish sub-sets. You can, therefore, find information items that belong to the intersection of the 'tea' subset and the 'green' subset and, hopefully, some of them are about 'green tea'.

The paths/folders are more like ordered tuples. It can be thought of as giving weights to the leading/parent folders. For example, a path of Education/computer could describing an item for educational community but contains information about computer usages. On the other hand, Computer/Education could be describing an item that are mainly for IT audiences but is related to education.

In terms of classifying an information item, the path/folder approach can give additional info about and item than tags, assuming using the same keywords. On the flip side, of cause, the classifying with paths are more involved. The other problem with paths is that, even though it provide weights, the weights are either 1 or 0 for parent and child folder.

The other issue to consider is the use of phrases in addition to words for tagging. There are times that tag with words is not the same as tag with a phrase. This applies to the paths too!

A possible alternative to tag and path is the weighted tag/phrase. In addition to plan tags/phases we can give them weights. The searching mechanism can now using both tag and weights to provide better results. In addition to that, the weight to a phrase can be distributed to words in the phrase in calculating the searching weights.

A note on ipfwadm spoof package protection

I was studying how to set up firewall rules for ipfwadm Linux command. In the process, I found some sample codes in filtering spoofed packets. However, I can't agree with the code.

Here is the code:

EXTERNNET="172.160.50.0/24"; # Subnetted external segment
ipfwadm -I -p accept
ipfwadm -I -a deny -V $IFINTERN -S $EXTERNNET -o

I thought about it and I think the code is in error. First of all, you can't defined an EXTERNNET. The EXTERNNET should be anything that is not your internal segment. But the ipfwadm syntax do not allow you to specify a negate or the supplement set. To over come this in this code setting, we should first allow the internal subnet:

ipfwadm -I -a accept -V $IFINTERN -S $INTERNNET -o

Then denied all others:

ipfwadm -I -a deny -V $IFINTERN -S $ANYWHERE -o

Of cause, depend on other rules, the deny statement may have to be moved toward the end so not to interfere with other rules.