Adventures with R - Cricket Analysis (creating a Player Database)

One of the things I had encountered in the main series was the inability to create a comprehensive player database. That part of the code was fairly manual.

I kept noodling and tinkering around to get an approach for creating a comprehensive player database. This would replace the manual approach of extracting one player link at a time.

I had initially set out to use the excellent Rvest package but ran into some issues trying to decipher the xpath that is required to make the link work. I believe that the player information is coded directly as html tags on crickinfo and it would have taken me a couple of xpath loops to get the player name and then then the player profile id out. Definitely doable (but will keep Rvest for another code I have in mind..

I focused on using dplyr, tidyr to do my heavy lifting

The code can be found HERE

Code walk through. I am reproducing the first part of the code, the main code is available for everyone to look at

library(stringr)
library(sqldf)
library(dplyr)
library(tidyr)

 batsmen_list <- list()
i<- 0

 # Hard coding pages to 50 pages. Each page has 50 players so we will have a database of 2500 top players, all time 
# These are some more types to generate more players
# Class 11 is all Test/ODI/T20 combined, Class 1 is top batsmen for Tests, Class 2 is top batsmen for ODI, Class 3 is top batsmen for T20
# First Run is for Overall Top Batsmen 

# Each page has 50 batsmen, focusing on the top 2500 batsmen
for(i in 1:50){
  
  print(i)
  
  main <- "http://stats.espncricinfo.com/ci/engine/stats/index.html?class=11;page="
  page <- i
  main1 <- ";template=results;type=batting"
  
  url <- sprintf("%s%s%s", main, page, main1)
  
  lines <- readLines(url)
  
  lines <- trimws(lines)
  
  lines <- lines[lines != '']
  
  link <- grep('href="', lines, value = TRUE) %>% 
    gsub('.*?href=\"(*ci/content/player/*)\">', '\\1', .)
  
  link <- paste0('http://stats.espncricinfo.com', link)
  
  batsmen_list[[i]] <- data.frame(link = link, stringsAsFactors = FALSE)
  
}

 lapply(batsmen_list, nrow)

 batsmen_df <- data.table::rbindlist(batsmen_list) %>% as.data.frame
batsmen_df_clean <- sqldf("select * from batsmen_df where link like '%/ci/content/player%' ")

 #Extending the table to have numeric player id

 numextract <- function(string){ 
  str_extract(string, "\\-*\\d+\\.*\\d*")
} 

 batsmen_df_clean$PlayerID = numextract(batsmen_df_clean$link)
batsmen_df_clean$PlayerID <- as.numeric(as.character(batsmen_df_clean$PlayerID))

 batsmen_df_clean$PlayerName <- as.character(sub(".*> *(.*?) *profile.*", "\\1", batsmen_df_clean$link))

 batsmen_df_clean <- batsmen_df_clean[-grep("http", batsmen_df_clean$PlayerName),]
batsmen_df_clean <- batsmen_df_clean[ -c(1)]
batsmen_df_clean <- unique(batsmen_df_clean)
batsmen_Overall <- batsmen_df_clean

The final batsmen_Overall provides a listing of 2500 batsmen with their profile id which cane be used in the main code

Comments

Post a Comment

Popular posts from this blog

Balkanization of Pakistan

Image
Pakistan - Should it be Balkanized ? While discussing my earlier post on 'A Weak India' with people, I realized that there was a total lack of awareness among the Indian Diaspora about what is going on back home and in general the subcontinent. I did get a few intelligent responses and one particularly interesting discussion was what is the rationale of having Pakistan as an independent country ? Life is the gift of nature, but beautiful living is the gift of wisdom", says a Greek adage. Maverick Pakistan President General Pervez Musharraf needs to suck up this sage saying down to his glib throat; whether he and his predecessors have been helpful in providing a " beautiful living" to the people of Afghanistan, their own denizens who had clamoured for Pakistan, and to the hapless Moslems of Kashmir, on the eastern fringe of truncated Pakistan. The leadership in undivided Pakistan, by pursuing unsound policies, had on the contrary played fraud on its own people.
Read more

Film Reviews on IndiaFM

The media manipulation with respect to movie reviews has already been well chronicled. It is a very well known fact that journalists covering movies are either in pro-director or anti-director groups. Consequently, you have well known reviewers like Roger Ebert giving thumbs up to movies which would not be worth even a first glance. These movies then go on to do a reasonable amount of business because people would think, "If Roger has endorsed it, we should go and check it out". This trend is all the more amplified in Bollywood. "Journalists" like Taran Adarsh of IndiaFM fame routinely pan work other than that by either the Chopras or Karan Johar. Of course, any movie starring SRK gets the most favorable rating. While I doubt the real impact of his rating scale (after all, SRK has his die-hard fans, review or no review), it is his "negative shade" about other movies that makes people think and then Not see the movie. A case in point is the movie "Seha
Read more

Outsourcing - the new wave !!

First they outsourced their customer service, then their market research analysis, then teaching and then some more....... Well, if Time is to be believed, the day is not far when we have the US Army outsourcing its 'war on terror' to where else ? - INDIA !! What do these "soldiers" get - US Citizenship. According to "Fast Track" in the August 7, 2006 issue of Time. "When President Bush visited Walter Reed Army Medical Center last week to witness three U.S. soldiers seriously wounded in Iraq become U.S. citizens, he said that "if somebody's willing to risk their lives for our country, they ought to be full participants in our country." It was an executive order signed by Bush four years ago that allowed this trio, and 35,000 like them, to petition for U.S. citizenship as soon as they donned a U.S. military uniform, instead of waiting the customary three years. So far, 26,000 of them have achieved that goal. Be it patriotism or the easier p
Read more