I used Jason Davies’ website to generate these word clouds.
Nirvana – Smells Like Teen Spirit

Party All The Time – Eddie Murphy

I used Jason Davies’ website to generate these word clouds.
Nirvana – Smells Like Teen Spirit

Party All The Time – Eddie Murphy

Data is from https://nycopendata.socrata.com
All 311 Service Requests from 2010 to present. This information is automatically updated daily.
These are all the complaint types and their counts for NYPD.
These are a couple of heatmaps for all noise related calls :
“Noise – Commercial”, “Noise – Street/Sidewalk”, “Noise – House of Worship”, “Noise – Vehicle”, “Noise – Park”
The data is from https://data.cityofchicago.org/
This dataset is a listing of all current City of Chicago employees, complete with full names, departments, positions, and annual salaries. For hourly employees the annual salary is estimated. Data Owner: Human Resources. Frequency: Data is updated quarterly.
In this post I will programmatically pass level 1 in Prince of Persia using c#.

This game came out in 1989, it is now freeware.
In order to run MS DOS Prince of Persia(In 2013 win7 era), I had to download DOSBOX.
Dos Box Website: http://www.dosbox.com/
I used this input simulator in my application:
Windows input Simulator http://inputsimulator.codeplex.com/.
Here is a fun video I uploaded to youtube:
C# Code used.
using System; using System.Collections.Generic; using System.ComponentModel; using System.Data; using System.Drawing; using System.Linq; using System.Text; using System.Windows.Forms; using System.Runtime.InteropServices; using System.Diagnostics; using System.Threading; using WindowsInput; |
namespace Prince { public partial class Form1 : Form { [DllImport("kernel32.dll", SetLastError = true)] [return: MarshalAs(UnmanagedType.Bool)] static extern bool AllocConsole(); public Form1() { InitializeComponent(); } private void Form1_Load(object sender, EventArgs e) { AllocConsole(); } [DllImport("USER32.DLL", CharSet = CharSet.Unicode)] public static extern IntPtr FindWindow(string lpClassName, string lpWindowName); [DllImport("USER32.DLL")] public static extern bool SetForegroundWindow(IntPtr hWnd); private void button1_Click(object sender, EventArgs e) { Process[] localAll = Process.GetProcesses(); IntPtr calculatorHandle = FindWindow(null, "DOSBox 0.74, Cpu speed: 3000 cycles, Frameskip 0, Program: PRINCE"); if (calculatorHandle == IntPtr.Zero) return; SetForegroundWindow(calculatorHandle); Thread.Sleep(500); if (SetForegroundWindow(calculatorHandle)) { commandPrince(VirtualKeyCode.NUMPAD6, 3700); commandPrince(VirtualKeyCode.SHIFT, 2000); commandPrince(VirtualKeyCode.NUMPAD4, 4200); Thread.Sleep(500); commandPrince(VirtualKeyCode.NUMPAD8, 2000); Thread.Sleep(1400); //.... This ^ is a small sample of commands I give to Prince //.... Full level 1 commands are below in the post. } } public void standingJumpForw(VirtualKeyCode key, VirtualKeyCode key2, int ms) { Console.WriteLine("Standing Forward Jump"); InputSimulator.SimulateKeyDown(key); Thread.Sleep(100); InputSimulator.SimulateKeyDown(key2); Thread.Sleep(ms); InputSimulator.SimulateKeyUp(key2); InputSimulator.SimulateKeyUp(key); Console.WriteLine(DateTime.Now.ToString("H:mm:ss") + " " + key.ToString() + " up "); } public void commandClibUp(VirtualKeyCode key, VirtualKeyCode key2, int ms) { Console.WriteLine(DateTime.Now.ToString("H:mm:ss") + " " + key.ToString() + " down "); InputSimulator.SimulateKeyDown(key); Thread.Sleep(ms); InputSimulator.SimulateKeyDown(key2); Thread.Sleep(500); InputSimulator.SimulateKeyUp(key2); InputSimulator.SimulateKeyUp(key); Console.WriteLine(DateTime.Now.ToString("H:mm:ss") + " " + key.ToString() + " up "); } public void commandPrince(VirtualKeyCode key, int ms) { Console.WriteLine(DateTime.Now.ToString("H:mm:ss") + " " + key.ToString() + " down "); InputSimulator.SimulateKeyDown(key); Thread.Sleep(ms); InputSimulator.SimulateKeyUp(key); Console.WriteLine(DateTime.Now.ToString("H:mm:ss") + " " + key.ToString() + " up "); } public void runningJump(VirtualKeyCode key, VirtualKeyCode key2, int ms) { Console.WriteLine("Running Jump "); InputSimulator.SimulateKeyDown(key); Thread.Sleep(ms); InputSimulator.SimulateKeyDown(key2); Thread.Sleep(550); InputSimulator.SimulateKeyUp(key2); InputSimulator.SimulateKeyUp(key); } public void runningJumpFast(VirtualKeyCode key, VirtualKeyCode key2, int ms) { Console.WriteLine("Running Jump Fast"); InputSimulator.SimulateKeyDown(key); Thread.Sleep(ms); InputSimulator.SimulateKeyDown(key2); Thread.Sleep(550); InputSimulator.SimulateKeyUp(key2); InputSimulator.SimulateKeyUp(key); } } } |
How to get the name of all possible worksheets in a Microsoft Excel File.

Imports System.IO Imports System.Data Imports System.Data.OleDb |
Dim con As System.Data.OleDb.OleDbConnection con = New System.Data.OleDb.OleDbConnection("provider=Microsoft.Jet.OLEDB.4.0;" & "data source=C:\ExcelFile.xls;Extended Properties=Excel 8.0;") con.Open() Dim dtSheets As DataTable = con.GetOleDbSchemaTable(OleDbSchemaGuid.Tables, Nothing) Dim drSheet As DataRow For Each drSheet In dtSheets.Rows ''Here are all your possible worksheets. ''drSheet("TABLE_NAME").ToString() Next |
Data is from City of Chicago Data Portal
This dataset displays location for vehicles that have been towed and impounded by the City of Chicago within the last 90 days. Illegally parked vehicles, abandoned vehicles and vehicles used for illegal activities may be towed by the Chicago Police Department, the Department of Streets and Sanitation, the Department of Revenue, Aviation and the office of the City Clerk.
TIME RANGE 06/25/2013 – 09/23/2013 (90 Days)
My goal with this post is to parse out all the horses which came in 1st to 3rd place from each race that took place at Arlington Park this season.
The website http://www.arlingtonpark.com/racing-handicapping/equibase/charts provides a calendar which links to the result file for each race date from the past. Clicking on a particular date will link you to a pdf which contains various statistics.
So lets download all of these programmatically. To achieve this, I wrote a small program to iterate over dates in the uri, requesting every race date pdf file.
Then, I save them into a folder on my c: drive called \HORSEDATA\. I also, wait 1 -3 second before each request, alternate the web client’s header, and delete the downloaded file if it is too small (No Race happened on that date).
DateTime Today = DateTime.Now; int randomWait = 0; Random random = new Random(); for (int i = 0; i < 200; i++) { randomWait = random.Next(1000, 3000); Thread.Sleep(randomWait); Console.WriteLine("Downloading : " + Today.ToString("d")); Today = Today.AddDays( -1 ); try { using (WebClient client = new WebClient()) { client.Headers.Add("user-agent", "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.2; .NET CLR 1.0.370"+i+";)"); client.DownloadFile("http://www.equibase.com/premium/eqbPDFChartPlus.cfm?RACE=A&BorP=P&TID=AP&CTRY=USA&DT=" + Today.ToString("d") + "&DAY=D&STYLE=EQB", @"C:\HORSEDATA\\" + Today.ToString("MM-dd-yy") + ".pdf"); } } catch (Exception x) {Console.WriteLine("No Race on this Date.");} FileInfo f = new FileInfo(@"C:\HORSEDATA\\" + Today.ToString("MM-dd-yy") + ".pdf"); long s1 = f.Length; Console.WriteLine(s1.ToString()); if (s1 < 4000) File.Delete(@"C:\HORSEDATA\\" + Today.ToString("MM-dd-yy") + ".pdf"); } |
Now your \HORSEDATA\ folder should contain the proper pdfs.

Next, read through all the race date PDFs. For each PDF, I want to get the names of horses which came in 1st to 3rd place, and count up the amount of times each unique horse at least showed.
Using iTextSharp library to read the PDFs and count up my horses.
using iTextSharp; using iTextSharp.text.pdf; using System.IO; using iTextSharp.text.pdf.parser; using System.Text.RegularExpressions; |
Console.BufferHeight = 6000; Dictionary<string, int> dictHorseShowCount = new Dictionary<string, int>(); LinkedList uniqueHorse = new LinkedList(); string[] filePaths = Directory.GetFiles(@"c:\HORSEDATASET\"); foreach (string file in filePaths) { Console.WriteLine("Reading : " + file); PdfReader reader = new PdfReader(file); StringWriter output = new StringWriter(); for (int i = 1; i <= reader.NumberOfPages; i++) output.WriteLine(PdfTextExtractor.GetTextFromPage(reader, i, new SimpleTextExtractionStrategy())); Regex regPattern = new Regex(@"Total WPS Pool(.*?)Copyright", RegexOptions.Singleline); MatchCollection matchX = regPattern.Matches(output.ToString()); foreach (Match match in matchX) { Match match2 = Regex.Match(match.ToString(), @"Win Place Show(.*?)Wager Type", RegexOptions.Singleline); if (match2.Success) { string [] lineparts = Regex.Split(match2.ToString(),"\n"); for( int i = 1; i < lineparts.Length - 1; i++ ) { string horseOnly = Regex.Replace(lineparts[i].ToString(), "[^a-zA-Z -]", ""); if (!uniqueHorse.Contains(horseOnly.Trim())) uniqueHorse.AddLast(horseOnly.Trim()); if (dictHorseShowCount.ContainsKey(horseOnly.Trim())) { int existing_int = dictHorseShowCount[horseOnly.Trim()]; existing_int += 1; dictHorseShowCount[horseOnly.Trim()] = existing_int; } else dictHorseShowCount[horseOnly.Trim()] = 1; } } } } var sortedDict = (from entry in dictHorseShowCount orderby entry.Value ascending select entry) .ToDictionary(pair => pair.Key, pair => pair.Value); foreach (string key in sortedDict.Keys) Console.WriteLine(key + " " + sortedDict[key]); Console.WriteLine(""); Console.WriteLine("Distinct Horse Showed : " + uniqueHorse.Count); Console.WriteLine(""); |
JAIL DATA!
Data is from
National Archive of Criminal Justice Data
http://www.icpsr.umich.edu/icpsrweb/NACJD/studies/33722
Use the dataset’s cookbook file to make sense of specific headers
states are coded as number, create a hash for all numbers

Create a hash table for each race with the State as the Key. Then Parse the file, and populate each hash with the inmate count.

This morning I discovered an interesting data set
Historical Data About Every Reported United States Drone Strike
from http://dronestre.am/

THIS DATA WAS ACCESSED 8/31/2013
So I decided to make some quick visualizations using Google api.
The Data can be downloaded in JSON format and looks like this:

Parse out the coordinates of each strike using ruby:

Accumulate strike totals by country:

Code to display the date of a strike, and its attributes; deaths, civilians, injuries, and children.

Reported US Drone Strikes 2012 – 2013 stacked chart

Entire Time Range of drone.am dataset accessed 8/31/2013
http://projecteuler.net/problem=11
Largest product in a grid
What is the greatest product of four adjacent numbers in the same direction (up, down, left, right, or diagonally) in the 20
20 grid?