Author Archives: MantasCode

Ruby: Visualizing 2013 City of Chicago Top 25 Yearly Salaries.

Leave a reply

The data is from https://data.cityofchicago.org/

This dataset is a listing of all current City of Chicago employees, complete with full names, departments, positions, and annual salaries. For hourly employees the annual salary is estimated. Data Owner: Human Resources. Frequency: Data is updated quarterly.

POLICE
mc_Police

FIRE
mc_fire

HEALTH
mc_Health

AVIATION
mc_Aviation

MAYOR’S OFFICE
mc_mayor

How the Data was parsed and formatted.
parsingthechicagodata

C#: Programmatically Playing Prince of Persia.

Leave a reply

In this post I will programmatically pass level 1 in Prince of Persia using c#.

Prince of Persia
This game came out in 1989, it is now freeware.

In order to run MS DOS Prince of Persia(In 2013 win7 era), I had to download DOSBOX.
Dos Box Website: http://www.dosbox.com/

I used this input simulator in my application:
Windows input Simulator http://inputsimulator.codeplex.com/.

Here is a fun video I uploaded to youtube:

C# Code used.

using System;
using System.Collections.Generic;
using System.ComponentModel;
using System.Data;
using System.Drawing;
using System.Linq;
using System.Text;
using System.Windows.Forms;
using System.Runtime.InteropServices;
using System.Diagnostics;
using System.Threading;
using WindowsInput;
namespace Prince
{
    public partial class Form1 : Form
    {
        [DllImport("kernel32.dll", SetLastError = true)]
        [return: MarshalAs(UnmanagedType.Bool)]
        static extern bool AllocConsole();
        public Form1()
        {
            InitializeComponent();
        }
        private void Form1_Load(object sender, EventArgs e)
        {
            AllocConsole();
        }
        [DllImport("USER32.DLL", CharSet = CharSet.Unicode)]
        public static extern IntPtr FindWindow(string lpClassName,
            string lpWindowName);
        [DllImport("USER32.DLL")]
        public static extern bool SetForegroundWindow(IntPtr hWnd);
        private void button1_Click(object sender, EventArgs e)
        {
            Process[] localAll = Process.GetProcesses();
            IntPtr calculatorHandle = FindWindow(null, "DOSBox 0.74, Cpu speed:     3000 cycles, Frameskip  0, Program:   PRINCE");
             if (calculatorHandle == IntPtr.Zero)
                 return;
             SetForegroundWindow(calculatorHandle);
             Thread.Sleep(500);
             if (SetForegroundWindow(calculatorHandle))
             {
                    commandPrince(VirtualKeyCode.NUMPAD6, 3700);
                    commandPrince(VirtualKeyCode.SHIFT, 2000);
                    commandPrince(VirtualKeyCode.NUMPAD4, 4200);
                    Thread.Sleep(500);
                    commandPrince(VirtualKeyCode.NUMPAD8, 2000);
                    Thread.Sleep(1400);
 
                    //....  This ^ is a small sample of commands I give to Prince
                    //....  Full level 1 commands are below in the post.
 
             } 
        }
        public void standingJumpForw(VirtualKeyCode key, VirtualKeyCode key2, int ms)
        {
            Console.WriteLine("Standing Forward Jump");
            InputSimulator.SimulateKeyDown(key);
            Thread.Sleep(100);
            InputSimulator.SimulateKeyDown(key2);
            Thread.Sleep(ms);
            InputSimulator.SimulateKeyUp(key2);
            InputSimulator.SimulateKeyUp(key);
            Console.WriteLine(DateTime.Now.ToString("H:mm:ss") + " " + key.ToString() + " up ");
        }
        public void commandClibUp(VirtualKeyCode key, VirtualKeyCode key2, int ms)
        {
            Console.WriteLine(DateTime.Now.ToString("H:mm:ss") + " " + key.ToString() + " down ");
            InputSimulator.SimulateKeyDown(key);
            Thread.Sleep(ms);
            InputSimulator.SimulateKeyDown(key2);
            Thread.Sleep(500);
            InputSimulator.SimulateKeyUp(key2);
            InputSimulator.SimulateKeyUp(key);
            Console.WriteLine(DateTime.Now.ToString("H:mm:ss") + " " + key.ToString() + " up ");
        }
        public void commandPrince(VirtualKeyCode key, int ms)
        {
            Console.WriteLine(DateTime.Now.ToString("H:mm:ss") + " " + key.ToString() + " down ");
            InputSimulator.SimulateKeyDown(key);
            Thread.Sleep(ms);
            InputSimulator.SimulateKeyUp(key);
            Console.WriteLine(DateTime.Now.ToString("H:mm:ss") + " " + key.ToString() + " up ");
        }
 
        public void runningJump(VirtualKeyCode key, VirtualKeyCode key2, int ms)
        {
            Console.WriteLine("Running Jump ");
            InputSimulator.SimulateKeyDown(key);
            Thread.Sleep(ms);
            InputSimulator.SimulateKeyDown(key2);
            Thread.Sleep(550);
            InputSimulator.SimulateKeyUp(key2);
            InputSimulator.SimulateKeyUp(key);
        }
        public void runningJumpFast(VirtualKeyCode key, VirtualKeyCode key2, int ms)
        {
            Console.WriteLine("Running Jump Fast");
            InputSimulator.SimulateKeyDown(key);
            Thread.Sleep(ms);
            InputSimulator.SimulateKeyDown(key2);
            Thread.Sleep(550);
            InputSimulator.SimulateKeyUp(key2);
            InputSimulator.SimulateKeyUp(key);
        }
    }
}

Continue reading →

ASP.NET: VB Ole Get the Names of Potential Worksheets in MS Excel File.

Leave a reply

How to get the name of all possible worksheets in a Microsoft Excel File.
ExcelWorksheets

Imports System.IO
Imports System.Data
Imports System.Data.OleDb
Dim con As System.Data.OleDb.OleDbConnection
con = New System.Data.OleDb.OleDbConnection("provider=Microsoft.Jet.OLEDB.4.0;" 
& "data source=C:\ExcelFile.xls;Extended Properties=Excel 8.0;")
con.Open()
Dim dtSheets As DataTable =
    con.GetOleDbSchemaTable(OleDbSchemaGuid.Tables, Nothing)
Dim drSheet As DataRow
For Each drSheet In dtSheets.Rows
    ''Here are all your possible worksheets.
    ''drSheet("TABLE_NAME").ToString()
Next

Ruby: Graphing Chicago Towed Vehicles by Brand and Color

1 Reply

cartowedformat

Data is from City of Chicago Data Portal

This dataset displays location for vehicles that have been towed and impounded by the City of Chicago within the last 90 days. Illegally parked vehicles, abandoned vehicles and vehicles used for illegal activities may be towed by the Chicago Police Department, the Department of Streets and Sanitation, the Department of Revenue, Aviation and the office of the City Clerk.

TIME RANGE 06/25/2013 – 09/23/2013 (90 Days)

VEHICLE COLOR
ChicagoTowedByColor

VEHICLE BRAND
ChicagoTowed by brand

Ruby Code used to parse and manicure data.
rubycarparse

C#: Parsing Horses.

1 Reply

Horserace

My goal with this post is to parse out all the horses which came in 1st to 3rd place from each race that took place at Arlington Park this season.

The website http://www.arlingtonpark.com/racing-handicapping/equibase/charts provides a calendar which links to the result file for each race date from the past. Clicking on a particular date will link you to a pdf which contains various statistics.

calendar

So lets download all of these programmatically. To achieve this, I wrote a small program to iterate over dates in the uri, requesting every race date pdf file.

Then, I save them into a folder on my c: drive called \HORSEDATA\. I also, wait 1 -3 second before each request, alternate the web client’s header, and delete the downloaded file if it is too small (No Race happened on that date).

DateTime Today = DateTime.Now;
int randomWait = 0;
Random random = new Random();
for (int i = 0; i < 200; i++)
{
    randomWait = random.Next(1000, 3000);
    Thread.Sleep(randomWait);
    Console.WriteLine("Downloading : " + Today.ToString("d"));
    Today =  Today.AddDays( -1 );
    try
    {
        using (WebClient client = new WebClient())
        {
client.Headers.Add("user-agent", "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.2; .NET CLR 1.0.370"+i+";)");
client.DownloadFile("http://www.equibase.com/premium/eqbPDFChartPlus.cfm?RACE=A&BorP=P&TID=AP&CTRY=USA&DT=" 
    + Today.ToString("d") + "&DAY=D&STYLE=EQB", @"C:\HORSEDATA\\" + Today.ToString("MM-dd-yy") + ".pdf");
        }
    }
    catch (Exception x)
    {Console.WriteLine("No Race on this Date.");}
    FileInfo f = new FileInfo(@"C:\HORSEDATA\\" + Today.ToString("MM-dd-yy") + ".pdf");
    long s1 = f.Length;
    Console.WriteLine(s1.ToString());
    if (s1 < 4000)
        File.Delete(@"C:\HORSEDATA\\" + Today.ToString("MM-dd-yy") + ".pdf");
}

Now your \HORSEDATA\ folder should contain the proper pdfs.
horsedatapdf

Next, read through all the race date PDFs. For each PDF, I want to get the names of horses which came in 1st to 3rd place, and count up the amount of times each unique horse at least showed.

3 horse parse

Using iTextSharp library to read the PDFs and count up my horses.

using iTextSharp;
using iTextSharp.text.pdf;
using System.IO;
using iTextSharp.text.pdf.parser;
using System.Text.RegularExpressions;
Console.BufferHeight = 6000;
Dictionary<string, int> dictHorseShowCount = new Dictionary<string, int>();
LinkedList uniqueHorse = new LinkedList();
string[] filePaths = Directory.GetFiles(@"c:\HORSEDATASET\");
foreach (string file in filePaths)
{
    Console.WriteLine("Reading : " + file);
    PdfReader reader = new PdfReader(file); 
    StringWriter output = new StringWriter();  
    for (int i = 1; i <= reader.NumberOfPages; i++)
        output.WriteLine(PdfTextExtractor.GetTextFromPage(reader, i, new SimpleTextExtractionStrategy()));
    Regex regPattern = new Regex(@"Total WPS Pool(.*?)Copyright", RegexOptions.Singleline);
    MatchCollection matchX = regPattern.Matches(output.ToString());
    foreach (Match match in matchX)
    {
        Match match2 = Regex.Match(match.ToString(), @"Win Place Show(.*?)Wager Type", RegexOptions.Singleline);
        if (match2.Success)
        {
            string [] lineparts = Regex.Split(match2.ToString(),"\n");
            for( int i = 1; i < lineparts.Length - 1; i++ )
            {
                string horseOnly = Regex.Replace(lineparts[i].ToString(), "[^a-zA-Z -]", "");
                if (!uniqueHorse.Contains(horseOnly.Trim()))
                    uniqueHorse.AddLast(horseOnly.Trim());
                if (dictHorseShowCount.ContainsKey(horseOnly.Trim()))
                {
                    int existing_int = dictHorseShowCount[horseOnly.Trim()];
                    existing_int += 1;
                    dictHorseShowCount[horseOnly.Trim()] = existing_int;
                }
                else
                    dictHorseShowCount[horseOnly.Trim()] = 1;
            }
        }
    }
}
var sortedDict = (from entry in dictHorseShowCount orderby entry.Value ascending select entry)
.ToDictionary(pair => pair.Key, pair => pair.Value);
foreach (string key in sortedDict.Keys)
    Console.WriteLine(key + "   "   + sortedDict[key]);
Console.WriteLine("");
Console.WriteLine("Distinct Horse Showed : " + uniqueHorse.Count);
Console.WriteLine("");

Output
outputHORSES

Ruby: Visualizing United States Jail Data by Race and State.

Leave a reply

JAIL DATA!

SomeJailFromWikipedia

Data is from
National Archive of Criminal Justice Data
http://www.icpsr.umich.edu/icpsrweb/NACJD/studies/33722

Use the dataset’s cookbook file to make sense of specific headers
states are coded as number, create a hash for all numbers
hashnew

Create a hash table for each race with the State as the Key. Then Parse the file, and populate each hash with the inmate count.
step2

Export JSON in GoogleCharts Acceptable format
step3

Here’s a Stacked Column Chart
MantasCodeRaceJailChart

Visualizing US Drone Strike Data.

1 Reply

This morning I discovered an interesting data set

Historical Data About Every Reported United States Drone Strike
from http://dronestre.am/
homepic

THIS DATA WAS ACCESSED 8/31/2013

So I decided to make some quick visualizations using Google api.
The Data can be downloaded in JSON format and looks like this:
json

Parse out the coordinates of each strike using ruby:
ruby_heatmap

Heat Maps
HeatMapSatDroneStirkeAfgan_pak_borderheatmap

Accumulate strike totals by country:
pirbycountry

Pie Chart By Country
Country Pie

Code to display the date of a strike, and its attributes; deaths, civilians, injuries, and children.
stackedareaRuby

Reported US Drone Strikes 2012 – 2013 stacked chart
2012-2013Drone Strikes

Entire Time Range of drone.am dataset accessed 8/31/2013

EndtireDataSetImage

DataViz: Comparing Chicago Crime Data to Los Angeles Crime Data for 2012

2 Replies

According to Google’s approximation of the city populations :
approxPops

2012 Chicago Crime Data From :

https://data.cityofchicago.org/
“Data is extracted from the Chicago Police Department’s CLEAR (Citizen Law Enforcement Analysis and Reporting) system.”

2012 Los Angeles Crime Data :

http://www.lasdhq.org/sites/CAASS/desc.html
Los Angeles Count Sherrif’s Department

Now I have two sets of data. I have a (Primary Crime Type or Category of Crime) and Frequency;
for both Chicago and Los Angeles.

Here are what they look like.
LABEL CORRELATION

CPD labels their crimes differently from the LAPD.

Here is how I will be correlating some of the crimes between the two cities. Comparing the Chicago classifications to Los Angeles.
ManualCorrelate

Next I will manicure my data into a Google Column Chart acceptable JSON object.

CPD vs LAPD

Ruby: Using Google Maps Heat Map to Display Chicago Crime Statistics

3 Replies

Each crime report contains latitude and longitude coordinates.  I used ruby to parse out each coordinate from this source :  https://data.cityofchicago.org.  Then I appended those points into a heat map.

Update: This dataset has been expanded on, in this post DataViz: Visualizing Chicago Narcotics Crime Data.

Chicago DECEPTIVE PRACTICE – 8/8/2012 to 8/8/2013

DECEPTIVE PRACTICE

Chicago ALL CRIMES – 8/8/2012 to 8/8/2013
HEATMAPOFCRIME

Chicago NARCOTICS – 8/8/2012 to 8/8/2013
NARCOTICS

Chicago HOMICIDE – 8/8/2012 to 8/8/2013
murderonlysat

Ruby example to parse all coordinates for Primary Crime Type: DECEPTIVE PRACTICE
RUBY

Output
output