Category Archives: C#

Philadelphia’s Drugs Drunks and Prostitute Maps

A look into Philadelphia’s Drugs Drunks and Prostitutes. The data is collected by the Philadelphia Police Department. Data can be accessed here https://www.opendataphilly.org/. Old CARTO was used to create visualizations. Click on static maps below for full size.

Continue reading →

Iowa’s Liquor Sale Maps and Visualizations 2020

2 Replies

A look into Iowa’s liquor sales for the last 2.5 years. Time spans from January 2018 through June 2020. The data can be found at data.iowa.gov. Click on images below to enlarge.

Distinct Coordinate by Volume Sold (Liters)

Pandemic Upswing

Top Bottle Sold by Distinct Location

Top Bottle

Top Category Per Distinct Location

C# Console Application was used to parse, aggregate, and output the data into desired structure for Google Charts and Carto. The script below can be used as a reference if you would like to investigate Iowa’s liquor sale data yourself. Note that this particular script will only output the data source of the last map in this post (Top Category Per Distinct Location sized by Bottles Sold).

using CsvHelper;
using System;
using System.Collections.Generic;
using System.Globalization;
using System.IO;
using System.Linq;
using System.Text;
using System.Text.RegularExpressions;
using System.Threading.Tasks;
 
namespace IowaSpirits2018plus
{
    class Program
    {
        static void Main(string[] args)
        {
            Console.WriteLine("Iowa Spirits..");
            Console.BufferHeight = 4000;
            Dictionary<string, Dictionary<string, int>> dictCoord_Item_Count = new Dictionary<string, Dictionary<string, int>>();
            string lastdate = "";
            using (var reader = new StreamReader("C:\\IOWASPIRITS\\iowaspirits2018plus.csv"))
            using (var csv = new CsvReader(reader, CultureInfo.InvariantCulture))
            {
                csv.Read();
                csv.ReadHeader();
                while (csv.Read())
                {
                    string date = csv.GetField("Date").ToString();
                    if ( lastdate != date)
                        Console.WriteLine(date);
                    lastdate = date;
                    DateTime fDate = csv.GetField("Date");
                    string city = csv.GetField("City").ToString();
                    string categoryName = csv.GetField("Category Name").ToString();
                    string bottlesSold = csv.GetField("Bottles Sold").ToString();
                    int iBottlesSold = int.Parse(csv.GetField("Bottles Sold").ToString());
                    string litersSold = csv.GetField("Volume Sold (Liters)").ToString();
                    string itemDescription = csv.GetField("Item Description").ToString();
                    string storelocation = csv.GetField("Store Location").ToString();
                    storelocation = Regex.Replace(storelocation, "POINT \\(", "");
                    storelocation = Regex.Replace(storelocation, "\\)", "");
                    storelocation = Regex.Replace(storelocation, " ", ",");
 
                    if (dictCoord_Item_Count.ContainsKey(storelocation))
                    {
                        Dictionary&lt;string, int&gt; existingDict = new Dictionary&lt;string, int&gt;();
                        existingDict = dictCoord_Item_Count[storelocation];
                        if (existingDict.ContainsKey(categoryName))
                        {
                            int existingBottleSold = existingDict[categoryName];
                            existingBottleSold += iBottlesSold;
                            existingDict[categoryName] = existingBottleSold;
                        }
                        else
                        {
                            existingDict.Add(categoryName, iBottlesSold);
                        }
                        dictCoord_Item_Count[storelocation] = existingDict;
                    }
                    else
                    {
                        Dictionary&lt;string, int&gt; tempDictItemCount = new Dictionary&lt;string, int&gt;();
                        tempDictItemCount.Add(categoryName, iBottlesSold);
                        dictCoord_Item_Count.Add(storelocation, tempDictItemCount);
                    }
                }
            }
            foreach ( string parent_key in dictCoord_Item_Count.Keys)
            {
                Console.Write(parent_key + ",");
                foreach (var item in dictCoord_Item_Count[parent_key].OrderByDescending(r =&gt; r.Value))
                {
                    Console.Write("{0}, {1}", item.Key, item.Value);
                    break;
                }
                Console.WriteLine();
            }
        }
    }
}

using CsvHelper; using System; using System.Collections.Generic; using System.Globalization; using System.IO; using System.Linq; using System.Text; using System.Text.RegularExpressions; using System.Threading.Tasks; namespace IowaSpirits2018plus { class Program { static void Main(string[] args) { Console.WriteLine("Iowa Spirits.."); Console.BufferHeight = 4000; Dictionary<string, Dictionary<string, int>> dictCoord_Item_Count = new Dictionary<string, Dictionary<string, int>>(); string lastdate = ""; using (var reader = new StreamReader("C:\\IOWASPIRITS\\iowaspirits2018plus.csv")) using (var csv = new CsvReader(reader, CultureInfo.InvariantCulture)) { csv.Read(); csv.ReadHeader(); while (csv.Read()) { string date = csv.GetField("Date").ToString(); if ( lastdate != date) Console.WriteLine(date); lastdate = date; DateTime fDate = csv.GetField("Date"); string city = csv.GetField("City").ToString(); string categoryName = csv.GetField("Category Name").ToString(); string bottlesSold = csv.GetField("Bottles Sold").ToString(); int iBottlesSold = int.Parse(csv.GetField("Bottles Sold").ToString()); string litersSold = csv.GetField("Volume Sold (Liters)").ToString(); string itemDescription = csv.GetField("Item Description").ToString(); string storelocation = csv.GetField("Store Location").ToString(); storelocation = Regex.Replace(storelocation, "POINT \\(", ""); storelocation = Regex.Replace(storelocation, "\\)", ""); storelocation = Regex.Replace(storelocation, " ", ","); if (dictCoord_Item_Count.ContainsKey(storelocation)) { Dictionary<string, int> existingDict = new Dictionary<string, int>(); existingDict = dictCoord_Item_Count[storelocation]; if (existingDict.ContainsKey(categoryName)) { int existingBottleSold = existingDict[categoryName]; existingBottleSold += iBottlesSold; existingDict[categoryName] = existingBottleSold; } else { existingDict.Add(categoryName, iBottlesSold); } dictCoord_Item_Count[storelocation] = existingDict; } else { Dictionary<string, int> tempDictItemCount = new Dictionary<string, int>(); tempDictItemCount.Add(categoryName, iBottlesSold); dictCoord_Item_Count.Add(storelocation, tempDictItemCount); } } } foreach ( string parent_key in dictCoord_Item_Count.Keys) { Console.Write(parent_key + ","); foreach (var item in dictCoord_Item_Count[parent_key].OrderByDescending(r => r.Value)) { Console.Write("{0}, {1}", item.Key, item.Value); break; } Console.WriteLine(); } } } }

UK Bicycle Theft in Major Cities 2017 to 2020

Leave a reply

Revisiting UK’s bicycle thefts. The data comes from https://data.police.uk/. The date ranged used was from May 2017 through April 2020. All surrounding police forces were used to obtain a comprehensive list of bicycle theft incidents. Previous post from 2016 can be found here. Carto was used to render the maps. Cities include: London, Manchester, Birmingham, and Bristol

London Click to enlarge 3.12 mb

London Click to enlarge 3.73 mb

Manchester Click to enlarge 1.07 mb

Birmingham Click to enlarge 706 kb

Bristol Click to enlarge 656 kb

If you would like to create your own UK crime maps, and are frustrated at the way data.police.uk presents its data. You may use this simple C# console application to aggregate it yourself. Simply paste the downloadables into a directory called LONDON4, and use the script below to generate an output file containing incidents from multiple police forces. Cheers.

C# Console Application

string dirPath = @"C:\LONDON4\";
List dirs = new List(Directory.EnumerateDirectories(dirPath));
Dictionary&lt;string, int&gt; dictUniqueCrime = new Dictionary&lt;string, int&gt;();
foreach (var dir in dirs)
{
    Console.WriteLine("{0}", dir.Substring(dir.LastIndexOf("\\") + 1));
    string filepath = @"C:\LONDON4\";
    filepath += dir.Substring(dir.LastIndexOf("\\") + 1);
    Console.WriteLine(filepath);
    string line;
    string[] fileEntries = Directory.GetFiles(filepath);
    foreach (string fileName in fileEntries)
    {
        Console.WriteLine(fileName);
        System.IO.StreamReader file =
            new System.IO.StreamReader(fileName);
        while ((line = file.ReadLine()) != null)
        {
            string[] parts = line.Split(',');
            string month = parts[1];
            string longitude = parts[4];
            string latitude = parts[5];
            string crimetype = parts[9];
            if (crimetype == "Bicycle theft")
            {
                string lines = month + "-1, " + longitude + ", " + latitude + "";
                System.IO.StreamWriter file1 = new System.IO.StreamWriter("C:\\Airplanes\\____BRITISHBICYCLES.csv", true);
                file1.WriteLine(lines);
                file1.Close();
            }
        }
        file.Close();
    }
}

Top 50 most influential subreddit moderators – May 2020

1 Reply

Plotting number of subreddits moderated within top 1,000 most subscribed subs, vs. Subreddit Subscriber Aggregate.

Click on images below to enlarge.

The data is from http://redditlist.com/ and https://www.reddit.com/. C# was written to parse both websites and generate output formatted for a GoogleCharts Bubble Chart. This script was ran on 5/14/2020.

If you would like to parse the top 1,000 most subscribed-to subreddit mod usernames, and aggregate their frequency yourself. Feel free to download visual studio and run the code below in a simple .NET Console Application. Of course this being a hard parse it is only bound to work so long as redditlist and reddit don’t change their html structure, otherwise it can be used as a reference.

Crawl redditlist.com
Collect the middle column for the first 8 pages listing 125 each.

Crawl reddit.com
Iterate over each subreddit’s /about/moderators page and collect usernames shown below.

Regular Expression patterns used to collect information:

@"_blank(.*?)</div>"
@">(.*?)</a>"
@"listing-stat'>(.*?)</span>"
@"<span class=""user""><a href=""https://old.reddit.com/user/(.*?)/"

C# .NET Console Application

using System;
using System.Collections.Generic;
using System.IO;
using System.Linq;
using System.Net;
using System.Text;
using System.Text.RegularExpressions;
using System.Threading.Tasks;
 
namespace RedditMegaModBubbleParse
{
    class Program
    {
        static void Main(string[] args)
        {
 
            Dictionary<string, int> dictSub_SubscriberCount = new Dictionary<string, int>();
 
            //PARSE TOP 1000 Subreddit names and subscribers//
            List<string> top500Subreddits = new List<string>();
            for (int page = 1; page < 9; page++)
            {
                WebClient wc = new WebClient();
                string htmlString = wc.DownloadString("http://redditlist.com/?page=" + page);
                MatchCollection matches = Regex.Matches(htmlString, @"_blank(.*?)</div>", RegexOptions.Singleline);
                int count = 0;
                foreach (Match match in matches)
                {
                    if (count > 125 && count <= 250)
                    {
                        Console.WriteLine(count);
                        string sHtmlChunk = match.Groups[1].Value;
                        Match mSubName = Regex.Match(sHtmlChunk, @">(.*?)</a>");
                        Match mSubscriberCount = Regex.Match(sHtmlChunk, @"listing-stat'>(.*?)</span>");
                        if (mSubName.Success && mSubscriberCount.Success)
                        {
                            string sSubName = mSubName.Groups[1].Value;
                            Console.WriteLine("Subreddit Name : " + sSubName);
                            string sSubscriberCount = mSubscriberCount.Groups[1].Value;
                            int iSubscriberCount = int.Parse(Regex.Replace(sSubscriberCount, ",", ""));
                            Console.WriteLine("Subreddit Count : " + iSubscriberCount);
                            dictSub_SubscriberCount.Add(sSubName, iSubscriberCount);
                        }
                        Console.WriteLine();
                    }
                    count += 1;
                }
            }
 
            Dictionary<string, int> dictModCount = new Dictionary<string, int>();
            Dictionary<string, int> dictModSubscriberAggregate = new Dictionary<string, int>();
 
            int count_sub = 0;
            foreach (string subreddit_name in dictSub_SubscriberCount.Keys)
            {
                try
                {
                    count_sub += 1;
                    Console.WriteLine("*** Subreddit " + count_sub + " ~~~ " + "[" + subreddit_name + "] ***");
                    WebClient wc = new WebClient();
                    string subreddit_clean = Regex.Replace(subreddit_name, " ", "");
                    string url_glue = "https://old.reddit.com/r/" + subreddit_clean + "/about/moderators";
                    Console.WriteLine(url_glue);
                    string htmlString = wc.DownloadString(url_glue);
                    MatchCollection matches = Regex.Matches(htmlString, @"<span class=""user""><a href=""https://old.reddit.com/user/(.*?)/", RegexOptions.Singleline);
                    foreach (Match match in matches)
                    {
                        Console.WriteLine(match.Groups[1].Value);
                        string modname = match.Groups[1].Value;
 
                        if (dictModCount.ContainsKey(modname))
                        {
                            int existingcount = dictModCount[modname];
                            existingcount += 1;
                            dictModCount[modname] = existingcount;
 
                            int existingSubscriberCount = dictModSubscriberAggregate[modname];
                            existingSubscriberCount += dictSub_SubscriberCount[subreddit_name];
                            dictModSubscriberAggregate[modname] = existingSubscriberCount;
 
                        }
                        else
                        {
                            dictModCount.Add(modname, 1);
                            dictModSubscriberAggregate.Add(modname, dictSub_SubscriberCount[subreddit_name]);
                        }
                    }
                }
                catch (Exception x)
                {
                    Console.WriteLine("ERROR ACCESSING SUBREDDIT  INVITE ONLY!!!!!!!");
                }
            }
 
            Console.WriteLine();
 
            string masteroutput = "";
            foreach (KeyValuePair<string, int> item in dictModCount.OrderByDescending(key => key.Value))
            {
                Console.WriteLine("['"+item.Key + "', " + item.Value+", "+ dictModSubscriberAggregate[item.Key]+", '1' , "+ dictModSubscriberAggregate[item.Key] + " ],");
                masteroutput += "['" + item.Key + "', " + item.Value + ", " + dictModSubscriberAggregate[item.Key] + ", '1' , " + dictModSubscriberAggregate[item.Key] + " ],\r\n";
            }
 
            //save to C:\Airplanes\
            StreamWriter streamWrite;
            streamWrite = File.AppendText("C:\\Airplanes\\output.txt");
            streamWrite.WriteLine(masteroutput);
            streamWrite.Close();
 
            Console.ReadLine();
 
        }
    }
}

using System; using System.Collections.Generic; using System.IO; using System.Linq; using System.Net; using System.Text; using System.Text.RegularExpressions; using System.Threading.Tasks; namespace RedditMegaModBubbleParse { class Program { static void Main(string[] args) { Dictionary<string, int> dictSub_SubscriberCount = new Dictionary<string, int>(); //PARSE TOP 1000 Subreddit names and subscribers// List<string> top500Subreddits = new List<string>(); for (int page = 1; page < 9; page++) { WebClient wc = new WebClient(); string htmlString = wc.DownloadString("http://redditlist.com/?page=" + page); MatchCollection matches = Regex.Matches(htmlString, @"_blank(.*?)</div>", RegexOptions.Singleline); int count = 0; foreach (Match match in matches) { if (count > 125 && count <= 250) { Console.WriteLine(count); string sHtmlChunk = match.Groups[1].Value; Match mSubName = Regex.Match(sHtmlChunk, @">(.*?)</a>"); Match mSubscriberCount = Regex.Match(sHtmlChunk, @"listing-stat'>(.*?)</span>"); if (mSubName.Success && mSubscriberCount.Success) { string sSubName = mSubName.Groups[1].Value; Console.WriteLine("Subreddit Name : " + sSubName); string sSubscriberCount = mSubscriberCount.Groups[1].Value; int iSubscriberCount = int.Parse(Regex.Replace(sSubscriberCount, ",", "")); Console.WriteLine("Subreddit Count : " + iSubscriberCount); dictSub_SubscriberCount.Add(sSubName, iSubscriberCount); } Console.WriteLine(); } count += 1; } } Dictionary<string, int> dictModCount = new Dictionary<string, int>(); Dictionary<string, int> dictModSubscriberAggregate = new Dictionary<string, int>(); int count_sub = 0; foreach (string subreddit_name in dictSub_SubscriberCount.Keys) { try { count_sub += 1; Console.WriteLine("*** Subreddit " + count_sub + " ~~~ " + "[" + subreddit_name + "] ***"); WebClient wc = new WebClient(); string subreddit_clean = Regex.Replace(subreddit_name, " ", ""); string url_glue = "https://old.reddit.com/r/" + subreddit_clean + "/about/moderators"; Console.WriteLine(url_glue); string htmlString = wc.DownloadString(url_glue); MatchCollection matches = Regex.Matches(htmlString, @"<span class=""user""><a href=""https://old.reddit.com/user/(.*?)/", RegexOptions.Singleline); foreach (Match match in matches) { Console.WriteLine(match.Groups[1].Value); string modname = match.Groups[1].Value; if (dictModCount.ContainsKey(modname)) { int existingcount = dictModCount[modname]; existingcount += 1; dictModCount[modname] = existingcount; int existingSubscriberCount = dictModSubscriberAggregate[modname]; existingSubscriberCount += dictSub_SubscriberCount[subreddit_name]; dictModSubscriberAggregate[modname] = existingSubscriberCount; } else { dictModCount.Add(modname, 1); dictModSubscriberAggregate.Add(modname, dictSub_SubscriberCount[subreddit_name]); } } } catch (Exception x) { Console.WriteLine("ERROR ACCESSING SUBREDDIT INVITE ONLY!!!!!!!"); } } Console.WriteLine(); string masteroutput = ""; foreach (KeyValuePair<string, int> item in dictModCount.OrderByDescending(key => key.Value)) { Console.WriteLine("['"+item.Key + "', " + item.Value+", "+ dictModSubscriberAggregate[item.Key]+", '1' , "+ dictModSubscriberAggregate[item.Key] + " ],"); masteroutput += "['" + item.Key + "', " + item.Value + ", " + dictModSubscriberAggregate[item.Key] + ", '1' , " + dictModSubscriberAggregate[item.Key] + " ],\r\n"; } //save to C:\Airplanes\ StreamWriter streamWrite; streamWrite = File.AppendText("C:\\Airplanes\\output.txt"); streamWrite.WriteLine(masteroutput); streamWrite.Close(); Console.ReadLine(); } } }

Congrats to the May 2020 top 50 Reddit Megamods
In order of the most subreddits modded within top 1,000 most subscribed-to subreddits.

AutoModerator MAGIC_EYE_BOT BotTerminator Blank-Cheque RepostSentinel cyXie Umbresp AssistantBOT BotDefense metastasis_d awkwardtheturtle LeafSamurai Merari01 IranianGenius commonvanilla GallowBoob N8theGr8 love_the_heat ManWithoutModem greatyellowshark justcool393 Lil_SpazJoekp siouxsie_siouxv2 SuzyModQ pHorniCaiTe babar77 PowerModerator maybesaydie davidreiss666 Sunkisty yummytuber SEO_Nuke PhlogistonAster daninger4995 kjoneslol sloth_on_meth ani625 Tornado9797 sidshembekar RalphiesBoogers RepostSleuthBot Noerdy stuffed02 whyhellomichael qgyh2 Llim ModeratelyHelpfulBot EpicEngineer T_Dumbsford Kesha_Paul

Proximity Vegetation

Leave a reply

This is a game mechanic concept whipped up using Unity3d and C#.

I figured it might look interesting to see procedurally generated trees spawning in, and growing, as the player moves. The tree variance, growth rate, and spawn locations are randomized. New tree spawns are triggered by the player’s movement. Something I’ve never seen done with foliage in a video game before.

Watch the video.

Generating Procedural Trees using Random in Unity3d

Leave a reply

Experimenting with some coroutines and System.Random

Experimenting with Unity3d

Leave a reply

Playing around with Unity3D.

Marble Race

Ball Blender

Top military expenditures by country.

London Drug Map

Leave a reply

A look into open data about crime and policing in England.

Found at https://data.police.uk/. I downloaded a date range from 2011 through 2016, ~5 years. The uncompressed filesize for all select-able policing forces was approximately 5.45 GB. A quick C# script was written to filter out only the “Drug” crime type category, and it’s long/lat degree coordinates.

The output was used to make the map below. Click map for larger size.

The downloadable data comes in a multi-directory/multi-file structure. Below is a c# console application which was created to parse the root folder, and export the specific crime type’s coordinates throughout out all the files.
Continue reading →

Coke Piano The Coca Cola Company (KO) 2015 Daily Stock Value

1 Reply

Winforms c# application animating the daily market values of The Coca-Cola Company with piano emphasis.

Chicago Police Gun Related Incident Visualizations 2008 – 2016

Leave a reply

A look into Chicago Police Department’s gun related incident reports from January 2008 through February 2016. The data is from data.cityofchicago.org.

Stacked bar chart of monthly incidents by primary type.
Click chart image to expand

Line chart of monthly incidents by primary type.
Click chart image to expand