Assignment 3¶
Objectives of This Analysis¶
- First, I decided it would be interesting to look at the initial charges, themselves, to see if different groups tend to be charged with higher or lower class felonies from the beginning. Such differences might reflect biases in policing or in the early stages of prosecution, or could just be the result of different classes of people committing different crimes at different rates.
- My main goal was to analyze the generosity or fairness of the Cook County Attorney's office over the years. I created a dependent variable I called felony_class_difference and set equal to the change in felony class from arraignment to disposition for each defendant. I think this is an imperfect measure of fairness, but it is the best I could find from the data as it exists today. I then looked at how the independent variables race, gender and age impact the felony_class_difference to see if there appears to be any systematic bias for or against any particular groups.
Part 1: Reading in and exploring the initiation data¶
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns
from matplotlib.pyplot import figure
%matplotlib inline
import warnings
sns.set(style="darkgrid")
warnings.filterwarnings('ignore')
import datetime
initiation = pd.read_csv('Initiation.csv', low_memory=False)
initiation.head()
initiation.shape
sns.heatmap(initiation.isnull(),yticklabels=False,cbar=False,cmap='viridis')
initiation.drop(['INCIDENT_END_DATE','UNIT'], axis=1, inplace=True)
initiation.dropna(inplace=True)
sns.heatmap(initiation.isnull(),yticklabels=False,cbar=False,cmap='viridis')
initiation['CLASS'].value_counts()
initiation.columns
initiation_filtered = initiation[['CASE_PARTICIPANT_ID', 'CHARGE_ID', 'CHARGE_OFFENSE_TITLE', 'CLASS','AGE_AT_INCIDENT', 'GENDER',
'RACE', 'LAW_ENFORCEMENT_AGENCY', 'ARRAIGNMENT_DATE', 'UPDATED_OFFENSE_CATEGORY']]
initiation_filtered.shape
initiation_filtered.iloc[:,-2] = initiation_filtered.iloc[:, -2].apply(pd.to_datetime, errors='coerce')
initiation_filtered['arraignment_year'] = initiation_filtered.ARRAIGNMENT_DATE.dt.year
initiation_filtered.drop('ARRAIGNMENT_DATE', axis=1, inplace=True)
initiation_filtered.head()
initiation_filtered.rename(columns={'CASE_PARTICIPANT_ID':'participant_id', 'CHARGE_ID':'charge_id','CHARGE_OFFENSE_TITLE':'charge_offence_title',
'CLASS':'felony_class_initiation', 'AGE_AT_INCIDENT':'age','GENDER':'gender','RACE':'race',
'LAW_ENFORCEMENT_AGENCY':'law_enforcement_agency', 'UPDATED_OFFENSE_CATEGORY':'updated_offense_category'}, inplace=True)
initiation_filtered.head()
initiation_filtered = initiation_filtered[initiation_filtered['felony_class_initiation'].isin(['X','1','2','3','4'])]
initiation_filtered['felony_class_initiation'] = initiation_filtered['felony_class_initiation'].map({'X': 0,'1': 1,'2':2,'3':3, '4':4})
initiation_filtered.race.value_counts()
initiation_filtered['race'] = initiation_filtered['race'].str.lower()
initiation_filtered.race.value_counts()
initiation_filtered = initiation_filtered[initiation_filtered['race'].isin(['black',
'white [hispanic or latino] ', 'white', 'hispanic','asian',
'white/black [hispanic or latino]'])]
initiation_filtered.gender.value_counts()
initiation_filtered = initiation_filtered[initiation_filtered['gender'].isin(['Male', 'Female'])]
Data Visualization¶
ax = sns.countplot(initiation_filtered['race'])
ax.set_xticklabels(ax.get_xticklabels(), rotation=40, ha="right")
ax.set_title('Breakdown by Race of People Charged With Felonies in Cook County from 2011 to 2018')
plt.show()
Observation: Black people comprised more than 2/3 of all people charged with felonies during this time period.¶
ax = sns.countplot(initiation_filtered['gender'])
ax.set_xticklabels(ax.get_xticklabels(), rotation=40, ha="right")
ax.set_title('Breakdown by Gender of People Charged With Felonies in Cook County from 2011 to 2018')
plt.show()
Observation: Male felony defendants outnumber female by about nine to one.¶
Note: I included only Male and Female gender because the data for other categories of gender is insufficient.
initiation_filtered[['age','arraignment_year']] = initiation_filtered[['age','arraignment_year']].fillna(0.0).astype(int)
initiation_filtered.arraignment_year.value_counts()
I decided to work with data between 2011 and 2018 only because the data for the other years is incomplete.
initiation_filtered = initiation_filtered[(initiation_filtered['arraignment_year'] > 2010) & (initiation_filtered['arraignment_year'] < 2019)]
initiation_filtered.arraignment_year.value_counts()
multi_plot = initiation_filtered.groupby(['arraignment_year', 'race']).felony_class_initiation.mean()
multi_plot = multi_plot.unstack()
multi_plot
multi_plot.loc[:, :].plot()
plt.legend(bbox_to_anchor=(1.75, 1), loc='upper right', ncol=1)
plt.plot(multi_plot['black'], marker='', color='black', linewidth=3, alpha=0.7)
plt.plot(multi_plot['white'], marker='', color='white', linewidth=3, alpha=1)
plt.plot(multi_plot['hispanic'], marker='', color='green', linewidth=2, alpha=1)
ax = plt.gca()
leg = ax.get_legend()
leg.legendHandles[1].set_color('black')
leg.legendHandles[3].set_color('white')
plt.ylabel('Felony class')
plt.title('Initial felony class for each race by year')
plt.show()
Observations:
The average felony class for people in the Hispanic class was much lower compared to the average of all other races.
People in the Black, Asian, and White/Black [Hispanic or Lation} categories were on average charged with the more serious felonies, although there were years where people in the latter two categories moved into the middle range occupied by people in the White and White [Hispanic or Latino] categories.
by_gender = initiation_filtered.groupby(['arraignment_year', 'gender']).felony_class_initiation.mean()
by_gender = by_gender.unstack()
by_gender
by_gender.loc[:, :].plot()
plt.legend(bbox_to_anchor=(1, 1), loc='upper right', ncol=1)
plt.ylabel('Felony class')
plt.title('Initial felony class for each gender by year')
plt.show()
Observation: The average felony class for Female is much lower compared to the average of the Male gender. Note: I did not include defendants from other gender classes because their numbers were too small to be statistically meaningful.
Part2: Exploring changes in felony class as defendants move from initiation through disposition. This entailed merging two data sets.¶
disposition = pd.read_csv('Dispositions.csv', low_memory=False)
disposition.head()
disposition.shape
disposition.info()
merged = pd.merge(initiation, disposition, on=['CASE_PARTICIPANT_ID', 'CHARGE_ID'])
merged.shape
merged.head()
merged_2 = merged[['CHARGE_ID', 'UPDATED_OFFENSE_CATEGORY_x',
'AGE_AT_INCIDENT_x','GENDER_x','RACE_x',
'ARREST_DATE_x','ARRAIGNMENT_DATE_x',
'DISPO_DATE','LAW_ENFORCEMENT_AGENCY_x',
'CLASS_x', 'CLASS_y', 'CHARGE_DISPOSITION']]
merged_2.iloc[:,5:8].head(1)
merged_2.iloc[:,5:8] = merged_2.iloc[:, 5:8].apply(pd.to_datetime, errors='coerce')
merged_2['arrest_year'] = merged_2.ARREST_DATE_x.dt.year
merged_2['arrest_month'] = merged_2.ARREST_DATE_x.dt.month
merged_2['arraignment_year'] = merged_2.ARRAIGNMENT_DATE_x.dt.year
merged_2['arraignment_month'] = merged_2.ARRAIGNMENT_DATE_x.dt.month
merged_2['disposition_year'] = merged_2.DISPO_DATE.dt.year
merged_2['disposition_month'] = merged_2.DISPO_DATE.dt.month
merged_2.rename(columns={'UPDATED_OFFENSE_CATEGORY_x':'type_of_crime','AGE_AT_INCIDENT_x':'age',
'GENDER_x':'gender','RACE_x':'race', 'LAW_ENFORCEMENT_AGENCY_x':'law_enforcement_agency',
'CHARGE_DISPOSITION':'charge_disposition', 'CLASS_x':'felony_class_initiation',
'CLASS_y':'felony_class_disposition', 'ARREST_DATE_x':'arrest_date',
'ARRAIGNMENT_DATE_x':'arraignment_date','DISPO_DATE':'disposition_date'}, inplace=True)
merged_2.head()
merged_3 = merged_2[merged_2['felony_class_initiation'].isin(['X','1','2','3','4'])
& merged_2['felony_class_disposition'].isin(['X','1','2','3','4'])]
merged_3 = merged_3[merged_3['gender'].isin(['Male','Female'])]
merged_3.shape
In order to get a quantitative measure of changes in felony class as a defendant moved from initiation to disposition, I had to first convert the most serious class from the non-numeric X to the numeric 0. The other four felony classes were already in numeric form, ranging from 1 (the most serious after X) to 4 (the least serious). I then defined felony_class_difference as the change in felonly class value from initiation to dispositon. For example, a defendant moving from Class 0 to Class 3 would have a felony_class_difference of 3, while one moving from 2 to 3 would have felony_class_difference of 1.
Note: there were some defendents whose felony class actually became more severe (e.g. moved from a 1 to a 0 or a 3 to 2). I did not include these in the analysis because their numbers were small and not very statistically meaningful. However, they might be worth investigating in a later analysis.
merged_3['felony_class_initiation'] = merged_3['felony_class_initiation'].map({'X': 0,'1': 1,'2':2,'3':3, '4':4})
merged_3['felony_class_disposition'] = merged_3['felony_class_disposition'].map({'X': 0, '1': 1,'2':2,'3':3, '4':4})
merged_3['felony_class_difference'] = merged_3['felony_class_disposition'] - merged_3['felony_class_initiation']
merged_3['felony_class_difference'].value_counts()
merged_3 = merged_3[merged_3['felony_class_difference'].isin([0,1,2,3,4])]
merged_3.head()
merged_3.info()
merged_3[['age','arraignment_year','disposition_year', 'arrest_year']] = merged_3[['age','arraignment_year',
'disposition_year', 'arrest_year']].fillna(0.0).astype(int)
merged_3 = merged_3[(merged_3.arraignment_year < 2019) & (merged_3.arraignment_year > 2010) ]
merged_3.arraignment_year.value_counts()
merged_3.race.value_counts()
merged_3['race'] = merged_3['race'].str.lower()
merged_3.race.value_counts()
merged_3 = merged_3[merged_3['race'].isin(['black',
'white [hispanic or latino]',
'white', 'hispanic','asian',
'white/black [hispanic or latino]'])]
grouped_by_race = merged_3.groupby(['arraignment_year', 'race']).felony_class_difference.mean()
grouped_by_race = grouped_by_race.unstack()
grouped_by_race
grouped_by_race.loc[:, :].plot()
plt.legend(bbox_to_anchor=(1.75, 1), loc='upper right', ncol=1)
plt.plot(grouped_by_race['black'], marker='', color='black', linewidth=3, alpha=0.7)
plt.plot(grouped_by_race['white'], marker='', color='white', linewidth=3, alpha=1)
plt.plot(grouped_by_race['hispanic'], marker='', color='green', linewidth=2, alpha=1)
ax = plt.gca()
leg = ax.get_legend()
leg.legendHandles[1].set_color('black')
leg.legendHandles[3].set_color('white')
plt.ylabel('Felony class drop')
plt.title('Average felony class drop by year for every race')
plt.show()
Observations: Upon initial examination, one might think the system is biased towards Black people and against Hispanic people since the data shows Blacks and White/Black [Hispanic or Latino] having the largest average reductions in felony class from initiation to disposition and Hispanics having the least. However, this may in part be explained by the observation made in Part 1 above that Hispanics had much lower average felonly class averages at initiation than Black or White/Black [Hispanic or Latino} people. Since Hispanic defendants started with lesser charges, there was less room for them to move down during the plea bargain process. Asians were the one class that started out with among the most severe average charges yet had among the lowest average felony_class_drop of any of the racial groups. This does not prove bias, since it's possible that the circumstances of their individual cases could have warranted this, but it does suggest that further investigation may be warranted.
grouped_by_gender = merged_3.groupby(['arraignment_year', 'gender']).felony_class_difference.mean()
grouped_by_gender = grouped_by_gender.unstack()
grouped_by_gender.loc[:, :].plot()
plt.legend(bbox_to_anchor=(1, 1), loc='upper right', ncol=1)
plt.ylabel('Felony class drop')
plt.title('Average felony class drop for each gender by year')
plt.show()
Observations: In recent years except for 2018, Female defendants have enjoyed larger average felony_class_drops than their Male counterparts, and this is despite the observation made in Part 1 that Females began with much less serious initial charges. Again, this does not prove bias, but it does warrant further investigation.
I created 5 age categories in order to create heatmap. I mapped each age category to a number from 0 to 6, e.g. 0 = teenager (13-18), 1 = young adult (18-25), 2 = adult (25-40), etc.
merged_3.loc[merged_3['age'] <= 18, 'age'] = 0
merged_3.loc[(merged_3['age'] > 18) & (merged_3['age'] <= 25), 'age'] = 1
merged_3.loc[(merged_3['age'] > 25) & (merged_3['age'] <= 40), 'age'] = 2
merged_3.loc[(merged_3['age'] > 40) & (merged_3['age'] <= 55), 'age'] = 3
merged_3.loc[(merged_3['age'] > 55) & (merged_3['age'] <= 70), 'age'] = 4
merged_3.loc[merged_3['age'] > 70, 'age'] = 5
grouped_by_age = merged_3.groupby(['arraignment_year', 'age']).felony_class_difference.mean()
grouped_by_age = grouped_by_age.unstack()
grouped_by_age
Seems like the most of the felony class drop occurs between the age 40 and 70, especially in recent years.
ax = plt.axes()
sns.heatmap(grouped_by_age,cmap="YlGnBu")
ax.set_title('Average felony class drop for each age rank by year')
plt.show()
Part 3: Analyzing types of crimes¶
merged_3.head()
merged_3['type_of_crime'].nunique()
f, ax = plt.subplots(figsize=(18,5))
ax = merged_3['type_of_crime'].value_counts().plot(kind='bar')
plt.title('Breakdown by Type of Crime in Cook County from 2011 to 2018')
plt.xlabel('Type of Crime')
plt.ylabel('Number of Crime')
merged_4 = merged_3
merged_4['felony_class_difference'] = merged_3.felony_class_difference.astype('object')
merged_4 = merged_3[merged_3['felony_class_difference'].isin([1,2,3,4])]
felony_charge_drop_by_race = pd.crosstab(index=merged_4['race'], columns=merged_4['felony_class_difference'], normalize='index')
felony_charge_drop_by_race
felony_charge_drop_by_race.plot(kind="bar", figsize=(12,8), stacked=True)
plt.title('The Percentage of Each Felony Charge Drop by Race for 2011-2018')
plt.show()
Observations: Even if we mentioned in Part 2 that Hispanic people had the lowest initial felony classes, we notice from the plot above, they are still getting pretty significant felony drop. Black and White/Black[Hispanic or Latino] have been getting the highest felony drops among the rest.
by_crime = merged_3[merged_3['type_of_crime'].isin(['UUW - Unlawful Use of Weapon',
'Narcotics', 'Aggravated DUI', 'Sex Crimes',
'Driving With Suspended Or Revoked License'])]
by_crime = merged_3[merged_3['type_of_crime'].isin(['UUW - Unlawful Use of Weapon',
'Narcotics', 'Aggravated DUI', 'Sex Crimes',
'Driving With Suspended Or Revoked License'])]
Below I made the table using the highest crime types' values and showed their percentage by each race.
crime_table = pd.crosstab(index=by_crime['race'], columns=by_crime['type_of_crime'],normalize='index')
crime_table
crime_table.plot(kind="bar", figsize=(12,8), stacked=True)
plt.title('The Percentage of Each Types of Crime by Race for 2011-2018')
plt.show()
Observations: It seems there is a similar pattern between Black and White/Black[Hispanic or Latino] peoples' crime types. Both races lead on UUW following by narcotics. Asian people have the highest record on Sex Crimes.
Idea for Further Improvements to Analysis¶
The aspect of the work done by the Cook County Attorney's Office that most impacts a defendant is what sentence that person ultimately receives at disposition. It would be very interesting to repeat the above analysis but replacing felony_class_difference as the dependent variable with something like "actual_sentence_versus_expectations", which I would define as follows:
- find the average sentence given in Cook County for all defendants convicted of a given crime during the period (from 2011 to 2018, say) and call this the "expected_sentence".
- add a column to the database that shows what sentence each defendant that plead guilty actually received at disposition. Call this the "actual_sentence". I am sure this data exists but wasn't sure where to find it.
- for each defendant who took a plea bargain during the period, subtract the actual_sentence they received at disposition from the expected_sentence for the most serious crime they were charged with at initiation to get the actual_sentence_versus_expectations for that defendant.
After seeing the data, I may want to make some further analyses related to the above. For example, for defendants with multiple charges against them, it might also be interesting to add up the expected sentences for all the charges against them at initiation to get a "total_expected_sentence" at initiation, then compare this to the actual_sentence in the same manner. It might also be necessary to make some adjustments to the data to avoid distortions. For example, if a defendant is sentenced to multiple lifetimes, I might treat that as one lifetime of say 60 years or something since, in terms of what it actually means to a defendant, one lifetime is no different than 10 or a thousand.
- I would then look at how the various independent variables (race, gender, age) affect the actual_sentence_versus_expectations.