Project Goals¶
Human migration is one of the defining global trends of the 21st century, whether it is political, economic, or conflit-driven. The goal of this project is to investigate patterns in the movement of Ukrainian refugees as a result of the February 2022 Russian invasion. I will also be analyzing the relationships between refugee flows and host country indicators, with the final goal of identifying the political and economic indicators that are most predictive of having larger refugee populations.
The Ukrainian refugee crisis is the fastest growing and largest displacement crisis in Europe since WWII (UNHCR). Political institutional quality and economic strength are big determinants of integration success, and knowing which indicators are most related to refugee flows can help governments and aid agencies help refugees more effectively.
Collaboration Plan (N/A)¶
As a Master's student, I am working alone and will not need a specialized plan.
ETL: Project Dataset 1¶
The first dataset that I will use comes from the Refugee Data Finder at the United Nations High Commissioner for Refugees (UNHCR), the leading international institution on migration data. This database contains yearly population counts of refugee, asylum-seeker, and internally displaced persons, as well as their countries of origin and asylum. I am interested in this dataset because it is the most basic form of tracking refugee migration destinations. I will use it to answer questions like "Which countries have received the largest volumes of refugees?" or "Have refugee populations increased, decreased, or stabilized over the course of the war?"
%cd /content/drive/MyDrive/CMPS 6160 (Data Sci)/Proj Data
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns
/content/drive/MyDrive/CMPS 6160 (Data Sci)/Proj Data
df_RefCounts_Orig = pd.read_csv('UKR Refugee Counts.csv')
df_RefCounts_Orig
| Year | Country of Asylum | Country of Origin | Country of Asylum ISO | Country of Origin ISO | Refugees | Asylum-seekers | IDPs | Other people in need of international protection | |
|---|---|---|---|---|---|---|---|---|---|
| 0 | 2022 | Albania | Ukraine | ALB | UKR | 2542 | 5 | 0 | 0 |
| 1 | 2023 | Albania | Ukraine | ALB | UKR | 4959 | 0 | 0 | 0 |
| 2 | 2024 | Albania | Ukraine | ALB | UKR | 9250 | 0 | 0 | 0 |
| 3 | 2022 | Egypt | Ukraine | EGY | UKR | 0 | 5 | 0 | 0 |
| 4 | 2023 | Egypt | Ukraine | EGY | UKR | 0 | 9 | 0 | 0 |
| ... | ... | ... | ... | ... | ... | ... | ... | ... | ... |
| 236 | 2023 | Uruguay | Ukraine | URY | UKR | 18 | 0 | 0 | 0 |
| 237 | 2024 | Uruguay | Ukraine | URY | UKR | 18 | 5 | 0 | 0 |
| 238 | 2022 | United States of America | Ukraine | USA | UKR | 3063 | 12369 | 0 | 0 |
| 239 | 2023 | United States of America | Ukraine | USA | UKR | 3299 | 17773 | 0 | 0 |
| 240 | 2024 | United States of America | Ukraine | USA | UKR | 3397 | 18658 | 0 | 0 |
241 rows × 9 columns
Because we are only studying refugee populations originating from Ukraine and going to other countries, we can drop the variables about country of origin and drop the three observations about Ukraine itself. I will also drop the "Other people in need of international protection" and "IDPs" columns because they contain no relevant data to my study and are filled only with zeros. I will also rename some of the asylum countries for convenience sake. Lastly, I will create a joint measure of refugees and asylum seekers. Since the distinction between the two classifications is primarily legal, if we want to know generally about all the people the war has displaced, we should use a combined count.
# Dropping variables and UKR observations:
df_RefCounts = df_RefCounts_Orig.drop(columns=['Country of Origin', 'Country of Origin ISO', 'Other people in need of international protection', 'IDPs'])
df_RefCounts = df_RefCounts[df_RefCounts['Country of Asylum ISO'] != 'UKR']
# Renaming countries:
df_RefCounts["Country of Asylum"] = df_RefCounts["Country of Asylum"].replace({
'Bolivia (Plurinational State of)': 'Bolivia',
'United Kingdom of Great Britain and Northern Ireland': 'United Kingdom',
'United States of America': 'United States',
'Rep. of Korea': 'South Korea',
'Rep. of Moldova': 'Moldova',
'Netherlands (Kingdom of the)': 'Netherlands',
'Russian Federation': 'Russia',
'Serbia and Kosovo: S/RES/1244 (1999)': 'Serbia',
'United Rep. of Tanzania': 'Tanzania'
})
#Joint measure
df_RefCounts['Ref_Asylum'] = df_RefCounts['Refugees'] + df_RefCounts['Asylum-seekers']
df_RefCounts
| Year | Country of Asylum | Country of Asylum ISO | Refugees | Asylum-seekers | Ref_Asylum | |
|---|---|---|---|---|---|---|
| 0 | 2022 | Albania | ALB | 2542 | 5 | 2547 |
| 1 | 2023 | Albania | ALB | 4959 | 0 | 4959 |
| 2 | 2024 | Albania | ALB | 9250 | 0 | 9250 |
| 3 | 2022 | Egypt | EGY | 0 | 5 | 5 |
| 4 | 2023 | Egypt | EGY | 0 | 9 | 9 |
| ... | ... | ... | ... | ... | ... | ... |
| 236 | 2023 | Uruguay | URY | 18 | 0 | 18 |
| 237 | 2024 | Uruguay | URY | 18 | 5 | 23 |
| 238 | 2022 | United States | USA | 3063 | 12369 | 15432 |
| 239 | 2023 | United States | USA | 3299 | 17773 | 21072 |
| 240 | 2024 | United States | USA | 3397 | 18658 | 22055 |
238 rows × 6 columns
The data above is in a basic tidy format. While proper database management would create a system of relational tables, this will suffice for our purposes. Next, we must check the dtypes. We see that they are already correctly coded, with our categorical variables as objects and our quantitative variables as int64:
df_RefCounts.dtypes
| 0 | |
|---|---|
| Year | int64 |
| Country of Asylum | object |
| Country of Asylum ISO | object |
| Refugees | int64 |
| Asylum-seekers | int64 |
| Ref_Asylum | int64 |
ETL: Project Dataset 2¶
My next dataset comes from the World Bank's World Development Indicators (WDI). This database offers more than 1,500 different variables at the national level and covers broad topics like the environment, education, and inequality, to name just a few. I used their built-in database query interface to select variables focusing on my outcomes of interest, primarily those related to the economy, education, and demography. I also narrowed the database down to the three years of interest (2022-2024). Later on, I will conduct a left join on country and year using the df_RefCounts as df1, so it only keeps country/year combinations that have recorded Ukrainian refugee data.
With this data, I hope to build contextual profiles for each country of asylum. It will help me answer questions like, "What are the strongest correlates of refugee population in the context of this conflict?", and "How much of the refugee population is absorbed by countries with high vs. low levels of development?"
df_WDI_Orig = pd.read_csv('WDI 1.csv')
df_WDI_Orig
| Country Name | Country Code | Series Name | Series Code | 2022 [YR2022] | 2023 [YR2023] | 2024 [YR2024] | |
|---|---|---|---|---|---|---|---|
| 0 | Afghanistan | AFG | Adjusted net enrollment rate, primary (% of pr... | SE.PRM.TENR | .. | .. | .. |
| 1 | Afghanistan | AFG | Adjusted net national income per capita (curre... | NY.ADJ.NNTY.PC.CD | .. | .. | .. |
| 2 | Afghanistan | AFG | Coverage of social safety net programs (% of p... | per_sa_allsa.cov_pop_tot | .. | .. | .. |
| 3 | Afghanistan | AFG | Educational attainment, at least Bachelor's or... | SE.TER.CUAT.BA.ZS | 3.98425006866455 | .. | .. |
| 4 | Afghanistan | AFG | Educational attainment, at least completed pri... | SE.PRM.CUAT.ZS | 21.0254802703857 | .. | .. |
| ... | ... | ... | ... | ... | ... | ... | ... |
| 10416 | NaN | NaN | NaN | NaN | NaN | NaN | NaN |
| 10417 | NaN | NaN | NaN | NaN | NaN | NaN | NaN |
| 10418 | NaN | NaN | NaN | NaN | NaN | NaN | NaN |
| 10419 | Data from database: World Development Indicators | NaN | NaN | NaN | NaN | NaN | NaN |
| 10420 | Last Updated: 10/07/2025 | NaN | NaN | NaN | NaN | NaN | NaN |
10421 rows × 7 columns
Upon loading in this table, we see we must reshape it from wide to long format. First, let's melt the years data into one column, clean up the syntax of the Year column, and then pivot the table so each indicator is its own column.
# Melt
df_WDI_melt = df_WDI_Orig.melt(
id_vars=['Country Name', 'Country Code', 'Series Name', 'Series Code'],
var_name='Year',
value_name='Value'
)
# Replace year values to clean syntax and convert to int64s
df_WDI_melt['Year'] = df_WDI_melt['Year'].replace({
'2022 [YR2022]': '2022',
'2023 [YR2023]': '2023',
'2024 [YR2024]': '2024'
}).astype(int)
# Before pivoting must drop the footnote observations at the end.
display(df_WDI_melt.tail(6))
df_WDI_melt = df_WDI_melt.iloc[:31258]
| Country Name | Country Code | Series Name | Series Code | Year | Value | |
|---|---|---|---|---|---|---|
| 31257 | Zimbabwe | ZWE | Unemployment, male (% of male labor force) (na... | SL.UEM.TOTL.MA.NE.ZS | 2024 | .. |
| 31258 | NaN | NaN | NaN | NaN | 2024 | NaN |
| 31259 | NaN | NaN | NaN | NaN | 2024 | NaN |
| 31260 | NaN | NaN | NaN | NaN | 2024 | NaN |
| 31261 | Data from database: World Development Indicators | NaN | NaN | NaN | 2024 | NaN |
| 31262 | Last Updated: 10/07/2025 | NaN | NaN | NaN | 2024 | NaN |
I was still getting a Duplicate Index Entries error when pivoting, so let's search for and drop duplicates:
dupes = df_WDI_melt[df_WDI_melt.duplicated(
subset=['Country Name', 'Country Code', 'Year', 'Series Name'],
keep=False
)]
print(dupes)
df_WDI_melt = df_WDI_melt.drop(index=dupes.index)
Country Name Country Code Series Name Series Code Year Value 10416 NaN NaN NaN NaN 2022 NaN 10417 NaN NaN NaN NaN 2022 NaN 10418 NaN NaN NaN NaN 2022 NaN 20837 NaN NaN NaN NaN 2023 NaN 20838 NaN NaN NaN NaN 2023 NaN 20839 NaN NaN NaN NaN 2023 NaN
# Now pivot
df_WDI_pivot = df_WDI_melt.pivot(
index=['Country Name', 'Country Code', 'Year'],
columns='Series Name',
values='Value'
).reset_index()
# Drop the column labeled NaN
df_WDI_pivot.drop(columns=[np.nan], inplace=True)
df_WDI_pivot
| Series Name | Country Name | Country Code | Year | Adjusted net enrollment rate, primary (% of primary school age children) | Adjusted net national income per capita (current US$) | Coverage of social safety net programs (% of population) | Educational attainment, at least Bachelor's or equivalent, population 25+, total (%) (cumulative) | Educational attainment, at least completed primary, population 25+ years, total (%) (cumulative) | Educational attainment, at least completed upper secondary, population 25+, total (%) (cumulative) | GDP growth (annual %) | ... | School enrollment, secondary (% net) | School enrollment, tertiary (% gross) | Unemployment with advanced education (% of total labor force with advanced education) | Unemployment with basic education (% of total labor force with basic education) | Unemployment, female (% of female labor force) (modeled ILO estimate) | Unemployment, female (% of female labor force) (national estimate) | Unemployment, male (% of male labor force) (modeled ILO estimate) | Unemployment, male (% of male labor force) (national estimate) | Unemployment, total (% of total labor force) (modeled ILO estimate) | Unemployment, total (% of total labor force) (national estimate) |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 0 | Afghanistan | AFG | 2022 | .. | .. | .. | 3.98425006866455 | 21.0254802703857 | 12.5973300933838 | -6.24017199240269 | ... | .. | .. | .. | .. | 26.742 | .. | 13.168 | .. | 14.1 | .. |
| 1 | Afghanistan | AFG | 2023 | .. | .. | .. | .. | .. | .. | 2.26694373649188 | ... | .. | .. | .. | .. | 26.57 | .. | 13.066 | .. | 13.991 | .. |
| 2 | Afghanistan | AFG | 2024 | .. | .. | .. | .. | .. | .. | .. | ... | .. | .. | .. | .. | 25.636 | .. | 12.394 | .. | 13.295 | .. |
| 3 | Albania | ALB | 2022 | .. | .. | .. | 19.8346436163563 | 97.8531397166699 | 53.2684854264147 | 4.82669611912765 | ... | .. | 62.647921513098 | 10.228 | 9.437 | 10.405 | 11.235 | 9.917 | 10.398 | 10.137 | 10.785 |
| 4 | Albania | ALB | 2023 | .. | .. | .. | 17.5100002288818 | 99.2799987792969 | 53.9700012207031 | 3.93661689192599 | ... | .. | 64.7293520237257 | 8.947 | 9.694 | 10.376 | 10.958 | 9.888 | 10.421 | 10.108 | 10.669 |
| ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... |
| 650 | Zambia | ZMB | 2023 | .. | .. | .. | 2.80119454477244 | .. | 33.5884544067257 | 5.36794285493059 | ... | .. | .. | 9.702 | 3.688 | 6.207 | 6.037 | 5.646 | 5.71 | 5.905 | 5.861 |
| 651 | Zambia | ZMB | 2024 | .. | .. | .. | .. | .. | .. | 4.0431482994443 | ... | .. | .. | .. | .. | 6.25 | .. | 5.713 | .. | 5.961 | .. |
| 652 | Zimbabwe | ZWE | 2022 | .. | .. | .. | 0.913877996562985 | .. | .. | 6.1391761744749 | ... | .. | .. | 5.945 | 8.878 | 10.562 | 10.461 | 9.615 | 9.554 | 10.087 | 10.001 |
| 653 | Zimbabwe | ZWE | 2023 | .. | .. | .. | 1.13487348253901 | .. | .. | 5.33677037339527 | ... | .. | .. | 5.933 | 8.493 | 8.824 | 9.46 | 8.694 | 9.126 | 8.759 | 9.29 |
| 654 | Zimbabwe | ZWE | 2024 | .. | .. | .. | .. | .. | .. | 2.02948414337123 | ... | .. | 7.74912957845971 | .. | .. | 8.587 | .. | 8.521 | .. | 8.554 | .. |
655 rows × 51 columns
Shape looks good and footnotes are gone. Now we can check our dtypes for the WDI dataset, but it doesn't look good. All of our variables besides Country Name and Country Code should be numeric:
df_WDI_pivot.dtypes
| 0 | |
|---|---|
| Series Name | |
| Country Name | object |
| Country Code | object |
| Year | int64 |
| Adjusted net enrollment rate, primary (% of primary school age children) | object |
| Adjusted net national income per capita (current US$) | object |
| Coverage of social safety net programs (% of population) | object |
| Educational attainment, at least Bachelor's or equivalent, population 25+, total (%) (cumulative) | object |
| Educational attainment, at least completed primary, population 25+ years, total (%) (cumulative) | object |
| Educational attainment, at least completed upper secondary, population 25+, total (%) (cumulative) | object |
| GDP growth (annual %) | object |
| GDP per capita (current US$) | object |
| GDP per capita growth (annual %) | object |
| GNI per capita growth (annual %) | object |
| GNI per capita, Atlas method (current US$) | object |
| Gini index | object |
| Government Effectiveness: Percentile Rank | object |
| Labor force participation rate for ages 15-24, total (%) (modeled ILO estimate) | object |
| Labor force participation rate for ages 15-24, total (%) (national estimate) | object |
| Labor force participation rate, total (% of total population ages 15+) (modeled ILO estimate) | object |
| Labor force participation rate, total (% of total population ages 15+) (national estimate) | object |
| Labor force participation rate, total (% of total population ages 15-64) (modeled ILO estimate) | object |
| Labor force with advanced education (% of total working-age population with advanced education) | object |
| Labor force with basic education (% of total working-age population with basic education) | object |
| Labor force with intermediate education (% of total working-age population with intermediate education) | object |
| Labor force, female (% of total labor force) | object |
| Net official flows from UN agencies, UNHCR (current US$) | object |
| Political Stability and Absence of Violence/Terrorism: Percentile Rank | object |
| Population ages 15-64 (% of total population) | object |
| Population ages 15-64, female | object |
| Population ages 15-64, male | object |
| Population ages 15-64, total | object |
| Population growth (annual %) | object |
| Population, female (% of total population) | object |
| Population, male (% of total population) | object |
| Poverty headcount ratio at national poverty lines (% of population) | object |
| Prevalence of moderate or severe food insecurity in the population (%) | object |
| Primary completion rate, total (% of relevant age group) | object |
| Rule of Law: Percentile Rank | object |
| School enrollment, primary (% gross) | object |
| School enrollment, primary (% net) | object |
| School enrollment, secondary (% gross) | object |
| School enrollment, secondary (% net) | object |
| School enrollment, tertiary (% gross) | object |
| Unemployment with advanced education (% of total labor force with advanced education) | object |
| Unemployment with basic education (% of total labor force with basic education) | object |
| Unemployment, female (% of female labor force) (modeled ILO estimate) | object |
| Unemployment, female (% of female labor force) (national estimate) | object |
| Unemployment, male (% of male labor force) (modeled ILO estimate) | object |
| Unemployment, male (% of male labor force) (national estimate) | object |
| Unemployment, total (% of total labor force) (modeled ILO estimate) | object |
| Unemployment, total (% of total labor force) (national estimate) | object |
# Select numeric columns
numeric_cols = df_WDI_pivot.columns.difference(['Country Name', 'Country Code'])
# Force to numeric
df_WDI_pivot[numeric_cols] = df_WDI_pivot[numeric_cols].apply(pd.to_numeric, errors='coerce')
# Also, fill blank values with NaN instead of ".."
df_WDI_pivot.replace('..', np.nan, inplace=True)
display(df_WDI_pivot.dtypes)
#Much better!
| 0 | |
|---|---|
| Series Name | |
| Country Name | object |
| Country Code | object |
| Year | int64 |
| Adjusted net enrollment rate, primary (% of primary school age children) | float64 |
| Adjusted net national income per capita (current US$) | float64 |
| Coverage of social safety net programs (% of population) | float64 |
| Educational attainment, at least Bachelor's or equivalent, population 25+, total (%) (cumulative) | float64 |
| Educational attainment, at least completed primary, population 25+ years, total (%) (cumulative) | float64 |
| Educational attainment, at least completed upper secondary, population 25+, total (%) (cumulative) | float64 |
| GDP growth (annual %) | float64 |
| GDP per capita (current US$) | float64 |
| GDP per capita growth (annual %) | float64 |
| GNI per capita growth (annual %) | float64 |
| GNI per capita, Atlas method (current US$) | float64 |
| Gini index | float64 |
| Government Effectiveness: Percentile Rank | float64 |
| Labor force participation rate for ages 15-24, total (%) (modeled ILO estimate) | float64 |
| Labor force participation rate for ages 15-24, total (%) (national estimate) | float64 |
| Labor force participation rate, total (% of total population ages 15+) (modeled ILO estimate) | float64 |
| Labor force participation rate, total (% of total population ages 15+) (national estimate) | float64 |
| Labor force participation rate, total (% of total population ages 15-64) (modeled ILO estimate) | float64 |
| Labor force with advanced education (% of total working-age population with advanced education) | float64 |
| Labor force with basic education (% of total working-age population with basic education) | float64 |
| Labor force with intermediate education (% of total working-age population with intermediate education) | float64 |
| Labor force, female (% of total labor force) | float64 |
| Net official flows from UN agencies, UNHCR (current US$) | float64 |
| Political Stability and Absence of Violence/Terrorism: Percentile Rank | float64 |
| Population ages 15-64 (% of total population) | float64 |
| Population ages 15-64, female | float64 |
| Population ages 15-64, male | float64 |
| Population ages 15-64, total | float64 |
| Population growth (annual %) | float64 |
| Population, female (% of total population) | float64 |
| Population, male (% of total population) | float64 |
| Poverty headcount ratio at national poverty lines (% of population) | float64 |
| Prevalence of moderate or severe food insecurity in the population (%) | float64 |
| Primary completion rate, total (% of relevant age group) | float64 |
| Rule of Law: Percentile Rank | float64 |
| School enrollment, primary (% gross) | float64 |
| School enrollment, primary (% net) | float64 |
| School enrollment, secondary (% gross) | float64 |
| School enrollment, secondary (% net) | float64 |
| School enrollment, tertiary (% gross) | float64 |
| Unemployment with advanced education (% of total labor force with advanced education) | float64 |
| Unemployment with basic education (% of total labor force with basic education) | float64 |
| Unemployment, female (% of female labor force) (modeled ILO estimate) | float64 |
| Unemployment, female (% of female labor force) (national estimate) | float64 |
| Unemployment, male (% of male labor force) (modeled ILO estimate) | float64 |
| Unemployment, male (% of male labor force) (national estimate) | float64 |
| Unemployment, total (% of total labor force) (modeled ILO estimate) | float64 |
| Unemployment, total (% of total labor force) (national estimate) | float64 |
Due to variation in data availability and measurement, it is common for some World Development Indicators to contain no data. Here, we'll drop any variables that contain no information for any of our country-year observations. We will also recode the burdensome variable names exported from the WDIs.
#Drop columns that have NaN for all values
df_WDI_pivot.dropna(axis=1, how='all', inplace=True)
#Rename using dictionary
rename_dict = {
'Country Name': 'Country Name',
'Country Code': 'ISO',
'Year': 'Year',
'Coverage of social safety net programs (% of population)': 'SocialNet_Cov',
"Educational attainment, at least Bachelor's or equivalent, population 25+, total (%) (cumulative)": 'Edu_Bach25p',
'Educational attainment, at least completed primary, population 25+ years, total (%) (cumulative)': 'Edu_Prim25p',
'Educational attainment, at least completed upper secondary, population 25+, total (%) (cumulative)': 'Edu_UpSec25p',
'GDP growth (annual %)': 'GDP_Growth',
'GDP per capita (current US$)': 'GDP_pc',
'GDP per capita growth (annual %)': 'GDP_pc_Growth',
'GNI per capita growth (annual %)': 'GNI_pc_Growth',
'GNI per capita, Atlas method (current US$)': 'GNI_pc',
'Gini index': 'Gini',
'Government Effectiveness: Percentile Rank': 'GovEffect_rank',
'Labor force participation rate for ages 15-24, total (%) (modeled ILO estimate)': 'LFPR_15_24_ILO',
'Labor force participation rate for ages 15-24, total (%) (national estimate)': 'LFPR_15_24_Nat',
'Labor force participation rate, total (% of total population ages 15+) (modeled ILO estimate)': 'LFPR_15p_ILO',
'Labor force participation rate, total (% of total population ages 15+) (national estimate)': 'LFPR_15p_Nat',
'Labor force participation rate, total (% of total population ages 15-64) (modeled ILO estimate)': 'LFPR_15_64_ILO',
'Labor force with advanced education (% of total working-age population with advanced education)': 'LF_AdvEdu',
'Labor force with basic education (% of total working-age population with basic education)': 'LF_BasicEdu',
'Labor force with intermediate education (% of total working-age population with intermediate education)': 'LF_IntEdu',
'Labor force, female (% of total labor force)': 'LF_Fem',
'Net official flows from UN agencies, UNHCR (current US$)': 'Net_UNHCR',
'Political Stability and Absence of Violence/Terrorism: Percentile Rank': 'Pol_Stab_rank',
'Population ages 15-64 (% of total population)': 'Pop_15_64_pct',
'Population ages 15-64, female': 'Pop_15_64_Fem',
'Population ages 15-64, male': 'Pop_15_64_Male',
'Population ages 15-64, total': 'Pop_15_64_Tot',
'Population growth (annual %)': 'Pop_Growth',
'Population, female (% of total population)': 'Pop_Fem_pct',
'Population, male (% of total population)': 'Pop_Male_pct',
'Poverty headcount ratio at national poverty lines (% of population)': 'Poverty_pct',
'Prevalence of moderate or severe food insecurity in the population (%)': 'FoodInsec_pct',
'Primary completion rate, total (% of relevant age group)': 'PrimComp_pct',
'Rule of Law: Percentile Rank': 'RuleLaw_rank',
'School enrollment, primary (% gross)': 'Enroll_Prim',
'School enrollment, secondary (% gross)': 'Enroll_Sec',
'School enrollment, tertiary (% gross)': 'Enroll_Tert',
'Unemployment with advanced education (% of total labor force with advanced education)': 'Unemp_AdvEdu',
'Unemployment with basic education (% of total labor force with basic education)': 'Unemp_BasicEdu',
'Unemployment, female (% of female labor force) (modeled ILO estimate)': 'Unemp_Fem_ILO',
'Unemployment, female (% of female labor force) (national estimate)': 'Unemp_Fem_Nat',
'Unemployment, male (% of male labor force) (modeled ILO estimate)': 'Unemp_Male_ILO',
'Unemployment, male (% of male labor force) (national estimate)': 'Unemp_Male_Nat',
'Unemployment, total (% of total labor force) (modeled ILO estimate)': 'Unemp_Tot_ILO',
'Unemployment, total (% of total labor force) (national estimate)': 'Unemp_Tot_Nat'
}
df_WDI_pivot = df_WDI_pivot.rename(columns=rename_dict)
df_WDI_pivot
| Series Name | Country Name | ISO | Year | SocialNet_Cov | Edu_Bach25p | Edu_Prim25p | Edu_UpSec25p | GDP_Growth | GDP_pc | GDP_pc_Growth | ... | Enroll_Sec | Enroll_Tert | Unemp_AdvEdu | Unemp_BasicEdu | Unemp_Fem_ILO | Unemp_Fem_Nat | Unemp_Male_ILO | Unemp_Male_Nat | Unemp_Tot_ILO | Unemp_Tot_Nat |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 0 | Afghanistan | AFG | 2022 | NaN | 3.984250 | 21.025480 | 12.597330 | -6.240172 | 357.261153 | -7.576669 | ... | NaN | NaN | NaN | NaN | 26.742 | NaN | 13.168 | NaN | 14.100 | NaN |
| 1 | Afghanistan | AFG | 2023 | NaN | NaN | NaN | NaN | 2.266944 | 413.757895 | 0.106093 | ... | NaN | NaN | NaN | NaN | 26.570 | NaN | 13.066 | NaN | 13.991 | NaN |
| 2 | Afghanistan | AFG | 2024 | NaN | NaN | NaN | NaN | NaN | NaN | NaN | ... | NaN | NaN | NaN | NaN | 25.636 | NaN | 12.394 | NaN | 13.295 | NaN |
| 3 | Albania | ALB | 2022 | NaN | 19.834644 | 97.853140 | 53.268485 | 4.826696 | 6846.426694 | 6.108948 | ... | 97.322471 | 62.647922 | 10.228 | 9.437 | 10.405 | 11.235 | 9.917 | 10.398 | 10.137 | 10.785 |
| 4 | Albania | ALB | 2023 | NaN | 17.510000 | 99.279999 | 53.970001 | 3.936617 | 8575.171346 | 5.137124 | ... | 96.136877 | 64.729352 | 8.947 | 9.694 | 10.376 | 10.958 | 9.888 | 10.421 | 10.108 | 10.669 |
| ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... |
| 650 | Zambia | ZMB | 2023 | NaN | 2.801195 | NaN | 33.588454 | 5.367943 | 1330.727806 | 2.464640 | ... | NaN | NaN | 9.702 | 3.688 | 6.207 | 6.037 | 5.646 | 5.710 | 5.905 | 5.861 |
| 651 | Zambia | ZMB | 2024 | NaN | NaN | NaN | NaN | 4.043148 | 1235.084665 | 1.158387 | ... | 56.788929 | NaN | NaN | NaN | 6.250 | NaN | 5.713 | NaN | 5.961 | NaN |
| 652 | Zimbabwe | ZWE | 2022 | NaN | 0.913878 | NaN | NaN | 6.139176 | 2040.546587 | 4.343582 | ... | NaN | NaN | 5.945 | 8.878 | 10.562 | 10.461 | 9.615 | 9.554 | 10.087 | 10.001 |
| 653 | Zimbabwe | ZWE | 2023 | NaN | 1.134873 | NaN | NaN | 5.336770 | 2156.034093 | 3.584903 | ... | NaN | NaN | 5.933 | 8.493 | 8.824 | 9.460 | 8.694 | 9.126 | 8.759 | 9.290 |
| 654 | Zimbabwe | ZWE | 2024 | NaN | NaN | NaN | NaN | 2.029484 | 2656.409377 | 0.228944 | ... | NaN | 7.749130 | NaN | NaN | 8.587 | NaN | 8.521 | NaN | 8.554 | NaN |
655 rows × 47 columns
We also need the total population variable from the same database, which I had forgotten to export at first. I will quickly load and reshape it so that it can be merged into the joint dataset using similar steps as the other WDIs above.
df_pop = pd.read_csv('Pop22-24.csv')
df_pop = df_pop.melt(id_vars=['Country Name', 'Country Code', 'Series Name', 'Series Code'], var_name='Year', value_name='Value') # Reshape
df_pop['Year'] = df_pop['Year'].replace({'2022 [YR2022]': '2022', '2023 [YR2023]': '2023', '2024 [YR2024]': '2024'}).astype(int) # Fix year syntax
df_pop['Value'] = df_pop['Value'].apply(pd.to_numeric, errors='coerce') # Fix dtype of pop value
df_pop.drop(columns=['Series Name', 'Series Code'], inplace=True) # Drop series name and code
df_pop.rename(columns={'Value': 'Tot_Pop'}, inplace=True) # Rename population variable
df_pop.dtypes
| 0 | |
|---|---|
| Country Name | object |
| Country Code | object |
| Year | int64 |
| Tot_Pop | float64 |
Now our World Bank indicators are ready for merging and further analysis.
Exploratory Data Analysis (EDA)¶
2024 Refugee Counts Pie Chart¶
To begin our EDA, we will create a pie chart to answer the question, "What countries have the largest Ukrainian refugee populations today?" This is a simple yet helpful snapshot of how the refugees are distributed across host countries. Later analysis will tell us things like how these refugee flows have fluctuated over the course of the war.
# Isolate latest year
RefCounts_2024 = df_RefCounts[df_RefCounts['Year']== 2024].copy()
RefCounts_2024 = RefCounts_2024.sort_values('Ref_Asylum', ascending=False)
# Build df of top n countries
top_n = 9
top_2024 = RefCounts_2024.head(top_n)
# Sum the rest of the counts into a one-observation df for "Other"
others_sum = RefCounts_2024['Ref_Asylum'].iloc[top_n:].sum()
df_other = pd.DataFrame({'Country of Asylum':['Other'],'Ref_Asylum':[others_sum]})
# Concatenate the two
df_piechart = pd.concat([top_2024, df_other], ignore_index=True)
# Now we actually make the chart
plt.figure(figsize=(6,6))
plt.pie(
df_piechart['Ref_Asylum'],
labels = df_piechart['Country of Asylum'],
autopct = '%1.1f%%',
startangle = 140
)
plt.title('2024 Refugee and Asylum-seekers by Country')
plt.show()
As we can see, the top countries are all in Europe, and about 50% of all Ukrainian refugees today are located in only 3 countries: Germany, Poland, and Czechia. In fact, the top 9 countries host 71.6% of all refugees. Knowing this can help us narrow down our expectations of what variables should predict higher refugee populations.
Histogram of Refugee Populations¶
Another interesting graphic we can make to show the distribution is a histogram of our 82 host countries by their 2024 Ukrainian refugee/asylum-seeker populations:
plt.figure(figsize=(7,4))
plt.hist(RefCounts_2024['Ref_Asylum'], bins=20)
plt.xlabel('Refugee and Asylum-seeker Population')
plt.ylabel('Number of Countries')
plt.title(f'Distribution of Displaced Ukrainians Across Countries in 2024')
plt.show()
We clearly see the data has a large right skew, representing how a few countries have large numbers of refugees. To correct for this, we can transform the data. Once done, this transformed data will serve as our dependent variable in OLS regression. Here, I've selected a log transformation because it will compress extreme values to a roughly symmetrical distribution (shown below) and still allow for easy coefficient interpretations in later regressions.
#Log-transform refugee counts into a more evenly distributed shape, creating a new column in the main RefCounts df.
x = df_RefCounts['Ref_Asylum']
df_RefCounts['ln_Ref_Asylum'] = np.log(x + 1) # Add 1 to account for zeros
#Now filter again to latest year
RefCounts_2024 = df_RefCounts[df_RefCounts['Year'] == 2024].copy()
#Calculate skew and check log-transformation is sufficiently symmetrical
from scipy.stats import skew
print("Skew (original):", skew(RefCounts_2024['Ref_Asylum']))
print("Skew (log-transformed):", skew(RefCounts_2024['ln_Ref_Asylum'])) #Should be near zero
plt.figure(figsize=(7,4))
plt.hist(RefCounts_2024['ln_Ref_Asylum'], bins=20)
plt.xlabel('ln(Refugee Population)')
plt.ylabel('Number of Countries')
plt.title(f'Refugee/Asylum-seeker Populations Across Countries in 2024')
plt.show()
Skew (original): 5.077644846318671 Skew (log-transformed): -0.15662866559786284
Merging for further EDA¶
Now we will merge our datasets together on their ISO codes, dropping the redundant key columns.
# Merge refugee counts with WDIs
df_RefWDI = pd.merge(df_RefCounts, df_WDI_pivot, how='left', left_on=['Country of Asylum ISO', 'Year'], right_on=['ISO', 'Year'])
# Merge in total population variable too
df_RefWDI = pd.merge(df_RefWDI, df_pop, how='left', left_on=['Country of Asylum ISO', 'Year'], right_on=['Country Code', 'Year'])
# Drop redundant key columns
df_RefWDI.drop(columns=['Country Name_x', 'Country Name_y', 'ISO', 'Country Code'], inplace=True)
df_RefWDI
| Year | Country of Asylum | Country of Asylum ISO | Refugees | Asylum-seekers | Ref_Asylum | ln_Ref_Asylum | SocialNet_Cov | Edu_Bach25p | Edu_Prim25p | ... | Enroll_Tert | Unemp_AdvEdu | Unemp_BasicEdu | Unemp_Fem_ILO | Unemp_Fem_Nat | Unemp_Male_ILO | Unemp_Male_Nat | Unemp_Tot_ILO | Unemp_Tot_Nat | Tot_Pop | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 0 | 2022 | Albania | ALB | 2542 | 5 | 2547 | 7.843064 | NaN | 19.834644 | 97.853140 | ... | 62.647922 | 10.228 | 9.437 | 10.405 | 11.235 | 9.917 | 10.398 | 10.137 | 10.785 | 2777689.0 |
| 1 | 2023 | Albania | ALB | 4959 | 0 | 4959 | 8.509161 | NaN | 17.510000 | 99.279999 | ... | 64.729352 | 8.947 | 9.694 | 10.376 | 10.958 | 9.888 | 10.421 | 10.108 | 10.669 | 2745972.0 |
| 2 | 2024 | Albania | ALB | 9250 | 0 | 9250 | 9.132487 | NaN | NaN | NaN | ... | 81.998627 | NaN | NaN | 10.527 | NaN | 10.023 | NaN | 10.250 | NaN | 2714617.0 |
| 3 | 2022 | Egypt | EGY | 0 | 5 | 5 | 1.791759 | NaN | NaN | 66.608430 | ... | 36.462231 | 15.183 | 4.705 | 18.283 | 18.516 | 4.987 | 5.051 | 7.344 | 7.344 | 112618250.0 |
| 4 | 2023 | Egypt | EGY | 0 | 9 | 9 | 2.302585 | NaN | 18.052291 | 69.221964 | ... | 37.900558 | 14.175 | 4.313 | 18.234 | 15.329 | 4.972 | 4.617 | 7.334 | 6.753 | 114535772.0 |
| ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... |
| 233 | 2023 | Uruguay | URY | 18 | 0 | 18 | 2.944439 | NaN | NaN | 92.800000 | ... | 79.953690 | 2.034 | 9.926 | 9.435 | 9.462 | 7.408 | 7.429 | 8.355 | 8.355 | 3388081.0 |
| 234 | 2024 | Uruguay | URY | 18 | 5 | 23 | 3.178054 | NaN | 12.929680 | 93.832077 | ... | NaN | 1.909 | 9.789 | 9.757 | 9.416 | 7.209 | 7.201 | 8.401 | 8.209 | 3386588.0 |
| 235 | 2022 | United States | USA | 3063 | 12369 | 15432 | 9.644263 | NaN | 37.660000 | 99.029999 | ... | 79.361931 | 2.321 | 5.507 | 3.613 | 3.614 | 3.680 | 3.681 | 3.650 | 3.650 | 334017321.0 |
| 236 | 2023 | United States | USA | 3299 | 17773 | 21072 | 9.955748 | NaN | 38.601131 | 99.075211 | ... | NaN | 2.373 | 5.591 | 3.450 | 3.454 | 3.795 | 3.800 | 3.638 | 3.638 | 336806231.0 |
| 237 | 2024 | United States | USA | 3397 | 18658 | 22055 | 10.001340 | NaN | 39.029478 | 99.682239 | ... | NaN | 2.608 | 6.501 | 3.892 | 3.917 | 4.285 | 4.115 | 4.106 | 4.022 | 340110988.0 |
238 rows × 52 columns
Normalized Refugee Counts¶
To be able to better compare countries of different sizes in our future model, we should also create a new refugee count variable normalized by the host population. While the logged values from earlier are useful for analyzing total flows, a measurement like refugees per-100 people can help us better understand the relative burden of migrant flows on host countries.
When comparing the top host countries in total count then normalized count below, we can see that larger-population countries like Germany, Russia, and the U.K. drop out of the top 10. This also emphasizes the impact of the refugee crisis on countries like Poland, Czechia, and Slovakia, which have large refugee populations in both level and ratio terms.
df_RefWDI['Refs_per_100'] = df_RefWDI['Ref_Asylum'] / df_RefWDI['Tot_Pop'] * 100
# Compare top rankings with level variable
top_refs = df_RefWDI.groupby('Country of Asylum')[['Ref_Asylum', 'Tot_Pop']].mean().reset_index().sort_values('Ref_Asylum', ascending=False)
display(top_refs.head(10))
top_normed = df_RefWDI.groupby('Country of Asylum')[['Refs_per_100', 'Tot_Pop']].mean().reset_index().sort_values('Refs_per_100', ascending=False)
display(top_normed.head(10))
| Country of Asylum | Ref_Asylum | Tot_Pop | |
|---|---|---|---|
| 28 | Germany | 1.103695e+06 | 8.373695e+07 |
| 65 | Poland | 9.700287e+05 | 3.668794e+07 |
| 68 | Russia | 8.369263e+05 | 1.438656e+08 |
| 19 | Czechia | 3.988330e+05 | 1.080611e+07 |
| 80 | United Kingdom | 2.217697e+05 | 6.844067e+07 |
| 74 | Spain | 1.922450e+05 | 4.831372e+07 |
| 38 | Italy | 1.624677e+05 | 5.899772e+07 |
| 70 | Slovakia | 1.206673e+05 | 5.426854e+06 |
| 67 | Romania | 1.198760e+05 | 1.905911e+07 |
| 52 | Moldova | 1.195533e+05 | 2.458571e+06 |
| Country of Asylum | Refs_per_100 | Tot_Pop | |
|---|---|---|---|
| 53 | Montenegro | 6.176500 | 6.236503e+05 |
| 52 | Moldova | 4.878297 | 2.458571e+06 |
| 19 | Czechia | 3.693071 | 1.080611e+07 |
| 24 | Estonia | 3.040658 | 1.363704e+06 |
| 65 | Poland | 2.644142 | 3.668794e+07 |
| 44 | Latvia | 2.310928 | 1.873090e+06 |
| 48 | Lithuania | 2.249647 | 2.863760e+06 |
| 70 | Slovakia | 2.223784 | 5.426854e+06 |
| 36 | Ireland | 1.764716 | 5.300231e+06 |
| 11 | Bulgaria | 1.506933 | 6.452020e+06 |
Statistics for Time-Series Analysis¶
Another important aspect of my project is its temporal dimension; the refugee data I have collected is a 3-year long panel of different countries. While the number of periods in the panel is low, there are still important statistics we can generate that will help our later modeling. For example, lags of refugee counts are essential in autoregressive modeling if I want to account for the serial correlation between observations. I can also compute discrete growth rates and log-differences of a country's refugee population from year to year, revealing how volatile or stable the flows are over time.
# Lags for the variable Ref_Asylum
df_RefWDI['Ref_Asylum_t-1'] = df_RefWDI.groupby('Country of Asylum')['Ref_Asylum'].shift(1)
df_RefWDI['Ref_Asylum_t-2'] = df_RefWDI.groupby('Country of Asylum')['Ref_Asylum'].shift(2)
# Replace NaNs in lagged refugee counts with zeros because pre-war, they had zero refugees.
df_RefWDI.fillna({'Ref_Asylum_t-1': 0}, inplace = True)
df_RefWDI.fillna({'Ref_Asylum_t-2': 0}, inplace = True)
# Now log transform these lags as well to make them compatible with our dependent variable.
df_RefWDI['ln_Ref_Asylum_t-1'] = np.log(df_RefWDI['Ref_Asylum_t-1'] + 1)
df_RefWDI['ln_Ref_Asylum_t-2'] = np.log(df_RefWDI['Ref_Asylum_t-2'] + 1)
# Also compute log-differences for Ref_Asylum
df_RefWDI['Ref_Growth_LD'] = df_RefWDI['ln_Ref_Asylum'] - df_RefWDI['ln_Ref_Asylum_t-1']
# Discrete growth rates
df_RefWDI['Ref_Growth'] = (df_RefWDI['Ref_Asylum'] - df_RefWDI['Ref_Asylum_t-1']) / df_RefWDI['Ref_Asylum_t-1'] * 100
display(df_RefWDI.sort_values(by='Ref_Growth_LD', ascending=False).head(5))
display(df_RefWDI[df_RefWDI['Ref_Growth'] != np.inf].sort_values(by='Ref_Growth', ascending=False).head(5))
| Year | Country of Asylum | Country of Asylum ISO | Refugees | Asylum-seekers | Ref_Asylum | ln_Ref_Asylum | SocialNet_Cov | Edu_Bach25p | Edu_Prim25p | ... | Unemp_Tot_ILO | Unemp_Tot_Nat | Tot_Pop | Refs_per_100 | Ref_Asylum_t-1 | Ref_Asylum_t-2 | ln_Ref_Asylum_t-1 | ln_Ref_Asylum_t-2 | Ref_Growth_LD | Ref_Growth | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 204 | 2022 | Russia | RUS | 1275315 | 610 | 1275925 | 14.059183 | NaN | NaN | 100.000000 | ... | 3.867 | 3.867 | 144236933.0 | 0.884604 | 0.0 | 0.0 | 0.0 | 0.0 | 14.059183 | inf |
| 84 | 2022 | Germany | DEU | 1002999 | 1383 | 1004382 | 13.819884 | NaN | 29.575744 | 100.000000 | ... | 3.120 | 3.137 | 83797985.0 | 1.198575 | 0.0 | 0.0 | 0.0 | 0.0 | 13.819884 | inf |
| 194 | 2022 | Poland | POL | 958147 | 372 | 958519 | 13.773146 | NaN | NaN | 99.512287 | ... | 2.811 | 2.811 | 36821749.0 | 2.603133 | 0.0 | 0.0 | 0.0 | 0.0 | 13.773146 | inf |
| 57 | 2022 | Czechia | CZE | 433501 | 137 | 433638 | 12.979968 | NaN | 22.287664 | 99.882802 | ... | 2.224 | 2.224 | 10672118.0 | 4.063280 | 0.0 | 0.0 | 0.0 | 0.0 | 12.979968 | inf |
| 78 | 2022 | United Kingdom | GBR | 182423 | 976 | 183399 | 12.119425 | NaN | NaN | NaN | ... | 3.730 | 3.768 | 67604000.0 | 0.271284 | 0.0 | 0.0 | 0.0 | 0.0 | 12.119425 | inf |
5 rows × 59 columns
| Year | Country of Asylum | Country of Asylum ISO | Refugees | Asylum-seekers | Ref_Asylum | ln_Ref_Asylum | SocialNet_Cov | Edu_Bach25p | Edu_Prim25p | ... | Unemp_Tot_ILO | Unemp_Tot_Nat | Tot_Pop | Refs_per_100 | Ref_Asylum_t-1 | Ref_Asylum_t-2 | ln_Ref_Asylum_t-1 | ln_Ref_Asylum_t-2 | Ref_Growth_LD | Ref_Growth | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 31 | 2023 | Brazil | BRA | 1129 | 30 | 1159 | 7.056175 | NaN | NaN | 85.200000 | ... | 7.947 | 7.947 | 211140729.0 | 0.000549 | 108.0 | 0.0 | 4.691348 | 0.0 | 2.364827 | 973.148148 |
| 13 | 2023 | Australia | AUS | 7472 | 809 | 8281 | 9.021840 | NaN | 35.764835 | 99.491927 | ... | 3.668 | 3.668 | 26652777.0 | 0.031070 | 2029.0 | 0.0 | 7.615791 | 0.0 | 1.406049 | 308.132085 |
| 208 | 2023 | Serbia | SRB | 4063 | 0 | 4063 | 8.309923 | NaN | 18.669201 | 98.741946 | ... | 8.268 | 8.268 | 6623183.0 | 0.061345 | 1108.0 | 0.0 | 7.011214 | 0.0 | 1.298709 | 266.696751 |
| 28 | 2023 | Bolivia | BOL | 7 | 10 | 17 | 2.890372 | NaN | 20.118389 | 71.232811 | ... | 3.024 | 2.948 | 12244159.0 | 0.000139 | 5.0 | 0.0 | 1.791759 | 0.0 | 1.098612 | 240.000000 |
| 189 | 2023 | Peru | PER | 30 | 90 | 120 | 4.795791 | NaN | NaN | 75.800000 | ... | 4.899 | 4.899 | 33845617.0 | 0.000355 | 36.0 | 0.0 | 3.610918 | 0.0 | 1.184873 | 233.333333 |
5 rows × 59 columns
As we can see from the above tables, in the log-differences version, the single largest growth episodes were in the first year of the war to countries like Russia, Germany, and Poland, and at a massive scale; Russia's log-difference of 14.1 indicates growth in refugee population by a factor of about 1.2 million. However, this is expected because log-difference growth starting from zero would be very large compared to growth between years with non-zero refugee counts. Furthermore, in such an explosive conflict, it makes sense contextually that the first year would see the largest outflow of Ukrainian refugees.
Looking at discrete growth, since the 2022 rates start from zero, those observed rates are mathematically infinite. Excluding those occurrences to look at growth after the first year of arrivals in a given country, we can see that Brazil had the next largest relative increase in refugee population, up 973% from 108 to 1159 in 2023. In fact, we can see that all of the top 5 single largest increases (outside 2022) occurred in 2023, so we know that the rate of refugee flows appears to have started high and slowly decreased from year to year in this data set.
Handling Missing Data¶
The World Development Indicators are often full of missing data, often due to varying data collection capacities between countries. We must be able to handle the missing observations as part of our analysis. We can create a missing data heatmap to see at a glance which variables have many missing variables.
plt.figure(figsize=(12,6))
sns.heatmap(df_RefWDI.isnull(), cbar=False)
plt.title("Missing Data Heatmap")
plt.show()
We can also generate missing value counts and percentages for a more explicit breakdown by variable.
missing_count = df_RefWDI.isna().sum()
missing_pct = (missing_count/len(df_RefWDI))*100
missing_summary = pd.DataFrame({
'missing_count': missing_count,
'missing_pct': missing_pct
}).sort_values(by='missing_pct', ascending=False)
missing_summary_full = missing_summary.copy() # Save a copy for later
missing_summary
| missing_count | missing_pct | |
|---|---|---|
| SocialNet_Cov | 225 | 94.537815 |
| Net_UNHCR | 194 | 81.512605 |
| Poverty_pct | 166 | 69.747899 |
| Gini | 134 | 56.302521 |
| Edu_Bach25p | 114 | 47.899160 |
| PrimComp_pct | 103 | 43.277311 |
| FoodInsec_pct | 100 | 42.016807 |
| Edu_UpSec25p | 91 | 38.235294 |
| Enroll_Tert | 86 | 36.134454 |
| Enroll_Sec | 85 | 35.714286 |
| Edu_Prim25p | 85 | 35.714286 |
| Pol_Stab_rank | 81 | 34.033613 |
| RuleLaw_rank | 81 | 34.033613 |
| GovEffect_rank | 81 | 34.033613 |
| Enroll_Prim | 75 | 31.512605 |
| LF_IntEdu | 52 | 21.848739 |
| Unemp_AdvEdu | 51 | 21.428571 |
| Unemp_BasicEdu | 51 | 21.428571 |
| GNI_pc_Growth | 50 | 21.008403 |
| LF_BasicEdu | 50 | 21.008403 |
| LF_AdvEdu | 50 | 21.008403 |
| Unemp_Fem_Nat | 38 | 15.966387 |
| Unemp_Male_Nat | 38 | 15.966387 |
| Unemp_Tot_Nat | 38 | 15.966387 |
| LFPR_15_24_Nat | 38 | 15.966387 |
| LFPR_15p_Nat | 36 | 15.126050 |
| GDP_Growth | 5 | 2.100840 |
| GNI_pc | 5 | 2.100840 |
| GDP_pc_Growth | 5 | 2.100840 |
| LFPR_15p_ILO | 4 | 1.680672 |
| LFPR_15_64_ILO | 4 | 1.680672 |
| Unemp_Tot_ILO | 4 | 1.680672 |
| LF_Fem | 4 | 1.680672 |
| Unemp_Fem_ILO | 4 | 1.680672 |
| Unemp_Male_ILO | 4 | 1.680672 |
| LFPR_15_24_ILO | 4 | 1.680672 |
| GDP_pc | 3 | 1.260504 |
| Year | 0 | 0.000000 |
| Country of Asylum ISO | 0 | 0.000000 |
| Asylum-seekers | 0 | 0.000000 |
| Ref_Asylum | 0 | 0.000000 |
| Refugees | 0 | 0.000000 |
| Country of Asylum | 0 | 0.000000 |
| ln_Ref_Asylum | 0 | 0.000000 |
| Pop_Male_pct | 0 | 0.000000 |
| Pop_15_64_pct | 0 | 0.000000 |
| Pop_15_64_Tot | 0 | 0.000000 |
| Pop_15_64_Male | 0 | 0.000000 |
| Pop_Fem_pct | 0 | 0.000000 |
| Pop_Growth | 0 | 0.000000 |
| Pop_15_64_Fem | 0 | 0.000000 |
| Tot_Pop | 0 | 0.000000 |
| Refs_per_100 | 0 | 0.000000 |
| Ref_Asylum_t-1 | 0 | 0.000000 |
| Ref_Asylum_t-2 | 0 | 0.000000 |
| ln_Ref_Asylum_t-1 | 0 | 0.000000 |
| ln_Ref_Asylum_t-2 | 0 | 0.000000 |
| Ref_Growth_LD | 0 | 0.000000 |
| Ref_Growth | 0 | 0.000000 |
As we can see, there are many variables missing large amounts of data. To start, we will drop columns that are missing 50% or more of their observations.
# Turn columns index into list of columns
to_drop = missing_summary[missing_summary['missing_pct'] > 50].index.tolist()
# Drop columns/rows from missing summary and the main dataframe
missing_summary.drop(to_drop, inplace=True)
df_RefWDI.drop(columns=to_drop, inplace=True)
I also made a modified version of our dataframe with only the remaining variables that are missing 20% or more of their observations to see if there are any visible patterns in the missing data
# Vars missing over 20% of observations
manymiss_vars = missing_summary[missing_summary['missing_pct'] > 20].index
to_drop = ['ln_Ref_Asylum_t-2', 'Ref_Growth', 'ln_Ref_Asylum_t-1', 'Ref_Asylum_t-1']
manymiss_vars_clean = manymiss_vars.difference(to_drop)
# See main df with only these variables to look for patterns
to_keep = ['Country of Asylum', 'Year'] + list(manymiss_vars_clean)
df_manymiss = df_RefWDI[to_keep]
df_manymiss
| Country of Asylum | Year | Edu_Bach25p | Edu_Prim25p | Edu_UpSec25p | Enroll_Prim | Enroll_Sec | Enroll_Tert | FoodInsec_pct | GNI_pc_Growth | GovEffect_rank | LF_AdvEdu | LF_BasicEdu | LF_IntEdu | Pol_Stab_rank | PrimComp_pct | RuleLaw_rank | Unemp_AdvEdu | Unemp_BasicEdu | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 0 | Albania | 2022 | 19.834644 | 97.853140 | 53.268485 | 96.371230 | 97.322471 | 62.647922 | 32.2 | 7.856992 | 56.132076 | 77.472 | 54.670 | 65.703 | 49.528301 | 101.176280 | 47.169811 | 10.228 | 9.437 |
| 1 | Albania | 2023 | 17.510000 | 99.279999 | 53.970001 | 93.713812 | 96.136877 | 64.729352 | 33.0 | 7.289707 | 60.377357 | 82.056 | 54.455 | 68.530 | 51.658768 | 95.135152 | 45.754719 | 8.947 | 9.694 |
| 2 | Albania | 2024 | NaN | NaN | NaN | 111.511650 | 108.355392 | 81.998627 | NaN | NaN | NaN | NaN | NaN | NaN | NaN | 107.345322 | NaN | NaN | NaN |
| 3 | Egypt | 2022 | NaN | 66.608430 | 54.313658 | 89.510087 | NaN | 36.462231 | 29.8 | 4.736284 | 34.433964 | 64.216 | 27.450 | 47.904 | 14.622642 | NaN | 42.924530 | 15.183 | 4.705 |
| 4 | Egypt | 2023 | 18.052291 | 69.221964 | 55.692370 | 95.921707 | 84.474862 | 37.900558 | 30.8 | 0.896229 | 41.981133 | 64.715 | 29.346 | 49.534 | 16.587677 | 93.841399 | 44.339622 | 14.175 | 4.313 |
| ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... |
| 233 | Uruguay | 2023 | NaN | 92.800000 | 34.000000 | 104.532466 | 119.105006 | 79.953690 | 16.2 | 2.573633 | 79.245285 | 81.355 | 61.637 | 72.827 | 81.990524 | 101.169008 | 75.943398 | 2.034 | 9.926 |
| 234 | Uruguay | 2024 | 12.929680 | 93.832077 | 35.009258 | NaN | NaN | NaN | NaN | 2.610834 | NaN | 81.471 | 62.368 | 74.021 | NaN | NaN | NaN | 1.909 | 9.789 |
| 235 | United States | 2022 | 37.660000 | 99.029999 | 91.190002 | 96.971733 | 97.473488 | 79.361931 | 9.3 | 1.852199 | 87.264153 | 71.780 | 42.572 | 56.076 | 46.226414 | 95.743523 | 88.679245 | 2.321 | 5.507 |
| 236 | United States | 2023 | 38.601131 | 99.075211 | 91.461288 | NaN | NaN | NaN | 10.3 | 1.821216 | 87.735847 | 71.866 | 44.160 | 56.415 | 47.393364 | NaN | 88.679245 | 2.373 | 5.591 |
| 237 | United States | 2024 | 39.029478 | 99.682239 | 95.447222 | NaN | NaN | NaN | NaN | 1.633877 | NaN | 71.592 | 44.352 | 56.369 | NaN | NaN | NaN | 2.608 | 6.501 |
238 rows × 19 columns
In this table above, we can see that a lot of the missing observations come in the year 2024, indicating that it's likely due to a lag in reporting of the most recent data. This trend persists across many different countries, so it is safe to use mean imputation on this data if we assume the 2024 indicators have no reason to be significantly different from their 2022 and 2023 values within a given country.
# Make list of just the numeric columns, except year.
numeric_cols = df_RefWDI.select_dtypes(include=np.number).columns.drop('Year', errors='ignore')
# Apply mean imputation, grouping by country, saving to a new df
df_RefWDI_meanimpute = df_RefWDI.copy()
df_RefWDI_meanimpute[numeric_cols] = df_RefWDI.groupby('Country of Asylum')[numeric_cols].transform(lambda x: x.fillna(x.mean()))
df_RefWDI_meanimpute
| Year | Country of Asylum | Country of Asylum ISO | Refugees | Asylum-seekers | Ref_Asylum | ln_Ref_Asylum | Edu_Bach25p | Edu_Prim25p | Edu_UpSec25p | ... | Unemp_Tot_ILO | Unemp_Tot_Nat | Tot_Pop | Refs_per_100 | Ref_Asylum_t-1 | Ref_Asylum_t-2 | ln_Ref_Asylum_t-1 | ln_Ref_Asylum_t-2 | Ref_Growth_LD | Ref_Growth | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 0 | 2022 | Albania | ALB | 2542 | 5 | 2547 | 7.843064 | 19.834644 | 97.853140 | 53.268485 | ... | 10.137 | 10.785 | 2777689.0 | 0.091695 | 0.0 | 0.0 | 0.000000 | 0.000000 | 7.843064 | inf |
| 1 | 2023 | Albania | ALB | 4959 | 0 | 4959 | 8.509161 | 17.510000 | 99.279999 | 53.970001 | ... | 10.108 | 10.669 | 2745972.0 | 0.180592 | 2547.0 | 0.0 | 7.843064 | 0.000000 | 0.666097 | 94.699647 |
| 2 | 2024 | Albania | ALB | 9250 | 0 | 9250 | 9.132487 | 18.672322 | 98.566569 | 53.619243 | ... | 10.250 | 10.727 | 2714617.0 | 0.340748 | 4959.0 | 2547.0 | 8.509161 | 7.843064 | 0.623326 | 86.529542 |
| 3 | 2022 | Egypt | EGY | 0 | 5 | 5 | 1.791759 | 18.052291 | 66.608430 | 54.313658 | ... | 7.344 | 7.344 | 112618250.0 | 0.000004 | 0.0 | 0.0 | 0.000000 | 0.000000 | 1.791759 | inf |
| 4 | 2023 | Egypt | EGY | 0 | 9 | 9 | 2.302585 | 18.052291 | 69.221964 | 55.692370 | ... | 7.334 | 6.753 | 114535772.0 | 0.000008 | 5.0 | 0.0 | 1.791759 | 0.000000 | 0.510826 | 80.000000 |
| ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... |
| 233 | 2023 | Uruguay | URY | 18 | 0 | 18 | 2.944439 | 12.929680 | 92.800000 | 34.000000 | ... | 8.355 | 8.355 | 3388081.0 | 0.000531 | 18.0 | 0.0 | 2.944439 | 0.000000 | 0.000000 | 0.000000 |
| 234 | 2024 | Uruguay | URY | 18 | 5 | 23 | 3.178054 | 12.929680 | 93.832077 | 35.009258 | ... | 8.401 | 8.209 | 3386588.0 | 0.000679 | 18.0 | 18.0 | 2.944439 | 2.944439 | 0.233615 | 27.777778 |
| 235 | 2022 | United States | USA | 3063 | 12369 | 15432 | 9.644263 | 37.660000 | 99.029999 | 91.190002 | ... | 3.650 | 3.650 | 334017321.0 | 0.004620 | 0.0 | 0.0 | 0.000000 | 0.000000 | 9.644263 | inf |
| 236 | 2023 | United States | USA | 3299 | 17773 | 21072 | 9.955748 | 38.601131 | 99.075211 | 91.461288 | ... | 3.638 | 3.638 | 336806231.0 | 0.006256 | 15432.0 | 0.0 | 9.644263 | 0.000000 | 0.311485 | 36.547434 |
| 237 | 2024 | United States | USA | 3397 | 18658 | 22055 | 10.001340 | 39.029478 | 99.682239 | 95.447222 | ... | 4.106 | 4.022 | 340110988.0 | 0.006485 | 21072.0 | 15432.0 | 9.955748 | 9.644263 | 0.045592 | 4.664958 |
238 rows × 55 columns
# Now regenerate the missing summary to see how it went
missing_count = df_RefWDI_meanimpute.isna().sum()
missing_pct = (missing_count/len(df_RefWDI_meanimpute))*100
missing_summary = pd.DataFrame({
'missing_count': missing_count,
'missing_pct': missing_pct
}).sort_values(by='missing_pct', ascending=False)
missing_summary
| missing_count | missing_pct | |
|---|---|---|
| Edu_Bach25p | 63 | 26.470588 |
| PrimComp_pct | 48 | 20.168067 |
| Edu_UpSec25p | 45 | 18.907563 |
| LF_IntEdu | 40 | 16.806723 |
| Unemp_BasicEdu | 37 | 15.546218 |
| LF_AdvEdu | 37 | 15.546218 |
| LF_BasicEdu | 37 | 15.546218 |
| Unemp_AdvEdu | 37 | 15.546218 |
| Edu_Prim25p | 36 | 15.126050 |
| FoodInsec_pct | 27 | 11.344538 |
| LFPR_15_24_Nat | 25 | 10.504202 |
| Enroll_Tert | 23 | 9.663866 |
| Unemp_Male_Nat | 22 | 9.243697 |
| Unemp_Fem_Nat | 22 | 9.243697 |
| Unemp_Tot_Nat | 22 | 9.243697 |
| GNI_pc_Growth | 20 | 8.403361 |
| LFPR_15p_Nat | 19 | 7.983193 |
| Enroll_Sec | 19 | 7.983193 |
| Enroll_Prim | 9 | 3.781513 |
| RuleLaw_rank | 4 | 1.680672 |
| GovEffect_rank | 4 | 1.680672 |
| Pol_Stab_rank | 4 | 1.680672 |
| GDP_pc_Growth | 3 | 1.260504 |
| Unemp_Tot_ILO | 3 | 1.260504 |
| LFPR_15_64_ILO | 3 | 1.260504 |
| LFPR_15_24_ILO | 3 | 1.260504 |
| LFPR_15p_ILO | 3 | 1.260504 |
| GNI_pc | 3 | 1.260504 |
| LF_Fem | 3 | 1.260504 |
| Unemp_Fem_ILO | 3 | 1.260504 |
| Unemp_Male_ILO | 3 | 1.260504 |
| GDP_Growth | 3 | 1.260504 |
| Year | 0 | 0.000000 |
| Country of Asylum | 0 | 0.000000 |
| Country of Asylum ISO | 0 | 0.000000 |
| Refugees | 0 | 0.000000 |
| Asylum-seekers | 0 | 0.000000 |
| Ref_Asylum | 0 | 0.000000 |
| ln_Ref_Asylum | 0 | 0.000000 |
| GDP_pc | 0 | 0.000000 |
| Pop_Male_pct | 0 | 0.000000 |
| Pop_15_64_Tot | 0 | 0.000000 |
| Pop_Fem_pct | 0 | 0.000000 |
| Pop_15_64_Fem | 0 | 0.000000 |
| Pop_15_64_Male | 0 | 0.000000 |
| Pop_Growth | 0 | 0.000000 |
| Pop_15_64_pct | 0 | 0.000000 |
| Tot_Pop | 0 | 0.000000 |
| Refs_per_100 | 0 | 0.000000 |
| Ref_Asylum_t-1 | 0 | 0.000000 |
| Ref_Asylum_t-2 | 0 | 0.000000 |
| ln_Ref_Asylum_t-1 | 0 | 0.000000 |
| ln_Ref_Asylum_t-2 | 0 | 0.000000 |
| Ref_Growth_LD | 0 | 0.000000 |
| Ref_Growth | 0 | 0.000000 |
After mean imputation, we still have a decent number of NaNs in some of our variables. This is because some of the countries have missing values for all three years, so there is no data to average in the first imputation method. There are also some countries that only appear in the dataset once (i.e. they only recorded Ukrainian refugees in one year) and are missing an indicator value, so they cannot be averaged either. To fill in these values, we can use SciKit Learn to run a K-nearest neighbors imputation on the remaining NaNs.
from sklearn.impute import KNNImputer
from sklearn.preprocessing import StandardScaler
imputer = KNNImputer(n_neighbors=5)
scaler = StandardScaler()
# Make copy
df_knnimpute = df_RefWDI_meanimpute.copy()
# Select numeric columns except year and engineered variables
numeric_cols = df_knnimpute.select_dtypes(include=np.number).columns.drop(['Year', 'Ref_Asylum_t-1', 'Ref_Asylum_t-2', 'ln_Ref_Asylum_t-1', 'ln_Ref_Asylum_t-2', 'Ref_Growth_LD', 'Ref_Growth', 'Refs_per_100'], errors='ignore')
# Scale
df_scaled = scaler.fit_transform(df_knnimpute[numeric_cols])
# Impute
df_sca_imp = imputer.fit_transform(df_scaled)
# Reverse the scaling so the values are interpretable again
df_imputed = scaler.inverse_transform(df_sca_imp)
# Put values back in original dataframe
df_knnimpute[numeric_cols] = df_imputed
df_final = df_knnimpute
df_final
| Year | Country of Asylum | Country of Asylum ISO | Refugees | Asylum-seekers | Ref_Asylum | ln_Ref_Asylum | Edu_Bach25p | Edu_Prim25p | Edu_UpSec25p | ... | Unemp_Tot_ILO | Unemp_Tot_Nat | Tot_Pop | Refs_per_100 | Ref_Asylum_t-1 | Ref_Asylum_t-2 | ln_Ref_Asylum_t-1 | ln_Ref_Asylum_t-2 | Ref_Growth_LD | Ref_Growth | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 0 | 2022 | Albania | ALB | 2542.0 | 5.0 | 2547.0 | 7.843064 | 19.834644 | 97.853140 | 53.268485 | ... | 10.137 | 10.785 | 2777689.0 | 0.091695 | 0.0 | 0.0 | 0.000000 | 0.000000 | 7.843064 | inf |
| 1 | 2023 | Albania | ALB | 4959.0 | 0.0 | 4959.0 | 8.509161 | 17.510000 | 99.279999 | 53.970001 | ... | 10.108 | 10.669 | 2745972.0 | 0.180592 | 2547.0 | 0.0 | 7.843064 | 0.000000 | 0.666097 | 94.699647 |
| 2 | 2024 | Albania | ALB | 9250.0 | 0.0 | 9250.0 | 9.132487 | 18.672322 | 98.566569 | 53.619243 | ... | 10.250 | 10.727 | 2714617.0 | 0.340748 | 4959.0 | 2547.0 | 8.509161 | 7.843064 | 0.623326 | 86.529542 |
| 3 | 2022 | Egypt | EGY | 0.0 | 5.0 | 5.0 | 1.791759 | 18.052291 | 66.608430 | 54.313658 | ... | 7.344 | 7.344 | 112618250.0 | 0.000004 | 0.0 | 0.0 | 0.000000 | 0.000000 | 1.791759 | inf |
| 4 | 2023 | Egypt | EGY | 0.0 | 9.0 | 9.0 | 2.302585 | 18.052291 | 69.221964 | 55.692370 | ... | 7.334 | 6.753 | 114535772.0 | 0.000008 | 5.0 | 0.0 | 1.791759 | 0.000000 | 0.510826 | 80.000000 |
| ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... |
| 233 | 2023 | Uruguay | URY | 18.0 | 0.0 | 18.0 | 2.944439 | 12.929680 | 92.800000 | 34.000000 | ... | 8.355 | 8.355 | 3388081.0 | 0.000531 | 18.0 | 0.0 | 2.944439 | 0.000000 | 0.000000 | 0.000000 |
| 234 | 2024 | Uruguay | URY | 18.0 | 5.0 | 23.0 | 3.178054 | 12.929680 | 93.832077 | 35.009258 | ... | 8.401 | 8.209 | 3386588.0 | 0.000679 | 18.0 | 18.0 | 2.944439 | 2.944439 | 0.233615 | 27.777778 |
| 235 | 2022 | United States | USA | 3063.0 | 12369.0 | 15432.0 | 9.644263 | 37.660000 | 99.029999 | 91.190002 | ... | 3.650 | 3.650 | 334017321.0 | 0.004620 | 0.0 | 0.0 | 0.000000 | 0.000000 | 9.644263 | inf |
| 236 | 2023 | United States | USA | 3299.0 | 17773.0 | 21072.0 | 9.955748 | 38.601131 | 99.075211 | 91.461288 | ... | 3.638 | 3.638 | 336806231.0 | 0.006256 | 15432.0 | 0.0 | 9.644263 | 0.000000 | 0.311485 | 36.547434 |
| 237 | 2024 | United States | USA | 3397.0 | 18658.0 | 22055.0 | 10.001340 | 39.029478 | 99.682239 | 95.447222 | ... | 4.106 | 4.022 | 340110988.0 | 0.006485 | 21072.0 | 15432.0 | 9.955748 | 9.644263 | 0.045592 | 4.664958 |
238 rows × 55 columns
# Check entire dataframe for NaNs
df_final.isna().sum().sum().item()
0
Now there is no more missing data!
Regressor Selection: Correlation Matrix¶
Here, we will look at the linear correlations between a number of macroeconomic variables and our prospective dependent variables. This is an important step because when it comes time to select OLS regressors, these correlations help us understand which variables may be most telling.
corr_vars = ['ln_Ref_Asylum', 'ln_Ref_Asylum_t-1', 'GDP_pc', 'Unemp_AdvEdu','Pol_Stab_rank', 'RuleLaw_rank', 'GovEffect_rank', 'Unemp_Tot_ILO', 'PrimComp_pct', 'FoodInsec_pct', 'LFPR_15_64_ILO', 'Edu_Bach25p']
fig, ax = plt.subplots(figsize=(10,10))
sns.heatmap(df_RefWDI[corr_vars].corr(), annot=True, cmap='coolwarm', ax=ax)
<Axes: >
Already, we can see that ln_Ref_Asylum has moderately high correlations with host factors like its first lag, bachelor's degree attainment, food insecurity rate, and percentile ranks in rule of law and government effectiveness. There are also moderate correlations with political stability rank, labor force participation rate, and advanced-education unemployment rate. This information can help inform our regressor selection.
We can also loop through all of our indicators to find the variables with the strongest correlations and sort them in order, merging with the original missing summary so we don't lose sight of how the precision we imposed via imputation may impact each variable's correlation:
all_numeric_cols = df_final.select_dtypes(include=np.number).columns.drop(['Year', 'Ref_Asylum_t-1', 'Ref_Asylum_t-2', 'Ref_Growth', 'Refs_per_100'], errors='ignore')
corr_list = []
for col in all_numeric_cols:
corr_val = df_final[col].corr(df_final['ln_Ref_Asylum'])
corr_list.append((col, corr_val, abs(corr_val)))
corr_df = pd.DataFrame(corr_list, columns=['Variable', 'Correlation', 'AbsCorrelation']).sort_values('AbsCorrelation', ascending=False)
# Reset index for missing_summary_full
missing_summary_full.reset_index(inplace=True)
missing_summary_full.rename(columns={'index': 'Variable'}, inplace=True)
# Merge with corr_df on variable name
corr_df = pd.merge(corr_df, missing_summary_full, on='Variable', how='left')
corr_df = corr_df.drop(columns='missing_count')
corr_df
| Variable | Correlation | AbsCorrelation | missing_pct | |
|---|---|---|---|---|
| 0 | ln_Ref_Asylum | 1.000000 | 1.000000 | 0.000000 |
| 1 | Edu_Prim25p | 0.764332 | 0.764332 | 35.714286 |
| 2 | Edu_UpSec25p | 0.725495 | 0.725495 | 38.235294 |
| 3 | FoodInsec_pct | -0.693349 | 0.693349 | 42.016807 |
| 4 | Edu_Bach25p | 0.586804 | 0.586804 | 47.899160 |
| 5 | ln_Ref_Asylum_t-1 | 0.552996 | 0.552996 | 0.000000 |
| 6 | RuleLaw_rank | 0.545991 | 0.545991 | 34.033613 |
| 7 | GovEffect_rank | 0.539856 | 0.539856 | 34.033613 |
| 8 | LF_Fem | 0.527798 | 0.527798 | 1.680672 |
| 9 | LF_BasicEdu | -0.521405 | 0.521405 | 21.008403 |
| 10 | Enroll_Sec | 0.516126 | 0.516126 | 35.714286 |
| 11 | Unemp_AdvEdu | -0.512115 | 0.512115 | 21.428571 |
| 12 | LFPR_15_64_ILO | 0.497893 | 0.497893 | 1.680672 |
| 13 | Ref_Asylum | 0.489153 | 0.489153 | 0.000000 |
| 14 | Refugees | 0.487681 | 0.487681 | 0.000000 |
| 15 | GNI_pc | 0.485333 | 0.485333 | 2.100840 |
| 16 | Enroll_Tert | 0.485075 | 0.485075 | 36.134454 |
| 17 | Pol_Stab_rank | 0.438945 | 0.438945 | 34.033613 |
| 18 | Pop_Fem_pct | 0.383133 | 0.383133 | 0.000000 |
| 19 | Pop_Male_pct | -0.383133 | 0.383133 | 0.000000 |
| 20 | GDP_pc | 0.382734 | 0.382734 | 1.260504 |
| 21 | Unemp_Fem_Nat | -0.355100 | 0.355100 | 15.966387 |
| 22 | Unemp_Fem_ILO | -0.348388 | 0.348388 | 1.680672 |
| 23 | Unemp_BasicEdu | 0.313410 | 0.313410 | 21.428571 |
| 24 | ln_Ref_Asylum_t-2 | 0.311366 | 0.311366 | 0.000000 |
| 25 | PrimComp_pct | 0.310162 | 0.310162 | 43.277311 |
| 26 | Pop_Growth | -0.303291 | 0.303291 | 0.000000 |
| 27 | Ref_Growth_LD | 0.278001 | 0.278001 | 0.000000 |
| 28 | GDP_Growth | -0.239881 | 0.239881 | 2.100840 |
| 29 | Unemp_Tot_Nat | -0.210714 | 0.210714 | 15.966387 |
| 30 | Unemp_Tot_ILO | -0.203330 | 0.203330 | 1.680672 |
| 31 | Asylum-seekers | 0.198408 | 0.198408 | 0.000000 |
| 32 | LF_AdvEdu | 0.186202 | 0.186202 | 21.008403 |
| 33 | Tot_Pop | -0.183711 | 0.183711 | 0.000000 |
| 34 | Pop_15_64_Fem | -0.181670 | 0.181670 | 0.000000 |
| 35 | Pop_15_64_Tot | -0.181136 | 0.181136 | 0.000000 |
| 36 | Pop_15_64_Male | -0.180623 | 0.180623 | 0.000000 |
| 37 | GDP_pc_Growth | -0.119371 | 0.119371 | 2.100840 |
| 38 | Unemp_Male_Nat | -0.101633 | 0.101633 | 15.966387 |
| 39 | Unemp_Male_ILO | -0.098559 | 0.098559 | 1.680672 |
| 40 | Enroll_Prim | 0.078320 | 0.078320 | 31.512605 |
| 41 | LFPR_15p_Nat | 0.045324 | 0.045324 | 15.126050 |
| 42 | LFPR_15_24_Nat | 0.042950 | 0.042950 | 15.966387 |
| 43 | GNI_pc_Growth | -0.030822 | 0.030822 | 21.008403 |
| 44 | Pop_15_64_pct | -0.026782 | 0.026782 | 0.000000 |
| 45 | LFPR_15_24_ILO | 0.025132 | 0.025132 | 1.680672 |
| 46 | LFPR_15p_ILO | 0.016677 | 0.016677 | 1.680672 |
| 47 | LF_IntEdu | 0.012487 | 0.012487 | 21.848739 |
Now this is a great place to start with our model design in the next phase! As done below, we can pick and choose regressors, being careful to select correlated regressors without overselecting those that had lots of missing data originally.
Modeling¶
Now we are ready for the final phase of the project: modeling. This OLS regression model will be help tell us which political and economic indicators in a country may make them more likely to take more refugees in times of crisis.
OLS Regression¶
One important objective in OLS regressions is avoiding multicollinearity between regressors. If two or more variables are too highly correlated (meaning they would essentially measure the same relationship with the outcome variable) or have direct linear dependency, the model will struggle to estimate the unique effect of each variable. This would also lead to unstable coefficients and inflated standard errors, making proper statistical inference impossible.
I choose to modify two existing variables to avoid this. Below, we can see the coefficients between the measured government rankings are quite high, so I combine these into a composite rank index with equal weighting. Like with my dependent variable, I also log transform GNI per capita because it follows a similarly right-skewed distribution, and transforming it will enable me to find a more precise pattern from this variable.
# Calculate correlation between 'RuleLaw_rank', 'GovEffect_rank', and 'Pol_Stab_rank'
display(df_final[['RuleLaw_rank', 'GovEffect_rank', 'Pol_Stab_rank']].corr())
# Build composite index out of governance ranks due to correlation
df_final['Governance_index'] = (df_final['RuleLaw_rank'] + df_final['GovEffect_rank'] + df_final['Pol_Stab_rank']) / 3
# Log transform GNI_pc to fix skew; rich-country income differences are small relative to global range
df_final['ln_GNI_pc'] = np.log(df_final['GNI_pc'])
| RuleLaw_rank | GovEffect_rank | Pol_Stab_rank | |
|---|---|---|---|
| RuleLaw_rank | 1.000000 | 0.946841 | 0.830163 |
| GovEffect_rank | 0.946841 | 1.000000 | 0.786557 |
| Pol_Stab_rank | 0.830163 | 0.786557 | 1.000000 |
Here, we run our final model. When I first ran this model with each regressor in level terms (except ln_GNI_pc and the lags, which are already transformed by definition), because the variables were in such vastly different unit spaces, the condition number was too high for proper inference. However, once I z-standardized the regressors, these estimates stabilized.
The only regressors that were not scaled are the two lags of refugee count and ln_GNI_pc to preserve comparability with the DV, as well as clean interpretability as elasticities.
import statsmodels.api as sm
regressors = ['ln_Ref_Asylum_t-1', 'ln_Ref_Asylum_t-2', 'Governance_index', 'LF_Fem', 'LFPR_15_64_ILO', 'Edu_UpSec25p', 'Unemp_Tot_ILO',
'GDP_Growth', 'ln_GNI_pc', 'Pop_Growth', 'Pop_15_64_pct', 'FoodInsec_pct']
target = 'ln_Ref_Asylum'
X = df_final[regressors]
y = df_final[target].loc[X.index]
noscale_vars = ['ln_Ref_Asylum_t-1', 'ln_Ref_Asylum_t-2', 'ln_GNI_pc']
vars_to_scale = [col for col in regressors if col not in noscale_vars]
# Scale non-lag regressors to avoid high condition number caused by unit imbalance
X_scaled = X.copy()
X_scaled[vars_to_scale] = scaler.fit_transform(X[vars_to_scale])
# Add constant
X_scaled = sm.add_constant(X_scaled)
# Model with SEs clustered by country to correct for panel autocorrelation
model = sm.OLS(y, X_scaled).fit(cov_type='cluster', cov_kwds={'groups': df_final['Country of Asylum ISO']})
print(model.summary())
OLS Regression Results
==============================================================================
Dep. Variable: ln_Ref_Asylum R-squared: 0.728
Model: OLS Adj. R-squared: 0.714
Method: Least Squares F-statistic: 85.42
Date: Mon, 31 Aug 2026 Prob (F-statistic): 4.25e-41
Time: 18:39:26 Log-Likelihood: -497.77
No. Observations: 238 AIC: 1022.
Df Residuals: 225 BIC: 1067.
Df Model: 12
Covariance Type: cluster
=====================================================================================
coef std err z P>|z| [0.025 0.975]
-------------------------------------------------------------------------------------
const -2.8515 4.308 -0.662 0.508 -11.295 5.592
ln_Ref_Asylum_t-1 0.2056 0.036 5.748 0.000 0.136 0.276
ln_Ref_Asylum_t-2 -0.0331 0.015 -2.175 0.030 -0.063 -0.003
Governance_index -0.1069 0.441 -0.242 0.809 -0.972 0.758
LF_Fem 0.4259 0.372 1.144 0.253 -0.304 1.156
LFPR_15_64_ILO -0.1538 0.456 -0.337 0.736 -1.047 0.740
Edu_UpSec25p 0.9690 0.372 2.607 0.009 0.241 1.697
Unemp_Tot_ILO 0.0009 0.232 0.004 0.997 -0.454 0.455
GDP_Growth 0.1970 0.160 1.231 0.218 -0.117 0.511
ln_GNI_pc 0.9772 0.446 2.190 0.029 0.103 1.852
Pop_Growth -0.8084 0.242 -3.334 0.001 -1.284 -0.333
Pop_15_64_pct -0.4313 0.249 -1.733 0.083 -0.919 0.056
FoodInsec_pct -0.7850 0.278 -2.820 0.005 -1.331 -0.239
==============================================================================
Omnibus: 3.396 Durbin-Watson: 0.972
Prob(Omnibus): 0.183 Jarque-Bera (JB): 3.091
Skew: -0.210 Prob(JB): 0.213
Kurtosis: 3.368 Cond. No. 259.
==============================================================================
Notes:
[1] Standard Errors are robust to cluster correlation (cluster)
Results and Interpretation¶
In the regression table, we should only look at the statistically significant coefficients, meaning they have p-values less than or equal to 0.05. From our results, besides the two lag values that are obviously highly explanatory, the most significant predictive indicators are Edu_UpSec25p, ln_GNI_pc, Pop_Growth, and FoodInsec_pct. Significant at the 10% threshold is also Pop_15_64_pct.
For the scaled regressors, coefficients must be exponentiated and are interpreted in terms of standard deviations. For ln_GNI_pc, this result is interpreted as an elasticity because both it and the outcome variable are log-transformed.
For example, a one-SD increase in the share of the host population with educational attainment of at least upper secondary is associated with a 0.969 increase in log refugee popoulation, on average.
- e^0.969 - 1 = 163% increase in refugees
Second example: A one-SD increase in the share of the population suffering from food insecurity is associated with a -0.785 decrease in logged refugees, on average.
- e^-0.785 - 1 = 54.4% decrease in refugee population
Additionally, on average, a 1% increase in GNI per capita is associated with a 0.98% increase in refugee population: a near proportionate elasticity. This can be scaled up linearly to predict flows based on however much a host country may have increased their GNI per capita by in a given year, holding other factors constant.
Conclusion¶
Based on the positivity and negativity of certain coefficients in our regressions, we can identify various "push" and "pull" factors in the context of refugee movement. Positive coefficients on Edu_UpSec25p and ln_GNI_pc indicate that refugees cluster in wealthier, higher-educated, and institutionally capable socities. Negative coefficients on Pop_Growth and FoodInsec_pct indicate that refugees tend to not go to societies with existing internal socioeconomic stress. The hope is that insights like these can help governments and organizations be better prepared for future migrant crises.
Thanks for following along with this project!
Mount and Extract:¶
from google.colab import drive
drive.mount('/content/drive')
Drive already mounted at /content/drive; to attempt to forcibly remount, call drive.mount("/content/drive", force_remount=True).
!jupyter nbconvert --to html "/content/drive/MyDrive/CMPS 6160 (Data Sci)/Milestone 3"
[NbConvertApp] Converting notebook /content/drive/MyDrive/CMPS 6160 (Data Sci)/Milestone 3 to html [NbConvertApp] Writing 523006 bytes to /content/drive/MyDrive/CMPS 6160 (Data Sci)/Milestone .html