Determination of African countries with food deficit¶

1) How much human food resources are available in African countries?¶

1.1) Preprocessing¶

To answer this important question, we will need to import data from the FAO Dataset. More specifically, we will focus on the section Food Balance Sheet with respect to African countries only.

Firstly, we will remove all the columns with title "Y----F" as they contain information about how the data was obtained (Calculated, Regression, Aggregate, FAO Estimation). In this context we will consider that FAO is a highly renowned Agency and hence we can assume these values are truthful without loss of generality. Furthermore we thought that it would be very handy to have numbers as columns representing years instead of "Y----". We proceed on removing the letter Y. The helper functions clean_Fs_and_years does this cleaning on the dataframe.

Secondly, we replace all the NAN values with 0 as Item was not available.

The third step to complete the cleaning of food_balance_africa consists of adapting names of countries in order to have consistency along our different dataframes. Since some countries changed their name over the years we will rename them. In particular, Swaziland to Eswatini and South Africa to Southern Africa. The function replace_names_of_countries takes the dataframe and the country names which should be replaced.

Our Dataframe looks like this:

Analysing our DataFrame food_balance_africa we can see that it's already well structured since it contains many key - value couples such as Item Code - Item and Element Code - Element . More specifically, we will take advantage of this structure to filter out only rows characterized by Grand total as an Item and Food supply (kcal/capita/day) as an Element. The corresponding key-values are (Item Code, 2901) and (Element Code, 664). A reference to the documentation in the FAO Website explains the legend for Element Code and Element Item extensively.

In order to keep our original Dataframe food_balance_africa as a reference we create a new Dataframe food_supply_africa in which we just keep countries and food supplies for every year.

We can now group group by Area and see the supplies derived from each item available in countries for that particular year.

In order to check for anomalies in our data, we would like to analyze the timeline. We therefore transpose the dataframe and plot the timeline of how food supply in different countries evolved. Legend was suppressed as it is too large.

This analysis shows that there are two inconsistencies. We therefore check for countries containing values equal to zero.

We notice that Sudan and Ethiopia appear twice as "Sudan" and "Sudan (former)" and "Ethiopia" and "Ethiopia PDR" respectively. This is due to the fact that South Sudan gained independence in 2011 (reference to https://en.wikipedia.org/wiki/South_Sudan), and the foundation of the Federal Democratic Republic of Ethiopia (reference to https://en.wikipedia.org/wiki/Ethiopia) in 1991. From then on, Ethiopia PDR was listed as Ethiopia. With food supply being consistently constant even after division, the newly introduced country "Sudan" is assumed to further on have accounted for both countries. For this reason, we will consider them to be one single country. Consequently, the two countries' data is merged into one continuous set each. The function merge_countries takes care of this, by substituting each key in dictionary (the second argument) with its value(s).

Let's plot the newly generated data:

Next, we want to add more columns representing future years until 2020 to prepare cells for extrapolation to make predictions about possible scenarios. prepare_future is used for this.

1.2) Extrapolation¶

First of all, we want to simulate data until 2020 to match the population data. Furthermore, we also want to be able to make predicitions for individual countries to assess if they might run into food shortages in the near future.
The prediction for the new years are done by using a "Recurrent Neural Network (RNN)" and a window of size 10. Basically what we will do here is using all the past history of each country (windowed in block of 10 years each) to run a neural network and try to predict the future behaviour (up to 2020). During our test we found that the neural networks are able to predict good estimations.
As we don't want precise data, the estimations achieved by using ML are in this case more than acceptable for our purpose.
Credits: We don't know much about RNN, so the network used here is adapted from the Time series forecasting tutorial on Tensorflow, available here
Note: we already ran the networks and saved the results on Colab, running them each time requires more than an hour on Colab. For this reason, we just use the pickle here instead of running the network. This is achieved by the function predict_future.

Plotting the results:

When looking at the predicted period (after 2013), a linear approximation can be observed with some countries being forecasted to oscillate. This can be explained by the fact that the neural network method recognised a recurring pattern in the data and extrapolates this behaviour. Among all the tested methods, this one is expected to reproduce the most realistic data, as no odd developments can be observed.

1.3) Visualizing the data interactively¶

We observe geojson_world is a GeoDataFrame. Let's see how the data are sorted:

We have to make sure that geojson_world has data for every country under our analysis. In this case, we need to check for every country in Africa and more specifically every country taken into account in food_supply_africa. Since the name in geojson_world is not a good index to match and we don't have any comparable id in food_supply_africa we have to do this job by hand, which is feasible since the size is small enough. As a result, we filter geojson_world to geojson_africa. Let's display this new GeoDataFrame.

The ordered list african_country_codes contains all the countries available to be plotted as geometry is available. We found out manually that Cabo Verde, Sao Tome and Principe and Mauritius are not in this list. For the sake of simplicity, we will remove these three countries as they don't affect our analysis.

Now we can move to plot the Food supply for each country. All of this is done in the plot_map function. This function plots a world map centered in the zone of interest with interactive values while scrolling over the individual.
As we are particularly interested in the situation in 2020, we'll plot our prediction of supply for this year in the notebook map. Could be interesting also to look at the evolving of the situation over the last 50 years, so we plot a map for each decade from 1970 to 2020.
Note: the map visualized here is just for 2020, if maybe will not load. If this happens, click here

The maps for the previous decade are available here:

• 1970
• 1980
• 1990
• 2000
• 2010

By analyzing the trend over the decades, we discover that the situation in the African region partially improved over the years, especially if we compare the 2020 with the 1970.
There are some countries, especially in North Africa, that improved their supplies by 1000 kcal. In contrast, the situation in the central-southern region is almost the same as 50 years ago. Another important point to see is that there is a bit of fluctuation of the values.

2) What is the ideal amount of kcal each African country needs?¶

In this first part, we compute kcal demand for males and females for every age group. Secondly, we will conduct an extensive analysis on African demographics. Finally, we will be able to combine kcal demand with African population data into a unique dataframe that will be the answer of our inital question: What is the kcal demand of a regular person in order to be healthy?

2.1) How many kilocalories does a regular person need daily?¶

First of all, we load the calories demand datasets scraped from the webpage Calories. This information will be matched with the population datsets to receive the total caloric demand in each country, each year.

In order to better work with the information we have collected, we will make some simplifications on the data. Mainly, we will:

• take the active lifestyle column in the calories demands database. According to the World Health Organization, regular physical activity helps to maintain a healthy body and reduces the risk of disease.
• group the ages into ranges that match the ranges provided in the World Population Database

We have now obtained a caloric demand for simpler calculations in the future and stored in the two previous datasets.
Now, we need a way to match the age groups in this dataframe to the ones in the population database we obtained. As such, let's analyse how ages are represented in our caloric demand dataframes.

We can see that there are ranges of ages with different sizes (which makes sense, because different age groups have different caloric needs). The function explode_age returns the dataframe with one row per individual age.

We apply the function to our two dataframes:

Ages are now unique in each dataframe ( male_calories and female_calories ) and there's a caloric input value for each of them.

The last step to allow the match with the population database is to build the same age groups we have in that set. The compress_ages function takes care of the differences between datasets by grouping the ages into the same ranges as in the population dataset (and calculating the average needs).

We can lastly apply the function to the dataframes:

We also use the age group as new index and rename the columns:

Let's have a look at the result we have achieved and collected in our matchable dataframe male_calories and female_calories. The unit here is kcal/person/day.

As as we can see, in general, males need more calories than females. For both genders, the maximum demand occurs at the age of 15-30.

In our data story, we present a pyramid plot to show the same results. The code used is presented below.

2.2) How many people live in Africa?¶

2.2.1) Preprocessing¶

In this second part of our analysis, we load the list of African countries. Secondly, we load the World Population Database (United Nation) and therefore we obtain two dataframes: one for males and the other one for females.

We need to check if the FAO Database contains data regarding every country in Africa. We will check the intersection with the list af_countries.

As expected, many countries were not present in the FAO Database. The countries to remove are the following: Eritrea, Burundi, Comoros, Democratic Republic of the Congo, Equatorial Guinea, Libya, Seychelles, Western Sahara, South Sudan, and Somalia. Furthermore, Mayotte and Réunion are French territory islands so they will be removed as well.

Now we can proceed to load the population data and clean it

The function clean_pop_df takes care of all the required cleaning. What it does is removing all the unnecessary columns and retaining just the rows of the needed countries. Then it multiplies the value by 1000 (Since the population data is per 1000 people, all entries are multiplied by the same factor (1000) to return to the real value).

World Population Database (United Nation) is now loaded and cleaned. This preprocessing is necessary in order to sort things out for next more complex steps.

Let's have a look at the final version of male population data grouped by age:

2.2.2) Interpolating the data on African population¶

In this context, the population dataframe for males pop_male and for females pop_female contains measurements of population censi, for years from 1950 to 2020 with a frequency of 5 years. Next, data is interpolated in order to obtain values for intermediate years.

For our interpolation method to work, we need to know if the population evolution in these intervals of 5 years is linear. In order to do so, we need to visualize the growth of the population for a group of countries (plotting all of them would be confusing). So we select 3 countries randomly and plot a simple animation of the growth over time.
Note: the code below is used to generate the HTML animation. If it doesn't load click on this link

The animation shows an almost linear growth for the 3 countries considered (with their respective scale), so we can continue with the interpolation.

Now we can apply our function interpolate_years, a simple linear interpolation, in order to obtain a frequency of 1 year.

Let's see how the new dataframes for males and females population look like:

2.2.3) Computing the total African population¶

Lastly, we will compute the total population per year. This new dataframe pop_tot will be useful for the next section of our analysis. obtain_total_pop does just this, combining the population in pop_male_africa and pop_female_africa.

For the next analysis we will need to match this data with the food_balance_africa. We proceed to give to our population data the same shape as the other datasets by using the function reshape_pop_dataframe.

2.3) Estimantion of ideal human food demand in Africa¶

Now we multiply each column of the population data for each matching age_group in the calories table (which is squeezed to enable multiplication, similar to transposing rows/columns of the dataset). All of this is done in get_calories_need
We obtain two datasets: male_cal_need_africa and female_cal_need_africa reporting total calories needed for each country in each year per age group per gender. The unit here is kcal/day.

Once we have the calories needed for both genders, we can aggregate the total caloric need of african countries into total_cal_need_africa. The function obtain_total_cal_need does just this, returning a dataframe with the calories needed for all the population of a country in a year, in the unit kcal/year.

Let's take a look at the total calories dataframe total_cal:

For the sake of consistency, we will now reshape our dataframe total_cal into a new one total_cal_final according to the same schema seen for food_supply_africa.

Drawing a sample of the final shaped dataframe total calories total_cal_need_africa:

Let's go on with a interactive visualization of the data in order to understand the trend over countries. The following map is based on data for 2020.
However, it's important also to understand what are the possible changing over time, so we plot the same map over the range from 1970 to 2020 (step of 10 years).
If the maps for 2020 doesn't show click here
Link to the other maps:

• 1970
• 1980
• 1990
• 2000
• 2010

By looking at the scale change over the years, it's possible to note how the needs of the African population have constantly grown (connected to the increase of population too). This is not reflected in the African food supply, that doesn't increase at the same rate as the population.

3) Which countries are in food deficit?¶

Next, an interesting comparison is introduced between the two dataframes we have obtained in the fist two parts of our analysis. More specifically, the analysis will take into account the total population dataframe pop_tot_africa and the food_supply_africa. With regard to the FAO Dataframe of food supply, we will need to transform the unit in kcal/year in order to compare results appropriately.

The function obtain_difference takes into account our dataframes to compute which countris have enough caloric food supply to actually meet their needs.

3.1) Visualizing the data¶

Let's start by doing a simple barplot of the deficit per persona/year in each country. As our main point of interest is the present, we will start with a graph showing next year sitution:

This plot already suggests that only by smart redistribution, Africa could sustain its own food demand. However, having the capabilities and know-how to efficiently set up a food aid operation is harder than it seems. European countries on the other hand have a lot more experience in this field, and they are expected to have an even higher amount of excess food, making it easier to provide for this whole operation. Therefore They will be considered to provide the difference in African countries.

For a better visualization, it is convenient to see the evolution of the kcal demand for all the years of interest. By using the draw_demand_bar, we wil plot the information for every year combined with an animation to move back and forth in time.
Note: the code below is used to generate the HTML animation. If the animation doesn't work inside the notebook, click on this link

As the animation shows, the African situation sensitevely improved over the last 60 years, starting from a share of 75% of starving countries in 1961 to "just" 21% in 2020.
During the years the situation improved differently. The improvement was basic up to the nineties, than dramatically changed from 2000 to now.

For our data story we plotted the same animation in a slider manner, by using the below function for Africa's and Europe's surplus situation.

Let's now move on to a more understandable map visualization. As usual, we are interested in 2020 for our endpoint, but also on the changing over the years. Note: if it doesn't show, the 2020 map can be found here
Links to the other years:

• 1970
• 1980
• 1990
• 2000
• 2010

We found an analysis here about the GDP (Gross Domestic Product) in African countries, which we think can be a good way to evaluate the correctness of our analysis. We show to the left the heat map of Africa in terms of each countries GDP per capita.

Observing the evolution over the last 50 years allow us to better understand and comment the animation above. As we can see from the 1970 map, almost all Africa lived in starving condition with the lowest peak in Guinea-Bissau (at huge deficit of 700 kcal/persona/day needed).

Individual case studies prove the data's accuracy. For example, the Ethiopian famine in the 80's is impressively reproduced and manifests itself as Ethiopia drops to the lowest rank. In general, a very positive development can be observed, as more and more countries manage to reach a surplus regime every single decade, with the highest increase occurring just recently in the last 10 years.

Ever since 2001, more than half of the examined countries show a net surplus. As of 2019, all of the countries in red were determined to be either war-riddled or politically fragile. Exceptions to the rule are Namibia and Eswatini, which both boast a relatively high GDP per capita (ranked 10th and 11th for the African continent, respectively). Thus, the only explanation would be inequality amongst the population or insufficient distribution of available resources.

Determination of European countries with food surplus¶

1) How much human food resources are available for European countries?¶

1.1) Preprocessing¶

To answer this important question, we will need to import data from the FAO Dataset. More specifically, we will focus on the section Food Balance Sheet with respect to European countries only.

European countries will be analysed following the same strategy we used for African countries in order to be consistent also in the way by which we assess whether countries are in deficit or have a surplus. To start off, we will:

• remove all the columns with title "Y----F".
• replace all the NAN values with 0 as Item was not available.

The third step to complete the cleaning of food_balance_europe consists on adapting names of countries in order to have consistency along our different dataframes.

The easiest of these changes that we observe in our dataframe is The former Yugoslav Republic of Macedonia should become North Macedonia.

Our Dataframe looks like this:

Given the European countries analysis, and since the structure of this dataset is equivalent to that one, we can again obtain the pairs (Item Code, 2901) and (Element Code, 664) for our Europe analysis.

We can now group by Area and see the supplies derived from each item available in countries for each particular year.

In order to check for anomalies in our data, we would like to analyze the timeline. We therefore transpose the dataframe and plot the timeline of how food supply in different countries evolved. Legend was suppressed as it is too large.

We can observe a similar situation in this graph as we did in the African countries analysis. As such, we can assume that some countries may have changed names or maybe some other situations occured, which caused countries data to stop being recorded/collected.

Let's then see which countries have zeros in the values for food supplies.

Respecting the cronology, we observe that there were some countries on which data stopped being collected, as well as others from which data started being collected after the initial collection year of 1950. After some research on these countries, we found out that:

• USSR (Union of Soviet Socialist Republics) was a union of a lot of countries, some of which were not even located in Europe. The Union was dissolved in 26 December 1991, and so the countries Belarus, Ukraine, Estonia, Republic of Moldova, Russian Federation, Latvia, Lithuania are the European countries that became independent in Europe.
• Yugoslav SFR was made up of the countries that became Bosnia and Herzegovina, Croatia, Serbia and Montenegro, North Macedonia, Slovenia, all of which obtained their independence between 25 June 1991 - 27 April 1992
• Czechoslovakia was dissoluted in 1 January 1993, after a period called The Velvet Revolution (because of the peaceful ways), and split into the two countries Czechia and Slovakia
• Serbia and Montenegro separated themselves in 2006, breaking up the last union still recognized as a successor of Yougoslavia
• Belgium-Luxembourg (2003) does not have a main reason to exist, as there is no major event happening in 2003 that would justify the mixing of the two countries. Our understanding lead us to believe this was probably just a grouping created in FAO when collecting the data from the two countries.

Let's then take care of all these cases..

We observe a drop in the food supply from most countries that came from the former USSR, which we find to be interesting. While it might not be the case for all of these countries, some of them are vey poor and as such, when separated from the world power that was the USSR, their ability to provide and produce for themselves may have dropped significantly, which is coherent with what we observe. As such, we continue our analysis with these values.

1.2) Interpolation¶

Next, we want to add more columns representing future years until 2020 to prepare cells for extrapolation to make predictions about possible scenarios.

Similar to the analysis of African countries, the same conclusions can be drawn about using the neural network method.

Visualizing our results with a interactive heatmap:

We notice, after our manual analysis on the countries that Malta is not displayed in the json. Hence, we won't consider it for our analysis

Moving into the plot of the European countries to see their actual and past supply. Note: the map for 2020 (in case it doesn't load) is available here
Links to other years:

• 1970
• 1980
• 1990
• 2000
• 2010

Analyzing the data shows a trend scaled up but mostly similar African case. The available food supply in Europe over the last 50 years remained mostly the same, with an available supply between 2500 to 3800 kcal/persona/day.
Interesting to note the drop the ex USSR had in 2000 after the division, drop already emerged during the previous cleaning of the dataset.

2) What is the ideal amount of kcal each European country needs?¶

Once again, we do a very similar analysis on European countries as we did for the African ones.

2.1) How many people live in Europe?¶

2.1.1) Preprocessing¶

In this second part of our analysis, we load the list of European countries. Secondly, we load the World Population Database (United Nation) and therefore we obtain two dataframes: one for males and the other one for females.

We need to check if the FAO Database contains data regarding every country in Europe. We will check the intersection with the list eu_countries.

Because there are less countries in Europe, and also because most European countries are part of ONU, we expected most countries to be present in both the FAO database and the population database. These Channel Islands are a small set of islands in the English Channel, and because they are so small, we can safely remove them from our scope of interest.

We now obtain the population for each gender in all european countries.

Let's have a look at the final version of male population data grouped by age:

2.1.2) Interpolation¶

Similarly to the analysis performed on Africa, we once again interpolate the years with a frequency of 5 years in pop_male_europe, to have a frequency of 1 year. Before doing so, we check again if the evolution in these 5 years intervals occurs in a linear manner.
Note: as usual, if the animation is not visible in the notebook, click here

Even if the growth is not so linear as a whole as before, the animation clearly shows that it is linear inside each group of 5 year. As our interpolation works by interpolating on each of these groups, we can proceed with it

Let's see how the new dataframes for males and females population look like:

2.1.3) Computing total European population¶

Lastly, we will compute the total population per year. This new dataframe pop_tot will be useful for the next section of our analysis.

For the next analysis we will need to match this data with the food_balance_europe. We proceed to give to our population data the same shape as the other dataset

3) Estimantion of ideal human food demand in Europe¶

Now we multiply each column of the population data for each matching age_group in the calories table (that here we squeeze to allow the multiplication, similar to a transpose rows/columns of the dataset).
We obtain two datasets: male_cal_need_europe and female_cal_need_europe reporting total calories needed for each country in each year per age group per gender. The unit here is kcal/day.