Tutorial 1.¶

Basic statistics¶

In this tutorial we will load data from a csv file in jupyter to obtain statistical information about the data domain using several Pharo frameworks.

First of all, be sure that NeoCSV, DataFrame, Roassal and JupyterTalk are installed in your kernel Image. Although it is possible to load packages from the jupyter client, we suggest you to load it using the workspace or the catalog browser and save it to keep the changes. Doing "Smalltalk saveSession" could break the image. Normally the kernel will lose the connection and you will be forced to close and halt the kernel from the File Menu but sometimes the Pharo Image could be damaged.

"Install NeoCSV"
Gofer it
   smalltalkhubUser: 'SvenVanCaekenberghe' project: 'Neo';
   configurationOf: 'NeoCSV';
   loadStable.
"install Roassal"
Gofer it
    smalltalkhubUser: 'ObjectProfile' project: 'Roassal2';
    configurationOf: 'Roassal2';
    loadStable.
    
"Install DataFrame and NeoCSV"
Metacello new
  baseline: 'DataFrame';
  repository: 'github://PolyMathOrg/DataFrame';
  load.
    
Metacello new 
    baseline: 'JupyterTalk';
    repository: 'github://jmari/JupyterTalk/repository';
    load:'all'

"this file is windows-1258 encoded so we have to load into Pharo kernel using the correct encoding"

stream := ZnCharacterReadStream 
                on:'/Users/Cat/Dropbox/Master Ciencia de les dades/S1.1.mineria de dades/PAC2/countries.csv'
                        asFileReference binaryReadStream
                encoding: #windows1258.
                
                  
arrayOfRows := (NeoCSVReader on: stream)
                            separator: $;;
                            upToEnd.

paisos := DataFrame fromRows:(arrayOfRows copyFrom:2 to:arrayOfRows size).
paisos columnNames: (arrayOfRows at:1)

Loading CSV content.¶

Before loading country.csv file, examine this file opening it in your favourite text editor, check the character encoding, its character separator and decimal point character. As you will see, it's windows-1258 encoded, separated by ';' and it uses european floating point format, ',' is the decimal point.
We use Zinc Streams and NeoCSV to encode the content and load it into an Array.
Let's create a DataFrame, since this Array has a first row containing the column headers we need to create the DataFrame from rows starting from two up to the end. We will assing the column names from the first row in the array.

Let's show rows from one to thirty three including its heading. Note that encoding is correct

"JupyterTalk will transform those Strings to utf-8, look at:#32 Cameroon | Yaoundé"
self display openInJupyter: (OrderedCollection new 
                                    add:paisos columnNames;
                                    addAll: (paisos asArrayOfRows copyFrom:1 to:33);
                                    yourself) .

Drawing a Boxplot.¶

Now, let's get a basic boxplot from 'URBAN_POPULATION' column. We need to change the column type from String to Integer.
The BoxPlot is part of the Roassal Package. DataFrame has helper methods to draw basic statistical drawings using Roassal.

"CONVERT URBAN_POPULATION TO INTEGER"

newCol := (paisos column:#URBAN_POPULATION) collect:[:v| v ifNil:[0] ifNotNil:[v asInteger]].
paisos column:#URBAN_POPULATION put:newCol.

b := (paisos column:#URBAN_POPULATION) boxplot.
self display openInJupyter: b

Draw a Histogram.¶

Now, let's create a histogram from a quantitative column. We will use POPULATION column so first we need to change column type to Integer and calculate maximun and minimun using DataSerie methods min and max.
We will use a Bag in order to categorize the POPULATION column.

"CONVERT POPULATION TO INTEGER"

newCol := (paisos column:#POPULATION) collect:[:v| v ifNil:[nil] ifNotNil:[v asInteger]].
paisos column:#POPULATION put:newCol.
maxPopulation := (paisos column:#POPULATION) max.
minPopulation := (paisos column:#POPULATION) min.
domainSize := maxPopulation - minPopulation.
bag := Bag new.
newCol do:[:each| bag add:((((each - minPopulation)/ domainSize*100) asInteger) ) ]

b:= RTGrapher new.
ds := RTData new.
ds barShape width: 10.
ds points: bag valuesAndCounts.
b add: ds.

uuid :=self display openInJupyter: b  extent:600@650

We can modify an object shown above. #openInJupyter returns an uuid for each displayed object. we can use this uuid to refresh the picture after we change any of its properties.

"we can modify an object shown in other response."
b axisX title:'Population in million of people'.
b axisY title:'Number of countries'.
self display refresh: uuid

Interactive visualization with Roassal.¶

Let's visualize a nice drawing using Roassal. We can interact with our drawing.
We need to include the Roassal javascript library in our jupyter markdown document.

self loadScript: IPRoassal js

| view coll col2 n b |
view := RTView new.
coll := (paisos column:#POPULATION).
n := RTMultiLinearColorForIdentity new objects: coll.
coll
    doWithIndex: [ :r :index | 
        view
            add:
            ((RTBox new
                color: [ :value | n rtValue: r ];
                size: r/ 1000)
            elementOn: index) ].
col2 := (paisos column:#NAME).
RTFlowLayout new applyOn: view elements.
view elements do: [ :e | e @ (RTPopup text: [ :el | col2 at:el ]) ].

b := RTAxisAdaptedBuilder new.
b view: view.
b margin: 20.
b objects: view elements.
b build.

self display 
        interactionOn;
        openInJupyter: b extent:650@640

Drawing a Bar chart.¶

Now, let's get a bar chart showing the number of countries in each continent. We use a Bag to get a count for each continent.

bag := Bag new.
(paisos column:#CONTINENT) do:[:each| bag add:each ]

b:= RTGrapher new.
ds := RTData new.
ds barShape width: 20.
ds interaction highlight.
ds points: bag valuesAndCounts associations.
ds y: [:each| each value].
ds barChartWithBarTitle: [:each|each key].
b add: ds.
b axisX
	noTick;
	noLabel.

self display openInJupyter: b extent:640@600

Basic statistics.¶

We will finish this tutorial getting basic statistcs from the column population.

self display openInJupyter:(paisos column:#POPULATION) summary asStringTable

         |     POPULATION  ---------+-----------------    Min  |          16954  1st Qu.  |        1952975   Median  |        5989065     Mean  |  3.122843657e7  3rd Qu.  |       20013660      Max  |     1215609480

NAME	CAPITAL	TOTAL_AREA_KM2	POPULATION	DENSITY_KM2	CONTINENT	URBAN_POPULATION	RURAL_POPULATION	CRUDE_OIL_BAR_DAY	OIL_RES_MIL_BAR	GAS_TONS	GAS_RES_MIL_M_3	GDP_MILLION_$	GDP_GROW_RATE	GDP_$_PER_CAPITA	DOCTORS	ILLITERACY	PRIMARY_SECTOR	SECONDARY_SECTOR	TERTIARY_SECTOR	PHONES_RATE	NATURAL_GROWTH	MEM_LIFE_EXP	WOMEN_LIFE_EXP	MAIN_SECTOR	MAIN_GROUP
Afghanistan	Kabul	647500	24326959	37,6	Asia	20	80	nil	nil	nil	nil	0	0	600	4797	nil	57	13	29	0	26	47	46	Primary	Rural
Albania	Tirane	28750	3453505	120,1	Europe	37	63	nil	nil	nil	nil	4100	6	1210	574	0	50	nil	18	nil	19,9	69,9	75,5	Primary	Rural
Algeria	Algiers	2381740	29181456	12,3	Africa	56	44	1395000	9200	59300000	3700000	108700	3,5	3800	1119	50	13	24	36	0,03	28,3	65,8	66,6	Tertiary	Urban
Andorra	Andorra la Vella	450	67569	150,2	Europe	nil	nil	nil	nil	nil	nil	1000	0	16200	nil	nil	nil	nil	nil	nil	nil	nil	nil	nil	nil
Angola	Luanda	1246700	10339364	8,3	Africa	32	68	nil	nil	nil	nil	7400	4	700	13489	72	71	nil	18	0	27	42,9	46,1	Primary	Rural
Antigua & Barbuda	Saint John's	440	65619	149,1	America	36	64	nil	nil	nil	nil	0	4,2	6600	nil	nil	nil	18	72	nil	9,5	70	73	Tertiary	Rural
Argentina	Buenos Aires	2766890	34673391	12,5	America	88	12	805000	nil	nil	nil	0	-4,4	8100	340	nil	nil	30	50	0,1	11	67	74	Tertiary	Urban
Armenia	Yerevan	29800	3590722	120,5	Asia	69	31	nil	nil	nil	nil	0	5,2	2560	nil	nil	nil	nil	nil	nil	15	nil	nil	nil	Urban
Australia	Canberra	7686850	18562252	2,4	Oceania	85	15	615000	nil	nil	nil	405400	3,3	22100	438	nil	5	15	72	0,47	8,4	73,3	79,6	Tertiary	Urban
Austria	Vienna	83850	8014617	95,6	Europe	56	44	nil	nil	nil	nil	152000	2,4	19000	345	0	7	35	55	0,53	0,9	72,1	78,8	Tertiary	Urban
Azerbaijan	Baku	86600	7892712	91,1	Asia	56	44	nil	nil	nil	nil	0	-17	1480	nil	nil	nil	nil	nil	nil	26,4	nil	nil	nil	Urban
Bahamas; The	Nassau	13940	259413	18,6	America	87	13	nil	nil	nil	nil	0	2	18700	809	5	1	11	62	0,48	14,3	67	74	Tertiary	Urban
Bahrain	Manama	620	590784	952,9	Asia	90	10	nil	nil	nil	nil	0	-2	12000	713	nil	nil	32	65	0,23	23,3	65	68,4	Tertiary	Urban
Bangladesh	Dhaka	144000	131066751	910,2	Asia	18	82	nil	nil	nil	nil	0	4,6	1130	6166	nil	57	nil	30	0	21,6	56	56	Primary	Rural
Barbados	Bridgetown	430	257010	597,7	America	47	53	nil	nil	nil	nil	0	2	9800	1042	2	5	nil	nil	0,41	8,2	71,9	76,9	Primary	Rural
Belarus	Minsk	207600	10468730	50,4	Europe	71	29	nil	nil	nil	nil	49200	-10	4700	nil	nil	nil	nil	nil	nil	nil	nil	nil	nil	Urban
Belgium	Brussels	30510	10099019	331	Europe	97	3	nil	nil	nil	nil	197000	2,3	19500	317	0	2	23	62	0,5	1,4	70	77	Tertiary	Urban
Belize	Belmopan	22960	219241	9,5	America	47	53	nil	nil	nil	nil	0	2	2750	2046	7	27	13	45	0,07	30	nil	nil	Tertiary	Rural
Benin	Porto-Novo	112620	5706582	50,7	Africa	31	69	nil	nil	nil	nil	7600	6	1380	16025	nil	67	7	24	nil	29,9	nil	nil	Primary	Rural
Bhutan	Thimphu	47000	1822305	38,8	Asia	6	94	nil	nil	nil	nil	0	6	730	nil	nil	87	0	3	nil	21,9	49,2	47,8	Primary	Rural
Bolivia	Sucre / La Paz	1098580	8073920	7,3	America	61	39	nil	nil	nil	nil	0	3,7	2530	2214	nil	42	17	38	0,02	29	nil	nil	Primary	Urban
Bosnia - Herzegovina	Sarajevo	51233	3222635	62,9	Europe	49	51	nil	nil	nil	nil	1000	0	300	nil	nil	nil	nil	nil	nil	nil	nil	nil	nil	Rural
Botswana	Gaborone	600370	1425275	2,4	Africa	28	72	nil	nil	nil	nil	4500	1	3200	7185	nil	4	nil	66	0,02	35,6	52,7	59,3	Tertiary	Rural
Brazil	Brasilia	8511965	162698486	19,1	America	78	22	800000	nil	nil	nil	0	4,2	6100	685	nil	24	21	5	0,08	20,7	62,3	67,6	Primary	Urban
Brunei	Bandar Seri Begawan	5770	299953	52	Asia	58	42	nil	nil	nil	nil	0	2	15800	1469	nil	3	17	69	0,16	24,3	72,6	76,4	Tertiary	Urban
Bulgaria	Sofia	110910	8753260	78,9	Europe	71	29	nil	nil	nil	nil	43200	2,4	4920	nil	nil	nil	48	33	nil	11,7	68,2	74,4	Secondary	Urban
Burkina	Ouagadougou	274200	10713625	39,1	Africa	27	73	nil	nil	nil	nil	7400	4	700	1359	nil	92	2	4	0	28,7	45,6	48,9	Primary	Rural
Burma	Rangoon	678500	45933719	67,7	Asia	nil	nil	nil	nil	nil	nil	0	6,8	1000	nil	nil	nil	nil	nil	nil	nil	nil	nil	nil	nil
Burundi	Bujumbura	27830	6398950	229,9	Africa	8	92	nil	nil	nil	nil	4000	2,7	600	572	50	93	2	4	0	32	50	54	Primary	Rural
Cambodia	Phnom Penh	181040	10860260	60	Asia	21	79	nil	nil	nil	nil	0	6,7	660	27000	52	70	nil	14	0	24,9	46,5	49,4	Primary	Rural
Cameroon	Yaoundé	475440	13915813	29,3	Africa	45	55	nil	nil	nil	nil	16500	1,8	1200	12540	nil	nil	6	nil	0	32,6	51	54	nil	Rural
Canada	Ottawa	9976140	28744482	2,9	America	77	23	2460000	6900	137700000	1900000	0	2,1	24400	449	4	3	19	68	0,7	7,6	73,3	80	Tertiary	Urban
Cape Verde	Praia	4030	448975	111,4	Africa	54	46	nil	nil	nil	nil	440	4,6	1040	4208	nil	33	32	14	0,01	24,4	63	67	Primary	Urban