Software and stuff

Necessary things to do:

• Windows users: Install Windows Subsystem for Linux and a Unix distribution

• Install Julia

• Install these Julia packages: Expectations, Distributions, LinearAlgebra, BenchmarkTools

These slides are based on Software Carpentry, notes by Grant Mcdermott, QuantEcon, Julia documentation, etc

Why learn the shell?

What is the shell?

The shell is the interface for interacting with your operating system, typically we are referring to the command line interface (terminal, command prompt, bash, etc)

A lot of what you can do in the shell can be done in Julia itself, why bother with it?

Why learn the shell?

Not everything can be done directily in your usual programming language

Command line is fast, powerful, and relatively easy to use, especially with modern shells like zsh or fish

Writing shell scripts is reproducible and fast, unlike clicking buttons on a GUI

If you want to use servers or any high performance computing you are likely to need to use shell

Why learn the shell?

You can automate your entire research pipeline with shell scripts (e.g. write something that calls multiple languages to execute your code then compiles your latex for the paper)

It really gets to the fundamentals of interacting with a computer (loops, tab-completions, saving scripts, etc)

It gets you understanding how to write code in terms of functions which will be important for any programming you do in scripting languages like Julia, R, MATLAB, or Stata

What is the shell?

The shell is basically just a program where you can type in commands to interact with the kernel and hardware

What is the shell?

The most common one is bash, Bourne again shell, because it comes default on Macs and Linux

I use fish, friendly interactive shell, because it comes default with a lot of nice features, all the commands still work identically to bash

Shell basics

When you open up the shell you should see a prompt, usually starting with $ (don't type this)

$

We can type in one command, ls which lists the contents of your current directory

$ ls

## 2a_coding.Rmd
## 2a_coding.html
## 2a_coding_files
## figures
## my-css.css
## sandbox

My current directory is the one for this set of slides

Shell basics

Commands come with potential options or flags that modify how they act

$ ls

## 2a_coding.Rmd
## 2a_coding.html
## 2a_coding_files
## figures
## my-css.css
## sandbox

$ ls -l # long form command

## total 360
## -rw-r--r--@ 1 ir229  staff   43236 Jan 29 14:39 2a_coding.Rmd
## -rw-r--r--@ 1 ir229  staff  107662 Jan 29 14:37 2a_coding.html
## drwxr-xr-x  7 ir229  staff     224 Jan 29 14:39 2a_coding_files
## drwxr-xr-x  9 ir229  staff     288 Jan 28 15:22 figures
## -rw-r--r--@ 1 ir229  staff    6573 Jan 22 12:44 my-css.css
## drwxr-xr-x  8 ir229  staff     256 Jan 29 14:39 sandbox

Shell basics

Options start with a dash and then
a sequence of letters denoting which options you want

e.g. this lists files in long form l, sorted descending by size (S), with sizes in a human-readable format (h)

$ ls -lSh

## total 360
## -rw-r--r--@ 1 ir229  staff   105K Jan 29 14:37 2a_coding.html
## -rw-r--r--@ 1 ir229  staff    42K Jan 29 14:39 2a_coding.Rmd
## -rw-r--r--@ 1 ir229  staff   6.4K Jan 22 12:44 my-css.css
## drwxr-xr-x  9 ir229  staff   288B Jan 28 15:22 figures
## drwxr-xr-x  8 ir229  staff   256B Jan 29 14:39 sandbox
## drwxr-xr-x  7 ir229  staff   224B Jan 29 14:39 2a_coding_files

Shell basics

Finally commands have an argument that the command operates on

The previous ls calls were operating on the current directory,
but we could use it on any directory we want

$ ls -lSh ~/Desktop/git

## total 0
## drwxr-xr-x  98 ir229  staff   3.1K Mar  2  2018 rec_markets
## drwxr-xr-x  64 ir229  staff   2.0K Nov 15  2018 lrr-mcmc-dice
## drwxr-xr-x@ 45 ir229  staff   1.4K Mar  4  2018 rc_paper
## drwxr-xr-x  44 ir229  staff   1.4K May 20  2019 steering-the-climate-system
## drwxr-xr-x  40 ir229  staff   1.3K Dec  1  2018 hr_recs
## drwxr-xr-x  30 ir229  staff   960B Jul 19  2019 robust-control-jl
## drwxr-xr-x  26 ir229  staff   832B May 20  2019 dynamic-stochastic-dice
## drwxr-xr-x  26 ir229  staff   832B Jan 19 16:10 optimal-climate-policy
## drwxr-xr-x  24 ir229  staff   768B Mar 12  2019 ND-Flaring
## drwxr-xr-x  24 ir229  staff   768B Sep 30 20:43 optimal-climate-policy-aej
## drwxr-xr-x  19 ir229  staff   608B Jan 15 18:23 irudik.github.io
## drwxr-xr-x  15 ir229  staff   480B Jan 24 11:00 hurricane_forecasts
## drwxr-xr-x  14 ir229  staff   448B Jan 27 13:59 seere-lab
## drwxr-xr-x  11 ir229  staff   352B Apr 29  2019 drillinginfo-data-import
## drwxr-xr-x  11 ir229  staff   352B Jan 21 21:11 ed_rubin_class
## drwxr-xr-x  11 ir229  staff   352B Aug 18 17:13 growth-versus-levels
## drwxr-xr-x  11 ir229  staff   352B Jan 20 12:51 purple_air
## drwxr-xr-x  10 ir229  staff   320B May 20  2019 external-impacts-rps
## drwxr-xr-x  10 ir229  staff   320B Sep 17 19:23 nascar_and_lead
## drwxr-xr-x   9 ir229  staff   288B Oct 27 18:05 aem-7xxx
## drwxr-xr-x   6 ir229  staff   192B May  7  2019 workflow
## drwxr-xr-x   5 ir229  staff   160B Jan 19 15:42 aem7130
## drwxr-xr-x   5 ir229  staff   160B Feb  1  2018 climate-learning-hpc

Shell basics

To see what commands and their options do, use the man (manual) command

q exits the man page, and spacebar lets you skip down by a page

$ man ls

## 
## LS(1)                     BSD General Commands Manual                    LS(1)
## 
## NNAAMMEE
##      llss -- list directory contents
## 
## SSYYNNOOPPSSIISS
##      llss [--AABBCCFFGGHHLLOOPPRRSSTTUUWW@@aabbccddeeffgghhiikkllmmnnooppqqrrssttuuwwxx11] [_f_i_l_e _._._.]
## 
## DDEESSCCRRIIPPTTIIOONN
##      For each operand that names a _f_i_l_e of a type other than directory, llss
##      displays its name as well as any requested, associated information.  For
##      each operand that names a _f_i_l_e of type directory, llss displays the names
##      of files contained within that directory, as well as any requested, asso-
##      ciated information.
## 
##      If no operands are given, the contents of the current directory are dis-
##      played.  If more than one operand is given, non-directory operands are
##      displayed first; directory and non-directory operands are sorted sepa-
##      rately and in lexicographical order.
## 
##      The following options are available:
## 
##      --@@      Display extended attribute keys and sizes in long (--ll) output.
## 
##      --11      (The numeric digit ``one''.)  Force output to be one entry per
##              line.  This is the default when output is not to a terminal.
## 
##      --AA      List all entries except for _. and _._..  Always set for the super-
##              user.
## 
##      --aa      Include directory entries whose names begin with a dot (_.).
## 
##      --BB      Force printing of non-printable characters (as defined by
##              ctype(3) and current locale settings) in file names as \_x_x_x,
##              where _x_x_x is the numeric value of the character in octal.
## 
##      --bb      As --BB, but use C escape codes whenever possible.
## 
##      --CC      Force multi-column output; this is the default when output is to
##              a terminal.
## 
##      --cc      Use time when file status was last changed for sorting (--tt) or
##              long printing (--ll).
## 
##      --dd      Directories are listed as plain files (not searched recursively).
## 
##      --ee      Print the Access Control List (ACL) associated with the file, if
##              present, in long (--ll) output.
## 
##      --FF      Display a slash (`/') immediately after each pathname that is a
##              directory, an asterisk (`*') after each that is executable, an at
##              sign (`@') after each symbolic link, an equals sign (`=') after
##              each socket, a percent sign (`%') after each whiteout, and a ver-
##              tical bar (`|') after each that is a FIFO.
## 
##      --ff      Output is not sorted.  This option turns on the --aa option.
## 
##      --GG      Enable colorized output.  This option is equivalent to defining
##              CLICOLOR in the environment.  (See below.)
## 
##      --gg      This option is only available for compatibility with POSIX; it is
##              used to display the group name in the long (--ll) format output
##              (the owner name is suppressed).
## 
##      --HH      Symbolic links on the command line are followed.  This option is
##              assumed if none of the --FF, --dd, or --ll options are specified.
## 
##      --hh      When used with the --ll option, use unit suffixes: Byte, Kilobyte,
##              Megabyte, Gigabyte, Terabyte and Petabyte in order to reduce the
##              number of digits to three or less using base 2 for sizes.
## 
##      --ii      For each file, print the file's file serial number (inode num-
##              ber).
## 
##      --kk      If the --ss option is specified, print the file size allocation in
##              kilobytes, not blocks.  This option overrides the environment
##              variable BLOCKSIZE.
## 
##      --LL      Follow all symbolic links to final target and list the file or
##              directory the link references rather than the link itself.  This
##              option cancels the --PP option.
## 
##      --ll      (The lowercase letter ``ell''.)  List in long format.  (See
##              below.)  If the output is to a terminal, a total sum for all the
##              file sizes is output on a line before the long listing.
## 
##      --mm      Stream output format; list files across the page, separated by
##              commas.
## 
##      --nn      Display user and group IDs numerically, rather than converting to
##              a user or group name in a long (--ll) output.  This option turns on
##              the --ll option.
## 
##      --OO      Include the file flags in a long (--ll) output.
## 
##      --oo      List in long format, but omit the group id.
## 
##      --PP      If argument is a symbolic link, list the link itself rather than
##              the object the link references.  This option cancels the --HH and
##              --LL options.
## 
##      --pp      Write a slash (`/') after each filename if that file is a direc-
##              tory.
## 
##      --qq      Force printing of non-graphic characters in file names as the
##              character `?'; this is the default when output is to a terminal.
## 
##      --RR      Recursively list subdirectories encountered.
## 
##      --rr      Reverse the order of the sort to get reverse lexicographical
##              order or the oldest entries first (or largest files last, if com-
##              bined with sort by size
## 
##      --SS      Sort files by size
## 
##      --ss      Display the number of file system blocks actually used by each
##              file, in units of 512 bytes, where partial units are rounded up
##              to the next integer value.  If the output is to a terminal, a
##              total sum for all the file sizes is output on a line before the
##              listing.  The environment variable BLOCKSIZE overrides the unit
##              size of 512 bytes.
## 
##      --TT      When used with the --ll (lowercase letter ``ell'') option, display
##              complete time information for the file, including month, day,
##              hour, minute, second, and year.
## 
##      --tt      Sort by time modified (most recently modified first) before sort-
##              ing the operands by lexicographical order.
## 
##      --uu      Use time of last access, instead of last modification of the file
##              for sorting (--tt) or long printing (--ll).
## 
##      --UU      Use time of file creation, instead of last modification for sort-
##              ing (--tt) or long output (--ll).
## 
##      --vv      Force unedited printing of non-graphic characters; this is the
##              default when output is not to a terminal.
## 
##      --WW      Display whiteouts when scanning directories.  (--SS) flag).
## 
##      --ww      Force raw printing of non-printable characters.  This is the
##              default when output is not to a terminal.
## 
##      --xx      The same as --CC, except that the multi-column output is produced
##              with entries sorted across, rather than down, the columns.
## 
##      The --11, --CC, --xx, and --ll options all override each other; the last one
##      specified determines the format used.
## 
##      The --cc and --uu options override each other; the last one specified deter-
##      mines the file time used.
## 
##      The --BB, --bb, --ww, and --qq options all override each other; the last one
##      specified determines the format used for non-printable characters.
## 
##      The --HH, --LL and --PP options all override each other (either partially or
##      fully); they are applied in the order specified.
## 
##      By default, llss lists one entry per line to standard output; the excep-
##      tions are to terminals or when the --CC or --xx options are specified.
## 
##      File information is displayed with one or more <blank>s separating the
##      information associated with the --ii, --ss, and --ll options.
## 
##    TThhee LLoonngg FFoorrmmaatt
##      If the --ll option is given, the following information is displayed for
##      each file: file mode, number of links, owner name, group name, number of
##      bytes in the file, abbreviated month, day-of-month file was last modi-
##      fied, hour file last modified, minute file last modified, and the path-
##      name.  In addition, for each directory whose contents are displayed, the
##      total number of 512-byte blocks used by the files in the directory is
##      displayed on a line by itself, immediately before the information for the
##      files in the directory.  If the file or directory has extended
##      attributes, the permissions field printed by the --ll option is followed by
##      a '@' character.  Otherwise, if the file or directory has extended secu-
##      rity information (such as an access control list), the permissions field
##      printed by the --ll option is followed by a '+' character.
## 
##      If the modification time of the file is more than 6 months in the past or
##      future, then the year of the last modification is displayed in place of
##      the hour and minute fields.
## 
##      If the owner or group names are not a known user or group name, or the --nn
##      option is given, the numeric ID's are displayed.
## 
##      If the file is a character special or block special file, the major and
##      minor device numbers for the file are displayed in the size field.  If
##      the file is a symbolic link, the pathname of the linked-to file is pre-
##      ceded by ``->''.
## 
##      The file mode printed under the --ll option consists of the entry type,
##      owner permissions, and group permissions.  The entry type character
##      describes the type of file, as follows:
## 
##            bb     Block special file.
##            cc     Character special file.
##            dd     Directory.
##            ll     Symbolic link.
##            ss     Socket link.
##            pp     FIFO.
##            --     Regular file.
## 
##      The next three fields are three characters each: owner permissions, group
##      permissions, and other permissions.  Each field has three character posi-
##      tions:
## 
##            1.   If rr, the file is readable; if --, it is not readable.
## 
##            2.   If ww, the file is writable; if --, it is not writable.
## 
##            3.   The first of the following that applies:
## 
##                       SS     If in the owner permissions, the file is not exe-
##                             cutable and set-user-ID mode is set.  If in the
##                             group permissions, the file is not executable and
##                             set-group-ID mode is set.
## 
##                       ss     If in the owner permissions, the file is exe-
##                             cutable and set-user-ID mode is set.  If in the
##                             group permissions, the file is executable and set-
##                             group-ID mode is set.
## 
##                       xx     The file is executable or the directory is search-
##                             able.
## 
##                       --     The file is neither readable, writable, exe-
##                             cutable, nor set-user-ID nor set-group-ID mode,
##                             nor sticky.  (See below.)
## 
##                 These next two apply only to the third character in the last
##                 group (other permissions).
## 
##                       TT     The sticky bit is set (mode 1000), but not execute
##                             or search permission.  (See chmod(1) or
##                             sticky(8).)
## 
##                       tt     The sticky bit is set (mode 1000), and is search-
##                             able or executable.  (See chmod(1) or sticky(8).)
## 
## EEXXAAMMPPLLEESS
##      The following is how to do an llss listing sorted by increasing size
## 
##            ls -lrS
## 
## DDIIAAGGNNOOSSTTIICCSS
##      The llss utility exits 0 on success, and >0 if an error occurs.
## 
## EENNVVIIRROONNMMEENNTT
##      The following environment variables affect the execution of llss:
## 
##      BLOCKSIZE       If the environment variable BLOCKSIZE is set, the block
##                      counts (see --ss) will be displayed in units of that size
##                      block.
## 
##      CLICOLOR        Use ANSI color sequences to distinguish file types.  See
##                      LSCOLORS below.  In addition to the file types mentioned
##                      in the --FF option some extra attributes (setuid bit set,
##                      etc.) are also displayed.  The colorization is dependent
##                      on a terminal type with the proper termcap(5) capabili-
##                      ties.  The default ``cons25'' console has the proper
##                      capabilities, but to display the colors in an xterm(1),
##                      for example, the TERM variable must be set to
##                      ``xterm-color''.  Other terminal types may require simi-
##                      lar adjustments.  Colorization is silently disabled if
##                      the output isn't directed to a terminal unless the
##                      CLICOLOR_FORCE variable is defined.
## 
##      CLICOLOR_FORCE  Color sequences are normally disabled if the output isn't
##                      directed to a terminal.  This can be overridden by set-
##                      ting this flag.  The TERM variable still needs to refer-
##                      ence a color capable terminal however otherwise it is not
##                      possible to determine which color sequences to use.
## 
##      COLUMNS         If this variable contains a string representing a decimal
##                      integer, it is used as the column position width for dis-
##                      playing multiple-text-column output.  The llss utility cal-
##                      culates how many pathname text columns to display based
##                      on the width provided.  (See --CC and --xx.)
## 
##      LANG            The locale to use when determining the order of day and
##                      month in the long --ll format output.  See environ(7) for
##                      more information.
## 
##      LSCOLORS        The value of this variable describes what color to use
##                      for which attribute when colors are enabled with
##                      CLICOLOR.  This string is a concatenation of pairs of the
##                      format _f_b, where _f is the foreground color and _b is the
##                      background color.
## 
##                      The color designators are as follows:
## 
##                            aa     black
##                            bb     red
##                            cc     green
##                            dd     brown
##                            ee     blue
##                            ff     magenta
##                            gg     cyan
##                            hh     light grey
##                            AA     bold black, usually shows up as dark grey
##                            BB     bold red
##                            CC     bold green
##                            DD     bold brown, usually shows up as yellow
##                            EE     bold blue
##                            FF     bold magenta
##                            GG     bold cyan
##                            HH     bold light grey; looks like bright white
##                            xx     default foreground or background
## 
##                      Note that the above are standard ANSI colors.  The actual
##                      display may differ depending on the color capabilities of
##                      the terminal in use.
## 
##                      The order of the attributes are as follows:
## 
##                            1.   directory
##                            2.   symbolic link
##                            3.   socket
##                            4.   pipe
##                            5.   executable
##                            6.   block special
##                            7.   character special
##                            8.   executable with setuid bit set
##                            9.   executable with setgid bit set
##                            10.  directory writable to others, with sticky bit
##                            11.  directory writable to others, without sticky
##                                 bit
## 
##                      The default is "exfxcxdxbxegedabagacad", i.e. blue fore-
##                      ground and default background for regular directories,
##                      black foreground and red background for setuid executa-
##                      bles, etc.
## 
##      LS_COLWIDTHS    If this variable is set, it is considered to be a colon-
##                      delimited list of minimum column widths.  Unreasonable
##                      and insufficient widths are ignored (thus zero signifies
##                      a dynamically sized column).  Not all columns have
##                      changeable widths.  The fields are, in order: inode,
##                      block count, number of links, user name, group name,
##                      flags, file size, file name.
## 
##      TERM            The CLICOLOR functionality depends on a terminal type
##                      with color capabilities.
## 
##      TZ              The timezone to use when displaying dates.  See
##                      environ(7) for more information.
## 
## CCOOMMPPAATTIIBBIILLIITTYY
##      The group field is now automatically included in the long listing for
##      files in order to be compatible with the IEEE Std 1003.2 (``POSIX.2'')
##      specification.
## 
## LLEEGGAACCYY DDEESSCCRRIIPPTTIIOONN
##      In legacy mode, the --ff option does not turn on the --aa option and the --gg,
##      --nn, and --oo options do not turn on the --ll option.
## 
##      Also, the --oo option causes the file flags to be included in a long (-l)
##      output; there is no --OO option.
## 
##      When --HH is specified (and not overridden by --LL or --PP) and a file argument
##      is a symlink that resolves to a non-directory file, the output will
##      reflect the nature of the link, rather than that of the file.  In legacy
##      operation, the output will describe the file.
## 
##      For more information about legacy mode, see compat(5).
## 
## SSEEEE AALLSSOO
##      chflags(1), chmod(1), sort(1), xterm(1), compat(5), termcap(5),
##      symlink(7), sticky(8)
## 
## SSTTAANNDDAARRDDSS
##      The llss utility conforms to IEEE Std 1003.1-2001 (``POSIX.1'').
## 
## HHIISSTTOORRYY
##      An llss command appeared in Version 1 AT&T UNIX.
## 
## BBUUGGSS
##      To maintain backward compatibility, the relationships between the many
##      options are quite complex.
## 
## BSD                              May 19, 2002                              BSD

Shell basics

Pressing h within a man page brings up the help page for how to navigate them

/terms_here lets you search within the man page for particular terms

Use n and shift+n to move forward and backward between matches

Navigation

We already learned how to list the files in a particular directory, but we need a few other tools to navigate around our machine

We often will want to know our current working directory so we know where we are before we start running commands

We do this with pwd

$ pwd

## /Users/ir229/Desktop/git/aem7130/spring-2020/lecture_notes/lecture_2

Navigation

Directories are organized in a hierarchical structure, at the top is the root directory, /

$ ls /

## Applications
## Library
## Network
## System
## Users
## Volumes
## anaconda3
## bin
## cores
## dev
## etc
## home
## installer.failurerequests
## net
## opt
## private
## sbin
## tmp
## usr
## var

Navigation

The root directory contains everything else

Other directories are inside the root directory and come afterward in the file path separated by forward slashes /

$ ls -lSh /Users

## total 0
## drwxr-xr-x+ 79 ir229          staff      2.5K Jan 26 12:52 ir229
## drwxr-xr-x+ 15 ag-wt94-doc10  11103514   480B Jan 10  2018 ag-wt94-doc10
## drwxr-xr-x+ 12 Guest          _guest     384B Nov 27  2017 Guest
## drwxr-xr-x  11 cals_oit       staff      352B Jan  9  2018 cals_oit
## drwxrwxrwt   7 root           wheel      224B Jun 28  2019 Shared
## drwxr-xr-x   5 ir229          admin      160B Dec 16  2018 ivan
## drwxr-xr-x   3 root           staff       96B Nov 17  2017 root

Navigation

The root directory contains everything else

Other directories are inside the root directory and come afterward in the file path separated by forward slashes /

$ ls -lSh /Users/ir229

## total 8
## drwx------+ 841 ir229  staff    26K Jan 29 11:59 Downloads
## drwx------+  71 ir229  staff   2.2K Sep 23 18:25 Library
## -rw-r--r--@   1 ir229  staff   1.0K Nov 25 13:39 artelys_lic_trial_artelys_knitro_academic_bb-8b-8e-28-3d.txt
## drwx------@  25 ir229  staff   800B Jan 27 14:00 Dropbox
## drwxr-xr-x   21 ir229  staff   672B Jan 17  2019 reveal.js
## drwx------+   9 ir229  staff   288B Jan 28 07:53 Documents
## drwx------+   7 ir229  staff   224B Jan 28 15:22 Desktop
## drwxr-xr-x    7 ir229  staff   224B Jan 26 12:49 HEG
## drwxr-xr-x    7 ir229  staff   224B Jan 26 12:50 wekafiles
## drwx------+   4 ir229  staff   128B Dec 21  2017 Music
## drwxr-xr-x+   4 ir229  staff   128B Nov 17  2017 Public
## drwx------@   3 ir229  staff    96B Nov 24  2017 Applications
## drwx------@   3 ir229  staff    96B Apr 10  2019 Google Drive
## drwx------+   3 ir229  staff    96B Nov 17  2017 Movies
## drwx------+   3 ir229  staff    96B Dec 21  2017 Pictures
## drwx------    2 ir229  staff    64B Jan 29  2018 Box Sync

Navigation

Next we need to be able to change directories, we can do this with cd (change directory)

$ cd /Users/ir229/Desktop/git
$ pwd

## /Users/ir229/Desktop/git

When navigating, it is often easier and more reproducible to use relative paths

This is when arguments are relative to your current working directory, instead of using absolute paths (e.g. /Users/ir229/Desktop/git)

Navigation

There's a few expressions that make this possible

• ~ is your home directory
• . is your current directory
• .. is the parent directory
• - is the previous directory you were in

Navigation

$ cd ~ # move to home directory (will vary computer-to-computer)
$ pwd
$ cd - # move to previous directory (lecture notes 2 directory)
$ cd .. # move to parent directory (general lecture notes directory)
$ pwd
$ cd . # move to current directory (nothing changes)
$ pwd

## /Users/ir229
## /Users/ir229/Desktop/git/aem7130/spring-2020/lecture_notes/lecture_2
## /Users/ir229/Desktop/git/aem7130/spring-2020/lecture_notes
## /Users/ir229/Desktop/git/aem7130/spring-2020/lecture_notes

You can see . and .. in your current directory when using ls with the a flag

$ ls -a

## .
## ..
## .DS_Store
## .Rhistory
## 2a_coding.Rmd
## 2a_coding.html
## 2a_coding_files
## figures
## my-css.css
## sandbox

Navigation

This makes navigation much easier

If we wanted to move from the current directory to the parent directory for this year's course we can just do

$ pwd
$ cd ../..
$ pwd

## /Users/ir229/Desktop/git/aem7130/spring-2020/lecture_notes/lecture_2
## /Users/ir229/Desktop/git/aem7130/spring-2020

instead of

$ cd /Users/ir229/Desktop/git/aem7130/spring-2020
$ pwd

## /Users/ir229/Desktop/git/aem7130/spring-2020

Navigation

Relative paths are very important for reproducible code

Your directory structure starting at root / will not be the same as someone else's

Using relative paths lets you circumvent this
as long as the project's directory structure is consistent

If you use Git or Dropbox it should be

Creating files and directories

We learned how to move around directories but how do we make them?

We do so with mkdir (make directory)

$ mkdir test_directory
$ ls

## 2a_coding.Rmd
## 2a_coding.html
## 2a_coding_files
## figures
## my-css.css
## sandbox
## test_directory

Creating files and directories

We create blank files using touch

$ touch test_directory/test.txt test_directory/test1.txt
$ ls test_directory

## test.txt
## test1.txt

touch is useful if you have a program that can't create a file itself but can edit them

Creating files and directories

If you have a Unix system pre-installed with nano you can use nano to create and edit the file

$ nano test_directory/test.txt

Creating files and directories

Here are some tips for naming files and directories

1. DON'T USE SPACES
   • Spaces are used to separate commands, you generally want to avoid them in names in favor of underscores or dashes
2. Use letters, numbers, underscores, periods, and dashes only

Creating files and directories

If you really, really, want to use spaces in names you'll have to do one of two things, enclose in quotes or backslash the space

$ mkdir "123test directory"
$ ls

## 123test directory
## 2a_coding.Rmd
## 2a_coding.html
## 2a_coding_files
## figures
## my-css.css
## sandbox
## test_directory

$ mkdir 123test\ directory\ 2
$ ls

## 123test directory
## 123test directory 2
## 2a_coding.Rmd
## 2a_coding.html
## 2a_coding_files
## figures
## my-css.css
## sandbox
## test_directory

## 2a_coding.Rmd
## 2a_coding.html
## 2a_coding_files
## figures
## my-css.css
## sandbox
## test_directory

Moving files and directories

We can move files and directories with mv (move)

$ mv test_directory/test.txt ..
$ ls ..

## figures
## lecture_1
## lecture_2
## test.txt

The first argument is the (relative) path of the file you want to move, the second argument is where you're moving it to

We moved the test.txt file from test_directory to the parent directory

Moving files and directories

$ mv ../test.txt test_directory
$ ls test_directory

## test.txt
## test1.txt

Here we moved it from the parent directory back to test_directory

Note that mv will overwrite any file with the move, use the -i option to make it ask you for confirmation

Moving files and directories

mv can also be used to rename files by just moving them to the same directory

$ mv test_directory/test.txt test_directory/test_new_name.txt
$ ls test_directory

## test1.txt
## test_new_name.txt

$ mv test_directory/test_new_name.txt test_directory/test.txt
$ ls test_directory

## test.txt
## test1.txt

Moving files and directories

Now that we've made the directory and file, how do we get rid of them? With rm

$ rm test_directory/test.txt
$ ls

## 2a_coding.Rmd
## 2a_coding.html
## 2a_coding_files
## figures
## my-css.css
## sandbox
## test_directory

Moving files and directories

We remove directories with rmdir

$ rmdir test_directory
$ ls

## rmdir: test_directory: Directory not empty
## 2a_coding.Rmd
## 2a_coding.html
## 2a_coding_files
## figures
## my-css.css
## sandbox
## test_directory

Notice that if a directory isn't empty you can't delete it

Moving files and directories

To delete a non-empty directory, you need to use rm on the directory, but apply the recursive option -r to delete everything inside of it first

$ rm -r test_directory
$ ls

## 2a_coding.Rmd
## 2a_coding.html
## 2a_coding_files
## figures
## my-css.css
## sandbox

Sometimes you might want to add the force option -f so it doesn't ask you if you want to delete each file

Copying files and directories

To copy files and directories just use cp, it works similarly to mv

$ mkdir test_directory
$ touch test_directory/test_copy.txt
$ cp test_directory/test_copy.txt .
$ ls test_directory

## test_copy.txt

$ ls

## 2a_coding.Rmd
## 2a_coding.html
## 2a_coding_files
## figures
## my-css.css
## sandbox
## test_copy.txt
## test_directory

Copying files and directories

You copy directories the same way, but if you want to copy the full file contents you need to apply the recursive option -r

$ cp -r test_directory ..
$ ls .. test_directory

## ..:
## figures
## lecture_1
## lecture_2
## test_directory
## 
## test_directory:
## test_copy.txt

$ ls ../test_directory

## test_copy.txt

Copying multiple files

How do we copy multiple files?

Let's make two directories for copying and a set of similar files

$ mkdir main_directory copy_directory
$ touch main_directory/file1.txt main_directory/file2.txt main_directory/file3.txt
$ ls main_directory

## file1.txt
## file2.txt
## file3.txt

Copying multiple files

To copy them we can just use cp as we did before

$ cp main_directory/file1.txt main_directory/file2.txt main_directory/file3.txt copy_directory
$ ls copy_directory

## file1.txt
## file2.txt
## file3.txt

We remove them the same way with rm

$ rm main_directory/file1.txt main_directory/file2.txt main_directory/file3.txt

Or we could use the mv rename trick into a new directory named copy_directory

Renaming multiple files

We can rename multiple files in an easier way using rename (brew install rename to install using Homebrew)

We can change all of our txt files to csvs, -s indicates that the first argument is to be the text we are changing, and the second argument is the text we are changing it to, the third argument is the location of the files we are renaming

$ ls copy_directory

## file1.txt
## file2.txt
## file3.txt

$ rename -s .txt .csv copy_directory/*
$ ls copy_directory

## file1.csv
## file2.csv
## file3.csv

Accessing multiple files

We can access multiple things at once using wildcards *, which replaces zero to any number of characters in the expression

$ touch copy_directory/test1.txt copy_directory/test2.txt copy_directory/test3.txt copy_directory/test123.txt
$ ls copy_directory/* # return everything in copy_directory

## copy_directory/file1.csv
## copy_directory/file2.csv
## copy_directory/file3.csv
## copy_directory/test1.txt
## copy_directory/test123.txt
## copy_directory/test2.txt
## copy_directory/test3.txt

$ ls copy_directory/*.csv # return everything in copy_directory with the csv extension

## copy_directory/file1.csv
## copy_directory/file2.csv
## copy_directory/file3.csv

Word count

The shell really shines when you try to combine multiple commands into one

Lets play around with the sandbox directory and count the number of words in animals.txt using wc

$ ls sandbox
$ wc sandbox/animals.txt

## animals.txt
## classes
## hey_jude.txt
## lucy_in_the_sky.txt
## trees.txt
##        0       7      33 sandbox/animals.txt

The first number is the number of lines, the second is the number of words, and the third is the number of characters

Word count

We can run this using the wildcard for all text files and also get the totals

$ wc sandbox/*.txt

##        0       7      33 sandbox/animals.txt
##       29     195     979 sandbox/hey_jude.txt
##       45     220    1191 sandbox/lucy_in_the_sky.txt
##        0       8      46 sandbox/trees.txt
##       74     430    2249 total

Redirecting

Now suppose we had 1 million files and wanted to find the one with the most words? Just printing to the screen doesn't work, we'd want to save the output and use it somewhere else, we can do that by redirecting with the greater than symbol >

$ wc -w sandbox/*.txt > sandbox/lengths.txt
$ ls sandbox

## animals.txt
## classes
## hey_jude.txt
## lengths.txt
## lucy_in_the_sky.txt
## trees.txt

Printing and cating

We can print the file to the screen using cat (print the full file) or less (one screenful)

$ cat sandbox/lengths.txt

##        7 sandbox/animals.txt
##      195 sandbox/hey_jude.txt
##      220 sandbox/lucy_in_the_sky.txt
##        8 sandbox/trees.txt
##      430 total

The w option made it so we only got the number of words, not characters or lines

Sorting

If we want to return the output sorted we can use sort

$ sort -n sandbox/lengths.txt

##        7 sandbox/animals.txt
##        8 sandbox/trees.txt
##      195 sandbox/hey_jude.txt
##      220 sandbox/lucy_in_the_sky.txt
##      430 total

where the n option means to sort numerically

Sorting

We can look at only the first few lines using head, (tail gets the last lines)

$ head -n 1 sandbox/lengths.txt

##        7 sandbox/animals.txt

Where the 1 means we only want the first line

Redirecting

> will always overwrite a file, we can use the double greater than symbol >> to append to a file

Lets use echo for an example which prints text

$ echo Hello world!

## Hello world!

$ echo \ walnut >> sandbox/trees.txt
$ cat sandbox/trees.txt

## maple pine birch oak beechnut palm fig redwood walnut

Piping

We've learned a few options for manipulating text files, we can combine them in easy ways using piping (same idea as Julia's queryverse and R's tidyverse)

Pipes | allow you sequentially write out commands that use the previous command's output as the next command's input

Suppose we wanted to find the file in a directory with the most number of characters, we could do this with

$ wc -m sandbox/* | sort -n | tail -n 2

## wc: sandbox/classes: read: Is a directory
##     1191 sandbox/lucy_in_the_sky.txt
##     2395 total

lucy_in_the_sky.txt is the longest in the sandbox directory

Piping

Look at the file sandbox/hey_jude.txt, how would we get the second verse?

We can pipe a head and tail together:

$ head -n 9 sandbox/hey_jude.txt | tail -n 4

## Hey jude, don't be afraid.
## You were made to go out and get her.
## The minute you let her under your skin,
## Then you begin to make it better.

head grabs the first two verses (with the empty line inbetween),
tail grabs second verse

Looping example 1

What if we wanted the second verse of multiple songs?

We can do that with a loop

$ for thing in list
$ do
$     operation_using $thing    # Indentation is good style
$ done

$ preprends any variables, here the variables are the things we're looping over

Looping example 1

$ for song in sandbox/hey_jude.txt sandbox/lucy_in_the_sky.txt
$ do
$     head -n 9 $song | tail -n 4
$ done

## Hey jude, don't be afraid.
## You were made to go out and get her.
## The minute you let her under your skin,
## Then you begin to make it better.
## Cellophane flowers of yellow and green
## Towering over your head
## Look for the girl with the sun in her eyes
## And she's gone

Looping example 2

How about a more realistic one that is real world useful (taken from Grant Mcdermott)

Let's combine a bunch of csvs using the shell

This is particularly useful with many or large datasets, because when done through shell, you do not need to load them into memory

The files are in /sandbox/classes and report a fake class schedule

Let's combine them into one

Looping example 2

First we need to make our class schedule file

$ touch sandbox/classes/class_schedule.csv

Then we need to add each day's schedule to the file

$ for day in $(ls sandbox/classes/*day.csv)
$ do
$     cat $day >> sandbox/classes/class_schedule.csv
$ done

where we treat what ls returns as a variable since its the output of a command

Looping example 2

$ cat sandbox/classes/class_schedule.csv

## day,morning,afternoon,evening
## friday,nothing,workshop,nothing
## day,morning,afternoon,evening
## monday,micro,macro,metrics
## day,morning,afternoon,evening
## thursday,game theory,seminar,nothings
## day,morning,afternoon,evening
## tuesday,game theory,seminar,nothings
## day,morning,afternoon,evening
## wednesday,micro,macro,metrics

Looks like it worked but we have the header every other line, how do we get rid of it?

Hint: we only need the header once, and then we want the last line of the csv for each file

Looping example 2

First lets remove the old file

$ rm -f sandbox/classes/class_schedule.csv

Next create the new file by grabbing the header from Monday

$ head -1 sandbox/classes/monday.csv > sandbox/classes/class_schedule.csv
$ cat sandbox/classes/class_schedule.csv

## day,morning,afternoon,evening

Looping example 2

So we've got the file started, now we need to fill in the days using our looping skills

We need to add each day's schedule to the file

$ for day in $(ls sandbox/classes/*day.csv)
$ do
$     tail -1 $day | cat >> sandbox/classes/class_schedule.csv
$ done
$ cat sandbox/classes/class_schedule.csv

## day,morning,afternoon,evening
## friday,nothing,workshop,nothing
## monday,micro,macro,metrics
## thursday,game theory,seminar,nothings
## tuesday,game theory,seminar,nothings
## wednesday,micro,macro,metrics

Finding things

How can we find things within files?

We use the command grep (global/regular expression/print)

grep finds and prints lines that match a certain pattern

For example, lets find the lines in Hey Jude that contain "make"

$ grep make sandbox/hey_jude.txt

## Hey jude, don't make it bad.
## Take a sad song and make it better.
## Then you can start to make it better.
## Then you begin to make it better.
## Then you can start to make it better.
## Hey jude, don't make it bad.
## Take a sad song and make it better.
## Then you'll begin to make it

Finding things

$ grep make sandbox/hey_jude.txt

Here make is the pattern we are searching for inside Hey Jude

Finding things

Now lets search Lucy in the Sky for "in"

$ grep in sandbox/lucy_in_the_sky.txt | head -5

## Picture yourself in a boat on a river
## With tangerine trees and marmalade skies
## Towering over your head
## Look for the girl with the sun in her eyes
## Lucy in the sky with diamonds

This gave us words that contained "in" but weren't actually the word "in"

Finding things

We can restrict the search to words with the w option

$ grep -w in sandbox/lucy_in_the_sky.txt | head -5

## Picture yourself in a boat on a river
## Look for the girl with the sun in her eyes
## Lucy in the sky with diamonds
## Lucy in the sky with diamonds
## Lucy in the sky with diamonds

Grepping

greps real power comes from using regular expressions

These are complex expressions that allow you to search for very specific things

For example, lets find lines with "a" as the second letter

$ grep -E "^.a" sandbox/lucy_in_the_sky.txt | head -5

## Waiting to take you away

grep shows up in most programming languages as well

You can imagine using it to do things like dynamically renaming a set of variables, dealing with weirdly reported FIPS codes, etc

Shell scripts

A nice thing about shell that's pretty underused by economists is putting the commands into scripts so we can re-use them

$ # writing a shell script using echo is kind of silly
$ # but I want to show you what I'm doing on the slides
$ touch sandbox/shell_script.sh
$ echo echo "Hello World!" >> sandbox/shell_script.sh
$ cat sandbox/shell_script.sh

## echo Hello World!

We can the execute it using bash

$ bash sandbox/shell_script.sh

## Hello World!

Why am I doing this to you?

Why learn Julia?

1. It's a high-level language, much easier to use than C++, Fortran, etc
2. It delivers C++ and Fortran speed

Intro to programming

Programming $\equiv$ writing a set of instructions

1. There are hard and fast rules you can't break if you want it to work
2. There are elements of style (e.g. Strunk and White) that make for clearer and more efficient code

If you will be doing computational work there are:

1. Language-independent coding basics you should know
   • Arrays are stored in memory in particular ways
2. Language-independent best practices you should use
   • Indent to convey program structure (or function in Python)
3. Language-dependent idiosyncracies that matter for function, speed, etc
   • Julia: type stability; R: vectorize

Intro to programming

Learning these early will:

1. Make coding a lot easier
2. Reduce total programmer time
3. Reduce total computer time
4. Make your code understandable by someone else or your future self
5. Make your code flexible

A broad view of programming

Your goal is to make a program

A program is made of different components and sub-components

The most basic component is a statement, more commonly called a line of code

A broad view of programming

Here's pseudoprogram:

deck = ["4 of hearts", "King of clubs", "Ace of spades"]
shuffled_deck = shuffle(deck)
first_card = shuffled_deck[1]
println("The first drawn card was " * shuffled_deck ".")

This program is real simple:

1. Create a deck of cards

A broad view of programming

Here's pseudoprogram:

deck = ["4 of hearts", "King of clubs", "Ace of spades"]
shuffled_deck = shuffle(deck)
first_card = shuffled_deck[1]
println("The first drawn card was " * shuffled_deck ".")

This program is real simple:

1. Create a deck of cards
2. Shuffle the deck

A broad view of programming

Here's pseudoprogram:

deck = ["4 of hearts", "King of clubs", "Ace of spades"]
shuffled_deck = shuffle(deck)
first_card = shuffled_deck[1]
println("The first drawn card was " * shuffled_deck ".")

This program is real simple:

1. Create a deck of cards
2. Shuffle the deck
3. Draw the top card

A broad view of programming

Here's pseudoprogram:

deck = ["4 of hearts", "King of clubs", "Ace of spades"]
shuffled_deck = shuffle(deck)
first_card = shuffled_deck[1]
println("The first drawn card was " * shuffled_deck ".")

This program is real simple:

1. Create a deck of cards
2. Shuffle the deck
3. Draw the top card
4. Print it

A broad view of programming

deck = ["4 of hearts", "King of clubs", "Ace of spades"]
shuffled_deck = shuffle(deck)
first_card = shuffled_deck[1]
println("The first drawn card was " * shuffled_deck ".")

What are the parentheses and why are they different from square brackets?

How does shuffle work?

What’s println?

It’s important to know that a good program has understandable code

Julia specifics

We will discuss coding in the context of Julia
but a lot of this ports to Python, MATLAB, etc

To do:

1. Types
2. Operators
3. Scope
4. Generic functions
5. Multiple dispatch

Types

All languages have some kind of data types like integers or arrays

The first type you will often use is a boolean (Bool) variable that takes on a value of true or false:

 x = true

## true

 typeof(x)

## Bool

Types

We can save the boolean value of actual statements in variables this way:

 @show y = 1 > 2

## y = 1 > 2 = false

## false

@show is a Julia macro for showing the operation

Numbers

Two other data types you will use frequently are integers

 typeof(1)

## Int64

and floating point numbers

 typeof(1.0)

## Float64

Recall from lecture 1 the 64 means 64 bits of storage for the number, which is probably the default on your machine

Numbers

You can always instantiate alternative floating point number types

 converted_int = convert(Float32, 1.0);
 typeof(converted_int)

## Float32

Numbers

Math works like you would expect:

 a = 2; b = 1.0;
 a * b

## 2.0

 a^2

## 4

Numbers

 2a - 4b

## 0.0

 @show 4a + 3b^2

## 4a + 3 * b ^ 2 = 11.0

## 11.0

You dont need * inbetween numeric literals (numbers) and variables

Strings

Strings store sequences of characters

You implement them with double quotations:

 x = "Hello World!";
 typeof(x)

## String

Note that ; suppresses output for that line of code but is unnecessary in Julia

Strings

It's easy to work with strings, use $ to interpolate a variable/expression

 x = 10; y = 20; println("x + y =  $(x+y).")

## x + y =  30.

Use * to concatenate strings

 a = "Aww"; b = "Yeah!!!"; println(a * " " * b)

## Aww Yeah!!!

You probably won't use strings too often unless you're working with text data or printing output

Containers

Containers are types that store collections of data

The most basic container is the Array which is denoted by square brackets

 a1 = [1 2; 3 4]; typeof(a1)

## Array{Int64,2}

Arrays are mutable which means you can change their values

 a1[1,1] = 5; a1

## 2×2 Array{Int64,2}:
##  5  2
##  3  4

You reference elements in a container with square brackets

Containers

An alternative to the Array is the Tuple which is denoted by parentheses

 a2 = (1, 2, 3, 4); typeof(a2)

## NTuple{4,Int64}

a2 is a Tuple of 4 Int64s, tuples have no dimension

Tuples are immutable which means you can't change their values

 try
   a2[1,1] = 5;
 catch
   println("Error, can't change value of a tuple.")
 end

## Error, can't change value of a tuple.

Containers

Tuples don't need parentheses (but it's probably best practice for clarity)

 a3 = 5, 6; typeof(a3)

## Tuple{Int64,Int64}

Containers

Tuples can be unpacked (see NamedTuple for an alternative and more efficient container)

 a3_x, a3_y = a3;
 a3_x

## 5

 a3_y

## 6

This is basically how functions return output when you call them

Containers

A Dictionary is the last main container type,
they are arrays but are indexed by keys (names) instead of numbers

 d1 = Dict("class" => "AEM7130", "grade" => 97);
 typeof(d1)

## Dict{String,Any}

d1 is a dictionary where the key are strings and the values are any kind of type

Containers

Reference specific values you want in the dictionary by referencing the key

 d1["class"]

## "AEM7130"

 d1["grade"]

## 97

Containers

If you just want all the keys or all the values you can use the base functions

 keys_d1 = keys(d1)

## Base.KeySet for a Dict{String,Any} with 2 entries. Keys:
##   "class"
##   "grade"

 values_d1 = values(d1)

## Base.ValueIterator for a Dict{String,Any} with 2 entries. Values:
##   "AEM7130"
##   97

Iterating

As in other languages we have loops at our disposal:

for loops iterate over containers

 for count in 1:10
   random_number = rand()
   if random_number > 0.2
     println("We drew a $random_number.")
   end
 end

## We drew a 0.7600855101338988.
## We drew a 0.49274639430432443.
## We drew a 0.8921567760692246.
## We drew a 0.27352582996706154.
## We drew a 0.790319051812356.
## We drew a 0.2353900368896369.
## We drew a 0.8048663968302878.
## We drew a 0.7067448362153019.

Iterating

while loops iterate until a logical expression is false

 while rand() > 0.5
   random_number = rand()
   if random_number > 0.2
     println("We drew a $random_number.")
   end
 end

## We drew a 0.5576232957214851.

Iterating

An Iterable is something you can loop over, like arrays

 actions = ["codes well", "skips class"];
 for action in actions
     println("Charlie $action")
 end

## Charlie codes well
## Charlie skips class

Iterating

There's a type that's a subset of iterables, Iterator, that are particularly convenient

These include things like the dictionary keys:

 for key in keys(d1)
   println(d1[key])
 end

## AEM7130
## 97

Iterating

Iterating on Iterators is more memory efficient than iterating on arrays

Here's a very simple example, the top function iterates on an Array, the bottom function iterates on an Iterator:

 function show_array_speed()
   m = 1
   for i = [1, 2, 3, 4, 5, 6]
     m = m*i
   end
 end;
 function show_iterator_speed()
   m = 1
   for i = 1:6
     m = m*i
   end
 end;

Iterating

 using BenchmarkTools
 @btime show_array_speed()

##   26.849 ns (1 allocation: 128 bytes)

 @btime show_iterator_speed()

##   1.279 ns (0 allocations: 0 bytes)

The Iterator approach is faster and allocates no memory

@btime is a macro from BenchmarkTools that shows you the elasped time and memory allocation

Neat looping

The nice thing about Julia vs MATLAB is your loops can be much neater since you don't need to index if you just want the container elements

 f(x) = x^2;
 x_values = 0:20:100;
 for x in x_values
   println(f(x))
 end

## 0
## 400
## 1600
## 3600
## 6400
## 10000

Neat looping

The loop directly assigns the elements of x_values to x instead of having to do something clumsy like x_values[i]

0:20:100 creates something called a StepRange (a type of Iterator) which starts at 0, steps up by 20 and ends at 100

Neat looping

You can also pull out an index and the element value by enumerating

 f(x) = x^2;
 x_values = 0:20:100;
 for (index, x) in enumerate(x_values)
   println("f(x) at value $index is $(f(x)).")
 end

## f(x) at value 1 is 0.
## f(x) at value 2 is 400.
## f(x) at value 3 is 1600.
## f(x) at value 4 is 3600.
## f(x) at value 5 is 6400.
## f(x) at value 6 is 10000.

enumerate basically assigns an index vector

Neat looping

There is also a lot of Python-esque functionality

For example: zip lets you loop over multiple different iterables at once

 last_name = ("Lincoln", "Bond", "Walras");
 first_name = ("Abraham", "James", "Leon");
 for (first_idx, last_idx) in zip(first_name, last_name)
   println("The name's $last_idx, $first_idx $last_idx.")
 end

## The name's Lincoln, Abraham Lincoln.
## The name's Bond, James Bond.
## The name's Walras, Leon Walras.

Neat looping

Nested loops can also be made very neatly

 for x in 1:3, y in 3:-1:1
   println(y-x)
 end

## 2
## 1
## 0
## 1
## 0
## -1
## 0
## -1
## -2

The first loop is the inner loop, the second loop is the outer loop

Comprehensions: the neatest looping

Comprehensions are super nice ways to use iterables that make your code cleaner and more compact

 squared = [y^2 for y in 1:2:11]

## 6-element Array{Int64,1}:
##    1
##    9
##   25
##   49
##   81
##  121

This created a 1-dimension Array using one line

Comprehensions: the neatest looping

We can also use nested loops for comprehensions

 squared_2 = [(y+z)^2 for y in 1:2:11, z in 1:6]

## 6×6 Array{Int64,2}:
##    4    9   16   25   36   49
##   16   25   36   49   64   81
##   36   49   64   81  100  121
##   64   81  100  121  144  169
##  100  121  144  169  196  225
##  144  169  196  225  256  289

This created a 2-dimensional Array

Use this (and the compact nested loop) sparingly since it's hard to follow

Dot syntax: broadcasting/vectorization

Vectorizing operations (e.g. applying it to a whole array or vector at once) is easy in Julia, just use dot syntax like you would in MATLAB, etc

 g(x) = x^2;
 squared_2 = g.(1:2:11)

## 6-element Array{Int64,1}:
##    1
##    9
##   25
##   49
##   81
##  121

This is actually called broadcasting

When broadcasting, you might want to consider pre-allocating arrays

Dot syntax: broadcasting/vectorization

Vectorization creates temporary allocations, temporary arrays in the middle of the process that aren't actually needed for the final product

Julia can do broadcasting in a nicer, faster way by fusing operations together and avoiding these temporary allocations

Dot syntax: broadcasting/vectorization

Let's write two functions that do the same thing:

 function show_vec_speed(x) 
   out = [3x.^2 + 4x + 7x.^3 for i = 1:1]
 end

## show_vec_speed (generic function with 1 method)

 function show_fuse_speed(x) 
   out = @. [3x.^2 + 4x + 7x.^3 for i = 1:1]
 end

## show_fuse_speed (generic function with 1 method)

The top one is vectorized for the operations, the @. in the bottom one vectorizes everything in one swoop: the function call, the operation, and the assignment to a variable

Dot syntax: broadcasting/vectorization

First, precompile the functions

 x = rand(10^6);
 @time show_vec_speed(x);
 @time show_fuse_speed(x);

 @time show_vec_speed(x)

##   0.019510 seconds (20 allocations: 45.777 MiB, 23.83% gc time)

## 1-element Array{Array{Float64,1},1}:
##  [12.861723629700903, 1.9288570130193439, 4.618121381082656, 2.1092997815476298, 0.5474929676796775, 0.2581580004559404, 1.7316931735226495, 0.865498488679512, 7.489015929309849, 1.3240002299519937  …  8.87972822880598, 11.058919443149918, 5.79373631326248, 7.363341963385606, 0.026949311578706444, 3.358930924502511, 1.2591943206571903, 12.868332924140653, 2.3547706579519305, 0.1613862421281777]

 @time show_fuse_speed(x)

##   0.001814 seconds (9 allocations: 7.630 MiB)

## 1-element Array{Array{Float64,1},1}:
##  [12.861723629700903, 1.9288570130193439, 4.618121381082656, 2.1092997815476298, 0.5474929676796775, 0.2581580004559404, 1.7316931735226495, 0.865498488679512, 7.489015929309849, 1.3240002299519937  …  8.87972822880598, 11.058919443149918, 5.79373631326248, 7.363341963385606, 0.026949311578706444, 3.358930924502511, 1.2591943206571903, 12.868332924140653, 2.3547706579519305, 0.1613862421281777]

Full vectorization using @. is 10x faster with 1/6 of the memory allocation

Dot syntax: vectorization

Not pre-allocated:

 h(y,z) = y^2 + sin(z);   # function to evaluate
 y = 1:2:1e6+1;           # input y
 z = rand(length(y));     # input z

Dot syntax

Here we are vectorizing the function call

 # precompile h so first timer isn't picking up on compile time
 h(1,2)

 @time out_1 = h.(y,z)    # evaluate h.(y,z) and time

##   0.041467 seconds (147.64 k allocations: 10.681 MiB, 13.87% gc time)

## 500001-element Array{Float64,1}:
##    1.5282379529505645  
##    9.346421517864995   
##   25.735112300725387   
##   49.719218820749624   
##   81.6045065941119     
##  121.44590854348961    
##  169.4543440659069     
##  225.51640067832085    
##  289.5748078394144     
##  361.3057619181106     
##    ⋮                   
##    9.999700002256343e11
##    9.999740001690128e11
##    9.999780001214305e11
##    9.999820000811464e11
##    9.999860000495673e11
##    9.999900000257694e11
##    9.999940000093579e11
##    9.999980000016255e11
##    1.000002000001051e12

Dot syntax: vectorization

Here we are vectorizing the function call and assignment

 out_2 = similar(out_1)

 @time out_2 .= h.(y,z)

##   0.024322 seconds (21.73 k allocations: 995.816 KiB)

## 500001-element Array{Float64,1}:
##    1.5282379529505645  
##    9.346421517864995   
##   25.735112300725387   
##   49.719218820749624   
##   81.6045065941119     
##  121.44590854348961    
##  169.4543440659069     
##  225.51640067832085    
##  289.5748078394144     
##  361.3057619181106     
##    ⋮                   
##    9.999700002256343e11
##    9.999740001690128e11
##    9.999780001214305e11
##    9.999820000811464e11
##    9.999860000495673e11
##    9.999900000257694e11
##    9.999940000093579e11
##    9.999980000016255e11
##    1.000002000001051e12

Dot syntax: vectorization

Here we are vectorizing the function call, assignment, and operations

 out_3 = similar(out_1)

 @time @. out_3 = h(y,z)

##   0.006950 seconds (6 allocations: 240 bytes)

## 500001-element Array{Float64,1}:
##    1.5282379529505645  
##    9.346421517864995   
##   25.735112300725387   
##   49.719218820749624   
##   81.6045065941119     
##  121.44590854348961    
##  169.4543440659069     
##  225.51640067832085    
##  289.5748078394144     
##  361.3057619181106     
##    ⋮                   
##    9.999700002256343e11
##    9.999740001690128e11
##    9.999780001214305e11
##    9.999820000811464e11
##    9.999860000495673e11
##    9.999900000257694e11
##    9.999940000093579e11
##    9.999980000016255e11
##    1.000002000001051e12

Logical operators work like you'd think

• == (equal equal) tests for equality

 1 == 1

## true

• != (exclaimation point equal) tests for inequality

 2 != 2

## false

• You can also test for approximate equality with $\approx$ (type \approx<TAB>)

 1.00000001 ≈ 1

## true

Scope

The scope of a variable name determines when it is valid to refer to it

Scope can be a frustrating concept if you haven't used a similarly scoped language before

If you want to dive into the details: the type of scoping in Julia is called lexical scoping

Different scopes can have the same name, i.e. saving_rate, but be assigned to different variables

Let's walk through some simple examples to see how it works

Scope

First, functions have their own local scope

 ff(xx) = xx^2;
 yy = 5;
 ff(yy)

## 25

xx isn't bound to any values outside the function ff

This is pretty natural for those of you who have done any programming before

Scope

Locally scoped functions allow us to do things like:

 xx = 10;
 fff(xx) = xx^2;
 fff(5)

## 25

Although xx was declared equal to 10, the function still evaluated at 5

This is all kind of obvious so far

Scope

But, this type of scoping also has (initially) counterintuitive results like:

 zz = 0;
 for ii = 1:10
   zz = ii
 end
 println("zz = $zz")

## zz = 0

Scope

What happened?

The zz outside the for loop has a different scope,
the global scope, than the zz inside it

The global scope is the outer most scope, outside all functions and loops

The zz inside the for loop has a scope local to the loop

Since the outside zz has global scope the locally scoped variables in the loop can't change it

Scope

Generally you want to avoid global scope because it can cause conflicts, slowness, etc, but you can use global to force it if you want something to have global scope

 zz = 0;
 for ii = 1:10
   global zz
   zz = ii
 end
 println("zz = $zz")

## zz = 10

Scope

Local scope kicks in whenever you have a new block keyword (i.e. you indented something) except for if

Global variables inside a local scope are inherted for reading, not writing

 x, y = 1, 2;
 function foo()
   x = 2        # assignment introduces a new local
   return x + y # y refers to the global
 end;
 foo()

## 4

 x

## 1

Scope

Important piece: nested functions can modify their parent scope's local variables

 x, y = 1, 2; # set globals
 function f_outer()
   x = 2                # introduces a new local
   function f_inner()
     x = 10             # modifies the parent's x
     return x + y       # y is global
   end
   return f_inner() + x # 12 + 10 (x is modified in call of f_inner())
 end;
 f_outer()
 x, y                   # verify that global x and y are unchanged

Scope

 function f_outer()
   x = 2                # introduces a new local
   function f_inner()
     x = 10             # modifies the parent's x
     return x + y       # y is global
   end
   return f_inner() + x # 12 + 10 (x is modified in call of f_inner())
 end;
 f_outer()

## 22

 x, y                   # verify that global x and y are unchanged

## (1, 2)

If f_inner was not nested and was in the global scope we'd get 14 not 22, this is also a way to handle the issue with loops editing variables not created in their local scope

Scope

We can fix looping issues with global scope by using a wrapper function that doesn't do anything but change the parent scope so it is not global

 function wrapper()
   zzz = 0;
   for iii = 1:10
     zzz = iii
   end
   println("zzz = $zzz")
 end

## wrapper (generic function with 1 method)

 wrapper()

## zzz = 10

Closures

These inner functions we've been looking at are called closures

When a function f is parsed in Julia, it looks to see if any of the variables have been previously defined in the current scope

 a = 0.2;
 f(x) = a * x^2;    # refers to the `a` in the outer scope

## f (generic function with 1 method)

 f(1)               # univariate function

## 0.2

Closures

 function g(a)
     f(x) = a * x^2; # refers to the `a` passed in the function
     f(1);           # univariate function
 end

## g (generic function with 2 methods)

 g(0.2)

## 0.2

In both of these examples f is a closure designed to capture a variable from an outer scope

Closures

Here's a complicated example that actually returns a closure (a function!) itself:

 x = 0;
 function toplevel(y)
   println("x = ", x, " is a global variable")
   println("y = ", y, " is a parameter")
   z = 2
   println("z = ", z, " is a local variable")
   function closure(v)
     println("v = ", v, " is a parameter")
     w = 3
     println("w = ", w, " is a local variable")
     println("x = ", x, " is a global variable")
     println("y = ", y, " is a closed variable (a parameter of the outer function)")
     println("z = ", z, " is a closed variable (a local of the outer function)")
   end;
   return closure
 end;

What will be returned when we call these functions?

Closures

Here's a complicated example:

 c_func = toplevel(10)

## x = 0 is a global variable
## y = 10 is a parameter
## z = 2 is a local variable

## (::getfield(Main, Symbol("#closure#2372")){Int64,Int64}) (generic function with 1 method)

 c_func(20)

## v = 20 is a parameter
## w = 3 is a local variable
## x = 0 is a global variable
## y = 10 is a closed variable (a parameter of the outer function)
## z = 2 is a closed variable (a local of the outer function)

The returned closure still has access to the outer function's local scope!

Generic functions

If you use Julia to write code for research you should aim to write generic functions

These are functions that are flexible (e.g. can deal with someone using an Int instead of a Float) and have high performance (e.g. comparable speed to C)

Functions are made generic by paying attention to types and making sure types are stable

Type stability: Given an input into a function, operations on that input should maintain the type so Julia knows what its type will be throughout the full function call

This allows it to compile type-specialized versions of the functions, which will yield higher performance

Generic functions

The question you might have is: Type stability sounds like mandating types (e.g. what C and Fortran do, not what R/Python/etc do), so how do we make it flexible?

We'll see next

These two functions look the same, but are they?

 function t1(n)
   s = 0
   t = 1
   for i in 1:n
      s += s/i
      t = div(t, i)
   end
   return t
 end

## t1 (generic function with 1 method)

 function t2(n)
   s  = 0.0
   t = 1
   for i in 1:n
      s += s/i
      t = div(t, i)
   end
   return t
 end

## t2 (generic function with 1 method)

No! t1 is not type stable

t1 starts with s as an Int64 but then we have s += s/i which will mean it must hold a Float64

It must be converted to Float so it is not type stable

No! t1 is not type stable

We can see this when calling the macro @code_warntype where it reports t1 at some point handles s that has type Union{Float64,Int64}, either Float64 or Int64

Julia now can't assume s's type and produce pure integer or floating point code $\rightarrow$ performance degradation

THIS MATTERS

Type stability matters for speed, here there's about a 100% difference

 @btime t1(5)

##   21.519 ns (0 allocations: 0 bytes)

## 0

 @btime t2(5)

##   13.985 ns (0 allocations: 0 bytes)

## 0

THIS MATTERS

Here's an order of magnitude difference for a similar function

 # Type instable
 function g()
   x=1
   for i = 1:10
     x = x/2
   end
   return x
 end
 # Type stable
 function h()
   x=1.0
   for i = 1:10
     x = x/2
   end
   return x
 end

THIS MATTERS

Here's an order of magnitude difference for a similar function

 @btime g()

##   10.412 ns (0 allocations: 0 bytes)

## 0.0009765625

 @btime h()

##   1.312 ns (0 allocations: 0 bytes)

## 0.0009765625

Concrete vs abstract types

A concrete type is one that can be instantiated (Float64 Bool Int32)

An abstract type cannot (Real, Number, Any)

Abstract types are for organizing the types

You can check where types are in the hierarchy

 @show Float64 <: Real

## Float64 <: Real = true

## true

 @show Array <: Real

## Array <: Real = false

## false

 @show Number <: Any

## Number <: Any = true

## true

Concrete vs abstract types

You can see the type hierarchy with the supertypes and subtypes commands

 using Base: show_supertypes
 show_supertypes(Float64)

## Float64 <: AbstractFloat <: Real <: Number <: Any

Creating new types

We can actually create new composite types using struct

 struct FoobarNoType # This will be immutable by default
   a
   b
   c
 end

Creating new types

This creates a new type called FoobarNoType, and we can generate a variable of this type using its constructor which will have the same name

 newfoo = FoobarNoType(1.3, 2, "plzzz");
 typeof(newfoo)

## FoobarNoType

 newfoo.a

## 1.3

You should always declare types for the fields of a new composite type

Creating new types

You can declare types with the double colon

 struct FoobarType # This will be immutable by default
   a::Float64
   b::Int
   c::String
 end

Creating new types

 newfoo_typed = FoobarType(1.3, 2, "plzzz");
 typeof(newfoo_typed)

## FoobarType

 newfoo.a

## 1.3

This lets the compiler generate efficient code because it knows the types of the fields when you construct a FoobarType

Declaring abstract types isn't good enough, you need to declare concrete types

Or do we?

Parametric types are what help deliver flexibility

We can create types that hold different types of fields
by declaring subsets of abstract types

 struct FooParam{t1 <: Real, t2 <: Real, t3 <: AbstractArray{<:Real}}
   a::t1
   b::t2
   c::t3
 end
 newfoo_para = FooParam(1.0, 7, [1., 4., 6.])

## FooParam{Float64,Int64,Array{Float64,1}}(1.0, 7, [1.0, 4.0, 6.0])

The curly brackets declare all the different type subsets we will use in FooParam

This actually delivers high performance code!

Delivering flexibility

We want to make sure types are stable but code is flexible

Ex: if want to preallocate an array to store data,
how do we know how to declare it's type?

We don't need to

Delivering flexibility

 using LinearAlgebra               # necessary for I
 function sametypes(x)
   y = similar(x)                  # creates an array that is `similar` to x, use this for preallocating
   z = I                           # creates a scalable identity matrix
   q = ones(eltype(x), length(x))  # one is a type generic array of ones, fill creates the array of length(x)
   y .= z * x + q
   return y
 end

## sametypes (generic function with 1 method)

 x = [5.5, 7.0, 3.1];
 y = [7, 8, 9];

Delivering flexibility

We did not declare any types but the function is type stable

 sametypes(x)

## 3-element Array{Float64,1}:
##  6.5
##  8.0
##  4.1

 sametypes(y)

## 3-element Array{Int64,1}:
##   8
##   9
##  10

There's a lot of other functions out there that help with writing flexible, type stable code

Multiple dispatch

Why type stability really matters: multiple dispatch

Neat thing about Julia: the same function name can perform different operations depending on the underlying type of the inputs

A function specifies different methods, each of which operates on a specific set of types

When you write a function that's type stable, you are actually writing many different methods, each of which are optimized for certain types

If your function isn't type stable, the optimized method may not be used

This is why Julia can achieve C speeds: it compiles to C (or faster) code

Multiple dispatch

/ has 103 different methods depending on the input types, these are 103 specialized sets of codes

 methods(/)

## # 103 methods for generic function "/":
## [1] /(a::Float16, b::Float16) in Base at float.jl:392
## [2] /(x::Float32, y::Float32) in Base at float.jl:400
## [3] /(x::Float64, y::Float64) in Base at float.jl:401
## [4] /(z::Complex{Float64}, w::Complex{Float64}) in Base at complex.jl:361
## [5] /(::Missing, ::Missing) in Base at missing.jl:93
## [6] /(::Missing, ::Number) in Base at missing.jl:94
## [7] /(x::BigInt, y::BigInt) in Base.GMP at gmp.jl:428
## [8] /(x::BigInt, y::Union{Int16, Int32, Int64, Int8, UInt16, UInt32, UInt64, UInt8}) in Base.GMP at gmp.jl:477
## [9] /(x::BigFloat, c::BigInt) in Base.MPFR at mpfr.jl:464
## [10] /(x::BigFloat, y::BigFloat) in Base.MPFR at mpfr.jl:421
## [11] /(x::BigFloat, c::Union{UInt16, UInt32, UInt64, UInt8}) in Base.MPFR at mpfr.jl:428
## [12] /(x::BigFloat, c::Union{Int16, Int32, Int64, Int8}) in Base.MPFR at mpfr.jl:440
## [13] /(x::BigFloat, c::Union{Float16, Float32, Float64}) in Base.MPFR at mpfr.jl:452
## [14] /(x::Union{Int128, Int16, Int32, Int64, Int8, UInt128, UInt16, UInt32, UInt64, UInt8}, y::Union{Int128, Int16, Int32, Int64, Int8, UInt128, UInt16, UInt32, UInt64, UInt8}) in Base at int.jl:59
## [15] /(x::Union{Int16, Int32, Int64, Int8, UInt16, UInt32, UInt64, UInt8}, y::BigInt) in Base.GMP at gmp.jl:478
## [16] /(x::T, y::T) where T<:Integer in Base at int.jl:57
## [17] /(x::Rational, y::Complex{#s75} where #s75<:Union{Integer, Rational}) in Base at rational.jl:266
## [18] /(a::R, z::S) where {R<:Real, S<:Complex} in Base at complex.jl:324
## [19] /(z::Complex, x::Real) in Base at complex.jl:325
## [20] /(z::Complex{T}, w::Complex{T}) where T<:Union{Float16, Float32} in Base at complex.jl:351
## [21] /(a::Complex{T}, b::Complex{T}) where T<:Real in Base at complex.jl:328
## [22] /(x::Rational, y::Rational) in Base at rational.jl:265
## [23] /(c::Union{UInt16, UInt32, UInt64, UInt8}, x::BigFloat) in Base.MPFR at mpfr.jl:433
## [24] /(c::Union{Int16, Int32, Int64, Int8}, x::BigFloat) in Base.MPFR at mpfr.jl:445
## [25] /(c::Union{Float16, Float32, Float64}, x::BigFloat) in Base.MPFR at mpfr.jl:457
## [26] /(x::AbstractIrrational, y::AbstractIrrational) in Base at irrationals.jl:137
## [27] /(x::T, y::T) where T<:Number in Base at promotion.jl:392
## [28] /(x::Number, y::Number) in Base at promotion.jl:316
## [29] /(x::Base.TwicePrecision, v::Number) in Base at twiceprecision.jl:301
## [30] /(x::Base.TwicePrecision, y::Base.TwicePrecision) in Base at twiceprecision.jl:305
## [31] /(r::StepRangeLen{#s75,#s74,S} where S where #s74<:Base.TwicePrecision where #s75<:Real, x::Real) in Base at twiceprecision.jl:482
## [32] /(B::BitArray, x::Number) in Base at bitarray.jl:1103
## [33] /(X::Union{DenseArray{P,N}, Base.ReinterpretArray{P,N,S,A} where S where A<:Union{SubArray{T,N,A,I,true} where I<:Union{Tuple{Vararg{Real,N} where N}, Tuple{AbstractUnitRange,Vararg{Any,N} where N}} where A<:DenseArray where N where T, DenseArray}, Base.ReshapedArray{P,N,A,MI} where MI<:Tuple{Vararg{Base.MultiplicativeInverses.SignedMultiplicativeInverse{Int64},N} where N} where A<:Union{Base.ReinterpretArray{T,N,S,A} where S where A<:Union{SubArray{T,N,A,I,true} where I<:Union{Tuple{Vararg{Real,N} where N}, Tuple{AbstractUnitRange,Vararg{Any,N} where N}} where A<:DenseArray where N where T, DenseArray} where N where T, SubArray{T,N,A,I,true} where I<:Union{Tuple{Vararg{Real,N} where N}, Tuple{AbstractUnitRange,Vararg{Any,N} where N}} where A<:DenseArray where N where T, DenseArray}, SubArray{P,N,A,I,L} where L where I<:Tuple{Vararg{Union{Int64, AbstractRange{Int64}, Base.AbstractCartesianIndex},N} where N} where A<:Union{Base.ReinterpretArray{T,N,S,A} where S where A<:Union{SubArray{T,N,A,I,true} where I<:Union{Tuple{Vararg{Real,N} where N}, Tuple{AbstractUnitRange,Vararg{Any,N} where N}} where A<:DenseArray where N where T, DenseArray} where N where T, Base.ReshapedArray{T,N,A,MI} where MI<:Tuple{Vararg{Base.MultiplicativeInverses.SignedMultiplicativeInverse{Int64},N} where N} where A<:Union{Base.ReinterpretArray{T,N,S,A} where S where A<:Union{SubArray{T,N,A,I,true} where I<:Union{Tuple{Vararg{Real,N} where N}, Tuple{AbstractUnitRange,Vararg{Any,N} where N}} where A<:DenseArray where N where T, DenseArray} where N where T, SubArray{T,N,A,I,true} where I<:Union{Tuple{Vararg{Real,N} where N}, Tuple{AbstractUnitRange,Vararg{Any,N} where N}} where A<:DenseArray where N where T, DenseArray} where N where T, DenseArray}} where N, y::Real) where P<:Dates.Period in Dates at /Users/sabae/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.2/Dates/src/deprecated.jl:46
## [34] /(A::SymTridiagonal, B::Number) in LinearAlgebra at /Users/sabae/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.2/LinearAlgebra/src/tridiag.jl:162
## [35] /(A::Tridiagonal, B::Number) in LinearAlgebra at /Users/sabae/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.2/LinearAlgebra/src/tridiag.jl:621
## [36] /(A::UpperTriangular, x::Number) in LinearAlgebra at /Users/sabae/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.2/LinearAlgebra/src/triangular.jl:736
## [37] /(A::UnitUpperTriangular, x::Number) in LinearAlgebra at /Users/sabae/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.2/LinearAlgebra/src/triangular.jl:739
## [38] /(A::LowerTriangular, x::Number) in LinearAlgebra at /Users/sabae/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.2/LinearAlgebra/src/triangular.jl:736
## [39] /(A::UnitLowerTriangular, x::Number) in LinearAlgebra at /Users/sabae/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.2/LinearAlgebra/src/triangular.jl:739
## [40] /(A::Symmetric, x::Number) in LinearAlgebra at /Users/sabae/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.2/LinearAlgebra/src/symmetric.jl:486
## [41] /(A::Hermitian, x::Real) in LinearAlgebra at /Users/sabae/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.2/LinearAlgebra/src/symmetric.jl:487
## [42] /(D::Diagonal, x::Number) in LinearAlgebra at /Users/sabae/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.2/LinearAlgebra/src/diagonal.jl:161
## [43] /(A::Bidiagonal, B::Number) in LinearAlgebra at /Users/sabae/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.2/LinearAlgebra/src/bidiag.jl:327
## [44] /(x::Union{SubArray{T,1,#s617,Tuple{Base.Slice{Base.OneTo{Int64}},Int64},false} where #s617<:SparseArrays.SparseMatrixCSC, SparseArrays.SparseVector{T,Ti} where Ti<:Integer} where T, a::Number) in SparseArrays at /Users/sabae/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.2/SparseArrays/src/sparsevector.jl:1409
## [45] /(A::AbstractArray, B::Number) in Base at arraymath.jl:55
## [46] /(A::Union{BitArray{2}, BitArray{1}}, B::Union{BitArray{2}, BitArray{1}}) in Base at bitarray.jl:1100
## [47] /(x::Number, B::BitArray) in Base at bitarray.jl:1104
## [48] /(::Number, ::Missing) in Base at missing.jl:95
## [49] /(x::P, y::P) where P<:Dates.Period in Dates at /Users/sabae/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.2/Dates/src/periods.jl:80
## [50] /(x::P, y::Real) where P<:Dates.Period in Dates at /Users/sabae/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.2/Dates/src/periods.jl:81
## [51] /(X::Union{DenseArray{P,N}, Base.ReinterpretArray{P,N,S,A} where S where A<:Union{SubArray{T,N,A,I,true} where I<:Union{Tuple{Vararg{Real,N} where N}, Tuple{AbstractUnitRange,Vararg{Any,N} where N}} where A<:DenseArray where N where T, DenseArray}, Base.ReshapedArray{P,N,A,MI} where MI<:Tuple{Vararg{Base.MultiplicativeInverses.SignedMultiplicativeInverse{Int64},N} where N} where A<:Union{Base.ReinterpretArray{T,N,S,A} where S where A<:Union{SubArray{T,N,A,I,true} where I<:Union{Tuple{Vararg{Real,N} where N}, Tuple{AbstractUnitRange,Vararg{Any,N} where N}} where A<:DenseArray where N where T, DenseArray} where N where T, SubArray{T,N,A,I,true} where I<:Union{Tuple{Vararg{Real,N} where N}, Tuple{AbstractUnitRange,Vararg{Any,N} where N}} where A<:DenseArray where N where T, DenseArray}, SubArray{P,N,A,I,L} where L where I<:Tuple{Vararg{Union{Int64, AbstractRange{Int64}, Base.AbstractCartesianIndex},N} where N} where A<:Union{Base.ReinterpretArray{T,N,S,A} where S where A<:Union{SubArray{T,N,A,I,true} where I<:Union{Tuple{Vararg{Real,N} where N}, Tuple{AbstractUnitRange,Vararg{Any,N} where N}} where A<:DenseArray where N where T, DenseArray} where N where T, Base.ReshapedArray{T,N,A,MI} where MI<:Tuple{Vararg{Base.MultiplicativeInverses.SignedMultiplicativeInverse{Int64},N} where N} where A<:Union{Base.ReinterpretArray{T,N,S,A} where S where A<:Union{SubArray{T,N,A,I,true} where I<:Union{Tuple{Vararg{Real,N} where N}, Tuple{AbstractUnitRange,Vararg{Any,N} where N}} where A<:DenseArray where N where T, DenseArray} where N where T, SubArray{T,N,A,I,true} where I<:Union{Tuple{Vararg{Real,N} where N}, Tuple{AbstractUnitRange,Vararg{Any,N} where N}} where A<:DenseArray where N where T, DenseArray} where N where T, DenseArray}} where N, y::P) where P<:Dates.Period in Dates at /Users/sabae/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.2/Dates/src/deprecated.jl:46
## [52] /(u::Adjoint{T,#s617} where #s617<:(AbstractArray{T,1} where T) where T, A::Transpose{#s617,#s616} where #s616<:Union{LowerTriangular, UpperTriangular} where #s617) in LinearAlgebra at /Users/sabae/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.2/LinearAlgebra/src/triangular.jl:2547
## [53] /(u::Adjoint{T,#s617} where #s617<:(AbstractArray{T,1} where T) where T, A::Transpose{#s617,#s616} where #s616<:Union{UnitLowerTriangular, UnitUpperTriangular} where #s617) in LinearAlgebra at /Users/sabae/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.2/LinearAlgebra/src/triangular.jl:2548
## [54] /(u::Adjoint{T,#s617} where #s617<:(AbstractArray{T,1} where T) where T, A::Transpose{#s617,#s616} where #s616<:(AbstractArray{T,2} where T) where #s617) in LinearAlgebra at /Users/sabae/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.2/LinearAlgebra/src/adjtrans.jl:282
## [55] /(u::Adjoint{T,#s617} where #s617<:(AbstractArray{T,1} where T) where T, A::Union{LowerTriangular, UpperTriangular}) in LinearAlgebra at /Users/sabae/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.2/LinearAlgebra/src/triangular.jl:2542
## [56] /(u::Adjoint{T,#s617} where #s617<:(AbstractArray{T,1} where T) where T, A::Union{UnitLowerTriangular, UnitUpperTriangular}) in LinearAlgebra at /Users/sabae/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.2/LinearAlgebra/src/triangular.jl:2543
## [57] /(u::Adjoint{T,#s617} where #s617<:(AbstractArray{T,1} where T) where T, A::Adjoint{#s617,#s616} where #s616<:Union{LowerTriangular, UpperTriangular} where #s617) in LinearAlgebra at /Users/sabae/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.2/LinearAlgebra/src/triangular.jl:2545
## [58] /(u::Adjoint{T,#s617} where #s617<:(AbstractArray{T,1} where T) where T, A::Adjoint{#s617,#s616} where #s616<:Union{UnitLowerTriangular, UnitUpperTriangular} where #s617) in LinearAlgebra at /Users/sabae/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.2/LinearAlgebra/src/triangular.jl:2546
## [59] /(u::Adjoint{T,#s617} where #s617<:(AbstractArray{T,1} where T) where T, A::AbstractArray{T,2} where T) in LinearAlgebra at /Users/sabae/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.2/LinearAlgebra/src/adjtrans.jl:280
## [60] /(u::Transpose{T,#s617} where #s617<:(AbstractArray{T,1} where T) where T, A::Adjoint{#s617,#s616} where #s616<:Union{LowerTriangular, UpperTriangular} where #s617) in LinearAlgebra at /Users/sabae/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.2/LinearAlgebra/src/triangular.jl:2553
## [61] /(u::Transpose{T,#s617} where #s617<:(AbstractArray{T,1} where T) where T, A::Adjoint{#s617,#s616} where #s616<:Union{UnitLowerTriangular, UnitUpperTriangular} where #s617) in LinearAlgebra at /Users/sabae/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.2/LinearAlgebra/src/triangular.jl:2554
## [62] /(u::Transpose{T,#s617} where #s617<:(AbstractArray{T,1} where T) where T, A::Adjoint{#s617,#s616} where #s616<:(AbstractArray{T,2} where T) where #s617) in LinearAlgebra at /Users/sabae/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.2/LinearAlgebra/src/adjtrans.jl:283
## [63] /(u::Transpose{T,#s617} where #s617<:(AbstractArray{T,1} where T) where T, A::Union{LowerTriangular, UpperTriangular}) in LinearAlgebra at /Users/sabae/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.2/LinearAlgebra/src/triangular.jl:2550
## [64] /(u::Transpose{T,#s617} where #s617<:(AbstractArray{T,1} where T) where T, A::Union{UnitLowerTriangular, UnitUpperTriangular}) in LinearAlgebra at /Users/sabae/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.2/LinearAlgebra/src/triangular.jl:2551
## [65] /(u::Transpose{T,#s617} where #s617<:(AbstractArray{T,1} where T) where T, A::Transpose{#s617,#s616} where #s616<:Union{LowerTriangular, UpperTriangular} where #s617) in LinearAlgebra at /Users/sabae/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.2/LinearAlgebra/src/triangular.jl:2555
## [66] /(u::Transpose{T,#s617} where #s617<:(AbstractArray{T,1} where T) where T, A::Transpose{#s617,#s616} where #s616<:Union{UnitLowerTriangular, UnitUpperTriangular} where #s617) in LinearAlgebra at /Users/sabae/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.2/LinearAlgebra/src/triangular.jl:2556
## [67] /(u::Transpose{T,#s617} where #s617<:(AbstractArray{T,1} where T) where T, A::AbstractArray{T,2} where T) in LinearAlgebra at /Users/sabae/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.2/LinearAlgebra/src/adjtrans.jl:281
## [68] /(A::LowerTriangular, B::LowerTriangular) in LinearAlgebra at /Users/sabae/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.2/LinearAlgebra/src/triangular.jl:1730
## [69] /(A::LowerTriangular, B::UnitLowerTriangular) in LinearAlgebra at /Users/sabae/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.2/LinearAlgebra/src/triangular.jl:1737
## [70] /(A::UpperTriangular, B::UpperTriangular) in LinearAlgebra at /Users/sabae/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.2/LinearAlgebra/src/triangular.jl:1744
## [71] /(A::UpperTriangular, B::UnitUpperTriangular) in LinearAlgebra at /Users/sabae/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.2/LinearAlgebra/src/triangular.jl:1751
## [72] /(A::LowerTriangular, xformB::Adjoint{#s617,#s616} where #s616<:UpperTriangular where #s617) in LinearAlgebra at /Users/sabae/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.2/LinearAlgebra/src/triangular.jl:1765
## [73] /(A::LowerTriangular, xformB::Adjoint{#s615,#s614} where #s614<:UnitUpperTriangular where #s615) in LinearAlgebra at /Users/sabae/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.2/LinearAlgebra/src/triangular.jl:1774
## [74] /(A::UpperTriangular, xformB::Adjoint{#s613,#s612} where #s612<:LowerTriangular where #s613) in LinearAlgebra at /Users/sabae/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.2/LinearAlgebra/src/triangular.jl:1783
## [75] /(A::UpperTriangular, xformB::Adjoint{#s611,#s610} where #s610<:UnitLowerTriangular where #s611) in LinearAlgebra at /Users/sabae/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.2/LinearAlgebra/src/triangular.jl:1792
## [76] /(A::LowerTriangular, xformB::Transpose{#s617,#s616} where #s616<:UpperTriangular where #s617) in LinearAlgebra at /Users/sabae/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.2/LinearAlgebra/src/triangular.jl:1765
## [77] /(A::LowerTriangular, xformB::Transpose{#s615,#s614} where #s614<:UnitUpperTriangular where #s615) in LinearAlgebra at /Users/sabae/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.2/LinearAlgebra/src/triangular.jl:1774
## [78] /(A::UpperTriangular, xformB::Transpose{#s613,#s612} where #s612<:LowerTriangular where #s613) in LinearAlgebra at /Users/sabae/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.2/LinearAlgebra/src/triangular.jl:1783
## [79] /(A::UpperTriangular, xformB::Transpose{#s611,#s610} where #s610<:UnitLowerTriangular where #s611) in LinearAlgebra at /Users/sabae/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.2/LinearAlgebra/src/triangular.jl:1792
## [80] /(A::AbstractArray{T,1} where T, B::Union{UnitLowerTriangular, UnitUpperTriangular}) in LinearAlgebra at /Users/sabae/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.2/LinearAlgebra/src/triangular.jl:1921
## [81] /(A::AbstractArray{T,1} where T, adjB::Adjoint{#s617,#s616} where #s616<:Union{UnitLowerTriangular, UnitUpperTriangular} where #s617) in LinearAlgebra at /Users/sabae/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.2/LinearAlgebra/src/triangular.jl:1928
## [82] /(A::AbstractArray{T,1} where T, transB::Transpose{#s615,#s614} where #s614<:Union{UnitLowerTriangular, UnitUpperTriangular} where #s615) in LinearAlgebra at /Users/sabae/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.2/LinearAlgebra/src/triangular.jl:1936
## [83] /(A::AbstractArray{T,1} where T, B::Union{LowerTriangular, UpperTriangular}) in LinearAlgebra at /Users/sabae/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.2/LinearAlgebra/src/triangular.jl:1947
## [84] /(A::AbstractArray{T,1} where T, adjB::Adjoint{#s617,#s616} where #s616<:Union{LowerTriangular, UpperTriangular} where #s617) in LinearAlgebra at /Users/sabae/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.2/LinearAlgebra/src/triangular.jl:1954
## [85] /(A::AbstractArray{T,1} where T, transB::Transpose{#s615,#s614} where #s614<:Union{LowerTriangular, UpperTriangular} where #s615) in LinearAlgebra at /Users/sabae/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.2/LinearAlgebra/src/triangular.jl:1962
## [86] /(A::AbstractArray{T,2} where T, B::Union{UnitLowerTriangular, UnitUpperTriangular}) in LinearAlgebra at /Users/sabae/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.2/LinearAlgebra/src/triangular.jl:1921
## [87] /(A::AbstractArray{T,2} where T, adjB::Adjoint{#s617,#s616} where #s616<:Union{UnitLowerTriangular, UnitUpperTriangular} where #s617) in LinearAlgebra at /Users/sabae/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.2/LinearAlgebra/src/triangular.jl:1928
## [88] /(A::AbstractArray{T,2} where T, transB::Transpose{#s615,#s614} where #s614<:Union{UnitLowerTriangular, UnitUpperTriangular} where #s615) in LinearAlgebra at /Users/sabae/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.2/LinearAlgebra/src/triangular.jl:1936
## [89] /(A::AbstractArray{T,2} where T, B::Union{LowerTriangular, UpperTriangular}) in LinearAlgebra at /Users/sabae/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.2/LinearAlgebra/src/triangular.jl:1947
## [90] /(A::AbstractArray{T,2} where T, adjB::Adjoint{#s617,#s616} where #s616<:Union{LowerTriangular, UpperTriangular} where #s617) in LinearAlgebra at /Users/sabae/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.2/LinearAlgebra/src/triangular.jl:1954
## [91] /(A::AbstractArray{T,2} where T, transB::Transpose{#s615,#s614} where #s614<:Union{LowerTriangular, UpperTriangular} where #s615) in LinearAlgebra at /Users/sabae/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.2/LinearAlgebra/src/triangular.jl:1962
## [92] /(Da::Diagonal, Db::Diagonal) in LinearAlgebra at /Users/sabae/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.2/LinearAlgebra/src/diagonal.jl:306
## [93] /(A::Union{LinearAlgebra.AbstractTriangular, Union{DenseArray{T,2}, Base.ReinterpretArray{T,2,S,A} where S where A<:Union{SubArray{T,N,A,I,true} where I<:Union{Tuple{Vararg{Real,N} where N}, Tuple{AbstractUnitRange,Vararg{Any,N} where N}} where A<:DenseArray where N where T, DenseArray}, Base.ReshapedArray{T,2,A,MI} where MI<:Tuple{Vararg{Base.MultiplicativeInverses.SignedMultiplicativeInverse{Int64},N} where N} where A<:Union{Base.ReinterpretArray{T,N,S,A} where S where A<:Union{SubArray{T,N,A,I,true} where I<:Union{Tuple{Vararg{Real,N} where N}, Tuple{AbstractUnitRange,Vararg{Any,N} where N}} where A<:DenseArray where N where T, DenseArray} where N where T, SubArray{T,N,A,I,true} where I<:Union{Tuple{Vararg{Real,N} where N}, Tuple{AbstractUnitRange,Vararg{Any,N} where N}} where A<:DenseArray where N where T, DenseArray}, SubArray{T,2,A,I,L} where L where I<:Tuple{Vararg{Union{Int64, AbstractRange{Int64}, Base.AbstractCartesianIndex},N} where N} where A<:Union{Base.ReinterpretArray{T,N,S,A} where S where A<:Union{SubArray{T,N,A,I,true} where I<:Union{Tuple{Vararg{Real,N} where N}, Tuple{AbstractUnitRange,Vararg{Any,N} where N}} where A<:DenseArray where N where T, DenseArray} where N where T, Base.ReshapedArray{T,N,A,MI} where MI<:Tuple{Vararg{Base.MultiplicativeInverses.SignedMultiplicativeInverse{Int64},N} where N} where A<:Union{Base.ReinterpretArray{T,N,S,A} where S where A<:Union{SubArray{T,N,A,I,true} where I<:Union{Tuple{Vararg{Real,N} where N}, Tuple{AbstractUnitRange,Vararg{Any,N} where N}} where A<:DenseArray where N where T, DenseArray} where N where T, SubArray{T,N,A,I,true} where I<:Union{Tuple{Vararg{Real,N} where N}, Tuple{AbstractUnitRange,Vararg{Any,N} where N}} where A<:DenseArray where N where T, DenseArray} where N where T, DenseArray}} where T}, D::Diagonal) in LinearAlgebra at /Users/sabae/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.2/LinearAlgebra/src/diagonal.jl:378
## [94] /(A::Bidiagonal{T,V} where V<:AbstractArray{T,1}, adjB::Adjoint{#s617,#s616} where #s616<:AbstractArray{T,1} where #s617) where T in LinearAlgebra at /Users/sabae/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.2/LinearAlgebra/src/bidiag.jl:574
## [95] /(A::Bidiagonal{T,V} where V<:AbstractArray{T,1}, B::AbstractArray{T,1}) where T in LinearAlgebra at /Users/sabae/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.2/LinearAlgebra/src/bidiag.jl:573
## [96] /(A::Union{AbstractArray{T,1}, AbstractArray{T,2}} where T, B::Union{AbstractArray{T,1}, AbstractArray{T,2}} where T) in LinearAlgebra at /Users/sabae/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.2/LinearAlgebra/src/generic.jl:977
## [97] /(x::Number, v::AbstractArray{T,1} where T) in LinearAlgebra at /Users/sabae/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.2/LinearAlgebra/src/generic.jl:982
## [98] /(B::AbstractArray{T,2} where T, A::LU) in LinearAlgebra at /Users/sabae/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.2/LinearAlgebra/src/lu.jl:627
## [99] /(J1::UniformScaling, J2::UniformScaling) in LinearAlgebra at /Users/sabae/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.2/LinearAlgebra/src/uniformscaling.jl:202
## [100] /(J::UniformScaling, A::AbstractArray{T,2} where T) in LinearAlgebra at /Users/sabae/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.2/LinearAlgebra/src/uniformscaling.jl:203
## [101] /(A::AbstractArray{T,2} where T, J::UniformScaling) in LinearAlgebra at /Users/sabae/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.2/LinearAlgebra/src/uniformscaling.jl:204
## [102] /(J::UniformScaling, x::Number) in LinearAlgebra at /Users/sabae/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.2/LinearAlgebra/src/uniformscaling.jl:206
## [103] /(x::RObject, y::RObject) in RCall at /Users/ir229/.julia/packages/RCall/g7dhB/src/operators.jl:6

Coding practices etc

See JuliaPraxis for best practices for naming, spacing, comments, etc