Go Module Proxy

Internals and Pitfalls

Andreas Linz—klingt.net

2019-11-14T19+01:00

A short History

$GOPATH/src

  • no version management everything was “latest”

GO15VENDOREXPERIMENT=on

  • In mid of 2015 Go 1.5 introduced experimental /vendor support
  • /vendor supplies manual managed dependencies
  • dependencies need to be part of the repository
  • (nobody likes to work with git submodules, git clone --recursive ...)

Go 2017 Survey Results:

When asked about the biggest challenges to their own personal use of Go, users clearly conveyed that lack of dependency management…

Go 2018 Survey Results:

[One of] The top three major challenges we identified are:

Package management (e.g., “Keeping up with vendoring”, “dependency / packet [sic] management / vendoring not unified”) …

🤔

Era of Package Managers

  • godm
  • gv
  • gom
  • Godep
  • Glide
  • dep 🦥
  • …

list of package managers

  • many Go developers were unhappy with package management
  • no official solution, only more or less stable third-party package management tools
  • you still needed to work inside the $GOPATH (or use “clever” hacks)

Dave Cheney said in 2016:

In my experience, many newcomers to Go are frustrated with the single workspace $GOPATH model.

With the end of 2016 a default GOPATH is set (~/go).

GO111MODULES=on

What’s a Module?

  • module: collection of related Go packages that are versioned together as single unit
  • e.g. a simple webserver using net/http from Go 1.1x with gorilla/mux as router is a collection of packages
  • modules add precise dependency requirements (go.mod) → reproducible builds
  • support will be finalized with Go 1.14

Modules documentation

vgo

Terminology

  • a repository contains one or more modules
  • a module contains one or more packages
  • package is a single directory of .go files

How to Setup a Modules Project?

$ go mod init github.com/example/project
  • migrates existing dependency definitions if present (e.g. Gopkg.toml from dep)
  • creates go.mod and go.sum files

Commit both but never edit them manually!

(go get and friends will do this for you)

Add Dependency in Specific Version

$ go get github.com/example/project@v1.2.3
  • the version can be a git reference as well (hash/branch…)

Go Modules Proxy

🕵️

  • potential information leak
  • set GOPRIVATE=my.secret.git.com[,…] for private repositories

Advantages of Using a Proxy

  • faster builds (less stress on the git server)
  • persistent depencies (if upstream is down)
  • reproducible builds (no git push -f)

Proxy Implementations

Write ony Yourself?

  • learn about modules
  • get to know the module proxy API
  • simpler access to private repositories

Private Repositories

  • go get knows nothing about authentication
  • common solution HTTPS to SSH rewrite:
$ git config --global url."git@git.company.com:".insteadOf https://git.company.com/

Module Proxy API

  • go help goproxy
  • only GET resources
module proxy protocol

$GOPROXY/<module>/@v/list

  • list of known versions
  • line by line
$ curl 'https://proxy.golang.org/github.com/gorilla/mux/@v/list'
v1.3.0
v1.7.0
v1.6.0
…

$GOPROXY/<module>/@v/<version>.info

  • returns version details and date of creation
$ curl 'https://proxy.golang.org/github.com/gorilla/mux/@v/v1.7.0.info'
{"Version":"v1.7.0","Time":"2019-01-25T16:05:53Z"}

$GOPROXY/<module>/@latest

  • not documented in go help goproxy
  • returns details of the latest available version
$ curl 'https://proxy.golang.org/github.com/gorilla/mux/@latest'
{"Version":"v1.7.3","Time":"2019-06-30T04:17:52Z"}

$GOPROXY/<module>/@v/<version>.mod

  • the module’s go.mod file
$ curl -s 'https://proxy.golang.org/github.com/gorilla/mux/@v/v1.7.0.mod' | head
module github.com/gorilla/mux

$GOPROXY/<module>/@v/<version>.zip

  • zip archive of the module at <version>
$ curl -s 'https://proxy.golang.org/github.com/gorilla/mux/@v/v1.7.0.zip' | bsdtar --list -zf- | head -n10
github.com/gorilla/mux@v1.7.0/.github/release-drafter.yml
github.com/gorilla/mux@v1.7.0/.github/stale.yml
github.com/gorilla/mux@v1.7.0/.travis.yml
github.com/gorilla/mux@v1.7.0/AUTHORS
github.com/gorilla/mux@v1.7.0/ISSUE_TEMPLATE.md
github.com/gorilla/mux@v1.7.0/LICENSE
github.com/gorilla/mux@v1.7.0/README.md
github.com/gorilla/mux@v1.7.0/bench_test.go
github.com/gorilla/mux@v1.7.0/context.go
github.com/gorilla/mux@v1.7.0/context_test.go
…

The Zip Archive

  • is pretty special

  • root folder must be:

    <import-path>@<version>/

  • it contains no metadata:

    • modification timestamps are DOS zero time (1979-11-30)
    • everything has 0644 permissions

You Should not Zip them Yourself

  • very hard to reproduce
  • only achieved archives with equal byte length (compared to proxy.golang.org)
  • checksums were still different (official zips used weird file metadata)

But…

So…

Should I Write one Myself?

  • For learning, yes!
  • For production, maybe?
  • documentation is still unfinished and all over the place
  • if a project is not using semantic versioning the API falls back to pseudo-versions:

Pseudo-Version

  • Example: v0.0.0-20191025081138-a37363377ac6
  • uses a custom date format 20060102150405 😐

Pseudo Version Format:

  • vX.0.0-yyyymmddhhmmss-abcdefabcdef
  • vX.Y.Z-pre.0.yyyymmddhhmmss-abcdefabcdef
  • vX.Y.(Z+1)-0.yyyymmddhhmmss-abcdefabcdef

Not much fun to handle them.

Further Reading

Questions?

Summary

A module proxy gives you:

  • faster builds
  • reproducible builds
  • persistent dependencies

Additionally, the checksum database eliminiates MITM modification of modules.