monochromatic

monochromatic blog: http://blog.z3bra.org
git clone git://z3bra.org/monochromatic
Log | Files | Refs

commit 251e9d348d825c5b45f0ecef5d19692bdc25ef01
parent c324148a7e8621ec00c94808b1e52ff8fd0f2f1b
Author: z3bra <willyatmailoodotorg>
Date:   Fri, 16 Sep 2016 00:57:56 +0200

New article about software packaging

Diffstat:
2016/09/pack-it-up.txt | 220+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
config.mk | 3++-
index.txt | 1+
3 files changed, 223 insertions(+), 1 deletion(-)

diff --git a/2016/09/pack-it-up.txt b/2016/09/pack-it-up.txt @@ -0,0 +1,220 @@ +# [Pack it up!](#) +## &mdash; 16 September, 2016 + +Today was a big day for me. It was the day all my software could play +together to reach a single goal: + +*Maintaining and hosting software packages!* + +Here are the tools: + +* [pm](http://z3bra.org/pm) - manage local packs +* [sick](http://z3bra.org/sick) - sign and check files +* [synk](http://z3bra.org/synk) - synchronize files between hosts +* [wendy](http://z3bra.org/wendy) - run a command on filesystem change + +Tied together with a fistful of scripts, and some easy preparation, +they help medeploy and manage my own software accross multiple operating +systems without efforts. + +This process includes three main tasks: packaging, deploying and +installing. This whole process is still at and early stage, and will +get stronger and sharper with time (and shell script will most likely +turn into actual programs). +Here is an explanation of how it works: + +0. pack an application and sign the tarball +1. upload the tarball to an online repository +2. get the latest tarball for a utility +3. install/update it locally + +The process is straigh-forward, and multiple parts can be automated. + +### Packaging + +In order to build my softwares, I need either `make` or +[`mk`](http://doc.cat-v.org/plan_9/4th_edition/papers/mk) (it's tending +to be `mk` only, but for now I still need both). Then process is then +fairly simple: + + $ cd $utility + $ mk + # mk install + +I wrote a quick script to "pack" these utilities for me. It will build +the software, install it to a temporary location, create a tarball out +of it, and sign this tarball with my private key: + + #!/bin/sh + + # user specific variables + SICKKEY=$HOME/.sick.d/$USER.key + REPO=/var/www/htdocs/dl.z3bra.org/pack + + # guess the pack name from current directory, or use given name + DIR=$(basename `pwd`) + PKG=${1:-$DIR} + + # set version from latest git tag, if applicable + TAG=$(git tag | sed -n '$p' | tr -d a-z) + VER=${TAG:-0.0} + + # this part should die... + test -f mkfile && MK=mk || MK=make + + # build pack and install to ./rootfs/usr + $MK + $MK DESTDIR=$(pwd)/rootfs PREFIX=/usr MANDIR=/usr/share/man install + ( + cd rootfs + mkdir -p $REPO + # pack and sign the installed utility + tar cvj * | sick -s -f ${SICKKEY} > $REPO/${PKG}#${VER}.tar.bz2 + ) + rm -rf rootfs + + # simply ensure that the file has been created correctly + echo + ls $REPO/${PKG}#${VER}.tar.bz2 + +At this point, to pack one of my utilities, all I need is: + + $ git clone git://z3bra.org/skroll + $ cd skroll + $ pack + +And I'm done :) + +### Deploying + +This part require a bit of setup. My current repository is at +http://dl.z3bra.org/pack, which is, locally in +`/var/www/htdocs/dl.z3bra.org/pack`. My tool `synk` can get a file +synchronized between two peers, but they will have the same path, +which is why I also created this directory on my local machine. +I also need to upload my public key (for `sick` checks) and a list +of what's currently in the repo. + +First, here is the `repogen` script, which will list the content of the +local repo, and write the pack names and version available to a file: + + #!/bin/sh + + REPO=/var/www/htdocs/dl.z3bra.org/pack + for tarball in $(find $REPO -name '*.tar.bz2'); do + pkg=$(basename $tarball | cut -d# -f1) + ver=$(basename $tarball | cut -d# -f2 | sed 's/.tar.bz2//') + printf '%s\t%s\n' "$pkg" "$ver" + done | sort | tee $REPO/.list + +I also copied the public key as `.key` in the directory. + +Now everything is ready for `synk`ronisation (over a VPN, in this case): + + find /var/www/htdocs/dl.z3bra.org/pack -type f | synk -h apophis.2f30 + +It can be automated using `wendy`, so that everytime the `.list` file +is modified by `repogen`, everything is replicated on the remote repo: + + wendy -m 8 -f $REPO/.list sh -c "find $REPO -type f | synk -h apophis.2f30" + +### Installing + +Now that we can create packs and upload them quickly to the repository, +it's time to install them! + +Using the `.list` file, we can check what's available. With the `.key` +file, we can ensure that no-one tampered with our pack during the +retrieval process. Using `pm`, we can install and update our packs for +daily use. + +All we need now, is a utility to fetch packs from the repo. I named this +script "`repo`"! + + + #!/bin/sh + + url="http://dl.z3bra.org/pack" + cache="$HOME/.cache/repo" + keyring="$HOME/.sick.d" + + usage() { + echo "usage: $(basename $0) [-s] [-t reponame] [PKG..]" >&2 + exit 1 + } + + reposync() { + mkdir -p ${cache} + curl -Ls ${url}/.list | tee ${cache}/.list + } + + repocheck() { + sick -f $HOME/.sick.d/egull.pub + } + + repotrust() { + curl -sL ${url}/.key > ${keyring}/${name}.pub + } + + repoget() { + pkg="$1" + ver=$(grep -E "^${pkg} " ${cache}/.list | tac | sed 1q | cut -f2) + + file="${pkg}#${ver}.tar.bz2" + html="${pkg}%23${ver}.tar.bz2" + + curl -sL ${url}/${html} | repocheck | ifne sponge ${cache}/${file} + test -f ${cache}/${file} \ + && readlink -f ${cache}/${file} \ + || echo "$pkg: signature check failed" >&2 + } + + repolist() { + pg -e ${cache}/.list + } + + test $# -eq 0 && { repolist && exit 0; } + + case $1 in + -s) reposync; exit 0 ;; + -l) repolist; exit 0 ;; + -t) test -z "$2" && usage || { repotrust $2; exit 0; } ;; + esac + + for n in $@; do + repoget $n + done + + exit 0 + +First, we retrieve the ed25519 public key from the repo: + + $ repo -t z3bra + +Then, the package list: + + $ repo -s + libwm 1.1 + skroll 0.6 + +And finally, install it! + + $ ROOT=$HOME/.local + $ export ROOT + $ pm -a $(repo skroll) + $ pm -i + skroll 0.6 + $ echo amazing! | skroll + +### Conclusion + +Ok, so this whole post was rather long, and not especially good at +describing the actual workflow. So as usual. Here is a quick video to +show off the whole process! + +<video controls> + <source src='/vid/20160916-skroll-install.webm' type='video/webm'> +</video> +<span class='caption'>Packagin, deploying and installing +[`skroll`](http://z3bra.org/skroll) on my system</span> + diff --git a/config.mk b/config.mk @@ -32,7 +32,8 @@ PAGES = index.html \ 2015/08/install-alpine-at-onlinenet.html \ 2016/01/make-your-own-distro.html \ 2016/03/hand-crafted-containers.html \ - 2016/08/desktop-streaming.html + 2016/08/desktop-streaming.html \ + 2016/09/pack-it-up.html FEEDS = rss/feed.xml EXTRA = css img vid data errors favicon.ico diff --git a/index.txt b/index.txt @@ -1,3 +1,4 @@ +* 0x001d - [Pack it up!](/2016/09/pack-it-up.html) * 0x001c - [Desktop streaming](/2016/08/desktop-streaming.html) * 0x001b - [Hand-crafted containers](/2016/03/hand-crafted-containers.html) * 0x001a - [Make your own distro](/2016/01/make-your-own-distro.html)