monochromatic

monochromatic blog: http://blog.z3bra.org
git clone git://z3bra.org/monochromatic
Log | Files | Refs

pack-it-up.txt (5915B)


      1 # [Pack it up!](#)
      2 ## — 16 September, 2016
      3 
      4 Today was a big day for me. It was the day all my software could play
      5 together to reach a single goal:
      6 
      7 *Maintaining and hosting software packages!*
      8 
      9 Here are the tools:
     10 
     11 * [pm](http://z3bra.org/pm) - manage local packs
     12 * [sick](http://z3bra.org/sick) - sign and check files
     13 * [synk](http://z3bra.org/synk) - synchronize files between hosts
     14 * [wendy](http://z3bra.org/wendy) - run a command on filesystem change
     15 
     16 Tied together with a fistful of scripts, and some easy preparation,
     17 they help me deploy and manage my own utilities accross multiple operating
     18 systems without efforts.
     19 
     20 This process includes three main tasks: packaging, deploying and
     21 installing. This whole process is still at and early stage, and will
     22 get stronger and sharper with time (and shell script will most likely
     23 turn into actual programs).  
     24 Here is an explanation of how it works:
     25 
     26 0. pack an application and sign the tarball
     27 1. upload the tarball to an online repository
     28 2. get the latest tarball for a utility
     29 3. install/update it locally
     30 
     31 The process is straigh-forward, and multiple parts can be automated.
     32 
     33 ### Packaging
     34 
     35 In order to build my softwares, I need either `make` or
     36 [`mk`](http://doc.cat-v.org/plan_9/4th_edition/papers/mk) (it's tending
     37 to be `mk` only, but for now I still need both). Then process is then
     38 fairly simple:
     39 
     40 	$ cd $utility
     41 	$ mk
     42 	# mk install
     43 
     44 I wrote a quick script to "pack" these utilities for me. It will build
     45 the software, install it to a temporary location, create a tarball out
     46 of it, and sign this tarball with my private key:
     47 
     48 	#!/bin/sh
     49 
     50 	# user specific variables
     51 	SICKKEY=$HOME/.sick.d/$USER.key
     52 	REPO=/var/www/htdocs/dl.z3bra.org/pack
     53 
     54 	# guess the pack name from current directory, or use given name
     55 	DIR=$(basename `pwd`)
     56 	PKG=${1:-$DIR}
     57 
     58 	# set version from latest git tag, if applicable
     59 	TAG=$(git tag | sed -n '$p' | tr -d a-z)
     60 	VER=${TAG:-0.0}
     61 	
     62 	# this part should die...
     63 	test -f mkfile && MK=mk || MK=make
     64 	
     65 	# build pack and install to ./rootfs/usr
     66 	$MK
     67 	$MK DESTDIR=$(pwd)/rootfs PREFIX=/usr MANDIR=/usr/share/man install
     68 	(
     69 		cd rootfs
     70 		mkdir -p $REPO
     71 		# pack and sign the installed utility
     72 		tar cvj * | sick -s -f ${SICKKEY} > $REPO/${PKG}#${VER}.tar.bz2
     73 	)
     74 	rm -rf rootfs
     75 	
     76 	# simply ensure that the file has been created correctly
     77 	echo
     78 	ls $REPO/${PKG}#${VER}.tar.bz2
     79 
     80 At this point, to pack one of my utilities, all I need is:
     81 
     82 	$ git clone git://z3bra.org/skroll
     83 	$ cd skroll
     84 	$ pack
     85 
     86 And I'm done :)
     87 
     88 ### Deploying
     89 
     90 This part require a bit of setup. My current repository is at
     91 http://dl.z3bra.org/pack, which is, locally in
     92 `/var/www/htdocs/dl.z3bra.org/pack`. My tool `synk` can get a file
     93 synchronized  between two peers, but they will have the same path,
     94 which is why I also created this directory on my local machine.
     95 I also need to upload my public key (for `sick` checks) and a list
     96 of what's currently in the repo.
     97 
     98 First, here is the `repogen` script, which will list the content of the
     99 local repo, and write the pack names and version available to a file:
    100 
    101 	#!/bin/sh
    102 
    103 	REPO=/var/www/htdocs/dl.z3bra.org/pack
    104 	for tarball in $(find $REPO -name '*.tar.bz2'); do
    105 	        pkg=$(basename $tarball | cut -d# -f1)
    106 	        ver=$(basename $tarball | cut -d# -f2 | sed 's/.tar.bz2//')
    107 	        printf '%s\t%s\n' "$pkg" "$ver"
    108 	done | sort | tee $REPO/.list
    109 
    110 I also copied the public key as `.key` in the directory.
    111 
    112 Now everything is ready for `synk`ronisation (over a VPN, in this case):
    113 
    114 	find /var/www/htdocs/dl.z3bra.org/pack -type f | synk -h apophis.2f30
    115 
    116 It can be automated using `wendy`, so that everytime the `.list` file
    117 is modified by `repogen`, everything is replicated on the remote repo:
    118 
    119 	wendy -m 8 -f $REPO/.list sh -c "find $REPO -type f | synk -h apophis.2f30"
    120 
    121 ### Installing
    122 
    123 Now that we can create packs and upload them quickly to the repository,
    124 it's time to install them!
    125 
    126 Using the `.list` file, we can check what's available. With the `.key`
    127 file, we can ensure that no-one tampered with our pack during the
    128 retrieval process. Using `pm`, we can install and update our packs for
    129 daily use.
    130 
    131 All we need now, is a utility to fetch packs from the repo. I named this
    132 script "`repo`"!
    133 
    134 
    135 	#!/bin/sh
    136 	
    137 	url="http://dl.z3bra.org/pack"
    138 	cache="$HOME/.cache/repo"
    139 	keyring="$HOME/.sick.d"
    140 	
    141 	usage() {
    142 		echo "usage: $(basename $0) [-s] [-t reponame] [PKG..]" >&2
    143 		exit 1
    144 	}
    145 	
    146 	reposync() {
    147 		mkdir -p ${cache}
    148 		curl -Ls ${url}/.list | tee ${cache}/.list
    149 	}
    150 	
    151 	repocheck() {
    152 		sick -f $HOME/.sick.d/egull.pub
    153 	}
    154 	
    155 	repotrust() {
    156 		curl -sL ${url}/.key > ${keyring}/${name}.pub
    157 	}
    158 	
    159 	repoget() {
    160 		pkg="$1"
    161 		ver=$(grep -E "^${pkg}	" ${cache}/.list | tac | sed 1q | cut -f2)
    162 	
    163 		file="${pkg}#${ver}.tar.bz2"
    164 		html="${pkg}%23${ver}.tar.bz2"
    165 	
    166 		curl -sL ${url}/${html} | repocheck | ifne sponge ${cache}/${file}
    167 		test -f ${cache}/${file} \
    168 			&& readlink -f ${cache}/${file} \
    169 			|| echo "$pkg: signature check failed" >&2
    170 	}
    171 	
    172 	repolist() {
    173 		pg -e ${cache}/.list
    174 	}
    175 	
    176 	test $# -eq 0 && { repolist && exit 0; }
    177 	
    178 	case $1 in
    179 		-s) reposync; exit 0 ;;
    180 		-l) repolist; exit 0 ;;
    181 		-t) test -z "$2" && usage || { repotrust $2; exit 0; } ;;
    182 	esac
    183 	
    184 	for n in $@; do
    185 		repoget $n
    186 	done
    187 	
    188 	exit 0
    189 
    190 First, we retrieve the ed25519 public key from the repo:
    191 
    192 	$ repo -t z3bra
    193 
    194 Then, the package list:
    195 
    196 	$ repo -s
    197 	libwm   1.1
    198 	skroll  0.6
    199 
    200 And finally, install it!
    201 
    202 	$ ROOT=$HOME/.local
    203 	$ export ROOT
    204 	$ pm -a $(repo skroll)
    205 	$ pm -i
    206 	skroll	0.6
    207 	$ echo amazing! | skroll
    208 
    209 ### Conclusion
    210 
    211 Ok, so this whole post was rather long, and not especially good at
    212 describing the actual workflow. So as usual. Here is a quick video to
    213 show off the whole process!
    214 
    215 <video controls>
    216         <source src="http://pub.z3bra.org/monochromatic/vid/20160916-skroll-install.webm" type="video/webm">
    217 </video>
    218 <span class='caption'>Packaging, deploying and installing
    219 [`skroll`](http://z3bra.org/skroll) on my system</span>
    220