This repository contains infrastructure-as-code ansible configurations for various pieces of the EmacsConf infrastructure. ansible-galaxy collection install community.general Production: needs prod-vars.yml, see prod-vars.yml.sample Docker: needs docker-vars.yml, see docker-vars.yml.sample * How to use this playbook 1. Install ansible on your local machine and check out this repo. 2. Copy the ansible_vars block from conf.org to prod-vars.yml in this repo (alongside inventory.yml), or set emacsconf-ansible-directory in Emacs and then use emacsconf-ansible-tangle-vars to tangle the file. 3. scp orga@res.emacsconf.org:~/authorized_keys . (if you're setting up any user accounts) 4. Find the ansible-playbook command you want to run and try it out. Debugging: add -v or -vv to the =ansible-playbook= command. * Setting up a vault :PROPERTIES: :CUSTOM_ID: vault :END: You can store passwords in vault files if you like. [[https://stackoverflow.com/questions/37297249/how-to-store-ansible-become-pass-in-a-vault-and-how-to-use-it][More info]] Put this text into =host_vars/media/plain= and =host_vars/upload/plain=: #+begin_example ansible_become_pass: "{{ vaulted_become_pass }}" #+end_example Use =ansible-vault create host_vars/media/crypted= and =ansible-vault create host_vars/upload/crypted= to create files with the contents: #+begin_example vaulted_become_pass: "yourpasswordhere" #+end_example To set the password for this console session: #+begin_src sh :eval no export ANSIBLE_PASSWORD=... #+end_src * Processes At the start of the conference preparation period, change =emacsconf-year= in [[file:group_vars/all.yml]] To start a local copy of the wiki for testing, see [[#wiki-docker][Ikiwiki - Docker]]. * Wiki ** Ikiwiki *** Prod When you update htmlscrubber.pm in wiki/templates: ansible-playbook -i inventory.yml prod-playbook.yml --tags wiki-plugins ansible-playbook -i docker-inventory.yml docker-reuse-playbook.yml --tags wiki-plugins ansible-playbook -i inventory.yml prod-playbook.yml --tags wiki *** Docker :PROPERTIES: :CUSTOM_ID: wiki-docker :END: Goal: - [X] Load the wiki at http://localhost:28080 - [X] Add SSH key - [X] Add as remote - [X] Push to the wiki - [X] Have the changes show up automatically - [X] Have ansible copy the SSH key file:/docker:emacsconf-front:/home/ikiwiki/emacsconf.setup Creating: ansible-playbook -i docker-inventory.yml docker-playbook.yml --tags wiki Reusing: ansible-playbook -i docker-inventory.yml docker-reuse-playbook.yml --tags wiki Restarting after a reboot: docker restart emacsconf-front Copying your SSH key: set the docker_ssh_key Ansible variable to the path of your public key or docker cp ~/.ssh/id_rsa.pub emacsconf-front:/home/ikiwiki/.ssh/authorized_keys2 docker exec emacsconf-front chown ikiwiki:ikiwiki /home/ikiwiki/.ssh/authorized_keys2 docker exec emacsconf-front chmod 600 /home/ikiwiki/.ssh/authorized_keys2 http://localhost:28080/ ssh localhost -p 2222 docker exec -it emacsconf-front /bin/bash git remote add docker ssh://ikiwiki@127.0.0.1:2222/var/www/wiki.git Debugging ssh wiki 'cd /var/www/wiki.git; git update-ref refs/heads/master HEAD^' && git push docker 2022-pages Stuck wiki: ssh ikiwiki@localhost -p 2222 ikiwiki --setup /home/ikiwiki/emacsconf.setup -v * Processing prerecs 1. Update =group_vars/all.yml=: set =emacsconf_year=. 2. In the conf.org file, call =M-x emacsconf-set-file-prefixes= to set the file prefixes. Tweak as needed. 3. Export the talks.json with =M-x emacsconf-ansible-export-talks=. 4. ansible-playbook -i inventory.yml prod-playbook.yml --tags prerec 5. ansible-playbook -i inventory.yml prod-playbook.yml --tags caption When you receive a file, create a directory for it named =~/current/files/$slug=. Copy the uploaded file as =$video_slug--original.$extension=, or use =rename-original.sh $slug $file=. Then call =process-prerec.sh $file=. It will launch some screen sessions for reencoding the file and creating the VTT. * Setting up the backstage area 1. Doublecheck the host in [[file:inventory.yml]] and the variables in [[file:roles/media/defaults/main.yml]]. 2. ansible-playbook -i inventory.yml prod-playbook.yml --tags media --ask-become-pass (or =ansible-playbook -i inventory.yml prod-playbook.yml --tags media --ask-vault-pass= if you've [[#vault][stored it in a vault]]) 3. Update the following variables in your Emacs configuration: - emacsconf-backstage-dir - emacsconf-backstage-phase * Upload service =ansible-playbook -i inventory.yml prod-playbook.yml --tags upload --ask-become-pass= (or =ansible-playbook -i inventory.yml prod-playbook.yml --tags upload --ask-vault-pass= if you've [[#vault][stored it in a vault]]) /ssh:media|sudo:upload@media:~upload /ssh:media|sudo::/etc/nginx/sites-available sudo service upload start Next step, check firewall * Publishing Goals: - [X] Set up Emacs 28.2 or a newer one - [X] Check out the repositories - [X] Load the configuration - [X] Publish the backstage index - [X] Publish the watchpages - [X] Publish schedule to the wiki and push - [ ] Have nice interactive setup - [ ] Publish backstage index on a hook - [ ] Connect to IRC and announce talks - [ ] Push talk info the text files on the stream - [ ] Start mpv in the right display - [ ] Publish the prerec files - [ ] Publish the prerec on the page ** Prod To run the playbook and publish the main schedule: #+begin_src sh ansible-playbook -i inventory.yml prod-playbook.yml --tags wiki-publish --extra-vars='{"force_publish": true}' #+end_src #+RESULTS: :results: :end: Update a specific talk's before/nav and the main schedule: (ex: wayland) ansible-playbook playbook.yml -e '{"slug": "wayland"}' -i inventory.yml --tags publish Force-publish the schedule: ansible-playbook -i inventory.yml prod-playbook.yml --tags publish -e force_publish=true ** Docker Creating: ansible-playbook -i docker-inventory.yml docker-playbook.yml --tags wiki,publish Reusing: ansible-playbook -i docker-inventory.yml docker-reuse-playbook.yml --tags publish With docker: https://stackoverflow.com/questions/24738264/how-to-test-ansible-playbook-using-docker * Pad ** Production ansible-playbook -i inventory.yml prod-playbook.yml --tags pad,proxy To fall back to wikimedia rewrite: ansible-playbook -i inventory.yml prod-playbook.yml --tags proxy --extra-vars='{"use_wikimedia": true}' You can still access pads directly with direct/p like this: https://pad.emacsconf.org/direct/p/2022-journalism To undo wikimedia rewrite: ansible-playbook -i inventory.yml prod-playbook.yml --tags proxy To prepare for a load test: ansible-playbook -i inventory.yml prod-playbook.yml --tags pad --extra-vars='{"etherpad_load_test": true}' ** Docker Creating: ansible-playbook -i docker-inventory.yml docker-playbook.yml --tags pad Reusing an existing container: ansible-playbook -i docker-inventory.yml docker-reuse-playbook.yml --tags pad Connecting: docker exec -it emacsconf-pad /bin/bash Creating pads ansible-playbook -i docker-inventory.yml docker-reuse-playbook.yml --tags create-pads file:/docker:emacsconf-pad:/home/etherpad/etherpad/ Getting the API key #+NAME: pad-key #+begin_src sh docker exec emacsconf-pad cat /home/etherpad/etherpad/APIKEY.txt #+end_src #+RESULTS: pad-key :results: b7a15dc34cc7f6917cca6cd9a2b4b92145af7c7cd9b341af34869ab8cd3568be :end: #+begin_src sh :var padkey=pad-key echo curl "http://localhost:9001/api/1/createPad?apikey=$padkey&padID=emacsconf-2022" curl "http://localhost:9001/api/1/createPad?apikey=$padkey&padID=emacsconf-2022" #+end_src #+RESULTS: :results: curl http://localhost:9001/api/1/createPad?apikey=b7a15dc34cc7f6917cca6cd9a2b4b92145af7c7cd9b341af34869ab8cd3568be&padID=emacsconf-2022 {"code":0,"message":"ok","data":null} :end: ** Useful https://github.com/systemli/ansible-role-etherpad https://gist.github.com/aaronpk/7307172 * Pad proxy ansible-playbook -i inventory.yml prod-playbook.yml --tags proxy --extra-vars='{"use_wikimedia": false}' ansible-playbook -i inventory.yml prod-playbook.yml --tags proxy --extra-vars='{"use_wikimedia": true}' * Stream ** Prod Setting up icecast: ansible-playbook -i inventory.yml prod-playbook.yml --tags stream ** Testing Runs the ffmpeg command on res ansible-playbook -i inventory.yml prod-playbook.yml --tags test -e icecast_test_file=/home/orga/test.webm -e icecast_test=file -e icecast_test_track=dev Play the stream with MPV: ansible-playbook -i inventory.yml prod-playbook.yml --tags test -e icecast_test_track=dev -e icecast_test=mpv Use a test pattern (don't know if this works) ansible-playbook -i inventory.yml prod-playbook.yml --tags test -e icecast_test_track=dev -e icecast_test=pattern ** Creating the fallback files ffmpeg -y -f lavfi -i anullsrc=channel_layout=stereo:sample_rate=48000 -loop 1 -r 20 -t 10 -i sorry.png -c:v libvpx -c:a libvorbis -color_primaries 1 -color_trc 1 -colorspace 1 -crf 30 -g 120 -minrate 1.5M -b:v 1500 -g 120 -maxrate 1.5M -cluster_time_limit 5100 -shortest sorry.webm ffmpeg -y -f lavfi -i anullsrc=channel_layout=stereo:sample_rate=48000 -loop 1 -r 20 -t 10 -i sorry.png -vf scale=854:480 -c:v libvpx -c:a libvorbis -color_primaries 1 -color_trc 1 -colorspace 1 -crf 30 -g 120 -minrate 1.5M -b:v 1500 -g 120 -maxrate 1.5M -cluster_time_limit 5100 -shortest sorry-480p.webm * OBS ansible-playbook -i inventory.yml prod-playbook.yml --tags obs Resizing VNC after connection xrandr -s 1280x720 Firefox profiles like to be created in an X environment firefox -no-remote -CreateProfile "{{ emacsconf_id }}-{{ item.item.id }} Update scenes from the gen copy: ssh emacsconf-gen@res.emacsconf.org -p 46668 "cat ~/.config/obs-studio/basic/scenes/emacsconf.json" | jq 'walk(if type == "string" then gsub("emacsconf"; "{{ emacsconf_id }}") else . end)' > roles/obs/templates/scenes.json ansible-playbook -i inventory.yml prod-playbook.yml --tags obs-scene * Media ansible-playbook -i inventory.yml prod-playbook.yml --tags media * Captioning Set up whisper: ansible-playbook -i inventory.yml prod-playbook.yml --tags caption Update caption script: ansible-playbook -i inventory.yml prod-playbook.yml --tags process-captions ffmpeg -y -i handwritten/reencode.webm -t 60 -vcodec copy -acodec copy test.webm * Other useful things nodemon -w . -e yml -x 'ansible-playbook -i inventory.yml prod-playbook.yml --tags vnc; true'