This repository contains infrastructure-as-code ansible configurations for various pieces of the EmacsConf infrastructure. ansible-galaxy collection install community.general Production: needs prod-vars.yml, see prod-vars.yml.sample Docker: needs docker-vars.yml, see docker-vars.yml.sample * How to use this playbook 1. Install ansible on your local machine and check out this repo. 2. Copy the ansible_vars block from conf.org to prod-vars.yml in this repo (alongside inventory.yml), or set emacsconf-ansible-directory in Emacs and then use emacsconf-ansible-tangle-vars to tangle the file. 3. scp orga@res.emacsconf.org:~/authorized_keys . (if you're setting up any user accounts) 4. Find the ansible-playbook command you want to run and try it out. Debugging: add -v or -vv to the =ansible-playbook= command. * Setting up a vault :PROPERTIES: :CUSTOM_ID: vault :END: You can store passwords in vault files if you like. [[https://stackoverflow.com/questions/37297249/how-to-store-ansible-become-pass-in-a-vault-and-how-to-use-it][More info]] Put this text into =host_vars/media/plain= and =host_vars/upload/plain=: #+begin_example ansible_become_pass: "{{ vaulted_become_pass }}" #+end_example Use =ansible-vault create host_vars/media/crypted= and =ansible-vault create host_vars/upload/crypted= to create files with the contents: #+begin_example vaulted_become_pass: "yourpasswordhere" #+end_example To set the password for this console session: #+begin_src sh :eval no export VAULT_PASSWORD=... #+end_src To change the password for a file: #+begin_src sh :eval no ansible-vault rekey $FILE --ask-vault-pass #+end_src * Processes At the start of the conference preparation period, change =emacsconf-year= in [[file:group_vars/all.yml]] To start a local copy of the wiki for testing, see [[#wiki-docker][Ikiwiki - Docker]]. * Wiki ** Ikiwiki *** Prod When you update htmlscrubber.pm in wiki/templates: ansible-playbook -i inventory.yml prod-playbook.yml --tags wiki-plugins ansible-playbook -i docker-inventory.yml docker-reuse-playbook.yml --tags wiki-plugins ansible-playbook -i inventory.yml prod-playbook.yml --tags wiki *** Docker :PROPERTIES: :CUSTOM_ID: wiki-docker :END: Goal: - [X] Load the wiki at http://localhost:28080 - [X] Add SSH key - [X] Add as remote - [X] Push to the wiki - [X] Have the changes show up automatically - [X] Have ansible copy the SSH key file:/docker:emacsconf-front:/home/ikiwiki/emacsconf.setup Creating: ansible-playbook -i docker-inventory.yml docker-playbook.yml --tags wiki Reusing: ansible-playbook -i docker-inventory.yml docker-reuse-playbook.yml --tags wiki Restarting after a reboot: docker restart emacsconf-front Copying your SSH key: set the docker_ssh_key Ansible variable to the path of your public key or docker cp ~/.ssh/id_rsa.pub emacsconf-front:/home/ikiwiki/.ssh/authorized_keys2 docker exec emacsconf-front chown ikiwiki:ikiwiki /home/ikiwiki/.ssh/authorized_keys2 docker exec emacsconf-front chmod 600 /home/ikiwiki/.ssh/authorized_keys2 http://localhost:28080/ ssh localhost -p 2222 docker exec -it emacsconf-front /bin/bash git remote add docker ssh://ikiwiki@127.0.0.1:2222/var/www/wiki.git Debugging ssh wiki 'cd /var/www/wiki.git; git update-ref refs/heads/master HEAD^' && git push docker 2022-pages Stuck wiki: ssh ikiwiki@localhost -p 2222 ikiwiki --setup /home/ikiwiki/emacsconf.setup -v * Processing prerecs 1. Update =group_vars/all.yml=: set =emacsconf_year=. 2. In the conf.org file, call =M-x emacsconf-set-file-prefixes= to set the file prefixes. Tweak as needed. 3. Export the talks.json with =M-x emacsconf-ansible-export-talks=. 4. ansible-playbook -i inventory.yml prod-playbook.yml --tags prerec 5. ansible-playbook -i inventory.yml prod-playbook.yml --tags caption When you receive a file, create a directory for it named =~/current/files/$slug=. Copy the uploaded file as =$video_slug--original.$extension=, or use =rename-original.sh $slug $file=. Then call =process-prerec.sh $file=. It will launch some screen sessions for reencoding the file and creating the VTT. * Setting up the backstage area 1. Doublecheck the host in [[file:inventory.yml]] and the variables in [[file:roles/media/defaults/main.yml]]. 2. ansible-playbook -i inventory.yml prod-playbook.yml --tags media --ask-become-pass (or =ansible-playbook -i inventory.yml prod-playbook.yml --tags media --ask-vault-pass= if you've [[#vault][stored it in a vault]]) 3. Update the following variables in your Emacs configuration: - emacsconf-backstage-dir - emacsconf-backstage-phase 4. Create ~/proj/emacsconf/{year}/cache 5. elisp:emacsconf-publish-talks-json-to-files 6. [[elisp:emacsconf-publish-backstage-index]] * Upload service =ansible-playbook -i inventory.yml prod-playbook.yml --tags upload --ask-become-pass= (or =ansible-playbook -i inventory.yml prod-playbook.yml --tags upload --ask-vault-pass= if you've [[#vault][stored it in a vault]]) /ssh:media|sudo:upload@media:~upload /ssh:media|sudo::/etc/nginx/sites-available sudo service upload start Next step, check firewall * Publishing Goals: - [X] Set up Emacs 28.2 or a newer one - [X] Check out the repositories - [X] Load the configuration - [X] Publish the backstage index - [X] Publish the watchpages - [X] Publish schedule to the wiki and push - [ ] Have nice interactive setup - [ ] Publish backstage index on a hook - [ ] Connect to IRC and announce talks - [ ] Push talk info the text files on the stream - [ ] Start mpv in the right display - [ ] Publish the prerec files - [ ] Publish the prerec on the page ** Prod To run the playbook and publish the main schedule: #+begin_src sh ansible-playbook -i inventory.yml prod-playbook.yml --tags wiki-publish --extra-vars='{"force_publish": true}' #+end_src #+RESULTS: :results: :end: Update a specific talk's before/nav and the main schedule: (ex: wayland) ansible-playbook playbook.yml -e '{"slug": "wayland"}' -i inventory.yml --tags publish Force-publish the schedule: ansible-playbook -i inventory.yml prod-playbook.yml --tags publish -e force_publish=true ** Development ** Docker Creating: ansible-playbook -i docker-inventory.yml docker-playbook.yml --tags wiki,publish Reusing: ansible-playbook -i docker-inventory.yml docker-reuse-playbook.yml --tags publish With docker: https://stackoverflow.com/questions/24738264/how-to-test-ansible-playbook-using-docker * Pad Before generating pads, use elisp:emacsconf-publish-talks-json-to-files to create the talks.json used. ** Production ansible-playbook -i inventory.yml prod-playbook.yml --tags pad,proxy To fall back to wikimedia rewrite: ansible-playbook -i inventory.yml prod-playbook.yml --tags proxy --extra-vars='{"use_wikimedia": true}' You can still access pads directly with direct/p like this: https://pad.emacsconf.org/direct/p/2022-journalism To undo wikimedia rewrite: ansible-playbook -i inventory.yml prod-playbook.yml --tags proxy To prepare for a load test: ansible-playbook -i inventory.yml prod-playbook.yml --tags pad --extra-vars='{"etherpad_load_test": true}' To create pads: ansible-playbook -i inventory.yml prod-playbook.yml --tags create-pads ** Docker Creating: ansible-playbook -i docker-inventory.yml docker-playbook.yml --tags pad Reusing an existing container: ansible-playbook -i docker-inventory.yml docker-reuse-playbook.yml --tags pad Connecting: docker exec -it emacsconf-pad /bin/bash Creating pads ansible-playbook -i docker-inventory.yml docker-reuse-playbook.yml --tags create-pads file:/docker:emacsconf-pad:/home/etherpad/etherpad/ Getting the API key #+NAME: pad-key #+begin_src sh docker exec emacsconf-pad cat /home/etherpad/etherpad/APIKEY.txt #+end_src #+RESULTS: pad-key :results: b7a15dc34cc7f6917cca6cd9a2b4b92145af7c7cd9b341af34869ab8cd3568be :end: #+begin_src sh :var padkey=pad-key echo curl "http://localhost:9001/api/1/createPad?apikey=$padkey&padID=emacsconf-2022" curl "http://localhost:9001/api/1/createPad?apikey=$padkey&padID=emacsconf-2022" #+end_src #+RESULTS: :results: curl http://localhost:9001/api/1/createPad?apikey=b7a15dc34cc7f6917cca6cd9a2b4b92145af7c7cd9b341af34869ab8cd3568be&padID=emacsconf-2022 {"code":0,"message":"ok","data":null} :end: ** Useful https://github.com/systemli/ansible-role-etherpad https://gist.github.com/aaronpk/7307172 * Pad proxy ansible-playbook -i inventory.yml prod-playbook.yml --tags proxy --extra-vars='{"use_wikimedia": false}' ansible-playbook -i inventory.yml prod-playbook.yml --tags proxy --extra-vars='{"use_wikimedia": true}' * Stream ** Prod Setting up icecast: ansible-playbook -i inventory.yml prod-playbook.yml --tags stream ** Testing Runs the ffmpeg command on res ansible-playbook -i inventory.yml prod-playbook.yml --tags test -e icecast_test_file=/home/orga/test.webm -e icecast_test=file -e icecast_test_track=dev Play the stream with MPV: ansible-playbook -i inventory.yml prod-playbook.yml --tags test -e icecast_test_track=dev -e icecast_test=mpv Use a test pattern (don't know if this works) ansible-playbook -i inventory.yml prod-playbook.yml --tags test -e icecast_test_track=dev -e icecast_test=pattern ** Creating the fallback files ffmpeg -y -f lavfi -i anullsrc=channel_layout=stereo:sample_rate=48000 -loop 1 -r 20 -t 10 -i sorry.png -c:v libvpx -c:a libvorbis -color_primaries 1 -color_trc 1 -colorspace 1 -crf 30 -g 120 -minrate 1.5M -b:v 1500 -g 120 -maxrate 1.5M -cluster_time_limit 5100 -shortest sorry.webm ffmpeg -y -f lavfi -i anullsrc=channel_layout=stereo:sample_rate=48000 -loop 1 -r 20 -t 10 -i sorry.png -vf scale=854:480 -c:v libvpx -c:a libvorbis -color_primaries 1 -color_trc 1 -colorspace 1 -crf 30 -g 120 -minrate 1.5M -b:v 1500 -g 120 -maxrate 1.5M -cluster_time_limit 5100 -shortest sorry-480p.webm * OBS ansible-playbook -i inventory.yml prod-playbook.yml --tags obs Resizing VNC after connection xrandr -s 1280x720 ** Firefox *** Firefox profiles like to be created in an X environment firefox -no-remote -CreateProfile "{{ emacsconf_id }}-{{ item.item.id }} *** Install Tampermonkey extension and scripts :manual: For each track: 1. Install the Tampermonkey extension by going to https://addons.mozilla.org/en-US/firefox/addon/tampermonkey/ . 2. Install the BBB script by clicking on the Tampermonkey extension, choosing *Install New Script*, and pasting in the following: #+begin_src js :eval no // ==UserScript== // @name Emacsconf BBB setup // @namespace https://emacsconf.org/ // @version 0.1 // @description Join BBB and set things up // @author You // @match https://bbb.emacsverse.org/* // @icon https://www.google.com/s2/favicons?sz=64&domain=emacsverse.org // @grant none // ==/UserScript== ( async function() { 'use strict'; const NAME = 'emacsconf'; async function waitUntil(conditionFunc, interval=500, timeout=null) { let initResult = conditionFunc(); if (initResult) return initResult; return new Promise((resolve, reject) => { let timeSoFar = 0; let timer = setInterval(() => { let result = conditionFunc(); if (result) { clearInterval(timer); resolve(result); } timeSoFar += interval; if (timeout && timeSoFar > timeout) { clearInterval(timer); reject(); } }, interval); }); } if (document.querySelector('input.join-form')) { document.querySelector('input.join-form').value = NAME; document.querySelector('#room-join').click(); return; } await waitUntil(() => document.querySelector('.icon-bbb-listen')).then((e) => e.closest('button').click()); await waitUntil(() => document.querySelector('.icon-bbb-user')).then((e) => e.closest('button').click()); })(); #+end_src Press =Ctrl+s= to save. 3. Add this script for IRC: #+begin_src js :eval no // ==UserScript== // @name Connect to EmacsConf chat automatically // @namespace https://emacsconf.org/ // @version 0.1 // @description try to take over the world! // @author You // @match https://chat.emacsconf.org/* // @icon https://www.google.com/s2/favicons?sz=64&domain=emacsconf.org // @grant none // ==/UserScript== (function() { 'use strict'; setTimeout(() => { if (document.querySelector('.connect-row')) { document.querySelector('.connect-row').closest('form').querySelector('button').click(); } }, 1000); })(); #+end_src 4. Join an BBB meeting and switch out of full-screen with F11. Check the address bar to see if autoplay is disabled (crossed-out autoplay icon). If it is, click on it and change *Block audio* to *Allow audio and video*. ** How to update scenes from the gen copy ssh emacsconf-gen@res.emacsconf.org -p 46668 "cat ~/.config/obs-studio/basic/scenes/emacsconf.json" | jq 'walk(if type == "string" then gsub("emacsconf"; "{{ emacsconf_id }}") else . end)' > roles/obs/templates/scenes.json ansible-playbook -i inventory.yml prod-playbook.yml --tags obs-scene * Media ansible-playbook -i inventory.yml prod-playbook.yml --tags media * Captioning Set up whisper: ansible-playbook -i inventory.yml prod-playbook.yml --tags caption Update caption script: ansible-playbook -i inventory.yml prod-playbook.yml --tags process-captions ffmpeg -y -i handwritten/reencode.webm -t 60 -vcodec copy -acodec copy test.webm * Other useful things nodemon -w . -e yml -x 'ansible-playbook -i inventory.yml prod-playbook.yml --tags vnc; true' * Restreaming Add something like this to your ~prod-vars.yml~: #+begin_src emacs-lisp restreaming_platforms: - name: youtube streams: - name: gen key: xxxx-xxxx-xxxx-xxxx-xxxx url: https://www.youtube.com/watch?v=xxxxxxxxxxx studio: https://studio.youtube.com/video/xxxxxxxxxxx/livestreaming source: gen.webm - name: dev key: xxxx-xxxx-xxxx-xxxx-xxxx url: https://www.youtube.com/watch?v=xxxxx-xxxxx studio: https://studio.youtube.com/video/xxxxx-xxxxx/livestreaming source: dev.webm - name: test key: xxxx-xxxx-xxxx-xxxx-xxxx studio: https://studio.youtube.com/video/xxxxxxxxxxx/livestreaming url: https://youtu.be/xxxxxxxxxxx source: gen.webm stream_url: rtmp://a.rtmp.youtube.com/live2 backup_stream: rtmp://b.rtmp.youtube.com/live2?backup=1 - name: toobnix stream_url: rtmp://toobnix.org:1935/live streams: - name: gen key: xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx url: https://toobnix.org/w/xxxxxxxxxxxxxxxxxxxxxx source: gen.webm - name: dev key: xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx url: https://toobnix.org/w/xxxxxxxxxxxxxxxxxxxxxx source: dev.webm - name: test key: xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx url: https://toobnix.org/w/xxxxxxxxxxxxxxxxxxxxxx source: gen.webm #+end_src It doesn't get automatically started, so you'll also need to call ~screen -S restream-$TRACK_ID-youtube~ and ~screen -S restream-$TRACK_ID-toobnix~.