summaryrefslogtreecommitdiffstats
path: root/README.org
blob: fe36befdd7f9751875044d2c32c7afaa95ac04ce (plain) (blame)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
This repository contains infrastructure-as-code ansible configurations
for various pieces of the EmacsConf infrastructure.

ansible-galaxy collection install community.general

Production: needs prod-vars.yml, see prod-vars.yml.sample
Docker: needs docker-vars.yml, see docker-vars.yml.sample

* How to use this playbook

1. Install ansible on your local machine and check out this repo.
2. Copy the ansible_vars block from conf.org to prod-vars.yml in this repo (alongside inventory.yml), or set emacsconf-ansible-directory in Emacs and then use emacsconf-ansible-tangle-vars to tangle the file.
3. scp orga@res.emacsconf.org:~/authorized_keys .    (if you're setting up any user accounts)
4. Find the ansible-playbook command you want to run and try it out.

Debugging: add -v or -vv to the =ansible-playbook= command.

* Setting up a vault
:PROPERTIES:
:CUSTOM_ID: vault
:END:

You can store passwords in vault files if you like. [[https://stackoverflow.com/questions/37297249/how-to-store-ansible-become-pass-in-a-vault-and-how-to-use-it][More info]]

Put this text into =host_vars/media/plain= and =host_vars/upload/plain=:

#+begin_example
ansible_become_pass: "{{ vaulted_become_pass }}"
#+end_example

Use =ansible-vault create host_vars/media/crypted= and =ansible-vault create host_vars/upload/crypted= to create files with the contents:

#+begin_example
vaulted_become_pass: "yourpasswordhere"
#+end_example

To set the password for this console session:

#+begin_src sh :eval no
  export VAULT_PASSWORD=...
#+end_src

To change the password for a file:

#+begin_src sh :eval no
ansible-vault rekey $FILE --ask-vault-pass
#+end_src

* Processes

At the start of the conference preparation period, change
=emacsconf-year= in [[file:group_vars/all.yml]]

To start a local copy of the wiki for testing, see [[#wiki-docker][Ikiwiki - Docker]].

* Wiki
** Ikiwiki
*** Prod

When you update htmlscrubber.pm in wiki/templates:

ansible-playbook -i inventory.yml prod-playbook.yml --tags wiki-plugins
ansible-playbook -i docker-inventory.yml docker-reuse-playbook.yml --tags wiki-plugins

ansible-playbook -i inventory.yml prod-playbook.yml --tags wiki
*** Docker
:PROPERTIES:
:CUSTOM_ID: wiki-docker
:END:

Goal:
- [X] Load the wiki at http://localhost:28080
- [X] Add SSH key
- [X] Add as remote
- [X] Push to the wiki
- [X] Have the changes show up automatically
- [X] Have ansible copy the SSH key
  
file:/docker:emacsconf-front:/home/ikiwiki/emacsconf.setup

Creating:
ansible-playbook -i docker-inventory.yml docker-playbook.yml --tags wiki

Reusing:
ansible-playbook -i docker-inventory.yml docker-reuse-playbook.yml --tags wiki

Restarting after a reboot:
docker restart emacsconf-front

Copying your SSH key:
set the docker_ssh_key Ansible variable to the path of your public key
or
docker cp ~/.ssh/id_rsa.pub emacsconf-front:/home/ikiwiki/.ssh/authorized_keys2
docker exec emacsconf-front chown ikiwiki:ikiwiki /home/ikiwiki/.ssh/authorized_keys2
docker exec emacsconf-front chmod 600 /home/ikiwiki/.ssh/authorized_keys2

http://localhost:28080/
ssh localhost -p 2222

docker exec -it emacsconf-front /bin/bash

git remote add docker ssh://ikiwiki@127.0.0.1:2222/var/www/wiki.git

Debugging
ssh wiki 'cd /var/www/wiki.git; git update-ref refs/heads/master HEAD^' && git push docker 2022-pages

Stuck wiki:
ssh ikiwiki@localhost -p 2222 ikiwiki --setup /home/ikiwiki/emacsconf.setup -v

* Processing prerecs

1. Update =group_vars/all.yml=: set =emacsconf_year=.
2. In the conf.org file, call =M-x emacsconf-set-file-prefixes= to set the file prefixes. Tweak as needed.
3. Export the talks.json with =M-x emacsconf-ansible-export-talks=.
4. ansible-playbook -i inventory.yml prod-playbook.yml --tags prerec
5. ansible-playbook -i inventory.yml prod-playbook.yml --tags caption

When you receive a file, create a directory for it named =~/current/files/$slug=. Copy the uploaded file as =$video_slug--original.$extension=, or use =rename-original.sh $slug $file=.

Then call =process-prerec.sh $file=. It will launch some screen sessions for reencoding the file and creating the VTT.


* Setting up the backstage area

1. Doublecheck the host in [[file:inventory.yml]] and the variables in [[file:roles/media/defaults/main.yml]].
2. ansible-playbook -i inventory.yml prod-playbook.yml --tags media --ask-become-pass
   (or =ansible-playbook -i inventory.yml prod-playbook.yml --tags media --ask-vault-pass= if you've [[#vault][stored it in a vault]])
3. Update the following variables in your Emacs configuration:
   - emacsconf-backstage-dir
   - emacsconf-backstage-phase
4. Create ~/proj/emacsconf/{year}/cache
5. elisp:emacsconf-publish-talks-json-to-files
6. [[elisp:emacsconf-publish-backstage-index]]
* Upload service

=ansible-playbook -i inventory.yml prod-playbook.yml --tags upload --ask-become-pass=
(or =ansible-playbook -i inventory.yml prod-playbook.yml --tags upload --ask-vault-pass= if you've [[#vault][stored it in a vault]])

/ssh:media|sudo:upload@media:~upload
/ssh:media|sudo::/etc/nginx/sites-available

sudo service upload start

Next step, check firewall

* Publishing

Goals:
- [X] Set up Emacs 28.2 or a newer one
- [X] Check out the repositories
- [X] Load the configuration
- [X] Publish the backstage index
- [X] Publish the watchpages
- [X] Publish schedule to the wiki and push
- [ ] Have nice interactive setup
- [ ] Publish backstage index on a hook
- [ ] Connect to IRC and announce talks
- [ ] Push talk info the text files on the stream
- [ ] Start mpv in the right display
- [ ] Publish the prerec files
- [ ] Publish the prerec on the page

** Prod
To run the playbook and publish the main schedule:

#+begin_src sh
ansible-playbook -i inventory.yml prod-playbook.yml --tags wiki-publish --extra-vars='{"force_publish": true}'
#+end_src

#+RESULTS:
:results:
:end:


Update a specific talk's before/nav and the main schedule: (ex: wayland)
ansible-playbook playbook.yml -e '{"slug": "wayland"}' -i inventory.yml --tags publish

Force-publish the schedule:
ansible-playbook -i inventory.yml prod-playbook.yml --tags publish -e force_publish=true
** Development
** Docker
Creating:
ansible-playbook -i docker-inventory.yml docker-playbook.yml --tags wiki,publish

Reusing:
ansible-playbook -i docker-inventory.yml docker-reuse-playbook.yml --tags publish

With docker:
https://stackoverflow.com/questions/24738264/how-to-test-ansible-playbook-using-docker
* Pad

Before generating pads, use elisp:emacsconf-publish-talks-json-to-files to create the talks.json used.

** Production
ansible-playbook -i inventory.yml prod-playbook.yml --tags pad,proxy

To fall back to wikimedia rewrite:
ansible-playbook -i inventory.yml prod-playbook.yml --tags proxy --extra-vars='{"use_wikimedia": true}'
You can still access pads directly with direct/p like this:
https://pad.emacsconf.org/direct/p/2022-journalism

To undo wikimedia rewrite:
ansible-playbook -i inventory.yml prod-playbook.yml --tags proxy 

To prepare for a load test:
ansible-playbook -i inventory.yml prod-playbook.yml --tags pad --extra-vars='{"etherpad_load_test": true}'

To create pads:
ansible-playbook -i inventory.yml prod-playbook.yml --tags create-pads
** Docker
Creating:
ansible-playbook -i docker-inventory.yml docker-playbook.yml --tags pad

Reusing an existing container:
ansible-playbook -i docker-inventory.yml docker-reuse-playbook.yml --tags pad

Connecting:
docker exec -it emacsconf-pad /bin/bash

Creating pads
ansible-playbook -i docker-inventory.yml docker-reuse-playbook.yml --tags create-pads

file:/docker:emacsconf-pad:/home/etherpad/etherpad/

Getting the API key

#+NAME: pad-key
#+begin_src sh
docker exec  emacsconf-pad cat /home/etherpad/etherpad/APIKEY.txt
#+end_src

#+RESULTS: pad-key
:results:
b7a15dc34cc7f6917cca6cd9a2b4b92145af7c7cd9b341af34869ab8cd3568be
:end:


#+begin_src sh :var padkey=pad-key
echo curl "http://localhost:9001/api/1/createPad?apikey=$padkey&padID=emacsconf-2022"
curl "http://localhost:9001/api/1/createPad?apikey=$padkey&padID=emacsconf-2022"
#+end_src

#+RESULTS:
:results:
curl http://localhost:9001/api/1/createPad?apikey=b7a15dc34cc7f6917cca6cd9a2b4b92145af7c7cd9b341af34869ab8cd3568be&padID=emacsconf-2022
{"code":0,"message":"ok","data":null}
:end:
** Useful
https://github.com/systemli/ansible-role-etherpad
https://gist.github.com/aaronpk/7307172
* Pad proxy

ansible-playbook -i inventory.yml prod-playbook.yml --tags proxy --extra-vars='{"use_wikimedia": false}'

ansible-playbook -i inventory.yml prod-playbook.yml --tags proxy --extra-vars='{"use_wikimedia": true}'

* Stream
** Prod
Setting up icecast:

ansible-playbook -i inventory.yml prod-playbook.yml --tags stream



** Testing

Runs the ffmpeg command on res

ansible-playbook -i inventory.yml prod-playbook.yml --tags test -e icecast_test_file=/home/orga/test.webm -e icecast_test=file -e icecast_test_track=dev

Play the stream with MPV:

ansible-playbook -i inventory.yml prod-playbook.yml --tags test -e icecast_test_track=dev -e icecast_test=mpv

Use a test pattern (don't know if this works)

ansible-playbook -i inventory.yml prod-playbook.yml --tags test -e icecast_test_track=dev -e icecast_test=pattern
** Creating the fallback files
ffmpeg -y -f lavfi -i anullsrc=channel_layout=stereo:sample_rate=48000 -loop 1 -r 20 -t 10 -i sorry.png -c:v libvpx -c:a libvorbis -color_primaries 1 -color_trc 1 -colorspace 1 -crf 30 -g 120 -minrate 1.5M -b:v 1500 -g 120 -maxrate 1.5M -cluster_time_limit 5100 -shortest sorry.webm

ffmpeg -y -f lavfi -i anullsrc=channel_layout=stereo:sample_rate=48000 -loop 1 -r 20 -t 10 -i sorry.png -vf scale=854:480 -c:v libvpx -c:a libvorbis -color_primaries 1 -color_trc 1 -colorspace 1 -crf 30 -g 120 -minrate 1.5M -b:v 1500 -g 120 -maxrate 1.5M -cluster_time_limit 5100 -shortest sorry-480p.webm

* OBS

ansible-playbook -i inventory.yml prod-playbook.yml --tags obs


Resizing VNC after connection
xrandr -s 1280x720
** Firefox
*** Firefox profiles like to be created in an X environment
firefox -no-remote -CreateProfile "{{ emacsconf_id }}-{{ item.item.id }}
*** Install Tampermonkey extension and scripts                     :manual:

For each track:

1. Install the Tampermonkey extension by going to https://addons.mozilla.org/en-US/firefox/addon/tampermonkey/ .

2. Install the BBB script by clicking on the Tampermonkey extension, choosing *Install New Script*, and pasting in the following:

   #+begin_src js :eval no
   // ==UserScript==
   // @name         Emacsconf BBB setup
   // @namespace    https://emacsconf.org/
   // @version      0.1
   // @description  Join BBB and set things up
   // @author       You
   // @match        https://bbb.emacsverse.org/*
   // @icon         https://www.google.com/s2/favicons?sz=64&domain=emacsverse.org
   // @grant        none
   // ==/UserScript==
   (
       async function() {
           'use strict';
           const NAME = 'emacsconf';
           async function waitUntil(conditionFunc, interval=500, timeout=null) {
               let initResult = conditionFunc();
               if (initResult) return initResult;
               return new Promise((resolve, reject) => {
                   let timeSoFar = 0;
                   let timer = setInterval(() => {
                       let result = conditionFunc();
                       if (result) {
                           clearInterval(timer);
                           resolve(result);
                       }
                       timeSoFar += interval;
                       if (timeout && timeSoFar > timeout) {
                           clearInterval(timer);
                           reject();
                       }
                   }, interval);
               });
           }
           if (document.querySelector('input.join-form')) {
               document.querySelector('input.join-form').value = NAME;
               document.querySelector('#room-join').click();
               return;
           }
           await waitUntil(() => document.querySelector('.icon-bbb-listen')).then((e) => e.closest('button').click());
           await waitUntil(() => document.querySelector('.icon-bbb-user')).then((e) => e.closest('button').click());
   })();
   #+end_src

   Press =Ctrl+s= to save.

3. Add this script for IRC:

   #+begin_src js :eval no
// ==UserScript==
// @name         Connect to EmacsConf chat automatically
// @namespace    https://emacsconf.org/
// @version      0.1
// @description  try to take over the world!
// @author       You
// @match        https://chat.emacsconf.org/*
// @icon         https://www.google.com/s2/favicons?sz=64&domain=emacsconf.org
// @grant        none
// ==/UserScript==

(function() {
    'use strict';
    setTimeout(() => {
        if (document.querySelector('.connect-row')) {
            document.querySelector('.connect-row').closest('form').querySelector('button').click();
        }
    }, 1000);
})();
   #+end_src


4. Join an BBB meeting and switch out of full-screen with F11. Check the address bar to see if autoplay is disabled (crossed-out autoplay icon). If it is, click on it and change *Block audio* to *Allow audio and video*.



** How to update scenes from the gen copy
ssh emacsconf-gen@res.emacsconf.org -p 46668 "cat ~/.config/obs-studio/basic/scenes/emacsconf.json" | jq 'walk(if type == "string" then gsub("emacsconf"; "{{ emacsconf_id }}") else . end)' > roles/obs/templates/scenes.json
ansible-playbook -i inventory.yml prod-playbook.yml --tags obs-scene

* Media

ansible-playbook -i inventory.yml prod-playbook.yml --tags media
* Captioning

Set up whisper:

ansible-playbook -i inventory.yml prod-playbook.yml --tags caption

Update caption script:

ansible-playbook -i inventory.yml prod-playbook.yml --tags process-captions

ffmpeg -y -i handwritten/reencode.webm -t 60 -vcodec copy -acodec copy test.webm 
* Other useful things

nodemon -w . -e yml -x 'ansible-playbook -i inventory.yml prod-playbook.yml --tags vnc; true'

* Restreaming

Add something like this to your ~prod-vars.yml~:

#+begin_src emacs-lisp
restreaming_platforms:
  - name: youtube
    streams:
      - name: gen
        key: xxxx-xxxx-xxxx-xxxx-xxxx
        url: https://www.youtube.com/watch?v=xxxxxxxxxxx
        studio: https://studio.youtube.com/video/xxxxxxxxxxx/livestreaming
        source: gen.webm
      - name: dev
        key: xxxx-xxxx-xxxx-xxxx-xxxx
        url: https://www.youtube.com/watch?v=xxxxx-xxxxx
        studio: https://studio.youtube.com/video/xxxxx-xxxxx/livestreaming
        source: dev.webm
      - name: test
        key: xxxx-xxxx-xxxx-xxxx-xxxx
        studio: https://studio.youtube.com/video/xxxxxxxxxxx/livestreaming
        url: https://youtu.be/xxxxxxxxxxx
        source: gen.webm
    stream_url: rtmp://a.rtmp.youtube.com/live2
    backup_stream: rtmp://b.rtmp.youtube.com/live2?backup=1
  - name: toobnix
    stream_url: rtmp://toobnix.org:1935/live
    streams:
      - name: gen
        key: xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
        url: https://toobnix.org/w/xxxxxxxxxxxxxxxxxxxxxx
        source: gen.webm
      - name: dev
        key: xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
        url: https://toobnix.org/w/xxxxxxxxxxxxxxxxxxxxxx
        source: dev.webm
      - name: test
        key: xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
        url: https://toobnix.org/w/xxxxxxxxxxxxxxxxxxxxxx
        source: gen.webm
#+end_src

It doesn't get automatically started, so you'll also need to call ~screen -S restream-$TRACK_ID-youtube~ and ~screen -S restream-$TRACK_ID-toobnix~.