#+PROPERTY: header-args:emacs-lisp :tangle playbook/emacsconf-playbook.el :eval no
#+OPTIONS: toc:4
#+begin_export md
<!-- This file was automatically generated from playbook.org. Instead of editing the .md, please edit the .org and republish. Thanks! -->
#+end_export

* Commands :noexport:

- [[elisp:(progn (org-md-export-to-markdown) (org-babel-tangle))][Export and tangle]]

* Roles/values
:PROPERTIES:
:CUSTOM_ID: roles
:END:

| STREAM | main organizer                                            | bandali |
| CHECK  | check-ins                                                 | sachac  |
| HOST   | hosting, intros, reading, moving questions/answers around | zaeph   |

| ${protected}     | [[/ssh:front:/var/www/media.emacsconf.org/2021/protected/]] |
| ${stream-status} | [[/ssh:front:/var/www/live.emacsconf.org/index.html]] |
| ${upcoming}      | upcoming.org shared over CRDT                           |
| ${conf}          | conf.org shared over CRDT                               |
| ${playbook}      | playbook.org shared over CRDT                           |
| ${dump}          | [[/ssh:live:/data/]]                                        |
| ${media}         | [[/ssh:front:/var/www/media.emacsconf.org/2021/]]           |

* Thursday or Friday before the conference
:PROPERTIES:
:CUSTOM_ID: days-before
:END:

- STREAM: Download prerecorded videos from ${protected}
  #+begin_src sh :eval no
  rsync -avzue ssh front:/var/www/media.emacsconf.org/2021/protected/*--main* .
  #+end_src

* On the day of the conference
:PROPERTIES:
:CUSTOM_ID: day-of
:END:
** Set up
:PROPERTIES:
:CUSTOM_ID: setup
:END:
*** Arrange screens
:PROPERTIES:
:CUSTOM_ID: screens
:END:

- CHECK:
  - Share ${upcoming}, ${playbook}, and ${conf} via CRDT: =conf-crdt-connect-and-share=
    - Current schedule, filenames/commands for playing, Q&A preference, IRC nick, pronunciation, intro notes, prerec duration, emergency contact information
    - =conf-upcoming-add-subtree=
  - Have #emacsconf-org and #emacsconf open
  - Start backup process for pad
    #+begin_src sh :eval no :tangle playbook/backup-pad.sh
    while true; do
      curl https://etherpad.wikimedia.org/p/emacsconf-2021/export/html > emacsconf-$(date +"%Y%m%d-%H%M%S").html
      sleep 15m
    end
    #+end_src
  - Computer for alternate streaming:
    - Open browser for joining BBB
    - Open MPV for playing https://live0.emacsconf.org/main.webm
- HOST:
  - Check OBS scenes for sharing windows/tabs as a virtual camera:
    - chat.emacsconf.org with #emacsconf
    - Etherpad
    - Schedule
    - next talk page
    - Clock with current time on screen: =watch TZ=America/Toronto date=
  - Set up backchannel for easy viewing
    - ${upcoming}
    - #emacsconf-org and #emacsconf channels
  - (?) Join organizer room S
- STREAM:
  - (?) Join organizer BBB room
  - Check OBS scenes
    - Prerecorded video in MPV
    - HOST's stream in MPV
  - Backchannel for easy viewing
    
*** Start streaming                                                :stream:
:PROPERTIES:
:CUSTOM_ID: start-streaming
:END:

- HOST: Display getting-ready message and start streaming to Icecast
- STREAM: Load HOST stream in MPV
- STREAM: Start streaming from OBS to local Icecast, relay to live0 icecast
- HOST: Confirm that the stream is live at https://live.emacsconf.org/main.webm
- STREAM: Update ${status} to say that the stream is live
- CHECK: Start low-resolution stream, confirm at https://live.emacsconf.org/main-480p.webm
  Call this on live0 with $CONF480PASS as the first parameter. The Icecast configuration is on =live0= at =/etc/icecast2/icecast.xml=.
  #+begin_src sh :eval no :tangle "playbook/restream-lowres.sh"
     PASS=$1
     while true; do ffmpeg -loglevel 24 -f webm -reconnect_at_eof 1 -reconnect_streamed 1  -re -i http://localhost:8000/main.webm -vf scale=854:480 -f webm -c:a copy -b:v 500k -maxrate 1M -bufsize 1M  -content_type video/webm -c:v libvpx icecast://ec2020main480pmu:$PASS@localhost:8000/main-480p.webm; done
  #+end_src
- CHECK: Start Youtube and Toobnix streams. Call this with $YOUTUBE1PASS, $YOUTUBE2PASS, or $TOOBNIX as the parameter
  #+begin_src sh :eval no :tangle "playbook/restream-flv.sh"
     MOUNT=$1
     while true; do ffmpeg -loglevel 24 -i http://localhost:8000/main.webm -cluster_size_limit 2M -cluster_time_limit 5100 -b:v 1M -crf 30 -g 125 -deadline good -threads 4 -vcodec libx264 -acodec libmp3lame -f flv $MOUNT; done
  #+end_src
- CHECK: Verify YouTube and Toobnix streams and the CPU load on live0.
- HOST: Verify with #emacsconf that the stream is active.
- CHECK: Play main stream on alternate laptop. Start alternate stream and verify. Update ${status}.
 
** Talk process
:PROPERTIES:
:CUSTOM_ID: talk
:END:
*** Check in a speaker
:PROPERTIES:
:CUSTOM_ID: check-in
:END:

Exception: [[*CHECK is unavailable][CHECK is unavailable]]

- Speaker checks in on #emacsconf-org via IRC or via e-mail ~30m before
- CHECK notes IRC nick for speaker.
- CHECK confirms Q&A preference: live/IRC/Etherpad, preferred way of getting questions
- [? unknown] How would you like to handle questions today? Live video, #emacsconf, Etherpad?
- [? IRC] You can keep an eye on #emacsconf for questions and discussions related to your talk. Feel free to answer, skip answering, tell people that you'll follow up afterwards, etc. We'll also copy questions from the Etherpad so that you can see them in one place. Let us know if you need help or if you want to get set up for live Q&A! (If the volume gets overwhelming, we can copy questions into #emacsconf-questions for you.)
- [? Etherpad] The Etherpad is at ${etherpad-url} . Would you like to find the section for your talk and get comfortable? We'll copy questions from IRC over to the pad so that you can see them. Feel free to answer, skip answering, tell people that you'll follow up afterwards, etc. Let us know if you need help or if you want to get set up for live Q&A!
- [? live] Are you thinking of handling questions over a web
  conference? I'll set you up in a BigBlueButton room so that you can
  get settled in. Let us know if you want to handle questions by IRC
  or Etherpad instead.
  - CHECK directs speaker to available room with =/checkin <room> <nick>=
  - Speaker joins talk room
  - CHECK makes speaker presenter and moderator, does last-minute tech check
    - Hello, thanks
    - Speaker tries screen sharing and webcam (optional)
    - check screen readability
    - CHECK briefs speaker on process, including:
      - live Q&A: reading questions themselves (can do in any order,
        can skip; coach possible responses for awkward things) or asking HOST to read questions to them
        - HOST can share the pad or IRC; speaker shares screen only if doing demo
      - encouragement of webcam, although it's optional
      - how HOST will join shortly before the prerec ends and then
        give them the go-ahead
      - closing any tabs watching the stream as their talk starts
        (otherwise the audio is confusing)
    - If the speaker will be giving a live presentation, CHECK
      collects emergency contact information (in case of technical
      issues) and shares it with HOST in the CRDT buffer
    - Okay to do other things until the prerec ends
  - CHECK updates ${upcoming} with link to the talk room and
    preferences for Q&A-. CHECK will also /msg the relevant
    information.

*** Present talk
:PROPERTIES:
:CUSTOM_ID: present
:END:

- CHECK marks previous talk as done and notes the time finished:
  - ${conf}
  - ${upcoming} - remove finished one
  - wiki  
- PAD clears pad colours.
- [? prerec]
  - (TODO: needed?) STREAM and HOST switch to the organizer room in BBB.
  - STREAM switches to MPV scene in OBS and plays the video (with captions if available).
    - Exception: [[*Last-minute prerecording submission][Last-minute prerecording submission]]
    - Exception: [[*Last-minute caption update][Last-minute caption update]]
  - [[*Publish information][CHECK publishes information]]
  - [[*Handle Q&A][HOST gets a head start on handling Q&A]]
  - When prerec finishes, STREAM switches to HOST stream in MPV.
- [? live]
  - Exception: [[*Speaker does not have a prerec and has not checked in][Speaker does not have a prerec and has not checked in]]
  - HOST joins the BBB room and double-checks that recording is on
  - CHECK notes that this is a live presentation and that the recording will be posted after the conference.
  - CHECK notes the time that the presentation started, recalculates schedule, and publishes it.
  - CHECK-alternate joins the BBB room and pauses main MPV.
  - STREAM switches to HOST stream in MPV.
  - Speaker presents.
    - Exception: [[*Technical issues during a live presentation][Technical issues during a live presentation]]
  - [? talk needs to be wrapped up]
    - HOST nudges speaker verbally.
  
*** Publish information
:PROPERTIES:
:CUSTOM_ID: publish
:END:

  - CHECK updates the schedule in:
    - ${conf}
    - ${upcoming}
    - wiki
  - CHECK publishes the video to media.emacsconf.org using =conf-publish-files=
  - CHECK commits the wiki page and the captions for the talk.
  - CHECK publishes the video on YouTube using =conf-video-share=.
    - Update description:
      #+begin_example
This video is available under the terms of the
Creative Commons Attribution-ShareAlike 4.0 International (CC
BY-SA 4.0) license.

You can view it using free and open source software at
${url}

${description}
      #+end_example
    - Mark it as public.
    - Add it to EmacsConf 2021 playlist.
  - CHECK publishes the video on ToobNix.
    - Copy title and description.
    - Mark it as public.
    - Add it to the EmacsConf 2021 playlist.
  - [? live sections ]
    - CHECK does a rough-cut of the recording from ${dump} to get the last X minutes or by time range. There's about a 1-2 minute delay.
      Ex: =(kill-new (conf-dump-get "alt" "10:24" "10:30" "qa_"))=
    - When there's an opportunity to do so:
      - CHECK finetunes the rough-cut recording (trim start and end) and posts it to:
        - media.emacsconf.org/2021
        - wiki page for talk
    
*** Handle Q&A
:PROPERTIES:
:CUSTOM_ID: questions
:END:

Exceptions:
- [[*Speaker has not checked in][Speaker has not checked in]]

- [? live]
  - CHECK-alternate joins the BBB room and pauses MPV.
  - HOST joins the BBB room
  - HOST starts recording in BBB or confirms that it's already recording
  - STREAM switches to the HOST stream and gives HOST the go-ahead via IRC.
  - HOST describes how to ask questions.
  - [? No questions yet]
    - HOST thanks speaker, says nice things about talk, and asks a couple of prepared questions
  - [? Awkward question]
    - HOST can try rephrasing the question.
    - HOST adds note to IRC/Etherpad that speakers can answer in any order, skip questions, answer afterwards, etc.
  - [? Q&A needs to be wrapped up]
    - HOST writes in Etherpad/IRC or nudges speaker verbally.
  - CHECK notes the time that the live Q&A finished and switches back to the main stream on CHECK-alternate.
- [? IRC/pad]
  - HOST switches to pad/chat OBS scene.
  - STREAM switches to HOST view and gives HOST the go-ahead.
  - HOST describes Q&A method and shows it on the screen.
  - While there's buffer time before the next talk, HOST can read out
    questions and answers, or transition to the next talk early
  - HOST: It's time for the next talk, but if you want to keep
    discussing the previous talk, please feel free to continue doing
    so on IRC or the pad.
- [? speaker will answer after the conference]
  - HOST switches to pad/chat OBS scene.
  - STREAM switches to HOST view and gives HOST the go-ahead.
  - HOST says the speaker is not available right now, but we'll
    forward the questions to the speaker and we'll post the speaker's
    answers on the wiki page. Leave your contact information if you
    want to be notified, or subscribe to the emacsconf-? mailing
    list to get the announcement. Please feel free to continue
    discussing the talk on IRC or the pad.
- [[*Present talk][Present next talk]]
  
** Break time
:PROPERTIES:
:CUSTOM_ID: break
:END:

- CHECK stops and restarts the Toobnix restreaming process, and re-checks the stream
- CHECK uses =conf-upcoming-add-subtree= to add the afternoon talks to upcoming.org
- HOST doublechecks network transfer limit and server health
  
** End of stream
:PROPERTIES:
:CUSTOM_ID: end
:END:

- CHECK stops ffmpeg process for Youtube
- CHECK stops ffmpeg process for Toobnix
- CHECK stops ffmpeg process for main-480p
- STREAM stops streaming
  
** In case of...
:PROPERTIES:
:CUSTOM_ID: exceptions
:END:
*** Last-minute prerecording submission
:PROPERTIES:
:CUSTOM_ID: last-minute-prerec
:END:

- CHECK will copy it from the FTP upload server to ${protected} and name it appropriately.
- CHECK will notify STREAM with the scp command and the mpv command so that STREAM can choose.
  
*** Last-minute caption update
:PROPERTIES:
:CUSTOM_ID: last-minute-captions
:END:

- CHECK uploads the --main.vtt file to ${protected}
- CHECK notifies STREAM via ${upcoming}
- STREAM uses the provided commands to download the VTT file and load it into MPV with =--sub-file=
  
*** Speaker has not checked in
:PROPERTIES:
:CUSTOM_ID: missing
:END:
- Let the previous talk run a little longer for Q&A; end at least in time for the prerec
- After the previous Q&A wraps up, play the prerec
- [? still not around after prerec finishes]
  - HOST: Speaker might be having some difficulty connecting, but we'll collect your questions on the pad and send them afterwards.
  - Can play next prerec a few minutes early
*** Speaker does not have a prerec and has not checked in
:PROPERTIES:
:CUSTOM_ID: really-missing
:END:
- Let the previous talk do live Q&A/demo if ready
- Close to the time of the missing talk:
  - See if any of the previous speakers want to be set up for an impromptu talk/extension in a BBB room, just in case
  - HOST: The next speaker might be having some difficulty connecting. In the meantime, let's...
    - OR:
      - highlight ongoing discussions
      - invite another speaker for an impromptu extension; mplsCorwin will keep a list of possible speakers who are still active
      - replay a short prerec
      - let mplsCorwin or zaeph fill in
*** CANCELLED Speaker whose talk was reallocated shows up and has a prerec
:PROPERTIES:
:CUSTOM_ID: reallocated-prerec
:END:
- CHECK copies it to ${protected}
- Plan to play prerec at the end of the day, or in any gaps if a live talk falls through
*** CANCELLED Speaker whose talk was reallocated shows up and wants to do it live
:PROPERTIES:
:CUSTOM_ID: reallocated-live
:END:
- See if there's enough time if buffers are shuffled back; if so, set up for a live presentation
- Check for alternate stream volunteers
- [? not enough time] Offer to set up a BBB room for recording or to
  accept a prerecording afterwards, then include it on the site and in
  post-conference communication
*** Alternate stream volunteer wants to stream  (nice to have, at risk)
:PROPERTIES:
:CUSTOM_ID: alternate
:END:
- CHECK gives ALTERNATE the BBB room URL for the talk they are interested in
- ALTERNATE starts streaming to assigned end point
- CHECK confirms stream
- CHECK updates ${stream-status}
- CHECK notifies STREAM and HOST
  - After prerec plays:
    - HOST: This talk has an extended demo/Q&A. You can go to ${alternate-url} to watch it, and we'll post a recording afterwards.
    - HOST sends ${alternate-url} to IRC: Alternate stream for ${title}: ${alternate-url}
- ALTERNATE notifies #emacsconf-org when the stream is done.
- CHECK updates ${stream-status} to note that the alternate stream is finished.

  FFMPEG process for sending the desktop and audio to the $CONFALT mountpoint on Linux with X11 and Alsa:

  1. Set the CONFALT environment variable to icecast://user:password@live0.emacsconf.org:8000/alt.webm
  2. Install pavucontrol if you don't have it already.
  3. Start the following command ([[file:playbook/stream-desktop-and-audio.sh]]:
     #+begin_src sh :eval no :tangle playbook/stream-desktop-and-audio.sh
     while true; do ffmpeg -loglevel 0 -ar 48000 -f alsa -channels 2 -sample_rate 48000 -i default -re -video_size 1280x720 -framerate 25 -f x11grab -i :0.0  -cluster_size_limit 2M -cluster_time_limit 5100 -content_type video/webm -c:v libvpx -b:v 1M -crf 30 -g 125 -deadline good -threads 4  -f webm $CONFALT; done
     #+end_src
  4. Use pavucontrol to set the recording source for the ffmpeg
     command to be the audio monitor, so you get system output as
     well.
     - OR:
       - [? splitting audio] [[#sound][Set up sinks for sound]]
       - [? same audio]
     - Set up audio monitor as the input for FFMPEG
     - MPV goes to MPV sink, browser goes to recording sink, FFMPEG takes in recording monitor

*** Pad malfunction or mess-up
:PROPERTIES:
:CUSTOM_ID: pad-broken
:END:

- PAD resets the pad using https://etherpad.wikimedia.org/p/emacsconf-2021/timeslider
- [? still not recovered]
  - PAD reimports the pad from backup
    
*** CRDT malfunctions
:PROPERTIES:
:CUSTOM_ID: crdt-broken
:END:

- HOST notifies CHECK and tries reconnecting
- [? still doesn't work]
  - Switch to backup Etherpad
    
*** Conduct guidelines issue
:PROPERTIES:
:CUSTOM_ID: conduct
:END:

- HOST addresses it (on-camera if needed) with a reminder and/or a kick or a ban
  
*** CHECK is unavailable
:PROPERTIES:
:CUSTOM_ID: check-gone
:END:

- HOST does check-ins
- HOST refers to conf.org for Q&A preference etc.
- STREAM checks conf.org for prerec filenames etc.
- Dropped goals:
  - Publishing recordings ASAP
  - Updating schedule/wiki on the fly
    
*** HOST is unavailable
:PROPERTIES:
:CUSTOM_ID: host-gone
:END:

- STREAM joins the BBB room and streams directly from there
- Either STREAM or CHECK does hosting
    
*** Technical issues during a live presentation
:PROPERTIES:
:CUSTOM_ID: live-issues
:END:

- HOST tries to contact the speaker
- [? back on track]
  - [? can be squeezed into remaining time]: Continue
  - [? need extra time]: CHECK fiddles with buffer of following talks in conf.org and updates schedule
  - [? need too much extra time (ex: 10min)]: HOST acknowledges
    technical issues and says we may be able to follow up after the
    conference
- [? can't resume]: HOST acknowledges technical issues and says we may
  be able to follow up after the conference
  
*** Big technical issues with streaming
:PROPERTIES:
:CUSTOM_ID: stream-issues
:END:

- HOST notifies #emacsconf and #emacsconf-org and pad
- HOST updates the 2021.md wiki page
- CHECK publishes prerecordings
  - media.emacsconf.org
  - wiki
  - Toobnix
  - Peertube
- STREAM e-mails the mailing list
  
*** live0 can't handle the load or is close to network transfer limit
:PROPERTIES:
:CUSTOM_ID: network
:END:

- OR:
  - Redirect some viewers via asking in #emacsconf:
    - watch via main-480p
    - watch via Toobnix (if we can get that working)
  - Consider dropping the restream to Toobnix (lower audience?) or to Youtube
  - Add additional node to Linode account for shared transfer pool (TODO: doublecheck)
    
* Before the conference
:PROPERTIES:
:CUSTOM_ID: before
:END:
** Help speakers with tech checks
:PROPERTIES:
:CUSTOM_ID: tech-checks
:END:
- Explain process
- Test audio, webcam, screensharing, collaborative pad
  - Music demos and other things that use system audio will need to be prerecorded (or done through virtual loopback device, maybe? Technical risk.)
  - Multi-monitor setups might not be handled well by BBB; share window instead of desktop
- Check if comfortable checking into IRC: chat.emacsconf.org/?join=emacsconf,emacsconf-org
- Get IRC nick, phone number for emergency contact, store in private wiki
- Try to record name pronunciation
- Encourage webcam for Q&A, although make it clear that it's totally optional
- Possible picture-in-picture approach to maximize screen real estate
** Experiment with setup to allow MPV / BBB sound isolation
:PROPERTIES:
:CUSTOM_ID: sound
:END:

Set MIC and OUTPUT appropriately.

#+begin_src sh :eval no
MIC=alsa_input.usb-046d_0819_A68D6BE0-02.mono-fallback
OUTPUT=alsa_output.pci-0000_00_1b.0.analog-stereo
if [ "$1" == "-u" ]; then
  pactl unload-module module-loopback
  pactl unload-module module-null-sink
else
  pacmd load-module module-null-sink sink_name=recording sink_properties=device.description=recording channels=2
  pacmd load-module module-null-sink sink_name=mpv sink_properties=device.description=mpv
  pacmd load-module module-loopback source=$MIC sink=recording
  pacmd load-module module-loopback source=mpv.monitor sink=recording
  pacmd load-module module-loopback source=recording.monitor sink=$OUTPUT
fi
#+end_src

You can probably then use =pavucontrol= (choose All Streams instead of
Applications on the Playback tab) or something like the following to
redirect the mpv output to the mpv sink.

#+begin_src sh :eval no
pacmd list-sink-inputs  # notice the ID for the process you want to redirect
pacmd move-sink-input 1 mpv
#+end_src