add upload to organizers notebookHEADmaster

1 files changed, 324 insertions, 51 deletions

diff --git a/organizers-notebook.md b/organizers-notebook.md
index ef153a5f..2ff20b0d 100644
--- a/organizers-notebook.md
+++ b/organizers-notebook.md
@@ -20,43 +20,48 @@ This document is the general organizers' notebook that can be used as a starting
     -   [Create the public organizers’ notebook in the wiki](#starting-up-the-conference-planning-process-create-the-public-organizers-notebook-in-the-wiki)
 -   [Phases and lessons learned](#phases-and-lessons-learned)
     -   [Draft CFP](#phases-and-lessons-learned-draft-cfp)
-    -   [Announce](#org897f0c8)
-    -   [Process proposals](#orgd12f3b1)
-    -   [Accept proposals](#orgcbfca65)
+    -   [Announce](#phases-and-lessons-learned-announce)
+    -   [Process proposals](#phases-and-lessons-learned-process-proposals)
+    -   [Accept proposals](#phases-and-lessons-learned-accept-proposals)
     -   [Draft schedule](#draft-schedule)
-    -   [Set up backstage and upload](#orgdbb77f2)
-    -   [Post the schedule](#org9456d25)
-    -   [Process uploaded files](#org75e87bb)
-    -   [Generate assets](#org125bd75)
-    -   [Prepare the infrastructure](#org9759aa9)
-    -   [Getting ready for captions](#org9f978d1)
+    -   [Set up backstage and upload](#phases-and-lessons-learned-set-up-backstage-and-upload)
+    -   [Post the schedule](#phases-and-lessons-learned-post-the-schedule)
+    -   [Process uploaded files](#phases-and-lessons-learned-process-uploaded-files)
+    -   [Generate assets](#phases-and-lessons-learned-generate-assets)
+    -   [Prepare the infrastructure](#phases-and-lessons-learned-prepare-the-infrastructure)
+    -   [Getting ready for captions](#phases-and-lessons-learned-getting-ready-for-captions)
     -   [Getting ready for the conference](#pre-conf)
     -   [During the conference](#conf)
--   [General infrastructure](#org85c6f49)
+    -   [After the conference](#phases-and-lessons-learned-after-the-conference)
+-   [General infrastructure](#general-infrastructure)
+    -   [Linode instance sizes](#general-infrastructure-linode-instance-sizes)
+    -   [File uploads](#upload)
     -   [Icecast](#icecast)
     -   [Watching pages](#watch)
     -   [VNC](#vnc)
     -   [Streaming with OBS](#obs)
-    -   [Shell scripts](#orgab7108a)
+    -   [Shell scripts](#general-infrastructure-shell-scripts)
     -   [Backstage](#backstage)
-    -   [IRC web client](#org5545d42)
-    -   [Etherpad](#orgd7fd9ee)
+    -   [IRC web client](#general-infrastructure-irc-web-client)
+    -   [Etherpad](#general-infrastructure-etherpad)
     -   [Publishing media to the server and to the wiki](#media)
-    -   [Automated IRC announcements](#org75cad9b)
+    -   [Automated IRC announcements](#general-infrastructure-automated-irc-announcements)
     -   [BigBlueButton](#bbb)
-    -   [Manual IRC announcements](#org5a3d818)
-    -   [Low-res stream](#orge08d62d)
-    -   [Upload talks to YouTube](#org7f2e95e)
-    -   [Restream to YouTube](#org0cc19c3)
+    -   [Manual IRC announcements](#general-infrastructure-manual-irc-announcements)
+    -   [Low-res stream](#general-infrastructure-low-res-stream)
+    -   [Upload talks to YouTube](#general-infrastructure-upload-talks-to-youtube)
+    -   [Restream to YouTube](#general-infrastructure-restream-to-youtube)
 -   [Other process notes](#other)
     -   [Adding another user to BBB](#bbb-user)
-    -   [When a talk is added after the schedule has already been drafted](#org4efd774)
+    -   [When a talk is added after the schedule has already been drafted](#other-process-notes-when-a-talk-is-added-after-the-schedule-has-already-been-drafted)
     -   [When a talk is cancelled](#cancel-talk)
+    -   [When a talk changes title](#other-process-notes-when-a-talk-changes-title)
     -   [When a talk Q&A changes method](#change-talk-qa)
-    -   [When a system is down](#org17cc58c)
-    -   [When we need to bring on a last-minute volunteer](#org4636e4e)
-    -   [Checking people in](#org811d67a)
-    -   [Ansible](#org5d78595)
+    -   [To play the other stream](#other-process-notes-to-play-the-other-stream)
+    -   [When a system is down](#other-process-notes-when-a-system-is-down)
+    -   [When we need to bring on a last-minute volunteer](#other-process-notes-when-we-need-to-bring-on-a-last-minute-volunteer)
+    -   [Checking people in](#other-process-notes-checking-people-in)
+    -   [Ansible](#other-process-notes-ansible)
 
 
 <a id="starting-up-the-conference-planning-process"></a>
@@ -157,7 +162,7 @@ Put inside double square brackets: `!template id=pagedraft`
 -   Put the speaker introduction right after the talk description for easier web page copying.
 
 
-<a id="org897f0c8"></a>
+<a id="phases-and-lessons-learned-announce"></a>
 
 ## Announce
 
@@ -165,7 +170,25 @@ Put inside double square brackets: `!template id=pagedraft`
 -   Submit to  <https://foss.events/>
 
 
-<a id="orgd12f3b1"></a>
+### Promote the conference
+
+Submit to <https://foss.events/>
+
+Maybe it would be good to get in touch with podcasts like
+
+-   [System Crafters](https://systemcrafters.net/)
+-   [This Week in Linux](https://www.youtube.com/playlist?list=PLbFVcOQ-YH_LRP687N0YeN78YZmBp5wqF)
+-   [Linux Unplugged](https://linuxunplugged.com/)
+-   [Ask Noah](http://asknoahshow.com/)
+-   [Linux After Dark](https://linuxafterdark.net/)
+-   [Lispy Gopher Show](https://anonradio.net/)
+
+etc to give a heads up on EmacsConf before it
+happens and also mention to them when videos are
+available to spark more interest.
+
+
+<a id="phases-and-lessons-learned-process-proposals"></a>
 
 ## Process proposals
 
@@ -194,7 +217,7 @@ When proposals come in:
     PUBLIC\_EMAIL
 
 
-<a id="orgcbfca65"></a>
+<a id="phases-and-lessons-learned-accept-proposals"></a>
 
 ## Accept proposals
 
@@ -227,7 +250,7 @@ For each talk:
 -   Add missing items to plan
 
 
-<a id="orgdbb77f2"></a>
+<a id="phases-and-lessons-learned-set-up-backstage-and-upload"></a>
 
 ## Set up backstage and upload
 
@@ -238,7 +261,7 @@ Lessons learned:
 -   Next year, let&rsquo;s get upload.emacsconf.org in place as the domain name so that people don&rsquo;t try to FTP files to it.
 
 
-<a id="org9456d25"></a>
+<a id="phases-and-lessons-learned-post-the-schedule"></a>
 
 ## Post the schedule
 
@@ -249,7 +272,7 @@ Lessons learned:
 -   Update talks.md to refer to watch.md as well
 
 
-<a id="org75e87bb"></a>
+<a id="phases-and-lessons-learned-process-uploaded-files"></a>
 
 ## Process uploaded files
 
@@ -274,7 +297,7 @@ Process:
     3.  <elisp:emacsconf-publish-backstage-index>
 
 
-<a id="org125bd75"></a>
+<a id="phases-and-lessons-learned-generate-assets"></a>
 
 ## Generate assets
 
@@ -292,7 +315,7 @@ Process:
 -   Splice together into a file that will be processed with subed-record-compile-video.
 
 
-<a id="org9759aa9"></a>
+<a id="phases-and-lessons-learned-prepare-the-infrastructure"></a>
 
 ## Prepare the infrastructure
 
@@ -314,7 +337,7 @@ Checklist:
 -   [ ] Icecast
 
 
-<a id="org9f978d1"></a>
+<a id="phases-and-lessons-learned-getting-ready-for-captions"></a>
 
 ## Getting ready for captions
 
@@ -344,6 +367,9 @@ Go through the :tminustwo: tags.
 
 Go through the :tminusone: tags.
 
+-   Set `emacsconf-publishing-phase` to `conference`.
+-   Set emacsconf-publish-include-pads to t and republish the info pages.
+
 
 <a id="conf"></a>
 
@@ -352,11 +378,198 @@ Go through the :tminusone: tags.
 <elisp:emacsconf-pad-open-shift-hyperlist>
 
 
-<a id="org85c6f49"></a>
+<a id="phases-and-lessons-learned-after-the-conference"></a>
+
+## After the conference
+
+
+### Capture stats
+
+2024 stats
+
+Saturday:
+gen: 177 peak + 14 peak lowres
+dev: 226 peak + 79 peak lowres
+
+Sunday:
+gen: 89 peak + 10 peak lowres
+
+meet peak 409% CPU (100% is 1 CPU), average 69.4%
+front peak 70.66% CPU (100% is 1 CPU)
+live peak 552% CPU (100% is 1 CPU) average 144%
+res peak 81.54% total CPU (each OBS ~250%), mem 7GB used
+
+so far we have used 1.2TB of transfer this month
+
+
+### Back up the data
+
+
+#### Icecast
+
+rsync -avze ssh live:&rsquo;/data/emacsconf-2024-\*&rsquo; media:~/2024
+
+rsync -avze ssh live:&rsquo;/data/emacsconf-2024-\*&rsquo; .
+
+
+#### BigBlueButton
+
+rsync -avze ssh root@bbb:/var/bigbluebutton/ bigbluebutton/
+
+ssh root@66.175.208.243 &rsquo;dd if=/dev/sda bs=5M &rsquo; | dd of=bbb-img-2024-12-08.img status=progress
+
+The img can be loaded with
+sudo losetup &#x2013;find &#x2013;show bbb-img-2024-12-08.img
+sudo mount /dev/loop10 test
+cd test
+
+
+##### Reviewing all the Q&A durations and chat messages
+
+    (let ((default-directory "/home/sacha/proj/emacsconf/2024/backups/bigbluebutton/published/video"))
+    	(mapconcat
+    	 (lambda (dir)
+    		 (let ((chat (xml-parse-file
+    									(expand-file-name "video.xml" dir)))
+    					 (metadata (xml-parse-file
+    											(expand-file-name "metadata.xml" dir))))
+    			 (concat
+    				"- "
+    				(dom-text (car (dom-by-tag metadata 'meetingName))) " ("
+    				(format-seconds  "%.2h:%z%.2m:%.2s"
+    												 (/ (string-to-number (dom-text (dom-by-tag metadata 'duration)))
+    														1000))
+    				")"
+    				"\n"
+    				(if (dom-by-tag chat 'chattimeline)
+    						(mapconcat (lambda (node)
+    												 (concat "  - " (dom-attr node 'name) ": "
+    																 (dom-attr node 'message) "\n"))
+    											 (dom-by-tag chat 'chattimeline)
+    											 "")
+    					"")
+    				"")
+    			 )
+    		 )
+    	 (directory-files "." nil "-"))
+    	)
+
+-   Copy IRC logs to cache and therefore to backstage: (ex: 2024-12-07-emacsconf-gen.txt)
+
+
+### Combine the IRC logs and the pads, and add them to the wiki
+
+(did we e-mail the speakers before or after?)
+
+
+### Process the Q&A recordings and live presentations
+
+-   Save answers videos as &#x2013;answers.webm.
+-   From the pad file: emacsconf-extract-copy-pad-to-wiki
+-   update cache
+-   Set `QA_PUBLIC` property to t as each Q&A gets processed.
+-   change status to `TO_INDEX_QA`
+-   emacsconf-publish-media-files-on-change
+-   emacsconf-publish-info-pages-for-talk
+
+Indexing
+
+-   emacsconf-extract-insert-note-with-question-heading
+-   emacsconf-subed-make-chapter-file-based-on-comments
+-   change status to `TO_CAPTION_QA`
+
+Also copy the YouTube comments and IRC comments
+emacsconf-extract-irc-anonymize-log
+
+
+### Send thank-you emails to the speakers
+
+
+### Notify emacs-discuss that the Q&A is up
+
+<https://lists.gnu.org/archive/html/emacsconf-discuss/2024-12/msg00000.html>
+
+    Hello, everyone!
+    
+    https://emacsconf.org/2024/talks and https://media.emacsconf.org/2024
+    should have the talks and Q&A sessions now, hooray, including chapter
+    indices for the Q&A. I've also copied over notes from Etherpad and IRC
+    on the talk pages, and I've e-mailed them to the speakers so they can
+    update the wiki directly or send in additional answers.
+    
+    BigBlueButton audio mixing is kinda iffy, so I'm holding off on
+    uploading the Q&A videos (and the live talk for secrets) to
+    YouTube/Toobnix until Leo and other volunteers have a listen to see
+    whether the audio needs to be tweaked, and maybe even to do some kind of
+    normalization. Unfortunately, recordings with separate audio tracks per
+    participant aren't available--it would be cool to figure that out next
+    year--so some talks might need some manual processing. (Please feel free
+    to volunteer if that's up your alley!) In the meantime, you can check
+    out the Q&A videos from the talk pages on the wiki. Please let us know
+    if there's anything that needs tweaking.
+    
+    Enjoy!
+    
+    Sacha
+
+
+### Write a report
+
+Sample: <https://emacsconf.org/2023/report/>
+
+
+### Send thank-you emails to the volunteers
+
+emacsconf-mail-template-ask-volunteer-for-mailing-address
+
+
+### Process the lessons learned
+
+
+<a id="general-infrastructure"></a>
 
 # General infrastructure
 
 
+<a id="general-infrastructure-linode-instance-sizes"></a>
+
+## Linode instance sizes
+
+
+### Production
+
+-   front0 shared Linode 32GB
+-   live0 shared Linode 64GB
+-   meet Dedicated CPU 16GB
+
+
+### Dormant
+
+Resize disk to 25GB, copying large files to media if needed
+
+-   nanode front0
+-   nanode live0
+-   delete meet after downloading all the recordings, or resize down to nanode
+
+
+<a id="upload"></a>
+
+## File uploads
+
+[psi-4ward/psitransfer: Simple open source self-hosted file sharing solution](https://github.com/psi-4ward/psitransfer)
+
+Before the conference:
+
+1.  Set `upload_enabled` to `true` in `prod-vars.yml` and run `ansible-playbook -i inventory.yml prod-playbook.yml --tags upload`.
+2.  Confirm that <https://upload.emacsconf.org> has the upload interface and works with the password.
+3.  Use `emacsconf-mail-template-to-all-groups` and `emacsconf-mail-upload-and-backstage-info` to e-mail the upload information to all speakers.
+
+After the conference:
+
+1.  Set `upload_enabled` to `false` in `prod-vars.yml` and run `ansible-playbook -i inventory.yml prod-playbook.yml --tags upload`.
+2.  Confirm that <https://upload.emacsconf.org> shows the in-between page.
+
+
 <a id="icecast"></a>
 
 ## Icecast
@@ -516,7 +729,7 @@ so that the year is updated in the configuration.
 This is on live.emacsconf.org and can be restarted with `/etc/init.d/emacsconf restart`.
 
 
-<a id="orgab7108a"></a>
+<a id="general-infrastructure-shell-scripts"></a>
 
 ## Shell scripts
 
@@ -630,7 +843,7 @@ Explanation of files:
 Other files might also have been uploaded by the speaker, such as slides or notes.
 
 
-<a id="org5545d42"></a>
+<a id="general-infrastructure-irc-web-client"></a>
 
 ## IRC web client
 
@@ -652,7 +865,7 @@ connection exhaustion errors. Is that something we can plan now or would
 you like me to bring it up a few days before the conference?
 
 
-<a id="orgd7fd9ee"></a>
+<a id="general-infrastructure-etherpad"></a>
 
 ## Etherpad
 
@@ -661,6 +874,10 @@ Success:
 -   [X] pad.emacsconf.org redirects to this year&rsquo;s entry
 -   [X] Each talk has its own Etherpad
 
+Lessons learned:
+
+-   After 2024: Added pronouns and pronunciation to the pad template since that&rsquo;s what the hosts will be looking at
+
 
 <a id="media"></a>
 
@@ -688,7 +905,7 @@ Switching it back to `TO_STREAM` and calling
 You can generate the index with `emacsconf-publish-update-media`.
 
 
-<a id="org75cad9b"></a>
+<a id="general-infrastructure-automated-irc-announcements"></a>
 
 ## Automated IRC announcements
 
@@ -712,8 +929,6 @@ ssh root@bbb
 cd greenlight-v3
 docker compose restart
 
-Does not seem to work in Firefox private browsing; documented at <https://emacsconf.org/2024/watch/>.
-
 
 ### System audio limitations
 
@@ -728,7 +943,41 @@ BBB says sharing system audio works only if you use Chrome on Mac OS X or Micros
 Related: <https://github.com/bigbluebutton/bigbluebutton/issues/8632>
 
 
-<a id="org5a3d818"></a>
+### Creating talk BBB rooms
+
+-   `docker exec -it greenlight-v3 /bin/bash`
+-   `bundle exec rails console`
+-   `user_id = User.find_by_email("sacha@sachachua.com").id`
+
+    (mapconcat (lambda (group)
+    					(format
+    					 "Room.create(user_id: user_id, name: \"%s - %s\")\n"
+    					 (plist-get (cadr group) :speakers)
+    					 (string-join (mapcar (lambda (talk) (plist-get talk :slug))
+    																			 (cdr group)))))
+    				(emacsconf-mail-groups (emacsconf-active-talks (emacsconf-get-talk-info)))
+    				"")
+
+-   <elisp:emacsconf-publish-bbb-static-redirects>
+
+Print out the room IDs with
+
+    Room.all.each { |x| puts x.friendly_id + " " + x.name }; nil
+
+
+### Setting up moderator access codes
+
+
+### Backing up BBB     :backup:
+
+rsync -avze ssh root@bbb:/var/bigbluebutton/ bigbluebutton/
+
+ssh root@bbb &rsquo;tar zcvf - /var/bigbluebutton /etc/bigbluebutton /root/greenlight-v3 /usr/local/bigbluebutton /usr/share/bbb-web&rsquo; > bbb-backup-$(date &ldquo;+%Y-%m-%d&rdquo;).tar.gz
+
+ssh root@bbb &rsquo;dd if=/dev/sda bs=5M &rsquo; | dd of=bbb-img-$(date &ldquo;+%Y-%m-%d&rdquo;).img status=progress
+
+
+<a id="general-infrastructure-manual-irc-announcements"></a>
 
 ## Manual IRC announcements
 
@@ -737,7 +986,7 @@ Success:
 -   [X] You can /opall, /conftopic, and /broadcast
 
 
-<a id="orge08d62d"></a>
+<a id="general-infrastructure-low-res-stream"></a>
 
 ## Low-res stream
 
@@ -748,7 +997,7 @@ backup on live:
 -   /usr/local/bin/emacsconf-lowres-dev-on-connect
 
 
-<a id="org7f2e95e"></a>
+<a id="general-infrastructure-upload-talks-to-youtube"></a>
 
 ## Upload talks to YouTube
 
@@ -784,7 +1033,7 @@ backup on live:
      (emacsconf-publish-prepare-for-display (emacsconf-get-talk-info)))
 
 
-<a id="org0cc19c3"></a>
+<a id="general-infrastructure-restream-to-youtube"></a>
 
 ## Restream to YouTube
 
@@ -802,7 +1051,7 @@ backup on live:
     docker exec -it greenlight-v2 bundle exec rake user:create["USERNAME","EMAIL","PASSWORD","user"]
 
 
-<a id="org4efd774"></a>
+<a id="other-process-notes-when-a-talk-is-added-after-the-schedule-has-already-been-drafted"></a>
 
 ## When a talk is added after the schedule has already been drafted
 
@@ -826,12 +1075,28 @@ backup on live:
 
 ## When a talk is cancelled
 
-1.  Update the status to CANCELLED.
+1.  Update the status to CANCELLED. Update that specific talk with <elisp:emacsconf-publish-info-pages-for-talk>.
 2.  Rejig the schedule if needed, and update with <elisp:emacsconf-schedule-update-from-info>
 3.  Update the published schedule with <elisp:emacsconf-update-schedule> and push the wiki.
 4.  Regenerate the intros (before, after). <elisp:emacsconf-stream-generate-in-between-pages>, then use <elisp:subed-record-compile-video> in the intros file.
-5.  Sync the new intros to the /data/emacsconf/shared/$YEAR/assets/intros directory.
-6.  Regenerate the pads: <elisp:emacsconf-pad-prepopulate-all-talks>
+5.  Resync the cache directory.
+6.  Consider updating the screenshots for sat-open and sun-open.
+
+
+<a id="other-process-notes-when-a-talk-changes-title"></a>
+
+## When a talk changes title
+
+1.  Update the title in conf.org.
+2.  Delete the FILE\_PREFIX property.
+3.  Call <elisp:emacsconf-set-file-prefix-if-needed>.
+4.  Rename existing files with <elisp:emacsconf-update-file-prefixes>.
+5.  Update the intro VTT file with the new title.
+6.  Update the wiki page for the talk.
+7.  Regenerate the in-between images with <elisp:emacsconf-stream-generate-in-between-pages>
+8.  Rerecord and recompile the intro, and recompile the intro for the talk after it.
+9.  <elisp:emacsconf-update-schedule>
+10. <elisp:emacsconf-pad-prepopulate-hyperlists>
 
 
 <a id="change-talk-qa"></a>
@@ -846,7 +1111,15 @@ backup on live:
 6.  Regenerate talks.json: <elisp:emacsconf-publish-talks-json-to-files>
 
 
-<a id="org17cc58c"></a>
+<a id="other-process-notes-to-play-the-other-stream"></a>
+
+## To play the other stream
+
+-   mpv &#x2013;profile=full <https://live0.emacsconf.org/gen.webm>
+-   Change the channel topic to note that discussion can be in the other channel
+
+
+<a id="other-process-notes-when-a-system-is-down"></a>
 
 ## When a system is down
 
@@ -859,7 +1132,7 @@ Update the status page:
 /broadcast <message> may also be helpful
 
 
-<a id="org4636e4e"></a>
+<a id="other-process-notes-when-we-need-to-bring-on-a-last-minute-volunteer"></a>
 
 ## When we need to bring on a last-minute volunteer
 
@@ -868,7 +1141,7 @@ Update the status page:
 will give them the backstage credentials and links to the index
 
 
-<a id="org811d67a"></a>
+<a id="other-process-notes-checking-people-in"></a>
 
 ## Checking people in
 
@@ -876,7 +1149,7 @@ will give them the backstage credentials and links to the index
 /room nick   (should be automatic if live, but there just in case)
 
 
-<a id="org5d78595"></a>
+<a id="other-process-notes-ansible"></a>
 
 ## Ansible