From 5cbebe97a64117710d4c8654370ef6395485b1d8 Mon Sep 17 00:00:00 2001 From: Sacha Chua Date: Fri, 17 Jan 2025 15:25:25 -0500 Subject: add upload to organizers notebook --- organizers-notebook.md | 375 ++++++++++++++++++++++++++++++++++++++++++------- 1 file changed, 324 insertions(+), 51 deletions(-) (limited to 'organizers-notebook.md') diff --git a/organizers-notebook.md b/organizers-notebook.md index ef153a5f..2ff20b0d 100644 --- a/organizers-notebook.md +++ b/organizers-notebook.md @@ -20,43 +20,48 @@ This document is the general organizers' notebook that can be used as a starting - [Create the public organizers’ notebook in the wiki](#starting-up-the-conference-planning-process-create-the-public-organizers-notebook-in-the-wiki) - [Phases and lessons learned](#phases-and-lessons-learned) - [Draft CFP](#phases-and-lessons-learned-draft-cfp) - - [Announce](#org897f0c8) - - [Process proposals](#orgd12f3b1) - - [Accept proposals](#orgcbfca65) + - [Announce](#phases-and-lessons-learned-announce) + - [Process proposals](#phases-and-lessons-learned-process-proposals) + - [Accept proposals](#phases-and-lessons-learned-accept-proposals) - [Draft schedule](#draft-schedule) - - [Set up backstage and upload](#orgdbb77f2) - - [Post the schedule](#org9456d25) - - [Process uploaded files](#org75e87bb) - - [Generate assets](#org125bd75) - - [Prepare the infrastructure](#org9759aa9) - - [Getting ready for captions](#org9f978d1) + - [Set up backstage and upload](#phases-and-lessons-learned-set-up-backstage-and-upload) + - [Post the schedule](#phases-and-lessons-learned-post-the-schedule) + - [Process uploaded files](#phases-and-lessons-learned-process-uploaded-files) + - [Generate assets](#phases-and-lessons-learned-generate-assets) + - [Prepare the infrastructure](#phases-and-lessons-learned-prepare-the-infrastructure) + - [Getting ready for captions](#phases-and-lessons-learned-getting-ready-for-captions) - [Getting ready for the conference](#pre-conf) - [During the conference](#conf) -- [General infrastructure](#org85c6f49) + - [After the conference](#phases-and-lessons-learned-after-the-conference) +- [General infrastructure](#general-infrastructure) + - [Linode instance sizes](#general-infrastructure-linode-instance-sizes) + - [File uploads](#upload) - [Icecast](#icecast) - [Watching pages](#watch) - [VNC](#vnc) - [Streaming with OBS](#obs) - - [Shell scripts](#orgab7108a) + - [Shell scripts](#general-infrastructure-shell-scripts) - [Backstage](#backstage) - - [IRC web client](#org5545d42) - - [Etherpad](#orgd7fd9ee) + - [IRC web client](#general-infrastructure-irc-web-client) + - [Etherpad](#general-infrastructure-etherpad) - [Publishing media to the server and to the wiki](#media) - - [Automated IRC announcements](#org75cad9b) + - [Automated IRC announcements](#general-infrastructure-automated-irc-announcements) - [BigBlueButton](#bbb) - - [Manual IRC announcements](#org5a3d818) - - [Low-res stream](#orge08d62d) - - [Upload talks to YouTube](#org7f2e95e) - - [Restream to YouTube](#org0cc19c3) + - [Manual IRC announcements](#general-infrastructure-manual-irc-announcements) + - [Low-res stream](#general-infrastructure-low-res-stream) + - [Upload talks to YouTube](#general-infrastructure-upload-talks-to-youtube) + - [Restream to YouTube](#general-infrastructure-restream-to-youtube) - [Other process notes](#other) - [Adding another user to BBB](#bbb-user) - - [When a talk is added after the schedule has already been drafted](#org4efd774) + - [When a talk is added after the schedule has already been drafted](#other-process-notes-when-a-talk-is-added-after-the-schedule-has-already-been-drafted) - [When a talk is cancelled](#cancel-talk) + - [When a talk changes title](#other-process-notes-when-a-talk-changes-title) - [When a talk Q&A changes method](#change-talk-qa) - - [When a system is down](#org17cc58c) - - [When we need to bring on a last-minute volunteer](#org4636e4e) - - [Checking people in](#org811d67a) - - [Ansible](#org5d78595) + - [To play the other stream](#other-process-notes-to-play-the-other-stream) + - [When a system is down](#other-process-notes-when-a-system-is-down) + - [When we need to bring on a last-minute volunteer](#other-process-notes-when-we-need-to-bring-on-a-last-minute-volunteer) + - [Checking people in](#other-process-notes-checking-people-in) + - [Ansible](#other-process-notes-ansible) @@ -157,7 +162,7 @@ Put inside double square brackets: `!template id=pagedraft` - Put the speaker introduction right after the talk description for easier web page copying. - + ## Announce @@ -165,7 +170,25 @@ Put inside double square brackets: `!template id=pagedraft` - Submit to - +### Promote the conference + +Submit to + +Maybe it would be good to get in touch with podcasts like + +- [System Crafters](https://systemcrafters.net/) +- [This Week in Linux](https://www.youtube.com/playlist?list=PLbFVcOQ-YH_LRP687N0YeN78YZmBp5wqF) +- [Linux Unplugged](https://linuxunplugged.com/) +- [Ask Noah](http://asknoahshow.com/) +- [Linux After Dark](https://linuxafterdark.net/) +- [Lispy Gopher Show](https://anonradio.net/) + +etc to give a heads up on EmacsConf before it +happens and also mention to them when videos are +available to spark more interest. + + + ## Process proposals @@ -194,7 +217,7 @@ When proposals come in: PUBLIC\_EMAIL - + ## Accept proposals @@ -227,7 +250,7 @@ For each talk: - Add missing items to plan - + ## Set up backstage and upload @@ -238,7 +261,7 @@ Lessons learned: - Next year, let’s get upload.emacsconf.org in place as the domain name so that people don’t try to FTP files to it. - + ## Post the schedule @@ -249,7 +272,7 @@ Lessons learned: - Update talks.md to refer to watch.md as well - + ## Process uploaded files @@ -274,7 +297,7 @@ Process: 3. - + ## Generate assets @@ -292,7 +315,7 @@ Process: - Splice together into a file that will be processed with subed-record-compile-video. - + ## Prepare the infrastructure @@ -314,7 +337,7 @@ Checklist: - [ ] Icecast - + ## Getting ready for captions @@ -344,6 +367,9 @@ Go through the :tminustwo: tags. Go through the :tminusone: tags. +- Set `emacsconf-publishing-phase` to `conference`. +- Set emacsconf-publish-include-pads to t and republish the info pages. + @@ -352,11 +378,198 @@ Go through the :tminusone: tags. - + + +## After the conference + + +### Capture stats + +2024 stats + +Saturday: +gen: 177 peak + 14 peak lowres +dev: 226 peak + 79 peak lowres + +Sunday: +gen: 89 peak + 10 peak lowres + +meet peak 409% CPU (100% is 1 CPU), average 69.4% +front peak 70.66% CPU (100% is 1 CPU) +live peak 552% CPU (100% is 1 CPU) average 144% +res peak 81.54% total CPU (each OBS ~250%), mem 7GB used + +so far we have used 1.2TB of transfer this month + + +### Back up the data + + +#### Icecast + +rsync -avze ssh live:’/data/emacsconf-2024-\*’ media:~/2024 + +rsync -avze ssh live:’/data/emacsconf-2024-\*’ . + + +#### BigBlueButton + +rsync -avze ssh root@bbb:/var/bigbluebutton/ bigbluebutton/ + +ssh root@66.175.208.243 ’dd if=/dev/sda bs=5M ’ | dd of=bbb-img-2024-12-08.img status=progress + +The img can be loaded with +sudo losetup –find –show bbb-img-2024-12-08.img +sudo mount /dev/loop10 test +cd test + + +##### Reviewing all the Q&A durations and chat messages + + (let ((default-directory "/home/sacha/proj/emacsconf/2024/backups/bigbluebutton/published/video")) + (mapconcat + (lambda (dir) + (let ((chat (xml-parse-file + (expand-file-name "video.xml" dir))) + (metadata (xml-parse-file + (expand-file-name "metadata.xml" dir)))) + (concat + "- " + (dom-text (car (dom-by-tag metadata 'meetingName))) " (" + (format-seconds "%.2h:%z%.2m:%.2s" + (/ (string-to-number (dom-text (dom-by-tag metadata 'duration))) + 1000)) + ")" + "\n" + (if (dom-by-tag chat 'chattimeline) + (mapconcat (lambda (node) + (concat " - " (dom-attr node 'name) ": " + (dom-attr node 'message) "\n")) + (dom-by-tag chat 'chattimeline) + "") + "") + "") + ) + ) + (directory-files "." nil "-")) + ) + +- Copy IRC logs to cache and therefore to backstage: (ex: 2024-12-07-emacsconf-gen.txt) + + +### Combine the IRC logs and the pads, and add them to the wiki + +(did we e-mail the speakers before or after?) + + +### Process the Q&A recordings and live presentations + +- Save answers videos as –answers.webm. +- From the pad file: emacsconf-extract-copy-pad-to-wiki +- update cache +- Set `QA_PUBLIC` property to t as each Q&A gets processed. +- change status to `TO_INDEX_QA` +- emacsconf-publish-media-files-on-change +- emacsconf-publish-info-pages-for-talk + +Indexing + +- emacsconf-extract-insert-note-with-question-heading +- emacsconf-subed-make-chapter-file-based-on-comments +- change status to `TO_CAPTION_QA` + +Also copy the YouTube comments and IRC comments +emacsconf-extract-irc-anonymize-log + + +### Send thank-you emails to the speakers + + +### Notify emacs-discuss that the Q&A is up + + + + Hello, everyone! + + https://emacsconf.org/2024/talks and https://media.emacsconf.org/2024 + should have the talks and Q&A sessions now, hooray, including chapter + indices for the Q&A. I've also copied over notes from Etherpad and IRC + on the talk pages, and I've e-mailed them to the speakers so they can + update the wiki directly or send in additional answers. + + BigBlueButton audio mixing is kinda iffy, so I'm holding off on + uploading the Q&A videos (and the live talk for secrets) to + YouTube/Toobnix until Leo and other volunteers have a listen to see + whether the audio needs to be tweaked, and maybe even to do some kind of + normalization. Unfortunately, recordings with separate audio tracks per + participant aren't available--it would be cool to figure that out next + year--so some talks might need some manual processing. (Please feel free + to volunteer if that's up your alley!) In the meantime, you can check + out the Q&A videos from the talk pages on the wiki. Please let us know + if there's anything that needs tweaking. + + Enjoy! + + Sacha + + +### Write a report + +Sample: + + +### Send thank-you emails to the volunteers + +emacsconf-mail-template-ask-volunteer-for-mailing-address + + +### Process the lessons learned + + + # General infrastructure + + +## Linode instance sizes + + +### Production + +- front0 shared Linode 32GB +- live0 shared Linode 64GB +- meet Dedicated CPU 16GB + + +### Dormant + +Resize disk to 25GB, copying large files to media if needed + +- nanode front0 +- nanode live0 +- delete meet after downloading all the recordings, or resize down to nanode + + + + +## File uploads + +[psi-4ward/psitransfer: Simple open source self-hosted file sharing solution](https://github.com/psi-4ward/psitransfer) + +Before the conference: + +1. Set `upload_enabled` to `true` in `prod-vars.yml` and run `ansible-playbook -i inventory.yml prod-playbook.yml --tags upload`. +2. Confirm that has the upload interface and works with the password. +3. Use `emacsconf-mail-template-to-all-groups` and `emacsconf-mail-upload-and-backstage-info` to e-mail the upload information to all speakers. + +After the conference: + +1. Set `upload_enabled` to `false` in `prod-vars.yml` and run `ansible-playbook -i inventory.yml prod-playbook.yml --tags upload`. +2. Confirm that shows the in-between page. + + ## Icecast @@ -516,7 +729,7 @@ so that the year is updated in the configuration. This is on live.emacsconf.org and can be restarted with `/etc/init.d/emacsconf restart`. - + ## Shell scripts @@ -630,7 +843,7 @@ Explanation of files: Other files might also have been uploaded by the speaker, such as slides or notes. - + ## IRC web client @@ -652,7 +865,7 @@ connection exhaustion errors. Is that something we can plan now or would you like me to bring it up a few days before the conference? - + ## Etherpad @@ -661,6 +874,10 @@ Success: - [X] pad.emacsconf.org redirects to this year’s entry - [X] Each talk has its own Etherpad +Lessons learned: + +- After 2024: Added pronouns and pronunciation to the pad template since that’s what the hosts will be looking at + @@ -688,7 +905,7 @@ Switching it back to `TO_STREAM` and calling You can generate the index with `emacsconf-publish-update-media`. - + ## Automated IRC announcements @@ -712,8 +929,6 @@ ssh root@bbb cd greenlight-v3 docker compose restart -Does not seem to work in Firefox private browsing; documented at . - ### System audio limitations @@ -728,7 +943,41 @@ BBB says sharing system audio works only if you use Chrome on Mac OS X or Micros Related: - +### Creating talk BBB rooms + +- `docker exec -it greenlight-v3 /bin/bash` +- `bundle exec rails console` +- `user_id = User.find_by_email("sacha@sachachua.com").id` + + (mapconcat (lambda (group) + (format + "Room.create(user_id: user_id, name: \"%s - %s\")\n" + (plist-get (cadr group) :speakers) + (string-join (mapcar (lambda (talk) (plist-get talk :slug)) + (cdr group))))) + (emacsconf-mail-groups (emacsconf-active-talks (emacsconf-get-talk-info))) + "") + +- + +Print out the room IDs with + + Room.all.each { |x| puts x.friendly_id + " " + x.name }; nil + + +### Setting up moderator access codes + + +### Backing up BBB :backup: + +rsync -avze ssh root@bbb:/var/bigbluebutton/ bigbluebutton/ + +ssh root@bbb ’tar zcvf - /var/bigbluebutton /etc/bigbluebutton /root/greenlight-v3 /usr/local/bigbluebutton /usr/share/bbb-web’ > bbb-backup-$(date “+%Y-%m-%d”).tar.gz + +ssh root@bbb ’dd if=/dev/sda bs=5M ’ | dd of=bbb-img-$(date “+%Y-%m-%d”).img status=progress + + + ## Manual IRC announcements @@ -737,7 +986,7 @@ Success: - [X] You can /opall, /conftopic, and /broadcast - + ## Low-res stream @@ -748,7 +997,7 @@ backup on live: - /usr/local/bin/emacsconf-lowres-dev-on-connect - + ## Upload talks to YouTube @@ -784,7 +1033,7 @@ backup on live: (emacsconf-publish-prepare-for-display (emacsconf-get-talk-info))) - + ## Restream to YouTube @@ -802,7 +1051,7 @@ backup on live: docker exec -it greenlight-v2 bundle exec rake user:create["USERNAME","EMAIL","PASSWORD","user"] - + ## When a talk is added after the schedule has already been drafted @@ -826,12 +1075,28 @@ backup on live: ## When a talk is cancelled -1. Update the status to CANCELLED. +1. Update the status to CANCELLED. Update that specific talk with . 2. Rejig the schedule if needed, and update with 3. Update the published schedule with and push the wiki. 4. Regenerate the intros (before, after). , then use in the intros file. -5. Sync the new intros to the /data/emacsconf/shared/$YEAR/assets/intros directory. -6. Regenerate the pads: +5. Resync the cache directory. +6. Consider updating the screenshots for sat-open and sun-open. + + + + +## When a talk changes title + +1. Update the title in conf.org. +2. Delete the FILE\_PREFIX property. +3. Call . +4. Rename existing files with . +5. Update the intro VTT file with the new title. +6. Update the wiki page for the talk. +7. Regenerate the in-between images with +8. Rerecord and recompile the intro, and recompile the intro for the talk after it. +9. +10. @@ -846,7 +1111,15 @@ backup on live: 6. Regenerate talks.json: - + + +## To play the other stream + +- mpv –profile=full +- Change the channel topic to note that discussion can be in the other channel + + + ## When a system is down @@ -859,7 +1132,7 @@ Update the status page: /broadcast may also be helpful - + ## When we need to bring on a last-minute volunteer @@ -868,7 +1141,7 @@ Update the status page: will give them the backstage credentials and links to the index - + ## Checking people in @@ -876,7 +1149,7 @@ will give them the backstage credentials and links to the index /room nick (should be automatic if live, but there just in case) - + ## Ansible -- cgit v1.2.3