From ed2d0d854f36a0c68b9a11d474a5532edfe29dc6 Mon Sep 17 00:00:00 2001 From: Sacha Chua Date: Thu, 20 Oct 2022 11:34:12 -0400 Subject: update tasks --- 2022/organizers-notebook.md | 745 ++++++++++++++++++++++---------------------- 1 file changed, 371 insertions(+), 374 deletions(-) (limited to '2022/organizers-notebook.md') diff --git a/2022/organizers-notebook.md b/2022/organizers-notebook.md index ae5ab8eb..6b17a943 100644 --- a/2022/organizers-notebook.md +++ b/2022/organizers-notebook.md @@ -28,18 +28,13 @@ Time-sensitive tasks: Other projects: -- [Figure out web-based file upload](#upload):needsowner:sachac:ansible: - [Coordinate and help volunteers](#coordinate-volunteers):sachac: - [Prepare for prerecs](#prepare-prerec-process):zaeph: -- [Plan Etherpad use and hosting](#etherpad):sachac:ansible: -- [Update IRC instructions because of multiple tracks](#irc) - [Investigate streaming options, maybe OBS in the cloud](#streaming):corwin: - [Work on the OBS scenes](#obs-scenes):zaeph:corwin: - [Build up the ansible playbook](#ansible):sachac:opal: - [Make something to display between talks](#intro) -- [Move scheduling and publishing code to Emacs on a VPS so that other people can help out](#publishing-sched):sachac: - [Create protocol for live Q&A](#live-protocol):zaeph: -- [Set up BBB rooms and update conf.org](#create-bbb):sachac: - [Add nice-to-have stuff to prepare.md](#extra-prepare):zaeph: - [Find volunteers for tech-checks](#tech-checks):zaeph: - [Update viewing instructions and watch pages](#write-viewing):sachac: @@ -51,6 +46,8 @@ Other projects: - [Figure out why ikiwiki is slow](#ikiwiki-regex):infra:wiki: - [Add category tags and possibly links between talks across 2022 and all previous years](#link-pages):needsowner:wiki: - [Turn off file upload service on media.emacsconf.org](#upload-off):sachac:infra: +- [Plan Etherpad use and hosting](#etherpad):sachac:ansible: +- [Set up BBB rooms and update conf.org](#create-bbb):sachac: @@ -512,18 +509,13 @@ Sacha Chua # Projects and other long-running tasks -- [Figure out web-based file upload](#upload):needsowner:sachac:ansible: - [Coordinate and help volunteers](#coordinate-volunteers):sachac: - [Prepare for prerecs](#prepare-prerec-process):zaeph: -- [Plan Etherpad use and hosting](#etherpad):sachac:ansible: -- [Update IRC instructions because of multiple tracks](#irc) - [Investigate streaming options, maybe OBS in the cloud](#streaming):corwin: - [Work on the OBS scenes](#obs-scenes):zaeph:corwin: - [Build up the ansible playbook](#ansible):sachac:opal: - [Make something to display between talks](#intro) -- [Move scheduling and publishing code to Emacs on a VPS so that other people can help out](#publishing-sched):sachac: - [Create protocol for live Q&A](#live-protocol):zaeph: -- [Set up BBB rooms and update conf.org](#create-bbb):sachac: - [Add nice-to-have stuff to prepare.md](#extra-prepare):zaeph: - [Find volunteers for tech-checks](#tech-checks):zaeph: - [Update viewing instructions and watch pages](#write-viewing):sachac: @@ -535,161 +527,12 @@ Sacha Chua - [Figure out why ikiwiki is slow](#ikiwiki-regex):infra:wiki: - [Add category tags and possibly links between talks across 2022 and all previous years](#link-pages):needsowner:wiki: - [Turn off file upload service on media.emacsconf.org](#upload-off):sachac:infra: +- [Plan Etherpad use and hosting](#etherpad):sachac:ansible: +- [Set up BBB rooms and update conf.org](#create-bbb):sachac: Ordered by importance. - - -## DONE Figure out web-based file upload :needsowner:sachac:ansible: - -zaeph: I can work on it, but I’m not experienced in this domain, so I’d prefer to be a back-up. -task is currently with bandali - -Lesson learned from last year: “Since people kept running into ftp -problems, we might want to set up a web-frontend next year to minimise -problems.” - -Maybe we could ask some of the volunteers who wanted to help us with -the infra? It shouldn’t be complicated to deploy a ready-made -solution. - -“file drop” is a common keyword for looking for information. -Considerations: - -- Probably run it on media.emacsconf.org -- Bonus features: - - password-protected or hidden behind some kind of authentication or hidden behind some kind of URL, so we don’t have to worry too much about spam - - extra points for sending speakers links to upload to specific folders so that we can separate resources by talk - - Resumable uploads would be good, since some speakers had a hard time with unreliable connections -- What other conferences do: - - LibrePlanet uses plain FTP and recommends FileZilla - - FOSSDEM uses Pentabarf to receive uploads - - DebConf uses SReview(?) - - FOSSGIS uses Seafile -- Some options: - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
ProjectDockerBusterBaseNotes
https://www.projectsend.org/  php+mysql 
https://github.com/pomf/pomf  php+mysql 
https://github.com/psi-4ward/psitransferofficial nodecan set upload password, resumable; data volume needs uid 1000
https://github.com/epoupon/fileshelterofficialppaC++ 
https://github.com/mickael-kerjean/filestashofficial  general FTP client
https://gitlab.com/moejo42/Jirafeauofficial php 
https://github.com/YouTransfer/YouTransferofficial nodelooking for maint
https://github.com/dutchcoders/transfer.sh/    
https://github.com/eikek/sharry    
- - - NextCloud - - anon FTP upload currently goes to *srv/ftp/anon* on media.emacsconf.org - - - - Maybe I’ll do psitransfer as a direct install - - -### DONE Create 2022/upload.md with the same workflow as last year for a start - - -### DONE Implement new workflow - - ## TODO Coordinate and help volunteers :sachac: @@ -791,124 +634,6 @@ We need time after the prerecs get submitted to: Make changes in [As prerecorded talks come in](#prerec-process) - - -## INPROGRESS Plan Etherpad use and hosting :sachac:ansible: - -- Relevant links: - - Per-pad, nicely structured info with abstract, watching information, etc. CarpentryCon 2022 Schedule • CarpentryCon 2022 - - One pad per session: - - [Etherpad integration in Jitsi ](https://community.jitsi.org/t/tutorial-etherpad-integration-in-jitsi-meetings/99697) - -- Good: One pad per session -- Better: Some kind of monitoring so that we can link to the pads or embed - the pads even before the conference starts while not risking too - much vandalism -- Best: - - Pad can be easily regenerated from Emacs Lisp with a check to see if people have been adding to it - - Pad links to next talks - -Where should we host this? - -- live0: gets scaled up the most, lots of people connect to it for the conference, didn’t hit performance constraints last time -- front0: lower risk of interfering with stream -- bbb: will already be put on strain with the concurrent streams (test showed it was stable with up to 10 concurrent video streams and 40 total participants) -- opal’s: no news from owner, but beefy server that we used for reencode last year -- new Linode (probably 4GB or 8GB shared CPU): can easily be spun up, yay Ansible - -[Ansible notes](#ansible) - -Consider if we need extra scaling beyond being on a beefy live0? - -- Scale calculator: - - assuming 3 concurrent authors, 100 lurkers per pad, 3 concurrent pads - - 1 core, 4GB RAM, bandwidth Mb/s: 7.488 -- -- - -Will need to try this again when we resize nodes. Probably just the extra memory will be enough and the CPU use from node won’t step on the streaming, but not sure - - -### DONE Use the API to create pages based on all the slugs - - -### CANCELLED Figure out monitoring; maybe get everything daily and commit to git repo? - -History or regeneration will be fine. We’ll probably link to it shortly before the event, and we can also turn off the service until we’re ready. - - -### DONE Generate talk-specific pad content - - -### DONE Set up nginx reverse proxy - - -### DONE Load-test - -tl;dr: Either a separate 4GB Linode or being on live0 will probably be -okay, but I’m not 100% sure due to the limitations of my load-testing -setup. I don’t know if we need to shard by pad. - -It looks like etherpad-load-test tends to max out at ~40 connections -on a specific node. I used GNU Parallel to run the loadtesting tool -against a 4GB Linode instance (shared CPU) from five nodes at the same -time (my X220, my 2GB Linode instance, front0, and the node with the -pad), and they all reached about 35-45 clients before failure (not -updating within 100ms). - - echo 'node node_modules/etherpad-load-test/app.js http://170.187.195.5:9001 -d 120 > ~/output.txt' | parallel -J loadtest -j 1 --onall --verbose --tag - - echo 'grep Local ~/output.txt | tail -1' | parallel -J loadtest -j 1 --onall --tag - -Files were created at roughly the same time, so the max loads probably -overlapped. It would be good to have finer control over the -etherpad-load-test tool. Haven’t figured out how to properly use `-l` -and `-a` yet. - - echo 'stat -c %y ~/output.txt' | parallel -J loadtest -j 1 --onall --tag - -CPU graph went only up to 5%. Network max was 2.36 Mb/s in, 1.25 Mb/s out. -looking at top, CPU seems to go only up to about 12% or so. - - -### DONE Set up pad.emacsconf.org to point to live0.emacsconf.org :bandali: - - -### DONE Set up letsencrypt - -Waiting for DNS -add to /etc/dehydrated/domains.txt -sudo sh -x /etc/cron.daily/renew-https-cert-local - - -### TODO Link to pad from talk page - - -### DONE Add links to general conference pad - -for collecting feedback - - -### TODO Prototype shift pads for easier scrolling - -if we can get the anchor plugin - - -### DONE Be able to fall back to wikimedia if necessary, maybe with nginx redirects - - -### DONE Export pad initial content HTML to make it easier to reimport into wikimedia or elsewhere - -`emacsconf-pad-export-initial-content-for-all-talks` - - - - -## DONE Update IRC instructions because of multiple tracks - -Added to watch pages - - ## TODO Investigate streaming options, maybe OBS in the cloud :corwin: @@ -1009,28 +734,6 @@ See break commercials - - -## DONE Move scheduling and publishing code to Emacs on a VPS so that other people can help out :sachac: - -Ideal: - -- Update pages with watching information, additional resources, etc. as talks go live -- Update the schedule as needed (cancelled or reordered talks, etc.) - -Where: - -- front? my own VPS? - - Nice if there’s an Ansible playbook - - sachac’s notes: - -- probably good to set it up on front - -It’s now on front. - - ## TODO Create protocol for live Q&A :zaeph: @@ -1044,28 +747,6 @@ Comments: - It would be nice to have a protocol with speakers to tell us that they’re uncomfortable with a question, just so that we can find a smooth way to skip it. - - -## DONE Set up BBB rooms and update conf.org :sachac: - -1. Log on to bbb.emacsverse.org as an admin. -2. Create a room. Enable **Mute users when they join**. - - The code below doesn’t quite work, but might be a good starting point for future automation. - - (setq list (seq-drop (emacsconf-bbb-room-title-list) 3)) ; skip some if needed - (progn - (setq name (pop list)) - (kill-new (format "name=\"%s\";$('#create-room-block').click();$('#create-room-name').val(name);$('#room_mute_on_join').click();$('.create-room-button').click();\n" - name)) - (sleep-for 1) - (shell-command "xdotool key alt+Tab sleep 3 key ctrl+v sleep 1 key Return")) - -console.log(JSON.stringify([…document.querySelectorAll(’.delete-room’)].map((o) => { return { name: o.getAttribute(’data-name’), path: o.getAttribute(’data-path’) }}).filter((o) => o.name.match(*^ec22*)))) - -see conf.org for the rest of the process - - ## STANDBY Add nice-to-have stuff to prepare.md :zaeph: @@ -1204,75 +885,211 @@ so that people on other platforms can come across EmacsConf while true; do ffmpeg -loglevel 24 -i http://localhost:8000/main.webm -cluster_size_limit 2M -cluster_time_limit 5100 -b:v 1M -crf 30 -g 125 -deadline good -threads 4 -vcodec libx264 -acodec libmp3lame -f flv $MOUNT; done - + + +## TODO Ask speakers for bios or support nudges to include on their talk pages + +maybe after we get the prerecs +ex: liberapay, patreon, anyone looking for a job, etc. + + + + +## BLOCKED Do a dry run + +zaeph is available on 2022-11-06 + +- Streams + - [ ] General stream + - [ ] Development stream + - [ ] 480p streams + - [ ] Toobnix stream + - [ ] YouTube stream +- Scenes + - [ ] Intro + - [ ] Prerec + - [ ] Q&A: live + - [ ] Q&A: IRC + - [ ] Q&A: pad +- [ ] IRC channels + - [ ] Announce + - [ ] Question handling + - [ ] Timers +- [ ] Watching pages + - [ ] Before launch + - [ ] Streaming + - [ ] Emergency announcement +- [ ] Pads +- [ ] Wiki + - [ ] Schedule update + - [ ] Message on the schedule + - [ ] Prerec live +- [ ] Wind everything down + + + + +## TODO Figure out why ikiwiki is slow :infra:wiki: + +complex regular expression issues? +should the captions be outside the wiki? + + + + +## TODO Add category tags and possibly links between talks across 2022 and all previous years :needsowner:wiki: + +- Prerequisite: Can edit wiki pages () +- Goal: Make it easier for people to discover interesting related talks + +- List of talks for EmacsConf 2022: +- List of talks: +- List of categories: +- To add something to a category, add `\[[!taglink CategoryName]]` to the bottom of its talk page (ex: 2022/talks/maint.md is in CategoryCommunity) +- You can create new categories by making up new CategoryNames. +- You can also link to a talk with a link like this: `\[[/2022/talks/maint|Maintaining the Maintainers: Attribution as an Economic Model for Open Source]]` + You can make a new heading called `# Related talks` + + + + +## TODO Turn off file upload service on media.emacsconf.org :sachac:infra: + + + + +## INPROGRESS Plan Etherpad use and hosting :sachac:ansible: + +- Relevant links: + - Per-pad, nicely structured info with abstract, watching information, etc. CarpentryCon 2022 Schedule • CarpentryCon 2022 + - One pad per session: + - [Etherpad integration in Jitsi ](https://community.jitsi.org/t/tutorial-etherpad-integration-in-jitsi-meetings/99697) + +- Good: One pad per session +- Better: Some kind of monitoring so that we can link to the pads or embed + the pads even before the conference starts while not risking too + much vandalism +- Best: + - Pad can be easily regenerated from Emacs Lisp with a check to see if people have been adding to it + - Pad links to next talks + +Where should we host this? + +- live0: gets scaled up the most, lots of people connect to it for the conference, didn’t hit performance constraints last time +- front0: lower risk of interfering with stream +- bbb: will already be put on strain with the concurrent streams (test showed it was stable with up to 10 concurrent video streams and 40 total participants) +- opal’s: no news from owner, but beefy server that we used for reencode last year +- new Linode (probably 4GB or 8GB shared CPU): can easily be spun up, yay Ansible + +[Ansible notes](#ansible) + +Consider if we need extra scaling beyond being on a beefy live0? + +- Scale calculator: + - assuming 3 concurrent authors, 100 lurkers per pad, 3 concurrent pads + - 1 core, 4GB RAM, bandwidth Mb/s: 7.488 +- +- + +Will need to try this again when we resize nodes. Probably just the extra memory will be enough and the CPU use from node won’t step on the streaming, but not sure + + +### DONE Use the API to create pages based on all the slugs + + +### CANCELLED Figure out monitoring; maybe get everything daily and commit to git repo? + +History or regeneration will be fine. We’ll probably link to it shortly before the event, and we can also turn off the service until we’re ready. + + +### DONE Generate talk-specific pad content + + +### DONE Set up nginx reverse proxy + + +### DONE Load-test + +tl;dr: Either a separate 4GB Linode or being on live0 will probably be +okay, but I’m not 100% sure due to the limitations of my load-testing +setup. I don’t know if we need to shard by pad. + +It looks like etherpad-load-test tends to max out at ~40 connections +on a specific node. I used GNU Parallel to run the loadtesting tool +against a 4GB Linode instance (shared CPU) from five nodes at the same +time (my X220, my 2GB Linode instance, front0, and the node with the +pad), and they all reached about 35-45 clients before failure (not +updating within 100ms). + + echo 'node node_modules/etherpad-load-test/app.js http://170.187.195.5:9001 -d 120 > ~/output.txt' | parallel -J loadtest -j 1 --onall --verbose --tag + + echo 'grep Local ~/output.txt | tail -1' | parallel -J loadtest -j 1 --onall --tag + +Files were created at roughly the same time, so the max loads probably +overlapped. It would be good to have finer control over the +etherpad-load-test tool. Haven’t figured out how to properly use `-l` +and `-a` yet. + + echo 'stat -c %y ~/output.txt' | parallel -J loadtest -j 1 --onall --tag + +CPU graph went only up to 5%. Network max was 2.36 Mb/s in, 1.25 Mb/s out. +looking at top, CPU seems to go only up to about 12% or so. + + +### DONE Set up pad.emacsconf.org to point to live0.emacsconf.org :bandali: + + +### DONE Set up letsencrypt + +Waiting for DNS +add to /etc/dehydrated/domains.txt +sudo sh -x /etc/cron.daily/renew-https-cert-local + + +### TODO Link to pad from talk page -## TODO Ask speakers for bios or support nudges to include on their talk pages -maybe after we get the prerecs -ex: liberapay, patreon, anyone looking for a job, etc. +### DONE Add links to general conference pad +for collecting feedback - -## BLOCKED Do a dry run +### TODO Prototype shift pads for easier scrolling -zaeph is available on 2022-11-06 +if we can get the anchor plugin -- Streams - - [ ] General stream - - [ ] Development stream - - [ ] 480p streams - - [ ] Toobnix stream - - [ ] YouTube stream -- Scenes - - [ ] Intro - - [ ] Prerec - - [ ] Q&A: live - - [ ] Q&A: IRC - - [ ] Q&A: pad -- [ ] IRC channels - - [ ] Announce - - [ ] Question handling - - [ ] Timers -- [ ] Watching pages - - [ ] Before launch - - [ ] Streaming - - [ ] Emergency announcement -- [ ] Pads -- [ ] Wiki - - [ ] Schedule update - - [ ] Message on the schedule - - [ ] Prerec live -- [ ] Wind everything down +### DONE Be able to fall back to wikimedia if necessary, maybe with nginx redirects - -## TODO Figure out why ikiwiki is slow :infra:wiki: +### DONE Export pad initial content HTML to make it easier to reimport into wikimedia or elsewhere -complex regular expression issues? -should the captions be outside the wiki? +`emacsconf-pad-export-initial-content-for-all-talks` - + -## TODO Add category tags and possibly links between talks across 2022 and all previous years :needsowner:wiki: +## DONE Set up BBB rooms and update conf.org :sachac: -- Prerequisite: Can edit wiki pages () -- Goal: Make it easier for people to discover interesting related talks +1. Log on to bbb.emacsverse.org as an admin. +2. Create a room. Enable **Mute users when they join**. + + The code below doesn’t quite work, but might be a good starting point for future automation. + + (setq list (seq-drop (emacsconf-bbb-room-title-list) 3)) ; skip some if needed + (progn + (setq name (pop list)) + (kill-new (format "name=\"%s\";$('#create-room-block').click();$('#create-room-name').val(name);$('#room_mute_on_join').click();$('.create-room-button').click();\n" + name)) + (sleep-for 1) + (shell-command "xdotool key alt+Tab sleep 3 key ctrl+v sleep 1 key Return")) -- List of talks for EmacsConf 2022: -- List of talks: -- List of categories: -- To add something to a category, add `\[[!taglink CategoryName]]` to the bottom of its talk page (ex: 2022/talks/maint.md is in CategoryCommunity) -- You can create new categories by making up new CategoryNames. -- You can also link to a talk with a link like this: `\[[/2022/talks/maint|Maintaining the Maintainers: Attribution as an Economic Model for Open Source]]` - You can make a new heading called `# Related talks` +console.log(JSON.stringify([…document.querySelectorAll(’.delete-room’)].map((o) => { return { name: o.getAttribute(’data-name’), path: o.getAttribute(’data-path’) }}).filter((o) => o.name.match(*^ec22*)))) +see conf.org for the rest of the process - -## TODO Turn off file upload service on media.emacsconf.org :sachac:infra: +### TODO Add volunteers to the BBB rooms @@ -2490,7 +2307,7 @@ Probably focus on grabbing the audio first and seeing what’s worth keeping Make a table of the form - +
@@ -3250,6 +3067,186 @@ Sacha It would be nice to have track-specific icals as well. + + +## DONE Figure out web-based file upload :needsowner:sachac:ansible: + +zaeph: I can work on it, but I’m not experienced in this domain, so I’d prefer to be a back-up. +task is currently with bandali + +Lesson learned from last year: “Since people kept running into ftp +problems, we might want to set up a web-frontend next year to minimise +problems.” + +Maybe we could ask some of the volunteers who wanted to help us with +the infra? It shouldn’t be complicated to deploy a ready-made +solution. + +“file drop” is a common keyword for looking for information. +Considerations: + +- Probably run it on media.emacsconf.org +- Bonus features: + - password-protected or hidden behind some kind of authentication or hidden behind some kind of URL, so we don’t have to worry too much about spam + - extra points for sending speakers links to upload to specific folders so that we can separate resources by talk + - Resumable uploads would be good, since some speakers had a hard time with unreliable connections +- What other conferences do: + - LibrePlanet uses plain FTP and recommends FileZilla + - FOSSDEM uses Pentabarf to receive uploads + - DebConf uses SReview(?) + - FOSSGIS uses Seafile +- Some options: + +
+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
ProjectDockerBusterBaseNotes
https://www.projectsend.org/  php+mysql 
https://github.com/pomf/pomf  php+mysql 
https://github.com/psi-4ward/psitransferofficial nodecan set upload password, resumable; data volume needs uid 1000
https://github.com/epoupon/fileshelterofficialppaC++ 
https://github.com/mickael-kerjean/filestashofficial  general FTP client
https://gitlab.com/moejo42/Jirafeauofficial php 
https://github.com/YouTransfer/YouTransferofficial nodelooking for maint
https://github.com/dutchcoders/transfer.sh/    
https://github.com/eikek/sharry    
+ + - NextCloud + + anon FTP upload currently goes to *srv/ftp/anon* on media.emacsconf.org + + + + Maybe I’ll do psitransfer as a direct install + + +### DONE Create 2022/upload.md with the same workflow as last year for a start + + +### DONE Implement new workflow + + + + +## DONE Update IRC instructions because of multiple tracks + +Added to watch pages + + + + +## DONE Move scheduling and publishing code to Emacs on a VPS so that other people can help out :sachac: + +Ideal: + +- Update pages with watching information, additional resources, etc. as talks go live +- Update the schedule as needed (cancelled or reordered talks, etc.) + +Where: + +- front? my own VPS? + + Nice if there’s an Ansible playbook + + sachac’s notes: + +- probably good to set it up on front + +It’s now on front. + + # Communications -- cgit v1.2.3