diff options
| author | Sacha Chua <sacha@sachachua.com> | 2022-10-20 11:34:12 -0400 |
|---|---|---|
| committer | Sacha Chua <sacha@sachachua.com> | 2022-10-20 11:34:12 -0400 |
| commit | ed2d0d854f36a0c68b9a11d474a5532edfe29dc6 (patch) | |
| tree | 3d9b9ef73790c502bd058340146aef88e3dd0716 | |
| parent | 6e3169a9d5a310a58bea4a6756e0dfce1e9975a1 (diff) | |
| download | emacsconf-wiki-ed2d0d854f36a0c68b9a11d474a5532edfe29dc6.tar.xz emacsconf-wiki-ed2d0d854f36a0c68b9a11d474a5532edfe29dc6.zip | |
update tasks
| -rw-r--r-- | 2022/organizers-notebook.md | 645 | ||||
| -rw-r--r-- | 2022/organizers-notebook/index.org | 438 |
2 files changed, 540 insertions, 543 deletions
diff --git a/2022/organizers-notebook.md b/2022/organizers-notebook.md index ae5ab8eb..6b17a943 100644 --- a/2022/organizers-notebook.md +++ b/2022/organizers-notebook.md @@ -28,18 +28,13 @@ Time-sensitive tasks: Other projects: -- [Figure out web-based file upload](#upload):needsowner:sachac:ansible: - [Coordinate and help volunteers](#coordinate-volunteers):sachac: - [Prepare for prerecs](#prepare-prerec-process):zaeph: -- [Plan Etherpad use and hosting](#etherpad):sachac:ansible: -- [Update IRC instructions because of multiple tracks](#irc) - [Investigate streaming options, maybe OBS in the cloud](#streaming):corwin: - [Work on the OBS scenes](#obs-scenes):zaeph:corwin: - [Build up the ansible playbook](#ansible):sachac:opal: - [Make something to display between talks](#intro) -- [Move scheduling and publishing code to Emacs on a VPS so that other people can help out](#publishing-sched):sachac: - [Create protocol for live Q&A](#live-protocol):zaeph: -- [Set up BBB rooms and update conf.org](#create-bbb):sachac: - [Add nice-to-have stuff to prepare.md](#extra-prepare):zaeph: - [Find volunteers for tech-checks](#tech-checks):zaeph: - [Update viewing instructions and watch pages](#write-viewing):sachac: @@ -51,6 +46,8 @@ Other projects: - [Figure out why ikiwiki is slow](#ikiwiki-regex):infra:wiki: - [Add category tags and possibly links between talks across 2022 and all previous years](#link-pages):needsowner:wiki: - [Turn off file upload service on media.emacsconf.org](#upload-off):sachac:infra: +- [Plan Etherpad use and hosting](#etherpad):sachac:ansible: +- [Set up BBB rooms and update conf.org](#create-bbb):sachac: <a id="cadence"></a> @@ -512,18 +509,13 @@ Sacha Chua # Projects and other long-running tasks -- [Figure out web-based file upload](#upload):needsowner:sachac:ansible: - [Coordinate and help volunteers](#coordinate-volunteers):sachac: - [Prepare for prerecs](#prepare-prerec-process):zaeph: -- [Plan Etherpad use and hosting](#etherpad):sachac:ansible: -- [Update IRC instructions because of multiple tracks](#irc) - [Investigate streaming options, maybe OBS in the cloud](#streaming):corwin: - [Work on the OBS scenes](#obs-scenes):zaeph:corwin: - [Build up the ansible playbook](#ansible):sachac:opal: - [Make something to display between talks](#intro) -- [Move scheduling and publishing code to Emacs on a VPS so that other people can help out](#publishing-sched):sachac: - [Create protocol for live Q&A](#live-protocol):zaeph: -- [Set up BBB rooms and update conf.org](#create-bbb):sachac: - [Add nice-to-have stuff to prepare.md](#extra-prepare):zaeph: - [Find volunteers for tech-checks](#tech-checks):zaeph: - [Update viewing instructions and watch pages](#write-viewing):sachac: @@ -535,161 +527,12 @@ Sacha Chua - [Figure out why ikiwiki is slow](#ikiwiki-regex):infra:wiki: - [Add category tags and possibly links between talks across 2022 and all previous years](#link-pages):needsowner:wiki: - [Turn off file upload service on media.emacsconf.org](#upload-off):sachac:infra: +- [Plan Etherpad use and hosting](#etherpad):sachac:ansible: +- [Set up BBB rooms and update conf.org](#create-bbb):sachac: Ordered by importance. -<a id="upload"></a> - -## DONE Figure out web-based file upload :needsowner:sachac:ansible: - -zaeph: I can work on it, but I’m not experienced in this domain, so I’d prefer to be a back-up. -task is currently with bandali - -Lesson learned from last year: “Since people kept running into ftp -problems, we might want to set up a web-frontend next year to minimise -problems.” - -Maybe we could ask some of the volunteers who wanted to help us with -the infra? It shouldn’t be complicated to deploy a ready-made -solution. - -“file drop” is a common keyword for looking for information. -Considerations: - -- Probably run it on media.emacsconf.org -- Bonus features: - - password-protected or hidden behind some kind of authentication or hidden behind some kind of URL, so we don’t have to worry too much about spam - - extra points for sending speakers links to upload to specific folders so that we can separate resources by talk - - Resumable uploads would be good, since some speakers had a hard time with unreliable connections -- What other conferences do: - - LibrePlanet uses plain FTP and recommends FileZilla <https://libreplanet.org/wiki/Video_upload_instructions> - - FOSSDEM uses Pentabarf to receive uploads <https://archive.fosdem.org/2022/manuals/program/speaker/> - - DebConf uses SReview(?) - - FOSSGIS uses Seafile <https://vmx.cx/cgi-bin/blog/index.cgi/video-uploads-for-an-online-conference%3A2021-06-12%3Aen%2Cconference%2Cgeo> -- Some options: - - <table border="2" cellspacing="0" cellpadding="6" rules="groups" frame="hsides"> - - - <colgroup> - <col class="org-left" /> - - <col class="org-left" /> - - <col class="org-left" /> - - <col class="org-left" /> - - <col class="org-left" /> - </colgroup> - <tbody> - <tr> - <td class="org-left">Project</td> - <td class="org-left">Docker</td> - <td class="org-left">Buster</td> - <td class="org-left">Base</td> - <td class="org-left">Notes</td> - </tr> - - - <tr> - <td class="org-left"><a href="https://www.projectsend.org/">https://www.projectsend.org/</a></td> - <td class="org-left"> </td> - <td class="org-left"> </td> - <td class="org-left">php+mysql</td> - <td class="org-left"> </td> - </tr> - - - <tr> - <td class="org-left"><a href="https://github.com/pomf/pomf">https://github.com/pomf/pomf</a></td> - <td class="org-left"> </td> - <td class="org-left"> </td> - <td class="org-left">php+mysql</td> - <td class="org-left"> </td> - </tr> - - - <tr> - <td class="org-left"><a href="https://github.com/psi-4ward/psitransfer">https://github.com/psi-4ward/psitransfer</a></td> - <td class="org-left">official</td> - <td class="org-left"> </td> - <td class="org-left">node</td> - <td class="org-left">can set upload password, resumable; data volume needs uid 1000</td> - </tr> - - - <tr> - <td class="org-left"><a href="https://github.com/epoupon/fileshelter">https://github.com/epoupon/fileshelter</a></td> - <td class="org-left">official</td> - <td class="org-left">ppa</td> - <td class="org-left">C++</td> - <td class="org-left"> </td> - </tr> - - - <tr> - <td class="org-left"><a href="https://github.com/mickael-kerjean/filestash">https://github.com/mickael-kerjean/filestash</a></td> - <td class="org-left">official</td> - <td class="org-left"> </td> - <td class="org-left"> </td> - <td class="org-left">general FTP client</td> - </tr> - - - <tr> - <td class="org-left"><a href="https://gitlab.com/moejo42/Jirafeau">https://gitlab.com/moejo42/Jirafeau</a></td> - <td class="org-left">official</td> - <td class="org-left"> </td> - <td class="org-left">php</td> - <td class="org-left"> </td> - </tr> - - - <tr> - <td class="org-left"><a href="https://github.com/YouTransfer/YouTransfer">https://github.com/YouTransfer/YouTransfer</a></td> - <td class="org-left">official</td> - <td class="org-left"> </td> - <td class="org-left">node</td> - <td class="org-left">looking for maint</td> - </tr> - - - <tr> - <td class="org-left"><a href="https://github.com/dutchcoders/transfer.sh/">https://github.com/dutchcoders/transfer.sh/</a></td> - <td class="org-left"> </td> - <td class="org-left"> </td> - <td class="org-left"> </td> - <td class="org-left"> </td> - </tr> - - - <tr> - <td class="org-left"><a href="https://github.com/eikek/sharry">https://github.com/eikek/sharry</a></td> - <td class="org-left"> </td> - <td class="org-left"> </td> - <td class="org-left"> </td> - <td class="org-left"> </td> - </tr> - </tbody> - </table> - - - NextCloud - - anon FTP upload currently goes to *srv/ftp/anon* on media.emacsconf.org - - <file:///ssh:media|sudo> - - Maybe I’ll do psitransfer as a direct install - - -### DONE Create 2022/upload.md with the same workflow as last year for a start - - -### DONE Implement new workflow - - <a id="coordinate-volunteers"></a> ## TODO Coordinate and help volunteers :sachac: @@ -791,124 +634,6 @@ We need time after the prerecs get submitted to: Make changes in [As prerecorded talks come in](#prerec-process) -<a id="etherpad"></a> - -## INPROGRESS Plan Etherpad use and hosting :sachac:ansible: - -- Relevant links: - - Per-pad, nicely structured info with abstract, watching information, etc. CarpentryCon 2022 Schedule • CarpentryCon 2022 <https://2022.carpentrycon.org/> - - One pad per session: <https://meta.wikimedia.org/wiki/Arctic_Knot_Conference_2021/Program#Friday,_June_25th> - - [Etherpad integration in Jitsi ](https://community.jitsi.org/t/tutorial-etherpad-integration-in-jitsi-meetings/99697) - -- Good: One pad per session -- Better: Some kind of monitoring so that we can link to the pads or embed - the pads even before the conference starts while not risking too - much vandalism -- Best: - - Pad can be easily regenerated from Emacs Lisp with a check to see if people have been adding to it - - Pad links to next talks - -Where should we host this? - -- live0: gets scaled up the most, lots of people connect to it for the conference, didn’t hit performance constraints last time -- front0: lower risk of interfering with stream -- bbb: will already be put on strain with the concurrent streams (test showed it was stable with up to 10 concurrent video streams and 40 total participants) -- opal’s: no news from owner, but beefy server that we used for reencode last year -- new Linode (probably 4GB or 8GB shared CPU): can easily be spun up, yay Ansible - -[Ansible notes](#ansible) - -Consider if we need extra scaling beyond being on a beefy live0? - -- Scale calculator: <https://scale.etherpad.org/> - - assuming 3 concurrent authors, 100 lurkers per pad, 3 concurrent pads - - 1 core, 4GB RAM, bandwidth Mb/s: 7.488 -- <https://mclear.co.uk/2021/09/08/deploying-etherpad-at-scale-in-one-minute/> -- <https://github.com/ether/etherpad-load-test> - -Will need to try this again when we resize nodes. Probably just the extra memory will be enough and the CPU use from node won’t step on the streaming, but not sure - - -### DONE Use the API to create pages based on all the slugs - - -### CANCELLED Figure out monitoring; maybe get everything daily and commit to git repo? - -History or regeneration will be fine. We’ll probably link to it shortly before the event, and we can also turn off the service until we’re ready. - - -### DONE Generate talk-specific pad content - - -### DONE Set up nginx reverse proxy - - -### DONE Load-test - -tl;dr: Either a separate 4GB Linode or being on live0 will probably be -okay, but I’m not 100% sure due to the limitations of my load-testing -setup. I don’t know if we need to shard by pad. - -It looks like etherpad-load-test tends to max out at ~40 connections -on a specific node. I used GNU Parallel to run the loadtesting tool -against a 4GB Linode instance (shared CPU) from five nodes at the same -time (my X220, my 2GB Linode instance, front0, and the node with the -pad), and they all reached about 35-45 clients before failure (not -updating within 100ms). - - echo 'node node_modules/etherpad-load-test/app.js http://170.187.195.5:9001 -d 120 > ~/output.txt' | parallel -J loadtest -j 1 --onall --verbose --tag - - echo 'grep Local ~/output.txt | tail -1' | parallel -J loadtest -j 1 --onall --tag - -Files were created at roughly the same time, so the max loads probably -overlapped. It would be good to have finer control over the -etherpad-load-test tool. Haven’t figured out how to properly use `-l` -and `-a` yet. - - echo 'stat -c %y ~/output.txt' | parallel -J loadtest -j 1 --onall --tag - -CPU graph went only up to 5%. Network max was 2.36 Mb/s in, 1.25 Mb/s out. -looking at top, CPU seems to go only up to about 12% or so. - - -### DONE Set up pad.emacsconf.org to point to live0.emacsconf.org :bandali: - - -### DONE Set up letsencrypt - -Waiting for DNS -add to /etc/dehydrated/domains.txt -sudo sh -x /etc/cron.daily/renew-https-cert-local - - -### TODO Link to pad from talk page - - -### DONE Add links to general conference pad - -for collecting feedback - - -### TODO Prototype shift pads for easier scrolling - -if we can get the anchor plugin - - -### DONE Be able to fall back to wikimedia if necessary, maybe with nginx redirects - - -### DONE Export pad initial content HTML to make it easier to reimport into wikimedia or elsewhere - -`emacsconf-pad-export-initial-content-for-all-talks` - - -<a id="irc"></a> - -## DONE Update IRC instructions because of multiple tracks - -Added to watch pages - - <a id="streaming"></a> ## TODO Investigate streaming options, maybe OBS in the cloud :corwin: @@ -1009,28 +734,6 @@ See break commercials <https://www.collabmagazine.com/organizing-a-multi-track-virtual-conference-with-microsoft-teams-live-events-a-technical-playbook-and-lessons-learned/> -<a id="publishing-sched"></a> - -## DONE Move scheduling and publishing code to Emacs on a VPS so that other people can help out :sachac: - -Ideal: - -- Update pages with watching information, additional resources, etc. as talks go live -- Update the schedule as needed (cancelled or reordered talks, etc.) - -Where: - -- front? my own VPS? - - Nice if there’s an Ansible playbook - - sachac’s notes: - <file:///home/sacha/code/docker/emacsconf-publish/> -- probably good to set it up on front - -It’s now on front. - - <a id="live-protocol"></a> ## TODO Create protocol for live Q&A :zaeph: @@ -1044,28 +747,6 @@ Comments: - It would be nice to have a protocol with speakers to tell us that they’re uncomfortable with a question, just so that we can find a smooth way to skip it. -<a id="create-bbb"></a> - -## DONE Set up BBB rooms and update conf.org :sachac: - -1. Log on to bbb.emacsverse.org as an admin. -2. Create a room. Enable **Mute users when they join**. - - The code below doesn’t quite work, but might be a good starting point for future automation. - - (setq list (seq-drop (emacsconf-bbb-room-title-list) 3)) ; skip some if needed - (progn - (setq name (pop list)) - (kill-new (format "name=\"%s\";$('#create-room-block').click();$('#create-room-name').val(name);$('#room_mute_on_join').click();$('.create-room-button').click();\n" - name)) - (sleep-for 1) - (shell-command "xdotool key alt+Tab sleep 3 key ctrl+v sleep 1 key Return")) - -console.log(JSON.stringify([…document.querySelectorAll(’.delete-room’)].map((o) => { return { name: o.getAttribute(’data-name’), path: o.getAttribute(’data-path’) }}).filter((o) => o.name.match(*^ec22*)))) - -see conf.org for the rest of the process - - <a id="extra-prepare"></a> ## STANDBY Add nice-to-have stuff to prepare.md :zaeph: @@ -1275,6 +956,142 @@ should the captions be outside the wiki? ## TODO Turn off file upload service on media.emacsconf.org :sachac:infra: +<a id="etherpad"></a> + +## INPROGRESS Plan Etherpad use and hosting :sachac:ansible: + +- Relevant links: + - Per-pad, nicely structured info with abstract, watching information, etc. CarpentryCon 2022 Schedule • CarpentryCon 2022 <https://2022.carpentrycon.org/> + - One pad per session: <https://meta.wikimedia.org/wiki/Arctic_Knot_Conference_2021/Program#Friday,_June_25th> + - [Etherpad integration in Jitsi ](https://community.jitsi.org/t/tutorial-etherpad-integration-in-jitsi-meetings/99697) + +- Good: One pad per session +- Better: Some kind of monitoring so that we can link to the pads or embed + the pads even before the conference starts while not risking too + much vandalism +- Best: + - Pad can be easily regenerated from Emacs Lisp with a check to see if people have been adding to it + - Pad links to next talks + +Where should we host this? + +- live0: gets scaled up the most, lots of people connect to it for the conference, didn’t hit performance constraints last time +- front0: lower risk of interfering with stream +- bbb: will already be put on strain with the concurrent streams (test showed it was stable with up to 10 concurrent video streams and 40 total participants) +- opal’s: no news from owner, but beefy server that we used for reencode last year +- new Linode (probably 4GB or 8GB shared CPU): can easily be spun up, yay Ansible + +[Ansible notes](#ansible) + +Consider if we need extra scaling beyond being on a beefy live0? + +- Scale calculator: <https://scale.etherpad.org/> + - assuming 3 concurrent authors, 100 lurkers per pad, 3 concurrent pads + - 1 core, 4GB RAM, bandwidth Mb/s: 7.488 +- <https://mclear.co.uk/2021/09/08/deploying-etherpad-at-scale-in-one-minute/> +- <https://github.com/ether/etherpad-load-test> + +Will need to try this again when we resize nodes. Probably just the extra memory will be enough and the CPU use from node won’t step on the streaming, but not sure + + +### DONE Use the API to create pages based on all the slugs + + +### CANCELLED Figure out monitoring; maybe get everything daily and commit to git repo? + +History or regeneration will be fine. We’ll probably link to it shortly before the event, and we can also turn off the service until we’re ready. + + +### DONE Generate talk-specific pad content + + +### DONE Set up nginx reverse proxy + + +### DONE Load-test + +tl;dr: Either a separate 4GB Linode or being on live0 will probably be +okay, but I’m not 100% sure due to the limitations of my load-testing +setup. I don’t know if we need to shard by pad. + +It looks like etherpad-load-test tends to max out at ~40 connections +on a specific node. I used GNU Parallel to run the loadtesting tool +against a 4GB Linode instance (shared CPU) from five nodes at the same +time (my X220, my 2GB Linode instance, front0, and the node with the +pad), and they all reached about 35-45 clients before failure (not +updating within 100ms). + + echo 'node node_modules/etherpad-load-test/app.js http://170.187.195.5:9001 -d 120 > ~/output.txt' | parallel -J loadtest -j 1 --onall --verbose --tag + + echo 'grep Local ~/output.txt | tail -1' | parallel -J loadtest -j 1 --onall --tag + +Files were created at roughly the same time, so the max loads probably +overlapped. It would be good to have finer control over the +etherpad-load-test tool. Haven’t figured out how to properly use `-l` +and `-a` yet. + + echo 'stat -c %y ~/output.txt' | parallel -J loadtest -j 1 --onall --tag + +CPU graph went only up to 5%. Network max was 2.36 Mb/s in, 1.25 Mb/s out. +looking at top, CPU seems to go only up to about 12% or so. + + +### DONE Set up pad.emacsconf.org to point to live0.emacsconf.org :bandali: + + +### DONE Set up letsencrypt + +Waiting for DNS +add to /etc/dehydrated/domains.txt +sudo sh -x /etc/cron.daily/renew-https-cert-local + + +### TODO Link to pad from talk page + + +### DONE Add links to general conference pad + +for collecting feedback + + +### TODO Prototype shift pads for easier scrolling + +if we can get the anchor plugin + + +### DONE Be able to fall back to wikimedia if necessary, maybe with nginx redirects + + +### DONE Export pad initial content HTML to make it easier to reimport into wikimedia or elsewhere + +`emacsconf-pad-export-initial-content-for-all-talks` + + +<a id="create-bbb"></a> + +## DONE Set up BBB rooms and update conf.org :sachac: + +1. Log on to bbb.emacsverse.org as an admin. +2. Create a room. Enable **Mute users when they join**. + + The code below doesn’t quite work, but might be a good starting point for future automation. + + (setq list (seq-drop (emacsconf-bbb-room-title-list) 3)) ; skip some if needed + (progn + (setq name (pop list)) + (kill-new (format "name=\"%s\";$('#create-room-block').click();$('#create-room-name').val(name);$('#room_mute_on_join').click();$('.create-room-button').click();\n" + name)) + (sleep-for 1) + (shell-command "xdotool key alt+Tab sleep 3 key ctrl+v sleep 1 key Return")) + +console.log(JSON.stringify([…document.querySelectorAll(’.delete-room’)].map((o) => { return { name: o.getAttribute(’data-name’), path: o.getAttribute(’data-path’) }}).filter((o) => o.name.match(*^ec22*)))) + +see conf.org for the rest of the process + + +### TODO Add volunteers to the BBB rooms + + <a id="maybe-projects"></a> # Projects to bear in mind but which are not actual @@ -2490,7 +2307,7 @@ Probably focus on grabbing the audio first and seeing what’s worth keeping Make a table of the form -<table id="org95fa28c" border="2" cellspacing="0" cellpadding="6" rules="groups" frame="hsides"> +<table id="orgecfc040" border="2" cellspacing="0" cellpadding="6" rules="groups" frame="hsides"> <colgroup> @@ -3250,6 +3067,186 @@ Sacha It would be nice to have track-specific icals as well. +<a id="upload"></a> + +## DONE Figure out web-based file upload :needsowner:sachac:ansible: + +zaeph: I can work on it, but I’m not experienced in this domain, so I’d prefer to be a back-up. +task is currently with bandali + +Lesson learned from last year: “Since people kept running into ftp +problems, we might want to set up a web-frontend next year to minimise +problems.” + +Maybe we could ask some of the volunteers who wanted to help us with +the infra? It shouldn’t be complicated to deploy a ready-made +solution. + +“file drop” is a common keyword for looking for information. +Considerations: + +- Probably run it on media.emacsconf.org +- Bonus features: + - password-protected or hidden behind some kind of authentication or hidden behind some kind of URL, so we don’t have to worry too much about spam + - extra points for sending speakers links to upload to specific folders so that we can separate resources by talk + - Resumable uploads would be good, since some speakers had a hard time with unreliable connections +- What other conferences do: + - LibrePlanet uses plain FTP and recommends FileZilla <https://libreplanet.org/wiki/Video_upload_instructions> + - FOSSDEM uses Pentabarf to receive uploads <https://archive.fosdem.org/2022/manuals/program/speaker/> + - DebConf uses SReview(?) + - FOSSGIS uses Seafile <https://vmx.cx/cgi-bin/blog/index.cgi/video-uploads-for-an-online-conference%3A2021-06-12%3Aen%2Cconference%2Cgeo> +- Some options: + + <table border="2" cellspacing="0" cellpadding="6" rules="groups" frame="hsides"> + + + <colgroup> + <col class="org-left" /> + + <col class="org-left" /> + + <col class="org-left" /> + + <col class="org-left" /> + + <col class="org-left" /> + </colgroup> + <tbody> + <tr> + <td class="org-left">Project</td> + <td class="org-left">Docker</td> + <td class="org-left">Buster</td> + <td class="org-left">Base</td> + <td class="org-left">Notes</td> + </tr> + + + <tr> + <td class="org-left"><a href="https://www.projectsend.org/">https://www.projectsend.org/</a></td> + <td class="org-left"> </td> + <td class="org-left"> </td> + <td class="org-left">php+mysql</td> + <td class="org-left"> </td> + </tr> + + + <tr> + <td class="org-left"><a href="https://github.com/pomf/pomf">https://github.com/pomf/pomf</a></td> + <td class="org-left"> </td> + <td class="org-left"> </td> + <td class="org-left">php+mysql</td> + <td class="org-left"> </td> + </tr> + + + <tr> + <td class="org-left"><a href="https://github.com/psi-4ward/psitransfer">https://github.com/psi-4ward/psitransfer</a></td> + <td class="org-left">official</td> + <td class="org-left"> </td> + <td class="org-left">node</td> + <td class="org-left">can set upload password, resumable; data volume needs uid 1000</td> + </tr> + + + <tr> + <td class="org-left"><a href="https://github.com/epoupon/fileshelter">https://github.com/epoupon/fileshelter</a></td> + <td class="org-left">official</td> + <td class="org-left">ppa</td> + <td class="org-left">C++</td> + <td class="org-left"> </td> + </tr> + + + <tr> + <td class="org-left"><a href="https://github.com/mickael-kerjean/filestash">https://github.com/mickael-kerjean/filestash</a></td> + <td class="org-left">official</td> + <td class="org-left"> </td> + <td class="org-left"> </td> + <td class="org-left">general FTP client</td> + </tr> + + + <tr> + <td class="org-left"><a href="https://gitlab.com/moejo42/Jirafeau">https://gitlab.com/moejo42/Jirafeau</a></td> + <td class="org-left">official</td> + <td class="org-left"> </td> + <td class="org-left">php</td> + <td class="org-left"> </td> + </tr> + + + <tr> + <td class="org-left"><a href="https://github.com/YouTransfer/YouTransfer">https://github.com/YouTransfer/YouTransfer</a></td> + <td class="org-left">official</td> + <td class="org-left"> </td> + <td class="org-left">node</td> + <td class="org-left">looking for maint</td> + </tr> + + + <tr> + <td class="org-left"><a href="https://github.com/dutchcoders/transfer.sh/">https://github.com/dutchcoders/transfer.sh/</a></td> + <td class="org-left"> </td> + <td class="org-left"> </td> + <td class="org-left"> </td> + <td class="org-left"> </td> + </tr> + + + <tr> + <td class="org-left"><a href="https://github.com/eikek/sharry">https://github.com/eikek/sharry</a></td> + <td class="org-left"> </td> + <td class="org-left"> </td> + <td class="org-left"> </td> + <td class="org-left"> </td> + </tr> + </tbody> + </table> + + - NextCloud + + anon FTP upload currently goes to *srv/ftp/anon* on media.emacsconf.org + + <file:///ssh:media|sudo> + + Maybe I’ll do psitransfer as a direct install + + +### DONE Create 2022/upload.md with the same workflow as last year for a start + + +### DONE Implement new workflow + + +<a id="irc"></a> + +## DONE Update IRC instructions because of multiple tracks + +Added to watch pages + + +<a id="publishing-sched"></a> + +## DONE Move scheduling and publishing code to Emacs on a VPS so that other people can help out :sachac: + +Ideal: + +- Update pages with watching information, additional resources, etc. as talks go live +- Update the schedule as needed (cancelled or reordered talks, etc.) + +Where: + +- front? my own VPS? + + Nice if there’s an Ansible playbook + + sachac’s notes: + <file:///home/sacha/code/docker/emacsconf-publish/> +- probably good to set it up on front + +It’s now on front. + + <a id="comms"></a> # Communications diff --git a/2022/organizers-notebook/index.org b/2022/organizers-notebook/index.org index a3e7b779..490f41d0 100644 --- a/2022/organizers-notebook/index.org +++ b/2022/organizers-notebook/index.org @@ -266,58 +266,6 @@ Sacha Chua #+TOC: headlines 1 local Ordered by importance. -** DONE [#A] Figure out web-based file upload :needsowner:sachac:ansible: -CLOSED: [2022-10-19 Wed 09:20] SCHEDULED: <2022-10-18 Tue> -:PROPERTIES: -:CUSTOM_ID: upload -:END: -zaeph: I can work on it, but I’m not experienced in this domain, so I’d prefer to be a back-up. -task is currently with bandali - -Lesson learned from last year: "Since people kept running into ftp -problems, we might want to set up a web-frontend next year to minimise -problems." - -Maybe we could ask some of the volunteers who wanted to help us with -the infra? It shouldn’t be complicated to deploy a ready-made -solution. - -"file drop" is a common keyword for looking for information. -Considerations: -- Probably run it on media.emacsconf.org -- Bonus features: - - password-protected or hidden behind some kind of authentication or hidden behind some kind of URL, so we don't have to worry too much about spam - - extra points for sending speakers links to upload to specific folders so that we can separate resources by talk - - Resumable uploads would be good, since some speakers had a hard time with unreliable connections -- What other conferences do: - - LibrePlanet uses plain FTP and recommends FileZilla https://libreplanet.org/wiki/Video_upload_instructions - - FOSSDEM uses Pentabarf to receive uploads https://archive.fosdem.org/2022/manuals/program/speaker/ - - DebConf uses SReview(?) - - FOSSGIS uses Seafile https://vmx.cx/cgi-bin/blog/index.cgi/video-uploads-for-an-online-conference%3A2021-06-12%3Aen%2Cconference%2Cgeo -- Some options: - - | Project | Docker | Buster | Base | Notes | - | https://www.projectsend.org/ | | | php+mysql | | - | https://github.com/pomf/pomf | | | php+mysql | | - | https://github.com/psi-4ward/psitransfer | official | | node | can set upload password, resumable; data volume needs uid 1000 | - | https://github.com/epoupon/fileshelter | official | ppa | C++ | | - | https://github.com/mickael-kerjean/filestash | official | | | general FTP client | - | https://gitlab.com/moejo42/Jirafeau | official | | php | | - | https://github.com/YouTransfer/YouTransfer | official | | node | looking for maint | - | https://github.com/dutchcoders/transfer.sh/ | | | | | - | https://github.com/eikek/sharry | | | | | - - NextCloud - - anon FTP upload currently goes to /srv/ftp/anon/ on media.emacsconf.org - - [[file:/ssh:media|sudo::/etc/vsftpd.conf]] - - Maybe I'll do psitransfer as a direct install - -*** DONE Create 2022/upload.md with the same workflow as last year for a start -CLOSED: [2022-10-18 Tue 14:37] -*** DONE Implement new workflow -CLOSED: [2022-10-18 Tue 14:37] ** TODO Coordinate and help volunteers :sachac: :PROPERTIES: :CUSTOM_ID: coordinate-volunteers @@ -405,129 +353,6 @@ We need time after the prerecs get submitted to: Make changes in [[#prerec-process]] -** INPROGRESS Plan Etherpad use and hosting :sachac:ansible: -DEADLINE: <2022-10-15 Sat> -:PROPERTIES: -:CUSTOM_ID: etherpad -:END: - -- Relevant links: - - Per-pad, nicely structured info with abstract, watching information, etc. CarpentryCon 2022 Schedule • CarpentryCon 2022 https://2022.carpentrycon.org/ - - One pad per session: https://meta.wikimedia.org/wiki/Arctic_Knot_Conference_2021/Program#Friday,_June_25th - - [[https://community.jitsi.org/t/tutorial-etherpad-integration-in-jitsi-meetings/99697][Etherpad integration in Jitsi ]] - -- Good: One pad per session -- Better: Some kind of monitoring so that we can link to the pads or embed - the pads even before the conference starts while not risking too - much vandalism -- Best: - - Pad can be easily regenerated from Emacs Lisp with a check to see if people have been adding to it - - Pad links to next talks - -Where should we host this? -- live0: gets scaled up the most, lots of people connect to it for the conference, didn't hit performance constraints last time -- front0: lower risk of interfering with stream -- bbb: will already be put on strain with the concurrent streams (test showed it was stable with up to 10 concurrent video streams and 40 total participants) -- opal’s: no news from owner, but beefy server that we used for reencode last year -- new Linode (probably 4GB or 8GB shared CPU): can easily be spun up, yay Ansible - -[[#ansible][Ansible notes]] - -Consider if we need extra scaling beyond being on a beefy live0? -- Scale calculator: https://scale.etherpad.org/ - - assuming 3 concurrent authors, 100 lurkers per pad, 3 concurrent pads - - 1 core, 4GB RAM, bandwidth Mb/s: 7.488 -- https://mclear.co.uk/2021/09/08/deploying-etherpad-at-scale-in-one-minute/ -- https://github.com/ether/etherpad-load-test - -Will need to try this again when we resize nodes. Probably just the extra memory will be enough and the CPU use from node won't step on the streaming, but not sure - -*** DONE Use the API to create pages based on all the slugs -CLOSED: [2022-10-11 Tue 20:41] -*** CANCELLED Figure out monitoring; maybe get everything daily and commit to git repo? -CLOSED: [2022-10-12 Wed 20:31] -History or regeneration will be fine. We'll probably link to it shortly before the event, and we can also turn off the service until we're ready. -*** DONE Generate talk-specific pad content -CLOSED: [2022-10-12 Wed 20:31] -*** DONE Set up nginx reverse proxy -CLOSED: [2022-10-13 Thu 12:19] -*** DONE Load-test -CLOSED: [2022-10-13 Thu 12:32] - -tl;dr: Either a separate 4GB Linode or being on live0 will probably be -okay, but I'm not 100% sure due to the limitations of my load-testing -setup. I don't know if we need to shard by pad. - -It looks like etherpad-load-test tends to max out at ~40 connections -on a specific node. I used GNU Parallel to run the loadtesting tool -against a 4GB Linode instance (shared CPU) from five nodes at the same -time (my X220, my 2GB Linode instance, front0, and the node with the -pad), and they all reached about 35-45 clients before failure (not -updating within 100ms). - -#+begin_src sh :eval no -echo 'node node_modules/etherpad-load-test/app.js http://170.187.195.5:9001 -d 120 > ~/output.txt' | parallel -J loadtest -j 1 --onall --verbose --tag -#+end_src - -#+begin_src sh :eval no :results output replace -echo 'grep Local ~/output.txt | tail -1' | parallel -J loadtest -j 1 --onall --tag -#+end_src - -#+RESULTS: -:results: -: Local Clients Connected: 43 -front Local Clients Connected: 43 -live Local Clients Connected: 44 -root@170.187.195.5 Local Clients Connected: 38 -web Local Clients Connected: 42 -:end: - -Files were created at roughly the same time, so the max loads probably -overlapped. It would be good to have finer control over the -etherpad-load-test tool. Haven't figured out how to properly use ~-l~ -and ~-a~ yet. - -#+begin_src sh :eval no :results output replace -echo 'stat -c %y ~/output.txt' | parallel -J loadtest -j 1 --onall --tag -#+end_src - -#+RESULTS: -:results: -: 2022-10-13 12:42:32.212744774 -0400 -front 2022-10-13 16:42:34.466605650 +0000 -live 2022-10-13 16:38:46.818016379 +0000 -root@170.187.195.5 2022-10-13 16:39:01.738965567 +0000 -web 2022-10-13 12:39:14.417086960 -0400 -:end: - -CPU graph went only up to 5%. Network max was 2.36 Mb/s in, 1.25 Mb/s out. -looking at top, CPU seems to go only up to about 12% or so. -*** DONE Set up pad.emacsconf.org to point to live0.emacsconf.org :bandali: -CLOSED: [2022-10-17 Mon 00:31] DEADLINE: <2022-11-12 Sat> -*** DONE Set up letsencrypt -CLOSED: [2022-10-16 Sun 14:42] -Waiting for DNS -add to /etc/dehydrated/domains.txt -sudo sh -x /etc/cron.daily/renew-https-cert-local -*** TODO Link to pad from talk page -SCHEDULED: <2022-11-15 Tue> -*** DONE Add links to general conference pad -CLOSED: [2022-10-19 Wed 09:22] -for collecting feedback -*** TODO Prototype shift pads for easier scrolling -if we can get the anchor plugin -*** DONE Be able to fall back to wikimedia if necessary, maybe with nginx redirects -CLOSED: [2022-10-13 Thu 13:55] -*** DONE Export pad initial content HTML to make it easier to reimport into wikimedia or elsewhere -CLOSED: [2022-10-13 Thu 13:46] -~emacsconf-pad-export-initial-content-for-all-talks~ -** DONE Update IRC instructions because of multiple tracks -CLOSED: [2022-10-16 Sun 17:39] -:PROPERTIES: -:CUSTOM_ID: irc -:END: - -Added to watch pages ** TODO Investigate streaming options, maybe OBS in the cloud :corwin: DEADLINE: <2022-11-20 Sun> :PROPERTIES: @@ -609,25 +434,6 @@ pads See break commercials https://www.collabmagazine.com/organizing-a-multi-track-virtual-conference-with-microsoft-teams-live-events-a-technical-playbook-and-lessons-learned/ -** DONE Move scheduling and publishing code to Emacs on a VPS so that other people can help out :sachac: -:PROPERTIES: -:CUSTOM_ID: publishing-sched -:END: -Ideal: -- Update pages with watching information, additional resources, etc. as talks go live -- Update the schedule as needed (cancelled or reordered talks, etc.) - -Where: -- front? my own VPS? - - Nice if there's an Ansible playbook - - sachac's notes: - [[file:~/code/docker/emacsconf-publish/]] -- probably good to set it up on front - -It's now on front. - ** TODO Create protocol for live Q&A :zaeph: :PROPERTIES: :CUSTOM_ID: live-protocol @@ -637,31 +443,6 @@ Objective: Comments: - It would be nice to have a protocol with speakers to tell us that they’re uncomfortable with a question, just so that we can find a smooth way to skip it. -** DONE Set up BBB rooms and update conf.org :sachac: -CLOSED: [2022-10-17 Mon 21:40] -:PROPERTIES: -:CUSTOM_ID: create-bbb -:END: - -1. Log on to bbb.emacsverse.org as an admin. -2. Create a room. Enable *Mute users when they join*. - - The code below doesn't quite work, but might be a good starting point for future automation. - - #+begin_src emacs-lisp :eval no -(setq list (seq-drop (emacsconf-bbb-room-title-list) 3)) ; skip some if needed -(progn - (setq name (pop list)) - (kill-new (format "name=\"%s\";$('#create-room-block').click();$('#create-room-name').val(name);$('#room_mute_on_join').click();$('.create-room-button').click();\n" - name)) - (sleep-for 1) - (shell-command "xdotool key alt+Tab sleep 3 key ctrl+v sleep 1 key Return")) -#+end_src - -console.log(JSON.stringify([...document.querySelectorAll('.delete-room')].map((o) => { return { name: o.getAttribute('data-name'), path: o.getAttribute('data-path') }}).filter((o) => o.name.match(/^ec22/)))) - -see conf.org for the rest of the process - ** STANDBY Add nice-to-have stuff to prepare.md :zaeph: :PROPERTIES: :CUSTOM_ID: extra-prepare @@ -844,6 +625,147 @@ SCHEDULED: <2022-12-14 Wed> :PROPERTIES: :CUSTOM_ID: upload-off :END: +** INPROGRESS [#C] Plan Etherpad use and hosting :sachac:ansible: +DEADLINE: <2022-10-15 Sat> +:PROPERTIES: +:CUSTOM_ID: etherpad +:END: + +- Relevant links: + - Per-pad, nicely structured info with abstract, watching information, etc. CarpentryCon 2022 Schedule • CarpentryCon 2022 https://2022.carpentrycon.org/ + - One pad per session: https://meta.wikimedia.org/wiki/Arctic_Knot_Conference_2021/Program#Friday,_June_25th + - [[https://community.jitsi.org/t/tutorial-etherpad-integration-in-jitsi-meetings/99697][Etherpad integration in Jitsi ]] + +- Good: One pad per session +- Better: Some kind of monitoring so that we can link to the pads or embed + the pads even before the conference starts while not risking too + much vandalism +- Best: + - Pad can be easily regenerated from Emacs Lisp with a check to see if people have been adding to it + - Pad links to next talks + +Where should we host this? +- live0: gets scaled up the most, lots of people connect to it for the conference, didn't hit performance constraints last time +- front0: lower risk of interfering with stream +- bbb: will already be put on strain with the concurrent streams (test showed it was stable with up to 10 concurrent video streams and 40 total participants) +- opal’s: no news from owner, but beefy server that we used for reencode last year +- new Linode (probably 4GB or 8GB shared CPU): can easily be spun up, yay Ansible + +[[#ansible][Ansible notes]] + +Consider if we need extra scaling beyond being on a beefy live0? +- Scale calculator: https://scale.etherpad.org/ + - assuming 3 concurrent authors, 100 lurkers per pad, 3 concurrent pads + - 1 core, 4GB RAM, bandwidth Mb/s: 7.488 +- https://mclear.co.uk/2021/09/08/deploying-etherpad-at-scale-in-one-minute/ +- https://github.com/ether/etherpad-load-test + +Will need to try this again when we resize nodes. Probably just the extra memory will be enough and the CPU use from node won't step on the streaming, but not sure + +*** DONE Use the API to create pages based on all the slugs +CLOSED: [2022-10-11 Tue 20:41] +*** CANCELLED Figure out monitoring; maybe get everything daily and commit to git repo? +CLOSED: [2022-10-12 Wed 20:31] +History or regeneration will be fine. We'll probably link to it shortly before the event, and we can also turn off the service until we're ready. +*** DONE Generate talk-specific pad content +CLOSED: [2022-10-12 Wed 20:31] +*** DONE Set up nginx reverse proxy +CLOSED: [2022-10-13 Thu 12:19] +*** DONE Load-test +CLOSED: [2022-10-13 Thu 12:32] + +tl;dr: Either a separate 4GB Linode or being on live0 will probably be +okay, but I'm not 100% sure due to the limitations of my load-testing +setup. I don't know if we need to shard by pad. + +It looks like etherpad-load-test tends to max out at ~40 connections +on a specific node. I used GNU Parallel to run the loadtesting tool +against a 4GB Linode instance (shared CPU) from five nodes at the same +time (my X220, my 2GB Linode instance, front0, and the node with the +pad), and they all reached about 35-45 clients before failure (not +updating within 100ms). + +#+begin_src sh :eval no +echo 'node node_modules/etherpad-load-test/app.js http://170.187.195.5:9001 -d 120 > ~/output.txt' | parallel -J loadtest -j 1 --onall --verbose --tag +#+end_src + +#+begin_src sh :eval no :results output replace +echo 'grep Local ~/output.txt | tail -1' | parallel -J loadtest -j 1 --onall --tag +#+end_src + +#+RESULTS: +:results: +: Local Clients Connected: 43 +front Local Clients Connected: 43 +live Local Clients Connected: 44 +root@170.187.195.5 Local Clients Connected: 38 +web Local Clients Connected: 42 +:end: + +Files were created at roughly the same time, so the max loads probably +overlapped. It would be good to have finer control over the +etherpad-load-test tool. Haven't figured out how to properly use ~-l~ +and ~-a~ yet. + +#+begin_src sh :eval no :results output replace +echo 'stat -c %y ~/output.txt' | parallel -J loadtest -j 1 --onall --tag +#+end_src + +#+RESULTS: +:results: +: 2022-10-13 12:42:32.212744774 -0400 +front 2022-10-13 16:42:34.466605650 +0000 +live 2022-10-13 16:38:46.818016379 +0000 +root@170.187.195.5 2022-10-13 16:39:01.738965567 +0000 +web 2022-10-13 12:39:14.417086960 -0400 +:end: + +CPU graph went only up to 5%. Network max was 2.36 Mb/s in, 1.25 Mb/s out. +looking at top, CPU seems to go only up to about 12% or so. +*** DONE Set up pad.emacsconf.org to point to live0.emacsconf.org :bandali: +CLOSED: [2022-10-17 Mon 00:31] DEADLINE: <2022-11-12 Sat> +*** DONE Set up letsencrypt +CLOSED: [2022-10-16 Sun 14:42] +Waiting for DNS +add to /etc/dehydrated/domains.txt +sudo sh -x /etc/cron.daily/renew-https-cert-local +*** TODO Link to pad from talk page +SCHEDULED: <2022-11-15 Tue> +*** DONE Add links to general conference pad +CLOSED: [2022-10-19 Wed 09:22] +for collecting feedback +*** TODO Prototype shift pads for easier scrolling +if we can get the anchor plugin +*** DONE Be able to fall back to wikimedia if necessary, maybe with nginx redirects +CLOSED: [2022-10-13 Thu 13:55] +*** DONE Export pad initial content HTML to make it easier to reimport into wikimedia or elsewhere +CLOSED: [2022-10-13 Thu 13:46] +~emacsconf-pad-export-initial-content-for-all-talks~ +** DONE Set up BBB rooms and update conf.org :sachac: +CLOSED: [2022-10-17 Mon 21:40] +:PROPERTIES: +:CUSTOM_ID: create-bbb +:END: + +1. Log on to bbb.emacsverse.org as an admin. +2. Create a room. Enable *Mute users when they join*. + + The code below doesn't quite work, but might be a good starting point for future automation. + + #+begin_src emacs-lisp :eval no +(setq list (seq-drop (emacsconf-bbb-room-title-list) 3)) ; skip some if needed +(progn + (setq name (pop list)) + (kill-new (format "name=\"%s\";$('#create-room-block').click();$('#create-room-name').val(name);$('#room_mute_on_join').click();$('.create-room-button').click();\n" + name)) + (sleep-for 1) + (shell-command "xdotool key alt+Tab sleep 3 key ctrl+v sleep 1 key Return")) +#+end_src + +console.log(JSON.stringify([...document.querySelectorAll('.delete-room')].map((o) => { return { name: o.getAttribute('data-name'), path: o.getAttribute('data-path') }}).filter((o) => o.name.match(/^ec22/)))) + +see conf.org for the rest of the process +*** TODO Add volunteers to the BBB rooms * Projects to bear in mind but which are not actual :PROPERTIES: :CUSTOM_ID: maybe-projects @@ -2658,6 +2580,84 @@ CLOSED: [2022-10-19 Wed 18:46] It would be nice to have track-specific icals as well. +** DONE [#A] Figure out web-based file upload :needsowner:sachac:ansible: +CLOSED: [2022-10-19 Wed 09:20] SCHEDULED: <2022-10-18 Tue> +:PROPERTIES: +:CUSTOM_ID: upload +:END: +zaeph: I can work on it, but I’m not experienced in this domain, so I’d prefer to be a back-up. +task is currently with bandali + +Lesson learned from last year: "Since people kept running into ftp +problems, we might want to set up a web-frontend next year to minimise +problems." + +Maybe we could ask some of the volunteers who wanted to help us with +the infra? It shouldn’t be complicated to deploy a ready-made +solution. + +"file drop" is a common keyword for looking for information. +Considerations: +- Probably run it on media.emacsconf.org +- Bonus features: + - password-protected or hidden behind some kind of authentication or hidden behind some kind of URL, so we don't have to worry too much about spam + - extra points for sending speakers links to upload to specific folders so that we can separate resources by talk + - Resumable uploads would be good, since some speakers had a hard time with unreliable connections +- What other conferences do: + - LibrePlanet uses plain FTP and recommends FileZilla https://libreplanet.org/wiki/Video_upload_instructions + - FOSSDEM uses Pentabarf to receive uploads https://archive.fosdem.org/2022/manuals/program/speaker/ + - DebConf uses SReview(?) + - FOSSGIS uses Seafile https://vmx.cx/cgi-bin/blog/index.cgi/video-uploads-for-an-online-conference%3A2021-06-12%3Aen%2Cconference%2Cgeo +- Some options: + + | Project | Docker | Buster | Base | Notes | + | https://www.projectsend.org/ | | | php+mysql | | + | https://github.com/pomf/pomf | | | php+mysql | | + | https://github.com/psi-4ward/psitransfer | official | | node | can set upload password, resumable; data volume needs uid 1000 | + | https://github.com/epoupon/fileshelter | official | ppa | C++ | | + | https://github.com/mickael-kerjean/filestash | official | | | general FTP client | + | https://gitlab.com/moejo42/Jirafeau | official | | php | | + | https://github.com/YouTransfer/YouTransfer | official | | node | looking for maint | + | https://github.com/dutchcoders/transfer.sh/ | | | | | + | https://github.com/eikek/sharry | | | | | + - NextCloud + + anon FTP upload currently goes to /srv/ftp/anon/ on media.emacsconf.org + + [[file:/ssh:media|sudo::/etc/vsftpd.conf]] + + Maybe I'll do psitransfer as a direct install + +*** DONE Create 2022/upload.md with the same workflow as last year for a start +CLOSED: [2022-10-18 Tue 14:37] +*** DONE Implement new workflow +CLOSED: [2022-10-18 Tue 14:37] +** DONE Update IRC instructions because of multiple tracks +CLOSED: [2022-10-16 Sun 17:39] +:PROPERTIES: +:CUSTOM_ID: irc +:END: + +Added to watch pages +** DONE Move scheduling and publishing code to Emacs on a VPS so that other people can help out :sachac: +:PROPERTIES: +:CUSTOM_ID: publishing-sched +:END: +Ideal: +- Update pages with watching information, additional resources, etc. as talks go live +- Update the schedule as needed (cancelled or reordered talks, etc.) + +Where: +- front? my own VPS? + + Nice if there's an Ansible playbook + + sachac's notes: + [[file:~/code/docker/emacsconf-publish/]] +- probably good to set it up on front + +It's now on front. + * Communications :PROPERTIES: :CUSTOM_ID: comms |