Wednesday, 2023-08-16

opendevreviewJeremy Stanley proposed opendev/system-config master: Restart Mailman 3 containers when configs change
fungiClark[m]: like that? ^00:13
Clark[m]fungi: you need to replace the uwsgi check with a mailman service check if appropriate00:14
Clark[m]Or is it uwsgi that mailman runs?00:14
fungidjango is running from uwsgi00:15
Clark[m]Ah ok. I guess in my head Django has it's own thing but I guess not00:15
fungii figured it was the easiest thing to check for00:15
fungirather than arbitrary python processes for queue runners00:15
Clark[m]But I don't think it will trigger that task to list the uwsgi stuff it will only trigger your restart task use to the name you notify00:16
fungii think django has its own inbuilt server as an option, but the deployment uses uwsgi00:16
fungihow does the one for jitsi know to run the similar check it has?00:17
fungiplaybooks/roles/letsencrypt-create-certs/handlers/restart_graphite.yaml also has a similar one00:18
Clark[m]It uses include tasks to run all the tasks in the included file when triggered here
Clark[m]So the name on like 44 there is what you notify them when it runs it runs everything in the included task file00:19
fungiso probably can't put those in the same handler file as the apache reload/restart00:19
Clark[m]Correct 00:23
fungiso how do i make sure it runs only once regardless of how many config files change, and only if the containers are already running?00:27
Clark[m]That is what Ansible handlers do. They only run once at the end00:42
Clark[m]So you notify the top level handler. It includes the multiple tasks and they run but only once00:42
fungier, so the file itself is the top-level handler?00:50
Clark[m]The handlers/main.yaml is where your top level goes. This is the one whose name you notify. That single task can then include tasks from another adjacent file just like the LE example which does all the steps you need in multiple tasks00:53
fungioh, got it00:54
opendevreviewJeremy Stanley proposed opendev/system-config master: Restart Mailman 3 containers when configs change
fungiokay, so that i guess ^01:00
Clark[m]Ya that looks correct. I'll take a closer look in the morning 01:02
*** dmellado819181 is now known as dmellado8191804:45
opendevreviewMd Irshad Sheikh proposed openstack/project-config master: Add Intel Device Plugins app to StarlingX
opendevreviewMd Irshad Sheikh proposed openstack/project-config master: Add Intel Device Plugins app to StarlingX
opendevreviewMd Irshad Sheikh proposed openstack/project-config master: Add Intel Device Plugins app to StarlingX
opendevreviewMd Irshad Sheikh proposed openstack/project-config master: Add Intel Device Plugins app to StarlingX
opendevreviewMd Irshad Sheikh proposed openstack/project-config master: Add Intel Device Plugins app to StarlingX
opendevreviewDmitriy Rabotyagov proposed openstack/diskimage-builder master: Set platform argument in container build command for cross-arch builds
opendevreviewJeremy Stanley proposed opendev/system-config master: Restart Mailman 3 containers when configs change
fungianyone up for reviewing a minor git-review fix as a break the monotony and/or excitement of their day?
*** jroll8 is now known as jroll12:09
fricklerfungi: that sounds interesting enough to make me have a look ;)12:23
* frickler offers a boring in return12:24
fungiooh, don't mind if i do12:24
frickler"Tisza Gergő created this story on 2012-07-12 at 19:54:03 " wow12:25
fungii forgot we had a storyboard deployment that long ago12:30
fungithat was basically right when i got involved in openstack12:30
fungioh! that's an import from launchpad12:31
fungibug number should match12:31
opendevreviewJeremy Stanley proposed opendev/git-review master: Use GIT_SSH for the SSH executable
fungifrickler: ^ does that suit you?13:00
fungireduced the entire change to just one line, since there was only one place that new variable was getting used anyway13:02
fungimaking a separate assignment just for that seemed unnecessary13:02
fricklerfungi: ah, one step further even than my proposal, ack13:03
*** blarnath is now known as d34dh0r5313:31
opendevreviewMerged openstack/project-config master: Add Intel Device Plugins app to StarlingX
opendevreviewMerged opendev/git-review master: Use GIT_SSH for the SSH executable
clarkbfungi: left a small concern on the mm3 notify change15:41
clarkbalso I think I'm mostly awake now if we want to approve the etherpad upgrade change15:41
fungicool, i'm ready when you are15:42
fungiwant me to approve that one?15:42
fungialso i agree with your comment on the mm3 change, i was parroting what the grafana/jitsi cert update tasks do, but they're just restarting containerized web servers15:43
clarkbfungi: sure go for the approval15:43
fungii'll fix that15:43
clarkbby the time it actually lands I should have had something to eat and loaded my ssh keys :)15:44
opendevreviewClark Boylan proposed opendev/system-config master: Update to Gitea 1.20
opendevreviewJeremy Stanley proposed opendev/system-config master: Restart Mailman 3 containers when configs change
fungiclarkb: is it safe enough to just do that ^ or does the first task need to to register a result that the second requires?16:23
clarkbfungi: if the down fails then the ansible run should stop there and not up the system which is probably what we want?16:23
fungiyes, okay i wasn't sure if it would try to run the second task regardless16:24
clarkbthe first task looking for uwsgi processes needs the ignore errors flag set to true because we expect it to occasionally exit non zero16:25
clarkbI guess one downside to this approach is that we're doing a docker-compose up -d here so we may restart twice16:26
fungiaha, so the absence of "ignore_errors: yes" in "down containers" means that if it fails then "up containers" will get skipped16:26
clarkbI think that is ok for bootstrapping and actually that may make the restart later fine16:26
clarkbso maybe we didn't need to rewrite the last patchset...16:27
opendevreviewMerged openstack/project-config master: Allow neutron-core to act as osc/sdk service-core
opendevreviewMerged openstack/project-config master: Remove ocata related definitions
clarkbup -d will restart things if new container images are present otherwise it is a noop16:28
clarkband we have to up things there in the playbook because we need to check db state for bootstrapping steps so we can't move everything back into the handler at the end16:28
clarkbso ya maybe the restart would be fine? I think both approaches are ok I guess just occasionalyl we'll do a restart too many16:29
opendevreviewMerged opendev/system-config master: Update etherpad to 1.9.1
fungii'm not following... should i replace the "Run docker-compose up" task with a notify to the restart handler instead?16:29
opendevreviewMerged openstack/project-config master: Remove publish-install-guide ocata job
fungithough we have subsequent tasks that rely on the container being up, so the handler may run too late if we did that16:30
clarkbfungi: no you shouldn't do that for the reason I stated above. We need the processes up and running so that all the bootstrapping stuff here can run16:30
clarkbbasically we cannot get away with a simple down/up -d in the handler as that occurs too late16:30
fungiokay, so basically what's going to happen is that tasks/main.yaml bootstraps the server and starts the container, but then the notify will result in downing and upping the container again at the very end?16:31
clarkband it might be worth thinking about the ramifications of that a bit before we land the change16:31
fungiand that's presumably not only safe but could also serve as a good test that things are restarting correctly16:31
clarkbya. It will just result in the occasionally longer than necessary outage16:33
clarkbetherpad has updated16:40
clarkbyesterday's etherpad for zuul fixups loads for me
clarkbas far as I can tell it is working16:41
clarkbkeep an eye out for problems but my first impressions is that this is working as expected16:44
fungilgtm too, yep16:51
fungiworth noting, that indentationOnNewLine default doesn't seem to actually do anything16:53
fungiit still indented after a :16:53
clarkbfungi: also possible the example file is not showing defaults but an opinionated config which would be weird but possible16:54
clarkbwe could try explicitly setting it to the value we want in our own config16:54
fungiyeah, could be16:54
fungiwell, it would be a behavior change, so i'm not sure the disruption is warranted16:55
clarkbour backspace keys can get more exercise16:55
fungipeople are used to deleting the extra spaces at the start of the next line if they didn't want it16:55
clarkbfungi: idea: we could register the output of the first docker-compose up -d maybe and check that in the handler17:07
clarkbif the first one actually started everything we can skip the handler restart?17:08
clarkbpossible that is too fragile to bother with though17:08
fungiis it possible to register that in the tasks and then check it in the handler?17:09
fungiif so, i'm cool with it, doesn't seem especially fragile17:10
clarkbyes I think so. The fragility would be with the output of docker-compose being consistent and parseable. I think we do this elsewhere somewhere already though17:11
clarkbbut I can't remember where17:11
fungican't just check the exit code?17:11
clarkbNo because I ran and didn't restart anything because everything is up to date is exit code zero 0 iirc17:12
clarkbdocker-compose up -d will stop and create new containers if new images are available. It will also take action if no containers are running17:12
clarkbBut it can also say "no work to do" and noop17:12
clarkbits the no work to do and noop case that we want the handler to run17:12
fungiyeah, i can see where that would lead to issues17:13
clarkbok that last gitea patchset fixed the access logs.  Ithink we're very close to being ready with to land that as long as we accept we have to add a bunch of config for oauth which we disable...17:52
clarkbI'm going to get a node held and we can decide if we are comfortable with that after reviewing a running server17:52
opendevreviewClark Boylan proposed opendev/system-config master: DNM intentional gitea failure to hold a node
fungiinfra-root: ildikov directly sent me a specific list of matrix space and channel names the starlingx community has requested we add to our homeserver, so in keeping with the consensus we reached in our meeting i'm proceeding to set those up now19:01
fungianybody happen to know if there's a lag when setting icons for matrix channels? as homeserver admin i added the starlingx logo as the icon for their space and each of their channels and i see it in the tab i have logged in for that account, but when i join them from my personal account i just see the letter "s"19:26
fricklerwhat's the space name? I can double-check with my acc19:28
clarkbI don't know19:28
fungishould be a purple x-like icon if it's loading correctly19:29
fungialso, do we need to use the admin account to set channel descriptions, or is that something which can be delegated to the project?19:30
fricklerI can confirm that I only see a green "S". for our downstream space there is an icon, but the person who set that up is on PTO19:31
fungias for the icon, i definitely see icons for other channels we have, e.g. #zuul, so it could just be something that doesn't propagate straight away19:31
fungicorvus: ^ also possible you remember whether there were any special steps when you added a logo icon for the zuul channel19:36
fungimaybe accounts on other homeservers have to wait for icons to replicate across the federated network19:36
clarkbfungi: I don't think we've delegated any permissions for zuul. So the admin account has to do it all currently19:39
fungisure, the question was more whether such delegation is possible19:44
clarkbit is possible19:44
ildikovfungi: Thank you! The space looks great overall. Can you also create a 'StarlingX General' room in addition to the project team rooms? I think that's the only one that seems to be missing.19:45
clarkbfungi: in element if you open the people panel on the right side and hover on users admin is power:100 and regular users are power:019:45
clarkbyou can asign power values to people which allow them to do more stuff. I'm not sure what specific value is needed for topic setting though19:45
fungiildikov: oops, thanks i missed that because it was in a different section of your list all by itself. adding it now!19:46
fungiclarkb: aha, thanks, so "power users" is matrix's approximation for irc channel operators?19:46
clarkblooks like users can give other users power up to their power level (very dragon ball z like all of a sudden)19:46
clarkbfungi: its a 0-100 scale19:46
clarkbso more nuanced than irc19:46
clarkbsounds like normal user is typically 0, moderator is 50, and admin is 100 for channels though that may partly be by convention19:47
fungiildikov: it's added now19:48
ildikovfungi: awesome, thank you!19:48
clarkblooks like the recommendation for the fedora community is that admin be a very small group (we alredy do this). Then the majority of people who need extra perms get power:50 moderator power. THen you configure the rooms to let moderators do things like set topic19:48
clarkbwith zuul we've kept it simple though and it has been working fine19:48
fungiclarkb: i guess then it's a matter of figuring out what "power level" is required for things like channel description/topic19:48
clarkbfungi: yes, though it sounds like you configure that as admin and by default it is proabably requiring power level 10019:49
clarkbbut you'd find those options and set them to 50 and promote a small number of moderators to power level 5019:49
fungimakes sense. i was going to see if i could work out how to give ildikov the ability to set the descriptions for these initial 11 channels since i lack the context to come up with them on my own19:50
fungiaha, there's a "roles and permissions" in the space settings19:50
clarkboh so you can do it at a top level for all space rooms at once?19:51
clarkbthats a nice efficiency thing19:51
fungii'll find out i suppose. it may be that the channels don't inherit roles from the space and have to be set individually19:51
clarkbhttps:// gitea 1.20.2 running via our config mangaement19:51
ildikovyeah, I was looking for something like 'moderator' level, or smth that allows me to take care of the StarlingX space19:52
ildikovso I don't need to bug y'all with every little nuance19:52
fungiildikov: ironically, it's called "moderator" even ;)19:52
clarkbThe theme color thing I did for mobile isn't working. However it seems to not work with the existing deployment on 1.19 either. I wonder if that is a dark vs light theme thing overriding our color19:52
ildikovyeah, I was influenced by the convo here ;)19:52
clarkbaha yup that is the issue. If I hardset theme to light in my mobile browser both the test node and production gitea use the pink color theme19:53
fungithere is a named power level called "moderator" and by default it has the roles: change settings, remove users, ban users, change space name, change main address for the space, change space avatar, manage rooms in this space, change description19:54
clarkbhrm but maybe that was sticky from Retrying with the test node doesn't make it pink19:54
clarkbfungi: I would remove change main address for the space at least19:55
fungiadditionally it gets the "invite users" role which the default power level has (so everyone can do it)19:55
ildikovfungi: that sounds like a reasonable list of things!19:55
clarkbbut otherwise those seem reasonable for a trusted group of moderators19:55
fungithe one role moderator lacks is "change permissions" (which only the admin power level has)19:55
ildikovI think that's fine19:56
fungiand yeah, it does seem like the ability to change the main address is a bit of a foot cannon19:56
clarkbfungi: ya so moderator cannot give themselves new permissions19:56
clarkbwhich seems correct19:56
clarkbbut also it seems like those are space specific and not room inheritable perms?19:57
clarkbI guess that makes sense since arbitrary rooms can be added to arbitrary spaces19:57
fungiso if i open the settings for a specific room there are more roles19:57
fungidefault power level: send messages, send reactions, remove messages sent by me19:58
fungimoderator power level: invite users, change settings, remove users, ban users, remove messages sent by others, notify everyone, change room name, change main address for the room, change room avatar, change topic, send events, send events, send events, modify widgets, voice broadcasts20:00
clarkbside note if you add the channel recording bot it will record deleted messages forever20:00
fungiadmin power level: change permissions, change history visibility, upgrade the room, change server acls, enable room encryption20:00
clarkbfungi: I think changing the main address for the room should be removed, but otherwise those seems reasonable20:00
clarkbfor moderator I mean20:01
fungiokay, i figured out where to get at the top-level settings panel for the homeserver too20:01
corvusfungi: i think the avatar was just a moderator-level setting on the room20:02
fungiseeing if there's somewhere to set the default role profile server-wide20:02
fungicorvus: yeah, setting the avatar is, i'm wondering if i'm seeing a propagation delay for avatars between homeservers20:02
corvusfungi: be careful of the format; i think it may silently ignore nonconforming images20:03
fungioh, good to know. it showed up for me on the admin account for the homeserver but not for my account on another homeserver, so maybe it won't replicate if it's not in the right format?20:03
fungihmm, what i thought was the top-level settings for the homeserver was actually just the personal settings for the admin account's client session. i don't see anywhere obvious we can set a default inheritance for permission levels. i think if we want to make "change address" unavailable to moderators then we have to do that one at a time on every channel20:07
clarkbin my head changing an address should almost never happen. If you want a new channel make a new channel. So giving that permission to anone but admins seems wrong20:08
corvuskeep in mind that an address is just a pointer, so even if (very hypothetically) the starlingx mods go rogue and defect from the community and change the room's main address, the address should still exist and work20:10
corvus(i still think it's something that is in appropriate for mod-level perms in our community; but it sort of makes sense from a decentralized system standpoint)20:11
corvusfungi: i'm not at the moment sure what the image format is supposed to be.  but i also don't see it (unless i click on the "S" and then it does pop up).20:12
fungiokay, i think i got "change main address" delegation set to admin instead of moderator in all 11 of the initial starlingx rooms and the space. we'll just want to remember to add that for any other channels that are added20:12
fungicorvus: yeah it's weird, with the homeserver admin account logged in with the element browser client, it shows the avatar i set in the left sidebar for every channel, but not when joining them from my account20:13
corvusfungi: i believe i used pngs, and i think maybe you used an svg?20:13
fungioho, so it was20:13
fricklerthat seems to be an svg icon? pretty sure you'll want some scaled down png or jpg20:13
frickleror what corvus says ;)20:14
fungii'll try to swap it out for another format. thanks for spotting20:14
corvusif i'm looking at the right thing, i think the zuul logo may be 120x120 png20:17
corvusi have another room logo that's 512 square though20:18
fungiwell, when i preview the space, element helpfully converted it to a 96x96 pixel png20:23
fungi(from svg), so maybe that's a sign that there's no need to make it any bigger20:23
clarkbI think bigger pngs will do better on higher resolution displays20:29
clarkbassuming you aren't already in mac display territory on your system20:29
fungino matter how much border i try to put around the logo, matrix still wants to crop it and round the corners. maybe i need to embed a subtle border20:40
fungier, it crops away the padding i mean20:40
funginever mind, pebcak: was missing that i needed to click the "save" button in the settings20:51
clarkbI can see the meta theme-color information in my browser so my template overriding is working20:53
clarkbstackoverflow says "20:53
clarkbchrome requires you to have a valid certificate for the theme-bar color to work20:53
clarkbso maybe we just send it when we are ready and hope it works?20:53
clarkband if not its a super minor thing anyway that doesn't work by default on my dark theme phone setup20:53
fungicorvus: frickler: thanks! switching to png got everything working20:54
corvuslooks good from here too!20:56
fungiildikov: i've made you a moderator for the space itself, but i expect i'll need to still set you as a moderator individually in each room, and i'm not sure how to do that unless you join all 11 of those channels first20:57
clarkbI see them in the public room listing for with logos20:57
fungiildikov: but once you're a moderator, i think you should be able to set others as moderators too (or if you can't yet i think we can configure that option)20:58
clarkbfungi: I think anyone can set anyone else to the same powerlevel or lower than themselves20:58
fungii need to take a break and cook dinner, but will get back to this once eating has concluded20:59
clarkbfungi: it looks like docker-compose echos 'foo-container is up-to-date' for every container that it noops on when running up -d22:03
clarkbfungi: I think this means we can check for 'is up-to-date' in the stdout of the up -d command and if that is present then restart things. Otherwise we know that all the containers were either just started for the first time or were all restarted for updates22:04
clarkbI think that is the most correct thing to avoid unnecessary downtime one after another when updating both configs and container images at the same time22:04
clarkbI tested this on jvb01 because we keep those up to date and restarting them should it restart has minimal impact22:05
clarkbif you want to confirm just cd into that dir (remember it has env files so you have to be in that dir) then run docker-compose up -d22:05
fungiwell, there's also a d-c option to give the path to the env file, so technically not necessary to cd22:07
ildikovfungi: sorry, I was on calls. I joined all the rooms now, and I will need someone to set moderator right for me there too22:10
ildikovEverything else looks great!22:10
ildikovI can see the logo populated not too, which is nice22:11
fungiildikov: no worries, just saw you arrive and have started to set you to a mod22:11
ildikovPerfect, thank you so much!22:11
fungiildikov: looks like it says you're a mod now in the space and also all 11 initial rooms22:13
fungiand yeah, it looks like there's no way to set someone as a moderator in a room until they join that room, which is why it had to wait22:13
fungiand also, moderator level isn't inherited from the space itself, so has to be done independently for each room22:14
ildikovYep, I can confirm all that22:14
ildikovI will set topics, etc and then update the StarlingX ML about the new setup22:15
fungisounds great. let me know if you run into any other gotchas22:15
ildikovWill do, thank you!22:16
fungiany time22:16
opendevreviewJeremy Stanley proposed opendev/system-config master: Restart Mailman 3 containers when configs change

Generated by 2.17.3 by Marius Gedminas - find it at!