Thursday, 2023-08-24

fungicorvus: will do. lots of those were old old, some like 2019 old00:42
funginow we have a baseline to see what/when is still happening00:43
opendevreviewJay Faulkner proposed openstack/diskimage-builder master: DNM: Testing Gentoo CI job against merged-usr profile
opendevreviewGuillaume Boutry proposed openstack/project-config master: Add OpenStack K8S Magnum charms
TheJuliagood morning13:29
TheJuliawrong channel, but still good to say :)13:30
fungiit's appropriate for any channel, good morning to you too!13:40
*** dviroel_ is now known as dviroel14:03
clarkbfungi: not sure what your day looks like. I've got a good stretch until this afternoon where I've got school prep stuff to do. Happy to try and upgrade gitea and or mailman3 or whatever if you think we're in a good spot for it14:57
fungiclarkb: yeah, i'm semi-distracted by painters at the moment but expect they'll be wrapping up for the day here shortly15:03
*** TheJulia is now known as confused15:34
*** confused is now known as TheJulia15:34
opendevreviewMerged openstack/project-config master: Add OpenStack K8S Magnum charms
clarkbfungi: because I have a bad habit of rerevieing things: I've just (re)discovered a todo I left myself in which is to determine if that work path value matches what we've got on the current nodes since we aren't starting from scratch there16:30
clarkb says there are a number of ways this can be set so I've got to cross check those methods against prod to see if the paths are the same on 1.19 as they will be in 1.20 with that change16:31
clarkbI suspect it is fine and we're using a default that is consistent across the board but better safe than sorry (we could upgrade into empty repos)16:35
clarkbI haven't tracked it down in our config management yet. I will need to ssh into the hosts and inspect the running systems16:35
fungilooks like we map /var/gitea/data in as /data and there is a /var/gitea/data/gitea folder16:36
fungiso in theory /data/gitea inside the container ends up there16:36
fungiother than existing, what does appropriateness entail?16:37
fungilooks like it's all owned uid=1000 gid=1000 which is what we tell the gitea-web container to use, so its processes should be able to write there16:38
fungianything else you think needs to be checked?16:38
fungiit's on the rootfs, but at least on gitea09 there's 113gb free16:39
fungiclarkb: other than revising the change to drop the todo comment, i guess it's fine?16:40
clarkbsorry went to grab my ssh keys to check myself and got distracted16:57
clarkbfungi: I think the main thing is to confirm the current 1.19 installs have work path set to /data/gitea and/or that the dir strucutures on the held node match our prod nodes16:57
clarkbfungi: it looks like the held gitea node logged the work path on startup. The logs for gitea14 seemed to have rolled over since last restart but I'm trying to double check that. We might be able to restart one of the prod nodes to have them log this info and confirm for us17:02
clarkbya logs seemed to have all rolled over.17:03
clarkbfungi: look in the /proc/$PID/environ "file" for both gitea14 and the held gitea99. GITEA_WORK_DIR differs. It is /data/gitea on the held 1.20 node and /app/gitea in prod :/17:05
clarkbfungi: ok I did some prior debugging of this too
clarkbwow gitea's documentation documents APP_DATA_PATH twice with two different sets of information17:11
fungihuh. odd17:14
clarkbI'm understanding this a bit better. Basically WORK_PATH is the default root that everything else is mounted to when using relative paths. Our config seems to use rooted paths for everything already in /data/gitea or /data/git17:14
clarkbthe only thing in /app/gitea in prod is the gitea binary itself17:15
clarkbso on 1.19 we haven't created any new directories relative to the WORK_PATH17:15
clarkbI'll corss check the contents of /data/gitea on 1.20 with 1.19 to see if 1.20 is creating any new paths17:15
clarkbgitea 1.19 has /var/gitea/data/gitea/packages and 1.20 does not. gitea 1.20 has /var/gitea/data/gitea/sessions and 1.19 does not17:17
clarkbfor the second item both 1.19 and 1.20 app.ini configs set session.PROVIDER_CONFIG = /data/gitea/sessions17:18
clarkbI'm guessing that is created on demand when sessions are created then?17:19
clarkbthe packages dir is empty on 1.19. We disable packages on both 1.19 and 1.20. I think this may be a side effect of actually disabling more stuff17:21
fungii don't see a /data/gitea/sessions on gitea0917:21
clarkbfungi: ya I think gitea99 has one because I actually logged in as root17:21
clarkband its probably created on demand when we login. We can test that by logging into gitea09 and seeing if we get a sessions dir17:21
fungiaha, so user login sessions to the webui (and not authenticated git push)17:22
clarkbI suspect both the sessions dir and packages dirs are fine. sessions beacuse we explciitly configure the path the same on both 1.19 and 1.20 and packages because we disable it and the contents are empty in 1.19 so even if it moves later that won't really affect us17:22
clarkball that to say I think we are fine. If we had content in /app/gitea/ that was not the gitea binary we would be hurtying17:22
clarkbbut we don't so yay17:22
clarkband all the content in /var/gitea/data is bind mounted to /data so everything under that including /data/gitea and /data/git is preserved and that doesn't change17:24
clarkball that to have a really long winded way of saying: I think this doesn't affect us because we haven't been relying on the default of appending relative paths to the WORK_DIR. Instead every subcomponent path has been explicitly stuck under /data/git or /data/gitea with fully rooted paths. The default prefix management being changed doesn't impact us as a result17:25
clarkbif that makes sense to you I think we can proceed. I'm happy to remove the todo in the app.ini file first if you prefer as well17:26
clarkblet me know if that all makes sense and what you think the best next step is17:29
fungilooks that way, yes. happy to approve as-is and then remove the todo comment separately17:33
fungii can be around for a while still to check on the deployment17:33
clarkbthat works for me. I too can be around if we want ot approve it now17:33
clarkbdo you want to +A or should I?17:33
fungii've approved it now17:35
clarkbyou might also want to double check the secret was added on bridge. But I'm pretty confident I did that correctly :)17:35
opendevreviewJames E. Blair proposed opendev/system-config master: Update gerrit image to bookworm
opendevreviewJames E. Blair proposed opendev/system-config master: Update hound image to bookworm
opendevreviewJames E. Blair proposed opendev/system-config master: Update jinja-init image to bookworm
opendevreviewJames E. Blair proposed opendev/system-config master: Update gerrit image to bookworm
opendevreviewJames E. Blair proposed opendev/system-config master: Update zookeeper-statsd image to bookworm
fungiwill do17:51
fungiyeah, i see a gitea_oauth2_jwt_secret set in the gitea group vars17:53
fungiwhich matches the variable name in the template17:53
clarkbin theory the value itself shouldnt matter17:53
fungiright, since we're just setting but not consuming it anywhere17:56
opendevreviewJames E. Blair proposed opendev/system-config master: Test hound image build
opendevreviewJames E. Blair proposed opendev/system-config master: Update gerrit image to bookworm
opendevreviewJames E. Blair proposed opendev/system-config master: Update gerrit image to bookworm
opendevreviewJames E. Blair proposed opendev/system-config master: Update jinja-init image to bookworm
opendevreviewJames E. Blair proposed opendev/system-config master: Update zookeeper-statsd image to bookworm
opendevreviewJames E. Blair proposed opendev/system-config master: Update hound image to bookworm
clarkbfungi: I'm not seeing starlingx matrix room logs on yet. Maybe nothing has happened since the bot joined up?18:46
clarkbbut that may need further debugging18:46
fungiyeah, i wonder if it requires a restart18:56
clarkbthe gitea change should merge any minute now if it isn't going to hit its timeout18:56
fungiroot     1376910  0.0  1.0 138588 20468 ?        Ssl  Jul26  12:22 /usr/local/bin/python /usr/local/bin/eavesdrop18:58
clarkbprobably a good hunch on needing to restart it then18:58
opendevreviewMerged opendev/system-config master: Update to Gitea 1.20
fungi-rw-r--r-- 1 root root 1203 Aug 23 15:26 /var/lib/matrix-eavesdrop/config/config.yaml18:59
clarkbhere we go18:59
fungiso unless the daemon reads file updates automatically or gets soft reloaded somewhere, yeah i think we're missing a restart handler18:59
clarkbthe gitea deploy job just started. It should update gitea09 first. I'm watching it19:00
clarkb loads for me and reports the version I expect to see19:02
clarkbI'm able to clone the repo from that url as well19:02
clarkbI think the other major thing to check is replication once we've got things merging (we can also check refs/changes/ updates but not as easily)19:03
clarkbThe mobile theming thing seems to work now that we have a real cert too19:04
clarkb(this was expected but difficult to test until we upgraded)19:04
clarkb9,10, and 11 are updated at this point. And things look fine from here19:05
clarkbwe are waiting for containers to stop on 1219:06
clarkbone of three has stopped but the other two have not yet19:07
clarkbthere they go19:07
fungiall of the above are working for me19:10
clarkbwe are done with all but 14 now19:10
clarkball six servers are done and the job reports success19:14
clarkbthe service itself seems to work for me. Let's just keep an eye out for replication happenign19:14
fungiyeah, if i spot something merging soon i'll check behind it19:15
clarkbsounds good. Since nothing is obviously wrong I'm going to go eat a sandwich for lunch19:15
fungii too am going to pop away for some refreshments and return soonish19:17
opendevreviewHarry Kominos proposed openstack/diskimage-builder master: feat: Add new fail2ban elemenent
fungiokay, heading out, bbiab19:23
opendevreviewJames E. Blair proposed opendev/system-config master: Restart matrix-eavesdrop when config changes
opendevreviewJames E. Blair proposed opendev/system-config master: Update gerrit image to bookworm
clarkbcorvus: there was some weird gerrit + java 17 thing I made a fix for but we decided to stick with 11. I think 11 is still available in bookworm too. Let me dig it up20:08
clarkbits possible they fixed that problem too20:08
clarkbthey did fix it. Neat
corvusclarkb: okay cool, then my latest rev might work, let's see!  i'm not married to any of this, just trying to see what's easy or needs more work :)20:13
clarkbcorvus: reading the fix on that makes me wonder how helpful the java 17 chagne was. Basically you can't access a protected method directly when using reflection anymore. So you make a subclass that overrides the method and makes it public instead20:22
clarkbI don't do language design so I'm probably missing how it is better to make people do ^ than just do it directly20:22
corvusi guess more explicit is better?  but yeah, there's probably more subtle stuff going on20:24
fungiit's been quiet. none of the channels i'm lurking in (and there are a lot) have gerritbot announcements of changes merged after the gitea upgrades20:30
fungimaybe that's a sign i should review some things20:30
fungiclarkb: 892700 looks straightforward and fairly low impact if it goes wrong somehow, care for being a second pair of eyes on it?20:32
fungii'm going to try to whip up a restart handler for the matrix-eavesdrop bot and notify it on config changes20:36
clarkbdid we want to manually restart it to confirm that is the issue?20:36
corvusfungi: see
corvusmaybe that will solve 2 problems at once :)20:36
fungicorvus: wow, it's like you traveled back in time and wrote the thing i was about to make. or maybe i'm just that far behind on reviews. yeah, probably that actually20:37
corvusit does not reload its configuration under any circumstances, so i strongly suspect it's the issue.  i think restarting it to effectuate a fix is a good idea, or you could tag on a no-nop config file change after 892713 if you want to see it in action.20:37
corvusfungi: i ate lunch at my desk20:37
corvusthere may be spaghetti sauce on that change, watch out20:37
fungiwhat i did to test a similar recent change to the mailman3 containers was manually roll back the configuration and let it get reapplied to trigger the handler20:38
corvusoh that's a good idea20:38
fungieven cheaper, i stuck a comment line at the top of /var/lib/matrix-eavesdrop/config/config.yaml on the server so ansible will see a difference and replace it20:40
fungiso once that merges and deploys here shortly, we should ideally see it restart the service20:41
clarkband then we'll just need people to use the channels :) though it sounds like people are filtering in finally20:44
fungioh, for starlingx? yeah there's been conversations in about half those channels in the past day20:44
fungiit's picking up steam nicely20:44
clarkbfungi: for mm3 we run db migrations everytime the container starts iirc. Is that hte only upgrade step that needs to be done?20:45
fungii had a lengthy talk in about replicating zuul jobs locally, pointing out where the images we build can be found, our configuration for how we build images, what other issues to look out for...20:45
clarkb(also I think thos emigrations happen automatically)20:45
fungiclarkb: yes, db migrations are the only transformations of data. everything else is config changes we merge in ansible or updates to the software/defaults in docker image building20:46
clarkbok cool. I have to pop out for school things in about 45 minutes, not sure if we want ot proceed with upgrading mm3 today. Happy to help if I can though20:47
fungii'm around to keep an eye on the deployment if you approve it20:48
clarkbfungi: do we want to do that now or would you prefer waiting for tommorrow (your day is going to wind down soon too I think?)20:49
fungii'm fine waiting for tomorrow. it's still possible we'll find something not quite right with today's gitea upgrade so spacing them out makes sense for our own peace of mind20:49
clarkbsounds good20:50
fungifor the record, there is a mailman 3 upgrade guide, but there have been no special steps since 3.3.1:
clarkbgood to know20:53
fungiand the standard migrate/compress/collectstatic/compilemessages steps are handled on container start20:53
corvusthe gerrit image did actually pass tests with jdk17 and bookworm20:55
fungi is where we do the mailman automated upgrade steps20:56
fungiso everything in the upgrade guide seems to me to be covered20:57
fungiPython 3.11.5, 3.10.13, 3.9.18, and 3.8.18 just released, including some security fixes21:01
* fungi takes a recompiling break21:01
opendevreviewMerged opendev/system-config master: Update hound image to bookworm
fungiokay let's see if that replicated21:03
corvuspython:3.11-bullseye still appears to be .421:03
clarkbthey usually rebuild within a day21:03
fungic2ea42a86..1c5bee394  master            -> origin/master21:03
clarkbthe docker hub library images I mean21:03
clarkb yup its there in gitea21:04
fungihead of master i just remote updated is Merge "Update hound image to bookworm"21:04
fungiso yes, seems to have replicated fine21:04
opendevreviewMerged opendev/system-config master: Restart matrix-eavesdrop when config changes
fungiand after another remote update it's Merge "Restart matrix-eavesdrop when config changes"21:05
funginow to see if we get a service restart when that deploys21:05
fungimy dummy comment is still present at the top of /var/lib/matrix-eavesdrop/config/config.yaml for the moment21:05
clarkbfungi: jobs are waiting behind the hourly deployment jobs21:06
clarkbwe run eavesdrop hourl so the hourly job will do it and it may not have the update to restart containers in it?21:16
clarkbwill do it -> update the config file you modified21:16
clarkbyou might need to modify that file again if it doesn't restart things21:16
clarkblooks like it restarted 10 minutes ago21:28
fungiyeah, it worked21:28
clarkband there are log files. THough the cron that spits out the nicer format stuff hasn't run yet21:28
fungi[and there was much rejoicing]21:29

Generated by 2.17.3 by Marius Gedminas - find it at!