| frickler | actually I'm only out after tomorrow, but cleaning up those holds is fine anyway | 03:59 |
|---|---|---|
| opendevreview | Tony Breeds proposed opendev/system-config master: Also include tzdata when installing ARA https://review.opendev.org/c/opendev/system-config/+/923684 | 07:21 |
| opendevreview | Tony Breeds proposed opendev/system-config master: Update ansible-devel job to run on a newer bridge https://review.opendev.org/c/opendev/system-config/+/930538 | 07:21 |
| opendevreview | Merged opendev/irc-meetings master: Change to odd weeks for irc meetings for publiccloud-sig. https://review.opendev.org/c/opendev/irc-meetings/+/930334 | 09:24 |
| Clark[m] | A jitsi docker release just happened if anyone wants to try manually doing the update steps performed by our daily job maybe we can sort out the issue with updating those containers? | 14:15 |
| fungi | trying to recall what problem we've observed... container not starting correctly/at all? | 14:27 |
| clarkb | ya its like it does an incomplete restart and something fails to come up | 14:50 |
| clarkb | so probably need to manually step through theansible actions to see where it goes sideways then hopefully determine a resolution | 14:51 |
| fungi | clarkb: this right? https://opendev.org/opendev/system-config/src/branch/master/playbooks/roles/jitsi-meet/tasks/main.yaml#L83-L90 | 15:05 |
| fungi | seems like there's an obvious step missing, or else docker-compose is smarter than i realize | 15:05 |
| clarkb | fungi: yup. And ya the down shouldn't technically be necessary with docker compose figuring out what needs to be stopped and then started but maybe that is what the problem is for some reason | 15:06 |
| fungi | does up imply down first if the image has changed? | 15:06 |
| clarkb | yes | 15:06 |
| clarkb | only for the updated images | 15:06 |
| clarkb | and maybe if there are depenednecies of updated images? But maybe that is the issue some subtle bheavior where we really need to restart everythingand we are not? | 15:06 |
| fungi | okay. so yeah maybe that's it... basically i just need to log into the servers and do a pull followed by up and see if it breaks again? | 15:06 |
| fungi | or should i try an explicit down before the up? | 15:07 |
| clarkb | fungi: maybe we do a pull, down, then up and if that works we just update the ansible to match? | 15:08 |
| fungi | can do | 15:08 |
| clarkb | we know the current ansible behavior is likely to leave things not working so if a different process gives us a working end result we can switch to that and see if it improves | 15:08 |
| frickler | always doing the down would reset running calls, wouldn't it? | 15:08 |
| clarkb | oh yes we need to check if the images updated | 15:09 |
| clarkb | there are examples of that in other role | 15:09 |
| fungi | have we been seeing the same problem occur on both the primary and jvb servers? | 15:14 |
| frickler | I only remember the meetpad server, but not too confident about it | 15:18 |
| opendevreview | Stephen Finucane proposed openstack/project-config master: Retire x/nova-lxd https://review.opendev.org/c/openstack/project-config/+/930600 | 15:24 |
| opendevreview | Stephen Finucane proposed openstack/project-config master: Retire x/nova-lxd https://review.opendev.org/c/openstack/project-config/+/930600 | 16:23 |
| opendevreview | Stephen Finucane proposed openstack/project-config master: Remove x/nova-lxd from zuul https://review.opendev.org/c/openstack/project-config/+/930600 | 16:24 |
| opendevreview | Merged openstack/project-config master: Remove x/nova-lxd from zuul https://review.opendev.org/c/openstack/project-config/+/930600 | 16:51 |
| frickler | corvus: not sure if you had seen this already, but https://review.opendev.org/c/opendev/sandbox/+/915197 broke .gitreview by reverting to an old repo name. it has since been fixed, but there still is an old link in the readme | 17:00 |
| corvus | ack; maybe we should fix that and then make a safe tag that we can revert to | 17:01 |
| fungi | clarkb: i did a docker-compose pull on meetpad02 but it didn't say "downloaded newer image" | 17:37 |
| fungi | i suppose it may have already pulled them down earlier? | 17:37 |
| clarkb | maybe https://github.com/jitsi/docker-jitsi-meet/releases/tag/stable-9753 is what I got a notification for in email | 17:38 |
| clarkb | prosody has matchign images for that in docker hub | 17:38 |
| fungi | docker image list mentions images created 4 hours ago so that seems to correspond | 17:39 |
| corvus | docker image ls says that there is a newer copy of all 4 images locally, from 4 hours ago | 17:39 |
| clarkb | fungi: if you do a `docker image list` the images did uget pulled | 17:39 |
| corvus | docker inspect jitsi-meet-docker_web_1 says it's running the older image | 17:39 |
| fungi | yep, so it pulled the new images but didn't restart any containers | 17:40 |
| corvus | i would conclude the state is equivalent to docker-compose pull has been run but nothing else | 17:40 |
| clarkb | so either docker-compose isn't logging that anymore (but our experience with gitea upgrades yesterday indicates it must?) or something else pulled them? | 17:40 |
| fungi | yeah, i'm hesitant to make the restart conditional on output from the pull command if we have something else also pulling images | 17:40 |
| clarkb | I think more likely your pull did pull them | 17:41 |
| fungi | i'll compare on jvb02 | 17:41 |
| fungi | yeah, you're probably right, i only see the old image on jvb02 | 17:42 |
| clarkb | let me dig up an exampel from gitea jobs for the command output | 17:43 |
| clarkb | but docker-compose usually makes it pretty obvious if it is pulling something | 17:43 |
| fungi | is is what my terminal history looks like: https://paste.opendev.org/show/bPKTZBiMvftcGIrgl3pF/ | 17:43 |
| corvus | if the goal is "update if there is a newer image" maybe instead of relying on the pull, we should do some api comparisons. like "if the sha for the desired image tag does not match what the container is running, run down and up" | 17:43 |
| fungi | maybe i need to make it think it's not on a tty? | 17:43 |
| corvus | you can get docker to spit out json and run that through jq to do things like that | 17:44 |
| fungi | i wonder if docker-compose has magic "keep output pretty" functionality that hides download info | 17:44 |
| clarkb | https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_5f3/930217/1/gate/system-config-run-gitea/5f3f224/bridge99.opendev.org/ara-report/results/259.html | 17:44 |
| clarkb | the difference is having a term or not I Think | 17:44 |
| clarkb | if you watch it interactively it does say all that stuff but rather tahn scrolling output it is doing it in place | 17:44 |
| fungi | i'll do a 2>&1|cat on jvb02 for confirmation | 17:45 |
| clarkb | you're seeing just the last line for each image | 17:45 |
| clarkb | in your paste I mean | 17:45 |
| fungi | yeah, when i dump it through cat i do get all the lines rather than just the final one | 17:46 |
| fungi | Pulling jvb ... status: downloaded newer image fo... | 17:46 |
| opendevreview | Jeremy Stanley proposed opendev/system-config master: Explicitly down Jitsi-Meet containers on upgrade https://review.opendev.org/c/opendev/system-config/+/930637 | 17:52 |
| clarkb | fungi: cool so doing a down and up after a pull caused everything to come up cleanly? | 17:53 |
| fungi | seems that way | 17:53 |
| clarkb | ya docker ps shows things updated and I was able to join a room then quit without issue | 17:53 |
| fungi | i didn't get any errors from the up -d on either server | 17:54 |
| frickler | why is https://review.opendev.org/c/openstack/openstack-zuul-jobs/+/930599/3 still failing? the depends-on is merged, the parent shows no errors | 18:01 |
| clarkb | have we reloaded the zuul tenant to remove the project? | 18:03 |
| clarkb | it says last reconfigured 39 minutes ago /me does some math | 18:04 |
| fungi | its depends-on deployed to the server at 17:04, if that helps | 18:04 |
| clarkb | 30 minutes ago was 17:34, 39 is 17:25. The recheck happened at 10:19 | 18:04 |
| fungi | (probably slightly before, but that's when the deploy buildset reported) | 18:04 |
| clarkb | fungi: ya I think it takes about 20 minutes to full reload the tenant though which aligns with deploying at 17:04 and reloading the tenant at 17:25 | 18:05 |
| fungi | i agree | 18:05 |
| clarkb | anyway it may be worth rechecking one more time since the previous recheck was before the tenant reloaded. Then if it still fails we dig into logs | 18:05 |
| corvus | clarkb: mouseover the reconfigure time and you'll get the real time | 18:05 |
| clarkb | oh cool 17:24 UTC ya | 18:05 |
| corvus | (every timestamp in zuul-web should have an absolute value) | 18:05 |
| clarkb | oh its the parent change! | 18:06 |
| clarkb | https://review.opendev.org/c/openstack/openstack-zuul-jobs/+/930615 has configuration error not the depends on | 18:06 |
| fungi | aha! | 18:06 |
| fungi | that explains it | 18:06 |
| clarkb | also the fix is to put the one we're rechecking first | 18:06 |
| fungi | the parent was passing in check earlier, but something merged to break it | 18:06 |
| clarkb | yes nova-lxd was removed so we can't refer to it anymore which the change we were originally looking at should fix | 18:07 |
| fungi | right, didn't see that until i approved it moments ago | 18:08 |
| frickler | the parent didn't have the error when I looked at it, but that explains it, thx | 18:09 |
| fungi | yeah, the error wasn't reported on the parent until i hit approve on it | 18:11 |
| frickler | but I think this needs another cleanup patch, then, the error location isn't touched by 930599 | 18:11 |
| clarkb | ah yup looks like another job? | 18:13 |
| clarkb | actually no I think thats it | 18:14 |
| clarkb | I would rebase and flip the order of those two changes and see if zuul complains any further | 18:14 |
| frickler | ah, yes, I misread the zuul note. let me just shuffle things, needs some local editing due to conflicts | 18:17 |
| NeilHanlon | infra-root: though I'm sure it's not, one may want to uninstall cups from your servers if it's there. | 18:21 |
| * NeilHanlon disappears into the night | 18:21 | |
| fungi | aha, so that was the answer to the recent riddle | 18:24 |
| frickler | reordering the stack seems to have severly confused gerrit's "Relation Chain" view :-/ I think I'll need to rebase the whole stack manually | 18:25 |
| NeilHanlon | fungi: at least partially, it seems. the disclosure has been moved up to... an hour from now | 18:57 |
| NeilHanlon | something something cat something something bag | 18:57 |
| fungi | indeed | 18:58 |
| * fungi sighs | 18:58 | |
| fungi | luckily i can't find anywhere we need to be concerned | 19:00 |
| frickler | soomething like https://github.com/OpenPrinting/cups-browsed/issues/36 likely? | 19:00 |
| Clark[m] | Oh cool I get lunch early so I can go on a bike ride and the fun starts | 19:08 |
| fungi | in our case it seems to have been much ado about nothing | 19:12 |
| mordred | the discussion in that PR is very respectful and productive | 19:15 |
| mordred | s/PR/issue/ | 19:16 |
| fungi | isn't it tho | 19:17 |
| clarkb | desktop flavors of various distro probably do have cups running though so check your laptops/desktops | 19:20 |
| clarkb | the writeup is public now | 20:01 |
| fungi | whee! | 20:03 |
| NeilHanlon | https://www.evilsocket.net/2024/09/26/Attacking-UNIX-systems-via-CUPS-Part-I/ | 20:14 |
| NeilHanlon | weeee indeed | 20:14 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!