| tonyb | clarkb: the plan is to be in MN during the pre-PTG | 01:56 |
|---|---|---|
| *** darmach45 is now known as darmach4 | 08:20 | |
| stephenfin | fungi: there are more changes (proper pyproject.toml support), but I don't think I'll be able to get them done in the next two weeks, so we should probably release what we have | 09:23 |
| stephenfin | iirc pbr isn't released with the usual tooling so that's presumably on ye to do? | 09:23 |
| frickler | stephenfin: what about https://review.opendev.org/c/openstack/pbr/+/942594 and the testing that fungi asked for? do you want to work on that or maybe kevko can? | 11:54 |
| gouthamr | o/ are we aware of any problems with meetpad today? | 14:16 |
| gouthamr | we have a bunch of people getting booted | 14:16 |
| fungi | this is the first i've heard it mentioned | 14:21 |
| Clark[m] | I used meetpad yesterday with no issues. I don't see any emails announcing jitsi meet releases so don't think it updated overnight but I haven't checked the server directly | 14:21 |
| fungi | i'm on my way out to run some quick errands so can't look into it just yet | 14:21 |
| fungi | are they all coming from the same network? same region of the world? any other commonality | 14:22 |
| clarkb | ya any more info you can share to help narrow this down will be helpful. I'll spot check the servers momentarily | 14:43 |
| clarkb | containers have been up for 12 days confirming that we haven't updated them since yesterday when fungi and I successfully used meetpad | 14:44 |
| opendevreview | James E. Blair proposed zuul/zuul-jobs master: Synchronize upload-image-s3 to upload-image-swift https://review.opendev.org/c/zuul/zuul-jobs/+/956378 | 14:45 |
| clarkb | looking at the jicofo logs on meetpad02 I see what appears to be clients leaving normally as far as the server is concerned | 14:48 |
| clarkb | ah the jvb logs indicate client connectivity checks are failing | 14:50 |
| clarkb | "is considered disconnected. No activity | 14:51 |
| clarkb | and "(stream.RTP), failing" | 14:51 |
| clarkb | however that should just be for the video stream. It looks like the udp packets are not flowing to those clients (possible firewall problems?) | 14:51 |
| clarkb | gouthamr: I think my suggestion would be to disable video and see if that helps since the jvb portion is just video | 14:52 |
| gouthamr | unfortunately we had to switch :( because we were in the thick of a design call.. hate using Google Meet, but we resorted to it | 14:55 |
| opendevreview | James E. Blair proposed openstack/project-config master: Update opendev and zuul tenants to Ansible 11 https://review.opendev.org/c/openstack/project-config/+/956686 | 14:55 |
| clarkb | I guess it would be useful to know if this client had ever used the service successfully before from that network location. But I'm thinking this is a network problem on the client side or at least between the client and server now | 14:56 |
| gouthamr | thank you, I’ll check on the call | 14:58 |
| clarkb | infra-root I've updated the pre ptg etherpad with three time blocks: october 7 1800-2000 UTC (this replaces our meeting that week), october 8 1500-1700 UTC and october 9 1500-1700 UTC. Since tonyb expects to be in MN that week I think those times should work for all of us timezone wise. Let me know if we need small adjustments to account for existing obligations | 14:58 |
| clarkb | https://etherpad.opendev.org/p/opendev-preptg-october-2025 is the etherpad | 14:58 |
| gouthamr | one of the attendees was from China, and another from India - will gather their browser deets etc | 14:58 |
| clarkb | gouthamr: but also if they want to test later with video disabled (assuming it wasn't already) that might be helpful too. As the only problems I can see in the log are in the jvb service which is what processes video streams | 14:59 |
| clarkb | stephenfin: pbr uses the normal release process | 14:59 |
| opendevreview | James E. Blair proposed zuul/zuul-jobs master: Synchronize upload-image-s3 to upload-image-swift https://review.opendev.org/c/zuul/zuul-jobs/+/956378 | 15:04 |
| fungi | "normal" for openstack release process anyway | 15:09 |
| fungi | propose a tag to the openstack/releases repo, get the oslo ptl or release liaison to +1 it, release managers approve | 15:10 |
| fungi | clarkb: should i go ahead with approving the gitea upgrade change? i'm going to grab lunch real quick but will be around long before it merges | 15:11 |
| clarkb | fungi: ++ I'm around now | 15:11 |
| fungi | in it goes | 15:11 |
| fungi | popping out for a bit, will be back within the hour | 15:12 |
| stephenfin | clarkb: fungi: ack, I proposed https://review.opendev.org/c/openstack/releases/+/956699 | 15:25 |
| clarkb | stephenfin: thanks the tag version and commit hash both seem fine to me. I +1'd but that is a normal code review. | 15:27 |
| stephenfin | ack | 15:28 |
| opendevreview | Merged openstack/project-config master: Update opendev and zuul tenants to Ansible 11 https://review.opendev.org/c/openstack/project-config/+/956686 | 15:43 |
| corvus | clarkb: i'd like to try migrating the image upload role to zuul-jobs again -- when you have a chance, could you take a look at https://review.opendev.org/956377 and https://review.opendev.org/956378 ? | 15:44 |
| corvus | that should bring the zuul-jobs version of the role up to what we're doing in zuul-providers, and then we can try the switch again | 15:44 |
| corvus | that second change is an update to the s3 role, but i think it's worth reviewing and merging together, since it parallels the swift role, but we have actual testing for s3, unlike swift :( | 15:45 |
| clarkb | corvus: I guess the connection backoff stuff didn't end up helping us with the specific issue we had but it may help us in general so we're keeping it? | 15:50 |
| corvus | clarkb: oh... didn't it? i thought it was like one of the 3 or 4 things we were having problems with at the same time | 15:51 |
| corvus | or... maybe it just turned it into a different error? :) | 15:51 |
| clarkb | I thought we got back the http return code that caused requests to not backoff | 15:51 |
| corvus | oh yeah. anyway, yes i think it's harmless to keep. | 15:52 |
| clarkb | but now that I'm thinking about it we didn't change anything with the swift deployments so maybe this did help. Or the cloud magically fixed it behind the scenes. In any case I don't think it hurts | 15:52 |
| clarkb | corvus: both lgtm but I left a note on https://review.opendev.org/c/zuul/zuul-jobs/+/956378 you may want to consider for an update or followup | 15:55 |
| corvus | oh yeah let's fix that now | 15:58 |
| opendevreview | James E. Blair proposed zuul/zuul-jobs master: Synchronize upload-image-s3 to upload-image-swift https://review.opendev.org/c/zuul/zuul-jobs/+/956378 | 16:00 |
| clarkb | apparently if you run btrfs with linux 6.15.3+ you'll want to grab a fix for log tree corruption | 16:04 |
| fungi | okay, back, i see 956591 is estimating another 23 minutes to finish system-config-run-gitea in the gate | 16:09 |
| opendevreview | Clark Boylan proposed opendev/system-config master: Reapply "Move system-config irc bots into quay.io" https://review.opendev.org/c/opendev/system-config/+/956706 | 16:15 |
| clarkb | that moves about half of the bots. statusbot, ptgbot and gerritbot are separate | 16:17 |
| stephenfin | frickler: sorry, I missed your reply earlier. I was hoping someone else could do that. I just rebased it | 16:25 |
| fungi | yeah, i didn't have time to fix up the change either, was hoping the author would | 16:26 |
| opendevreview | Merged zuul/zuul-jobs master: Update upload-image-swift https://review.opendev.org/c/zuul/zuul-jobs/+/956377 | 16:28 |
| opendevreview | James E. Blair proposed opendev/zuul-providers master: Switch to zuul-jobs upload-image-swift https://review.opendev.org/c/opendev/zuul-providers/+/951018 | 16:34 |
| corvus | okay there's the update | 16:34 |
| opendevreview | Merged zuul/zuul-jobs master: Synchronize upload-image-s3 to upload-image-swift https://review.opendev.org/c/zuul/zuul-jobs/+/956378 | 16:35 |
| clarkb | corvus: does that need to set the build image name? | 16:35 |
| clarkb | specifically here: https://review.opendev.org/c/opendev/zuul-providers/+/951018/3/playbooks/opendev-build-diskimage-base/post-inner.yaml | 16:36 |
| clarkb | or do the new defaults cover that for us already? | 16:36 |
| corvus | clarkb: i think the new defaults should cover that; in all the situations this will run in, we'll use the zuul value | 16:37 |
| clarkb | got it | 16:37 |
| opendevreview | James E. Blair proposed openstack/project-config master: Remove nodepool configuration/elements https://review.opendev.org/c/openstack/project-config/+/956184 | 16:42 |
| corvus | https://review.opendev.org/c/openstack/openstack-zuul-jobs/+/956707 Remove project-config-dib job [NEW] is a companion to that change ^ | 16:44 |
| opendevreview | Merged opendev/system-config master: Upgrade Gitea to 1.24.4 https://review.opendev.org/c/opendev/system-config/+/956591 | 16:52 |
| clarkb | finally. | 16:53 |
| fungi | sure took its time | 16:54 |
| fungi | i expect we have options for optimizing that job | 16:55 |
| clarkb | gitea09 is updated | 16:56 |
| clarkb | browsing and git clone seem to work | 16:57 |
| clarkb | all the giteas are updated now. The job should be finishing up momentarily | 17:06 |
| fungi | yep, all working for me | 17:06 |
| clarkb | and the buildset was successful. | 17:07 |
| opendevreview | Clark Boylan proposed opendev/gerritbot master: Reapply "Move gerritbot to quay.io" https://review.opendev.org/c/opendev/gerritbot/+/956712 | 17:10 |
| opendevreview | Clark Boylan proposed opendev/system-config master: Pull gerritbot from quay.io https://review.opendev.org/c/opendev/system-config/+/956714 | 17:13 |
| clarkb | https://opendev.org/opendev/system-config/commit/db552d55763598f34a554d2a1594ec77462cd294 replication seems to be working too | 17:13 |
| fungi | clarkb: speaking of images on quay, 956354 wants to mirror another version of docker registry into our quay account | 17:21 |
| clarkb | oh i thought I had +2'd that one. Let me rereview after the comments | 17:22 |
| fungi | doesn't look like you reviewed it | 17:25 |
| clarkb | ya I must've just looked at it then not commented. probably because Friday was busy | 17:25 |
| clarkb | anyway it looks like this was a shot in the dark attempt at fixing a thing that was actually a problem in the client not the registry | 17:26 |
| clarkb | bbezak: should we abandon that change? | 17:26 |
| bbezak | indeed it was different problem. however at some point we would want newer (maintained) registry version - 3. | 17:27 |
| bbezak | I think we can proceed with that change then | 17:28 |
| bbezak | clarkb | 17:30 |
| clarkb | ya so for now lets not land it until we decide we need it | 17:30 |
| bbezak | ok | 17:31 |
| opendevreview | Clark Boylan proposed opendev/statusbot master: Reapply "Move statusbot to quay.io" https://review.opendev.org/c/opendev/statusbot/+/956716 | 17:43 |
| opendevreview | Clark Boylan proposed opendev/system-config master: Fetch statusbot from quay https://review.opendev.org/c/opendev/system-config/+/956717 | 17:44 |
| opendevreview | Clark Boylan proposed opendev/system-config master: Pull ptgbot from quay.io https://review.opendev.org/c/opendev/system-config/+/956719 | 17:50 |
| clarkb | ok that should be all of the changes necessary to update eavesdrop images to publish to quay then have eavesdrop pull them from quay cc infra-root | 17:51 |
| opendevreview | Jeremy Stanley proposed openstack/project-config master: Prepare for retirement of RefStack repositories https://review.opendev.org/c/openstack/project-config/+/947859 | 17:51 |
| opendevreview | Merged openstack/project-config master: Add jobs for openstack/grian-ui repo https://review.opendev.org/c/openstack/project-config/+/948250 | 17:52 |
| clarkb | the refstack wind down change runs a lot of jobs | 18:02 |
| fungi | yeah... | 18:05 |
| opendevreview | Merged openstack/project-config master: Drop retired openstack/identity-api https://review.opendev.org/c/openstack/project-config/+/955419 | 18:05 |
| fungi | also touches a lot of files, so i guess that's expected | 18:05 |
| opendevreview | Merged openstack/project-config master: Add notification about heat-translator and tosca-parser https://review.opendev.org/c/openstack/project-config/+/955420 | 18:08 |
| opendevreview | Merged openstack/project-config master: Remove networking-midonet https://review.opendev.org/c/openstack/project-config/+/919415 | 18:08 |
| clarkb | ya I don't think it is a problem. More just unexpected | 18:10 |
| clarkb | I think it is due to modifying the run-base playbook and the templates for jobs | 18:10 |
| clarkb | fungi: fyi I went ahead and approved https://review.opendev.org/c/openstack/ptgbot/+/930781 and its parent | 18:22 |
| fungi | thanks | 18:22 |
| clarkb | I think the parent is necessary to fix pep8 jobs which failed on my upload to quay change | 18:22 |
| fungi | hopefully they still work almost a year later | 18:23 |
| clarkb | and we're far enough out from the next ptg that now seems like a good time tostart getting those in. The actual switch to py312 change failed but no logs anymore so I'll recheck that one | 18:23 |
| fungi | k | 18:24 |
| fungi | okay, they did actually merge. good | 18:31 |
| opendevreview | Merged opendev/system-config master: Wind down refstack.openstack.org https://review.opendev.org/c/opendev/system-config/+/947856 | 19:06 |
| opendevreview | James E. Blair proposed openstack/project-config master: Remove nodepool configuration/elements https://review.opendev.org/c/openstack/project-config/+/956184 | 19:38 |
| opendevreview | James E. Blair proposed openstack/project-config master: Stop running project-config-dib https://review.opendev.org/c/openstack/project-config/+/956726 | 19:38 |
| opendevreview | Merged opendev/system-config master: Apply UserAgentFilter to every vhost on static https://review.opendev.org/c/opendev/system-config/+/955616 | 19:39 |
| opendevreview | Merged openstack/project-config master: Stop running project-config-dib https://review.opendev.org/c/openstack/project-config/+/956726 | 20:35 |
| clarkb | fungi: I think the system-config container image move change for irc bots probably wants to go in on friday to avoid restarting limnoria at a bad time. But ptgbot, gerritbot, and statusbot can probably go in at any time? Any concerns with the timing of those three? | 20:45 |
| fungi | none whatsoever | 21:06 |
| fungi | the sooner the better | 21:06 |
| clarkb | fungi: maybe we should start with ptgbot since it is the least used right now? | 21:09 |
| clarkb | I can +A the change for that one in order if you think that is a good start | 21:09 |
| fungi | or i can, just didn't know if we were ready to start monitoring those | 21:12 |
| clarkb | ya I think so. The only primary concern I have is not restarting limnoria during a meeting so that change is a good firday option. The others are probably ok nowish | 21:12 |
| fungi | ptgbot change is approved, once that's done i can approve the corresponding system-config one | 21:13 |
| clarkb | awesome | 21:17 |
| clarkb | the change has merged now we need the iamge to promote | 21:21 |
| fungi | yep | 21:21 |
| fungi | it finished, yeah? | 21:22 |
| clarkb | https://quay.io/repository/opendevorg/ptgbot/manifest/sha256:b8edacc3782a0dacfb965fe0b37d178f311cfbc7b0917885c4e91361e531365b that just updated so I think we'regood to proceed now | 21:22 |
| clarkb | https://quay.io/repository/opendevorg/ptgbot?tab=tags has the last modified timestamp in its table | 21:22 |
| fungi | it's in | 21:22 |
| fungi | i also went ahead and approved the publishing changes for statusbot and gerritbot | 21:24 |
| opendevreview | Merged opendev/statusbot master: Reapply "Move statusbot to quay.io" https://review.opendev.org/c/opendev/statusbot/+/956716 | 21:35 |
| opendevreview | Merged opendev/gerritbot master: Reapply "Move gerritbot to quay.io" https://review.opendev.org/c/opendev/gerritbot/+/956712 | 21:36 |
| fungi | promote completed for both of those as well | 21:38 |
| clarkb | https://quay.io/repository/opendevorg/gerritbot?tab=tags and https://quay.io/repository/opendevorg/statusbot?tab=tags both show updated timestamps | 21:40 |
| opendevreview | Merged opendev/system-config master: Pull ptgbot from quay.io https://review.opendev.org/c/opendev/system-config/+/956719 | 21:51 |
| clarkb | that has deployed and docker ps -a seems to show it running the quay image on eavesdrop02 | 21:57 |
| clarkb | so I think that is happy | 21:57 |
| clarkb | openinfraptg is in #openinfra-events. Probably good enough as a verification? | 21:58 |
| clarkb | fungi: should we proceed wtih gerritbot and statusbot? Those are a bit moe impactful | 22:01 |
| fungi | 21:55 --> openinfraptg (~openinfra@eavesdrop02.opendev.org) has joined #openinfra-events | 22:01 |
| fungi | it dropped and rejoined as intended | 22:01 |
| fungi | i've approved statusbot/gerritbot config changes now | 22:02 |
| opendevreview | Clark Boylan proposed opendev/system-config master: Run eavesdrop jobs when gerritbot role updates https://review.opendev.org/c/opendev/system-config/+/956735 | 22:27 |
| clarkb | I noticed ^ after you approved those two changes. Not an urgent update as statusbot does trigger the jobs and is queued up behind the gerritbot change | 22:27 |
| opendevreview | Merged opendev/system-config master: Pull gerritbot from quay.io https://review.opendev.org/c/opendev/system-config/+/956714 | 22:28 |
| clarkb | ya as expected ^ that isn't triggering the deployment. but the next change for statusbot should and it will update both | 22:29 |
| fungi | oh good | 22:31 |
| fungi | and i've approved 956735 so that won't be an issue in the future. thanks! | 22:32 |
| clarkb | and we can plan to migrate the other three images within system-config on friday to avoid meeting interference | 22:38 |
| opendevreview | Merged opendev/system-config master: Fetch statusbot from quay https://review.opendev.org/c/opendev/system-config/+/956717 | 22:40 |
| clarkb | both statusbot and gerritbot have been restared on the quay image | 22:44 |
| clarkb | should we #status log Moved statusbot image to quay as a test? | 22:44 |
| fungi | go for it | 22:45 |
| fungi | i saw them both exit and statusbot rejoin too | 22:45 |
| fungi | obviously gerritbot will rejoin when it feels it has something to say to us | 22:45 |
| clarkb | #status log Moved statusbot image to quay.io | 22:45 |
| clarkb | when 956735 lands we should get a gerritbot message | 22:46 |
| fungi | statusbot may still be busy joining the remaining channels in its list | 22:47 |
| opendevstatus | clarkb: finished logging | 22:47 |
| opendevreview | Merged opendev/system-config master: Run eavesdrop jobs when gerritbot role updates https://review.opendev.org/c/opendev/system-config/+/956735 | 22:47 |
| fungi | there it is | 22:47 |
| clarkb | I went looking for statusbot logs and couldn't find them. Any idea where they go? | 22:47 |
| fungi | also confirmed https://fosstodon.org/@opendevinfra and https://wiki.openstack.org/wiki/Infrastructure_Status got your log entry | 22:47 |
| fungi | /var/log/statusbot/ has some logs | 22:48 |
| clarkb | aha I see that bind mount now too | 22:48 |
| clarkb | we also have it set up to journald log but it must not emit anything to stdout | 22:49 |
| clarkb | ok I think these are happy. Thanks | 22:49 |
| fungi | looks like we're through all the ones that won't impact meetbot, so i'm going to call it a night | 22:49 |
Generated by irclog2html.py 4.0.0 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!