| opendevreview | Clark Boylan proposed opendev/system-config master: Update Gitea to version 1.22 https://review.opendev.org/c/opendev/system-config/+/920580 | 02:31 |
|---|---|---|
| clarkb | dtantsur: fungi: I think that is a warning because it isn't a fatal handling of the situation within zuul. For example a lot of things are a hard stop and are fatal like not being abel to find a nodeset or a secret. But in this case if you recheck after the parent lands it will run hence the warning | 02:32 |
| clarkb | as a heads up I'm likely going to get a late start tomorrow. It has been really hot here lately so if I want to do anything outside I need to do it first thing in the morning which I'll likely end up doing tomorrow | 02:36 |
| frickler | infra-root: seems openmetal people updated the glance config as discussed, so I'm bumping the max-servers setting for the cloud now again and will watch the results | 05:04 |
| mattoliver | hey all, meetpad seems to be down. I've been using it mentor some students in qatar about openstack developement as summer project for them. Is it a service that is only available around PTG times (And that's why it's down) or is it having issues? No stress either way, it was just super convenient to talk to the students. I can always use some other service but meetpad's etherpad connection was/is pretty sweet :) | 06:16 |
| tonyb | hey mattoliver let me look | 06:17 |
| mattoliver | thanks tonyb | 06:17 |
| tonyb | It's totally having issues | 06:23 |
| tonyb | mattoliver: try now? | 06:25 |
| tonyb | mattoliver: https://meetpad.opendev.org/isitbroken | 06:25 |
| tonyb | mattoliver: Looks like it's working for me. No errors in the logs etc. | 06:56 |
| mattoliver | looks like it's working now! Thanks tonyb !! | 07:15 |
| mattoliver | Sorry it took so long to check, was in the meeting. | 07:16 |
| *** thuvh1 is now known as thuvh | 09:06 | |
| tonyb | mattoliver: all good. hopefully it's working? | 10:48 |
| *** ykarel__ is now known as ykarel | 10:54 | |
| opendevreview | Stephen Finucane proposed openstack/project-config master: Add Python 3.8 requirements job https://review.opendev.org/c/openstack/project-config/+/923639 | 12:17 |
| opendevreview | Stephen Finucane proposed openstack/project-config master: Add Python 3.12 (and 3.11) requirements job https://review.opendev.org/c/openstack/project-config/+/923640 | 12:17 |
| opendevreview | Stephen Finucane proposed openstack/project-config master: Add Python 3.8 requirements job https://review.opendev.org/c/openstack/project-config/+/923639 | 12:25 |
| opendevreview | Stephen Finucane proposed openstack/project-config master: Add Python 3.12 (and 3.11) requirements job https://review.opendev.org/c/openstack/project-config/+/923640 | 12:25 |
| fungi | tonyb: any indication what happened to meetpad earlier? or any guess as to how long it was offline? | 12:55 |
| frickler | the job has been failing for 5 days, I assume this to be related, but didn't check logs yet https://zuul.opendev.org/t/openstack/builds?job_name=infra-prod-service-meetpad | 12:59 |
| fungi | oh, probably so | 13:00 |
| frickler | the job was failing in TASK [jitsi-meet : Run docker-compose up] ************************************** | 13:01 |
| frickler | looks like upstream released new docker images around that time, but let's wait for tonyb to confirm what he did https://github.com/jitsi/docker-jitsi-meet/releases/tag/stable-9584-1 | 13:10 |
| frickler | openmetal cloud seems to be doing fine so far, reaching the full 50 nodes being in use at times. I'll wait until tomorrow and then remove nl02 from emergency https://grafana.opendev.org/d/75a5d1cf58/nodepool3a-openmetal?orgId=1&from=now-24h&to=now | 13:18 |
| fungi | excellent | 13:28 |
| fungi | thanks for keeping up with that! | 13:28 |
| clarkb | ++ thank you for keeping that moving forward | 15:30 |
| clarkb | I'll get a node hold up for the gitea 1.22.1 upgrade. Then after that is done we can start planning for the db doctoring. Also there is an etherpad update that I'll push change(s) for shortly | 15:31 |
| clarkb | heh etherpad added back apikey support.... | 15:36 |
| clarkb | the default is oauth 2.0 which we already converted to so I think we acn stick with that | 15:36 |
| opendevreview | Clark Boylan proposed opendev/system-config master: DNM intentional Gitea failure to hold a node https://review.opendev.org/c/opendev/system-config/+/848181 | 15:45 |
| clarkb | ok a hold has been requested for ^ | 15:46 |
| Madelino | hello | 16:00 |
| clarkb | hello | 16:01 |
| opendevreview | Clark Boylan proposed opendev/system-config master: Upgrade to etherpad 2.1.1 https://review.opendev.org/c/opendev/system-config/+/923661 | 16:04 |
| clarkb | if ^ passes testing I'll put a hold in place for it as well | 16:04 |
| opendevreview | Clark Boylan proposed opendev/system-config master: DNM force etherpad failure to hold node https://review.opendev.org/c/opendev/system-config/+/840972 | 16:38 |
| clarkb | autohold is in place for ^ now too | 16:39 |
| clarkb | held etherpad 2.1.1 is at 149.202.167.222 and held gitea is at https://104.130.219.4:3081/opendev/system-config You will need to edit /etc/hosts for etherpad | 17:43 |
| clarkb | I'm using the clarkb-test pad on the held etherpad | 17:44 |
| clarkb | etherpad is looking good to me. I suspect we can probably go ahead and land that one once people have done reviews | 17:46 |
| clarkb | the gitea upgrade will require more careful review. Even the chagne log admits it is a very big update :) that asid we have decent testing so probably not much to worry about | 17:48 |
| frickler | fyi several gate failures indicate pypi is having some issues, these jobs passed just hours ago https://zuul.opendev.org/t/openstack/build/b818f72a12fb4fefae22ffa0a111b0f9 https://zuul.opendev.org/t/openstack/build/a0395437c4b44d01bc95685abaf6be0a | 18:42 |
| clarkb | I'm going to take the openmetal stuff off the meeting agenda since that is back to working again | 20:06 |
| fungi | sounds good, thanks! | 20:07 |
| clarkb | hrm no corvus here | 20:09 |
| clarkb | corvus if you see this my some magic in the matrix bridge do you want to keep the nodepool in zuul stuff on the agenda? | 20:09 |
| corvus | clarkb: i can see you! you can strike it from the agenda | 20:16 |
| clarkb | oh cool so thats a bridge optimization to contineu to show you content without having you actually be in the room | 20:17 |
| clarkb | thanks, I figured we could add it back once we start having changes and decisions to make in opendev | 20:17 |
| *** corvus is now known as notcorvus | 20:18 | |
| *** notcorvus is now known as corvus | 20:18 | |
| clarkb | any other updates I should make to the meeting agenda? I'll try and hold off sending it until the very end of my day as that seems to work better for tonyb's schedule | 20:24 |
| tonyb | clarkb: I dropped the collecting backlog point from the agenda as there isn't anything to talk about there. | 21:59 |
| tonyb | clarkb: I don't have any other updates/edits | 21:59 |
| tonyb | clarkb: so send it whenever you get a chance | 22:00 |
| clarkb | tonyb: thanks! | 22:08 |
| clarkb | and sent | 22:13 |
| clarkb | interestingly I dont' see it in my inbox yet. Maybe we do still have some mail delivery delays? | 22:16 |
| tonyb | fungi, frickler: The web and parapsody services had both exited with SIGTERM, but the containers were still present which was stopping them from coming back with a `docker-compose up`. | 22:17 |
| tonyb | The timing is suspicious, WRT to the new container release | 22:18 |
| tonyb | clarkb: I have it in mine | 22:18 |
| clarkb | tonyb: ack maybe something with my mail service then | 22:18 |
| clarkb | tonyb: re meetpad that almost sounds like sigterms were issued by docker compose to replace the conatiners but then docker never recognized the containers as shutdown? | 22:19 |
| clarkb | it delivered for me 6 minutes after I sent it. Could be nothing to worry about if it was something on my mailserver end | 22:20 |
| tonyb | clarkb: Yeah given the timing that's my thinking | 22:21 |
| tonyb | clarkb: we don't have great logs to confirm that though | 22:21 |
| tonyb | Oh interesting there was a container tag 6 days ago and another 5 days ago | 22:22 |
| clarkb | maybe the older one was broken causing this issue then they pushed an update a day later? | 22:27 |
| tonyb | Perhaps. I'll keep looking | 22:27 |
| tonyb | Somewhat related, we don't seem to have anything like: https://opendev.org/opendev/system-config/src/branch/master/playbooks/roles/etherpad/templates/docker-compose.yaml.j2#L29-L32 | 22:28 |
| clarkb | I think the jitsi meet docker compose file was heavily adapted from upstram so ya may be missing stuff that is more opendev psecific likethat | 22:29 |
| tonyb | for the meetpad containers which I think is part of the magic to get the container logs into /var/log/conatiners/* | 22:29 |
| clarkb | yes that is one half of the magic for the /var/log/conatiners stuff ( I thin kthe other half lives in syslog config so that it knows what to do with docker logs sent to it) | 22:30 |
| tonyb | clarkb: Okay I'll add the logging stuff | 22:30 |
| tonyb | clarkb: Yeah we have the rsyslog part in place | 22:31 |
| clarkb | ya I think we just set that globally to make it easy | 22:31 |
| opendevreview | Tony Breeds proposed opendev/system-config master: Add logging tags to various jitsi_meet containers https://review.opendev.org/c/opendev/system-config/+/923680 | 22:42 |
| fungi | for the jitsi docker configs, i tried to be very explicit about recording in comments what commits they originated in and what edits we made to them, in order to make future updating/resyncing easier | 22:42 |
| opendevreview | Tony Breeds proposed opendev/system-config master: Add logging tags to various jitsi_meet containers https://review.opendev.org/c/opendev/system-config/+/923680 | 22:45 |
| tonyb | ^^ should add something along those lines. | 22:45 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!