| *** elodilles is now known as elodilles_pto | 06:33 | |
| *** jroll03 is now known as jroll0 | 07:39 | |
| *** tkajinam is now known as Guest8639 | 09:04 | |
| fungi | #status log Restarted the opendevreview IRC bot (gerritbot) in order to force it to reconnect to the event stream following this weekend's unexpected Gerrit service outage | 13:02 |
|---|---|---|
| opendevstatus | fungi: finished logging | 13:03 |
| fungi | apparently gerritbot-matrix didn't need restarting, since i see it reporting change events in #zuul:opendev.org after the outage | 13:03 |
| opendevreview | Jeremy Stanley proposed opendev/gerritbot master: Feature: Keep alive interval of 60 by default https://review.opendev.org/c/opendev/gerritbot/+/941117 | 15:22 |
| opendevreview | Merged openstack/project-config master: Revert "Temporarily remove zuul-providers from zuul tenant" https://review.opendev.org/c/openstack/project-config/+/940902 | 15:43 |
| fungi | deploy for ^ failed | 16:09 |
| fungi | TASK [sync-project-config : Sync project-config repo] | 16:09 |
| fungi | fatal: [zm07.opendev.org]: FAILED! | 16:10 |
| fungi | but it's no_log so that's all i know without investigating on the server itself | 16:10 |
| fungi | i don't see right off that there's anything broken in /opt/project-config | 16:11 |
| fungi | maybe a random git connection error hitting gitea | 16:12 |
| clarkb | I think that sync uses rsync from bridge to the host | 16:12 |
| clarkb | but I'm not sure of that | 16:12 |
| fungi | cloning project-config from gitea in my homedir on zm07 is kinda slow | 16:13 |
| fungi | but yeah, i think you're right about it being pushed from bridge instead for consistency in the deploy job | 16:13 |
| clarkb | yes that task uses the synchronize module | 16:14 |
| fungi | i guess we should just keep an eye out for similar failures | 16:15 |
| clarkb | /opt isn't full? you indicated it looked ok | 16:15 |
| fungi | 7% used | 16:16 |
| clarkb | odd | 16:16 |
| fungi | no errors in dmesg either | 16:16 |
| clarkb | I Guess network trouble is the most likely explanation in that case | 16:16 |
| clarkb | and now it is time for reboots for local weekly updates. Back soon | 16:16 |
| opendevreview | Clark Boylan proposed opendev/system-config master: Switch codesearch to journald logging https://review.opendev.org/c/opendev/system-config/+/941129 | 16:29 |
| opendevreview | Clark Boylan proposed opendev/system-config master: Test codesearch on Noble https://review.opendev.org/c/opendev/system-config/+/941130 | 16:29 |
| clarkb | if 941130 comes back clean I'll boot a new codesearch server using our noble image | 16:30 |
| clarkb | codesearch does not currently mount things with a volume. I'll preserve that behavior. If space gets tight we should be able to move the git repo content and indexes onto /opt | 16:32 |
| clarkb | fungi: and did we want to land those bindep changes? | 16:32 |
| opendevreview | Clark Boylan proposed opendev/system-config master: Test zuul-lb on noble https://review.opendev.org/c/opendev/system-config/+/941134 | 16:43 |
| fungi | clarkb: yeah, i guess if you're good with them and nobody else is interested in reviewing, we could go ahead | 16:57 |
| opendevreview | Clark Boylan proposed opendev/system-config master: Test codesearch on Noble https://review.opendev.org/c/opendev/system-config/+/941130 | 17:05 |
| clarkb | I'm staged up on bridge to deploy zuul-lb02 and codsearch02 on noble nodes as soon as the ci results come back clean | 17:07 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/project-config master: Stop using storyboard by Freezer https://review.opendev.org/c/openstack/project-config/+/941138 | 17:40 |
| opendevreview | Clark Boylan proposed opendev/zone-opendev.org master: Add codesearch02 to DNS https://review.opendev.org/c/opendev/zone-opendev.org/+/941140 | 17:47 |
| opendevreview | Clark Boylan proposed opendev/system-config master: Deploy codesearch02 https://review.opendev.org/c/opendev/system-config/+/941130 | 17:51 |
| clarkb | I think ^ and its two parents are ready for review | 17:51 |
| opendevreview | Clark Boylan proposed opendev/system-config master: Test zuul-lb on noble https://review.opendev.org/c/opendev/system-config/+/941134 | 18:00 |
| clarkb | I don't have zuul-lb02 working in CI yet (I hope that last ps fixes it though). But I'm going to get the node deployed just to keep things rolling forward | 18:11 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/project-config master: Remove Freezer DR from infra https://review.opendev.org/c/openstack/project-config/+/938184 | 18:20 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/project-config master: Stop using storyboard by Freezer https://review.opendev.org/c/openstack/project-config/+/941138 | 18:20 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/project-config master: Stop using storyboard by Freezer https://review.opendev.org/c/openstack/project-config/+/941138 | 18:22 |
| opendevreview | Clark Boylan proposed opendev/zone-opendev.org master: Add zuul-lb02 to DNS https://review.opendev.org/c/opendev/zone-opendev.org/+/941146 | 18:22 |
| opendevreview | Clark Boylan proposed opendev/system-config master: Deploy zuul-lb02 https://review.opendev.org/c/opendev/system-config/+/941134 | 18:25 |
| clarkb | hitting docker hub rate limits and ubuntu ppa errors on the codesearch change. I don't think either are related to what I've edited though so can be rechecked once all jobs report. Or maybe wait a bit and see if docker hub rate limits are happier | 18:47 |
| clarkb | It seems like we do hit them far less often now indicating improvements with our migrations away from docker hub but still more to be done I guess | 18:47 |
| clarkb | I left a review on https://review.opendev.org/c/openstack/tacker/+/941115 with ci resource consumption math for the tacker project if anyone else is interested | 19:12 |
| clarkb | I tried to illustrate the scope of the problem and offer suggestions on how things might be improved | 19:12 |
| clarkb | looks like the formater consumed my *'s | 19:14 |
| clarkb | I believe that single change used 403 nodes if anyone is wodnering | 19:29 |
| clarkb | my math is in the comment | 19:29 |
| opendevreview | Clark Boylan proposed opendev/zone-opendev.org master: Add zuul-lb02 to DNS https://review.opendev.org/c/opendev/zone-opendev.org/+/941146 | 20:53 |
| opendevreview | Clark Boylan proposed opendev/zone-opendev.org master: Add codesearch02 to DNS https://review.opendev.org/c/opendev/zone-opendev.org/+/941140 | 20:53 |
| clarkb | I'm flipping the order on those so that I can do zuul-lb02 first rather than codesearch. The reason for that is CI jobs for codesearch are sad on docker rate limits | 20:53 |
| clarkb | if anyone has time to reivew 941146 and 941134 that would be helpful otherwise I may just proceed with them as they are somewhat cookie cutter | 20:53 |
| clarkb | reviews welcome on the codesearch changes too | 20:54 |
| clarkb | oh and a reminder to edit the meeting agenda or let me know what needs to be edited | 21:09 |
| corvus | clarkb: +3 | 21:15 |
| clarkb | thanks. zuul got sad beacuse the dns chagne doesn't share a pipeline with the system-config chagne so I approved the dns update then will recheck the system-config change | 21:16 |
| clarkb | I've got the codesaerch change rechecking too. I'm hopeful I might be able to swing back around to that before the end of the day | 21:17 |
| opendevreview | Merged opendev/zone-opendev.org master: Add zuul-lb02 to DNS https://review.opendev.org/c/opendev/zone-opendev.org/+/941146 | 21:19 |
| opendevreview | Merged opendev/system-config master: Deploy zuul-lb02 https://review.opendev.org/c/opendev/system-config/+/941134 | 22:41 |
| opendevreview | Clark Boylan proposed opendev/zone-opendev.org master: Switch zuul.o.o to zuul-lb02 https://review.opendev.org/c/opendev/zone-opendev.org/+/941167 | 22:43 |
| opendevreview | Clark Boylan proposed opendev/zone-opendev.org master: Cleanup zuul-lb01 and reset zuul ttl https://review.opendev.org/c/opendev/zone-opendev.org/+/941168 | 22:43 |
| clarkb | these two dns updates depend on the deployment of 941134 being happy. zuul-lb jobs are near the end of the list so I expect this to be a bit | 22:44 |
| clarkb | also the dns updates conflcit with codesearch (I'll rebase as necessary depending on how either one these advance) | 22:44 |
| clarkb | I'm self approving https://review.opendev.org/c/opendev/system-config/+/941129 to update codesearch logging from syslog to journald and can manually restart services on the existing server to pick that up once it lands | 22:45 |
| clarkb | this is a prep step for the change to deploy codesearch to the new noble host (the change for that has been failing on docker rate limits) | 22:45 |
| clarkb | I've done a first pass on the meeting agenda. Let me know if there are dditional edits that should be made | 22:55 |
| corvus | maybe we should reverse-sort our inventory hosts ;) | 22:56 |
| opendevreview | Merged opendev/system-config master: Switch codesearch to journald logging https://review.opendev.org/c/opendev/system-config/+/941129 | 23:26 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!