| clarkb | https://github.com/ether/etherpad-lite/issues/7211 | 00:00 |
|---|---|---|
| clarkb | I think they did move the tag | 00:00 |
| clarkb | when I first tested it the tag was "correct" for some value of correct in that it contained the fix for our css issue | 00:00 |
| clarkb | and since then the tag appears to have moved and now it does not include the fix for the css issue which is https://github.com/ether/etherpad-lite/pull/7204 | 00:01 |
| clarkb | I'm going to respond to 7211 with what I've found | 00:01 |
| clarkb | as mentioned I think we're ok. The 2.5.2 tag that exists now appears to be 2.5.1 with a single commit to fix the changelog script | 00:03 |
| clarkb | the page is useable under firefox just looks bad and hopefully they make a 2.5.3 that we can upgrade to. | 00:03 |
| clarkb | corvus: do you know why the docker image build stuff with buildx doesn't emit the actual buid logs to the job-output.txt file? they are in the json file so the console viewer in the zuul web ui renders them | 00:06 |
| corvus | can you share a build url you're looking at? | 00:07 |
| clarkb | corvus: https://zuul.opendev.org/t/openstack/build/3ec999c2acde470e959985f7340ae108/console#4/0/76/ubuntu-noble this is where it is rendered but if you switch to job-output.txt the actual docker build output doesnt' show up | 00:08 |
| corvus | "zuul_log_id":"in-loop-ignore" | 00:10 |
| corvus | short answer: loops are complicated and the live log streaming isn't sophisticated enough to handle that case | 00:11 |
| corvus | but that could conceivably change | 00:11 |
| clarkb | ack thanks. fwiw I'm able to get what I need from the console so it isn't urgent | 00:12 |
| clarkb | also I discovered I have a local cached version of that tag so I'm forever out of sync with upstream unless I intervene explicitly to fix that tag | 00:12 |
| corvus | to anyone reading who may be unfamiliar with the subtext: never change a git tag (on a public repo). | 00:15 |
| clarkb | https://github.com/ether/etherpad-lite/issues/7211 has been updated | 00:16 |
| clarkb | to summarize we're effectively running v2.5.1 which does not include our desired fix because the upstream updated tags | 00:19 |
| clarkb | this is ok other than the known issue with firefox | 00:19 |
| clarkb | I've also discovered we're setting an explicit tag for etherpad docker images which is still 2.2.6. We deploy :latest so we're basically ignoring that but when upstream fixes this I'll try to remember to update that explicit tag value to match | 00:20 |
| clarkb | we could build with the old commit that v2.5.2 pointed to or grab the image from the intermediate registry that I built and tested already and then promote it. But upstream needs to fix this anyway so we should just rely on that imo | 00:21 |
| clarkb | and to confirm I cleaned up /etc/hosts on etherpad02 | 00:28 |
| clarkb | so I think we're basically in steady state there now | 00:29 |
| clarkb | fungi: I've got a todo list item to drop review03 from the emergecny file but you did that already right? | 00:32 |
| opendevreview | Dr. Jens Harbott proposed opendev/irc-meetings master: Revert "kolla: Move meeting one hour backwards (DST)" https://review.opendev.org/c/opendev/irc-meetings/+/966172 | 13:13 |
| opendevreview | Merged opendev/irc-meetings master: Revert "kolla: Move meeting one hour backwards (DST)" https://review.opendev.org/c/opendev/irc-meetings/+/966172 | 13:37 |
| opendevreview | Michal Nasiadka proposed zuul/zuul-jobs master: Use mirror_info in configure-mirrors role https://review.opendev.org/c/zuul/zuul-jobs/+/966187 | 14:54 |
| fungi | clarkb: yes, review03 was dropped from the emergency file yesterday during the meeting when we discussed it, same time i removed the jitsi-meet servers | 14:55 |
| fungi | the main page at https://etherpad.opendev.org/ is readable for me on qutebrowser (also webkit-based), though looks like it's missing styling. text isn't trying to escape from any of the butttons | 14:59 |
| fungi | ah, sounds like they (accidentally) rolled back the tag, so not a useful test | 15:00 |
| fungi | unrelated, should we consider mirroring the etherpad container image into quay? or is the forced ipv4 workaround our go-to now? | 15:03 |
| Clark[m] | fungi: that image isn't pushed to quay yet because the server is still on jammy running docker not podman. Moving it to quay would break speculative testing | 15:22 |
| Clark[m] | Which to be fair upstream just negated most of the benefits for | 15:23 |
| fungi | oh right | 15:25 |
| mnasiadka | Clark[m], fungi: While doing Alma 10 I posted a glean patch (https://review.opendev.org/c/opendev/glean/+/962567) - it seems it’s not required - should I abandon it? | 15:41 |
| mnasiadka | Yeah, it matches el10 for NM on Alma, so I’ll just abandon it. | 15:43 |
| fungi | if things are working without it, then sure, no need to add further complexity | 15:43 |
| clarkb | ++ | 15:45 |
| opendevreview | Michal Nasiadka proposed zuul/zuul-jobs master: Use mirror_info in configure-mirrors role https://review.opendev.org/c/zuul/zuul-jobs/+/966187 | 15:45 |
| clarkb | infra-root I'm going to approve https://review.opendev.org/c/opendev/system-config/+/964728 now to force gitea access through the lb. That will take about an hour to gate so we have time to block it if there is reason to | 15:49 |
| clarkb | then I can look at increasing the memcached memroy limit next | 15:50 |
| clarkb | then we can look at the 1.25.1 upgrade | 15:50 |
| clarkb | I decided that doing our best to improve performance generally is good for the upgrade process too so want to do them in that order | 15:50 |
| opendevreview | Michal Nasiadka proposed zuul/zuul-jobs master: Use mirror_info in configure-mirrors role https://review.opendev.org/c/zuul/zuul-jobs/+/966187 | 15:54 |
| opendevreview | Michal Nasiadka proposed opendev/zuul-providers master: Add trixie-arm64 https://review.opendev.org/c/opendev/zuul-providers/+/966200 | 16:02 |
| fungi | gonna go grab a quick lunch, back soon | 16:20 |
| opendevreview | Michal Nasiadka proposed opendev/zuul-providers master: Add trixie-arm64 https://review.opendev.org/c/opendev/zuul-providers/+/966200 | 16:49 |
| opendevreview | Michal Nasiadka proposed opendev/zuul-providers master: Add trixie-arm64 https://review.opendev.org/c/opendev/zuul-providers/+/966200 | 16:53 |
| opendevreview | Merged opendev/system-config master: Force gitea http(s) connectivity through the load balancer https://review.opendev.org/c/opendev/system-config/+/964728 | 17:07 |
| clarkb | that is currently waiting for the hourly deployments to complete | 17:10 |
| clarkb | iptables has updated on gitea09 at least | 17:20 |
| clarkb | I think iptables has updated on all backends now | 17:20 |
| clarkb | I'm still able to reach https://opendev.org via my browser | 17:21 |
| clarkb | doesn't appear that we've broken anything. Not sure if anyone else wants to do a quick test, but once this deployment is done and I can check the overall result I'll plan to approve https://review.opendev.org/c/opendev/system-config/+/965420 if everything continues to look happy | 17:22 |
| clarkb | git clone works too | 17:22 |
| clarkb | hrm the deployment failed | 17:24 |
| clarkb | mirror01.dfw3.raxflex.opendev.org is unreachable | 17:25 |
| clarkb | which I knwe but forgot that had impacts here. I'll put it in the emergency file now | 17:25 |
| clarkb | but I don't think I need to recheck or reenqueue anything as the firewall updates are in place and things seem to be working | 17:25 |
| clarkb | I've approved https://review.opendev.org/c/opendev/system-config/+/965420 to increase the memcached cache size | 17:26 |
| fungi | quickly tested gitea and seems to still be working for me through haproxy but i get icmp network unreachable responses for gitea09 directly on 80, 443, 3000, 3080 and 3081 | 18:09 |
| fungi | so just what we wanted | 18:09 |
| fungi | i can also still ssh into it | 18:09 |
| clarkb | thanks for checking | 18:12 |
| opendevreview | Merged opendev/system-config master: Increase Gitea memacached limit to 4096MB https://review.opendev.org/c/opendev/system-config/+/965420 | 18:20 |
| clarkb | all giteas are running with the bigger cache now and the deployment job succeeded | 18:43 |
| clarkb | and the service is still accessible to me | 18:43 |
| fungi | me as well | 18:45 |
| clarkb | if you notice opendev.org slowness please let us know. It might be helpful for us to evaluate how impactful these two changes are in improving general service uptimes and responsiveness | 18:46 |
| clarkb | vexxhost has responded and confirms it was a memory consumption problem. They have indicated steps have been taken on the hypervisor to free up resources and better accomodate the instance | 19:13 |
| fungi | thanks! (to them and to you for reaching out) | 19:15 |
| mnasiadka | clarkb: I’d say bigger cache made the UI and git clone operations somewhat faster, but maybe it’s just a late hour EU wise, and there’s less traffic from my side of the planet | 19:37 |
| mnasiadka | I’ll have a look tomorrow in EU office hours to see if it persists | 19:37 |
| fungi | i'd refrain from judging cache performance until things run uninterrupted for a few days so we're looking at a more populated/full cache | 19:40 |
| clarkb | I think only the UI is affected by this cache fwiw. And yes probably tneed to wait a bit to measure | 20:23 |
| clarkb | I've just realized that when testing held gitea nodes we now need to hit the haproxy due to the firewall. My current 1.25.1 held node was held before the firewall update so isn't affected but talking to its haproxy works fine as a check that we can do so in the future: https://158.69.67.86/opendev/system-config | 23:03 |
| clarkb | but also ^ is an easy way to test gitea 1.25.1 | 23:03 |
| clarkb | etherpad upstream reports the release/tag issue was due to a broken script which they have now fixed and made a 2.5.3 release to address | 23:11 |
| clarkb | I'll work on getting another set of changes up to upgrade to that version now | 23:11 |
| corvus | good to hear that was accidental | 23:14 |
| opendevreview | Clark Boylan proposed opendev/system-config master: Upgrade to etherpad 2.5.3 https://review.opendev.org/c/opendev/system-config/+/966233 | 23:16 |
| clarkb | corvus: ya I think 2.5.3 is basically what 2.5.2 was expected to be + the fix to the release script. but I'll hold a node for testing to double check since we've alreadybeen surprised once this week :) | 23:17 |
| opendevreview | Clark Boylan proposed opendev/system-config master: DNM force etherpad failure to hold node https://review.opendev.org/c/opendev/system-config/+/840972 | 23:17 |
Generated by irclog2html.py 4.0.0 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!