| fungi | i've powered off lists.openstack.org and am making an image of it now | 00:11 |
|---|---|---|
| fungi | also removed it from the disable list on bridge | 00:11 |
| fungi | when i wake up tomorrow, assuming image creation succeeds, i'll delete the server | 00:11 |
| *** yadnesh4 is now known as yadnesh | 08:57 | |
| *** hindret_ is now known as hindret | 08:57 | |
| *** open10k8s_ is now known as open10k8s | 08:57 | |
| *** tonyb_ is now known as tonyb | 08:57 | |
| *** ildikov_ is now known as ildikov | 08:57 | |
| *** eball_ is now known as eball | 08:57 | |
| *** ShadowJonathan_ is now known as ShadowJonathan | 08:57 | |
| *** erbarr_ is now known as erbarr | 08:57 | |
| *** ricolin_ is now known as ricolin | 08:57 | |
| *** TheJulia_ is now known as TheJulia | 08:57 | |
| *** snbuback_ is now known as snbuback | 08:57 | |
| *** mnasiadka_ is now known as mnasiadka | 08:57 | |
| *** simondodsley_ is now known as simondodsley | 08:57 | |
| *** clarkb is now known as Guest5393 | 08:59 | |
| *** diablo_rojo_phone is now known as Guest5405 | 08:59 | |
| opendevreview | Dr. Jens Harbott proposed opendev/irc-meetings master: Revert "kolla: Move meeting one hour backwards (DST)" https://review.opendev.org/c/opendev/irc-meetings/+/899821 | 09:32 |
| *** clarkb is now known as Guest5418 | 10:45 | |
| fungi | server image lists.openstack.org-final-20231101-before-deletion exists in rackspace's dfw region now, reporting 47.33 GB in size | 13:09 |
| fungi | o | 13:09 |
| fungi | i'll proceed with the server deletion | 13:10 |
| fungi | #status log Deleted the original lists.openstack.org server after creating an archival image, 11.5 years from the anniversary of its creation (you'll be missed old friend!) | 13:13 |
| opendevstatus | fungi: finished logging | 13:13 |
| opendevreview | Merged opendev/irc-meetings master: Revert "kolla: Move meeting one hour backwards (DST)" https://review.opendev.org/c/opendev/irc-meetings/+/899821 | 13:14 |
| *** dansmith_ is now known as dansmith | 13:40 | |
| *** Guest5418 is now known as clarkb | 15:03 | |
| clarkb | woo I'm me again | 15:06 |
| TheJulia | was there ever a question? | 15:08 |
| TheJulia | :) | 15:08 |
| clarkb | I got guestified | 15:11 |
| clarkb | looks like it was oftc specific though so not something on my end | 15:11 |
| clarkb | or if it was something specific to talking to oftc | 15:11 |
| *** sfinucan is now known as stephenfin | 16:15 | |
| clarkb | fungi: I'm skimming the etherpad config. Most things seem to have sane defaults taht we don't need to explicitly add. socketTransportProtocols is something that we should maybe sync up though. We can also copy over the new stuff and override defaults if we want | 16:17 |
| clarkb | all that to say I suspect we can just remove the logging config in the 1.9.4 change. Then do a followup with config edits fi we want | 16:17 |
| fungi | i'm about halfway through updating settings.json.docker from the version in 1.9.4, preserving our edits | 16:19 |
| clarkb | fungi: you've decided to just be explicit about the defaults then? Thats fine too. I just don't think it is necessary as part of the 1.9.4 upgrade if we want to split them | 16:20 |
| fungi | and then i'll do settings.json.js to upstream settings.json.template | 16:20 |
| fungi | er, settings.json.j2 to upstream settings.json.template | 16:20 |
| clarkb | fungi: oh wait for settings.json.docker we don't actually use that file do we have edits in it? | 16:20 |
| clarkb | I think you can just copy it over directly? | 16:21 |
| clarkb | its there to be in sync with the upstream docker image builds, but then we supply a config file that mounts over it when we run the container | 16:21 |
| fungi | yes, we have some divergence. if we don't need it for testing the built image or something, then i'll just forward to the upstream version | 16:21 |
| fungi | working on settings.json.j2 now | 16:22 |
| clarkb | I don't think that divergence is from our side | 16:23 |
| clarkb | upstream has changed the .docker file contents and created divergence themselves, but it was always a straight copy over to satisfy the image build system's need for that file | 16:23 |
| fungi | agreed, i think it's just long-past changes to the file upstream we've missed | 16:23 |
| clarkb | and then on the actual config side (not the one we stuff in the image for compat with upstream image building) socketTransportProtocols is the only setting where we've diverged and may want to resync. The others all seem like reasonable defaults to me (in part because they've been used for some time now) | 16:24 |
| fungi | we also override skinName to no-skin instead of upstream's colibris default | 16:29 |
| fungi | that was intentional to avoid ending up with a google docs like "page view" interface | 16:30 |
| tonyb | While looking at the mirror replacements I came across ianw's switch to kafs, that seems neat but I want to better understand AFS before I tackle that part. | 16:30 |
| fungi | i do think kafs is likely to be a good path forward for our clients, then we only have to worry about openafs for the afs servers themselves at least | 16:31 |
| clarkb | tonyb: ya I think that can wait. Its something that apparently gets better and better with time and at this point may just work for our read only nodes like the mirrors. But jammy for example includes a reasonable openafs package version so no reason to not use it? | 16:31 |
| clarkb | last time I tried kafs it did not work for me but that was a while back now | 16:31 |
| fungi | if memory serves, kafs was working great for read-only/unauthenticated afs access since a while, and more recently for authenticated/write access? | 16:31 |
| tonyb | Fair. FWIW, I run kafs locally and /afs/openstack.org is "just there". Mind you I'm a very light user | 16:32 |
| clarkb | neat. I tried it a while back on opensuse tumbleweed beacuse there were no openafs pacakges. It did not work so I want back to doing afs things on our servers rather than via my local machines. And that has worked so I haven't tried in quite some time | 16:32 |
| clarkb | I think it did list the path fwiw but I couldn't list subdirs or read files | 16:32 |
| fungi | "didn't work" for anonymous read access even? | 16:33 |
| fungi | what was the kernel version roughly, do you recall? | 16:33 |
| clarkb | correct operations would just hang and I think I would have to reboot to unload the module to make things happy even | 16:34 |
| clarkb | I don't. This was quite some time ago so I wouldn't bother to debug it further. Just try on modern kernels and take it from there | 16:34 |
| opendevreview | Julia Kreger proposed openstack/diskimage-builder master: Remove cloud-init when using simple-init https://review.opendev.org/c/openstack/diskimage-builder/+/899885 | 16:41 |
| opendevreview | Julia Kreger proposed openstack/diskimage-builder master: ConfigDrive: Disable cloud-init auto dhcp by default https://review.opendev.org/c/openstack/diskimage-builder/+/899886 | 16:41 |
| *** promethe- is now known as prometheanfire | 16:49 | |
| *** prometheanfire is now known as Guest5452 | 16:49 | |
| *** Guest5452 is now known as prometheanfire | 16:51 | |
| fungi | clarkb: i see we set maxAge to 21600000 instead of 21600, looks like the typical js 1k timer resolution multiplier, but upstream comments 21600 as "6 hours" | 17:13 |
| fungi | (i.e. they're not multiplying by 1k) | 17:13 |
| clarkb | fungi: if you look at keyRotationInterval they do multiple by 1k there for the ms to s conversion | 17:14 |
| clarkb | I suspect that ours is correct | 17:14 |
| clarkb | fungi: 05fcf4ba4a9a63c2007eef31405c2a3712776241 is the commit that changed it upstream | 17:15 |
| fungi | yeah, i just found https://github.com/ether/etherpad-lite/commit/c407ad3d | 17:15 |
| clarkb | that didn't come with a unit conversion internally | 17:15 |
| clarkb | I think their code is broken? | 17:15 |
| clarkb | er config not cod | 17:15 |
| fungi | "re-enabled maxAge setting. maxAge is now in seconds, not milliseconds." | 17:15 |
| clarkb | oh so they've got multiple changes affecting it fun | 17:16 |
| fungi | looks like that changed in 1.1.1 so we've been running at 6k hours maxage since then | 17:16 |
| clarkb | fungi: except the config file had the *1k conversion until 05fcf4ba4a9a63c2007eef31405c2a3712776241 | 17:17 |
| fungi | yeah | 17:17 |
| fungi | the things you find when you start comparing divergent configs | 17:17 |
| clarkb | this may explain why you need a hard refresh after some etherpad upgrades | 17:17 |
| clarkb | its setting cache control max-age values | 17:18 |
| clarkb | which is indeed a seconds value | 17:18 |
| clarkb | I like that Gerrit takes human readable units for things like this :) | 17:19 |
| clarkb | that must've persisted in our config since we first deployed etherpad. I wrote the initial config management for it in puppet back in 2012 ish. I think we had it for the san diego summit so would've been between may and october? | 17:22 |
| clarkb | fungi: yay for finding 11 year old bugs and deleting 11 year old servers :) | 17:22 |
| fungi | yup | 17:22 |
| fungi | clarkb: i think this is the diff of upstream's settings.json.template to our settings.json.j2 after refactoring all the comments and whitespace and removing the log4js stuff we added... does it look right to you? https://paste.opendev.org/show/b5upAReDaZrkSnvmd9TH/ | 17:25 |
| clarkb | fungi: yes I think so | 17:28 |
| opendevreview | Jeremy Stanley proposed opendev/system-config master: Upgrade Etherpad to 1.9.4 https://review.opendev.org/c/opendev/system-config/+/896454 | 17:29 |
| opendevreview | Jeremy Stanley proposed opendev/system-config master: DNM force etherpad failure to hold node https://review.opendev.org/c/opendev/system-config/+/840972 | 17:29 |
| clarkb | fungi: hem for userColor we converted that from null to false to fix the color issues we had | 17:30 |
| fungi | oh, yep i can amend it to keep that edit | 17:30 |
| clarkb | fungi: that one will probably need to be figured out to determine if the upstream template is wrong or if they converted back to using null instead of false | 17:30 |
| fungi | should we test it out first and see if it still needs fixing? | 17:31 |
| clarkb | ++ | 17:31 |
| fungi | i have an autohold set for 840972 | 17:31 |
| clarkb | oh wait its only the docker side that is false and now everything else is null | 17:31 |
| clarkb | I think maybe this is correct | 17:31 |
| clarkb | fungi: it won't work. I've foudn another issue. But want to do more complete comments before posting | 17:33 |
| clarkb | I think it will be easier via gerrit | 17:33 |
| fungi | wfm | 17:34 |
| fungi | thanks | 17:34 |
| clarkb | ok posted | 17:40 |
| fungi | relying on diff to compare files makes it too easy to miss when sections are moving in and out of /* */ wrapper lines | 17:57 |
| clarkb | fungi: I use vimdiff then za to expand everything | 18:05 |
| clarkb | then I get the helpful hints but full context | 18:05 |
| fungi | clarkb: do you think the skinVariants is going to cause a problem? if it's simply ignored by the skin we're using then leaving it in there seems fine | 18:16 |
| fungi | it does say those options are "for the colibris skin only" but doesn't say they need to be commented out with other skins | 18:17 |
| clarkb | ya it might ignore them. We can try it that way I guess | 18:18 |
| clarkb | it should be pretty aparent if etherpad looks different on the held node | 18:18 |
| fungi | here's what the revised diff looks like after your corrections: https://paste.opendev.org/show/bP5bdVGWosjcki1ncqZa/ | 18:19 |
| opendevreview | Jeremy Stanley proposed opendev/system-config master: Upgrade Etherpad to 1.9.4 https://review.opendev.org/c/opendev/system-config/+/896454 | 18:21 |
| opendevreview | Jeremy Stanley proposed opendev/system-config master: DNM force etherpad failure to hold node https://review.opendev.org/c/opendev/system-config/+/840972 | 18:21 |
| clarkb | that looks better | 18:22 |
| fungi | i've replaced the autohold with a fresh one | 18:24 |
| clarkb | I still think we should consider applying the config update before upgrading to 1.9.4 but this config does look like how I would expect it to for that or the upgrade | 18:37 |
| fungi | oh, i can split the two changes apart if you prefer | 18:40 |
| fungi | yeah, now i see i overlooked that comment from patchset 3 | 18:41 |
| clarkb | fungi: just thinking that if we had merged this config update as part of the upgrade we would have no idea if the upgrade or the config update were the source of our resulting database problems | 18:41 |
| clarkb | well the config update that was initially proposed | 18:41 |
| opendevreview | Jeremy Stanley proposed opendev/system-config master: Upgrade Etherpad to 1.9.4 https://review.opendev.org/c/opendev/system-config/+/896454 | 18:54 |
| opendevreview | Jeremy Stanley proposed opendev/system-config master: DNM force etherpad failure to hold node https://review.opendev.org/c/opendev/system-config/+/840972 | 18:54 |
| opendevreview | Jeremy Stanley proposed opendev/system-config master: Update Etherpad settings from upstream https://review.opendev.org/c/opendev/system-config/+/899894 | 18:54 |
| fungi | 104.239.240.40 is the held node with etherpad 1.9.4 deployed | 19:37 |
| fungi | i'm testing on https://etherpad.opendev.org/p/isitbroken with my /etc/hosts overridden to that | 19:39 |
| fungi | everything seems to work for me, tested from both ff and cr | 19:45 |
| fungi | even ☃︎ | 19:49 |
| fungi | infra-root: please test the held node ^ and then we can plan for merging the upgrade series | 19:50 |
| clarkb | will do | 19:51 |
| fungi | also tested out the timeslider, read-only link, fiddled with settings | 19:54 |
| clarkb | fungi: that all lgtm | 19:57 |
| clarkb | now to review the changes | 19:57 |
| clarkb | fungi: on both the chagnes above we seem to continue to get logging for the service where we expect logs. Additionally on the config update change you can see in those logs where it attempts to connect to mysql and fails because the db is starting up. There isn't an explicit indication that the db successfully connects later but we can infer that from the other happy logs I think | 20:24 |
| clarkb | all that in addition to the held node looking good results in +2s from me. Thank you for putting that together | 20:25 |
| clarkb | fungi: I've realized that the start time for mailman upgrades conflicts with the gerrit community meeting if it starts at 15:00 UTC. Less of a problem if it starts at 16:00 UTC> I wouldn't reschedule as those meetings have been quiet and I should be able to multitask | 20:45 |
| clarkb | I went ahead and approved tonyb's mirror testing changes since they only affect testing and probably don't need multiple reviewers | 20:56 |
| fungi | sure, i'll probably approve the mailman upgrade change around 15:00 but honestly if there are problems we likely won't know until later in the day anyway | 21:21 |
| opendevreview | Merged opendev/system-config master: [testinfra] Add port into curl's --resolve arg. https://review.opendev.org/c/opendev/system-config/+/899762 | 21:28 |
| opendevreview | Merged opendev/system-config master: Add a jammy test node for regional mirrors https://review.opendev.org/c/opendev/system-config/+/899710 | 22:21 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!