Wednesday, 2023-11-01

fungii've powered off and am making an image of it now00:11
fungialso removed it from the disable list on bridge00:11
fungiwhen i wake up tomorrow, assuming image creation succeeds, i'll delete the server00:11
*** yadnesh4 is now known as yadnesh08:57
*** hindret_ is now known as hindret08:57
*** open10k8s_ is now known as open10k8s08:57
*** tonyb_ is now known as tonyb08:57
*** ildikov_ is now known as ildikov08:57
*** eball_ is now known as eball08:57
*** ShadowJonathan_ is now known as ShadowJonathan08:57
*** erbarr_ is now known as erbarr08:57
*** ricolin_ is now known as ricolin08:57
*** TheJulia_ is now known as TheJulia08:57
*** snbuback_ is now known as snbuback08:57
*** mnasiadka_ is now known as mnasiadka08:57
*** simondodsley_ is now known as simondodsley08:57
*** clarkb is now known as Guest539308:59
*** diablo_rojo_phone is now known as Guest540508:59
opendevreviewDr. Jens Harbott proposed opendev/irc-meetings master: Revert "kolla: Move meeting one hour backwards (DST)"
*** clarkb is now known as Guest541810:45
fungiserver image exists in rackspace's dfw region now, reporting 47.33 GB in size13:09
fungii'll proceed with the server deletion13:10
fungi#status log Deleted the original server after creating an archival image, 11.5 years from the anniversary of its creation (you'll be missed old friend!)13:13
opendevstatusfungi: finished logging13:13
opendevreviewMerged opendev/irc-meetings master: Revert "kolla: Move meeting one hour backwards (DST)"
*** dansmith_ is now known as dansmith13:40
*** Guest5418 is now known as clarkb15:03
clarkbwoo I'm me again15:06
TheJuliawas there ever a question?15:08
clarkbI got guestified15:11
clarkblooks like it was oftc specific though so not something on my end15:11
clarkbor if it was something specific to talking to oftc15:11
*** sfinucan is now known as stephenfin16:15
clarkbfungi: I'm skimming the etherpad config. Most things seem to have sane defaults taht we don't need to explicitly add. socketTransportProtocols is something that we should maybe sync up though. We can also copy over the new stuff and override defaults if we want16:17
clarkball that to say I suspect we can just remove the logging config in the 1.9.4 change. Then do a followup with config edits fi we want16:17
fungii'm about halfway through updating settings.json.docker from the version in 1.9.4, preserving our edits16:19
clarkbfungi: you've decided to just be explicit about the defaults then? Thats fine too. I just don't think it is necessary as part of the 1.9.4 upgrade if we want to split them16:20
fungiand then i'll do settings.json.js to upstream settings.json.template16:20
fungier, settings.json.j2 to upstream settings.json.template16:20
clarkbfungi: oh wait for settings.json.docker we don't actually use that file do we have edits in it?16:20
clarkbI think you can just copy it over directly?16:21
clarkbits there to be in sync with the upstream docker image builds, but then we supply a config file that mounts over it when we run the container16:21
fungiyes, we have some divergence. if we don't need it for testing the built image or something, then i'll just forward to the upstream version16:21
fungiworking on settings.json.j2 now16:22
clarkbI don't think that divergence is from our side16:23
clarkbupstream has changed the .docker file contents and created divergence themselves, but it was always a straight copy over to satisfy the image build system's need for that file16:23
fungiagreed, i think it's just long-past changes to the file upstream we've missed16:23
clarkband then on the actual config side (not the one we stuff in the image for compat with upstream image building) socketTransportProtocols is the only setting where we've diverged and may want to resync. The others all seem like reasonable defaults to me (in part because they've been used for some time now)16:24
fungiwe also override skinName to no-skin instead of upstream's colibris default16:29
fungithat was intentional to avoid ending up with a google docs like "page view" interface16:30
tonybWhile looking at the mirror replacements I came across ianw's switch to kafs, that seems neat but I want to better understand AFS before I tackle that part.16:30
fungii do think kafs is likely to be a good path forward for our clients, then we only have to worry about openafs for the afs servers themselves at least16:31
clarkbtonyb: ya I think that can wait. Its something that apparently gets better and better with time and at this point may just work for our read only nodes like the mirrors. But jammy for example includes a reasonable openafs package version so no reason to not use it?16:31
clarkblast time I tried kafs it did not work for me but that was a while back now16:31
fungiif memory serves, kafs was working great for read-only/unauthenticated afs access since a while, and more recently for authenticated/write access?16:31
tonybFair.  FWIW, I run kafs locally and /afs/ is "just there".  Mind you I'm a very light user16:32
clarkbneat. I tried it a while back on opensuse tumbleweed beacuse there were no openafs pacakges. It did not work so I want back to doing afs things on our servers rather than via my local machines. And that has worked so I haven't tried in quite some time16:32
clarkbI think it did list the path fwiw but I couldn't list subdirs or read files16:32
fungi"didn't work" for anonymous read access even?16:33
fungiwhat was the kernel version roughly, do you recall?16:33
clarkbcorrect operations would just hang and I think I would have to reboot to unload the module to make things happy even16:34
clarkbI don't. This was quite some time ago so I wouldn't bother to debug it further. Just try on modern kernels and take it from there16:34
opendevreviewJulia Kreger proposed openstack/diskimage-builder master: Remove cloud-init when using simple-init
opendevreviewJulia Kreger proposed openstack/diskimage-builder master: ConfigDrive: Disable cloud-init auto dhcp by default
*** promethe- is now known as prometheanfire16:49
*** prometheanfire is now known as Guest545216:49
*** Guest5452 is now known as prometheanfire16:51
fungiclarkb: i see we set maxAge to 21600000 instead of 21600, looks like the typical js 1k timer resolution multiplier, but upstream comments 21600 as "6 hours"17:13
fungi(i.e. they're not multiplying by 1k)17:13
clarkbfungi: if you look at keyRotationInterval they do multiple by 1k there for the ms to s conversion17:14
clarkbI suspect that ours is correct17:14
clarkbfungi: 05fcf4ba4a9a63c2007eef31405c2a3712776241 is the commit that changed it upstream17:15
fungiyeah, i just found
clarkbthat didn't come with a unit conversion internally17:15
clarkbI think their code is broken?17:15
clarkber config not cod17:15
fungi"re-enabled maxAge setting. maxAge is now in seconds, not milliseconds."17:15
clarkboh so they've got multiple changes affecting it fun17:16
fungilooks like that changed in 1.1.1 so we've been running at 6k hours maxage since then17:16
clarkbfungi: except the config file had the *1k conversion until 05fcf4ba4a9a63c2007eef31405c2a371277624117:17
fungithe things you find when you start comparing divergent configs17:17
clarkbthis may explain why you need a hard refresh after some etherpad upgrades17:17
clarkbits setting cache control max-age values17:18
clarkbwhich is indeed a seconds value17:18
clarkbI like that Gerrit takes human readable units for things like this :)17:19
clarkbthat must've persisted in our config since we first deployed etherpad. I wrote the initial config management for it in puppet back in 2012 ish. I think we had it for the san diego summit so would've been between may and october?17:22
clarkbfungi: yay for finding 11 year old bugs and deleting 11 year old servers :)17:22
fungiclarkb: i think this is the diff of upstream's settings.json.template to our settings.json.j2 after refactoring all the comments and whitespace and removing the log4js stuff we added... does it look right to you?
clarkbfungi: yes I think so17:28
opendevreviewJeremy Stanley proposed opendev/system-config master: Upgrade Etherpad to 1.9.4
opendevreviewJeremy Stanley proposed opendev/system-config master: DNM force etherpad failure to hold node
clarkbfungi: hem for userColor we converted that from null to false to fix the color issues we had17:30
fungioh, yep i can amend it to keep that edit17:30
clarkbfungi: that one will probably need to be figured out to determine if the upstream template is wrong or if they converted back to using null instead of false17:30
fungishould we test it out first and see if it still needs fixing?17:31
fungii have an autohold set for 84097217:31
clarkboh wait its only the docker side that is false and now everything else is null17:31
clarkbI think maybe this is correct17:31
clarkbfungi: it won't work. I've foudn another issue. But want to do more complete comments before posting17:33
clarkbI think it will be easier via gerrit17:33
clarkbok posted17:40
fungirelying on diff to compare files makes it too easy to miss when sections are moving in and out of /* */ wrapper lines17:57
clarkbfungi: I use vimdiff then za to expand everything18:05
clarkbthen I get the helpful hints but full context18:05
fungiclarkb: do you think the skinVariants is going to cause a problem? if it's simply ignored by the skin we're using then leaving it in there seems fine18:16
fungiit does say those options are "for the colibris skin only" but doesn't say they need to be commented out with other skins18:17
clarkbya it might ignore them. We can try it that way I guess18:18
clarkbit should be pretty aparent if etherpad looks different on the held node18:18
fungihere's what the revised diff looks like after your corrections:
opendevreviewJeremy Stanley proposed opendev/system-config master: Upgrade Etherpad to 1.9.4
opendevreviewJeremy Stanley proposed opendev/system-config master: DNM force etherpad failure to hold node
clarkbthat looks better18:22
fungii've replaced the autohold with a fresh one18:24
clarkbI still think we should consider applying the config update before upgrading to 1.9.4 but this config does look like how I would expect it to for that or the upgrade18:37
fungioh, i can split the two changes apart if you prefer18:40
fungiyeah, now i see i overlooked that comment from patchset 318:41
clarkbfungi: just thinking that if we had merged this config update as part of the upgrade we would have no idea if the upgrade or the config update were the source of our resulting database problems18:41
clarkbwell the config update that was initially proposed18:41
opendevreviewJeremy Stanley proposed opendev/system-config master: Upgrade Etherpad to 1.9.4
opendevreviewJeremy Stanley proposed opendev/system-config master: DNM force etherpad failure to hold node
opendevreviewJeremy Stanley proposed opendev/system-config master: Update Etherpad settings from upstream
fungi104.239.240.40 is the held node with etherpad 1.9.4 deployed19:37
fungii'm testing on with my /etc/hosts overridden to that19:39
fungieverything seems to work for me, tested from both ff and cr19:45
fungieven ☃︎19:49
fungiinfra-root: please test the held node ^ and then we can plan for merging the upgrade series19:50
clarkbwill do19:51
fungialso tested out the timeslider, read-only link, fiddled with settings19:54
clarkbfungi: that all lgtm19:57
clarkbnow to review the changes19:57
clarkbfungi: on both the chagnes above we seem to continue to get logging for the service where we expect logs. Additionally on the config update change you can see in those logs where it attempts to connect to mysql and fails because the db is starting up. There isn't an explicit indication that the db successfully connects later but we can infer that from the other happy logs I think20:24
clarkball that in addition to the held node looking good results in +2s from me. Thank you for putting that together20:25
clarkbfungi: I've realized that the start time for mailman upgrades conflicts with the gerrit community meeting if it starts at 15:00 UTC. Less of a problem if it starts at 16:00 UTC> I wouldn't reschedule as those meetings have been quiet and I should be able to multitask20:45
clarkbI went ahead and approved tonyb's mirror testing changes since they only affect testing and probably don't need multiple reviewers20:56
fungisure, i'll probably approve the mailman upgrade change around 15:00 but honestly if there are problems we likely won't know until later in the day anyway21:21
opendevreviewMerged opendev/system-config master: [testinfra] Add port into curl's --resolve arg.
opendevreviewMerged opendev/system-config master: Add a jammy test node for regional mirrors

Generated by 2.17.3 by Marius Gedminas - find it at!