Tuesday, 2021-07-27

ianwyeah i was thinking abstract out the steps into roles, and then call the same roles from both places.  but it looks to fiddly to bother00:02
opendevreviewSteve Baker proposed openstack/diskimage-builder master: Depend on dib-python to set DIB_PYTHON_VERSION  https://review.opendev.org/c/openstack/diskimage-builder/+/80240800:03
ianwstill i feel like import_playbook might work ... or adding it as another playbook in the test00:03
Clark[m]ianw: the older code added it as another playbook in the test but these require specific vars to be passed in which is awkward for the current test setup. I'll think on making it less clunky00:17
ianwyeah i get that.  would just be good if the logs weren't jumbled with another nested ansible run.  could possibly just use ANSIBLE_LOG_PATH and collect that log file as something easier to examine00:20
opendevreviewTristan Cacqueray proposed opendev/system-config master: Run matrix-gerritbot on eavesdrop  https://review.opendev.org/c/opendev/system-config/+/80050602:20
*** marios is now known as marios|ruck05:04
*** amoralej|off is now known as amoralej06:46
*** marios|ruck is now known as marios07:01
*** marios is now known as marios|ruck07:03
*** ykarel|away is now known as ykarel07:28
*** rpittau|afk is now known as rpittau07:29
*** ykarel is now known as ykarel|lunch08:47
yoctozeptohmm, got something new from gerrit today09:55
yoctozeptoerror: remote unpack failed: error Missing tree d700505440e0092db1a7831758dc10b6ef46a56f09:55
yoctozeptofatal: Unpack error, check server log09:55
yoctozeptothe next try succeeded09:56
*** ykarel|lunch is now known as ykarel10:07
fungiyoctozepto: we've seen similar errors in the past we think may be related to thin pushes from cgit to jgit, and clarkb added a --no-thin command line option to git-review to serve as a potential workaround11:50
yoctozeptoack, thanks fungi11:54
opendevreviewAnanya proposed opendev/elastic-recheck master: Run elastic-recheck container  https://review.opendev.org/c/opendev/elastic-recheck/+/72962311:55
opendevreviewAnanya proposed opendev/elastic-recheck master: Run elastic-recheck container  https://review.opendev.org/c/opendev/elastic-recheck/+/72962312:03
fungiyoctozepto: though previously we'd seen errors like that be persistent for a particular commit history (otherwise requiring something like a rebase to solve)12:07
yoctozeptointeresting, I have only retried12:39
*** amoralej is now known as amoralej|lunch13:02
*** amoralej|lunch is now known as amoralej13:43
opendevreviewAnanya proposed opendev/elastic-recheck master: Run elastic-recheck container  https://review.opendev.org/c/opendev/elastic-recheck/+/72962313:44
fungiyoctozepto: was it a single change or a series of dependent changes?13:45
fungithere is a bug report about it at https://storyboard.openstack.org/#!/story/133254913:49
fungii'll update that with info about the --no-thin workaround in 2.1.013:49
yoctozeptofungi: single, just one commit on top of current master in the releases repo13:58
yoctozeptovery simple13:58
*** ykarel is now known as ykarel|away14:03
clarkbI think yoctozepto was one of the people that hit this a few months back which caused us to dig into it?14:11
clarkbwith openstack-ansible repo iirc14:11
yoctozeptoI would bet it was kolla-ansible instead14:11
clarkbAnyway, current suspicion is that a particular client interaction between C git and jgit causes it to happen. And using the --no-thin flag tells git to not use the optimized negotiation and use the more brute force version which seems to work reliably14:11
yoctozeptobut then there at least was a stack of changes and it did not give up14:11
clarkbI wouldn't use --no-thin for every push, just if you hit this14:11
yoctozeptowell, second try just worked (TM)14:12
fungiright, that's why it's not set up as a config option14:12
fungithin pushes should be much more efficient when they work, but in rare cases jgit and cgit don't seem to agree on what should be included14:13
yoctozeptoodd, perhaps the jgit workers get clumsy14:13
clarkbdpawlik: got back to me with the details I need to clean up that conflict. I'll do that one today rather than batching it up since I have user info confirming the correct fix14:16
dpawlikclarkb, fungi: thank you folks :)14:17
fungii've updated story 1332549 with details of the --no-thin option in git-review 2.1.0 but i left the task in a todo state because it's not clear to me whether we should be treating this as a git-review bug (it's most likely a bug in jgit), but also if we close it then it's less likely future users who run into the error will find that report14:51
clarkband the issue may go away on its own as people update git? I dunno I've never seen the issue locally myself and run a fairly up to date git14:52
fungii haven't either, but on the other hand i likely don't push changes to gerrit nearly as often as some users14:52
fungiso statistically speaking, my sample size may not be significant enough to expect to have encountered it14:53
clarkbyoctozepto: out of curiosity are you using bionic's git? another group that hit this was using bionic iirc15:03
yoctozeptoclarkb: no, focal's15:04
yoctozeptogit version 2.25.115:04
opendevreviewTristan Cacqueray proposed opendev/system-config master: Run matrix-gerritbot on eavesdrop  https://review.opendev.org/c/opendev/system-config/+/80050615:27
*** marios|ruck is now known as marios15:37
*** marios is now known as marios|ruck15:39
sshnaidmhi, how do I get "public key" of project ansible-collections-openstack ?15:51
opendevreviewAbhishek Kekane proposed openstack/project-config master: Remove 'glance' group from project entry  https://review.opendev.org/c/openstack/project-config/+/80254815:52
*** marios|ruck is now known as marios15:56
*** marios is now known as marios|ruck15:57
fungisshnaidm: see https://docs.opendev.org/opendev/infra-manual/latest/drivers.html#using-secrets for an example16:02
corvussshnaidm: also https://zuul-ci.org/docs/zuul-client/commands.html#encrypt is an option16:03
*** marios|ruck is now known as marios|out16:13
sshnaidmfungi, can you please take a look? is it the right pipeline? https://review.opendev.org/c/openstack/ansible-collections-openstack/+/80237916:21
fungicorvus: currently the opendev infrastructure manual's section on zuul secrets links to https://docs.opendev.org/opendev/infra-manual/latest/drivers.html#using-secrets which has been updated with a (broken) link to zuul-client, do you think we should link directly to the encrypt command doc for it instead? (also i'll push a fix momentarily for that link markup error in the zuul docs)16:26
fungican also update the example in infra-manual to use zuul-client16:27
corvusfungi: i don't see the broken link to zuul-client16:29
fungicorvus: https://zuul-ci.org/docs/zuul/discussion/encryption.html last paragraph just before the usage output16:30
fungifixup proposed in 80255416:31
corvusfungi: and yeah, i lean toward updating the manual to link to zuul-client; i don't think we're in a rush to remove that script, but it's probably the better long-term option.  i've been trying to use it myself.16:31
corvusfungi: got it (re link)16:31
fungicool, i'll freshen up the infra-manual section in that case. thanks!16:32
*** sshnaidm is now known as sshnaidm|afk16:34
*** rpittau is now known as rpittau|afk16:40
*** amoralej is now known as amoralej|off17:11
clarkbdpawlik: ok your extra account should be cleaned up now. If you want to double check the active used account is working as expected that would be great (though I anticipate it is fine)17:39
clarkbI've got a new consistency check running to output a new set of input conflicts to the audit script. Then I'll get the audit results and the next batch or proposed cleanups onto review02 for review17:41
dmsimardI haven't sent a patch to gerrit in a while and it's not accepting my ssh key anymore, is there a gotcha or new crypto requirement ? I tried removing my key and adding it back, didn't work either17:42
clarkbdmsimard: are you on fedora and using an rsa key?17:45
dmsimardyes and yes17:46
* fungi loves this one17:46
dmsimardwhat did I miss ? :D17:46
fungifedora decided to be proactive abd opaquely break ssh-rsa which is merely deprecated in openssh17:47
opendevreviewJay Faulkner proposed openstack/diskimage-builder master: Permit specification of extra bootstrap packages  https://review.opendev.org/c/openstack/diskimage-builder/+/80259217:47
fungiyour easiest solution is to switch to an ecdsa key, because the fedora maintainers blindly broke rsa2 support for ssh in situations where the server doesn't support dynamic negotiation options in the protocol17:48
clarkbyup and on top of that when breaking ssh-rsa in the way they broke it they were supposed to update client defaults to fallback to the rsa version that is allowed17:48
clarkbbut they didn't do that. They only half made the change and now you get this behavior17:48
dmsimardI see, searching for that does yield some results which I'll dig into -- I'm due for a key rotation soon anyway17:49
clarkbWhat Fedora should do is also update their openssh client to fallback to rsa-sha2 variants rather than the upstream default of sha117:49
clarkbthen their change would likely just work in 99% of cases. As is it is far more likely to break17:49
dmsimardI've found that adding "PubkeyAcceptedKeyTypes +ssh-rsa" to ~/.ssh/config for review.opendev.org fixes it for now so I'll run with that today but will renew my ssh key soon, thanks for the help <317:50
clarkbdmsimard: be careful doing that because I think people said that made a global change17:51
dmsimardoh ? despite being under a specific host ?17:51
clarkbdmsimard: basically you can ssh rsa to all the hosts now or something which is why we stopped recommending people do that and instead suggest using a non rsa key for fedora users17:51
clarkbdmsimard: ya it has to do with how ssh overrides those defaults17:51
clarkb(it is confusing and I may be misremembering)17:51
fungiit's possible that constraining it to a specific host entry only downgrades security for connections to that specific remote host17:52
clarkbanyway, I'm not a fedora user nor have I ever been. But if fedora users want to suggest to fedora that they supplement the disabling of rsa sha1 variants with a fallback in openssh client to rsa sha2 variants instead of sha1 that would be great17:53
clarkbThe ssh rfc says this should be done when the sha1 variants are disabled17:53
fungibut you're choosing a platform which has decided it's in your best interests to not use sha1 in the ssh protocol key exchanges, so we avoid recommending to users that they go against the security guidance of their chosen platfirms17:53
clarkbits just that no one but fedora has done that yet so the clients have gotten those updates17:53
clarkbinfra-root I've got everything on review02 except for the latest audit results as the audit is still running17:54
clarkband now the audit results are in place. infra-root you can now review my proposed cleanups with the latest audit results yaml file to cross check against (or do your own cross checking)17:57
clarkbinfra-root while I had extra perms I took the chance to look at melody. It reports we maxed out at 84GB of memory in gerrit18:00
clarkbbasically double what we could provide the server before18:00
fungithat's awesome18:01
fungiand awe-inspiring18:02
fungimnaser might like that tidbit too18:02
* melwitt loves faster gerrit18:22
clarkbyuriys: fyi we can talk about the cloud fixup here if you're comfortable with that.18:32
mordredthat's amazing - also - WOW that's a lot of RAM18:38
yuriysYep that's fine. The two instances still appear to be stuck, currently in a meeting, but after will continue to see how to clean those two up. Just seems that I'll have to manually purge from mariadb, which is my least favorite resolution method.18:45
clarkbok. The other thing I was going to try was a yum update and reboots for kernel patches18:45
clarkbWe've got our team meeting in about 15 minutes, then I need lunch then I'll look at that18:46
clarkbThen I'm reviewing a stack of zuul changes18:46
yuriysSounds like a plan.18:46
clarkbhopefully in the shade under a tree outside :)18:46
yuriysI'm curious whether or not it's also a good idea to pull the latest victoria container images.18:46
yuriysOn top of the DNF updates.18:46
fungii have no objection to that suggestion18:50
fungiit's non-impacting for us and we can always do a fresh redeploy there worst case18:51
yuriysSounds good.18:51
fungirebuilding the mirror vm is likely to be the most involved step if we redeploy18:52
fungibut most of that is automated as well18:52
yuriysI want to say I'd like to avoid that... that really shouldn't be normal for cloud 'updates' lol.18:52
fungisure, but this is openstack ;)18:52
fungiin openstack, our motto is "anything's possible" *wink*18:52
fungi(not officially our motto, but should be, right after "if it's not tested it's broken")18:53
clarkbya I'm good with experimenting here. If we can help you all learn stuff I'm game :)18:54
yuriysYeah for sure. I'm down to jump in a meets and screenshare all the funsies as usual. My typical workflow. Been a while since I've talked to you guys.18:55
yuriysBut if not, it's cool, I SEE HOW IT IS18:55
clarkbI'm open to that though that might be better for tomorrow (I'm wide open tomorrow if that works for you)18:55
clarkband we can do the kernel updates and reboots and all that fun then too18:55
yuriysoo yeah, same, wednesday way nicer for me.18:56
clarkbyuriys: I can be around as early as 8am pacific if you want to pick a time18:56
fungiyeah, i'm multi-tasking irc and backlogged yardwork, so teleconferencing isn't super compatible with my afternoon (not that my presence is strictly necessary either)18:56
fungimy tomorrow is also mostly open18:57
yuriysLets just figure it out tomorrow lol.18:57
clarkbworks for me :)18:57
mordredcorvus: I +2'd the matrix changes but didn't +A anything20:15
corvusmordred: thanks!20:45
opendevreviewJay Faulkner proposed openstack/diskimage-builder master: Permit specification of extra bootstrap packages  https://review.opendev.org/c/openstack/diskimage-builder/+/80259220:47
clarkbservice coordinator election email has been sent. You should see it momentarily20:53
opendevreviewIan Wienand proposed openstack/project-config master: Revert "nodepool: pause centos-8-stream builds"  https://review.opendev.org/c/openstack/project-config/+/80261920:56
ianwi'll check in on the builders a bit later and monitor ^ ... those images are getting very old now; but we should be gtg with new dib release deployed20:58
ianwalso noticed the arm64 job timed out, e.g. https://zuul.opendev.org/t/openstack/build/1e6031f1b07f43488f04acb2a736e653/logs21:01
ianwi'll look into that; it always suggests to me an issue where we're building too many wheels21:01
clarkbianw: on the nodepool side?21:02
clarkb/t/openstack shouldn't be nodepool though21:02
ianwno that was runtime system-config job21:02
clarkbthe review-test cleanup change lgtm and I approved the docker-compose restart flags change21:05
clarkbI'm going to go scope out some shade in a moment and dig into the zuul changes21:06
clarkbinfra-root does anyone know if we prefer to split https://review.opendev.org/c/openstack/project-config/+/790093 into to changes so that we can land them normally or do we force merge to get around the zuul error?22:53
clarkbthat change includes the zuul/main.yaml tenant update to include the new project as well as the layout change and zuul won't ever +1 that as is but we can force merge after we rename in gerrit to address that22:54
fungiin past renames since the advent of zuul v3 we've bypassed gating to merge the changes whole23:07
opendevreviewMerged openstack/project-config master: Revert "nodepool: pause centos-8-stream builds"  https://review.opendev.org/c/openstack/project-config/+/80261923:22
opendevreviewClark Boylan proposed opendev/system-config master: Test the rename_repos playbook  https://review.opendev.org/c/opendev/system-config/+/80211223:54
clarkbianw: fungi: that attempts to address the concern with nested ansible using import_playbook (I half expect that will break in unexpected ways but if it works it should solve this well) and it adds an explicit test for the CI-tools-updated group rename23:55
ianwi feel like that ansible should already be looking at playbooks etc from the zuul checkout, so i think it has a > 0 chance of working :)23:56
clarkbya I'm mostly concerend there is some gotcha with import_playbook that makes it unusable here as imports and includes always seem clunky23:56
clarkbbut if it does work then it should work well23:56

Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!