Friday, 2020-11-06

ianwsean-k-mooney / zbr : LMN if you want +2 on pypa/project-config.  we shouldn't need to fiddle it much at this point00:20
*** DSpider has quit IRC00:47
*** mlavalle has quit IRC01:07
*** iurygregory has quit IRC01:26
kevinzclarkb: kevinz: thanks! Then we'd like to start contribute the DIB first :-)02:24
*** ykarel|away has joined #opendev05:01
*** ykarel|away is now known as ykarel05:02
*** rpittau|afk is now known as rpittau06:47
*** ysandeep|away is now known as ysandeep|ruck06:53
*** ralonsoh has joined #opendev06:56
*** Tengu has joined #opendev07:09
*** Tengu has quit IRC07:12
*** sboyron has joined #opendev07:19
*** Tengu has joined #opendev07:22
*** fressi has joined #opendev07:39
*** Green_Bird has quit IRC07:43
*** sboyron has quit IRC07:43
*** Green_Bird has joined #opendev07:43
*** eolivare has joined #opendev08:05
*** ysandeep|ruck is now known as ysandeep|lunch08:07
*** iurygregory_ has joined #opendev08:09
*** sboyron has joined #opendev08:10
*** andrewbonney has joined #opendev08:11
*** iurygregory_ is now known as iurygregory08:14
*** fressi has quit IRC08:25
*** dtantsur|afk is now known as dtantsur08:42
*** fressi has joined #opendev09:07
*** ysandeep|lunch is now known as ysandeep|ruck09:45
*** sboyron has quit IRC09:54
*** sboyron has joined #opendev09:54
*** slaweq has joined #opendev09:56
*** DSpider has joined #opendev10:12
*** hashar has joined #opendev10:54
*** dtantsur is now known as dtantsur|bbl11:13
*** ysandeep|ruck is now known as ysandeep|afk11:30
*** janders has quit IRC12:19
*** janders has joined #opendev12:20
*** janders has quit IRC12:20
*** ysandeep|afk is now known as ysandeep12:25
*** janders has joined #opendev12:29
*** fressi has quit IRC12:31
*** janders has quit IRC12:32
*** janders has joined #opendev12:33
*** janders has quit IRC12:33
*** janders has joined #opendev12:36
*** ykarel has quit IRC12:40
*** ykarel has joined #opendev12:41
*** ysandeep is now known as ysandeep|ruck12:48
*** tosky has joined #opendev13:07
*** fressi has joined #opendev13:13
*** d34dh0r53 has quit IRC13:43
*** ysandeep|ruck is now known as ysandeep|brb13:53
openstackgerritzbr proposed opendev/elastic-recheck master: WIP: Run elastic-recheck container
*** d34dh0r53 has joined #opendev14:01
*** dtantsur|bbl is now known as dtantsur14:20
*** hashar has quit IRC14:23
openstackgerritThierry Carrez proposed openstack/project-config master: Create inspur/venus repository
*** ysandeep|brb is now known as ysandeep|ruck15:00
*** ysandeep|ruck is now known as ysandeep|away15:15
*** rpittau is now known as rpittau|afk15:18
*** sshnaidm_ has joined #opendev16:12
*** sshnaidm|afk has quit IRC16:13
openstackgerritzbr proposed opendev/elastic-recheck master: WIP: Run elastic-recheck container
openstackgerritzbr proposed opendev/elastic-recheck master: Make file writing atomic
clarkbinfra-root fungi and I have restored review-test to a november 5 state. If we're gonna try the surrogate gerrit idea we need to probably start that process now from this 2.13 state. Concerns are we need to configure another gerrit with enough config in vexxhost to do a proper migration without adding unwanted external connectivity. Then when done we need to figure out what to sync back in order to get a16:16
clarkbworking prod but not too much as the configs differ.16:16
clarkbI'm starting to lena towards doing the process in place out of simplicity and since we only need to do it once16:17
clarkbany thoughts on that?16:17
clarkbmnaser: ^ it was your initial idea too so you may have input16:17
fungiyeah, thinking through what's entailed, we need to decide what we should copy from review-test to the instance in vexxhost, and also work out a database configuration for a local mysqld there (review and review-test are currently still relying on trove instances in rackspace for their databases)16:18
fungiand then also be careful not to modify the tweaked configuration back to the original instance when we're copying back the upgraded data16:18
mnaserclarkb: honestly, people seem to be ok with the outage and given that there's many other fires, maybe it might be ok..16:19
openstackgerritzbr proposed opendev/elastic-recheck master: Make file writing atomic
fungithe up-front effort for not doing an in-place upgrade will mostly revolve around those factors (making sure there's enough gerrit and supporting files in place on the surrogate in vexxhost, making sure we copy the right content from the original server/database, and making sure we copy only the relevant upgraded content back to the original again)16:20
*** sshnaidm_ is now known as sshnaidm|afk16:21
*** mlavalle has joined #opendev16:23
*** eolivare has quit IRC16:23
openstackgerritMerged zuul/zuul-jobs master: More E208
openstackgerritzbr proposed opendev/elastic-recheck master: Make file writing atomic
*** fressi has quit IRC16:55
*** ykarel has quit IRC17:05
*** ykarel_ has joined #opendev17:05
*** dtantsur is now known as dtantsur|afk17:13
*** ykarel_ has quit IRC17:19
openstackgerritJeremy Stanley proposed openstack/project-config master: Run publish-openstack-artifacts on ubuntu-focal
fungisean-k-mooney: did you intend for your ci system to report on changes like that ^ one?17:52
sean-k-mooneywhich one17:53
fungithe one i just pushed there17:53
sean-k-mooneyam no it should not but i think that is becasue of a zuul quirck17:53
sean-k-mooneythis is my projects definition
sean-k-mooneyso it should not be running any jobs on that porject17:55
sean-k-mooneybut the syntax thing is not a job17:55
sean-k-mooneybut i think i have my tenent config wrong17:55
sean-k-mooneylet me check17:56
clarkbthe issue is that blovk17:56
clarkbif project-config didnt match any pipeline config it should be ignored17:56
clarkbwhat you can do is in the tenant config not load the project pipeline stuff then drop your default .* "nop" stuff17:57
sean-k-mooneyi tought i alredady didnt17:57
sean-k-mooneythe  include: []17:57
clarkbthen you shouldnt need that first block you linked17:57
clarkbzuul by default doesnt do anything17:57
clarkband if you arent loading the configs there wont be anything to do17:58
sean-k-mooneyoh ok but it shoudl not break anything right17:58
clarkbI dont think so17:58
sean-k-mooneyya so im not sure why its triggering17:58
clarkbbecauseif the .* match17:58
clarkbthat matches all projects so it will parse them17:58
sean-k-mooneyoh ok17:59
sean-k-mooneyill remove that and see if it helps17:59
sean-k-mooneyok pushed
sean-k-mooneylet me know if it still happens and ill try somethign else18:03
fungii'll keep an eye out for any more, thanks for the quick resolution sean-k-mooney!18:04
sean-k-mooneyno worries i was debating if i shoudl host this in the x namespace or soemthign just so ye could tweak it if there was an issue18:04
sean-k-mooneybut if you notice anything just let me know and ill try to fix it18:05
fungisounds great18:05
fungiwill do18:05
*** andrewbonney has quit IRC18:07
sean-k-mooneyby the way is there an ansible lint job in zuul-jobs? i think there is a yamlint one18:11
sean-k-mooneywhich i might use im going to move most of my custom jobs to a different repo but was wondering what the best way to lint the config repo is18:12
sean-k-mooneyit wont run the jobs right18:12
sean-k-mooneyso i guess pre-commit would be the best way to go for now18:12
*** ralonsoh has quit IRC18:13
fungisean-k-mooney: we've been running it from tox18:15
sean-k-mooneyon the config repo18:15
sean-k-mooneyor just in general18:15
sean-k-mooney i was going to run it via tox for the other repos18:16
fungion our config repo:
sean-k-mooneybut that needs to be run manually right18:16
sean-k-mooneyit cant be run via zuul18:16
sean-k-mooneyor am i missunderstanding that.18:17
sean-k-mooneynew jobs dont take effect but will merged jobs run18:17
fungiwe have a job which basically runs `tox -e linters`18:17
sean-k-mooneyoh sorry so ya then merged jobs do run im just being dumb18:17
clarkbfungi: +2 on the focal nodeset change. Not sure if others are around to review today18:18
sean-k-mooneyim mixing up the fact that proposed changes to zuul on config repos dont take effect until they are merged and job running at all18:18
fungiit's only speculative job configuration changes which aren't used to determine the behavior of the jobs for trusted config projects, you can still test the changed files themselves with no problem zuul somply isn't reading them as its configuration18:18
sean-k-mooneycool ill do that so18:19
sean-k-mooneyonce i figure out how to disable line lenght checks :)18:19
fungibut yeah, zuul checks out the proposed change for your config repo with no problem. it doesn't rely on the contents of that change to configure the builds which run on it, but that doesn't prevent you from linting the content or whatever18:20
sean-k-mooneyfungi: that is so you cant steal secrets and such18:20
sean-k-mooneyor allow things to merge via a depends on18:20
*** Tengu has quit IRC18:22
*** Tengu has joined #opendev18:22
fungiit's mostly to protect secrets, but more generally to avoid people proposing changes which work around safety measures enforced in config repos via depends-on to an unreviewed config repo change18:27
*** mlavalle has quit IRC18:41
*** mlavalle has joined #opendev18:43
fricklerfungi: +2 on that focal change, not sure you'd want to approve and watch it now, otherwise I can do that on Monday19:39
fungifrickler: i'll coordinate it with the release team19:56
clarkbdoes anyone know how you're supposed to map an image's digest on docker hub to the image you have locally?20:15
clarkbI want to restart nodepool launchers and I'm 99% sure the latest image is in place but the digest and image id locally on nl02 don't match what is latest on docker hub20:15
sean-k-mooneyclarkb: you could do a docker pull20:16
clarkbsean-k-mooney: right I did that, and now I'm trying to check :)20:16
sean-k-mooneybut i dont know how to actully verify it20:16
clarkbbut docker seems to publish a bunch of hashes and none of them make sense20:16
sean-k-mooneymaybed do a sha256sum on the export of the image20:18
sean-k-mooneyyou can pull by digest e.g. docker pull ubuntu@sha256:45b23dee08af5e43a7fea6c4cf9c25ccf269ee113168c19722f87876677c5cb220:18
clarkbif I docker image inspect it then it shows the change I expect20:19
clarkbso I'm now pretty confident hte image is from my change. But I can't figure out how to map what I've got locally to what docker hub advertises20:19
sean-k-mooneythere is a RepoDigests and digest string in the inspect output20:22
clarkbI think this is a multiarch image and I wonder if the digest I'm seeing locally is for the image proper and not the amd64 specific digest?20:22
clarkbsean-k-mooney: yup and that is different than what docker hub shows hence my confusion20:22
sean-k-mooneyfor me its also different then "podman image save | sha256sum" creates20:22
clarkbzuul/nodepool-launcher@sha256:4b3472a14e913861a16bc8579d7d7b5d045db59150df54d11707c19d73f32248 is what I've got locally20:23
clarkbsha256:73ce0f1a5481c110bc7060e5b165790c4781d07db512e45b08d33d742c8c1f1c is what docker hub says for the amd64 image20:23
clarkbI'm guessing 4b34 is for the multiarch digest and not the arch specific digest20:23
clarkbit just seems like if we're hashing these things lining them up would be a nice feature20:23
clarkbanyway I'm resetarting nl02's launcher now20:24
clarkblimestone and airshup-citycloud providers should check the ssh key scanning for us20:24
sean-k-mooneythat is what i get when i pull and inspect it20:25
sean-k-mooneyso it has both20:25
clarkbinteresting that you see both20:25
sean-k-mooneythat was with podman20:26
sean-k-mooneymaybe it depens on the docker client version20:26
clarkbok I bet docker doesn't know how to properly expose the multiarch stuff20:26
sean-k-mooneyor podman in that case20:26
clarkbbut podman is20:26
sean-k-mooneyi have docker somewhere ill check with that20:27
sean-k-mooneyya i only see the first one20:29
clarkbnow just waiting for nodes to be allocated from those two providers20:29
clarkbso far it has been quiet :/ I guess that is what we get on a friday20:29
clarkbsean-k-mooney: thank you for checking, I think that confirms well enough my suspcions20:29
sean-k-mooneyso docker 19.03.11 doesent seam to show it podman 2.0.4 does20:30
sean-k-mooneyand i also just see the "zuul/nodepool-launcher@sha256:4b3472a14e913861a16bc8579d7d7b5d045db59150df54d11707c19d73f32248" one20:30
fungiremember: podman, skopeo, et cetera are dirty words if you want docker inc to give your open source project unlimited image downloads ;)20:30
sean-k-mooneyi still prefer docker20:31
sean-k-mooneyi have finally figured out how to run podman conaienr with systemd so they restart20:31
sean-k-mooneybut i have not been able to make it run rootless so its really not adding much over docker20:31
sean-k-mooneyi still need to lanuch them with sudo20:32
clarkbI'm 95% sure is a job running with the extra keys being scanned20:33
clarkbseems like ansible is working just fine so no issues iwth the extra keys20:33
sean-k-mooneyevery so often i give in to the peer pressure fo my fellow redhatters and try the new redhatty way fo doing things but i always end up going back to the things that jsut work20:33
clarkbI'll let it run a bit before we decide its all happy, but next step will be restarting nl01, 03 and 0420:33
clarkbthen let ade_lee know to retry the fips stuff20:34
sean-k-mooneyim guessing this is for fedora?20:34
sean-k-mooneywith the elipctic curve keys by defualt?20:34
sean-k-mooneyor is it somethign else20:34
clarkbsean-k-mooney: its for fips actually. Paramiko defaults to using ed25519 keys first which are no enabled by fips. What happened before is we scanned just the ed25119 key since that was the default with paramiko then later when the job enables fips and reboots the server presents an ecdsa key and ansible won't ssh anymore due to a hostkey mismatch20:35
sean-k-mooneyah ok20:35
clarkbsean-k-mooney: the change I implemented is to have nodepool do a proper ssh keyscan and find all the valid hostkeys that paramiko can handle (for most of our servers that is ed25519, rsa, and ecdsa) then when the host changes its fips policy ansible should still find a valid key in what it scanned20:35
clarkbnow just confirming this doesn't break zuul in an unexpected way. The existing key exchange protocol between nodepool and zuul assumed a list and seems to be working fine20:36
sean-k-mooneywell sicne im not doing fips stuff i guess i wont hit that then20:36
sean-k-mooneythat said i think im goint to redpleoy everthing in conteianr eventurally20:36
sean-k-mooneyjust to make updates simpler20:37
sean-k-mooneyso ill get that change when i do that20:37
sean-k-mooneythat said i installed form git so maybe its more work then doign an git pull and service restart hehe20:38
clarkbit was an interesting problem. I ended up reading some of the ssh rfcs. Keyscanning isn't actually built into the protocol as far as I can tell so you are forced to set a single keytype from the client side and check if remote will negotiate with that, repeat for each key type20:38
sean-k-mooneyi did not know that20:39
clarkbthe client and server typically send each other lists ordered by priority and you pick the first one they both agree on (I think client list is preferred)20:39
sean-k-mooneybut it would be a pain to debug that20:39
sean-k-mooneyya i have seen that when i have used -v with ssh20:39
sean-k-mooneywhen i was debuggeing x11 forwarding before20:39
*** fressi has joined #opendev20:50
clarkbfungi: have you seen anything that would indicate I shouldn't restart the rest of the launchers for the ssh key scanning now?21:15
clarkbseems like the things I've checked are fine21:15
clarkbok proceeding with the other launcher restarts since I haven't seen problems21:22
clarkb#status log Restarted nodepool launchers to pick up an ssh keyscanning fix. This should grab all valid ssh hostkeys for test nodes enabling testing with fips in jobs.21:23
openstackstatusclarkb: finished logging21:23
clarkbI've rechecked
fungiclarkb: yeah, have a belated "fine by me"21:25
*** melwitt is now known as jgwentworth21:26
*** whoami-rajat__ has quit IRC21:26
fungihad to step away for a few to deal with sudden mail-order food deliveries21:26
clarkbI'm not sure is properly enabling fips looking at its console log21:45
clarkbbut the reboot succeeded...21:45
fungiimprovement at least21:49
openstackgerritBrian Haley proposed zuul/zuul-jobs master: Decrease MTU to account for IPv6 header
openstackgerritAlbin Vass proposed zuul/zuul-jobs master: Add nimble roles and job
openstackgerritAlbin Vass proposed zuul/zuul-jobs master: Remove unecessary files attributes from child jobs
openstackgerritAlbin Vass proposed zuul/zuul-jobs master: Remove unecessary files attributes from child jobs
*** hamalq has joined #opendev23:05
openstackgerritAlbin Vass proposed zuul/zuul-jobs master: Add option to install kubernetes with kind
openstackgerritAlbin Vass proposed zuul/zuul-jobs master: DNM: Add unified synchronize-repos role that works with linux and windows
openstackgerritAlbin Vass proposed zuul/zuul-jobs master: Enable progressive mode with ansible-lint
openstackgerritMerged zuul/zuul-jobs master: Enable progressive mode with ansible-lint

Generated by 2.17.2 by Marius Gedminas - find it at!