| ianw | sean-k-mooney / zbr : LMN if you want +2 on pypa/project-config. we shouldn't need to fiddle it much at this point | 00:20 |
|---|---|---|
| *** DSpider has quit IRC | 00:47 | |
| *** mlavalle has quit IRC | 01:07 | |
| *** iurygregory has quit IRC | 01:26 | |
| kevinz | clarkb: kevinz: thanks! Then we'd like to start contribute the DIB first :-) | 02:24 |
| *** ykarel|away has joined #opendev | 05:01 | |
| *** ykarel|away is now known as ykarel | 05:02 | |
| *** rpittau|afk is now known as rpittau | 06:47 | |
| *** ysandeep|away is now known as ysandeep|ruck | 06:53 | |
| *** ralonsoh has joined #opendev | 06:56 | |
| *** Tengu has joined #opendev | 07:09 | |
| *** Tengu has quit IRC | 07:12 | |
| *** sboyron has joined #opendev | 07:19 | |
| *** Tengu has joined #opendev | 07:22 | |
| *** fressi has joined #opendev | 07:39 | |
| *** Green_Bird has quit IRC | 07:43 | |
| *** sboyron has quit IRC | 07:43 | |
| *** Green_Bird has joined #opendev | 07:43 | |
| *** eolivare has joined #opendev | 08:05 | |
| *** ysandeep|ruck is now known as ysandeep|lunch | 08:07 | |
| *** iurygregory_ has joined #opendev | 08:09 | |
| *** sboyron has joined #opendev | 08:10 | |
| *** andrewbonney has joined #opendev | 08:11 | |
| *** iurygregory_ is now known as iurygregory | 08:14 | |
| *** fressi has quit IRC | 08:25 | |
| *** dtantsur|afk is now known as dtantsur | 08:42 | |
| *** fressi has joined #opendev | 09:07 | |
| *** ysandeep|lunch is now known as ysandeep|ruck | 09:45 | |
| *** sboyron has quit IRC | 09:54 | |
| *** sboyron has joined #opendev | 09:54 | |
| *** slaweq has joined #opendev | 09:56 | |
| *** DSpider has joined #opendev | 10:12 | |
| *** hashar has joined #opendev | 10:54 | |
| *** dtantsur is now known as dtantsur|bbl | 11:13 | |
| *** ysandeep|ruck is now known as ysandeep|afk | 11:30 | |
| *** janders has quit IRC | 12:19 | |
| *** janders has joined #opendev | 12:20 | |
| *** janders has quit IRC | 12:20 | |
| *** ysandeep|afk is now known as ysandeep | 12:25 | |
| *** janders has joined #opendev | 12:29 | |
| *** fressi has quit IRC | 12:31 | |
| *** janders has quit IRC | 12:32 | |
| *** janders has joined #opendev | 12:33 | |
| *** janders has quit IRC | 12:33 | |
| *** janders has joined #opendev | 12:36 | |
| *** ykarel has quit IRC | 12:40 | |
| *** ykarel has joined #opendev | 12:41 | |
| *** ysandeep is now known as ysandeep|ruck | 12:48 | |
| *** tosky has joined #opendev | 13:07 | |
| *** fressi has joined #opendev | 13:13 | |
| *** d34dh0r53 has quit IRC | 13:43 | |
| *** ysandeep|ruck is now known as ysandeep|brb | 13:53 | |
| openstackgerrit | zbr proposed opendev/elastic-recheck master: WIP: Run elastic-recheck container https://review.opendev.org/729623 | 13:58 |
| *** d34dh0r53 has joined #opendev | 14:01 | |
| *** dtantsur|bbl is now known as dtantsur | 14:20 | |
| *** hashar has quit IRC | 14:23 | |
| openstackgerrit | Thierry Carrez proposed openstack/project-config master: Create inspur/venus repository https://review.opendev.org/761751 | 14:50 |
| *** ysandeep|brb is now known as ysandeep|ruck | 15:00 | |
| *** ysandeep|ruck is now known as ysandeep|away | 15:15 | |
| *** rpittau is now known as rpittau|afk | 15:18 | |
| *** sshnaidm_ has joined #opendev | 16:12 | |
| *** sshnaidm|afk has quit IRC | 16:13 | |
| openstackgerrit | zbr proposed opendev/elastic-recheck master: WIP: Run elastic-recheck container https://review.opendev.org/729623 | 16:16 |
| openstackgerrit | zbr proposed opendev/elastic-recheck master: Make file writing atomic https://review.opendev.org/761764 | 16:16 |
| clarkb | infra-root fungi and I have restored review-test to a november 5 state. If we're gonna try the surrogate gerrit idea we need to probably start that process now from this 2.13 state. Concerns are we need to configure another gerrit with enough config in vexxhost to do a proper migration without adding unwanted external connectivity. Then when done we need to figure out what to sync back in order to get a | 16:16 |
| clarkb | working prod but not too much as the configs differ. | 16:16 |
| clarkb | I'm starting to lena towards doing the process in place out of simplicity and since we only need to do it once | 16:17 |
| clarkb | any thoughts on that? | 16:17 |
| clarkb | mnaser: ^ it was your initial idea too so you may have input | 16:17 |
| fungi | yeah, thinking through what's entailed, we need to decide what we should copy from review-test to the instance in vexxhost, and also work out a database configuration for a local mysqld there (review and review-test are currently still relying on trove instances in rackspace for their databases) | 16:18 |
| fungi | and then also be careful not to modify the tweaked configuration back to the original instance when we're copying back the upgraded data | 16:18 |
| mnaser | clarkb: honestly, people seem to be ok with the outage and given that there's many other fires, maybe it might be ok.. | 16:19 |
| openstackgerrit | zbr proposed opendev/elastic-recheck master: Make file writing atomic https://review.opendev.org/761764 | 16:20 |
| fungi | the up-front effort for not doing an in-place upgrade will mostly revolve around those factors (making sure there's enough gerrit and supporting files in place on the surrogate in vexxhost, making sure we copy the right content from the original server/database, and making sure we copy only the relevant upgraded content back to the original again) | 16:20 |
| *** sshnaidm_ is now known as sshnaidm|afk | 16:21 | |
| *** mlavalle has joined #opendev | 16:23 | |
| *** eolivare has quit IRC | 16:23 | |
| openstackgerrit | Merged zuul/zuul-jobs master: More E208 https://review.opendev.org/761293 | 16:25 |
| openstackgerrit | zbr proposed opendev/elastic-recheck master: Make file writing atomic https://review.opendev.org/761764 | 16:41 |
| *** fressi has quit IRC | 16:55 | |
| *** ykarel has quit IRC | 17:05 | |
| *** ykarel_ has joined #opendev | 17:05 | |
| *** dtantsur is now known as dtantsur|afk | 17:13 | |
| *** ykarel_ has quit IRC | 17:19 | |
| openstackgerrit | Jeremy Stanley proposed openstack/project-config master: Run publish-openstack-artifacts on ubuntu-focal https://review.opendev.org/761776 | 17:49 |
| fungi | sean-k-mooney: did you intend for your ci system to report on changes like that ^ one? | 17:52 |
| sean-k-mooney | which one | 17:53 |
| fungi | the one i just pushed there | 17:53 |
| fungi | https://review.opendev.org/761776 | 17:53 |
| sean-k-mooney | am no it should not but i think that is becasue of a zuul quirck | 17:53 |
| sean-k-mooney | this is my projects definition https://github.com/SeanMooney/ci-sean-mooney/blob/main/zuul.d/projects.yaml#L2-L14 | 17:54 |
| sean-k-mooney | so it should not be running any jobs on that porject | 17:55 |
| sean-k-mooney | but the syntax thing is not a job | 17:55 |
| sean-k-mooney | but i think i have my tenent config wrong | 17:55 |
| sean-k-mooney | let me check | 17:56 |
| clarkb | the issue is that blovk | 17:56 |
| clarkb | if project-config didnt match any pipeline config it should be ignored | 17:56 |
| clarkb | what you can do is in the tenant config not load the project pipeline stuff then drop your default .* "nop" stuff | 17:57 |
| sean-k-mooney | http://paste.openstack.org/show/799796/ | 17:57 |
| sean-k-mooney | i tought i alredady didnt | 17:57 |
| sean-k-mooney | the include: [] | 17:57 |
| clarkb | then you shouldnt need that first block you linked | 17:57 |
| clarkb | zuul by default doesnt do anything | 17:57 |
| clarkb | and if you arent loading the configs there wont be anything to do | 17:58 |
| sean-k-mooney | oh ok but it shoudl not break anything right | 17:58 |
| clarkb | I dont think so | 17:58 |
| sean-k-mooney | ya so im not sure why its triggering | 17:58 |
| clarkb | becauseif the .* match | 17:58 |
| clarkb | that matches all projects so it will parse them | 17:58 |
| sean-k-mooney | oh ok | 17:59 |
| sean-k-mooney | ill remove that and see if it helps | 17:59 |
| sean-k-mooney | ok pushed https://github.com/SeanMooney/ci-sean-mooney/blob/main/zuul.d/projects.yaml | 18:03 |
| sean-k-mooney | let me know if it still happens and ill try somethign else | 18:03 |
| fungi | i'll keep an eye out for any more, thanks for the quick resolution sean-k-mooney! | 18:04 |
| sean-k-mooney | no worries i was debating if i shoudl host this in the x namespace or soemthign just so ye could tweak it if there was an issue | 18:04 |
| sean-k-mooney | but if you notice anything just let me know and ill try to fix it | 18:05 |
| fungi | sounds great | 18:05 |
| fungi | will do | 18:05 |
| *** andrewbonney has quit IRC | 18:07 | |
| sean-k-mooney | by the way is there an ansible lint job in zuul-jobs? i think there is a yamlint one | 18:11 |
| sean-k-mooney | which i might use im going to move most of my custom jobs to a different repo but was wondering what the best way to lint the config repo is | 18:12 |
| sean-k-mooney | it wont run the jobs right | 18:12 |
| sean-k-mooney | so i guess pre-commit would be the best way to go for now | 18:12 |
| *** ralonsoh has quit IRC | 18:13 | |
| fungi | sean-k-mooney: we've been running it from tox | 18:15 |
| sean-k-mooney | on the config repo | 18:15 |
| sean-k-mooney | or just in general | 18:15 |
| sean-k-mooney | i was going to run it via tox for the other repos | 18:16 |
| fungi | on our config repo: https://opendev.org/openstack/project-config/src/branch/master/tox.ini#L10-L52 | 18:16 |
| sean-k-mooney | right | 18:16 |
| sean-k-mooney | but that needs to be run manually right | 18:16 |
| sean-k-mooney | it cant be run via zuul | 18:16 |
| sean-k-mooney | or am i missunderstanding that. | 18:17 |
| sean-k-mooney | new jobs dont take effect but will merged jobs run | 18:17 |
| fungi | we have a job which basically runs `tox -e linters` | 18:17 |
| sean-k-mooney | oh sorry so ya then merged jobs do run im just being dumb | 18:17 |
| clarkb | fungi: +2 on the focal nodeset change. Not sure if others are around to review today | 18:18 |
| sean-k-mooney | im mixing up the fact that proposed changes to zuul on config repos dont take effect until they are merged and job running at all | 18:18 |
| fungi | it's only speculative job configuration changes which aren't used to determine the behavior of the jobs for trusted config projects, you can still test the changed files themselves with no problem zuul somply isn't reading them as its configuration | 18:18 |
| sean-k-mooney | cool ill do that so | 18:19 |
| sean-k-mooney | once i figure out how to disable line lenght checks :) | 18:19 |
| fungi | but yeah, zuul checks out the proposed change for your config repo with no problem. it doesn't rely on the contents of that change to configure the builds which run on it, but that doesn't prevent you from linting the content or whatever | 18:20 |
| sean-k-mooney | https://opendev.org/openstack/project-config/src/branch/master/.ansible-lint#L10 | 18:20 |
| sean-k-mooney | fungi: that is so you cant steal secrets and such | 18:20 |
| sean-k-mooney | or allow things to merge via a depends on | 18:20 |
| *** Tengu has quit IRC | 18:22 | |
| *** Tengu has joined #opendev | 18:22 | |
| fungi | it's mostly to protect secrets, but more generally to avoid people proposing changes which work around safety measures enforced in config repos via depends-on to an unreviewed config repo change | 18:27 |
| *** mlavalle has quit IRC | 18:41 | |
| *** mlavalle has joined #opendev | 18:43 | |
| frickler | fungi: +2 on that focal change, not sure you'd want to approve and watch it now, otherwise I can do that on Monday | 19:39 |
| fungi | frickler: i'll coordinate it with the release team | 19:56 |
| clarkb | does anyone know how you're supposed to map an image's digest on docker hub to the image you have locally? | 20:15 |
| clarkb | I want to restart nodepool launchers and I'm 99% sure the latest image is in place but the digest and image id locally on nl02 don't match what is latest on docker hub | 20:15 |
| sean-k-mooney | clarkb: you could do a docker pull | 20:16 |
| clarkb | sean-k-mooney: right I did that, and now I'm trying to check :) | 20:16 |
| sean-k-mooney | but i dont know how to actully verify it | 20:16 |
| sean-k-mooney | ya | 20:16 |
| clarkb | but docker seems to publish a bunch of hashes and none of them make sense | 20:16 |
| sean-k-mooney | same | 20:16 |
| sean-k-mooney | maybed do a sha256sum on the export of the image | 20:18 |
| sean-k-mooney | you can pull by digest e.g. docker pull ubuntu@sha256:45b23dee08af5e43a7fea6c4cf9c25ccf269ee113168c19722f87876677c5cb2 | 20:18 |
| sean-k-mooney | https://docs.docker.com/engine/reference/commandline/pull/#pull-an-image-by-digest-immutable-identifier | 20:19 |
| clarkb | if I docker image inspect it then it shows the change I expect | 20:19 |
| clarkb | so I'm now pretty confident hte image is from my change. But I can't figure out how to map what I've got locally to what docker hub advertises | 20:19 |
| sean-k-mooney | there is a RepoDigests and digest string in the inspect output | 20:22 |
| clarkb | I think this is a multiarch image and I wonder if the digest I'm seeing locally is for the image proper and not the amd64 specific digest? | 20:22 |
| clarkb | sean-k-mooney: yup and that is different than what docker hub shows hence my confusion | 20:22 |
| sean-k-mooney | for me its also different then "podman image save docker.io/zuul/zuul-web | sha256sum" creates | 20:22 |
| clarkb | zuul/nodepool-launcher@sha256:4b3472a14e913861a16bc8579d7d7b5d045db59150df54d11707c19d73f32248 is what I've got locally | 20:23 |
| clarkb | sha256:73ce0f1a5481c110bc7060e5b165790c4781d07db512e45b08d33d742c8c1f1c is what docker hub says for the amd64 image | 20:23 |
| clarkb | I'm guessing 4b34 is for the multiarch digest and not the arch specific digest | 20:23 |
| clarkb | it just seems like if we're hashing these things lining them up would be a nice feature | 20:23 |
| clarkb | anyway I'm resetarting nl02's launcher now | 20:24 |
| clarkb | limestone and airshup-citycloud providers should check the ssh key scanning for us | 20:24 |
| sean-k-mooney | http://paste.openstack.org/show/799800/ | 20:25 |
| sean-k-mooney | that is what i get when i pull and inspect it | 20:25 |
| sean-k-mooney | so it has both | 20:25 |
| clarkb | interesting that you see both | 20:25 |
| sean-k-mooney | that was with podman | 20:26 |
| sean-k-mooney | maybe it depens on the docker client version | 20:26 |
| clarkb | ok I bet docker doesn't know how to properly expose the multiarch stuff | 20:26 |
| sean-k-mooney | or podman in that case | 20:26 |
| clarkb | but podman is | 20:26 |
| sean-k-mooney | i have docker somewhere ill check with that | 20:27 |
| sean-k-mooney | ya i only see the first one | 20:29 |
| clarkb | now just waiting for nodes to be allocated from those two providers | 20:29 |
| clarkb | so far it has been quiet :/ I guess that is what we get on a friday | 20:29 |
| clarkb | sean-k-mooney: thank you for checking, I think that confirms well enough my suspcions | 20:29 |
| sean-k-mooney | so docker 19.03.11 doesent seam to show it podman 2.0.4 does | 20:30 |
| sean-k-mooney | and i also just see the "zuul/nodepool-launcher@sha256:4b3472a14e913861a16bc8579d7d7b5d045db59150df54d11707c19d73f32248" one | 20:30 |
| fungi | remember: podman, skopeo, et cetera are dirty words if you want docker inc to give your open source project unlimited image downloads ;) | 20:30 |
| sean-k-mooney | i still prefer docker | 20:31 |
| sean-k-mooney | i have finally figured out how to run podman conaienr with systemd so they restart | 20:31 |
| sean-k-mooney | but i have not been able to make it run rootless so its really not adding much over docker | 20:31 |
| sean-k-mooney | i still need to lanuch them with sudo | 20:32 |
| clarkb | I'm 95% sure https://zuul.opendev.org/t/openstack/stream/e05b1b1cd97d4143b9596234c8e41899?logfile=console.log is a job running with the extra keys being scanned | 20:33 |
| clarkb | seems like ansible is working just fine so no issues iwth the extra keys | 20:33 |
| sean-k-mooney | every so often i give in to the peer pressure fo my fellow redhatters and try the new redhatty way fo doing things but i always end up going back to the things that jsut work | 20:33 |
| clarkb | I'll let it run a bit before we decide its all happy, but next step will be restarting nl01, 03 and 04 | 20:33 |
| clarkb | then let ade_lee know to retry the fips stuff | 20:34 |
| sean-k-mooney | im guessing this is for fedora? | 20:34 |
| sean-k-mooney | with the elipctic curve keys by defualt? | 20:34 |
| sean-k-mooney | or is it somethign else | 20:34 |
| clarkb | sean-k-mooney: its for fips actually. Paramiko defaults to using ed25519 keys first which are no enabled by fips. What happened before is we scanned just the ed25119 key since that was the default with paramiko then later when the job enables fips and reboots the server presents an ecdsa key and ansible won't ssh anymore due to a hostkey mismatch | 20:35 |
| sean-k-mooney | ah ok | 20:35 |
| clarkb | sean-k-mooney: the change I implemented is to have nodepool do a proper ssh keyscan and find all the valid hostkeys that paramiko can handle (for most of our servers that is ed25519, rsa, and ecdsa) then when the host changes its fips policy ansible should still find a valid key in what it scanned | 20:35 |
| clarkb | now just confirming this doesn't break zuul in an unexpected way. The existing key exchange protocol between nodepool and zuul assumed a list and seems to be working fine | 20:36 |
| sean-k-mooney | well sicne im not doing fips stuff i guess i wont hit that then | 20:36 |
| sean-k-mooney | that said i think im goint to redpleoy everthing in conteianr eventurally | 20:36 |
| sean-k-mooney | just to make updates simpler | 20:37 |
| sean-k-mooney | so ill get that change when i do that | 20:37 |
| sean-k-mooney | that said i installed form git so maybe its more work then doign an git pull and service restart hehe | 20:38 |
| clarkb | it was an interesting problem. I ended up reading some of the ssh rfcs. Keyscanning isn't actually built into the protocol as far as I can tell so you are forced to set a single keytype from the client side and check if remote will negotiate with that, repeat for each key type | 20:38 |
| sean-k-mooney | i did not know that | 20:39 |
| clarkb | the client and server typically send each other lists ordered by priority and you pick the first one they both agree on (I think client list is preferred) | 20:39 |
| sean-k-mooney | but it would be a pain to debug that | 20:39 |
| sean-k-mooney | ya i have seen that when i have used -v with ssh | 20:39 |
| sean-k-mooney | when i was debuggeing x11 forwarding before | 20:39 |
| *** fressi has joined #opendev | 20:50 | |
| clarkb | fungi: have you seen anything that would indicate I shouldn't restart the rest of the launchers for the ssh key scanning now? | 21:15 |
| clarkb | seems like the things I've checked are fine | 21:15 |
| clarkb | ok proceeding with the other launcher restarts since I haven't seen problems | 21:22 |
| clarkb | #status log Restarted nodepool launchers to pick up an ssh keyscanning fix. This should grab all valid ssh hostkeys for test nodes enabling testing with fips in jobs. | 21:23 |
| openstackstatus | clarkb: finished logging | 21:23 |
| clarkb | I've rechecked https://review.opendev.org/#/c/760665/4 | 21:25 |
| fungi | clarkb: yeah, have a belated "fine by me" | 21:25 |
| *** melwitt is now known as jgwentworth | 21:26 | |
| *** whoami-rajat__ has quit IRC | 21:26 | |
| fungi | had to step away for a few to deal with sudden mail-order food deliveries | 21:26 |
| clarkb | I'm not sure https://review.opendev.org/#/c/760665/4 is properly enabling fips looking at its console log | 21:45 |
| clarkb | but the reboot succeeded... | 21:45 |
| fungi | improvement at least | 21:49 |
| openstackgerrit | Brian Haley proposed zuul/zuul-jobs master: Decrease MTU to account for IPv6 header https://review.opendev.org/761800 | 22:35 |
| openstackgerrit | Albin Vass proposed zuul/zuul-jobs master: Add nimble roles and job https://review.opendev.org/747865 | 22:57 |
| openstackgerrit | Albin Vass proposed zuul/zuul-jobs master: Remove unecessary files attributes from child jobs https://review.opendev.org/761802 | 23:00 |
| openstackgerrit | Albin Vass proposed zuul/zuul-jobs master: Remove unecessary files attributes from child jobs https://review.opendev.org/761802 | 23:02 |
| *** hamalq has joined #opendev | 23:05 | |
| openstackgerrit | Albin Vass proposed zuul/zuul-jobs master: Add option to install kubernetes with kind https://review.opendev.org/740935 | 23:06 |
| openstackgerrit | Albin Vass proposed zuul/zuul-jobs master: DNM: Add unified synchronize-repos role that works with linux and windows https://review.opendev.org/740005 | 23:12 |
| openstackgerrit | Albin Vass proposed zuul/zuul-jobs master: Enable progressive mode with ansible-lint https://review.opendev.org/760691 | 23:14 |
| openstackgerrit | Merged zuul/zuul-jobs master: Enable progressive mode with ansible-lint https://review.opendev.org/760691 | 23:49 |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!