| *** bhagyashris_ is now known as bhagyashris | 08:26 | |
| rpittau | good morning! I'm testing some jobs on centos-9 nodesets and I think there is a bit of a problem with the protocol for the ssh key generated on the node, we currently use RSA by default which apparently doesn't work well with the new openssl | 09:16 |
|---|---|---|
| rpittau | I tried using zuul_ssh_key_algorithm to change that to edcsa but it seems ignoring the value I set and it kept rsa | 09:16 |
| rpittau | the error specifically is: "debug1: send_pubkey_test: no mutual signature algorithm" | 09:51 |
| rpittau | which indicates that the algorithm is not supported by the client | 09:51 |
| jrosser_ | rpittau: there are notes about quite near the top of here https://www.openssh.com/releasenotes.html | 10:22 |
| jrosser_ | in order to submit a patch from a centos-9 node i have added "PubkeyAcceptedKeyTypes +ssh-rsa" in my ssh config for that host | 10:23 |
| rpittau | jrosser_: yeah, exactly, but I was wondering if there is also a way to change the algorithm used to create the ssh key on the nodeset, I mean, it should be possible using zuul_ssh_key_algorithm but it doesn't seem to work | 10:34 |
| rpittau | accepting rsa is not recommended anyway | 10:37 |
| rpittau | I guess it's acceptable for CI, but not for production | 10:37 |
| jrosser_ | well it depends what yuo want to do | 10:41 |
| jrosser_ | in my case i want to use git-review to submit patches to gerrit from an centos-9 node, so it's necessary to force a suitable key to be negotiated | 10:41 |
| rpittau | jrosser_: AFAIK gerrit accepts ecdsa, so no reason to use deprecated rsa | 10:47 |
| rpittau | anyway, I'm more focused on the reason why zuul_ssh_key_algorithm doesn't work | 10:49 |
| opendevreview | Merged openstack/devstack master: Fix tempest upper-constraints https://review.opendev.org/c/openstack/devstack/+/823322 | 10:57 |
| frickler | slaweq: do you have contacts at redhat interested in fips that might want to avoid this revert by finding a proper solution instead? https://review.opendev.org/c/openstack/devstack/+/823303 | 10:57 |
| slaweq | frickler: let me check who in Red Hat is mostly working on that | 10:59 |
| slaweq | frickler: I think it's mostly ade_lee who proposed original patch which You try to revert | 11:00 |
| slaweq | You should conctact him as he is driving this effort in Red Hat | 11:01 |
| *** dansmith is now known as Guest10351 | 11:26 | |
| opendevreview | Maciej Szwed proposed openstack/devstack stable/xena: [Stable-only] Fix tempest upper-constraints https://review.opendev.org/c/openstack/devstack/+/823313 | 11:27 |
| frickler | slaweq: I would, but they don't seem to be around here | 11:31 |
| slaweq | frickler: I will try to ping ade_lee later today about it | 11:32 |
| slaweq | but I'm not sure if he is working this week | 11:32 |
| opendevreview | Andre Aranha proposed openstack/tempest master: WIP - Refactor ssh.Client to allow other clients https://review.opendev.org/c/openstack/tempest/+/820860 | 12:10 |
| *** bhagyashris_ is now known as bhagyashris | 13:19 | |
| opendevreview | Jorhson Deng proposed openstack/os-testr master: Updating python testing classifier as per Yoga testing runtime https://review.opendev.org/c/openstack/os-testr/+/822864 | 13:28 |
| opendevreview | Radosław Piliszek proposed openstack/devstack stable/xena: Fix tempest upper-constraints https://review.opendev.org/c/openstack/devstack/+/823313 | 13:30 |
| opendevreview | Radosław Piliszek proposed openstack/devstack stable/wallaby: Fix tempest upper-constraints https://review.opendev.org/c/openstack/devstack/+/823462 | 13:31 |
| opendevreview | Radosław Piliszek proposed openstack/devstack stable/victoria: Fix tempest upper-constraints https://review.opendev.org/c/openstack/devstack/+/823463 | 13:31 |
| *** whoami-rajat__ is now known as whoami-rajat | 13:59 | |
| mtreinish | gmann, kopecmartin: Just a heads up I saw you talking about --black-regex vs --exclude-regex yesterday. I started the stestr 4.0.0 release prep with the PR to remove old deprecated names: https://github.com/mtreinish/stestr/pull/317 | 14:30 |
| mtreinish | I'm planning to release that in the next month or 2 | 14:31 |
| ade_lee | frickler, hey there - slaweq told me you were looking for regarding https://review.opendev.org/c/openstack/devstack/+/823303 | 14:57 |
| ade_lee | frickler, maybe making the change dependent on not being on openeuler is the right approach | 15:03 |
| gmann | mtreinish: sure, tempest has also deprecated the old param and convert them to new one while passing to stestr. so we are good from tempest side. | 15:29 |
| opendevreview | Dr. Jens Harbott proposed openstack/devstack stable/xena: Fix mysqladmin failure for Fedora 34 and mariadb https://review.opendev.org/c/openstack/devstack/+/823468 | 15:34 |
| mtreinish | gmann: ok cool. If it was an issue I could delay the release. But glad it's all good | 15:39 |
| gmann | yeah | 15:40 |
| lajoskatona | gmann: Hi & Happy New Year! | 15:57 |
| lajoskatona | gmann: shall I ask about deprecating lib/neutron in devstack, I mean how we can do it | 15:57 |
| gmann | lajoskatona: thanks. you too. | 15:57 |
| lajoskatona | gmann: we discussed it yesterday during the team meeting: https://meetings.opendev.org/meetings/networking/2022/networking.2022-01-04-14.04.log.html#l-51 | 15:57 |
| lajoskatona | gmann: the issue is that we have lib/neutron and lib/neutron-legacy and the later one is "undeprecated" now | 15:58 |
| gmann | lajoskatona: yeah so neutron-legacy is one we are going with right? | 15:59 |
| lajoskatona | gmann: exactly | 15:59 |
| gmann | lajoskatona: there is no standard way to deprecate in devstack. what we can do is 1. echo warning with removal deadline of say 2 release in lib/neutron (in case anyone replying on that) 2. after 2 release means in AA we can remove it AND we can rename lib/neutron-legacy to lib/neutron if that does not require more changes | 16:02 |
| gmann | frickler: anything else we need to do on this ? | 16:02 |
| gmann | and in 1st step we can send on ML too | 16:03 |
| frickler | gmann: lajoskatona: well there's two things here. one is deprecating the scripts in lib/neutron and maybe renaming it to lib/neutron-deadend or similar | 16:04 |
| frickler | the other is whether we'd still want to move the service names away from quantum. like use neutron-server instead of q-svc etc. | 16:05 |
| frickler | ade_lee: yes, would be great if you could propose that as a followup patch | 16:06 |
| gmann | frickler: service name rename is good but I am worried it might need more work like changes in ENABLE_SREVICE configuration etc | 16:08 |
| lajoskatona | gmann: +1 | 16:08 |
| gmann | as first step, we can go with deprecating/removing the script /lib/neutron | 16:08 |
| ade_lee | frickler, how does one determine that you're on openeuler? | 16:09 |
| lajoskatona | frickler: as q-* things didn't raised alarm for any lawyers I think we can keep them as most folks used to them | 16:09 |
| gmann | it will be big breaking change for many jobs/3rd party CI | 16:10 |
| frickler | ade_lee: they added a check function, see e.g. https://review.opendev.org/c/openstack/devstack/+/760790/41/lib/apache#88 | 16:11 |
| frickler | lajoskatona: gmann: hmm, o.k., quantum forever it is, then ;) | 16:12 |
| lajoskatona | frickler: :-) | 16:14 |
| gmann | frickler: lajoskatona or we can do 1. change in logs 2. accept both saying q-* are deprecated in doc and later we see if we can remove the q-* from configuration ? | 16:15 |
| gmann | 1. change the log file name | 16:15 |
| gmann | but that can be done as separate from deprecating/removing script lib/neutron | 16:16 |
| lajoskatona | gmann: You mean from screen-q-svc.txt to screen-neutron-server.txt or similar? | 16:17 |
| gmann | lajoskatona: yeah | 16:17 |
| gmann | lajoskatona: I was thinking to keep accepting -* in enable_service etc and internally we check both q-* or neutron-* services in lib/neutron-legacy or plugins and everything like log file name service name in systemd etc we use neutron-* | 16:19 |
| gmann | * keep accepting q-* | 16:19 |
| gmann | basically accept both q-* and neutron-* and later try to kill q-* if we can | 16:20 |
| ade_lee | frickler, ok thanks - I'll submit a follow up patch that uses the check function | 16:20 |
| lajoskatona | gmann: that sounds good | 16:21 |
| ade_lee | gmann, so I'm still trying to understand your comment here -- https://review.opendev.org/c/openstack/tempest/+/807465/16/tempest/lib/services/compute/keypairs_client.py#88 | 16:23 |
| gmann | ade_lee: i mean lib/services/compute/keypairs_client keep accepting kwargs only and pass it to nova as it is. and its tempest tests who can generate the ecdsa key and pass in kwargs | 16:24 |
| gmann | keypairs_client does not need to do the things by itself instead just accept and pass the request params to nova | 16:25 |
| ade_lee | gmann, right - I agree in general - its just that nova doesn't know yet how to generate an ecdsa key | 16:25 |
| ade_lee | we need to add that functionality to nova | 16:26 |
| ade_lee | and I can put up a review to do that - but I don't know how long it will take to merge , if I need a spec etc. | 16:26 |
| ade_lee | temporarily, can we generate the key here in this case? | 16:26 |
| gmann | ade_lee: yeah, that is fine, we can do that while we will add tests in tempest. I am fine to merge the tempest patch 807465 as it is because it does not change the default behavior and provide way to test the change needed in nova side or so | 16:27 |
| ade_lee | cool | 16:28 |
| gmann | ade_lee: once you remove the key generation from keypairs_client and releasenote nit update I am +2 | 16:28 |
| gmann | and later once nova changes are done then we can add more tests testing with ecdsa key | 16:29 |
| ade_lee | gmann, wait -- I thought I understand what we agreed on - but now , not so sure .. | 16:30 |
| gmann | ade_lee: I mean I am ok to merge the 807465 once you fix my comment for keypairs_client . I will not wait for nova change to merge and everything work | 16:31 |
| ade_lee | gmann, the reason I wanted to keep the key generation in here is so that I wouldn't have to change tempest tests all over the place to generate a key on the client side. | 16:31 |
| ade_lee | once we add the support on the nova side, we can take out the key generation .. | 16:32 |
| ade_lee | the key generation stuff only affects ecdsa keys .. | 16:32 |
| gmann | ade_lee: i understand but things is keypairs_client is lib function and we cannot change it once nova generate the key by its own. we need to go with deprecation phase or so | 16:32 |
| gmann | ade_lee: I was thinking, add new test with ecdsa key and test the nova change. and later once nova generate it by own then all tests can be run with ecsda key | 16:33 |
| gmann | "once we add the support on the nova side, we can take out the key generation .." so this can be done with single new tests instead of adding key generation in lib keypairs_client | 16:35 |
| ade_lee | gmann, yeah - it just means that I'm going to have go through all the places where we use this lib function and if using ecdsa keys, replicate this code there. | 16:37 |
| ade_lee | gmann, that could be a bunch of places - because, of course, I'm using this to get fips tests to pass across many projects | 16:38 |
| gmann | ade_lee: do we need to do all the places? I think doing in one new tests can serve the purpose of testing the nova change? | 16:38 |
| gmann | new tests - only run if ecdsa key is configured in tempest - generate the ecdsa key and pass it to keypairs_client | 16:39 |
| gmann | that should be enough right? | 16:39 |
| gmann | and later once nova changes are merged then all tests can be run on ecdsa if configured | 16:39 |
| ade_lee | gmann, let me look to see where changes will be required .. again, I'm running tempest tests with fips across nova, neutron manila, cinder .. etc .. I don't know if their tempest plugins make calls to generate keys | 16:41 |
| ade_lee | most do not generate their own keys and rely on nova to do it | 16:41 |
| gmann | ade_lee: sure, I am saying we will run all with fips once nova changes are done until then just add few tests with explicit generation of key in one/few tests. Adding temporary things in user facing lib API (keypairs_client ) is not good idea. | 16:43 |
| *** Guest10351 is now known as dansmith | 16:46 | |
| *** akahat|ruck is now known as akahat|out | 17:35 | |
| opendevreview | Sofia Enriquez proposed openstack/devstack-plugin-ceph master: WIP Allow the usage of Clone v2 API https://review.opendev.org/c/openstack/devstack-plugin-ceph/+/782624 | 17:51 |
| opendevreview | Sofia Enriquez proposed openstack/tempest master: DNM test snapshot deletion with deps on ceph https://review.opendev.org/c/openstack/tempest/+/782392 | 17:57 |
| opendevreview | Merged openstack/devstack stable/xena: Fix tempest upper-constraints https://review.opendev.org/c/openstack/devstack/+/823313 | 17:59 |
| opendevreview | Merged openstack/devstack stable/victoria: Fix tempest upper-constraints https://review.opendev.org/c/openstack/devstack/+/823463 | 18:17 |
| sean-k-mooney | hi are ye currently tracking any buge with sudo -H -E python3.8 /opt/repos/devstack/files/get-pip.py | 19:29 |
| sean-k-mooney | failing with an assertion error | 19:29 |
| sean-k-mooney | assert '_distutils' in core.__file__, core.__file__ | 19:29 |
| sean-k-mooney | https://paste.opendev.org/show/811945/ | 19:30 |
| sean-k-mooney | thats on Ubuntu 20.04.3 LTS | 19:30 |
| sean-k-mooney | i guess its this https://github.com/pypa/setuptools/issues/2993 | 19:35 |
| sean-k-mooney | looks like its fixed in setuptools https://github.com/pypa/setuptools/commit/41fa663c9da93ca1af98ce2ae397c02e4b3062e8 | 19:38 |
| sean-k-mooney | so we need setuptools v60.3.0 | 19:38 |
| sean-k-mooney | we currently cap it to 60.2.0 | 19:39 |
| sean-k-mooney | https://github.com/openstack/requirements/blob/master/upper-constraints.txt#L633 | 19:39 |
| sean-k-mooney | so we need to fix that | 19:39 |
| sean-k-mooney | clarkb: frickler ^ have we started to see that break the gate jobs yet | 19:40 |
| sean-k-mooney | it should cause all the devstack jobs to fail on ubuntu | 19:40 |
| jamesbenson | FYI: I found that if I run refstack (main branch) on my wallaby openstack with tempest version 29.1.0 tests pass, but if I use 29.2.0, I'm unable to SSH into the VM's that use random passwords resulting in ~6 failed tests. | 19:58 |
| fungi | sean-k-mooney: is there a constraints update in review yet? | 20:11 |
| sean-k-mooney | i dont think so | 20:11 |
| sean-k-mooney | the bot might have one | 20:11 |
| sean-k-mooney | i can check | 20:11 |
| sean-k-mooney | i have one locally but as you know having ssh issues | 20:12 |
| fungi | yeah, i'm pulling it up | 20:12 |
| fungi | sean-k-mooney: https://review.opendev.org/823447 was the last bot proposed one and it merged roughly 14 hours ago | 20:13 |
| sean-k-mooney | 20:14 | |
| sean-k-mooney | https://review.opendev.org/c/openstack/requirements/+/823556 is the most recent on | 20:14 |
| sean-k-mooney | but it does not have it | 20:14 |
| sean-k-mooney | oh | 20:15 |
| sean-k-mooney | its not on pypi | 20:15 |
| sean-k-mooney | its tagged https://github.com/pypa/setuptools/releases/tag/v60.3.0 | 20:15 |
| fungi | different bot too, that's the one which proposes targeted constraints updates for specific project releases | 20:15 |
| fungi | 823447 is from the general daily mass update | 20:16 |
| sean-k-mooney | but not here https://pypi.org/project/setuptools/#history | 20:16 |
| fungi | 823556 was merely for ovsdbapp getting a new release | 20:16 |
| sean-k-mooney | ack | 20:16 |
| fungi | setuptools' release autobuilders may not be done yet | 20:16 |
| sean-k-mooney | the other way to fix this would be to downgrade to < 60 or blcklist the know broken ones | 20:17 |
| fungi | the good news is we may not see it break our jobs since we have a separate ensure-pip role, so it may not end up triggering the buggy codepath | 20:17 |
| sean-k-mooney | ya i think ill just comment out the get pip bit in devstack locally | 20:18 |
| sean-k-mooney | and wait for the package to be released | 20:18 |
| fungi | does the get pip bit in devstack break if pip is already installed on the system? | 20:18 |
| sean-k-mooney | actully it might we worth addign a devstack macro/varible for that | 20:18 |
| sean-k-mooney | yes | 20:18 |
| fungi | ahh, okay, then we may see job failures in our ci as well | 20:19 |
| sean-k-mooney | stack@cloud:/opt/repos/devstack$ pip --version | 20:19 |
| sean-k-mooney | pip 21.3.1 from /usr/local/lib/python3.8/dist-packages/pip (python 3.8) | 20:19 |
| sean-k-mooney | it looks like we have cod for fedoa and rhel 9 to not install pip with get pip | 20:22 |
| sean-k-mooney | do you think it would make sense to add a config flag and default it to false so we use disto pip by default ? | 20:22 |
| sean-k-mooney | if pip is installed already | 20:23 |
| fungi | not sure, that's more of a question for the devstack maintainers | 20:24 |
| sean-k-mooney | hum that seam to be working locally at least | 20:32 |
| sean-k-mooney | i can propose a patch and see what they say | 20:32 |
| sean-k-mooney | fungi: i wonder if we have export SETUPTOOLS_USE_DISTUTILS=stdlib | 20:49 |
| sean-k-mooney | in our jobs | 20:49 |
| sean-k-mooney | that is the other workaround | 20:49 |
| sean-k-mooney | we do in some code paths | 20:50 |
| sean-k-mooney | https://opendev.org/openstack/devstack/src/branch/master/inc/python#L179 | 20:50 |
| sean-k-mooney | for https://github.com/pypa/setuptools/issues/2232 | 20:51 |
| sean-k-mooney | but i guess we need to also do that now for git pip effectivly | 20:51 |
| sean-k-mooney | not just when using it | 20:52 |
| fungi | yeah, i see a number of examples at https://codesearch.opendev.org/?q=SETUPTOOLS_USE_DISTUTILS | 20:53 |
| sean-k-mooney | oh lovely | 20:59 |
| sean-k-mooney | https://github.com/openstack/devstack/commit/e06d954229fc4fca827105f5bb0809a19075d590 | 20:59 |
| sean-k-mooney | that is also broken | 20:59 |
| sean-k-mooney | its almost right but nova thows an error if you set cpu_models and you do not have cpu_mode=custom | 21:00 |
| sean-k-mooney | so that should also have an if around it | 21:00 |
| sean-k-mooney | actully that wont fully fix things i use the config overrid syntax not the macro sysntax but i can explcitly set it to cpu_models= i think too | 21:05 |
| opendevreview | Ade Lee proposed openstack/devstack master: Only set chap algorithms if not openeuler https://review.opendev.org/c/openstack/devstack/+/823580 | 21:28 |
| ade_lee | frickler, ^^ | 21:28 |
| *** artom__ is now known as artom | 23:15 | |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!