| opendevreview | Ghanshyam proposed openstack/tempest master: Introduce @serial test execution decorator https://review.opendev.org/c/openstack/tempest/+/821732 | 02:45 |
|---|---|---|
| gmann | kopecmartin: ^^ looks good to me now, please check | 02:46 |
| *** yadnesh|away is now known as yadnesh | 04:43 | |
| *** jpena|off is now known as jpena | 08:21 | |
| *** yadnesh is now known as yadnesh|afk | 08:26 | |
| *** soniya29|rover is now known as soniya29|rover|lunch | 08:55 | |
| *** soniya29|rover|lunch is now known as soniya29|rover | 10:06 | |
| opendevreview | Sylvain Bauza proposed openstack/tempest master: DNM: Test renaming OOMkilled test to be called earlier https://review.opendev.org/c/openstack/tempest/+/870913 | 10:19 |
| opendevreview | Bas de Bruijne proposed openstack/tempest master: tempest cleanup - don't initialize admin id's https://review.opendev.org/c/openstack/tempest/+/870930 | 12:31 |
| *** gthiemon1e is now known as gthiemonge | 13:09 | |
| *** soniya29|rover is now known as soniya29|rover|brb | 13:20 | |
| opendevreview | Balazs Gibizer proposed openstack/tempest master: DNM: Test renaming OOM killed test to be called later https://review.opendev.org/c/openstack/tempest/+/870947 | 13:26 |
| frickler | ianw: do you plan to move devstack to f37? or should we rather deprecate fedora support due to lack of contributors keeping it up to date? | 13:28 |
| dansmith | gmann: so I got a fresh jammy, | 15:52 |
| dansmith | got farther with that, had a config problem, | 15:53 |
| dansmith | unstack, restack, and I'm back at the version install problem! | 15:53 |
| dansmith | sure seems like there's something up with the keystone deps | 15:53 |
| gmann | humm | 15:53 |
| gmann | dansmith: did you try clean.sh ? not 100% sure if that will help to cleanup deps | 15:54 |
| dansmith | yes always unstack and clean | 15:54 |
| opendevreview | Sylvain Bauza proposed openstack/tempest master: DNM: Test renaming OOMkilled test to be called earlier https://review.opendev.org/c/openstack/tempest/+/870913 | 15:55 |
| opendevreview | Balazs Gibizer proposed openstack/tempest master: Skip test_attach_scsi_disk_with_config_drive https://review.opendev.org/c/openstack/tempest/+/870974 | 16:45 |
| johnsom | gmann Question for you related to grenade and SRBAC enforce new defaults. Nova just enabled that (and scoped tokens which I don't understand why), which now breaks all of the legacy accounts because they don't have the "member" role. So, on upgrade all non-admin accounts are blocked access to nova. How is upgrade supposed to work with this? Does every installation have to modify every account's roles? | 16:47 |
| dansmith | gmann: okay, so the ceph OOM issue is because one test is downloading the entire image that we use into memory, so it can re-upload it | 16:50 |
| dansmith | gmann: on the ceph job, that's 1G, and of course anyone using tempest in a real cloud for verification is likely using more than a 16MB cirros image | 16:51 |
| gmann | johnsom: accessing API via non-'member' role and consider that token as owner of project is one of the bug we fixed in the new RBAC right | 16:51 |
| gmann | johnsom: in grenade or devstack if we are doing the same we should add 'member' (or 'reader') role in the access creds | 16:52 |
| johnsom | The nova rule: https://github.com/openstack/nova/blob/master/nova/policies/base.py#L94 | 16:52 |
| johnsom | The grenade failure (after upgrade): https://zuul.opendev.org/t/openstack/build/4a98acf0e7a24967a2fa607547c0ecf0/log/job-output.txt#22005 | 16:52 |
| johnsom | gmann Ok, so every deployment will have to modify every user role to include member/reader on or before upgrade to Antelope. Correct? | 16:53 |
| gmann | johnsom: that is expectation but I am thinking what else we can do other than mentioning it in release notes. something in upgrade-checks tool? | 16:54 |
| johnsom | Yeah, I think upgrade-checks is 100% a good idea | 16:54 |
| gmann | johnsom: because the bug was role 'foo' behave as owner of project and when we have 'reader' role it has to be fixed otherwise we cannot enforce 'reader' role capability | 16:54 |
| gmann | johnsom: sure, let me check on that. and will update you. | 16:55 |
| gmann | dansmith: ohk, which test. I thought test create/download the smaller image not the actual or 1 G image | 16:56 |
| dansmith | gmann: https://github.com/openstack/tempest/blob/6fa213cc0fcab744b46109e5f07cb58b0df4a314/tempest/api/compute/admin/test_volume.py#L52-L63 | 16:56 |
| johnsom | Yeah, I get it. Thanks for the info. I will propose fixes for tempest-plugin in Octavia to deal with the grenade issue. Easy enough | 16:57 |
| dansmith | gmann: so I was looking into making our image download client able to do the chunked download like the upload, | 16:57 |
| dansmith | but there's a loooot of plumbing to the actual http client.. I didn't realize this was all so custom | 16:57 |
| gmann | johnsom: thanks | 16:57 |
| dansmith | gmann: so we're going to skip that test for now (see gibi's patch above) | 16:57 |
| dansmith | gmann: I figured the right thing is to make the download chunk-able, but please confirm before I go try to do that :) | 16:58 |
| dansmith | or if there's some way I'm missing | 16:58 |
| gmann | dansmith: +1 I think that is better. | 16:59 |
| gmann | dansmith: ok for skipping it for now, patch you mean this one? https://review.opendev.org/c/openstack/tempest/+/870974 | 16:59 |
| dansmith | gmann: yeah | 17:00 |
| dansmith | gmann: okay so we have chunkable upload but just confirming, no chunkable download that you know of? I'm thinking this needs to be some sort of "just give me the raw response so I can chunk it" flag, if you think that's okay | 17:00 |
| gmann | dansmith: yeah, not aware of chunkable download. I think doing it based on flag seems better and we can do if test needs to do it | 17:02 |
| dansmith | gmann: okay, lemme see what I can work out | 17:03 |
| dansmith | gmann: btw, my latest stack failed due to the same keystone "no compute registered" thing locally | 17:03 |
| dansmith | so that's clearly a problem not just related to the gate | 17:03 |
| gmann | dansmith: thanks | 17:04 |
| gmann | dansmith: not sure why endpoint issue happening. and it is at the start itself right? not just compute endpoint missing | 17:07 |
| *** yadnesh|afk is now known as yadnesh|away | 17:13 | |
| dansmith | gmann: I haven't debugged it but compute is running and keystone shows no errors in the log | 17:26 |
| dansmith | gmann: the gate jobs show some amount of 404 talking to keystone I think | 17:27 |
| dansmith | I guess if I hit it again I'll try to dig in | 17:27 |
| gmann | ack | 17:28 |
| dansmith | and just like that, my stack is about to fail here | 17:28 |
| dansmith | guh | 17:28 |
| *** gmann is now known as gmann_afk | 17:29 | |
| dansmith | okay, my local reason is different than the one slaweq posted | 17:35 |
| dansmith | his is because the second compute worker can't access rabbit: | 17:36 |
| dansmith | Jan 13 15:16:45.704745 np0032723689 nova-compute[56832]: ERROR oslo.messaging._drivers.impl_rabbit [None req-655fe0a4-5254-4f7e-aa3b-4e8fe744637b None None] Connection failed: [Errno 111] ECONNREFUSED (retrying in 15.0 seconds): ConnectionRefusedError: [Errno 111] ECONNREFUSED | 17:36 |
| dansmith | however, mine is because placement can't start (which means compute can't start) and I can't believe it, but it's related to this weird keystone version problem: | 17:36 |
| dansmith | Jan 18 17:25:19 jammy devstack@placement-api.service[108112]: ERROR placement raise InvalidVersion(f"Invalid version: '{version}'") | 17:36 |
| dansmith | this is on a completely clean jammy | 17:36 |
| dansmith | it's where keystone goes to generate the user-agent, looks up package versions of distro-info, and chokes on an invalid version | 17:37 |
| dansmith | this package: | 17:37 |
| dansmith | ii python3-distro-info 1.1build1 all information about distributions' releases (Python 3 module) | 17:37 |
| *** gmann_afk is now known as gmann | 17:41 | |
| *** jpena is now known as jpena|off | 17:42 | |
| dansmith | gmann: full trace: https://paste.opendev.org/show/b8rr5zVkr3D2uSXZQDpo/ | 17:43 |
| dansmith | basically the same I was seeing while trying to install keystone in devstack before I cleaned my system | 17:44 |
| dansmith | logstash says we're not hitting this in the gate anywhere, but I don't know why | 17:44 |
| clarkb | we use a fairly minimal image in nodepool. It may not include python3-distro-info. The devstack jobs log a dpkg -l which should confirm | 17:45 |
| dansmith | it's in ubuntu-minimal | 17:45 |
| clarkb | ubuntu-minimal as defined by? | 17:45 |
| clarkb | I mean its easy to check /me looks | 17:46 |
| dansmith | that's a dpkg install group | 17:46 |
| dansmith | so I figured we were probably at least at that | 17:46 |
| clarkb | https://zuul.opendev.org/t/openstack/build/cc2fc794eef7491fbfc2b36df1684b31/log/controller/logs/dpkg-l.txt its not in there | 17:47 |
| dansmith | okay, lemme try to purge it | 17:47 |
| dansmith | I would imagine anyone running in production will have that installed so... | 17:47 |
| dansmith | with that gone, it now chokes on python3-debian | 17:48 |
| clarkb | how is it breaking? | 17:48 |
| clarkb | pip saying it can't uninstall the package or something? | 17:48 |
| dansmith | because pkg_resources now enforces python's definition of a valid version number | 17:48 |
| dansmith | clarkb: placement is crashing at startup when it loads the keystoneclient, | 17:49 |
| dansmith | but earlier I saw it failing to install keystone from pip/source yeah | 17:49 |
| dansmith | clarkb: see my pastebin above? | 17:49 |
| clarkb | oh neat, thats basically going to explode against all the distro python versions due to how distros version things | 17:49 |
| dansmith | okay, with all that gone I can get it started | 17:49 |
| dansmith | clarkb: right it explodes everything | 17:49 |
| clarkb | another reason to move devstack into a venv, but seems we can never get critical mass to push that over the hump | 17:50 |
| dansmith | this is the first time I'm seeing it explode in openstack land, but it's been exploding other stuff | 17:50 |
| dansmith | I understand why they want consistent version numbers for dep checking, for non-dep things, they hard explode things, which is really uncool | 17:51 |
| dansmith | so non-library application type things, it's a real effing pain | 17:51 |
| frickler | I see this deprecation warning on my local jammy https://paste.opendev.org/show/bvOoK1yKsbVCqGnGncar/ | 17:55 |
| frickler | so I wonder how you got a newer pkg_resources version | 17:55 |
| dansmith | frickler: yeah that's what you get before you upgrade past a certain version of some tools I think | 17:55 |
| dansmith | frickler: I dunno, clean jammy, fully upgrade'd | 17:55 |
| knikolla[m] | living on the edge | 17:55 |
| dansmith | I fully upgraded to try to fix this problem, fwiw | 17:56 |
| dansmith | so I think we must be pulling in some other stuff as a result | 17:56 |
| frickler | that's what I have here, too. from the warnings I see, that looks like an ubuntu packaging issue, though. might want to check with ubuntu python team | 17:56 |
| dansmith | perhaps I started with a more conventional jammy install than you? | 17:56 |
| frickler | I started from impish a year ago or so | 17:56 |
| dansmith | because if you have a very barebones one, you won't have that package (apparently) | 17:57 |
| frickler | and it's a full desktop install, so I have that package, hence the warning | 17:57 |
| dansmith | it's not a packaging problem, it's a packaging convention that python libraries aren't going to tolerate anymore, | 17:57 |
| dansmith | it's us loading the distro-info package, presumably so we can put "ubuntu/jammy" in the user-agent string, but pkg_resources won't tolerate ubuntu's own version number of the package | 17:58 |
| dansmith | frickler: ack, I didn't see the package name in there, but I see | 17:58 |
| *** gmann is now known as gmann_afk | 18:06 | |
| dansmith | now I'm getting this in devstack setup: | 18:43 |
| dansmith | https://paste.opendev.org/show/bRQHzgFYiohYMcXQRRrV/ | 18:43 |
| dansmith | unable to create tempest's venv | 18:43 |
| dansmith | I think maybe I'll just skip 2023 | 18:43 |
| dansmith | okay maybe tox4 issues | 18:47 |
| *** gmann_afk is now known as gmann | 18:56 | |
| ianw | frickler: yeah i would like to update to f37, hopefully "soon" | 20:58 |
| opendevreview | Dan Smith proposed openstack/tempest master: TURBOWIP: Chunked download https://review.opendev.org/c/openstack/tempest/+/871000 | 21:06 |
| opendevreview | Ghanshyam proposed openstack/tempest master: Skip test_attach_scsi_disk_with_config_drive https://review.opendev.org/c/openstack/tempest/+/870974 | 21:08 |
| *** dmellado_ is now known as dmellado | 22:31 | |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!