Wednesday, 2022-11-23

fungiianw: do you have more details? i'm able to `sudo ~fungi/foo/bin/openstack --os-cloud=openstackci-rax --os-region-name=DFW server list` on bridge01 and `sudo ~fungi/foo/bin/pip freeze|grep openstackclient` reports python-openstackclient==6.0.000:03
ianwfungi: this is on my local machine, i made a venv, installed python-openstackclient and added a clouds.yaml and did "server list"00:04
fungiwhich version were you seeing an error with?00:04
fungisomething newer than 6.0.0?00:04
ianw$ ./venv/bin/openstack --version00:04
ianwopenstack 6.0.000:04
fungiodd, i wonder why that version works on bridge00:04
fungicould it be related to the openstacksdk version instead?00:05
fungior keystoneauth?00:05
ianw    raise exceptions.UnsupportedVersion(msg)00:08
ianwcinderclient.exceptions.UnsupportedVersion: Invalid client version '2.0'. Major part should be '3'00:08
ianwit's the volume bit maybe00:08
fungiyou need python-cinderclient<800:09
fungi8.0.0 dropped support for volume api v200:09
fungiand rackspace doesn't have v300:09
ianwok, i see what i've done, i've commented out the volume_api_version: 2 when testing also00:10
ianw*but* ... rax connections do still work without identity_api_version: 200:10
ianw... and fails with identity_api_version: 300:11
ianwso, it must know how to autonegotiate it00:12
ianwok, that change is probably just a red-herring.  i guess we can remove the explicit pin, if we feel confident openstacksdk knows what to do, but it's a minor point00:12
fungiyeah, we could probably remove the identity_api_version: 3 from the others too, given that00:13
ianwI can replicate the "BadRequestException: 400: Client Error for url:, Bad networks format"00:13
ianwit looks like what we send is00:13
ianw-d '{"server": {"config_drive": true, "max_count": 1, "min_count": 1, "networks": "auto", "imageRef": "c2e5b7be-32ea-4f74-bb88-1c9a4104f8ca", "flavorRef": "6", "name": "ianw-test"}}'00:13
ianwso, presumably "networks: auto" is the thread to pull on ...00:14
fungioh, so this is the api responding with an error because it doesn't understand networks:auto, not the client bombing because it doesn't understand the response?00:16
ianwRESP BODY: {"badRequest": {"message": "Bad networks format", "code": 400}}00:16
ianwi think so00:16
fungithat does look to be the case, yes00:16
ianwhrm, maybe i should first try the same thing on an older client and see if i can pick out the same request and see what it is00:17
fungiianw: in that case you want openstacksdk<0.9900:20
fungiwhich ends up being 0.61.000:20
fungithat's what got the error to go away for me00:20
ianwhrm, openstacksdk<0.99 doesn't work with python3.1100:20
fungioh, ouch00:21
clarkbthe reason is a bit :( too00:21
fungiyeah, i guess we're still using 3.10 on bridge0100:21
clarkbit uses some deprecated intepreter introspection method that was finally removed in 3.1100:21
ianw... ok, so using a python3.8 venv, and openstacksdk 0.9900:22
ianw -d '{"server": {"config_drive": true, "max_count": 1, "min_count": 1, "imageRef": "c2e5b7be-32ea-4f74-bb88-1c9a4104f8ca", "flavorRef": "6", "name": "ianw-test"}}'00:22
clarkb0.99 is what failed right?00:22
ianw(this didn't work, but also it didn't give the same error)00:22
ianwwhich "failed" are we talking about? :)00:23
ianwanyway, it looks like openstacksdk 0.99 *doesn't* send "networks:" in the server create request00:24
fungiopenstacksdk>=0.99.0 is what was sending network:auto and resulting in the network error response00:24
fungiianw: openstacksdk 0.99 or <0.99?00:24
ianwaah yes sorry correct!  <0.9900:24
ianwlet me install 0.99 and just 100% confirm00:24
fungiso 0.61.0 (being the last tag before 0.99.0)00:25
ianwyes correct.  so ...00:25
fungiso whatever is happening with that additional parameter seems to have come in with the major refactor from the feature branch00:25
ianw0.61.0 does *not* send "networks: auto" and 0.99.0 *does* send "networks: auto" and that makes RAX barf00:26
clarkbI'm guessing that and update was made to ste networks: auto if you don't specify any network info00:26
ianwyes, i guess the next step is code inspection, or if that doesn't help, i can bisect00:26
ianw should be it00:27
ianw looks like it was the modification00:29
ianw# If user has not passed networks - let Nova try the best.00:29
ianw            kwargs['networks'] = 'auto'00:29
clarkbseems like a situation where we want to let nova make its best guess but by passing it explicit info we've invalidated that00:30
ianw... to the nova source ...00:31
fungithat feature was probably added to nova very recently in 201500:32
clarkb"Starting with microversion 2.37, this field is required and the special string values auto and none can be specified for networks. auto tells the Compute service to use a network that is available to the project, if one exists."00:32
clarkbianw: ^ from the nova compute api docs00:32
ianwheh, yeah, newton release notes ...00:32
clarkbso ya this has essentially broken all nova pre microversion 2.3700:32
fungiso the sdk should only do that if if detects a sufficient microversion, but either doesn't check or checks incorrectly00:32
ianwwell i think we have enough for an openstacksdk patch at any rate00:33
clarkbfungi: its not checking it seems to blindly add the argument00:33
clarkbbasically if you don't set networks it sets networks: auto for you00:33
fungiin good news, odds are we can use latest sdk with rackspace if that gets fixed00:34
ianwyeah a quick check for 2.37 microversion gets past that00:37
clarkbianw: the utils file in there has a few functions that might be helpful for doing this properly00:39
clarkbbsaiclly if supports_microversion(2.37) or something00:39
ianwi'll take a poke at testing00:46
ianwi guess that would simplify things down to just needing the older cinderclient00:46
clarkbit would also allow us to update in nodepool potentially00:48
fungiyes, on both counts00:49
opendevreviewJeremy Stanley proposed opendev/system-config master: Improve launch-node deps and fix script bugs
opendevreviewJeremy Stanley proposed opendev/ master: Add lists01 to DNS
opendevreviewJeremy Stanley proposed opendev/system-config master: Add to the inventory
fungiianw: another detail i just spotted... launch-node outputs command lines to use the rackdns tool for setting reverse dns entries in rackspace, but it doesn't seem we copied that to bridge01 (i'll probably just resort to using the webui)02:38
ianwoh, hrm, yeah02:42
ianwi felt like for some reason that stopped working.  but maybe it did start working again02:42
Clark[m]That was something that got on the etherpad for bridge iirc02:45
opendevreviewIan Wienand proposed opendev/system-config master: rax: remove identity_api_version 2 pin
ianw does not look compelling :/02:49
ianwprobably it would be easier to repurpose the rax dns backup script02:49
*** pojadhav|afk is now known as pojadhav03:12
*** yadnesh|away is now known as yadnesh04:03
ianwthat can work, i'll clean it up.  i don't want to spend too much time on rax dns tools 04:04
Clark[m]Do the existing tools not work?04:22
Clark[m]Maybe we would need python2 on the new bridge?04:22
*** ysandeep is now known as ysandeep|ruck04:29
ianwthe existing tool i linked to above is abandoned for a long time -- i think we can get a token and do the post in a much more straight forward fashion05:25
*** ysandeep|ruck is now known as ysandeep|ruck|brb05:34
*** ysandeep|ruck|brb is now known as ysandeep|ruck05:53
*** jpena|off is now known as jpena08:29
*** yadnesh is now known as yadnesh|afk08:47
opendevreviewMerged openstack/project-config master: Add Allow-Post-Review flag to OpenStackSDK project
opendevreviewMerged openstack/project-config master: Add post-review pipeline
*** yadnesh|afk is now known as yadnesh09:22
*** ysandeep__ is now known as ysandeep|ruck10:39
*** rlandy|out is now known as rlandy|rover11:10
*** dviroel|afk is now known as dviroel11:15
*** soniya29 is now known as soniya29|afk11:23
*** ysandeep|ruck is now known as ysandeep|out11:57
opendevreviewMerged opendev/git-review master: Add message option
opendevreviewCedric Jeanneret proposed opendev/base-jobs master: Ensure NetworkManager will not override /etc/resolv.conf
*** dasm|off is now known as dasm13:59
opendevreviewCedric Jeanneret proposed openstack/project-config master: Ensure NetworkManager doesn't override /etc/resolv.conf
opendevreviewCedric Jeanneret proposed opendev/base-jobs master: Ensure NetworkManager will not override /etc/resolv.conf
opendevreviewCedric Jeanneret proposed opendev/base-jobs master: Ensure NetworkManager will not override /etc/resolv.conf
*** ysandeep__ is now known as ysandeep|out14:50
*** dviroel is now known as dviroel|lunch15:07
*** soniya29|afk is now known as soniya2915:48
*** dviroel_ is now known as dviroel16:15
*** marios is now known as marios|out16:31
*** yadnesh is now known as yadnesh|away16:41
*** jpena is now known as jpena|off17:30
clarkbfungi: is there a maxking PR for the update yet? I can go thumbs up it or whatever github wants to bring attention with. Also I can write it if that helps (I alredy have a fork setup in githu and all that)17:51
opendevreviewVishal Manchanda proposed zuul/zuul-jobs master: [DNM] Change browser name chromium-browser->chromium for debian bullseye
opendevreviewClark Boylan proposed zuul/zuul-jobs master: Add support for nodejs testing on Debian
fungiclarkb: there is not, no. been sidelined by other work unfortunately. if you're up for proposing it i won't object, you likely already have a personal fork to push the pr from18:33
clarkbya I've got all that set up already so its a few steps easier18:34
clarkbcheckout the job list on
clarkbI wonder if that is a record for us18:45
clarkb~91 jobs18:46
opendevreviewJeremy Stanley proposed opendev/system-config master: Improve launch-node deps and fix script bugs
opendevreviewJeremy Stanley proposed opendev/system-config master: Improve launch-node deps and fix script bugs
fungiclarkb: when you have time, the lists01 inventory and dns additions are pending approval under topic:mailman319:48
fungialready got +2 from ianw19:48
funginew major versions of flake8 and pyflakes today, minor update to pycodestyle (formerly pep8), and yet another patch update for setuptools19:55
fungiit's a python community holiday tradition19:55
clarkbfungi: and we don't expect email to be sent right?20:03
clarkbon the test nodes we caught all outbound email but won't do that on this node20:03
fungiclarkb: we haven't imported any subscribers yet, so there should be no digests to send20:04
clarkband it doesn't seem to alert list owners when we create the lists right? oh also before we land this, are there any list cleanups we need to do. I should've mentioned that when we landed the previous changes20:04
clarkbbasically is our list of lists in sync with mm2 to reduce pruning needed later20:04
clarkbthe summit program committee list for example got renamed?20:05
fungii think i also renamed it in the patch?20:05
clarkboh it looks like we'll only configure opendev lists to start. The others are commented out20:06
clarkbwhich means syncing isn't critical20:07
fungiyeah, but also i did update summit-programming-committee to summit-track-chairs in the commented one too20:07
fungiand in the test version20:07
fungiso that all matches still20:07
clarkblast thing I'm double checking is letsencrypt which I don't see config for in our host vars20:08
fungioh, i thought we added that when we did the test one20:08
clarkbmaybe? I don't see it in the role20:08
fungidoes it need to be done separately for production?20:08
clarkboh because that is done by groups?20:08
fungilists[0-9]* is in the letsencrypt list in inventory/service/groups.yaml20:09
clarkbya and we use the mailman3 group to specify the cert names20:10
clarkbI'm now trying to figure out if we need dns updates or if this will fail20:10
clarkbI think it willwork since we already set acme records for the current prod server and those should work here as well20:11
clarkbfungi: I do notice that is not in the borg backup group20:11
clarkbour test server, lists99, is20:11
clarkbI've approved the dns update and +2'd the inventory update. I'm not approving the inventory update as I need to step out now for lunch but feel free to if you can watch it20:12
fungiam i misreading that entry?20:13
clarkboh heh I was looking for it below where the other entry is20:13
clarkbso ya don't even need a followup for that one20:13
opendevreviewMerged opendev/ master: Add lists01 to DNS
*** dviroel is now known as dviroel|afk21:25
*** rlandy|rover is now known as rlandy|out22:25
opendevreviewIan Wienand proposed opendev/system-config master: rax: remove identity_api_version 2 pin
*** dasm is now known as dasm|off23:49

Generated by 2.17.3 by Marius Gedminas - find it at!