| opendevreview | Merged openstack/nova master: Drop lower-constraints.txt and its testing https://review.opendev.org/c/openstack/nova/+/840021 | 00:01 |
|---|---|---|
| opendevreview | Julia Kreger proposed openstack/nova-specs master: Discussion: Options to address ironic driver rebalance https://review.opendev.org/c/openstack/nova-specs/+/842015 | 01:14 |
| melwitt | elodilles: would like to have your opinion about one-off func test change vs backporting func test refactor https://review.opendev.org/c/openstack/nova/+/841483/2#message-295349b50b8233180cd5015ab8b977bab074765b at your convenience | 03:14 |
| *** gibi_pto is now known as gibi | 05:51 | |
| opendevreview | Rico Lin proposed openstack/nova-specs master: Add vIOMMU device support for libvirt driver https://review.opendev.org/c/openstack/nova-specs/+/840310 | 07:35 |
| elodilles | melwitt: i've commented on the patch. (in general one-off maybe better, but if this would be needed multiple times and considering this smaller functional test refactor, maybe it is acceptable to backport it) | 08:37 |
| *** akekane_ is now known as abhishekk | 09:36 | |
| gibi | sean-k-mooney: hi! I'm about to start fixing nits in the PCI tracking spec, but if you already in the middle of reviewing it then I can hold until your comments | 10:10 |
| sean-k-mooney | im in the middel of reviewing but sofar i have no comments to add :) | 10:18 |
| sean-k-mooney | so if you want to fix them go ahead | 10:18 |
| gibi | OK thans. I will fix the nits then soon | 10:22 |
| opendevreview | Balazs Gibizer proposed openstack/nova-specs master: PCI device tracking in Placement https://review.opendev.org/c/openstack/nova-specs/+/791047 | 10:38 |
| gibi | sean-k-mooney, stephenfin, melwitt: thanks for the reviews so far I fixed up the last batch of nits in the spec ^^ | 10:38 |
| erlon | sean-k-mooney: melwitt: hey guys, good morning, can I get some eyes on the pre-livemigration rollback patch? | 11:48 |
| erlon | https://review.opendev.org/c/openstack/nova/+/836014 | 11:48 |
| sean-k-mooney | yep ill take a look although i currently dont have stable rights but ill confrim it hte patch looks sane | 11:54 |
| sean-k-mooney | just finsihing a spec review but ill be doen shortly | 11:54 |
| sean-k-mooney | gibi: +1 some nits and one question inline but im more or less +2 | 12:25 |
| gibi | sean-k-mooney: thanks. I can trim down the required / forbidden traits tag to just having ``traits`` and implement required traits now similar in the first round. We can see if we need forbidden traits later ever | 12:30 |
| sean-k-mooney | im not pushed really either way i was just thinkign about it as one filed wiht forbiden triat modedl with a - or ! if supproted | 12:32 |
| sean-k-mooney | i dont think its much work to supprot forbidien traits so i dont really see any harm in including it | 12:33 |
| sean-k-mooney | it wont expand scope that much | 12:33 |
| sean-k-mooney | if other are ok with what you ahve propsoed im fine to upgrade to +2 | 12:33 |
| sean-k-mooney | but i want to wait of melwitt and stephenfin to comment | 12:33 |
| gibi | sure, lets see what the others think | 12:34 |
| gibi | and thanks again for the review | 12:34 |
| sean-k-mooney | no worreis. thanks for volenterring to take this on :) | 12:34 |
| gibi | :) | 12:34 |
| opendevreview | ribaudr proposed openstack/python-novaclient master: Microversion 2.91: Support specifying destination host to unshelve https://review.opendev.org/c/openstack/python-novaclient/+/831651 | 12:44 |
| sean-k-mooney | bauzas: i assume you are respining the keypair spec | 13:01 |
| bauzas | sean-k-mooney: yes, I was about to ping you and gmann | 13:07 |
| bauzas | about the x508 type | 13:08 |
| bauzas | I think we should continue to accept this parameter | 13:08 |
| sean-k-mooney | for listing | 13:09 |
| sean-k-mooney | not for generation right | 13:09 |
| sean-k-mooney | so we will keep type for show and import | 13:10 |
| sean-k-mooney | but remove generation for all types in the microversion | 13:10 |
| bauzas | sean-k-mooney: you can import a x508 public key https://github.com/openstack/nova/blob/972c06c608f0b00e9066d7f581fd81197065cf49/nova/api/openstack/compute/keypairs.py#L105 | 13:11 |
| sean-k-mooney | yes i know | 13:11 |
| sean-k-mooney | that was never in doubt at least in my mind | 13:11 |
| bauzas | if so, we create a fingerprint | 13:12 |
| sean-k-mooney | i was not sure if we could generate them but we can | 13:12 |
| bauzas | sean-k-mooney: you can generate them too | 13:12 |
| sean-k-mooney | yes i know i found the code | 13:12 |
| sean-k-mooney | so i dont want the capablities to differ per type | 13:12 |
| bauzas | yes | 13:12 |
| sean-k-mooney | so i dont want to kep generateion for x509 but drop it for ssh in teh new microverion | 13:13 |
| bauzas | so I'll remove this in the spec | 13:13 |
| sean-k-mooney | ack | 13:13 |
| sean-k-mooney | so we will remvoe generation for all types | 13:13 |
| sean-k-mooney | but keep type in the api for import and list | 13:13 |
| sean-k-mooney | correct? | 13:13 |
| sean-k-mooney | and keep the generation code for older microverions | 13:13 |
| bauzas | sean-k-mooney: yes, just stoping to accept no public key | 13:14 |
| bauzas | by the new microversion, yes | 13:15 |
| sean-k-mooney | that works for me | 13:15 |
| opendevreview | Sylvain Bauza proposed openstack/nova-specs master: Proposes to remove keypair generation https://review.opendev.org/c/openstack/nova-specs/+/840217 | 13:29 |
| *** haleyb_ is now known as haleyb | 13:38 | |
| kashyap | gibi: FFS, now CentOS9S seem to fail this - https://review.opendev.org/c/openstack/nova/+/838926 | 13:54 |
| kashyap | I just can't seem to get rid off my back | 13:54 |
| kashyap | It's another unrelated random timeout: https://zuul.opendev.org/t/openstack/build/bcaf2154eeb545cdb7907d0c807513e9 | 13:55 |
| *** dasm|off is now known as dasm | 13:56 | |
| gibi | kashyap: that was another disk detach issue | 14:23 |
| sean-k-mooney | should we make the c9s job non-voting | 14:31 |
| sean-k-mooney | i have seen that fail quite offen with the detach issue | 14:31 |
| kashyap | gibi: Oh, damn; is that a real one? | 14:34 |
| kashyap | sean-k-mooney: Yeah, we should until that known issue is resolved. I wonder if others disagree | 14:35 |
| sean-k-mooney | kashyap: i dont think its realated to your patch if that is what you are asking | 14:35 |
| gibi | I think we are still in the progress of landing the tempest patches that adds the waiter unit the guest boots up | 14:35 |
| sean-k-mooney | kashyap: but device detach is really really falky on centos 9 | 14:35 |
| kashyap | sean-k-mooney: No; I was asking if it's an unrelated new issue :) | 14:35 |
| gibi | it is not new | 14:35 |
| gibi | and it is not related to your patch | 14:35 |
| sean-k-mooney | kashyap: its unrelated to your patch | 14:35 |
| kashyap | (Yep, noted) | 14:36 |
| sean-k-mooney | gibi: i tought the tempest chagnes were landed on master | 14:38 |
| sean-k-mooney | https://review.opendev.org/q/topic:wait_until_sshable_pingable | 14:39 |
| sean-k-mooney | so https://review.opendev.org/c/openstack/tempest/+/817772? | 14:39 |
| sean-k-mooney | hum tempest.api.compute.volumes.test_attach_volume.AttachVolumeMultiAttachTest | 14:41 |
| sean-k-mooney | is what failed | 14:41 |
| sean-k-mooney | so maybe that is not useing that yet | 14:41 |
| sean-k-mooney | apprently it is https://github.com/openstack/tempest/blob/master/tempest/api/compute/volumes/test_attach_volume.py | 14:42 |
| sean-k-mooney | perhaps test_resize_server_with_multiattached_volume is not | 14:42 |
| gibi | I noted this particular failure in https://bugs.launchpad.net/nova/+bug/1960346/comments/29 | 14:43 |
| sean-k-mooney | ah | 14:43 |
| sean-k-mooney | https://github.com/openstack/tempest/blob/master/tempest/api/compute/volumes/test_attach_volume.py#L495= | 14:43 |
| sean-k-mooney | so we need to wait after that resize | 14:43 |
| gibi | after resize we only wait for ACTIVE state then go and detach | 14:43 |
| sean-k-mooney | or in teh resize call we need to wait for it to be sshable again | 14:43 |
| gibi | yepp | 14:44 |
| sean-k-mooney | ya | 14:44 |
| sean-k-mooney | ok i dont see a patch for that in the seriese at least not on the topic | 14:44 |
| sean-k-mooney | but that makes sense why this is still failing | 14:44 |
| gibi | probably it is easier to grep for detach calls and put a wait before them | 14:44 |
| gibi | gmann: fyi ^^ https://bugs.launchpad.net/nova/+bug/1960346/comments/29 | 14:45 |
| sean-k-mooney | perhaps | 14:45 |
| sean-k-mooney | so we could put a wait here https://github.com/openstack/tempest/blob/569c7a89f54c94494fde46ce2aa4fbd26492e640/tempest/api/compute/base.py#L459-L469= | 14:45 |
| sean-k-mooney | or | 14:46 |
| sean-k-mooney | we could put a wait here https://github.com/openstack/tempest/blob/569c7a89f54c94494fde46ce2aa4fbd26492e640/tempest/api/compute/base.py#L547= | 14:46 |
| sean-k-mooney | in detach | 14:46 |
| sean-k-mooney | if we do it in detach it shoudl alway ensure that we check its sshabel when we are about to detach but we woudl likely need a flag to hadnel teh vm state | 14:47 |
| sean-k-mooney | e.g. if its not active | 14:47 |
| gibi | yeah | 14:48 |
| sean-k-mooney | personally while i dont really liek the wackamole approch i would prefer not to put it in detach | 14:49 |
| sean-k-mooney | so put it in reszie ectra as those operations already have the instance and are changing the state | 14:49 |
| gibi | but not all VM lifecycle operation, not all resize, will be followed by a detach | 14:52 |
| gibi | so it is a waste to wait if no detach is done later | 14:53 |
| sean-k-mooney | yes but we could wait in all cases whre we expect the vm to be active | 14:53 |
| sean-k-mooney | ya so it a questoin of how much knolage we want the autor and review of tempest chagne to need | 14:53 |
| sean-k-mooney | we can be extra safe and alway wait when ever we expect a vm to be active | 14:54 |
| sean-k-mooney | or we can put it in only when we expect to do a detach | 14:54 |
| sean-k-mooney | in which case we shoudl not modify resize or detach and just add the wait in the respective tests | 14:54 |
| sean-k-mooney | i think the pattern they are taking is to pass sshable to wait_until | 14:56 |
| sean-k-mooney | and do that at the call site | 14:56 |
| sean-k-mooney | as was done here https://review.opendev.org/c/openstack/tempest/+/840112/2/tempest/api/compute/base.py | 14:57 |
| sean-k-mooney | so we woudl add wait_until='ACTIVE' a paramater to resize | 14:57 |
| sean-k-mooney | and pass wait_until='SSHABLE' in that test that is failing when we call resize_server | 14:58 |
| gibi | I think that wait_until thing is specificly for create_server | 14:58 |
| gibi | we need to extend resize_server to make the wait | 14:59 |
| gibi | there is even something called as a validation resource to be passed around | 15:00 |
| gibi | ... I'm putting something together... | 15:01 |
| sean-k-mooney | i tought it was passed to waiters.wait_for_server_status | 15:03 |
| sean-k-mooney | hum perhaps not | 15:04 |
| sean-k-mooney | gibi: it was not ment to be specific to create server | 15:04 |
| sean-k-mooney | we have the genirc waiters https://review.opendev.org/c/openstack/tempest/+/817635/15/tempest/common/waiters.py#576 | 15:05 |
| sean-k-mooney | but ya its not plumed into the wait for status waiter at leas not in that patch | 15:06 |
| gibi | yeah it is complicated as you need ssh set up including networking and keyts | 15:06 |
| gibi | keys | 15:06 |
| sean-k-mooney | yep the pingable version was ment to aovid that | 15:07 |
| sean-k-mooney | you still need sec groups and network config | 15:07 |
| sean-k-mooney | but slightly less overhead | 15:07 |
| sean-k-mooney | in anycase it makes sense why resize is still flaky | 15:08 |
| gibi | yepp | 15:09 |
| bauzas | reminder : nova meeting in 47 mins here | 15:13 |
| gmann | bauzas: sean-k-mooney: I am on same page for key generation things 1. remove only key generation support 2. keep 'type' as it is | 15:15 |
| bauzas | ++ | 15:15 |
| gmann | gibi: sean-k-mooney I am not surprise on c9s job unstable. | 15:19 |
| gibi | sean-k-mooney, kashyap, gmann: https://review.opendev.org/c/openstack/tempest/+/842140 | 15:19 |
| sean-k-mooney | gmann: do you know why that was made voting? | 15:19 |
| sean-k-mooney | i was not expecting to have a voting job yet | 15:20 |
| gmann | gibi: sean-k-mooney but we should not make SSH-able by deafult in base class for all test but 842140 approach is good | 15:20 |
| gmann | sean-k-mooney: because it was passing and we want to see how long it will :)as non voting goes unmonitored | 15:20 |
| sean-k-mooney | gmann: right but for nova we had said we did want it to be voting intially at the ptg | 15:21 |
| gmann | gibi: sean-k-mooney I think we can do those grep for detach and make them SSH-able. I tried to do in that series but might not have finished | 15:21 |
| sean-k-mooney | so i was surpsed ot see it in the nova gate | 15:21 |
| sean-k-mooney | sicne we had not got around to creating the nova-next centos job yet | 15:21 |
| sean-k-mooney | but i guess the templats got updated? | 15:21 |
| gmann | sean-k-mooney: you mean 'did not want it to be voting' if 'did want it to be voting' ? | 15:21 |
| gibi | actually I'm fine that is voting, this way we are forced to fix these missing waiters | 15:22 |
| sean-k-mooney | we intially did not want he centos job to be voting in nova. i was suggesting adding one based on nova-next | 15:22 |
| kashyap | gibi: Oh, cool | 15:22 |
| kashyap | wait_until in ("SSHABLE", "PINGABLE") and | 15:22 |
| kashyap | CONF.validation.run_validation | 15:22 |
| sean-k-mooney | gibi: well im ok with it but i was expecting use to let it bake for a while | 15:23 |
| gmann | if detach things is only failing we ill keep identified the test waiting for SSH-able or can grep in advance. | 15:23 |
| gmann | and if more other failure start then we can think of making it non voting/ | 15:23 |
| gibi | gmann: ack | 15:24 |
| sean-k-mooney | gmann: gibi actully i guess the issue is i missed this https://github.com/openstack/nova/commit/ed3abea3b239044bf72277d548da5e2277aed8f5 | 15:25 |
| sean-k-mooney | gibi: i did not think we had added centos based testing in or check/gate pipeline yet at all | 15:26 |
| sean-k-mooney | so when i brought it up in the ptg i was epecting to let it bake for a few months | 15:26 |
| gmann | sean-k-mooney: yeah, centos stream is first we are trying i think and c9s as voting | 15:26 |
| sean-k-mooney | before making it voting | 15:26 |
| sean-k-mooney | gmann: is this happing in other project too | 15:27 |
| gmann | but I am not saying we have to it is like if they are stable we keep them voting and if unstable and no help from centos community then we can think of removing the distro testing frmo testing runtime. example opensuse | 15:27 |
| sean-k-mooney | because we wanted to add it to nova to test very speicific things namely newer libvirt and q35 by defualt | 15:27 |
| gmann | sean-k-mooney: happening failure or making it voitng? | 15:28 |
| sean-k-mooney | gmann: the testing runtime dose not define what os we test on in the indivigual projects | 15:28 |
| gmann | sean-k-mooney: we define minimum expectation of testing and those are needed to be stable as base distro or QA support | 15:29 |
| sean-k-mooney | as in centos being listed in the testing runtime has never ment that nova or neutron tested with it | 15:29 |
| sean-k-mooney | gmann: centos has been in the pti for years but it was only added to nova in yoga by gibis patch | 15:29 |
| gmann | sean-k-mooney: it is not like that, it is meant as a minimum expectation from every project but as devstack itself and our CI/CD is mostly on ubuntu we are doing that only and centos etc are mostly tested in tripleo or so | 15:30 |
| sean-k-mooney | gmann: right i am privding feed back that that is not how it has worked previously | 15:30 |
| gmann | sean-k-mooney: not by gibi patch, it is made voting in tempest side I think lee adedd this c8s integrated job | 15:30 |
| sean-k-mooney | so form my persepitve this is a change in expecation from teh QA team | 15:30 |
| gmann | sean-k-mooney: I know, we are trying if c9s as base can be stable as that is must needed for FIPS testing also which a community wide goal now. | 15:31 |
| sean-k-mooney | yep im aware of the fips intersect | 15:31 |
| gmann | sean-k-mooney: if c9s is not stable we will make it clear 1. can centos community help in making it stable 2. if no then let's not spend time on this and drop the entire support like opensuse we did | 15:31 |
| bauzas | I said it downstream but I saw something fun : https://blueprints.launchpad.net/nova/+spec/update-userdata Registered by Steve Baker on 2013-10-07 | 15:32 |
| gmann | and having them voting is my main goal to know the data | 15:32 |
| bauzas | on that date, I was just starting to work on Nova :) | 15:32 |
| sean-k-mooney | gmann: right so when i brogt it up at the ptg for the last 3 ptgs or more | 15:32 |
| bauzas | and my kid was 3yo :) | 15:32 |
| sean-k-mooney | i started with not having it voting ot not blocke patche merging until it was table | 15:33 |
| sean-k-mooney | gmann: i wanted to have it voting eventuly too | 15:33 |
| sean-k-mooney | and mensiton using fips testing as one of the reasons to do this as well as teting newer libvirt | 15:33 |
| sean-k-mooney | gmann: so i think we are aligned | 15:33 |
| sean-k-mooney | just was not expecting it to be voting until m2 honestly | 15:34 |
| gmann | sean-k-mooney: yeah, I think we are saying same thing. if that become unstable we will make it non voting even that is what QA plan is. but as long as it is passing (we will fix the detach things) it is ok to try | 15:34 |
| sean-k-mooney | gmann: untill we fix the detach thing i think its unstable which is why | 15:34 |
| sean-k-mooney | lee had a patch to change it to voting at the end of the sshable series | 15:35 |
| sean-k-mooney | https://review.opendev.org/c/openstack/devstack/+/834546 | 15:35 |
| sean-k-mooney | i guess is what made the change | 15:35 |
| gmann | yeah that is what I did, fixed most flaky test and see | 15:36 |
| sean-k-mooney | it was proably meged too soone but that fine | 15:36 |
| gmann | but sure, let me make all detach tests SSH-able whatever remaining | 15:36 |
| * sean-k-mooney opens https://review.opendev.org/c/openstack/tempest/+/842140 | 15:37 | |
| gmann | gibi: left 1 comment, rest lgtm https://review.opendev.org/c/openstack/tempest/+/842140/1/tempest/api/compute/base.py#482 | 15:38 |
| sean-k-mooney | hehe | 15:38 |
| sean-k-mooney | we basically asked the same question | 15:38 |
| sean-k-mooney | but i dont think we can hard code | 15:39 |
| sean-k-mooney | to active | 15:39 |
| sean-k-mooney | we can only set it to active if it was "sshaable" or "pingable" | 15:39 |
| sean-k-mooney | if the instance was stopped when you resize it | 15:39 |
| sean-k-mooney | it wont go to active after its resized | 15:39 |
| gibi | it was hardcoded to ACTIVE before the change | 15:40 |
| gibi | so I think we can hardcode it now too :) | 15:40 |
| sean-k-mooney | i guess we dont resize stoped guests? | 15:40 |
| gibi | I guess so | 15:40 |
| sean-k-mooney | ok then they can update that when the add a test for that | 15:40 |
| sean-k-mooney | so ok hardcode it | 15:40 |
| gibi | cool | 15:40 |
| gibi | I will respin | 15:40 |
| gmann | if run validation is disabled then hard coded is ok | 15:40 |
| gmann | and we know detach will be the issue that time | 15:41 |
| gmann | these carry forwards comments from previous PS are becoming more iterating now. I am sure 3rd person reviewing this first time will be confused on it if they are fixed or not - https://review.opendev.org/c/openstack/tempest/+/842140/2/tempest/api/compute/base.py | 15:46 |
| gmann | *irritating :) | 15:47 |
| gmann | may be just for me | 15:47 |
| sean-k-mooney | ya they are | 15:47 |
| sean-k-mooney | i have started activly cleaning them up when i do reviews now | 15:47 |
| sean-k-mooney | so i take a pass over a review comparing the old patch to new and and mark them as done if they are | 15:48 |
| sean-k-mooney | then i do a reivew after the patch is "clean" | 15:48 |
| sean-k-mooney | its a pain | 15:49 |
| gmann | yeah | 15:52 |
| bauzas | #startmeeting nova | 16:02 |
| opendevmeet | Meeting started Tue May 17 16:02:02 2022 UTC and is due to finish in 60 minutes. The chair is bauzas. Information about MeetBot at http://wiki.debian.org/MeetBot. | 16:02 |
| opendevmeet | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 16:02 |
| opendevmeet | The meeting name has been set to 'nova' | 16:02 |
| bauzas | sorry, was late | 16:02 |
| gibi | o/ | 16:02 |
| dansmith | o/ | 16:02 |
| elodilles | o/ | 16:02 |
| Uggla | o/ | 16:02 |
| bauzas | #link https://wiki.openstack.org/wiki/Meetings/Nova#Agenda_for_next_meeting | 16:02 |
| * bauzas was having his brain locked due to some previous meeting | 16:02 | |
| bauzas | and the heat wave here makes me melting | 16:03 |
| bauzas | let's start then | 16:03 |
| bauzas | #topic Bugs (stuck/critical) | 16:03 |
| bauzas | #info No Critical bug | 16:03 |
| bauzas | #link https://bugs.launchpad.net/nova/+bugs?search=Search&field.status=New 14 new untriaged bugs (-5 since the last meeting) | 16:03 |
| bauzas | kudos to Uggla for this excellent work | 16:03 |
| bauzas | he wrote a triage etherpad (yet again, this wasn't mandatory to do it) but in case you wanna know what he triaged https://etherpad.opendev.org/p/nova-bug-triage-20220510 | 16:04 |
| bauzas | #link https://storyboard.openstack.org/#!/project/openstack/placement 26 open stories (0 since the last meeting) in Storyboard for Placement | 16:04 |
| bauzas | #info Add yourself in the team bug roster if you want to help https://etherpad.opendev.org/p/nova-bug-triage-roster | 16:04 |
| bauzas | which leads me to the last point | 16:05 |
| bauzas | sean-k-mooney: are you okay with holding the bug baton for this week ? | 16:05 |
| sean-k-mooney | yes | 16:06 |
| bauzas | gracias | 16:06 |
| bauzas | #info Next bug baton is passed to sean-k-mooney | 16:06 |
| bauzas | any bugs to discuss before we move on ? | 16:06 |
| bauzas | #topic Gate status | 16:07 |
| bauzas | #link https://bugs.launchpad.net/nova/+bugs?field.tag=gate-failure Nova gate bugs | 16:07 |
| bauzas | #link https://zuul.openstack.org/builds?project=openstack%2Fplacement&pipeline=periodic-weekly Placement periodic job status | 16:07 |
| bauzas | #link https://zuul.opendev.org/t/openstack/builds?job_name=nova-emulation&pipeline=periodic-weekly&skip=0 Emulation periodic job runs | 16:07 |
| bauzas | all the above seems fine ^ | 16:07 |
| bauzas | and not fine like in a fire | 16:07 |
| artom | I think Uggla had a bug he wanted to double-check? Dunno if you want to leave that for the open discussion | 16:07 |
| artom | triaged | 16:07 |
| artom | 1959186 (appears to be valid TBC tomorrow) | 16:07 |
| bauzas | artom: we can | 16:08 |
| sean-k-mooney | https://bugs.launchpad.net/nova/+bug/1959186 | 16:08 |
| bauzas | I'll leave it for open discussion then | 16:08 |
| bauzas | people can start digesting it meanwhile | 16:09 |
| * bauzas has zero context yet | 16:09 | |
| sean-k-mooney | initally this does not seam like a bug | 16:09 |
| bauzas | #info Please look at the gate failures and file a bug report with the gate-failure tag. | 16:09 |
| sean-k-mooney | but know behavior but ya | 16:09 |
| bauzas | #info STOP DOING BLIND RECHECKS aka. 'recheck' https://docs.openstack.org/project-team-guide/testing.html#how-to-handle-test-failures | 16:09 |
| sean-k-mooney | lets wait till later | 16:09 |
| bauzas | #topic Release Planning | 16:09 |
| bauzas | #link https://releases.openstack.org/zed/schedule.html | 16:09 |
| bauzas | #info Zed-1 is due in 2 days | 16:09 |
| bauzas | \o/ | 16:09 |
| bauzas | before we discuss about this milestone | 16:10 |
| bauzas | #info Spec review day happened last week on May 10th | 16:10 |
| bauzas | #info 2 specs were approved but a lot of them were reviewed. Kudos to the team for this hard work. | 16:10 |
| gibi | <3 | 16:10 |
| bauzas | I've seen lot of feedback there | 16:10 |
| bauzas | very happy | 16:10 |
| bauzas | now, | 16:10 |
| bauzas | zed-1 | 16:10 |
| bauzas | in theory, we only have one thing to take care of for this milestone | 16:11 |
| bauzas | libraries releases | 16:11 |
| sean-k-mooney | yep the release patches have been proposed | 16:12 |
| sean-k-mooney | i have not looked at them yet | 16:12 |
| sean-k-mooney | are theree any patches peopel would like to wait for? | 16:13 |
| bauzas | that's my humble question | 16:13 |
| sean-k-mooney | there is one patch in flight in os-vif but i dotn think we have to wait for that | 16:13 |
| bauzas | do people care of something they'd like to see published before end of this cycle for the libs ? | 16:13 |
| gibi | no need to wait, we can push out a new lib release any time it is freeee | 16:13 |
| bauzas | os-traits seems interesting to look at | 16:13 |
| sean-k-mooney | yep we use cycle with intermdiaries | 16:14 |
| sean-k-mooney | which requries at least 1 release per cycle | 16:14 |
| sean-k-mooney | but we can have as many as we want | 16:14 |
| bauzas | https://review.opendev.org/q/project:os-traits+is:open is empty :) | 16:14 |
| bauzas | oh shit | 16:14 |
| sean-k-mooney | yep i think we close to approving spec that would add some | 16:14 |
| bauzas | https://review.opendev.org/q/project:openstack/os-traits+is:open | 16:14 |
| bauzas | better now:) | 16:14 |
| sean-k-mooney | so the manial share spec is still pending | 16:15 |
| bauzas | ok, only the manila-related traits | 16:15 |
| sean-k-mooney | so Uggla's patch cant merge yet | 16:15 |
| bauzas | and yeah the spec is pending | 16:15 |
| bauzas | correct | 16:15 |
| bauzas | so, let's release os-traits by now | 16:15 |
| sean-k-mooney | we can merge teh runtim patch but that is not urgent | 16:15 |
| sean-k-mooney | +1 | 16:16 |
| bauzas | and we'll publish a newer os-traits release once the spec is approved | 16:16 |
| bauzas | Uggla: ^ | 16:16 |
| elodilles | note: os-traits is in independent release model, so release patch won't be generated by release team | 16:17 |
| bauzas | https://review.opendev.org/q/project:openstack/os-resource-classes+is:open is a bit more crap | 16:17 |
| bauzas | oh shit you're right | 16:17 |
| bauzas | I was looking at https://governance.openstack.org/tc/reference/projects/nova.html#deliverables | 16:17 |
| Uggla | bauzas, \o/ | 16:17 |
| bauzas | but it's not describing the release models | 16:17 |
| sean-k-mooney | elodilles: os-traits isnt independint is it | 16:17 |
| bauzas | I'm lost with all the references | 16:18 |
| sean-k-mooney | oh it is | 16:18 |
| bauzas | for release models, this is another website unrelated to governance | 16:18 |
| sean-k-mooney | well that does not make sense | 16:18 |
| sean-k-mooney | because placment is tightly coupled ot it | 16:18 |
| elodilles | sean-k-mooney: it is | 16:18 |
| sean-k-mooney | elodilles: placmentn currently only works with exactly one version of os-triats | 16:18 |
| bauzas | ok, found the right doc https://releases.openstack.org/teams/nova.html | 16:18 |
| sean-k-mooney | https://github.com/openstack/releases/blob/master/deliverables/_independent/os-traits.yaml | 16:19 |
| bauzas | this leaves os-r-c and os-traits be independent | 16:19 |
| sean-k-mooney | right but placment assert the exact numober of triat and resouce classes in its test | 16:19 |
| bauzas | https://releases.openstack.org/teams/nova.html#independent | 16:19 |
| bauzas | sean-k-mooney: we discussed this at the PTG right ? | 16:20 |
| sean-k-mooney | so it only works form a unit test perspecive with exactly one version of os-traits/os-rescoue classes | 16:20 |
| bauzas | and we agreed on a proposal | 16:20 |
| sean-k-mooney | yep | 16:20 |
| sean-k-mooney | relax the test | 16:20 |
| bauzas | yup | 16:20 |
| bauzas | so let's do it | 16:20 |
| sean-k-mooney | yep | 16:20 |
| sean-k-mooney | just pointing out that indpenent did not make sense wiht that context | 16:20 |
| sean-k-mooney | but ok we can wait to release it until we have new traits | 16:21 |
| sean-k-mooney | and fix the test in the interim | 16:21 |
| bauzas | point is, we should only care of os-vif and client releases for placement and nova, that's it | 16:21 |
| bauzas | (for zed-1 I mean) | 16:21 |
| sean-k-mooney | yep | 16:21 |
| bauzas | gmann: btw. this is not going well for https://review.opendev.org/c/openstack/os-vif/+/840020 | 16:23 |
| bauzas | but let's discuss this on open discussion ^ | 16:23 |
| bauzas | and let's continue the agenda | 16:23 |
| bauzas | #topic Review priorities | 16:23 |
| bauzas | #link https://review.opendev.org/q/status:open+(project:openstack/nova+OR+project:openstack/placement+OR+project:openstack/os-traits+OR+project:openstack/os-resource-classes+OR+project:openstack/os-vif+OR+project:openstack/python-novaclient+OR+project:openstack/osc-placement)+label:Review-Priority%252B1 | 16:23 |
| bauzas | #link https://review.opendev.org/c/openstack/project-config/+/837595 Gerrit policy for Review-prio contributors flag. Naming bikeshed in there. | 16:23 |
| bauzas | reviews are welcome on ^ | 16:24 |
| bauzas | I see gibi having concerns on the naming | 16:24 |
| gibi | I have no good suggestion | 16:24 |
| gibi | so feel free to ignore me | 16:24 |
| gibi | my problem on the current naming is that is sounds like a core approves that something is a prioirty | 16:27 |
| gibi | but the aim is instead that the core says "I will review this" | 16:27 |
| bauzas | gibi: I can propose something else | 16:27 |
| bauzas | to unblock the patch | 16:27 |
| bauzas | #link https://docs.openstack.org/nova/latest/contributor/process.html#what-the-review-priority-label-in-gerrit-are-use-for Documentation we already have | 16:27 |
| sean-k-mooney | bauzas: sure go for it | 16:27 |
| bauzas | next topic, | 16:28 |
| bauzas | #topic Stable Branches | 16:28 |
| bauzas | elodilles: I haven't seen you updating the section | 16:28 |
| bauzas | so far, so good ? | 16:28 |
| elodilles | #info ussuri and older branches are still blocked, newer branches should be OK | 16:29 |
| bauzas | I think we said last week, newer branches *are* OK :) | 16:29 |
| elodilles | i haven't got any further with the investigation of ussuri branch :/ | 16:29 |
| *** erlon_ is now known as erlon | 16:29 | |
| *** ganso_ is now known as ganso | 16:29 | |
| *** hemna5 is now known as hemna | 16:29 | |
| *** andrewbonney_ is now known as andrewbonney | 16:29 | |
| *** ricolin_ is now known as ricolin | 16:29 | |
| elodilles | bauzas: yes, unfortunately nothing news :( | 16:30 |
| bauzas | damn, looks like we have a netsplit at the time of the meeting | 16:30 |
| elodilles | yepp, it seems :S | 16:30 |
| sean-k-mooney | people will be back soon proably | 16:30 |
| bauzas | hopefully yes | 16:30 |
| sean-k-mooney | looks like mainly NA folks | 16:31 |
| bauzas | let's continue then | 16:31 |
| bauzas | sean-k-mooney: yeah, one OFTC server seems to have killed the conns | 16:31 |
| elodilles | nothing more to add for now :X | 16:31 |
| bauzas | elodilles: at one time, we'll need to cut the rope on the older branches | 16:31 |
| sean-k-mooney | older starting form ? | 16:32 |
| elodilles | bauzas: indeed. now ussuri is in bad shape for weeks now... | 16:32 |
| sean-k-mooney | hum | 16:32 |
| bauzas | sean-k-mooney: ussuri | 16:32 |
| sean-k-mooney | so we woudl be droping train and below too | 16:33 |
| elodilles | actually it fails due to multiple intermittent failures | 16:33 |
| bauzas | we need help or some action in order to mitigate the blocker | 16:33 |
| bauzas | elodilles: yup, constant rechecks | 16:33 |
| sean-k-mooney | bauzas: what is the blocker exactly | 16:33 |
| bauzas | the failure ratio is so high it becomes unpractical to merge | 16:34 |
| sean-k-mooney | right but on what issue | 16:34 |
| bauzas | a couple of them | 16:34 |
| * sean-k-mooney has not looked at stable/ussiri | 16:34 | |
| bauzas | elodilles: we tracked them, right | 16:34 |
| sean-k-mooney | ok i was wonderign if there was one we shoudl look at specificly | 16:34 |
| elodilles | sean-k-mooney: guest kernel panics, volume timeouts, | 16:34 |
| sean-k-mooney | as in volume detach | 16:35 |
| sean-k-mooney | is it resize related? | 16:35 |
| sean-k-mooney | becasue we noticed today that reize is not suing hte sshable change yet | 16:35 |
| sean-k-mooney | gibi proposed a patch to cover that | 16:35 |
| elodilles | also this bug comes up often: https://bugs.launchpad.net/nova/+bug/1901739 | 16:36 |
| sean-k-mooney | im not familar with that | 16:36 |
| sean-k-mooney | looks like lee fixed that in V but it has not been backported | 16:37 |
| gibi | it is a mix of fixes already due to zuul migration and ubuntu version cahgne | 16:37 |
| gibi | chnage | 16:37 |
| sean-k-mooney | ok we can proably move on but if you have a list elodilles please share | 16:37 |
| elodilles | sean-k-mooney: well it's on the recheck list on this patch: https://review.opendev.org/c/openstack/nova/+/838033 | 16:38 |
| elodilles | :/ | 16:38 |
| sean-k-mooney | can we ask infra to force merge that | 16:39 |
| elodilles | sean-k-mooney: it won't help to other patches, they will need the same rechecks | 16:39 |
| sean-k-mooney | yes i know | 16:40 |
| bauzas | in general, we can try to release the jobs | 16:40 |
| bauzas | in order to merge one specific patch we want | 16:40 |
| elodilles | yes, one way is to decrease the test coverage | 16:41 |
| gmann | +1, better than force merge | 16:41 |
| bauzas | we did that in some past | 16:41 |
| sean-k-mooney | ok | 16:41 |
| gibi | here that would mean to cut out devstack based jobs | 16:41 |
| sean-k-mooney | we can make some non voting for now | 16:41 |
| gibi | an rely on unit and functional | 16:41 |
| gibi | *and | 16:41 |
| bauzas | this was ages ago | 16:41 |
| gmann | yeah just for temporary time | 16:41 |
| elodilles | gibi: or some volume related tests. though i don't know which is worse :S | 16:41 |
| bauzas | but I still remember the magic formula | 16:41 |
| gibi | gmann: it would not be temporary if we disable jobs we will never fix tem | 16:42 |
| gibi | them | 16:42 |
| sean-k-mooney | well i asusme we have a limited set of patches that we want to merge and we woudl turn these abck on like droping lc jobs | 16:42 |
| sean-k-mooney | so i was epecting disable patch then revert | 16:42 |
| bauzas | gibi: the idea is to relax the jobs before merging patches we want and then enabling again the jobs | 16:43 |
| gibi | bauzas: and will we do it for every patch we want to merge? | 16:43 |
| bauzas | and see whether the failure ratio drops again | 16:43 |
| gmann | yeah that is what I thought, enabling them again | 16:43 |
| gibi | bauzas: it would work iff we would have patches in queue that fixes those breaks but we dont | 16:43 |
| bauzas | gibi: no, we need to identify a set of patches we'd like to merge in order to help with CI stability | 16:43 |
| gibi | we don't have the fixes :/ | 16:44 |
| gibi | it is not like merge these 5 patches to stabilize ussuri, we don't have those 5 patches ready | 16:44 |
| gibi | we have random patches backported and waiting | 16:44 |
| gmann | humm | 16:44 |
| bauzas | ok, we won't obviously solve this problem by now | 16:45 |
| gmann | one thing to note if we are removing integration tests coverage then it is better to make branch EOL like we did for ocata | 16:45 |
| gibi | so something like: 1) collect the issue to be fixed in ussurit to stabilize CI 2) create fixes for them 3) force merge them 4) profit | 16:45 |
| bauzas | let's state we all know about this problem and we'll figure out the solutions later | 16:45 |
| bauzas | gibi: yeah, sounds 1/ and 2/ are not done yet | 16:45 |
| elodilles | maybe if that "SSHABLE" fix helps things in ussuri we have one, but still we probably have other failures | 16:45 |
| gibi | elodilles: a buncs of SSHABLE already landed, so we should see them help | 16:46 |
| gmann | elodilles: that is again difficult as we are not supposed to use tempest master for ussuri so not all master change will go for ussuri testing | 16:46 |
| bauzas | elodilles: ideally an etherpad of doom may help gathering ideas and feedback | 16:46 |
| gibi | gmann: ahh | 16:46 |
| gibi | that explains | 16:46 |
| gmann | I have patch up to cap tempest for ussuri but that is not merged yet | 16:46 |
| sean-k-mooney | gmann: wait we are not? | 16:46 |
| sean-k-mooney | gmann: temest is ment to be branchless upstream | 16:47 |
| gmann | but we have stopped ussuri support in tempest master | 16:47 |
| sean-k-mooney | hum | 16:47 |
| sean-k-mooney | because of what reason? | 16:47 |
| gmann | sean-k-mooney: yes and master tempest is for SUpported branch and we cannot support EM branches | 16:47 |
| sean-k-mooney | python version ? | 16:47 |
| sean-k-mooney | shoudl that not be a project choice | 16:48 |
| gmann | EM branches - #link https://docs.openstack.org/tempest/latest/stable_branch_support_policy.html | 16:48 |
| sean-k-mooney | tempest should not have to support it | 16:48 |
| sean-k-mooney | but project that support em branches shoudl still be able to choose which tempest to run | 16:48 |
| gmann | sean-k-mooney: we have to pin tempest for EM branch so that using master there can break them anytime | 16:48 |
| gmann | sean-k-mooney: yeah, you can but no guarantee if master tempest will work | 16:49 |
| gmann | as long as it is passing it is ok to use | 16:49 |
| sean-k-mooney | ack | 16:49 |
| sean-k-mooney | so i think i would prefer to keep tempest uncapped until ti breaks | 16:50 |
| elodilles | (well, i guess changes on tempest master can affect not only EM branches, but they are affected most likely) | 16:50 |
| sean-k-mooney | elodilles: well nova uses microversion | 16:50 |
| sean-k-mooney | so it should work form our point of view | 16:50 |
| gmann | elodilles: for supported branches we make sure tempest does not break them but for EM it is not the case | 16:50 |
| sean-k-mooney | chagnes to tempest should not break our stable branches expcti if there ar package dep issues | 16:50 |
| gmann | plan is : by default devstack will cap the tempest in ussuri and project can override it in jobs | 16:51 |
| bauzas | I need to timestop the conversation | 16:51 |
| sean-k-mooney | which i guess is why this was done | 16:51 |
| gmann | yeah, we can discuss it in qa or nova channel after meeting | 16:51 |
| sean-k-mooney | +1 | 16:51 |
| elodilles | +1 | 16:51 |
| bauzas | gmann: sean-k-mooney: gibi: you're all free to continue the conversation after the meeting | 16:51 |
| bauzas | cool | 16:51 |
| gmann | sure | 16:51 |
| bauzas | moving on quickly | 16:52 |
| * gibi will pass out after the meet :/ | 16:52 | |
| bauzas | #topic Open discussion | 16:52 |
| bauzas | Uggla: do you want to discuss https://bugs.launchpad.net/nova/+bug/1959186 | 16:52 |
| bauzas | ? | 16:52 |
| bauzas | Uggla: you left this bug with comments on your triage etherpad | 16:52 |
| Uggla | yep | 16:52 |
| Uggla | It appears valid to me, but I would like a crosscheck. | 16:53 |
| sean-k-mooney | i dont think it is | 16:53 |
| sean-k-mooney | there is a long runnign knwon issue that you cant delete in use image form glance if it uses ceph | 16:54 |
| sean-k-mooney | that only happens if you use raw images | 16:54 |
| sean-k-mooney | at that enabel our copy on write shallow clone code path | 16:54 |
| bauzas | sounds at least unrelated to nova | 16:54 |
| sean-k-mooney | which i suspect is what is happening here with the snapshot | 16:54 |
| bauzas | maybe valid but not on our side | 16:54 |
| bauzas | right? | 16:54 |
| dansmith_ | it's really desired behavior even | 16:55 |
| sean-k-mooney | there have been some feature request in this area | 16:55 |
| sean-k-mooney | some peopel woudl like too break the shallow copy | 16:55 |
| dansmith_ | I think it'd be a feature on the glance side, IIRC | 16:55 |
| *** dansmith_ is now known as dansmith | 16:55 | |
| sean-k-mooney | ya so they have not providded enough info in the bug | 16:56 |
| sean-k-mooney | we do not know the image type | 16:56 |
| dansmith | nova doesn't even know about the linkage after it's created right? so if the link is broken, nova is fine with it | 16:56 |
| sean-k-mooney | to confirm wone way or another | 16:56 |
| sean-k-mooney | am im not sure | 16:56 |
| bauzas | dansmith: hence my point, unrelated to the project | 16:57 |
| bauzas | => Opinion | 16:57 |
| levy14 | have a feature proposal, but not in the agenda yet. may I ask here and gauge interest/feasibility? | 16:57 |
| dansmith | bauzas: yeah | 16:57 |
| sean-k-mooney | there is some metadata on teh image but i dont know if we read that when we create the new vm | 16:57 |
| sean-k-mooney | or tack it on our side | 16:57 |
| bauzas | sean-k-mooney: we can add glance as a service project in the bug report | 16:57 |
| bauzas | and mark it invalid on our side | 16:57 |
| bauzas | we'll see it coming back if that's really a nova bug | 16:58 |
| bauzas | Uggla: works for you ? | 16:58 |
| Uggla | yes | 16:58 |
| bauzas | OK, sold | 16:58 |
| bauzas | #agreed https://bugs.launchpad.net/nova/+bug/1959186 to be punted to Glance | 16:59 |
| sean-k-mooney | ill do that and ask for more info in the bug | 16:59 |
| bauzas | I wanted to discuss https://review.opendev.org/c/openstack/os-vif/+/840020 but we're at time | 17:00 |
| bauzas | thanks all | 17:00 |
| bauzas | #endmeeting | 17:00 |
| opendevmeet | Meeting ended Tue May 17 17:00:16 2022 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 17:00 |
| opendevmeet | Minutes: https://meetings.opendev.org/meetings/nova/2022/nova.2022-05-17-16.02.html | 17:00 |
| opendevmeet | Minutes (text): https://meetings.opendev.org/meetings/nova/2022/nova.2022-05-17-16.02.txt | 17:00 |
| opendevmeet | Log: https://meetings.opendev.org/meetings/nova/2022/nova.2022-05-17-16.02.log.html | 17:00 |
| bauzas | gmann: https://review.opendev.org/c/openstack/os-vif/+/840020 seems problematic to merge | 17:00 |
| sean-k-mooney | bauzas: the failures are unrelated i think | 17:00 |
| sean-k-mooney | there were db issues | 17:00 |
| gmann | yeah let's see I did recheck on this | 17:00 |
| sean-k-mooney | the patch is just remvoing lower constraitns there are no code changes | 17:01 |
| bauzas | sean-k-mooney: ok | 17:01 |
| bauzas | all good then | 17:01 |
| * bauzas has to stop in order to see some folk face-to-face | 17:01 | |
| sean-k-mooney | bauzas: i looked durign the metting thanks for bring it ot my attention i tought that merged already | 17:02 |
| bauzas | np | 17:02 |
| bauzas | happy to help | 17:02 |
| gibi | levy14: I suggest to type your idea in here, or add it to the agenda for next week | 17:02 |
| * bauzas has to invest time and money to figure out how to max out his new 10gbps fiber connection | 17:02 | |
| levy14 | My org found a lot of code that runs in VMs contains credentials in code or config files. I would like to be able to make calls (HTTP requests) to a secret store (Barbican) without providing credentials, and Nova to add them automatically, based on the VM's identity. Much like AWS instance profiles and Azure managed identities work. How should I proceed with this? | 17:03 |
| levy14 | will try to add it to teh agenda for the next time | 17:03 |
| bauzas | levy14: feel free to add it by yourself in https://wiki.openstack.org/wiki/Meetings/Nova#Agenda_for_next_meeting | 17:03 |
| bauzas | you only need LP creds | 17:03 |
| levy14 | so no need to create a blueprint for it? | 17:04 |
| bauzas | levy14: that's what we'll figure out at the meeting :) | 17:04 |
| levy14 | cool, will add it. if any feedback here, will try to capture that too. | 17:04 |
| Uggla | bauzas, leaving IRC, see you in 1h call me on my cell if needed. | 17:05 |
| sean-k-mooney | levy14: that kind of sound like something that was propsoed before | 17:06 |
| bauzas | Uggla: bien sûr et à toute :) | 17:06 |
| sean-k-mooney | related to jwt tokenes or something like that | 17:06 |
| levy14 | sean-k-moonehy: any idea why it is not yet available? any serious blocker? as we see it as a huge security improvement. | 17:07 |
| sean-k-mooney | levy14: so you would lke to have the ablity to request that a set of barbican securest are automaticly injected wehn a vm is created | 17:07 |
| sean-k-mooney | levy14: well you woudl have to opt into this behavior on a pervm basis | 17:07 |
| sean-k-mooney | and that woudl requrie an api change to request it on boot | 17:07 |
| sean-k-mooney | to say which secret or secrets to inject | 17:08 |
| levy14 | nope, I would like to be able to write code that just calls barbican to get a secret, without having to deal with any credentials. nor inject them. | 17:08 |
| sean-k-mooney | levy14: so basically becacuse no oen has asked for it or step up to do the work | 17:08 |
| sean-k-mooney | wehre would that code run | 17:08 |
| levy14 | in the vm | 17:09 |
| sean-k-mooney | well you can do that today | 17:09 |
| levy14 | in whatever language | 17:09 |
| sean-k-mooney | call barbical from the vm | 17:09 |
| sean-k-mooney | but you would need an applciation credental or simialr in the vm to make that call | 17:09 |
| sean-k-mooney | levy14: nova cant run code in the vm for you by the way | 17:09 |
| levy14 | that is what I want to avoid. people are putting those in code or in config files. or injecting them, without properly handling it. and it's a mess. | 17:10 |
| levy14 | how about not needint that at all. all all https calls going of from a vm to (at least) barbican get added a (temp) access token of the vm's identity | 17:10 |
| sean-k-mooney | that would be a security risk | 17:11 |
| sean-k-mooney | since by default nova is not allows to access the users secrets | 17:11 |
| levy14 | it is not about running code in the vm by nova. it is about intercepting outbound http(s) calls and if the destination is barbican, add the token | 17:11 |
| sean-k-mooney | even admin cannot get them by default | 17:11 |
| dmendiza[m] | 👀 | 17:12 |
| sean-k-mooney | levy14: you would need to intercept the request using a man in the midel proxy | 17:12 |
| sean-k-mooney | kind of like the metadta proxy | 17:12 |
| sean-k-mooney | levy14: i think what you want is more like this https://review.opendev.org/c/openstack/keystone/+/784558 | 17:13 |
| levy14 | well, can we use the vm's identity (I don't even know if there is such a think in nova) | 17:13 |
| sean-k-mooney | its nto what you asked for but providign a jwt token to the vm via metadta | 17:13 |
| sean-k-mooney | levy14: vms dont have identity's | 17:13 |
| sean-k-mooney | a vm is a resouce that is owned by a project not a user | 17:14 |
| levy14 | ok, is there a good reason why it cannot get an identity? for this specific purpose. to allow the vm to impersonate a user and access secrets? | 17:15 |
| sean-k-mooney | you mean grab the token form the request that was used to boot the vm and then use that to call barbican | 17:15 |
| levy14 | that would be an option, yes | 17:15 |
| sean-k-mooney | well that would be a giant red flag form an auti perspetivce | 17:16 |
| levy14 | although e.g. what aws does is generate short lived temp tokens on request for this purpose | 17:16 |
| levy14 | ut in the end, any solution that allows users to grab secrets from bbarbican with a well-known identity, and without being a security risk, would be fine | 17:17 |
| levy14 | I just want to end this cancer of credentials in code repos | 17:17 |
| sean-k-mooney | usign jwt is proably trhe way to go | 17:18 |
| sean-k-mooney | there was someone looking at extening keystone to supprot just json web tokens to be able to issue a new keystone token | 17:18 |
| sean-k-mooney | and then useing vendor data or nova metadta to provide the token ot the vm | 17:19 |
| sean-k-mooney | via the metadtat servie | 17:19 |
| levy14 | I am not sure I understand how those would help | 17:19 |
| sean-k-mooney | the vm could get a jwt token form the metadata endpoing without auth based in the vm mac and ip via curl | 17:19 |
| sean-k-mooney | and then use that to issue a full keystone token | 17:20 |
| sean-k-mooney | and make requesuts | 17:20 |
| levy14 | and how would the metadata endpoint know what identity to issue a jwt token for? | 17:21 |
| levy14 | I mean where is the ip/mac <-> identity info coming from? | 17:21 |
| levy14 | so that vm creators/users can control it (so that that user gets access to the ecrets the code in the vm will try to fetch) | 17:22 |
| sean-k-mooney | levy14: that comes for nova/neutron | 17:22 |
| levy14 | so in the end nova/neutron does have an identity for each vm? | 17:23 |
| sean-k-mooney | no | 17:23 |
| sean-k-mooney | it used an external service | 17:23 |
| sean-k-mooney | nova suport external dynimc vender data | 17:23 |
| sean-k-mooney | service like nova-join can use that to add extra info into the metadata aviable to the vm | 17:24 |
| levy14 | interesting | 17:25 |
| sean-k-mooney | https://review.opendev.org/q/topic:bp%252Fjson-web-tokens | 17:25 |
| sean-k-mooney | i think that is the keystone feature | 17:25 |
| sean-k-mooney | let me see if i can find any more info | 17:25 |
| levy14 | I am very curious in any solution that gets me closer to this | 17:25 |
| sean-k-mooney | its been a while since this came up | 17:25 |
| sean-k-mooney | like 2 years or more | 17:25 |
| sean-k-mooney | levy14: https://specs.openstack.org/openstack/keystone-specs/specs/keystone/stein/json-web-tokens.html | 17:26 |
| levy14 | thanks for the links, will do some reading | 17:27 |
| levy14 | and add this to the next meeting's agenda | 17:27 |
| sean-k-mooney | levy14: this likely woudl have to be implemted outsdie fo nova. if it was in nova we would need a very detailed spec of how to do this safely | 17:29 |
| sean-k-mooney | just to set expecations | 17:29 |
| levy14 | I understand | 17:30 |
| levy14 | we're a federation of openstack sites, so solutions that require each site to config this would be problematic. I was hoping for a nova-native solution. even if it's going to take a while. | 17:31 |
| sean-k-mooney | levy14: jrosser was asking about this before | 17:31 |
| sean-k-mooney | levy14: just checked my irc logs let me get you a link | 17:32 |
| sean-k-mooney | they did a poc of injecting the token https://paste.opendev.org/show/798996/ | 17:32 |
| sean-k-mooney | https://meetings.opendev.org/irclogs/%23openstack-nova/%23openstack-nova.2020-10-13.log.html#t2020-10-13T15:30:05 | 17:33 |
| sean-k-mooney | levy14: it was based on https://cloud.google.com/compute/docs/instances/verifying-instance-identity | 17:33 |
| jrosser | sean-k-mooney: I totally ran out of time to take that further | 17:33 |
| jrosser | but still interesting if it can be made to work with an external thing like step-ca (that was my use case) | 17:34 |
| sean-k-mooney | jrosser: maybe you can expalin it to levy14 and they could find resouce to work on it | 17:34 |
| sean-k-mooney | jrosser: i have not been following keystoen recently but i belive tehy are also working on some oath/openid connect supprot this cycle | 17:35 |
| sean-k-mooney | not sure if that is related | 17:35 |
| sean-k-mooney | this https://review.opendev.org/q/topic:bp%252Foauth2-client-credentials-ext | 17:36 |
| levy14 | in any case, adding it to the agenda and we'll see next time | 17:37 |
| levy14 | thanks for the links | 17:37 |
| jrosser | levy14: I have POC code to insert a signed jwt into the metadata | 17:42 |
| jrosser | https://github.com/bbc/nova/commit/c0e9a98fb96d0dff73d0a5c5e7ebad8078fd85a1 | 17:44 |
| sean-k-mooney | jrosser: didnt knwo you worked for/with the bbc | 17:45 |
| sean-k-mooney | or that they used openstack | 17:45 |
| sean-k-mooney | thats cool | 17:45 |
| jrosser | we use it in the R&D labs | 17:45 |
| sean-k-mooney | even still that is nice to hear | 17:45 |
| sean-k-mooney | even if its only for R&D | 17:46 |
| sean-k-mooney | and not production | 17:46 |
| sean-k-mooney | jrosser: your poc is that integrated with keystone in any way. can the jwt token be sued to auth with keystone | 17:47 |
| sean-k-mooney | to then bootstrap to calling barbican | 17:47 |
| sean-k-mooney | or did you just use it for the vm verifcation usecase | 17:47 |
| jrosser | the idea (pretty much copied from what GCP do) is that you could validate that jwt against a published CA cert at some well known location | 17:48 |
| sean-k-mooney | ya ok | 17:48 |
| jrosser | and that is sufficient to say that the VM is what it claims to be | 17:48 |
| sean-k-mooney | so keystone i think now support using jwt to auth and get a keystone token | 17:49 |
| sean-k-mooney | so your poc is not tied into that | 17:49 |
| jrosser | right - though i guess the JWT payload can be pretty arbitrary so i'm not sure how much that would intersect | 17:49 |
| jrosser | hmmm hashicorp vault can authenticate with a JWT | 17:53 |
| jrosser | now that would be really interesting to make work | 17:54 |
| sean-k-mooney | jrosser: levy14's usecases is to use something to allow the vm to auth to barbican | 17:55 |
| sean-k-mooney | jrosser: basically so that you dont need to sotre creds in teh applicaitons configs | 17:56 |
| jrosser | right - i believe that such things are very straightforward in aws | 17:56 |
| sean-k-mooney | jrosser: and since keystone now accpets jwt tokes to auth with my theory was | 17:56 |
| sean-k-mooney | if we added a jwt token in the metadta you coudl use that to get a keystone token and then use that to call barbican | 17:57 |
| sean-k-mooney | and sicne jwt tokesn can expire you could allow that only for x mintues after teh server is booted for intial bootstrap | 17:57 |
| sean-k-mooney | anyway its not my usecase so i wont worry about it for now | 17:58 |
| sean-k-mooney | but it could be a ncie feature if openstack could provide a simialr workflow | 17:59 |
| sean-k-mooney | its proably not a nova feature however | 17:59 |
| sean-k-mooney | ot not mainly | 17:59 |
| sean-k-mooney | a nova feature | 17:59 |
| *** melwitt_ is now known as melwitt | 18:08 | |
| jrosser | seems it is fernet or jws for keystone, not a mixture | 18:09 |
| jrosser | and it does seem more clean to have keystone issue a token then then is in the metadata | 18:10 |
| sean-k-mooney | well for there usecase tehy are tryign to prevent having to put credeitals in the vm to begin with | 18:10 |
| sean-k-mooney | so they need somthign to bootstrap form | 18:10 |
| sean-k-mooney | i think k8s solves this by definign env vars that are only accepable in the container | 18:11 |
| sean-k-mooney | so you can get your pods secret | 18:11 |
| sean-k-mooney | the metadta serivce is the closest thing we have to that | 18:11 |
| jrosser | oh i am confusing the existing jws stuff in keystone with that new jwt auth patch | 18:13 |
| * sean-k-mooney just skimed the open patches so may also be confusign some thigns | 18:14 | |
| *** artom__ is now known as artom | 19:08 | |
| *** ianw_ is now known as ianw | 19:11 | |
| *** hemna6 is now known as hemna | 19:34 | |
| *** dasm is now known as dasm|off | 21:49 | |
| *** mfo is now known as Guest729 | 21:52 | |
| *** mfo_ is now known as mfo | 21:52 | |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!