Monday, 2022-01-24

*** melwitt is now known as Guest34402:26
*** bhagyashris is now known as bhagyashris|ruck03:04
*** bhagyashris is now known as bhagyashris|ruck07:05
AmmadHello,07:23
AmmadAnyone available from nova team ?07:23
AmmadI am able to live migrate vm that is on local disk of compute node to other compute node in wallaby but its not working in xena. 07:24
AmmadTimed out during operation: cannot acquire state change lock (held by monitor=remoteDispatchDomainMigratePrepare3Params)07:25
AmmadSeeing above error from libvirt.07:25
*** hemna7 is now known as hemna07:37
plibeau3hello, when you have time to review https://review.opendev.org/c/openstack/nova/+/82053108:03
opendevreviewFederico Ressi proposed openstack/nova master: Debug Nova APIs call failures  https://review.opendev.org/c/openstack/nova/+/80668308:22
*** bhagyashris_ is now known as bhagyashris|ruck09:52
IPOHello !11:47
IPOIs there any plans for https://review.opendev.org/c/openstack/nova/+/805649 ?11:48
sean-k-mooney[m]i have upgraded my vote to a +211:52
sean-k-mooney[m]IPO so i would like to see that merged sooner rather then later11:53
sean-k-mooney[m]stephenfin not sure if you are arround but you might have input on ^ otherwise bauza or gibi perhaps.11:54
IPOthx for info !11:55
sean-k-mooney[m]stephenfin by the way i just noticed you have an old series up for removing vcpu_pin_set. i assume you wont have time to work on that. one of the patches is removing the reshape did we settle on that being an ok thing to do11:59
sean-k-mooney[m]at this point i think it makes sense to remove it and do the other clean up but i know there was some concern about that in the past.12:00
gibisean-k-mooney[m]: ack, queued but my queue getting long this time as I'm spending quality time with placement SQL queries12:04
sean-k-mooney[m]gibi hopefully someone else will get to it before you, how is the placement work going12:16
*** dasm|afk is now known as dasm12:38
gibisean-k-mooney[m]: some initial patches are up, but there is still work to do13:26
gibiI see you found them 13:27
gibithanks for the initial look13:28
gibiI will get to them13:28
sean-k-mooneymy main uncertentiy for the first too is should we be returnning a 400 for the repated args since it not relaly supported and the current behvior is not what most would expect13:29
gibiyeah I felt the same13:29
sean-k-mooneybut as i said inline i dont know if that requries a microverions13:29
sean-k-mooneyif it does what you have is the best we can do13:29
sean-k-mooneywell pluse a docs not about the behviaor13:29
gibiI will pull in others to agree on to make a separate fix to have that as http400 without a version bump13:29
sean-k-mooneyack13:30
gibias I'm against silently ignoring things13:30
sean-k-mooneyill try and take a look at the rest of the series but swapping to email for a bit13:30
gibisean-k-mooney[m]: thanks, no need to rush, I'm still working on the test for the top patch of the series then I will move from the db layer to the object / api 13:31
sean-k-mooneyack. do you have any nova feature that will consome this this cycle by the way13:31
gibino not at this cycle13:36
gibiin the future I'd like to extend the qos support for  multisegment networks13:37
gibiwhere a port might belong to physnet A or physnet B hence the any-trait support need13:37
sean-k-mooneyright although really your talking about adding suport for multi segment provider networks13:37
sean-k-mooneyand just making sure it works with qos too13:38
sean-k-mooneysince we dont really support the multi provider physnet extnsion at all in nova today13:38
sean-k-mooneywe just use the first one always13:38
gibiyeah we have a big hole in that support, but in a certain edge case it works :)13:38
sean-k-mooneybut yes that is a good use case for it13:38
sean-k-mooneyedge case beign you dont use sriov or qos13:39
gibiif you use both, and the SRIOV is your first segment and you dont use numa aware vswitches then it works :)13:39
sean-k-mooneygibi: once we have the ablity to supprot multi segment network we can do generic physnet arare shculding 13:39
gibisean-k-mooney: yes, and I will look into making the generic case work of course13:40
sean-k-mooneyack which woudl be a nice win13:40
gibiI agree13:40
sean-k-mooneyits kind of amazing that we will only strat checkign if the phynest is on thet host in the 26th release :)13:41
gibibetter late than never :)13:41
sean-k-mooneygibi: by the way the totaly generic case required neturon changes or addtional placment changes to allow matching tratis form sharing RPs without resouce requests13:51
sean-k-mooneye.g. we need to be abel to create shareign RPs for the physnets and assocaited them with the compute nodes via aggreate13:52
sean-k-mooneythos can be resouceless or they can contian invnetories of subnets/ip wand the port can request an allocation form them13:52
sean-k-mooneyso i suspect it will be simpler for you to get it working with qos ports first13:53
gibihm13:53
gibicurrently we have physnet RPs on the bridge RP or on the PF RP13:53
sean-k-mooneythen netron can report the ip inventorires or port invetories or whatever we use so that all ports can have a resouce request form the provier or the trait13:54
sean-k-mooneygibi: yep we do13:54
gibiahh I see now13:54
gibiwe need a resource for the non QoS ports13:54
gibiand that would be the IP13:54
sean-k-mooneyyes13:54
sean-k-mooneyip or port or something else universal13:54
gibihm, there is a spec for ip less ports13:54
sean-k-mooneyor a placment feature to allow matching tratis agiasts resouceless rps13:54
gibiso the ip is not universal13:55
sean-k-mooneyya port likely is the best candiate13:55
gibiOK, lets revisit this for the next cycle13:55
gibibut good points13:55
gibithanks13:55
sean-k-mooneyyep i dont see this as a blocker to what you want to enable13:55
sean-k-mooneyjust to the end goal fo network aware schdulign so it a sperate spec for the fully generic case13:55
sean-k-mooneyshduing based on ip aviablity is partly there for routed networks but as you and bauzas found out when impleenting it they are currently doign a hack with the reserved value13:56
sean-k-mooneywhich we shoudl clean up eventually13:56
gibiyepp13:58
*** bhagyashris is now known as bhagyashris|ruck14:35
opendevreviewDan Smith proposed openstack/nova master: Add service version check workaround for FFU  https://review.opendev.org/c/openstack/nova/+/82609714:57
Guest295sean-k-mooney: gibi ^14:57
*** Guest295 is now known as dansmith14:58
gibidansmith: ack , will check14:58
dansmithgibi: I assume you saw that thread on the ML14:58
gibiyes14:58
dansmithgibi: I've been working on an FFU grenade lately and poked that pretty easy :/14:58
gibiand we talked it through with sean-k-mooney last week14:59
gibiI'm not against the WA flag 14:59
gibias it is disabled by default14:59
dansmithI think it sucks to need it, especially to set it for an upgrade, but I think it's probably the most straightforward mitigation at the moment15:00
gibiyeah. I'm +2 on it, it is simple15:01
sean-k-mooneyi would proably have gone with "compute_service_check_is_fatal" to align with teh vif_plug checks but ya as a temp fix i think its the simplest thing to do15:01
sean-k-mooneyi can take a looks at it in a few just finishing up something15:02
dansmithsean-k-mooney: workarounds were all supposed to be boolean, =False by default and opt-in to some alternate behavior15:02
dansmithso that they should all be "off" in normal operation15:02
dansmithwe've deviated from that quite a bit unfortunately, but I hold the flame :)15:02
sean-k-mooneyya i agree it should be off by default15:02
sean-k-mooneyand is_fatal woudl be one by default so what you suggested is more correct15:03
sean-k-mooneyi was orginially thinking this would not be a workaround15:03
sean-k-mooneyi guess my main question is do we know what we want to replace it with long term15:03
dansmithwe probably need to figure out how to fix this without a workaround, as noted, but this is a simple backport to get people out of the box15:03
dansmithyeah, I dunno15:03
sean-k-mooneyyou made a good point that looking at "up" is potentially racy15:04
dansmithyeah15:05
sean-k-mooneyso for your grenade job i assuem you are just going to set this to true15:05
sean-k-mooneywe need to backport this to wallaby before that job can merge howere right15:06
sean-k-mooneysince it need to be set pre upgrade15:06
sean-k-mooneyor i guess xena not wallaby15:06
dansmithsean-k-mooney: no we only need it on the target15:06
dansmithbut we need to backport it for people15:07
sean-k-mooneywell we only need it on the target but we are not ment to requrie config updates on upgrade15:07
sean-k-mooneyso to have both be true we should have the config option avaiable in the source version right15:07
sean-k-mooneyso just another reason to backport15:07
sean-k-mooneyactully so i guess design question for FFU are we going to assume the "no config updates are requried" part still hold true15:08
sean-k-mooneyor is that just for n to n+115:08
dansmithwell, this is really a FFU-specific config option, so it's appropriate for grenade until it's fixed, IMHO15:09
dansmithrunning (base) with this enabled would be wrong and not like production15:09
sean-k-mooneyack ok i can buy that.15:09
dansmithno config updates from n-2 to n is a different thing, and I don't think we need to stick to that, no15:09
dansmithhowever, this is an upgrade bug/quirk mitigation15:09
opendevreviewLee Yarwood proposed openstack/nova master: WIP libvirt: Register defaults for undefined hw image properties  https://review.opendev.org/c/openstack/nova/+/80070815:29
opendevreviewLee Yarwood proposed openstack/nova master: WIP manage: Add image_property commands  https://review.opendev.org/c/openstack/nova/+/82439215:29
sean-k-mooneydansmith: https://review.opendev.org/c/openstack/nova/+/826097/1/nova/service.py#265 do you want this to work for just the conductor/schduler or also the api15:30
dansmithsean-k-mooney: conductor is the important one, because it's how computes update their record.. does the api check separately?15:30
sean-k-mooneyyes15:31
sean-k-mooneyhttps://github.com/openstack/nova/blob/909cfc76369b94b026cf42b86fb5a310dce21a8c/nova/api/openstack/wsgi_app.py#L5015:31
dansmithah, for wsgi yeah15:31
sean-k-mooneywhen i was check rpc compatiabliy for train-> wallaby i had to commet out both15:31
dansmithyeah I'll update15:31
dansmithI imagine systemd is restarting api enough that it wasn't a problem for me in my grenade15:32
dansmithI'm running a job on top of that now, but I will update when it's done15:32
dansmiththanks for catching15:32
sean-k-mooneyno worries ping me when its up and ill rereview.15:36
opendevreviewDan Smith proposed openstack/nova master: Add service version check workaround for FFU  https://review.opendev.org/c/openstack/nova/+/82609715:54
opendevreviewBalazs Gibizer proposed openstack/placement master: Extend the RP tree DB query to support any-traits  https://review.opendev.org/c/openstack/placement/+/82584916:01
opendevreviewBalazs Gibizer proposed openstack/placement master: Extend the RP tree DB query to support any-traits  https://review.opendev.org/c/openstack/placement/+/82584916:06
*** Guest344 is now known as melwitt16:18
dansmithhuzzah https://review.opendev.org/c/openstack/grenade/+/82610116:52
sean-k-mooneynice 16:55
artomWow17:33
artomAlso, I will never not think of https://knowyourmeme.com/memes/rage-guy-fffffuuuuuuuu17:33
opendevreviewsean mooney proposed openstack/nova master: [WIP] add initial healthcheck support  https://review.opendev.org/c/openstack/nova/+/82501517:48
ade_lee__gmann, sean-k-mooney trying to decide how to handle issue in https://review.opendev.org/c/openstack/tempest/+/81080818:22
ade_lee__gmann, sean-k-mooney it sounds to me like we're never going to fix the plain encryptor provider .18:23
gmannade_lee__: I am very unclear on how many tests we are going to skip or fix for FIPs mode. so my suggestion is to exclude the tests run using --exclude-regex/--exclude-list instead of permanently marking those tests as skip in code. 18:27
gmannthat is how we do for ceph case, 'xyz list of tests does not work for ceph backend so just do not run'18:28
ade_lee__ok18:29
sean-k-mooneygmann: the regex approch makes sense to me18:41
sean-k-mooneyproably using the exclude regex in this case to skip non fips complent tests18:42
gmanncool18:42
gmannyeah18:42
sean-k-mooneythe disadvantage to that is running locally18:42
sean-k-mooneyso can we do it with a tox env18:42
sean-k-mooneyrather then in the job18:42
sean-k-mooneyso you can repoduce locally too18:43
sean-k-mooneye.g. like https://github.com/openstack/tempest/blob/master/tox.ini#L151-L16218:43
sean-k-mooneywe can define an integrated-fips target18:44
gmannyeah, we can do that. 18:45
sean-k-mooneyade_lee__: does ^ work for you18:46
ade_lee__gmann, sean-k-mooney we can do that.  18:48
sean-k-mooneyi think from a downstream pserspcitive that wil make things simpler since we wont have to translate the regex into a jenkins jobs it will just invoke the fips target and get teh same set of test as upstream18:48
gmannade_lee__: cool, so let's add that in follow up patch after paramiko one which has already +2 and under zuul result. 18:49
sean-k-mooneywith that said i have not looked at what our downstream jobs will look like18:49
ade_lee__gmann, sean-k-mooney I suspect for nova though the test suite might pass without the skip, given that the test did not have the fips flag set18:50
ade_lee__so this may be moot for right now for the nova fips test -- it shows up for sure in the cinder tests though18:50
sean-k-mooneyi dont really see how this would be project specific18:51
sean-k-mooneywe are both using the the same integrated-fips jobs no?18:51
ade_lee__sean-k-mooney, its more a matter of which tests run in which test jobs ..18:51
sean-k-mooneywell my point is the latest version fo the nova patch uses a common job18:52
sean-k-mooneyhttps://review.opendev.org/c/openstack/nova/+/790519/20/.zuul.yaml18:52
gmannyeah it run all the tests18:52
sean-k-mooneyits not nova specific18:52
ade_lee__sean-k-mooney, gmann where the fips flag was shown to be needed was here -- https://review.opendev.org/c/openstack/cinder/+/790535/24/.zuul.yaml#13418:53
ade_lee__one or more of those in any case ..18:53
sean-k-mooneythat because fo ceph?18:54
sean-k-mooneywhere is the common job currently defiend18:54
ade_lee__tempest-integrated-storage-fips I think ..18:54
gmannthis one  https://github.com/openstack/tempest/blob/master/zuul.d/integrated-gate.yaml#L30118:55
sean-k-mooneyhere https://github.com/openstack/tempest/blob/master/zuul.d/integrated-gate.yaml#L300-L31418:55
ade_lee__sean-k-mooney, remember this test is for skipping something related to encrypted volumes ..18:55
sean-k-mooneyya18:55
gmannnot sure how that flag which is not there will pass the test. let's see in result18:55
sean-k-mooneyright but in genally we woudl expect those test to run on nova too18:55
gmannyes, it will run18:55
gmannits tempest-full run18:56
sean-k-mooneyade_lee__: i would guess this is speicifc to useing ceph18:56
sean-k-mooneyas by default we will use lvm + iscsi18:56
sean-k-mooneyor rather looking at https://review.opendev.org/c/openstack/cinder/+/79053518:56
sean-k-mooneyit need in a few non lvm/isci default cases18:57
sean-k-mooneyits proably related to   ISCSID_ENABLE_FIPS: True18:57
sean-k-mooneythat is set in all the jobs that failed18:58
sean-k-mooneybut its not in tempest-centos8-stream-fips18:58
ade_lee__sean-k-mooney, that just sets the iscsi chap algorithms to not use md5 -- we actually don't need that any more18:58
ade_lee__sean-k-mooney, so it has no effect.  18:58
ade_lee__sean-k-mooney, abishop and eharney are still investigating the failures 18:59
ade_lee__sean-k-mooney, there is something going on with cryptsetup and fips18:59
sean-k-mooneyi see well right now the generic fips env is running https://github.com/openstack/tempest/blob/master/tox.ini#L101-L11318:59
ade_lee__even in the luks case18:59
sean-k-mooneythe fips jobs seam to be all using the singel node node sets is that intentinal19:00
ade_lee__sean-k-mooney, no - not intentional - thats what we had defeined for nova while testing19:02
sean-k-mooneygmann: actully looking at how those run it proably does not make sense to use a tox target19:03
ade_lee__sean-k-mooney, we can change if needed19:03
sean-k-mooneyade_lee__: for nova we would want multi node for live migration and colde migration coverage19:03
ade_lee__sean-k-mooney, ack - so do we want to change in the generic job as defined here - or define a new job? gmann ^^?19:04
sean-k-mooneygmann: we will need to exclude non fips compatible test but we need to include tests form the tempest plugins for ceph ectra19:05
sean-k-mooneyso doing that in the job configuation actully proable makes more sense then backing it into a bunch of tox envs19:05
sean-k-mooneyade_lee__: porbaly have both so just inhirt form the current jobs and defien the nodeset to be a multi node variant19:06
sean-k-mooneythen the porject can decided which one they want to included19:07
ade_lee__sean-k-mooney, sounds reasonable to me 19:07
gmannsean-k-mooney: ade_lee__ ok, so for multinode case we can inherit from tempest-multinode-full-py3 https://github.com/openstack/tempest/blob/master/zuul.d/integrated-gate.yaml#L23019:08
sean-k-mooneythat also works19:08
sean-k-mooneyi was thinking of just inhirting form tempest-centos8-stream-fips and setting the nodeset19:09
sean-k-mooneyso you dont have to redfiene the fips specific things in two places19:09
ade_lee__sean-k-mooney, though presumably you have to refine all the multinode things ..19:10
sean-k-mooneyno19:10
sean-k-mooneyall the devstack jobs inhrit form multinode19:10
sean-k-mooneyso a single node job is a multi node jobs without a second node defiend19:11
ade_lee__sean-k-mooney, I meant all the things in https://github.com/openstack/tempest/blob/master/zuul.d/integrated-gate.yaml#L232-L24219:11
gmannade_lee__: you do not need USE_PYTHON3 as it is default to true in devstack now19:12
sean-k-mooney ade_lee__  none of that is relevent to multi node19:12
gmannade_lee__: sean-k-mooney so you are saying two jobs 1. existing  tempest-centos8-stream-fips 2. new multinode one ?19:13
sean-k-mooneywhat we need to do is https://github.com/openstack/devstack/blob/master/.zuul.yaml#L179-L20719:13
sean-k-mooneyfor centos19:13
gmannif so why not converting the existing one to multinode 19:13
sean-k-mooneygmann: we could convert it to multinode19:13
sean-k-mooneybut we still need to override the nodeset to use centos-8-stream19:13
sean-k-mooneysince the current jobs assumes that19:14
gmannyeah19:14
sean-k-mooneyso either in devstack or in tempest we need a openstack-two-node-centos-8-stream nodeset19:14
gmannwe need to define that in devstack19:15
sean-k-mooneygmann: i was assumign it would be simpler to get the fips jobs stable single node then add multi node19:15
sean-k-mooneygmann: by convention yes. althoguh its not required by zuul19:15
ade_lee__sean-k-mooney, we should probably stick with convention ..19:16
gmannsean-k-mooney: we can see what all test fails and if that is multinode related or not. if so then we can separate the efforty 19:16
sean-k-mooneyack19:16
sean-k-mooneyok im going to go cook dinner so ill let ye figure out the best way forward19:16
ade_lee__sean-k-mooney, gmann -- so , as I understand the plan19:17
gmannsean-k-mooney: have a good one. lunch time for  me too19:17
gmannade_lee__: thanks 19:17
* sean-k-mooney waits for summary19:17
ade_lee__1. merge current patches for single-node (assuming all tests pass, as I suspect they will)19:17
ade_lee__2. add nodeset definition to devstack19:18
ade_lee__3. add multinode job to tempest with that nodeset -- basically same as single + new nodeset19:18
ade_lee__4. add to nova jobs / replace existing job19:19
ade_lee__thats it .. I think ..19:20
gmannade_lee__: in 3rd and 4th - let's modify existing job  tempest-centos8-stream-fips to use multinode nodeset19:20
sean-k-mooneyya that sounds resonable to me i suspect say keystone wont care about multi node so having both makes sense to me but 3 could be "update single node job to multi node " too as an alternitive19:20
ade_lee__jinx :)19:20
sean-k-mooneygmann: im ok with either option so ill defer to your preferocne on one job or two19:20
ade_lee__deferring to ya'll - but I think its likely other projects may not care about multinode too19:21
gmannsean-k-mooney: ade_lee__ let's start with multinode and if things fails more or so then we can have single node and then multinode separate jobs19:21
gmannade_lee__: sean-k-mooney you mean keystone only right? other projects anyways need to define the new jobs to include their tempest tests from their plugin19:22
ade_lee__gmann, I'm just thinking that given that we have a common place for fips jobs, other projects might want to use or inherit from them19:23
gmannade_lee__: ok, inherit is good point. 19:23
gmannok. let's define both in that case. 19:24
ade_lee__cool19:24
sean-k-mooneygmann: i picked keystone at randmon bascially because noting in keystone should affect cold/live migration so keystone proabley dont want to waste ci resouces testing multi node19:24
sean-k-mooneysame for glance swift ectra19:24
gmannyeah and even it can be treated as base for other tempest plugin jobs too19:24
gmannso single and multinode jobs as separate sounds good19:25
ade_lee__cool - sounds like a plan ! thanks ya'll19:25
ade_lee__go eat!19:25
gmannthanks19:25
gmann:)19:25
sean-k-mooneyo/19:26
*** melwitt is now known as Guest44019:54
*** melwitt_ is now known as melwitt20:55
*** melwitt is now known as Guest45020:59
-opendevstatus- NOTICE: review.opendev.org will have a few short outages over the next few hours (beginning 22:00 UTC) while we rename projects and then upgrade to Gerrit 3.4. See https://lists.opendev.org/pipermail/service-announce/2022-January/000030.html for details.21:04
*** melwitt_ is now known as melwitt21:05
*** melwitt is now known as Guest45121:07
*** Guest451 is now known as melwitt21:10
*** dasm is now known as dasm|off21:30
*** ministry is now known as __ministry21:52
-opendevstatus- NOTICE: The review.opendev.org maintenance work is beginning now. Expect Gerrit outages over the next couple of hours. See https://lists.opendev.org/pipermail/service-announce/2022-January/000030.html for details.22:02
*** melwitt is now known as Guest45622:04
*** Guest456 is now known as melwitt22:12
*** melwitt is now known as Guest45822:13
*** Guest458 is now known as melwitt22:15

Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!