Friday, 2021-09-10

opendevreviewxiaoxin yang proposed openstack/nova master: Secure boot requires SMM feature enabled  https://review.opendev.org/c/openstack/nova/+/80812602:12
opendevreviewJorhson Deng proposed openstack/nova master: remove some redundant parameters in migrate_server  https://review.opendev.org/c/openstack/nova/+/80814302:41
*** legochen_ is now known as legochen03:36
opendevreviewJorhson Deng proposed openstack/nova master: remove some redundant parameters in migrate_server  https://review.opendev.org/c/openstack/nova/+/80814306:08
admin1seeking info on what happens during a migration .. say i have h1 ( hypervisor1 ) and h2 ( hypervisor2 ) .. when I want to migrate vm1 on h1 -> h2,  what makes the ssh conection .. is it   h1 nova -> h2 nova .. or h1 libvirt -> h2 libvirt or h1 root -> h2 root .. 07:00
gibibauzas: o/ friendly reminder that we need to land melwitt's placemnt fix in Xena RC1 https://review.opendev.org/c/openstack/placement/+/80701407:24
gibimelwitt, dansmith: we are out of EU cores for now, so could you please look at a small doc patch before RC1 https://review.opendev.org/c/openstack/nova/+/807564 ?07:25
gibimelwitt, dansmith: also the reno prelude is ready https://review.opendev.org/c/openstack/nova/+/80778607:25
bauzasgibi: yeah I know and I started to review it07:56
bauzasbut y'know how it's a bit difficult07:56
bauzas+ loving escalations during the last 2 days :(07:56
gibibauzas: thanks. feel free to ask questions about the patch I should be able to answer them07:57
gibiahh escalations. :/07:57
bauzasdon't know what happended07:59
bauzasmaybe because of the moon or because of puppets07:59
bauzasbut looks like we have a lot of customers asking us to help them07:59
gibimaybe they are back from summer vacation :)08:00
bauzasthey're not French :p08:01
gibithen they are just nice and waited for the French to return from their summer vacation :P08:08
admin1one more question .. if cold migration works but live does not -- ceph backend .. where do i need to start looking ..  the nova logs ( debug true ) does not provide any actionable insight .. 08:28
gibibauzas: do you agree with my triage here https://bugs.launchpad.net/nova/+bug/1939856 that it is probably a new feature not a bugfix?09:30
* bauzas looks09:31
bauzasgibi: fwiw, I +wd https://review.opendev.org/c/openstack/nova/+/79235709:34
gibibauzas: thanks09:35
bauzasso we can start to use the new label09:35
bauzasbut then I'll provide a new change explaining what I'd like to have to helping contributors09:35
gibiOK, I will check that09:35
bauzasgibi: about the bug you asked me09:36
bauzasit's honestly a bug but Wishlist looks good to me09:37
bauzasas you said, given the instance was offloaded from the compute, it's no longer possible to ask nova to wait for deleting it09:37
*** elodilles_pto is now known as elodilles09:37
gibiI would be affraid to change the delete behavior of offloaded instances as the current behavior was there forever and it is not really leaking any resources (I checked ports and volumes are cleaned up) 09:42
gibiso probalby people out there are adapted to this09:43
bauzasgibi: we can fix this by documenting :)09:51
gibiI can accept a documentation patch for it sure09:51
kevkoHi folks, in victoria I am hitting this -> /var/log/kolla/nova/nova-compute.log:2021-09-10 09:49:09.815 8 ERROR nova.virt.libvirt.driver [req-8082a663-ca30-4845-9e93-2429c500ba76 31031550a7c94928abbd9dfa8634fba1 7b2ac086026741aa82c6c9c96ded42ee - default default] [instance: cf3ae6f9-a971-4123-ae73-9b597364985a] attaching network adapter failed.: libvirt.libvirtError: internal error: unable to execute QEMU command 'netdev_add': Invalid parameter 10:13
kevkotype for 'vhost', expected: boolean10:13
kevkois it bug ? 10:13
sean-k-mooneykevko: can you past the libvirt xml somewhere so we can see what was generated10:25
sean-k-mooneyalso what libvirt verions are you useing10:25
kevkosean-k-mooney: https://paste.opendev.org/show/809223/10:28
kevkodebian version 10:28
kevkobuster one 10:28
sean-k-mooneythat meets our min libvirt if only barely https://github.com/openstack/nova/blob/stable/victoria/nova/virt/libvirt/driver.py#L22510:32
sean-k-mooneycan you provide teh xml that was generated so we can see where the type element was added10:32
sean-k-mooneyi assume this was an interface attach?10:32
kevkosean-k-mooney: well, i'm running tempests against stack .. if i turn on debug .. will i see xml ? 10:33
kevkosean-k-mooney: yeah, interface attach 10:33
opendevreviewDmitrii Shcherbakov proposed openstack/nova master: [PoC][yoga] Off-path Networking Backends Support  https://review.opendev.org/c/openstack/nova/+/80819910:33
sean-k-mooneyyes if you trun on debug you see a about a 8 line segment of xml printed for attach10:33
sean-k-mooneywe print the full xml when startign the vm and just the bit we are appending for attach10:34
kevkook, give me 5 minutes cca 10:34
kevkosean-k-mooney: https://paste.opendev.org/show/809224/10:37
kevkosean-k-mooney: bigger log -> https://paste.opendev.org/show/809225/10:39
sean-k-mooneythat is odd that looks normal to me10:39
sean-k-mooneyoh what is your qemu version10:40
sean-k-mooneyis it below 4.0.010:40
kevkohttps://paste.opendev.org/show/809226/10:40
kevkonope, it's 5.210:40
sean-k-mooneyok i was wondering if it was older then libvirt suported10:41
sean-k-mooneythye xml we are generating looks corect to me10:41
sean-k-mooneyso i suspect the issue is in the qemu args that libvirt is creeating10:41
sean-k-mooneyhave you looked in the qemu instance long10:42
sean-k-mooney*log10:42
sean-k-mooneyin /var/log/libvirt/qemu/instace*10:42
sean-k-mooneyis there any lower level error there10:42
kevkolet me check10:42
kevkosean-k-mooney: nothing https://paste.opendev.org/show/809227/10:45
sean-k-mooneyodd i dont even see thte attach10:46
sean-k-mooneyalthough it might not log that if it does not work10:46
sean-k-mooneykevko: other then comparing the full xml to the fragment to see if there is any delta (comparing to the exising interface)10:49
sean-k-mooneyim not sure we can be much more help this looks like a libvirt bug10:49
kevkoi'm trying to replace victoria libvrit container for wallaby 10:49
kevkolet me check what will happen :P10:49
sean-k-mooneyit likely should work  nova support a fairly wide range10:50
sean-k-mooneyand libvirt has a pretty stable api10:50
kevkosean-k-mooney: success 10:55
kevkosean-k-mooney: it's working 10:55
kevkonice 10:55
kevkoso minimal version 5.0.0 is buggy 10:55
gibihm, this sounds like https://bugzilla.redhat.com/show_bug.cgi?id=183531210:55
gibithat points to https://bugzilla.redhat.com/show_bug.cgi?id=183271010:55
gibiwhich has a lot of libvirt commits to fix10:55
gibiso yeah, possible libvirt - qemu incompatibility 10:56
kevkoaaaaa10:57
kevkogibi: yeah, it looks like that is the same thing as i met10:57
sean-k-mooneyya since libvirt dont actully maintian stable branchs upstream frocing the distros to do it your milage might vary on the min version10:57
sean-k-mooneyi dont think we actully test our min version in the ci10:58
sean-k-mooneywe choose it based on the api feature set10:58
opendevreviewMerged openstack/nova master: Describe the use of Review-Priority label  https://review.opendev.org/c/openstack/nova/+/79235710:58
sean-k-mooneyand remove any condtional check for api compatiablity below that level10:58
sean-k-mooneymost of our testing id done with whatever version is shiped with ubuntu 20.04 10:59
sean-k-mooneywhich is 6.0.011:00
sean-k-mooneykevko: so anything above 5.0.0 should be api compatibale based on teh feature we use but we have only tested with 6.0.0+ for victoria11:01
sean-k-mooneyhuh https://bugzilla.redhat.com/show_bug.cgi?id=1832710 well that is fun its nice when we are not told about this type of fix...11:05
kevkowell, i'm using kolla debian images which are based on buster in victoria (wallaby is already bullseye ..)11:14
kevkoso 6+ should be ok 11:15
kashyapgibi: FOr that netdev_add thing - so long as you're not mixing "old libvirt and new QEMU" for a given distro, you should not hit that issue.12:26
kashyapgibi: I.e. if you're using CentOS 8.2, just stick to libvirt and QEMU from 8.2; don't mix versions from 8.312:27
gibikashyap: it was kevko who had an env with the issue. I'm not sure about the host OS in kevko's env12:42
kashyapgibi: Sorry, yes; noticed it after you responded.  (Didn't read the KM-long scrollback, afraid/)12:42
gibino worries12:42
gibikevko solved the issue by using newer libvirt12:43
*** legochen_ is now known as legochen13:39
bauzasmelwitt: gibi: eventually +Wd https://review.opendev.org/c/openstack/placement/+/807014 for the create project  bugfix, but with some nits13:50
*** artom_ is now known as artom14:06
gibibauzas: thanks will read shortly14:16
gibibauzas: I agree with your nits. thanks again14:27
bauzasgibi: ack, no worries, it's just nits15:09
* bauzas stops to work for the day15:09
gibibauzas: have a nice weekend15:10
bauzasyou too15:11
* bauzas needs to work for a gate 15:11
bauzas(but not the CI one ;) )15:11
gibi:)15:17
melwittbauzas: thanks, I will make a followup patch to fix the nits15:19
opendevreviewOpenStack Release Bot proposed openstack/os-vif stable/xena: Update .gitreview for stable/xena  https://review.opendev.org/c/openstack/os-vif/+/80845215:20
opendevreviewOpenStack Release Bot proposed openstack/os-vif stable/xena: Update TOX_CONSTRAINTS_FILE for stable/xena  https://review.opendev.org/c/openstack/os-vif/+/80845315:20
opendevreviewOpenStack Release Bot proposed openstack/os-vif master: Update master for stable/xena  https://review.opendev.org/c/openstack/os-vif/+/80845415:20
opendevreviewOpenStack Release Bot proposed openstack/os-vif master: Add Python3 yoga unit tests  https://review.opendev.org/c/openstack/os-vif/+/80845515:20
gibimelwitt: placment the patch just bounced from the gate with strange functional test failures...15:20
opendevreviewOpenStack Release Bot proposed openstack/osc-placement stable/xena: Update .gitreview for stable/xena  https://review.opendev.org/c/openstack/osc-placement/+/80845615:21
opendevreviewOpenStack Release Bot proposed openstack/osc-placement stable/xena: Update TOX_CONSTRAINTS_FILE for stable/xena  https://review.opendev.org/c/openstack/osc-placement/+/80845715:21
melwittgibi: 😩 oh no. I will look15:21
gibihttps://597ae34c2915b12741a5-9637172b828bef74a2ccde1fe8027212.ssl.cf1.rackcdn.com/807014/6/gate/openstack-tox-functional-py38/bc74f18/testr_results.html15:21
opendevreviewOpenStack Release Bot proposed openstack/osc-placement master: Update master for stable/xena  https://review.opendev.org/c/openstack/osc-placement/+/80845815:21
opendevreviewOpenStack Release Bot proposed openstack/python-novaclient stable/xena: Update .gitreview for stable/xena  https://review.opendev.org/c/openstack/python-novaclient/+/80845915:22
opendevreviewOpenStack Release Bot proposed openstack/python-novaclient stable/xena: Update TOX_CONSTRAINTS_FILE for stable/xena  https://review.opendev.org/c/openstack/python-novaclient/+/80846015:22
gibiI can reproduce it on master locally without your patch so this is probably due to requirement change ... :/15:22
opendevreviewOpenStack Release Bot proposed openstack/python-novaclient master: Update master for stable/xena  https://review.opendev.org/c/openstack/python-novaclient/+/80846115:22
opendevreviewOpenStack Release Bot proposed openstack/python-novaclient master: Add Python3 yoga unit tests  https://review.opendev.org/c/openstack/python-novaclient/+/80846215:22
melwittgibi: oh hm15:23
gibiit is not visible in https://zuul.openstack.org/builds?project=openstack%2Fplacement&pipeline=periodic-weekly so it happened in the last ~5 days15:24
gibihm, the global req bumped oslo.db to 11.0.0 two days ago, that is a likely candidate15:26
melwittagree. aside, there likely isn't a cross-placement job on global bump. I wonder if we should add one15:28
gibiOK I confirmed that those tests fail with oslo.db 11.0.0 but passing with oslo.db 10.0.0 (the previous version)15:31
gibiyeah I don't think we have a placement functional job on the global req repo 15:31
sean-k-mooneyi dont see one in https://github.com/openstack/requirements/blob/master/.zuul.d/cross-jobs.yaml15:31
gibibut we should15:31
sean-k-mooneywe have a nova one but you are correct no placment one https://github.com/openstack/requirements/blob/master/.zuul.d/project.yaml#L4215:32
gibiok I failed to figure out how to reformulate those tests with the new sqlalchemy and I have to drop soon. sorry15:58
melwittthanks gibi. I'm also trying to fix, will let you know if I find any success15:59
gibithanks15:59
gibiand good luck15:59
melwittthanks. have a nice weekend o/15:59
gibisame to you too eventually o/16:00
melwitt:)16:01
opendevreviewArtom Lifshitz proposed openstack/nova master: WIP: Gracefully power off guest on instance delete  https://review.opendev.org/c/openstack/nova/+/80847416:03
sean-k-mooneyartom: ok so that jsut add a call self._power_off_instance16:29
sean-k-mooneyare you sure we cant remove   self._shutdown_instance(context, instance, bdms)16:29
artomsean-k-mooney, maybe? Seems it does more than just power if off though...16:29
sean-k-mooneyit delete the volume attachments and detory the domain xml16:31
sean-k-mooneythat seam to be about it16:31
sean-k-mooneypresumable _cleanup_volumes would do the same16:32
sean-k-mooneyok _cleanup_volumes does not handel the attachments16:32
sean-k-mooneyor calling terminat_connection16:32
sean-k-mooneywhat about powerofof16:32
artomlulz we just suck at reusing code, do we?16:32
sean-k-mooneypoweroff instance just calls driver.power off16:33
sean-k-mooneywhat we proably shoudl do is remove the call to driver.destry in self._shutdown_instance16:34
dasmhey team. i'm trying to wrap my head around placement for nova. When VCPUs number are changed? When VM is stopped? Or only when deleted? I found this old thread: http://lists.openstack.org/pipermail/openstack/2014-June/008022.html but not sure if it's still valid.16:34
sean-k-mooneyand have it call _power_off_isntace16:34
artomsean-k-mooney, yeah, maybe. One thing I don't like about my current (admitedly very quickly hacked up) thing is that we're leaking libvirt driver stuff into the compute manager16:35
artomThough I guess it *kinda* makes sense?16:35
opendevreviewmelanie witt proposed openstack/placement master: Call Engine.execute() in func tests for oslo.db 11.0.0  https://review.opendev.org/c/openstack/placement/+/80847716:36
artomOne _power_off_instance() is graceful shutdown, _shutdown_instance() pulls the plug.16:36
artomThat's not really driver-specific...16:36
sean-k-mooneyand they are reverst in libvirt too16:36
sean-k-mooneypoweroff with virsh pulls the plug16:37
sean-k-mooneyif i rememebr correctly16:37
sean-k-mooneyi think you can just call power_off here https://github.com/openstack/nova/blob/e276184892250922c96c38415eabbaf7507f7e17/nova/compute/manager.py#L2897-L289816:37
artomy haev clear werd wen complixaeted werds maek tings moar fun16:37
sean-k-mooneybecasue it will call driver.destroy if the gracefull approch does not work right16:37
artomI think so? _power_off_instance() does the timeout calc for us...16:39
sean-k-mooneyyes16:40
sean-k-mooneyhttps://github.com/openstack/nova/blob/e276184892250922c96c38415eabbaf7507f7e17/nova/compute/manager.py#L2833-L285016:40
sean-k-mooneyhuh apparently that is setabel in the image16:41
artomFun surprises every day16:41
sean-k-mooneythat is also not the correct way to get image properties16:41
sean-k-mooney i mean it works16:41
sean-k-mooneybut we shoudl be using hte objects16:41
sean-k-mooneyoh i see why its not useing the object16:42
sean-k-mooneyits not in the object16:42
sean-k-mooneyso image_os_shutdown_timeout or os_shutdown_timeout is not a legal image property16:43
artomAt this point in the rabbit hole we're going to poke out in Australia16:43
sean-k-mooneythis is just code form before we made the image props ovos16:44
kashyapartom: I politely "well-actuallyed" (I strongly avoid this) here on destroy() behaviour: https://review.opendev.org/c/openstack/nova/+/808474.  Hope you don't mind. :)16:44
sean-k-mooneythat was likely an extenion 16:44
kashyapartom: But your broader goal is a good catch -- with volumes, i.e16:44
* kashyap needs to head out; will check back in later. Have a good weekend!16:45
artomkashyap, valid points16:45
artomWell, I guess you're heading out, will continue in the review16:45
sean-k-mooneyartom: care to add a todo to remove supprot for image_os_shutdown_timeout or add it to the image proprties properly in the future16:46
artomsean-k-mooney, sure, just stick it in gerrit please16:46
sean-k-mooneyartom: ok it should be all there16:57
artomCheers!17:01
melwittgibi: here's my patch for fixing the placement func test jobs https://review.opendev.org/c/openstack/placement/+/80847717:40
dansmithmelwitt: +2d that17:49
dansmithmelwitt: did the other thing to separate the transactions merge?17:49
melwittdansmith: no it's stuck behind this recent fail. I'm about to rebase onto ^17:49
dansmithoh really, okay dang17:50
melwittalso thanks for the +217:50
melwittyeah :(17:50
dansmithwas going to say, we were still "recheck nova fail"-ing again yesterday17:50
dansmithwho else is around to slam this trivial fix in I wonder?17:50
melwittdansmith: yeah bauzas +W'ed earlier today (the transaction thing)17:51
melwittoh maybe gmann 17:51
melwittgmann: are you around to review this gate fix for placement? https://review.opendev.org/c/openstack/placement/+/80847717:52
dansmith++17:52
gmannmelwitt: sure, checking17:52
melwitt\o/17:53
melwitthm actually I'm not sure whether I will need to rebase on the fix. I get confused about whether/when zuul does it automatically17:53
gmannmelwitt: +A17:57
melwittgmann: thank you!! :)17:57
dansmithmelwitt: rebase.. rebase like the wind!17:58
gmanndansmith: melwitt as it is different file it should not require rebase, and zuul should merge it once fix is merged17:58
gmannwe can try recheck once 808477 is merged17:58
dansmithgmann: there's another fix that depends on this17:58
dansmithrebase so we can make sure this lands before that17:59
gmann807014 this one ?17:59
dansmithyeah18:00
gmannI think 807014 should pass with recheck once 808477  is merged18:01
dansmithright,18:01
dansmithbut 807014 is breaking lots of *other* gates, so we want that in the queue ASAP18:02
gmannohk. 18:03
gmannor try re/another +w can trigger gate pipeline directly ?18:03
gmannbut it might do check also. yeah rebase is more safer to put it queue asap18:05
melwittgmann: it's this if you want to try https://review.opendev.org/c/openstack/placement/+/80701418:05
melwittok, will do then18:05
dansmithI think rebasing is better,18:05
dansmithbecause if they get kicked out, we want them to stay in the right order for the next trip18:05
gmannok18:05
opendevreviewmelanie witt proposed openstack/placement master: Narrow scope of set allocations database transaction  https://review.opendev.org/c/openstack/placement/+/80701418:06
dansmithmelwitt: got it18:06
melwittthanks!18:07
opendevreviewMerged openstack/placement master: Call Engine.execute() in func tests for oslo.db 11.0.0  https://review.opendev.org/c/openstack/placement/+/80847719:00
admin1hi all .. is this s known error ? is this a known error coming up when launching new instances "Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible. <class 'amqp.exceptions.InternalError'> (HTTP 500) (Request-ID: req-17bb244e-46c6-4f2f-a213-3a1ae786afc2)" ? 19:58
melwittadmin1: that means an unhandled exception bubbled up to the api. need to check the nova-api.log to see what happened20:03
opendevreviewMerged openstack/placement master: Narrow scope of set allocations database transaction  https://review.opendev.org/c/openstack/placement/+/80701421:09
opendevreviewMerged openstack/nova master: [doc] port-resource-request-groups not landed in Xena  https://review.opendev.org/c/openstack/nova/+/80756423:35

Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!