Wednesday, 2018-11-14

*** spatel has quit IRC00:02
*** k_mouza has joined #openstack-nova00:12
*** k_mouza has quit IRC00:17
*** _alastor_ has quit IRC00:17
*** k_mouza has joined #openstack-nova00:20
*** k_mouza has quit IRC00:24
*** tbachman has joined #openstack-nova00:44
*** tbachman_ has joined #openstack-nova00:49
*** tbachman has quit IRC00:49
*** tbachman_ is now known as tbachman00:49
*** yedongcan has joined #openstack-nova01:20
*** gyee has quit IRC01:23
*** hshiina has joined #openstack-nova01:24
*** _alastor_ has joined #openstack-nova01:25
*** _alastor_ has quit IRC01:29
*** imacdonn has quit IRC01:38
*** imacdonn has joined #openstack-nova01:39
*** cfriesen has quit IRC02:18
*** mrsoul has joined #openstack-nova02:37
*** psachin has joined #openstack-nova02:49
*** mhen has quit IRC02:56
*** mhen has joined #openstack-nova02:57
*** cfriesen has joined #openstack-nova03:10
openstackgerrityenai proposed openstack/nova master: Rollback detaching when reserving volume is failed  https://review.openstack.org/61786103:17
*** Bhujay has joined #openstack-nova03:53
*** udesale has joined #openstack-nova04:05
*** sridharg has joined #openstack-nova04:43
*** udesale has quit IRC04:50
*** lpetrut has joined #openstack-nova04:51
*** udesale has joined #openstack-nova05:02
*** efried has joined #openstack-nova05:13
*** lpetrut has quit IRC05:25
*** Bhujay has quit IRC05:29
*** _alastor_ has joined #openstack-nova05:33
*** poojaj has joined #openstack-nova05:33
*** efried has quit IRC05:36
*** _alastor_ has quit IRC05:38
*** moshele has joined #openstack-nova05:42
*** janki has joined #openstack-nova05:44
*** dave-mccowan has joined #openstack-nova05:48
*** moshele has quit IRC05:51
*** Dinesh_Bhor has joined #openstack-nova05:53
*** jichenjc has joined #openstack-nova05:53
*** dave-mccowan has quit IRC05:54
*** Dinesh_Bhor has quit IRC05:58
*** jding1_ has joined #openstack-nova06:04
*** spatel has joined #openstack-nova06:05
*** poojaj has quit IRC06:05
*** poojaj has joined #openstack-nova06:08
*** jackding has quit IRC06:08
*** poojaj has quit IRC06:10
*** spatel has quit IRC06:10
*** alex_xu has quit IRC06:11
*** alex_xu has joined #openstack-nova06:14
*** poojaj has joined #openstack-nova06:15
*** tetsuro has joined #openstack-nova06:30
*** threestrands has quit IRC06:34
*** ratailor has joined #openstack-nova06:35
*** cfriesen has quit IRC06:41
*** poojajadhav has joined #openstack-nova06:50
*** poojaj has quit IRC06:53
sean-k-mooneythe summit starts in 1 hour right? at least the keynote should?06:58
sean-k-mooneyor it could have stared yesterday...06:58
*** poojajadhav has quit IRC07:01
tetsurosean-k-mooney: The summit started yesterday. The keynote of Day 2 starts in 1 hour.07:17
sean-k-mooneytetsuro: hehe yes i started watching the youtube vids07:17
sean-k-mooneytetsuro: are you at the summit this time07:18
tetsuroYes I am, but I didn't see you.07:18
tetsuroyou aren't this time?07:18
sean-k-mooneyno i choose not to travel this time as i have done quite alot of travel, at least for me already this year07:19
tetsuroI see. I hope I can see you next Denver :)07:19
tetsuroAnyway, gotta leave the hotel07:19
sean-k-mooneyyes same, im hoping to be in denver. enjoy07:20
tetsurobye.07:20
*** tetsuro has quit IRC07:20
*** lpetrut has joined #openstack-nova07:21
*** brinzhang has joined #openstack-nova07:23
*** annp has joined #openstack-nova07:30
*** pooja_jadhav has joined #openstack-nova07:39
*** mriedem has joined #openstack-nova07:40
openstackgerritMatt Riedemann proposed openstack/nova master: Add DownCellFixture  https://review.openstack.org/61481007:41
*** pcaruana has joined #openstack-nova07:42
*** lpetrut has quit IRC07:44
*** rodolof has joined #openstack-nova07:58
openstackgerritDan Smith proposed openstack/nova master: Add CellsV2 FAQ about API design decisions  https://review.openstack.org/61789807:59
*** jichenjc has quit IRC08:01
*** cdent has joined #openstack-nova08:04
*** sahid has joined #openstack-nova08:05
*** rodolof has quit IRC08:05
*** rodolof has joined #openstack-nova08:05
mriedemdansmith: this is pretty simple, something we talked about at the ptg to fix a few issues in scheduling to a specific node https://review.openstack.org/#/c/603585/08:08
*** lbragstad has joined #openstack-nova08:08
*** ratailor_ has joined #openstack-nova08:09
*** ratailor__ has joined #openstack-nova08:11
*** ratailor has quit IRC08:12
*** ratailor has joined #openstack-nova08:14
*** ratailor_ has quit IRC08:14
*** ratailor__ has quit IRC08:16
dansmithmriedem: okay I +Wd the spec, but I wish it included more about the actual nova side that actually solves a problem instead of just a placement-side API that _could_ be used to solve a problem08:16
*** Dinesh_Bhor has joined #openstack-nova08:17
sean-k-mooneydansmith: the usecase we discussed at the ptg was related to numa affinity right08:18
dansmithsean-k-mooney: no08:19
dansmithsean-k-mooney: boot or live migration with a target host08:19
sean-k-mooneydansmith: the in_tree paramater?08:20
*** ratailor has quit IRC08:20
mriedemcorrect08:20
mriedemor rebuild with an image change08:20
mriedemor evac with a target host08:20
mriedemor cold migrate with a target host... :)08:21
sean-k-mooneyon the in_tree is set to the compute node uuid08:21
sean-k-mooneyya that makes sense too08:21
openstackgerritMerged openstack/nova-specs master: Spec: allocation candidates in tree  https://review.openstack.org/60358508:22
*** ratailor has joined #openstack-nova08:22
*** erlon has joined #openstack-nova08:36
*** ygk_12345 has joined #openstack-nova08:40
mriedemjroll: just some updates on https://review.openstack.org/#/c/609709/ and i'm happy08:45
*** erlon has quit IRC08:48
*** jpena|off is now known as jpena08:54
openstackgerritsean mooney proposed openstack/os-vif master: always create ovs port during plug  https://review.openstack.org/60238408:58
openstackgerritsean mooney proposed openstack/os-vif master: add isolate_vif config option  https://review.openstack.org/61253408:59
openstackgerritsean mooney proposed openstack/os-vif stable/rocky: always create ovs port during plug  https://review.openstack.org/60985009:02
*** markvoelker has joined #openstack-nova09:03
*** Dinesh_Bhor has quit IRC09:03
*** Dinesh_Bhor has joined #openstack-nova09:04
*** brinzhang has quit IRC09:10
*** pvradu has joined #openstack-nova09:13
*** ralonsoh has joined #openstack-nova09:16
*** Dinesh_Bhor has quit IRC09:21
*** pvradu has quit IRC09:23
*** pvradu has joined #openstack-nova09:23
*** markvoelker has quit IRC09:24
openstackgerritsean mooney proposed openstack/os-vif stable/rocky: add isolate_vif config option  https://review.openstack.org/61628509:31
openstackgerritIvaylo Mitev proposed openstack/nova master: VMware: Attach volumes using adapter type from instance  https://review.openstack.org/61659909:34
*** _alastor_ has joined #openstack-nova09:35
*** lee2 has joined #openstack-nova09:38
*** lee2 is now known as lyarwood09:38
jrollmriedem: yep, doing it this morning09:39
*** _alastor_ has quit IRC09:39
*** derekh has joined #openstack-nova09:40
*** cdent has quit IRC09:43
*** Bhujay has joined #openstack-nova09:50
*** Bhujay has quit IRC09:51
*** Bhujay has joined #openstack-nova09:51
*** dave-mccowan has joined #openstack-nova09:54
mriedemKevin_Zheng: when you get a chance, can you update the root volume detach spec with the resize issue i told you about the other night? https://review.openstack.org/#/c/600628/09:57
Kevin_Zhengmriedem: Oh the spec, go it, I was only thinking about the implementation09:58
*** Dinesh_Bhor has joined #openstack-nova09:59
*** lbragstad has quit IRC10:03
mriedemi just want to make sure it's recorded in the spec so we don't forget about it during review10:04
mriedemsince the unshelve/start limitations are listed in the spec10:04
openstackgerritJim Rollenhagen proposed openstack/nova-specs master: Use conductor groups to partition nova-compute services for Ironic  https://review.openstack.org/60970910:05
mriedemresize revert/confirm would have the same limitations - fail if there is no root volume10:05
jrollmriedem: ^ should finally be good to go there now10:05
*** cdent has joined #openstack-nova10:06
mriedemack10:07
*** dave-mccowan has quit IRC10:07
mriedemsean-k-mooney: if you want to follow along https://etherpad.openstack.org/p/BER-nfv-hpc-pain-points10:07
openstackgerritBalazs Gibizer proposed openstack/nova master: Calculate port_id rp_uuid mapping for binding  https://review.openstack.org/61623910:08
openstackgerritBalazs Gibizer proposed openstack/nova master: Pass allocations and traits to neturonv2 api  https://review.openstack.org/61624010:08
openstackgerritBalazs Gibizer proposed openstack/nova master: Send RP uuid  in the port binding  https://review.openstack.org/56945910:08
openstackgerritBalazs Gibizer proposed openstack/nova master: Test boot with more ports with bandwidth request  https://review.openstack.org/57331710:08
sean-k-mooneymriedem: yep i already had it open :)10:09
*** ygk_12345 has quit IRC10:09
*** priteau has joined #openstack-nova10:14
openstackgerritBalazs Gibizer proposed openstack/nova master: Test boot with more ports with bandwidth request  https://review.openstack.org/57331710:21
mriedemlogan-: did you get that resize bug opened?10:26
*** ccamacho has joined #openstack-nova10:27
*** ccamacho has quit IRC10:27
*** ccamacho has joined #openstack-nova10:27
*** priteau has quit IRC10:39
*** mriedem has quit IRC10:40
*** Dinesh_Bhor has quit IRC10:40
*** yedongcan has quit IRC10:41
sean-k-mooneylooks like there was a good conversation in https://etherpad.openstack.org/p/BER-nfv-hpc-pain-points10:44
*** lbragstad has joined #openstack-nova10:49
*** alexchadin has joined #openstack-nova10:49
*** yedongcan has joined #openstack-nova10:50
*** yedongcan has left #openstack-nova10:55
*** udesale has quit IRC10:59
*** priteau has joined #openstack-nova11:02
*** pooja_jadhav has quit IRC11:05
*** dave-mccowan has joined #openstack-nova11:06
logan-mriedem: https://bugs.launchpad.net/nova/+bug/180333111:13
openstackLaunchpad bug 1803331 in OpenStack Compute (nova) "Root disk lost when resizing instance from imagebackend to rbd backed flavor" [Undecided,New]11:13
*** alexchadin has quit IRC11:16
*** lbragstad has quit IRC11:18
lyarwoodlogan-: https://review.openstack.org/#/c/578846/ might resolve that assuming this is a cause of us cleaning up aggresively after a failure, hard to tell from the trace alone.11:20
sean-k-mooneylogan-: how do you have an RBD backed flavor? isnt the storage backend determined by the nova.conf not the flaovor?11:22
lyarwoodsean-k-mooney: host aggregates11:23
sean-k-mooneylyarwood: right but just because you are afinities flaovrs to host aggrages and that chnages your network backend does not mean this is related to flaovrs11:24
openstackgerritSean McGinnis proposed openstack/nova master: Remove v1 check in Cinder client version lookup  https://review.openstack.org/61792711:24
sean-k-mooneythis is simply a bug with rezieing between image backed nodes and rbd nodes11:25
lyarwoodsean-k-mooney: imagebackend and yeah agreed I was just passing on what was described in the bug11:25
*** mdbooth has joined #openstack-nova11:25
lyarwoodsean-k-mooney: this is all imagebackend, I'm not sure why logan- has called one out as imagebackend and the other as rbd11:25
mdboothlyarwood: reading11:25
sean-k-mooneyreading the bug it seams the issues is with migrating from imagebacked nodes to rbd backed nodes no?11:26
logan-Sorry if it is confusing. Yes these are different nodes with different storage backends configured in nova.conf11:27
logan-sean-k-mooney: correct11:27
sean-k-mooneyi would assume the same behavior would exist if you just migrated and did not reszie11:27
mdboothSounds like there's 2 issues there11:27
mdboothFirstly, resizing between compute hosts using different imagebackends isn't supported11:27
openstackgerritBalazs Gibizer proposed openstack/nova master: Pass allocations and traits to neturonv2 api  https://review.openstack.org/61624011:28
openstackgerritBalazs Gibizer proposed openstack/nova master: Send RP uuid  in the port binding  https://review.openstack.org/56945911:28
openstackgerritBalazs Gibizer proposed openstack/nova master: Test boot with more ports with bandwidth request  https://review.openstack.org/57331711:28
mdboothSo, I guess that's not an issue, except that we could do better at preventing the attempt11:28
mdboothI think sean-k-mooney mentioned that the other week.11:28
logan-mdbooth: right, I don’t expect it to be. I just expect it to fail and not destroy the instance :)11:28
mdboothSecondly, we seem to have lost the root disk, which is extremely bad11:28
*** dave-mccowan has quit IRC11:28
sean-k-mooneylogan-: i actully dont think its unresobaly to support we jsut dont today11:28
mdboothsean-k-mooney: Conceptually it's not unreasonable, it's just that the imagebackend code is utterly awful.11:29
* mdbooth has refactored it twice, but it didn't land.11:30
*** ratailor has quit IRC11:30
sean-k-mooneyi was going to suggest makeing https://bugs.launchpad.net/nova/+bug/1803331 more general and simply stating "live migration between image backend types fails with loss of the root disk."11:30
openstackLaunchpad bug 1803331 in OpenStack Compute (nova) "Root disk lost when resizing instance from imagebackend to rbd backed flavor" [Undecided,New]11:30
mdboothsean-k-mooney: This is resize11:31
mdboothsean-k-mooney: Completely different code path.11:31
sean-k-mooneymdbooth: yes it is but it would also break on migreate right11:31
mdboothLive migration might actually work.11:31
sean-k-mooneysorry cold migrate11:31
*** priteau has quit IRC11:32
sean-k-mooneycold migrate and resize would both triger this. live migrate who knows11:32
mdboothlogan-: At this point if you were a RH customer I'd ask you for sosreports.11:32
mdboothBasically I'd want to see full logs from the initial error which caused the root disk to be deleted.11:32
mdboothThe subsequent error after the root disk has been deleted isn't all that interesting.11:33
sean-k-mooneymdbooth: in theory this would alow apply to any missmatch in image backend right e.g. any combinantion lvm or image or rbd where the backend changes.11:34
mdboothsean-k-mooney: Yes. The code explicitly assumes that the layouts are identical on source and dest.11:34
sean-k-mooneyso this seams like something we should add to the host state object and then check in the scheuler in the future11:35
logan-Ok I can pull more logs this afternoon. What would help? nova-compute in debug on both source & dest I’m guessing?11:35
mdboothlogan-: Yep, specifically for the resize operation itself, not just the subsequent failure to boot11:36
logan-Got it11:36
logan-Will do. Thanks11:36
*** cdent has quit IRC11:36
mdboothlogan-: If you look in the instance directory on the destination you should find the root disk there11:40
mdboothlogan-: Also, you should be able to revert resize11:40
mdboothI don't think we've actually deleted the root disk here, it's just that the dest is looking for it in the wrong place.11:41
sean-k-mooneywe might have deleted it on the source node11:42
sean-k-mooneyor do we not do that until we confirm and your guessing on revert it tries to find it using the location form the dest11:43
mdboothNot until confirm11:45
mdboothRevert will use the copy which remains on the source11:45
mdboothSo actually I think this is just a case of extremely bad user experience. The operation isn't supported, but we could warn earlier rather than going down in flames. I don't believe any data has been lost.11:46
mdboothlogan-: If ^^^ is correct, no need for those logs, btw.11:47
openstackgerritMerged openstack/nova stable/rocky: Set defult value of num_nvme_discover_tries=5  https://review.openstack.org/60868311:56
openstackgerritMartin Midolesov proposed openstack/nova master: Allow driver to specify switch&port for faster lookup  https://review.openstack.org/61769512:00
*** ygk_12345 has joined #openstack-nova12:00
*** hshiina has quit IRC12:07
*** tbachman has quit IRC12:09
*** cdent has joined #openstack-nova12:12
*** priteau has joined #openstack-nova12:16
*** mriedem has joined #openstack-nova12:19
*** dave-mccowan has joined #openstack-nova12:26
*** ygk_12345 has quit IRC12:27
*** jpena is now known as jpena|lunch12:31
*** tbachman has joined #openstack-nova12:34
*** priteau has quit IRC12:36
*** tetsuro has joined #openstack-nova12:37
*** tbachman_ has joined #openstack-nova12:38
*** tbachman has quit IRC12:38
*** tbachman_ is now known as tbachman12:38
*** bnemec has joined #openstack-nova12:39
*** panda is now known as panda|rover12:42
*** Dinesh_Bhor has joined #openstack-nova12:47
openstackgerritMerged openstack/nova-specs master: Use conductor groups to partition nova-compute services for Ironic  https://review.openstack.org/60970912:52
*** tetsuro has quit IRC12:57
*** lbragstad has joined #openstack-nova13:01
*** tetsuro has joined #openstack-nova13:01
*** moshele has joined #openstack-nova13:02
*** bnemec has quit IRC13:08
openstackgerritChris Dent proposed openstack/nova master: WIP: Use external placement in functional tests  https://review.openstack.org/61794113:08
*** jaosorior has joined #openstack-nova13:14
*** Bhujay has quit IRC13:14
*** ThomasWhite has quit IRC13:16
*** _alastor_ has joined #openstack-nova13:17
*** moshele has quit IRC13:18
*** jaosorior has quit IRC13:18
*** ivaat has joined #openstack-nova13:18
logan-updated the bug with additional details mdbooth13:19
openstackgerritChris Dent proposed openstack/nova master: WIP: Use external placement in functional tests  https://review.openstack.org/61794113:19
openstackgerritBalazs Gibizer proposed openstack/nova master: Calculate port_id rp_uuid mapping for binding  https://review.openstack.org/61623913:23
openstackgerritBalazs Gibizer proposed openstack/nova master: Pass allocations and traits to neturonv2 api  https://review.openstack.org/61624013:23
openstackgerritBalazs Gibizer proposed openstack/nova master: Send RP uuid  in the port binding  https://review.openstack.org/56945913:23
openstackgerritBalazs Gibizer proposed openstack/nova master: Test boot with more ports with bandwidth request  https://review.openstack.org/57331713:23
mdboothlogan-: Can you retry the revert after setting instance state to 'resize_confirm' ?13:26
*** udesale has joined #openstack-nova13:27
ivaathi i am seeking help for this: https://bugs.launchpad.net/nova/+bug/180132613:32
openstackLaunchpad bug 1801326 in OpenStack Compute (nova) "CRITICAL keystonemiddleware.auth_token" [Undecided,New]13:32
*** tetsuro has quit IRC13:32
*** tbachman has quit IRC13:34
*** jpena|lunch is now known as jpena13:37
*** tetsuro has joined #openstack-nova13:37
*** tbachman has joined #openstack-nova13:37
*** lbragstad has quit IRC13:38
openstackgerritChris Dent proposed openstack/nova master: WIP: Use external placement in functional tests  https://review.openstack.org/61794113:42
*** skatsaounis has quit IRC13:45
*** skatsaounis has joined #openstack-nova13:46
logan-mdbooth: I guess that will require database shenanigans? "openstack server set: error: argument --state: invalid choice: u'resize_confirm' (choose from 'active', 'error')"13:47
mriedemlogan-: heh, reset state won't allow you to set it back to VERIFY_RESIZE13:50
mriedemso yes, db surgery13:50
logan-ok i'll give it a shot13:51
*** ccamacho has quit IRC13:51
*** hongda has joined #openstack-nova13:51
*** openstackgerrit has quit IRC13:51
mriedembecause you need to get to this https://github.com/openstack/nova/blob/master/nova/compute/api.py#L350613:51
*** ccamacho has joined #openstack-nova13:53
ivaathi! i am seeking help for this: https://bugs.launchpad.net/nova/+bug/180132613:53
openstackLaunchpad bug 1801326 in OpenStack Compute (nova) "CRITICAL keystonemiddleware.auth_token" [Undecided,New]13:53
ivaati am sure it is bug13:54
*** tetsuro has quit IRC13:55
*** tetsuro has joined #openstack-nova13:55
*** hongda has quit IRC13:55
*** zigo has joined #openstack-nova13:57
mriedemivaat: see my comment in the bug, it's not a bug, unless you can investigate and prove otherwise13:58
mriedemit looks like a misconfiguration13:58
ivaatmriedem, well i attached nova.conf as well. tried user with logging in with same user. works13:59
ivaatif it looks like misconfiguration then could you point out what part in config? i have gone trough config several times. followed official doc14:00
mriedemsee my comment on the bug14:00
ivaatbug seems to be deleted14:00
ivaatsry going directly with bug link i can see it now14:01
ivaatmriedem, wtf. i just confirmed i have done this several times.. just take your time when helping. conf was attached to see what is wrong14:02
*** dave-mccowan has quit IRC14:03
logan-i tried hacking instances.vm_state='resized' and now it says "Instance has not been resized." so I guess there is some other state fields that need to be updated to hack it back to a revertable state14:03
mriedemivaat: i looked at the paste in the bug, i didn't see [keystone_authtoken] in there14:05
mriedemlogan-: so what did you do? vm was in error and you fixed the disk and tried to confirm again but that failed b/c it was in error rather than resized status?14:07
ivaatat moment when i installed was no guide to add keystone_authtoken14:07
mriedemivaat: i don't know what install guide you were looking at14:07
*** tetsuro has quit IRC14:07
mriedemmaybe something very old14:08
*** aspiers[m] has joined #openstack-nova14:08
*** cfriesen has joined #openstack-nova14:09
ivaatalright, i will test it out14:09
ivaatthx for now14:09
aspiers[m]mriedem: does it make sense to aim for a quick hallway discussion about SEV while we're here? sean-k-mooney too if you're around14:10
ivaateven nova conf is missing such keyword.. since it is section it should have been there14:10
ivaati think i got previous version conf14:10
*** moshele has joined #openstack-nova14:10
logan-mriedem: ya, resize -> error -> unable to revert resize due to error state14:10
sean-k-mooneyaspiers[m]: im not in berlin unfortunetly but im on irc14:11
*** tetsuro has joined #openstack-nova14:11
aspiers[m]sean-k-mooney: OK thanks. We can continue on the review of course, but feel free to ping me here too if you prefer14:12
logan-So looking at how to revert the resize and return it back to the source node where the disk is still present in /var/lib/nova/instances/<uuid>_resize/*14:12
sean-k-mooneyif im needed i could dial in to a googel hangout but im sure there are enough people to sync up with to figure things out there without me14:12
aspiers[m]I'm currently liasing with our SEV engineers and AMD's to get more clarity around the questions you guys raised14:12
*** spatel has joined #openstack-nova14:12
*** erlon has joined #openstack-nova14:13
aspiers[m]guessing a hangout won't be needed but will certainly let you know if I'm wrong14:15
*** moshele has quit IRC14:15
sean-k-mooneyaspiers[m]: ok just read your responces. if we can just document what will work and what wont then im more or less happy.14:15
sean-k-mooneyaspiers[m]: im also fine with an fix constatin + vm ram + video ram14:16
aspiers[m]sean-k-mooney: absolutely - next patch set should give much more clarity over what is expected to work vs. not14:16
sean-k-mooneybut if we go that way i would make the constant a config value14:16
*** efried has joined #openstack-nova14:16
aspiers[m]OK that sounds like a good idea as a short-term workaround until qemu exposes that calculation14:16
aspiers[m]so you mean a fixed configurable value for the overhead per VM, right? so <hard_limit> would be calculated as flavor ram plus that value14:17
sean-k-mooneythe issue is qemu cant really know this up front as it depends on several things that can cahgne at runtime and from version to version14:17
aspiers[m]possibly with an adjustment if guest video ram is specified via extra specs or image props14:18
*** _alastor_ has quit IRC14:18
*** openstackgerrit has joined #openstack-nova14:18
openstackgerritChris Dent proposed openstack/nova master: WIP: Use external placement in functional tests  https://review.openstack.org/61794114:18
sean-k-mooneyaspiers[m]: ya it could be a config value for the over head or a flavor extraspec14:18
sean-k-mooneyjust some way to reduce or increase the 512 without code change if we mesuer it is too big or two small later14:19
sean-k-mooneyso it would be ram + video ram + a configurable constat overhead14:19
*** itlinux has joined #openstack-nova14:20
sean-k-mooneythat said i think we do this somewhere already14:20
sean-k-mooneyaspiers[m]: we support setting the hardlimit in https://github.com/openstack/nova/blob/88951ca98e1b286b58aa1ad94f9af40b8260c01f/nova/virt/libvirt/config.py#L2102-L2133 and i belive we use it for realtime instances14:23
*** lbragstad has joined #openstack-nova14:23
aspiers[m]sean-k-mooney: ah yeah, I guess probably for the same reason (memory pinning)14:25
*** awaugama has joined #openstack-nova14:25
sean-k-mooneyaspiers[m]: yes we use it so we can memlock the memory for realtime instaces. also all hugepage backed instace will also have the hugepages memlocked which we use for ovs-dpdk so it can do dma into the guest memory14:25
sean-k-mooneyim trying not to be angry about the fact that apparently we only use this in unit tests.... http://codesearch.openstack.org/?q=hard_limit&i=nope&files=&repos=nova14:29
aspiers[m]haha :)14:30
sean-k-mooneyaspiers[m]: well the code is there for you to use...14:31
aspiers[m]indeed. I don't think the actual coding is gonna be too hard14:34
*** dklyle has quit IRC14:39
*** tetsuro has quit IRC14:44
*** tetsuro has joined #openstack-nova14:48
*** _alastor_ has joined #openstack-nova14:50
*** tbachman has quit IRC14:51
*** tbachman has joined #openstack-nova14:52
*** _alastor_ has quit IRC14:56
*** tetsuro has quit IRC14:56
*** spatel has quit IRC14:59
*** itlinux has quit IRC15:00
*** itlinux has joined #openstack-nova15:02
*** lbragstad has quit IRC15:04
*** mriedem has quit IRC15:06
*** lbragstad has joined #openstack-nova15:06
*** itlinux has quit IRC15:07
*** Luzi has joined #openstack-nova15:08
*** cdent has quit IRC15:09
*** N3l1x has joined #openstack-nova15:10
*** cdent has joined #openstack-nova15:12
*** erlon has quit IRC15:16
*** mriedem has joined #openstack-nova15:20
mriedemdansmith: so belmiro gave me another issue to worry about, for which he's going to open a bug. they drop the allocation candidates limit down to 10 to make scheduling faster, but they also at times disable a large number of computes in a cell for $reasons (mnaser also does that for load balancing the cloud); the problem then is b/c of the low alloc candidates limit, placement gives back 10 disabled computes and gets novalidhost15:21
mriedemto workaround it he set reserved = total for inventory in placement on the disabled nodes,15:22
mriedembut that sucks for hundreds of vms15:22
dansmithouch15:22
mriedemi suggested a pre-request placement filter to somehow filter out disabled nodes15:22
mriedemmaybe forbidden trait/aggregate, idk15:22
dansmithhmm15:22
dansmiththey disable compute or disable a cell?15:22
dansmith*computes15:23
sean-k-mooneymriedem: i was going to say yes tag with a trait and use forbindin traits but then we are using placmeent to sotre metadata15:23
dansmithmriedem: I think the right thing to do is to make placement results better so they don't need such a tiny result set to mitigate performance issues15:24
dansmithbut, regardless of what that number is, the same problem could happen15:24
dansmithso if it's a disabled compute situation, I think having compute decorate its RP in some way is probably the thing to do there,15:25
dansmithalthough it generates a lot of chatty traffic from api->compute->placement15:25
sean-k-mooneystephenfin: bauzas by the way is there  a nova cyborg session at the ptg? they demoed something that does not work upstream in the key note. im reviewing the spec again but it would be good to check in with them.15:25
openstackgerritChris Dent proposed openstack/nova master: WIP: Use external placement in functional tests  https://review.openstack.org/61794115:25
*** efried has quit IRC15:26
mriedemi would really want to avoid needing to make placement api calls from the pre-placement filter for sure15:28
mriedemi.e. to rope disabled computes into a resource provider aggregate and then tag a forbidden aggregate filter on the request15:28
dansmithno,15:28
dansmithyou can't really do that I don't think15:29
mriedemthe compute tagging it's resource provider as disabled with a trait is one way yeah - not really the same for ironic (they set reserved = total)15:29
dansmithI meant have computes decorate their own RP with a trait or reserved=total15:29
openstackgerritmelanie witt proposed openstack/nova-specs master: Propose counting quota usage from placement and API database  https://review.openstack.org/50904215:29
mriedemsean-k-mooney: traits are going to be used as metadata in placement15:29
mriedemdansmith: right15:29
mriedemanyway, i told belmiro to open a bug so we don't lose track of the issue15:29
mriedemwe also talked about the upt stuff in compute15:30
dansmithack15:30
mriedemi think he's ok with what's going on in efried's series to clean that up and sighup on changes15:30
mriedemand my short history lesson on how we got here15:30
mriedemmy reward was his other issues15:31
*** erlon has joined #openstack-nova15:31
dansmithheh15:34
*** Luzi has quit IRC15:35
*** tetsuro has joined #openstack-nova15:37
sean-k-mooneymriedem: ya traits are kind of metata but im not sure how peopel would fell about a mainantnce trait or whatever we would use to make the compute node RP as down in placemetn15:39
dansmithI would expect jaypipes to not want that15:40
dansmithsince "cant do stuff" isn't a capability15:40
dansmithreserved=total is a lot more accurate15:40
dansmith"these resources aren't schedulable right now"15:40
sean-k-mooneyyes that was my original concern with a trait15:40
mriedemyeah, plus that matches what the ironic driver already does today15:40
dansmithyep15:40
sean-k-mooneyya that does seem alot cleaner15:41
mriedemi'm not sure if that happens immediately or in a periodic? when the admin disables the compute service in the api, does the api set reserved=total for all inventory on the compute node resource provider?15:41
mriedemseems it should be immediately mirrored15:42
dansmithdisable is an rpc call right?15:42
mriedemno15:42
dansmithbecause xen used to do stuff when you disabled it I think15:42
dansmithmaybe I'm mixing in the maintenance mode call15:42
mriedemit's just a db change15:42
mriedemyeah i think yo uare15:42
dansmithokay, so I was going to say let the compute service do the disable,15:43
dansmithbut same difference just having api do it15:43
sean-k-mooneymriedem: well renebaling it would need the virt driver to resore the old reserved value right15:43
dansmithit's more better to let compute do it I think, just because it's the one that owns the resources15:43
dansmithand the RP15:43
mriedemset_host_enabled was the thing you're thinking of15:43
mriedemin the os-hosts api15:43
jaypipessean-k-mooney, dansmith, mriedem: I'm still reviewing Virgil's Aeneid^W^Wthe Cyborg-nova interaction spec.15:44
dansmithand that doesn't get called when disable?15:44
sean-k-mooneymriedem: what inventor was reserver=total set on by the way15:44
mriedemdansmith: no15:44
mriedemPUT /os-services/{compute_node_id} is just a db change on the services record15:44
*** rodolof has quit IRC15:44
sean-k-mooneyjaypipes: im reviewing that currently too15:44
sean-k-mooneyjaypipes: this one right https://review.openstack.org/#/c/603955/10/specs/stein/approved/nova-cyborg-interaction.rst,unified15:44
dansmithmriedem: okay, well, it's rather unfortunate to have the api muck with the inventory of the compute node, but mechanically it would work15:44
jaypipessean-k-mooney: yes. that one.15:45
dansmithmriedem: un-disabling is a good thought though, since you'd really not want to do that from the api I think15:45
openstackgerritTakashi NATSUME proposed openstack/nova master: Add API ref guideline for body text  https://review.openstack.org/60562815:45
sean-k-mooneyjaypipes: i jsut pushed what i had it take a while to get through that end to end15:45
mriedemdansmith: you mean fixing the reserved value once the service is re-enabled?15:45
openstackgerritTakashi NATSUME proposed openstack/nova master: Remove mox in libvirt/test_driver.py (7)  https://review.openstack.org/57199215:45
dansmithmriedem: yeah15:46
openstackgerritTakashi NATSUME proposed openstack/nova master: Remove mox in libvirt/test_driver.py (8)  https://review.openstack.org/57199315:46
mriedemyeah i don't think we can do that from api, compute knows that info15:46
mriedemsome of it is stored in the compute node record, but maybe not all inventory records15:46
dansmithmriedem: right.. also there's a potential race I guess,15:46
mriedemonly ram/vcpu/disk15:46
mriedemyeah15:46
mriedemlet the compute b/c the source of truth for inventory (unless the API reserves it all...?)15:46
*** artom has joined #openstack-nova15:46
dansmithmriedem: which is api racing with compute to update the inventory.. they could both use generation to catch the race, but probably better to just let one own it15:46
dansmithyeah15:46
openstackgerritTakashi NATSUME proposed openstack/nova master: Add description of custom resource classes  https://review.openstack.org/61672115:47
*** artom has quit IRC15:47
openstackgerritTakashi NATSUME proposed openstack/nova master: Remove mox in virt/test_block_device.py  https://review.openstack.org/56615315:47
*** artom has joined #openstack-nova15:47
*** pvradu has quit IRC15:47
openstackgerritTakashi NATSUME proposed openstack/nova master: api-ref: Add descriptions for vol-backed snapshots  https://review.openstack.org/61508415:48
openstackgerritTakashi NATSUME proposed openstack/nova master: Use links to placement docs in nova docs  https://review.openstack.org/61405615:48
sean-k-mooneyjaypipes: are you at the summit by the way. you ususally only attend the ptgs as of late15:48
openstackgerritTakashi NATSUME proposed openstack/nova master: Remove Placement API reference  https://review.openstack.org/61443715:48
*** sahid has quit IRC15:48
jaypipessean-k-mooney: no. I'm not in Berlin15:48
openstackgerritTakashi NATSUME proposed openstack/nova master: Transform compute_task notifications  https://review.openstack.org/48262915:49
sean-k-mooneyjaypipes: ah ok same. its weird not being there but also i got to review that spec again15:49
openstackgerritTakashi NATSUME proposed openstack/nova master: Use oslo_db.sqlalchemy.test_fixtures  https://review.openstack.org/60935215:49
*** sahid has joined #openstack-nova15:49
openstackgerritTakashi NATSUME proposed openstack/nova master: Fix best_match() deprecation warning  https://review.openstack.org/61120415:50
openstackgerritTakashi NATSUME proposed openstack/nova master: Fix server query examples  https://review.openstack.org/61683415:50
openstackgerritChris Dent proposed openstack/nova master: WIP: Use external placement in functional tests  https://review.openstack.org/61794115:51
*** dklyle has joined #openstack-nova15:55
jaypipessean-k-mooney: hell of a reward...15:56
*** lbragstad has quit IRC16:00
*** dave-mccowan has joined #openstack-nova16:03
mriedemha16:04
*** cdent has quit IRC16:06
*** _alastor_ has joined #openstack-nova16:07
*** ccamacho has quit IRC16:07
*** sridharg has quit IRC16:08
*** irclogbot_2 has quit IRC16:08
*** mriedem has quit IRC16:10
*** mriedem has joined #openstack-nova16:12
*** tbachman has quit IRC16:15
*** irclogbot_2 has joined #openstack-nova16:16
*** dave-mccowan has quit IRC16:20
*** ccamacho has joined #openstack-nova16:23
*** tbachman has joined #openstack-nova16:24
*** tetsuro has quit IRC16:26
*** pcaruana has quit IRC16:29
*** k_mouza has joined #openstack-nova16:30
*** tbachman has quit IRC16:35
*** jding1_ has quit IRC16:35
*** ccamacho has quit IRC16:35
*** k_mouza has quit IRC16:36
*** udesale has quit IRC16:36
*** jackding has joined #openstack-nova16:36
*** dims has quit IRC16:37
*** dims has joined #openstack-nova16:41
*** tbachman has joined #openstack-nova16:41
*** k_mouza has joined #openstack-nova16:44
*** cdent has joined #openstack-nova16:46
*** k_mouza has quit IRC16:48
openstackgerritChris Dent proposed openstack/nova master: WIP: Use external placement in functional tests  https://review.openstack.org/61794116:50
*** jmlowe has joined #openstack-nova16:52
*** mriedem has quit IRC16:53
openstackgerritChris Dent proposed openstack/nova master: WIP: Use external placement in functional tests  https://review.openstack.org/61794116:56
*** psachin has quit IRC16:58
*** k_mouza has joined #openstack-nova16:58
*** Dinesh_Bhor has quit IRC17:02
*** k_mouza has quit IRC17:03
*** erlon has quit IRC17:04
*** pcaruana has joined #openstack-nova17:11
*** janki has quit IRC17:14
*** gyee has joined #openstack-nova17:14
*** mrjk_ has quit IRC17:16
*** tbachman has quit IRC17:21
*** k_mouza has joined #openstack-nova17:25
*** k_mouza has quit IRC17:29
*** k_mouza has joined #openstack-nova17:29
*** k_mouza has quit IRC17:30
*** k_mouza_ has joined #openstack-nova17:30
*** k_mouza_ has quit IRC17:35
openstackgerritChris Dent proposed openstack/nova master: WIP: Use external placement in functional tests  https://review.openstack.org/61794117:35
*** cdent has quit IRC17:38
*** tbachman has joined #openstack-nova17:40
*** k_mouza has joined #openstack-nova17:43
*** panda|rover is now known as panda|rover|off17:47
*** k_mouza has quit IRC17:48
*** k_mouza has joined #openstack-nova17:54
*** k_mouza has quit IRC17:55
*** derekh has quit IRC18:04
*** k_mouza has joined #openstack-nova18:11
*** psachin has joined #openstack-nova18:12
*** jpena is now known as jpena|off18:16
*** k_mouza has quit IRC18:16
*** k_mouza has joined #openstack-nova18:20
*** k_mouza has quit IRC18:24
*** psachin has quit IRC18:25
*** tbachman has quit IRC18:26
*** sahid has quit IRC18:37
*** ralonsoh has quit IRC18:38
*** k_mouza has joined #openstack-nova19:05
*** k_mouza has quit IRC19:09
*** spatel has joined #openstack-nova19:45
spatelsean-k-mooney: ^^19:45
spatelif i have 8 vCPU on instance then its ok to set  ethtool -L eth0 combined 8 ?19:46
spatelor i should set 4  (50% of number of cpu)19:46
*** jmlowe has quit IRC19:50
*** pcaruana has quit IRC19:54
*** artom has quit IRC20:22
*** tetsuro has joined #openstack-nova20:23
*** tetsuro has quit IRC20:25
openstackgerritBalazs Gibizer proposed openstack/nova master: Pass allocations and traits to neturonv2 api  https://review.openstack.org/61624020:32
openstackgerritBalazs Gibizer proposed openstack/nova master: Send RP uuid  in the port binding  https://review.openstack.org/56945920:32
openstackgerritBalazs Gibizer proposed openstack/nova master: Test boot with more ports with bandwidth request  https://review.openstack.org/57331720:32
*** fyx has joined #openstack-nova20:40
*** artom has joined #openstack-nova20:55
*** k_mouza has joined #openstack-nova21:25
*** tbachman has joined #openstack-nova21:27
*** k_mouza_ has joined #openstack-nova21:29
*** k_mouza has quit IRC21:30
*** threestrands has joined #openstack-nova21:32
*** k_mouza_ has quit IRC21:33
*** k_mouza has joined #openstack-nova21:36
*** k_mouza_ has joined #openstack-nova21:39
*** k_mouza has quit IRC21:39
*** k_mouza_ has quit IRC21:43
*** ivaat has quit IRC22:02
*** markvoelker has joined #openstack-nova22:10
*** gokhan has quit IRC22:15
*** awaugama has quit IRC22:34
*** spatel has quit IRC23:02
*** dave-mccowan has joined #openstack-nova23:12
*** maciejjozefczyk has joined #openstack-nova23:13
*** maciejjozefczyk has quit IRC23:32

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!