Wednesday, 2017-05-17

*** yamahata has quit IRC00:00
*** jogo has quit IRC00:00
*** markvoelker has quit IRC00:01
*** crushil has quit IRC00:01
*** crushil has joined #openstack-nova00:02
*** markvoelker has joined #openstack-nova00:02
*** markvoelker has quit IRC00:04
*** markvoelker has joined #openstack-nova00:04
*** baoli has quit IRC00:07
*** liverpooler has quit IRC00:08
*** yamahata has joined #openstack-nova00:15
*** chyka has joined #openstack-nova00:19
*** tjones has joined #openstack-nova00:19
*** nic has quit IRC00:23
*** thorst has joined #openstack-nova00:25
*** thorst has quit IRC00:27
*** crushil has quit IRC00:32
openstackgerritArtom Lifshitz proposed openstack/nova master: Use VIR_DOMAIN_BLOCK_REBASE_COPY_DEV when rebasing  https://review.openstack.org/46520500:33
*** ijw has quit IRC00:36
*** ijw has joined #openstack-nova00:36
*** hongbin has quit IRC00:37
*** fragatin_ has joined #openstack-nova00:38
*** fragatin_ has quit IRC00:39
*** fragatin_ has joined #openstack-nova00:39
*** dimtruck is now known as zz_dimtruck00:40
*** gianpietro has quit IRC00:41
*** fragatina has quit IRC00:41
*** crushil has joined #openstack-nova00:41
*** zz_dimtruck is now known as dimtruck00:42
*** fragatin_ has quit IRC00:44
*** crushil has quit IRC00:50
*** dixiaoli has joined #openstack-nova00:51
*** hieulq has joined #openstack-nova00:51
*** ijw has quit IRC00:54
*** tjones has quit IRC00:54
armaxmriedem: ping00:55
mriedemarmax: pong00:55
armaxquick question00:55
armaxsince I know how much you love numa and sriov code in nova00:55
armaxdo you happen to know anything about this bug https://bugs.launchpad.net/mos/+bug/1658070?00:56
openstackLaunchpad bug 1658070 in Mirantis OpenStack "Failed SR_IOV evacuation with host" [Medium,Confirmed] - Assigned to Sergey Nikitin (snikitin)00:56
armaxapparently it’s targeted to MOS00:56
armaxbut I fear that’s plainly affecting nova00:56
*** tbachman has quit IRC00:56
mriedemhmm, doesn't ring a bell, but i bet cfriesen might00:56
mriedemi think snikitin is off openstack now too00:57
armaxok thanks00:57
armaxI’ll poke around00:57
*** thorst has joined #openstack-nova00:57
mriedemdoesn't mention which release either00:58
mriedemmitaka i guess?00:58
*** diablo_rojo has joined #openstack-nova01:01
mriedemarmax: hmm https://github.com/openstack/nova/commit/dfdae01bd03f522ffab7876b253ec4164193470201:01
mriedem^ went back to newton01:01
armaxI see01:02
*** trungnv has joined #openstack-nova01:02
armaxthat’s good to know01:02
mriedembut that bug was around awhile01:02
mriedemi remember01:02
mriedemit was blocking migration with pci01:02
mriedemhttps://bugs.launchpad.net/nova/+bug/151288001:03
openstackLaunchpad bug 1512880 in OpenStack Compute (nova) newton "Failed cold migration with SR-IOV" [Medium,Fix released]01:03
armaxthat seems like it’s a slightly different use case01:03
*** tbachman has joined #openstack-nova01:03
mriedemmaybe01:04
mriedemevacuate and migrate are pretty similar though01:04
armaxyeah01:04
mriedemcould also be some stuff in the resource tracker in the compute, but the bug report said that if they specify a host it fails, but if they don't it works01:04
armaxperhaps bug https://bugs.launchpad.net/mos/+bug/1658070 is newton?01:04
openstackLaunchpad bug 1658070 in Mirantis OpenStack "Failed SR_IOV evacuation with host" [Medium,Confirmed]01:04
mriedemmoshele and cfriesen are definitely the guys to ask01:04
armaxas the workaround says to let nova figure out the host01:05
mriedemyeah that part confuses me01:05
mriedemunless when you don't specify a host, the scheduler picks the right how to migrate to01:05
mriedem*host01:05
armaxright, though we’d have to figure out whether all hosts are equal or the failure is genuine01:06
*** itlinux has quit IRC01:07
armaxthanks for the feedback, I’ll try to get confirm whether this is an issue still affecting master01:07
mriedemsure, if all else fails moshele in the morning (he's in israel i think)01:08
mriedemcfriesen has no excuse for not being around right now working at all hours, he's in canada01:08
*** harlowja has quit IRC01:08
artomYeah, our country's so big we encompass all timezones, and therefore work around the clock01:11
mriedemhttps://review.openstack.org/#/q/topic:bug/1512880+status:merged+branch:master01:11
mriedemarmax: ^ was really a whole series of changes to get cold migration working with pci devices01:11
mriedemartom: amen01:11
armaxmriedem: so I guess we’ll have to see whether the issue exists on master or not01:12
mriedemi wouldn't be surprised if it does, evacuate isn't integration tested01:13
mriedemanything with pci/sriov is not tested heavily, it's all 3rd party and a limited scope i think01:13
armaxyup01:13
armaxmakes sense01:13
mriedembut the mellanox guys were running cold migration with sriov in their ci i think01:14
mriedembut we can't test evacuate in tempest since you have to take down the single compute service01:14
armaxyeah, that’s not really reasonable on a continuous basis01:14
artomSurely we can hack something around that...01:14
artomBut maybe it's just my late night brain talking01:15
mriedemartom: yeah we've talked about ways to hack that01:15
mriedemcould be in the multinode job but you have to basically lock all other tests from running at the same time01:15
mriedemwould be best if we just had some integration tests to run serially in-tree01:15
artomNothing in evacuate actually depends on the compute being down, no?01:16
*** thorst has quit IRC01:16
*** itlinux has joined #openstack-nova01:16
mriedemyes, the api :)01:16
mriedemhttps://github.com/openstack/nova/blob/master/nova/compute/api.py#L408201:16
artomRight... we could, err, jiggle that check a bit ;)01:16
artomIt's extremely ugly01:16
*** gjayavelu has quit IRC01:17
artomBut it would allow us to run evacuations tests in parallel01:17
mriedemartom: for the last time i wish you'd stop talking about jiggling things around me01:17
artomI made the conscious effort of saying jiggle and not diddle, you should thank me01:17
mriedemi'm sensitive because my wife said i looked apple shaped after i got back from boston01:17
mriedemthose are nowhere near as bad as what that tubaguy says constantly01:18
mriedemwhich i shall not repeat01:18
artomCan Nova know it's running in CI?01:18
mriedemprobably shouldn't,01:18
mriedemi wouldn't do that for tempest01:19
mriedemwe could probably do that with in-tree functional tests01:19
artomI said it was gonna be ugly :/01:19
mriedemwhere you can have multiple compute services running01:19
artomThat seems like a whole lot of scaffolding01:19
mriedemhttps://github.com/openstack/nova/blob/master/nova/tests/functional/regressions/test_bug_1671648.py01:20
mriedem^ is an example of a multi-compute functional test01:20
mriedemto test retries in conductor01:20
armaxthis use case probably falls in the category of issues that are two expensive to test on a continuous basis01:20
armaxtwo -> too01:21
mriedemwe test live migration and cold migration and resize on a continuous basis01:21
mriedemanything involving snapshots would probably take more time than this01:22
mriedemevacuate is just rebuild on another host01:22
*** Daisy has joined #openstack-nova01:22
mriedemidk, you could probably test most of the functionality through the entire stack with what's setup in https://github.com/openstack/nova/blob/master/nova/tests/functional/regressions/test_bug_1671648.py01:22
*** gyee has quit IRC01:22
mriedemanything external like neutron and cinder are faked out, but...01:22
mriedemanywho, i need to go spend time with http://www.adultswim.com/videos/your-pretty-face-is-going-to-hell/01:23
armaxmriedem: have fun and thanks for the feedback01:24
*** cNilesh has joined #openstack-nova01:24
artomWhoa, I parsed 'adultswim' waaay wrong01:24
*** gianpietro has joined #openstack-nova01:24
*** oomichi has quit IRC01:25
*** salv-orlando has quit IRC01:25
mriedemi highly recommend http://www.adultswim.com/videos/your-pretty-face-is-going-to-hell/hammerman/01:25
*** oomichi has joined #openstack-nova01:25
*** gyee has joined #openstack-nova01:26
*** tovin07_ has joined #openstack-nova01:26
*** dave-mccowan has quit IRC01:28
*** Apoorva_ has joined #openstack-nova01:29
*** gianpietro has quit IRC01:29
openstackgerritZhenyu Zheng proposed openstack/nova master: Use plain routes list for server-password endpoint instead of stevedore  https://review.openstack.org/46204901:30
*** dixiaoli has quit IRC01:30
*** thorst has joined #openstack-nova01:31
*** Apoorva has quit IRC01:32
*** Apoorva_ has quit IRC01:33
*** thorst has quit IRC01:33
*** Kevin_Zheng has joined #openstack-nova01:36
*** gianpietro has joined #openstack-nova01:37
*** suresh12 has quit IRC01:41
openstackgerritZhenyu Zheng proposed openstack/nova master: Use plain routes list for server-password endpoint instead of stevedore  https://review.openstack.org/46204901:41
*** Daisy has quit IRC01:42
openstackgerritArtom Lifshitz proposed openstack/nova master: Plumbing for tagged nic attachment  https://review.openstack.org/41562201:42
openstackgerritArtom Lifshitz proposed openstack/nova master: Delete nic metadata when detaching interface  https://review.openstack.org/43538701:42
openstackgerritArtom Lifshitz proposed openstack/nova master: Libvirt support for tagged volume attachment  https://review.openstack.org/41810001:42
openstackgerritArtom Lifshitz proposed openstack/nova master: Delete disk metadata when detaching volume  https://review.openstack.org/43508201:42
*** Daisy has joined #openstack-nova01:42
openstackgerritArtom Lifshitz proposed openstack/nova master: Plumbing for tagged volume attachment  https://review.openstack.org/39194101:42
openstackgerritArtom Lifshitz proposed openstack/nova master: API support for tagged device attachment  https://review.openstack.org/37602101:42
openstackgerritArtom Lifshitz proposed openstack/nova master: Libvirt support for tagged nic attachment  https://review.openstack.org/41552001:42
*** gcb has joined #openstack-nova01:43
*** gianpietro has quit IRC01:45
*** Daisy has quit IRC01:46
*** gongysh has joined #openstack-nova01:47
*** Daisy has joined #openstack-nova01:51
*** itlinux has quit IRC01:51
*** dave-mccowan has joined #openstack-nova01:52
*** Sukhdev has joined #openstack-nova01:55
*** coreywright has quit IRC01:55
*** Sukhdev has quit IRC01:56
*** itlinux has joined #openstack-nova02:00
*** tjones has joined #openstack-nova02:00
*** takashin has quit IRC02:04
*** suresh12 has joined #openstack-nova02:04
*** moshele has joined #openstack-nova02:09
*** suresh12 has quit IRC02:09
*** slaweq has joined #openstack-nova02:10
*** smatzek has joined #openstack-nova02:10
*** coreywright has joined #openstack-nova02:14
*** slaweq has quit IRC02:15
*** yamahata has quit IRC02:16
*** thorst has joined #openstack-nova02:18
*** thorst has quit IRC02:19
*** gianpietro has joined #openstack-nova02:21
*** tjones has quit IRC02:21
*** bkopilov_ has quit IRC02:21
*** bkopilov has quit IRC02:22
*** smatzek has quit IRC02:25
*** gianpietro has quit IRC02:25
*** itlinux has quit IRC02:25
*** jaypipes has joined #openstack-nova02:27
*** thorst has joined #openstack-nova02:29
*** thorst has quit IRC02:29
openstackgerritHuan Xie proposed openstack/nova master: WIP: XenAPI use os-xenapi v2 in nova  https://review.openstack.org/45349302:30
*** tbachman has quit IRC02:32
*** gyee has quit IRC02:35
*** itlinux has joined #openstack-nova02:36
*** moshele has quit IRC02:37
*** moshele has joined #openstack-nova02:37
*** liusheng has quit IRC02:39
*** jaypipes has quit IRC02:40
*** fragatina has joined #openstack-nova02:41
*** dave-mccowan has quit IRC02:42
*** Shunli has joined #openstack-nova02:44
*** fragatina has quit IRC02:45
*** itlinux has quit IRC02:48
*** sree has joined #openstack-nova02:49
*** nicolasbock has quit IRC02:51
*** sree has quit IRC02:53
*** xinliang has quit IRC02:54
*** abhi89 has joined #openstack-nova02:54
*** tjones has joined #openstack-nova02:54
*** crushil has joined #openstack-nova02:55
*** tbachman has joined #openstack-nova02:56
*** fragatina has joined #openstack-nova02:56
*** hongbin has joined #openstack-nova02:57
*** fragatina has quit IRC02:57
*** itlinux has joined #openstack-nova02:57
*** fragatina has joined #openstack-nova02:58
*** diablo_rojo has quit IRC02:59
*** abhishek has joined #openstack-nova03:00
*** thorst has joined #openstack-nova03:00
*** tjones has quit IRC03:03
*** abhi89 has quit IRC03:03
*** xinliang has joined #openstack-nova03:06
*** abhishek has quit IRC03:06
*** abhi89 has joined #openstack-nova03:07
*** ociuhandu has joined #openstack-nova03:08
*** ociuhandu has quit IRC03:10
*** liusheng_ has joined #openstack-nova03:12
*** oomichi has quit IRC03:14
*** ociuhandu has joined #openstack-nova03:15
*** thorst has quit IRC03:18
*** oomichi has joined #openstack-nova03:19
*** itlinux has quit IRC03:24
*** fragatina has quit IRC03:24
*** hongbin has quit IRC03:24
*** Sukhdev has joined #openstack-nova03:27
*** crushil has quit IRC03:31
*** Daisy has quit IRC03:34
*** Daisy has joined #openstack-nova03:34
*** slaweq has joined #openstack-nova03:38
*** Daisy has quit IRC03:39
*** bkopilov_ has joined #openstack-nova03:39
*** dikonoor has joined #openstack-nova03:42
*** bkopilov has joined #openstack-nova03:42
*** links has joined #openstack-nova03:42
*** kfarr has quit IRC03:43
*** slaweq has quit IRC03:43
*** takashin has joined #openstack-nova03:44
*** zsli_ has joined #openstack-nova03:49
*** links has quit IRC03:51
*** Shunli has quit IRC03:52
openstackgerritOpenStack Proposal Bot proposed openstack/nova master: Updated from global requirements  https://review.openstack.org/46528303:53
*** links has joined #openstack-nova03:54
openstackgerritOpenStack Proposal Bot proposed openstack/os-vif master: Updated from global requirements  https://review.openstack.org/45104903:56
*** gouthamr has quit IRC03:57
openstackgerritOpenStack Proposal Bot proposed openstack/python-novaclient master: Updated from global requirements  https://review.openstack.org/46537103:58
*** deepak has quit IRC03:58
*** vks1 has joined #openstack-nova04:01
*** abhi89 has quit IRC04:04
*** xinliang has quit IRC04:04
*** xinliang has joined #openstack-nova04:05
*** suresh12 has joined #openstack-nova04:07
*** gomarivera_ has joined #openstack-nova04:13
*** chyka has quit IRC04:13
*** chyka has joined #openstack-nova04:15
*** thorst has joined #openstack-nova04:15
*** gongysh has quit IRC04:15
*** chyka has quit IRC04:19
*** thorst has quit IRC04:20
*** tbachman has quit IRC04:23
*** moshele has quit IRC04:24
*** links has quit IRC04:24
*** dikonoor has quit IRC04:32
*** adisky_ has joined #openstack-nova04:32
*** harlowja has joined #openstack-nova04:33
*** vancsics has quit IRC04:36
*** ayogi has joined #openstack-nova04:37
*** dikonoor has joined #openstack-nova04:50
*** Daisy has joined #openstack-nova04:52
*** gongysh has joined #openstack-nova04:53
*** prateek has joined #openstack-nova04:54
*** Daisy_ has joined #openstack-nova04:56
*** Daisy has quit IRC04:56
*** ociuhandu has quit IRC05:00
*** links has joined #openstack-nova05:06
*** suresh12 has quit IRC05:11
openstackgerritGhanshyam Mann proposed openstack/nova master: Add policy description for 'os-hide-server-addresses'  https://review.openstack.org/46172805:16
*** thorst has joined #openstack-nova05:16
*** abhi89 has joined #openstack-nova05:17
*** trinaths has joined #openstack-nova05:19
*** abhishek has joined #openstack-nova05:20
*** gjayavelu has joined #openstack-nova05:20
openstackgerritjichenjc proposed openstack/nova-specs master: WIP: add z/VM Driver  https://review.openstack.org/46491505:21
*** thorst has quit IRC05:21
*** abhishek has quit IRC05:22
*** Sukhdev has quit IRC05:23
*** abhi89 has quit IRC05:23
*** dikonoor has quit IRC05:26
*** yamahata has joined #openstack-nova05:28
*** sree has joined #openstack-nova05:29
*** ratailor has joined #openstack-nova05:29
openstackgerritLuo Gangyi proposed openstack/nova master: convert unicode to string before we connect to rados  https://review.openstack.org/46540405:32
*** salv-orlando has joined #openstack-nova05:32
*** armax has quit IRC05:40
*** dikonoor has joined #openstack-nova05:43
*** litao__ has joined #openstack-nova05:47
*** pcaruana has joined #openstack-nova05:48
openstackgerritSongming Yan proposed openstack/nova-specs master: Sriov-bond  https://review.openstack.org/46352605:49
openstackgerritSongming Yan proposed openstack/nova-specs master: Sriov-bond  https://review.openstack.org/46352605:49
*** sridharg has joined #openstack-nova05:53
*** harlowja has quit IRC05:54
*** ralonsoh has joined #openstack-nova05:58
*** fragatina has joined #openstack-nova05:58
*** liusheng_ has quit IRC06:12
*** liusheng_ has joined #openstack-nova06:13
*** Oku_OS-away is now known as Oku_OS06:13
*** fragatina has quit IRC06:15
*** thorst has joined #openstack-nova06:17
*** Daisy_ has quit IRC06:17
*** cNileshc has joined #openstack-nova06:17
*** Daisy has joined #openstack-nova06:18
*** rcernin has joined #openstack-nova06:18
*** cNilesh has quit IRC06:19
*** Daisy has quit IRC06:22
*** thorst has quit IRC06:22
*** markus_z has joined #openstack-nova06:22
*** Daisy has joined #openstack-nova06:28
*** gongysh has quit IRC06:31
*** Daisy has quit IRC06:32
*** liusheng_ has quit IRC06:34
*** Daisy has joined #openstack-nova06:35
openstackgerritWang Shilong proposed openstack/nova master: Lustre support  https://review.openstack.org/44628806:36
*** gongysh has joined #openstack-nova06:43
*** satyar has joined #openstack-nova06:48
*** belmoreira has joined #openstack-nova06:51
*** mgagne has quit IRC06:53
*** mgagne has joined #openstack-nova06:56
*** mgagne is now known as Guest2410306:56
*** abalutoiu has quit IRC06:57
alex_xujohnthetubaguy: gmann , I'm at bug smash, and there is event at evening, not sure I can back to the hotel in time. If i'm late on the api meeting, would you mind to run the api meeting first?06:57
*** rcernin has quit IRC06:57
openstackgerritTakashi NATSUME proposed openstack/nova master: api-ref: Example verification for servers-actions.inc  https://review.openstack.org/45456507:00
openstackgerritZhenyu Zheng proposed openstack/nova master: Catch exception.OverQuota when create image for volume backed instance  https://review.openstack.org/46454807:03
openstackgerritGhanshyam Mann proposed openstack/nova master: Add policy description for 'limits'  https://review.openstack.org/46542607:03
*** salv-orlando has quit IRC07:09
*** ltomasbo|away is now known as ltomasbo07:10
*** mdnadeem has joined #openstack-nova07:11
*** suresh12 has joined #openstack-nova07:11
*** hieulq_ has joined #openstack-nova07:12
*** hieulq_ has quit IRC07:13
*** rcernin has joined #openstack-nova07:13
*** tetsuro has joined #openstack-nova07:13
*** danpawlik_absent is now known as dpawlik07:14
*** tetsuro has quit IRC07:14
*** karimb has joined #openstack-nova07:15
*** suresh12 has quit IRC07:16
*** thorst has joined #openstack-nova07:18
openstackgerritGhanshyam Mann proposed openstack/nova master: Add policy description for multinic  https://review.openstack.org/46543307:19
*** thorst has quit IRC07:22
*** sridharg has quit IRC07:23
*** cfriesen has quit IRC07:26
openstackgerritfengzhr proposed openstack/nova master: Modify the description of flat_injected in nova.conf  https://review.openstack.org/45878907:26
*** mriedem has quit IRC07:27
*** burt has quit IRC07:27
*** burt has joined #openstack-nova07:28
*** rcernin has quit IRC07:29
*** sridharg has joined #openstack-nova07:30
*** prateek is now known as prateek|lunch07:32
*** rcernin has joined #openstack-nova07:35
*** rcernin has quit IRC07:35
*** rcernin has joined #openstack-nova07:35
*** jpena|off is now known as jpena07:36
openstackgerritGhanshyam Mann proposed openstack/nova master: Add policy description for server_usage  https://review.openstack.org/46543707:42
*** hshiina has joined #openstack-nova07:48
*** jaosorior_away is now known as jaosorior07:48
*** Daisy has quit IRC07:50
*** Daisy has joined #openstack-nova07:50
*** prateek|lunch is now known as prateek07:53
*** lpetrut has joined #openstack-nova07:54
*** Daisy has quit IRC07:54
*** gongysh has quit IRC07:56
*** damien_r has joined #openstack-nova07:58
*** damien_r has quit IRC07:58
openstackgerritLuo Gangyi proposed openstack/nova master: convert unicode to string before we connect to rados  https://review.openstack.org/46540407:59
*** zzzeek has quit IRC08:00
openstackgerritGhanshyam Mann proposed openstack/nova master: Add policy description for networks_associate  https://review.openstack.org/46544308:00
openstackgerritJay Pipes proposed openstack/nova master: placement: test for agg association not sharing  https://review.openstack.org/46190108:03
*** sridharg has quit IRC08:03
*** zzzeek has joined #openstack-nova08:04
*** djohnsto has joined #openstack-nova08:04
*** mlakat has joined #openstack-nova08:04
*** dimtruck is now known as zz_dimtruck08:08
*** salv-orlando has joined #openstack-nova08:10
*** Daisy has joined #openstack-nova08:10
*** gszasz has joined #openstack-nova08:12
*** lucas-afk is now known as lucasagomes08:13
*** salv-orlando has quit IRC08:14
johnthetubaguyalex_xu: no worries, I will try be around for the meeting08:14
alex_xujohnthetubaguy: thanks!08:15
*** karimb has quit IRC08:16
openstackgerritGhanshyam Mann proposed openstack/nova master: Add policy description for os-networks  https://review.openstack.org/46545208:17
*** trinaths has quit IRC08:18
*** thorst has joined #openstack-nova08:18
*** yamahata has quit IRC08:20
*** thorst has quit IRC08:23
*** dikonoo has joined #openstack-nova08:24
*** dikonoor has quit IRC08:24
*** Drankis has joined #openstack-nova08:27
*** gszasz has quit IRC08:27
*** udesale has joined #openstack-nova08:27
*** derekh has joined #openstack-nova08:30
openstackgerritGhanshyam Mann proposed openstack/nova master: Add policy description for os-host  https://review.openstack.org/46545808:31
gmannalex_xu: sure, i will also be there08:31
alex_xugmann: thanks a lot08:31
openstackgerritGhanshyam Mann proposed openstack/nova master: Add policy description for os-fixed-ips  https://review.openstack.org/46172408:32
openstackgerritGhanshyam Mann proposed openstack/nova master: Add policy description for 'os-hide-server-addresses'  https://review.openstack.org/46172808:34
*** gongysh has joined #openstack-nova08:34
gmannalex_xu: johnthetubaguy ^^ with this policy doc are finished :  https://review.openstack.org/#/q/topic:bp/policy-docs+status:open+project:+openstack/nova08:35
johnthetubaguygmann: superstar, I should take a look at those soon08:35
gmannjohnthetubaguy: thanks08:36
*** trinaths has joined #openstack-nova08:36
alex_xugmann: yea, superstar :)08:38
openstackgerritArtom Lifshitz proposed openstack/nova master: Use VIR_DOMAIN_BLOCK_REBASE_COPY_DEV when rebasing  https://review.openstack.org/46520508:39
*** karimb has joined #openstack-nova08:39
*** gjayavelu has quit IRC08:39
artomkashyap, mind taking a look at ^^ when you have a moment? You offered help with the downstream bug )08:40
artom;)08:40
*** trinaths has left #openstack-nova08:49
*** aarefiev_afk is now known as aarefiev08:51
*** karimb has quit IRC09:00
*** salv-orlando has joined #openstack-nova09:00
*** takashin has quit IRC09:02
openstackgerritZhenyu Zheng proposed openstack/nova master: Support tag instances when boot(3/4)  https://review.openstack.org/45959309:03
*** gszasz has joined #openstack-nova09:03
*** Daisy has quit IRC09:04
*** Daisy has joined #openstack-nova09:04
*** ralonsoh has quit IRC09:04
*** ralonsoh_ has joined #openstack-nova09:04
*** salv-orlando has quit IRC09:06
*** satyar has quit IRC09:06
*** karimb has joined #openstack-nova09:07
openstackgerritZhenyu Zheng proposed openstack/nova master: Catch exception.OverQuota when create image for volume backed instance  https://review.openstack.org/46454809:09
*** djohnsto has quit IRC09:11
*** gongysh has quit IRC09:14
*** gongysh has joined #openstack-nova09:15
*** thorst has joined #openstack-nova09:19
openstackgerritTao Li proposed openstack/nova master: Fix the display of updated_at time when using memcache driver.  https://review.openstack.org/41446309:23
*** gcb has quit IRC09:27
*** karimb has quit IRC09:27
*** zsli_ has quit IRC09:30
*** karimb has joined #openstack-nova09:33
*** markus_z has quit IRC09:33
*** yingjun has quit IRC09:38
*** thorst has quit IRC09:39
openstackgerritBéla Vancsics proposed openstack/nova master: [WIP] Transform keypair.delete notification  https://review.openstack.org/46394609:40
openstackgerritFerenc Horváth proposed openstack/nova master: Add AggregateMultiTenancyIsolation filter tests  https://review.openstack.org/46400209:41
openstackgerritAlex Szarka proposed openstack/nova master: Transform instance.resize.error notifications  https://review.openstack.org/41788209:47
*** links has quit IRC09:48
*** Daisy has quit IRC09:49
*** markus_z has joined #openstack-nova09:50
*** tbachman has joined #openstack-nova09:53
*** tovin07_ has quit IRC09:56
*** slaweq has joined #openstack-nova09:56
*** yamamoto has quit IRC09:56
*** markus_z has quit IRC09:57
*** yingjun has joined #openstack-nova09:58
*** links has joined #openstack-nova10:00
openstackgerritjichenjc proposed openstack/nova-specs master: WIP: add z/VM Driver  https://review.openstack.org/46491510:01
*** GnomeKing has quit IRC10:01
*** slaweq has quit IRC10:01
*** johnthetubaguy has left #openstack-nova10:06
*** cNilesh has joined #openstack-nova10:06
*** cNileshc has quit IRC10:09
*** salv-orlando has joined #openstack-nova10:10
*** suresh12 has joined #openstack-nova10:12
*** dikonoo has quit IRC10:12
*** faizy has joined #openstack-nova10:14
*** salv-orlando has quit IRC10:15
*** Daviey_ is now known as Daviey10:15
*** markus_z has joined #openstack-nova10:16
*** suresh12 has quit IRC10:16
*** cNilesh has quit IRC10:19
*** sdague has joined #openstack-nova10:29
*** hferenc has quit IRC10:32
*** slaweq has joined #openstack-nova10:32
*** thorst has joined #openstack-nova10:35
*** slaweq has quit IRC10:37
*** bkopilov_ has quit IRC10:39
*** bkopilov has quit IRC10:39
*** thorst has quit IRC10:40
*** yingjun has quit IRC10:43
*** yamamoto has joined #openstack-nova10:45
*** moshele has joined #openstack-nova10:46
openstackgerritSandeep Pawar proposed openstack/nova master: Libvirt volume driver for Veritas HyperScale  https://review.openstack.org/44395110:52
*** ayogi has quit IRC10:52
-openstackstatus- NOTICE: gerrit is being restarted to help stuck git replication issues10:54
*** nicolasbock has joined #openstack-nova10:54
openstackgerritStephen Finucane proposed openstack/python-novaclient master: Remove the `--volume-service-name` option  https://review.openstack.org/46550211:01
*** links has quit IRC11:05
*** gszasz has quit IRC11:05
*** gomarivera_ has quit IRC11:05
openstackgerritjichenjc proposed openstack/nova master: WIP: remove flavor policy  https://review.openstack.org/46550511:07
*** salv-orlando has joined #openstack-nova11:11
*** smatzek has joined #openstack-nova11:12
*** hferenc has joined #openstack-nova11:15
*** salv-orlando has quit IRC11:16
*** gongysh has quit IRC11:18
*** gongysh has joined #openstack-nova11:18
*** rfolco has joined #openstack-nova11:19
*** gongysh has quit IRC11:19
*** cdent has joined #openstack-nova11:19
*** gongysh has joined #openstack-nova11:19
openstackgerritStephen Finucane proposed openstack/python-novaclient master: Remove --cell_name, --tenant options  https://review.openstack.org/46550911:19
*** gongysh has quit IRC11:19
openstackgerritjichenjc proposed openstack/nova master: WIP: Remove os_compute_api:os-server-groups policy  https://review.openstack.org/46551011:19
*** gongysh has joined #openstack-nova11:20
*** gongysh has quit IRC11:20
*** links has joined #openstack-nova11:22
*** gianpietro has joined #openstack-nova11:22
openstackgerritjichenjc proposed openstack/nova master: WIP: remove flavor policy  https://review.openstack.org/46550511:23
*** gianpietro has quit IRC11:27
*** TTris has joined #openstack-nova11:31
*** sean-k-mooney has quit IRC11:32
*** Kevin_Zheng has quit IRC11:36
*** zul has quit IRC11:39
openstackgerritAlex Szarka proposed openstack/nova master: [WIP] Transform instance.exists notification  https://review.openstack.org/40366011:40
*** zul has joined #openstack-nova11:41
*** karimb has quit IRC11:44
*** abalutoiu has joined #openstack-nova11:44
*** Jack_Iv has joined #openstack-nova11:46
*** thorst has joined #openstack-nova11:46
openstackgerritAndrey Volkov proposed openstack/nova master: PoC: Select PCI devices with distinct tag values  https://review.openstack.org/44800811:48
*** gongysh has joined #openstack-nova11:49
*** litao__ has quit IRC11:50
*** gianpietro has joined #openstack-nova11:50
*** udesale has quit IRC11:52
*** yolkfull has quit IRC11:52
openstackgerritAlex Szarka proposed openstack/nova master: Add method for verify multiple notifications  https://review.openstack.org/46552611:53
*** tbachman has quit IRC11:54
*** gianpietro has quit IRC11:56
openstackgerritAlex Szarka proposed openstack/nova master: Add method for verify multiple notifications  https://review.openstack.org/46552611:57
*** claudiub has joined #openstack-nova11:57
*** gongysh has quit IRC11:58
*** lucasagomes is now known as lucas-hungry11:59
*** Alex______ has joined #openstack-nova11:59
*** sean-k-mooney has joined #openstack-nova12:02
*** ratailor has quit IRC12:03
openstackgerritjichenjc proposed openstack/nova master: WIP: remove flavor policy  https://review.openstack.org/46550512:07
*** Daisy has joined #openstack-nova12:07
*** gianpietro has joined #openstack-nova12:10
*** ericyoung has quit IRC12:11
*** Daisy has quit IRC12:12
*** vks1 has quit IRC12:15
*** claudiub has quit IRC12:15
*** jwcroppe has quit IRC12:16
*** jwcroppe has joined #openstack-nova12:16
*** mriedem has joined #openstack-nova12:17
*** mriedem has quit IRC12:18
*** mriedem has joined #openstack-nova12:18
*** liverpooler has joined #openstack-nova12:19
*** jwcroppe has quit IRC12:20
*** gszasz has joined #openstack-nova12:21
openstackgerritAlex Szarka proposed openstack/nova master: [WIP] Transform instance.exists notification  https://review.openstack.org/40366012:21
openstackgerritAlex Szarka proposed openstack/nova master: [WIP] Transform instance.exists notification  https://review.openstack.org/40366012:22
*** gianpietro has quit IRC12:31
*** karimb has joined #openstack-nova12:31
*** gianpietro has joined #openstack-nova12:32
*** ayogi has joined #openstack-nova12:33
*** Miouge has joined #openstack-nova12:34
*** zz_dimtruck is now known as dimtruck12:34
MiougeIs there an extra spec (flavor or image) to configure <cpu mode='host-passthrough'> with libvirt? I know of cpu_mode in nova.conf but that’s for a whole compute node12:35
*** gianpietro has quit IRC12:36
openstackgerritAlex Szarka proposed openstack/nova master: Add method for verify multiple notifications  https://review.openstack.org/46552612:37
openstackgerritAlex Szarka proposed openstack/nova master: [WIP] Transform instance.exists notification  https://review.openstack.org/40366012:38
*** gouthamr has joined #openstack-nova12:39
*** salv-orlando has joined #openstack-nova12:39
sfinucanMiouge: not that I'm aware of, and it's not something we'd be eager to enable12:39
artommriedem, I thought it was OK to continue using mox when it's already in place?12:40
artomI essentially copy-pasted an existing test and tweaked it a bit12:40
*** salv-orlando has quit IRC12:40
*** edmondsw has joined #openstack-nova12:41
*** salv-orlando has joined #openstack-nova12:41
Miougesfinucan: curious to why not something eager to enable, recommended to use cpu_nova.conf and a separate host aggregate instead?12:42
mriedemartom: not for new tests12:42
artommriedem, anyways, I think we had this discussion previously, and it was only dansmith who persuaded you to let it go, so I'll just fix my test, shall I12:42
artom;)12:42
mriedemartom: new tests should be written in mock12:42
mriedemartom: using mox to tweak existing tests that are using mox is acceptable12:42
*** mdnadeem has quit IRC12:43
sfinucanMiouge: The comments in this spec should explain the position :) https://review.openstack.org/#/c/387295/12:43
sfinucan...and host aggregates would be preferred, yes12:44
artommriedem, fair enough. If I ever find the discussion I'm thinking of I'll wave it in your face, but yeah, I won't be lazy and do the new test with mock ;)12:44
mriedemartom: i remember it, it was some nova-network live migration thing12:45
*** bhagyashris has quit IRC12:45
mriedemand i believe dansmith went back afterward and re-wrote the tests in mock12:45
*** tbachman has joined #openstack-nova12:45
*** kylek3h has quit IRC12:46
mriedemartom: see #4 here https://docs.openstack.org/infra/manual/developers.html#peer-review12:46
*** dane-fichter has joined #openstack-nova12:46
*** jpena is now known as jpena|lunch12:47
*** sree has quit IRC12:48
*** sree has joined #openstack-nova12:49
*** hshiina has quit IRC12:50
dane-fichterhey dansmith, could you take a look at my trusted certs patch when you get a chance?12:51
dane-fichterhttps://review.openstack.org/#/c/457711/12:51
dane-fichterIt's passing unit tests, not sure if there's anything else I need to implement here12:52
sdaguedansmith / mriedem: thinking about the thing that came up in the cells forum session around searching by ip ....12:53
sdagueit's really pathologically bad in our code12:53
*** sree has quit IRC12:54
*** sree has joined #openstack-nova12:54
mriedemi looked it up after the session,12:54
mriedemand it wasn't mysql or pg specific12:54
sdaguenope12:54
mriedemit was originally just about a security bug12:54
sdaguemriedem: what was the security bug?12:54
mriedemapi would call down to nova-network which would pull all instances from the db and iterate them filtering on ip, something like that12:55
sdagueok, so removing nova-network out of the flow, we're now just loading every instance, loading every network info cache, and grepping through each json?12:56
sdagueespecially in the all_tenants case12:56
*** lucas-hungry is now known as lucasagomes12:57
mriedemhttps://github.com/openstack/nova/commit/ae781ee97947c33d6d43e4c21df4f338c875bf1c was the original change12:57
*** lyan has joined #openstack-nova12:58
*** liangy has joined #openstack-nova12:58
*** lyan has quit IRC12:58
mriedemsdague: not sure how you'd do the filtering in the db when the instance_info_cache.network_data is serialized json12:58
*** xyang1 has joined #openstack-nova12:58
mriedem*network_info12:59
sdagueright, in order to make it efficient we'd need to extract the search keys that we wanted to hit into a column12:59
*** mdnadeem has joined #openstack-nova12:59
sdaguewhich you could do, just denormalize it a bit12:59
*** ayogi has quit IRC13:00
*** esberglu has joined #openstack-nova13:00
sdaguegiven that the ip search is regex, you could even just make it a comma separated list, and still do the second stage filter, but it would drop a lot of db bloat13:00
sdagueanyway, just pondering that, given it seemed like it was a query many wanted to do, and it's one of our most pathologically bad ones13:01
mriedemand it's also been this way forever..13:01
mriedemif we had numbers on how long it takes to do the filter on an all_tenants call with 1000 instances or something, then it might be more interesting13:02
*** gomarivera_ has joined #openstack-nova13:02
sdaguemriedem: yeh data would be good13:02
*** eharney has joined #openstack-nova13:03
sdagueit does also mean though if you exceed 1000 instances in your filter set before ips, you'll get the wrong answer, right?13:03
*** bhagyashris has joined #openstack-nova13:03
sdaguethat's actually the thing I'm more concerned about13:03
*** johnthetubaguy has joined #openstack-nova13:03
*** salv-orlando has quit IRC13:03
mriedemtrue, if the ip you're looking for is the 1001 instance, you're not going to get anything back13:03
*** salv-orlando has joined #openstack-nova13:04
sdagueif nothing else, we should describe that issue13:04
mriedemyeah, definitely,13:05
mriedemcould go into the api-ref13:05
mriedemi feel like there was something else we said we'd doc in the api-ref but i'm forgetting - i think notes are in etherpads though, and i haven't digested those yet13:05
*** tuan_luong has joined #openstack-nova13:06
sdaguewell, in mysql we could - https://dev.mysql.com/doc/refman/5.7/en/json-search-functions.html#function_json-contains13:06
mriedemip (Optional) query string An IPv4 address to filter results by.13:06
mriedem^ is the api-ref and doesn't mention anything about this13:06
sdaguemriedem: yep, I know13:06
sdagueI honestly didn't realize how bad it was13:06
*** rabel has joined #openstack-nova13:07
*** gomarivera_ has quit IRC13:07
mriedemlooks like postgresql does too https://www.postgresql.org/docs/9.3/static/functions-json.html13:07
mriedemhttp://schinckel.net/2014/05/25/querying-json-in-postgres/13:08
*** salv-orlando has quit IRC13:08
rabelHi there, I'm a little bit confused about quotas in nova. To start with: Can anyone tell me, what a quota class is? And how are those related to default quotas? How do default quotas work in detail? For example why does a project have default quotas? I thought default quotas are global quotas that take effect for projects without quotas?13:09
*** artom has quit IRC13:11
mriedemrabel: se13:12
mriedem*sec13:12
*** Jack_Iv has quit IRC13:12
mriedemrabel: some notes here https://docs.openstack.org/developer/nova/quotas.html13:12
mriedemrabel: there is only one quota class and it's "default"13:12
*** Jack_Iv has joined #openstack-nova13:12
*** Jack_Iv has quit IRC13:13
*** Jack_Iv has joined #openstack-nova13:13
mriedemsome more info here https://review.openstack.org/#/c/411035/13:13
*** Jack_Iv has quit IRC13:13
rabelmriedem: thank you very much! do you know if other openstack components differ in this regard?13:14
*** gianpietro has joined #openstack-nova13:14
*** mdrabe has joined #openstack-nova13:14
*** kfarr has joined #openstack-nova13:15
*** pchavva has joined #openstack-nova13:15
*** mkoderer has joined #openstack-nova13:17
*** marst has quit IRC13:18
mkoderermriedem: ping13:18
*** marst has joined #openstack-nova13:18
bhagyashrisHi all, Can anyone please guide me How to check the memory consumption or memory during the process initialization suppose for n-cpu service? Is there any command to check this?13:18
*** kylek3h has joined #openstack-nova13:19
*** jamesdenton has joined #openstack-nova13:20
rabelmriedem: what is still confusing to me is the --default option of "openstack quota show", which shows default quotas for a given project or class. that does not seem to make a lot of sense. what do you think?13:22
*** vladikr has joined #openstack-nova13:22
*** baoli has joined #openstack-nova13:23
*** cleong has joined #openstack-nova13:30
mriedemrabel: i think that's trying to match the nova quota-defaults CLI13:30
*** sree has quit IRC13:31
mriedemmkoderer: pong13:31
mkoderermriedem: dgonzalez and I wanted to discuss a tricky bug with you: https://bugs.launchpad.net/nova/+bug/169113113:32
openstackLaunchpad bug 1691131 in OpenStack Compute (nova) "IO stuck causes nova compute agent outage" [Undecided,New]13:32
mkoderernova-compute stops sending it's heartbeats if we have a NFS overload13:32
mkodererand with this we can't control the cloud anylonger since stopping of vms is not possible anylonger13:33
mriedemmkoderer: which release?13:33
mkodererwe are on liberty - but the code is the same on master13:33
mriedem"The nova-compute services scans over all attached volume paths"13:34
mriedemwhich part specifically are you referring to there?13:34
rabelmriedem: so, let me summarize: the only quota class is "default" and the concept of quota classes will probably soon be removed. default quotas are quotas that take effect if there is no project-specific quotas. and having default quotas for a project/class does not make sense. is this correct in your opinion?13:34
mriedem"and having default quotas for a project/class does not make sense" - why does that not make sense?13:35
mriedemrabel: having default quota (limits) is fine13:35
mriedemhaving an api and database for quota classes when there is only ever a single entry though, is weird13:36
mriedem*database table13:36
*** thorst is now known as thorst_afk13:36
mkoderermriedem: this part blocks: https://github.com/openstack/nova/blob/902b7bf6f5425824a0661b8e4beac4a894749c03/nova/virt/libvirt/driver.py#L702213:36
*** johnthetubaguy has left #openstack-nova13:37
dansmithsdague: mriedem: setting the precedent of loading json in mysql seems not awesome13:37
rabelmriedem: default quotas do make sense. but how are they connected to a specific project? isn't the idea of default quotas that they are global?13:37
dansmithsdague: mriedem: I'd almost rather just build a separate index of that thing, if it's a thing people do often13:37
dansmithsdague: mriedem: also, it's something rather easy that a client can do right? find the port/device_id in neutron and then show the instance in nova?13:38
*** markus_z has quit IRC13:39
*** cleong has quit IRC13:41
bauzasdansmith: edleafe: tbc, I'm working on updating my series given dan's comments13:41
*** mdnadeem has quit IRC13:41
dansmithbauzas: sweet :)13:42
bauzasjust has a problem with a 2-instances request13:42
bauzasbecause I need more hosts than just 413:42
edleafebauzas: cool. I'll be pushing an update to my series, but it's just a pep8 fix, so don't freak out13:42
sdaguedansmith: expect, neutron api isn't always exposed correctly directly to users13:42
*** zzzeek has quit IRC13:43
*** diablo_rojo has joined #openstack-nova13:43
sdaguedansmith: yeh, secondary index would be fine as well13:43
bauzasedleafe: which one do you think you want to update ? the ones about passing the UUIDs and the scheduler claims?13:43
*** jpena|lunch is now known as jpena13:44
edleafebauzas: yep13:44
bauzasok13:44
dansmithsdague: could we also just special case that search by asking neutron ourselves and then pulling the relevant instances out ?13:45
openstackgerritOpenStack Proposal Bot proposed openstack/nova master: Updated from global requirements  https://review.openstack.org/46528313:45
sdaguedansmith: they support search by regex?13:45
sdagueI think it gets complicated when there are multiple filters13:46
sdaguebut especially what we allow in our filters today13:46
*** johnthetubaguy has joined #openstack-nova13:46
dansmithsdague: I dunno, they'd have to for the client approach to work, but I didn't realize that searching by regex was a thing, I thought the use-case was "find me the instance with this ip"13:46
sdagueit's definitely a regex today, which is kind of suck13:47
sdaguebut that's what it is13:47
sdagueand has been13:47
*** dimtruck is now known as zz_dimtruck13:47
*** tbachman has quit IRC13:47
mriedemmkoderer: that's in a ploop block, are you using ploop?13:48
*** mlavalle has joined #openstack-nova13:48
*** zzzeek has joined #openstack-nova13:49
mriedemmkoderer: is there a way to detect a bad mountpoint with nfs?13:49
*** READ10 has joined #openstack-nova13:50
*** lucasagomes is now known as lucas-brb13:52
dgonzalezmriedem: mkoderer sent the wrong line, the correct line is https://github.com/openstack/nova/blob/master/nova/virt/libvirt/driver.py#L7025 (below the loop block)13:53
openstackgerritBéla Vancsics proposed openstack/nova master: Transform keypair.delete notification  https://review.openstack.org/46394613:53
*** tbachman has joined #openstack-nova13:53
dgonzalezmriedem: But both lines basically have the same content, so I guess the issue could occur in both blocks13:53
*** kfarr has quit IRC13:54
mriedemi'm going to be away for a couple of hours13:54
*** mriedem is now known as mriedem_afk13:54
*** baoli_ has joined #openstack-nova13:55
*** Drankis has quit IRC13:55
*** baoli has quit IRC13:57
alex_xujohnthetubaguy: sdague it would be great to merge this one first when you have free time, it can reduce some conflict in later patch. thanks in advance13:58
sdaguealex_xu: which patch?13:59
alex_xusdague: oops, https://review.openstack.org/46282513:59
*** crushil has joined #openstack-nova13:59
*** salv-orlando has joined #openstack-nova14:00
sdaguealex_xu: done14:00
alex_xusdague: thanks!14:01
johnthetubaguywill take a look at that series next, looking a the policy docs first14:02
johnthetubaguyI also need to work out why that highlight didn't make any noise in this (franctionally less stupid) IRC client14:02
*** rabel has quit IRC14:03
*** jwcroppe has joined #openstack-nova14:04
*** p01nt3r75 has joined #openstack-nova14:04
*** djohnsto has joined #openstack-nova14:05
*** vks1 has joined #openstack-nova14:06
*** rabel has joined #openstack-nova14:07
*** coreywright has quit IRC14:07
*** zz_dimtruck is now known as dimtruck14:08
rabelmriedem: I was shortly disconnected. have you by chance answered my last question?14:10
*** awaugama has joined #openstack-nova14:12
*** suresh12 has joined #openstack-nova14:12
*** itlinux has joined #openstack-nova14:12
*** tbachman has quit IRC14:14
*** suresh12 has quit IRC14:16
dimssdague : Do we need to suppress this ERROR traceback? http://logs.openstack.org/58/465458/1/check/gate-tempest-dsvm-neutron-full-ubuntu-xenial/d240f9b/logs/screen-n-sch.txt.gz#_May_17_08_54_38_99695814:19
*** coreywright has joined #openstack-nova14:20
sdaguedims: yeh, probably14:22
*** adisky_ has quit IRC14:22
sdaguewe should support at least a 60 second grace before we emit that14:22
sdagueit's a service race startup14:22
*** marst has quit IRC14:23
*** tbachman has joined #openstack-nova14:24
*** tjones has joined #openstack-nova14:24
*** karimb has quit IRC14:25
*** trinaths has joined #openstack-nova14:26
*** tuan_luong has quit IRC14:26
*** gszasz has quit IRC14:27
*** moshele has quit IRC14:28
*** jwcroppe has quit IRC14:28
*** udesale has joined #openstack-nova14:29
*** jwcroppe has joined #openstack-nova14:29
*** karimb has joined #openstack-nova14:29
*** marst has joined #openstack-nova14:30
*** gjayavelu has joined #openstack-nova14:31
*** Guest24103 is now known as mgagne14:32
*** mgagne has quit IRC14:32
*** mgagne has joined #openstack-nova14:32
*** yamamoto_ has joined #openstack-nova14:32
*** jwcroppe_ has joined #openstack-nova14:32
*** gyee has joined #openstack-nova14:33
*** lucas-brb is now known as lucasagomes14:33
*** jwcroppe has quit IRC14:34
*** yamamoto has quit IRC14:36
*** dane-fichter has quit IRC14:36
edleafeThis isn't the best forum for discussion, but I wanted some clarification about the request from ops for additional host for retries. Was this simply for Nova? I remember talking about getting rid of the dependence on max_attempts in the scheduler, because it was a nova-ism.14:37
edleafeBasing the number of returned hosts on max_attempts seems like we're once again linking nova to the scheduler. IOW, if max_attempts is 5, then *every* service that might ever use the scheduler will get back 5 resource providers per request, whether they need them or not. Wouldn't it be a better design to have the service calling select_destinations ask for more if they want more than 1?14:37
edleafe^^ bauzas dansmith cdent jaypipes14:38
*** dave-mccowan has joined #openstack-nova14:38
bauzasedleafe: because the conductor would be cell-local14:38
*** moshele has joined #openstack-nova14:39
bauzasedleafe: so it would be an upcall which we disagree with14:39
*** slaweq has joined #openstack-nova14:39
edleafebauzas: you're missing the point14:39
edleafeI'm not saying returning more than one host is a bad thing14:39
*** itlinux has quit IRC14:39
edleafeI'm asking if we want to tie the number of returned RPs to nova's needs14:39
bauzasno14:39
bauzasthe RP call returns the full list14:40
cdentedleafe: when you say "returned RPs" do you mean placement -> scheduler or scheduler -> something else14:40
*** armax has joined #openstack-nova14:40
bauzasthen the filters/weighters filters out that list14:40
edleafecdent: returned from scheduler14:40
cdent14:40
edleafecdent: I still try to think of scheduler as independent from nova14:41
edleafe(as a design goal)14:41
*** artom has joined #openstack-nova14:41
cdentseems a reasonable practice, although I've always assumed that while placement would always be independent, it would have as clients lots of different "things that do filtering" (including nova-scheduler)14:41
edleafecdent: oh definitely for placement, since we got to create it separately14:42
*** prateek has quit IRC14:42
bauzasedleafe: no, we clearly said during Forum that while placement could be eventually independent, scheduler is a nova-ism14:42
edleafeI'm wondering if we're giving up on the notion of the scheduler ever being useful for anything besides nova14:42
bauzasedleafe: at least jay told that in his presentations14:43
cdentI think maintaining a strong contractual boundary is always good14:43
edleafeso it was decided that every service that needs scheduling will create its own?14:43
sean-k-mooneysfinucan: can you look at https://review.openstack.org/#/c/451049/9 the os-vif requirements file on master has conflicting with the global requirement for a while so we should probably merge this14:44
edleafeor just that nova doesn't want to waste cycles on making that happen14:44
cdentso I would think that the answer to your original question is "yes, we should be explicit"14:44
sfinucansean-k-mooney: on it14:44
bauzasedleafe: every service could be using Placement differently14:44
edleafebauzas: nobody's talking about placement14:45
bauzasedleafe: and scheduler could be a library by Queens14:45
sean-k-mooneysfinucan: cool thanks, just going true my review queue which is always fun :)14:45
bauzasbut that's still a nova thing14:45
edleafeI'm talking about making a change that ties the scheduler closer to nova14:45
edleafethat we will decide to undo in N cycles because the winds have changed direction14:46
*** vks1 has quit IRC14:47
*** burt has quit IRC14:48
*** burt has joined #openstack-nova14:48
*** slaweq has quit IRC14:48
*** udesale has quit IRC14:50
dansmithedleafe: I think we've been on a path to keep scheduler closely tied to nova and split out placement as the generic thing, ever since placement was a thing14:50
*** udesale has joined #openstack-nova14:50
dansmiththat's been my understanding14:50
*** ociuhandu has joined #openstack-nova14:50
*** vks1 has joined #openstack-nova14:50
edleafedansmith: so it's OK to add more things to tie them together?14:53
dansmithto tie nova and the scheduler together?14:53
dansmithor nova and "nova's scheduler" I should say14:54
*** udesale has quit IRC14:54
edleafeI think you've answered my question14:54
dansmithheh14:54
edleafebleh14:54
openstackgerritSylvain Bauza proposed openstack/nova master: Scheduler returning alternatives per destination  https://review.openstack.org/46517514:54
openstackgerritSylvain Bauza proposed openstack/nova master: Delete existing allocations when retrying  https://review.openstack.org/46043314:54
openstackgerritSylvain Bauza proposed openstack/nova master: WIP: Allocate instance by the conductor  https://review.openstack.org/46045514:54
dansmithmaking the claim in the scheduler is tying them closer together right?14:54
edleafeno14:54
*** cfriesen has joined #openstack-nova14:54
dansmithhmm, okay :)14:55
bauzasdansmith: edleafe: just in case you're interested in, I just updated my series14:55
*** moshele has quit IRC14:55
*** links has quit IRC14:55
dansmithbauzas: cool14:55
bauzasworking now back on the conductor change now that we have the alternatives14:55
edleafeassuming an independent scheduler, if a service requests some resources, the scheduler would make a selection, claim the resources to ensure that nothing else uses them, and then returns the selected RP14:55
edleafeNothing nova-y about it14:56
bauzasedleafe: again, like I said, we plan to make scheduler a library for Queens14:56
bauzasso it would eventually be possible for other services to use that lib14:56
bauzasbut for the moment, consider it as a nova-ism14:56
edleafebauzas: that seems kind of silly if only Nova can use it14:56
bauzasbaby steps at least14:57
bauzasie. the plan is14:57
bauzas1/ claims in Pike14:57
bauzas2/ scheduler as a lib in Queens14:57
edleafebauzas: baby steps forward, while taking full steps back14:57
bauzas3/ conductor uses that lib for filtering/weighting in Queens/Rocky14:57
bauzas4/ other services could possibly be using that scheduler lib by Rocky14:58
*** tbachman has quit IRC14:58
bauzas(c) by dansmith for the library thing14:58
bauzaswhich I totally agree with14:58
edleafebauzas: no they couldn't, if we start tying scheduler to nova more and more14:58
edleafeinstead of getting rid of a nova-ism and making it part of the interface, we're now making scheduler more dependent on that nova-ism14:59
*** salv-orlando has quit IRC14:59
bauzasedleafe: that's a truism, and I don't disagree with your statement14:59
dansmithbauzas: no, it was johnthetubaguy's idea, I just fell in love with it14:59
edleafemaking it even less likely that any other service will ever use scheduler14:59
bauzaswhat I disagree is with you saying that we go into that direction14:59
*** salv-orlando has joined #openstack-nova15:00
bauzasdansmith: oops, hail to johnthetubaguy then :)15:00
edleafedansmith: that contradicts what you were saying earlier: that the scheduler is nova's scheduler15:00
bauzasanyway, back to work for updating the conductor now15:00
edleafewhy do any of those steps if the end goal isn't to make it useful for other services?15:00
dansmithedleafe: I don't want it as a library for other projects to use, I want it to be nothing but code in nova (i.e. not a service)15:01
dansmithedleafe: because placement is the generic thing useful for other services15:01
johnthetubaguyplacement is for everyone right, and +1 what dansmith just said15:01
*** ociuhandu has quit IRC15:01
edleafeok, then let's tie scheduler to nova ever more tightly15:01
cdentlibrary is a bit of a misnomer, I guess? We mean something more like "module"?15:02
dansmithcdent: yeah15:02
edleafecdent: yeah, "library" means generally useful15:02
cdentlibrary would be fun, but seems unlikely. Also, why would we replicate that confusing mass?15:02
johnthetubaguymy think was, stop making RPC API calls to it, since it the process is no longer special, but thats a different thing15:03
edleafecdent: because the concept of choosing one RP over another is rather generic15:03
dansmithedleafe: I think we've been on this "keep scheduler in nova, split out placement" path for quite a while, but I'm not sure the distinction was made explicit and given the "split the scheduler" talk before that, you're probably not the only one confused by the "plan"15:03
cdentthere could be a library for things that filter resource providers that is implemented outside of nova, using more clear techniques15:03
cdentI think that would be useful15:04
dansmithcould be,15:04
edleafecdent: sure, but there was never any work even considered for that15:04
johnthetubaguycdent: yeah, thats a sepearate, quite interesting thing, that is a library15:04
*** rcernin has quit IRC15:04
dansmithand you could also build a generic scheduling library that makes it clear we should dump much of what nova's scheduler does15:04
*** salv-orlando has quit IRC15:04
dansmithand when you guys build that, we can evaluate :)15:04
edleafecdent: whereas we've spent a lot of effort on cleaning up the interface between nova and the scheduler15:04
cdentedleafe: I still think that interface should be kept as clean and explicit as possible. That's simply good practice.15:05
* dansmith nods15:05
edleafecdent: and if the scheduler were an independent library, it could be cleaned up radically, as long as the interface contract was maintained15:05
* johnthetubaguy nods at cdent15:05
edleafecdent: well, that's why I started this conversation.15:05
* cdent nods at everyone15:05
edleafeWe're making the interface dirtier15:06
cdentcan someone point me at this dirtying?15:06
johnthetubaguyedleafe: where?15:06
edleafebasing the number of returned hosts on max_attempts15:06
edleafewhich is a nova-ism that we've discussed removing in the past15:06
bauzasI would certainly appreciate reviews on my changes15:06
edleafeIOW, any other potential user of scheduler would now get back the number of hosts that nova wants15:07
edleafeIMO, it would be cleaner to make the number of hosts to return per request explicit15:07
edleafeand move max_attempts out of the scheduler config15:08
johnthetubaguyedleafe: so the retry configuration needs to be set by the operator where?15:08
edleafejohnthetubaguy: somewhere in nova's config, not in scheduler's config15:08
johnthetubaguyscheduler config is nova config right now, and there is no plan to change that right?15:08
edleafejohnthetubaguy: Unless, of course, we are deciding to keep the two locked together15:09
johnthetubaguyplacement config is a totally different thing15:09
edleafejohnthetubaguy: no plan, correct. It's been pointed out several times as something we need to change15:09
edleafeif we want an independent scheduler15:09
cdentfwiw, edleafe, I agree with you: the config of attempts should be outside of the nova-scheduler's domain, it should accept an arg15:09
*** itlinux has joined #openstack-nova15:09
cdent_even_ if that code never becomes a lib15:10
*** vks1 has quit IRC15:10
*** karimb has quit IRC15:10
bauzascdent: well, technically, we said to return alternatives by the number of retries15:11
johnthetubaguyseems like you want the retry conf set centrally not different on every compute node, thats the bit that matters I guess?15:11
bauzascdent: and atm we have that flag in the scheduler group15:11
bauzasalthough it's settable per compute given it's used by the scheduler.utils module15:12
johnthetubaguybauzas: hmm, joy15:12
edleafejohnthetubaguy: by not passing the number of hosts to return to the scheduler, it doesn't matter what config file it's set in.15:13
bauzasagain, I beg people to review https://review.openstack.org/#/c/465175/ if they have matters15:13
edleafeit's set and unchangeable for any other service15:13
edleafeit seems like since keeping the interface clean is more work, we'll just keep doing things the old way with tying scheduler to nova15:13
edleafewhich dansmith says is what we're going to do15:14
*** salv-orlando has joined #openstack-nova15:14
johnthetubaguyI am all for keeping the interface clean, it just feels like that belongs inside the scheduler to me, but I am kinda just stuck in the old ways I guess15:14
edleafejohnthetubaguy: so you don't ever see any service other than nova ever ever ever using the scheduler?15:15
*** karimb has joined #openstack-nova15:15
johnthetubaguyedleafe: basically no, I could see another seam being added to extract the filter/weigher stuff thats been cut and paste all over the place, and some placement helper utils, but that feels like a set of problems we should ignore for now, so we can make quicker progress15:16
edleafejohnthetubaguy: agree that we shouldn't spend time on it now15:17
dansmithjohnthetubaguy: agree15:17
edleafejohnthetubaguy: disagree that we should actively make things worse15:17
edleafeworse == tighter coupling15:17
johnthetubaguyright now there is random util dust sprinkled all over, thats bad, I need to go review the proposed code15:19
sean-k-mooneydansmith: johnthetubaguy  out of interest do you ever see a case where i would be allowed to pass a set of claime to nova boot? not saying that is desireable but has that come up as a topic before?15:20
dansmithsean-k-mooney: it has15:20
johnthetubaguysean-k-mooney: yeah, came up with the watcher folks I think15:20
johnthetubaguyreservations, etc15:20
*** trinaths has left #openstack-nova15:20
sean-k-mooneycool was any desion reached on that or was it defered to later15:20
dansmithsean-k-mooney: what do you think? :P15:21
sean-k-mooneymuch later haha15:21
dansmithwe can't even agree which color to paint the bike shed15:21
dansmithwell, some of us can, but .. :P15:21
*** trinaths1 has joined #openstack-nova15:21
*** hongbin has joined #openstack-nova15:22
*** salv-orlando has quit IRC15:22
*** felipemonteiro has joined #openstack-nova15:22
sean-k-mooneyok cool as long as we can at least all agree to use the placement api for track resources and allocations of those resources thats better then what we have today15:23
*** tbachman has joined #openstack-nova15:23
sean-k-mooneywell thats perhaps unfair neutron is already using it  for ip pools for the l3 routing stuff so we have made more progress then just agreeing to use the placement api15:24
*** trinaths1 has quit IRC15:24
*** rabel_b1 has joined #openstack-nova15:27
*** rabel has quit IRC15:27
sean-k-mooneydansmith: by the way you summit talk on the cells v2 work was really good. jays talks on the placement apis were also good to watch15:27
dansmithsean-k-mooney: thanks :)15:28
*** armax has quit IRC15:28
*** p01nt3r75 has quit IRC15:29
*** abalutoiu has quit IRC15:30
*** p01nt3r75 has joined #openstack-nova15:31
*** shewless has quit IRC15:31
*** Oku_OS is now known as Oku_OS-away15:33
*** mdrabe has quit IRC15:36
*** Miouge has quit IRC15:37
*** hieulq_ has joined #openstack-nova15:38
*** psachin has joined #openstack-nova15:38
*** Sukhdev has joined #openstack-nova15:40
*** gomarivera_ has joined #openstack-nova15:40
cfriesenkashyap: what would you think of https://review.openstack.org/#/c/461456 ?  The rationale being that it's defined as a realtime mask so if the user specifies "1-2" they should get vCPUs 1 and 2 as realtime.15:43
*** mdrabe has joined #openstack-nova15:44
sfinucancfriesen: I took a look at it. Looks sensible to me. Do we document support image metadata anywhere though?15:45
sfinucan...or is that up to Glance to do?15:45
sfinucandansmith: RE: https://review.openstack.org/#/c/408242/, what would be the preferred pattern -15:47
*** mlakat has quit IRC15:48
sfinucanhardening all the access to various InstanceNUMACell attributes to check if the attribute is set first15:48
johnthetubaguysfinucan: did you see the image meta object stuff?15:48
sfinucanor always setting those attributes anywhere we create an object?15:48
johnthetubaguysfinucan: we should probably generate docs from that at some point15:48
*** sree has joined #openstack-nova15:48
*** chyka has joined #openstack-nova15:48
cfriesensfinucan: not actually sure15:48
sfinucanjohnthetubaguy: To which "stuff" do you refer? :)15:48
*** gjayavelu has quit IRC15:48
johnthetubaguysfinucan: https://github.com/openstack/nova/blob/master/nova/objects/image_meta.py15:49
sfinucanjohnthetubaguy: Oh, yeah - I knew where that was defined, alright. I meant docs that you don't need to read the code for :)15:49
johnthetubaguysfinucan: we should generate some docs off that I think, I suspect a few have got into the admin guides, but unsure on the details there15:50
cfriesensfinucane: glance has this https://docs.openstack.org/developer/glance/common-image-properties.html  but there's not much there15:50
johnthetubaguysfinucan: its better than extra specs though...15:50
cfriesensfinucan: johnthetubaguy: image_meta.py just says "CPU mask indicates which vCPUs will have realtime enable" which doesn't really match the current code (hence my patch)15:51
dansmithsfinucan: what do you mean? those patches do the right thing.. we merged one already right?15:51
openstackgerritDaniel Gonzalez Nothnagel proposed openstack/nova master: WIP: Get disk size of instance in separate process  https://review.openstack.org/46565315:51
sfinucanjohnthetubaguy: I'll go look for stuff now. Admin guide seems like the correct place to put them, even though we're going to move that into nova soon15:52
sfinucandansmith: Aye, but those patches that merged mix the two. https://review.openstack.org/gitweb?p=openstack/nova.git;a=commitdiff;h=99f3bbc1ac7ddcde36931f590b6df3811e596d8c15:52
*** sree_ has joined #openstack-nova15:53
sfinucandansmith: So in some cases, we've done "if ('key' in object) and object.key:"15:53
*** lucasagomes is now known as lucas-afk15:53
*** sree_ is now known as Guest3926315:53
sfinucan...but in others, we just set the various fields to None in the tests, sidestepping the issue15:53
dansmithsfinucan: being defensive in the code that uses those fields is the wy15:54
dansmith*way15:54
sfinucanOK, I'll do that so15:54
dansmithsfinucan: i.e. the "if foo in obj and obj.foo15:54
dansmithsfinucan: cool, thanks for picking those back up15:54
* sfinucan is trying to fix https://review.openstack.org/#/c/408242/15:54
sfinucandansmith: no problemo15:54
*** sree has quit IRC15:55
*** gomarivera_ has quit IRC15:55
*** sree has joined #openstack-nova15:56
*** Apoorva has joined #openstack-nova15:57
*** Guest39263 has quit IRC15:58
*** gouthamr_ has joined #openstack-nova15:58
*** karimb_ has joined #openstack-nova15:58
*** gouthamr has quit IRC15:59
*** nic has joined #openstack-nova16:00
*** karimb has quit IRC16:00
*** nic1 has joined #openstack-nova16:00
*** lpetrut has quit IRC16:01
*** tjones has quit IRC16:03
*** moshele has joined #openstack-nova16:03
*** sree has quit IRC16:03
*** sree has joined #openstack-nova16:04
*** nic has quit IRC16:05
mriedem_afkrabel_b1: you can have global default quotas (limits), project-specific default quotas, and then just project-specific quotas, and i understand that the latter two are confusing, and i'm not sure if/how we treat them differently, unless you have default project limits and user-specific limits within that project16:05
*** mriedem_afk is now known as mriedem16:05
mriedemmkoderer: dgonzalez: so anytime anything hits https://github.com/openstack/nova/blob/master/nova/virt/libvirt/driver.py#L7025 in the compute it locks up the entire process?16:06
mriedemmkoderer: dgonzalez: because that method is called from a few different operations16:06
rabel_b1mriedem: ok, thank you very much.16:08
*** hieulq_ has quit IRC16:09
*** hieulq_ has joined #openstack-nova16:09
mriedemdgonzalez: ah i see your patch now16:09
*** hieulq_ has quit IRC16:10
*** hieulq_ has joined #openstack-nova16:10
*** gyee_ has joined #openstack-nova16:11
*** pcaruana has quit IRC16:11
*** mdnadeem has joined #openstack-nova16:12
*** hieulq__ has joined #openstack-nova16:13
*** ociuhandu has joined #openstack-nova16:16
*** jaosorior is now known as jaosorior_away16:17
*** mdnadeem has quit IRC16:17
*** hieulq__ has quit IRC16:18
*** belmoreira has quit IRC16:20
*** p01nt3r75 has quit IRC16:20
*** gyee has quit IRC16:20
*** ociuhandu has quit IRC16:25
*** gouthamr_ has quit IRC16:28
*** marst has quit IRC16:28
*** aarefiev is now known as aarefiev_afk16:31
*** sree_ has joined #openstack-nova16:32
*** sree_ is now known as Guest1642016:32
*** harlowja has joined #openstack-nova16:32
*** Guest16420 has quit IRC16:33
*** sree has quit IRC16:34
*** ltomasbo is now known as ltomasbo|away16:40
*** rabel_b1 has quit IRC16:41
openstackgerritArtom Lifshitz proposed openstack/nova master: Plumbing for tagged nic attachment  https://review.openstack.org/41562216:43
openstackgerritArtom Lifshitz proposed openstack/nova master: Delete nic metadata when detaching interface  https://review.openstack.org/43538716:44
openstackgerritArtom Lifshitz proposed openstack/nova master: Libvirt support for tagged volume attachment  https://review.openstack.org/41810016:44
openstackgerritArtom Lifshitz proposed openstack/nova master: Delete disk metadata when detaching volume  https://review.openstack.org/43508216:44
openstackgerritArtom Lifshitz proposed openstack/nova master: Plumbing for tagged volume attachment  https://review.openstack.org/39194116:44
openstackgerritArtom Lifshitz proposed openstack/nova master: API support for tagged device attachment  https://review.openstack.org/37602116:44
openstackgerritsean mooney proposed openstack/nova master: do not set vcpu_pin elements when virt_type=qemu  https://review.openstack.org/46516016:44
*** eharney has quit IRC16:44
openstackgerritArtom Lifshitz proposed openstack/nova master: Plumbing for tagged nic attachment  https://review.openstack.org/41562216:47
openstackgerritArtom Lifshitz proposed openstack/nova master: Delete nic metadata when detaching interface  https://review.openstack.org/43538716:47
openstackgerritArtom Lifshitz proposed openstack/nova master: Libvirt support for tagged volume attachment  https://review.openstack.org/41810016:47
openstackgerritArtom Lifshitz proposed openstack/nova master: Delete disk metadata when detaching volume  https://review.openstack.org/43508216:47
openstackgerritArtom Lifshitz proposed openstack/nova master: Plumbing for tagged volume attachment  https://review.openstack.org/39194116:47
openstackgerritArtom Lifshitz proposed openstack/nova master: API support for tagged device attachment  https://review.openstack.org/37602116:47
openstackgerritArtom Lifshitz proposed openstack/nova master: Libvirt support for tagged nic attachment  https://review.openstack.org/41552016:47
*** armax has joined #openstack-nova16:48
*** eharney has joined #openstack-nova16:48
*** gianpietro has quit IRC16:55
*** karimb_ has quit IRC16:59
*** gouthamr has joined #openstack-nova17:02
*** derekh has quit IRC17:02
*** fragatina has joined #openstack-nova17:02
*** fragatina has quit IRC17:02
*** fragatina has joined #openstack-nova17:03
*** dimtruck is now known as zz_dimtruck17:03
*** hieulq__ has joined #openstack-nova17:03
*** hieulq_ has quit IRC17:04
*** tbachman has quit IRC17:04
*** zz_dimtruck is now known as dimtruck17:04
cfriesenwhere does ServerExternalEvents run?  is this part of nova-api?17:04
cfriesenbah, never mind, just noticed the full path to the file17:05
*** hieulq__ has quit IRC17:07
*** sambetts is now known as sambetts|afk17:09
*** stvnoyes1 has left #openstack-nova17:11
openstackgerritMatthew Booth proposed openstack/nova master: Add a method signature test for all virt drivers  https://review.openstack.org/46567617:11
mdboothHonestly I was expecting that to find things. Surprised it didn't.17:12
*** jwcroppe_ has quit IRC17:13
mdboothlyarwood: Added ^^^ when I noticed your change to swap_volume, also your previous reluctance to change detach_volume. This test should make those things easier.17:14
*** suresh12 has joined #openstack-nova17:15
*** hieulq_ has joined #openstack-nova17:15
*** fragatina has quit IRC17:16
*** hieulq_ has quit IRC17:18
*** felipemonteiro has quit IRC17:18
*** gianpietro has joined #openstack-nova17:19
artommdbooth, I thought we already tested virt drivers's method signatures... Were some drivers missing from the list?17:20
*** ralonsoh_ has quit IRC17:21
mdboothartom: Where do we test it?17:22
mdboothAlthough that would explain why they all match :)17:22
artommdbooth, ah, I think they're individual tests in each driver17:22
artomhttp://paste.openstack.org/show/609828/17:23
artomI know I remembered getting method signature errors17:23
*** gianpietro has quit IRC17:24
mdboothThat's incredibly similar to mine, except not recursive17:25
mdboothI needed recursive for the volume drivers17:25
mdboothAlthough I hadn't noticed that existed already17:26
*** harlowja has quit IRC17:26
mdboothI wonder why it doesn't have to handle wrapped methods17:26
*** ericyoung has joined #openstack-nova17:28
artommdbooth, I agree your way is better - it doesn't depend on each driver doing their own tests correctly17:28
mdboothAh... HyperV munges it.17:28
mdboothartom: Meh, it's only important that it's checked, tbh.17:29
artommdbooth, I suppose the only issue with the current tests is if a new virt driver gets added and forgets them17:29
artomBut that doesn't seem likely17:29
*** jpena is now known as jpena|off17:30
*** gaurangt has joined #openstack-nova17:30
mdboothWell powervm.PowerVMDriver already wasn't added to the DriverMethodTestCase17:30
artomAlso, you've included PowerVM17:30
mdboothAlthough that case is stupid17:30
artomWhich isn't in the list I got17:30
mdboothartom: Can you note on the review that we already do that? I'll probably abandon it, and/or update my recursive test to use the existing method.17:32
artommdbooth, sure17:32
mdboothartom: Thanks17:32
*** Sukhdev has quit IRC17:35
*** tbachman has joined #openstack-nova17:35
artommdbooth, there. I didn't even -1 it ;)17:35
*** crushil has quit IRC17:37
mriedemcfriesen: did you happen to catch the sriov bug armax was asking about last night?17:38
*** kaisers has joined #openstack-nova17:39
*** felipemonteiro has joined #openstack-nova17:39
*** yamamoto_ has quit IRC17:39
*** kfarr has joined #openstack-nova17:40
armaxmriedem: btw talking to gyee_ it seems the issue happens in Newton still17:40
*** fragatina has joined #openstack-nova17:40
*** chyka_ has joined #openstack-nova17:41
*** yamamoto has joined #openstack-nova17:43
*** jwcroppe has joined #openstack-nova17:43
*** salv-orlando has joined #openstack-nova17:43
gyee_I am curious why we are using NopClaim if host is specified https://github.com/openstack/nova/blob/master/nova/compute/manager.py#L277117:43
gyee_does specifying a host implies force evacuate?17:44
*** crushil has joined #openstack-nova17:44
*** chyka has quit IRC17:45
*** salv-orlando has quit IRC17:48
*** gjayavelu has joined #openstack-nova17:49
*** felipemonteiro has quit IRC17:49
gyee_meanwhile if SR-IOV is enabled, we are trying to get the PCI mapping via the migration context, which is only populated at rebuild_claim. https://github.com/openstack/nova/blob/master/nova/network/neutronv2/api.py#L246217:49
*** itlinux has quit IRC17:50
*** jamesdenton has quit IRC17:50
mriedemhttps://github.com/openstack/nova/commit/dc0221d7240326a2d1b467e2a367bebb7e764e61 added that code in the compute manager about the nop claim, which implies resources were already claimed, but i'd have to dig into that17:51
*** felipemonteiro has joined #openstack-nova17:52
*** kfarr has quit IRC17:54
mriedemyikes, that was part of a big series https://review.openstack.org/#/q/topic:bug/141766717:56
*** felipemonteiro has quit IRC17:56
*** gianpietro has joined #openstack-nova17:57
gyee_but I don't see where the resource is being claimed, if the host is specified17:57
openstackgerritGage Hugo proposed openstack/nova master: Admin API Policy contingent on is_admin_project  https://review.openstack.org/38414817:57
mriedemgyee_: i'm also looking for that17:58
*** jamesdenton has joined #openstack-nova17:58
*** jamielennox is now known as jamielennox|away17:59
*** rabel_b1 has joined #openstack-nova18:02
*** liangy has quit IRC18:02
gyee_mriedem: I don't think that's being done. The migration_context in instance_extra table is empty if host specified, regardless if SR-IOV is enabled or not18:03
*** smatzek is now known as smatzek_away18:05
*** jamielennox|away is now known as jamielennox18:05
mriedemgyee_: as far as i can tell, the only way to get the noop claim is if the user specifies the host and --force18:06
*** zul has quit IRC18:06
mriedemhttps://github.com/openstack/nova/blob/master/nova/compute/api.py#L411518:07
mriedemif the host is specified but force is not, then the api nulls out host when calling conductor18:07
gyee_I didn't use the --force option. Just merely specifying the host.18:07
mriedemwhich gets to this else block https://github.com/openstack/nova/blob/master/nova/conductor/manager.py#L75118:07
mriedemnode would eventually be set here https://github.com/openstack/nova/blob/master/nova/conductor/manager.py#L77018:08
mriedemand passed to compute as scheduled_node18:08
mriedemwhich should get the rebuild_claim here https://github.com/openstack/nova/blob/master/nova/compute/manager.py#L276918:08
mriedemdansmith: do you remember anything about not doing a claim if the host is forced during evacuate?18:08
*** yamahata has joined #openstack-nova18:09
dansmithmriedem: no18:09
dansmithmriedem: I would think you'd get a new claim on the new compute node, and then drop the old one when the dead compute comes up a deletes the old instance18:10
*** rabel_b1 has quit IRC18:11
*** Sukhdev has joined #openstack-nova18:11
*** yamamoto has quit IRC18:11
mriedemi'm not sure why we have this then https://github.com/openstack/nova/blob/master/nova/compute/manager.py#L277118:12
*** yamamoto has joined #openstack-nova18:12
dansmithmriedem: well for a regular rebuild18:12
mriedemoh...yeah18:12
dansmithregular rebuild is in place18:12
mriedemoriginal commit message says "Also - if we called the scheduler, we also know the node of the chosen host, and the limits set, and this information is needed for doing claims."18:13
mriedembut yeah,18:13
*** claudiub has joined #openstack-nova18:13
mriedemi suppose the noop claim is there for rebuild on the same host18:13
mriedemnot evacuate18:13
sdagueefried: I'm pushing in your devstack console color fix, thanks again for that18:13
mriedemgod18:13
mriedemat some point we should just split the evacuate and rebuild code apart in the compute manager18:13
gyee_++18:14
mriedemi think you can also specify a host for rebuild18:15
mriedemoh no nvm18:15
mriedemi was thinking of another bug,18:15
mriedemwhere you change the image and the new image meta for the rebuild doesn't work with the original host, but we don't validate that18:16
mriedemgyee_: then i think all you need to do is add an "or recreate" to this conditional https://github.com/openstack/nova/blob/master/nova/compute/manager.py#L276718:16
mriedemgyee_: do you have a recreate of the bug?18:17
*** moshele has quit IRC18:17
gyee_mriedem: I haven't file a bug for Nova yet18:19
gyee_about to do so18:19
mriedemgyee_: oh we have a bug18:19
mriedemwell MOS does18:19
mriedembut we can add nova to that18:19
cfriesenquestion about RPC (with standard rabbitmq/kombu messaging): are incoming messages on nova-compute immediately grabbed by an eventlet?  I'm seeing huge lock contention on the 'refresh_cache-%s' lock between the live migration code and external events coming in from neutron18:19
gyee_do I have perm to do that? Sorry I am still new to Nova.18:19
mriedemgyee_: https://bugs.launchpad.net/mos/+bug/165807018:20
openstackLaunchpad bug 1658070 in Mirantis OpenStack "Failed SR_IOV evacuation with host" [Medium,Confirmed]18:20
*** sree has joined #openstack-nova18:20
gyee_oh good :-)18:20
*** psachin has quit IRC18:23
*** jamesdenton has quit IRC18:23
*** chyka has joined #openstack-nova18:23
*** moshele has joined #openstack-nova18:24
cdentbauzas, dansmith: sorry for beating a dead horse, but I need some help understanding https://review.openstack.org/#/c/465175/ (returning alternatives)18:24
*** sree has quit IRC18:25
cdentedleafe too18:25
gyee_mriedem: thanks for the help!18:25
*** chyka_ has quit IRC18:25
dansmithcdent: what about it?18:25
mriedemgyee_: np, you did the hard debug already,18:25
cdentin the collection of hosts being returned from select_destinations there is a list of dicts18:25
cdentin each dict is one host which is the chosen host18:25
mriedemgyee_: if you get a recreate and have that fix handy ping me to review18:25
cdentand then a key 'alternatives' which are other hosts that might work18:26
cdentis each chosen host (across the entire selected_hosts) different or does every single host satisfy the request_spec?18:26
gyee_mriedem: definitely, just need to add unit tests before pushing it upstream18:27
cdentor to put it another way: what is the reason for not having simply "more hosts" instead of the nested "alternatives"?18:27
*** crushil has quit IRC18:27
dansmithcdent: each host could satisfy the request.. meaning all the primaries and alternates18:27
dansmithcdent: I had thought about it in terms of just a list of hosts, and not the hierarchy, however:18:28
dansmithcdent: I think the point is that we will, for a single num_instances>1 request, not choose the same primary host for two instances, but the alternates can overlap18:28
dansmithcdent: so the distinction is made such that we consume the primaries18:28
mriedemgyee_: i'm still a bit confused about how this happens if that force flag isn't specified18:29
cdentdansmith: is the additional code complexity worth the win?18:29
dansmithcdent: well, if we don't, then we'll choose hostA, (hostB, C) for the first instance and hostB, (HostA, C) for the second, right? that's a bad decision if we fail on hostB and the first instance filled up hostA18:30
*** suresh12 has quit IRC18:30
dansmithcdent: also, I think we have to consume the hosts as primaries to keep the existing scheduler behavior, right?18:30
dansmiththe previous code did that such that num_instances>1 would not pack them all onto the same host, IIUC18:31
*** salv-orlando has joined #openstack-nova18:31
dansmithI dunno, I guess I'd wait for either bauzas or johnthetubaguy to comment on that18:31
* dansmith shuts up18:31
cdentthanks dansmith, I'll have to read through it a bit more18:32
cdentit feels like it could be simpler, but feelings may not matter18:32
*** jamesdenton has joined #openstack-nova18:32
dansmithwell, FWIW, when I first read that code I was kinda horrified with the nested dicting,18:33
dansmithbut I didn't complain because I think it probably makes the smallest behavioral change18:33
dansmithit's more code, but I'm not sure it's _that_ much more complex18:33
*** suresh12 has joined #openstack-nova18:34
dansmithit's not an unreasonable opinion though, IMHO18:34
*** jwcroppe has quit IRC18:34
dansmithrecording it, even if it is shot down with technical reasons is probably a good idea18:34
cdentroger that18:34
gyee_mriedem: maybe none != False?18:35
EmilienMmriedem, cdent: super quick fyi: I tested again nova/wsgi and it works fine18:35
cdentyay!18:35
EmilienMthis is our config: http://logs.openstack.org/81/462281/3/check/gate-puppet-openstack-integration-4-scenario001-tempest-centos-7/ce2e89d/logs/apache_config/10-nova_api_wsgi.conf.txt.gz18:35
EmilienMif you have any suggestion, feel free to let me know18:35
EmilienMthe proof it works http://logs.openstack.org/81/462281/3/check/gate-puppet-openstack-integration-4-scenario001-tempest-centos-7/ce2e89d/logs/apache/nova_api_wsgi_access_ssl.txt.gz18:35
*** itlinux has joined #openstack-nova18:36
EmilienMmwhahaha: ^ fyi also18:36
*** moshele has quit IRC18:36
mriedemgyee_: i think you might be right https://github.com/openstack/nova/blob/stable/newton/nova/api/openstack/compute/evacuate.py#L9218:37
cdentEmilienM: is cgi-bin/nova/nova-api the new wsgi script that pbr creates?18:37
EmilienMcdent: yes, nova-api-wsgi or something18:37
*** zul has joined #openstack-nova18:37
mriedemgyee_: gah, yeah, that's it18:37
edleafecdent: my understanding is that if N instances are requested, we'd pick N hosts from the list of weighed hosts. Then for each, we'd get the cell uuid and from the remaining, find enough in the same cell to satisfy the number of alternates needed18:37
EmilienMcdent: /usr/bin/nova-api-wsgi18:37
gyee_mriedem: two bugs then18:37
mriedemso if you're using microversion<2.29 then force is None18:37
mriedemyup18:37
EmilienMcdent: I need to go now, but you can PM me if you have more comments, i'll catch up later18:38
edleafeSo I didn't envision overlap at all18:38
edleafebut I suppose it wouldn't be terrible, since the claim would simply fail on a retry18:38
*** salv-orlando has quit IRC18:39
melwittdansmith, sdague: I created a WIP devstack patch to add the --tcp option to dstat and added Depends-On it for the caching db connections patch, and compared the output (using active tcp as a hint to db connections)18:40
sdaguemelwitt: cool, honestly, just add that if you like18:40
melwittI put them in a spreadsheet here https://docs.google.com/spreadsheets/d/1DIfFfX3kaA_SRoCM-aO7BN4IBEShChXLztOBFeKryt4/edit?usp=sharing18:40
sdaguedstat is really there to cross check all these things for runs, so no reason to not report it all the time18:40
melwittcool. it makes the csv less human readable but I think it's probably worth it for us to be able to have a high level indicator of how many connections we're using http://logs.openstack.org/94/465194/1/check/gate-tempest-dsvm-neutron-full-ubuntu-xenial/7a6b700/logs/dstat-csv_log.txt.gz18:42
dansmithmelwitt: not sure I get it.. both tabs top out at ~40 right?18:43
sdagueyeh, the non csv version is still emitted as well18:43
dansmithoh18:44
melwittdansmith: that's listening. I highlighted the "active/established" column, col AC. comparing a gate-tempest-dsvm-neutron-full-ubuntu-xenial in both cases, without caching active connections were 1495 at the end of the run, with caching it was 44518:44
dansmithyeah18:44
dansmithI hadn't scrolled over enough18:45
*** yamamoto has quit IRC18:45
dansmithwell, I guess that's it18:45
dansmithand needing of a backport18:45
dansmithI just dunno why we're never gc'ing those18:45
melwittah, I had hoped it would show the scrolling I did to ppl shared with18:45
melwittyeah, I don't get it either. I thought it should have worked fine the way it was18:46
mriedemmelwitt: for your todo in the wip you could use oslo.cache which i think has a default timer18:46
mriedemand can be configured for memcache18:46
dansmithmriedem: will that put it in memcache or dogpile?18:47
dansmithbecause we probably don't want that18:47
mriedemdefaults to dogpile18:47
mriedemin memory18:47
mriedemwell, i think18:47
*** MVenesio has joined #openstack-nova18:47
mriedemhttps://docs.openstack.org/developer/oslo.cache/opts.html18:47
dansmithwe just don't want those shared across machines in any way, and not serialized either18:47
*** salv-orlando has joined #openstack-nova18:48
melwittmriedem: I have looked at oslo.cache briefly before and wasn't sure about whether or not to use it. I don't think I found examples in nova18:49
*** felipemonteiro has joined #openstack-nova18:49
dansmithwell, for the backport we can ignore the cache purge,18:49
dansmithsince the other lists aren't purged either18:49
melwittdansmith: I added a thing to clear it upon SIGHUP. as far as expiring based on a timer, the utility of it seems to be mostly around not keeping a connection open to a cell that's seldom used18:51
dansmithright, I'm just saying we needn't worry about any of this for the first patch that we want to be simple for backporting18:52
dansmithI wouldn't even hook up SIGHUP since the other lists are't either18:52
melwittokay18:53
dansmithrelated tangent,18:55
dansmithreporting the tcp connection count like this all the time seems like a good thing, and is another thing we could measure and report on independent of worker performance18:55
dansmithi.e "ending tcp connections is >100% higher than expected"18:55
dansmithre: the perf-monitoring forum session18:55
melwittyeah, I was thinking the same18:55
dansmithI also wonder if we should open a bug against oslo-db for this18:56
melwittgoing to un-WIP the devstack patch. sdague, where is the non-csv dstat in the logs? I was wondering should we put --tcp in for both csv and non-csv? I did but I wasn't sure18:56
*** mtanino_ has joined #openstack-nova18:58
melwittdansmith: yeah, should at least ask zzzeek about it and open a bug if it's not an expected thing18:58
*** catintheroof has joined #openstack-nova18:59
*** salv-orlando has quit IRC18:59
sdaguemelwitt: it's in a screen19:00
sdaguewell, it's in the journal :)19:00
sdaguebut we dump it out to screen19:00
sdaguefiles19:00
sdaguefor compat in gate runs19:00
melwittcool, thanks19:01
*** crushil has joined #openstack-nova19:03
*** READ10 has quit IRC19:03
*** lpetrut has joined #openstack-nova19:03
*** Vek has quit IRC19:04
*** mordred has quit IRC19:05
*** jwcroppe has joined #openstack-nova19:06
openstackgerritArtom Lifshitz proposed openstack/nova master: Plumbing for tagged nic attachment  https://review.openstack.org/41562219:06
openstackgerritArtom Lifshitz proposed openstack/nova master: Delete nic metadata when detaching interface  https://review.openstack.org/43538719:06
openstackgerritArtom Lifshitz proposed openstack/nova master: Libvirt support for tagged volume attachment  https://review.openstack.org/41810019:06
openstackgerritArtom Lifshitz proposed openstack/nova master: Delete disk metadata when detaching volume  https://review.openstack.org/43508219:06
openstackgerritArtom Lifshitz proposed openstack/nova master: Plumbing for tagged volume attachment  https://review.openstack.org/39194119:06
openstackgerritArtom Lifshitz proposed openstack/nova master: API support for tagged device attachment  https://review.openstack.org/37602119:06
openstackgerritArtom Lifshitz proposed openstack/nova master: Libvirt support for tagged nic attachment  https://review.openstack.org/41552019:06
*** mordred has joined #openstack-nova19:06
openstackgerritSteve Noyes proposed openstack/nova master: WIP: update live migration to use v3 cinder api  https://review.openstack.org/46398719:10
melwittsdague: patch has been un-WIP https://review.openstack.org/#/c/46519419:11
sdaguemelwitt: +2, thanks19:12
melwittthx19:12
*** artom has quit IRC19:13
*** stvnoyes has joined #openstack-nova19:14
cdentdansmith, bauzas, edleafe I tried to leave some questions on https://review.openstack.org/#/c/465175/ to capture my concerns, but not sure if they are clear19:16
openstackgerritMatt Riedemann proposed openstack/nova master: Destroy the ReqSpec object when deleting the instance  https://review.openstack.org/39106019:17
*** gomarivera_ has joined #openstack-nova19:22
*** karimb has joined #openstack-nova19:26
*** gomarivera_ has quit IRC19:27
*** mkoderer has quit IRC19:27
*** awaugama has quit IRC19:28
*** jose-phillips has quit IRC19:34
*** clenimar has joined #openstack-nova19:34
*** gianpietro has quit IRC19:35
*** NostawRm has quit IRC19:36
*** yamahata has quit IRC19:37
*** harlowja has joined #openstack-nova19:39
openstackgerritMatt Riedemann proposed openstack/nova master: Plumbing for tagged nic attachment  https://review.openstack.org/41562219:40
openstackgerritMatt Riedemann proposed openstack/nova master: Delete nic metadata when detaching interface  https://review.openstack.org/43538719:40
openstackgerritMatt Riedemann proposed openstack/nova master: Libvirt support for tagged volume attachment  https://review.openstack.org/41810019:40
openstackgerritMatt Riedemann proposed openstack/nova master: Delete disk metadata when detaching volume  https://review.openstack.org/43508219:40
openstackgerritMatt Riedemann proposed openstack/nova master: Plumbing for tagged volume attachment  https://review.openstack.org/39194119:40
openstackgerritMatt Riedemann proposed openstack/nova master: API support for tagged device attachment  https://review.openstack.org/37602119:40
openstackgerritMatt Riedemann proposed openstack/nova master: Libvirt support for tagged nic attachment  https://review.openstack.org/41552019:40
*** xyang1 has quit IRC19:43
*** xyang1 has joined #openstack-nova19:45
melwittcfriesen: had you filed a bug for the increase in DB connections in Newton? I'll need one to refer to in my patch19:45
*** yamamoto has joined #openstack-nova19:45
*** jose-phillips has joined #openstack-nova19:46
*** Sukhdev has quit IRC19:47
*** lpetrut has quit IRC19:47
*** crushil has quit IRC19:49
*** jose-phillips has quit IRC19:52
*** harlowja has quit IRC19:53
openstackgerritMatt Riedemann proposed openstack/nova master: Remove DeviceIsBusy exception  https://review.openstack.org/46572319:56
*** karimb has quit IRC19:56
*** nic1 is now known as nic19:57
*** harlowja has joined #openstack-nova19:57
*** karimb has joined #openstack-nova19:57
*** karimb has quit IRC19:57
*** harlowja has quit IRC20:00
openstackgerritmelanie witt proposed openstack/nova master: Cache database and message queue connection objects  https://review.openstack.org/46504220:01
*** yamamoto has quit IRC20:02
*** cdent has quit IRC20:05
zzzeekmelwitt: can you tell me about https://review.openstack.org/#/c/465042/3/nova/context.py20:06
*** crushil has joined #openstack-nova20:06
melwittzzzeek: yes! I was hoping you would be able to comment on this. recently we've seen a significant increase in number of DB connections since we've been doing DB connection switching for cells20:07
zzzeekmelwitt: so...did you try changing pool_size and max_overflow ?20:08
zzzeekmelwitt: also when you say "switching" - are you retrieving independent connections per cell DB or are you "switching" a single connection to refer to different databases within a single MySQL server?20:08
zzzeekIIUC "cells" would be in different DB servers altogether so they'd have to have independent DB connections20:09
dansmithzzzeek: correct, we're just surprised that when we create and stop using one, they don't seem to be gc'd20:09
melwittzzzeek: independent connection. to make a connection we create a new oslo.db transaction context manager20:09
dansmithzzzeek: i.e. every time we do that switch (i.e. new connection), we get a connection that never goes away20:09
zzzeekdansmith / melwitt OK, so, the SQLAlchemy engine by default uses a connection pool20:09
zzzeekif you don't want it to pool anything, you need to turn it off20:10
*** smatzek_away has quit IRC20:10
*** dnovosel has joined #openstack-nova20:10
dansmithzzzeek: the pool is keeping references to all the connections then?20:10
*** pchavva has quit IRC20:10
zzzeekdansmith: it keeps refernces to as many connections as you've told it to.   our default in oslo.db is probably 5 at the moment20:10
dansmithzzzeek: we end up with ~1400 connections after a tempest run20:11
dansmithI'm guessing it doesn't know that each re-connection to the same db is reusable though? so it's keeping 5 connections of every transaction manager?20:11
zzzeekdansmith: we end up with 1.4K connections on an openstack even w/o separate cell DBs if we have dozens of processes.  it's kind of a problem to which there are a few solutions20:11
dansmithif that's the case, we should just not do that thing (i.e. do what melwitt has proposed)20:11
melwittzzzeek: I had thought the pool would hand back an object that is the same DB as before20:11
dansmithzzzeek: well, it doesn't happen with melwitt's patch20:12
zzzeekdansmith: right but, you don't need a patch.  just turn down the pooling20:12
*** suresh12 has quit IRC20:12
zzzeekdansmith: with one exception, which is that to disable the pool entirely, you need to use the "nullpool" implementation and we might need to add a hook to oslo.db if you really want that. but that is not what melwitt's patch is doing.20:12
dansmithwell, we get different behavior than what we have today.. meaning we defeat the benefits of the connection pooling , if there are benefits20:13
zzzeekdansmith: just in the last few IRC lines im not sure if all three of us are clear on every concept here, for example, I don't know what a "Transaction manager" is20:13
dansmithzzzeek: the oslo.db context20:13
melwittit's the _TransactionContextManager class in oslo.db20:14
dansmithhttps://review.openstack.org/#/c/465042/3/nova/context.py20:14
dansmithL384 there ^20:14
zzzeekalso, the idea of a "pool" is that yes, it has a connection that it will reuse as needed20:14
melwittokay, so I don't know what we're doing to not get the reusability20:14
dansmithmelwitt: the pool manager has to know that a repeated connection attempt is "the same" in order to re-use an existing connection20:15
dansmithso I'm wondering if we're confusing it such that it isn't re-using every time we create a new context there20:15
zzzeekdansmith: ok more terms that confuse me, what is "the pool manager"20:15
melwittright. how does it normally know that I guess? yeah20:15
dansmithzzzeek: I'm sure I'm not using the right term.. whatever the thing is in sqla that does the pooling.. you said "nullpool" above, which would be one of those things20:16
dansmithmelwitt: well, it can do connection pooling with one context easily20:16
melwittright ...20:16
dansmithand that's all we've needed in the past20:16
dansmithbut now we're creating lots of those things20:17
zzzeekdansmith: OK.  so we are talking aobut SQLAlchemy QueuePool or NullPool.  it doesn't know about connection attempts being "the same" or anything like that, it basically is asked for a DB connection, and it gives you one from its pool.  if it doesnt have any, it either decides to give you an ad-hoc connection based on what you've configured "max_overflow" towards, or, it makes you wait20:17
melwittzzzeek: this is how we establish db connections (static) and what we did to "switch" is just create a new one of those each time https://github.com/openstack/nova/blob/master/nova/db/sqlalchemy/api.py#L79-L8020:17
mriedemi'd think it would key off the connection url20:17
melwittmriedem: that's what I had thought20:17
dansmithmaybe, but that has creds in it20:17
dansmithzzzeek: but is that pool global, or per-context or what?20:17
*** dimtruck is now known as zz_dimtruck20:18
zzzeekdansmith: if nova is architected here per best practices, you'd have one pool per unique database URL20:18
dansmithheh20:18
zzzeekdansmith: if OTOH you are creating brand new engines every time you get a requset, then you're making new pools each time and that is the wrong way to do that20:18
dansmithwe're a long ways away from the pool in nova, so it's hard to say20:19
zzzeekdansmith: so in this patch i see "db_connection", and if that is in fact an "Engine" object, or say the enginefacade factory around that, then that's OK, that's what should happen here.20:19
zzzeekthat's not a "connection", it's a factory for connections20:19
zzzeekif that's the case, I dont really know what this is20:19
dansmithenginefacade.transaction_context()20:20
dansmithit's that ^20:20
melwittit's a _TransactionContextManager20:20
zzzeekdansmith / melwitt ok20:20
*** gianpietro has joined #openstack-nova20:20
*** marst has joined #openstack-nova20:20
melwittI modeled it off how it's done in our db/sqlalchemy/api.py for the static connections20:20
zzzeekmelwitt: OK then that's fine let me update my thing20:21
efriedsdague Thinking I should blast the ML to let 'em know about journalctl -a, yah?20:21
zzzeekmelwitt: to clarify, previosly, everyime new cell DB were used, you'd build up whole new _TransactionContextManager per request ?20:21
efriedI mean, it's in the doc now (when does that get flushed to docs.openstack.org?) but who's gonna go reread it?20:22
*** suresh12 has joined #openstack-nova20:23
melwittzzzeek: yeah. and that was based on my limited knowledge of how to use oslo.db, if I want to establish a new connection from only a connection url, it seemed like that's the only way?20:23
melwittis to use enginefacade.transaction_context()20:23
zzzeekmelwitt: OK so, is that version of the code what's released in production and all that? because that will perform very, very badly20:23
cfriesenwho knows neutron and live-migration?   Do we really need to process "network-changed" events on the dest compute node during a migration when we're going to explicitly call network_info = self.network_api.get_instance_nw_info() anyways in post_live_migration_at_destination()?20:23
melwittzzzeek: yes. that's why at the moment, it seems like caching these is the way to fix this, and backport it all the way to newton20:24
zzzeekmelwitt: it is.    So, background, the transactioncontextmanager is  a facade around the concept of the SQLAlchemy engine20:24
mriedemcfriesen: we probably don't need to, but does refreshing the network info twice do any harm?20:25
zzzeekthis is why it has a method "get_engine()".   when you use it as a decorator or context manager, it goes in to a "block" in which scope it actually procures an ORM session which uses an actual sqlalchemy.engine.Connection object20:25
zzzeekmelwitt: the concept of the Engine is described at http://docs.sqlalchemy.org/en/rel_1_1/core/engines.html http://docs.sqlalchemy.org/en/rel_1_1/core/connections.html http://docs.sqlalchemy.org/en/rel_1_1/core/pooling.html and would provide some background on what's going on20:26
*** suresh12 has quit IRC20:27
melwittzzzeek: thanks20:27
zzzeekmelwitt: the history of "enginefacade" itself is that when I came to work on openstack, there was already an "EngineFacade" local to openstack / oslo that evreyone was using, so the curent oslo.db version tries to improve upon that.   however, like all layers of indirection, it is confusing.  I wrote it and I never remember how it is organized when I go back and look at it, because it had to work on top of all these existing20:27
zzzeekpatterns20:27
cfriesenmriedem: turns out that it does when you have a lot (like 16) vifs on an instance.  The live migration code can end up blocking for like 35+ seconds waiting to get the 'refresh_cache-%s' lock because it's racing against the incoming network-change events20:28
zzzeekmelwitt: so short story, if you make a new engine (and therefore a new connection pool) on every request, then yes, the "pool" is not working because you are making a whole new pool each time20:29
melwittzzzeek: I see. I too, find confusing :)20:29
mriedemcfriesen: ok i wondered about the lock20:29
cfriesenmriedem: the lack of fair locking means that the events are likely to drop/regain the lock before the live migration code gets it20:29
zzzeekmelwitt: ive tried for many years to get the docs / API to express this concept that when you have a URL in SQLAclhemy, you get a *factory*, and not the connection directly, but then with enginefacade in the mix, the bar for explaining is higher.20:29
cfriesenmriedem: adding a greenthread.sleep(0) in the event path helps a lot, but that only works if you're using the "blocking" rpc executor20:30
melwittzzzeek: gotcha. thanks for explaining. I had been expecting something like a tcp socket pool where it would see us connecting to the same place repeatedly and reuse objects. but I think I understand now why it wouldn't be able to match those things up20:30
mriedemcfriesen: what is the 'blocking' rpc executor?20:30
cfriesenmriedem: oslo_messaging/_executors/impl_blocking.py20:32
cfriesenmriedem: default RPC executor in oslo messaging20:32
mriedemcfriesen: i see, https://docs.openstack.org/developer/oslo.messaging/executors.html20:34
*** itlinux_ has joined #openstack-nova20:34
mriedemcfriesen: well, we could potentially ignore the event if we know the instance is undergoing a certain task state transition20:34
mriedemcfriesen: looking at nova.rpc, we use eventlet for the executor20:36
mriedemexecutor='eventlet',20:36
cfriesenmriedem: we were just wondering internally whether we could ignore them if it was in the middle of a live migration.  do you think the state contained in "instance" at the time it was pulled from the DB in ServerExternalEventsController would be recent enough or would we need to check the DB20:37
*** itlinux has quit IRC20:37
mriedemcfriesen: between the time that the api pulls the instance from the db and the time it rpc casts to the compute and checks the state, i don't imagine there would be a huge difference, but it does become a race and you could end up ignoring an event that you actually wanted to process20:38
*** slaweq has joined #openstack-nova20:38
*** slaweq has quit IRC20:38
zzzeekmelwitt: oh, like an implicit global pool per URL.  yeah, we don't do it that way.  SQLA doesn't have any global registries.  However, you're right that oslo.db *could* serve this purpose.   it's just that a URL is bundled with all kinds of other things like pooling config, tranasction isolation, SQL mode stuff, encodings, etc.20:38
*** jose-phillips has joined #openstack-nova20:38
cfriesenmriedem: hmm...that's sort of odd, I wasn't seeing a bunch of events getting bunched up all waiting on the lock.   In that case the yield isn't guaranteed to buy you anything and only "fair locks" would provide a guarantee20:39
zzzeek(sqla has some global registries but not linked to things like strings)20:40
openstackgerritMatt Riedemann proposed openstack/nova master: Plumbing for tagged nic attachment  https://review.openstack.org/41562220:41
*** slaweq has joined #openstack-nova20:41
melwittzzzeek: out of curiosity, is there a better way to do something like connecting to several different DBs? or is what we have reasonable20:41
openstackgerritMatt Riedemann proposed openstack/nova master: Delete nic metadata when detaching interface  https://review.openstack.org/43538720:41
openstackgerritMatt Riedemann proposed openstack/nova master: Libvirt support for tagged volume attachment  https://review.openstack.org/41810020:41
openstackgerritMatt Riedemann proposed openstack/nova master: Delete disk metadata when detaching volume  https://review.openstack.org/43508220:41
openstackgerritMatt Riedemann proposed openstack/nova master: Plumbing for tagged volume attachment  https://review.openstack.org/39194120:41
openstackgerritMatt Riedemann proposed openstack/nova master: API support for tagged device attachment  https://review.openstack.org/37602120:41
openstackgerritMatt Riedemann proposed openstack/nova master: Libvirt support for tagged nic attachment  https://review.openstack.org/41552020:41
*** zz_dimtruck is now known as dimtruck20:41
zzzeekmelwitt: in here I see "db.create_context_manager()" which seems pretty clean.   what determines the other settings for this CELL db, they are taken from the global settings?  like how do we know it's utf-8 encoding, stuff like that ?20:42
melwittzzzeek: they all use the same config, they're just clones of what we call the "main DB" they all have the same schema20:43
dansmithclone in the schema sense20:43
melwittyeah, sorry20:43
zzzeekmelwitt: yup.   OK so oslo.db could have a featrue like this.  like in enginefacade, you give it the URL and some flag, "use_existing" or something, pulls it from a dictionaar like you have here keyed to URL20:44
*** gouthamr has quit IRC20:44
melwittzzzeek: okay, cool. thanks. just wanted a sanity check of whether there were more appropriate objects to use that we didn't use20:47
sdagueefried: totally feel free to, it was a nice add20:48
*** zul has quit IRC20:51
efriedsdague Pursuant to your colors-by-hash idea, it occurred to me that it might be more useful to do that by service, now that we've got converged logs.20:52
sdagueefried: you don't have access that early20:53
efriedJust thinking whether consumer is more likely to grep away by req-id or by service - and I'm thinking it's the former by a mile.20:53
sdaguethat's set by systemd itself20:53
sdagueefried: you don't need to grep by service though20:53
sdaguebecause -u does that for you20:53
efriedsdague Yeah, exactly my point.20:54
efriedIt would be a visual break showing you when you're jumping services.20:54
cfriesentask_states.MIGRATING is only for live-migration, right?20:54
sdagueright, so the reason I thought it would be nice to rainbow hash the req-id is that it becomes more clear when you have competing flows happening at the some time20:54
sdaguemost of our races are actually related to that20:54
efriedsdague Makes sense.20:54
sdagueit's potentially totally gratuitous, but it was enough to wonder "hey, that might be useful"20:55
efriedsdague If it's just coloring the req-id string itself (not the whole message), it should be harmless at worst... unless you wind up printing black-on-black.20:55
melwittcells meeting in a few min20:57
sdagueefried: yeh, I was thinking just the req-id string20:57
*** Vek has joined #openstack-nova21:00
*** catintheroof has quit IRC21:01
*** Apoorva_ has joined #openstack-nova21:01
*** armax has quit IRC21:02
*** thorst_afk has quit IRC21:03
*** rfolco has quit IRC21:03
*** jwcroppe has quit IRC21:03
*** dnovosel has quit IRC21:05
*** Apoorva has quit IRC21:05
*** Sukhdev has joined #openstack-nova21:05
*** hongbin has quit IRC21:08
*** gomarivera_ has joined #openstack-nova21:11
*** MVenesio has quit IRC21:11
*** itlinux_ has quit IRC21:14
*** gouthamr has joined #openstack-nova21:14
*** ijw has joined #openstack-nova21:15
*** gomarivera_ has quit IRC21:15
*** hongbin has joined #openstack-nova21:16
*** kylek3h has quit IRC21:16
*** artom has joined #openstack-nova21:17
*** ijw has quit IRC21:19
*** esberglu has quit IRC21:19
*** ijw has joined #openstack-nova21:20
*** ijw has quit IRC21:20
*** ijw has joined #openstack-nova21:20
*** mnestratov has joined #openstack-nova21:22
*** sdague has quit IRC21:22
*** kylek3h has joined #openstack-nova21:22
*** ijw has quit IRC21:25
*** ijw has joined #openstack-nova21:26
*** edmondsw has quit IRC21:27
*** edmondsw has joined #openstack-nova21:28
*** edmondsw_ has joined #openstack-nova21:31
*** edmondsw has quit IRC21:32
*** marst has quit IRC21:33
melwittdansmith: got a bunch of unit test fails on my patch bc I think I need to clear that cache between tests21:34
dansmithmelwitt: yep21:34
dansmithjust like we do for the others21:34
*** ijw has quit IRC21:34
*** rfolco has joined #openstack-nova21:34
melwittI just grepped for CELLS in the tests and didn't find anything. where are you doing it?21:34
melwitter, didn't find a clearing of it21:34
*** ijw has joined #openstack-nova21:35
*** edmondsw_ has quit IRC21:36
melwittoh, derp, I grepped in nova/tests but it's in nova/test.py21:36
melwittnvm21:36
dansmithyepo21:36
*** avolkov has quit IRC21:36
*** ijw has quit IRC21:39
*** ijw has joined #openstack-nova21:40
cfriesenmriedem: dansmith: would something like this seem okay to you? http://paste.openstack.org/show/609851/   I'm a little bit worried about _rollback_live_migration() which only calls setup_networks_on_host() and not get_instance_nw_info() like the network-changed notifier does21:41
*** baoli_ has quit IRC21:41
*** burt has quit IRC21:41
*** mtanino_ has quit IRC21:44
*** ijw has quit IRC21:47
*** ijw has joined #openstack-nova21:48
*** Sukhdev has quit IRC21:50
*** ijw has quit IRC21:52
*** ijw has joined #openstack-nova21:52
openstackgerritmelanie witt proposed openstack/nova master: Cache database and message queue connection objects  https://review.openstack.org/46504221:53
*** mnestratov has quit IRC21:53
*** ijw has quit IRC21:54
*** slaweq has quit IRC21:54
*** mdrabe has quit IRC21:55
*** ijw has joined #openstack-nova21:55
*** jamesdenton has quit IRC21:57
*** xyang1 has quit IRC21:58
*** rfolco has quit IRC21:58
*** thorst_afk has joined #openstack-nova22:00
mriedemcfriesen: i don't really care for the instance.refresh thing22:02
*** harlowja has joined #openstack-nova22:02
*** kylek3h has quit IRC22:12
*** dimtruck is now known as zz_dimtruck22:14
*** zz_dimtruck is now known as dimtruck22:14
*** harlowja has quit IRC22:18
*** thorst_afk has quit IRC22:20
*** dimtruck is now known as zz_dimtruck22:23
*** mlavalle has quit IRC22:29
*** Apoorva_ has quit IRC22:33
*** Apoorva has joined #openstack-nova22:33
*** itlinux has joined #openstack-nova22:33
*** suresh12 has joined #openstack-nova22:38
cburgessIs it actually possible to select additional scheduler filters to use on the boot command?22:46
cfriesenmriedem: the point of that is to query from the DB the current task_state to see if it's migrating...got a better way to do that?22:46
*** baoli has joined #openstack-nova22:48
*** baoli has quit IRC22:48
*** baoli has joined #openstack-nova22:49
dansmithcfriesen: I don't see why we should skip refreshes altogether for migrating instances, nor do I think it's reasonable to do that heavy lookup every time22:49
dansmithwe already send events to both host and destination during migration22:49
*** itlinux has quit IRC22:49
dansmithif you ask for any pre-review, can you do it in gerrit please? it's annoying to see only ten lines of context22:49
cfriesendansmith: we've seen the refresh take 35+ seconds due to lock contention with the incoming network-change events when there are 16 vifs.  the events kept on acquiring the lock even though the migration code tried to get it first.22:50
cfriesendansmith: lack of fair locking was giving grief22:50
*** eharney has quit IRC22:52
cfriesendansmith: with the 2-sec delay due to neutron batching, the DB lookup from instance.refresh() is way faster22:52
cfriesendansmith: and the reason to skip them for live-migrating instances is that after the migration is done we're going to refresh the cache explicitly22:53
dansmithbut for everyone else that's an extra db lookup on every event22:53
*** zz_dimtruck is now known as dimtruck22:54
cfriesendansmith: got any suggestions?  fair locking would help unblock the migration code but we'd still have the 2-sec neutron delays (and that seems serialized somehow so it takes ~8sec to finish getting all the network-changed events after a migration22:55
*** erhudy has quit IRC22:55
mriedemcfriesen: do you really need the refresh at all?22:56
mriedemwhat are the chances that the live migration completes and the task_state changes in the window of time between when the api gets the instances for the events and when you check in the compute?22:56
cfriesenmriedem: that's why I asked you earlier...seems like there'd be a small race window where we might ignore an event we should have processed22:58
cfriesenwhether that would actually cause problems...I don't know22:58
melwittcburgess: what do you mean by select? other than adding them to enabled_filters in your nova.conf?22:58
*** itlinux has joined #openstack-nova22:59
cfriesenmriedem: dansmith: to ignore it the instance would have been migrating at the time nova-api pulled the instance from the DB, so I *think* it should be valid to ignore the event23:00
cburgessmelwitt Exactly. We have available_filters and default_filters in nova.conf. default gets used when you do a launch but is there a way as part of a boot to add a filter to that list or use a different filter?23:00
cfriesencburgess: I don't see a way...get_filtered_hosts() calls self.filter_handler.get_filtered_objects(self.enabled_filters....) where self.enabled_filters is used as the list of filters to evaluate23:01
melwittcburgess: you mean potentially use different filters for different boot requests? I don't think so23:01
dansmithcfriesen: I don't see why it's valid to ignore it ever, but especially not just because you might've lost the race where we already looked it up in the api23:02
cfriesendansmith: the event handler for the network-changed message just calls self.network_api.get_instance_nw_info().  the live-migration code will call that explicitly when the migration finishes.23:04
cfriesenthat's a relatively expensive call, so we'd like to minimize the number of times we call it23:04
cfriesenand we'd like to avoid the main migration code contending on the lock with the events23:05
cburgessmelwitt cfriesen So whats what I thought. Which begs the question.. why do we have 2 config values?23:05
melwittcburgess: I think the only way to get different behavior for different boot would be scheduler hints (and add a filter that handles that hint) or extra specs to do it via flavor23:05
cburgessmelwitt Ahh that could be why hints. I haven't looked at that.23:06
melwittcburgess: that's a good question ... checking23:06
cfriesencburgess: 2 config values for what/23:06
cfriesen?23:06
melwittlike available filters vs enabled filters23:06
cburgesscfriesen scheduler_default_filters vs scheduler_available_filters23:07
dansmithcfriesen: some virt driver could need to wait for some update during live migration and need to see that refresh to proceed, no?23:07
melwittI actually don't know why23:07
dansmithcfriesen: sounds like you want to fix the locking problem instead of just whack-a-mole'ing your immediate problem23:07
*** liverpooler has quit IRC23:08
cfriesendansmith: there are a few issues:  1) lock isn't fair, 2) the call to neutron is expensive, 3) what does it buy us to call self.network_api.get_instance_nw_info() a bunch of times during a single live-migration?23:08
dansmithcfriesen: fix #1, skip duplicate instances as you are, and I just wrote a virt driver that depends on those refreshes to complete phases of my live migration.23:09
*** lbragstad has quit IRC23:10
cburgessmelwitt Yeah ok thats what I thought.. superfluous... I'll keep digging into the hints thing but it really look to me like its redundant.23:12
cfriesendansmith: would a virt driver ever use InstanceInfoCache?23:12
*** gouthamr has quit IRC23:12
dansmithcfriesen: sure, the one I just wrote does23:13
melwittcburgess: yeah, I'm not seeing any clues as to why there are two options. maybe a legacy thing that used to have a purpose. jaypipes, bauzas, or edleafe might know23:13
cfriesen:)23:13
*** mtanino has joined #openstack-nova23:13
cfriesendansmith: okay, so assuming we do the locking thing...do you think we would want to modify the standard lock (likely making it somewhat slower in the fastpath) or introduce a new "fair" lock and hope services switch to using it?23:14
dansmithcfriesen: there's lots of network_info walking throughout the xen driver at least..23:15
dansmithcfriesen: I have no idea what you want to do, and would suggest you ask the lockutils people what they think23:15
cfriesendansmith: I think it'd be nuts to rely on incoming network-changed events during a live migration since they can be processed in any order due to RPC and locking contention.  But who knows what crazy stuff xen might do.23:16
cfriesendansmith: fair enough23:16
dansmithif you're waiting for some property on a vif to become a value, I see no reason why that wouldn't be reasonable23:17
dansmithanyway, EOD and EOP for me23:17
* dansmith &23:17
cfriesenthanks for the feedback23:17
*** dikonoor has joined #openstack-nova23:19
*** gouthamr has joined #openstack-nova23:20
*** thorst_afk has joined #openstack-nova23:20
*** claudiub has quit IRC23:23
*** seba has quit IRC23:24
* mriedem wonders what virt driver dan wrote23:24
mriedemSteelyDanHypervisor?23:24
*** jgrassler has quit IRC23:25
mriedemi'm going to need someone to run the nova team meeting tomorrow...23:26
cfriesen10thDanBlackBeltHypervisor23:26
*** itlinux has quit IRC23:27
mriedemalso, i wonder if dan is aware of dan's opinions on out of tree virt drivers23:27
mriedemoomichi: were you interested in taking over either https://blueprints.launchpad.net/nova/+spec/live-migration-per-instance-timeout or https://blueprints.launchpad.net/nova/+spec/live-migration-force-after-timeout ?23:31
*** crushil has quit IRC23:32
*** seba has joined #openstack-nova23:34
*** karthiks has quit IRC23:35
*** jgrassler has joined #openstack-nova23:36
*** dimtruck is now known as zz_dimtruck23:37
*** pewp has quit IRC23:38
*** pewp has joined #openstack-nova23:38
*** ijw has quit IRC23:38
*** ijw has joined #openstack-nova23:39
*** thorst_afk has quit IRC23:41
*** mtanino has quit IRC23:44
*** chyka has quit IRC23:45
*** chyka has joined #openstack-nova23:45
*** felipemonteiro has quit IRC23:49
*** chyka has quit IRC23:50
openstackgerritEd Leafe proposed openstack/nova master: Have scheduler claim resources  https://review.openstack.org/46399323:51
openstackgerritEd Leafe proposed openstack/nova master: Remove selecting host based on host_subset_size  https://review.openstack.org/46517423:51
openstackgerritEd Leafe proposed openstack/nova master: Make the method to put allocations public  https://review.openstack.org/46408623:51
openstackgerritEd Leafe proposed openstack/nova master: Don't delete allocation if instance being scheduled  https://review.openstack.org/46017723:51
openstackgerritEd Leafe proposed openstack/nova master: Pass a list of instance UUIDs to scheduler  https://review.openstack.org/46517123:51
openstackgerritEd Leafe proposed openstack/nova master: WIP - Pass the number of hosts to return explicitly  https://review.openstack.org/46576723:51
openstackgerritEd Leafe proposed openstack/nova master: WIP - return multiple hosts from select_destinations  https://review.openstack.org/46576823:51
mriedemmelwitt: do you want to tweak the reno at all in this? https://review.openstack.org/#/c/465042/23:51
mriedemif not, i'm +223:51
melwittmriedem: yeah, I can change it to your suggest and flip the test assertions. I wasn't thinking when I copied them23:52
melwitt*suggestion23:52
*** armax has joined #openstack-nova23:53
openstackgerritHongbin Lu proposed openstack/nova master: Handle exception on adding secgroup  https://review.openstack.org/46517323:53
cfriesenmriedem: so it looks like _build_network_info_model() costs about 200ms plus about 125ms per port since we query each port separatly from neutron.  and the refresh_cache lock is held the whole time23:55
cfriesenmriedem: I wonder if we could issues those requests in parallel for all the ports?23:56
mriedemcfriesen: we've talked for years about needing to do bulk queries from neutron23:56
*** hongbin has quit IRC23:56
mriedemwell,23:57
mriedemwhere do we re-query per port, after we get the list of ports from data = client.list_ports(**search_opts)23:57
cfriesen_nw_info_get_ips and _nw_info_get_subnets both take noticeable time23:58
cfriesenfor each port23:58
mriedemthe first is going to iterate over all fixed_ips for a port and get floating IPs for each fixed_ip23:59

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!