Wednesday, 2017-09-27

*** acormier has joined #openstack-nova00:01
*** penick has quit IRC00:02
*** tetsuro has quit IRC00:03
*** mriedem1 has joined #openstack-nova00:05
*** acormier has quit IRC00:05
*** mriedem has quit IRC00:05
*** mriedem1 is now known as mriedem00:08
*** thorst has joined #openstack-nova00:08
*** thorst has quit IRC00:09
*** yangyapeng has quit IRC00:11
*** yangyapeng has joined #openstack-nova00:12
mriedemdansmith: interesting, listing without details, 500 error (cell0) and 500 active (cell1) is a lot faster than the 1000 active,00:13
mriedemi suppose because we don't have as much to join00:13
dansmithmriedem: with my patch or before?00:13
mriedemoh shit, nvm - copy paste error00:13
mriedemwas using the compute endpoint url from my other devstack :)00:13
mriedem"wow this is fast!"00:14
mriedemhah, here we go, nice and slow00:15
mriedemfault loading mofos00:15
*** yufei has joined #openstack-nova00:18
*** yufei has quit IRC00:18
mriedem4.495s with GET /servers, 1000 ACTIVE vms. 11.185s with 500 error, 500 active00:20
*** moshele has quit IRC00:27
*** smatzek_ has quit IRC00:29
mriedem24.125s to list them with details00:30
*** hemna_ has joined #openstack-nova00:34
*** nicolasbock has quit IRC00:35
*** yamahata has quit IRC00:36
*** Apoorva_ has joined #openstack-nova00:40
*** nicolasbock has joined #openstack-nova00:42
openstackgerritMichael Still proposed openstack/nova master: Move ploop commands to privsep.
openstackgerritMichael Still proposed openstack/nova master: Read from console ptys using privsep.
openstackgerritMichael Still proposed openstack/nova master: Don't shell out to mkdir, use ensure_tree()
openstackgerritMichael Still proposed openstack/nova master: Cleanup mount / umount and associated rmdir calls
openstackgerritMichael Still proposed openstack/nova master: Move lvm handling to privsep.
openstackgerritMichael Still proposed openstack/nova master: Move shred to privsep.
openstackgerritMichael Still proposed openstack/nova master: Move xend existence probes to privsep.
openstackgerritMichael Still proposed openstack/nova master: Move the idmapshift binary into privsep.
openstackgerritMichael Still proposed openstack/nova master: Move loopback setup and removal to privsep.
openstackgerritMichael Still proposed openstack/nova master: Move nbd commands to privsep.
openstackgerritMichael Still proposed openstack/nova master: Move kpartx calls to privsep.
openstackgerritMichael Still proposed openstack/nova master: Move blkid calls to privsep.
*** Apoorva has quit IRC00:43
mriedeminteresting, listing with details and microversion 2.53 is not much worse than with microversion 2.1 for the error/active mix case - it was nearly double between microversions when all were active00:44
mriedemdansmith: time for your change, do i need or just the one below it?00:47
dansmithmriedem: the one below it should orphan those so they're never called00:48
dansmithso you shouldn't notice any difference afaik00:48
*** acormier has joined #openstack-nova00:49
*** acormier has quit IRC00:50
*** acormier has joined #openstack-nova00:51
*** jichen has joined #openstack-nova00:52
*** acormier has quit IRC00:53
*** acormier has joined #openstack-nova00:54
*** Apoorva_ has quit IRC00:54
*** acormier_ has joined #openstack-nova00:56
*** Shunli has joined #openstack-nova00:58
*** acormier has quit IRC00:58
*** esberglu has joined #openstack-nova00:59
*** yamamoto has quit IRC01:03
*** yamamoto_ has joined #openstack-nova01:03
*** phuongnh has joined #openstack-nova01:03
*** esberglu has quit IRC01:03
*** Sukhdev has quit IRC01:03
*** thorst has joined #openstack-nova01:06
*** thorst has quit IRC01:06
*** litao__ has joined #openstack-nova01:06
*** Tom_ has joined #openstack-nova01:13
*** yingjun has joined #openstack-nova01:24
*** Tom_ has quit IRC01:25
*** Tom_ has joined #openstack-nova01:25
*** Tom_ has quit IRC01:30
*** coreywright has quit IRC01:32
*** shaner has quit IRC01:33
*** ijw has quit IRC01:33
*** shaner has joined #openstack-nova01:34
*** yufei has joined #openstack-nova01:35
mriedemdansmith: ok i have results in
mriedemwith your change01:38
dansmithis that faster/same except for details?01:39
mriedemcompared to w/o your change, (1) GET /servers with microversion 2.1 is slightly faster01:39
mriedemGET /servers/detail with microversion is about the same, a bit faster01:39
mriedembut, GET /server/details with microversion 2.53 is slower01:39
mriedemnot a ton, but it's slower01:40
mriedem25.78 compared to 30.1001:40
mriedembut, it's not a huge different01:40
dansmithoh only detail with the later microversion01:40
mriedemsomething about >2.1 always makes listing with details slower01:40
dansmithand there's some fault handling behavior difference?01:40
mriedemat least because of the joins on the (1) services table and (2) tags table01:40
mriedemi don't think there is any fault handling behavior differences with microversion >2.101:41
*** mingyu has quit IRC01:41
mriedemif there was, that might explain it01:41
dansmithokay I thought you were saying there was01:41
dansmithI dunno why because I'm pre-joining it when we were loading them separate01:41
*** mingyu has joined #openstack-nova01:41
mriedemthe only other joins i can think of right now with microversion >2.1 is on the services table (2.16) and tags able (2.26)01:41
mriedemstill, it's a difference of about 4 seconds, which isn't huge here01:42
dansmithso aside from fault, there's no difference in what I'm doing vs what we do currently,01:42
dansmithother than we're not serializing the queries01:42
*** avolkov has quit IRC01:43
dansmithwithout my change we issue the cell0 one and then the cell1 one, where now we're doing both at once01:43
dansmithis this a devstack vm on your laptop or something better?01:43
mriedemit's in a vexxhost vm01:43
mriedemthe fault stuff is the only major difference i can think of, since we'll be joining on fault all the time, rather than just for instances in ERROR state01:44
mriedemmaybe that is equaling things out somehow, idk, like if i had 1000 all in ERROR state before/after your change, that might be different in favor of yours01:45
dansmithhmm, yeah, I guess maybe that might be it01:45
*** chyka has quit IRC01:46
mriedemi do have the numbers from yesterday before your change with 1000 ACTIVE,01:46
mriedemso tomorrow i could run yours through with all active and see if there is a bigger difference because of the fault join01:46
dansmithwell, I guess we could go back to the not automatic loading of fault01:46
*** crushil has joined #openstack-nova01:46
mriedemi'll run that all active scenario tomorrow to see if it could be the fault stuff,01:46
mriedemit's nearly 9pm so i'm not going to do it tonight01:46
dansmiththere was something the API was doing that made it seem way better to do this than what it was doing01:47
dansmithbut it's been a while now01:47
dansmithwe could also plumb the logic of when to load the fault into the lower layers01:48
mriedemyup i was thinking that too01:49
mriedemanother thing that might be causing the microversion bloat, is maybe the microversion to pull the embedded flavor out of the instance01:49
mriedemadded in pike01:49
dansmithyou could run through each microversion and see where the spike is01:50
*** coreywright has joined #openstack-nova01:50
dansmiththe sorting layer on top of this really has nothing to do with what we're sorting though01:50
dansmithit doesn't make any more copies of things, nor iterate the list more times01:50
dansmithso, the change right before the switchover should do the fault loading but not the sorting, so you could run against that and see if it's more like the earlier or more like the later01:51
*** chyka has joined #openstack-nova01:52
mriedem ?01:53
mriedemlike, revert that on top of the change that uses the new code in the API01:53
mriedemoh, nvm,01:54
dansmithoh, I guess you were running on master already?01:54
mriedemyes, new devstack as of today01:54
dansmithyeah, okay01:54
*** Apoorva has joined #openstack-nova01:54
mriedemso 2.16 makes us join on services, 2.26 makes us join on tags, 2.47 returns instance.flavor, and your change always joins on faults01:55
mriedem2.47 is suspicious01:55
mriedemsince that's from instance_extra01:55
dansmithbut again, it shouldn't be any different01:55
mriedemyeah, nvm, we also didn't start loading that in the api as of 2.47, we already pulled out instance.flavor to get the link stuff01:57
*** hongbin has joined #openstack-nova01:58
mriedemtotally unrelated, but when we lazy-load instance.flavor, we're still joining on system_metadata now, we should be able to stop doing that01:59
*** takashin has quit IRC02:02
mriedemok, i'll run through with 1000 active instances tomorrow with your change and see if that makes a big difference, and if so, it could be the fault thing02:04
*** thorst has joined #openstack-nova02:07
*** Tom_ has joined #openstack-nova02:24
*** Tom__ has joined #openstack-nova02:27
*** gbarros has quit IRC02:28
*** yushb has joined #openstack-nova02:28
*** Tom___ has joined #openstack-nova02:29
*** Tom_ has quit IRC02:29
yushbJOIN #openstack-karbor02:29
*** yushb has left #openstack-nova02:30
*** Tom__ has quit IRC02:31
*** vladikr has quit IRC02:31
*** vladikr has joined #openstack-nova02:32
*** yushb has joined #openstack-nova02:32
*** esberglu has joined #openstack-nova02:42
*** acormier has joined #openstack-nova02:46
*** esberglu has quit IRC02:47
*** erlon has quit IRC02:48
*** thorst has quit IRC02:48
*** mingyu has quit IRC02:48
*** acormier_ has quit IRC02:49
*** csuttles has quit IRC02:52
*** csuttles has joined #openstack-nova02:53
*** csuttles has quit IRC02:54
*** csuttles has joined #openstack-nova02:55
*** acormier has quit IRC02:56
dansmithmriedem: I don't auto-join fault until:
dansmithso I don't think it's the fault03:01
*** pino has joined #openstack-nova03:04
*** mingyu has joined #openstack-nova03:06
*** tojuvone has quit IRC03:08
*** itlinux has joined #openstack-nova03:12
*** ijw has joined #openstack-nova03:12
*** vladikr has quit IRC03:13
*** vladikr has joined #openstack-nova03:13
*** ijw has quit IRC03:17
*** nicolasbock has quit IRC03:21
*** TuanLA has joined #openstack-nova03:21
*** mingyu has quit IRC03:23
*** yufei has quit IRC03:27
*** tojuvone has joined #openstack-nova03:27
*** yushb has quit IRC03:29
*** yangyapeng has quit IRC03:32
*** yangyapeng has joined #openstack-nova03:32
*** gyee has quit IRC03:33
openstackgerritMichael Still proposed openstack/nova master: Move ploop commands to privsep.
openstackgerritMichael Still proposed openstack/nova master: Read from console ptys using privsep.
openstackgerritMichael Still proposed openstack/nova master: Don't shell out to mkdir, use ensure_tree()
openstackgerritMichael Still proposed openstack/nova master: Cleanup mount / umount and associated rmdir calls
openstackgerritMichael Still proposed openstack/nova master: Move lvm handling to privsep.
openstackgerritMichael Still proposed openstack/nova master: Move shred to privsep.
openstackgerritMichael Still proposed openstack/nova master: Move xend existence probes to privsep.
openstackgerritMichael Still proposed openstack/nova master: Move the idmapshift binary into privsep.
openstackgerritMichael Still proposed openstack/nova master: Move loopback setup and removal to privsep.
openstackgerritMichael Still proposed openstack/nova master: Move nbd commands to privsep.
openstackgerritMichael Still proposed openstack/nova master: Move kpartx calls to privsep.
openstackgerritMichael Still proposed openstack/nova master: Move blkid calls to privsep.
*** yangyapeng has quit IRC03:37
*** itlinux has quit IRC03:41
*** thorst has joined #openstack-nova03:45
*** Sukhdev has joined #openstack-nova03:46
*** Tom___ has quit IRC03:49
*** Tom has joined #openstack-nova03:50
*** Atom1234 has joined #openstack-nova03:51
*** vladikr has quit IRC03:52
*** owalsh_ has joined #openstack-nova03:52
*** yangzhenyu has quit IRC03:52
*** diga has joined #openstack-nova03:53
openstackgerritMerged openstack/nova master: Add slowest command to tox.ini
*** Tom has quit IRC03:55
*** owalsh has quit IRC03:56
*** crushil has quit IRC03:57
*** vladikr has joined #openstack-nova04:00
*** krtaylor has quit IRC04:00
*** krtaylor has joined #openstack-nova04:02
*** ratailor has joined #openstack-nova04:08
*** krtaylor has quit IRC04:08
*** thorst has quit IRC04:16
*** jaosorior has joined #openstack-nova04:19
*** Apoorva_ has joined #openstack-nova04:21
*** hongbin has quit IRC04:21
*** Atom1234 has quit IRC04:21
*** kashyap` has joined #openstack-nova04:22
*** mhenkel_ has joined #openstack-nova04:23
*** spotz_ has joined #openstack-nova04:24
*** migi_ has joined #openstack-nova04:26
*** mdnadeem has joined #openstack-nova04:26
*** csuttles_ has joined #openstack-nova04:27
*** Sukhdev has quit IRC04:29
*** csuttles has quit IRC04:29
*** Apoorva has quit IRC04:29
*** hemna_ has quit IRC04:29
*** mriedem has quit IRC04:29
*** xinliang has quit IRC04:29
*** danpawlik has quit IRC04:29
*** cburgess has quit IRC04:29
*** spotz has quit IRC04:29
*** jcook has quit IRC04:29
*** sambetts|afk has quit IRC04:29
*** ujjain- has quit IRC04:29
*** mhenkel has quit IRC04:29
*** s1061123 has quit IRC04:29
*** ericyoung has quit IRC04:29
*** migi has quit IRC04:29
*** stephenfin has quit IRC04:29
*** obre has quit IRC04:29
*** kashyap has quit IRC04:29
*** gryf has quit IRC04:29
*** stephenfin has joined #openstack-nova04:30
*** s1061123 has joined #openstack-nova04:30
*** sambetts_ has joined #openstack-nova04:30
*** cburgess has joined #openstack-nova04:30
*** ericyoung has joined #openstack-nova04:30
*** claudiub has joined #openstack-nova04:30
*** ujjain has joined #openstack-nova04:31
*** ujjain has quit IRC04:31
*** ujjain has joined #openstack-nova04:31
*** jcook has joined #openstack-nova04:32
*** john5223_ has quit IRC04:32
*** slagle has quit IRC04:34
*** ansiwen has quit IRC04:34
*** mdbooth has quit IRC04:34
*** crushil has joined #openstack-nova04:34
*** udesale has joined #openstack-nova04:36
*** xinliang has joined #openstack-nova04:36
*** obre has joined #openstack-nova04:36
*** gryf has joined #openstack-nova04:37
*** hemna_ has joined #openstack-nova04:37
*** danpawlik has joined #openstack-nova04:37
*** manasm has joined #openstack-nova04:39
*** Apoorva_ has quit IRC04:40
*** sree has joined #openstack-nova04:40
*** slagle has joined #openstack-nova04:41
*** armax has joined #openstack-nova04:42
*** Shunli has quit IRC04:42
*** pino has quit IRC04:43
*** vladikr has quit IRC04:46
*** psachin has joined #openstack-nova04:47
*** mdbooth has joined #openstack-nova04:48
*** ansiwen has joined #openstack-nova04:48
*** vladikr has joined #openstack-nova04:50
*** vladikr has quit IRC04:56
*** acormier has joined #openstack-nova04:56
*** vladikr has joined #openstack-nova04:59
*** acormier has quit IRC05:00
*** chyka has quit IRC05:11
*** zhurong has joined #openstack-nova05:13
*** thorst has joined #openstack-nova05:13
*** vladikr has quit IRC05:14
*** vladikr has joined #openstack-nova05:17
*** phuongnh has quit IRC05:19
*** sridharg has joined #openstack-nova05:21
*** crushil has quit IRC05:21
*** vladikr has quit IRC05:22
*** lajoskatona has joined #openstack-nova05:23
*** esberglu has joined #openstack-nova05:25
*** Atom1234 has joined #openstack-nova05:26
*** Tom has joined #openstack-nova05:28
*** esberglu has quit IRC05:29
*** trinaths1 has joined #openstack-nova05:31
*** trinaths1 has left #openstack-nova05:31
*** Eran_Kuris has joined #openstack-nova05:38
*** elod has joined #openstack-nova05:43
*** avolkov has joined #openstack-nova05:46
*** thorst has quit IRC05:47
*** Tom has quit IRC05:49
*** ltomasbo has quit IRC05:49
*** markmc has quit IRC05:50
*** Atom1234 has quit IRC05:50
*** Tom has joined #openstack-nova05:51
*** Atom1234 has joined #openstack-nova05:51
*** jpena|off has quit IRC05:52
openstackgerritjichenjc proposed openstack/nova master: check query param for used_limits function
*** Oku_OS-away is now known as Oku_OS06:10
*** avolkov has quit IRC06:13
*** avolkov has joined #openstack-nova06:14
*** liuyulong has joined #openstack-nova06:16
*** armax has quit IRC06:19
lennybHi, I am working on devstack master, and my n-cond-cell1.service got stucked during stop #link  it's log is empty, no exceptions in other logs. Any tips/ideas will be appreciated.06:21
*** ltomasbo has joined #openstack-nova06:21
*** Tom has quit IRC06:26
*** edand has joined #openstack-nova06:30
*** NehaAlhat has joined #openstack-nova06:34
*** markmc has joined #openstack-nova06:35
*** moshele has joined #openstack-nova06:36
*** andreas_s has joined #openstack-nova06:36
*** neha_alhat has quit IRC06:36
*** Atom1234 has quit IRC06:36
openstackgerritjichenjc proposed openstack/nova master: check query param for server groups function
*** moshele has quit IRC06:38
openstackgerritYikun Jiang proposed openstack/nova master: Update Instance action's updated_at when action event updated.
*** rcernin has joined #openstack-nova06:39
*** armax has joined #openstack-nova06:41
*** yamamoto_ has quit IRC06:41
*** thorst has joined #openstack-nova06:45
*** yamamoto has joined #openstack-nova06:45
*** cfriesen_ has quit IRC06:46
*** Tom has joined #openstack-nova06:47
*** armax has quit IRC06:50
openstackgerritLajos Katona proposed openstack/nova master: Moving more utils to ServerResourceAllocationTestBase
openstackgerritLajos Katona proposed openstack/nova master: factor out compute service start in ServerMovingTest
openstackgerritLajos Katona proposed openstack/nova master: Test resource allocation during soft delete
openstackgerritAlex Xu proposed openstack/nova-specs master: Add trait support in the allocation candidates API
*** sshwarts has joined #openstack-nova06:59
*** yamamoto_ has joined #openstack-nova07:04
*** yamamoto has quit IRC07:04
*** sree has quit IRC07:06
*** sree has joined #openstack-nova07:07
openstackgerritYikun Jiang proposed openstack/nova-specs master: Add pagination and changes since filter support for os-instance-action API
*** yamamoto_ has quit IRC07:10
*** sree has quit IRC07:11
*** bhagyashri_s has joined #openstack-nova07:11
*** ps_jadhav has joined #openstack-nova07:11
*** neha_alhat has joined #openstack-nova07:11
*** tssurya has joined #openstack-nova07:13
*** neha_alhat has quit IRC07:13
*** bhagyashri_s has quit IRC07:13
*** esberglu has joined #openstack-nova07:13
*** sree has joined #openstack-nova07:13
*** NehaAlhat has quit IRC07:13
*** bhagyashris has quit IRC07:14
*** pooja_jadhav has quit IRC07:14
*** ps_jadhav has quit IRC07:14
*** Dinesh_Bhor has quit IRC07:17
*** thorst has quit IRC07:17
*** esberglu has quit IRC07:17
*** phuongnh has joined #openstack-nova07:18
*** yamamoto has joined #openstack-nova07:18
openstackgerritYikun Jiang proposed openstack/nova-specs master: Add pagination and changes since filter support for os-instance-action API
*** neha_alhat has joined #openstack-nova07:20
*** bhagyashris has joined #openstack-nova07:20
*** pooja_jadhav has joined #openstack-nova07:21
*** Dinesh_Bhor has joined #openstack-nova07:21
*** tesseract has joined #openstack-nova07:21
openstackgerritZhenyu Zheng proposed openstack/nova stable/pike: Mention API behavior change when over quota limit
*** jaosorior has quit IRC07:31
*** moshele has joined #openstack-nova07:37
*** ragiman has joined #openstack-nova07:39
*** manasm has quit IRC07:40
*** manasm has joined #openstack-nova07:41
*** pooja-jadhav has joined #openstack-nova07:46
*** jaosorior has joined #openstack-nova07:46
*** jpena has joined #openstack-nova07:47
*** ijw has joined #openstack-nova07:47
*** pooja_jadhav has quit IRC07:48
*** tetsuro has joined #openstack-nova07:52
*** alexchadin has joined #openstack-nova07:53
*** kashyap` is now known as kashyap07:57
openstackgerritZhenyu Zheng proposed openstack/nova master: nova-manage db archive_deleted_rows is not multi-cell aware
*** ijw has quit IRC08:02
*** yamahata has joined #openstack-nova08:02
*** ralonsoh has joined #openstack-nova08:04
*** mingyu has joined #openstack-nova08:04
*** Tom has quit IRC08:06
*** efried has quit IRC08:08
*** zhurong has quit IRC08:11
*** thorst has joined #openstack-nova08:14
*** Tom has joined #openstack-nova08:16
*** efried has joined #openstack-nova08:19
*** Tom has quit IRC08:20
*** owalsh_ is now known as owalsh08:24
*** brault has quit IRC08:24
*** baoli has joined #openstack-nova08:28
*** yamahata has quit IRC08:29
*** Tom has joined #openstack-nova08:29
*** zhurong has joined #openstack-nova08:30
gibijohnthetubaguy: hi! do you have any comments on ? mikal has already stated that rackspace private cloud is not using it
*** baoli has quit IRC08:32
*** mingyu has quit IRC08:40
*** mingyu has joined #openstack-nova08:41
*** alexchadin has quit IRC08:41
*** alexchadin has joined #openstack-nova08:42
*** derekh has joined #openstack-nova08:47
*** alexchadin has quit IRC08:48
openstackgerritBalazs Gibizer proposed openstack/nova master: Remove dead code of api.fault notification sending
*** thorst has quit IRC08:50
*** hferenc has joined #openstack-nova08:54
ralonsohdansmith: hi, can I talk about
*** udesale has quit IRC08:57
*** udesale has joined #openstack-nova08:57
johnthetubaguygibi: I am very far removed from if that would be used now08:58
johnthetubaguygibi: I believe the context around that was providing an SLA that involved knowing about the 500 errors in the API, and doing bug fixes to reduce them08:59
johnthetubaguygibi: i.e. the SLA is related to the % of 5xx errors from the API, so errors in the API are really important to those folks08:59
johnthetubaguygibi: I don't remember using those notifications myself, mostly got that data from ELK when I was last looking at that stuff09:00
*** esberglu has joined #openstack-nova09:01
johnthetubaguygibi: tl;dr +1 on killing it09:01
gibijohnthetubaguy: thanks for the info09:02
johnthetubaguygibi: np09:02
stephenfingibi: Super easy docs patch needing another +2 here, if you're twiddling your thumbs at any point today :)
stephenfinWell, kinda easy09:04
gibistephenfin: I will look shortly09:04
stephenfinno rush09:04
*** claudiub|2 has joined #openstack-nova09:05
*** esberglu has quit IRC09:06
*** claudiub|3 has joined #openstack-nova09:06
*** alexchadin has joined #openstack-nova09:07
*** alexchadin has quit IRC09:08
*** claudiub has quit IRC09:08
*** alexchadin has joined #openstack-nova09:08
*** claudiub|2 has quit IRC09:10
*** alexchadin has quit IRC09:11
*** hferenc has quit IRC09:13
*** yamamoto has quit IRC09:19
*** tssurya has quit IRC09:22
*** sambetts_ is now known as sambetts09:27
openstackgerritMoshe Levi proposed openstack/nova master: Don't overwrite binding-profile
*** brault has joined #openstack-nova09:33
*** mvk has quit IRC09:34
*** josecastroleon has joined #openstack-nova09:41
gibistephenfin: left some comments in
stephenfingibi: On it. Thanks :)09:42
gibistephenfin: sorry for being picky. I can accept if most of my comments are fixed in a followup09:42
gibistephenfin: the only thing that I think is a must
*** thorst has joined #openstack-nova09:47
gibistephenfin: if you need something to review then I can suggest this test improvement series :)09:47
*** afazekas is now known as afazekas|seek4fo09:48
openstackgerritZhenyu Zheng proposed openstack/nova master: nova-manage db archive_deleted_rows is not multi-cell aware
*** sdague has joined #openstack-nova09:50
*** Atom1234 has joined #openstack-nova09:52
*** gszasz has joined #openstack-nova09:55
*** diga has quit IRC09:55
*** diga has joined #openstack-nova09:57
*** esberglu has joined #openstack-nova10:00
*** edand has quit IRC10:00
*** josecastroleon has quit IRC10:00
*** diga has quit IRC10:04
*** esberglu has quit IRC10:04
*** yingjun has quit IRC10:05
*** TuanLA has quit IRC10:06
*** pooja-jadhav is now known as pooja_jadhav10:11
openstackgerritStephen Finucane proposed openstack/nova master: doc: Rework man pages
*** tetsuro has quit IRC10:16
*** thorst has quit IRC10:18
*** jichen has quit IRC10:19
*** Tom has quit IRC10:19
*** zhurong has quit IRC10:20
*** Tom has joined #openstack-nova10:20
*** yamamoto has joined #openstack-nova10:20
*** Tom has quit IRC10:21
*** rmart04 has joined #openstack-nova10:25
*** yamahata has joined #openstack-nova10:27
*** manasm has quit IRC10:30
*** manasm has joined #openstack-nova10:32
*** andreas_s_ has joined #openstack-nova10:37
*** andreas_s has quit IRC10:40
*** syqian has joined #openstack-nova10:44
*** andreas_s_ has quit IRC10:45
*** edand has joined #openstack-nova10:47
*** diga has joined #openstack-nova10:47
*** andreas_s has joined #openstack-nova10:47
*** mvk has joined #openstack-nova10:48
*** udesale has quit IRC10:52
openstackgerritLee Yarwood proposed openstack/python-novaclient stable/newton: Fix aggregate_update name and availability_zone clash
*** smatzek_ has joined #openstack-nova10:52
*** yassine has quit IRC10:55
*** Atom1234 has quit IRC10:57
*** syqian has quit IRC10:57
*** hemna_ has quit IRC10:57
*** yassine has joined #openstack-nova10:58
*** mingyu has quit IRC11:00
*** nicolasbock has joined #openstack-nova11:00
*** nicolasbock has quit IRC11:04
alex_xugmann: sorry, I can't make the api meeting today, have cold today, headache11:06
*** jaosorior is now known as jaosorior_sick11:12
*** cdent has joined #openstack-nova11:13
*** thorst has joined #openstack-nova11:15
*** phuongnh has quit IRC11:18
*** diga has quit IRC11:23
*** acormier has joined #openstack-nova11:24
*** _pewp_ has quit IRC11:24
*** mingyu has joined #openstack-nova11:26
*** thorst has quit IRC11:27
cdentefried: you on the scene yet?11:27
*** pcaruana has joined #openstack-nova11:27
*** udesale has joined #openstack-nova11:28
cdentefried: when you are, I’ve got a few questions about non-openstack managed workloads11:28
*** liuyulong has quit IRC11:30
*** _pewp_ has joined #openstack-nova11:31
*** tylerderosagrund has joined #openstack-nova11:39
openstackgerritLajos Katona proposed openstack/nova master: Change live_migrate tests to use fakedriver
*** dave-mccowan has joined #openstack-nova11:50
*** acormier has quit IRC11:53
*** trinaths has joined #openstack-nova11:54
*** jpena is now known as jpena|lunch11:54
*** thorst has joined #openstack-nova11:55
manasmbauzas: you may want to take a look at .11:57
openstackLaunchpad bug 1719859 in OpenStack Compute (nova) "Resize failure due to instance group being None in request spec" [Undecided,New]11:57
*** tylerderosagrund has quit IRC11:57
*** sahid has joined #openstack-nova12:01
*** dave-mccowan has quit IRC12:01
*** edand has quit IRC12:01
*** dave-mccowan has joined #openstack-nova12:04
*** tssurya has joined #openstack-nova12:04
*** litao__ has quit IRC12:04
*** chyka has joined #openstack-nova12:07
*** afazekas|seek4fo is now known as afazekas12:09
*** chyka has quit IRC12:11
*** acormier has joined #openstack-nova12:15
*** smatzek_ is now known as smatzek12:16
*** dtantsur|afk is now known as dtantsur12:16
*** alexchadin has joined #openstack-nova12:16
*** edmondsw has joined #openstack-nova12:17
*** pino has joined #openstack-nova12:18
*** vladikr has joined #openstack-nova12:23
*** acormier has quit IRC12:29
*** moshele has quit IRC12:29
*** gouthamr has joined #openstack-nova12:30
*** yangyapeng has joined #openstack-nova12:34
openstackgerritYikun Jiang proposed openstack/nova master: Update Instance action's updated_at when action event updated.
*** edand has joined #openstack-nova12:35
*** gouthamr_ has joined #openstack-nova12:36
*** pchavva has joined #openstack-nova12:37
*** yangyapeng has quit IRC12:38
*** gouthamr has quit IRC12:38
*** pino has quit IRC12:39
*** manasm has quit IRC12:40
gmannalex_xu: oh, i also do not have much for this week. we can cancel the meeting12:42
gmannalex_xu: take rest and get well soon12:42
*** trinaths has left #openstack-nova12:43
*** gcb has quit IRC12:45
*** jmlowe has quit IRC12:48
tssuryaHi, I was just playing around with cell_v2 and came across the nova-manage cell_v2 map_instances command; so I went through the source code ( but was not able to find a connection as to which sql_connection string is used to query the 'instances' table in the nova DB. It should be taking this from the conf file ? But could someone point me to the section of code that does this ?12:53
*** alexchadin has quit IRC12:55
*** mingyu has quit IRC12:55
*** alexchadin has joined #openstack-nova12:56
*** jpena|lunch is now known as jpena12:57
*** mingyu has joined #openstack-nova12:57
*** takashin has joined #openstack-nova12:58
*** esberglu has joined #openstack-nova12:58
sdaguetssurya: it's coming from the same nova.conf that your API server is expected to be using13:00
*** ratailor has quit IRC13:01
sdaguetssurya: all nova-manage commands are assumed to be run from the shell on the API server (and have the same db access)13:01
*** eharney has joined #openstack-nova13:01
openstackgerritzhangyangyang proposed openstack/nova master: Move libvirts qemu-img support to privsep.
*** liverpooler has joined #openstack-nova13:03
tssuryasdague : thanks for the reply, so here is my situation : I have some instances lying in my nova_cell1 DB which need to be mapped to my new cell (cell1). however the db access URL for this is inside nova_cell1.conf ? so when I run the map instances command, they do not get mapped.13:05
*** smatzek has quit IRC13:05
tssuryasdague: I think its because the queries are sent to the nova_cell0 DB since nova.conf has that access url.13:06
efriedcdent Yo13:07
openstackgerritzhangyangyang proposed openstack/nova master: Move libvirts qemu-img support to privsep
cdentefried: yo13:08
*** mriedem has joined #openstack-nova13:09
cdentefried: my question is basically: in powervm does the thing which is the actual hypervisor ever host workloads that are not managed by nova. The comments on this bug for context:
openstackLaunchpad bug 1718212 in OpenStack Compute (nova) "Compute resource tracker does not report correct information for drivers such as vSphere" [Medium,In progress] - Assigned to Radoslav Gerganov (rgerganov)13:09
cdentsince placement is authoritative for dynamic workloads if nova is not doing all the managing, there may be need for additional things talking to placement13:10
efriedcdent That's a multi-pronged question.13:10
cdentIt’s a full on dinner fork13:10
openstackgerritSean Dague proposed openstack/nova master: Support qemu >= 2.10
efriedFirst, nothing is stopping the user from creating VMs outside the auspices of OpenStack.13:11
*** jistr is now known as jistr|call13:12
efriedSecond, there's a mode (the most common one, historically) where the I/O virtualization is done by one (or two, cause redundancy/HA) separate partitions, in which case those guys are consuming resources outside of Nova's purview.13:12
efriedcdent Now I'll look at the bug...13:13
bauzasefried: creating VMs by the hypervisor directly is not supported by Nova13:13
*** pino has joined #openstack-nova13:13
*** moshele has joined #openstack-nova13:13
*** krtaylor has joined #openstack-nova13:13
efriedbauzas Oh, Nova won't pick them up, for sure.  But there's nothing stopping the user from doing it.13:14
cdentefried: So, short answer is “yes” so second question is “Do you yet have a plan on how to manage it”13:14
*** lucasxu has joined #openstack-nova13:14
bauzasefried: if the operator does that, then any bug related to that should be Wontfix13:14
*** lyan has joined #openstack-nova13:14
efriedbauzas Well...13:14
efriedI agree in principle.13:14
bauzasefried: lemme find where we say that we don't support direct hypervisor calls13:14
efriedBut that doesn't mean we won't try to accomodate out-of-band partitions when reporting inventory/usage.13:15
openstackgerritMatt Riedemann proposed openstack/nova master: Log consumer uuid when retrying claims in the scheduler
efriedbauzas Sure, but I totally believe you.13:15
openstackgerritAndrey Volkov proposed openstack/nova master: [WIP] List instances performace optimization
*** MVenesio has joined #openstack-nova13:16
bauzasefried: I didn't found any explanations either in or
bauzasmaybe we should say that13:17
efriedcdent Okay, in general, we have very similar issues as described in comment #6 in that bug.13:17
*** chyka has joined #openstack-nova13:18
efriedWhen I started looking into converting over to get_inventory, it was going to be a matter of reporting "reserved" amounts based on whatever's going on OOB.13:18
johnthetubaguyI thought we were heading down the not doing live updates of resource usage?13:18
cdentefried: do you have a convenient way of distinguishing between nova managed and not-nova managed stuff?13:18
efried...and it's not trivial to figure out what's OOB.13:19
efried(I was just about to say :)13:19
efriedI can know off the bat to account for my Novalink partition (the node on which the compute service runs) and my Virtual I/O Servers.13:19
cdentjohnthetubaguy: that seems to work okay for libvirt, but not so great otherwise13:19
efriedBut any plain ol' worker bees I have to figure out whether they came from Nova.13:20
johnthetubaguycdent: I guess I am not seeing why, is that vmware bug the example?13:20
openstackgerritRodolfo Alonso Hernandez proposed openstack/os-vif master: Migrate from 'ip' commands to 'pyroute2'
cdentjohnthetubaguy: at this stage efried, rgerganov and I are just sort of having a chat, not making any decisions13:20
bhagyashrisjohnthetubaguy, mriedem: Hi,13:20
efriedSimplest would be, every time get_inventory/get_available_resource is called, I ask Nova for the full list of instances it knows about, ask my hypervisor for the instances IT knows about, and do a set-diff.13:21
johnthetubaguyI am just curious where the problem isn't nova driver expected resource usage13:21
cdentjohnthetubaguy: that’s a specific case of a general issue of “things other than the nova-compute node using the stuff that nova-compute is also using”13:21
efriedjust so.13:21
johnthetubaguyI think I need a more concrete example13:21
mriedemcdent: i think the vcenter vms have some nova metadata associated with them, so you can tell which ones are nova-managed and which were created oob13:22
efriedjohnthetubaguy You try to spawn an instance and ask for 3 VCPU.  Resource tracker reports you've got 3 VCPU, so the claim passes.13:22
efriedjohnthetubaguy But in fact, one VCPU is being consumed by a VM that was spawned out of band13:22
rgerganovmriedem, that may work for vcpu and memory but it won't work well for storage13:22
*** chyka has quit IRC13:22
johnthetubaguyits the out of band VMs, I thought we explicitly didn't support that13:22
efriedjohnthetubaguy Yeah, bauzas said the same, but didn't find where that's documented.13:23
efriedNot saying that means it's supported, or that it should be :)13:23
bauzasso, Nova isn't a proxy layer for hypervisors13:23
johnthetubaguyyeah, I could have swarn it was in here, but I don't see it:
bauzasjohnthetubaguy: yeah, I verified that13:24
bauzaslemme provide a change for it13:24
johnthetubaguywell, its a bit late I guess13:24
cdentSo, even if the statement is “we don’t do that” the problem still holds for shared storage13:25
mriedemnova doesn't import existing vms on the hypervisor,13:25
mriedembut that doesn't mean we don't try to adjust inventory based on things running on the hypervisor host13:25
cdentwhere “the problem” is the generic notion of mixed accounting13:25
efriedmriedem Which actually makes it more problematic.13:25
johnthetubaguycdent: yeah, that sounds like a real thing we have to support13:25
mriedemthat's why we have reserved13:25
mriedemeven libvirt hosts have to account for things like ovs running on the same host13:26
cdentRight, so one of the underlying questions is:13:26
cdentDo we intend/expect that reserved will by dynamically adjuted, frequently13:26
*** baoli has joined #openstack-nova13:26
cdentOr in the cases where we want it to be dynamically adjusted we should instead make allocations, via some third party?13:26
efriedRight, back to what johnthetubaguy mentioned earlier: "not doing live updates of resource usage"13:26
efriedThat is, will get_inventory() eventually be a thing that's run only once, rather than on a periodic?13:27
bhagyashrisjohnthetubaguy, mriedem: Could you please review patch: ? Addressed all review comments. Thank you :)13:27
*** lbragstad has joined #openstack-nova13:27
mriedemi don't think it will no13:27
bauzasI'm not seeing the reserved bit to be that dynamicv13:27
mriedemthe operator can adjust reserved space dynamically13:27
johnthetubaguycdent: I separate allocations are easier to update than a single reserved value13:27
efriedI've heard tell (possibly from jay) that it's a no-no to update the *total* amounts.13:27
mriedemwe talked about this in boston13:27
*** sbezverk has joined #openstack-nova13:27
bauzasbut yeah, the operator can just set or raise the reserved bit when they want13:27
mriedemabout how we don't account for overhead, and operators would have to handle that by toggling reserved values13:27
cdentmriedem: yes, but the new ingredient here is the dynamism13:28
bauzasmriedem: do you think we should clearly state that in ?13:28
efried...unless, like, they hot-plug a whole new disk, or jack up the disk size on the SAN or whatever.  In that case, we should be able to bump the totals, yes?13:28
bauzasthe fact that Nova doesn't support direct hypervisor VMs, but can leave room for them13:28
cdentefried: yes13:28
efriedSame actually applies to CPUs13:28
efriedNot sure if more primitive hypervisors have this, but there's a dynamic entitlement thing where you can unlock previously-unavailable CPUs on the fly.13:29
*** mingyu has quit IRC13:29
johnthetubaguybhagyashris: there was a -1 review on that patch with no answer when I last looked, so I skipped looking any deeper, did you respond to their questions yet?13:29
*** josecastroleon has joined #openstack-nova13:30
johnthetubaguymriedem: it totally feels like the toggling reserved values ticks the 80% case, with very few surpizes13:30
*** takashin has quit IRC13:30
cdentefried, mriedem, bauzas, johnthetubaguy, rgerganov: it feels like we probably have the tooling to make something workable for this, but as we figure it out it would be good to document some kind of (hate this phrase) best practice,13:31
efriedMe, I'm concerned about the penalty of getting the full list of instances every time get_inventory is run.13:31
johnthetubaguyits feels like making the 80% case work really well should be step 1? And some rough best practices for the other bits for now, that could work.13:32
cdentefried: instances from the hypervisor’s point of view, nova’s point of view or placement’s point of view? or some combination? That is, which one worries you most?13:33
johnthetubaguyhonestly doing both of those worry me13:33
efriedYeah, I have to do both.13:33
johnthetubaguyI like the system doing nothing when its idle, in an ideal world of course.13:33
*** sridharg has quit IRC13:33
bauzaswe shouldn't really create a way where people would directly call their VMs if they wish, because that would mean we would silently say we support that13:33
efriedCause the alternative is maintaining some kind of cache and trying to keep it up to date every time a VM gets spawned or deleted (I can get notifications for the OOB ones, and update the in-band ones from spawn/destroy).13:34
bhagyashrisjohnthetubaguy: Actually I have not respond yet, but Andrey Volkov is given -1 and he suggested that we can fix this issue at schema level (same thing I have proposed in the patch set 3) but after some discussion we decided to fix this in logic and at schema level.13:34
bauzasif the operator doesn't know the dedicated amount their wanna reserve, then either they should disable some compute from the pool, or just set a high value13:34
*** gbarros has joined #openstack-nova13:34
*** jmlowe has joined #openstack-nova13:34
johnthetubaguybut isn't the easy case much easier here? deployer sets the reserved values, no period updates, worry about special cases later?13:35
bauzasproviding a way to dynamically adjust resources in Nova would just mean we create things to support13:35
cdentjohnthetubaguy, bauzas: perhaps sad, but true, there are in tree hypervisors where oob vms are fairly common. nova in itself may not need to support that, but the hypervisors will likely continue doing it13:35
bhagyashriss/and at schema level/and not at schema level13:35
bauzasjohnthetubaguy: that's my point13:35
dansmithmriedem: have you seen the stuff I added to the etherpad last night?13:36
bauzasjohnthetubaguy: if operators really want a space for out-of-band VMs, then they have the reserved bit they can set and try to modify when they feel necessary13:36
bhagyashrisjohnthetubaguy: i will respond to that question.13:36
johnthetubaguycdent: but even those drivers, there are folks who run dedicated clusters just for OpenStack13:36
bauzasbut trying to provide some mechanism for having a dynamic reserved bit seems to me something we shouldn't do13:36
bauzasand honestly, wouldn't it be better to just disable a host and play with it, if that's really needed ?13:37
cdentjohnthetubaguy: yes, totally13:37
johnthetubaguycdent: that was me arguing for making the sync optional, for when you know Nova owns the world13:37
* cdent nods13:37
rgerganovjohnthetubaguy, most users runs dedicated clusters but also used shared storage13:37
dansmithjohnthetubaguy: are you talking about the periodic that will try to delete unknown vms?13:38
mriedemdansmith: sure did13:38
johnthetubaguybhagyashris: thanks, responding in the review will help wehn I get back to that13:38
mriedemdansmith: nope, talking about update_available_resource13:38
mriedemand using get_inventory to report reserved space for other things running on the hypervisor outside of nova's control13:38
johnthetubaguydansmith: well, I was thinking about that early, with all the logging by default, but there I was meaning the resource tracker updater13:38
dansmithmriedem: eff that. there's a reserved count for that kind of stuff.13:39
bhagyashrisjohnthetubaguy: Thank you :)13:39
mriedemdansmith: meaning the config option?13:39
*** tetsuro has joined #openstack-nova13:39
dansmithmeaning counting other things and subtracting it ourselves13:39
mriedemi think the config option is what's used to report reserved inventory in the RT today13:40
dansmithwell, actually, I dunno that it is today, but it should be, and that should be your outlet to carve out space13:40
*** takashin has joined #openstack-nova13:40
*** cleong has joined #openstack-nova13:41
dansmithit is13:41
dansmithmake those hot-reloadable if you want, but that's the path, IMHO13:41
*** mingyu has joined #openstack-nova13:42
efrieddansmith That guy still gets run on a periodic basis, right?  I.e. dynamic tweaking of reserved amounts is supported (at least by the driver - not asking if those conf opts are dynamic)13:42
johnthetubaguywe should make SIG_HUP trigger a resource refresh?13:43
dansmithjohnthetubaguy: HUP already triggers a config re-read13:43
johnthetubaguyI was meaning a resource tracker update13:43
dansmithjohnthetubaguy: so if they were reloadable you'd get a new value the next time inventory runs13:44
johnthetubaguyso we don't have to keep the periodic thing, just do it on events13:44
dansmithI'm not sure there's any reason not to do it perioically13:44
johnthetubaguysystem load, all that lists of instances, etc13:44
dansmithit doesn't need to anymore13:45
dansmithin pike, once there aren't any ocata computes around we're not doing healing for instances other than deleted ones13:46
johnthetubaguyI guess I am confused13:46
* cdent gives johnthetubaguy a membership card13:46
*** lifeless has quit IRC13:46
johnthetubaguyI thought we were left with no periodic updates in queens13:47
johnthetubaguyto inventory, etc13:47
johnthetubaguyalthough... ironic needs those13:47
johnthetubaguydang it13:47
dansmithI'm saying, we can remove a bunch of the RT stuff for healing instances now I think,13:48
*** awaugama has joined #openstack-nova13:48
johnthetubaguyah, OK, so maybe I am agreeing with you13:48
dansmithwhich will make the periodic mostly an inventory mechanism, which doesn't need to be listing instances and such13:48
efriedThe periodic will use driver.get_inventory() to update inventories?13:49
dansmithor get_available_resource() if it doesn't have that13:50
dansmithor whatever that method is called13:50
*** burt has joined #openstack-nova13:50
efriedSo I think the point is that, in order to do dynamic adjustment of 'reserved' amounts properly, get_inventory itself may need to be listing instances and such.13:50
dansmithinventory has nothing to do with instances13:51
*** alexchadin has quit IRC13:51
rgerganovdansmith, there could be no way to calculate reserve without listing instances13:51
dansmithreserved is a conf value13:52
bauzasthat's my whole point13:52
dansmithyou all were talking about other vms not owned by nova on a hypervisor right? and something external adding that value to reserved so nova doesn't try to use it, right?13:52
bauzasreserved is set by the operator13:52
bauzasso, if the operator wants to raise that value, fair enough13:53
*** takashin has quit IRC13:53
dansmithin that case, some other thing could be updating CONF.reserved_whatever and HUPing nova so that the next time the periodic runs, it just stabs the new scalar value into inventory.. no counting required.13:53
bauzasbut I don't see why Nova should care of thaty13:53
dansmithbauzas: most definitely13:53
bauzasdansmith: +113:53
*** crushil has joined #openstack-nova13:53
*** lifeless has joined #openstack-nova13:54
*** Tom_ has joined #openstack-nova13:54
*** dave-mccowan has quit IRC13:54
mriedembauzas: in case you didn't see this
openstackLaunchpad bug 1719730 in OpenStack Compute (nova) pike "Reschedule after the late affinity check fails with "'NoneType' object is not iterable"" [High,Confirmed]13:55
*** udesale__ has joined #openstack-nova13:55
efriedAs user experiences go, "Extend SAN disk, edit conf file, HUP n-cpu" ain't as friendly as "Extend SAN disk".13:55
mriedembauzas: melwitt might be working on a fix, i'm not sure13:55
mriedembauzas: but it's a regression in pike13:55
bauzasmriedem: yup, I just provided my thoughts litterally 5 mins ago :p13:55
bauzasmriedem: tl;dr I don't understand how we can get a ReqSpec group info that isn't accurate13:55
rgerganovefried +113:55
dansmithefried: nova wouldn't be reporting the inventory for a SAN disk, so no problem there :)13:56
dansmithefried: and anything that an agent is surveying would be updated automatically13:56
*** udesale has quit IRC13:56
dansmithefried: if you hotplug some memory into your compute node, then nova-compute would automatically report that on the next periodic13:56
dansmithefried: what you are talking about is updating the reserved value, which is static, operator-configured or operator-scripted13:57
mriedembauzas: provided your thoughts where? don't see anything in the bug or the linked pike change13:57
dansmithefried: the HUP is so we notice the change to the config file.. certainly you don't want us to be live reloading the config file whenever we feel like it13:57
bauzasmriedem: oh snap, duplicate bug13:57
*** Tom_ has quit IRC13:58
openstackLaunchpad bug 1719859 in OpenStack Compute (nova) "Resize failure due to instance group being None in request spec" [Undecided,New]13:58
*** hemna_ has joined #openstack-nova13:58
*** Tom_ has joined #openstack-nova13:58
efrieddansmith Right, extending SAN was a bad example.13:58
bauzasmriedem: mmm, nevermind, different issue, it seems13:58
* efried is pretty good at coming up with bad examples that get us talking about the wrong part of a problem :(13:58
*** rtjure has quit IRC13:58
*** baoli has quit IRC13:58
efriedNot being super well-versed on how OVS comes into play for libvirt, can we apply the same principle to networking resources?13:59
mriedembauzas: ok, mel's is pretty straight forward13:59
*** baoli has joined #openstack-nova13:59
mriedemwe used to set group_members in the filter_properties,13:59
mriedemyour change in pike stopped using filter_properties,13:59
efriedI.e. similar UX statement: "Do OVS thingy, edit conf file, HUP n-cpu" not as friendly as "Do OVS thingy"13:59
mriedembut forgot to include the group_members stuff14:00
mriedemshe already said that the 1 line change fixes the issue14:00
mriedemi just asked that a functional regression test get created14:00
bauzasmriedem: okay, I'll wait for melwitt's patch14:00
dansmithefried: no, nova-compute isn't reporting any network resources14:00
*** ragiman has quit IRC14:00
bauzasmriedem: thanks for the catch14:00
mriedembauzas: which reminds me, did you cleanup that num_instances one?14:01
bauzasmriedem: I got some late pings yesterday14:01
mriedemmelwitt found the problem, i just helped debug14:01
dansmithefried: I think the block you're stumbling over is that nova does not count things it does not count. Therefore, it does not dynamically update inventory for things it does not count. If you want to override the inventory for a thing it _does_ count to account for things it does not count, then you put that in config an HUP it to notice.14:01
*** manasm has joined #openstack-nova14:01
bauzasmriedem: yeah, just needs rebase14:01
mriedemi need to start making an etherpad of bug fixes we need in pike...14:01
*** lajoskatona has left #openstack-nova14:01
bauzasmriedem: I also have a couple of bugfixes that are on hold in my queue14:02
bauzaslike the fact we don't destroy the Spec objects14:02
bauzasmriedem: an etherpad seems a good idea to me14:02
bauzasthat would also help me focusing on rebasing such bugs14:02
*** rtjure has joined #openstack-nova14:03
*** coreywright has quit IRC14:04
tssuryadansmith, mriedem, sdague : so basically I have a simple set up using devstack with one cell, then I added a new 'cell2' that has some unmapped instances in its DB. However when I run the map_instances command specifying the cell uuid, it does not work since like sdague said earlier, the db url is being taken from nova.conf (which is that of cell0) while the db url for the new cell is inside the cell_mappings table of nova_api db. So I14:04
*** acormier has joined #openstack-nova14:04
dansmithtssurya: create a small config with just the db url you need14:05
efrieddansmith Thanks, that makes sense.14:05
*** Drankis has joined #openstack-nova14:06
tssuryadansmith : okay! so that means the current working is as expected ?14:06
dansmithtssurya: it's been a while since I looked at that command, but I would guess so14:07
mriedemtssurya: nova.conf in default devstack points at cell0 on purpose14:07
mriedemtssurya: nova-manage uses nova.conf if no other --config-file is specified14:08
mriedemon the command line14:08
*** chyka has joined #openstack-nova14:08
mriedemso if you want to do things to cell1, you would do something like: nova-manage --config-file /etc/nova/nova_cell1.conf db archive_deleted_instances --verbose --until-complete14:08
mriedemfor archiving deleted instances in cell114:08
stephenfinefried: You didn't start work converting this to a seqdiag, did you?
openstackgerritEric Berglund proposed openstack/nova master: WIP(5): PowerVM driver: ovs vif
stephenfinI think mriedem asked for it?14:09
tssuryamriedem, dansmith : okay thanks that makes sense14:09
dansmithtssurya: yeah look at the docstring on that method, it says it assumes the config points at the right database14:09
mriedemtssurya: related
openstackLaunchpad bug 1719487 in OpenStack Compute (nova) "nova-manage db archive_deleted_rows is not multi-cell aware" [Wishlist,Triaged] - Assigned to Zhenyu Zheng (zhengzhenyu)14:09
dansmithtssurya: you could make it smarter :)14:09
openstackgerritOpenStack Proposal Bot proposed openstack/os-vif stable/pike: Updated from global requirements
mriedemstephenfin: that's already done14:09
*** sree has quit IRC14:09
efriedstephenfin Yes, merged.14:10
mriedemthere is one thing missing14:10
mriedemi noticed yesterday14:10
*** yamamoto has quit IRC14:10
mriedemon the failure path, it doesn't have a box saying that the source node is running the _rollback_live_migration method14:10
mriedemwhich does some stuff on the source node, and calls the dest node to cleanup14:10
tssuryadansmith , mriedem : ok, will have a look at it :)14:10
*** felipemonteiro has joined #openstack-nova14:10
openstackgerritOpenStack Proposal Bot proposed openstack/python-novaclient stable/pike: Updated from global requirements
efriedmriedem That would be like a box to the left of that final 'call' arrow at the bottom?14:11
*** sree has joined #openstack-nova14:11
mriedemefried: i think more like how the post_live_migration box is in the success path on the source node14:11
*** felipemonteiro_ has joined #openstack-nova14:12
*** smatzek has joined #openstack-nova14:13
* cdent goes to find some coffee to sit with sdague’s slides14:14
efriedsdague cdent Care to render an opinion/ruling on jichenjc's concerns here: ?14:14
*** smatzek has quit IRC14:14
*** smatzek has joined #openstack-nova14:15
*** ragiman has joined #openstack-nova14:15
*** yamamoto has joined #openstack-nova14:15
*** felipemonteiro has quit IRC14:16
*** sree has quit IRC14:16
*** smatzek has quit IRC14:16
*** smatzek_ has joined #openstack-nova14:16
cdentefried: remind me when I’m back in about 20 mins14:17
*** cdent has quit IRC14:17
efriedack, thx14:17
*** coreywright has joined #openstack-nova14:17
openstackgerritEric Fried proposed openstack/nova master: _rollback_live_migration in live-migration seqdiag
*** haobing has joined #openstack-nova14:18
efriedmriedem ^14:18
*** mdnadeem has quit IRC14:18
*** alex_xu has quit IRC14:18
*** cdent has joined #openstack-nova14:19
mriedemclaudiub|3: hyperv ci seems to have gone crazy14:19
mriedemnot sure if that's a zuulv3 side effect or what14:20
*** yamamoto has quit IRC14:20
*** cdent has quit IRC14:20
*** alex_xu has joined #openstack-nova14:21
openstackgerritMatt Riedemann proposed openstack/nova master: xenapi: pass migrate_data to recover_method if live migrate fails
*** smatzek_ is now known as smatzek14:22
*** haobing has quit IRC14:22
openstackgerritOpenStack Proposal Bot proposed openstack/os-vif stable/pike: Updated from global requirements
openstackgerritOpenStack Proposal Bot proposed openstack/python-novaclient stable/pike: Updated from global requirements
*** hemna_ has quit IRC14:25
*** takashin has joined #openstack-nova14:26
*** dave-mccowan has joined #openstack-nova14:27
*** tidwellr has joined #openstack-nova14:27
*** hongbin has joined #openstack-nova14:27
mriedemgibi: appears that test_live_migrate_delete has introduced a race
*** jistr|call is now known as jistr14:27
mriedemi was noticing some stuff like this yesterday when writing another test,14:28
mriedemi noticed that we set migration and instance state before we're actually done cleaning up live migration things,14:28
mriedemso tests that rely on asserting the cleanups can get racy14:28
*** takashin has quit IRC14:29
gibimriedem: I have to check if we can wait for notification or instance action to avoid the race14:30
*** jmlowe has quit IRC14:31
gibibut I guess we need a bug so I can go and file it if you haven't already filed it14:31
openstackLaunchpad bug 1719915 in OpenStack Compute (nova) "test_live_migrate_delete race fail when checking allocations: MismatchError: 2 != 1" [Medium,Confirmed]14:32
*** hemna_ has joined #openstack-nova14:34
gibimriedem: thanks, I go and dig for a solution14:34
gibimriedem: btw, I will be mostly unavailable tomorrow and on Friday. Could you do the reporting about the notification subteam meeting on the weekly nova meeting?14:36
openstackgerritOpenStack Proposal Bot proposed openstack/os-vif stable/pike: Updated from global requirements
mriedemgibi: sure14:37
gibimriedem: thanks a lot14:37
*** jmlowe has joined #openstack-nova14:37
johnthetubaguymriedem: I was looking at the requesting traits in flavors, did we ever talk about requesting the absence of a trait (like request no CPU_FLAG_X available)14:38
openstackgerritOpenStack Proposal Bot proposed openstack/python-novaclient stable/pike: Updated from global requirements
*** stvnoyes has quit IRC14:38
*** udesale__ has quit IRC14:38
*** dave-mccowan has quit IRC14:38
efriedjohnthetubaguy That's a fun thought.  Is there a real use case for it?14:39
*** josecastroleon has quit IRC14:39
johnthetubaguyefried: disable the VT CPU flag?14:39
johnthetubaguyor disable hyperthreading14:39
dansmithjohnthetubaguy: not that I'm aware of, just required and preferred14:39
*** Drankis has quit IRC14:40
efriedjohnthetubaguy I have been chastised already to make the distinction between "Ask for a resource that *can* do this thing" and "Ask for a resource and then switch this thing on".14:40
johnthetubaguyI am happy to ignore all that for now14:41
johnthetubaguyneed the basic thing first14:41
efriedjohnthetubaguy (or off).  The latter thing definitely needs to be supported - just not via the placement/traits paths, I am led to understand.14:41
efriedI can't remember whether it was a review or IRC where jaypipes and I had that talk.  Looking...14:42
efriedjohnthetubaguy Whee:
*** yassine has quit IRC14:43
johnthetubaguycool, thanks, will have a read14:43
*** yassine has joined #openstack-nova14:43
*** alex_xu has quit IRC14:44
claudiub|3mriedem: cool, ty for the heads up. should be fine now.14:44
*** dave-mccowan has joined #openstack-nova14:44
openstackgerritStephen Finucane proposed openstack/nova-specs master: PCI NUMA Policies
tssuryadansmith, mriedem : so like the bug here even for the map instances, instead of querying with respect to the config file, we could extract the info from the cell_mappings of API database ?14:44
openstackLaunchpad bug 1719487 in OpenStack Compute (nova) "nova-manage db archive_deleted_rows is not multi-cell aware" [Wishlist,Triaged] - Assigned to Zhenyu Zheng (zhengzhenyu)14:44
dansmithtssurya: yeah14:45
tssuryadansmith : so to start working on this I would need a bug ?14:47
dansmithtssurya: it's not really a bug, just an enhancement, but mriedem loves paperwork, so probably easiest to have one14:47
*** jmlowe has quit IRC14:48
*** jmlowe has joined #openstack-nova14:48
*** cdent has joined #openstack-nova14:48
tssuryadansmith : yes :D will open one then14:48
openstackgerritOpenStack Proposal Bot proposed openstack/os-vif stable/ocata: Updated from global requirements
*** jaypipes has joined #openstack-nova14:49
*** alex_xu has joined #openstack-nova14:49
*** dave-mcc_ has joined #openstack-nova14:49
*** mnestratov has joined #openstack-nova14:50
*** rmart04 has quit IRC14:50
mriedemnote that was marked as wishlist14:51
*** dave-mccowan has quit IRC14:51
mriedembecause yeah it's not really a bug14:51
mriedemjohnthetubaguy: idk,14:51
mriedemlogic operators on required and preferred traits is not something i'm thinking about14:51
*** edand has quit IRC14:51
cdentnot feels inevitable, but when the query parameters for allocation candidates becomes skynet, it’s not my fault14:55
tssuryamriedem : yes I understand its not exactly a bug ;14:56
*** baoli has quit IRC14:56
*** baoli has joined #openstack-nova14:59
*** cfriesen_ has joined #openstack-nova15:00
mriedemi wonder if anyone has actually tried benchmarking and comparing ocata to pike for claims in the scheduler, and with multiple scheduler processes, since we claim that's a safe thing to do now
mriedem"The FilterScheduler driver now provides allocations to the Placement API, which helps concurrent schedulers to verify resource consumptions directly without waiting for compute services to ask for a reschedule in case of a race condition. That is an important performance improvement that includes allowing one to use more than one scheduler worker if there are capacity concerns. For more details, see the Pike Upgrade Notes fo15:01
dansmithmriedem: so along those lines, I noticed that we are pegging the crap out of placement during scheduling 100 instances15:01
mriedemdansmith: red hat has a perf lab right? is ^ something they have talked about?15:01
dansmitheven on my fast piece of hardware, placement pegs a CPU15:01
dansmithmriedem: not that I've heard15:01
openstackgerritOpenStack Proposal Bot proposed openstack/os-vif stable/pike: Updated from global requirements
cdentyeah, we had “do some performance testing” in the weekly rp update for so long that I eventually took it out from apparent lack of interest15:02
mriedemok. our public cloud guys have made tweaks to the scheduler for performance in mitaka, and lots of those tweaks i've said, "this thing in pike should resolve/replace that" but i don't have hard evidence15:02
cdentIt would surprise me not one tiny bit that it is not as performant as expected, because the only testing I’m aware of was done using mostly just a database, and not any of the other bits15:02
cdentand since then the queries have modified quite a bit15:02
cdentand we’ve added more objects15:02
*** jmlowe has quit IRC15:03
mriedemcdent: so i don't think you were around yesterday when we were talking about this,15:03
openstackgerritOpenStack Proposal Bot proposed openstack/python-novaclient stable/pike: Updated from global requirements
mriedembut i realized, after several hours, yesterday why i couldn't burst 500 (fake) guest vms on my single node devstack15:03
mriedemand they were all going novalidhost15:03
cdentI had an afternoon in an attorney’s office ...15:03
dansmithalso keep in mind that a little slower scheduler performance compares very favorably to 10% retries in the background because we choose bad computes15:03
cdentmriedem: what was the cause?15:04
mriedemdansmith: that's why i'd want to compare ocata to pike15:04
mriedemcdent: the ultimate cause was a 409 response from placement when putting the allocations during scheduling15:04
mriedemwe retry that 3 times,15:04
dansmithmriedem: yeah, I'm just saying you have to consider "time to all active" not just "time to building" or something15:04
mriedembut it wasn't enough apparently15:04
mriedemdansmith: agree15:04
*** andreas_s_ has joined #openstack-nova15:04
cdentthe 409 was for generation mismatch?15:04
*** jmlowe has joined #openstack-nova15:04
mriedemdansmith: with all the spinning plates, i've been thinking about starting a perf scenario etherpad, polish that up and then send out to see if people can help15:04
*** gyee has joined #openstack-nova15:05
dansmithcdent: it's allocation, so it's the internal rp generation conflict I think15:05
mriedemcdent: this is the response, "There was a conflict when trying to complete your request.\n\n Inventory changed while attempting to allocate: Another thread concurrently updated the data. Please retry your update"15:05
dansmithcdent: and I'm not sure why placement isn't just retrying that for us15:05
mriedemi didn't see any actual inventory updates from the virt driver, which shouldn't happen since the inventory in this case is static15:05
dansmithah sweet15:05
mriedemso the message was a bit misleading15:05
cdentdansmith: yeah, that’s what I meant by generation mismatch15:05
dansmithcdent: yeah, I know where it's happening, but hadn't seen that TODO from jaypipes15:06
dansmithso that's cool15:06
dansmithcdent: I'm not sure why we're hitting it though,15:06
dansmithcdent: since it's a single thread of allocating for things, with no inventory updates coming from the compute15:06
*** ragiman has quit IRC15:06
dansmithso I'm not sure what is racing15:06
cdentthinking out loud: every time we write an allocation we update the generation15:07
*** andreas_s has quit IRC15:07
cdentand we compare the generation with what the generation was before we entered the transaction15:07
cdentso we race to get the transaction15:07
dansmithand we conflict if something else changes the generation while we're trying to15:07
gibimriedem: I don't think we saw a real race on master see my comment in the bug
openstackLaunchpad bug 1719915 in OpenStack Compute (nova) "test_live_migrate_delete race fail when checking allocations: MismatchError: 2 != 1" [Medium,Confirmed] - Assigned to Balazs Gibizer (balazs-gibizer)15:07
cdentwe create an rp object for each allocation at the http layer15:08
cdentthat’s the generation that’s being used15:08
* cdent confirms that guess15:08
*** yangyapeng has joined #openstack-nova15:08
*** yamamoto has joined #openstack-nova15:10
cdentmost straightforward thing to do, presumably, is to do the TODO, and retry 10 times server side, so the client would be effectively retrying 30 times?15:10
*** stvnoyes has joined #openstack-nova15:10
dansmithcdent: sure, we should be retrying server side,15:10
dansmithcdent: my point is I don't know why we'd be hitting this need to retry with a single thread of allocations15:10
cdent(efried I haven’t got an opinion on that conf/ issue)15:11
mriedemright, we process the instances in a for loop in the scheduler15:11
efriedcdent Ack, thanks for looking.15:11
mriedemso we're put'ing the allocations to the same host, but in order15:11
mriedemand the compute shouldn't be changing any inventory since it's static15:11
mriedemi grep'ed the logs for PUT.*inventories and there was nothing15:11
cdentis there anything else putting allocations?15:12
dansmithcdent: no, single 100-instance boot, so one for loop15:12
mriedemwould have to audit that, i didn't dig yet15:12
mriedemcdent: like the compute?15:12
dansmithI mean.. "shouldn't be"15:12
mriedemright, nothing else shoudl be15:12
mriedemsince we're not doing any moves or anything15:12
cdentyeah, I’m wondering if we left something else somewhere that we forgot about?15:12
cdentI know it’s not supposed to be, but given everything...15:13
dansmithmriedem: remember I suggested to see if the compute was doing ocata fallback behavior for some reason15:13
gibimriedem: OK, thats a different failure than the what originally was pasted to the bug report.  I continue digging...15:13
cdentanother possibility is that uwsgi is (somehow, who knows) letting things get out of order15:13
mriedemdansmith: do we log anything specific in that case?15:13
mriedemi see a buttload of the "we're on a pike compute with all pike computes, so not healing allocations" all the time15:13
dansmithmriedem: placement will log it15:13
dansmithmriedem: okay then that probably means it's not15:13
mriedembut ^ is from the periodic15:13
mriedemthere are paths in the RT that go into that code w/o consciously passing the has_ocata_computes flag15:14
mriedembut i think it defaults to False anyway15:14
*** Oku_OS is now known as Oku_OS-away15:14
cdentmriedem: have you got a set of logs you can make available?15:15
openstackgerritOpenStack Proposal Bot proposed openstack/os-vif stable/pike: Updated from global requirements
cdentthis doesn’t feel like something it’s going to be easy to reason about without some files to grep15:16
dansmithcdent: it should be pretty easy to reproduce (or not) in a devstack and then you can instrument the code as needed15:16
openstackgerritOpenStack Proposal Bot proposed openstack/python-novaclient stable/pike: Updated from global requirements
mriedemcdent: no, it's all local15:17
mriedemwell, in this devstack vm which is not local15:17
mriedembut yeah i have the local.conf for the devstack if you want to reproduce15:17
cdentmriedem: sure, but you have tar and such?15:17
mriedemis there a standard way to tar up the journald logs?15:18
cdentballs, I forgot about journald, meh15:18
mriedemit's tar'ed up in devstack-gate15:18
mriedemso i can just copy whatever we do in CI15:18
* cdent nods15:18
openstackgerritRodolfo Alonso Hernandez proposed openstack/nova master: Change 'InstancePCIRequest' spec field
cdentI can’t really look with any real attention until about 3 hours from now, but if you get a chance to do it, that’s great it will useful, if not, just the local.conf will do15:19
*** manasm has quit IRC15:20
*** ragiman has joined #openstack-nova15:20
mriedemyou know, i could just do this with a devstack patch15:21
mriedemthat's easier15:21
mriedemlet the ci do the work15:21
*** josecastroleon has joined #openstack-nova15:21
*** ragiman has quit IRC15:24
mriedemneedless to say, i'm doing a terrible job of reviewing code or specs, or writing specs15:25
*** moshele has quit IRC15:25
mriedemsdague: can you get this stable/pike novaclient bug fix backport?
mriedempretty nasty and we need to release it15:26
sdaguemriedem: looking15:26
*** gouthamr_ has quit IRC15:26
*** lpetrut has joined #openstack-nova15:27
dansmithmriedem: speaking of reviewing, I'm not sure what else to do on the base switchover patch since the difference is not measurable on my box. I'm poking at the fault thing in the later patch, but we can just strip that out and work on it in parallel15:27
mriedemi haven't reviewed the actual change yet15:27
mriedemwas just getting 1000 active vms locally to test the fault thing we talked about last night15:28
mriedemi agree that what we found last night, for numbers, isn't worth holding things up15:28
mriedemsdague: thanks15:28
dansmithmriedem: okay, like I said in the etherpad, I don't think we're doing the fault thing in that patch yet15:28
dansmithand applying the one that does it definitely has an impact15:28
mriedemnegative impact?15:29
*** manasm has joined #openstack-nova15:30
mriedembecause we've added a new unconditional join i suppose15:30
*** Sukhdev has joined #openstack-nova15:30
dansmithit's not a new join, but it's a new query yeah15:30
mriedemyeah, was just thinking that15:30
dansmithI'm messing with the later patch though to see if I can squash it out15:30
mriedemlike you said, we could plumb that in the db api15:30
openstackgerritOpenStack Proposal Bot proposed openstack/os-vif stable/newton: Updated from global requirements
dansmiththat's what I'm doing yeah15:31
mriedemif faults is in expected_attrs, get the instances in deleted/error and include the faults on those?15:31
*** jmlowe has quit IRC15:32
*** dtantsur is now known as dtantsur|afk15:35
cdentdo dansmith and mriedem have an exciting new etherpad they are willing to share?15:36
dansmithcdent: but that's just incidental to him finding the problem15:37
dansmithit's not about the thing we were describing15:37
* cdent nods15:37
cdentyou know I’m a total junkie for all the info15:37
mriedemstarted as getting a benchmark for testing before and after the big instance list change15:37
mriedembut have also added todos for weird things we've seen, like the concurrent update detected thing with 500 instances15:38
*** andreas_s_ has quit IRC15:38
*** psachin has quit IRC15:40
mriedemsdague: can i put [[post-config|$NOVA_CONF]] in stackrc?15:42
cdentmriedem, dansmith, (and sean mooney): If any of you get a chance to look at the discussion on (limiting GET /allocation_candidates ) it’s gotten to the point where we are trying to decide what it is that we are actually optimizing for, so could do with more input15:42
sdaguemriedem: in local.conf15:43
*** rcernin has quit IRC15:43
*** crushil_ has joined #openstack-nova15:43
mriedemsdague: yeah, locally, but this is for running something through ci15:43
*** josecastroleon has quit IRC15:44
sdaguethere isn't really anyway to jam it into stackrc, you need to do project-config changes for stuff like that15:44
sdagueif this is just for hacktastic stuff, just iniset whatever you want in lib/nova15:45
mriedemok, just checking, i can hack this other ways15:45
mriedemyeah that's what i'm doing15:45
*** yamahata has quit IRC15:45
*** mnestratov has quit IRC15:53
*** penick has joined #openstack-nova15:54
dansmithmriedem: so with /servers/details on 2.53, we're pegging conductor real hard. With 2.1 there is no conductor interaction at all15:55
dansmithso we must be doing something stupid in the later microversion15:55
dansmithbecause we shouldn't be making rpc calls at all15:55
dansmithalso, defeating all fault loading and all tag loading doesn't make anything faster15:55
*** erlon has joined #openstack-nova15:55
*** xyang1 has joined #openstack-nova15:56
mriedemhow about services?15:56
dansmithservices is included in the 2.1 columns so I didn't try15:57
cdentjohnthetubaguy: you still want to hold your -2 on ? It’s got a blueprint now15:59
mriedemoh yeah16:00
melwittmriedem: I'm working on the regression func test for the reschedule bug, FYI16:00
dansmithhowever, I can stop compute and conductor and still do the list with no failure (and no difference in speed)16:00
mriedemcdent: the spec on that isn't approved16:01
*** bswartz has joined #openstack-nova16:01
mriedemcdent: see my comment from may 26 on that patch16:01
*** yamahata has joined #openstack-nova16:01
cdentmriedem: okay16:02
openstackgerritBalazs Gibizer proposed openstack/nova master: Fix race in delete allocation in ServerMovingTests
gibimriedem: I've pushed a fix for ^^16:03
openstackLaunchpad bug 1719915 in OpenStack Compute (nova) "test_live_migrate_delete race fail when checking allocations: MismatchError: 2 != 1" [Medium,In progress] - Assigned to Balazs Gibizer (balazs-gibizer)16:03
*** sbezverk has quit IRC16:03
cdentmriedem: I shall continue to remind the keepers of the CI16:04
*** Apoorva has joined #openstack-nova16:04
*** crushil_ has quit IRC16:05
dansmithmriedem: 2.26 adds ~3s to my runtime16:06
dansmithmriedem: 2.16 adds about 0.250s16:07
cdentsdague: you’ve comment on the bug report related to , can you comment on the fix when you get a chance. _might_ have backport potential. stephenfin you willing to upgrade your +1?16:08
dansmithmriedem: 2.26 was tags, btw16:08
dansmithmriedem: so this is 3s with tags short-circuited at the db layer, so I think the 3s is all api overhead for empty things16:08
stephenfincdent: Yup, happy to +2 once someone sdague or dansmith has looked at it (I'm no expert in that area)16:09
cdentthanks stephenfin16:09
johnthetubaguycdent: its normally dropped when the blueprint is approved, I don't remember how spec-less get approved now16:10
cdentjohnthetubaguy: s’okay, matt’s cleared things up: until CI is super happy the blueprint won’t get approved16:10
cdenti hadn’t seen his comment in the middle of the stack16:10
johnthetubaguycdent: ah, cool, I should read the scrollback better16:10
cdentand I should read the comments better :)16:11
johnthetubaguyoh yeah, I see now16:11
*** edand has joined #openstack-nova16:11
*** smatzek_ has joined #openstack-nova16:11
*** r-daneel has joined #openstack-nova16:12
mriedemdansmith: cdent: this is my super hack devstack patch to try and recreate the 500 instance burst failure
*** smatzek has quit IRC16:15
*** hemna_ has quit IRC16:15
mriedemdansmith: ok so we still don't know which microversion is making instance list go back through conductor16:16
dansmithmriedem: I have conductor stopped and nothing else is failing16:16
dansmithwhich I can't explain16:16
mriedemhmm, api going straight to db somewhere?16:17
dansmithapi should be going straight to the db everywhere16:17
dansmithI'm not sure why conductor was doing anything during a list in the first place16:18
dansmithI tried turning it off to see what broke and nothing did16:18
mriedemmaybe you hit a window where a periodic was hitting conductor at the same time as you were doing the instance list?16:18
dansmithcould be, but I did it a few times16:18
dansmitheither way, I'm going to go measure the 2.26 impact with just my change (not the short-circuiting i've done) and on master and see what the diff is16:19
mriedemi'm going to go preheat the oven because it's going to be pot pie time in about an hour16:20
*** crushil has quit IRC16:23
*** tesseract has quit IRC16:24
*** shardy has joined #openstack-nova16:25
openstackgerritBalazs Gibizer proposed openstack/nova master: Fix race in delete allocation in ServerMovingTests
*** Tom_ has quit IRC16:28
*** lpetrut has quit IRC16:28
*** Tom_ has joined #openstack-nova16:29
*** yamahata has quit IRC16:30
openstackgerritMerged openstack/nova master: Fix IoOpsFilter test case class name.
*** Tom_ has quit IRC16:34
openstackgerritMerged openstack/nova master: libvirt: bandwidth param should be set in guest migrate
openstackgerritMerged openstack/nova stable/ocata: Provide hints when nova-manage db sync fails to sync cell0
openstackgerritMerged openstack/nova master: Ensure errors_out_migration errors out migration
*** jmlowe has joined #openstack-nova16:35
efriedsdague got an opinion on ?16:35
johnsomI have an instance booted in nova (master) that nova/neutron shows two plugged ports, but the kernel is not seeing the second network interface.  It was hot-plugged with attach.  Any pointers for debugging this?16:36
johnsomthe qemu process command line (ps -ef) only shows one interface, but I'm not sure if it should show a hot-plugged network interface or not.16:37
johnsomWe have been seeing this in our gates off and on during Pike, but I just had it happen local so I can debug, etc.16:37
openstackgerritChris Dent proposed openstack/nova master: DNM: Don't monkey patch eventlet in functional
openstackgerritChris Dent proposed openstack/nova master: Do not monkey patch eventlet in unit tests
openstackgerritMerged openstack/python-novaclient stable/pike: Allow boot server with multiple nics
*** david-lyle has quit IRC16:40
*** manasm has quit IRC16:41
*** bhagyashris has quit IRC16:42
*** gouthamr has joined #openstack-nova16:43
*** bhagyashris has joined #openstack-nova16:43
*** mvk has quit IRC16:44
*** crushil has joined #openstack-nova16:44
*** hemna_ has joined #openstack-nova16:45
*** gouthamr_ has joined #openstack-nova16:45
johnthetubaguyjohnsom: were there any nova-compute logs about why the attach failed?16:46
johnsomI am looking through and collecting those, n-cpu?16:46
johnthetubaguyyeah, thats the ones I think16:47
johnsomnova show lists it as ACTIVE16:47
*** cdent has quit IRC16:47
johnthetubaguyit wouldn't got to an error state if it failed16:48
openstackgerritMatt Riedemann proposed openstack/nova master: Support qemu >= 2.10
johnsomYeah, n-cpu doesn't have any ERROR level messages.  I can see some vif lines related to the interface, but no ERROR16:48
*** gouthamr has quit IRC16:49
johnthetubaguyits probably a warn16:49
johnthetubaguydo you see the OVS bits wired up?16:49
*** derekh has quit IRC16:49
johnsomSep 27 08:48:44 devstackpy27-2 nova-compute[21517]: WARNING nova.compute.manager [None req-48a86b9a-bf96-4f0e-bc60-00682c991e35 service nova] [instance: fb013f87-2e20-42d7-950d-bc9add853f2c] Received unexpected event network-vif-plugged-37ea16ee-b9bc-48c8-b23b-1221bece7c9a for instance with vm_state active and task_state None.16:49
johnthetubaguyhow did you do the attach?16:50
johnthetubaguyvia the Nova API?16:50
johnthetubaguyits probably a case of tracing that through the code following the logs, seeing where it failed, I suspect in n-cpu but it could have been earlier16:51
johnsomSep 27 08:48:44 devstackpy27-2 devstack@n-api.service[21452]: [pid: 21460|app: 016:52
johnsom|req: 28/58] () {62 vars in 1337 bytes} [Wed Sep 27 08:48:38 2017]16:52
johnsom POST /compute/v2.1/servers/fb013f87-2e20-42d7-950d-bc9add853f2c/os-interface =>16:52
johnsom generated 280 bytes in 5420 msecs (HTTP/1.1 200) 9 headers in 359 bytes (1 swit16:52
johnsomches on core 0)16:52
sdagueefried: the list_opts thing is fine16:52
*** xyang1 has quit IRC16:52
*** sshwarts has quit IRC16:53
johnsomOk, well, I am going to attempt to collect world+dog logs and info to open a bug.  Just wanted to ask if there were specific things I should look at while I have a "live" system.16:53
*** penick has quit IRC16:53
johnthetubaguyjohnsom: you probably want to trace it to this code:
johnthetubaguyjohnsom: in the n-cpu logs16:54
johnthetubaguyI was expecting to see this one I think:
johnthetubaguybut sounds like you hit a different failure16:55
*** yamamoto has quit IRC16:55
johnthetubaguyprobably in
johnthetubaguylooks like an exception in there would not get logged properly16:56
*** penick has joined #openstack-nova16:57
johnthetubaguyjohnsom: sadly that means you need to look through where it got in here:
johnthetubaguyjohnsom: best of luck!16:57
*** gszasz has quit IRC16:58
*** yamamoto has joined #openstack-nova16:59
*** smatzek_ is now known as smatzek17:00
johnsomSep 27 08:48:39 devstackpy27-2 nova-compute[21517]: DEBUG [None req-cfaf7680-4a74-45a7-9dc8-fd793b93fde5 admin admin] [instance: fb013f87-2e20-42d7-950d-bc9add853f2c] Successfully updated port: 37ea16ee-b9bc-48c8-b23b-1221bece7c9a {{(pid=21517) _update_port /opt/stack/nova/nova/network/neutronv2/}}17:00
*** sree has joined #openstack-nova17:00
johnsomYeah, this is going to take some time.17:01
*** penick_ has joined #openstack-nova17:01
*** Apoorva_ has joined #openstack-nova17:03
*** penick has quit IRC17:03
*** pino has quit IRC17:04
*** jpena is now known as jpena|off17:05
*** Apoorva has quit IRC17:06
*** sree has quit IRC17:06
*** tbachman has quit IRC17:06
*** dave-mcc_ has quit IRC17:06
*** cdent has joined #openstack-nova17:09
*** dave-mccowan has joined #openstack-nova17:11
johnthetubaguydansmith: traits for drivers that are not ironic, is it the driver that is meant to be reporting them upwards, or is that config, or both?17:12
johnthetubaguyI guess I was meaning that as a more general question really17:13
*** david-lyle has joined #openstack-nova17:13
*** pcaruana has quit IRC17:14
*** itlinux has joined #openstack-nova17:17
dansmithjohnthetubaguy: at some point I think it'll be a little of both17:25
mriedemcould be an external service17:25
dansmithjohnthetubaguy: some things the compute manager probably adds to a list of virty things that the driver exposes17:25
mriedemthis reminds me, i was going to put something in our "nova is not a metrics gatherer" policy doc about this17:25
mriedembecause of the thing at the ptg where intel wanted nova-compute reporting some crazy cpu traits17:25
johnthetubaguyits just for ironic it feels like the virt driver pushes those up from iroinc17:25
mriedemi think ideally we don't want the ironic driver being a proxy to placement for this stuff17:26
johnthetubaguythe problem is when an admin deletes a trait in ironic, how do we know to delete it in placement17:26
dansmithjohnthetubaguy: ironic could do it itself for sure17:26
dansmithjohnthetubaguy: for libvirt it'd be the virt driver17:26
johnthetubaguythe problem is the nova creates the resource provider right now, using the compute node name, hashring details, etc17:26
dansmithjohnthetubaguy: no, the rp uuid is the ironic node uuid17:26
dansmithjohnthetubaguy: nova creates it if it's not there already, ironic could have done it17:27
johnthetubaguyhmm, I thought it had both for some reason, I need to trace that all properly so its clear in my head17:28
johnthetubaguyso I thought we said at the PTG the ironic virt driver would push this all up, but I am not totally against ironic doing that17:28
*** Tengu has joined #openstack-nova17:29
*** shardy has quit IRC17:29
TenguI'm having some issues setting up host aggregation and flavor matching (i.e. "flavor m1.medium shall start only on that aggregate"17:29
*** gjayavelu has joined #openstack-nova17:30
*** yamamoto has quit IRC17:31
*** yamamoto has joined #openstack-nova17:31
openstackgerritOpenStack Proposal Bot proposed openstack/os-vif stable/pike: Updated from global requirements
*** markmc has quit IRC17:32
*** markmc has joined #openstack-nova17:33
openstackgerritOpenStack Proposal Bot proposed openstack/python-novaclient stable/pike: Updated from global requirements
openstackgerritmelanie witt proposed openstack/nova master: Set group_members when converting to legacy request spec
melwittmriedem: ^ I wrote that test by working from nova/tests/functional/regressions/ and just now realized I guess I could have just added an instance group to the existing test to also test this. but maybe it's better to have the tests separated17:38
*** vvargaszte has joined #openstack-nova17:39
melwittfood for thought17:39
mriedemdansmith: L135 are my results for 1000 active instances with your change17:40
mriedemi'm pretty surprised at the improvements there17:40
cdentmriedem: which job results on are my best target for pokage?17:42
dansmithmriedem: hmm17:42
dansmithmriedem: if you roll back to the other patch does it go back to the perf you measured before?17:43
dansmithmriedem: with my patch we iterate the list fewer times17:43
*** vvargaszte has quit IRC17:43
dansmithI'd be surprised if it made that much difference, but it should make some17:44
mriedemcan try that in a bit17:44
dansmithmy microversion survey results are interesting17:44
dansmithand not good17:44
dansmithwill be done in a few minutes17:44
mriedemcdent: i'd think just the normal tempest dsvm job
cdentroger that17:45
mriedemlots of copied bash in here so i likely screwed something up17:45
mriedemhmm, didn't even get to my stuff17:46
*** tbachman has joined #openstack-nova17:47
openstackgerritMatt Riedemann proposed openstack/nova master: Remove dest node allocations during live migration rollback
*** Swami has joined #openstack-nova17:48
openstackgerritMatt Riedemann proposed openstack/nova master: Remove dest node allocations during live migration rollback
dansmithmriedem: check that out:
dansmithsdague: you too ^17:50
mriedemjesus, graphs?!17:50
mriedemwell, looking at 2.46 and 2.47, i think it's 2.47
dansmith2.26 was not free but 2.47 is killing us17:51
mriedemthis is 500 error/500 active right?17:51
mriedemdansmith: want to report a bug with details?17:51
dansmithmriedem: yes, 500/50017:51
*** itlinux has quit IRC17:51
*** sambetts is now known as sambetts|afk17:52
dansmithso that was on my patch17:53
dansmithI'm going to run it again on master but starting at maybe 2.24 to make sure the knees are in the same place17:53
sdaguedansmith: interesting.... any idea why the display on the embedded data is causing things to go nuts17:53
mriedemi did notice this w/o your change too though17:53
mriedemi wonder if we're lazy-loading the flavor extra specs?17:53
dansmithmriedem: yeah, I don't see differences in my numbers so I'm just doing it for completeness17:53
dansmithsdague: I haven't looked yet17:54
sdaguemriedem: ah, right, that's probably it17:54
dansmithwe shouldn't be lazy-loading extra specs17:54
dansmiththey should be in the flavor in the instance17:54
sdagueI mean, it is a bunch more data. I guess it could just be serialization cost of more data, though it seems weird.17:55
mriedemwell, we were always getting the flavor17:56
openstackLaunchpad bug 1719966 in OpenStack Compute (nova) "Microversion 2.47 punches nova in its special place" [Undecided,New]17:56
mriedemeven before 2.2717:56
dansmithsdague: right, no difference in what we're pulling from the db across that boundary, just what we do with it in the api17:56
dansmithsdague: (I checked)17:57
*** cfriesen_ has quit IRC17:57
mriedemso we were always pulling it
mriedemand we were always joining on it in the db
mriedemso why is this so much slower?
mriedemthe policy check for each instance?17:58
*** penick has joined #openstack-nova17:58
dansmiththe only thing we can lazy-load from flavor isprojects, BTW17:59
dansmithnot extra_specs or anything else17:59
mriedemok i thought that we always had extra_specs but didn't go back to check18:00
sdaguemriedem: yeh, the policy check is going to be per instance18:00
sdaguethe policy check is a fs.stat as well18:00
dansmithwe should check it once and pass it to the per-instance flavor method right?18:00
*** vvargaszte has joined #openstack-nova18:00
sdaguebecause policy file is dynamically reread18:00
dansmiththat might explain why mriedem sees a bigger hit18:00
dansmithyou know what18:01
mriedemwhere is cfriesen when it's time to talk about performance degradation?18:01
dansmithI think we might want to backport this fix18:01
mriedemwe for sure do18:01
dansmithI mean.. maybe18:01
*** penick_ has quit IRC18:01
mriedemthe policy thing is backportable18:01
mriedemcheck once18:01
dansmithwe could leave it and just further relegate pike to the trashcan of releases18:01
mriedemocata is already in that trashcan18:02
mriedemi've literally been sending emails internally for weeks saying, "once you upgrade to pike, this should all be much better"18:02
mriedem*: not actual statement of fact backed up by any evidence18:02
*** cfriesen_ has joined #openstack-nova18:03
sdagueit would be nice if a context only evaluated a particular policy rule once18:05
openstackgerritJohn Garbutt proposed openstack/nova-specs master: Support traits in the Ironic driver
*** ralonsoh has quit IRC18:06
sdaguebecause it's going to be a little awkward to handle cases where you need to send down the preevaluted permission through a bunch of function calls18:06
mriedemthis one shoudn't be too terrible18:07
mriedemdansmith: are you started on the fix?18:07
dansmithright this one should be easy18:07
dansmithmriedem: no but I can18:07
dansmithI'd rather fix this and you finish reviewing my patch18:07
mriedemwhich patch? the kahuna?18:08
mriedemi can't finish what i haven't started18:08
dansmithI'd rather fix this and you start reviewing my patch18:08
Tenguhello! anyone can point me a valid doc for pike and host aggregation + flavor pinning? I'm stuck right now trying to get all working, and I find contradictory docs :/18:08
mriedemdefine "flavor pinning"18:09
mriedemyou can associate a host aggregate with a specific flavor via metadata / extra specs18:09
Tengumriedem: "m1.small must run on that aggregate, while m2.small must run on this aggregate"18:09
mriedemhowever, any other aggregate which is not tied to that flavor can still use it18:09
mriedemthere is no exclusion18:09
mriedemTengu: i think you're looking for this then
Tenguwhat should I put in the metadata?18:10
Tenguah, will check that18:10
Tengu3 days ago? darn… pretty fresh18:11
mriedemthat spec has been around quite awhile18:11
mriedemalmost a year18:11
Tenguwe saw something like that for Icehouse18:11
mriedemsee L466 here
mriedemapparently the stakeholders were not yet synergized as promised18:12
Tenguah, that's for queens… we're running pike :/. don't tell me there isn't anything working right now?18:12
mriedemwell, read the spec first and confirm if that's what you're asking for18:12
Tengulooks like what we want, yes. but that's strange, I found some doc, even at Redhat, saying "it works" but without proper example.18:13
mriedemto summarize, we talked about this at the pike ptg in february, we needed to have the various use cases documented in the spec to make sure the solution would cover them, and there were at least 2 stakeholders in the room saying, "we have an out of tree filter that does something like this" and we said, ok read this and tell us if it will replace your out of tree filter, and those people never replied to ack that it does18:14
Tengumay I explain what I did?18:14
Tenguand point to the doc I followed - maybe a solution might be found18:15
*** pcaruana has joined #openstack-nova18:15
*** NostawRm has quit IRC18:15
*** NostawRm has joined #openstack-nova18:15
Tengumriedem: I followed - I think that one has some equivalent in openstack "open" doc18:16
*** Sukhdev has quit IRC18:17
Tengumriedem: I activated AggregateInstanceExtraSpecsFilter filter in nova.conf, and created two aggregate - all hosts are in those aggregates (in fact, for now, only two hosts - hence once per group).18:17
Tengumriedem: the metadata is like "gen1=true" for first aggregate, "gen2=true" for the second.18:17
melwittTengu: I think if you tag your flavors with extra_specs and then use the AggregateInstanceExtraSpecsFilter you can do what you want18:17
Tenguafter that, the flavor were created, and a metadata was added in the form "gen1=true" for m1.medium, and "gen2=true" for m2.medium18:18
mriedemthe problem is,18:18
mriedemflavor1 is associated to agg1 and flavor2 is associated to agg2,18:18
mriedembut that doesn't exclude agg2 from using flavor118:18
mriedemand vice versa18:18
mriedemthat's the strict isolation problme18:18
Tenguhmm ok.18:18
Tengunot a really big issue - for now, we have "no host found" in fact18:18
melwittI thought if the flavors were tagged it would require that key to pass?18:19
mriedemhonestly i'd have to re-read
melwittI'm reading it again now18:19
mriedemi am definitely not an expert here on the existing capabilities and gaps18:19
Tengumelwitt: same for me - actually, for now, we're unable to start any instance because it doesn't find any host to run it18:19
Tengumriedem: but maybe it's "just" the metadata format that fails me. is there any doc for that?18:20
melwittTengu: and you added gen1=true and gen2=true to your host aggregates?18:20
Tenguyup, as a metadata as well18:21
mriedemthe now deleted ops guide might have had something specific for this18:21
cdentit got moved to the wiki?18:22
TenguI found a doc saying the metadata on the flavor should be in the form aggregate_instance_extra_specs:gen1='true'18:22
Tengubut that doesn't work either18:22
cfriesen_mriedem: dansmith: just saw the mention of microversion 2.47...there was already a call to "instance.get_flavor()" previously, so I had assumed it would get the whole flavor.  I suspect you're right that it's lazy-loading extra-specs.18:22
mriedemit would be in here if it existed
dansmithcfriesen_: no it's policy18:22
dansmithcfriesen_: the policy is checked per instance now, which is an fs call at least18:22
cfriesen_dansmith: ah...I had a networking glitch, missed some irc.18:23
*** yamahata has joined #openstack-nova18:23
cfriesen_dansmith: fix is what, cache the policy?18:23
dansmithcfriesen_: check once per list and not once per instance18:23
dansmithcfriesen_: I'm cooking it up now18:24
dansmithsmells like bacon18:24
cfriesen_dansmith: do we even need that?  couldn't we check it the first time and cache it?18:24
dansmithcfriesen_: that's what I just said18:24
cfriesen_I meant the first time on process startup18:24
mriedemTengu: i've seen a better doc than that red hat one, sec18:24
dansmithcfriesen_: it depends per request18:25
Tengumriedem: that would be nice :)18:25
cfriesen_dansmith: ah, of course18:25
melwittTengu: I found this doc
dansmithmriedem: confirmed the knee in the same place on master18:26
TenguI've also followed - but failed.18:26
Tengumelwitt: ah, ocata, might work, pike is just one version ahead. will check that, thanks!18:26
mriedemmelwitt: yeah
mriedemopenstack flavor set --property aggregate_instance_extra_specs:ssd=true ssd.large18:26
Tenguduh… ok, I was also on that one -.-'18:27
melwittTengu: the main thing I saw ppl run into a snag is that you apparently have to use that prefix when you set the key on the flavor but NOT use it when you set the key on the aggregate18:27
Tengumelwitt: yup, I have done that18:27
Tengubut to no success until now.18:27
mriedemand that key prefix is only used with AggregateInstanceExtraSpecsFilter18:27
cfriesen_was just going to mention the filter18:27
mriedemand you have to make sure you have that enabled18:27
melwittTengu: yeah, did you add that filter to your configured filters for the FilterScheduler?18:28
melwittin nova.conf18:28
Tenguit's enabled. should it be in the first position?18:28
Tengumelwitt: yep, it's present18:28
mriedemorder only matters for performance18:28
Tenguand I rebooted the controllers in order to ensure all is running at the latest config version18:28
melwittTengu: no but you will want to check nova-scheduler logs to make sure some other filter isn't rejecting it18:28
Tengumriedem: hmm ok.18:28
melwittat DEBUG log level. it's possible something else is going wrong and not the key match for the metadata18:29
Tengumelwitt: yup, but I didn't see anything. the instance "directory" was created on the right node in /var/lib/nova/instances18:29
melwittif you're getting NoValidHost you should see something18:29
Tengubut after a while, paff, directory is removed, and crash, "no host found"… although it actually HAD found a host18:29
melwittunless a compute host rejected the request in which case you should see an error in the nova-compute logs or the nova-conductor logs18:30
Tenguwill check that one.18:30
melwittthe way it works is if scheduling filters all pass, it goes to nova-compute, if something fails while it builds it, it will tear it down, log stuff, and try to reschedule to another host if you have retries configured18:30
Tenguwhat would be the patter of a rejection in nova-compute.log ?18:31
Tenguhmm ok.18:31
TenguI have the retryfilter18:31
melwittshould see something logged at ERROR level I think18:31
melwittin nova-compute18:31
Tenguthink this one will try to re-schedule18:31
Tengucorrupted image download o_O18:32
*** pino has joined #openstack-nova18:32
Tenguthat might explain a bit. but that would point the glance storage18:32
openstackgerritEric Berglund proposed openstack/nova master: PowerVM Driver: config drive
openstackgerritEric Berglund proposed openstack/nova master: WIP(5): PowerVM driver: ovs vif
Tengualthough… hmm. timestamp doesn't really match. will dig a bit more.18:33
melwittTengu: yeah, so you have other issues there. but as long as the instance is always landing on the host where the aggregate meta matches the flavor, you know at least the extra specs filtering is working correctly18:34
melwitt(for your original concern)18:34
*** acormier has quit IRC18:34
Tengumelwitt: right.18:35
Tenguso my debug steps weren't that wrong. I should have had a better look to the nova-compute.log file though.18:35
mriedemyou can also trace the request id and/or instance id through the logs if you have your logs pumped to an ELK stack18:36
mriedemor journald like in devstack18:37
Tengufor now we don't have an ELK (it will run on the openstack… well, yes, that might cause some issues at some point ;)).18:38
Tengubut we want to do that, yep.18:38
dansmithmriedem: over about 300 runs, my patch is consistently faster than master18:39
mriedemoh that's w/o the policy fix :)18:39
mriedemi was like, wtf18:39
Tengubut the image corruption is the best hint for now. Have to check why - the ceph cluster isn't a cluster for now and we have some failed disks on it, so it can explain a lot. it's not in prod for now, this also explain some issues18:39
mriedemdansmith: throw that in somewhere so we don't lose it18:41
stvnoyeshi mriedem, if you get a change to re-review it would be great. I am on vacation next week so there's still some time this week for me to turn the review around again if it's needed. thanks.18:42
mriedemdansmith: totally unrelated, but i'm think about throwing the ceph job in the experimental queue
stvnoyesjohnthetubaguy: pls take a look at when you get a chance. it's a pretty small change, and it's needed for the cinder v3 live migrate change. thanks.18:44
dansmithmelwitt: ^18:44
melwittit would take me awhile to unroll what is going on with that job18:46
mriedemthis is compared to the normal dsvm tempest job
melwittyeah, hm. so it was tracking okay until around the 16th18:47
*** lpetrut has joined #openstack-nova18:47
melwittI'll dig into it18:48
mriedemwell, there were also spikes in the normal job then too, just not as bad18:48
mriedem9/23 is where it goes nuts18:48
melwittyeah. last we discussed it was at the last PTG and jbernard had some TODOs but I didn't know details about what they were. I thought the first thing was something to do with a job timeout being too short18:49
mriedemhe was going to start restricting the tests18:49
melwittrestricting in what way?18:49
melwittokay. so that would be my starting point, aside from the latest craziness. which might just be more timeout and OOM stuff (have to dig)18:50
*** lpetrut has quit IRC18:50
efriedmriedem Re-request tough love on please18:51
*** lpetrut has joined #openstack-nova18:51
mriedemmelwitt: merged on the 22nd18:51
mriedemcinder.tests.tempest.api.volume.test_volume_revert.VolumeRevertTests.test_volume_revert_to_snapshot_after_extended is what i see failing18:51
mriedemso my guess is, the ceph job doesn't care for live snapshots18:51
melwittthanks for that info18:52
*** penick has quit IRC18:53
mriedemalthough the test that's failing is a cinder api test18:54
*** sbezverk has joined #openstack-nova18:54
mriedemand i don't see any related errors in n-cpu18:54
*** moshele has joined #openstack-nova18:55
openstackgerritDan Smith proposed openstack/nova master: Fix policy check performance in 2.47
*** cleong has quit IRC18:58
dansmithoops, didn't finish the commit message on that one18:59
dansmithI'm just testing it in my devstack rig anyway18:59
*** slaweq_ has joined #openstack-nova19:00
*** pcaruana has quit IRC19:00
*** sree has joined #openstack-nova19:01
openstackgerritMatt Riedemann proposed openstack/nova master: doc: make host aggregates examples more discoverable
*** hemna_ has quit IRC19:02
mriedemnot even the bug link19:02
*** r-daneel has quit IRC19:02
mriedemyou were so excited19:02
mriedemabout getting punched in the naughty parts19:02
mriedemTengu: melwitt:
mriedem^ should help a bit with doc discovery19:03
*** r-daneel has joined #openstack-nova19:03
Tengumriedem: \o/ thanks !19:03
Tengufor now I'm digging in glance, as apparently it's crashed.19:04
*** sree has quit IRC19:06
dansmithsdague: mriedem: cfriesen_:
mriedemdansmith: awesome19:07
mriedemi ran the 2.53 microversion, GET /servers/detail thing again w/o your patch, to see why i had just a big difference in numbers, and you're right, it's the vm19:08
mriedemso w/o your patch, it's still closer to with your patch,19:08
mriedemand over half of what it was the other day on the other vm19:08
mriedemso just need to chalk that up to public cloud19:08
cdentwhat happened at 2.47?19:09
dansmithmriedem: cool19:09
mriedemcdent: we started checking policy per instance when listing instances19:09
mriedemwhich adds up when you're listing 1000 instances19:09
*** penick has joined #openstack-nova19:10
openstackgerritDan Smith proposed openstack/nova master: Fix policy check performance in 2.47+
*** liangy has joined #openstack-nova19:11
cfriesen_cdent: my bad, I didn't realize policy check was expensive19:11
*** lyan has quit IRC19:11
mriedemi'm sure i approved the change so don't worry about it19:12
cdentcfriesen_: a reasonable thing to assume in a reasonable universe, but we probably left that one long ago19:12
sdagueI kind of wonder if there are other places with embedded policy checks like that are expensive19:12
sdaguecfriesen_: there is an implicit fstat because policy is live reread19:13
cdentspeaking of, that’s a potential next microoptimization in the unit tests. that file gets read over and over and over over and over and over and ...19:13
sdaguehonestly, it might behoove us to change that behavior entirely, as we've got the hup handler now19:14
bauzasdansmith: you trampled me19:15
*** sahid has quit IRC19:15
dansmithbauzas: I did?19:15
bauzasdansmith: with Twitter19:15
*** edmondsw has quit IRC19:15
bauzasso, maybe you should be the next US president given you use Twitter for trampling folks :p19:16
bauzasmmm, maybe "trample" is not the right verb19:16
dansmithI'm not sure what trampling I did, but I definitely need not be president19:16
penicktoo late i'm writing you in19:16
bauzasI mean, I chilled :p19:16
mriedemyou made sylvain spit out his coffee19:17
mriedemyou "floored" him19:17
bauzaswhen I saw the tweet for 2.47 :p19:17
bauzassorry for "trampling"19:17
dansmithbauzas: okay I replied to you about two seconds before you pinged me here so I thought you meant my reply was rude in some way19:18
bauzasemacron: maybe you should ask French folks to stop using French but rather English ?19:18
*** edmondsw has joined #openstack-nova19:18
bauzasdansmith: sorry, the verb wasn't good :)19:19
bauzas"chilling" is better19:19
bauzasdansmith: anyway, thanks for your tweet19:20
*** liverpooler has quit IRC19:23
*** edmondsw has quit IRC19:23
*** liverpooler has joined #openstack-nova19:23
*** jmlowe has quit IRC19:25
cfriesen_dansmith: reviewing your patch.  I assume the version check is a performance optimization to avoid the policy check if we can?19:25
*** penick_ has joined #openstack-nova19:27
mriedemcfriesen_: it's because we only ever care about showing flavor extra specs if you're requesting 2.47 or above19:28
mriedemso don't even make the policy check otherwise19:28
*** liangy has quit IRC19:28
dansmithcfriesen_: yeah19:28
sdaguedansmith: so one thing to consider on that test, there is nothing in that test asserting the server list is > 1 right now19:28
sdaguebecause it's all common setup19:28
dansmithsdague: true, but I did check that its 419:28
dansmithI can add another19:29
sdagueI thought it was 519:29
sdagueI was just running it19:29
dansmithit was 419:29
mriedemso just self.assertGreater(len(instances), 1) ?19:29
dansmithoh no,19:29
dansmithtop index was 419:29
sdagueyeh, something like that19:30
*** penick has quit IRC19:30
sdaguejust so the test is more concisely valid19:31
sdaguethe reset seems fine to me19:31
openstackgerritDan Smith proposed openstack/nova master: Fix policy check performance in 2.47+
*** shaner has quit IRC19:36
*** liverpooler has quit IRC19:36
*** shaner has joined #openstack-nova19:36
cfriesen_just throwing this out there...could we use a global variable for show_extra_specs such that it's None for the first instance and then the calculated value is used for subsequent ones?  That'd avoid the API changes, but globals are icky.19:36
mriedemthis isn't an api change19:37
cfriesen_picky picky...function signature changes19:37
dansmithcfriesen_: that doesn't work19:37
dansmithcfriesen_: because this code is running lots of lists for lots of people, some of which do and some of which don't have that permission19:37
cfriesen_set it to None at the beginning of each call19:38
*** gjayavelu has quit IRC19:38
mriedemthat's what this does...19:38
mriedemw/o a global19:38
dansmithcfriesen_: that also won't work19:38
*** liverpooler has joined #openstack-nova19:38
dansmithcfriesen_: because we don't necessarily complete a whole call through the stack before we go on to the next one19:38
dansmithcfriesen_: that would be known as a "CVE"19:39
cfriesen_dansmith: due to eventlets I guess?19:39
dansmiththreads in general19:39
mriedemi wonder if the fedex guy is required to jog from the truck to the house and back19:40
mriedemlike, is there a camera watching him to make sure he jogs?19:40
dansmiththere is at my house19:40
dansmithI call and complain any time he saunters instead of jogs19:40
mriedemwhat if he mosey's?19:40
cfriesen_dansmith: I thought we were using processes for nova-api, not threads19:41
dansmithmosey is a saunter with slightly more vigor19:41
mriedemmore tude19:41
dansmithcfriesen_: there are threads (greenthreads currently) in each process19:41
cfriesen_ah, got it.19:41
dansmithcfriesen_: but seriously, setting a global for a permission flag and hoping it gets reset before the next call is like the worst idea ever :)19:41
cfriesen_I'm pretty sure I've had worse.  :)    I just didn't like the fact that we were checking it in two different places depending on flow.19:42
cfriesen_In C/C++ I'd just pass a pointer or pass it by reference.19:43
mriedemin fortran i'd goto that mothertrucker19:43
* dansmith moves on19:43
cdentdansmith: None and False meaning different  things. ballsy.19:43
cfriesen_I saw old fortran back in my engineering days that had goto with multiple targets....that was messed up.19:43
dansmithcdent: they are wholly different things :)19:44
sdaguecfriesen_: if you wanted to handle it in a different way, the nova local caching model would be just to make context.can cache the parameter lists and answers19:44
cdentsure, but still19:44
sdaguebecause contexts are constructed all the time19:44
*** eharney has quit IRC19:44
sdagueand that would fastpath the check in tight loops like this19:44
dansmithsdague: so I was thinking about a way to make the fixture raise if a test checked the same exactly policy action/target in a single run19:45
*** liverpooler has quit IRC19:45
cfriesen_sdague: yes, that'd be nice.  but this isn't horrible19:45
dansmithsdague: to ferret out some of what you were saying might be there19:45
dansmithcfriesen_: sdague: this has to be minimal backport though19:45
dansmithbecause this _has_ to be backported19:45
dansmithelse we lose our license to code19:45
mriedemretain our license to ill?19:45
sdaguedansmith: yeh, I think context.can is the right place to shadow that19:46
sdaguein tests19:46
dansmithsdague: yeah19:46
sdagueI also really wonder how much better perf would get if we remove the policy reload entirely19:46
sdaguebecause that's just kind of there because of the early rax operational model19:47
dansmithsdague: if you can give me a line to comment out I can run it on my rig while it's still built19:47
mriedemi think dims was looking at making the policy check call an external service...if that makes you worry at all19:47
*** edmondsw has joined #openstack-nova19:47
sdaguemriedem: yeh, I think this small thing has pretty much shown you can only do that if it's a load on startup19:47
*** lyan has joined #openstack-nova19:47
sdagueotherwise you are toast19:47
sdaguedansmith: yeh, let me go look, it's been a minute since I've been in that code19:48
dansmithmriedem: jesus19:48
*** edmondsw has quit IRC19:48
*** edmondsw has joined #openstack-nova19:48
*** markvoelker has quit IRC19:48
bauzasdansmith: sdague: mriedem: any reason why isn't yet +W'd ?19:49
bauzascan I pull the trigger?19:49
mriedemapparently there is a cache
*** markvoelker has joined #openstack-nova19:49
*** shaner has quit IRC19:50
dansmithmriedem: that means we're still doing the fstat() each time right?19:50
dansmiththat's the painful bit I think, especially given policy files are empty now19:50
*** shaner has joined #openstack-nova19:50
dansmithactually two calls19:50
dansmithexists and getmtime19:50
mriedemjust return True from _is_directory_updated all the time?19:50
dansmithor only check the directory if it's been 30s since we last did it19:51
dansmithor only check it after you've sighup'd :)19:51
dansmithI guess you meant for this q&d check, yeah19:52
mriedemyeah just change that to return False all the time19:53
mriedemit would be False based on how it's used19:53
*** pino has quit IRC19:53
mriedemsee if you shave a few seconds19:53
*** pino has joined #openstack-nova19:53
sdaguemriedem: yeh, that's the stat calls19:54
sdague_is_directory_updated is different though19:54
sdaguebecause that's the different terrible issue of a policy.d19:55
sdaguewhich makes this *N worse19:55
openstackgerritMatt Riedemann proposed openstack/nova stable/pike: Fix policy check performance in 2.47+
dansmithbauzas: pull the trigger19:55
dansmithsdague: so wait, is there more I need to kill?19:55
openstackgerritmelanie witt proposed openstack/nova master: Set group_members when converting to legacy request spec
bauzashonestly, I think fixing it now is better anyway19:55
bauzashah, jinxed19:56
*** moshele has quit IRC19:56
sdaguedansmith: I think you want to short circuit here -
*** vvargaszte has quit IRC19:58
sdagueand just return (True, "")19:58
sdagueactually (False, "")19:58
mriedem"A tuple with a boolean specifying if the data is fresh or not" in the docstring doesn't seem to match the code19:59
sdaguemriedem: yeh it does20:00
mriedemif the data "was refreshed"20:00
sdaguemriedem: ah, ok, english binary reversal20:00
sdaguehonestly, I read it in the reversed context the first time20:00
dansmithI thought I should return (False, cache['filename']['data']) no?20:01
mriedemit's not not unfresh?!20:01
sdaguedansmith: it could, but it could also just return garbage20:01
sdaguebecause if the first param is False, nothing is done with it20:01
dansmithah, okay20:01
sdague"" would be the production default state20:01
dansmithoh, right,20:02
sdagueas policy.json is blank20:02
*** chyka has quit IRC20:02
dansmithwas thinking I had to return the actual thing20:02
*** chyka has joined #openstack-nova20:02
sdaguethough, I think the impact is going to be minimal, given that the issue was 1000 of these calls, which added 3s in your env, so were looking at 3ms per policy check20:05
sdagueso, unless we have another nested thing, it's going to be hard to see the impact20:05
dansmithI see no impact as the first numbers are coming out20:05
*** yamamoto has quit IRC20:08
mriedemmelwitt: couple comments on the test, but looks great otherwise20:09
melwittthanks, looking20:13
dansmithmriedem: sdague: not likely any difference:
*** edand has quit IRC20:14
*** r-daneel has quit IRC20:14
*** mnestratov has joined #openstack-nova20:15
*** r-daneel has joined #openstack-nova20:15
mriedemsdague: i just git log -i --grep'ed for the first time i think20:18
mriedemafter you've said it in channel once per week i think20:18
mriedemit might be the first thing you say when you wake up20:19
mriedemnot sure20:19
efriedoo.  TIL.20:22
sdaguedansmith: yeh, it's clear the current call is down to 2 policy checks, which is good20:26
*** yamamoto has joined #openstack-nova20:27
openstackgerritMerged openstack/nova master: _rollback_live_migration in live-migration seqdiag
*** smatzek has quit IRC20:29
openstackgerritmelanie witt proposed openstack/nova master: Make setenv consistent for functional and api_sample_tests
*** yamamoto has quit IRC20:31
mriedemcdent: i know what's going on in my devstack patch20:32
*** gouthamr_ has quit IRC20:32
mriedemoh do i ever20:32
*** belmoreira has joined #openstack-nova20:32
mriedemi've been mtreinish'd i think20:32
dansmithmriedem: unfortunately a unit test failure snuck past me in that fix patch20:32
mriedemwhere is my gd trombone20:33
openstackgerritDan Smith proposed openstack/nova master: Fix policy check performance in 2.47+
*** jpena|off has quit IRC20:33
*** ltomasbo has quit IRC20:34
mriedemcdent: this case statement used to have a wildcard
openstackgerritmelanie witt proposed openstack/nova master: Set group_members when converting to legacy request spec
*** gouthamr has joined #openstack-nova20:35
cdentmriedem: whyreaka20:36
sdaguemriedem: ok, so what's the issue?20:37
*** ltomasbo has joined #openstack-nova20:37
mriedemVIRT_DRIVER=fake means you don't get any default image downloaded20:37
*** jpena|off has joined #openstack-nova20:38
mriedemso i assume you don't want the wildcard back20:38
johnsomFYI, after collecting a mountain of logs on this missing network interface issue I decided to force a PCI bus rescan inside the instance, boom, the interface appears as it should have. So, ubuntu/kernel/something issue and not nova/neutron20:38
mriedemso i'll just add a case for fake and make it the same as libvirt?20:38
cdentsounds right to me20:39
johnsomstock, current, ubuntu 16.04 cloud image20:39
cdentI’m dead, will check back in the morn20:39
*** cdent has quit IRC20:39
sdaguemriedem: ok, because later everything assumes a real image?20:39
*** gouthamr has quit IRC20:39
mriedemsdague: well, tempest tries to get the image from glance to put in tempest.conf20:40
mriedemthe image id i mean20:40
mriedemworking a patch20:41
*** gouthamr has joined #openstack-nova20:42
sdagueyeh, so, honestly we should probably figure out a better setup strategy for "throw away this image"20:43
sdaguemriedem: also, the qemu 2.10 thing wasn't a race, it's actually python2.7 and 3.5 evaluating the mock sentinel differently in the comparison20:44
sdaguefor one of them it passes a >= check and the other it does not20:44
*** belmoreira has quit IRC20:44
*** gouthamr has quit IRC20:45
openstackgerritSean Dague proposed openstack/nova master: Support qemu >= 2.10
*** esberglu has quit IRC20:47
*** esberglu has joined #openstack-nova20:48
mriedemmelwitt: pep820:49
mriedem./nova/tests/functional/regressions/ F401 'cast_as_call' imported but unused20:50
melwittsigh, of course20:50
mriedemi still have the trombone out if you want to borrow it20:50
melwitthaha yeah20:50
melwittI thought, pep8 can't complain if I just deleted a line. yeah it could20:51
*** lucasxu has quit IRC20:52
openstackgerritmelanie witt proposed openstack/nova master: Set group_members when converting to legacy request spec
*** esberglu has quit IRC20:52
melwittgdi locally I'm failing a notifications func test because I'm missing a notification from conductor. but the code that's supposed to send that notification is running, so I don't know how it could be missing20:57
*** LeoBud has joined #openstack-nova20:58
*** pchavva has quit IRC20:58
*** smatzek has joined #openstack-nova21:02
*** mvk has joined #openstack-nova21:02
*** sree has joined #openstack-nova21:02
*** penick_ has quit IRC21:03
*** eharney has joined #openstack-nova21:04
*** penick has joined #openstack-nova21:07
*** sree has quit IRC21:07
*** esberglu has joined #openstack-nova21:08
*** thorst has quit IRC21:09
*** crushil has quit IRC21:09
*** penick has quit IRC21:11
*** thorst has joined #openstack-nova21:11
*** MVenesio has quit IRC21:13
*** MVenesio has joined #openstack-nova21:13
*** bnemec has quit IRC21:14
*** thorst has quit IRC21:15
*** jpena|off has quit IRC21:16
*** ltomasbo has quit IRC21:16
*** MVenesio has quit IRC21:17
*** ltomasbo has joined #openstack-nova21:18
*** jpena|off has joined #openstack-nova21:18
*** LeoBud has quit IRC21:20
*** yamamoto has joined #openstack-nova21:28
*** thorst has joined #openstack-nova21:29
*** slaweq_ has quit IRC21:32
*** thorst has quit IRC21:34
*** yamamoto has quit IRC21:34
*** yamahata has quit IRC21:36
*** itlinux has joined #openstack-nova21:36
*** yamahata has joined #openstack-nova21:36
openstackgerritMerged openstack/nova master: Have one list of reboot task_states
openstackgerritMichael Still proposed openstack/nova master: Move ploop commands to privsep.
openstackgerritMichael Still proposed openstack/nova master: Read from console ptys using privsep.
openstackgerritMichael Still proposed openstack/nova master: Don't shell out to mkdir, use ensure_tree()
openstackgerritMichael Still proposed openstack/nova master: Cleanup mount / umount and associated rmdir calls
openstackgerritMichael Still proposed openstack/nova master: Move lvm handling to privsep.
openstackgerritMichael Still proposed openstack/nova master: Move shred to privsep.
openstackgerritMichael Still proposed openstack/nova master: Move xend existence probes to privsep.
openstackgerritMichael Still proposed openstack/nova master: Move the idmapshift binary into privsep.
openstackgerritMichael Still proposed openstack/nova master: Move loopback setup and removal to privsep.
openstackgerritMichael Still proposed openstack/nova master: Move nbd commands to privsep.
openstackgerritMichael Still proposed openstack/nova master: Move kpartx calls to privsep.
openstackgerritMichael Still proposed openstack/nova master: Move blkid calls to privsep.
*** gjayavelu has joined #openstack-nova21:40
mriedemsydney travel approved, hotel booked!21:42
*** tetsuro has quit IRC21:42
*** awaugama has quit IRC21:43
*** gyee has quit IRC21:44
mriedemtime to go through the blistering visa process21:44
*** dave-mccowan has quit IRC21:45
melwittit's easy to do on their site. there's no free option for US ppl, have to get the ETA visa and it costs $20 AUD21:45
*** penick has joined #openstack-nova21:46
mriedemyeah, i was joking21:46
openstackgerritMatt Riedemann proposed openstack/nova stable/pike: Fix policy check performance in 2.47+
mriedemooo the hotel has an infinity pool21:48
mriedemcan we do the sessions from poolside?!21:48
mriedemwith kangerbangers21:49
melwittwhat? what kinda fancy hotel are you staying at21:49
mriedemthe sofitel next to the conference center, for the non-refundable pay immediately and never get your money back rate21:49
mtreinishmriedem: the sofitel has an inifinity pool? I must have missed that on the website21:51
mtreinishnot that I looked closely21:51
openstackgerritMerged openstack/nova master: Implement query param schema for agent index
mriedemwell, it will, it's not open yet21:53
*** baoli has quit IRC21:54
melwitthm, it looks like the prices are lower than when I first looked several weeks ago21:54
cfriesen_melwitt: yeah, I was surprised that Canadians don't get the free option.  I figured what with commonwealth and all...21:54
mriedemthe only thing you share,21:55
*** penick has quit IRC21:55
mriedemis your love of groveling for the royal family21:55
mriedemthere are no other benefits21:55
cfriesen_our love of beer, more like.21:55
mriedemgermans don't love beer?21:55
cfriesen_I didn't say it was exclusive21:55
mriedembut germans can't love the royalty the same way21:56
cfriesen_half the british royalty was german anyway21:56
mriedemoh breeding21:56
mriedemi plan to marry my daughter to a fellow in the next county21:57
mriedemfor the inter-county alliance21:57
mriedemagainst NE iowa21:57
*** yamamoto has joined #openstack-nova21:57
*** mriedem is now known as mriedem_away21:57
*** yamamoto has quit IRC21:57
*** dave-mccowan has joined #openstack-nova21:57
*** thorst has joined #openstack-nova21:58
*** lpetrut has quit IRC21:59
*** lyan has quit IRC21:59
*** vladikr has quit IRC22:02
*** vladikr has joined #openstack-nova22:02
*** slaweq_ has joined #openstack-nova22:03
*** dave-mccowan has quit IRC22:03
*** takashin has joined #openstack-nova22:06
takashinSpec cores, would you review ? It got one +2.22:09
openstackgerritMerged openstack/nova-specs master: Move pike implemented specs
*** smatzek has quit IRC22:15
*** gbarros has quit IRC22:22
*** edmondsw has quit IRC22:23
*** gbarros has joined #openstack-nova22:23
*** mnestratov has quit IRC22:28
*** tidwellr has quit IRC22:30
*** krtaylor has quit IRC22:30
*** gjayavelu has quit IRC22:31
*** slaweq_ has quit IRC22:36
*** slaweq_ has joined #openstack-nova22:39
*** krtaylor has joined #openstack-nova22:43
*** penick has joined #openstack-nova22:45
*** jaypipes has quit IRC22:48
*** gbarros has quit IRC22:50
*** penick has quit IRC22:52
*** yamamoto has joined #openstack-nova22:58
openstackgerritMerged openstack/nova master: Fix --max-count handling for nova-manage cell_v2 map_instances
*** rtjure has quit IRC23:01
*** claudiub|3 has quit IRC23:03
cfriesen_is there a reason why nova uses named indexes in the DB rather than using "index=True" as part of the Column definition?23:03
*** pino has quit IRC23:04
*** yamamoto has quit IRC23:05
*** slaweq_ has quit IRC23:13
*** baoli has joined #openstack-nova23:18
*** slaweq_ has joined #openstack-nova23:18
*** baoli_ has joined #openstack-nova23:22
*** baoli has quit IRC23:22
*** tidwellr has joined #openstack-nova23:22
*** hongbin has quit IRC23:25
*** gjayavelu has joined #openstack-nova23:26
*** felipemonteiro_ has quit IRC23:29
*** thorst has quit IRC23:30
mriedem_awayeasier to change them later if needed23:31
mriedem_awaycfriesen_: ^23:31
mriedem_awaysince different backends name them if you do it automatically23:31
*** mriedem_away is now known as mriedem23:31
*** Sukhdev has joined #openstack-nova23:32
*** jmlowe has joined #openstack-nova23:32
mriedemalthough i think in recent years we just identify an index by the columns in it23:32
*** krtaylor has quit IRC23:35
*** edmondsw has joined #openstack-nova23:40
*** ijw has joined #openstack-nova23:43
*** edmondsw has quit IRC23:44
*** moshele has joined #openstack-nova23:44
*** ijw has quit IRC23:45
*** ijw has joined #openstack-nova23:45
*** slaweq_ has quit IRC23:48
*** gjayavelu has quit IRC23:50
*** jmlowe has quit IRC23:52
*** slaweq_ has joined #openstack-nova23:52
*** bnemec has joined #openstack-nova23:52
*** jmlowe has joined #openstack-nova23:53
*** r-daneel has quit IRC23:58
*** yamahata has quit IRC23:58

Generated by 2.15.3 by Marius Gedminas - find it at!