Monday, 2018-01-22

*** karthiks has joined #openstack-nova00:01
*** yankcrime has quit IRC00:04
*** _nick has joined #openstack-nova00:05
*** sree has joined #openstack-nova00:08
*** karthiks has quit IRC00:09
*** armax has quit IRC00:10
*** sree has quit IRC00:12
*** dave-mccowan has joined #openstack-nova00:18
*** karthiks has joined #openstack-nova00:22
*** yingjun has joined #openstack-nova00:22
*** sree has joined #openstack-nova00:27
*** sree has quit IRC00:32
*** suresh12 has joined #openstack-nova00:38
*** edmondsw has joined #openstack-nova00:46
*** hiro-kobayahi has joined #openstack-nova00:48
*** liuzz has joined #openstack-nova00:48
*** yassine has joined #openstack-nova00:51
*** edmondsw has quit IRC00:51
*** yassine is now known as Guest7870300:51
*** Tom-Tom has joined #openstack-nova00:52
*** Tom-Tom has quit IRC00:59
*** Tom-Tom has joined #openstack-nova01:00
*** Tom-Tom has quit IRC01:01
*** Tom-Tom has joined #openstack-nova01:01
*** Tom-Tom_ has joined #openstack-nova01:03
*** hshiina has joined #openstack-nova01:03
*** Tom-Tom has quit IRC01:06
*** phuongnh has joined #openstack-nova01:06
*** zhaochao has joined #openstack-nova01:08
*** hieulq has joined #openstack-nova01:12
*** tiendc has joined #openstack-nova01:14
*** yangyapeng has joined #openstack-nova01:15
*** tuanla____ has joined #openstack-nova01:24
*** zhurong has joined #openstack-nova01:29
*** suresh12 has quit IRC01:29
*** suresh12 has joined #openstack-nova01:31
*** zhurong has quit IRC01:33
openstackgerritYikun Jiang (Kero) proposed openstack/nova master: Fix 500 in test_resize_server_negative_invalid_state
openstackgerritYikun Jiang (Kero) proposed openstack/nova master: Add index(instance_uuid, updated_at) on instance_actions table
*** suresh12 has quit IRC01:36
*** markvoelker has joined #openstack-nova01:47
*** gongysh has joined #openstack-nova01:49
*** hongbin has joined #openstack-nova02:02
*** fragatina has quit IRC02:05
*** Dinesh_Bhor has joined #openstack-nova02:06
openstackgerritjichenjc proposed openstack/nova-specs master: add z/VM Driver in Rocky
*** gcb has joined #openstack-nova02:09
*** crushil_ has joined #openstack-nova02:11
*** weshay has quit IRC02:14
openstackgerritint32bit proposed openstack/nova master: Fix wrong disk size for sparse flat files
*** weshay has joined #openstack-nova02:15
*** markvoelker has quit IRC02:21
openstackgerritTakashi NATSUME proposed openstack/nova master: [placement] Add sending global request ID in delete (3)
*** takashin has left #openstack-nova02:26
*** zhurong has joined #openstack-nova02:32
openstackgerritYa Lian Pan proposed openstack/nova master: z/VM Driver: Initial change set of z/VM driver
openstackgerritYa Lian Pan proposed openstack/nova master: z/VM Driver: Spawn and destroy function of z/VM driver
*** takashin has joined #openstack-nova02:34
*** annp has joined #openstack-nova02:50
openstackgerritTakashi NATSUME proposed openstack/nova master: api-ref: Parameter verification for
openstackgerritTakashi NATSUME proposed openstack/nova master: api-ref: Example verification for
*** crushil_ has quit IRC02:53
*** yamamoto has joined #openstack-nova02:57
*** hiro-kobayahi has quit IRC03:12
*** ljjjustin has joined #openstack-nova03:16
openstackgerritTakashi NATSUME proposed openstack/nova master: [placement] Add sending global request ID in post
*** markvoelker has joined #openstack-nova03:17
openstackgerritTakashi NATSUME proposed openstack/nova master: [placement] Add sending global request ID in put (1)
openstackgerritTakashi NATSUME proposed openstack/nova master: [placement] Add sending global request ID in put (2)
*** karthiks has quit IRC03:18
*** sree has joined #openstack-nova03:24
*** ljjjustin has quit IRC03:27
*** karthiks has joined #openstack-nova03:31
*** psachin has joined #openstack-nova03:39
*** hshiina has quit IRC03:43
openstackgerritTakashi NATSUME proposed openstack/nova master: [placement] Add sending global request ID in put (3)
*** hshiina has joined #openstack-nova03:47
*** suresh12 has joined #openstack-nova03:49
*** markvoelker has quit IRC03:51
*** Tom-Tom_ has quit IRC03:52
*** yangyapeng has quit IRC03:55
*** hiro-kobayashi has joined #openstack-nova03:56
*** chyka has joined #openstack-nova03:56
*** links has joined #openstack-nova04:00
*** abhishekk has joined #openstack-nova04:05
*** chyka has quit IRC04:09
*** crushil_ has joined #openstack-nova04:10
*** Tom-Tom has joined #openstack-nova04:12
*** sapd has quit IRC04:14
*** Tom-Tom has quit IRC04:16
*** lyan has quit IRC04:17
*** dave-mccowan has quit IRC04:21
*** edmondsw has joined #openstack-nova04:22
*** suresh12 has quit IRC04:23
*** chyka has joined #openstack-nova04:26
*** edmondsw has quit IRC04:27
*** sapd has joined #openstack-nova04:28
*** chyka has quit IRC04:30
*** bhujay has joined #openstack-nova04:32
*** yangyapeng has joined #openstack-nova04:34
*** crushil_ has quit IRC04:36
*** zhurong has quit IRC04:39
*** tetsuro_ has joined #openstack-nova04:40
*** vivsoni has joined #openstack-nova04:46
vivsonihi team, i am having master setup created 2 days back, 'nova net-list' command not found04:46
vivsonierror: argument <subcommand>: invalid choice: u'net-list'04:47
vivsoniis this command deprecated ?04:47
*** gongysh has quit IRC04:47
*** markvoelker has joined #openstack-nova04:48
takashinvivsoni: 'nova net-list' has been removed in Pike. See .04:51
*** karthiks has quit IRC04:58
openstackgerritTakashi NATSUME proposed openstack/nova master: [placement] Add sending global request ID in get
vivsonitakashin: thanks :)05:01
jianghuawvivsoni, now you can use "openstack network list" instead.05:02
*** hongbin has quit IRC05:03
openstackgerritTakashi NATSUME proposed openstack/python-novaclient master: Add CLI to show instance usage audit logs
*** suresh12 has joined #openstack-nova05:04
*** Tom-Tom has joined #openstack-nova05:13
openstackgerritDeepak Mourya proposed openstack/nova master: Handle TZ change in iso8601 >=1.12.0
*** yangyapeng has quit IRC05:16
vivsonijianghuaw: ohk.. i was searching for that only. Thanks.. also it would be great if you can share link from which in future i can refer to05:20
*** markvoelker has quit IRC05:21
jianghuawvivsoni, the commands are included in openstack client. See here:
vivsonijianghuaw: Thanks !!!05:25
*** gongysh has joined #openstack-nova05:26
*** chyka has joined #openstack-nova05:30
*** trinaths has joined #openstack-nova05:33
*** chyka has quit IRC05:34
*** suresh12 has quit IRC05:38
openstackgerritMerged openstack/nova master: Fix instance evacuation with PCI devices
openstackgerritTakashi NATSUME proposed openstack/nova master: [placement] Add sending global request ID in put (1)
*** threestrands has quit IRC05:40
*** yangyapeng has joined #openstack-nova05:41
*** zhurong has joined #openstack-nova05:44
*** yangyapeng has quit IRC05:45
*** hiro-kobayashi has quit IRC05:50
*** ratailor has joined #openstack-nova05:51
*** chyka has joined #openstack-nova05:51
*** sapd has quit IRC05:51
*** gongysh has quit IRC05:52
*** gongysh has joined #openstack-nova05:55
*** oanson has quit IRC05:55
*** chyka has quit IRC05:56
*** sapd has joined #openstack-nova05:56
*** hshiina has quit IRC05:56
*** yamahata has quit IRC05:56
*** yangyapeng has joined #openstack-nova05:59
*** Dinesh_Bhor has quit IRC06:02
*** Dinesh_Bhor has joined #openstack-nova06:04
*** suresh12 has joined #openstack-nova06:08
*** Dinesh_Bhor has quit IRC06:08
*** edmondsw has joined #openstack-nova06:11
*** suresh12 has quit IRC06:12
*** Dinesh_Bhor has joined #openstack-nova06:14
*** Dinesh_Bhor has quit IRC06:14
*** edmondsw has quit IRC06:15
*** markvoelker has joined #openstack-nova06:18
openstackgerritTakashi NATSUME proposed openstack/nova master: [placement] Add sending global request ID in put (2)
*** jmlowe_ has joined #openstack-nova06:22
*** sapd has quit IRC06:24
*** sapd has joined #openstack-nova06:24
*** jmlowe has quit IRC06:26
openstackgerritTakashi NATSUME proposed openstack/nova master: [placement] Add sending global request ID in put (3)
*** lajoskatona has joined #openstack-nova06:30
openstackgerritTakashi NATSUME proposed openstack/nova master: [placement] Add sending global request ID in get
openstackgerritTakashi NATSUME proposed openstack/nova master: [cellv2] Improve getting BDMs in multiple cells
*** trinaths has quit IRC06:32
*** trinaths has joined #openstack-nova06:33
*** sree_ has joined #openstack-nova06:33
*** sree_ is now known as Guest8625006:34
*** sree has quit IRC06:36
*** ratailor has quit IRC06:39
*** ratailor has joined #openstack-nova06:40
*** tianhui has quit IRC06:45
*** tianhui has joined #openstack-nova06:45
*** hieulq has quit IRC06:45
*** markvoelker has quit IRC06:51
*** sridharg has joined #openstack-nova06:51
*** vipul has quit IRC06:54
*** dikonoor has joined #openstack-nova06:59
*** hieulq has joined #openstack-nova07:00
*** ttallgren has joined #openstack-nova07:01
*** edand has joined #openstack-nova07:02
*** oanson has joined #openstack-nova07:02
*** alexchadin has joined #openstack-nova07:05
openstackgerritshuangyang.qian proposed openstack/nova master: Fix the metadata re to match the Chinese
*** qsyqian has joined #openstack-nova07:15
*** rcernin has quit IRC07:17
*** andreas_s has joined #openstack-nova07:26
*** gszasz has joined #openstack-nova07:26
*** chyka has joined #openstack-nova07:27
*** Guest86250 has quit IRC07:32
*** sree has joined #openstack-nova07:32
*** yingjun has quit IRC07:32
*** ratailor is now known as ratailor|Lunch07:34
*** fragatina has joined #openstack-nova07:36
*** chyka has quit IRC07:36
*** avolkov has joined #openstack-nova07:40
*** slaweq has joined #openstack-nova07:47
*** markvoelker has joined #openstack-nova07:48
*** sapd_ has joined #openstack-nova07:48
*** sapd has quit IRC07:48
*** sahid has joined #openstack-nova07:49
*** pcaruana has joined #openstack-nova07:52
*** janki has joined #openstack-nova07:55
*** edmondsw has joined #openstack-nova07:59
*** matrohon has joined #openstack-nova07:59
openstackgerritshuangyang.qian proposed openstack/nova master: Fix the wrong user_id record in the log
*** edmondsw has quit IRC08:03
*** rgerganov has joined #openstack-nova08:07
*** Tom-Tom has quit IRC08:08
*** chyka has joined #openstack-nova08:08
*** Tom-Tom has joined #openstack-nova08:09
*** chyka has quit IRC08:13
*** Tom-Tom has quit IRC08:13
*** Tom-Tom has joined #openstack-nova08:14
*** qsyqian has quit IRC08:14
*** alexchadin has quit IRC08:15
*** AlexeyAbashkin has joined #openstack-nova08:15
*** alexchadin has joined #openstack-nova08:18
*** tesseract has joined #openstack-nova08:19
*** ratailor|Lunch is now known as ratailor08:20
*** markvoelker has quit IRC08:21
*** fragatina has quit IRC08:22
*** tssurya has joined #openstack-nova08:23
*** ralonsoh has joined #openstack-nova08:26
*** danpawlik has joined #openstack-nova08:29
*** zhaochao has quit IRC08:31
*** slaweq_ has joined #openstack-nova08:31
*** zhaochao has joined #openstack-nova08:31
*** slaweq_ has quit IRC08:36
*** giblet is now known as gibi08:39
gibigood morning nova08:39
Spazmoticmornin gibi08:40
*** damien_r has joined #openstack-nova08:41
SpazmoticUK Slowly waking up and my 3rd shift US butt over here just sitting over here all lonely :D08:41
*** tovin07 has joined #openstack-nova08:43
*** andreas_s has quit IRC08:43
*** tovin07 has quit IRC08:43
*** andreas_s has joined #openstack-nova08:44
*** sidx64 has joined #openstack-nova08:46
*** jpena|off is now known as jpena08:46
*** migi has quit IRC08:49
*** migi has joined #openstack-nova08:50
openstackgerritTakashi NATSUME proposed openstack/nova master: Transform instance-evacuate notification
*** slaweq_ has joined #openstack-nova08:53
*** andreas_s has quit IRC08:53
*** sidx64 has quit IRC08:56
*** gaurangt has quit IRC08:56
*** gaurangt has joined #openstack-nova08:56
*** sidx64 has joined #openstack-nova08:57
*** slaweq_ has quit IRC08:57
*** andreas_s has joined #openstack-nova08:58
*** alexchadin has quit IRC08:59
openstackgerritTakashi NATSUME proposed openstack/nova master: Fix a comment in a notification functional test
*** andreas_s has quit IRC09:07
*** andreas_s has joined #openstack-nova09:09
*** jaosorior has joined #openstack-nova09:10
bauzasgood morning folks09:10
*** damien_r has quit IRC09:13
ameedacan you please check my comments here ?
*** damien_r has joined #openstack-nova09:14
*** mgoddard_ has joined #openstack-nova09:16
*** andreas_s has quit IRC09:16
*** andreas_s has joined #openstack-nova09:17
*** Eran_Kuris has quit IRC09:19
*** markvoelker has joined #openstack-nova09:19
openstackgerritint32bit proposed openstack/nova master: Rollback original image info if server fail to rebuild
*** Eran_Kuris has joined #openstack-nova09:19
openstackgerritAlex Xu proposed openstack/nova master: placement: enable required traits from the flavor extra specs
*** markvoelker has quit IRC09:23
*** derekh has joined #openstack-nova09:25
*** _nick is now known as yankcrime09:31
*** gcb has quit IRC09:32
*** gszasz has quit IRC09:35
*** Tom-Tom has quit IRC09:36
*** Tom-Tom has joined #openstack-nova09:37
*** Tom-Tom has quit IRC09:37
*** Tom-Tom has joined #openstack-nova09:37
*** Tom-Tom has quit IRC09:38
*** Tom-Tom has joined #openstack-nova09:39
*** Tom-Tom_ has joined #openstack-nova09:43
*** Tom-Tom has quit IRC09:44
*** bhagyashris has joined #openstack-nova09:44
*** edmondsw has joined #openstack-nova09:47
*** edmondsw has quit IRC09:52
*** rmart04 has joined #openstack-nova09:52
*** raorn has joined #openstack-nova09:53
*** pooja_jadhav has joined #openstack-nova09:54
*** pooja-jadhav has joined #openstack-nova09:54
*** pooja-jadhav has quit IRC09:54
*** yamamoto has quit IRC09:57
*** chyka has joined #openstack-nova09:58
*** yamamoto has joined #openstack-nova09:58
*** chyka has quit IRC10:02
*** szaher has joined #openstack-nova10:02
*** yamamoto has quit IRC10:03
*** dtantsur|afk is now known as dtantsur10:05
*** sree has quit IRC10:06
*** sree has joined #openstack-nova10:06
*** yamamoto has joined #openstack-nova10:09
*** trinaths has left #openstack-nova10:10
*** yamamoto has quit IRC10:10
*** sree has quit IRC10:11
*** yamamoto has joined #openstack-nova10:12
*** mgoddard_ has quit IRC10:13
*** alexchadin has joined #openstack-nova10:16
*** yangyapeng has quit IRC10:18
*** takashin_ has joined #openstack-nova10:19
*** takashin has quit IRC10:21
openstackgerritTakashi NATSUME proposed openstack/nova master: api-ref: Fix parameter type in
*** sidx64 has quit IRC10:22
*** Tom-Tom_ has quit IRC10:23
*** Tom-Tom has joined #openstack-nova10:24
*** mgoddard_ has joined #openstack-nova10:25
*** Tom-Tom has quit IRC10:29
*** Tom-Tom has joined #openstack-nova10:29
*** liuzz has quit IRC10:30
*** tuanla____ has quit IRC10:30
*** tetsuro_ has quit IRC10:30
openstackgerritTakashi NATSUME proposed openstack/nova master: api-ref: Fix parameter order in rebuild
*** itlinux has joined #openstack-nova10:32
*** zhurong has quit IRC10:33
*** Tom-Tom has quit IRC10:34
openstackgerritsahid proposed openstack/nova stable/pike: Fixes 'Not enough available memory' log message
*** gszasz has joined #openstack-nova10:36
*** sidx64 has joined #openstack-nova10:37
*** yangyapeng has joined #openstack-nova10:38
*** sidx64 has quit IRC10:39
*** itlinux has quit IRC10:41
*** rmart04 has quit IRC10:42
*** yangyapeng has quit IRC10:44
*** Tom-Tom has joined #openstack-nova10:45
*** yangyapeng has joined #openstack-nova10:49
*** Tom-Tom has quit IRC10:49
*** zhurong has joined #openstack-nova10:51
*** yangyapeng has quit IRC10:53
*** sdague has joined #openstack-nova10:54
*** ratailor has quit IRC10:58
*** lpetrut has joined #openstack-nova11:00
*** sidx64 has joined #openstack-nova11:02
openstackgerritMatthew Booth proposed openstack/nova master: Pass DriverBlockDevice to driver.attach_volume
openstackgerritMatthew Booth proposed openstack/nova master: Use real block_device_info data in libvirt tests
openstackgerritMatthew Booth proposed openstack/nova master: Fix libvirt volume tests passing invalid disk_info
openstackgerritMatthew Booth proposed openstack/nova master: Pass disk_info dict to libvirt_info
openstackgerritMatthew Booth proposed openstack/nova master: Expose volume host type and path independent of libvirt config
openstackgerritMatthew Booth proposed openstack/nova master: Don't generate fake disk_info in swap_volume
openstackgerritMatthew Booth proposed openstack/nova master: Local disk serial numbers for the libvirt driver
*** abhishekk has quit IRC11:02
*** lpetrut has quit IRC11:03
*** ratailor has joined #openstack-nova11:05
*** andreas_s has quit IRC11:07
*** andreas_s has joined #openstack-nova11:08
*** andreas_s has quit IRC11:12
*** kwathore_ has quit IRC11:13
*** vivsoni has quit IRC11:13
*** kwathore has joined #openstack-nova11:14
*** vivsoni has joined #openstack-nova11:14
*** yangyapeng has joined #openstack-nova11:17
*** andreas_s has joined #openstack-nova11:17
*** markvoelker has joined #openstack-nova11:19
*** chyka has joined #openstack-nova11:20
*** chyka has quit IRC11:21
openstackgerritJiri Suchomel proposed openstack/nova master: Do not multiply megabytes with 1024 to get gigabytes.
*** annp has quit IRC11:22
*** links has quit IRC11:23
*** gaurangt has quit IRC11:24
*** gaurangt has joined #openstack-nova11:25
*** alexchadin has quit IRC11:28
openstackgerritint32bit proposed openstack/nova master: Rollback original image info if server fail to rebuild
*** gongysh has quit IRC11:30
*** andreas_s has quit IRC11:32
*** AlexeyAbashkin has quit IRC11:32
*** andreas_s has joined #openstack-nova11:33
*** openstackgerrit has quit IRC11:33
*** alexchadin has joined #openstack-nova11:33
*** links has joined #openstack-nova11:35
*** yangyapeng has quit IRC11:37
*** yangyapeng has joined #openstack-nova11:37
*** sidx64 has quit IRC11:39
*** AlexeyAbashkin has joined #openstack-nova11:41
*** andreas_s has quit IRC11:42
*** sidx64 has joined #openstack-nova11:43
*** janki has quit IRC11:43
*** andreas_s has joined #openstack-nova11:47
*** sree has joined #openstack-nova11:49
*** andreas_s has quit IRC11:49
*** andreas_s has joined #openstack-nova11:50
*** psachin has quit IRC11:53
*** sree has quit IRC11:53
*** markvoelker has quit IRC11:54
*** tesseract-RH has joined #openstack-nova11:55
*** chyka has joined #openstack-nova11:57
*** tesseract has quit IRC11:57
*** openstackgerrit has joined #openstack-nova11:59
openstackgerritRodolfo Alonso Hernandez proposed openstack/nova master: Format NIC features using os-traits definitions
openstackgerritRodolfo Alonso Hernandez proposed openstack/os-vif master: Add support for Windows network commands
openstackgerritBalazs Gibizer proposed openstack/nova master: Transform instance.exists notification
*** chyka has quit IRC12:02
*** sidx64 has quit IRC12:07
openstackgerritRadoslav Gerganov proposed openstack/nova master: WIP: VMware: implement update_provider_tree()
*** sidx64 has joined #openstack-nova12:08
*** janki has joined #openstack-nova12:10
*** dave-mccowan has joined #openstack-nova12:13
*** cdent has joined #openstack-nova12:17
*** bhujay has quit IRC12:18
*** bhujay has joined #openstack-nova12:18
*** krtaylor has joined #openstack-nova12:19
*** dave-mcc_ has joined #openstack-nova12:21
*** dave-mccowan has quit IRC12:22
*** sidx64 has quit IRC12:23
*** claudiub has joined #openstack-nova12:25
*** yikun_ has joined #openstack-nova12:28
SpazmoticPoor Xenapi code~12:28
*** ttsiouts has joined #openstack-nova12:29
openstackgerritLee Yarwood proposed openstack/nova master: DNM: Test LM with encrypted volumes
*** sidx64 has joined #openstack-nova12:33
openstackgerritint32bit proposed openstack/nova master: Refix disk size during live migration with disk over-commit
*** sidx64 has quit IRC12:33
*** jpena is now known as jpena|lunch12:33
*** smatzek has joined #openstack-nova12:33
*** sidx64 has joined #openstack-nova12:34
*** alexchadin has quit IRC12:35
*** bhujay has quit IRC12:39
*** phuongnh has quit IRC12:39
*** dikonoor has quit IRC12:39
*** phuongnh has joined #openstack-nova12:40
*** alexchadin has joined #openstack-nova12:42
*** smatzek has left #openstack-nova12:44
*** phuongnh has quit IRC12:49
*** markvoelker has joined #openstack-nova12:51
*** sidx64 has quit IRC12:52
*** ratailor has quit IRC12:53
openstackgerritLajos Katona proposed openstack/nova master: WIP: Add nested resources to server moving tests
*** yamamoto has quit IRC12:59
openstackgerritMatthew Booth proposed openstack/nova master: Automatically revert resize which fails on destination
*** dikonoor has joined #openstack-nova13:02
*** markvoelker has quit IRC13:02
*** markvoelker has joined #openstack-nova13:02
lyarwoodmdbooth: - Anything left to get your +1 on that?13:06
lyarwoodmdbooth: FWIW is testing this at the moment, I'll add a note if/when I respin the original.13:07
mdboothNope. Do you think you'll address the diff comment if you respin?13:07
lyarwoodmdbooth: ah yeah, I can look at that now13:08
mdboothlyarwood: You can have my +1 without it, though.13:08
*** tiendc has quit IRC13:08
lyarwoodmdbooth: if you could for now, just writing up a ML post to get additional reviewers before FF13:08
mdboothlyarwood: kk13:12
*** moshele has joined #openstack-nova13:13
*** chyka has joined #openstack-nova13:15
*** crushil_ has joined #openstack-nova13:15
*** artom has joined #openstack-nova13:17
openstackgerritJiri Suchomel proposed openstack/nova master: Do not multiply megabytes with 1024 to get gigabytes.
*** yamamoto has joined #openstack-nova13:19
*** chyka has quit IRC13:19
*** Tom-Tom has joined #openstack-nova13:21
alex_xucdent: thanks for
cdentalex_xu: does it do the right thing?13:22
alex_xucdent: working on it now13:22
*** jpena|lunch is now known as jpena13:26
*** gongysh has joined #openstack-nova13:27
*** zhurong has quit IRC13:28
efriedGreetings all13:29
efriedcdent Welcome back.13:29
cdenthold efried13:31
efriedThat too13:31
*** Eran_Kuris has quit IRC13:31
*** fanzhang has quit IRC13:31
cdentexpect me to leave again, because I left my house, therefore any moment now will be ill13:31
*** fanzhang has joined #openstack-nova13:32
*** andreas_s has quit IRC13:35
alex_xucdent: I guess that required to sort the response 'GET /allocation_candidates' API also, since that API response is unorder also13:35
cdentefried: is there a pending implementation of the in virt driver side of update provider tree somewhere?13:35
*** andreas_s has joined #openstack-nova13:35
efriedcdent Xen and VMWare are both working on that.13:36
cdentalex_xu: no, the idea is that whatever the order of the output, you can sort it in gabbi to alpha, and then compare with an alpha expected result13:36
alex_xucdent: ah, I got it13:36
cdentefried: yeah, was looking at the VMware one this morning (with the author) and was hoping for something with more flesh on the bones13:36
*** Drankis has joined #openstack-nova13:36
sean-k-mooneyalex_xu: qq do you know if ironic is tagging resouce providers for ironic nodes with traits? just want to chenc if your traits work will only be consumable with nova vm based driver in queeens or if you can use traits with ironic in queens also.13:37
cdentefried: because rado had some questions about the use of the 'generation' parameter in update_inventory13:37
efriedcdent Oh, I haven't seen the vmware one yet, though I chatted with rgerganov about it last week.  Link please?13:37
efriedah, okay.13:37
efriedLet me find the xen series, though I'm not sure it has any more "flesh"...13:37
efriedsean-k-mooney Yes, there's a series for that.13:38
cdentefried: see especially the commens at
efriedsean-k-mooney They're implementing a get_traits() driver method, which is going to be a kind of bastardized bridge between get_inventory and update_provider_tree, hopefully short-lived.13:38
sean-k-mooneyefried: oh ok cool. thats what i was hoping to hear. so provided the ironic discoverd or the admin applys traits to the resouce providers then it should work13:38
*** gcb has joined #openstack-nova13:39
efriedsean-k-mooney I believe that's the theory.  They're staying pretty close to the blueprint from what I've seen so far.13:39
alex_xusean-k-mooney: yea, efried already answer that13:39
sean-k-mooneyefried: ah ok so it will be ironic backend driver dependent then.13:39
efriedsean-k-mooney Let me find the series so you can go after the authors for specifics...13:40
sean-k-mooneyalex_xu: yep sound like ye have that covered :)13:40
alex_xuyea, my traits works just generic thing, needs each driver to report traits13:40
sean-k-mooneyefried: its on i can follow up myself dont put yourself out if you dont have it to had.13:40
alex_xucdent: it works \o/13:40
cdentalex_xu: yay!13:41
*** liverpooler has joined #openstack-nova13:41
efriedcdent Series is a bit wobbly at this point, but kinda starts here:
rgerganovefried, the doc string of provider_tree.update_inventory says that it will update the generation or the RP which I find strange13:42
efriedcdent (That's the xen one)13:42
cdentthanks efried13:42
efriedrgerganov The *placement* side will update the generation, yes.13:42
efriedcdent Here's where they actually implement update_provider_tree, which just calls the helpers in the other couple of patches in the "series":
efriedrgerganov There has been some confusion (not least on my part) about how the generation works.13:43
*** andreas_s has quit IRC13:43
*** andreas_s has joined #openstack-nova13:44
cdentefried: in this case the question is why is update_inventory accepting a generation param?13:44
efriedsean-k-mooney Here's mgoddard's series on ironic traits:
*** Eran_Kuris has joined #openstack-nova13:44
efriedcdent We have to send the generation as we know it back to the placement API so it can detect concurrent updates.13:44
cdentefried: yes, but doesn't the tree already know it?13:45
efriedcdent rgerganov I do a GET, and receive inventory at generation 1.  Joe-Bob's server over there does a GET and receives the same inventory, generation 1.  Now Joe-Bob does a PUT with a new version of the inventory, which is now at generation 2.  When I do my PUT, my payload would blow away Joe-Bob's.13:46
* alex_xu happy with dublin warm than beijing13:46
*** edmondsw_ has joined #openstack-nova13:46
efriedSo the solution: I send "generation 1" when I do my PUT, and placement says, "whoah, I'm at generation 2, 409".13:46
efriedSo I have to re-GET so I can make sure my updates to the inventory are still valid in the context of Joe-Bob's.13:47
efriedCan you dig it, dogg?13:47
*** itlinux has joined #openstack-nova13:47
openstackgerritAlex Xu proposed openstack/nova master: placement: support traits in allocation candidates API
openstackgerritAlex Xu proposed openstack/nova master: placement: using the dict format for the allocations
openstackgerritAlex Xu proposed openstack/nova master: placement: enable required traits from the flavor extra specs
cdentefried: so here the expectatin is that gen =  provider_tree.generation ?13:48
alex_xugibi: cdent yikun_ ^ updated13:48
cdentthanks alex_xu will look soon13:48
alex_xucdent: thanks13:48
efriedohh, cdent sorry, I was thinking of report client's update_inventory_for_provider_or_whatever_it's_called.13:49
efriedcdent Why does ProviderTree.update_inventory take a generation.13:49
efriedOne sec.13:49
efriedcdent rgerganov It's so report client can keep its internal cache up to date.13:50
efriedcdent rgerganov I agree it shouldn't be required - for any of the ProviderTree update_* methods - because update_provider_tree shouldn't use it, because update_provider_tree shouldn't be talking directly to placement.13:50
efriededleafe You've got the sched meeting today, yes?13:51
bauzasjianghuaw: hola13:51
bauzasjianghuaw: I'm hardly testing my vGPU changes for libvirt13:51
bauzasjianghuaw: I saw some problems due to libvirt13:51
bauzasjianghuaw: have you also tested for example suspending your instances ?13:52
*** pchavva has joined #openstack-nova13:52
sean-k-mooneyefried: thanks :) it will make my pm happy to know that this is well in hand. it was on our potential gaps list for a while.13:52
*** esberglu has joined #openstack-nova13:55
*** edmondsw_ is now known as edmondsw13:56
*** yangyapeng has quit IRC13:58
*** ttallgren has quit IRC13:58
edleafeefried: yes, scheduler meeting in 2 minutes in #openstack-nova-alt13:58
efriedOr #openstack-meeting-alt13:58
* efried rolls eyes13:58
edleafeand lack of caffeine :(13:59
bauzasnova alternative ?13:59
* bauzas looks at Intel ;)13:59
edleafetoo early to have to think clearly13:59
bauzasjust rename nova-alt to ciao14:00
bauzasshorter FTW14:00
jrollhey nova friends, the ironic rolling upgrade testing is down. while we try to track down why nova-conductor segfaults after upgrading ironic (without restarting n-cond), this patch will help us work around it by allowing nova queens to work with ironic pike (which is a good thing for users anyway). reviews would be super helpful, thank you :)
*** weshay is now known as weshay|rover14:00
sean-k-mooneybauzas: :) but its in go so obvioulsy better even though it has no epa/numa or any other kind of plathform awerness14:01
edleafeScheduler subteam meeting running now in #openstack-meeting-alt14:02
sean-k-mooneybauzas: by the way if its not clear i am not a ciao fan14:03
bauzassean-k-mooney: that's fine14:03
bauzassean-k-mooney: just say now you're a Kata fan :p14:04
bauzasbe corp, man14:04
sean-k-mooneybauzas: haha well kata is a merger of clear containers and hyper right. its more of a libvirt alternitve then nova as far as i understand14:04
bauzasthat's at least a big question I have in mind14:05
bauzasand I saw noone talking about that14:05
bauzasI should learn Go, if I'm not foolish14:05
sean-k-mooneyreading go is not that hard, reversing the type/name order of things when writing it will drive me nuts for at least 6 months if i ever try to write it14:06
*** sree has joined #openstack-nova14:06
edmondswgibi finucannot think you'll be able to get to the PowerVM SEA networking patch? should be easy after the OVS one14:07
finucannotedmondsw: Sure will. Got four series on my backlog but I'm working through an emulator threads bug today. Will probably be tomorrow, I'd say14:08
*** lyan has joined #openstack-nova14:08
*** finucannot is now known as stephenfin14:08
edmondswfinucannot thanks!14:08
*** itlinux has quit IRC14:09
openstackgerritEric Berglund proposed openstack/nova master: PowerVM Driver: SEA
edmondsw^ is just a rebase14:11
*** yamamoto has quit IRC14:17
*** tbachman has joined #openstack-nova14:18
*** yamamoto has joined #openstack-nova14:18
*** dillaman has quit IRC14:20
lyarwoodstephenfin: - Do you have time to go over this today? :)14:22
openstackgerritEric Berglund proposed openstack/nova master: PowerVM Driver: vSCSI volume driver
stephenfinlyarwood: Depends on how big it is. See above :) (tl;dr: /me side-tracked by a bug today)14:23
*** yamamoto has quit IRC14:23
*** alexchadin has quit IRC14:25
lyarwoodstephenfin: kk, pretty big but tomorrow (morning?) would be fine if that's possible14:25
stephenfinlyarwood: It's top of my list14:26
stephenfinSorry edmondsw :) You're next in line14:26
*** eharney has joined #openstack-nova14:26
edmondswstephenfin sure :)14:26
*** mriedem has joined #openstack-nova14:27
*** bhujay has joined #openstack-nova14:27
*** zhurong has joined #openstack-nova14:28
*** Guest86649 is now known as dansmith14:29
*** tssurya has quit IRC14:29
sean-k-mooneystephenfin: i was talking to rodoflo eairlier regarding he needs to move on to yardstick work which means he wont be able to work on this before the code freeze14:29
sean-k-mooneystephenfin: im going to try an pick it up14:29
sean-k-mooneystephenfin: you had some changes you wanted regarding the spec dict/object can i ping you later once i have it setup locally to confirm what needs to be done14:30
stephenfinsean-k-mooney: Sure can. I _think_ they make sense but I'll leave that to you to decide :)14:30
sean-k-mooneystephenfin: this barly missed pike then we made a lot of change in queens so dont want it to slip to rocky if it can be avoided14:31
stephenfinAgreed. I'd like to get that in, if at all possible14:31
*** yangyapeng has joined #openstack-nova14:32
sean-k-mooneystephenfin: cool am i need to get dan smit to look at that too as he previous gave feedback re using objects14:32
stephenfinsean-k-mooney: Yup, about versioning. I think the tl;dr: of it was that you can't really remove a field, even if it's unused, and type changes have to have backwards compatibility wrappers provided14:33
ameedajaypipes: are you around ?14:33
*** lucasagomes is now known as lucas-hungry14:34
sean-k-mooneystephenfin: yes he suggested synatsizing the new field form the old using a lazy loader if it was not set. i think rodlofo has that done i just want to make sure he is ok with the filed change you asked for too as i think that field existed before the patch so we cant just convert it to an object.14:35
*** jackie-truong has joined #openstack-nova14:35
sean-k-mooneystephenfin: i need to read the patch again since its been a few weeks since i did so i may be mis remembering14:36
*** yikun_ has quit IRC14:39
*** trozet has quit IRC14:39
*** takashin_ has left #openstack-nova14:40
*** takashin has joined #openstack-nova14:40
* edleafe goes to brew more coffee14:40
gibialex_xu: thanks for the update. I'm +2 on the bottom patch. I will review further in that chain soon14:43
efriedameeda I believe Jay is trying to find a spot to work from at the moment.14:43
*** tssurya has joined #openstack-nova14:43
gibiedmondsw: the SEA patch is on my list14:44
edmondswgibi great, tx14:44
ameedaefried: hehe, so can you help me ?14:44
efriedameeda Gosh, that depends.  What's going on?14:45
mriedemstephenfin: were you working on a nit fixes patch for the websocket proxy security series?14:45
ameedaefried: can you please check this "" and notice the scenario from the bug side ?14:45
stephenfinmriedem: Yup, it merged. Lemme know if there's stuff I missed
*** trozet has joined #openstack-nova14:46
mriedemstephenfin: ah ok - was just wondering if you wanted to update that docs patch for my one comment or do it in a follow up?14:46
efriedameeda Oh, this patch.  Yeah, I looked it over a bit last week and accepted that it's not really in my wheelhouse, sorry.14:47
stephenfinmriedem: If it's just that, I can edit on Gerrit14:47
stephenfinmriedem: reply left, in any case14:47
mriedemstephenfin: just edit inline and i'll +W14:48
bauzasmriedem: welcome back14:48
bauzasmriedem: for your pleasure, we have
openstackLaunchpad bug 1744325 in OpenStack Compute (nova) "If a rebuild is refused by the scheduler, the instance's imageref is not rolled back" [Critical,In progress] - Assigned to int32bit (int32bit)14:48
openstackgerritStephen Finucane proposed openstack/nova master: doc: Document TLS security setup for noVNC proxy
*** yamamoto has joined #openstack-nova14:49
stephenfinmriedem: Done (y)14:49
mriedembauzas: tagged for rc potential but not going to look at it for awhile14:49
bauzasmriedem: I'm following the change14:49
bauzasI'll see how it goes, and try to help next week if we haven't reached to a clear solution14:50
ameedaefried: thank you !14:51
mriedemit's not fun but i wouldn't say it's the worst bug i've ever seen,14:51
mriedemconsidering until queens, if you tried rebuilding a volume-backed server, we'd always fail to do so and still change the imageref on the instance14:51
*** jaypipes has joined #openstack-nova14:52
bauzasmriedem: hmmm, you're right14:54
*** yamamoto has quit IRC14:54
*** gongysh has quit IRC14:54
*** jaypipes has quit IRC14:55
*** moshele has quit IRC14:59
*** takashin has left #openstack-nova15:00
cdentedleafe, efried: would be good to get this stack merged before we start merging all the big stuff so that it is useful in the inevitable debugging we will need to do:
edleafecdent: ack15:04
*** awaugama has joined #openstack-nova15:04
*** archit has joined #openstack-nova15:04
Roamer`mriedem, welcome back! What are the chances of the StorPool volume attachment driver? I know it's close to the deadline, but you said if we got our ducks in a row... and the last duck, 533046, has been merged for a week15:05
*** namnh has joined #openstack-nova15:05
*** jaypipes has joined #openstack-nova15:05
Roamer`and right now we're in a weird position where the main use case for our Cinder and os-brick drivers is not in OpenStack proper :)15:05
mriedembauzas: comments inline in
bauzasmriedem: k, will look15:06
mriedembauzas: it's breaking evacuate potentially, and it's also missing rolling back a bunch of other properties, so it's probably not worth trying to rollback any properties since we'd be totally half assing it15:06
*** armax has joined #openstack-nova15:06
mriedemmight as well just put the instance into ERROR state and make the user rebuild with a valid image to fix things15:06
Roamer`mriedem, sorry, of course I'm talking about (certainly this is not the only thing on your plate...)15:06
mriedemRoamer`: i'll take a look15:07
bauzasmriedem: oh, excellent catch15:07
mriedemstill trying to get my bearings15:07
*** Tom-Tom has quit IRC15:07
Roamer`mriedem, of course, thanks a lot!15:07
mriedembauzas: can you re-approve ?15:07
*** Tom-Tom has joined #openstack-nova15:08
mriedemso we can get the series of novaclient microversion changes flushed for a final release15:08
bauzasmriedem: I'm half-reviewing, half-testing my own series, so apologies for not fully getting all the context in mind15:08
mriedembauzas: np15:08
bauzasmriedem: oh, the novaclient change is just a rebase ? +wd15:08
*** r-daneel has joined #openstack-nova15:09
bauzasmriedem: I tried to look at the multiattach thing while you were on vacations15:09
bauzasmriedem: that's an excellent job you made honestly15:09
bauzaswe're still missing the API bits, but I understand now the logic15:10
bauzasI like the fact to short-cut the multiattach if we don't support that by the compute15:10
*** crushil_ has quit IRC15:11
*** Tom-Tom has quit IRC15:12
mriedembauzas: i saw thanks for reviewing those,15:12
mriedemthe short circuit was copied from artom's tagged attach stuff15:12
*** bhujay has quit IRC15:13
*** Eran_Kuris has quit IRC15:14
efriedcdent I'm still pretty confused what you're getting at here
bauzasmriedem: btw. I have a question for you15:14
bauzasmriedem: I'm hardly-testing my vGPU changes on a dedicated box15:15
bauzasmriedem: and I'm testing all the instance operations15:15
bauzasmriedem: for providing a good release note about what can be supported and what not15:15
*** stelucz_ has joined #openstack-nova15:15
*** mlavalle has joined #openstack-nova15:15
openstackgerritMatthew Booth proposed openstack/nova master: Local disk serial numbers for the libvirt driver
efriedcdent If ironic gets its traits and they're not different from what's in the cache, we won't make the API call.  Which is as it should be.  Am I missing something?15:15
*** yamamoto has joined #openstack-nova15:15
bauzasmriedem: so, the point is, when I tried to suspend an instance, libvirt popped up an exception15:15
*** yamamoto has quit IRC15:15
cdentefried: I'm asking for the authors of the patch to acknowledge that they are aware of the quoted lines of provider tree code. This is because elsewhere in the stack there's been discussion of "this doesn't work with provider tree yet"15:15
bauzasmriedem: basically because we don't accept to hot-unplug mediated devices15:16
cdentIt could be all that is needed is "yeah, we know, it's fine"15:16
bauzasmriedem: now, the question is, would you be OK if I would just transform that libvirt exception in a more-readable exception like "sorry, but you can't suspend your instance because $vGPUs"15:16
*** stelucz_ is now known as stelucz15:17
bauzasand leave the instance into ERROR15:17
bauzasof course, the user can reset the state or rebuild15:17
*** larsks has left #openstack-nova15:17
mriedembauzas: i don't think the instance should be in ERROR state in that case,15:17
mriedemsince we didn't change the guest15:17
mriedemyou'd get a fault recorded as to why the suspend failed15:17
mriedemwhich is probably good enough15:18
bauzasset it back to ACTIVE15:18
bauzasmmm, good call15:18
*** sree_ has joined #openstack-nova15:18
mriedemyeah i guess you'd revert to the original vm_state, which right now can only be active in the API15:18
*** sree_ is now known as Guest876115:18
mriedemjohnthetubaguy: replied to your question in the multiattach api change - i think we're covered for the attach flow, but in a different way15:19
*** armax has quit IRC15:20
johnthetubaguymriedem: ah... got it15:20
*** edand has quit IRC15:21
*** armax has joined #openstack-nova15:21
*** hongbin has joined #openstack-nova15:22
*** sree has quit IRC15:22
cdentefried: so, based on your response are you aserting "don't worry, the provider tree will always be right because we've merged enough code for that to be true". If so, cool, but that wasn't clear from earlier discussion on the patches.15:22
*** lucas-hungry is now known as lucasagomes15:23
efriedcdent I'll grant you we still have work to do on concurrency management, though we've asserted that that's only a theoretical problem at the moment due to The Big Semaphore and the single-source-of-control-ness for compute node RPs at the moment.15:24
mgoddard_cdent, efried: I think a simple answer here is that the resource tracker will always have called set_inventory_for_provider prior to calling set_traits_for_provider, and this ensures that the RPs are present15:24
efriedmgoddard_ cdent Ohhh, are we worried about having populated the cache with the relevant provider at this point?  I didn't pick up on that at all.15:25
cdentefried: yes, my query has been, all along: is the provider tree active for this code path?15:25
efriedcdent That's definitely a legitimate concern, because as documented on set_traits_for_provider ( we don't attempt to create the provider.  But furthermore, we don't do _ensure_provider either, so there had better have been something prior that populated the cache for that guy.15:26
efriedcdent I'm sure I'm just being obtuse, but I didn't understand that from your comments at all :(15:27
mgoddard_efried: yes, it's set_inventory_for_provider15:27
efriedmgoddard_ That's good; and I think it's worth adding a code comment to that effect to affirm that it's been considered.  Good call cdent15:28
cdentmgoddard_: how/where does set_inventory_for_provider get involved in the management of the ProviderTree?15:29
efriedcdent It calls _ensure_resource_provider first thing.15:29
openstackgerritMatt Riedemann proposed openstack/nova master: Add the nova-multiattach job
efriedcdent Which creates the provider if it doesn't exist, but in any case populates/refreshes the cache15:30
cdentokay, that's the missing piece of the pie, thank you.15:31
mgoddard_efried: good call, I'll add a comment15:31
efriedcdent The other code path, update_compute_node, does the same (_ensure_resource_provider)15:31
cdentI just wanted to be sure, because of what people had said earlier about the potential for confusion15:31
*** trinaths has joined #openstack-nova15:32
cdentthose comments about potential for confusion had made it seem like there was a chance that the provider tree could either be wrong or even not exist15:32
cdentbut since it is established by _ensure_* is cool15:32
efriedcdent I doubt it's perfect15:32
cdentand how does that make you feel?15:33
efriedcdent Dirty.  So dirty.15:34
efriedcdent For one thing, as noted above, I've convinced myself that concurrency isn't an issue YET.15:35
efriedcdent And we're also working on the theory of merge big stuff early so we can shake out the bugs.15:35
efriedcdent Not that I think it's a great policy to count on shaking out bugs later rather than avoiding them by careful inspection beforehand...15:35
efriedcdent But we also can't get into analysis paralysis.  Gotta walk the line.15:36
cdenti like merge big stuff early15:36
cdentas long as we actually exercise15:36
edleafe"early" != "days before feature freeze"15:36
johnthetubaguymriedem: gibi: apologies, lots of things been getting in my way, but I am +2 on the multi-attach now, went back though the merged chain, I don't feel qualified for +W for some reason, but gibi you might be happy with that?15:37
efriedWell, unless feature freeze is deliberately early in the cycle, to allow time to exercise.15:37
*** zhaochao has quit IRC15:37
*** namnh has quit IRC15:38
mriedemjohnthetubaguy: thanks; just cleaning up the patch that adds the CI job so it's run in the check/gate queue15:38
*** namnh has joined #openstack-nova15:39
*** namnh has quit IRC15:39
johnthetubaguymriedem: ah, cool, I did see the -1 on there15:39
mriedemi think zuul got lost in the long chain of deps15:39
mriedemrechecking it15:39
*** namnh has joined #openstack-nova15:39
*** namnh has quit IRC15:39
*** namnh has joined #openstack-nova15:40
*** namnh has quit IRC15:40
mriedemjohnthetubaguy: sdague: we'll need this bug fix to get the multiattach job in - relies on not using pike UCA
*** namnh has joined #openstack-nova15:40
*** ameade has quit IRC15:40
*** markmc has quit IRC15:40
*** DuncanT has quit IRC15:40
*** andrewbogott has quit IRC15:40
*** catmando has quit IRC15:40
*** BlackDex has quit IRC15:40
*** Adri2000 has quit IRC15:40
*** ekhugen_alt has quit IRC15:40
mriedemwhich means you can't snapshot a paused instance with older libvirt15:40
*** markmc has joined #openstack-nova15:40
*** ekhugen_alt has joined #openstack-nova15:40
*** Adri2000 has joined #openstack-nova15:40
*** Adri2000 has quit IRC15:40
*** Adri2000 has joined #openstack-nova15:40
*** BlackDex has joined #openstack-nova15:40
*** catmando has joined #openstack-nova15:41
*** andrewbogott has joined #openstack-nova15:41
*** andrewbogott has quit IRC15:41
*** andrewbogott has joined #openstack-nova15:41
*** ameade has joined #openstack-nova15:41
*** DuncanT has joined #openstack-nova15:41
gibijohnthetubaguy: thanks for the review15:43
gibimriedem: is it OK for you that I +W the multiattach api patch as john is +2 on it or I should wait for new job to run?15:45
efriedrgerganov If you're willing to have rebased onto instead of I'll keep it up to date as I work on those WIPs.15:46
mriedemgibi: should be fine to +W - the CI results have already passed for awhile now15:46
efriedrgerganov At the moment you're off on a side branch15:46
mriedemi'm just changing the job config15:46
gibimriedem: OK, thanks15:46
*** yikun_jiang has quit IRC15:47
efriedgibi Thanks for the reviews!  Knocking 'em out today15:47
*** yikun has joined #openstack-nova15:48
*** felipemonteiro_ has joined #openstack-nova15:49
mriedemedleafe: looks like needs a rebase?15:50
*** slaweq has quit IRC15:50
*** tovin07 has joined #openstack-nova15:51
jaypipescdent: I see you liked my country music song joke.15:51
jaypipesthere's been a plethora of jokes and movie references in reviews on efried's latest patch series.15:52
jaypipesI've been having quite a bit of fun.15:52
*** yikun_jiang has joined #openstack-nova15:52
*** yikun has quit IRC15:52
edleafemriedem: working on it15:52
*** efried has left #openstack-nova15:52
*** efried has joined #openstack-nova15:52
edleafemultiattach stepped on RPC versions15:52
mriedemoh yeah15:53
mriedemit was a race15:53
amorinhello everybody15:53
*** david-lyle has joined #openstack-nova15:53
*** tidwellr has joined #openstack-nova15:53
jaypipesamorin: mornin.15:54
*** julienc has joined #openstack-nova15:54
amorinI'd like to know if there is a reason of this if iso9660 line:15:54
*** felipemonteiro__ has joined #openstack-nova15:55
amorinI mean, other possibility is vfat afaik15:55
amorinwhat if nova is transfering the config drive from remote if it is vfat?15:55
*** derekh has quit IRC15:56
*** derekh has joined #openstack-nova15:56
amorinjaypipes: evenin :p15:56
amorin(almost 5 pm here)15:56
tovin07mriedem, hi15:56
jaypipesamorin: well, evening then :)15:57
*** felipemonteiro_ has quit IRC15:58
jaypipesamorin: as for your question... no idea. perhaps mdbooth or lyarwood might know the answer on that one.16:00
* mdbooth looks16:00
amorinjaypipes mdbooth thanks16:00
*** bhujay has joined #openstack-nova16:01
mdboothamorin: What's the question?16:02
mdboothamorin: Ah, you're wondering why the handling difference between iso9660 and vfat?16:02
amorinwhat if we copy the config-drive no matter its kind (vfat or iso)16:02
efriedcdent "For future reference, in the future this loop could be replaced with a single request to POST /allocations, clearing the allocations for all the consumers." <== This must have been a difficult comment to write.  The war between "use lots of little API calls" and "use stuff I wrote!"  :P16:03
amorinI understand that libvirt is able to copy it if its vfat16:03
amorinbut it seems that if nova copy it first,16:03
mdboothIt was (is? but I doubt it) a bug in libvirt/qemu in the handling of iso9660 disks16:03
amorinthen libvirt will do nothing16:03
efriedcdent Only joshing you of course.  The POST is a great idea there.16:03
mdboothDid you look at the referenced lp bug?16:03
amorinmdbooth: yes16:03
amorinseems that libvirt is still failing with iso16:04
amorinI was just wondering if copying vfat with nova is a bad idea or not16:04
sean-k-mooneyamorin: mdbooth i would guesss the bug in libvirt is related to iso beeing treated as cdroms and vfat ect disk being considered hdds or somthin in that vain?16:05
*** tssurya has quit IRC16:06
sean-k-mooneyamorin: well one way to check would be remove that line and look at the livemigration gate jobs. it might result in both nova and neutron coping the config drive16:06
*** chyka has joined #openstack-nova16:06
openstackgerritMark Goddard proposed openstack/nova master: Send traits to ironic on server boot
openstackgerritMark Goddard proposed openstack/nova master: Add get_traits() method to ComputeDriver
openstackgerritMark Goddard proposed openstack/nova master: Implement get_traits() for the ironic virt driver
amorinproblem is, imagine you already spawn instances with iso kind, nova needs to copy it before (because of libvirt bug), but in the meantime, you updated the nova config, so CONF.config_drive = vfat16:06
amorinthen you never enter this if, and live-migration fail16:06
*** lpetrut has joined #openstack-nova16:07
mdboothsean-k-mooney: Pretty sure it's something like that. Long time since I had this cached.16:07
mdboothamorin: So, the other thing that happened since then is that we explicitly specify which disks to migrate using migrateToURI316:07
amorinsean-k-mooney: I did try to remove it in my lab, seems to work, I'll try to submit and patch and see the results in gate jobs16:07
amorinmdbooth: ok, got it, migrateToURI3 is then able to migrate the vfat config-drive16:08
amorinmdbooth: but are we sure that migrateToURI3 is able to handle the iso kind?16:10
amorini'll check16:10
mdboothIIRC it's because cd-rom drives are read-only, and qemu won't let us write to a read-only disk, even during live migration16:10
mdboothamorin: The pertinent point about migrateToURI3 is that we specify a list of disks to block migrate explicitly16:10
zzzeekjaypipes: my connection monitoring thing currently lets you put ?plugin=connmon on the SQLAlchemy URL. But Nova Cells shoves DB urls into the database the first time it runs and then they never change :(.  So need to add a config flag to oslo.db.  But then Nova hardcodes all the oslo flags :)16:10
mdboothWhereas before it was all disks16:10
sean-k-mooneyamorin: is there a reason we migate configdirve instead of recreating on remote side our of interest. technically with vfat it can be readwrite but they are intended to be readonly16:11
*** chyka has quit IRC16:11
sean-k-mooneymdbooth: ^16:11
mdboothamorin: So previously we didn't avoid the problematic disks, iirc16:11
mdboothsean-k-mooney: I believe we do recreate it now16:11
*** tidwellr has quit IRC16:12
*** tidwellr_ has joined #openstack-nova16:12
*** mvk has quit IRC16:13
tovin07this patch passed all zuul gate (had 2+2, A+1). however, it's still open.
jaypipeszzzeek: that is indeed the suck :(16:14
zzzeekjaypipes: this "nova puts the URLs into the DB16:14
zzzeek" thing has been a really huge problem16:14
jaypipeszzzeek: as opposed to, say, a config management system? :)16:14
zzzeekit is far and away the worst design decision16:14
zzzeekjaypipes: it means I cannot change the URL in the config file and have it take effect, *and* it means whatever is in the config file on one server is *implicitly shared* with all other nova servers on other machines16:15
jaypipeszzzeek: well, I don't disagree it's a bad design but it's not something we can "fix" right away...16:15
zzzeekjaypipes: we have all kind of workarounds.   it's just v v inconvenient to constantly hit it16:15
jaypipeszzzeek: is there a hack/workaround that will allow you to proceed with your ?plugin=XXX enhancements?16:16
*** yamamoto has joined #openstack-nova16:16
zzzeekjaypipes: yes I am going to add it to oslo.db so it's a separate config option outside of the URL16:16
jaypipeszzzeek: again, I don't disagree with you at all16:16
zzzeekjaypipes: which means when I document how to use connmon w/ nova, the answer will be "it depends :) "16:16
zzzeeklike older nova, OK there's no cells, put it in the URL.  newer nova, OK we have the olso.db thing.   middle-nova, erg, rewrite your cells URLs w/ the command line thing16:17
*** itlinux has joined #openstack-nova16:17
jaypipeszzzeek: another annoying this about oslo.cfg: you can't tell whether a CONF option has been set or not. i.e. has the CONF option value been set (via either argparse or configparser) to any value.16:17
sean-k-mooneystephenfin: just looking at the clean up you are asking for are not directly related to the patch rodlfo is doing so i should probably pull it out into another patch above or below rodolfos. is that ok? and do you have a prefernece?16:17
jaypipeszzzeek: s/this/thing16:17
zzzeekjaypipes: I might have observed the other day that putting something in [DEFAULT] doesn't actually "default" the value elsewhere?   i struggled for days getting region_name / os_region_name setup to work16:18
jaypipeszzzeek: yeah16:18
*** david-lyle has quit IRC16:19
bauzasdoes anyone know more than just by the name the "nova backup" command ?16:19
*** armax has quit IRC16:19
zzzeekjaypipes: and another thing!  why is the keystone auth URL in nova.conf but then if my region is wrong it seems to ask keystone for the *other* URL then it starts using that one (and fails) ?16:19
bauzasI guess the retention period is just for calculating whether we should delete an old image or not16:19
bauzasbut hopefully nova isn't periodically running a task to clean that up16:19
bauzas(ie. that's a per-call retention calculation)16:20
stephenfinsean-k-mooney: What cleanup is this now?16:20
*** Guest8761 has quit IRC16:20
bauzasI'm just amazed to discover such API after 5 years working for Nova16:20
*** sree has joined #openstack-nova16:20
bauzasnevermind, got my answer
sean-k-mooneystephenfin: basically the comment here
*** tssurya has joined #openstack-nova16:21
*** armax has joined #openstack-nova16:22
stephenfinsean-k-mooney: Aye, the lines I actually commented on are a cleanup task. However, my comments about what the object should actually look like are not16:22
sean-k-mooneystephenfin: they are against lines not modified by rodolfos patch so im wondering should it be a seperate patch or in this one16:22
mriedembauzas: note you can't use backup with a volume-backed instance, but you can snapshot a volume-backed instance, another fun wrinkle, and something people have wanted to fix for probably 5 years as well16:22
stephenfinsean-k-mooney: What I was suggesting is that rather than having two generic container fields on the object, we actually define the fields we want16:23
bauzasmriedem: but you can backup a volume, right?16:23
bauzasnot with nova, of course16:23
stephenfinsean-k-mooney: The two generic container fields being 'dict_of_lists' and 'dict_of_strings'16:23
bauzasanyway, I'm just testing my patches against
bauzaswhatever the backup does, it works16:24
mriedembackup creates a snapshot with a rotating retention period yeah16:24
jaypipeszzzeek: see: vestigial tail? :)16:24
stephenfinsean-k-mooney: Rather than using 'dict_of_lists', add a ListOpt for each key that we'd expect to store in there. Similarly, instead of 'dict_of_strings', add a ListOpt for each key16:24
*** namnh has quit IRC16:24
stephenfinDoes that make sense?16:24
*** lajoskatona has left #openstack-nova16:25
*** namnh has joined #openstack-nova16:25
bauzasmriedem: it tags the snapshot, I see16:25
*** sree has quit IRC16:25
bauzasmriedem: my question was just about what was enforcing that retention16:25
bauzasbut that's fine, I see that16:25
sean-k-mooneyok i can do that would you like it in this patch or a seperate one16:25
stephenfinsean-k-mooney: That one, please. If not, we're going to have to immediately issue a MAJOR version bump on the object so we can remove the 'dict_of_*' fields16:26
stephenfinWhich seems awfully silly :)16:26
*** itlinux has quit IRC16:26
*** yamamoto has quit IRC16:27
*** AlexeyAbashkin has quit IRC16:27
sean-k-mooneystephenfin: the network capablites will still need to remain a list of stings in the object as it can technicall by any trait includeing custom_ ones16:27
*** eharney has quit IRC16:27
sean-k-mooneystephenfin: ok will do.16:27
stephenfinsean-k-mooney: Yup, I'd expect to see 'capabilities = ListOpt(...)'16:27
stephenfinsean-k-mooney: Shhhhooound16:28
*** _pewp_ has quit IRC16:28
*** david-lyle has joined #openstack-nova16:28
sean-k-mooneyam that might also want to be an object actully capablities:{ network = listOpts(...);}16:29
*** _pewp_ has joined #openstack-nova16:29
sean-k-mooneyfor pci device we could have compute or other non netowrk capablites later16:29
*** vladikr has joined #openstack-nova16:30
*** namnh has quit IRC16:30
openstackgerritMatt Riedemann proposed openstack/python-novaclient master: Microversion 2.59 - Migrations list pagination
sean-k-mooneye.g. gpus could have opengl/cuda level as a trait but they are still pcidevices.16:30
lyarwoodkashyap: random question, have you ever had to file a bug against libvirt as shipped in UCA?16:31
*** zhurong has quit IRC16:31
kashyaplyarwood: Good random question16:31
kashyaplyarwood: I haven't, but what I did was---16:32
openstackgerritEd Leafe proposed openstack/nova master: Change compute RPC to use alternates for resize
lyarwoodkashyap: hitting this that I think is the result of libvirt being compiled without gnutls installed, somehow.16:32
kashyapJump on their IRC channel, and some friendly Ubuntu person did i16:32
edleafemriedem: jaypipes: ^^ rebased w/new RPC version16:32
kashyaps/did i/it/16:32
lyarwoodkashyap: kk, which channel?16:32
edmondswzzzeek if you're referring to the keystone_authtoken section, keystone url has to be there so that nova knows how to talk to keystone at all16:32
kashyaplyarwood: #ubuntu-dev16:32
*** tovin07 has quit IRC16:32
kashyaplyarwood: If it's a packaging but, #ubuntu-packaging16:32
zzzeekedmondsw: yes, but I have observed that once it does that initial talk to keystone thing, it looks into endpoints and gets whatever it finds there for keystone, then starts talking to that16:33
kashyapWhy are they compiling _out_ GnuTLS?  Libvirt depends on it, as you know16:33
edmondswzzzeek yep. Which should be all well and good unless you've got something misconfigured16:33
lyarwoodkashyap: no idea :)16:33
*** sree has joined #openstack-nova16:33
kashyaplyarwood: Try if you can find Christian Ehrhardt from Canonical on those channels16:34
kashyaplyarwood: He's the active libvirt dev from Canononical that I see16:34
kashyap(On the upstream lists)16:34
zzzeekedmondsw: sure.  these things are just v hard to debug in an HA setup16:34
*** namnh has joined #openstack-nova16:35
edmondswzzzeek yeah :( I'm not sure the exact reason there (or if there is a good reason). lbragstad would have a better idea16:35
*** moshele has joined #openstack-nova16:36
openstackgerritMerged openstack/python-novaclient master: Add support for the 2.57 microversion
openstackgerritMerged openstack/python-novaclient master: Microversion 2.58 - Instance actions list pagination
zzzeekedmondsw: well w regions I guess you are saying "here we'd prefer you use this address for this service" so even though you already gave it a "bootstrap" keystone, sure I'd probably like that it does that.  depends on wht you're using regions for16:37
edmondswzzzeek yeah, regions get all kinds of funky16:38
*** andreas_s has quit IRC16:39
*** namnh has quit IRC16:39
*** moshele has quit IRC16:40
kashyaplyarwood: Probably best to report it on the tracker, to save your time16:41
openstackgerritJackie Truong proposed openstack/python-novaclient master: Microversion 2.60 - Add trusted_image_certificates
kashyaplyarwood: I see UCA libvirt bugs are filed just "normally" like this --
openstackLaunchpad bug 1670220 in Fuel for OpenStack ocata "Libvirt service name is different for UCA" [High,Fix committed] - Assigned to Vladimir Kuklin (vkuklin)16:41
lyarwoodkashyap: kk, I created
openstackLaunchpad bug 1744758 in libvirt (Ubuntu) "libvirt 2.5.0 appears to be compiled without gnutls" [Undecided,New]16:42
*** julienc has left #openstack-nova16:42
kashyaplyarwood: Yeah, checking w/ a libvirt upstream dev, seems there's no 'special' flag, but certainly something is wonky in the build16:43
kashyapSo your bug looks fine16:43
*** andreas_s has joined #openstack-nova16:44
*** sridharg has quit IRC16:44
lyarwoodkashyap: yeah, there's no flag for this, we just check if gnutls_cipher_encrypt is available during the build.16:45
lyarwoodkashyap: thanks for asking :)16:45
*** jackie-truong has quit IRC16:46
*** penick has joined #openstack-nova16:46
*** jpena is now known as jpena|brb16:46
jaypipeslyarwood: quick ? for you on
lyarwoodjaypipes: looking16:47
*** chyka has joined #openstack-nova16:47
*** dikonoor has quit IRC16:47
lyarwoodjaypipes: yeah correct, _get_guest_xml calls down into _get_guest_storage_config that eventually calls _connect_volume16:50
*** tesseract-RH has quit IRC16:51
bauzascdent: saw your -1 but that seems a bit harsh given it's for grammatical issues16:52
bauzascdent: would you like to see the fixes by a follow-up ?16:53
*** _pewp_ has quit IRC16:53
bauzascdent: context is
*** andreas_s has quit IRC16:53
cdentbauzas: is fine as a followup, but it seemed pretty relevant since it was a release note16:53
cdenthowever, since there are later changes to the same release note, followups would be fine16:53
*** _pewp_ has joined #openstack-nova16:53
dansmithbauzas: I think getting the reno right is worthwhile, FWIW16:53
dansmithit's not just one typo16:54
bauzasdansmith: cdent: okay, I'm just afraid of any possible gate slowness so close by the milestone16:54
bauzasbut I'll spin another rev16:54
dansmithbauzas: I'll pounce on it as soon as you do16:54
bauzasI'll also clarify what has been tested and what not16:54
cdentjaypipes: feh, I struggle to remember '"special" filters and weighers'16:56
bauzasalso, I'm interested in getting some Oxford/Thesaurus link about the difference between a "this" and a "that" :p16:57
bauzasbecause we don't have that with "ce truc" :p16:57
cdentthis is here, that is there16:58
bauzasbut we have a "tu/vous" rule that is silly16:58
cdentenglish is full of silly16:58
dansmithcdent: heh, was trying to think of a way to explain that16:58
*** sree has quit IRC16:58
bauzasI have a 7-yr old daughter just learning French grammar16:58
*** trinaths has quit IRC16:58
bauzasnever say English is full of silly, or I'll just compare with French16:59
efriedMost Americans never learn English grammar, so...16:59
*** sree has joined #openstack-nova16:59
efriedbauzas this ~= celui-çi; that ~= celui-là17:00
bauzasthis = person, that = object ?17:00
openstackgerritJim Rollenhagen proposed openstack/nova master: Ironic: negotiate microversion to allow downgrade to Pike
*** suresh12 has joined #openstack-nova17:00
efriedjust ceci vs cela17:00
*** derekh_ has joined #openstack-nova17:00
jrollhey (again) nova friends, the ironic rolling upgrade testing is down. while we try to track down why nova-conductor segfaults after upgrading ironic (without restarting n-cond), this patch will help us work around it by allowing nova queens to work with ironic pike (which is a good thing for users anyway). reviews would be super helpful, thank you :)
efriedbauzas Mais je peux voir que "ça" peut servir pour les deux17:01
bauzasefried: 'Remove that code' looks to be wrong17:02
bauzasthat means "ce" to me17:02
bauzasjust rebasing now17:02
*** smatzek has joined #openstack-nova17:02
mriedemedleafe: ok comments inline17:02
efriedbauzas Depends on the context.  And it's definitely subtle.  If I highlighted a chunk and made a gerrit comment on it, I would sai "Remove this code".  If I was talking about some code somewhere else, or in the future, I would say "Remove that code".17:03
*** matrohon has quit IRC17:03
*** derekh has quit IRC17:03
*** felipemonteiro__ has quit IRC17:04
*** ralonsoh_ has joined #openstack-nova17:04
* efried doesn't speculate on who would say, "Keep all the code"17:04
bauzasefried: I see, it's more the intent itself17:05
bauzasI'll try to play with this17:06
bauzas(and not that)17:06
efriedBonne chance.17:06
edleafemriedem: will review in a bit17:06
efriedI'll argue English-is-sillier-than-French any day.17:07
*** fragatina has joined #openstack-nova17:07
*** ralonsoh has quit IRC17:08
*** bhujay has quit IRC17:08
*** Yikun has joined #openstack-nova17:10
*** yikun_jiang has quit IRC17:11
*** pcaruana has quit IRC17:11
*** sree has quit IRC17:11
jaypipeslyarwood: thx. just wanted to be sure.17:11
*** jpena|brb is now known as jpena17:11
*** moshele has joined #openstack-nova17:13
*** hieulq has quit IRC17:16
mriedemstvnoyes: your swap volume multiattach tempest test patch is a merge conflict with the resize test, i'm going to rebase those so they are in a series, otherwise i can't test them together17:16
*** janki has quit IRC17:17
*** armax has quit IRC17:18
openstackgerritEric Berglund proposed openstack/nova master: PowerVM Driver: SEA
*** armax has joined #openstack-nova17:20
esberglugibi: Removed the nova-net references from ^ so it's ready whenever you have a chance. Thanks for the reviews!17:22
*** smatzek has quit IRC17:22
*** smatzek has joined #openstack-nova17:23
*** archit is now known as amodi17:23
*** suresh12 has quit IRC17:24
*** moshele has quit IRC17:24
openstackgerritMatt Riedemann proposed openstack/nova master: [api] Allow multi-attach in compute api
openstackgerritMatt Riedemann proposed openstack/nova master: Add the nova-multiattach job
mriedemah crap17:25
mriedemgibi: accidental rebase on - want to +W again?17:25
*** jose-phillips has joined #openstack-nova17:26
*** smatzek_ has joined #openstack-nova17:27
*** smatzek has quit IRC17:28
*** suresh12 has joined #openstack-nova17:30
dansmithjroll: is that a thing you want separate from working around the segv issue? because I thought we always expect ironic to be newer and this changes that17:32
*** smatzek_ has quit IRC17:33
jrolldansmith: yes, we want to be able to go one release either way between ironic and nova. click the bug there, there's a plan where we drop most of the negotiation in the client to do so, this is just the quick way to do it17:34
*** suresh12 has quit IRC17:34
openstackgerritMerged openstack/nova master: doc: Document TLS security setup for noVNC proxy
*** smatzek has joined #openstack-nova17:35
mriedemefried: thanks for running the nova meeting last week17:35
efriedmriedem Sure thing, for what it was worth :)  I didn't update the agenda or anything, so it was a bit slipshod.17:35
mriedemwas fine17:36
mriedemyou are now, incidentally, the PTL for Rocky, congratulations; it was a trap.17:36
efriedhahahaha, so THAT's all it takes?17:37
dansmithjroll: mkay, I'm not sure how I feel about this.. the mechanism hasn't changed since this was titled "hacky negotiation to let us do a thing" and ... I agree17:37
*** tidwellr_ has quit IRC17:37
jrolldansmith: I mean, it's a bit of a hack but it works. the client patches are nearly done to do the better way, but it likely won't make this cycle. our options are basically do this or release without tested rolling upgrades, AIUI17:38
dansmithmriedem: you marked this bug as wishlist so I assume you're okay with it? (having a random previous ironic microversion the driver is willing to use)17:38
dansmithjroll: because some library is crashing conductor?17:39
jrolldansmith: suggestions welcome. especially if you can figure out the conductor exploding17:39
dansmithjroll: everything we add after this will have to either bump that "previous" thing or add infrastructure and handling for the newer feature, IFF the version is new enough17:40
edleafeefried: it's so cute that you think mriedem is kidding.17:40
dansmithjroll: is it some action that causes conductor to crash and is a restart enough? It's odd since conductor shouldn't be related to ironic at all17:40
mriedemdansmith: which bug?17:41
dansmithmriedem: well, it's a combo thing.. they want "N-1 ironic support in the driver" as well as this configdrive optionality which uses it, which I guess works around some other unrelated issue by chance17:41
jrolldansmith: debugging notes so far are at line 47 here: . as far as we can tell, upon upgrade, some shared library is being deleted, that is being loaded at runtime in the conductor17:42
jrollsee also
openstackLaunchpad bug 1744139 in Ironic "Grenade job fails due to systemd stopping n-cpu" [Critical,In progress] - Assigned to Julia Kreger (juliaashleykreger)17:42
dansmithjroll: right, so just a restart of the service because of that issue would be all we need right?17:42
dansmiththat seems like a reasonable thing to have in a from-foo script in grenade with a note about the shared library thing17:43
jrolldansmith: pike conductor won't start with queens version of global-reqs17:43
dansmithjroll: yikes, well that kinda breaks upgrades across the board for everyone then17:43
* jroll attempting to find details17:43
jrolldansmith: yeah, you're the first to respond to cries for help here :P17:44
mriedemjroll: note that dan and i were out last week too on unrelated adventures17:45
sean-k-mooneydansmith: only if you use systemd. if you use kolla you should be fine17:45
sean-k-mooneydansmith: well i guess it breaks the gate for everyone17:45
dansmithjroll: this N-1 patch is a really bad way to work around this grenade breakage, IMHO, because it sounds like you're suggesting changing the ordering to "fix" things with help from this17:45
*** penick has quit IRC17:45
jrollsean-k-mooney: s/use systemd/run other services co-located with nova conductor without some isolation/17:46
*** jdillaman has joined #openstack-nova17:47
jrolldansmith: don't disagree - I know smart people spent a lot of time debugging the real issue and didn't get very far. I jumped in to help write this patch, which is something we want anyway. (of course we'll need to document this special case as well)17:47
mriedemhas grenade ever supported n-1 nova-conductor with n requirements? i thought with multinode grenade, we always brought down the controller services and brought everything up with N code and dependencies, and left the subnode n-cpu at N-1?17:48
dansmithmriedem: right, conductor can't be backlevel17:48
jrollironic's grenade has always supported n-1 nova with n ironic17:48
dansmithzero detail on the reason for the segv in the logs it looks like17:48
sean-k-mooneyjohnthetubaguy: oh ya i had not read the full bug but i see its really a python lib issue so ya unless you use contaienrs or other isolation mechanisum(venvs ...) your kindo of screwed17:48
sean-k-mooney* jroll  not johnthetubaguy17:49
mriedemjroll: sure but that's different,17:49
dansmithjroll: we're talking about nova-conductor and the rest of nova17:49
*** Eran_Kuris has joined #openstack-nova17:49
mriedemand the release notes in nova have always said, "make sure you upgrade ironic first before nova" which is how we always hedged against version discovery/negotiation17:49
dansmithso I'm pretty hard -2 on working around this issue, which will break lots of people, by changing the upgrade order rules in the eleventh hour17:49
*** mjturek has joined #openstack-nova17:50
dansmithindependent of whether or not we support an n-1 ironic17:50
jrolldansmith: mriedem: right, we haven't upgraded nova at all. but upgrading ironic upgrades a number of libraries. we don't restart n-cond with the new libraries, but somehow at runtime it's causing these segfaults17:50
jrollsure, if someone can help fix the actual issue, I'm happy to kill this patch17:50
dansmithjroll: yep, I get that as the cause here17:50
*** hoonetorg has quit IRC17:50
dansmithwe probably need gate/devstack/grenade people help here to capture a core or something17:50
dansmithjroll: is there any temporal locality about when this started happening related to some bump in g-reqs?17:51
jrollsorry, was pointing out that how nova does grenade is irrelevant here17:51
*** amorin is now known as amorin[away]17:51
dansmithjroll: sure, the ordering of services within grenade shouldn't change to cover this up, is what Imeant17:51
jrolldansmith: not sure offhand, TheJulia do you know?17:51
* TheJulia context switches17:51
dansmithI see now why it affects conductor, because of the worker forking, which likely means it could affect other things the same way under load17:52
TheJuliaironic is a plugin, aiui, we're stuck with the order of ironic after nova17:52
openstackgerritJay Pipes proposed openstack/nova master: tests for alloc candidates with nested and traits
openstackgerritJay Pipes proposed openstack/nova master: add test for scenario with sum of child resources
openstackgerritJay Pipes proposed openstack/nova master: _get_trees_matching_all_resources() sum children
openstackgerritJay Pipes proposed openstack/nova master: add _get_usages_from_trees() function
openstackgerritJay Pipes proposed openstack/nova master: WIP - populate alloc cands for trees
openstackgerritJay Pipes proposed openstack/nova master: validate provider summary structure
dansmithno, ironic goes before nova, which is why this is happening, right?17:52
jrollwe disable the nova upgrade altogether in our rolling upgrades test17:53
dansmithah, well, then the same difference17:53
TheJuliathe operational order of upgade as defined in grenade is nova before ironic can be even upgraded, however things go sideways in the logs starting around swift/glance being updated17:53
openstackgerritSylvain Bauza proposed openstack/nova master: Recreate mediated devices on reboot
openstackgerritSylvain Bauza proposed openstack/nova master: libvirt: pass the mdevs when rebooting the guest
openstackgerritSylvain Bauza proposed openstack/nova master: WIP: Avoid suspending guest with attached vGPUs
dansmithgoing before isn't a solution because it changes the ordering, which you're forcing by just disabling the nova upgrade17:54
*** AlexeyAbashkin has joined #openstack-nova17:54
*** Tom-Tom has joined #openstack-nova17:54
TheJuliagoing before requires retooling grenade _or_ moving ironic's upgrade scripting into grenade17:54
TheJuliaboth have been objected to in the past17:54
dansmithand changing the expectations of the operators, documentation, and anything else that may be different as a result17:54
jrollsorry, we got off-track a bit - TheJulia, I meant to ask if you know if there are any g-r bumps around the same time as this started happening? (I'm attempting to find that right now)17:55
TheJuliaIronic has always stated that ironic must be upgraded prior to nova, however some nova contributors have expressed desire that ironic can be upgrded earlier. The API pin is what kills us17:55
*** mvenesio has joined #openstack-nova17:56
TheJuliajroll: I don't believe17:56
dansmithyou said "goes before" and "people want earlier" which doesn't seem different to me17:56
*** AlexeyAbashkin has quit IRC17:56
TheJuliaI don't believe so, however we had two different grenade breaks in about 24 hours17:56
dansmithwhy is nova-conductor not segfaulting like crazy after the thing that upgrades g-reqs (swift/glance) in other jobs?17:56
TheJuliaI think because it gets restarted17:56
dansmithor is it and we don't notice because we don't care until after?17:56
*** Eran_Kuris has quit IRC17:57
jrollI think nova gets upgraded first in most grenade runs, right?17:57
dansmithokay and so it matters for you because you never upgrade nova at the end?17:57
TheJuliathe state it is left in prior to that restart is referring to a bunch of modules/libraries/paths that have been deleted due to prior python module upgrades17:57
*** Eran_Kuris has joined #openstack-nova17:57
dansmithjroll: no I don't think so17:57
*** jaypipes has quit IRC17:58
TheJuliajust restarting didn't work because of a sqlalchemy change17:58
*** amodi has quit IRC17:58
TheJuliasince python requirements no longer match up because some processes have upgraded parts and pieces of the libraries17:58
*** Tom-Tom has quit IRC17:58
sean-k-mooneyTheJulia: ie ironic installed new sqlalcamy and since nova is not upgraded its not compatible?17:59
TheJuliasean-k-mooney: no17:59
TheJuliaIronic has not yet upgraded at that point17:59
TheJuliait is earlier on in the glance execution17:59
dansmithwhat is the sqlalchemy fail? because that seems like a bad thing17:59
*** derekh_ has quit IRC18:00
TheJuliayes, an argument was removed from sqlalchemy and oslo_db changed accordingly18:00
TheJuliaI believe18:00
* TheJulia looks in notes for link18:00
* jroll puts for the record the first message he sees in logstash is at 2018-01-17T09:52:37.119Z18:01
*** ralonsoh_ has quit IRC18:02
jrolllooks like oslo.middleware 3.34.0, oslo.utils 3.35.0, and osprofiler 1.15.1 are the most relevant g-r updates18:03
TheJuliajroll: but also, our grenade jobs were failing due to a neutron change for about 24-36 hours prior to that18:03
dansmithso some glue around this change might get us out of the jam?
sean-k-mooneyis this for master? g-r for SQLAlchemy on master has a minium of 1.0.10 and pike has an upperconstaint of 1.1.12 so you should be able to install master without needing to upgrade SQLAlchemy18:03
*** penick has joined #openstack-nova18:03
jrollTheJulia: oh right, were they getting into the upgrade bits?18:03
sean-k-mooneyi guess it upgrades because we allway install the newest requirement allowed in greade18:03
dansmithI'd guess that's just a library API breakage in oslo.db18:03
TheJuliajroll: what do you mean?18:04
jrollTheJulia: with the neutron failure, was that before or after the upgrade, do you remember?18:04
dansmithTheJulia: yeah, so that commit you linked from oslo.db needs to be graceful I think as it's an API breakage, AFAICT18:05
TheJuliabefore, neutron's upgrade would fail18:05
jrollTheJulia: ah, thanks.18:05
TheJuliabecause of networking-generic-swtich18:05
dansmithor, maybe since it was deprecated we need to backport a fix to nova pike to stop passing it?18:05
*** mgoddard_ has quit IRC18:05
jrolloslo.concurrency feels like a likely suspect in that range then, too (believe that touches some C libraries)18:06
mriedemhow is nova-conductor even picking up the new oslo.db if it's not being restarted?18:06
jrolldansmith: that would let us restart it, yeah, though that isn't ideal18:06
dansmithmriedem: worker forking18:06
jrollwell, there's two things going on18:07
*** hoonetorg has joined #openstack-nova18:07
jrollthe worker fork segfaults18:07
dansmithjroll: ah right I got lost that this can't be the segv issue, it's the breakage that prevents the restart, correct18:07
jrollif we restart the main process, it fails due to oslo.db18:07
dansmithright right18:07
mriedemah ok, so not an intentional restart18:07
mriedemsomething triggers a failure and restart, which then fails18:07
dansmithmriedem: it's just workers being cycled in and out I think, not failure related initially18:08
mriedemi'll go ahead and say i don't understand18:09
mriedemi welcome the ridicule18:09
sean-k-mooneydansmith:  the minium version of SQLAlchemy on master is below the max on pike currently. the commit TheJulia referenced does not seem to indicate what version of SQLAlchemy removed the retry arg. it sound like there is a min version bump missing also if that change is not graceful18:09
*** david-lyle has quit IRC18:09
*** dtantsur is now known as dtantsur|afk18:09
jrollmriedem: it looks like this
TheJuliaif the fork causes a dynamic library to be referenced that hasn't already been opened by the parent process, that would explain the segfault in that some of the things the parent was still running with that spanws the worker is gone because pip deleted them18:09
jrolland then systemd starts killing n-cpu and such, because insanity18:09
TheJuliaand then people begin drinking fine spirits18:10
*** moshele has joined #openstack-nova18:10
dansmithI have to run to a thing for a bit, back in a bit18:11
*** felipemonteiro has joined #openstack-nova18:12
jrollI feel like this is related but I can't prove it
sean-k-mooneyjroll: i would assume systemd did not kill n-cpu and it segfaulted by trying to deref a fuction pointer from a module that nologer existed due to the upgrade and systemd just noticed the process died.18:12
jrollsean-k-mooney: n-conductor does the segfaulting, n-cpu gets killed by systemd18:13
mriedemif this has been happening for <=10 days we could hopefully figure out when it started from logstash18:14
TheJuliajroll: that is correct18:14
*** jackie-truong has joined #openstack-nova18:14
jrollmriedem: we had another failure for about 24-36 hours before that, so it's a bit masked, but the first instance is 2018-01-17T09:52:37.119Z in logstash18:14
*** felipemonteiro_ has joined #openstack-nova18:15
mriedem was 1/1618:16
mriedemnot sure that commit should be a problem though since something has to opt into passing the new kwarg, else it defaults to the same as before18:18
*** jpena is now known as jpena|off18:18
jrollah true18:18
mriedemand i don't think we run with osprofiler enabled in this job do we?18:18
mriedemworth checking18:18
*** felipemonteiro has quit IRC18:18
jrollnot that I know of18:18
jrollI don't think any ironic jobs have ever enabled that18:19
*** jackie-truong has quit IRC18:19
mriedemJan 18 04:49:32.628690 ubuntu-xenial-inap-mtl01-0001976291 nova-conductor[19600]: DEBUG oslo_service.service [None req-db5ca533-98f7-4358-8448-0201fe9a04d8 None None] profiler.enabled               = False {{(pid=19600) log_opt_values /usr/local/lib/python2.7/dist-packages/oslo_config/}}18:19
mriedemyeah osprofiler isn't enabled18:19
mriedem was 1/1718:20
*** armax has quit IRC18:20
jrollcould be18:20
mriedemcould try reverting that back to 1.11.2 and run a depends-on with an ironic patch18:21
jrollnothing O_o18:21
*** Eran_Kuris has quit IRC18:21
*** jackie-truong has joined #openstack-nova18:21
jrolloh, they don't tag, good18:21
*** armax has joined #openstack-nova18:22
jrollwindows, py3, meh18:22
mriedemthat's just the stuff they call out in the release notes18:22
*** sree has joined #openstack-nova18:22
mriedembut yeah18:22
*** suresh12 has joined #openstack-nova18:23
mriedemnote the "and introduces a file handel leak  due to an   upstream bug in pyroute2"18:24
sean-k-mooneyi dont think os-vif is the cause as in 1.7 we did not use pyroute2 and in 1.9.0 we have disabled and use ip tools instead.18:26
jrolldoesn't look like os-vif gets upgraded, though:
sean-k-mooneyjroll: the only project useing os-vif are nova and kuyr-kubernetes18:27
*** sree has quit IRC18:27
efriedrgerganov yt?18:27
sean-k-mooneyim assumeing you dont have the later installed in this gate and you disable nova upgrade right ?18:27
mriedemyeah os-vif is 1.7.0 in pip freeze18:28
jrollso in this run, it started segfaulting at 05:49:48.64284018:28
jrollhere is grenade.txt at that time-ish
jrollgotta be one of those first few imo18:29
cdentefried: he's a time zone before me, and usually pretty sane with regards to going home, so he's probably not around18:29
*** felipemonteiro_ has quit IRC18:29
efriedcdent Swhat I figured, but it was worth a shot.  Perhaps you can see it:
jrollsimplejson does lots of C things:
* cdent looks18:31
mriedemsimplejson 3.13.2 has been in u-c for 2 months though18:32
mriedemso it's not that18:32
mriedemlooking at things in from 1/17 that are in that list18:32
mriedemand which nova usess18:32
*** pcaruana has joined #openstack-nova18:32
mriedemi think that would only be babel, sphinx and cffi, and runtime code doesn't use sphinx18:33
mriedemnor babel i don't think18:33
mriedemso my money is on cffi18:33
jrollcffi isn't being upgraded at that time though18:33
cdentefried: I think you're right18:34
efriedcdent Thanks for looking.18:34
efriedcdent I coded it all up as if I wasn't, then started writing tests...18:34
*** jackie-truong has quit IRC18:34
jrollmriedem: hrm, there's also 2018-01-18 05:49:32.589 | + /opt/stack/new/grenade/projects/50_neutron/ :   sudo apt-get -y install python-qpid18:34
edleafeefried: just looked, too, and I can't find any way it could be None, either18:35
efriededleafe Thank you.18:35
*** jackie-truong has joined #openstack-nova18:35
sean-k-mooneyjroll: the gate runs on rabbitmq but perhaps python-qupid has a dep that upgraded something18:35
jrollmriedem: oh, and from apt-get update about 3 minutes before crashy crashy
cdentefried: I think one of the thing that makes me confused about the ProviderTree stuff is that it behaves as if it is a strong Type. Which may make sense in this context, but I struggle to get used to it; it changes some idioms.18:35
efriedcdent What do you mean by "strong Type"?18:36
jrollnova definitely uses python-libvirt18:36
mriedemqpid shouldn't be getting installed18:36
edleafecdent: you mean static type?18:36
sean-k-mooneyfinding the lib that change however wont resolve the issue will it. this seams like a general class of problem.18:36
sean-k-mooneypython-libvirt is used by n-cpu but not the conductor18:36
jrollthere's definitely a possibility it's imported by the conductor, though18:37
jrollin some way18:37
*** jmlowe_ has quit IRC18:37
*** AlexeyAbashkin has joined #openstack-nova18:38
jrollsean-k-mooney: yes, it's a general class of problem with a known solution: don't use system packages :P18:38
cdentedleafe, efried: I guess static could work as well, but what I mean is that it is written as if we're in a strongly typed language and the ProviderTree itself is a (static) type, made of up of statically typed things. <- This is not a relevant though, really, I'm just trying to suss out some of the sources of my anxiety with ProviderTree so I can flush them.18:38
jrollI gotta step away for a bit so I can eat and stuff, sorry18:39
sean-k-mooneyjroll: i dont think the conductor is aware of the hypervisor so i dont think it would ever import phyton-libvirt. unless for the livemigration events?18:39
jrollcdent: short for stuff my face :D18:39
jrollsean-k-mooney: not on purpose, but it's a tangled web18:39
* jroll bbiab18:39
*** tidwellr has joined #openstack-nova18:40
mriedemconductor only talks to n-cpu over rpc18:40
cdentbut yes, edleafe, I'm noodling in a way that uses terms incorrectly18:40
edleafecdent: no worries; just wanted to be sure I understood your thoughts18:41
sean-k-mooneymriedem: ya i think python-libvirts use shoudl be limited to just the libvirt virt driver correct18:41
*** AlexeyAbashkin has quit IRC18:42
*** esberglu has quit IRC18:46
mriedemjroll: for when you get back, it looks like ovs was getting updated too around the time things started crashing
mriedem2018-01-18T05:46:42.826Z|05730|rconn|INFO|brbm<->tcp: connection closed by peer18:48
*** eharney has joined #openstack-nova18:48
*** esberglu has joined #openstack-nova18:49
mriedem2018-01-18T05:46:38.988Z|00007|jsonrpc|WARN|tcp: receive error: Connection reset by peer18:50
mriedemand neutron is upgraded right before ironic when things start blowing up18:51
*** cdent has quit IRC18:52
*** lpetrut has quit IRC18:54
*** links has quit IRC18:55
mriedem2018-01-18 05:46:41.213 | + /opt/stack/new/devstack/functions-common:stop_process:1528 :   sudo systemctl stop devstack@q-agt.service18:57
*** fragatina has quit IRC18:58
*** chyka has quit IRC19:00
*** jmlowe has joined #openstack-nova19:00
sean-k-mooneymriedem: looking at the systemd service files on my local system there is no depency between q-agt or n-cpu and n-cond or the supper conductor for that matter so that service stop should not propegate to nova services19:01
sean-k-mooneyif the connect reset is in ovs-vswitchd.txt.gz  it indicates teh q-agt was upgraded rather then ovs19:02
*** amodi has joined #openstack-nova19:03
mriedemyeah q-agt is being stopped at 5:46:4219:03
sean-k-mooneyso its likely one (proably several) of neutron deps is common with nova19:04
*** lucasagomes is now known as lucas-afk19:04
sean-k-mooneyin the normal upgrade sequence nova would upgrade before neutron correct19:04
sean-k-mooneyi mean technicall i dont think it has to but i think that is correct for greade19:05
*** penick has quit IRC19:05
mriedemneutron goes before nova19:05
sean-k-mooneyoh ok19:05
mriedemJan 18 05:48:49 ubuntu-xenial-inap-mtl01-0001976291 kernel: traps: nova-compute[1669] general protection ip:4a7f54 sp:7ffd13de8328 error:0 in python2.7[400000+2e9000]19:06
mriedemJan 18 05:49:48 ubuntu-xenial-inap-mtl01-0001976291 kernel: nova-conductor[20023]: segfault at d0 ip 0000000000500211 sp 00007ffdae038760 error 4 in python2.7[400000+2e9000]19:07
sean-k-mooneymriedem: ya so that basically a segfault19:07
mriedemyeah, and not helpful as to what failed19:07
*** jackie-truong has quit IRC19:07
*** david-lyle has joined #openstack-nova19:07
mriedemright before it:19:08
mriedemJan 18 05:49:36 ubuntu-xenial-inap-mtl01-0001976291 sudo[24307]:    stack : TTY=unknown ; PWD=/opt/stack/old/tempest ; USER=root ; ENV=http_proxy= https_proxy= no_proxy= PIP_FIND_LINKS= SETUPTOOLS_SYS_PATH_TECHNIQUE=rewrite ; COMMAND=/usr/local/bin/pip2.7 install -c /opt/stack/new/requirements/upper-constraints.txt -r /opt/stack/new/neutron/test-requirements.txt -e /opt/stack/new/neutron19:08
mriedemneutron is installing stuff19:08
*** AlexeyAbashkin has joined #openstack-nova19:08
sean-k-mooneywell it happend 5 seconds after we start installing the swift test-requirements
sean-k-mooneythe sudo process for install the swift test requiremetns pid:15728 end at
sean-k-mooneyso the compute agent segfault while we are upgradeing the test requiremets for swift19:13
*** sree has joined #openstack-nova19:14
sean-k-mooneythat said nothing in jumps out at me19:14
mriedemi don't know why test-requirement would be getting installed in a devstack env anyway19:15
mriedemtest-requirements should be for unit tests19:15
sean-k-mooneymriedem: that is a dispute i have had with devstack for 2-3 releases19:16
sean-k-mooneydevstack installs all test-requiremets systemwide19:16
sdaguetest-requirements also got overloaded by lots of projects as "optional requirements" and people put all their drivers in there19:16
sdagueit's never had clear definitions19:17
sdagueit was made to always install... 3 years ago?19:17
mriedemok so neutron reqs are installed here for queens19:17
mriedemJan 18 05:49:36 ubuntu-xenial-inap-mtl01-0001976291 sudo[24307]:    stack : TTY=unknown ; PWD=/opt/stack/old/tempest ; USER=root ; ENV=http_proxy= https_proxy= no_proxy= PIP_FIND_LINKS= SETUPTOOLS_SYS_PATH_TECHNIQUE=rewrite ; COMMAND=/usr/local/bin/pip2.7 install -c /opt/stack/new/requirements/upper-constraints.txt -r /opt/stack/new/neutron/test-requirements.txt -e /opt/stack/new/neutron19:17
sean-k-mooneysdague: yes but if they are optional they should only be installed in the gates that need them :)19:17
mriedemthen nova-conductor segfaults Jan 18 05:49:48 ubuntu-xenial-inap-mtl01-0001976291 kernel: nova-conductor[20023]: segfault at d0 ip 0000000000500211 sp 00007ffdae038760 error 4 in python2.7[400000+2e9000]19:17
mriedemand nova-conductor is pike at this point19:18
sean-k-mooneysdague: yep but it only bit me with the flake8 import ording plugin that was added then blacklisted to nova test-requriement that ended up brakeing our ovs-dpdk gates because ovs enabled pip but disable that check internally19:19
*** sree has quit IRC19:19
sean-k-mooneymriedem: well that appears to be the same memory adress in both python2.7[400000+2e9000]19:19
*** AlexeyAbashkin has quit IRC19:20
sean-k-mooneymriedem: this is a time issue tool any of the previous pip installs could have cased it, as we cant know how long it will be betwen the module being replaced and the dangeling function pointer being deferenced19:22
sean-k-mooney* timing issue too19:23
openstackgerritMark Goddard proposed openstack/nova master: Implement get_traits() for the ironic virt driver
*** itlinux has joined #openstack-nova19:23
*** jaypipes has joined #openstack-nova19:24
*** sree has joined #openstack-nova19:25
efriededleafe jaypipes Just found a nice leetle bug19:25
sean-k-mooneysdague: by the way that was not a did at you. i know why you made the change i.e the multiple code paths and the dep issue it was causeing.19:25
mriedemhmm, so n-cpu starts failing around 5:46 and fails to connect to the ironic api19:26
efriedsdague too19:26
openstackgerritMark Goddard proposed openstack/nova master: Send traits to ironic on server boot
openstackgerritMark Goddard proposed openstack/nova master: Add get_traits() method to ComputeDriver
openstackgerritMark Goddard proposed openstack/nova master: Implement get_traits() for the ironic virt driver
mriedemand i think that's around when glance is being upgraded/restarted19:26
*** penick has joined #openstack-nova19:26
mriedemmaybe some weird uwsgi stuff?19:26
efriededleafe jaypipes sdague The bug is that e.g. [] is a valid json payload.19:27
sdagueefried: nice19:27
jaypipesefried: sure, I suppose. though I prefer DELETE :)19:28
sean-k-mooneykeysonten is dead from 5:4719:28
*** itlinux has quit IRC19:28
efriedjaypipes No DELETE API for aggs :(19:29
owalshdansmith: hey, hitting an issue with cell_v2 setup
openstackLaunchpad bug 1744780 in tripleo "Nova::Cell_v2::Simple_setup/Nova_cell_v2[cell0] fails on master promote" [Critical,Triaged]19:29
jaypipesefried: ooh, yeah...19:29
mriedem[Thu Jan 18 05:50:49.905807 2018] [wsgi:warn] [pid 30759:tid 139966291453824] mod_wsgi: Compiled for Python/
jaypipesefried: prolly should have one, but yeah19:29
mriedem[Thu Jan 18 05:50:49.905865 2018] [wsgi:warn] [pid 30759:tid 139966291453824] mod_wsgi: Runtime using Python/
*** sree has quit IRC19:30
mriedemmtreinish: is anything using this code yet?
mriedemi guess not
melwittowalsh: that's a recent change that we're not allowing creation/update of a cell with a DB connection + transport URL combination that already exists19:33
owalshmelwitt: yea, but it's not removing the target cell from the cells it checks for dups19:34
owalshmelwitt: so it fails on itself19:34
*** mgoddard has quit IRC19:36
* owalsh biab19:36
*** mgoddard has joined #openstack-nova19:36
efriedjaypipes edleafe sdague FYI19:37
openstackLaunchpad bug 1744786 in OpenStack Compute (nova) "SchedulerReportClient.put with empty (not None) payload errors 415" [Undecided,New]19:37
jrollmriedem: looks like n-cpu is because we shut down ironic-api while upgrading things (which seems weird to me, but)19:40
*** itlinux has joined #openstack-nova19:43
*** pcaruana has quit IRC19:43
*** sree has joined #openstack-nova19:45
melwittowalsh: I see what you mean. so the case you're hitting is trying to update_cell with only one of DB connection or transport URL being different then?19:46
dansmithmelwitt: yeah I expect so19:46
*** itlinux has quit IRC19:46
*** fragatina has joined #openstack-nova19:47
melwittyeah, just looking at the unit tests and the case of both being different should go through. guh19:47
dansmithmelwitt: er, what I meant was I think they're likely just running update_cell all the time, expecting it to be idempotent19:47
dansmithso it might not be different, but the pre-check finds an existing (identical) record and so it fails19:48
melwittoh, I see19:48
dansmithwhere it should only fail if they set cell1  to the same thing as cell219:48
* melwitt nods19:48
mriedemstephenfin: jaypipes: i don't see the reason for this change at all
mriedemthose tests all default to send 2.1 so explicitly hard-coding something isn't necessary19:50
*** sree has quit IRC19:50
melwittdansmith: do you want to cook up a patch for that or should I?19:51
dansmithmelwitt: grilling one currently19:51
jaypipesmriedem: that was originally in one of jackie's other patches that was unrelated and I had asked her to split it out.19:53
*** suresh12 has quit IRC19:54
openstackgerritEric Fried proposed openstack/nova master: Report Client: PUT empty (not None) JSON data
efriedsdague jaypipes dansmith ^19:54
efriedWould rather not put it in the dep chain of the update_provider_tree series, but it's gonna foul my test cases for update_from_provider_tree (which is how I found it)19:54
mriedemjaypipes: ok19:54
jaypipesmriedem: without it, patches that test the new certificates REST API functionality were failing due to the personality stuff still being around for some reason19:54
mriedemnew tests should have just overrode the personality files stuff rather than need to hack up old tests19:55
mriedemand...the mailman is stuck outside my house19:56
openstackgerritSen Yang proposed openstack/python-novaclient master: Implement hypervisor hostname exact patten match for server cold migrate
mriedemhe knows i work from home, so i guess i have to suit up now19:56
* efried admires mriedem for letting the mailman hang out for an extra few seconds while he updates us.19:56
*** slaweq has joined #openstack-nova19:57
openstackgerritDan Smith proposed openstack/nova master: Fix update_cell to ignore existing identical cells
dansmithmelwitt: owalsh: ^19:59
*** dave-mcc_ is now known as dave-mcowan19:59
*** dave-mcowan is now known as dave-mccowan19:59
*** dave-mccowan is now known as dave-mcc_20:00
*** jmlowe has quit IRC20:00
*** namnh has joined #openstack-nova20:02
*** armax has quit IRC20:02
*** namnh has quit IRC20:03
*** namnh has joined #openstack-nova20:03
*** armax has joined #openstack-nova20:03
*** suresh12_ has joined #openstack-nova20:04
*** harlowja has joined #openstack-nova20:04
*** jmlowe has joined #openstack-nova20:06
*** chyka has joined #openstack-nova20:06
*** namnh has quit IRC20:07
*** armax has quit IRC20:08
openstackgerritEd Leafe proposed openstack/nova master: Change compute RPC to use alternates for resize
edleafemriedem: ^^ responded to your comments20:09
*** sree has joined #openstack-nova20:12
*** amodi has quit IRC20:13
dansmithmelwitt: I think you meant assertFalse() when you pointed out my mind-o20:13
openstackgerritDan Smith proposed openstack/nova master: Fix update_cell to ignore existing identical cells
dansmithmelwitt: I had another bug in the test that that uncovered, but the functional code worked thankfully20:14
melwittdansmith: yes, heh. I was thinking 0 == no error == True for some reason20:14
*** Drankis has quit IRC20:15
mtreinishmriedem: yeah, not yet. But I need get that landed in glance soon20:15
*** kwathore_ has joined #openstack-nova20:16
*** sree has quit IRC20:17
*** suresh12_ has quit IRC20:19
*** kwathore has quit IRC20:20
*** suresh12 has joined #openstack-nova20:20
mtreinishmriedem: were you looking to use that option for something?20:22
*** avolkov has quit IRC20:23
*** amodi has joined #openstack-nova20:26
*** moshele has quit IRC20:27
*** penick has quit IRC20:32
*** namnh_ has joined #openstack-nova20:32
*** mhenkel_ has quit IRC20:32
*** mjturek has quit IRC20:33
*** namnh_ has quit IRC20:33
*** namnh_ has joined #openstack-nova20:33
*** johnthetubaguy has quit IRC20:33
*** jcosmao has joined #openstack-nova20:36
*** itlinux has joined #openstack-nova20:38
*** smatzek has quit IRC20:38
*** mhenkel has joined #openstack-nova20:39
*** jmlowe has quit IRC20:39
*** johnthetubaguy has joined #openstack-nova20:40
*** awaugama has quit IRC20:40
*** jcosmao has left #openstack-nova20:41
*** pooja has joined #openstack-nova20:41
*** sree has joined #openstack-nova20:44
*** namnh_ has quit IRC20:45
owalshmelwitt, dansmith: LTGM, thanks!20:45
*** r-daneel has quit IRC20:46
*** sree has quit IRC20:49
*** gszasz has quit IRC20:51
ildikovgibi: johnthetubaguy: mriedem: thank you!!! :)20:52
bauzasmriedem: dansmith: gibi: rebased for cdent's comments about relnotes20:53
*** mhenkel has quit IRC20:54
*** andreas_s has joined #openstack-nova20:56
*** suresh12 has quit IRC20:58
*** liverpooler has quit IRC20:59
*** mhenkel has joined #openstack-nova21:00
*** andreas_s has quit IRC21:00
*** Tom-Tom has joined #openstack-nova21:03
openstackgerritEric Berglund proposed openstack/nova master: Add PowerVM to feature-classification
*** r-daneel has joined #openstack-nova21:04
*** sree has joined #openstack-nova21:04
*** mhenkel has quit IRC21:04
*** dave-mcc_ has quit IRC21:05
*** penick has joined #openstack-nova21:05
*** suresh12 has joined #openstack-nova21:06
mriedemmtreinish: no, trying to figure out why the ironic multinode grenade job starting having random seg faults since 1/1721:06
*** mhenkel has joined #openstack-nova21:06
poojaHello! Is it possible to use the Placement API in Newton with a non-cells deployment?21:06
mriedempooja: yes21:07
poojamriedem: Thanks! I am trying to go from Newton non-cells to Pike release with cells v2.21:07
*** Tom-Tom has quit IRC21:08
*** jackie-truong has joined #openstack-nova21:08
poojaSo adding placement api in Newton and then upgrading packages to Pike and creating nova_cell0 and nova_cell1 databases would work?21:08
*** sree has quit IRC21:09
mriedempooja: placement and cells v2 don't really have anything to do with each other, so yes that should be fine21:09
poojaah okay, thanks! My understanding was it needed a cell for filtering hosts.21:09
mriedemno placement doesn't know anything about nova internals21:13
mriedemor deployment setup like cellsv221:13
dansmithand cells are not visible outside nova21:13
*** esberglu has quit IRC21:16
*** esberglu has joined #openstack-nova21:16
mriedemjesus tempest-full is now 2 hours long21:17
mriedemwell maybe depends on the node you land on21:18
mriedemrax dfw must be nice and slow21:19
melwittyeah, I was about to say we've been getting lots more job run fails related to timeouts, and I'm not sure if it's about the nodes or if it's about tempest run time creeping over the timeout line21:19
*** esberglu_ has joined #openstack-nova21:19
mriedemone tempest-full i saw was 2 hours and 8 minutes, another was 1 hour and 20 minutes21:20
melwittah, okay21:20
*** esberglu has quit IRC21:20
*** sahid has quit IRC21:21
*** sahid has joined #openstack-nova21:22
mriedemseeing rpc messaging timeouts in cinder logs too on rax dfw21:25
*** suresh12 has quit IRC21:26
*** jaypipes has quit IRC21:27
*** amodi has quit IRC21:28
openstackgerritCarlos Goncalves proposed openstack/os-vif master: [DNM] vif_plug_ovs: add should_provide_port
*** dave-mccowan has joined #openstack-nova21:30
*** armax has joined #openstack-nova21:32
*** jmlowe has joined #openstack-nova21:34
openstackgerritMatt Riedemann proposed openstack/nova master: Start moving users of parted to privsep.
*** jaypipes has joined #openstack-nova21:40
*** suresh12 has joined #openstack-nova21:40
*** gszasz has joined #openstack-nova21:44
*** lpetrut has joined #openstack-nova21:47
*** jaypipes has quit IRC21:49
*** gszasz has quit IRC21:50
*** gszasz has joined #openstack-nova21:50
*** edmondsw has quit IRC21:52
*** dave-mccowan has quit IRC21:52
*** dave-mcc_ has joined #openstack-nova21:52
hongbinmriedem: hi matt, a question. i am looking at this bug: , the use case there seems to require nova to add an api to support filter instances by floating ip. do you know if it makes sense to have this api in nova?21:53
openstackLaunchpad bug 1723026 in neutron "[RFE]Support get device_ids from floatingips" [Wishlist,Confirmed] - Assigned to Hongbin Lu (
*** liverpooler has joined #openstack-nova21:58
*** yamahata has joined #openstack-nova22:00
*** pchavva has quit IRC22:01
*** mvenesio has quit IRC22:06
*** yamahata has quit IRC22:11
mriedemhongbin: that doesn't seem like a nova api change, sounds like a neutron api change22:11
mriedemhongbin: nova has deprecated all of it's floating IP related APIs since they were just proxies to neutron and nova-network is deprecated22:13
*** jaosorior has quit IRC22:13
hongbinmriedem: ack, i see22:13
mriedemhongbin: is that RFE asking that GET /servers adds a query parameter to filter instances by floating IP substring? like the "ip" filter which filters on the fixed IP?22:14
hongbinmriedem: thanks for the remarks22:14
hongbinmriedem: per my understanding, yes22:14
*** mvk has joined #openstack-nova22:15
hongbinmriedem: but if nova thinks this doesn't make sense, i need to re-discuss this ref in neutron22:15
mriedemwell, i could see the floatingip filter param to GET /servers maybe, but it's just another proxy;22:16
mriedemthe best thing is probably to add something to neutron to allow filtering ports by floating IP substring?22:16
*** slaweq has quit IRC22:17
hongbinyes, if nova is removing proxies to neutron, then this request doesn't seem to fit into the scope22:17
hongbinyes, this can be done22:18
*** openstackgerrit has quit IRC22:18
hongbinhowever, the question from neutron team is if this will be used by nova22:19
mriedemit won't,22:19
*** rcernin has joined #openstack-nova22:19
mriedemunless we plumb something into the compute API to proxy that through when listing servers22:19
hongbinyes, get it22:19
hongbinmlavalle: ping, fyi (above)22:20
*** sree has joined #openstack-nova22:20
mriedemif the requestor says, "well then i get the device_id from the port(s) and then i have to call the compute API to get server details and that's too much work" then i'd ask what they need from the server details beyond just the server uuid22:20
*** openstackgerrit has joined #openstack-nova22:21
openstackgerritMerged openstack/nova master: libvirt: Collocate encryptor and volume driver calls
openstackgerritMerged openstack/nova master: libvirt: Introduce disk encryption config classes
openstackgerritMerged openstack/nova master: [api] Allow multi-attach in compute api
efriedmriedem dansmith Easy fix, one +2 already, can I get a quick look please?
hongbinmriedem: my understanding is that they want a new filter in nova (i.e. nova list --floating-ip <xxx>), and under the hook, nova calls neutron api with the device_id to query the floating ips22:23
hongbinand the addition of device_id in neutron side is for nova to query the floating ips fast22:24
*** lpetrut has quit IRC22:24
mriedemhongbin: we do'nt know the device_id until we get the ports, which would be filtered by the floating IP the user passes in22:24
*** amodi has joined #openstack-nova22:24
*** sree has quit IRC22:25
*** lpetrut has joined #openstack-nova22:25
hongbinmriedem: yes, so the proposal said "make the device_id of floating ip same as the instance uuid"22:26
hongbinthen, nova uses the uuid as the device_id to query22:26
hongbins/uuid/instance uuid/22:27
*** claudiub has quit IRC22:27
mriedemso (1) list all instances, (2) then call neutron with all of those instance uuids to get the ports AND also filter the floating IPs from that list of ports by the provided floating IP substring match?22:28
mriedemthe port.device_id is the same as the instance uuid already22:29
mriedemthe port and floating IP have a relationship somewhere22:29
mriedemso if you can list/filter ports by floating IP, you can build a list of server uuids22:29
sean-k-mooneymriedem: i think the owner of teh floating ip in neutron is the port uuid22:29
*** lpetrut has quit IRC22:30
sean-k-mooneyso really what you want to do is say neutron give be the owner of the port that has this uuid22:30
*** lpetrut has joined #openstack-nova22:30
sean-k-mooney* floating ip22:30
hongbin(2) is performed only if users insert the --floating-ip filter when listing the instances22:31
*** lyan has quit IRC22:33
*** jackie-truong has quit IRC22:34
hongbinmriedem: yes, it is confusing, i will ask the bug reportor to clarify22:36
mriedemif the request is just, "do the same thing as the ip filter but for floating ips when listing instances" then that's easy to understand22:38
mriedemthe ip filter works on fixed ips, the floating ip filter (being requested) works on floating ips22:38
sean-k-mooneyhongbin: mriedem this is the basic workflow today
*** smatzek has joined #openstack-nova22:38
*** jaosorior has joined #openstack-nova22:39
sean-k-mooneythe device id of the neutron port is the nova instance id so today tyou do a floating ip show by it ip then get the port_id it is bound to look up the port and then get the device id of the port22:40
mriedemsean-k-mooney: the floating IP has a fixed IP on it22:40
mriedemnova list --ip
sean-k-mooneythe fixed ips can overlap22:40
*** sahid has quit IRC22:40
sean-k-mooneye.g. one tenant can have two networks with the same subnet22:41
mriedemi have a request,22:41
mriedemcan one of you hand me that shotgun in the corner?22:41
sean-k-mooneyif its any concilation the floating ip has to be unique. i think...22:42
*** smatzek has quit IRC22:43
sean-k-mooneymriedem: if the floating ip show included the device id direclty would that work for you22:43
mriedemsean-k-mooney: any of this being handled outside of nova works for me22:44
sean-k-mooneythen its jsut nova show ${device-id}22:44
sean-k-mooneywell to be honest its only 3 command today22:44
*** sahid has joined #openstack-nova22:44
sean-k-mooneyi can proably make that a 1 liner with fome bash foo22:44
mriedemso is creating a volume of a specific type and then creating a server from that volume, but people have always wanted to proxy through the volume type for boot from volume22:45
sean-k-mooneyi think the real issue is if you support regex or partical matches on the ip in stead of exact match22:45
mriedembecause bash is hard22:45
sean-k-mooneymriedem: thats fair was the original request to do this in the client or api22:46
hongbinthe scalability is also an issue, i.e. how to list the instances with a specific floating ip when there are thousands of instances22:46
*** lpetrut has quit IRC22:47
melwittmriedem: easy fix for 'nova-manage cell_v2 update_cell' where it wasn't excluding 'self' in its search for duplicate DB connection or transport URL
mriedemsean-k-mooney: the api22:48
mriedemhongbin: if you filter the ports by the floating ip first, then you've already narrowed down the instances because the port has the device_id22:49
*** sahid has quit IRC22:49
hongbinmriedem: i see, if what they want is one instance, then it is not a problem22:50
hongbinif they want a list instances with a floating ip prefix, then it is22:50
sean-k-mooneyhongbin: well the oneline is openstack server show $(openstack port show $(openstack ip floating show --column port_id -f value) -f value --column device_id) but yes you would have to call this for all instance in the prfix in a loop22:51
*** suresh12 has quit IRC22:51
hongbinsean-k-mooney: ack22:51
sean-k-mooneyso ideally you would extend floating ip list to accpet a cider to match agenst22:51
*** suresh12 has joined #openstack-nova22:52
sean-k-mooneyand also add the device id to the floating ip so you can just do a show on the nova instance that the end22:52
dansmithefried: got it22:53
*** dikonoor has joined #openstack-nova22:53
mriedemmelwitt: ack; currently trudging through internal email backlog of misery23:00
sean-k-mooneyhongbin: mriedem  looking at
sean-k-mooneyhongbin: mriedem what we really want in the neutron api is the ablity to do /v2.0/floatingips/?floatingip=<cidr>&fields=device_id23:01
sean-k-mooneythe blocker being current floatingip=<cidr> has to be an exact match not a cider and the port device_id is not returned as part of the floating_ip object23:02
hongbinsean-k-mooney: yes, agree23:03
*** liverpooler has quit IRC23:04
hongbinif neutron supports matching floating ip prefix, and adding the device_id to the floating ip resource, it seems to resolve the problem23:04
efrieddansmith Thank you sir.  Now if we can just get zuul to cooperate23:05
dansmithgood luck23:05
efriedikr.  It's been awful last couple weeks.23:05
sean-k-mooneyhongbin: based on they already support some simple filtering so addign cidr support should be possible instead of exact match23:06
hongbinsean-k-mooney: yes, technically, it is an easy job23:08
*** eharney has quit IRC23:09
sean-k-mooneywell you say that but it depends on where the filtering is done. if its in sql whic i would guess it is based on what is currently supported that less trivial23:09
hongbini think what needs to be done in neutron side is to change the sql query from '==' to 'like'23:11
hongbine.g. input==<exact_floating_ip> to'%floating_ip_substring%')23:12
sean-k-mooneyhongbin: if you wnat to suport queries like  all fluting ips containing "192.168" yes but you cant say all floating ips in
sean-k-mooneyim not sure what the original goal was23:13
hongbinyes, i also a bit confusing, asking my colleague to clarify23:14
*** yasemin has quit IRC23:14
sean-k-mooneyadding the device_id to the floating ip need a change to the floating ip api which i think is defiended in neutron-lib and that cant be reved until rocky as we are past the non-client lib freeze23:14
hongbinyes, know that23:15
sean-k-mooneyhongbin: i was just checking for my own sake but this is what would have to be modified
hongbinsean-k-mooney: like this ?23:18
*** AlexeyAbashkin has joined #openstack-nova23:18
hongbinso i already have a poc for that23:18
sean-k-mooneyhongbin: oh haha yes exactly like that :)23:19
sean-k-mooneyhongbin: so ya all that is missing then is adding support for a partial match on the floating ip  so input==<exact_floating_ip> to'%floating_ip_substring%') as you said above23:21
*** vladikr has quit IRC23:22
hongbinyes, and also need a clear picture that how nova will use it23:22
*** mlavalle has quit IRC23:23
*** AlexeyAbashkin has quit IRC23:23
sean-k-mooneyprosumably nova would call /v2.0/floatingips/?floatingip=<ip substing>&fields=device_id then do a a server list for each insance uuid returned.23:23
sean-k-mooneybut mriedem  could comment on that better then i ^23:24
mriedemi think i said that like an hour ago23:25
mriedem"do the same thing as the ip filter but with a floating ip"23:25
mriedemquestion was why does nova need to add this proxy filter support at all23:25
*** vladikr has joined #openstack-nova23:25
*** betherly has quit IRC23:26
*** kmalloc has quit IRC23:26
*** DuncanT has quit IRC23:26
*** icey has quit IRC23:26
*** catmando has quit IRC23:26
*** andrewbogott has quit IRC23:26
*** rajinir has quit IRC23:26
*** gregwork has quit IRC23:27
*** catmando has joined #openstack-nova23:27
*** betherly has joined #openstack-nova23:27
*** Hazelesque has quit IRC23:27
*** kmalloc has joined #openstack-nova23:27
*** icey has joined #openstack-nova23:27
*** andrewbogott has joined #openstack-nova23:27
*** rajinir has joined #openstack-nova23:27
*** suresh12 has quit IRC23:27
*** gregwork has joined #openstack-nova23:27
*** lennyb has quit IRC23:27
*** DuncanT has joined #openstack-nova23:27
sean-k-mooneyi think when this came up in the ptg i liked the idea of the neutorn change but fealt the nova change did not need to be in the api and could be in the client23:27
openstackgerritMatt Riedemann proposed openstack/python-novaclient master: Add support for microversion 2.60 - volume multiattach
*** Adri2000 has quit IRC23:28
*** Guest78703 has quit IRC23:28
*** lennyb has joined #openstack-nova23:28
mriedemsean-k-mooney: this floating ip filtering thing didn't come up in denver23:28
mriedemat least that i can recall - we talked about how to improve the performance of the *existing* ip filter when listing instances23:29
*** Adri2000 has joined #openstack-nova23:29
*** Hazelesque has joined #openstack-nova23:29
mriedemand that's done:
sean-k-mooneymriedem: are you sure im pretty sure i asked migle about adding the api change to neutron in dever23:29
sean-k-mooneymriedem: oh maybe that is what im thinking about23:30
*** suresh12 has joined #openstack-nova23:31
sean-k-mooneymriedem: so this is the same just adding support for floating ip23:31
mriedemsounds like it yes23:31
mriedemmelwitt: dansmith: so on we're just wanting to like change the name of an existing cell right?23:34
dansmithmriedem: or puppet is just running update_cell all the time because that's how it works23:34
dansmiths/all the/every/23:35
*** Guest78703 has joined #openstack-nova23:35
dansmithlike, there's probably a piece of puppet that says "ensure that cell0 has db url $foo and mq url $bar" and they just run update_cell blindly23:35
melwittmriedem: yeah, the update name would be one but the situation reported in the bug was that update_cell is not idempotent23:35
melwittI think it would also fail if you wanted to change only one of DB connection or transport URL with update_cell23:36
dansmithno, I think it shouldn't because that wouldn't match one of the existing ones right?23:36
*** suresh12 has quit IRC23:37
melwittit would match one of them with itself (before the fix) I think23:37
dansmithoh you're right, it's an or23:37
dansmithso yeah23:37
openstackgerritEric Fried proposed openstack/nova master: WIP: SchedulerReportClient.update_from_provider_tree
openstackgerritEric Fried proposed openstack/nova master: WIP: Use update_provider_tree from resource tracker
openstackgerritEric Fried proposed openstack/nova master: Fix nits in update_provider_tree series
openstackgerritEric Fried proposed openstack/nova master: Move refresh time from report client to prov tree
openstackgerritEric Fried proposed openstack/nova master: ProviderTree.get_provider_uuids: Top-down ordering
openstackgerritEric Fried proposed openstack/nova master: set_{aggregates|traits}_for_provider: tolerate set
sean-k-mooneymriedem: this makes me sad device_id should be product_id according to and other tests23:38
melwittyeah. historically it was an 'and' but recently it was changed to an 'or' since global MQ can't be a thing anyway and each cell should have unique DB and transport URL combo23:38
sean-k-mooneymriedem: unless wew chanded for device_id to product id a some point. those thest have been using incorrect mocks for 3 years23:39
mriedemsean-k-mooney: that's a pci request, not a port23:39
mriedemor you're just talking about something different and i have no idea23:40
sean-k-mooneymriedem: sorry yes this was what i was working on before the floating ip conversation23:40
melwitt(corrects self) er, or there shouldn't be two cells with dupe DB connection or dupe MQ URL23:41
mriedemsean-k-mooney: well you'll be happy to know it's just a json blob of whackiness so it doesn't matter
openstackgerritMerged openstack/nova master: Generalize DB conf group copying
sean-k-mooneymriedem: thats what im changing at stephenfin request23:41
openstackgerritMerged openstack/nova master: Recreate mediated devices on reboot
sean-k-mooneymriedem: which is why my tests are failing because i dont allow device id since the spec says it product_id23:42
sean-k-mooneymriedem: im reworking with stephenfin comments since rodolfo had to move on to opnfv work23:43
openstackgerritmelanie witt proposed openstack/nova master: Detach volume after deleting instance with no host
sean-k-mooneyi think this is really just at test bug but it still hurts to look at the git blame. anyway i think ill get back to this in the morning since its almost midnight my time23:45
*** penick has quit IRC23:46
*** efried is now known as efried_back_wed23:46
*** suresh12 has joined #openstack-nova23:50
*** takashin has joined #openstack-nova23:52
*** liverpooler has joined #openstack-nova23:55
*** openstack has quit IRC23:55
*** openstack has joined #openstack-nova23:57
*** ChanServ sets mode: +o openstack23:57
openstackgerritTakashi NATSUME proposed openstack/nova master: List/show all server migration types (1/2)

Generated by 2.15.3 by Marius Gedminas - find it at!