| *** tbachman has joined #openstack-nova | 00:02 | |
| *** jawad_axd has joined #openstack-nova | 00:04 | |
| *** jawad_axd has quit IRC | 00:08 | |
| *** artom has joined #openstack-nova | 00:11 | |
| *** Liang__ has quit IRC | 00:12 | |
| *** ociuhandu has joined #openstack-nova | 00:12 | |
| *** rajinir has quit IRC | 00:19 | |
| *** ociuhandu has quit IRC | 00:21 | |
| *** ociuhandu has joined #openstack-nova | 00:22 | |
| *** rajinir has joined #openstack-nova | 00:24 | |
| *** ivve has quit IRC | 00:25 | |
| *** ociuhandu has quit IRC | 00:26 | |
| *** Liang__ has joined #openstack-nova | 00:33 | |
| *** Liang__ has quit IRC | 00:36 | |
| *** ociuhandu has joined #openstack-nova | 00:36 | |
| *** jawad_axd has joined #openstack-nova | 00:46 | |
| *** dave-mccowan has joined #openstack-nova | 00:47 | |
| *** gyee has quit IRC | 00:48 | |
| *** rajinir has quit IRC | 00:48 | |
| *** ociuhandu has quit IRC | 00:49 | |
| *** macz has quit IRC | 00:49 | |
| *** jawad_axd has quit IRC | 00:50 | |
| *** mriedem has joined #openstack-nova | 00:51 | |
| *** rajinir has joined #openstack-nova | 00:54 | |
| openstackgerrit | Merged openstack/nova stable/train: Nova compute: add in log exception to help debug failures https://review.opendev.org/692662 | 00:54 |
|---|---|---|
| openstackgerrit | Matt Riedemann proposed openstack/nova stable/stein: Nova compute: add in log exception to help debug failures https://review.opendev.org/692936 | 00:56 |
| *** TxGirlGeek has quit IRC | 01:00 | |
| *** jawad_axd has joined #openstack-nova | 01:07 | |
| *** jawad_axd has quit IRC | 01:11 | |
| *** bnemec has quit IRC | 01:12 | |
| gibi_ptg | efried: thanks for pushing the evacute patches for qos. I think https://blueprints.launchpad.net/nova/+spec/support-move-ops-with-qos-ports-ussuri is incorrectly put into completed state. I still need to implement support for live migrate and unshelve with qos as the bp describes | 01:20 |
| *** ociuhandu has joined #openstack-nova | 01:23 | |
| *** ociuhandu has quit IRC | 01:34 | |
| *** slaweq_ has joined #openstack-nova | 01:35 | |
| *** slaweq__ has joined #openstack-nova | 01:39 | |
| *** slaweq_ has quit IRC | 01:41 | |
| *** dave-mccowan has quit IRC | 01:43 | |
| *** adrianc_ has joined #openstack-nova | 01:45 | |
| *** jawad_axd has joined #openstack-nova | 01:48 | |
| *** rajinir has quit IRC | 01:48 | |
| *** tkajinam has joined #openstack-nova | 01:52 | |
| *** jawad_axd has quit IRC | 01:53 | |
| *** rajinir has joined #openstack-nova | 01:54 | |
| *** bnemec has joined #openstack-nova | 01:58 | |
| *** dave-mccowan has joined #openstack-nova | 02:00 | |
| mriedem | gibi_ptg: fixed | 02:00 |
| gibi_ptg | mriedem: thanks | 02:02 |
| *** dave-mccowan has quit IRC | 02:08 | |
| mriedem | it's alive http://www.danplanet.com/blog/2019/11/04/start-and-monitor-image-pre-cache-operations-in-nova/ | 02:15 |
| *** adrianc_ has quit IRC | 02:16 | |
| *** adrianc_ has joined #openstack-nova | 02:16 | |
| gibi_ptg | mriedem: I will mention it in the nova project update as something we already done from the Ussuri plans | 02:18 |
| mriedem | the image cache thing? yeah good idea. | 02:19 |
| gibi_ptg | yepp | 02:19 |
| gibi_ptg | the image download already come up in the war stories session yesterday | 02:19 |
| *** slaweq__ has quit IRC | 02:23 | |
| *** ociuhandu has joined #openstack-nova | 02:28 | |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: WIP: Create BDM in API during attach volume https://review.opendev.org/692940 | 02:29 |
| *** ociuhandu has quit IRC | 02:36 | |
| *** bnemec has quit IRC | 02:37 | |
| *** Liang__ has joined #openstack-nova | 02:39 | |
| *** tkajinam has quit IRC | 02:40 | |
| *** slaweq has joined #openstack-nova | 02:45 | |
| *** adrianc_ has quit IRC | 02:46 | |
| *** adrianc_ has joined #openstack-nova | 02:46 | |
| *** mriedem has quit IRC | 02:47 | |
| *** rajinir has quit IRC | 02:48 | |
| *** jawad_axd has joined #openstack-nova | 02:50 | |
| *** bnemec has joined #openstack-nova | 02:52 | |
| *** ricolin has joined #openstack-nova | 02:53 | |
| *** adrianc__ has joined #openstack-nova | 02:53 | |
| *** rajinir has joined #openstack-nova | 02:53 | |
| *** adrianc_ has quit IRC | 02:54 | |
| *** adrianc_ has joined #openstack-nova | 02:54 | |
| *** jawad_axd has quit IRC | 02:55 | |
| *** yaawang_ has joined #openstack-nova | 02:55 | |
| *** Liang__ has quit IRC | 02:56 | |
| *** adrianc__ has quit IRC | 02:58 | |
| *** yaawang has quit IRC | 02:58 | |
| *** adrianc__ has joined #openstack-nova | 03:11 | |
| *** adrianc_ has quit IRC | 03:14 | |
| openstackgerrit | melanie witt proposed openstack/nova stable/stein: Add integration testing for heal_allocations https://review.opendev.org/692923 | 03:16 |
| openstackgerrit | melanie witt proposed openstack/nova stable/stein: Add regression test for bug 1824435 https://review.opendev.org/692906 | 03:16 |
| openstack | bug 1824435 in OpenStack Compute (nova) stein "fill_virtual_interface_list migration fails on second attempt" [Medium,In progress] https://launchpad.net/bugs/1824435 - Assigned to melanie witt (melwitt) | 03:16 |
| openstackgerrit | melanie witt proposed openstack/nova stable/stein: Remove redundant call to get/create default security group https://review.opendev.org/692907 | 03:16 |
| *** adrianc__ has quit IRC | 03:16 | |
| *** adrianc__ has joined #openstack-nova | 03:32 | |
| *** ricolin has quit IRC | 03:35 | |
| *** psachin has joined #openstack-nova | 03:38 | |
| *** slaweq has quit IRC | 03:40 | |
| *** ricolin has joined #openstack-nova | 03:41 | |
| *** bnemec has quit IRC | 03:56 | |
| *** ricolin has quit IRC | 03:57 | |
| *** bnemec has joined #openstack-nova | 03:57 | |
| *** ricolin has joined #openstack-nova | 03:57 | |
| *** adrianc__ has quit IRC | 03:58 | |
| *** adrianc__ has joined #openstack-nova | 04:07 | |
| *** ricolin has quit IRC | 04:13 | |
| *** takashin has joined #openstack-nova | 04:23 | |
| *** yaawang_ has quit IRC | 04:23 | |
| *** adrianc__ has quit IRC | 04:25 | |
| *** yaawang_ has joined #openstack-nova | 04:26 | |
| *** bnemec has quit IRC | 04:26 | |
| *** adrianc__ has joined #openstack-nova | 04:29 | |
| *** adrianc__ has quit IRC | 04:30 | |
| *** bhagyashris has joined #openstack-nova | 04:36 | |
| *** larainema has joined #openstack-nova | 04:41 | |
| *** Liang__ has joined #openstack-nova | 05:02 | |
| *** Liang__ has quit IRC | 05:13 | |
| *** brault has quit IRC | 05:17 | |
| gibi_ptg | nova project update went OK in Shanghai. We got questions about SRIOV live migration with bonded virtio interface for failover. adrianc wil; lead a discussion around it on the PTG | 05:20 |
| *** ricolin has joined #openstack-nova | 05:23 | |
| *** ileixe has quit IRC | 05:24 | |
| *** slaweq has joined #openstack-nova | 05:24 | |
| *** slaweq_ has joined #openstack-nova | 05:28 | |
| *** ileixe has joined #openstack-nova | 05:28 | |
| *** slaweq has quit IRC | 05:29 | |
| *** ileixe has quit IRC | 05:33 | |
| *** ileixe has joined #openstack-nova | 05:33 | |
| *** pcaruana has joined #openstack-nova | 05:41 | |
| *** ileixe has quit IRC | 05:44 | |
| *** slaweq_ has quit IRC | 05:45 | |
| *** ileixe has joined #openstack-nova | 05:47 | |
| *** links has joined #openstack-nova | 05:47 | |
| *** rajinir has quit IRC | 05:48 | |
| *** rajinir has joined #openstack-nova | 05:49 | |
| *** ricolin has quit IRC | 05:50 | |
| *** bhagyashris has quit IRC | 06:08 | |
| *** yaawang_ has quit IRC | 06:22 | |
| *** yaawang_ has joined #openstack-nova | 06:23 | |
| *** ratailor has joined #openstack-nova | 06:24 | |
| *** bhagyashris has joined #openstack-nova | 06:56 | |
| *** Liang__ has joined #openstack-nova | 07:07 | |
| *** Liang__ has quit IRC | 07:14 | |
| *** Liang__ has joined #openstack-nova | 07:15 | |
| *** jawad_axd has joined #openstack-nova | 07:15 | |
| *** lpetrut has joined #openstack-nova | 07:16 | |
| *** slaweq_ has joined #openstack-nova | 07:16 | |
| *** lpetrut has quit IRC | 07:16 | |
| *** lpetrut has joined #openstack-nova | 07:17 | |
| *** ricolin has joined #openstack-nova | 07:17 | |
| *** slaweq__ has joined #openstack-nova | 07:20 | |
| *** slaweq_ has quit IRC | 07:22 | |
| *** dpawlik has joined #openstack-nova | 07:31 | |
| *** ircuser-1 has joined #openstack-nova | 07:34 | |
| *** ociuhandu has joined #openstack-nova | 07:37 | |
| *** ociuhandu has quit IRC | 07:37 | |
| *** ociuhandu has joined #openstack-nova | 07:38 | |
| *** luksky has joined #openstack-nova | 07:39 | |
| *** ociuhandu has quit IRC | 07:43 | |
| *** ricolin_ has joined #openstack-nova | 07:45 | |
| *** Liang__ has quit IRC | 07:46 | |
| *** ricolin has quit IRC | 07:47 | |
| *** bhagyashris has quit IRC | 07:55 | |
| *** ricolin_ has quit IRC | 08:04 | |
| *** Liang__ has joined #openstack-nova | 08:04 | |
| *** ricolin_ has joined #openstack-nova | 08:04 | |
| *** damien_r has joined #openstack-nova | 08:11 | |
| *** bhagyashris has joined #openstack-nova | 08:12 | |
| *** ociuhandu has joined #openstack-nova | 08:14 | |
| *** slaweq__ has quit IRC | 08:19 | |
| *** slaweq has joined #openstack-nova | 08:19 | |
| *** tesseract has joined #openstack-nova | 08:24 | |
| *** bnemec has joined #openstack-nova | 08:25 | |
| openstackgerrit | Tushar Patil proposed openstack/nova-specs master: Allow compute nodes to use DISK_GB from shared storage RP https://review.opendev.org/650188 | 08:26 |
| *** takashin has left #openstack-nova | 08:30 | |
| *** ociuhandu has quit IRC | 08:30 | |
| *** ociuhandu has joined #openstack-nova | 08:31 | |
| *** ociuhandu has quit IRC | 08:33 | |
| *** ociuhandu has joined #openstack-nova | 08:34 | |
| *** Liang__ has quit IRC | 08:35 | |
| *** ociuhandu has quit IRC | 08:35 | |
| *** ociuhandu has joined #openstack-nova | 08:36 | |
| *** ricolin_ has quit IRC | 08:37 | |
| *** zbr|ooo is now known as zbr | 08:38 | |
| *** bhagyashris has quit IRC | 08:39 | |
| *** bhagyashris has joined #openstack-nova | 08:40 | |
| *** ociuhandu has quit IRC | 08:40 | |
| *** ociuhandu has joined #openstack-nova | 08:41 | |
| *** ociuhandu has quit IRC | 08:42 | |
| *** ociuhandu has joined #openstack-nova | 08:43 | |
| *** mkrai has joined #openstack-nova | 08:44 | |
| *** ociuhandu has quit IRC | 08:44 | |
| *** ociuhandu has joined #openstack-nova | 08:45 | |
| *** bnemec has quit IRC | 08:50 | |
| *** ociuhandu has quit IRC | 08:51 | |
| *** ociuhandu has joined #openstack-nova | 08:52 | |
| *** ralonsoh has joined #openstack-nova | 08:52 | |
| *** ociuhandu has quit IRC | 08:55 | |
| *** ociuhandu has joined #openstack-nova | 08:56 | |
| *** ociuhandu has quit IRC | 08:57 | |
| *** ociuhandu has joined #openstack-nova | 08:58 | |
| *** ociuhandu has quit IRC | 08:59 | |
| *** ociuhandu has joined #openstack-nova | 09:00 | |
| *** ociuhandu has quit IRC | 09:03 | |
| *** ociuhandu has joined #openstack-nova | 09:03 | |
| *** slaweq has quit IRC | 09:04 | |
| *** bhagyashris_ has joined #openstack-nova | 09:06 | |
| *** bhagyashris has quit IRC | 09:09 | |
| *** jistr is now known as jistr|mtgs | 09:10 | |
| *** mvkr has quit IRC | 09:17 | |
| *** bhagyashris_ has quit IRC | 09:18 | |
| *** bhagyashris has joined #openstack-nova | 09:18 | |
| *** bhagyashris_ has joined #openstack-nova | 09:18 | |
| *** ociuhandu has quit IRC | 09:23 | |
| *** ociuhandu has joined #openstack-nova | 09:23 | |
| *** arxcruz is now known as arxcruz|off | 09:24 | |
| *** ociuhandu has quit IRC | 09:25 | |
| *** ociuhandu has joined #openstack-nova | 09:25 | |
| *** ociuhandu has quit IRC | 09:27 | |
| *** ociuhandu has joined #openstack-nova | 09:27 | |
| *** ociuhandu has quit IRC | 09:29 | |
| *** ociuhandu has joined #openstack-nova | 09:30 | |
| *** ociuhandu has quit IRC | 09:31 | |
| *** ociuhandu has joined #openstack-nova | 09:32 | |
| *** ociuhandu has quit IRC | 09:39 | |
| *** ociuhandu has joined #openstack-nova | 09:40 | |
| *** ociuhandu has quit IRC | 09:41 | |
| *** ociuhandu has joined #openstack-nova | 09:42 | |
| *** bhagyashris__ has joined #openstack-nova | 09:43 | |
| *** derekh has joined #openstack-nova | 09:43 | |
| *** bhagyashris_ has quit IRC | 09:45 | |
| *** bhagyashris_ has joined #openstack-nova | 09:46 | |
| *** bhagyashris has quit IRC | 09:46 | |
| *** mkrai has quit IRC | 09:56 | |
| *** derekh has quit IRC | 10:01 | |
| *** derekh has joined #openstack-nova | 10:01 | |
| *** ygk_12345 has joined #openstack-nova | 10:10 | |
| *** ociuhandu has quit IRC | 10:23 | |
| *** bhagyashris__ has quit IRC | 10:23 | |
| *** bhagyashris_ has quit IRC | 10:23 | |
| *** ociuhandu has joined #openstack-nova | 10:23 | |
| *** ociuhandu has quit IRC | 10:25 | |
| *** ociuhandu has joined #openstack-nova | 10:26 | |
| *** dtantsur|afk is now known as dtantsur | 10:26 | |
| *** bhagyashris__ has joined #openstack-nova | 10:30 | |
| *** bhagyashris_ has joined #openstack-nova | 10:30 | |
| *** davee__ has quit IRC | 10:31 | |
| *** ygk_12345 has quit IRC | 10:32 | |
| *** davee__ has joined #openstack-nova | 10:32 | |
| *** davee__ has quit IRC | 10:33 | |
| *** davee__ has joined #openstack-nova | 10:34 | |
| *** davee__ has quit IRC | 10:35 | |
| *** ivve has joined #openstack-nova | 10:46 | |
| *** dpawlik has quit IRC | 10:48 | |
| *** mkrai has joined #openstack-nova | 10:53 | |
| *** ociuhandu has quit IRC | 10:57 | |
| *** ociuhandu has joined #openstack-nova | 10:57 | |
| *** ociuhandu has quit IRC | 10:59 | |
| *** ociuhandu has joined #openstack-nova | 11:00 | |
| openstackgerrit | Adit Sarfaty proposed openstack/nova master: Avoid fetching metadata when no subnets found https://review.opendev.org/679247 | 11:06 |
| *** dpawlik has joined #openstack-nova | 11:06 | |
| openstackgerrit | Adit Sarfaty proposed openstack/nova master: Avoid fetching metadata when no subnets found https://review.opendev.org/679247 | 11:13 |
| *** dpawlik has quit IRC | 11:14 | |
| *** dpawlik has joined #openstack-nova | 11:16 | |
| *** ociuhandu has quit IRC | 11:18 | |
| *** mkrai has quit IRC | 11:21 | |
| *** ratailor has quit IRC | 11:30 | |
| *** ociuhandu has joined #openstack-nova | 11:34 | |
| *** ociuhandu has quit IRC | 11:45 | |
| *** tbachman has quit IRC | 11:48 | |
| *** mvkr has joined #openstack-nova | 12:00 | |
| *** dviroel has joined #openstack-nova | 12:06 | |
| *** dpawlik has quit IRC | 12:10 | |
| *** luksky has quit IRC | 12:13 | |
| *** henriqueof has quit IRC | 12:16 | |
| *** bhagyashris__ has quit IRC | 12:28 | |
| *** bhagyashris_ has quit IRC | 12:28 | |
| *** ociuhandu has joined #openstack-nova | 12:30 | |
| *** larainema has quit IRC | 12:32 | |
| *** dpawlik has joined #openstack-nova | 12:44 | |
| *** ociuhandu has quit IRC | 12:50 | |
| openstackgerrit | ya.wang proposed openstack/nova-specs master: Re-propose expose-auto-converge-post-copy spec for Ussuri https://review.opendev.org/687199 | 12:50 |
| *** ociuhandu has joined #openstack-nova | 12:50 | |
| *** Garyx has quit IRC | 12:53 | |
| *** ociuhandu has quit IRC | 12:56 | |
| *** tbachman has joined #openstack-nova | 13:06 | |
| efried | gibi_ptg: thanks for the update, and thanks to y'all for covering the session. | 13:07 |
| efried | and sorry about prematurely completing that bp, forgot about the lm bits. | 13:07 |
| *** derekh has quit IRC | 13:09 | |
| *** jangutter has joined #openstack-nova | 13:10 | |
| *** ygk_12345 has joined #openstack-nova | 13:11 | |
| sean-k-mooney | is the PTG starting tonight | 13:14 |
| *** jistr|mtgs is now known as jistr | 13:14 | |
| gibi_ptg | sean-k-mooney: it starting wednesday morning shanghai time | 13:17 |
| gibi_ptg | efried: no problemo | 13:17 |
| *** tbachman has quit IRC | 13:17 | |
| sean-k-mooney | gibi_ptg: ya so thats 12-16 from now right is tuesday evenig/night for you now? | 13:18 |
| *** tbachman has joined #openstack-nova | 13:18 | |
| sean-k-mooney | ye are utc+8 right? | 13:19 |
| gibi_ptg | 21:19 local time | 13:19 |
| sean-k-mooney | ya that is what i guessed cool | 13:19 |
| *** Liang__ has joined #openstack-nova | 13:24 | |
| *** ociuhandu has joined #openstack-nova | 13:28 | |
| *** luksky has joined #openstack-nova | 13:37 | |
| *** ociuhandu has quit IRC | 13:37 | |
| openstackgerrit | Merged openstack/nova stable/stein: Nova compute: add in log exception to help debug failures https://review.opendev.org/692936 | 13:50 |
| openstackgerrit | Merged openstack/nova stable/queens: cleanup evacuated instances not on hypervisor https://review.opendev.org/687873 | 13:50 |
| openstackgerrit | Merged openstack/nova stable/queens: Add functional test for resize crash compute restart revert https://review.opendev.org/687874 | 13:50 |
| openstackgerrit | Merged openstack/nova stable/queens: Prevent init_host test to interfere with other tests https://review.opendev.org/687876 | 13:50 |
| openstackgerrit | Merged openstack/nova stable/queens: Functional reproduce for bug 1833581 https://review.opendev.org/687877 | 13:50 |
| openstack | bug 1833581 in OpenStack Compute (nova) queens "instance stuck in BUILD state if nova-compute is restarted" [Low,In progress] https://launchpad.net/bugs/1833581 - Assigned to Elod Illes (elod-illes) | 13:50 |
| openstackgerrit | Merged openstack/nova stable/queens: Error out interrupted builds https://review.opendev.org/687878 | 13:50 |
| *** eharney has quit IRC | 13:53 | |
| *** macz has joined #openstack-nova | 13:55 | |
| *** mriedem has joined #openstack-nova | 13:58 | |
| *** amodi has quit IRC | 13:58 | |
| openstackgerrit | Matt Riedemann proposed openstack/nova stable/rocky: Nova compute: add in log exception to help debug failures https://review.opendev.org/693009 | 14:06 |
| *** mkrai has joined #openstack-nova | 14:07 | |
| *** xek has joined #openstack-nova | 14:11 | |
| *** mvkr has quit IRC | 14:19 | |
| *** yaawang_ has quit IRC | 14:19 | |
| *** yaawang has joined #openstack-nova | 14:20 | |
| *** lbragstad_ has joined #openstack-nova | 14:22 | |
| *** lbragstad has quit IRC | 14:23 | |
| mriedem | dansmith: here is that follow up https://review.opendev.org/#/c/692856/ - the next one after it is really simple as well | 14:24 |
| *** slaweq has joined #openstack-nova | 14:27 | |
| *** jaosorior has joined #openstack-nova | 14:29 | |
| *** slaweq_ has joined #openstack-nova | 14:32 | |
| *** slaweq has quit IRC | 14:32 | |
| *** mvkr has joined #openstack-nova | 14:32 | |
| *** jangutter has quit IRC | 14:33 | |
| *** ygk_12345 has left #openstack-nova | 14:36 | |
| *** xek has quit IRC | 14:40 | |
| *** amodi has joined #openstack-nova | 14:42 | |
| *** links has quit IRC | 14:48 | |
| *** eharney has joined #openstack-nova | 14:49 | |
| *** mvkr has quit IRC | 14:56 | |
| *** xek has joined #openstack-nova | 14:58 | |
| *** tbachman has quit IRC | 14:59 | |
| *** slaweq_ is now known as slaweq | 14:59 | |
| *** jawad_axd has quit IRC | 15:00 | |
| *** derekh has joined #openstack-nova | 15:02 | |
| *** ociuhandu has joined #openstack-nova | 15:03 | |
| *** jawad_axd has joined #openstack-nova | 15:04 | |
| *** dpawlik has quit IRC | 15:05 | |
| artom | mriedem, left a thought on https://review.opendev.org/#/c/594139/1 | 15:06 |
| *** watersj has joined #openstack-nova | 15:07 | |
| *** ociuhandu has quit IRC | 15:07 | |
| watersj | How do I cancel a pre-migration task? It is stuck and will not complete | 15:07 |
| watersj | ? | 15:07 |
| *** slaweq has quit IRC | 15:08 | |
| *** jawad_axd has quit IRC | 15:09 | |
| mriedem | watersj: i think you're looking for https://docs.openstack.org/python-novaclient/latest/cli/nova.html#nova-live-migration-abort | 15:09 |
| *** mvkr has joined #openstack-nova | 15:09 | |
| *** slaweq__ has joined #openstack-nova | 15:11 | |
| *** jmlowe has quit IRC | 15:11 | |
| watersj | mriedem, I get instance is in an invalid state 409 | 15:12 |
| *** jamesdenton has quit IRC | 15:13 | |
| *** jmlowe has joined #openstack-nova | 15:14 | |
| *** jamesdenton has joined #openstack-nova | 15:15 | |
| *** jawad_axd has joined #openstack-nova | 15:18 | |
| efried | mriedem: not sure if we do this for storyboard bugs, but https://review.opendev.org/693028 | 15:18 |
| mriedem | watersj: what is the migration status for that instance? you can use https://docs.openstack.org/python-novaclient/latest/cli/nova.html#nova-server-migration-list | 15:19 |
| mriedem | efried: e-r doesn't support storyboard | 15:19 |
| *** slaweq__ has quit IRC | 15:19 | |
| mriedem | https://storyboard.openstack.org/#!/story/2004745 | 15:20 |
| efried | oh. Oh well. The logstash query works. Is there a way to make it stick? | 15:20 |
| dansmith | mriedem: WAT :P | 15:20 |
| mriedem | efried: you could use https://bugs.launchpad.net/python-openstackclient/ ? | 15:20 |
| mriedem | oops | 15:20 |
| efried | yeah | 15:20 |
| mriedem | efried: i meant https://bugs.launchpad.net/openstack-gate | 15:20 |
| mriedem | that's the catch all | 15:20 |
| efried | Okay. | 15:21 |
| mriedem | artom: i'm looking and responding, but it's...complicated | 15:21 |
| artom | mriedem, I believe you :) | 15:22 |
| *** macz has quit IRC | 15:23 | |
| *** macz has joined #openstack-nova | 15:23 | |
| efried | mriedem: done, thanks. | 15:24 |
| *** macz has quit IRC | 15:25 | |
| mriedem | artom: replied | 15:28 |
| artom | mriedem, ack, lemme dig into that | 15:30 |
| mriedem | tl;dr the neutronv2/api code is a gd mess when it comes to migrations | 15:30 |
| mriedem | there are like at least 6 methods involved in mucking with port bindings | 15:30 |
| *** tbachman has joined #openstack-nova | 15:30 | |
| artom | mriedem, I guess it comes down to "are we backporting this?" | 15:37 |
| artom | If not, we can invest in a more thorough refactor/rethink | 15:37 |
| mriedem | it probably depends on if sean-k-mooney can recreate the failure in a real env and confirm the bug and the fix | 15:38 |
| mriedem | on...rocky | 15:38 |
| mriedem | like i said in the change over a year ago, "Waiting for Sean Mooney to test this out in his environment since he was able to generate the issue and report the bug, so I want to know if this fixes it. Also, I want Miguel to take a look at this given my assumptions about how I thought neutron would handle the source host binding when the dest binding is deleted." | 15:38 |
| mriedem | even if we did backport, i could do either of my 1 or 2 options and make it backportable | 15:39 |
| mriedem | and add a wrapper/convenience method which doesn't get backported that calls both the activate and teardown methods under the covers | 15:40 |
| *** ociuhandu has joined #openstack-nova | 15:40 | |
| sean-k-mooney | am what do i need to repoduce? reads back | 15:41 |
| *** jaosorior has quit IRC | 15:41 | |
| mriedem | https://review.opendev.org/#/c/594139/ | 15:41 |
| artom | I kinda wonder what ovs-dpdk and/or hugepages have to do with it | 15:42 |
| sean-k-mooney | mriedem: that sounds like https://review.opendev.org/#/c/653506/ | 15:42 |
| artom | Seems like we could repro it in func tests with an instance with some ports | 15:42 |
| sean-k-mooney | although i need to reload the context for both back into my head | 15:43 |
| mriedem | artom: the neutron fixture would have to start tracking port bindings and do it correctly, which i'm not entirely confident in how that api works - which is why we have this bug in the first place | 15:43 |
| mriedem | i.e. deleting an inactive port binding does not automatically activate the other port binding | 15:44 |
| mriedem | but activating an inactive port binding automatically deactivates the previously active port binding | 15:44 |
| artom | mriedem, ah, right | 15:44 |
| mriedem | from my commit message, "I had assumed neutron would automatically activate the old remaining port binding when the dest host port binding was deleted, like how neutron automatically deactivates the source host binding when the dest host binding is activated, but apparently that is not the case, so we have to do it ourselves on rollback." | 15:44 |
| artom | mriedem, yep - as I said in my first comment, I can't speak to the raison d'etre of the patch | 15:45 |
| artom | Just the mechanics (assuming we need it in the first place) | 15:46 |
| mriedem | so i think tl;dr on live migration rollback since rocky we leave the source host port bindings inactive and you likely need to do something to resolve that, i'm not sure that the heal instance periodic fixes it | 15:46 |
| sean-k-mooney | yep | 15:46 |
| sean-k-mooney | well we have two patach to resovle to related edgecases already | 15:46 |
| sean-k-mooney | artom: what bug are you looking at | 15:47 |
| artom | sean-k-mooney, https://bugs.launchpad.net/nova/+bug/1788014 | 15:47 |
| openstack | Launchpad bug 1788014 in OpenStack Compute (nova) "when live migration fails due to a internal error rollback is not handeled correctly." [Medium,In progress] - Assigned to Matt Riedemann (mriedem) | 15:47 |
| watersj | mriedem, vm status is pre-migrating | 15:47 |
| mriedem | https://review.opendev.org/#/c/603844/ landed in stein but i'm not sure that heals the port binding on the source host either | 15:47 |
| *** slaweq__ has joined #openstack-nova | 15:47 | |
| mriedem | watersj: i wanted to know the migration record status, not the server vm_state | 15:48 |
| mriedem | or maybe that's what you meant | 15:48 |
| sean-k-mooney | so that is a duplicate of https://bugs.launchpad.net/nova/+bug/1822884 | 15:48 |
| openstack | Launchpad bug 1822884 in OpenStack Compute (nova) "live migration fails due to port binding duplicate key entry in post_live_migrate" [Medium,In progress] - Assigned to sean mooney (sean-k-mooney) | 15:48 |
| watersj | mriedem, output from server-migration-list is empty | 15:48 |
| *** ociuhandu has quit IRC | 15:48 | |
| mriedem | i guess because the api doesn't consider pre-migrating as in-progress...i think i found that out recently | 15:49 |
| artom | mriedem, wait, that rings a bell | 15:49 |
| mriedem | anyway, https://docs.openstack.org/python-novaclient/latest/cli/nova.html#nova-live-migration-abort won't do anything for a pre-migrating migration, only running/queued/preparing | 15:49 |
| mriedem | artom: yeah i noticed it when reviewing your numa live migration series | 15:49 |
| sean-k-mooney | there was a seperate issue with evacuate and pre-migrationg that artom might be thinking of | 15:50 |
| artom | mriedem, oh right - https://review.opendev.org/#/c/684409/ | 15:50 |
| watersj | this is stein that I am running | 15:50 |
| mriedem | because in the long ago live migration records didn't have pre-migrating since that was only done during the RT claim | 15:50 |
| artom | But my patch only changes the non-API version of that query | 15:50 |
| artom | Which is your comment on https://review.opendev.org/#/c/684409/1/nova/db/sqlalchemy/api.py@4431 | 15:51 |
| mriedem | right, so the api only shows in-progress live migrations when they have status "'queued', 'preparing', 'running'" | 15:52 |
| mriedem | which watersj just confirmed | 15:52 |
| mriedem | even though arguably 'accepted' and 'pre-migrating' could be considered in-progress | 15:52 |
| mriedem | or as in-progress as queued | 15:52 |
| *** slaweq__ has quit IRC | 15:52 | |
| mriedem | watersj: did you trace down where the migration request failed and got stuck? | 15:53 |
| artom | mriedem, I think that has to do with when they get put in the queue | 15:53 |
| artom | IOW, you can't abort it if it's not in the queue yet | 15:53 |
| mriedem | watersj: if the guest is still on the source host then nothing has really changed. you would likely have to manually change the migration record status to 'error' and reset the instance vm_state to 'active' and task_state to None | 15:53 |
| *** lpetrut has quit IRC | 15:54 | |
| watersj | no, looking at nova-compute logs did not give me much info. Currently vm is down and I trying to get it back on | 15:54 |
| *** ociuhandu has joined #openstack-nova | 15:54 | |
| watersj | i did change its state ti error | 15:54 |
| watersj | and then active | 15:54 |
| mriedem | watersj: ok, so you'll have to set the migration record status to 'error' and hard reboot the guest i think | 15:54 |
| sean-k-mooney | artom: mriedem by the way i though when we got to post-copy stage we cant rollback anymore | 15:54 |
| mriedem | watersj: the instances table record or the migrations table record? | 15:54 |
| mriedem | those are different records | 15:54 |
| *** TxGirlGeek has joined #openstack-nova | 15:55 | |
| watersj | 1sec | 15:55 |
| artom | sean-k-mooney, yeah, but... is that relevant here? | 15:56 |
| watersj | i ran nova reset-state <name> | 15:56 |
| artom | Also, which "here"? watersj's issue, or https://review.opendev.org/#/c/594139/1 ? | 15:56 |
| watersj | i ran nova reset-state --active <name> | 15:56 |
| mriedem | watersj: that just updates the server's vm/task states, it doesn't do anything with a stopped guest | 15:56 |
| sean-k-mooney | artom: for mriedem commit "If we've reached the point of post-copy during live migration on the source host and activated the destination host port bindings, but then something fails and we have to rollback," | 15:56 |
| watersj | the guest is not running | 15:56 |
| watersj | that is confirmed | 15:56 |
| mriedem | right, that's why i said you'll have to hard reboot it | 15:57 |
| *** watersj has left #openstack-nova | 15:57 | |
| mriedem | to get it back to active/running for real | 15:57 |
| mriedem | bye! | 15:57 |
| artom | You're welcome? | 15:57 |
| artom | sean-k-mooney, huh, good spot | 15:57 |
| *** watersj has joined #openstack-nova | 15:57 | |
| artom | Once we're in post-copy, we can't rollback, as the VM is now running on the dest | 15:57 |
| sean-k-mooney | yes | 15:57 |
| artom | https://wiki.qemu.org/Features/PostCopyLiveMigration#Summary | 15:57 |
| sean-k-mooney | and libvirt cant rollback at that point | 15:58 |
| artom | Although I think that's just the commit message | 15:58 |
| *** JamesBenson has joined #openstack-nova | 15:59 | |
| artom | Your bug (https://bugs.launchpad.net/nova/+bug/1788014) doesn't mention post-copy | 15:59 |
| openstack | Launchpad bug 1788014 in OpenStack Compute (nova) "when live migration fails due to a internal error rollback is not handeled correctly." [Medium,In progress] - Assigned to Matt Riedemann (mriedem) | 15:59 |
| *** ociuhandu has quit IRC | 15:59 | |
| artom | And the new code is being called from the existing _rollback_live_migration() method, so it's not like we're attempting to rollback a post-copy migration | 16:00 |
| mriedem | i likely mentioned post-copy in the commit message b/c the lifecycle handler callback thing will activate the dest host port binding if the migration switches to post-copy | 16:00 |
| mriedem | this code https://github.com/openstack/nova/blob/master/nova/compute/manager.py#L1252 | 16:00 |
| watersj | Instance not resizing, skipping migration, and looks to be staying there | 16:01 |
| watersj | from nova-compute logs | 16:01 |
| mriedem | but it doesn't have to be post-copy to switch bindings https://github.com/openstack/nova/blob/master/nova/compute/manager.py#L1203 | 16:01 |
| mriedem | watersj: yes, i realize b/c the resource tracker ignores live migrations in stein | 16:01 |
| mriedem | watersj: like i said, you'll have to find and manually change the migrations table record to 'error' status and then hard reboot the server | 16:02 |
| watersj | ah, k | 16:02 |
| *** jawad_axd has quit IRC | 16:02 | |
| mriedem | the former takes it out of the RT loop which is why you're seeing "Instance not resizing, skipping migration" and the latter is to get the guest running agian | 16:02 |
| mriedem | *again | 16:02 |
| watersj | ty, give it try | 16:02 |
| mriedem | something like this should find the migration: | 16:03 |
| artom | mriedem, so EVENT_LIFECYCLE_POSTCOPY_STARTED and EVENT_LIFECYCLE_MIGRATION_COMPLETED mean the same thing in that context: the VM is now irreversibly running on the dest | 16:03 |
| artom | mriedem, so it makes sense to update the port bindings at that point | 16:03 |
| mriedem | select * from nova_cell1.migrations where instance_uuid=<server id> and deleted=0 and status='pre-migrating'; | 16:03 |
| sean-k-mooney | artom: not nessisarally | 16:03 |
| sean-k-mooney | EVENT_LIFECYCLE_MIGRATION_COMPLETED i think can be sent for failure | 16:03 |
| mriedem | yup | 16:04 |
| mriedem | which is another patch of mine | 16:04 |
| mriedem | https://review.opendev.org/#/c/594527/ | 16:04 |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: libvirt: check job status for VIR_DOMAIN_EVENT_SUSPENDED_MIGRATED event https://review.opendev.org/594527 | 16:05 |
| artom | Aha, TIL | 16:05 |
| artom | (Actually, I think I already checked this when faking it for my NUMA LM func tests, but forgot) | 16:05 |
| artom | (Which, btw, are waiting for review - but given the last thing was Stephen's -1 which I addressed, maybe wait for him to get pack from PTG) | 16:06 |
| artom | We've long passed any sense of urgency, at this point | 16:06 |
| sean-k-mooney | oh actully i think i know how to repoduce the failure by the way | 16:06 |
| sean-k-mooney | i think i found this issue while trying to live migrate between kerenl ovs and ovs dpdk were the virtio feature flags change | 16:07 |
| sean-k-mooney | due to setting the mtu in one and not in the other | 16:07 |
| sean-k-mooney | strangely enough i set up a dpdk env to work on that and and another dpdk thing on friday | 16:08 |
| *** tbachman has quit IRC | 16:09 | |
| sean-k-mooney | artom: this was part of the orginal error http://paste.openstack.org/show/728539/ | 16:10 |
| artom | sean-k-mooney, ah, so you hit this because the source sent some XML the dest couldn't handle, and QEMU died | 16:11 |
| sean-k-mooney | yep | 16:12 |
| sean-k-mooney | libvirt was fine with it | 16:12 |
| sean-k-mooney | well you can see the error | 16:12 |
| artom | Right, the XML itself was "fine" | 16:12 |
| artom | But the generated qemu command line exploded it | 16:12 |
| sean-k-mooney | ya but not for the migration | 16:12 |
| sean-k-mooney | qemu also acccpeted teh commandline | 16:13 |
| sean-k-mooney | it was only an issue when the migration started | 16:13 |
| mriedem | so if https://review.opendev.org/#/c/594527/ fixes that to not activate the dest host port bindings, then we don't need to rollback and maybe i can just abandon https://review.opendev.org/594139 | 16:13 |
| sean-k-mooney | ya maybe | 16:14 |
| artom | I wonder if we could reproduce that in the gate, with a DNM patch that borks the LM on purpose | 16:14 |
| sean-k-mooney | i can try and recreate teh excat error i had and do some testing i guess | 16:14 |
| *** jawad_axd has joined #openstack-nova | 16:14 | |
| sean-k-mooney | am i might be able to do that yes... | 16:14 |
| sean-k-mooney | it woudl be quicker to do it locally first | 16:15 |
| *** tbachman has joined #openstack-nova | 16:15 | |
| sean-k-mooney | i have to go fix my dpdk job | 16:15 |
| *** links has joined #openstack-nova | 16:15 | |
| * artom is lazy, and likes having the gate do all the setup for him | 16:15 | |
| artom | (Apologies to infra and the nodepool donors) | 16:15 |
| sean-k-mooney | if i make it work again i could have 1 node be kernel ovs and the other dpdk and then replicte this with a simple live migrate test | 16:16 |
| artom | I'm assuming hacking Nova to just generate broken XML for the dest won't be enough? | 16:16 |
| sean-k-mooney | well yes | 16:16 |
| sean-k-mooney | to get teh conflcit one of the xmls shoudl have the mtu set and the udpated one not | 16:17 |
| watersj | mriedem, just to verify update migrations set status = 'error' where id = 76; | 16:17 |
| sean-k-mooney | that will change between the 32bit feature flags and teh 64bit feature flags and will cause the error | 16:17 |
| watersj | id 76 is record in question | 16:18 |
| artom | sean-k-mooney, so why not do that? DNM patch that adds a "break_xml()" to _update_xml() on the source | 16:18 |
| artom | Remove all the non-LM testing from Zuul | 16:18 |
| artom | And play around with that | 16:18 |
| sean-k-mooney | artom: so your going to write that patch ya :) | 16:18 |
| artom | sean-k-mooney, I don't mind | 16:18 |
| *** links has quit IRC | 16:19 | |
| *** jawad_axd has quit IRC | 16:19 | |
| mriedem | watersj: yeah | 16:19 |
| sean-k-mooney | well it shoudl be pretty easy to do i guess. am i need to look at that downstream bug breifly but i can also maybe look at this with you after i figure out if we can backport that fix | 16:20 |
| artom | sean-k-mooney, there's a downstream bug for that? | 16:20 |
| sean-k-mooney | that fix bing the binding_failded thing | 16:20 |
| sean-k-mooney | no a different thing | 16:21 |
| artom | Ah | 16:21 |
| artom | Because one of the things we were discussing is are we backporting the fix | 16:21 |
| artom | Which influences how invasive it can be | 16:21 |
| mriedem | artom: (10:13:25 AM) mriedem: so if https://review.opendev.org/#/c/594527/ fixes that to not activate the dest host port bindings, then we don't need to rollback and maybe i can just abandon https://review.opendev.org/594139 | 16:21 |
| sean-k-mooney | mriedem: if i can recreate it ill apply that patch and see if it fixes it | 16:22 |
| *** mkrai has quit IRC | 16:22 | |
| mriedem | ack | 16:22 |
| artom | mriedem, ah, yeah, it we never update the port bindings for a failed LM, we don't need to roll them back | 16:22 |
| artom | So, would there be value in a DNM to repro that in the gate? | 16:23 |
| artom | *DNM patch | 16:23 |
| watersj | I'm getting, There are allocations remaining against the source host that might need to be removed | 16:26 |
| *** igordc has joined #openstack-nova | 16:27 | |
| *** igordc has quit IRC | 16:27 | |
| sean-k-mooney | artom: if you can repro it in the gate we can use that to test both of mriedem patches so yes i think there is still value | 16:27 |
| sean-k-mooney | although long term a functional regression test would be optimal | 16:28 |
| artom | sean-k-mooney, mriedem was saying we need to fix the Neutron fixture for that | 16:28 |
| sean-k-mooney | proably | 16:28 |
| artom | sean-k-mooney, so what was that thing about the MTU? Removing it should trigger thing? Seems weird | 16:28 |
| sean-k-mooney | it does not really do the multiple port bindings workflow right | 16:29 |
| sean-k-mooney | artom: ya remove it form the port forces libvirt to change form teh 64bit encodeing to the 32bit encoding | 16:29 |
| sean-k-mooney | well adding it forces it | 16:29 |
| *** damien_r has quit IRC | 16:30 | |
| sean-k-mooney | anyway the root of that secific libvirt/qemu issue is they ran out of feature bits and adding and removing hte mtu is one way to chagne it | 16:31 |
| *** luksky has quit IRC | 16:31 | |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: WIP: Create BDM in API during attach volume https://review.opendev.org/692940 | 16:31 |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: Provide a better error when _verify_response hits a TypeError https://review.opendev.org/693042 | 16:31 |
| mriedem | watersj: ah yeah...so in placement the migration record (id=76) holds the allocations for the source host and the instance holds allocations for the dest host, and you need to move the instance allocations back to the source host and drop the allocations from the dest host, which doesn't sound fun right | 16:33 |
| mriedem | luckily i just wrote a guide on how to do this https://review.opendev.org/#/c/691427/ | 16:33 |
| watersj | lol | 16:34 |
| mriedem | you might want to just skip straight to the heal_allocations part | 16:34 |
| mriedem | i.e. delete the allocations for both the migration and instance consumer and then heal allocations for the instance | 16:34 |
| mriedem | https://docs.openstack.org/osc-placement/latest/cli/index.html#resource-provider-allocation-delete | 16:34 |
| mriedem | you'd pass the migration uuid and instance uuid to ^ to remove the allocations in placement | 16:35 |
| *** jawad_axd has joined #openstack-nova | 16:35 | |
| mriedem | then run: nova-manage placemet heal_allocations | 16:35 |
| mriedem | https://docs.openstack.org/nova/stein/cli/nova-manage.html#placement | 16:35 |
| mriedem | if you want to target the specific instance, or do a dry-run heal_allocations, you'd need to setup a train nova instance in a vm/container/virtualenv https://docs.openstack.org/nova/train/cli/nova-manage.html#placement since --dry-run and --instance options aren't in stein | 16:36 |
| watersj | have backup of volume | 16:37 |
| watersj | ready really break it | 16:37 |
| mriedem | i don't know what that means | 16:37 |
| mriedem | oh you mean you do'nt need a dry run | 16:37 |
| mriedem | heal allocations just recreates the missing allocation records in placement for the instance using the flavor data associated with the instance | 16:37 |
| watersj | mean I ok to try without dryrun capabilities | 16:37 |
| mriedem | doesn't touch the guest or anything like that | 16:37 |
| openstackgerrit | Artom Lifshitz proposed openstack/nova master: DNM: break live migration to trigger bug 1788014 https://review.opendev.org/693045 | 16:38 |
| openstack | bug 1788014 in OpenStack Compute (nova) "when live migration fails due to a internal error rollback is not handeled correctly." [Medium,In progress] https://launchpad.net/bugs/1788014 - Assigned to Matt Riedemann (mriedem) | 16:38 |
| watersj | k, ty for the help | 16:38 |
| openstackgerrit | Merged openstack/nova stable/stein: Add integration testing for heal_allocations https://review.opendev.org/692923 | 16:39 |
| *** jawad_axd has quit IRC | 16:39 | |
| *** ociuhandu has joined #openstack-nova | 16:46 | |
| *** mvkr has quit IRC | 16:48 | |
| *** tesseract has quit IRC | 16:50 | |
| *** ociuhandu has quit IRC | 16:50 | |
| mriedem | dansmith: replied in https://review.opendev.org/#/c/635668/ - i'm not sure exactly what you're looking for so i gave some options | 16:57 |
| mriedem | if you want to see a paste of what i'm talking about i can do that quick | 16:57 |
| *** luksky has joined #openstack-nova | 16:57 | |
| openstackgerrit | Artom Lifshitz proposed openstack/nova master: DNM: break live migration to trigger bug 1788014 https://review.opendev.org/693045 | 16:59 |
| openstack | bug 1788014 in OpenStack Compute (nova) "when live migration fails due to a internal error rollback is not handeled correctly." [Medium,In progress] https://launchpad.net/bugs/1788014 - Assigned to Matt Riedemann (mriedem) | 16:59 |
| dansmith | mriedem: seems like you get it to me :) | 17:01 |
| mriedem | ok | 17:02 |
| mriedem | updating thusly | 17:02 |
| *** psachin has quit IRC | 17:20 | |
| melwitt | whoa, one of my things merged the first time | 17:23 |
| * melwitt faints | 17:23 | |
| dansmith | you know | 17:26 |
| dansmith | it has been said that on several providers we're our own noisy neighbor | 17:26 |
| dansmith | if we see a massive drop in the IO-based failures this week, | 17:26 |
| dansmith | that would probably just go to confirm that as a likely problem | 17:26 |
| dansmith | less volume for samples for sure, but.. | 17:26 |
| melwitt | yeah, I was wondering that too. summit/ptg and if suddenly less gate failures | 17:29 |
| mriedem | gate has been smooth has rich creamery butter this week | 17:30 |
| mriedem | https://www.youtube.com/watch?v=3ds0vWfoTwU | 17:32 |
| *** gyee has joined #openstack-nova | 17:33 | |
| melwitt | lol | 17:33 |
| mriedem | i didn't actually think of the simpsons when i said that, but now i realize it's just baked into my brain from years of watching those recorded episodes on vhs as a kid | 17:35 |
| mriedem | "we didn't have streaming in my day..." | 17:35 |
| melwitt | that's impressive | 17:35 |
| sean-k-mooney | even with out streaming like friends the simsons is one of those shows that is on a loop on cable/satialite tv constantly | 17:36 |
| mriedem | we didn't have cable until i was 17 | 17:36 |
| mriedem | hence the vhs | 17:36 |
| mriedem | and then it was all WWE and Love Line years after my friends were sick of those shows | 17:37 |
| sean-k-mooney | some what similar i was maybe 12/14 but that just becasue ireland and it not being a thing until the 90s | 17:37 |
| * dansmith tweaks the topic for this channel | 17:37 | |
| dansmith | "This channel is for Nova development and mriedem's childhood nostalgia" | 17:38 |
| sean-k-mooney | dansmith: and music references form the 80s | 17:38 |
| dansmith | sean-k-mooney: same category | 17:38 |
| mriedem | let it be so | 17:39 |
| sean-k-mooney | well now that im donw backporting a patch we backported 6 months ago i think im going to get food then do something useful... | 17:40 |
| *** jamesdenton has quit IRC | 17:46 | |
| *** mvkr has joined #openstack-nova | 17:47 | |
| *** jamesdenton has joined #openstack-nova | 17:49 | |
| openstackgerrit | melanie witt proposed openstack/nova stable/stein: Add regression test for bug 1824435 https://review.opendev.org/692906 | 17:50 |
| openstack | bug 1824435 in OpenStack Compute (nova) stein "fill_virtual_interface_list migration fails on second attempt" [Medium,In progress] https://launchpad.net/bugs/1824435 - Assigned to melanie witt (melwitt) | 17:50 |
| openstackgerrit | melanie witt proposed openstack/nova stable/stein: Remove redundant call to get/create default security group https://review.opendev.org/692907 | 17:50 |
| *** jamesdenton has quit IRC | 17:59 | |
| *** derekh has quit IRC | 18:00 | |
| *** ociuhandu has joined #openstack-nova | 18:02 | |
| watersj | is openstack resource provider a command available to stein? or am I supposed to replace "resource provider" with something else ? | 18:06 |
| melwitt | watersj: you need to install osc-placement in order to get the placement cli commands | 18:07 |
| *** ociuhandu has quit IRC | 18:07 | |
| watersj | melwitt, ty | 18:10 |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: Execute CrossCellMigrationTask from MigrationTask https://review.opendev.org/635668 | 18:17 |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: Refresh instance in MigrationTask.execute Exception handler https://review.opendev.org/669012 | 18:17 |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: Plumb allow_cross_cell_resize into compute API resize() https://review.opendev.org/635684 | 18:17 |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: Filter duplicates from compute API get_migrations_sorted() https://review.opendev.org/636224 | 18:17 |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: Start functional testing for cross-cell resize https://review.opendev.org/636253 | 18:17 |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: Handle target host cross-cell cold migration in conductor https://review.opendev.org/642591 | 18:17 |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: Validate image/create during cross-cell resize functional testing https://review.opendev.org/642592 | 18:17 |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: Add zones wrinkle to TestMultiCellMigrate https://review.opendev.org/643450 | 18:17 |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: Add negative test for cross-cell finish_resize failing https://review.opendev.org/643451 | 18:17 |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: Add negative test for prep_snapshot_based_resize_at_source failing https://review.opendev.org/669013 | 18:17 |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: Add confirm_snapshot_based_resize_at_source compute method https://review.opendev.org/637058 | 18:17 |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: Add ConfirmResizeTask https://review.opendev.org/637070 | 18:17 |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: Add confirm_snapshot_based_resize conductor RPC method https://review.opendev.org/637075 | 18:17 |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: Confirm cross-cell resize from the API https://review.opendev.org/637316 | 18:17 |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: Add revert_snapshot_based_resize_at_dest compute method https://review.opendev.org/637630 | 18:17 |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: Deal with cross-cell resize in _remove_deleted_instances_allocations https://review.opendev.org/639453 | 18:17 |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: WIP: Add finish_revert_snapshot_based_resize_at_source compute method https://review.opendev.org/637647 | 18:17 |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: WIP: Add RevertResizeTask https://review.opendev.org/638046 | 18:17 |
| *** jamesdenton has joined #openstack-nova | 18:19 | |
| dansmith | mriedem: I'm missing something | 18:25 |
| dansmith | okay I think I see, but I feel like now the flow is more confusing than it was before | 18:27 |
| dansmith | all the nested conditions make it hard for me to tell what states you can end up in after your fall out of all of it | 18:28 |
| dansmith | and having the two "restricting" cases is not helping | 18:28 |
| dansmith | I feel like that is a lot more complicated just because we're trying to log a message there about something that will happen later based on a flag we're not even concerned with | 18:29 |
| dansmith | maybe it's more complicated because of the later change you commented about but... it's pretty confusing | 18:30 |
| dansmith | ....which adds another level of nested conditional | 18:30 |
| dansmith | and more logs | 18:30 |
| dansmith | maybe it's just me, and sorry if so, but I think that end state tripley-nested conditional thing is just a total nightmare | 18:31 |
| openstackgerrit | Archit Modi proposed openstack/nova master: Fix block-device-mapping parameter naming https://review.opendev.org/693065 | 18:39 |
| dansmith | mriedem: totally untested, but I think this rearrangement of the final code works the same and is easier to follow? https://pastebin.com/Hq61MM2V | 18:40 |
| *** CeeMac has quit IRC | 18:45 | |
| *** ralonsoh has quit IRC | 18:54 | |
| mriedem | that paste could work, it's fine with me. it's after the targeted host thing so https://review.opendev.org/#/c/635668/ would be different | 18:57 |
| mriedem | unless you're suggesting munging those changes together | 18:58 |
| dansmith | mriedem: no, I'm not suggesting muging, I just chose the final state to demonstrate what I think it needs to look like | 19:00 |
| *** jaosorior has joined #openstack-nova | 19:03 | |
| mriedem | i'm tinkering | 19:03 |
| melwitt | mriedem: I added you to what's turning out to be a hairy review for a bfv docs fix https://review.opendev.org/693065 | 19:04 |
| melwitt | I think it probably just needs to be updated to use --boot-from-volume (new in 4.0.0 added by you) because it's a confusing mix of wrong osc commands and correct nova boot commands | 19:05 |
| melwitt | but you know best | 19:05 |
| openstackgerrit | Dustin Cowles proposed openstack/nova master: Provider Config File: YAML file loading and schema validation https://review.opendev.org/673341 | 19:05 |
| openstackgerrit | Dustin Cowles proposed openstack/nova master: Provider Config File: Function to further validate and retrieve configs https://review.opendev.org/676029 | 19:05 |
| openstackgerrit | Dustin Cowles proposed openstack/nova master: WIP: Provider Config File: Merge provider configs to provider tree https://review.opendev.org/676522 | 19:05 |
| mriedem | melwitt: ok so this was likely a case of a doc being straight up changed from nova boot to just openstack server create with no thought to the implications of the actual option names or what was supported at the time | 19:06 |
| mriedem | fun | 19:06 |
| melwitt | exactly | 19:06 |
| mriedem | melwitt: left some thoughts on how to maybe break that down into more manageable chunks | 19:12 |
| mriedem | b/c i really hate fixing lots of random shit like this in the docs in the same patch | 19:12 |
| mriedem | easy to get something wrong | 19:12 |
| artom | alias 'openstack server create' 'nova boot' amirite | 19:12 |
| melwitt | cool thanks | 19:12 |
| mriedem | dansmith: ok so this would be pass 1 http://paste.openstack.org/show/785820/ | 19:13 |
| *** xek_ has joined #openstack-nova | 19:13 | |
| mriedem | for https://review.opendev.org/#/c/635668/ | 19:13 |
| *** ociuhandu has joined #openstack-nova | 19:14 | |
| *** xek has quit IRC | 19:14 | |
| mriedem | which i'd build on with the targeted host stuff in https://review.opendev.org/#/c/642591/ later | 19:14 |
| mriedem | to be more like your final paste | 19:14 |
| *** jaosorior has quit IRC | 19:15 | |
| melwitt | mriedem: so first step is just replace 'openstack server create' with 'nova boot' to make everything work correctly, backport that. then next step is just convert everything to osc commands properly. I don't think I understand the middle step you described | 19:18 |
| melwitt | or rather I'm seeing 2 steps instead of 3 | 19:18 |
| mriedem | no there is still a nova boot example in there https://review.opendev.org/#/c/693065/1/doc/source/user/launch-instance-from-volume.rst@224 | 19:20 |
| mriedem | so i was saying: | 19:20 |
| mriedem | 1. revert to using nova boot everywhere | 19:20 |
| mriedem | which is backportable | 19:20 |
| *** ociuhandu has quit IRC | 19:21 | |
| dansmith | mriedem: did I miss that requested_destination can be set and None? | 19:21 |
| mriedem | 2. fix the non nova boot (so not https://review.opendev.org/#/c/693065/1/doc/source/user/launch-instance-from-volume.rst@224) openstack server create usage | 19:21 |
| mriedem | 3. replace the remaining nova boot example with openstack server create and mention --boot-from-volume since osc 4.0.0 | 19:21 |
| mriedem | dansmith: that was the existing logic and is what we do elsewhere for setting the cell when moving the server | 19:21 |
| dansmith | aight | 19:22 |
| mriedem | melwitt: that's just how i'd tackle that to try and keep it small and sane for each thing getting fixed | 19:22 |
| *** rouk has joined #openstack-nova | 19:22 | |
| mriedem | rather than do it all in a single change stephen style :P and wait for 8 months for someone to review the doc fix | 19:22 |
| melwitt | mriedem: yeah, sorry. I'm just thinking after we revert to nova boot everywhere, there won't be any non nova boot for step 2 | 19:23 |
| mriedem | dansmith: this is my end product http://paste.openstack.org/show/785821/ | 19:23 |
| mriedem | no test changes | 19:23 |
| mriedem | melwitt: oh yeah you're right | 19:23 |
| dansmith | mriedem: hundy times better, IMHO | 19:23 |
| mriedem | dansmith: ok pushing | 19:23 |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: Execute CrossCellMigrationTask from MigrationTask https://review.opendev.org/635668 | 19:25 |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: Refresh instance in MigrationTask.execute Exception handler https://review.opendev.org/669012 | 19:25 |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: Plumb allow_cross_cell_resize into compute API resize() https://review.opendev.org/635684 | 19:25 |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: Filter duplicates from compute API get_migrations_sorted() https://review.opendev.org/636224 | 19:25 |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: Start functional testing for cross-cell resize https://review.opendev.org/636253 | 19:25 |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: Handle target host cross-cell cold migration in conductor https://review.opendev.org/642591 | 19:25 |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: Validate image/create during cross-cell resize functional testing https://review.opendev.org/642592 | 19:25 |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: Add zones wrinkle to TestMultiCellMigrate https://review.opendev.org/643450 | 19:25 |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: Add negative test for cross-cell finish_resize failing https://review.opendev.org/643451 | 19:25 |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: Add negative test for prep_snapshot_based_resize_at_source failing https://review.opendev.org/669013 | 19:25 |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: Add confirm_snapshot_based_resize_at_source compute method https://review.opendev.org/637058 | 19:25 |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: Add ConfirmResizeTask https://review.opendev.org/637070 | 19:25 |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: Add confirm_snapshot_based_resize conductor RPC method https://review.opendev.org/637075 | 19:25 |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: Confirm cross-cell resize from the API https://review.opendev.org/637316 | 19:25 |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: Add revert_snapshot_based_resize_at_dest compute method https://review.opendev.org/637630 | 19:25 |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: Deal with cross-cell resize in _remove_deleted_instances_allocations https://review.opendev.org/639453 | 19:25 |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: WIP: Add finish_revert_snapshot_based_resize_at_source compute method https://review.opendev.org/637647 | 19:25 |
| mriedem | melwitt: i guess the alternative is just (1) fix the existing busted osc usage in there, and (2) convert the remaining nova boot to osc and also mention --boot-from-volume | 19:25 |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: WIP: Add RevertResizeTask https://review.opendev.org/638046 | 19:25 |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: Add revert_snapshot_based_resize conductor RPC method https://review.opendev.org/638047 | 19:25 |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: Revert cross-cell resize from the API https://review.opendev.org/638048 | 19:25 |
| mriedem | the change right now just seems like it's trying to do too much and getting some of what it's trying to fix wrong - or incomplete | 19:25 |
| melwitt | mriedem: yeah, I feel like reverting to nova boot and backport is the right first step | 19:26 |
| mriedem | that works for me. | 19:26 |
| melwitt | because actually making it correct for osc is not straightforward | 19:26 |
| mriedem | btw, i thought rhosp had it's own docs and didn't use upstream? | 19:26 |
| melwitt | and would be better off as a separate change | 19:26 |
| amodi | mriedem: melwitt sure, updting the patch to use nova boot | 19:27 |
| amodi | yes, we dont use upstream, except when we do :) | 19:28 |
| mriedem | btw, this was the breaking change in the manuals repo https://review.opendev.org/#/c/404623/ | 19:28 |
| *** dpawlik has joined #openstack-nova | 19:28 | |
| melwitt | yeah, that was part of the big sweep that did some find/replaces. other docs of ours became wrong in that sweep, can't remember which atm | 19:29 |
| melwitt | oh, the quota stuf | 19:29 |
| melwitt | stuff | 19:29 |
| mriedem | yup the quota one was annoying | 19:30 |
| mriedem | i left an updated comment on amodi's change hopefully clearing it up | 19:30 |
| mriedem | amodi: btw, you should create a doc bug for this since we'll backport the fix | 19:31 |
| mriedem | and now i'm off to vote on raising my taxes for schools because i love the children | 19:31 |
| amodi | mriedem: ack, will do, and im just updating this doc | 19:32 |
| amodi | or if u want me to update all the other ones in the referenced changed, lmk | 19:32 |
| mriedem | amodi: just that doc. i didn't look through https://review.opendev.org/#/c/404623/ to see if it broke other stuf | 19:33 |
| mriedem | *stuff | 19:33 |
| melwitt | it broke the boot from ISO image too (still looking) | 19:33 |
| amodi | https://review.opendev.org/#/c/404623/8/doc/user-guide/source/cli-nova-launch-instance-using-ISO-image.rst this has the --block-device used with osc command | 19:33 |
| melwitt | the other stuff is probably ok | 19:34 |
| mriedem | ah yeah that is wrong, osc before 4.0 didn't have a way to boot with an image and create a blank volume | 19:34 |
| mriedem | yeah the rest looked ok | 19:34 |
| amodi | cool, will update thanks! | 19:35 |
| mriedem | actually i'm not even sure latest osc allows creating a server and telling nova to create and attach a blank volume | 19:35 |
| mriedem | add it to the gap pile! | 19:35 |
| mriedem | https://etherpad.openstack.org/p/compute-api-microversion-gap-in-osc | 19:35 |
| mriedem | artom: you broke nothing https://review.opendev.org/#/c/693045/ | 19:36 |
| openstackgerrit | Boris Bobrov proposed openstack/nova master: Also enable iommu for virtio controllers and video in libvirt https://review.opendev.org/684825 | 19:36 |
| openstackgerrit | Boris Bobrov proposed openstack/nova master: Create a controller for qga when SEV is used https://review.opendev.org/693072 | 19:36 |
| melwitt | hoo that is a lot of gaps | 19:37 |
| mriedem | (1) i thought you were artom (2) saying that is a lot of gas | 19:38 |
| *** mriedem is now known as mriedem_afk | 19:38 | |
| melwitt | osc might cause gas | 19:40 |
| artom | mriedem_afk, I know, I suck | 19:40 |
| artom | Or... blow? Which way is the gas supposed to go? | 19:40 |
| dansmith | suck squeeze bang blow | 19:40 |
| artom | That's the one | 19:41 |
| artom | Applies to ICEs and other things | 19:41 |
| dansmith | why do you think mriedem_afk is afk right now? | 19:41 |
| dansmith | HEY-O | 19:41 |
| dansmith | oh, voting, nevermind | 19:41 |
| artom | In some ways it's the same thing | 19:42 |
| dansmith | heh | 19:42 |
| artom | I also like how he keeps -1'ing the _DNM_ change :) | 19:47 |
| *** eharney has quit IRC | 19:52 | |
| *** watersj has quit IRC | 19:52 | |
| openstackgerrit | Artom Lifshitz proposed openstack/nova master: DNM: break live migration to trigger bug 1788014 https://review.opendev.org/693045 | 19:58 |
| openstack | bug 1788014 in OpenStack Compute (nova) "when live migration fails due to a internal error rollback is not handeled correctly." [Medium,In progress] https://launchpad.net/bugs/1788014 - Assigned to Matt Riedemann (mriedem) | 19:58 |
| sean-k-mooney | artom: its possibel the breakage has something to with vhost-user other then the mtu or libvirt could have fixed the issue and now always use the 64bit encoding instead | 20:07 |
| artom | sean-k-mooney, I added an extra debug log, we'll see | 20:09 |
| artom | If the latter, we can try and find another way to trigger this | 20:09 |
| sean-k-mooney | it been over a year since i actully hit that error and it was on cent7.5 so the libvirt we are using in the gate in the ubuntu vms is alot different | 20:09 |
| *** amodi has quit IRC | 20:10 | |
| efried | sean-k-mooney: what do we do about tempest tests that need a certain minimum of e.g. libvirt or qemu? | 20:10 |
| efried | Do all the CI nodes have the same (latest) version? | 20:11 |
| efried | hm, I guess they probably do if they all use the same image... | 20:11 |
| sean-k-mooney | it depends | 20:11 |
| sean-k-mooney | in some cases we have feature flags in the tempest config | 20:11 |
| sean-k-mooney | but in general tempest does not have test that care | 20:11 |
| sean-k-mooney | in the nova code we have checks if feature need a newer version then our minium | 20:12 |
| efried | I'm going to be getting into vTPM with encryption this cycle, which requires a very recent libvirt/qemu. I don't know yet whether I'll be adding CI for it, but was curious how that would happen if I did. | 20:12 |
| efried | yeah, but that'll just cause the build to fail, right? | 20:12 |
| *** dpawlik has quit IRC | 20:13 | |
| sean-k-mooney | am its a pain but we used to use fedora with the virt preview repo for that | 20:13 |
| *** jawad_axd has joined #openstack-nova | 20:13 | |
| sean-k-mooney | if that is not an option we also have a devstack plugin you could use | 20:13 |
| sean-k-mooney | it will compile libvirt and qemu from source | 20:13 |
| efried | eek | 20:14 |
| sean-k-mooney | personally i found that beeter then random fedora virt preview package | 20:14 |
| efried | I don't think I need *unreleased* versions. Just pretty recent ones. | 20:14 |
| sean-k-mooney | efried: do you know what version you need | 20:14 |
| efried | yeah, one sec... | 20:15 |
| * artom goes for an early school run | 20:15 | |
| *** ociuhandu has joined #openstack-nova | 20:16 | |
| efried | sean-k-mooney: libvirt 5.6.0. qemu at least 1.11, but possibly higher, not sure. | 20:16 |
| efried | sorry, 2.11 | 20:16 |
| sean-k-mooney | ok fedroa 20 might have that 5.6 is pretty new i think | 20:16 |
| sean-k-mooney | https://opendev.org/x/devstack-plugin-libvirt-qemu | 20:16 |
| sean-k-mooney | i can try and build those with that too | 20:16 |
| sean-k-mooney | i havent really been maintaining that | 20:17 |
| sean-k-mooney | but its what we used to use at intel to test prerelease stuff | 20:17 |
| *** mriedem_afk is now known as mriedem | 20:18 | |
| mriedem | artom: -1ing your DNM changes is how i stay up on stackalytics | 20:18 |
| *** ociuhandu has quit IRC | 20:21 | |
| mriedem | the bottom 2 trivial changes in the cross-cell resize series are begging for another core https://review.opendev.org/#/c/692856/ | 20:21 |
| mriedem | they are, not me | 20:22 |
| sean-k-mooney | efried: ill test it but i think you just need to add http://paste.openstack.org/show/785823/ for the devstack plugin | 20:22 |
| *** amodi has joined #openstack-nova | 20:23 | |
| openstackgerrit | Archit Modi proposed openstack/nova master: Revert "openstack server create" to "nova boot" in nova docs https://review.opendev.org/693065 | 20:26 |
| *** zbr is now known as zbr|ooo | 20:29 | |
| efried | whoah, I appear to have 4.0.0 on my bionic | 20:30 |
| efried | is the apt package level the number I want to be looking at? | 20:30 |
| sean-k-mooney | 4.0 of? | 20:30 |
| sean-k-mooney | libvirt you check with libvirtd --version | 20:31 |
| efried | yeah, 4.0.0 | 20:31 |
| sean-k-mooney | ya and ubuntu 18.04 has 2.11 libvirt | 20:31 |
| sean-k-mooney | *qemu | 20:32 |
| sean-k-mooney | so the qemu in the gate should be fine | 20:32 |
| efried | confirmed 2.11 | 20:32 |
| efried | Is there a way *other* than building from source (which I have chronically shit luck with, no matter what I'm trying to build) to get a higher libvirt on my bionic? | 20:33 |
| sean-k-mooney | i dont think there is a ppa but it migth be in the ubuntu cloud archive | 20:33 |
| sean-k-mooney | let me check | 20:33 |
| sean-k-mooney | ya they have 5.4 | 20:34 |
| sean-k-mooney | http://ubuntu-cloud.archive.canonical.com/ubuntu/pool/main/libv/libvirt/ | 20:34 |
| sean-k-mooney | you need 5.6 however right | 20:35 |
| efried | okay, so if I need 5.6.0 I need to build | 20:35 |
| efried | yeah. | 20:35 |
| efried | wah wah wahhhhh | 20:35 |
| sean-k-mooney | even the libvirt ppa is 5.4 https://launchpad.net/ubuntu/+source/libvirt | 20:36 |
| sean-k-mooney | ill let you know in a few minute i the plug still work. | 20:37 |
| efried | sean-k-mooney: I gotta do this locally first anyway | 20:37 |
| sean-k-mooney | i havenet used it in like 2 years but i wrote the first version of it to solve this exact problem | 20:37 |
| efried | unless you're saying I can use the plugin to do it locally.... which would make a certain amount of sense ;P | 20:38 |
| sean-k-mooney | yes you can | 20:38 |
| sean-k-mooney | its just a devstack plugin so you can run it locally by adding the 3 lines i sent you to your local.conf | 20:38 |
| sean-k-mooney | it should intall all the deps to build it and checkout the version you set as part of stacking | 20:39 |
| efried | oo | 20:40 |
| efried | mriedem: done | 20:44 |
| efried | sean-k-mooney: trying.... | 20:46 |
| mriedem | efried: thanks | 20:47 |
| sean-k-mooney | efried: libvirt just finsihed building for me | 20:47 |
| efried | I couldn't just sit here listening to all the begging. | 20:47 |
| sean-k-mooney | its working on qemu now | 20:47 |
| efried | sean-k-mooney: cool, so given that I'm running 1/128th of the cores you are, mine will finish some time tomorrow? | 20:47 |
| sean-k-mooney | its not that bad. im limited by disk io | 20:48 |
| sean-k-mooney | it will add maybe 5-10 mins on the build at most and i setup the pluging to cache the build | 20:49 |
| sean-k-mooney | it creates a BUILD_COMPLETE file in /opt/stack/libvirt and /opt/stack/qemu | 20:49 |
| *** xek_ has quit IRC | 20:49 | |
| sean-k-mooney | so you dont need to do it again unless you want too | 20:49 |
| sean-k-mooney | unstack does not delete that file but clean will | 20:50 |
| *** xek has joined #openstack-nova | 20:50 | |
| efried | nice | 20:50 |
| efried | By sheer luck, I totally asked the right person. | 20:50 |
| efried | ...unless it doesn't work for me. In which case you're fired. | 20:51 |
| sean-k-mooney | assuming it still works you can also give it a set of patchwork urls and to test with unmerged patche to both too | 20:51 |
| efried | hum, I seem to still be running under py2, which probably isn't my best call. | 20:51 |
| sean-k-mooney | it will proably be fine | 20:51 |
| efried | Do I just fix that by setting USE_PYTHON3=true in my local.conf? | 20:51 |
| sean-k-mooney | for about another week or two | 20:52 |
| sean-k-mooney | yes | 20:52 |
| efried | well, I don't really care except that I'm trying to be close to realistic here. | 20:52 |
| sean-k-mooney | if your talking to a bank they will be running python 2 for the next decade | 20:52 |
| sean-k-mooney | setting USE_PYTHON3=true wont remove the py2 stuff so its best to do that on a clean vm | 20:53 |
| efried | would you be disgusted if I told you I'm not running in a VM? | 20:53 |
| rouk | s/python 2/COBOL or some variant of BASIC or java 1.0/g | 20:53 |
| sean-k-mooney | efried: not really if you have a dev server | 20:54 |
| efried | indeed rouk. FORTRAN etc. Course, they're probably not running OpenStack in those languages. | 20:54 |
| sean-k-mooney | i would not run devstack on my laptop directly however | 20:54 |
| efried | yeah, I've got some dev NUCs. | 20:54 |
| efried | I think it's building libvirt... | 20:55 |
| sean-k-mooney | just finished qemu | 20:56 |
| *** eharney has joined #openstack-nova | 20:57 | |
| sean-k-mooney | ... and it failed... sudo addgroup -f -r libvirt | 20:57 |
| sean-k-mooney | Option f is ambiguous (firstuid, force-badname) | 20:57 |
| sean-k-mooney | Unknown option: r | 20:57 |
| sean-k-mooney | why do pepole change things like addgroup | 20:57 |
| *** xek has quit IRC | 20:59 | |
| sean-k-mooney | oh those are the optiongs for groupadd | 21:00 |
| sean-k-mooney | efried: if you have a different terminal just run "sudo groupadd -f -r libvirt" | 21:01 |
| sean-k-mooney | and it will continue instaed failing or you can leave it fail and run it after and then just stack and unstack | 21:02 |
| efried | thanks sean-k-mooney. Why is it ambiguous in flight and not when I run it?? | 21:04 |
| sean-k-mooney | its trying to use addgroup not groupadd | 21:04 |
| sean-k-mooney | https://opendev.org/x/devstack-plugin-libvirt-qemu/src/branch/master/devstack/libs/libvirt#L157 | 21:05 |
| sean-k-mooney | its a bug but this has not been tested on 18.04 before | 21:05 |
| sean-k-mooney | i last used it on 16.04 and centos7 | 21:05 |
| openstackgerrit | Archit Modi proposed openstack/nova master: Revert "openstack server create" to "nova boot" in nova docs https://review.opendev.org/693065 | 21:06 |
| efried | o | 21:06 |
| sean-k-mooney | ill fix it up after i test it a bit | 21:06 |
| *** tbachman has quit IRC | 21:13 | |
| openstackgerrit | Archit Modi proposed openstack/nova master: Revert "openstack server create" to "nova boot" in nova docs https://review.opendev.org/693065 | 21:13 |
| efried | sean-k-mooney: http://paste.openstack.org/raw/785825/ FYI | 21:15 |
| sean-k-mooney | do you have /usr/bin/python | 21:18 |
| openstackgerrit | melanie witt proposed openstack/nova stable/stein: Add regression test for bug 1824435 https://review.opendev.org/692906 | 21:19 |
| openstack | bug 1824435 in OpenStack Compute (nova) stein "fill_virtual_interface_list migration fails on second attempt" [Medium,In progress] https://launchpad.net/bugs/1824435 - Assigned to melanie witt (melwitt) | 21:19 |
| openstackgerrit | melanie witt proposed openstack/nova stable/stein: Remove redundant call to get/create default security group https://review.opendev.org/692907 | 21:19 |
| efried | sean-k-mooney: yes, it points to python2.7 | 21:19 |
| sean-k-mooney | that shoudl be fine i have the same its form here https://opendev.org/x/devstack-plugin-libvirt-qemu/src/branch/master/devstack/libs/libvirt#L209 | 21:20 |
| sean-k-mooney | you could proably comment that line out and install libvirt-python by hand | 21:22 |
| sean-k-mooney | the plugin will be cloned locally to /opt/stack | 21:22 |
| efried | sean-k-mooney: I tried, it fails the same way. I even tried installing a lower version. | 21:22 |
| sean-k-mooney | so installing libvirt python just fails in general form pip | 21:23 |
| efried | oh, hold on, it's failing a different way when I do it manually... | 21:24 |
| efried | sean-k-mooney: http://paste.openstack.org/raw/785826/ | 21:26 |
| efried | sorry, the original thing happened when I tried it manually the first time | 21:28 |
| efried | then I unstacked | 21:28 |
| efried | then after the unstack it started to fail the second way. That's probably because unstack unset the path to libvirt. | 21:28 |
| sean-k-mooney | maybe what to you get if you run this | 21:29 |
| sean-k-mooney | ubuntu@dev:/opt/repos/devstack$ pkg-config libvirt ; echo $? | 21:29 |
| sean-k-mooney | 0 | 21:29 |
| efried | 1 right now, lemme stack again and see... | 21:29 |
| *** ociuhandu has joined #openstack-nova | 21:29 | |
| sean-k-mooney | had you stacked on the system before | 21:30 |
| sean-k-mooney | or was it clean | 21:30 |
| sean-k-mooney | im currenly not using a clean vm | 21:30 |
| efried | before before? yes | 21:30 |
| sean-k-mooney | ok same then | 21:30 |
| efried | I unstacked, cleaned, upgraded all packages, rebooted, then started stacking with RECLONE and your plugin, other stuff unchanged. | 21:30 |
| sean-k-mooney | do you have the libvirt.so? | 21:31 |
| sean-k-mooney | ubuntu@dev:/opt/repos/devstack$ ll /usr/lib/libvirt.so | 21:31 |
| sean-k-mooney | lrwxrwxrwx 1 root root 19 Nov 5 21:04 /usr/lib/libvirt.so -> libvirt.so.0.5008.0* | 21:31 |
| openstackgerrit | melanie witt proposed openstack/nova master: Revert "openstack server create" to "nova boot" in nova docs https://review.opendev.org/693065 | 21:32 |
| sean-k-mooney | that shoudl have been install when the pluging did "sudo make install" in the /opt/stack/libvirt folder | 21:32 |
| openstackgerrit | melanie witt proposed openstack/nova master: Revert "openstack server create" to "nova boot" in nova docs https://review.opendev.org/693065 | 21:33 |
| efried | I'll check when I get back to that point | 21:33 |
| efried | I've got another NUC doing the same thing, but with USE_PYTHON3=yes | 21:33 |
| efried | we'll see if that hits the same problem. | 21:33 |
| *** amodi has quit IRC | 21:34 | |
| *** ociuhandu has quit IRC | 21:39 | |
| *** tbachman has joined #openstack-nova | 21:40 | |
| *** amodi has joined #openstack-nova | 21:40 | |
| efried | hm, despite USE_PYTHON3=yes it still appears to be using py2 | 21:42 |
| *** ociuhandu has joined #openstack-nova | 21:47 | |
| efried | nevertheless the supposed-py3 one seems to have gotten past that point. | 21:49 |
| sean-k-mooney | i guess that is a good thing | 21:49 |
| sean-k-mooney | im expect apparmor to be unhappy by the way. im currently fixing that on my system | 21:51 |
| *** takashin has joined #openstack-nova | 21:52 | |
| *** macz has joined #openstack-nova | 21:55 | |
| openstackgerrit | Dustin Cowles proposed openstack/nova master: Provider Config File: Function to further validate and retrieve configs https://review.opendev.org/676029 | 21:56 |
| openstackgerrit | Dustin Cowles proposed openstack/nova master: WIP: Provider Config File: Merge provider configs to provider tree https://review.opendev.org/676522 | 21:56 |
| *** ociuhandu has quit IRC | 21:57 | |
| efried | sean-k-mooney: what's apparmor? | 21:57 |
| efried | Is it related to this? | 21:57 |
| efried | Connection to libvirt failed: authentication unavailable: no polkit agent available to authenticate action 'org.libvirt.unix.manage': libvirtError: authentication unavailable: no polkit agent available to authenticate action 'org.libvirt.unix.manage' | 21:57 |
| sean-k-mooney | its the securaty context that ubuntu use instead of se linux | 21:57 |
| sean-k-mooney | sudo ln -s /etc/apparmor.d/usr.sbin.libvirtd /etc/apparmor.d/disable/usr.sbin.libvirtd | 21:58 |
| sean-k-mooney | will diable the libvirt profie allowing soruce build to work | 21:58 |
| sean-k-mooney | but you have to do a reboot for it to take effect | 21:58 |
| sean-k-mooney | you are ment to be able to just use aa-complain | 21:58 |
| sean-k-mooney | to put it in warning/complain mode but it was not working for me | 21:58 |
| efried | would this be... not a problem if I compiled a different version of libvirt? | 21:59 |
| sean-k-mooney | i need to write a readme for that plugin and doument this and fix the other issue | 21:59 |
| sean-k-mooney | no i hit the same issue | 21:59 |
| efried | oh, the readme isn't "Come find sean-k-mooney on IRC"? | 21:59 |
| efried | well, I didn't change 5.8 to 5.6 in my env | 21:59 |
| sean-k-mooney | well it basicaly is right now | 21:59 |
| efried | swhy I'm asking. | 21:59 |
| efried | and if I reboot I have to restack, right? | 22:00 |
| sean-k-mooney | there is another thing you coudl try first | 22:00 |
| sean-k-mooney | but i did it the reboot way | 22:00 |
| *** TxGirlGeek has quit IRC | 22:00 | |
| sean-k-mooney | you can add security_driver="none" | 22:01 |
| *** TxGirlGeek has joined #openstack-nova | 22:01 | |
| sean-k-mooney | to /etc/libvirt/qemu.conf and restart libvirt-bin | 22:01 |
| sean-k-mooney | but if you rerun stack.sh it will overwrite it | 22:01 |
| sean-k-mooney | ill fix this up and automated in the plugin | 22:02 |
| sean-k-mooney | its ment to do this for you | 22:02 |
| sean-k-mooney | but as i said it has not been updated for ubuntu 18.04 | 22:02 |
| *** pcaruana has quit IRC | 22:03 | |
| efried | sean-k-mooney: and tbc, is this the same issue as | 22:04 |
| efried | Connection to libvirt failed: authentication unavailable: no polkit agent available to authenticate action 'org.libvirt.unix.manage': libvirtError: authentication unavailable: no polkit agent available to authenticate action 'org.libvirt.unix.manage' | 22:04 |
| efried | ? | 22:04 |
| sean-k-mooney | yes | 22:04 |
| efried | k | 22:04 |
| sean-k-mooney | polkit is used to interact with selinux via dbus on fedora/centos/rhel | 22:05 |
| sean-k-mooney | is apparently the libvirt default | 22:05 |
| openstackgerrit | Takashi NATSUME proposed openstack/nova master: Update keypairs in saving an instance object https://review.opendev.org/683043 | 22:05 |
| sean-k-mooney | im just not passing the correct flags when building it to have it auto install the apparmor files | 22:05 |
| sean-k-mooney | i should prably just ask danpb how to do it correctly | 22:06 |
| melwitt | mriedem: do you remember any func test change from the past made to avoid "{"forbidden": {"code": 403, "message": "Maximum number of ports exceeded"}}" | 22:11 |
| mriedem | are you backporting something to queens or newton or something and hitting that? | 22:11 |
| melwitt | I'm backporting stuff and hitting this because I'm presumably missing some underlying func test improvement | 22:11 |
| melwitt | yes | 22:11 |
| melwitt | I thought if anyone might remember something related to that, it would be you | 22:12 |
| *** jawad_axd has quit IRC | 22:12 | |
| mriedem | not specifically, but i'd look for port quota stuff on the NeutronFixture | 22:12 |
| melwitt | thanks, any direction helps a lot :) | 22:12 |
| mriedem | https://review.opendev.org/#/q/I1dbccc2be6ba79bf267edac9208c80e187e6256a ? | 22:13 |
| mriedem | https://review.opendev.org/#/c/587412/4/nova/tests/fixtures.py@1347 is the thing i'd think you want | 22:13 |
| melwitt | looks promising | 22:14 |
| *** ociuhandu has joined #openstack-nova | 22:18 | |
| *** ociuhandu has quit IRC | 22:23 | |
| *** tbachman has quit IRC | 22:24 | |
| sean-k-mooney | efried: ok i think i have fixed the issue locally ill work on a patch. but basically i need to tweak the /etc/libvirt/libvirtd.conf sligtly and nova will be happy | 22:25 |
| *** ociuhandu has joined #openstack-nova | 22:25 | |
| sean-k-mooney | efried: http://paste.openstack.org/show/785827/ | 22:25 |
| openstackgerrit | Merged openstack/nova master: Follow up to I3e28c0163dc14dacf847c5a69730ba2e29650370 https://review.opendev.org/692856 | 22:27 |
| efried | sean-k-mooney: okay. I tried doing that conf change and now it's doing this | 22:27 |
| efried | Connection to libvirt failed: Failed to connect socket to '/var/run/libvirt/libvirt-sock': No such file or directory: libvirtError: Failed to connect socket to '/var/run/libvirt/libvirt-sock': No such file or directory | 22:27 |
| sean-k-mooney | where is that error? nova? virsh? | 22:28 |
| *** JamesBen_ has joined #openstack-nova | 22:28 | |
| efried | nova compute startup (journalctl) | 22:28 |
| sean-k-mooney | ok what do you get if you do "sudo systemctl status libvirt-bin" | 22:29 |
| *** rcernin has joined #openstack-nova | 22:29 | |
| *** ociuhandu has quit IRC | 22:30 | |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: Improve metadata server performance with large security groups https://review.opendev.org/656084 | 22:30 |
| mriedem | this is a simple performance improvement fix, been sitting a long time waiting for the originaly author to touch it up ^ | 22:30 |
| mriedem | and when i went to do it, i realized there was a much simpler fix | 22:30 |
| efried | d'oh, sorry, already started unstacking. | 22:31 |
| *** JamesBenson has quit IRC | 22:31 | |
| sean-k-mooney | no worries | 22:31 |
| efried | sean-k-mooney: I'm adding those extra libvirtd.conf lines (from within the plugin) and will restack. | 22:32 |
| sean-k-mooney | by the way i have code in the pluging to fix the app armor stuff | 22:32 |
| sean-k-mooney | sudo apparmor_parser -R /etc/apparmor.d/usr.sbin.libvirtd || /bin/true | 22:32 |
| sean-k-mooney | im guessing that is failing | 22:32 |
| sean-k-mooney | because i thinkg the package name chagned | 22:32 |
| *** JamesBen_ has quit IRC | 22:32 | |
| sean-k-mooney | so im fining that too | 22:32 |
| efried | are we not set -e here? | 22:33 |
| efried | oh, || true would ignore :( | 22:33 |
| sean-k-mooney | yes | 22:33 |
| sean-k-mooney | the /bin/ture was to ignore it on subsequent runs but its really just a hack | 22:34 |
| sean-k-mooney | install_build_deps for libvirt-bin used to pull in the apparmor utils but i apparently does not anymore so ill install them direcly | 22:35 |
| sean-k-mooney | im guessing you dont have apparmor_parser right | 22:35 |
| sean-k-mooney | i should not really depend on the transitive depencies like that | 22:36 |
| *** ociuhandu has joined #openstack-nova | 22:36 | |
| efried | sean-k-mooney: I have an apparmor_parser in /sbin | 22:37 |
| *** TxGirlGeek has quit IRC | 22:38 | |
| sean-k-mooney | hum ok it should have remvoed the proifle then but ill document the other way in anycase | 22:38 |
| efried | the file it's referring to doesn't exist fwiw | 22:39 |
| melwitt | mriedem: you were right, that solved the 403. now it's on to 409 "Multiple possible networks found, use a Network ID to be more specific". I'm sifting through NeutronFixture changes | 22:41 |
| *** TxGirlGeek has joined #openstack-nova | 22:41 | |
| efried | sean-k-mooney: with those extra lines, now stack fails when libvirt-bin won't start. | 22:41 |
| sean-k-mooney | efried: there is still a profile in /etc/apparmor.d/libvirt/ in my case | 22:41 |
| * melwitt pans for gold | 22:42 | |
| efried | that line is skipping | 22:42 |
| efried | +/opt/stack/devstack-plugin-libvirt-qemu/devstack/libs/libvirt:install_libvirt_qemu_src:210 sudo apparmor_parser -R /etc/apparmor.d/usr.sbin.libvirtd | 22:42 |
| efried | File /etc/apparmor.d/usr.sbin.libvirtd not found, skipping... | 22:42 |
| *** ociuhandu has quit IRC | 22:43 | |
| efried | the libvirt-bin service is failing thusly: | 22:43 |
| efried | Nov 05 16:40:14 nucle systemd[1]: libvirtd.service: Service hold-off time over, scheduling restart. | 22:43 |
| efried | Nov 05 16:40:14 nucle systemd[1]: libvirtd.service: Scheduled restart job, restart counter is at 5. | 22:43 |
| efried | Nov 05 16:40:14 nucle systemd[1]: Stopped Virtualization daemon. | 22:43 |
| efried | Nov 05 16:40:14 nucle systemd[1]: libvirtd.service: Start request repeated too quickly. | 22:43 |
| efried | Nov 05 16:40:14 nucle systemd[1]: libvirtd.service: Failed with result 'exit-code'. | 22:43 |
| efried | Nov 05 16:40:14 nucle systemd[1]: Failed to start Virtualization daemon. | 22:43 |
| sean-k-mooney | ya it has obviously moved | 22:43 |
| efried | I see these | 22:43 |
| efried | /etc/apparmor.d/cache/usr.sbin.libvirtd | 22:43 |
| efried | /opt/stack/libvirt/src/security/apparmor/usr.sbin.libvirtd | 22:43 |
| sean-k-mooney | do you have anything in the journal | 22:43 |
| *** ociuhandu has joined #openstack-nova | 22:43 | |
| efried | Assuming I would find it with journalctl -u libvirt-bin, no | 22:44 |
| sean-k-mooney | it would be libvirtd-bin | 22:44 |
| sean-k-mooney | but i dont think that is where its logging | 22:44 |
| efried | either way | 22:44 |
| efried | no | 22:44 |
| sean-k-mooney | ok give me a sec to push this up and test it | 22:45 |
| efried | crap, I f'ed up the libvirtd.conf hack, sec. | 22:45 |
| melwitt | mriedem: ok, looks like I just needed more of that change and now things are working. gonna try backporting the whole thing | 22:47 |
| mriedem | melwitt: which target branch? is this upstream? | 22:48 |
| mriedem | because i don't think we want to backport that whole thing... | 22:49 |
| *** ociuhandu has quit IRC | 22:49 | |
| melwitt | mriedem: this is downstream queens. really? ok | 22:50 |
| mriedem | well, it's big | 22:50 |
| mriedem | note you already have an abandoned backport of that upstream on stable/queens | 22:50 |
| mriedem | https://review.opendev.org/#/c/599764/ | 22:50 |
| melwitt | cause it looks like I need the bits removing the validate_networks stub, add the shared network stuff, and add stuff from this change too https://review.opendev.org/#/c/585385/15/nova/tests/fixtures.py | 22:51 |
| melwitt | lol welp | 22:51 |
| mriedem | you can't just do fake_network.set_stub_network_methods again? | 22:51 |
| melwitt | maybe I can. I have zero memory of doing that before | 22:51 |
| melwitt | I'll try it | 22:51 |
| melwitt | that would be way easier | 22:52 |
| sean-k-mooney | efried: im testing https://review.opendev.org/#/c/693094/1 now | 22:54 |
| *** luksky has quit IRC | 22:54 | |
| sean-k-mooney | efried: assuming that correct the issue ill see about setting up a tempest gate jobs that uses it and then you can just reuse/extend it for your own stuff | 22:55 |
| *** mriedem has quit IRC | 22:58 | |
| *** zbr|ooo has quit IRC | 23:00 | |
| *** yoctozepto has quit IRC | 23:04 | |
| *** yoctozepto has joined #openstack-nova | 23:04 | |
| openstackgerrit | Takashi NATSUME proposed openstack/python-novaclient master: doc: Fix supported version descriptions https://review.opendev.org/693097 | 23:05 |
| *** tbachman has joined #openstack-nova | 23:23 | |
| sean-k-mooney | efried: looks like i should be adding "--with-apparmor_profiles" when compiling to have it install them correctly. | 23:25 |
| sean-k-mooney | of cource the only mention of that flag in the libvirt repo is in the configure script ... | 23:26 |
| sean-k-mooney | sometimes i forget how "fun" C is | 23:27 |
| *** ociuhandu has joined #openstack-nova | 23:29 | |
| efried | sean-k-mooney: how do I check my version of qemu? | 23:34 |
| efried | n-cpu failed to start with | 23:35 |
| efried | Nova requires QEMU version 2.8.0 or greater. | 23:35 |
| efried | but the plugin should have set up 4.1.0, no? | 23:35 |
| *** takashin has quit IRC | 23:35 | |
| sean-k-mooney | correct | 23:35 |
| sean-k-mooney | http://paste.openstack.org/show/785829/ | 23:35 |
| sean-k-mooney | i have 4.1 | 23:35 |
| *** ociuhandu has quit IRC | 23:36 | |
| *** Liang__ has quit IRC | 23:38 | |
| openstackgerrit | melanie witt proposed openstack/nova stable/stein: Add regression test for bug 1824435 https://review.opendev.org/692906 | 23:39 |
| openstack | bug 1824435 in OpenStack Compute (nova) stein "fill_virtual_interface_list migration fails on second attempt" [Medium,In progress] https://launchpad.net/bugs/1824435 - Assigned to melanie witt (melwitt) | 23:39 |
| openstackgerrit | melanie witt proposed openstack/nova stable/stein: Remove redundant call to get/create default security group https://review.opendev.org/692907 | 23:39 |
| sean-k-mooney | efried: ill try running this all again on a clean ububut cloud image in the morning and ill provide you with the local.conf and anychagnes that are needed on the system if any in an update to the readme. | 23:45 |
| sean-k-mooney | one way or another its in my best interest to get this documented and updated in the repo because i will need this again at somepoint in the futrue | 23:46 |
| efried | sean-k-mooney: I gotta run and break some arms, but when I run the qemu commands I see 4.1, but obv nova thinks I've got something lesser | 23:53 |
| efried | anyway, thanks for the help so far, I'll bug you again tomorrow. | 23:53 |
| sean-k-mooney | no worries sorry it didnt "just work" | 23:54 |
| efried | :) | 23:55 |
| *** bnemec has joined #openstack-nova | 23:57 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!