| *** munimeha1 has quit IRC | 00:06 | |
| *** itlinux has joined #openstack-nova | 00:06 | |
| *** wolverineav has quit IRC | 00:14 | |
| *** wolverineav has joined #openstack-nova | 00:18 | |
| *** dklyle has quit IRC | 00:21 | |
| *** macza has quit IRC | 00:23 | |
| *** wolverineav has quit IRC | 00:23 | |
| *** itlinux_ has joined #openstack-nova | 00:25 | |
| *** itlinux has quit IRC | 00:28 | |
| *** dklyle has joined #openstack-nova | 00:34 | |
| *** xek__ has joined #openstack-nova | 00:37 | |
| *** mlavalle has quit IRC | 00:38 | |
| *** xek_ has quit IRC | 00:39 | |
| openstackgerrit | Merged openstack/nova master: Fix typo https://review.openstack.org/624442 | 00:40 |
|---|---|---|
| *** wolverineav has joined #openstack-nova | 00:48 | |
| *** Swami has quit IRC | 00:49 | |
| *** itlinux_ has quit IRC | 00:49 | |
| mriedem | dansmith: i think i've figured out why we're getting that ocata api service version warning in the logs - https://review.openstack.org/#/c/624438/ - something is re-entering _get_instance with a cell1-targeted context, so it's failing to find any nova-osapi_compute services in the cell1 db (as expected), i'm just not sure where the re-entry is coming from. | 00:52 |
| *** dakshina-ilangov has quit IRC | 00:53 | |
| mriedem | and that's why we don't get a 404 from the Instance.get_by_uuid call because the context is already targeted | 00:56 |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: DNM: add debug logging for bug 1807723 https://review.openstack.org/624181 | 01:04 |
| openstack | bug 1807723 in tempest "test_volume_swap_with_multiattach intermittently fails during cleanup" [High,In progress] https://launchpad.net/bugs/1807723 - Assigned to Matt Riedemann (mriedem) | 01:04 |
| *** dklyle has quit IRC | 01:04 | |
| *** brinzhang has joined #openstack-nova | 01:09 | |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: DNM: debug old compute API service version lookup https://review.openstack.org/624438 | 01:14 |
| *** dave-mccowan has joined #openstack-nova | 01:14 | |
| *** mriedem has quit IRC | 01:23 | |
| *** yedongcan has joined #openstack-nova | 01:34 | |
| *** erlon has quit IRC | 01:35 | |
| *** markvoelker has quit IRC | 01:41 | |
| *** cfriesen has quit IRC | 01:59 | |
| *** Dinesh_Bhor has joined #openstack-nova | 02:01 | |
| *** mrsoul has quit IRC | 02:09 | |
| *** brinzhang has quit IRC | 02:10 | |
| *** brinzhang has joined #openstack-nova | 02:10 | |
| openstackgerrit | Yikun Jiang proposed openstack/nova master: Add live migration timeout action https://review.openstack.org/619143 | 02:13 |
| *** wolverineav has quit IRC | 02:17 | |
| *** wolverineav has joined #openstack-nova | 02:18 | |
| *** bhagyashris_ has joined #openstack-nova | 02:21 | |
| *** wolverineav has quit IRC | 02:22 | |
| *** Dinesh_Bhor has quit IRC | 02:27 | |
| *** mhen has quit IRC | 02:28 | |
| *** mhen has joined #openstack-nova | 02:31 | |
| *** Dinesh_Bhor has joined #openstack-nova | 02:33 | |
| openstackgerrit | Yikun Jiang proposed openstack/nova master: Add live migration timeout action https://review.openstack.org/619143 | 02:41 |
| *** cfriesen has joined #openstack-nova | 02:44 | |
| *** hongbin has joined #openstack-nova | 02:44 | |
| *** igordc has quit IRC | 03:07 | |
| openstackgerrit | Yikun Jiang proposed openstack/nova master: Remove live_migration_progress_timeout config https://review.openstack.org/619142 | 03:26 |
| *** psachin has joined #openstack-nova | 03:27 | |
| *** wolverineav has joined #openstack-nova | 03:30 | |
| *** Dinesh_Bhor has quit IRC | 03:34 | |
| openstackgerrit | Merged openstack/nova master: Only warn about not having computes nodes once in rpcapi https://review.openstack.org/623282 | 03:35 |
| *** dave-mccowan has quit IRC | 03:52 | |
| openstackgerrit | Merged openstack/nova master: Handle tags in _bury_in_cell0 https://review.openstack.org/621856 | 03:58 |
| *** udesale has joined #openstack-nova | 04:17 | |
| *** Dinesh_Bhor has joined #openstack-nova | 04:42 | |
| *** hongbin has quit IRC | 04:55 | |
| *** brault has joined #openstack-nova | 05:09 | |
| *** brault has quit IRC | 05:14 | |
| *** wolverineav has quit IRC | 05:17 | |
| *** maciejjozefczyk has quit IRC | 05:19 | |
| *** rtjure has quit IRC | 05:19 | |
| *** maciejjozefczyk has joined #openstack-nova | 05:19 | |
| *** ratailor has joined #openstack-nova | 05:20 | |
| *** itlinux has joined #openstack-nova | 05:21 | |
| *** rtjure has joined #openstack-nova | 05:22 | |
| *** wolverineav has joined #openstack-nova | 05:24 | |
| *** dklyle has joined #openstack-nova | 05:51 | |
| *** itlinux has quit IRC | 05:51 | |
| *** dklyle has quit IRC | 05:56 | |
| *** _alastor_ has joined #openstack-nova | 06:06 | |
| *** wolverineav has quit IRC | 06:09 | |
| *** Dinesh_Bhor has quit IRC | 06:13 | |
| *** sridharg has joined #openstack-nova | 06:15 | |
| *** slaweq has joined #openstack-nova | 06:29 | |
| *** lpetrut has joined #openstack-nova | 06:30 | |
| *** cfriesen has quit IRC | 06:31 | |
| *** _alastor_ has quit IRC | 06:45 | |
| *** Dinesh_Bhor has joined #openstack-nova | 06:47 | |
| *** moshele has joined #openstack-nova | 07:00 | |
| *** Luzi has joined #openstack-nova | 07:01 | |
| openstackgerrit | Michael Still proposed openstack/nova master: Imagebackend should call processutils.execute directly. https://review.openstack.org/619704 | 07:09 |
| openstackgerrit | Michael Still proposed openstack/nova master: Remove final users of utils.execute() in libvirt. https://review.openstack.org/619705 | 07:09 |
| openstackgerrit | Michael Still proposed openstack/nova master: Remove the final user of utils.execute() from virt.images https://review.openstack.org/620007 | 07:09 |
| openstackgerrit | Michael Still proposed openstack/nova master: Remove utils.execute() from the hyperv driver. https://review.openstack.org/620008 | 07:09 |
| openstackgerrit | Michael Still proposed openstack/nova master: Remove utils.execute() from virt.disk.api. https://review.openstack.org/620009 | 07:09 |
| openstackgerrit | Michael Still proposed openstack/nova master: Move a generic bridge helper to a linux_net privsep file. https://review.openstack.org/620010 | 07:09 |
| openstackgerrit | Michael Still proposed openstack/nova master: Move bridge creation to privsep. https://review.openstack.org/620180 | 07:09 |
| openstackgerrit | Michael Still proposed openstack/nova master: Move some linux network helpers to use privsep. https://review.openstack.org/621398 | 07:09 |
| openstackgerrit | Michael Still proposed openstack/nova master: Move simple execute call to processutils. https://review.openstack.org/621527 | 07:09 |
| openstackgerrit | Michael Still proposed openstack/nova master: Move interface enabling to privsep. https://review.openstack.org/621528 | 07:09 |
| openstackgerrit | Michael Still proposed openstack/nova master: Move setting mac addresses for network devices to privsep. https://review.openstack.org/621529 | 07:09 |
| openstackgerrit | Michael Still proposed openstack/nova master: Move interface disabling to privsep. https://review.openstack.org/622150 | 07:09 |
| openstackgerrit | Michael Still proposed openstack/nova master: Move binding ips to privsep. https://review.openstack.org/622151 | 07:09 |
| openstackgerrit | Michael Still proposed openstack/nova master: create_veth_pair is unused, remove it. https://review.openstack.org/624226 | 07:09 |
| openstackgerrit | Michael Still proposed openstack/nova master: Create specialist set_macaddr_and_vlan helper. https://review.openstack.org/624227 | 07:09 |
| openstackgerrit | Michael Still proposed openstack/nova master: Move create_tap_dev into privsep. https://review.openstack.org/624228 | 07:09 |
| openstackgerrit | Michael Still proposed openstack/nova master: Move set_vf_interface_vlan to be with its only caller. https://review.openstack.org/624229 | 07:09 |
| openstackgerrit | Michael Still proposed openstack/nova master: Move DHCP releasing to privsep. https://review.openstack.org/624230 | 07:09 |
| *** masayukig[m] has joined #openstack-nova | 07:13 | |
| *** wolverineav has joined #openstack-nova | 07:25 | |
| *** openstackgerrit has quit IRC | 07:29 | |
| *** dklyle has joined #openstack-nova | 07:29 | |
| *** wolverineav has quit IRC | 07:29 | |
| *** rcernin has quit IRC | 07:30 | |
| *** ivve has quit IRC | 07:30 | |
| *** alexchadin has joined #openstack-nova | 07:31 | |
| *** sahid has joined #openstack-nova | 07:43 | |
| *** lpetrut has quit IRC | 07:46 | |
| *** trident has quit IRC | 07:58 | |
| *** trident has joined #openstack-nova | 08:00 | |
| *** imacdonn has quit IRC | 08:23 | |
| *** imacdonn has joined #openstack-nova | 08:23 | |
| *** brault has joined #openstack-nova | 08:23 | |
| *** brault has quit IRC | 08:24 | |
| *** brault has joined #openstack-nova | 08:24 | |
| *** helenafm has joined #openstack-nova | 08:24 | |
| *** dklyle has quit IRC | 08:32 | |
| *** ccamacho has joined #openstack-nova | 08:37 | |
| *** ralonsoh has joined #openstack-nova | 08:41 | |
| *** markvoelker has joined #openstack-nova | 08:44 | |
| *** markvoelker has quit IRC | 08:49 | |
| *** jonher_ has joined #openstack-nova | 08:57 | |
| *** tssurya has joined #openstack-nova | 08:59 | |
| *** jonher has quit IRC | 09:00 | |
| *** jonher_ is now known as jonher | 09:00 | |
| *** ivve has joined #openstack-nova | 09:04 | |
| *** openstackgerrit has joined #openstack-nova | 09:10 | |
| openstackgerrit | Zhenyu Zheng proposed openstack/nova master: WIP run metadata api per cell https://review.openstack.org/624612 | 09:10 |
| *** mtreinish has quit IRC | 09:11 | |
| *** k_mouza has joined #openstack-nova | 09:13 | |
| *** yan0s has joined #openstack-nova | 09:15 | |
| *** dtantsur|afk is now known as dtantsur | 09:19 | |
| yan0s | hi all | 09:19 |
| *** k_mouza has quit IRC | 09:19 | |
| yan0s | why isn't there a "nova quota-class-list" command? | 09:20 |
| yan0s | or "openstack quota list --class" | 09:20 |
| *** k_mouza has joined #openstack-nova | 09:20 | |
| yan0s | how can I get list of defined quota classes? | 09:20 |
| *** psachin is now known as psachin|session | 09:35 | |
| *** markvoelker has joined #openstack-nova | 09:39 | |
| *** lpetrut has joined #openstack-nova | 09:41 | |
| *** moshele has quit IRC | 09:42 | |
| *** derekh has joined #openstack-nova | 09:44 | |
| *** lpetrut has quit IRC | 09:45 | |
| ondrejme | yan0s: nova quota-defaults ? | 09:51 |
| *** ivve has quit IRC | 09:52 | |
| *** bhagyashris_ has quit IRC | 09:53 | |
| *** ohorecny2 has joined #openstack-nova | 09:54 | |
| *** mtreinish has joined #openstack-nova | 09:57 | |
| *** cdent has joined #openstack-nova | 10:20 | |
| *** Dinesh_Bhor has quit IRC | 10:24 | |
| openstackgerrit | Lee Yarwood proposed openstack/nova master: libvirt: Add workaround to cleanup instance dir during evac with rbd https://review.openstack.org/618478 | 10:40 |
| *** Dinesh_Bhor has joined #openstack-nova | 10:49 | |
| *** erlon has joined #openstack-nova | 10:57 | |
| *** rodolof has joined #openstack-nova | 11:02 | |
| *** udesale has quit IRC | 11:02 | |
| *** udesale has joined #openstack-nova | 11:03 | |
| *** lpetrut has joined #openstack-nova | 11:03 | |
| *** moshele has joined #openstack-nova | 11:13 | |
| *** yedongcan has left #openstack-nova | 11:14 | |
| *** markvoelker has quit IRC | 11:30 | |
| *** Eran_Kuris has joined #openstack-nova | 11:30 | |
| *** Dinesh_Bhor has quit IRC | 11:33 | |
| yan0s | ondrejme: it is possible to set different quota per class | 11:34 |
| yan0s | and it possible to see the quota set in this class if you know the class name | 11:34 |
| yan0s | but it is not possible to get a list of available classes | 11:35 |
| yan0s | the command you sent me returns the default quota values | 11:36 |
| openstackgerrit | Lee Yarwood proposed openstack/nova stable/rocky: Only warn about not having computes nodes once in rpcapi https://review.openstack.org/624677 | 11:42 |
| *** ttsiouts has joined #openstack-nova | 11:47 | |
| *** k_mouza has quit IRC | 11:51 | |
| *** yikun_ has quit IRC | 11:51 | |
| stephenfin | lyarwood: I was wondering why nova-api (or was it the conductor?) was spewing all those logs ^ | 11:53 |
| *** jamesdenton has quit IRC | 11:56 | |
| lyarwood | stephenfin: that's dansmith's fix, I'm just pulling it back into stable/rocky | 11:56 |
| *** dtantsur is now known as dtantsur|brb | 11:56 | |
| *** jamesdenton has joined #openstack-nova | 12:00 | |
| *** moshele has quit IRC | 12:03 | |
| *** moshele has joined #openstack-nova | 12:04 | |
| *** markvoelker has joined #openstack-nova | 12:05 | |
| *** ttsiouts has quit IRC | 12:07 | |
| *** ttsiouts has joined #openstack-nova | 12:07 | |
| *** k_mouza has joined #openstack-nova | 12:09 | |
| *** ttsiouts has quit IRC | 12:12 | |
| openstackgerrit | Chris Dent proposed openstack/os-resource-classes master: Sync STANDARDS with placement/nova rc_fields https://review.openstack.org/624370 | 12:12 |
| openstackgerrit | Chris Dent proposed openstack/os-resource-classes master: Tune up documentation to be more useful https://review.openstack.org/624384 | 12:13 |
| *** brault has quit IRC | 12:34 | |
| *** dave-mccowan has joined #openstack-nova | 12:40 | |
| *** ratailor has quit IRC | 12:54 | |
| *** ttsiouts has joined #openstack-nova | 12:54 | |
| openstackgerrit | Brin Zhang proposed openstack/nova-specs master: Specifying az when restore shelved server https://review.openstack.org/624689 | 12:56 |
| openstackgerrit | Brin Zhang proposed openstack/nova-specs master: Specifying az when restore shelved server https://review.openstack.org/624689 | 12:58 |
| *** moshele has quit IRC | 12:59 | |
| *** brinzhang has quit IRC | 13:00 | |
| *** Eran_Kuris has quit IRC | 13:04 | |
| *** k_mouza has quit IRC | 13:19 | |
| *** dtantsur|brb is now known as dtantsur | 13:20 | |
| *** k_mouza has joined #openstack-nova | 13:27 | |
| *** markvoelker has quit IRC | 13:28 | |
| *** k_mouza_ has joined #openstack-nova | 13:30 | |
| *** tbachman has joined #openstack-nova | 13:30 | |
| *** k_mouza__ has joined #openstack-nova | 13:31 | |
| *** k_mouza has quit IRC | 13:32 | |
| *** k_mouza_ has quit IRC | 13:34 | |
| adrianc | sean-k-mooney: Hi, regarding the issue you experienced yesterday with live-migration, ping me when you are here | 13:39 |
| *** eharney has joined #openstack-nova | 13:40 | |
| sean-k-mooney | adrianc: ping | 13:46 |
| sean-k-mooney | :) | 13:46 |
| adrianc | sean-k-mooney: so apparently there is some weird behavior with sriov macvtap when you tear down the VM the VF netdev remains up | 13:49 |
| adrianc | so you have a scenario of duplicate macs in the network | 13:49 |
| adrianc | doing ifconfig <VF-netdev> down on the source VF should fix the issue (at least on my setup it did) | 13:50 |
| adrianc | sean-k-mooney: i had a discussion with moshele about it, apparently this is an old issue which branches off this mail: https://www.redhat.com/archives/libvir-list/2016-September/msg00075.html | 13:52 |
| sean-k-mooney | ah ok | 13:53 |
| sean-k-mooney | so i had considered creating an os-vif plugin for sriov this cycle. | 13:53 |
| adrianc | sriov netdev mac address will be re-allocated only on driver rebind on the VF | 13:53 |
| sean-k-mooney | we can do it in the nova tree but we could also jsut create the plugin and have it set the vf down | 13:54 |
| adrianc | if you do passthrough then it will be properly cleaned, however on macvtap this is not the case | 13:54 |
| sean-k-mooney | ya | 13:54 |
| adrianc | i added a hook in my code to do: interface down in vif.py under unplug_hw_veb() | 13:54 |
| sean-k-mooney | ok cool that should work for now. | 13:55 |
| sean-k-mooney | i would like to move that into os-vif later i also want to move the trusted vf logic into os-vif which was why i was orignally gong to create the sriov plugin | 13:56 |
| *** dtantsur is now known as dtantsur|brb | 13:56 | |
| sean-k-mooney | adrianc: do you have a new patchset up with that change. ill redeploy and test | 13:56 |
| adrianc | not yet, i need to deploy the change and make sure it actually does the job. it worked manually. | 13:58 |
| adrianc | sean-k-mooney: it should not be tied to the neutron-sriov-live-migration work as it serves as a WA to an existing problem. | 14:00 |
| *** markvoelker has joined #openstack-nova | 14:01 | |
| adrianc | let me know if doing ifconfig <netdev> down on the source vf netdev solves the issue for you. (after migration) | 14:01 |
| sean-k-mooney | that is true but you could make it the base patch of the series so its easy to test all of them | 14:01 |
| sean-k-mooney | sure i need to restack but ill let you know in an hour or so | 14:02 |
| adrianc | sean-k-mooney: you will still need some POC code in neutron for multiple port binding | 14:02 |
| sean-k-mooney | ya i am useing that patch too | 14:03 |
| sean-k-mooney | devstack can deploy with unmerged patches from gerrit | 14:04 |
| sean-k-mooney | but it cant combine two sets fo unrelated patches | 14:04 |
| sean-k-mooney | so to test at the moment i jsut need to point to the top patch on the chain in nova and the neutron patch | 14:04 |
| adrianc | can be applied manually after deploy :) | 14:05 |
| sean-k-mooney | yep ill just create a branch locally and cherrypick them | 14:06 |
| sean-k-mooney | its fine | 14:06 |
| *** mriedem has joined #openstack-nova | 14:13 | |
| *** jonher_ has joined #openstack-nova | 14:17 | |
| *** rodolof has quit IRC | 14:20 | |
| *** jonher has quit IRC | 14:21 | |
| *** jonher_ is now known as jonher | 14:21 | |
| *** rodolof has joined #openstack-nova | 14:21 | |
| mriedem | melwitt: a change of mine hit the same vif plug timeout thing on that test with 7 ports http://logs.openstack.org/46/623246/3/check/nova-multiattach/efa830b/logs/screen-n-cpu.txt.gz?level=TRACE#_Dec_12_00_01_13_179474 | 14:28 |
| mriedem | bauzas: can you look at this? https://review.openstack.org/#/c/615347/ - it drops old confusing compat which is actually contributing to lots of misleading warnings in the n-api logs and gate race failures | 14:29 |
| bauzas | mriedem: sure | 14:29 |
| sean-k-mooney | mriedem ya i have seen that a few times | 14:29 |
| *** k_mouza has joined #openstack-nova | 14:32 | |
| *** irclogbot_2 has quit IRC | 14:32 | |
| mriedem | it's always the first port that we're missing | 14:33 |
| mriedem | we get 6 of the 7 network-vif-plugged events | 14:33 |
| sean-k-mooney | mriedem: one edgecase that i am awre of is in some configuration we start waithing for the network events too late | 14:33 |
| sean-k-mooney | we start waiting for the event just before we call plug on the interface | 14:34 |
| sean-k-mooney | for some neutron backend they send that event when we call bind | 14:34 |
| sean-k-mooney | i assume this is ml2/ovs however correct? | 14:34 |
| sean-k-mooney | that should not hit the race but i wonde if ther i a missed event in the logs | 14:35 |
| mriedem | yes ovs | 14:35 |
| *** k_mouza__ has quit IRC | 14:35 | |
| *** k_mouza has quit IRC | 14:36 | |
| *** k_mouza has joined #openstack-nova | 14:37 | |
| sean-k-mooney | mriedem: http://logs.openstack.org/46/623246/3/check/nova-multiattach/efa830b/logs/screen-n-cpu.txt.gz?#_Dec_12_00_01_07_481084 | 14:37 |
| mriedem | we plug it here | 14:37 |
| mriedem | Dec 11 23:56:11.855811 ubuntu-xenial-inap-mtl01-0001136812 nova-compute[29399]: INFO os_vif [None req-f43f3c35-dc0b-4aa4-bfd0-4046045b315e tempest-TaggedBootDevicesTest-250467933 tempest-TaggedBootDevicesTest-250467933] Successfully plugged vif VIFOpenVSwitch(active=False,address=fa:16:3e:fb:dd:9f,bridge_name='br-int',has_traffic_filtering=True,id=472ab433-bc18-41b5-85ae-009611117b70,network=Network(4988ef3d-6a8f-477f-b0a2-29 | 14:37 |
| mriedem | 019494),plugin='ovs',port_profile=VIFPortProfileOpenVSwitch,preserve_on_delete=False,vif_name='tap472ab433-bc') | 14:37 |
| mriedem | hmm wtf | 14:38 |
| mriedem | i also see this right before that port is plugged | 14:38 |
| mriedem | Dec 11 23:55:46.981194 ubuntu-xenial-inap-mtl01-0001136812 neutron-server[20899]: DEBUG neutron.notifiers.nova [-] Sending events: [{'tag': u'472ab433-bc18-41b5-85ae-009611117b70', 'name': 'network-changed', 'server_uuid': u'18e5987e-a535-4044-84aa-dd8f023cedcc'}] {{(pid=21001) send_events /opt/stack/new/neutron/neutron/notifiers/nova.py:245}} | 14:38 |
| mriedem | sean-k-mooney: that's a different port | 14:39 |
| mriedem | the unplugged event | 14:39 |
| sean-k-mooney | mriedem: yes i just noticed that | 14:39 |
| *** irclogbot_2 has joined #openstack-nova | 14:40 | |
| mriedem | asking in -neutron | 14:42 |
| *** moshele has joined #openstack-nova | 14:42 | |
| *** mmethot has quit IRC | 14:43 | |
| *** mmethot has joined #openstack-nova | 14:46 | |
| *** moshele has quit IRC | 14:47 | |
| *** irclogbot_2 has quit IRC | 14:52 | |
| frickler | nova stable cores: https://review.openstack.org/619254 is waiting for a second +2, if it gets that, I can start nagging mriedem about stable releases instead ;-) | 14:53 |
| *** yan0s has quit IRC | 14:55 | |
| *** mlavalle has joined #openstack-nova | 15:02 | |
| *** irclogbot_2 has joined #openstack-nova | 15:03 | |
| jangutter | mriedem: yonks ago I saw something similar that happened if the neutron-ovsdb linkage got busted. | 15:07 |
| mriedem | jangutter: i did see this in the neutron agent logs | 15:09 |
| mriedem | Dec 11 23:56:12.725010 ubuntu-xenial-inap-mtl01-0001136812 neutron-openvswitch-agent[21879]: INFO neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [None req-cc26f856-2521-411e-ae10-34ad96b7c665 None None] Port 472ab433-bc18-41b5-85ae-009611117b70 was not found on the integration bridge and will therefore not be processed | 15:09 |
| *** awaugama has joined #openstack-nova | 15:09 | |
| jangutter | mriedem: there's some extra metadata added by libvirt that links the port uuid. years ago I saw neutron triggering off of that. | 15:10 |
| adrianc | sean-k-mooney: OK, so apparently on my setup i need to unbind/bind the VF, taking down the interface is not sufficient. this may be NIC vendor specific, let me know if interface down works for you OR you need to rebind the VF to flush out the MAC from the source node | 15:11 |
| *** awaugama has quit IRC | 15:14 | |
| *** awaugama has joined #openstack-nova | 15:15 | |
| *** markvoelker has quit IRC | 15:18 | |
| *** yan0s has joined #openstack-nova | 15:20 | |
| sean-k-mooney | adrianc: rebinding the vf will be a problem if needed | 15:21 |
| sean-k-mooney | nova doen not always have permissions to do that. | 15:22 |
| *** alexchadin has quit IRC | 15:22 | |
| *** lpetrut has quit IRC | 15:22 | |
| mriedem | hello people - easy fix to get the postgresql job working again https://review.openstack.org/#/c/619061/ | 15:23 |
| mriedem | cdent: ^ since you seem to care about pg | 15:23 |
| mriedem | slaweq: you too ^ | 15:23 |
| adrianc | sean-k-mooney: the fix should be in libvirt imo, once the guest is not on the machine, it should properly clean-up. however for the direct SRIOV you will not have this problem | 15:23 |
| cdent | mriedem: it's less care about pg, more care about not-just-mysql | 15:24 |
| cdent | but yeah, looking | 15:24 |
| sean-k-mooney | adrianc: well libvirt or qemu but yes one of the two shoudl notify the pf to reset the vf | 15:24 |
| sean-k-mooney | adrianc: yes direct mode should work as the vf will be reset when it detached form the guest kernel and released to the host kernel | 15:25 |
| jangutter | sean-k-mooney, adrianc: I think qemu does a function-level-reset on the VF when doing iommu passthrough. | 15:25 |
| sean-k-mooney | jangutter: on a live migration | 15:25 |
| jangutter | sean-k-mooney, adrianc: and libvirt does driver binding and unbinding. | 15:25 |
| *** dtantsur|brb is now known as dtantsur | 15:25 | |
| sean-k-mooney | jangutter: in the macvtap case you do not bind the diriver to vfio-pci | 15:25 |
| slaweq | mriedem: it is this fix which will finally fix neutron pgsql periodic job, right? | 15:26 |
| sean-k-mooney | jangutter: you leave it bound to the host kernel driver at all times | 15:26 |
| jangutter | sean-k-mooney: yep, but permission-wise, libvirt could bind/unbind it as a workaround. | 15:26 |
| jangutter | sean-k-mooney: and generally, you _don't_ want to do a FLR on something bound to a driver. | 15:26 |
| sean-k-mooney | jangutter: it could yes but it wont today meaning we would need a new libvirt release and need to check for it in the code | 15:26 |
| jangutter | sean-k-mooney: yep, tell me about it, I know all about people wanting to propagate libvirt "fixes". | 15:27 |
| sean-k-mooney | i think this would be somthing that the sriov nic agent could resonalby handel too | 15:28 |
| sean-k-mooney | well maybe not | 15:28 |
| sean-k-mooney | that could be racy | 15:28 |
| sean-k-mooney | it would have to be nova/libvirt to avoid races with other vms booting | 15:29 |
| adrianc | sean-k-mooney: there is no FLR involved during detach/attach, the driver unbinds from the guest, and binds on the host whether or not it triggers an FLR is something the driver decides AFAIK | 15:29 |
| jangutter | adrianc: yep, FLR is mandatory on iommu passthrough I think, but not for driver bind/unbind | 15:29 |
| sean-k-mooney | well it depend on if the vf is bound to say ixgbeVf when its detach or if it stays with vfio-pci | 15:30 |
| jangutter | sean-k-mooney: yeah. | 15:30 |
| adrianc | would have been easier if setting VF mac address via ip link on the VF idx would have synced the mac address on the corresponding VF netdev :) | 15:31 |
| sean-k-mooney | in any case however in macvtap mode we aparent dont update the driver the vf is bound to or trigger a vf reset | 15:32 |
| sean-k-mooney | adrianc: that silly talk why would a hardware/driver implementer ever do a thing like that :P | 15:32 |
| openstackgerrit | Balazs Gibizer proposed openstack/nova master: Add python 3.7 unit and functional tox jobs https://review.openstack.org/624055 | 15:32 |
| openstackgerrit | Balazs Gibizer proposed openstack/nova master: Replace ThreadPoolExecutor with GreenThreadPoolExecutor https://review.openstack.org/624740 | 15:32 |
| gibi | cdent: ^^ | 15:32 |
| cdent | roger | 15:32 |
| mriedem | slaweq: yeah | 15:33 |
| jangutter | sean-k-mooney, adrianc: years ago we had looked at how to propagate MAC config from host to the VF and the conclusion we came to was that at the FLR is the safest way to do so. I don't think there was an async event able to propagate that info from the device to the driver without either a reset or rebind. | 15:33 |
| slaweq | mriedem: thx for taking care of it :) | 15:33 |
| sean-k-mooney | adrianc: speacking of ip link ther eis no way to reset it via ip tools right. | 15:33 |
| jangutter | sean-k-mooney, adrianc: the only way I know to trigger an FLR is via setpci... | 15:34 |
| sean-k-mooney | jangutter: i was hoping there would be a standard sysfs file we could echo too but i guess not | 15:35 |
| aspiers | mriedem: if I understand the code right, the config drive disk mapping is built alongside the other disk mappings prior to instance launch (even though the config drive itself seems to be built late via post_xml_callback). so it's not hotplugged post-launch. Also I think it will either use "ide" or "virtio" bus (depending on whether iso9660 or vfat) and therefore we don't anticipate any problems with | 15:35 |
| aspiers | SEV. Nevertheless we're testing to make sure | 15:35 |
| jangutter | sean-k-mooney: there might be these days, though. | 15:35 |
| sean-k-mooney | jangutter: it would still likely be vendor specific | 15:35 |
| jangutter | sean-k-mooney: very, I believe our driver only recently could propagate MAC on rebind. Previously you _had_ to get an FLR. | 15:36 |
| sean-k-mooney | adrianc: im just getting around to setting up that test env now but i will try a few different things and let you know | 15:36 |
| sean-k-mooney | should i put this in the spec by the way. im inclined to say no as this is too technical/hardware specific | 15:37 |
| adrianc | sean-k-mooney: ack, let me know if you can find a nicer WA | 15:37 |
| adrianc | well i dont think its related to the spec, its a long existing problem IMO and not related to live-migration | 15:38 |
| sean-k-mooney | we need to adress this in the code obviously but i think code commetns/dev docs | 15:38 |
| *** mchlumsky has quit IRC | 15:38 | |
| sean-k-mooney | adrianc: i guess you could consider it a cold migration bug | 15:38 |
| adrianc | there are additional gaps with macVtap like multiple port binding so | 15:38 |
| *** rodolof has quit IRC | 15:38 | |
| *** rodolof has joined #openstack-nova | 15:39 | |
| *** mchlumsky has joined #openstack-nova | 15:40 | |
| *** maciejjozefczyk has quit IRC | 15:40 | |
| adrianc | sean-k-mooney: it can happen also without LM being involved, easiest is to boot a VM on node A delete and boot it on node B | 15:40 |
| adrianc | the mac will be reused and you hit the issue | 15:41 |
| sean-k-mooney | adrianc: reusing the same neutorn port so that the mac and vlan remain the same | 15:41 |
| adrianc | adrianc: yes, using the same neutron port | 15:41 |
| adrianc | sean-k-mooney: ^ | 15:42 |
| sean-k-mooney | ya ok that is proably ground enought to just treat this as a seperate bug and maybe backport the fix | 15:42 |
| openstackgerrit | Lee Yarwood proposed openstack/nova master: libvirt: Add workaround to cleanup instance dir during evac with rbd https://review.openstack.org/618478 | 15:43 |
| lyarwood | mriedem / mdbooth / melwitt ; ^ thanks again for the reviews, I appreciate just how ugly this is but hopefully this is btw but with downstream customers asking for this to be fixed back in Newton I'm being forced into a corner here tbh. | 15:45 |
| lyarwood | thanks weechat-- | 15:46 |
| sean-k-mooney | lyarwood: what wrong with weechat++ | 15:46 |
| lyarwood | ^ the above mess, I was trying to edit the line and it decided to send it out, fun times. | 15:47 |
| *** lpetrut has joined #openstack-nova | 15:48 | |
| *** lpetrut has quit IRC | 15:56 | |
| *** tssurya has quit IRC | 15:57 | |
| *** Luzi has quit IRC | 15:57 | |
| mriedem | melwitt: did you see my question in your spec for configurable max volumes? https://review.openstack.org/#/c/597306/1/specs/stein/approved/conf-max-attach-volumes.rst@44 | 16:02 |
| melwitt | mriedem: I did, sorry. the immediate answer was "not 100% sure, need to check my notes again". it was "fun" tracing it all and I can't remember whether bfv vs attach additional end up using the same code path that leads to the limit of 26. I need to double check it in order to reply and haven't gotten to it yet | 16:05 |
| mriedem | ok, but i'm assuming even if it's not the same code path, bfv would need a check somewhere for the config option to limit the number of volumes attached to an instance during spawn | 16:09 |
| mriedem | it might have to be here https://github.com/openstack/nova/blob/master/nova/compute/manager.py#L1598 | 16:10 |
| mriedem | or a new driver method called from _prep_block_devices | 16:10 |
| mriedem | because you don't want to create or attach more than the limit before you get to driver.spawn just to have it fail and have to roll those all back | 16:10 |
| *** gyee has joined #openstack-nova | 16:19 | |
| *** udesale has quit IRC | 16:19 | |
| melwitt | mriedem: yeah, I guess the context is in my WIP patch for the implementation I've put the check at a lower level where all of the code paths end up https://review.openstack.org/616777 there's a default impl in compute/utils that all drivers other than libvirt use, and libvirt has its own override | 16:20 |
| melwitt | I don't have any func test coverage yet to verify it all works right, so I'm in process of writing those | 16:20 |
| melwitt | but if you want to take a quick look to point out anything bad that jumps out, I'd appreciate that | 16:20 |
| mriedem | yeah that won't cut it for bfv | 16:27 |
| mriedem | i'll leave a comment | 16:27 |
| melwitt | thanks | 16:28 |
| *** ttsiouts has quit IRC | 16:29 | |
| *** ttsiouts has joined #openstack-nova | 16:30 | |
| *** ttsiouts has quit IRC | 16:30 | |
| *** ttsiouts has joined #openstack-nova | 16:31 | |
| *** _alastor_ has joined #openstack-nova | 16:32 | |
| *** macza has joined #openstack-nova | 16:34 | |
| openstackgerrit | Merged openstack/nova master: Drop pre-cellsv2 compat in compute API.get() https://review.openstack.org/615347 | 16:38 |
| *** moshele has joined #openstack-nova | 16:40 | |
| *** _alastor_ has quit IRC | 16:40 | |
| *** cfriesen has joined #openstack-nova | 16:42 | |
| *** cfriesen has quit IRC | 16:42 | |
| *** cfriesen has joined #openstack-nova | 16:42 | |
| *** sahid has quit IRC | 16:47 | |
| *** sahid has joined #openstack-nova | 16:47 | |
| *** tssurya has joined #openstack-nova | 16:51 | |
| *** itlinux has joined #openstack-nova | 16:53 | |
| openstackgerrit | Chris Dent proposed openstack/os-resource-classes master: Sync STANDARDS with placement/nova rc_fields https://review.openstack.org/624370 | 16:53 |
| openstackgerrit | Chris Dent proposed openstack/os-resource-classes master: Tune up documentation to be more useful https://review.openstack.org/624384 | 16:53 |
| moshele | jangutter: hi | 16:56 |
| jangutter | moshele: hi, long time no chat! | 16:57 |
| moshele | jangutter: yes indeed | 16:57 |
| moshele | jangutter: I am not sure I understand you comments | 16:58 |
| jangutter | moshele: sorry, it's a bit late in the year.... | 16:58 |
| moshele | jangutter: https://review.openstack.org/#/c/619920/ | 16:58 |
| dansmith | mriedem: was your lazy-load patch trying to get rid of the billion "Lock placement_client acquired" messages on startup? | 16:58 |
| *** yan0s has quit IRC | 16:59 | |
| jangutter | moshele: the first question is possibly the easiest: on the NIC you have two ports, B and C. I would have thought that it makes sense specifying "connect B and C" to the bridge? | 17:00 |
| mriedem | dansmith: yes | 17:02 |
| mriedem | https://review.openstack.org/#/c/623246/ | 17:02 |
| dansmith | mriedem: yeah that's pretty bad | 17:03 |
| mriedem | it now removes the lock as well on jay's request | 17:03 |
| dansmith | mriedem: just noticed while reviewing the logs forthis: https://review.openstack.org/#/c/623284/4 | 17:06 |
| *** ohorecny2 has quit IRC | 17:06 | |
| dansmith | mriedem: which does now select the right rpc version once computes get started | 17:06 |
| dansmith | which was not happening before | 17:06 |
| sean-k-mooney | adrianc: so yes if i just do "sudo ifconfig enp1s17f5 0" i does not fix the issue either | 17:07 |
| openstackgerrit | Balazs Gibizer proposed openstack/nova master: Ensure that allocated PF matches the used PF https://review.openstack.org/623543 | 17:07 |
| mriedem | dansmith: which job was that message from? | 17:08 |
| mriedem | because on tempest-full nova-api starts up before there are any computes | 17:08 |
| *** helenafm has quit IRC | 17:08 | |
| dansmith | mriedem: tempest-full | 17:08 |
| sean-k-mooney | adrianc: sorry "sudo ifconfig enp1s17f5 down" | 17:08 |
| dansmith | mriedem: right you see it fail once first, and then once computes are started, it finds the version | 17:08 |
| dansmith | mriedem: before that "all cells" patch, you saw the failure on start and it never finds them after that | 17:08 |
| mriedem | ah yeah http://logs.openstack.org/84/623284/4/check/tempest-full/23568c2/controller/logs/screen-n-api.txt.gz#_Dec_11_20_56_12_002867 | 17:09 |
| mriedem | http://logs.openstack.org/84/623284/4/check/tempest-full/23568c2/controller/logs/screen-n-api.txt.gz#_Dec_11_21_00_06_274322 | 17:09 |
| dansmith | yep | 17:09 |
| mriedem | huh wonder if i made devstack switch upgrade_levels=auto in stein or rocky... | 17:09 |
| dansmith | it's been a while I thought, so I'm surprised this didn't break us | 17:10 |
| dansmith | but we also haven't had many rpc versions recently | 17:10 |
| mriedem | https://review.openstack.org/#/c/550108/ | 17:10 |
| mriedem | we went to auto upgrade levels in rocky | 17:10 |
| dansmith | well, we'll definitely want to backport this | 17:11 |
| mriedem | how far back do you think we need this to go? | 17:11 |
| mriedem | pike? | 17:11 |
| dansmith | pike was when you could actually have multiple cells right? so yeah, there | 17:11 |
| mriedem | i guess no one is multicell in pike right now, that we know of | 17:11 |
| mriedem | but yeah | 17:11 |
| mriedem | cern would be the only one to have hit this | 17:11 |
| dansmith | that we know of | 17:12 |
| mriedem | right | 17:12 |
| mriedem | could be chinese telecoms that have backported this to havana already | 17:12 |
| dansmith | melwitt: ready to approve this? https://review.openstack.org/#/c/623284/4 | 17:12 |
| sean-k-mooney | adrianc: actully this time the vf still had the vm mac for some reason if i changed it with "sudo ip link set enp1s0f1 vf 6 mac 1e:d5:87:7c:6f:b1" i can ping | 17:12 |
| melwitt | dansmith: yeah, hang on | 17:13 |
| dansmith | mriedem: why was that thing @synchronized in the first place? | 17:15 |
| dansmith | changing that makes me more nervous than the mechanical lazy-loading | 17:15 |
| mriedem | dansmith: (1) yes i said the same, i.e. i didn't really want to munge that into the lazy-load patch i had, and (2) looking at the change that added the lock, it was added as a precaution - i linked to that from the commit message | 17:16 |
| dansmith | ah sorry | 17:16 |
| dansmith | so you agree it's concerning but changed it anyway? | 17:17 |
| mriedem | jay, chris and eric seemed to think it was fine to remove the lock | 17:17 |
| mriedem | i didn't really want to change it within the same change that does the lazy-load, | 17:17 |
| melwitt | I wondered the same. last efried had mentioned about it, I thought he had said the lock was needed. but didn't get into details | 17:17 |
| mriedem | but jay -1ed me over it | 17:17 |
| melwitt | but maybe I missed convo after that | 17:17 |
| mriedem | i don't care if we remove the lock or not really | 17:17 |
| mriedem | i didn't really want to do squash both things into the same patch, but again, the -1 | 17:18 |
| mriedem | honestly i just wanted to move forward with it, | 17:18 |
| mriedem | but now jay and eric are gone | 17:18 |
| dansmith | I guess I'd just rather remove it but in a separate patch to avoid co-mingling the effects of these two changes, if we have to revert one, etc | 17:18 |
| dansmith | but I'm probably over-thinking it | 17:18 |
| mriedem | i thought the same | 17:19 |
| dansmith | heh, your logs are flooded with the rpc thing, while looking for flooding with placement messages | 17:19 |
| dansmith | getting all these committed would be, like, good | 17:19 |
| mriedem | yes, i'm spinning several gate plates right now | 17:19 |
| mriedem | thought about sending a summary to the ML | 17:20 |
| melwitt | I'd want them separate for the potential revert reason too but since you've already been through the wringer on it, so we could just go ahead and do a partial revert if we have to for the lock thing *shrug* | 17:20 |
| *** psachin|session has quit IRC | 17:20 | |
| mriedem | i can also just remove the lock part of this if dansmith and melwitt agree to approve the lazy-load today | 17:20 |
| mriedem | and throw the lock remove at the end | 17:20 |
| mriedem | i will knife someone if i have to redo this again and then wait | 17:20 |
| dansmith | I was about to +2, so I'd definitely +2 two patches instead yeah | 17:20 |
| dansmith | but whatever | 17:20 |
| melwitt | I'm fine with removing the lock since it seems like everyone who knows about it said to just do it. just we're aware there might be fallout and have to revert that piece | 17:21 |
| mriedem | i shall redo it then | 17:21 |
| sean-k-mooney | adrianc: http://paste.openstack.org/show/737154/ seting the vf mac via the pf fixes the issue | 17:21 |
| dansmith | mriedem: I will stand by for immediate +2age | 17:21 |
| * melwitt looks at it now | 17:22 | |
| *** sridharg has quit IRC | 17:26 | |
| *** dtantsur is now known as dtantsur|afk | 17:26 | |
| *** ttsiouts has quit IRC | 17:33 | |
| *** ttsiouts has joined #openstack-nova | 17:34 | |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: Only construct SchedulerReportClient on first access from API https://review.openstack.org/623246 | 17:35 |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: DRY up SchedulerReportClient init https://review.openstack.org/623247 | 17:35 |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: Remove lock on SchedulerReportClient._create_client https://review.openstack.org/624770 | 17:35 |
| mriedem | dansmith: melwitt: ^ | 17:35 |
| dansmith | you know, thinking about this.. maybe they should be one patch. | 17:35 |
| openstackgerrit | Stephen Finucane proposed openstack/nova master: Fail to live migration if instance has a NUMA topology https://review.openstack.org/611088 | 17:35 |
| mriedem | i hope you're just fucking with me | 17:36 |
| dansmith | well, hear me out here | 17:36 |
| *** k_mouza_ has joined #openstack-nova | 17:37 | |
| * mriedem sharpens the blade | 17:37 | |
| dansmith | heh | 17:37 |
| *** ttsiouts has quit IRC | 17:38 | |
| mriedem | dansmith: remember looking at this? https://bugs.launchpad.net/nova/+bug/1799298 | 17:38 |
| openstack | Launchpad bug 1799298 in OpenStack Compute (nova) rocky "Metadata API cross joining instance_metadata and instance_system_metadata" [Medium,Triaged] | 17:38 |
| mriedem | did anything come of that? | 17:38 |
| *** sahid has quit IRC | 17:38 | |
| dansmith | the thing that came out of it is I'm not sure it's worth the pain, but I did start on it | 17:39 |
| mriedem | well, as a simple partial fix, we could just stop joining on system_metadata | 17:39 |
| mriedem | since as far as i know that's only used if you've got a dynamic vendordata provider | 17:39 |
| *** spatel has joined #openstack-nova | 17:40 | |
| spatel | sean-k-mooney: Hey! | 17:40 |
| dansmith | yeah | 17:40 |
| mriedem | we likely don't need to join on security_groups either... | 17:40 |
| *** k_mouza has quit IRC | 17:41 | |
| spatel | I have question, i have VLAN 200 in openstack with /24 public IP | 17:41 |
| *** k_mouza_ has quit IRC | 17:41 | |
| sean-k-mooney | sure ok | 17:41 |
| spatel | that /24 about to get full, is it possible i can add new /24 pool in same VLAN 200? | 17:41 |
| sean-k-mooney | yes neutron allows you to add multiple subnets to the same network | 17:41 |
| sean-k-mooney | you will need to add the second subnet to the router also | 17:42 |
| *** panda is now known as panda|off | 17:43 | |
| spatel | I have VLAN base networking not tenant base.. | 17:43 |
| spatel | This is the command i used to create my first subnet.. | 17:43 |
| spatel | neutron subnet-create net_pub_vlan_200 71.xxx.xxx.0/24 --name sub_pub_vlan_200 --allocation-pool start=71.xxx.xxx.2,end=71.xxx.xxx.254 --dns-nameservers 8.8.8.8 --gateway=71.xxx.xxx.1 | 17:43 |
| sean-k-mooney | spatel: we should peroably swap to the neutron channel but that is fine | 17:44 |
| sean-k-mooney | vlan networks are still teant networks the terminology is a littel overloaded | 17:45 |
| spatel | You are saying i add just add new public IP in same "net_pub_vlan_200" network? | 17:45 |
| sean-k-mooney | yes you can | 17:45 |
| spatel | Something like this? | 17:46 |
| spatel | neutron subnet-create net_pub_vlan_200 72.xxx.xxx.0/24 --name new_subnet_vlan_200 --allocation-pool start=72.xxx.xxx.2,end=72.xxx.xxx.254 --dns-nameservers 8.8.8.8 --gateway=72.xxx.xxx.1 | 17:46 |
| sean-k-mooney | yep | 17:46 |
| spatel | net_pub_vlan_200 will be same right? | 17:46 |
| spatel | perfect!! thanks for confirmation :) | 17:47 |
| sean-k-mooney | the only other thing you need to do is add a new prot to the openstack or phyical router to connect to the new subnet | 17:47 |
| spatel | new port? | 17:48 |
| sean-k-mooney | if you are using a phyical provider router you need to asing 72.xxx.xxx.1 to it so it can act as a gatway but if you are using neutron routers then you just need to add a port on the neutron router to connet to n 72 subnet | 17:49 |
| spatel | oh! yes i am running HSRP so yes.. i have to setup HSRP on my gateway switches :) | 17:50 |
| sean-k-mooney | spatel: you can use the openstack router add subnet <route> 72.xxx.xxx.2 | 17:50 |
| sean-k-mooney | ah ok then ya you are using phyical routers instead of neutron routers for you public network so you can skip the openstack router add subnet command | 17:52 |
| *** rodolof has quit IRC | 17:56 | |
| spatel | yes!! | 17:59 |
| spatel | sean-k-mooney: quick question, now i am planning to build new Openstack cloud with NATing style floating ip, not VLAN base | 18:00 |
| spatel | how do i scale that network? we have super high PPS rate in network... does virtual router can handle that traffic? | 18:00 |
| spatel | I have option for DVR | 18:00 |
| sean-k-mooney | spatel: lets take the converstation to #openstack-neutron | 18:01 |
| *** _alastor_ has joined #openstack-nova | 18:03 | |
| *** derekh has quit IRC | 18:04 | |
| spatel | sean-k-mooney: thanks :) | 18:06 |
| mriedem | dansmith: can you think of anything besides flavor that we would have data migrated from system_metadata in the meta-api? i was thinking keypairs but that was different i think - it was a separate table and the data was migrated to extras | 18:07 |
| dansmith | yeah, not keypairs | 18:08 |
| dansmith | nothing off the top of my head, but I haven't thought about it much | 18:08 |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: Be smarter about DB joins in the metadata API https://review.openstack.org/624778 | 18:18 |
| mriedem | easy peasy ^ | 18:18 |
| *** mriedem is now known as mriedem_lunch | 18:21 | |
| *** ralonsoh has quit IRC | 18:25 | |
| *** cdent has quit IRC | 18:31 | |
| *** Swami has joined #openstack-nova | 18:44 | |
| *** moshele has quit IRC | 18:44 | |
| *** wolverineav has joined #openstack-nova | 18:46 | |
| *** wolverineav has quit IRC | 18:46 | |
| *** wolverineav has joined #openstack-nova | 18:47 | |
| openstackgerrit | Chris Dent proposed openstack/nova master: Use external placement in functional tests https://review.openstack.org/617941 | 18:47 |
| openstackgerrit | Chris Dent proposed openstack/nova master: Delete the placement code https://review.openstack.org/618215 | 18:50 |
| *** tssurya has quit IRC | 18:51 | |
| *** wolverineav has quit IRC | 18:57 | |
| *** wolverineav has joined #openstack-nova | 19:03 | |
| *** spatel has quit IRC | 19:07 | |
| *** wolverineav has quit IRC | 19:07 | |
| *** wolverineav has joined #openstack-nova | 19:09 | |
| *** mriedem_lunch is now known as mriedem | 19:11 | |
| *** tbachman has quit IRC | 19:23 | |
| *** erlon has quit IRC | 19:33 | |
| *** tbachman has joined #openstack-nova | 19:35 | |
| *** rodolof has joined #openstack-nova | 19:56 | |
| *** wolverineav has quit IRC | 20:05 | |
| *** kmalloc is now known as notmorgan | 20:10 | |
| *** notmorgan is now known as morgan | 20:10 | |
| *** itlinux_ has joined #openstack-nova | 20:14 | |
| *** itlinux has quit IRC | 20:15 | |
| *** tbachman has quit IRC | 20:23 | |
| *** _alastor_ has quit IRC | 20:32 | |
| *** dklyle has joined #openstack-nova | 20:39 | |
| *** wolverineav has joined #openstack-nova | 20:44 | |
| *** wolverineav has quit IRC | 20:52 | |
| *** wolverineav has joined #openstack-nova | 20:55 | |
| *** rcernin has joined #openstack-nova | 21:06 | |
| openstackgerrit | Matt Riedemann proposed openstack/nova stable/rocky: Handle tags in _bury_in_cell0 https://review.openstack.org/624813 | 21:08 |
| *** jmlowe has quit IRC | 21:10 | |
| *** lchabert2 has joined #openstack-nova | 21:11 | |
| *** jmlowe has joined #openstack-nova | 21:11 | |
| *** jmlowe has quit IRC | 21:13 | |
| *** jmlowe has joined #openstack-nova | 21:14 | |
| *** xek__ has quit IRC | 21:18 | |
| *** tbachman has joined #openstack-nova | 21:19 | |
| *** lchabert2 has quit IRC | 21:22 | |
| *** lchabert has joined #openstack-nova | 21:22 | |
| *** rodolof has quit IRC | 21:32 | |
| *** rodolof has joined #openstack-nova | 21:32 | |
| *** awaugama has quit IRC | 21:40 | |
| *** brandor5 has joined #openstack-nova | 21:43 | |
| brandor5 | hello everyone: can anyone tell me where I can find the newton nova api reference? | 21:43 |
| *** lpetrut has joined #openstack-nova | 21:44 | |
| *** _alastor_ has joined #openstack-nova | 21:46 | |
| jonher | brandor5: this? https://developer.openstack.org/api-ref/compute/ | 21:48 |
| brandor5 | jonher: that's latest, I need the one for newton | 21:48 |
| brandor5 | I'm trying to use the api to confirmResize after a migrate and I'm getting an error | 21:49 |
| brandor5 | Cannot 'confirmResize' instance e1782f5c-7559-4cac-a628-6d4a75a015f4 while it is in vm_state stopped | 21:49 |
| *** slaweq has quit IRC | 21:49 | |
| brandor5 | If i run `nova resize-confirm <instance_id>` it works and I'm running the same curl that it runs... | 21:50 |
| mriedem | brandor5: the api-ref on master should be the same for newton - where there are functional differences they should be called out with notes or microversions | 21:51 |
| mriedem | i.e. the api should be backward compatible | 21:51 |
| brandor5 | mriedem: oh cool, thanks... any idea on the error I'm seeing? | 21:51 |
| brandor5 | especially that it works when I use the command line client | 21:51 |
| * mriedem pulls out the ledger | 21:51 | |
| mriedem | can't confirm a resize on a stopped vm in newton https://github.com/openstack/nova/blob/newton-eol/nova/compute/api.py#L2924 | 21:52 |
| mriedem | nor master https://github.com/openstack/nova/blob/master/nova/compute/api.py#L3489 | 21:52 |
| brandor5 | hmm that's weird because `nova resize-confirm <instance id>` definitely works | 21:53 |
| mriedem | works works? or just doesn't say it failed? | 21:53 |
| brandor5 | works works | 21:53 |
| mriedem | which version of novaclient? | 21:53 |
| brandor5 | 10.1.0 | 21:53 |
| brandor5 | and when I run nova in debug I've verified that I'm running the same 'curl' that novaclient is running | 21:54 |
| *** takashin has joined #openstack-nova | 21:54 | |
| mriedem | no magic there https://github.com/openstack/python-novaclient/blob/10.1.0/novaclient/v2/servers.py#L1533 | 21:55 |
| mriedem | can you show the server status before doing each (cli and curl) to verify? | 21:55 |
| brandor5 | yeah, I'm doing that... I have a watch running `openstack server show <instance_id>` | 21:56 |
| brandor5 | if I try to hit the api directly I get a 409 | 21:56 |
| mriedem | so you have some script or something that's just auto-confirming a resize as soon as the instance status is VERIFY_RESIZE? | 21:56 |
| brandor5 | if I use novaclient it works | 21:56 |
| brandor5 | correct | 21:56 |
| mriedem | there is a config option for auto-confirming migrations | 21:56 |
| mriedem | .. | 21:56 |
| brandor5 | yeah, that's across the board and the docs say it's not recommended... my script is just for putting a compute into 'maintenance' mode | 21:57 |
| brandor5 | It's actually ansible playbook that sets maintenance mode, checks for errored instances, manually moves shutoff instances, and then blasts the active instances' | 21:58 |
| brandor5 | unfortunately right now I can't use the command line clients because ansible tower doesn't have them installed... (that's a ticket i have open with redhat) | 21:59 |
| mriedem | gd red hat | 21:59 |
| mriedem | i can't see anything that would be a difference between the client and straight curl | 21:59 |
| brandor5 | haha | 21:59 |
| brandor5 | i'll keep poking then... double check my headers and whatnot | 21:59 |
| brandor5 | thanks for the help | 21:59 |
| *** wwriverrat_ has joined #openstack-nova | 21:59 | |
| mriedem | are you polling on the 'status' field in the server show response or something like vm_state? | 22:00 |
| *** itlinux_ has quit IRC | 22:00 | |
| *** dklyle has quit IRC | 22:00 | |
| *** itlinux has joined #openstack-nova | 22:00 | |
| brandor5 | https://pastebin.com/ijVe0J42 | 22:01 |
| melwitt | mriedem: what's the trick to bfv with more than one --block-device? I keep getting ERROR state saying 'no image supplied' even though the first one I give is bootindex=0 with source=image | 22:01 |
| mriedem | brandor5: that looks ok | 22:02 |
| mriedem | melwitt: are you trying to attach data volumes or have nova create the volumes? | 22:02 |
| *** mdbooth_ has joined #openstack-nova | 22:02 | |
| mriedem | it's probably easiest to just boot and attach 2 data volumes (so not boot_index=0) | 22:02 |
| melwitt | mriedem: I'm consistently trying to have nova create the boot volume, and attach a non-bootable volume I created in cinder ahead | 22:03 |
| mriedem | i.e. pre-existing volumes | 22:03 |
| mriedem | what is your source_type and destination_type? | 22:03 |
| melwitt | this was my latest attempt http://paste.openstack.org/show/737169/ | 22:04 |
| mriedem | --block-device source=image,id=d7b0d7cb-1373-456d-bcec-29fbec2f84e1,dest=volume,size=1,shutdown=preserve,bootindex=0 | 22:04 |
| mriedem | is that id an image id? | 22:04 |
| melwitt | yes | 22:04 |
| mriedem | --block-device source=volume,id=33550efe-6a7b-4956-a7e0-5bd50004ec30,dest=volume,size=1,shutdown=preserve | 22:04 |
| mriedem | and that id is a volume id? | 22:04 |
| melwitt | and that one is a volume id | 22:04 |
| mriedem | you don't need to specify size on the latter, but nova should ignore that | 22:05 |
| melwitt | I think it can be done but I don't know how. I think I've seen launchpad bug reports where people were booting with multiple volumes | 22:05 |
| *** mdbooth has quit IRC | 22:05 | |
| *** slaweq has joined #openstack-nova | 22:06 | |
| mriedem | where are you seeing the "no image supplied"? | 22:06 |
| melwitt | in the nova show fault field | 22:06 |
| mriedem | http://codesearch.openstack.org/?q=no%20image%20supplied&i=nope&files=&repos= | 22:06 |
| mriedem | that's not in the code that i can see | 22:06 |
| mriedem | besides the show response | 22:06 |
| mriedem | which is normal | 22:06 |
| mriedem | paste your fault | 22:06 |
| melwitt | http://paste.openstack.org/show/737170/ | 22:07 |
| melwitt | oh sorry | 22:07 |
| melwitt | dammmit | 22:07 |
| melwitt | that was the image field | 22:08 |
| Kevin_Zheng | mriedem: just shelve the vm and all the volume will be in reserved status and cannot be detached | 22:08 |
| mriedem | yeah that's normal | 22:08 |
| openstackgerrit | Matt Riedemann proposed openstack/nova stable/rocky: Handle tags in _bury_in_cell0 https://review.openstack.org/624813 | 22:08 |
| mriedem | b/c there is no image ref if you bfv | 22:08 |
| mriedem | Kevin_Zheng: we have a tempest test that does exactly that | 22:08 |
| melwitt | yeah, I am so rusty on this. that's embarrassing | 22:09 |
| Kevin_Zheng | I didn’t check yet but that’s the result I got | 22:09 |
| mriedem | good thing i don't have a twitter account to let everyone know about this | 22:09 |
| Kevin_Zheng | And also the code does it | 22:09 |
| melwitt | I was thinking that sentence was the fault field, but the fault field wasn't even showing because I wasn't authed as admin. lots of good stuff going on here | 22:09 |
| melwitt | haha yup | 22:10 |
| *** slaweq has quit IRC | 22:10 | |
| mriedem | Kevin_Zheng: is this vanilla devstack or something with your detach boot volume changes applied? | 22:10 |
| melwitt | failure to prep block device, which is probably my conf limit code firing | 22:10 |
| mriedem | melwitt: yes probably, and we don't reschedule on failures from _prep_block_device i don't think | 22:11 |
| mriedem | so we just crap out | 22:11 |
| brandor5 | mriedem: i found the problem, the resize call was firing too fast... I put a "sleep" in for a few seconds and it worked | 22:11 |
| Kevin_Zheng | DevStack | 22:11 |
| mriedem | brandor5: hmm, that's weird, because as soon as the server status is VERIFY_RESIZE is should be ok to call confirmResize | 22:12 |
| mriedem | especially because i'm sure tempest does exactly the same thing, | 22:12 |
| mriedem | poll for VERIFY_RESIZE and then immediately confirm | 22:12 |
| brandor5 | yeah, that's what I'm doing | 22:13 |
| brandor5 | if I sleep for 3 seconds after that it works | 22:13 |
| Kevin_Zheng | I did not change the shelve flow, I only change the detach attach flow and it will not affect as it was blocked in the very early stage calling begin-detaching | 22:13 |
| *** sambetts_ has quit IRC | 22:13 | |
| brandor5 | unless my polling isn't working the way I think | 22:13 |
| brandor5 | I'll double check that really quick | 22:14 |
| mriedem | this is where nova-compute changes the status https://github.com/openstack/nova/blob/newton-eol/nova/compute/manager.py#L3943 | 22:14 |
| mriedem | this is the api code for the status field https://github.com/openstack/nova/blob/newton-eol/nova/api/openstack/common.py#L79 | 22:15 |
| mriedem | so once compute does the update to the db, and the api returns VERIFY_RESIZE is should be ok to confirm | 22:15 |
| *** sambetts_ has joined #openstack-nova | 22:15 | |
| mriedem | Kevin_Zheng: did you look at the tempest test i pointed out? | 22:15 |
| *** slaweq has joined #openstack-nova | 22:15 | |
| Kevin_Zheng | Not yet, I will check it today | 22:16 |
| mriedem | good morning btw | 22:16 |
| mriedem | 6am what are you doing | 22:17 |
| Kevin_Zheng | My kid fall down the bed so everyone is up:) | 22:17 |
| mriedem | ha | 22:18 |
| mriedem | time for guard rails | 22:18 |
| mriedem | or strapts | 22:18 |
| mriedem | *straps | 22:18 |
| brandor5 | mriedem: found the problem... it was an error on my part... if you look at the paste I linked the register,until, retries, and delay options were indented incorrectly and not being honored....... so it was just running the cehck once and then moving on through | 22:18 |
| mriedem | brandor5: ah ok | 22:19 |
| mriedem | i knew it couldn't be nova's fault, nova is never at fault | 22:19 |
| brandor5 | hehe | 22:19 |
| Kevin_Zheng | :) go back to sleep now | 22:19 |
| brandor5 | it's dns | 22:19 |
| mriedem | it probably worked in the cli just b/c of a time delay | 22:19 |
| brandor5 | or the network | 22:19 |
| mriedem | building the client and such | 22:19 |
| mriedem | or that | 22:19 |
| brandor5 | those are my favorits :D | 22:19 |
| *** slaweq has quit IRC | 22:20 | |
| openstackgerrit | Matt Riedemann proposed openstack/nova stable/queens: Handle tags in _bury_in_cell0 https://review.openstack.org/624824 | 22:21 |
| melwitt | confirmed, it was my limit code. so it does work for bfv. and it didn't create a new volume before it failed. I will test it more, maybe it's too good to be true | 22:22 |
| mriedem | we should get that fault message cleaned up to actually have the error message in it | 22:22 |
| melwitt | aye | 22:23 |
| *** mnasiadka_ has joined #openstack-nova | 22:23 | |
| *** s1061123 has joined #openstack-nova | 22:23 | |
| mriedem | could have sworn i did something in that code for that, but might have been logging | 22:23 |
| *** awestin1_ has joined #openstack-nova | 22:23 | |
| mriedem | yeah https://review.openstack.org/#/q/I20a5e8e5e10dd505c1b24c208f919c6550e9d1a4 | 22:24 |
| mriedem | maybe "Failure prepping block device" is just your error message | 22:25 |
| *** obre_ has joined #openstack-nova | 22:25 | |
| melwitt | indeed in the log: Failure prepping block device: TooManyDiskDevices: The maximum allowed number of disk devices to attach to a single instance has bee | 22:25 |
| melwitt | n exceeded | 22:25 |
| *** _alastor_ has quit IRC | 22:26 | |
| melwitt | le sigh, newline | 22:26 |
| *** mnasiadka has quit IRC | 22:29 | |
| *** andreykurilin has quit IRC | 22:29 | |
| *** masayukig[m] has quit IRC | 22:29 | |
| *** obre has quit IRC | 22:29 | |
| *** s1061123_ has quit IRC | 22:29 | |
| *** awestin1 has quit IRC | 22:29 | |
| *** mlavalle has quit IRC | 22:29 | |
| *** smcginnis has quit IRC | 22:29 | |
| *** McNinja has quit IRC | 22:29 | |
| *** logan- has quit IRC | 22:29 | |
| *** awestin1_ is now known as awestin1 | 22:29 | |
| *** mnasiadka_ is now known as mnasiadka | 22:29 | |
| *** mlavalle has joined #openstack-nova | 22:30 | |
| *** wolverineav has quit IRC | 22:31 | |
| *** logan- has joined #openstack-nova | 22:31 | |
| *** trident has quit IRC | 22:32 | |
| *** gouthamr has quit IRC | 22:32 | |
| *** wolverineav has joined #openstack-nova | 22:32 | |
| *** lpetrut has quit IRC | 22:32 | |
| *** trident has joined #openstack-nova | 22:34 | |
| *** itlinux has quit IRC | 22:35 | |
| *** gouthamr has joined #openstack-nova | 22:35 | |
| openstackgerrit | Matt Riedemann proposed openstack/nova stable/pike: Handle tags in _bury_in_cell0 https://review.openstack.org/624825 | 22:36 |
| *** brandor5 has left #openstack-nova | 22:37 | |
| *** slaweq has joined #openstack-nova | 22:42 | |
| *** slaweq has quit IRC | 22:47 | |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: Replace ThreadPoolExecutor with GreenThreadPoolExecutor https://review.openstack.org/624740 | 22:48 |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: Add python 3.7 unit and functional tox jobs https://review.openstack.org/624055 | 22:48 |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: Add py36/py37 functional jobs to the experimental queue https://review.openstack.org/624826 | 22:48 |
| melwitt | found that the error message gets squashed here https://github.com/openstack/nova/blob/master/nova/compute/manager.py#L2341-L2343 and that's what ends up in the instance fault | 22:48 |
| melwitt | feels like a separate patch for improving the information there, I can try it and see how it looks, going to be a much longer string | 22:49 |
| *** mchlumsky has quit IRC | 22:57 | |
| *** smcginnis has joined #openstack-nova | 23:08 | |
| *** dtrainor has quit IRC | 23:09 | |
| *** mlavalle has quit IRC | 23:13 | |
| openstackgerrit | Matt Riedemann proposed openstack/os-resource-classes master: Sync STANDARDS with placement/nova rc_fields https://review.openstack.org/624370 | 23:15 |
| openstackgerrit | Matt Riedemann proposed openstack/os-resource-classes master: Tune up documentation to be more useful https://review.openstack.org/624384 | 23:15 |
| *** dave-mccowan has quit IRC | 23:19 | |
| *** rodolof has quit IRC | 23:21 | |
| mriedem | melwitt: oh ok, we should just do similar in https://github.com/openstack/nova/blob/master/nova/compute/manager.py#L2341 then and probably include the underlying reason | 23:21 |
| mriedem | also, duh duh duh nova-lvm job is busted (privsep) http://logs.openstack.org/43/619143/12/check/nova-lvm/786180c/logs/screen-n-cpu.txt.gz?level=TRACE#_Dec_12_12_35_39_607002 | 23:21 |
| *** rodolof has joined #openstack-nova | 23:21 | |
| melwitt | mriedem: yeah, doing it now. figured I should write up a bug too | 23:22 |
| mriedem | https://bugs.launchpad.net/nova/+bug/1808247 | 23:23 |
| openstack | Launchpad bug 1808247 in OpenStack Compute (nova) "nova-lvm job failing with ArgsAlreadyParsedError (privsep)" [Medium,Confirmed] | 23:23 |
| mriedem | if only mikal were on irc | 23:24 |
| openstackgerrit | Rodolfo Alonso Hernandez proposed openstack/os-vif master: Create iproute.IPRoute() inside a context https://review.openstack.org/624831 | 23:28 |
| openstackgerrit | Merged openstack/nova master: Make compute rpcapi version calculation check all cells https://review.openstack.org/623284 | 23:29 |
| openstackgerrit | Merged openstack/nova master: Remove allocations before setting vm_status to SHELVED_OFFLOADED https://review.openstack.org/623596 | 23:29 |
| openstackgerrit | melanie witt proposed openstack/nova master: Propagate exception message from _prep_block_device https://review.openstack.org/624832 | 23:31 |
| *** dtrainor has joined #openstack-nova | 23:31 | |
| melwitt | ah, I guess I need a test to show it going to the fault message. or maybe just that the message from _prep_block_device gets included | 23:33 |
| *** rodolof has quit IRC | 23:35 | |
| *** rodolof has joined #openstack-nova | 23:35 | |
| *** dklyle has joined #openstack-nova | 23:35 | |
| *** rodolof has quit IRC | 23:37 | |
| *** rodolof has joined #openstack-nova | 23:38 | |
| openstackgerrit | Rodolfo Alonso Hernandez proposed openstack/os-vif master: Create iproute.IPRoute() inside a context https://review.openstack.org/624831 | 23:41 |
| *** dklyle has quit IRC | 23:45 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!