imacdonn | it doesn't exist until libvirt creates the domain | 00:00 |
---|---|---|
*** ssurana has joined #openstack-nova | 00:01 | |
imacdonn | https://github.com/openstack/nova/blob/master/nova/virt/libvirt/driver.py#L5079 | 00:02 |
mriedem | do you see Converting VIF in the nova-compute logs at debug level? | 00:02 |
imacdonn | note that plug_vifs() is called before _create_domain() | 00:03 |
mriedem | right that's where we start with os-vif | 00:03 |
mriedem | yeah | 00:03 |
mriedem | kevinbenton: are you around? | 00:03 |
kevinbenton | mriedem: AHOy! | 00:04 |
mriedem | kevinbenton: ^ whats set the MTU on the tap interface? | 00:04 |
mriedem | for vif_type == bridge? | 00:04 |
*** lyan has quit IRC | 00:04 | |
mriedem | i thought it was os-vif | 00:05 |
kevinbenton | mriedem: no | 00:05 |
kevinbenton | mriedem: it's either libvirt internals or it inherits from bridge as part of linux | 00:05 |
mriedem | or does neutron do it once we plug the vif and neutron agent picks that up? | 00:05 |
mriedem | hmm | 00:05 |
kevinbenton | mriedem: this is sort of why i wanted to move away from libvirt plugging things into the bridge | 00:06 |
imacdonn | inheriting from the bridge doesn't work, because the bridge is initially created with MTU 1500, and it won't go any higher until a interface is added .. then it assumes the lowest MTU amongst all member interfaces | 00:06 |
kevinbenton | imacdonn: hang on | 00:06 |
imacdonn | k | 00:06 |
kevinbenton | MTU is set on the bridge by os-vif | 00:06 |
mriedem | i thought it was this https://github.com/openstack/os-vif/blob/master/vif_plug_linux_bridge/linux_net.py#L91 | 00:07 |
imacdonn | it can't be | 00:07 |
imacdonn | or .. maybe I have a bug in the OS | 00:07 |
*** rfolco_ has quit IRC | 00:07 | |
imacdonn | # brctl addbr test-bridge | 00:08 |
imacdonn | # ip l set test-bridge mtu 9000 | 00:08 |
imacdonn | RTNETLINK answers: Invalid argument | 00:08 |
kevinbenton | oh hang on | 00:09 |
kevinbenton | maybe not | 00:09 |
kevinbenton | i think that branch in os-vif is not exercised with neutron | 00:09 |
kevinbenton | because https://github.com/openstack/os-vif/blob/master/vif_plug_linux_bridge/linux_bridge.py#L98 is not true with neutron | 00:10 |
imacdonn | yeah, I only have should_provide_bridge | 00:10 |
mriedem | ok, well i guess that's why then | 00:15 |
*** ijw has joined #openstack-nova | 00:15 | |
kevinbenton | https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1399064 | 00:16 |
openstack | Launchpad bug 1399064 in linux (Ubuntu) "bridges cannot have a mtu > 1500 by themselves" [Medium,Triaged] | 00:16 |
kevinbenton | so i wonder if we have a race condition here | 00:16 |
kevinbenton | where nova adding a tap interface to the bridge results in 1500 mtu | 00:16 |
kevinbenton | since it's first | 00:16 |
kevinbenton | and then neutron adds vxlan interface | 00:17 |
kevinbenton | with real network mtu | 00:17 |
imacdonn | yeah, so then when I destroy nova instance, and the tap interface goes away, the bridge inherits the MTU of the VLAN interface, which is 9000 (in my case) | 00:18 |
imacdonn | then I launch another nova instance, and its tap interface picks up the bridge's MTU of 9000, and it works | 00:18 |
kevinbenton | ack | 00:18 |
kevinbenton | that sounds exactly like the case then | 00:18 |
imacdonn | it's not really a "race condition" in the multi-threading sense | 00:18 |
kevinbenton | well it is in the sense that the neutron agent could conceivably beat nova | 00:19 |
kevinbenton | after libvirt creates bridge | 00:19 |
imacdonn | even if it was possible to set the MTU on an empty bridge, there's still nothign in the code that will set the bridge's MTU | 00:19 |
kevinbenton | but before it adds interface | 00:19 |
kevinbenton | neutron could wire up the vlan/vxlan interface | 00:19 |
imacdonn | oh, you;re saying that if neutron added the VLAN interface before libvirt created the domain, it'd work ? | 00:20 |
kevinbenton | right | 00:20 |
imacdonn | that can't happen, because the neutron l2 agent triggers on the tap interface's existence | 00:20 |
kevinbenton | oh duh | 00:21 |
imacdonn | well, I guess it could happen if there's a window where the tap interface exists but hasn't been added the bridge yet | 00:21 |
kevinbenton | yeah | 00:21 |
kevinbenton | but the reason we don't see this in the gate is probably because of the dhcp agents, etc | 00:21 |
imacdonn | BTW, this was solved in the neutron agent in the past | 00:21 |
imacdonn | but then it got un-solved by https://git.openstack.org/cgit/openstack/neutron/commit/?id=d352661c56d5f03713e615b7e0c2c9c8688e0132 | 00:22 |
*** brault has quit IRC | 00:22 | |
kevinbenton | how would that solve it? | 00:22 |
imacdonn | the previous solution does seem a little kludgey | 00:22 |
*** gjayavelu has quit IRC | 00:22 | |
kevinbenton | bridge MTU would also be too small | 00:22 |
imacdonn | it looks at the the MTU of the VLAN interface and forces it onto the tap interface | 00:22 |
*** edmondsw has joined #openstack-nova | 00:23 | |
kevinbenton | ah, i see | 00:23 |
*** ijw_ has joined #openstack-nova | 00:23 | |
kevinbenton | it's ignoring bridge mtu completely | 00:24 |
imacdonn | yeah, the bridge just inherits the lowest amongst all its members | 00:24 |
*** ijw has quit IRC | 00:25 | |
kevinbenton | yes, so this was result in packet loss every time a vm connected | 00:25 |
imacdonn | I wonder if libvirt provides any way to specify the MTU for a bridge interface | 00:26 |
imacdonn | seems that that's probably where it should be done ? | 00:26 |
clarkb | this is only a problem if trying to use larger MTUs right? | 00:26 |
clarkb | thats why we wouldnt see it in the gate | 00:26 |
imacdonn | yeah | 00:27 |
kevinbenton | well we also wouldn't see it in single node gate | 00:27 |
kevinbenton | because dhcp agent would get attached before VM | 00:27 |
kevinbenton | and DHCP agent plugs using correct MTU | 00:27 |
clarkb | kevinbenton: right but until we test with jumbo frames we would never see it regardless of multinode or single node | 00:27 |
clarkb | (and we have multinode testing) | 00:27 |
imacdonn | FWIW, the dhcp_agent's tap interface does get set to that of the network .. but that's obviously nothing to do with nova | 00:27 |
*** brault has joined #openstack-nova | 00:28 | |
*** edmondsw has quit IRC | 00:28 | |
kevinbenton | so yeah, there are two possible fixes i can see | 00:28 |
kevinbenton | tell libvirt about MTU if it supports that | 00:28 |
*** gcb has quit IRC | 00:29 | |
imacdonn | yeah, if | 00:29 |
kevinbenton | or proceed with https://review.openstack.org/#/c/447150/ | 00:29 |
kevinbenton | which will stop libvirt from plugging stuff into linux bridges | 00:29 |
kevinbenton | and then re-add something like https://git.openstack.org/cgit/openstack/neutron/commit/?id=d352661c56d5f03713e615b7e0c2c9c8688e0132 | 00:29 |
imacdonn | hmm ineresting | 00:29 |
kevinbenton | to set MTU before it gets added to the bridge | 00:29 |
imacdonn | interesting* | 00:29 |
openstackgerrit | Takashi NATSUME proposed openstack/nova master: Avoid forcing translation on logging calls https://review.openstack.org/413876 | 00:29 |
*** ngupta has joined #openstack-nova | 00:30 | |
*** Apoorva_ has joined #openstack-nova | 00:30 | |
*** gcb has joined #openstack-nova | 00:31 | |
imacdonn | interesting hackaround ... http://linuxaleph.blogspot.com/2013/01/how-to-network-jumbo-frames-to-kvm-guest.html | 00:31 |
kevinbenton | imacdonn: can you check if libvirt can take MTU as an option? | 00:33 |
imacdonn | Googling on that now | 00:33 |
kevinbenton | because the problem with my patch is that it requires a permissions change due to qemu sillyness | 00:34 |
kevinbenton | it requires more permissions to not create a bridge than to create one :) | 00:34 |
imacdonn | heh | 00:34 |
*** Apoorva has quit IRC | 00:34 | |
clarkb | kevinbenton: well qemu-bridge-helper is a thing, maybe you just make that nop? | 00:34 |
clarkb | (thought that might confuse people greatly) | 00:34 |
*** Apoorva_ has quit IRC | 00:35 | |
kevinbenton | clarkb: yeah, that also makes detection harder because i wouldn't be using a different vif_type on the neutron side | 00:35 |
kevinbenton | it would be hard to distinguish in the agent if a given nova port was going to try to plug itself into a bridge | 00:36 |
clarkb | gotcha | 00:36 |
clarkb | (you only need to attach to the bridge if on a network connected to something?) | 00:37 |
*** thorst has joined #openstack-nova | 00:37 | |
imacdonn | https://libvirt.org/formatnetwork.html <- trying to comprehend this .. I think it's saying that it only works for libvirt-managed networks | 00:37 |
imacdonn | "Network types not specifically mentioned here don't support having an MTU set in the libvirt network config." | 00:38 |
kevinbenton | clarkb: we attach ports to bridges that were created by agents. we leave attaching to bridge up to nova for compute ports | 00:38 |
kevinbenton | clarkb: so we skip based on device_owner 'compute' right now | 00:39 |
kevinbenton | clarkb: ideally i want to get rid of that, but for one cycle i will need to know about old compute ports trying to plug themselves in | 00:39 |
kevinbenton | clarkb: so with my current approach we just set the vif_type='tap' for any newly bound linux bridge ports | 00:40 |
kevinbenton | clarkb: and that will make it easy to tell | 00:40 |
clarkb | and is there a reason you can't set the mtu on the bridge when it is created (because you don't know what the mtu of the devices that attach to it later will be?) | 00:41 |
kevinbenton | clarkb: we could | 00:42 |
imacdonn | linux doesn't seem to allow that .. at least the kernel I'm running | 00:42 |
kevinbenton | clarkb: but that leads to the other bug | 00:42 |
*** thorst has quit IRC | 00:42 | |
kevinbenton | clarkb: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1399064 | 00:42 |
openstack | Launchpad bug 1399064 in linux (Ubuntu) "bridges cannot have a mtu > 1500 by themselves" [Medium,Triaged] | 00:42 |
clarkb | ah | 00:42 |
clarkb | we set them lower in the multinode tests when we used linux bridges and not ovs | 00:42 |
clarkb | good to know thanks | 00:42 |
*** karimb has quit IRC | 00:43 | |
kevinbenton | imacdonn: yeah, that doesn't sound promising for libvirt | 00:45 |
imacdonn | ack | 00:45 |
*** Jack_Iv has joined #openstack-nova | 00:45 | |
kevinbenton | imacdonn: i suppose we can put the logic back in the neutron agent | 00:46 |
kevinbenton | the tap device inherits from the bridge, right? | 00:46 |
imacdonn | it appears so, yes | 00:46 |
imacdonn | well... | 00:47 |
kevinbenton | so once the bridge is embiggened, we don't reduce by adding a tap device | 00:47 |
imacdonn | it picks up the bridge's MTU when it gets added to the bridge | 00:47 |
kevinbenton | :) | 00:47 |
imacdonn | if you change the bridge later, I don't think the tap will follow | 00:47 |
kevinbenton | yeah, that's fine | 00:47 |
kevinbenton | i'm just trying to make sure the bridge wouldn't shrink every time | 00:47 |
imacdonn | right | 00:47 |
kevinbenton | until the agent went through and corrected | 00:47 |
kevinbenton | ok, then we can fix this in neutron | 00:47 |
kevinbenton | imacdonn: if you file a bug i can revert that patch | 00:48 |
kevinbenton | well not revert | 00:48 |
kevinbenton | but put in logic to match to the network MTU | 00:48 |
imacdonn | hang on | 00:48 |
imacdonn | I might have been looking at the wrong libvirt doc | 00:49 |
imacdonn | https://libvirt.org/formatdomain.html#mtu | 00:49 |
imacdonn | this is fairly new .. I'm on libvirt 1.2.x | 00:49 |
*** jerrygb has joined #openstack-nova | 00:50 | |
*** Jack_Iv has quit IRC | 00:50 | |
imacdonn | well, a mix of 1.2.x and 2.0.x, but not 3.1 | 00:50 |
kevinbenton | i gtg for a bit. if it's not looking good for libvirt, just let me know | 00:52 |
kevinbenton | the logic should probably exist in linuxbridge agent anyway to deal with older ones | 00:52 |
imacdonn | I'll file the bug anyway .. think we at least need something to get us by until your other solution | 00:52 |
*** catintheroof has quit IRC | 00:52 | |
imacdonn | yeah, I think that requiring libvirt 3.1.x would be a problem for many | 00:53 |
imacdonn | thanks! I'll get the bug in | 00:53 |
*** tovin07_ has joined #openstack-nova | 00:54 | |
*** jerrygb has quit IRC | 00:56 | |
*** baoli has joined #openstack-nova | 00:56 | |
*** phuongnh has joined #openstack-nova | 01:02 | |
*** gyee has quit IRC | 01:03 | |
*** cNilesh has joined #openstack-nova | 01:05 | |
*** MasterOfBugs has quit IRC | 01:07 | |
imacdonn | https://bugs.launchpad.net/neutron/+bug/1684326 | 01:08 |
openstack | Launchpad bug 1684326 in neutron "MTU not set on nova instance's vif_type=bridge tap interface" [Undecided,New] | 01:08 |
*** thorst has joined #openstack-nova | 01:13 | |
*** NikhilS has joined #openstack-nova | 01:14 | |
*** jamesden_ has joined #openstack-nova | 01:17 | |
openstackgerrit | Matt Riedemann proposed openstack/nova master: Deprecate os-hosts API https://review.openstack.org/456504 | 01:19 |
*** yingwei has joined #openstack-nova | 01:20 | |
*** jamesden_ has quit IRC | 01:20 | |
*** mriedem has quit IRC | 01:22 | |
*** jamesden_ has joined #openstack-nova | 01:23 | |
*** Sukhdev has quit IRC | 01:25 | |
*** thorst has quit IRC | 01:25 | |
*** MasterOfBugs has joined #openstack-nova | 01:25 | |
*** stvnoyes has quit IRC | 01:26 | |
*** stvnoyes has joined #openstack-nova | 01:26 | |
*** rfolco has joined #openstack-nova | 01:31 | |
*** rfolco has quit IRC | 01:31 | |
*** jamesden_ has quit IRC | 01:31 | |
*** thorst has joined #openstack-nova | 01:33 | |
*** lyan has joined #openstack-nova | 01:35 | |
*** david-lyle has quit IRC | 01:35 | |
*** hongbin has joined #openstack-nova | 01:36 | |
*** thorst has quit IRC | 01:36 | |
*** yongjiexu has joined #openstack-nova | 01:39 | |
*** jerrygb has joined #openstack-nova | 01:43 | |
*** yingwei has quit IRC | 01:46 | |
*** stvnoyes has quit IRC | 01:46 | |
*** stvnoyes has joined #openstack-nova | 01:47 | |
*** zz_dimtruck is now known as dimtruck | 01:48 | |
openstackgerrit | Zhenyu Zheng proposed openstack/nova master: Support tag instances when boot https://review.openstack.org/394321 | 01:49 |
*** david-lyle has joined #openstack-nova | 01:50 | |
*** dave-mcc_ has joined #openstack-nova | 01:51 | |
*** annegentle has quit IRC | 01:51 | |
*** kevinz has joined #openstack-nova | 01:52 | |
*** dave-mccowan has quit IRC | 01:53 | |
*** baoli has quit IRC | 01:55 | |
*** baoli has joined #openstack-nova | 01:56 | |
*** amotoki has joined #openstack-nova | 02:02 | |
*** jaypipes has quit IRC | 02:05 | |
*** thorst has joined #openstack-nova | 02:07 | |
*** dave-mccowan has joined #openstack-nova | 02:10 | |
*** zhurong has joined #openstack-nova | 02:10 | |
*** yamahata has quit IRC | 02:11 | |
openstackgerrit | zhao mingjun proposed openstack/nova master: follow-up https://review.openstack.org/458298 | 02:11 |
*** Shunli has joined #openstack-nova | 02:11 | |
*** bkopilov has quit IRC | 02:13 | |
*** dave-mcc_ has quit IRC | 02:13 | |
*** korean101 has quit IRC | 02:17 | |
*** tbachman has quit IRC | 02:22 | |
*** thorst has quit IRC | 02:23 | |
*** ssurana has quit IRC | 02:30 | |
*** lyan has quit IRC | 02:31 | |
*** coreywright has quit IRC | 02:37 | |
*** ngupta has quit IRC | 02:38 | |
*** ngupta has joined #openstack-nova | 02:38 | |
*** gouthamr has joined #openstack-nova | 02:45 | |
*** coreywright has joined #openstack-nova | 02:50 | |
*** thorst has joined #openstack-nova | 02:54 | |
*** thorst has quit IRC | 02:54 | |
*** ngupta has quit IRC | 02:54 | |
*** jerrygb has quit IRC | 02:55 | |
*** gjayavelu has joined #openstack-nova | 02:59 | |
openstackgerrit | Huan Xie proposed openstack/nova master: WIP: XenAPI use os-xenapi v2 in nova https://review.openstack.org/453493 | 02:59 |
*** yongjiexu has quit IRC | 03:03 | |
*** gongysh has joined #openstack-nova | 03:03 | |
*** yingwei has joined #openstack-nova | 03:03 | |
openstackgerrit | Huan Xie proposed openstack/nova master: WIP: XenAPI use os-xenapi v2 in nova https://review.openstack.org/453493 | 03:06 |
*** Fdaisuke_ has joined #openstack-nova | 03:08 | |
*** Fdaisuke has quit IRC | 03:09 | |
openstackgerrit | Zhenyu Zheng proposed openstack/nova master: Support tag instances when boot https://review.openstack.org/394321 | 03:18 |
*** zhurong has quit IRC | 03:21 | |
*** thorst has joined #openstack-nova | 03:26 | |
*** zhurong has joined #openstack-nova | 03:28 | |
*** imacdonn has quit IRC | 03:28 | |
*** imacdonn has joined #openstack-nova | 03:28 | |
*** Matias has quit IRC | 03:30 | |
*** zhurong has quit IRC | 03:31 | |
*** nicolasbock has quit IRC | 03:31 | |
*** Matias has joined #openstack-nova | 03:32 | |
*** Sukhdev has joined #openstack-nova | 03:33 | |
*** gjayavelu has quit IRC | 03:38 | |
*** Sukhdev has quit IRC | 03:40 | |
*** bkopilov has joined #openstack-nova | 03:44 | |
*** thorst has quit IRC | 03:44 | |
*** vladikr has quit IRC | 03:47 | |
*** armax has quit IRC | 03:51 | |
*** fandi has joined #openstack-nova | 03:55 | |
*** ngupta has joined #openstack-nova | 03:55 | |
*** fandi has quit IRC | 03:56 | |
*** fandi has joined #openstack-nova | 03:57 | |
*** fandi has quit IRC | 03:59 | |
*** edmondsw has joined #openstack-nova | 04:00 | |
*** fandi has joined #openstack-nova | 04:00 | |
*** fandi has quit IRC | 04:01 | |
*** Shashi has joined #openstack-nova | 04:02 | |
Shashi | Hello Team, | 04:02 |
openstackgerrit | Zhenyu Zheng proposed openstack/nova master: Support tag instances when boot https://review.openstack.org/394321 | 04:02 |
Shashi | Getting AttributeError: 'module' object has no attribute 'VIR_MIGRATE_POSTCOPY' error and n-cpu services are not starting up | 04:03 |
*** fandi has joined #openstack-nova | 04:03 | |
*** Fdaisuke has joined #openstack-nova | 04:03 | |
*** edmondsw has quit IRC | 04:04 | |
*** baoli has quit IRC | 04:05 | |
*** Fdaisuke_ has quit IRC | 04:06 | |
*** dave-mccowan has quit IRC | 04:07 | |
*** baoli has joined #openstack-nova | 04:07 | |
*** ratailor has joined #openstack-nova | 04:11 | |
*** trinaths has joined #openstack-nova | 04:12 | |
imacdonn | Shashi, if this is a deployment issue, better to ask in #openstack ... but also see https://bugs.launchpad.net/openstack-devel/+bug/1682692 | 04:12 |
openstack | Launchpad bug 1682692 in OpenStack Development Infrastructure "nova fails with error: 'module' object has no attribute 'VIR_MIGRATE_POSTCOPY' after using libvirt 2.5.0 from UCA" [Undecided,New] | 04:12 |
*** neilsun has joined #openstack-nova | 04:14 | |
Shashi | Thanks imacdonn | 04:15 |
*** jerrygb has joined #openstack-nova | 04:15 | |
*** fragatin_ has joined #openstack-nova | 04:15 | |
*** eantyshev has quit IRC | 04:15 | |
*** salv-orlando has joined #openstack-nova | 04:19 | |
*** fragatina has quit IRC | 04:19 | |
*** fragatin_ has quit IRC | 04:20 | |
*** MasterOfBugs has quit IRC | 04:20 | |
*** salv-orlando has quit IRC | 04:23 | |
*** adisky_ has joined #openstack-nova | 04:28 | |
*** baoli has quit IRC | 04:31 | |
*** baoli has joined #openstack-nova | 04:31 | |
*** zhurong has joined #openstack-nova | 04:31 | |
*** hongbin has quit IRC | 04:32 | |
*** jerrygb has quit IRC | 04:32 | |
*** jerrygb has joined #openstack-nova | 04:32 | |
*** salv-orlando has joined #openstack-nova | 04:33 | |
*** trinaths has left #openstack-nova | 04:34 | |
*** fragatina has joined #openstack-nova | 04:35 | |
*** baoli has quit IRC | 04:35 | |
*** fragatina has quit IRC | 04:39 | |
*** thorst has joined #openstack-nova | 04:41 | |
*** kaisers has joined #openstack-nova | 04:42 | |
*** psachin has joined #openstack-nova | 04:44 | |
*** ayogi has joined #openstack-nova | 04:44 | |
*** vks1 has joined #openstack-nova | 04:45 | |
*** Jack_Iv has joined #openstack-nova | 04:45 | |
*** thorst has quit IRC | 04:46 | |
*** yamahata has joined #openstack-nova | 04:46 | |
*** kaisers has quit IRC | 04:47 | |
*** jerrygb has quit IRC | 04:47 | |
*** kaisers has joined #openstack-nova | 04:48 | |
*** Jack_Iv has quit IRC | 04:50 | |
*** Jack_Iv has joined #openstack-nova | 04:52 | |
*** xinliang has quit IRC | 04:52 | |
*** fragatina has joined #openstack-nova | 04:55 | |
*** Jack_Iv has quit IRC | 04:56 | |
*** Shashi has quit IRC | 04:56 | |
*** mdnadeem has joined #openstack-nova | 04:57 | |
*** fragatina has quit IRC | 04:57 | |
*** fragatina has joined #openstack-nova | 04:58 | |
*** ducnc has joined #openstack-nova | 05:00 | |
*** ducnc has quit IRC | 05:00 | |
*** Shashi has joined #openstack-nova | 05:00 | |
*** sree has joined #openstack-nova | 05:04 | |
*** moshele has joined #openstack-nova | 05:05 | |
*** sj_ has joined #openstack-nova | 05:06 | |
*** udesale has joined #openstack-nova | 05:06 | |
sj_ | what's the concept of cells in nova ? | 05:07 |
sj_ | m not getting it | 05:07 |
sj_ | ny1 ? | 05:07 |
sj_ | ?? | 05:08 |
*** guchihiro has joined #openstack-nova | 05:12 | |
*** fragatina has quit IRC | 05:17 | |
*** coreywright has quit IRC | 05:17 | |
*** dikonoor has joined #openstack-nova | 05:18 | |
*** lucky__ has joined #openstack-nova | 05:21 | |
*** nkorabli has joined #openstack-nova | 05:21 | |
*** prateek has joined #openstack-nova | 05:26 | |
*** nkorabli has quit IRC | 05:27 | |
*** nkorabli has joined #openstack-nova | 05:28 | |
*** Jack_Iv has joined #openstack-nova | 05:30 | |
*** nkorabli has quit IRC | 05:33 | |
*** coreywright has joined #openstack-nova | 05:35 | |
*** ijw_ has quit IRC | 05:38 | |
*** nkorabli has joined #openstack-nova | 05:42 | |
*** thorst has joined #openstack-nova | 05:42 | |
*** fandi has quit IRC | 05:45 | |
*** thorst has quit IRC | 05:46 | |
*** gcb has quit IRC | 05:51 | |
openstackgerrit | Huan Xie proposed openstack/nova master: WIP: XenAPI use os-xenapi v2 in nova https://review.openstack.org/453493 | 05:51 |
*** lucky__ has quit IRC | 05:52 | |
*** sj_ has quit IRC | 05:52 | |
*** xinliang has joined #openstack-nova | 05:53 | |
*** xinliang has quit IRC | 05:53 | |
*** xinliang has joined #openstack-nova | 05:53 | |
*** gouthamr has quit IRC | 05:54 | |
*** damien_r has joined #openstack-nova | 05:54 | |
*** CristinaPauna has quit IRC | 05:57 | |
*** neilsun has quit IRC | 05:58 | |
*** neilsun has joined #openstack-nova | 05:59 | |
*** edmondsw has joined #openstack-nova | 06:00 | |
*** Jack_Iv has quit IRC | 06:02 | |
*** winston-d_ has joined #openstack-nova | 06:03 | |
*** Oku_OS-away is now known as Oku_OS | 06:04 | |
*** ijw has joined #openstack-nova | 06:04 | |
*** edmondsw has quit IRC | 06:05 | |
*** ijw has quit IRC | 06:08 | |
*** Jack_Iv has joined #openstack-nova | 06:11 | |
*** zhurong has quit IRC | 06:12 | |
*** salv-orlando has quit IRC | 06:12 | |
*** Jack_Iv has quit IRC | 06:15 | |
*** kornicameister has joined #openstack-nova | 06:16 | |
*** ijw has joined #openstack-nova | 06:20 | |
*** vks1 has quit IRC | 06:22 | |
*** david-lyle has quit IRC | 06:23 | |
*** nkorabli has quit IRC | 06:23 | |
*** nkorabli has joined #openstack-nova | 06:24 | |
openstackgerrit | Alex Xu proposed openstack/nova master: Deprecate Multinic, floatingip action and os-virtual-interface API https://review.openstack.org/457181 | 06:26 |
*** Jack_Iv has joined #openstack-nova | 06:27 | |
openstackgerrit | Lei Zhang proposed openstack/nova master: Add sync traits command for placement https://review.openstack.org/450125 | 06:27 |
*** vks1 has joined #openstack-nova | 06:27 | |
*** Jack_Iv has quit IRC | 06:27 | |
*** voelzmo has joined #openstack-nova | 06:27 | |
*** nkorabli has quit IRC | 06:28 | |
*** rcernin has joined #openstack-nova | 06:28 | |
openstackgerrit | Huan Xie proposed openstack/nova master: WIP: XenAPI use os-xenapi v2 in nova https://review.openstack.org/453493 | 06:28 |
openstackgerrit | Alex Xu proposed openstack/nova master: Use plain routes list for flavors-access endpoint instead of stevedore https://review.openstack.org/456589 | 06:33 |
openstackgerrit | Alex Xu proposed openstack/nova master: Use plain routes list for '/servers' endpoint instead of stevedore https://review.openstack.org/445864 | 06:33 |
openstackgerrit | Alex Xu proposed openstack/nova master: Use plain routes list for flavor endpoint instead of stevedore[1] https://review.openstack.org/456549 | 06:33 |
openstackgerrit | Alex Xu proposed openstack/nova master: Use plain routes list for '/os-aggregates' endpoint instead of stevedore https://review.openstack.org/456825 | 06:33 |
openstackgerrit | Alex Xu proposed openstack/nova master: Use plain routes list for flavors-extraspecs endpoint instead of stevedore https://review.openstack.org/456585 | 06:33 |
openstackgerrit | Alex Xu proposed openstack/nova master: Use plain routes list for '/os-keypairs' endpoint instead of stevedore https://review.openstack.org/456478 | 06:33 |
*** slaweq has joined #openstack-nova | 06:37 | |
*** david-lyle has joined #openstack-nova | 06:37 | |
*** neilsun has quit IRC | 06:38 | |
*** andreas_s has joined #openstack-nova | 06:40 | |
*** Zhaomingjun has joined #openstack-nova | 06:40 | |
*** pcaruana has joined #openstack-nova | 06:40 | |
*** dimtruck is now known as zz_dimtruck | 06:46 | |
*** nkorabli has joined #openstack-nova | 06:49 | |
*** ralonsoh has joined #openstack-nova | 06:51 | |
*** tesseract has joined #openstack-nova | 06:56 | |
*** damien_r has quit IRC | 06:57 | |
*** david-lyle has quit IRC | 07:02 | |
*** david-lyle has joined #openstack-nova | 07:03 | |
*** takashin has joined #openstack-nova | 07:03 | |
*** markus_z has joined #openstack-nova | 07:07 | |
*** ijw has quit IRC | 07:07 | |
*** ijw has joined #openstack-nova | 07:10 | |
*** jaosorior_away is now known as jaosorior | 07:11 | |
*** salv-orlando has joined #openstack-nova | 07:11 | |
*** karimb has joined #openstack-nova | 07:12 | |
*** zhurong has joined #openstack-nova | 07:12 | |
*** sridharg has joined #openstack-nova | 07:13 | |
*** jhesketh has quit IRC | 07:14 | |
*** ijw has quit IRC | 07:15 | |
*** damien_r has joined #openstack-nova | 07:15 | |
*** Drankis has joined #openstack-nova | 07:19 | |
*** ltomasbo|away is now known as ltomasbo | 07:21 | |
*** mlakat has joined #openstack-nova | 07:24 | |
*** Zhaomingjun has quit IRC | 07:26 | |
*** jamielennox has quit IRC | 07:28 | |
*** neilsun has joined #openstack-nova | 07:33 | |
*** Drankis has quit IRC | 07:38 | |
*** jamielennox has joined #openstack-nova | 07:42 | |
*** jpena|off is now known as jpena | 07:42 | |
*** thorst has joined #openstack-nova | 07:44 | |
*** jhesketh has joined #openstack-nova | 07:44 | |
*** MasterOfBugs has joined #openstack-nova | 07:44 | |
*** karimb has quit IRC | 07:45 | |
*** slaweq has quit IRC | 07:48 | |
*** slaweq has joined #openstack-nova | 07:49 | |
*** ratailor is now known as ratailor|Lunch | 07:49 | |
*** Drankis has joined #openstack-nova | 07:51 | |
*** zzzeek has quit IRC | 08:00 | |
*** zzzeek has joined #openstack-nova | 08:00 | |
*** thorst has quit IRC | 08:03 | |
*** karimb has joined #openstack-nova | 08:08 | |
*** slaweq has quit IRC | 08:08 | |
*** slaweq has joined #openstack-nova | 08:09 | |
*** ijw has joined #openstack-nova | 08:10 | |
*** CristinaPauna has joined #openstack-nova | 08:11 | |
*** ijw has quit IRC | 08:16 | |
*** mpaolino has joined #openstack-nova | 08:19 | |
openstackgerrit | Zhenyu Zheng proposed openstack/nova master: Support tag instances when boot https://review.openstack.org/394321 | 08:21 |
*** efoley has joined #openstack-nova | 08:25 | |
*** CristinaPauna has quit IRC | 08:25 | |
*** CristinaPauna has joined #openstack-nova | 08:26 | |
*** efoley_ has joined #openstack-nova | 08:27 | |
*** aarefiev_afk is now known as aarefiev | 08:29 | |
*** efoley has quit IRC | 08:31 | |
*** lucas-afk is now known as lucasagomes | 08:31 | |
*** tovin07__ has joined #openstack-nova | 08:35 | |
*** ralonsoh_ has joined #openstack-nova | 08:36 | |
*** tovin07_ has quit IRC | 08:36 | |
*** ralonsoh_ has quit IRC | 08:36 | |
*** ralonsoh_ has joined #openstack-nova | 08:37 | |
*** ralonsoh has quit IRC | 08:39 | |
*** ratailor|Lunch is now known as ratailor | 08:40 | |
*** Fdaisuke_ has joined #openstack-nova | 08:40 | |
*** guchihiro has quit IRC | 08:40 | |
*** MasterOfBugs has quit IRC | 08:42 | |
*** Fdaisuke has quit IRC | 08:42 | |
openstackgerrit | Lei Zhang proposed openstack/nova master: Add sync traits command for placement https://review.openstack.org/450125 | 08:44 |
*** mnestratov has joined #openstack-nova | 08:47 | |
*** david-lyle has quit IRC | 08:47 | |
openstackgerrit | Stephen Finucane proposed openstack/nova master: conf: Rename '[neutron] ovs_bridge' https://review.openstack.org/450266 | 08:49 |
openstackgerrit | Stephen Finucane proposed openstack/nova master: conf: Move 'floating_ips' opts into 'network' https://review.openstack.org/431039 | 08:49 |
openstackgerrit | Stephen Finucane proposed openstack/nova master: conf: Deprecate 'default_floating_pool' https://review.openstack.org/431038 | 08:49 |
openstackgerrit | Stephen Finucane proposed openstack/nova master: conf: Add neutron.default_floating_pool https://review.openstack.org/431037 | 08:49 |
openstackgerrit | Dao Cong Tien proposed openstack/nova master: Stop using mox int unit/virt/xenapi/image/test_utils.py https://review.openstack.org/342626 | 08:53 |
*** moshele has quit IRC | 08:56 | |
*** abalutoiu__ has joined #openstack-nova | 08:58 | |
*** abalutoiu__ is now known as abalutoiu | 09:01 | |
*** abalutoiu_ has quit IRC | 09:01 | |
*** efoley__ has joined #openstack-nova | 09:01 | |
*** karimb has quit IRC | 09:02 | |
openstackgerrit | Stephen Finucane proposed openstack/nova master: libvirt: Make 'get_domain' private https://review.openstack.org/417378 | 09:04 |
*** efoley_ has quit IRC | 09:04 | |
*** karimb has joined #openstack-nova | 09:06 | |
openstackgerrit | Stephen Finucane proposed openstack/nova master: conf: Set default for live_migration_scheme https://review.openstack.org/457716 | 09:06 |
openstackgerrit | Stephen Finucane proposed openstack/nova master: conf: Add three new '[libvirt] live_migration_*' options https://review.openstack.org/456571 | 09:06 |
openstackgerrit | Stephen Finucane proposed openstack/nova master: conf: Gather 'live_migration_scheme', 'live_migration_inbound_addr' https://review.openstack.org/456572 | 09:06 |
openstackgerrit | Stephen Finucane proposed openstack/nova master: conf: Convert 'live_migration_inbound_addr' to HostAddressOpt https://review.openstack.org/456573 | 09:06 |
*** faizy_ has joined #openstack-nova | 09:08 | |
*** neilsun has quit IRC | 09:09 | |
*** faizy_ has quit IRC | 09:09 | |
*** faizy_ has joined #openstack-nova | 09:09 | |
*** faizy_ has quit IRC | 09:11 | |
*** faizy has quit IRC | 09:11 | |
*** salv-orlando has quit IRC | 09:13 | |
*** ijw has joined #openstack-nova | 09:13 | |
*** NikhilS has quit IRC | 09:13 | |
openstackgerrit | Andy McCrae proposed openstack/nova master: Allow CONTENT_LENGTH to be present but empty https://review.openstack.org/455710 | 09:15 |
*** moshele has joined #openstack-nova | 09:17 | |
*** ijw has quit IRC | 09:18 | |
*** neilsun has joined #openstack-nova | 09:23 | |
*** tovin07_ has joined #openstack-nova | 09:25 | |
*** zenoway has joined #openstack-nova | 09:27 | |
*** tovin07__ has quit IRC | 09:27 | |
*** yamahata has quit IRC | 09:29 | |
*** cdent has joined #openstack-nova | 09:30 | |
*** lpetrut has joined #openstack-nova | 09:32 | |
*** derekh has joined #openstack-nova | 09:33 | |
*** sambetts|afk is now known as sambetts | 09:33 | |
*** karimb has quit IRC | 09:41 | |
*** gszasz has joined #openstack-nova | 09:44 | |
*** rmart04 has joined #openstack-nova | 09:51 | |
*** zenoway has quit IRC | 09:54 | |
*** karimb has joined #openstack-nova | 09:56 | |
*** zenoway has joined #openstack-nova | 09:57 | |
*** cdent_ has joined #openstack-nova | 09:57 | |
*** cdent has quit IRC | 09:58 | |
*** cdent_ is now known as cdent | 09:58 | |
*** tovin07_ has quit IRC | 09:59 | |
*** thorst has joined #openstack-nova | 10:00 | |
*** nicolasbock has joined #openstack-nova | 10:02 | |
*** thorst has quit IRC | 10:04 | |
*** haplo37_ has quit IRC | 10:05 | |
*** hferenc has quit IRC | 10:07 | |
*** karimb has quit IRC | 10:08 | |
*** karimb has joined #openstack-nova | 10:09 | |
*** ijw has joined #openstack-nova | 10:14 | |
*** haplo37_ has joined #openstack-nova | 10:16 | |
*** thomasem has quit IRC | 10:17 | |
*** thomasem has joined #openstack-nova | 10:18 | |
*** slaweq has quit IRC | 10:19 | |
*** zhurong has quit IRC | 10:19 | |
*** ijw has quit IRC | 10:19 | |
*** sdague has joined #openstack-nova | 10:21 | |
*** sree has quit IRC | 10:29 | |
kashyap | johnthetubaguy: Hi there | 10:30 |
kashyap | johnthetubaguy: You abandoned this one with: "There are some other threads on this now." | 10:30 |
kashyap | https://review.openstack.org/#/c/377756/ | 10:30 |
kashyap | Care to point those threads? | 10:30 |
*** thomasem_ has joined #openstack-nova | 10:36 | |
*** cdent_ has joined #openstack-nova | 10:38 | |
*** kevinz has quit IRC | 10:39 | |
openstackgerrit | Huan Xie proposed openstack/nova master: WIP: XenAPI use os-xenapi v2 in nova https://review.openstack.org/453493 | 10:40 |
*** cdent has quit IRC | 10:40 | |
*** cdent_ is now known as cdent | 10:40 | |
openstackgerrit | Takashi NATSUME proposed openstack/python-novaclient master: Microversion 2.43 - Enable cold migration with target host https://review.openstack.org/406707 | 10:41 |
openstackgerrit | Takashi NATSUME proposed openstack/python-novaclient master: Microversion 2.43 - List/Show all server migration types https://review.openstack.org/430839 | 10:42 |
openstackgerrit | Takashi NATSUME proposed openstack/python-novaclient master: Microversion 2.43 - List/Show all server migration types https://review.openstack.org/430839 | 10:42 |
*** ociuhandu has joined #openstack-nova | 10:45 | |
*** rmart04 has quit IRC | 10:45 | |
*** fragatina has joined #openstack-nova | 10:47 | |
*** cNilesh has quit IRC | 10:48 | |
*** gongysh has quit IRC | 10:50 | |
*** rmart04 has joined #openstack-nova | 10:50 | |
*** karimb has quit IRC | 10:57 | |
*** ralonsoh__ has joined #openstack-nova | 10:57 | |
*** slaweq has joined #openstack-nova | 11:00 | |
*** ralonsoh_ has quit IRC | 11:01 | |
*** thorst has joined #openstack-nova | 11:01 | |
*** fragatina has quit IRC | 11:03 | |
*** winston-d_ has quit IRC | 11:03 | |
*** fragatina has joined #openstack-nova | 11:03 | |
*** karimb has joined #openstack-nova | 11:03 | |
*** thorst has quit IRC | 11:06 | |
*** markvoelker has quit IRC | 11:06 | |
*** efoley_ has joined #openstack-nova | 11:06 | |
*** markvoelker has joined #openstack-nova | 11:06 | |
*** mvk has quit IRC | 11:06 | |
*** bkopilov has quit IRC | 11:07 | |
*** efoley__ has quit IRC | 11:09 | |
*** jaosorior has quit IRC | 11:09 | |
johnthetubaguy | kashyap: the API WG has a spec, and we have a new one, I should have linked to those | 11:10 |
kashyap | johnthetubaguy: Thanks. | 11:10 |
kashyap | It's just dizzying about where we stand with all those abandoned specs | 11:10 |
johnthetubaguy | kashyap: I think this is the new one: https://review.openstack.org/454792 | 11:11 |
*** markvoelker has quit IRC | 11:11 | |
johnthetubaguy | kashyap: they were all abandoned largely due to un answered -1s for several months | 11:11 |
johnthetubaguy | kashyap: but like we do for patches | 11:11 |
kashyap | johnthetubaguy: jaypipes: Is this also reproposed? "Standardize capabilities using Enums" -- https://review.openstack.org/#/c/309762/ | 11:11 |
kashyap | johnthetubaguy: Yeah, figured as much. Can't keep inactive specs around forever | 11:12 |
johnthetubaguy | kashyap: I think thats alex_xu's work around traits | 11:12 |
*** nkorabli has quit IRC | 11:12 | |
*** zenoway has quit IRC | 11:13 | |
johnthetubaguy | kashyap: yeah, exactly that | 11:13 |
alex_xu | kashyap: johnthetubaguy, yea that is replaced by trait | 11:13 |
alex_xu | kashyap: the standard trait is in https://github.com/openstack/os-traits now | 11:14 |
* kashyap reads | 11:14 | |
alex_xu | kashyap: this is for traits api http://specs.openstack.org/openstack/nova-specs/specs/pike/approved/resource-provider-traits.html | 11:14 |
kashyap | johnthetubaguy: alex_xu: So, about instance live resize up / down, etc, was there a discussion about its status yet? | 11:14 |
kashyap | (I mean at the past Summit) | 11:15 |
* kashyap should probably look at the Etherpad | 11:15 | |
johnthetubaguy | kashyap: at the ptg I think we said still blocked behind getting agreement on capabilities | 11:15 |
kashyap | johnthetubaguy: Ah-ha | 11:15 |
*** karimb has quit IRC | 11:15 | |
johnthetubaguy | ... but the folks who stepped up to do that just got laid off, so thats going to need another new owner | 11:15 |
*** ijw has joined #openstack-nova | 11:16 | |
alex_xu | :( | 11:16 |
kashyap | johnthetubaguy: Oh, from which company are the said folks, if I may ask? | 11:17 |
*** kaisers_ has joined #openstack-nova | 11:17 | |
openstackgerrit | Takashi NATSUME proposed openstack/nova-specs master: Abort Cold Migration https://review.openstack.org/334732 | 11:17 |
*** zhurong has joined #openstack-nova | 11:17 | |
johnthetubaguy | rackspace/OSIC | 11:17 |
kashyap | Ah-ha | 11:17 |
openstackgerrit | Takashi NATSUME proposed openstack/nova-specs master: Specify keymap on server boot https://review.openstack.org/448830 | 11:17 |
*** zenoway has joined #openstack-nova | 11:18 | |
johnthetubaguy | kashyap: I think matt is getting a list of all the impacted priority items that need owners now, but still not sure whats happening to everyone yet | 11:18 |
kashyap | johnthetubaguy: Yeah, understood. Thank you | 11:19 |
*** ratailor has quit IRC | 11:19 | |
*** kaisers has quit IRC | 11:20 | |
*** jaosorior has joined #openstack-nova | 11:20 | |
*** ijw has quit IRC | 11:21 | |
*** BlackDex has quit IRC | 11:21 | |
*** smatzek has joined #openstack-nova | 11:23 | |
*** jpena is now known as jpena|lunch | 11:25 | |
*** smatzek has quit IRC | 11:25 | |
*** smatzek has joined #openstack-nova | 11:25 | |
*** thorst has joined #openstack-nova | 11:26 | |
*** alexpilotti has quit IRC | 11:26 | |
*** trinaths has joined #openstack-nova | 11:26 | |
*** alexpilotti has joined #openstack-nova | 11:27 | |
*** zeroDivisible has quit IRC | 11:27 | |
*** zenoway has quit IRC | 11:27 | |
openstackgerrit | Zhenyu Zheng proposed openstack/nova master: Support tag instances when boot https://review.openstack.org/394321 | 11:29 |
*** lennyb has quit IRC | 11:29 | |
*** lennyb has joined #openstack-nova | 11:30 | |
*** zeroDivisible has joined #openstack-nova | 11:31 | |
*** alexpilotti has quit IRC | 11:31 | |
*** cdent has quit IRC | 11:31 | |
zioproto | dansmith: hey are here ? | 11:32 |
*** openstackgerrit has quit IRC | 11:32 | |
zioproto | dansmith: remember when we fixed something in the nova migrations for the db upgrade from mitaka to newton ? | 11:32 |
zioproto | dansmith: I am testing now the migration of my production database... and I get these output | 11:33 |
zioproto | dansmith: https://pastebin.com/NU85iyqN any ideas ? | 11:34 |
zioproto | all those ERROR message dont look any good | 11:35 |
*** neilsun has quit IRC | 11:36 | |
johnthetubaguy | I see "not all services that access the DB directly are updated to the latest version" | 11:36 |
johnthetubaguy | zioproto: not sure if that helps, but has you got everything upgrade before you do the data migrations for that release? | 11:37 |
*** djohnsto has joined #openstack-nova | 11:37 | |
*** phuongnh has quit IRC | 11:37 | |
zioproto | what I am doing here | 11:38 |
zioproto | I just get a dump of the production database | 11:38 |
zioproto | and I put it on a new db, that no other software touches | 11:38 |
zioproto | I install nova on this server | 11:38 |
zioproto | and I just run the commands you see in the pastebin | 11:38 |
zioproto | it is just to test the migrations | 11:38 |
zioproto | the database is my mitaka production database | 11:39 |
zioproto | and the packages I install are ubuntu nova Installed: 2:14.0.4-0ubuntu1.2~cloud0 | 11:39 |
zioproto | johnthetubaguy: makes sense to do the test in this way ? | 11:39 |
*** rfolco has joined #openstack-nova | 11:43 | |
*** alexpilotti has joined #openstack-nova | 11:43 | |
*** djohnsto has quit IRC | 11:45 | |
*** abalutoiu has quit IRC | 11:46 | |
*** Shunli has quit IRC | 11:46 | |
*** alexpilotti has quit IRC | 11:47 | |
*** jerrygb has joined #openstack-nova | 11:49 | |
*** jerrygb has quit IRC | 11:54 | |
*** openstackgerrit has joined #openstack-nova | 11:57 | |
openstackgerrit | Takashi NATSUME proposed openstack/nova master: Fix unnecessary code block in a release note https://review.openstack.org/458472 | 11:57 |
johnthetubaguy | zioproto: sorry, just heading for my lunch, I think you need to update the service records to pretend they are running newton, before you can run the newton data migrations | 11:57 |
*** pchavva has joined #openstack-nova | 11:59 | |
*** zenoway has joined #openstack-nova | 11:59 | |
*** lyan has joined #openstack-nova | 12:00 | |
*** alexpilotti has joined #openstack-nova | 12:00 | |
*** jpena|lunch is now known as jpena | 12:01 | |
*** Jack_Iv has joined #openstack-nova | 12:02 | |
*** Jack_Iv has quit IRC | 12:02 | |
*** neilsun has joined #openstack-nova | 12:04 | |
*** alexpilotti has quit IRC | 12:04 | |
*** alexpilotti has joined #openstack-nova | 12:05 | |
zioproto | johnthetubaguy: thanks I will try that | 12:07 |
*** kevinz has joined #openstack-nova | 12:08 | |
*** alexpilotti has quit IRC | 12:09 | |
*** karimb has joined #openstack-nova | 12:09 | |
*** jerrygb has joined #openstack-nova | 12:09 | |
*** mpaolino_ has joined #openstack-nova | 12:10 | |
*** vks1 has quit IRC | 12:12 | |
*** edmondsw has joined #openstack-nova | 12:12 | |
*** zenoway has quit IRC | 12:13 | |
*** mpaolino has quit IRC | 12:13 | |
*** jerrygb has quit IRC | 12:14 | |
*** mvk has joined #openstack-nova | 12:15 | |
*** zenoway has joined #openstack-nova | 12:16 | |
*** ijw has joined #openstack-nova | 12:17 | |
*** kevinz has quit IRC | 12:18 | |
*** kevinz has joined #openstack-nova | 12:19 | |
*** sree has joined #openstack-nova | 12:22 | |
*** ijw has quit IRC | 12:22 | |
*** fragatina has quit IRC | 12:22 | |
*** jaypipes has joined #openstack-nova | 12:23 | |
*** fragatina has joined #openstack-nova | 12:23 | |
*** abalutoiu has joined #openstack-nova | 12:24 | |
zioproto | johnthetubaguy: I had to fix the thing with 'delete from services where deleted_at is not NULL;' | 12:25 |
zioproto | looks like if you have in your database services flagged as deleted this can break the migration | 12:25 |
*** dave-mccowan has joined #openstack-nova | 12:27 | |
*** catintheroof has joined #openstack-nova | 12:27 | |
*** markus_z has quit IRC | 12:28 | |
*** catintheroof has quit IRC | 12:30 | |
*** catintheroof has joined #openstack-nova | 12:30 | |
johnthetubaguy | zioproto: ah, that rings a bell, I think dansmith has patch that already https://github.com/openstack/nova/commit/2e05c82b807624b8420df4c94d68b75563b230d2 I guess its not in the package | 12:32 |
johnthetubaguy | actually, yeah, looks like thats not in a tagged release quite yet | 12:32 |
asettle | nicolasbock: Hey if you're around, I need a nova docs CPL ;) https://bugs.launchpad.net/openstack-manuals/+bug/1683104 and https://bugs.launchpad.net/openstack-manuals/+bug/1683071 | 12:33 |
openstack | Launchpad bug 1683104 in openstack-manuals "instance is not created rightly" [Undecided,New] | 12:33 |
openstack | Launchpad bug 1683071 in openstack-manuals "error installing nova-common" [Undecided,New] | 12:33 |
asettle | I just need someone to help verify :( | 12:33 |
zioproto | Yes but this commit is in Mitaka | 12:33 |
zioproto | I am confused | 12:33 |
nicolasbock | asettle, ok, let me have a look | 12:33 |
asettle | Thanks :) | 12:33 |
*** mdrabe has joined #openstack-nova | 12:34 | |
*** salv-orlando has joined #openstack-nova | 12:35 | |
*** hshiina has joined #openstack-nova | 12:35 | |
zioproto | johnthetubaguy: maybe you are right, this https://review.openstack.org/#/c/438630/ is only in 14.0.5 | 12:36 |
zioproto | and I have 14.0.4 | 12:36 |
*** Shashi has quit IRC | 12:37 | |
*** annegentle has joined #openstack-nova | 12:37 | |
*** markvoelker has joined #openstack-nova | 12:39 | |
*** catintheroof has quit IRC | 12:40 | |
*** catintheroof has joined #openstack-nova | 12:41 | |
*** catintheroof has quit IRC | 12:41 | |
*** catintheroof has joined #openstack-nova | 12:41 | |
*** gcb has joined #openstack-nova | 12:43 | |
*** catintheroof has quit IRC | 12:43 | |
*** catintheroof has joined #openstack-nova | 12:43 | |
*** liverpooler has joined #openstack-nova | 12:46 | |
*** vladikr has joined #openstack-nova | 12:49 | |
*** yingwei has quit IRC | 12:50 | |
*** bkopilov has joined #openstack-nova | 12:50 | |
*** eharney has joined #openstack-nova | 12:51 | |
*** zenoway has quit IRC | 12:51 | |
*** zenoway has joined #openstack-nova | 12:52 | |
*** jerrygb has joined #openstack-nova | 12:56 | |
*** mpaolino_ has quit IRC | 12:56 | |
*** djohnsto has joined #openstack-nova | 12:57 | |
*** sree has quit IRC | 12:57 | |
*** annegentle has quit IRC | 12:57 | |
*** sree has joined #openstack-nova | 12:58 | |
*** ababich has quit IRC | 12:59 | |
*** alexpilotti has joined #openstack-nova | 13:01 | |
*** slaweq has quit IRC | 13:03 | |
*** cleong has joined #openstack-nova | 13:03 | |
*** zenoway has quit IRC | 13:04 | |
*** zenoway has joined #openstack-nova | 13:04 | |
*** mpaolino_ has joined #openstack-nova | 13:13 | |
*** sree has quit IRC | 13:14 | |
*** sree has joined #openstack-nova | 13:15 | |
*** ijw has joined #openstack-nova | 13:18 | |
*** alexpilo_ has joined #openstack-nova | 13:19 | |
*** sree has quit IRC | 13:19 | |
*** cdent has joined #openstack-nova | 13:20 | |
*** mriedem has joined #openstack-nova | 13:20 | |
*** CristinaPauna has quit IRC | 13:21 | |
efried | mriedem Good morning. When you have a sec - I noticed in your "Reflections" email the term "shared storage pools" - was wondering what that means? | 13:21 |
*** claudiub has quit IRC | 13:21 | |
*** slaweq has joined #openstack-nova | 13:22 | |
*** slaweq has quit IRC | 13:22 | |
*** abalutoiu has quit IRC | 13:22 | |
*** alexpilotti has quit IRC | 13:22 | |
*** burt has joined #openstack-nova | 13:22 | |
*** erhudy has joined #openstack-nova | 13:23 | |
mriedem | a pool, of shared storage | 13:23 |
mriedem | efried: https://specs.openstack.org/openstack/nova-specs/specs/newton/implemented/generic-resource-pools.html | 13:23 |
*** alexpilotti has joined #openstack-nova | 13:24 | |
*** ijw has quit IRC | 13:24 | |
*** traskat has joined #openstack-nova | 13:24 | |
*** hypothermic_cat is now known as ildikov | 13:25 | |
*** ociuhandu has quit IRC | 13:25 | |
efried | mriedem Okay, thanks. PowerVM has Shared Storage Pools (capitals intentional) which is a very specific technology based on a cluster of VIOSes etc. Just wanting to anticipate future confusion that will be caused by e.g. https://review.openstack.org/443189 | 13:25 |
mriedem | it's just a file share isn't it? | 13:26 |
dansmith | zioproto: yep, what johnthetubaguy said.. there are some data migrations that won't run without the services having been updated yet | 13:26 |
dansmith | zioproto: specifically one, that PciDevice one | 13:26 |
*** alexpilo_ has quit IRC | 13:26 | |
dansmith | zioproto: if you don't use any pci devices in your deployment then it won't do anything when it runs anyway | 13:26 |
johnthetubaguy | dansmith: it sounds like he was missing your fix up for the delete services thingy | 13:27 |
*** lpetrut has quit IRC | 13:27 | |
johnthetubaguy | old packages | 13:27 |
johnthetubaguy | well... older | 13:27 |
traskat | hey, what i have to do apply to apply extra specs to running VMs for examaple disk IO limitations? | 13:27 |
dansmith | johnthetubaguy: ah maybe | 13:27 |
sfinucan | mriedem: Could you take a look at these today? https://review.openstack.org/#/q/status:open+project:openstack/nova+branch:master+topic:bump-min-libvirt | 13:27 |
mriedem | sfinucan: i can | 13:27 |
sfinucan | mriedem: Cheers | 13:28 |
efried | mriedem PowerVM SSP? No. The VIOSes share a pool of SAN disks and carve out LUs, which can then be attached to VMs. Things like thin clones are supported. Aforementioned patch allows multiple separate hosts (which otherwise don't communicate with each other) to share a SSP, including sharing images. | 13:28 |
efried | The VIOSes use an AIX-specific cluster technology, and an AIX-specific shared file system type. | 13:29 |
mriedem | efried: at a high level it's the same concept for modeling it in the placement service | 13:29 |
mriedem | it's a shared storage resource provider, | 13:29 |
mriedem | which will be in an aggregate relationship with the compute node resource providers that it serves disk to | 13:29 |
*** trinaths has left #openstack-nova | 13:30 | |
efried | mriedem Cool, maybe we ( thorst ) can look into folding PowerVM SSP into that concept. Thanks. | 13:30 |
mriedem | so rather than each compute node resource provider reporting it has 100TB of disk or whatever the pool size is, which would be a lie if you have 10 computes for the same storage (10*100TB), you'd model it so that the 10 compute RPs share the same disk RP with 100TB | 13:30 |
mriedem | efried: it's an operator thing to set it up properly | 13:31 |
efried | But we would have to enable/support it. | 13:31 |
mriedem | jaypipes or cdent or edleafe can correct me if i'm wrong | 13:31 |
* jaypipes reads back | 13:31 | |
edleafe | mriedem: correctamundo | 13:32 |
jaypipes | efried: totes. the VIOSes in AIX on PVM SSPs on SAN. yep, totes. | 13:33 |
*** ociuhandu has joined #openstack-nova | 13:34 | |
openstackgerrit | Sandeep Pawar proposed openstack/nova master: Libvirt volume driver for Veritas HyperScale https://review.openstack.org/443951 | 13:34 |
*** ig0r_ has joined #openstack-nova | 13:34 | |
*** sree has joined #openstack-nova | 13:34 | |
thorst | efried: yeah, we should definitely fold into that concept... (was afk) | 13:36 |
*** felipemonteiro has joined #openstack-nova | 13:36 | |
efried | thorst adreznec Let's queue it up for discussion in our box doc. | 13:36 |
thorst | yep | 13:37 |
thorst | we may have some FS ones in the same boat. | 13:37 |
*** sree has quit IRC | 13:38 | |
efried | thorst Though it looks like maybe there is no affordance for it to be automated - the deployer just has to know which hosts are sharing an SSP and run these resource-provider set inventory commands accordingly. | 13:39 |
*** smatzek has quit IRC | 13:39 | |
thorst | well, we've got a command to tell them all the hosts sharing...so. Maybe something to doc for that process. | 13:40 |
*** abalutoiu has joined #openstack-nova | 13:40 | |
*** gcb has quit IRC | 13:40 | |
efried | Would be kinda neat if there was a method on the compute driver that could be asked to provide some kind of identifier for the shared resource (the SSP UUID in this case?) so the controller can figure out the pool on its own. | 13:40 |
efried | mriedem edleafe jaypipes Future enhancement ^^ ? | 13:40 |
efried | (though I haven't finished reading the spec yet ;-) | 13:41 |
jaypipes | efried: in case it wasn't obvious, I was being sarcastic above. :) | 13:41 |
*** ociuhandu has quit IRC | 13:41 | |
efried | jaypipes Oh, it wasn't. | 13:42 |
*** abalutoiu has quit IRC | 13:42 | |
efried | To me | 13:42 |
jaypipes | efried: I was poking fun at your TLAs :P | 13:42 |
efried | Gotcha. IBM does love its TLAs. | 13:42 |
*** abalutoiu has joined #openstack-nova | 13:42 | |
jaypipes | efried: joking aside, I don't see anything that would prevent SSPs from being used as shared resource providers, no. | 13:42 |
openstackgerrit | Roman Podoliaka proposed openstack/osc-placement master: CLI for resource providers https://review.openstack.org/457532 | 13:43 |
openstackgerrit | Roman Podoliaka proposed openstack/osc-placement master: CLI for inventories https://review.openstack.org/457533 | 13:43 |
openstackgerrit | Roman Podoliaka proposed openstack/osc-placement master: CLI for allocations https://review.openstack.org/457534 | 13:43 |
openstackgerrit | Roman Podoliaka proposed openstack/osc-placement master: CLI for usages https://review.openstack.org/457535 | 13:43 |
efried | jaypipes Agree. But seems like it should be possible without operator intervention - at least in the narrow case I'm thinking about. | 13:43 |
jaypipes | efried: yes, indeed. | 13:44 |
efried | Not sure if possible in the general case. | 13:44 |
*** jamesden_ has joined #openstack-nova | 13:44 | |
efried | I would think it should be possible if the shared storage was via NFS. | 13:44 |
efried | As long as the computes had a reliable way to canonicalize the host and mountpoint. | 13:45 |
rpodolyaka | cdent: thanks for your input on osc-placement changes! it's good to have a pair of fresh eyes on this | 13:45 |
thorst | could do the old - have each compute create a file on the share...and then have the compute drivers report back what they see. From that you could figure out who can see who... | 13:46 |
openstackgerrit | Lee Yarwood proposed openstack/nova master: encryptors: Switch to os-brick encryptor classes https://review.openstack.org/391597 | 13:46 |
*** david-lyle has joined #openstack-nova | 13:46 | |
cdent | rpodolyaka: you're very welcome | 13:47 |
lyarwood | mriedem: would you be willing to approve the bp for ^ https://blueprints.launchpad.net/nova/+spec/switch-to-os-brick-encryptor-classes or should I bring it up during the meeting today? | 13:48 |
*** abalutoiu has quit IRC | 13:49 | |
*** ekuris has quit IRC | 13:50 | |
mriedem | lyarwood: bring it up during open discussion | 13:51 |
lyarwood | mriedem: ack thanks | 13:51 |
*** mlavalle has joined #openstack-nova | 13:52 | |
cdent | johnthetubaguy: I've seen your comments on the service stuff on https://review.openstack.org/#/c/457283/ and I'm confused on one point that you might be able to help me on: | 13:53 |
cdent | johnthetubaguy: do we expect the host field in the service record to be the same or different for each instance of the wsgi app? | 13:53 |
johnthetubaguy | cdent: I think we want an entry for each version that is installed | 13:54 |
johnthetubaguy | cdent: so we can tell when the upgrade has finished | 13:54 |
johnthetubaguy | (or not) | 13:54 |
cdent | that doesn't really answer my question :) | 13:54 |
johnthetubaguy | cdent: true, I think the answer is yes, but the above is why | 13:55 |
johnthetubaguy | oops, by yes, I mean different for each instance | 13:55 |
johnthetubaguy | the problem is cleaning up old ones, etc, its a bit... unexpected | 13:55 |
cdent | so many knobs | 13:56 |
cdent | okay, I'll make some adjustments. using CONF.host seems like a reasonable way to go | 13:56 |
*** diegows has joined #openstack-nova | 13:56 | |
johnthetubaguy | cdent: I think this is the extra context: https://review.openstack.org/#/c/384724/ and https://launchpad.net/bugs/1631430 | 13:56 |
openstack | Launchpad bug 1631430 in OpenStack Compute (nova) "Only one entry created for nova-osapi_compute and nova-metadata services in multi-n-api environment" [Medium,In progress] - Assigned to Sujitha (sujitha-neti) | 13:56 |
mriedem | nova meeting in 3 minutes in #openstack-meeting | 13:57 |
cdent | thanks johnthetubaguy | 13:57 |
openstackgerrit | sahid proposed openstack/nova master: network: add command to configure trusted mode for VFs https://review.openstack.org/458513 | 13:58 |
openstackgerrit | sahid proposed openstack/nova master: libvirt: configure trust mode for vfs https://review.openstack.org/458514 | 13:58 |
mdbooth | sfinucan johnthetubaguy mriedem: I'd really like to push this one over the line: https://review.openstack.org/#/c/383859/ . Appreciate it's a pretty complex, though. I was thinking about splitting out the code which resets on libvirt reconnection into a separate patch. I still think this code adds robustness, but splitting it out might make it simpler to review the most important part of the patch. Worth it? | 13:58 |
*** smatzek has joined #openstack-nova | 13:59 | |
mriedem | i wouldn't split it | 13:59 |
sfinucan | I wouldn't | 13:59 |
mdbooth | Glad I asked :) | 13:59 |
sfinucan | Yeah, I'm generally happy with it as is. The tests are by far the most complex piece but copious comments do help | 14:00 |
johnthetubaguy | yeah, I think I am quite close to +2 on that now, its dropped off my radar for a bit | 14:00 |
mdbooth | Yeah, the tests are a nightmare | 14:00 |
johnthetubaguy | will try taking another peak today | 14:00 |
*** shaner has quit IRC | 14:00 | |
mdbooth | Ok, thanks. I'll update with sfinucan's comments, but otherwise leave it alone. | 14:01 |
*** eharney has quit IRC | 14:01 | |
sfinucan | (y) | 14:02 |
*** ayogi has quit IRC | 14:03 | |
*** esberglu has joined #openstack-nova | 14:05 | |
*** awaugama has joined #openstack-nova | 14:05 | |
*** CristinaPauna has joined #openstack-nova | 14:09 | |
*** tbachman has joined #openstack-nova | 14:12 | |
*** eharney has joined #openstack-nova | 14:14 | |
*** david-lyle has quit IRC | 14:14 | |
*** ngupta has quit IRC | 14:17 | |
*** ngupta has joined #openstack-nova | 14:17 | |
*** zz_dimtruck is now known as dimtruck | 14:18 | |
*** gouthamr has joined #openstack-nova | 14:19 | |
*** Jeffrey4l has quit IRC | 14:19 | |
*** ijw has joined #openstack-nova | 14:20 | |
traskat | hey, what i have to do to apply extra specs to running VMs for examaple disk IO limitations without restart the VM? | 14:21 |
*** slaweq has joined #openstack-nova | 14:22 | |
*** claudiub has joined #openstack-nova | 14:23 | |
*** fragatina has quit IRC | 14:23 | |
*** shaner has joined #openstack-nova | 14:23 | |
efried | traskat Resize? | 14:25 |
*** alexpilotti has quit IRC | 14:25 | |
efried | (I don't know the answer, just an idea.) | 14:25 |
*** ijw has quit IRC | 14:25 | |
*** alexpilotti has joined #openstack-nova | 14:26 | |
*** lucasxu has joined #openstack-nova | 14:26 | |
*** dikonoor has quit IRC | 14:26 | |
*** slaweq has quit IRC | 14:27 | |
traskat | efried, you mean to another flavor ? | 14:29 |
efried | traskat I would think so. Where else would you get the new extra_specs? | 14:29 |
efried | traskat Prolly depends on the driver whether the resize can happen without a reboot... | 14:30 |
efried | But then, the extra_specs will be driver-specific anyway, não é? | 14:30 |
*** abalutoiu has joined #openstack-nova | 14:32 | |
*** kfarr has joined #openstack-nova | 14:34 | |
openstackgerrit | Mathieu Gagné proposed openstack/nova master: Add ability to signal and perform online volume size change https://review.openstack.org/454322 | 14:39 |
imacdonn | mdbooth, around ? | 14:40 |
mdbooth | imacdonn: Yep | 14:40 |
mriedem | takashin: alex_xu: ok did you want to talk about the server migrations API? | 14:41 |
imacdonn | hi! Could you take a peek at https://bugs.launchpad.net/nova/+bug/1683972 ? | 14:41 |
openstack | Launchpad bug 1662483 in OpenStack Compute (nova) "duplicate for #1683972 detach_volume races with delete" [Undecided,In progress] - Assigned to Matthew Booth (mbooth-9) | 14:41 |
mriedem | or wait for me to write something up? | 14:41 |
alex_xu | mriedem: yea | 14:41 |
takashin | mriedem: yes. | 14:41 |
alex_xu | I just didn't remember why we only show in-progress initially | 14:41 |
*** kevinz has quit IRC | 14:41 | |
mriedem | alex_xu: because the api was written solely to cancel in-progress live migratoins | 14:41 |
mriedem | *migrations | 14:41 |
mdbooth | imacdonn: Looking | 14:41 |
imacdonn | mdbooth, thanks | 14:41 |
takashin | alex_xu: There is os-migrations API. | 14:42 |
*** psachin has quit IRC | 14:42 | |
takashin | alex_xu: We can use it to list all migrations. | 14:42 |
mriedem | alex_xu: so rather than just having to validate when a request is made to cancel a live migration, we filtered out the migrations you could pick from to do the cancel | 14:42 |
*** dgonzalez_ has joined #openstack-nova | 14:43 | |
*** mkoderer_ has joined #openstack-nova | 14:43 | |
takashin | mriedem: Yes. To cancel a live migration. | 14:43 |
mriedem | https://github.com/openstack/nova/blob/master/nova/api/openstack/compute/migrations.py#L62 | 14:44 |
mriedem | so before 2.23, we can list all migrations, but we don't expose the type | 14:44 |
mdbooth | imacdonn: Fun | 14:44 |
mriedem | because add_link defaults to False | 14:44 |
alex_xu | mriedem: that works for me. I guess initialy we think there is only one migration, and we didn't want to trouble with user to find out which migration is in-progress. | 14:44 |
imacdonn | mdbooth :) indeed | 14:44 |
*** dgonzalez_ has quit IRC | 14:45 | |
*** mkoderer_ has quit IRC | 14:45 | |
alex_xu | with filter, we can reach the same goal | 14:45 |
mriedem | at microversion >= 2.23 with os-migrations, we provide a link and migration_type in the response for an in-progress live migration, otherwise just the migration_type | 14:45 |
mriedem | alex_xu: note that os-migrations also takes a filter parameter | 14:45 |
mriedem | https://github.com/openstack/nova/blob/master/nova/api/openstack/compute/migrations.py#L77 | 14:45 |
alex_xu | so i'm ok with that. and the user have one more way to query all the migraitons for a specific servers, so nothing bad at here | 14:45 |
mriedem | it's not validated, but it's there | 14:45 |
openstackgerrit | Dan Smith proposed openstack/nova master: DNM: Canary test against multi-cell devstack https://review.openstack.org/458537 | 14:46 |
mdbooth | imacdonn: Let me think about that for a bit. lyarwood may also be interested in https://bugs.launchpad.net/nova/+bug/1683972 | 14:46 |
openstack | Launchpad bug 1662483 in OpenStack Compute (nova) "duplicate for #1683972 detach_volume races with delete" [Undecided,In progress] - Assigned to Matthew Booth (mbooth-9) | 14:46 |
alex_xu | takashin: what would you think? | 14:47 |
mriedem | alex_xu: this is how os-migrations gets the migrations from the db + filters https://github.com/openstack/nova/blob/master/nova/db/sqlalchemy/api.py#L4789 | 14:47 |
imacdonn | mdbooth, OK, will check that one out too. I think my problem is a bit more extensive than 1683972, but maybe we can find a solution that covers both | 14:47 |
mdbooth | imacdonn: It's the same bug, just repeating for lyarwood | 14:48 |
*** marst has quit IRC | 14:48 | |
mdbooth | I just unlinked the bug it was marked as a dup of, btw | 14:48 |
mriedem | alex_xu: so technically you could get all migration types via os-migrations v2.1 and filter on instance_uuid | 14:48 |
imacdonn | mdbooth, oh, duh .. still working on my first coffee :) | 14:48 |
traskat | efried, are there other options then changing the flavor. i tried now a reboot but that didn't work either. the flavor i use for the rebooted instance has the extra specs before. | 14:48 |
takashin | alex_xu: So I would like to make new APIs to show in-progress migrations only. | 14:48 |
mdbooth | I don't think they're the same, although they're cousins | 14:48 |
imacdonn | mdbooth, agreed | 14:48 |
zioproto | dansmith: I upgraded to 14.0.5 but there are still some strange different messages https://pastebin.com/vLNWdiRZ | 14:49 |
mriedem | takashin: why would you rather filter all migrations and return only in-progress migrations without any option for the user to choose the filtering? | 14:49 |
zioproto | looks like a problem with the keypairs | 14:50 |
efried | traskat Sorry, I'm not familiar enough with this to be of help. The resize was just an off-the-cuff idea. But you probably want to take this discussion to the #openstack channel (see channel topic). | 14:50 |
traskat | ok thanks | 14:50 |
mriedem | takashin: that seems inconsistent behavior with the rest of our apis, which if you list resources, you get undeleted resources in their current state, | 14:50 |
mriedem | you can take actions on those resources, and those actions may be invalid for the state that the resource is in, | 14:51 |
mriedem | in which case, the api will tell you that with a 400 response | 14:51 |
dansmith | zioproto: hmm | 14:51 |
takashin | mriedem: Because we have os-migrations API and there should be a difference between new APIs and old one. | 14:51 |
lyarwood | mdbooth: thanks, I missed that you had a change up to add locking to detach_volume in the compute layer somehow | 14:51 |
zioproto | dansmith: Some instances are still missing keypair information. Unable to run keypair migration at this time. | 14:51 |
dansmith | zioproto: yeah, but the problem is the "error attempting to run ..." one | 14:52 |
dansmith | zioproto: just a sec | 14:52 |
mdbooth | lyarwood: Not related to this issue. I should probably abandon that patch as it's more complex than I was going for. | 14:52 |
mriedem | takashin: so if i understand your point, if i want to list all migrations, regardless of type or status, i should use os-migrations | 14:52 |
*** CristinaPauna has quit IRC | 14:52 | |
alex_xu | takashin: /servers/{id}/migrations is more easy for user to query migration for specific server | 14:52 |
mriedem | takashin: but if i want to list migrations for a specific server in order to cancel the migration, i should use /servers/server_id/migrations | 14:52 |
zioproto | dansmith: I wait no problem :) | 14:52 |
*** tbachman has quit IRC | 14:53 | |
takashin | mriedem: alex_xu: Yes. In specific state. | 14:53 |
mriedem | takashin: the thing that worries me about saying just use os-migrations, is the docs have a warning on it https://developer.openstack.org/api-ref/compute/#migrations-os-migrations-frozen | 14:53 |
alex_xu | takashin: From the ML, operators have use-case to query specific server's migrations. so | 14:53 |
mriedem | basically saying, don't use this, use the other API | 14:53 |
lyarwood | mdbooth: is it not? wouldn't locking against instance.uuid for both attach and detach help here? | 14:54 |
dansmith | zioproto: well, I was looking to see if we had any updates to that migration but I don't think we have | 14:54 |
*** bkopilov has quit IRC | 14:54 | |
mriedem | alex_xu: argubally they could, i.e. GET /os-migrations?instance_uuid=1234 | 14:54 |
dansmith | zioproto: you can make a code tweak to dump more info since this is a test right? | 14:54 |
mdbooth | lyarwood: Don't think so. This is racing with cinder internals. | 14:54 |
*** prateek has quit IRC | 14:54 | |
imacdonn | lyarwood, mdbooth, my problem is that the iSCSI connection termination is separate from detach | 14:54 |
*** marst has joined #openstack-nova | 14:55 | |
mdbooth | imacdonn: If it rescans again before the lun has been reallocated, does the device go away? | 14:55 |
dansmith | zioproto: can you do this and see what it shows us? https://pastebin.com/3RJdsJE1 | 14:55 |
lyarwood | imacdonn: it shouldn't be anymore, we've moved everything into the bdm.detach() call | 14:55 |
alex_xu | mriedem: but more straight forward with /servers/{id}/migrations? but actually we should think about more how the API user write the client code | 14:55 |
lyarwood | imacdonn: so locking that against instance.uuid should help avoid this now unless I'm missing something | 14:56 |
mdbooth | lyarwood: terminate connection isn't atomic, right? | 14:56 |
mdbooth | lyarwood: So on completion of the call to terminate connection, the connection hasn't necessarily been terminated? | 14:56 |
mdbooth | atomic is the wrong word. Synchronous. | 14:56 |
lyarwood | mdbooth: yeah good point | 14:56 |
mriedem | alex_xu: we actually have CLIs for these in novaclient already | 14:56 |
gibi | mriedem: hi! I'm sorry for missed the nova meeting, today is a pretty crazy day. Thanks for giving the subteam update. | 14:56 |
mriedem | gibi: np | 14:57 |
imacdonn | lyarwood, mdbooth, terminate connection is NOT part of bdm.detach, though | 14:57 |
mdbooth | lyarwood: That's a question, btw. Do we know for sure that is/isn't the case? | 14:57 |
mriedem | alex_xu: nova migration-list [--host] [--status] | 14:57 |
*** kevinz has joined #openstack-nova | 14:57 | |
mdbooth | imacdonn: If it was synchronous it would at least be a lever we could use to fix the problem. | 14:57 |
*** amotoki has quit IRC | 14:57 | |
mriedem | alex_xu: we don't allow listing migrations via os-migrations and filtering on instance uuid from the CLI | 14:57 |
mriedem | we could certainly add that | 14:57 |
mdbooth | If it's async this could be messy. | 14:57 |
* mdbooth strongly suspects it's async. | 14:58 | |
mriedem | alex_xu: then we also have nova server-migration-list <server> | 14:58 |
mriedem | which is using /servers/server_id/migrations | 14:58 |
mriedem | from a CLI perspective, it's very confusing | 14:58 |
zioproto | dansmith: https://pastebin.com/n0THHGSG | 14:58 |
imacdonn | mdbooth, you mean like a "callback when it's done" thing ? | 14:58 |
dansmith | zioproto: interesting | 14:58 |
mdbooth | I mean: don't return control until it's done | 14:58 |
zioproto | dansmith: my production database is always interesting :( | 14:59 |
dansmith | zioproto: no, this is super helpful | 14:59 |
mdbooth | If nova knew that immediately after terminate connection returned the connection was definitely already be gone, we could use that | 14:59 |
imacdonn | mdbooth, I believe that's how it is already ... cinderclient waits for the response to an HTTP request | 14:59 |
lyarwood | imacdonn: sorry, I'm talking about the driver bdm - https://github.com/openstack/nova/blob/master/nova/virt/block_device.py#L294-L352 | 14:59 |
*** adisky_ has quit IRC | 14:59 | |
mriedem | alex_xu: takashin: so maybe we just leave /servers/server_id/migrations as doing the server-side in-progress filtering as it does today, and expand the "nova migration-list" CLI to allow filtering on instance uuid too | 14:59 |
zioproto | dansmith: I have upgrade date on Saturday 29th of April :) Do you think I can find a fix by then ? :) | 14:59 |
mriedem | and clarify the help text in the api ref docs and CLI help text | 15:00 |
dansmith | zioproto: yep | 15:00 |
mriedem | alex_xu: takashin: part of the problem is we just don't document os-migrations very well in the api-ref | 15:00 |
zioproto | dansmith: I have the feeling everytime I chat with you a new Nova release comes out :D | 15:00 |
dansmith | zioproto: heh | 15:00 |
mdbooth | lyarwood kashyap: Any idea what the status of qemu iscsi multipath is? | 15:01 |
lyarwood | mdbooth: yeah looking at the cinderv2 code it looks synchronous | 15:01 |
* kashyap reads | 15:01 | |
mdbooth | lyarwood: Got a link? | 15:01 |
zioproto | I am building the ubuntu packages for 14.0.5 ... but I will wait a 14.0.6 if something still needs to be changed for the database migrations | 15:01 |
*** aarefiev is now known as aarefiev_afk | 15:01 | |
*** rcernin has quit IRC | 15:01 | |
lyarwood | mdbooth: AFAIK it's in and we have some config support for it in the libvirt driver but it's not wired up at the moment | 15:01 |
dansmith | zioproto: so I'm going to give you another patch to try and see if you get farther, but it's not the fix I want, so I'll ask you to test the real fix as well, okay? | 15:01 |
lyarwood | mdbooth: https://github.com/openstack/cinder/blob/master/cinder/api/contrib/volume_actions.py#L192-L208 | 15:01 |
mriedem | lyarwood: mdbooth: terminate_connection in cinder is an RPC call to the cinder-volume backend | 15:01 |
zioproto | dansmith: sure ! | 15:01 |
mdbooth | lyarwood: Sweet, because the real solution here is to stop doing this and use qemu. | 15:02 |
mriedem | lyarwood: that's not what you want | 15:02 |
alex_xu | mriedem: at least that match the requirement from the operator | 15:02 |
mriedem | lyarwood: https://github.com/openstack/cinder/blob/master/cinder/volume/api.py#L749 | 15:02 |
*** kevinz has quit IRC | 15:02 | |
mriedem | https://github.com/openstack/cinder/blob/master/cinder/volume/rpcapi.py#L217 | 15:02 |
mriedem | alex_xu: ok so i guess i'll just go with that then, and unblock takashin | 15:03 |
*** gongysh has joined #openstack-nova | 15:03 | |
alex_xu | mriedem: thanks | 15:03 |
mriedem | i can fix the os-migrations api-ref docs (i've been meaning to do that for awhile now), | 15:03 |
mdbooth | mriedem: It's a call, which is nice. | 15:03 |
lyarwood | mriedem: yeah I was just getting there, thanks for the links :) | 15:03 |
takashin | mriedem: alex_xu: Thank you. | 15:03 |
mriedem | and add --server as a filter option to nova migration-list | 15:03 |
mriedem | mdbooth: well normally a block rpc call from the api is not nice | 15:03 |
mriedem | *blocking | 15:03 |
dansmith | zioproto: https://pastebin.com/yHueQi4s | 15:04 |
*** mlakat has quit IRC | 15:05 | |
alex_xu | mriedem: I can help you on those | 15:05 |
mdbooth | mriedem: Nice for me :) | 15:05 |
mdbooth | But yeah | 15:05 |
openstackgerrit | Matt Riedemann proposed openstack/python-novaclient master: Microversion 2.42 - Fix tag attribute disappearing https://review.openstack.org/429512 | 15:05 |
mriedem | sdague: another one if you have time https://review.openstack.org/#/c/429512/ - unblocks novaclient for new microversions in pike | 15:05 |
*** moshele has quit IRC | 15:05 | |
*** armax has joined #openstack-nova | 15:06 | |
zioproto | dansmith: should I revert the previous patch with the 'raise' ? | 15:06 |
dansmith | zioproto: no | 15:06 |
dansmith | zioproto: the raise lets us see if I typo'd this one :P | 15:07 |
*** yingwei has joined #openstack-nova | 15:07 | |
zioproto | dansmith: https://pastebin.com/eLsiFn9W | 15:07 |
*** Oku_OS is now known as Oku_OS-away | 15:07 | |
kashyap | mdbooth: So, no, upstream QEMU's built-in iSCSI initiator doesn't support multipathing yet | 15:08 |
kashyap | mdbooth: And I haven't seen any related to it for over 2 years | 15:08 |
dansmith | zioproto: hmm, that ignoring line should mean we're skipping the thing that gives us the error | 15:09 |
dansmith | zioproto: I hate to ask, but did you apply with patch or paste it in by hand? | 15:09 |
mdbooth | kashyap: Sucks :( | 15:09 |
zioproto | I pasted by hand ! | 15:09 |
dansmith | zioproto: any chance you missed the 'continue' line under the log? | 15:10 |
dansmith | zioproto: L35 in the paste | 15:10 |
lyarwood | kashyap / mdbooth ; AFAIK the lack of multipath support is the reason it was never wired up in Nova | 15:10 |
*** felipemonteiro__ has joined #openstack-nova | 15:10 | |
zioproto | and you are right ! | 15:10 |
mdbooth | lyarwood: Right. | 15:10 |
kashyap | mdbooth: Hey, on the brighter hand, QEMU 2.9 just released, with some "robust" improvements | 15:10 |
*** felipemonteiro has quit IRC | 15:11 | |
*** udesale has quit IRC | 15:11 | |
zioproto | dansmith: https://pastebin.com/LeH3HudT | 15:11 |
zioproto | sorry for missing the 'continue' | 15:12 |
*** ayogi has joined #openstack-nova | 15:12 | |
openstackgerrit | STEW TY proposed openstack/nova master: Transform instance.unrescue notifications https://review.openstack.org/388275 | 15:12 |
dansmith | zioproto: no problem :) | 15:12 |
dansmith | zioproto: okay, so you have a bunch of instances with keypairs that have since been deleted... | 15:13 |
*** david-lyle has joined #openstack-nova | 15:13 | |
mriedem | efried: one new question from me inline https://review.openstack.org/#/c/438598/12 | 15:13 |
zioproto | also the instances have been deleted I guess | 15:13 |
*** takashin has left #openstack-nova | 15:13 | |
efried | mriedem Ack, looking... | 15:13 |
openstackgerrit | Steve Noyes proposed openstack/nova master: Update detach to use V3 Cinder API https://review.openstack.org/438750 | 15:13 |
zioproto | dansmith: I think we are talking about the instances that are marked as deleted | 15:13 |
*** hongbin has joined #openstack-nova | 15:13 | |
zioproto | dansmith: are you going to submit this patch for review in master ? | 15:14 |
dansmith | zioproto: no, let me get you an example | 15:14 |
mriedem | efried: same question about exception handling in power_off | 15:14 |
mriedem | we try to avoid handling Exception if possible | 15:14 |
dansmith | zioproto: 2017-04-20 15:11:10.584 787 WARNING nova.objects.instance [req-d1cbe1e7-1f03-46b6-9009-bba793eb73ba - - - - -] Instance b750d102-4fbb-4401-80b2-cc36888c9749 keypair c_rally_5aabcf4c_Xg4N7es6 not found | 15:14 |
*** alexpilo_ has joined #openstack-nova | 15:14 | |
*** alexpilotti has quit IRC | 15:14 | |
*** pcaruana has quit IRC | 15:14 | |
efried | mriedem Roger that. Responding. | 15:14 |
dansmith | zioproto: yeah, we'll fix in master and backport | 15:14 |
sdague | mriedem: so, given that powervm driver is pretty self contained, is there particular primary concerns before merging? | 15:15 |
*** nkorabli has joined #openstack-nova | 15:15 | |
bauzas | oh man, /me needs to get drugs in order to think about https://review.openstack.org/#/c/437424/ | 15:15 |
bauzas | let's prepare a new battle | 15:15 |
zioproto | dansmith: ok, please give me the review number, I will cherry-pick this patch in my deployment or I cant upgrade next week | 15:15 |
openstackgerrit | Roman Podoliaka proposed openstack/osc-placement master: CLI for resource providers https://review.openstack.org/457532 | 15:15 |
openstackgerrit | Roman Podoliaka proposed openstack/osc-placement master: CLI for inventories https://review.openstack.org/457533 | 15:15 |
openstackgerrit | Roman Podoliaka proposed openstack/osc-placement master: CLI for allocations https://review.openstack.org/457534 | 15:15 |
openstackgerrit | Roman Podoliaka proposed openstack/osc-placement master: CLI for usages https://review.openstack.org/457535 | 15:15 |
openstackgerrit | Roman Podoliaka proposed openstack/osc-placement master: tests: add a hook for functional testing in the gate https://review.openstack.org/452122 | 15:15 |
dansmith | zioproto: yeah, but we need to get the problem fully fixed first :) | 15:15 |
mriedem | sdague: i had questions in the 2nd patch | 15:15 |
zioproto | I am available to test more patches :) | 15:16 |
*** lucasxu has quit IRC | 15:16 | |
zioproto | dansmith: should I open a bug on LP ? | 15:16 |
dansmith | zioproto: yep, working on a new patch | 15:16 |
dansmith | zioproto: yeah, please do file a bug, we'll need it for the review | 15:17 |
*** lucasxu has joined #openstack-nova | 15:17 | |
efried | sdague Not sure if you've been aware, but FYI we have a CI running (a subset of) tempest on the in-tree patches, and publishing to gerrit. | 15:17 |
sdague | efried: while you are changing things, I think you misunderstand LOG.exception(e) :) | 15:18 |
efried | sdague Entirely possible ;-) | 15:18 |
efried | Please enlighten me. | 15:18 |
imacdonn | mdbooth, you asked earlier if a rescan would make the device(s) go away .. if I'm interpreting your question properly, the answer is "no" ... rescan picks up new LUNs, but doesn't do anything about ones that have become unresponsive | 15:19 |
efried | Thought it was LOG.error + stack trace | 15:19 |
sdague | efried: right | 15:19 |
mdbooth | imacdonn: Ok, thanks | 15:19 |
sdague | efried: comment left | 15:19 |
efried | sdague Thanks. | 15:19 |
*** kristian__ has joined #openstack-nova | 15:20 | |
lyarwood | mdbooth / imacdonn ; https://bugzilla.redhat.com/show_bug.cgi?id=1367805 is a dup of this that I could never reproduce correctly FWIW | 15:20 |
openstack | bugzilla.redhat.com bug 1367805 in openstack-nova "Previously removed iSCSI multipath volumes can be rediscovered by future disconnect_volume calls before being unmapped on the backend by os-terminate_connection." [High,Closed: insufficient_data] - Assigned to lyarwood | 15:20 |
mdbooth | imacdonn lyarwood: So what I think we need to achieve here is that after calling disconnect nothing is subsequently allowed to connect until the associated terminate connection has defintely resulted in the device having been killed by cinder | 15:21 |
imacdonn | lyarwood, \o/ I'm not alone ;) | 15:21 |
*** ijw has joined #openstack-nova | 15:21 | |
openstackgerrit | STEW TY proposed openstack/nova master: Transform instance.unrescue notifications https://review.openstack.org/388275 | 15:22 |
lyarwood | mdbooth: yes and the locking in your change would do that now | 15:22 |
*** sridharg has quit IRC | 15:22 | |
imacdonn | mdbooth, right .. and I've sortof hacked that by putting a lock on things that cover that window .. like _shutdown_instance() in compute.manager | 15:22 |
imacdonn | mdbooth, and the same lock on connect_volume() in virt.libvirt.volume.iscsi | 15:23 |
*** alexpilotti has joined #openstack-nova | 15:23 | |
efried | sdague So I don't need to pass the exception itself to LOG.exception - it picks it up automatically? | 15:23 |
*** mdnadeem has quit IRC | 15:23 | |
mriedem | efried: yeah | 15:23 |
efried | Beautiful, thanks. | 15:23 |
mriedem | LOG.exception() just takes a message if you want to say something | 15:23 |
mriedem | and it dumps the current stacktrace | 15:23 |
mdbooth | lyarwood: Assuming we can verify the sync behaviour of terminate connection all the way down | 15:23 |
sdague | yeh, it's probably worth logging the instance there, I put a suggested string in a comment right now | 15:24 |
imacdonn | mdbooth, lyarwood AFAIK, it's sync as far as nova-compute is concerned - it's making an API call to cinder ... it's an RPC call within cinder, but nova-compute doesn't know about that | 15:24 |
*** rmart04 has quit IRC | 15:24 | |
zioproto | dansmith: https://bugs.launchpad.net/nova/+bug/1684861 | 15:25 |
openstack | Launchpad bug 1684861 in OpenStack Compute (nova) "Database migrations from mitaka to newton fail due to missing keypairs" [Undecided,New] | 15:25 |
*** zenoway has quit IRC | 15:25 | |
zioproto | wait the title is wrong, I just realized | 15:25 |
dansmith | zioproto: thanks, still working on the fix if you can hang on for a bit | 15:25 |
zioproto | https://bugs.launchpad.net/nova/+bug/1684861 | 15:26 |
*** alexpilo_ has quit IRC | 15:26 | |
zioproto | I updated the title | 15:26 |
imacdonn | mdbooth, lyarwood If those calls were not sync, we'd have other problems .. like we'd ask cinder to connect a volume, then do the rescan "too early" | 15:26 |
zioproto | if you link the fix to the bug I will track it | 15:26 |
*** ijw has quit IRC | 15:26 | |
mdbooth | imacdonn: I've learned not to assume these things :) | 15:27 |
imacdonn | hah ;) | 15:27 |
dansmith | zioproto: it looks to me like you have something that shouldn't be allowed with the foreign key constraint | 15:27 |
dansmith | zioproto: have you disabled those constraints on your database by chance? | 15:27 |
zioproto | how can I check ? | 15:27 |
zioproto | it is possible something happened in this database since Icehouse | 15:27 |
zioproto | is possible we messed up something in the upgrade Juno to Kilo if I remember correctly | 15:28 |
dansmith | zioproto: well, can you check that you have something in instance_extra with instance_uuid=e7eb4196-994e-4968-bcf0-aa6f783bbb18, but nothing in instances.uuid for that? | 15:28 |
*** voelzmo has quit IRC | 15:29 | |
dansmith | zioproto: are you using a hand-rolled db archiving script or something? | 15:29 |
dansmith | zioproto: something like this maybe? https://tecadmin.net/temporarily-disable-foreign-key-checks-in-mysql/ | 15:29 |
*** slaweq has joined #openstack-nova | 15:29 | |
*** zenoway has joined #openstack-nova | 15:30 | |
*** coreywright has quit IRC | 15:31 | |
zioproto | dansmith: yes I have data in instance_extra but not in instances | 15:31 |
zioproto | I know what happened | 15:31 |
dansmith | zioproto: that shouldn't be possible | 15:31 |
zioproto | we were upgrading to Kilo to Liberty | 15:31 |
*** Jeffrey4l has joined #openstack-nova | 15:31 | |
zioproto | and when doing the prod update we had a problem with the database migration | 15:31 |
zioproto | and we had to do some fast operations on the database | 15:31 |
*** vks1 has joined #openstack-nova | 15:32 | |
zioproto | it is possible that we deleted some very old instances | 15:32 |
sdague | efried: is there a reason why this goes fatal - https://review.openstack.org/#/c/391288/51/nova/virt/powervm/vm.py@405 | 15:32 |
zioproto | and we disabled the foreigh checks to do that | 15:32 |
*** artom has quit IRC | 15:32 | |
sdague | because I don't see any higher level error recovery for that exception | 15:32 |
dansmith | zioproto: ah, okay that would explain it | 15:32 |
*** artom has joined #openstack-nova | 15:32 | |
*** moshele has joined #openstack-nova | 15:33 | |
zioproto | dansmith: we probably dropped a lot of stuff from table instances to make a kilo to liberty database migration go through, while we were in maintanance window | 15:33 |
zioproto | that is why now we improved and we do these migrations before | 15:33 |
zioproto | dansmith: so this means we behaved bad and we dont get a patch that fixes the problem ?? :| | 15:33 |
efried | sdague If we got there, somebody passed an extra spec with a 'powervm:' prefix, but that we don't recognize. | 15:33 |
efried | We definitely want to fail the deploy, nah? | 15:34 |
dansmith | zioproto: okay, I'm trying to decide if we should try to handle that possibility in the mainstream code or not.. because technically you broke the data model :) | 15:34 |
*** artom has quit IRC | 15:34 | |
*** artom has joined #openstack-nova | 15:34 | |
*** zenoway has quit IRC | 15:34 | |
*** annegentle has joined #openstack-nova | 15:34 | |
*** satyar has joined #openstack-nova | 15:35 | |
dansmith | zioproto: I'm leaning towards catching the problem, logging it, but making you do your own cleanup to fix it | 15:35 |
dansmith | zioproto: which would just mean deleting the orphaned instance_extra records | 15:35 |
dansmith | mriedem: you following this at all? | 15:35 |
efried | sdague I *suppose* an argument could be made for ignoring, so if we add a new spec in a later release, you can still use a flavor containing it on an older compute node... | 15:35 |
efried | thorst thoughts? | 15:35 |
*** tbachman has joined #openstack-nova | 15:35 | |
*** vks1 has quit IRC | 15:36 | |
thorst | efried: I just logged a comment. I think its sufficient to just log a warning saying 'Unknown' or something | 15:36 |
sdague | efried: maybe? though by throwing an attribute error, it's not super clear to me where things go into sane recovery to the user to explain to them what went wrong | 15:36 |
*** alexpilo_ has joined #openstack-nova | 15:36 | |
thorst | but I agree...full stop, meh | 15:36 |
zioproto | dansmith: of course all other operators that asked my how I upgraded from kilo to liberty did the same. | 15:36 |
thorst | warning tells them that something came in you didn't understand. | 15:36 |
dansmith | zioproto: yeah, I'm sure you're not the only one | 15:36 |
sdague | efried: I guess the real question is who can solve the problem | 15:36 |
efried | thorst ...IF they look at the log. | 15:36 |
mriedem | dansmith: not a word | 15:36 |
sdague | is it someone that can change their API request? in which case tell them over the API channel how to fix it | 15:37 |
*** moshele has quit IRC | 15:37 | |
sdague | but if only admins can fix it, then log warning seems better | 15:37 |
dansmith | mriedem: so, zioproto disabled FK constraints in kilo, deleted a bunch of instances, and then re-enabled them, which orphaned some instance extra records, which now don't behave properly since they assume they have a FK to instances | 15:37 |
efried | sdague thorst I guess if they misspell a keyword, they'll not get the corresponding behavior. Which they may not notice for... a while. And once they do, they'll have to look at the log to figure out why that happened. | 15:37 |
dansmith | mriedem: which is why their migration is failing | 15:38 |
openstackgerrit | Balazs Gibizer proposed openstack/nova master: add tags field to instance.update notification https://review.openstack.org/407228 | 15:38 |
dansmith | mriedem: handling it is a little tricky, so I'm leaning towards catching the error, logging it and the uuid so they can manually clean the affected instance extra records, but not actually try to magically handle this, | 15:38 |
dansmith | mriedem: in case it's actually for some other reason | 15:38 |
mriedem | why were the FK constraints disabled? | 15:38 |
*** alexpilo_ has quit IRC | 15:39 | |
*** alexpilotti has quit IRC | 15:39 | |
dansmith | mriedem: especially since the way you get here is breaking the model | 15:39 |
dansmith | mriedem: dunno, they probably had stale/corrupt data and wanted to delete some broken instances, and didn't realize they should have deleted the things depending on the instance records instead of disabling FKs | 15:39 |
*** alexpilotti has joined #openstack-nova | 15:39 | |
mdbooth | lyarwood imacdonn: So this call isn't synchronous unfortunately, or at least it's driver dependent and for zfssa iscsi it's not sychronous | 15:39 |
*** lucasxu has quit IRC | 15:39 | |
mriedem | dansmith: ummm | 15:40 |
mriedem | it seems weird to put something into code for a thing you shouldn't have done in the first place, | 15:40 |
openstackgerrit | Pavel Gluschak (scsnow) proposed openstack/nova master: VStorage: changed default log path https://review.openstack.org/458557 | 15:40 |
mriedem | and if you did, then you (as the person that did it) patch the fix on your end | 15:40 |
dansmith | mriedem: well, right, so that's why I just want to log/explain and be graceful | 15:40 |
imacdonn | mdbooth lyarwood: where do you see that ? | 15:40 |
mriedem | even that is odd | 15:40 |
mdbooth | The cinder api makes a synchronous rest call to c-vol, which synchronously makes a rest call to zfssa | 15:40 |
dansmith | mriedem: not fix the problem, but just warn about the affected uuid(s) | 15:40 |
mriedem | "someone reported they did this odd thing they shouldn't have, so we're going to handle and log it for them" | 15:40 |
mdbooth | However, the rest call explicitly isn't synchronous | 15:41 |
zioproto | dansmith: I have the proof https://bugs.launchpad.net/nova/+bug/1511466/comments/9 | 15:41 |
openstack | Launchpad bug 1511466 in OpenStack Compute (nova) "migrate flavor data impossible" [Medium,Confirmed] | 15:41 |
mriedem | dansmith: i know, i'm saying it's weird | 15:41 |
dansmith | mriedem: right now they just get "error, failed, kthx" | 15:41 |
zioproto | dansmith: somebody wrote on LP to do that, and I did it ! | 15:41 |
mdbooth | The driver waits for the return, but the return it's looking for is ACCEPTED | 15:41 |
dansmith | zioproto: do you do everything random people tell you to do? nevermind, don't answer that :P | 15:41 |
*** lucasxu has joined #openstack-nova | 15:41 | |
mdbooth | Which doesn't indicate the requested change has happened yet, only that it has been requested | 15:41 |
mriedem | dansmith: so are people doing this because of some latent bug that forced them to do it, but we have since fixed? | 15:41 |
dansmith | mriedem: well, yeah, but especially if it was advice on a LP bug that multiple people followed, :/ | 15:41 |
imacdonn | mdbooth lyarwood OK, yeah, but doesn't cinder-api wait for the response from cinder-volume before responding to the (cinder) API request? | 15:42 |
*** eharney has quit IRC | 15:42 | |
dansmith | mriedem: it doesn't matter, disabling FKs isn't the right workaround for a problem anyway, it's deleting extra, then deleting instance, etc | 15:42 |
*** annegentle has quit IRC | 15:42 | |
mdbooth | It does, yes, but that response only indicates that zfssa has accepted the request | 15:42 |
mriedem | dansmith: i agree | 15:42 |
*** lucasxu has quit IRC | 15:42 | |
mdbooth | It doesn't indicate that the request has been processed | 15:42 |
mriedem | which is why it's odd to put something into code that is for "people worked around this incorrectly, but we're going to patch something in for them" | 15:42 |
dansmith | mriedem: but regardless, it's out there so I at least want to log/warn and not just fail quietly with no indication about why | 15:42 |
* mdbooth expects a wide range of variation between drivers on this | 15:43 | |
imacdonn | mdbooth lyarwood: The request to the ZFSSA is another REST API .. it also waits for the operation to complete before returning | 15:43 |
*** lucasxu has joined #openstack-nova | 15:43 | |
mriedem | dansmith: where does this handle/log thing go? a migration? an object? | 15:43 |
dansmith | mriedem: well, I wasn't going to be that nice, I was going to say "Instance Extra (uuid) has no Instance, unable to migrate" | 15:43 |
*** Tahvok has joined #openstack-nova | 15:43 | |
mdbooth | imacdonn: It waits for the rest call to return an http response | 15:43 |
imacdonn | mdbooth lyarwood I know this quite well, because it times out sometimes ;) | 15:43 |
dansmith | mriedem: in the online migration function that nova-manage is calling | 15:43 |
mdbooth | But the http resonse doesn't (according to the docs) indicate that it has completed | 15:43 |
*** annegentle has joined #openstack-nova | 15:43 | |
mriedem | dansmith: as you like | 15:44 |
mriedem | i won't block it | 15:44 |
dansmith | mriedem: well, I'll put it up and you can decide then | 15:44 |
mriedem | geddy lee told me that if i choose not to decide i've still made a choice | 15:44 |
*** coreywright has joined #openstack-nova | 15:44 | |
Tahvok | I'm installing nova on ubuntu, and according to the guide I need to create the cell1 cell. In the command they say I need to run it with some sort of id? 109e1d4b-536a-40d0-83c6-5f121b82b650 | 15:44 |
imacdonn | mdbooth lyarwood empirically, I can say that it does wait for the operation to complete .... how else would the status be returned ?? | 15:44 |
Tahvok | What is that id? https://docs.openstack.org/ocata/install-guide-ubuntu/nova-controller-install.html | 15:44 |
*** zenoway has joined #openstack-nova | 15:44 | |
dansmith | mriedem: heh | 15:44 |
mdbooth | imacdonn: It returns 202 ACCEPTED | 15:44 |
mriedem | Tahvok: which command? nova-manage cell_v2 create_cell? | 15:45 |
Tahvok | mriedem: yes | 15:45 |
mriedem | su -s /bin/sh -c "nova-manage cell_v2 create_cell --name=cell1 --verbose" nova | 15:45 |
*** djohnsto has quit IRC | 15:45 | |
mriedem | that doesn't say anything about providing an id | 15:45 |
Tahvok | Oh, it's the output | 15:45 |
mriedem | it outputs an id | 15:45 |
mriedem | yes | 15:45 |
Tahvok | Yes, I thought it's part of the command | 15:45 |
Tahvok | mriedem: thanks! | 15:45 |
imacdonn | mdbooth: the ZFSSA REST API call ?? | 15:46 |
mriedem | Tahvok: also https://docs.openstack.org/developer/nova/man/nova-manage.html#nova-cells-v2 in case you need more info on the commands | 15:46 |
lyarwood | mdbooth: sorry back, isn't this a bug within this and any other driver that doesn't terminate_connection synchronously then? | 15:46 |
mdbooth | imacdonn: Yeah | 15:46 |
*** eharney has joined #openstack-nova | 15:46 | |
lyarwood | mdbooth: I skimmed over a few target drivers and they appeared to be sync | 15:46 |
mdbooth | lyarwood: Right, we're discussing zfssa, but other drivers will be different | 15:46 |
Tahvok | mriedem: is that create_cell command idempotent? | 15:46 |
imacdonn | mdbooth: that doesn't sound right ... looking for a debug log to verify | 15:47 |
Tahvok | Could I run it again, or it will try creating another cell? | 15:47 |
lyarwood | mdbooth: yeah, I think there's still value to adding the lock on our side if we also follow up with cinder to get all drivers in line there | 15:47 |
zioproto | dansmith: the bug is still open https://bugs.launchpad.net/nova/+bug/1511466 SOOO I guess any operator with a big deployment that managed to upgrade to liberty probably used this workaround. This means that soon this problem will explode :D | 15:48 |
openstack | Launchpad bug 1511466 in OpenStack Compute (nova) "migrate flavor data impossible" [Medium,Confirmed] | 15:48 |
mriedem | Tahvok: it's not | 15:48 |
mdbooth | imacdonn: That's definitely what it returns, and the documented semantics of that are that it's an async call which may not have completed. That doesn't necessarily mean that's what it's doing, of course. | 15:48 |
mriedem | Tahvok: it would fail with a duplicate entry error | 15:48 |
dansmith | zioproto: yeah, you know what you need to do to fix your data right? | 15:48 |
mriedem | Tahvok: "Returns 0 if the cell mapping was successfully created, 1 if the transport url or database connection was missing, and 2 if a cell is already using that transport url and database connection combination." | 15:48 |
Tahvok | mriedem: that's fine as long as it does not create any new records | 15:48 |
*** andreas_s has quit IRC | 15:48 | |
mriedem | Tahvok: $? would be 2 | 15:48 |
Tahvok | mriedem: thanks a lot! | 15:48 |
imacdonn | mdbooth: heh, OK ... I'm fairly sure there's nothing in the ZFSSA driver that gets a result of the operation by any means other than the response to the REST request | 15:49 |
*** salv-orl_ has joined #openstack-nova | 15:49 | |
zioproto | dansmith: so you mean that if I delete from the instance_extra table all the records where the uuid is not in instances, then I dont need any patch ? | 15:49 |
mdbooth | imacdonn: https://docs.oracle.com/cd/E51475_01/html/E52433/makehtml-id-181.html | 15:50 |
mdbooth | Except it's doing a PUT, not a POST | 15:50 |
zioproto | dansmith: I have to rush away from the office... I will ping you tomorrow about this | 15:50 |
dansmith | zioproto: yeah, that's my expectation.. let me know if it doesn't and we can see if something else is wrong too | 15:50 |
dansmith | zioproto: okay thanks | 15:50 |
imacdonn | mdbooth: Yeah, so does "successful response" mean "I've completed it"? In practice I believe it does .. so not sure why they're using 202 | 15:51 |
*** salv-orlando has quit IRC | 15:51 | |
*** felipemonteiro__ has quit IRC | 15:51 | |
mdbooth | Meh, I wonder if it's reasonable to assert this is sync and make it the cinder driver's problem if it isn't | 15:51 |
mdbooth | lyarwood: ^^^ ? | 15:52 |
imacdonn | mdbooth: in any case, I think that nova-compute has to assume that the cinder API call won't return until the work as completed .. if that's not happening, it's either a bug in cinder, or a large other can of worms (need an async mechanism between nova and cinder | 15:52 |
lyarwood | mdbooth: yeah that's my view | 15:52 |
mdbooth | lyarwood: Ok | 15:52 |
*** vks1 has joined #openstack-nova | 15:53 | |
lyarwood | mdbooth: but we should really make that assumption known to the cinder folks | 15:53 |
lyarwood | mdbooth: using this driver as an example | 15:53 |
mdbooth | Yep | 15:53 |
openstackgerrit | Dan Smith proposed openstack/nova master: Warn the user about orphaned extra records during keypair migration https://review.openstack.org/458564 | 15:53 |
dansmith | mriedem: this is what I was thinking ^ | 15:53 |
*** zenoway has quit IRC | 15:53 | |
dansmith | mriedem: I'll mark the bug as invalid and also comment on the original bug warning people that they eff'd themselves | 15:54 |
*** chyka has joined #openstack-nova | 15:54 | |
*** zenoway has joined #openstack-nova | 15:54 | |
zioproto | dansmith: with this patch https://review.openstack.org/#/c/458564 I should be able to go on without cleaning the db, right ? | 15:54 |
zioproto | it is the patch I applied manually, right? | 15:54 |
*** zenoway has quit IRC | 15:55 | |
dansmith | zioproto: no | 15:55 |
dansmith | zioproto: it's just a warning | 15:55 |
*** zenoway has joined #openstack-nova | 15:55 | |
dansmith | zioproto: it won't fix the problem, which will make the second keypair migration unable to proceed | 15:55 |
dansmith | zioproto: you have to clean the database | 15:55 |
sfinucan | vladikr, sahid: Little cleanup here, if you could take a gawk https://review.openstack.org/#/c/417378/ | 15:56 |
vladikr | sfinucan, sure | 15:57 |
*** felipemonteiro has joined #openstack-nova | 15:58 | |
*** amotoki has joined #openstack-nova | 15:58 | |
openstackgerrit | Matt Riedemann proposed openstack/nova master: Deprecate os-hosts API https://review.openstack.org/456504 | 15:58 |
*** dtp has joined #openstack-nova | 15:58 | |
mriedem | alex_xu: ^ fixed that unit test | 15:59 |
*** zenoway has quit IRC | 15:59 | |
*** david-lyle has quit IRC | 15:59 | |
*** mpaolino_ has quit IRC | 15:59 | |
*** afazekas_ is now known as afazekas | 16:00 | |
mdbooth | lyarwood: My patch won't fix this, btw, because it needs to be global: https://review.openstack.org/#/c/430213/2/nova/compute/manager.py | 16:00 |
*** mdrabe has quit IRC | 16:00 | |
*** ig0r_ has quit IRC | 16:01 | |
*** liverpooler has quit IRC | 16:01 | |
mdbooth | lyarwood: When using iscsi, I think all connect and disconnect calls must be serialised, where disconnect also include terminate connection | 16:01 |
lyarwood | mdbooth: global in terms of the compute node right? | 16:02 |
mdbooth | lyarwood: Yes | 16:02 |
lyarwood | mdbooth: I assumed the manager was, maybe I'm missing something | 16:02 |
mdbooth | lyarwood: The lock has to be global | 16:03 |
mdbooth | In my patch it's not, it's keyed on instance uuid | 16:03 |
imacdonn | lyarwood: https://github.com/openstack/nova/blob/master/nova/compute/manager.py#L2261 | 16:03 |
*** amotoki has quit IRC | 16:03 | |
*** liverpooler has joined #openstack-nova | 16:03 | |
*** zenoway has joined #openstack-nova | 16:03 | |
mdbooth | i.e. No 2 connect/disconnect operations affecting iscsi can occur concurrently, regardless of target or instance | 16:04 |
mdbooth | But that's not true of other volume drivers, which is evil | 16:04 |
mdbooth | So unless we want to push this restriction onto all volume backends, terminate connection would have to be managed by os-brick | 16:06 |
*** dave-mccowan has quit IRC | 16:06 | |
*** mdrabe has joined #openstack-nova | 16:07 | |
mdbooth | lyarwood johnthetubaguy: ^^^ | 16:07 |
lyarwood | mdbooth: yeah sorry the cinderv3 meeting just started, I see what you mean know re global | 16:07 |
lyarwood | now* | 16:07 |
*** david-lyle has joined #openstack-nova | 16:07 | |
sdague | so, given the issues with https://bugs.launchpad.net/nova/+bug/1511466, how about bump to critical, and provide some better guidance to get through it | 16:08 |
openstack | Launchpad bug 1511466 in OpenStack Compute (nova) "migrate flavor data impossible" [Critical,Confirmed] | 16:08 |
*** zenoway has quit IRC | 16:08 | |
sdague | because it does seem like it's coming up on google hunts :) | 16:08 |
openstackgerrit | Eric Fried proposed openstack/nova master: PowerVM Driver: spawn/destroy #2: functional https://review.openstack.org/438598 | 16:09 |
efried | sdague mriedem ^^ Thanks for the review, back atcha. | 16:10 |
openstackgerrit | Chris Dent proposed openstack/nova master: Update resource tracker to PUT custom resource classes https://review.openstack.org/456915 | 16:11 |
openstackgerrit | Chris Dent proposed openstack/nova master: [placement] Idempotent PUT /resource_classes/{name} https://review.openstack.org/448791 | 16:11 |
*** zhurong has quit IRC | 16:11 | |
*** nic has joined #openstack-nova | 16:12 | |
*** traskat has quit IRC | 16:12 | |
*** dikonoor has joined #openstack-nova | 16:12 | |
openstackgerrit | Balazs Gibizer proposed openstack/nova master: Using max api version in notification sample test https://review.openstack.org/443686 | 16:15 |
*** Drankis has quit IRC | 16:15 | |
openstackgerrit | Balazs Gibizer proposed openstack/nova master: Using max api version in notification sample test https://review.openstack.org/443686 | 16:17 |
*** dave-mccowan has joined #openstack-nova | 16:21 | |
openstackgerrit | Eric Fried proposed openstack/nova master: PowerVM Driver: power_on/off and reboot https://review.openstack.org/427380 | 16:22 |
*** dave-mcc_ has joined #openstack-nova | 16:23 | |
*** ijw has joined #openstack-nova | 16:23 | |
openstackgerrit | Balazs Gibizer proposed openstack/nova master: Using max api version in notification sample test https://review.openstack.org/443686 | 16:24 |
openstackgerrit | Balazs Gibizer proposed openstack/nova master: add tags field to instance.update notification https://review.openstack.org/407228 | 16:24 |
*** dave-mccowan has quit IRC | 16:26 | |
gibi | mriedem: I update the patch that adds tags to the instance.update notification as we agreed https://review.openstack.org/#/c/407228/ | 16:26 |
*** gongysh has quit IRC | 16:26 | |
openstackgerrit | Dan Smith proposed openstack/nova master: Warn the user about orphaned extra records during keypair migration https://review.openstack.org/458564 | 16:26 |
openstackgerrit | Dan Smith proposed openstack/nova master: Make server groups api aware of multiple cells for membership https://review.openstack.org/457338 | 16:27 |
openstackgerrit | Dan Smith proposed openstack/nova master: Sort CellMappingList.get_all() for safety https://review.openstack.org/443174 | 16:27 |
openstackgerrit | Dan Smith proposed openstack/nova master: Clean up ClientRouter debt https://review.openstack.org/444487 | 16:27 |
*** ngupta has quit IRC | 16:27 | |
openstackgerrit | Dan Smith proposed openstack/nova master: DNM: Canary test against multi-cell devstack https://review.openstack.org/458537 | 16:27 |
openstackgerrit | Dan Smith proposed openstack/nova master: Add workaround to disable group policy check upcall https://review.openstack.org/442736 | 16:27 |
*** ijw has quit IRC | 16:29 | |
openstackgerrit | Eric Fried proposed openstack/nova master: PowerVM Driver: spawn/destroy #3: TaskFlow https://review.openstack.org/438729 | 16:29 |
*** damien_r has quit IRC | 16:30 | |
*** lucasagomes is now known as lucas-afk | 16:30 | |
*** chyka has quit IRC | 16:30 | |
sfinucan | johnthetubaguy: could you take a look at https://review.openstack.org/#/c/449288/ at some point? I'd like to know if there are backwards-compatibility implications for doing something like that | 16:30 |
*** chyka has joined #openstack-nova | 16:31 | |
*** karimb has quit IRC | 16:34 | |
*** tesseract has quit IRC | 16:34 | |
*** chyka_ has joined #openstack-nova | 16:35 | |
*** chyka has quit IRC | 16:35 | |
*** annegentle has quit IRC | 16:38 | |
*** annegentle has joined #openstack-nova | 16:39 | |
*** ralonsoh__ has quit IRC | 16:40 | |
sfinucan | mdbooth: Happy to push on with https://review.openstack.org/#/c/383859/ if you fancy sticking in that context comment | 16:42 |
sfinucan | (in case that wasn't obvious from earlier) | 16:42 |
*** lucasxu has quit IRC | 16:44 | |
*** damien_r has joined #openstack-nova | 16:44 | |
*** fragatina has joined #openstack-nova | 16:45 | |
*** derekh has quit IRC | 16:48 | |
*** ssurana has joined #openstack-nova | 16:48 | |
*** jaosorior has quit IRC | 16:49 | |
*** stvnoyes has quit IRC | 16:50 | |
*** damien_r has quit IRC | 16:50 | |
*** stvnoyes has joined #openstack-nova | 16:51 | |
*** yingwei has quit IRC | 16:52 | |
*** kfarr has quit IRC | 16:53 | |
*** esberglu has quit IRC | 16:55 | |
*** lucasxu has joined #openstack-nova | 16:55 | |
*** mlavalle has quit IRC | 16:56 | |
*** alexpilo_ has joined #openstack-nova | 16:57 | |
openstackgerrit | Eric Fried proposed openstack/nova master: PowerVM Driver: spawn/destroy #4: full flavor https://review.openstack.org/391288 | 16:58 |
clarkb | mriedem: just fyi I'm fixing the live migration job fails on ocata right now | 16:59 |
clarkb | mriedem: only a portion of the UCA stuff was backported... | 16:59 |
*** hemna is now known as willynilly | 16:59 | |
*** willynilly is now known as hemna | 16:59 | |
mriedem | clarkb: ok, thanks | 17:00 |
mriedem | didn't realize it was broken, but i see that now | 17:01 |
*** alexpilotti has quit IRC | 17:01 | |
*** gjayavelu has joined #openstack-nova | 17:01 | |
*** alexpilo_ has quit IRC | 17:01 | |
clarkb | mriedem: I've been checking e-r every morning and looking at the top gate bug :) just to make sure everything is happy | 17:02 |
*** liverpooler has quit IRC | 17:03 | |
*** hshiina has quit IRC | 17:03 | |
*** mvk has quit IRC | 17:03 | |
*** liverpooler has joined #openstack-nova | 17:06 | |
johnthetubaguy | mriedem: so I just -1ed the thing I just +Wed https://review.openstack.org/#/c/438750 | 17:08 |
johnthetubaguy | mriedem: with a note about that begin_detaching stuff | 17:08 |
johnthetubaguy | mriedem: if what I said is total nonsense, I am happy for someone to over-ride that worry, the rest of it looks fine | 17:08 |
*** jpena is now known as jpena|off | 17:10 | |
*** cdent has quit IRC | 17:10 | |
*** efoley_ has quit IRC | 17:12 | |
*** david-lyle_ has joined #openstack-nova | 17:17 | |
*** david-lyle has quit IRC | 17:17 | |
*** ltomasbo is now known as ltomasbo|away | 17:17 | |
*** alexpilotti has joined #openstack-nova | 17:18 | |
*** yamahata has joined #openstack-nova | 17:18 | |
*** lyan has quit IRC | 17:19 | |
openstackgerrit | Lee Yarwood proposed openstack/nova master: encryptors: Switch to os-brick encryptor classes https://review.openstack.org/391597 | 17:19 |
ildikov | johnthetubaguy: isn't begin_detaching an API-level call? | 17:21 |
ildikov | johnthetubaguy: I mean it gets called before we even get deeper down to detach with the new flow | 17:22 |
*** david-lyle_ has quit IRC | 17:23 | |
*** mvk has joined #openstack-nova | 17:25 | |
*** Apoorva has joined #openstack-nova | 17:28 | |
*** cdent has joined #openstack-nova | 17:30 | |
*** ig0r_ has joined #openstack-nova | 17:32 | |
*** esberglu has joined #openstack-nova | 17:34 | |
mriedem | johnthetubaguy: i don't follow the concern in this patch | 17:39 |
mriedem | johnthetubaguy: as mentioned during the meeting, i thought begin_detaching could be used in both the old and new flows | 17:40 |
*** salv-orl_ has quit IRC | 17:44 | |
*** salv-orlando has joined #openstack-nova | 17:44 | |
ildikov | mriedem: I think it's currently used before we get into the actual detach parts with both flows right now | 17:45 |
mriedem | yes it is | 17:45 |
mriedem | b/c it's used in the api | 17:45 |
*** vks1 has quit IRC | 17:45 | |
ildikov | that's what I thought that it's in a different layer so it happens before we get to choose between the new/old flow | 17:46 |
*** lyan has joined #openstack-nova | 17:47 | |
*** sambetts is now known as sambetts|afk | 17:50 | |
*** alexpilotti has quit IRC | 17:53 | |
*** dillaman has quit IRC | 17:56 | |
*** vks1 has joined #openstack-nova | 17:57 | |
*** fragatina has quit IRC | 17:58 | |
*** amotoki has joined #openstack-nova | 18:00 | |
*** ijw has joined #openstack-nova | 18:01 | |
*** vks1 has quit IRC | 18:01 | |
*** avolkov has quit IRC | 18:02 | |
*** amotoki has quit IRC | 18:05 | |
*** karimb has joined #openstack-nova | 18:05 | |
*** salv-orlando has quit IRC | 18:06 | |
openstackgerrit | Jay Pipes proposed openstack/nova master: placement: implement get_inventory() for libvirt https://review.openstack.org/457782 | 18:06 |
*** karimb has quit IRC | 18:08 | |
*** ayogi has quit IRC | 18:11 | |
*** gszasz has quit IRC | 18:12 | |
*** voelzmo has joined #openstack-nova | 18:12 | |
*** dillaman has joined #openstack-nova | 18:12 | |
*** vks1 has joined #openstack-nova | 18:16 | |
*** kristian__ has quit IRC | 18:16 | |
openstackgerrit | Ildiko Vancsa proposed openstack/nova master: Add Cinder v3 detach to swap_volume https://review.openstack.org/456971 | 18:18 |
openstackgerrit | Ildiko Vancsa proposed openstack/nova master: Add Cinder v3 detach call to _terminate_volume_connections https://review.openstack.org/456896 | 18:18 |
openstackgerrit | Ildiko Vancsa proposed openstack/nova master: Add Cinder v3 detach call to post_live_migration https://review.openstack.org/456988 | 18:18 |
*** cdent has quit IRC | 18:21 | |
*** MasterOfBugs has joined #openstack-nova | 18:21 | |
*** voelzmo has quit IRC | 18:21 | |
*** stvnoyes has quit IRC | 18:24 | |
*** stvnoyes has joined #openstack-nova | 18:25 | |
*** ge0rgi has joined #openstack-nova | 18:27 | |
ildikov | mriedem: lyarwood: johnthetubaguy: fixed up the chain ^^ | 18:29 |
lyarwood | ildikov: cool thanks | 18:30 |
ildikov | mriedem: lyarwood: johnthetubaguy: these two are not relying on _terminate_volume_connections: https://review.openstack.org/#/c/456851/ https://review.openstack.org/#/c/456877/ | 18:30 |
mriedem | ok | 18:30 |
ildikov | the last two need the almost approved base change as that's where attachment_delete is added | 18:31 |
ildikov | I can move those on top of that patch if we think it's going to be blocked for longer | 18:31 |
ildikov | I hope not though | 18:31 |
mriedem | you could just use the Depends-On tag | 18:31 |
ildikov | oh, ok good point | 18:32 |
ildikov | I always thought that's more for cross-repo dependencies... | 18:32 |
*** vks1 has quit IRC | 18:33 | |
ildikov | lyarwood: all the patches will need some additional tests, so that can be picked up by anyone who's available | 18:33 |
ildikov | lyarwood: stvnoyes helped out with the base patch | 18:33 |
ildikov | lyarwood: stvnoyes: the two independent patches could be good candidates if either of you have bandwidth | 18:34 |
mriedem | ok i just commented on the lack of a test in the local delete in api one | 18:34 |
ildikov | lyarwood: stvnoyes: I can babysit the chain | 18:34 |
lyarwood | yeah I can dig into that tomorrow | 18:34 |
*** Sukhdev has joined #openstack-nova | 18:34 | |
ildikov | mriedem: yep, tnx, that's a "known issue" at least | 18:34 |
ildikov | lyarwood: sounds good, tnx, please ping stvnoyes and me if you start with either of those or both so we don't work on it in parallel | 18:35 |
lyarwood | will do, thanks | 18:35 |
*** artom has quit IRC | 18:37 | |
ge0rgi | Hello I am a CS working on my master thesis. I have a simple question: How do I raise an exception in nova compute manager so that the error message from the exception is displayed in horizon as well? Can I use notifications for this? I saw the notify_about_instance_usage method and that it accepts a fault parameter, however I don't see the message displayed. | 18:37 |
*** artom has joined #openstack-nova | 18:37 | |
stvnoyes | lyarwood, ildikov- i am up for either one, how about I take https://review.openstack.org/#/c/456877/ ? | 18:38 |
ildikov | stvnoyes: sounds good to me, tnx! | 18:38 |
*** jerrygb has quit IRC | 18:39 | |
*** Sukhdev has quit IRC | 18:40 | |
*** Sukhdev has joined #openstack-nova | 18:40 | |
*** Sukhdev has quit IRC | 18:43 | |
*** lucasxu has quit IRC | 18:44 | |
*** lucasxu has joined #openstack-nova | 18:45 | |
*** dikonoor has quit IRC | 18:45 | |
*** Sukhdev has joined #openstack-nova | 18:50 | |
*** jerrygb has joined #openstack-nova | 18:51 | |
*** alexpilotti has joined #openstack-nova | 18:52 | |
*** alexpilotti has quit IRC | 18:57 | |
*** mnestratov has quit IRC | 19:00 | |
*** lucasxu has quit IRC | 19:00 | |
*** lucasxu has joined #openstack-nova | 19:01 | |
*** liusheng has quit IRC | 19:01 | |
*** liusheng has joined #openstack-nova | 19:02 | |
*** lucasxu has quit IRC | 19:04 | |
*** jerrygb has quit IRC | 19:07 | |
*** abalutoiu has quit IRC | 19:11 | |
*** nkorabli_ has joined #openstack-nova | 19:13 | |
*** liusheng has quit IRC | 19:13 | |
*** liusheng has joined #openstack-nova | 19:14 | |
rajinir | Are live migration settings removed fom nova.conf? Looking at live_migration_progress_timeout | 19:16 |
*** kristian__ has joined #openstack-nova | 19:16 | |
*** fragatina has joined #openstack-nova | 19:16 | |
*** nkorabli has quit IRC | 19:16 | |
*** kristian__ has quit IRC | 19:21 | |
*** karimb has joined #openstack-nova | 19:22 | |
efried | Is it the done thing to use save_and_reraise_exception instead of plain raise everywhere? | 19:31 |
efried | Like, are we really worried about something as simple as LOG.* changing the active exception context? | 19:32 |
efried | mriedem ^^ | 19:32 |
*** felipemonteiro has quit IRC | 19:33 | |
*** david-lyle has joined #openstack-nova | 19:33 | |
*** jerrygb has joined #openstack-nova | 19:36 | |
sdague | efried: you got an example? | 19:38 |
*** awaugama has quit IRC | 19:38 | |
efried | sdague https://review.openstack.org/#/c/438598/13/nova/virt/powervm/vm.py@185 | 19:38 |
*** kristian__ has joined #openstack-nova | 19:38 | |
efried | etc. | 19:38 |
*** kristian__ has quit IRC | 19:40 | |
sdague | if it's legit just a log message, it's probably fine, we have a number of those in the code | 19:41 |
*** salv-orlando has joined #openstack-nova | 19:41 | |
sdague | if you are doing anything more complicated, the save_and_reraise is probably better | 19:41 |
*** kristian__ has joined #openstack-nova | 19:41 | |
edmondsw | sdague, the comments at the top of save_and_reraise_exception indicate problems can be created switching greenthreads, so does it even matter how complicated what you're doing is? | 19:42 |
*** voelzmo has joined #openstack-nova | 19:42 | |
*** pchavva has quit IRC | 19:42 | |
edmondsw | https://github.com/openstack/oslo.utils/blob/master/oslo_utils/excutils.py#L151 | 19:43 |
efried | edmondsw I don't see how it helps, considering the greenthread switch could happen in the middle of the save_and_reraise() setup anyway. | 19:44 |
efried | And that would have to be a base python bug if it really worked that way, nah? | 19:44 |
sdague | edmondsw: so maybe, but if that's the case we've got hundreds of issues to clean up in Nova | 19:44 |
cfriesen | sfinucan: you around? | 19:45 |
efried | sdague 56 ;-) | 19:45 |
sdague | efried: ok, 56 :) | 19:45 |
efried | We counted ;-) | 19:45 |
edmondsw | efried sdague I don't know, I'd just err on the side of caution and use the method... if it doesn't help, at least it doesn't hurt | 19:45 |
edmondsw | and hopefully it helps | 19:45 |
sdague | still, anyway, it seems like a standard we don't really need to hold the powervm folks to for new code | 19:45 |
edmondsw | sdague ok | 19:45 |
*** kristian__ has quit IRC | 19:46 | |
sdague | edmondsw: we would probably not be opposed to additional patches cleaning up that all over the place | 19:47 |
edmondsw | sdague sure :) | 19:48 |
*** ijw has quit IRC | 19:48 | |
efried | If it's the preferred SOP, I'll do it. Just extra code/logic for what seems like a fairly paranoid reason. | 19:50 |
openstackgerrit | Dan Smith proposed openstack/nova master: DNM: Canary test against multi-cell devstack https://review.openstack.org/458537 | 19:50 |
openstackgerrit | Dan Smith proposed openstack/nova master: WIP: Move to proper target_cell calling convention https://review.openstack.org/458634 | 19:50 |
openstackgerrit | Dan Smith proposed openstack/nova master: Make target_cell() yield a new context https://review.openstack.org/458635 | 19:50 |
openstackgerrit | Dan Smith proposed openstack/nova master: WIP: De-complicate some of the instance delete path https://review.openstack.org/458636 | 19:50 |
dansmith | mriedem: I think you will want to look at this: https://review.openstack.org/458636 | 19:50 |
*** kristian__ has joined #openstack-nova | 19:52 | |
*** abalutoiu has joined #openstack-nova | 19:52 | |
*** satyar has quit IRC | 19:53 | |
*** djohnsto has joined #openstack-nova | 19:53 | |
*** fragatina has quit IRC | 19:55 | |
*** chyka_ has quit IRC | 19:55 | |
*** chyka has joined #openstack-nova | 19:57 | |
*** voelzmo has quit IRC | 19:59 | |
*** amotoki has joined #openstack-nova | 20:01 | |
mriedem | oh come on | 20:01 |
mriedem | dansmith: https://review.openstack.org/#/c/455007/ | 20:01 |
*** salv-orlando has quit IRC | 20:01 | |
mriedem | but yeah, that was part of my master plan at some point here too, removing the context managers | 20:02 |
mriedem | since they are redundant since we stash the cell on the context | 20:02 |
mriedem | when looking up the instance | 20:02 |
dansmith | um, we can just remove that whole block? | 20:02 |
mriedem | and do normal local_delete? | 20:02 |
mriedem | or you mean this? https://review.openstack.org/#/c/455007/2/nova/compute/api.py | 20:03 |
dansmith | that yeah | 20:03 |
mriedem | it appears it's ok except for something with cells 1 | 20:03 |
mriedem | which i haven't debugged yet | 20:03 |
dansmith | okay, that's a different block than I'm working on right? | 20:03 |
mriedem | that whole block was written before https://review.openstack.org/#/c/367557/ | 20:04 |
mriedem | yes | 20:04 |
mriedem | ^ is what deletes the instance in conductor | 20:04 |
mriedem | if the build request is gone | 20:04 |
mriedem | and that code likely suffers from the same quota/local delete mess that _delete handles | 20:04 |
mriedem | i.e. we do the same gd local delete thing in 3 places | 20:04 |
dansmith | okay, I'm confused | 20:04 |
*** amotoki has quit IRC | 20:05 | |
dansmith | I don't think we can remove the targeting dance that does different things with the instance and the quota stuff, until the patches I have below this that avoid mutating the context, right? | 20:06 |
mriedem | i haven't seen your patches yet | 20:06 |
mriedem | so idk | 20:06 |
*** sree has joined #openstack-nova | 20:06 | |
mriedem | but that's why i didn't dig into removing the context dance yet | 20:06 |
mriedem | that's why i started with https://review.openstack.org/#/c/455007/ | 20:06 |
dansmith | right, so I guess Idon't understand the "oh come on" comment :) | 20:06 |
mriedem | since there was a big fat note in there about removing it after we deleted the instance in conducto | 20:07 |
mriedem | "oh come on" as in i thought we were doing the same thing, before i actually looked at your change | 20:07 |
mriedem | beyond the commit title | 20:07 |
mriedem | sucka | 20:07 |
mriedem | we are trying to converge on the same thing though - which is cleanup the 3-headed local delete monster we have | 20:08 |
dansmith | ...okay | 20:08 |
mriedem | or not | 20:08 |
*** eharney has quit IRC | 20:08 | |
*** salv-orlando has joined #openstack-nova | 20:09 | |
*** smatzek has quit IRC | 20:09 | |
*** sree has quit IRC | 20:11 | |
*** spotz is now known as spotz_zzz | 20:15 | |
*** chyka has quit IRC | 20:16 | |
*** liverpooler has quit IRC | 20:16 | |
*** chyka has joined #openstack-nova | 20:17 | |
*** spotz_zzz is now known as spotz | 20:17 | |
*** chyka has quit IRC | 20:22 | |
openstackgerrit | Eric Fried proposed openstack/nova master: PowerVM Driver: spawn/destroy #2: functional https://review.openstack.org/438598 | 20:23 |
efried | edmondsw ^^ Done and done. Thanks for the review. | 20:23 |
*** chyka has joined #openstack-nova | 20:27 | |
*** felipemonteiro has joined #openstack-nova | 20:28 | |
*** Sukhdev has quit IRC | 20:28 | |
*** mlavalle has joined #openstack-nova | 20:29 | |
openstackgerrit | Eric Fried proposed openstack/nova master: PowerVM Driver: spawn/destroy #2: functional https://review.openstack.org/438598 | 20:34 |
*** kaisers_ has quit IRC | 20:35 | |
*** dtp has quit IRC | 20:35 | |
mriedem | are we going to be backporting this? https://review.openstack.org/#/c/456397/ | 20:35 |
mriedem | dansmith: ^? | 20:35 |
openstackgerrit | Eric Fried proposed openstack/nova master: PowerVM Driver: power_on/off and reboot https://review.openstack.org/427380 | 20:35 |
*** annegentle has quit IRC | 20:35 | |
openstackgerrit | Matt Riedemann proposed openstack/python-novaclient master: 2.43: createImage/createBackup image_id is in response body https://review.openstack.org/457008 | 20:36 |
*** abalutoiu_ has joined #openstack-nova | 20:40 | |
openstackgerrit | Eric Fried proposed openstack/nova master: PowerVM Driver: spawn/destroy #3: TaskFlow https://review.openstack.org/438729 | 20:42 |
*** abalutoiu has quit IRC | 20:44 | |
*** cleong has quit IRC | 20:46 | |
*** ijw has joined #openstack-nova | 20:49 | |
cfriesen | any libvirt experts here? apparently libvirt doesn't like "cache=writethrough io=native" for a disk, but I don't see nova being aware of this. (We hit the issue by trying to use a disk with only 4K blocks, so it failed the direct-io test and set a cache mode of "writethrough") | 20:49 |
*** edmondsw has quit IRC | 20:51 | |
*** edmondsw has joined #openstack-nova | 20:51 | |
openstackgerrit | Matt Riedemann proposed openstack/nova master: api-ref: move createBackup to server-actions https://review.openstack.org/452971 | 20:52 |
openstackgerrit | Matt Riedemann proposed openstack/nova master: 2.43: Remove Location header from createImage and createBackup responses https://review.openstack.org/455443 | 20:52 |
openstackgerrit | Eric Fried proposed openstack/nova master: PowerVM Driver: spawn/destroy #4: full flavor https://review.openstack.org/391288 | 20:54 |
cfriesen | second question...if we have a disk with 4K blocks and no 512B fallback, and we use "cache=none", does the guest need to be able to handle 4KB blocks or does qemu handle that? | 20:54 |
cfriesen | I realize it's not really a nova question...but I thought there might be folks on here who've run into it. | 20:54 |
*** ijw has quit IRC | 20:56 | |
*** edmondsw has quit IRC | 20:56 | |
*** david-lyle has quit IRC | 20:57 | |
*** annegentle has joined #openstack-nova | 21:00 | |
*** rfolco has quit IRC | 21:01 | |
*** fragatina has joined #openstack-nova | 21:01 | |
*** amotoki has joined #openstack-nova | 21:02 | |
*** cdent has joined #openstack-nova | 21:03 | |
*** jdurgin has quit IRC | 21:03 | |
openstackgerrit | Eric Fried proposed openstack/nova master: PowerVM Driver: spawn/destroy #2: functional https://review.openstack.org/438598 | 21:04 |
*** amotoki has quit IRC | 21:07 | |
*** dtp has joined #openstack-nova | 21:10 | |
*** thorst has quit IRC | 21:11 | |
*** burt has quit IRC | 21:13 | |
mriedem | efried: i hope you were done making changes to that patch | 21:14 |
efried | mriedem Which 'un? | 21:14 |
mriedem | https://review.openstack.org/438598 | 21:14 |
efried | mriedem I certainly hope so too ;-) | 21:14 |
efried | Thanks! | 21:14 |
*** dimtruck is now known as zz_dimtruck | 21:16 | |
*** krtaylor has quit IRC | 21:16 | |
*** edmondsw has joined #openstack-nova | 21:20 | |
*** nkorabli_ has quit IRC | 21:22 | |
*** nkorabli has joined #openstack-nova | 21:22 | |
*** sdague has quit IRC | 21:23 | |
*** krtaylor has joined #openstack-nova | 21:24 | |
*** edmondsw has quit IRC | 21:25 | |
*** gjayavelu has quit IRC | 21:31 | |
*** karimb has quit IRC | 21:34 | |
*** catintheroof has quit IRC | 21:34 | |
*** kaisers has joined #openstack-nova | 21:35 | |
*** felipemonteiro has quit IRC | 21:42 | |
mriedem | easy patches in need of a +W https://review.openstack.org/#/q/topic:bump-min-libvirt+status:open | 21:44 |
dansmith | mriedem: yeah we shoud, you want me to? | 21:46 |
dansmith | *should | 21:46 |
*** esberglu has quit IRC | 21:46 | |
*** sneti_ has joined #openstack-nova | 21:46 | |
mriedem | dansmith: sure | 21:46 |
*** jamesden_ has quit IRC | 21:46 | |
*** esberglu has joined #openstack-nova | 21:46 | |
*** sneti_ has quit IRC | 21:47 | |
*** nkorabli has quit IRC | 21:47 | |
*** nkorabli has joined #openstack-nova | 21:48 | |
*** salv-orl_ has joined #openstack-nova | 21:49 | |
dansmith | mriedem: https://review.openstack.org/#/q/Ia7e773674a4976fc03deee3f08a6ddb45568ec11,n,z | 21:49 |
*** cdent has quit IRC | 21:50 | |
*** esberglu has quit IRC | 21:51 | |
*** salv-orlando has quit IRC | 21:51 | |
mriedem | ooo and you didn't even go lazy and backport both via the master change using gerrit | 21:52 |
mriedem | you knew i'd check that | 21:52 |
*** ijw has joined #openstack-nova | 21:52 | |
*** nkorabli has quit IRC | 21:52 | |
mriedem | +2s all around | 21:53 |
*** Guest86170 has quit IRC | 21:53 | |
dansmith | I didn't | 21:53 |
dansmith | I just never use the gerrit ui for much of anything | 21:53 |
dansmith | 'cause I don't trust it | 21:53 |
*** cdent has joined #openstack-nova | 21:54 | |
mriedem | i have the gerrit ui do my taxes and travel planning | 21:54 |
dansmith | heh | 21:54 |
*** kristian__ has quit IRC | 21:57 | |
*** gouthamr has quit IRC | 21:57 | |
*** kristian__ has joined #openstack-nova | 21:57 | |
*** gjayavelu has joined #openstack-nova | 21:57 | |
*** ijw has quit IRC | 21:58 | |
*** Guest86170 has joined #openstack-nova | 21:59 | |
efried | I always trust the gerrit UI to do rebases & cherry-picks when it's able. Until now. It just broke me. | 21:59 |
*** ge0rgi has quit IRC | 22:00 | |
clarkb | fwiw you shouldn't really need gerrit to do rebases for you | 22:02 |
*** kristian__ has quit IRC | 22:02 | |
clarkb | it can't resolve conflicts and thats the only time rebases should be necessary | 22:02 |
*** annegentle has quit IRC | 22:02 | |
efried | clarkb Unless you're dealing with a stack of changes. | 22:02 |
clarkb | oh I always curate the stack locally | 22:03 |
*** zz_dimtruck is now known as dimtruck | 22:03 | |
efried | clarkb When it's a trivial rebase, I like the two-clicks-on-gerrit rather than the multi-step download/rebase/upload. All that extra typing! | 22:03 |
openstackgerrit | Eric Fried proposed openstack/nova master: PowerVM Driver: spawn/destroy #4: full flavor https://review.openstack.org/391288 | 22:04 |
efried | 'cept ^^ where it effed up. | 22:05 |
clarkb | right I mean rather than push just a parent change in a stack then clicking buttons I will push the entire stack to avoid pushing any buttons | 22:05 |
*** Sukhdev has joined #openstack-nova | 22:06 | |
efried | clarkb I dig it. | 22:06 |
*** thorst has joined #openstack-nova | 22:06 | |
*** mdrabe has quit IRC | 22:06 | |
*** thorst has quit IRC | 22:07 | |
openstackgerrit | Eric Fried proposed openstack/nova master: PowerVM Driver: console https://review.openstack.org/409402 | 22:08 |
*** Apoorva_ has joined #openstack-nova | 22:08 | |
*** Apoorva has quit IRC | 22:11 | |
*** jdurgin has joined #openstack-nova | 22:12 | |
*** marst_ has joined #openstack-nova | 22:17 | |
*** marst_ has quit IRC | 22:18 | |
*** marst_ has joined #openstack-nova | 22:18 | |
*** djohnsto has quit IRC | 22:19 | |
*** marst has quit IRC | 22:21 | |
mriedem | always rebase an entire stack locally | 22:22 |
*** salv-orl_ has quit IRC | 22:22 | |
mriedem | unless you're crazy | 22:22 |
*** lyan has quit IRC | 22:22 | |
*** salv-orlando has joined #openstack-nova | 22:23 | |
*** annegentle has joined #openstack-nova | 22:24 | |
*** marst_ has quit IRC | 22:25 | |
*** lyan has joined #openstack-nova | 22:25 | |
*** jdurgin has quit IRC | 22:25 | |
*** jamielennox is now known as jamielennox|away | 22:25 | |
*** salv-orlando has quit IRC | 22:27 | |
openstackgerrit | Eric Fried proposed openstack/nova master: PowerVM Driver: SSP emphemeral disk support https://review.openstack.org/443189 | 22:28 |
*** annegentle has quit IRC | 22:29 | |
*** jamielennox|away is now known as jamielennox | 22:29 | |
mriedem | efried: https://review.openstack.org/#/c/427380/ failed powervm in-tree ci | 22:34 |
mriedem | error: could not apply fb15915... PowerVM Driver: spawn/destroy #3: TaskFlow | 22:34 |
mriedem | you need to rebase that change don't you? | 22:35 |
*** thorst has joined #openstack-nova | 22:39 | |
*** Apoorva_ has quit IRC | 22:41 | |
*** Apoorva has joined #openstack-nova | 22:41 | |
*** ijw has joined #openstack-nova | 22:55 | |
*** thorst has quit IRC | 22:56 | |
*** amotoki has joined #openstack-nova | 22:56 | |
*** cdent has quit IRC | 22:59 | |
*** ijw has quit IRC | 23:00 | |
*** nic has quit IRC | 23:00 | |
*** amotoki has quit IRC | 23:01 | |
*** sdague has joined #openstack-nova | 23:06 | |
openstackgerrit | christopher.uhler proposed openstack/nova master: Libvirt volume driver for Veritas HyperScale https://review.openstack.org/443951 | 23:08 |
*** ssurana has quit IRC | 23:08 | |
*** Sukhdev has quit IRC | 23:08 | |
*** ssurana has joined #openstack-nova | 23:08 | |
mriedem | smcginnis: what is https://www.openstack.org/summit/boston-2017/summit-schedule/events/18681/track-chairs-recap-and-discussion ? | 23:11 |
*** kaisers has quit IRC | 23:19 | |
*** slaweq has quit IRC | 23:20 | |
*** slaweq has joined #openstack-nova | 23:20 | |
*** jdurgin has joined #openstack-nova | 23:22 | |
*** slaweq has quit IRC | 23:24 | |
*** chyka has quit IRC | 23:33 | |
*** gouthamr has joined #openstack-nova | 23:33 | |
*** mlavalle has quit IRC | 23:36 | |
*** lyan has quit IRC | 23:37 | |
openstackgerrit | Matt Riedemann proposed openstack/nova master: Using max api version in notification sample test https://review.openstack.org/443686 | 23:38 |
openstackgerrit | Matt Riedemann proposed openstack/nova master: Adding auto_disk_config field to InstancePayload https://review.openstack.org/419185 | 23:38 |
openstackgerrit | Matt Riedemann proposed openstack/nova master: add tags field to instance.update notification https://review.openstack.org/407228 | 23:38 |
*** markvoelker has quit IRC | 23:39 | |
mriedem | gibi: for your morning coffee, the rebuild notifications in this change have duplicate entries for auto_disk_config https://review.openstack.org/#/c/419185/21/doc/notification_samples/instance-rebuild-end.json | 23:39 |
*** takashin has joined #openstack-nova | 23:41 | |
mriedem | maybe just a carried over typo | 23:41 |
*** sree has joined #openstack-nova | 23:42 | |
*** jerrygb has quit IRC | 23:45 | |
*** sree has quit IRC | 23:46 | |
openstackgerrit | Matt Riedemann proposed openstack/nova master: Adding auto_disk_config field to InstancePayload https://review.openstack.org/419185 | 23:47 |
openstackgerrit | Takashi NATSUME proposed openstack/python-novaclient master: Microversion 2.43 - Enable cold migration with target host https://review.openstack.org/406707 | 23:48 |
openstackgerrit | Takashi NATSUME proposed openstack/python-novaclient master: Microversion 2.43 - List/Show all server migration types https://review.openstack.org/430839 | 23:48 |
*** thorst has joined #openstack-nova | 23:53 | |
*** salv-orlando has joined #openstack-nova | 23:54 | |
*** ijw has joined #openstack-nova | 23:56 |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!