| *** zhugaoxiao has quit IRC | 00:08 | |
| *** wolverineav has joined #openstack-neutron | 00:10 | |
| *** claudiub has joined #openstack-neutron | 00:10 | |
| openstackgerrit | Jianghua Wang proposed openstack/neutron: XenAPI: Support daemon mode for rootwrap https://review.openstack.org/390931 | 00:13 |
|---|---|---|
| *** wolverineav has quit IRC | 00:15 | |
| *** marlinc has quit IRC | 00:18 | |
| *** thorst has joined #openstack-neutron | 00:21 | |
| *** thorst has quit IRC | 00:23 | |
| *** gbutnaru has joined #openstack-neutron | 00:29 | |
| *** wolverineav has joined #openstack-neutron | 00:30 | |
| *** wolverineav has quit IRC | 00:34 | |
| *** salv-orl_ has quit IRC | 00:38 | |
| *** wolverineav has joined #openstack-neutron | 00:40 | |
| *** tommylikehu_ has joined #openstack-neutron | 00:43 | |
| *** wolverineav has quit IRC | 00:44 | |
| *** tommylikehu_ has quit IRC | 00:47 | |
| *** tommylikehu_ has joined #openstack-neutron | 00:48 | |
| *** tommylikehu_ has quit IRC | 00:48 | |
| *** tommylikehu_ has joined #openstack-neutron | 00:49 | |
| *** tommylikehu_ has quit IRC | 00:49 | |
| *** tommylikehu_ has joined #openstack-neutron | 00:49 | |
| *** tommylikehu_ has quit IRC | 00:50 | |
| *** tommylikehu_ has joined #openstack-neutron | 00:50 | |
| *** roeyc has quit IRC | 00:50 | |
| *** tommylikehu_ has quit IRC | 00:51 | |
| *** tommylikehu_ has joined #openstack-neutron | 00:51 | |
| *** tommylikehu_ has quit IRC | 00:51 | |
| *** tommylikehu_ has joined #openstack-neutron | 00:52 | |
| *** tommylikehu_ has quit IRC | 00:52 | |
| *** tommylikehu_ has joined #openstack-neutron | 01:06 | |
| *** tommylikehu_ has quit IRC | 01:06 | |
| *** tommylikehu_ has joined #openstack-neutron | 01:07 | |
| *** tommylik_ has joined #openstack-neutron | 01:08 | |
| *** tommylikehu_ has quit IRC | 01:08 | |
| *** wolverineav has joined #openstack-neutron | 01:09 | |
| *** thorst has joined #openstack-neutron | 01:11 | |
| *** thorst has quit IRC | 01:11 | |
| *** sterdnotshaken has joined #openstack-neutron | 01:13 | |
| *** sterdnotshaken has quit IRC | 01:13 | |
| *** wolverineav has quit IRC | 01:14 | |
| *** yamahata has joined #openstack-neutron | 01:17 | |
| *** marlinc has joined #openstack-neutron | 01:23 | |
| *** gooduone has quit IRC | 01:27 | |
| *** wolverineav has joined #openstack-neutron | 01:29 | |
| *** grantma has joined #openstack-neutron | 01:30 | |
| *** wolverineav has quit IRC | 01:33 | |
| *** salv-orlando has joined #openstack-neutron | 01:38 | |
| *** claudiub has quit IRC | 01:40 | |
| *** salv-orlando has quit IRC | 01:43 | |
| *** marlinc has quit IRC | 01:43 | |
| *** wolverineav has joined #openstack-neutron | 01:48 | |
| *** marlinc has joined #openstack-neutron | 01:53 | |
| *** wolverineav has quit IRC | 01:53 | |
| *** tflynn has joined #openstack-neutron | 01:59 | |
| *** bobmel has joined #openstack-neutron | 02:03 | |
| *** bobmel has quit IRC | 02:08 | |
| *** wolverineav has joined #openstack-neutron | 02:08 | |
| *** shaner_ has quit IRC | 02:13 | |
| *** wolverineav has quit IRC | 02:13 | |
| openstackgerrit | Kevin Benton proposed openstack/neutron: Decompose router extra attr processing https://review.openstack.org/417604 | 02:16 |
| openstackgerrit | David Hill proposed openstack/neutron: Escape quotes in conn_uri declaration https://review.openstack.org/417641 | 02:26 |
| *** wolverineav has joined #openstack-neutron | 02:28 | |
| *** tflynn has quit IRC | 02:30 | |
| *** wolverineav has quit IRC | 02:33 | |
| *** thorst has joined #openstack-neutron | 02:35 | |
| *** thorst has quit IRC | 02:35 | |
| *** salv-orlando has joined #openstack-neutron | 02:40 | |
| *** yamahata has quit IRC | 02:44 | |
| *** salv-orlando has quit IRC | 02:44 | |
| *** wolverineav has joined #openstack-neutron | 02:48 | |
| *** shaner_ has joined #openstack-neutron | 02:51 | |
| *** wolverineav has quit IRC | 02:52 | |
| *** fzdarsky__ has joined #openstack-neutron | 02:57 | |
| *** fzdarsky_ has quit IRC | 03:01 | |
| *** wolverineav has joined #openstack-neutron | 03:07 | |
| *** ramishra has quit IRC | 03:11 | |
| *** shaner has quit IRC | 03:11 | |
| *** wolverineav has quit IRC | 03:11 | |
| *** ramishra has joined #openstack-neutron | 03:12 | |
| *** bobmel has joined #openstack-neutron | 03:16 | |
| *** bobmel has quit IRC | 03:20 | |
| *** jhershbe_ has joined #openstack-neutron | 03:20 | |
| *** wolverineav has joined #openstack-neutron | 03:27 | |
| *** wolverineav has quit IRC | 03:32 | |
| *** itisha has quit IRC | 03:32 | |
| *** salv-orlando has joined #openstack-neutron | 03:41 | |
| *** reedip_outofmemo has joined #openstack-neutron | 03:42 | |
| *** salv-orlando has quit IRC | 03:45 | |
| *** wolverineav has joined #openstack-neutron | 03:46 | |
| *** wolverineav has quit IRC | 03:51 | |
| *** prateek has joined #openstack-neutron | 03:54 | |
| *** dkehn has quit IRC | 03:57 | |
| *** wolverineav has joined #openstack-neutron | 03:57 | |
| *** dkehn has joined #openstack-neutron | 03:57 | |
| *** wolverineav has quit IRC | 04:01 | |
| *** wolverineav has joined #openstack-neutron | 04:06 | |
| *** wolverineav has quit IRC | 04:11 | |
| *** tbachman has joined #openstack-neutron | 04:19 | |
| *** tflynn has joined #openstack-neutron | 04:20 | |
| *** wolverineav has joined #openstack-neutron | 04:26 | |
| *** wolverineav has quit IRC | 04:31 | |
| *** wolverineav has joined #openstack-neutron | 04:36 | |
| *** wolverineav has quit IRC | 04:41 | |
| *** reedip_outofmemo has quit IRC | 04:41 | |
| *** salv-orlando has joined #openstack-neutron | 04:41 | |
| *** salv-orlando has quit IRC | 04:46 | |
| *** wolverineav has joined #openstack-neutron | 04:55 | |
| *** belharar has joined #openstack-neutron | 05:00 | |
| *** belharar_ has joined #openstack-neutron | 05:00 | |
| *** wolverineav has quit IRC | 05:00 | |
| *** gkadam has joined #openstack-neutron | 05:03 | |
| *** tommylik_ has quit IRC | 05:15 | |
| *** tflynn has quit IRC | 05:18 | |
| *** tflynn has joined #openstack-neutron | 05:18 | |
| *** nyechiel has joined #openstack-neutron | 05:25 | |
| *** bkopilov has joined #openstack-neutron | 05:29 | |
| *** gongysh has joined #openstack-neutron | 05:32 | |
| *** tommylikehu_ has joined #openstack-neutron | 05:42 | |
| *** salv-orlando has joined #openstack-neutron | 05:42 | |
| *** salv-orlando has quit IRC | 05:47 | |
| *** gvrangan has joined #openstack-neutron | 05:53 | |
| *** kobis1 has quit IRC | 06:00 | |
| *** garyk has joined #openstack-neutron | 06:05 | |
| *** itzikb has joined #openstack-neutron | 06:10 | |
| *** Alex_Stef has joined #openstack-neutron | 06:14 | |
| *** ekuris has joined #openstack-neutron | 06:15 | |
| *** moshele has joined #openstack-neutron | 06:18 | |
| *** mkolesni has joined #openstack-neutron | 06:25 | |
| openstackgerrit | Kevin Benton proposed openstack/neutron: Decompose router extra attr processing https://review.openstack.org/417604 | 06:27 |
| *** tommylikehu_ has quit IRC | 06:31 | |
| *** gkadam has quit IRC | 06:32 | |
| *** nmagnezi has joined #openstack-neutron | 06:35 | |
| *** brault has joined #openstack-neutron | 06:39 | |
| *** thorst has joined #openstack-neutron | 06:40 | |
| *** tommylikehu_ has joined #openstack-neutron | 06:43 | |
| *** brault has quit IRC | 06:43 | |
| *** salv-orlando has joined #openstack-neutron | 06:43 | |
| *** kobis has joined #openstack-neutron | 06:45 | |
| *** thorst has quit IRC | 06:45 | |
| *** Jeffrey4l_ has joined #openstack-neutron | 06:49 | |
| *** kobis has quit IRC | 06:49 | |
| *** salv-orlando has quit IRC | 06:50 | |
| *** kobis has joined #openstack-neutron | 06:51 | |
| *** thorst has joined #openstack-neutron | 06:52 | |
| *** jhershbe_ has quit IRC | 06:53 | |
| *** thorst has quit IRC | 06:56 | |
| *** jhershbe_ has joined #openstack-neutron | 07:07 | |
| *** jhershbe_ has quit IRC | 07:13 | |
| *** gcheresh_ has joined #openstack-neutron | 07:14 | |
| openstackgerrit | QunyingRan proposed openstack/neutron: Remove used ip which not in subnet pools https://review.openstack.org/322387 | 07:17 |
| *** prateek has quit IRC | 07:20 | |
| *** prateek has joined #openstack-neutron | 07:26 | |
| *** yamahata has joined #openstack-neutron | 07:33 | |
| *** jhershbe has joined #openstack-neutron | 07:35 | |
| *** salv-orlando has joined #openstack-neutron | 07:46 | |
| *** salv-orlando has quit IRC | 07:51 | |
| *** Alex_Stef has quit IRC | 07:59 | |
| *** moshele has quit IRC | 08:00 | |
| *** moshele has joined #openstack-neutron | 08:01 | |
| *** moshele has quit IRC | 08:03 | |
| openstackgerrit | Kevin Benton proposed openstack/neutron: Decompose router extra attr processing https://review.openstack.org/417604 | 08:08 |
| *** reedip has quit IRC | 08:12 | |
| *** pck has quit IRC | 08:15 | |
| *** pck has joined #openstack-neutron | 08:16 | |
| *** yfried has joined #openstack-neutron | 08:17 | |
| openstackgerrit | Kevin Benton proposed openstack/neutron: Decompose router extra attr processing https://review.openstack.org/417604 | 08:19 |
| openstackgerrit | Kevin Benton proposed openstack/neutron: Simplify AZ string conversion https://review.openstack.org/417661 | 08:19 |
| openstackgerrit | Kevin Benton proposed openstack/neutron: Get rid of l3 ha delete_router inheritance https://review.openstack.org/399507 | 08:19 |
| openstackgerrit | Kevin Benton proposed openstack/neutron: Get rid of L3 HA override of _update_router_db https://review.openstack.org/399506 | 08:19 |
| openstackgerrit | Kevin Benton proposed openstack/neutron: Get rid of create_router override in l3_ha mixin https://review.openstack.org/399505 | 08:19 |
| *** reedip has joined #openstack-neutron | 08:25 | |
| openstackgerrit | Kevin Benton proposed openstack/neutron: Simplify AZ string conversion https://review.openstack.org/417661 | 08:34 |
| openstackgerrit | Kevin Benton proposed openstack/neutron: Decompose router extra attr processing https://review.openstack.org/417604 | 08:34 |
| openstackgerrit | Kevin Benton proposed openstack/neutron: Get rid of create_router override in l3_ha mixin https://review.openstack.org/399505 | 08:35 |
| openstackgerrit | Kevin Benton proposed openstack/neutron: Get rid of L3 HA override of _update_router_db https://review.openstack.org/399506 | 08:35 |
| openstackgerrit | Kevin Benton proposed openstack/neutron: Get rid of l3 ha delete_router inheritance https://review.openstack.org/399507 | 08:35 |
| *** grantma has quit IRC | 08:38 | |
| *** kobis has quit IRC | 08:39 | |
| *** tflynn has quit IRC | 08:41 | |
| *** tflynn has joined #openstack-neutron | 08:42 | |
| *** kobis has joined #openstack-neutron | 08:45 | |
| *** tflynn has quit IRC | 08:46 | |
| *** salv-orlando has joined #openstack-neutron | 08:47 | |
| *** jhershbe has quit IRC | 08:48 | |
| kevinbenton | jschwarz: ping | 08:49 |
| jschwarz | kevinbenton, morning (or evening for you) | 08:50 |
| *** salv-orlando has quit IRC | 08:51 | |
| kevinbenton | jschwarz: moring | 08:51 |
| openstackgerrit | Kevin Benton proposed openstack/neutron: Get rid of l3 ha delete_router inheritance https://review.openstack.org/399507 | 08:52 |
| kevinbenton | jschwarz: have some patches in flight that i would like you to review | 08:52 |
| kevinbenton | jschwarz: ^^ | 08:52 |
| kevinbenton | that's the first | 08:52 |
| jschwarz | kevinbenton, will do :) | 08:52 |
| jschwarz | kevinbenton, it looks like https://review.openstack.org/#/c/317949/ introduced some race, btw | 08:52 |
| *** thorst has joined #openstack-neutron | 08:52 | |
| jschwarz | kevinbenton, 2 ha_vr_ids are allocated for the same router, and since only one is initialized the l3 agent also crashes on it | 08:53 |
| jschwarz | kevinbenton, https://bugs.launchpad.net/neutron/+bug/1654032 | 08:53 |
| openstack | Launchpad bug 1654032 in tripleo "CI: unable to ping floating-ip in pingtest" [Critical,In progress] - Assigned to Emilien Macchi (emilienm) | 08:53 |
| kevinbenton | jschwarz: 2 vr_ids? | 08:53 |
| kevinbenton | jschwarz: why don't we have a unique constraint on that? | 08:53 |
| jschwarz | kevinbenton, I dunno | 08:54 |
| jschwarz | though that would be a solution, I would like to understand the exact race first | 08:54 |
| jschwarz | I can't really reproduce this yet :< | 08:54 |
| jschwarz | meh, I'm gonna put in more debugs on that shit | 08:55 |
| *** thorst has quit IRC | 08:57 | |
| kevinbenton | jschwarz: there is another change i have that may affect that behavior | 08:57 |
| jschwarz | kevinbenton, well actually it is a primary key so it should be already unique | 08:57 |
| jschwarz | kevinbenton, I think it might be overriding the old one so it doesn't affect it | 08:58 |
| jschwarz | kevinbenton, what change? | 08:58 |
| *** roeyc has joined #openstack-neutron | 08:59 | |
| kevinbenton | https://review.openstack.org/#/c/399505/ | 08:59 |
| *** kobis has quit IRC | 08:59 | |
| *** zzzeek has quit IRC | 09:00 | |
| kevinbenton | jschwarz: after that vr_id will be allocated as part of router transaction | 09:00 |
| *** zzzeek has joined #openstack-neutron | 09:01 | |
| jschwarz | kevinbenton, aye I see.. though it can still be creation from the l3_agent_scheduler logic | 09:02 |
| *** jhershbe has joined #openstack-neutron | 09:02 | |
| kevinbenton | jschwarz: L3HARouterVRIdAllocation doesn't have unique constraint on router | 09:03 |
| jschwarz | kevinbenton, aye, but it has a primary key with the HA network (which is per tenant) so it should cover it | 09:03 |
| jschwarz | am I mistaken? | 09:03 |
| *** iranzo has joined #openstack-neutron | 09:04 | |
| kevinbenton | jschwarz: right, but there may be a race where two things allocate different VR IDs | 09:04 |
| kevinbenton | jschwarz: for the same router | 09:05 |
| kevinbenton | jschwarz: although we should just end up with an orphaned VR_ID | 09:05 |
| jschwarz | kevinbenton, only if something deletes the old one | 09:05 |
| jschwarz | which I don't think is happening | 09:06 |
| *** slaweq_ has quit IRC | 09:08 | |
| kevinbenton | jschwarz: so if the agent calls create_ha_port_and_bind right at the same time the ha network is being allocated | 09:08 |
| jschwarz | kevinbenton, but it shouldn't (because ALLOCATING) | 09:09 |
| jschwarz | kevinbenton, also I've gone through the logs very thoroughly - I don't think the agent is aware of the router until after the first routers_updated notification | 09:09 |
| *** slaweq_ has joined #openstack-neutron | 09:10 | |
| kevinbenton | jschwarz: so where are you seeing two vr_ids? | 09:10 |
| jschwarz | kevinbenton, and when it does, the scheduler says that the router has already been bound and ignores it | 09:10 |
| kevinbenton | jschwarz: the model only has one volumn | 09:10 |
| jschwarz | kevinbenton, here: | 09:10 |
| kevinbenton | column | 09:10 |
| jschwarz | kevinbenton, hold on, the log is loading :| | 09:11 |
| *** Alex_Stef has joined #openstack-neutron | 09:12 | |
| *** ushkalim has joined #openstack-neutron | 09:13 | |
| *** moshele has joined #openstack-neutron | 09:14 | |
| jschwarz | kevinbenton, so here: http://logs.openstack.org/89/417089/3/experimental-tripleo/gate-tripleo-ci-centos-7-ovb-ha/19f0335/logs/overcloud-controller-1/var/log/neutron/l3-agent.txt.gz#_2017-01-07_10_37_25_400 | 09:14 |
| jschwarz | kevinbenton, you can see that the agent is aware of ha_vr_id = 1 for this router | 09:15 |
| jschwarz | but later on it has ha_vr_id=2 | 09:15 |
| jschwarz | (when an exception occurs a bit down, it prints the router_db | 09:15 |
| jschwarz | kevinbenton, http://logs.openstack.org/89/417089/3/experimental-tripleo/gate-tripleo-ci-centos-7-ovb-ha/19f0335/logs/overcloud-controller-1/var/log/neutron/l3-agent.txt.gz#_2017-01-07_10_37_31_322 | 09:16 |
| openstackgerrit | John Schwarz proposed openstack/neutron: DNM: debug for ha https://review.openstack.org/417089 | 09:17 |
| openstackgerrit | John Schwarz proposed openstack/neutron: DNM: debug for ha https://review.openstack.org/417089 | 09:21 |
| jschwarz | kevinbenton, anyway I'm not clear on what's calling that code again so I'm a bit lost... added some more debug logs and will reproduce it again at the gate (though it may take some time) | 09:23 |
| *** kobis has joined #openstack-neutron | 09:23 | |
| jschwarz | kevinbenton, do you have some time to talk about our lords and saviors https://review.openstack.org/#/c/357966/ and https://review.openstack.org/#/c/386077/ while we're at it? : | 09:23 |
| jschwarz | :) | 09:23 |
| kevinbenton | jschwarz: hang on, looking at these logs | 09:23 |
| jschwarz | ack | 09:24 |
| kevinbenton | jschwarz: so after this change | 09:25 |
| kevinbenton | jschwarz: routers no longer have an interface on the HA network after being created | 09:25 |
| kevinbenton | jschwarz: so something could come in a delete the HA network... :/ | 09:26 |
| jschwarz | kevinbenton, say what? | 09:26 |
| jschwarz | after what change? | 09:26 |
| kevinbenton | jschwarz: after our change, we no longer create HA interfaces on the HA network during router creation | 09:27 |
| kevinbenton | jschwarz: after your change | 09:27 |
| kevinbenton | jschwarz: whatever you want to call it | 09:27 |
| jschwarz | kevinbenton, you mean https://review.openstack.org/#/c/317949/ ? | 09:27 |
| kevinbenton | jschwarz: yep | 09:27 |
| jschwarz | kevinbenton, ahhh | 09:27 |
| kevinbenton | jschwarz: so if another HA router gets deleted on that tenant in the interim | 09:28 |
| jschwarz | kevinbenton, so by the time https://review.openstack.org/#/c/317949/28/neutron/scheduler/l3_agent_scheduler.py@326 is ran, the ha network has already been deleted? | 09:28 |
| kevinbenton | jschwarz: yes, i think something else must come along and wipe out the HA network | 09:28 |
| kevinbenton | jschwarz: but I'm trying to confirm | 09:28 |
| jschwarz | well that's a bummer | 09:28 |
| jschwarz | let me run this locally | 09:28 |
| jschwarz | kevinbenton, but if the HA network has been deleted, the vr_id should also have been deleted | 09:31 |
| jschwarz | and the re-creation of it should have the value 1 | 09:31 |
| jschwarz | no? | 09:31 |
| kevinbenton | jschwarz: yeah, it doesn't look like that's the problem | 09:31 |
| kevinbenton | jschwarz: because i see a mention of the HA network ID | 09:31 |
| kevinbenton | jschwarz: before even the first sync | 09:31 |
| jschwarz | and it doesn't change? | 09:31 |
| kevinbenton | jschwarz: well i can't tell which version the l3 agent first got | 09:32 |
| kevinbenton | jschwarz: but the network existing before that first sync | 09:33 |
| kevinbenton | jschwarz: so it wasn't created afterwards | 09:34 |
| kevinbenton | jschwarz: and you're right that it should have been the same vr-ID of 1 | 09:34 |
| jschwarz | kevinbenton, was a nice idea though :) | 09:34 |
| kevinbenton | jschwarz: we can lose the HA network now though, right? or do we have a relational constraint that would stop network deletion if an HA router exists | 09:35 |
| jschwarz | kevinbenton, I think it can be delete | 09:36 |
| jschwarz | d | 09:36 |
| jschwarz | kevinbenton, but the scheduler should be smart enough to re-create it with the create_with_dependency | 09:36 |
| kevinbenton | jschwarz: then the scheduling would just create a new one | 09:36 |
| kevinbenton | jschwarz: yeah | 09:36 |
| *** shausy has joined #openstack-neutron | 09:40 | |
| kevinbenton | jschwarz: the ALLOCATING filter only applies to what is returned to the agent, right? | 09:44 |
| jschwarz | kevinbenton, right | 09:44 |
| kevinbenton | jschwarz: couldn't that still trigger interface building then? | 09:44 |
| *** shausy has quit IRC | 09:45 | |
| kevinbenton | jschwarz: i see here the router is in ALLOCATING in process_sync_data http://logs.openstack.org/89/417089/3/experimental-tripleo/gate-tripleo-ci-centos-7-ovb-ha/19f0335/logs/overcloud-controller-1/var/log/neutron/server.txt.gz#_2017-01-07_10_37_22_002 | 09:45 |
| jschwarz | kevinbenton, yes, but process_sync_data should eventually filter that one out, and it shouldn't be able to modify anything | 09:47 |
| kevinbenton | jschwarz: when is that interface building triggered? | 09:47 |
| jschwarz | kevinbenton, only on create_router, update_router and the scheduler | 09:47 |
| kevinbenton | jschwarz: right, but when is the scheduler called? | 09:48 |
| kevinbenton | jschwarz: before or after filtering? | 09:48 |
| jschwarz | kevinbenton, after | 09:48 |
| kevinbenton | jschwarz: sck | 09:48 |
| kevinbenton | jschwarz: ack | 09:48 |
| jschwarz | kevinbenton, the scheduler filters it out, and create_router calls it on its own when it's already filtered | 09:48 |
| kevinbenton | jschwarz: well create_router doesn't call it at all anymore | 09:48 |
| kevinbenton | jschwarz: same with update_router | 09:49 |
| kevinbenton | jschwarz: at least not interfaces, right? | 09:49 |
| jschwarz | why not? | 09:49 |
| jschwarz | ah right | 09:49 |
| jschwarz | kevinbenton, but see here: | 09:49 |
| jschwarz | kevinbenton, https://github.com/openstack/neutron/blob/master/neutron/db/l3_hamode_db.py#L417 | 09:49 |
| jschwarz | kevinbenton, this in turn triggers the scheduler's creation of the interfaces | 09:49 |
| kevinbenton | jschwarz: oh | 09:50 |
| kevinbenton | jschwarz: so we still end up creating interfaces as part of create_router | 09:50 |
| jschwarz | kevinbenton, yes - after RouterL3AgentBinding | 09:50 |
| jschwarz | but that's later down the line | 09:50 |
| kevinbenton | jschwarz: oh, i wonder if the issue is that we are setting the vr_id on the extra attrs outside of a transaction | 09:52 |
| kevinbenton | jschwarz: and so it randomly gets flushed to the DB at a later time | 09:52 |
| kevinbenton | jschwarz: https://github.com/openstack/neutron/blob/master/neutron/db/l3_hamode_db.py#L191-L193 | 09:52 |
| jschwarz | kevinbenton, :S | 09:52 |
| kevinbenton | jschwarz: we aren't calling that inside of a transaction anymore | 09:52 |
| jschwarz | kevinbenton, but https://github.com/openstack/neutron/blob/master/neutron/db/l3_hamode_db.py#L158 is giving us a transaction anyway, no? | 09:53 |
| kevinbenton | jschwarz: that transaction is closed when that function exits | 09:53 |
| kevinbenton | jschwarz: well when that context manager exits | 09:53 |
| kevinbenton | jschwarz: so we allocate one | 09:53 |
| kevinbenton | jschwarz: but then the assingment to extra attrs model is outisde | 09:54 |
| jschwarz | ermmm | 09:54 |
| jschwarz | kevinbenton, so shouldn't _allocate_vr_id fail the second time? | 09:55 |
| jschwarz | because the ha network already has a vr_id? | 09:55 |
| kevinbenton | jschwarz: ha networks have multiple vr ids | 09:55 |
| jschwarz | kevinbenton, no they don't | 09:55 |
| kevinbenton | jschwarz: yes they do | 09:55 |
| kevinbenton | jschwarz: why do you think that table exists :P | 09:55 |
| jschwarz | kevinbenton, ah right | 09:56 |
| jschwarz | I was thinking about something else :P | 09:56 |
| kevinbenton | jschwarz: let me do some testing locally really quick | 09:56 |
| jschwarz | kevinbenton, likewise | 09:56 |
| kevinbenton | jschwarz: i've run into issues before where making modifications to a model outside of a transaction led to late commited or even lost changes | 09:57 |
| kevinbenton | jschwarz: this probem existed with the quota engine | 09:57 |
| jschwarz | kevinbenton, but my patch didn't change the behavior - it was already outside a transaction before | 09:58 |
| jschwarz | kevinbenton, why did it only now start failing? | 09:58 |
| kevinbenton | jschwarz: well we did change the timing of when the attribute was next referenced | 10:00 |
| kevinbenton | jschwarz: https://review.openstack.org/#/c/317949/28/neutron/db/l3_hamode_db.py | 10:00 |
| kevinbenton | jschwarz: L435 of the old version | 10:00 |
| kevinbenton | jschwarz: that may have flushed the change to the db | 10:00 |
| jschwarz | aye | 10:00 |
| kevinbenton | jschwarz: before schedule_router | 10:00 |
| jschwarz | well that's a bummer :P | 10:00 |
| kevinbenton | jschwarz: but I need to confirm the behavior is indeed weird with something concrete | 10:01 |
| kevinbenton | jschwarz: the fix will be really simple if that's the problem | 10:01 |
| jschwarz | kevinbenton, ack | 10:01 |
| jschwarz | kevinbenton, but even if it wasn't yet committed, shouldn't create a new vr_id | 10:01 |
| jschwarz | kevinbenton, the ha network is created and committed so it shouldn't need to create a new vr_id | 10:01 |
| kevinbenton | jschwarz: yes it does, vr_id's aren't per ha network :) | 10:02 |
| kevinbenton | jschwarz: they are per router | 10:02 |
| jschwarz | kevinbenton, erm right again :P | 10:02 |
| *** kobis has quit IRC | 10:02 | |
| jschwarz | kevinbenton, so the logic in https://review.openstack.org/#/c/317949/28/neutron/scheduler/l3_agent_scheduler.py@326 isn't really correct then | 10:03 |
| kevinbenton | jschwarz: yeah, actually i'm not sure how this is supposed to work :) | 10:03 |
| jschwarz | kevinbenton, if the ha network exists but no vr_id exists for that router, one won't be created | 10:03 |
| kevinbenton | jschwarz: right | 10:04 |
| * jschwarz oopses | 10:04 | |
| kevinbenton | jschwarz: but unless it couldn't find the ha network, i don't see what would trigger a new allocation | 10:04 |
| jschwarz | kevinbenton, aye, that's what puzzling me | 10:05 |
| *** mickeys has quit IRC | 10:07 | |
| *** prateek has quit IRC | 10:09 | |
| jschwarz | kevinbenton, so the first API request was create_router | 10:09 |
| jschwarz | kevinbenton, if the second one was something like add_router_interface, that would trigger a schedule_routers() call | 10:10 |
| jschwarz | this might lead to a re-creation of vr id if some data is not comitted | 10:10 |
| kevinbenton | jschwarz: but the network should be committing fine | 10:11 |
| jschwarz | alas, everything other than the extra_attrs.ha_vr_id is in a transaction | 10:11 |
| *** kevo has joined #openstack-neutron | 10:16 | |
| *** prateek has joined #openstack-neutron | 10:19 | |
| kevinbenton | jschwarz: definitely keeps the same ha network | 10:28 |
| kevinbenton | jschwarz: http://logs.openstack.org/89/417089/3/experimental-tripleo/gate-tripleo-ci-centos-7-ovb-ha/19f0335/logs/overcloud-controller-0/var/log/neutron/server.txt.gz#_2017-01-07_10_37_18_985 | 10:28 |
| kevinbenton | jschwarz: 22f0fb93 is the uuid of the network when vr_id 2 shows up later | 10:28 |
| jschwarz | brb | 10:29 |
| kevinbenton | jschwarz: and the request ID of that line belongs to the HTTP POST | 10:29 |
| kevinbenton | jschwarz: ok | 10:29 |
| *** yamahata has quit IRC | 10:30 | |
| jschwarz | kevinbenton, :< | 10:30 |
| jschwarz | kevinbenton, so wth | 10:30 |
| jschwarz | kevinbenton, ok so look | 10:32 |
| jschwarz | kevinbenton, nvm :< dead end | 10:32 |
| *** gvrangan has quit IRC | 10:40 | |
| * jschwarz likes Sushi for lunch | 10:48 | |
| *** salv-orlando has joined #openstack-neutron | 10:49 | |
| kevinbenton | jschwarz: that does make a good lunch :) | 10:49 |
| jschwarz | kevinbenton++ | 10:49 |
| jschwarz | and to think I didn't like Sushi until 6 months ago | 10:49 |
| *** claudiub has joined #openstack-neutron | 10:51 | |
| *** salv-orlando has quit IRC | 10:53 | |
| kevinbenton | jschwarz: it's addictive :) | 10:54 |
| kevinbenton | jschwarz: picking up smoking might be cheaper :P | 10:54 |
| *** jhershbe has quit IRC | 10:55 | |
| jschwarz | kevinbenton, lol, yeah | 10:55 |
| *** slaweq_ has quit IRC | 11:02 | |
| *** slaweq_ has joined #openstack-neutron | 11:03 | |
| jschwarz | kevinbenton, hopefully the gate for the latest patch will fail and will give a proper traceback explaining what flow is triggering the allocation | 11:04 |
| jschwarz | I'm lost :< | 11:04 |
| jschwarz | it should finish in ~15 minutes | 11:04 |
| *** mickeys has joined #openstack-neutron | 11:07 | |
| openstackgerrit | Kevin Benton proposed openstack/neutron: test https://review.openstack.org/417672 | 11:11 |
| kevinbenton | jschwarz: more stuff to tell if it's being called && | 11:11 |
| kevinbenton | ^^ | 11:11 |
| kevinbenton | jschwarz: i need a 'check experimental' right? | 11:11 |
| *** mickeys has quit IRC | 11:12 | |
| jschwarz | kevinbenton, yes, but you want to "Depends-On: I5aa89db3624bd4fd7da8681d7b6e86618b93e4cb" | 11:12 |
| jschwarz | which adds debug logs for the neutron processes | 11:12 |
| kevinbenton | ah | 11:12 |
| openstackgerrit | Kevin Benton proposed openstack/neutron: test https://review.openstack.org/417672 | 11:13 |
| kevinbenton | jschwarz: it's not in the tripleo queue | 11:14 |
| kevinbenton | jschwarz: is there a diff keyword for that? | 11:14 |
| jschwarz | kevinbenton, nope | 11:14 |
| kevinbenton | jschwarz: oh, wrong queue | 11:14 |
| kevinbenton | jschwarz: i was looking at check-tripleo | 11:14 |
| jschwarz | kevinbenton, yep :) | 11:14 |
| *** jhershbe has joined #openstack-neutron | 11:15 | |
| kevinbenton | jschwarz: it could be that we have another bug lurking and this just altered the timing enough to reveal it | 11:15 |
| jschwarz | kevinbenton, agreed | 11:16 |
| jschwarz | kevinbenton, the run I started didn't fail | 11:17 |
| jschwarz | kevinbenton, :< need to recheck | 11:17 |
| jschwarz | with my luck, the logs I added changed the timings and now everything works | 11:17 |
| jschwarz | kevinbenton, hold on, it did fail | 11:19 |
| jschwarz | kevinbenton, http://logs.openstack.org/89/417089/5/experimental-tripleo/gate-tripleo-ci-centos-7-ovb-ha/9999697/logs/ :) | 11:19 |
| *** bigjools_ is now known as bigjools | 11:19 | |
| *** tbachman has quit IRC | 11:19 | |
| *** bigjools has quit IRC | 11:20 | |
| *** bigjools has joined #openstack-neutron | 11:20 | |
| kevinbenton | jschwarz: what's the word? | 11:20 |
| jschwarz | looking | 11:20 |
| jschwarz | kevinbenton, my browser isn't working with me very nicely, need to wget the files... :< | 11:21 |
| kevinbenton | jschwarz: yeah, these are brutal | 11:21 |
| jschwarz | kevinbenton, allocation #1: http://logs.openstack.org/89/417089/5/experimental-tripleo/gate-tripleo-ci-centos-7-ovb-ha/9999697/logs/overcloud-controller-0/var/log/neutron/server.txt.gz#_2017-01-08_11_10_07_890 | 11:21 |
| jschwarz | kevinbenton, allocation #2: http://logs.openstack.org/89/417089/5/experimental-tripleo/gate-tripleo-ci-centos-7-ovb-ha/9999697/logs/overcloud-controller-0/var/log/neutron/server.txt.gz#_2017-01-08_11_10_10_956 | 11:22 |
| jschwarz | kevinbenton, so uhh | 11:23 |
| jschwarz | let me pastebin | 11:24 |
| jschwarz | kevinbenton, http://pastebin.com/uNZGz00Y | 11:24 |
| jschwarz | kevinbenton, the first allocation is from inside the schedule(), the second one is from create_router's call to _set_vr_id_and_ensure_network | 11:24 |
| jschwarz | ah wow | 11:25 |
| jschwarz | I didn't even think of this flow! | 11:25 |
| jschwarz | the super().create_router() calls schedule() on it's own | 11:25 |
| jschwarz | :S | 11:25 |
| jschwarz | wtf this is some crazy ass bug | 11:25 |
| jschwarz | kevinbenton, anyways I'll solve it by removing one of those calls (the schedule() from the super() shouldn't happen) and put in a transaction for _set_vr_id | 11:27 |
| jschwarz | kevinbenton, I assume a normal subtransaction is ok? | 11:30 |
| kevinbenton | jschwarz: yep | 11:30 |
| *** belharar has quit IRC | 11:31 | |
| kevinbenton | jschwarz: just approve my patch :) | 11:31 |
| kevinbenton | jschwarz: and it should fix it | 11:31 |
| jschwarz | kevinbenton, lol :P | 11:32 |
| jschwarz | kevinbenton, jenkins doesn't agree with you :P | 11:32 |
| kevinbenton | jschwarz: jenkins is having a bad day | 11:32 |
| jschwarz | kevinbenton, :) | 11:33 |
| kevinbenton | jschwarz: since the base thing calls schedule, we can actually get rid of the schedule call completely in l3 HA | 11:33 |
| kevinbenton | jschwarz: once my patch is in | 11:33 |
| *** kevo has quit IRC | 11:34 | |
| kevinbenton | jschwarz: we have to be careful ripping it out of the base one because other l3 plugins not based on HA may depend on it | 11:34 |
| *** slaweq_ has quit IRC | 11:34 | |
| jschwarz | kevinbenton, the base code doesn't always call schedule though | 11:34 |
| kevinbenton | jschwarz: ok, well we'll have to leave it then | 11:34 |
| jschwarz | kevinbenton, it only does if external_gateway was provided when create_router was called | 11:34 |
| kevinbenton | jschwarz: but once we are on callbacks, the order will be fixed | 11:34 |
| jschwarz | kevinbenton, I'll review your code after I submit a patch to fix the gate failure | 11:35 |
| *** slaweq_ has joined #openstack-neutron | 11:36 | |
| jschwarz | kevinbenton, adding "context.session.begin(subtransactions=True)" to _set_vr_id doesn't like me | 11:36 |
| jschwarz | kevinbenton, "sqlalchemy.exc.InvalidRequestError: A transaction is already begun. Use subtransactions=True to allow subtransactions." for the begin() in _allocate_vr_id | 11:36 |
| kevinbenton | jschwarz: oh, we must call it in a transaction from somewhere then | 11:37 |
| kevinbenton | jschwarz: leave it be for now :) | 11:37 |
| jschwarz | kevinbenton, lol | 11:37 |
| jschwarz | kevinbenton, EmilienM wants this fixed asap for the tripleo gate though | 11:37 |
| jschwarz | it's breaking them | 11:37 |
| jschwarz | and I want to fix it because it bothers me :P | 11:37 |
| kevinbenton | jschwarz: how are you going to fix it? | 11:37 |
| jschwarz | kevinbenton, putting the "router_db.extra_attributes.ha_vr_id = XXX" in a transaction, and removing the call to _set_vr_id_and_ensure_network from create_router and update_router | 11:38 |
| jschwarz | ah but then I also need to fix the bad logic in the schedulers | 11:39 |
| jschwarz | kevinbenton, can we change https://github.com/openstack/neutron/blob/master/neutron/db/l3_hamode_db.py#L158 to an autonested_transaction? | 11:42 |
| kevinbenton | jschwarz: no | 11:43 |
| kevinbenton | jschwarz: that breaks retries | 11:43 |
| kevinbenton | jschwarz: what you can do though is split them into two lines | 11:43 |
| kevinbenton | jschwarz: this i mean | 11:43 |
| kevinbenton | https://github.com/openstack/neutron/blob/master/neutron/db/l3_hamode_db.py#L192-L193 | 11:43 |
| kevinbenton | jschwarz: allocate outside of transaction | 11:43 |
| jschwarz | kevinbenton, ack | 11:44 |
| kevinbenton | jschwarz: then assign to the extra attr inside a transaction | 11:44 |
| jschwarz | kevinbenton, looks better | 11:45 |
| jschwarz | right, working on a fix, will let you know when it's up | 11:46 |
| jschwarz | kevinbenton, when are you planning to log off? | 11:46 |
| kevinbenton | jschwarz: will be around for a while longer | 11:46 |
| kevinbenton | jschwarz: want to get my patches fixed up | 11:46 |
| *** rdo has quit IRC | 11:50 | |
| *** rdo has joined #openstack-neutron | 11:51 | |
| jschwarz | kevinbenton, so I'm thinking we're gonna want a unique constrain for the vr id allocation | 11:57 |
| jschwarz | kevinbenton, like you talked about before | 11:57 |
| kevinbenton | jschwarz: unfortunately we can't have contract scripts anymore | 11:58 |
| jschwarz | kevinbenton, yay us | 11:58 |
| kevinbenton | jschwarz: so i think we should probably just have a one line query to check that the router doesn't have a VR_ID already | 11:59 |
| jschwarz | kevinbenton, but if 2 threads concurrently check that query... | 11:59 |
| jschwarz | kevinbenton, like, 2 auto_schedule_routers() running and bind_router was successful for 2 different agents | 11:59 |
| kevinbenton | jschwarz: i don't think 2 threads can land there can they? | 12:00 |
| jschwarz | they can override its other's extra_attrs.ha_vr_id | 12:00 |
| jschwarz | it's possible | 12:00 |
| kevinbenton | jschwarz: how? | 12:00 |
| jschwarz | kevinbenton, unless the chosen_agents is sorted, in which case they can't | 12:00 |
| kevinbenton | jschwarz: we don't schedule until vr_id is allocated | 12:00 |
| jschwarz | kevinbenton, that's what's happening now | 12:01 |
| jschwarz | we create the RouterL3AgentBinding and only then realize we need a new ha_vr_id | 12:01 |
| kevinbenton | jschwarz: no, what's happening now is we are scheduling before VR_ID | 12:01 |
| kevinbenton | jschwarz: and that's what you're fixing, right? | 12:02 |
| openstackgerrit | John Schwarz proposed openstack/neutron: DNM: debug for ha https://review.openstack.org/417089 | 12:02 |
| jschwarz | kevinbenton, have a look at ^ | 12:02 |
| jschwarz | (a wip fix of what I had in mind) | 12:02 |
| jschwarz | well, we could fix it in another, simpler way | 12:03 |
| jschwarz | of just calling _set_vr_id_and_ensure_network before we start iterating over the chosen_agents | 12:03 |
| jschwarz | kevinbenton, in https://review.openstack.org/#/c/417089/6/neutron/scheduler/l3_agent_scheduler.py@392 | 12:04 |
| jschwarz | kevinbenton, then we'll really schedule until vr_id is allocated | 12:04 |
| jschwarz | s/until/after/g | 12:04 |
| jschwarz | thoughts? | 12:04 |
| kevinbenton | every time you call _set_vr_id_and_ensure_network it allocates another vr_id | 12:05 |
| jschwarz | kevinbenton, agreed | 12:05 |
| jschwarz | kevinbenton, but we can't call _set_vr_id without the safe creation loop for the ha network | 12:05 |
| jschwarz | we can do a "if not router_db.extra_attributes.ha_vr_id:", but that's not concurrently safe | 12:06 |
| kevinbenton | jschwarz: actually, if you move router_db.extra_attributes.ha_vr_id assingment into the allocate function | 12:07 |
| kevinbenton | jschwarz: so it's part of the same transaction | 12:07 |
| kevinbenton | jschwarz: it will become safe | 12:07 |
| jschwarz | then it's safe | 12:07 |
| jschwarz | and if it's created more than once - so be it | 12:07 |
| kevinbenton | jschwarz: because if something else races, one will hit a staledataerror | 12:07 |
| *** mickeys has joined #openstack-neutron | 12:08 | |
| *** iranzo has quit IRC | 12:12 | |
| *** mickeys has quit IRC | 12:13 | |
| *** iranzo has joined #openstack-neutron | 12:14 | |
| *** gongysh has quit IRC | 12:14 | |
| *** markvoelker has quit IRC | 12:15 | |
| *** brault has joined #openstack-neutron | 12:23 | |
| *** jhershbe has quit IRC | 12:26 | |
| *** thorst has joined #openstack-neutron | 12:30 | |
| *** bkopilov has quit IRC | 12:31 | |
| *** thorst has quit IRC | 12:32 | |
| *** nyechiel has quit IRC | 12:38 | |
| openstackgerrit | John Schwarz proposed openstack/neutron: l3scheduler: create ha_vr_id more robustly https://review.openstack.org/417089 | 12:42 |
| jschwarz | kevinbenton, ^ | 12:42 |
| jschwarz | kevinbenton, will be happy if we can merge it quickly so reviews would be obviously appreciated ;-) | 12:43 |
| kevinbenton | jschwarz: missing a check | 12:44 |
| kevinbenton | jschwarz: left a couple of comments | 12:44 |
| jschwarz | kevinbenton, ack | 12:45 |
| *** jhershbe has joined #openstack-neutron | 12:47 | |
| *** salv-orlando has joined #openstack-neutron | 12:50 | |
| *** kobis has joined #openstack-neutron | 12:51 | |
| *** salv-orlando has quit IRC | 12:55 | |
| openstackgerrit | John Schwarz proposed openstack/neutron: l3scheduler: create ha_vr_id more robustly https://review.openstack.org/417089 | 12:55 |
| jschwarz | kevinbenton, ^ :) | 12:55 |
| kevinbenton | jschwarz: is 0 a valid VR_ID? | 12:56 |
| *** tommylikehu has quit IRC | 12:56 | |
| jschwarz | kevinbenton, nope | 12:56 |
| jschwarz | it starts from 1 | 12:56 |
| *** tommylikehu has joined #openstack-neutron | 12:57 | |
| kevinbenton | jschwarz: you don't want to get rid of _set_vr_id? | 12:57 |
| jschwarz | kevinbenton, forgot to post replies | 12:57 |
| jschwarz | kevinbenton, it's still used by _set_vr_id_and_ensure_network (to extrapolate network_id) | 12:57 |
| kevinbenton | jschwarz: why can't it call _allocate_vr_id ? | 12:58 |
| kevinbenton | jschwarz: adjust the signature of _allocate_vr_id to accept the same thing _set_vr_id used to | 12:58 |
| jschwarz | kevinbenton, ah | 13:00 |
| jschwarz | right, silly me O.o | 13:00 |
| oanson | Hi. I have a strange issue with openstack: It returns "'NoneType' object is not iterable" when I create a router (router is created) | 13:01 |
| oanson | It seems availability_zone_hints is None, rather than a list. | 13:02 |
| oanson | Is this known? Where there changes in that area? | 13:03 |
| kevinbenton | oanson: not that i've heard of | 13:03 |
| jschwarz | oanson, can you provide a full traceback? | 13:04 |
| oanson | Yes. One sec | 13:04 |
| oanson | Available here: http://paste.fedoraproject.org/522599/80694148 | 13:05 |
| oanson | I added a try/catch clause myself. Otherwise the traceback wouldn't appear. | 13:05 |
| jschwarz | oanson, sorry, not familiar with that code | 13:06 |
| oanson | jschwarz, kevinbenton, thanks. I'll keep digging. | 13:07 |
| *** mickeys has joined #openstack-neutron | 13:09 | |
| *** tbachman has joined #openstack-neutron | 13:13 | |
| *** mickeys has quit IRC | 13:13 | |
| *** markvoelker has joined #openstack-neutron | 13:16 | |
| jschwarz | kevinbenton, done | 13:16 |
| openstackgerrit | John Schwarz proposed openstack/neutron: l3scheduler: create ha_vr_id more robustly https://review.openstack.org/417089 | 13:16 |
| *** yfried has quit IRC | 13:18 | |
| *** prateek has quit IRC | 13:19 | |
| roeyc | oanson: have a look here https://bugs.launchpad.net/python-openstackclient/+bug/1652025 | 13:20 |
| openstack | Launchpad bug 1652025 in python-openstackclient "Traceback after subnet create" [Undecided,In progress] - Assigned to Roey Chen (roeyc) | 13:20 |
| oanson | roeyc, yes. This is it! Thanks! | 13:21 |
| *** markvoelker has quit IRC | 13:21 | |
| *** gkadam has joined #openstack-neutron | 13:22 | |
| openstackgerrit | Kevin Benton proposed openstack/neutron: Simplify AZ string conversion https://review.openstack.org/417661 | 13:22 |
| *** tbachman has quit IRC | 13:23 | |
| *** bobmel has joined #openstack-neutron | 13:23 | |
| *** tbachman has joined #openstack-neutron | 13:23 | |
| *** prateek has joined #openstack-neutron | 13:24 | |
| kevinbenton | jschwarz: what is the purpose of changing https://review.openstack.org/#/c/417089/9/neutron/scheduler/l3_agent_scheduler.py ? | 13:25 |
| *** bobmel_ has joined #openstack-neutron | 13:26 | |
| kevinbenton | jschwarz: wasn't it fine the way it was? | 13:26 |
| jschwarz | kevinbenton, nope | 13:26 |
| kevinbenton | jschwarz: why? | 13:26 |
| jschwarz | kevinbenton, well it was since _set_vr_id_and_ensure_network was called from the create_router() logic | 13:26 |
| jschwarz | but now that it isn't, in case an ha network already exists, it would never create a vr_id allocation now | 13:27 |
| kevinbenton | jschwarz: oh, i see | 13:27 |
| *** bobmel has quit IRC | 13:28 | |
| kevinbenton | jschwarz: two more small changes | 13:28 |
| jschwarz | kevinbenton, will do | 13:29 |
| *** moshele has quit IRC | 13:32 | |
| *** moshele has joined #openstack-neutron | 13:33 | |
| *** tbachman has quit IRC | 13:33 | |
| *** moshele has quit IRC | 13:34 | |
| *** moshele has joined #openstack-neutron | 13:34 | |
| openstackgerrit | garyk proposed openstack/neutron: Remove python 3.4 support https://review.openstack.org/415952 | 13:38 |
| *** bobmel_ has quit IRC | 13:38 | |
| *** tbachman has joined #openstack-neutron | 13:39 | |
| *** garyk has quit IRC | 13:41 | |
| *** tbachman has quit IRC | 13:43 | |
| *** bobmel has joined #openstack-neutron | 13:45 | |
| jschwarz | kevinbenton, and yet another ping ^_^ | 13:46 |
| *** tbachman has joined #openstack-neutron | 13:46 | |
| openstackgerrit | John Schwarz proposed openstack/neutron: l3scheduler: create ha_vr_id more robustly https://review.openstack.org/417089 | 13:46 |
| *** tbachman has quit IRC | 13:50 | |
| kevinbenton | jschwarz: lgtm | 13:51 |
| *** salv-orlando has joined #openstack-neutron | 13:52 | |
| jschwarz | kevinbenton, pleasure doing business with you :) | 13:52 |
| *** xinliang has quit IRC | 13:53 | |
| *** limao has joined #openstack-neutron | 13:55 | |
| *** salv-orlando has quit IRC | 13:56 | |
| *** mriedem1 has joined #openstack-neutron | 13:59 | |
| *** mriedem has quit IRC | 13:59 | |
| *** bobmel has quit IRC | 14:02 | |
| *** wolverineav has joined #openstack-neutron | 14:03 | |
| *** xinliang has joined #openstack-neutron | 14:06 | |
| *** mickeys has joined #openstack-neutron | 14:10 | |
| *** gkadam is now known as newnick | 14:12 | |
| *** newnick is now known as gkadam | 14:12 | |
| *** itzikb has quit IRC | 14:13 | |
| *** mickeys has quit IRC | 14:14 | |
| openstackgerrit | John Schwarz proposed openstack/neutron: Revert "Add ALLOCATING state to routers" https://review.openstack.org/357966 | 14:16 |
| *** markvoelker has joined #openstack-neutron | 14:17 | |
| openstackgerrit | Kevin Benton proposed openstack/neutron: Decompose router extra attr processing https://review.openstack.org/417604 | 14:18 |
| kevinbenton | jschwarz: what else did you want to chat about? | 14:19 |
| jschwarz | kevinbenton, https://review.openstack.org/357966 and https://review.openstack.org/#/c/386077/ | 14:19 |
| jschwarz | kevinbenton, the last missing pieces of The Great L3 Scheduler Refactor | 14:19 |
| jschwarz | TGLSR! | 14:19 |
| *** gkadam has quit IRC | 14:20 | |
| *** shausy has joined #openstack-neutron | 14:21 | |
| *** gkadam has joined #openstack-neutron | 14:21 | |
| *** markvoelker has quit IRC | 14:22 | |
| *** mriedem1 is now known as mriedem | 14:22 | |
| openstackgerrit | John Schwarz proposed openstack/neutron: Refactor L3 scheduler (unify code paths) https://review.openstack.org/386077 | 14:25 |
| jschwarz | kevinbenton, reviewing your patches next | 14:26 |
| *** wolverineav has quit IRC | 14:26 | |
| jschwarz | kevinbenton, re: https://review.openstack.org/#/c/399505, the commit message mentions that if the scheduler threw an error it's now ignored (puts the router into ERROR) | 14:28 |
| *** shausy has quit IRC | 14:28 | |
| jschwarz | worth mentioning that the scheduler should be safe as it is (doesn't throw exceptions afair) | 14:28 |
| kevinbenton | jschwarz: not even if all agents disappear? | 14:29 |
| *** wolverineav has joined #openstack-neutron | 14:29 | |
| *** shausy has joined #openstack-neutron | 14:29 | |
| *** wolverineav has quit IRC | 14:30 | |
| jschwarz | kevinbenton, I didn't check all the possible scenarios, mind you ;-) | 14:30 |
| *** bobmel has joined #openstack-neutron | 14:32 | |
| *** thorst has joined #openstack-neutron | 14:32 | |
| openstackgerrit | John Schwarz proposed openstack/neutron: Revert "Add ALLOCATING state to routers" https://review.openstack.org/357966 | 14:35 |
| *** tommylikehu_ has quit IRC | 14:36 | |
| *** thorst has quit IRC | 14:37 | |
| *** armax has joined #openstack-neutron | 14:37 | |
| *** reedip has quit IRC | 14:39 | |
| *** wolverineav has joined #openstack-neutron | 14:45 | |
| *** belharar has joined #openstack-neutron | 14:46 | |
| *** moshele has quit IRC | 14:47 | |
| *** bobmel has quit IRC | 14:49 | |
| *** fzdarsky__ has quit IRC | 14:49 | |
| *** wolverineav has quit IRC | 14:50 | |
| *** jhershbe has quit IRC | 14:51 | |
| *** bobmel has joined #openstack-neutron | 14:52 | |
| *** gus has quit IRC | 14:52 | |
| *** salv-orlando has joined #openstack-neutron | 14:52 | |
| *** reedip has joined #openstack-neutron | 14:53 | |
| *** gus has joined #openstack-neutron | 14:54 | |
| *** nyechiel has joined #openstack-neutron | 14:56 | |
| openstackgerrit | John Schwarz proposed openstack/neutron: Revert "Add ALLOCATING state to routers" https://review.openstack.org/357966 | 14:56 |
| *** salv-orlando has quit IRC | 14:57 | |
| *** wolverineav has joined #openstack-neutron | 14:57 | |
| *** fzdarsky has joined #openstack-neutron | 15:00 | |
| *** bobmel has quit IRC | 15:01 | |
| *** wolverineav has quit IRC | 15:02 | |
| jschwarz | kevinbenton, replied on https://review.openstack.org/#/c/386077/5 | 15:05 |
| * jschwarz likes the fast iterations | 15:05 | |
| *** moshele has joined #openstack-neutron | 15:05 | |
| kevinbenton | jschwarz: so we still have two different ways scheduling can occur | 15:07 |
| kevinbenton | jschwarz: either right after router creation (or interface attachment in legacy case) | 15:07 |
| kevinbenton | jschwarz: or if auto_schedule_routers is called | 15:07 |
| *** bobmel has joined #openstack-neutron | 15:08 | |
| kevinbenton | jschwarz: right? | 15:08 |
| jschwarz | kevinbenton, correct | 15:08 |
| jschwarz | kevinbenton, but they both use the same code paths now | 15:08 |
| kevinbenton | jschwarz: when does the l3 agent call auto_schedule_routers | 15:09 |
| kevinbenton | jschwarz: only on a sync right? | 15:09 |
| jschwarz | kevinbenton, on get_router_ids (the one before the sync) | 15:10 |
| *** armax has quit IRC | 15:10 | |
| kevinbenton | jschwarz: but it only calls in reaction to something else is what i mean | 15:10 |
| kevinbenton | jschwarz: on startup or if something updates? | 15:10 |
| kevinbenton | jschwarz: i'm trying to understand when auto_schedule_routers will be called | 15:11 |
| jschwarz | kevinbenton, yes | 15:11 |
| *** mickeys has joined #openstack-neutron | 15:11 | |
| jschwarz | kevinbenton, on startup of an agent, or on a full_sync | 15:11 |
| kevinbenton | jschwarz: and a full sync now only occurs on error? | 15:11 |
| jschwarz | kevinbenton, iirc yes | 15:11 |
| *** fzdarsky is now known as fzdarsky|afk | 15:12 | |
| kevinbenton | jschwarz: ok. it's out of scope of this patch, but I think this needs to go away in its current form | 15:12 |
| *** fzdarsky|afk has quit IRC | 15:13 | |
| jschwarz | kevinbenton, I agree | 15:13 |
| jschwarz | kevinbenton, once we have all the OVOs for routers, we can do some good push notification logic here and change a lot of how it works | 15:13 |
| kevinbenton | jschwarz: regardless of that, it's just stupid scheduling | 15:13 |
| jschwarz | kevinbenton, lol | 15:13 |
| kevinbenton | jschwarz: first router that happens to hit this will consume every unscheduled router | 15:14 |
| kevinbenton | jschwarz: even if it's fully loaded | 15:14 |
| jschwarz | underscheduled, you mean | 15:14 |
| kevinbenton | jschwarz: either | 15:14 |
| jschwarz | kevinbenton, yep | 15:14 |
| kevinbenton | <1 for legacy | 15:14 |
| jschwarz | kevinbenton, we can change this into some kind of a global scheduler for all the routers and all the agents | 15:14 |
| jschwarz | like a load-balancer scheduler or something | 15:14 |
| kevinbenton | jschwarz: we already have that | 15:15 |
| kevinbenton | jschwarz: it happens during normal router scheduling | 15:15 |
| kevinbenton | jschwarz: or maybe that's just dhcp agents? | 15:15 |
| jschwarz | kevinbenton, if we use the LeastRoutersScheduler, yes | 15:15 |
| kevinbenton | jschwarz: right | 15:16 |
| kevinbenton | jschwarz: but auto schedule just completely ignores that | 15:16 |
| jschwarz | kevinbenton, but that doesn't work the same for auto_schedule_routers | 15:16 |
| jschwarz | yep | 15:16 |
| kevinbenton | jschwarz: i'm not sure what the point of auto schedule routers is now | 15:16 |
| *** bobmel has quit IRC | 15:16 | |
| *** mickeys has quit IRC | 15:16 | |
| jschwarz | kevinbenton, the point was always "here's a new agent, lets share the load". whether or not it's smart about it is a different story | 15:17 |
| *** wolverineav has joined #openstack-neutron | 15:17 | |
| kevinbenton | jschwarz: but it doesn't share the load | 15:17 |
| *** bobmel has joined #openstack-neutron | 15:17 | |
| kevinbenton | jschwarz: no routers should ever match the criteria unless there was an error scheduling before | 15:17 |
| jschwarz | kevinbenton, on HA routers there could be a case where it does - if max_l3_agents is high enough | 15:17 |
| jschwarz | but yeah, that's sucky | 15:17 |
| *** markvoelker has joined #openstack-neutron | 15:18 | |
| jschwarz | you'd expect something like a rescheduling of some of the existing routers to the new agent | 15:18 |
| jschwarz | or something like that | 15:18 |
| kevinbenton | jschwarz: i can see the case for HA adding more instances | 15:18 |
| jschwarz | kevinbenton, how does the dhcp scheduler does it? | 15:19 |
| kevinbenton | jschwarz: there is no auto_schedule | 15:20 |
| kevinbenton | jschwarz: that i can think of | 15:20 |
| kevinbenton | jschwarz: nothing triggered by the agent | 15:20 |
| *** wolverineav has quit IRC | 15:21 | |
| *** markvoelker has quit IRC | 15:22 | |
| *** dave-mccowan has joined #openstack-neutron | 15:25 | |
| kevinbenton | jschwarz: let me ponder on it over the rest of the weekend :) | 15:26 |
| kevinbenton | jschwarz: i'm wondering if it's time to dump auto_schedule_routers | 15:26 |
| jschwarz | kevinbenton, meh | 15:27 |
| jschwarz | kevinbenton, we could have done that one like a year ago, and it would have prevented 99% of the races we fixed :P | 15:27 |
| jschwarz | kevinbenton, jokes aside though, I do like the general idea of the auto schedule concept. that it's bad is something else and we can improve it over time (that's what my patches intend to start to do) | 15:28 |
| kevinbenton | jschwarz: can you explain to me what you want it to do though? | 15:28 |
| kevinbenton | jschwarz: is it just for adding more HA instances? | 15:29 |
| jschwarz | kevinbenton, the long game plan is making it actually useful | 15:30 |
| kevinbenton | jschwarz: how? | 15:30 |
| kevinbenton | jschwarz: all routers are scheduled after creation | 15:30 |
| kevinbenton | jschwarz: there shouldn't be a case for it to do something | 15:30 |
| kevinbenton | jschwarz: that's what I'm trying to wrap my head around | 15:30 |
| *** bobmel has quit IRC | 15:30 | |
| jschwarz | kevinbenton, as in, if I have 50 routers on node 1 and 50 routers on node 2, surely an addition of a new node can unschedule some of these routers to make it a 33/33/34 | 15:30 |
| jschwarz | or something along those lines | 15:31 |
| kevinbenton | jschwarz: no, that disrupts traffic | 15:31 |
| jschwarz | kevinbenton, not sure it distrupts floatingips traffic | 15:32 |
| kevinbenton | jschwarz: for ha maybe you could shuffle some standby ones around | 15:32 |
| jschwarz | kevinbenton, snat traffic can be managed through contrackd I think (but didn't make sure) | 15:32 |
| kevinbenton | jschwarz: so you want to implement live migration of conntrackd state? :) | 15:32 |
| jschwarz | kevinbenton, YES! :D | 15:33 |
| kevinbenton | jschwarz: you would also need to have the notion of a think being scheduled to both agents while its happening | 15:33 |
| kevinbenton | a thing* | 15:33 |
| jschwarz | lol | 15:33 |
| jschwarz | problems for future me ;-) | 15:33 |
| kevinbenton | jschwarz: we used to have a config option for auto_schedule_routers | 15:33 |
| *** thorst has joined #openstack-neutron | 15:34 | |
| jschwarz | kevinbenton, don't we still? | 15:34 |
| *** mriedem has quit IRC | 15:34 | |
| *** thorst has quit IRC | 15:34 | |
| jschwarz | pretty sure we still do | 15:34 |
| *** bobmel has joined #openstack-neutron | 15:34 | |
| kevinbenton | jschwarz: not that grep showed me | 15:34 |
| kevinbenton | jschwarz: unless it's named something weird | 15:34 |
| jschwarz | kevinbenton, ./neutron/db/l3_agentschedulers_db.py:55: cfg.BoolOpt('router_auto_schedule', default=True, | 15:35 |
| jschwarz | of course, why not switch some words around to make it interesting to find? :P | 15:35 |
| kevinbenton | jschwarz: ah, the dyslexic defense device | 15:35 |
| *** rossella_s has joined #openstack-neutron | 15:39 | |
| *** bobmel has quit IRC | 15:40 | |
| kevinbenton | jschwarz: https://github.com/openstack/neutron/commit/4c41a259c93e326d5635164047c72c9de207fde5 | 15:43 |
| kevinbenton | jschwarz: L131 of neutron/db/l3_dvr_db.py | 15:43 |
| kevinbenton | jschwarz: on the new side | 15:43 |
| jschwarz | kevinbenton, yes? | 15:43 |
| kevinbenton | jschwarz: why did you do it on every router update instead of just on migration? | 15:44 |
| *** gcheresh_ has quit IRC | 15:44 | |
| jschwarz | looking | 15:44 |
| *** kobis has quit IRC | 15:44 | |
| kevinbenton | jschwarz: i.e. put under the "if migrating_to_distributed:" branch | 15:45 |
| *** Alex_Stef has quit IRC | 15:45 | |
| jschwarz | kevinbenton, I didn't introduce the original code | 15:45 |
| jschwarz | kevinbenton, https://review.openstack.org/#/c/404215/2/neutron/db/l3_dvr_db.py | 15:45 |
| *** yfried has joined #openstack-neutron | 15:45 | |
| kevinbenton | jschwarz: ah | 15:45 |
| jschwarz | kevinbenton, the _update_distributed_attr was already there, I only made it more generic so I can reuse it | 15:45 |
| jschwarz | kevinbenton, didn't want to also touch the logic as we needed it for some downstream milestone | 15:46 |
| kevinbenton | jschwarz: makese sense | 15:46 |
| kevinbenton | jschwarz: i'm afraid to touch it right now :) | 15:46 |
| kevinbenton | jschwarz: hence the question | 15:46 |
| jschwarz | kevinbenton, man that code is scary | 15:46 |
| kevinbenton | jschwarz: well it's becoming much shorter | 15:47 |
| kevinbenton | jschwarz: to the point where just that is starting to stand out as odd | 15:47 |
| jschwarz | kevinbenton, but yeah, I agree it should be moved to inside the 'if' | 15:47 |
| jschwarz | those are some good patches actually :) | 15:47 |
| *** fidgi has joined #openstack-neutron | 15:47 | |
| *** moshele has quit IRC | 15:47 | |
| *** moshele has joined #openstack-neutron | 15:48 | |
| *** ekuris has quit IRC | 15:50 | |
| * jschwarz is logging off | 15:55 | |
| jschwarz | see you guys tomorrow :) | 15:55 |
| *** shausy has quit IRC | 15:56 | |
| *** wolverineav has joined #openstack-neutron | 15:56 | |
| *** shausy has joined #openstack-neutron | 15:56 | |
| *** bkopilov has joined #openstack-neutron | 15:59 | |
| kevinbenton | jschwarz: bye! | 15:59 |
| *** wolverineav has quit IRC | 16:00 | |
| *** wolverineav has joined #openstack-neutron | 16:06 | |
| *** fidgi has left #openstack-neutron | 16:08 | |
| *** yfried has quit IRC | 16:11 | |
| *** wolverineav has quit IRC | 16:11 | |
| *** roeyc has quit IRC | 16:11 | |
| *** mickeys has joined #openstack-neutron | 16:13 | |
| *** wolverineav has joined #openstack-neutron | 16:16 | |
| *** mickeys has quit IRC | 16:17 | |
| *** markvoelker has joined #openstack-neutron | 16:19 | |
| *** wolverineav has quit IRC | 16:20 | |
| openstackgerrit | Kevin Benton proposed openstack/neutron: Decompose router extra attr processing https://review.openstack.org/417604 | 16:22 |
| openstackgerrit | Kevin Benton proposed openstack/neutron: Only migrate ports on DVR migration https://review.openstack.org/417699 | 16:22 |
| *** markvoelker has quit IRC | 16:23 | |
| *** gkadam has quit IRC | 16:24 | |
| *** moshele has quit IRC | 16:26 | |
| *** shausy has quit IRC | 16:30 | |
| *** moshele has joined #openstack-neutron | 16:30 | |
| *** shausy has joined #openstack-neutron | 16:30 | |
| openstackgerrit | Kevin Benton proposed openstack/neutron: Get rid of create_router override in l3_ha mixin https://review.openstack.org/399505 | 16:34 |
| *** mdbooth has quit IRC | 16:34 | |
| openstackgerrit | Kevin Benton proposed openstack/neutron: Get rid of L3 HA override of _update_router_db https://review.openstack.org/399506 | 16:35 |
| *** moshele has quit IRC | 16:35 | |
| *** salv-orlando has joined #openstack-neutron | 16:35 | |
| *** mdbooth has joined #openstack-neutron | 16:35 | |
| *** ushkalim has quit IRC | 16:36 | |
| *** mriedem has joined #openstack-neutron | 16:36 | |
| *** nyechiel has quit IRC | 16:41 | |
| *** wolverineav has joined #openstack-neutron | 16:45 | |
| *** markvoelker has joined #openstack-neutron | 16:47 | |
| *** armax has joined #openstack-neutron | 16:48 | |
| *** wolverineav has quit IRC | 16:50 | |
| *** markvoelker has quit IRC | 16:52 | |
| *** limao has quit IRC | 16:59 | |
| *** wolverineav has joined #openstack-neutron | 17:05 | |
| *** wolverineav has quit IRC | 17:10 | |
| *** nyechiel has joined #openstack-neutron | 17:11 | |
| *** mickeys has joined #openstack-neutron | 17:14 | |
| *** mickeys has quit IRC | 17:18 | |
| *** moshele has joined #openstack-neutron | 17:21 | |
| *** wolverineav has joined #openstack-neutron | 17:25 | |
| *** teclator has quit IRC | 17:26 | |
| *** teclator has joined #openstack-neutron | 17:26 | |
| *** prateek has quit IRC | 17:26 | |
| *** shausy has quit IRC | 17:27 | |
| *** shausy has joined #openstack-neutron | 17:27 | |
| *** haplo37_ has quit IRC | 17:29 | |
| *** salv-orlando has quit IRC | 17:29 | |
| *** wolverineav has quit IRC | 17:30 | |
| *** prateek has joined #openstack-neutron | 17:32 | |
| *** haplo37_ has joined #openstack-neutron | 17:32 | |
| *** thorst has joined #openstack-neutron | 17:35 | |
| *** armax has quit IRC | 17:35 | |
| *** igordcard has quit IRC | 17:38 | |
| *** igordcard has joined #openstack-neutron | 17:38 | |
| *** teclator has quit IRC | 17:38 | |
| *** gkadam has joined #openstack-neutron | 17:39 | |
| *** teclator has joined #openstack-neutron | 17:39 | |
| *** thorst has quit IRC | 17:39 | |
| *** wolverineav has joined #openstack-neutron | 17:41 | |
| *** shausy has quit IRC | 17:42 | |
| openstackgerrit | Margaret Frances proposed openstack/neutron-lib: FWaaS v2 API reference https://review.openstack.org/391338 | 17:44 |
| *** wolverineav has quit IRC | 17:45 | |
| *** neiljerram has quit IRC | 17:47 | |
| *** teclator has quit IRC | 18:02 | |
| *** teclator has joined #openstack-neutron | 18:02 | |
| *** gkadam has quit IRC | 18:06 | |
| *** thorst has joined #openstack-neutron | 18:07 | |
| *** thorst has quit IRC | 18:08 | |
| *** ijw has joined #openstack-neutron | 18:10 | |
| *** belharar has quit IRC | 18:11 | |
| *** mickeys has joined #openstack-neutron | 18:14 | |
| *** duartea__ has joined #openstack-neutron | 18:17 | |
| *** mickeys has quit IRC | 18:18 | |
| *** adduarte has quit IRC | 18:20 | |
| *** moshele has quit IRC | 18:22 | |
| *** nyechiel has quit IRC | 18:25 | |
| *** Jack_V has joined #openstack-neutron | 18:25 | |
| *** wolverineav has joined #openstack-neutron | 18:26 | |
| *** salv-orlando has joined #openstack-neutron | 18:30 | |
| *** markmcclain has quit IRC | 18:30 | |
| *** markmcclain has joined #openstack-neutron | 18:31 | |
| *** wolverineav has quit IRC | 18:31 | |
| *** tbachman has joined #openstack-neutron | 18:31 | |
| *** tbachman_ has joined #openstack-neutron | 18:33 | |
| *** salv-orlando has quit IRC | 18:35 | |
| *** tbachman_ has quit IRC | 18:36 | |
| *** tbachman has quit IRC | 18:36 | |
| *** belharar has joined #openstack-neutron | 18:37 | |
| *** grantma has joined #openstack-neutron | 18:40 | |
| *** belharar has quit IRC | 18:42 | |
| *** bobmel has joined #openstack-neutron | 18:42 | |
| *** mickeys has joined #openstack-neutron | 18:44 | |
| *** dave-mccowan has quit IRC | 18:46 | |
| *** wolverineav has joined #openstack-neutron | 18:46 | |
| *** bobmel has quit IRC | 18:47 | |
| *** nyechiel has joined #openstack-neutron | 18:48 | |
| *** wolverineav has quit IRC | 18:51 | |
| *** g3ek has quit IRC | 18:54 | |
| *** nyechiel has quit IRC | 19:02 | |
| *** prateek has quit IRC | 19:03 | |
| *** pcaruana has quit IRC | 19:03 | |
| *** g3ek has joined #openstack-neutron | 19:04 | |
| *** wolverineav has joined #openstack-neutron | 19:06 | |
| *** bobmel has joined #openstack-neutron | 19:08 | |
| *** moshele has joined #openstack-neutron | 19:08 | |
| *** wolverineav has quit IRC | 19:10 | |
| *** matrohon has joined #openstack-neutron | 19:11 | |
| *** prateek has joined #openstack-neutron | 19:13 | |
| *** wolverineav has joined #openstack-neutron | 19:25 | |
| *** yfried has joined #openstack-neutron | 19:29 | |
| *** matrohon has quit IRC | 19:30 | |
| *** wolverineav has quit IRC | 19:30 | |
| *** salv-orlando has joined #openstack-neutron | 19:31 | |
| *** fnaval has joined #openstack-neutron | 19:31 | |
| *** nicolasbock has joined #openstack-neutron | 19:32 | |
| *** fnaval has quit IRC | 19:33 | |
| *** fnaval has joined #openstack-neutron | 19:34 | |
| *** salv-orlando has quit IRC | 19:35 | |
| *** wolverineav has joined #openstack-neutron | 19:35 | |
| *** thorst has joined #openstack-neutron | 19:37 | |
| *** fnaval has quit IRC | 19:38 | |
| *** Jack_V has quit IRC | 19:40 | |
| *** wolverineav has quit IRC | 19:40 | |
| *** thorst has quit IRC | 19:41 | |
| *** fnaval has joined #openstack-neutron | 19:43 | |
| *** fnaval has quit IRC | 19:43 | |
| *** fnaval has joined #openstack-neutron | 19:44 | |
| johnsom | Is anyone looking at this issue? | 19:44 |
| johnsom | ++ /opt/stack/new/octavia/devstack/plugin.sh:build_mgmt_network:221 : openstack security group create lb-mgmt-sec-grp | 19:44 |
| johnsom | 'SecurityGroup' object has no attribute 'keys' | 19:44 |
| *** wolverineav has joined #openstack-neutron | 19:45 | |
| johnsom | Looks like it started failing sometime after 1/6 11:28am pacific time | 19:48 |
| *** bobmel_ has joined #openstack-neutron | 19:49 | |
| *** bobmel has quit IRC | 19:49 | |
| *** wolverineav has quit IRC | 19:50 | |
| *** thorst has joined #openstack-neutron | 19:53 | |
| *** thorst has quit IRC | 19:54 | |
| *** rkukura has quit IRC | 19:57 | |
| *** yfried has quit IRC | 19:58 | |
| johnsom | It looks like q-svc returned a 201, so maybe a client issue? | 20:03 |
| *** ircuser-1 has quit IRC | 20:06 | |
| *** wolverineav has joined #openstack-neutron | 20:14 | |
| *** bobmel_ has quit IRC | 20:16 | |
| *** yfried has joined #openstack-neutron | 20:18 | |
| *** wolverineav has quit IRC | 20:19 | |
| *** bobmel has joined #openstack-neutron | 20:21 | |
| *** wolverineav has joined #openstack-neutron | 20:25 | |
| johnsom | Yep, if I run with python-openstackclient 3.5.0 no problem, but when I upgrade to 3.6.0 I get the failure | 20:26 |
| openstackgerrit | fumihiko kakuma proposed openstack/neutron-dynamic-routing: Enable gate_hook for tempest api test https://review.openstack.org/417707 | 20:27 |
| *** wolverineav has quit IRC | 20:29 | |
| *** bobmel has quit IRC | 20:31 | |
| *** salv-orlando has joined #openstack-neutron | 20:32 | |
| johnsom | Put a bug in: https://bugs.launchpad.net/python-openstackclient/+bug/1654887 | 20:33 |
| openstack | Launchpad bug 1654887 in python-openstackclient "Upgrade to 3.6.0 causes AttributeError: 'SecurityGroup' object has no attribute 'keys'" [Undecided,New] | 20:33 |
| *** rossella_s has quit IRC | 20:36 | |
| *** salv-orlando has quit IRC | 20:36 | |
| *** adriant has joined #openstack-neutron | 20:41 | |
| openstackgerrit | fumihiko kakuma proposed openstack/neutron-dynamic-routing: Enable gate_hook for tempest api test https://review.openstack.org/417707 | 20:43 |
| *** tbachman has joined #openstack-neutron | 20:43 | |
| *** wolverineav has joined #openstack-neutron | 20:44 | |
| *** tbachman has quit IRC | 20:46 | |
| *** wolverineav has quit IRC | 20:48 | |
| *** yfried has quit IRC | 20:49 | |
| *** wolverineav has joined #openstack-neutron | 20:54 | |
| *** moshele has quit IRC | 20:57 | |
| *** wolverineav has quit IRC | 20:59 | |
| *** gvrangan has joined #openstack-neutron | 20:59 | |
| *** baoli has joined #openstack-neutron | 21:04 | |
| *** gvrangan has quit IRC | 21:13 | |
| openstackgerrit | fumihiko kakuma proposed openstack/neutron-dynamic-routing: Enable gate_hook for tempest api test https://review.openstack.org/417707 | 21:17 |
| *** salv-orlando has joined #openstack-neutron | 21:33 | |
| *** Jeffrey4l_ has quit IRC | 21:35 | |
| *** salv-orlando has quit IRC | 21:37 | |
| *** nicolasbock has quit IRC | 21:39 | |
| *** Alex_Stef has joined #openstack-neutron | 21:41 | |
| *** salv-orlando has joined #openstack-neutron | 21:47 | |
| *** Jeffrey4l_ has joined #openstack-neutron | 21:47 | |
| *** Alex_Stef has quit IRC | 21:51 | |
| *** catintheroof has joined #openstack-neutron | 21:52 | |
| *** Jeffrey4l_ has quit IRC | 21:57 | |
| *** Jeffrey4l_ has joined #openstack-neutron | 21:58 | |
| *** prateek has quit IRC | 22:00 | |
| *** tflynn has joined #openstack-neutron | 22:02 | |
| *** aspiers has quit IRC | 22:03 | |
| *** teclator has quit IRC | 22:12 | |
| *** teclator has joined #openstack-neutron | 22:12 | |
| *** neiljerram has joined #openstack-neutron | 22:20 | |
| *** salv-orl_ has joined #openstack-neutron | 22:22 | |
| *** salv-orlando has quit IRC | 22:22 | |
| *** thorst has joined #openstack-neutron | 22:24 | |
| *** thorst has quit IRC | 22:24 | |
| *** zhhuabj has quit IRC | 22:25 | |
| *** tflynn has quit IRC | 22:31 | |
| *** fnaval has quit IRC | 22:35 | |
| *** jamielennox is now known as jamielennox|away | 22:37 | |
| *** zhhuabj has joined #openstack-neutron | 22:38 | |
| *** bobmel has joined #openstack-neutron | 22:39 | |
| *** neiljerram has quit IRC | 22:40 | |
| *** bobmel has quit IRC | 22:43 | |
| *** nmagnezi has quit IRC | 22:44 | |
| *** jamielennox|away is now known as jamielennox | 22:53 | |
| *** ianw_pto is now known as ianw | 23:01 | |
| *** grantma has quit IRC | 23:01 | |
| *** tommylikehu_ has joined #openstack-neutron | 23:08 | |
| *** ihrachys has joined #openstack-neutron | 23:11 | |
| *** tommylikehu_ has quit IRC | 23:12 | |
| *** reedip has quit IRC | 23:17 | |
| *** reedip has joined #openstack-neutron | 23:29 | |
| *** wolverineav has joined #openstack-neutron | 23:31 | |
| *** gouthamr has joined #openstack-neutron | 23:32 | |
| *** teclator_ has joined #openstack-neutron | 23:34 | |
| *** teclator has quit IRC | 23:34 | |
| *** wolverineav has quit IRC | 23:36 | |
| *** thorst has joined #openstack-neutron | 23:38 | |
| *** kevo has joined #openstack-neutron | 23:38 | |
| *** thorst has quit IRC | 23:39 | |
| *** catintheroof has quit IRC | 23:41 | |
| *** itisha has joined #openstack-neutron | 23:43 | |
| *** reedip has quit IRC | 23:43 | |
| *** teclator_ has quit IRC | 23:46 | |
| *** teclator has joined #openstack-neutron | 23:46 | |
| *** salv-orlando has joined #openstack-neutron | 23:48 | |
| *** salv-orl_ has quit IRC | 23:48 | |
| *** baoli has quit IRC | 23:48 | |
| *** wolverineav has joined #openstack-neutron | 23:51 | |
| *** wolverineav has quit IRC | 23:56 | |
| *** reedip has joined #openstack-neutron | 23:57 | |
| *** ijw has quit IRC | 23:59 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!