Monday, 2013-12-23

« Sunday, 2013-12-22 Index Tuesday, 2013-12-24 »
*** markwash has joined #openstack-neutron00:40
*** WackoRobie has joined #openstack-neutron00:40
*** yamahata has joined #openstack-neutron00:47
*** clev has quit IRC00:52
*** markwash has quit IRC01:19
*** WackoRobie has quit IRC01:20
*** WackoRobie has joined #openstack-neutron01:21
*** markwash has joined #openstack-neutron01:21
*** WackoRob_ has joined #openstack-neutron01:24
*** WackoRobie has quit IRC01:24
*** WackoRob_ has quit IRC01:26
*** zzelle has quit IRC01:36
*** harlowja has joined #openstack-neutron01:55
*** Jianyong has joined #openstack-neutron02:02
*** krast has joined #openstack-neutron02:15
*** WackoRobie has joined #openstack-neutron02:16
*** WackoRobie has quit IRC02:16
openstackgerritGary Duan proposed a change to openstack/neutron: FWaaS integration with service type framewrok  https://review.openstack.org/6069902:19
*** clev has joined #openstack-neutron02:22
*** krast has quit IRC02:28
openstackgerritA change was merged to openstack/python-neutronclient: Remove a debugging print statement  https://review.openstack.org/6261702:31
*** krast has joined #openstack-neutron02:31
*** WackoRobie has joined #openstack-neutron02:31
*** clev has quit IRC02:32
*** WackoRobie has quit IRC02:32
*** markwash has quit IRC02:32
*** markwash has joined #openstack-neutron02:35
*** alexpilotti has quit IRC02:38
*** vkozhukalov has joined #openstack-neutron02:54
*** harlowja is now known as harlowja_away02:55
*** alexpilotti has joined #openstack-neutron02:59
*** alexpilotti has quit IRC03:07
openstackgerritZhang Hua proposed a change to openstack/neutron: Clean up ML2 Manager  https://review.openstack.org/6135103:08
*** yamahata has quit IRC03:25
*** yamahata has joined #openstack-neutron03:28
*** WackoRobie has joined #openstack-neutron03:42
*** WackoRobie has quit IRC03:47
*** gongysh has joined #openstack-neutron04:01
*** jecarey has quit IRC04:08
*** HenryG has quit IRC04:22
*** harlowja_away is now known as harlowja04:31
*** harlowja has quit IRC04:35
*** networkstatic is now known as networkstatic_zZ04:37
*** networkstatic_zZ is now known as networkstatic04:43
*** chandankumar has joined #openstack-neutron04:44
*** HenryG has joined #openstack-neutron04:51
*** markwash has quit IRC05:08
*** yfried has quit IRC05:29
*** markwash has joined #openstack-neutron05:30
*** markwash has quit IRC05:36
openstackgerritenikanorov proposed a change to openstack/neutron: Fix empty network deletion in db_base_plugin for postgesql  https://review.openstack.org/6359706:15
*** irenab_ has joined #openstack-neutron06:16
*** yfried has joined #openstack-neutron06:19
*** ljjjustin has joined #openstack-neutron06:22
*** AMike has quit IRC06:30
*** ashaikh has joined #openstack-neutron06:32
*** gdubreui has quit IRC06:34
openstackgerritJenkins proposed a change to openstack/neutron: Imported Translations from Transifex  https://review.openstack.org/6305606:38
*** vkozhukalov has quit IRC06:50
*** ashaikh has quit IRC06:53
*** otherwiseguy has quit IRC06:54
*** yfried has quit IRC06:56
*** yfried has joined #openstack-neutron07:02
*** yfried has quit IRC07:05
*** yfried has joined #openstack-neutron07:06
*** garyk has joined #openstack-neutron07:15
*** networkstatic has quit IRC07:21
*** jlibosva has joined #openstack-neutron07:39
*** evgenyf has joined #openstack-neutron07:43
marunsalv-orlando: poing07:51
salv-orlandomarun: piong07:52
marunsalv-orlando: I'm not sure I see how the notification patch could possibly cause problems.07:52
salv-orlandomarun: I'm pretty sure it's not your patch causing those failures07:52
salv-orlandoI just wanted a confirmation from you07:52
marunsalv-orlando: i'm also not clear on why successful resource creation has anything to do with agent's being up or down07:53
marunsalv-orlando: I'm tempted to -1 the devstack patch accordingly07:53
salv-orlandogo for it07:53
salv-orlandobut you should not -1 something because it's not clear for you :)07:53
marunsalv-orlando: i don't want to be a jerk, though.  what am i missing?07:53
salv-orlandoseriously, I can explain. Have you looked at the comment I left on bug 125389607:54
*** csd has quit IRC07:54
marunsalv-orlando: why on earth would resource creation (setting db state) have anything to do with propagating that state to an agent?07:54
marunsalv-orlando: looking...07:54
salv-orlandoI can explain here...07:54
marunsalv-orlando: ok07:54
salv-orlandobasically that happens because when you create a port with the ml2 plugin, it does something which is a called a binding.07:55
salv-orlandobasically associate either the local VLAN or the provider attributes07:55
salv-orlandoto do so the ml2 plugin calls the "agent mechanism driver"07:55
salv-orlandowhich if there is no agent available, just does not create this binding07:55
salv-orlandothen when the agent asks for the port details, since there are no bindings the plugin does not return the details07:56
salv-orlandoand the port is not wired.07:56
salv-orlandoNow devstack starts neutron server, creates the resources and then starts the agent.07:56
salv-orlandoYour next question should be "why this does not happen always then?" :)07:57
salv-orlandobecause the port which does not get wired is the DHCP port, which is created asynchronously when the dhcp agent receives a notification that the subnet has been created.07:57
marunso is the ml2 plugin just retarded??07:57
salv-orlandoTherefore we are observing what is a race between the dhcp and the ovs agent07:57
*** amuller has joined #openstack-neutron07:58
salv-orlandoI think there might be something to fix wrt this behaviour in the ml2 plugin as well.07:58
salv-orlandoindeed if you're unable to bind a port to a "segment", one should not put the port "DOWN" as the consumers of the neutron API might expect it to go UP eventually07:58
salv-orlandothe port should be put in ERROR07:58
marunsalv-orlando: right07:59
salv-orlandobut perhaps there's even more to do in the ml2 plugin. It's just that we need to involve the people which have been developing the mechanism driver framework07:59
marunsalv-orlando: <expletive deleted>07:59
salv-orlandoand I would like to stop seeing these failures for  bug 125389607:59
marunsalv-orlando: can you explain this port binding thing again?08:00
salv-orlandomarun: if what you've deleted relates to ml2 developers not being involved as expected in bug triaging and fixing, I kind of understand your concern.08:00
salv-orlandomarun: sure.08:00
salv-orlandowhen a port is created, ml2 plugin binds it to a "segment". A segment is either a local vlan identifier or a provider network specification. In the case of gate tests it's always a local vlan id. This local vlan id will be set by the agent on the ovs port.08:01
marunlocal vlan id??08:01
marunare you saying the ml2 plugin now allocates the local vlan on the neutron server instead of it being done at the agent level like with the ovs plugin?08:02
* marun confused08:02
*** amuller has quit IRC08:03
salv-orlandothis is what I read from the code. I am confused either because  I did not understand the necessity of it. However, I do not have enough info on the thought process which led to this decision.08:03
*** bvandenh has quit IRC08:03
salv-orlandoBottom line is that if there is no agent for a port, this "segment binding" is not created.08:03
*** amuller has joined #openstack-neutron08:03
marunsalv-orlando: <expletive deleted>08:04
salv-orlandowhen this happens, get_device_details in neutron/plugins/ml2/rpc.py does not return details for the port, and as a consequence the agent puts the port on the DEAD VLAN08:04
salv-orlandoas you might understand, I agree there is a probably an action to be taken on the ML2 plugin, but yesterday I thought that this might involve engaging with the developers, rediscussing the issue, reaching consensus on a solution, and then finding someone to implement it.08:05
marunsalv-orlando: ok, so I'll agree to not -1 the devstack patch if we file a bug ensuring this massive cock-up gets fixed.08:05
salv-orlandomarun: and this is why I went for the devstack patch.08:05
marunsalv-orlando: understood.08:06
salv-orlandoI will create a bug report, hopefully tonight in the neutron meeting I will find some time to query some ml2 dev08:06
marunsalv-orlando: cool.08:06
* marun regrets not looking closely at ml2 08:06
salv-orlandoI tried yesterday on IRC, but as you might expect the room was desert.08:06
marunsalv-orlando: weekend before western holidays?  yeah08:07
salv-orlandomarun: I think I already told you what I regret about ml208:07
marunsalv-orlando: I don't think so - maybe you can enlighten me?08:07
salv-orlandoWe probably rushed a bit in making it the default plugin.08:07
marunsalv-orlando: ah, yes.08:08
marunsalv-orlando: given the bug in question, I would entirely agree with that assessment.08:08
marunsalv-orlando: The fact that it was allowed to introduce a dependency on agent liveness indicates a lamentable lack of distributed systems experience on the part of everyone involved.  :(08:09
marunsalv-orlando: (though that criticism could probably be leveled at large swaths of openstack)08:10
*** dzyu has joined #openstack-neutron08:10
salv-orlandomarun: your last comment is correct. The fact is that I'm the first who make mistakes when it comes to distributed systems designs. And I did not get a good peer review as I usually do, I would have made errors which are far worse.08:11
salv-orlandolast sentence was supposed to start with "And If I did not get..."08:11
marunsalv-orlando: Hey, doesn't armax have a phd in distributed systems?08:14
*** ljjjustin has quit IRC08:14
salv-orlandomarun: technically I too have it08:14
salv-orlandonote the "technically"08:14
salv-orlandomarun: and I even TA-ed the distributed system class and implemented demo versions for student of several consensus protocols including paxos08:15
salv-orlandomarun: but still, I make poor mistakes08:15
marunsalv-orlando: oy.  optimistically, at least, things can only improve ;)08:16
marunsalv-orlando: and I get it - this stuff is hard.  we don't know our own ignorance08:17
salv-orlandomarun: right. I will report this bug, and then seek to elicit some feedback from the ML2 team. I might be ok to keep agents and server loosely coupled as they are now, I think the server should just not made assumptions like the one that an agent is always available, and if it does, take appropriate reactions - like failing the port create or setting it in ERROR state as like nova does when there is no compute node to schedule an instance08:19
marunsalv-orlando: ok08:20
marunsalv-orlando: why is it ok to require the agent to be running though?08:20
marunsalv-orlando: why can't the server decide and notify the agent?08:21
marunsalv-orlando: or is the implication that both the server and the agent are now making decisions that have to be synchronized?08:21
* marun feels a headache coming on08:21
salv-orlandomarun: I can't give an answer to this question without hearing from the people who designed and developed the ml2 mechanism framework.08:27
marunsalv-orlando: fair enough08:27
salv-orlandoguesswork leads to headaches, and I don't take pills08:27
marunsalv-orlando: :)08:27
*** vkozhukalov has joined #openstack-neutron08:29
*** ygbo has joined #openstack-neutron08:43
*** rkukura has quit IRC08:45
*** rkukura has joined #openstack-neutron08:45
marunrkukura: are you really up08:47
marun?08:47
*** fouxm has joined #openstack-neutron08:50
*** jlibosva has quit IRC08:50
*** jlibosva has joined #openstack-neutron08:51
*** salv-orlando_ has joined #openstack-neutron08:54
*** rossella_s has joined #openstack-neutron08:56
*** salv-orlando has quit IRC08:57
*** salv-orlando_ is now known as salv-orlando08:57
*** safchain has joined #openstack-neutron08:58
openstackgerritAaron Rosen proposed a change to openstack/neutron: Bump api_workers from 0 to 4  https://review.openstack.org/5978709:02
*** salv-orlando has quit IRC09:07
*** salv-orlando has joined #openstack-neutron09:07
*** ijw has joined #openstack-neutron09:13
*** ihrachyshka has quit IRC09:14
openstackgerritSylvain Afchain proposed a change to openstack/neutron: Dnsmasq uses all agent IPs as nameservers  https://review.openstack.org/6106709:15
marunsalv-orlando: ping09:18
salv-orlandohi martun09:18
salv-orlandomarun09:18
marunsalv-orlando: so, I'm guilty of thinking too much...09:18
marunsalv-orlando: I have one question, and then I'm hoping you would be willing to review the first part of a design doc regarding the dhcp agent09:19
salv-orlandomarun: and now you've got an headache? I'm sorry I'm out of drugs09:19
marunsalv-orlando: (i promise, it's not too painful)09:19
salv-orlandomarun: go ahead09:19
marunsalv-orlando: are networks that aren't isolated possible?09:20
marunsalv-orlando: i.e. is neutron going to be ns-isolated only at some point in the future09:20
marunsalv-orlando: the question is related to the safety of applying state changes to multiple networks out-of-order09:21
salv-orlandomarun: I think so, but I don't know when that will happen. I think the option of running neutron without overlapping ips (hence no namespaces), will stay for havana09:22
marunsalv-orlando: I guess the potential exists in that scenario for db changes across network to be applied in the wrong order by the dhcp agent09:22
salv-orlandomarun: perhaps that will go away once all major linux distros will support namespace, but then there's the user perspective (there are some who really don't want to use namespaces - not guessing here, I've met at least one of those users)09:22
marunsalv-orlando: Arg09:23
salv-orlandomarun: then that could be a problem regardless of whether we're using namespaces or not, I guess?09:23
marunsalv-orlando: No, I don't think so.09:23
marunsalv-orlando: so, example.09:23
marunsalv-orlando: network A has allocation range X1,X2 and network B has allocation range Y1,Y209:24
marunsalv-orlando: network A kills all ports on allocation range X2 and drops it, network B adds allocation range equivalent to X2 and adds ports09:24
marunsalv-orlando: with network isolation, this doesn't matter.09:24
*** ijw has quit IRC09:25
marunsalv-orlando: without network isolation, well, it's possible that if we don't globally order operations we could have ports on X2 still there while ports are allocated on the new range on network b09:25
marunsalv-orlando: does that make sense?09:25
*** ijw has joined #openstack-neutron09:25
marunsalv-orlando: In havana, well, there is no guarantee of notification ordering, so things are just going to suck anyway09:26
marunsalv-orlando: if we can guarantee notification ordering per-network, though, network isolation will prevent anything bad from happening cross-network09:26
salv-orlando609:26
salv-orlando3509:26
salv-orlandomarun: sorry cat on keyboard09:26
marunheh09:27
salv-orlandomarun: summarising, the current handling of port_update, but even just stashing all the messages in a set and then processing them in the main loop creates a window of opportunity for some range of IPs to be defined at the same time on more than one dnsmasq instance.09:28
salv-orlandois that correct?09:28
marunsalv-orlando: yes09:29
salv-orlandoand the way for sorting it would be to put notifications in a queue and serially processing them?09:29
marunsalv-orlando: I have something prepared, I'll email.09:29
salv-orlandomarun: the only thing I have to say abut serial processing is that in our cloud, where we use the dhcp agent, we found out that at some scale (500 networks for instance), the dhcp agent would take about 35 minutes to do the initial synchronization09:31
salv-orlandothis is why arosen did that think about parallel processing for dhcp namespaces09:31
salv-orlandothink/thing09:31
marunhold that thought09:31
salv-orlandoholding that09:31
marunsent to your gmail09:31
salv-orlandook09:31
marunsalv-orlando: this is just the start - I haven't written the part about how notifications are actually processed.  suffice to say, though, that operation ordering only has to be (necessarily) per-network, and there will be room to optimize by properly separating db to agent sync from agent to dnsmasq sync.09:34
salv-orlandoI am reading it, but I won't be able to answer immediately. At first glance it makes sense - the problem with agent counters reminds me of vector clocks but I'm not sure. I think it's a good idea to send it to the mailing list.09:34
marunsalv-orlando: I intend to post it as a blueprint design, I was just hoping to get some initial feedback from you as to whether I was missing anything obvious.09:34
marunsalv-orlando: it is kind of like vector clocks, but since we have the db we don't have to take into account multiple hosts.  we have one source of truth - the db.09:35
salv-orlandoI will be able to send you my feedback in a few hours - I'm sorry for the delay, but I have something else I'm trying to finish for today.09:35
openstackgerritenikanorov proposed a change to openstack/neutron: Add security groups tables for ML2 plugin via migration  https://review.openstack.org/6358509:36
salv-orlandoanyway, from a process perspective, something else to think about is how to implement this incrementally so we do not end up with a bunch of large patches to review a few days before the I-3 deadline09:36
marunsalv-orlando: no worries.  I'm going to sleep now anyway.  :)09:36
*** ijw has quit IRC09:36
marunsalv-orlando: I don't think this will be too complicated, and I'm hoping to make the rational clear to reviewers via a clear design doc.09:37
salv-orlandomarun: cool.09:37
marunsalv-orlando: the 3 phases I anticipate are 1) ordered notifications, 2) separation of db to agent and agent to dnsmasq sync, and 3) optimize for performance09:38
marunsalv-orlando: I'm hoping this effort might prove useful for the other agents if they have a similar issue with processing notifications out of order.09:39
salv-orlandoI'm not sure if markmcclain is cooking as well something for #2, you might try to ping him tomorrow09:39
marunsalv-orlando: i think he is, yes.09:40
marunsalv-orlando: I'm less clear on whether he's doing #1, though, so that effort at least might be useful.09:42
salv-orlandomarun: I and mark have never talked about #109:44
marunsalv-orlando: I think it's going to impact all of the agents.09:45
marunsalv-orlando: especially when multiple api workers are used.09:45
salv-orlandomarun: Sure. It does not have to be something specific to the dhcp agent.09:46
marunsalv-orlando: 'nite!09:51
salv-orlandomarun: good night09:51
marunsalv-orlando: thank you for the conversation, helpful as always.09:51
marun:)09:51
*** majopela|lunch has joined #openstack-neutron09:51
salv-orlandomarun: you're wlecome09:51
*** chandankumar has quit IRC09:56
openstackgerritYves-Gwenael Bourhis proposed a change to openstack/neutron: Make dnsmasq aware of all names  https://review.openstack.org/5293009:58
*** chandankumar has joined #openstack-neutron09:59
*** ijw has joined #openstack-neutron10:00
*** dzyu has quit IRC10:01
*** chandankumar has quit IRC10:06
*** chandankumar has joined #openstack-neutron10:08
*** majopela|lunch has quit IRC10:10
enikanorov__salv-orlando: hi. I wanted to ask about whether fixing bugs of ovs plugin is something desirable? afaik it is deprecated in favor of ml210:17
salv-orlandoidk but as long as it's there if there's a bug i'd fix it, unless it requires major changes in the plugin code10:23
enikanorov__ok i see10:23
*** evgenyf has quit IRC10:24
*** rossella_s has quit IRC10:28
gongyshping garyk10:28
gongyshping amotoki10:29
gongyshping safchain10:30
safchaingongysh, Hi10:30
gongyshsafchain: how is the l3 agent HA going?10:31
safchaingongysh, I hope and I plan to submit the first patches next week10:31
*** ijw has quit IRC10:32
gongyshok thanks10:33
*** gongysh has quit IRC10:33
*** evgenyf has joined #openstack-neutron10:48
*** rossella_s has joined #openstack-neutron10:50
*** Jianyong has quit IRC10:52
*** bvandenh has joined #openstack-neutron10:58
*** markvoelker has quit IRC11:04
*** bvandenh has quit IRC11:09
QlawyHi, I have an issue with neutron and traffic to "outside"11:13
QlawyI have all-in-one installation11:14
QlawyI can ping vms from openstack, I can ping openstack from vms11:14
Qlawybut11:14
Qlawywhen I try to ping vms from another machine which is in same network as openstack I cant do this11:14
Qlawywhen I want to ping that host from VM, I also cant do this11:15
Qlawyi can bet its some kind of security-group/rules issue but no idea how to fix it11:16
Qlawyanother way to solve it is to connect that "another machine" into neutrons network ;/11:17
openstackgerritBerezovsky Irena proposed a change to openstack/neutron: Add update from agent to plugin on device up  https://review.openstack.org/5360911:29
*** rossella_s has quit IRC11:29
*** krast has quit IRC11:31
openstackgerritBerezovsky Irena proposed a change to openstack/neutron: Add update from agent to plugin on device up  https://review.openstack.org/5360911:54
*** zzelle has joined #openstack-neutron11:56
openstackgerritmouad benchchaoui proposed a change to openstack/neutron: Make the metadata namespace proxy transparent  https://review.openstack.org/2813712:00
openstackgerritOleg Bondarev proposed a change to openstack/neutron: LBaaS: agent monitoring and instance rescheduling  https://review.openstack.org/5974312:03
*** jlibosva has quit IRC12:38
sdaguesalv-orlando: so I think https://bugs.launchpad.net/tempest/+bug/1253896 has changed it's signature, I'm now seeing a lot of no route to host issues12:42
sdaguehttp://logs.openstack.org/30/63530/3/gate/gate-tempest-dsvm-neutron/11c8e98/console.html#_2013-12-23_12_39_15_61612:47
*** jlibosva has joined #openstack-neutron12:52
*** b3nt_pin has quit IRC12:55
*** ashaikh has joined #openstack-neutron12:59
*** yamahata has quit IRC13:02
*** yamahata has joined #openstack-neutron13:03
*** aymenfrikha has joined #openstack-neutron13:10
*** yamahata has quit IRC13:15
*** b3nt_pin has joined #openstack-neutron13:19
*** b3nt_pin is now known as beagles13:20
*** markwash has joined #openstack-neutron13:31
*** yamahata has joined #openstack-neutron13:32
*** Jianyong has joined #openstack-neutron13:37
*** irenab_ has quit IRC13:47
*** jdev789 has joined #openstack-neutron13:48
*** julim has joined #openstack-neutron13:54
openstackgerritSylvain Afchain proposed a change to openstack/neutron: L3 Agent can handle many external networks  https://review.openstack.org/5935913:55
openstackgerritSylvain Afchain proposed a change to openstack/neutron: L3 Agent can handle many external networks  https://review.openstack.org/5935913:58
*** WackoRobie has joined #openstack-neutron14:00
*** yfried has quit IRC14:00
*** yfried has joined #openstack-neutron14:13
*** ihrachys has quit IRC14:16
*** ihrachys has joined #openstack-neutron14:17
*** ihrachys has quit IRC14:17
*** ihrachys has joined #openstack-neutron14:17
*** ihrachys has quit IRC14:18
*** ihrachys has joined #openstack-neutron14:19
*** alexpilotti has joined #openstack-neutron14:20
*** markwash has quit IRC14:24
*** ihrachys has quit IRC14:30
*** ihrachys has joined #openstack-neutron14:31
*** rossella_s has joined #openstack-neutron14:41
*** bvandenh has joined #openstack-neutron14:46
*** clev has joined #openstack-neutron14:48
*** nijaba has quit IRC14:50
openstackgerritSylvain Afchain proposed a change to openstack/neutron: L3 Agent can handle many external networks  https://review.openstack.org/5935914:51
*** nijaba has joined #openstack-neutron14:57
*** nijaba has quit IRC14:57
*** nijaba has joined #openstack-neutron14:57
*** rwsu has joined #openstack-neutron14:59
*** rustlebee is now known as russellb14:59
*** nijaba has quit IRC15:01
*** nijaba has joined #openstack-neutron15:04
*** WackoRobie has quit IRC15:19
*** Jianyong has quit IRC15:28
*** jlibosva has quit IRC15:30
openstackgerritXiang Hui proposed a change to openstack/neutron: Add options for all commands executing run_vsctl  https://review.openstack.org/5895415:32
*** otherwiseguy has joined #openstack-neutron15:39
*** ashaikh has quit IRC15:46
openstackgerritAndreas Jaeger proposed a change to openstack/python-neutronclient: Fix description of ListSubnet  https://review.openstack.org/6361915:46
*** rossella_s_ has joined #openstack-neutron15:51
openstackgerritAleks Chirko proposed a change to openstack/neutron: Bugfix and refactoring for ovs_lib flow mgnt methods  https://review.openstack.org/5853315:52
*** rossella_s has quit IRC15:52
*** rossella_s_ is now known as rossella_s15:52
*** yfried has quit IRC15:56
*** yfried has joined #openstack-neutron15:57
*** vkozhukalov has quit IRC15:57
*** ihrachys has quit IRC16:03
*** SumitNaiksatam has quit IRC16:08
*** yfried has quit IRC16:08
*** garyk has quit IRC16:13
openstackgerritA change was merged to openstack/neutron: BigSwitch: Fixes floating IP backend updates  https://review.openstack.org/6304716:13
openstackgerritAndreas Jaeger proposed a change to openstack/python-neutronclient: Fix description of ListSubnet  https://review.openstack.org/6361916:13
*** WackoRobie has joined #openstack-neutron16:18
*** SumitNaiksatam has joined #openstack-neutron16:30
*** rossella_s_ has joined #openstack-neutron16:48
*** rossella_s has quit IRC16:49
*** rossella_s_ is now known as rossella_s16:49
*** rossella_s has quit IRC16:58
*** yamahata has quit IRC16:58
*** jdev789 has quit IRC17:02
*** rossella_s has joined #openstack-neutron17:05
*** alexpilotti has quit IRC17:07
*** mlavalle has joined #openstack-neutron17:11
anteayarossella_s mlavalle beagles hello17:12
anteayaI have to leave for a meeting very soon17:12
beagleshi17:12
mlavalleanteaya: hello17:12
anteayaso if you three want to have an update17:12
anteayaI encourage it17:12
anteayaand I will read the backscroll when I return17:12
anteayahow are you doing?17:12
mlavallewe are scheduled to talk at 18:00UTC17:12
rossella_santeaya: hello17:13
anteayaokay great17:14
anteaya45 minutes17:14
anteayaI won't be here for it, so don't wait for me17:14
mlavalleok, will do  and keep you posted17:15
anteayathanks17:17
*** bvandenh has quit IRC17:18
*** ygbo has quit IRC17:27
*** AndreyGrebenniko has quit IRC17:27
*** akamyshnikova_ has quit IRC17:27
*** skraynev has quit IRC17:27
*** obondarev has quit IRC17:27
*** enikanorov has quit IRC17:27
*** skraynev has joined #openstack-neutron17:28
*** akamyshnikova_ has joined #openstack-neutron17:28
*** enikanorov has joined #openstack-neutron17:28
*** obondarev has joined #openstack-neutron17:30
openstackgerritAleks Chirko proposed a change to openstack/neutron: Bugfix and refactoring for ovs_lib flow mgnt methods  https://review.openstack.org/5853317:30
*** ijw has joined #openstack-neutron17:30
openstackgerritSvetlana Dobogoeva proposed a change to openstack/neutron: Added unit tests for module neutron/plugins/nicira/api_client/client.py  https://review.openstack.org/5994817:33
*** amuller has quit IRC17:33
anteayaenikanorov__: this is currently the #2 gate blocking bug: https://bugs.launchpad.net/nova/+bug/125489017:36
anteayadims has offered a patch which he links to at the bottom of the bug report17:36
anteayaif you have the time to evaluate the patch and review it prior to the meeting, we might be able to get that patched merged and that bug addressed today17:37
anteayawould be great if that happened, if you feel the patch addresses the bug17:37
anteayathanks17:37
enikanorov__anteaya: yes, i have seen this. however i think that the patch needs to be tested against jobs on which the failures are seen17:37
enikanorov__i mean neutron-pg and neutron-pg-isolated17:37
anteayaenikanorov__: great17:37
enikanorov__and nova patches are not tested against them17:37
anteayaokay17:37
anteayawhat needs to happen?17:37
*** yfried has joined #openstack-neutron17:37
enikanorov__so in fact i don't mind letting this patch to land, but i'm not a nova expert and neither nova core17:38
enikanorov__i just would like to see it tested with those jobs17:38
anteayayfried: you would like a patch tested against https://review.openstack.org/#/c/62702/ which is failing in jenkins17:39
anteayayfried: what do you need to do to edit the patch so that it passes tests?17:39
anteayaenikanorov__: that is fair17:39
anteayaenikanorov__: I would like to stay and help you get the testing done to your statisfaction17:40
anteayabut if I do that I will be late for an appointment17:40
anteayacan you ask clarkb or jeblair perhaps in -infra to see if they can help you learn how to get the tests to run on or with that patch?17:41
enikanorov__i'll talk to dims about it17:41
anteayaI think you may have asked on the weekend but it was very quiet on the weekend17:41
anteayathanks17:41
enikanorov__afaik that will require to create specifal configuration so 'check experimental' could run neutron-pg* jobs agains nova patches17:41
* anteaya is afk for the rest of the day17:41
anteayaenikanorov__: okay17:41
anteayado you know how to create a check experiemental job?17:42
anteayathis patch might help provide a bit of a template: https://review.openstack.org/#/c/62930/17:42
*** safchain has quit IRC17:42
*** amrit has quit IRC17:43
*** harlowja has joined #openstack-neutron17:50
*** ijw has quit IRC17:50
*** rossella_s_ has joined #openstack-neutron17:52
*** rossella_s has quit IRC17:54
*** rossella_s has joined #openstack-neutron17:55
*** garyk has joined #openstack-neutron17:56
*** rossella_s_ has quit IRC17:57
*** ijw has joined #openstack-neutron18:00
rossella_smlavalle: beagles: hello! shall we start?18:01
mlavallerossella_s, beagles: hi18:01
mlavalleyeah18:01
mlavallelet's do it18:01
rossella_sdid you receive my email?18:01
beaglesyup18:01
rossella_shttps://etherpad.openstack.org/p/full-neutron-gate-tests18:01
mlavallerossella_s: thank you for th runs and the log files18:01
mlavalleand the etherpad18:02
rossella_smlavalle: my pleasure18:02
mlavalleAt first look, the picture doesn't look that ugly18:02
rossella_sindeed18:02
mlavallethere are 3 or 4 tests failing consistently18:02
rossella_syes18:02
rossella_sbut there's always some random failure18:03
mlavalleand 3 of them have to do with ssh to a floating ip18:03
rossella_syes18:03
rossella_swhat's usually the cause of that?18:03
mlavalleright now thee is a lot of activity trying to fix floating ip propagation and timeouts18:04
beaglestime seems to be the cause of that most of the time I think18:04
mlavalleI know yfried has been working over the past few days specifically on that18:04
mlavallein fact, he is been working with two of the scripts we are seeing failling18:05
mlavalletest_network_basic_ops and test_cross_tenant_connect|vity18:05
beaglesmmm.. afaik that is really only that he is removing the timeout on the floating IP propagation, that won't affect the other failure modes18:06
mlavalleso it seems to me that it is logical we are seeing these issues in this jjob18:06
rossella_sbeagles: what are the other failure mode?18:06
rossella_ss/mode/modes18:06
beaglesping and ssh themselves failing18:06
beaglesthe floating IP check was something I added because the ping test was failing on me all fo the time and I felt it was cart before horse to ping before we had some idea that it had actually been alloated18:07
mlavalleahhhh, ok18:07
beaglesbut really in the end I don't think it'll make a difference.. it takes too long for the floating IP to become active in some cases18:07
beaglesand to fix the tests, we'll need to fix that performance issue... still...18:08
beaglesif it takes a long time for the vm to become active, it will still fail18:08
mlavalleso, beagles, is that performance issue with the FLIP's and VM's something you are activey involved?18:09
beaglesum not active in the openstack sense, sorry... but with actually booting etc.18:09
rossella_swho's working on that?18:09
mlavallewould you say we need to involve make yfried aware of what we are seeing?18:09
beaglesmlavalle: not at the moment. You could say that it is vying for top spot18:09
beaglesyfried is aware.. or should be, we discussed it a length18:10
beaglesthere is an email thread as well... test_network_basic_ops and the "FloatingIPChecker" control point18:10
beagleson openstack-dev18:10
mlavalleso I suggest that one action is to track what is happening with this discussion on test_network_basic_ops and the "FloatingIPChecker". If that gets fixed, I think we won't see these issues in the job anymore18:13
mlavalledoes this makes sense to you?18:13
rossella_smlavalle: I agree18:13
beaglessure18:13
mlavallebefore, we distribute tasks, let's discuss another issue18:14
rossella_sthere's another test that is consistently failing, tempest.cli.simple_read_only.test_neutron.SimpleReadOnlyNeutronClientTest.test_neutron_security_group_rule_list18:14
mlavalleI didn't have time to analyze that one18:14
mlavallewhat did you find?18:14
rossella_smlavalle: sorry, I haven't checked either18:14
rossella_sI can do that later18:14
mlavallethat's fine, I thought you had gone through it18:15
mlavalleso, let me bring up another issue18:15
*** vkozhukalov has joined #openstack-neutron18:15
mlavalleI went through the console log of the job18:15
mlavalleyou will fin several instances of skipped tests18:16
mlavallefor example tempest.api.compute.servers.test_virtual_interfaces.VirtualInterfacesTestJSON.test_list_virtual_interfaces[gate] ... skipped u'Skipped until Bug: 1183436 is resolved.18:17
*** rossella_s_ has joined #openstack-neutron18:17
mlavalleif you look at bug 1183436, https://bugs.launchpad.net/tempest/+bug/118343618:18
*** rossella_s has quit IRC18:19
*** rossella_s_ is now known as rossella_s18:19
mlavalleyou will see that the tests method is skipped because Neutron cannot list virtual interfaces18:19
mlavallethe way nova can18:19
mlavalleso a few months ago, someone decided to just skip the test18:20
mlavalleam I making sense so far?18:20
beaglesyup18:20
mlavalleso, when we go to the nova / nautron parity conversation, what do these skips mean?18:21
mlavalleI can assure you there are several of them18:21
mlavallein the tempest code18:21
beaglesyeah, I would expect so.. 1s18:21
*** WackoRobie has quit IRC18:21
dimsenikanorov, hop onto -infra channel18:22
beagleshrmm18:22
mlavalleI think we need to bring this to the attention to the larger Neutron team18:22
mlavalleand as a team make the decision whether we skip those tests or we are going to work to make those tests run18:22
beagleswell.. as part of the parity question we need to triage the bugs related to same and figure out if they need to be implemented or not18:23
beaglesin September there were several APIs in the nova-neutron interface that hadn't been implemented18:24
mlavallebeagles: yeap, his is the conversation I want to have18:24
beaglesarosen, by virtue of volunteering to take the lead on that module, could use that info for sure18:25
mlavalleit seems to me that the very next step in this regard is to go thorugh the log of the job and find all the skips that might be relevant and then proceed to that triaging18:25
*** alexpilotti has joined #openstack-neutron18:25
mlavallemakes sense?18:25
beaglesyup18:25
mlavallehow about you rossella_s18:26
mlavalle?18:26
rossella_smlavalle: yes, makes sense18:26
beaglesin some cases it is pretty easy on the code from, just search for NotImplementedError() exceptions :)18:26
beagless/from/front/18:26
rossella_s:)18:27
mlavalleok, so to summarize, we have two next steps:18:27
mlavalle1) follow up with the FLIP propagation / timeout fixes and make sure they are going to contribute to fix this job18:28
mlavalle2) Compile all the skipped tests in the job that are relevant to the Neutron / nova parity convrsation and start triaging them18:29
mlavalleam I missing something?18:29
rossella_s3) check the SimpleReadOnlyNeutronClientTest.test_neutron_security_group_rule_list failure18:29
mlavalleyes. good catch18:30
mlavallebeagles: could you help with 1? seems like you've been closer to the FLIP's conversation18:32
beaglesmlavalle: yes18:32
mlavallerossella_s: would you take 2 and 3?18:33
*** fouxm has quit IRC18:33
rossella_smlavalle: yes, np. But I'm on holidays till the 3rd.18:34
*** fouxm has joined #openstack-neutron18:34
beaglesactually same here.. who is *not* off until 2014?18:35
rossella_s:D18:35
mlavallerossella_s: let's do something then…. I will make as much progress as I can on this from now until the 3rd18:35
mlavallewhen you come back and you take it from there18:35
rossella_smlavalle: ok, deal!18:35
mlavalleI might not be able to do a lot, becuase I'm working on the api tests stuff18:35
mlavallebut i'll give it a try18:35
rossella_sI am working on that too. I hope my patches get merged today18:36
mlavalleone final thing< I would like to update the larger team today on what we have agreed18:36
mlavallerossella_s: would you like to do that update?18:37
rossella_smlavalle: you mean at the neutron meeting?18:37
mlavalleyes18:37
rossella_syes, ok.18:38
rossella_s:)18:38
mlavalleok, I will put your name down in the agenda18:38
mlavalleanything else?18:38
rossella_sso what should I say? just a small summary of what we have been doing and our action list?18:38
mlavalleyeah, just that we've been working on this and that we have 3 action items18:39
rossella_smlavalle: ok18:39
rossella_ssound good18:39
rossella_ss/sound/sounds18:39
mlavalleI think we are done18:39
rossella_syes! bye! see you later at the neutron meeting!18:40
mlavalleenjoy your time off!!!18:40
mlavalleHappy holidays!18:40
rossella_syes! you too, I hope you are taking some day off18:40
mlavalleyeah, some days18:40
mlavalle:-)18:40
rossella_senjoy then!18:41
mlavallebeagles: talk to you later18:41
beaglescheers18:41
*** markmcclain has joined #openstack-neutron18:46
*** ijw has quit IRC18:47
*** ijw has joined #openstack-neutron18:47
*** rossella_s_ has joined #openstack-neutron18:51
*** rossella_s has quit IRC18:52
*** rossella_s_ is now known as rossella_s18:52
*** fouxm has quit IRC18:54
*** rossella_s_ has joined #openstack-neutron18:59
*** rossella_s has quit IRC19:02
*** rossella_s_ is now known as rossella_s19:02
yfriedmlavalle: beagles: can you guys fill me in on your discussion? Are you aware of this bug https://launchpad.net/bugs/1262529 and these patches https://review.openstack.org/#/c/63637/ https://review.openstack.org/#/c/63627/19:04
*** otherwiseguy has quit IRC19:04
beaglesyfried: yup yup and yup19:04
yfriedI would like to help, especially if this is an issue I'm already part of, but this participating during this hour is really hard for me19:05
beaglesyfried: basically it was a question regarding timeout failures that were occuring in the experimental queue with various connectivity failures19:05
yfriedbeagles: they are completely untouched. I was hoping to have your opinion on them19:05
beaglesyfried: in some cases it was ssh timing out, in one case the floating IP propagation test was timing out19:06
beaglesyfried: yeah, I'll get to them today19:08
*** ijw has quit IRC19:09
yfriedbeagles: mlavalle: I'm mostly unavailable now, so I will appreciate if you could send me your conclusions. I would like to participate in solving this issue. I will make effort to go over this discussion later (probably tomorrow) but if you could post it to the mailing list as part of the thread it would be great.19:10
*** rossella_s_ has joined #openstack-neutron19:12
mlavalleyfried: will do, thanks for your help19:12
*** ijw has joined #openstack-neutron19:12
*** garyk has quit IRC19:12
beaglesokay, who has seen "Duplicate test id detected......." in tempest.log when running testr?19:13
beaglesbloody annoying19:13
*** rossella_s has quit IRC19:14
*** rossella_s_ is now known as rossella_s19:14
*** yamahata__ has joined #openstack-neutron19:19
sdaguebeagles: it's usually always correct. You have an autosave file getting picked up or something?19:19
beaglessdague: hah, i bet that's it19:20
*** geekinutah1 has joined #openstack-neutron19:23
beaglessdague: mmm... not quite it19:25
*** marios_ has joined #openstack-neutron19:25
*** ijw has quit IRC19:25
*** ijw has joined #openstack-neutron19:26
beaglessdague: tox -esmoke seems to run, but testr based invocations do not19:26
*** yamahata_ has quit IRC19:26
*** geekinutah has quit IRC19:26
*** marios has quit IRC19:26
*** ijw_ has joined #openstack-neutron19:30
*** ijw has quit IRC19:30
sdaguebeagles: ValueError: Duplicate test id detected: tempest.api.compute.test_quotas.QuotasTestJSON.test_compare_tenant_quotas_with_default_quotas[gate,smoke]19:32
sdague ?19:32
beaglessdague: Duplicate test id detected: tempest.api.compute.test_authorization.AuthorizationTestJSON.test_change_password_for_alt_account_fails19:32
*** rossella_s_ has joined #openstack-neutron19:34
sdaguehmmm... I'm getting a different duplicate error on list here19:36
*** geekinutah1 is now known as geekinutah19:36
*** rossella_s has quit IRC19:38
*** rossella_s_ is now known as rossella_s19:38
beaglessdague: I just tried hacking testsuite.py to dump the list to stdout.. and after a bit of hacking about, there is definitely a bit of duplication where it isn't expected19:46
sdagueyeh, wonder how close to being awake lifeless is19:56
sdaguethis doesn't seem right19:56
sdagueI even went and purged all my pyc files19:56
beaglessame here19:57
*** aymenfrikha has quit IRC19:59
beaglessdague: code in testtools.py was recently (is Oct 25 recent?) changed20:05
*** armax has joined #openstack-neutron20:06
beaglesfor py 3.3 compat issue20:06
sdagueyeh, the fact that it works under tox is interesting though20:08
beaglessdague: mmm yeah20:09
lifelessbeagles: sdague: I'm just stepping out20:11
sdaguelifeless: ok, when are you stepping back?20:11
lifelesstry testtools 0.9.34 and failing that shoot me a mail and I can look later20:11
lifelesssdague: in 3 weeks :)20:11
*** ijw has joined #openstack-neutron20:11
sdaguealready using testtools 0.9.3420:12
lifelesssdague: [I will have a look my evening - about 9 hours from now probably]20:12
lifelesshowever, that test is robust, so I think it's basically an issue in tempest; may be the loader logic isn't quite right, or another possible cause20:12
lifelessis importing a class rather than a module20:13
lifelessso file a) test_foo.py20:13
lifelessclass TestBar(TestSuite):....20:13
lifelessfile b20:13
lifelessfrom test_foo import TestBar20:13
lifelesswill cause TestBase to be instantiated twice in two files.20:13
lifelessHTH, must run now.20:13
sdagueok20:13
sdaguewell I can 100% reproduce this issue20:15
*** ijw_ has quit IRC20:15
sdagueoh, I think it's the eval error problem20:17
sdaguewhere there is a bad test somewhere, and because testr / subunit / discover swallow all the useful output it shows up as a duplicate test id error20:17
beaglesoh20:18
beagleshah20:18
*** ijw has quit IRC20:19
*** ijw has joined #openstack-neutron20:20
beaglessdague: python-mock20:23
*** gdubreui has joined #openstack-neutron20:23
beaglessdague: imported indirectily through tempest/tests/test_rest_client.py20:23
beaglessdague: installing on my system resolved the issue at any rate20:23
sdagueah, yeh, that's why we created the tox targets for various test sets20:25
beaglesyup, makes sense20:25
sdagueI really just wish the toolchain would fail with messages that make debugging possible :)20:25
beaglessdague: +10020:25
*** rossella_s_ has joined #openstack-neutron20:32
*** ijw_ has joined #openstack-neutron20:33
*** rossella_s has quit IRC20:34
*** rossella_s_ is now known as rossella_s20:34
*** ijw has quit IRC20:36
*** rossella_s_ has joined #openstack-neutron20:43
*** rossella_s has quit IRC20:47
*** rossella_s_ is now known as rossella_s20:47
*** amotoki_ has joined #openstack-neutron21:00
*** WackoRobie has joined #openstack-neutron21:02
*** amotoki is now known as __amotoki__21:06
*** amotoki_ is now known as amotoki21:06
*** WackoRobie has quit IRC21:07
*** carl_baldwin has joined #openstack-neutron21:12
*** irenab_ has joined #openstack-neutron21:14
*** ijw_ has quit IRC21:28
*** vkozhukalov has quit IRC21:35
*** mestery has joined #openstack-neutron21:50
openstackgerritClaudiu Belu proposed a change to openstack/neutron: Fixes Hyper-V ceilometer metrics enabling after service restart  https://review.openstack.org/6384022:00
markmcclaindkehn, salv-orlando, marun: picking up where we left off22:01
salv-orlandoI'm here22:02
markmcclainas salv-orlando pointed out we do have a tricky problem22:02
dkehnyes22:02
markmcclainhow to avoid branches22:02
salv-orlandopatch the code to run the schema changes at startup? idk if that can possibly work22:03
markmcclainnot excited about magic code at start-up22:03
marunhmm, that begs the question.  is it possible to add new migrations (i.e. change schema) to stable via a backport?22:04
salv-orlandoor add a havna-2 revision between havana and the first icehouse revision22:04
salv-orlandoand tell users to do a neutron-db-manage upgrade head?22:04
markmcclainI think we insert a havana2 migration22:04
markmcclainthat fixes the problem for stable22:04
salv-orlandomarun: db migration in stable are -2'ed unless there a seriously good reason for them22:04
marunsalv-orlando: is eventual consistency a seriously good reason enough?22:05
marunsalv-orlando: i'm not trying to be a dick, it is an honest question.22:05
markmcclainfor those running trunk we could create a smarter revision that checks for the presence of the proper table22:05
markmcclainso that way we can fix it both places22:06
salv-orlandomarun: I think the question to ask is whether there is a fundamental use case broken rather than whether there's eventual consistency or not. But let's first finish this other discussion22:06
marunsalv-orlando: ok22:06
*** amotoki has quit IRC22:06
markmcclainthe havana-2 migration would need the same smarts22:06
markmcclainwe'd just have to insert into two places on the timeline22:06
salv-orlandomarkmcclain: how can we avoid then to run it twice for new deployments?22:07
dkehnmarkmcclain: whoim or what check for a proper table the migration or something else22:07
salv-orlandowith an explicit check in the 2nd?22:07
markmcclainI think the migration has to check to make sure the state exists to modify the tables22:07
markmcclainif not then skip22:07
markmcclainthat way the 2nd time alembic sees it22:08
markmcclainthe migration would be a no-op22:08
marun+122:08
*** rossella_s has quit IRC22:08
*** ijw has joined #openstack-neutron22:08
marunthat's a nice property of code-based migrations - they can be smart22:08
markmcclainfor havana.2 release we'll just have to update the release notes22:09
*** irenab_ has quit IRC22:09
markmcclainI'd be happy to work on this22:09
markmcclaindkehn: sorry didn't clarify that alembic would check22:10
salv-orlandook sounds like it's sorted?22:10
dkehnmarkmcclain: thx, I guess the question is why aren't we doing it now? just curious22:10
salv-orlandoI think both migration should also be aware that if the lbaas plugin is enabled they should not run22:10
markmcclainmaybe.. I'd prefer for the migration to be aware of the requirement that the table exist at the specific moment in the timeline if ml2 is enable22:11
marunsalv-orlando: does that imply that the plugin's migration isn't smart enough to skip if the state is already as desired?22:12
markmcclainso if case there is some other drive migration we missed this one will still correct the db22:12
salv-orlandomake sense to me.22:12
markmcclainmarun: right now.. no22:12
markmcclainalembic's smarts are at migration creation time22:12
marunmarkmcclain: i guess it would be a lot of work to always do that :/22:12
markmcclainso I'll be adding a bit of defensive code22:13
*** ijw has quit IRC22:13
markmcclainunless someone thinks I'm heading down the wrong path22:13
markmcclainI'll work on coding this up and add you all as reviewers :)22:13
markmcclainnow onto the eventual consistency question….22:14
*** armax has left #openstack-neutron22:14
dkehnmarkmcclain: in the for what its worth catagory the enabling q-lbaas does get the devstack up and working, wainting on all the exercises22:15
marunI have heard lots of complaints that Neutron falls over when booting large numbers of vm's concurrently.22:16
dkehnmarkmcclain: still fails on the floating_ips part22:16
marunGiven that scalability is an issue, having multiple workers for api and rpc is a matter of when, not if.22:17
markmcclainmarun: right we run 10+ instances in our cloud22:17
markmcclainmarun: I've got to clean up a patch that separates the api and rpc processes22:17
markmcclainso that they can be scaled independently22:18
markmcclainright now their entangled22:18
markmcclains/their/they're/22:18
marunscaling them independently is the starting point22:18
marunbut once that is done, the dhcp agent, at least, is a walking zombie of race conditions22:18
marunso eventual consistency is needed to make sure that those races don't exhibit22:19
markmcclainI wouldn't limit to dhcp.. l3 has the same basic workflow22:19
maruni'm just talking about what I know, hopefully the solution for dhcp can be applied generally.22:19
salv-orlandowell, the l2 agent has no workflow at all then.22:19
marunSo, that said, the issue isn't that we're going to have eventually consistent agents22:20
marunthe issue is whether the patches that implement it can/should be backported to havana22:20
maruni have some people clamoring for fixes for grizzly, but that seems like a bridge too far22:21
markmcclainmarun: if they fix bugs and can be deployed without migrations or invocation changes they I think we should consider them for backport22:21
marunit will require some minor db changes - an extra field to the networks table should be it22:21
markmcclainthen sadly folks will Icehouse to enable it22:21
marun:(22:22
marunthat really sucks22:22
markmcclaingiven current velocity of new features22:22
marunis there no way at least to give people the option?22:22
markmcclainmarun new migrations aren't allowed22:22
marunIt's pretty embarrassing to be working on a project that is so broken in grizzly and havana by default, and not be allowed to fix it.22:23
marunBut I guess them's the breaks.22:23
markmcclainmarun: yes there are bugs but that is the nature of releasing every 6 mos22:23
markmcclainnot sure how we provide an optional backport22:24
markmcclainit's also one of the reason several different deployments track trunk22:24
marunWell, for RH we have to support what we ship.22:24
salv-orlandoun022:24
markmcclainmarun: I understand that22:24
marungrizzly/havana basically can't scale with the oss plugins, so I guess we'll have to maintain our own branches until icehouse ships22:25
salv-orlandounless one deem current code is so broken a rewrite is needed. Then back port will be easy ;)22:25
markmcclainmarun: yeah not excited about random forks of backports either22:25
markmcclainbecause that makes bug triaging harder22:25
markmcclainwill have to think about how we can approach it22:26
marunmarkmcclain: yes, food for thought.22:26
markmcclainat best Havana is as far back as I'd like to go22:26
marunmarkmcclain: I would agree with that.  Maintaining grizzly isn't in the cards.22:26
marunmarkmcclain: So, to keep in mind - eventual consistency should require at most one field addition per agent-type.  Possibly fewer.22:27
markmcclainyeah.. I think the first step is to get this fixed in trunk22:27
marunmarkmcclain: understood.22:27
markmcclainthen we'll be able to definitively be able to catalog the changes needed22:27
markmcclainwhich will probably tell us whether we can make a reasonable backport available22:28
marunagreed.22:29
markmcclainI'm afraid that if we begin this discussion without specific it will get derailed quickly22:29
marunok, that's all.  thank you for giving me some perspective on the issue.22:30
markmcclainbecause folks will fill the gaps how they see fit22:30
*** yamahata has joined #openstack-neutron22:30
marunI wasn't expecting anything definitive, I just wanted to know in general some of the challenges ahead.22:30
markmcclainok22:30
markmcclainmigrations and changes in invocation, configuration are the biggest no-nos22:31
markmcclainbecause those require changes to deployment tools22:31
marunright22:31
markmcclainotherwise logic changes are acceptable in backports22:31
markmcclainwe're really not changing the API so we should be good on that front22:32
marunand if there is sufficient perceived demand for a given patch, even if it does break deployment tools, then it's possible at least.22:32
markmcclainyeah if the fixes are good enough22:32
markmcclainwe can consider ways to enable it optionally in backports22:33
markmcclainthe migrations also cause the same problems we were discussing with ml2 bugs22:33
maruni'm afraid i'm still not entirely clear on the migrations issue…22:33
markmcclainso adding that fix is actually a good test for how we could make changes like this22:33
marunis it that we can't interject a migration in the sequence?22:33
markmcclainso we cannot change or alter migrations once we release22:34
markmcclainwe the problem is some are running the latest release and others are running trunk22:34
markmcclainso the insertion point is different for those users22:34
marunthat would seem like a failing of the migration mechanism22:34
marunso long as a migration can be made no-op if it is not required22:34
marunand then it can be injected earlier in the chain without issue (if sufficiently orthogonal)22:35
markmcclainwell we're going to have to add code to make those migrations sufficiently optional22:35
markmcclainthe ml2 fix will be a good proving ground for this22:35
marunah, ok.22:36
marunok, that's all from me.22:36
*** carl_baldwin has quit IRC22:37
*** mlavalle has quit IRC22:37
*** HenryG has quit IRC22:38
*** jaypipes has joined #openstack-neutron22:38
*** HenryG has joined #openstack-neutron22:38
*** HenryG has quit IRC22:43
*** HenryG has joined #openstack-neutron22:44
*** zzelle has quit IRC22:50
*** julim has quit IRC22:56
*** clev has quit IRC22:57
*** carl_baldwin has joined #openstack-neutron23:00
*** carl_baldwin has quit IRC23:03
*** carl_baldwin2 has joined #openstack-neutron23:03
*** markwash has joined #openstack-neutron23:09
*** carl_baldwin2 has quit IRC23:12
*** yamahata has quit IRC23:20
*** markmcclain has quit IRC23:24
*** markmcclain has joined #openstack-neutron23:26
*** ijw has joined #openstack-neutron23:29
*** harlowja_ has joined #openstack-neutron23:29
*** harlowja has quit IRC23:29
jaypipessalv-orlando: ahoy. anything I can do to assist with https://bugs.launchpad.net/tempest/+bug/1253896? It's blocking some patches I have for Keystone and have a few cycles if you need assistance.23:31
anteayajaypipes: this patch is about to merge23:32
anteayahttps://review.openstack.org/#/c/63641/23:32
salv-orlandojaypipes: analyse log fle and found root causes. What is reported in bug 1253896 is not a bug but rather a failure manifestation. We have found several root causes now, but looking at failure reports I know there are others23:32
anteayapray it addressed 125389623:33
salv-orlandoThere is also a patch from marun which addresses another failure mode of bug 125389623:33
*** ijw has quit IRC23:33
jaypipessalv-orlando: ok... will do my best to dig around.23:33
anteayasalv-orlando: awesome, which patch?23:33
*** ijw has joined #openstack-neutron23:34
salv-orlandohttps://review.openstack.org/#/c/61168/23:34
salv-orlandoanteaya: ^23:34
* anteaya clicks23:34
salv-orlandomarun did not add to the commit message this bug, but the patch will prevent failures in dhcp setup because of missed notifications23:34
*** SumitNaiksatam has quit IRC23:35
salv-orlandothat failure will manifest in tempest as ssh timeout, but the truth is that the ssh connection fails because the vm never got an ip23:35
jaypipessalv-orlando: patch now merged.23:36
jaypipessalv-orlando: the first patch, that is.. to devstack.23:36
salv-orlandojaypipes: good. We'll see if failure rate goes down. The next 7-8 days won't count however23:37
*** markwash has quit IRC23:37
jaypipessalv-orlando: how come?23:37
salv-orlandojaypipes: christmas and new years' day23:38
jaypipessalv-orlando: oh, I see what you mean... yes :)23:38
openstackgerritA change was merged to openstack/neutron: Send DHCP notifications regardless of agent status  https://review.openstack.org/6116823:40
openstackgerritA change was merged to openstack/neutron: Imported Translations from Transifex  https://review.openstack.org/6305623:40
jaypipessalv-orlando: and now marun's patch is merged... :)23:41
salv-orlandook23:41
jaypipessalv-orlando: what is the difference between an agent's admin_state_up and active flags?23:44
*** nijaba has quit IRC23:45
jaypipessalv-orlando: oh, never mind... see it now...23:45
jaypipessorry for the noise!23:45
salv-orlandojaypipes: cool23:45
salv-orlandono worries23:45
*** nijaba has joined #openstack-neutron23:45
*** nijaba has quit IRC23:45
*** nijaba has joined #openstack-neutron23:45
*** ijw_ has joined #openstack-neutron23:52
*** ijw has quit IRC23:56
« Sunday, 2013-12-22 Index Tuesday, 2013-12-24 »

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!