Thursday, 2013-12-05

bgorskiIs there a better way than getting router port list and choosing the one from external network?00:01
bgorskiWhat the IP address is not present in gateway_info inside router show command?00:02
openstackgerritJon Grimm proposed a change to openstack/neutron: Fix ml2 plugin for allowedaddresspairs tests  https://review.openstack.org/5889600:03
openstackgerritdekehn proposed a change to openstack/neutron: extra_dhcp_opt add checks for empty strings  https://review.openstack.org/5985800:05
*** openstackgerrit has quit IRC00:06
*** openstackgerrit has joined #openstack-neutron00:06
*** aymenfrikha has quit IRC00:07
*** aymenfrikha has joined #openstack-neutron00:07
*** alexpilotti has joined #openstack-neutron00:07
*** pete5 has quit IRC00:18
*** mlavalle has quit IRC00:19
*** matsuhashi has joined #openstack-neutron00:21
*** yfujioka has joined #openstack-neutron00:24
*** jp_at_hp has quit IRC00:26
*** SumitNaiksatam has joined #openstack-neutron00:27
*** alexpilotti has quit IRC00:28
*** layer427expert has joined #openstack-neutron00:36
layer427expertdoes anyone have 5 minutes to answer a question on filter?00:38
layer427expertI need to get the list of members that have the same address from a given tenant00:38
layer427expertfrom looking at the code it is just a k,v list of field:value is this correct?00:39
layer427expertfilter_foo = {"tenant_id":pool['tenant_id'], "address":"10.0.0.3"}00:40
layer427expert                    debug(self.plugin.get_members(self, context, fil, fields=None))00:40
layer427expertexample: debug(self.plugin.get_members(self, context, filter_foo, fields=None))00:41
*** dims has joined #openstack-neutron00:44
*** dims has quit IRC00:45
*** jgrimm has quit IRC00:55
*** Abhishe__ has quit IRC01:07
*** matsuhashi has quit IRC01:31
*** matsuhas_ has joined #openstack-neutron01:33
*** dims has joined #openstack-neutron01:39
*** Jianyong has joined #openstack-neutron01:46
*** dzyu has joined #openstack-neutron01:53
openstackgerritAaron Rosen proposed a change to openstack/python-neutronclient: Fix neutron port-create --no-security-groups  https://review.openstack.org/5983601:56
*** netavenger-jr has joined #openstack-neutron01:58
*** yuyangbj has joined #openstack-neutron01:58
yuyangbjHi sc68cal, are you there?01:59
*** carl_baldwin has quit IRC01:59
*** yuyangbj has quit IRC02:03
*** aymenfrikha has quit IRC02:08
openstackgerritShiv Haris proposed a change to openstack/neutron: Implementaion of Mechanism driver for Brocade VDX cluster of switches.  https://review.openstack.org/6012902:13
*** yuyangbj has joined #openstack-neutron02:16
*** openstack has joined #openstack-neutron02:17
*** openstackgerrit has quit IRC02:24
*** openstackgerrit has joined #openstack-neutron02:24
openstackgerritXiang Hui proposed a change to openstack/neutron: Sync dhcp_agent.ini with the codes  https://review.openstack.org/5911802:25
*** bgorski has quit IRC02:33
*** mengxd has joined #openstack-neutron02:33
*** aymenfrikha has joined #openstack-neutron02:33
*** matsuhas_ has quit IRC02:35
*** dims has quit IRC02:37
*** banix has joined #openstack-neutron02:46
*** xdmeng has joined #openstack-neutron02:47
*** mengxd has quit IRC02:49
*** aymenfrikha has quit IRC02:59
*** Jianyong has quit IRC03:04
*** aveiga has joined #openstack-neutron03:11
*** HenryG has quit IRC03:11
*** chandankumar has joined #openstack-neutron03:17
*** Jianyong has joined #openstack-neutron03:17
*** Jianyong has quit IRC03:23
*** alagalah_ has joined #openstack-neutron03:32
*** alagalah_ has left #openstack-neutron03:33
*** banix has quit IRC03:33
openstackgerritBrian Haley proposed a change to openstack/neutron: Change l3-agent to periodically check children are running  https://review.openstack.org/5999703:35
*** Jianyong has joined #openstack-neutron03:35
*** suresh12 has quit IRC03:36
*** suresh12 has joined #openstack-neutron03:37
*** banix has joined #openstack-neutron03:37
*** suresh12 has quit IRC03:41
*** suresh12 has joined #openstack-neutron03:42
*** shashank_ has quit IRC03:43
*** Jianyong has quit IRC03:44
*** suresh12 has quit IRC03:45
*** Jianyong has joined #openstack-neutron03:56
*** suresh12 has joined #openstack-neutron03:56
*** suresh12 has quit IRC04:00
*** Jianyong has quit IRC04:01
*** x86brandon has joined #openstack-neutron04:02
*** aveiga has quit IRC04:07
*** networkstatic has joined #openstack-neutron04:11
openstackgerritAaron Rosen proposed a change to openstack/neutron: Bump api_workers from 0 to 4  https://review.openstack.org/5978704:13
*** matsuhashi has joined #openstack-neutron04:14
*** banix has quit IRC04:26
*** otherwiseguy has quit IRC04:40
*** Jianyong has joined #openstack-neutron04:43
*** otherwiseguy has joined #openstack-neutron04:52
*** amotoki has joined #openstack-neutron04:53
*** suresh12 has joined #openstack-neutron04:55
openstackgerritberlin proposed a change to openstack/neutron: Add Error Handling to NVP advanced LBaaS/FWaaS  https://review.openstack.org/5962504:57
*** suresh12 has quit IRC05:01
*** marun has joined #openstack-neutron05:03
openstackgerritBrian Haley proposed a change to openstack/neutron: Change l3-agent to periodically check children are running  https://review.openstack.org/5999705:14
*** yfried has quit IRC05:14
*** layer427expert has quit IRC05:14
*** Jianyong has quit IRC05:20
*** changbl has joined #openstack-neutron05:25
*** netavenger-jr has quit IRC05:31
*** garyk has quit IRC05:44
*** yamahata_ has quit IRC05:55
*** yfried has joined #openstack-neutron06:03
*** Jianyong has joined #openstack-neutron06:07
*** matsuhashi has quit IRC06:14
*** matsuhashi has joined #openstack-neutron06:14
openstackgerritA change was merged to openstack/python-neutronclient: Fix neutron port-create --no-security-groups  https://review.openstack.org/5983606:14
openstackgerritIsaku Yamahata proposed a change to openstack/python-neutronclient: setup logger name of NeutronCommand automatically  https://review.openstack.org/5931006:30
openstackgerritJenkins proposed a change to openstack/neutron: Imported Translations from Transifex  https://review.openstack.org/6015606:31
*** gdubreui has quit IRC06:32
*** alex_klimov has joined #openstack-neutron06:33
*** Abhishek has joined #openstack-neutron06:34
*** alex_klimov has quit IRC06:41
openstackgerritXiang Hui proposed a change to openstack/neutron: Sync dhcp_agent.ini with the codes  https://review.openstack.org/5911806:42
*** amritanshu_RnD has joined #openstack-neutron06:45
*** x86brandon has quit IRC06:48
*** Abhishek has quit IRC06:56
*** jpich has joined #openstack-neutron07:00
*** alex_klimov has joined #openstack-neutron07:06
openstackgerritA change was merged to openstack/neutron: Fix misspells  https://review.openstack.org/5980907:08
*** alex_klimov has quit IRC07:08
*** alex_klimov has joined #openstack-neutron07:08
*** alex_klimov has quit IRC07:09
*** alex_klimov1 has joined #openstack-neutron07:10
*** dzyu has quit IRC07:14
*** matsuhashi has quit IRC07:17
*** jlibosva has joined #openstack-neutron07:18
*** nati_ueno has quit IRC07:24
*** jpich has quit IRC07:56
*** xdmeng has quit IRC07:57
*** mengxd has joined #openstack-neutron07:57
openstackgerritAkihiro Motoki proposed a change to openstack/neutron: Return request-id in API response  https://review.openstack.org/5827007:58
*** harlowja has quit IRC08:08
*** nati_ueno has joined #openstack-neutron08:15
*** yuyangbj has left #openstack-neutron08:28
*** fouxm has joined #openstack-neutron08:32
*** salv-orlando has joined #openstack-neutron08:34
*** fouxm has quit IRC08:35
*** networkstatic has quit IRC08:36
*** fouxm has joined #openstack-neutron08:41
*** networkstatic has joined #openstack-neutron08:42
*** jp_at_hp has joined #openstack-neutron08:46
marunamotoki: ping08:49
amotokimarun: pong08:49
marunamotoki: I have mixed feelings about the change to the request id patch.  Given that it's something we're always going to want, why does it need to be a separate wsgi app?08:50
marunamotoki: I'm not sure swift is a good example. framkly.08:50
marunfrankly08:50
amotokimarun: in my first implementation request-id is added in several places. I think it is better request-id is added in a single place in the pipeline.08:51
marunhmmm08:52
marunamotoki: I guess if that's the way to do it.08:53
marunamotoki: I'm going to suggest breaking the patch apart, though.08:53
marunamotoki: I think adding the faultwrap middleware should be one patch, and then a second patch can add request id support.08:54
amotokimarun: wsgi.Router is another poiint to add request-id.08:54
marunamotoki: If you think that the best way to do it is via middleware, I think that's ok.  I simply think that adding the middleware should be added separately.  It seems that it's primary value is actually in error handling, and then becomes a convenient place to add request id.08:56
amotokimarun: agree.08:56
marunamotoki: Is there a bug or otherwise a perceived need for the faultwrap middleware?  i.e. are we leaking errors?08:56
amotokimarun: AFAIK no bug.08:57
marunamotoki: do you think it makes sense to add one?  Or should it simply be tagged with Related-Bug?08:57
amotokimarun: when I add a code to raise exception in keystone context middleware, i see it in the response.08:57
*** nati_ueno has quit IRC08:57
amotokimarun: it is worht filed as a separate bug though it is a potential bug.08:58
marunamotoki: ah.  and looking at the pipeline it looks like any global error handling in the neutron app wouldn't necessarily cover extensions anyway08:58
*** jistr has joined #openstack-neutron08:58
marunamotoki: +1 on filing a bug then.  I think it's a good find.08:58
amotokimarun: will file it.08:59
marunamotoki: One more thing, have you seen the thread that discussed cross-application request id logging?08:59
amotokimarun: is it a part of the thread of request-id?09:00
marunamotoki: yes09:00
amotokimarun: i haven't caught up with the thread yet.09:00
marunamotoki: The part I think is important is that there appears to be consensus on how to handle cross-app request ids.09:01
marunamotoki: Basically, a service like neutron will check for a client-supplied request id and log it with a newly generated request id.09:01
marunamotoki: now that I think about it, though, that seems problematic :(09:02
marunamotoki: nothing stopping a malicious user from passing request ids that they receive back from a given service request and effectively poisoning the logs.09:03
marunamotoki: nevermind.09:03
amotokimarun: another idea is to log a returned request-id in a client side.09:04
marunamotoki: we can certainly do that, but I'm hoping to have the ability to debug nova-neutron integration.09:04
marunamotoki: ideally a call to nova to boot a vm would return a request id that could be used to track all the operations in the log associated with the initial request in both neutron and nova.09:05
amotokimarun: I think both approach will work though there is some difference.09:05
marunamotoki: ah, you mean that nova would log the returned request id… hmmm, I think that's a safer bet.09:08
amotokimarun: yes.09:08
marunamotoki: I'll suggest that on the mailing list09:08
amotokimarun: to do so, neutronclient needs to report request-id.09:09
marunamotoki: It would be more involved, mind you.  It would require that every client request to explicitly log the request id when cross-service requests are made.09:09
marunamotoki: won't it be doing so anyway?09:09
openstackgerritA change was merged to openstack/neutron: Handle exceptions on create_dhcp_port  https://review.openstack.org/5781209:10
amotokimarun: I think so too. it is not a case of nova neutron integration. It is better all clients support it.09:12
marunamotoki: agreed.  I'm just using it as an example since it's the important one for neutron.09:12
marunamotoki: ideally this middleware becomes part of oslo and everyone uses the same one.09:12
amotokimarun: agree. the middleware should be a part of oslo.09:13
amotokimarun: if you have a better name of the middleware, please advise me. I think faultwrap is not best....09:16
*** yamahata_ has joined #openstack-neutron09:16
marunamotoki: what about using the swift name?09:16
anteayayeah I agree faultwrap is not a good name09:18
anteayadon't ask mordred for name suggestions though09:18
anteayawhat is the swift name?09:18
amotoki A name like catch_error or faultwrap seems it focuses on error handling...09:18
amotokiI think the middleware is to do common things for all requests.09:19
amotokiAt least i agree catch_error is better than faultwrap.09:20
marunamotoki: well, the primary utility of request id is in debugging faults09:20
marunamotoki: so at least it's peripherally related09:20
anteayasitting here with dhellman, he doesn't have irc09:21
anteayahe asked for this idea to be posted to the mailing list, when you are ready09:21
anteayadoes that sound reasonable?09:21
*** nati_ueno has joined #openstack-neutron09:22
marunamotoki: what idea?09:22
marunoops, anteaya ^09:22
amotoki?09:22
marunanteaya: the name?09:22
anteayathat this is a middleware that becomes part of oslo09:22
marunanteaya: right09:22
anteayaor did I mis-understand09:22
marunanteaya: no, I think we're on the same page09:23
anteayaokay09:23
marunanteaya: does doug want it to go into oslo asap or can the neutron patch be the POC?09:23
marunanteaya: we need request id traceability yesterday is all09:23
*** jpich has joined #openstack-neutron09:24
anteayahi, marun, dhellmann taking over anteaya's keyboard for a sec09:24
marunanteaya: hi :)09:25
anteayamarun: you should probably go ahead and bring it into oslo asap09:25
anteayamarun: is it already used in >1 project?09:26
marunanteaya: I think swift is doing something similar, but I don't know what their attitude towards oslo is09:26
marunanteaya: we'll need this in both neutron and nova though09:26
marunanteaya: …and everywhere, actually.09:26
anteayamarun: ok, the typical path is to pull it into oslo if we know >1 proj wants it, but the api is not stable09:27
marunanteaya: I'd say we have a good case then.09:27
anteayamarun: if we can make the oslo version compatible for swift, great, but if not that is not a blocker09:27
marunamotoki: ^^09:27
anteayayep09:27
anteayamarun: any questions on process?09:28
marunamotoki: Sounds like the patch should be retargeted for oslo09:28
marunanteaya: I'm afraid I'm just the reviewer.  amotoki has been working on the patch.09:28
amotokimarun: yeah.09:28
marunanteaya, amotoki: I think the patch as is can probably go in.  I'd like to see a follow-on that ensures cross-service request id logging, but I think we still need to hash out the best approach on the mailing list.09:29
amotokimarun: swift one is a bit different: it uses X-Trans-Id and swift does not depend on webob.09:29
anteayaamotoki: send email to the -dev list if you have issues/questions and i or someone on the oslo team will help out09:29
marunamotoki: ah, sounds like you'll have to implement it fresh then.09:29
anteayamarun: sure, that might even be separate middleware but we can discuss later09:29
marunanteaya: fyi the subject of the list convo is '[openstack-dev] request-id in API response'09:30
anteayamarun: great, I'll look for that and reply under my own name :-)09:30
amotokimarun: anteaya: is it better to move the patch to oslo soon, or we continue to discuss in neutron?09:30
marunanteaya: cool, and thanks for stepping in.09:30
marunamotoki: I think the consensus was to do it in oslo instead of neutron.09:31
amotokineutron-nova integration has high priority bugs.09:31
amotokithis is the only reason of my question.09:31
anteayaamotoki, marun: if you want to work it out in neutron first, that's ok too, but you're welcome to work through the oslo repo -- do the most pragmatic thing09:32
marunamotoki: I guess that's a question for anteaya - how fast can we bring a change in from oslo once merged.09:32
anteayamarun: that part should be easy, it's getting the change into oslo in the first place that may cause delay09:32
anteayawe tend to be picky about our apis :-)09:32
marunanteaya: it's not so complicated, at least.  I'm guessing nova has already implemented request ids, too, so they probably have an opinion on the version that goes into oslo09:33
anteayamarun: probably want to add some of those devs as reviewers on the changeset, then09:34
marunanteaya: understood09:34
marunamotoki: are you familiar with the nova implementation of request ids?09:35
amotokimarun: to some extent. I read request-id nova code too.09:35
anteayamarun, amotoki: I need to go back to listening to the conference, but if you bring up questions on the -dev ml I will see them later today (use [oslo] in the subject)09:35
marunanteaya: ok, thanks for the help!09:36
amotokianteaya: marun: thanks. I will reply to -dev list, but it will be after a couple of hours.09:37
*** xdmeng has joined #openstack-neutron09:38
*** mengxd has quit IRC09:39
*** xdmeng has quit IRC09:39
*** mengxd has joined #openstack-neutron09:39
anteayaI'm back as myself now09:40
anteayaglad you were able to get some direction on this09:40
*** rossella_s has joined #openstack-neutron09:40
anteayaspeak up if you need more, and like Doug said, post with [oslo] in the subject09:40
*** mengxd has quit IRC09:41
anteayahey rossella_s09:41
rossella_shi anteaya09:41
anteayahow is your work with tempest going?09:41
rossella_sthanks for asking. Well I send 2 crs with api tests09:41
rossella_sI am trying now to analyze now the failure of the full tempest test for neutron. It's going pretty slow09:42
anteayacrs?09:43
rossella_scode review09:43
rossella_spatch09:44
anteayaah09:44
*** amuller has joined #openstack-neutron09:45
openstackgerritgongysh proposed a change to openstack/neutron: add router binding into firewall  https://review.openstack.org/6019009:45
openstackgerritA change was merged to openstack/neutron: Imported Translations from Transifex  https://review.openstack.org/6015609:46
openstackgerritgongysh proposed a change to openstack/neutron: add router binding into firewall  https://review.openstack.org/6019009:46
anteayarossella_s: can you post the urls for the patches?09:46
rossella_shttps://review.openstack.org/#/c/56349/09:47
rossella_shttps://review.openstack.org/#/c/56680/09:47
rossella_sthis is related, I've found a bug in the neutron xml client for tempest https://review.openstack.org/#/c/59029/09:48
anteayaslow wifi here, I am looking at the status of the patches09:49
*** gongysh has joined #openstack-neutron09:49
rossella_sthey are in a good shape09:50
rossella_sonly this needs a core reviewer to take a look https://review.openstack.org/#/c/56349/09:50
openstackgerritA change was merged to openstack/neutron: atomically setup ovs ports  https://review.openstack.org/5961209:52
anteayaI'm trying to find out how to contact Zhi Kun Liu09:52
*** markmcclain has joined #openstack-neutron09:54
anteayahedoesn't seem to be a foundation memeber09:54
*** gongysh has quit IRC09:55
rossella_sI don't know him09:56
anteayaMark found him in the foundation dirctory09:57
*** yamahata_ has quit IRC09:58
anteayahe seems to be a tempest dev09:58
*** markmcclain has quit IRC09:59
anteayahis irc nick is zhikunliu09:59
*** nati_ueno has quit IRC10:06
anteayarossella_s: 56349 has a new patchset, so you are looking for a neutron core to give a review?10:06
rossella_santeaya: yes indeed10:07
* rossella_s coffee10:08
anteayaokay10:09
*** bvandenh has joined #openstack-neutron10:14
*** Jianyong has left #openstack-neutron10:31
openstackgerritJianing Yang proposed a change to openstack/neutron: Fix utils.str2dict and utils.dict2str's incorrect behavior on certain inputs  https://review.openstack.org/6019510:33
openstackgerritIsaku Yamahata proposed a change to openstack/neutron: l3-agent-consolidation(WIP): framework for consolidating l3 agents  https://review.openstack.org/5762710:45
openstackgerritA change was merged to openstack/neutron: Fix metering iptables driver doesn't read root_helper param  https://review.openstack.org/5905710:56
*** amuller has quit IRC10:58
*** yfried has quit IRC10:58
openstackgerritA change was merged to openstack/neutron: Improve unit test coverage for Cisco plugin base code  https://review.openstack.org/5888211:00
openstackgerritA change was merged to openstack/neutron: Fix bad call in port_update in linuxbridge agent  https://review.openstack.org/5943111:00
openstackgerritA change was merged to openstack/neutron: metaplugin: use correct parameter to call neutron client  https://review.openstack.org/5738311:00
*** yamahata_ has joined #openstack-neutron11:01
openstackgerritBerezovsky Irena proposed a change to openstack/neutron: Change daemon_endpoint default port  https://review.openstack.org/6020311:08
*** yfried has joined #openstack-neutron11:13
*** jprovazn has joined #openstack-neutron11:15
*** yamahata_ has quit IRC11:15
fouxmHi, I'd like to give a MTU of 9000 in my VM. My physical interface is at 9050 but neutron intitates the tap at 1500. Is there a way to force the MTU with linuxbridge plugin (conf havana RC3, neutron / linuxbridge / vlan). Thx11:15
*** yfried has quit IRC11:18
*** yfried has joined #openstack-neutron11:30
openstackgerritenikanorov proposed a change to openstack/neutron: Introduce Loadbalancer instance  https://review.openstack.org/6020711:31
*** HenryG has joined #openstack-neutron11:32
*** pcm_ has joined #openstack-neutron11:37
*** pcm_ has quit IRC11:38
*** pcm_ has joined #openstack-neutron11:39
*** amuller has joined #openstack-neutron11:45
*** networkstatic is now known as networkstatic_zZ11:48
*** Sreedhar has joined #openstack-neutron11:48
*** lori has left #openstack-neutron11:49
openstackgerritJianing Yang proposed a change to openstack/neutron: Fix str2dict and dict2str's incorrect behavior  https://review.openstack.org/6019511:49
*** rkukura has quit IRC11:58
*** amuller has quit IRC12:03
*** amuller_ has joined #openstack-neutron12:03
*** amuller_ has quit IRC12:09
*** bashok has joined #openstack-neutron12:10
*** jojo_ has joined #openstack-neutron12:17
jojo_have been stressing add/remove floating-ips (10) between several instances (20) and after a while floating-ip network stops working12:19
jojo_its an havana/neutron/ovs/gre12:19
jojo_and if l2 agent is restarted on compute nodes, service is restored12:20
*** amuller_ has joined #openstack-neutron12:25
*** yfujioka has quit IRC12:27
Sreedharmarun, slav-orlando: For the neutron server performance issue during concurrent instance deployment, After merging the changes with multiple neutron-server process patch https://review.openstack.org/#/c/37131/ able to deploy 240 instances in Havana as well. 2 instances out of 240 went to error state, thats due to Too many connections to SQL Database12:39
Sreedharmarun, slav-orlando: But during the instance first boot only 81 instances could get an IP. Network ports are getting created and IP Address are allocating very fast but by the time port is made up its taking close to 153 seconds.. In Grizzly i have tested many times we did not had these issues till 210 instances12:41
Sreedharmarun, slav-orlando:In Grizzly just a reboot of the instance could get the IP Address during boot, but in Havana we need to restart the DHCP agent, dnsmasq process followed by instance reboot. This clearly shows there is a performance regression in Neutron/Horizon compared to Grizzly12:42
*** dims has joined #openstack-neutron12:51
HenryGSreedhar: great info!12:54
*** bashok has quit IRC12:54
HenryGSreedhar: how many api_workers did you configure?12:54
HenryGSreedar: you should report this info to the dev ML. Not everyone catches all the IRC msgs12:55
*** ihrachyska has quit IRC13:01
HenryGSreedhar: I am looking for the change(s) that caused the need to restart the dhcp agent and dnsmasq process. Do you know the change(s)?13:03
SreedharHenryG: My VM has 8cores/16 hyperthreads, so i set api_workers to 813:04
*** ihrachyska has joined #openstack-neutron13:04
SreedharHenryG: Have updated my findings in this bug - https://bugs.launchpad.net/neutron/+bug/119238113:05
SreedharHenryG: When doing the tests with Grizzly initially i had the same issues like need to restart DHCP agent/dnsmasq to get IP address. But after tuning the SQL pool parameters, agent_down_time and report_interval, just rebooting the instance would have suffice13:06
*** salv-orlando_ has joined #openstack-neutron13:06
*** bvandenh has quit IRC13:07
SreedharHenryG: But in Havana, I have tuned the same values but still having the issue13:08
*** salv-orlando has quit IRC13:09
*** salv-orlando_ is now known as salv-orlando13:09
*** amuller_ has quit IRC13:09
HenryGSreedhar: Thanks, great info to have!13:10
SreedharHenryG: In the bug https://bugs.launchpad.net/neutron/+bug/1192381, issue documented is ports info is not getting updated in dnsmasq host file, But in my case, ports info is getting updated in dnsmasq host file but with a delay of close to 153 seconds13:11
HenryGSreedhar: so that sounds like the underlying issue is the time taken to bring the ports up?13:13
HenryGSreedhar: if the time is too long, something times out and the dnsmasq host file never gets updated?13:14
SreedharHenryG: In my case dnmasq is getting updated but with a delay13:18
SreedharHenryG: I could even ping the IP address of those instances. Due to this delay, tools like cloud-init could not configure due to network not available at that time. But if we reboot the instance (without rebooting dnsmasq) instance not getting the IP and i can't even ping the IP address13:19
*** yamahata_ has joined #openstack-neutron13:25
*** yamahata_ has quit IRC13:26
*** yamahata_ has joined #openstack-neutron13:26
marunSreedhar: hi13:26
Sreedharmarun: Hi, multiple api workers patch changes helped to create 240 instances in havana as well. Without that patch and single worker thread, i could create only 150 instances with 30 concurrent requests13:27
anteayaHenryG: the channel is logged and the logs are found here: http://eavesdrop.openstack.org/irclogs/%23openstack-neutron/13:28
anteayaI'm encouraging conversations to take place in the channel and for people to read the logs13:28
anteayait helps improve the speed of addressing topics13:29
anteayaand mailing list for items that need more input including input from other projects13:30
marunSreedhar: so, improvement.13:30
marunSreedhar: I was wondering whether it might make sense to have multiple workers for rpc processing as well13:30
*** amuller_ has joined #openstack-neutron13:30
marunSreedhar: since the bottleneck now appears to be rpc-related rather than wsgi-related13:31
marunSreedhar: The other option is to simply profile the code paths associated with booting a VM.  There is clearly room for improvement.13:31
*** amuller_ is now known as amuller13:31
Sreedharmarun: multiple rpc processing might help.. Note, these kind of issues we get with high concurrent requests with X number of neutron ports already exists13:32
marunSreedhar: The load is tiny, though.13:32
marunSreedhar: Or rather, should be.13:32
Sreedharmarun: I think delay is not in booting the VM paths.. Delay is between port create/ip allocate till port is made up - In the email, i have shared the timing and logs13:33
*** matrohon has quit IRC13:33
marunSreedhar: I'm afraid I'm not being clear13:33
marunSreedhar: The delay is related to nova as much as anything else, since creating a port in neutron is only part of the story.13:34
marunSreedhar: The port allocation on the compute node is triggered by nova-compute.13:34
marunSreedhar: I'm saying, there needs to be profiling of all the pieces in the chain to find the bottlenecks and speed them up.13:35
*** changbl has quit IRC13:35
marunSreedhar: recording the total time and taking potshots at what we think might be the problem isn't a very effective strategy.13:35
marunSreedhar: does that make sense?13:37
Sreedharmarun: During the instance creation, network ports are being created and IP is getting allocated very fast from the logs. In each compute node approximately 2 instances would create in each batch.  But the OVS agent in compute nodes are getting many rpc timeout messages for security_group_rules_for_devices13:37
Sreedharmarun: If you can guide me where to profile, i am happy to do that and collect logs13:37
marunSreedhar: Right, because those rpc requests are all hitting a single process.13:38
marunSreedhar: I notice from your email that you are using m1.small.  Are you aware that devstack actually has an option for m1.nano, consuming only 64gb of ram?13:39
Sreedharmarun: yes, during my tests, i wanted to simulate close to customer environment, so i have used m1.small.13:41
marunSreedhar: So you're not using cirros images then?13:41
Sreedharmarun: I am using the Ubuntu Cloud Archive13:42
marunSreedhar: For the purposes of testing the kinds of issues we are seeing, I don't see any advantage to using large VMs13:42
marunSreedhar: In fact, the large ops gate uses a null virt driver (I think) that doesn't even use real vm's13:42
marunSreedhar: Narrowing issues related to networking doesn't actually require VM's, just endpoints that can behave as they do.13:43
marunsalv-orlando: are there any docs on the large ops gate?  I'd like to start using that in the hopes of speeding up iteration around these issues.13:44
salv-orlandomarun: I don't there is any documentation. I learned how it works looking at the configuration options for tempest and devstack in devstack-gate.yaml (project openstack-infra/config.git)13:46
marunsalv-orlando: gotcha, thanks13:46
*** matrohon has joined #openstack-neutron13:47
salv-orlandomarun: https://github.com/openstack-infra/config/blob/master/modules/openstack_project/files/jenkins_job_builder/config/devstack-gate.yaml#L37413:47
marunsalv-orlando: hmmm.  what do those environment variables configure?13:48
marunsalv-orlando: devstack? or some helper script?13:48
salv-orlandoThey pass parameters into the wrapper script of devstack-gate13:49
salv-orlandoopenstack-infra/devstack-gate.git13:49
marunah, ok13:49
salv-orlandoif you want to use that for reproducing a similar job locally on a infrastructure which is not the same as the upstream, they're very little useful13:50
marundefinitely13:51
maruni feel pretty dumb for having been using real vm's up until now13:51
salv-orlandothe bottom line for the large ops is that:13:51
salv-orlando1) set the number of vm to 150, 2) use the fake_virt driver, 3) run tox -elarge-ops13:52
jojo_any clue about connection on floating ips being lost after stressing add-remove operations?13:52
salv-orlandojojo_: this is something which we thought has been fixed in havana; notifications where treated in the wrong way on the agent allowing processing messages out of order13:53
salv-orlandoIf you have a repro script that shows this is happening still, please share it13:53
salv-orlandomarun: and btw for testing scalability of nova/neutron the fake virt driver is good, but for many tests you need real vms13:54
salv-orlandobecause otherwise you won't be stressing the agents, as no VIFs would show up on the integration bridge13:54
marunsalv-orlando: hmm, I didn't realize that13:55
marunsalv-orlando: I think it's heinous that the lxc support was dropped13:55
jojo_well, i am able to reproduce it in a small setup, 2 compute nodes, with 20 vms and rotating 5 floating-ips13:55
marunsalv-orlando: it would have been perfect for neutron testing13:55
jojo_connection is restored when l2 agent is restarted on compute nodes13:55
salv-orlandojojo_: the l2 agent? That's interesting. Are you rotating floating IPs or also rebooting instances?13:56
jojo_its a neutron/ovs/gre setup13:56
jojo_when neutron-openvswitch-plugin is restarted13:57
*** jistr has quit IRC13:57
jojo_conenction is working again13:57
jojo_i am rebooting instances too13:57
jojo_but when they have no floating-ip attached13:58
*** jistr has joined #openstack-neutron13:58
salv-orlandojojo_: got it. I was asking about the reboot thing because I am trying to understand if this is an issue so far unknown on the l3 agent or one of the issues we already working on the ovs agent and dhcp agent13:59
*** jecarey has joined #openstack-neutron14:00
jojo_ok14:00
salv-orlandoA perfect test for isolating the problem would be to a simple scree with keeps disassociating floating IPs from an instance and association it to another instance14:00
salv-orlandowithout rebooting any.14:00
salv-orlandoIs that a test you could do? If you don't have a test-env I can do it or find somebody to do that, but it might take a while.14:01
jojo_didnt think rebooting had anything to do14:01
jojo_i can do it14:01
salv-orlandook, that would be great; and if you collect the logs that would allows to gain an understanding of what's going on.14:01
jojo_right now, gonna run a test where floating-ip are rotated continuosly14:01
openstackgerritÉdouard Thuleau proposed a change to openstack/neutron: Add local ARP responder to the OVS agent (WIP)  https://review.openstack.org/4922714:12
*** yamahata_ has quit IRC14:12
*** yamahata_ has joined #openstack-neutron14:12
*** julim has joined #openstack-neutron14:12
*** dims has quit IRC14:12
Sreedharmarun: regarding instance not able to get ip, we see this issue more consistency if we have already some number of neutron ports exists. Does the active number of neutron ports play any role in rpc timeouts. If yes, does tuning more number of nova-conductor process helps anyway14:12
marunSreedhar: the number of database records is likely a contributor to performance degredation14:12
marunSreedhar: nova-conductor wouldn't have anything to do with it - that's nova only14:12
openstackgerritÉdouard Thuleau proposed a change to openstack/neutron: Add local ARP responder to the OVS agent  https://review.openstack.org/4922714:12
*** dims has joined #openstack-neutron14:12
marunSreedhar: oh, wait.  the mysql error is nova14:12
openstackgerritÉdouard Thuleau proposed a change to openstack/neutron: Add local ARP responder to the OVS agent  https://review.openstack.org/4922714:12
marunSreedhar, salv-orlando: The QueuePool settings are db-connection related, so doesn't there need to be coordination between all openstack services to ensure that the mysql connection limit is not reached?14:12
Sreedharmarun: I am using mysql as a database. Does any tuning to database will help (I am planning to increase max_connections). or more number of RabbitMQ servers helps14:12
openstackgerritA change was merged to openstack/neutron: Midonet to support port association at floating IP creation  https://review.openstack.org/5579214:12
marunSreedhar: It's a db issue.  I seriously doubt that rabbitmq is overloaded in these scenarios.14:12
*** heyongli has joined #openstack-neutron14:12
marunSreedhar: The problem is that the processing of rpc messages is db-bound in general and cpu bound in the case of the neutron service since it's single process.14:12
*** carl_baldwin has joined #openstack-neutron14:12
salv-orlandomarun: if you don't tune those parameters appropriately you risk getting the "mysql has gone away" error for instance, because all the project might use more connection than what mysql-server allows14:13
Sreedharmarun: With carl's patch when i set api_workers to 8, it started 8 neutron-server process. You mean to say these 8 process only serve api requests, but single neutron-server process will use for rpc requests14:13
marunSreedhar: correct.14:14
salv-orlandobut whether the coordination should pertain to openstack, that's a question I'm not able to answer. I see it as a configuration management problem, but I am far from being an expert on the matter.14:14
marunsalv-orlando: funny how development and operations bleed together....14:15
salv-orlandoSreedhar: yeah, that was the part we made sure of during review. There's always only one worker handling AMQP messages.14:15
jojo_salv-orlando: the test is already running for 15 minutes and connection is up for all 8 floating-ips14:15
jojo_salv-orlando: gonna let it run for 15-20 more mins14:15
salv-orlandojojo_: let's play the waiting game then...14:15
marunsalv-orlando, Sreedhar: I've asked carl_baldwin on the mailing list whether it would be possible to add multiple workers for RPC.14:15
jojo_;)14:15
marunsalv-orlando: is there any overriding reason not to?  i.e. not safe for concurrency?14:16
salv-orlandojojo_: neutron is more "eventually fallible" than "eventually consistent"14:16
salv-orlandoMeaning that if you keep it running, it will eventually fail :)14:16
marunlol14:16
jojo_:|14:17
salv-orlandomarun: we discussed this with markmclain. Do not remember the reason on top of my head, but we were worried about the fact this might have increased the occurrence of mysql/eventlet deadlocks14:18
marun*sigh*14:18
marunI'll have to censor myself at this point.'14:18
salv-orlandoor you can move to a non-logged channel if you are looking for venting some frustration14:19
marunAlthough, throwing eventlet under a bus doesn't seem like such a bad idea.14:19
marungood old process-based concurrency is looking pretty good right about now.  all the problems one has to worry about are up-front and not hidden, and I think that's a bette approach.14:19
*** afazekas has quit IRC14:20
salv-orlandothe other reason was that with multiple consumers/producers on the AMQP bus, there was a chance of out-of-order message processing and repeated notifications14:21
*** peristeri has joined #openstack-neutron14:21
marunsalv-orlando: that sounds like a problem somebody in the universe might have already given some thought to, no?14:22
*** Ken has joined #openstack-neutron14:22
*** Ken is now known as Guest8487414:23
marunsalv-orlando: sorry :(14:24
*** jojo_ is now known as wolf14:24
*** wolf is now known as jojo_14:24
*** clev has joined #openstack-neutron14:27
salv-orlandomarun: of course I would not dare to say the problem can't be solved. Perhaps I won't be able to solve it, but this is another story ;) I was just saying that it was a risk we did not want to take in the last days of the last milestone14:28
marunsalv-orlando: yeah, that's why I was apologizing.  We're gated by non-technical concerns as much as anything at this point.14:28
salv-orlandopretty much the same reason for which we decided not to default to ovsdbmonitor polling manager for havana14:29
marunsalv-orlando: don't you mean 'decided not to merge polling minimization for havana?' ;)14:30
salv-orlandono we merged it I think - in rc phase?14:30
salv-orlandowe decided not to make it the default approach14:31
*** otherwiseguy has quit IRC14:31
*** Guest84874 has quit IRC14:31
*** otherwiseguy has joined #openstack-neutron14:31
marunsalv-orlando: hmmm, maybe my patch was just half-assed. that sounds more likely.  good thing it wasn't made the default!14:32
*** markmcclain has joined #openstack-neutron14:32
marunsalv-orlando: nope, I was right the first time.  It didn't make it into havana at all.14:32
marunsalv-orlando: we still have to backport it.14:32
salv-orlandomarun: sorry about that I thought we merged it by RC-214:33
marunsalv-orlando: hoping to lean on otherwiseguy for that14:33
salv-orlandoI am happy to support the back port even if the patches are rather large14:33
salv-orlandothough I don't have +2 power on stable patches14:33
marunsalv-orlando: who does?  I think we're going to need more manpower on that this month14:33
carl_baldwinmarun: Just found your question in the mailing list.  Will respond.14:34
salv-orlandomarun from our team I think it's arosen and markmcclain14:34
salv-orlandoand garyk14:34
maruncarl_baldwin: appreciated.  though I think salv-orlando might have answered it in the channel (order-of-operations and mysql/eventlet deadlocks)14:34
*** carl_baldwin has quit IRC14:34
*** rkukura has joined #openstack-neutron14:35
anteayamarun: did you want to be a stable patch approver?14:35
anteayaI agree it would be great to have more14:35
markmcclainwe've have 3 stable reviewers on our team14:36
markmcclainarosen, garyk, and me14:36
markmcclainif you propose the backports we'll review them14:36
marunmarkmcclain: is there a need for more?14:36
jojo_salv-orlando: well, i am pretty sure there's no problem now, 35 minutes of test14:36
marunmarkmcclain: or is the current team sufficient?14:36
markmcclainwe could use a few more, but teh stable team looks for a consistent record of reviewing stable patches14:37
anteayahow's your review record of stable patches, marun?14:37
markmcclainstable core is also cross project14:37
marunanteaya: middling: http://russellbryant.net/openstack-stats/neutron-reviewers-30.txt14:37
markmcclainso it requires proposal and lazy consensus14:38
marunanteaya: scratch that, non-existent.  I have only been reviewing master.14:38
salv-orlandojojo_: perhaps the reboots have a role then. This might explain why you need to restart the l2 agent. I would look at port occasionally not being rewired or being rewired very late14:38
salv-orlandoI need to switch off for 2.5 hours now. Will back later.14:38
anteayamarun: ah okay, so I encourage stable approval if that is something that you want14:39
salv-orlando* Will be back14:39
jojo_hummmm14:39
anteayamarun: sounds like you have an idea of the path to get there, yeah?14:39
jojo_will try to get more info14:39
openstackgerritA change was merged to openstack/neutron: Improve unit test coverage for Cisco plugin nexus code  https://review.openstack.org/5710014:39
marunanteaya: yes14:39
jojo_this sounds like a big drawback14:39
marunmarkmcclain: so, dumb question.14:39
anteayamarun: go you \o/14:39
jojo_if no tunning/tweaking possible to fix it14:39
markmcclainmarun: no dumb questions14:40
marunmarkmcclain: I see an awful lot of patches being proposed that don't seem to apply to 1) stability or 2) parity14:40
marunmarkmcclain: wasn't the goal to minimize the review load while we are in crisis mode?  Or are some people just not getting the memo?14:40
markmcclainthe goal was focus on stability and parity14:40
* anteaya thinks marun is asking a great question14:40
markmcclainfor the most part we've done it14:41
markmcclainthere are also some bug fixes for existing code14:41
markmcclainwhich are generally acceptable14:41
marunHow does a reviewer know whether a patch is valid or not?14:41
marunpeople can keep churning on 'bug fixes' ad infinitum, though.  there are tons of things to 'fix' that aren't necessarily important14:41
markmcclainit's the bug severity is critical/high14:41
marunah, ok.14:42
markmcclainthen that is acceptable14:42
markmcclainif there are low priority items14:42
anteayaperhaps I need to create a list somewhere to help reviewers?14:42
markmcclainit is really up to the reviewer if they want to review14:42
anteayawould that help?14:42
anteayasimilar to the gate blocking bug roundup on the meeting agenda14:42
markmcclainif the severity of the bug is unknown feel free to ping the channel14:42
marunmarkmcclain: That sounds alot like 'anything goes' though.  so long as reviewers are willing.14:43
markmcclainwell low priority items are always on a has time available basis14:43
anteayaI'm in agreement with marun, it would be great to have some sort of system to assist with reviewer focus14:43
markmcclainif you've reviewed all of the urgent items14:43
*** mjbright_ has joined #openstack-neutron14:43
markmcclainthen it is still helpful to review the items with lesser priority14:43
marunmarkmcclain:  I think my question is as much about the patches people are pushing as the cost of reviewing them, tough.14:44
markmcclainsome of hte bug fixes actually improve the stability of certain plugins14:44
*** amotoki_ has joined #openstack-neutron14:44
marunmarkmcclain: Why are people allowed to push low priority patches when we have urgent issues that need addressing?14:44
markmcclainand once more entities implement 3rd party testing I expect there will more of those type bugs14:44
markmcclainwe are an open community, so anyone can propose anything14:45
*** carl_baldwin has joined #openstack-neutron14:45
marunmarkmcclain: …with predictable results, frankly.14:45
markmcclainbut as a community we can decide the priority that we review the proposals14:45
anteayagreat14:45
anteayaso if I were to create an etherpad with links to reviews14:46
anteayaperhaps as a group we can discuss the priority?14:46
anteayawith bug fix priority referenced for the patch url14:46
anteayaI'll do that and then let's see how that gets used14:46
marunanteaya: I think mark might have been referring to the priority in launchpad14:46
markmcclainyou can just use this link: https://bugs.launchpad.net/neutron/+bugs?search=Search&field.importance=Critical&field.importance=High&field.status=New&field.status=Confirmed&field.status=Triaged&field.status=In+Progress14:47
mesteryso, along those lines, I see a lot of reviews proposed with the bug priority set to unknown ...14:47
mesteryIn cases like that, I usually set the bug to low priority, but it's somewhat bothered me that people submit patches for bugs without a priority.14:47
markmcclainwell not everyone can set priority14:48
mesteryAh, got it. Thanks markmcclain, didn't know that.14:48
markmcclainsomeone with triage rights has to set it14:48
mesteryOK, so I guess I'm actually doing my job by setting priority then :)14:48
marunit kind of bothers me when people submit patches without talking to anybody about it, frankly.14:48
anteayamarkmcclain: who has triage rights currently?14:49
markmcclainthe priority of a bug is wrong or unknown feel free to ping the room14:49
markmcclainthe core team should have it14:49
mesterygot it, thanks14:49
markmcclainif you don't PM me14:49
* mestery nods14:49
* markmcclain steps away from keyboard for a bit14:49
*** markmcclain has quit IRC14:49
*** mjbright_ has quit IRC14:49
carl_baldwinmarun: Mysql/eventlet blocking seems to be the deeper problem.  Scaling out workers was meant to be a stop-gap to alleviate it.14:53
*** jojo_ is now known as _jj_14:53
HenryGDoes gerrit have a way of ordering the review queue by priority of the bug(s) they reference?14:53
maruncarl_baldwin: so I guess the short-term solution for rpc will be profiling and optimizing the hot paths14:53
marunHenryG: I don't believe so, but you should be able to work some magic with one of the gerrit cli tools.14:54
anteayamarun: thanks for bringing up the conversation14:55
anteayaI agree that patch submission/review should be following our stated focus for the project14:55
anteayaand outliers to that focus should be discussed14:55
carl_baldwinHenryG: This isn't exactly what you asked for but this page is an attempt at ordering reviews by priority:  http://status.openstack.org/reviews/.   It would be nicer if gerrit could do this.  it would need to get the priority from launchpad.  I'm not sure if it can do that.14:56
marunanteaya: :\  a bit too agro.  Even knowing what to expect with opensource I still find it frustrating that we can't coordinate on priorities more effectively.  But as Mark suggests, participants can't be ordered about.14:57
carl_baldwinmarun: Multiple RPC workers could also alleviate it in the short term.  Is there a case where the RPC process is still CPU bound after separating out the API worker processes?  (Sorry if I missed some context up in the channel)14:58
*** s3wong has joined #openstack-neutron14:59
maruncarl_baldwin: when we use multiple wsgi workers, rpc requests from the agents become the bottleneck15:00
carl_baldwinmarun: I see15:01
*** networkstatic_zZ is now known as networkstatic15:01
maruncarl_baldwin: but according to markmcclain and salv-orlando, it isn't possible to trivially use multiple workers for rpc because processing rpc requests out of sequence can be dangerous15:02
*** networkstatic has quit IRC15:02
*** Mr_W has joined #openstack-neutron15:02
*** networkstatic has joined #openstack-neutron15:03
enikanorov_marun: I'd like to comment on your question15:03
maruncarl_baldwin: i was talking to salv-orlando yesteray about this and I think we came to agree that rate limiting and optimizing hot spots might be the best short-term solution, short of rearchitecture15:03
marunenikanorov_: do tell15:03
enikanorov_I think we should not prohibit patches which introduce new features. Or otherwise we will distract a part of community15:04
enikanorov_that are expecting them and working on them15:04
enikanorov_poeople are aware of what core team focus is right now15:04
marunenikanorov_: you don't have to worry.  It's only me that thinks its stupid to continue business as usual.15:04
carl_baldwinmarun: was that discussion in the channel?  I could go read the logs.15:04
maruncarl_baldwin: yes, it was in the channel15:04
*** alagalah has joined #openstack-neutron15:05
openstackgerritA change was merged to openstack/neutron: Fix MeteringLabel model to not clear router's tenant id on deletion  https://review.openstack.org/5604115:05
enikanorov_the 'business' is actually core's which prioritize their time. and lots of other folks may be not so deeply involved in neutron to help with quite complex problems that gate tests has uncovered15:05
*** alagalah has left #openstack-neutron15:06
marunenikanorov_: I think feature development is completely broken, at present.15:06
marunenikanorov_: people are free to merge patches without real tests.15:06
marunenikanorov_: unit tests prove nothing15:06
enikanorov_marun: I think Icehouse has a new policy for that15:06
marunenikanorov_: so 'feature' additions are as likely to make stability worse.15:06
enikanorov_to address this exact issue15:07
enikanorov_in our process15:07
marunenikanorov_: how is it being enforced exactly?15:07
marunenikanorov_: people merge patches15:07
*** Vijay__ has joined #openstack-neutron15:07
marunenikanorov_: eventually (hopefully) tempest or functional patches merge15:07
enikanorov_well, it is enforced as usual15:07
marunenikanorov_: if they don't, how do do we remove those patches?15:07
marunenikanorov_: i.e. it is not enforced.15:07
*** mjbright_ has joined #openstack-neutron15:07
enikanorov_you go and put -2 until you see a patch passing specific tempest tests15:07
*** heyongli has quit IRC15:07
marunenikanorov_: that means absolutely nothing without good tempest coverage15:08
enikanorov_which are not present15:08
enikanorov_so you -2 until tempest has test15:08
marunenikanorov_: except that we can't merge a tempest test until the feature exists15:08
marunenikanorov_: --> broken15:08
enikanorov_well, there could be such cases as well15:08
carl_baldwinmarun: I will read up on it in the log.  I think it is already possible to run more than one RPC message processor.  If the neutron server process is run on multiple hosts in active/active I think you end up getting multiple independent RPC processing threads unless I'm missing something.15:09
marunenikanorov_: so this policy, does it require that a given feature have an integration test in review before merge?15:09
marunenikanorov_: I haven't seen evidence of this, but I might simply be ignorant.15:09
maruncarl_baldwin: is active/active an option?15:10
enikanorov_marun: that is how I understand the policy. just like you said.15:12
*** _jj_ has quit IRC15:12
marunenikanorov_: Is there documentation you can point me to?15:12
openstackgerritdekehn proposed a change to openstack/neutron: extra_dhcp_opt add checks for empty strings  https://review.openstack.org/5985815:12
carl_baldwinmarun: I think so but I will confirm that is the case.  I'm running neutron server on multiple hosts and I think that the RPC processor is active on both.  I'm certain that the API server is active on both.15:12
marunenikanorov_: It sounds like I am simply ignorant.15:12
enikanorov_marun: no, i based by understanding on the summit session15:12
*** _jj_ has joined #openstack-neutron15:12
enikanorov_where it was discussed15:12
marunenikanorov_: fair enough.  I worry that this detail is getting lost in the shuffle.15:13
enikanorov_it was primarily about vendor-specific stuff, but obviously as we targeting stability, the same should be applied to core functionality as well15:13
marunenikanorov_: I would tend to agree.15:13
maruncarl_baldwin: Hmm…  I'd be interested in hearing what you find out.  My expectation would be that active/active would have the potential to process RPC messages out of sequence and introduce consistency problems.15:16
carl_baldwinmarun: I'll be sure to let you know what I find.15:16
*** yamahata_ has quit IRC15:17
*** yamahata_ has joined #openstack-neutron15:17
*** yfried has quit IRC15:18
*** _jj_ has quit IRC15:20
*** carl_baldwin has quit IRC15:22
marun01738715:23
*** marun has quit IRC15:23
*** devlaps has joined #openstack-neutron15:26
*** bvandenh has joined #openstack-neutron15:30
*** jgrimm has joined #openstack-neutron15:31
*** amritanshu_RnD has quit IRC15:32
*** banix has joined #openstack-neutron15:36
openstackgerritJianing Yang proposed a change to openstack/neutron: Fix str2dict and dict2str's incorrect behavior  https://review.openstack.org/6019515:37
anteayamarun: that is true15:39
anteayabut we can work towards focus regarding reviews and that can help15:39
anteayamarios_: I have a question, what do you mean by agro?15:40
anteayaI need to spend some time with the gerrit ssh commands and will report any useful findings I trip over15:41
anteayamarun: the question I gave to marios_ was meant for you15:44
anteayaand I think that we need to have more discussion so that everyone can see how our new policy is being enforced15:44
anteayaif you can't see it and point to it, they there are others who have no reference for its enforcement as well15:45
anteayaI'm not core, and only cores can approve15:45
anteayaI can support the conversation but my direct actions don't have an effect on enforcement15:46
*** Vijay__ has quit IRC15:50
*** jprovazn has quit IRC15:58
*** aymenfrikha has joined #openstack-neutron16:00
*** amuller has quit IRC16:04
*** mlavalle has joined #openstack-neutron16:05
*** yamahata_ has quit IRC16:06
*** carl_baldwin has joined #openstack-neutron16:07
*** rudrarugge has joined #openstack-neutron16:09
anteayawell one can conduct a query on review.openstack.org targetting the topic: ssh -p 29418 anteaya@review.openstack.org gerrit query --format=JSON status:open project:openstack/tempest topic:^bug.* limit:1016:11
anteayaso if we developed a culture where the status of the bug is included in the topic16:12
*** jprovazn has joined #openstack-neutron16:12
anteayafor instance topic:^bug/critical/.*16:12
anteayathen the query could return all critical bugs16:13
anteayabe we would need to manifest that internally for it to work16:13
anteayamarun: ^16:13
*** rossella_s has quit IRC16:15
anteayamlavalle: hello16:15
mlavalleanteaya: hi16:15
*** changbl has joined #openstack-neutron16:17
anteayahave you taken a look at the addition I made to the api gap anaylsis on the etherpad?16:20
mlavalleanteaya: give me a minute16:21
anteayak16:22
mlavalleanteaya: do you mean at the top about selecting a topic  from the list, put your name, etc...?16:23
anteayahttps://etherpad.openstack.org/p/icehouse-summit-qa-neutron16:23
anteayayes16:23
anteayadoes that make sense to you?16:23
mlavalleanteaya: that's perfect16:23
*** alex_klimov1 has quit IRC16:23
anteayaawesome16:23
anteayaplease encourage people to create a bug16:24
mlavalleby this coming Monday I will have a wiki page with instructions aimed at new contributors walking them through the process of developing API tests specifically for Neutron16:24
anteayaif you could do one or two as an example  that might help seed the direction16:25
anteayawoohoo16:25
anteayago you16:25
anteayathat will be awesome16:25
mlavalleOnce the wiki is up, I will send a message to the whole openstack mail list16:26
anteayasounds good16:26
anteayado you want to post in channel first and get some feedback from others in this room?16:26
anteayajust to clear up any instruction bugs16:27
anteayaand then to the ml?16:27
mlavalleperfect, that's a great idea16:27
anteayagreat16:27
mlavalleI also made a little change to the Neutron wiki: https://wiki.openstack.org/wiki/Neutron16:28
anteayahow is your current understanding of tempest?16:28
mlavalleLook at the Related Projects section16:28
anteayado you feel you have enough knowledge to help other -neutron devss?16:28
* anteaya clicks16:28
mlavalleYes, I have enough understanding to help other Neutron debs with Tempest16:28
anteayathat looks great16:29
anteayawell done16:29
anteayaperfect16:29
anteayaare you able to be available more on irc?16:30
mlavalleI am connected every day16:30
anteayaon more of a daily basis, during your working days?16:30
mlavalleall day long16:30
anteayahmmmm, I seem to miss you16:30
mlavallewe use it at work also16:30
anteayaokay great16:30
anteayado you read the logs? http://eavesdrop.openstack.org/irclogs/%23openstack-neutron/16:31
anteayawhat TZ are you in?16:31
mlavallethe other thing you should be aware of is that the gap analysis is not over yet…. I have finished core and l3 level, but over the next few days I will continue doing L4 to L716:31
anteayaokay16:32
mlavalleI am not in the habit of reading the logs, but I will try to start doing it16:32
mlavalleI am in Texas, thats CST16:32
anteayalet's see if we can get people to turn the gaps into bugs in what you have covered, while you continue16:32
mlavallecool16:33
anteayathanks, yes I tend to just post in the channel as I think of points16:33
anteayaokay CST16:33
mlavallei belive you are in Toronto, that's EST, right16:33
anteayaI'll probably overlap with you more next week then16:33
anteayaEST when I'm home, yes16:34
anteayacurrently GST +116:34
mlavalleso you are 1 hour ahead of me16:34
anteayausually yes16:34
mlavalleok16:34
anteayacan you connect with rosella_s and see how her tempest understanding is coming along?16:35
mlavallesure, i'll try to do it this afternoon…. where is she located? do you know?16:35
anteayashe is currently bogged down in her assessment and if you could help unblock her, that would be great16:35
anteayashe is in Spain16:35
mlavalleah ok, I am originally from Mexico City, so I speak Spanish16:36
anteayaso would have to be morning for you and later for her16:36
anteayacool16:36
anteayaher English is great, but whatever works for the two of you16:36
mlavalleI'll follow up with her16:36
anteayathank you16:36
*** pete5 has joined #openstack-neutron16:43
*** SumitNaiksatam has quit IRC16:43
anteayaat a conference, talks are wrapping up, soon going to be offline until tomorrow16:45
anteayabut my server is listening to post questions you have and I will do my best to address them when I am next online16:45
anteayathanks16:45
*** litong has joined #openstack-neutron16:46
anteayaroaet: oh before I go, that bug https://bugs.launchpad.net/neutron/+bug/125144816:46
anteayaroaet: any updates on this bug?16:46
roaetanteaya: not at the moment, we have a meeting friday to discuss it with our nova folk and neutron folk16:47
roaetSince it appears to be a race condition16:47
anteayaokay thanks16:47
anteayagreat, can you update the bug report with any comment after that discussion16:47
roaetof course16:48
anteayaeven if it is just noting taht the discussion happened16:48
anteayathanks16:48
anteayajog0: can you take another look at the logstash for https://bugs.launchpad.net/neutron/+bug/125016816:50
*** networkstatic has quit IRC16:50
anteayaI am seeing no hits for it since Dec. 316:50
anteayacan you confirm?16:51
*** rossella_s has joined #openstack-neutron16:52
anteayajog0: can I also get a logstash url in https://bugs.launchpad.net/nova/+bug/1210483?16:52
anteayasorry if it obvious how to do that, I think I need to up my logstash foo16:52
anteayaokay now I am afk16:53
*** x86brandon has joined #openstack-neutron16:54
*** mjbright_ has quit IRC16:56
*** rpodolyaka has left #openstack-neutron16:56
*** _jj_ has joined #openstack-neutron16:57
_jj_well, about the issue with floating-ip change and instance reboot16:58
_jj_have been conducting tests for hours16:59
_jj_it has something to do for sure with rebooting but i cannot see anything abnormal16:59
openstackgerritA change was merged to openstack/neutron: Ensure NVP API connection port is always an integer  https://review.openstack.org/5875116:59
_jj_in my two node setup, right now have all the instances on host A without public connection and all the instance on host B with connection up and runing17:00
*** s3wong has quit IRC17:01
_jj_any suggestion where to look at?17:01
openstackgerritAnn Kamyshnikova proposed a change to openstack/neutron: Fix downgrade in migration  https://review.openstack.org/5309117:03
*** SumitNaiksatam has joined #openstack-neutron17:04
*** clev_ has joined #openstack-neutron17:05
*** clev has quit IRC17:05
beagles_jj_, I'd look for changes in the ports and mac addrs17:05
beaglesnot ports as in IP ports, but neutron ports17:06
*** rudrarugge has quit IRC17:08
openstackgerritDirk Mueller proposed a change to openstack/python-neutronclient: Use assertIn where appropriate  https://review.openstack.org/5916517:11
*** julim has quit IRC17:16
*** julim has joined #openstack-neutron17:17
_jj_beagles: well  i have not seen anything weird in neutron ports17:22
_jj_i keep on testing17:22
*** armax has joined #openstack-neutron17:24
*** armax has quit IRC17:25
*** yfried has joined #openstack-neutron17:26
*** chandankumar has quit IRC17:26
*** armax has joined #openstack-neutron17:27
openstackgerritÉdouard Thuleau proposed a change to openstack/neutron: Add local ARP responder to the OVS agent  https://review.openstack.org/4922717:27
*** armax has quit IRC17:28
*** armax has joined #openstack-neutron17:28
*** armax has quit IRC17:32
*** armax has joined #openstack-neutron17:34
*** armax has quit IRC17:34
*** armax has joined #openstack-neutron17:34
*** clev_ has quit IRC17:34
pete5arson, could please you comment on why https://launchpad.net/neutron/+bug/1177973 isn't havana backport potential anymore. I see that you removed the tag a couple of weeks ago. Thanks!17:40
salv-orlando_jj_: I am back; after rebooting an instance, it would be good to understand whether it's the floating ip not working for some reason or is networking not ready on the instance.17:40
pete5arosen (not arson :-) ) could please you comment on why https://launchpad.net/neutron/+bug/1177973 isn't havana backport potential anymore. I see that you removed the tag a couple of weeks ago. Thanks!17:40
*** jistr has quit IRC17:41
*** armax has quit IRC17:42
*** armax has joined #openstack-neutron17:43
*** alagalah has joined #openstack-neutron17:49
*** armax has quit IRC17:49
*** armax has joined #openstack-neutron17:49
*** rudrarugge has joined #openstack-neutron17:55
*** mlavalle has quit IRC18:02
*** nati_ueno has joined #openstack-neutron18:02
*** rossella_s has quit IRC18:03
*** mlavalle has joined #openstack-neutron18:04
arosenpete5: I'll re-add the tag. I just looked at the patch again. Seems fine. Usually adding additional config options in backports isn't really great.18:10
salv-orlandoarosen: isn't there also an additional dependency on psutil? Or am I dreaming of it?18:12
arosensalv-orlando:  doesn't look like it: https://review.openstack.org/#/c/45678/1118:13
*** harlowja has joined #openstack-neutron18:14
salv-orlandoarosen: cool I can't find it either in that patch nor the dependent ones, but I remember having to re-create the tox env because of psutil while reviewing the patch. But it could have been just me being high on caffeine18:16
pete5Does anybody know what Kyle Mestery's IRC handle is?18:17
*** csd_ has joined #openstack-neutron18:18
mesterymestery18:18
arosenha18:18
mestery:)18:18
*** csd_ is now known as csd18:18
pete5mestery: :-)    When you've got a moment, could you comment on the ongoing discussion on https://review.openstack.org/#/c/58854/18:18
mesterypete518:19
mesterypete5: Sure, will do, thanks for pointing this out!18:19
*** csd has quit IRC18:26
*** jlibosva has quit IRC18:27
*** jlibosva has joined #openstack-neutron18:30
*** ihrachyska has quit IRC18:37
*** csd has joined #openstack-neutron18:38
*** fouxm has quit IRC18:42
*** rudrarugge has quit IRC18:43
*** clev has joined #openstack-neutron18:47
*** terrylhowe has joined #openstack-neutron18:48
terrylhoweA subnet and a port can be added to a router.  Are there any other likely use cases for interface add?18:50
*** suresh12 has joined #openstack-neutron18:50
*** Abhishe__ has joined #openstack-neutron18:51
terrylhoweI'm working on the unified CLI and I was thinking of making the commands "router add port" and "router add subnet"18:53
*** ihrachyska has joined #openstack-neutron18:53
*** mlavalle has quit IRC18:55
openstackgerritEd Bak proposed a change to openstack/neutron: Change to improve dhcp-agent sync_state  https://review.openstack.org/5986319:03
*** armax has quit IRC19:11
openstackgerritShree Duth Awasthi proposed a change to openstack/python-neutronclient: Misc typo in neutronclient  https://review.openstack.org/6032419:12
*** yfried has quit IRC19:13
*** armax has joined #openstack-neutron19:13
*** armax has quit IRC19:14
*** alagalah has left #openstack-neutron19:14
*** armax has joined #openstack-neutron19:16
jog0anteaya: for 1210483?19:17
jog0anteaya: I need a entry in nova-compute or a neutron service to use as a fingerprint19:18
*** ywu has quit IRC19:19
jog0and for https://bugs.launchpad.net/neutron/+bug/1250168 your analysis is correct it turns out that was the new hpcloud going online19:19
*** armax has quit IRC19:19
jog0we were using it in gate and it was slowe. also we dropped the large-ops test down to 100 instances because 150 was too close to a failure case on slower clouds.  And although we should make that test work for numbers larger then 150 we have much bigger issues to sort out19:20
*** armax has joined #openstack-neutron19:20
*** armax has quit IRC19:20
jog0anteaya: https://review.openstack.org/#/c/60000/19:21
*** armax has joined #openstack-neutron19:21
*** clev has quit IRC19:30
*** clev has joined #openstack-neutron19:34
*** Sreedhar has quit IRC19:35
openstackgerritJon Grimm proposed a change to openstack/neutron: Openvswitch update_port should return updated port info  https://review.openstack.org/5884719:39
*** jpich has quit IRC19:40
*** aveiga has joined #openstack-neutron19:46
openstackgerritJon Grimm proposed a change to openstack/neutron: Fix ml2 plugin for allowedaddresspairs tests  https://review.openstack.org/5889619:51
*** armax has quit IRC19:55
*** armax has joined #openstack-neutron19:55
*** jlibosva has quit IRC19:55
*** SridarK has joined #openstack-neutron19:57
*** ihrachyska has quit IRC19:59
*** terrylhowe has left #openstack-neutron20:08
*** bvandenh has quit IRC20:09
*** julim has quit IRC20:16
*** suresh12 has quit IRC20:17
*** rkukura has quit IRC20:17
*** Abhishe__ has quit IRC20:25
openstackgerritJon Grimm proposed a change to openstack/neutron: Fix ml2 & nec plugins for allowedaddresspairs tests  https://review.openstack.org/5889620:26
*** suresh12 has joined #openstack-neutron20:28
*** armax has left #openstack-neutron20:32
*** aymenfrikha has quit IRC20:39
*** aymenfrikha has joined #openstack-neutron20:42
*** jprovazn has quit IRC20:50
*** mlavalle has joined #openstack-neutron20:50
*** ywu has joined #openstack-neutron20:50
*** banix has quit IRC20:51
*** networkstatic has joined #openstack-neutron20:55
*** julim has joined #openstack-neutron20:56
*** sbasam has joined #openstack-neutron20:56
*** Abhishek_ has joined #openstack-neutron20:57
openstackgerritShiv Haris proposed a change to openstack/neutron: Implementaion of Mechanism driver for Brocade VDX cluster of switches  https://review.openstack.org/6012920:59
*** pcm_ has quit IRC21:03
*** banix has joined #openstack-neutron21:04
openstackgerritCarl Baldwin proposed a change to openstack/neutron: Use information from the dnsmasq hosts file to call dhcp_release  https://review.openstack.org/5626321:12
*** banix has quit IRC21:14
*** banix has joined #openstack-neutron21:18
openstackgerritSalvatore Orlando proposed a change to openstack/neutron: Test commit for testing parallel job in experimental queue  https://review.openstack.org/5742021:25
*** yfried has joined #openstack-neutron21:36
*** yamahata_ has joined #openstack-neutron21:37
*** arosen has quit IRC21:43
*** arosen has joined #openstack-neutron21:44
*** yfried has quit IRC21:49
asadoughineutron security-group-show is hard to read on a non-giant display21:50
*** banix has quit IRC21:50
*** banix has joined #openstack-neutron21:51
*** networkstatic has quit IRC22:01
*** banix has quit IRC22:03
*** litong has quit IRC22:17
*** jgrimm has quit IRC22:18
openstackgerritDane LeBlanc proposed a change to openstack/neutron: Improve unit test coverage for Cisco plugin common code  https://review.openstack.org/6037022:18
*** clev has quit IRC22:19
*** aveiga has quit IRC22:24
*** arosen has quit IRC22:32
*** arosen has joined #openstack-neutron22:32
*** changbl has quit IRC22:32
*** sbasam has left #openstack-neutron22:35
*** banix has joined #openstack-neutron22:37
*** gdubreui has joined #openstack-neutron22:41
*** yamahata_ has quit IRC22:45
*** rkukura has joined #openstack-neutron22:46
*** openstackgerrit has quit IRC22:48
*** openstackgerrit has joined #openstack-neutron22:48
*** rkukura has quit IRC23:04
*** nati_uen_ has joined #openstack-neutron23:04
*** nati_ueno has quit IRC23:07
*** clev has joined #openstack-neutron23:13
*** jecarey has quit IRC23:22
*** nati_uen_ has quit IRC23:34
*** nati_ueno has joined #openstack-neutron23:34
*** clev has quit IRC23:36
openstackgerritEdgar Magana proposed a change to openstack/neutron: Implements provider network support in PLUMgrid plugin  https://review.openstack.org/6038323:46
*** openstackgerrit has quit IRC23:47
*** openstackgerrit has joined #openstack-neutron23:47
*** peristeri has quit IRC23:49
*** mlavalle has quit IRC23:54
*** carl_baldwin has quit IRC23:59

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!