| *** gdubreui has quit IRC | 00:02 | |
| *** markmcclain has joined #openstack-neutron | 00:05 | |
| *** carl_baldwin has quit IRC | 00:15 | |
| *** banix has quit IRC | 00:17 | |
| *** dave_tucker has quit IRC | 00:17 | |
| *** yfujioka has joined #openstack-neutron | 00:26 | |
| openstackgerrit | Shiv Haris proposed a change to openstack/neutron: Testing https://review.openstack.org/61273 | 00:29 |
|---|---|---|
| *** iwamoto has joined #openstack-neutron | 00:34 | |
| *** dave_tucker has joined #openstack-neutron | 00:37 | |
| *** djbkd has quit IRC | 00:40 | |
| *** SumitNaiksatam has joined #openstack-neutron | 00:41 | |
| *** networkstatic has quit IRC | 00:43 | |
| *** yamahata has joined #openstack-neutron | 00:47 | |
| *** klindgren_ has quit IRC | 00:49 | |
| *** gdubreui has joined #openstack-neutron | 00:50 | |
| *** mlavalle has quit IRC | 00:57 | |
| *** dave_tucker has quit IRC | 01:01 | |
| *** dave_tucker has joined #openstack-neutron | 01:09 | |
| *** dhellmann has joined #openstack-neutron | 01:10 | |
| *** dhellmann has left #openstack-neutron | 01:11 | |
| jog0 | anteaya: can you make sure this one gets hunted down https://bugs.launchpad.net/neutron/+bug/1244255 | 01:22 |
| * anteaya clicks | 01:25 | |
| *** dave_tucker has left #openstack-neutron | 01:26 | |
| anteaya | jog0: this is the fingerprint I have for it: http://bit.ly/1i05EwL | 01:27 |
| anteaya | walk with me for a minute on this one | 01:27 |
| *** dt has joined #openstack-neutron | 01:27 | |
| jog0 | anteaya: you can see the fingerprint here status.openstack.org/elastic-recheck/ | 01:27 |
| anteaya | how do we have 39 failures all at the same time for this fingerprint? | 01:27 |
| jog0 | anteaya: I know nothing about that bug | 01:27 |
| jog0 | just that we are seeing it and its stalled | 01:27 |
| *** dt has left #openstack-neutron | 01:28 | |
| *** ashaikh has quit IRC | 01:28 | |
| anteaya | help me with the logstash here for a minute | 01:28 |
| anteaya | so on 2013-12-16 | 01:28 |
| anteaya | there are 39 hits with teh exact same timestamp | 01:28 |
| anteaya | can you confirm? | 01:28 |
| *** dave_tucker has joined #openstack-neutron | 01:28 | |
| jog0 | checking | 01:28 |
| anteaya | the first 5 hits all have the same build_uuid e7f0fb8a68b84bad8f26b502fff76545 | 01:29 |
| anteaya | those 39 hits | 01:31 |
| anteaya | only have 3 unique build_uuids | 01:31 |
| anteaya | so does this mean that there are actually only 3 hits for that bug at the time? | 01:31 |
| anteaya | the build_uuid is a unique identifier for that job run, is it not? | 01:32 |
| jog0 | anteaya: look and the build_change | 01:32 |
| *** dave_tucker has left #openstack-neutron | 01:32 | |
| jog0 | they are all for https://review.openstack.org/#/c/59787/ | 01:32 |
| * anteaya clicks the build change | 01:32 | |
| jog0 | so this bug may not be in gate after all | 01:32 |
| *** dave_tucker has joined #openstack-neutron | 01:33 | |
| *** dave_tucker is now known as dave_tucker_zzz | 01:33 | |
| jog0 | I need to head out, can you update that bug with our findings | 01:34 |
| anteaya | jog0: same build here http://bit.ly/1cPJycf | 01:35 |
| anteaya | what did we find? | 01:35 |
| anteaya | multiple hits for the same build_change? | 01:35 |
| anteaya | is this a bug, or expected logstash behaviour? | 01:35 |
| anteaya | I don't know | 01:35 |
| jog0 | anteaya: that this has only been seen in one patch set https://review.openstack.org/#/c/59787/ | 01:36 |
| anteaya | can do | 01:36 |
| jog0 | anteaya: and yeah expected behavior | 01:36 |
| jog0 | I'll explain tomorrow | 01:36 |
| anteaya | so perhaps the patch is the issue? | 01:36 |
| anteaya | and thanks | 01:36 |
| jog0 | in short logstash treats each line in a log file as a seperate item | 01:36 |
| anteaya | ah | 01:38 |
| anteaya | build_change is my new best friend | 01:38 |
| *** dave_tucker_zzz is now known as dave_tucker | 01:38 | |
| anteaya | happy next item on your calendar | 01:38 |
| anteaya | tty tomorrow | 01:38 |
| *** dzyu has joined #openstack-neutron | 01:39 | |
| jog0 | o/ | 01:40 |
| *** dave_tucker is now known as dave_tucker_zzz | 01:41 | |
| openstackgerrit | Micheal Thompson proposed a change to openstack/neutron: Corrected PEP8 issues Made corrections to apiObj https://review.openstack.org/62792 | 01:44 |
| anteaya | jog0 commented | 02:01 |
| anteaya | jog0: all hits for that bug right now are from a single patch | 02:04 |
| anteaya | I also commented on the patch, since it might indeed be the source of a problem | 02:04 |
| anteaya | and the tests might indeed be failing appropriately | 02:04 |
| *** banix has joined #openstack-neutron | 02:08 | |
| *** dave_tucker_zzz has quit IRC | 02:08 | |
| *** dave_tucker has joined #openstack-neutron | 02:14 | |
| *** dave_tucker is now known as dave_tucker_zzz | 02:15 | |
| openstackgerrit | Micheal Thompson proposed a change to openstack/neutron: Corrected PEP8 issues https://review.openstack.org/62792 | 02:15 |
| *** dave_tucker_zzz is now known as dave_tucker | 02:16 | |
| *** krast has joined #openstack-neutron | 02:23 | |
| *** dave_tucker is now known as dave_tucker_zzz | 02:23 | |
| *** gdubreui has quit IRC | 02:26 | |
| *** vkozhukalov has joined #openstack-neutron | 02:54 | |
| *** gdubreui has joined #openstack-neutron | 03:00 | |
| *** haleyb has quit IRC | 03:03 | |
| *** nati_uen_ has quit IRC | 03:04 | |
| *** alexpilotti has quit IRC | 03:11 | |
| *** networkstatic has joined #openstack-neutron | 03:16 | |
| openstackgerrit | Micheal Thompson proposed a change to openstack/neutron: Added A10 LBaaS Driver to branch. https://review.openstack.org/62464 | 03:30 |
| openstackgerrit | Micheal Thompson proposed a change to openstack/neutron: Corrected PEP8 issues Made corrections to apiObj https://review.openstack.org/62792 | 03:30 |
| marun | beagles: ping | 03:39 |
| *** jp_at_hp has quit IRC | 03:49 | |
| *** harlowja is now known as harlowja_away | 03:51 | |
| *** ljjjustin has joined #openstack-neutron | 03:54 | |
| openstackgerrit | Micheal Thompson proposed a change to openstack/neutron: modified: .gitignore modified: neutron/services/loadbalancer/drivers/a10networks/a10_api_2_1/core/a10logger.py modified: neutron/services/loadbalancer/drivers/a10networks/a10_api_2_1/core/apiObj.py modified: neutron/services/loadbalancer/drive https://review.openstack.org/62803 | 03:56 |
| *** terry_howe has left #openstack-neutron | 03:56 | |
| *** layer427expert has joined #openstack-neutron | 03:57 | |
| layer427expert | is there anyone who can answer a question about git review? | 03:57 |
| dkehn | I'll try | 03:58 |
| *** terry_howe has joined #openstack-neutron | 03:59 | |
| layer427expert | So a couple of things I am trying to setup an automated system for testing and commit | 03:59 |
| layer427expert | can we go to another channel? | 03:59 |
| dkehn | ? | 03:59 |
| dkehn | whats up | 03:59 |
| dkehn | sure | 04:00 |
| layer427expert | Sorry the issues is when I commit and then review I notice that my changes are going in as | 04:00 |
| layer427expert | new commits when I am trying to patch. | 04:00 |
| dkehn | is this after you do the inital commit | 04:01 |
| layer427expert | yes | 04:01 |
| dkehn | are you using the --amend to keep the same CHAGE_ID | 04:01 |
| dkehn | change_id | 04:01 |
| dkehn | change-is | 04:01 |
| dkehn | change-id, crap | 04:01 |
| layer427expert | ya | 04:01 |
| *** ashaikh has joined #openstack-neutron | 04:01 | |
| layer427expert | then he walks away | 04:01 |
| layer427expert | no I am missing that | 04:02 |
| layer427expert | thanks | 04:02 |
| layer427expert | I know it was something simple... thanks for the help.. | 04:02 |
| *** aymenfrikha has quit IRC | 04:03 | |
| *** suresh12 has quit IRC | 04:03 | |
| dkehn | so after you do the inital commit and the git review, you will notice a change-id: see https://review.openstack.org/#/c/58860/, after that point you just do a git commit -a --amend which will keep the change-id, and you don't need to change anything in the commit | 04:03 |
| layer427expert | OIC.... | 04:04 |
| dkehn | layer427expert: np, I'm off to bed | 04:04 |
| openstackgerrit | Dave Cahill proposed a change to openstack/neutron: Midonet plugin: Fix source NAT https://review.openstack.org/62805 | 04:04 |
| layer427expert | ythank gn | 04:04 |
| *** inara has quit IRC | 04:14 | |
| *** chandankumar has joined #openstack-neutron | 04:15 | |
| *** julim has quit IRC | 04:18 | |
| *** inara has joined #openstack-neutron | 04:22 | |
| *** jecarey has quit IRC | 04:26 | |
| openstackgerrit | Micheal Thompson proposed a change to openstack/neutron: Added A10 LBaaS Driver to branch. https://review.openstack.org/62464 | 04:28 |
| *** yfried has joined #openstack-neutron | 04:34 | |
| *** layer427expert has quit IRC | 04:54 | |
| *** bashok has joined #openstack-neutron | 04:56 | |
| *** krast_ has joined #openstack-neutron | 05:06 | |
| *** krast has quit IRC | 05:07 | |
| *** csd_ has quit IRC | 05:16 | |
| *** jecarey has joined #openstack-neutron | 05:16 | |
| *** csd_ has joined #openstack-neutron | 05:17 | |
| *** yfried has quit IRC | 05:18 | |
| *** garyk has quit IRC | 05:19 | |
| *** rohit404 has quit IRC | 05:20 | |
| *** rohit404 has joined #openstack-neutron | 05:22 | |
| *** gdubreui has quit IRC | 05:28 | |
| openstackgerrit | Dazhao Yu proposed a change to openstack/neutron: Calculate stateless IPv6 address https://review.openstack.org/56184 | 05:33 |
| *** rpodolyaka1 has joined #openstack-neutron | 05:34 | |
| *** layer427expert has joined #openstack-neutron | 05:35 | |
| *** rpodolyaka1 has left #openstack-neutron | 05:43 | |
| openstackgerrit | Micheal Thompson proposed a change to openstack/neutron: Fixed PEP8 issues hopefully. Corrected Cookie issue https://review.openstack.org/62464 | 05:50 |
| *** nati_ueno has joined #openstack-neutron | 05:51 | |
| rohit404 | anyone here know what's the use case of multi-provider network extension ? | 05:52 |
| *** jecarey has quit IRC | 05:53 | |
| marun | salv-orlando: ping | 05:56 |
| openstackgerrit | Berezovsky Irena proposed a change to openstack/neutron: Change default eswitchd port to avoid conflict https://review.openstack.org/60203 | 05:56 |
| *** csd_ has quit IRC | 05:57 | |
| *** csd_ has joined #openstack-neutron | 05:58 | |
| *** csd_ has quit IRC | 06:00 | |
| *** csd_ has joined #openstack-neutron | 06:01 | |
| *** ashaikh has quit IRC | 06:02 | |
| *** suresh12 has joined #openstack-neutron | 06:04 | |
| *** banix has quit IRC | 06:13 | |
| *** vkozhukalov has quit IRC | 06:15 | |
| *** vkozhukalov has joined #openstack-neutron | 06:15 | |
| *** csd_ has quit IRC | 06:22 | |
| *** csd_ has joined #openstack-neutron | 06:23 | |
| *** Jianyong has joined #openstack-neutron | 06:24 | |
| *** yfried has joined #openstack-neutron | 06:32 | |
| openstackgerrit | Dazhao Yu proposed a change to openstack/neutron: Calculate stateless IPv6 address https://review.openstack.org/56184 | 06:34 |
| *** alex_klimov has joined #openstack-neutron | 06:36 | |
| openstackgerrit | Jenkins proposed a change to openstack/neutron: Imported Translations from Transifex https://review.openstack.org/62812 | 06:37 |
| *** csd_ has quit IRC | 06:37 | |
| *** csd_ has joined #openstack-neutron | 06:39 | |
| *** dkehn has quit IRC | 06:40 | |
| *** rwsu_ has quit IRC | 06:41 | |
| *** dkehn has joined #openstack-neutron | 06:42 | |
| *** csd__ has joined #openstack-neutron | 06:42 | |
| *** csd_ has quit IRC | 06:44 | |
| *** amritanshu_RnD has joined #openstack-neutron | 06:49 | |
| *** vkozhukalov has quit IRC | 06:52 | |
| *** dims has quit IRC | 06:55 | |
| *** layer427expert has quit IRC | 06:59 | |
| *** garyk has joined #openstack-neutron | 07:07 | |
| *** layer427expert has joined #openstack-neutron | 07:09 | |
| *** ljjjustin has quit IRC | 07:09 | |
| *** ljjjustin has joined #openstack-neutron | 07:11 | |
| *** wenjianhn has joined #openstack-neutron | 07:23 | |
| openstackgerrit | Oleg Bondarev proposed a change to openstack/neutron: LBaaS: agent monitoring and instance rescheduling https://review.openstack.org/59743 | 07:27 |
| *** ljjjustin has quit IRC | 07:34 | |
| yfried | beagles: here? | 07:39 |
| *** dims_ has joined #openstack-neutron | 07:43 | |
| openstackgerrit | Micheal Thompson proposed a change to openstack/neutron: Fixed PEP8 issues hopefully. Corrected Cookie issue https://review.openstack.org/62464 | 07:44 |
| *** rohit404 has quit IRC | 07:48 | |
| *** rohit404 has joined #openstack-neutron | 07:49 | |
| *** suresh12 has quit IRC | 07:53 | |
| openstackgerrit | Xiang Hui proposed a change to openstack/neutron: Raise the RuntimeError encountered when adding port https://review.openstack.org/58954 | 08:02 |
| *** layer427expert has quit IRC | 08:02 | |
| *** jlibosva has joined #openstack-neutron | 08:06 | |
| *** iwamoto has quit IRC | 08:13 | |
| *** fouxm has joined #openstack-neutron | 08:14 | |
| *** fouxm has quit IRC | 08:19 | |
| *** fouxm has joined #openstack-neutron | 08:20 | |
| *** SushilKM has joined #openstack-neutron | 08:22 | |
| openstackgerrit | Tomoko Inoue proposed a change to openstack/neutron: Can't create a firewall for admin tenant when at least one other tenant has a firewall. https://review.openstack.org/62600 | 08:22 |
| salv-orlando | marun: still around? | 08:23 |
| marun | salv-orlando: yes | 08:23 |
| marun | salv-orlando: your ovs l2 patch works very well | 08:24 |
| marun | salv-orlando: i'm not sure there's as much performance improvement to be had from the dhcp agent, i'm afraid. | 08:25 |
| marun | salv-orlando: it's looking like the main benefit of a refactor will be increasing resiliency to failure | 08:25 |
| marun | salv-orlando: in your testing of the l2 agent, how do you usually test? with real vm's (how many) or fakevirt? | 08:26 |
| salv-orlando | l2 agent needs real vms | 08:26 |
| salv-orlando | marun: if the tap is not created (as the fake_virt driver does), nothing happens on the l2 agent | 08:27 |
| openstackgerrit | Oleg Bondarev proposed a change to openstack/neutron: Mock looping_call in metadata agent tests https://review.openstack.org/62822 | 08:27 |
| marun | salv-orlando: shame we don't have a way to fake it. | 08:28 |
| marun | salv-orlando: how many vm's can you boot simultaneously? | 08:28 |
| openstackgerrit | Tomoko Inoue proposed a change to openstack/neutron: Can't create a firewall for admin tenant when at least one other tenant has a firewall. https://review.openstack.org/62600 | 08:30 |
| *** vkozhukalov has joined #openstack-neutron | 08:31 | |
| salv-orlando | marun: my limit is the size of my testbed; the one I am using currently does not allow me to go beyond 10. | 08:35 |
| salv-orlando | but I think if I have time over the weekend I can add a few compute nodes and get up to 40-50 | 08:35 |
| *** ygbo has joined #openstack-neutron | 08:35 | |
| marun | salv-orlando: I can boot 40 no problem. Around 50 it becomes problematic - nova starts reporting connection errors in contacting neutron | 08:37 |
| *** roeyc has joined #openstack-neutron | 08:37 | |
| marun | salv-orlando: but at least now all the vm's that are marked 'active' are actually active | 08:38 |
| salv-orlando | are you enabling multiple api workers? | 08:39 |
| marun | salv-orlando: yes, 4 | 08:39 |
| marun | salv-orlando: ah, nuts. i'm wrong :( | 08:40 |
| marun | salv-orlando: i thought i'd set it in local.conf but I guess not. | 08:40 |
| salv-orlando | marun: nova does network configuration asynchronously, so it bombs neutron with requests; I wondeer why the large_ops jobs with the fake driver succeeds for 150 whereas the same operation with a real driver fails for 50 vms | 08:41 |
| salv-orlando | nova/neutron calls are the same regardless of the virt driver. The extra load might be because of l2 agent callbacks calling into the server? | 08:41 |
| marun | salv-orlando: it may be. i was seeing elevated load on the l2 agent as the number of vm's rose | 08:43 |
| salv-orlando | marun: the load gets big for processing the taps, mostly for doing iptables-{save|restore}, ovs-vsctl and ovs-ofctl. But the code in trunk currently triggers a good deal of callbacks from the agent to the server, which also increase, unnecessarily the load on the server. For instance refresh_firewall is always invoked when handling port_update and it always gets all the security group rules which apply to a given list of devices | 08:46 |
| marun | salv-orlando: ah, that sounds like a good target for further optimization then | 08:48 |
| salv-orlando | and if you look at trunk you'll find out the l2 agent is now pretty much notified on almost every change. This is a bug, I think, and I have a patch for it. The patch has 2 +2 already, but since it failed during verify | 08:48 |
| *** amuller has joined #openstack-neutron | 08:48 | |
| salv-orlando | I am not running more checks to ensure it is not the patch adding more bugs | 08:48 |
| salv-orlando | marun: this patch -> https://review.openstack.org/#/c/58860/ | 08:48 |
| openstackgerrit | Sylvain Afchain proposed a change to openstack/neutron: Add param and iptables rules to protect dnsmasq ports https://review.openstack.org/61994 | 08:54 |
| kruskakli | anteaya: I think I figured out how to run my Jenkins test now, I need to export the GERRIT_REFSPEC in my execut-shell command, then do, typically: git fetch https://review.openstack.org/openstack/neutron $GERRIT_REFSPEC && git checkout FETCH_HEAD before staring my unit tests | 08:54 |
| *** jlibosva has quit IRC | 09:10 | |
| *** ihrachyshka has quit IRC | 09:10 | |
| *** markmcclain has quit IRC | 09:10 | |
| *** salv-orlando has quit IRC | 09:10 | |
| *** unicell has quit IRC | 09:10 | |
| *** openstackgerrit has quit IRC | 09:11 | |
| *** enikanorov__ has quit IRC | 09:11 | |
| *** vkozhukalov has quit IRC | 09:11 | |
| *** alex_klimov has quit IRC | 09:11 | |
| *** Jianyong has quit IRC | 09:11 | |
| *** bashok has quit IRC | 09:11 | |
| *** briancline has quit IRC | 09:11 | |
| *** otherwiseguy has quit IRC | 09:11 | |
| *** zigo has quit IRC | 09:11 | |
| *** morganfainberg has quit IRC | 09:11 | |
| *** rohit404 has quit IRC | 09:11 | |
| *** wenjianhn has quit IRC | 09:11 | |
| *** garyk has quit IRC | 09:11 | |
| *** nati_ueno has quit IRC | 09:11 | |
| *** krast_ has quit IRC | 09:11 | |
| *** HenryG has quit IRC | 09:11 | |
| *** marios has quit IRC | 09:11 | |
| *** dims_ has quit IRC | 09:11 | |
| *** csd__ has quit IRC | 09:11 | |
| *** dkehn has quit IRC | 09:11 | |
| *** yfried has quit IRC | 09:11 | |
| *** dzyu has quit IRC | 09:11 | |
| *** yfujioka has quit IRC | 09:11 | |
| *** AndreyGrebenniko has quit IRC | 09:11 | |
| *** pashi_ has quit IRC | 09:11 | |
| *** jianingy_afk has quit IRC | 09:11 | |
| *** thansen has quit IRC | 09:11 | |
| *** _cerberus_ has quit IRC | 09:11 | |
| *** sdague has quit IRC | 09:11 | |
| *** ivoks has quit IRC | 09:11 | |
| *** metral has quit IRC | 09:11 | |
| *** inara has quit IRC | 09:11 | |
| *** Qlawy has quit IRC | 09:11 | |
| *** pasquier-s has quit IRC | 09:11 | |
| *** terry_howe has quit IRC | 09:11 | |
| *** doude_ has quit IRC | 09:11 | |
| *** jog0 has quit IRC | 09:11 | |
| *** MichielHN has quit IRC | 09:11 | |
| *** mrsnivvel has quit IRC | 09:11 | |
| *** obondarev has quit IRC | 09:11 | |
| *** harlowja_away has quit IRC | 09:11 | |
| *** lifeless has quit IRC | 09:11 | |
| *** kashyap has quit IRC | 09:11 | |
| *** roeyc has quit IRC | 09:11 | |
| *** ywu has quit IRC | 09:11 | |
| *** dsockwell has quit IRC | 09:11 | |
| *** skraynev has quit IRC | 09:11 | |
| *** Mierdin has quit IRC | 09:11 | |
| *** chandankumar has quit IRC | 09:11 | |
| *** dave_tucker_zzz has quit IRC | 09:11 | |
| *** rdo has quit IRC | 09:11 | |
| *** fcoj has quit IRC | 09:11 | |
| *** rkukura has quit IRC | 09:11 | |
| *** benner has quit IRC | 09:11 | |
| *** ekarlso has quit IRC | 09:11 | |
| *** amritanshu_RnD has quit IRC | 09:11 | |
| *** networkstatic has quit IRC | 09:11 | |
| *** yamahata has quit IRC | 09:11 | |
| *** kruskakli has quit IRC | 09:11 | |
| *** mattymo has quit IRC | 09:11 | |
| *** larsks has quit IRC | 09:11 | |
| *** sgran has quit IRC | 09:11 | |
| *** fouxm has quit IRC | 09:11 | |
| *** mordred has quit IRC | 09:11 | |
| *** pbeskow_ has quit IRC | 09:11 | |
| *** sc68cal has quit IRC | 09:11 | |
| *** pvo has quit IRC | 09:11 | |
| *** ajo has quit IRC | 09:11 | |
| *** EmilienM has quit IRC | 09:11 | |
| *** russellb has quit IRC | 09:11 | |
| *** yongli has quit IRC | 09:11 | |
| *** akamyshnikova_ has quit IRC | 09:11 | |
| *** lari__ has quit IRC | 09:11 | |
| *** Makdaam has quit IRC | 09:11 | |
| *** aryan has quit IRC | 09:11 | |
| *** cburgess has quit IRC | 09:11 | |
| *** dosaboy has quit IRC | 09:11 | |
| *** christophk has quit IRC | 09:11 | |
| *** zhhuabj has quit IRC | 09:11 | |
| *** angryjesters has quit IRC | 09:11 | |
| *** roaet has quit IRC | 09:11 | |
| *** gizmoguy_ has quit IRC | 09:11 | |
| *** Apsu has quit IRC | 09:11 | |
| *** decede has quit IRC | 09:11 | |
| *** markvoelker has quit IRC | 09:11 | |
| *** ogelbukh has quit IRC | 09:11 | |
| *** annegentle_ has quit IRC | 09:11 | |
| *** enikanorov has quit IRC | 09:11 | |
| *** jhurlbert has quit IRC | 09:11 | |
| *** nijaba has quit IRC | 09:11 | |
| *** amuller has quit IRC | 09:11 | |
| *** ygbo has quit IRC | 09:11 | |
| *** SushilKM has quit IRC | 09:11 | |
| *** SumitNaiksatam has quit IRC | 09:11 | |
| *** ijw has quit IRC | 09:11 | |
| *** mtreinish has quit IRC | 09:11 | |
| *** yamahata__ has quit IRC | 09:11 | |
| *** marun has quit IRC | 09:11 | |
| *** JoeHazzers has quit IRC | 09:11 | |
| *** asadoughi has quit IRC | 09:11 | |
| *** anteaya has quit IRC | 09:11 | |
| *** lowkey has quit IRC | 09:11 | |
| *** terry_howe has joined #openstack-neutron | 09:13 | |
| *** jpich has joined #openstack-neutron | 09:13 | |
| *** majopela has joined #openstack-neutron | 09:13 | |
| *** amuller has joined #openstack-neutron | 09:13 | |
| *** roeyc has joined #openstack-neutron | 09:13 | |
| *** ygbo has joined #openstack-neutron | 09:13 | |
| *** vkozhukalov has joined #openstack-neutron | 09:13 | |
| *** SushilKM has joined #openstack-neutron | 09:13 | |
| *** fouxm has joined #openstack-neutron | 09:13 | |
| *** jlibosva has joined #openstack-neutron | 09:13 | |
| *** rohit404 has joined #openstack-neutron | 09:13 | |
| *** dims_ has joined #openstack-neutron | 09:13 | |
| *** wenjianhn has joined #openstack-neutron | 09:13 | |
| *** garyk has joined #openstack-neutron | 09:13 | |
| *** amritanshu_RnD has joined #openstack-neutron | 09:13 | |
| *** csd__ has joined #openstack-neutron | 09:13 | |
| *** dkehn has joined #openstack-neutron | 09:13 | |
| *** alex_klimov has joined #openstack-neutron | 09:13 | |
| *** yfried has joined #openstack-neutron | 09:13 | |
| *** Jianyong has joined #openstack-neutron | 09:13 | |
| *** nati_ueno has joined #openstack-neutron | 09:13 | |
| *** krast_ has joined #openstack-neutron | 09:13 | |
| *** bashok has joined #openstack-neutron | 09:13 | |
| *** inara has joined #openstack-neutron | 09:13 | |
| *** chandankumar has joined #openstack-neutron | 09:13 | |
| *** dave_tucker_zzz has joined #openstack-neutron | 09:13 | |
| *** dzyu has joined #openstack-neutron | 09:13 | |
| *** yamahata has joined #openstack-neutron | 09:13 | |
| *** SumitNaiksatam has joined #openstack-neutron | 09:13 | |
| *** yfujioka has joined #openstack-neutron | 09:13 | |
| *** markmcclain has joined #openstack-neutron | 09:13 | |
| *** briancline has joined #openstack-neutron | 09:13 | |
| *** salv-orlando has joined #openstack-neutron | 09:13 | |
| *** ihrachyshka has joined #openstack-neutron | 09:13 | |
| *** ijw has joined #openstack-neutron | 09:13 | |
| *** Mierdin has joined #openstack-neutron | 09:13 | |
| *** HenryG has joined #openstack-neutron | 09:13 | |
| *** otherwiseguy has joined #openstack-neutron | 09:13 | |
| *** mtreinish has joined #openstack-neutron | 09:13 | |
| *** yamahata__ has joined #openstack-neutron | 09:13 | |
| *** unicell has joined #openstack-neutron | 09:13 | |
| *** marun has joined #openstack-neutron | 09:13 | |
| *** marios has joined #openstack-neutron | 09:13 | |
| *** zhhuabj has joined #openstack-neutron | 09:13 | |
| *** openstackgerrit has joined #openstack-neutron | 09:13 | |
| *** enikanorov__ has joined #openstack-neutron | 09:13 | |
| *** thansen has joined #openstack-neutron | 09:13 | |
| *** mordred has joined #openstack-neutron | 09:13 | |
| *** gizmoguy_ has joined #openstack-neutron | 09:13 | |
| *** jianingy_afk has joined #openstack-neutron | 09:13 | |
| *** metral has joined #openstack-neutron | 09:13 | |
| *** ivoks has joined #openstack-neutron | 09:13 | |
| *** pashi_ has joined #openstack-neutron | 09:13 | |
| *** AndreyGrebenniko has joined #openstack-neutron | 09:13 | |
| *** sdague has joined #openstack-neutron | 09:13 | |
| *** zigo has joined #openstack-neutron | 09:13 | |
| *** morganfainberg has joined #openstack-neutron | 09:13 | |
| *** _cerberus_ has joined #openstack-neutron | 09:13 | |
| *** rdo has joined #openstack-neutron | 09:13 | |
| *** Qlawy has joined #openstack-neutron | 09:13 | |
| *** yongli has joined #openstack-neutron | 09:13 | |
| *** pasquier-s has joined #openstack-neutron | 09:13 | |
| *** akamyshnikova_ has joined #openstack-neutron | 09:13 | |
| *** doude_ has joined #openstack-neutron | 09:13 | |
| *** Apsu has joined #openstack-neutron | 09:13 | |
| *** annegentle_ has joined #openstack-neutron | 09:13 | |
| *** skraynev has joined #openstack-neutron | 09:13 | |
| *** decede has joined #openstack-neutron | 09:13 | |
| *** markvoelker has joined #openstack-neutron | 09:13 | |
| *** mattymo has joined #openstack-neutron | 09:13 | |
| *** jog0 has joined #openstack-neutron | 09:13 | |
| *** MichielHN has joined #openstack-neutron | 09:13 | |
| *** dsockwell has joined #openstack-neutron | 09:13 | |
| *** mrsnivvel has joined #openstack-neutron | 09:13 | |
| *** ogelbukh has joined #openstack-neutron | 09:13 | |
| *** lowkey has joined #openstack-neutron | 09:13 | |
| *** obondarev has joined #openstack-neutron | 09:13 | |
| *** harlowja_away has joined #openstack-neutron | 09:13 | |
| *** enikanorov has joined #openstack-neutron | 09:13 | |
| *** lifeless has joined #openstack-neutron | 09:13 | |
| *** jhurlbert has joined #openstack-neutron | 09:13 | |
| *** kashyap has joined #openstack-neutron | 09:13 | |
| *** pbeskow_ has joined #openstack-neutron | 09:13 | |
| *** angryjesters has joined #openstack-neutron | 09:13 | |
| *** fcoj has joined #openstack-neutron | 09:13 | |
| *** lari__ has joined #openstack-neutron | 09:13 | |
| *** rkukura has joined #openstack-neutron | 09:13 | |
| *** dosaboy has joined #openstack-neutron | 09:13 | |
| *** kruskakli has joined #openstack-neutron | 09:13 | |
| *** benner has joined #openstack-neutron | 09:13 | |
| *** ekarlso has joined #openstack-neutron | 09:13 | |
| *** ywu has joined #openstack-neutron | 09:13 | |
| *** JoeHazzers has joined #openstack-neutron | 09:13 | |
| *** sc68cal has joined #openstack-neutron | 09:13 | |
| *** pvo has joined #openstack-neutron | 09:13 | |
| *** Makdaam has joined #openstack-neutron | 09:13 | |
| *** aryan has joined #openstack-neutron | 09:13 | |
| *** asadoughi has joined #openstack-neutron | 09:13 | |
| *** roaet has joined #openstack-neutron | 09:13 | |
| *** larsks has joined #openstack-neutron | 09:13 | |
| *** sgran has joined #openstack-neutron | 09:13 | |
| *** cburgess has joined #openstack-neutron | 09:13 | |
| *** nijaba has joined #openstack-neutron | 09:13 | |
| *** ajo has joined #openstack-neutron | 09:13 | |
| *** EmilienM has joined #openstack-neutron | 09:13 | |
| *** russellb has joined #openstack-neutron | 09:13 | |
| *** christophk has joined #openstack-neutron | 09:13 | |
| *** anteaya has joined #openstack-neutron | 09:13 | |
| *** Mierdin has quit IRC | 09:13 | |
| marun | salv-orlando: i'm still seeing pretty high cpu usage on the agent, even with the patch | 09:13 |
| salv-orlando | marun: that patch is not final - look at the todos | 09:14 |
| openstackgerrit | Hyunsun Moon proposed a change to openstack/neutron: Fix port binding host is set None when port update https://review.openstack.org/62829 | 09:14 |
| salv-orlando | marun: I am talking about the l2 agent patch | 09:14 |
| marun | salv-orlando: ah, right. | 09:14 |
| EmilienM | i'm currently working on puppet-neutron, and try to understand the changes made over the last days about security group configuration. Is the firewall_driver should be configured on both plugin & agent ? | 09:14 |
| EmilienM | my concern is https://review.openstack.org/#/c/62535 - where a guy tries to configure firewall_driver on ml2.ini, but I thought this flag was only for the agent (OVS or LB) | 09:15 |
| salv-orlando | marun: basically what happens now is that even if update a port changing its name, it reprocesses the vif (unnecessarily) and reapplies filters (even more unnecessarily). When you boot a vm there are both port-create and port-update calls from nova (that's another thing arosen is optimising), thus the high load. | 09:16 |
| *** jlibosva has quit IRC | 09:16 | |
| *** ihrachyshka has quit IRC | 09:16 | |
| *** markmcclain has quit IRC | 09:16 | |
| *** salv-orlando has quit IRC | 09:16 | |
| *** unicell has quit IRC | 09:16 | |
| *** openstackgerrit has quit IRC | 09:16 | |
| *** enikanorov__ has quit IRC | 09:16 | |
| *** vkozhukalov has quit IRC | 09:16 | |
| *** alex_klimov has quit IRC | 09:16 | |
| *** Jianyong has quit IRC | 09:16 | |
| *** bashok has quit IRC | 09:16 | |
| *** briancline has quit IRC | 09:16 | |
| *** otherwiseguy has quit IRC | 09:16 | |
| *** zigo has quit IRC | 09:16 | |
| *** morganfainberg has quit IRC | 09:16 | |
| *** terry_howe has quit IRC | 09:16 | |
| *** rohit404 has quit IRC | 09:16 | |
| *** wenjianhn has quit IRC | 09:16 | |
| *** garyk has quit IRC | 09:16 | |
| *** nati_ueno has quit IRC | 09:16 | |
| *** krast_ has quit IRC | 09:16 | |
| *** HenryG has quit IRC | 09:16 | |
| *** marios has quit IRC | 09:16 | |
| *** dims_ has quit IRC | 09:16 | |
| *** csd__ has quit IRC | 09:16 | |
| *** dkehn has quit IRC | 09:16 | |
| *** yfried has quit IRC | 09:16 | |
| *** dzyu has quit IRC | 09:16 | |
| *** yfujioka has quit IRC | 09:16 | |
| *** AndreyGrebenniko has quit IRC | 09:16 | |
| *** pashi_ has quit IRC | 09:16 | |
| *** jianingy_afk has quit IRC | 09:16 | |
| *** thansen has quit IRC | 09:16 | |
| *** _cerberus_ has quit IRC | 09:16 | |
| *** sdague has quit IRC | 09:16 | |
| *** ivoks has quit IRC | 09:16 | |
| *** metral has quit IRC | 09:16 | |
| *** inara has quit IRC | 09:16 | |
| *** Qlawy has quit IRC | 09:16 | |
| *** pasquier-s has quit IRC | 09:16 | |
| *** doude_ has quit IRC | 09:16 | |
| *** jog0 has quit IRC | 09:16 | |
| *** MichielHN has quit IRC | 09:16 | |
| *** mrsnivvel has quit IRC | 09:16 | |
| *** obondarev has quit IRC | 09:16 | |
| *** harlowja_away has quit IRC | 09:16 | |
| *** lifeless has quit IRC | 09:16 | |
| *** kashyap has quit IRC | 09:16 | |
| *** roeyc has quit IRC | 09:16 | |
| *** ywu has quit IRC | 09:16 | |
| *** majopela has quit IRC | 09:16 | |
| *** dsockwell has quit IRC | 09:16 | |
| *** skraynev has quit IRC | 09:16 | |
| *** chandankumar has quit IRC | 09:16 | |
| *** dave_tucker_zzz has quit IRC | 09:16 | |
| *** rdo has quit IRC | 09:16 | |
| *** fcoj has quit IRC | 09:16 | |
| *** rkukura has quit IRC | 09:16 | |
| *** benner has quit IRC | 09:16 | |
| *** ekarlso has quit IRC | 09:16 | |
| *** jpich has quit IRC | 09:16 | |
| *** amritanshu_RnD has quit IRC | 09:16 | |
| *** yamahata has quit IRC | 09:16 | |
| *** kruskakli has quit IRC | 09:16 | |
| *** mattymo has quit IRC | 09:16 | |
| *** larsks has quit IRC | 09:16 | |
| *** sgran has quit IRC | 09:16 | |
| *** fouxm has quit IRC | 09:16 | |
| *** mordred has quit IRC | 09:16 | |
| *** pbeskow_ has quit IRC | 09:16 | |
| *** sc68cal has quit IRC | 09:16 | |
| *** pvo has quit IRC | 09:16 | |
| *** ajo has quit IRC | 09:16 | |
| *** EmilienM has quit IRC | 09:16 | |
| *** russellb has quit IRC | 09:16 | |
| *** yongli has quit IRC | 09:16 | |
| *** akamyshnikova_ has quit IRC | 09:16 | |
| *** lari__ has quit IRC | 09:16 | |
| *** Makdaam has quit IRC | 09:16 | |
| *** aryan has quit IRC | 09:16 | |
| *** cburgess has quit IRC | 09:16 | |
| *** dosaboy has quit IRC | 09:16 | |
| *** christophk has quit IRC | 09:16 | |
| *** zhhuabj has quit IRC | 09:16 | |
| *** angryjesters has quit IRC | 09:16 | |
| *** roaet has quit IRC | 09:16 | |
| *** gizmoguy_ has quit IRC | 09:16 | |
| *** Apsu has quit IRC | 09:16 | |
| *** decede has quit IRC | 09:16 | |
| *** markvoelker has quit IRC | 09:16 | |
| *** ogelbukh has quit IRC | 09:16 | |
| *** annegentle_ has quit IRC | 09:16 | |
| *** enikanorov has quit IRC | 09:16 | |
| *** jhurlbert has quit IRC | 09:16 | |
| *** nijaba has quit IRC | 09:16 | |
| *** amuller has quit IRC | 09:16 | |
| *** ygbo has quit IRC | 09:16 | |
| *** SushilKM has quit IRC | 09:16 | |
| *** SumitNaiksatam has quit IRC | 09:16 | |
| *** ijw has quit IRC | 09:16 | |
| *** mtreinish has quit IRC | 09:17 | |
| *** yamahata__ has quit IRC | 09:17 | |
| *** marun has quit IRC | 09:17 | |
| *** JoeHazzers has quit IRC | 09:17 | |
| *** asadoughi has quit IRC | 09:17 | |
| *** anteaya has quit IRC | 09:17 | |
| *** lowkey has quit IRC | 09:17 | |
| *** matrohon has quit IRC | 09:17 | |
| *** alex_klimov has joined #openstack-neutron | 09:22 | |
| *** rohit404 has joined #openstack-neutron | 09:22 | |
| *** matrohon has joined #openstack-neutron | 09:22 | |
| *** terry_howe has joined #openstack-neutron | 09:22 | |
| *** jpich has joined #openstack-neutron | 09:22 | |
| *** majopela has joined #openstack-neutron | 09:22 | |
| *** amuller has joined #openstack-neutron | 09:22 | |
| *** roeyc has joined #openstack-neutron | 09:22 | |
| *** ygbo has joined #openstack-neutron | 09:22 | |
| *** vkozhukalov has joined #openstack-neutron | 09:22 | |
| *** SushilKM has joined #openstack-neutron | 09:22 | |
| *** fouxm has joined #openstack-neutron | 09:22 | |
| *** jlibosva has joined #openstack-neutron | 09:22 | |
| *** dims_ has joined #openstack-neutron | 09:22 | |
| *** wenjianhn has joined #openstack-neutron | 09:22 | |
| *** garyk has joined #openstack-neutron | 09:22 | |
| *** amritanshu_RnD has joined #openstack-neutron | 09:22 | |
| *** csd__ has joined #openstack-neutron | 09:22 | |
| *** dkehn has joined #openstack-neutron | 09:22 | |
| *** yfried has joined #openstack-neutron | 09:22 | |
| *** Jianyong has joined #openstack-neutron | 09:22 | |
| *** nati_ueno has joined #openstack-neutron | 09:22 | |
| *** krast_ has joined #openstack-neutron | 09:22 | |
| *** bashok has joined #openstack-neutron | 09:22 | |
| *** inara has joined #openstack-neutron | 09:22 | |
| *** chandankumar has joined #openstack-neutron | 09:22 | |
| *** dave_tucker_zzz has joined #openstack-neutron | 09:22 | |
| *** dzyu has joined #openstack-neutron | 09:22 | |
| *** yamahata has joined #openstack-neutron | 09:22 | |
| *** SumitNaiksatam has joined #openstack-neutron | 09:22 | |
| *** yfujioka has joined #openstack-neutron | 09:22 | |
| *** markmcclain has joined #openstack-neutron | 09:22 | |
| *** briancline has joined #openstack-neutron | 09:22 | |
| *** salv-orlando has joined #openstack-neutron | 09:22 | |
| *** ihrachyshka has joined #openstack-neutron | 09:22 | |
| *** ijw has joined #openstack-neutron | 09:22 | |
| *** HenryG has joined #openstack-neutron | 09:22 | |
| *** otherwiseguy has joined #openstack-neutron | 09:22 | |
| *** mtreinish has joined #openstack-neutron | 09:22 | |
| *** yamahata__ has joined #openstack-neutron | 09:22 | |
| *** unicell has joined #openstack-neutron | 09:22 | |
| *** marun has joined #openstack-neutron | 09:22 | |
| *** marios has joined #openstack-neutron | 09:22 | |
| *** zhhuabj has joined #openstack-neutron | 09:22 | |
| *** openstackgerrit has joined #openstack-neutron | 09:22 | |
| *** enikanorov__ has joined #openstack-neutron | 09:22 | |
| *** thansen has joined #openstack-neutron | 09:22 | |
| *** mordred has joined #openstack-neutron | 09:22 | |
| *** gizmoguy_ has joined #openstack-neutron | 09:22 | |
| *** jianingy_afk has joined #openstack-neutron | 09:22 | |
| *** metral has joined #openstack-neutron | 09:22 | |
| *** ivoks has joined #openstack-neutron | 09:22 | |
| *** pashi_ has joined #openstack-neutron | 09:22 | |
| *** AndreyGrebenniko has joined #openstack-neutron | 09:22 | |
| *** sdague has joined #openstack-neutron | 09:22 | |
| *** zigo has joined #openstack-neutron | 09:22 | |
| *** morganfainberg has joined #openstack-neutron | 09:22 | |
| *** _cerberus_ has joined #openstack-neutron | 09:22 | |
| *** rdo has joined #openstack-neutron | 09:22 | |
| *** Qlawy has joined #openstack-neutron | 09:22 | |
| *** yongli has joined #openstack-neutron | 09:22 | |
| *** pasquier-s has joined #openstack-neutron | 09:22 | |
| *** akamyshnikova_ has joined #openstack-neutron | 09:22 | |
| *** doude_ has joined #openstack-neutron | 09:22 | |
| *** Apsu has joined #openstack-neutron | 09:22 | |
| *** annegentle_ has joined #openstack-neutron | 09:22 | |
| *** skraynev has joined #openstack-neutron | 09:22 | |
| *** decede has joined #openstack-neutron | 09:22 | |
| *** markvoelker has joined #openstack-neutron | 09:22 | |
| *** mattymo has joined #openstack-neutron | 09:22 | |
| *** jog0 has joined #openstack-neutron | 09:22 | |
| *** MichielHN has joined #openstack-neutron | 09:22 | |
| *** dsockwell has joined #openstack-neutron | 09:22 | |
| *** mrsnivvel has joined #openstack-neutron | 09:22 | |
| *** ogelbukh has joined #openstack-neutron | 09:22 | |
| *** lowkey has joined #openstack-neutron | 09:22 | |
| *** obondarev has joined #openstack-neutron | 09:22 | |
| *** harlowja_away has joined #openstack-neutron | 09:22 | |
| *** enikanorov has joined #openstack-neutron | 09:22 | |
| *** lifeless has joined #openstack-neutron | 09:22 | |
| *** jhurlbert has joined #openstack-neutron | 09:22 | |
| *** kashyap has joined #openstack-neutron | 09:22 | |
| *** pbeskow_ has joined #openstack-neutron | 09:22 | |
| *** angryjesters has joined #openstack-neutron | 09:22 | |
| *** fcoj has joined #openstack-neutron | 09:22 | |
| *** lari__ has joined #openstack-neutron | 09:22 | |
| *** rkukura has joined #openstack-neutron | 09:22 | |
| *** dosaboy has joined #openstack-neutron | 09:22 | |
| *** kruskakli has joined #openstack-neutron | 09:22 | |
| *** benner has joined #openstack-neutron | 09:22 | |
| *** ekarlso has joined #openstack-neutron | 09:22 | |
| *** ywu has joined #openstack-neutron | 09:22 | |
| *** JoeHazzers has joined #openstack-neutron | 09:22 | |
| *** sc68cal has joined #openstack-neutron | 09:22 | |
| *** pvo has joined #openstack-neutron | 09:22 | |
| *** Makdaam has joined #openstack-neutron | 09:22 | |
| *** aryan has joined #openstack-neutron | 09:22 | |
| *** asadoughi has joined #openstack-neutron | 09:22 | |
| *** roaet has joined #openstack-neutron | 09:22 | |
| *** larsks has joined #openstack-neutron | 09:22 | |
| *** sgran has joined #openstack-neutron | 09:22 | |
| *** cburgess has joined #openstack-neutron | 09:22 | |
| *** nijaba has joined #openstack-neutron | 09:22 | |
| *** ajo has joined #openstack-neutron | 09:22 | |
| *** EmilienM has joined #openstack-neutron | 09:22 | |
| *** russellb has joined #openstack-neutron | 09:22 | |
| *** christophk has joined #openstack-neutron | 09:22 | |
| *** anteaya has joined #openstack-neutron | 09:22 | |
| salv-orlando | marun: ovs does not know anything about IP info, as the IP is actually in the guests. I was thinking of adding some intelligence to port_updated notifications | 09:22 |
| marun | salv-orlando: ah, right. | 09:22 |
| EmilienM | matrohon: the guy was not able to run SG using ML2 without the trick | 09:22 |
| marun | salv-orlando: yes, I vote for better notifications. | 09:22 |
| salv-orlando | now it dumps all the details, and that's not a great idea. since it is the same problem that caused the l3 agent to go crazy during havana | 09:22 |
| marun | salv-orlando: fyi, with your patches I can now boot 40 vm's at a time without error. 50 errors out like crazy, though. | 09:22 |
| salv-orlando | Why don't you post your errors somewhere? | 09:22 |
| salv-orlando | marun: to cut a long story short, I was thinking of augmenting the notification with a few flags that tell what kind of change we are looking at. Something like (port_id, admin_state_change, mac_ip_config_change) | 09:23 |
| marun | salv-orlando: that would only seem to make sense. if we have the info the agent will need, why not send it directly? | 09:24 |
| openstackgerrit | mouad benchchaoui proposed a change to openstack/neutron: Make the metadata namespace proxy transparent https://review.openstack.org/28137 | 09:24 |
| marun | salv-orlando: ack, my errors are again my stupidity. quotas too low and duplicate ports being created per vm | 09:25 |
| salv-orlando | sending the actual info, like the new ips, is actually dangerous; this caused us troubles with the l3 agent; nati-ueno spotted indeed an issue were there where two changes to the same floating IP in the API (say X and Y, with X -> Y) and then the l3 agent applied Y -> X | 09:25 |
| marun | salv-orlando: i remember seeing on the bug about why duplicate ports were being created. that's a nova issue? | 09:25 |
| marun | salv-orlando: so ordering of amqp messages isn't reliable by default? | 09:26 |
| salv-orlando | in the queue, yes | 09:26 |
| salv-orlando | once you pick it from the queue and process it with event let not | 09:26 |
| salv-orlando | is there anything reliable about eventlet? | 09:27 |
| marun | salv-orlando: it is if we use it correctly | 09:27 |
| marun | salv-orlando: i think it's just as annoying as real threads in that regard | 09:27 |
| marun | salv-orlando: instead of managing access to shared data, we have to manage execution paths much more carefully | 09:27 |
| marun | salv-orlando: but for example, the preliminary work I've done for the dhcp agent is to simply collect notifications and stuff them into a local queue | 09:28 |
| *** zoresvit has joined #openstack-neutron | 09:28 | |
| marun | salv-orlando: probably a better option is just dispensing with the lame oslo rpc dispatch mechanism entirely, but i wanted something minimally invasive for now | 09:28 |
| marun | salv-orlando: once things are in a local queue, we can process the queue with only a single greenthread and be guaranteed ordering consistency | 09:29 |
| openstackgerrit | Simon Pasquier proposed a change to openstack/neutron: neutron-rootwrap-xen-dom0 handles data from stdin https://review.openstack.org/62346 | 09:29 |
| marun | salv-orlando: and, when a resync is required, coordination becomes easy as opposed to using explicit file-based locks | 09:30 |
| salv-orlando | it's fine to choose something minimally invasive; pushing events in LIFO queue should guarantee ordering consistency as long as we can be reasonably sure they are pushed in the same order as they're picked from the queue. | 09:31 |
| salv-orlando | marun: therefore the rpc callback should probably have a critical section inside? | 09:31 |
| salv-orlando | sorry not the rpc callback, the function called by the rpc message | 09:31 |
| marun | salv-orlando: the rpc callback just pushes to a queue. there should be no blocking io that would relinquish control to another greenthread | 09:32 |
| marun | salv-orlando: and then all the notifications are simply called serially | 09:32 |
| salv-orlando | marun: I got that, but if you have multiple concurrent notifications, the rpc callback also pushes concurrently in the que | 09:32 |
| * salv-orlando queue | 09:33 | |
| marun | salv-orlando: uh, there's no such thing as concurrency in eventlet | 09:33 |
| marun | salv-orlando: and control only ever switches if a greenthread gives up control | 09:33 |
| salv-orlando | marun: or if some routine you've called has been monkey patched by eventlet and yields | 09:34 |
| salv-orlando | but if you just add to a queue, I think it's sage | 09:34 |
| salv-orlando | safe | 09:34 |
| marun | salv-orlando: I agree, I don't mean to argue. You're right that one has to take care to avoid relinquishing control during something important. | 09:34 |
| marun | salv-orlando: I'd be happy to see eventlet put in the bin, frankly. It seriously violates the 'explicit is better than implicit' mantra of python. | 09:35 |
| salv-orlando | marun: I'm never in a mood for an argument :) I'm just saying that eventlet to same extent encourages bad concurrent programming patters | 09:35 |
| salv-orlando | patterns | 09:35 |
| marun | salv-orlando: I couldn't agree more. | 09:35 |
| marun | salv-orlando: at least with real threading or process-based concurrency one knows that one is handling something dangerous, and work carefully to avoid problems. | 09:36 |
| salv-orlando | so ideally I would code as if you're using pthreads anyway, and don't bother that eventlet is not really concurrent | 09:36 |
| marun | salv-orlando: eventlet encourages 'not my problem' | 09:36 |
| openstackgerrit | Sushil Kumar proposed a change to openstack/python-neutronclient: Updates .gitignore https://review.openstack.org/62180 | 09:37 |
| marun | salv-orlando: well, it's been important for me to know that only a single thread executes at a time. | 09:37 |
| marun | salv-orlando: combined with the knowledge of how control is given up, eventlet is no longer a black box for me. | 09:37 |
| marun | salv-orlando: The lack of good synchronization primitives for eventlet has lead to pervasive use of utils.synchronize, which in my book indicates a failure to think the problem through in most cases. | 09:38 |
| *** roeyc has quit IRC | 09:39 | |
| marun | salv-orlando: in real threading, using a synchronization primitive would make sense, but in eventlet I have yet to see a compelling reason. | 09:39 |
| salv-orlando | there might be a reason only when you need to process larger routines which should not be interrupted and make a call which could yield | 09:40 |
| salv-orlando | think about the l2 agent, which might yield in treat_vif_port. I think the following sequence of event is possible: | 09:41 |
| salv-orlando | A) the main rpc_loop finds a port, gets device details from server, and finds out the port if administratively down, then reaches treat_vif_port | 09:42 |
| salv-orlando | B) the main rpc_loop yields to port_update notification for bringing that same port up, and yields to some other thread (maybe the state report thing), before doing treat_vif_port | 09:43 |
| marun | salv-orlando: again, bad design | 09:43 |
| salv-orlando | C) control goes back to the port_update thread which treats the port on the agent assigning the local vlan and bringing it up | 09:43 |
| salv-orlando | marun: gotcha, I got your point. | 09:43 |
| salv-orlando | basically you're saying if you do the design correctly you probably won't need to deal with explicit synchronization | 09:44 |
| marun | salv-orlando: That was my conjecture, but I'm wrong. | 09:44 |
| marun | salv-orlando: :) | 09:45 |
| marun | salv-orlando: I do think it would be better for notifications to be locally queued and processed by a single event loop. | 09:45 |
| salv-orlando | meh: we could talk about this for ages, but perhaps it's better we just look at problems and find the most efficient and reliable way to process them. | 09:45 |
| marun | salv-orlando: actually, i think i still am right. | 09:45 |
| marun | salv-orlando: no, wrong. | 09:46 |
| marun | salv-orlando: as you say, its complicated | 09:46 |
| salv-orlando | marun: I agree with you; I was just musing on the opportunities for using shared resource locking primitives with eventlet as well | 09:46 |
| salv-orlando | a single loop is the most efficient way, agreed. period. | 09:46 |
| openstackgerrit | Zhang Hua proposed a change to openstack/neutron: Clean up ML2 Manager https://review.openstack.org/61351 | 09:46 |
| marun | salv-orlando: Having a single event loop in one greenthread and another greenthread queueing notifications makes sense to me. I guess the key point to understand is that things break down if the eventloop ever spawns new greenthreads that it doesn't wait for. | 09:48 |
| marun | salv-orlando: ala the dhcp agent fix that went in to wait for the pool of threads doing the syncing. without that wait, things just don't work. | 09:48 |
| *** jroovers has joined #openstack-neutron | 09:48 | |
| marun | salv-orlando: so fyi, the breakage I was seeing with 50 vm's was more PEBKAC | 09:49 |
| marun | salv-orlando: port quota was set at 50, and duplicate ports were being created as per that bug | 09:49 |
| marun | salv-orlando: increasing the quota and trying again | 09:49 |
| salv-orlando | marun: so you are still seeing the duplicate ports. I never understood whether that was more a neutron or a nova bug. But if that's still happening, the bug should be reopened. | 09:50 |
| openstackgerrit | Armando Migliaccio proposed a change to openstack/neutron: Add migration support from agent to NSX dhcp/metadata services https://review.openstack.org/51325 | 09:50 |
| marun | salv-orlando: I think it's a nova/neutron integration problem, as per the comment on the bug https://bugs.launchpad.net/neutron/+bug/1160442 | 09:51 |
| openstackgerrit | Yves-Gwenael Bourhis proposed a change to openstack/neutron: Make dnsmasq aware of all names https://review.openstack.org/52930 | 09:51 |
| marun | salv-orlando: Ian Well's comment about it being nova rescheduling makes the most sense | 09:52 |
| marun | salv-orlando: it is re-opened, and jlibosva is assigned to work on it | 09:52 |
| salv-orlando | marun: just seen that | 09:52 |
| marun | salv-orlando: so what to set the status to - triaged? | 09:53 |
| marun | salv-orlando: or confirmed? | 09:53 |
| salv-orlando | if you know the root cause, is triaged imho | 09:55 |
| marun | ok | 09:55 |
| *** Jianyong has left #openstack-neutron | 10:04 | |
| *** dzyu has quit IRC | 10:06 | |
| *** krast_ has quit IRC | 10:10 | |
| marun | jlibosva: ping | 10:12 |
| *** dhellmann has joined #openstack-neutron | 10:15 | |
| *** jp_at_hp has joined #openstack-neutron | 10:19 | |
| *** salv-orlando has quit IRC | 10:23 | |
| *** dave_tucker_zzz is now known as dave_tucker | 10:28 | |
| openstackgerrit | Salvatore Orlando proposed a change to openstack/neutron: Process port_update notifications in the main agent loop https://review.openstack.org/61964 | 10:30 |
| *** nati_ueno has quit IRC | 10:34 | |
| *** dave_tucker is now known as dave_tucker_zzz | 10:35 | |
| *** jp_at_hp has quit IRC | 10:37 | |
| *** yamahata has quit IRC | 10:42 | |
| *** yamahata__ has quit IRC | 10:42 | |
| *** yamahata__ has joined #openstack-neutron | 10:42 | |
| *** yamahata__ has quit IRC | 10:43 | |
| *** yamahata__ has joined #openstack-neutron | 10:43 | |
| *** dave_tucker_zzz is now known as dave_tucker | 10:43 | |
| openstackgerrit | Jianing Yang proposed a change to openstack/neutron: Implement basic functionalities for port forwarding https://review.openstack.org/60512 | 10:44 |
| *** safchain has joined #openstack-neutron | 10:44 | |
| *** yfujioka has quit IRC | 10:45 | |
| *** dave_tucker is now known as dave_tucker_zzz | 10:47 | |
| *** jecarey has joined #openstack-neutron | 10:52 | |
| *** nati_ueno has joined #openstack-neutron | 11:13 | |
| *** jp_at_hp has joined #openstack-neutron | 11:19 | |
| *** jp_at_hp has quit IRC | 11:19 | |
| *** jp_at_hp has joined #openstack-neutron | 11:19 | |
| *** dave_tucker_zzz is now known as dave_tucker | 11:20 | |
| *** dave_tucker is now known as dave_tucker_zzz | 11:30 | |
| yfried | beagles: I have some questions about floating_ip_tracker | 11:31 |
| *** dave_tucker_zzz is now known as dave_tucker | 11:32 | |
| *** pcm has joined #openstack-neutron | 11:33 | |
| openstackgerrit | Oleg Bondarev proposed a change to openstack/neutron: LBaaS: agent monitoring and instance rescheduling https://review.openstack.org/59743 | 11:35 |
| *** pcm has quit IRC | 11:35 | |
| *** pcm has joined #openstack-neutron | 11:36 | |
| *** dave_tucker is now known as dave_tucker_zzz | 11:39 | |
| *** dave_tucker_zzz is now known as dave_tucker | 11:46 | |
| *** marun has quit IRC | 11:48 | |
| openstackgerrit | Jianing Yang proposed a change to openstack/neutron: Implement basic functionalities for port forwarding https://review.openstack.org/60512 | 11:48 |
| *** bashok has quit IRC | 11:49 | |
| *** rohit404 has quit IRC | 11:54 | |
| *** dave_tucker is now known as dave_tucker_zzz | 11:55 | |
| *** afazekas_ has joined #openstack-neutron | 12:01 | |
| *** aymenfrikha has joined #openstack-neutron | 12:06 | |
| *** yfried has quit IRC | 12:07 | |
| *** yfried has joined #openstack-neutron | 12:08 | |
| *** salv-orlando has joined #openstack-neutron | 12:16 | |
| *** rohit404 has joined #openstack-neutron | 12:18 | |
| *** jamespage has joined #openstack-neutron | 12:21 | |
| openstackgerrit | Sylvain Afchain proposed a change to openstack/neutron: L3 Agent can handle many external networks https://review.openstack.org/59359 | 12:23 |
| openstackgerrit | Oleg Bondarev proposed a change to openstack/neutron: LBaaS: agent monitoring and instance rescheduling https://review.openstack.org/59743 | 12:30 |
| *** yfried has quit IRC | 12:32 | |
| *** Abhishek has joined #openstack-neutron | 12:34 | |
| *** yfried has joined #openstack-neutron | 12:45 | |
| ijw | marun: I see i was being talked about | 12:46 |
| ijw | marun: salv-orlando: that was arosen's idea about what was going on, I think, tell him to bloody well move the port-create call | 12:47 |
| ijw | If we only used port-update in nova-compute, and only ever passed ports (not networks) to nove-compute, the code would be less racy and simultaneously simpler. | 12:47 |
| ijw | The only question is whether or how you could unwind the port-create if the scheduling fails | 12:48 |
| salv-orlando | arosen is working on that indeed. technically if the port has not been yet wired, undoing is easy | 12:48 |
| ijw | Yeah, it's more that it's nova that knows the unwinding is required (or you do it in the background) but I rememebr working through the logic for that, maybe on the mailing list | 12:49 |
| salv-orlando | or you don't even need to undo. The port is merely an entry on the db, and as long as scheduling is not done, the host binding part isn't populated | 12:49 |
| ijw | That much is true, but it's automatically created and will persist forever if you don't clean it up | 12:49 |
| salv-orlando | ijw: you're correct this happens currently because nova creates the ports for neutron after scheduling | 12:49 |
| salv-orlando | arose wants to move the logical bits of providing the network resource to before the scheduling | 12:50 |
| ijw | Mind, maybe we were going to get shot of it in nova delete. There was logic to this but it's been a while | 12:50 |
| salv-orlando | s/arose/arosen | 12:50 |
| ijw | Definitely the right answer. Is he going to have time for it? | 12:50 |
| ijw | It's been on his to-do list for a while. | 12:50 |
| ijw | Also, on a more important topic, you around for a Christmas pint next week? | 12:51 |
| *** yfried has quit IRC | 12:55 | |
| *** aymenfrikha has quit IRC | 13:00 | |
| *** aymenfrikha has joined #openstack-neutron | 13:01 | |
| *** jroovers has quit IRC | 13:09 | |
| *** aymenfrikha has quit IRC | 13:13 | |
| jlibosva | anteaya: Hi, is there a way how can I connect to machine that is running the grenade gate test? to see whether the problem is the same I have locally? | 13:21 |
| openstackgerrit | Darragh O'Reilly proposed a change to openstack/neutron: linuxbridge-agent: process port update notifications in the main loop https://review.openstack.org/62875 | 13:26 |
| *** clev has joined #openstack-neutron | 13:29 | |
| *** dhellmann has left #openstack-neutron | 13:31 | |
| *** yfried has joined #openstack-neutron | 13:32 | |
| *** alagalah has joined #openstack-neutron | 13:34 | |
| beagles | yfried: fire away | 13:38 |
| *** banix has joined #openstack-neutron | 13:38 | |
| *** banix has quit IRC | 13:40 | |
| *** jlibosva has quit IRC | 13:43 | |
| openstackgerrit | Oleg Bondarev proposed a change to openstack/neutron: LBaaS: agent monitoring and instance rescheduling https://review.openstack.org/59743 | 13:47 |
| beagles | yfried: actually brb.. gotta do a quick errand | 13:48 |
| *** beagles is now known as beagles_brb | 13:48 | |
| *** jpich has quit IRC | 13:52 | |
| yfried | beagles_brb: In the FLIP_tracker - you're waiting for the FLIP to show up on the VMs DB (~ nova show <VM>) which takes a lot of time. isn't it enough to check the neutron DB? | 13:52 |
| *** jpich has joined #openstack-neutron | 13:53 | |
| openstackgerrit | A change was merged to openstack/neutron: fix --excluded of meter-label-rule-create is not working https://review.openstack.org/61344 | 13:53 |
| openstackgerrit | Oleg Bondarev proposed a change to openstack/neutron: LBaaS: agent monitoring and instance rescheduling https://review.openstack.org/59743 | 13:55 |
| openstackgerrit | Salvatore Orlando proposed a change to openstack/neutron: Process port_update notifications in the main agent loop https://review.openstack.org/61964 | 13:55 |
| *** amotoki has joined #openstack-neutron | 13:55 | |
| *** mattymo has quit IRC | 13:55 | |
| *** mattymo has joined #openstack-neutron | 13:55 | |
| *** julim has joined #openstack-neutron | 13:58 | |
| *** aymenfrikha has joined #openstack-neutron | 14:00 | |
| *** jroovers has joined #openstack-neutron | 14:04 | |
| *** jorisroovers has joined #openstack-neutron | 14:06 | |
| *** ashaikh has joined #openstack-neutron | 14:08 | |
| *** jroovers has quit IRC | 14:09 | |
| jorisroovers | Could someone have a look at https://review.openstack.org/#/c/62867/ for me? Thanks | 14:09 |
| *** jlibosva has joined #openstack-neutron | 14:10 | |
| *** nati_ueno has quit IRC | 14:10 | |
| *** ijw has quit IRC | 14:12 | |
| *** roeyc has joined #openstack-neutron | 14:13 | |
| *** ijw has joined #openstack-neutron | 14:13 | |
| *** peristeri has joined #openstack-neutron | 14:13 | |
| openstackgerrit | Avishay Balderman proposed a change to openstack/neutron: LBaaS L7 model (WIP) https://review.openstack.org/61721 | 14:19 |
| *** yfujioka has joined #openstack-neutron | 14:21 | |
| *** jorisroovers has quit IRC | 14:25 | |
| *** jlibosva has quit IRC | 14:27 | |
| *** richardboswell2 has joined #openstack-neutron | 14:28 | |
| *** wenjianhn has quit IRC | 14:29 | |
| *** otherwiseguy has quit IRC | 14:37 | |
| *** ashaikh has quit IRC | 14:38 | |
| anteaya | kruskakli: awesome, thanks for the feedback | 14:38 |
| anteaya | kruskakli: there is a meeting for people working on third party tests, the meeting it on Thursday at 2200 utc in #openstack-meeting-alt, it would be wonderful if you could attend and share at the meeting what you just shared with me | 14:39 |
| *** jlibosva has joined #openstack-neutron | 14:41 | |
| *** tongli has joined #openstack-neutron | 14:43 | |
| openstackgerrit | Oleg Bondarev proposed a change to openstack/neutron: LBaaS: agent monitoring and instance rescheduling https://review.openstack.org/59743 | 14:44 |
| *** dims_ has quit IRC | 14:44 | |
| openstackgerrit | Darragh O'Reilly proposed a change to openstack/neutron: linuxbridge-agent: process port updates in the main loop https://review.openstack.org/62875 | 14:47 |
| *** insanidade has joined #openstack-neutron | 14:47 | |
| insanidade | hi all. where do I find the recorded webinars from yesterday's PTL meeting? | 14:48 |
| anteaya | jlibosva: I can ask fungi to access a running vm, you will have to tell him what commands you want him to run on it, I am asssuming you want the vm to be running your patch when you access it? | 14:48 |
| *** dims_ has joined #openstack-neutron | 14:48 | |
| anteaya | jlibosva: please join #openstack-infra if you haven't already and I will see about getting that coordinated | 14:49 |
| jlibosva | anteaya: ok, thanks. | 14:49 |
| *** SushilKM has quit IRC | 14:49 | |
| anteaya | insanidade: http://eavesdrop.openstack.org/meetings/project/2013/ | 14:49 |
| jlibosva | anteaya: I want to see the tags of ports in ovs. When I deploy devstack stable/havana on my ubuntu, I see instances have tags 2 while dhcp and router is 1 | 14:50 |
| anteaya | insanidade: there are no webinars, all openstack meetings take place on irc, in #openstack-meeting and #openstack-meeting-alt so they can be logged | 14:50 |
| anteaya | we are an opensource project and we use opensource tools | 14:51 |
| anteaya | jlibosva: let's see what is on fungi's agenda already | 14:51 |
| *** aveiga has joined #openstack-neutron | 14:53 | |
| *** djbkd has joined #openstack-neutron | 14:54 | |
| *** pcm is now known as Guest99997 | 14:55 | |
| *** pcm_ has joined #openstack-neutron | 14:57 | |
| insanidade | anteaya: I mean those: http://www.openstack.org/blog/ (see first post) | 14:58 |
| insanidade | are they webinars ? | 14:58 |
| anteaya | ah summit talks | 14:59 |
| anteaya | the PTL meets every Tuesday, yesterday and the meetings are logged on irc | 14:59 |
| insanidade | yeah. I'd like to see yesterday's discussion but can't find where its recording was made available | 15:00 |
| *** yfujioka has quit IRC | 15:01 | |
| *** amuller has quit IRC | 15:02 | |
| yfried | beagles_brb: are you back? | 15:03 |
| *** beagles_brb is now known as beagles | 15:04 | |
| beagles | yikes | 15:04 |
| beagles | yeah | 15:04 |
| beagles | sorry | 15:04 |
| beagles | went down a rat hole | 15:04 |
| yfried | beagles: how do I post links to lines in the master branch? | 15:04 |
| *** akamyshnikova has joined #openstack-neutron | 15:04 | |
| *** terry_howe has left #openstack-neutron | 15:05 | |
| beagles | good question... | 15:05 |
| beagles | you could just tell me the lines you are referring to :) | 15:05 |
| *** terry_howe has joined #openstack-neutron | 15:05 | |
| yfried | beagles: http://git.openstack.org/cgit/openstack/tempest/tree/tempest/scenario/test_network_basic_ops.py#n56 | 15:06 |
| dkehn | beagles: did you ever get devstack to run straight from trunk, without migration issues? Just asking before I go there | 15:06 |
| yfried | beagles: so you are fetching the server data from nova to check if floating ip has reached it | 15:06 |
| beagles | dkehn: I did a couple of fresh ones in the last few days and it was cool | 15:06 |
| yfried | beagles: right? | 15:06 |
| beagles | yfried: yup | 15:06 |
| dkehn | beagles: great thx | 15:07 |
| yfried | beagles: why? | 15:07 |
| beagles | yfried: why checking if it reaches it or why ask each time? | 15:07 |
| *** alex_klimov has quit IRC | 15:07 | |
| beagles | yfried: if the latter.. well, you have to for the former to make sense :) | 15:07 |
| yfried | beagles: In my experience you should be able to reach the VM well before the FLIP has updated nova's DB | 15:08 |
| yfried | beagles: why not look at neutron's DB - ie port list | 15:08 |
| beagles | yfried: the former, it is meant as a control point | 15:08 |
| yfried | beagles: I'm asking because it seems to crush often in case of stress in the gate | 15:09 |
| beagles | yfried: because the logic I was using was meant to be independent of neutron... ie it would be applicable to nova-networking as well as neutron | 15:10 |
| beagles | yfried: two things about that | 15:10 |
| yfried | so I'm guessing it reaches timeout before VM is update | 15:10 |
| mestery | Question for people: I'm having trouble running unit tests on the latest Neutron. I am on Fedora 20, but what I see fail is something like this: http://paste.openstack.org/raw/55321/ | 15:10 |
| mestery | Anyone seen anything like that recently? | 15:10 |
| mestery | I'm running them with: tox -v -epy27 neutron.tests.unit.ml2.test_mechanism_ncs (as an example) | 15:11 |
| anteaya | insanidade: margie callard wrote the blog post: http://www.openstack.org/foundation/staff | 15:11 |
| beagles | yfried: pinging would timeout as well on the gate... and it would be interpreted as a failure in neutron, whereas it could simply be load | 15:11 |
| anteaya | insanidade: you can email her and ask, personally I have no idea | 15:11 |
| beagles | yfried: if nova has the floating ip, an ping fails then it is definitely the network implementation that has screwed up | 15:11 |
| yfried | beagles: https://review.openstack.org/#/c/55101/ https://review.openstack.org/#/c/62697/ | 15:11 |
| yfried | check out these 2 patches | 15:12 |
| yfried | the first is my test, creating 2 VMs with FLIP | 15:12 |
| yfried | the second adds your check to the test | 15:12 |
| yfried | the second fails half of the time | 15:13 |
| yfried | the first runs without issues | 15:13 |
| beagles | yfried: interesting.. because it was the reverse for me | 15:14 |
| beagles | yfried: in my own environment that is | 15:14 |
| anteaya | vote for J: http://lists.openstack.org/pipermail/openstack/2013-December/004029.html | 15:14 |
| beagles | yfried: there is another facet to this... | 15:14 |
| insanidade | anteaya: thanks :) | 15:15 |
| beagles | yfried: is it a bug if the floating IP does not appear in the nova list in a timely fashion... that being said, the check could be moved until after the connectivity tests and still be valid | 15:15 |
| yfried | beagles: do you know if the gate VMs (the VMs that run the devstack gate) are created by openstack as well? if so - you have an issue of very slow FLIP propagation, and even slower DB update | 15:15 |
| anteaya | mestery: did unit tests work for you previously on the same environment using the same command syntax? | 15:16 |
| mestery | anteaya: Apologies, I have just now realized they pass without some local changes. Rookie mistake. :( | 15:16 |
| * mestery crawls back into his hole. | 15:16 | |
| anteaya | it happens | 15:16 |
| yfried | beagles: why do you need the check if the connection is validated? | 15:16 |
| mestery | :) | 15:16 |
| anteaya | don't do that, stay with us | 15:16 |
| beagles | yfried: what? | 15:16 |
| * mestery could have sworn he checked that last night. | 15:16 | |
| anteaya | morning eyes | 15:17 |
| anteaya | happens to everybody | 15:17 |
| beagles | yfried: I don't understand the question I think | 15:17 |
| anteaya | insanidade: let us know what you find out | 15:17 |
| yfried | beagles: the last one? | 15:17 |
| beagles | yfried: oh, if the floating IP appears in the list? | 15:18 |
| yfried | beagles: let me restart - | 15:18 |
| yfried | beagles: you said that we can move the tracker check to after the connectivity check. am I right? | 15:19 |
| beagles | yfried: you could if you felt it was invalid to put it before pinging.. which would be contrary to its original purpose, which was to add a control point to add sanity to the connectivity checks. | 15:19 |
| *** ashaikh has joined #openstack-neutron | 15:19 | |
| beagles | yfried: bear with me and I'll elaborate on my thinking... | 15:20 |
| beagles | yfried: so basically what we are trying to do is perform functional.. and in some cases very rudimentary system tests (which the scenarios seem to be)... | 15:21 |
| beagles | yfried: automating these things means we have to take the highly asynchronous nature of the system into account... | 15:22 |
| beagles | yfried: for example, even though an initiating call might be synchronous, the resultant changes on the system may be initiated in an asynchronous fashion... | 15:22 |
| beagles | yfried: which means what feedback you do get may be misleading... the ACTIVE state on a VM for example... | 15:23 |
| beagles | yfried: so for system tests to be directed at testing functionality there has to be a "settling phase" to the test in which you verify certain control points... | 15:23 |
| beagles | yfried: testing functionality before settling occurs results in red-herring chases because errors will point to functions failing... | 15:24 |
| beagles | yfried: that is not to say time limits are verbotten, it is just that the time limits are on the control points in the settling phase... | 15:24 |
| pbeskow_ | \ | 15:24 |
| beagles | yfried: so if the settling doesn't occur, the system is still bugged, but it isn't in the functionality, it is in the settling of the asynchronous setup operations... | 15:25 |
| beagles | yfried: in this particular case, ping was used as a control point and I changed it to a round trip outcome... | 15:25 |
| beagles | yfried: if pinging failed after the round trip then neutron is for certain to blame (with the current implementation)... | 15:26 |
| beagles | yfried: if the control point of waiting for the floating IP fails, the issue is a performance issue with propagating the floating ip... | 15:26 |
| beagles | yfried: admittedly what would be best is for there to be some other way to have such a control point... like some kind of feedback that system changes have been applied | 15:27 |
| beagles | yfried that would be perfect | 15:27 |
| *** amritanshu_RnD has quit IRC | 15:27 | |
| yfried | beagles: I followed until the last post | 15:28 |
| beagles | yfried, I think in a meeting yesterday one of the QA guys warned of using the word non-determinism to apply to timing errors that are in fact race conditions.... | 15:28 |
| beagles | yfried: which is valid in a sense, but the race condition is where? the implementation or the test or both... | 15:29 |
| *** otherwiseguy has joined #openstack-neutron | 15:29 | |
| beagles | yfried: pragmatically from a testing view it makes no difference, from our perspective it is all of the difference in the world in where we focus energy | 15:29 |
| beagles | yfried: ok.... last post | 15:29 |
| beagles | yfried: so basically what I was referring to is that in almost every case where we rely on feedback from openstack to know when to continue a test there is an inherent uncertainty because the information is premature... | 15:30 |
| beagles | yfried: eg. the active status on a VM... sure the virt driver says it is active, but the vm might be fscking or something... it isn't really ready for anything, including connecting to it... | 15:31 |
| beagles | yfried: if there were someway to *know* that the VM was ready before trying the next step in that case, everything would be gravy... | 15:31 |
| beagles | yfried: the same goes for changes that are made by neutron | 15:31 |
| beagles | yfried: I haven't come up with a good way to do that.. but I think about it all the time | 15:33 |
| *** roeyc has quit IRC | 15:33 | |
| beagles | yfried: there is something interesting, did you say your test always passes without the check? | 15:35 |
| yfried | beagles: yes | 15:35 |
| yfried | you see the patch log | 15:36 |
| beagles | yfried: network_basic_ops as well? | 15:36 |
| *** fouxm has quit IRC | 15:36 | |
| yfried | I never had issues with network_basic_ops before your change or after | 15:36 |
| beagles | yfried: interesting... it was failing a lot for me when I wrote that | 15:37 |
| yfried | beagles: locally or on the gate? | 15:37 |
| beagles | yfried: locally | 15:37 |
| yfried | where you using more that a single VM\FLIP? | 15:38 |
| beagles | yfried: nope | 15:38 |
| *** csd__ has quit IRC | 15:39 | |
| *** carl_baldwin has joined #openstack-neutron | 15:40 | |
| *** vkozhukalov has quit IRC | 15:40 | |
| yfried | beagles: do you know if anyone else had the same issue as you? | 15:41 |
| yfried | beagles: anyway - since network tests are done with neutron, I think waiting for nova DB isn't the answer. it might work on your system or for a single VM test, but on a more complicated test it fails | 15:41 |
| beagles | yfried: network_basic_ops timing out? | 15:41 |
| yfried | beagles: yeah | 15:41 |
| *** clev has quit IRC | 15:42 | |
| beagles | yfried: re the test.. I was under the impression that this was failing for people.. certainly ping timeouts were known to occur | 15:43 |
| beagles | yfried: regarding the whether to wait or not... I'm fine either way as long as everybody else is, but I say this... | 15:44 |
| beagles | yfried: the fact that floating IP doesn't show up is a bug | 15:44 |
| beagles | yfried: if we want a separate test, then cool... but we should test for it | 15:44 |
| *** jdev789 has joined #openstack-neutron | 15:45 | |
| *** mlavalle has joined #openstack-neutron | 15:45 | |
| *** fouxm has joined #openstack-neutron | 15:45 | |
| mlavalle | enikanorov: ping | 15:45 |
| yfried | beagles: maybe we should put a bigger timeout? | 15:46 |
| yfried | *longer | 15:46 |
| beagles | yfried: not a big fan of longer timeouts... we need a better check for failure... | 15:46 |
| ijw | If this is the timing out test, then arguably if it's missing a timeout that large it *is* a failure | 15:46 |
| yfried | beagles: any Ideas? | 15:47 |
| *** dkehn has quit IRC | 15:48 | |
| beagles | yfried: its tough.. the best checks are system introspective and not implemenation agnostic.. eg. checking the firewall rules or whether the VM is even accessible via its private IP | 15:49 |
| beagles | yfried: and at some point you start muddying the waters between what is acceptable from tempest tests | 15:49 |
| yfried | beagles: we can't do the 2nd due to netns. what about the first? | 15:49 |
| beagles | yfried: same thing.. the rules are in a ns | 15:50 |
| *** dkehn has joined #openstack-neutron | 15:50 | |
| beagles | 1s | 15:50 |
| mlavalle | yfried: thanks for the patch sets…. I will take a look on them around my lunch time (i'm in UTC -6) | 15:53 |
| yfried | mlavalle: the 2nd fails even thought the first works. if you are able to see why I would love to konw | 15:55 |
| yfried | know | 15:55 |
| mlavalle | yfried: ok, I'll try find out…. | 15:55 |
| yfried | beagles: I need to leave. maybe we should move this conv to the mailing list? | 15:56 |
| mlavalle | yfried: how are you located in relationship to UTC? + 2? + 3? | 15:56 |
| beagles | yfried: sure... quick question if I can | 15:56 |
| beagles | yfried: is this failing most of the time in your environment or at the gate? | 15:56 |
| yfried | mlavalle: UTC+2 | 15:57 |
| mlavalle | so, enjoy dinner :-) | 15:57 |
| * beagles proceeds to kill firefox | 15:58 | |
| yfried | mlavalle: I have a long bus ride before and kid to put to bed, but tnx | 15:58 |
| beagles | yfried: the reason I ask is that I'm only seeing one such failure on the gate.. but I'm still looking in case I missed it | 15:58 |
| *** banix has joined #openstack-neutron | 15:59 | |
| yfried | beagles: haven't seen this locally. I'm guessing it's a stress issue, or maybe devstack just takes longer to assign FLIP and the 2nd VM sometimes hits the threshold. | 16:00 |
| yfried | beagles: could you add the timeout value to the log so we can see if the tests that do pass are anywhere near the threshold? | 16:01 |
| yfried | beagles: because we are looking at it as a pass/fail, but if everything is hitting close to the timeout this is a bug and we need to know | 16:02 |
| yfried | beagles: what I'm asking is - since your patch tracks the time it takes for the FLIP to hit the DB, can we log it to the debug logs and see this for all the neutron gate tests for a day? | 16:03 |
| yfried | yfried: I'd do it myself if I knew how | 16:03 |
| beagles | yfried: I'm thinking that that information is already there.. just checking on it | 16:04 |
| beagles | yfried: my firefox is not happy with me | 16:04 |
| *** jdev789 has quit IRC | 16:04 | |
| yfried | beagles: I really need to go now. too bad I can't continue this on my phone but kit-kat killed my IRC app. can you please post this issue and your findings to the mailing list so we can carry this offline and also have more eyes on it? | 16:05 |
| *** ihrachyshka has quit IRC | 16:05 | |
| beagles | yfried: no it's cool.. and will do! cheers | 16:05 |
| yfried | beagles: tnx. | 16:06 |
| *** Abhishek has quit IRC | 16:07 | |
| *** jlibosva has quit IRC | 16:08 | |
| yfried | beagles: check this out: http://logs.openstack.org/01/55101/34/check/check-tempest-dsvm-neutron/7bb7612/console.html | 16:08 |
| yfried | beagles: nm. false alarm. bye now | 16:09 |
| *** ihrachyshka has joined #openstack-neutron | 16:09 | |
| beagles | yfried: :) cheers ... fwiw the calls are more frequent than I anticipated... every second | 16:09 |
| beagles | that's kinda silly.. when you are wating on steady state, the wait intervals have to be reasonable enough to allow steadying.. hammering is only going to make things worse | 16:10 |
| *** SushilKM__ has joined #openstack-neutron | 16:10 | |
| yfried | mlavalle: sdague: I just lost your reviews due to rebase and bug. can you please re post them https://review.openstack.org/#/c/55101/ | 16:10 |
| beagles | I guess it must be modified from the default in the upstream env | 16:11 |
| openstackgerrit | Avishay Balderman proposed a change to openstack/neutron: LBaaS L7 model (WIP) https://review.openstack.org/61721 | 16:12 |
| * beagles thinks on that a minute | 16:13 | |
| *** yamahata has joined #openstack-neutron | 16:13 | |
| *** yfried has quit IRC | 16:15 | |
| *** clev has joined #openstack-neutron | 16:18 | |
| *** Abhishek has joined #openstack-neutron | 16:20 | |
| *** markmcclain has quit IRC | 16:25 | |
| openstackgerrit | Sean M. Collins proposed a change to openstack/neutron: Create a new attribute for subnets, to store v6 dhcp options https://review.openstack.org/52983 | 16:26 |
| *** SushilKM__ has quit IRC | 16:27 | |
| *** roeyc has joined #openstack-neutron | 16:28 | |
| openstackgerrit | Avishay Balderman proposed a change to openstack/neutron: LBaaS L7 model (WIP) https://review.openstack.org/61721 | 16:33 |
| *** SushilKM__ has joined #openstack-neutron | 16:35 | |
| *** fouxm has quit IRC | 16:39 | |
| *** fouxm has joined #openstack-neutron | 16:40 | |
| *** fouxm has quit IRC | 16:44 | |
| openstackgerrit | Svetlana Dobogoeva proposed a change to openstack/neutron: Added unit tests for module neutron/plugins/nicira/api_client/client.py https://review.openstack.org/59948 | 16:50 |
| openstackgerrit | Svetlana Dobogoeva proposed a change to openstack/neutron: Added unit tests for module neutron/plugins/nicira/api_client/client.py https://review.openstack.org/59948 | 16:56 |
| *** kashyap has left #openstack-neutron | 16:58 | |
| *** fouxm has joined #openstack-neutron | 17:01 | |
| *** eezhova has joined #openstack-neutron | 17:03 | |
| openstackgerrit | A change was merged to openstack/neutron: ml2: gre, vxlan type driver can leak segment_id https://review.openstack.org/61694 | 17:04 |
| openstackgerrit | A change was merged to openstack/neutron: Change default eswitchd port to avoid conflict https://review.openstack.org/60203 | 17:06 |
| *** dosaboy has quit IRC | 17:07 | |
| openstackgerrit | A change was merged to openstack/neutron: Imported Translations from Transifex https://review.openstack.org/62812 | 17:07 |
| openstackgerrit | A change was merged to openstack/neutron: Midonet plugin: Fix source NAT https://review.openstack.org/62805 | 17:07 |
| *** dosaboy has joined #openstack-neutron | 17:08 | |
| *** rwsu has joined #openstack-neutron | 17:08 | |
| *** ygbo has quit IRC | 17:08 | |
| *** dosaboy has quit IRC | 17:11 | |
| *** dosaboy has joined #openstack-neutron | 17:11 | |
| *** suresh12 has joined #openstack-neutron | 17:13 | |
| *** fouxm has quit IRC | 17:15 | |
| *** fouxm has joined #openstack-neutron | 17:15 | |
| *** fouxm has quit IRC | 17:15 | |
| anteaya | eezhova: hello | 17:19 |
| eezhova | hi | 17:20 |
| anteaya | mlavalle: ping | 17:20 |
| enikanorov | mlavalle: pong | 17:20 |
| *** SushilKM__ has quit IRC | 17:20 | |
| anteaya | is ann online as well? | 17:20 |
| akamyshnikova | anteaya, hi, I'm here too | 17:20 |
| *** SushilKM has joined #openstack-neutron | 17:20 | |
| anteaya | akamyshnikova: great | 17:20 |
| anteaya | glad to meet you both | 17:20 |
| mlavalle | enikanorov; I want to follow up on your offer of 2 developers | 17:20 |
| anteaya | mlavalle: here they are | 17:20 |
| anteaya | eezhova: and akamyshnikova | 17:21 |
| eezhova | so am I | 17:21 |
| anteaya | so I wanted to ensure that you meet mlavalle | 17:21 |
| anteaya | so that he can review your patches | 17:21 |
| anteaya | and he can talk to you about any blockages you are facing in your work | 17:21 |
| anteaya | can you share the urls of your current neutron tempest patches, please? | 17:22 |
| eezhova | https://review.openstack.org/#/c/58697/ | 17:22 |
| eezhova | https://review.openstack.org/#/c/59729/ | 17:22 |
| eezhova | https://review.openstack.org/#/c/62662/ | 17:22 |
| akamyshnikova | https://review.openstack.org/61118 | 17:23 |
| akamyshnikova | https://review.openstack.org/62620 | 17:23 |
| eezhova | mine are scenario tests for load balancer | 17:23 |
| mlavalle | eezhova and akamyshinikova: very nice meeting you. I'll take a look on those reviews and keep track of them | 17:24 |
| mlavalle | do you have any questions at this point in time? | 17:24 |
| akamyshnikova | I haven't resolve all comments to my patches. I continue working on them tomorrow | 17:24 |
| mlavalle | if you add me (minsel) to the reviewers, they will show up in my gerrit page | 17:25 |
| mlavalle | I can do that myself, though…. don't worry about ir | 17:27 |
| eezhova | well, the only thing that concerns me now is that there is a bug in neutron that was discovered while I was writing the test for health monitor. There is a proposed fix but since it hasn't been reviewed this bug blocks my changes. | 17:27 |
| enikanorov | eezhova: does it block the basic scenario? | 17:28 |
| eezhova | no, just the the succeeding patches | 17:28 |
| enikanorov | ok. i think the first one is ready | 17:30 |
| mlavalle | eezhova, akamyshnikova: I added myself to your patch sets…. I'll track them and you can always ping me here…. Right now I have to run to a meeting. Very glad to meet you | 17:31 |
| eezhova | okay, thank you | 17:31 |
| akamyshnikova | mlavalle, thank you! | 17:31 |
| mlavalle | enikanorov: thanks for the follow up | 17:31 |
| anteaya | eezhova: what patch is the proposed bug fix you need? | 17:32 |
| eezhova | https://review.openstack.org/#/c/61673/ | 17:33 |
| anteaya | if you have patches you need merged for your work, be sure to post those patches in the channel to mlavalle's attention and mine so we can follow them up so you can do your work | 17:33 |
| * anteaya clicks | 17:33 | |
| enikanorov | bugfixes for the advance features are not getting much attention recently | 17:34 |
| *** vkozhukalov has joined #openstack-neutron | 17:34 | |
| anteaya | enikanorov: it is hard for all the reviewers to review | 17:35 |
| anteaya | if the patch is blocking gate bugs or tempest tests, I will do my best to get reviewers on them | 17:35 |
| enikanorov | yeah, i know since i'm one of them | 17:35 |
| anteaya | gate and tempest are my priorities | 17:35 |
| anteaya | yes | 17:36 |
| anteaya | obondarev: are you available? | 17:36 |
| anteaya | obondarev: can you update 61673 with a comment providing some clarity on the outcome of your discussion with enikanorov about your patch | 17:36 |
| anteaya | it will help reviewers to review | 17:37 |
| enikanorov | obondarev has adressed the issues that i pointed out on review in a separate patch | 17:37 |
| enikanorov | so i'm ok with this review | 17:38 |
| *** krast has joined #openstack-neutron | 17:38 | |
| *** jp_at_hp has quit IRC | 17:41 | |
| *** garyk has quit IRC | 17:42 | |
| anteaya | enikanorov: can you take a moment and state as such in a comment on review 61673? | 17:43 |
| enikanorov | ok | 17:44 |
| anteaya | as a reviewer, it just seems like the discussion was never resolved, so it is hard for me to say "yes I am comfortable with this patch" | 17:44 |
| anteaya | thanks | 17:44 |
| *** chandankumar has quit IRC | 17:45 | |
| enikanorov | the thing is that we working closely with obondarev so we discuss things in person often | 17:46 |
| *** suresh12 has quit IRC | 17:56 | |
| *** amotoki has quit IRC | 17:56 | |
| *** jpich has quit IRC | 17:56 | |
| *** clev has quit IRC | 17:56 | |
| *** mlavalle has quit IRC | 17:56 | |
| *** pcm_ has quit IRC | 17:56 | |
| *** rohit404 has quit IRC | 17:56 | |
| *** yamahata__ has quit IRC | 17:56 | |
| *** unicell has quit IRC | 17:56 | |
| *** openstackgerrit has quit IRC | 17:56 | |
| *** enikanorov__ has quit IRC | 17:56 | |
| *** Abhishek has quit IRC | 17:56 | |
| *** aveiga has quit IRC | 17:56 | |
| *** mattymo has quit IRC | 17:56 | |
| *** briancline has quit IRC | 17:56 | |
| *** zigo has quit IRC | 17:56 | |
| *** morganfainberg has quit IRC | 17:56 | |
| *** vkozhukalov has quit IRC | 17:56 | |
| *** peristeri has quit IRC | 17:56 | |
| *** jecarey has quit IRC | 17:56 | |
| *** terry_howe has quit IRC | 17:56 | |
| *** HenryG has quit IRC | 17:56 | |
| *** marios has quit IRC | 17:56 | |
| *** afazekas_ has quit IRC | 17:56 | |
| *** safchain has quit IRC | 17:56 | |
| *** jamespage has quit IRC | 17:56 | |
| *** AndreyGrebenniko has quit IRC | 17:56 | |
| *** pashi_ has quit IRC | 17:56 | |
| *** jianingy_afk has quit IRC | 17:56 | |
| *** dosaboy has quit IRC | 17:56 | |
| *** eezhova has quit IRC | 17:56 | |
| *** yamahata has quit IRC | 17:56 | |
| *** ihrachyshka has quit IRC | 17:56 | |
| *** thansen has quit IRC | 17:56 | |
| *** _cerberus_ has quit IRC | 17:56 | |
| *** sdague has quit IRC | 17:56 | |
| *** ivoks has quit IRC | 17:56 | |
| *** metral has quit IRC | 17:56 | |
| *** inara has quit IRC | 17:57 | |
| *** Qlawy has quit IRC | 17:57 | |
| *** pasquier-s has quit IRC | 17:57 | |
| *** doude_ has quit IRC | 17:57 | |
| *** jog0 has quit IRC | 17:57 | |
| *** MichielHN has quit IRC | 17:57 | |
| *** mrsnivvel has quit IRC | 17:57 | |
| *** obondarev has quit IRC | 17:57 | |
| *** harlowja_away has quit IRC | 17:57 | |
| *** lifeless has quit IRC | 17:57 | |
| *** ywu has quit IRC | 17:57 | |
| *** carl_baldwin has quit IRC | 17:57 | |
| *** tongli has quit IRC | 17:57 | |
| *** ijw has quit IRC | 17:57 | |
| *** salv-orlando has quit IRC | 17:57 | |
| *** majopela has quit IRC | 17:57 | |
| *** dsockwell has quit IRC | 17:57 | |
| *** skraynev has quit IRC | 17:57 | |
| *** dkehn has quit IRC | 17:57 | |
| *** otherwiseguy has quit IRC | 17:57 | |
| *** dims_ has quit IRC | 17:57 | |
| *** julim has quit IRC | 17:57 | |
| *** matrohon has quit IRC | 17:57 | |
| *** dave_tucker_zzz has quit IRC | 17:57 | |
| *** rdo has quit IRC | 17:57 | |
| *** fcoj has quit IRC | 17:57 | |
| *** rkukura has quit IRC | 17:57 | |
| *** benner has quit IRC | 17:57 | |
| *** ekarlso has quit IRC | 17:57 | |
| *** krast has quit IRC | 17:57 | |
| *** rwsu has quit IRC | 17:57 | |
| *** banix has quit IRC | 17:57 | |
| *** kruskakli has quit IRC | 17:57 | |
| *** larsks has quit IRC | 17:57 | |
| *** sgran has quit IRC | 17:57 | |
| *** ashaikh has quit IRC | 17:57 | |
| *** richardboswell2 has quit IRC | 17:57 | |
| *** mordred has quit IRC | 17:57 | |
| *** pbeskow_ has quit IRC | 17:57 | |
| *** sc68cal has quit IRC | 17:57 | |
| *** pvo has quit IRC | 17:57 | |
| *** ajo has quit IRC | 17:57 | |
| *** EmilienM has quit IRC | 17:57 | |
| *** russellb has quit IRC | 17:57 | |
| *** yongli has quit IRC | 17:57 | |
| *** akamyshnikova_ has quit IRC | 17:57 | |
| *** lari__ has quit IRC | 17:57 | |
| *** Makdaam has quit IRC | 17:57 | |
| *** aryan has quit IRC | 17:57 | |
| *** cburgess has quit IRC | 17:57 | |
| *** christophk has quit IRC | 17:57 | |
| *** zhhuabj has quit IRC | 17:57 | |
| *** angryjesters has quit IRC | 17:57 | |
| *** roaet has quit IRC | 17:57 | |
| *** gizmoguy_ has quit IRC | 17:57 | |
| *** Apsu has quit IRC | 17:57 | |
| *** decede has quit IRC | 17:57 | |
| *** markvoelker has quit IRC | 17:57 | |
| *** ogelbukh has quit IRC | 17:57 | |
| *** annegentle_ has quit IRC | 17:57 | |
| *** enikanorov has quit IRC | 17:57 | |
| *** jhurlbert has quit IRC | 17:57 | |
| *** nijaba has quit IRC | 17:57 | |
| *** SushilKM has quit IRC | 17:57 | |
| *** roeyc has quit IRC | 17:57 | |
| *** akamyshnikova has quit IRC | 17:57 | |
| *** aymenfrikha has quit IRC | 17:57 | |
| *** zoresvit has quit IRC | 17:57 | |
| *** SumitNaiksatam has quit IRC | 17:57 | |
| *** djbkd has quit IRC | 17:57 | |
| *** alagalah has quit IRC | 17:57 | |
| *** mtreinish has quit IRC | 17:57 | |
| *** insanidade has quit IRC | 17:57 | |
| *** JoeHazzers has quit IRC | 17:57 | |
| *** asadoughi has quit IRC | 17:57 | |
| *** anteaya has quit IRC | 17:57 | |
| *** lowkey has quit IRC | 17:57 | |
| *** openstack has joined #openstack-neutron | 18:02 | |
| -hobana.freenode.net- [freenode-info] why register and identify? your IRC nick is how people know you. http://freenode.net/faq.shtml#nicksetup | 18:02 | |
| *** openstack has joined #openstack-neutron | 18:06 | |
| *** akamyshnikova has quit IRC | 18:09 | |
| *** harlowja_away is now known as harlowja | 18:09 | |
| *** mestery has joined #openstack-neutron | 18:12 | |
| *** beagles has joined #openstack-neutron | 18:12 | |
| *** changbl has joined #openstack-neutron | 18:12 | |
| *** roeyc has quit IRC | 18:13 | |
| *** yfried has joined #openstack-neutron | 18:13 | |
| *** SushilKM has quit IRC | 18:13 | |
| *** Abhishek has quit IRC | 18:15 | |
| *** Abhishek has joined #openstack-neutron | 18:17 | |
| *** nati_ueno has joined #openstack-neutron | 18:22 | |
| *** eezhova has left #openstack-neutron | 18:22 | |
| *** majopela has quit IRC | 18:23 | |
| *** majopela has joined #openstack-neutron | 18:24 | |
| *** suresh12 has joined #openstack-neutron | 18:25 | |
| *** majopela has quit IRC | 18:26 | |
| *** Mierdin has joined #openstack-neutron | 18:27 | |
| *** jpich has quit IRC | 18:38 | |
| *** djbkd has left #openstack-neutron | 18:44 | |
| *** markmcclain has joined #openstack-neutron | 18:45 | |
| *** salv-orlando_ has joined #openstack-neutron | 18:49 | |
| *** salv-orlando has quit IRC | 18:50 | |
| *** salv-orlando_ is now known as salv-orlando | 18:50 | |
| *** mlavalle has joined #openstack-neutron | 18:50 | |
| *** SushilKM has joined #openstack-neutron | 18:50 | |
| *** afazekas_ has quit IRC | 18:59 | |
| terry_howe | markmcclain or salv-orlando_ do you guys know if the DHCP agent to network is one-to-one, one-to-many or many-to-many | 19:01 |
| markmcclain | terry_howe: each agent manages the dnsmasq instances for multiple networks | 19:03 |
| terry_howe | thanks | 19:04 |
| *** Abhishek has quit IRC | 19:07 | |
| anteaya | enikanorov: this bug is now #2 in the gate and rising: https://bugs.launchpad.net/nova/+bug/1254890 | 19:08 |
| *** yfried has quit IRC | 19:09 | |
| anteaya | enikanorov: have you had a chance to look at it at all? | 19:09 |
| *** richardboswell2 has quit IRC | 19:09 | |
| anteaya | dims_: you also are on this bug, any thoughts you might be able to share with enikanorov? | 19:09 |
| enikanorov | anteaya: i've looked into other one yet. will switch | 19:09 |
| anteaya | enikanorov: which other one? | 19:10 |
| *** yfried has joined #openstack-neutron | 19:10 | |
| *** yfried has quit IRC | 19:10 | |
| *** SumitNaiksatam has joined #openstack-neutron | 19:12 | |
| enikanorov | https://bugs.launchpad.net/neutron/+bug/1210483 | 19:12 |
| enikanorov | I've managed to repro this one like once in 50 test runs. haven't managed to identify the root cause of the race | 19:12 |
| anteaya | :( | 19:13 |
| *** rohit404 has quit IRC | 19:13 | |
| anteaya | can you update the report for 1210483 before you switch | 19:13 |
| anteaya | they are both in the gate | 19:13 |
| *** networkstatic has quit IRC | 19:13 | |
| anteaya | let me dig into 1210483 some more on logstash and see if I can get you something helpful | 19:14 |
| anteaya | and thank you | 19:14 |
| *** yfried has joined #openstack-neutron | 19:14 | |
| anteaya | *they are both on the elastic-recheck page indicating they are affecting the gate | 19:16 |
| enikanorov | for 1210483 logs doesn't produce anything specific except tempest failure itself, that's the problem. looks like it's some imternal timing issue between several calls from nova to neutron | 19:26 |
| yfried | beagles: did you happen to send a mail about our discussion? I can't see it | 19:36 |
| anteaya | enikanorov: okay, I am going through logstash now to see if I can find something more helpful | 19:38 |
| enikanorov | anteaya: ok | 19:39 |
| anteaya | enikanorov: do you think there needs to be an increase in logging somewhere to get more accurate information? | 19:39 |
| enikanorov | anteaya: I'm thinking about it... once i find something which will help to identify the issue I'll may be push a patch or something. currently I'm just going to continue with code analysis | 19:41 |
| *** ajo is now known as zz_ajo | 19:42 | |
| anteaya | enikanorov: okay, yes if you need better logging to get to the bottom of 1210483, please submit a patch | 19:43 |
| anteaya | jog0: are you available? | 19:44 |
| anteaya | I have come across something odd in logstash and I am hoping you are able to help me understand it | 19:44 |
| *** insanida1e has joined #openstack-neutron | 19:45 | |
| *** insanidade has quit IRC | 19:48 | |
| dims_ | anteaya, i did not find out much, need to look again | 19:49 |
| anteaya | dims_: great thanks enikanorov is also working on it, #2 bug in the gate and rising | 19:50 |
| anteaya | if you are able to work together, that would be awesome | 19:50 |
| *** zz_ajo is now known as ajo | 19:55 | |
| *** SumitNaiksatam has quit IRC | 19:57 | |
| *** dave_tucker_zzz is now known as dave_tucker | 19:57 | |
| *** otherwiseguy has quit IRC | 20:00 | |
| salv-orlando | Random question for all the people that every day run the l3 agent. If I ask you something like "what the highest frequency of router updated messages your l3 agent can handle safely, where safely mean they are all processed within a minute?" would there be somebody with an answer? | 20:01 |
| salv-orlando | or at least "how long on average does it take to process a router"? | 20:01 |
| *** ashaikh has quit IRC | 20:07 | |
| openstackgerrit | Amir Sadoughi proposed a change to openstack/python-neutronclient: Added --source-port-range-min, --source-port-range-max https://review.openstack.org/62130 | 20:12 |
| beagles | yfried: not yet | 20:13 |
| yfried | beagles: ok | 20:13 |
| beagles | yfried: fwiw though you can see how long it took for the checks to pass in the tempest.log | 20:14 |
| beagles | yfried: what I also noticed that the check is cycling every second... that's too fast IMO | 20:14 |
| beagles | yfried: because it results in a query to nova which is just causing a load on something that is already loaded | 20:15 |
| beagles | yfried: well... probably calls neutron too | 20:15 |
| beagles | yfried: so .. that needs to be backed off, because that is just making the situation worse | 20:15 |
| *** SumitNaiksatam has joined #openstack-neutron | 20:19 | |
| *** dave_tucker is now known as dave_tucker_zzz | 20:20 | |
| *** ashaikh has joined #openstack-neutron | 20:20 | |
| *** dave_tucker_zzz is now known as dave_tucker | 20:22 | |
| openstackgerrit | A change was merged to openstack/neutron: ml2/type_gre: Adds missing clear_db to test_type_gre.py https://review.openstack.org/62530 | 20:27 |
| *** vkozhukalov has quit IRC | 20:28 | |
| *** gizmoguy_ is now known as gizmoguy | 20:31 | |
| *** networkstatic has joined #openstack-neutron | 20:43 | |
| *** dave_tucker is now known as dave_tucker_zzz | 20:53 | |
| salv-orlando | fellow neutron core dev: a +A might be welcome here - https://review.openstack.org/#/c/58860/ | 20:55 |
| salv-orlando | this patch was already approved, but I blocked it since it failed the gate for bug 1253896. After verifying the logs and several rechecks I think it's safe to send it again through the gare | 20:56 |
| *** ajo is now known as zz_ajo | 20:56 | |
| *** marun has joined #openstack-neutron | 20:56 | |
| *** aveiga has quit IRC | 21:17 | |
| *** geekinutah has joined #openstack-neutron | 21:17 | |
| *** clev has quit IRC | 21:21 | |
| *** mlavalle has quit IRC | 21:22 | |
| *** mlavalle has joined #openstack-neutron | 21:29 | |
| *** otherwiseguy has joined #openstack-neutron | 21:32 | |
| anteaya | otherwiseguy: hey there | 21:33 |
| openstackgerrit | Cyril Roelandt proposed a change to openstack/python-neutronclient: Sync with oslo https://review.openstack.org/62991 | 21:34 |
| jog0 | anteaya: pong | 21:34 |
| otherwiseguy | anteaya: hello | 21:35 |
| anteaya | jog0: hello there | 21:36 |
| anteaya | otherwiseguy: haven't see you in a bit, glad to have you around | 21:36 |
| jog0 | you had a logstash question | 21:36 |
| anteaya | yes | 21:37 |
| anteaya | http://bit.ly/19di5CB | 21:37 |
| anteaya | let's start here | 21:37 |
| anteaya | do you see the build hitting the top failure | 21:37 |
| otherwiseguy | anteaya: glad to be around. :) hopefully getting caught up on my RH havana release responsibilities this week. | 21:37 |
| anteaya | jog0: confirm https://review.openstack.org/#/c/62530 | 21:37 |
| anteaya | otherwiseguy: great, it will be great to have you back on upstream | 21:38 |
| anteaya | we need you | 21:38 |
| otherwiseguy | upstream work is certainly a lot more fun. ;) | 21:38 |
| jog0 | anteaya: yeah I see that | 21:38 |
| anteaya | do you see in the patch | 21:38 |
| anteaya | elastic-recheck suggests recheck bug 1254890 | 21:39 |
| anteaya | for that failure | 21:39 |
| anteaya | and the logs have timeout waiting for thing: http://logs.openstack.org/30/62530/2/gate/gate-tempest-dsvm-neutron-pg/8a0bbff/console.html#_2013-12-18_17_36_51_094 | 21:40 |
| openstackgerrit | Micheal Thompson proposed a change to openstack/neutron: Fixed PEP8 issues hopefully. Corrected Cookie issue Patch 7 Fixed hopfully the last pep8 issue. Pep8 validation passing in local environment but not Remote https://review.openstack.org/62464 | 21:40 |
| anteaya | which is 1254890 | 21:40 |
| jog0 | anteaya: athat is a different test | 21:40 |
| jog0 | err build UUID | 21:40 |
| jog0 | so what you are seeing is build uuid afed48ac95da4bfb80524e52cedd0319 ran but for some reason didn't become the one used by zuul | 21:41 |
| jog0 | because it was a gate failure | 21:41 |
| anteaya | oh | 21:41 |
| jog0 | if something in front of that patch fails in the gate, then all subsequent results are thrown out, but logstash recordds them anyway | 21:41 |
| jog0 | anteaya: that confused me for a while too | 21:42 |
| anteaya | so logstash records them but I can't access the logs to follow it up | 21:42 |
| anteaya | hrumph | 21:42 |
| jog0 | you can, just see log_url | 21:42 |
| jog0 | http://logs.openstack.org/30/62530/2/gate/gate-tempest-dsvm-neutron-pg-isolated/afed48a/console.html | 21:42 |
| anteaya | okay, do I have access to anything in addition to the console.html output? | 21:43 |
| jog0 | yes, delete the console.html from the URL | 21:44 |
| anteaya | thanks | 21:45 |
| anteaya | that was all I needed | 21:46 |
| anteaya | knowing what logstash is doing really helps | 21:46 |
| jog0 | glad to help | 21:46 |
| anteaya | jog0: this bug is rising: https://bugs.launchpad.net/tempest/+bug/1258682 | 21:47 |
| anteaya | not a neutron bug but noone is assigned | 21:47 |
| anteaya | thought you might want to know | 21:47 |
| anteaya | and can we finish off our discussion from last night? | 21:48 |
| openstackgerrit | Salvatore Orlando proposed a change to openstack/neutron: Commit for testing router processing times on gate https://review.openstack.org/62994 | 21:49 |
| jog0 | anteaya: sure | 21:50 |
| anteaya | jog0: we had been discussing https://bugs.launchpad.net/neutron/+bug/1244255 | 21:50 |
| jog0 | FYI for 1258682 the issue right now is we don't collect logs | 21:50 |
| anteaya | can you look at my notes at the bottom and perhaps we can go from there | 21:50 |
| *** roeyc has joined #openstack-neutron | 21:50 | |
| jog0 | so working on getting that done first | 21:51 |
| jog0 | then we can identify what is causing tempest to timeout | 21:51 |
| anteaya | jog0: ah okay, add me to the patch once it is up, who is working on it? | 21:51 |
| jog0 | anteaya: Changing priority to low. The particular symptoms are cured for now, but the issue remains. | 21:51 |
| openstackgerrit | Cyril Roelandt proposed a change to openstack/python-neutronclient: Use six.moves.cStringIO rather than cStringIO https://review.openstack.org/62995 | 21:51 |
| jog0 | ignore the last comment | 21:52 |
| * anteaya ignores | 21:52 | |
| jog0 | for 1258682 | 21:53 |
| jog0 | https://review.openstack.org/#/c/62784/ | 21:53 |
| jog0 | and https://review.openstack.org/#/c/62786/ | 21:53 |
| openstackgerrit | A change was merged to openstack/neutron: Add support for NSX/NVP Metadata services https://review.openstack.org/49415 | 21:53 |
| jog0 | ok on to 1244255 | 21:54 |
| anteaya | neither mention bug 1258682 | 21:54 |
| jog0 | anteaya: yeah :( | 21:54 |
| jog0 | they are related to that bug but not exclusively | 21:55 |
| jog0 | will update my patch to link it | 21:55 |
| *** suresh12 has quit IRC | 21:56 | |
| jog0 | ok 1244255 | 21:57 |
| *** hichihara has joined #openstack-neutron | 21:58 | |
| jog0 | I agree with your comment, this isn't a gate issue and priority can chance accordingly | 21:58 |
| jog0 | although IIRC the patch that caused all those issues was a tempest patch | 21:58 |
| anteaya | hmmm | 21:59 |
| anteaya | do you remember which tempest patch? | 21:59 |
| mlavalle | anteaya: hi, we are supposed to talk at 22:00 UTC | 22:00 |
| jog0 | oh was wrong https://review.openstack.org/#/c/59787/ | 22:00 |
| beagles | o/ | 22:00 |
| anteaya | we are | 22:00 |
| anteaya | so here we are | 22:01 |
| jog0 | its the api workers | 22:01 |
| *** tongli has quit IRC | 22:01 | |
| anteaya | jog0: yes | 22:01 |
| jog0 | so that feature not working is bad | 22:01 |
| anteaya | yes | 22:01 |
| *** suresh12 has joined #openstack-neutron | 22:01 | |
| anteaya | and it hits that failure a lot | 22:01 |
| jog0 | as it means right now setting api_workersfrom 0 to 4 goes boom | 22:01 |
| anteaya | it does | 22:01 |
| *** dave_tucker_zzz is now known as dave_tucker | 22:01 | |
| anteaya | mlavalle beagles so what is new? | 22:01 |
| anteaya | who wants to go first? | 22:02 |
| beagles | I think rossella_s is still ill, but I started going through each test in the "full" suite yesterday | 22:02 |
| anteaya | awesome, thank you | 22:02 |
| anteaya | what did you find? | 22:02 |
| beagles | the faulty cleanup breaks many things which means pinpointing "blame neutron" tests laborsome | 22:03 |
| anteaya | where is the cleanup faulty | 22:03 |
| anteaya | in the neutron tempest tests? | 22:03 |
| beagles | in some cases, no | 22:03 |
| beagles | I will have a complete report sometime tomorrow | 22:03 |
| anteaya | from your gut for now | 22:04 |
| anteaya | what do you think? | 22:04 |
| anteaya | if it changes tomorrow that is fine | 22:04 |
| beagles | k 1 s | 22:04 |
| beagles | so far I've run 1402 tests in isolation | 22:05 |
| mlavalle | anteaya: followed up with enikanorov about 2 developers, met them, got reviews from them. Reviewed several patches for yfried. We are working together in debugging one of them. unstuck a review for joris Roovers. Later tonight I will continue ading to the api gap analysis. That's all | 22:05 |
| anteaya | mlavalle: okay thanks, good work | 22:06 |
| anteaya | keep doing that | 22:06 |
| anteaya | beagles: awesome | 22:06 |
| beagles | and 106 have failed | 22:06 |
| anteaya | any pattern to the 106 that failed? | 22:06 |
| mlavalle | ok, got run…. se you around | 22:06 |
| anteaya | and yay for the 1296 that passed | 22:07 |
| beagles | in some cases, they are not neutron at all but errors with "image" related tests | 22:07 |
| anteaya | mlavalle: k | 22:07 |
| beagles | it breaks several | 22:07 |
| anteaya | "image" related tests | 22:07 |
| anteaya | glance? | 22:07 |
| beagles | 1s.. I'll get you an example | 22:07 |
| beagles | tempest.api.compute.images.test_image_metadata.ImagesMetadataTestJSON.test_update_image_metadata.result.txt | 22:07 |
| beagles | at the moment I'm weeding them out and continuing | 22:08 |
| beagles | but keeping track, all of the same | 22:08 |
| beagles | I'll need to share with the parties who might be interested | 22:08 |
| *** jroovers has joined #openstack-neutron | 22:09 | |
| beagles | many of the neutron only tests appear to pass just fine, but I don't have a concrete number | 22:09 |
| anteaya | okay thanks for your early findings | 22:09 |
| anteaya | I'll see if I can get a glance person in channel to listen in | 22:09 |
| *** aymenfrikha has quit IRC | 22:09 | |
| beagles | I am working on a run-in-isolation-of-each-process that will detect broken cleanups of different types and continue with the next test | 22:10 |
| beagles | with the hope of getting a better picture of what is really not running | 22:10 |
| *** peristeri has quit IRC | 22:10 | |
| beagles | and what tests are broken with cleanups | 22:10 |
| *** markwash has joined #openstack-neutron | 22:11 | |
| anteaya | thanks markwash | 22:11 |
| anteaya | beagles was just updating me on this progress going over the tempest tests for neutron individually | 22:12 |
| anteaya | he hasn't finished his assessment yet | 22:12 |
| markwash | yeah, I'm interested to see what could be going on, if there's any way we can help | 22:12 |
| anteaya | thank you | 22:12 |
| anteaya | beagles: did you want to repeat some of your thoughts for markwash's benefit? | 22:12 |
| beagles | sure and greetings to markwash | 22:13 |
| markwash | greetings! | 22:13 |
| beagles | so, basically in an effort to isolate individual failures of tests for neutron, I've taken a listing of tests from the "full" run and have begun working with breaking them into groups and isolating failures | 22:14 |
| beagles | in some cases failures of the tests themselves or of setup or teardown leaves detritus around that interferes with further testing | 22:14 |
| beagles | in the case of some of the tests I was running, cleanups were failing to delete VMs left behind resulting in quotas being exceeded and interfering with subsequent tests | 22:15 |
| jroovers | hi enikanorov | 22:15 |
| enikanorov | hi | 22:15 |
| beagles | I'm not sure if these belong to glance or not, but I guess since the term "image" shows up in some .. | 22:15 |
| jroovers | enikanorov, question related to https://review.openstack.org/#/c/62867/ | 22:16 |
| markwash | beagles: are some images getting left behind with that detritus? | 22:16 |
| enikanorov | jroovers: go ahead | 22:16 |
| beagles | actually.. you know these tests are pretty clearly compute not glance | 22:16 |
| jroovers | enikanorov: I'm quite new to openstack/neutron/tempest dev | 22:16 |
| beagles | tempest.api.compute.images.test_image_metadata.ImagesMetadataTestXML.test_get_image_metadata_item.result.txt | 22:16 |
| beagles | maybe? | 22:16 |
| jroovers | enikanorov: so looking for a bit of guidance, what would your recommendation be. Wait or rebase? | 22:17 |
| markwash | beagles: It probably is still talking to glance in some way | 22:17 |
| markwash | beagles: so suspicion isn't completely lifted | 22:17 |
| enikanorov | jroovers: doesn't matter actually. It's up to you | 22:17 |
| beagles | markwash: okay an example would be a NotFound: Object not found | 22:18 |
| beagles | Details: {"itemNotFound": {"message": "Image not found.", "code": 404}} | 22:18 |
| beagles | exception in tempest.api.compute.images.test_image_metadata.ImagesMetadataTestJSON | 22:18 |
| jroovers | enikanorov: so, if I'm understanding your comments correctly, they only relate to the network_client part right? | 22:18 |
| jroovers | enikanorov: the tests itself are good? | 22:18 |
| enikanorov | jroovers: i was mainly concerned about adding stuff that i tried to remove from the clients | 22:18 |
| markwash | beagles: in one case before we saw some issues where an image was being deleted before it's PUT had finished | 22:18 |
| markwash | and tempest was angry somehow, or something was complaining about errors being in the logs | 22:19 |
| enikanorov | jroovers: the tests look good, however I don't know if this set is enough or not | 22:19 |
| jroovers | enikanorov: its based on the doc | 22:19 |
| beagles | markwash: ah cool | 22:19 |
| jroovers | enikanorov: https://etherpad.openstack.org/p/icehouse-summit-qa-neutron | 22:19 |
| enikanorov | jroovers: also I don't know if they should be placed in a separate file (is it a common practice?) | 22:19 |
| enikanorov | jroovers: got it | 22:20 |
| jroovers | enikanorov: yes, separate file is a tempest guideline | 22:20 |
| enikanorov | separate file for negative, right? | 22:20 |
| jroovers | enikanorov: yes | 22:20 |
| enikanorov | ok then | 22:20 |
| enikanorov | tests look good | 22:20 |
| jroovers | enikanorov: so would it be good to just remove the client part then? | 22:20 |
| jroovers | that would essentially mean that the tests would only work for xml now | 22:21 |
| jroovers | enikanorov: sorry, err, only for json | 22:21 |
| enikanorov | if you could rebase on some of my patches, you should be able to run the tests for both xml and json | 22:21 |
| jroovers | ok, I'll try that | 22:21 |
| markwash | beagles: if there are some failures you want me to take a look at I'd probably have some time over the next day or so to dig into logs | 22:21 |
| beagles | markwash: as there are lot of failures and eventually each subsequent test simply fails because quotas eventually get blown, maybe it would be best for me to simply publish the entire results tomorrow when I get the neutron stuff straightened | 22:21 |
| jroovers | enikanorov: don't have any experience with that, but I'll figure it out | 22:22 |
| beagles | markwash: +1 | 22:22 |
| enikanorov | yeah, i think it's quite simple. initialy clients had lots of similar code that i just tried to automate. that however slightly changes client's usage. | 22:22 |
| anteaya | beagles markwash it looks like we are in a good place to go forward | 22:22 |
| enikanorov | ok, it's 2:20 am here so i'm going to sleep a bit | 22:23 |
| jroovers | enikanorov: thanks for the help, I'll get back to you if I have additional questions | 22:23 |
| jroovers | enikanorov: good night! | 22:23 |
| enikanorov | jroovers: sure. | 22:23 |
| anteaya | markwash: thanks for jumping in, you are welcome to idle, would be great to have you available for pinging if things come up | 22:23 |
| beagles | anteaya agreed | 22:23 |
| *** hichihara has quit IRC | 22:23 | |
| markwash | okay I'll lurk for a bit | 22:23 |
| anteaya | awesome, great work beagles | 22:23 |
| anteaya | markwash: thank you | 22:23 |
| anteaya | and if there is anything we/I can do to help glance be sure to ask | 22:24 |
| beagles | in addition, the use case stuff proceeds, but I want to give this breakdown a nudge priority wise .. | 22:24 |
| beagles | so we can parcel out work | 22:24 |
| anteaya | it would be good to know what does work | 22:24 |
| beagles | :) indeed it would | 22:24 |
| anteaya | and if there is a pattern to the failures | 22:24 |
| anteaya | so great direction | 22:24 |
| jroovers | anteaya or mlavalle , can I ask you a question about https://review.openstack.org/#/c/62867/ | 22:24 |
| anteaya | jroovers: ask away | 22:25 |
| beagles | btw mlavalle ( I guess arosen isnt' around) | 22:25 |
| anteaya | I haven't seen arosen lately | 22:25 |
| beagles | I think he's travelling | 22:25 |
| jroovers | anteaya: ok, thanks :-) So I was wondering why there is a test failing | 22:25 |
| beagles | but anteaya this is for you to :) | 22:25 |
| anteaya | though he does tend to be kind of quiet | 22:25 |
| beagles | or too | 22:25 |
| jroovers | anteaya: first it was check-tempest-dsvm-full | 22:26 |
| beagles | I was doings some testing of searching by ip from nova | 22:26 |
| beagles | so you do something like nova list --ip 10.0.0\..* or whatever | 22:26 |
| jroovers | anteaya: sorry, first it was check-grenade-dsvm and now it is check-tempest-dsvm-full | 22:26 |
| jroovers | anteaya: I have the feeling like this is something outside of my control | 22:26 |
| anteaya | jroovers: hve you looked at the logs for the failures? | 22:26 |
| beagles | it doesn't seem to work | 22:27 |
| anteaya | start with the console.html log first | 22:27 |
| mlavalle | jroovers: the first time you submitted your patch set, the check-grenade-dsvm failed to build devstack…. it had nothing to do with your test…. that's why I reched it | 22:27 |
| beagles | so I've got to do a bit more analysis including checking if it is already fixed since havana dropped | 22:27 |
| anteaya | beagles: can you paste what output you do get from that command? | 22:27 |
| beagles | but in the meantime heads up | 22:27 |
| beagles | anteaya: that's kind of the thing.. there is no output | 22:28 |
| jroovers | mlavalle: how did you initiate that recheck? | 22:28 |
| beagles | I need to check with trunk... | 22:28 |
| * anteaya looks quizzical | 22:28 | |
| anteaya | hmmm | 22:28 |
| mlavalle | jroovers: if it failed again after that recheck, we have to look at the logs of the job where it failed. | 22:28 |
| anteaya | alright thanks for the heads up | 22:28 |
| beagles | nova list --ip 10.0.0\.* .. you get no results | 22:28 |
| anteaya | hmmmm | 22:28 |
| anteaya | I have no input | 22:28 |
| anteaya | other than to say I would expect something | 22:29 |
| beagles | :) | 22:29 |
| anteaya | mlavalle: let's take a moment and ensure that jroovers develops good habits for addressing this test feedback from jenkins | 22:29 |
| anteaya | jroovers: so did you look at the logs of both failed tests? | 22:29 |
| beagles | yeah.. at any rate it is a parity bug if it still exists, I just need to do some sleuthing to verify | 22:29 |
| jroovers | anteaya: I did look at the first one, and am doing that again right now | 22:30 |
| anteaya | beagles: happy sleuthing | 22:30 |
| anteaya | jroovers: link to the error please once you find it | 22:30 |
| beagles | anteaya mlavalle: and that is all I have at the moment | 22:30 |
| mlavalle | jroovers: I am busy right now, but I will take a look again later tonight. In general, you don't recheck a patchset unless you went thorugh the logs of the failing job (which I did with your patchset) and you know that the patchset is not the cause | 22:30 |
| anteaya | beagles: great report, thanks | 22:30 |
| mlavalle | jroovers: but your initial assumption should be that the gate is pointing to a bug in your patchset | 22:32 |
| mlavalle | that is why we have a gate | 22:32 |
| jroovers | mlavalle: ok, thanks. Just as a clarification, what button do you click to recheck, or do you just add a comment and does jenkins automatically rechecks | 22:32 |
| mlavalle | yes, you just do a review with a recheck comment | 22:32 |
| jroovers | mlavalle, definitely agreed on checking your own code first. | 22:32 |
| jroovers | mlavalle: ok, good to know, thanks | 22:33 |
| openstackgerrit | Cyril Roelandt proposed a change to openstack/python-neutronclient: Remove an unused imported module https://review.openstack.org/63003 | 22:34 |
| jroovers | anteaya, so related to the first issue (http://logs.openstack.org/67/62867/1/check/check-grenade-dsvm/b0941ce/console.html) | 22:34 |
| jroovers | anteaya: the problem is that keystone did not upgrade correctly | 22:35 |
| salv-orlando | beagles: not sure I understood everything by scrolling back but neutron does not support wildcard matching on ip addresses. | 22:35 |
| jroovers | anteaya: "./grenade.sh:253 Failure in upgrade-keystone" | 22:35 |
| anteaya | jroovers: great work | 22:35 |
| anteaya | and you can click on the timestamp beside that error for it to be added to the url | 22:35 |
| beagles | salv-orlando: on port-list? | 22:35 |
| anteaya | thus | 22:36 |
| anteaya | http://logs.openstack.org/67/62867/1/check/check-grenade-dsvm/b0941ce/console.html#_2013-12-18_13_16_25_614 | 22:36 |
| salv-orlando | I think grenade is non-voting for neutron at the moment, people are still working on getting the job right | 22:36 |
| salv-orlando | yeas neutron commands. | 22:36 |
| beagles | salv-orlando: or rather through the ports.json? | 22:36 |
| jroovers | anteaya: yes | 22:36 |
| beagles | salv-orlando: through the cli it definitely doesn't | 22:36 |
| jroovers | anteaya: sorry, btw, didn't know you could link to lines | 22:36 |
| salv-orlando | beagles: neutron api (either neutron port-list or GET /v2.0/ports.jsom | 22:36 |
| beagles | salv-orlando: interesting | 22:36 |
| jroovers | anteaya: so is the thought process that because I did not make any changes to keystone, it must be a issue with devstack or grenade? | 22:36 |
| salv-orlando | there was a patch a while ago but I do not remember why we did not approve it | 22:36 |
| salv-orlando | it did regex matching | 22:37 |
| beagles | salv-orlando: that would at least explain why it wouldn't work | 22:37 |
| anteaya | jroovers: many people don't know that, which is why I wanted to take you through it and show you a few things | 22:37 |
| anteaya | jroovers: that is certainly a possbility | 22:38 |
| anteaya | it was an upgrade command was it not? | 22:38 |
| jroovers | anteaya: yes | 22:38 |
| anteaya | so the error could be in one of our repos or mirrors failing to have the package available when devstack was looking for it | 22:38 |
| anteaya | that is possible as well | 22:39 |
| beagles | salv-orlando: so.. mmm that's a parity issue then | 22:39 |
| *** yamahata has quit IRC | 22:39 | |
| anteaya | jroovers: so the first step is find the error | 22:40 |
| jroovers | anteaya: well it says that keystone did not start http://logs.openstack.org/67/62867/1/check/check-grenade-dsvm/b0941ce/console.html#_2013-12-18_13_16_25_580 | 22:40 |
| anteaya | I'm trying to get in the habit of finding the error and using the timestamp in the url to get a useful link to use in my comment on the patch | 22:40 |
| beagles | salv-orlando: the alternative would be to list all ports available to the integration library and filter them there | 22:40 |
| anteaya | so I can annotate my detective work on my own test failures | 22:41 |
| beagles | salv-orlando: which wouldn't be the end of the world | 22:41 |
| jroovers | anteaya: that is good advice, I'll starting doing that as well :-) | 22:41 |
| anteaya | it helps another to point out what is obvious might you might have overlooked | 22:41 |
| anteaya | then I write my assessment | 22:42 |
| anteaya | something like this is a failure to upgrade and start keystone, this patch contains code to <purpose of patch> which does not affect starting keystone | 22:43 |
| anteaya | then I would see if I can find a bug in devstack or grenade or keystone that matches | 22:43 |
| mlavalle | jroovers: the big picture is the following: each one of those gate jobs creates a devstack instance and the runsn against that devstack instance a battery of tempest tests, including your patchset. So he console log, from a big picture perpective, has two sections. The first one is the creation of the devstack isntance. The second one is the execution of the tempest tests. In the case we are talking about, it didn't finish th | 22:43 |
| mlavalle | devstack instance creation. Never got to run the tempest tests with your patchset | 22:43 |
| jroovers | mlavalle: makes sense now, thanks | 22:44 |
| mlavalle | jroovers: you will find useful going to the console log of one of the jobs that returned green in that patch submission. You will see the two sections | 22:45 |
| anteaya | if you find a bug, add to the bug report if you can and use recheck bug <bug number> | 22:45 |
| anteaya | if you can't find it, file a bug and then use your new bug number in the recheck | 22:45 |
| anteaya | recheck for the check queue | 22:45 |
| anteaya | reverify for the gate queue (for patches that have been approved for merging) | 22:45 |
| mlavalle | jroovers: here's the interesting case: a gate job that succesfully built the devstack instance, executed the tempst tests against it and one or several of the tests failed. In that case, the job will be also red in gerrit. And you should see what tempest tests failed and how they are related to your patch | 22:47 |
| mlavalle | makes sense? | 22:48 |
| jroovers | mlavalle, yes. And I think I'm experiencing that now http://logs.openstack.org/67/62867/1/check/check-tempest-dsvm-full/853cd04/console.html#_2013-12-18_22_13_13_977 | 22:48 |
| jroovers | mlavalle, several compute tests failing | 22:49 |
| mlavalle | jroovers: ok try to debug it. As I said, I am doing something else right now. But I'll get again to it later tonight. | 22:50 |
| *** markwash has quit IRC | 22:51 | |
| mlavalle | i'll leave comments in gerrit for you so you can see them tomorrow morning, your time | 22:51 |
| jroovers | mlavalle, yes, but I'm going to rebase based on enikanorov's comments first. Thanks for the help. I'll probably won't be around later as it is getting late here | 22:51 |
| jroovers | mlavalle: much appreciated :-) | 22:51 |
| mlavalle | I know, you are 7 / 8 hours ahead of me | 22:51 |
| mlavalle | :-) | 22:52 |
| jroovers | mlavalle: approaching midnight here :-) | 22:52 |
| mlavalle | so go have a beer | 22:52 |
| jroovers | mlavalle: will be doing that right now :-) | 22:52 |
| jroovers | thanks again | 22:52 |
| jroovers | anteaya, thanks to you too | 22:53 |
| *** ameade has joined #openstack-neutron | 22:53 | |
| jroovers | anteaya, mlavalle, you're my heroes! | 22:53 |
| * jroovers starts saving to buy beers for anteaya and mlavalle :-) | 22:53 | |
| mlavalle | anteaya: I will add a "wrestling with the gate" section to that wiki page I put together for new developers | 22:54 |
| mlavalle | over the wekend | 22:54 |
| anteaya | mlavalle: sounds like a great addition | 22:54 |
| anteaya | thanks mlavalle | 22:55 |
| anteaya | jroovers: looks like you are in a good spot | 22:55 |
| anteaya | jroovers: keep us updated on your progress | 22:55 |
| anteaya | and be sure to ask questions | 22:55 |
| marun | salv-orlando: so, I have a dumb discovery | 22:55 |
| jroovers | anteaya: will do, thanks! | 22:55 |
| *** roeyc has quit IRC | 22:56 | |
| *** dims_ has quit IRC | 22:57 | |
| *** markwash has joined #openstack-neutron | 22:59 | |
| *** gdubreui has joined #openstack-neutron | 23:00 | |
| beagles | mlavalle anteaya: do either of you have anything to share with relevance to parity? | 23:05 |
| mlavalle | beagles: not right now | 23:05 |
| beagles | mlavalle: if I gave you a list of the neutron methods called by way of the nova/neutron integration library, could you cross reference it with your API coverage gap analysis? | 23:06 |
| mlavalle | beagles: don't know but it doesn't hurt to try…. send it over. Most likley< I will try to do that over the weekend | 23:07 |
| anteaya | beagles: if you do it on an etherpad we can all look | 23:08 |
| mlavalle | beagles: I kind of suspect where you are trying to go | 23:08 |
| beagles | mlavalle: ack.. I think I can pull that out pretty quickly tomorrow a.m. iirc you are west coast.. I think I'm probably something like 5 hours behind | 23:08 |
| beagles | or ahead sorry | 23:08 |
| anteaya | and I have no valuable info to share with relevance to parity, no | 23:08 |
| mlavalle | I'm us central | 23:09 |
| beagles | okay.. not so much.. but a few hours :) | 23:09 |
| mlavalle | Texas, to be precise | 23:09 |
| beagles | nice! | 23:09 |
| *** SushilKM has quit IRC | 23:09 | |
| beagles | and anteaya, yeah.. it'll go into an etherpad to start at least | 23:10 |
| beagles | gets it out there | 23:10 |
| anteaya | awesome thank you | 23:12 |
| openstackgerrit | Carl Baldwin proposed a change to openstack/neutron: Use information from the dnsmasq hosts file to call dhcp_release https://review.openstack.org/56263 | 23:12 |
| anteaya | <-- dinner | 23:12 |
| beagles | I think we mentioned that tomorrow wasn't going to work for syncing up at 22:00 UTC | 23:12 |
| beagles | I'll be in contact though during the day so we'll at least have some idea of what would be said at that time anyways | 23:12 |
| *** dims_ has joined #openstack-neutron | 23:13 | |
| beagles | if 22:00 on Friday doesn't work for anybody, we should try to grab each other sometime earlier if possible | 23:14 |
| *** markwash has quit IRC | 23:14 | |
| beagles | if I can come up with some concrete work items, I might be able to scrounge some bodies to work on them | 23:14 |
| *** gdubreui has quit IRC | 23:16 | |
| beagles | anyways... I'm off to do domestic things for the rest of the evening and maybe wrap some gifts... till tomorrow | 23:16 |
| *** banix has quit IRC | 23:17 | |
| *** carl_baldwin has quit IRC | 23:19 | |
| *** hichihara has joined #openstack-neutron | 23:21 | |
| *** hichihara has quit IRC | 23:22 | |
| anteaya | beagles: sounds good to me | 23:24 |
| *** jorisroovers has joined #openstack-neutron | 23:26 | |
| *** jroovers has quit IRC | 23:26 | |
| openstackgerrit | Micheal Thompson proposed a change to openstack/neutron: Fixed PEP8 import alphabetical order https://review.openstack.org/62464 | 23:28 |
| dkehn | whaaaaaa | 23:31 |
| *** layer427expert has joined #openstack-neutron | 23:31 | |
| *** layer427expert has quit IRC | 23:38 | |
| *** networkstatic is now known as networkstatic_zZ | 23:44 | |
| *** jorisroovers has quit IRC | 23:58 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!