kevinbenton | marun: ready https://review.openstack.org/#/c/80413/ | 00:01 |
---|---|---|
*** julim has quit IRC | 00:03 | |
marun | kevinbenton: ok | 00:04 |
openstackgerrit | Evgeny Fedoruk proposed a change to openstack/neutron: Add 2-leg configuration to Radware LBaaS Driver https://review.openstack.org/69009 | 00:04 |
*** ijw has joined #openstack-neutron | 00:05 | |
*** julim has joined #openstack-neutron | 00:05 | |
kfox1111 | so, neutron-server is stuck, but its not doing anything with mysql... | 00:15 |
*** hogepodge has quit IRC | 00:15 | |
kevinbenton | kfox1111: is it logging anything? | 00:16 |
kfox1111 | no. I don't have debug level logging turned on though. | 00:18 |
kfox1111 | Would the message its choking on still be in the qpid queue or does it take it off before doing it? | 00:20 |
*** ijw has quit IRC | 00:21 | |
*** zhipeng has quit IRC | 00:21 | |
*** suresh12 has quit IRC | 00:21 | |
kevinbenton | kfox1111: i think it will have taken it off | 00:21 |
kevinbenton | kfox1111: what operation were you doing when it happened? | 00:22 |
kfox1111 | hmm... thats why it probably recovers right after being restarted... | 00:22 |
*** zhipeng has joined #openstack-neutron | 00:22 | |
kfox1111 | no clue. big enough cloud that lots of things are happening. | 00:22 |
kfox1111 | Its happening randomly between once and 3 times a day now. :/ | 00:22 |
kevinbenton | kfox1111: hmmm. is it the reference plugin? | 00:22 |
kfox1111 | core_plugin = neutron.plugins.linuxbridge.lb_neutron_plugin.LinuxBridgePluginV2 | 00:23 |
kfox1111 | rpc_backend = neutron.openstack.common.rpc.impl_qpid | 00:23 |
kfox1111 | havana from rdo. | 00:23 |
*** ijw has joined #openstack-neutron | 00:23 | |
*** hogepodge has joined #openstack-neutron | 00:24 | |
kevinbenton | is turning on debugging an option? it sounds like something is causing a deadlock | 00:24 |
kevinbenton | it would be useful to have the log of what it was doing last when it froze up | 00:25 |
*** hogepodge has quit IRC | 00:25 | |
*** baoli has quit IRC | 00:25 | |
kfox1111 | yeah. I will probably turn up the logs tomorrow morning. :/ | 00:25 |
*** baoli has joined #openstack-neutron | 00:26 | |
*** thuc has quit IRC | 00:26 | |
kevinbenton | kfox1111: sorry i couldn't be of more help without it. a few to many unknowns | 00:26 |
*** thuc has joined #openstack-neutron | 00:26 | |
kfox1111 | hmm... and now the linuxbridge agent has 72.6k messages stuck in its queue. :/ | 00:26 |
*** xianghui has joined #openstack-neutron | 00:26 | |
kevinbenton | is it slowly draining them? | 00:27 |
*** BuSerD has quit IRC | 00:27 | |
kfox1111 | nope. stuck. | 00:27 |
kevinbenton | what message is at the head of the queue? | 00:27 |
kfox1111 | ok. just restarted it... seems happier now... | 00:27 |
kfox1111 | gota recreate my stacks now. :/ | 00:28 |
kevinbenton | i wonder if there is some malformed message that blocks up the queue processor | 00:28 |
*** suresh12 has joined #openstack-neutron | 00:28 | |
*** matsuhashi has joined #openstack-neutron | 00:29 | |
kfox1111 | could be. I straced the neutron-server process but it seemed to be pretty idle. | 00:29 |
*** zhipeng has quit IRC | 00:30 | |
marun | kevinbenton: fyi the change is being bumped to 2nd in the queue | 00:30 |
kevinbenton | marun: awesome. like a fast-pass at six flags.... only for a patch | 00:31 |
*** thuc has quit IRC | 00:31 | |
kevinbenton | kfox1111: well once you get some debug logs, feel free to ping me tomorrow | 00:31 |
*** ijw has quit IRC | 00:31 | |
kfox1111 | k. thanks. | 00:32 |
kfox1111 | hmm.. the bridge agent queue seems to be filling up again... | 00:32 |
kfox1111 | I'm seeing lots of: | 00:32 |
kfox1111 | 2014-03-19 17:32:36.401 41084 INFO neutron.agent.securitygroups_rpc [-] Refresh firewall rules | 00:32 |
kfox1111 | in the logs. | 00:32 |
*** manishg has quit IRC | 00:33 | |
kfox1111 | I'm seeing it drail a little and add more. slowly increasing. :/ | 00:34 |
kfox1111 | 327... | 00:34 |
kfox1111 | 335... | 00:34 |
*** salv-orlando has joined #openstack-neutron | 00:35 | |
kfox1111 | 357... | 00:35 |
kfox1111 | This seems to be a problem. Is there a way to tell where the messages came from? | 00:35 |
kfox1111 | I'm guessing they are a ll firewall refresh maybe. | 00:35 |
*** ijw has joined #openstack-neutron | 00:35 | |
kevinbenton | i don't think you should be getting that many messages to refresh the firewall unless a port change occurs or security group change occurs | 00:36 |
kfox1111 | yeah, I don't think there are too many of those... But it looks like its trying over and over to refresh, and getting backlogged. | 00:37 |
*** dguitarbite_ has quit IRC | 00:38 | |
kfox1111 | 536. 534.. 541... | 00:38 |
kevinbenton | kfox1111: is there an easy way to look at the current messages in the queues? | 00:38 |
kevinbenton | with quid? | 00:39 |
kevinbenton | qpid* | 00:39 |
kfox1111 | not sure.... | 00:39 |
*** dguitarbite has joined #openstack-neutron | 00:40 | |
kevinbenton | kfox1111: i have to head out. i'll be back around tomorrow | 00:41 |
kfox1111 | k. me too. thanks for the help. | 00:41 |
*** suresh12 has quit IRC | 00:42 | |
*** devlaps has quit IRC | 00:47 | |
*** suresh12 has joined #openstack-neutron | 00:47 | |
*** yamahata has joined #openstack-neutron | 00:48 | |
*** kfox1111 has quit IRC | 00:50 | |
*** ijw has quit IRC | 00:52 | |
*** ijw has joined #openstack-neutron | 00:52 | |
*** ijw_ has joined #openstack-neutron | 00:55 | |
salv-orlando | marun: you still around? | 00:56 |
salv-orlando | So I've looked you amended the patch for the semaphore to accommodate rkukura complains about that patch interfering with his work… but does moving the semaphore within a function called by get_device_details solve the problem? | 00:57 |
*** ijw has quit IRC | 00:57 | |
salv-orlando | kevinbenton: you've updated the patch but not the commit message. Now we have a commit message that does something and a patch that does something else... | 00:58 |
salv-orlando | and most importantly something I forgot to ask earlier… why does get_device_details update a port a status?????? | 00:59 |
marun | salv-orlando: I was asking the same thing… it feels wrong and and rkukura is proposing to do yet more of that (wrt binding) | 00:59 |
salv-orlando | yeah I don't want to comment on that work before I see the code however | 01:00 |
salv-orlando | I don't think we have a spec or whatever to discuss | 01:00 |
marun | salv-orlando: my fault on not forcing an update to the commit message. | 01:00 |
marun | salv-orlando: do you want to stop the gate? | 01:01 |
salv-orlando | marun… if we can get kevinbenton to do that we can both reapprove | 01:01 |
kevinbenton | salv-orlando: i can update it | 01:01 |
marun | salv-orlando: we pushed it to the top of the queue so it might be running already | 01:01 |
*** armax has left #openstack-neutron | 01:01 | |
*** alagalah has quit IRC | 01:02 | |
salv-orlando | well, just remember to not go back to that change and say "WTF" when you see the change does not do what the commit message says | 01:02 |
marun | salv-orlando: it was bob's comment that prompted moving the lock inside the method, so that it wouldn't be held indefinitely in the case of the change he's working on. | 01:02 |
marun | salv-orlando: yeah, I need to be nicer about wtf's in general :( | 01:02 |
marun | it'll bite me in the a** at some point | 01:02 |
salv-orlando | marun: I'd rather get a genuine wtf rather than a fake "thanks for submitting this" | 01:03 |
marun | salv-orlando: so, do you want to stop the gate? | 01:03 |
salv-orlando | anyway… a change which alters again binding behaviour in ml2, and promotes a loop in get_device_details as suggested by bob at first glance sounds like something we should not merge so late in the release cycle. | 01:03 |
kevinbenton | actually, what will happen if i update the commit message now? infra recognizes a trivial update | 01:03 |
salv-orlando | kevinbenton: the patch will be bounced anyway out of the gate | 01:04 |
marun | salv-orlando: I would tend to agree. | 01:04 |
salv-orlando | marun: anyway, nothing can be said so far… I guess | 01:04 |
kevinbenton | salv-orlando: shall i do it for the sanctity of good commit messages? | 01:04 |
marun | salv-orlando: if we don't have a spec or patch in review by now, i don't know how it could possibly go into icehouse | 01:04 |
marun | kevinbenton: *sigh* I guess yeah, do it. | 01:05 |
salv-orlando | kevinbenton: it's not pedantry it's just that some developers like me look at commit message history to understand what's going on | 01:05 |
salv-orlando | we can get infra people to re-promote it to the top of the gate queue | 01:05 |
salv-orlando | or who know zuul might be so kind to keep in the top | 01:05 |
marun | kevinbenton: my only relunctance is that it was my lack of care that approved it and i'll have to admit as much to ingra | 01:05 |
kevinbenton | salv-orlando: i didn't say it was pedantry :-) | 01:05 |
marun | kevinbenton: so, do it. | 01:05 |
*** leseb has joined #openstack-neutron | 01:05 | |
marun | (arg, need to take a break, spalling errers everiwhare | 01:06 |
salv-orlando | marun: but at the end of the day, did you figured out why get_device_details updates the port status? | 01:06 |
marun | salv-orlando: no | 01:07 |
marun | salv-orlando: I was complaining to rkukura that modifying things in something that looks like an accessor violates a fundamental tenant of oo design | 01:07 |
marun | salv-orlando: but that conversation never came to a resolution | 01:07 |
salv-orlando | marun: is that really a need for a resolution? | 01:08 |
marun | salv-orlando: instead i asked him to ensure that I was a reviewer on his new patch | 01:08 |
marun | salv-orlando: well, I mean that the conversation was derailed about just getting something working. | 01:09 |
salv-orlando | marun: I also wondered how moving the lock into a method called by get_device_details rather than in get_device_details itself would solve the problem he had with semaphore not allowing his patch to work. But I guess we need to wait and see. | 01:09 |
*** leseb has quit IRC | 01:10 | |
openstackgerrit | Kevin Benton proposed a change to openstack/neutron: Add a semaphore to some ML2 operations https://review.openstack.org/80413 | 01:10 |
kevinbenton | salv-orlando: so i think he was concerned that get_device_details could have long calls in it depending on the drivers or something along those lines | 01:11 |
kevinbenton | salv-orlando: so he wanted to avoid getting the semaphore until as late as possible | 01:11 |
marun | salv-orlando: so, updated patch submitted. | 01:11 |
salv-orlando | thanks | 01:11 |
marun | salv-orlando: do you want to re-approve or are you sufficiently concerned that you don't want it to merge as is? | 01:12 |
salv-orlando | marun, kevinbenton - I've give my +2 | 01:12 |
marun | salv-orlando: ok. | 01:12 |
marun | salv-orlando: we can always reverse if needed. | 01:12 |
salv-orlando | I went over the code and get_device_details should not hit the DB lock until the semaphore is hit | 01:12 |
*** _cjones_ has quit IRC | 01:13 | |
salv-orlando | when the semaphore is hit, it should yield again, and then it should eventually allow delete_port to complete and release the lock | 01:13 |
kevinbenton | salv-orlando: yes, if it can't get the sem, it will yield | 01:14 |
marun | salv-orlando: am I supposed to wait until checks pass before I approved or does that happen automatically now? | 01:14 |
kevinbenton | salv-orlando: i believe that's automatic. gate won't start until checks pass | 01:15 |
salv-orlando | marun: you should wait for checks or tell infra we've just updated the commit message so they won't yell at you | 01:15 |
salv-orlando | the funny thing is that delete_port is doing select .. for update before deleting in order to avoid postgres errors with read committed isolation levels | 01:15 |
marun | salv-orlando: ok, i should unapprove then? | 01:15 |
salv-orlando | marun: even if you approved as kevinbenton said, it's now going to the check queue anyway | 01:15 |
marun | that's an unfortunate kind of funnhy | 01:15 |
marun | salv-orlando: Can I leave it to you to get infra to push it to the top of queue when the checks pass? | 01:16 |
salv-orlando | marun: I would love to but it's 1.20 AM my time and this might be slip in the morning | 01:17 |
marun | salv-orlando: ah, ok. | 01:17 |
marun | salv-orlando: I'll keep responsibility then. | 01:17 |
salv-orlando | ok thanks | 01:17 |
salv-orlando | anyway, we basically have a situation where we either screw postgres or mysql. Something not right somewhere ;) | 01:17 |
kevinbenton | marun: i can tell them when it's ready if you want since i watch the jenkins notifications | 01:17 |
marun | kevinbenton: much appreciated | 01:18 |
marun | salv-orlando: I've been slowly trying to educate myself about these locking/isolation issues. I'm finding it a slog. | 01:18 |
marun | salv-orlando: and eventlet is not helping | 01:19 |
marun | salv-orlando: talking with comstud about the issue at length I'm going to work on a transaction wrapper that can detect when a yield is happening so that we can track them down. | 01:19 |
marun | salv-orlando: It's unlikely that this type of problem won't recur in different places in the future. | 01:20 |
salv-orlando | marun: sure I already know about 3 more places | 01:20 |
salv-orlando | marun: one is the place you fixed | 01:20 |
salv-orlando | another is update vips | 01:20 |
marun | salv-orlando: ah right, the vips | 01:20 |
marun | salv-orlando: I was seeing that in the logstash query | 01:21 |
jaypipes | sbalukoff: done with your audit, eh? ;) | 01:21 |
salv-orlando | and the last one is written on a piece of paper I have in my bag, and I don't remember now | 01:21 |
kevinbenton | salv-orlando: create_port .... | 01:21 |
salv-orlando | marun: interestingly they're unlikely to be critical | 01:21 |
salv-orlando | nope kevinbenton, it shows up during a SELECT on ports | 01:21 |
salv-orlando | but if you've seen them in create_port too | 01:21 |
salv-orlando | that's another place to look for | 01:21 |
marun | salv-orlando: anything that touches ports can be affected by a SELECT FOR UPDATE | 01:22 |
salv-orlando | marun: still we don't want lock wait timeouts ever | 01:22 |
marun | salv-orlando: preventing yields should hold us until we have a better solution | 01:22 |
marun | salv-orlando: from what i hear asyncio holds promise for python3, and there is a backport to python2 | 01:22 |
marun | salv-orlando: it would require a rewrite, though. | 01:22 |
salv-orlando | marun: right. At the end of the day it's not a db programming problem. It's an eventlet issue. | 01:23 |
kevinbenton | salv-orlando: in create_port the deadlock is on ip allocation | 01:23 |
marun | salv-orlando: it's explicit, not implicit. | 01:23 |
salv-orlando | kevinbenton: marun sorted that out because it was deadlock between allocate ip and deallocate ip | 01:23 |
kevinbenton | salv-orlando: see http://paste.openstack.org/show/68417/ | 01:23 |
marun | salv-orlando: agreed | 01:23 |
kevinbenton | salv-orlando: oh okay | 01:23 |
salv-orlando | kevinbenton: well… unless they keep showing up of course | 01:24 |
salv-orlando | marun: so replacing eventlet should be on the agenda, but it does not just depend on neutron alone. | 01:24 |
marun | kevinbenton: that is from feb 21. | 01:24 |
kevinbenton | salv-orlando: that's an old one from the bug report so it could be fixed | 01:24 |
salv-orlando | I think oslo.rpc too should move way | 01:24 |
marun | salv-orlando: agreed | 01:24 |
salv-orlando | and the wsgi framework | 01:24 |
marun | kevinbenton: https://review.openstack.org/#/c/81196/ | 01:25 |
marun | kevinbenton: that merged today, should fix that contention due to removal of SELECT FOR UPDATE on IPAllocation | 01:25 |
kevinbenton | marun: excellent | 01:25 |
marun | salv-orlando: I hope there's discussion at the summit about this. | 01:26 |
kevinbenton | marun: that would be good i think | 01:27 |
marun | salv-orlando: I think asyncio would be a lot of work, but it would provide forward-looking compatibility for python3 and a switch to explicit concurrency would go a long way towards preventing people from doing silly things. | 01:27 |
salv-orlando | I actually hope the discussion happens before the summit, so that people will be able to assign actionable items ther | 01:27 |
salv-orlando | e | 01:27 |
kevinbenton | it's not very intuitive what causes the problem | 01:27 |
marun | salv-orlando: agreed | 01:27 |
*** networkstatic has quit IRC | 01:27 | |
*** alexpilotti_ has joined #openstack-neutron | 01:28 | |
HenryG | salv-orlando: sorry to interrupt, question regarding https://review.openstack.org/40296 | 01:28 |
HenryG | I have rebased and am testing | 01:28 |
salv-orlando | HenryG: listening | 01:28 |
HenryG | Getting a lot of "OperationalError: no such table" | 01:29 |
salv-orlando | marun: are already aware of this: https://www.mail-archive.com/openstack-dev@lists.openstack.org/msg15979.html ? | 01:29 |
HenryG | How should the unit tests be loading the db/tables now? | 01:29 |
salv-orlando | HenryG: I think… if my memory does not fail me... | 01:30 |
*** alexpilotti has quit IRC | 01:30 | |
*** alexpilotti_ is now known as alexpilotti | 01:30 | |
salv-orlando | that they should be now inheriting from this class: https://review.openstack.org/#/c/40296/6/neutron/tests/unit/testlib_api.py | 01:30 |
salv-orlando | SqlTestCase | 01:30 |
HenryG | salv-orlando: thanks, will look into that | 01:31 |
salv-orlando | and if they call register_models, they should not do that anymore | 01:31 |
marun | salv-orlando: hmmm, couldn't we use transactions to empty the tables rather than dropping/creating on every test? | 01:32 |
*** alagalah has joined #openstack-neutron | 01:32 | |
*** samuelbercovici has quit IRC | 01:32 | |
*** jobewan has joined #openstack-neutron | 01:33 | |
marun | salv-orlando: also, iirc, we're not using in-memory db for unit tests | 01:34 |
marun | salv-orlando: or maybe I've missed something. | 01:34 |
marun | salv-orlando: where is the db url defined for the tests? | 01:34 |
marun | salv-orlando: is it in tests/etc/ ? | 01:34 |
*** sbalukoff has quit IRC | 01:34 | |
salv-orlando | marun, it used to use 'sqlite://' but honestly I have not looked at that in ages | 01:35 |
salv-orlando | we had initially code that emptied tables, I don't remember why it got changed | 01:35 |
salv-orlando | I do apologise, but I'm going to disconnect now | 01:35 |
HenryG | It is still 'sqlite://', in neutron/tests/etc/neutron.conf.test | 01:35 |
salv-orlando | good night ;) | 01:35 |
*** salv-orlando has quit IRC | 01:35 | |
HenryG | salv-orlando: you shouldn't be up at this hour anyway | 01:36 |
marun | HenryG: sqlite:// is not in memory, btw | 01:36 |
marun | HenryG: that would be sqlite:///:memory: | 01:37 |
*** thuc has joined #openstack-neutron | 01:37 | |
HenryG | marun: I understand, I was just confirming the current status of things | 01:37 |
marun | HenryG: ah, sorry | 01:37 |
openstackgerrit | Lee Li proposed a change to openstack/neutron: NsxApiException should return str https://review.openstack.org/81685 | 01:38 |
*** xuhanp has joined #openstack-neutron | 01:39 | |
HenryG | marun: Items to add to your refactor etherpad: empty tables by transactions; use in-memory DB ? | 01:39 |
*** SumitNaiksatam has quit IRC | 01:39 | |
*** xianghui has quit IRC | 01:39 | |
*** xianghui has joined #openstack-neutron | 01:39 | |
marun | HenryG: I think the former is worth doing, but the latter I'm not sure about. | 01:39 |
marun | HenryG: I think switching from in-memory db was due to memory issues caused by it. | 01:39 |
marun | HenryG: On second thought, it should be there so that we can at least investigate. | 01:40 |
marun | HenryG: I'm afraid I have little progress beyond getting the functional job fixed. | 01:40 |
marun | HenryG: working on bugs has seemed more important, but that's a rational that could keep me from doing anything else! | 01:40 |
*** alagalah has quit IRC | 01:41 | |
HenryG | marun: np. I am also caught up in bugs right now. | 01:41 |
*** thuc has quit IRC | 01:42 | |
marun | HenryG: thanks for the pointers, I've updated the etherpad. | 01:42 |
HenryG | marun: But I will start asking annoying questions sooner or later :) | 01:42 |
HenryG | Yesterday I set up floobits. A pretty cool thingy. | 01:43 |
*** zhipeng has joined #openstack-neutron | 01:43 | |
*** alexpilotti has quit IRC | 01:44 | |
*** dvorkinista has quit IRC | 01:44 | |
marun | HenryG: I'm going to try to do a walkthrough for a bunch of folks at once using floobits. | 01:44 |
marun | HenryG: want to try out the terminal sharing for that | 01:45 |
marun | HenryG: I'm hoping it will prove better that screen sharing for tiny text. | 01:45 |
HenryG | marun: sounds like a good idea. Let me know if you need a guinea pig | 01:46 |
*** sfox has joined #openstack-neutron | 01:46 | |
marun | HenryG: the cool thing about floobits is you can be the guinea pig yourself.. | 01:46 |
marun | log in via 2 different browsers, or a browser and an editor, viola! | 01:47 |
marun | HenryG: when you find yourself wanting to work on code, though, i'm happy to make time. | 01:47 |
HenryG | marun: Great, thanks. The first thing I want to understand is your retargetable api test. Although, I guess maybe step back and get an overview of the concept and goals first. | 01:50 |
*** ijw_ has quit IRC | 01:51 | |
*** manishg has joined #openstack-neutron | 01:53 | |
*** chandan_kumar has joined #openstack-neutron | 01:53 | |
*** manishg_ has joined #openstack-neutron | 01:56 | |
*** manishg has quit IRC | 01:58 | |
*** shakamunyi has joined #openstack-neutron | 02:00 | |
marun | would you like to schedule a time? | 02:01 |
*** manishg_ has quit IRC | 02:02 | |
marun | HenryG: ^ | 02:03 |
* HenryG looks at his schedule ... | 02:04 | |
*** ijw has joined #openstack-neutron | 02:04 | |
HenryG | marun: tomorrow is a bit full until about 4:30 pm eastern. Friday is wide open. | 02:07 |
marun | HenryG: Friday works better for me too | 02:08 |
marun | HenryG: I can be available between noon and 2:30 eastern | 02:09 |
HenryG | marun: great, let's do 1:00 eastern | 02:09 |
marun | HenryG: Ok, sounds good. | 02:10 |
*** ijw has quit IRC | 02:11 | |
*** thuc has joined #openstack-neutron | 02:12 | |
*** alagalah has joined #openstack-neutron | 02:12 | |
*** alagalah has quit IRC | 02:17 | |
*** SumitNaiksatam has joined #openstack-neutron | 02:22 | |
*** thuc_ has joined #openstack-neutron | 02:25 | |
*** dvorkinista has joined #openstack-neutron | 02:25 | |
*** thuc has quit IRC | 02:28 | |
openstackgerrit | jun xie proposed a change to openstack/neutron: Use different name for the same constraint https://review.openstack.org/81488 | 02:29 |
*** thuc_ has quit IRC | 02:31 | |
*** thuc has joined #openstack-neutron | 02:32 | |
*** banix has joined #openstack-neutron | 02:33 | |
openstackgerrit | A change was merged to openstack/neutron: Show neutron API request body with debug enabled https://review.openstack.org/81109 | 02:34 |
*** networkstatic has joined #openstack-neutron | 02:34 | |
*** networkstatic has quit IRC | 02:35 | |
*** mwagner__ is now known as mwagner__zzzz | 02:37 | |
*** sfox has quit IRC | 02:40 | |
*** dvorkinista has quit IRC | 02:45 | |
openstackgerrit | Isaku Yamahata proposed a change to openstack/neutron: tests/unit: refactor reading neutron.conf.test https://review.openstack.org/76418 | 02:48 |
openstackgerrit | Isaku Yamahata proposed a change to openstack/neutron: tests/unit: clean up notification driver https://review.openstack.org/76139 | 02:48 |
*** dvorkinista has joined #openstack-neutron | 02:48 | |
*** tomoe_ has joined #openstack-neutron | 02:50 | |
*** dvorkinista has quit IRC | 02:51 | |
*** dvorkinista has joined #openstack-neutron | 02:53 | |
*** dvorkinista has quit IRC | 02:54 | |
*** suresh12 has quit IRC | 02:54 | |
*** thuc has quit IRC | 02:59 | |
*** thuc has joined #openstack-neutron | 03:00 | |
*** tomoe_ has quit IRC | 03:00 | |
*** thuc has quit IRC | 03:04 | |
*** xianghui has quit IRC | 03:04 | |
*** tomoe_ has joined #openstack-neutron | 03:05 | |
*** sfox has joined #openstack-neutron | 03:18 | |
*** chandan_kumar has quit IRC | 03:22 | |
*** harlowja is now known as harlowja_away | 03:23 | |
*** thuc has joined #openstack-neutron | 03:24 | |
*** dvorkinista has joined #openstack-neutron | 03:25 | |
*** sbalukoff has joined #openstack-neutron | 03:26 | |
*** dvorkinista has quit IRC | 03:27 | |
*** dvorkinista has joined #openstack-neutron | 03:28 | |
*** tomoe_ has quit IRC | 03:29 | |
*** jecarey has joined #openstack-neutron | 03:30 | |
*** dvorkinista has quit IRC | 03:33 | |
openstackgerrit | Henry Gessau proposed a change to openstack/neutron: Remove auto-generation of db schema from models at startup https://review.openstack.org/40296 | 03:45 |
*** matsuhashi has quit IRC | 03:47 | |
openstackgerrit | Henry Gessau proposed a change to openstack/neutron: Remove auto-generation of db schema from models at startup https://review.openstack.org/40296 | 03:57 |
*** arosen1 has quit IRC | 04:02 | |
*** tomoe_ has joined #openstack-neutron | 04:08 | |
*** saju_m has joined #openstack-neutron | 04:12 | |
*** banix has quit IRC | 04:12 | |
*** Akshik has joined #openstack-neutron | 04:12 | |
*** tomoe_ has quit IRC | 04:19 | |
*** tomoe_ has joined #openstack-neutron | 04:19 | |
*** thuc has quit IRC | 04:20 | |
*** thuc has joined #openstack-neutron | 04:21 | |
*** sweston has joined #openstack-neutron | 04:21 | |
*** tomoe_ has quit IRC | 04:24 | |
*** thuc has quit IRC | 04:25 | |
*** matsuhashi has joined #openstack-neutron | 04:31 | |
openstackgerrit | Yuuichi Fujioka proposed a change to openstack/neutron: change to allow setting ip address of floating ip https://review.openstack.org/70286 | 04:36 |
openstackgerrit | A change was merged to openstack/neutron: Imported Translations from Transifex https://review.openstack.org/81185 | 04:38 |
openstackgerrit | A change was merged to openstack/neutron: Change report_interval from 4 to 30, agent_down_time from 9 to 75 https://review.openstack.org/80829 | 04:39 |
*** yfujioka has joined #openstack-neutron | 04:40 | |
*** thuc has joined #openstack-neutron | 04:41 | |
*** thuc has quit IRC | 04:49 | |
*** xianghui has joined #openstack-neutron | 04:49 | |
openstackgerrit | Henry Gessau proposed a change to openstack/neutron: Remove auto-generation of db schema from models at startup https://review.openstack.org/40296 | 04:52 |
*** Akshik_ has joined #openstack-neutron | 04:56 | |
*** Akshik has quit IRC | 05:00 | |
*** yfujioka has quit IRC | 05:02 | |
*** carl_baldwin has joined #openstack-neutron | 05:08 | |
*** chandan_kumar has joined #openstack-neutron | 05:10 | |
*** suresh12 has joined #openstack-neutron | 05:11 | |
*** carl_baldwin has quit IRC | 05:17 | |
*** chandan_kumar has quit IRC | 05:17 | |
*** sridhar has joined #openstack-neutron | 05:17 | |
*** yongli has joined #openstack-neutron | 05:19 | |
openstackgerrit | A change was merged to openstack/neutron: Don't use root to list namespaces https://review.openstack.org/81098 | 05:20 |
*** pradipta_away is now known as pradipta | 05:25 | |
*** amotoki has joined #openstack-neutron | 05:29 | |
*** tomoe_ has joined #openstack-neutron | 05:31 | |
oda-g | amotoki: ping | 05:31 |
*** tomoe_ has joined #openstack-neutron | 05:32 | |
*** vkozhukalov_ has joined #openstack-neutron | 05:33 | |
*** tomoe_ has quit IRC | 05:41 | |
amotoki | oda-g: pong | 05:42 |
oda-g | amotoki: Would you have a look at https://review.openstack.org/#/c/79858/ ? I clarified the coverage of the patch in the commit message. | 05:42 |
amotoki | oda-g: will look it later. I am now discussing in another channel. | 05:44 |
oda-g | amotoki: I see. | 05:44 |
*** irenab has joined #openstack-neutron | 05:46 | |
*** gdubreui has quit IRC | 05:47 | |
*** tomoe_ has joined #openstack-neutron | 05:56 | |
*** thuc has joined #openstack-neutron | 05:57 | |
*** networkstatic has joined #openstack-neutron | 05:57 | |
*** thuc has quit IRC | 06:02 | |
*** saju_m has quit IRC | 06:02 | |
*** nati_ueno has joined #openstack-neutron | 06:03 | |
*** saju_m has joined #openstack-neutron | 06:04 | |
oda-g | nati_ueno: ping | 06:04 |
openstackgerrit | Kevin Benton proposed a change to openstack/neutron: Add decorator to prevent eventlet from yielding https://review.openstack.org/81709 | 06:04 |
*** saju_m has quit IRC | 06:07 | |
*** saju_m has joined #openstack-neutron | 06:08 | |
nati_ueno | oda-g: pong | 06:09 |
oda-g | nati_ueno: Would you approve https://review.openstack.org/#/c/79858/ again ? I fixed the commit message to clarify the coverage of the patch that is pointed out from Motoki-san. | 06:10 |
nati_ueno | oda-g: let me check | 06:10 |
*** tomoe_ has quit IRC | 06:13 | |
*** jlibosva has joined #openstack-neutron | 06:18 | |
*** vkozhukalov_ has quit IRC | 06:19 | |
openstackgerrit | Kevin Benton proposed a change to openstack/neutron: Add decorator to prevent eventlet from yielding https://review.openstack.org/81709 | 06:20 |
*** sridhar has quit IRC | 06:23 | |
*** sfox has quit IRC | 06:27 | |
openstackgerrit | Jenkins proposed a change to openstack/neutron: Imported Translations from Transifex https://review.openstack.org/81713 | 06:28 |
*** matsuhashi has quit IRC | 06:28 | |
*** matsuhashi has joined #openstack-neutron | 06:30 | |
*** tomoe_ has joined #openstack-neutron | 06:32 | |
*** saju_m has quit IRC | 06:35 | |
*** networkstatic has quit IRC | 06:35 | |
*** sfox has joined #openstack-neutron | 06:35 | |
*** sridhar has joined #openstack-neutron | 06:41 | |
*** irenab has quit IRC | 06:43 | |
obondarev | amotoki: ping | 06:47 |
openstackgerrit | berlin proposed a change to openstack/neutron: Fix duplicate name of NVP LBaaS objs not allowed on vShield Edge https://review.openstack.org/73905 | 06:50 |
openstackgerrit | Nachi Ueno proposed a change to openstack/neutron: Add enable_security_group option https://review.openstack.org/67281 | 06:50 |
*** nati_ueno has quit IRC | 06:51 | |
*** suresh12 has quit IRC | 06:58 | |
*** bvandenh has joined #openstack-neutron | 06:59 | |
*** oda-g has quit IRC | 07:00 | |
*** sweston has quit IRC | 07:02 | |
*** bashok has joined #openstack-neutron | 07:06 | |
*** bvandenh has quit IRC | 07:07 | |
*** tomoe_ has quit IRC | 07:11 | |
*** jlibosva has joined #openstack-neutron | 07:12 | |
*** tomoe_ has joined #openstack-neutron | 07:12 | |
*** shakamunyi has quit IRC | 07:15 | |
*** jlibosva has quit IRC | 07:16 | |
*** Akshik_ has quit IRC | 07:19 | |
*** Akshik_ has joined #openstack-neutron | 07:19 | |
*** Akshik has joined #openstack-neutron | 07:22 | |
*** Akshik_ has quit IRC | 07:22 | |
*** rotbeard has joined #openstack-neutron | 07:23 | |
*** arosen1 has joined #openstack-neutron | 07:25 | |
*** lukego has joined #openstack-neutron | 07:27 | |
*** sfox has quit IRC | 07:29 | |
*** bvandenh has joined #openstack-neutron | 07:29 | |
*** zhipeng has quit IRC | 07:36 | |
*** jobewan has quit IRC | 07:38 | |
*** nati_ueno has joined #openstack-neutron | 07:39 | |
*** shakamunyi has joined #openstack-neutron | 07:41 | |
*** arosen1 has quit IRC | 07:41 | |
*** zhipeng has joined #openstack-neutron | 07:43 | |
*** jlibosva has joined #openstack-neutron | 07:43 | |
*** ramishra has joined #openstack-neutron | 07:43 | |
*** leseb has joined #openstack-neutron | 07:44 | |
*** skraynev_afk is now known as skraynev | 07:45 | |
*** shakamunyi has quit IRC | 07:45 | |
*** irenab has joined #openstack-neutron | 07:46 | |
*** vkozhukalov_ has joined #openstack-neutron | 07:49 | |
*** mrsnivvel has joined #openstack-neutron | 07:51 | |
*** leseb has quit IRC | 07:52 | |
*** Jabadia has joined #openstack-neutron | 07:53 | |
*** saju_m has joined #openstack-neutron | 07:56 | |
*** arosen1 has joined #openstack-neutron | 08:01 | |
*** irenab has quit IRC | 08:01 | |
*** Jabadia has quit IRC | 08:05 | |
*** suresh12 has joined #openstack-neutron | 08:08 | |
*** Jabadia has joined #openstack-neutron | 08:10 | |
*** suresh12 has quit IRC | 08:12 | |
*** ajo has joined #openstack-neutron | 08:13 | |
*** Jabadia has quit IRC | 08:20 | |
*** Jabadia has joined #openstack-neutron | 08:21 | |
*** Akshik_ has joined #openstack-neutron | 08:25 | |
*** jgallard has joined #openstack-neutron | 08:28 | |
*** Akshik has quit IRC | 08:29 | |
*** jgallard has quit IRC | 08:33 | |
*** jgallard has joined #openstack-neutron | 08:33 | |
*** zhipeng has quit IRC | 08:34 | |
*** SumitNaiksatam has quit IRC | 08:35 | |
*** shakamunyi has joined #openstack-neutron | 08:42 | |
*** shakamunyi has quit IRC | 08:46 | |
*** nati_ueno has quit IRC | 08:47 | |
*** ygbo has joined #openstack-neutron | 08:48 | |
*** Akshik_ has quit IRC | 08:48 | |
*** morganfainberg is now known as morganfainberg_Z | 08:54 | |
*** ihrachys|afk is now known as ihrachys | 08:57 | |
*** leseb has joined #openstack-neutron | 08:58 | |
*** nati_ueno has joined #openstack-neutron | 08:58 | |
*** nati_uen_ has joined #openstack-neutron | 08:59 | |
*** bada_ has quit IRC | 08:59 | |
*** bada_ has joined #openstack-neutron | 09:00 | |
*** nati_ueno has quit IRC | 09:03 | |
*** dguitarbite has quit IRC | 09:04 | |
*** jp_at_hp has joined #openstack-neutron | 09:04 | |
*** Akshik has joined #openstack-neutron | 09:06 | |
*** Akshik_ has joined #openstack-neutron | 09:07 | |
*** Akshik has quit IRC | 09:11 | |
*** amuller has joined #openstack-neutron | 09:13 | |
openstackgerrit | Oleg Bondarev proposed a change to openstack/neutron: agent report_states tests should consider parallel execution https://review.openstack.org/81740 | 09:23 |
*** safchain has joined #openstack-neutron | 09:27 | |
*** ramishra has quit IRC | 09:30 | |
*** orion195 has quit IRC | 09:35 | |
*** roeyc has joined #openstack-neutron | 09:35 | |
*** safchain has quit IRC | 09:37 | |
openstackgerrit | A change was merged to openstack/neutron: Cisco plugin fails with ParseError no elem found https://review.openstack.org/81304 | 09:41 |
openstackgerrit | A change was merged to openstack/neutron: return false or true according to binding result https://review.openstack.org/81457 | 09:42 |
*** shakamunyi has joined #openstack-neutron | 09:43 | |
*** tomoe_ has quit IRC | 09:45 | |
*** arosen1 has quit IRC | 09:47 | |
*** shakamunyi has quit IRC | 09:47 | |
*** safchain has joined #openstack-neutron | 09:50 | |
*** SumitNaiksatam has joined #openstack-neutron | 09:53 | |
*** yamahata has quit IRC | 10:02 | |
*** overlayer has joined #openstack-neutron | 10:03 | |
*** djoreilly has joined #openstack-neutron | 10:04 | |
*** bada_ has quit IRC | 10:15 | |
*** rwsu has quit IRC | 10:18 | |
*** safchain has quit IRC | 10:18 | |
*** safchain has joined #openstack-neutron | 10:20 | |
*** xuhanp has quit IRC | 10:21 | |
openstackgerrit | A change was merged to openstack/neutron: NSX: Fix router-interface-delete returns 404 when router not in nsx https://review.openstack.org/80177 | 10:22 |
*** ramishra has joined #openstack-neutron | 10:25 | |
*** geekinutah has quit IRC | 10:25 | |
*** geekinutah has joined #openstack-neutron | 10:25 | |
*** pcm_ has joined #openstack-neutron | 10:29 | |
*** pcm_ has quit IRC | 10:31 | |
*** pcm_ has joined #openstack-neutron | 10:31 | |
*** afazekas has joined #openstack-neutron | 10:31 | |
*** rwsu has joined #openstack-neutron | 10:33 | |
*** nati_uen_ has quit IRC | 10:43 | |
*** shakamunyi has joined #openstack-neutron | 10:43 | |
*** nati_ueno has joined #openstack-neutron | 10:44 | |
*** shakamunyi has quit IRC | 10:48 | |
*** nati_ueno has quit IRC | 10:48 | |
*** yamahata has joined #openstack-neutron | 10:53 | |
openstackgerrit | Yaguang Tang proposed a change to openstack/python-neutronclient: Make neutronclient parse keystone v3 endpoints correctly https://review.openstack.org/81767 | 10:54 |
openstackgerrit | shihanzhang proposed a change to openstack/neutron: Prevent dhcp port deletion from the API https://review.openstack.org/81768 | 10:59 |
*** chandankumar_ has quit IRC | 11:00 | |
*** chandankumar has joined #openstack-neutron | 11:01 | |
*** matsuhashi has quit IRC | 11:01 | |
*** ihrachys is now known as ihrachys|lunch | 11:02 | |
*** roeyc has quit IRC | 11:08 | |
*** leseb has quit IRC | 11:14 | |
*** leseb has joined #openstack-neutron | 11:15 | |
*** devvesa has joined #openstack-neutron | 11:17 | |
*** leseb has quit IRC | 11:19 | |
openstackgerrit | Oleg Bondarev proposed a change to openstack/neutron: agent report_states tests should consider parallel execution https://review.openstack.org/81740 | 11:23 |
*** matsuhashi has joined #openstack-neutron | 11:24 | |
*** jgallard has quit IRC | 11:25 | |
*** mflobo has joined #openstack-neutron | 11:27 | |
*** xianghui has quit IRC | 11:33 | |
*** markmcclain has joined #openstack-neutron | 11:35 | |
*** markmcclain1 has joined #openstack-neutron | 11:35 | |
*** markmcclain1 has quit IRC | 11:35 | |
*** overlayer has quit IRC | 11:36 | |
*** markmcclain1 has joined #openstack-neutron | 11:36 | |
*** alexpilotti has joined #openstack-neutron | 11:37 | |
*** overlayer has joined #openstack-neutron | 11:37 | |
*** markmcclain1 has quit IRC | 11:37 | |
*** markmcclain has quit IRC | 11:39 | |
*** markmcclain has joined #openstack-neutron | 11:41 | |
*** safchain_ has joined #openstack-neutron | 11:42 | |
*** safchain has quit IRC | 11:42 | |
*** shakamunyi has joined #openstack-neutron | 11:44 | |
*** irenab has joined #openstack-neutron | 11:45 | |
*** salv-orlando has joined #openstack-neutron | 11:47 | |
*** shakamunyi has quit IRC | 11:48 | |
*** dave_tucker_zzz is now known as dave_tucker | 11:55 | |
HenryG | salv-orlando: markmcclain: https://review.openstack.org/40296 <--- success! But ... | 11:56 |
salv-orlando | HenryG… But? | 11:57 |
HenryG | salv-orlando: markmcclain: quotas table migration is missing from ML2. Not sure how to fix. | 11:57 |
salv-orlando | quotas have always been pesky. I would not rule that it does not work with migrations even in master | 11:58 |
markmcclain | HenryG: we'll probably have to add a quota migration to make it work correctly | 11:59 |
*** banix has joined #openstack-neutron | 11:59 | |
*** overlayer has quit IRC | 12:00 | |
HenryG | markmcclain: I did that in the patch, don't know if I got it right. | 12:05 |
HenryG | https://review.openstack.org/#/c/40296/9/neutron/db/migration/alembic_migrations/versions/5ac71e65402c_ml2_initial.py | 12:05 |
HenryG | also, extradhcpopts table migration needed fixing | 12:06 |
*** matsuhashi has quit IRC | 12:08 | |
*** crc32 has joined #openstack-neutron | 12:09 | |
*** saju_m has quit IRC | 12:09 | |
*** sballe has quit IRC | 12:12 | |
*** matsuhashi has joined #openstack-neutron | 12:12 | |
*** sballe has joined #openstack-neutron | 12:12 | |
openstackgerrit | Ann Kamyshnikova proposed a change to openstack/neutron: WIP Implement test https://review.openstack.org/76520 | 12:13 |
openstackgerrit | Ann Kamyshnikova proposed a change to openstack/neutron: WIP Add testing for database from oslo https://review.openstack.org/76519 | 12:13 |
*** tomoe_ has joined #openstack-neutron | 12:14 | |
*** matsuhashi has quit IRC | 12:18 | |
*** matsuhashi has joined #openstack-neutron | 12:24 | |
*** saju_m has joined #openstack-neutron | 12:26 | |
*** dvorkinista has joined #openstack-neutron | 12:30 | |
*** banix has quit IRC | 12:31 | |
*** dvorkinista has quit IRC | 12:35 | |
*** xuhanp has joined #openstack-neutron | 12:35 | |
*** devvesa has quit IRC | 12:40 | |
*** leseb has joined #openstack-neutron | 12:40 | |
*** roeyc has joined #openstack-neutron | 12:41 | |
openstackgerrit | shihanzhang proposed a change to openstack/neutron: Prevent dhcp port deletion from the API https://review.openstack.org/81768 | 12:42 |
*** vkozhukalov_ has quit IRC | 12:42 | |
*** shakamunyi has joined #openstack-neutron | 12:45 | |
*** tongli has joined #openstack-neutron | 12:47 | |
*** matsuhashi has quit IRC | 12:47 | |
*** shakamunyi has quit IRC | 12:49 | |
sballe | Could somebody confirm taht the LBaaS weekly meeting is still at 9amEST? or 1400 UTC onIRC channel: #openstack-meeting | 12:50 |
*** saju_m has quit IRC | 12:53 | |
*** vkozhukalov has joined #openstack-neutron | 12:58 | |
*** jgallard has joined #openstack-neutron | 13:00 | |
*** ramishra has quit IRC | 13:00 | |
*** dguerri has joined #openstack-neutron | 13:01 | |
*** ramishra has joined #openstack-neutron | 13:02 | |
*** thuc has joined #openstack-neutron | 13:03 | |
*** mwagner__zzzz is now known as mwagner_ | 13:04 | |
*** thuc_ has joined #openstack-neutron | 13:04 | |
*** saju_m has joined #openstack-neutron | 13:06 | |
*** thuc has quit IRC | 13:07 | |
*** overlayer has joined #openstack-neutron | 13:08 | |
HenryG | salv-orlando: markmcclain: I see the missing quotas table issue with the openvswitch legacy plugin too. Not at all clear to me how to fix that. | 13:13 |
salv-orlando | HenryG: cool I'll check and fle a bug if necessary | 13:14 |
markmcclain | sballe: remember the LBaaS starts in 45mins | 13:15 |
markmcclain | UTC does not shift to summer time | 13:16 |
sballe | markmcclain, thanks. I had forgotten. ;-) | 13:16 |
*** changbl has quit IRC | 13:16 | |
*** ihrachys|lunch is now known as ihrachys | 13:17 | |
markmcclain | HenryG: :( need to think on what is the right way for that | 13:18 |
*** ramishra has quit IRC | 13:20 | |
*** saju_m has quit IRC | 13:23 | |
*** pradipta is now known as pradipta_away | 13:23 | |
openstackgerrit | A change was merged to openstack/neutron: Add a semaphore to some ML2 operations https://review.openstack.org/80413 | 13:25 |
dguerri | hello, is there a way to retrieve all the ports of a given agent (API and/or CLI)? | 13:30 |
*** shakamunyi has joined #openstack-neutron | 13:32 | |
openstackgerrit | A change was merged to openstack/neutron: NEC plugin: Honor Retry-After response from OFC https://review.openstack.org/81472 | 13:33 |
*** peristeri has joined #openstack-neutron | 13:33 | |
HenryG | salv-orlando: markmcclain: missing quotas table for linuxbridge legacy plugin too. :( | 13:36 |
*** salv-orlando has quit IRC | 13:38 | |
*** saju_m has joined #openstack-neutron | 13:39 | |
*** dave_tucker is now known as dave_tucker_zzz | 13:43 | |
*** ramishra has joined #openstack-neutron | 13:44 | |
*** yamahata has quit IRC | 13:48 | |
*** saju_m has quit IRC | 13:48 | |
*** yamahata has joined #openstack-neutron | 13:49 | |
*** dave_tucker_zzz is now known as dave_tucker | 13:58 | |
*** sbalukoff1 has joined #openstack-neutron | 13:58 | |
*** sbalukoff has quit IRC | 14:01 | |
*** jorgem has joined #openstack-neutron | 14:02 | |
openstackgerrit | Hareesh Puthalath proposed a change to openstack/neutron: Include cisco plugin in migration plugins with ovs https://review.openstack.org/80406 | 14:02 |
*** ramishra has quit IRC | 14:03 | |
*** ramishra has joined #openstack-neutron | 14:03 | |
*** sbalukoff1 is now known as sbalukoff | 14:07 | |
openstackgerrit | Robert Kukura proposed a change to openstack/neutron: ML2: Remove validate_port_binding() and unbind_port() https://review.openstack.org/79511 | 14:07 |
*** ramishra has quit IRC | 14:08 | |
*** tvardeman has joined #openstack-neutron | 14:08 | |
*** ramishra has joined #openstack-neutron | 14:09 | |
*** blogan has joined #openstack-neutron | 14:09 | |
*** Jianyong has joined #openstack-neutron | 14:10 | |
*** devvesa has joined #openstack-neutron | 14:10 | |
*** Akshik_ has quit IRC | 14:14 | |
*** Xtnn has joined #openstack-neutron | 14:17 | |
*** ramishra_ has joined #openstack-neutron | 14:17 | |
*** carlp has joined #openstack-neutron | 14:17 | |
*** ramishra has quit IRC | 14:17 | |
*** nati_ueno has joined #openstack-neutron | 14:19 | |
*** Xtnn has left #openstack-neutron | 14:19 | |
*** ramishra_ has quit IRC | 14:19 | |
*** irenab has quit IRC | 14:19 | |
*** ramishra has joined #openstack-neutron | 14:20 | |
*** ramishra has quit IRC | 14:24 | |
*** Jianyong has quit IRC | 14:24 | |
*** ryanpetrello has joined #openstack-neutron | 14:26 | |
*** liudong_ has joined #openstack-neutron | 14:27 | |
*** ryanpetrello has left #openstack-neutron | 14:28 | |
*** thuc_ has quit IRC | 14:30 | |
*** thuc has joined #openstack-neutron | 14:31 | |
sdague | who is handling lbaas bugs? | 14:32 |
sdague | because currently it's randomly falling over in the gate a bit | 14:32 |
sdague | and I'm pretty close to just disabling either the service or the tempest tests on it | 14:33 |
sdague | http://logs.openstack.org/78/81778/1/check/check-tempest-dsvm-neutron/0a85d7c/console.html#_2014-03-20_13_43_58_797 as an example | 14:33 |
*** armax has joined #openstack-neutron | 14:35 | |
*** thuc has quit IRC | 14:35 | |
HenryG | sdague: lbaas meeting going on in #openstack-meeting right now. | 14:35 |
*** sfox has joined #openstack-neutron | 14:37 | |
devvesa | #openstack-meeting-3 | 14:44 |
*** salv-orlando has joined #openstack-neutron | 14:45 | |
*** dvorkinista has joined #openstack-neutron | 14:49 | |
enikanorov__ | sdague: hi | 14:52 |
enikanorov__ | sdague: we're looking into it | 14:52 |
obondarev | sdague: there is a bug https://bugs.launchpad.net/tempest/+bug/1294603 | 14:52 |
*** carl_baldwin has joined #openstack-neutron | 14:52 | |
*** nati_ueno has quit IRC | 14:52 | |
sdague | this is not a tempest bug | 14:53 |
*** dvorkinista has quit IRC | 14:53 | |
sdague | at least not as far as I can tell | 14:53 |
sdague | please merge bugs and make it critical on the neutron side | 14:54 |
obondarev | sdague: 1294603 is opened for both tempest and neutron side | 14:54 |
sdague | obondarev: ok, it's not a tempest bug | 14:55 |
*** leseb has quit IRC | 14:55 | |
obondarev | sdague: probably disabling the test would be the best option at the moment | 14:55 |
*** mwagner is now known as mwagner_notHere | 14:55 | |
sdague | I think we should disable lbaas instead | 14:56 |
obondarev | sdague: to unlock the gate while we are working on the bug | 14:56 |
sdague | we really shouldn't be running a service that we don't have testing on | 14:56 |
markmcclain | obondarev: can you propose skipping that test | 14:56 |
obondarev | sdague: why? | 14:56 |
*** thuc has joined #openstack-neutron | 14:56 | |
obondarev | markmcclain: you mean submit a patch for skipping? | 14:56 |
*** otherwiseguy has joined #openstack-neutron | 14:56 | |
sdague | because we should ensure everything we're running in the gate is actually being tested | 14:56 |
markmcclain | obondarev: yes | 14:57 |
obondarev | sdague: that test was merged a couple of days ago | 14:57 |
obondarev | markmcclain: ok will do | 14:57 |
sdague | obondarev: what is testing that lbaas is working besides that test? | 14:58 |
obondarev | sdague: yes, and there are no changes to lbaas since that test was merged | 14:59 |
sdague | obondarev: I understand that | 14:59 |
sdague | it means lbaas always wasn't working | 14:59 |
sdague | now we just test it to know | 14:59 |
sdague | that it randomly breaks a lot | 14:59 |
obondarev | sdague: not sure, need to investigate more on this | 14:59 |
sdague | which is why I asked the question about what other tests are validating that lbaas is working correctly in the gate | 15:00 |
obondarev | sdague: only api tests | 15:00 |
sdague | obondarev: and those don't test actual lbaas functionality, right? | 15:01 |
obondarev | sdague: correct | 15:01 |
sdague | which means lbaas is, and has been, broken | 15:01 |
*** thuc has quit IRC | 15:02 | |
sdague | so we can take a skip for now to relieve the gate, but if we don't have a fix within a week, I think we should stop running the service in the gate entirely | 15:02 |
*** thuc has joined #openstack-neutron | 15:02 | |
*** thedodd has joined #openstack-neutron | 15:02 | |
*** nati_ueno has joined #openstack-neutron | 15:03 | |
enikanorov__ | sdague: why so? | 15:03 |
sdague | enikanorov: because it doesn't work | 15:04 |
obondarev | sdague: will try to address the issue within a week | 15:04 |
enikanorov__ | grep "skip_because(bug=" * -r | wc -l | 15:05 |
enikanorov__ | 41 | 15:05 |
enikanorov__ | sure will try to fix that | 15:05 |
enikanorov__ | *we'll | 15:05 |
*** mwagner_ has quit IRC | 15:06 | |
*** leseb has joined #openstack-neutron | 15:10 | |
*** overlayer has quit IRC | 15:12 | |
sdague | I also think a big issue is the review that turned it on slipped the skip removal in the middle of a refactor - https://review.openstack.org/#/c/71898/ | 15:13 |
*** bashok has quit IRC | 15:13 | |
*** overlayer has joined #openstack-neutron | 15:14 | |
enikanorov__ | sdague: https://review.openstack.org/#/c/81831/ | 15:14 |
sdague | yep, +A | 15:14 |
sdague | thanks | 15:14 |
*** liudong_ has quit IRC | 15:15 | |
*** banix has joined #openstack-neutron | 15:15 | |
enikanorov__ | sdague: it was intentional, because adding floating IP fixed the root caused of why it was skipped | 15:15 |
*** yfried1 has quit IRC | 15:16 | |
sdague | enikanorov__: except, it didn't | 15:17 |
*** networkstatic has joined #openstack-neutron | 15:18 | |
*** changbl has joined #openstack-neutron | 15:18 | |
enikanorov__ | well, it did/ it worked well untill few days ago | 15:18 |
*** networkstatic has quit IRC | 15:20 | |
salv-orlando | it *merged* a few days ago (cheeky comment) | 15:20 |
enikanorov__ | >:E | 15:20 |
*** carl_baldwin_ has joined #openstack-neutron | 15:22 | |
*** carl_baldwin has quit IRC | 15:22 | |
*** carl_baldwin_ is now known as carl_baldwin | 15:22 | |
*** yfried has joined #openstack-neutron | 15:24 | |
*** sfox has quit IRC | 15:24 | |
*** sfox1 has joined #openstack-neutron | 15:24 | |
*** Jabadia has quit IRC | 15:27 | |
sdague | enikanorov: look at logstash - http://logstash.openstack.org/#eyJmaWVsZHMiOltdLCJzZWFyY2giOiJtZXNzYWdlOlwiRkFJTDogdGVtcGVzdC5zY2VuYXJpby50ZXN0X2xvYWRfYmFsYW5jZXJfYmFzaWMuVGVzdExvYWRCYWxhbmNlckJhc2ljLnRlc3RfbG9hZF9iYWxhbmNlcl9iYXNpY1wiIEFORCBmaWxlbmFtZTpjb25zb2xlLmh0bWwiLCJ0aW1lZnJhbWUiOiJhbGwiLCJncmFwaG1vZGUiOiJjb3VudCIsIm9mZnNldCI6MCwidGltZSI6eyJ1c2VyX2ludGVydmFsIjowfSwic3RhbXAiOjEzOTUzMjkxMzE0MTJ9 | 15:27 |
sdague | this has been failing since the test was turned on | 15:27 |
*** Jabadia has joined #openstack-neutron | 15:27 | |
sdague | 220 fails in 14 days | 15:27 |
sdague | so this isn't just the last couple of days | 15:27 |
*** krtaylor has quit IRC | 15:27 | |
*** shakamunyi has quit IRC | 15:29 | |
openstackgerrit | A change was merged to openstack/neutron: BigSwitch: Widen range of HTTPExceptions caught https://review.openstack.org/79064 | 15:30 |
*** Jabadia has quit IRC | 15:31 | |
kevinbenton | has the gate failure rate reduced since https://review.openstack.org/#/c/80413/ merged? | 15:32 |
kevinbenton | due to db lock issues | 15:32 |
*** overlayer has quit IRC | 15:36 | |
openstackgerrit | enikanorov proposed a change to openstack/neutron: Fix typo in lbaas agent exception message https://review.openstack.org/81835 | 15:36 |
*** banix has quit IRC | 15:38 | |
*** sbalukoff has quit IRC | 15:39 | |
enikanorov__ | sdague: the failure patterns are different. looking into older ones | 15:40 |
enikanorov__ | the spike of test failures happened 1-2 days ago | 15:40 |
openstackgerrit | Avishay Balderman proposed a change to openstack/neutron: LBaaS L7 Rules Implementation https://review.openstack.org/61721 | 15:42 |
*** otherwiseguy has quit IRC | 15:43 | |
openstackgerrit | Kevin Benton proposed a change to openstack/neutron: Add decorator to prevent eventlet from yielding https://review.openstack.org/81709 | 15:43 |
openstackgerrit | A change was merged to openstack/neutron: ML2 BigSwitch: Don't modify parent context https://review.openstack.org/81360 | 15:44 |
openstackgerrit | A change was merged to openstack/neutron: Allow CIDRs with non-zero masked portions https://review.openstack.org/81137 | 15:46 |
openstackgerrit | A change was merged to openstack/neutron: Process ICMP type for iptables firewall https://review.openstack.org/78835 | 15:46 |
*** crc32 has quit IRC | 15:46 | |
openstackgerrit | A change was merged to openstack/neutron: BigSwitch: Use eventlet.sleep in watchdog https://review.openstack.org/80130 | 15:47 |
sdague | kevinbenton: it's probably too early to tell | 15:47 |
*** claudiub_ has joined #openstack-neutron | 15:48 | |
kevinbenton | sdague: is this an appropriate query for that? | 15:48 |
kevinbenton | http://logstash.openstack.org/#eyJmaWVsZHMiOltdLCJzZWFyY2giOiJtZXNzYWdlOlwiTG9jayB3YWl0IHRpbWVvdXQgZXhjZWVkZWRcIiBBTkQgZmlsZW5hbWU6Y29uc29sZS5odG1sIiwidGltZWZyYW1lIjoiYWxsIiwiZ3JhcGhtb2RlIjoiY291bnQiLCJvZmZzZXQiOjAsInRpbWUiOnsidXNlcl9pbnRlcnZhbCI6MH0sInN0YW1wIjoxMzk1MzMwMzA4ODE0fQ== | 15:48 |
*** devlaps has joined #openstack-neutron | 15:49 | |
sdague | kevinbenton: I don't think so | 15:50 |
sdague | http://status.openstack.org/elastic-recheck/index.html has the logstash link we are tracking | 15:50 |
openstackgerrit | Nachi Ueno proposed a change to openstack/neutron: Add enable_security_group option https://review.openstack.org/67281 | 15:53 |
kevinbenton | sdague: i see. thx | 15:54 |
*** rossella_s has quit IRC | 15:54 | |
sdague | I have not "yet" directly noticed that failure in the gate since the merge, which is something | 15:55 |
*** alagalah has joined #openstack-neutron | 15:55 | |
sdague | but it's all statistics, so we need to build enough data to know for sure | 15:55 |
*** rossella_s has joined #openstack-neutron | 15:56 | |
kevinbenton | right | 15:56 |
claudiub_ | hello | 15:56 |
claudiub_ | if i could take a bit of your time to review a small bug fix, i'd appreciate it. :) | 15:57 |
claudiub_ | ~20-30 lines. | 15:57 |
claudiub_ | https://review.openstack.org/#/c/81382/ | 15:58 |
*** networkstatic has joined #openstack-neutron | 15:58 | |
*** devvesa has quit IRC | 15:58 | |
*** YorikSar has joined #openstack-neutron | 16:00 | |
*** xuhanp has quit IRC | 16:03 | |
*** carl_baldwin has quit IRC | 16:05 | |
*** carl_baldwin_ has joined #openstack-neutron | 16:05 | |
*** carl_baldwin has joined #openstack-neutron | 16:05 | |
*** shakamunyi has joined #openstack-neutron | 16:06 | |
*** devvesa has joined #openstack-neutron | 16:06 | |
*** otherwiseguy has joined #openstack-neutron | 16:07 | |
*** carl_baldwin has quit IRC | 16:09 | |
*** shakamunyi has quit IRC | 16:10 | |
*** carl_baldwin has joined #openstack-neutron | 16:11 | |
*** leseb has quit IRC | 16:12 | |
*** sfox1 has quit IRC | 16:12 | |
*** carl_baldwin has quit IRC | 16:13 | |
*** _cjones_ has joined #openstack-neutron | 16:16 | |
*** manishg has joined #openstack-neutron | 16:17 | |
openstackgerrit | Kevin Benton proposed a change to openstack/neutron: Add decorator to prevent eventlet from yielding https://review.openstack.org/81709 | 16:19 |
*** yfried has quit IRC | 16:20 | |
*** thuc has quit IRC | 16:23 | |
*** thuc has joined #openstack-neutron | 16:23 | |
*** thuc has quit IRC | 16:28 | |
*** thuc has joined #openstack-neutron | 16:28 | |
*** SumitNaiksatam has quit IRC | 16:28 | |
*** banix has joined #openstack-neutron | 16:30 | |
*** yfried has joined #openstack-neutron | 16:31 | |
*** banix has quit IRC | 16:32 | |
*** sballe has quit IRC | 16:34 | |
*** dims has quit IRC | 16:37 | |
*** leseb has joined #openstack-neutron | 16:43 | |
*** thuc has quit IRC | 16:43 | |
*** thuc_ has joined #openstack-neutron | 16:43 | |
*** SumitNaiksatam has joined #openstack-neutron | 16:43 | |
*** overlayer has joined #openstack-neutron | 16:43 | |
*** YorikSar has left #openstack-neutron | 16:44 | |
*** leseb has quit IRC | 16:44 | |
*** leseb has joined #openstack-neutron | 16:44 | |
*** leseb has quit IRC | 16:47 | |
*** leseb has joined #openstack-neutron | 16:47 | |
*** dims has joined #openstack-neutron | 16:49 | |
*** suresh12 has joined #openstack-neutron | 16:50 | |
*** suresh12 has quit IRC | 16:51 | |
*** carl_baldwin has joined #openstack-neutron | 16:51 | |
*** otherwiseguy has quit IRC | 16:52 | |
*** spandhe has joined #openstack-neutron | 16:57 | |
*** leseb has quit IRC | 16:58 | |
*** leseb has joined #openstack-neutron | 16:58 | |
*** roeyc has quit IRC | 16:59 | |
*** changbl has quit IRC | 17:02 | |
*** leseb has quit IRC | 17:02 | |
*** sfox has joined #openstack-neutron | 17:04 | |
*** devvesa has quit IRC | 17:05 | |
*** mwagner_ has joined #openstack-neutron | 17:07 | |
*** nati_ueno has quit IRC | 17:08 | |
*** nati_ueno has joined #openstack-neutron | 17:08 | |
*** dvorkinista has joined #openstack-neutron | 17:09 | |
*** skraynev is now known as skraynev_afl | 17:10 | |
*** skraynev_afl is now known as skraynev_afk | 17:10 | |
*** nati_ueno has quit IRC | 17:12 | |
*** dvorkinista has quit IRC | 17:13 | |
*** carl_baldwin has quit IRC | 17:14 | |
*** vkozhukalov has quit IRC | 17:17 | |
*** carl_baldwin has joined #openstack-neutron | 17:18 | |
*** jobewan has joined #openstack-neutron | 17:18 | |
jogo | sdague kevinbenton: it looks like there were gate hits for the bug | 17:20 |
kevinbenton | jogo: do you have a link to a log? | 17:21 |
jogo | kevinbenton: http://logs.openstack.org/08/74508/4/gate/gate-tempest-dsvm-neutron/6f857b4/logs/screen-q-svc.txt | 17:21 |
jogo | http://logs.openstack.org/91/81091/1/gate/gate-tempest-dsvm-sahara-neutron/e5bf7db/logs/screen-q-svc.txt | 17:21 |
*** leseb has joined #openstack-neutron | 17:23 | |
*** otherwiseguy has joined #openstack-neutron | 17:23 | |
*** morganfainberg_Z is now known as morganfainberg | 17:23 | |
*** claudiub_ has quit IRC | 17:23 | |
kevinbenton | jogo: thx. looks like it's a vip issue | 17:24 |
kevinbenton | jogo: need another lock... | 17:24 |
*** tomoe_ has quit IRC | 17:24 | |
*** tomoe_ has joined #openstack-neutron | 17:25 | |
*** krtaylor has joined #openstack-neutron | 17:27 | |
*** tomoe_ has quit IRC | 17:29 | |
enikanorov__ | jogo: kevinbenton: what bug are you discussing? | 17:30 |
jogo | enikanorov__: top bug on http://status.openstack.org/elastic-recheck/gate.html | 17:30 |
*** afazekas has quit IRC | 17:30 | |
kevinbenton | enikanorov__: the eventlet/mysql deadlock issues | 17:31 |
enikanorov__ | i see | 17:31 |
*** banix has joined #openstack-neutron | 17:34 | |
*** changbl has joined #openstack-neutron | 17:34 | |
jogo | kevinbenton: if you have another patch that will help, once its in gate queue ping me or sdague and you we can help get that patch to the top of the gate (by asking infra very nicely) | 17:36 |
*** sbalukoff has joined #openstack-neutron | 17:38 | |
kevinbenton | jogo: sounds good | 17:39 |
*** crc32 has joined #openstack-neutron | 17:39 | |
jogo | because if we can't fix this soon ... well we are in trouble | 17:39 |
jogo | any ideas of what happend on mar 12 to cause this bug? | 17:39 |
*** banix has quit IRC | 17:41 | |
kevinbenton | jogo: it's hard to tell. something that causes a thread to yield after it has a DB lock | 17:42 |
kevinbenton | jogo: and mar 12 could have just been a change that adjusted the timing in the tests to expose it | 17:42 |
kevinbenton | jogo: the bug could have been there for quite a while | 17:43 |
jogo | kevinbenton: right | 17:45 |
*** carl_baldwin has quit IRC | 17:45 | |
*** tomoe_ has joined #openstack-neutron | 17:45 | |
*** banix has joined #openstack-neutron | 17:47 | |
*** dfarrell07 has joined #openstack-neutron | 17:48 | |
*** banix has quit IRC | 17:49 | |
*** ygbo has quit IRC | 17:49 | |
openstackgerrit | mark mcclain proposed a change to openstack/neutron: Disable XML tests on Py26 https://review.openstack.org/81865 | 17:50 |
jogo | I69fe9cff4b550240215329e1082a63cd11617faa | 17:51 |
jogo | Ifed2505a97ceca56c4cedd22efa49adcdf3317b8 | 17:52 |
*** banix has joined #openstack-neutron | 17:53 | |
jogo | both are in the right time period (roughly) | 17:53 |
*** banix has quit IRC | 17:54 | |
jogo | not sure if they made things worse or not | 17:55 |
jogo | kevinbenton: ^^ | 17:55 |
kevinbenton | jogo: the disassociate one possibly just by changing the order of DB transactions | 17:55 |
*** safchain_ has quit IRC | 17:55 | |
jogo | so if other neutron cores agree with you, and its safe to revert ... I think its worth trying | 17:58 |
kevinbenton | jogo: it's hard to tell. is there a mechanism to launch several simultaneous checks? | 17:59 |
jogo | kevinbenton: no :/ but with how frequent this bug is you can just run recheck a few times | 18:00 |
jogo | other option is to push up afew duplicate patches | 18:00 |
kevinbenton | jogo: give me a few mins, working on the vip issue | 18:05 |
*** ijw has joined #openstack-neutron | 18:06 | |
jogo | kevinbenton: well if the fix works, thats always better then a revert | 18:07 |
*** fouxm has quit IRC | 18:08 | |
*** tomoe_ has quit IRC | 18:08 | |
*** leseb has quit IRC | 18:08 | |
*** tomoe_ has joined #openstack-neutron | 18:09 | |
*** jorgem has quit IRC | 18:09 | |
*** jorgem has joined #openstack-neutron | 18:11 | |
*** tomoe_ has quit IRC | 18:13 | |
*** amuller has quit IRC | 18:15 | |
*** otherwiseguy has quit IRC | 18:16 | |
marun | jogo: hmmm, looking at those patches now | 18:19 |
*** yfried has quit IRC | 18:20 | |
kevinbenton | salv-orlando: ping | 18:20 |
marun | jogo: the fact that disassociation now occurs after a lock is acquired could be a problem | 18:21 |
salv-orlando | kevinbenton: ping | 18:21 |
marun | kevinbenton: so is the problem just for the vips? | 18:21 |
marun | kevinbenton: or is there still contention on the ports table? | 18:21 |
kevinbenton | marun: i think the vip update_status operation can be blocked by a lock on the port table | 18:21 |
kevinbenton | marun: it's joined to it via port_id | 18:22 |
marun | that would do it. | 18:22 |
kevinbenton | salv-orlando, marun: i have another patch for you to look at | 18:22 |
salv-orlando | jogo: I try to gues… disassociate_floating_ips change and updatE_floating_ip_status | 18:22 |
openstackgerrit | Kevin Benton proposed a change to openstack/neutron: Add semaphore to LB VIP status update https://review.openstack.org/81877 | 18:22 |
salv-orlando | reverts are already unde review for both | 18:23 |
marun | that second patch won't affect anything b/c the metaplugin is not tested in gate | 18:23 |
salv-orlando | marun: yeah not metaplugin | 18:23 |
marun | salv-orlando: both of which? | 18:23 |
kevinbenton | salv-orlando, marun: https://review.openstack.org/81877 | 18:23 |
marun | salv-orlando: I69fe9cff4b550240215329e1082a63cd11617faa I get | 18:23 |
salv-orlando | the two patches we attempt to revert in the past | 18:23 |
salv-orlando | but have seen lock timeout failures in the past 4 hours? | 18:23 |
*** yfried has joined #openstack-neutron | 18:23 | |
salv-orlando | not from what I gather | 18:24 |
salv-orlando | jogo: log stash still reports last failure at 12:28 GMT today | 18:25 |
salv-orlando | and the patch supposed to fix merged shortly after | 18:25 |
jogo | http://logs.openstack.org/08/74508/4/gate/gate-tempest-dsvm-neutron/6f857b4/logs/screen-q-svc.txt | 18:25 |
jogo | there appear to be noticably less | 18:25 |
jogo | oh wait that is a successfull run | 18:26 |
*** packet has joined #openstack-neutron | 18:26 | |
jogo | salv-orlando: now I see the same thing yu do | 18:26 |
jogo | 12:28 | 18:26 |
jogo | once I ignore successful runs | 18:26 |
salv-orlando | jogo: successful runs with lock wait timeout errors | 18:26 |
*** jlibosva has quit IRC | 18:26 | |
salv-orlando | are mostly because of a similar problem in the bit kevinbenton addresses in 81877 | 18:27 |
jogo | ahh cool | 18:27 |
jogo | so things are looking promising! | 18:27 |
jogo | we also have the libvirt fix in and that is looking good too | 18:27 |
*** jorgem has quit IRC | 18:29 | |
enikanorov__ | kevinbenton: commented on https://review.openstack.org/#/c/81877/ | 18:31 |
*** thuc_ has quit IRC | 18:32 | |
*** claudiub_ has joined #openstack-neutron | 18:32 | |
*** thuc has joined #openstack-neutron | 18:33 | |
*** markmcclain has quit IRC | 18:34 | |
*** jorgem has joined #openstack-neutron | 18:34 | |
kevinbenton | enikanorov__: is everyone comfortable with a change like that at this time? | 18:34 |
enikanorov__ | kevinbenton: it should be safe change if it passes UTs (and I'm sure it will) | 18:35 |
kevinbenton | also, synchronization doesn't need to work across servers. the problem is specific to threads in same pool | 18:35 |
*** jgallard has quit IRC | 18:35 | |
*** manishg_ has joined #openstack-neutron | 18:37 | |
*** thuc has quit IRC | 18:37 | |
*** manishg has quit IRC | 18:39 | |
*** manishg_ is now known as manishg | 18:39 | |
enikanorov__ | kevinbenton: so, if ports are locked, could not the same issue appear on another server where semaphore isn't locked? | 18:40 |
enikanorov__ | disjoining tables will solve tis once and for all | 18:40 |
*** stefanb has joined #openstack-neutron | 18:40 | |
*** alagalah has quit IRC | 18:42 | |
openstackgerrit | enikanorov proposed a change to openstack/neutron: Return meaningful error message on pool creation error https://review.openstack.org/81883 | 18:44 |
stefanb | is the device_owner property of a port something that can be set by a client or is it being set when the port is connected to 'something' ? | 18:44 |
*** alagalah has joined #openstack-neutron | 18:44 | |
stefanb | specifically I would like to create a port for a VM where IP spoofing is disabled; I see code in neutron not applying spoofing rules if the device_owner starts with 'network:' | 18:46 |
stefanb | The code to create such a port would be located inside nova; it would be similar to the cloud-pipe code | 18:47 |
kevinbenton | enikanorov__: on another server, it would just block until the other server with the lock finishes what it's doing | 18:47 |
kevinbenton | enikanorov__: the port fix is better | 18:48 |
enikanorov__ | kevinbenton: port fix is what? | 18:48 |
kevinbenton | enikanorov__: what you suggested | 18:48 |
kevinbenton | enikanorov__: because it should just eliminate the contention | 18:48 |
enikanorov__ | yep | 18:48 |
enikanorov__ | kevinbenton: can you explain the issue with mysql and eventlet? so as i understand, when you query the row, it blocks on port table and greenthread yields; does it take control back? | 18:50 |
kevinbenton | enikanorov__: the issue is that the greenthread doesn't yield when mysql blocks | 18:52 |
*** manishg has quit IRC | 18:53 | |
enikanorov__ | so it's there stuck forever? | 18:53 |
enikanorov__ | for particular query or what? | 18:53 |
kevinbenton | enikanorov__: until the mysql lock on that table is released | 18:53 |
kevinbenton | enikanorov__: at least this is the problem with any UPDATE statements | 18:54 |
enikanorov__ | i see. is the root cause of such behavior known? | 18:54 |
*** banix has joined #openstack-neutron | 18:55 | |
kevinbenton | enikanorov__: yes, it's a fundamental issue with mysql/eventlet. the mysql call goes into a mysql C library that eventlet doesn't understand so it can't recognize blocking calls and yield | 18:55 |
kevinbenton | https://docs.google.com/drawings/d/13A2x4AWbf8zmzeGApUmYVlBrW8CMTPFTCBGSP_nTzDA/edit?usp=sharing | 18:55 |
enikanorov__ | thanks for the details | 18:56 |
kevinbenton | enikanorov__: | 18:56 |
kevinbenton | see diagram | 18:56 |
*** sfox has quit IRC | 18:56 | |
*** manishg has joined #openstack-neutron | 18:56 | |
*** dvorkini_ has joined #openstack-neutron | 18:56 | |
enikanorov__ | yeah | 18:56 |
armax | the lock is per row, afaik | 18:56 |
*** armitage81 has joined #openstack-neutron | 18:57 | |
kevinbenton | enikanorov__: shoot. i have a meeting for an hour or so. can you look at how i will disable the join in the patch without changing get_by_id? | 18:57 |
armax | so locking the entire table is caused by the scope of the query | 18:57 |
enikanorov__ | kevinbenton: sure, i'll update your patch | 18:57 |
*** armitage81 has quit IRC | 18:57 | |
*** armitage81 has joined #openstack-neutron | 18:57 | |
*** sfox has joined #openstack-neutron | 18:58 | |
*** jorgem has quit IRC | 19:00 | |
*** sfox has quit IRC | 19:02 | |
*** jorgem has joined #openstack-neutron | 19:03 | |
*** thuc has joined #openstack-neutron | 19:04 | |
rkukura | kevinbenton: Wish I had seen that drawing before! | 19:06 |
*** rotbeard has quit IRC | 19:06 | |
mestery | rkukura kevinbenton: +1000 | 19:07 |
claudiub_ | hello. | 19:08 |
claudiub_ | could anyone help me with a review for a small bug fix (3-4 lines of code) on neutron? :) | 19:09 |
claudiub_ | https://review.openstack.org/#/c/81382/ | 19:09 |
mestery | claudiub_: Does the same fix need to be made to the Hyper-V ML2 driver? | 19:11 |
*** thuc_ has joined #openstack-neutron | 19:12 | |
claudiub_ | no, it is strictly hyper-v related | 19:12 |
*** thuc has quit IRC | 19:12 | |
claudiub_ | the way the security groups ACLs are added to a hyper-v switch port | 19:12 |
mestery | OK, cool. Just was curious because of the Hyper-V neutron agent is there too. | 19:12 |
HenryG | mestery: Looks like the claudiub_ change is to the agent only, so should cover ML2 use case too? | 19:14 |
mestery | HenryG: Yes, noticed that as well :) | 19:14 |
*** vkozhukalov has joined #openstack-neutron | 19:15 | |
claudiub_ | thx guys for the help. :) | 19:17 |
*** kfox1111 has joined #openstack-neutron | 19:25 | |
kfox1111 | I have a sytem that is constantly neutron.agent.securitygroups_rpc refreshing firewall rules. What would cause that? | 19:26 |
*** aveiga has joined #openstack-neutron | 19:32 | |
*** packet has quit IRC | 19:33 | |
*** packet has joined #openstack-neutron | 19:34 | |
openstackgerrit | enikanorov proposed a change to openstack/neutron: Get rid of additional db contention on fetching VIP https://review.openstack.org/81877 | 19:36 |
enikanorov__ | kevinbenton: ^^ | 19:36 |
*** overlayer has quit IRC | 19:41 | |
*** vu has joined #openstack-neutron | 19:46 | |
*** sballe has joined #openstack-neutron | 19:48 | |
*** djoreilly has quit IRC | 19:48 | |
kevinbenton | salv-orlando, marun: ^^ | 19:49 |
kfox1111 | kevinbenton: hi. | 19:51 |
kfox1111 | what process inserts firewall refresh rpc's in linuxbridge-agent's queue? | 19:51 |
*** alagalah has quit IRC | 19:56 | |
*** alagalah has joined #openstack-neutron | 19:57 | |
*** otherwiseguy has joined #openstack-neutron | 19:58 | |
*** thuc_ has quit IRC | 19:58 | |
*** thuc has joined #openstack-neutron | 19:58 | |
*** dvorkinista has joined #openstack-neutron | 19:59 | |
*** thuc_ has joined #openstack-neutron | 19:59 | |
*** dvorkini_ has quit IRC | 20:00 | |
*** markwash has joined #openstack-neutron | 20:01 | |
kevinbenton | kfox1111: one sec. i believe any port changes will notify agents and they will check to see if their port is modified, and if so, refresh the firewall | 20:01 |
kevinbenton | kfox1111: the other i thought was just a rule change | 20:02 |
*** markwash has quit IRC | 20:02 | |
*** markwash has joined #openstack-neutron | 20:02 | |
*** thuc has quit IRC | 20:03 | |
marun | kevinbenton: looking | 20:04 |
kfox1111 | I don't think either of those should be happening at the frequency I'm seeing them. I'm seeing so many its not able to keep up. | 20:05 |
kevinbenton | kfox1111: so are they messages on the bus, or just locally showing firewall_refresh in the agent logs? | 20:07 |
kevinbenton | kfox1111: because the agent can refresh itself | 20:07 |
kfox1111 | good question. the queue is getting backed up. the only thing in the logs is firewall_refresh, which is constantly spitting out. | 20:09 |
kfox1111 | let me turn up debugging and see what happens. | 20:09 |
enikanorov__ | kevinbenton: marun: salv-orlando: need to update the patch, i see an issue, sorry | 20:09 |
kevinbenton | kfox1111: it might be worth turning up the debugging on the server too to see what it's sending | 20:10 |
kevinbenton | enikanorov__: ok | 20:11 |
openstackgerrit | A change was merged to openstack/neutron: NSX: Sync do not pass around model object https://review.openstack.org/74508 | 20:11 |
kfox1111 | I'm seeing lots of stuff going on now... | 20:11 |
kevinbenton | kfox1111: http://paste.openstack.org/ | 20:12 |
kfox1111 | Is there anything in here that is sensitive? | 20:12 |
kfox1111 | also, how much do you want? | 20:12 |
kevinbenton | kfox1111: neutron.openstack.common.rpc.amqp | 20:14 |
*** thuc has joined #openstack-neutron | 20:14 | |
kevinbenton | kfox1111: grep for those ones | 20:14 |
kfox1111 | k. | 20:14 |
kevinbenton | kfox1111: those should be safe from sensitive stuff | 20:15 |
claudiub_ | hey guys, i also have a backport patch for some time, could you please take a look at it? :) | 20:15 |
claudiub_ | https://review.openstack.org/#/c/63775/ | 20:15 |
kfox1111 | http://pastebin.com/mK4ypQT8 | 20:15 |
kfox1111 | same thing over and over. | 20:15 |
*** overlayer has joined #openstack-neutron | 20:16 | |
kfox1111 | hmm.. the messages look like its looping over each port... | 20:17 |
*** thuc_ has quit IRC | 20:18 | |
kfox1111 | grepping for linux.utils, | 20:18 |
kfox1111 | its calling iptables-save and iptables-restore alot. | 20:18 |
*** thuc has quit IRC | 20:18 | |
kfox1111 | hmm... this is interesting. top is showing neutron-server on the core being between 14 and 30% cpu load continuously... | 20:19 |
kfox1111 | maybe that is related. | 20:19 |
kfox1111 | neutron-linuxbridge-agent looks like its about similarly loaded too. | 20:20 |
openstackgerrit | enikanorov proposed a change to openstack/neutron: Get rid of additional db contention on fetching VIP https://review.openstack.org/81877 | 20:20 |
kevinbenton | kfox1111: this pastebin is from the server, right? | 20:21 |
*** mflobo_ has joined #openstack-neutron | 20:22 | |
kfox1111 | from linuxbridge-agent. you want neutron-server? | 20:22 |
kfox1111 | http://pastebin.com/7ghQYf3Y <- neutron-server amqp debugging logs. | 20:24 |
*** mflobo has quit IRC | 20:25 | |
sballe | enikanorov, I am trying to understand if the neutron LBaaS scheduler could schedule LBaaS e.g. HA proxies the same way that Nova can schedule VMs? | 20:26 |
sballe | I am looking at this: https://wiki.openstack.org/wiki/Neutron/LBaaS/Architecture/Scheduler and I am not clear on he answer | 20:27 |
kevinbenton | kfox1111: i wonder if the constant port updates from the dhcp-agent are the problem | 20:27 |
kevinbenton | in the linuxbridge-agent | 20:27 |
*** julim has quit IRC | 20:28 | |
kfox1111 | hmm... let me look at the dhcp-agent... | 20:29 |
enikanorov__ | pcm_: hi. replied to your comment: https://review.openstack.org/#/c/79599/ | 20:29 |
kfox1111 | hmm.... | 20:29 |
kfox1111 | its in a slightly bad way... lets see... | 20:30 |
kfox1111 | I'm seeing this about every 10 seconds: | 20:31 |
kfox1111 | http://pastebin.com/RLNiJ6gm | 20:31 |
kevinbenton | kfox1111: i would say that's the root cause | 20:33 |
kfox1111 | oh, interesting... | 20:33 |
kfox1111 | 3e4d979e-c5cc-4414-9c03-e08831259276 is our external network. | 20:33 |
kevinbenton | kfox1111: dhcp agent failing to start | 20:33 |
kfox1111 | it looks like someone added a ipv6 subnet to it... | 20:33 |
kfox1111 | eeep. and dhcp is enabled on it. thats probably it... | 20:34 |
kfox1111 | how do I use neutron subnet-update to turn off dhcp? | 20:35 |
*** banix has quit IRC | 20:35 | |
kfox1111 | hmm.. ok. using an argument the same name as the table row worked. | 20:37 |
*** carl_baldwin has joined #openstack-neutron | 20:37 | |
*** rwsu has quit IRC | 20:37 | |
kfox1111 | its not dhcp agaent is not spewing anymore... | 20:38 |
*** armitage81 has quit IRC | 20:38 | |
*** aveiga has quit IRC | 20:39 | |
kfox1111 | ah... neutron server load is much down now. | 20:40 |
kfox1111 | thanks for the debugging help... | 20:40 |
*** crc32 has quit IRC | 20:41 | |
kevinbenton | kfox1111: hey, sorry about the delay. you should have been able to do it from the horizon interface for the subnet | 20:43 |
*** fouxm has joined #openstack-neutron | 20:45 | |
*** peristeri has quit IRC | 20:45 | |
*** dims has quit IRC | 20:46 | |
*** mwagner_ has quit IRC | 20:46 | |
kfox1111 | Ah, I see. I was trying to edit from the project's tab, and it wasn't giving me any editing boxes for it. But if Igo to the admin tab, then I can do it. | 20:47 |
kfox1111 | yeah. the queue seems to be drained nicely now. I think that was it. | 20:47 |
*** armitage81 has joined #openstack-neutron | 20:52 | |
*** armitage81 has quit IRC | 20:53 | |
*** armitage81 has joined #openstack-neutron | 20:53 | |
*** marun has quit IRC | 20:53 | |
kfox1111 | thanks again for all the help. | 20:53 |
kevinbenton | kfox1111: no prob. glad you got it fixed | 20:54 |
*** alagalah has quit IRC | 20:57 | |
*** ajo has quit IRC | 20:57 | |
*** krtaylor has quit IRC | 20:57 | |
kfox1111 | hmm... now on to figuring out why the load ballancer is not passing stuff through... | 20:58 |
*** dvorkinista has quit IRC | 21:01 | |
*** dims has joined #openstack-neutron | 21:01 | |
kfox1111 | everything looks to be hooked up right. | 21:02 |
*** thuc has joined #openstack-neutron | 21:03 | |
*** zhipeng has joined #openstack-neutron | 21:04 | |
*** rwsu has joined #openstack-neutron | 21:04 | |
*** jaypipes has quit IRC | 21:07 | |
*** tongli has quit IRC | 21:07 | |
*** dvorkinista has joined #openstack-neutron | 21:08 | |
kfox1111 | hmm... does a vip have a security group associated with it? | 21:09 |
*** lukego has quit IRC | 21:09 | |
*** dvorkinista has quit IRC | 21:09 | |
*** carl_baldwin has quit IRC | 21:09 | |
*** carl_baldwin has joined #openstack-neutron | 21:09 | |
*** carl_baldwin has quit IRC | 21:11 | |
kevinbenton | kfox1111: unfortunately i don't know anything about the load balancer config, so someone else will have to help you out with that one | 21:11 |
kfox1111 | I had this working at one point.... For some reason, if I ip netns exec into the namespace, the load ballancer is working fine. If I try and curl to the load ballancer from one of the hosts the load ballancer talks to, it cant talk to the load ballancer. very strange. | 21:12 |
kfox1111 | I can ping the load ballancer though. | 21:12 |
*** carl_baldwin has joined #openstack-neutron | 21:12 | |
kevinbenton | marun, salv-orlando: did you get a chance to look at the 3rd patch uploaded by enikanorov__? | 21:14 |
*** jaypipes has joined #openstack-neutron | 21:14 | |
kfox1111 | yeah. the load balancer is not even seeing the request. | 21:15 |
*** rudrarugge has joined #openstack-neutron | 21:15 | |
*** thuc has quit IRC | 21:15 | |
*** thuc has joined #openstack-neutron | 21:16 | |
*** ijw has quit IRC | 21:16 | |
kfox1111 | yeah... doing the ping from qdhcp-the_network to the load ballancer fails too. | 21:17 |
rudrarugge | i have a couple of questions on the behavior of security groups in neutron | 21:19 |
*** mlavalle has joined #openstack-neutron | 21:19 | |
rudrarugge | "default" security group is created for a project and has two rules: #1 allow ingress from same SG #2 allow all egress | 21:19 |
rudrarugge | can these two rules be edited/deleted? | 21:20 |
kfox1111 | rudrarugge: yes. | 21:20 |
*** carlp has quit IRC | 21:21 | |
rudrarugge | openstack does not allow delete of default security group - is this understanding correct? | 21:23 |
kfox1111 | I'm not sure. I just know you can change whats in it just like any other group. | 21:23 |
*** tvardeman has quit IRC | 21:24 | |
*** carl_baldwin has quit IRC | 21:24 | |
rudrarugge | arosen: can you confirm this information for security groups. looking at the doc that you have published for port of SG from nova to quantum/neutron | 21:24 |
*** vkozhukalov has quit IRC | 21:25 | |
rudrarugge | what is the behavior if all rules from the default security group are deleted? | 21:25 |
*** banix has joined #openstack-neutron | 21:25 | |
rudrarugge | anyone? | 21:26 |
*** jorgem1 has joined #openstack-neutron | 21:27 | |
kfox1111 | I beleive its completely additive permissions. So 100% blocked I think. | 21:27 |
rudrarugge | thanks kfox1111 | 21:29 |
*** jorgem has quit IRC | 21:29 | |
*** banix has quit IRC | 21:29 | |
kfox1111 | sure. :) | 21:29 |
*** vu has quit IRC | 21:32 | |
*** dvorkinista has joined #openstack-neutron | 21:34 | |
rudrarugge | Based on the following link looks like the default rules in the default security group cannot be edited/deleted in neutron http://www.gossamer-threads.com/lists/openstack/dev/36779?page=last | 21:35 |
kfox1111 | I believe they are talking about the rules that all new tenant networks get. Not that you can't delete the rules in the security group after its created. But I don't think I've ever tried it. | 21:39 |
kfox1111 | Only one way to find out... :) | 21:39 |
*** aveiga has joined #openstack-neutron | 21:47 | |
*** networkstatic is now known as networkstatic_zZ | 21:51 | |
openstackgerrit | A change was merged to openstack/neutron: BigSwitch: Sync workaround for port del deadlock https://review.openstack.org/80688 | 21:52 |
rms_13 | rudrarugge: Do you think it would be a good idea to make default sg optional/configurable? | 21:54 |
rudrarugge | default sg is fine for each project and preventing delete of default sg is also fine | 21:55 |
rudrarugge | its the behavior of the rules that i am curious about: are the 2 default rules implicit or explicit. can users edit/delete the rules? what is the behavior if the rules are deleted | 21:56 |
kfox1111 | it should be the same as any other group. | 21:56 |
arosen | rudrarugge: kfox1111 that is correct if you remove all the egress rules an instance can no longer send out. | 21:56 |
kfox1111 | it just pre-create's it for the user from whta I understand. | 21:56 |
kfox1111 | Its like /etc/skel on a unix box. | 21:56 |
arosen | rudrarugge: users can delete all rules. | 21:57 |
kfox1111 | on user create, it just copies /etc/skel to /home/username | 21:57 |
kfox1111 | the user is free to do whatever they want with it afterwards. | 21:57 |
rudrarugge | arosen: if all default rules are deleted then its in locked down mode | 21:57 |
arosen | rudrarugge: yup | 21:57 |
rudrarugge | arosen: thanks | 21:58 |
arosen | rudrarugge: sure np! | 21:58 |
rudrarugge | arosen: can user delete default sg? | 21:58 |
arosen | rudrarugge: on the admin can | 21:58 |
arosen | every instance that's created gets a default security group | 21:58 |
*** jgrimm has quit IRC | 21:59 | |
arosen | if you precreate the ports in neutron though with --no-security-groups you can launch an instance without a security group. | 21:59 |
arosen | Nova by default addds the default sg | 21:59 |
rudrarugge | got it | 22:00 |
rudrarugge | thanks | 22:00 |
*** alagalah has joined #openstack-neutron | 22:00 | |
arosen | rudrarugge: same behavior as ec2 btw (or should be) | 22:02 |
arosen | i confirmed that it was when we were figuring out how it should work. | 22:02 |
manishg | arosen, what do you mean by 'preceate ports' ? do you mean when creating an instance I can specify now '--no-security-groups' and it will (via compute) translate into port-create (w/ no-security-group option)? | 22:05 |
arosen | manishg: i mean: neutron port-create --no-security-groups <network>; then nova boot --nic port-id=<port> | 22:07 |
arosen | manishg: otherwise nova adds them. | 22:07 |
*** marun has joined #openstack-neutron | 22:07 | |
manishg | arosen. got it. and for havana, there seems to be also 'port-update --no-security-groups' . wondering in case it wasn't precreated then would port-update also work? | 22:07 |
arosen | manishg: yea you can always update the port(s) and remove them or add them at anytime. | 22:08 |
manishg | don't recall the no-security-groups option in grizzly. | 22:08 |
arosen | manishg: i don't think we had added those bits to the python-neutronclient but it would have worked via curl in grizzly as well. | 22:08 |
manishg | arosen: ah, ok. thanks. would be good to have an option for this when creating an instance instead of pre-creating and/or updating. do you know if there is any effort in that direction? | 22:10 |
*** ajo has joined #openstack-neutron | 22:13 | |
*** pcm_ has quit IRC | 22:13 | |
*** dvorkinista has quit IRC | 22:15 | |
*** banix has joined #openstack-neutron | 22:18 | |
arosen | manishg: yup, there is a way to do this if you're plugin implements the port_security_enabled extension | 22:20 |
*** jorgem1 has quit IRC | 22:20 | |
*** Sukhdev has joined #openstack-neutron | 22:20 | |
arosen | basically you can create a network with; neutron net-create --port-security-enabled=False blah; then ports created on it won't have a security group | 22:20 |
arosen | also you're able to launch instances on it even though it doesn't have a subnet. | 22:20 |
arosen | manishg: anyways, in nova's v3 api we're not going to have nova proxy calls and create ports in neutron. When this happens we'll be able to let neutron decide if something needs a security group or not. | 22:21 |
arosen | right now nova automatically adds 'default' for you. | 22:22 |
arosen | if you don't provide one. | 22:22 |
*** overlayer has quit IRC | 22:22 | |
*** sungju_ has joined #openstack-neutron | 22:23 | |
*** ajo has quit IRC | 22:24 | |
manishg | arosen: thanks. yeah, that's what we noticed (that nova adds the default automatically) and so had to change code to avoid using security groups. Didn't see any option in net-create for disabling security groups at the time (in grizzly) for the subnet. | 22:25 |
*** sungju- has joined #openstack-neutron | 22:25 | |
arosen | manishg: yea i believe this is only supported if you're using NSX | 22:25 |
*** nati_ueno has joined #openstack-neutron | 22:26 | |
*** sungju_ has quit IRC | 22:27 | |
*** armax has quit IRC | 22:28 | |
*** blogan has quit IRC | 22:28 | |
*** gdubreui has joined #openstack-neutron | 22:29 | |
*** thuc_ has joined #openstack-neutron | 22:31 | |
*** thuc has quit IRC | 22:34 | |
*** thuc_ has quit IRC | 22:35 | |
*** thuc has joined #openstack-neutron | 22:36 | |
*** alagalah has quit IRC | 22:39 | |
*** thuc has quit IRC | 22:45 | |
*** thuc has joined #openstack-neutron | 22:46 | |
*** manishg has quit IRC | 22:47 | |
*** changbl has quit IRC | 22:47 | |
*** dfarrell07 has quit IRC | 22:54 | |
*** rudrarugge has quit IRC | 22:57 | |
*** jobewan has quit IRC | 22:57 | |
*** ajo has joined #openstack-neutron | 22:58 | |
*** networkstatic_zZ is now known as networkstatic | 22:58 | |
*** salv-orlando has quit IRC | 23:06 | |
*** salv-orlando has joined #openstack-neutron | 23:07 | |
*** zzelle has joined #openstack-neutron | 23:07 | |
*** mlavalle has quit IRC | 23:12 | |
*** armax has joined #openstack-neutron | 23:13 | |
*** ijw has joined #openstack-neutron | 23:15 | |
*** ajo has quit IRC | 23:15 | |
*** thedodd has quit IRC | 23:17 | |
*** banix has quit IRC | 23:19 | |
*** claudiub_ has quit IRC | 23:30 | |
kfox1111 | ok. so the lbaas instance seems to have the default security group associated with it, which is totally wrong. :/ | 23:33 |
*** mwagner_notHere has quit IRC | 23:37 | |
openstackgerrit | Aaron Rosen proposed a change to openstack/python-neutronclient: Allow user ID for authentication https://review.openstack.org/78596 | 23:37 |
*** banix has joined #openstack-neutron | 23:43 | |
kfox1111 | yeah. thats the problem. | 23:44 |
kfox1111 | how do you port-update the security group? | 23:44 |
kfox1111 | 400-{u'NeutronError': {u'message': u"Invalid input for operation: '6' is not an integer or uuid.", u'type': u'InvalidInput', u'detail': u''}} | 23:44 |
*** aveiga has quit IRC | 23:49 | |
*** alagalah has joined #openstack-neutron | 23:50 | |
openstackgerrit | Nachi Ueno proposed a change to openstack/neutron: Add enable_security_group option https://review.openstack.org/67281 | 23:50 |
*** alagalah has quit IRC | 23:54 | |
*** alexpilotti has quit IRC | 23:55 | |
*** zzelle has quit IRC | 23:56 | |
*** nati_ueno has quit IRC | 23:56 | |
*** nati_ueno has joined #openstack-neutron | 23:57 |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!