| *** rarcea has quit IRC | 00:10 | |
| *** zeih has joined #openstack-operators | 00:14 | |
| *** dminer has quit IRC | 00:14 | |
| *** zeih has quit IRC | 00:18 | |
| *** haukebruno has joined #openstack-operators | 00:32 | |
| *** vsedelnik has joined #openstack-operators | 00:48 | |
| *** saneax is now known as saneax_AFK | 00:50 | |
| *** vsedelnik has quit IRC | 00:53 | |
| *** zeih has joined #openstack-operators | 01:00 | |
| *** zeih has quit IRC | 01:05 | |
| *** piet has quit IRC | 01:07 | |
| *** lhcheng has quit IRC | 01:16 | |
| *** ducttape_ has joined #openstack-operators | 01:24 | |
| *** ducttape_ has quit IRC | 01:25 | |
| *** ducttape_ has joined #openstack-operators | 01:25 | |
| *** jmckind has joined #openstack-operators | 01:28 | |
| *** ducttape_ has quit IRC | 01:31 | |
| *** VW has joined #openstack-operators | 01:48 | |
| *** vsedelnik has joined #openstack-operators | 01:49 | |
| *** dmsimard has quit IRC | 01:50 | |
| *** dmsimard has joined #openstack-operators | 01:51 | |
| *** VW has quit IRC | 01:53 | |
| *** apoorvad has quit IRC | 01:53 | |
| *** vsedelnik has quit IRC | 01:53 | |
| *** zeih has joined #openstack-operators | 02:01 | |
| *** esker has joined #openstack-operators | 02:01 | |
| *** zeih has quit IRC | 02:05 | |
| *** esker has quit IRC | 02:07 | |
| *** vilobhmm11 has quit IRC | 02:09 | |
| *** Vinsh has quit IRC | 02:19 | |
| *** fawadkhaliq has quit IRC | 02:20 | |
| *** zengine has joined #openstack-operators | 02:22 | |
| *** jmckind is now known as jmckind_ | 02:24 | |
| *** ducttape_ has joined #openstack-operators | 02:25 | |
| *** apoorvad has joined #openstack-operators | 02:36 | |
| *** esker has joined #openstack-operators | 02:43 | |
| *** esker has quit IRC | 02:48 | |
| *** vsedelnik has joined #openstack-operators | 02:50 | |
| *** piet has joined #openstack-operators | 02:53 | |
| *** vsedelnik has quit IRC | 02:54 | |
| *** cdelatte has joined #openstack-operators | 02:59 | |
| *** zeih has joined #openstack-operators | 03:02 | |
| *** zeih has quit IRC | 03:07 | |
| *** piet has quit IRC | 03:13 | |
| *** piet has joined #openstack-operators | 03:14 | |
| *** vilobhmm11 has joined #openstack-operators | 03:15 | |
| *** esker has joined #openstack-operators | 03:18 | |
| *** vilobhmm111 has joined #openstack-operators | 03:22 | |
| *** vilobhmm11 has quit IRC | 03:25 | |
| *** zengine has quit IRC | 03:28 | |
| *** ducttape_ has quit IRC | 03:36 | |
| *** fawadkhaliq has joined #openstack-operators | 03:38 | |
| *** vilobhmm111 has quit IRC | 03:39 | |
| *** dims has quit IRC | 03:40 | |
| *** ducttape_ has joined #openstack-operators | 03:42 | |
| *** dims has joined #openstack-operators | 03:43 | |
| *** dims has quit IRC | 03:44 | |
| *** apoorvad has quit IRC | 03:45 | |
| *** esker has quit IRC | 03:49 | |
| *** vsedelnik has joined #openstack-operators | 03:50 | |
| *** vsedelnik has quit IRC | 03:55 | |
| *** ducttape_ has quit IRC | 03:56 | |
| *** zeih has joined #openstack-operators | 04:02 | |
| *** zeih has quit IRC | 04:07 | |
| *** piet has quit IRC | 04:27 | |
| *** esker has joined #openstack-operators | 04:27 | |
| *** openstack has joined #openstack-operators | 04:29 | |
| *** openstack has quit IRC | 04:30 | |
| *** openstack has joined #openstack-operators | 04:32 | |
| *** jlk has joined #openstack-operators | 04:33 | |
| *** jlk has quit IRC | 04:33 | |
| *** jlk has joined #openstack-operators | 04:33 | |
| *** esker has quit IRC | 04:33 | |
| *** kfjohnson_ has quit IRC | 04:34 | |
| *** openstack has quit IRC | 04:34 | |
| *** openstack has joined #openstack-operators | 14:07 | |
| *** cdelatte has quit IRC | 14:08 | |
| *** ctrath has joined #openstack-operators | 14:09 | |
| *** zeih has joined #openstack-operators | 14:10 | |
| *** dminer has joined #openstack-operators | 14:11 | |
| *** VW has quit IRC | 14:14 | |
| *** mriedem has joined #openstack-operators | 14:15 | |
| *** VW has joined #openstack-operators | 14:15 | |
| *** piet has quit IRC | 14:16 | |
| *** ducttape_ has joined #openstack-operators | 14:21 | |
| *** eddima has quit IRC | 14:22 | |
| *** ducttape_ has quit IRC | 14:26 | |
| *** dansmith is now known as superdan | 14:29 | |
| *** eddima has joined #openstack-operators | 14:32 | |
| *** piet has joined #openstack-operators | 14:36 | |
| *** openstackgerrit_ has quit IRC | 14:36 | |
| *** openstackgerrit_ has joined #openstack-operators | 14:37 | |
| *** ducttape_ has joined #openstack-operators | 14:56 | |
| *** piet has quit IRC | 14:56 | |
| *** admin0 has quit IRC | 15:00 | |
| *** admin0 has joined #openstack-operators | 15:00 | |
| *** piet has joined #openstack-operators | 15:04 | |
| *** zeih has quit IRC | 15:07 | |
| *** vsedelnik has quit IRC | 15:09 | |
| *** markvoelker has quit IRC | 15:10 | |
| *** markvoelker has joined #openstack-operators | 15:10 | |
| *** krobzaur has quit IRC | 15:11 | |
| *** VW has quit IRC | 15:13 | |
| *** esker has joined #openstack-operators | 15:14 | |
| *** esker has quit IRC | 15:15 | |
| *** esker has joined #openstack-operators | 15:15 | |
| *** bsilverman^AFK is now known as bsilverman | 15:19 | |
| *** piet has quit IRC | 15:21 | |
| *** david_lyle__ has joined #openstack-operators | 15:34 | |
| klindgren | Has anyone else experieince the nuetron-api stop responding to api requests and the last thing logged was a bunch of: [req-5b1dd446-b139-48b0-a4a1-8d0d81c07bcf None None] Notify callbacks for port, after_update | 15:35 |
|---|---|---|
| klindgren | logs | 15:35 |
| klindgren | then it bascially jsut goes dead | 15:35 |
| *** simon-AS559 has joined #openstack-operators | 15:36 | |
| klindgren | we have had this happen twice in the past week, fix is to restart neutron-server and requests magically start flowing again. The issue is that we have 3 api servers, just one of them doing this bascailly kills everything | 15:36 |
| *** david-lyle has quit IRC | 15:37 | |
| *** regXboi has joined #openstack-operators | 15:40 | |
| *** mdorman has joined #openstack-operators | 15:45 | |
| *** dims is now known as dimsum_ | 15:53 | |
| *** k_stev has joined #openstack-operators | 15:55 | |
| klindgren | right before this happens I see: 8:47] | 15:55 |
| klindgren | 2016-03-03 20:56:50.950 392 INFO neutron.wsgi [-] (392) wsgi exited, is_accepting=True | 15:55 |
| klindgren | 2016-03-03 20:57:01.298 393 INFO neutron.wsgi [-] (393) wsgi exited, is_accepting=True | 15:55 |
| klindgren | [8:47] | 15:55 |
| klindgren | 2016-03-03 20:59:13.857 394 INFO neutron.wsgi [-] (394) wsgi exited, is_accepting=True | 15:55 |
| klindgren | 2016-03-03 20:59:37.068 390 INFO neutron.wsgi [-] (390) wsgi exited, is_accepting=True | 15:55 |
| klindgren | looks like the wsgi process inside neutron died | 15:55 |
| *** mriedem is now known as mriedem_meeting | 15:55 | |
| *** belmoreira has quit IRC | 16:07 | |
| haukebruno | klindgren, I don't know about the issue you are facing but maybe enabling verbose and debug will give more information | 16:10 |
| haukebruno | (hopefully_ | 16:10 |
| klindgren | we are running verbose - the only issue with debug is the sheer volume of logs thats comes out. But I can try it - what sucks is I have exactly 0 things to go on why the wsgi side would die or what causes it. Its never happened before then today ~5 days apart from the first time it happened. | 16:12 |
| *** busterswt has joined #openstack-operators | 16:13 | |
| klindgren | So getting nailed twice in 1 week hurts | 16:14 |
| *** Marga_ has quit IRC | 16:17 | |
| *** david_lyle__ is now known as david_lyle | 16:17 | |
| *** eddima has quit IRC | 16:18 | |
| *** rarcea has quit IRC | 16:21 | |
| *** racedo has joined #openstack-operators | 16:22 | |
| *** piet has joined #openstack-operators | 16:27 | |
| *** armax has quit IRC | 16:29 | |
| haukebruno | klindgren, it is just a try. imho it is worth it to provide a lot of diskspace for logs to openstack services :p | 16:29 |
| *** armax has joined #openstack-operators | 16:29 | |
| *** krobzaur has joined #openstack-operators | 16:32 | |
| dimsum_ | klindgren : liberty? earlier? | 16:33 |
| klindgren | dimsum_, kilo | 16:34 |
| *** haukebruno has quit IRC | 16:34 | |
| klindgren | we were/are focusing on getting some other services started up before focusing on reving everything up to a newer version | 16:35 |
| dimsum_ | klindgren : understood. seems to be coming from eventlet - https://github.com/eventlet/eventlet/blob/master/eventlet/wsgi.py#L885 | 16:39 |
| dimsum_ | klindgren : looking at the "while is_accepting" loop looks like there may have been some exceptions right there causing it to break out and exit | 16:39 |
| *** mriedem_meeting is now known as mriedem | 16:40 | |
| dimsum_ | klindgren : any network event in syslog or something around the time you hit "wsgi exited"? | 16:41 |
| klindgren | the interesting to me atleast thing is that it still listens on the socket for requests | 16:41 |
| klindgren | nope syslog/dmesg clean in both of those events | 16:41 |
| klindgren | they happened on different servers | 16:41 |
| klindgren | the other 2 servers continued on without any issue and they are in the same rack/set of switches | 16:42 |
| dimsum_ | klindgren : at least now you can add more logs in one spot to see if it happens again :) | 16:42 |
| klindgren | waiting to see if our security team was doing a security team against the servers/vip at that time | 16:43 |
| klindgren | we have seen that blow up other services before | 16:43 |
| dimsum_ | cool thanks | 16:43 |
| klindgren | security scan** | 16:45 |
| *** apoorvad has joined #openstack-operators | 16:49 | |
| *** simon-AS559 has quit IRC | 16:50 | |
| *** zengine has quit IRC | 16:51 | |
| *** rcernin has quit IRC | 16:51 | |
| klindgren | security scans did not happen at that time | 16:59 |
| *** matrohon has quit IRC | 17:02 | |
| *** admin0 has quit IRC | 17:02 | |
| *** cfarquhar has quit IRC | 17:03 | |
| *** zengine has joined #openstack-operators | 17:04 | |
| *** zengine_ has joined #openstack-operators | 17:06 | |
| *** zengine has quit IRC | 17:09 | |
| *** zengine_ is now known as zengine | 17:09 | |
| dimsum_ | klindgren :( | 17:14 |
| *** racedo has quit IRC | 17:18 | |
| *** racedo has joined #openstack-operators | 17:18 | |
| *** VW has joined #openstack-operators | 17:18 | |
| *** pilgrimstack has quit IRC | 17:30 | |
| *** racedo has quit IRC | 17:36 | |
| *** VW has quit IRC | 17:43 | |
| *** vilobhmm11 has joined #openstack-operators | 17:46 | |
| *** vilobhmm111 has joined #openstack-operators | 17:49 | |
| *** vilobhmm11 has quit IRC | 17:51 | |
| *** simon-AS559 has joined #openstack-operators | 17:54 | |
| *** gyee has joined #openstack-operators | 17:55 | |
| *** simon-AS5591 has joined #openstack-operators | 17:55 | |
| *** simon-AS559 has quit IRC | 17:58 | |
| *** admin0 has joined #openstack-operators | 18:02 | |
| *** esker has quit IRC | 18:02 | |
| *** electrofelix has quit IRC | 18:03 | |
| *** derekh has quit IRC | 18:11 | |
| *** permalac has quit IRC | 18:12 | |
| *** vilobhmm111 has quit IRC | 18:16 | |
| *** lhcheng has joined #openstack-operators | 18:19 | |
| *** piet has quit IRC | 18:19 | |
| *** piet has joined #openstack-operators | 18:21 | |
| *** ducttape_ has quit IRC | 18:26 | |
| *** portante has quit IRC | 18:29 | |
| *** shakamunyi has joined #openstack-operators | 18:29 | |
| *** portante has joined #openstack-operators | 18:31 | |
| *** zeih_ has joined #openstack-operators | 18:33 | |
| *** britthouser has quit IRC | 18:35 | |
| *** zeih_ has quit IRC | 18:38 | |
| *** ducttape_ has joined #openstack-operators | 18:38 | |
| *** piet has quit IRC | 18:38 | |
| *** pcaruana has quit IRC | 18:41 | |
| *** piet has joined #openstack-operators | 18:41 | |
| *** mriedem has quit IRC | 18:47 | |
| *** mriedem has joined #openstack-operators | 18:48 | |
| *** vilobhmm11 has joined #openstack-operators | 18:48 | |
| admin0 | if I want MPLS vpn , what SDN is the way to go ? | 18:58 |
| klindgren | and it just happened again. So its happened once on each server. | 19:12 |
| *** apoorvad has quit IRC | 19:22 | |
| *** gyee has quit IRC | 19:23 | |
| *** apoorvad has joined #openstack-operators | 19:26 | |
| *** k_stev has quit IRC | 19:28 | |
| *** gyee has joined #openstack-operators | 19:31 | |
| *** ctrath has quit IRC | 19:34 | |
| *** piet has quit IRC | 19:35 | |
| *** piet has joined #openstack-operators | 19:36 | |
| *** ctrath has joined #openstack-operators | 19:37 | |
| *** vsedelnik has joined #openstack-operators | 19:38 | |
| *** vilobhmm111 has joined #openstack-operators | 19:42 | |
| *** vilobhmm11 has quit IRC | 19:45 | |
| *** admin0 has quit IRC | 19:52 | |
| *** fawadkhaliq has joined #openstack-operators | 19:52 | |
| *** pilgrimstack has joined #openstack-operators | 19:55 | |
| *** pilgrimstack has quit IRC | 19:58 | |
| *** pilgrimstack has joined #openstack-operators | 19:58 | |
| *** pilgrimstack has quit IRC | 20:00 | |
| *** k_stev has joined #openstack-operators | 20:05 | |
| *** derekh has joined #openstack-operators | 20:06 | |
| *** admin0 has joined #openstack-operators | 20:09 | |
| *** Marga_ has joined #openstack-operators | 20:10 | |
| *** piet has quit IRC | 20:11 | |
| *** vilobhmm111 has quit IRC | 20:13 | |
| *** vilobhmm11 has joined #openstack-operators | 20:13 | |
| *** vsedelnik has quit IRC | 20:14 | |
| *** ctrath has quit IRC | 20:15 | |
| *** regXboi has quit IRC | 20:16 | |
| *** derekh has quit IRC | 20:18 | |
| *** ctrath has joined #openstack-operators | 20:22 | |
| *** piet has joined #openstack-operators | 20:25 | |
| *** rarcea has joined #openstack-operators | 20:26 | |
| *** bvandenh has quit IRC | 20:33 | |
| *** zengine has quit IRC | 20:36 | |
| *** k_stev has quit IRC | 20:40 | |
| *** cfarquhar has joined #openstack-operators | 20:40 | |
| *** cfarquhar has joined #openstack-operators | 20:40 | |
| *** k_stev has joined #openstack-operators | 20:42 | |
| *** k_stev has quit IRC | 20:51 | |
| *** delattec has quit IRC | 20:58 | |
| *** k_stev has joined #openstack-operators | 21:07 | |
| *** piet has quit IRC | 21:07 | |
| *** apoorvad has quit IRC | 21:10 | |
| *** rarcea has quit IRC | 21:12 | |
| *** vsedelnik has joined #openstack-operators | 21:14 | |
| *** vsedelnik has quit IRC | 21:19 | |
| *** dmsimard has quit IRC | 21:23 | |
| *** k_stev has left #openstack-operators | 21:35 | |
| *** apoorvad has joined #openstack-operators | 21:36 | |
| *** pcaruana has joined #openstack-operators | 21:58 | |
| *** vsedelnik has joined #openstack-operators | 22:15 | |
| *** mdorman has quit IRC | 22:18 | |
| *** vsedelnik has quit IRC | 22:20 | |
| *** lhcheng has quit IRC | 22:25 | |
| *** busterswt has quit IRC | 22:28 | |
| *** lhcheng has joined #openstack-operators | 22:33 | |
| *** piet has joined #openstack-operators | 22:42 | |
| *** ctrath has quit IRC | 22:43 | |
| *** mriedem has quit IRC | 22:46 | |
| *** vsedelnik has joined #openstack-operators | 23:16 | |
| *** admin0 has quit IRC | 23:21 | |
| *** piet has quit IRC | 23:21 | |
| *** vsedelnik has quit IRC | 23:22 | |
| *** Trident has quit IRC | 23:33 | |
| *** fawadkhaliq has quit IRC | 23:38 | |
| *** fawadkhaliq has joined #openstack-operators | 23:39 | |
| *** ctrath has joined #openstack-operators | 23:52 | |
| *** ctrath has quit IRC | 23:52 | |
| *** ducttape_ has quit IRC | 23:55 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!