| *** openstackgerrit has joined #openstack-zaqar | 00:24 | |
| openstackgerrit | Merged openstack/zaqar-specs: Removes unnecessary utf-8 coding https://review.openstack.org/414797 | 00:24 |
|---|---|---|
| openstackgerrit | Merged openstack/zaqar-specs: H803 hacking have been deprecated https://review.openstack.org/412424 | 00:24 |
| openstackgerrit | Merged openstack/zaqar-specs: Changed the home-page link https://review.openstack.org/390076 | 00:24 |
| *** sebasmag_ has joined #openstack-zaqar | 00:42 | |
| *** sebasmagri has quit IRC | 00:44 | |
| *** sebasmag_ has quit IRC | 00:44 | |
| *** wanghao has joined #openstack-zaqar | 00:48 | |
| *** robcresswell has quit IRC | 01:13 | |
| *** flaper87 has quit IRC | 01:13 | |
| *** wxy_ has quit IRC | 01:13 | |
| *** andreaf has quit IRC | 01:13 | |
| *** kgriffs has quit IRC | 01:13 | |
| *** boris-42 has quit IRC | 01:13 | |
| *** kgriffs has joined #openstack-zaqar | 01:14 | |
| *** andreaf has joined #openstack-zaqar | 01:16 | |
| *** robcresswell has joined #openstack-zaqar | 01:19 | |
| *** boris-42 has joined #openstack-zaqar | 01:25 | |
| *** wxy_ has joined #openstack-zaqar | 01:29 | |
| *** wxy_ has quit IRC | 01:34 | |
| *** wxy has joined #openstack-zaqar | 01:38 | |
| *** yanyanhu has joined #openstack-zaqar | 01:53 | |
| *** yanyanhu_ has joined #openstack-zaqar | 02:02 | |
| *** yanyanhu has quit IRC | 02:04 | |
| *** mpanetta has quit IRC | 02:20 | |
| *** Eva-i has joined #openstack-zaqar | 02:25 | |
| *** mars has quit IRC | 03:06 | |
| *** mars has joined #openstack-zaqar | 03:24 | |
| *** lhx_ has joined #openstack-zaqar | 03:51 | |
| *** flwang1 has quit IRC | 03:56 | |
| *** itisha has quit IRC | 04:02 | |
| *** rbrady is now known as rbrady-afk | 04:26 | |
| *** wxy has quit IRC | 04:28 | |
| *** wanghao_ has joined #openstack-zaqar | 04:39 | |
| *** wanghao has quit IRC | 04:41 | |
| *** sebasmagri has joined #openstack-zaqar | 04:41 | |
| *** wanghao has joined #openstack-zaqar | 04:41 | |
| *** wanghao_ has quit IRC | 04:44 | |
| *** lhx_ has quit IRC | 05:03 | |
| *** ryansb has quit IRC | 05:31 | |
| *** lhx_ has joined #openstack-zaqar | 06:09 | |
| *** wanghao_ has joined #openstack-zaqar | 06:30 | |
| *** wanghao has quit IRC | 06:33 | |
| *** rcernin has quit IRC | 06:55 | |
| *** tesseract has joined #openstack-zaqar | 07:09 | |
| *** pcaruana has joined #openstack-zaqar | 07:10 | |
| *** rcernin has joined #openstack-zaqar | 07:13 | |
| *** nadya has joined #openstack-zaqar | 07:21 | |
| *** nadya has quit IRC | 07:38 | |
| *** nadya has joined #openstack-zaqar | 07:49 | |
| *** nadya has quit IRC | 07:56 | |
| *** nadya has joined #openstack-zaqar | 08:00 | |
| *** nadya has quit IRC | 08:07 | |
| *** nadya has joined #openstack-zaqar | 08:07 | |
| *** nadya has quit IRC | 08:08 | |
| *** robcresswell has quit IRC | 08:31 | |
| *** kgriffs has quit IRC | 08:31 | |
| *** harlowja has quit IRC | 08:31 | |
| *** jtomasek has quit IRC | 08:31 | |
| *** openstackgerrit has quit IRC | 08:31 | |
| *** Eva-i has quit IRC | 08:31 | |
| *** lhx_ has quit IRC | 08:31 | |
| *** kong has quit IRC | 08:31 | |
| *** godlike has quit IRC | 08:31 | |
| *** boris-42 has quit IRC | 08:31 | |
| *** rcernin has quit IRC | 08:31 | |
| *** yanyanhu_ has quit IRC | 08:31 | |
| *** Guest66666 has quit IRC | 08:31 | |
| *** rbrady-afk has quit IRC | 08:31 | |
| *** sebasmagri has quit IRC | 08:31 | |
| *** tonyb has quit IRC | 08:31 | |
| *** wanghao_ has quit IRC | 08:31 | |
| *** pcaruana has quit IRC | 08:31 | |
| *** tesseract has quit IRC | 08:31 | |
| *** andreaf has quit IRC | 08:31 | |
| *** AAzza has quit IRC | 08:31 | |
| *** mars has quit IRC | 08:31 | |
| *** david-lyle has quit IRC | 08:31 | |
| *** nadya has joined #openstack-zaqar | 08:37 | |
| *** yanyanhu_ has joined #openstack-zaqar | 08:37 | |
| *** lhx_ has joined #openstack-zaqar | 08:37 | |
| *** rcernin has joined #openstack-zaqar | 08:37 | |
| *** pcaruana has joined #openstack-zaqar | 08:37 | |
| *** tesseract has joined #openstack-zaqar | 08:37 | |
| *** wanghao_ has joined #openstack-zaqar | 08:37 | |
| *** sebasmagri has joined #openstack-zaqar | 08:37 | |
| *** mars has joined #openstack-zaqar | 08:37 | |
| *** Eva-i has joined #openstack-zaqar | 08:37 | |
| *** boris-42 has joined #openstack-zaqar | 08:37 | |
| *** robcresswell has joined #openstack-zaqar | 08:37 | |
| *** andreaf has joined #openstack-zaqar | 08:37 | |
| *** kgriffs has joined #openstack-zaqar | 08:37 | |
| *** openstackgerrit has joined #openstack-zaqar | 08:37 | |
| *** harlowja has joined #openstack-zaqar | 08:37 | |
| *** AAzza has joined #openstack-zaqar | 08:37 | |
| *** jtomasek has joined #openstack-zaqar | 08:37 | |
| *** tonyb has joined #openstack-zaqar | 08:37 | |
| *** Guest66666 has joined #openstack-zaqar | 08:37 | |
| *** kong has joined #openstack-zaqar | 08:37 | |
| *** godlike has joined #openstack-zaqar | 08:37 | |
| *** david-lyle has joined #openstack-zaqar | 08:37 | |
| *** rbrady-afk has joined #openstack-zaqar | 08:38 | |
| *** ameade has quit IRC | 08:40 | |
| *** nadya has quit IRC | 08:44 | |
| *** wanghao_ has quit IRC | 09:06 | |
| *** wanghao has joined #openstack-zaqar | 09:07 | |
| *** ameade has joined #openstack-zaqar | 09:12 | |
| *** wanghao has quit IRC | 09:28 | |
| *** nadya has joined #openstack-zaqar | 09:49 | |
| *** yanyanhu_ has quit IRC | 10:23 | |
| *** lhx_ has quit IRC | 10:28 | |
| *** lhx_ has joined #openstack-zaqar | 10:31 | |
| *** AAzza has quit IRC | 10:43 | |
| *** AAzza has joined #openstack-zaqar | 10:43 | |
| *** mars has quit IRC | 11:39 | |
| *** ryansb has joined #openstack-zaqar | 13:01 | |
| *** ryansb has quit IRC | 13:01 | |
| *** ryansb has joined #openstack-zaqar | 13:01 | |
| *** rbrady-afk is now known as rbrady | 13:07 | |
| *** lhx_ has quit IRC | 13:16 | |
| *** nadya has quit IRC | 13:38 | |
| *** AAzza has quit IRC | 13:59 | |
| *** AAzza has joined #openstack-zaqar | 14:00 | |
| *** rbrady has quit IRC | 14:02 | |
| *** rbrady has joined #openstack-zaqar | 14:02 | |
| *** lhx_ has joined #openstack-zaqar | 14:19 | |
| *** nadya has joined #openstack-zaqar | 14:24 | |
| *** nadya has quit IRC | 14:25 | |
| *** mpanetta has joined #openstack-zaqar | 15:05 | |
| *** mpanetta has quit IRC | 15:09 | |
| *** mpanetta has joined #openstack-zaqar | 15:09 | |
| *** nadya has joined #openstack-zaqar | 15:38 | |
| *** sebasmagri has quit IRC | 15:42 | |
| *** nadya has quit IRC | 15:52 | |
| *** rcernin has quit IRC | 15:53 | |
| *** sebasmagri has joined #openstack-zaqar | 16:03 | |
| *** sebasmagri has quit IRC | 16:05 | |
| *** sebasmagri has joined #openstack-zaqar | 16:06 | |
| *** nadya has joined #openstack-zaqar | 16:24 | |
| *** dtrainor has joined #openstack-zaqar | 16:35 | |
| *** itisha has joined #openstack-zaqar | 16:39 | |
| dtrainor | howdy. i'm troubleshooting some reliability issues between the UI component and Zaqar. Sometimes we see that some messages don't appear to make it to the UI, so data in the UI does not get refreshed. There's no pattern or predictability to it, it just happens some times. Most often, it happens when there's a lot of data requested by the UI (during introspection, or deployment progress polling). I'm trying to repeat this wi | 16:40 |
| dtrainor | th zaqar debugging enabled, and saw something interesting that seems to happen around the same time I expect to see an update in the UI, but don't. I'm having trouble interpreting the message, but this may be due to my inexperience with zaqar http://paste.openstack.org/show/594134/ | 16:40 |
| dtrainor | we've noticed these inconsistencies more on high latency connections, if that helps | 16:40 |
| *** tesseract has quit IRC | 16:47 | |
| dtrainor | jtomasek, this is related to the message reliability issues we were discussing | 16:52 |
| *** lhx_ has quit IRC | 16:58 | |
| jtomasek | rbrady maybe? ^ | 16:59 |
| rbrady | dtrainor: when this occurs - is the zaqar service crashing? | 17:01 |
| rbrady | wondering if that's the source of the "connection refused" message | 17:01 |
| dtrainor | it does not crash, nope | 17:03 |
| dtrainor | i was trying to find some pattern or other effects of this, but I can't seem to find anything | 17:05 |
| dtrainor | i'm reading up on connection pooling, it feels like we're hitting a connection pool limit or something | 17:14 |
| dtrainor | causing some service to actively refuse new connections, and messages end up getting passively dropped (from the viewpoint of the ui, anyway) | 17:14 |
| *** nadya has quit IRC | 17:24 | |
| rbrady | dtrainor: unfortunately I don't have any answers today. Please document as much as possible and file a bug. it will be interesting to see if it's zaqar or if it's another component that is the root cause | 17:29 |
| dtrainor | will do! thanks! | 17:29 |
| *** nadya has joined #openstack-zaqar | 17:45 | |
| *** pcaruana has quit IRC | 18:01 | |
| *** sebasmag_ has joined #openstack-zaqar | 18:16 | |
| *** sebasmagri has quit IRC | 18:16 | |
| *** rbrady is now known as rbrady-afk | 18:47 | |
| *** AAzza has quit IRC | 18:50 | |
| *** AAzza has joined #openstack-zaqar | 18:53 | |
| *** nadya has quit IRC | 19:35 | |
| *** rbrady-afk is now known as rbrady | 19:54 | |
| *** Eva-i has quit IRC | 20:23 | |
| *** Eva-i has joined #openstack-zaqar | 21:17 | |
| therve | dtrainor, Still around? I can help maybe | 21:50 |
| dtrainor | i sure am | 21:50 |
| therve | I wrote that code, so I know a little bit the quirks :) | 21:51 |
| dtrainor | haha so you "know a guy" | 21:51 |
| therve | The WSGI server sends a HTTP request to the websocket server when you have a subscriber | 21:51 |
| dtrainor | right, that's what i'm gathering | 21:51 |
| dtrainor | (my background in websockets is a little weak, learning a lot today) | 21:51 |
| therve | The websocket server binds a random port at startup | 21:52 |
| therve | Which it uses to tell the wsgi server "hey, talk to me there" | 21:52 |
| therve | In your logs, we can see 2 different ports | 21:52 |
| therve | Which would mean that the websocket server got restarted (I think?) | 21:52 |
| dtrainor | i see. | 21:53 |
| therve | Unless you have a multinode architecture, but I doubt it on the undercloud | 21:53 |
| dtrainor | in that error i pasted, I only see one port being used? | 21:53 |
| therve | In the "Notifying subscriber..." line you can see the other | 21:53 |
| therve | http://undercloud.localdomain:35674 | 21:53 |
| dtrainor | ooh ok i see it, sorry. | 21:54 |
| dtrainor | where does the wsgi server run? | 21:54 |
| dtrainor | is it part of zaqar-server? | 21:55 |
| therve | You have 2 zaqar-server instances I believe on the undercloud? | 21:55 |
| therve | One is wsgi, the other websocket | 21:55 |
| dtrainor | that looks correct, yep | 21:55 |
| dtrainor | ah ok, so zaqar-server@1 service is my websocket service | 21:55 |
| dtrainor | transport=websocket | 21:55 |
| therve | Yep | 21:56 |
| dtrainor | ok, that explains a lot. | 21:56 |
| therve | Now, I don't believe that error ought to be fatal | 21:57 |
| therve | IE it shouldn't prevent the message to be sent to the subscriber correctly | 21:57 |
| *** sebasmag_ has quit IRC | 21:58 | |
| dtrainor | it doesn't appear to be fatal to zaqar, neither the websocket nor the wsgi service restart or crash. in fact i believe it's only temproary, as subsequent requests are likely to work | 21:58 |
| therve | But you observe message being lost at that time? | 21:59 |
| dtrainor | at that time, yep. if i force the ui to refresh, either by forcing it or just hitting f5, the message will likely make it to ui | 21:59 |
| therve | Hum | 22:00 |
| dtrainor | having limted experience troubleshooting zaqar, the only method i've been working with to test this is refreshing the ui to make it re-request the data | 22:00 |
| dtrainor | i'm sure there are more robust testing tools available, and i'll find 'em, just wanted to make sure i was going down the right path | 22:01 |
| therve | dtrainor, When you refresh the UI, you fetch the messages from the queue, no? | 22:01 |
| therve | IE it doesn't use the subscription? | 22:01 |
| dtrainor | you know, i can't give you a confident answer on that. | 22:02 |
| dtrainor | maybe jtomasek can if he's around | 22:02 |
| therve | Regarding the server being restarted, do you know why that would be the case? | 22:03 |
| therve | If you have the full logs of both processes that'd be helpful | 22:05 |
| dtrainor | i've been looking at logs and can't confirm that the server is being restarted, rbrady was asking about this earlier but i saw no indication | 22:05 |
| dtrainor | sure, i can pull more logs | 22:05 |
| therve | I don't think we can get into that particular situation without some kind of restart, but we never know | 22:07 |
| dtrainor | understood - one moment, pulling logs | 22:07 |
| dtrainor | going through the process again to generate fresh logs | 22:10 |
| dtrainor | of course, now when i try to reproduce it, it works | 22:14 |
| therve | :) | 22:14 |
| therve | I got to go now, but please open a bug, at least with the traceback you gave, I'll have a look | 22:15 |
| dtrainor | https://bugzilla.redhat.com/show_bug.cgi?id=1410894 :) | 22:15 |
| openstack | bugzilla.redhat.com bug 1410894 in rhosp-director-ui "Occasionally, UI does not update with correct information" [Medium,New] - Assigned to jrist | 22:15 |
| dtrainor | thank you | 22:15 |
| dtrainor | i need to run shortly too | 22:15 |
| therve | Thanks! | 22:15 |
| dtrainor | i appreciate the time, looking forward to resuming on monday | 22:15 |
| *** sebasmagri has joined #openstack-zaqar | 22:57 | |
| *** dtrainor has quit IRC | 23:15 | |
| *** flwang1 has joined #openstack-zaqar | 23:17 | |
| *** flwang1 has quit IRC | 23:26 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!