| * diablo_rojo opens reviews | 00:04 | |
| diablo_rojo | fungi, looks good to me and sounds like that's what SotK was asking for so I think I will just push it through. | 00:05 |
|---|---|---|
| diablo_rojo | =2+Wed | 00:08 |
| diablo_rojo | +2+Wed | 00:08 |
| fungi | thanks! | 00:11 |
| fungi | cloudnull: once it merges, it'll be autodeployed in ~30-60 minutes and you should be able to retry creating boards and worklists | 00:12 |
| fungi | assuming you're still around that late | 00:12 |
| cloudnull | will give it a go later tonight and report back for sure. | 00:12 |
| fungi | i should be up and around for a while too | 00:12 |
| diablo_rojo | cloudnull, thanks! | 00:43 |
| diablo_rojo | I'll be around either here or on my other nick. | 00:44 |
| openstackgerrit | Merged opendev/storyboard master: Correct team iterator lists in worklist creation https://review.opendev.org/667248 | 00:45 |
| cloudnull | ^ Woot! | 00:56 |
| diablo_rojo | Woot indeed :) | 00:59 |
| fungi | cloudnull: a quick check tells me it's already gotten installed on storyboard.o.o | 01:21 |
| cloudnull | so I'm still not able to create a new board - "500: POST /api/v1/worklists: _CallbackResult was already set" <- thats the error I'm seeing now | 02:04 |
| *** whoami-rajat has joined #storyboard | 02:07 | |
| cloudnull | I can however create a worklist | 02:08 |
| cloudnull | diablo_rojo fungi ^ | 02:08 |
| cloudnull | and if I try and create a new board, with no columns, I get - 500: POST /api/v1/boards: 'NoneType' object has no attribute 'name' | 02:11 |
| cloudnull | same error if I create a board with 1 column | 02:12 |
| *** diablo_rojo has quit IRC | 02:17 | |
| *** armstrong has quit IRC | 02:45 | |
| *** udesale has joined #storyboard | 03:53 | |
| *** jtomasek has joined #storyboard | 05:51 | |
| *** altlogbot_0 has quit IRC | 06:46 | |
| *** altlogbot_1 has joined #storyboard | 06:56 | |
| SotK | looks like boards had the same bug, I'll send a patch | 06:56 |
| SotK | sorry for the painful experience cloudnull | 06:57 |
| openstackgerrit | Adam Coldrick proposed opendev/storyboard master: Correct team iterator lists in board creation https://review.opendev.org/667275 | 06:58 |
| *** jpich has joined #storyboard | 07:07 | |
| *** tosky has joined #storyboard | 07:28 | |
| *** jtomasek has quit IRC | 07:31 | |
| *** jtomasek has joined #storyboard | 07:32 | |
| *** udesale has quit IRC | 08:01 | |
| *** udesale has joined #storyboard | 08:03 | |
| *** udesale has quit IRC | 08:03 | |
| *** udesale has joined #storyboard | 08:03 | |
| *** udesale has quit IRC | 11:06 | |
| *** udesale has joined #storyboard | 11:08 | |
| openstackgerrit | Merged opendev/storyboard master: Correct team iterator lists in board creation https://review.opendev.org/667275 | 11:36 |
| fungi | and it's also merged, should take effect here shortly | 11:42 |
| SotK | thanks :) | 12:01 |
| fungi | thank you for finding and fixing! | 12:08 |
| *** whoami-rajat has quit IRC | 13:16 | |
| *** openstackgerrit has quit IRC | 13:48 | |
| cloudnull | fungi SotK will give it a spin again in just a few | 13:50 |
| cloudnull | thanks! | 13:50 |
| fungi | and thanks for bearing with us on these regressions | 13:50 |
| *** udesale has quit IRC | 14:55 | |
| *** udesale has joined #storyboard | 16:03 | |
| *** diablo_rojo has joined #storyboard | 16:07 | |
| cloudnull | hey all, just tried again to create a new board - https://i.snag.gy/FiXSmC.jpg - same error as before :'( | 16:47 |
| cloudnull | also same error when adding a story with multiple tasks - https://i.snag.gy/5p1TCt.jpg | 16:49 |
| fungi | the multiple tasks problem isn't expected to be fixed yet, but i've reset the task in https://storyboard.openstack.org/#!/story/2000983 to a todo state to indicate that we don't think it's fixed | 16:52 |
| fungi | since we discovered the new way that's breaking after the prior attempt at solving it | 16:52 |
| fungi | pip says the version with https://review.opendev.org/667275 is installed, so presumably we're still missing a spot. i'll check the apache logs again | 16:53 |
| *** jpich has quit IRC | 16:56 | |
| fungi | that looks like a new error, fwiw | 16:56 |
| fungi | oh fun, that's an assertionerror exception too | 16:56 |
| SotK | fungi: is there any chance you can try to get the full traceback of that error please? | 16:58 |
| fungi | i'm nearly done getting it onto paste.o.o | 16:58 |
| SotK | heh, thanks | 16:58 |
| fungi | for some reason my pastebinit install has been broken for a while, so copying and pasting around | 16:59 |
| fungi | voila: http://paste.openstack.org/show/753376/ | 16:59 |
| fungi | duplicate callback for the event stream? | 17:00 |
| SotK | oh huh, I thought we'd blocked publishing events to rabbitmq for worklists and boards when we had a similar issue to this ages and ages ago with the intention of making it work "one day" | 17:03 |
| SotK | ah I guess its because its a timeline event thing that triggered it not a worklist | 17:03 |
| * SotK finishes what he's doing so he can pay better attention | 17:04 | |
| fungi | right, just happened to get called through the worklist creation codepath | 17:04 |
| *** udesale has quit IRC | 17:07 | |
| *** jtomasek has quit IRC | 17:21 | |
| SotK | cloudnull, fungi: can either of you reproduce that board error? | 17:58 |
| fungi | i'm not certain what steps cloudnull took to get it | 17:58 |
| SotK | we started publishing timeline events for worklists a long time ago and I've never seen this error before, and I can't reproduce it locally or on storyboard-dev (assuming that the reproduction steps are the try to make a board like the one in the screenshot) | 18:00 |
| SotK | fungi: what version of pika is installed on storyboard.o.o out of interest? | 18:07 |
| fungi | pika==0.13.0 | 18:09 |
| fungi | would an api method to get detail on what dependency versions are present be worthwhile? along with one to find out what version of the api server is running? | 18:09 |
| fungi | i've written something similar for a personal application i maintain, which i'd be happy to hack in an implementation of | 18:10 |
| SotK | I think that would be very useful, yeah | 18:11 |
| fungi | i'll add that to my list | 18:11 |
| fungi | useful to query the environment when the service is started, so that you know what versions were present to be imported at start time rather than possibly upgraded while it was running and so may not be the actual version in use | 18:13 |
| fungi | granted, it's fairly easy to return both | 18:13 |
| SotK | the traceback looks like the problem is happening when pika itself is handling a `socket.error` | 18:17 |
| SotK | no idea what is causing it | 18:17 |
| fungi | there are a bunch of socket disconnects in the wsgi error log | 18:26 |
| fungi | i had assumed they were unrelated, from clients aborting calls | 18:26 |
| fungi | [Tue Jun 25 16:57:42.304761 2019] [wsgi:error] [pid 4996:tid 140067423065856] 2019-06-25 16:57:42.304 4996 WARNING storyboard.notifications.publisher [-] Attempted to send message on closed connection.: ConnectionClosed: (-1, "error(32, 'Broken pipe')") | 18:27 |
| fungi | maybe a sign rabbit is acting out? | 18:28 |
| fungi | rabbitmq was last started 2019-02-08 so it's been running a while | 18:28 |
| fungi | =INFO REPORT==== 25-Jun-2019::16:57:47 === accepting AMQP connection <0.12169.47> (127.0.0.1:45838 -> 127.0.0.1:5672) | 18:29 |
| fungi | =ERROR REPORT==== 25-Jun-2019::17:00:47 === closing AMQP connection <0.12169.47> (127.0.0.1:45838 -> 127.0.0.1:5672): Missed heartbeats from client, timeout: 60s | 18:30 |
| fungi | that's happening every few minutes | 18:30 |
| fungi | oh, or maybe the api server is simply too busy to send heartbeats | 18:30 |
| cloudnull | SotK I can give it another spin ? | 18:44 |
| fungi | yeah, see if you can get it to happen again | 18:44 |
| fungi | i have a feeling we're seeing something involving rabbitmq disconnects | 18:44 |
| cloudnull | basically all I did was login, click boards, select create a board from the drop down, add a title, description, and a couple columns, then click save | 18:45 |
| * cloudnull doing it now | 18:45 | |
| fungi | thanks! | 18:45 |
| cloudnull | it worked :) | 18:46 |
| cloudnull | https://storyboard.openstack.org/#!/board/155 | 18:46 |
| fungi | for the record, we've only seen that assertion error once today (so far) suggesting it's not happening with great frequency | 18:47 |
| cloudnull | I can go create a mess of boards to see if it happens again ? | 18:48 |
| fungi | well, once in today's log file anyway | 18:48 |
| fungi | prior to log rotation there was an occurrence at 02:04:04 utc | 18:48 |
| cloudnull | does storyboard have a REST API I can use to create stories/tasks ? | 18:49 |
| cloudnull | and assign them to a board ? | 18:49 |
| fungi | prior to that, the most recent occurrence was 2019-06-13 19:41:01 utc, so ~11 days ago? | 18:49 |
| *** ianychoi_ has quit IRC | 18:49 | |
| fungi | and was also just the one, so this seems to be an infrequent race/whatever which has been around for a while | 18:50 |
| fungi | cloudnull: yes, it even has documentation... https://docs.openstack.org/infra/storyboard/webapi/v1.html | 18:50 |
| *** ianychoi_ has joined #storyboard | 18:50 | |
| cloudnull | nice! | 18:50 |
| * cloudnull reading | 18:50 | |
| fungi | the design is api-first, so anything the webclient can do has to be supported by the api | 18:51 |
| fungi | cloudnull: there's also a console client... https://pypi.org/project/boartty/ | 18:52 |
| fungi | cloudnull: and a python sdk... https://docs.openstack.org/infra/python-storyboardclient/ | 18:53 |
| cloudnull | very cool ! this will help my trello to storyboard migration :) | 18:54 |
| fungi | source for all of those things reside in opendev's gerrit, in case you find bugs you want to fix | 18:58 |
| fungi | (or improvements you want to make) | 18:58 |
| fungi | same goes for the webclient too (opendev/storyboard-webclient repo) | 18:59 |
| fungi | SotK: anyway, in total three occurrences of "AssertionError: _CallbackResult was already set" in the past 16 days | 19:01 |
| fungi | so doesn't seem to be happening with any extreme frequency | 19:01 |
| cloudnull | I'm running into the following when trying to use the python-storyboardclient - https://pasted.tech/pastes/d5c987f1fdca2acb41d86aa933132d8ab6f9540c.raw - seems to be happening w/ both py27 and py37 from within a venv | 19:14 |
| cloudnull | happens from `pip install git+https://opendev.org/openstack-infra/python-storyboardclient` using a local clone, or using the pypi package. | 19:16 |
| fungi | huh, i'll try the same | 19:17 |
| cloudnull | oh, i guess I need to RTFM harder. if `cd` into the local clone and install it works | 19:19 |
| *** jtomasek has joined #storyboard | 19:19 | |
| cloudnull | git+ and the pypi package are not working :'( | 19:19 |
| fungi | it worked for me a moment ago, but yes if my cwd is not a git repo then it breaks (whether installed from pypi or not) | 19:24 |
| fungi | i wonder if this is a pbr problem | 19:24 |
| fungi | it's certainly strange | 19:26 |
| fungi | i've only ever seen this if package metadata is missing or pbr can't find where pip wrote it | 19:26 |
| cloudnull | I've only ever encountered this as a PBR bug | 19:34 |
| fungi | yes, discussing it in the infra meeting now in fact | 19:35 |
| fungi | there was a very old pbr bug, but this is happening with latest pbr | 19:35 |
| *** tosky has quit IRC | 19:40 | |
| fungi | cloudnull: okay, so clarkb spotted the problem there | 20:00 |
| fungi | likely a very-long-standing error in storyboardclient/__init__.py which has gone unnoticed (though i haven't done a git blame on it) | 20:01 |
| fungi | we need to explicitly pass in the package name as it's not the same as the module name | 20:01 |
| fungi | by default pbr assumes they match | 20:01 |
| cloudnull | ah ha! | 20:02 |
| cloudnull | at least it sounds like a simple thing to resolve | 20:03 |
| fungi | yeah, either he or i will push a patch in just a moment | 20:03 |
| fungi | (a very trivial one) | 20:03 |
| *** openstackgerrit has joined #storyboard | 20:09 | |
| openstackgerrit | Jeremy Stanley proposed opendev/python-storyboardclient master: Correct the distname in PBR version discovery https://review.opendev.org/667455 | 20:09 |
| fungi | cloudnull: ^ | 20:09 |
| cloudnull | nice! | 20:09 |
| fungi | git blame indicates that bug has existed ever since the beginning | 20:10 |
| fungi | all the way back to the initial cookiecutter commit https://review.openstack.org/138090 in 2014 | 20:11 |
| * fungi notes that neither the author nor the four people who reviewed that change (nor the ci account which verified it!) is around our community any longer | 20:13 | |
| diablo_rojo | Lol | 20:26 |
| diablo_rojo | thats a piece of history right there | 20:26 |
| Zara | heh, now I feel old, having met most of those involved at least once | 20:43 |
| *** ianychoi_ has quit IRC | 20:59 | |
| *** ianychoi_ has joined #storyboard | 21:00 | |
| *** ianychoi_ has quit IRC | 21:05 | |
| *** ianychoi_ has joined #storyboard | 21:09 | |
| *** openstackgerrit has quit IRC | 21:18 | |
| *** ianychoi_ is now known as ianychoi | 21:18 | |
| *** jtomasek has quit IRC | 21:24 | |
| *** diablo_rojo has quit IRC | 22:54 | |
| *** diablo_rojo has joined #storyboard | 23:31 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!
