openstackgerrit | Mohammed Naser proposed openstack-infra/zuul master: Hide queue headers for empty queues when filtering https://review.openstack.org/572588 | 00:02 |
---|---|---|
mnaser | mordred: ^ this one is for you | 00:03 |
mnaser | this is what the result looks like https://usercontent.irccloud-cdn.com/file/9vii67C7/Screen%20Shot%202018-06-05%20at%208.03.28%20PM.png | 00:04 |
mnaser | yay for uncluttered status pages? | 00:04 |
mnaser | (to test locally, `yarn install` and `npm start` in your terminal) | 00:05 |
openstackgerrit | Clark Boylan proposed openstack-infra/zuul master: Fix race in slow start test https://review.openstack.org/572584 | 00:16 |
pabelanger | mnaser: sweat! now do a button / toggle to switch from https:// to finger:// please for console logs | 00:20 |
clarkb | I don't see the difference | 00:21 |
clarkb | but I've been fixing races in the tests so my eyes are probably not working proeprly | 00:21 |
pabelanger | mnaser: oh, another one is getting estimate time back when you hover over status bar, not sure where we lost that | 00:24 |
pabelanger | but is a nice thing to have | 00:24 |
*** weshay_ has joined #zuul | 00:28 | |
*** mhu` has joined #zuul | 00:28 | |
*** pabelanger_ has joined #zuul | 00:28 | |
*** _ari__ has joined #zuul | 00:29 | |
*** mhu has quit IRC | 00:30 | |
*** myoung|off has quit IRC | 00:31 | |
*** pabelanger has quit IRC | 00:31 | |
*** _ari_ has quit IRC | 00:32 | |
*** weshay has quit IRC | 00:32 | |
*** myoung has joined #zuul | 00:33 | |
mnaser | clarkb: here is nova filtered on what is deployed live https://usercontent.irccloud-cdn.com/file/bZx5bxNJ/Screen%20Shot%202018-06-05%20at%208.34.41%20PM.png | 00:35 |
mnaser | note the lack of useless Queue: .. | 00:35 |
mnaser | pabelanger_: those are easy quick ones to do | 00:35 |
* mnaser will hack on this a bit later today | 00:35 | |
clarkb | oh gotcha | 00:37 |
*** acozine1 has joined #zuul | 00:41 | |
*** mhu has joined #zuul | 00:42 | |
*** pabelanger has joined #zuul | 00:42 | |
*** mhu` has quit IRC | 00:42 | |
*** _ari_ has joined #zuul | 00:43 | |
*** pabelanger_ has quit IRC | 00:43 | |
*** myoung_ has joined #zuul | 00:43 | |
*** weshay has joined #zuul | 00:43 | |
*** weshay_ has quit IRC | 00:44 | |
openstackgerrit | Clark Boylan proposed openstack-infra/zuul master: Fix race in slow start test https://review.openstack.org/572584 | 00:44 |
*** _ari__ has quit IRC | 00:44 | |
*** myoung has quit IRC | 00:44 | |
clarkb | I fixed one race but I Think that just exposed another ... I'll have to pick that up tomorrow probably | 00:46 |
*** harlowja has quit IRC | 01:10 | |
*** xinliang has quit IRC | 01:12 | |
*** threestrands_ has joined #zuul | 01:33 | |
*** threestrands has quit IRC | 01:36 | |
*** nguyenhai93 has quit IRC | 01:47 | |
*** nguyenhai93 has joined #zuul | 01:48 | |
*** yolanda_ has joined #zuul | 01:56 | |
*** yolanda_ has quit IRC | 01:57 | |
*** yolanda_ has joined #zuul | 01:58 | |
*** yolanda has quit IRC | 01:59 | |
openstackgerrit | Clark Boylan proposed openstack-infra/zuul master: Fix race in slow start test https://review.openstack.org/572584 | 03:00 |
clarkb | that deserves careful review but I believe I may have reached a non racy test_slow_start state | 03:01 |
clarkb | commit message tries to go into much detail about what I have done | 03:01 |
*** josefwells has joined #zuul | 03:39 | |
josefwells | hey, I was on earlier with some issues with github setup | 03:41 |
josefwells | I just pushed my docker setup for zookeeper+nodepool+zuul | 03:41 |
josefwells | https://github.com/josefwells/zuul-docker | 03:42 |
clarkb | josefwells: let me take a quick look | 03:42 |
josefwells | I'm pretty new at docker, but I think it is pretty good | 03:42 |
josefwells | hey clarkb, thanks! | 03:42 |
josefwells | I had been following the zuul from scratch and github instructions here: https://zuul-ci.org/docs/zuul/admin/github_setup.html | 03:43 |
josefwells | so some of the repository assumes some of that has been done | 03:43 |
josefwells | setting up webhook, getting zuul.pem | 03:44 |
clarkb | josefwells: have the error message you get in the logs? | 03:44 |
josefwells | funny enough it tends to not be reproducible, but it complains about the zuul-test-config yaml file, I see it when I start zuul-scheduler | 03:46 |
clarkb | josefwells: reading that config you need to have a zuul-git connection | 03:46 |
clarkb | the zuul from scratch document explains this (we should maybe make that more explciit in the github config doc since its used there) | 03:47 |
josefwells | extra keys not allowed @ data['trigger']['github'] | 03:47 |
josefwells | that is one | 03:47 |
josefwells | oh, you MUST have zuul-git? | 03:47 |
josefwells | my stuff is behind some painful firewalls | 03:47 |
clarkb | you don't have to have it but you configured it in main.yaml but there is no corresponding connection in zuul.conf | 03:48 |
josefwells | oh, I did eventually remove that | 03:48 |
clarkb | as for the github key I think the issue there is you have named the github connection "my_github" not "github" | 03:48 |
clarkb | so they don't match | 03:48 |
josefwells | sorry, I did some sanitzation to remove some internal info | 03:48 |
clarkb | that is fine, you just need to make sure that the connection name for github in zuul.conf is what you use in your pipeline trigger config | 03:49 |
clarkb | github isn't a generic there, it is referring to a specific named connection | 03:49 |
josefwells | oh, the trigger itself is referring to the specific connection? ok | 03:51 |
clarkb | yup | 03:51 |
clarkb | its saying trigger when connection foo has these events. And you need to match that to a connection confi | 03:51 |
*** bhavik1 has joined #zuul | 03:51 | |
clarkb | this allows you to talk to github.com and github enterprise install locally if necesary | 03:51 |
clarkb | or multiple github installs locally, etc | 03:53 |
josefwells | makes perfect sense, just didn't realize | 03:54 |
josefwells | looks like zuul-scheduler likes it now! | 03:54 |
josefwells | progress! | 03:54 |
josefwells | think any of the docker-goo would be helpful to you guys? | 03:55 |
clarkb | josefwells: I know mordred has plans to try and automate some of the image building on a simple base then bind mount in configs or similar iirc | 03:55 |
clarkb | but I'm not full up to date on all those details. pbrx is the name of the tool being built iirc | 03:56 |
josefwells | hmm, still no web interface visible | 03:57 |
josefwells | should just be able to load it over port 9000 right? | 03:58 |
clarkb | josefwells: if the zuul-web process is running yes I would expect so | 03:58 |
clarkb | I think it may listen on locahost only by default though | 03:58 |
josefwells | 2018-06-05 18:04:50,353 INFO zuul.WebServer: Zuul Web Server starting | 03:58 |
clarkb | possible that it isn't listening on an ip that you can hit? | 03:58 |
josefwells | it is listening on 0.0.0.0, should be all interfaces | 03:59 |
clarkb | do you get anything if you curl the root? | 03:59 |
josefwells | ERR_EMPTY_RESPONSE | 03:59 |
clarkb | what about /api/info | 04:00 |
josefwells | {"info": {"capabilities": {"job_history": false}, "stats": {"url": null, "prefix": null, "type": "graphite"}, "websocket_url": null}} | 04:01 |
clarkb | ok so it is there and working | 04:01 |
josefwells | yeah, I never set up statsd | 04:01 |
josefwells | not sure if it is required | 04:01 |
clarkb | no it shouldn't be | 04:01 |
josefwells | I haven't pushed anything to zuul-test, just zuul-test-config | 04:01 |
josefwells | also the zuul.conf in zuul-test-config only has pipeline, no jobs or other stuff, again, just following the simple setup stuff I found | 04:02 |
clarkb | looks like the status page lives at status.html not index.html or similar | 04:03 |
clarkb | try getting /status.html and see if that igves you a status page | 04:03 |
clarkb | (we rewrite / to status.html in our webserver) | 04:03 |
clarkb | corvus: ^ we might want to add a route for / to status.html by default in cherrypy | 04:03 |
josefwells | plain text: 404: Not Found | 04:04 |
clarkb | hrm /tenants.html is the other root path I would expect towork, is that one any better? | 04:05 |
clarkb | the pretending to not be multitenant while being multitenant has made some of this confusing | 04:06 |
josefwells | tried that, also tried puting my tenant name in the url localhost:9000/github-test/status.html | 04:07 |
clarkb | ok if that isn't working my best guess is that the static path isn't right or we aren't installing the build artifacts for the client side of zuul-web | 04:08 |
clarkb | anything in the zuul-web logs related to trying to fetch the html files? | 04:08 |
josefwells | no, just that zuul-web started | 04:08 |
josefwells | gonna kill/restart it | 04:08 |
josefwells | same deal, only stopping/started added to log | 04:09 |
josefwells | odd that the base url returns empty, while any specific page gives me a 404 page | 04:10 |
clarkb | you aren't setting a static_path under [web] in zuul.conf so it should look at $zuul_install_path/zuul/web/static/ for those files | 04:10 |
*** rlandy|rover|bbl is now known as rlandy|rover | 04:10 | |
clarkb | josefwells: I think that is due to how the routes are setup in zuul-web a specific page will end up matching the route for file lookups but the root doesn't have one | 04:10 |
josefwells | perhaps my pip -u installs are breaking the lookup | 04:11 |
josefwells | I don't see anything in ~/.local/lib/python3.5/site-packages/zuul/web just handler.py _init_.py and _pycache_ | 04:12 |
clarkb | https://zuul-ci.org/docs/zuul/developer/javascript.html that may be helpful in figuring out how to bootstrap those static files | 04:13 |
josefwells | odd that api/info works | 04:14 |
clarkb | josefwells: the api data is coming out of zuul itself and none of it relies on prerendering static assets | 04:15 |
josefwells | ah, ok | 04:15 |
josefwells | still, I was stuck on that github trigger business for a while today, thanks so much for your help | 04:15 |
josefwells | I'll dig into this web stuff tomorrow | 04:16 |
josefwells | I gotta roll, when I get it going I'll feedback some info for you | 04:17 |
*** harlowja has joined #zuul | 04:17 | |
clarkb | looks like we've got apache set up to direclty serve the static assets on our install so I'm less help | 04:19 |
tobiash | clarkb: looks like we're missing the yarn stuff from zfs docs | 04:20 |
clarkb | tobiash: I want to say that the wheels are supposed to have it installed so maybe a problem with that | 04:20 |
tobiash | clarkb: the wheels are missing the static stuff yet | 04:21 |
clarkb | ah ok, then ya we need to have that in the install doc likely | 04:21 |
tobiash | clarkb: we may want to fix the wheels and switch zfs to not install from source | 04:22 |
tobiash | And maybe a fatal installation error if yarn is missing during installation | 04:23 |
tobiash | mordred: thoughts? | 04:24 |
*** bhavik1 has quit IRC | 04:24 | |
tobiash | Not sure if that's possible without breaking tox | 04:24 |
tobiash | At least we may want to check for the static stuff during zuul-web startup | 04:25 |
clarkb | tobiash: well you may offload serving of those files like openstack does. It is a tricky problem | 04:26 |
clarkb | maybe a warning | 04:26 |
*** josefwells has quit IRC | 04:26 | |
tobiash | hrm, I don't like warnings if I don't care in one case and want an error in the other case | 04:29 |
tobiash | maybe a fatal check that can be disabled in config | 04:29 |
*** harlowja has quit IRC | 04:45 | |
clarkb | tobiash: I put up a change ti fix test_slow_start if you are wanting to review things and updated the log stteaming change | 04:49 |
clarkb | ee test for ^ | 04:49 |
tobiash | clarkb: I already saw it. I'll look at it later @work | 04:50 |
openstackgerrit | Merged openstack-infra/zuul-jobs master: Add a job variable to control javascript link copying https://review.openstack.org/572542 | 05:23 |
*** sshnaidm|brq has joined #zuul | 05:42 | |
*** sshnaidm|brq has quit IRC | 05:47 | |
*** pcaruana has joined #zuul | 06:44 | |
openstackgerrit | Merged openstack-infra/zuul-jobs master: Fix deprecation warning of ansible-2.5 https://review.openstack.org/571996 | 06:47 |
*** openstackgerrit has quit IRC | 07:19 | |
*** sshnaidm|brq has joined #zuul | 07:21 | |
*** hashar has joined #zuul | 07:41 | |
*** jpena|off is now known as jpena | 07:52 | |
*** threestrands_ has quit IRC | 08:03 | |
*** gtema has joined #zuul | 08:17 | |
*** electrofelix has joined #zuul | 08:49 | |
*** hashar is now known as hasharAway | 09:50 | |
*** openstackgerrit has joined #zuul | 09:56 | |
openstackgerrit | Fabien Boucher proposed openstack-infra/zuul master: Add a loading_errors info to {tenant}/info endpoint https://review.openstack.org/553873 | 09:56 |
openstackgerrit | Markos Chandras (hwoarang) proposed openstack-infra/nodepool master: Add opensuse-150 to nodepool dsvm https://review.openstack.org/572416 | 10:13 |
*** yolanda_ has quit IRC | 11:00 | |
*** yolanda_ has joined #zuul | 11:00 | |
andreaf | tobiash ping | 11:03 |
tobiash | andreaf: hi | 11:05 |
andreaf | I'm interested in setting up zuul (for demo purposes) on k8s - clarkb mentioned yesterday that you do that today | 11:06 |
*** jpena is now known as jpena|lunch | 11:06 | |
andreaf | so I was wondering if you have docker images and/or helm charts that you could share | 11:06 |
tobiash | I'm running zuul on openshift (without helm) | 11:08 |
gtema | that's interesting. Was going to start thinking about that next week. | 11:09 |
gtema | are you using buildah or s2i for building zuul images? | 11:10 |
tobiash | pabelanger also asked me about my dockerfiles a few days ago: http://eavesdrop.openstack.org/irclogs/%23zuul/%23zuul.2018-05-31.log.html#t2018-05-31T17:50:32 | 11:10 |
tobiash | gtema: I'm using traditional dockerfiles | 11:10 |
openstackgerrit | Fabien Boucher proposed openstack-infra/zuul master: Add a loading_errors info to {tenant}/info endpoint https://review.openstack.org/553873 | 11:11 |
tobiash | I currently have a zuul-base image and some layers on top which are special to my deployment (e.g. generation of main.yaml from a repo) | 11:11 |
gtema | tobiash: I see. I want however to try buildah or s2i - seems promising | 11:11 |
tobiash | gtema: there is also interest in providing official zuul docker images | 11:12 |
andreaf | gtema: this may be interesting as well https://github.com/GoogleContainerTools/kaniko | 11:12 |
tobiash | gtema: there is a zuul container spec | 11:12 |
gtema | tobiash: are you using templates to describe whole installation? | 11:13 |
andreaf | tobiash: yeah I would be interested in building a chart as well | 11:13 |
andreaf | tobiash: if we could get a zuul chart into the incubator repo it would be great | 11:13 |
tobiash | gtema: yes, we use templates (and a small script that assembles the template from a folder structure) | 11:13 |
tobiash | the container spec: https://review.openstack.org/560136 | 11:14 |
andreaf | tobiash thanks for sharing the zuul docker image - do you have something similar for nodepool? | 11:14 |
tobiash | yes | 11:14 |
tobiash | just a sec | 11:14 |
gtema | andreaf: agree. Chart, OpenShift template, and probably OpenShift ADB seems nice possibilities to get this done | 11:14 |
tobiash | andreaf: nodepool dockerfile: http://paste.openstack.org/show/722790/ | 11:16 |
gtema | would be probably nice to integrate it into windmill or create a separate project for that | 11:16 |
tobiash | andreaf: but beware that needs some updates and currently needs to be in place in contrast to the zuul dockerfile | 11:16 |
tobiash | my nodepool dockerfile got less love in the last few months ;) | 11:18 |
andreaf | tobiash thanks, I'll keep that in mind | 11:18 |
gtema | tobiash: is it your "production" installation, or simply sandbox? | 11:18 |
andreaf | tobiash: thanks for sharing again | 11:18 |
tobiash | production | 11:19 |
andreaf | tobiash: do you have manifests as well? something pruned of any prod value that you could share? | 11:19 |
gtema | great. Any problems/notes on running? | 11:19 |
tobiash | manifests? | 11:19 |
andreaf | k8s manifests? like deployments, services, ingresses, configmaps etc | 11:20 |
andreaf | I never used openshift, but at the end it's built around k8s right? | 11:21 |
tobiash | andreaf: openshift adds further stuff like build configs, image streams which we currently use to build the images directly in openshift | 11:21 |
gtema | I guess openshift template is a combination of manifests (kind of) | 11:21 |
tobiash | yes, together with a simple parameter substitution | 11:22 |
gtema | tobiash: would you, please, also share template? | 11:22 |
tobiash | have to check first what I need to strip from | 11:23 |
gtema | sure | 11:23 |
andreaf | tobiash ok thanks | 11:23 |
andreaf | tobiash do you have any problem if I end up putting something built on those in a public repo under apache 2.0 license? | 11:24 |
andreaf | s/problem/concern | 11:24 |
tobiash | andreaf: I'll strip out all special stuff from us and share it as a zuul review (which never will merge). Once it's there it's apache 2 as zuul... ;) | 11:40 |
andreaf | cool thanks! | 11:40 |
gtema | thanks a lot Tobias | 11:42 |
*** jluk has joined #zuul | 11:55 | |
jluk | hi, any idea why periodic pipeline triggered from zuul CLI reports "Executing jobs for change <Branch 0x7fd deletes refs/heads/master from 000000" while a genuine timer trigger produces "<Branch 0x7fd refs/heads/master updated None..None>"? | 12:06 |
*** GonZo2000 has joined #zuul | 12:07 | |
*** GonZo2000 has quit IRC | 12:07 | |
*** GonZo2000 has joined #zuul | 12:07 | |
openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: DNM: Add OpenShift templates https://review.openstack.org/572746 | 12:08 |
tobiash | andreaf, gtema: ^ | 12:08 |
tobiash | andreaf, gtema: that | 12:08 |
tobiash | that won't run out of the box but should give you a starting point | 12:09 |
*** gtema has quit IRC | 12:09 | |
andreaf | tobiash: that's really really useful, thank you! | 12:09 |
tobiash | had to strip out a lot of our special stuff (e.g. on the fly config generation, master key generation) | 12:09 |
andreaf | sure | 12:10 |
*** jpena|lunch is now known as jpena | 12:10 | |
*** GonZo2000 has quit IRC | 12:19 | |
tobiash | pabelanger: you might be interested in that too ^ | 12:25 |
*** jluk has quit IRC | 12:31 | |
*** hasharAway is now known as hashar | 12:32 | |
*** gtema has joined #zuul | 12:50 | |
fungi | tobiash: clarkb: catching up, if the wheels get fixed then a `pip install .` in the source tree should install those static assets because modern pip builds a wheel from the source and installs that behind the scenes | 13:14 |
*** myoung_ is now known as myoung | 13:21 | |
tobiash | Shrews: our nodepool now starts building qcow2 images and raw images while we only have raw image clouds defined | 14:06 |
tobiash | it may be caused by the driver refactoring, but have to analyse first | 14:06 |
gtema | tobiash: I think it may be from me: https://review.openstack.org/#/c/566437/ | 14:07 |
Shrews | tobiash: api refactor doesn't touch builder code. check I27f8f45ba62e5ee0cca0ea7cd27893092b726ebd | 14:07 |
tobiash | gtema, Shrews: thanks for the hint | 14:08 |
gtema | shouldn't be, as it was set only if nothing else was present. But who knows | 14:08 |
Shrews | tobiash: i never reviewed that, but if you could identify and fix, it would be appreciated | 14:08 |
tobiash | Shrews: I think I see the error | 14:08 |
tobiash | I'm currently relying on the auto-detection via clouds.yaml and don't define the formats in the nodepool config | 14:09 |
openstackgerrit | David Shrewsbury proposed openstack-infra/nodepool master: Create a base Driver class https://review.openstack.org/571585 | 14:09 |
*** hashar is now known as hasharAway | 14:09 | |
tobiash | so auto-detection detects raw and the default adds a further qcow2 | 14:09 |
tobiash | so I can workaround that in the config and afterwards try to fix that | 14:10 |
gtema | ok | 14:11 |
Shrews | corvus: https://review.openstack.org/571585 test failures are confusing me if you get a chance to look | 14:12 |
Shrews | corvus: TypeError: getProviderConfig() missing 1 required positional argument: 'provider' | 14:12 |
gtema | tobiash: I think in that case the solution might be to set a default in build.py:_buildImage as I wanted initially and not to set in in config reading | 14:14 |
tobiash | gtema: yes probably | 14:15 |
tobiash | ok, the config workaround works | 14:16 |
tobiash | just noticed that the builder doesn't seem to reload the config in contrast to the launcher | 14:17 |
gtema | was also suspecting this | 14:17 |
Shrews | tobiash: it's supposed to | 14:21 |
tobiash | hrm, updated the config and it still built both, after restarting the builder it built the correct formats | 14:22 |
tobiash | have to check that again when I have time | 14:22 |
tobiash | Shrews: it's supposed to reload or not reload the config? | 14:23 |
Shrews | that's a bug then, b/c it is getting reloaded | 14:23 |
Shrews | both BuildWorker and UploadWorker threads reload the config | 14:23 |
Shrews | maybe something needs reset during the reload | 14:24 |
tobiash | maybe | 14:24 |
mnaser | if anyone is around, i made a thing to make things pretty -- https://review.openstack.org/#/c/572588/ | 14:26 |
*** pcaruana has quit IRC | 15:23 | |
openstackgerrit | Monty Taylor proposed openstack-infra/zuul master: Upgrade from angularjs (v1) to angular (v6) https://review.openstack.org/551989 | 15:29 |
mordred | tristanC: ok. I think that ^^ is finaly ready | 15:30 |
mordred | mnaser: ^^ you too, since you grok web things | 15:30 |
mordred | mnaser: also - YAY on your patch | 15:30 |
openstackgerrit | Monty Taylor proposed openstack-infra/zuul master: Translate zuulStartStream into typescript https://review.openstack.org/558618 | 15:35 |
openstackgerrit | Monty Taylor proposed openstack-infra/zuul master: Shift log streaming code into StreamComponent https://review.openstack.org/558619 | 15:35 |
clarkb | if people are doing zuul stuff this morning, I've still got a handful of changes up from yesterday related to fixing some of the brokeness we found after restarting services on Monday | 15:36 |
clarkb | I think I even have a fix for test_slow_start which seems to fail 1/2 job runs | 15:37 |
*** sshnaidm|brq has quit IRC | 15:37 | |
corvus | clarkb: yeah, i'm about 30 minutes into reviewing that one now | 15:45 |
clarkb | corvus: cool let me know if you have questions, it started by fixing the easy race with build1.waiting then running into the other stuff needing to be updated because we were running in a bit more lockstep | 15:46 |
clarkb | I tried to capture what and why I did things in the commit message | 15:47 |
mnaser | thats a terrifying big patch but i'll try to dig through it haha | 15:57 |
clarkb | mnaser: no kidding | 15:58 |
mnaser | mordred: going through it, yay, actual routes | 16:00 |
openstackgerrit | Michael Johnson proposed openstack-infra/zuul-jobs master: Collect the coverage report for npm test jobs https://review.openstack.org/570260 | 16:00 |
mordred | mnaser: yah - sorry it's so big - but hopefully it sets up enough things for us to be able to make big improvements | 16:00 |
*** hasharAway is now known as hashar | 16:01 | |
mnaser | "<zuul-dashboard></zuul-dashboard>" | 16:01 |
mnaser | lovely | 16:01 |
mordred | mnaser: also - http://logs.openstack.org/89/551989/38/check/zuul-build-dashboard/53d282b/npm/html/status.html and http://logs.openstack.org/89/551989/38/check/zuul-build-dashboard-multi-tenant/769ae05/npm/html/tenants.html should both fully work | 16:03 |
mnaser | oh i was about to start it locally | 16:03 |
mnaser | but that works too | 16:03 |
mnaser | :P | 16:03 |
mnaser | thats a relaly cool job | 16:04 |
mnaser | helps merge changes easier for reviewers | 16:04 |
mordred | yah - I think otherwise sometimes it's too hard | 16:04 |
clarkb | may want fungi to sanity check that http://logs.openstack.org/89/551989/38/check/zuul-build-dashboard/53d282b/npm/html/status.html doesnt perform worse on his browser | 16:05 |
clarkb | seems to work fine in mine | 16:05 |
gtema | in Chrome on the Builds page the "refresh" button is bigger, that search input boxes. In FF - same size (of the button). In FF rounded corners, while in Chrome - not. Also in this place there is no padding/margin at all - would be nicer | 16:10 |
clarkb | gtema: this is with the link above? or already merged status page? | 16:12 |
fungi | my browser is firefox 60.0.1 with a handful of extensions (tridactyl, eff privacy badger, duckduckgo privacy essentials, multi-account containers) | 16:12 |
fungi | seems to render that draft fine | 16:12 |
gtema | clarkb: http://logs.openstack.org/89/551989/38/check/zuul-build-dashboard-multi-tenant/769ae05/npm/html/t/openstack.org/builds.html | 16:13 |
fungi | it's slow-ish for me, but my browser is somewhat overloaded at the moment anyway | 16:13 |
mnaser | fungi: i think that with time, we'll be able to improve things significantly | 16:13 |
fungi | still performs better than the "old" status page from, say, a week or two ago | 16:13 |
mnaser | the infrastructure mordred is putting down means that we don't have to redraw the world | 16:13 |
AJaeger | tobiash, clarkb, mnaser, I see a change by clarkb that has two +2s - but was not approved: https://review.openstack.org/#/c/562042/ . Should we merge this finally? Or is there a reason that nobody gave the +A? | 16:14 |
mnaser | at some point only the things that have to be updated will be updated | 16:14 |
mnaser | AJaeger: i left a +2 but not +A because i wanted other cores to be aware of that change because it affects $world | 16:14 |
mnaser | should have probably clarified | 16:14 |
mnaser | i assume that was tobiash intention too | 16:14 |
clarkb | AJaeger: that one became less important, hwoever I think the consistency will be nice for readers | 16:15 |
tobiash | AJaeger, mnaser: yupp ;) | 16:15 |
mordred | mnaser: yes- I look forward to being able to make that status page more directly reactive - I think it should decrease the browser cpu usage significantly | 16:16 |
tobiash | one caveat is that the job info is on one line then | 16:16 |
AJaeger | so, anybody wants to +A - or what shall we do with it? | 16:18 |
clarkb | AJaeger: I think we should consider what people find more readable. We fixed the logstash issue externally. If they find the new version better lets merge it, but I am happy to abandon if that is not the case | 16:19 |
gtema | and overall this filter block on Build page was "part of" navbar, while now it is an additional rowm which takes place | 16:19 |
AJaeger | clarkb: how do you want to reach that consensus? | 16:19 |
* AJaeger has to step out for a bit now... | 16:20 | |
clarkb | AJaeger: maybe we just ask the existing reviewers, tobiash mnaser on https://review.openstack.org/#/c/562042/2 do you think the new content is more readable or prefer the old one? | 16:20 |
tobiash | Actually looking again onto the result: | 16:20 |
tobiash | http://logs.openstack.org/42/562042/2/check/openstack-infra-base-integration-centos-7/c07855f/job-output.txt.gz#_2018-05-24_04_20_58_062439 | 16:20 |
tobiash | I find it quite hard to visually separate the three pieces of information on that single line | 16:21 |
tobiash | oh actually four pieces of information... | 16:22 |
tobiash | while this looks ugly but the information is better readable: | 16:23 |
tobiash | http://logs.openstack.org/42/572342/1/check/tox-cover/bcb3341/job-output.txt.gz#_2018-06-05_11_12_48_114940 | 16:23 |
tobiash | I guess making that a localhost shell task would be visually best but limit it to trusted jobs | 16:24 |
*** elyezer has quit IRC | 16:24 | |
tobiash | clarkb: we may also be able to fix that in the ansible callback? | 16:25 |
clarkb | tobiash: so that it adds timestamps to each line? possibly | 16:26 |
mordred | I really need to make some progress on the in-broswer rendering of the json ... | 16:26 |
tobiash | clarkb: yes, that's the idea | 16:26 |
tobiash | clarkb: I think that would ideal | 16:27 |
tobiash | that would solve the problem while staying nicely readable | 16:27 |
clarkb | ok why don't I abandon the existing change then | 16:27 |
tobiash | ++ | 16:28 |
mordred | mnaser: thanks for those reviews - they're super helpful - and yes, I do think we need the routerlinkactive on the li element - lemme try updating it | 16:30 |
*** elyezer has joined #zuul | 16:31 | |
mnaser | mordred: cool, yup, done too much bootstrap in my life time | 16:31 |
clarkb | tobiash: what code path does that go through? it isn't the regular callback as that is just for shell and command streaming from a remote right? | 16:33 |
mordred | mnaser: woot! I fixed the active thing. now for the container thing | 16:34 |
mnaser | should probably look into making the leap to bs4 at some point too | 16:35 |
mordred | mnaser: yes. I may leave that patch to you ;) | 16:35 |
tobiash | clarkb: I thought it's the same callback as shell and branches off to command streaming for shell and command | 16:36 |
mnaser | if i have spare time i'd probably focus that on making things more reactive cause thats probably far more important | 16:36 |
tobiash | but have to look again at that | 16:36 |
clarkb | oh _process_result_for_localhost is what it is I think | 16:36 |
openstackgerrit | Artem Goncharov proposed openstack-infra/nodepool master: retire shade in favor of openstacksdk https://review.openstack.org/572829 | 16:37 |
mnaser | i am pretty excited im sure people will tell the difference with their status pages being cleaner :) | 16:37 |
mnaser | mordred: i wonder if i can borrow your job for my change to make it easier to get reviews.. unless you aim to merge your stuff soon | 16:37 |
tobiash | clarkb: check out v2_runner_on_ok | 16:40 |
tobiash | line 375 | 16:41 |
tobiash | that's handling debug results | 16:41 |
gtema | mordred: please look at the "builds" page. The row coloring is missing: <tr class="undefined". Another is that the filter section does not look nice. At least adding padding-right:5px for label and input there will be better | 16:42 |
clarkb | tobiash: ya, but what calls that? is that part of the ansible api? I think for some reason I assumed it was zuul v2 | 16:42 |
clarkb | grep in ansible tree says it is ansible api | 16:42 |
tobiash | clarkb: that's a callback called by ansible itself | 16:42 |
tobiash | clarkb: so actually _log would need to split 'msg' | 16:43 |
clarkb | tobiash: ya _log_message already does the splitting so I think we just need to stop treating debug: msg as special there | 16:46 |
clarkb | I'm reading _log_message to make sure I understand it | 16:46 |
corvus | clarkb: did you actually encounter any races after you added the build1.waiting/sleep loop? | 16:46 |
mnaser | is there a known bug right now in console logs | 16:46 |
clarkb | corvus: the test consistently failed after that because we were accepting jobs at that point. | 16:47 |
tobiash | clarkb: _log_message adds the hostname and calls _log | 16:47 |
clarkb | corvus: the race was older code could be not accepting (which is what we were asserting) I think. Then waiting there prevented that race from happening and I had to go step by step | 16:47 |
clarkb | tobiash: it also sets the result dict output | 16:48 |
clarkb | corvus: I think what the waiting check did was expose that we were racing on the subsequent assertFalse(accepting) and that could never be the case without tweaking the min and max values | 16:49 |
clarkb | * could never be the case now that we were sequencing things a bit more rigidly | 16:49 |
clarkb | tobiash: I'll do the easy update of iterating over msg if it is multiple lines | 16:50 |
*** myoung is now known as myoung|lunch | 16:50 | |
tobiash | ++ | 16:50 |
corvus | clarkb: i'm confused. if i add only the build1.waiting/sleep loop in, i can't get the test to fail, nor do i see how it could. let me push up a change. | 16:51 |
clarkb | corvus: we set max accepting to 3 but then only start 2 jobs so we should be accepting from that point forward unless we change the number down to 2 | 16:52 |
clarkb | at least that was my reading of it | 16:52 |
openstackgerrit | Monty Taylor proposed openstack-infra/zuul master: Upgrade from angularjs (v1) to angular (v6) https://review.openstack.org/551989 | 16:52 |
mordred | mnaser: ^^ that fixes oth things - thank you, I'd forgotten that both of those regressions were bothering me :) | 16:53 |
openstackgerrit | Tobias Henkel proposed openstack-infra/nodepool master: Fix adding qcow2 format without need https://review.openstack.org/572836 | 16:53 |
tobiash | Shrews, gtema: I think that should fix the image format issue ^ | 16:54 |
tobiash | but I should add a test case ;) | 16:54 |
gtema | tobiash: yes. Looks nice | 16:54 |
mordred | mnaser: I would like to merge this soon - it's been lurking out there WAY too long - maybe just rebase yours on top of mine and you'll get the jobs? | 16:54 |
mordred | gtema: oh - hrm. wheredid the row coloring go ... | 16:55 |
clarkb | corvus: similarly because min accepting was set to 1 we would transition to accepting on the next manageLoad() which made it harder to assert that state as I had to accomodate each manageLoad() I simplified by setting min to 0 so it wouldn't flip unless jobs finished or max was increased | 16:55 |
openstackgerrit | James E. Blair proposed openstack-infra/zuul master: Fix race in slow start test https://review.openstack.org/572842 | 17:12 |
openstackgerrit | James E. Blair proposed openstack-infra/zuul master: Make test_slow_start more robust https://review.openstack.org/572843 | 17:12 |
corvus | clarkb: ^ okay, i wrote up why i think the test should work more-or-less as-written (but with your initial fix). | 17:13 |
corvus | clarkb: setting the min to 0 is kind of a red flag for me -- that's not possible in production use | 17:14 |
*** gtema has quit IRC | 17:14 | |
clarkb | corvus: https://review.openstack.org/#/c/572843/1/zuul/executor/server.py is another way to approach that | 17:15 |
clarkb | (I'm fine with that but still need to finish reviewing it) | 17:15 |
corvus | clarkb: i think a key nuance is that the test actually relies on manageLoad *not* being called too often. if it is called too often, it will notice that jobs are no longer "starting". | 17:15 |
clarkb | ya | 17:15 |
corvus | (by the time a job hits the "hold_jobs_in_build" wait, it's no longer starting, it's running) | 17:16 |
clarkb | corvus: I think it would be nice to keep the raise in waitForExecutorBuild but I can add that back in in a followup | 17:16 |
clarkb | it just makes that case of error easier to understand when reading the test fail log | 17:16 |
corvus | we could add a new "hold jobs in start" feature, and i got halfway through doing that, but it's a lot of code for basically just this test | 17:16 |
clarkb | corvus: that was why I went with min accepting 0 :) | 17:16 |
corvus | clarkb: agreed re: exception | 17:16 |
corvus | clarkb: i'm just really surprised you hit a race with just 572842; i've run it a lot locally with no failures -- the races that 572843 are really pathological cases (which, to be fair, we get sometimes around here, but usually not reliably) | 17:19 |
corvus | clarkb: actually, i think waitForBuildCompletion can just be replaced with another iterate_timeout call | 17:19 |
corvus | (it's only a single check) | 17:20 |
clarkb | to check the starting attribute | 17:20 |
corvus | sorry, waitForWorkerCompletion | 17:20 |
corvus | it just checks whether build.uuid is in job_workers | 17:21 |
clarkb | corvus: https://review.openstack.org/#/c/572842/1/tests/unit/test_executor.py line 550 of the new side of the diff is what gave me much trouble. That won't be reliable until the child change update to ordering of manageLoad() and starting work. Do we want to combine the two changes for that reason? | 17:21 |
clarkb | corvus: ah | 17:21 |
openstackgerrit | Monty Taylor proposed openstack-infra/zuul master: Upgrade from angularjs (v1) to angular (v6) https://review.openstack.org/551989 | 17:22 |
mordred | mnaser: ^^ that should fix the thing gtema pointed out (missing color bars on builds page) | 17:22 |
mordred | mnaser: I'm going to need some help to figure out how to get the filter boxes up into the nav bar | 17:23 |
mordred | mnaser: since the navbar is a component now - we can probably do that as a followup though | 17:23 |
mordred | mnaser: oh - interesting. the header works differently in produciton for status and for builds/jobs | 17:24 |
corvus | clarkb: python has to essentially only run the job thread all the way until it stops to hold in build and then switch back to the executor thread to call manageLoad, and it has to do that *twice* (one for each job) in order for that race to fail the test. i'm happy to squash the change, but also, you should buy a lottery ticket. :) | 17:24 |
mordred | corvus: if you have a sec, could you look at the navbar header on https://zuul.openstack.org/builds.html vs https://zuul.openstack.org/status.html | 17:24 |
openstackgerrit | James E. Blair proposed openstack-infra/zuul master: Fix race in slow start test https://review.openstack.org/572842 | 17:26 |
mordred | corvus: one spans the entire page, one doesn't - in the angular patch it becomes the same component so will be consistent, but we should pick which one we want to be consistent to | 17:27 |
corvus | mordred: i agree they look different. builds has a margin | 17:27 |
mordred | yah | 17:27 |
mordred | I *think* I prefer full page width - but I don't have a strong opinion | 17:27 |
corvus | mordred: i think i like status (w/o margin). but it's almost a toss up. | 17:27 |
corvus | mordred: i think we said the same thing :) | 17:27 |
mordred | corvus: :) | 17:28 |
mordred | that's also what mnaser suggested in an earlier review, so it's what the patch does now - so yay! | 17:28 |
clarkb | corvus: +2 I will do a followup to add the raise. Also I think the child cahnge can be abandoned? | 17:30 |
corvus | done | 17:31 |
corvus | clarkb: i was suggesting instead of adding the raise, replace that entire method with an iterate_timeout | 17:31 |
*** bhavik1 has joined #zuul | 17:32 | |
openstackgerrit | Clark Boylan proposed openstack-infra/zuul master: Add timestamps to multiline debug message logs https://review.openstack.org/572845 | 17:33 |
clarkb | corvus: oh gotch | 17:33 |
clarkb | tobiash: ^ that is the fix for logging with test update too | 17:33 |
mnaser | mordred: I can provide brain cycles on how to best wire this up | 17:34 |
mnaser | But I can’t commit to time to figuring that part out because I’m a little overcommitted right now | 17:34 |
openstackgerrit | Clark Boylan proposed openstack-infra/zuul master: Treat ::1 as localhost in addition to 127.0.0.1 https://review.openstack.org/572847 | 17:34 |
clarkb | that was somethign I noticed when fixing the log message thing above. Not sure if strictly necessary | 17:35 |
*** gtema has joined #zuul | 17:35 | |
corvus | Shrews: i'm looking at 571585 | 17:36 |
Shrews | corvus: relooking too now that i'm back from lunch. i suspect it has something to do with the class loading aspect | 17:37 |
Shrews | maybe the driver needs instantiation? | 17:37 |
openstackgerrit | Artem Goncharov proposed openstack-infra/nodepool master: retire shade in favor of openstacksdk https://review.openstack.org/572829 | 17:38 |
Shrews | get_provider_config() used to do the instantiation. doesn't do that now | 17:40 |
*** jpena is now known as jpena|off | 17:40 | |
openstackgerrit | Tobias Henkel proposed openstack-infra/nodepool master: Fix adding qcow2 format without need https://review.openstack.org/572836 | 17:40 |
tobiash | Shrews, gtema: with test now ^ | 17:40 |
tobiash | the test was helpful because PS1 was actually broken :) | 17:41 |
gtema | tobiash: great | 17:41 |
Shrews | yay tests | 17:41 |
tobiash | that adds test for gtema's use case as well as a test for my regression | 17:42 |
corvus | Shrews: yeah, i think you're right. left comments on the change. | 17:44 |
openstackgerrit | Clark Boylan proposed openstack-infra/zuul master: Remove unnecessary method in test_slow_start https://review.openstack.org/572850 | 17:46 |
clarkb | corvus: something like that I Guess | 17:46 |
*** bhavik1 has quit IRC | 17:47 | |
*** gtema has quit IRC | 17:48 | |
corvus | clarkb: ya | 17:52 |
openstackgerrit | Tobias Henkel proposed openstack-infra/nodepool master: Fix adding qcow2 format without need https://review.openstack.org/572836 | 17:52 |
mordred | mnaser: no worries - I don't think it's a huge issue we need to solve today - and I found some web blog things with suggestions | 17:54 |
mordred | mnaser: they involve words I don't understand yet of course :) | 17:54 |
mnaser | mordred: i can help with translating things :P | 18:05 |
mnaser | but yeah moving the filter into the navbar is nice but from an angular pov i don't think it's right because we're taking a component all together and splicing it out of it | 18:05 |
mnaser | imagine someone taking the dashboard downstream and wanting to use particular components | 18:06 |
mnaser | like say i load the libraries and just want to add a `<zuul-status server="zuul.openstack.org" />` using the libraries | 18:06 |
mnaser | we're pretty far away from that but the idea of creating these components is letting (hopefully) downstream users easily integrate things like this | 18:06 |
mnaser | unless we find a way to wire it up to the search like <zuul><zuul-filter /><zuul-status .../></zuul> that way its all in there together | 18:07 |
mnaser | it's a fun discussion we can have | 18:07 |
mnaser | </irc-blog-post> | 18:07 |
mordred | mnaser: ++ | 18:09 |
*** electrofelix has quit IRC | 18:10 | |
openstackgerrit | Merged openstack-infra/zuul master: Fix race in slow start test https://review.openstack.org/572842 | 18:11 |
openstackgerrit | David Shrewsbury proposed openstack-infra/nodepool master: Create a base Driver class https://review.openstack.org/571585 | 18:11 |
Shrews | corvus: ^^ passes locally now | 18:12 |
*** myoung|lunch is now known as myoung | 18:18 | |
openstackgerrit | Merged openstack-infra/zuul master: Remove unnecessary method in test_slow_start https://review.openstack.org/572850 | 18:19 |
openstackgerrit | Tobias Henkel proposed openstack-infra/nodepool master: Move QuotaInformation to driver utils https://review.openstack.org/572860 | 18:23 |
*** robled has quit IRC | 18:32 | |
*** robled has joined #zuul | 18:36 | |
*** robled has quit IRC | 18:36 | |
*** robled has joined #zuul | 18:36 | |
openstackgerrit | Monty Taylor proposed openstack-infra/zuul master: Translate zuulStartStream into typescript https://review.openstack.org/558618 | 18:58 |
openstackgerrit | Monty Taylor proposed openstack-infra/zuul master: Shift log streaming code into StreamComponent https://review.openstack.org/558619 | 18:58 |
openstackgerrit | David Shrewsbury proposed openstack-infra/nodepool master: Have Drivers create Providers https://review.openstack.org/571586 | 18:58 |
mordred | corvus, tristanC, clarkb: https://review.openstack.org/#/c/551989 https://review.openstack.org/#/c/558618 and https://review.openstack.org/#/c/558619/ should all be good to go now | 19:01 |
mordred | Shrews: I like how that patch has a net-zero line impact | 19:03 |
Shrews | mordred: a well balanced nodepool | 19:04 |
kmalloc | whoa, i didn't know nodepool used the old "class X:" form. threw me for a loop having not seen that in a while /embarassing :P | 19:13 |
kmalloc | (embarassing for me not recognizing it as valid python) | 19:13 |
Shrews | kmalloc: yeah I noticed a couple of those today. probably should clean those up | 19:14 |
kmalloc | Shrews: ++ thankfully that is super easy | 19:15 |
tobiash | mordred: comment on https://review.openstack.org/558619 | 19:20 |
tobiash | mordred: oh actually that's already broken in 551989 | 19:23 |
mordred | tobiash: ah - cool. thanks | 19:41 |
tobiash | Shrews: is the driver class ready to go? | 19:44 |
Shrews | tobiash: seems to be | 19:45 |
Shrews | tobiash: maybe we should add a release note for your qcow2 fix? | 19:47 |
clarkb | qcow2 fix? | 19:47 |
tobiash | clarkb: https://review.openstack.org/572836 | 19:48 |
tobiash | Shrews: was the bug introduced in a previous release? | 19:48 |
clarkb | I want to say that the autodetection from clouds.yaml is relatively new | 19:49 |
clarkb | bug has probably been in there since we added that functionality | 19:49 |
Shrews | tobiash: oh, maybe not | 19:49 |
Shrews | clarkb: no, i think this was a recent-ish change | 19:49 |
tobiash | clarkb: I use the autodetection from clouds.yaml since the beginning | 19:49 |
tobiash | Shrews: I don't see a tag after the bug was introduced | 19:50 |
clarkb | ah | 19:50 |
Shrews | tobiash: was there a release note on the breaking change? | 19:50 |
tobiash | nope | 19:50 |
Shrews | k | 19:50 |
Shrews | probably should have been | 19:50 |
openstackgerrit | Merged openstack-infra/zuul-jobs master: Include hostnames in /etc/hosts https://review.openstack.org/572313 | 19:51 |
Shrews | we need to be more diligent with that | 19:51 |
tobiash | Shrews: do you want a release not for fallback qcow2 feature? | 19:51 |
tobiash | s/not/note | 19:51 |
Shrews | tobiash: seems worth mentioning, yeah? | 19:52 |
tobiash | I'll add one | 19:52 |
Shrews | i'm going to have to add one for all the driver api changes, i think | 19:54 |
Shrews | though we always said that wasn't set in stone yet | 19:54 |
openstackgerrit | Merged openstack-infra/nodepool master: Create a base Driver class https://review.openstack.org/571585 | 19:54 |
corvus | Shrews: yeah, i don't think we've made any user-visible changes yet. except that when it's all done, we should be able to add a note about how 'nodepool list' will show static ready nodes. | 19:55 |
openstackgerrit | Merged openstack-infra/nodepool master: Have Drivers create Providers https://review.openstack.org/571586 | 19:55 |
Shrews | corvus: so we don't want to use reno for driver api changes? | 19:56 |
Shrews | just user visible ones? | 19:57 |
corvus | Shrews: i think so. at least until we're ready to consider it a public api. | 19:57 |
Shrews | i guess most nodepool users don't care about the apit | 19:57 |
corvus | (but it's still an internal api right now) | 19:58 |
Shrews | s/apit/api | 19:58 |
Shrews | k | 19:58 |
corvus | developers can read the git log :) | 19:58 |
Shrews | lol | 19:58 |
openstackgerrit | Tobias Henkel proposed openstack-infra/nodepool master: Fix adding qcow2 format without need https://review.openstack.org/572836 | 19:58 |
Shrews | "code is self documenting!" | 19:58 |
corvus | i just wrote a bunch of documentation for it, so that seems to be working ;) | 19:59 |
Shrews | indeed | 19:59 |
tobiash | Shrews: now with release note ^ | 19:59 |
tobiash | corvus: are you running 572210 on your instance currentlz? | 20:05 |
tobiash | I thought about upgrading our instance and I'm not sure if it will break without 572210 | 20:06 |
corvus | tobiash: yes i think it's needed | 20:06 |
tobiash | k, thx | 20:10 |
*** acozine1 has quit IRC | 20:38 | |
corvus | also, i think there's an issue with websocket streaming still, working on that now | 21:00 |
*** myoung is now known as myoung|off | 21:06 | |
openstackgerrit | Merged openstack-infra/nodepool master: Fix adding qcow2 format without need https://review.openstack.org/572836 | 21:18 |
*** dims has left #zuul | 21:31 | |
openstackgerrit | James E. Blair proposed openstack-infra/zuul master: Handle more than one websocket streamer at a time https://review.openstack.org/573023 | 22:25 |
openstackgerrit | James E. Blair proposed openstack-infra/zuul master: Handle more than one websocket streamer at a time https://review.openstack.org/573023 | 22:28 |
corvus | tobiash: ^ that's also necessary | 22:30 |
corvus | Shrews, mordred: ^ may be of interest | 22:31 |
*** rlandy|rover is now known as rlandy|rover|bbl | 22:37 | |
*** threestrands has joined #zuul | 22:51 | |
*** eandersson_ is now known as eandersson | 22:59 | |
*** hashar has quit IRC | 23:00 | |
clarkb | fungi: I reworkflowed it as I'm satisfied with corvus' response | 23:01 |
clarkb | I'll need to rebase my change to that test once it goes in | 23:01 |
fungi | wfm | 23:02 |
openstackgerrit | James E. Blair proposed openstack-infra/zuul master: Handle websocket client hangups better https://review.openstack.org/573034 | 23:06 |
corvus | clarkb, fungi: that's a non-urgent improvement, but you might want to go ahead and review it while it's fresh | 23:07 |
corvus | (the current behavior is okay, but results in things hanging around too long and spurious exceptions in the log) | 23:07 |
openstackgerrit | Merged openstack-infra/zuul master: Handle more than one websocket streamer at a time https://review.openstack.org/573023 | 23:12 |
openstackgerrit | James E. Blair proposed openstack-infra/zuul master: Fix zuul-web sql connections https://review.openstack.org/572210 | 23:13 |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!