| openstackgerrit | Corey Bryant proposed openstack-infra/zuul-jobs master: Add ensure-python role https://review.openstack.org/609761 | 01:45 |
|---|---|---|
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/nodepool master: Implement a Kubernetes driver https://review.openstack.org/535557 | 01:47 |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/nodepool master: Implement an OpenShift resource provider https://review.openstack.org/570667 | 01:49 |
| *** zigo has quit IRC | 02:29 | |
| *** zigo has joined #zuul | 02:31 | |
| *** rlandy|bbl is now known as rlandy | 03:00 | |
| *** rlandy has quit IRC | 03:02 | |
| *** mhu has quit IRC | 04:06 | |
| *** mhu has joined #zuul | 04:07 | |
| *** EmilienM is now known as EvilienM | 04:13 | |
| openstackgerrit | Merged openstack-infra/zuul-sphinx master: Add attr-overview directive https://review.openstack.org/604980 | 04:16 |
| *** bhavikdbavishi has joined #zuul | 04:17 | |
| openstackgerrit | Merged openstack-infra/zuul master: Add a quick-start test job https://review.openstack.org/609514 | 04:50 |
| *** toabctl has joined #zuul | 05:07 | |
| *** ssbarnea has quit IRC | 05:11 | |
| *** pcaruana has joined #zuul | 05:15 | |
| *** ssbarnea has joined #zuul | 05:30 | |
| *** chkumar|off is now known as chandankumar | 05:34 | |
| openstackgerrit | Simon Westphahl proposed openstack-infra/zuul master: Check paused parent on node failure of child job https://review.openstack.org/609911 | 05:38 |
| *** jesusaur has quit IRC | 05:41 | |
| *** jesusaur has joined #zuul | 06:12 | |
| openstackgerrit | Merged openstack-infra/zuul-jobs master: Install docker from distro properly https://review.openstack.org/609844 | 07:03 |
| *** jesusaur has quit IRC | 07:08 | |
| *** jesusaur has joined #zuul | 07:13 | |
| *** bhavikdbavishi has quit IRC | 07:16 | |
| openstackgerrit | Merged openstack-infra/zuul master: Fixes for quick-start playbooks https://review.openstack.org/609551 | 07:23 |
| openstackgerrit | Merged openstack-infra/zuul master: Remove disable-recommends in quick-start test https://review.openstack.org/609553 | 07:23 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: Retry jobs failed with MERGER_FAILURE https://review.openstack.org/609945 | 07:53 |
| *** electrofelix has joined #zuul | 08:04 | |
| *** slaweq has quit IRC | 08:46 | |
| openstackgerrit | Merged openstack-infra/zuul-jobs master: Add ensure-python role https://review.openstack.org/609761 | 08:56 |
| *** slaweq has joined #zuul | 09:57 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: wip: add status graph rendering https://review.openstack.org/610001 | 10:27 |
| tristanC | 610001 test the react-force-graph library, it enables d3 style of rendering, but using webgl :) | 10:31 |
| *** pabelanger has quit IRC | 10:40 | |
| *** jpena|off has quit IRC | 10:41 | |
| *** jpena|off has joined #zuul | 10:42 | |
| *** sshnaidm is now known as sshnaidm|off | 10:57 | |
| *** bhavikdbavishi has joined #zuul | 11:37 | |
| *** bhavikdbavishi has quit IRC | 11:46 | |
| *** bhavikdbavishi has joined #zuul | 11:46 | |
| openstackgerrit | Tobias Henkel proposed openstack-infra/nodepool master: Fix skipping of max-servers if not set https://review.openstack.org/610019 | 12:06 |
| dmsimard | tristanC: o_O | 12:39 |
| *** rlandy has joined #zuul | 12:40 | |
| openstackgerrit | Tobias Henkel proposed openstack-infra/nodepool master: Ensure that completed handlers are removed frequently https://review.openstack.org/610029 | 13:07 |
| mordred | tristanC: http://logs.openstack.org/01/610001/1/check/zuul-build-dashboard/aba87b3/job-output.txt.gz#_2018-10-12_11_01_35_020184 sad panda | 13:11 |
| *** bhavikdbavishi1 has joined #zuul | 13:13 | |
| *** bhavikdbavishi has quit IRC | 13:13 | |
| *** bhavikdbavishi1 has quit IRC | 13:17 | |
| *** ssbarnea_ has quit IRC | 13:18 | |
| openstackgerrit | David Shrewsbury proposed openstack-infra/nodepool master: Implement a Kubernetes driver https://review.openstack.org/535557 | 13:39 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/nodepool master: Add tox functional testing for drivers https://review.openstack.org/609515 | 13:39 |
| Shrews | tristanC: i think ^^ will take care of both starting ZK and not running func tests in the tox-py* jobs | 13:40 |
| Shrews | but i've disabled all other tests for the time being | 13:42 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/nodepool master: Ensure that completed handlers are removed frequently https://review.openstack.org/610029 | 13:42 |
| Shrews | tired of waiting for nodes :/ | 13:42 |
| mordred | Shrews: waiting is the worst | 13:53 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/nodepool master: Add tox functional testing for drivers https://review.openstack.org/609515 | 13:54 |
| Shrews | mordred: i though worst cats were the worst? | 13:55 |
| mordred | Shrews: well played | 13:55 |
| mordred | OH GOOD | 13:56 |
| mordred | corvus: re: the container-image-not-containing-javascript-files issue ... | 13:56 |
| mordred | corvus: pbrx build-images is totally working and doing the right thing when I buld locally | 13:57 |
| mordred | so some how pbrx build-images in the same repo on the same base os is doing something different on my laptop and on a gate node | 13:58 |
| Shrews | nice | 13:58 |
| Shrews | mordred: we could change the job to send you an SMS to notify you to build and upload new images | 13:59 |
| mordred | Shrews: I think that's TOTALLY the right way to do it | 14:02 |
| tobiash | mordred: one day of swapping jobs with zuul ;) | 14:06 |
| mordred | tobiash, Shrews: ok - I found one difference between the two systems ... I did not have thelatest python:alpine image | 14:08 |
| Shrews | tristanC: hrm, functional tests seems to be running now with latest patchset (watching the live stream), but WOW is it creating a lot of requests for nodes. Something is definitely off | 14:09 |
| mordred | trying again after doing a docker pull ... do you think we should have pbrx do a docker pull if the image is already there? | 14:09 |
| Shrews | tristanC: http://logs.openstack.org/15/609515/7/check/nodepool-functional-k8s/382e16f/job-output.txt.gz#_2018-10-12_14_03_24_424799 | 14:09 |
| Shrews | mordred: seems like a good way to cut down on this type of discrepancy | 14:11 |
| mordred | Shrews: I mean - it's still working locally | 14:11 |
| mordred | because of course it is | 14:11 |
| Shrews | mordred: even with the latest image??? | 14:12 |
| mordred | yeah | 14:12 |
| Shrews | of course | 14:12 |
| mordred | the logging from the early parts of python setup.py bdist_wheel is different, which is strange to me | 14:13 |
| mordred | I think I want to add a debug logging line to pbrx which prints the sha of the base image is uses | 14:13 |
| Shrews | can't help you there. wheel building is foreign to me | 14:13 |
| *** pabelanger has joined #zuul | 14:16 | |
| corvus | pabelanger, clarkb: did anyone re-run the publication jobs for nodepool? | 14:21 |
| corvus | fungi: ? ^ | 14:21 |
| pabelanger | corvus: I did yes, but we noticed that we only build a python2 wheel, not python3. see http://tarballs.openstack.org/nodepool/ I created https://review.openstack.org/609866/ last night, but need to still confirm it is right, it assumes python3 in installed | 14:23 |
| pabelanger | maybe we should use the build-python-release role in zuul-jobs? | 14:24 |
| pabelanger | not really sure, mordred ideas?^ | 14:24 |
| openstackgerrit | Monty Taylor proposed openstack/pbrx master: Add debug info about base image sha https://review.openstack.org/610052 | 14:27 |
| openstackgerrit | Monty Taylor proposed openstack/pbrx master: Add debug info about base image sha https://review.openstack.org/610052 | 14:27 |
| corvus | pabelanger: 866 seems a plausible thing for us to do | 14:28 |
| mordred | looking | 14:28 |
| pabelanger | https://git.zuul-ci.org/cgit/zuul-jobs/tree/roles/build-python-release seems to be the same idea too | 14:28 |
| mordred | yes. we can also add the universal wheel flag to setup.cfg, sincewe don't contain c extension code | 14:29 |
| mordred | however, since we only support python3 - maybe building wheels with python3 only is a more better choice? | 14:30 |
| pabelanger | yah, that was my only thought about universal wheels | 14:30 |
| mordred | otoh - version specific wheel just means someone would download the tarball on python2 - so it doesn't REALY protect anything | 14:34 |
| openstackgerrit | Monty Taylor proposed openstack-infra/nodepool master: Add universal flag to setup.cfg https://review.openstack.org/610057 | 14:35 |
| mordred | I'll leave that there for discussion purposes | 14:35 |
| mordred | pabelanger: we could also totally use build-python-release role | 14:48 |
| pabelanger | mordred: k, I'll push up a change shortly for that | 14:53 |
| mordred | corvus, Shrews: \o/ I've got a local failure reproduced for the wheel thing | 14:58 |
| corvus | mordred: yay, cause i was just squinting at 609764 and making a weird face.... | 14:59 |
| mordred | corvus: I tried running pbrx build-images in a fresh clone of the zuul repo - and that fails | 15:00 |
| mordred | corvus: so I've got something in my local zuul dir that's making it work | 15:00 |
| openstackgerrit | Merged openstack/pbrx master: Add debug info about base image sha https://review.openstack.org/610052 | 15:00 |
| corvus | mordred: i'm guessing you have 6 years of stuff in your local zuul dir... | 15:00 |
| mordred | yeah | 15:01 |
| mordred | trying different things now ... | 15:01 |
| clarkb | mordred: pabelanger there is a way to indicate a package is python version dependent. black does this to be >=3.6 | 15:02 |
| mordred | clarkb: https://github.com/ambv/black/blob/master/setup.cfg#L1 | 15:03 |
| mordred | we should maybe do that | 15:03 |
| pabelanger | +1 | 15:06 |
| mordred | corvus: I think I have it. I cannot explain WHY - but *shrug* | 15:08 |
| corvus | mordred: does it involve https://review.openstack.org/609764 ? | 15:09 |
| corvus | please say no | 15:09 |
| mordred | corvus: no | 15:09 |
| corvus | good | 15:09 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul master: Add __init__.py to web directory https://review.openstack.org/610080 | 15:11 |
| mordred | corvus: well, you'r enot going to be *happier* | 15:11 |
| mordred | corvus: because it makes just a little sense as 609764 | 15:11 |
| mordred | in fact, it might make _less_ sense | 15:11 |
| corvus | mordred: well, if __init__.py is required at all, it seems to make sense it would be required in the whole hierarchy | 15:12 |
| mordred | corvus: ya - but web/ isn't in the actual resulting hierarchy | 15:13 |
| mordred | it's a top level container with the javascript source code in it | 15:13 |
| corvus | oooh | 15:13 |
| corvus | mmmkay. | 15:13 |
| mordred | this does not cause a top-level web/ to get put in the wheel - or any of the source code | 15:13 |
| mordred | only the built artifacts | 15:13 |
| corvus | mordred: then may i suggest adding a comment to that effect in that file? :) | 15:13 |
| mordred | corvus: :) | 15:14 |
| corvus | i'm certain if we don't, someone will clean it up for us | 15:14 |
| corvus | (though, by that point, we'll be gating on functioning built images, but still.... save folks some time) | 15:14 |
| corvus | mordred: and can we abandon 609764? | 15:14 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul master: Add __init__.py to web directory https://review.openstack.org/610080 | 15:15 |
| mordred | corvus: done | 15:15 |
| mordred | corvus: we can - although we could also land it - I _personally_ like having the hook refer to the real locations of the python files and not the symlinked paths through the javascript source dir | 15:16 |
| mordred | corvus: but I don't feel strongly enough about it to argue in favor of it any more than that sentence | 15:16 |
| mordred | it should be a completely no-op patch | 15:16 |
| *** bhavikdbavishi has joined #zuul | 15:16 | |
| corvus | mordred: can you explain it better in the commit message then? because i can't make heads or tails of the change. | 15:17 |
| corvus | it's probably the 'something something symlinks' part that's getting me :) | 15:21 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul master: Use zuul/web/static in setup_hook https://review.openstack.org/609764 | 15:26 |
| mordred | corvus: done | 15:26 |
| mordred | corvus: well - at the time I wrote it originally it was a complete stab in the dark because my network wasn't good enough to do the container builds locally on my laptop | 15:27 |
| mordred | corvus: so 'something something symlinks' was the best I had :) | 15:27 |
| *** dkehn has joined #zuul | 15:29 | |
| corvus | mordred: that makes much more sense! thanks :) | 15:32 |
| mordred | clarkb: the ambv/black thing in setup.cfg won't help us - it's basically the opposite of the universal thing | 15:35 |
| mordred | clarkb: they have a line in their setup.py | 15:35 |
| clarkb | mordred: there is a thing with pypi where pypi knows not to give python3.5 a black package (this caught dmsimard unawares recnetly) | 15:35 |
| clarkb | is that all baked in through that wheel stuff? | 15:35 |
| mordred | clarkb: assert sys.version_info >= (3, 6, 0), "black requires Python 3.6+" | 15:35 |
| clarkb | mordred: if you pip install black under python3.5 it errors to find a valid package | 15:36 |
| mordred | really? ok - I'll dig in further then | 15:36 |
| clarkb | ya dmsimard thought our mirror was broken | 15:36 |
| mordred | AH I see it now | 15:36 |
| clarkb | (the error is unfortunately not the most clear) | 15:36 |
| mordred | python_requires=">=3.6", | 15:36 |
| mordred | that's in their setup invocation - lemme see if that's a thi ngwe can set | 15:36 |
| dmsimard | yeah, it requires 3.6 | 15:39 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul master: Add line to setup.cfg limiting to python >=3.5 https://review.openstack.org/610089 | 15:42 |
| mordred | clarkb, corvus: ^^ | 15:42 |
| openstackgerrit | Monty Taylor proposed openstack-infra/nodepool master: Add line to setup.cfg limiting to python >=3.5 https://review.openstack.org/610090 | 15:42 |
| clarkb | mordred: those changes might catch anyhwere where we run python2 to do stuff (like with the release jobs) probably a good thing overall but may be a short period of fixing weird stuff | 15:43 |
| mordred | clarkb: yah | 15:43 |
| mordred | clarkb: it's actually making me want to put in a sanity check to pbr | 15:43 |
| mordred | clarkb: so that if python-requires is set, even running setup.py will bail on you | 15:44 |
| mordred | but then I'd have to do that | 15:44 |
| pabelanger | mordred: I mean, I don't mind you doing it :) | 15:45 |
| *** rlandy is now known as rlandy|biab | 15:45 | |
| corvus | py36 seems to be failing on random things a lot more often than py35 | 15:46 |
| tobiash | yes, atm there the random failures are increasing as it seems | 15:47 |
| corvus | i looked at one failure earlier, and afaict, zuul ran ansible and then ansible just didn't do anything. | 15:47 |
| tobiash | I also saw a broken zookeeper today in nodepool | 15:48 |
| tobiash | seems really random :( | 15:48 |
| corvus | the second failure i looked at today looks like more "normal" hard timeouts | 15:48 |
| corvus | yeah. i'm looking for races, but i'm not seeing them. | 15:48 |
| tobiash | so maybe tweaking some timeouts could help | 15:49 |
| clarkb | probably worth a sanity check that we aren't leaking memory like that one set of test suite bugs that caused instability | 15:49 |
| tobiash | so you have something like sar-logging for the duration of the job? | 15:49 |
| clarkb | (thinking about processes doing nothing and swapping/ooming might explain that?) | 15:49 |
| tobiash | we have that (optional) in our base job | 15:50 |
| clarkb | tobiash: our devstack jobs run a dstat in the background. When zuul test suite had the memory leaks last time I ran it locally and profiled it that way. I have no evidence this is happening now other than processes being "slow" doing nothing can be memory pressure | 15:50 |
| tobiash | and generates more system stats graphs than I can think of :) | 15:50 |
| mordred | corvus: fwiw, py36 has a new dictionary implementation | 15:51 |
| mordred | corvus: which is supposed to be better from a memory usage perspective as well as faster - and it is also ordered by default rather than unordered | 15:51 |
| corvus | mordred: yep | 15:51 |
| *** ssbarnea has quit IRC | 16:00 | |
| *** panda has quit IRC | 16:01 | |
| *** panda has joined #zuul | 16:02 | |
| openstackgerrit | Merged openstack-infra/zuul master: Check paused parent on node failure of child job https://review.openstack.org/609911 | 16:10 |
| openstackgerrit | Merged openstack-infra/zuul master: Optionally disable service worker in zuul-web https://review.openstack.org/609626 | 16:10 |
| clarkb | corvus: which bit of config for the quick start sets up the +/-2 values and review categories in gerrit? iirc newer gerrit stopped setting those by default? | 16:15 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul master: WIP: Run dstat and generate graphs in unit tests https://review.openstack.org/610100 | 16:17 |
| corvus | clarkb, tobiash: ^ while i dig up the answer to clarkb's question, that might be interesting | 16:17 |
| corvus | (i mean, cloning something from github and running it is terrible, but just as something to play with for now...) | 16:17 |
| corvus | clarkb: the setup playbook pushes http://git.zuul-ci.org/cgit/zuul/tree/doc/source/admin/examples/playbooks/project.config#n61 to All-Projects config | 16:19 |
| clarkb | corvus: thanks | 16:19 |
| *** rlandy|biab is now known as rlandy | 16:24 | |
| *** ssbarnea has joined #zuul | 16:25 | |
| corvus | we could probably remove the workflow-1 from that | 16:28 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul master: Add reference to svg icon in the PWA manifest https://review.openstack.org/610105 | 16:33 |
| mordred | corvus, tristanC: If I'm understanding correctly, I *think* that ^^ should fix the issue with not having a logo for zuul saved to a phone screen | 16:34 |
| corvus | mordred: why isn't the favicon used? | 16:37 |
| mordred | corvus: different sizes of things I believe - this is what shoudl be used for the icon for the 'app' | 16:38 |
| mordred | corvus: https://medium.com/front-end-hacking/build-a-realtime-pwa-with-react-99e7b0fd3270 and search for "The manifest.json file" and then also https://stackoverflow.com/questions/48839338/which-icon-sizes-are-required-for-progressive-web-apps-pwa-as-of-q1-2018 both have good info that I haven't _fully_ digested | 16:39 |
| mordred | ah - I missed this snippet: "icons must include a 192px and a 512px sized icon" | 16:39 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul master: Add icons in the PWA manifest https://review.openstack.org/610105 | 16:46 |
| *** shanemcd has joined #zuul | 17:03 | |
| shanemcd | o/ pabelanger | 17:03 |
| pabelanger | shanemcd: hello there | 17:03 |
| shanemcd | oh hi | 17:03 |
| openstackgerrit | Merged openstack-infra/nodepool master: Add universal flag to setup.cfg https://review.openstack.org/610057 | 17:04 |
| *** SpamapS has quit IRC | 17:13 | |
| pabelanger | corvus: mordred: do we also want to do ^ for zuul? | 17:15 |
| *** SpamapS has joined #zuul | 17:15 | |
| clarkb | corvus: I'm looking at trying to get the submit logstash jobs role to distinguish between different types of failures. Currently if we fail in pre-run that isn't indicated to elasticsearch because it is recorded as FAILURE. Reading the runAnsiblePlaybook() method in the executor the success var is a boolean (true/false, success/failure) | 17:24 |
| clarkb | corvus: I think at this point we probably have to treat that as existing contract? would it be acceptable to keep an "earliest_failure" zuul var that can be exposed to playbooks to try and expose that further downstream? | 17:25 |
| clarkb | or maybe a "will be retried" boolean value | 17:25 |
| mordred | pabelanger: prolly so - and also 610090 | 17:26 |
| clarkb | I don't think this needs to be solved today, but it will help the long term presentation of bugs affecting gate data | 17:27 |
| *** caphrim007 has joined #zuul | 17:28 | |
| openstackgerrit | Clark Boylan proposed openstack-infra/zuul master: WIP: Run dstat and generate graphs in unit tests https://review.openstack.org/610100 | 17:32 |
| clarkb | previous run failed because yo ucan't use > with command, needs shell | 17:33 |
| corvus | clarkb: thx. and let me think on your failure question a bit. | 17:36 |
| mordred | corvus, tristanC, tobiash: I ran the google lighthouse tool on zuul.openstack.org and it produced this report: inaugust.com/zuul.openstack.org_2018-10-12_11-49-45.report.html | 17:43 |
| dmsimard | mordred: " Device: Emulated Nexus 5X " ? | 17:43 |
| mordred | I think we might want to either add running that to a job (can be installed via yarn) - r just run it manually from time to time | 17:43 |
| dmsimard | Is there a non-mobile version of it ? | 17:44 |
| mordred | dunno - I was actually doing this as part of poking at some of the weirdness on mobile | 17:44 |
| tobiash | Cool | 17:44 |
| mordred | even so - there's things in there - such as apparently we're not setting cache headers on the hashed js/css build artifacts | 17:44 |
| mordred | which is sad, since they can all have an infinite cache ttl, being hashed in the filename and all | 17:45 |
| dmsimard | mordred: thanks for sharing, I'll need to run it on ara too :p | 17:46 |
| mordred | dmsimard: there's also a chrome extension you can use | 17:46 |
| mordred | dmsimard: https://developers.google.com/web/tools/lighthouse/ | 17:46 |
| *** ssbarnea_ has joined #zuul | 18:00 | |
| pabelanger | mordred: corvus: mind adding https://review.openstack.org/608472/ to your review pipeline for +3, we're using that today in ansible-network and want to upstream it | 18:01 |
| mordred | dmsimard: the in-browser version of lighthouse lets you also do it for desktop | 18:03 |
| mordred | dmsimard: chrome -> developer tools -> audits | 18:04 |
| mordred | corvus, tobiash: that __init__.py fix did not work | 18:07 |
| mordred | corvus: I have a new fix coming | 18:11 |
| openstackgerrit | Merged openstack-infra/zuul master: Add __init__.py to web directory https://review.openstack.org/610080 | 18:20 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul master: Simplify Builds page https://review.openstack.org/610137 | 18:30 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul master: Add more information to build page https://review.openstack.org/610138 | 18:30 |
| corvus | mordred: how did you determine it didn't work before it merged? | 18:30 |
| Shrews | i'm guessing he pulled the image down from dockerhub and checked that | 18:36 |
| Shrews | oh, that's a post job | 18:36 |
| Shrews | he's psychic? :) | 18:36 |
| corvus | yeah, the timing is fishy... | 18:37 |
| corvus | the post job is still running | 18:38 |
| corvus | it *just* finished pushing | 18:38 |
| Shrews | corvus: would be nice if we could make the post job use your new docker-compose job to validate good images before we push | 18:39 |
| corvus | Shrews: 609763 turns it into a gating job, so should have the same effect | 18:40 |
| clarkb | http://logs.openstack.org/00/610100/2/check/tox-py36/08ab6f4/dstat.html hrm dstat doesn't have any contents | 18:40 |
| pabelanger | Shrews: corvus: this is exciting! | 18:40 |
| pabelanger | I haven't tried docker compose yet, but looked at the job. Super cool | 18:40 |
| corvus | Shrews: when we incorporate tobiash's 'run a docker registry', we should be able to use the same images the pbrx job is running... then, once i finish the promote pipeline, we should be able to publish the same images we test. | 18:41 |
| Shrews | neat | 18:41 |
| corvus | (put another way, we build the images 3 times now, and are pretty sure they're the same. next step is build them 2 times. then once. :) | 18:41 |
| pabelanger | \o/ | 18:41 |
| corvus | the build api has landed, so i can pick up work on promote pipeline soonish. | 18:42 |
| pabelanger | much requested feature, looking forward to promote pipeline :) | 18:42 |
| corvus | however mordred determined they didn't work, i agree. i just pulled a new zuul-web image and still no joy | 18:44 |
| mordred | corvus: I looked at the output from the job | 18:45 |
| mordred | corvus: specifically, this: http://logs.openstack.org/80/610080/2/check/pbrx-build-container-images/08658e7/job-output.txt.gz#_2018-10-12_15_51_18_559935 | 18:45 |
| mordred | corvus: in a correct build, you'll see the javascript stuff go there | 18:45 |
| corvus | mordred: aha! thanks | 18:45 |
| *** kmalloc is now known as needscoffee | 18:48 | |
| corvus | clarkb: is that link from your ps2? | 18:48 |
| clarkb | corvus: yes | 18:48 |
| corvus | clarkb: http://logs.openstack.org/00/610100/2/check/tox-py36/08ab6f4/job-output.txt.gz#_2018-10-12_18_21_32_997050 | 18:48 |
| clarkb | huh ara doesn't seem to have that in its log for that command | 18:49 |
| clarkb | oh wait this is the pre side ofthings | 18:50 |
| corvus | yeah, we failed to run dstat | 18:50 |
| clarkb | it is the >& | 18:50 |
| clarkb | because sh is used and not bash | 18:50 |
| clarkb | need 2>&1 ? | 18:50 |
| clarkb | er wait no 2>/dev/null ? | 18:50 |
| corvus | or switch to bash? | 18:50 |
| clarkb | or switch to bash | 18:51 |
| corvus | incoming | 18:51 |
| mordred | or switch to bash! | 18:51 |
| *** bhavikdbavishi has quit IRC | 18:51 | |
| clarkb | corvus: do yo uwant to write that change or should I push it up? | 18:51 |
| pabelanger | ha | 18:51 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul master: WIP: Run dstat and generate graphs in unit tests https://review.openstack.org/610100 | 18:51 |
| corvus | clarkb: ^ fixed another thing i anticipate being a problem | 18:51 |
| dmsimard | corvus: that reminds me I never finished a dstat implementation for zuul-jobs https://review.openstack.org/#/c/518374/ | 18:53 |
| dmsimard | I used a systemd unit approach instead of backgrounding the process | 18:54 |
| openstackgerrit | Monty Taylor proposed openstack/pbrx master: Run an sdist before the bdist_wheel command https://review.openstack.org/610142 | 18:55 |
| mordred | tobiash, corvus, Shrews: ^^ | 18:55 |
| corvus | dmsimard: i would use that if you finish it :) | 18:57 |
| dmsimard | I can try that and add your graph generation bits to it | 18:58 |
| dmsimard | should it have a generic name ? like job-metrics or job-graphs | 18:59 |
| corvus | dmsimard: i'd probably -1 the graph generation bits right now -- that's not how we should run software. i just wanted to try it out. but getting dstat running would be great. | 18:59 |
| corvus | (mostly, i want to see if it's worth figuring out how to do the graph generation bits correctly :) | 18:59 |
| dmsimard | sure | 18:59 |
| Shrews | mordred: gee i wish pbrx had tests to validate (and help me understand) what that does. we should work on that at some point :) | 19:01 |
| mordred | Shrews: ++ | 19:01 |
| openstackgerrit | Merged openstack-infra/zuul-jobs master: Create ansible-galaxy-import role https://review.openstack.org/608472 | 19:02 |
| AJaeger | corvus: did you see ianw's comment on https://review.openstack.org/#/c/592850/1 ? IS your -1 still valid? | 19:06 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul master: Remove unneeded cruft from bdist_wheel fix attempts https://review.openstack.org/610143 | 19:08 |
| mordred | Shrews, corvus: ^^ that removes the various other things we tried (turns out none needed) - we should be able to verify the results by looking at the pbrx job log for that patchset | 19:08 |
| *** dkehn has quit IRC | 19:17 | |
| *** dkehn has joined #zuul | 19:18 | |
| clarkb | mordred: could we fix the setup hook to have it do the right hting for bdist wheel? I ask because if you are running setup.py or pip locally it probably won'twork either? | 19:20 |
| openstackgerrit | David Moreau Simard proposed openstack-infra/zuul-jobs master: Add a role to install and configure dstat to run in the background https://review.openstack.org/518374 | 19:20 |
| openstackgerrit | David Moreau Simard proposed openstack-infra/zuul-jobs master: Add a role to install and configure dstat to run in the background https://review.openstack.org/518374 | 19:20 |
| mordred | clarkb: I have not been able to figure out how to - but pip install . or setup.py should both work fine | 19:23 |
| mordred | clarkb: it's really *only* if you're running bdist_wheel - because of what it does with build directories | 19:23 |
| clarkb | does pip wheel work? | 19:24 |
| clarkb | most people will install from our wheels so I guess as long as we get those right its ok. Speaking of do the sdist and wheel publish jobs do the right thing? | 19:24 |
| mordred | pip wheel does something different | 19:25 |
| mordred | and yes | 19:25 |
| mordred | sdist works fine | 19:25 |
| mordred | and our publish job just run python setup.py sdist bdist_wheel | 19:25 |
| mordred | clarkb: I'll keep looking for ways to improve the setup hook in zuul to see if we can hook in to the process earlier - but that's down in the bowels of some deep dark magic | 19:26 |
| logan- | dmsimard: I like the dstat change. would definitely move to use that in OSA. right now we're doing dstat csv+charts in the gate with https://github.com/openstack/openstack-ansible/blob/3200de090d44419f8a80835424c140fcd69e2a1e/scripts/scripts-library.sh#L244-L277 | 19:26 |
| dmsimard | logan-: it's an old patch I revived when I saw https://review.openstack.org/610100 | 19:27 |
| clarkb | mordred: getting the published builds right will cover it for the vast majority of people. Then its likely folks like infra, tobiash and SpamapS installing from source (maybe?) that we'd want to have update their installations if necessary | 19:28 |
| logan- | ah nice, using the same charting and everything :P | 19:29 |
| mordred | clarkb: yah. and even they will be fine as long as their process isn't 'git clone https://git.zuul-ci.org/zuul ; cd zuul ; python setup.py bdist_wheel ; pip install dist/*whl' | 19:29 |
| mordred | clarkb: andif that is their process ... what? | 19:30 |
| mordred | cause that's so much more typing than 'git clone https://git.zuul-ci.org/zuul ; cd zuul ; pip install .' | 19:30 |
| tobiash | clarkb: we're installing from source. Will that get broken? | 19:32 |
| clarkb | tobiash: if you pip install /path/to/zuul it should work as mordred says above | 19:32 |
| clarkb | tobiash: it will only break if you run python setup.py bdist_wheel in an isolated build | 19:32 |
| tobiash | Ok, then I'm probably fine, thx | 19:33 |
| AJaeger | that reminds me: Let's merge fungi's change https://review.openstack.org/599607 for zuul-base-jobs, please. Any reviewer? | 19:34 |
| *** electrofelix has quit IRC | 19:37 | |
| mordred | clarkb, corvus, Shrews, tobiash: http://logs.openstack.org/42/610142/1/check/pbrx-build-zuul-containers/ca53e1e/job-output.txt.gz#_2018-10-12_19_36_59_439944 <-- js content working | 19:44 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul master: Add line to setup.cfg limiting to python >=3.5 https://review.openstack.org/610089 | 19:54 |
| clarkb | http://logs.openstack.org/00/610100/3/check/tox-py36/6647194/dstat.html is pretty neat | 19:58 |
| clarkb | memory use looks pretty good actually | 20:02 |
| clarkb | load average is pretty high peaking around 13 and system cpu usage is also high | 20:02 |
| clarkb | socket count seems high as well. Those are timed wait? so sockets closed and just making sure they clean up nicely? | 20:02 |
| clarkb | there is some really high cpu wai too | 20:05 |
| pabelanger | nice! Should see about running that in ansible-network also | 20:06 |
| clarkb | the iops scales are definitely different between py35 and py36 | 20:07 |
| clarkb | unsure if cloud related or distro (xenial vs bionic) | 20:07 |
| clarkb | this sort of data likely would be useful for hte timeouts or ansible not doing anything | 20:07 |
| openstackgerrit | Merged openstack/pbrx master: Run an sdist before the bdist_wheel command https://review.openstack.org/610142 | 20:09 |
| clarkb | the thing that stands out to me there on py35 vs py36 is the cpu wai difference. If we see that consistently on py36 jobs across different clouds that may be a breadcrumb to follow | 20:12 |
| * clarkb rechecks the change to see if it will run jobs on other clouds | 20:12 | |
| mordred | clarkb: ++ | 20:13 |
| corvus | clarkb: cool! it does strain my browser a bit. | 20:16 |
| corvus | i'm going to be afk for the rest of the afternoon running errands | 20:16 |
| clarkb | see you monday | 20:16 |
| *** ssbarnea_ has quit IRC | 20:17 | |
| *** needscoffee is now known as kmalloc | 20:20 | |
| openstackgerrit | Merged openstack-infra/nodepool master: Add line to setup.cfg limiting to python >=3.5 https://review.openstack.org/610090 | 20:21 |
| openstackgerrit | Merged openstack-infra/zuul master: Quick-start: remove ansible Dockerfile https://review.openstack.org/609555 | 20:21 |
| clarkb | I've approved the change to make quick start gate zuul | 20:26 |
| clarkb | we may want to add that change to nodepool too? since we run nodepool containers too | 20:26 |
| clarkb | then that can replace the zuul nodepool functional job? | 20:26 |
| clarkb | Shrews: ^ | 20:26 |
| openstackgerrit | Clark Boylan proposed openstack-infra/nodepool master: Run zuul-quick-start job https://review.openstack.org/610159 | 20:35 |
| *** ianychoi has joined #zuul | 20:35 | |
| clarkb | that should make sure nodepool doesn't break zuul gate | 20:35 |
| clarkb | we can decide if the zuul nodepool functional job can be removed in a followup, mostly want to avoid nodepool breaking zuul gate with ^ | 20:36 |
| mordred | clarkb: I'm excited about this | 20:37 |
| Shrews | clarkb: I suggested the same sometime recently | 20:48 |
| openstackgerrit | Merged openstack-infra/zuul master: Build new container images for quick-start https://review.openstack.org/609763 | 20:51 |
| openstackgerrit | Merged openstack-infra/zuul master: Add line to setup.cfg limiting to python >=3.5 https://review.openstack.org/610089 | 20:52 |
| mordred | corvus, clarkb, Shrews, tobiash: http://logs.openstack.org/43/610143/1/check/pbrx-build-container-images/2e640a6/job-output.txt.gz#_2018-10-12_19_53_13_352062 looks like it worked properly | 20:56 |
| mordred | so https://review.openstack.org/#/c/610143/ should be good | 20:56 |
| clarkb | mordred: +2 | 20:57 |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul-jobs master: Fix typo with ansible_galaxy_info.token https://review.openstack.org/610164 | 21:00 |
| pabelanger | mordred: clarkb: could I get a quick review on ^ to fix a typo, I didn't properly setup the token | 21:01 |
| pabelanger | and now matches README.rst file | 21:01 |
| mordred | pabelanger: wfm | 21:01 |
| pabelanger | ty | 21:01 |
| clarkb | I've approved it | 21:01 |
| openstackgerrit | Merged openstack-infra/zuul master: Quick-start: switch to polygerrit https://review.openstack.org/609828 | 21:08 |
| clarkb | we don't see the large cpu wai on rax py36 | 21:18 |
| clarkb | but do still see high sys cpu | 21:18 |
| openstackgerrit | Merged openstack-infra/zuul master: Remove unneeded cruft from bdist_wheel fix attempts https://review.openstack.org/610143 | 21:27 |
| pabelanger | I just restored https://review.openstack.org/567696/, the ability to allow a user to toggle compressing console logs. It would be great to get some reviews on it | 21:28 |
| pabelanger | would be one less step somebody needed to do for a POC with zuul publishing jobs | 21:29 |
| clarkb | pabelanger: fwiw I think corvus' quick start handles all of that? | 21:30 |
| pabelanger | clarkb: let me look | 21:30 |
| pabelanger | clarkb: ah, I suspect the http.conf we include has gzip enabled | 21:32 |
| pabelanger | I still think it is valid, to allow a user to disable compression. Without having them fiddle with httpd | 21:33 |
| clarkb | pabelanger: the change as is going to stop compressing the logs for existing users right? | 21:35 |
| clarkb | we probably want to set the compress flag in infra and notify other users if we move forward with that | 21:35 |
| pabelanger | clarkb: yah, good point. I can send out an email to zuul-discuss again | 21:36 |
| *** rlandy has quit IRC | 21:54 | |
| *** pcaruana has quit IRC | 22:20 | |
| openstackgerrit | Merged openstack-infra/zuul-jobs master: Fix typo with ansible_galaxy_info.token https://review.openstack.org/610164 | 22:26 |
| clarkb | of course as soon as you start measuring things they stop failing | 22:49 |
| clarkb | I wonder if we are our own noisy neihbor and that is the problem | 22:49 |
| clarkb | later on friday everything is fine with less noise | 22:49 |
| *** caphrim007 has quit IRC | 23:21 | |
| *** slaweq has quit IRC | 23:32 | |
| *** EvilienM is now known as EmilienM | 23:57 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!