| *** tbachman has joined #openstack-infra | 00:01 | |
| *** tbachman has left #openstack-infra | 00:02 | |
| *** tosky has quit IRC | 00:02 | |
| *** rcernin has joined #openstack-infra | 00:03 | |
| *** jamesmcarthur has joined #openstack-infra | 00:09 | |
| *** jamesmcarthur has quit IRC | 00:13 | |
| *** jesusaur has joined #openstack-infra | 00:18 | |
| *** dave-mccowan has quit IRC | 00:18 | |
| *** agopi has joined #openstack-infra | 00:19 | |
| *** dave-mccowan has joined #openstack-infra | 00:19 | |
| *** harlowja has joined #openstack-infra | 00:25 | |
| *** rcernin has quit IRC | 00:39 | |
| *** shoogz has quit IRC | 00:41 | |
| *** panda is now known as panda|off | 00:42 | |
| *** shoogz has joined #openstack-infra | 00:49 | |
| *** shoogz has joined #openstack-infra | 00:50 | |
| *** bobh has quit IRC | 00:55 | |
| *** dave-mccowan has quit IRC | 01:10 | |
| *** mrsoul has quit IRC | 01:19 | |
| *** diablo_rojo has joined #openstack-infra | 01:45 | |
| openstackgerrit | Merged openstack-infra/project-config master: Remove Fedora Job for Kata project https://review.openstack.org/613690 | 01:54 |
|---|---|---|
| *** yamamoto has quit IRC | 02:03 | |
| *** diablo_rojo has quit IRC | 02:07 | |
| *** jamesmcarthur has joined #openstack-infra | 02:25 | |
| *** jamesmcarthur has quit IRC | 02:29 | |
| *** cfriesen has quit IRC | 02:43 | |
| *** mrsoul has joined #openstack-infra | 02:44 | |
| *** icey has quit IRC | 02:57 | |
| *** icey has joined #openstack-infra | 02:57 | |
| *** diablo_rojo has joined #openstack-infra | 02:59 | |
| *** icey has quit IRC | 03:00 | |
| *** icey has joined #openstack-infra | 03:01 | |
| *** yamamoto has joined #openstack-infra | 03:04 | |
| *** diablo_rojo has quit IRC | 03:20 | |
| *** yamamoto has quit IRC | 03:21 | |
| *** yamamoto has joined #openstack-infra | 03:30 | |
| *** bobh has joined #openstack-infra | 03:41 | |
| *** bobh has quit IRC | 03:42 | |
| *** bobh has joined #openstack-infra | 03:47 | |
| *** bobh has quit IRC | 03:49 | |
| *** diablo_rojo has joined #openstack-infra | 03:49 | |
| *** bobh has joined #openstack-infra | 04:03 | |
| *** yamamoto has quit IRC | 04:19 | |
| *** cfriesen has joined #openstack-infra | 04:24 | |
| *** bobh has quit IRC | 04:26 | |
| *** cfriesen has quit IRC | 04:28 | |
| *** lpetrut has joined #openstack-infra | 04:34 | |
| *** yamamoto has joined #openstack-infra | 04:37 | |
| *** yamamoto has quit IRC | 05:07 | |
| *** yamamoto has joined #openstack-infra | 05:09 | |
| *** jaosorior has quit IRC | 05:20 | |
| *** ykarel|away has joined #openstack-infra | 05:26 | |
| *** carl_cai has joined #openstack-infra | 05:30 | |
| *** d0ugal has quit IRC | 05:32 | |
| *** d0ugal has joined #openstack-infra | 05:33 | |
| *** lpetrut has quit IRC | 05:46 | |
| *** yamamoto has quit IRC | 05:46 | |
| *** d0ugal has quit IRC | 05:55 | |
| *** sthussey has quit IRC | 06:04 | |
| *** diablo_rojo has quit IRC | 06:08 | |
| *** ykarel|away has quit IRC | 06:26 | |
| *** kjackal has joined #openstack-infra | 06:34 | |
| *** yamamoto has joined #openstack-infra | 06:40 | |
| *** roman_g has quit IRC | 06:57 | |
| *** d0ugal has joined #openstack-infra | 07:05 | |
| *** ChosSimbaOne has quit IRC | 07:16 | |
| *** ChosSimbaOne has joined #openstack-infra | 07:16 | |
| *** kjackal has quit IRC | 07:19 | |
| *** d0ugal has quit IRC | 07:30 | |
| *** d0ugal has joined #openstack-infra | 07:34 | |
| *** imacdonn has joined #openstack-infra | 07:36 | |
| *** ccamacho has quit IRC | 07:38 | |
| *** ccamacho has joined #openstack-infra | 07:39 | |
| *** carl_cai has quit IRC | 07:39 | |
| *** d0ugal has quit IRC | 07:42 | |
| *** apetrich has quit IRC | 07:42 | |
| openstackgerrit | Merged openstack-infra/project-config master: Remove airship-armada jobs, as they are all in project https://review.openstack.org/611013 | 07:50 |
| *** slaweq has quit IRC | 07:50 | |
| *** apetrich has joined #openstack-infra | 07:59 | |
| *** weshay has quit IRC | 07:59 | |
| *** apetrich has quit IRC | 08:10 | |
| ssbarnea | fungi : 1.6days queue time... and the graph shows how it varies in time http://dashboard-ci.tripleo.org/d/cEEjGFFmz/cockpit?orgId=1&from=now-90d&to=now (90 days). I wonder what can we do to improve it? | 08:24 |
| ssbarnea | i was starting to get used to ~8h as i was translating it to "probably next morning", but with ~37h this becomes a huge problem, especially as I seen a ~50:50 chance to fail to merge/conflict/infra-failure, which more than doubles the effective ETA for a change. | 08:27 |
| *** apetrich has joined #openstack-infra | 08:28 | |
| *** ccamacho has quit IRC | 08:28 | |
| *** pcaruana has joined #openstack-infra | 08:30 | |
| *** jaosorior has joined #openstack-infra | 08:38 | |
| *** yamamoto has quit IRC | 08:44 | |
| *** yamamoto has joined #openstack-infra | 08:44 | |
| *** yamamoto has quit IRC | 08:46 | |
| *** e0ne has joined #openstack-infra | 08:52 | |
| *** e0ne_ has joined #openstack-infra | 08:56 | |
| *** e0ne has quit IRC | 08:58 | |
| *** jaosorior has quit IRC | 09:05 | |
| *** slaweq has joined #openstack-infra | 09:11 | |
| *** slaweq has quit IRC | 09:15 | |
| *** yamamoto has joined #openstack-infra | 09:18 | |
| *** e0ne_ has quit IRC | 09:39 | |
| ssbarnea | i seen something weird about ImportError: No module named html on "Set pypi_twine_executable" -- on http://logs.openstack.org/25/613725/1/check/test-release-openstack-python3/df34317/job-output.txt.gz | 09:52 |
| *** rcernin has joined #openstack-infra | 09:54 | |
| *** carl_cai has joined #openstack-infra | 09:57 | |
| *** d0ugal has joined #openstack-infra | 10:07 | |
| *** d0ugal has quit IRC | 10:18 | |
| *** d0ugal has joined #openstack-infra | 10:21 | |
| ssbarnea | anyone knows were to find "ensure-twine" role? http://codesearch.openstack.org/?q=ensure-twine&i=nope&files=&repos= does not help here. | 10:37 |
| *** slaweq has joined #openstack-infra | 10:42 | |
| *** kjackal has joined #openstack-infra | 10:42 | |
| *** d0ugal has quit IRC | 10:46 | |
| *** pbourke has quit IRC | 10:48 | |
| *** pbourke has joined #openstack-infra | 10:49 | |
| clarkb | ssbarnea: zuul-jobs repo in roles dir | 10:50 |
| ssbarnea | clarkb thanks, not sure why search didn't work, probably because i should have search for path only. | 10:59 |
| ssbarnea | apparently is missing the html requirement and this is why the test jobs fails. | 10:59 |
| *** kjackal has quit IRC | 11:02 | |
| openstackgerrit | Sorin Sbarnea proposed openstack-infra/zuul-jobs master: Add missing optiuonal dependencies to ensure-twine https://review.openstack.org/613726 | 11:06 |
| openstackgerrit | Sorin Sbarnea proposed openstack-infra/zuul-jobs master: Add missing optional dependencies to ensure-twine https://review.openstack.org/613726 | 11:07 |
| *** rcernin has quit IRC | 11:13 | |
| *** slaweq has quit IRC | 11:15 | |
| openstackgerrit | Sorin Sbarnea proposed openstack-infra/zuul-jobs master: Updated bindep to cover for MacOS requirments https://review.openstack.org/613727 | 11:26 |
| openstackgerrit | Sorin Sbarnea proposed openstack-infra/zuul-jobs master: Updated bindep to cover for MacOS requirements https://review.openstack.org/613727 | 11:27 |
| *** kjackal has joined #openstack-infra | 11:31 | |
| openstackgerrit | Sorin Sbarnea proposed openstack-infra/zuul-jobs master: Makes queue import cross py2/py3 compatible https://review.openstack.org/613728 | 11:51 |
| *** tosky has joined #openstack-infra | 12:04 | |
| *** carl_cai has quit IRC | 12:06 | |
| *** mrsoul has quit IRC | 12:09 | |
| *** slaweq has joined #openstack-infra | 12:11 | |
| ssbarnea | clarkb : check https://review.openstack.org/#/c/613728/ -- it seems that to make one fix one needs to add 3 others :D | 12:13 |
| *** yamamoto has quit IRC | 12:13 | |
| *** yamamoto has joined #openstack-infra | 12:14 | |
| *** yamamoto has quit IRC | 12:15 | |
| *** jaosorior has joined #openstack-infra | 12:15 | |
| *** yamamoto has joined #openstack-infra | 12:16 | |
| *** yamamoto has quit IRC | 12:17 | |
| *** dave-mccowan has joined #openstack-infra | 12:29 | |
| *** dave-mccowan has quit IRC | 12:33 | |
| *** mrsoul has joined #openstack-infra | 12:39 | |
| *** slaweq has quit IRC | 12:44 | |
| *** yamamoto has joined #openstack-infra | 12:54 | |
| ssbarnea | AJaeger coreycb : a hotfix https://review.openstack.org/#/c/613728/ | 12:56 |
| openstackgerrit | Sorin Sbarnea proposed openstack-infra/zuul-jobs master: Add missing optional dependencies to ensure-twine https://review.openstack.org/613726 | 12:59 |
| *** yamamoto has quit IRC | 13:08 | |
| *** yamamoto has joined #openstack-infra | 13:08 | |
| AJaeger | ssbarnea: let's test that 613726 does the right thing - please show the result of a good run with depends-on | 13:14 |
| *** yamamoto has quit IRC | 13:20 | |
| *** yamamoto has joined #openstack-infra | 13:37 | |
| openstackgerrit | Merged openstack-infra/zuul-jobs master: Makes queue import cross py2/py3 compatible https://review.openstack.org/613728 | 13:38 |
| *** yamamoto has quit IRC | 13:45 | |
| *** yamamoto has joined #openstack-infra | 13:54 | |
| *** auristor has quit IRC | 13:59 | |
| *** yamamoto has quit IRC | 13:59 | |
| fungi | ssbarnea: ways to improve the time through the tripleo shared gate queue: fix nondeterminism in jobs, use fewer nodes, run shorter jobs, run fewer jobs, approve fewer changes... | 14:02 |
| *** dklyle has quit IRC | 14:04 | |
| fungi | though at the moment there's almost nothing running, so seems we caught up when people stopped pushing/approving things | 14:04 |
| *** dklyle has joined #openstack-infra | 14:06 | |
| fungi | the main reason the tripleo gate pipeline has been running long is because when things are active we don't have enough nodes to satisfy the growing backlog of requests because tripleo is starving those resources (clarkb took preliminary measurements that indicate that half the time our nodes spend running jobs, they're running jobs for tripleo repos) | 14:08 |
| fungi | so significantly reducing the node count, run time and nondeterministic failures for tripleo's jobs is about the only thing that will solve the tripleo gate backlog | 14:09 |
| fungi | likely the biggest impact on queue throughput for tripleo will come from fixing the nondeterministic failures, leading to fewer gate resets, leading to greater parallelism in testing approved changes | 14:11 |
| ssbarnea | fungi: this was my impression too, i will talk with wes next week, we need to prio this work. | 14:12 |
| fungi | but understandably, when there aren't enough test resources to go around, developing and testing and proving those fixes becomes harder, so using fewer resources overall will be a big help there | 14:12 |
| *** auristor has joined #openstack-infra | 14:12 | |
| ssbarnea | i raised the issue of non-deterministic failures (i call them random) two months ago, but we always had something more important | 14:13 |
| fungi | the irony is that the "more important" things will take a lot longer to get done if the job failures aren't addresses ;) | 14:14 |
| fungi | er, addressed | 14:14 |
| ssbarnea | one of the things we are moving now is standalone deployment, this is in progress, so will happen soon. | 14:14 |
| fungi | yeah, that sounds like it will result in a significant reduction in node consumption. now that we have measurements we can use to compare, it will be easier to find out exactly how effective too | 14:15 |
| *** slaweq has joined #openstack-infra | 14:19 | |
| ssbarnea | the queue length is very good long term measurement, also the failure rate is useful but we need to find a way to measure only the random failure (rate). | 14:21 |
| *** jamesmcarthur has joined #openstack-infra | 14:22 | |
| ssbarnea | i do not exaggerate when I say that on average I need to do 2.5x rechecks before it merges. so it may be true that we have acceptable level of resources but not enough predictability on outcome. | 14:23 |
| ssbarnea | damn, now i see again the abandoning of job to free gate, this is not normal. | 14:24 |
| *** slaweq has quit IRC | 14:25 | |
| fungi | well, from a measurement standpoint, i meant now that we have detailed information on the number of node-seconds consumed per repo, we can find out how all of those various efforts put together are reducing tripleo's consumption of job resources in the overall system | 14:25 |
| *** slaweq has joined #openstack-infra | 14:26 | |
| *** slaweq has quit IRC | 14:32 | |
| *** lpetrut has joined #openstack-infra | 14:35 | |
| *** yamamoto has joined #openstack-infra | 14:36 | |
| *** lpetrut has quit IRC | 14:50 | |
| *** dklyle has quit IRC | 15:03 | |
| *** dklyle has joined #openstack-infra | 15:03 | |
| ssbarnea | AJaeger: I seen your msg and replied on https://review.openstack.org/#/c/613725/ -- it seems that depends-on does not work in this case. | 15:11 |
| *** bobh has joined #openstack-infra | 15:12 | |
| AJaeger | ssbarnea: it should work - please show me the example | 15:13 |
| AJaeger | ah, 613725 is the example... | 15:13 |
| AJaeger | ssbarnea: indeed - the role is used by a job defined in project-config and thus we do not allow this change since project-config is trusted. | 15:18 |
| AJaeger | ssbarnea: I'll comment on the review and will +2 | 15:18 |
| ssbarnea | thanks | 15:19 |
| ssbarnea | in fact this new job came as a surprise as it was not my original intention to add it, but what it does is really useful. many problems can be related to the publishing part. | 15:20 |
| AJaeger | ssbarnea: indeed - the release team introduced that to the template to catch those publishing problems | 15:24 |
| fungi | we've had many, many release failures in the past because of various aspects of projects not being set up correctly for building release artifacts, or building artifacts which pypi subsequently rejects | 15:25 |
| AJaeger | fungi, are you happy with 613725 ? Or should we ask release team to review? | 15:25 |
| fungi | wouldn't be a bad idea to get input from smcginnis maybe (i think dhellmann is still out?) | 15:26 |
| fungi | just to make sure | 15:26 |
| * AJaeger added smcginnis to https://review.openstack.org/#/c/613725/ now | 15:26 | |
| fungi | i assume this is breaking because we hadn't exercised it against markdown-flavored readmes? | 15:27 |
| AJaeger | agreed | 15:27 |
| fungi | hrm, the readme for jjb is rst not md | 15:28 |
| fungi | so that doesn't explain it | 15:28 |
| fungi | perhaps a new release of twine? | 15:28 |
| *** slaweq has joined #openstack-infra | 15:28 | |
| fungi | Last released: Sep 24, 2018 | 15:30 |
| fungi | so that's not it | 15:30 |
| fungi | https://pypi.org/project/readme_renderer/ | 15:31 |
| fungi | Last released: About 3 days ago | 15:31 |
| fungi | https://github.com/pypa/readme_renderer/issues/133 | 15:32 |
| fungi | Warning message says to install readme_render[md] but cmarkgfm must be installed | 15:32 |
| fungi | https://github.com/pypa/readme_renderer/pull/130 | 15:34 |
| fungi | Move cmarkgfm dependency into [md] extra | 15:34 |
| fungi | that appeared in the latest release | 15:34 |
| AJaeger | ah | 15:36 |
| fungi | AJaeger: ssbarnea: https://github.com/pypa/readme_renderer/pull/134 is going to fix it in the next readme_renderer release, looks like | 15:40 |
| fungi | it removed that import | 15:41 |
| ssbarnea | fungi: still we have the md support to cover, so this change should fix more than one problem. | 15:43 |
| AJaeger | fungi: can we wait for that release instead of merging 613725? Or what do you propose? | 15:43 |
| ssbarnea | unless you know that "html" could produce problems,... | 15:43 |
| fungi | well, the addition of html will become unnecessary with the next readme_renderer release | 15:44 |
| ssbarnea | funny, I see 'asottile' everywhere I go | 15:44 |
| AJaeger | ssbarnea: so, let's add a TODO to remove it - ok? | 15:44 |
| fungi | the inclusion of readme_renderer[md] will enable projects using that job to have markdown-flavored readmes in addition to rst, and will silence a warning from twine, so seems fine | 15:45 |
| fungi | but yeah, if we get a todo comment in there about removing html once there is a readme_renderer release available containing pr#134, i think that change is otherwise fine | 15:46 |
| fungi | and get a little more accurate detail in the commit message now that we know this is a regression introduced by the last readme_renderer release a few days ago | 15:47 |
| *** jamesmcarthur has quit IRC | 15:48 | |
| *** jamesmcarthur has joined #openstack-infra | 15:49 | |
| ssbarnea | i asked on #pypa for a hotfix release, lets wait a bit and see if they make it | 15:52 |
| ssbarnea | hopefully it is only about adding a tag | 15:52 |
| *** jamesmcarthur has quit IRC | 15:53 | |
| fungi | thanks! | 15:53 |
| fungi | if we do need to fix this sooner we can emergency approve your change, otherwise you can just make it about adding optional markdown checking | 15:54 |
| fungi | usually if i see a job start systematically failing when i know it's already heavily-used across projects and hasn't been altered recently, i go looking to see how recently any of the modules listed in the traceback have had new releases. more often than not that leads to the actual cause | 15:57 |
| *** dklyle has quit IRC | 16:15 | |
| openstackgerrit | Sorin Sbarnea proposed openstack-infra/zuul-jobs master: Add missing optional dependencies to ensure-twine https://review.openstack.org/613726 | 16:18 |
| ssbarnea | AJaeger fungi : I think I found an alternative: https://review.openstack.org/#/c/613726/ -- i guess the original twine condition has a similar root cause. | 16:25 |
| ssbarnea | i am confident that next version of readme_renderer will contain the fix. | 16:25 |
| openstackgerrit | Sorin Sbarnea proposed openstack-infra/zuul-jobs master: Updated bindep to cover for MacOS requirements https://review.openstack.org/613727 | 16:31 |
| *** e0ne has joined #openstack-infra | 16:39 | |
| *** slaweq has quit IRC | 16:42 | |
| *** slaweq has joined #openstack-infra | 16:49 | |
| *** e0ne has quit IRC | 16:53 | |
| openstackgerrit | Sorin Sbarnea proposed openstack-infra/infra-specs master: WIP: Propose introduction of priority queues to zuul https://review.openstack.org/613753 | 16:55 |
| *** slaweq has quit IRC | 17:34 | |
| *** bobh has quit IRC | 17:35 | |
| *** bobh has joined #openstack-infra | 17:51 | |
| *** HenryG has quit IRC | 18:31 | |
| *** HenryG has joined #openstack-infra | 18:32 | |
| *** icey has quit IRC | 18:50 | |
| *** icey has joined #openstack-infra | 18:57 | |
| *** d0ugal has joined #openstack-infra | 18:58 | |
| *** apetrich has quit IRC | 19:02 | |
| *** d0ugal has quit IRC | 19:04 | |
| *** apetrich has joined #openstack-infra | 19:19 | |
| *** bobh has quit IRC | 19:28 | |
| *** bobh has joined #openstack-infra | 19:29 | |
| *** mattoliverau has joined #openstack-infra | 19:33 | |
| *** dhill_ has quit IRC | 20:18 | |
| *** imacdonn has quit IRC | 20:32 | |
| *** imacdonn has joined #openstack-infra | 20:32 | |
| *** armax has quit IRC | 21:57 | |
| *** slaweq has joined #openstack-infra | 22:26 | |
| *** jesusaur has quit IRC | 22:27 | |
| *** pcaruana has quit IRC | 22:33 | |
| *** tosky has quit IRC | 22:52 | |
| *** otherwiseguy has quit IRC | 23:38 | |
| *** haleyb has quit IRC | 23:39 | |
| *** slaweq has quit IRC | 23:46 | |
| *** smarcet has joined #openstack-infra | 23:46 | |
| *** Adri2000 has quit IRC | 23:49 | |
| *** d0ugal has joined #openstack-infra | 23:54 | |
| *** Adri2000 has joined #openstack-infra | 23:55 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!