| tristanC | fungi: mordred: the linter job goern is using is proposed here: https://review.openstack.org/#/c/530682/ . The sf-jobs projects we setup with software-factory is a staging projects which just contains all those reviews: https://review.openstack.org/#/q/project:openstack-infra/zuul-jobs+owner:self+status:open | 01:25 |
|---|---|---|
| tristanC | and openstack-infra/zuul-jobs shadow sf-jobs so that there is no conflict if these reviews ever got merged | 01:26 |
| *** spsurya has joined #zuul | 02:51 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul master: WIP: Report to gerrit over HTTP https://review.openstack.org/577027 | 03:10 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul master: WIP: Report to gerrit over HTTP https://review.openstack.org/577027 | 03:11 |
| *** yolanda has quit IRC | 03:12 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul master: WIP: Report to gerrit over HTTP https://review.openstack.org/577027 | 03:59 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul master: WIP: Support line comments in Gerrit https://review.openstack.org/577035 | 03:59 |
| tobiash | corvus: what I'm asking myself regarding line comments is that the line numbers in the jobs may not match the line numbers in gerrit/github because of zuul's wpeculative merging | 04:13 |
| tobiash | I don't have an idea how to solve that though | 04:14 |
| *** SpamapS has quit IRC | 04:29 | |
| *** bhavik1 has joined #zuul | 04:59 | |
| *** swest has joined #zuul | 05:02 | |
| *** nchakrab has joined #zuul | 05:08 | |
| *** nchakrab has quit IRC | 05:21 | |
| *** bhavik1 has quit IRC | 05:23 | |
| *** yolanda has joined #zuul | 05:34 | |
| corvus | tobiash: good point. maybe it'll be good enough most of the time? we could try to reconcile them -- find out what line number in the patch corresponds to the line number on disk. or, folks could choose to use it without the speculative merge (ie, check out the actual change -- it's still in the git history somewhere). or folks could choose to disable file comments if there's a merge involved. | 05:44 |
| corvus | tobiash: also... i was not expecting you to be awake. :) | 05:44 |
| tobiash | corvus: that's my normal time to be awake ;) | 05:46 |
| tobiash | corvus: thinking about the logs, my problem is that I need the debug logging but the start/stop messages don't scale well with the number of tenants/pipelines | 05:47 |
| tobiash | corvus: is there a way that I can filter them without sacrificing debug logging? | 05:47 |
| corvus | tobiash: i basically picked the gerrit data structure for file comments -- it seemed like a good one; if you have a second to think about whether it would pose any problems for github, i'd appreciate it. | 05:48 |
| tobiash | corvus: regarding github there is that new checks api | 05:48 |
| tobiash | I think that's also line based | 05:48 |
| tobiash | so the best thing I can come up with is to document that there might be mismatches and how to check out the original commit within the job (which would be useful for linting jobs anyway) | 05:49 |
| corvus | tobiash: what's your goal with the logging? i'd like to get everything to a point where an operator can know what's happening with just the info log. debug should only be useful for.. well, actualy debugging of zuul, where we need lots of spam... | 05:49 |
| tobiash | corvus: in openshift I the console logs are aggregated and displayed and that's my primary source of coping with 'oh hey something was wrong with my change' can you take a look at it? | 05:50 |
| tobiash | and the info log is pretty limited | 05:50 |
| corvus | tobiash: can you help identify what's missing from the info log for your use cases? maybe we can promote some debug lines to info? | 05:51 |
| tobiash | corvus: I could try to set it to info and promote anything which prevents common analysis over time | 05:52 |
| tobiash | gah, github3.py levels are also weird | 05:53 |
| tobiash | 2018-06-21 05:51:26,667 INFO github3: JSON was returned | 05:53 |
| corvus | yeah, that doesn't look very info | 05:53 |
| tobiash | that's useless for info | 05:53 |
| corvus | oh but that's github3 | 05:53 |
| corvus | we shouldn't be logging that anyway | 05:53 |
| corvus | we should be supressing github3 to warning or above | 05:54 |
| tobiash | I log github3 info level (not debug) to be able to see the requests to our github | 05:54 |
| *** nchakrab has joined #zuul | 05:54 | |
| tobiash | but that's my personal log config for that | 05:54 |
| tobiash | that makes it easy to spot too chatty communication with github | 05:55 |
| corvus | ah ok | 05:55 |
| corvus | well, you could do github3@info to your debug log, and github3@warn to the info log | 05:55 |
| tobiash | actually github3.py info is pretty ok beside that json message | 05:56 |
| tobiash | maybe I'll create a pull request for that | 05:56 |
| corvus | maybe a pr.. ya :) | 05:56 |
| *** Rohaan has joined #zuul | 06:43 | |
| *** gtema has joined #zuul | 06:46 | |
| *** abelur has quit IRC | 07:03 | |
| *** abelur has joined #zuul | 07:09 | |
| *** Rohaan has quit IRC | 07:13 | |
| *** pcaruana has joined #zuul | 07:22 | |
| *** sshnaidm|off has quit IRC | 07:40 | |
| *** jpena|off is now known as jpena | 07:42 | |
| *** electrofelix has joined #zuul | 07:59 | |
| *** sshnaidm has joined #zuul | 08:18 | |
| *** hashar has joined #zuul | 08:20 | |
| *** nchakrab has quit IRC | 08:42 | |
| *** nchakrab has joined #zuul | 08:55 | |
| *** gtema has quit IRC | 09:13 | |
| *** sshnaidm is now known as sshnaidm|lnch | 09:26 | |
| *** spsurya has quit IRC | 09:58 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/zuul-jobs master: Use https for security.debian.org https://review.openstack.org/577119 | 10:08 |
| *** snapiri has joined #zuul | 10:29 | |
| *** sshnaidm|lnch is now known as sshnaidm | 11:04 | |
| *** nchakrab has quit IRC | 12:12 | |
| *** rlandy has joined #zuul | 12:14 | |
| *** swest has quit IRC | 13:07 | |
| *** myoung|off is now known as myoung | 13:09 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: gerrit: remove suspicious change number str cast https://review.openstack.org/577154 | 13:11 |
| mhu | Hello, I'm trying to set up an environment to run unit tests, tox -ep35 fails for me with "Exception: More than one thread is running" on master. Anybody knows how to fix that? | 13:17 |
| tristanC | corvus: we are having an odd behavior since we upgraded to gerrit-2.14.7, we suspect 577154 to be the root cause, but we could use your expert eyes on that matter | 13:18 |
| tristanC | corvus: basically child change doesn't get dequeue when parent changes are updated, here is the relevant debug logs: http://paste.openstack.org/show/724023/ | 13:18 |
| tristanC | tobiash: perhaps you had a similar issue after a gerrit upgrade? or did you stay in 2.13? | 13:25 |
| *** nchakrab has joined #zuul | 13:27 | |
| *** gtema has joined #zuul | 13:29 | |
| tristanC | oops, paste got truncated, complete version is: https://paste.fedoraproject.org/paste/mDbj1L-1RYeOleiGSAOVJw | 13:34 |
| tobiash | tristanC: I don't manage a gerrit anymore and the one we currenty use in addition to github is an externally managed gerrit 2.9 | 13:37 |
| *** spsurya has joined #zuul | 13:44 | |
| tristanC | fbo: btw, the change that introduced this cast ( https://review.openstack.org/433748 ) mention an issue with some code that isn't there anymore | 13:47 |
| fbo | tristanC: ah yes so it seems to be safe to remove then | 13:52 |
| tobiash | clarkb: I'm a step further into understanding the github multi-enqueue issue | 13:57 |
| tobiash | and I'm afraid it's a complicated heavily timing based thing | 13:57 |
| tobiash | so the use case is you have pull request 1,aaaaa with a review | 13:58 |
| tobiash | now you push an update to that and get pull request 1,bbbbb | 13:58 |
| tobiash | github sends two events in this case: dismissed stale review and synchronized | 13:59 |
| tobiash | if zuul gets the dismissed stale review first it updates its data of 1,aaaaa | 14:00 |
| tobiash | that includes the github supplied updated_at timestamp of PR #1 (note that github has no patchset concept as gerrit) | 14:00 |
| tobiash | then zuul processes 1,bbbbb, enqueues it into the pipeline | 14:01 |
| tobiash | during that it checks if 1,bbbbb is an update of 1,aaaaa | 14:01 |
| tobiash | note that because github only knows PR 1 both PullRequest objects in zuul get the same timestamp | 14:02 |
| tobiash | and then the isUpdateOf check fails: http://git.openstack.org/cgit/openstack-infra/zuul/tree/zuul/driver/github/githubmodel.py#n45 | 14:03 |
| tobiash | and if that fails zuul doesn't dequeue the old pr | 14:03 |
| tobiash | at first glance I don't know a way how we could fix that | 14:08 |
| *** nchakrab has quit IRC | 14:08 | |
| *** nchakrab has joined #zuul | 14:09 | |
| *** acozine1 has joined #zuul | 14:14 | |
| clarkb | maybe dont update the timestamp using the dismissal event? | 14:16 |
| clarkb | then the original push timestamp remains and b canupdate it and be newer? | 14:16 |
| tobiash | clarkb: the problem is that zuul *updates* this timestamp on the dismissal event | 14:16 |
| tobiash | it uses the updated_at timestamp of the PR which is the same for any version when newly requested | 14:17 |
| clarkb | right can we special case the dismissal event to not update the timestamp? | 14:17 |
| tobiash | it's not only the dismissal event but any event that causes an update of the pr | 14:18 |
| tobiash | so we would need to switch that to *never* update the timestamp | 14:18 |
| tobiash | (if we already have one) | 14:18 |
| clarkb | hrm | 14:18 |
| tobiash | or we don't rely on the timestamp and just compare the git sha's and hope that if they differ that's an update | 14:19 |
| tobiash | that's the second thing I could think of | 14:19 |
| tobiash | the problem is that the timestamp in github is not attached to the commit but to the pull request | 14:20 |
| tobiash | in gerrit there's no problem with that as we have a 1:1 relationship between zuul changes and gerrit patchsets | 14:20 |
| tobiash | in github we have a n:1 relation between zuul changes and the github pr | 14:21 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: WIP First try to solve github multi-enqueue error https://review.openstack.org/577173 | 14:23 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: WIP First try to solve github multi-enqueue error https://review.openstack.org/577173 | 14:24 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: WIP: Second try to fix github multi-enqueue issue https://review.openstack.org/577175 | 14:25 |
| tobiash | clarkb: that would be basically the two ideas to fix that ^ | 14:26 |
| *** pcaruana has quit IRC | 14:56 | |
| *** nchakrab has quit IRC | 15:18 | |
| *** ianychoi has joined #zuul | 15:48 | |
| *** spsurya has quit IRC | 16:28 | |
| *** nchakrab has joined #zuul | 16:59 | |
| *** nchakrab has quit IRC | 17:04 | |
| *** elyezer has quit IRC | 17:06 | |
| *** zaro__ is now known as zaro | 17:08 | |
| *** jpena is now known as jpena|off | 17:12 | |
| openstackgerrit | Matthieu Huin proposed openstack-infra/zuul master: [WIP] web: add tenant-scoped autohold, enqueue https://review.openstack.org/576907 | 17:53 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/nodepool master: Remove a done TODO in static driver https://review.openstack.org/577244 | 18:01 |
| *** gtema has quit IRC | 18:08 | |
| openstackgerrit | Matthieu Huin proposed openstack-infra/zuul master: [WIP] web: add tenant-scoped autohold, enqueue https://review.openstack.org/576907 | 18:18 |
| openstackgerrit | Merged openstack-infra/nodepool master: Pass zk connection to ProviderManager.start() https://review.openstack.org/574895 | 19:00 |
| openstackgerrit | Merged openstack-infra/nodepool master: Make a proper driver notification API https://review.openstack.org/574896 | 19:01 |
| *** weshay|ruck is now known as weshay_PTO | 19:12 | |
| openstackgerrit | Merged openstack-infra/nodepool master: Pre-register static nodes https://review.openstack.org/574897 | 19:16 |
| openstackgerrit | Merged openstack-infra/nodepool master: Clarify static node name uniqueness in docs https://review.openstack.org/575159 | 19:17 |
| Shrews | oh yay. those are satisfying merges | 19:20 |
| *** SpamapS has joined #zuul | 19:23 | |
| tobiash | looks like we have a race in tox-remote: http://logs.openstack.org/22/572522/3/gate/zuul-tox-remote/168c0a1/job-output.txt.gz#_2018-06-20_14_12_23_726596 | 19:26 |
| mordred | Shrews: \o/ | 19:27 |
| *** acozine1 has quit IRC | 19:29 | |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: Fix race in tox-remote test_command https://review.openstack.org/577279 | 19:30 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: Fix race in tox-remote test_command https://review.openstack.org/577279 | 19:32 |
| *** trishnag has quit IRC | 19:32 | |
| tobiash | that should fix the race ^ | 19:33 |
| mordred | tobiash: lgtm | 19:35 |
| tobiash | \o/ | 19:35 |
| tobiash | mordred: that would be an easy review too: https://review.openstack.org/573641 | 19:35 |
| tobiash | reduces logging noise | 19:36 |
| tobiash | (test logging noise) | 19:36 |
| tobiash | Shrews: I guess 572342 is obsolete now? | 19:38 |
| openstackgerrit | Merged openstack-infra/nodepool master: Remove a done TODO in static driver https://review.openstack.org/577244 | 19:39 |
| Shrews | tobiash: i think so | 19:39 |
| tobiash | k, thx | 19:39 |
| mordred | tobiash: done | 19:42 |
| tobiash | :) | 19:42 |
| openstackgerrit | Merged openstack-infra/zuul master: Don't use GRANT to create new MySQL users https://review.openstack.org/573641 | 20:06 |
| *** yolanda_ has joined #zuul | 20:07 | |
| openstackgerrit | Merged openstack-infra/zuul master: Make streaming test run longer than connection timeout https://review.openstack.org/572522 | 20:07 |
| *** trishnag has joined #zuul | 20:09 | |
| *** trishnag has quit IRC | 20:09 | |
| *** trishnag has joined #zuul | 20:09 | |
| *** yolanda has quit IRC | 20:11 | |
| openstackgerrit | Merged openstack-infra/zuul master: Fix race in tox-remote test_command https://review.openstack.org/577279 | 20:20 |
| *** myoung is now known as myoung|off | 21:30 | |
| *** hashar has quit IRC | 22:12 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: gerrit: cast change and patchset numbers to str https://review.openstack.org/577154 | 23:00 |
| *** rlandy has quit IRC | 23:09 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!