| *** yolanda__ has joined #zuul | 00:09 | |
| *** yolanda_ has quit IRC | 00:13 | |
| *** ianychoi_ is now known as ianychoi | 01:16 | |
| *** yolanda_ has joined #zuul | 01:51 | |
| *** yolanda__ has quit IRC | 01:54 | |
| *** yolanda__ has joined #zuul | 01:59 | |
| *** yolanda_ has quit IRC | 02:02 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: wip: add github event filter debug https://review.openstack.org/580547 | 03:23 |
|---|---|---|
| *** swest has joined #zuul | 05:11 | |
| *** yolanda_ has joined #zuul | 05:48 | |
| openstackgerrit | Tobias Henkel proposed openstack-infra/nodepool master: Add test with unresolvable static node https://review.openstack.org/580063 | 05:50 |
| *** yolanda__ has quit IRC | 05:52 | |
| *** fbo|off is now known as fbo | 06:04 | |
| *** nchakrab has joined #zuul | 06:12 | |
| *** nchakrab_ has joined #zuul | 06:13 | |
| *** nchakrab has quit IRC | 06:17 | |
| openstackgerrit | Tobias Henkel proposed openstack-infra/nodepool master: Fix relaunch attempts when hitting quota errors https://review.openstack.org/536930 | 06:39 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/nodepool master: Fix relaunch attempts when hitting quota errors https://review.openstack.org/536930 | 06:41 |
| *** dmellado has joined #zuul | 06:46 | |
| openstackgerrit | Tobias Henkel proposed openstack-infra/nodepool master: Fix race at shutdown https://review.openstack.org/580564 | 06:46 |
| *** gtema_ has joined #zuul | 06:57 | |
| *** yolanda_ is now known as yolanda | 07:12 | |
| *** nchakrab_ has quit IRC | 07:20 | |
| *** nchakrab has joined #zuul | 07:40 | |
| *** nchakrab_ has joined #zuul | 07:42 | |
| *** nchakrab has quit IRC | 07:46 | |
| *** jpena|off is now known as jpena | 07:47 | |
| *** yolanda_ has joined #zuul | 07:54 | |
| *** yolanda has quit IRC | 07:56 | |
| *** hashar has joined #zuul | 09:29 | |
| *** abelur has quit IRC | 10:07 | |
| *** abelur has joined #zuul | 10:07 | |
| *** nchakrab_ has quit IRC | 10:38 | |
| *** nchakrab has joined #zuul | 10:39 | |
| *** nchakrab has quit IRC | 10:53 | |
| *** nchakrab has joined #zuul | 10:54 | |
| *** yolanda_ is now known as yolanda | 11:12 | |
| *** jpena is now known as jpena|lunch | 11:47 | |
| *** rlandy has joined #zuul | 12:37 | |
| *** nchakrab_ has joined #zuul | 12:39 | |
| *** nchakrab has quit IRC | 12:42 | |
| *** nchakrab_ has quit IRC | 12:43 | |
| mordred | tobiash: your quota patch makes me think we should be raising a QuotaExceeded error in openstacksdk so you don't have to do a string match on the exception text | 12:50 |
| tobiash | mordred: that absolutely makes sense :) | 12:50 |
| mordred | tobiash: I'll put it on my list | 12:51 |
| *** yolanda_ has joined #zuul | 12:54 | |
| *** yolanda has quit IRC | 12:57 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul master: Add job to build container images using pbrx https://review.openstack.org/580160 | 12:58 |
| *** jpena|lunch is now known as jpena | 12:58 | |
| *** nchakrab has joined #zuul | 13:06 | |
| openstackgerrit | Merged openstack-infra/zuul-jobs master: Install build bindep profiles alongside doc and test https://review.openstack.org/580521 | 13:08 |
| openstackgerrit | Merged openstack-infra/nodepool master: Fix race at shutdown https://review.openstack.org/580564 | 13:15 |
| *** pawelzny has quit IRC | 13:27 | |
| *** gtema_ has quit IRC | 13:39 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul master: Add job to build container images using pbrx https://review.openstack.org/580160 | 13:42 |
| corvus | logan-: i don't think it needs to be added to the config for depends-on to work, but it would need to be for required-projects (though i'd like to make that unecessary as well) | 14:19 |
| corvus | mordred, Shrews: quick question on 572829 | 14:33 |
| *** nchakrab has quit IRC | 14:33 | |
| corvus | also, gtema says that 566158 should pass tests when based on 572829 so i'm going to add a depends-on to that. hope that's ok. | 14:34 |
| *** nchakrab has joined #zuul | 14:34 | |
| openstackgerrit | James E. Blair proposed openstack-infra/nodepool master: Use openstacksdk instead of os-client-config https://review.openstack.org/566158 | 14:34 |
| *** nchakrab has quit IRC | 14:38 | |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/zuul master: Change "core developer" reference https://review.openstack.org/579241 | 14:45 |
| *** acozine1 has joined #zuul | 14:56 | |
| *** yolanda__ has joined #zuul | 15:00 | |
| Shrews | corvus: i think that has to do with shade now using openstacksdk for cloud config instead of occ. mordred can probably explain better | 15:00 |
| * Shrews has to afk for a bit | 15:00 | |
| corvus | i was especially wondering about dropping the **args at the end | 15:01 |
| Shrews | err, not shade, but openstacksdk has it's own version of config | 15:01 |
| *** yolanda_ has quit IRC | 15:03 | |
| *** acozine1 has quit IRC | 15:10 | |
| corvus | clarkb: i don't understand the "until converted to a string" part of your comment on line 119 of model.py on 579997 | 15:17 |
| clarkb | corvus: the hash functions explicitly converts the attributes to str, if the objects are not str prior to that they may hash to the same value afterwards but fail the __eq__ check | 15:18 |
| *** openstackgerrit has quit IRC | 15:19 | |
| corvus | clarkb: ah, i see, thanks. the answer to that is the same as hashing in general: things in python that use __hash__ (like set membership) also use __eq__ to account for hash collisions. | 15:19 |
| tobiash | corvus: I spent quite some time again grepping and scrolling through the logs I have and wrote some things up in an etherpad: https://etherpad.openstack.org/p/GQYVEEv2D3 | 15:20 |
| tobiash | the relevant logs I have are 3.5 minutes of scheduler logs in which the state must got broken | 15:21 |
| tobiash | that's 24000 lines of log :/ | 15:21 |
| tobiash | so I tried to filter some stuff there, if you want me to grep for additional data I can maybe add more data to that | 15:22 |
| tobiash | unfortunately I have no merger and executor logs of that timeframe | 15:22 |
| *** gtema has joined #zuul | 15:23 | |
| corvus | tobiash: cool, if you can save the log for a bit, that'd be great :) | 15:23 |
| tobiash | that log is on my hard disk and I won't delete it :) | 15:23 |
| *** nchakrab has joined #zuul | 15:28 | |
| *** hashar is now known as hasharAway | 15:29 | |
| *** nchakrab has quit IRC | 15:29 | |
| *** fbo is now known as fbo|off | 15:34 | |
| tobiash | So at least what I can tell from the logs is that there is no cat job and no reconfig in between | 15:37 |
| tobiash | So I think we can rule out a locking problem on executor/merger | 15:38 |
| *** openstackgerrit has joined #zuul | 15:41 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul master: Report config errors when removing cross-repo jobs https://review.openstack.org/579997 | 15:41 |
| corvus | clarkb: ^ with many more things in the hash soup :) | 15:41 |
| clarkb | corvus: cool, I'll rereview shortly | 15:43 |
| corvus | tobiash: can you double check that zuul-conf-global never shows up as an untrusted project (ie, it wasn't accidentally added as an untrusted project to one of the tenants)? | 15:43 |
| corvus | tobiash: did the zuul-conf-global PR Depends-On anything? did any other change Depends-On it? | 15:45 |
| corvus | tobiash: i've added those two questions and one more to the etherpad. the third question i just put there because i'm looking for some log lines. | 15:55 |
| *** acozine1 has joined #zuul | 16:10 | |
| *** nchakrab has joined #zuul | 16:11 | |
| *** nchakrab has quit IRC | 16:13 | |
| tobiash | corvus: updated the pad | 16:17 |
| tobiash | I've also added the relevant parts of the tenant config | 16:18 |
| tobiash | there is also shadowing regarding zuul-jobs involved if that might matter | 16:18 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul master: Report config errors when removing cross-repo jobs https://review.openstack.org/579997 | 16:20 |
| corvus | tobiash: yeah, shadowing is good to keep in mind | 16:21 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul master: Add job to build container images using pbrx https://review.openstack.org/580160 | 16:21 |
| corvus | tobiash: in the pad i pointed where i think there are still missing log lines | 16:24 |
| tobiash | corvus: checking | 16:24 |
| tobiash | corvus: created layout id was missing | 16:26 |
| tobiash | added to these lines | 16:26 |
| clarkb | corvus: heh I left a comment on ps2 of the error change and found there was a ps3 already addressing my comment | 16:27 |
| tobiash | corvus: what I also noticed is that all except one tenant exclude project on that repo but every tenant scheduled a merge and calculated a layout to just notice in check that nothing needs to be done | 16:28 |
| corvus | tobiash: i pasted in the 3 log lines, one of which we should see after loading phase 1 | 16:29 |
| tobiash | that is probably unrelated though | 16:29 |
| corvus | tobiash: yeah, good to keep in mind | 16:29 |
| tobiash | phase 2 is definitly missing in the log | 16:30 |
| corvus | yeah, i don't expect the phase 2 message in this case | 16:30 |
| corvus | but we should definitely have exactly one of the 3 that follows | 16:30 |
| tobiash | corvus: I cannot see any of these three | 16:31 |
| corvus | the phase 1 message is inside the exception handler for the last one, so we know we've entered that, so if there was an exception, it should be reported there | 16:31 |
| corvus | tobiash: interesting. what git sha are you on, and do you have any local patches? | 16:31 |
| corvus | i need to make sure i'm looking at exactly the right code now :) | 16:31 |
| corvus | tobiash: specifically, any local patches which touch manager/__init__.py | 16:32 |
| tobiash | uhm, have to check what state that was at that time | 16:33 |
| tobiash | just a sec | 16:33 |
| tobiash | corvus: I cannot tell precisely but it was based on the master from 7 days ago | 16:42 |
| tobiash | I'm currently using a moving branch and probably should tag all deployments... | 16:42 |
| tobiash | I'll try to prepare an equivalent branch | 16:42 |
| corvus | tobiash: oh, i see it, there is a path that ends with phase 1 | 16:45 |
| tobiash | corvus: is that good or bad? | 16:46 |
| *** yolanda_ has joined #zuul | 16:46 | |
| corvus | tobiash: it's good, it makes sense for this case | 16:46 |
| corvus | tobiash: this line: http://git.zuul-ci.org/cgit/zuul/tree/zuul/manager/__init__.py#n452 | 16:47 |
| tobiash | corvus: I tried to reproduce the branch as precise as possible https://github.com/tobiashenkel/zuul/tree/debug | 16:48 |
| corvus | tobiash: thanks | 16:48 |
| tobiash | the base must have been 02682856 or one before that | 16:49 |
| *** yolanda__ has quit IRC | 16:49 | |
| tobiash | and on top my current branches where I reverted one which was created after that | 16:49 |
| tobiash | but no changes to manager/__init__.py as far as I know | 16:49 |
| Shrews | tobiash: approved https://review.openstack.org/536930, but i think you left a couple of debugging lines in the test we should clean up (noted in comments) | 17:16 |
| tobiash | Shrews: oops | 17:17 |
| Shrews | tobiash: oh actually... i spotted something | 17:20 |
| Shrews | tobiash: left another comment in launchesCompleted. I'll leave it to you if you'd rather clean that up in a follow too, or just change that review | 17:22 |
| tobiash | looking | 17:23 |
| tobiash | Shrews: I think that was needed to get the test green, but I can double check | 17:24 |
| Shrews | tobiash: why did you change that for loop to use self.nodeset[:] from self.nodeset? | 17:25 |
| tobiash | to be safe that I can remove items during iteration | 17:25 |
| tobiash | otherwise I also can loop again after that loop to remove the aborted nodes from the nodeset | 17:25 |
| Shrews | oh, does that make a copy? | 17:26 |
| tobiash | yes | 17:26 |
| *** jpena is now known as jpena|off | 17:26 | |
| Shrews | ah, ok. | 17:26 |
| tobiash | I also can change that to self.nodeset().copy() if you like | 17:26 |
| tobiash | that's the same | 17:26 |
| Shrews | i think that's more clear, tbh | 17:27 |
| tobiash | no problem | 17:27 |
| Shrews | tobiash: don't you love how i come up with questions AFTER i click the approve button??? | 17:28 |
| Shrews | :) | 17:28 |
| tobiash | :) | 17:29 |
| corvus | time is meaningless | 17:29 |
| tobiash | Shrews: ok, just double checked, the zk.ABORTED addition is really needed | 17:29 |
| Shrews | corvus: not if you can bottle it | 17:29 |
| *** yolanda__ has joined #zuul | 17:29 | |
| Shrews | tobiash: hrm... trying to make the connection in my head | 17:30 |
| tobiash | is this a copy of the nodeset somewhere else? | 17:30 |
| Shrews | tobiash: oh, they aren't removed immediately, only during poll | 17:31 |
| Shrews | i see it now | 17:31 |
| tobiash | ah that was it | 17:31 |
| *** hasharAway has quit IRC | 17:32 | |
| Shrews | tobiash: ok, i'm comfortable approving again. the comments and copy() can be a followup if you want | 17:32 |
| *** yolanda_ has quit IRC | 17:32 | |
| tobiash | Shrews: as you like, it's already in my working copy (uncommitted yet) | 17:33 |
| Shrews | or i can do it if you are EOD | 17:33 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul master: Report config errors when removing cross-repo jobs https://review.openstack.org/579997 | 17:33 |
| Shrews | tobiash: approved | 17:33 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/nodepool master: Cleanup test_over_quota https://review.openstack.org/580728 | 17:35 |
| tobiash | and the followup ^ | 17:35 |
| Shrews | thx | 17:35 |
| tobiash | well I'm officially eod since more than thre hours ;) | 17:36 |
| Shrews | oh, sorry. feel free to ignore us after EOD :) | 17:36 |
| gtema | yeah, EOD is too early in Germany ;-) | 17:36 |
| tobiash | I enjoy working with you guys :) | 17:37 |
| gtema | :-) | 17:37 |
| openstackgerrit | Merged openstack-infra/nodepool master: Fix relaunch attempts when hitting quota errors https://review.openstack.org/536930 | 17:41 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul-jobs master: Install config pointing docker at the dockerhub mirror https://review.openstack.org/580730 | 17:48 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul master: Add job to build container images using pbrx https://review.openstack.org/580160 | 17:50 |
| *** gtema has quit IRC | 17:57 | |
| *** yolanda_ has joined #zuul | 18:07 | |
| *** yolanda__ has quit IRC | 18:10 | |
| *** acozine1 has quit IRC | 18:18 | |
| *** acozine1 has joined #zuul | 18:20 | |
| *** robled has quit IRC | 18:23 | |
| *** robled has joined #zuul | 18:24 | |
| *** robled has quit IRC | 18:24 | |
| *** robled has joined #zuul | 18:24 | |
| tobiash | corvus: we are getting increasing problems with reconfigurations blocking the whole system and we are thinking about ways how we can improve that | 18:38 |
| tobiash | I think our main problem with that is that the scheduler has a shared event queue for all tenants that is blocked by any reconfiguration event regardless if that's relevant for the tenant or not | 18:40 |
| tobiash | corvus: what do you think about the possibilities to migrate to an event queue per tenant? | 18:41 |
| *** acozine1 has quit IRC | 18:42 | |
| tobiash | I think that could help in further tenant isolation, less blocking of tenants and it could be a first step in direction of a scale out scheduler | 18:42 |
| *** acozine1 has joined #zuul | 18:47 | |
| corvus | tobiash: i think there are things that can be done to make the existing reconfiguration less painful. what issue are you seeing specifically? | 19:06 |
| corvus | tobiash: ie, what aspect is slow? | 19:06 |
| tobiash | the full reconfiguration takes around 5 minutes | 19:09 |
| corvus | tobiash: when you add tenants or projects? | 19:09 |
| tobiash | especially in a multi tenant setup the full reconfig is inefficient because tenants are reconfigured sequentially and thus we cannot use the mergers efficiently | 19:10 |
| tobiash | adding projects | 19:10 |
| tobiash | which we do a lot currently | 19:10 |
| tobiash | also a tenant reconfiguration within several tenants is also taking almost a minute blocking all other tenants | 19:11 |
| corvus | tobiash: tenant reconfiguration should be fast, it uses mostly cached data | 19:11 |
| corvus | tobiash: re-freezing the jobs is maybe what's taking a long time there? | 19:11 |
| corvus | (re-enqueuing changes) | 19:12 |
| tobiash | unfortunately the tenant reconfiguration of our most active tenant is taking around 40s currently | 19:13 |
| corvus | tobiash: it might be useful to have a breakdown of that 40s | 19:14 |
| tobiash | we're also working on a change that skips tenant reconfiguration if only branches are deleted and no branch contained cached config | 19:15 |
| tobiash | much of that is asking github for branches | 19:15 |
| corvus | tobiash: but back to full-reconfiguration: we can probably add a new kind of reconfiguration which uses cached config, and use that for changes to main.yaml | 19:16 |
| corvus | tobiash: so that nothing automatically does a full reconfiguration. we can relegate full reconfiguration to just a manual operation that an operator could perform in case something has gone wrong. | 19:16 |
| corvus | tobiash: i think the tenant reconfiguration, and the proposed semi-full reconfiguration could both use cached branches (except when a project has changed). that should reduce a lot of time | 19:17 |
| tobiash | corvus: you mean comparing the changes to main.yaml and just delete caches of removed projects, then trigger a tenant configuration for every tenant? | 19:17 |
| corvus | tobiash: something like that, yes. | 19:17 |
| tobiash | corvus: a break down of the 40s is about: get branches: 7s, reenqueueing changes: 30s | 19:24 |
| corvus | tobiash: we might also be able to optimize adding/removing projects (especially if they have no configuration) -- in those cases, we might be able to short-circuit reconfiguration altogether. | 19:27 |
| corvus | tobiash: other than that, i don't think there's much we can do about the reenqueing until we parellelize queue processors (zuulv4) | 19:28 |
| tobiash | corvus: in that example 7 changes were re-enqueued | 19:30 |
| tobiash | that took around 5s per change | 19:30 |
| corvus | tobiash: we might be able to make job freezing more efficient | 19:31 |
| corvus | i have to run now | 19:32 |
| *** yolanda__ has joined #zuul | 19:56 | |
| *** yolanda_ has quit IRC | 19:59 | |
| *** yolanda_ has joined #zuul | 20:22 | |
| corvus | tobiash: do you have any log entries related to the branch creation event? | 20:25 |
| *** yolanda__ has quit IRC | 20:26 | |
| *** yolanda__ has joined #zuul | 20:31 | |
| *** yolanda_ has quit IRC | 20:34 | |
| tobiash | corvus: added the branch push event | 20:39 |
| corvus | tobiash: it didn't do any reconfiguration due to the branch creation? (it shouldn't -- i just wanted to make sure the logs confirm that) | 20:39 |
| corvus | (ie, there's no "Submitting tenant reconfiguration event" after that) | 20:39 |
| tobiash | corvus: no, there was no reconfiguration in that time frame | 20:40 |
| corvus | tobiash: anything about shadow jobs after the layout was created? (see line 59 in etherpad) | 20:41 |
| tobiash | corvus: before 576538 it would have done a tenant reconfiguration but that was already deployed at that time | 20:41 |
| corvus | tobiash: yeah, that agrees with the code on your github debug branch | 20:42 |
| tobiash | a grep for shadow returns nothing | 20:48 |
| corvus | tobiash: thanks -- you don't have any shadow jobs? | 20:48 |
| tobiash | corvus: I'm not sure, the only shadow config is with zuul-jobs and I think the base job there was removed | 20:49 |
| corvus | yeah, moved into zuul-base-jobs | 20:50 |
| tobiash | corvus: I think the base job was the only reason in the past to shadow zuul-jobs so that's probably not needed anymore | 20:52 |
| corvus | tobiash: was there reconfiguration of any kind between the branch creation and the PR? | 20:52 |
| tobiash | corvus: no | 20:52 |
| tobiash | corvus: the first reconfigurations happened after the first occurrence of the wrong config | 20:53 |
| tobiash | and a full reconfig an hour later fixed it | 20:53 |
| corvus | tobiash: your clone-timeout merge pulled in some more recent fixes | 21:00 |
| corvus | https://github.com/tobiashenkel/zuul/commit/fbbab57ff2ae11e104e32a36aeaeecae1b9600fd | 21:00 |
| tobiash | corvus: oh, I overlooked that, the clone-timeout was added today | 21:02 |
| corvus | tobiash: did you say you were working on a test? | 21:04 |
| tobiash | corvus: I tried that but had no luck and zuul behaved as expected | 21:05 |
| corvus | tobiash: is it something you can push up to gerrit? | 21:06 |
| tobiash | corvus: I have to check but I think I've thrown it frustrated away ;) | 21:06 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul master: Report config errors when removing cross-repo jobs https://review.openstack.org/579997 | 21:08 |
| corvus | tobiash: 'git reflog' ? :) | 21:08 |
| tobiash | corvus: found it, just a sec | 21:08 |
| tobiash | corvus: I wasn't sure that I even did a commit | 21:08 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: WIP: unsuccessful attempt to reproduce zuul erratic behavior https://review.openstack.org/580765 | 21:09 |
| tobiash | corvus: ^ really dirty | 21:09 |
| corvus | tobiash: do you think we can do that with fake github instead of fake gerrit? one of the things that strikes me as interesting about this is that a branch is created on a config project | 21:10 |
| tobiash | corvus: should be possible | 21:11 |
| tobiash | corvus: fake gerrit was just the quickest copy&paste start | 21:11 |
| tobiash | corvus: the test case also misses assertions as I just added a break point on the pass and inspected the messages on B | 21:12 |
| corvus | i usually just run in the foreground and look at stdout | 21:12 |
| tobiash | I like the debugger :) | 21:13 |
| corvus | that can be useful -- but it's good to look at the logs emitted by new tests to make sure they're what we need in prod | 21:14 |
| tobiash | that's true | 21:14 |
| corvus | anyway -- whatever works best for debugging of course -- i just want to advocate occasionally looking at test logs for that purpose. :) | 21:14 |
| corvus | tobiash: do you want to take a stab at doing a fake github version of that test? | 21:15 |
| corvus | (also, fwiw, i threw a change at openstack's zuul and did not see any issues: https://review.openstack.org/580759 ) | 21:16 |
| tobiash | corvus: I'm going to sleep in a few minutes so probably on monday | 21:16 |
| corvus | tobiash: cool. i think the most interesting parts of this are multi-tenant, github, and creating the branch on the config project | 21:17 |
| corvus | i don't see anything jumping out at me as obviously the problem, but those things seem like they could have plausible connections | 21:17 |
| tobiash | yes, so I'll try to put all three things into the test case | 21:18 |
| corvus | tobiash: i'm assuming this has only happened once -- have you seen this *not* happen with the same circumstances? | 21:18 |
| tobiash | corvus: it was kind of a standard behavior but I've just thrown a similar change into that repo | 21:22 |
| tobiash | gah | 21:22 |
| tobiash | broken | 21:22 |
| corvus | tobiash: yay it's reproducible :) | 21:23 |
| corvus | tobiash: with any luck, everyone else is asleep so the logs will be much smaller now, with fewer things going on? | 21:23 |
| tobiash | corvus: I assume so | 21:25 |
| tobiash | corvus: so I'll save the logs and repair prod and then go to sleep ;) | 21:25 |
| corvus | tobiash: sounds good -- we'll have more data and more ideas to try next week! thanks! | 21:26 |
| tobiash | ok, now I have logs of scheduler, all 8 mergers and all 8 executors | 21:32 |
| tobiash | and reconfig triggered | 21:33 |
| tobiash | corvus: ok, just did a last test for today. A recheck of that PR also broke it so the branch creation is probably unrelated. | 21:42 |
| corvus | tobiash: oh good idea | 21:42 |
| corvus | tobiash: we should still keep in mind that it exists on a branch of the repo though | 21:42 |
| tobiash | yah | 21:43 |
| tobiash | but now time for bed | 21:43 |
| tobiash | n8 | 21:43 |
| *** yolanda_ has joined #zuul | 21:53 | |
| *** yolanda__ has quit IRC | 21:56 | |
| *** acozine1 has quit IRC | 22:10 | |
| *** yolanda__ has joined #zuul | 23:04 | |
| *** yolanda_ has quit IRC | 23:06 | |
| *** yolanda has joined #zuul | 23:07 | |
| *** yolanda__ has quit IRC | 23:09 | |
| *** rlandy has quit IRC | 23:16 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul-jobs master: WIP: Add a role to return file comments https://review.openstack.org/579033 | 23:25 |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul-jobs master: configure-mirrors: add set_epel_repository option https://review.openstack.org/580781 | 23:39 |
| *** yolanda_ has joined #zuul | 23:43 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul-jobs master: configure-mirrors: skip system mirrors if zuul can't sudo https://review.openstack.org/580782 | 23:45 |
| *** yolanda has quit IRC | 23:46 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!