| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: configloader: skip merger:cat when no items are included https://review.openstack.org/576016 | 01:42 |
|---|---|---|
| *** ianychoi_ has joined #zuul | 02:58 | |
| *** ianychoi_ has quit IRC | 03:01 | |
| *** ianychoi has quit IRC | 03:02 | |
| *** ianychoi_ has joined #zuul | 03:02 | |
| *** rlandy has joined #zuul | 03:08 | |
| *** bhavik1 has joined #zuul | 03:12 | |
| *** rlandy has quit IRC | 03:21 | |
| *** bhavik1 has quit IRC | 03:43 | |
| tobiash | tristanC: added a question on ^ | 05:15 |
| *** jesusaur has quit IRC | 05:17 | |
| *** swest has quit IRC | 05:24 | |
| *** swest has joined #zuul | 05:25 | |
| tristanC | tobiash: thanks, replied with more informations | 05:26 |
| tristanC | tobiash: also, regarding configloader scale issue, I'd like to propose a mechanism to handle project-created event, something that would auto-add new projects to avoid having to reload the scheduler for every project addition | 05:27 |
| tobiash | tristanC: in that case you probably want regex matching of projects in the tenant config? | 05:28 |
| tristanC | tobiash: yes, something along those line would help | 05:28 |
| tristanC | not sure if github sends such event though | 05:28 |
| tristanC | maybe we could associate the event to a custom config-reload pipeline, like that we could use "zuul enqueue" to manually inject new projects in zuul | 05:30 |
| tobiash | tristanC: I understand what you're trying to achieve in 576016 but still don't understand that from a data structure point of view as that what you're checking for None should already evaluate to an empty list above | 05:30 |
| tobiash | tristanC: github sends such an event: https://developer.github.com/v3/activity/events/types/#projectevent | 05:30 |
| tristanC | tobiash: a project defined like "- org/project1: { include: [] }" is having all objects in its load_classes | 05:32 |
| tristanC | we need a special case where user set include to empty list (!= None) to skip load_classes | 05:32 |
| tristanC | the groups4.yaml test show the issue, without this extra "is None" check, unwanted merger:cat job are used | 05:34 |
| tobiash | tristanC: but then only the None check should be sufficient as in the case 'not project_include' is always true | 05:35 |
| tobiash | that if statement is pretty confusing | 05:35 |
| tristanC | i agree :-) | 05:36 |
| tristanC | we just changed our zuul configuration generator, and it's now using per project settings instead of projects groups. Took some time to figure-out why reload took *much* longer... | 05:37 |
| tristanC | (most our projects are -distgit with include:[], no .zuul.yaml support) | 05:39 |
| tobiash | tristanC: it might make sense to change that to if ... is None project_include = current_include else project_include = frozenset( ... | 05:40 |
| tristanC | tobiash: good idea, let's give it a try | 05:41 |
| tobiash | and add a short comment to that explains why we're checking to none instead of empty set here | 05:41 |
| tobiash | I think that could make it cleare | 05:42 |
| tobiash | clearer | 05:42 |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: configloader: skip merger:cat when no items are included https://review.openstack.org/576016 | 05:47 |
| *** Rohaan has joined #zuul | 06:08 | |
| Rohaan | tristanC: Is there any specific order in which you need to start Zookeeper, nodepool and zuul services? I've been getting /zuul/api/tenant/local/status: Service Unavailable since this morning :( . I'm trying to restart zuul services after oc cluster up like this: https://pastebin.com/dz9sS0Y7 | 06:26 |
| tristanC | Rohaan: zookeeper needs to be restarted first | 06:27 |
| tristanC | Rohaan: and since you are re-creating the openshift cluster, i think you can also rm -Rf /var/lib/zookeeper/version-2/* | 06:28 |
| *** gtema has joined #zuul | 06:31 | |
| *** pcaruana has joined #zuul | 06:35 | |
| *** dmellado has joined #zuul | 06:41 | |
| Rohaan | tristanC: I tried these steps but still I get this error :( Is there something I could be missing? I get EndOfStreamException in zookeeper status logs https://pastebin.com/7Hid2GR5. | 06:43 |
| *** yolanda__ is now known as yolanda | 06:43 | |
| tristanC | Rohaan: seems like nodepool (12:04:06 IST; 7min ago) needs to be restarted after zookeper restart (12:07:21 IST; 3min 52s ago) | 06:45 |
| tristanC | Rohaan: zookeeper is like the state machine of zuul and nodepool, it needs to be running early on | 06:46 |
| *** hashar has joined #zuul | 06:51 | |
| Rohaan | tristanC: Does any of these services need to be restarted at all after oc cluster up?? | 06:55 |
| tristanC | Rohaan: no, nodepool should self-heal when openshift provider is restarted | 06:56 |
| Rohaan | Hm | 06:56 |
| tristanC | Rohaan: but iirc, there are issue when your server restart, nodepool and zuul really needs access to a running zookeeper service | 06:56 |
| Rohaan | tristanC: Oh I see. | 06:59 |
| Rohaan | let me wait for some time and see if nodepool makes a connection to zookeeper | 07:00 |
| *** jesusaur has joined #zuul | 07:03 | |
| tristanC | tobiash: before and after the "include: []" (restart at 06:55) fix: https://softwarefactory-project.io/grafana/d/000000001/zuul-status?panelId=44&fullscreen&orgId=1&from=1529304005142&to=1529305862695 :) | 07:11 |
| tobiash | :) | 07:13 |
| *** jpena|off is now known as jpena | 07:43 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/nodepool master: zk: use kazoo retry facilities https://review.openstack.org/535537 | 07:51 |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/nodepool master: zk: retry initial zookeeper connection attempts https://review.openstack.org/576047 | 07:51 |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: zk: use kazoo retry facilities https://review.openstack.org/536209 | 07:55 |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: zk: retry initial zookeeper connection attempts https://review.openstack.org/576048 | 07:55 |
| *** sshnaidm has quit IRC | 08:09 | |
| *** sshnaidm has joined #zuul | 09:34 | |
| *** Rohaan___ has joined #zuul | 10:23 | |
| *** Rohaan has quit IRC | 10:23 | |
| *** Rohaan___ is now known as Rohaan | 11:05 | |
| *** jpena is now known as jpena|lunch | 11:08 | |
| *** Rohaan has quit IRC | 11:14 | |
| *** jpena|lunch is now known as jpena | 12:03 | |
| *** rlandy has joined #zuul | 12:17 | |
| Shrews | A wrist injury from a few years ago is giving me major problems. Headed to the dr today so won't be around much. Very hard to type atm | 12:37 |
| tobiash | Shrews: oh, hope it'll get better soon | 12:49 |
| tobiash | corvus, fungi: during working on that ansible logging problem I totally forgot to send the security notice about the no_log issue | 13:04 |
| tobiash | corvus, fungi: draft here: https://etherpad.openstack.org/p/rK1f4cRdHq | 13:05 |
| tobiash | dmsimard: ^ | 13:06 |
| fungi | tobiash: thanks, i noticed we also didn't include a release note about it (at least not that i could find). i might as well go ahead and get a cve issued for it as well | 13:06 |
| dmsimard | tobiash: the scope is a bit larger, sec | 13:07 |
| dmsimard | tobiash: there is a CVE | 13:07 |
| dmsimard | sec | 13:07 |
| fungi | i thought the cve was for a defect in ansible, and our devect was similar but not the same? or was the patch to zuul basically identical to the patch for ansible? | 13:08 |
| tobiash | fungi: it basically was as we did the same mistake in the callback as upstream ansible | 13:08 |
| tobiash | s/was as/was the same as/ | 13:09 |
| dmsimard | tobiash: which is still vulnerable despite ansible being fixed ? | 13:10 |
| fungi | i guess what i'm asking is whether it's identical enough to the defect in ansible that mitre would prefer reusing that cve and updating it in their database vs issuing a new cve for zuul | 13:10 |
| tobiash | dmsimard: I assume so | 13:11 |
| tobiash | actually we had two bugs in there | 13:11 |
| dmsimard | my understanding is that the defect was in ansible and it was "cascading" down to Zuul (and ARA) as a result but tobiash came up with a zuul fix that I haven't looked at in depth | 13:11 |
| dmsimard | tobiash: ok so I think it's fair to talk about the two seperate ones and how they might be related | 13:12 |
| dmsimard | i.e, in any case operators will want to update to the latest 2.4/2.5 dot releases once they are shipped | 13:12 |
| dmsimard | 2.5.5 is already out with the fix since the 14th | 13:13 |
| dmsimard | 2.4.5 is in rc1 | 13:13 |
| tobiash | dmsimard: I'm not sure but we at least had the a check in our callback that was broken in a very similar way than upstream ansible | 13:14 |
| dmsimard | ++ | 13:14 |
| tobiash | the second bug was that we didn't handle v2_runner_on_unreachable in the zuul_stream callback | 13:15 |
| tobiash | fungi: should we then get a cve for zuul (given that we didn't do that on the last issues)? | 13:16 |
| tristanC | tobiash: fwiw, anyone can request a cve using this form: https://cveform.mitre.org/ | 13:17 |
| *** elyezer has joined #zuul | 13:17 | |
| tristanC | it seems like a good idea considering zuul>=3.0<3.1 is affected | 13:19 |
| dmsimard | tobiash: added some notes on the etherpad | 13:25 |
| dmsimard | tobiash: brb | 13:25 |
| tobiash | dmsimard: cool, thx | 13:26 |
| *** gtema has quit IRC | 13:32 | |
| fungi | tobiash: what was the last issue? | 13:33 |
| tobiash | fungi: we basically had the same user visible error in both callbacks (zuul_stream, zuul_json) | 13:34 |
| fungi | if you're referring to the ones we announced before 3.0 was released, then those didn't need a cve. cve is for tracking vulnerabilities in released versions of software and at least as far as i was aware this was the first exploitable vulnerability we had discovered after 3.0.0 was released | 13:34 |
| tobiash | ah ok, I think the last one was before the 3.0 release | 13:34 |
| fungi | but entirely possible i missed the discussion around a previous one | 13:34 |
| corvus | "exploitable" is iffy for this one, but if ansible figured it was, no reason to argue. :) | 13:37 |
| tobiash | corvus: it even might be exploitable by shutting down the test node during the build | 13:38 |
| corvus | tobiash: fair. etherpad lgtm | 13:39 |
| fungi | yeah, i suppose in this case it's more the risk that you accidentally expose credentials in published logs people could search through | 13:39 |
| corvus | fungi: that's definitely bad, the thing i wasn't clear about was whether you could intentionally trigger it. | 13:40 |
| fungi | do we want a patch to add a 3.1.0 release note for this fix too? | 13:40 |
| corvus | fungi: seems reasonable | 13:41 |
| corvus | tobiash: if you're ready to send that message, i can approve it now, before i hop on the plane | 13:42 |
| tobiash | corvus: so send it without a cve? | 13:42 |
| fungi | tobiash: did you want me to fill out the cve request form? if so i can do that and then include the cve number in the release note patch | 13:42 |
| corvus | oh, you're waiting on the cve. nm | 13:42 |
| fungi | you can always announce without a cve assignment and then i can reply with the cve number. there's really no need to wait for cve assignments to disclose something | 13:43 |
| corvus | tobiash: any of the infra-root folks should be able to approve the moderation request in my absence | 13:43 |
| tobiash | fungi: that would be great, thx | 13:44 |
| tobiash | fungi, corvus: so I'll send the mail now and the cve will be part of the release notes patch? | 13:44 |
| tobiash | that's ok? | 13:44 |
| fungi | yep | 13:44 |
| corvus | wfm | 13:45 |
| fungi | because i haven't completed my vmt process fork for zuul yet, i'll be following https://security.openstack.org/vmt-process.html#send-cve-request but s/openstack/zuul/ for the "vendor" field | 13:45 |
| tobiash | corvus: mail sent and safe travels | 13:47 |
| corvus | tobiash: approved, thanks! | 13:49 |
| *** sshnaidm_ has joined #zuul | 13:54 | |
| *** sshnaidm has quit IRC | 13:57 | |
| fungi | i've gotten confirmation that my cve request was received. hopefully will have an assignment in a few hours, or else they'll tell us to reuse the same cve as ansible | 14:31 |
| fungi | either way, i'll push up a review to add the release note once i get something back from mitre | 14:31 |
| *** acozine1 has joined #zuul | 15:00 | |
| *** sshnaidm_ has quit IRC | 15:06 | |
| *** dtruong_ has joined #zuul | 15:07 | |
| *** dtruong has quit IRC | 15:12 | |
| *** sshnaidm_ has joined #zuul | 15:19 | |
| *** sshnaidm_ has quit IRC | 15:48 | |
| openstackgerrit | Fabien Boucher proposed openstack-infra/zuul master: Prevent Zuul scheduler to crash at startup if gerrit down https://review.openstack.org/576192 | 16:00 |
| *** gtema has joined #zuul | 16:08 | |
| *** sshnaidm_ has joined #zuul | 16:16 | |
| *** pcaruana has quit IRC | 16:20 | |
| *** sshnaidm_ is now known as sshnaidm|afk | 16:20 | |
| *** hashar is now known as hasharAway | 17:05 | |
| Shrews | tobiash: thx but looking like another surgery might be needed :( | 17:05 |
| *** dkranz has quit IRC | 17:06 | |
| *** dkranz has joined #zuul | 17:08 | |
| *** jpena is now known as jpena|off | 17:15 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul-jobs master: Just use chmod instead of file for log permissions https://review.openstack.org/576215 | 17:20 |
| *** myoung is now known as myoung|lunch | 18:08 | |
| *** pcaruana has joined #zuul | 18:25 | |
| *** pcaruana has quit IRC | 18:39 | |
| openstackgerrit | Merged openstack-infra/zuul-jobs master: Just use chmod instead of file for log permissions https://review.openstack.org/576215 | 18:45 |
| *** sshnaidm|afk has quit IRC | 18:49 | |
| *** gtema has quit IRC | 18:50 | |
| *** myoung|lunch is now known as myoung | 19:16 | |
| *** sshnaidm|afk has joined #zuul | 19:24 | |
| *** acozine1 has quit IRC | 20:47 | |
| openstackgerrit | Mohammed Naser proposed openstack-infra/zuul-jobs master: Add PATH to `ip` command execution https://review.openstack.org/576274 | 21:37 |
| *** EmilienM is now known as EmilienM_PTO | 21:45 | |
| *** hasharAway has quit IRC | 22:07 | |
| *** myoung is now known as myoung|off | 22:07 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!