| *** jamesmcarthur has quit IRC | 00:26 | |
| *** jamesmcarthur has joined #zuul | 01:35 | |
| *** jamesmcarthur has quit IRC | 02:12 | |
| *** jamesmcarthur has joined #zuul | 02:17 | |
| *** jamesmcarthur has quit IRC | 02:21 | |
| *** jamesmcarthur has joined #zuul | 02:36 | |
| *** jamesmcarthur has quit IRC | 03:02 | |
| *** raukadah is now known as chandankumar | 03:59 | |
| *** kmalloc has quit IRC | 05:02 | |
| *** saneax has joined #zuul | 05:23 | |
| *** evrardjp has quit IRC | 05:35 | |
| *** evrardjp has joined #zuul | 05:35 | |
| *** threestrands has joined #zuul | 06:04 | |
| *** threestrands has quit IRC | 06:16 | |
| *** felixedel has joined #zuul | 06:41 | |
| *** dpawlik has joined #zuul | 06:46 | |
| *** yolanda has joined #zuul | 06:47 | |
| *** dpawlik has quit IRC | 06:57 | |
| *** dpawlik has joined #zuul | 07:08 | |
| *** saneax has quit IRC | 07:17 | |
| *** nborg1 has joined #zuul | 07:50 | |
| *** nborg1 has left #zuul | 07:50 | |
| *** NBorg has joined #zuul | 07:50 | |
| NBorg | When I'm adding a gitlab project with the git driver, how should I refer to the project? <path>/<repo> or <path>/<repo>.git? | 08:09 |
|---|---|---|
| NBorg | (The url in gitlab says .git, so I suspect that, but something is wrong) | 08:10 |
| *** jcapitao has joined #zuul | 08:17 | |
| *** tosky has joined #zuul | 08:23 | |
| *** jpena|off is now known as jpena | 08:23 | |
| *** saneax has joined #zuul | 08:23 | |
| *** mgoddard has quit IRC | 08:27 | |
| *** mgoddard has joined #zuul | 08:33 | |
| fbo | NBorg: We have some gitlab projects connected to Zuul with the git driver and we use the <path>/<repo>.git | 08:38 |
| fbo | NBorg: also if you run a recent version of Zuul, a Gitlab driver exists but still experimental. | 08:42 |
| openstackgerrit | Tobias Henkel proposed zuul/zuul-jobs master: Optimize repos for size when preparing for nodecache https://review.opendev.org/710720 | 09:03 |
| *** jcapitao has quit IRC | 09:03 | |
| *** jcapitao has joined #zuul | 09:04 | |
| *** hashar has joined #zuul | 09:13 | |
| *** dmellado has quit IRC | 09:23 | |
| *** dmellado has joined #zuul | 09:26 | |
| *** dmellado has quit IRC | 09:33 | |
| *** dmellado has joined #zuul | 09:35 | |
| *** hashar has quit IRC | 09:52 | |
| *** sshnaidm is now known as sshnaidm|pto | 09:59 | |
| *** jfoufas1 has joined #zuul | 10:09 | |
| *** Defolos has joined #zuul | 10:23 | |
| *** sshnaidm|pto has quit IRC | 10:26 | |
| openstackgerrit | Felix Edel proposed zuul/zuul master: Dequeue changes via github checks API https://review.opendev.org/709135 | 10:32 |
| *** jfoufas1 has quit IRC | 10:34 | |
| *** sshnaidm has joined #zuul | 10:42 | |
| *** felixedel has quit IRC | 10:49 | |
| *** sshnaidm_ has joined #zuul | 11:12 | |
| *** sshnaidm has quit IRC | 11:12 | |
| *** sshnaidm_ is now known as sshnaidm|pto | 11:15 | |
| *** sshnaidm|pto has quit IRC | 11:23 | |
| *** sshnaidm has joined #zuul | 11:23 | |
| *** sshnaidm is now known as sshnaidm|pto | 11:24 | |
| *** felixedel has joined #zuul | 11:27 | |
| *** jcapitao is now known as jcapitao|lunch | 11:46 | |
| tobiash | zuul-maint: all py35 jobs (zuul and zuul-jobs at least) started to fail during 'run tox without tests' with an import error | 11:48 |
| tobiash | https://zuul.opendev.org/t/zuul/build/0cff305fed704da0a34b108d8812155b | 12:00 |
| tobiash | seems like '/usr/bin/python3 -m virtualenv --no-download --python /usr/bin/python3 py35' fails, but I cannot reproduce it on a bionic with py35 | 12:03 |
| tobiash | also cannot reproduce in xenial container | 12:05 |
| *** sugaar has quit IRC | 12:07 | |
| *** yolanda has quit IRC | 12:10 | |
| *** yolanda has joined #zuul | 12:11 | |
| *** harrymichal has joined #zuul | 12:12 | |
| openstackgerrit | Tobias Henkel proposed zuul/zuul master: DNM: Cap alembic https://review.opendev.org/710757 | 12:28 |
| *** harrymichal has quit IRC | 12:28 | |
| *** jfoufas1 has joined #zuul | 12:35 | |
| *** jpena is now known as jpena|lunch | 12:36 | |
| tobiash | that was a shot in the dark but didn't work ^ | 12:41 |
| *** rlandy has joined #zuul | 12:57 | |
| NBorg | fb: Thank you. I've found a problem elsewhere. | 12:58 |
| swest | tobiash: it's an updated dependency of tox: https://pypi.org/project/importlib-resources/ | 13:10 |
| swest | https://gitlab.com/python-devs/importlib_resources/-/commit/344fb6a0780053834d29ed95b17f32f0c7522d1a | 13:10 |
| tobiash | upstream issue: https://gitlab.com/python-devs/importlib_resources/issues/83 | 13:12 |
| *** jcapitao|lunch is now known as jcapitao | 13:15 | |
| *** jpena|lunch is now known as jpena | 13:32 | |
| *** toabctl has joined #zuul | 13:41 | |
| *** tosky_ has joined #zuul | 13:47 | |
| *** tosky has quit IRC | 13:47 | |
| *** tosky_ is now known as tosky | 13:47 | |
| *** jfoufas1 has quit IRC | 14:04 | |
| *** Goneri has joined #zuul | 14:16 | |
| *** felixedel has quit IRC | 14:25 | |
| *** felixedel has joined #zuul | 14:38 | |
| *** erbarr has joined #zuul | 14:39 | |
| *** zbr is now known as zbr|pto | 14:43 | |
| corvus | tobiash, swest: do you think pinning importlib-resources would work? or is the virtualenv fix from frickler likely to help? | 15:06 |
| tobiash | corvus: it must be fixed before running tox | 15:06 |
| corvus | ok, so structurally like the virtualenv fix | 15:07 |
| tobiash | yes | 15:07 |
| corvus | should i push up frickler's patch to zuul then to see if it fixes this? | 15:08 |
| tobiash | maybe something like $PIP install --user importlib-resources==1.0.2 | 15:08 |
| corvus | ok, so you think importlib-resources and virtualenv are two different problems? | 15:08 |
| corvus | (it's not clear to me whether pinning virtualenv will also take care of the importlibe problem) | 15:09 |
| tobiash | corvus: there is a py27 problem (I think frickler's patch addresses this) and a py35 problem | 15:09 |
| tobiash | those are two different problems (that might be solveable with the same approach | 15:09 |
| tobiash | the py35 problem is caused by a release of importlib-resources yesterday | 15:10 |
| tobiash | corvus: virtualenv seems to just require importlib-resources>=1.0: https://github.com/pypa/virtualenv/blob/master/setup.cfg#L48 | 15:11 |
| pabelanger | tobiash: https://github.com/ansible/project-config/blob/master/zuul.d/pipelines.yaml#L56 is the triggers we have for gate, both label and status however, I don't see 'check' api info now. Especially if we remove setting the status in reporter | 15:11 |
| tobiash | so I think importlib needs to be gapped | 15:11 |
| corvus | tobiash: so the importlib fix alone won't be sufficient | 15:12 |
| corvus | er | 15:12 |
| tobiash | corvus: I think we need both | 15:12 |
| corvus | tobiash: so the virtualenv fix alone won't be sufficient :) | 15:12 |
| tobiash | importlib will fix py35 and virtualenv will fix 27 | 15:12 |
| corvus | tobiash: right, but for zuul specifically, we only need the importlib fix | 15:12 |
| tobiash | yes | 15:13 |
| corvus | if mordred were around, i think he would suggest another option: drop 3.5 support | 15:13 |
| corvus | (though that's still what opendev and probably others are running, so there's more to consider there) | 15:14 |
| tobiash | ok, I'll take his part: should we drop 3.5 support? ;) | 15:16 |
| mordred | let's drop 3.5 support! | 15:19 |
| corvus | well, it's still the default under xenial... and i'm not sure what else... maybe if things get worse? but maybe if they fix this in a day or so it's not a big deal? | 15:19 |
| openstackgerrit | James E. Blair proposed zuul/zuul master: DNM: Test fixing py35 by pinning importlib-resources https://review.opendev.org/710794 | 15:19 |
| corvus | tobiash: ^ something like that? | 15:20 |
| corvus | the other thing we could do is just make 35 non-voting and see if this gets fixed upstream in a day or two.... | 15:20 |
| tobiash | corvus: I think that should work | 15:20 |
| *** jamesmcarthur has joined #zuul | 15:21 | |
| *** jamesmcarthur has quit IRC | 15:27 | |
| *** jamesmcarthur has joined #zuul | 15:28 | |
| *** goneri_ has joined #zuul | 15:28 | |
| *** goneri_ has quit IRC | 15:28 | |
| tobiash | corvus: still fails | 15:31 |
| tobiash | for those not watching the live log: http://paste.openstack.org/show/790200/ | 15:32 |
| corvus | tobiash: did that run in python 2.7? | 15:33 |
| tobiash | weird | 15:33 |
| tobiash | maybe force usage of pip3 | 15:33 |
| corvus | yeah, we probably have both pip and pip3 on those nodes? | 15:33 |
| mordred | yeah - we do | 15:34 |
| tobiash | I don't know which python version tox is using, but I guess 35 since this didn't fix it | 15:34 |
| mordred | and yeah, I think we should just use pip3 in that fix playbook | 15:34 |
| openstackgerrit | James E. Blair proposed zuul/zuul master: DNM: Test fixing py35 by pinning importlib-resources https://review.opendev.org/710794 | 15:34 |
| corvus | done | 15:34 |
| *** chandankumar is now known as raukadah | 15:36 | |
| *** sshnaidm|pto has quit IRC | 15:41 | |
| *** avass has joined #zuul | 15:46 | |
| *** jamesmcarthur has quit IRC | 15:46 | |
| *** avass has quit IRC | 15:50 | |
| *** sshnaidm|pto has joined #zuul | 15:55 | |
| *** avass has joined #zuul | 15:56 | |
| *** dmellado has quit IRC | 15:57 | |
| avass | Does anyone know if there's any change related to the gerrit trigger enabling filtering on groups like you can do with usernames? https://zuul-ci.org/docs/zuul/reference/drivers/gerrit.html#attr-pipeline.trigger.%3Cgerrit%20source%3E.username | 15:57 |
| avass | We might need it, so if there isn't we're going to have to implement that :) | 15:58 |
| *** dmellado has joined #zuul | 16:00 | |
| corvus | avass: that doesn't sound familiar | 16:00 |
| *** jamesmcarthur has joined #zuul | 16:04 | |
| *** jamesmcarthur has quit IRC | 16:04 | |
| *** jamesmcarthur has joined #zuul | 16:05 | |
| *** felixedel has quit IRC | 16:07 | |
| *** armstrongs has joined #zuul | 16:07 | |
| *** jcapitao is now known as jcapitao_afk | 16:12 | |
| *** mattw4 has joined #zuul | 16:12 | |
| *** dmellado has quit IRC | 16:18 | |
| avass | corvus: alright | 16:18 |
| *** avass has quit IRC | 16:18 | |
| *** dmellado has joined #zuul | 16:20 | |
| corvus | tobiash, mordred, frickler: still no joy on https://review.opendev.org/710794 | 16:30 |
| mordred | corvus: importlib-resources is at 1.1.0 inside of the virtualenv | 16:32 |
| mordred | corvus: do we also need to pin it somehow inside of the tox venv? | 16:32 |
| corvus | i guess i need to go read clarkb's spec to try to get my bearings | 16:33 |
| mordred | corvus: https://zuul.opendev.org/t/zuul/build/afbd2b6548d543398e872582c5fd68e4/log/job-output.txt#1663 | 16:34 |
| clarkb | mordred: corvus tox runs from a global install doesnt' it? | 16:34 |
| clarkb | its possible dropping the --user and adding some sudo will fix things? | 16:34 |
| mordred | clarkb: tox does - but we're getting an error from inside of the virtualenv | 16:34 |
| clarkb | ah | 16:34 |
| corvus | but i thought this was related to tox | 16:35 |
| clarkb | I'm still trying to catch up on the issue. The upstream bug implies it is something that virtuaelnv itself does that imports a lib that does not exist in python3.5 | 16:36 |
| corvus | clarkb: if i were to modify the global tox env, should i use pip or pip3? | 16:36 |
| clarkb | corvus: I believe pip3, but I'm not 100% sure of that | 16:36 |
| openstackgerrit | James E. Blair proposed zuul/zuul master: DNM: Test fixing py35 by pinning importlib-resources https://review.opendev.org/710794 | 16:38 |
| corvus | clarkb, mordred: ^? | 16:38 |
| mordred | I think we're seeing a different prolbme | 16:39 |
| *** igordc has joined #zuul | 16:39 | |
| mordred | zuul installs virtualenv as part of its requirements | 16:39 |
| mordred | as part of doing that, it's pulling in importlib-resources 1.1.0 into the virtualenv, which is then breaking cherrypy | 16:39 |
| clarkb | mordred: ok I think the pin corvus is doing should avoid that too because we don't pip install -U do we? | 16:40 |
| mordred | I think all we need to do is add importlib-resources==1.0.2 before virtualenv in the requirements.txt file | 16:40 |
| clarkb | ya that should work too | 16:40 |
| mordred | corvus' fix is a fix to the tox install | 16:40 |
| clarkb | oh right global vs venv context (may need both?) | 16:40 |
| mordred | it does not impact the content _inside_ of the tox managed virtualenv - which is where the failure in that patch happened | 16:41 |
| mordred | yeah. I don't know that we need the global - but it also shouldn't hurt. | 16:41 |
| corvus | i assumed it failed there due to some weird copying thing | 16:41 |
| corvus | i was basing the global fix on the bug report | 16:41 |
| clarkb | corvus: ya I think we may need both things | 16:41 |
| corvus | ok, i will push up 2 more changes | 16:41 |
| clarkb | however the global fix wouldn't be necessary until our images updated. | 16:41 |
| corvus | then we'll have all 3 | 16:41 |
| clarkb | so its possible we started failing in the mordred identified case first, then later we'll start failing early on the global issue | 16:42 |
| mordred | clarkb: yah | 16:42 |
| corvus | clarkb: you don't think we're running new enough to hit the global issue? | 16:42 |
| openstackgerrit | James E. Blair proposed zuul/zuul master: DNM: Test fixing py35 by pinning importlib-resources in reqs https://review.opendev.org/710826 | 16:44 |
| corvus | mordred: that's your idea i think? ^ | 16:44 |
| mordred | corvus: yes - except it needs to go on the line before virtualenv | 16:45 |
| corvus | haha | 16:45 |
| mordred | because there is no depsolver | 16:45 |
| openstackgerrit | James E. Blair proposed zuul/zuul master: DNM: Test fixing py35 by pinning importlib-resources in reqs https://review.opendev.org/710826 | 16:46 |
| openstackgerrit | James E. Blair proposed zuul/zuul master: DNM: Test fixing py35 by pinning importlib-resources both places https://review.opendev.org/710828 | 16:46 |
| corvus | and that's both changes combined | 16:46 |
| *** dmellado has quit IRC | 16:46 | |
| mordred | corvus: same thing with position of importlib-resources in the combined change | 16:47 |
| *** jcapitao_afk is now known as jcapitao | 16:48 | |
| clarkb | corvus: or we werent whenever the failures occurred that mordred was looking at | 16:48 |
| tobiash | that's getting tricky | 16:48 |
| corvus | mordred: doh, cherry-picked wrong version | 16:48 |
| clarkb | also another option here may be to usevenv not virtualenv. Itspython3 only but zuul is python3 only so thats fine | 16:49 |
| *** dmellado has joined #zuul | 16:49 | |
| openstackgerrit | James E. Blair proposed zuul/zuul master: DNM: Test fixing py35 by pinning importlib-resources both places https://review.opendev.org/710828 | 16:49 |
| tobiash | corvus, clarkb, mordred: a different solution would be to upgrade python to the latest 3.5 release (which would solve this issue as well) | 16:50 |
| corvus | tobiash: yes, but we may be running into a case where using the old broken python supplied by the os vendor is actually good | 16:50 |
| corvus | like, if one of our goals is to make "pip3 install zuul" work on ubuntu xenial, then this is a correct failure we should see if we can fix | 16:51 |
| corvus | or, at least, it may be? i'm not sure i fully understand this yet ): | 16:52 |
| tobiash | yeah, installing third party python is probably not the thing we want | 16:52 |
| tobiash | corvus: btw, I couldn't reproduce the issue in a xenial container with apt-get installed tox | 16:52 |
| corvus | tobiash: oy, that undermines my assumptions a bit | 16:53 |
| tobiash | or let me rephrase, with an apt-get installed virtualenv and running the command that failed | 16:54 |
| tobiash | I'll try with real tox again | 16:54 |
| corvus | if i understand mordred's theory correctly, it is that since zuul uses virtualenv (ie, the executor's management of ansible venvs), it's hitting this error *inside* the venv that tox creates. then clarkb's theory is that, as described by the upstream bug, we may also hit this in tox itself setting up that venv. | 16:54 |
| corvus | and there is some thought that we would hit both of those | 16:55 |
| corvus | we might be able to go back to the old build logs and see if the error was from inside or outside of the inner venv | 16:55 |
| tobiash | initially I saw the error in zuul-jobs where it hits the error during setting up the virtualenv by tox | 16:56 |
| tobiash | apt-get installed tox is too old :/ | 16:57 |
| corvus | in fact: https://zuul.opendev.org/t/zuul/build/03d30ebb30e14d3aa36ee204785763d7 vs https://zuul.opendev.org/t/zuul/build/b7544f27c98f4f0585616aae07e227a5 are different -- the first is my first patch (the one that didn't do anything because we ran the wrong pip); the second is the most recent version where it's now updating the pip global install. it looks like the first may be an outer error, and the | 16:58 |
| corvus | second an inner error? | 16:58 |
| mordred | corvus: yes - I think the first is the outer error | 16:58 |
| mordred | corvus: and yes, the second is the inner | 16:59 |
| corvus | light dawns on corvus | 16:59 |
| *** dmellado has quit IRC | 17:01 | |
| tobiash | corvus: I think you need to do three things | 17:01 |
| tobiash | outer error: pip, inner error #1 requirements and inner error #2 ansible venv requirements | 17:02 |
| tobiash | hrm, but the second log indicates inner error #1 | 17:03 |
| corvus | tobiash: i don't think we've gotten far enough to see #2 yet? | 17:04 |
| mordred | I agree | 17:04 |
| tobiash | yes, I just noticed that | 17:04 |
| corvus | tobiash: so if we see #2, we would need to pin in zuul/lib/ansible-config.conf ? | 17:06 |
| *** dmellado has joined #zuul | 17:07 | |
| corvus | (either that, or inside of the executor itself where it creates the virtualenv?) | 17:07 |
| openstackgerrit | James E. Blair proposed zuul/zuul master: DNM: Test fixing py35 by pinning importlib-resources in three places https://review.opendev.org/710833 | 17:08 |
| corvus | tobiash: ^ maybe like that? that's a new change so we can evaluate these separately | 17:09 |
| mordred | corvus: yah. although we might want that first in the list just in case - but yeah - I think that's potentially right | 17:09 |
| Shrews | such a confusing issue | 17:12 |
| clarkb | corvus: that change lgtm. | 17:13 |
| *** dmellado has quit IRC | 17:19 | |
| tobiash | corvus: in a xenial container I got it working with: 1. install tox in venv (/opt/tox), 2. install importlib in this venv, 3. patch zuul's requirements.txt | 17:22 |
| tobiash | patching ansible conf doesn't seem to be required | 17:22 |
| corvus | oh good | 17:23 |
| *** dmellado has joined #zuul | 17:25 | |
| tobiash | the difference I did (not sure if relevant) is that I installed tox into a venv | 17:25 |
| *** NBorg has quit IRC | 17:32 | |
| *** evrardjp has quit IRC | 17:35 | |
| *** evrardjp has joined #zuul | 17:35 | |
| Shrews | i don't understand the reasoning in that bug report, but it seems only early versions of 3.5.x are affected? i wonder what version we have | 17:36 |
| Shrews | doesn't seem to be indicated in logs | 17:37 |
| *** jpena is now known as jpena|off | 17:39 | |
| corvus | tobiash, clarkb, mordred, Shrews: the "both places" change seems to be running tests | 17:40 |
| clarkb | Shrews: ubuntu xenial is 3.5.2 and is affected (as noted in the bug) | 17:41 |
| mordred | corvus: running tests is a wonderful state to reach | 17:41 |
| clarkb | Shrews: newer python3.5 never properly branched from python3.6 so I'm guessing those trees are pretty close to each other (which is a bit crazy for other reasons imo) | 17:41 |
| Shrews | i guess that explains why patching ansible cfg is not necessary (likely installs a newer 3.5.x) | 17:42 |
| *** igordc has quit IRC | 17:52 | |
| *** jcapitao is now known as jcapitao_off | 17:54 | |
| *** saneax has quit IRC | 18:03 | |
| *** igordc has joined #zuul | 18:15 | |
| *** tjgresha has quit IRC | 18:31 | |
| clarkb | corvus: https://review.opendev.org/#/c/710833/ passed testing. Do you want to make it mergeable and/or maybe do similar for the global case in the base job? | 18:44 |
| *** harrymichal has joined #zuul | 18:46 | |
| *** jcapitao_off has quit IRC | 18:49 | |
| corvus | clarkb: seems like that's the way to go -- i probably won't be able to work on the base job until after lunch if you have time to get started on that. i will rework 833 to be mergable asap, and assuming it merges soonish, we can pull the global part of it back out after the base job change | 18:50 |
| corvus | clarkb: or rather, i'm going to make 710828 mergable | 18:50 |
| clarkb | ok I'll start on the global side | 18:50 |
| clarkb | in base jobs i mean | 18:50 |
| corvus | i don't think we need 833 | 18:50 |
| openstackgerrit | James E. Blair proposed zuul/zuul master: Fix py35 by pinning importlib-resources https://review.opendev.org/710828 | 18:54 |
| clarkb | corvus: what is the difference between the two? | 18:55 |
| *** harrymichal has quit IRC | 19:01 | |
| corvus | clarkb: the extra place where we don't appear to need the pinning is inside the ansible virtualenvs the executor uses | 19:07 |
| clarkb | ah | 19:08 |
| corvus | #1 global tox install; #2 venv created by tox; #3 venv created by zuul for ansible. testing says we need #1 and #2. we're pleasantly surprised we don't need #3, though i don't know that we know why we don't. :) | 19:09 |
| Shrews | corvus: that's what i was trying to figure out earlier. my assumption was that the venv created for ansible installs a python version recent enough to avoid the issue | 19:11 |
| corvus | Shrews: i don't think a new version of python is installed; i think it uses the system version | 19:13 |
| clarkb | corvus: ya I think that is what is happening. Because we don't pip install -U zuul | 19:14 |
| clarkb | without the -U it decides its fine with the current version? | 19:14 |
| Shrews | oh does it? well that invalidates that theory then | 19:14 |
| tristanC | could it be that the ansible venv virtualenv version is affected but it is not used? | 19:20 |
| mordred | I think it's that we aren't using any functions inside of ansible that are affected | 19:27 |
| mordred | #2 above was bombing out in a codepath that imported cherrypy which used importlib which hit the bad codepath - in the ansiblevenvs we're just running ansible things, so potentially not running code that hits the affected codepath in importlib | 19:28 |
| *** jamesmcarthur has quit IRC | 19:40 | |
| *** jamesmcarthur has joined #zuul | 19:40 | |
| *** jamesmcarthur has quit IRC | 19:45 | |
| corvus | yeah, that sounds plausible | 19:45 |
| openstackgerrit | Merged zuul/zuul master: Fix py35 by pinning importlib-resources https://review.opendev.org/710828 | 20:11 |
| armstrongs | hey has something changed with the way zuul ships logs to the logging server in 3.17.0 it doesn't seem to be able to ssh to the logging hosts but in 3.16.1 and before it is fine. | 20:46 |
| tristanC | armstrongs: is there a log of the failure? | 20:47 |
| clarkb | armstrongs: this is to copy the log files at the end of the job? | 20:47 |
| armstrongs | yeah it is | 20:48 |
| armstrongs | i dont have logs as they havent been shipped | 20:48 |
| armstrongs | "msg": "SSH Error: data could not be sent to remote host \"zuullogging-tcp.paas.je-labs.com\". Make sure this host can be reached over ssh" | 20:48 |
| armstrongs | is all i have to go on | 20:48 |
| armstrongs | when i roll back it works | 20:48 |
| clarkb | armstrongs: is that hostname correct? | 20:49 |
| armstrongs | yeah exact same config works in 3.16.1 as i deploy logging seperate | 20:49 |
| armstrongs | hostname is correct | 20:50 |
| clarkb | armstrongs: have you overridden the ssh executable value? (is it possible that something like rsync is doing that?0 | 20:50 |
| mordred | armstrongs: and no changes to zuul-jobs when you do those changes? | 20:51 |
| *** michael-beaver has joined #zuul | 20:52 | |
| armstrongs | yeah no changes to the jobs or logging server. The only change is 3.16.1 to 13.17.0 on an ec2. Base image is the same as well | 20:52 |
| armstrongs | i roll back and it works | 20:52 |
| armstrongs | i will dig some more. Just wanted to check something new hasnt been introduced :) | 20:54 |
| mordred | armstrongs: not intentionally :) | 20:54 |
| mordred | armstrongs: we did put in some blacklisting that prevents setting ssh command line arguments in job content | 20:54 |
| tristanC | armstrongs: not sure if it is related, but we had issue with ssh public key format when updating zuul/ansible/paramiko, and switching to ssh-rsa usually solved our issue | 20:54 |
| mordred | oh - that could maybe be related too | 20:55 |
| armstrongs | ok that is a great start will have a look at those. Thanks | 20:55 |
| mordred | I don't see anything else between 3.16.1 and 3.17.0 that might have that sort of impact | 20:55 |
| corvus | the zuul quick start does use a static logging host, so i think that code path should generally be tested | 20:55 |
| mordred | yeah - I'll be interested in finding out what the issue is here | 20:56 |
| armstrongs | i will compare your latest upload logging role to mine. We host them locally so they may now have some drift | 20:56 |
| armstrongs | based on what you said | 20:57 |
| *** jamesmcarthur has joined #zuul | 20:59 | |
| corvus | armstrongs: keep an eye out for any variables like "ansible_ssh_*" -- those are the ones we blacklisted in 3.17.0; if you are setting those somewhere, that may be the problem | 21:01 |
| armstrongs | the log roles are the same as upstream still.hmmm. Will take a look at vars and keys. | 21:02 |
| *** sgw has quit IRC | 21:04 | |
| *** tobiash has quit IRC | 21:13 | |
| *** tobiash has joined #zuul | 21:15 | |
| *** mattw4 has quit IRC | 21:18 | |
| clarkb | https://github.com/asottile/flake8-typing-imports#rationale is being used to deal with that importlib-resources issue | 21:29 |
| clarkb | we may want to consider using similar guards in zuul with its type checking | 21:29 |
| clarkb | I think that may allow us to do some richer type checking while still running under older python | 21:29 |
| clarkb | (but also it apparently helps to manage the myriad of changes to the type checking interfaces) | 21:29 |
| *** mattw4 has joined #zuul | 21:48 | |
| openstackgerrit | Merged zuul/zuul master: Refactor branch protection test infrastructure https://review.opendev.org/709834 | 22:07 |
| *** mattw4 has quit IRC | 22:17 | |
| openstackgerrit | Merged zuul/zuul master: Test canMerge also with checks api https://review.opendev.org/709835 | 22:19 |
| *** mattw4 has joined #zuul | 22:22 | |
| openstackgerrit | Merged zuul/zuul master: Deprecate -d switch for running in foreground https://review.opendev.org/705185 | 22:31 |
| clarkb | jamesmcarthur oh you are here. I'll repost my notes re the test survey | 22:34 |
| clarkb | question 14 needs to be split into two. One for zuul and one for nodepool. Also after taking it I think the $TIME questions are too abstract. We should probably ask. How many jobs do you run each day? How many nodes do you use each day? | 22:35 |
| clarkb | and finally question 28 should be deKata'd and zuulified | 22:35 |
| jamesmcarthur | hello :) | 22:36 |
| jamesmcarthur | Thanks for clarification on Question 14. I was wondering about that | 22:36 |
| jamesmcarthur | I'll take care of it | 22:36 |
| *** igordc has quit IRC | 22:38 | |
| corvus | jamesmcarthur: i sent an email, but SM is being very weird for me | 22:42 |
| jamesmcarthur | corvus: I've been making live updates, so that's my best guess. | 22:43 |
| jamesmcarthur | but I don't like weird | 22:43 |
| corvus | i filled out the 1st page, clicked next, and then i think it sent me back to the start page... | 22:43 |
| corvus | i started again and the 1st page was blank | 22:43 |
| jamesmcarthur | corvus: we changed the name of the survey mid-go, so that might have something to do with it. I had a weird blip as well | 22:43 |
| corvus | i went through it again, and now i'm on to the second page, but when i click next, i'm sent back to the start again. now it's go the saved answers on the first page, but second is blank again | 22:44 |
| corvus | jamesmcarthur: okay, maybe it was those updates? | 22:44 |
| corvus | i'm on to page 3 now | 22:44 |
| corvus | jamesmcarthur: okay, i made it through :) | 22:45 |
| jamesmcarthur | whew | 22:45 |
| jamesmcarthur | sorry for the hiccups | 22:45 |
| jamesmcarthur | i probably released it a bit soon | 22:46 |
| corvus | np, hopefully that was it. i'll try again after the next round of stabilization | 22:46 |
| clarkb | ya I'll happily give it another go when you are ready for us to kick the tires again | 22:48 |
| jamesmcarthur | clarkb: corvus: still working on question 14, so give me a bit on this | 22:49 |
| jamesmcarthur | i'll update this channel after that | 22:49 |
| jamesmcarthur | and then we can give some time for feedback | 22:49 |
| jamesmcarthur | I've incorporated all other feedback so far. | 22:49 |
| clarkb | jamesmcarthur: cool. I've edited the etherpad with changes to how various cloud providers seem to refer to themselves | 22:52 |
| clarkb | trying to make things stylistically consistent for answers | 22:53 |
| tristanC | i'm still worried such long list of questions might scares user away... shouldn't it at least starts by 'why do you use Zuul for?' ? | 22:55 |
| tristanC | s/why/what/ | 22:56 |
| clarkb | tristanC: https://etherpad.openstack.org/p/Zuul-User-Survey is the etherpad that is being used as input to create the survey. I think having that around line 16/17 would be good | 22:57 |
| clarkb | from a time consumption perspective determining the number of jobs/day nodes/day would be my biggest time cost | 22:58 |
| tristanC | and aren't "What deployment model do you use?" equivalent to "What deployment mechanism do you use?" ? | 22:58 |
| clarkb | tristanC: no, deployment model is "releases|releaes + lcoal patches|continuous deployment from master" vs deployment mechanism "puppet|k8s operator| ansible|etc" | 22:59 |
| clarkb | tristanC: its possible we may need to refine those questions as the answers do help determine what it is we are looking for | 22:59 |
| tristanC | clarkb: perhaps usage metrics could be log-scaled, e.g. 1~10, 10~100, 100~1000 ? | 22:59 |
| clarkb | tristanC: ya ballparking the numbers might make it easier for peopel to answer. That way they don't go looking for the specific number | 23:00 |
| mordred | ++ | 23:02 |
| tristanC | "Which areas of Zuul require further enhancement?" also sounds similar to "List missing trigger drivers", "List missing nodepool drivers", (and the probably missing "repporter drivers" question too) | 23:05 |
| clarkb | Maybe we can put that at the end and say "Is there anything else you would like to see added to Zuul?" | 23:07 |
| clarkb | I think it is important to get an idea of what people are plugging zuul into | 23:07 |
| * clarkb leaves that comment on the etherpad | 23:08 | |
| tristanC | clarkb: agree, knowing which drivers are used is valuable | 23:08 |
| clarkb | if my browser will cooperate.. | 23:08 |
| mordred | clarkb: browsers | 23:08 |
| clarkb | it has decided that now is a good time for a lunch braek | 23:09 |
| mordred | clarkb: I left a comment about deployment tools - feels like there's 2 different questions interleaved | 23:09 |
| *** armstrongs has quit IRC | 23:12 | |
| *** jamesmcarthur has quit IRC | 23:16 | |
| clarkb | mordred: ya I think that is good feedback | 23:22 |
| *** michael-beaver has quit IRC | 23:22 | |
| tristanC | jamesmcarthur left, but he commented on my suggestions that open ended questions are tougher to track year over year. < i think it depends on the amount of answers, and perhaps it will not be difficult to reduce the collected answer to common group | 23:24 |
| tristanC | i realize my suggestions actually extends the list, while i think the list should be shorter :) so please feel free to discard them (their background is orangish) | 23:34 |
| openstackgerrit | Tristan Cacqueray proposed zuul/zuul-registry master: config: add support for %(var)s syntax https://review.opendev.org/710886 | 23:39 |
| *** armstrongs has joined #zuul | 23:49 | |
| armstrongs | found the issue we had ansible_ssh_common_args: "-o StrictHostKeyChecking=no" set on add_host guess that falls under the blocked list | 23:50 |
| armstrongs | on the new version | 23:50 |
| *** tosky has quit IRC | 23:57 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!