| *** ociuhandu has joined #openstack-infra | 00:00 | |
| *** ociuhandu has quit IRC | 00:05 | |
| *** diablo_rojo has quit IRC | 00:41 | |
| *** diablo_rojo__ has joined #openstack-infra | 00:41 | |
| *** jamesmcarthur has joined #openstack-infra | 00:54 | |
| *** jamesmcarthur has quit IRC | 01:03 | |
| *** ociuhandu has joined #openstack-infra | 01:27 | |
| *** harlowja has quit IRC | 01:29 | |
| *** jamesmcarthur has joined #openstack-infra | 01:34 | |
| *** ociuhandu has quit IRC | 01:36 | |
| *** slaweq has joined #openstack-infra | 01:46 | |
| *** harlowja has joined #openstack-infra | 01:50 | |
| *** slaweq has quit IRC | 01:50 | |
| *** armax has quit IRC | 01:52 | |
| *** jamesmcarthur has quit IRC | 02:03 | |
| *** slaweq has joined #openstack-infra | 02:23 | |
| *** slaweq has quit IRC | 02:28 | |
| *** diablo_rojo__ has quit IRC | 02:30 | |
| *** ociuhandu has joined #openstack-infra | 02:33 | |
| *** ociuhandu has quit IRC | 02:38 | |
| *** diablo_rojo__ has joined #openstack-infra | 02:50 | |
| *** diablo_rojo__ has quit IRC | 03:17 | |
| *** armax has joined #openstack-infra | 03:17 | |
| *** armax has quit IRC | 03:32 | |
| *** armax has joined #openstack-infra | 03:33 | |
| *** armax has quit IRC | 03:35 | |
| *** ociuhandu has joined #openstack-infra | 03:50 | |
| *** ociuhandu has quit IRC | 03:55 | |
| *** harlowja has quit IRC | 04:10 | |
| *** harlowja has joined #openstack-infra | 04:11 | |
| *** ricolin has joined #openstack-infra | 04:20 | |
| *** harlowja has quit IRC | 04:25 | |
| *** ociuhandu has joined #openstack-infra | 04:29 | |
| *** ociuhandu has quit IRC | 04:33 | |
| *** ricolin has quit IRC | 04:36 | |
| *** smarcet has quit IRC | 04:51 | |
| *** rfolco has joined #openstack-infra | 05:11 | |
| *** rfolco has quit IRC | 05:16 | |
| *** harlowja has joined #openstack-infra | 06:25 | |
| *** ociuhandu has joined #openstack-infra | 07:02 | |
| *** ociuhandu has quit IRC | 07:06 | |
| *** ociuhandu has joined #openstack-infra | 07:31 | |
| *** ociuhandu has quit IRC | 07:37 | |
| *** ociuhandu has joined #openstack-infra | 08:00 | |
| *** ociuhandu has quit IRC | 08:05 | |
| *** corvus has quit IRC | 09:23 | |
| *** corvus has joined #openstack-infra | 09:24 | |
| *** ociuhandu has joined #openstack-infra | 09:42 | |
| *** slaweq has joined #openstack-infra | 09:46 | |
| *** ociuhandu has quit IRC | 09:48 | |
| *** jklare has quit IRC | 09:57 | |
| *** jklare has joined #openstack-infra | 09:59 | |
| *** ociuhandu has joined #openstack-infra | 10:42 | |
| *** slaweq has quit IRC | 10:43 | |
| *** ociuhandu has quit IRC | 10:49 | |
| *** ianychoi_ has joined #openstack-infra | 11:13 | |
| *** ianychoi has quit IRC | 11:15 | |
| *** kjackal has joined #openstack-infra | 11:49 | |
| *** slaweq has joined #openstack-infra | 11:58 | |
| *** tosky has joined #openstack-infra | 12:05 | |
| *** ociuhandu has joined #openstack-infra | 12:08 | |
| *** slaweq has quit IRC | 12:10 | |
| *** zigo has quit IRC | 12:10 | |
| *** zigo has joined #openstack-infra | 12:13 | |
| *** ociuhandu has quit IRC | 12:22 | |
| *** tosky has quit IRC | 12:25 | |
| *** tosky has joined #openstack-infra | 12:25 | |
| *** kjackal has quit IRC | 12:52 | |
| *** ociuhandu has joined #openstack-infra | 13:02 | |
| *** ociuhandu has quit IRC | 13:07 | |
| *** ociuhandu has joined #openstack-infra | 13:09 | |
| *** kjackal has joined #openstack-infra | 13:16 | |
| *** pkopec has joined #openstack-infra | 13:17 | |
| *** ociuhandu has quit IRC | 13:17 | |
| *** ociuhandu has joined #openstack-infra | 13:18 | |
| *** ociuhandu has quit IRC | 13:24 | |
| *** Lucas_Gray has joined #openstack-infra | 13:42 | |
| yoctozepto | infra-root, several jobs (requirements-check, grenade, test-release-openstack) break on "distutils.errors.DistutilsError: the `allow-hosts` option is not supported when using pip to install requirements. " across all openstack projects, due to new setuptools release yesterday | 14:49 |
|---|---|---|
| *** brwyatt has quit IRC | 14:49 | |
| yoctozepto | looks like config issue in infra, not able to reproduce locally on my centos 7 | 14:51 |
| *** brwyatt has joined #openstack-infra | 14:52 | |
| yoctozepto | I reported upstream https://github.com/pypa/setuptools/issues/1916 (mostly about the message being misleading) | 14:53 |
| fungi | yoctozepto: interesting, we use that to mark the mirrors as allowed package sources | 14:59 |
| yoctozepto | fungi: thought so, though now it is gone and breaking | 15:00 |
| fungi | yoctozepto: specifically, we set it here: https://opendev.org/zuul/zuul-jobs/src/branch/master/roles/configure-mirrors/templates/.pydistutils.cfg.j2#L4 | 15:03 |
| mordred | fungi, yoctozepto: the internet tells me that pip wants "trusted-host" | 15:03 |
| yoctozepto | fungi: ah, explains with codesearch with allow-hosts returned nothing, doh | 15:03 |
| yoctozepto | explains why* | 15:03 |
| yoctozepto | mordred, fungi: we need that line gone anyway, because it breaks raw setup.py as well | 15:04 |
| fungi | yoctozepto: also, setuptools *is* calling pip apparently, see the top entry in the changelog: https://github.com/pypa/setuptools/blob/master/CHANGES.rst | 15:04 |
| yoctozepto | fungi: ah, right | 15:05 |
| yoctozepto | should have read bolded releasenotes :D | 15:05 |
| fungi | we need that entry for older setuptools, but when it's calling out to pip that gets handled for us here instead: https://opendev.org/zuul/zuul-jobs/src/branch/master/roles/configure-mirrors/templates/etc/pip.conf.j2#L5 | 15:06 |
| fungi | backward-compatibility would be nice so we don't have to alter .pydistutils.cfg depending on what version of setuptools gets invoked (there may even be more than one on the same system thanks to virtualenvs) | 15:06 |
| yoctozepto | fungi: hmm, maybe join the discussion on upstream issue, might be they care to make it more backwards compatible and do a quick new release... | 15:08 |
| fungi | apparently firefox wants to restart on me, so may be a few minutes | 15:18 |
| fungi | okay, feature request for backward-compat added to the issue you reported, yoctozepto | 15:25 |
| fungi | you might consider updating the issue title to be a little more accurate | 15:25 |
| yoctozepto | fungi: right | 15:26 |
| yoctozepto | fungi: did a few edits around | 15:30 |
| yoctozepto | in the meantime - would it be possible to pin setuptools? otherwise we are waiting for a move from pypa team to unblock many pending changes... | 15:31 |
| fungi | not sure... i want to say virtualenv/pip/something always grabs the latest setuptools version? | 15:35 |
| fungi | so it could be tricky | 15:35 |
| fungi | yoctozepto: can you provide a link to a breaking job? maybe we can figure out from that where/when setuptools gets upgraded to the new version | 15:36 |
| mordred | fungi: do we actually having code using setuptools old enough to need the backwards compat? | 15:36 |
| fungi | i honestly have no clue | 15:37 |
| yoctozepto | fungi: seems not all jobs, py* jobs and lower-constraints pass fine | 15:37 |
| yoctozepto | gimme a sec | 15:37 |
| mordred | I *think* we pretty much use latest setuptools everywhere - for anything that uses pip to install | 15:37 |
| mordred | unless I'm wrong :) | 15:37 |
| yoctozepto | fungi: https://review.opendev.org/694646 | 15:37 |
| fungi | yeah, that's why i want an actual example to start from | 15:38 |
| yoctozepto | here is a pretty summary of fail/not fail | 15:38 |
| yoctozepto | so it's either different config | 15:38 |
| yoctozepto | or different setuptools :-) | 15:38 |
| fungi | or not calling setuptools at all | 15:38 |
| mordred | we've been trying to prevent things from using easy_install paths for years | 15:39 |
| *** ccamacho has joined #openstack-infra | 15:41 | |
| yoctozepto | fungi: oh, I'm pretty sure they are called | 15:43 |
| yoctozepto | I would blame tox venvs having old (non-upgraded) setuptools | 15:43 |
| corvus | there was some recursive case at some point (like pip install foo caused setuptools install bar as a dependency) | 15:43 |
| yoctozepto | mordred: still we have python setup.py around... ;-) | 15:43 |
| corvus | don't know if that still applies | 15:43 |
| mordred | corvus: yeah - there were some edge cases back in the day - but I think they've been sorted | 15:45 |
| fungi | newer pip masquerades as easy_install so that things you install via pip don't actually recurse to it | 15:46 |
| mordred | yoctozepto: yeah - but I'm pretty sure it's using pip behind the scenes in all of the scenarios that mattter anymore | 15:46 |
| fungi | but i forget when they implemented that, been a few years i think | 15:46 |
| yoctozepto | mordred: if it's upgrade | 15:46 |
| yoctozepto | upgraded* | 15:47 |
| yoctozepto | fungi, mordred, corvus: https://github.com/pypa/setuptools/issues/1916 | 15:47 |
| yoctozepto | I think you guys are better-suited to replying to our break-creator :-) | 15:48 |
| yoctozepto | https://github.com/pypa/setuptools/commit/d6948c636f5e657ac56911b71b7a459d326d8389 ;-) | 15:48 |
| *** Lucas_Gray has quit IRC | 15:49 | |
| yoctozepto | fungi: "try to avoid calling into setuptools" how so? not install anything? :D | 15:54 |
| *** ccamacho has quit IRC | 15:55 | |
| fungi | yoctozepto: run wheel and pip directly for things | 15:58 |
| yoctozepto | fungi: well, pip breaks too, e.g. in grenade | 15:58 |
| fungi | or use system packages maybe? | 15:59 |
| fungi | at any rate, yes probably not a great option | 15:59 |
| fungi | my point there was kinda that | 15:59 |
| fungi | we have several not-great options to suggest to users as workarounds | 15:59 |
| mordred | well - if our primary interface is pip (which it has been for *years*) and pydistutils.cfg is there only to catch the occasional accidental uses of easy_install ... and pydistutils.cfg is now breaking pip | 16:02 |
| fungi | (breaking pip indirectly by breaking setuptools) | 16:02 |
| mordred | I'd say we should stop configuring pydistutils.cfg - the only downside of which is making easy_install using codepaths less reliable | 16:02 |
| fungi | but yes | 16:02 |
| mordred | and if someone finds they are hitting that less reliable codepath, they can update their code to use pip directly and avoid easy_install which is what they should have been doing anyway | 16:03 |
| fungi | well, less reliable or inefficient/nonfunctional in the case of anything relying on prebuilt wheels | 16:04 |
| fungi | but sure | 16:04 |
| mordred | yeah. but - yeah - maybe better to find what should be a vanishingly small number of cases where people are relying on something we don't value anyway | 16:04 |
| fungi | we can declare a moratorium on supporting easy_install with our default configurations | 16:04 |
| mordred | yeah | 16:05 |
| fungi | i support that | 16:05 |
| yoctozepto | +1 | 16:06 |
| openstackgerrit | Monty Taylor proposed zuul/zuul-jobs master: Stop installing pydistutils.cfg https://review.opendev.org/695821 | 16:07 |
| mordred | we can discuss on that ^^ | 16:07 |
| *** ccamacho has joined #openstack-infra | 16:07 | |
| *** kjackal has quit IRC | 16:08 | |
| mordred | but yeah - also - *many* jobs are broken | 16:09 |
| mordred | fungi: https://zuul.opendev.org/t/openstack/build/7794ca7653e24fa69b32b193ad1b07b7 | 16:09 |
| mordred | fungi: https://review.opendev.org/#/c/695795/ | 16:09 |
| yoctozepto | mordred: yeah, that's why I alarmed you | 16:10 |
| yoctozepto | (you = infra) | 16:10 |
| yoctozepto | (to avoid CI Blue Monday) | 16:10 |
| *** Lucas_Gray has joined #openstack-infra | 16:14 | |
| fungi | yoctozepto: you could also depends-on that change i think to confirm whether that solves the problem | 16:17 |
| yoctozepto | fungi: all right | 16:17 |
| yoctozepto | https://review.opendev.org/#/c/694646/ | 16:18 |
| yoctozepto | should be all green (ignoring randomness) | 16:19 |
| *** ociuhandu has joined #openstack-infra | 16:19 | |
| yoctozepto | fungi: still "sees" allow-hosts https://9b6a46031744f34b6eb1-08136eee394ca142f86118487824fe1a.ssl.cf1.rackcdn.com/694646/3/check/test-release-openstack/e9c7147/job-output.json | 16:22 |
| fungi | yoctozepto: mordred: i suspect that role was merely replacing the default copy installed by https://opendev.org/openstack/diskimage-builder/src/branch/master/diskimage_builder/elements/pypi/pre-install.d/04-configure-pypi-mirror#L67 | 16:24 |
| mordred | fungi: oh. yeah | 16:24 |
| yoctozepto | empty it instead :D | 16:24 |
| openstackgerrit | Monty Taylor proposed zuul/zuul-jobs master: Remove pydistutils.cfg if it's there https://review.opendev.org/695822 | 16:26 |
| yoctozepto | fungi: https://review.opendev.org/#/c/694646/ | 16:28 |
| openstackgerrit | Monty Taylor proposed openstack/diskimage-builder master: Stop installing pydistutils.cfg https://review.opendev.org/695823 | 16:28 |
| mordred | fungi, yoctozepto: there's a remove the file - and a dib cleanup | 16:28 |
| yoctozepto | fungi, mordred: aaand it still see allow-hosts https://76e83cb25f5bb60c76b4-73e606dbd0efde3a717e3340853334ea.ssl.cf5.rackcdn.com/694646/4/check/test-release-openstack/a68d709/job-output.json | 16:34 |
| *** kjackal has joined #openstack-infra | 16:34 | |
| yoctozepto | by the logs | 16:35 |
| yoctozepto | it looks like it still tries to template it | 16:35 |
| yoctozepto | so depends-on did not act upon it | 16:36 |
| *** kjackal has quit IRC | 16:36 | |
| fungi | ahh, yeah, it shows opendev.org/zuul/zuul-jobs/roles/configure-mirrors is still installing /home/zuul/.pydistutils.cfg | 16:40 |
| yoctozepto | fungi, mordred: more problematic | 16:40 |
| yoctozepto | is that dib jobs will not pass now | 16:41 |
| yoctozepto | because of allow-hosts | 16:41 |
| yoctozepto | and opensuse15 breaks zuul fixes, oh well | 16:44 |
| yoctozepto | :D | 16:44 |
| corvus | it may run in a trusted playbook (i don't think we ever finished moving to pabelanger's base job scheme); if so, we'd need to land the change blind or do the base-test dance. | 16:45 |
| yoctozepto | ^ and fix opensuse15 to get them merging | 16:48 |
| *** ociuhandu has quit IRC | 16:48 | |
| *** ociuhandu has joined #openstack-infra | 16:49 | |
| *** ociuhandu has quit IRC | 16:54 | |
| openstackgerrit | Monty Taylor proposed zuul/zuul-jobs master: Stop installing pydistutils.cfg https://review.opendev.org/695821 | 17:04 |
| mordred | there it is squashed, in case we want to force-land it | 17:04 |
| yoctozepto | mordred: unfortunately you might have to by the looks of it | 17:09 |
| yoctozepto | +1 from me though | 17:09 |
| yoctozepto | fungi, corvus: ^^ | 17:12 |
| *** ccamacho has quit IRC | 17:13 | |
| *** Lucas_Gray has quit IRC | 18:49 | |
| *** ociuhandu has joined #openstack-infra | 18:51 | |
| *** ociuhandu has quit IRC | 19:00 | |
| *** takamatsu has quit IRC | 19:10 | |
| *** takamatsu has joined #openstack-infra | 19:11 | |
| *** ociuhandu has joined #openstack-infra | 19:18 | |
| *** ociuhandu has quit IRC | 19:23 | |
| *** ociuhandu has joined #openstack-infra | 19:36 | |
| *** ociuhandu has quit IRC | 19:36 | |
| *** ociuhandu has joined #openstack-infra | 19:37 | |
| *** ociuhandu has quit IRC | 19:50 | |
| *** ociuhandu has joined #openstack-infra | 19:51 | |
| *** goldyfruit_ has quit IRC | 20:24 | |
| *** ociuhandu has quit IRC | 20:28 | |
| *** ociuhandu has joined #openstack-infra | 20:31 | |
| *** ociuhandu has quit IRC | 20:35 | |
| *** takamatsu has quit IRC | 20:42 | |
| *** takamatsu has joined #openstack-infra | 20:44 | |
| ianw | hrm, i'll be fully online in ~ 1 hour and can take a look ... | 21:05 |
| *** hamzy has joined #openstack-infra | 21:08 | |
| *** ociuhandu has joined #openstack-infra | 21:13 | |
| mordred | ianw: you're going to "enjoy" this one | 21:15 |
| *** ociuhandu has quit IRC | 21:19 | |
| *** ociuhandu has joined #openstack-infra | 21:55 | |
| ianw | ok, i have no idea what's going on, but i'm here :) | 21:57 |
| *** ociuhandu has quit IRC | 22:04 | |
| *** armax has joined #openstack-infra | 22:06 | |
| ianw | ok, removing the file seems like the way forwrad | 22:07 |
| ianw | 2019-11-24 17:09:40.519649 | opensuse-15 | Repository 'repo-update' is invalid. | 22:07 |
| ianw | 2019-11-24 17:09:40.519751 | opensuse-15 | [repo-update|http://mirror.ord.rax.opendev.org/opensuse/update/leap/15.1/oss/] Valid metadata not found at specified URL | 22:07 |
| *** dchen has quit IRC | 22:09 | |
| ianw | afaics, this is synced corrrectly | 22:10 |
| ianw | + k5start -t -f /etc/opensuse.keytab service/opensuse-mirror -- timeout -k 2m 30m rsync -rlptDvz --delete --stats --delete-excluded --exclude=src/ --exclude=nosrc/ --exclude=.~tmp~ rsync://mirrors.rit.edu/opensuse/update/leap/15.1/ /afs/.o | 22:11 |
| ianw | penstack.org/mirror/opensuse/update/leap/15.1/ | 22:11 |
| ianw | Total transferred file size: 0 bytes | 22:11 |
| ianw | i.e. the latest run had nothing to do | 22:11 |
| ianw | last time it ran successfully was 2019-11-21T22:18:28, which leaves a large window of possible breaking updates | 22:13 |
| ianw | ok, i also think not much uses the dib codepath, so that's less urgent | 22:23 |
| *** armax has quit IRC | 22:24 | |
| *** goldyfruit has joined #openstack-infra | 22:28 | |
| *** ociuhandu has joined #openstack-infra | 22:31 | |
| *** ociuhandu has quit IRC | 22:36 | |
| openstackgerrit | Merged zuul/zuul-jobs master: Stop installing pydistutils.cfg https://review.opendev.org/695821 | 22:37 |
| ianw | #status log force merged pydistutils.cfg removal patch https://review.opendev.org/695821 ; see notes in change about reasoning | 22:37 |
| openstackstatus | ianw: finished logging | 22:37 |
| *** adam_g has quit IRC | 22:39 | |
| *** adam_g has joined #openstack-infra | 22:40 | |
| fungi | thanks ianw! if you get a sec, assuming it seems to be working, could you send a quick couple sentences to the zuul-discuss ml letting folks know this changed since it's a behavior change to a role in the zuul-jobs repo and not just something opendev-specific? | 22:41 |
| ianw | fungi: yep, just looking at the opensuse job to get the gate back, then will do | 22:41 |
| fungi | awesome, if i weren't in the middle of beating a batch of fudge i'd send it | 22:42 |
| *** jamesmcarthur has joined #openstack-infra | 22:47 | |
| *** rcernin has joined #openstack-infra | 22:57 | |
| ianw | ]$ grep 'zuul-jobs-test-install-kubernetes-docker' * | 22:57 |
| ianw | container-roles-jobs.yaml: name: zuul-jobs-test-install-kubernetes-docker | 22:57 |
| ianw | container-roles-jobs.yaml: name: zuul-jobs-test-install-kubernetes-docker | 22:57 |
| ianw | we appear to define the same job twice ... let the yak shaving begin :) | 22:57 |
| *** tkajinam has joined #openstack-infra | 23:03 | |
| *** pkopec has quit IRC | 23:10 | |
| *** dchen has joined #openstack-infra | 23:17 | |
| *** rosmaita has left #openstack-infra | 23:20 | |
| *** rfolco has joined #openstack-infra | 23:27 | |
| *** rfolco has quit IRC | 23:29 | |
| *** rfolco has joined #openstack-infra | 23:29 | |
| *** jamesmcarthur has quit IRC | 23:30 | |
| *** ociuhandu has joined #openstack-infra | 23:30 | |
| *** rfolco has quit IRC | 23:34 | |
| *** rfolco has joined #openstack-infra | 23:35 | |
| *** ociuhandu has quit IRC | 23:35 | |
| *** rfolco has quit IRC | 23:40 | |
| *** rfolco has joined #openstack-infra | 23:56 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!