| fungi | openstack victoria will be targeting ubuntu 20.04 lts, so you'll at least have newer sqlite available next cycle | 00:00 |
|---|---|---|
| clarkb | right maybe not ubuntu has newer sqlite | 00:00 |
| fungi | centos 8, opensuse 15... | 00:00 |
| *** eernst has quit IRC | 00:00 | |
| clarkb | I don't know what the implications of setting atomic to false are either | 00:01 |
| adriant | I do mostly container stuff lately so I rarely run into older versions of things :( | 00:01 |
| clarkb | but might be an option for testing? | 00:01 |
| adriant | I'm doing that now | 00:01 |
| adriant | may just be a safe migration to change | 00:02 |
| *** slaweq has quit IRC | 00:04 | |
| *** mattw4 has quit IRC | 00:08 | |
| adriant | tests running with the new value, but if it does pass then I have to go back and retest against mariadb versions :( | 00:12 |
| adriant | yep, seems to be passing | 00:13 |
| adriant | onwards to mariadb testing | 00:13 |
| *** jamesmcarthur has joined #openstack-infra | 00:14 | |
| clarkb | fwiw I think a lot of the openstack projects only use sqlite for simple testing, The migrations are never tested against sqlite because people aren't expected to use sqlite in production | 00:14 |
| *** jamesmcarthur has quit IRC | 00:19 | |
| clarkb | also you mentioned containers earlier, nothing prevents you from using containers on our CI system. Though it may be at odds with the project testing interface depending on how you use them | 00:19 |
| *** jamesmcarthur has joined #openstack-infra | 00:24 | |
| *** jamesmcarthur has quit IRC | 00:29 | |
| adriant | clarkb: I just set "atomic = False" and it worked, and turns out after some more digging, what that enables is a feature only sqlite and postgres have anyway, so i don't even need to retest on mariadb | 00:44 |
| adriant | and I don't test the migrations on the ci system, they just happen to run to initialise the tests | 00:44 |
| adriant | django spins up tmp db to have the code run against | 00:45 |
| adriant | and by default that is sqlite, although it could well point to a mariadb and the tests will initialise the tables as needed | 00:47 |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: afs-release: fix cron job output capture https://review.opendev.org/707033 | 00:50 |
| *** gfidente|afk has quit IRC | 00:51 | |
| clarkb | ianw: any idea why https://review.opendev.org/#/c/697589/ the afs role there is failing on centos 8 job? | 01:05 |
| clarkb | seems like the openafs service isn't starting on centos8? | 01:06 |
| fungi | and looks like that may have just started? | 01:14 |
| fungi | well, "just" in the past couple weeks | 01:15 |
| clarkb | ya I assume the test was running successfully previously. But I thought the rpms were pretty carefully managed (and by that I really mean we don't update it much) | 01:15 |
| fungi | the job passed as recently as 2020-01-28 | 01:16 |
| fungi | and we don't collect system logs, so hard to know why just from the build results | 01:18 |
| *** rfolco has quit IRC | 01:19 | |
| *** dciabrin_ has joined #openstack-infra | 01:21 | |
| *** dciabrin has quit IRC | 01:22 | |
| *** yamamoto has joined #openstack-infra | 01:30 | |
| *** armax has quit IRC | 01:39 | |
| ianw | sorry, was just eating lunch, looking | 01:49 |
| ianw | where is the "install kernel modes" step happening? | 01:51 |
| clarkb | is that dfrom the package? | 01:52 |
| ianw | ... interesting, i do not see them in the console view, but see it in the txt output | 01:52 |
| ianw | https://zuul.opendev.org/t/openstack/build/267b48d246364f9da3773bda52a6c287/console | 01:52 |
| ianw | versus | 01:52 |
| ianw | https://zuul.opendev.org/t/openstack/build/267b48d246364f9da3773bda52a6c287/log/job-output.txt#368 | 01:53 |
| ianw | if i'm reading it right, it only took about 40 seconds on the install kernel modules step; i think that is too short and suggests maybe they didn't build correctly | 01:54 |
| ianw | http://zuul.openstack.org/build/bd2e393ff1e340c6a7cdc4af6eaa41af was the last good run | 01:56 |
| ianw | same kernel according to host info | 01:57 |
| ianw | 2020-01-28 20:09:11.533830 | TASK [openafs-client : Install kernel modules] | 01:57 |
| ianw | 2020-01-28 20:20:09.390162 | base | changed | 01:57 |
| ianw | that's more about how long it takes to compile everything | 01:58 |
| ianw | when i look at the json https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_0b4/697589/2/check/system-config-zuul-role-integration-centos-8/0b4f90f/job-output.json | 01:58 |
| ianw | "Install kernel modules" has a start, but no end -- i think this would explain why it's missing from console | 01:59 |
| ianw | ... so i think we now have a lot more questions than answers :/ | 02:01 |
| ianw | the install is a call to yum: @ https://opendev.org/opendev/system-config/src/branch/master/roles/openafs-client/tasks/openafs-client/CentOS.yaml#L31 | 02:04 |
| *** david-lyle is now known as dklyle | 02:05 | |
| *** psachin has joined #openstack-infra | 02:06 | |
| ianw | i think this must be a bug in the yum module somehow that it misses the failure | 02:09 |
| *** jamesmcarthur has joined #openstack-infra | 02:09 | |
| clarkb | huh | 02:09 |
| *** rlandy has quit IRC | 02:09 | |
| clarkb | would dnf ve better here or is that the same thing? | 02:10 |
| ianw | clarkb: jinx, i was just reading that :) | 02:11 |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: openafs-client: use dnf for CentOS 8 https://review.opendev.org/707037 | 02:16 |
| ianw | we need the bare call (not package:) because of the enablerepo | 02:16 |
| kevinz | ianw: sure, I see. Thanks, will sort out the image deletion problem in Linaro London | 02:17 |
| *** yamamoto has quit IRC | 02:24 | |
| *** zxiiro has quit IRC | 02:25 | |
| ianw | "Your kernel headers for kernel 4.18.0-147.3.1.el8_1.x86_64 cannot be found at" ... maybe something changed and we're not install header packages on hosts nay more? | 02:28 |
| *** nicolasbock has quit IRC | 02:28 | |
| ianw | # echo $? | 02:28 |
| ianw | 0 | 02:28 |
| ianw | how nice that it still returns 0 | 02:28 |
| *** jamesmcarthur has quit IRC | 02:28 | |
| openstackgerrit | Steve Baker proposed zuul/zuul master: Scheduler: make autohold hold_list configurable https://review.opendev.org/632498 | 02:31 |
| *** jamesmcarthur has joined #openstack-infra | 02:31 | |
| *** roman_g has quit IRC | 02:34 | |
| openstackgerrit | Merged opendev/system-config master: afs-release: fix cron job output capture https://review.opendev.org/707033 | 02:34 |
| *** jamesmcarthur has quit IRC | 02:42 | |
| ianw | i think our centos8 builds have fallen behind and now there's a new kernel, and ... everything is messed up | 02:47 |
| clarkb | oh hrm | 02:48 |
| ianw | it looks like there's stuck yum processes from old builds on nb01 and nb02, and it's prevented new builds | 02:51 |
| ianw | been around since feb09 | 02:51 |
| ianw | i'm going to reboot the hosts and clear out crap | 02:51 |
| *** yamamoto has joined #openstack-infra | 03:00 | |
| *** yamamoto has quit IRC | 03:03 | |
| *** gyee has quit IRC | 03:11 | |
| *** fdegir has quit IRC | 03:14 | |
| *** fdegir has joined #openstack-infra | 03:14 | |
| ianw | #status log rebooted nb01 & nb02 and cleared out tmp directories as yum-based builds were getting stuck on old, unexited dib processes | 03:22 |
| openstackstatus | ianw: finished logging | 03:22 |
| ianw | the problem is that it boots one kernel, then installs "kernel-headers" which is now the new kernel package. so it can't find the headers for the running kernel | 03:25 |
| ianw | this should sort itself out with new builds so that everything is consistent | 03:26 |
| *** dychen has joined #openstack-infra | 03:34 | |
| *** dchen has quit IRC | 03:36 | |
| *** sgw has joined #openstack-infra | 03:38 | |
| fungi | really? they don't keep multiple versioned kernel-headers packages coinstallable? | 03:42 |
| fungi | i don't know how i'd survive if debian didn't do that, what with all the lkm compiling i do against non-live kernel versions before rebooting | 03:42 |
| fungi | for openafs in particular, when i install a new kernel dkms gets called form a post-install hook to build the lkm for the new version so whenever i get around to rebooting it's already installed | 03:44 |
| fungi | er, dkms gets called FROM a post-install hook | 03:44 |
| *** hongbin has joined #openstack-infra | 03:48 | |
| *** dychen has quit IRC | 03:53 | |
| *** dchen has joined #openstack-infra | 03:54 | |
| ianw | i think it's a always use the latest approach, similar to the 8.0 -> 8.1 transition | 03:57 |
| fungi | except that requires a reboot, which doesn't happen immediately | 04:03 |
| fungi | i guess the bootloader doesn't offer the prior kernel version as a fallback option either? | 04:03 |
| ianw | i think it clears out after the new package goes in? i don't see the bootloader prompt much! | 04:14 |
| kevinz | ianw: image from Linaro-london has been deleted | 04:15 |
| kevinz | ianw: ceph issue here in Linaro-london has been fixed | 04:15 |
| *** udesale has joined #openstack-infra | 04:19 | |
| *** ykarel|away is now known as ykarel | 04:20 | |
| *** yamamoto has joined #openstack-infra | 04:37 | |
| *** yamamoto has quit IRC | 04:40 | |
| *** hongbin has quit IRC | 04:41 | |
| *** goldyfruit has quit IRC | 04:49 | |
| ianw | kevinz: thanks! | 05:00 |
| ianw | kevinz: that seems to have fixed all our leaked nodes; nodepool was blocking trying to delete and not cleaning up, but now it's fixed things are progressing again | 05:04 |
| ianw | i think that should make both the linaro-us and london cloud fully operational now | 05:05 |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: Migrate AFS publishing to mirror-update.opendev.org https://review.opendev.org/706731 | 05:15 |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: Add project.tarballs to periodic AFS release https://review.opendev.org/706732 | 05:15 |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: afs-release: run every 5 minutes https://review.opendev.org/707060 | 05:15 |
| *** vesper has quit IRC | 05:18 | |
| *** ramishra_ has joined #openstack-infra | 05:19 | |
| *** vesper11 has joined #openstack-infra | 05:23 | |
| *** raukadah is now known as chkumar|rover | 05:28 | |
| *** evrardjp has quit IRC | 05:34 | |
| *** evrardjp has joined #openstack-infra | 05:34 | |
| *** igordc has joined #openstack-infra | 05:51 | |
| mnaser | mordred: OSA does deploy on el7 but using py2 because some packages like selinux under py3 don’t exist. noonedeadpunk knows a bit more on those details | 05:53 |
| *** igordc has quit IRC | 06:00 | |
| *** lmiccini has joined #openstack-infra | 06:30 | |
| openstackgerrit | Merged zuul/zuul master: Install kubectl/oc into executor container image https://review.opendev.org/706995 | 06:46 |
| *** ccamacho has quit IRC | 06:50 | |
| *** jtomasek has joined #openstack-infra | 07:01 | |
| *** lpetrut has joined #openstack-infra | 07:03 | |
| *** arif-ali has joined #openstack-infra | 07:11 | |
| rm_work | dunno if its come up yet, but virtualenv 20.0 released yesterday, and it broke our workflow by using more symlinks rather than copying in system libs like setuptools... makes things less portable :( | 07:14 |
| rm_work | so just an FYI | 07:14 |
| kevinz | ianw: Good to hear, thanks for help! | 07:24 |
| *** pgaxatte has joined #openstack-infra | 07:36 | |
| openstackgerrit | Felix Schmidt proposed zuul/zuul master: Implement basic github checks API workflow https://review.opendev.org/705168 | 07:37 |
| zbr | i can confirm that virtuaenv broke almost everything, even bindep https://dashboard.zuul.ansible.com/t/ansible/build/1e148fb6d2684932a0620d6b23e5909d/console | 07:37 |
| zbr | https://github.com/pypa/virtualenv/issues/1551 | 07:37 |
| *** ykarel is now known as ykarel|lunch | 07:45 | |
| *** udesale has quit IRC | 07:46 | |
| *** udesale has joined #openstack-infra | 07:47 | |
| *** Lucas_Gray has joined #openstack-infra | 07:52 | |
| *** imacdonn has quit IRC | 07:53 | |
| *** imacdonn has joined #openstack-infra | 07:53 | |
| fungi | rm_work: zbr: the case we hit in zuul's testing seems to be solvable by using --seeder=pip in the virtualenv command-line | 08:09 |
| fungi | the default app-data seeder is what does the symlinking | 08:09 |
| rm_work | hmm interesting | 08:10 |
| rm_work | i'll test that | 08:10 |
| rm_work | for now i pinned it in our CI | 08:10 |
| zbr | fungi: are we talking about the same issue with newer venv or a different one? mine was caused by incompatibility with older six | 08:12 |
| rm_work | yep seeder=pip seems to fix the issue i was looking at, thanks fungi | 08:13 |
| *** dchen has quit IRC | 08:13 | |
| fungi | zbr: the issue with virtualenv 20 symlinking its seed packages into ~/.local/share/ | 08:14 |
| zbr | what i did not understood is who installs newer virtualenv because if we cannot even do a "virtualenv --help" we are in deep,... | 08:14 |
| fungi | manifests as things like pkg_resources import failures | 08:15 |
| rm_work | yep | 08:15 |
| zbr | so gabor found a sneeky way to for us to upgrade the toolchain :D | 08:15 |
| rm_work | we bundle venvs into RPMs T_T | 08:15 |
| rm_work | so they end up places with broken symlinks | 08:15 |
| fungi | the "good news" is that virtualenv 20 claims to create relocateable environments now | 08:16 |
| fungi | at least per the release notes | 08:16 |
| rm_work | hmmm | 08:16 |
| fungi | i haven't tried relocating one yet | 08:16 |
| rm_work | obviously not by default | 08:16 |
| rm_work | since whatever seeder they use by default does tons of painful symlinking | 08:17 |
| fungi | though *if* the symlinks are canonical and not relative then they're still relocatable on the same system for the same user | 08:17 |
| fungi | but yeah, not exactly my definition of relocateable | 08:17 |
| rm_work | yeah | 08:18 |
| rm_work | they claimed to have an option --always-copy | 08:18 |
| rm_work | but it ... didn't seem to do anything? so maybe i was misunderstanding its purpose | 08:18 |
| fungi | granted it used to be just renaming the directory for a virtualenv would render it broken | 08:18 |
| zbr | fungi: imho https://opendev.org/zuul/zuul-jobs/src/branch/master/roles/bindep/tasks/install.yaml#L12 is now broken | 08:22 |
| zbr | and w/o way to fix it. | 08:23 |
| zbr | because I cannot tell ansible to add an extra new parameter when trying to create this virtualenv | 08:23 |
| zbr | ... only when virtualenv is >20.... | 08:23 |
| *** tosky has joined #openstack-infra | 08:27 | |
| *** priteau has joined #openstack-infra | 08:28 | |
| *** tesseract has joined #openstack-infra | 08:29 | |
| openstackgerrit | Sorin Sbarnea proposed zuul/zuul-jobs master: WIP: Make bindep role compatible with newer virtualenv https://review.opendev.org/707078 | 08:30 |
| *** Lucas_Gray has quit IRC | 08:39 | |
| openstackgerrit | Sorin Sbarnea proposed zuul/zuul-jobs master: WIP: Make bindep role compatible with newer virtualenv https://review.opendev.org/707078 | 08:39 |
| *** ccamacho has joined #openstack-infra | 08:46 | |
| *** Lucas_Gray has joined #openstack-infra | 08:47 | |
| *** ykarel|lunch is now known as ykarel | 08:50 | |
| *** ralonsoh has joined #openstack-infra | 08:52 | |
| *** jpena|off is now known as jpena | 08:54 | |
| *** rpittau|afk is now known as rpittau | 08:55 | |
| *** tommylikehu_ has joined #openstack-infra | 08:57 | |
| *** Lucas_Gray has quit IRC | 09:00 | |
| *** ccamacho has quit IRC | 09:01 | |
| *** Lucas_Gray has joined #openstack-infra | 09:01 | |
| *** elod has quit IRC | 09:03 | |
| *** slaweq has joined #openstack-infra | 09:07 | |
| *** dtantsur|afk is now known as dtantsur | 09:12 | |
| *** elod has joined #openstack-infra | 09:12 | |
| frickler | ftr, devstack at least on stable/rocky seems to be also affected by some related issue https://zuul.opendev.org/t/openstack/build/5ad657e46ca04f73af7fbe1604ab4dc1 | 09:13 |
| *** lucasagomes has joined #openstack-infra | 09:16 | |
| *** roman_g has joined #openstack-infra | 09:20 | |
| openstackgerrit | Merged zuul/zuul master: Extract allow/disallow filter into util function https://review.opendev.org/706144 | 09:21 |
| *** pkopec has joined #openstack-infra | 09:22 | |
| *** sshnaidm|afk is now known as sshnaidm | 09:25 | |
| *** takamatsu has joined #openstack-infra | 09:28 | |
| *** ociuhandu has joined #openstack-infra | 09:28 | |
| *** ociuhandu has quit IRC | 09:29 | |
| openstackgerrit | Matthieu Huin proposed zuul/zuul master: JWT drivers: Deprecate RS256withJWKS, introduce OpenIDConnect https://review.opendev.org/701972 | 09:30 |
| openstackgerrit | Matthieu Huin proposed zuul/zuul master: Authorization rules: add templating https://review.opendev.org/705193 | 09:30 |
| *** Lucas_Gray has quit IRC | 09:33 | |
| rm_work | yep T_T chaos | 09:40 |
| openstackgerrit | Neil Jerram proposed openstack/project-config master: End gating for networking-calico, as retiring project from OpenStack https://review.opendev.org/707086 | 09:41 |
| *** psachin has quit IRC | 09:42 | |
| *** derekh has joined #openstack-infra | 09:44 | |
| *** psachin has joined #openstack-infra | 09:47 | |
| *** Lucas_Gray has joined #openstack-infra | 09:51 | |
| *** psachin has quit IRC | 09:54 | |
| *** slaweq has quit IRC | 09:56 | |
| *** psachin has joined #openstack-infra | 10:02 | |
| *** ociuhandu has joined #openstack-infra | 10:06 | |
| *** xek has joined #openstack-infra | 10:07 | |
| *** ociuhandu has quit IRC | 10:09 | |
| *** ociuhandu has joined #openstack-infra | 10:09 | |
| *** ociuhandu has quit IRC | 10:10 | |
| noonedeadpunk | fungi: can I ask for another autohold? I'm pretty sure that directory is created and I can't reproduce failure locally.... https://zuul.opendev.org/t/openstack/build/70dca1b262264ec98c5e623892cfa8a1/log/job-output.txt#17538 | 10:11 |
| *** ociuhandu has joined #openstack-infra | 10:11 | |
| *** ociuhandu has quit IRC | 10:18 | |
| *** psachin has quit IRC | 10:20 | |
| *** ociuhandu has joined #openstack-infra | 10:20 | |
| *** psachin has joined #openstack-infra | 10:21 | |
| *** ociuhandu has quit IRC | 10:21 | |
| openstackgerrit | Tobias Henkel proposed zuul/zuul master: DNM: Try quickstart with gerrit and pure ssh https://review.opendev.org/707092 | 10:21 |
| openstackgerrit | Tobias Henkel proposed zuul/zuul master: Offload repo reset to processes https://review.opendev.org/706827 | 10:22 |
| mnaser | does infra ship 'virtualenv' inside its images? | 10:22 |
| *** ociuhandu has joined #openstack-infra | 10:26 | |
| *** ociuhandu has quit IRC | 10:27 | |
| *** priteau has quit IRC | 10:28 | |
| *** ociuhandu has joined #openstack-infra | 10:29 | |
| *** ociuhandu has quit IRC | 10:30 | |
| *** pkopec_ has joined #openstack-infra | 10:32 | |
| *** pkopec has quit IRC | 10:33 | |
| openstackgerrit | Tobias Henkel proposed zuul/zuul master: DNM: Offload repo reset fixed with ssh https://review.opendev.org/707095 | 10:33 |
| *** ramishra_ has quit IRC | 10:35 | |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed opendev/irc-meetings master: Add Ansible SIG meeting https://review.opendev.org/697278 | 10:40 |
| zbr | mnaser: i am afraid that the answer is yes. | 10:47 |
| zbr | and that now is a broken one. | 10:47 |
| zbr | i supposed that https://opendev.org/openstack/diskimage-builder/src/branch/master/diskimage_builder/elements/pip-and-virtualenv/install.d/pip-and-virtualenv-source-install/04-install-pip is still used to build images. | 10:47 |
| frickler | mnaser: this is the matching excerpt from the latest ubuntu build http://paste.openstack.org/show/789419/ | 10:49 |
| frickler | this is the full log https://nb01.openstack.org/ubuntu-bionic-0000098344.log | 10:51 |
| openstackgerrit | Sorin Sbarnea proposed openstack/diskimage-builder master: WIP: Validate virtualenv and pip https://review.opendev.org/707104 | 10:58 |
| *** priteau has joined #openstack-infra | 11:03 | |
| *** ociuhandu has joined #openstack-infra | 11:04 | |
| *** dSrinivas has quit IRC | 11:05 | |
| zbr | it seems that outdated pypi mirrors is becoming a sure, 3rd time this week and I see somethink similar to ERROR: Could not find a version that satisfies the requirement mox3===1.0.0 | 11:05 |
| zbr | when the package exists on pypi. | 11:05 |
| zbr | https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_5a0/707104/1/check/openstack-tox-py27/5a05fed/job-output.txt | 11:05 |
| frickler | zbr: mox3 requires py>=36 | 11:08 |
| frickler | I verified on a held node that simply just downgrading virtualenv<20 fixes at least the initial issue for devstack | 11:08 |
| zbr | frickler: yep, but that job fails on https://review.opendev.org/#/c/707104/ change which has nothing to so with it. | 11:09 |
| *** ociuhandu has quit IRC | 11:09 | |
| *** ociuhandu has joined #openstack-infra | 11:13 | |
| frickler | zbr: yes, that needs a fix in reqs to make the u-c for mox3 split for py2 and py3. happened for a lot of other pkgs recently, like https://review.opendev.org/#/c/706165/ | 11:14 |
| zbr | frickler: pip knows to read metadata but i suspect our mirrors are plain text ones, breaking pip ability to find dependencies. | 11:15 |
| zbr | (with conditions) | 11:15 |
| frickler | zbr: with conditions, yes, but not if pip is given a constraints file telling it to install mox3==1.0.0 | 11:17 |
| *** ociuhandu has quit IRC | 11:18 | |
| zbr | frickler: https://review.opendev.org/#/c/707112/1 | 11:21 |
| *** rpittau is now known as rpittau|bbl | 11:21 | |
| frickler | zbr: actually https://review.opendev.org/#/c/707064/1/upper-constraints.txt already has the correct thing afaict, just needs merging. which in turn likely needs a fix for the virtualenv issue | 11:22 |
| zbr | tbh, the virtualenv===20.0.1 there gives me... it was released yesterday. | 11:24 |
| *** fungi has quit IRC | 11:24 | |
| openstackgerrit | Merged zuul/zuul master: JWT drivers: Deprecate RS256withJWKS, introduce OpenIDConnect https://review.opendev.org/701972 | 11:24 |
| *** fungi has joined #openstack-infra | 11:26 | |
| *** yamamoto has joined #openstack-infra | 11:41 | |
| *** yamamoto has quit IRC | 11:49 | |
| *** ociuhandu has joined #openstack-infra | 11:51 | |
| frickler | so the "ImportError: cannot import name ensure_text" comes from combining pip<10 with virtualenv>=20. it goes away when I upgrade both pip and virtualenv to latest. the workaround to downgrade in devstack seems to work, but will need to get into all older stable branches first | 11:53 |
| frickler | https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_20a/707109/1/check/devstack/20ac4f0/ | 11:53 |
| *** yamamoto has joined #openstack-infra | 11:55 | |
| *** ociuhandu has quit IRC | 11:57 | |
| *** Lucas_Gray has quit IRC | 11:59 | |
| *** Wasaac has quit IRC | 12:00 | |
| *** Wasaac has joined #openstack-infra | 12:00 | |
| *** yamamoto has quit IRC | 12:01 | |
| *** ociuhandu has joined #openstack-infra | 12:01 | |
| *** nicolasbock has joined #openstack-infra | 12:01 | |
| *** rfolco has joined #openstack-infra | 12:02 | |
| openstackgerrit | Jan Kubovy proposed zuul/zuul master: WIP: Store unparsed branch config in Zookeeper https://review.opendev.org/705716 | 12:03 |
| *** ricolin has quit IRC | 12:10 | |
| *** udesale_ has joined #openstack-infra | 12:11 | |
| *** Wasaac has quit IRC | 12:11 | |
| *** Wasaac has joined #openstack-infra | 12:12 | |
| *** udesale has quit IRC | 12:13 | |
| *** jaosorior has joined #openstack-infra | 12:15 | |
| *** gfidente has joined #openstack-infra | 12:21 | |
| *** dtantsur is now known as dtantsur|brb | 12:27 | |
| *** ociuhandu has quit IRC | 12:31 | |
| *** jaosorior has quit IRC | 12:32 | |
| *** yamamoto has joined #openstack-infra | 12:32 | |
| *** ociuhandu has joined #openstack-infra | 12:34 | |
| *** ykarel is now known as ykarel|afk | 12:34 | |
| *** dpawlik has joined #openstack-infra | 12:35 | |
| *** Lucas_Gray has joined #openstack-infra | 12:36 | |
| *** jpena is now known as jpena|lunch | 12:38 | |
| openstackgerrit | Sorin Sbarnea proposed openstack/diskimage-builder master: WIP: Validate virtualenv and pip https://review.opendev.org/707104 | 12:41 |
| *** rlandy has joined #openstack-infra | 12:51 | |
| *** adriancz has joined #openstack-infra | 13:00 | |
| *** yamamoto has quit IRC | 13:02 | |
| *** adriant has quit IRC | 13:02 | |
| *** adriant has joined #openstack-infra | 13:02 | |
| openstackgerrit | Sorin Sbarnea proposed openstack/diskimage-builder master: WIP: Validate virtualenv and pip https://review.opendev.org/707104 | 13:06 |
| *** rh-jelabarre has joined #openstack-infra | 13:07 | |
| *** takamatsu has quit IRC | 13:14 | |
| *** jamesmcarthur has joined #openstack-infra | 13:15 | |
| *** adriancz has quit IRC | 13:17 | |
| *** Lucas_Gray has quit IRC | 13:18 | |
| *** adriancz has joined #openstack-infra | 13:18 | |
| *** adriancz has quit IRC | 13:18 | |
| *** adriancz has joined #openstack-infra | 13:18 | |
| *** Lucas_Gray has joined #openstack-infra | 13:18 | |
| *** goldyfruit has joined #openstack-infra | 13:18 | |
| *** rpittau|bbl is now known as rpittau | 13:22 | |
| *** Lucas_Gray has quit IRC | 13:24 | |
| *** jamesmcarthur has quit IRC | 13:24 | |
| *** jamesmcarthur has joined #openstack-infra | 13:25 | |
| *** derekh has quit IRC | 13:27 | |
| *** jamesmcarthur has quit IRC | 13:28 | |
| *** jamesmcarthur has joined #openstack-infra | 13:28 | |
| *** jpena|lunch is now known as jpena | 13:32 | |
| *** jamesmcarthur has quit IRC | 13:35 | |
| *** takamatsu has joined #openstack-infra | 13:37 | |
| *** Lucas_Gray has joined #openstack-infra | 13:37 | |
| *** jamesmcarthur has joined #openstack-infra | 13:43 | |
| *** dtantsur|brb is now known as dtantsur | 13:49 | |
| *** Lucas_Gray has quit IRC | 13:50 | |
| *** Lucas_Gray has joined #openstack-infra | 13:52 | |
| *** sgw has quit IRC | 13:53 | |
| *** aaronsheffield has joined #openstack-infra | 13:56 | |
| *** tkajinam has joined #openstack-infra | 13:59 | |
| *** derekh has joined #openstack-infra | 14:01 | |
| *** ramishra has joined #openstack-infra | 14:04 | |
| *** ykarel|afk is now known as ykarel | 14:05 | |
| *** Goneri has joined #openstack-infra | 14:12 | |
| *** dSrinivas has joined #openstack-infra | 14:13 | |
| dSrinivas | Hi All, I am unable to attach the Nodepool vm to the jenkins dashboard http://paste.openstack.org/show/789424/ | 14:13 |
| mordred | frickler: did we wind up with images with old pip and new virtualenv? | 14:17 |
| mordred | zbr, fungi: the bindep role issue isn't an issue with --seeder=pip - it's a _different_ pip/virtualenv breakage | 14:21 |
| openstackgerrit | Merged zuul/zuul master: Uncap virtualenv https://review.opendev.org/706871 | 14:28 |
| *** takamatsu has quit IRC | 14:29 | |
| EmilienM | infra-root: could we please expediate https://review.opendev.org/#/c/707062 to the gate? It's the virtualenv issue, that affects all our jobs. It would help. Thanks. cc weshay | 14:30 |
| zbr | if would avoid using virtualenv 20+ until they fix it, its is breaking too many things. | 14:30 |
| weshay | thanks EmilienM | 14:30 |
| fungi | i don't have much expectation that the virtualenv maintainers are going to roll back these behaviors. they were intentional (re)design choices, and they published a beta of v20 weeks ago for anyone to try out | 14:32 |
| fungi | so temporarily pinning <20 is probably not a "until they fix it" thing | 14:33 |
| zbr | fungi: they published the beta, they received feedback including this bug and refused to fix it before the release. | 14:33 |
| fungi | and i have a feeling they'll continue to refuse to fix it if they don't consider it broken | 14:33 |
| zbr | yes, because they assumed virtualenv will not be installed with an older version of six. somehow we are breaking this requriments. | 14:34 |
| fungi | the beta seemed to me like more of a "here's the changes we're making, i hope you're ready" | 14:34 |
| dSrinivas | mordred, ianw: http://paste.openstack.org/show/789424/ | 14:34 |
| zbr | and what is sad is that issue with six was avoidable | 14:35 |
| mordred | wait - what's the _six_ issue? | 14:35 |
| mordred | jeez. ok | 14:36 |
| * mordred has read it | 14:37 | |
| zbr | mainly virtualenv>=20.0 chokes if run with six<1.14 | 14:37 |
| mordred | yeah | 14:38 |
| *** goldyfruit has quit IRC | 14:38 | |
| mordred | so - I'd like to repeat my long standing advice - NEVER mix distro installed python with pip installed python | 14:38 |
| *** goldyfruit has joined #openstack-infra | 14:38 | |
| mordred | it ALWAYS breaks | 14:38 |
| mordred | but I think that patch shoudl unbreak people for now | 14:40 |
| mordred | zbr: has anyone considered getting a tox package made for centos8? seems like, since so much else that's being done is done with packages that it would improve life for folks | 14:43 |
| zbr | mordred: :( | 14:43 |
| mordred | like epel has one for centos7- https://centos.pkgs.org/7/epel-x86_64/python-tox-1.4.2-8.el7.noarch.rpm.html | 14:44 |
| zbr | did you check the version? is useless. | 14:44 |
| zbr | 1.4.2, current is 3.9.x | 14:44 |
| zbr | almost everywhere I used it a have a min version of 3.4, 3.8, 3.9 in few cases. | 14:45 |
| zbr | anyway, today's issue is not really caused by tox. | 14:45 |
| fungi | it's not being caused by `pip install tox` dragging in a newer virtualenv than you want? | 14:47 |
| mordred | yeah - that's where the issue is coming from that I've seen | 14:47 |
| *** chkumar|rover is now known as raukadah | 14:50 | |
| *** ykarel is now known as ykarel|away | 14:51 | |
| mordred | fungi: could I beg reviews for https://review.opendev.org/#/c/705878 and parents? | 14:51 |
| *** sgw has joined #openstack-infra | 14:56 | |
| *** michael-beaver has joined #openstack-infra | 14:56 | |
| *** ociuhandu has quit IRC | 14:57 | |
| *** elod has quit IRC | 14:57 | |
| *** elod has joined #openstack-infra | 14:58 | |
| *** pkopec_ is now known as pkopec | 14:59 | |
| openstackgerrit | Monty Taylor proposed zuul/zuul master: Protect fail if unsafe from None value https://review.opendev.org/707167 | 15:00 |
| *** artom has joined #openstack-infra | 15:04 | |
| *** ysastri has joined #openstack-infra | 15:06 | |
| ysastri | Hello I am getting ERROR: not a btrfs filesystem: /var/lib/lxc in zuul pipeline check. I tried two times and it is same. Any idea?? | 15:07 |
| *** artom has quit IRC | 15:09 | |
| *** zxiiro has joined #openstack-infra | 15:10 | |
| fungi | mordred: that chains back up to 705670 where i had a question last week. sorry i should have probably left a -1 for improved visibility | 15:10 |
| mordred | fungi: oh - sorry - I didn't see that | 15:10 |
| *** takamatsu has joined #openstack-infra | 15:11 | |
| mordred | fungi: it's a good question! yes - we probably should do that | 15:11 |
| mordred | I'll start working on that in just a smidge, thanks | 15:11 |
| *** artom has joined #openstack-infra | 15:12 | |
| *** artom has quit IRC | 15:12 | |
| fungi | np. i'll start looking through the rest | 15:12 |
| *** artom has joined #openstack-infra | 15:13 | |
| *** sreejithp has joined #openstack-infra | 15:21 | |
| mordred | fungi: I think, even though it's not strictly required, I'll get a review-dev.openstack.org too and do the redirect, so that we can just put the redirect vhost in for both | 15:22 |
| *** Lucas_Gray has quit IRC | 15:23 | |
| *** jamesmcarthur has quit IRC | 15:23 | |
| *** Lucas_Gray has joined #openstack-infra | 15:25 | |
| *** jamesmcarthur has joined #openstack-infra | 15:26 | |
| fungi | that's probably good to help test the redirect config, yeah | 15:27 |
| *** gfidente has quit IRC | 15:27 | |
| clarkb | ysastri: it helps to link to logs | 15:27 |
| *** ociuhandu has joined #openstack-infra | 15:28 | |
| *** tkajinam has quit IRC | 15:29 | |
| clarkb | ysastri: I am not familiar with that error though | 15:30 |
| *** armax has joined #openstack-infra | 15:30 | |
| *** gfidente has joined #openstack-infra | 15:32 | |
| corvus | i've asked openstack-release to hold off on releases so i can restart zuul; expecting to do that in ~20 minutes | 15:37 |
| *** pgaxatte has quit IRC | 15:38 | |
| *** jamesmcarthur has quit IRC | 15:38 | |
| mordred | fungi: LE dns needs on openstack.org are simply the _acme-challenge record? | 15:39 |
| mordred | (the cname to acme.opendev.org) | 15:39 |
| ysastri | clarkb I am getting the same error in all the zuul checks https://zuul.opendev.org/t/openstack/build/c4e14eba89244eb298831ca9f92c9ce1 | 15:42 |
| *** cshen has joined #openstack-infra | 15:45 | |
| *** jamesmcarthur has joined #openstack-infra | 15:46 | |
| corvus | infra-root: i'm going to restart zuul now | 15:50 |
| clarkb | mordred: yes | 15:51 |
| openstackgerrit | Monty Taylor proposed zuul/zuul master: Run yarn explicitly in Dockerfile https://review.opendev.org/707182 | 15:52 |
| *** udesale_ has quit IRC | 15:52 | |
| *** zbr_ has joined #openstack-infra | 15:53 | |
| *** zbr_ has quit IRC | 15:53 | |
| *** udesale_ has joined #openstack-infra | 15:53 | |
| openstackgerrit | Monty Taylor proposed zuul/zuul master: Run yarn explicitly in Dockerfile https://review.opendev.org/707182 | 15:54 |
| *** ociuhandu has quit IRC | 15:54 | |
| frickler | mordred: I didn't check, but I think the images also have fresh pip, but devstack in stable branches still caps pip<10. and the uncap isn't really backportable | 15:55 |
| *** cshen has quit IRC | 15:55 | |
| mordred | uhm | 15:56 |
| *** KeithMnemonic has quit IRC | 15:56 | |
| mordred | we're capping pip in stable branches? I guess I shouldn't ask | 15:56 |
| mordred | if we're capping pip in stable devstack then we probably need to cap virtualenv in devstack in stable branches | 15:57 |
| clarkb | mordred: its due to pip refusing to install packages if it can't unistnall the existing distutils package versions | 15:57 |
| mordred | *joy* | 15:57 |
| clarkb | and yes capping virtualenv in stable devstack is probably simpler than trying to unwind all of the deps that don't like newer pip | 15:57 |
| mordred | yah | 15:57 |
| mordred | I think for master ensuring everything is new enough is the better road forward | 15:58 |
| frickler | https://review.opendev.org/707109 is my attempt to fix for stable/rocky, probably needs to cycle through all branches | 15:59 |
| clarkb | ysastri: https://zuul.opendev.org/t/openstack/build/c4e14eba89244eb298831ca9f92c9ce1/log/job-output.txt#14993 is that the actual error? | 16:00 |
| frickler | hmm, interesting comment in https://review.opendev.org/#/c/707133/1/tools/fixup_stuff.sh , maybe we can bump six a bit instead? | 16:00 |
| *** lpetrut has quit IRC | 16:01 | |
| mordred | frickler: yeah - I think bumping six is probably the more surgical thing to do | 16:01 |
| mordred | bumping pip is actually just a shotgun approach which happens to cause six to get upgraded | 16:02 |
| *** Lucas_Gray has quit IRC | 16:02 | |
| openstackgerrit | Federico Ressi proposed zuul/zuul-jobs master: Allow to force bindep installation https://review.opendev.org/707185 | 16:04 |
| ysastri | clarkb: yes you are right that is the actual error. | 16:05 |
| *** udesale_ has quit IRC | 16:06 | |
| *** tommylikehu_ has quit IRC | 16:06 | |
| *** udesale_ has joined #openstack-infra | 16:07 | |
| mordred | #status log added LE records for review-dev.o.o and review.o.o to rax dns and made review-dev.o.o a cname to review-dev.opendev.org | 16:08 |
| openstackstatus | mordred: finished logging | 16:08 |
| mordred | fungi, clarkb: https://review.opendev.org/#/c/705673/ tiny readme update | 16:09 |
| *** ociuhandu has joined #openstack-infra | 16:11 | |
| *** rf0lc0 has joined #openstack-infra | 16:11 | |
| *** ociuhandu has quit IRC | 16:12 | |
| *** ociuhandu has joined #openstack-infra | 16:12 | |
| *** rfolco has quit IRC | 16:13 | |
| frickler | mordred: right, seems we would need to increase the cap for six in reqs in order for that to work, I have asked the requirements team for feedback on that | 16:15 |
| frickler | but I verified on my help node that pip==9.0.3 with six==1.12.0 and virtualenv==20.0.1 work fine | 16:16 |
| frickler | held* | 16:16 |
| *** udesale_ has quit IRC | 16:16 | |
| frickler | btw., there's a couple of pretty ancient nodes being held for/by corvus and efried, please check whether you still need those | 16:17 |
| fungi | noonedeadpunk: did anyone help you yet with that hold you were pinging me about at 10:11z? | 16:19 |
| fungi | that just reminded me | 16:19 |
| frickler | oh, seems I missed that, sorry | 16:20 |
| openstackgerrit | Tristan Cacqueray proposed zuul/zuul master: kubernetes-operator: change attribute to camelCase https://review.opendev.org/707190 | 16:21 |
| frickler | mordred: I also just checked that stable/stein has six==1.12.0 in u-c, so the issue really only affects <= rocky | 16:21 |
| noonedeadpunk | fungi: not really :p | 16:22 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Get LE certs for review.o.o https://review.opendev.org/705670 | 16:22 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Remove review-dev01.openstack.org https://review.opendev.org/705671 | 16:22 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Use LE certs for Apache https://review.opendev.org/705690 | 16:22 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Update letsencrypt_gid documentation https://review.opendev.org/705878 | 16:22 |
| fungi | noonedeadpunk: i'm in a meeting but should be able to get it set up, just a minute | 16:22 |
| openstackgerrit | Tobias Henkel proposed zuul/zuul master: Support pausing merge jobs https://review.opendev.org/707192 | 16:22 |
| noonedeadpunk | yeah. no hurry | 16:22 |
| mordred | fungi: ^^ that should add the redirects and all of the needed LE certs | 16:23 |
| openstackgerrit | Tristan Cacqueray proposed zuul/zuul-operator master: Update attributes to camelCase https://review.opendev.org/707193 | 16:24 |
| openstackgerrit | Iury Gregory Melo Ferreira proposed openstack/diskimage-builder master: Add efi packages for ironic-agent https://review.opendev.org/707195 | 16:27 |
| fungi | noonedeadpunk: sudo zuul autohold --tenant openstack --project openstack/openstack-ansible-os_manila --job openstack-ansible-deploy-aio_metal-ubuntu-bionic --change 705222 --reason "noonedeadpunk investigating missing ceph temporary directory" --count 1 | 16:27 |
| fungi | recheck at will | 16:27 |
| noonedeadpunk | fungi: thank you! | 16:28 |
| fungi | yw! | 16:28 |
| mnasiadka | frickler: we're getting these errors in centos7 kolla-ansible master jobs - https://zuul.opendev.org/t/openstack/build/acae2fb98da545b8a0ade1e4c5070c1f/log/job-output.txt#347 - seems to me like six is not in proper version in bindep venv? | 16:32 |
| *** gyee has joined #openstack-infra | 16:32 | |
| clarkb | mnasiadka: that isn't six | 16:33 |
| clarkb | mnasiadka: that is the pkg_resources issue, which I'm surprised hits the bindep env | 16:33 |
| clarkb | permissions issue there maybe rather than broken symlinks | 16:34 |
| *** dtantsur is now known as dtantsur|afk | 16:34 | |
| mnasiadka | clarkb: ok, surprised or not - we would be interested in getting this fixed :) | 16:34 |
| clarkb | sure, I just wanted to clarify it isn't six (you can clearly see it failed on import pkg_resources) | 16:34 |
| mnasiadka | ah right, looked at too many similar errors today - thanks for correcting :) | 16:35 |
| corvus | #status log restarted all of zuul at 3.16.0 | 16:36 |
| openstackstatus | corvus: finished logging | 16:36 |
| clarkb | yes confirmed that is the issue | 16:37 |
| clarkb | because dr-xr-x---. 5 root root 4096 Feb 11 16:31 root | 16:37 |
| clarkb | as root I am able to import pkg_resources but not as the zuul user | 16:37 |
| frickler | clarkb: is that 750 for /usr/bindep-env ? or which dir? | 16:39 |
| mordred | for /root | 16:39 |
| mnasiadka | ok, interesting that it showed up when new virtualenv version broke a lot of things here and there ;-) | 16:39 |
| mordred | because root created the bindep-env and new virtualenv symlinks things to ~/ | 16:39 |
| clarkb | frickler: mordred is correct | 16:39 |
| mordred | mnasiadka: new virtualenv broke many things | 16:39 |
| clarkb | its /root then the libs are in /root/.local | 16:39 |
| frickler | ah, fancy | 16:39 |
| mordred | yeah. "awesome" | 16:40 |
| clarkb | I'm working on a patch to use --seeder=pip | 16:40 |
| mordred | ++ | 16:40 |
| smcginnis | Not sure if it is new since the restart or if I just noticed it, but from the zuul dashboard, there are now warning alerts in the top right corner. | 16:40 |
| clarkb | its funny the top reason to use virtualenv over venv is that venv is slower because it doesn't use this app-data thing that breaks everyone :) | 16:40 |
| mordred | smcginnis: I agree - there are warnings there | 16:40 |
| smcginnis | A bunch of "'NoneType' object is not iterable" notifications. | 16:40 |
| mordred | yah | 16:41 |
| mordred | corvus: ^^ | 16:41 |
| mordred | seems to all be for external repos we're tracking | 16:41 |
| smcginnis | For your convenience and pleasure - https://zuul.opendev.org/t/openstack/config-errors | 16:41 |
| corvus | indeed | 16:41 |
| openstackgerrit | Clark Boylan proposed openstack/project-config master: Use virtualenv --seeder=pip so that libs are accessible https://review.opendev.org/707199 | 16:43 |
| clarkb | infra-root mnasiadka ^ that and an image rebuild should fix the problem I Think | 16:43 |
| mordred | clarkb: should we add a $something to base to unbreak things while waiting for that? | 16:43 |
| clarkb | mordred: I'm not sure what that would be, possibly reinstalling the venv at runtime? | 16:44 |
| mordred | clarkb: we could chmod -R o+rx /root to make the files accessible | 16:44 |
| clarkb | oh ya that would probably do it too | 16:45 |
| fungi | clarkb: https://github.com/pypa/virtualenv/issues/1563 suggests that the performance gains are maybe only on ms windows | 16:45 |
| mordred | SO HAPPY | 16:45 |
| clarkb | "also --copies does not do copies:" | 16:45 |
| fungi | in my opinion, virtualenv stopped being for linux users as soon as they're able to ditch python 2.7 | 16:46 |
| clarkb | I've decided I have to laugh at that rather than cry :) | 16:46 |
| fungi | i switched my python3-only projects to use the tox-venv plugin and never looked back | 16:46 |
| *** jpena is now known as jpena|brb | 16:46 | |
| fungi | i've approved 707199 but we'll obviously need to build new images once that lands | 16:47 |
| *** slaweq has joined #openstack-infra | 16:47 | |
| fungi | and i need to go run some lunch errands but will be back as soon as i can (well before the infra meeting for sure) | 16:47 |
| clarkb | fungi: I think if we delete the newest set that will retrigger rebuilds and hopefully fall back to good image in the interim | 16:47 |
| fungi | i concur | 16:48 |
| frickler | can we just revert to older images in the meantime? | 16:48 |
| *** Wasaac has quit IRC | 16:48 | |
| fungi | frickler: we can as soon as that change merges | 16:48 |
| clarkb | ya nodepool makes a proper revert tricky now that it checks to build images constantly | 16:48 |
| fungi | deleting the newest images will fall back to the previous ones *and* begin building the newer fixed ones | 16:48 |
| clarkb | to avoid ^ we could set all the image to paused | 16:48 |
| *** Wasaac has joined #openstack-infra | 16:49 | |
| fungi | also possible | 16:49 |
| clarkb | but that will take as long as it does to land the fix | 16:49 |
| fungi | anyway, heading out, will try to be quick | 16:49 |
| clarkb | (either via normal channels or hold in ansible then manually land) | 16:49 |
| openstackgerrit | Monty Taylor proposed opendev/base-jobs master: Workaround root virtualenv break with v20 https://review.opendev.org/707200 | 16:51 |
| mordred | clarkb, fungi: something like that ^^ ? | 16:51 |
| mordred | of course, doing the base-dev dance would also take longer than just landing the fix | 16:52 |
| clarkb | does X mean only on dirs? | 16:52 |
| clarkb | mordred: I think that will work and take effect as soon as it lands | 16:52 |
| clarkb | frickler: ^ I think that may be our best bet for quick fix | 16:52 |
| mordred | clarkb: yes - X means only on dirs | 16:53 |
| clarkb | then we can revert as soon as new images are uploaded | 16:53 |
| mordred | my biggest concern is that testing that is likely to be very time consuming - but if we're comfortable with it like it is | 16:53 |
| mordred | I guess worst case we force-revert | 16:53 |
| clarkb | I mean its basically broken already | 16:54 |
| openstackgerrit | Monty Taylor proposed opendev/base-jobs master: Revert "Workaround root virtualenv break with v20" https://review.opendev.org/707202 | 16:54 |
| mordred | clarkb: it's ony broken for people on centos | 16:54 |
| clarkb | I guess the difference is it is only broken for jobs that use bindep but if this is broken it will break for everything | 16:54 |
| clarkb | mordred: until ubuntu images build :) | 16:54 |
| clarkb | mordred: I think its largely a race against dib right now | 16:54 |
| mordred | I don't think ubuntu sets 770 on /root | 16:54 |
| mnasiadka | mordred: on centos7 to be precise, centos8 works fine. | 16:54 |
| clarkb | oh | 16:54 |
| clarkb | got it | 16:54 |
| jrosser | what is only broken on centos? | 16:55 |
| mordred | which is to say - it's not like the entire system is broken - even though there is an immediate breakage we're working on fixing | 16:55 |
| clarkb | jrosser: permissions on /root prevent non root users from accessing packages in root installed virtualenvs | 16:55 |
| clarkb | jrosser: this is new behavior with virtualenv 20.0.0 release beacuse it by default doesn't install packages in the venv | 16:56 |
| mordred | jrosser: using the bindep venv created by root with the new symlink behavior in virtualenv >20 | 16:56 |
| clarkb | (well not all packages) | 16:56 |
| openstackgerrit | Tobias Henkel proposed zuul/zuul master: Revert "Fix github app authentication to work with checks API endpoints" https://review.opendev.org/707205 | 16:56 |
| mordred | it's an "optimization" | 16:56 |
| frickler | hmm, I'm not too fond of fiddling with /root permissions, but I need to have a break and get some food anyways | 16:56 |
| corvus | infra-root, smcginnis: we think we tracked down the bug in zuul. it should only be affecting the github projects that we run in third-party-check. the result is that zuul is essentially ignoring them. i'd like to restart zuul again to confirm the fix so we can issue a new release. is that ok? (should be faster this time) | 16:56 |
| clarkb | corvus: are we just restarting the scheduler this time? | 16:56 |
| clarkb | (to understand why it would be faster) | 16:57 |
| corvus | clarkb: yes | 16:57 |
| smcginnis | corvus: We have a few releases in queue again. Is it safe just restarting the scheduler, or should we hold off on things and wait for the queue to drain? | 16:57 |
| clarkb | I think that is ok, a lot of stuff is on fire today and we don't have proper fixes for the virtualenv stuff in all places. However we probably want https://review.opendev.org/707199 to land sooner than later | 16:57 |
| clarkb | corvus: if you restart can you enqueue ^ to the gate maybe? | 16:57 |
| corvus | smcginnis: we should wait for it to drain | 16:57 |
| jrosser | ok i was just clarifying if it was the permissions issue or virtualenv 20 which was specific to centos | 16:58 |
| smcginnis | corvus: Ack, will hold off and make sure the others now in -releases | 16:58 |
| corvus | clarkb: i suggest you go aheand and enqueue that now while we wait for releases to finish | 16:58 |
| clarkb | corvus: ok should I do that or will you? | 16:59 |
| corvus | clarkb: you please | 16:59 |
| *** rpittau is now known as rpittau|afk | 17:00 | |
| *** ysastri has quit IRC | 17:00 | |
| *** lmiccini has quit IRC | 17:00 | |
| clarkb | done | 17:00 |
| *** priteau has quit IRC | 17:00 | |
| *** lucasagomes has quit IRC | 17:02 | |
| clarkb | as a heads up I'll probably need to pick up the kids from school today which conflicts with the end of our meeting. I can either end the meeting about 10-15 minutes early or someone can end meeting for me once the timer runs out (or I can figure out how to chair someone else) | 17:04 |
| *** dave-mccowan has joined #openstack-infra | 17:10 | |
| mordred | clarkb, fungi, frickler: https://zuul.opendev.org/t/openstack/build/53073321d769465dbd16baf72a3c3e08/log/applytest/puppetapplytest21.final.out.FAILED#112 \o/ | 17:10 |
| clarkb | mordred: oh that one is gonna hurt probably | 17:11 |
| clarkb | mordred: its beacuse virtualenv --version format output changed | 17:11 |
| clarkb | you'd think the one thing you could keep stable aws that | 17:11 |
| clarkb | so that things that detect versions would you know work | 17:12 |
| mordred | clarkb: I don't even see where we're trying to use that | 17:12 |
| mordred | I think it's deep in puppet | 17:12 |
| clarkb | mordred: I think there is a puppet virtualenv module and the way facts work is they are installed then all evaluated before running puppet | 17:12 |
| clarkb | mordred: step zero might be checking where we use that module and possibly we can delete it | 17:13 |
| clarkb | mordred: /etc/puppet/modules/python/lib/facter/virtualenv_version.rb | 17:13 |
| clarkb | and ya it appears to fail to parse because output changed (its got a regex there) | 17:14 |
| mordred | clarkb: we use it in 2 places | 17:15 |
| openstackgerrit | Julia Kreger proposed openstack/project-config master: retire coreos-image-builder https://review.opendev.org/707210 | 17:15 |
| clarkb | mordred: the "good" news is that puppet module hasn't been updated to fix this issue | 17:16 |
| clarkb | but even if it had it appears a release about 6 months ago dropped puppet 4 support | 17:16 |
| clarkb | we could fork it and modify the regex pretty easily likely as another option | 17:16 |
| *** michael-beaver has quit IRC | 17:16 | |
| clarkb | https://github.com/voxpupuli/puppet-python is the repo | 17:16 |
| mordred | yeah. or - just stop using python::virtualen | 17:16 |
| mordred | virtualenv | 17:16 |
| clarkb | ya | 17:16 |
| clarkb | well I think we have to drop the module entirely | 17:17 |
| openstackgerrit | Tobias Henkel proposed zuul/zuul master: Revert "Fix github app authentication to work with checks API endpoints" https://review.opendev.org/707205 | 17:17 |
| clarkb | because even if you don't use python::virtualenv it will try to evaluate the fact | 17:17 |
| mordred | I think the facter virtualenv_version only breaks us where we're using it | 17:17 |
| mordred | well - except we only see one failure in the puppet job | 17:17 |
| mordred | gah - in the one place where we use it | 17:17 |
| clarkb | mordred: oh it fails to create the venv version fact but then its not an error until we try and use the var | 17:17 |
| clarkb | that makes sense | 17:17 |
| mordred | so I don't think it's polluting anything else | 17:17 |
| mordred | yeah | 17:17 |
| mordred | so let me just update system-config to remove that use ot virtualenv | 17:18 |
| clarkb | I'm going to be a good user and file a bug with upstream | 17:18 |
| clarkb | then if they fix it we can always grab that diff | 17:18 |
| clarkb | (but I think I'm good with just not using that at all) | 17:18 |
| mordred | we only use this in two places, I'm more inclined to just accelerate our move off :) | 17:18 |
| clarkb | ++ | 17:18 |
| *** gmann is now known as gmann_afk | 17:20 | |
| clarkb | you know what kills a user happy to report a bug? a bug template that wants to know everything about you | 17:20 |
| corvus | clarkb: don't tell em anything you don't want to :) | 17:21 |
| openstackgerrit | Merged openstack/project-config master: Use virtualenv --seeder=pip so that libs are accessible https://review.opendev.org/707199 | 17:21 |
| ykarel|away | so now with ^^, when new nodepool images will be available? | 17:22 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Make small tweaks to launch node README https://review.opendev.org/705673 | 17:22 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Stop using python::virtualenv for virtualenvs https://review.opendev.org/707214 | 17:23 |
| mordred | clarkb: how does 707214 look? also - 705673 should tell us if it works | 17:23 |
| ykarel|away | mordred, any idea ^^? | 17:24 |
| clarkb | ykarel|away: probably several hours at least | 17:24 |
| clarkb | ykarel|away: it takes about an hour to build an image then variable time to upload it | 17:24 |
| *** ccamacho has joined #openstack-infra | 17:24 | |
| mordred | clarkb: should we risk the workaround? | 17:24 |
| *** igordc has joined #openstack-infra | 17:24 | |
| ykarel|away | clarkb, ohhk, i thought it just need to trigger image build and it will be available just after it's built | 17:25 |
| *** jpena|brb is now known as jpena | 17:25 | |
| clarkb | mordred: one small thing in 214 | 17:25 |
| mordred | ZOMG | 17:25 |
| clarkb | mordred: hrm? | 17:25 |
| mordred | can we just disable the check to see if there is an ansible module? | 17:25 |
| mordred | playbooks/base/pre.yaml:7: [E302] chmod used in place of argument mode to file module | 17:25 |
| mordred | I DO NOT CARE | 17:25 |
| *** ociuhandu_ has joined #openstack-infra | 17:26 | |
| clarkb | mordred: I'd be in favor though sometimes it is nice to know ansible can help | 17:26 |
| clarkb | ykarel|away: no we have to upload the image to all of our clouds too | 17:26 |
| mordred | clarkb: how do I disable it for that line again? | 17:27 |
| ykarel|away | clarkb, ack Thanks for the info, and there are plenty of cloud, so much time makes sens | 17:27 |
| *** kopecmartin is now known as kopecmartin|afk | 17:27 | |
| *** ramishra has quit IRC | 17:28 | |
| openstackgerrit | Monty Taylor proposed opendev/base-jobs master: Workaround root virtualenv break with v20 https://review.opendev.org/707200 | 17:29 |
| openstackgerrit | Monty Taylor proposed opendev/base-jobs master: Revert "Workaround root virtualenv break with v20" https://review.opendev.org/707202 | 17:29 |
| *** mattw4 has joined #openstack-infra | 17:29 | |
| mordred | clarkb: added a noqa ^^ | 17:29 |
| clarkb | mordred: I never remember | 17:29 |
| *** ociuhandu has quit IRC | 17:29 | |
| clarkb | mordred: will the # be evaluated in yaml or shell context there | 17:29 |
| mordred | yeah - lemme do a different thig | 17:30 |
| clarkb | https://github.com/voxpupuli/puppet-python/issues/534 submitted | 17:30 |
| *** ociuhandu_ has quit IRC | 17:30 | |
| openstackgerrit | Monty Taylor proposed opendev/base-jobs master: Workaround root virtualenv break with v20 https://review.opendev.org/707200 | 17:30 |
| openstackgerrit | Monty Taylor proposed opendev/base-jobs master: Revert "Workaround root virtualenv break with v20" https://review.opendev.org/707202 | 17:31 |
| clarkb | mordred: oh I see a bug with that | 17:31 |
| clarkb | mordred: that play runs on localhost but we want it to run on all | 17:31 |
| corvus | clarkb, mordred: do we put any private keys in /root (like the ones we generate for multinode jobs), or are they all in ~zuul? | 17:33 |
| *** evrardjp has quit IRC | 17:34 | |
| corvus | (i'm wondering if the chmod will trip openssh's key permission check) | 17:34 |
| *** evrardjp has joined #openstack-infra | 17:34 | |
| clarkb | corvus: good question. I think the old school d-g jobs may stash ssh keys there | 17:34 |
| clarkb | because live migration happens over a root ssh connection? | 17:34 |
| clarkb | we could chmod /root and then recursively on /root/.local | 17:35 |
| clarkb | to avoid changing /root/.ssh | 17:35 |
| corvus | that might be a safer/more targeted fix | 17:35 |
| corvus | i gotta say, i'm not a fan of the new virtualenv behavior | 17:36 |
| corvus | this one in particular seems like it's likely to hit a lot of folks :/ | 17:36 |
| mordred | yeah. I even replied to one of the bugs on the virtualenv repo in github | 17:36 |
| corvus | the zuul break was like "okay, sure, we're experts doing advanced things" but "installed as root and made available to users" is, er, not exotic. | 17:37 |
| clarkb | looks like it broke travis ci too fwiw | 17:38 |
| clarkb | so ya not an uncommon setup at least in CI systems | 17:38 |
| openstackgerrit | Monty Taylor proposed opendev/base-jobs master: Workaround root virtualenv break with v20 https://review.opendev.org/707200 | 17:38 |
| corvus | i would argue s/ci// | 17:38 |
| mordred | clarkb corvus: ^^ how's that? | 17:38 |
| mordred | yeah - we install tools in venvs on production servers | 17:38 |
| mordred | that are intended to be used as shared tools | 17:38 |
| mordred | it's not a ludicrous pattern only for CI | 17:39 |
| corvus | the releases have cleared out, i'm going to restart the scheduler now | 17:39 |
| clarkb | mordred: that ps lgtm | 17:39 |
| *** jamesmcarthur has quit IRC | 17:40 | |
| clarkb | oh looks like they are changing the behavior in the next release | 17:41 |
| clarkb | https://github.com/pypa/virtualenv/pull/1578 | 17:41 |
| clarkb | it will still use the bundled data by default but copy it raather than symlink it | 17:41 |
| clarkb | if we wait long enough this will just fix it self :) | 17:41 |
| clarkb | (I have no idea what their release plans are) | 17:42 |
| *** ccamacho has quit IRC | 17:42 | |
| clarkb | mordred: did you see my note on https://review.opendev.org/707214 | 17:45 |
| clarkb | mordred: I can push a new ps if you like | 17:45 |
| corvus | smcginnis: errors are gone now, thanks! you should be clear to resume releases | 17:47 |
| corvus | #status log restarted zuul scheduler at 3.16.0 with https://review.opendev.org/707205 manually applied | 17:48 |
| mordred | clarkb: oh, I can get that | 17:48 |
| openstackstatus | corvus: finished logging | 17:48 |
| mordred | clarkb: \o/ | 17:48 |
| smcginnis | corvus: ack | 17:48 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Stop using python::virtualenv for virtualenvs https://review.opendev.org/707214 | 17:50 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Make small tweaks to launch node README https://review.opendev.org/705673 | 17:50 |
| clarkb | mordred: now that zuul is back up we should decide if we want to land https://review.opendev.org/#/c/707200/4 I think I'm ok with it and reverting it if it causes widespread panic is easy | 17:51 |
| clarkb | I need to find some food and drink. I'll be back in a few minutes to help monitor ^ if we land it | 17:55 |
| *** derekh has quit IRC | 17:58 | |
| mordred | clarkb, corvus: I'mma land that patch - I'm here to watch for carnage | 18:00 |
| openstackgerrit | Monty Taylor proposed opendev/base-jobs master: Revert "Revert "Workaround root virtualenv break with v20"" https://review.opendev.org/707202 | 18:01 |
| *** igordc has quit IRC | 18:02 | |
| openstackgerrit | Monty Taylor proposed opendev/base-jobs master: Revert "Workaround root virtualenv break with v20" https://review.opendev.org/707202 | 18:02 |
| *** igordc has joined #openstack-infra | 18:06 | |
| *** jackedin has joined #openstack-infra | 18:08 | |
| fungi | i'm back, looks like i conveniently missed some excitement | 18:12 |
| *** xek_ has joined #openstack-infra | 18:12 | |
| mordred | fungi: we're landing a base-job change with no real testing - so there might still be excitement | 18:13 |
| openstackgerrit | Merged opendev/base-jobs master: Workaround root virtualenv break with v20 https://review.opendev.org/707200 | 18:13 |
| mnaser | fungi: ^ looks like you're here just in time. | 18:13 |
| *** xek has quit IRC | 18:15 | |
| *** Ramereth has quit IRC | 18:17 | |
| fungi | woo! | 18:18 |
| fungi | i'll put on my plastic mac | 18:19 |
| mordred | ykarel|away, weshay: ^^ we just landed a workaround for the virtualenv on centos issue | 18:19 |
| ykarel|away | mordred, Thanks | 18:19 |
| mordred | mnasiadka: ^^ you too | 18:19 |
| mnasiadka | mordred: was tracking, but thanks for update :) | 18:20 |
| mordred | ykarel|away: it went in without testing, because of where it sits in the stack, so it COULD cause the world to blow up | 18:20 |
| mordred | but fingers crossed | 18:20 |
| fungi | itym "...blow up MORE" | 18:20 |
| openstackgerrit | Monty Taylor proposed zuul/zuul master: Run yarn explicitly in Dockerfile https://review.opendev.org/707182 | 18:21 |
| clarkb | fungi: earlier at least :) | 18:21 |
| clarkb | I've got food and tea now | 18:22 |
| fungi | in proper esr fashion, we blow up early and often | 18:22 |
| ykarel|away | mordred, ack /me rechecked a patch where it will be tested https://review.opendev.org/#/c/707127/ | 18:22 |
| ykarel|away | but yes different jobs will be using it differently, so let's see | 18:23 |
| weshay | o hoo ray :) | 18:25 |
| *** Ramereth has joined #openstack-infra | 18:26 | |
| *** armax has quit IRC | 18:32 | |
| *** ralonsoh has quit IRC | 18:33 | |
| clarkb | /etc/project-config is updated on nb01 and nb02 so I'll trigger a centos 7 rebuild now | 18:35 |
| ykarel|away | mordred, it worked for the job i rechecked | 18:36 |
| mordred | ykarel|away: YAY! | 18:36 |
| *** jpena is now known as jpena|off | 18:36 | |
| mordred | corvus: zomg. we landed a change without testing and it didn't break! | 18:36 |
| clarkb | if other images are obvserved to have the same pkg_resources problem for the bindep venv we can do that for them too otherwise they'll get update on the normal 24 hour cycle | 18:36 |
| clarkb | mordred: is that a first?! | 18:37 |
| mordred | clarkb: I think so! | 18:38 |
| mordred | clarkb: we wont' see the problem on other images because of the workaround | 18:38 |
| clarkb | oh right | 18:39 |
| clarkb | well in that case maybe trigger all the images to rebuild? | 18:39 |
| mordred | clarkb: so really I think it's about waiting for new images - then we can revert - although maybe we should go through a whole base-test cycle on the revert | 18:39 |
| clarkb | then we can revert more quickly? | 18:39 |
| mordred | clarkb: yeah - might not be a bad idea | 18:39 |
| clarkb | ok I'll go ahead and trigger those builds now | 18:40 |
| mordred | clarkb: "Hello, a fix for this issue has been released via virtualenv 20.0.2" | 18:41 |
| clarkb | mordred: oh did that release happen? | 18:41 |
| mordred | clarkb: maybe we should see if the new behavior works for us without seeder | 18:41 |
| mordred | clarkb: I *just* got notificaiton of it | 18:42 |
| clarkb | ya I can test that locally really quickly | 18:42 |
| mordred | cool | 18:42 |
| clarkb | all images but fedora-30 and the arm64 images have rebuilds queued now | 18:42 |
| mordred | I think the new thing is that it's still going to do the ~/.local thing - but it's going to copy not symlink when it makes the venv | 18:42 |
| mordred | so that it's not using pip to download each time | 18:42 |
| clarkb | mordred: ya | 18:42 |
| mordred | which is a behavior i can get bheind | 18:42 |
| clarkb | also if they fix that we can simplify zuul too | 18:42 |
| clarkb | do a virtualenv !=20.0.0,!=20.0.1 and use old or new virtuaelvn | 18:43 |
| mordred | ++ | 18:43 |
| clarkb | mordred: https://review.opendev.org/#/c/706958/ is an easy one unrelated to anything venv if you have a minute | 18:43 |
| fungi | worth noting though, pip also maintains local caches (a file cache and a url cache) so wouldn't likely download again anyway | 18:44 |
| ykarel|away | mordred, chmod: cannot access '/root/.local': No such file or directory can be seen if image don't have that directory | 18:45 |
| clarkb | confirmed that there are no symlinks in venv/lib/pythonX/site-packages/ | 18:45 |
| mordred | ykarel|away: damn | 18:45 |
| ykarel|away | i tried in one of our image where it doesn't exist | 18:45 |
| clarkb | using virtualenv==20.0.2 | 18:45 |
| mordred | clarkb: I thnik we need to quick land the revert | 18:45 |
| mordred | oh wait - all of the intra images will have that | 18:46 |
| clarkb | mordred: as long as we have updated | 18:46 |
| clarkb | lets check nodepool iamge-list | 18:46 |
| mordred | clarkb: I'm getting a bunch of rechecks on system-config patches | 18:46 |
| clarkb | mordred: fedora-30 (a paused image) and opensuse-* are older but everything else appears to have updated in the last <24 hours | 18:48 |
| clarkb | I expect we are mostly ok as a result | 18:48 |
| *** gmann_afk is now known as gmann | 18:49 | |
| ykarel|away | hmm 20.0.2 is working | 18:49 |
| clarkb | new centos-7 is building | 18:50 |
| clarkb | ykarel|away: yes, they switched the default to copy not symlink | 18:50 |
| clarkb | and that just appeared like 5 minutes ago | 18:50 |
| ykarel|away | hmm i see created virtual environment CPython2Posix(dest=/home/centos/ven5, clear=False, global=False) with seeder FromAppData pip=latest setuptools=latest wheel=latest app_data_dir=/root/.local/share/virtualenv/seed-v1 via=copy while creating venv with 20.0.2 | 18:50 |
| clarkb | ykarel|away: yes it is more verbose too apparently | 18:50 |
| ykarel|away | yup, so we are good then | 18:51 |
| clarkb | I'm guessing that is to help bug reports | 18:51 |
| clarkb | ykarel|away: no, we still need new images either way | 18:51 |
| ykarel|away | clarkb, yes true | 18:51 |
| fungi | i'm happy to have been wrong that they wouldn't cave to pressure from complaints | 18:51 |
| mordred | we shoudl have them for centos now right? | 18:51 |
| clarkb | mordred: no its building now | 18:51 |
| mordred | fungi: there seem to be new humans | 18:51 |
| mordred | clarkb: nod | 18:51 |
| ykarel|away | clarkb, isn't there some basic tests for images, before they get pushed to clouds | 18:51 |
| mordred | clarkb: well - I'm pretty pessimistic about the workaround patch | 18:52 |
| clarkb | mordred: I think what we should do is set failed_when: false on those tasks | 18:52 |
| ykarel|away | to avoid future such issues | 18:52 |
| fungi | ykarel|away: what would you test? | 18:52 |
| clarkb | mordred: and then it is best effort if the dir is there, then remove the workaround entirely once we have new images | 18:52 |
| fungi | there was a time when we tested images with devstack | 18:52 |
| ykarel|away | fungi, some jobs | 18:52 |
| ykarel|away | agree everything can't be tested | 18:52 |
| clarkb | ykarel|away: no the problem space is too large and I don't want it to be our job to fix devstack (or tripleo or wahtever) when distros or pypa move forward | 18:52 |
| fungi | and went weeks without uploading new images because devstack itself was flaky | 18:52 |
| mordred | clarkb: oh - becaues we're only going to have /root/.local if we have a setuptools that does the linking | 18:52 |
| mordred | nod | 18:53 |
| clarkb | ykarel|away: this is actually broken in the real world we don't want to mask that | 18:53 |
| clarkb | mordred: ya | 18:53 |
| mordred | kk. let me add failed_when: false | 18:53 |
| ykarel|away | clarkb, fungi ack | 18:53 |
| clarkb | ykarel|away: our goal isn't to have jobs that pass all the time. It is to correctly determine when software is broken so that it can be fixed | 18:53 |
| ykarel|away | clarkb, ack got it | 18:54 |
| fungi | even if that software is external dependencies and test harnesses/frameworks | 18:54 |
| * clarkb preps for the meeting | 18:54 | |
| openstackgerrit | Monty Taylor proposed opendev/base-jobs master: Add failed_when false to workaround https://review.opendev.org/707234 | 18:54 |
| clarkb | the good news is through our feedback and others virtualenv has changed its behavior to be friendleir to users | 18:54 |
| clarkb | and we wouldn't have had that feedback if we masked it off | 18:55 |
| clarkb | on the long term this should be healthier for all in volved | 18:55 |
| openstackgerrit | Monty Taylor proposed opendev/base-jobs master: Revert "Workaround root virtualenv break with v20" https://review.opendev.org/707202 | 18:56 |
| mordred | clarkb: we'll need to force-merge the failed_when patch | 18:56 |
| *** gfidente is now known as gfidente|afk | 18:56 | |
| clarkb | mordred: +2 from me maybe fungi or corvus (or ianw if awake) can look too and if they are happy with it we land it | 18:57 |
| mordred | clarkb, fungi, corvus: I've got the force-merge bit - but does someone want to review that? | 18:57 |
| *** psachin has quit IRC | 18:57 | |
| clarkb | mordred: I +2'd it | 18:57 |
| mordred | clarkb: awesome - let's give corvus and fungi a sec | 18:57 |
| clarkb | 707234 to be clear | 18:57 |
| mordred | yeah | 18:57 |
| fungi | i +a'd both, (obviously the second won't merge until you remove the wip toggle) | 18:59 |
| corvus | mordred: ++ on the failed when | 19:01 |
| corvus | on the revert, we're expecting a new image build to get new virtualenv with the copy behavior, right? | 19:01 |
| clarkb | corvus: we landed a change to use --seeder=pip | 19:04 |
| clarkb | corvus: whcih we should be able to revert and then build using the new copy behavior | 19:04 |
| clarkb | (both should fix this) | 19:04 |
| corvus | ok. wasn't sure if we were going to progress through both of if we superceded pip with copy | 19:05 |
| openstackgerrit | Merged opendev/base-jobs master: Add failed_when false to workaround https://review.opendev.org/707234 | 19:05 |
| mordred | I have force-merged that and removed myself from the bootstrappers group | 19:06 |
| openstackgerrit | Sorin Sbarnea proposed zuul/zuul-jobs master: Install tox-venv alongside tox https://review.opendev.org/707237 | 19:18 |
| openstackgerrit | Sorin Sbarnea proposed zuul/zuul-jobs master: Install tox-venv alongside tox https://review.opendev.org/707237 | 19:20 |
| *** armax has joined #openstack-infra | 19:21 | |
| openstackgerrit | James E. Blair proposed zuul/zuul-jobs master: Add an ensure-tox test job https://review.opendev.org/706371 | 19:23 |
| openstackgerrit | Sorin Sbarnea proposed zuul/zuul-jobs master: Install tox-venv alongside tox https://review.opendev.org/707237 | 19:24 |
| *** eernst has joined #openstack-infra | 19:26 | |
| *** eernst has quit IRC | 19:28 | |
| openstackgerrit | James E. Blair proposed zuul/zuul-jobs master: Run ensure-tox on all platforms https://review.opendev.org/707238 | 19:30 |
| openstackgerrit | Sorin Sbarnea proposed zuul/zuul-jobs master: tox: change default calling as a module https://review.opendev.org/690057 | 19:34 |
| *** hashar has joined #openstack-infra | 19:35 | |
| *** jtomasek has quit IRC | 19:48 | |
| *** Goneri has quit IRC | 19:53 | |
| mordred | fungi: I don't have questinos - but could you just be entertaining? | 20:01 |
| * fungi juggles flaming carrots | 20:02 | |
| * fungi switches to juggling flaming code reviews | 20:03 | |
| mordred | mmm. flaming carrots are tasty | 20:04 |
| fungi | in an existential sort of way | 20:04 |
| fungi | https://en.wikipedia.org/wiki/Flaming_Carrot_Comics | 20:05 |
| mordred | fungi: :) - I was more thinking about something along these lines: https://www.epicurious.com/recipes/food/views/burnt-carrots-with-goat-cheese-parsley-arugula-and-crispy-garlic-chips-354729 | 20:06 |
| fungi | it's all the same, ultimately | 20:08 |
| *** Wasaac has quit IRC | 20:15 | |
| *** Wasaac_ has joined #openstack-infra | 20:15 | |
| *** Wasaac_ is now known as Wasaac | 20:16 | |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Stop using python::virtualenv for virtualenvs https://review.opendev.org/707214 | 20:29 |
| mordred | fungi, ianw: ^^ sigh | 20:29 |
| mordred | ianw: or - you said something about moving afsmon ... should I just delete that from the puppet for now? I defer to you judgement on that | 20:30 |
| *** owalsh has quit IRC | 20:30 | |
| openstackgerrit | Merged zuul/zuul master: Extract the watcher from git driver https://review.opendev.org/706092 | 20:35 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Make small tweaks to launch node README https://review.opendev.org/705673 | 20:39 |
| *** Lucas_Gray has joined #openstack-infra | 20:41 | |
| *** owalsh has joined #openstack-infra | 20:44 | |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Get LE certs for review.o.o https://review.opendev.org/705670 | 20:44 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Remove review-dev01.openstack.org https://review.opendev.org/705671 | 20:44 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Use LE certs for Apache https://review.opendev.org/705690 | 20:44 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Update letsencrypt_gid documentation https://review.opendev.org/705878 | 20:44 |
| ianw | mordred: i think i can move that mirror-update.opendev.org like the release script; i'll work on that today. if it's easier to just remove from puppet i think that's ok | 20:48 |
| mordred | ianw: well - let's see if that patch goes green - cause if it does, that's the easiest thing | 20:49 |
| mordred | ianw: if it _doesn't_ ... I'll move to removing | 20:49 |
| clarkb | for entertainment I've got a 4 yeard old that pulled a shopping cart over on herself | 20:49 |
| clarkb | I looked away for 5 seconds | 20:50 |
| ianw | :/ just to add to an interesting day | 20:51 |
| mordred | clarkb: sometimes it can be fun to pull a shopping cart over on oneself? | 20:52 |
| weshay | how's this look? https://review.opendev.org/#/c/707204/ | 21:01 |
| clarkb | mordred: well fun for everyone watxhing. she is ok | 21:03 |
| mordred | clarkb: good. I was assuming she was ok since you'd IRC'd about it | 21:05 |
| mordred | but I'm glad to hear that was a valid assumption | 21:05 |
| mordred | weshay: you're gonna want an upper-constraints.txt update in there too | 21:07 |
| openstackgerrit | James E. Blair proposed zuul/zuul master: Fix kubectl/oc install in container image https://review.opendev.org/707249 | 21:09 |
| weshay | mordred, https://opendev.org/openstack/requirements/src/branch/master/upper-constraints.txt#L573 bump that to 4.0.0? | 21:10 |
| *** Wasaac has quit IRC | 21:11 | |
| *** Wasaac has joined #openstack-infra | 21:12 | |
| weshay | mordred, mind if I do that in another patch? | 21:13 |
| *** Goneri has joined #openstack-infra | 21:13 | |
| openstackgerrit | James E. Blair proposed zuul/zuul master: Support pausing merge jobs https://review.opendev.org/707192 | 21:16 |
| openstackgerrit | James E. Blair proposed zuul/zuul master: Revert "Fix github app authentication to work with checks API endpoints" https://review.opendev.org/707205 | 21:16 |
| mordred | weshay: well - it's mostly that u-c isn't going to match what you just did - so you need to bump that to 4.0.0 - but you ALSO need to add a version constraint for it and then add a 2.7 constraint for 3.1.1 | 21:17 |
| mordred | so I don't think that's going to do what you think it's going to do - one sec | 21:17 |
| weshay | ya.. I see that it won't get to 4.0.0 but 3.1.1 is working for folks now | 21:18 |
| weshay | I'm not trying to get to 4.0.0 in py37 just trying to get jobs back to green.. priority #1, then I can bump the upper constraint | 21:18 |
| mordred | AH - I grok what you're saying | 21:19 |
| weshay | if I push another revision we're down for another day | 21:19 |
| mordred | sure then | 21:19 |
| weshay | ah.. cool. doing it now.. thanks | 21:19 |
| *** hashar has quit IRC | 21:21 | |
| *** Lucas_Gray has quit IRC | 21:22 | |
| openstackgerrit | Monty Taylor proposed zuul/zuul master: Move oc download to before src copy https://review.opendev.org/707250 | 21:22 |
| weshay | k.. patch is on top.. | 21:26 |
| mordred | weshay: comment left - you need a few more lines | 21:29 |
| mordred | ianw: GROSS - the askbot puppet is also bong. this is going to get worse | 21:29 |
| weshay | ah.. k | 21:30 |
| weshay | thank you | 21:30 |
| openstackgerrit | Monty Taylor proposed opendev/puppet-askbot master: Stop using python::virtualenv https://review.opendev.org/707255 | 21:33 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Stop using python::virtualenv for virtualenvs https://review.opendev.org/707214 | 21:35 |
| *** xek_ has quit IRC | 21:35 | |
| weshay | heh.. thanks smcginnis :) | 21:39 |
| mordred | weshay: he knows what's going on more than we do | 21:39 |
| smcginnis | weshay: Sorry, it was already on its way. :) | 21:39 |
| weshay | yup.. happy to let him do his thang | 21:39 |
| weshay | not a prob.. don't mind a bit | 21:39 |
| openstackgerrit | Merged zuul/zuul master: Run yarn explicitly in Dockerfile https://review.opendev.org/707182 | 21:47 |
| *** ahosam has joined #openstack-infra | 22:04 | |
| openstackgerrit | Ian Pittwood proposed openstack/project-config master: [WIP] Use GitHub Issues for airshipctl https://review.opendev.org/707262 | 22:05 |
| *** jamesmcarthur has joined #openstack-infra | 22:06 | |
| *** rcernin has joined #openstack-infra | 22:07 | |
| *** pkopec has quit IRC | 22:13 | |
| *** ahosam has quit IRC | 22:15 | |
| *** kaisers has joined #openstack-infra | 22:21 | |
| *** slaweq has quit IRC | 22:25 | |
| *** mattw4 has quit IRC | 22:27 | |
| *** rh-jelabarre has quit IRC | 22:31 | |
| openstackgerrit | Merged zuul/zuul master: Fix kubectl/oc install in container image https://review.opendev.org/707249 | 22:40 |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: Move afsmon to mirror-update.opendev.org https://review.opendev.org/707267 | 22:41 |
| *** jamesmcarthur has quit IRC | 22:41 | |
| ianw | mordred: ^ if that goes green, it's one less use at least :/ | 22:42 |
| *** jackedin has quit IRC | 22:47 | |
| *** slaweq has joined #openstack-infra | 22:50 | |
| *** iurygregory has quit IRC | 22:52 | |
| mordred | ianw: I think we're either going to have to force-merge a patch - or remove a testing | 22:54 |
| *** slaweq has quit IRC | 22:55 | |
| ianw | mordred: i'm just having a look at that askbot one | 22:55 |
| mordred | ianw: because we need your patch there plus the one in the puppet-askbot repo | 22:55 |
| mordred | ianw: https://review.opendev.org/#/c/707255/ fwiw | 22:55 |
| ianw | 020-02-11 21:46:01 +0000 Puppet (warning): Unknown variable: '::virtualenv_version'. at /etc/puppetlabs/code/modules/python/manifests/virtualenv.pp:121:22 | 22:55 |
| ianw | not sure how that's getting pulled in | 22:56 |
| ianw | https://zuul.opendev.org/t/openstack/build/8102215f00b1471c8c1f7e991bcfa5c0/log/applytest/puppetapplytest21.final.out.FAILED is the failure from 707255 | 22:56 |
| clarkb | ianw its from puppet-python | 22:56 |
| clarkb | oh thats there in the error | 22:57 |
| mordred | yeah | 22:57 |
| ianw | the other thing is, 707255 is making it a python3 virtualenv, which i'm 99% sure askbot isn't | 22:57 |
| mordred | ah - yeah | 22:57 |
| mordred | one sec - lemme update 707255 | 22:58 |
| mordred | but I think your patch + 707255 should fix the things | 22:58 |
| mordred | but we might have to force-merge one of them (I'd vote for puppet-askbot) | 22:58 |
| ianw | ++ | 22:58 |
| openstackgerrit | Monty Taylor proposed opendev/puppet-askbot master: Stop using python::virtualenv https://review.opendev.org/707255 | 23:00 |
| ianw | oh, that isn't the error in that test anyway | 23:01 |
| ianw | 'versioncmp' parameter 'a' expects a String value, got Undef at /etc/puppetlabs/code/modules/python/manifests/virtualenv.pp:121:11 | 23:01 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Get LE certs for review.o.o https://review.opendev.org/705670 | 23:02 |
| *** jamesmcarthur has joined #openstack-infra | 23:02 | |
| mordred | ianw, clarkb: that ^^ depeds-on both - so it should show us if force-merging askbot would be good | 23:02 |
| ianw | - virtualenv => true, | 23:03 |
| ianw | mordred: ^ is it possible puppet-python is unhappy without this set? | 23:03 |
| clarkb | if making a virtualenv then yes aiui | 23:03 |
| ianw | yeah, the python::virtualenv is *also* removed ... that's why it seems odd | 23:04 |
| mordred | ianw: I'm confused - which thing are you looking at now? | 23:04 |
| *** jamesmcarthur_ has joined #openstack-infra | 23:05 | |
| ianw | mordred: ohhh, i gotta read the errors better sorry | 23:05 |
| *** jamesmcarthur has quit IRC | 23:05 | |
| mordred | ianw: PHEW | 23:05 |
| ianw | /etc/puppetlabs/code/modules/openstack_project/manifests/mirror_update.pp:474 ... so that's the afsmon virtualenv | 23:05 |
| mordred | yeah - so your patch should take care of that | 23:05 |
| ianw | that's causing problems in the puppet-askbot repo ... which is the circular dependency you're talking about | 23:06 |
| mordred | and then the askbot patch should take care of the other | 23:06 |
| mordred | yup | 23:06 |
| ianw | heh, yeah, ok i think i'm synced now :) | 23:06 |
| mordred | but - 705670 should show us a potential green state | 23:06 |
| *** tkajinam has joined #openstack-infra | 23:06 | |
| ianw | so much fun | 23:07 |
| *** slaweq has joined #openstack-infra | 23:11 | |
| mordred | ianw: one question on https://review.opendev.org/#/c/707267 | 23:12 |
| ianw | mordred: replied, but i'm hoping that installing outside a virtualenv pip will notice them and not bring anything else in? | 23:14 |
| mordred | nope | 23:15 |
| mordred | it does not work that way at all | 23:15 |
| mordred | sadly | 23:15 |
| mordred | it would be neat if it did | 23:15 |
| mordred | ianw: at least, I'm 99% sure of that - with all the changes in pip, it's certainly worth checking - let me make a quick local test | 23:16 |
| *** aaronsheffield has quit IRC | 23:16 | |
| *** slaweq has quit IRC | 23:16 | |
| mordred | ianw: I stand drastically corrected | 23:19 |
| ianw | https://f4e8f4bf26e31661b712-346c02a9e1e5e1d1d02c825991cd779a.ssl.cf1.rackcdn.com/707267/1/check/system-config-run-mirror-update/4498c93/bridge.openstack.org/ara-report/result/6a31d930-395f-43a0-99fa-7e43c263bade/ | 23:20 |
| ianw | still pulls in babel and pytz | 23:20 |
| mordred | ianw: left a comment, but I think we should add python3-babel to your package list | 23:21 |
| mordred | whcih will also take care of pytz - but you could add python3-tz explicitly if you wanted | 23:21 |
| ianw | ++ i think it's only for documentation build, it probably shouldn't be in requirements.txt | 23:21 |
| ianw | but that can be a problem for another day | 23:21 |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: Move afsmon to mirror-update.opendev.org https://review.opendev.org/707267 | 23:23 |
| *** jamesmcarthur_ has quit IRC | 23:24 | |
| *** jamesmcarthur has joined #openstack-infra | 23:25 | |
| *** jamesmcarthur has quit IRC | 23:25 | |
| *** jamesmcarthur has joined #openstack-infra | 23:26 | |
| *** jamesmcarthur has quit IRC | 23:32 | |
| *** dchen has joined #openstack-infra | 23:32 | |
| *** armax has quit IRC | 23:33 | |
| *** nicolasbock has quit IRC | 23:36 | |
| *** igordc has quit IRC | 23:46 | |
| *** sreejithp has quit IRC | 23:48 | |
| *** sreejithp_ has joined #openstack-infra | 23:48 | |
| openstackgerrit | Merged openstack/os-performance-tools master: [ussuri][goal] Drop python 2.7 support and testing https://review.opendev.org/705520 | 23:50 |
| *** rcernin has quit IRC | 23:51 | |
| ianw | hrm, zuul didn't like the dependencies on https://review.opendev.org/#/c/705670/ | 23:56 |
| clarkb | ianw: because a new ps showed up ? | 23:57 |
| clarkb | it may just need a recheck | 23:57 |
| ianw | yeah, i looks like a lint error on the other one, let me see | 23:57 |
| openstackgerrit | Merged zuul/zuul master: Revert "Fix github app authentication to work with checks API endpoints" https://review.opendev.org/707205 | 23:58 |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!