| ianw | i'm not really seeing the error in https://zuul.opendev.org/t/openstack/build/4526f76400124cc3929afc3dd2210463/log/job-output.txt#2099 | 00:00 |
|---|---|---|
| clarkb | ianw: its the warnings below | 00:02 |
| clarkb | https://zuul.opendev.org/t/openstack/build/4526f76400124cc3929afc3dd2210463/log/job-output.txt#2100-2117 | 00:02 |
| clarkb | we have to remove the :: prefix because in puppet4 its redundant and the linter decided to stop allowing it because rewriting things like that is fun?! | 00:03 |
| ianw | that's a bunch of unrelated files, this job must have not run on this code since it was updated | 00:04 |
| clarkb | ya and its a relatively new linter things (within the last month iirc) | 00:04 |
| *** sreejithp_ has quit IRC | 00:05 | |
| *** rcernin has joined #openstack-infra | 00:07 | |
| *** artom has quit IRC | 00:10 | |
| *** slaweq has joined #openstack-infra | 00:11 | |
| openstackgerrit | Ian Wienand proposed opendev/puppet-askbot master: Stop using python::virtualenv https://review.opendev.org/707255 | 00:12 |
| openstackgerrit | Ian Wienand proposed opendev/puppet-askbot master: Remove ::'s that linter now complians about https://review.opendev.org/707276 | 00:12 |
| ianw | if that lints, i think we force-merge both | 00:13 |
| ianw | rechecking 705670 if it was an in-flight change | 00:13 |
| *** artom has joined #openstack-infra | 00:14 | |
| ianw | yeah, it's running | 00:15 |
| *** artom has quit IRC | 00:15 | |
| *** artom has joined #openstack-infra | 00:15 | |
| *** slaweq has quit IRC | 00:16 | |
| ianw | i have no idea why afsmon has "Babel!=2.4.0,>=2.3.4 # BSD" ... but guess what version of babel is packaged in bionic | 00:19 |
| clarkb | ha | 00:20 |
| clarkb | when it rains it pours | 00:20 |
| ianw | there aren't even any docs in afsmon, i have no idea why i included it | 00:24 |
| clarkb | babel is for translations? | 00:24 |
| clarkb | maybe it just got pulled in via some templated thing? | 00:24 |
| ianw | or pbr? | 00:25 |
| clarkb | I thought pbr avoided runtime deps | 00:27 |
| openstackgerrit | Ian Wienand proposed opendev/afsmon master: Remove Babel requirement https://review.opendev.org/707277 | 00:31 |
| clarkb | +2 | 00:31 |
| fungi | pbr does end up depending on pkg_resources, but lazy-loaded i think | 00:32 |
| clarkb | ya setuptools too | 00:32 |
| clarkb | (but it runs via setuptools so thats sort of an assumed dep) | 00:32 |
| fungi | sure | 00:34 |
| fungi | honestly, pip has more deps than pbr, though it vendors them | 00:34 |
| *** tosky has quit IRC | 00:38 | |
| ianw | sigh, now flake8, presumably a newer version, fails on afsmon | 00:41 |
| *** armax has joined #openstack-infra | 00:43 | |
| openstackgerrit | Ian Wienand proposed opendev/afsmon master: Remove Babel requirement https://review.opendev.org/707277 | 00:47 |
| openstackgerrit | Ian Wienand proposed opendev/afsmon master: flake8 regex fixes https://review.opendev.org/707281 | 00:47 |
| *** rf0lc0 has quit IRC | 00:50 | |
| *** rcernin has quit IRC | 00:55 | |
| *** rcernin has joined #openstack-infra | 00:55 | |
| ianw | 705670 still isn't quite happy, although i wonder if it has actually correctly deployed all the code from the depends-on | 00:57 |
| ianw | https://zuul.opendev.org/t/openstack/build/6a7ebef337af4a948dceb4250d6ce4da/log/applytest/puppetapplytest46.final.out.FAILED | 00:58 |
| ianw | https://zuul.opendev.org/t/openstack/build/6a7ebef337af4a948dceb4250d6ce4da/log/applytest/puppetapplytest47.final.out.FAILED | 00:58 |
| ianw | arrgh, no, i've stuffed up 707255 fixing the lint issues | 01:01 |
| openstackgerrit | Ian Wienand proposed opendev/puppet-askbot master: Stop using python::virtualenv https://review.opendev.org/707255 | 01:03 |
| ianw | this time for sure | 01:04 |
| openstackgerrit | Merged opendev/afsmon master: flake8 regex fixes https://review.opendev.org/707281 | 01:05 |
| openstackgerrit | Merged opendev/afsmon master: Remove Babel requirement https://review.opendev.org/707277 | 01:05 |
| *** masayukig is now known as migawa|AFK | 01:08 | |
| *** migawa|AFK is now known as migawa | 01:09 | |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: Move afsmon to mirror-update.opendev.org https://review.opendev.org/707267 | 01:10 |
| *** slaweq has joined #openstack-infra | 01:11 | |
| *** jamesmcarthur has joined #openstack-infra | 01:12 | |
| *** slaweq has quit IRC | 01:16 | |
| *** rlandy has quit IRC | 01:17 | |
| *** jamesmcarthur has quit IRC | 01:23 | |
| *** stevebaker has quit IRC | 01:24 | |
| *** stevebaker has joined #openstack-infra | 01:26 | |
| *** sgw has quit IRC | 01:26 | |
| *** jamesmcarthur has joined #openstack-infra | 01:36 | |
| openstackgerrit | Ian Wienand proposed opendev/puppet-askbot master: Stop using python::virtualenv https://review.opendev.org/707255 | 01:46 |
| *** yamamoto has joined #openstack-infra | 01:57 | |
| ianw | going afk for a little for lunch, then i think i can sort out getting the system-config gate up again | 02:00 |
| *** slaweq has joined #openstack-infra | 02:11 | |
| *** yamamoto has quit IRC | 02:12 | |
| *** yamamoto has joined #openstack-infra | 02:13 | |
| *** slaweq has quit IRC | 02:16 | |
| *** yamamoto has quit IRC | 02:17 | |
| *** ricolin has joined #openstack-infra | 02:19 | |
| *** roman_g has quit IRC | 02:34 | |
| *** jamesmcarthur has quit IRC | 02:38 | |
| *** yamamoto has joined #openstack-infra | 02:40 | |
| *** jamesmcarthur has joined #openstack-infra | 02:53 | |
| *** yamamoto has quit IRC | 03:01 | |
| *** yamamoto has joined #openstack-infra | 03:03 | |
| *** rcernin has quit IRC | 03:06 | |
| *** slaweq has joined #openstack-infra | 03:11 | |
| ianw | ok, so 707255 now only has one puppet failure, which is the afsmon virtualenv deployment. this is the circular dependency. so i am going to force merge 707276 (lint changes) & 707255 | 03:12 |
| openstackgerrit | Merged opendev/puppet-askbot master: Remove ::'s that linter now complians about https://review.opendev.org/707276 | 03:15 |
| openstackgerrit | Merged opendev/puppet-askbot master: Stop using python::virtualenv https://review.opendev.org/707255 | 03:16 |
| *** slaweq has quit IRC | 03:16 | |
| ianw | with the 1.2 release of afsmon, pip doesn't pull any non-deb-packaged dependencies in -> https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_6ec/707267/3/check/system-config-run-mirror-update/6ec965c/bridge.openstack.org/ara-report/result/52902a76-329c-440b-8dc3-f1c82f627a78/ | 03:17 |
| ianw | rechecking https://review.opendev.org/#/c/707267/ which should now pass zuul | 03:18 |
| ianw | and removed myself from bootstrappers | 03:18 |
| *** sgw has joined #openstack-infra | 03:21 | |
| *** psachin has joined #openstack-infra | 03:21 | |
| *** rcernin has joined #openstack-infra | 03:22 | |
| *** pcrews has joined #openstack-infra | 03:23 | |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: Move afsmon to mirror-update.opendev.org https://review.opendev.org/707267 | 03:39 |
| *** jamesmcarthur has quit IRC | 03:55 | |
| *** udesale has joined #openstack-infra | 04:05 | |
| *** gyee has quit IRC | 04:08 | |
| *** ramishra has joined #openstack-infra | 04:09 | |
| *** slaweq has joined #openstack-infra | 04:11 | |
| *** slaweq has quit IRC | 04:16 | |
| *** rcernin has quit IRC | 04:16 | |
| *** rcernin has joined #openstack-infra | 04:16 | |
| *** dSrinivas has quit IRC | 04:32 | |
| *** ramishra has quit IRC | 04:32 | |
| *** psachin has quit IRC | 04:33 | |
| *** artom has quit IRC | 04:35 | |
| weshay | ianw, hey buddy.. have an issue blocking container builds in tripleo.. getting an error re: blacklist on https://review.opendev.org/#/c/707204/ | 04:39 |
| weshay | https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_50e/707204/4/check/openstack-tox-validate/50e08c4/job-output.txt | 04:40 |
| weshay | any idea what's going on there? | 04:40 |
| prometheanfire | weshay: hi :D | 04:43 |
| weshay | howdy | 04:43 |
| *** dave-mccowan has quit IRC | 04:43 | |
| prometheanfire | python-ironicclient!=2.5.2,!=2.7.1,!=3.0.0,>=2.3.0;python_version>='3.5' | 04:44 |
| weshay | is the issue >=2.3.0 | 04:44 |
| prometheanfire | that doesn't need the minimum specified | 04:44 |
| weshay | and needs to be 2.3.1? | 04:44 |
| prometheanfire | same for the line above actually | 04:44 |
| prometheanfire | see how I'm doing https://review.opendev.org/707261 | 04:45 |
| weshay | k.. the error did indicate that.. /me looks | 04:45 |
| prometheanfire | add the cap for py27 and specifiy an uncapped line (same as before) for py3 | 04:46 |
| *** ramishra has joined #openstack-infra | 04:46 | |
| * weshay looking | 04:46 | |
| prometheanfire | and you py_version should be >=3.6, not 3.5 | 04:46 |
| prometheanfire | https://github.com/openstack/python-ironicclient/blob/4.0.0/setup.cfg specifies 3.6/3.7 only | 04:47 |
| prometheanfire | should really have a cap on python_version, but that'd just be annoying, and python is forward compat right? :D | 04:47 |
| weshay | k.. hearing slightly different things from diff folks.. so it goes :) | 04:47 |
| * prometheanfire stares at py39 | 04:47 | |
| weshay | aye | 04:47 |
| weshay | lolz | 04:47 |
| * prometheanfire is ptl of the reqs project if it helps | 04:48 | |
| prometheanfire | also #openstack-requirements | 04:48 |
| weshay | that always does :) | 04:48 |
| weshay | thanks for the assist | 04:48 |
| weshay | k.. let's see if I grok this.. making a change | 04:48 |
| prometheanfire | np, was trawling through the channels | 04:48 |
| *** migawa is now known as migawa|lunch|AFK | 04:50 | |
| weshay | so.. | 04:51 |
| weshay | python-ironicclient!=2.5.2,!=2.7.1,!=3.0.0,>=2.3.1,<4.0.0;python_version=='2.7' # Apache-2.0 | 04:51 |
| weshay | python-ironicclient!=2.5.2,!=2.7.1,!=3.0.0,>=2.3.1;python_version>='3.6' # Apache-2.0 | 04:51 |
| weshay | py27 is capped <4.0.0 | 04:51 |
| weshay | py3.6+ is not capped but retains the excludes certain versions | 04:51 |
| prometheanfire | you are still specifying the min version | 04:51 |
| weshay | k | 04:51 |
| prometheanfire | (you shouldn't be) | 04:52 |
| weshay | python-ironicclient!=2.5.2,!=2.7.1,!=3.0.0,>=2.3.1,<4.0.0;python_version=='2.7' # Apache-2.0 | 04:52 |
| weshay | python-ironicclient!=2.5.2,!=2.7.1,!=3.0.0;python_version>='3.6' # Apache-2.0 | 04:52 |
| prometheanfire | looks about right, submit it and I'll look at the diff in gertty | 04:52 |
| prometheanfire | might want to fix the commit message too (spacing) :D | 04:54 |
| weshay | prometheanfire, last question.. when we were running tests locally against the requirements for tripleo-common.. change https://review.opendev.org/#/c/707054/ | 04:54 |
| openstackgerrit | Merged opendev/system-config master: Move afsmon to mirror-update.opendev.org https://review.opendev.org/707267 | 04:54 |
| * prometheanfire now knows why kevin was interested in the review | 04:54 | |
| weshay | it seemed we had to match exactly what was in global requirements or the requirements-check job failed | 04:55 |
| prometheanfire | you are allowed to define minimums in your project's configs, but we don't define them globally | 04:55 |
| weshay | ah.. k.. that's probably what we did.. copy from tc to global | 04:55 |
| prometheanfire | what you had to match for the minimum was a lower-constraints file | 04:55 |
| weshay | k.. makes sense | 04:56 |
| weshay | k | 04:56 |
| weshay | thanks for the help :) | 04:56 |
| weshay | prometheanfire++ | 04:56 |
| prometheanfire | each project might specify a diferent minimum (swift likes to keep things nice and ancient), but they all must work with whatever is in upper-constraints | 04:56 |
| prometheanfire | np | 04:56 |
| *** ykarel|away is now known as ykarel | 04:58 | |
| prometheanfire | onemore change :P | 04:58 |
| weshay | oh both mins k | 04:59 |
| weshay | you can lead a horse | 04:59 |
| *** rcernin is now known as rcernin|lunch | 05:01 | |
| weshay | should be fixed now | 05:01 |
| prometheanfire | that one looks good | 05:01 |
| weshay | man.. my brain understood and fingers are too anxious to type | 05:01 |
| prometheanfire | :D | 05:02 |
| weshay | \0/ | 05:08 |
| *** slaweq has joined #openstack-infra | 05:11 | |
| *** slaweq has quit IRC | 05:16 | |
| *** migawa|lunch|AFK is now known as migawa|lunch | 05:18 | |
| timburke | "ancient", "stable" -- to-may-to, to-mah-to | 05:25 |
| prometheanfire | timburke: vulnerable? I'm sure there's something somewhere... | 05:30 |
| *** evrardjp has quit IRC | 05:34 | |
| *** evrardjp has joined #openstack-infra | 05:34 | |
| timburke | i don't feel like it's my decision to make. that's on distros, vendors, and deployers -- when it comes down to it, operators and their risk tolerances | 05:36 |
| timburke | ultimately, dropping support for older versions of my dependencies only increases upgrade friction | 05:36 |
| ianw | "Unable to find any of pip3 to use. pip needs to be installed." ... i feel like this is the same thing mordred saw | 05:37 |
| prometheanfire | timburke: sure, I'm just poking fun | 05:39 |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: afsmon: install python3-pip https://review.opendev.org/707313 | 05:39 |
| *** jamesmcarthur has joined #openstack-infra | 05:57 | |
| *** goldyfruit has quit IRC | 05:58 | |
| *** goldyfruit has joined #openstack-infra | 05:58 | |
| *** jamesmcarthur has quit IRC | 06:01 | |
| *** slaweq has joined #openstack-infra | 06:11 | |
| *** slaweq has quit IRC | 06:16 | |
| openstackgerrit | Merged opendev/system-config master: afs-release: run every 5 minutes https://review.opendev.org/707060 | 06:19 |
| *** goldyfruit has quit IRC | 06:30 | |
| *** goldyfruit has joined #openstack-infra | 06:30 | |
| *** lmiccini has joined #openstack-infra | 06:36 | |
| *** jtomasek has joined #openstack-infra | 06:51 | |
| *** slaweq has joined #openstack-infra | 07:08 | |
| *** slaweq has quit IRC | 07:15 | |
| *** ysastri has joined #openstack-infra | 07:15 | |
| *** cshen has joined #openstack-infra | 07:18 | |
| ysastri | Hello my zuul checks failed for the 3rd time with same error ERROR: not a btrfs filesystem: /var/lib/lxc. This time in TASK [haproxy_server : Ensure haproxy_hatop_download_path exists]. I don't have any idea how to proceed. https://zuul.opendev.org/t/openstack/build/553ab6b3936c4215b902fbfd3fd3d1d2 | 07:19 |
| *** pgaxatte has joined #openstack-infra | 07:19 | |
| *** rpittau|afk is now known as rpittau | 07:21 | |
| *** pgaxatte has quit IRC | 07:24 | |
| *** pgaxatte has joined #openstack-infra | 07:24 | |
| *** yoctozepto has quit IRC | 07:28 | |
| *** raukadah is now known as chkumar|rover | 07:28 | |
| *** yoctozepto has joined #openstack-infra | 07:28 | |
| *** lpetrut has joined #openstack-infra | 07:31 | |
| *** stevebaker has quit IRC | 07:31 | |
| *** yoctozepto has quit IRC | 07:33 | |
| *** ysastri has quit IRC | 07:35 | |
| *** iurygregory has joined #openstack-infra | 07:35 | |
| *** kopecmartin|afk is now known as kopecmartin | 07:44 | |
| *** ianychoi has joined #openstack-infra | 07:46 | |
| *** slaweq has joined #openstack-infra | 07:51 | |
| *** imacdonn has quit IRC | 07:54 | |
| *** imacdonn has joined #openstack-infra | 07:54 | |
| *** ykarel is now known as ykarel|lunch | 07:55 | |
| *** slaweq has quit IRC | 07:57 | |
| *** slaweq has joined #openstack-infra | 07:59 | |
| *** yoctozepto has joined #openstack-infra | 07:59 | |
| *** xek_ has joined #openstack-infra | 08:09 | |
| *** tkajinam has quit IRC | 08:10 | |
| *** dchen has quit IRC | 08:16 | |
| *** tosky has joined #openstack-infra | 08:22 | |
| *** ccamacho has joined #openstack-infra | 08:27 | |
| openstackgerrit | Tobias Henkel proposed zuul/zuul master: Support pausing merge jobs https://review.opendev.org/707192 | 08:28 |
| openstackgerrit | Sorin Sbarnea proposed zuul/zuul-jobs master: ensure-tox: include tox-venv plugin https://review.opendev.org/707237 | 08:29 |
| *** yamamoto has quit IRC | 08:30 | |
| *** roman_g has joined #openstack-infra | 08:31 | |
| *** pkopec has joined #openstack-infra | 08:36 | |
| *** ralonsoh has joined #openstack-infra | 08:38 | |
| openstackgerrit | Sorin Sbarnea proposed zuul/zuul-jobs master: WIP: ensure-tox to verify module calling https://review.opendev.org/707335 | 08:42 |
| *** yamamoto has joined #openstack-infra | 08:44 | |
| *** dtantsur|afk is now known as dtantsur | 08:45 | |
| *** jpena|off is now known as jpena | 08:50 | |
| openstackgerrit | Matthieu Huin proposed zuul/zuul master: Authorization rules: add templating https://review.opendev.org/705193 | 08:52 |
| *** gfidente|afk is now known as gfidente | 08:58 | |
| openstackgerrit | Noam Angel proposed openstack/diskimage-builder master: fix iscsi-boot element exiting build even if dracut-regenerate used https://review.opendev.org/707340 | 09:00 |
| *** ysastri has joined #openstack-infra | 09:03 | |
| *** lucasagomes has joined #openstack-infra | 09:06 | |
| openstackgerrit | Noam Angel proposed openstack/diskimage-builder master: fix iscsi-boot element exiting build even if dracut-regenerate used https://review.opendev.org/707340 | 09:09 |
| *** cshen has quit IRC | 09:17 | |
| jpena | hi! It looks like https://review.opendev.org/707199 has broken dib images for centos 8, since the "--seeder=pip" command line option is not supported by virtualenv 15.1.0 | 09:21 |
| *** iurygregory has quit IRC | 09:21 | |
| jpena | that is the version packaged by rhel/centos 8, and we can only install it from package | 09:21 |
| *** tetsuro has joined #openstack-infra | 09:27 | |
| *** cshen has joined #openstack-infra | 09:33 | |
| *** pkopec has quit IRC | 09:37 | |
| *** cshen has quit IRC | 09:37 | |
| openstackgerrit | Merged opendev/system-config master: afsmon: install python3-pip https://review.opendev.org/707313 | 09:38 |
| *** pkopec has joined #openstack-infra | 09:39 | |
| *** iurygregory has joined #openstack-infra | 09:39 | |
| openstackgerrit | Alfredo Moralejo proposed openstack/project-config master: Revert "Use virtualenv --seeder=pip so that libs are accessible" https://review.opendev.org/707346 | 09:39 |
| *** ykarel|lunch is now known as ykarel | 09:41 | |
| openstackgerrit | Alfredo Moralejo proposed openstack/project-config master: Revert "Use virtualenv --seeder=pip so that libs are accessible" https://review.opendev.org/707346 | 09:41 |
| openstackgerrit | Alfredo Moralejo proposed openstack/project-config master: Revert "Use virtualenv --seeder=pip so that libs are accessible" https://review.opendev.org/707346 | 09:42 |
| *** amoralej has joined #openstack-infra | 09:46 | |
| *** cshen has joined #openstack-infra | 09:56 | |
| *** ociuhandu has joined #openstack-infra | 09:56 | |
| *** derekh has joined #openstack-infra | 10:13 | |
| *** rcernin|lunch has quit IRC | 10:22 | |
| *** kaisers has quit IRC | 10:27 | |
| *** kaisers has joined #openstack-infra | 10:35 | |
| *** rcernin|lunch has joined #openstack-infra | 10:36 | |
| *** cshen has quit IRC | 10:46 | |
| *** pkopec has quit IRC | 10:47 | |
| openstackgerrit | Neil Jerram proposed openstack/project-config master: Remove networking-calico from infrastructure systems https://review.opendev.org/707360 | 10:47 |
| *** sshnaidm_ has joined #openstack-infra | 10:49 | |
| *** sshnaidm has quit IRC | 10:49 | |
| *** sshnaidm_ is now known as sshnaidm | 10:49 | |
| *** pkopec has joined #openstack-infra | 10:56 | |
| *** Nizars has joined #openstack-infra | 11:03 | |
| *** cshen has joined #openstack-infra | 11:03 | |
| *** cshen has quit IRC | 11:05 | |
| *** udesale has quit IRC | 11:06 | |
| *** cshen has joined #openstack-infra | 11:08 | |
| *** zxiiro has quit IRC | 11:13 | |
| *** armax has quit IRC | 11:14 | |
| ysastri | Zuul check failed again now 4th time starting from yesterday. https://zuul.opendev.org/t/openstack/build/3aa56b53f7c24f87b442608e9688df51 | 11:16 |
| *** armax has joined #openstack-infra | 11:19 | |
| *** Lucas_Gray has joined #openstack-infra | 11:27 | |
| amoralej | may i get attention on https://review.opendev.org/#/c/707346/ ? | 11:29 |
| amoralej | centos jobs are currently broken, i expect that to help | 11:30 |
| *** rpittau is now known as rpittau|bbl | 11:33 | |
| *** armax has quit IRC | 11:44 | |
| *** ysastri has quit IRC | 11:47 | |
| mnasiadka | amoralej: I don't know if there's any infra-root in european timezone... | 11:53 |
| amoralej | ack | 11:54 |
| amoralej | thx mnasiadka | 11:54 |
| mnasiadka | although I can confirm that virtualenv 20.0.2 works with six 1.14.0 | 11:54 |
| *** jaosorior has joined #openstack-infra | 12:01 | |
| *** nicolasbock has joined #openstack-infra | 12:06 | |
| *** ysastri has joined #openstack-infra | 12:06 | |
| *** ociuhandu has quit IRC | 12:09 | |
| *** ociuhandu has joined #openstack-infra | 12:10 | |
| *** rfolco has joined #openstack-infra | 12:10 | |
| *** sshnaidm has quit IRC | 12:20 | |
| *** ianychoi has quit IRC | 12:25 | |
| *** rfolco has quit IRC | 12:26 | |
| *** rfolco has joined #openstack-infra | 12:26 | |
| *** sshnaidm has joined #openstack-infra | 12:27 | |
| *** sshnaidm has quit IRC | 12:31 | |
| *** udesale has joined #openstack-infra | 12:34 | |
| *** sshnaidm has joined #openstack-infra | 12:41 | |
| *** jpena is now known as jpena|lunch | 12:49 | |
| *** amoralej is now known as amoralej|lunch | 12:56 | |
| *** rlandy has joined #openstack-infra | 12:57 | |
| *** sshnaidm has quit IRC | 13:02 | |
| *** rpittau|bbl is now known as rpittau | 13:05 | |
| *** jamesmcarthur has joined #openstack-infra | 13:09 | |
| *** ociuhandu has quit IRC | 13:11 | |
| *** ykarel is now known as ykarel|afk | 13:14 | |
| AJaeger_ | ysastri: infra images don't have btrfs, if you need that , you need to set it up. I wonder whether this is a new requirement due to an updated lxc package - or your setup is broken. Best ask on the #openstack-ansible channel | 13:16 |
| *** sshnaidm has joined #openstack-infra | 13:17 | |
| *** sshnaidm has quit IRC | 13:18 | |
| AJaeger_ | amoralej|lunch, sorry missing context to review that - this was done yesterday by some US based team members, let's wait for them | 13:18 |
| *** sshnaidm has joined #openstack-infra | 13:18 | |
| *** dave-mccowan has joined #openstack-infra | 13:19 | |
| jrosser | AJaeger_: its just noisy logging - if there were a btrfs filesystem it would print some info, the code is not smart enough to be quiet when there is no btrfs | 13:24 |
| jrosser | it's not actually what is failing the job at all | 13:24 |
| *** dave-mccowan has quit IRC | 13:26 | |
| *** sshnaidm has quit IRC | 13:28 | |
| *** sshnaidm has joined #openstack-infra | 13:29 | |
| *** derekh has quit IRC | 13:30 | |
| *** Lucas_Gray has quit IRC | 13:31 | |
| fungi | yeah, i'm catching up, looks like ianw got some of the blockers for system-config solved overnight while i was in low-power standby mode | 13:33 |
| *** jpena|lunch is now known as jpena | 13:33 | |
| *** Lucas_Gray has joined #openstack-infra | 13:34 | |
| *** jamesmcarthur has quit IRC | 13:35 | |
| AJaeger_ | jrosser: thanks for explanation, let's tell ysastri about it ^ | 13:37 |
| *** rh-jelabarre has joined #openstack-infra | 13:39 | |
| fungi | ysastri: i don't think that error message has anything to do with your job failure. you're probably better off asking in #openstack-ansible but i suspect the problem is missing selinux module for python: https://zuul.opendev.org/t/openstack/build/553ab6b3936c4215b902fbfd3fd3d1d2/log/job-output.txt#12612 | 13:40 |
| zbr | can we add a six>1.14.0 to https://opendev.org/zuul/zuul-jobs/src/branch/master/roles/ensure-tox/tasks/main.yaml#L11 ? | 13:41 |
| zbr | somehow this line bring incompatible virtualenv with it | 13:42 |
| zbr | personally I would cap the virtualenv to <20, but based on what I seen around, this would not be popular. | 13:42 |
| * AJaeger_ is back offline - too bad internet during a train ride ;( | 13:43 | |
| *** nicolasbock has quit IRC | 13:43 | |
| zbr | the reality is that this simple line brings incompatible virtualenv-six mix | 13:43 |
| fungi | jpena: amoralej|lunch: change 707346 would revert changes we made to how we invoke virtualenv when building centos-8 images, not to how jobs run the virtualenv command on centos-8 images. i agree the revert should be safe now, but a link to a job failure would be good so we can see what's actually breaking | 13:44 |
| fungi | as i doubt that particular revert is going to solve whatever job problems you're seeing | 13:45 |
| *** jamesmcarthur has joined #openstack-infra | 13:45 | |
| jpena | fungi: it's not just a job, it's diskimage-buider failing to build images. See https://softwarefactory-project.io/nodepool-log/upstream-centos-8-0000006898.log for an example | 13:46 |
| jpena | "venv: error: unrecognized arguments: --seeder=pip" | 13:46 |
| fungi | jpena: okay, thanks, so it's not a job failure at all | 13:46 |
| fungi | then i agree, we ought to be able to just revert | 13:46 |
| jpena | thanks! | 13:47 |
| fungi | though i find it interesting that software factory is utilizing elements from our project-config repo | 13:47 |
| fungi | i don't think this change broke *our* image building | 13:47 |
| fungi | because we install virtualenv from pypi on our images | 13:47 |
| *** nicolasbock has joined #openstack-infra | 13:47 | |
| jpena | fungi: we are building a centos-8 iamges using the same elements as upstream, for use by tripleo 3rd party ci | 13:48 |
| *** ociuhandu has joined #openstack-infra | 13:48 | |
| fungi | i strongly recommend not reusing our project-config dib elements out of context. they're tuned for our ci environment and make a lot of assumptions about how we build our opendev images | 13:48 |
| jpena | fungi: I think the opendev centos-8 image must be broken too, since centos8/rhel8 always installs virtualenv from package (as per diskimage-builder) | 13:48 |
| fungi | "venv: error: unrecognized arguments: --seeder=pip" https://nb01.openstack.org/centos-8-0000066750.log | 13:50 |
| fungi | yep | 13:50 |
| fungi | we didn't do a good enough job of making sure we replaced the system package | 13:50 |
| *** rcernin|lunch has quit IRC | 13:50 | |
| fungi | anyway, i approved the revert, we should hopefully have new centos-8 images once that lands | 13:51 |
| jpena | ack, thanks! | 13:51 |
| fungi | (and then they build and get uploaded to our providers anyway) | 13:51 |
| *** ociuhandu has quit IRC | 13:52 | |
| *** eharney has quit IRC | 13:55 | |
| ysastri | AJaeger_: Thanks I added the question in #openstack-ansible . I don't have any requirement for btrfs. | 13:56 |
| fungi | ysastri: but also, it looks like the error is earlier in the log, about missing python-libselinux | 13:57 |
| *** amoralej|lunch is now known as amoralej | 13:58 | |
| fungi | jpena: amoralej|lunch: in fact, the problem has nothing to do with how virtualenv is installed *in* the images because we're running virtualenv from outside the image chroot entirely. it broke all our image builds because the version of virtualenv installed system-wide on our nodepool builders is too old to understand the --seeder option: https://nb01.openstack.org/ubuntu-bionic-0000098357.log | 13:58 |
| openstackgerrit | Merged openstack/project-config master: Revert "Use virtualenv --seeder=pip so that libs are accessible" https://review.opendev.org/707346 | 14:00 |
| *** lbragstad has quit IRC | 14:02 | |
| *** lbragstad has joined #openstack-infra | 14:02 | |
| *** ociuhandu has joined #openstack-infra | 14:04 | |
| fungi | jpena: amoralej: okay, now i've convinced myself we're running the virtualenv installed inside the images, but that the version we're invoking even within the context of ubuntu-bionic images is still too old to support the --seeder option | 14:05 |
| fungi | anyway, thanks for catching that and pushing the revert | 14:05 |
| *** derekh has joined #openstack-infra | 14:05 | |
| *** ykarel|afk is now known as ykarel | 14:06 | |
| fungi | i still have concerns about software factory reusing opendev's custom dib elements directly from openstack/project-config, but that's worth a separate discussion | 14:06 |
| *** wissamsawah47 has joined #openstack-infra | 14:06 | |
| *** lbragstad has quit IRC | 14:08 | |
| *** jamesmcarthur has quit IRC | 14:11 | |
| *** jamesmcarthur has joined #openstack-infra | 14:12 | |
| *** wissamsawah47 has quit IRC | 14:12 | |
| *** pgaxatte has quit IRC | 14:12 | |
| openstackgerrit | Zygimantas Matonis proposed openstack/diskimage-builder master: Add Docker CE engine DIB element https://review.opendev.org/707391 | 14:12 |
| fungi | mordred: 705671 and its children need rebases now due to an outdated parent (705670) | 14:14 |
| *** pgaxatte has joined #openstack-infra | 14:14 | |
| *** ociuhandu has quit IRC | 14:15 | |
| *** ian-pittwood has joined #openstack-infra | 14:16 | |
| *** jamesmcarthur has quit IRC | 14:17 | |
| *** tesseract has quit IRC | 14:20 | |
| ian-pittwood | Hello, all. I have been working on helping transition airship/airshipctl to use GitHub Issues from Jira Cloud. As part of this transition, I realized that the project still had OpenStack Storyboard enabled. I created this patchset (https://review.opendev.org/#/c/707262/1) to disable Storyboard and I found in this document | 14:23 |
| ian-pittwood | (https://docs.openstack.org/infra/system-config/jeepyb.html) that it seems like we can indicate our use of GitHub Issues using the "has-issues" option. Unfortunately, the build for my change failed with an error that "has-issues" isn't a valid option. Does anyone know if projects have the ability to set "has-issues"? Am I doing something wrong? | 14:23 |
| fungi | ian-pittwood: i think the has-issues option is specifically for toggling the pull-request closer, though i don't know if unsetting that been exercised in a very long time | 14:26 |
| fungi | lemme go digging in the jeepyb source code | 14:26 |
| ian-pittwood | Thanks, fungi | 14:26 |
| *** Nizars has quit IRC | 14:27 | |
| fungi | regardless, we're not going to set up opendev integration into proprietary issue trackers, so you'll be best off mentioning the location in your readme files | 14:27 |
| zbr | i am trying a tox fix, but we cannot really rely on it https://github.com/tox-dev/tox/pull/1519/files | 14:28 |
| ian-pittwood | Well it's not supposed to be proprietary. Do you just mean external issue trackers? | 14:29 |
| fungi | github is not free/libre open source software | 14:29 |
| ian-pittwood | Well that's a fair point | 14:30 |
| fungi | ian-pittwood: so the has-issues toggle is used in manage_projects.py for a (deprecated) github project autocreation feature (so that we could turn off github issues on the autocreated mirror repos), nowhere else that i can find | 14:31 |
| ian-pittwood | Ok, well thanks for taking a look. I'll just remove it and put a link in the README as you suggested. | 14:31 |
| zbr | fungi: what counts as free/libre open source? can we be more specific? | 14:31 |
| fungi | zbr: osi-approved licenses | 14:31 |
| fungi | software distributed under them | 14:32 |
| ian-pittwood | In that same vein, is it possible to make per project gerrit hooks? I think the answer is probably no, but I thought I'd ask | 14:32 |
| *** jaosorior has quit IRC | 14:32 | |
| fungi | zbr: github doesn't even publish the source code for their services as far as i know, much less do so under an osi-approved free/libre open source license | 14:32 |
| *** jaosorior has joined #openstack-infra | 14:32 | |
| zbr | i asked as I was curious if jira would be acceptable, as the source is available. | 14:33 |
| fungi | ian-pittwood: we also want to get rid of the gerrit hook scripts, generally everything they do could be replaced by zuul jobs | 14:33 |
| openstackgerrit | Daniel Bengtsson proposed openstack/cookiecutter master: Update the minversion parameter. https://review.opendev.org/707397 | 14:33 |
| ian-pittwood | Ah, I see. I'll look into that instead. Thanks, fungi. | 14:34 |
| dulek | Folks, can you freeze a gate VM for me an my debugging? Talking about this thing: https://review.opendev.org/#/c/706529/. My SSH key is at https://github.com/dulek.keys. | 14:34 |
| *** sgw has quit IRC | 14:34 | |
| dulek | It's a classic - works locally, but doesn't on the gate due to some networking weirdness. | 14:34 |
| fungi | zbr: atlassian distributes the source code for jira (and confluence) under a proprietary license, last i looked | 14:34 |
| *** Lucas_Gray has quit IRC | 14:35 | |
| zbr | fungi: yep (if nothing changes recently). so if someone wants to integrate with any SaaS service that is not OSI, they would have to write a proxy that OSI. | 14:35 |
| *** dSrinivas has joined #openstack-infra | 14:36 | |
| dSrinivas | ianw: Hi, I am unbale to create the Node in jenkins using this code https://opendev.org/zuul/nodepool/src/tag/0.3.1/nodepool/myjenkins.py#L132 | 14:37 |
| dSrinivas | and it is failing with http://paste.openstack.org/show/789424/ | 14:37 |
| fungi | dulek: sudo zuul autohold --tenant openstack --project openstack/kuryr-kubernetes --job kuryr-kubernetes-tempest-containerized-ipv6 --change 706529 --reason "dulek investigating timeouts in new devstack ipv6 job" --count 1 | 14:37 |
| fungi | dulek: once you recheck and the job fails, any infra-root should be able to ssh in and add your ssh key | 14:37 |
| dSrinivas | ianw: Can you please help me to find out the issue. Thank you. | 14:37 |
| fungi | i need to disappear for a tax appointment now, but should be back in a few hours | 14:38 |
| dulek | fungi: Thanks! | 14:38 |
| fungi | ys! | 14:38 |
| fungi | er, you're welcome! | 14:38 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Remove review-dev01.openstack.org https://review.opendev.org/705671 | 14:40 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Use LE certs for Apache https://review.opendev.org/705690 | 14:40 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Update letsencrypt_gid documentation https://review.opendev.org/705878 | 14:40 |
| openstackgerrit | Ian Pittwood proposed openstack/project-config master: [WIP] Use GitHub Issues for airshipctl https://review.opendev.org/707262 | 14:41 |
| openstackgerrit | Ian Pittwood proposed openstack/project-config master: [WIP] Use GitHub Issues for airshipctl https://review.opendev.org/707262 | 14:41 |
| *** eharney has joined #openstack-infra | 14:41 | |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Make small tweaks to launch node README https://review.opendev.org/705673 | 14:42 |
| *** jamesmcarthur has joined #openstack-infra | 14:42 | |
| dSrinivas | fungi: Hi, Can you please help with the issue which i posted previously | 14:43 |
| *** jamesmcarthur has quit IRC | 14:47 | |
| openstackgerrit | Ian Pittwood proposed openstack/project-config master: Disable Storyboard for airshipctl https://review.opendev.org/707262 | 14:52 |
| *** ociuhandu has joined #openstack-infra | 14:53 | |
| *** jaosorior has quit IRC | 14:54 | |
| *** ociuhandu has quit IRC | 14:58 | |
| *** jamesmcarthur has joined #openstack-infra | 15:00 | |
| *** weshay is now known as weshay|ruck | 15:02 | |
| *** chkumar|rover is now known as raukadah | 15:02 | |
| *** sgw has joined #openstack-infra | 15:04 | |
| *** artom has joined #openstack-infra | 15:11 | |
| clarkb | dSrinivas: it hasbeen along time since we interacted with jenkins. It is possible the jenkins api haschanged and is no longer compatible with that library | 15:13 |
| clarkb | however, I would start with double checking your authentication. Also consider zuulv3 whoch doesnt need jenkins | 15:14 |
| *** priteau has joined #openstack-infra | 15:15 | |
| *** lbragstad has joined #openstack-infra | 15:15 | |
| *** sreejithp has joined #openstack-infra | 15:16 | |
| *** jamesmcarthur has quit IRC | 15:16 | |
| *** dtantsur is now known as dtantsur|brb | 15:18 | |
| *** jtomasek has quit IRC | 15:22 | |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Use LE certs for Apache https://review.opendev.org/705690 | 15:23 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Update letsencrypt_gid documentation https://review.opendev.org/705878 | 15:23 |
| mordred | corvus, clarkb: our "install-kubectl" role in system-config installs via snaps - which seem to break a non-zero number of times. we installed kubectl in the zuul images by just installing oc from the tarballs - maybe we should consider that for system-config too to make it simpler? | 15:23 |
| *** lpetrut has quit IRC | 15:24 | |
| frickler | mordred: do you know why it did break, is it when downloading the snap? is there a way we could mirror those like we mirror repos? | 15:25 |
| *** ociuhandu has joined #openstack-infra | 15:25 | |
| mordred | frickler: yeah - it's ... one sec, let me get a link | 15:26 |
| mordred | but yes, I'm sure we could do - but we use snaps for almost nothing, so I really wonder if it's worth the effort to add another mirror system | 15:26 |
| mordred | frickler: https://zuul.opendev.org/t/openstack/build/8f67d7f8346b4cc9bde59bf2749dc9a2/log/job-output.txt#9542 | 15:27 |
| mordred | frickler: doesn't even seem to be a download error as much as just snap derping | 15:27 |
| mordred | oh - no, there it is - store server returned 400 | 15:27 |
| frickler | it might get used in more locations once we move to focal, I'm not sure about the effort involved, just wondering whether that would be a possible option at all | 15:27 |
| mordred | frickler: nod. it's a good question - especially if it's going to be the new hotness in the future | 15:28 |
| openstackgerrit | Merged opendev/system-config master: Make small tweaks to launch node README https://review.opendev.org/705673 | 15:28 |
| frickler | jamespage or coreycb might know | 15:28 |
| *** jamesmcarthur has joined #openstack-infra | 15:31 | |
| *** rpittau is now known as rpittau|afk | 15:31 | |
| frickler | this one doesn't make me too optimistic, though https://forum.snapcraft.io/t/local-mirror-for-snaps/1903 . we might consider installing a specific version locally, but possibly still see the upgrade issue | 15:31 |
| jrosser | is there a short answer to where we are regarding infra images and yesterdays virtualenv 20 problems | 15:31 |
| clarkb | jrosser: I'm not fully caught up yet butimage rebuilds yesterday as well as mordreds base job workaround shouldve fixed most infra side issues | 15:32 |
| clarkb | the 20.0.2 release should also just fix it in general going forward so we can revert fixes carefully now | 15:33 |
| *** ysastri has quit IRC | 15:33 | |
| jrosser | i was trying to decide if it was worth diving in to fix * OSA jobs being broken, or just to hold off for other stuff to land first | 15:33 |
| frickler | jrosser: do you still see broken jobs currently? or are those broken in different ways? | 15:35 |
| clarkb | the 20.0.2 release doesnt fix issues woth sox version conflicts or --version output format changes though | 15:36 |
| clarkb | *with six" | 15:36 |
| *** ian-pittwood has quit IRC | 15:37 | |
| jrosser | frickler: heres a patch where the virtualenv stuff broke after PS1 https://review.opendev.org/#/c/706874/ | 15:38 |
| jrosser | and it's pretty much a bonfire after that | 15:38 |
| jrosser | and a lot of breakage like this https://zuul.opendev.org/t/openstack/build/73b4a388ffb4454a9354f51bc2cbae73/log/job-output.txt#15545 | 15:39 |
| jamespage | frickler: what's the question? snap mirrors? | 15:39 |
| *** cshen has quit IRC | 15:39 | |
| frickler | jamespage: yes, is there a way to do that by now? | 15:40 |
| jamespage | frickler: there is something - lemme find out | 15:41 |
| frickler | jrosser: oh, that last one looks like it may be similar to our puppet-venv issue. afaik the version output has changed, leading to errors parsing it | 15:41 |
| jamespage | not my area of expertise tbh | 15:41 |
| clarkb | jrosser: if that comes from parsing of `vitualenv --version` you'll need to fix that on your end | 15:41 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Install kubectl via openshift client tools https://review.opendev.org/707412 | 15:41 |
| clarkb | frickler: yes, exactly re venv version | 15:42 |
| mordred | frickler, clarkb: ^^ that's just sake of argument - although I'm also interested in learning about snap mirrors | 15:42 |
| jrosser | clarkb: yes seems i need to look at this https://opendev.org/openstack/ansible-role-python_venv_build/src/branch/master/tasks/python_venv_preflight.yml#L24-L41 | 15:43 |
| *** armax has joined #openstack-infra | 15:45 | |
| mordred | frickler, jamespage: it seems like mirrors are not supported. there is an "enterprise snap store proxy" - but it is "free for up to five devices" which makes it sound like something not for us | 15:46 |
| fungi | "enterprise" anything makes me shudder | 15:47 |
| frickler | mordred: is downloading from github actually more stable, though? | 15:47 |
| * fungi is back from his annual tax beating | 15:47 | |
| fungi | clarkb: turns out the image fixes yesterday never happened (and are reverted as of an hour or so ago) | 15:48 |
| fungi | the version of virtualenv we're installing, even in ubuntu-bionic chroots, is too old to support --seeder | 15:48 |
| mordred | frickler: it's what we're doing for this in the zuul images - but if we wanted we could probably pre-download/cache a thing if we need to | 15:48 |
| fungi | clarkb: so image builds all failed on an unrecognized option | 15:49 |
| mordred | frickler: but I honestly don't know - this is me thinking out loud :) | 15:49 |
| *** ykarel is now known as ykarel|afk | 15:49 | |
| mordred | fungi: why are we getting such an old virtualenv? | 15:49 |
| mordred | (also, at this point virtualenv 20.0.2 should also be solid) | 15:50 |
| fungi | mordred: that's a good question, and one i did not have time to look into yet | 15:50 |
| fungi | https://nb01.openstack.org/ubuntu-bionic-0000098357.log | 15:50 |
| fungi | which has now rotated out i guess | 15:51 |
| fungi | that's a 404 | 15:51 |
| mordred | it's because we're using pip-and-virtualenv which installs from packages | 15:51 |
| mordred | wow - I *so* thought we'd switched to installing via get-pip | 15:51 |
| mordred | oh | 15:52 |
| mordred | we do get-pip on not-ubuntu | 15:52 |
| fungi | https://nb01.openstack.org/ubuntu-bionic-0000098359.log shows "venv: error: unrecognized arguments: --seeder=pip" though it's from 14:28z so started before the revert landed | 15:52 |
| mordred | how about we change that element to do the get-pip logic _everywhere_ | 15:52 |
| fungi | i want to say we landed changes not long ago to switch to system-packaged pip on ubuntu because it was "new enough" | 15:53 |
| mordred | that's a bit crazy that we have ubuntu packages vs get-pip | 15:53 |
| mordred | ugh | 15:53 |
| fungi | maybe check git history around there | 15:53 |
| mordred | I will | 15:53 |
| mordred | I really think it's an ongoing source of strife to attempt to use distro pip and virtualenv ... but ignoring that, doing it differently on ubuntu vs non-ubuntu makes reasoning about issues like yesterday much harder | 15:54 |
| mordred | so I'm going to suggest we back that out for whatever reason we did it so that we can have consistency across our images | 15:54 |
| fungi | part of the friction also seems to be that software factory want to directly reuse our custom dib elements from project-config but at the same time install everything from distro packages (they're the ones which brought the image build failures to our attention earlier today) | 15:55 |
| frickler | if I read the log correctly, we install virtualenv==20.0.2 into py2, but the call to "python3 -m venv" is what fails later, maybe we need to run "pip install -U --force-reinstall virtualenv" also with pip3? | 15:56 |
| clarkb | fungi: how did jobs break on that then? | 15:56 |
| *** pgaxatte has quit IRC | 15:56 | |
| fungi | jobs did not break on that | 15:57 |
| mordred | clarkb: jobs only broke on centos | 15:57 |
| clarkb | I see so one subset of image was getting new virtualenv | 15:57 |
| fungi | software factory's nodepool image builds started failing because they use our project-config infra-needs dib elements | 15:57 |
| mordred | fungi: nod. well - fwiw, pip-and-virtualenv element is in dib directly, not in project-config | 15:57 |
| * frickler will be back later | 15:58 | |
| clarkb | did centos rebuild at least? | 15:58 |
| fungi | yeah, but what we patched was our bindep and similar virtualenv builds in infra-needs | 15:58 |
| clarkb | we can revert worlarounds in base if so | 15:58 |
| mordred | but - at this point I kind of think we should make an opendev version of it - and it should be opinionated for opendev | 15:58 |
| fungi | clarkb: centos-8 images were failing on the same error. virtualenv too old to support --seeder | 15:59 |
| fungi | i haven't looked to see if centos-7 updated | 15:59 |
| mordred | fungi: we do not override the pip and virtualenv installation in infra-needs | 15:59 |
| fungi | mordred: we call virtualenv in infra-needs | 15:59 |
| fungi | and we changed how we call it to add --seeder=pip | 15:59 |
| mordred | we also install from pacakges on centos8 | 15:59 |
| clarkb | fungi: it was 7 that had broken jobs | 15:59 |
| mordred | fungi: yes. sorry - I'm talking about a differen thing - I agree with you on that | 16:00 |
| fungi | and software factory uses our infra-needs element in their nodepool config | 16:00 |
| mordred | what I'm saying is that we install virtualenv from distro packages for centos8 and ubuntu but not for centos7 | 16:00 |
| fungi | ahh, yep | 16:00 |
| mordred | and I'm suggesting that we stop trying to install virtualenv and pip from distro packages anywhere | 16:00 |
| mordred | and just use get-pip everywhere since we have to use it on some of our images | 16:01 |
| mordred | so that we'll be consistent everywhere | 16:01 |
| mordred | because yesterday NONE of us knew we had a packages vs get-pip split in how this is installed | 16:01 |
| fungi | i don't contest that. but i think dib used to install virtualenv and pip from pypi everywhere and recently changed to use system packages on platforms which were deemed "new enough" so we should make sure we understand why they did that before proposing a revert of it | 16:01 |
| mordred | agree | 16:01 |
| *** udesale has quit IRC | 16:02 | |
| clarkb | also sf shouldnt use our infra elements | 16:02 |
| clarkb | or at least we dont make that an api | 16:02 |
| fungi | clarkb: yeah, i mentioned that a few times | 16:02 |
| *** udesale has joined #openstack-infra | 16:02 | |
| *** tesseract has joined #openstack-infra | 16:02 | |
| fungi | whether or not we think using that is a good idea, the bug they ran into was one we were also breaking on and just hadn't spotted yet | 16:02 |
| mordred | fungi: the package-installs file in pip-and-virtualenv element hasn't changed in this regard since 2016 | 16:03 |
| *** jamesmcarthur_ has joined #openstack-infra | 16:03 | |
| fungi | huh, interesting | 16:03 |
| mordred | it's possible what you're saying is right - but is just happened a LONG time ago | 16:03 |
| fungi | so maybe we've been installing it from packages since xenial | 16:03 |
| mordred | or maybe we swtiched to using upsream pip-and-virtualenv | 16:03 |
| mordred | and had a fork previously? | 16:03 |
| fungi | also possible | 16:03 |
| mordred | ok. so ... | 16:05 |
| clarkb | fungi: right and in this specific case we had planned to revert anyway once 20.0.2 happened. The problem is if we "fix" this by being consistent we'll break them again because tehy want distro packages | 16:06 |
| *** jamesmcarthur has quit IRC | 16:06 | |
| fungi | we could also consider transitioning to not preinstalling virtualenv (or maybe even pip for that matter) in images and instead do it at job runtime, but that's a deeper change likely to break a bunch of existing assumptions | 16:06 |
| jrosser | from my pov it has been a big surprise to find that in CI jobs i am not testing what end users will encounter on an actual deployment of say, bionic | 16:06 |
| mordred | I retract my earlier statement | 16:06 |
| mordred | jrosser: you mean that we instal from pip vs from packages? | 16:06 |
| jrosser | absolutely | 16:06 |
| jrosser | for things like devstack i can see that can be argued as a non issue | 16:07 |
| clarkb | fungi: the reason it is preinstalled is to avoid building out bindep venv in particular at runtime. I don't know what the cost is of that but it is possible its small enoguh that a runtime hit is fine | 16:07 |
| jrosser | but for deployment projects its a fairly scary thing | 16:07 |
| clarkb | I don't think the os-testr venv is used anymore but that is the other one we have | 16:07 |
| mordred | jrosser: the deployment projects that use packages use packages in the gate | 16:07 |
| mordred | to my knowledge | 16:07 |
| fungi | clarkb: in those cases we could consider either installing virtualenv to an alternative path and calling it directly there, or just uninstalling it when we're done using it within the chroot | 16:08 |
| jrosser | but for OSA, which deploys from source, into venvs, our world is not package centric | 16:08 |
| mordred | yeah. that's right | 16:08 |
| jrosser | and normalising and dealing with the actual differences between distros is the value add that we bring | 16:08 |
| mordred | osa isn't package based, so it _is_ testing what people are going ot be deploying in the real world | 16:08 |
| fungi | python3 -m venv /tmp/bootstrap && /tmp/bootstrap/bin/pip install virtualenv && /tmp/bootstrap/bin/virtualenv /usr/venv/bindep && /usr/venv/bindep/bin/pip install bindep | 16:09 |
| fungi | or whatever | 16:09 |
| clarkb | fungi: right but then you have to install python-venv | 16:09 |
| clarkb | which potentially brings its own conflicts | 16:09 |
| fungi | alternatively `/tmp/bootstrap/bin/virtualenv -p python2.7 /usr/venv/bindep` | 16:09 |
| fungi | yeah, python-venv could also be uninstalled, possibly more cleanly | 16:10 |
| fungi | or python3-venv really | 16:10 |
| mordred | btw - I said wrong thingns earlier - we do pip install virtualenv everywhere | 16:10 |
| mordred | we just install the distro packages FIRST - then we pip install over top of it | 16:10 |
| clarkb | and don't force -U? | 16:10 |
| mordred | we -U | 16:10 |
| fungi | that makes an even bigger mess since pip is going to invalidate a bunch of assumptions from the underlying package manager as well | 16:10 |
| mordred | so I don't know why we didn't have 20.0.2 | 16:11 |
| clarkb | fungi: except that doesn't seem to have happened because we didn't update virtualenv | 16:11 |
| fungi | mordred: are we installing virtualenv under python2 and then invoking the system-packaged version for python3? | 16:11 |
| clarkb | (otherwise --seeder=pip would've worked fine) | 16:11 |
| mordred | but yeah - the comments say we do this so that if someone does "$PKG_MGR install python-virtualenv later it doesn't step on the pip instal" | 16:11 |
| mordred | fungi: we are doing distro versions for both - the pip installing both over top | 16:11 |
| mordred | this is one of the most complex things I've seen in a while | 16:12 |
| fungi | except that apparently folks want to step on the pip install and use distro-packaged versions in some cases | 16:12 |
| mordred | people explicity want to use distro-packaged virtualenv? | 16:12 |
| fungi | that's what they're saying | 16:12 |
| mordred | wow | 16:12 |
| mordred | I can't even | 16:12 |
| fungi | jrosser above makes the point that his users in production are going to install python3-virtualenv from ubuntu's packages on their servers | 16:13 |
| mordred | oh - THAT was the distro comment - got it | 16:13 |
| fungi | and so testing against newer virtualenv invalidates the project's assumptions | 16:13 |
| mordred | I mean | 16:13 |
| mordred | I think those users shouldn't do that | 16:13 |
| jrosser | we take whatever the distro has, as there is no choice about that | 16:13 |
| mordred | but I get it :) | 16:13 |
| clarkb | jrosser: well there is a choice :) | 16:13 |
| jrosser | that is a starting point to build something sensible | 16:13 |
| jrosser | but we never *ever* attempt to mess with the system python with pip | 16:14 |
| jrosser | too many bad times trying to do that | 16:14 |
| fungi | ubuntu ostensibly provides security patches for their python3-virtualenv package and has features built in to apply those updates automatically without sysadmin intervention. i expect some people see that as an indispensable feature | 16:14 |
| mordred | I take the other route and never EVER install python packages from distro packages - but I can respect your approach | 16:14 |
| clarkb | fungi: yup, but also break stdlib venv :/ tehre are definitely trade offs there and which work best for your use case will differ depending on use case likely | 16:15 |
| mordred | shrug. we're not going to agree on this point ... but I can see and understand what the use case is | 16:15 |
| mordred | I honestly do not know how we can possibly support all of our current use cases | 16:15 |
| fungi | i take a middle of the road approach and always install packages from pypi into virtual environments and always call them directly from there so that system packages and pypi packages can coexist in entirely separate contexts | 16:15 |
| mordred | except by ceasing to pre-install pip and virtualenv at all | 16:15 |
| fungi | (and i use system packages to bootstrap the venvs) | 16:15 |
| clarkb | mordred: we can't we have to either support none of them or choose the one that aligns with our needs and ask others to manage | 16:16 |
| jrosser | ^ this is the OSA approach too | 16:16 |
| mordred | clarkb: right. I think I'm on none of them | 16:16 |
| clarkb | fungi: jrosser I use system package to bootstrap a utility venv and then I install virtualenv in that | 16:16 |
| fungi | yep, same here | 16:16 |
| mordred | because the thing we're doing now is super complicated and at the same time is undermining people's testing | 16:16 |
| jrosser | well anyway it's been bothering me a lot realising that bionic != bionic | 16:17 |
| jrosser | i don't know what the answer is though | 16:17 |
| fungi | i basically add a .../lib/virtualenv/bin/virtualenv explicitly to my path and keep it out of the way of the distro's version that way | 16:17 |
| clarkb | fungi: yup same here | 16:17 |
| clarkb | I symlink in #HOME/bin which is in path | 16:17 |
| mordred | jrosser: I think we need to make a plan to stop pre-installing - make a zuul role that does the install in a pre playbook for things like tox that need it | 16:18 |
| mordred | and figure out a plan for our system virtualenvs | 16:18 |
| clarkb | mordred: well we'd still preinstall but then probably uninstall | 16:18 |
| clarkb | (or we'd have to bootstrap bindep from scratch on every job) | 16:18 |
| mordred | yeah - that's fine | 16:18 |
| mordred | I thnik y'all were suggesting using distro virtualenv to get the bindep virtualenv | 16:19 |
| mordred | then distro removing it | 16:19 |
| mordred | yeah? | 16:19 |
| clarkb | that is probably the simplest way to remove things since apt/yum/etc will be good at it | 16:19 |
| mordred | that seems workable | 16:19 |
| mordred | clarkb: same with glean, yeah? | 16:19 |
| mordred | have we figured if anything uses that os-testr venv? | 16:20 |
| clarkb | mordred: I think glean is directly installed | 16:20 |
| clarkb | mordred: I can look into the os-testr venv | 16:20 |
| clarkb | (still unsure on that one) | 16:20 |
| fungi | it would be good to keep in mind that there are two basic classes of virtualenv users for us. those who need virtualenv as part of the test framework, and those whose software uses virtualenv for some function. the former are much less likely to care where virtualenv is coming from | 16:20 |
| clarkb | fungi: fwiw its ^ these changes that are likely to break sf | 16:20 |
| mordred | fungi: yes. I completely agree | 16:21 |
| mordred | fungi: I believe by and large we've focused on the former | 16:21 |
| fungi | but the latter use case can this way be addressed by adding virtualenv to bindep.txt or requirements.txt | 16:21 |
| mordred | but I can see how the results of that would be disturbing to the latter | 16:21 |
| fungi | and having jobs pull in the version they're expecting | 16:21 |
| mordred | fungi: I mean - not if we've overwritten virtualenv with pip already - we do some xargs rm stuff in pip-and-virtualenv :) | 16:22 |
| mordred | we seem to do a LOT of work to prevent people from distro installing virtualenv later and getting distro virtualenv | 16:22 |
| fungi | oh, right, i'm thinking if the pip-and-virtualenv element went away entirely | 16:22 |
| mordred | oh -yeah - TOTALLY | 16:22 |
| mordred | ++ | 16:22 |
| fungi | we could find a way to build the extra virtualenvs we need in our image chroots but still clean up after ourselves so there is no preinstalled pip or virtualenv on those images unless the platform itself would normally provide them | 16:23 |
| clarkb | a surprising amount of stuff uses os-testr-env. http://codesearch.openstack.org/?q=os-testr-env&i=nope&files=&repos= seems like mordred may have tried to start cleaning this up already though: https://opendev.org/zuul/zuul-jobs/src/branch/master/roles/fetch-subunit-output/files/find-subunit2html.sh#L39-L43 | 16:24 |
| clarkb | I think for a first pass on this we'll want to keep that venv | 16:24 |
| fungi | so that folks whose software relies on virtualenv can declare python3-virtualenv in their bindep.txt, perhaps (as long as they're calling our preinstalled /usr/venv/bindep/bin/bindep or whatever) and that provides accurate testing of what they also tell their users to install on that platform | 16:24 |
| *** zxiiro has joined #openstack-infra | 16:24 | |
| clarkb | we also preinstall tox | 16:24 |
| clarkb | which will pull in virtualenv iirc | 16:25 |
| clarkb | my hunch is that this is significantly large that we probably don't awnt to just start tackling it | 16:25 |
| clarkb | and instead fix the images we've got so that bindep works including on centos-7 | 16:25 |
| fungi | right, i think we *can* safely preinstall pip, virtualenv, tox, et cetera to places outside the execution path and the python library import path for bootstrapping purposes | 16:25 |
| clarkb | then have a longer term plan to cleaning this up | 16:25 |
| clarkb | fungi: but then how do jobs find them? | 16:26 |
| mordred | fungi: so install tox into a virtualenv and make a /usr/local/bin/tox symlink? | 16:26 |
| mordred | tox is the big one here | 16:26 |
| clarkb | mordred: if we make the symlink then we still potentially break assumptions about "I'm not using pip stuff" | 16:26 |
| fungi | if people want to use those preinstalled copies they can symlink into their path or call them directly. otherwise they can install a system-wide copy however they'd install other software in their jobs | 16:26 |
| mordred | the other options is just to stop preinstalling bindep or tox and let the zuul pre playbooks handle it | 16:26 |
| fungi | i said up front, this is likely to be disruptive and break a lot of existing assumptions in jobs | 16:27 |
| clarkb | fungi: not only that but the way ansible deals with PATH makes that really painful | 16:27 |
| clarkb | as mnaser recently discovered | 16:27 |
| mordred | yup | 16:27 |
| clarkb | because command doesn't evaluate rc or profile stuff | 16:27 |
| mnaser | hi what did I discover | 16:27 |
| mnaser | ah yes | 16:27 |
| fungi | right, adjusting the path envvar isn't likely a good way to go about it, which is why i suggested jobs could symlink into their path or call them directly | 16:28 |
| mordred | mnaser: we're just revelling in the hell that is pip vs. distro packaging | 16:28 |
| fungi | for example, we already don't put our preinstalled bindep into the system default execution path | 16:28 |
| clarkb | a good first step is likely to make the venvs we do have independent of a longer lived virtualenv install. Then if tox reinstalls virtualenv we can clean that up next | 16:28 |
| cmurphy | i'm seeing "ERROR:root:ImportError: cannot import name ensure_text" virtualenv problems on rocky branches, is that what you guys are working on or is this something else? | 16:29 |
| mordred | clarkb: yeah | 16:29 |
| mordred | cmurphy: yup. that's part of the hell | 16:29 |
| fungi | cmurphy: that one is too-old six with to-new virtualenv | 16:29 |
| clarkb | cmurphy: that is related to six versioning. is this a devstack job? if so frickler has fixes up | 16:29 |
| cmurphy | yay | 16:29 |
| clarkb | cmurphy: https://review.opendev.org/#/c/707109/ rocky devstack fix | 16:29 |
| cmurphy | clarkb: yes devstack https://zuul.opendev.org/t/openstack/build/4cb21274651344d68f3cf81a600008f7/log/job-output.txt | 16:29 |
| clarkb | cmurphy: I think if you recheck it will be good now since that chagne above is merged | 16:30 |
| cmurphy | great so i can rechekc | 16:30 |
| cmurphy | kk | 16:30 |
| cmurphy | ty | 16:30 |
| jrosser | fungi: you made a good observation that there were two classes of virtualenv users earlier | 16:32 |
| jrosser | would one answer be to create a more distro centric version of the images whilst leaving the exsiting one just as is | 16:32 |
| clarkb | we've done split images in the past and it is pain. Also distro centric images can't boot on all clouds | 16:33 |
| jrosser | right, i'm sure this is all a road already trodden :) | 16:33 |
| clarkb | I have very little interest in doubling the number of images we have to maintain and debugging why two different sets of init systems fail to init on other clouds | 16:33 |
| mordred | clarkb, fungi: I have an alternate suggestion | 16:34 |
| mordred | we keep our current system of pre-installing pip, virtualenv and tox via pip for our users who are just using those as a happenstance (which is the larger audience) ... BUT ... | 16:35 |
| *** raukadah has left #openstack-infra | 16:35 | |
| mordred | we simplify it and remove all of the "also install distro packages and then overwrite things to amke it hard for people to install from distro packages later" stuff ... AND ... | 16:35 |
| *** sgw has quit IRC | 16:36 | |
| fungi | mordred: that seems like a good first step regardless, sure | 16:36 |
| mordred | we make a cleanup screipt, since it's now easy to cleanup - that would delete the relevant pip/virtualenv/tox installs from where pip installed them - that folks like OSA can run in a pre playbook in zuul - that would result ina . clean system into which they can apt-get install python-virtualenv and they will get what they are expecting | 16:36 |
| mordred | that way we don't have to try to figure out every assumption people have out there based on our current pattern of pre-providing - but we provide jrosser and crew a *solid* way to get back to a clean system | 16:37 |
| clarkb | jrosser: as a side note, I don't think we are providing the virtualenv>=20.0.0 that is breaking your jobs because our image builds failed when trying to use --seeder=pip | 16:37 |
| clarkb | jrosser: so the cleanup mordred describes may not actually fix this for your jobs | 16:38 |
| clarkb | (and you may want to take a look at the jobs to see where that newer virtualenv comes from if you don't intend to have it) | 16:38 |
| mordred | if people are ok with considering the concept - I can make some WIP patches for us to discuss | 16:38 |
| zbr | mordred: clarkb: https://review.opendev.org/#/c/706371/ please. | 16:38 |
| *** priteau has quit IRC | 16:38 | |
| jrosser | clarkb: ok i will double check that, i think we did test the same (as far as you can outside zuul) thing in a standard cloud-image | 16:39 |
| *** priteau has joined #openstack-infra | 16:39 | |
| clarkb | I've confirmed that the centos-7 image did build and has uploaded | 16:42 |
| clarkb | I think we can revert mordred's workaround | 16:42 |
| *** Sundar has joined #openstack-infra | 16:43 | |
| Sundar | gmann: Hello, please ping me when you have a moment. | 16:43 |
| gmann | Sundar: hi | 16:44 |
| mordred | clarkb: \o/ | 16:44 |
| openstackgerrit | Clark Boylan proposed zuul/zuul master: Simplify virtualenv install and execution https://review.opendev.org/707428 | 16:45 |
| Sundar | gmann: Thanks for pinging back. We had an issue where the py37 and py38 Zuul jobs for Cyborg would time out. Normally, they take only a few min, so it dodn;t seem like increasing the time would help. After some investigating, we found that this patch https://review.opendev.org/#/c/706911/ fixes the problem. It essentially reverts an earlier patch | 16:46 |
| Sundar | that set ignore base python conflict flag. | 16:46 |
| Sundar | gmann: What is not clear is, why this 'ignore base python conflict' flag should cause timeouts. Hope you can throw some light on it. | 16:47 |
| *** lucasagomes has quit IRC | 16:47 | |
| fungi | Sundar: do you have a log from one of the timed-out builds? | 16:49 |
| *** ramishra has quit IRC | 16:49 | |
| gmann | Sundar: 'ignore base python conflict' should not cause timeout. it is flag to avoid the version conflict in case of basepython on common place. for example if you keep basepython-py3 in testenv then, running the tox env with default py version env can conflict. | 16:49 |
| Sundar | Yes, there is nothing in the logs indictaing which tests timed out. The tests mentioned in the logs all passed. | 16:49 |
| gmann | it is warning by tox as of now and than if this flag is not used it might be error when they turn warning to errorr | 16:49 |
| fungi | it's likely that ignore base python conflict resulted in you running the requested version of python, and that your tests actually time out on that version of python | 16:50 |
| clarkb | Sundar: https://tox.readthedocs.io/en/latest/config.html#conf-ignore_basepython_conflict | 16:50 |
| clarkb | Sundar: you have basepython set to python3 | 16:50 |
| Sundar | fungi: Example patch https://review.opendev.org/#/c/698846/10 . Example log: https://cd11bacc403da694b347-14879265802d142fe2b972960013518d.ssl.cf2.rackcdn.com/698846/4/check/openstack-tox-py37/1dcf585/ | 16:51 |
| clarkb | Sundar: that means when you run py37 but python3 is actually python3.6 it will ignore that error and run under python3 anyway | 16:51 |
| *** udesale has quit IRC | 16:51 | |
| clarkb | I think you've fixed this by not running against the expected version of python | 16:51 |
| clarkb | (and the expected version cause sthe timeouts somehow) | 16:51 |
| *** priteau has quit IRC | 16:51 | |
| *** udesale has joined #openstack-infra | 16:52 | |
| *** gyee has joined #openstack-infra | 16:52 | |
| Sundar | clarkb, gmann: I also would not have suspected this change. But, since reverting it, we haven't had any timeouts. Adding it as a base patch before another failing patch would make the latter pass. | 16:52 |
| gmann | Sundar: that patch passing jobs . any failed one | 16:52 |
| fungi | gmann: here's the zuul build page for the failed build log Sundar linked: https://zuul.opendev.org/t/openstack/build/1dcf58536e1f4c89aec287651c5321d8/ | 16:53 |
| Sundar | gmann: Older version of the same patch failed repeatedly, until we rebased it over this reverting patch. | 16:53 |
| *** sgw has joined #openstack-infra | 16:53 | |
| fungi | the log in that example goes silent between 2020-01-18 16:40:31.438574 and 2020-01-18 17:17:28.895598 | 16:54 |
| clarkb | https://zuul.opendev.org/t/openstack/build/98532e50f3bc4f79b4e5574d1f177b48/log/job-output.txt#803 shows that the passing job is using python3.6 as I suspected | 16:55 |
| clarkb | the failing jobs are using the expected python3 that is newer and timeout | 16:55 |
| clarkb | it is passing because it isn't testing what you intend to test | 16:55 |
| fungi | yep, what i also said. basically you have a timeout when testing under python 3.7 and so unsetting that tox option is masking the failure by testing with python3.7 in your 3.7 job | 16:55 |
| fungi | er, by testing with python 3.6 in your 3.7 job | 16:56 |
| Sundar | clarkb, fungi: I see. How would we start debugging the issue with Py 3.7? We don;t know which tests are failing. We have tried doin 'stestr --serial' in tox.ini, but that didn't help. | 16:57 |
| fungi | i suspect you have a unit test in a livelock of some sort under 3.7 but since the job times out before testr can report, it's hard to say | 16:57 |
| clarkb | Sundar: did it still timeout when run serially? | 16:57 |
| Sundar | clarkb: Yes | 16:58 |
| AJaeger_ | config-core, a couple of less exciting changes to review: https://review.opendev.org/706271 https://review.opendev.org/707086 https://review.opendev.org/707210 | 16:58 |
| *** iurygregory has quit IRC | 16:58 | |
| fungi | did you try it on a system which actually has python3.7 installed? | 16:58 |
| fungi | Sundar: ^ | 16:58 |
| clarkb | Sundar: ok, then you should be able to see which test is running when it stops. Then work backward from that test | 16:58 |
| clarkb | Sundar: step 0 is identify where it hangs. Then next is run that test alone with no other tests. See if it hangs or not. If it does great that is probably easier to debug | 16:59 |
| clarkb | and basically work backward from there | 16:59 |
| fungi | unless the job timeout was too short for serialized testing and so got cut short before it reached the spinning test | 16:59 |
| Sundar | I am having trouble installing Py3.7 in my CentOS VMs. Somebody is bringing up an Ubuntu VM with only Py3.7. SO, I haven't tested it locally. | 16:59 |
| gmann | logs seem py3.7 was used this and test running on - https://zuul.opendev.org/t/openstack/build/1dcf58536e1f4c89aec287651c5321d8/log/job-output.txt#459 | 16:59 |
| AJaeger_ | config-core, and some more reviews for zuul-jobs: https://review.opendev.org/706404 https://review.opendev.org/681882 https://review.opendev.org/681603 https://review.opendev.org/704414 | 16:59 |
| clarkb | gmann: you can't use the venv name | 16:59 |
| clarkb | gmann: the venv name will always show the tox target name | 16:59 |
| clarkb | you need to look at the python that was installed in that venv | 16:59 |
| fungi | tox names the venv based on whatever you called it | 16:59 |
| AJaeger_ | config-core, and final review request: https://review.opendev.org/706733 and https://review.opendev.org/706319 https://review.opendev.org/706070 - please help reviewing | 17:00 |
| gmann | not name the version installed in lib/ | 17:00 |
| clarkb | gmann: https://zuul.opendev.org/t/openstack/build/1dcf58536e1f4c89aec287651c5321d8/log/job-output.txt#859 shows it was python3.7 and that job timed out | 17:00 |
| fungi | clarkb: gmann: but yes, that's the failure example | 17:00 |
| fungi | look at the "passing" version from the patch which turns off conflict ignoring | 17:01 |
| fungi | it winds up using python3.6 in a virtualenv named py37 | 17:01 |
| *** lmiccini has quit IRC | 17:01 | |
| openstackgerrit | Merged zuul/zuul-jobs master: Add an ensure-tox test job https://review.opendev.org/706371 | 17:02 |
| gmann | Sundar: let me try py3.7 env locally and run serially. I will be able to do after finishing heat-tempest-plugin issue | 17:02 |
| clarkb | Sundar: gmann another option is to run each test independently | 17:03 |
| clarkb | (that might be really slow depending on the number of tests but is doable and would help pinpoint if a specific test or group of tests are to blame) | 17:03 |
| fungi | or bisect them | 17:03 |
| Sundar | clarkb: "you should be able to see which test is running when it stops" -- this is with '--serial' right? | 17:03 |
| openstackgerrit | Ian Pittwood proposed openstack/project-config master: Disable Storyboard for airshipctl https://review.opendev.org/707262 | 17:03 |
| fungi | Sundar: yes | 17:03 |
| clarkb | Sundar: no I mean in entirely separate test runs | 17:03 |
| clarkb | tox -epy37 --test1 ; tox -epy37 -- test2; and so on | 17:03 |
| clarkb | I would start with serially first and see if you can identify the cause that way. It will be fast | 17:04 |
| fungi | though if you run serially with a long timeout and it eventually hangs on one test, that's likely the culprit | 17:04 |
| clarkb | *faster | 17:04 |
| gmann | clarkb: ok. but both case stestr should log failed test at least | 17:04 |
| clarkb | gmann: not necessarily if the test is hanging | 17:04 |
| fungi | yeah, if control never returns to the caller then you never get a report | 17:04 |
| clarkb | you might not get anything logged (and so have to use subtraction of what was logged against a complete list to narrow the filed) or simply that a test started and has never finished | 17:05 |
| Sundar | clarkb, fungi, gmann: Got it. We will try both (a) using a local env with py37 and running individual tests, and (b) running with -serial and seeing what was the last failing test. | 17:06 |
| Sundar | Thanks a lot to all of you! | 17:06 |
| fungi | Sundar: you're welcome, let us know if you run into any other questions and we're happy to help | 17:07 |
| clarkb | Sundar: remember to run your testing on code that doesn't have the "fix" | 17:07 |
| *** udesale has quit IRC | 17:08 | |
| clarkb | the good news is it hangs when run serially | 17:09 |
| Sundar | Sure, will do | 17:09 |
| clarkb | so probably isn't an inter test process conflict (those are a pain to debug) | 17:09 |
| openstackgerrit | Thierry Carrez proposed openstack/project-config master: check-release-approval: log Gerrit data on error https://review.opendev.org/707432 | 17:11 |
| *** igordc has joined #openstack-infra | 17:17 | |
| clarkb | AJaeger_: I've pulled up those changes. Will review over breakfast. THanks | 17:18 |
| *** tesseract has quit IRC | 17:32 | |
| *** ociuhandu_ has joined #openstack-infra | 17:32 | |
| openstackgerrit | Tristan Cacqueray proposed zuul/zuul master: Gitlab - bootstrap the driver structure + Webhook support https://review.opendev.org/685682 | 17:33 |
| *** evrardjp has quit IRC | 17:34 | |
| *** evrardjp has joined #openstack-infra | 17:34 | |
| *** ociuhandu has quit IRC | 17:36 | |
| *** ociuhandu_ has quit IRC | 17:37 | |
| *** iurygregory has joined #openstack-infra | 17:40 | |
| openstackgerrit | James E. Blair proposed zuul/gcp-authdaemon master: Initial commit https://review.opendev.org/707438 | 17:41 |
| *** xek__ has joined #openstack-infra | 17:42 | |
| *** dkehn has quit IRC | 17:42 | |
| jrosser | clarkb: this is a trivial debug for where virtualenv is coming from in an OSA job http://paste.openstack.org/show/789480/ | 17:43 |
| jrosser | and also likley explains the later failure when parsing the version due to the extra stuff after the version number | 17:44 |
| *** xek_ has quit IRC | 17:44 | |
| clarkb | jrosser: right but what installs taht since apparently our image builds install 3.6.something | 17:45 |
| clarkb | (on bionic) | 17:45 |
| clarkb | I'm guessing some pre-run playbook maybe? | 17:45 |
| jrosser | i have no idea :/ | 17:45 |
| openstackgerrit | Thierry Carrez proposed openstack/project-config master: check-release-approval: log Gerrit data on error https://review.opendev.org/707432 | 17:46 |
| openstackgerrit | Clark Boylan proposed zuul/zuul master: Simplify virtualenv install and execution https://review.opendev.org/707428 | 17:47 |
| *** jamesmcarthur_ has quit IRC | 17:47 | |
| clarkb | I guess what I'm trying to get at is we could do all this work to "fix" the images but then the job side will still do what is unexpected so we need to understand that side too | 17:48 |
| openstackgerrit | Sorin Sbarnea proposed zuul/zuul-jobs master: WIP: Make ensure-tox pass cross-platform https://review.opendev.org/707439 | 17:48 |
| fungi | mordred: see review comment on https://review.opendev.org/705690 you've got something going wrong in the vhost config, looks like | 17:49 |
| noonedeadpunk | actually on pure bionic images no issues arise for OSA and correct virtualenv get's installed (which is provided by system packages) | 17:50 |
| clarkb | noonedeadpunk: we don't have those in the gate though and you aren't running all the zuul prerun playbooks I expect | 17:51 |
| jrosser | we are, but not that do any modifications to installations of things on the host | 17:52 |
| clarkb | noonedeadpunk: jrosser https://zuul.opendev.org/t/openstack/build/73b4a388ffb4454a9354f51bc2cbae73/log/job-output.txt#13842 at that point the version is still old I think beacuse your fact worked | 17:52 |
| jrosser | no thats different | 17:53 |
| jrosser | thats running in an lxc machine container, so is "proper bionic" environment | 17:53 |
| clarkb | I see so different fs context | 17:53 |
| jrosser | the one that fails is running against "aio1" which is the host | 17:53 |
| clarkb | jrosser: I understand that your jobs may not change virtualenv | 17:53 |
| clarkb | but you inherit from other jobs that may | 17:53 |
| openstackgerrit | Ghanshyam Mann proposed openstack/devstack-gate master: Cap virtualenv to a version < 20 https://review.opendev.org/707441 | 17:53 |
| clarkb | something is clearly switchign form distro install to latest from pip between when we build virtualenvs in the image and when your job fails | 17:54 |
| clarkb | I am saying we need to understand that. Can we stop arguing that this is happenign because it is clearly happening | 17:54 |
| clarkb | I don't understand why it is happening just that we need to understand it | 17:54 |
| clarkb | otherwise changing the image may not have the effect you want | 17:55 |
| openstackgerrit | Merged zuul/zuul-jobs master: Add tox env for update-test-platforms https://review.opendev.org/706404 | 17:55 |
| noonedeadpunk | Yeah. I just kinda checked zuul pre gate jobs and didn't find anything related... But, according to https://6c99932face18597ac21-e78425eb2af28756e9f2701e93cfe670.ssl.cf1.rackcdn.com/707212/2/check/openstack-ansible-deploy-aio_metal-ubuntu-bionic/abaa5d2/logs/ara-report/result/79ca9049-ba7f-4b75-b4a5-e08be03ffcd8/ it's preinstalled before our gre-gate script | 17:58 |
| clarkb | jrosser: noonedeadpunk is this virtualenv https://zuul.opendev.org/t/openstack/build/73b4a388ffb4454a9354f51bc2cbae73/log/job-output.txt#3519 the one with the new version? | 17:58 |
| openstackgerrit | Monty Taylor proposed openstack/project-config master: WIP Install pip and virtualenv only from get-pip https://review.opendev.org/707442 | 17:59 |
| clarkb | I'm not sure where https://zuul.opendev.org/t/openstack/build/73b4a388ffb4454a9354f51bc2cbae73/log/job-output.txt#15544 is being evaluated | 17:59 |
| jrosser | it's here https://opendev.org/openstack/ansible-role-python_venv_build/src/branch/master/tasks/python_venv_preflight.yml#L41 | 18:00 |
| noonedeadpunk | or wait... | 18:00 |
| jrosser | and i think that the ansible version test chokes on the extra stuff on --version | 18:01 |
| *** derekh has quit IRC | 18:01 | |
| clarkb | jrosser: noonedeadpunk right that seems to run against aio1 and I'm not sure what that is | 18:02 |
| mordred | clarkb: ^^ there's a sake-of-argument WIP patch doing the thing I think we want | 18:02 |
| clarkb | where virtualenv seems to be happy early in the job is against "ubuntu-bionic" | 18:02 |
| clarkb | "ubuntu-bionic" is what I expect to be our test VM | 18:02 |
| clarkb | but I don't know what aio1 is | 18:02 |
| openstackgerrit | Ghanshyam Mann proposed openstack/devstack-gate master: Cap virtualenv to a version < 20 https://review.opendev.org/707441 | 18:03 |
| noonedeadpunk | Ok, just to short the list of search - at this point virtualenv is already 20.0.1 https://zuul.opendev.org/t/openstack/build/abaa5d23dfb44caeb0220d8f82550624/log/job-output.txt#4972 | 18:03 |
| clarkb | is aio1 an alias for ubuntu-bionic in the ansible inventory? | 18:03 |
| jrosser | it is here https://github.com/openstack/openstack-ansible/blob/master/tests/roles/bootstrap-host/tasks/prepare_hostname.yml#L36 | 18:04 |
| jrosser | and then used in the inventory of the embedded ansible | 18:05 |
| clarkb | ok so it is effectively an alias (localhost has been renamed to aio1) | 18:05 |
| clarkb | mordred: fungi: is it possible the image build is using old virtualenv to install bindep env and friends, but then upgrading virtualenv after the fact? | 18:06 |
| clarkb | that could explain the confusion around "we thought it was always updating" | 18:06 |
| *** igordc has quit IRC | 18:08 | |
| clarkb | jrosser: noonedeadpunk out of curiousity, it seems like the osa job installs virtualenv in a virtualenv | 18:08 |
| clarkb | is that mostly just to be ignored? | 18:08 |
| mordred | clarkb: yes. | 18:09 |
| noonedeadpunk | hum? can you kindly point to that? | 18:09 |
| mordred | clarkb: both because that's likely, and because what's going on has grown so baroque that literally anything is possible at this point | 18:09 |
| clarkb | noonedeadpunk: https://zuul.opendev.org/t/openstack/build/73b4a388ffb4454a9354f51bc2cbae73/log/job-output.txt#3754 installs virtualenv at https://zuul.opendev.org/t/openstack/build/73b4a388ffb4454a9354f51bc2cbae73/log/job-output.txt#4005 | 18:10 |
| *** jpena is now known as jpena|off | 18:11 | |
| clarkb | its weird to me that you'd be relying on system virtualenv if you explictly pull it in (opensatck-ansible has it as a dep) | 18:11 |
| clarkb | but that doesn't explain why the system level virtualenv is unexpectedly new | 18:11 |
| clarkb | I'm going to look at image build logs now | 18:12 |
| mordred | clarkb: my biggest hunch would be that somthing installs tox somewhere | 18:12 |
| openstackgerrit | Merged opendev/system-config master: Get LE certs for review.o.o https://review.opendev.org/705670 | 18:12 |
| openstackgerrit | Merged opendev/system-config master: Remove review-dev01.openstack.org https://review.opendev.org/705671 | 18:12 |
| mordred | clarkb: yeah - infra-packge-needs pip installs tox | 18:13 |
| cmurphy | new issue? ":stderr: ERROR: Package 'stackviz' requires a different Python: 3.5.2 not in '>=3.6'" https://zuul.opendev.org/t/openstack/build/ba16721308514130819d1fdd7c2cb821 | 18:13 |
| mordred | clarkb: which is highly likely to pull in virtualenv - especially if virtualenv is only installed via system packages at that point | 18:13 |
| mordred | clarkb: (obviously verifying that from build logs is a good idea) | 18:14 |
| clarkb | mordred: ya the image logs show it installing 20 | 18:14 |
| clarkb | mordred: now I'm trying to sorto ut why bindep-env didn't work given ^ | 18:14 |
| clarkb | 2020-02-12 17:18:46.606 | Successfully installed appdirs-1.4.3 configparser-4.0.2 contextlib2-0.6.0.post1 distlib-0.3.0 filelock-3.0.12 importlib-metadata-1.5.0 importlib-resources-1.0.2 pathlib2-2.3.5 scandir-1.10.0 six-1.14.0 typing-3.7.4.1 virtualenv-20.0.3 zipp-1.1.0 | 18:14 |
| mordred | yeah - that's weid - tox is installed before bindep | 18:14 |
| clarkb | 2020-02-12 17:18:55.507 | + /usr/bin/python3 -m venv /usr/bindep-env | 18:15 |
| mordred | so bindep-env should be new | 18:15 |
| clarkb | that solves the mystery | 18:15 |
| mordred | yup | 18:15 |
| clarkb | ugh | 18:15 |
| clarkb | its because we don't use virtualenv to make those venvs on all hosts | 18:15 |
| clarkb | mordred: I think that maybe changes the calculus of the above stuff slightly | 18:15 |
| mordred | yeah | 18:15 |
| *** Sundar has quit IRC | 18:15 | |
| clarkb | cmurphy: I think that is different, and probably related to ypthon3 chagnes not virtualenv | 18:16 |
| mordred | like - I think we should focus on removing divergence as a step 0 | 18:16 |
| mordred | because it is *impossible* to keep all this in ones head | 18:16 |
| clarkb | cmurphy: I think stackviz might require older python3 and so pip refuses to install it | 18:16 |
| clarkb | mordred: yes I think we should make things consistent as much as possible, then we can sort out how to make them different | 18:16 |
| openstackgerrit | Jeremy Stanley proposed zuul/zuul master: Flesh out the glossary significantly https://review.opendev.org/704391 | 18:16 |
| clarkb | fungi: ^ not sure if you caught that but the reason seeder=pip didn't work is because we use python3 -mvenv not virtualenv | 18:17 |
| clarkb | not because virtualenv was too old | 18:17 |
| clarkb | virtualenv was plenty new and would have worked if we used virtualenv | 18:17 |
| openstackgerrit | Sorin Sbarnea proposed zuul/zuul-jobs master: ensure-tox: include tox-venv plugin https://review.opendev.org/707237 | 18:17 |
| *** jamesmcarthur has joined #openstack-infra | 18:18 | |
| mordred | clarkb: the dib element does python3 -m venv for python3 and -m virtualenv for python2 | 18:18 |
| noonedeadpunk | clarkb: that's a good point actually | 18:18 |
| clarkb | mordred: that makes sense beacuse no -m venv on python2 | 18:18 |
| *** eharney has quit IRC | 18:18 | |
| openstackgerrit | Monty Taylor proposed openstack/project-config master: WIP Install pip and virtualenv only from get-pip https://review.opendev.org/707442 | 18:19 |
| mordred | clarkb: ^^ that adds the environment setting for DIB_PYTHON_VIRTUALENV but hardcodes it to python3 -m virtualenv | 18:19 |
| fungi | clarkb: got it, so we weren't actually using virtualenv as our virtualenv command | 18:19 |
| clarkb | fungi: correct | 18:19 |
| mordred | clarkb: we have no images without python3 available right? | 18:19 |
| fungi | that makes perfect sense, in a twisted sort of way | 18:19 |
| clarkb | mordred: centos7 is quasi available | 18:20 |
| mordred | clarkb: yeah. there's the epel repo which is where it comes from | 18:20 |
| clarkb | mordred: no its in actual centos 7 now, no epel | 18:20 |
| mordred | oh yeah? NEAT | 18:20 |
| clarkb | but I don't know that it is installed by default | 18:20 |
| mordred | clarkb: that's ok - I've got package-installs.yaml for that | 18:21 |
| mordred | clarkb: also - I tested in a container - but it seems liek get-pip is installing into /usr/local on fedora | 18:21 |
| clarkb | oh neat they must've changed that on fedora then | 18:21 |
| mordred | yup. same on f27 | 18:22 |
| mordred | makes the cleanup script idea easier | 18:22 |
| clarkb | what about centos 8? | 18:23 |
| *** gfidente is now known as gfidente|afk | 18:23 | |
| clarkb | pretty sure it doesn't do that on centos 7 | 18:23 |
| openstackgerrit | Sorin Sbarnea proposed zuul/zuul-jobs master: bindep: use venv when possible https://review.opendev.org/707078 | 18:23 |
| *** ijw has joined #openstack-infra | 18:23 | |
| *** igordc has joined #openstack-infra | 18:23 | |
| clarkb | I really dislike that debuntu decided to remove venv from stdlib | 18:24 |
| ijw | Could someone +A https://review.opendev.org/#/c/706790/ for me? | 18:24 |
| openstackgerrit | Monty Taylor proposed openstack/project-config master: WIP Install pip and virtualenv only from get-pip https://review.opendev.org/707442 | 18:25 |
| mordred | clarkb: checking centos7 now | 18:25 |
| clarkb | mordred: in ^ the rhel stuff installed python2 then 3 in that order which was important accoprding to the comments but now you've gone the other way | 18:26 |
| ijw | Thanks clarkb | 18:26 |
| clarkb | mordred: I think because `pip` will be which ever version installs second | 18:26 |
| clarkb | ugh | 18:27 |
| fungi | the underlying reason for debian separating venv out of the stdlib package is that it depends on ensurepip which is also separated out | 18:27 |
| zbr | why we are not using ansible_python.executable to decide which python to use? | 18:27 |
| clarkb | zbr: because that is compeltely orthogonal | 18:27 |
| clarkb | zbr: all of this happens long before ansible is on the scene | 18:27 |
| *** eernst has joined #openstack-infra | 18:28 | |
| clarkb | mordred: fungi: now that I'ev got more of this mapped into my head. I think we should use python3 -m venv to create all virtualenvs that we create (and this should be possible on centos 7 now with python3) | 18:29 |
| zbr | i missed the context, i was referring to roles from zuul-roles. | 18:29 |
| clarkb | then we are consistent, use stdlib version and don't have to track virtualenv development | 18:29 |
| clarkb | zbr: this is image building | 18:29 |
| mordred | clarkb: /usr/local | 18:30 |
| clarkb | mordred: oh wow | 18:30 |
| clarkb | then for pip, pip3, and pip2 I think we should probably do python3 then python2 on centos7 and xenial. But can do python2 then python3 on everything else? | 18:30 |
| clarkb | I think that will roughly preserve the python2 defautl of those platforms | 18:31 |
| clarkb | while using python3 on newer systems | 18:31 |
| *** jamesmcarthur has quit IRC | 18:31 | |
| *** jamesmcarthur has joined #openstack-infra | 18:31 | |
| mordred | clarkb: oh. wow. this gets funner | 18:32 |
| mordred | clarkb: so - for 3 it all works like we want | 18:32 |
| mordred | clarkb: for 2 - on centos7 - it seems to be installing into /usr/ still | 18:32 |
| clarkb | mordred: probably because the python3 comes from centos8 | 18:32 |
| zbr | prefering python2 on centos7, but prefering python3 on 8. | 18:33 |
| mordred | clarkb: yeah | 18:33 |
| fungi | yeah, red hat's pip2 wants to install into the same path as its rpms | 18:33 |
| fungi | for $reasons | 18:33 |
| clarkb | fungi: re venv on debuntu you don't have to install a bunch of other stdlib that relies on extra packages separately though | 18:33 |
| fungi | long-standing disagreement with the direction debian went with a sieve for dist-packages and site-packages | 18:33 |
| mordred | but ... luckily - python2 on centos7 does not force-bring in python2-pip ... so we can still clean out /usr/bin/pip safely on the cleanup script | 18:33 |
| clarkb | things like curses and readline are just there | 18:33 |
| clarkb | they might be separately packaged but I don't have to think about them because if I install python I get them | 18:34 |
| zbr | yep, rpm systems do not have the luxury found on deb ones, where pip installs to different location than system packages. | 18:34 |
| fungi | clarkb: well, technically you do, but they flag it as important | 18:34 |
| mordred | zbr: they do on centos8 | 18:34 |
| mordred | it works properly for python3 on centos8 | 18:34 |
| zbr | yeah? i did not know that, that is clearly a positive change. | 18:35 |
| mordred | scuse me - it works properly for python3 | 18:35 |
| mordred | zbr: yah! I'm thrilled! | 18:35 |
| fungi | clarkb: the interpreter is in a separate python-minimal package which doesn't include stdlib | 18:35 |
| mordred | I guess python3 gave people the chance to walk back from the previous thing | 18:35 |
| clarkb | fungi: right but why doesnt' installing `python` including venv? | 18:35 |
| clarkb | it includes readline and curses etc | 18:35 |
| fungi | clarkb: because venv needs ensurepip to install things into the venv | 18:35 |
| clarkb | I guess the problem is in dependency lists | 18:35 |
| fungi | and ensurepip is separated out | 18:35 |
| clarkb | fungi: right and readline needs readline and curses curses | 18:36 |
| clarkb | and so on | 18:36 |
| clarkb | you can still have those dependencies | 18:36 |
| clarkb | even if separately packaged | 18:36 |
| fungi | sure, but the python package depends on those | 18:36 |
| *** amoralej is now known as amoralej|off | 18:36 | |
| clarkb | right I guess I'm suggesting the `python` package should depend on its stdlib packages | 18:37 |
| clarkb | (and those packages on their deps) | 18:37 |
| fungi | it doesn't depend on pip because the python packagers for debian have a fundamental disconnect when it comes to installing another package manager as part of the base stdlib | 18:37 |
| fungi | so pip and any other modules which need pip are separated from the stdlib package | 18:37 |
| clarkb | mordred: I'm now trying top puzzle through how we can make this transition a bit more gracefully. I thinkwe can probably do the python3 -mvenv switch first (adding in centos7 python3) | 18:38 |
| *** ykarel|afk is now known as ykarel|away | 18:39 | |
| clarkb | oh wait does what fungi say mean if we install python3-venv it will install system pip? | 18:39 |
| AJaeger_ | thanks for reviewing, clarkb | 18:39 |
| clarkb | in that case we may not want to use python -m venv as it would conflict with that leave system alone and make cleanup early plan | 18:39 |
| clarkb | AJaeger_: I'm slowly getting through themn | 18:39 |
| clarkb | (really slowly) | 18:39 |
| clarkb | fungi: mordred if that is the case then maybe just dealing with `virtualenv` is the best thing to keep the system cleanable | 18:40 |
| fungi | https://packages.ubuntu.com/bionic-updates/python3.6-venv shows it depends on python-pip-whl | 18:40 |
| fungi | see https://packages.ubuntu.com/bionic-updates/all/python-pip-whl/filelist for what that drags in | 18:41 |
| fungi | the argument there has some legitimacy | 18:41 |
| openstackgerrit | Monty Taylor proposed openstack/project-config master: WIP Install pip and virtualenv only from get-pip https://review.opendev.org/707442 | 18:41 |
| openstackgerrit | Merged openstack/project-config master: Remove networking-vpp entry https://review.opendev.org/706790 | 18:41 |
| clarkb | huh so it isn't really installing those packages as much as installing package files that venv can isntall | 18:42 |
| clarkb | neat | 18:42 |
| mordred | clarkb, fungi: ok - I think that version might be solidish - as well as the cleanup script | 18:42 |
| *** jamesmcarthur has quit IRC | 18:42 | |
| AJaeger_ | clarkb: as time permits - otherwise do them tomorrow ;) | 18:42 |
| *** AJaeger_ is now known as AJaeger | 18:42 | |
| mordred | but yeah - I think for the sake of sanity and cleanability - we want to just use virtualenv | 18:42 |
| fungi | clarkb: yeah, so you don't wind up with them in your system context at least | 18:43 |
| mordred | much as I approve of fungi and ianw's position of switching to venv :) | 18:43 |
| *** ahosam has joined #openstack-infra | 18:43 | |
| clarkb | mordred: it has the pip2 vs pip3 as `pip` problem as well as pulling in python-venv may pollute things (though not too bad according to the file list above) | 18:43 |
| *** jamesmcarthur has joined #openstack-infra | 18:43 | |
| clarkb | mordred: also does that install python3 on centos7? | 18:43 |
| mordred | yup | 18:43 |
| clarkb | I still don't understand how that works if so | 18:43 |
| mordred | it just works | 18:43 |
| mordred | yum install python3 works on centos7 | 18:43 |
| clarkb | oh ok | 18:44 |
| clarkb | so we don't need package mapping or anything | 18:44 |
| clarkb | mordred: does the empty map for python3 on centos mean use the original value or "does not exist"? | 18:44 |
| clarkb | mordred: line 10 https://review.opendev.org/#/c/707442/4/nodepool/elements/simple-pip/pkg-map | 18:44 |
| fungi | basically the problem is that venv needs to get pip from somewhere, and pip in turn has a bunch of dependencies (including things like requests), and in order to satisfy the dissident test et cetera you need to be able to make a venv without any network connectivity at all, so python-pip-whl ships built wheels of pip and all its dependency chain, in theory built from the corresponding source packages | 18:45 |
| fungi | in the archive for each of those python librarues | 18:45 |
| fungi | libraries | 18:45 |
| *** eernst has quit IRC | 18:45 | |
| fungi | so that the security team doesn't have to patch separate copies of them | 18:46 |
| fungi | but it's ugly enough that they didn't want to make the python stdlib package depend on all that mess | 18:46 |
| openstackgerrit | Monty Taylor proposed openstack/project-config master: WIP Install pip and virtualenv only from get-pip https://review.opendev.org/707442 | 18:46 |
| mordred | clarkb: it means skip | 18:46 |
| *** stevebaker has joined #openstack-infra | 18:46 | |
| clarkb | ok then I'm confused at how we install pytho3n again :) | 18:47 |
| mordred | clarkb: I just realized we need to not try to uinstall python2-pip on rh - it's hard-required | 18:47 |
| *** tosky has quit IRC | 18:47 | |
| mordred | clarkb: I just cleaned some things up | 18:47 |
| fungi | yes, you can't uninstall it on rhel/centos/fedora because it's part of the system required set | 18:47 |
| *** jamesmcarthur has quit IRC | 18:48 | |
| zbr | so who is against venv? https://review.opendev.org/#/c/707078/ | 18:49 |
| clarkb | zbr: well that change doesn't ensure venv is present | 18:49 |
| mordred | yah - but it's ok really. oh - you know - maybe we should not get-pip for the python2 on centos7 case and just let system pip be pip2 - might be the cleanest | 18:49 |
| clarkb | which we would need to do on debuntu | 18:50 |
| clarkb | zbr: I don't think anyone is against it so much as we're discovering just how complicated all of this is | 18:50 |
| mordred | zbr: also - I think we're discovering that as nice as venv is it opens up a giant can of worms | 18:50 |
| mordred | yeah | 18:50 |
| mordred | I think I'm now officially against it | 18:50 |
| mordred | because of the complexity | 18:50 |
| mordred | sadly | 18:50 |
| zbr | that is weird because venv is considered part of stdlib, so it should be preffered to anything outside the stdlib, like virtualenv. | 18:51 |
| clarkb | zbr: right but it isn't included on many distros (because debian doesn't include it) | 18:51 |
| clarkb | its still packaged but python3 doesn't imply venv is there | 18:51 |
| clarkb | you have to install it separately | 18:51 |
| clarkb | fungi and I just has a large converstaion about it a few minutes ago in scrollback | 18:52 |
| zbr | we can easily do "python -m venv ... || python -m virtualenv ..." | 18:52 |
| clarkb | right, butif we have to cover virtualenv anyway, and it works with all python versions maybe we are better off just doing that for consistency | 18:53 |
| clarkb | then we avoid the extremely confusing situation we found ourselves in where --seeder=pip didn't work on some images but did on others | 18:53 |
| *** ijw has quit IRC | 18:53 | |
| clarkb | in this case Ithink consistency is actually really helpful | 18:53 |
| clarkb | and venv makes it hard to be consistent ebcause it can't python2 | 18:53 |
| zbr | virtualenv>20 will not work on centos 7 and 8, without messing system package python[3]-six, that is known. | 18:54 |
| clarkb | on 8 too? | 18:54 |
| zbr | too | 18:54 |
| *** jamesmcarthur has joined #openstack-infra | 18:54 | |
| clarkb | fwiw I think centos7 + virtualenv20 worked fine on our images | 18:54 |
| clarkb | the key is you have to install six too | 18:54 |
| zbr | it works only if you upgrade six | 18:54 |
| zbr | that was the main issue, and I really do not like overriding six. | 18:55 |
| openstackgerrit | Monty Taylor proposed openstack/project-config master: WIP Install pip and virtualenv only from get-pip https://review.opendev.org/707442 | 18:55 |
| mordred | clarkb: how's that read in the install script ? | 18:55 |
| zbr | while there could be hope for a newer six in centos 8.x I have serious doubts it will ever happen for 7 | 18:55 |
| clarkb | ok, I'm going to argue for consistency here beacuse none of us understood the inconsistent behavior having mixed venv and virtualenv | 18:55 |
| clarkb | and it created a lot of confusion | 18:55 |
| clarkb | I don't think venv allows us to be consistent | 18:55 |
| clarkb | virtualenv does | 18:56 |
| zbr | mainly due to support policy | 18:56 |
| mordred | clarkb: I agree | 18:56 |
| clarkb | once python2 goes away (which won't happen any time soon) we can use venv | 18:56 |
| zbr | and the new feature introduce by virtualenv 20, counts as "consistency", for sure. :D | 18:56 |
| clarkb | zbr: consistency of behavior across platforms | 18:56 |
| mordred | zbr: we're not arguing virtualenv is better ... just simply that we can consistently at least install it across platforms | 18:56 |
| mordred | for sure | 18:56 |
| clarkb | if virtualenv updates it will potentially break but the break will be consistent | 18:56 |
| clarkb | and when we fix it we won't spend a day simply understanding the system | 18:57 |
| mordred | yup. and we won't spend 2 days trying to figure out why it's broken only on e a subset of things in one way and a different subset in a different way | 18:57 |
| *** jamesmcarthur has quit IRC | 18:57 | |
| *** jamesmcarthur has joined #openstack-infra | 18:58 | |
| zbr | i think a lot of pain comes from the decision to bake images with stuff that break the support-contract of the OS. I know well why we do it, but also this makes very hard to blame/coerce the distro to fix a problem. Hacks should be possible but not pre-baked. | 19:00 |
| clarkb | we might also wnat to have an ad hoc meeting with ianw once his day starts as ianw has been involved on the dib side of this | 19:00 |
| clarkb | zbr: ya its a balancing act between doing too much and doing enough that jobs are quick and reliable | 19:01 |
| clarkb | I'm not sure we've managed to correct balance here, but those are the forces pulling in different directions | 19:01 |
| fungi | well, none of the images we make have any os support contract, we don't even start from their published images, we build them up from scratch and a list of packages | 19:01 |
| clarkb | I do think not caching anything (like bindep) would be a step backwards | 19:01 |
| mordred | yup. and I think we're putting together a plan to get closer | 19:01 |
| *** yamamoto has quit IRC | 19:03 | |
| * fungi loves it when a plan comes together | 19:04 | |
| *** ralonsoh has quit IRC | 19:04 | |
| clarkb | from the infrastructure side I think the goals are reasonable consistency so that we can understand behavior even across different distros. Also a reasonable amount of precaching so that tools that are used in most (all) jobs are present and don't put unnecessary load on the mirrors | 19:05 |
| clarkb | From the testing side I agree that not providing an all together different experience is useful. However, we've got to accept that to run on certain clouds there will be differences | 19:05 |
| openstackgerrit | Tristan Cacqueray proposed zuul/zuul master: Gitlab - bootstrap the driver structure + Webhook support https://review.opendev.org/685682 | 19:05 |
| openstackgerrit | Tristan Cacqueray proposed zuul/zuul master: Gitlab - Basic handling of merge_requests event https://review.opendev.org/685990 | 19:05 |
| clarkb | and that requires us to balance those forces against each other | 19:06 |
| fungi | we might ought to transcribe some of that into https://docs.openstack.org/infra/manual/testing.html too | 19:07 |
| clarkb | unfortauntely this is also an area that doesn't get a lot of attention. Its difficult to test, slow going, and really not shiny | 19:07 |
| clarkb | that means there is like a grand total of 4 people that poke at this and our biases are likely to leak through | 19:07 |
| *** jamesmcarthur has quit IRC | 19:08 | |
| *** jamesmcarthur has joined #openstack-infra | 19:09 | |
| openstackgerrit | Merged zuul/zuul master: Simplify virtualenv install and execution https://review.opendev.org/707428 | 19:09 |
| clarkb | fungi: probably a good idea, though we may want to get some resolution on this particular topic first as we may invalidate the current setup shortly | 19:10 |
| fungi | yep, i meant once we figure out what we should be saying, of course ;) | 19:11 |
| *** jamesmcarthur has quit IRC | 19:13 | |
| clarkb | Thinking about base requirements for a second. We need python to be present for glean at the very least | 19:14 |
| clarkb | If we want to precache bindep venvs then we need pip/virtualenv/etc available at image build time to do that. But don't necessarily need them at image runtime? | 19:15 |
| clarkb | mordred's change approximates ^ by making it possible to clean up the extra tools at runtime | 19:15 |
| clarkb | That certainly makes it simpler for the image builds as we can do a transition bit by bit | 19:15 |
| clarkb | but maybe the eventual goal is to remove those tools by default at build time using a similar method? | 19:16 |
| *** eharney has joined #openstack-infra | 19:17 | |
| mordred | clarkb: I think tox pre-installed is the biggie - since that's a requirement for a giant number of our tests and for the most part people just want it to be there and don't care where it came from | 19:19 |
| clarkb | mordred: ya | 19:20 |
| mordred | that's why I think we need pip/virtualenv/etc available in the images - because otherwise we're installing tox 10k times a day which is a waste if we can remove the parts where it's an imposition on clean test environments for some fokls | 19:20 |
| clarkb | mordred: looking at the WIP change again I feel like I'm getting lost a bit and may need to take a break and come back to it with fresh eyes. However I'm still not quite sure https://review.opendev.org/#/c/707442/3..6/nodepool/elements/simple-pip/install.d/04-install-pip is correct as `pip` will be pip3 on older distros | 19:20 |
| mordred | it's dense - definitely good to take a break | 19:21 |
| clarkb | mordred: I think this was a major bit of what the old code was trying to do. Ensureing that pip and pip3 and pip2 were all mapped approprioately to the correct version | 19:21 |
| mordred | I believe the outcome of that should be that pip is NEVER pip3 | 19:21 |
| fungi | it's possible to install tox in a virtualenv and link it from /usr/local/bin | 19:21 |
| clarkb | and I think in simplifying that we may be inadverdently forcing more things to pip3 | 19:21 |
| fungi | then we could do a flag day where we drop the symlink | 19:21 |
| clarkb | fungi: it is, but that doesn't solve the case that jrosser and friends have where they use the tool without specifying a path and it is the "wrong" version | 19:21 |
| mordred | clarkb: rm -f /usr/local/bin/{pip,wheel,virtualenv} should remove the bare pip from /usr/local | 19:22 |
| fungi | it would solve it by eventually having the tool not be in the path at all unless their job installed/linked it somehow | 19:22 |
| clarkb | mordred: won't get-pip.py install pip as pip3 if run under python3? | 19:22 |
| clarkb | fungi: well /usr/local/bin is in your path by default | 19:22 |
| clarkb | at least on debuntu | 19:22 |
| clarkb | or are you saying add that link at runtime | 19:22 |
| mordred | clarkb: yes. but that's what we do first | 19:23 |
| fungi | hence my suggestion of a flag day where we remove that pre-installed symlink | 19:23 |
| mordred | I don't know what that's trying to solve | 19:23 |
| clarkb | mordred: ah yup the rm is what I'm missing | 19:23 |
| mordred | could we state the problem that installing tox in a virtualenv and symlinking then one day not symlinking would be solving? my brain may be too fried to follow or infer that one | 19:24 |
| clarkb | mordred: if we symlink into path then users just running `tox` will get an unexpected version not the version from the distro that they may have installed | 19:24 |
| clarkb | mordred: its basically the same issue we have with virtualenv today, except that we've tidied up the installation into a virtualenv | 19:24 |
| fungi | mordred: solving 1. the python dependencies for tox getting installed system-wide and so polluting the default execution path/python lib import path, and 2. the possibility that folks want to specify their own means of obtaining tox | 19:25 |
| mordred | yeah. I would like to say I do not accept a desire for distro tox as valid | 19:25 |
| mordred | like, I'm a hard -2 on that | 19:25 |
| mordred | tox is a test runner abstraction tool - it does not should not and must not matter to zuul users how tox got installed | 19:25 |
| mordred | 1) is a valid thing to strive for | 19:25 |
| fungi | distro tox may not be valid, but not polluting the system with pypi tox's dependencies by sticking it in a virtualenv is more the pint | 19:26 |
| fungi | point | 19:26 |
| mordred | yes - this is a goal I can get behind in general | 19:26 |
| clarkb | ya putting it in a virtualenv is likely desireable either way | 19:26 |
| clarkb | I'm just trying to reconcile the fact that virtualenv causes all this trouble because it updated and tox can do the same thing | 19:26 |
| clarkb | (though it has been a long time since tox decided to break everyone on release0 | 19:26 |
| clarkb | mordred: new question on https://review.opendev.org/#/c/707442/3..6/nodepool/elements/simple-pip/install.d/04-install-pip that is going to make `pip` always python2 right? | 19:27 |
| fungi | also if we make it widely known that we provide a convenience symlink at /usr/bin/tox to /usr/venv/tox/bin/tox then folks who do care about distro tox can just add a role which does `rm /usr/bin/tox` | 19:27 |
| clarkb | mordred: I think that is a chagne in behavior for bionic and centos-8 | 19:27 |
| fungi | er, at /usr/local/bin/tox i mean | 19:27 |
| clarkb | (this is where consistency and trying to do the right thing for the platform becomes tricky, btu Ibelieve this is what ianw aimed to ahcived in the dib pip stuff) | 19:28 |
| clarkb | maybe the thing to do here is take a break. Get ianw's thoughts on it and see where we end up then | 19:28 |
| clarkb | and then try and break whatever we end up with into as many bite size chunks as possible | 19:29 |
| mordred | clarkb: oh is it? I thought the contract was pip == python2 and pip3 == python3 but that people shoudl stop using pip directly and instead use pythonX -m pip to be sure | 19:29 |
| mordred | clarkb: ++ | 19:29 |
| mordred | I think that's a GREAT idea | 19:29 |
| mordred | because it's been a dense morning and my brain hurts | 19:29 |
| clarkb | fungi: we should symlink bindep simiarly too | 19:30 |
| clarkb | (going back to consistency, if we are going to do virtuaelnvs for utilties and have them accessible by jobs lets get them in PATH consistently too) | 19:30 |
| fungi | clarkb: if that's a pattern we want to retain anyway. right now we expect jobs to know where to find bindep (or install it themselves) because we don't link it into the path | 19:30 |
| *** jamesmcarthur has joined #openstack-infra | 19:30 | |
| clarkb | ya, we could transition away from that. | 19:31 |
| clarkb | That way we don't have to remember these things operate differenlty :) | 19:31 |
| fungi | i was considering the symlink for tox as a transitional step to just expecting jobs to know where we install the tox entrypoint | 19:31 |
| *** jamesmcarthur_ has joined #openstack-infra | 19:31 | |
| fungi | at which point we could drop the symlink for it | 19:31 |
| mordred | I'm not crazy about expecting jobs to know where we install the entrypoint | 19:31 |
| clarkb | hrm | 19:31 |
| clarkb | ya I think that makes generic roles in zuul-jobs a bit tricky | 19:31 |
| mordred | I think that's complexity for not a lot of value | 19:31 |
| mordred | especially for things like tox | 19:32 |
| fungi | but i agree with clarkb, if we want to provide tox in the path by default, then we should do it for tools like bindep as well | 19:32 |
| mordred | ++ | 19:32 |
| fungi | one or the other | 19:32 |
| mordred | I like the symlinking-into-path idea | 19:32 |
| mordred | well - that said ... | 19:33 |
| fungi | at least if we do them as a single simlink, then jobs which do care to override them only need to delete the symlink | 19:33 |
| mordred | there's the case to be made that jobs are already supporting variables like tox_executable because of the case where tox isn't pre-installed and the role installs it into ~/.local and we can't update ansible's path | 19:33 |
| mordred | and we're trying to make the job also able to work without root because people find that important | 19:33 |
| mordred | so - this may be one of those collect all the use cases things | 19:34 |
| clarkb | https://etherpad.openstack.org/p/pTFF4U9Klz I'm starting to take notes there to try and wrap my head around all of this | 19:34 |
| mordred | it's possible fungi's idea of symlinking as a transition is the right one | 19:34 |
| *** jamesmcarthur has quit IRC | 19:35 | |
| mordred | amusingly - this is a discussion we used to have back at MySQL in the 00s ... do we make the packages shipped by MySQL Inc all behave the same so that the experience of a user installing software "from MySQL" the same - or do we make each one behave like things "are supposed to" on each given platform, making things behave like a user of that platform expects but potentially divergent from other | 19:35 |
| mordred | upstream MySQL installs | 19:35 |
| mordred | it turns out there is not a good right answer, sadly | 19:36 |
| fungi | my main concern is to not break jobs which currently expect us to provide tox in the image's default exec path because we've done so since the dawn of openstack ci | 19:36 |
| mordred | yup | 19:37 |
| fungi | but it does seem tractable to wean our jobs off that expectation | 19:37 |
| mordred | it's gonna be a high bar to convince me to break that in the near future :) | 19:37 |
| *** yamamoto has joined #openstack-infra | 19:39 | |
| fungi | well, we used to preinstall piles of stuff in our images, as i'm sure you remember, and we (eventually) were able to stop | 19:41 |
| mordred | yes- thank heavens :) | 19:44 |
| *** stevebaker has quit IRC | 19:50 | |
| *** smarcet has joined #openstack-infra | 19:51 | |
| *** dtantsur|brb is now known as dtantsur|afk | 19:51 | |
| *** stevebaker has joined #openstack-infra | 19:51 | |
| *** yamamoto has quit IRC | 19:52 | |
| *** jamesmcarthur_ has quit IRC | 19:54 | |
| *** ahosam has quit IRC | 19:54 | |
| *** ahosam has joined #openstack-infra | 19:55 | |
| *** smarcet has quit IRC | 19:57 | |
| *** igordc has quit IRC | 19:58 | |
| fungi | mordred: see updated comment on 705690, it looks like you maybe copied erb from a puppet template into that j2 template for the vhost | 19:58 |
| fungi | templating language context switch | 19:59 |
| clarkb | ok I think https://etherpad.openstack.org/p/pTFF4U9Klz is fairly complete with what I managed to keep paged in | 19:59 |
| clarkb | others that have been involved feel free to update | 19:59 |
| clarkb | zbr: fungi mordred jrosser noonedeadpunk ^ | 19:59 |
| clarkb | mordred: fungi I'm kind of thinking maybe we can have pip be top level but then have everything else in some set of virtualenvs and symlinked into path as necessary | 20:00 |
| clarkb | mordred: that slightly changes your WIP change as virtualenv wouldn't need to be installed system wide | 20:00 |
| *** auristor has quit IRC | 20:01 | |
| clarkb | mordred: I think it simplifies the cleanup process (rm pip things and symlinks), while still allowing jobs that need to create a virtualenv or use tox without caring about what version they get | 20:01 |
| *** ahosam has quit IRC | 20:01 | |
| *** ahosam has joined #openstack-infra | 20:01 | |
| *** auristor has joined #openstack-infra | 20:02 | |
| mordred | clarkb: maybe ... I think we should double check jobs using virtualenv | 20:04 |
| clarkb | mordred: virtualenv will happily make a new virtualenv elsewhere even against different python (if you -p) | 20:04 |
| openstackgerrit | Merged openstack/project-config master: check-release-approval: log Gerrit data on error https://review.opendev.org/707432 | 20:04 |
| clarkb | though we should test that on a variety of sytstem as it may differ | 20:05 |
| *** ahosam has quit IRC | 20:05 | |
| mordred | oh - yeah - I get what you're saying | 20:05 |
| mordred | yeah - totally | 20:05 |
| *** ahosam has joined #openstack-infra | 20:06 | |
| * mordred has to run to lunch - will fix 705690 and do another pass on the image patch when he returns | 20:06 | |
| clarkb | I'm on parenting duty again this afternoon fwiw. Will be taking them to swim class | 20:06 |
| openstackgerrit | Tristan Cacqueray proposed zuul/zuul master: Gitlab - bootstrap the driver structure + Webhook support https://review.opendev.org/685682 | 20:16 |
| openstackgerrit | Tristan Cacqueray proposed zuul/zuul master: Gitlab - Basic handling of merge_requests event https://review.opendev.org/685990 | 20:16 |
| fungi | clarkb: mordred: i noted in the pad too, but running pip from a virtualenv really only breaks system-wide `sudo pip install ...` | 20:19 |
| fungi | `pip install --user ...` works fine from a virtualenv pip | 20:20 |
| fungi | and installs into your ~/.local rather than the virtualenv | 20:20 |
| fungi | granted, we likely have a ton of `sudo pip install ...` going on, even beyond devstack | 20:21 |
| fungi | though --root or --prefix could be used to work around that | 20:26 |
| *** armax has quit IRC | 20:45 | |
| *** armax has joined #openstack-infra | 20:48 | |
| *** ChanServ has quit IRC | 20:50 | |
| ianw | just catching up ... | 21:00 |
| ianw | i'm happy to expunge the pip-and-virtualenv element completely ... maybe whatever that breaks is no worse than whatever is broken now | 21:01 |
| clarkb | ianw: I've tried to distill things in https://etherpad.openstack.org/p/pTFF4U9Klz | 21:02 |
| ianw | ok, will read in a sec | 21:03 |
| openstackgerrit | Tristan Cacqueray proposed zuul/zuul master: Gitlab - bootstrap the driver structure + Webhook support https://review.opendev.org/685682 | 21:10 |
| openstackgerrit | Tristan Cacqueray proposed zuul/zuul master: Gitlab - Basic handling of merge_requests event https://review.opendev.org/685990 | 21:10 |
| *** artom has quit IRC | 21:12 | |
| *** ChanServ has joined #openstack-infra | 21:13 | |
| *** orwell.freenode.net sets mode: +o ChanServ | 21:13 | |
| ianw | clarkb: one bit of context i'm missing is why virtualenv/bindep matters for a tool that is supposed to be run basically as a command like bindep/glean? | 21:15 |
| ianw | sorry, virtualenv/venv i mean | 21:15 |
| *** rfolco has quit IRC | 21:18 | |
| *** rcernin has joined #openstack-infra | 21:18 | |
| clarkb | ianw: because the change to fix virtualenv yesterday was only needed on centos7 where we use actual virtualenv and broke everything else. Its mostly about creating consistency where possible yo avoid unexpectedbehaviors | 21:18 |
| clarkb | *to avoid | 21:18 |
| fungi | heading out for an early dinner, but will check back soon | 21:21 |
| TheJulia | out of curiosity, has anyone seen odd spikes in wait time on test vms? | 21:24 |
| TheJulia | recently | 21:24 |
| mordred | clarkb: back - so - can you tell me what you think the update to what Iv'e got should be? we should put tox and virtualenv pip each into virtualenvs - what makes those virtualenvs? | 21:29 |
| mordred | also - morning ianw! | 21:32 |
| mordred | clarkb: I think - from reading the summary in the etherpad - that for those steps to make sense, we still need the WIP patch as a baseline - that is "be able to install consistent pip / virtualenv in such a way that it can then be cleaned up" | 21:34 |
| mordred | but maybe ... maybe that means installing it from distro packages, using that to make the utility venvs, then distro package uninstalling | 21:35 |
| mordred | which - I think dib makes easy, since we can put in a finalize.d step to remove python-virtualenv again, yeah? | 21:35 |
| ianw | i feel like venv's are the right thing for utilities? (not virtualenvs?) | 21:36 |
| mordred | k. that makes sense to me and I can work on it | 21:36 |
| openstackgerrit | Tobias Henkel proposed zuul/zuul master: Report retried builds in a build set via mqtt. https://review.opendev.org/632727 | 21:36 |
| ianw | especially on centos8, where you want to install them under the "system python" which has venv but not virtualenv? | 21:36 |
| mordred | ianw: except we need python2 as well | 21:36 |
| mordred | since we need a pip2 | 21:36 |
| *** ociuhandu has joined #openstack-infra | 21:37 | |
| mordred | so I think just yum install python2-virtualenv python3-virtualenv ; # make virtualenvs ; yum remove python2-virtualenv python3-virtualenv should be fine | 21:37 |
| ianw | what do we need a pip2 for? | 21:38 |
| ianw | sorry i have to do a school run, bib | 21:38 |
| mordred | the idea from clark above is to not have pip or virtualenv installed at the system level at all | 21:38 |
| mordred | but instead to install them into virtualenvs and make symlinks into /usr/local/bin | 21:38 |
| mordred | so that if people (like OSA) don't want to use them, we can just remove the symlinks and it's all good | 21:39 |
| ianw | ohhh, right, sorry i thought we were talking about glean/bindep installs | 21:39 |
| mordred | (this is the extreme version - and might not work because of sudo pip install globally) | 21:39 |
| mordred | ianw: it's probably a better step to just start with those :) | 21:39 |
| mordred | and I agree - we can just use venv for those while we get our feet wet with this | 21:39 |
| ianw | i think we are, right? | 21:40 |
| mordred | if we switch all the way to providing pip2 via a virtualenv and symlink - we'll want to change those | 21:40 |
| mordred | no - we do them differently depending on os | 21:40 |
| ianw | well, only on centos7 | 21:40 |
| mordred | it's venv in some places, virtualenv on others | 21:40 |
| mordred | right - that's differently based on os | 21:40 |
| ianw | https://opendev.org/openstack/diskimage-builder/src/branch/master/diskimage_builder/elements/pip-and-virtualenv/README.rst#environment-variables | 21:40 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Use LE certs for Apache https://review.opendev.org/705690 | 21:44 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Update letsencrypt_gid documentation https://review.opendev.org/705878 | 21:44 |
| mordred | fungi: thanks - that should fix that one hopes | 21:45 |
| *** ociuhandu has quit IRC | 21:48 | |
| *** ociuhandu has joined #openstack-infra | 21:49 | |
| clarkb | mordred: ya I think whatever is easy to install then u install | 21:52 |
| *** ociuhandu has quit IRC | 21:52 | |
| clarkb | TheJulia: no I dont think so | 21:53 |
| openstackgerrit | Merged zuul/zuul master: Don't run jobs if only their file matchers are updated https://review.opendev.org/706399 | 21:53 |
| clarkb | TheJulia: it will vary based on demand and cloud error rate (as we retry per cloud 3 times) | 21:53 |
| openstackgerrit | Tobias Henkel proposed zuul/zuul master: Report retried builds in a build set via mqtt. https://review.opendev.org/632727 | 21:54 |
| TheJulia | clarkb: okay, I'm just looking at some logs and seeing neutron freaking out on message bus messages not responded suddenly correlating to a spike in wait time on the cpu but there really is no way to be sure :( | 21:55 |
| clarkb | oh cpu wait time | 21:56 |
| clarkb | not wait time for atest node | 21:56 |
| mordred | clarkb: ok. I think that sounds good - but I think it's maybe a followup | 21:56 |
| clarkb | mordred: ya I definitely think tackling this in small chunks is the way to go | 21:56 |
| mordred | primarily since we need to be careful with global pip installs | 21:56 |
| openstackgerrit | James E. Blair proposed zuul/zuul master: Don't report enqueue of non-live item https://review.opendev.org/707479 | 22:01 |
| clarkb | TheJulia: for cpu wait time I think we've found some clouds are worse than others for that and I think jobs swapping may have an impact on that (at least as our own noisy nieghbor) | 22:01 |
| *** pkopec has quit IRC | 22:03 | |
| ianw | infra-root: if i could get one more eye on https://review.opendev.org/#/c/706731/ i will monitor the migrated release process today | 22:04 |
| openstackgerrit | Merged zuul/zuul master: Gerrit: add polling support for refs https://review.opendev.org/706138 | 22:06 |
| TheJulia | clarkb: sadly, only about 60M in swap, and tons of free ram, but things did start going sideways right as memory pressure increased | 22:08 |
| clarkb | ianw: done | 22:08 |
| JayF | TheJulia: that makes me wonder if the cloud it's running on is doing ram overcommits, but that's a shot in the dark | 22:10 |
| JayF | TheJulia: (i.e.: it's swapping you on the hyp even if not in the VM) | 22:10 |
| TheJulia | JayF: That could distinctly be the issue, Basically see just enough io wait that messages don't make it across the message bus in time and neutron has a small freakout | 22:11 |
| TheJulia | and then things recover | 22:11 |
| clarkb | JayF: TheJulia it can also be other jobs swapping hard as we are often the only users of the hypervisors | 22:13 |
| clarkb | regular tempest jobs do swap a fair bit | 22:13 |
| JayF | I was just looking at her corrolation between memory pressure and things going upside-down | 22:13 |
| clarkb | its really hard for us to correlate that info though (but theoretically possible as we have nova hostids) | 22:14 |
| clarkb | ah ya that could be | 22:14 |
| TheJulia | yup, this happens to be our standalone job and we have some extra network booting which is super sensitive to neutron just... kind of getting delayed or slightly locked up | 22:14 |
| * TheJulia wonders if the headache of the single test is wroth it | 22:15 | |
| TheJulia | worth | 22:15 |
| mordred | clarkb: pip can uninstall itself | 22:15 |
| ianw | fungi: i started at having a look at migrating reprepro ... i guess gpg key import is step 0. ansible has a apt_gpg thing, but not a generic gpg key import | 22:16 |
| *** dSrinivas has quit IRC | 22:16 | |
| *** sshnaidm has quit IRC | 22:17 | |
| *** sshnaidm has joined #openstack-infra | 22:18 | |
| *** jamesmcarthur has joined #openstack-infra | 22:20 | |
| openstackgerrit | Monty Taylor proposed openstack/project-config master: Install pip and virtualenv only from get-pip https://review.opendev.org/707442 | 22:24 |
| *** jamesmcarthur has quit IRC | 22:24 | |
| clarkb | mordred: neat | 22:24 |
| mordred | clarkb: ok ^^ check that out now | 22:24 |
| *** jamesmcarthur has joined #openstack-infra | 22:25 | |
| prometheanfire | should a notice be sent to all channels that gate is broken (to stop the rechecks)? | 22:25 |
| clarkb | prometheanfire: is it broken? I think we've largely fixed things | 22:26 |
| mordred | prometheanfire: it's broken? | 22:26 |
| mordred | yeah - same question | 22:26 |
| prometheanfire | oh, I was under the impression we needed https://review.opendev.org/#/c/707441 | 22:26 |
| *** slaweq has quit IRC | 22:26 | |
| mordred | prometheanfire: nope. issues have been fixed in upstream virtualenv | 22:27 |
| prometheanfire | all the reqs grenade checks have failed for the last few hours | 22:27 |
| prometheanfire | so good to recheck? | 22:27 |
| clarkb | oh grenade is a d-g thing | 22:27 |
| clarkb | and only applies to jobs on xenial aiui | 22:27 |
| mordred | clarkb: is xenial still broken and we missed it? | 22:28 |
| clarkb | mordred: itsbecauseof the six thing | 22:28 |
| clarkb | or something like that | 22:28 |
| mordred | oh. blerg. is somethign instaling old python-six distro packages? | 22:28 |
| clarkb | not the symlinks | 22:28 |
| mordred | oh - or we're pinning six probably | 22:29 |
| mordred | something something | 22:29 |
| mordred | ? | 22:29 |
| prometheanfire | it is pinned in constraints | 22:29 |
| clarkb | ya or virtualenv doesnt set the lower bound properly and we fail to update | 22:29 |
| prometheanfire | so if you are using that | 22:29 |
| clarkb | ah ok constraints. that would do it I think | 22:29 |
| prometheanfire | heh | 22:29 |
| mordred | here's a recent fail https://zuul.opendev.org/t/openstack/build/2c509d93d10442e1a8765a957e9ef8bd | 22:29 |
| prometheanfire | depends, a surprising amount of stuff in gate doesn't use constraints | 22:29 |
| mordred | clarkb: "./safe-devstack-vm-gate-wrap.sh: line 508: /tmp/ansible/bin/pip3: No such file or directory" | 22:31 |
| mordred | JOY | 22:31 |
| prometheanfire | sounds familiar | 22:31 |
| clarkb | ya that is what 441 aims to fix | 22:31 |
| clarkb | I think it does fix it but I havent double checked an old branch run | 22:32 |
| openstackgerrit | Tristan Cacqueray proposed zuul/zuul master: Gitlab - Add zuul_event_id and set use get_annotated_logger https://review.opendev.org/692799 | 22:32 |
| ianw | mordred: how does 707442 interact with the extant pip-and-virtualenv package? | 22:32 |
| clarkb | its d-g specific though | 22:32 |
| mordred | that's going to need an -U | 22:33 |
| openstackgerrit | Tristan Cacqueray proposed zuul/zuul master: Gitlab - Minimal reporter ables to comment on MR https://review.opendev.org/694346 | 22:33 |
| clarkb | non legacy jobs should be ok I think | 22:33 |
| openstackgerrit | Tristan Cacqueray proposed zuul/zuul master: Gitlab - Implement the note event and the comment trigger action https://review.opendev.org/698964 | 22:33 |
| mordred | ianw: it replaces it - or tries to | 22:33 |
| ianw | mordred: ahh, yeah, i don't think that's going to purge it because simple-init will still bring it in as a dependency | 22:33 |
| clarkb | (because devstack is managing it directly now) | 22:33 |
| mordred | ianw: headdesk | 22:35 |
| ianw | the only other potential big users of this during dib i can think of is ironic-python-agent | 22:36 |
| clarkb | does anyone else have trouble resuming from suspend with newer kernels? | 22:38 |
| ianw | clarkb: that's why i upgrade laptops about once every 10 years, 3 years after getting a laptop it generally starts to suspend, and then you've got about 4 years of it working, and then it will break again :) | 22:40 |
| clarkb | ianw: I'm in the break again trough. This is a thinkpad x240 that is almost 6 years old | 22:40 |
| openstackgerrit | Merged opendev/system-config master: Migrate AFS publishing to mirror-update.opendev.org https://review.opendev.org/706731 | 22:40 |
| clarkb | mordred: it seems to work without -U if you hard set the version https://zuul.opendev.org/t/openstack/build/19807b442cfc44cd9b871ace9c0cc1e7/log/job-output.txt#3032 | 22:41 |
| ianw | mordred: maybe if we attack this from the "how are we going to install glean" side first? | 22:41 |
| clarkb | it semes like if I close lid again and wait for it to sleep then try resume again I have a high rate of success | 22:42 |
| fungi | okay, back from dinner | 22:42 |
| clarkb | prometheanfire: do you know if grenade jobs are failed to changes against master? | 22:43 |
| mordred | clarkb: neat | 22:43 |
| clarkb | prometheanfire: if so then I think this problem is different than we thought | 22:43 |
| fungi | catching up on all the chatty | 22:43 |
| clarkb | but if stable branch specific then ya I'm fairly certain it is the six thing | 22:43 |
| mordred | ianw: well - we could do that, but I don't think that's the problem | 22:44 |
| prometheanfire | clarkb: I can check some recent stuff, how far back should I look? | 22:44 |
| prometheanfire | https://review.opendev.org/707218 failed an hour ago | 22:44 |
| clarkb | prometheanfire: since this morning. Maybe last 8 hours or so | 22:44 |
| mordred | ianw: while there are _many_ use cases, I think the two fundamental problems are a) what is getting installed or not installed right now is super confusing b) we have users with legit use cases for wanting to not have pip installed virtualenv be the virtualenv they get | 22:45 |
| clarkb | hrm that is a bionic job against master | 22:45 |
| clarkb | did virtualenv regress furhter? | 22:45 |
| mordred | there is a 20.0.3 | 22:45 |
| clarkb | or is six 1.14.0 https://zuul.opendev.org/t/openstack/build/60f3cddb8c6449f8b5e1271bd5e9652c/log/job-output.txt#2980 not actually new enough | 22:45 |
| clarkb | 1.14.0 is latest | 22:46 |
| mordred | ianw: I think maybe what we shoudl do is migrate that patch to be one that adds a new element to dib - then make a virtual element that both pip-and-virtualenv and simple-pip provide | 22:46 |
| prometheanfire | ya, that matches UC | 22:46 |
| clarkb | oh | 22:46 |
| clarkb | I betcha it doesn't install a pip3 | 22:46 |
| * clarkb checks | 22:46 | |
| prometheanfire | heh | 22:46 |
| *** eernst has joined #openstack-infra | 22:46 | |
| mordred | just . pip? | 22:46 |
| mordred | I mean - if we're calling the pip in a virtualenv, there is only one python in that virtualenv, it's safe to call it pip | 22:47 |
| clarkb | mordred: ya | 22:47 |
| mordred | clarkb: you want me to do the patch? | 22:47 |
| clarkb | mordred: let me confirm really quickly then sure | 22:47 |
| clarkb | if you can patch ready I'll work on confirming behavior | 22:47 |
| openstackgerrit | Monty Taylor proposed openstack/devstack-gate master: Just use the pip from the virtualenv https://review.opendev.org/707490 | 22:48 |
| clarkb | mordred: confirmed, you get a pip2 but no pip3 | 22:49 |
| mordred | is that doing a python2 virtualenv then? | 22:49 |
| clarkb | and pip2 installs against python3 | 22:49 |
| mordred | wat? | 22:49 |
| clarkb | no I set -p and confirmed bin/python --version and bin/python3 --version | 22:49 |
| ianw | mordred: zero disagreement there about it being confusing and crufty. that was why dropped all installations on centos8 | 22:49 |
| clarkb | ya I think this is a new bug | 22:49 |
| mordred | clarkb: wow | 22:50 |
| clarkb | or new as of virtualenv 20 | 22:50 |
| mordred | well - the patch above will "fix" it | 22:50 |
| mordred | since we already asked for a python version - so just plain pip in the venv pathwill be the rightone | 22:50 |
| mordred | ianw: I mean - I'm *very* impressed by the levels of interwoven use cases the existing code is solving :) | 22:50 |
| clarkb | mordred: yup | 22:50 |
| clarkb | and we can revert gmann's change if it passes testing | 22:51 |
| mordred | clarkb: so that's a python2 virtualenv with -p python3 ? | 22:52 |
| clarkb | mordred: no its a python3 virtualenv with a `pip2` | 22:52 |
| clarkb | I've confirmed this behavior does not exist on virtualenv 16.7.5 | 22:52 |
| clarkb | mordred: run `virtualenv -p python3 foo` and check foo/bin | 22:53 |
| mordred | clarkb: right - but what I mean is - it's a virtualenv installed with python2 | 22:53 |
| clarkb | then do it again for old virtualenv | 22:53 |
| mordred | then that virtualenv was used with --python3 | 22:53 |
| clarkb | mordred: oh maybe. Let me check that | 22:53 |
| mordred | I just reprodiuced it in that case | 22:53 |
| clarkb | ok yup me too | 22:53 |
| clarkb | let me try with a python3 virtualenv | 22:53 |
| mordred | I did - got pip3 | 22:54 |
| *** tkajinam has joined #openstack-infra | 22:54 | |
| mordred | so I thnik that's a new bug - it's matching the pip suffix to the version of python from virtualenv, not the version of python installed IN the virtualenv | 22:54 |
| clarkb | yup agreed | 22:54 |
| mordred | it's in 20.0.2 too | 22:55 |
| clarkb | and also reproduced here | 22:55 |
| mordred | confirmed in 16.7.9 it does what's expected | 22:55 |
| mordred | I'l go file a bug | 22:55 |
| clarkb | what I don't understadn in the d-g case is that should be a python3 installed virtualenv creating a python3 virtualenv | 22:56 |
| clarkb | however, I'm probably wrong about ^ as it explains the failures | 22:56 |
| *** sgw has quit IRC | 22:59 | |
| mordred | virtualenv in the images is python2 | 23:01 |
| mordred | across the board - that's one of the things in the README for pip-and-virtualenv | 23:02 |
| clarkb | gotcha | 23:02 |
| clarkb | and really it shouldnt matter because -p | 23:02 |
| mordred | it's an explicit choice - I think because changing the python it's installed with changes some defaults for tox | 23:02 |
| clarkb | but new bugs :) | 23:02 |
| mordred | yah | 23:02 |
| *** jamesmcarthur has quit IRC | 23:02 | |
| mordred | woohoo - I got a round-numbered issue: https://github.com/pypa/virtualenv/issues/1600 | 23:03 |
| prometheanfire | lol | 23:03 |
| prometheanfire | might want to edit it though | 23:03 |
| prometheanfire | the pip commands installed in the virtualenv are pip and pip3 rather than pip and pip3 | 23:04 |
| *** rh-jelabarre has quit IRC | 23:04 | |
| fungi | that last statement confuses me | 23:04 |
| clarkb | fungi: its quoted from the issue | 23:04 |
| fungi | should one pip3 be a pip2? | 23:04 |
| clarkb | it should be are pip and pip2 rather than pip and pip3 | 23:04 |
| fungi | got it | 23:04 |
| prometheanfire | :D | 23:04 |
| clarkb | I think mordred has to edit it as the filer | 23:04 |
| fungi | and now i | 23:05 |
| fungi | 'm caught up | 23:05 |
| fungi | ianw: i'll be honest, i know nothing about possible ansible plugins/extensions/modules for gnupg/openpgp key management | 23:06 |
| mordred | fixed | 23:06 |
| prometheanfire | subscribed | 23:06 |
| mordred | prometheanfire: also, https://review.opendev.org/#/c/707490/ should fix you | 23:06 |
| prometheanfire | subscribed | 23:07 |
| clarkb | as will the change from gmann (bceause old virtualenv works) | 23:07 |
| clarkb | I think we land both then push a revert for gmann's change | 23:07 |
| clarkb | and if the revert passes testing we can land the revert | 23:07 |
| mordred | ++ | 23:07 |
| mordred | also - it's interesting that we install virtualenv in devstack-gate | 23:07 |
| clarkb | mordred: I think for third party ci | 23:08 |
| clarkb | since we don't control the images they run on | 23:08 |
| *** xek__ has quit IRC | 23:08 | |
| *** eernst has quit IRC | 23:08 | |
| clarkb | I bet a git log search will show we broke some third party ci assuming virtualenv was present :) | 23:08 |
| *** eernst has joined #openstack-infra | 23:08 | |
| mordred | clarkb: ah - nod | 23:08 |
| ianw | mordred: i know a) other fires and b) probably been through this but i'm trying to see there's a path with some notes at end of https://etherpad.openstack.org/p/pTFF4U9Klz | 23:10 |
| ianw | did we find any issues with clarkb's "provide latest virtualenv from a virtualenv" idea? | 23:11 |
| ianw | that's something i'd not considered before, but strikes me as a good way to provide the latest tools without taking over the system, and thus avoiding all this package pinning which has been a constant pain for people | 23:12 |
| clarkb | ianw: we'd still need to be able to install that virtualenv to install virtualenv but in theory we can clean that up at build time so that at job time its less dirty | 23:12 |
| clarkb | and I think it should work because its how I've been using virtualenv locally for a while | 23:13 |
| clarkb | (but my use cases are likely far less diverse than our CI systems) | 23:13 |
| ianw | clarkb: yep, i think that's what i'm getting at -- during the dib build, we have a way to make utility environments (which is forked, but by necessity -- virtualenv on centos7, system-python venv on centos8, python-venv package on debuntu) | 23:14 |
| mordred | yeah - we still have to do the bootstrap stage, and we still need to be able to clean up the bootstrap items | 23:14 |
| ianw | yep, the key is *only* use system packaged virtual environment tools for the utility envs | 23:15 |
| ianw | and then remove them | 23:15 |
| clarkb | ya Ithink that could work | 23:15 |
| mordred | the reason I'm pushing back on the forked thing so hard is that we'll have to develop similarly forked cleanup ... but - if you think that'll be easier I can be totally on board with that | 23:15 |
| mordred | the bigger issue would be pip itself | 23:15 |
| mordred | I don't think that one is workable | 23:15 |
| clarkb | mordred: maybe we punt on pip to start | 23:16 |
| mordred | I think we hav eto | 23:16 |
| clarkb | I agree, I just mean lets not worry about it either | 23:16 |
| mordred | we have too many things (looks at devstack-gate) that assume pip exists and works globally | 23:16 |
| clarkb | we can improve the other tools separately | 23:16 |
| mordred | no - I mean - I agree we have to punt | 23:16 |
| ianw | but we could install the latest pip in a virtualenv and symlink it to /usr/local/bin right? | 23:17 |
| mordred | I still think it's gonna be harder to think about though - although I agree things in venvs will totally work | 23:17 |
| mordred | no | 23:18 |
| mordred | that doesn't work - a pip inside of a virtualenv will install things into that virtualenv | 23:18 |
| clarkb | and then things won't end up in $PATH as expected | 23:18 |
| ianw | ahhhh | 23:18 |
| clarkb | it would probably mostly work if we didn't have to worry about paths | 23:18 |
| mordred | yup | 23:18 |
| ianw | of course | 23:18 |
| mordred | but - I kind think once things are broken via needing non-distro pip - it's cleaner to do no distro virtualenv | 23:19 |
| mordred | beceause it's honestly really easy to clean up from | 23:19 |
| clarkb | fwiw I also don't think this is an emergency. we can take our time to think this through and sleep on it (a few times if necessary) | 23:19 |
| *** Adri2000 has quit IRC | 23:19 | |
| mordred | OH TOTALLY | 23:19 |
| ianw | right ... the only problem with non-distro pip is bugs | 23:19 |
| mordred | ianw: if I just put an element-provides in that simple-pip element that said "pip-and-virtualenv" - wouldn't that cause dep resolution to not pull in pip-and-virtualenv? | 23:20 |
| mordred | ianw: I thnik you meant s/non-distro// | 23:20 |
| mordred | ;) | 23:20 |
| mordred | (if simple-pip was in the list, that is) | 23:21 |
| ianw | mordred: umm, maybe? i'm not sure if the ordering would get that right | 23:21 |
| mordred | I guess that's ordering specific | 23:21 |
| mordred | yeah | 23:21 |
| gmann | clarkb: mordred: ack. | 23:21 |
| clarkb | mordred: ianw: we could create a new simpler-init | 23:21 |
| clarkb | forklift to avoid unexpected conflicts | 23:22 |
| ianw | mordred: certainly we could do something to make it more deterministic and add the element's own element-provides last | 23:22 |
| ianw | clarkb: i think though, even if glean goes out of the picture as a python dependency, you still have other things | 23:22 |
| openstackgerrit | Monty Taylor proposed openstack/diskimage-builder master: Add simple-pip element https://review.opendev.org/707499 | 23:23 |
| ianw | mordred: yeah, distro pip. that's always been the problem; some pip/setuptools bug manifests and blocks all progress, and we can't wait for distros to fix it | 23:23 |
| clarkb | ianw: ya that is a good point | 23:24 |
| mordred | ianw: there's a dib repo based option | 23:24 |
| ianw | ok have to think about that, but it should be fully gate testable in dib | 23:25 |
| mordred | with that - I think we can then do glean, bindep and tox in virtualenvs pretty easily - the cleanup script will remove pip and virtualenv installed by pip from the system for people like OSA ... and putting it in dib we can adjust the simple-init depends | 23:25 |
| mordred | yeah | 23:25 |
| mordred | also - it's possible it's garbage :) | 23:25 |
| *** rlandy is now known as rlandy|bbl | 23:26 | |
| ianw | my inclination is that if we can have "get-pip.py" not involved at all, it's still better; i.e. use distro packages. i understand the point about divergence of tools -- although if we can see a future without centos7 it then really comes down to "install python-venv on debuntu" | 23:28 |
| clarkb | I suppose we could also do python3.6 on centos7 | 23:29 |
| clarkb | do we expect that will cause problems? | 23:29 |
| clarkb | (I'm imagining that delta might also cause jobs to be unhappy for reasons I can't comprehend) | 23:29 |
| ianw | yeah, we could. my only problem is people basing devstack jobs on epel python3 as RDO does not support that and it just seems like a house built on sand. but in terms of, say, having glean running from python3 on centos7 hosts, i don't have issues | 23:31 |
| clarkb | its also no epel anymore | 23:32 |
| clarkb | its in the main distro | 23:32 |
| *** dchen has joined #openstack-infra | 23:32 | |
| mordred | ianw: heh. I think this is just where we have different povs -- my inclination would be if we could never touch distro packages of pip or virtualenv the world would be much better | 23:33 |
| mordred | because the versions of those things are so wildly different | 23:34 |
| clarkb | mordred: I htink ianw's point is that it should be sufficient for boostrapping those tools in a virtuaelnv though | 23:34 |
| clarkb | and we can probably do that consistently? | 23:34 |
| mordred | I understand | 23:34 |
| mordred | but it's just such a complex complex complex thing | 23:34 |
| mordred | and the benefit it gets us over just doing get-pip.py everywhere with no divergence anywhere is fleeting | 23:34 |
| *** eernst has quit IRC | 23:35 | |
| clarkb | ya I think the complexity is my concern as well. Since now we are managing two systems rather than one that might be clunky in some cases | 23:35 |
| clarkb | I guess that is the tradeoff | 23:35 |
| ianw | i think though, we might be in a position that everywhere we care about has venv? which is a package install on debuntu | 23:35 |
| mordred | "on this platform on this verison of python we use this module but not this odule and on this other platform we hav eto install this othe rthing first" | 23:35 |
| mordred | ianw: nope. becaues python2 | 23:35 |
| ianw | but for our utility virtualenv's we don't need to care about that? | 23:35 |
| *** sgw has joined #openstack-infra | 23:36 | |
| mordred | we need to have python2 installed virtualenv on the systems | 23:36 |
| clarkb | can -m venv make a python2 venv? | 23:36 |
| mordred | oh - that's an interesting point | 23:36 |
| mordred | I mean - I still just don't understand the appeal | 23:36 |
| mordred | but if that's what y'all like | 23:36 |
| mordred | the thing I want is for every thing we've got to be identical and to not need to know that on centos7 virtualenv is actually in a box in the corner | 23:37 |
| clarkb | I'm reading docs and I don't see that it can | 23:37 |
| clarkb | I think its implied the venv is the same python running the module | 23:37 |
| ianw | mordred: as in /usr/local/bin/virtualenv creates a python2 virtualenv? | 23:37 |
| mordred | ianw: no -as in /usr/local/bin/virtualenv is itself installed with python2 | 23:37 |
| *** xek has joined #openstack-infra | 23:37 | |
| mordred | that is a current system assumption as to what unqualified virtualenv is - and it's one that was purposefully made | 23:38 |
| ianw | if virtualenv were installed in a venv in /opt, and /usr/local/bin/virtualenv was a script "#!/bin/bash /opt/virtualenv-env/bin/virtualenv -p python2" wouldn't that suffice? | 23:38 |
| ianw | (good god the levels of venv, virtualenv, and virtualenv in venv) | 23:39 |
| mordred | yeah. it might - but _why_ | 23:39 |
| mordred | that's so complex | 23:39 |
| clarkb | if the last -p wins it might work, but ya it starts to get really complicated | 23:39 |
| fungi | ianw: clarkb: if it helps, on my local systems i symlink ~/bin/virtualenv to ~/lib/virtualenv/bin/virtualenv (after a `python3 -m venv ~/lib/virtualenv`) so this is a thing which works generally | 23:40 |
| ianw | i don't really see it as that complex. the latest virtualenv is installed in /opt/virtualenv-env ... that's it | 23:40 |
| mordred | ianw: on your system it's ok for virtualenv to have been installed with python3 | 23:40 |
| fungi | mordred: ianw: pip run from a virtualenv will install into the virtualenv *by default* but can be overridden with command-line options | 23:40 |
| mordred | yes. but those command line options will not be the command line options someone who just runs pip is going to iuse | 23:41 |
| * fungi tried to capture some of this in the etherpad | 23:41 | |
| clarkb | right this is about preserving enough behavior that we don't make things worse | 23:41 |
| *** sgw has quit IRC | 23:41 | |
| ianw | fungi: so in a similar way, possibly /usr/local/bin/pip could be "#!/bin/bash /opt/virtualenv-env/bin/pip --some-option-to-break-venv" ? | 23:41 |
| mordred | other than ickiness from folks with distro backgrounds, what is the issue with using get-pip? | 23:42 |
| mordred | like, what use case does it break? and in what way is it complex? | 23:42 |
| clarkb | my concern would be that it pollutes its deps (I don't know that it does) | 23:42 |
| ianw | the problem has always seem to be that installing distro packages then wipes out get-pip | 23:42 |
| ianw | installed stuff. | 23:42 |
| clarkb | but say requests would be from pypi instead of distro potentially | 23:42 |
| mordred | YES .. but that's why we're adding a cleanup script | 23:42 |
| mordred | it is ABSOLUTELY unacceptable to install python-pip over top of get-pip installed pip | 23:43 |
| mordred | that will ALWAYS break | 23:43 |
| clarkb | right and I think the cleanup script addresses ianw's concern too | 23:43 |
| mordred | yeah | 23:43 |
| mordred | clarkb: could you say more words around "pollutes its deps"? | 23:44 |
| fungi | someone should probably just boot an ubuntu-bionic minimal image and run get-pip.py and then look at the result of `pip list | 23:44 |
| fungi | but pip has runtime dependencies on other python libs like requests | 23:44 |
| clarkb | mordred: python-requests and so on. Does get-pip.py pull those in as normal deps or are they all fully vendored | 23:44 |
| *** sreejithp has quit IRC | 23:45 | |
| *** goldyfruit has quit IRC | 23:45 | |
| clarkb | mordred: I think the cleanup scrip addresses this too fwiw, but you have latest requests because pip pulled it in, but expected the 2 year old distro requests instead | 23:45 |
| fungi | also do the "vendored" libs wind up in the normal python interpreter import search path | 23:45 |
| clarkb | fungi: right that | 23:45 |
| prometheanfire | this is why we restrict pip from installing system libs, pyenv exists as well if we wanted | 23:45 |
| *** goldyfruit has joined #openstack-infra | 23:45 | |
| mordred | clarkb: they're all fully vendored | 23:45 |
| clarkb | prometheanfire: the problem is we can't just say tomorrow everyone needs to use pyenv | 23:45 |
| fungi | prometheanfire: again, th etrick is making sure that gets called by people who don't know where it's installed | 23:46 |
| clarkb | prometheanfire: we need to have a graceful transition | 23:46 |
| mordred | sorry - I shoudl have mentioned - I did exactly what fungi mentioned on ubuntu and fedora and centos earlier today | 23:46 |
| prometheanfire | agreed, that is a longer term thing | 23:46 |
| fungi | mordred: ooh! details? | 23:46 |
| clarkb | mordred: ok if they are all fully vendored then I think having get-pip installed pip is fine if we say "use the cleanup script if you don't want it" | 23:46 |
| mordred | I installed get-pip directly on a clean install - both with and without python-pip installed, and I looked at the actual overlap | 23:46 |
| mordred | the list of packages in the cleanup script is exactly what get installed - and to the best of my abilities it seems as if the cleanup script commands completely clean them up | 23:47 |
| timburke | hi -infra! i've been seeing a bunch of swift failures in the last day or so: http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22AttributeError%3A%20'Namespace'%20object%20has%20no%20attribute%20'with_traceback'%5C%22 (you'll want to change the time period to last two days or so) | 23:47 |
| timburke | it seems to be fixed by virtualenv 20.0.3 -- i was just wondering how often those base images get rebuilt (ie, when i can start rechecking jobs so they'd have the new virtualenv) | 23:47 |
| ianw | mordred: why do we need to *use* the latest/non-distro pip? yes we want to *provide* it, but what do we need to run it for? | 23:47 |
| mordred | it's _entirely_ possible I derped that - but I was basing all of that element on doing stuff in clean containers of each distro to double check | 23:47 |
| mordred | ianw: I want to use it because I want pip to be identical across all of our nodes | 23:48 |
| clarkb | timburke: daily, can you link to a specific job example logs so that we can cross check with known virtualenv fallout? | 23:48 |
| mordred | becuase I dont;' want to be tracking down issues with pip v14 when something derps because some distro decided to patch it because they don't like pip or something | 23:48 |
| mordred | also - we've used latest pip for forever - so I don't understand the value of going to distro pop | 23:49 |
| ianw | mordred: so you feel strongly that glean should be installed with the latest pip? see i feel like that's a very separate thing to what nodes use during CI | 23:49 |
| timburke | clarkb, https://zuul.opendev.org/t/openstack/build/3eef852dfacf4a639500e09534090933 for example | 23:49 |
| mordred | ianw: oh - I don't care about what glean gets installed with | 23:49 |
| *** yamamoto has joined #openstack-infra | 23:49 | |
| mordred | ianw: I care that we have latest pip available for our users to use on their test nodes in tox and whatnot | 23:49 |
| mordred | but I agree - what glean gets installed with is completely separate | 23:50 |
| ianw | mordred: right, we totally agree on that. what i'm concerned about is during the dib build, we install pip with get-pip.py, and then something comes and install a system package, and then we have a big mess | 23:50 |
| clarkb | timburke: and you've confirmed that 20.0.3 fixes that? | 23:50 |
| mordred | ianw: that's already the case - and why we're adding the cleanup script for them to use | 23:51 |
| ianw | that's why i'm trying to think through making the dib build *only* use system packages -- but provide the latest stuff via links, etc | 23:51 |
| mordred | right now what happens is they install distro pip and they don't get it | 23:51 |
| clarkb | timburke: I think 20.0.3 is starting to end up on our images https://zuul.opendev.org/t/openstack/build/19807b442cfc44cd9b871ace9c0cc1e7/log/job-output.txt#3030 shows it was there then we downgrade it due to https://github.com/pypa/virtualenv/issues/1600 | 23:51 |
| timburke | clarkb, sure seems like it would be fixed by https://github.com/pypa/virtualenv/commit/77f0dd80a12105864d8475abb6609dfd693ef5ac#diff-79b6454ee141f415b84cfbe2e037cc60 | 23:51 |
| clarkb | timburke: I think that means you should start to see it getting better | 23:51 |
| ianw | mordred: sort of -- the pip-and-virtualenv element prevents this by pinning the distro packages | 23:51 |
| mordred | yeah - which is actually problematic for OSA | 23:52 |
| mordred | who use virtualenv as a part of their production deploy but who expect to use distro virtualenv and have no way to get it | 23:52 |
| clarkb | mordred: sort of, they explicitly dep on virtualenv from pypi and install it, then don't use it ... | 23:52 |
| ianw | right, but your proposed element doesn't. so from a generic dib POV, if someone uses it, it install pip from get-pip.py, but then their stuff could pull in system pip, and ... urgh | 23:52 |
| clarkb | I'm not sure which side of that they will end up deciding is a bug, but ya | 23:52 |
| mordred | ianw: right - but they MUST use the cleanup script first them | 23:53 |
| clarkb | ianw: oh that is a good point | 23:53 |
| mordred | that is why it's there - it is for their use case | 23:53 |
| ianw | moredred: in a dib element? | 23:53 |
| timburke | 👍 thanks, clarkb! | 23:53 |
| mordred | oh - generic dib | 23:53 |
| mordred | yeah - sorry - that's why this was in project-config to start :) | 23:53 |
| mordred | it's a very good point from a generic dib perspective | 23:53 |
| mordred | I'm purely thinking about infra nodes | 23:54 |
| *** yamamoto has quit IRC | 23:54 | |
| timburke | i'll probably wait a little longer then kick off some rechecks before i sign off for the day | 23:54 |
| fungi | timburke: why do today what can be put off 'til tomorrow? | 23:55 |
| * fungi has a collection of witticisms obtained from thrift store tee-shirts | 23:56 | |
| mordred | ianw: ok. it's dinner time for me - I'm sure we'll be back on this topic tomorrow - since it's a largely sisyphiean problem ;) | 23:56 |
| ianw | mordred: that's why i'm sort of thinking that we a) make sure venv is setup from system packages b) dib elements use DIB_PYTHON_VIRTUALENV and know, basically, they'll get a environment to install stuff c) purge the packages d) setup links/scripts so that /usr/local/bin/{virtualenv|pip} do what people think | 23:56 |
| ianw | mordred: yep, thanks for the back and forth. i like that this is getting attention as it's been a pain point that needed this discussion for a while | 23:56 |
| mordred | ++ | 23:57 |
| mordred | it's also SUCH a dense topic . - sometimes it takes days like this for us all to page it all in | 23:57 |
| fungi | i do think that making sure `python -m venv` works for the available python3 interpreters is a no-brainer, since we install the interpreters from system packages and venv is *supposed* to be part of stdlib... except that same argument doesn't seem to work for pip | 23:57 |
| fungi | in all honesty, it's a pain point which has needed addressing since before openstack | 23:58 |
| fungi | it's just gotten worse over time | 23:58 |
| mordred | we could always return to openstack circa 2010, ditch pip and do everything from deb packages | 23:59 |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!