| *** ysandeep|out is now known as ysandeep|PTO | 00:05 | |
| opendevreview | Ian Wienand proposed opendev/system-config master: infra-prod-bootstrap-bridge: fix typo in playbook name https://review.opendev.org/c/opendev/system-config/+/861805 | 00:11 |
|---|---|---|
| opendevreview | MICHAEL KELLY proposed zuul/zuul-jobs master: helm: Add job for linting helm charts https://review.opendev.org/c/zuul/zuul-jobs/+/861799 | 00:28 |
| *** rlandy is now known as rlandy|out | 00:32 | |
| opendevreview | Merged opendev/system-config master: infra-prod-bootstrap-bridge: fix typo in playbook name https://review.opendev.org/c/opendev/system-config/+/861805 | 00:34 |
| opendevreview | MICHAEL KELLY proposed zuul/zuul-jobs master: helm: Add job for linting helm charts https://review.opendev.org/c/zuul/zuul-jobs/+/861799 | 00:34 |
| opendevreview | Michael Kelly proposed zuul/zuul-jobs master: helm: Add job for linting helm charts https://review.opendev.org/c/zuul/zuul-jobs/+/861799 | 00:37 |
| opendevreview | Michael Kelly proposed zuul/zuul-jobs master: helm: Add job for linting helm charts https://review.opendev.org/c/zuul/zuul-jobs/+/861799 | 02:22 |
| opendevreview | Merged opendev/system-config master: bootstrap-bridge: fix redirect https://review.opendev.org/c/opendev/system-config/+/861673 | 02:24 |
| opendevreview | Ian Wienand proposed opendev/system-config master: bootstrap-bridge: fix production undefined variable https://review.opendev.org/c/opendev/system-config/+/861811 | 02:51 |
| opendevreview | Michael Kelly proposed zuul/zuul-jobs master: helm: Add job for linting helm charts https://review.opendev.org/c/zuul/zuul-jobs/+/861799 | 03:10 |
| opendevreview | Merged opendev/system-config master: bootstrap-bridge: fix production undefined variable https://review.opendev.org/c/opendev/system-config/+/861811 | 04:59 |
| opendevreview | Ian Wienand proposed opendev/system-config master: bootstrap-bridge: use hostname for play https://review.opendev.org/c/opendev/system-config/+/861818 | 05:35 |
| *** pojadhav- is now known as pojadhav | 06:08 | |
| *** bhagyashris is now known as bhagyashris|ruck | 06:24 | |
| opendevreview | Merged opendev/system-config master: bootstrap-bridge: use hostname for play https://review.opendev.org/c/opendev/system-config/+/861818 | 06:27 |
| opendevreview | Ian Wienand proposed opendev/system-config master: bridge-bootstrap: pass hostname for testing https://review.opendev.org/c/opendev/system-config/+/861824 | 07:14 |
| *** jpena|off is now known as jpena | 07:37 | |
| opendevreview | Dmitriy Rabotyagov proposed opendev/system-config master: Fix linuxcontainers proxy URL https://review.opendev.org/c/opendev/system-config/+/861827 | 07:41 |
| *** rlandy|out is now known as rlandy | 10:31 | |
| *** frenzy_friday is now known as frenzyfriday|rover | 12:52 | |
| *** dasm|off is now known as dasm | 13:55 | |
| opendevreview | Felipe Reyes proposed openstack/project-config master: Create 'Backport-Candidate' for Charms repos https://review.opendev.org/c/openstack/project-config/+/861892 | 14:29 |
| *** ykarel is now known as ykarel|afk | 15:27 | |
| opendevreview | David Wilde proposed opendev/irc-meetings master: Change chair contact information for keystone https://review.opendev.org/c/opendev/irc-meetings/+/861894 | 15:28 |
| opendevreview | David Wilde proposed opendev/irc-meetings master: Change chair contact information for keystone https://review.opendev.org/c/opendev/irc-meetings/+/861894 | 15:29 |
| *** dviroel is now known as dviroel|lunch | 15:46 | |
| *** ykarel|afk is now known as ykarel | 15:48 | |
| clarkb | fungi: I had your openstack security ptg meeting on in the background and meetpad seemed to do well | 16:02 |
| clarkb | I've since managed to track this specific pip caching issue down to 22.2.2 hashing a cache dir that includes the sdist sha256 sum in the input to the hashed cache dir path but 22.3 is not | 16:02 |
| fungi | yes, no complaints other than one attendee noted the shared document didn't open automatically for them and they had to open it from the ellipsis menu | 16:03 |
| clarkb | which explains why it doesn't work across pip versions (I don't understand the underlying reason for the different hash inputs yet). However, it still doesn't fully explain why my fix didn't fix it | 16:03 |
| clarkb | its possible my fix isn't leading to a consistent version of pip as I thought I guess. | 16:03 |
| clarkb | upstream has said use `pip wheel` I want to say we didn't do that because it doens't actually get all your deps but may be that is a bad assumption and we should just switch to pip wheel | 16:04 |
| clarkb | however, as a pip user I'm still a bit annoyed that every time I upgrade pip I might end up with a stale useless pip wheel cache | 16:04 |
| clarkb | I think there may still be a bug here despite upstreams assertion the cache isn't portable | 16:05 |
| corvus | clarkb: wasn't one of your attempts to use the same version of pip on both sides? | 16:10 |
| clarkb | corvus: yes, either I'm failing to do that properly in the docker images (one of the other things I need to look at is improving logging of that info in the image builds) or we're able to trigger a similar mismatch of wheel cache dirs across the same version of pip as well | 16:15 |
| clarkb | I was hoping that understanding the delta between 22.2.2 nd 22.3 in calculating the wheel cache dir hash would make it easier to evaluate that in the docker image builds | 16:16 |
| corvus | yeah. a cache that doesn't work on the same host with the same versions isn't much of a cache. | 16:17 |
| fungi | okay, puc is working again for me now | 16:21 |
| clarkb | all this started this mroning when I discovered the `pip cache` command which under 22.3 was able to list and happily show you a valid wheel for netifaces that was built with 22.2.2. This made me fairl certain the issue isn't a format change or some incompatibility. ANd sure enough through some printdebugging its calculating hash paths different acrsso the versions. I think I would | 16:21 |
| clarkb | be ok with "the cache isn't valid because we had to change formats" but "the cache isn't valid bceause we lost your data due to hash change" feels more buggy | 16:21 |
| clarkb | fungi: puc? | 16:21 |
| fungi | packages.ubuntu.com | 16:21 |
| fungi | i was trying to search it for the python3.11 rc, but it's only showing it in kinetic. likely it doesn't index -proposed-updates, only -updates | 16:22 |
| *** dviroel|lunch is now known as dviroel | 16:28 | |
| *** jpena is now known as jpena|off | 16:45 | |
| clarkb | I suspect bad03ef931d9b3ff4f9e75f35f9c41f45839e2a1 broke this | 16:48 |
| clarkb | yup I'm like 98% sure now. The issue is that Link() is instantiated with hashes passed in but not link_hash it appears | 16:53 |
| clarkb | in 22.2.2 it look at Link._hashes set from the hashes input but in 22.3 it does not | 16:53 |
| clarkb | I don't know their hash checking mechanism well enough but it is possible this could lead to not properly checking hashes of downloaded files? | 16:54 |
| fungi | that seems possible, at least | 16:57 |
| corvus | "bad03..." it's RIGHT THERE in the hash! | 16:59 |
| fungi | deadc0d3 | 17:00 |
| corvus | it's like the git equivalent of being rfc3514 compliant | 17:00 |
| clarkb | so basically pip creates these internal models of Links to packages and it builds them using Link.from_json(). That is called against simple api json results for a package. This returns hashes data for each package version. These are recorded but then subsequently ignored. I'm reasonably confident this is a real bug now | 17:02 |
| fungi | the ietf just doesn't get into april fool's jokes like it used to | 17:02 |
| clarkb | and understanding that, I'm less confident that my version stuff in he docker images is doing what I expect since this seems to be an actual bug and not a behavior thing? | 17:03 |
| corvus | clarkb: what "version stuff"? | 17:04 |
| clarkb | corvus: removing the -U pip to ensure that we get a consistent version of pip | 17:07 |
| clarkb | corvus: the testing we've got failed both for the uwsgi image and nodepool with the same error. In my debugging of this issue I'm fairly certain I understand why 22.2.2 and 22.3 don't share cache values. That implies to me that we've still got a pip version mismatch in our docker imags | 17:08 |
| opendevreview | Tony Breeds proposed zuul/zuul-jobs master: [docs][trivial] Remove redundant colon : https://review.opendev.org/c/zuul/zuul-jobs/+/861909 | 17:14 |
| clarkb | I've updated the github issue for the pip thing. I'm going to take a break, but afterwards my next step is adding mroe debugging to our image builds to cross check if we are mixing pip versions | 17:44 |
| opendevreview | Clark Boylan proposed opendev/system-config master: Stop updating pip in our docker assemble script https://review.opendev.org/c/opendev/system-config/+/861800 | 18:14 |
| opendevreview | Clark Boylan proposed opendev/system-config master: Record pip version in our python image builds https://review.opendev.org/c/opendev/system-config/+/861914 | 18:14 |
| clarkb | starting there just collecting data on pip versions | 18:14 |
| corvus | i wonder how silly it would be to attach all of that as image metadata | 18:16 |
| clarkb | corvus: how does that work? But ya having the python --version and pip --version info (and maybe setuptools and wheel and build too) is probably a good idea | 18:17 |
| corvus | clarkb: currently we stick some info in with cli args when building the image. theoretically, i'd look into trying to exfiltrate the info from the image maybe do a "docker run pip freeze"? and then adding it after the fact. but that might make another layer. so i don't know if it's worth it, etc. mostly just brainstorming/spitballing atm. | 18:20 |
| clarkb | ack. I think it is a good idea if feasible | 18:20 |
| clarkb | the thing that made our image build failures weird I noticed is that we update pip everywhere (I dind't realize that yesterday) which means in theory it should all be in sync | 18:25 |
| clarkb | its possible that the issue I found in pip creates problems with 22.3 and 22.3 rather than across versions, though I haven't been able to reproduce that locally yet. | 18:26 |
| clarkb | Anyway that extra logging should give us more details. And at some point I'll also try to look at using pip wheel instead of install with a wheel cache dir | 18:26 |
| fungi | maybe it's when pip calls the plaintext simple api vs the json simple api? | 18:27 |
| fungi | though i can't imagine what would be happening there to cause it to switch | 18:28 |
| clarkb | oh ! I have a theory | 18:38 |
| clarkb | we use a venv created by python3 -m venv in the assemble script. That will install the version of pip included with the python install but not the same as what we update to in the image for /usr/bin/pip ? | 18:39 |
| clarkb | its possible that the real fix here ws simply to rebuild all our imges so they end up on a consistent pip version (22.3.0) | 18:39 |
| clarkb | ya I think that may be it | 18:39 |
| clarkb | because the assemble script runs when we make the new layer on top. I'll have to think this over if there is a good way to avoid this issue generally. | 18:40 |
| clarkb | if 861914 doesn't all fail I think that supports this theory | 18:40 |
| clarkb | then what we do is land 861914 assuming we're happy with the extra logging. Not merge 861800. And then figure out how to make this more robust as a followup | 18:41 |
| fungi | yes, the venv module will use ensurepip's version | 18:41 |
| clarkb | ya so we actually probably do need to always pip install -U, except we also need to do it in the base layer... | 18:42 |
| clarkb | I guess switching to pip wheel might make this problem largely go away assuming we can make that work | 18:42 |
| fungi | we could take full advantage of pip 22.3's new features: splat its zipapp distribution somewhere, then call it and tell it to install packages in our venv rather than relying on in-venv pip | 18:42 |
| clarkb | ya uwsgi builds are succeeding on 861914. infra-root I think landing 861914 willallow us to build docker images for projects that have wheel deps that need to be built | 18:42 |
| clarkb | fungi: ya that is probably worth investigating too | 18:43 |
| fungi | their long-term goal is to no longer install pip in venvs unless the user expressly requests it | 18:44 |
| clarkb | and maybe they will release 22.3.1 with my patch in it :) | 18:44 |
| fungi | since it can now be run independent of any particular python environment (the python interpreting pip doesn't have to to be the same python as the one for the environment being managed) | 18:45 |
| clarkb | https://zuul.opendev.org/t/openstack/build/31f91f8fda1e4a82917cbb125b8d012b/log/job-output.txt#2910-2914 seems to confirm my theory which further backups up that rebuilding the images is the fix | 19:21 |
| clarkb | basically what that shows is when we upgrade pip in the venv we are upgrading from 22.2.2 to 22.3. That means when this first started we were using 22.3 to build the wheel cache then 22.2.2 to install it in install-from-bindep. Then when I stopped doing venv/bin/pip install -U pip what happened was the global pip was updated to 22.3 but we our venv pip was 22.2.2 and we had the same | 19:23 |
| clarkb | problem in the other direction | 19:23 |
| clarkb | simply rebuilding things and getting everything onto 22.3 fixes this | 19:23 |
| clarkb | fungi: frickler: any chance you have time to review https://review.opendev.org/c/opendev/system-config/+/861914 ? Once those image update and promote we can land updates to other images like nodepool | 19:23 |
| corvus | clarkb: more review is great (especially for visibility), but if ppl are busy today, i think we can land that with just our reviews. it's effectively a noop change. | 19:38 |
| fungi | i just now approved 861914 | 19:39 |
| corvus | fungi to the rescue :) | 19:41 |
| opendevreview | Merged opendev/system-config master: Fix linuxcontainers proxy URL https://review.opendev.org/c/opendev/system-config/+/861827 | 20:47 |
| *** dviroel is now known as dviroel|biab | 20:47 | |
| corvus | clarkb: https://zuul.opendev.org/t/openstack/build/47b46b7beb9c4897aea5227065d49cf8 is worth a look | 20:54 |
| clarkb | do we know why it seems to run install-from-bindep multiple times? | 20:58 |
| clarkb | I think this is related to the uwsgi build issues we looked at in the past. THe builder fails to build a wheel for uwsgi then builds uwsgi with setup.py allowing things to continue. Making assemble fail if no wheel is built would be a good improvement. Let me see if I can page the uwsgi build issues back in to see if we can improve that at all | 21:02 |
| clarkb | however I think it is safe to recheck (this is uwsgi specific) I will do so now | 21:02 |
| corvus | ok cool; it was close enough to the error du jour that i wasn't sure | 21:04 |
| clarkb | I believe https://github.com/unbit/uwsgi/pull/2384 is the fix in uwsgi but uwsgi hasn't made a release more recently than that merge | 21:05 |
| clarkb | pip has changed how it reports the error though so its slightly different in the job log | 21:06 |
| clarkb | but I'm like 99% sure its the same thing | 21:06 |
| clarkb | https://github.com/unbit/uwsgi/issues/2471 and https://github.com/unbit/uwsgi/issues/2398 are different similar issues too. | 21:12 |
| opendevreview | Merged opendev/system-config master: bridge-bootstrap: pass hostname for testing https://review.opendev.org/c/opendev/system-config/+/861824 | 21:22 |
| opendevreview | Ian Wienand proposed zuul/zuul-jobs master: stage-output: use links in docs https://review.opendev.org/c/zuul/zuul-jobs/+/861924 | 21:22 |
| opendevreview | Clark Boylan proposed opendev/system-config master: Attempt to make uWsgi builds more reliable https://review.opendev.org/c/opendev/system-config/+/861925 | 21:24 |
| clarkb | thats uwsgi specific and attempts to apply some of the suggestions from that set of issues | 21:24 |
| clarkb | I would remove uwsgi except that I think paste uses it and maybe refstack? | 21:25 |
| clarkb | so it is actually used. Maybe another option is to install distro uwsgi instead? | 21:25 |
| clarkb | 861925 might need to be squashed into 861914 which will make it difficult to tell if it actually helps or not. I hope we don't have to do that | 21:32 |
| *** dviroel|biab is now known as dviroel | 21:59 | |
| opendevreview | Merged opendev/system-config master: Record pip version in our python image builds https://review.opendev.org/c/opendev/system-config/+/861914 | 22:16 |
| clarkb | woot | 22:16 |
| fungi | no small fear | 22:17 |
| fungi | feat too | 22:17 |
| fungi | fear no smalls | 22:17 |
| clarkb | once the promotions happen I can go recheck some changes | 22:17 |
| *** dasm is now known as dasm|off | 22:19 | |
| opendevreview | Ian Wienand proposed opendev/system-config master: Move clouds definitions into control-planes-clouds group https://review.opendev.org/c/opendev/system-config/+/861130 | 22:49 |
| opendevreview | Ian Wienand proposed opendev/system-config master: Run jobs with a jammy bridge.openstack.org https://review.opendev.org/c/opendev/system-config/+/857799 | 22:49 |
| opendevreview | Ian Wienand proposed opendev/system-config master: testinfra: Update selenium calls https://review.opendev.org/c/opendev/system-config/+/858003 | 22:49 |
| opendevreview | Ian Wienand proposed opendev/system-config master: Abstract name of bastion host for testing path https://review.opendev.org/c/opendev/system-config/+/858476 | 22:49 |
| opendevreview | Ian Wienand proposed opendev/system-config master: bootstrap-bridge: use abstracted hostname https://review.opendev.org/c/opendev/system-config/+/861031 | 22:49 |
| opendevreview | Ian Wienand proposed opendev/system-config master: Convert production playbooks to bastion host group https://review.opendev.org/c/opendev/system-config/+/858486 | 22:49 |
| opendevreview | Ian Wienand proposed opendev/system-config master: Run a base test against "old" bridge https://review.opendev.org/c/opendev/system-config/+/860802 | 22:49 |
| opendevreview | Ian Wienand proposed opendev/system-config master: Switch testing bridge name to bridge01.opendev.org https://review.opendev.org/c/opendev/system-config/+/861112 | 22:49 |
| ianw | infra-root: if someone could double check 861130 that would be good. i'm 99% sure it's a no-op moving the variables | 22:56 |
| clarkb | ianw: yup agreed its a noop. But see note about the prod side of that data being elsewhere and maybe needing similar cleanup | 22:59 |
| clarkb | (I think as part of the bridge move we'll probably want to do a fair bit of cleanup of the prod hostvars and group vars just to limit the amount of confusion going forward on the new host and that can be done as a separate step) | 22:59 |
| ianw | ++ | 23:00 |
| *** dviroel is now known as dviroel|out | 23:03 | |
| clarkb | cool the cpu count flag setting on the uwsgi build does seem to limit it to two cpus. I'm sure this slows the jobs down but they still run in under 20 minutes | 23:53 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!