Thursday, 2022-10-20

opendevreviewIan Wienand proposed zuul/zuul-jobs master: stage-output: use links in docs
opendevreviewMerged opendev/system-config master: Move clouds definitions into control-planes-clouds group
*** rlandy is now known as rlandy|out00:53
fungiclarkb: i guess 861800 is no longer needed?00:55
opendevreviewMerged openstack/project-config master: Create 'Backport-Candidate' for Charms repos
opendevreviewMerged opendev/system-config master: Run jobs with a jammy
opendevreviewMerged opendev/system-config master: testinfra: Update selenium calls
*** ysandeep|PTO is now known as ysandeep05:07
opendevreviewRafal Lewandowski proposed openstack/diskimage-builder master: Added example configuration
*** ysandeep is now known as ysandeep|lunch09:08
*** rlandy|out is now known as rlandy10:29
opendevreviewMerged opendev/irc-meetings master: Change chair contact information for keystone
*** ysandeep|lunch is now known as ysandeep10:46
*** dviroel|out is now known as dviroel11:25
fungilooks like we got new pbr (5.11.0) and stevedore (4.1.0) releases12:00
*** ysandeep is now known as ysandeep|mtg12:15
*** dasm|off is now known as dasm13:13
*** ysandeep|mtg is now known as ysandeep13:44
clarkbfungi: correct 861800 has been abandoned14:03
*** dviroel is now known as dviroel|ptg14:08
clarkbmordred isn't here anymore, but the reason we don't use `pip wheel` in assemble is that we don't want the wheel(s) for the target install to mix with the wheels for the dependencies (which currently go in the wheel cache in a separate dir)14:25
clarkbI think this is beacuse we don't necessarily have shared info between assemble on the builder and install-from-bindep on the base image and it is easy to pip install /path/*.whl to work around that14:26
clarkbI think this is solveable though where we can do the wheel install in assemble, make note of the wheels that created and stash them in a file, run pip wheel against everything else. Then in install-from-bindep read the contents of that file and pip install those using find links?14:27
*** ysandeep is now known as ysandeep|dinner14:29
opendevreviewClark Boylan proposed opendev/system-config master: Switch python-builder/python-base to pip wheel
*** ysandeep|dinner is now known as ysandeep14:55
*** marios_ is now known as marios15:52
*** marios is now known as marios|out16:05
*** dviroel|ptg is now known as dviroel|lunch\16:06
*** ysandeep is now known as ysandeep|out16:20
*** jpena is now known as jpena|off16:39
*** dviroel|lunch\ is now known as dviroel16:53
gtemaclarkb: which cloud is still having cinderv1? I think what happened is that support for it got dropped from cinderclient which is used in OSC (we still haven't switched to SDK here)17:34
gtemaIf we could get access to this we may add support in SDK/OSC for that17:34
clarkbgtema: its rax (they do have v2 but its not complete and some operatins don't work so we've just fallen back to 1)17:34
gtemaAh great, will get access to it so can evaluate adding it17:35
fungialso i did some additional testing of python-openstackclient versions and the cinder v1 issues talking to rax seem to be independent of the python-cinderclient version i used (working venv with older osc worked vs newer osc did not even though both venvs had the same cinderclient version)17:59
fungii also pinned keystoneclient and ksa to the same version in both venvs17:59
fungiso those don't appear to be involved18:00
opendevreviewClark Boylan proposed opendev/system-config master: Switch python-builder/python-base to pip wheel
clarkbinfra-root thoughts on landing to restore the DNS record TTLs?18:11
clarkbit seems unlikely we'll need to revert to the old host at this point so TTLs being an hour is probbly fine. THen tomorrow? I'll plan to clean up the old server.18:11
clarkbpip CI is broken on a git submodule thing18:16
clarkbI ended up making a PR as requested by them for the cache thing. We'll see how this goes18:16
fungii've heard no complaints about gitea or the lb18:18
fungii guess i can go ahead and approve it, since the impact of that should be approximately nil18:19
clarkbI'll bundle up the gitea-lb01 removabl with jvb02 removal to reduce the rtt on those. (we definitely don't need the second jvb for meetpad after this ptg, it held up well but there wasn't concurrent load that would demand the extra jvb server)18:19
opendevreviewMerged opendev/ master: Reset TTLs for records
fungisounds good18:26
clarkbthe TTLs I see now are back to an hour19:11
opendevreviewClark Boylan proposed openstack/diskimage-builder master: DNM testing the combo of the depends on
clarkbok my pip wheel change in the base images does break siblings. The reason is we end up with multiple wheels for the same package (in this case dib in the nodepool siblings job) and pip cannot resolve that. I will have to think about how that can be resolved20:29
clarkbI was hoping that it would just pick the higher version and roll with it20:29
opendevreviewClark Boylan proposed opendev/system-config master: Switch python-builder/python-base to pip wheel
clarkbin my commit message I described an alternative appraoch that turns out to actually be the fix for this so I'm trying that here ^20:43
clarkbthe update to address is apparently affecting zuul and is also the cause of my failed pip ci jobs. Heads up here if other things run into it21:36
clarkbfocal and jammy both seem ot have updated by now in our CI system so should see fallout for that if it is there21:38
clarkbfirefox's about firefox menu button (to check if I need to restart for updates) is next to the report deceptive site button. Not great ui there22:03
fungihow are the security updates to git manifesting in job failures?22:07
clarkbin zuul's case zuul's submodule testing adding a repo with a file:// broke with default settings22:07
clarkbI think pip ci has a similar problem22:08
*** rlandy is now known as rlandy|bbl22:30
*** dasm is now known as dasm|off22:46
*** dviroel is now known as dviroel|out22:47
opendevreviewClark Boylan proposed openstack/diskimage-builder master: DNM testing the combo of the depends on
clarkbok I think the last recheck on ^ shows my pip wheel change may be working but this update should run all the functional nodepool + dib jobs too which will give us a better idea22:52
clarkb I think that line shows it successfully installing the sibling package of dib rather than the last actual release22:57
clarkbhowever I also note a lot of wheels get downloaded which I did not expect.22:57
clarkbdoes pip wheel maybe only preserve wheels that need to be built and aren't hosted on pypi?22:58
clarkbit definitely feels that way. Seems like all the things that it had to build before when it broke are there but then everything els eit downloads. I think that is ok though somewhat contrary to our original goals23:00
fungiit shouldn't. my local runs of tox show pip reusing cached wheels it downloaded previously23:04
fungiwith 22.323:05
clarkbfungi: using `pip wheel` and --find-links or `pip install` and --cache-dir?23:05
clarkbit does what you describe with `pip install` and --cache-dir but switching to the suggested `pip wheel` and --find-links like that change does seems to not23:05
fungiusing  `pip install` and letting it leverage its default cachedir of ~/.cache/pip23:05
clarkbI mean its still doing what we want which is build wheels in a disposable image and copy them over to the prod install to avoid needing build deps on the prod image23:06
fungioh, got it. pip wheel not reusing cached wheels23:06
clarkbright, upstream says doing that is bad and they want us to use `pip wheel` and --find-links23:06
fungithat *might* be intentional, or it could be a side effect nobody upstream has spotted23:06
clarkbbut ya I think I have pip wheel working in that stack now with the caveat I've just pointed out23:07
fungidoes pip wheel without --find-links do the same?23:07
clarkbthis second pass of nodepool + dib testing will tell us23:07
fungii mean, obviously --find-links is necessary for our use case23:07
fungijust wondering if the not reusing cached wheels (or not caching wheels?) is a side effect of passing --find-links23:07
clarkboh I stopped passing --cache-dir23:08
clarkbbut maybe I should do both?23:08
clarkbfungi: if you look at the install starting at it illustrates the behavior I'm talking about23:08
fungiif i just do pip wheel in one of my projects without --find-links it reuses the cached wheels for deps it needs23:08
clarkbright I guess what I'm confused about is it is supposed to write all the wheels for all the deps into the --wheel-dir but then when you use --find-links against that dir it seems to ignore those files if they also exist on pypi?23:09
clarkbessentially negating the need of a cache dir23:09
clarkbexcept that doesn't actually seem to be the case23:10
clarkb this is where all the wheels are written to the wheel dir23:10
clarkbso why is it ignoring those files later? Must be another pip bug...23:10
fungiand you're using a file:// url right?23:11
clarkbits only the wheels it builds at that it actually uses out of the find links dir later23:11
clarkbfungi: yes its pip install /path/to/foo.whl when we do the install later23:11
clarkbfungi: ianw: I still need to restart Gerrit to pick up is now a good time to do that?23:12
fungioh, i meant to the --find-links parameter23:12
fungiclarkb: now seems like a fine time for a gerrit restart, sure23:12
clarkbfungi: yes `--find-links /output/wheels`23:12
fungi--help suggests  it should be `--find-links file:///output/wheels`23:13
clarkbI think just /foo is equivalent23:13
fungioh, nevermind i misread. it does say local path *or* file:// url23:14
clarkbthe behavior makes me think that find links isn't disabling pypi and that it still prefers pypi to local wheels23:14
clarkbit can't find wheels for the other things on pypi so it fallsback to the local files there23:14
fungibut yes, it seems like --find-links is for adding wheels, not specifying a preferred location for them23:14
fungiit at least doesn't say it overrides the index or extra index urls23:14
clarkb"opendevorg/gerrit@sha256:385212f091675ee9b7702f2d59ed76bee87a154782918f8a92dec3b0c37366cd" via docker inspect of the image on review02 matches
fungiso i would infer the behavior you're observing is probably expected23:15
clarkbI'll go ahead and down then up -d the service now23:15
fungii have a feeling if you bring that --find-links behavior to their attention, their response will be that you should run a local on the machine23:18
clarkbor use --cache-dir :P23:18
clarkbit honestly seems backwards that the fix for this is to use the cache dir the way we've been using it23:18
clarkbI think I'll call that out on the issue since they keep saying using pip wheel23:18
fungi"what we want is a local cache of wheels which pip will reuse"23:19
fungibetter, which pip will populate and reuse23:19
clarkbgerrit web ui loads for me and other than complaints about not being able to compress log files that don't exist and more ssh clients that can't negotiate things23:19
fungiyeah, gerrit lgtm still23:20
clarkband now we're running the 3.5 tagged image23:21
clarkbone less thing on the pile now :)23:21
clarkb#status log Restarted Gerrit on our latest gerrit 3.5 image (change 861270) which resyncs our plugin versions to
opendevstatusclarkb: finished logging23:22
ianw++ thanks :)23:24
clarkblooks like we can use --no-index23:26
fungiaha, that's the ticket23:30
clarkb I left a comment there with what we found. It would be helpful for reviewers to weigh in on whether or not they want the new behavior or --no-index to emulate the old23:31

Generated by 2.17.3 by Marius Gedminas - find it at!