Monday, 2022-04-04

ianwyeah now we have the buster container we might need to reevaludate00:56
ianwit's probably a question of what else it drags in00:58
opendevreviewIan Wienand proposed openstack/diskimage-builder master: Test building jammy
*** lbragstad2 is now known as lbragstad02:37
*** raukadah is now known as chandankumar03:40
*** raukadah is now known as chandankumar04:22
*** ysandeep|out is now known as ysandeep04:26
opendevreviewMerged openstack/project-config master: Match configure-mirrors for CentOS wheel URLs
ianwfungi: ^ thanks for that.  i've remounted the wheel volumes without the -stream, and released 04:28
opendevreviewMerged zuul/zuul-jobs master: ensure-kubernetes: fix missing 02-crio.conf
fricklerianw: not sure why this doesn't show in the build log, testing locally I got: E: Extracting .//var/cache/apt/archives/base-files_12ubuntu3_amd64.deb requires the zstdcat command, which is not available06:29
fricklerzstd is only suggested by debootstrap, explicitly added it now for the next attempt06:29
*** ysandeep is now known as ysandeep|afk06:49
opendevreviewMerged zuul/zuul-jobs master: ensure-pip: fix typo in ensure_pip_virtualenv_command documentation
ianwhuh, weird.  the package format has changed in unstable?07:30
ianwclarkb: i made a glean 1.21.0 release with your fix and unpased centos 9; centos-9-stream-0000005135 is the fixed version.  it's uploading now07:34
ianwalso i had a poke at some of the deleting images nodepool is stuck on, seems a lot of leaks in osuosl.  i've sent an email to support about that07:35
*** jpena|off is now known as jpena07:36
fricklerRamereth: ^^08:40
fricklerianw: seems to be a different issue. maybe I have to try with the docker image locally08:42
*** ysandeep|afk is now known as ysandeep09:11
*** arxcruz|out is now known as arxcruz09:34
*** jpena is now known as jpena|off09:35
*** jpena|off is now known as jpena09:44
*** pojadhav- is now known as pojadhav10:58
*** pojadhav- is now known as pojadhav11:13
*** pojadhav is now known as pojadhav|afk11:34
*** dviroel|out is now known as dviroel11:38
fungiianw: thanks for moving the volumes too!11:49
fungifrickler: interesting, zstd is only a suggests in the debootstrap in debian/unstable, neither cdebootstrap nor mmdebstrap (what i usually use) even have it as a suggests. wonder if this is an ubuntu-specific modification?11:54
fungias in maybe ubuntu switched from zlib to zstd compression for their debs recently11:56
fungifrickler: yep, that's exactly it, apparently...
fungithe version in bookworm is too old, the version in bullseye/sid has support for unpacking ubuntu's newer format but seems to need zstd for doing so12:05
fungier, te version in bullseye is too old, the version in bookworm/sid is new enough12:05
fungitoo many "b" release names in a row12:05
fungia new enough version migrated to bookworm (testing) in november, so adding to bullseye-backports would probably be possible if requested12:07
fungiright now there are older versions in old-bpo (buster-backports) and o-o-bpo (stretch-backports) indicating it frequently gets backported for such reasons12:09
fricklerfungi: interesting, seems debootstrap on jammy itself doesn't have this dependency and works fine without zstd.12:21
fungiprobably it uses libzstd instead, or has it statically linked12:23
fricklerfungi: it uses dpkg-deb if that has zstd support, which it seems to have on ubuntu but not debian
fricklerand that in turn indeed uses libzstd12:34
*** ysandeep is now known as ysandeep|break12:40
*** pojadhav- is now known as pojadhav12:45
*** jpena is now known as jpena|off12:54
*** ysandeep|break is now known as ysandeep13:00
*** jpena|off is now known as jpena13:03
marioso/ folks can someone point me to the host pin for diablo? so we can start recording (for tripleo )13:04
*** timburke_ is now known as timburke13:04
fungimarios: zoom? maybe ping diablo_rojo in #openinfra-events, we don't manage that13:08
fungiinfra-root: ended up back on the spamhaus pbl... it looks like spamhaus has changed their processes, now the listing page mentions "NOTE: Exclusions are only valid for 1 year."13:09
fungiwe probably need to revisit all our pbl exclusions13:10
mariosfungi: yeah thanks found it13:11
mariosfungi: thankyou for checking13:11
mariossorry for noise13:11
fungino worries13:12
fungi#status log Requested Spamhaus PBL exclusion for the IPv4 address of lists.kata-containers.io13:21
opendevstatusfungi: finished logging13:21
opendevreviewMerged zuul/zuul-jobs master: Add tox-py310 job
fungimeetpad, jvb01 and jvb02 seem to be holding up well so far. can see some load on meetpad but nothing looks worrisome13:53
opendevreviewMerged opendev/git-review master: Switch to unittest.mock
jrosseri got an unexpected -2 from zuul on this patch
jrosserand the message was a warning that `Change 835548 in project openstack/openstack-ansible does not share a change queue with 833865 in project openstack/openstack-ansible-os_nova`15:09
jrosserand i'm OK with those not sharing a queue, and didn't expect a warning to fail the job completely15:10
jrosseronce the depends-on patch was merged, it went through and merged as expected but at the expense of rechecking15:11
clarkbjrosser: corvus can confirm but I think that may be zuuls new behavior of being more explicit about why things aren't enqueing? Either that or unexpected fallout from the topic submisson stuff /me checks if they shared a topic15:16
clarkbno does not look like they shared a topic15:16
clarkbianw: thank you for getting that release pushed. I see booted in-use instances in rax now15:21
clarkbI need to cleanup my test node now15:21
clarkband done15:34
clarkbfrickler: left a question on the nodepool change for debootstrap update. I'm +2 since its mostly just a thing about the comment15:36
clarkbI think I'm mostly caught up on things that happened over the weekend. I'm half avoiding PTG esssions this morning since I've got a DMV appointment midday today and don'y want to get sucked into anything but ping me if I should jump on a call15:39
prometheanfirefyi, I have to hold back paramiko due to it not working with dropbear anymore (closed as wontfix)
clarkbprometheanfire: thats not entirely true. It will work with dropbear but you either need a newre dropbera than is currently in cirros or you need to use a different key type than rsa15:42
clarkbprometheanfire: we've been suggesting that people use non rsa keys15:42
prometheanfireah, kk15:45
clarkbprometheanfire: it does look like paramiko is trying sha2 as the fallback which is good. Openssh doesn't do this and falls back to sha1 then fails due to its new sha1 policy which is a bit silly imo. I think that all clients should fallback to sha2 if they disallow sha1 then error as paramiko does if the server cannot sha215:45
prometheanfireas long as it's known about15:45
clarkbprometheanfire: basically I wouldn't pin paramiko I would force tempest et al to use a different key type. The support is already there for fips testing15:46
prometheanfireya, tempest is the right place for a fix15:46
prometheanfireif I see a meeting during the ptg I'll bring it up15:47
clarkbfungi: looks like virtualenv still hasn't updated for new setuptools.15:48
clarkbI was hoping we could revert some of thos ework around today. Oh well15:53
fungiyeah, tell me about it15:57
*** marios is now known as marios|out16:17
gibiit is just me or gerrit has issues managing ssh keys? 16:22
gibiI tried to delete old keys via the UI 16:23
gibifirst it seems it is deleted but after a refresh they are back16:23
clarkbgibi: you have to click save iirc16:23
clarkbbut no I've not had any issues removing keys16:23
gibi /o\16:24
gibiclarkb: thanks. that was the trick16:25
clarkbya I'm not sure I like that ui but it allows you to queue up a bunch of deletion sand additions and apply them at once16:26
clarkbI'm guessing that was useful to some user16:26
gibiclarkb: do you happen to know if our gerrit supports ed25519 keys?16:29
gibiI can add such key via the gui but it does not seems to be accepted via the ssh handshake16:30
*** jpena is now known as jpena|off16:31
clarkbgibi: it does16:31
clarkbI use one now16:31
clarkbfungi: I'm reading the tc ptl discussion notes and I think some of that feedback should be to opendev. Its a bit frustrating that we've done office hours for multiple PTGs and when we decide not to becuase no one engages us there is a bunch of feedback to the wrong group of people :/16:32
clarkbfungi: I wonder how we can communicate that better. Like hey we tried for a ocuple years and didn't get any input so stopped. How can we better collect that feedback? type messaging to the mailing list maybe16:33
gibiI use a yubikey with ed25519_sk resident key so it might be a problem on my side unlocking the ssh key with the yubikey during ssh16:33
clarkbgibi: _sk keys are different than regular keys. I don't know that gerrit supports them16:33
clarkbboth sides need to support them iirc16:33
fungisupport would need to be in apache mina-sshd i guess16:34
clarkbfungi: yup16:34
gibiclarkb, fungi: thanks for the info. I guess I will stick to a non _sk key then with gerrit for now16:35
clarkbyup looks like the sshd has to be aware of the key type at least for negotiation purposes16:38
clarkbit may be that the client does the heavy lifting, but seems like that key type must be explicitly recognized by both sides of the connection16:38
*** dviroel is now known as dviroel|lunch17:01
*** dviroel|lunch is now known as dviroel17:55
opendevreviewMerged zuul/zuul-jobs master: run-buildset-registry: Drop extra install packages task
*** rlandy is now known as rlandy|brb19:20
*** rlandy|brb is now known as rlandy19:44
johnsomHi, I'm having an odd issue with the openstack-tox-docs job on Wallaby. I see pip running a second time unconstrained, which pulls in a newer (broken) version of jinja2:
johnsomI am not seeing where that pip install is coming from. My code-search-foo isn't working today.20:27
johnsomDoes anyone have any thoughts?20:27
fungithe tox jobs normally run pip install twice, in order to separately pull in siblings projects for source installs. something there is probably interacting questionably20:28
fungilooking at the report20:28
johnsomYeah, I found the task in siblings.yaml:
fungijohnsom: for further reference, the module it's running is this:
johnsomThanks for the second set of eyes btw20:30
fungiit expects to handle constraints if supplied20:30
clarkbit should only update the packages that are found to be siblings iirc and then do that install with contsraints applied20:31
fungiit looks like a constraints file was supplied20:31
fungithat indicates it checked out the requirements repo at that path to stable/wallaby20:34
clarkblooking at the install siblings task it doesn't look like it installed any siblings?20:36
johnsomThe wallaby upper-constraint is: Jinja2===2.11.3 which is what it pulled the first time.20:36
clarkbI don't think it is the siblings install doing it20:38
clarkbsince it happens in the task before that20:38
clarkbRun tox without tests is where it happens20:38
johnsomThat task name matched20:39
clarkbI think it is installing the deps and that works with constraints but then when it installs the package itself it fails to use constraints20:41
*** dviroel is now known as dviroel|afk20:41
fungithe unconstrained pip install command seems to be coming from tox itself, yeah?  ubuntu-focal | [4348] /home/zuul/src/$ /home/zuul/src/ install --exists-action w -e .20:43
clarkbits the step that installs designate20:43
clarkbafter it installs all the deps20:43
clarkbI think that implies the package requirements want newer jinja and so pip pulls it in rather than seeing it s already satisifed?20:44
fungithis is tox's usedevelop behavior at work, right?20:45
clarkbhrm except the requirment is jinja2>=2.10 and 2.11.3 was installed previously20:45
clarkbfungi: yes I think so20:45
johnsomI see that set in the tox.ini for Designate:
clarkbI suspect this is reproduceable locally20:46
johnsomI have no idea why20:46
clarkbjust run tox -edevos20:46
clarkbheh if I can spoell20:46
clarkbtox -edocs20:46
fungijohnsom: you might see if dropping the testenv.usedevelop option from tox.ini results in the correct behavior there20:46
fungithis might be a tox regression20:46
fungii can't remember if usedevelop ever obeyed install_command20:47
clarkbwhat is expected to happen is we install all the deps with constraints then when it installs the package itself it hsould see all thoes packages are already installed at valid versions and not reinstall them20:47
fungioh, actually install_command isn't setting a constraints option, that's being set in the deps20:47
fungiso this makes a bit of sense20:47
clarkbfungi: well install_command isn't even used by projects like nova20:48
fungiusedevelop doesn't consult the deps which is where -c is being set20:48
johnsomso, locally, tox -e docs runs just fine for me20:48
clarkbjohnsom: you may need a -r too20:48
johnsomBut I probably don't have the latest tox20:48
johnsomYeah, it was a new clone, no previous env20:48
clarkbI think the issue is that the doc requirements don't fully overlap with the requirements.txt to install the pcake20:48
clarkband when pip reconciles all of those other missing package deps to install designate it pulls more stuff in20:49
clarkbI think you want to not install the package at all20:49
clarkbwhatever flag that is in tox20:49
clarkbor stop using doc specific requirements20:49
fungithough if you're using a sphinx extension which runs the program to capture cli output, for example, you may need it installed in the env. depends on what's in the project's docs as to whether that approach is needed20:50
* fungi needs to start preparing for the ptg to resume20:50
johnsomHmm, this just "magically" broke sometime between March 23rd and April 1st.20:50
clarkbya and if that is the case then you shouldn't have doc specific requirements that are a subset of normal isntall requirements20:51
clarkbjohnsom: tox hasn't updated since december, but maybe virtualenv/setuptools/pip shifted behavior20:51
clarkbjinja2 updated to be non backward compatible and it just worked installing the wrong version until then20:51
fungiit probably broke on april 1, just to prank you ;)20:51
clarkbanyway I'm pretty sur ethis is because it installs a lot of packages to install designate20:51
clarkband the constrained install is only the doc deps and that doesn't install enough under constraints20:52
fungijinja2 3.1.0 was released on march 24, and 3.1.1 a day after20:53
fungijohnsom: we probably have build history prior to that, if you want to check a successful run for that branch and see what version of jinja2 actually got installed in the docs builds20:54
fungii.e. whether it was 3.0.3 or what's in constraints20:55
johnsomAh, we do have logs for the last successful.
johnsomYeah, it was Jinja2 3.0.320:57
fungiso constraints was never actually applying there20:57
fungijinja2<3.1 was simply working20:58
johnsomStill, I'm trying to get my head around why it would install unconstrained. I'm going to read up on this usedevelop=True setting I don't know why is in tox.ini20:58
johnsomYeah, I think you are right20:58
clarkbjohnsom: is develop is not the problem20:58
fungi was what broke you, according to the changelog for 3.1.020:58
clarkbjohnsom: the problem is that your doc requirements are not a super set of the designate requirements20:58
clarkbI suspect this problem exists for much of openstack too20:59
fungiand is specific to the docs jobs if the project is being installed into the env20:59
johnsomYeah, I don't think any of the project I have seen are setup that way20:59
clarkbbecause the constraints only apply when installing the dependencies. Then when it installs designate (the package) the expectation is that all the deps are already installed so it will noop the installs at that point20:59
clarkbbut that doesn't work here becuase most of the dependencies are not listed as doc requirements and thus weren't installed and thus get pulled in unconstrained21:00
johnsomWe see in the section above it installed 2.11.3 then this second run uninstalled that and installed 3.1.121:02
clarkbjohnsom: yes becaus esome other dep depends on jinja2 and caused pip to decide to install the newer version21:02
clarkblikely because their dep is >2.11.321:03
clarkbI think there are two reasonable fixes here. First is to stop intsalling the package (designate) in doc builds. The other is to drop the docs specific requirements and install all the deps constraints21:03
clarkbjohnsom: ya so maybe pip dep solver is being to smart then21:04
clarkbeithre way the reason is some other package depends on jinja2 that wasn't preinstalled forcing jinja2 to get reinstalled and for some reason pip ignores the existing install21:04
johnsomdesignate has to be installed for the code indexes and such.21:05
clarkbthen you need to drop the doc requirements21:05
clarkbor rather do doc requirements + project requirements21:05
clarkbsince I guess there may be doc specific requirements that only get installed for documentation21:05
johnsomIt's definitely something wrong in pip. Above it sees it's installed then gets another >=2.10 and it goes and pulls it again.21:06
clarkbjohnsom: add a line here to add requirements21:06
clarkbjohnsom: well even if you fix the jinja2 problem some other dep could hvae that problem21:07
clarkbor if the thing pulling in jinja2 later says >=3.021:07
clarkbyes pip is maybe not doing the correct thing here but blaming pip and them fixing this specific issue is only a temporary/half fix21:07
clarkbsince jinja2>=3.0 in a dep that wasn't preinstalled would cause the same problem21:07
clarkband any other dep could do the same thing21:07
clarkbmy hunch i sthat the pip dep solver is related to why pip does that21:08
johnsomWell, that also passes local, so I will give that a try. 21:10
clarkbI've updated the meeting agenda on the wiki. I didn't really have anything new to add and trimmed out some stuff that occurred. Feel free to add topics for the next little bit and I'll get that sent out at the end of my day21:22
johnsomclarkb You are the pip whisperer. That worked. Thank you all for the insight.21:30
clarkbcool, it might be worth a thread on openstack-discuss to make sure that others are aware of this problem if they install the package and use a subset of deps in the doc requirements21:31
clarkbjohnsom: is that something you might want to draft up or should I send an email about it/21:31
johnsomI'm not sure I understand the issue well enough to be clear. 21:32
johnsomIf you give me a blurb on the why/how it's happening I can fill in the how-to-fix parts21:33
clarkbjohnsom: can do. I'll start an etherpad21:33
johnsomclarkb Cool, on it.21:37
clarkbjohnsom: I edited line 10 of that etherpad with a smal ledit to make it more clear21:55
johnsomclarkb Ok, I haven't poked at that yet.21:56
clarkbI'm going to drop off the security sig call now since it is noisy here at home today and I got my bit in :) I will see if I can join the qa things tomorrow but that beginning of that slot conflicts with school run21:59
clarkband other meetings21:59
fungiexplicitly installing designate with constraints before (or instead of) tox's usedevelop behavior could be another workaround22:19
johnsomfungi This worked:
johnsomI'm currently writing up the commit message for the real change and we will send out the email to the discuss list.22:20
fungiin a personal project i add {toxinidir} to the deps= in my testenv:docs instead of relying on usedevelop22:20
fungiso it's effectively installing "." but that way it can do so along with the normal deps installation rather than in a separate usedevelop phase22:21
fungisince your deps contain the -c, doing it that way would apply constraints to the designate installation and also avoid having the separate usedevelop invocation of pip22:22

Generated by 2.17.3 by Marius Gedminas - find it at!