Monday, 2020-08-31

ianwso setuptools 50.0 has reimplemented the change from h; see
ianwSETUPTOOLS_USE_DISTUTILS=distlib is an escape hatch ...01:00
ianwsorry, yes01:00
fungiat least according to the changelog01:01
ianwfor devstack, i think maybe putting that in pip_install01:01
fungii didn't test it though, nor have i looked closely at the code01:01
ianwi don't think we'd want it set for things using pip, say, from their own virtualenv01:01
funginah, virtualenvs/venvs should be unaffected01:02
fungithis will just be places where we're running pip install into the system context as root, and likely only on debian/ubuntu systems01:03
fungisince it's clobbering debian's patched distutils which installs stuff into /usr/local instead of /usr01:03
ianwright, what i mean is i don't think we want the env var set in that case01:04
fungii concur01:04
ianwso like a global export in isn't quite right01:05
ianwor we remove our /usr/local/bin references, i guess01:05
fungithat's an option so long as we're not worried about the possibility the same commands could exist elsewhere in the path but be somehow wrong/broken01:07
ianw        # nova defaults to /usr/local/bin, but fedora and suse pip like to01:07
ianw        # install things in /usr/bin01:07
ianw        iniset $NOVA_CONF DEFAULT bindir "/usr/bin"01:07
ianwwe already have stuff like ^ scattered around01:08
ianwanother option i guess is to use the packaged pip tools only.02:38
ianwthat's a pita though, and i guess things like bionic pip don't understand manylinux2014 wheels02:38
clarkbwhy nit siwtch to /usr/bin?03:06
clarkbmaybe not ideal but should work?03:06
ianwclarkb: that's not really the issue -- it installs all libraries to site-packages and the packaged python doesn't look in there03:07
ianwthat's by debuntu design -- dist-packages for .deb packaged python, and site-packages if you like to ./configure; make your own python03:08
ianwyeah, so trying to use the packaged pip tools it blows up building things like which have manylinux wheels that only later pips understand03:11
ianwi think there's something up with
ianw--- ping statistics ---04:08
ianw28 packets transmitted, 0 received, 100% packet loss, time 27628ms04:08
ianwit can't talk or ping to pypi.  but i can ssh in, and it can wget/ping google.com04:08
ianwhrm, this looks like the db8 addresses again04:16
ianwi have run "ip -6 addr list | grep 2001:db8 | awk '{ print $2 }' | xargs -I {} -n1 ip addr del {} dev ens3"04:17
ianwmnaser, infra-root: ^04:17
*** stephenfin has quit IRC05:35
*** stephenfin has joined #opendev05:42
*** stephenfin has quit IRC05:47
*** stephenfin has joined #opendev05:56
fricklermnaser: did you ever succeed in locating the sources for that? these were the current ones
*** stephenfin has quit IRC06:01
*** stephenfin has joined #opendev06:02
*** fressi has joined #opendev06:05
*** ysandeep|away is now known as ysandeep06:08
*** stephenfin has quit IRC06:09
*** stephenfin has joined #opendev06:16
*** sshnaidm|afk is now known as sshnaidm06:17
*** stephenfin has quit IRC06:27
*** stephenfin has joined #opendev06:29
*** stephenfin has quit IRC06:50
*** stephenfin has joined #opendev06:57
ianwfrickler: you probably noticed everything broken :)07:03
*** stephenfin has quit IRC07:04
ianwfrickler: two suggestions are to pin setuptools <50 ; or try to set the config variable in appropriate places (something like ... i've set it wip as i think there might be more)07:04
ianwi'm pretty much out for today, so ... yeah :/07:04
*** hashar has joined #opendev07:08
*** ysandeep is now known as ysandeep|lunch07:32
*** tosky has joined #opendev07:39
*** zigo has joined #opendev07:44
*** tosky has quit IRC08:00
*** sgw has quit IRC08:00
*** dirk has quit IRC08:00
*** moppy has quit IRC08:01
*** moppy has joined #opendev08:02
*** tosky has joined #opendev08:05
*** sgw has joined #opendev08:05
*** dirk has joined #opendev08:05
*** hemanth_n has joined #opendev08:14
louroto/ just checking, the last discussion(s) here are related to the py35 POST_FAILUREs we're seeing e.g. in , correct?08:16
fricklerlourot: I guess they are, although that's a failure mode I haven't seen yet. infra-root: ^^ that would be an affected location outside of devstack08:19
*** ysandeep|lunch is now known as ysandeep08:45
*** rgildein has joined #opendev09:01
*** rgildein has quit IRC09:01
ianw"SystemError: Parent module 'setuptools' not loaded, cannot perform relative import"09:03
ianwi guess that is related to setuptools 50 if i had to guess, but not sure exactly what part of it09:03
*** dtantsur|afk is now known as dtantsur09:11
frickler#status notice due to a new release of setuptools (50.0.0), a lot of jobs are currently broken, please do not recheck blindly. see
openstackstatusfrickler: sending notice09:15
-openstackstatus- NOTICE: due to a new release of setuptools (50.0.0), a lot of jobs are currently broken, please do not recheck blindly. see
openstackstatusfrickler: finished sending notice09:18
*** DSpider has joined #opendev09:49
*** hashar has quit IRC10:02
*** Gyuseok_Jung has joined #opendev10:03
*** tosky has quit IRC10:13
*** tosky has joined #opendev10:13
*** elod has quit IRC11:27
*** weshay is now known as weshay|ruck11:44
*** elod has joined #opendev11:52
*** sshnaidm has quit IRC11:57
*** sshnaidm has joined #opendev12:07
*** redrobot has joined #opendev12:34
*** ysandeep is now known as ysandeep|brb12:49
*** lpetrut has joined #opendev12:52
*** ysandeep|brb is now known as ysandeep12:57
*** kevinz has quit IRC13:59
*** ysandeep is now known as ysandeep|afk14:05
*** cloudnull has joined #opendev14:06
fungi#status log zm07 was rebooted by the provider at 04:00z due to unspecified hypervisor host issues14:11
openstackstatusfungi: finished logging14:12
fungilooks like it's working fine at this point, so i'll close out that ticket14:12
fricklerany puppet folk around? puppet-openstack-beaker-centos-8 seems to be running as non-voting job in gate14:12
fungialong with the one for zm04 last tuesday14:12
*** fressi has quit IRC14:22
*** openstackgerrit has quit IRC14:37
clarkbI'm trying to page back in the things that I had learned last time setuptools changed this behvaior nad I seem to recall that it was specific to the editable installs14:37
clarkbonce I've got some tea and breakfast I was going to test if that is still the case and if so we may be able to avoid a lot of trouble simply by dropping the -e from pip install14:38
fungiit seems like it's any installs, not just editable14:41
fungibut they also provided an envvar this time we can use to tell setuptools to use the distutils from stdlib rather than its own copy14:41
fricklerclarkb: dropping -e worked for devstack, but ianw's solution seemed better to me, see #-qa14:42
frickler vs.
clarkbfungi: I'm rerunning my old reproducers fromthe original bug to test (but with different setuptools versions)14:45
clarkbfrickler: ya for devstack editable installs are probably a reasonable feature to keep14:45
clarkbbut in the general case I don't think they are necessary14:45
*** ysandeep|afk is now known as ysandeep14:47
fricklerI haven't seen editable installs anywhere else yet. the os-testr failure on py35 seems a different thing, maybe updating setuptools after the venv was created?14:52
clarkbconfirmed removing the -e preserves the old behavior14:52
clarkbfungi: ^ fyi it is editable install specific14:53
fungioh, cool, so that does give us some additional options14:54
clarkb`sudo docker run -it --rm ubuntu:20.04 bash -c 'apt-get update && apt-get install -y python3-pip git && pip3 install -U pip && pip --version && pip install -U "setuptools" && git clone && pip install ./twine && which twine && pip uninstall -y twine && pip install ./twine && which twine'` is the non -e version14:54
clarkband it uses twine because that was pypa's example package when reproducing before14:54
clarkbadd a -e to the pip install to test the -e case14:55
*** qchris has quit IRC14:57
*** qchris has joined #opendev15:10
*** sshnaidm is now known as sshnaidm|bbl15:11
*** lpetrut has quit IRC15:11
*** lpetrut has joined #opendev15:20
*** erbarr has joined #opendev15:23
*** ysandeep is now known as ysandeep|away15:24
johnsomNot that it's as important an issue as the setuptools issue, but gerritbot doesn't seem to be happy. posted an update that wasn't announced.15:58
fungithe bot may be dead. i'll check it in a sec15:59
fungiit's still running and getting events from gerrit according to its log, but maybe it fell off irc and didn't notice16:01
johnsomI got booted out of freenode over the weekend, so maybe there are freenode issues at play today as well16:01
fungi2020-08-31 14:37:31     <--     openstackgerrit ( has quit (*.net *.split)16:01
fungilooks like it hit a split almost 1.5 hours ago16:02
johnsomThe update I linked was about eight minutes ago16:03
fungiyeah, i think it was hit with a netsplit over an hour ago and hasn't returned16:05
fungii'll try restarting it and see if it gets a connected server16:05
fungi#status log restarted gerritbot following a netsplit at 14:37:31 from which it never returned16:06
openstackstatusfungi: finished logging16:06
yoctozeptoah, you read my mind, I was about to report the gerrit bot missing ;d16:08
*** priteau has joined #opendev16:09
fungijohnsom is on the ball today16:09
*** wolsen has joined #opendev16:37
*** dtantsur is now known as dtantsur|afk17:13
*** tosky has quit IRC17:14
*** priteau has quit IRC17:29
clarkbmorning meetings are done. I've reviewed setuptools things in devstack land and going to pop out on a bike ride now18:05
clarkbwhen I get back I'd like to make a gerritlib release18:05
clarkb(any idea if setuptools update will impact that?)18:06
fungiprobably not, but if it does we can at least hopefully address it there since it's a lower impact project18:10
*** diablo_rojo has joined #opendev18:11
fungidoesn't look like openstack release management have approved any release requests since setuptools 50 happened18:11
*** lpetrut has quit IRC18:20
*** mattd01 has joined #opendev18:36
*** mattd01 has quit IRC18:37
*** mattd01 has joined #opendev18:38
clarkbinfra-root how does tagging 6058d994b8bd36a8cf46b14ae2f05f806b895191 as 0.9.0 in gerritlib look?20:12
clarkbIf we're happy with ^ I'll do that after I finish eating this sandwich20:12
*** tosky has joined #opendev20:15
fungi0.8.4 was the last release, 6058d99 is the master branch tip20:16
fungiall changes in that range are testing-related or bug fixes, except the last one which is a feature addition, so i agree with 0.9.020:17
fungiif not for "Add branches arg to createProject" i would have said 0.8.5, but yeah 0.9.0 there is best20:18
clarkbok making that tag now20:23
clarkband pushed20:26
clarkbonce that is on pypi the next step is toe land the jeepyb change which should build new gerrit images with the gerritlib and jeepyb updates included20:27
fungii guess this will also tell us if new setuptools broke general release automation jobs20:30
clarkbrelease-openstack-python succeeded at least20:30
fungia good sign, yep20:31
clarkb is present as well20:31
clarkbfungi: is the next thing to land, should I go ahead and do so now?20:32
fungisure, you've got 2x +2 on there20:33
clarkbk just going to double check that that builds gerrit imgaes20:33
clarkbhuh it triggers gerrit images but not for 2.1420:34
clarkbnot going to worry about that right now since we don't need 2.14 at the moment20:34
*** mattd01 has quit IRC20:51
mnaserHTTPError: 500 Server Error: Internal Server Error for url:
mnaserother requests went through just fine20:58
mnaserbut just a heads up in case something is brewing..20:58
*** openstackgerrit has joined #opendev21:04
openstackgerritClark Boylan proposed openstack/project-config master: Pin setuptools<50 in our image venvs
clarkbianw: ^ I think that is the workaround for python3.5 os-testr problems21:04
clarkbmnaser: which backend do you get that from (the cert names will tell you)21:04
clarkball 8 gitea backends load that page for me21:05
fungior was that in a job log?21:05
openstackgerritMerged opendev/jeepyb master: Set repo HEAD on gerrit project creation
clarkbinfra-root config-core ^ fyi that shouldn't impact new project creations but if it does let me know21:16
clarkbinfra-root config-core should fix the python3.5 job errors that people are reporting21:30
clarkbI'm happy to shepherd some image rebuilds if I can get reviews on that21:30
ianwclarkb: are we 100% sure it's using tox from that virtulaenv these days?21:40
ianwand not something setup by ensure-tox?21:40
clarkbianw: no, and really its just os-testr from the venv for it that I'm noticing failing21:40
clarkbI figured I'd apply the fix globally across the venvs though21:40
clarkb is an example21:41
clarkbre ensure-tox I believe we set the tox path to our venv and use the venv21:41
clarkbbut I'm not sure of that21:41
ianw ... i don't think so21:44
ianwanyway, kind of moot because it's not tox causing hte problem21:44
clarkbmaybe we should set tox_executable, but ya not important for right now21:45
openstackgerritMerged openstack/project-config master: Pin setuptools<50 in our image venvs
clarkbI've double checked and our new gerrit docker imag ehas gerritlib==0.9.0 in it as expected21:57
*** Dmitrii-Sh1 has joined #opendev22:03
*** Dmitrii-Sh has quit IRC22:05
*** Dmitrii-Sh1 is now known as Dmitrii-Sh22:05
*** stephenfin has joined #opendev22:37
clarkbI've started an ubuntu-xenial image build after confirming that project config has been updated on builders22:38
clarkbfocal and bionic just got new builds prior to that change landing so will need to be triggered as well (though so far only python3.5 jobs on xenial seem affected)22:41
*** tosky has quit IRC23:03
clarkbianw: should we keep the github third party ci meeting topic item?23:06
ianwclarkb: i think we can remove it for now.  upstream said that full release automation was on their radar for "soon" ... of course we could do that all for them right now :)23:07
clarkbianw: I did mean to ask why we only got a py35 arm64 wheel? Is that just a manylinux problem?23:07
clarkbor do we need to add more jobs etc23:08
ianwno, it's just that it's using the stable abi so it's forward compatible23:08
clarkbwill pip discover that?23:08
clarkbif so maybe I need to resurrect my nodepool arm64 image jobs23:08
ianwyep, recent enough pip will pull it in, as it's the absolute lowest-common-denominator that works everywhere23:09
clarkbneat I didn't know that23:09
ianwyes, sorry i meant to ping to try that again23:09
clarkbianw: feel free to update my changes or push new ones23:09
clarkbI'm running out of day and trying to get the agenda out and shepherd the vm image rebuilds23:09
clarkbI can likely look at nodepool tomorrow if you don't get there first23:09
ianwok; yeah was on my todo, till everything was broken yesterday :)23:09
clarkbianw: I think what we need/want is a revert chang eof the multi arch removal, then on top of that we can fiddle with mirror setup or whatever to speed things up23:11
clarkb(if necessary)23:11
ianwyeah, i'm not sure if ffi re-enabled their wheel build; do you remember if anything else was slowing it down?23:14
ianwoff hand23:14
fungiso pip on py38 will pull in cp35 wheels?23:15
fungii did not realize23:15
fungii was sure i had seen pip under newer interpreters ignore wheels for older abis23:16
ianwfungi: yeah, stable api
ianwnote that, annoyingly, the bionic pip doesn't23:19
clarkbya but how does pip know the cp35 annotated wheel is valid for cp38? I mean I expect that it does I just don't know the mechanism23:19
ianwit's the -abi3- in it i think23:20
fungii just hadn't seen one of those before23:20
ianwyes, i'm saying this like i'm not some sort of dilettante who had never built a manylinux wheel before last week :)23:21
clarkbyou're the expert now23:22
* fungi casts off one shoe23:23
ianwwe should be able to do pynacl23:24
clarkbfor nodepool we should see if we can run quickly enough now with or without the openstack wheel mirrors23:26
clarkbbut ya may need to add wheels for a small number of additional deps23:26
fungii suppose we could try to implement our full wheel builder job on arm64?23:27
fungimight run for days though23:27
ianwyeah, i think targeted approach is probably better23:31
clarkbfwiw I think pynacl is in the openstack wheel builds the problem we hit with it in zuul context is zuul doesn't use openstack's constraints23:32
fungihonestly, while it's a little more work, maybe this is a great case for generalizing the wheel builder job to take arbitrary requirements lists23:32
ianwthe thing with the manylinux builds though is i think it pretty much defeats any sort of generic approach to building23:32
clarkbmaybe we can have openstack's wheels builds build the constraints and whatever is most recent23:32
ianwlike the pyca one is using it's own custom docker image23:32
clarkbsince we'll expect openstack to use the recent version anyway23:32
fungiso we could say "run the wheel builder job for the list including these versions of cryptography and pynacl" (for starters)23:32
clarkbbasically do a forward looking build there then I think zuul's major issue with consuming from it to plug gaps will be addressed23:33
ianwyeah, i mean that was basically what we decided :)  but it's also great if we can do something like work with upstream because now the cryptography problem is solved for everyone23:33
fungioh, i get ya... yes, building manylinux wheels is likely a separate task from maintaining a platform-specific wheel cache, agreed23:33
clarkbya I'm not saying stop with the upstream builds23:33
clarkbmore just a how do we make things better downstream too23:34
ianwyeah, agree.  updating the building script to pull more arbitrary things is on my todo as well :)23:35
ianwlooks like cryptography have realised that the rust extensions they want to pull in don't work nice with the python limited api/abi23:36
fungiare they building those with maturin or something like setuptools-rust?23:37
ianwfungi: setuptools-rust and then ... umm ... i don't know what other wrappers are invovled23:52
clarkbthe xenial image is built and uploading to ovh now (the other 6 upload workers are uploading the bionic image that built just a bit too early)23:54
clarkbI was hoping to recheck before enqueuing rebuilds of the other images so I'll wait on bionic and focal and the debian images23:55
clarkbthey don't seem to have trouble currently23:55

Generated by 2.17.2 by Marius Gedminas - find it at!