Saturday, 2022-03-19

opendevreviewMerged opendev/system-config master: Rebuild Gerrit images
*** mazzy50988129295808594 is now known as mazzy509881292958085901:38
fricklerinfra-root: the container on nb03 keeps crashing with ModuleNotFoundError: No module named 'ibm_botocore'
fungii wonder if a deploy job broke or was interrupted11:39
fungier, except dependencies should be in the containers we build11:40
fungioh! that's the arm64 builder, so maybe our arm images are broken11:40
corvusthat should be provided by ibm-cos-sdk-core
corvusnote that amd downloads a targz; arm downloads a wheel, but i don't see a wheel listed on
corvusi do see it here:
corvusand it is missing the files16:17
corvussorry -- correction of earlier link:16:17
corvusso amd64 decided to install 2.11.0; and arm64 decided to install 1.2.0.  16:18
corvusfungifrickler could this be a situation where pypi served a wrong simple index?16:22
corvusthe thing where it is missing the version metadata or whatever that problem is?16:22
fungiwell, pip's behavior when installing an sdist is to install the newest one it thinks is valid, and if compilation fails then pick the next oldest one... lather rinse repeat until it builds successfully. also pip hides the build failure output for newer sdists if it manages to build an older one16:24
corvusbut wasn't there an issue where the simple index was returning data with incorrect version specifiers and therefore pip was choosing to download too-old versions?16:25
corvus(but this only happened for some fastly caches?)16:25
fungioften times you'll see this if you're missing a build dependency, pip will keep backtracking until it hits the version before the new dependency was introduced16:25
fungiyes there have been times when we've proxied/cached copies of the pip simple api pages which lacked requires_python metadata, but i don't see ibm-cos-sdk-core setting it16:26
fungiahh, no it does, i looked past it16:26
fungiRequires: Python >= 3.616:26
fungithat's for 2.11.016:26
fungiare we building the images with an older python than 3.6?16:27
fungiif not, then that's probably not involved16:27
corvuslooks like 3.916:27
fungimore likely to be a missing build-dep for some c extension16:28
fungiintroduced in 2.x16:28
fungiand pip is just hiding the compiler errors16:28
fungiif pip's verbosity were cranked up i think it would report them16:28
fungior we could add a ibm-cos-sdk-core>=216:29
corvusaccording to the package metadata:16:29
corvusrun_requires: "python-dateutil>=2.1,<3.0.0", "jmespath>=0.7.1,<1.0.0", "docutils>=0.10"16:29
corvusrequires: "ordereddict==1.1", "simplejson==3.3.0"16:29
fungithose are required python modules, but does it build any c extensions? i'm pulling up the source code now16:30
corvusi'll try `pip install -v ibm-cos-sdk-core` on an arm machine16:30
fungilooks like it may be pure python, so perhaps not compiler errors either16:31
corvusTemporary failure in name resolution')': /simple/ibm-cos-sdk-core/16:31
corvusdo we have working dns on nb03?16:32
corvusweird, it works on-host, but i can't seem to do any network ops in a container :/16:33
fungiwhy would it think /simple/ibm-cos-sdk-core/ was a hostname?16:34
corvusi have no idea; i can't seem to do even the simplest things in a on nb03.  i tried apt-get update and that's failing too16:35
corvusoh now it's working16:35
corvusit seems to have downloaded the right version16:36
corvusi'll try installing the whole requirements file16:36
fungiand that's inside the nodepool-builder docker container?16:37
corvusin the python-builder one (since that's where we run the pip commands when building the nodepool builder)16:38
fungiyep, okay16:39
corvusokay, that failed to build the netifaces wheel, but that's probably just me not doing enough prep work -- it did download 2.11.0 of cos-sdk16:39
corvusyeah, i'll install gcc :)16:40
fungithough that leaves me wondering if 2.x added a netifaces dep16:43
corvusthat was from the whole requirements.txt, so probably from something else16:44
fungiyeah, would have been an indirect dependency16:44
corvusi'm basically stepping through the build process manually...16:46
corvusand it worked:16:46
corvus  Created wheel for ibm-cos-sdk-core: filename=ibm_cos_sdk_core-2.11.0-py3-none-any.whl size=536334 sha256=f8bca20509ab93b526aaf59e697bb2ec816fe0b55036527240d89e8d54da35e316:46
corvusi'm starting to think this may be a transient error16:46
corvusmaybe we should just put a mininum version in our reqs so that if it happens again the build will fail?16:46
fungithat seems wise16:50
fungiand yeah, no significant changes that i can see in the between 1.2.1 and 2.0.016:50
corvusi pushed up a change: remote: Add a mininum version for ibm-cos-sdk-core [NEW]        16:50
corvusi'm going to restart the mergers and then the schedulers now16:51
fungiawesome. i'm around if you need help16:51
corvuszuul01 is down16:58
corvuszuul01 is starting up17:01
corvusit's up; stopping zuul02 now17:08
corvuszuul02 is starting up again17:11
corvusi am pleased with the performance of the lb :)17:11
corvusall done17:19
fungii stepped away for a few, i guess webui failover worked well?17:59
corvusyeah -- i didn't have a status screen going.. i should have done that... but i did check the components page periodically and did not notice an outage20:05
fungivery cool20:16

Generated by 2.17.3 by Marius Gedminas - find it at!