| opendevreview | Merged opendev/system-config master: Rebuild Gerrit images https://review.opendev.org/c/opendev/system-config/+/834244 | 00:01 |
|---|---|---|
| *** mazzy50988129295808594 is now known as mazzy5098812929580859 | 01:38 | |
| frickler | infra-root: the container on nb03 keeps crashing with ModuleNotFoundError: No module named 'ibm_botocore' https://paste.opendev.org/show/bt8UFSh13KK9KakDkUiP/ | 10:26 |
| fungi | i wonder if a deploy job broke or was interrupted | 11:39 |
| fungi | er, except dependencies should be in the containers we build | 11:40 |
| fungi | oh! that's the arm64 builder, so maybe our arm images are broken | 11:40 |
| corvus | that should be provided by ibm-cos-sdk-core https://pypi.org/project/ibm-cos-sdk-core/#files | 16:07 |
| corvus | amd64: https://zuul.opendev.org/t/zuul/build/f3022ec8e84e4af8bcdd826ef3063b20/log/job-output.txt#3107 | 16:13 |
| corvus | arm64: https://zuul.opendev.org/t/zuul/build/f3022ec8e84e4af8bcdd826ef3063b20/log/job-output.txt#5498 | 16:13 |
| corvus | note that amd downloads a targz; arm downloads a wheel, but i don't see a wheel listed on https://pypi.org/project/ibm-cos-sdk-core/#files | 16:13 |
| corvus | i do see it here: https://pypi.org/simple/ibm-cos-sdk-core/ | 16:16 |
| corvus | and it is missing the files | 16:17 |
| corvus | sorry -- correction of earlier link: | 16:17 |
| corvus | amd64: https://zuul.opendev.org/t/zuul/build/f3022ec8e84e4af8bcdd826ef3063b20/log/job-output.txt#3209 | 16:17 |
| corvus | so amd64 decided to install 2.11.0; and arm64 decided to install 1.2.0. | 16:18 |
| corvus | fungifrickler could this be a situation where pypi served a wrong simple index? | 16:22 |
| corvus | the thing where it is missing the version metadata or whatever that problem is? | 16:22 |
| fungi | well, pip's behavior when installing an sdist is to install the newest one it thinks is valid, and if compilation fails then pick the next oldest one... lather rinse repeat until it builds successfully. also pip hides the build failure output for newer sdists if it manages to build an older one | 16:24 |
| corvus | but wasn't there an issue where the simple index was returning data with incorrect version specifiers and therefore pip was choosing to download too-old versions? | 16:25 |
| corvus | (but this only happened for some fastly caches?) | 16:25 |
| fungi | often times you'll see this if you're missing a build dependency, pip will keep backtracking until it hits the version before the new dependency was introduced | 16:25 |
| fungi | yes there have been times when we've proxied/cached copies of the pip simple api pages which lacked requires_python metadata, but i don't see ibm-cos-sdk-core setting it | 16:26 |
| fungi | ahh, no it does, i looked past it | 16:26 |
| fungi | Requires: Python >= 3.6 | 16:26 |
| fungi | that's for 2.11.0 | 16:26 |
| fungi | are we building the images with an older python than 3.6? | 16:27 |
| fungi | if not, then that's probably not involved | 16:27 |
| corvus | looks like 3.9 | 16:27 |
| fungi | more likely to be a missing build-dep for some c extension | 16:28 |
| fungi | introduced in 2.x | 16:28 |
| fungi | and pip is just hiding the compiler errors | 16:28 |
| fungi | if pip's verbosity were cranked up i think it would report them | 16:28 |
| fungi | or we could add a ibm-cos-sdk-core>=2 | 16:29 |
| corvus | according to the package metadata: | 16:29 |
| corvus | run_requires: "python-dateutil>=2.1,<3.0.0", "jmespath>=0.7.1,<1.0.0", "docutils>=0.10" | 16:29 |
| corvus | requires: "ordereddict==1.1", "simplejson==3.3.0" | 16:29 |
| fungi | those are required python modules, but does it build any c extensions? i'm pulling up the source code now | 16:30 |
| corvus | i'll try `pip install -v ibm-cos-sdk-core` on an arm machine | 16:30 |
| fungi | looks like it may be pure python, so perhaps not compiler errors either | 16:31 |
| corvus | Temporary failure in name resolution')': /simple/ibm-cos-sdk-core/ | 16:31 |
| corvus | do we have working dns on nb03? | 16:32 |
| corvus | weird, it works on-host, but i can't seem to do any network ops in a container :/ | 16:33 |
| fungi | why would it think /simple/ibm-cos-sdk-core/ was a hostname? | 16:34 |
| corvus | i have no idea; i can't seem to do even the simplest things in a docker.io/opendevorg/python-builder:3.9-bullseye on nb03. i tried apt-get update and that's failing too | 16:35 |
| corvus | oh now it's working | 16:35 |
| corvus | https://paste.opendev.org/show/buxvhavyUCNAlYaV0eOd/ | 16:36 |
| corvus | it seems to have downloaded the right version | 16:36 |
| corvus | i'll try installing the whole requirements file | 16:36 |
| fungi | and that's inside the nodepool-builder docker container? | 16:37 |
| corvus | in the python-builder one (since that's where we run the pip commands when building the nodepool builder) | 16:38 |
| fungi | yep, okay | 16:39 |
| corvus | okay, that failed to build the netifaces wheel, but that's probably just me not doing enough prep work -- it did download 2.11.0 of cos-sdk | 16:39 |
| corvus | yeah, i'll install gcc :) | 16:40 |
| fungi | though that leaves me wondering if 2.x added a netifaces dep | 16:43 |
| corvus | that was from the whole requirements.txt, so probably from something else | 16:44 |
| fungi | yeah, would have been an indirect dependency | 16:44 |
| corvus | i'm basically stepping through the build process manually... | 16:46 |
| corvus | and it worked: | 16:46 |
| corvus | Created wheel for ibm-cos-sdk-core: filename=ibm_cos_sdk_core-2.11.0-py3-none-any.whl size=536334 sha256=f8bca20509ab93b526aaf59e697bb2ec816fe0b55036527240d89e8d54da35e3 | 16:46 |
| corvus | i'm starting to think this may be a transient error | 16:46 |
| corvus | maybe we should just put a mininum version in our reqs so that if it happens again the build will fail? | 16:46 |
| fungi | that seems wise | 16:50 |
| fungi | and yeah, no significant changes that i can see in the setup.py between 1.2.1 and 2.0.0 | 16:50 |
| corvus | i pushed up a change: remote: https://review.opendev.org/c/zuul/nodepool/+/834376 Add a mininum version for ibm-cos-sdk-core [NEW] | 16:50 |
| fungi | thanks! | 16:51 |
| corvus | i'm going to restart the mergers and then the schedulers now | 16:51 |
| fungi | awesome. i'm around if you need help | 16:51 |
| corvus | zuul01 is down | 16:58 |
| corvus | zuul01 is starting up | 17:01 |
| corvus | it's up; stopping zuul02 now | 17:08 |
| corvus | zuul02 is starting up again | 17:11 |
| corvus | i am pleased with the performance of the lb :) | 17:11 |
| corvus | all done | 17:19 |
| fungi | i stepped away for a few, i guess webui failover worked well? | 17:59 |
| corvus | yeah -- i didn't have a status screen going.. i should have done that... but i did check the components page periodically and did not notice an outage | 20:05 |
| fungi | very cool | 20:16 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!