| *** rlandy_ is now known as rlandy|out | 01:33 | |
| *** ysandeep|out is now known as ysandeep | 05:01 | |
| *** ysandeep is now known as ysandeep|afk | 06:35 | |
| *** ysandeep|afk is now known as ysandeep | 06:51 | |
| *** amoralej|off is now known as amoralej | 07:02 | |
| *** ysandeep is now known as ysandeep|lunch | 07:28 | |
| opendevreview | Merged openstack/project-config master: Retire security-specs repo https://review.opendev.org/c/openstack/project-config/+/825401 | 07:34 |
|---|---|---|
| *** ysandeep|lunch is now known as ysandeep | 08:29 | |
| *** jpena|off is now known as jpena | 08:38 | |
| *** ysandeep is now known as ysandeep|away | 10:01 | |
| *** pojadhav is now known as pojadhav|afk | 10:50 | |
| *** ysandeep|away is now known as ysandeep | 10:50 | |
| *** ralonsoh_ is now known as ralonsoh | 10:51 | |
| *** dviroel|out is now known as dviroel|river | 10:58 | |
| *** dviroel|river is now known as dviroel|rover | 10:58 | |
| *** dviroel|rover is now known as dviroel|ruck | 11:01 | |
| *** rlandy|out is now known as rlandy|ruck | 11:14 | |
| lourot | clarkb, fungi o/ I see the new project cirros/cirros ( https://review.opendev.org/c/openstack/project-config/+/827719/4/gerrit/projects.yaml ) validates importing a repo whose default branch is `main`. Did the experiment work? | 12:19 |
| *** dviroel|ruck is now known as dviroel|ruck|afk | 12:23 | |
| frickler | lourot: yes, both the import and running jobs seem to work fine, we just had to fix our project-config linting a bit | 12:53 |
| *** amoralej is now known as amoralej|lunch | 12:56 | |
| *** amoralej|lunch is now known as amoralej | 13:53 | |
| frickler | clicking "https://zuul.opendev.org/t/openstack/job/noop" gives "Something went wrong" and that message stays until I reload the page. all other jobs seem to display fine | 13:54 |
| fungi | interesting. i guess that's because there's no actual definition for the job, it's a built-in name | 13:58 |
| fungi | may need special handling | 13:58 |
| *** dviroel|ruck|afk is now known as dviroel|ruck | 14:03 | |
| ianychoi[m] | Hi all, openstack translation jobs are failing and it looks like that most errors come from "ModuleNotFoundError: No module named 'setuptools_rust'" while installing cryptography | 14:05 |
| ianychoi[m] | pip package - https://zuul.opendev.org/t/openstack/builds?job_name=propose-translation-update&job_name=upstream-translation-update&result=FAILURE . Any suggestions to solve such issues? | 14:05 |
| fungi | cryptography added a build-time dependency on rust a while back, but that's been quite some time | 14:07 |
| fungi | looks like it's succeeding sometimes but also a mix of failure and retry_limit results | 14:10 |
| fungi | have you noticed any pattern in which projects it's working for and which are failing? | 14:12 |
| fungi | taking openstack/nova as an example, it looks like that job was running successfully up through 2022-02-03 19:02:03 but had started breaking as of 2022-02-04 03:32:05 | 14:16 |
| fungi | in nova's case, the jobs are retrying and eventually returning a retry_limit result | 14:16 |
| fungi | the "ensure-sphinx: Install found doc requirements" in pre-run is complaining because wheel isn't present when installing pillow, and then fails to build the sdist of it due to missing jpeg libs | 14:20 |
| fungi | ianychoi[m]: we may be looking at multiple failure modes, hard to know if they're all related to the same cause yet | 14:20 |
| fungi | that 2022-02-03/2022-02-04 window may coincide with the release of pip 22.0.3 though odd earlier 22.x versions weren't causing issues | 14:22 |
| fungi | setuptools and virtualenv also had releases around that date | 14:23 |
| fungi | virtualenv 20.13.1 released saturday and updated its included pip to 22.0.3 and setuptools to 60.6.0, but we were seeing errors days prior to that | 14:30 |
| fungi | ahh, pillow itself had a new release on 2022-02-03 | 14:31 |
| fungi | i bet it's that we previously had prebuilt wheels of it and now we don't | 14:31 |
| fungi | looks like they didn't upload any python 3.6 wheels | 14:32 |
| fungi | and yep, those translation update jobs are still running on ubuntu-bionic nodes | 14:33 |
| fungi | ianychoi[m]: ^ we should try updating them to run on ubuntu-focal and see if that at least solves the pillow problem | 14:33 |
| fungi | if cryptography recently stopped uploading cp36 wheels in a recent release, it could explan the sudden emergence of the other error you noted | 14:47 |
| fungi | nope, that's not it, latest cryptography is from december and has cp36 wheels, i'll have to see if i can figure out why it wants to build it from sdist anyway | 14:48 |
| ianychoi[m] | fungi: thanks a lot for your investigation. Seeing total 5 different kind of errors now https://paste.opendev.org/show/812557/ | 15:09 |
| *** dviroel|ruck is now known as dviroel|ruck|lunch | 15:13 | |
| clarkb | frickler: did you end up setting the default branch for cirros in the zuul config somewhere? I seem to recall that is necessary but don't recall seeing that update. Maybe it happened in repo? | 15:38 |
| clarkb | frickler: mostly curious so I know how to direct future requests along these lines | 15:38 |
| *** akahat|rover is now known as akahat|dinner | 15:43 | |
| *** ysandeep is now known as ysandeep|dinner | 15:45 | |
| opendevreview | Aurelien Lourot proposed openstack/project-config master: Mirror charm-interface-openstack-loadbalancer to GitHub https://review.opendev.org/c/openstack/project-config/+/828166 | 15:47 |
| opendevreview | Aurelien Lourot proposed openstack/project-config master: Mirror charm-ops-interface-openstack-loadbalancer to GitHub https://review.opendev.org/c/openstack/project-config/+/828166 | 15:50 |
| frickler | clarkb: https://review.opendev.org/c/openstack/project-config/+/827719/4/gerrit/projects.yaml#147 and then in the repo in .gitreview, but nothing zuul-specific iirc | 15:52 |
| clarkb | frickler: ah ok. The zuul config may only matter if you rely on zuul's fallback behavior if branches don't align | 15:55 |
| fungi | right, more likely to crop up once you start trying to add devstack jobs | 15:55 |
| clarkb | for example if a change to nova stable/foo is tested with cirros by default zuul will look for a stable/foo branch in cirros. If it doesn't find that it will look for master as the fallback but cirros needs to tell ti to look for main | 15:55 |
| fungi | ianychoi[m]: looking through the 5 classes of errors you have there, all of them look related to dependencies dropping python 3.6 support. i expect running that job on ubuntu-focal nodes would address them | 16:00 |
| fungi | it's either things that have recently release but not provided cp36 wheels, projects which have made recent releases declaring requires_python >=3.7, constraints limited to python 3.8 because of openstack/releases master branch expecting focal nodes, and so on | 16:02 |
| fungi | cryptography failing to build from sdist because of expecting rust installed is the one outlier, i'm unclear on why that would have started recently | 16:03 |
| clarkb | perhaps pip was/is too old there to understand the abi3 wheels for cryptography? | 16:06 |
| clarkb | or maybe pip updated and has a regression around that? | 16:06 |
| fungi | yeah, but then why did it too only start on 2022-02-03? | 16:06 |
| fungi | https://zuul.opendev.org/t/openstack/builds?job_name=propose-translation-update&project=openstack/python-openstackclient | 16:06 |
| fungi | cryptography and setuptools_rust haven't seen new releases since december | 16:07 |
| fungi | unless maybe constraints changed on thursday | 16:08 |
| frickler | maybe we'd need to add main to https://opendev.org/openstack/devstack/src/branch/master/.zuul.yaml#L1-L7 for cirros, I still need to learn how to deal with artifacts before I get to test that | 16:08 |
| *** ysandeep|dinner is now known as ysandeep | 16:08 | |
| *** dviroel|ruck|lunch is now known as dviroel|ruck | 16:09 | |
| frickler | fungi: clarkb: ensure-sphinx switched to py3 https://opendev.org/zuul/zuul-jobs/commit/ab4042e60a5d1474438ffedd12ecfe9ea6d6338a | 16:11 |
| frickler | also switched from virtualenv to python3 -m venv iiuc | 16:12 |
| fungi | frickler: oh, great point, that may have been previously relying on older versions of dependencies as a result of using python 2.7 | 16:12 |
| fungi | and switching to python3 -m venv would explain why wheel wouldn't be preinstalled too | 16:13 |
| fungi | that commit did indeed merge on 2022-02-03 | 16:13 |
| frickler | ah, so just adding pip install wheel might be enough to solve this, right | 16:14 |
| fungi | well, it may solve the ones failing because of "no target bdist_wheel" or whatever, but not necessarily the others | 16:14 |
| fungi | though yes, adding wheel to the sphinx venv is almost certainly a good idea either way | 16:15 |
| fungi | by default, a python3 venv appears to come with only pip and setuptools preinstalled | 16:17 |
| clarkb | fungi: I thought no target bdist_wheel was supposed to be non fatal | 16:19 |
| clarkb | it should fallback to the old system iirc | 16:19 |
| fungi | right | 16:19 |
| fungi | but also if we're going to run these jobs with master branch of openstack/releases upper-constraints.txt then they're going to need to run on focal | 16:19 |
| fungi | ideally, projects would add the necessary packages for building their doc requirements to bindep.txt (tagged with a "doc" profile), but i have a feeling most of these are relying on us to have build wheels in our cache | 16:21 |
| *** amoralej is now known as amoralej|off | 16:22 | |
| fungi | and yeah, i expect the bdist_wheel warnings are a symptom of the job no longer finding approproate wheel packages for the dependencies it's being asked to install | 16:22 |
| frickler | for the cryptography issue, very likely pip from python3-pip on bionic is too old | 16:24 |
| clarkb | frickler: ya that is my suspicion thati t doesn't understand abi3 wheels | 16:25 |
| clarkb | fungi: to be fair I think even cryptography expects you to use their wheels because getting a rust toolchain isn't as simple as installing a bunch of pacakegs with apt | 16:26 |
| fungi | yeah, looks like it's been more than a year since we've built pillow for bionic: https://static.opendev.org/mirror/wheel/ubuntu-18.04-x86_64/p/pillow/ | 16:26 |
| fungi | ajaeger also pointed out by e-mail that one reason we didn't move those jobs to focal is that the zanata client won't work with the newer jdk | 16:42 |
| fungi | and suggests this could be the final nail in zanata's coffing, metaphorically (or maybe the exorcism which allows its zombified corpse to finally rest) | 16:44 |
| fungi | s/coffing/coffin/ | 16:44 |
| clarkb | ya we should probably start yelling about this more (though I think the i18n team has been well aware) | 16:44 |
| *** marios is now known as marios|out | 16:48 | |
| *** ysandeep is now known as ysandeep|out | 16:59 | |
| fungi | clarkb: in good news, my canary project has pbr.json in its sdists and wheels again now that pbr 5.8.1 is out | 17:07 |
| *** akahat|dinner is now known as akaha|rover | 17:08 | |
| clarkb | nice | 17:08 |
| fungi | now if i could just remember who originally brought the problem to our attention by asking how to find the git ref in an uploaded tarball | 17:09 |
| clarkb | mnasiadka called it out in scrollback of this channel | 17:10 |
| fungi | ahh, yep thanks. i didn't scroll up far enough in here | 17:10 |
| fungi | mnasiadka_: ^ any sdist tarballs (or wheels) built after 07:45 utc today should have a pbr.json with commit ref info in them again | 17:12 |
| fungi | (and also any made prior to the pbr 5.8.0 release on 2021-11-18) | 17:13 |
| fungi | specifically, if you parse that json blob, its top-level dict should have a key called "git_version" whose value is the 7-digit abbreviation of the commit from which it was built | 17:16 |
| fungi | note that this is only applicable for tarballs built in post-merge pipelines (those triggered by gerrit's ref-updated events like the "post" pipeline, or timer triggers like "periodic" or tags like "release") | 17:17 |
| fungi | but that's likely to be all the ones openstack uploads to the tarballs site anyway | 17:18 |
| clarkb | corvus: fungi: ok I've pulled up the zuul merger code and I think that the merger stop command is effectively a graceful stop already | 17:26 |
| clarkb | It sets the running flag in the main loop to false which means that it will either immediately exit if not performing work or exit after the current merge task completes | 17:27 |
| clarkb | I think we can just update our tooling/docs to say use the stop command. Would it make sense to add graceful as an alias for stop to be more explicit? | 17:27 |
| fungi | makes sense to me | 17:31 |
| clarkb | ya I think having a graceful command for other tools can give the impression that stop without graceful is never graceful. I'll work on a quick alias change and zuul reviewers can tell me if I am crazy :) | 17:32 |
| fungi | but also if it's decided later that there's something even more graceful to be done, that's the peg to hang it on | 17:34 |
| corvus | clarkb: okay cool, so all we need to do in our playbook is issue the stop command instead of running docker-compose | 17:38 |
| clarkb | corvus: yup I think so | 17:39 |
| opendevreview | James E. Blair proposed opendev/system-config master: Add the start of a Zuul rolling restart playbook https://review.opendev.org/c/opendev/system-config/+/828176 | 17:43 |
| *** jpena is now known as jpena|off | 17:43 | |
| opendevreview | James E. Blair proposed opendev/system-config master: Add the start of a Zuul rolling restart playbook https://review.opendev.org/c/opendev/system-config/+/828176 | 17:43 |
| clarkb | this gitea 1.16 update is fun. I'm having a hard time understanding how we set config for both the web and ssh daemons properly and 1.16 changed that a bit | 18:36 |
| clarkb | aha I think I get it now | 18:45 |
| clarkb | well at least one piece of the puzzle | 18:45 |
| opendevreview | Clark Boylan proposed opendev/system-config master: Update Gitea to 1.16.1 https://review.opendev.org/c/opendev/system-config/+/828184 | 19:19 |
| clarkb | I half expect ^ to fail but we have to start somewhere | 19:19 |
| jrosser | can I use job.vars to pass a setting into, let’s say, common-mirrors role? | 19:33 |
| *** noonedeadpunk_ is now known as noonedeadpunk | 19:34 | |
| opendevreview | Clark Boylan proposed opendev/system-config master: Update Gitea to 1.16.1 https://review.opendev.org/c/opendev/system-config/+/828184 | 19:36 |
| clarkb | jrosser: I think there are restrictions on that since the mirrors stuff is running in a trusted context | 19:37 |
| clarkb | corvus: ^ you can probably explain that better than I can | 19:37 |
| jrosser | I saw something about limited templating, in this case I was thinking about a bool | 19:37 |
| fungi | jrosser: where's the common-mirrors role? i can't find it | 19:37 |
| jrosser | https://opendev.org/zuul/zuul-jobs/src/branch/master/roles/configure-mirrors/tasks/mirror/CentOS-8.yaml | 19:38 |
| jrosser | argh | 19:38 |
| fungi | oh! configure-mirrors? | 19:38 |
| jrosser | my fault :) configure-mirrors | 19:38 |
| corvus | you can definitely pass vars into the role, but passing in ansible jinja templates as the content of those vars might not be permissible under some circumstances | 19:39 |
| jrosser | ok cool thankyou - I will think about it a bit first | 19:40 |
| clarkb | in general i think it would probably be more reliable for children jobs to update things like that when they start runningrather than trying to voerride the parent | 19:41 |
| clarkb | Since repo configs can be site dependent depending on how things are configured | 19:41 |
| corvus | a bool should be fine | 19:41 |
| corvus | (tho whether it's a good idea for configure-mirrors to trust a bool passed in is a different story -- i don't want to prejudge that) | 19:41 |
| fungi | "When running a trusted playbook, the value of variables will be frozen at the start of the job. Therefore if the value of the variable is an Ansible Jinja template, it may only reference values which are known at the start of the job, and its value will not change. Untrusted playbooks dynamically evaluate variables and are not limited by this restriction." | 19:43 |
| fungi | https://zuul-ci.org/docs/zuul/latest/config/job.html#attr-job.vars | 19:43 |
| clarkb | fungi: thanks! I was looking for that | 19:43 |
| clarkb | https://zuul-ci.org/docs/zuul/latest/job-content.html#user-jobs-job-variables I ended up there and wasn't finding it | 19:43 |
| clarkb | maybe we need to combine that type of content somehow in the docs | 19:43 |
| opendevreview | Clark Boylan proposed opendev/system-config master: Update Gitea to 1.16.1 https://review.opendev.org/c/opendev/system-config/+/828184 | 20:53 |
| clarkb | our periodic jobs haven't run since the 3rd. I believe this is why the ptg.opendev.org cert is going to expure in under 3 0days | 21:00 |
| clarkb | I'm not sure yet why this is happening. Zuul's error report page for the openstack tenant doesn't show anything for system-config | 21:01 |
| clarkb | I think I see it | 21:01 |
| opendevreview | Clark Boylan proposed opendev/system-config master: Update infra-prod-service-review dependencies https://review.opendev.org/c/opendev/system-config/+/828194 | 21:05 |
| clarkb | It would be great if Zuul can be more explicit about those issues | 21:05 |
| clarkb | I'm ot even sure that is all that is wrong right now :/ | 21:05 |
| fungi | ahh, thanks, i was meaning to look into that | 21:08 |
| opendevreview | Gage Hugo proposed openstack/project-config master: Retire security-specs repo - Step 3 https://review.opendev.org/c/openstack/project-config/+/827178 | 22:03 |
| *** dviroel|ruck is now known as dviroel|out | 22:41 | |
| clarkb | https://review.opendev.org/c/opendev/system-config/+/828184 for the gitea 1.16.1 upgrade passes testing now. I've WIP'd it because we should figure out testing of git replication over ssh first before landing that (since the ssh daemon is impacted by the breaking changes made to docker image stuff) | 22:54 |
| fungi | we do test that in our integration testing though, right? | 22:55 |
| clarkb | fungi: we do not. | 22:56 |
| clarkb | we don't currently have a job that deploys gitea and gerrit together | 22:56 |
| fungi | oh, huh. i thought that was already there | 22:56 |
| clarkb | and I think the system-config push into gitea is done over https not ssh | 22:56 |
| clarkb | ya confirmed the git push happens over https not ssh | 22:56 |
| clarkb | fungi: https://review.opendev.org/c/opendev/system-config/+/828194 landing that would be a good idea | 23:00 |
| opendevreview | Merged opendev/system-config master: Update infra-prod-service-review dependencies https://review.opendev.org/c/opendev/system-config/+/828194 | 23:13 |
| ianw | clarkb: ^ so the hope is that gets periodic jobs up again? just want to make sure i keep an eye | 23:46 |
| clarkb | ianw: yup | 23:55 |
| clarkb | its possible there are other issues lurking, but that one stood out to me based on timing | 23:56 |
| clarkb | (since we cleaned up the 3.3 images recently) | 23:56 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!