Monday, 2022-02-07

*** rlandy_ is now known as rlandy|out01:33
*** ysandeep|out is now known as ysandeep05:01
*** ysandeep is now known as ysandeep|afk06:35
*** ysandeep|afk is now known as ysandeep06:51
*** amoralej|off is now known as amoralej07:02
*** ysandeep is now known as ysandeep|lunch07:28
opendevreviewMerged openstack/project-config master: Retire security-specs repo
*** ysandeep|lunch is now known as ysandeep08:29
*** jpena|off is now known as jpena08:38
*** ysandeep is now known as ysandeep|away10:01
*** pojadhav is now known as pojadhav|afk10:50
*** ysandeep|away is now known as ysandeep10:50
*** ralonsoh_ is now known as ralonsoh10:51
*** dviroel|out is now known as dviroel|river10:58
*** dviroel|river is now known as dviroel|rover10:58
*** dviroel|rover is now known as dviroel|ruck11:01
*** rlandy|out is now known as rlandy|ruck11:14
lourotclarkb, fungi o/ I see the new project cirros/cirros ( ) validates importing a repo whose default branch is `main`. Did the experiment work?12:19
*** dviroel|ruck is now known as dviroel|ruck|afk12:23
fricklerlourot: yes, both the import and running jobs seem to work fine, we just had to fix our project-config linting a bit12:53
*** amoralej is now known as amoralej|lunch12:56
*** amoralej|lunch is now known as amoralej13:53
fricklerclicking "" gives "Something went wrong" and that message stays until I reload the page. all other jobs seem to display fine13:54
fungiinteresting. i guess that's because there's no actual definition for the job, it's a built-in name13:58
fungimay need special handling13:58
*** dviroel|ruck|afk is now known as dviroel|ruck14:03
ianychoi[m]Hi all, openstack translation jobs are failing and it looks like that most errors come from "ModuleNotFoundError: No module named 'setuptools_rust'" while installing cryptography14:05
ianychoi[m] pip package - . Any suggestions to solve such issues?14:05
fungicryptography added a build-time dependency on rust a while back, but that's been quite some time14:07
fungilooks like it's succeeding sometimes but also a mix of failure and retry_limit results14:10
fungihave you noticed any pattern in which projects it's working for and which are failing?14:12
fungitaking openstack/nova as an example, it looks like that job was running successfully up through 2022-02-03 19:02:03 but had started breaking as of 2022-02-04 03:32:0514:16
fungiin nova's case, the jobs are retrying and eventually returning a retry_limit result14:16
fungithe "ensure-sphinx: Install found doc requirements" in pre-run is complaining because wheel isn't present when installing pillow, and then fails to build the sdist of it due to missing jpeg libs14:20
fungiianychoi[m]: we may be looking at multiple failure modes, hard to know if they're all related to the same cause yet14:20
fungithat 2022-02-03/2022-02-04 window may coincide with the release of pip 22.0.3 though odd earlier 22.x versions weren't causing issues14:22
fungisetuptools and virtualenv also had releases around that date14:23
fungivirtualenv 20.13.1 released saturday and updated its included pip to 22.0.3 and setuptools to 60.6.0, but we were seeing errors days prior to that14:30
fungiahh, pillow itself had a new release on 2022-02-0314:31
fungii bet it's that we previously had prebuilt wheels of it and now we don't14:31
fungilooks like they didn't upload any python 3.6 wheels14:32
fungiand yep, those translation update jobs are still running on ubuntu-bionic nodes14:33
fungiianychoi[m]: ^ we should try updating them to run on ubuntu-focal and see if that at least solves the pillow problem14:33
fungiif cryptography recently stopped uploading cp36 wheels in a recent release, it could explan the sudden emergence of the other error you noted14:47
funginope, that's not it, latest cryptography is from december and has cp36 wheels, i'll have to see if i can figure out why it wants to build it from sdist anyway14:48
ianychoi[m]fungi: thanks a lot for your investigation. Seeing total 5 different kind of errors now
*** dviroel|ruck is now known as dviroel|ruck|lunch15:13
clarkbfrickler: did you end up setting the default branch for cirros in the zuul config somewhere? I seem to recall that is necessary but don't recall seeing that update. Maybe it happened in repo?15:38
clarkbfrickler: mostly curious so I know how to direct future requests along these lines15:38
*** akahat|rover is now known as akahat|dinner15:43
*** ysandeep is now known as ysandeep|dinner15:45
opendevreviewAurelien Lourot proposed openstack/project-config master: Mirror charm-interface-openstack-loadbalancer to GitHub
opendevreviewAurelien Lourot proposed openstack/project-config master: Mirror charm-ops-interface-openstack-loadbalancer to GitHub
fricklerclarkb: and then in the repo in .gitreview, but nothing zuul-specific iirc15:52
clarkbfrickler: ah ok. The zuul config may only matter if you rely on zuul's fallback behavior if branches don't align15:55
fungiright, more likely to crop up once you start trying to add devstack jobs15:55
clarkbfor example if a change to nova stable/foo is tested with cirros by default zuul will look for a stable/foo branch in cirros. If it doesn't find that it will look for master as the fallback but cirros needs to tell ti to look for main15:55
fungiianychoi[m]: looking through the 5 classes of errors you have there, all of them look related to dependencies dropping python 3.6 support. i expect running that job on ubuntu-focal nodes would address them16:00
fungiit's either things that have recently release but not provided cp36 wheels, projects which have made recent releases declaring requires_python >=3.7, constraints limited to python 3.8 because of openstack/releases master branch expecting focal nodes, and so on16:02
fungicryptography failing to build from sdist because of expecting rust installed is the one outlier, i'm unclear on why that would have started recently16:03
clarkbperhaps pip was/is too old there to understand the abi3 wheels for cryptography?16:06
clarkbor maybe pip updated and has a regression around that?16:06
fungiyeah, but then why did it too only start on 2022-02-03?16:06
fungicryptography and setuptools_rust haven't seen new releases since december16:07
fungiunless maybe constraints changed on thursday16:08
fricklermaybe we'd need to add main to for cirros, I still need to learn how to deal with artifacts before I get to test that16:08
*** ysandeep|dinner is now known as ysandeep16:08
*** dviroel|ruck|lunch is now known as dviroel|ruck16:09
fricklerfungi: clarkb: ensure-sphinx switched to py3
frickleralso switched from virtualenv to python3 -m venv iiuc16:12
fungifrickler: oh, great point, that may have been previously relying on older versions of dependencies as a result of using python 2.716:12
fungiand switching to python3 -m venv would explain why wheel wouldn't be preinstalled too16:13
fungithat commit did indeed merge on 2022-02-0316:13
fricklerah, so just adding pip install wheel might be enough to solve this, right16:14
fungiwell, it may solve the ones failing because of "no target bdist_wheel" or whatever, but not necessarily the others16:14
fungithough yes, adding wheel to the sphinx venv is almost certainly a good idea either way16:15
fungiby default, a python3 venv appears to come with only pip and setuptools preinstalled16:17
clarkbfungi: I thought no target bdist_wheel was supposed to be non fatal16:19
clarkbit should fallback to the old system iirc16:19
fungibut also if we're going to run these jobs with master branch of openstack/releases upper-constraints.txt then they're going to need to run on focal16:19
fungiideally, projects would add the necessary packages for building their doc requirements to bindep.txt (tagged with a "doc" profile), but i have a feeling most of these are relying on us to have build wheels in our cache16:21
*** amoralej is now known as amoralej|off16:22
fungiand yeah, i expect the bdist_wheel warnings are a symptom of the job no longer finding approproate wheel packages for the dependencies it's being asked to install16:22
fricklerfor the cryptography issue, very likely pip from python3-pip on bionic is too old16:24
clarkbfrickler: ya that is my suspicion thati t doesn't understand abi3 wheels16:25
clarkbfungi: to be fair I think even cryptography expects you to use their wheels because getting a rust toolchain isn't as simple as installing a bunch of pacakegs with apt16:26
fungiyeah, looks like it's been more than a year since we've built pillow for bionic:
fungiajaeger also pointed out by e-mail that one reason we didn't move those jobs to focal is that the zanata client won't work with the newer jdk16:42
fungiand suggests this could be the final nail in zanata's coffing, metaphorically (or maybe the exorcism which allows its zombified corpse to finally rest)16:44
clarkbya we should probably start yelling about this more (though I think the i18n team has been well aware)16:44
*** marios is now known as marios|out16:48
*** ysandeep is now known as ysandeep|out16:59
fungiclarkb: in good news, my canary project has pbr.json in its sdists and wheels again now that pbr 5.8.1 is out17:07
*** akahat|dinner is now known as akaha|rover17:08
funginow if i could just remember who originally brought the problem to our attention by asking how to find the git ref in an uploaded tarball17:09
clarkbmnasiadka called it out in scrollback of this channel17:10
fungiahh, yep thanks. i didn't scroll up far enough in here17:10
fungimnasiadka_: ^ any sdist tarballs (or wheels) built after 07:45 utc today should have a pbr.json with commit ref info in them again17:12
fungi(and also any made prior to the pbr 5.8.0 release on 2021-11-18)17:13
fungispecifically, if you parse that json blob, its top-level dict should have a key called "git_version" whose value is the 7-digit abbreviation of the commit from which it was built17:16
funginote that this is only applicable for tarballs built in post-merge pipelines (those triggered by gerrit's ref-updated events like the "post" pipeline, or timer triggers like "periodic" or tags like "release")17:17
fungibut that's likely to be all the ones openstack uploads to the tarballs site anyway17:18
clarkbcorvus: fungi: ok I've pulled up the zuul merger code and I think that the merger stop command is effectively a graceful stop already17:26
clarkbIt sets the running flag in the main loop to false which means that it will either immediately exit if not performing work or exit after the current merge task completes17:27
clarkbI think we can just update our tooling/docs to say use the stop command. Would it make sense to add graceful as an alias for stop to be more explicit?17:27
fungimakes sense to me17:31
clarkbya I think having a graceful command for other tools can give the impression that stop without graceful is never graceful. I'll work on a quick alias change and zuul reviewers can tell me if I am crazy :)17:32
fungibut also if it's decided later that there's something even more graceful to be done, that's the peg to hang it on17:34
corvusclarkb: okay cool, so all we need to do in our playbook is issue the stop command instead of running docker-compose17:38
clarkbcorvus: yup I think so17:39
opendevreviewJames E. Blair proposed opendev/system-config master: Add the start of a Zuul rolling restart playbook
*** jpena is now known as jpena|off17:43
opendevreviewJames E. Blair proposed opendev/system-config master: Add the start of a Zuul rolling restart playbook
clarkbthis gitea 1.16 update is fun. I'm having a hard time understanding how we set config for both the web and ssh daemons properly and 1.16 changed that a bit18:36
clarkbaha I think I get it now18:45
clarkbwell at least one piece of the puzzle18:45
opendevreviewClark Boylan proposed opendev/system-config master: Update Gitea to 1.16.1
clarkbI half expect ^ to fail but we have to start somewhere19:19
jrossercan I use job.vars to pass a setting into, let’s say, common-mirrors role?19:33
*** noonedeadpunk_ is now known as noonedeadpunk19:34
opendevreviewClark Boylan proposed opendev/system-config master: Update Gitea to 1.16.1
clarkbjrosser: I think there are restrictions on that since the mirrors stuff is running in a trusted context19:37
clarkbcorvus: ^ you can probably explain that better than I can19:37
jrosserI saw something about limited templating, in this case I was thinking about a bool19:37
fungijrosser: where's the common-mirrors role? i can't find it19:37
fungioh! configure-mirrors?19:38
jrossermy fault :) configure-mirrors19:38
corvusyou can definitely pass vars into the role, but passing in ansible jinja templates as the content of those vars might not be permissible under some circumstances19:39
jrosserok cool thankyou - I will think about it a bit first19:40
clarkbin general i think it would probably be more reliable for children jobs to update things like that when they start runningrather than trying to voerride the parent19:41
clarkbSince repo configs can be site dependent depending on how things are configured19:41
corvusa bool should be fine19:41
corvus(tho whether it's a good idea for configure-mirrors to trust a bool passed in is a different story -- i don't want to prejudge that)19:41
fungi"When running a trusted playbook, the value of variables will be frozen at the start of the job. Therefore if the value of the variable is an Ansible Jinja template, it may only reference values which are known at the start of the job, and its value will not change. Untrusted playbooks dynamically evaluate variables and are not limited by this restriction."19:43
clarkbfungi: thanks! I was looking for that19:43
clarkb I ended up there and wasn't finding it19:43
clarkbmaybe we need to combine that type of content somehow in the docs19:43
opendevreviewClark Boylan proposed opendev/system-config master: Update Gitea to 1.16.1
clarkbour periodic jobs haven't run since the 3rd. I believe this is why the cert is going to expure in under 3 0days21:00
clarkbI'm not sure yet why this is happening. Zuul's error report page for the openstack tenant doesn't show anything for system-config21:01
clarkbI think I see it21:01
opendevreviewClark Boylan proposed opendev/system-config master: Update infra-prod-service-review dependencies
clarkbIt would be great if Zuul can be more explicit about those issues21:05
clarkbI'm ot even sure that is all that is wrong right now :/21:05
fungiahh, thanks, i was meaning to look into that21:08
opendevreviewGage Hugo proposed openstack/project-config master: Retire security-specs repo - Step 3
*** dviroel|ruck is now known as dviroel|out22:41
clarkb for the gitea 1.16.1 upgrade passes testing now. I've WIP'd it because we should figure out testing of git replication over ssh first before landing that (since the ssh daemon is impacted by the breaking changes made to docker image stuff)22:54
fungiwe do test that in our integration testing though, right?22:55
clarkbfungi: we do not.22:56
clarkbwe don't currently have a job that deploys gitea and gerrit together22:56
fungioh, huh. i thought that was already there22:56
clarkband I think the system-config push into gitea is done over https not ssh22:56
clarkbya confirmed the git push happens over https not ssh22:56
clarkbfungi: landing that would be a good idea23:00
opendevreviewMerged opendev/system-config master: Update infra-prod-service-review dependencies
ianwclarkb: ^ so the hope is that gets periodic jobs up again?  just want to make sure i keep an eye23:46
clarkbianw: yup23:55
clarkbits possible there are other issues lurking, but that one stood out to me based on timing23:56
clarkb(since we cleaned up the 3.3 images recently)23:56

Generated by 2.17.3 by Marius Gedminas - find it at!