Thursday, 2021-11-18

Clark[m]Maybe land them one at a time to keep the Delta small if anything has a sad? But ya I don't expect issues00:23
ianwyeah for sure00:25
*** kevinz_ is now known as kevinz02:24
*** ysandeep|PTO is now known as ysandeep03:54
*** pojadhav|afk is now known as pojadhav04:01
opendevreviewMerged opendev/system-config master: infra-prod: remove duplicated bridge key setup
*** ysandeep is now known as ysandeep|afk05:13
ianwnext hourly run should exercise ^05:39
*** ysandeep|afk is now known as ysandeep06:19
opendevreviewMerged openstack/diskimage-builder master: Add openEuler jobs back
opendevreviewMerged openstack/diskimage-builder master: Revert "centos 9-stream: make non-voting for mirror issues"
opendevreviewMerged openstack/project-config master: Grafana: remove midonet and fwaas dashboards
opendevreviewMerged openstack/project-config master: Set Fedora 34 min-ready to zero
*** ysandeep is now known as ysandeep|lunch07:41
*** pojadhav- is now known as pojadhav08:08
*** jpena|off is now known as jpena08:36
*** ysandeep|lunch is now known as ysandeep08:38
soniya29|ruckhello all, it seems mirrors aren't reachable -
soniya29|ruckcan anybody help over this issue?08:54
*** ykarel is now known as ykarel|lunch09:07
opendevreviewDr. Jens Harbott proposed openstack/project-config master: Remove Fedora 34
fricklersoniya29|ruck: if the mirror weren't reachable, it couldn't respond with a 404. the mirrors might be in an inconsistent state though, likely caused by syncing from an in-flux upstream. not much we can do except choosing a different upstream if the issue persists after the next update09:42
opendevreviewThierry Carrez proposed openstack/project-config master: Fix ACLs for deliverables under release management
*** ysandeep is now known as ysandeep|afk10:09
ianwsoniya29|ruck: also note for some of those they're on rax and so are referring to mirror-int.<region> which is an internal rax-only IP.  so if you're trying to see if that's live from outside it won't work.  so that might be a red-herring10:14
ianw(at some point rax routing was such that using the external addresses was unreliable, that's how we ended up with that.  it's the only region like that, though)10:15
ianw has run with 818190 applied (and others).  so looking good for that10:16
soniya29|ruckfrickler, ianw, thanks :)10:29
*** ykarel|lunch is now known as ykarel11:03
*** ysandeep|afk is now known as ysandeep11:06
*** pojadhav is now known as pojadhav|brb11:35
*** jpena is now known as jpena|lunch12:23
*** pojadhav|brb is now known as pojadhav12:31
*** jpena|lunch is now known as jpena13:16
opendevreviewLajos Katona proposed openstack/project-config master: Grafana dashboard for instance usage of projects
*** ysandeep is now known as ysandeep|dinner14:10
opendevreviewClark Boylan proposed opendev/bindep master: Try out PBR pep 517 support
*** ysandeep|dinner is now known as ysandeep14:49
dtantsurhey folks, a zuul question follows. do we have a pattern for running unit tests of a service on a library?15:07
dtantsurkind of cross tests on requirements but on a library?15:07
fungidtantsur: yes, check out the oslo projects' test definitions15:08
fungithey have a bunch, e.g. run nova unit tests with a proposed patch for oslo.messaging or whatever15:08
dtantsuraha, a great pointer15:08
dtantsurfor some reason I forgot about Oslo Oo15:08
fungiaww, don't tell the oslo team that! ;)15:09
dtantsurtoo much time working on metal3 recently15:10
dtantsurhmm, cannot find anything suitable15:11
dtantsurmaybe lib-forward-testing-python315:11
fungii'll look15:12
*** ykarel is now known as ykarel|away15:13
*** pojadhav is now known as pojadhav|afk15:14
dtantsur is similar, but I need it to run on the library, not on the service15:15
dtantsur(running on the service will work as well, I guess)15:15
*** akahat|rover is now known as akahat|dinner15:19
fungidtantsur: there's openstack-tox-py36-with-oslo-master defined in openstack-zuul-jobs, that's probably what i was thinking about15:21
fungisimilarly openstack-tox-py36-with-neutron-lib-master15:21
dtantsuryeah, makes sense. I guess I should read what tox_install_siblings does and whether it matches my case15:21
fungidtantsur: a quick summary is that if a project is in the required-projects list for the job, then that role preinstalls the project into the tox env from (potentially speculative) git source rather than from pypi15:23
dtantsurgot it, thanks! but I guess if the job is running on $library, it will be the tox environment of the $library, not of the $service?15:24
clarkbdtantsur: yes but it would work the other direction too, just depends on which side you set it up on. Could be both15:25
dtantsurah, so if I define the job on $service and run it on $library, it will do the trick?15:25
clarkbNo I think the tox jobs assume they run against the context of the change event that triggered them15:26
dtantsurhmm, I see15:27
clarkbyou'd need to define a job on each side I think15:27
clarkbso that the context is correct, but I could be wrong about that. The tox jobs might be smarter than I expect them to be15:27
fungilooking at the tox role, it defaults a zuul_work_dir var to zuul.project.src_dir but you can override tat15:32
fungiif the other relevant roles those jobs are based on also respect zuul_work_dir then just setting it in your job definition would probably be most of what you need15:33
dtantsurthanks again! I'll look into it tomorrow (when I hopefully no longer have a migraine)15:36
*** ysandeep is now known as ysandeep|afk15:37
fungii hope thinking about job definitions didn't give you a migraine :/15:37
fungifeel better!15:37
clarkbpaladox: congrats on gerrit maintainership!15:50
fungiooh, yes congratulations on that!15:55
clarkbre centos-8 stream the upstream mirror shows which seems to match which is our upstream to populate
clarkbI'm pretty sure that means this isn't our fault, the actual centos package repository is broken16:00
clarkbrlandy|ruck: ^ does that seem correct to you? Do you know who to reach out to about fixing that?16:00
clarkbI don't understand rpm mirrors well enough to know what might be coming from causing 404s16:01
rlandy|ruckclarkb: try amoralej on #openstack-infra16:03
rlandy|ruckhe would know more about that16:03
opendevreviewClark Boylan proposed opendev/bindep master: Try out PBR pep 517 support
opendevreviewClark Boylan proposed opendev/bindep master: Update test tool to use stestr
clarkbfungi: ^ fyi trying to remove deprecation warnings with the stack now16:34
clarkbfungi: also it seems a few people haev noticed the git-review problem because arch upgraded. It occured to me that I'm not entirely sure what that flag was doing, but it is in the code path that most people will have undone16:41
clarkbwith that in mind should we maybe go ahead and land the fix and make a release? Or wait for more testing to happen from people with new git (I asked them to do that on the storyboard story)16:42
opendevreviewClark Boylan proposed opendev/bindep master: DNM test doc builds
clarkb818435 failed doc builds so I've pushed a noop change update to the docs t osee if that is a problem more generally16:47
clarkbfungi: it seems liek we're installin bindep as version 0.0.0 which is causing fallout. Now I wonder if new PBR is a problem somehow16:50
clarkbspot checking openstack unittest jobs we seem to get the correct version there16:53
clarkbhuh and a noop push worked fine so somethign I broke in the tox.ini update? Weird16:53
clarkboh wait no I got my changes backward. Of course. Its the update chagne that breaks whcih is good because that is the one that introduces new stuff. I feel much better now16:53
*** marios is now known as marios|out16:58
*** akahat|dinner is now known as akahat|rover16:58
clarkbok the issue is when you don't skipsdist tox uses python sdist which my change removes support for because it removes setup_requires for pbr16:59
*** ysandeep|afk is now known as ysandeep16:59
clarkbI think PBR is working fine, Its the tools around it that are gettign weird when we drop tooling17:01
opendevreviewClark Boylan proposed opendev/bindep master: Try out PBR pep 517 support
clarkbThe next thing on my list is gerritbot uid stuff. But I've just realized I don't see where/how we managed known_hosts for that so I need to do more investigating before proceeding17:10
opendevreviewClark Boylan proposed opendev/bindep master: Try out PBR pep 517 support
clarkbboom no deprecation warnings that I see17:31
clarkbfungi: ^ I think that shows that it is now possible to avoid those problems as well as stop relying on easy_install alltogether17:31
clarkbstill terrible for backward compatibility but possible :)17:32
clarkbkeynotes over, time for breakfast17:37
fungithanks, i'll take a look here shortly17:38
*** jpena is now known as jpena|off17:39
*** ysandeep is now known as ysandeep|out18:12
clarkbI think I can add the setup_requires back in and still avoid the deprecation warnings with that setup. But I'm not positive of that18:12
clarkbbut then we'd be backward compatible (but untested)18:12
clarkbgerritbot uses gerritlib which uses paramiko with a warning policy for known hosts. After inspecting the running container and the system it runs on I cannot see evidence that it is loading ssh keys from anywhere18:18
clarkbinfra-root any concern with landing given ^ maybe you want to double check those assertions?18:19
clarkbIf there aren't concerns I think we can go ahead and approve that change18:19
fungii went ahead and approved it18:30
clarkbthanks. I plan to be around all day except for the school run later so should be able to monitor18:30
*** tkajinam is now known as Guest622018:30
fungiyeah, same here (sans the school run, but i will break to cook dinner in a few hours)18:31
mgariepyhello i'm debugging mariadb for OSA and i would like to ahve access to a running node:
mgariepybefore it's delete.18:38
mgariepycan someone help me ?18:39
clarkbopenstack-ansible-deploy-infra_lxc-ubuntu-focal for seems to be the details. Give me a minute and I'll put a hold in plce18:39
mgariepythanks 18:40
clarkbthe hold requset is in place. Do you have an ssh public key published somewhere that I can add to the server once it is held?18:41
clarkbmgariepy: I guess we are just waiting for the job to timeout?18:44
mgariepyi guess18:45
mgariepyit should be soonish  :D18:45
clarkbmgariepy: root@
clarkbplease let us know when you are done and we can reclaim the instance. Also always a good idea to think about how jobs and logs can be improved to expose what is going on without direct access for the future once you know how it broke18:46
mgariepythis issue  i was able to reproduce on my machine every time but it seems that it's not 100% fixed.18:47
opendevreviewMerged opendev/system-config master: Run gerritbot with a user that will be shared with matrix-gerritbot
fungilooks like opendevreview exited around 25 minutes ago, i'll check to see if it started again20:00
clarkbfungi: hrm the job hasn't run for it yet20:01
clarkboh! I think the periodic job managed to get it20:01
clarkbfungi: docker says it is up and running20:01
fungiit definitely restarted and is using the correct account, looks like20:02
clarkbya I think what happened was we merged the change when the hourly run was happening but before the hourly run of eavesdrop20:02
clarkbthen the hourly run of eavesdrop updates from master because it is a peridoic job and it did the update then20:02
fungimakes sense. that's fine, i guess20:02
clarkbso the jobs for the change landing should noop now20:02
clarkband ya from what I see on the server it seems to be working if we can confirm it emits events to us here we should be good20:02
fungii don't have anything i need to push just this moment, but i'll check other channels to see if it already did so20:03
opendevreviewClark Boylan proposed opendev/bindep master: Try out PBR pep 517 support
fungiyeah, it's working. it announced 818458 in the ironic channel moments agio20:04
fungior that works too20:04
clarkbI needed to rever back to the backward compatible version of that change now that I had tested the forward looking version20:04
clarkbcool seems happy20:04
clarkbBefore we do the matrix-gerritbot I need to followup on how it does known_hosts too20:04
clarkbI can look at that after lunch I guess20:04
fungii was just looking through that pep 517 test change20:04
clarkbfungi: basically I think that change is mergeable again should we want to go that route and we might see interesting thigns about whether or not setup_requires are ignored in that situation20:07
fungigot sidetracked, apparently filelock released a new version on tuesday deprecating its poll_intervall parameter in favor of poll_interval and tox is just blowing up everywhere20:13
fungiyeah, looks like it was
fungiand, wow, it didn't actually correct its internal use of the name20:20
fungiahh, it kept the old name around and separate rather than as an alias for the new one20:22
clarkbis this affecting our CI jobs?20:23
fungino, because we don't run with pythonwarnings=error20:24
fungilooks like it may be virtualenv using the old parameter name still, tox is rather opaque about failures from virtualenv invocation, and swallows tracebacks entirely for some reason20:27
clarkbah you are trying to check that you can install stuff with pbr without that warning20:27
fungiyeah, there it is, in src/virtualenv/util/lock.py20:27
fungiright, i was trying to see if i could test with one of my pbr-using projects where i set tox to fail on deprecation warnings (or any warnings) unless i've expressly filtered them20:28
fungiit does appear latest master branch state for virtualenv still uses the deprecated poll_intervall when calling filelock20:29
fungii'll get an issue filed20:30
clarkbcan you force downgrade filelock?20:33
fungiyeah, just wanted to get the issue filed20:41
fungilooks like virtualenv has a --with-traceback option, for future reference20:42
fungifiled as
clarkbfungi: I'm wondering if we should start filing bugs against tox to stop using setup.py20:45
clarkbor at least make it an option to use something like build20:46
clarkbthe hacked up bindep tox.ini I wrote is basically a workaround for that20:46
fungiquite possibly20:47
fungiat least that one wasn't hard to add to my pythonwarnings ignore list20:51
fungitoo often i find messages i can't filter directly because they include "," (the pythonwarnings record separator) or ":" (the pythonwarnings field separator) directly in the exception text20:52
fungiunfortunately, testing these pbr changes without having them in a release is nontrivial, since we can't actually exercise whether pip properly fetches pbr as a build backend20:55
clarkbfungi: they are all in a pbr release now20:55
clarkbthat is why I picked this up again with bindep20:55
fungioh! i missed that20:55
fungiokay, so that's the pbr>=5.8.020:56
clarkbre matrix-gerritbot we supply known hosts to it so this shouldn't be a problem there. fungi  if you can review sometime before tomorrow I'm happy to try and land that one tomorrow next20:57
fungiokay, with that new filelock warning filtered, all my tox envs run clean on the project where i'm trying to replicate this20:58
clarkbI feel like t oday is an early stop day due to being up super early again. Better to do production changes tomorrow morning20:58
funginow to try replacing its with a pyproject.toml20:58
clarkbfungi: note you still need the setup.py20:58
clarkbas the setuptools build meta stuff executes setup.py20:58
fungier, right, just not the setup_requires in it20:58
clarkbyup that20:58
fungimmm, "UserWarning: Unknown distribution option: 'pbr'"21:05
clarkbthat is created via an entrypoint in pbr21:06
clarkbsetuptools loads options based on its entrpoints. So a pbr entrpoint becomes a pbr option. Then you can pass values to it and they are passed to the called entrypoint21:07
clarkblearning this was a huge part of figuring out why pbr wasn't working without setup_requires before21:07
clarkbas I had to braek the recursive calls to pbr() by updating that option after the first call to be None21:07
fungilooks like it's something calling `python egg_info`21:07
clarkbah yup that would do it if you have removed setup_requires21:08
fungibut i'm not clear on exactly what is calling it yet21:08
fungiside note, i'm not yet sure how i feel about the underlining in python 3.11 tracebacks21:09
clarkbfungi: in my setup_requires patchset I had to remove usedevelop=True beacuse that does call and instead put . in the deps list21:10
clarkbits this sort of thing that tox would ideally be updated to handle so that tox can be used without direct calls21:10
fungiahh, yeah, i was trying with usedevelop still on21:11
fungii'll try without it, that's probably the reason21:11
fungii do my deps in setup.cfg (options.install_requires) though, so i'll have to see if a similar solution works there21:13
clarkbfungi: I don't think that is what you want21:14
clarkbthis isn't about deps for your installation but about tox installing your software to the venv21:14
fungier, yeah, sorry i do manage deps in the tox testenvs too21:15
clarkbbindep's tests and doc builds run bindep commands so we need to install it21:15
clarkbotherwise it would be fine with just having the requirements and test-requirements in place and running the python directly21:15
fungideps=. doesn't seem to address it though, the same UserWarning is raised when it tries to `python -m pip install .`21:17
clarkbhuh pip isn't supposed to run I didn't think21:18
clarkbit should see your pyproject.toml and use that instead21:19
fungiooh, i wonder if this is because pbr is an install_requires for bindep so is being preinstalled21:19
funginope, adding it doesn't seem to solve that21:21
clarkbfungi: when I run `tox -vv -repy38` for the latest bindep patchset for testing it logs 'Installing build dependencies ... done' before installing requirements21:23
clarkbI believe that while pbr is in install_requires it isn't preventing the pyproject.toml config from taking precedence and running first21:23
clarkbhowever it doesn't log which build dependencies it has installed there21:23
opendevreviewMerged opendev/base-jobs master: infra-prod: override checkout on periodic/hourly
clarkbianw: good morning :)21:26
fungioh, okay i'm clearly testing too late in the day. i did a git clean when starting my tests, without git adding the pyproject.toml *sigh*21:27
clarkbthe gerritbot change went well. I'm about to eat lunch then I'll need to do a school run but I can push up a change to the dockerfile next as i think this shows my assumptions were correct. As long as you've got the file readable and executable for the random new user its fine21:27
clarkbit would be a bigger concern if we had to write stuff, but we don't in this case21:27
ianwclarkb: good afternoon :)21:28
fungiclarkb: okay, much better results when i don't blindly blow away the build backend definition before testing it!21:30
fungirerunning all my tox envs to get a new baseline with the pyproject.toml added and the setup_requires parameter removed21:31
fungithen i'll see what warnings i can remove from my filter21:32
Clark[m]fungi: fwiw I think you can leave setup_requires there for backward compat but it is easier to see things are working as intended by removing it temporarily first21:35
fungiconfirmed latest pbr lets me replace the setup_requires with a build-backend in pyproject.toml, and works as expected with all my various tox envs. after dinner i'll play around with seeing what warnings that obsoletes21:46
ianwinfra-root: i don't think my dstat_graph pull requests are going anywhere, i've proposed importing the project with
opendevreviewClark Boylan proposed opendev/gerritbot master: Update docker image to bullseye
opendevreviewClark Boylan proposed opendev/gerritbot master: Update the docker image to run as uid 11000
clarkbianw: ^ there is the promised image update as well as a bump up to bullseye while we are at it23:25
fungii suppose we have a bunch of images which need to switch to bullseye23:27
clarkbyes, I/we have been picking them off slowly over the last few weeks. but there are always more :)23:27
clarkbgerrit, zuul, nodepool, gitea are all done23:27

Generated by 2.17.2 by Marius Gedminas - find it at!