Thursday, 2022-04-14

*** rlandy is now known as rlandy|out00:52
opendevreviewMichael Johnson proposed openstack/diskimage-builder master: Mark our source trees as safe for git to use as other users
opendevreviewMatthew Thode proposed openstack/project-config master: update generate constraints to py38,39,310
*** pojadhav|afk is now known as pojadhav03:56
opendevreviewIan Wienand proposed openstack/diskimage-builder master: source-repositories : use explicit sudo/-C args when in REPO_DEST
opendevreviewchandan kumar proposed zuul/zuul-jobs master: Setup RDO gpg keys and repo for CentOS distros only
*** rlandy|out is now known as rlandy10:31
*** pojadhav is now known as pojadhav|afk11:00
*** dviroel|afk is now known as dviroel11:24
hrwcan zuul log pages get lighter?11:58
*** pojadhav|afk is now known as pojadhav11:58
hrwwhenever I go to pages like I hear how fans of my ryzen box start to spin up and firefox holds everything trying to deliver page11:59
hrwand switching from 'task summary' to 'logs' takes 10+ seconds before page is clickable again12:00
hrw29MB job-output.json12:03
fungihrw: yes, i think it's mainly a problem with very large job logs like that12:15
fungiit's worth discussing possible solutions in the zuul matrix channel12:15
hrwfungi: and debugging why it does not fetch job-output.json.gz12:16
fungiwell, job-output.json in our deployment is stored in swift, so your browser theoretically transparently negotiates compression with the server when requesting12:17
fungipre-compressing would break that12:17
fungialso i seriously doubt it's taking 10 seconds to fetch 29mb of data, it's the browser-side processing of that data which takes time12:18
hrwand then browser says 12:18
*** dviroel is now known as dviroel|brb12:19
fungiit's possible smarter indexing/summaries could be prepended to the json allowing more rapid rendering of the high-level details and then only spending time parsing the larger data set when specific operations request it (like expanding a playbook in the console view or something)12:20
fungifor that matter, something more drastic like serialized sqlite might allow faster indexing into specific slices of the data without having to parse the entire set every time12:21
fungii don't personally know what kinds of efficiency magic might be available since i'm not that well-versed in database theory12:21
hrwneither do I12:22
hrwwill dig out my matrix account12:22
fungiit's really fairly quick when you don't have a large json blob:
hrwtoo bad that I do 99% of work in kolla(-ansible) with insane logs12:24
hrwI may check is my logging ideas possible to implement again.12:25
fungithe job-output.json in that example is "only" 612kb (9633 lines)12:26
hrwso console output is show and all goes into files (instead of console+files like it is now)12:26
fungiclarkb: i've approved 827774 and will keep an eye on it12:47
fungialso, you determined that jeepyb/manage-projects probably came through unscathed, right?12:48
*** dviroel|brb is now known as dviroel12:51
*** artom_ is now known as artom13:29
hrwok. may make kolla logs shorter 13:37
jrosserfwiw for gigantic log files i get on fine with the 'raw' link in my browser but the pretty rendition that zuul makes bogs down quite badly13:41
opendevreviewMerged opendev/system-config master: Force case sensitive Gerrit users in config
fungiseems to have deployed successfully14:12
Clark[m]fungi 827774 probably deserves a Gerrit restart before the renaming just to be sure it acts as expected? I think 3.4 should ignore the setting and it only takes effect in 3.5. basically double check that noops.14:13
Clark[m]fungi: for jeepyb I pushed up a noop change to have it run it's tests and those passed as well14:13
fungiyeah, i figured 827774 wouldn't take effect yet, just wanted to be sure it would deplpy the config file14:15
fungii guess we can approve pending project creation changes. i'll do one that's ready and make sure it works as intended before we do any more14:15
dtantsurfolks, do you have plans on making {{ ansible_user_dir }}/src a safe directory for git or should we work around for our jobs?14:15
fungidtantsur: it's a safe directory for the zuul which jobs run as14:16
fungiare you performing git operations there as a different user, maybe via sudo?14:16
Clark[m]hrw jrosser for really large log files I have to use vim. There are definitely tiers of problem and the size of the data makes each tier progressively worse. Logging effectively is something that a lot of jobs struggle with imo. For example printing every passing test case to stdout is extremely noisy. A lot of openstack projects go even worse and print the warnings their tests generate to stdout too.14:16
dtantsurhmm, so it's not the same failure?
fungii meant to say "for the zuul user which jobs run as"14:17
dtantsurfungi: yeah, it has a become. hmm. maybe we can use `pip install --local`...14:17
hrwClark[m]: for very large logs I just run build locally ;D14:17
dtantsurthe affected task:
hrw is much more quiet kolla run14:18
fungidtantsur: if you're going to `sudo pip install .` for a git repo with a pbr-using project, then yes you'll need to do something to address that. you can set the dir as safe for the root account to use like the workaround which merged to devstack yesterday, or you can separate wheel building and installing if you don't need editable mode14:18
Clark[m]Or use a virtualenv14:19
dtantsurwe need an urgent fix for our CI, I don't feel like changing the way devstack works there right away14:19
fungipbr doesn't need to run git as root, so you can build wheels as the zuul user and install them with any account14:19
* dtantsur wonders if ansible has something for pip wheel14:20
fungidtantsur: devstack has a workaround merged already14:20
Clark[m]Changing the way devstack works is what devstack has already done14:20
dtantsurClark[m]: you mean, it defaults to virtualenvs?14:20
hrwdtantsur: you mean?14:20
dtantsurhrw: I'm looking for the smallest change I can make to to unbreak our CI14:20
dtantsurso far I'm inclined to go down the git config path and figure out later14:21
fungidtantsur: devstack sets project source trees it installs as safe for the root user to run git in14:21
Clark[m]No devstack sets the safe directory setting for each repo it clones14:21
hrwdtantsur: behave and use virtualenv?14:21
dtantsurhrw: *SMALLEST*14:22
hrwdtantsur: chown root:root whole git repo14:22
*** pojadhav is now known as pojadhav|afk14:22
fungidtantsur: is what merged to devstack as a workaround14:22
hrwdtantsur: so ansible runs pip as root, pbr sees git code as own and does the job14:22
dtantsuryep, I'm looking at it. I guess I'll do the same for the cloned dir.14:22
fungiit's a one-liner14:23
fungiso i'm not sure how much smaller you want ;)14:23
fungidtantsur: if you want to go the build/install split route instead, i have a working poc at but it's not as minimal as what we merged14:24
*** chkumar|ruck is now known as raukadah14:31
hrwnow is mostly base-jobs output rather than kolla one ;D14:36
hrwand job-output.json is 5MB instead of 29M14:37
Clark[m]hrw: I don't think it is bad to have output, more concern that we over do it. For me at least the top level logs should identify high level actions and errors with their details when they occur14:38
hrwClark[m]: we have all that output in separate log files14:38
funginoting that a job consisting of lots of small ansible tasks also bloats the json blob14:38
hrwClark[m]: and when firefox get 29MB of json to parse my ryzen system starts to levitate on air flowing from case fans14:39
hrwClark[m]: and those separate log files are *separate* so they are readable. contrary to basic output where everything is mixed14:40
hrwanyway waiting for other kolla cores to review14:40
Clark[m]Sure. I just don't want people to feel they have to delete logging. More I want to impress that it is about logging what is useful14:41
Clark[m]Typically a side effect of this is logs that are manageable.14:42
hrwClark[m]: sure. logs are useful. We just generated tons of them and not used them.14:43
opendevreviewMerged openstack/project-config master: Add Ganesha based Ceph NFS Charm
*** dviroel is now known as dviroel|lunch15:12
prometheanfiredo we have a image I can use to run generate-constraints that has py38,39,310?15:49
clarkbprometheanfire: rumor is focal might get a 3.10 package and then it could do that, but I am not aware of one that has all three currently15:51
fungithat sounds suspiciously like a question i raised on a review recently ;)15:51
* fungi looks up what's available15:51
clarkbeach of the centos' are single python3. Focal has 3.8 and 3.9. Debian bullseye has 3.9 and 3.1015:52
clarkboh wait no bullseye is just 3.915:52
*** marios is now known as marios|out15:52
fungi says not for focal yet15:52
clarkbI guess 3.10 comes out of unstable or testing or something15:52
prometheanfireugh, am I right that zed will support those three python versions officially?  that's all I'm looking to change to15:53
fungiyeah, 3.10 is in debian testing and unstable:
clarkbprometheanfire: I think that is the plan. But also I wouldn't be surprised if python3.6 panic occurs again15:53
prometheanfireI really want to remove 36 :P, if it's just 38/39 I'd be happy too15:54
fungi3.8 and 3.915:54
prometheanfireoh, that is easier then15:54
clarkbsort of15:54
clarkbthe intent is to have 3.10 testing15:54
clarkbso I guess it isn't official support but requirements likely needs to handle it anwyay15:55
fungithe "3.6 panic" is that there is still no rhel 9 (as far as i'm aware anyway), so latest rhel is still py3615:55
prometheanfireif I had more time to get a gentoo image working right we would still have to remove 36 lol15:55
fungiwhich means we may release a zed which can't run on rhel servers15:55
prometheanfirewe'd have 27 available though15:55
clarkbfungi: ya I feel like the current defintions for support are a bit flawed and openstack should target rocky to address that instead of centos15:55
clarkbbasicaly the idea originally was always to support rhel via a stand in15:56
clarkbwe've broken that itnent now because raisins I guess15:56
fungibut really, these are topics for #openstack-tc not #opendev, since the people who need to make decisions about that sort of thing pay more attention there15:56
prometheanfirerequirements was originally just for gate usage, but it's always morphed15:57
prometheanfiresure, joining15:57
clarkbbut to be clear I am not aware of any distro image we have that can do all three out of the box15:57
clarkbyou can do 3.8 + 3.9 or 3.9 + 3.1015:57
prometheanfirewell, we can start with what we officially support15:57
opendevreviewMatthew Thode proposed openstack/project-config master: update generate constraints to py38,39
fungiin hindsight, the py3k transition creating a combined constraints file was probably a poor choice. this would be a lot easier if we had pyver-specific constraints files and just applied the appropriate one for the interpreter version being used16:03
fungithen we could generate them independently in separate jobs16:03
fungiwe can still do that and then have a subsequent job which mashes them together, i guess16:03
clarkbI think sort -u can mash them together proeprly16:04
clarkbyou just need to remove duplicates16:04
fungiwell, i imagine you'd set pyver-specific envmarks on every line indicating the version for which that constraint was generated, and then just trivially concatenate16:06
fungino need to get complicated with it16:06
prometheanfiresuper ugly, but'd work lol16:06
clarkboh I thought the generation tooling already added the pyver envmarks when necessary16:07
clarkbso you get a subset rather than needing to expand it out. But ya expanding verbosely would work fine too16:07
prometheanfirewhen necessary, but not always16:07
fungiyeah, but if you do each one in a separate job you no longer know whether it's "necessary"16:07
fungithough you could postprocess the concatenated aggregate and collapse envmarks together16:07
fungifunctionally identical, just shortens the file16:08
clarkbisn't that independent of where the job runs? it looks at the pypi data I would've thought? Or is it diffing the pip list output?16:08
clarkbbut ya many ways to address this and not all of them complicated16:08
fungiit's not as simple as whether the package specifies a requires_python, you may need different versions on different interpreters because of some transitive dependency relationship16:09
*** dviroel|lunch is now known as dviroel16:20
*** rlandy is now known as rlandy|brb16:26
hrwprometheanfire: use manylinux2014 container?16:28
johnsomCan someone review and hopefully +W this DIB patch that fixes the git issue for Octavia image building?
johnsomSuccessful test patch here:
clarkbdoes ^ mean opendev has been unable to build iamges? I guess we'd need bullseye to update then land a nodepool change to pick up the git update?16:38
johnsomThe issue we hit was using the source-repositories element to get some git based packages. If the nodepool images aren't installing packages that way they were probably fine.16:40
clarkbjohnsom: we use source-repositories to cache the repos. But we do not install them from there16:40
clarkbjohnsom: ok I think I found a bug16:47
johnsomclarkb I think Ian is out for a few days, so we are probably on our own on this patch.16:50
clarkbI too am only about to be half here. But if people agree with my -1 pushing up a new patchset is probably a good idea16:50
johnsomYeah, I see it too. I will push an update and run the test patch again.16:51
opendevreviewMichael Johnson proposed openstack/diskimage-builder master: source-repositories : use explicit sudo/-C args when in REPO_DEST
*** rlandy|brb is now known as rlandy16:53
clarkbfungi: fyi!/story/200999317:05
johnsomclarkb The test patch successfully built the image with that pushd removed.17:23
clarkbjohnsom: and you verified the repo contents in the resulting image?17:23
johnsomha, well, give the test job a few more minutes. 17:24
clarkbok I +2'd it but  Ithink it would be good to check the image contents before approving (the dib CI will take a bit of time nayway)17:26
johnsomThere are a lot of different paths in that element. I'm not sure we even use that other path.17:26
clarkbI need to step away for a it but will check back in later. Also fungi can +A it if he feels comfortable17:26
clarkbjohnsom: ya thats possible, but just confirming as much as we can without a ton of extra effort is good17:26
fungii'm happy to approve it now. i need to run a quick errand, but it won't merge right away17:29
johnsomYeah, the image is good. The tempest tests are passing.17:30
fungistepping away for just a few, back shortly17:50
fungiand back, looks like it hasn't merged yet18:36
clarkbwe'll also likely need to do a release18:53
clarkbbut depending on how octavia is consuming dib that may not be as urgent18:53
clarkb I'm not sure that includes the git fix that would affect our builders18:54
clarkb I don't think so. That means we have some time on our end18:55
johnsomIt's in related projects, but yes, I suspect we will need a release18:55
opendevreviewMerged openstack/diskimage-builder master: source-repositories : use explicit sudo/-C args when in REPO_DEST
fungithere we go19:53
opendevreviewDavid Wilde proposed opendev/git-review master: Bypass Java >= 17 strong encapsulation for tests
opendevreviewDavid Wilde proposed opendev/git-review master: Bypass Java >= 17 strong encapsulation for tests
clarkbfungi: I'm going to start being here less as I have to help with the kids, but re project renames I think we need to review the openstack/project-config change(s) for the rename, push a opendev/project-config yaml file for the rename (this is the input to the rename playbook), then write down a plan to make sure we aren't missing anything?20:21
fungithat's pretty much it. i was planning to hack on it after dinner20:28
Clark[m]Cool I'll try to carve out time later to pull up things to review20:37
fungiyeah, i'll probably be free in ~ an hour20:38
clarkbquickly checking the dib release situation I think we could do a 3.20.3 release with only that last update in it. The previous commit was 3.20.220:59
opendevreviewDavid Wilde proposed opendev/git-review master: Bypass Java >= 17 strong encapsulation for tests
*** dviroel is now known as dviroel|out21:24
fungiclarkb: agreed, that seems prudent22:33
fungithe fix is pretty minimal22:34
fungistill just the one rename for tomorrow:
fungisingle repo changing namespace, no group renames included22:36
fungii'm mildly dubious about switching requireContributorAgreement to true, but will remind the tc that they need to double-check the status of all the prior contributors to that repo22:36
fungii don't think it should block the rename anyway22:37
fungii'll +2 and push up the metadata change22:37
opendevreviewJeremy Stanley proposed opendev/project-config master: Record project renames happening April 15, 2022
fungiclarkb: ^22:40
clarkbfungi: cool that change and hte actual rename change both lgtm22:44
clarkband since there is only a single rename we don't have to worry about order of operations between different renames or getting them in in a short period of time22:44
fungiyeah, i plan to just do a status notice as a heads up at 14z and then another at 15z when i start the outage, but basically just follow
fungiwe don't have any outstanding caveats we need to update the docs there for as far as you're aware, right?22:47
Clark[m]Did we manage to get all the updates into that doc after the last rename?22:47
Clark[m]Let's see if we can find the etherpad for it to compare22:48
Clark[m]fungi: ^ if you need to compare. That also had a Gerrit server upgrade at the end which made it a little different but I think you can largely ignore that22:51
fungiyeah, the only slightly out of the ordinary thing we have going on this time is that the updated gerrit config will take effect at restart to do nothing actually new because it just adds an option our gerrit version should ignore22:53
johnsomIt looks like diskimage-builder isn't release managed by the release team. What is the process to get that git fix released?23:16
Clark[m]johnsom: someone in the release group has to push a signed tag23:19
Clark[m]This is intentional to avoid chicken and egg problems between the release system and disk image builder releases23:20
johnsomSo pretty much infra. Would someone have a moment to hit the button on that? The Octavia grenade jobs use the released version.23:26
fungijohnsom: if i wait 30 minutes can i call it a friday release? ;)23:30
johnsomIt's a bit longer until Friday here.23:31
fungiwell, i mean utc friday. real friday ;)23:31
fungiClark[m]: it looks like ianw inadvertently tagged not-head for 3.20.223:32
fungithere's a merge commit which landed after it for a commit which is before it23:32
fungiso i think 3.20.3 is actually also going to include Update fedora element testing to F3523:34
fungii guess it's not a functional change for the software though, so doesn't really alter plans23:34
fungicommit 4cb3346fec6715d3f2ba7953face5296f497956a (HEAD, tag: 3.20.3, origin/master, origin/HEAD, gerrit/master, refs/changes/24/837824/2)23:37
fungithat look right?23:37
fungiif so, i'll heave23:37
johnsomIt's the right sha23:38
*** rlandy is now known as rlandy|out23:48
Clark[m]fungi: oh interesting but I agree a test only change isn't a big deal23:50
Clark[m]And ya that Sha looks right23:51
fungicool, pushing thanks!23:52
fungiand it's up23:52
fungirelease-openstack-python node request is taking longer than expected23:57
johnsomThank you!23:57

Generated by 2.17.3 by Marius Gedminas - find it at!