| *** elodilles is now known as elodilles_pto | 07:04 | |
| opendevreview | Merged openstack/os-brick stable/wallaby: NVMe-oF: Return right nqn when missing hostnqn https://review.opendev.org/c/openstack/os-brick/+/834604 | 09:46 |
|---|---|---|
| dc_arunkv | Hi Core Reviewers, can someone please review the patch: https://review.opendev.org/c/openstack/cinder/+/836996 | 09:49 |
| opendevreview | Merged openstack/os-brick stable/yoga: Fix encryption symlink issues https://review.opendev.org/c/openstack/os-brick/+/845845 | 10:43 |
| opendevreview | Tushar Trambak Gite proposed openstack/cinder master: Include volume type constraints in internal API https://review.opendev.org/c/openstack/cinder/+/846146 | 11:01 |
| opendevreview | Tushar Trambak Gite proposed openstack/cinder master: Include volume type constraints in internal API https://review.opendev.org/c/openstack/cinder/+/846146 | 11:04 |
| *** dviroel|afk is now known as dviroel | 11:18 | |
| opendevreview | Gorka Eguileor proposed openstack/cinder master: DNM: Testing devstack memory changes https://review.opendev.org/c/openstack/cinder/+/846149 | 11:41 |
| opendevreview | Gorka Eguileor proposed openstack/cinder master: DNM: Testing devstack memory changes https://review.opendev.org/c/openstack/cinder/+/846149 | 11:45 |
| HappyStacker1 | hello core team ! | 12:21 |
| HappyStacker1 | can someone review https://review.opendev.org/c/openstack/cinder/+/821749 ? | 12:21 |
| HappyStacker1 | We need +1 workflow | 12:21 |
| opendevreview | Brian Rosmaita proposed openstack/cinderlib master: Use stable/yoga override for CI jobs (conclusion) https://review.opendev.org/c/openstack/cinderlib/+/845272 | 12:43 |
| rosmaita | tosky: thanks for the review ... see if ^^ addresses your concerns | 12:43 |
| tosky | rosmaita: yes, thanks, but is that new file needed? It wasn't there in the first revision | 13:27 |
| rosmaita | oh crap | 13:27 |
| rosmaita | no, it shouldn't be there ... i will remove it after the os-brick-src job finishes | 13:28 |
| opendevreview | Stephen Finucane proposed openstack/cinder master: tests: Enable SQLAlchemy 2.0 deprecation warnings https://review.opendev.org/c/openstack/cinder/+/837163 | 13:54 |
| opendevreview | Stephen Finucane proposed openstack/cinder master: models: Remove implicit coercion of SELECT to scalar subquery https://review.opendev.org/c/openstack/cinder/+/837164 | 13:54 |
| opendevreview | Stephen Finucane proposed openstack/cinder master: db: Pass case.whens as positionals, not a list https://review.opendev.org/c/openstack/cinder/+/837165 | 13:54 |
| opendevreview | Stephen Finucane proposed openstack/cinder master: Resolve invalid UUID warning https://review.opendev.org/c/openstack/cinder/+/837353 | 13:54 |
| opendevreview | Stephen Finucane proposed openstack/cinder master: db: Don't pass strings to Connection.execute https://review.opendev.org/c/openstack/cinder/+/837166 | 13:54 |
| opendevreview | Stephen Finucane proposed openstack/cinder master: db: Don't use strings to indicate column, relationship paths https://review.opendev.org/c/openstack/cinder/+/837167 | 13:54 |
| opendevreview | Stephen Finucane proposed openstack/cinder master: db: Don't use legacy calling style of select() https://review.opendev.org/c/openstack/cinder/+/837354 | 13:54 |
| opendevreview | Stephen Finucane proposed openstack/cinder master: db: Don't use strings to indicate relationship names https://review.opendev.org/c/openstack/cinder/+/837355 | 13:54 |
| opendevreview | Stephen Finucane proposed openstack/cinder master: db: Remove use of 'as_scalar()' https://review.opendev.org/c/openstack/cinder/+/837356 | 13:54 |
| opendevreview | Stephen Finucane proposed openstack/cinder master: db: Remove resolved TODO https://review.opendev.org/c/openstack/cinder/+/837546 | 13:54 |
| opendevreview | Stephen Finucane proposed openstack/cinder master: db: Fix up some API signatures, other style issues https://review.opendev.org/c/openstack/cinder/+/837542 | 13:54 |
| opendevreview | Stephen Finucane proposed openstack/cinder master: db: Move helpers to cinder.db.sqlalchemy.api https://review.opendev.org/c/openstack/cinder/+/837543 | 13:54 |
| opendevreview | Stephen Finucane proposed openstack/cinder master: db: Remove unnecessary sqlalchemy abstraction https://review.opendev.org/c/openstack/cinder/+/813229 | 13:54 |
| opendevreview | Stephen Finucane proposed openstack/cinder master: tests: Fix invalid test https://review.opendev.org/c/openstack/cinder/+/846171 | 13:54 |
| opendevreview | Stephen Finucane proposed openstack/cinder master: tests: Address UserWarning in tests https://review.opendev.org/c/openstack/cinder/+/846172 | 13:54 |
| opendevreview | Stephen Finucane proposed openstack/cinder master: db: Use cinder.db.api directly https://review.opendev.org/c/openstack/cinder/+/846173 | 13:54 |
| whoami-rajat | and i thought we finished reviewing DB patches | 14:06 |
| geguileo | dansmith: you where saying that 900MB for n-cpu was a lot... Here you can see 2.2GB (without the changes I'm proposing) https://f6314bbe689272b182bf-704d2e5cde896695f5c12544f01f1d12.ssl.cf1.rackcdn.com/845806/2/check/tempest-slow-py3/363a876/compute1/logs/performance.json | 15:25 |
| geguileo | I really don't know what's going on there... | 15:25 |
| dansmith | geguileo: hmm, yeah I'm quite sure that's abnormal, so maybe something else happened | 15:51 |
| opendevreview | Brian Rosmaita proposed openstack/cinderlib master: Use stable/yoga override for CI jobs (conclusion) https://review.opendev.org/c/openstack/cinderlib/+/845272 | 15:51 |
| geguileo | dansmith: I'm seeing other CI jobs (multi-node) with 1.8GB, 1.74GB on the compute node | 15:52 |
| dansmith | ack, maybe a recentish merge caused some regression there and only you noticed it because you were looking at memory | 15:52 |
| dansmith | I'll poke a bit | 15:52 |
| geguileo | dansmith: Since I can't tell if the changes that fix c-bak help the other components I will change my devstack and tripleo patches to only change cinder-volume and cinder-backup services | 15:53 |
| geguileo | because I don't have time to dig more into it now :'-( | 15:53 |
| rosmaita | geguileo: that sounds like a good compromise | 15:53 |
| dansmith | geguileo: well, if you're bring it down from 1G+ to 900M, you're probably not making it worse, but yeah that seems like a reasonable thing to do | 15:54 |
| geguileo | dansmith: well, the thing is that I'm not sure if it's bringing it down for nova or not... | 15:54 |
| geguileo | dansmith: because that one consume 2.2GB without the devstack changes, but then I created a patch with a depends on the devstack patch and it consumed even more on that specific run... | 15:55 |
| geguileo | so I'd need to run things locally to be able to do proper test runs | 15:55 |
| geguileo | too much difference between runs to know if this is actually improving or not | 15:55 |
| dansmith | oh, I thought you said it went from 2.2G plain to 900M with your changes | 15:55 |
| dansmith | but ack | 15:56 |
| geguileo | dansmith: for backup it goes down on the CI from 700-900MB down to 140-200MB | 15:56 |
| geguileo | it's n-cpu that I can't tell if it helps or not | 15:56 |
| geguileo | too much difference in nova from runs | 15:57 |
| geguileo | this patch https://review.opendev.org/c/openstack/devstack/+/814891 | 15:57 |
| dansmith | yeah let me try to query out recent trends and see | 15:57 |
| geguileo | has on tempest-multinode-full-py3 one run with 1.74 GB, another with 970.89 MB, another with 1.80 GB... | 15:58 |
| geguileo | thanks | 15:58 |
| stephenfin | whoami-rajat: Almost :) | 15:58 |
| stephenfin | whoami-rajat: That sqlalchemy 2.0 series has been around for a while, but I wanted to wait until the other series was (mostly) done before rebasing and proceeding with it. You're going to need that before sqlalchemy 2.0 is released in the next month or two | 15:59 |
| stephenfin | blame zzzeek | 15:59 |
| dansmith | geguileo: okay maybe I'm lying by some amount | 16:01 |
| dansmith | one of my stored json files from several weeks ago has: | 16:02 |
| dansmith | | 540 | devstack@n-cpu.service | | 16:02 |
| dansmith | that's more than I was expecting, but pretty far off from 2G | 16:02 |
| geguileo | dansmith: was it a multinode one? | 16:02 |
| dansmith | no, that was tempest-integrated-compute | 16:02 |
| geguileo | because I've noticed that on the multinode ones the n-cpu on the compute is the one that goes crazy with RAM | 16:02 |
| dansmith | but I only have logs for the primary node on any multinode ones | 16:02 |
| geguileo | ok, those probably consume less memory | 16:03 |
| dansmith | ahh | 16:03 |
| dansmith | maybe a live migration thing then | 16:03 |
| dansmith | geguileo: aha, yeah: | 16:03 |
| dansmith | | 1563 | devstack@n-cpu.service | | 16:03 |
| dansmith | wowsers | 16:03 |
| geguileo | for the controller nodes consumption is lower: between 333 MB and 450 MB | 16:03 |
| geguileo | dansmith: yeah, that's what I'm seeing as well | 16:04 |
| dansmith | geguileo: this is why that other grok patch I sent captures point-in-time json files so we have something to compare $today to | 16:04 |
| dansmith | we thought maybe reset those at the beginning of every cycle or something, so we have some measurement of what might have changed | 16:04 |
| geguileo | dansmith: it would also be good to have the initial ones of previous releases | 16:05 |
| geguileo | to see long term trends | 16:05 |
| geguileo | but first we need to understand what's going on... | 16:05 |
| dansmith | sure, once it's in git you can always go pull that | 16:05 |
| geguileo | I think that 1.xGB thing on computes may be a matter of memory pressure being low so memory is not being released to the system because the system doesn't need it | 16:06 |
| geguileo | but it's just a wild, wild guess | 16:06 |
| dansmith | not sure what you mean.. python doesn't know whether the system needs memory in order to deflate its heap | 16:06 |
| dansmith | or are you thinking memorycurrent is rss and is shows less when part of us is swapped out? | 16:07 |
| geguileo | iiuc we have 2 memory management systems in place here | 16:08 |
| geguileo | Python memory management with its arenas, pools, and blocks | 16:08 |
| geguileo | and then glibc arenas | 16:08 |
| geguileo | and even if python releases memory it can happen that glibc does not | 16:09 |
| geguileo | it has its own algorithm to decide when to give it back to the system | 16:09 |
| dansmith | yeah, but I thought python basically never releases it fully to the system | 16:09 |
| geguileo | that can be looked into | 16:10 |
| geguileo | there is a way to ask python to give you the arenas/pool/block allocations | 16:10 |
| geguileo | but it needs to be run within python | 16:10 |
| geguileo | I did it when I was investigating the backup service | 16:11 |
| dansmith | ack, like guru meditation on USR1 | 16:11 |
| geguileo | yeah, it could be a guru meditation plugin | 16:11 |
| dansmith | man I can't find any definition of memorycurrent, but I was assuming it was virt and not rss | 16:12 |
| dansmith | because I think all the values in our list likely add up to more than 8G, but maybe not | 16:12 |
| geguileo | lol | 16:12 |
| geguileo | they don't add up more | 16:13 |
| geguileo | it's RSS | 16:13 |
| geguileo | I confirmed it locally | 16:13 |
| geguileo | it's not VSZ | 16:13 |
| geguileo | and I made a little change to your script to show me the total, and it's never more than 8GB | 16:14 |
| dansmith | hah, I just made that | 16:14 |
| dansmith | I also added cases for db and api, so I should post that (or propose to devstack/tools) | 16:14 |
| dansmith | that multinode one I was looking at is 6358M total | 16:14 |
| dansmith | so yeah okay | 16:15 |
| dansmith | so sean had noticed some differing behavior based on swap being configured or not, | 16:15 |
| dansmith | which I dismissed because I thought that was virt, but if it's rss, I guess that goes up as competition for swap increases | 16:15 |
| geguileo | oooooh, swap could indicate memory pressure related thingy | 16:16 |
| geguileo | memory pressure behaves differently when there is no swap to when there is swap (even if it's a 10MB swap) | 16:16 |
| dansmith | yeah | 16:16 |
| dansmith | geguileo: https://review.opendev.org/c/openstack/devstack/+/846198 | 16:19 |
| dansmith | geguileo: can we continue this in -qa? I'm also talking to sean in -nova and it'd be easier to find a common spot :) | 16:19 |
| geguileo | dansmith: joining | 16:20 |
| opendevreview | Jean Pierre Roquesalane proposed openstack/cinder master: PowerMax Driver - Manage volume into correct storage group https://review.opendev.org/c/openstack/cinder/+/797333 | 18:58 |
| opendevreview | Jean Pierre Roquesalane proposed openstack/cinder master: Add NVMe/TCP support to Dell EMC PowerStore driver https://review.opendev.org/c/openstack/cinder/+/819149 | 19:04 |
| HappyStacker1 | team, can someone look at https://review.opendev.org/c/openstack/cinder/+/844563 | 19:15 |
| *** dviroel is now known as dviroel|biab | 21:07 | |
| opendevreview | Brian Rosmaita proposed openstack/cinder master: DNM: CI test probe https://review.opendev.org/c/openstack/cinder/+/846230 | 22:05 |
| *** dviroel|biab is now known as dviroel | 23:07 | |
| *** dviroel is now known as dviroel|out | 23:16 | |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!