Friday, 2022-03-11

opendevreview王昊 proposed openstack/cinder master: Support multiple filters together for volume time comparison filter
opendevreviewVictoria Martinez de la Cruz proposed openstack/devstack-plugin-ceph master: Deploy with cephadm
opendevreviewVictoria Martinez de la Cruz proposed openstack/devstack-plugin-ceph master: Deploy with cephadm
*** dviroel|out is now known as dviroel|ruck11:12
*** dviroel is now known as dviroel|ruck12:20
hemnaarg more capacity issues.13:47
hemnaalso looks like the volume manager doesn't update the allocated_capacity_gb for migrated volumes.13:48
hemnawhich is also another reason why that can go negative13:48
hemnamigrate a volume from host a to b13:48
hemnadelete the volume.13:48
hemnaI'm seeing thick provisioned pools being over provisioned and I think this is one of the reasons it's happening.13:50
hemnaalso a side effect of the vmware driver allowing lazy creates13:51
hemnacinder asks it to create a 1TB volume, the driver says ok, but does nothing.13:51
hemnait creates the actual volume when someone wants to attach it.13:52
hemnasince the scheduler makes decisions based upon the reported free space for thick provisioned volumes, you get over provisioned thick pool.  :(13:52
eharneyshouldn't the driver account for those lazily created volumes in its stats reporting?13:55
hemnaIt's not currently, and if it was kept in memory, a restart would lose that info 13:55
hemnathe vmware driver is a mess13:56
hemnaand vmware behind it is a mess13:56
hemnathe vmware driver would have to go through every volume created against it and see if there is a backing shadow vm associated with it, if and only if that volume hasn't been attached ever.13:57
hemnathat would be crazy to do at startup13:57
hemnathat's all it does during lazy create13:58
rosmaitahemna: can you lead a discussion about the issues you are finding at the PTG?  i think eharney is correct that some issues seem to be scheduler related but others seem more driver concerns14:07
rosmaitaand like you point out, the vmware driver is kind of pathological14:07
rosmaitawe'll need to get agreement on how this all should  be fixed14:07
hemnawhen is the PTG ?14:08
rosmaitaapril 4-814:08
rosmaitacinder is meeting tuesday-friday14:08
hemnaok do we still need to register for it and all that?14:08
rosmaitayeah, the usual deal14:08
* hemna registers14:09
hemnaok thanks for the reminder14:10
opendevreviewBrian Rosmaita proposed openstack/cinder master: Add yoga release note prelude
rosmaitawhoami-rajat: abishop: let me know if i described your features correctly ^^14:21
abishoprosmaita: ack, left a +1 for mine14:27
whoami-rajatrosmaita, i think it's worth mentioning that it's only applicable for ``available`` and ``error`` volumes for now (we would not want to highlight it for for reserved volumes until the nova part is merged)14:28
whoami-rajatbut maybe that's more detail for a prelude so not sure ^14:28
rosmaitawhoami-rajat: yeah, maybe we should update the new feature part of your release note for that feature?14:29
* whoami-rajat checking the feature releasenote14:30
whoami-rajatrosmaita, yeah it does mention about reimaging reserved volumes although apart from nova, i don't think any end user/operator might use it14:31
whoami-rajatbut we can modify it to remove the reserved part14:31
rosmaitaok, you can propose a patch to stable/yoga (as soon as it's cut) and we can discuss there whether we need to mention it or not14:32
whoami-rajatI'm thinking again about when the feature merges in nova in Zed, will we propose an empty releasenote mentioning it in cinder? maybe the releasenote is fine and I'm just thinking too much14:33
whoami-rajatrosmaita, in any case, i think the prelude looks fine, +114:33
rosmaitacool, ty14:34
opendevreviewOpenStack Release Bot proposed openstack/devstack-plugin-ceph stable/yoga: Update .gitreview for stable/yoga
opendevreviewOpenStack Release Bot proposed openstack/devstack-plugin-ceph stable/yoga: Update TOX_CONSTRAINTS_FILE for stable/yoga
opendevreviewSofia Enriquez proposed openstack/cinder-tempest-plugin master: [tests] Deleting Volumes Which are consumed as a basis for others
hemnaultimately, I think cinder needs to fix the reporting of it's allocated space. and then use that to determine what it thinks a pool has free, and then compare that to what's being reported, and then use the smaller number between the 2.14:37
hemnabecause also, something else besides cinder could be allocated space on that pool.14:38
hemnaeven my new calculate_capacity_factors here: uses the reported free space for thick provisioning14:39
opendevreviewSofia Enriquez proposed openstack/cinder master: rbd: fix full restore ceph backups
*** hemna1 is now known as hemna15:27
dansmithrosmaita: can you look at a CI failure?
rosmaitadansmith: sure15:32
dansmithAFAICT, a lot of tests got stuck in trying to clean up volume backups which takes so long that the job timeout fired and killed it15:32
dansmiththis is on a simple devstack patch to to change how we generate our tls certs, so I wouldn't expect that to be anything related15:32
dansmiththis is just a one-off failure while running that job, but wondering if it looks like something you recognize.. I can't find anything obvious in cinder logs15:33
rosmaitaok, i am in a meeting for a bit, but will take a look15:34
dansmithack thanks15:34
*** dviroel|ruck is now known as dviroel|ruck|lunch15:48
opendevreviewAde Lee proposed openstack/cinder master: WIP/DNM: Add fips check jobs
opendevreviewMerged openstack/cinder master: Add yoga release note prelude
rosmaitawhoami-rajat: can you verify the hash on ?16:40
whoami-rajatlooks good16:45
*** dviroel|ruck|lunch is now known as dviroel|ruck16:46
whoami-rajatdid my +1 trigger PTL-Approved +1 ? didn't know that was in place16:50
rosmaitawelcome to the big time!16:53
opendevreviewEric Harney proposed openstack/cinder master: RBD: Use static methods where possible
opendevreviewMerged openstack/devstack-plugin-ceph stable/yoga: Update .gitreview for stable/yoga
opendevreviewBrian Rosmaita proposed openstack/cinder master: Update doc8 ignore-path
opendevreviewMerged openstack/devstack-plugin-ceph stable/yoga: Update TOX_CONSTRAINTS_FILE for stable/yoga
hemnaanother deficiency I noticed that there is a bit of a race condition between the time when a volume is created and the next time the get_volume_stats is called.20:40
hemnathe allocated_capacity_gb doesn't update 20:40
hemnaand you can create a bunch of volumes between then and the next update.20:40
hemnaI think the scheduler should automatically decrement that space in it's local setting until the next update comes20:43
hemnathis seems quite bad :(20:43
*** dviroel|ruck is now known as dviroel|out21:19

Generated by 2.17.3 by Marius Gedminas - find it at!