Wednesday, 2020-01-29

*** nweinber has joined #openstack-nova00:04
*** tosky has quit IRC00:08
*** dtruong has quit IRC00:12
*** nweinber has quit IRC00:12
*** dtruong has joined #openstack-nova00:12
*** munimeha1 has quit IRC00:21
*** nweinber has joined #openstack-nova00:24
*** mlavalle has quit IRC00:32
*** nweinber has quit IRC00:38
*** threestrands has joined #openstack-nova00:50
*** TxGirlGeek has quit IRC01:06
*** slaweq has joined #openstack-nova01:11
*** slaweq has quit IRC01:16
*** igordc has quit IRC01:23
*** sapd1_x has joined #openstack-nova01:34
*** spatel has joined #openstack-nova01:35
*** Liang__ has joined #openstack-nova01:52
*** gyee has quit IRC01:55
*** hamzy has quit IRC01:56
*** hamzy has joined #openstack-nova01:56
*** zzzeek has quit IRC02:03
*** TxGirlGeek has joined #openstack-nova02:04
*** damien_r has joined #openstack-nova02:05
*** rnoriega_ has joined #openstack-nova02:05
*** damien_r has quit IRC02:06
*** zzzeek has joined #openstack-nova02:08
*** zul has quit IRC02:12
*** damien_r has joined #openstack-nova02:45
*** tbachman has quit IRC02:47
*** damien_r has quit IRC02:57
*** spatel has quit IRC03:06
*** slaweq has joined #openstack-nova03:11
*** damien_r has joined #openstack-nova03:16
*** slaweq has quit IRC03:16
*** links has joined #openstack-nova03:18
*** damien_r has quit IRC03:20
*** TxGirlGeek has quit IRC03:50
openstackgerritMerged openstack/nova master: Handle cell failures in get_compute_nodes_by_host_or_node  https://review.opendev.org/70018603:52
openstackgerritMerged openstack/nova master: doc: define boot from volume in the glossary  https://review.opendev.org/69900903:52
*** Liang__ has quit IRC04:09
*** udesale has joined #openstack-nova04:13
*** damien_r has joined #openstack-nova04:16
*** factor has joined #openstack-nova04:24
*** Liang__ has joined #openstack-nova04:33
*** Liang__ has quit IRC04:38
*** abhishekk|away is now known as abhishekk04:39
*** TxGirlGeek has joined #openstack-nova04:50
*** damien_r has quit IRC04:56
openstackgerritHuachang Wang proposed openstack/nova master: [WIP] More tests for validating CPU policy by 'resources:PCPU' and 'resources:VCPU'  https://review.opendev.org/69600704:57
openstackgerritHuachang Wang proposed openstack/nova master: [WIP] Refactor the code path of creating instance with a NUMA topology  https://review.opendev.org/68893204:57
openstackgerritHuachang Wang proposed openstack/nova master: [WIP] Assign instance dedicated CPU set through `cpu_pinning` field  https://review.opendev.org/68893304:57
openstackgerritHuachang Wang proposed openstack/nova master: [WIP] Introduce a new instance CPU allocation policy: mixed  https://review.opendev.org/68893404:57
openstackgerritHuachang Wang proposed openstack/nova master: [WIP] support unbalanced dedicatd CPU distribution on instance NUMA nodes  https://review.opendev.org/69600804:57
openstackgerritHuachang Wang proposed openstack/nova master: [WIP] Create 'mixed' instance from PCPU and VCPU resources  https://review.opendev.org/69600904:57
openstackgerritHuachang Wang proposed openstack/nova master: [WIP] metadata: export the vCPU IDs that are pinning on the host CPUs  https://review.opendev.org/68893604:57
*** slaweq has joined #openstack-nova05:11
*** TxGirlGeek has quit IRC05:12
*** TxGirlGeek has joined #openstack-nova05:12
*** slaweq has quit IRC05:15
*** TxGirlGeek has quit IRC05:17
*** HagunKim has quit IRC05:32
*** mmethot_ has joined #openstack-nova06:17
*** sapd1_x has quit IRC06:17
*** mmethot has quit IRC06:20
*** sapd1_x has joined #openstack-nova06:30
*** ratailor has joined #openstack-nova06:45
*** alex_xu has quit IRC06:47
*** slaweq has joined #openstack-nova07:11
*** slaweq has quit IRC07:15
*** dpawlik has joined #openstack-nova07:16
*** luksky has joined #openstack-nova07:20
*** sapd1_x has quit IRC07:32
*** maciejjozefczyk has joined #openstack-nova07:54
*** dtantsur|afk is now known as dtantsur07:54
*** slaweq has joined #openstack-nova07:56
*** tesseract has joined #openstack-nova08:03
*** tkajinam has quit IRC08:08
*** rpittau|afk is now known as rpittau08:08
*** rcernin has quit IRC08:20
*** luksky has quit IRC08:28
*** luksky has joined #openstack-nova08:31
*** tosky has joined #openstack-nova08:35
*** mrch has joined #openstack-nova08:41
*** belmoreira has joined #openstack-nova08:57
*** ralonsoh has joined #openstack-nova08:59
*** martinkennelly has joined #openstack-nova09:17
*** luksky has quit IRC09:26
bauzasgood morning Nova09:31
stephenfino/09:38
*** iurygregory has joined #openstack-nova09:40
gibi\o09:43
openstackgerritStephen Finucane proposed openstack/nova master: Use COMPUTE_SAME_HOST_COLD_MIGRATE trait during migrate  https://review.opendev.org/69522009:45
*** threestrands has quit IRC09:47
openstackgerritBalazs Gibizer proposed openstack/nova master: Support unshelve with qos ports  https://review.opendev.org/70475909:48
openstackgerritStephen Finucane proposed openstack/nova master: Recalculate 'RequestSpec.numa_topology' on resize  https://review.opendev.org/66252210:05
*** xek has quit IRC10:10
*** derekh has joined #openstack-nova10:13
*** lifeless has quit IRC10:19
stephenfingibi, bauzas: Either of you care to sanity check this and then send it on its way? Just updating to reflect post-merge reality10:25
stephenfinhttps://review.opendev.org/#/c/666032/10:25
bauzasstephenfin: sure10:25
bauzasbtw. thanks stephenfin and gibi for my multiple GPU types spec reapproval10:25
stephenfinnp. Made sense10:26
bauzasI'm more concerned by the NUMA topology in placement spec that's blocked because of good gibi's and sean-k-mooney's finding10:26
*** luksky has joined #openstack-nova10:29
*** dviroel has joined #openstack-nova10:30
*** psachin has joined #openstack-nova10:36
*** xek has joined #openstack-nova10:55
*** pcaruana has quit IRC10:57
*** priteau has joined #openstack-nova11:00
*** udesale has quit IRC11:01
sean-k-mooneybauzas: im conserned about that spec too and the interaction with mix cpus11:02
sean-k-mooneybauzas: can we have a call or something to work on this together11:03
sean-k-mooneybauzas: that said im currently working on the backport. i did not feel great yesterday so did not get it finished :(11:04
openstackgerritStephen Finucane proposed openstack/nova-specs master: Re-propose the flavor extra spec validation spec  https://review.opendev.org/68265511:08
*** xek_ has joined #openstack-nova11:10
*** xek has quit IRC11:10
sean-k-mooneybauzas: if you have not already done so i think you shoudl review https://review.opendev.org/#/c/668656/ too by the way11:15
bauzassean-k-mooney: ack, will do later in the afternoon11:17
bauzasand thanks for the reviews11:17
*** psachin has quit IRC11:18
openstackgerritStephen Finucane proposed openstack/nova master: WIP: api: Add support for extra spec validation  https://review.opendev.org/70464311:24
gibistephenfin: looking11:29
gibistephenfin: +A11:38
stephenfingibi: ta11:39
*** pcaruana has joined #openstack-nova11:40
openstackgerritStephen Finucane proposed openstack/nova-specs master: Re-propose the flavor extra spec validation spec  https://review.opendev.org/68265511:43
openstackgerritMerged openstack/nova-specs master: Additional upgrade clarifications for cpu-resources  https://review.opendev.org/66603211:49
stephenfingibi: I know it's horrible stuff, but could I ask you to revisit https://review.opendev.org/#/c/662522 this week? There's another patch needed to improve how we rollback in a failure, but that should stand by itself all the same11:50
gibisure I added to my queue11:50
stephenfinthanks11:50
*** rpittau is now known as rpittau|bbl11:53
*** ociuhandu has joined #openstack-nova11:58
*** salmankhan has joined #openstack-nova12:23
*** links has quit IRC12:24
huaqiangstephenfin: Thanks your review for spec https://review.opendev.org/#/c/668656. I haven't responded to your comments in time because I am still in my vacation of the chinese new year,12:26
huaqiangI want to comfirm that you prefer the PCPU mask approach that we have dropped, right?12:27
*** ociuhandu has quit IRC12:31
*** mgariepy has joined #openstack-nova12:35
*** ociuhandu has joined #openstack-nova12:40
*** jawad_axd has joined #openstack-nova12:54
stephenfinhuaqiang: No problem. There's no rush. Yes, that's what I'm in favour of. I'll try discuss it with dansmith this week so we can have it resolved for when you're back13:04
*** artom has quit IRC13:06
*** alex_xu has joined #openstack-nova13:08
huaqiangstephenfin: thanks. It's pretty close to the freeze of spec, I planed to work but not work fully these days.13:12
huaqiangI'll keep my eye on the update of gerrit and irc.13:13
*** maciejjozefczyk has quit IRC13:14
*** damien_r has joined #openstack-nova13:17
*** ociuhandu has quit IRC13:17
*** ociuhandu has joined #openstack-nova13:21
*** ociuhandu has quit IRC13:26
*** nweinber has joined #openstack-nova13:27
*** rpittau|bbl is now known as rpittau13:30
*** ratailor has quit IRC13:32
*** maciejjozefczyk has joined #openstack-nova13:39
*** ratailor has joined #openstack-nova13:44
*** ratailor has quit IRC13:48
*** ratailor has joined #openstack-nova13:50
*** Liang__ has joined #openstack-nova13:53
*** Liang__ is now known as LiangFang13:53
*** ratailor has quit IRC13:59
*** ociuhandu has joined #openstack-nova14:00
*** shilpasd has joined #openstack-nova14:18
*** KeithMnemonic has joined #openstack-nova14:20
*** zzzeek has quit IRC14:22
*** zzzeek has joined #openstack-nova14:23
*** spatel has joined #openstack-nova14:36
openstackgerritLenny Verkhovsky proposed openstack/os-vif master: WIP: Testing CI  https://review.opendev.org/68693714:37
*** jawad_axd has quit IRC14:39
*** jawad_axd has joined #openstack-nova14:39
*** spatel has quit IRC14:40
*** jawad_ax_ has joined #openstack-nova14:43
*** jawad_axd has quit IRC14:45
*** spatel has joined #openstack-nova14:48
*** jawad_ax_ has quit IRC14:48
*** priteau has quit IRC14:50
*** priteau has joined #openstack-nova14:51
*** spatel has quit IRC14:54
stephenfingibi: Replied at https://review.opendev.org/#/c/682655/3/specs/ussuri/approved/flavor-extra-spec-validators.rst@194. I'll hold off addressing sean-k-mooney's comments until you've taken another look14:55
gibistephenfin: ack, looking14:56
*** priteau has quit IRC14:56
sean-k-mooneystephenfin: im reviewing you poc code at the moment14:56
sean-k-mooneyalmost done14:56
*** mrch has quit IRC14:58
efriedstephenfin: just posted another round of comments. Since sean-k-mooney and gibi are here, perhaps we can hash out the granular-control issue realtime.14:58
* gibi has 30 minutes before he needs to leave but sure let's do it14:59
sean-k-mooneywe might not need it but if we do add it rather then a config option i think the new microverions shoudl add a &validate=stric|permissive|off option to the post14:59
sean-k-mooneymaybe just strict|off based on stephens last comment14:59
gibistephenfin: do you think people will hack on a the in-tree validator yamls if it is in the python package dir?15:00
stephenfinI've no idea. I just know this would be the first time we'll have used YAML to define something that's core to nova15:00
stephenfinThere's no reason we couldn't define our nova.conf "schema" in YAML but we chose not to. I guess if we can figure out why we did that, we'd solve this too15:01
efriednova.conf is ini style15:02
stephenfinbut not the definitions for those config options15:02
efriedconceived, if I'm not mistaken, before yaml was sexy.15:02
*** dklyle has quit IRC15:02
efriedIt would be possible to do these validators ini-style and use oslo.conf15:02
sean-k-mooneygibi: so i really hate the idea of only doing this with yaml files but if we do then i think there is no excust to not use the json based metadef format that glance has and share the files as a lib15:03
stephenfineeew, I'm not writing that much JSON15:03
efriedYeah, definitely don't want the files to be json. yaml exists because json is ugly.15:03
*** dklyle has joined #openstack-nova15:03
stephenfinalso, I think I did look at that and it didn't offer everything we'd need15:03
sean-k-mooneybut we are needless reinventing the wheel here15:04
gibiI don't want to start a format war, I just wanted to avoid implementing two pieces of code when one piece would be enough technically15:04
stephenfinBut glance's stuff wasn't invented here, Sean.15:04
stephenfin:P15:04
sean-k-mooneythe have a format for declaring this. they expose it via an api for heat and horizon to consume to generate use15:04
bauzashonestly, what's the problem with stevedore ?15:04
efriedOh, yeah, IMO ini isn't powerful enough for long-term use, even for nova.conf (as demonstrated by the travesty that is passthrough_whitelist).15:04
bauzasnova was built around it15:04
bauzasplugins FTW15:04
sean-k-mooneywe could pull the galnce sutff into a lib and share it15:04
* bauzas jk15:04
efriedbauzas: to me, just unnecessary complexity and an additional thing we don't need.15:05
sean-k-mooneyif we go file based only i think that is what we shoudl do15:05
bauzasefried: entrypoints are complex to manage ?15:05
bauzashonestly, we haven't heard about this since nova exists15:05
efriedentrypoints plus python code plus packaging, versus putting a yaml file in a directory? Hell yes.15:05
sean-k-mooneybauzas: there is one complexity with them form a packaging point of view15:06
stephenfinsean-k-mooney: so we'd be using a package to distribute a YAML file?15:06
efriedugh, no, please.15:06
bauzasit's config15:06
efriedCan we not boil the ocean?15:06
sean-k-mooneyjson file and the code to do all the validation which they have laready written as far as i knwo15:06
stephenfinI'd personally rather strip the flavor stuff out of glance15:07
stephenfinWhy do they need to care about flavors? They're an image service15:07
*** priteau has joined #openstack-nova15:07
efriedhow did glance get involved here?15:07
sean-k-mooneythey care about it becasue we had this problem years ago and decied that they would be used as the catalog of support meatdata itmes15:07
bauzasI thought the spec was simple enough to just re-accept it15:08
efried(rhetorical question; can we stay on topic?)15:08
bauzasgiven it was accepted beofre15:08
bauzasbut now, looks like we rathole around any possible better way to provide config15:08
stephenfinso I really don't want another os-whatever library to maintain. This should be in-tree, with some kind of mechanism to export to glance if we really care15:09
stephenfinWhich I'd be surprised if we do, since sean-k-mooney is the only person that's ever heard of this feature in glance15:09
bauzasthat's exactly why we chose plugins before15:09
bauzasbut... whatever15:09
bauzaslooks like we have consensus around YAML15:09
bauzasthe glance issue shouldn't be taken care as of now, until we know whether glance needs it as well or not15:10
stephenfinso the question comes down to Python objects or YAML, and whether we need to make the strict behavior opt-in or not15:10
sean-k-mooneywell im going to be out voted so i guess yes15:10
dansmithyaml for what? rules of extra spec validation?15:10
stephenfinyup15:10
dansmithand... why?15:10
gibiI think having a simple api microversion to opt in is enough15:10
dansmithbecause we can share that file between us and glance?15:11
efriedno15:11
efriedso we can accommodate extra specs nova doesn't own or know about15:11
gibidansmith: because we need the yaml for deployer defined extra specs anyhow15:11
stephenfinso we don't have two different ways to represent a rule15:11
stephenfinhttps://review.opendev.org/#/c/704643/2/nova/api/validation/extra_specs/hw.py15:11
bauzashonestly, I was preferring the python approach thru stevedore plugins15:11
stephenfinwe want to provide a way for operators to specify their own custom extra specs for out of tree filters, or whatever15:11
sean-k-mooneydansmith: gibi does not like the python class way of doing it and suggeste that if we are going to support yaml anyway just use it for everything. fair summary?15:11
stephenfinsean-k-mooney: yup, that15:12
gibiI don't want to start a format war, I just wanted to avoid implementing two pieces of code when one piece would be enough technically15:12
dansmithso, this brings the policy problem screaming back from hell?15:12
stephenfinoslo.policy?15:12
sean-k-mooneydansmith: no that was brogh back for a different reason15:12
dansmithi.e. we provide a base yaml, they edit it to add their stuff and tweak their prefs and then when they go to upgrade they're sadface?15:12
stephenfinthat was another concern of mine, but I figured we'd just load multiple files15:13
sean-k-mooneye.g. if the new microversion enabled validateion and we extend the flavor update again in the future you cant use the new feautre with validation idsabled15:13
stephenfinI'm almost certain someone will go tweak the YAML file we provide though15:13
bauzasyuuuuup15:13
*** mriedem has joined #openstack-nova15:13
dansmithstephenfin: I would argue that we don't/shouldn't encode our baseline validations into yaml and distribute them that way, as it's just asking for that problem15:13
dansmithright15:13
sean-k-mooneydansmith: so i was suggesting add a &validate query arge to turn off validation in the new microversion15:13
efriedIf we do in-tree stuff with yaml, we put the nova-owned stuff within the nova package so nobody fs with it.15:13
dansmithefried: riiiight15:14
stephenfinsean-k-mooney: hold up, let's figure the YAML thing out first15:14
efriedriiiight they'll still f with it? Let 'em. That's breaking the seal, voiding the warranty, not different than if they go patch the code.15:14
sean-k-mooneyefried: distos will modify it every time we backport stuff15:14
gibiwe somebody want to tweak what is written in the python DSL for in-tree validation he can do it, it is almos like a yaml file15:14
dansmithsurely the yaml to validate something complex like numa whatever is going to be nasty or naive right?15:14
sean-k-mooneyit will be an interesting regex15:15
gibibut nothing more than a regex15:15
stephenfingibi: I don't think it's a DSL, really. It's just an object. I could use dicts if that would be preferable15:15
dansmithan "interesting" regex or some relatively simple python right?15:15
bauzas:)15:15
sean-k-mooneyactully its still a regex in stephns code15:15
sean-k-mooneywell poc15:15
stephenfinnot entirely15:15
sean-k-mooneybut it could be simpler15:15
stephenfinI let you specify enums etc.15:16
sean-k-mooneyyes15:16
stephenfingranted, all that gets converted into a regex but there's no reason I have to do that for all cases. It's a PoC after all15:16
sean-k-mooneyalthough you convert the enum int an bool filed back to regex form15:16
* gibi drops the case for yaml only, but still would like to see an implementation that does not duplicated code for intree and out of tree extra spec validation15:16
dansmithso I think my feeling on this is: (a) I'm not sure we need an interface for people to be able to specify their own validators, but (b) if we do, that seems like an obvious use case for entrypoints and a dead simple plugin interface with a super-controlled input and output behavior set15:17
efriedgibi: it won't be a lot of duplication, really. The parser will construct the same kind of python objects we have in tree and then feed all of them into the validation engine.15:17
stephenfingibi: yeah, what efried said15:18
gibiOK, I rest my case then15:18
stephenfinit's just a slightly different representation of the exact same data15:18
stephenfindansmith: So kill the YAML idea entirely?15:18
stephenfinIf so, the main issue I'd see with that is that they'd have to write and install a Python package for the custom stuff15:19
dansmithstephenfin: I don't think our validations should be in yaml, and I don't really see why custom ones should be either.. they'd be a lot harder to test than a single python function you can run with a string you would put in your flavor15:19
bauzasstephenfin: I don't see a problem here, we asked for that since a while15:20
efriedIf we do that, we can't have strict validation. Writing python code and figuring out stevedore for snowflake extra specs is too high a barrier to entry for that to be the only option.15:20
dansmithstephenfin: I think that would be a pretty simple example thing, they have to do that for scheduler filters, and I expect anyone that feels they need this (honestly I don't see it being that important) will be fine with that standard15:20
bauzas(like, Blazar was having a specific nova-ish package for nova plugins he needed)15:20
bauzasyeah, it's very similar to the scheduler filters case15:21
stephenfinI was about to ask what the story with extra scheduler filters was. Those are a package too so15:21
bauzasyup15:21
bauzasor delivered by other means15:21
dansmithefried: stevedore is on the loader side, they don't have to do anything like that.. entry points are a stock python thing15:21
stephenfinefried: Are snowflake extra specs a real thing people use?15:21
stephenfinFar as I knew, extra specs were for scheduler filters and virt drivers15:21
sean-k-mooneythey use custom one for filter15:21
efriedf man, even if you managed to miss one of the stock ones15:21
dansmithstephenfin: you can in conjunction with the json filter or yeah a custom scheduler filter15:22
dansmithstephenfin: if you're writing a custom scheduler flter, you're doing a package, so throwing a validator in there makes plenty of sense15:22
stephenfinIf it's a custom filter, we're saying they're using a package already so no big deal15:22
stephenfindansmith: yeah15:22
gibimy employer's downstream product has at least two snowflake extra spec15:22
gibibut sure I can implement a python filter for that15:22
sean-k-mooneysame with a thrid party virt dirver15:22
dansmithstephenfin: imagine the fame and fortune if you wrote up a nice blog post on how to do this super cleanly :D15:22
dansmithsean-k-mooney: shut yo' mouth! :D15:23
sean-k-mooney:)15:23
stephenfinIf I can do one thing well, it's write lots of docs and bug people continuously to review them15:23
bauzasscheduler filters are the exact reasons why we never constrained extra specs15:23
stephenfinno issues there15:23
bauzasso I guess 3rd-parties would *love* to consider filters and validation code to be considered equal15:24
sean-k-mooneyso that bring up stict vs permissive  vs off15:24
bauzasso they could package this at once15:24
stephenfinefried: I don't think that's as big a deal as you think. Realistically, how often do people create new flavors15:24
sean-k-mooneywe could skip all non namespaced extra space or ones in a namespace we dont know about15:24
sean-k-mooneyin permisive mode at least15:24
*** ociuhandu has quit IRC15:24
stephenfinThey try create one, we fail because we don't recognize the spec, they report and bug and we add/backport the missing validator15:24
stephenfinIt's just a bug15:24
*** ganso has quit IRC15:25
stephenfin*report a bug15:25
dansmithare you going to make admins able to create flavors that fail validation15:25
dansmith?15:25
*** ganso has joined #openstack-nova15:25
stephenfindansmith: that's the question15:25
stephenfinI want a microversion that starts failing extra specs that don't pass validation15:26
dansmithif there is a &force=yes then that gives them a way past the validation if there's a bug15:26
sean-k-mooneydansmith: that is what is was kind of suggestin with  &validate=strict|permissive|off15:26
stephenfinand I'm arguing that ^ is unnecessary since we should be able to audit the extra specs we have in-tree15:27
*** jmlowe has joined #openstack-nova15:27
dansmithsean-k-mooney: oh per-create, I thought you meant global config15:27
sean-k-mooneyno per request15:27
stephenfinefried suggested global config. I said no cos config-driven API behavior is bad15:27
dansmithstephenfin: yeah, I dunno.. I think some of them are complex enough that you might not be able to do that as well as you think15:27
dansmithstephenfin: agree15:27
dansmithstephenfin: but I think it's legit to have a validation=no per-request, but that can return 403 either because of policy or config if it's documented when you add it, no? so later we could turn that off or default it off15:28
*** jmlowe has quit IRC15:28
stephenfinhmm, that wouldn't be so bad15:29
dansmitheither way, I just think you might not want to assume you can enumerate every crazy way people have abused those extra_specs and having a bunch of angry people calling your home :D15:29
sean-k-mooneythe reason we would wont that is if we add this in 2.68 and in 2.69 we add another field to the flavor but we still want to disable validtion it would be nice to be able to turn it off15:29
bauzashonestly, leave them time to change their flavors15:30
stephenfinsean-k-mooney: yeah, I've been suggesting using the older microversion purely as a stop gap to allow us iron out the kinks15:30
dansmithIMHO, and AFAICT, validation is *just* to help make the admin's life easier so they don't have to keep booting instances to test that they completely broke an extra_spec right?15:30
bauzasalso, are we proposing to fix existing embedded flavors ?15:30
dansmithso being able to turn it off in an emergency doesn't seem like a bad thing to me15:30
sean-k-mooneybut that would not work if the thing i wanted to set was added in 2.6915:30
stephenfinThis also won't break existing flavors. It'll only prevent them creating new broken flavors15:30
dansmithyou're not going to periodically fsck your flavors so, what does it matter?15:30
bauzasstephenfin: I mean, instance's embedded flavors15:30
*** artom has joined #openstack-nova15:31
sean-k-mooneystephenfin: creating or updating15:31
stephenfinbauzas: Nope. Out of scope for this. This focuses purely on setting new flavor extra specs15:31
sean-k-mooneyyou want to be able to do the valiation optionaly on flaovr updates too15:31
dansmithsean-k-mooney: that's true, if you're updating a flavor that currently doesn't pass validation15:32
stephenfinWe could look into doing that with a nova-manage command (or nova-audit) command in the future but I'm not even thinking about that at the moment15:32
stephenfinsean-k-mooney: Well, setting or updating a new extra spec15:32
sean-k-mooneybauzas: ya for not if you need to update embeded flavor resize15:32
sean-k-mooneystephenfin: yes15:32
dansmither, wait, stephenfin are you just doing validation on the one spec you're setting during that operation or on the whole flavor as it would be with the spec set?15:33
stephenfindansmith: Just the one spec15:33
dansmithah okay15:33
bauzasI need to reread the spec because I'm unclear of some bits15:33
stephenfinI don't plan to e.g. audit the existing extra specs on a flavor when setting a new one15:33
* gibi needs to leave15:33
stephenfingibi: o/15:33
sean-k-mooneydansmith: the spec says that while there is a depency between some spec we wont enforce that15:33
dansmithstephenfin: aren't there some specs that depend on each other, like being able to set something that isn't valid unless some other spec says you have that thing?15:33
dansmithokay gotcha15:33
bauzasOK, flavor set, there is15:33
gibio/15:34
sean-k-mooneywe could add that in the future15:34
stephenfindansmith: There are but we can't enforce it at the API level15:34
dansmithstephenfin: because some have co-dependency?15:34
efriedwe would be able to enforce some interdependencies15:34
stephenfinotherwise you'd have to set e.g. 'hw:cpu_policy' before 'hw:cpu_dedicated_policy'15:34
stephenfinintroduces a whole slew of ordering issues15:34
efriedbut not things like "this extra spec is only relevant to the libvirt driver"15:34
sean-k-mooneystephenfin: in some case we can but often the depency can be fulfiled by the image too15:34
dansmithstephenfin: right, I dunno that that's bad necessarily15:34
dansmithstephenfin: you just couldn't enforce that "X and Y have to be both present or both absent" I think15:35
dansmithanyway, my concern over the update was if you were tweaking a flavor that currently doesn't pass and you can't get yourself out of trouble with single ops15:35
dansmithsounds like that's not a thing15:35
sean-k-mooneyya it should not be an issue in what is proposed for this cycle15:36
* dansmith nods15:36
stephenfinI could do that enforcement but I probably don't need to do it in the API15:36
dansmithwell, I'm not sure what the point is if not in the API :)15:36
sean-k-mooneyas i said in some cases the co depency can be fulfiled by the image so we often dont know if it will be fuliled until the boot request15:37
sean-k-mooneyso that a diffent level of validation15:37
dansmithsean-k-mooney: ah, that is a very good point15:37
*** tbachman has joined #openstack-nova15:37
stephenfinoh, good point15:37
sean-k-mooneyif it a co depency that is purly fulfiled in the flavor15:37
dansmithyup15:37
sean-k-mooneymaybe but that feels like a ux bug15:37
efriedthat ship has sailed though15:38
sean-k-mooneyyes15:38
stephenfinanyway, I think having the registry to validate extra spec names and the separate validators to handle the values gets us an awful lot over what we have at the moment15:38
* sean-k-mooney leaves this there for future reading https://docs.openstack.org/glance/pike/user/glancemetadefcatalogapi.html15:39
sean-k-mooneystephenfin: but yes i agree with that statemnet15:39
stephenfinand it's the lowest hanging fruit, which is nice :)15:39
stephenfinI'm going to update the spec to drop the YAML idea in favour of entrypoints and also add the escape hatch qparam15:40
stephenfindansmith, efried, sean-k-mooney, bauzas: That make sense to you folks?15:40
sean-k-mooneyya im fine with that15:41
dansmithyup15:41
dansmithI didn't catch earlier discussion, but was gibi opposed to all that?15:41
bauzascool15:41
efriedgrudgingly accept entrypoints over yaml. qparam ++15:41
sean-k-mooneyincidentally if you felt like writhing a blog on it or example plugin you could provide a yaml one15:41
stephenfinhe wanted the qparam too, I think, but also suggested we use YAML for everything if we were using it15:42
stephenfinso we didn't have two different representations of the same thing15:42
efriedso now we have one representation15:42
efriedit's python.15:42
sean-k-mooneyyes15:42
stephenfinif everything's an object, that concern's resolved. gibi will let me know if he doesn't agree, I'm sure15:42
stephenfinefried: Don't you love it15:43
sean-k-mooneywe can even assume its python 3 at long last15:43
dansmithcool15:43
efriedIf we're providing the qparam so I can push through my snowflake without writing python code for it, I'm okay.15:43
stephenfin\o/ sweet15:43
efriedI still think it would be a good idea to have a 'permissive' mode on that qparam15:44
stephenfinefried: what's the difference vs. enable/disable?15:44
efriedIf I'm constructing my flavor all at once, rather than one extra spec at a time, it gives me the advantage of validating the known/in-tree things while ignoring the snowflakes.15:44
sean-k-mooneypermissive mode i guess would log a warning or let you know it failed validation15:45
stephenfinoh, that's what I was going to do for off15:45
efriedusing two separate calls is a workaround to that15:45
efriedyeah, 'off' is 'warn'.15:45
efriedbut that's for values too15:45
sean-k-mooneyoh i was thing off ment you know off as in dont even check15:45
stephenfinno, I was thinking off means you'll never be able to disable validation for the in-tree stuff15:45
stephenfinsean-k-mooney: nope15:45
efried'permissive' means don't restrict to known keys.15:45
stephenfinwait, that sentence doesn't make sense15:46
* stephenfin retries15:46
stephenfinno, I was thinking off means don't worry about unrecognised keys but do validate values for the known ones15:46
stephenfinso  you'll never be able to disable validation for the in-tree stuff15:46
stephenfinbetter.15:46
sean-k-mooneyin my head 'stirct' = know keys only, 'permissive' mean allow unkonw keys but warn, off mean dont do validation at all15:47
stephenfinwould you ever want to turn it off for *everything*?15:47
efriedI'm not married to the names, but I thought three modes:15:47
efried- off: we don't 4xx for anything. We can still run the validators, but just warn if we find something awry.15:47
efried- permissive: only validate values, only for keys we recognize. Ignore unrecognized keys.15:47
efried- strict: above, plus fail on unrecognized keys.15:47
sean-k-mooneyits what you will get for the old microversion15:48
sean-k-mooneyso i thnk it makes sense to represent that code path as an option15:48
stephenfinefried: That mostly makes sense but I'm wondering what off is useful for15:48
sean-k-mooneystephenfin: off is basicaly whatever behavior you get with the old microverison15:49
stephenfinright, but is that behaviour ever desirable though?15:49
sean-k-mooneyso if im using a newer micoverion for the flavor endpoint that add something not in the old one i can still turn off the vlidation15:49
efriedstephenfin: We forgot or messed up an in-tree one, and don't want to wait for the fix. Could flip to the prior microversion, but the warnings would give me a way to sort of manually make sure everything else is kosher.15:49
sean-k-mooneystephenfin: yes15:49
sean-k-mooneystephenfin: there was a spec for composable flavor recently15:50
stephenfinefried: wdym by "everything else"15:50
efriedthe remainder of the extra specs in my request.15:50
sean-k-mooneyif that was approved and landed after this feature we might want to use that without validation15:50
stephenfinoh, yes, you can set multiple extra specs at once. duh15:50
efriedI try with strict/permissive, I get a bounce on extra spec X, I redrive with 'off' and make sure the only warning I get is on X.15:50
sean-k-mooneyefried: im more or less ok with your deffintions by the way your off just does a littel more then i expected15:51
efriedwe can call it 'warn'15:51
stephenfinefried: vs. just retrying one by one?15:51
efriedyes15:51
efriedor "I want to test drive this feature"15:51
sean-k-mooneyi suggested permissive to mirror selinux but warn would be fine15:51
efried"but still create my flavor"15:51
sean-k-mooneyalthough that implice that strict should be error15:51
efriedreally for completeness15:52
*** lbragstad has quit IRC15:52
stephenfinOkay. You're aware the warnings are only going to be in the logs though, yeah?15:52
stephenfinSo I'm not sure how much feedback you're going to be getting15:52
efriedIt would be nice to find a way to send them back to the CLI, but yeah, I get it.15:52
*** ociuhandu has joined #openstack-nova15:52
stephenfincompared to just trying one by one 'til you find the erroneous extra spec15:53
sean-k-mooneywell for errors we can put it in the resonce body15:53
stephenfinsean-k-mooney: right, but not for warnings15:53
efriedI think that ^ would be nice for a future microversion, don't need to do it now.15:53
sean-k-mooneyfor wraning maybe we could alter the respocne to include them too15:53
stephenfinfor those we already have a body - the updated flavor (I think)15:53
sean-k-mooneyya it should be the full flavor object15:53
sean-k-mooneybut we could extend it from the mircoverion on15:54
stephenfinyup, so no way to get back that warning via the API15:54
sean-k-mooneythat said its a nice to have15:54
efriedso then yeah, call it 'off' and the warnings are in the logs15:54
sean-k-mooneysure there is we just change the api respoce to also have validtion warning but lets leave that out of scope for now15:54
*** jmlowe has joined #openstack-nova15:54
stephenfinefried: ack15:54
* stephenfin goes to rework the spec15:55
efried(note that we technically wouldn't need 'strict', since that's the default, but again for completeness it would be nice to support it)15:55
stephenfinagreed15:55
sean-k-mooneyyep15:55
sean-k-mooneyi like the implcit state to be setable explictly even if its the default if you say nothing15:56
sean-k-mooneyit makes testing nicer imo15:56
sean-k-mooneyand it make documting the behavior simpler since you have a name for each mode15:56
efriedcould call 'permissive' 'values-only' or something, since 'permissive' isn't particularly descriptive15:56
efriedbikeshed away.15:57
*** lbragstad has joined #openstack-nova15:57
sean-k-mooneyya maybe again i just suggeste permissive since selinux in permissve mode doese policy validateion and log any violation but allow whatever the process was doing to continue15:57
sean-k-mooneyit seamed like a good analog but not many peopel might make that connection15:58
efried'ignore-unknown-keys'15:59
efried'ignore-(well-okay-log-a-warning)-unknown-keys'15:59
dansmithso,16:00
dansmithI had to deal with something and lost track here16:00
dansmithone reason to not allow the client to choose more than strict or not-strict is that if you have permission to create flavors within your tenant, and you can set validation=warn (in the logs), then you can spam the server-side logs with warnings16:00
dansmithwhich would be considered somewhat of a DoS16:00
stephenfinwe're adding a policy for this though, right?16:02
dansmitha policy for what? each of the values of the validation=(on|off|warn) ?16:03
stephenfin<dansmith> stephenfin: but I think it's legit to have a validation=no per-request, but that can return 403 either because of policy or config if it's documented when you add it, no? so later we could turn that off or default it off16:03
stephenfinwhat did you mean by that? ^16:03
stephenfinI assumed you meant for the ability to set '?validation=<anything>'16:04
dansmithstephenfin: I meant state that validation=no might be disabled (i.e. always required) in the docs, so that we can allow that to be turned off in config or policy in the future if the op wants to require all flavors be validated16:04
dansmithI'm just arguing against having a client-controlled way to emit WARNING messages in the server-side logs16:05
sean-k-mooneyor i guess restircted to a group e.g. admins although it is an admin only api already by default16:05
stephenfinI think WARNING might be a bit much, tbh. INFO seems appropriate if the user has explicitly requested no/limited validation16:06
sean-k-mooneythe policy.json seam like a better approch if we were to make it configurable16:06
stephenfinIf it was INFO, we'd be okay, right?16:06
stephenfinsince no operator will get paged for INFO logs16:07
sean-k-mooneyi think dansmith was concerned about ddos risks16:07
dansmithI dunno, I just don't like it in general16:07
dansmithright16:07
sean-k-mooneyinfor does not really help with hat16:07
sean-k-mooney*that16:07
dansmithbut it's not a sticking point16:07
dansmithsean-k-mooney: correct16:07
stephenfinthey could already ddos things but just repeatedly hitting the API with working extra specs16:07
stephenfinwe log each request iirc16:07
stephenfin*by just16:07
stephenfinthat's what API rate limiting is for16:08
sean-k-mooneywell i suspect the error could be long if you add a bunch of invalide extra specs so its a larger risk16:08
sean-k-mooneybut again its an admin only api by default16:08
dansmithand logging something that is a warning as info doesn't make it info16:08
sean-k-mooneyso the request will get rejected by a normal user well before it hit your code16:08
dansmithbut anyway, like I say, not a sticking point16:09
stephenfinIs it a warning though? Per above, info does seem appropriate if the user has explicitly requested no/limited validation16:09
*** eharney has quit IRC16:09
stephenfinIt's a warning if they didn't, which they'll see in their HTTP 4xx response16:09
dansmithseems like a warning to me :)16:10
dansmithanyway16:10
dansmithwe needn't argue about it16:10
dansmithI will just toldjaso if you get CVE paperwork over it :)16:10
stephenfinfine by me :)16:10
*** maciejjozefczyk has quit IRC16:11
*** gyee has joined #openstack-nova16:14
bauzasmmm, what the heck is this ?16:16
bauzasstestr: error: unrecognized arguments: --test-path=./nova/tests/functional run test_nova_manage16:16
*** mlavalle has joined #openstack-nova16:16
bauzasholy shit16:16
bauzasI probably need to upgrade stestr by hand16:16
bauzasbut I did recreated my venc16:16
bauzasvenv*16:17
stephenfintox -e function --recreate ?16:17
stephenfin*al16:17
stephenfinunless you're using stestr without tox, which is odd16:18
stephenfinseeing as you can do stuff like 'tox -e functional -- -n path_to_functional_test.py::TestClass.test_method'16:18
bauzasstephenfin: that's what I did16:22
bauzasand I ended up with stestr==2.6.016:22
bauzasI always asked to run a subset of tests by doing tox -efunc <my_regex>16:23
bauzasand it worked16:23
stephenfinbauzas: That's what I have too. Working fine here16:24
bauzaschecking with stestr==2.5.016:25
*** TxGirlGeek has joined #openstack-nova16:25
bauzasfunctional runtests: commands[0] | stestr --test-path=./nova/tests/functional run test_nova_manage16:25
bauzasmmmmm16:25
stephenfinbauzas: '$ tox -e functional -- test_nova_manage' wfm16:26
bauzasI never had to use the positional markers, but whatever, gonna try16:27
bauzascrazy, still same issue16:27
stephenfinwant to dump the complete output to paste.o.o16:28
stephenfinit's something simple, I'd suspect16:28
bauzasyeah me too, but can't see the problem16:30
bauzasit's just stestr which doesn't sound to accept --test-path16:30
bauzashttp://paste.openstack.org/show/788935/16:30
bauzasactually, nope16:31
bauzaswell, I'm puzzled16:33
stephenfinah, wait, have you added additional tests for nova-manage?16:34
stephenfinThere's a bug with oslo.config whereby the CLI parser is global'ish16:35
stephenfinYeah, '--remote_debug-port' is a nova-manage option. You're not mocking stuff properly16:35
stephenfinbauzas: mriedem saw the issue pop up in a unit test at https://review.opendev.org/#/c/694806/2/ and it's currently causing an issue with glance and the latest version of cliff16:38
*** lbragstad has quit IRC16:38
bauzasstephenfin: interesting, if I use the functional-py36 target, it does work16:40
stephenfineven if you rebuild the venv?16:40
bauzasit was created, I never used the target yet16:41
stephenfingotcha16:41
stephenfinweird16:41
stephenfinthen I'm not sure :(16:41
bauzasto make it clear, the functional-py36 target works, but not the standard one16:41
bauzaseither way, I know what to do, but I have to look at the target differences in tox.ini16:41
*** luksky has quit IRC16:42
efriedbauzas: would it be productive for me to read the current PS of the numa topo spec, or wait til you rev it?16:42
bauzasefried: I'm wraping my head around the group_policy issue, but your thoughts could be helpful16:43
bauzasefried: and we have a disagreement on the memory modeling with NUMA with sean-k-mooney, your opinion could help us finding a consensus16:43
bauzasefried: to answer your question, yeah comments would be appreciated on the current rev16:44
efriedokay, I'll give it a read. My view on group_policy is that we should be ignoring it. It doesn't really have a place with granular groups and the other knobs we put in in recent microversions.16:44
efriedI at least proposed (though I don't remember if we actually pulled the trigger on this) making it no longer required in a recent microversion.16:44
sean-k-mooneybauzas: sorry can we pick this up after the internal call16:45
sean-k-mooneye.g. in 15 mins16:45
bauzassean-k-mooney: yeah, and I even wanted to discuss it during the internal call :D16:45
bauzas(but I'm superseded by other topics :) )16:45
sean-k-mooneyya i saw16:45
efriedI won't be much help on the memory modeling, probably, as I really don't understand that level of detail of NUMA itself. But I'll give it a shot.16:47
bauzastbh, me too16:48
bauzasthe big question is should we iteratively model huge pages and have memory split now, or care about the whole now ?16:49
efriedI'll have to refresh my memory (heh) on whether we did things to support cross-provider accumulation of resources, e.g. so you could still land on a NUMA-modeled host with 128/128 if you asked for 256.16:51
*** eharney has joined #openstack-nova16:51
efriedI seem to recall we tried to KISS by saying you land on a NUMA-modeled host by asking for NUMA-modeled resources, and the converse, but that may have only been a point in time in the discussion, not where we landed.16:52
efriedI'll have to swap this all back in.16:52
bauzasefried: yeah that's what I recall too16:53
*** dviroel has left #openstack-nova16:55
*** jmlowe has quit IRC16:56
*** ociuhandu has quit IRC16:59
*** bnemec-ooo has quit IRC17:00
openstackgerritVladyslav Drok proposed openstack/nova master: Fix volume attachment rollback  https://review.opendev.org/70484717:03
*** jmlowe has joined #openstack-nova17:07
efriednts: we did abandon "can_split" https://review.opendev.org/#/c/658510/17:07
sean-k-mooneyyes we did17:08
* efried is rereading https://docs.openstack.org/placement/train/specs/train/implemented/2005575-nested-magic-1.html17:08
stephenfinsean-k-mooney: for that memory issue, is there a reason we need to continue to support floating memory when NUMA is used?17:09
stephenfindo we do strict NUMA affinity when using a NUMA topology at the moment?17:09
sean-k-mooneythat depends. yes but the question is do we care17:09
stephenfinfor memory, that is17:10
sean-k-mooneywe would have to pin the memoyy based on the placemetn allocation of all vms17:10
stephenfinI think we don't unless we explicitly request a pagesize, yeah?17:10
sean-k-mooneybut for vms with out a numa toplogy we could pin that vm aross host numa nodes17:10
stephenfinokay, so that leads to my question: do we need to support that model?17:11
stephenfinVMs with a NUMA topology alongside those without?17:11
efriedAgain, I think we decided it was acceptable to enforce a segregated data center17:11
sean-k-mooneyif we want to support vms without a numa toplogy yes17:11
efriedwhere some hosts are NUMA-aware and some are not.17:11
stephenfinsean-k-mooney: we could have a knob17:11
stephenfinuse_numa_in_placement17:11
efriedAnd if you ask for a NUMA topo, you land on the former, always, exclusively. And the converse.17:11
stephenfinefried: right, thing we're talking about the same thing17:11
stephenfin*think17:11
sean-k-mooneyi would prefer to handel it via resouce classes17:11
sean-k-mooneyso if you report memory_mb its non numa aware memory17:12
efriedExcept for the upgrade issue, I think it wouldn't matter.17:12
sean-k-mooneyif you reprot mempage_small its numa aware17:12
efriedUpgrade is my only concern about doing MEMORY_MB now and pages later.17:12
efriedbecause now you have flavors that straddle both models17:12
stephenfinsean-k-mooney: right, but that's duplication17:12
efriedbut they are mutually exclusive as to the hosts they work with.17:12
sean-k-mooneysame for the cpu if you use VCPU its not numa aware and on the root provider if it SCPU its numa aware shared cpu17:12
stephenfinand instances will have to consume both17:13
stephenfinassuming we report both17:13
sean-k-mooneystephenfin: no the host would only report one or the other17:13
efriedWe shouldn't have the same resource represented by two different resource classes. We've nacked that idea many times in the past.17:13
stephenfinwhat determines what type we report?17:13
sean-k-mooneywe did it with pcpus and vcpus17:13
sean-k-mooneybut if we say that then MEMORY_MB is not the correct baseline for memory17:14
efriedeh? No, the PCPU and VCPU counts represent different physical processors.17:14
*** martinkennelly has quit IRC17:14
stephenfinoh, so you're saying instead of having a "turn on NUMA" flag, we have two flags: one for non-NUMA memory and one for NUMA memory?17:14
efriedyeah, I think we ought to bite the bullet on memory. If we're ever going to want it to be something other than MEMORY_MB, we should do that now.17:14
sean-k-mooneythey represetn different pools of hardware threads17:14
sean-k-mooneybut they are both the resouces17:14
sean-k-mooneywe just treat them as if they are different17:14
efriedmy point is, they don't overlap17:14
*** jmlowe has quit IRC17:15
sean-k-mooneyright so i think we should be modeling memory as mempages of a specifc size17:15
*** lbragstad has joined #openstack-nova17:15
sean-k-mooneywe already do in the resouce tracker in the host numa toplogy blob17:15
efriedIf we have N pages that total to M mb of ram, we should *either* model PAGES=N *or* MEMORY_MB=M, but *never* both at the same time.17:15
efriedI think we're agreeing on that, just want to be clear.17:15
stephenfinYup17:15
sean-k-mooneyyep although its slightly more complicated17:16
efriedsean-k-mooney: I also want us to make sure we're not tying ourselves to libvirt unless that's inherently unavoidable.17:16
sean-k-mooneybut in pricipal yes always one or the other17:16
efriedI.e. are there other drivers that don't think of their NUMA-based memory in terms of pages?17:16
sean-k-mooneyyes i want to add an abstration17:16
stephenfinmaybe we should make like the 90s and teach placement about OOP17:16
efriedOr that think of them as pages of a different size?17:16
sean-k-mooneyso mempages_small and mempages_larage17:16
stephenfin1GB_PAGES is_a PAGES17:16
efriedhahaha, you're a funny guy.17:17
sean-k-mooneywith 1G or 2MB large ppages modeld as a trait17:17
efriedNO17:17
sean-k-mooneyyes because we support hw:mem_page_size=large17:17
stephenfinPAGES_1M, surely?17:17
efriedI hate that.17:17
stephenfinPAGES_1M, PAGES_2K, PAGES_1G?17:17
sean-k-mooneyand that cant be modeld as pages_2M and pages _1G17:17
efriedunless 1M is the only size we can ever have.17:17
*** jmlowe has joined #openstack-nova17:17
efriedNo17:18
efriedwe ought to be able to do that with MEMORY_MB with appropriate step_size.17:18
sean-k-mooneymy point is we cant make mem_page_size=large work if the resouce class has the size info17:18
efriedthen you can ask for so many MB, and you'll get the right number of pages chopped up according to that step_size.17:18
efriedpretty sure that's what step_size exists for.17:18
sean-k-mooneythe step_zize and min/max allocation works17:18
stephenfinhow does one determine the step size?17:19
efriedit's the page size17:19
sean-k-mooneyi have talked about that before17:19
*** dtantsur is now known as dtantsur|afk17:19
efriedthe virt driver knows how big a page is, yah?17:19
stephenfinlarge isn't a page size17:19
sean-k-mooneybut we cant explctly request 1G or 2mb pages in that case17:19
efriedno, 'large' isn't a thing.17:19
stephenfinit's anything bigger than 4k17:19
sean-k-mooneyefried: yes it is17:19
stephenfinwhich could be 1G on half the compute nodes17:19
sean-k-mooneyhw:mem_page_size=large is the recommend way to enable hugepages17:19
efriedwait, so there are flavors that care how many pages they get, but not how much memory that ends up being????17:20
sean-k-mooneyyes17:20
stephenfinno17:20
sean-k-mooneywell not pages17:20
sean-k-mooneybut page size17:20
stephenfinthey care about how much memory they get but not the exact size of the pages17:20
stephenfinjust so long as it's != 4k17:20
stephenfin2M, 1G, 8G (on POWER) - it's all fair game17:20
sean-k-mooneystephenfin: that is not alwasy true17:21
sean-k-mooneymany people do care about the page size17:21
efriedDid we ever implement required=in:T1,T2,T3?17:21
sean-k-mooneyno17:21
efriedso17:21
sean-k-mooneyif we did we could use that17:21
sean-k-mooneyalthough no it would not work because reqired in was for traits17:22
efriedif people care about specific page sizes, we have a trait that says PAGE_SIZES_HERE_ARE_4K.17:22
efriedBut if people care about 'large', where that's allowed to mean "bigger than X", that doesn't work.17:22
stephenfinright, and those people are probably explicitly saying e.g. 'hw:mem_page_size=2k'17:22
sean-k-mooneyyes17:22
stephenfinor know that their datacenter is configured with only 2k or 1G pages17:22
efried(is bauzas still listening btw?)17:22
stephenfinbauzas is on kid duty17:22
stephenfinbut has scrollback17:22
efriedk17:22
efriedyuh17:22
stephenfinefried: Still not sure how we do the step_size determination though17:23
sean-k-mooneyhttps://etherpad.openstack.org/p/mem_page_size_and_placement17:23
sean-k-mooneylets go there too17:23
efriedstephenfin: Doesn't the virt driver know how big its pages are?17:23
sean-k-mooneyyes17:23
*** jmlowe has quit IRC17:23
efriedthat's the step_size.17:23
sean-k-mooneyand a host can have multiple pages sizes17:23
efriedin the same numa cell?17:24
sean-k-mooneya host can have 4k 2mb and 1g in the same cell yes17:24
gibistephenfin: having a single representation is a positive thing for me. and I accept that it will be python instead of yaml17:24
stephenfinBut you don't know what virt driver you're going to use before you query placement17:24
efriedstephenfin: step_size is part of the model, not part of the query.17:24
sean-k-mooneyyep which is why large is a thing17:24
sean-k-mooneyefried: yep that was why step size did not work when i discussed this in the past17:24
efriedbecause you can have different size pages in the same cell.17:25
sean-k-mooneywe cant say 10G of ram in 1G increments17:25
sean-k-mooneyat least not with just stepsize17:25
stephenfinefried: you can, and not everything the same size17:25
efriedokay, then there's no way this works without an abstraction and/or simplification.17:25
*** jmlowe has joined #openstack-nova17:25
efriedso we need to decide what to cut17:25
stephenfinHow would you say "give me a compute node that can handle an 8GB instance with 1G pages"17:26
efriedbecause we're not going to make resource classes for PAGE_$size17:26
stephenfinI don't see how we can avoid it17:26
stephenfina 4k page != a 2M page17:26
efriedstephenfin: because then you *can't* say "give me an instance with 8GB"17:26
efriedwhich is surely the more common use case?17:27
stephenfinsure you can17:27
stephenfinwe just translate the request17:27
efriedyou would have to translate the request into multiple GET /a_c queries.17:27
efriedbecause you can build 8GB a zillion different ways17:27
efriedAND it requires discovery of which page-size-resource-classes exist in your deployment.17:27
efriedthat's a non-starter, sorry.17:27
stephenfinNo, you can't consume hugepages unless you explicitly request them17:28
sean-k-mooneyso by default if you dont specify hw:mem_page_size it has to be the smalles page size17:28
sean-k-mooneyefried: that is why i want mempage_small and mempage_large to be different resoce classes17:28
efriedbut "huge" isn't a discrete number?17:28
efriedso again, if you do that, you can't ask for 8GB17:28
stephenfinhugepages are offlimits for instances without 'hw:mem_page_size'17:28
stephenfinright, sean-k-mooney?17:28
efriedbecause you have to ask for a number of pages, but you don't know how big those pages are.17:28
stephenfinyeah, that one is a problem17:29
stephenfinbecause it's intentionally abstract17:29
sean-k-mooneyyes17:29
sean-k-mooneyif you dont set hw:mem_page_size you will only use 4k memory17:29
stephenfinright17:29
stephenfinso for non-hugepages instances, there will only ever be a single GET /a_c query17:29
efriedum17:30
stephenfinfor MEMORY_MB or PAGES_4K or whatever we call it17:30
sean-k-mooneyefried: its that way because people wanted to change for hugepages and people did not want all instacne to have a numa toplogy by defualt17:30
efriedthere will only ever be a single GET /a_c query.17:30
efriedever ever17:30
stephenfinnot for pinned instances, at the moment17:30
stephenfinbut that's a temporary thing17:30
stephenfinagreed17:30
sean-k-mooneyso i was proposeing useing MEMORY_MB for non numa small pages17:30
sean-k-mooneye.g. on the root RP17:31
efriedsean-k-mooney: we're simplifying the "all instances have a numa topology" aspect by saying that NUMA-aware requests only go to NUMA-modeled hosts and vice versa.17:31
stephenfinsean-k-mooney: agree on the first part. disagree on the second17:31
stephenfinit's on the root RP if that compute node is configured to be NUMA'y17:31
sean-k-mooneywell that is all hosts17:31
efriedThe small pages are still affined, yah?17:31
efriedSo they should be modeled on the NUMA RPs17:31
sean-k-mooneythere are no non numa host for the better part of a decade17:31
stephenfinefried: If you're using guest NUMA, they should be17:31
stephenfinsean-k-mooney: No, but there are people who pretend they don't exist17:32
stephenfin*they exist17:32
sean-k-mooneyefried: they have an affinty but they are allocated by the kernel17:32
efriedyeah, so given a strict NUMA-or-not split, I don't see a reason to keep any memory on the root17:32
sean-k-mooneyand they can move between numa nodes17:32
stephenfinby creating e.g. 128GB instances on a dual socket compute node with only 64GB RAM per node17:32
sean-k-mooneyso unless we pinn the memory to a numa node when we creat the vm the kernel manages where the pages are paged form17:33
sean-k-mooneyfor hugepages we allways pin them to a numa node17:33
efriedand why would we not pin the small pages?17:33
efriedwait17:33
efriedlet me turn that around:17:33
*** rpittau is now known as rpittau|afk17:33
efrieddo we ever want to make sure we *do* pin the small pages?17:33
sean-k-mooneywe have an option for that today17:34
sean-k-mooneyif you set hw:mem_page_size=small17:34
efriedk, then we should model them in the numa cells, and always pin them.17:34
sean-k-mooneyif you set hw:mem_page_size it will be pinned17:34
stephenfinIf we're modelling NUMA in placement, then yes. We want to make sure that the memory placement allocated came from the actual provider (the host NUMA node)17:34
efriedthat will mean there are a few cases where you didn't care about that and you don't land on a host whose small pages are sparse and spread across cells. But we've already said we're accepting that.17:35
sean-k-mooneyefried: we could but the implciation of that are 1 of the follow two things need to happen17:35
stephenfinotherwise placements says "here's 8GB from <RP mapping to host NUMA node #1>", but on the host that memory comes from node #017:35
stephenfinand placement doesn't reflect reality17:35
sean-k-mooney1 all vms must be numa vms and there fro cannot span host numa nodes unless you set hw:numa_nodes>117:36
efriedthis ^17:36
sean-k-mooneyor we have to allow a singel numa node guest to sapn host numa nodes based on the placement allcoation17:36
efriednope17:36
stephenfinalternative17:36
efriedYour VM is a numa VM17:36
stephenfinlet people turn off the NUMA reporting17:36
stephenfinmake everything flat again17:36
efriedIf it's not a numa VM, it doesn't land on a NUMA host17:36
efriedyes, that ^17:36
sean-k-mooneyok i have argued for your vm is a numa vm for years17:37
stephenfinYeah, I also want that17:37
efriedsorry, you're misunderstanding.17:37
stephenfinI know what efried is going to say17:37
stephenfinthe NUMA reporting has to be turned on?17:37
sean-k-mooneyok no i get what your saying17:37
sean-k-mooneybut to not land on a numa host i think we need to have 2 resouce classes17:37
stephenfindefaults to off17:37
efriedWe force deployers to split their data center.17:38
efriedN hosts are NUMA-modeled. All and only VMs with a NUMA topo land on those hosts.17:38
efriedM hosts are flat. All and only *non* NUMA topo VMs land on those hosts.17:38
sean-k-mooneyso in pricipal you shoudl be doing that partioning alreday17:38
sean-k-mooneyfor reason i wont go into today17:38
efriedexactly.17:38
stephenfinshe borked.17:38
stephenfin(if you don't do that partitioning)17:38
sean-k-mooneyelequently said17:39
efriedthere may be a way we can support a small amount of overlap in the future, but for the moment I think it wouldn't be a terrible idea to add (under the covers) a required or forbidden trait that makes sure you land on the right kind of host.17:39
efriedlike the NUMA_ROOT trait on the provider from which you're requesting $resource.17:39
efriedso let's get back to pages.17:39
stephenfinthat might be needed to handle the upgrade impact17:39
efriedHow many "abstract" page sizes are there?17:39
efriedsmall/large/huge?17:40
stephenfinwe can't expect users to go set the "this is NUMA node" flag ahead of time17:40
stephenfinefried: small, large17:40
stephenfinhey, I documented that for my validator17:40
efriedstephenfin: totally not, that happens under the covers, now and forever, if it happens at all.17:40
efriedokay, and how many *discrete* page sizes are there within those buckets?17:40
sean-k-mooneysmall/large/any  or a specifc page size expressed as an integer with an optional suffix17:40
stephenfinhttps://review.opendev.org/#/c/704643/2/nova/api/validation/extra_specs/hw.py@11317:41
efried\o/17:41
stephenfinnoting sean-k-mooney's comment17:41
sean-k-mooneythe integuer is unbounded but in pratice about 12ish17:41
stephenfinefried: architecture dependent17:41
stephenfinIntel supports 2M and 1G17:41
stephenfin*x8617:41
efriedlovely. Is there overlap on what's considered small and large?17:41
stephenfinPOWER supports 8G, iirc17:41
sean-k-mooneyefried: jay put up a patch to os resouces for the common ones a while ago17:41
stephenfinWho knows what ARM supports17:42
efriedthat is, are there discrete values that are considered small on some systems and large on others?17:42
stephenfinALL the pagesizes17:42
sean-k-mooneyefried: in practice not really but technically there can be17:42
stephenfinsean-k-mooney: 4k is a small page _everywhere_, right?17:42
sean-k-mooneysmall is almost always 4k17:42
sean-k-mooneyno17:42
efriedthose are different statements17:42
sean-k-mooneysmall is the native pagesize of the host17:42
sean-k-mooneyand the smallest in the available set17:43
sean-k-mooneyit is almost alwasy 4k and raely 16k or 64k17:43
stephenfinsean-k-mooney: The internet tells me 4k is hardcoded as the default page size in Linux17:43
stephenfinhttps://unix.stackexchange.com/a/12821817:43
sean-k-mooneyyes on x86, aarch64 and power917:44
stephenfinokay, we don't need to care about anything else, realistically17:44
sean-k-mooneythat why i said its almost alwasy 4k17:44
stephenfinwe can stick a giant TODO in somewhere in case someone wants to run openstack on obscure architecture17:45
sean-k-mooneywe can proably treat it as such17:45
stephenfinagreed17:45
stephenfins/TODO/NOTE/17:45
stephenfinefried: what are you referring to?17:45
*** TxGirlGeek has quit IRC17:45
sean-k-mooneylarge is defiend as any pagesize that is not the same as small on the plathform17:45
efriedokay, so I'm afraid the compromise we need to make is this:17:45
efriedRepresent all the memory as MEMORY_MB with a step_size that's the least common denominator of all the page sizes.17:45
efriedSupply traits saying things like I_HAVE_$SIZE_PAGES_HERE, where $SIZE can include both discrete and abstract sizes.17:45
efriedUse ^ where possible to make the placement pass a bit better17:45
efriedbut accept that it's not going to be perfect, and that we have to do the rest in the NTF.17:45
efried...which may entail bouncing a host late.17:46
stephenfinI don't understand how step_size would work17:46
sean-k-mooneyefried: if we do that we need to keep the hugepage code in the resouce tracker and numa toplogy filter for ever17:46
efriedI'm afraid that may be unavoidable, unless we make big changes in placement, or people agree to stop needing that shit.17:47
sean-k-mooneyefried: if we had a placement exteion weree we could pass a step size in the querry that would help17:47
stephenfinI have a host that have 16 1GB hugepages and the rest are normal small pages17:47
efriedstephenfin: example, if we have 4K, 1M, and 2G pages, the step_size has to be 4K17:47
stephenfinyou will always use 4k so17:47
stephenfinevery host is going to report small pages17:48
efriedokay, so be it.17:48
sean-k-mooneyefried: not all the moroy can be allocated in 4k17:48
sean-k-mooneyon that host17:48
sean-k-mooneyhugepages are preallcoted17:48
sean-k-mooneywith a given page size17:48
stephenfinI don't get how this solves anything17:48
efriedyeah, I understand that. We have to do that part via the NTF. I don't see a way around it.17:48
efried"solves"17:48
efriedit gives us a way to move forward17:48
efriedand support all the variants we need17:49
sean-k-mooneywell the thing is if we had different resocue classes hugepage was one of the things we had determin could be fully done by placment17:49
stephenfinthis is essentially not modelling different page sizes in placement17:49
sean-k-mooneyif we modeld the different pages sizes in placment17:49
efriedstephenfin: correct.17:49
stephenfinNUMA is all about memory17:49
sean-k-mooneyso thats a lot of work for very littel benifit17:49
sean-k-mooneyyep17:49
efriedwe cannot do that and also support requests for MEMORY_MB=$X that don't care about pages.17:49
stephenfinwhat's the point in modelling NUMA in placement if we don't do the memory modelling?17:50
efriedaffinity17:50
sean-k-mooneyefried: we can if MEMORY_MB is always for non numa memory17:50
stephenfinwe have the NUMA topology filter for that17:50
stephenfingood enough17:50
efriedstephenfin: this allows us to move a big swatch of the affinity filtering to the placement query.17:50
efriedjust not all of it.17:50
sean-k-mooneyefried: no it does not17:50
efriedand not as much as we would have liked17:50
efriedwha?17:50
stephenfinvery little of it17:50
efriedwha??17:51
sean-k-mooneywe will have to do it all again in the ntf17:51
stephenfinyup17:51
efriedum17:51
efriedno17:51
efriedlook17:51
sean-k-mooneywe may have less host to check17:51
*** salmankhan has quit IRC17:51
stephenfinwhat's the objection to modelling different page sizes as different resource types?17:51
stephenfinthey're different things17:51
sean-k-mooneybut we still need to check all aspecs again17:51
stephenfinsomething that ask for one can't consume the other17:51
efriedBecause it makes it impossible to request a certain amount of memory without asking for specific pages.17:51
sean-k-mooneythere is another way to do it17:51
sean-k-mooneywe can do it in 1 resouce class17:52
sean-k-mooneyMEMORY_MB17:52
stephenfinbut it doesn't17:52
efriedhow would you do it stephenfin?17:52
sean-k-mooneybut we need 1 RP per page size17:52
stephenfinwe report MEMORY_MB for normal 4k pages17:52
sean-k-mooneyand each RP would be under a numa node17:52
stephenfinand then PAGES_4M, PAGES_1G, etc. for the various hugepage sizes17:52
sean-k-mooneythat wont work17:53
stephenfinif instances aren't requesting page sizes, they get MEMORY_MB17:53
sean-k-mooneybecause it breaks large17:53
stephenfinit does, and we'll need to figure out a migration plan for that17:53
sean-k-mooneymem_page_size=large is the most common case17:53
efriedOkay, so if you don't ask for large pages, you only get small pages, and if the only way to satisfy your memory requirement would be by using the large pages, you just bounce.17:53
stephenfinyou can't use the large pages17:53
stephenfinunless you *explicitly* ask for them17:53
efriedis that what happens today?17:53
stephenfinyup17:53
efriedokay cool.17:54
sean-k-mooneyyes17:54
stephenfingiven a host with 8GB of RAM, of which 4GB is allocated to 1GB huge pages17:54
sean-k-mooneywhich is why you have to partion the host today17:54
sean-k-mooneywe have 2 memory tracker in nova today17:54
efriedis it possible today to ask for both small and large pages in the same request, or is mem_page_size=large all or nothing?17:54
stephenfinonly 4GB is actually usable17:54
sean-k-mooneyone is numa and page size aware and the other just looks at total memory17:54
stephenfinnope, all or nothing17:55
efriedcool17:55
efriedthen yes, split into two resource classes.17:55
sean-k-mooneyefried: there is mempage_size=any17:55
efriedugh17:55
sean-k-mooneybut that just means the image gets to choose17:55
stephenfinN resource classes17:55
efriedand if the image doesn't specify... bounce?17:55
sean-k-mooneyand it will be small if not stated17:55
efriedstephenfin: Still can't do N resource classes. Have to have two.17:55
stephenfinCan't do two17:55
efriedone for small, one for large.17:55
efriedEach one has a step_size that's the least common denominator of the pages in that range.17:56
stephenfinHost has some 1G hugepages and some 2M pages17:56
sean-k-mooneyefried: any was ment to give you the smallest pagesize available but we skimed on it and just do 4k pages i think17:56
stephenfinI can't request some of the former and some of the latter17:56
stephenfinIt has to be one or the other17:56
sean-k-mooneystephenfin: that is a nova limitation17:56
efriedokay, but if you say LARGE, we get to pick17:56
stephenfinSo placement could say "oh, I have hugepages", but it turns out they're different sizes17:56
sean-k-mooneynot a libvirt one but we should not mix them17:56
stephenfinand it can't actually fulfil that request17:56
efriedyeah, the resource class would not indicate number of pages, it's still a number of MB.17:57
efriedright, that's the part we would have to defer to the NTF17:57
efriedOtherwise you *still* can't say memory=4GB,page_size=large because we would have no way to translate that request.17:57
sean-k-mooneyyes17:57
efried...without doing multiple placement queries, which is a hard no.17:57
stephenfinthen find a way to kill page_size=large or convert it at the API level17:57
sean-k-mooneyso there is a way to do this with only one resocue class + traits17:57
stephenfinapi config option17:58
stephenfinlarge_mem_page_size = PAGE_2M17:58
sean-k-mooneybut to do that it need a 3 level resouce provider tree17:58
stephenfinthat's required from N+117:58
stephenfinas an interim step in N, we make two requests, one for PAGE_2M, one for PAGE_1G17:58
sean-k-mooneystephenfin: mem_page_size=large is the most comonly used value17:58
stephenfinlike we're doing for VCPU and PCPU rn17:58
stephenfinsean-k-mooney: yup, and we can continue supporting it17:59
efriedis that not config-driven API behavior?17:59
efriedthe thing you just told me was a no-no17:59
stephenfinwe just have to ask operators to define what large aliases to17:59
sean-k-mooneyno17:59
efriedbut that only works if the deployment only has one large page size?17:59
stephenfinon the assumption that people aren't mixing and matching different page sizes17:59
efriedand if that's true, then the whole issue is moot17:59
sean-k-mooneybecause its not uncommon for the same cloud to have both17:59
efriedif we can assume that, we're golden.18:00
stephenfinwhy would they do that?18:00
sean-k-mooneythat is not a safe assumtion18:00
sean-k-mooneybecause differnt applciation perfrom better18:00
* efried ==> bio break18:00
sean-k-mooneywith different page sizes18:00
stephenfinby application you mean something in an instance?18:00
stephenfinif so, are they actually using 'large'?18:01
stephenfinvs. the size that performs better18:01
stephenfin*best18:01
sean-k-mooneyalso 1G was added a lot later then 2mb18:01
sean-k-mooneyit requires specifc cpu support18:01
stephenfinokay, so this will force operators to choose one or the other for their deployment18:01
sean-k-mooneythey use large for applciation that can support either and 1G when needed18:01
sean-k-mooneyyes which i dont think we need to do18:02
sean-k-mooneyim going to skech somthing up for ye to review18:02
sean-k-mooneygive me 2 mins18:02
stephenfinyeah, good idea18:02
stephenfinI know what efried's approach is and I don't like it18:02
stephenfinDon't grok yours yet though18:02
stephenfinAlso, I'm supposed to be going out for dinner so I should probably scarper for now18:03
sean-k-mooneywhat is the ascii site we use for specs18:03
*** lbragsta_ has joined #openstack-nova18:03
stephenfinascii?18:03
stephenfinascii site?18:03
stephenfinfor drawing?18:03
sean-k-mooneyhttp://asciiflow.com/18:03
*** derekh has quit IRC18:03
sean-k-mooneyya i litrally ment draw since its clearer18:03
stephenfinoh, I was going to say just use https://www.draw.io/18:04
stephenfinor https://docs.google.com/drawings18:04
stephenfinanyway, let me know what you draft18:04
* stephenfin buggers off home18:04
*** TxGirlGeek has joined #openstack-nova18:09
sean-k-mooneystephenfin: im thinking 3 layers like this18:09
sean-k-mooneyhttps://etherpad.openstack.org/p/mem_page_size_and_placement18:09
*** luksky has joined #openstack-nova18:09
sean-k-mooneyefried: bauzas ^18:14
efriedsean-k-mooney: step_size=4 doesn't make sense unless we use MEMORY_KB18:14
sean-k-mooneywell actuly i gues it would be 102418:15
sean-k-mooneysince we limit flavor to 1mb granuarity18:15
efriedtbc, it's not really a problem to introduce new RCs for this, since we're doing the translation under the covers and only allowing NUMA-modeled VMs on NUMA-modeled hosts.18:15
sean-k-mooneyits only there becasue you can say mem_page_size=4 or 4k today18:15
efriedokay, that's fine.18:15
efriedThe three-tiered approach works IF you always get exactly one page size18:16
sean-k-mooneythis will allow all the sentinels to wrok and we can remove the hugepage page tracking from the numa toplogy filter/resouce tracker18:16
sean-k-mooneyefried: yes today we only allow 1 page size18:16
sean-k-mooneyso if we dont enable more flexiblity then today we can make that assumtion a requirement18:17
efriedso if I have three different hugepage sizes on the same numa cell, say 1G, 2G, 4G, then memory=8G,page_size=large will only ever get me one of18:18
efried[2 x 4G]18:18
efried[4 x 2G]18:18
efried[8 x 1G]18:18
efriedbut never e.g. [1 x 4G] + [4 x 1G]18:18
efriedright?18:18
sean-k-mooneyyes18:19
efriedcool, then this works, I like it.18:19
sean-k-mooneylibvirt support mixing and we intentionaly do not18:19
sean-k-mooneyi propsoed this in the past and the main push back is an extra layer adds well an extra layer18:19
efriedshall I write it up in the spec comments?18:19
efriedyeah, the layer doesn't bother me. It's totally abstracted from the user.18:20
*** lbragsta_ has quit IRC18:20
sean-k-mooneysure that would be awsome18:20
efriedcool, on it.18:20
sean-k-mooneywell when i first propsoed this we did not have the abitiy to query nested rps because i first brough up this design 3-4 releases ago18:21
sean-k-mooneyso people were more conserend about 3 level when 2 level did not work18:21
efriedwith placement today, this totally works.18:21
efriedyou can represent the affinity using same_subtree18:22
efriedso the extra layer doesn't break that.18:22
sean-k-mooneyya18:22
*** tosky has quit IRC18:22
sean-k-mooneyif we do it this way we can remove much of the logic form the NTF and numa resouce tracker18:22
sean-k-mooneyunlike cpu pinning hugepage just need to know how much of each page type is avaible per numa node18:23
sean-k-mooneyso placement with 3 level can model eveything we need to track18:23
sean-k-mooneyso we could remvoe all the mempage trackinging in the host numa topology blob and only compute that in memroy to update the palcement inventory18:24
melwittstephenfin: looks like your update covered most of what was there but looks like you left out the server names the test was filling in before? https://review.opendev.org/69522018:28
*** priteau has quit IRC18:33
*** eharney has quit IRC18:38
*** benj_ has quit IRC18:47
*** benj_ has joined #openstack-nova18:47
*** ralonsoh has quit IRC18:47
*** eharney has joined #openstack-nova18:51
*** iurygregory has quit IRC18:52
sean-k-mooneyefried: fyi the ther node would be for vgpus althoug they would proably go a the bottem level of the tree beside the memory ones18:54
sean-k-mooney* the other nodes below the RP18:54
efriedFor now we're punting on devices, so we'll leave the VGPUs where they are and not support affinity. Later, I agree, the providers representing PGPUs would be underneath the NUMA nodes, parallel to the memory providers.18:54
sean-k-mooneyyep18:55
openstackgerritBalazs Gibizer proposed openstack/nova master: Repro gen conflict in COMPUTE_STATUS_DISABLED handling  https://review.opendev.org/70486518:55
openstackgerritBalazs Gibizer proposed openstack/nova master: Reduce gen conflict in COMPUTE_STATUS_DISABLED handling  https://review.opendev.org/70486618:55
sean-k-mooneyefried: i know we have customer request for numa aware gpus/vgpus but we agreed even internally that that should wait until after the memory/cpu case is done18:56
sean-k-mooneyso im fine with that18:56
efriedyeah, sure, we're not going to not do it, we're just not going to do it now.18:56
sean-k-mooneyyep18:57
efriedsean-k-mooney: I'm going to put some examples in the etherpad, but will need you to go through and edit my spelling on the extra specs.18:57
sean-k-mooneyhehe i will try18:58
* sean-k-mooney editing spelling means breaking it or fixing it ...18:58
efriedI figure if I ask you to edit the spelling of real tokens I have a better chance.18:58
efriedvs. English.18:58
sean-k-mooneyyes you do18:59
efriedye gods, it just occurred to me: do you speak gaelic?18:59
efriedyour writing in gaelic must be IMPOSSIBLE to understand, given that gaelic is impossible to read in the first freakin place.18:59
*** tesseract has quit IRC18:59
sean-k-mooneyyes and no. i went to an all irish play school and got an irish langage exemption becaue i coudl not write in irish19:00
sean-k-mooneyi could speak it and could kind of read it19:00
efried"oh, let me just throw an extra consonant and an extra vowel in each syllable. Half of them will be silent. Which ones? Who knows? It's a surprise!"19:00
sean-k-mooneyhehe irish is interesting that way19:00
sean-k-mooneyespcially since spelling of a verb chagne with gender pluarlity and something the noun it is used ith19:01
*** spatel has joined #openstack-nova19:02
*** igordc has joined #openstack-nova19:02
sean-k-mooneyactully do you want that to be a numa instance example or not19:05
sean-k-mooneyi assume a numa one19:06
sean-k-mooneya non numa one would not have hw:mem_page_size set at all and not triat19:06
efriedeandersson: can you write me a flavor example with two numa nodes where one has small pages and one has large?19:10
efriedwhoops, eandersson disregard. ^ was for sean-k-mooney19:10
sean-k-mooneyno19:10
sean-k-mooneypagesize is vm wide19:10
efried(my IRC client has been dropping the first char of my messages lately)19:10
efriedokay, fine, can you write me a flavor with two numa nodes and an explicit page size?19:11
sean-k-mooneyyes ill add 219:11
efried(turns out 'sean<tab>' minus the first character is 'ean<tab>')19:11
sean-k-mooneyhehe ill remove our disscution on the first example to clean up the description19:12
*** jmlowe has quit IRC19:14
sean-k-mooneyyou can also split the cpu asemetircly but that is overkill for the exampels i think19:16
sean-k-mooneyfor the implict splitign its an error if the cpus and ram and not evenly devisable by the number of numa nodes19:17
efriedsean-k-mooney: I prefer leaving that third example as it was before, since it shows that we could land on either of the LARGE RPs.19:19
efried++19:19
sean-k-mooneyya19:19
sean-k-mooneythat is why i reverted it19:19
sean-k-mooneyany why i used large in the first place19:19
efriedsean-k-mooney: any other examples that we should throw in?19:19
sean-k-mooneyam maybe a non numa guests19:20
efriedyeah19:20
efriedto show that it would not land on the sample host at all.19:21
sean-k-mooneyyes19:22
sean-k-mooneyi was trying to figure out the not num host19:22
sean-k-mooney*numa19:22
sean-k-mooneyif we did not add the forbiden trait19:23
sean-k-mooneythen it could actully land on that host19:23
sean-k-mooneybut we would have to pin it to a numa node19:24
sean-k-mooneywhich we could do but we proably dont want too19:24
sean-k-mooneyso you can continue to create vms that span numa nodes if you dont care19:24
efriedright, that's what I was implying earlier when I said we could probably overlap, but should prevent that for now19:24
efriedoh, that wouldn't allow you to span numa nodes.19:24
sean-k-mooneyya19:24
sean-k-mooneyya it would not you are right19:24
efriedoh, I see, you mean your proc from one and your mem from another19:24
sean-k-mooneynot without can split19:24
sean-k-mooneyyou could have procs and memory span ya19:25
sean-k-mooneybut can_split would be need to have any one resouce span numa nodes19:25
efriedyeah, actually, the non-NUMA example can be simplified if we just use a granular group for the proc & mem....19:25
sean-k-mooneyya19:26
sean-k-mooneythat would work19:26
efriedsimpler, I like.19:26
*** eharney has quit IRC19:26
sean-k-mooneyoh we are missign something from all of them19:26
sean-k-mooneywe need to add group_policy=none19:26
sean-k-mooneybecause we are using granular groups19:26
sean-k-mooneywe still get the correct affintiy because fo same tree in the multi numa case19:27
sean-k-mooneyeven with group_policy=none19:28
efriedyeah, I was just going to check whether we in fact did remove group_policy in the latest microversions.19:28
efriedit's irrelevant in this case. We get the same result with isolate or none19:29
sean-k-mooneylooking at the api doces its still there19:29
efriedokay, boo. But as noted above, it's irrelevant, so set it to whatever. Do you agree?19:29
sean-k-mooneyit can be in this case19:29
sean-k-mooneybut isolate breaks eaisly19:30
sean-k-mooneyfor example 2 port with bandwith requests19:30
sean-k-mooneyso i dont like publicising its use19:30
efriedRight, so the point is, we can leave it to the user and/or the bandwidth code to decide on group_policy.19:30
*** spatel has quit IRC19:30
sean-k-mooneyyes19:30
efriedIIRC the bandwidth code is defaulting it rn.19:30
efriedor at least we talked about doing that.19:31
efriedWe can ask gibi to confirm19:31
sean-k-mooneyrn as in none?19:31
efried'right now'19:31
sean-k-mooneyoh i think its defaultin to none if it is19:31
sean-k-mooneythat should be our default unless you say otherwise19:31
sean-k-mooneyok so it required becasuee we are using groups but the value is not relevent to the spec19:32
sean-k-mooneyim fine with that19:32
sean-k-mooneyits an implemenation detail19:32
efriedyeah. That was a gripe I had during the nested magic design, that group_policy should default (can't remember which way I said) at this point because we can control everything we need using other, more granular (heh) mechanisms.19:33
efriedbut we didn't actually do that.19:33
efriedAnd, confirmed, we're defaulting to 'none' these days.19:33
efriedI found the reno yaml, lemme find where it is in the docs.19:33
sean-k-mooneywell really we should not have it be global either19:34
sean-k-mooneyit should be per set of groups or per subtreee or something19:34
efriedhttps://docs.openstack.org/releasenotes/nova/train.html#other-notes19:34
sean-k-mooneyanyway out of scope19:34
efriedright, that was my gripe. We now have same_subtree to dictate which can be shared and which can be spread. Because same_subtree also devolves to same rp in relevant cases.19:35
sean-k-mooney ya19:35
sean-k-mooneyprovide we model nics with an RP per PF same_subtree or the negation should hanel that too19:36
sean-k-mooneyso we might be able to remvoe the group polices thing at some point19:36
sean-k-mooneyanyway if this does not require it to make it work it makes me happy19:37
*** eharney has joined #openstack-nova19:39
sean-k-mooneyi have saved a copy of that etherpad locally just in case by the way.19:40
efriedcool. I've got my comment all composed, will post it when I'm done reviewing the rest. Hopefully bauzas will parlay the etherpad into the doc before the etherpad goes kablooey, as it inevitably will.19:41
sean-k-mooneywell we nerver delete them19:41
sean-k-mooneythe only go away if the db gets currpted19:41
sean-k-mooneyso you would be suprised how long they survie19:42
sean-k-mooneythat said we have been bitten enough times at ptgs that i make backups19:42
efriedI've seen enough corrupted19:42
efriedyeah.19:42
*** eharney has quit IRC19:48
*** dklyle has quit IRC19:56
*** gmann is now known as gmann_afk19:57
*** lifeless has joined #openstack-nova20:01
*** dklyle has joined #openstack-nova20:03
*** dklyle has quit IRC20:08
*** dklyle has joined #openstack-nova20:14
artomdansmith, around? So, are you completely opposed to https://review.opendev.org/#/c/672595/61/nova/tests/unit/virt/libvirt/fakelibvirt.py@489 ?20:15
artomBecause I've been trying to write something better, and I'm not sure what I have is worth it. Maybe I just suck, or am overthinking it?20:16
dansmithWhy is [1] harder than 1? Aside from all the test change20:16
dansmithor maybe you're stuck on the fact that it really should be named cpu_socket_map or something?20:17
dansmithI do think what you have is pretty ugly and would prefer a mutually exclusive additional arg to what you have It hink20:17
artomdansmith, well, the way I have it now is a full-on separate class20:17
artomWell, classes, because it trickles up and down20:18
artomNow = locally, not pushed yet20:18
dansmithand that's really just for tests?20:18
artomThe objective being, allow tests that don't need the sockets maps complexity to use the existing class, and have a new one for the tests that need it20:19
artomInstead of imposing sockets map on all tests20:19
*** LiangFang has quit IRC20:19
artomYou mean the HostInfo and NUMATopology classes? Yeah, just for tests20:20
dansmithRight, point is you've created a new class to wrap hostinfo just for tests20:20
dansmithseems kinda smelly since they're not doing what the production code does20:20
dansmithat least make that a fixture or utility thing in tests/20:20
artomHostInfo already exists just for tests20:21
*** slaweq has quit IRC20:22
dansmithohhh, sorry I maybe was thinking too fast.. doesn't it mirror something else in the main code?20:22
* dansmith notes it has been 12 days and a week of vacay since he last looked at this20:23
*** slaweq has joined #openstack-nova20:23
artomdansmith, no worries - I've been struggling with this for way too much time, too20:24
artomdansmith, so, the HostInfo is eventually passed down to the fake_connection here: https://github.com/openstack/nova/blob/master/nova/tests/functional/libvirt/base.py#L11120:25
artomdansmith, it's essentially our libvirt service stub20:25
dansmithit doesn't inherit from something real, so ignore my previous comments20:25
dansmitha wrapper class for ease makes perfect sense I think20:25
dansmithyup I'm caught up now20:25
artomOhhhh20:27
artomI think I just got it20:27
artomMake the "internal machinery" use sockets map20:27
artomAnd leave HostInfo on top as a compatibility shim20:27
artomAnd for tests that need it, they can use the "internal machinery" class directly20:28
dansmithyeah, i thought that was what you were suggesting20:32
artomNo, I had 2 classes in parallel20:32
*** jmlowe has joined #openstack-nova20:43
*** jawad_axd has joined #openstack-nova20:52
*** jmlowe has quit IRC20:52
*** slaweq has quit IRC20:54
*** jmlowe has joined #openstack-nova20:54
*** jawad_axd has quit IRC20:56
*** jmlowe has quit IRC20:57
*** lucidguy has quit IRC20:59
*** eharney has joined #openstack-nova21:03
*** slaweq has joined #openstack-nova21:04
*** jmlowe has joined #openstack-nova21:04
*** IvensZambrano has joined #openstack-nova21:05
*** jmlowe has quit IRC21:05
*** jmlowe has joined #openstack-nova21:07
*** artom has quit IRC21:08
*** slaweq has quit IRC21:09
*** jmlowe has quit IRC21:09
*** slaweq has joined #openstack-nova21:11
*** slaweq has quit IRC21:16
*** gmann_afk is now known as gmann21:17
*** jmlowe has joined #openstack-nova21:24
efriedbauzas, sean-k-mooney, stephenfin: Finished that review of the numa topo spec.21:24
*** jmlowe has quit IRC21:25
*** jmlowe has joined #openstack-nova21:31
*** rcernin has joined #openstack-nova21:32
sean-k-mooneyefried: cool i will try to catch up with the review tommorow21:34
*** jmlowe has quit IRC21:37
*** xek_ has quit IRC21:42
*** jmlowe has joined #openstack-nova21:45
*** jmlowe has quit IRC21:46
*** nweinber has quit IRC21:58
*** ociuhandu has joined #openstack-nova22:01
*** jmlowe has joined #openstack-nova22:06
*** ociuhandu has quit IRC22:06
*** tosky has joined #openstack-nova22:10
*** artom has joined #openstack-nova22:19
*** mriedem has left #openstack-nova22:35
*** damien_r has quit IRC22:54
*** kaisers has quit IRC23:06
*** tkajinam has joined #openstack-nova23:07
*** shilpasd has quit IRC23:10
*** TxGirlGeek has quit IRC23:11
*** slaweq has joined #openstack-nova23:11
*** mlavalle has quit IRC23:14
*** spatel has joined #openstack-nova23:14
*** slaweq has quit IRC23:16
*** TxGirlGeek has joined #openstack-nova23:19
*** spatel has quit IRC23:19
*** damien_r has joined #openstack-nova23:20
*** IvensZambrano has quit IRC23:21
*** kaisers has joined #openstack-nova23:23
*** jmlowe has quit IRC23:38
*** jmlowe has joined #openstack-nova23:43
*** tosky has quit IRC23:52

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!