Friday, 2019-08-23

*** markvoelker has joined #openstack-nova00:00
*** markvoelker has quit IRC00:05
*** dtruong has quit IRC00:07
*** dtruong has joined #openstack-nova00:07
*** owalsh has joined #openstack-nova00:13
*** TxGirlGeek has quit IRC00:17
alex_xumriedem_away: thanks :)00:19
*** markvoelker has joined #openstack-nova00:22
*** mlavalle has quit IRC00:25
*** mlavalle has joined #openstack-nova00:27
mriedem_awayi love SNES00:31
mriedem_awayand am playing FF1 on PS1 lately because00:32
mriedem_awaybut i was actually physically at a park with my kid00:32
mriedem_awaywith air and sun00:32
mriedem_awaywild00:32
*** mriedem_away is now known as mriedem00:32
*** mriedem has quit IRC00:33
*** gyee has quit IRC00:36
openstackgerritBrin Zhang proposed openstack/nova master: Remove the invalid AZ server create test  https://review.opendev.org/67811600:38
*** ozzzo has left #openstack-nova00:42
*** markvoelker has quit IRC00:47
*** brinzhang_ has joined #openstack-nova00:51
*** brinzhang has quit IRC00:54
*** bbowen has joined #openstack-nova00:58
*** larainema has joined #openstack-nova01:00
*** bhagyashris has joined #openstack-nova01:11
openstackgerritBrin Zhang proposed openstack/nova master: Specify availability_zone to unshelve  https://review.opendev.org/66385101:17
*** tbachman has quit IRC01:24
*** tbachman has joined #openstack-nova01:26
*** sapd1_x has joined #openstack-nova01:30
*** gbarros has joined #openstack-nova01:42
*** mmethot has joined #openstack-nova01:57
openstackgerritBhagyashri Shewale proposed openstack/nova master: Ignore root_gb for BFV in simple tenant usage API  https://review.opendev.org/61262601:58
*** gbarros has quit IRC02:05
openstackgerritMerged openstack/nova master: Add docstring to check_availability_zone function  https://review.opendev.org/67785802:11
*** dave-mccowan has joined #openstack-nova02:35
*** markvoelker has joined #openstack-nova02:40
*** boxiang has quit IRC02:44
*** zhubx has joined #openstack-nova02:44
*** markvoelker has quit IRC02:46
*** psachin has joined #openstack-nova03:02
*** KeithMnemonic1 has quit IRC03:06
*** KeithMnemonic has joined #openstack-nova03:12
*** markvoelker has joined #openstack-nova03:20
*** markvoelker has quit IRC03:25
openstackgerritGuo Jingyu proposed openstack/nova master: Define and catch InstanceAgentNotRunning for set-password failure  https://review.opendev.org/66146603:30
openstackgerritzhufl proposed openstack/nova master: [Trivial]Remove used helper get_vif_devname_with_prefix  https://review.opendev.org/67813603:38
openstackgerritBrin Zhang proposed openstack/nova master: Specify availability_zone to unshelve  https://review.opendev.org/66385103:43
*** mlavalle has quit IRC03:47
openstackgerritMerged openstack/nova master: objects: Rename 'nova.objects.instance_numa_topology'  https://review.opendev.org/67178903:54
openstackgerritMerged openstack/nova master: libvirt: Remove unnecessary try-catch around 'getCPUMap'  https://review.opendev.org/67179003:55
openstackgerritMerged openstack/nova master: libvirt: Remove unnecessary argument  https://review.opendev.org/67439603:55
openstackgerritMerged openstack/nova master: claims: Remove useless caching  https://review.opendev.org/67179103:55
*** ricolin has quit IRC04:14
*** ricolin has joined #openstack-nova04:21
*** udesale has joined #openstack-nova04:23
*** mkrai has joined #openstack-nova04:33
*** ratailor has joined #openstack-nova04:34
*** markvoelker has joined #openstack-nova04:40
*** markvoelker has quit IRC04:45
*** ash2307 has left #openstack-nova04:53
*** jaosorior has joined #openstack-nova05:01
*** ricolin has quit IRC05:02
*** ricolin has joined #openstack-nova05:03
*** janki has joined #openstack-nova05:08
*** dave-mccowan has quit IRC05:16
*** adriant has joined #openstack-nova05:23
*** tetsuro has joined #openstack-nova05:27
*** tetsuro has quit IRC05:27
openstackgerritzhufl proposed openstack/nova master: [Trivial]Remove unused helper get_vif_devname_with_prefix  https://review.opendev.org/67813605:40
*** udesale has quit IRC05:55
*** udesale has joined #openstack-nova05:55
*** boxiang has joined #openstack-nova05:56
*** boxiang has quit IRC05:57
*** boxiang has joined #openstack-nova05:57
*** zhubx has quit IRC05:58
*** ociuhandu has joined #openstack-nova06:03
*** udesale has quit IRC06:06
*** ociuhandu has quit IRC06:07
openstackgerritYongli He proposed openstack/nova master: Add server sub-resource topology API  https://review.opendev.org/62147606:16
openstackgerritYongli He proposed openstack/nova master: Add server sub-resource topology API  https://review.opendev.org/62147606:19
*** markvoelker has joined #openstack-nova06:40
*** markvoelker has quit IRC06:45
*** ash2307 has joined #openstack-nova06:47
*** shilpasd has joined #openstack-nova06:56
*** jawad_axd has joined #openstack-nova07:07
*** takamatsu has joined #openstack-nova07:08
*** ricolin has quit IRC07:15
*** rcernin has quit IRC07:15
*** trident has quit IRC07:25
*** lpetrut has joined #openstack-nova07:29
*** trident has joined #openstack-nova07:31
*** sapd1_ has joined #openstack-nova07:33
*** N3l1x has quit IRC07:34
*** ash2307 has left #openstack-nova07:36
*** xek has joined #openstack-nova07:37
*** sapd1 has quit IRC07:37
*** AJaeger has joined #openstack-nova07:40
AJaegernova stable cores, could you review https://review.opendev.org/677178, https://review.opendev.org/677176 https://review.opendev.org/677177 ? Those remove experimental openSUSE 42.3 jobs since the distro is end-of-life now.07:41
lyarwoodAJaeger: ack, I'll take a look now07:43
*** ociuhandu has joined #openstack-nova07:44
lyarwoodAJaeger: nothing on stable/stein?07:45
AJaegerlyarwood: no, that one uses the newer opensuse-150 job already07:45
AJaegerthanks, lyarwood07:45
*** ociuhandu has quit IRC07:46
lyarwoodIb0b49fead613d06001df348831c022edfe8d0fd1 - got it, thanks07:46
*** ralonsoh has joined #openstack-nova07:57
*** takashin has quit IRC08:06
*** dtantsur|afk is now known as dtantsur08:10
*** rpittau|afk is now known as rpittau08:14
*** jangutter has quit IRC08:15
*** tkajinam has quit IRC08:19
*** ociuhandu has joined #openstack-nova08:20
*** ociuhandu has quit IRC08:25
*** ivve has joined #openstack-nova08:26
*** lpetrut has quit IRC08:31
*** shilpasd has quit IRC08:33
*** jangutter has joined #openstack-nova08:33
*** mdbooth has joined #openstack-nova08:35
*** mkrai has quit IRC08:39
*** mkrai has joined #openstack-nova08:43
openstackgerritYongli He proposed openstack/nova master: Add server sub-resource topology API  https://review.opendev.org/62147608:47
*** sapd1_x has quit IRC08:47
*** janki has quit IRC08:48
cervignihello08:51
cervigniI am playing around with vgpu08:51
*** mdbooth has quit IRC08:52
cervigniIt seems that the nova scheduler does not assign properly the cpu-> gpu08:52
cervigniessentially the first vgpu that is always created by the scheduler, goes to my 2nd gpu08:52
cervigniwhile the cores are assigned to first cpu (first numa node)08:52
cervigniis there a way to tweak the scheduling algorithm or to specify somehow that there is an affinity between CPU and VGPUs ?08:53
cervignias it stands, instances always get first cpus / second gpu08:53
cervignitherefore there is a big numa issue and performance penality where instances are communicating with the "opposite" numa node in the other cpu socket08:54
*** mdbooth has joined #openstack-nova08:54
cervigniit seems to me a real nova scheduling issue, but I would like someone to confirm that is the case before opening a bug report08:56
openstackgerritMitsuhiro Tanino proposed openstack/nova master: Smaller project quota value than user quota value can't be set  https://review.opendev.org/67818308:56
*** shilpasd has joined #openstack-nova08:57
cervigniGPU 0 has a pci id lower than GPU 1, but for reasons I don't understand, the VGPUs are always first assigned to GPU108:57
cervigni|===============================+======================+======================|08:57
cervigni|   0  Tesla V100-PCIE...  On   | 00000000:37:00.0 Off |                  Off |08:57
cervigni| N/A   38C    P0    27W / 250W |     39MiB / 16383MiB |      0%      Default |08:57
cervigni+-------------------------------+----------------------+----------------------+08:57
cervigni|   1  Tesla V100-PCIE...  On   | 00000000:86:00.0 Off |                  Off |08:57
cervigni| N/A   35C    P0    29W / 250W |     51MiB / 16383MiB |      0%      Default |08:57
cervigni+-------------------------------+----------------------+----------------------+08:57
*** mdbooth has quit IRC08:57
*** mdbooth has joined #openstack-nova08:58
openstackgerritMitsuhiro Tanino proposed openstack/nova master: Smaller project quota value than user quota value can't be set  https://review.opendev.org/67818308:59
*** mdbooth has quit IRC08:59
*** mdbooth has joined #openstack-nova09:00
openstackgerritMitsuhiro Tanino proposed openstack/nova master: Smaller project quota than user quota shouldn't be set  https://review.opendev.org/67818309:05
*** owalsh is now known as owalsh|away09:16
*** cervigni has quit IRC09:27
*** ociuhandu has joined #openstack-nova09:30
*** ociuhandu has quit IRC09:34
*** brinzhang_ has quit IRC09:46
*** ociuhandu has joined #openstack-nova09:47
openstackgerritBhagyashri Shewale proposed openstack/nova master: Ignore root_gb for BFV in simple tenant usage API  https://review.opendev.org/61262609:53
*** psachin has quit IRC10:04
*** psachin has joined #openstack-nova10:06
*** maciejjozefczyk has joined #openstack-nova10:06
*** maciejjozefczyk has quit IRC10:07
*** ociuhandu has quit IRC10:10
*** jaosorior has quit IRC10:10
*** roukoswarf has quit IRC10:24
*** rouk has joined #openstack-nova10:24
*** mrjk has quit IRC10:27
*** jaosorior has joined #openstack-nova10:27
*** bhagyashris has quit IRC10:30
*** ociuhandu has joined #openstack-nova10:43
*** mkrai_ has joined #openstack-nova10:44
*** mkrai has quit IRC10:47
*** stakeda has quit IRC10:47
*** mkrai__ has joined #openstack-nova10:48
stephenfinalex_xu: You should probably look at https://review.opendev.org/#/c/674894/ given your comments at https://review.opendev.org/#/c/664420/42/nova/scheduler/request_filter.py@20610:48
*** mkrai_ has quit IRC10:52
*** mkrai__ has quit IRC10:55
*** lpetrut has joined #openstack-nova10:56
*** zhubx has joined #openstack-nova10:59
*** zhubx has quit IRC11:01
*** zhubx has joined #openstack-nova11:02
*** boxiang has quit IRC11:03
*** tesseract has joined #openstack-nova11:12
*** bauzas has quit IRC11:18
*** bauzas has joined #openstack-nova11:18
efriedsean-k-mooney: I agree. I skimmed over your review last night, sounds like what I was expecting.11:18
sean-k-mooneyefried: i might try and submit a basic patch to show how to do it as i suggested instead11:19
sean-k-mooneyefried: i assume we are talking about cyborg privsep11:19
efriedyes11:19
sean-k-mooneyi think i was a little harsh in the review but i also have strong feeling about this topic.11:20
*** jaosorior has quit IRC11:26
efriedSwhy I was hoping you would look at it.11:26
*** tbachman has quit IRC11:35
*** hemna has joined #openstack-nova11:40
*** jaosorior has joined #openstack-nova11:43
sean-k-mooneyefried: well i review in cyborg octionally and is mostly negitve when i do. not alway but my +/- raition there is proably not that positive. so i do try to make sure my review that i do there are at least constructive rather then saying "no this is all terible" and suggest how it can be improved. but if i do my usuall 50 billion comments on each indiviaual issue that can come across badly if in  a11:47
sean-k-mooney"drive by" review.11:47
openstackgerritEric Fried proposed openstack/nova master: Use SDK for setting instance id  https://review.opendev.org/65969011:53
openstackgerritEric Fried proposed openstack/nova master: Use SDK for add/remove instance info from node  https://review.opendev.org/65969111:53
openstackgerritEric Fried proposed openstack/nova master: Use SDK for getting network metadata from node  https://review.opendev.org/67021311:53
efriedsean-k-mooney: Any update on that libvirt attribute ordering bug? Not sure if I missed something going past...11:54
*** markvoelker has joined #openstack-nova11:57
sean-k-mooneyefried: its still on my todo list. i was dealing with a downstream backport the last day or two but i should get to it today or monday.11:58
efriedcool, thanks. It's a "requirements freeze" thing fwiw.11:58
sean-k-mooneyoh ya when is that next week?11:59
sean-k-mooneyi was aware of that but i proably should expidite looking at it11:59
sean-k-mooneynon-client lib frezee shoudl m3 -2 weeks11:59
efriedoh, I thought it was, but no, it's the week of Sept 911:59
sean-k-mooneyand oslo shoudl be m3 -312:00
sean-k-mooneyi guess requirement freeze is m3-112:00
sean-k-mooneyactully it looks like its m312:01
sean-k-mooneyso september 12th12:01
sean-k-mooneybut ill try and repoduce anyway this/next week and get a patch up so we dont run up against it12:01
sean-k-mooneyis this a shorter release by the way. it feels a little compressed. the freeze dates are all 1 week later then i expect in my head12:03
sean-k-mooneymaybe im just used to makeing sure everythin is in os-vif on time and i naturally try to have the last week to make sure we dont have any bugs before the freeze12:04
*** dave-mccowan has joined #openstack-nova12:06
*** larainema has quit IRC12:09
*** tbachman has joined #openstack-nova12:35
*** tbachman has quit IRC12:38
*** davee_ has joined #openstack-nova12:40
*** tbachman has joined #openstack-nova12:40
*** spatel has joined #openstack-nova12:43
*** ociuhandu has quit IRC12:44
*** dougsz has joined #openstack-nova12:47
*** udesale has joined #openstack-nova12:48
*** spatel has quit IRC12:48
dougsznova-manage db online_data_migrations <- Do I need to run that per cell, or is that a top level only thing? Ref: https://docs.openstack.org/nova/latest/user/upgrade.html12:49
AJaegernova stable cores, could I get a second +2 on https://review.opendev.org/677178, https://review.opendev.org/677176 https://review.opendev.org/677177, please ? Those remove experimental openSUSE 42.3 jobs since the distro is end-of-life now.12:51
*** Bidwe_jay has joined #openstack-nova12:52
sean-k-mooneyAJaeger: do we have a replacement suse job either based on tumbelweed or a more recent leap release12:54
sean-k-mooneyAJaeger: or rather do we have opensuse node sets so a replacment job could be built if needed12:55
AJaegersean-k-mooney: yes, in master and stein already12:57
*** gbarros has joined #openstack-nova12:57
sean-k-mooneycool12:57
AJaegerbut I didn't want to add new distro support to old stable branches that only have an experimental job12:57
*** aarents has joined #openstack-nova12:57
sean-k-mooneyya makes sense12:58
AJaegersean-k-mooney: there's job tempest-full-py3-opensuse150:12:58
AJaeger(still experimental) in master and stein12:58
sean-k-mooneyim not sure experimental jobs add much value on stable as they will be used even less often then master. but its still better then nothing12:59
*** roukoswarf has joined #openstack-nova12:59
*** rouk has quit IRC13:00
sean-k-mooneyor in other words removing an experimal job form stable that is on an unsupproted os/is broken is pretty safe13:01
AJaegeragreed13:02
*** ratailor has quit IRC13:04
*** eharney has joined #openstack-nova13:12
*** jaosorior has quit IRC13:14
*** spatel has joined #openstack-nova13:14
*** ociuhandu has joined #openstack-nova13:14
*** spatel has quit IRC13:17
*** ociuhandu has quit IRC13:19
*** ivve has quit IRC13:24
*** BjoernT has joined #openstack-nova13:27
*** zhubx has quit IRC13:31
*** zhubx has joined #openstack-nova13:31
*** bnemec has joined #openstack-nova13:34
*** bnemec is now known as beekneemech13:35
*** mriedem has joined #openstack-nova13:37
*** artom has joined #openstack-nova13:40
*** ociuhandu has joined #openstack-nova13:41
*** psachin has quit IRC13:47
*** jawad_axd has quit IRC13:49
*** jawad_axd has joined #openstack-nova13:53
*** ociuhandu has quit IRC13:54
openstackgerritBalazs Gibizer proposed openstack/nova master: Do not log token in the novncproxy logs  https://review.opendev.org/67823413:57
*** jawad_axd has quit IRC13:58
*** ociuhandu has joined #openstack-nova14:00
*** tbachman_ has joined #openstack-nova14:01
*** tbachman has quit IRC14:05
*** tbachman_ is now known as tbachman14:05
openstackgerritEric Fried proposed openstack/nova master: Update SDK fixture for openstacksdk 0.35.0  https://review.opendev.org/67823714:05
openstackgerritMatt Riedemann proposed openstack/nova master: Process [compute] in $NOVA_CPU_CONF in nova-next  https://review.opendev.org/67280014:11
alex_xustephenfin: nice14:11
alex_xumriedem: I replied your email, let me know what can make you have more confidence on this14:13
dansmithartom: you've got a bunch of legit unit test fails14:15
artomdansmith, yeah, those have been fixed locally14:16
dansmithartom: I'm also concerned about the validity of some of the other tests that aren't failing even though you're using the wrong dst_numa_config in some of the code and/or tests14:16
dansmithartom: okay cool, I just lightly dusted the bottom several patches with more comments14:16
artomdansmith, I think that's because I was asserting obj.dst_numa_config and setting it as well14:16
dansmithup to the one you said was the top of the things you've updated14:16
artomSo the code sets dst_numa_config, and the test checks it as well14:17
dansmithartom: okay, I guess just setting a non-field property on the object, okay14:17
artomdansmith, yeah14:17
dansmithwell, let's make sure to get all those right :D14:17
artomdansmith, so you're saying I need to squash the RPC changes into the top patch?14:17
dansmithI guess no real testing was done with the updated set since the top weren't updated anyway14:17
artomI split them to keep patch size as small as possible14:17
artomBut looks like it'll be unavoidable here :(14:18
artomdansmith, yeah, grep took care of that14:18
dansmithartom: squash the rpc one with the top one because of the version/capability advertisement?14:18
artomdansmith, yeah14:18
dansmiththat's probably too obtuse of a solution.. I mean it'll work I guess, but hopefully we can split them a little more than that14:19
dansmithflipping their order is probably better14:19
artomOhhh14:20
dansmithget the code in place, just don't allow it to be called from RPC until the second one, where you actually bump the versions and add the params14:20
dansmithand remove the extra service version bump of course14:20
dansmithamirite?14:20
*** dpawlik has quit IRC14:20
artomMakes sense, though I need to think through how that works for params14:20
dansmithit'll require some tweaking of things I'm sure, but that general ordering should work for you I think14:21
dansmithartom: I also think you probably need to maintain a -W on the bottom patch, if you're going to have it in various states of not-really-working (or I can -2 it)14:21
artomIt's that sort of expected as the series goes through review?14:21
dansmithartom: since it's going to mostly require a manual verification (which right now would fail hard), we probably want to wait to land much of it until it's actually proven to be working14:22
artomBut I can do that, sure14:22
artomAh, in that sense14:22
artomYeah, makes sense14:22
dansmithjust for someone not really paying attention,14:22
dansmithbecause your bottom few patches look devilishly small14:22
artomI'm devious like that :D14:23
artomActually the func test on top is only failing because I explicitly added an assertFalse because I wanted to see the logging14:23
dansmiththe claim one wouldn't be terrible to land in isolation, but the next one is an object schema commitment, which is still not ready to land, so...14:23
artomI can remove that so we can get at least some confidence in the gate14:24
dansmithokay I'm not sure how that can be the case with you using the wrong field names in the object,14:24
dansmithunless it's really that fake14:24
artomdansmith, well ok, it *was* failing because of the assertFalse14:24
dansmithoh okay I see14:24
artom*now* it might be broken for real :D14:24
dansmithI hope it is :)14:24
dansmithotherwise it doesn't do much for confidence :)14:24
artomdansmith, yeah: "AttributeError: 'module' object has no attribute 'NUMAHostInfo'"14:25
efriedartom: qq: Est-ce que c'est vrai que, si je dis, "on ne permet pas l'un ou l'autre," même si tu connais ce que je veux dire, ce n'est pas strictement correcte? Il faudrait dire, "on ne permet pas ni l'un ni l'autre"?14:27
artomefried, on ne permet ni l'un ni l'autre14:27
efriedsorry, yah14:27
artomOu même "aucun des deux n'est permis"14:28
dansmithartom: uh, that's not the one I was worried about, so maybe there's another problem?14:28
artomdansmith, it probably just hit that one first14:28
dansmithpoint is, I didn't catch that one14:28
*** mlavalle has joined #openstack-nova14:29
artomdansmith, oh, wait, that's from an unrelated stephenfin commit14:29
artomdansmith, tell you want, before addressing any other feedback, I'll un-WIP the func test14:29
artom*what14:30
dansmithun-WIP and fix?14:30
artomYeah, fix was implied :)14:30
dansmithjuuuuust checking :)14:30
artomHaha, there are limits to my lazyness ;)14:30
dansmithwell, you can see my level of confidence in your laziness I guess14:30
artom;_;14:31
dansmithefried: are you a candidate reviewer for this numa live migration stuff? seems like it's probably somewhat in your wheelhouse and sphere of interest14:32
dougszIn cells v2, do you run online_data_migrations only when *all* cells are upgraded?14:32
dansmithdougsz: you can run them against individual cells as they're upgraded or afterwards against all of them14:32
*** davee_ has quit IRC14:33
AJaegerefried, so are fine with https://review.opendev.org/677178, https://review.opendev.org/677176 and https://review.opendev.org/677177, please ?14:33
efrieddansmith: sphere of interest, as dictated by my employer, yes. Wheelhouse, not in the slightest. I've been hoping people who actually know wtf they're looking at will do the dirty work here.14:34
efriedAJaeger: I'm not stable. dansmith might be able to push those for you, though.14:35
dansmithefried: um, it's numa and little bit of rpc14:35
dansmithefried: but I'm trying to make sure it's clean and then will have to look for another reviewer to +W and figured that might be you14:35
dansmithefried: it's really not very complicated.. much less so than I originally thought when discussing the meta details14:35
dougszdansmith: thanks - so to upgrade one cell, i run online_data_migrations in the cell, and then again at the top level?14:35
dansmithdougsz: "run again at the top level" meaning "run again against all cells once they're all updated" ?14:36
efrieddansmith: I will dig in if I need to, but I'm still way low on the learning curve of both numa and rpc.14:36
mriedemalex_xu: ack, and replied14:36
dansmithefried: opportunity? :)14:36
mriedemalex_xu: tl;dr do whatever you want14:37
efriedyeah14:37
dansmithheh14:37
alex_xuI want to sleep now :)14:38
artomefried, FWIW, the crux of it is converting parts of https://libvirt.org/formatdomain.html#elementsNUMATuning and https://libvirt.org/formatdomain.html#elementsCPUTuning to objects and back14:39
artomSo it's not touching any of the nasty code in nova/virt/hardware.py14:39
artomOr just get alex_xu to do it, if he's more up to speed on the libvirt'y things ;)14:40
mriedemdansmith: i'm likely your other core for numa live migration14:40
AJaegerefried: ah, thanks14:40
dansmithmriedem: I didn't think you'd want to14:41
dansmithmriedem: and/or I didn't want to know what pound of flesh you'd want in return :D14:41
mriedemi've gone through it before...in rocky probably at tihs point14:41
dansmithyeah, I know14:41
mriedemand i'm more familiar with the live migration and rt claims flows that many14:41
mriedem*than14:41
dansmithI know, but .. see above :)14:41
mriedemhowever, i also have confidence issues with it14:42
mriedemso i've been waiting for another core to dig in first, root out the obvious stuff, and someone (intel/windriver) to come along and say they've actually tested it in a lab14:42
openstackgerritBalazs Gibizer proposed openstack/nova master: Mask the token used to allow access to consoles  https://review.opendev.org/22062214:42
dansmithwell, I'm doing the former14:43
mriedemyeah,14:43
dansmithand I think it's just going to be artom's hand testing14:43
mriedemand i think as of last friday sean-k-mooney said in his free time at home when he shouldn't be on his own personal hardware was trying to do the latter14:43
dansmithat least initially14:43
dansmithor that14:43
artommriedem, Dean told me he's brought it up the chain to try and get resources to test it, but not sure that's going to pan out14:43
mriedemi know intel is strapped for cash and resources these days14:43
mriedemit's sad14:43
mriedemsad folks14:44
alex_xumriedem: thanks anyway, sorry for pushing hard14:44
sean-k-mooneyreads back14:44
dansmithit's not going to happen in time to land this either I'm sure14:44
dansmithlol14:44
dougszdansmith: maybe I'm confused. If I run `nova-manage db online_data_migrations` using the cell conductor nova.conf, I update only the cell, if I run `nova-manage db online_data_migrations` using the top level nova.conf, I upgrade all cells?14:44
dansmithdougsz: no, hence my question ;)14:44
artomFWIW, I do have that whitebox tempest plugin code14:44
artomIt's nowhere upstream yet, but that's next on my list14:44
dansmithdougsz: the former per cell is how it works, which is why I asked why you'd run again "at the top level"14:44
alex_xuartom: emm..not sure what is it yet14:45
artomMove whitebox under openstack-qa, find hardware on which to run it. Fort Nebula is a good candidate14:45
sean-k-mooneyi do plan to test artoms stuff next week. i need to resintall the os on those servers. but i have already set up one  of them14:45
AJaegerdansmith: could you put the following three on your review list, please? https://review.opendev.org/677178, https://review.opendev.org/677176 and https://review.opendev.org/677177 all remove an experimental opensuse job for an eol distro from stable branches14:45
mriedemdougsz: iow, you have to run that command per cell14:45
*** jawad_axd has joined #openstack-nova14:46
dansmithmriedem: he said he used the cell conductor config, which is the per-cell bit, so I think he gets that14:46
mriedemok14:46
dansmithI guess we do have some migrations in the api database,14:46
dansmithso I guess running against the api config is also necessary, I dunno why I wasn't thinking about that initially14:47
dansmithdougsz: ^14:47
artomalex_xu, NUMA live migration, you looked at it a bit before, and there's some overlap with your vpmem work I think. dansmith and I are hunting for a second non-RH core14:48
dougszdansmith, mriedem: thanks, I think that makes sense. So per cell, *and* at the top level to do it everywhere.14:48
mriedem"to do it everywhere" isn't a thing14:48
mriedemthe command doesn't broadcast across all cells14:48
dansmithright,14:48
dansmiththat's the part I was taking exception with14:48
mriedemdan is just saying there are some migrations in the API DB which is "at the top"14:49
mriedem"top of the topology" iow14:49
alex_xuartom: yea, i can onboard the review, probably need one or two day, rampup your patches14:49
artomalex_xu, thank you, appreciated :)14:49
dougszmriedem: Ok, thanks, and I trigger those top level migrations via nova-manage api_db sync. I think I get it14:50
alex_xuartom: btw, I don't think it will overlap with vpmem anymore, but will check again14:50
openstackgerritEric Fried proposed openstack/nova master: Update SDK fixture for openstacksdk 0.35.0  https://review.opendev.org/67823714:50
efriedmriedem: more sdk turducken :(  ^14:50
mriedemdougsz: online_data_migrations is not the same as the sync command14:50
mriedemsync == upgrade db table schemas14:51
*** jawad_axd has quit IRC14:51
mriedemonline_data_migrations == migrate existing data in the table to some new format if necessary14:51
dougszgot it14:51
*** shilpasd has quit IRC14:53
dougszfinal question, If I upgrade a single cell, to get it to move to latest RPC version, I need to SIG-HUP the top level Nova services as well?14:53
*** lpetrut has quit IRC14:54
mriedemAJaeger: i got them14:54
efriedSIGHUP is still broken, until https://review.opendev.org/#/c/641907/14:54
AJaegerthanks, mriedem !14:55
*** ociuhandu has quit IRC14:55
dougszyeah, just thinking, I upgrade one cell, I need to restart all top level services because SIG-HUP doesn't work14:55
mriedemdougsz: well, you should be pinning the rpc upgrade level for the computes at the top level so when you upgrade your computes in the cell, the top level (api/conductor) isn't sending messages the computes won't understand14:55
mriedemdougsz: are you updating the control plane services first or last?14:56
dougszcontrol plane first14:56
efriedstephenfin: I'm waiting for a new push of the cpu-resources series, right?14:56
mriedemhttps://docs.openstack.org/nova/latest/user/upgrade.html14:56
stephenfinyup14:56
mriedemdougsz: https://docs.openstack.org/nova/latest/user/upgrade.html14:56
stephenfinworking on it as we speak14:56
dougszyeah, trying to follow that in a cells context14:56
efriedstephenfin: I'm hoping that if I read the code once a day for a week, I'll understand it.14:56
efriedCurrently it's you making changes I don't understand to code I don't understand.14:57
mriedemdougsz: cellsv2 yeah?14:57
dougszyeah14:57
mriedemdougsz: what services are running in the cells? just nova-compute?14:57
*** xek has quit IRC14:58
dougsz+ nova-ssh, nova-conductor14:58
mriedemidk what nova-ssh is14:58
mriedemanyway, if you upgrade the control plane first, you want to make sure those are configured for [upgrade_levels]/compute (either auto or whatever your n-1 release you're upgrading from is)14:59
mriedemthat way the controller services won't send N version messages to N-1 services14:59
mriedemi.e. pin the rpc versions to the lowest common version, roll through upgrades, and then unpin14:59
dougszok, thanks  mriedem, that bit makes sense. I will go and have a play15:00
*** dpawlik has joined #openstack-nova15:00
*** AJaeger has left #openstack-nova15:00
*** tbachman has quit IRC15:05
*** jawad_axd has joined #openstack-nova15:11
openstackgerritBalazs Gibizer proposed openstack/nova master: allow getting resource request of every bound ports of an instance  https://review.opendev.org/65511015:12
*** tbachman has joined #openstack-nova15:12
openstackgerritBalazs Gibizer proposed openstack/nova master: Pass network API to the conducor's MigrationTask  https://review.opendev.org/65511115:14
*** jawad_axd has quit IRC15:16
dansmithefried: sighup still does the thing he needs, just inefficiently, IIRC15:16
openstackgerritBalazs Gibizer proposed openstack/nova master: Add request_spec to server move RPC calls  https://review.opendev.org/65572115:16
dansmithefried: it effectively restarts the whole process, which is good enough, just too heavy15:16
efrieddansmith: I thought we decided SIGHUP also breaks privsep15:17
dansmithoh does it?15:17
efriedsomething about either restarting or not restarting the privsep daemon...15:18
dougszhttps://github.com/openstack/kolla-ansible/blob/master/ansible/roles/nova/handlers/main.yml#L21115:18
efriednow I can't remember where I saw it.15:18
dougszdansmith efried ^ That's the issue afaik15:18
dansmithnot restarting is fine, as long as it doesn't lose a reference to the pipe, which I thought was global15:18
*** damien_r has quit IRC15:18
*** tbachman has quit IRC15:18
dansmithdougsz: oh, right, right, ignore me15:19
dansmithI forgot the rabbit hole went deeper15:19
openstackgerritBalazs Gibizer proposed openstack/nova master: re-calculate provider mapping during migration  https://review.opendev.org/65511215:19
dougsznp!15:19
dansmiththe simple case is that it does a full restart for no reason, but I forgot it also breaks some state as a result15:19
*** dave-mccowan has quit IRC15:20
efriedsomeone stephenfin with oslo-core stephenfin should merge that patch15:20
stephenfinwhuh?15:21
efriedhttps://review.opendev.org/#/c/641907/15:21
openstackgerritBalazs Gibizer proposed openstack/nova master: update allocation in binding profile during migrate  https://review.opendev.org/65642215:21
stephenfinI'm going to finish this cpu-resources reshuffle, but I can take a look after that15:22
efriedcool. No hurry, clearly. It's been open since like rocky.15:22
efriedsure would be nice to stop having to tell people SIGHUP is broken tho.15:22
beekneemechIf you figure it out you'll be ahead of the rest of us. :-)15:22
efried(oh, did I trigger a keyword?)15:23
openstackgerritMatt Riedemann proposed openstack/nova stable/stein: doc: remove confusing docs about aggregate allocation ratios  https://review.opendev.org/67825415:23
beekneemechYeah, I added an oslo highlight after the Denver PTL discussion.15:23
openstackgerritBalazs Gibizer proposed openstack/nova master: Extend NeutronFixture to handle migrations  https://review.opendev.org/65511415:24
dansmithbeekneemech: efried: I think it might be worth testing that with a real nova15:24
dansmithbefore just firing it in15:24
dansmithhas anyone done that?15:24
mriedemcpu-resources reshuffle is the worst dance move yet15:24
efrieddansmith: I ripped it in a devstack, does that count?15:24
dansmithefried: does that mean "tested it" ?15:24
mriedemit means he farted i think15:25
efriedI didn't try to hit the "stuff that actually breaks" paths cause I didn't understand them well enough15:25
dansmiththen...no15:25
efriedbut I watched logs through SIGHUP before and after15:25
efriedthat analysis is in the patch15:25
openstackgerritBalazs Gibizer proposed openstack/nova master: prepare func test env for moving servers with bandwidth  https://review.opendev.org/65510915:26
dansmithyou should be able to validate that before the patch SIGHUP breaks booting instances, and after it doesn't right?15:26
*** macz has joined #openstack-nova15:26
efrieddansmith: I'm updating my devstack now, then will try that.15:27
mriedemdue to not processing the vif plug event right? you might be waiting 5 minutes if you don't tweak the config15:28
mriedemhttps://docs.openstack.org/nova/latest/configuration/config.html#DEFAULT.vif_plugging_timeout15:28
mriedemi'd drop that to 60 seconds15:28
openstackgerritEric Fried proposed openstack/nova master: Add emulated TPM support to Nova  https://review.opendev.org/63136315:28
openstackgerritEric Fried proposed openstack/nova master: Add support for resize and cold migration of emulated TPM files  https://review.opendev.org/63993415:28
openstackgerritBalazs Gibizer proposed openstack/nova master: Func test for migrate server with ports having resource request  https://review.opendev.org/65511315:28
*** CeeMac has joined #openstack-nova15:28
CeeMacafternoon15:28
CeeMacis there a way to configure nova to use cinder volumes for ephemeral storage and root disks?15:29
mriedemnope15:29
CeeMacthat answers that then15:29
mriedemit's come up many a time15:29
CeeMacso, the only option for shared storage backend is to mount the instance disk path to NFS?15:30
mriedemyou're talking about different things i think15:30
CeeMacits entirely possible15:30
openstackgerritBalazs Gibizer proposed openstack/nova master: Make _rever_allocation nested allocation aware  https://review.opendev.org/67613815:31
mriedemthe instance path files are not the same as the root disk - you can have the root disks in rbd but have the instance files (console log, config drive, etc) be on local disk on the compute node15:31
mriedemit gets confusing15:31
CeeMaci don't use ceph, if that helps15:31
dansmithartom: so, what do you think the timetable is for a respin of the numalm set? you said you've got unit tests fixed locally already, so can you do some swizzling of the top two patches and push that up today to get a run started?15:31
mriedemor you could have the instance path files on local disk or shared NFS/rbd, but the root disk is actually a volume in cinder15:31
CeeMaci was thinking in terms of when an instance is launched without specifying to use a cinder volume15:32
mriedemCeeMac: sure, just saying that's a common shared storage backend15:32
artomdansmith, working on updating the func test currently, since confidence seems to be the big thing15:32
CeeMacmriedem: got you, yeah its all very confusing.15:32
*** rpittau is now known as rpittau|afk15:32
mriedemCeeMac: assuming you're using libvirt, the image backend for the root disk is defined with this config https://docs.openstack.org/nova/latest/configuration/config.html#libvirt.images_type15:33
mriedemso you're probably using raw/flat/qcow2 yeah?15:33
openstackgerritBalazs Gibizer proposed openstack/nova master: Support reverting migration / resize with bandwidth  https://review.opendev.org/67614015:33
CeeMacqcow215:33
CeeMacbut i back images off onto volume, so it converts it raw iirc15:33
mriedemhttps://docs.openstack.org/nova/latest/configuration/config.html#DEFAULT.instances_path controls where the instance files are going to be then15:34
CeeMacright15:35
CeeMacthats what I was trying to get at i think15:35
openstackgerritBalazs Gibizer proposed openstack/nova master: Func test for migrate re-schedule with bandwidth  https://review.opendev.org/67697215:35
CeeMacwhen an instance is launched without specifying to use volume backing for root disk, the root disk will go there too by default?15:36
CeeMaci.e. not through horizon15:36
mriedemhorizon doesn't have anything to do with it (unless you mean horizon is somehow forcing you to use volume-backed servers)15:36
mriedembut yeah15:36
mriedemhttps://review.opendev.org/#/c/511965/ has some background details on the whole "configure nova to do volume-backed servers by default" thing which has had lots of discussion in the past like i said15:37
mriedemmany attempts, many failures15:37
sean-k-mooneyhorizon default to createing a volume if cinder is enabled15:37
mriedemsean-k-mooney: ok but it's not required15:37
mriedemnor does everyone use horizon15:37
sean-k-mooneyim not really sure why but it does. its not require correct15:37
CeeMacsean-k-mooney: thats my understanding too15:38
openstackgerritBalazs Gibizer proposed openstack/nova master: Support migrating SRIOV port with bandwidth  https://review.opendev.org/67698015:38
openstackgerritBalazs Gibizer proposed openstack/nova master: Allow migrating server with port resource request  https://review.opendev.org/67149715:38
sean-k-mooneyCeeMac: it defintly does not require it as i untick it all the time15:38
CeeMacmy issue comes when using rancher to deploy VMs, socker-volume isn't block-storage aware15:38
CeeMacI'm happy with horizon defaulting to volume, as thats where I'd prefer for the instance to go15:39
CeeMactrying to work out a way to force all instances to back off to block storage regardless of how they're requested15:39
CeeMacwas hoping there was an integrated cinder way15:39
sean-k-mooneyCeeMac: i kind of wish it default to delete the volume when you delete the vm too however15:39
CeeMacbut I'm guessing I'd need to go with mounting /var/lib/nova/instances to NFS on our SAN15:39
CeeMacwhich is where instances_path points to right now15:40
CeeMacin nova.config15:40
sean-k-mooneyCeeMac: i think we have a config option that will help15:40
sean-k-mooneyhttps://docs.openstack.org/nova/latest/configuration/config.html#DEFAULT.max_local_block_devices15:40
sean-k-mooneyif you set that to 0 you have to use cinder volumes15:40
sean-k-mooneynot sure if tha tis what you want15:40
CeeMacsean-k-mooney: that is a good start to prevent local disks being created and potentially filling up the (small) local disks on the compute15:41
sean-k-mooneythe other options open to you if you nova no/limnited local stoage and have ceph is to enable the RBD image backend15:41
CeeMacbut I haven't found a way to instruct rancher, or docker for that matter, to use volumes for images15:41
CeeMacsean-k-mooney: sadly no ceph here15:42
CeeMaciscsi backed cinder15:42
sean-k-mooneyah ok15:42
sean-k-mooneywe dont have a cinder or iscis image backend to force all instace to use cinder unfortunetly15:42
CeeMacis NFS the only/best/supported way of backing off the instance_path, or is iscsi an option?15:43
CeeMachaha, i think you answered that question then.15:43
sean-k-mooneyi think you would be stuck with nfs15:43
CeeMacyou could be right, at least for the short term until I can get/hack a workaround for rancher to be able to call for volume on instance launch15:44
CeeMacthanks for the help mriedem and sean-k-mooney appreciate it15:44
sean-k-mooneyrancher use docker to run all service rihgt15:44
sean-k-mooneyi have not looked at rahcner os much15:44
CeeMacyeah15:45
sean-k-mooneybut i think you can confugre docer to use remote sotorage15:45
CeeMacand docker-volume isn't block storage aware15:45
CeeMacyou feed it a flavour and image effectively15:45
sean-k-mooneye.g. i think you should be able to have the non system docker use a volume plugin and mont /var/libvirt/instace... or what ever on a network share15:45
*** gyee has joined #openstack-nova15:46
CeeMacif rancher was running inside openstack yeah, but we're connecting to openstack through api to a project from outside15:46
CeeMaci'll have a ponder15:47
sean-k-mooneyi know there used to be a tone of voluem plugins https://docs.docker.com/engine/extend/legacy_plugins/#volume-plugins but i dont know if any of thos worked with racner/still exists15:47
CeeMaci've seen a cinder driver for docker on github, but havent had a chance to test that yet15:48
mriedemsean-k-mooney: https://docs.openstack.org/nova/latest/configuration/config.html#DEFAULT.max_local_block_devices is not what you think it is15:49
mriedemsean-k-mooney: that's only if the bdm has destination_type='local'15:49
mriedemunless you mean for the implicit image-defined bdm15:49
sean-k-mooneyit prevent you spaning flavor tha trequest ephmeral/swap or root disk if you are using qcow15:49
mriedemsource_type='image', destination_type='local'15:49
sean-k-mooneyya i ment you can use that to ensure only bfv guest can boot on that node15:50
mriedem*any* node15:50
mriedemit's config-driven api15:50
sean-k-mooneyyeah thos are awsome... but ya15:50
sean-k-mooneyi was looking to do that in the past but since i was deploying ceph for cinder i just deploy with the rbd image backend instead15:51
*** udesale has quit IRC15:57
sean-k-mooneymriedem: im going to step away for an hour or so to have breakfast/lunch but i plan to adress your comments on https://review.opendev.org/#/c/669738/ when i get back since they have been waiting a while.15:57
*** udesale has joined #openstack-nova15:58
mriedembreakfast/lunch?15:58
mriedemit's 5pm15:58
sean-k-mooneylyarwood: mriedem if we can land https://review.opendev.org/#/c/678098/ on stable stine that would make mnaser happy15:59
sean-k-mooneyyes.. i normally eat once to twice a day. usually i try to grab something at around 315:59
*** hamzy has quit IRC15:59
sean-k-mooneyand then around 815:59
sean-k-mooneyor i jsut have one big meal around 6 and then snack on fruit/nuts/small things16:00
*** kmalloc is now known as needscoffee16:00
*** needscoffee is now known as needsSoMuchMoreC16:00
*** needsSoMuchMoreC is now known as needscoffee16:00
*** dtantsur is now known as dtantsur|afk16:00
mriedemi just got the image of you as the giant in jack and the beanstalk crushing human bones to make your bread16:02
mriedem"small things"16:02
*** jangutter has quit IRC16:03
dansmithartom: I looked at the top two patches with plans to prescribe an ordering and squash/split instructions, but I think it probably just needs to be fully squashed16:04
artomdansmith, yeah? I was hoping to avoid that, as it makes reviewing difficult16:04
dansmithI know16:04
dansmithI kept calling out lines as "this could be separate" and then removed those comments16:05
artomBut you're most likely right (duh!), as a whole bunch of things need to happen in one chunk16:05
dansmithyup, any ordering that I could come up with was dependent on a patch ahead of everything that was like "add a bunch of single-line unrelated changes to various places" followed by a bigger one that hooked them all up16:06
dansmithand it just doesn't make sense16:06
dansmithI kept having to jump back and forth between those two patches to work that out and ultimately fail, which means reviewing isn't really improved16:06
dansmithfewer lines, but not in a meaningful way, so...16:06
artomWell, at least the commit message is already in bullet points16:07
*** beekneemech has quit IRC16:07
dansmithusually that's a bad sign, but in this case, I don't really see a useful way around16:07
dansmithmriedem might disagree, but hopefully not16:07
dansmithhe likes to disagree with me periodically to make himself look smart though16:08
dansmithor to assert his smartness, maybe I should say16:08
artomOh, nice save16:08
dansmithheh16:08
mriedemhttps://www.youtube.com/watch?v=YLP1_jwT9_M16:10
dansmithheh16:11
*** tesseract has quit IRC16:11
dansmithhttps://www.youtube.com/watch?v=oGfkPCZYfFw16:11
artomHave we finally devolved to communicating via only YouTube videos?16:12
dansmithtrying to speak mriedem's language16:12
openstackgerritBalazs Gibizer proposed openstack/nova master: update allocation in binding profile during migrate  https://review.opendev.org/65642216:12
openstackgerritBalazs Gibizer proposed openstack/nova master: Extend NeutronFixture to handle migrations  https://review.opendev.org/65511416:12
*** dpawlik has quit IRC16:13
openstackgerritMatt Riedemann proposed openstack/nova master: Docs and functional test for max_local_block_devices  https://review.opendev.org/67826816:14
* mriedem goes to get bbq16:14
openstackgerritBalazs Gibizer proposed openstack/nova master: prepare func test env for moving servers with bandwidth  https://review.opendev.org/65510916:15
dougszOne other question (sorry!). If I upgrade one cell of many, run online_data_migrations in that cell only, set upgrade_levels to auto (or N-1 level) to pin RPC version, will there be a performance penalty in that cell, until all cells move to the latest RPC version?16:16
dansmithdougsz: you need to move the upgrade pin only once all the cells are updated16:17
dansmithotherwise the conductors will start speaking newer versions to the old cells16:17
dansmithalso,16:17
dougszso auto is not good enough?16:17
dansmithif you use auto, then it will always get the minimum support version, you don't need to tweak it before and after16:18
dansmithonline-data-migrations is unrelated to the rpc pin version fwiw16:18
*** tbachman has joined #openstack-nova16:19
*** TxGirlGeek has joined #openstack-nova16:19
openstackgerritBalazs Gibizer proposed openstack/nova master: Func test for migrate server with ports having resource request  https://review.opendev.org/65511316:19
openstackgerritBalazs Gibizer proposed openstack/nova master: Make _rever_allocation nested allocation aware  https://review.opendev.org/67613816:19
dougszok, thanks, that makes sense, i can upgrade a single cell and take my time checking that it doesn't catch fire before upgrading the others16:20
*** bnemec has joined #openstack-nova16:20
dansmiththeoretically yes16:21
dougsz:)16:21
dansmithI believe cern does this16:21
openstackgerritMerged openstack/nova master: trivial: Rename exception argument  https://review.opendev.org/67179516:23
openstackgerritMerged openstack/nova master: Remove 'hardware.get_host_numa_usage_from_instance'  https://review.opendev.org/67179716:23
openstackgerritMerged openstack/nova master: Remove 'hardware.host_topology_and_format_from_host'  https://review.opendev.org/67179816:23
openstackgerritMerged openstack/nova master: Remove 'hardware.instance_topology_from_instance'  https://review.opendev.org/67179916:23
openstackgerritMerged openstack/nova master: Rework 'hardware.numa_usage_from_instances'  https://review.opendev.org/67256516:23
openstackgerritBalazs Gibizer proposed openstack/nova master: Support reverting migration / resize with bandwidth  https://review.opendev.org/67614016:24
openstackgerritBalazs Gibizer proposed openstack/nova master: Func test for migrate re-schedule with bandwidth  https://review.opendev.org/67697216:27
stephenfinNice. Delighted to get rid of all those awful functions ^16:29
stephenfinThis is all going swimmingly16:29
openstackgerritBalazs Gibizer proposed openstack/nova master: Support migrating SRIOV port with bandwidth  https://review.opendev.org/67698016:29
*** dpawlik has joined #openstack-nova16:29
openstackgerritBalazs Gibizer proposed openstack/nova master: Allow migrating server with port resource request  https://review.opendev.org/67149716:31
*** dpawlik has quit IRC16:34
artomdansmith, about the is_trackable_migration removal...16:36
dansmithyar16:36
artomWe obviously need to get rid of it, because we want the resource tracker to track NUMA live migrations and update resource usage accordingly16:36
artomBut we're removing it entirely, not just for NUMA live migrations.16:37
dansmithisn't this just for reporting purposes?16:37
artomWell, resource usage as well16:37
dansmithI've mostly dumped this out of my brain because placement,16:37
artomNUMATopologyFilter still uses it16:37
dansmithbut does the scheduler actually use some of this information anymore?16:37
*** tbachman has quit IRC16:37
dansmithright, okay, the legacy things16:37
artomSo because we'll be tracking all live migrations, we need to do claims for all live migrations, not just NUMA ones16:38
artom(which is what's currently in the patch series)16:38
dansmithso right now we don't report this for live migrations, nor do we claim, so the numa filter will send us things we don't have room for, and we also won't do claims to check that it was wrong, right?16:38
artomYep16:38
*** markvoelker has quit IRC16:39
dansmithokay, but non-numa-affected live migrations are already protected and don't need anything else right?16:39
dansmithbecause we're depending on placement for the non-numa resources16:39
artomYeah, that should be fine16:39
artomI was more worried about things like PCI16:39
artomThough I guess we don't support live migrations for those at all, so it the claim fails, it'll just be a plus16:40
dansmithno live migration with pci devices yeah?16:40
dansmithright16:40
dansmithso to continue my thought above,16:40
*** tbachman has joined #openstack-nova16:40
artomThough that's completely untested16:40
dansmithI think you can save that change until the very end,16:40
dansmithbecause once we're claiming, we're doing better than we are today, and then we can start reporting them, which is another incremental improvement to make the scheduler make better decisions16:40
artomThe very end == the massive squashed patch, right?16:41
dansmithwell, wherever16:41
dansmithjust saing,16:41
dansmithit doesn't need to be synchronous with the change to claim I think16:41
artomOh, right, that's a given. I've addressed all that feedback, now working on squashing16:42
dansmithokay16:42
dansmithseems like it might be a good candidate for being its own patch though.. right?16:42
openstackgerritStephen Finucane proposed openstack/nova master: Remove 'hw:cpu_policy', 'hw:mem_page_size' extra specs from API samples  https://review.opendev.org/67533816:44
openstackgerritStephen Finucane proposed openstack/nova master: libvirt: Start reporting PCPU inventory to placement  https://review.opendev.org/67179216:44
openstackgerritStephen Finucane proposed openstack/nova master: libvirt: '_get_(v|p)cpu_total' to '_get_(v|p)cpu_available'  https://review.opendev.org/67269316:44
openstackgerritStephen Finucane proposed openstack/nova master: trivial: Rewrap definitions of 'NUMACell'  https://review.opendev.org/67439516:44
openstackgerritStephen Finucane proposed openstack/nova master: hardware: Differentiate between shared and dedicated CPUs  https://review.opendev.org/67180016:44
openstackgerritStephen Finucane proposed openstack/nova master: objects: Rename 'fields' import to 'obj_fields'  https://review.opendev.org/67410316:44
openstackgerritStephen Finucane proposed openstack/nova master: libvirt: Start reporting 'HW_CPU_HYPERTHREADING' trait  https://review.opendev.org/67557116:44
openstackgerritStephen Finucane proposed openstack/nova master: Add support for translating CPU policy extra specs, image meta  https://review.opendev.org/67180116:44
openstackgerritStephen Finucane proposed openstack/nova master: libvirt: Fold in argument to '_update_provider_tree_for_vgpu'  https://review.opendev.org/67672916:44
openstackgerritStephen Finucane proposed openstack/nova master: Add reshaper for PCPU  https://review.opendev.org/67489516:44
artomdansmith, don't think so - it depends on the RPC changes, and once we start claiming, we need all the other things as well - rollback/application of the migration context, etc16:44
dansmithum16:45
dansmithartom: the reporting stands alone, no?16:45
artomdansmith, it uses the resource tracker, which is affected by claims16:45
artomYou're making me doubt myself now16:46
dansmithartom: if we don't report (as we don't today) then the scheduler will keep making the same decisions it is today, but we can stop an operation if the claim tells us there isn't space16:46
dansmithwe don't alter the claims as part of that routine do we?16:46
openstackgerritStephen Finucane proposed openstack/nova master: libvirt: Start reporting PCPU inventory to placement  https://review.opendev.org/67179316:47
openstackgerritStephen Finucane proposed openstack/nova master: libvirt: '_get_(v|p)cpu_total' to '_get_(v|p)cpu_available'  https://review.opendev.org/67269316:47
openstackgerritStephen Finucane proposed openstack/nova master: trivial: Rewrap definitions of 'NUMACell'  https://review.opendev.org/67439516:47
openstackgerritStephen Finucane proposed openstack/nova master: hardware: Differentiate between shared and dedicated CPUs  https://review.opendev.org/67180016:47
openstackgerritStephen Finucane proposed openstack/nova master: objects: Rename 'fields' import to 'obj_fields'  https://review.opendev.org/67410316:47
openstackgerritStephen Finucane proposed openstack/nova master: libvirt: Start reporting 'HW_CPU_HYPERTHREADING' trait  https://review.opendev.org/67557116:47
openstackgerritStephen Finucane proposed openstack/nova master: Add support for translating CPU policy extra specs, image meta  https://review.opendev.org/67180116:47
openstackgerritStephen Finucane proposed openstack/nova master: libvirt: Fold in argument to '_update_provider_tree_for_vgpu'  https://review.opendev.org/67672916:47
openstackgerritStephen Finucane proposed openstack/nova master: Add reshaper for PCPU  https://review.opendev.org/67489516:47
artomdansmith, I might be misunderstanding you entirely, but reporting and claims go hand in hand - when we start claiming, we'll automatically get correct reporting16:47
*** tbachman has quit IRC16:47
*** markvoelker has joined #openstack-nova16:47
artomWell16:48
artomUnless we keep ignoreing live migration in the resource tracker16:48
stephenfinmriedem: How does one kick off tempest tests when they wouldn't normally run? https://review.opendev.org/#/c/677475/16:48
stephenfin(because irrelevant-files, I'm assuming)16:49
artomdansmith, is that what you meant? Move the "stop ignoreing live migrations in the resource tracker" to its own small patch on top?16:49
dansmithartom: yes, but, I need to re-read this claims stuff16:50
dansmithmaybe I'm making CLAIMS that are incorrect16:50
artomdansmith, ack. Sounds like a good place for me to get lunch16:50
artomdansmith, HI-OH16:50
openstackgerritStephen Finucane proposed openstack/nova master: setup.cfg: Cleanup  https://review.opendev.org/67796916:51
artomdansmith, want me to push the bottom fixes?16:51
dansmithI think it's because all the regular claim code has been removed16:51
dansmithartom: yes16:51
dansmithI thought the only bits left of the claiming stuff don't actually really interact that much with the resource tracker,16:52
dansmithand rather just see if the things being asked for exist on the compute.. meaning the pci device and the numa arrangement16:53
dansmithso I'm not sure why the reporting stuff is really related directly16:53
*** bnemec has quit IRC16:53
*** dougsz has quit IRC16:54
openstackgerritArtom Lifshitz proposed openstack/nova master: Introduce live_migration_claim()  https://review.opendev.org/63566916:54
openstackgerritArtom Lifshitz proposed openstack/nova master: New objects for NUMA live migration  https://review.opendev.org/63482716:54
openstackgerritArtom Lifshitz proposed openstack/nova master: LM: add support for augmenting migrate_data with info from claims  https://review.opendev.org/63482816:54
openstackgerritArtom Lifshitz proposed openstack/nova master: LM: add support for updating NUMA-related XML on the source  https://review.opendev.org/63522916:54
openstackgerritArtom Lifshitz proposed openstack/nova master: NUMA live migration support  https://review.opendev.org/63460616:54
openstackgerritArtom Lifshitz proposed openstack/nova master: Deprecate CONF.workarounds.enable_numa_live_migration  https://review.opendev.org/64002116:54
openstackgerritArtom Lifshitz proposed openstack/nova master: Functional test for NUMA live migration  https://review.opendev.org/67259516:54
efrieddansmith: I should only have to restart n-cpu to pick up a change in oslo.service, yah?16:54
efriedat least, the change we care about16:54
dansmithefried: if you're just testing that service then obviously :)16:55
efriedI didn't know if other services and their e.g. RPC objects would care.16:55
efrieddansmith: Then I've confirmed the oslo.service fix doesn't fix the problem.16:56
efried(which doesn't mean it's wrong, just not complete)16:56
dansmithartom: okay, I see now, I forgot how this works, but I see that if we don't remove that skip we won't do the actual checking of anything, so ignore me16:57
dansmithefried: sweet16:57
artomdansmith, yeah, it's coupling great?16:58
artom*isn't16:58
*** boxiang has joined #openstack-nova16:58
artomRight, pushed, bottom 4 should be good, now lunch16:59
*** ozzzo has joined #openstack-nova16:59
*** zhubx has quit IRC17:01
ozzzoIt looks like nova is keeping mysql connections open until they time out. Is there a way to stop these error messages?17:01
ozzzoAborted connection 10726 to db: 'nova' user: 'nova' host: 'asdf' (Got timeout reading communication packets)17:02
*** boxiang has quit IRC17:02
*** zhubx has joined #openstack-nova17:02
*** udesale has quit IRC17:05
*** udesale has joined #openstack-nova17:06
*** igordc has joined #openstack-nova17:09
*** udesale has quit IRC17:11
*** udesale has joined #openstack-nova17:12
mnaserefried: if you have a few seconds, i wanted to know if you had thoughts about https://review.opendev.org/#/c/676540/ -- should i mark it no longer as deprecated as it's clearly being used?17:16
efriedmnaser: or propose whatever alternative we would prefer. mriedem ^17:17
mnaseri mean if it's clearly being consumed, then i don't mind making the change to stop marking is an deprecated..17:18
*** tbachman has joined #openstack-nova17:22
openstackgerritMerged openstack/nova stable/rocky: Remove experimental job on openSUSE 42.3  https://review.opendev.org/67717617:25
openstackgerritMerged openstack/nova stable/queens: Remove experimental job on openSUSE 42.3  https://review.opendev.org/67717717:25
openstackgerritMerged openstack/nova stable/pike: Remove experimental job on openSUSE 42.3  https://review.opendev.org/67717817:25
efriedmriedem, dansmith: Posted details on the SIGHUP patch https://review.opendev.org/#/c/641907/17:28
efriedWe are indeed breaking the privsep pipe (/me says as if I know what I'm talking about)17:29
efriedunclear where that would need to be fixed.17:29
mriedemstephenfin: yeah irrelevant-files17:29
*** ralonsoh has quit IRC17:34
*** udesale has quit IRC17:35
*** hamzy has joined #openstack-nova17:35
sean-k-mooneystephenfin: add a followup [DNM] patch that add whitespace to a file that would trigger tempest or temporally remove one fo the fiels from irrelevent-files. i generall prefer to have a DNM testing patch as i can just run the job i care about.17:36
mriedemefried: mnaser: i think we likely should undeprecate the direct file download stuff since clearly there are users and those users have made attempts to upstream an alternative solution which hasn't been approved, but not wholesale rejected either from what i understand, so it's probably not good to just rip it out from them when we're not providing an alternative17:37
efriedwfm17:38
sean-k-mooneystephenfin: that chante will alter they way devstack install nova as nova installs all packages in requirements.txt and test-requiremnts.txt system wide by default so it should pass in the gate17:38
sean-k-mooneystephenfin: however if you enable installing in virtual envs it may not work anymore17:38
sean-k-mooneystephenfin: without a change to devstack nova would be relying on other pojects to install pymsql or psycopg2 for postgress17:39
efriedmriedem: any advice on next steps for copying mysql logs? https://review.opendev.org/#/c/678051/17:43
*** gbarros has quit IRC17:45
efriedmriedem:17:45
efriedWarning: Permanently added '166.78.116.9' (ECDSA) to the list of known hosts.17:45
efriedsudo: no tty present and no askpass program specified17:45
efriedrsync: connection unexpectedly closed (0 bytes received so far) [Receiver]17:45
efriedrsync error: error in rsync protocol data stream (code 12) at io.c(226) [Receiver=3.1.1]17:45
sean-k-mooneyi dont think you want to copy them to the executer liek that i think you want to drop them into the working directory on the vm and have the normal log copy task copy them at the end17:47
sean-k-mooneyi may be wrong about that but i think that is what i have seen done before17:48
*** cz2 has joined #openstack-nova17:48
artomsean-k-mooney, do you know if we would break anything if we started doing claims for all live migrations? I'm thinking of PCI stuff...17:49
artomPresumably claims for PCI aren't broken in and of themselves, since they're used to boot instances and for cold migrations17:50
sean-k-mooneywe do claims for pci deivces for migration already17:50
artom*live*?17:50
sean-k-mooneyyes17:50
sean-k-mooneywell for neutron sriov ports17:51
sean-k-mooneywe dont support live migration with other pci devices17:51
sean-k-mooneybut for sriov migration we added cliams17:51
sean-k-mooney*sriov live migration17:51
artomAre we talking about the same claims?17:51
sean-k-mooneyin the resouce tracker17:51
artomYeah17:52
sean-k-mooneywe do it as part of pre live migration at dest17:52
artomCould you point me to the review/code?17:52
*** tbachman has quit IRC17:52
*** TxGirlGeek has quit IRC17:52
sean-k-mooneylookign for it now17:52
sean-k-mooneyhttps://review.opendev.org/#/c/616120/17:53
sean-k-mooneyand i think we use them in https://review.opendev.org/#/c/620115/2817:53
sean-k-mooneythe pci manager works differently then the numa toplogy in the RT17:54
sean-k-mooneybut ehere is the claim in the compute manager17:54
sean-k-mooneyhttps://review.opendev.org/#/c/620115/28/nova/compute/manager.py17:55
openstackgerritEric Fried proposed openstack/nova master: DNM: Try to repro bug 1823251 with mysql logs  https://review.opendev.org/67805117:55
openstackbug 1823251 in OpenStack Compute (nova) "Spike in TestNovaMigrationsMySQL.test_walk_versions/test_innodb_tables failures since April 1 2019 on limestone-regionone" [High,Confirmed] https://launchpad.net/bugs/182325117:55
artomsean-k-mooney, it's not actually creating a MoveClaim object though, is it?17:56
artomsean-k-mooney, I'm thinking of a scenario where we live-migration a NUMA instance with neutron SRIOV ports17:57
sean-k-mooneyno its not as far as i rememeber17:57
artomWill the MoveClaim for the NUMA part conflict with the PCI claim for the SRIOV part17:57
sean-k-mooneyno we claim them in the rt and store the pci addresses in the live migration data object17:58
sean-k-mooneyi dont think we are using move claims at all17:58
*** munimeha1 has joined #openstack-nova17:59
sean-k-mooneywe might use move claims for cold migration i have not check that in a long time17:59
mriedemefried: i think it's failing because the dest doesn't exist when your task runs17:59
mriedemthat's done later here https://storage.bhs1.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/logs_51/678051/3/check/nova-tox-collect-py27/f64b3ba/ara-report/file/5ba0f501-9e58-4f71-859a-ef084a19e0be/#line-1118:00
efriedmriedem: infra said it was because sudo is disabled for tox by default. Latest respin (above) reparents to openstack-tox-with-sudo.18:00
artomsean-k-mooney, if you have the hardware to test those 2 together it would put many minds at east, I think18:00
sean-k-mooneyi plan to do that yes18:00
sean-k-mooneyill do it monday or tuesday if you like18:01
efriedmriedem: but maybe that too. Though I would have thought they would set up log dirs before invoking posts.18:01
sean-k-mooneyi have 1 of my sriov systems setup with 18.04 so i need to install 18.04 on the second one an then install centos on a third host to test something else18:02
sean-k-mooneyill kick of those now and i should be able to test it monday if you bug me or tuesday if you dont18:03
*** tbachman has joined #openstack-nova18:04
artomsean-k-mooney, appreciated :)18:05
mriedemefried: maybe it is https://storage.bhs1.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/logs_51/678051/3/check/nova-tox-collect-py27/f64b3ba/ara-report/file/c8a99499-4049-4878-b0a9-13a9292437f4/#line-118:06
mriedemefried: so maybe your dest needs to be '{{ zuul_log_path }}'18:06
mriedemer + /mysql/18:06
mriedemor {{ zuul_output_dir }}/logs/mysql18:07
mriedemhttps://storage.bhs1.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/logs_51/678051/3/check/nova-tox-collect-py27/f64b3ba/ara-report/file/f31abc72-92ae-4e7d-b37c-8c124f7f32bd/#line-118:07
*** gbarros has joined #openstack-nova18:08
*** ozzzo has quit IRC18:09
efriedmriedem: okay, can try something along those lines if this one borks again.18:09
mriedemok18:10
mriedemyou know about the ara report right?18:10
mriedemto see what runs before and after you with what values18:10
sean-k-mooneyartom: once i have it up an running i can give you remote acess to poke around if you want. i have a static ip so i can easily port forwad you to i the hosts. ubuntu is install now so i just need to add a couple of kernel arges to make sriov work then stack with your code18:13
artomsean-k-mooney, ack, that works18:14
artomsean-k-mooney, I'm thinking of this bit in the resource tracker: https://github.com/openstack/nova/blob/master/nova/compute/resource_tracker.py#L283-L29518:15
artomFor NUMA LM, we call that during check_live_migration_at_destination18:15
efriedmriedem: no, I knew it was there, but never really understood what it meant.18:16
mriedembtw, speaking of that nova.image.download module https://logs.opendev.org/84/676684/1/check/neutron-grenade-multinode/b0a76ac/logs/screen-n-api.txt#_Aug_15_13_25_04_74429318:18
mriedemi'm not sure why that's happening18:18
sean-k-mooney https://github.com/openstack/nova/blob/master/nova/compute/resource_tracker.py#L294-L295 might be a proablem18:19
sean-k-mooneysince we have already claimed them on the dest it could claim them twice18:19
mriedemoh gdi of course https://review.opendev.org/#/c/676684/18:19
mriedemhehh18:19
sean-k-mooneyi think currenlty we only use that for cold migration right18:19
artomsean-k-mooney, *currently*18:21
artomBut NUMA LM is based on doing a MoveClaim18:21
sean-k-mooneyright so we would have to modify this or it wont work with sriov live migration18:22
sean-k-mooneyor rather might not18:22
sean-k-mooneywill you be passing in somthing to indicate its a live migration18:23
artomOr just ignore pci_requests entirely if it's a live migration?18:23
sean-k-mooneythat is what i was going to suggest18:23
artomYeah, the migration will hava a type set18:23
openstackgerritAndreas Jaeger proposed openstack/nova master: PDF documentation build  https://review.opendev.org/67673018:23
*** gbarros has quit IRC18:23
sean-k-mooneyits nice that it is claiming alias based pci devices18:23
sean-k-mooneywe dont currently support that in the sriov migration18:24
sean-k-mooneybut that also means we woudl not update the xml18:24
artomYeah, we can't claim resource we have no intention of using18:24
sean-k-mooneyso what we need to to is 1 check that there is not alias based pci device and two not call the cpi claim bit if its a live migration18:24
artomWhy is 2 alone not enough?18:25
sean-k-mooneyit is but the other is a safty mesure18:25
sean-k-mooneye.g. we shoudl not attemt to migrate if you have alias based pci passthough18:25
sean-k-mooneyits not supported at all18:25
sean-k-mooneywe might already check that somewhere18:26
sean-k-mooneyif we dont we should in the conductor18:26
artomSeems like that should be separate from NUMA LM tho18:26
sean-k-mooneyya it is.18:26
artomNUMA LM just cares about not stepping on SRIOV LM's toes18:27
sean-k-mooneyso for numa the if(migration.type!=live) shoudl be enough18:27
artomWe have functional tests for SRIOV LM?18:28
artomI wonder if we could try both in a func test18:28
sean-k-mooneywe dont18:28
artomFunc tests for neutron SRIOV at all?18:29
sean-k-mooneynope18:29
sean-k-mooneywe have for flavor based pci passthough18:29
sean-k-mooneyhttps://github.com/openstack/nova/blob/master/nova/tests/functional/libvirt/test_pci_sriov_servers.py18:29
sean-k-mooneybut no fucntional test for neutorn sriov18:29
artomSo manual testing it is18:30
sean-k-mooneyso we would have to extend the neutron mock18:30
artomYeah18:30
sean-k-mooneybut we could add them if we did18:30
sean-k-mooneyi dont think we have any livemigration functional tests that use the libvirt fake driver18:30
artomWe didn't18:31
artomNUMA LM's going to be the first18:31
artomYou meant fakelibvirt, right?18:31
sean-k-mooneyso your going to implement the multiple port bidnign workflow in the neutron fixtures18:31
sean-k-mooneyyes18:31
artomBecause we have a couple that use fake virt18:31
sean-k-mooneyfakelibvirt18:31
artomsean-k-mooney, Ha. Ha. Ha.18:32
sean-k-mooneyartom: you will need that form live migration testing i think18:32
sean-k-mooneyyou would for sriov lm18:32
artomsean-k-mooney, heh, one thing at a time :)18:32
sean-k-mooneyfor numa you could fall back to the old codepath18:32
artomIf we decide neutron sriov func test is something we want, I don't mind doing it18:33
artomBut for NUMA LM it's irrelevant18:33
*** gbarros has joined #openstack-nova18:35
sean-k-mooneyit would be nice to have functional test for this at some point but i think we shoudl wait till U to do that18:36
sean-k-mooneyat least that we we can do some firstpart testing beyond unit tests18:36
mriedemsean-k-mooney: https://review.opendev.org/#/c/678098/2 needs to mention the reason for the difference in the backport (in the commit message)18:37
mriedemotherwise it looks OK18:37
sean-k-mooneyok ill respin it now18:37
sean-k-mooneyit was this https://github.com/openstack/os-vif/commit/ee124d2e98c2660ced7d71c1bd10525bf095a93918:38
mriedemyeah i just ofund that18:38
mriedemId8b71172fb06b435cf169a7e55c11233f22fa65b18:38
*** gyee has quit IRC18:42
openstackgerritsean mooney proposed openstack/os-vif stable/stein: only disable mac ageing for ovs hybrid plug  https://review.opendev.org/67809818:44
sean-k-mooneysomething like ^18:44
sean-k-mooneythe other option we have would be to backport Id8b71172fb06b435cf169a7e55c11233f22fa65b18:44
*** gyee has joined #openstack-nova18:46
mriedemno we don't want to backport that big refactor18:49
mriedem+2 on the other18:49
mriedemmelwitt: maybe you want to hit https://review.opendev.org/#/c/678098/ since lee is probably done18:49
*** xek has joined #openstack-nova18:58
mriedemmelwitt: any changes of getting https://review.opendev.org/#/c/507486/ and/or https://review.opendev.org/#/c/640898/ updated soonish before i lose context on those?18:58
mriedem*chance18:58
*** hamzy has quit IRC19:01
*** lyarwood has quit IRC19:04
openstackgerritArtom Lifshitz proposed openstack/nova master: Introduce live_migration_claim()  https://review.opendev.org/63566919:04
openstackgerritArtom Lifshitz proposed openstack/nova master: New objects for NUMA live migration  https://review.opendev.org/63482719:04
openstackgerritArtom Lifshitz proposed openstack/nova master: LM: add support for augmenting migrate_data with info from claims  https://review.opendev.org/63482819:04
openstackgerritArtom Lifshitz proposed openstack/nova master: LM: add support for updating NUMA-related XML on the source  https://review.opendev.org/63522919:04
openstackgerritArtom Lifshitz proposed openstack/nova master: NUMA live migration support  https://review.opendev.org/63460619:04
openstackgerritArtom Lifshitz proposed openstack/nova master: Deprecate CONF.workarounds.enable_numa_live_migration  https://review.opendev.org/64002119:04
openstackgerritArtom Lifshitz proposed openstack/nova master: Functional test for NUMA live migration  https://review.opendev.org/67259519:04
*** gbarros has quit IRC19:07
*** bnemec has joined #openstack-nova19:10
*** bnemec is now known as beekneemech19:11
*** eharney has quit IRC19:13
sean-k-mooneyartom: the last patch ^ is the one i should deploy yes?19:16
artomsean-k-mooney, hol'up, things are in motion19:16
sean-k-mooneyhave you added the sriov if?19:16
sean-k-mooneyok19:16
artomBut in general, yeah19:17
artomI did add the if19:17
sean-k-mooneyim not quite ready yet to run it but its close19:17
*** boxiang has joined #openstack-nova19:17
sean-k-mooneyi normlly run a vanila devstack install first then switch to the ting i want to test19:17
openstackgerritEric Fried proposed openstack/nova master: Allow strict_proxies for sdk Connection  https://review.opendev.org/67692619:18
efriedI think I oughtta rebase the ironic series on that ^ and switch it on.19:19
*** zhubx has quit IRC19:19
*** zhubx has joined #openstack-nova19:20
*** gbarros has joined #openstack-nova19:22
*** ozzzo has joined #openstack-nova19:23
openstackgerritEric Fried proposed openstack/nova master: Introduces SDK to IronicDriver and uses for node.get  https://review.opendev.org/64289919:24
*** boxiang has quit IRC19:24
openstackgerritEric Fried proposed openstack/nova master: Use SDK for node.list  https://review.opendev.org/65602719:24
openstackgerritEric Fried proposed openstack/nova master: Use SDK for validating instance and node  https://review.opendev.org/65602819:24
openstackgerritEric Fried proposed openstack/nova master: Use SDK for setting instance id  https://review.opendev.org/65969019:24
openstackgerritEric Fried proposed openstack/nova master: Use SDK for add/remove instance info from node  https://review.opendev.org/65969119:24
openstackgerritEric Fried proposed openstack/nova master: Use SDK for getting network metadata from node  https://review.opendev.org/67021319:24
efriedso there's that19:24
*** hamzy has joined #openstack-nova19:26
*** gbarros has quit IRC19:30
*** gbarros has joined #openstack-nova19:35
NobodyCamHappy Friday Nova folks19:38
dansmithdon't fall for it, it's a trap19:38
NobodyCamoff the wall question; is it possible to the kvm vgamem option per flavor or image?19:39
NobodyCamlol19:39
*** tbachman has quit IRC19:39
dansmithsee ^19:39
sean-k-mooneyNobodyCam: yes19:39
sean-k-mooneyyou can change it via the image19:39
NobodyCamNice! :)19:40
NobodyCamThank you19:40
sean-k-mooneyhttps://github.com/openstack/glance/blob/master/etc/metadefs/compute-libvirt-image.json#L57-L6219:40
sean-k-mooneyalso we dont test that so it might be broken19:40
sean-k-mooneybut it should work19:40
sean-k-mooneylet us know19:40
sean-k-mooneyoh and apparently you can set a limit in the flavor19:41
*** gbarros has quit IRC19:43
NobodyCamI'll play with that19:44
NobodyCam<model type='qxl' ram='65536' vram='65536' vgamem='16384' heads='1' primary='yes'/>19:44
NobodyCamis what I'm getting by default19:44
sean-k-mooneyya this is the code that uses it in the livbrt dirver19:45
sean-k-mooneyhttps://opendev.org/openstack/nova/src/branch/master/nova/virt/libvirt/driver.py#L4809-L484619:45
sean-k-mooneyapparently its supported by vmware too19:45
sean-k-mooneyi think that default is coming form libvirt19:46
NobodyCamyea looking at the code I'm not sure that will get me what I'm looking for19:48
sean-k-mooneywhat are you trying to do19:49
sean-k-mooneyoh you want to change vgamem19:49
sean-k-mooneynot vram19:49
sean-k-mooneyhttps://libvirt.org/formatdomain.html#elementsVideo looking at the docs19:50
sean-k-mooneyvgamem is used for the size of the fallback framebuffer for the qxl type only19:51
NobodyCamoh, yea reading that now19:52
sean-k-mooneyvram is basicaly the amount of ram reported to as on the grapics card to the guest19:52
sean-k-mooneyso its normally the only one you care about changing19:53
NobodyCamyea, Let me test that, Thank you sean-k-mooney :)19:54
sean-k-mooneyno worries. i added suport for requestion virtio a few weeks ago so it relitivly fresh in my mind since i wen though all the option at that time19:54
NobodyCam:)19:58
*** ivve has joined #openstack-nova20:03
*** mgariepy has quit IRC20:06
*** eharney has joined #openstack-nova20:10
*** weshay is now known as weshay_MOD20:10
openstackgerritMatt Riedemann proposed openstack/nova master: neutron: refactor nw info cache refresh out of associate_floating_ip  https://review.opendev.org/67830020:15
openstackgerritMatt Riedemann proposed openstack/nova master: Trap and log errors from _update_inst_info_cache_for_disassociated_fip  https://review.opendev.org/67830120:15
*** igordc has quit IRC20:37
sean-k-mooneyartom: can you login to "ssh stack@jump.seanmooney.info -p 10024"20:48
artomsean-k-mooney, I cannot, looks like a timeout20:48
artomAlso, this is a public channel20:48
sean-k-mooneyya i know i added your public keys20:48
sean-k-mooneyi was hoping it would jsut work for you and you would be wondering how20:49
artom*suspicious Fry* I don't remember sending you my public key20:49
sean-k-mooneyyou didnt but i got them form your github20:49
sean-k-mooneyi have the forwarding workin glocally but i have not test it form external and it look like it not right20:50
artomI have my pubkeys on github?20:52
artomOh, the oshax thing?20:52
sean-k-mooneyyep https://github.com/notartom.keys20:52
sean-k-mooneyall gitub user have there public keys on github if you uploade them20:53
artomTIL20:53
sean-k-mooneywant to try again20:53
sean-k-mooneyi change the port the nat was applying too20:53
sean-k-mooneyif it doesnt work no worries ill fix over the week end. still learnign how to configre my ubitui router20:54
NobodyCamFYI sean-k-mooney that setting seems to have taken care of my error ... Thank you (again) for your assistance :)20:54
sean-k-mooneyNobodyCam: cool glad it still works20:55
NobodyCamhehehe ;p20:55
artomsean-k-mooney, hanging on debug1: Connecting to jump.seanmooney.info [93.107.226.179] port 10024.21:04
artomJust... open a tmux :)21:05
sean-k-mooneyartom: ok its the rever nat that is not working yet then21:05
sean-k-mooneyi mean i could also jsut use the simple ui for doing it rthar then doing it manually21:06
sean-k-mooneythat might work21:07
sean-k-mooneyill fix it over teh weekend if not by connecting via my phone21:08
sean-k-mooneyon the plus side i think numa migration is working with your patches21:09
sean-k-mooneyi need to test it properly but i did just migrate a vm with hw:numa_nodes=221:10
sean-k-mooneyand one with a sriov macvtap21:14
sean-k-mooney+ numa21:14
mriedemhuh, i saw this unit test job timeout https://18e51e15a34f17ffbc81-ffc80d196410a18186442d9badd30b78.ssl.cf2.rackcdn.com/677819/2/gate/openstack-tox-py27/3446844/job-output.txt21:18
mriedemand if you look at the tests being run, they are being run in serial after some point21:18
*** xek has quit IRC21:25
*** needscoffee is now known as kmalloc21:32
sean-k-mooneyartom:  so... your numa code21:33
sean-k-mooneyhave you refactored it much lately21:33
artomsean-k-mooney, it's currently half-broken as I work through feedback and tests failing...21:33
sean-k-mooneyif by half broken you mean it nolonger updates the xml then yes21:33
sean-k-mooneyyes it is21:33
artomSounds about right.21:34
artomFields got renamed, so various if checks are failing.21:35
sean-k-mooneyon the plus side the vm was moved21:35
sean-k-mooneyso its not anymore broken the without your code21:35
openstackgerritArtom Lifshitz proposed openstack/nova master: Introduce live_migration_claim()  https://review.opendev.org/63566921:43
openstackgerritArtom Lifshitz proposed openstack/nova master: New objects for NUMA live migration  https://review.opendev.org/63482721:43
openstackgerritArtom Lifshitz proposed openstack/nova master: LM: add support for augmenting migrate_data with info from claims  https://review.opendev.org/63482821:43
openstackgerritArtom Lifshitz proposed openstack/nova master: LM: add support for updating NUMA-related XML on the source  https://review.opendev.org/63522921:43
openstackgerritArtom Lifshitz proposed openstack/nova master: NUMA live migration support  https://review.opendev.org/63460621:43
openstackgerritArtom Lifshitz proposed openstack/nova master: Deprecate CONF.workarounds.enable_numa_live_migration  https://review.opendev.org/64002121:43
openstackgerritArtom Lifshitz proposed openstack/nova master: Functional test for NUMA live migration  https://review.opendev.org/67259521:43
* artom calls it a week21:44
sean-k-mooneyo/ ill test that on monday21:44
sean-k-mooneyor when your finsihed erworking it21:45
*** igordc has joined #openstack-nova21:45
*** takamatsu has quit IRC21:55
*** rcernin has joined #openstack-nova22:11
*** KeithMnemonic has quit IRC22:13
*** ivve has quit IRC22:17
*** BjoernT has quit IRC22:18
*** kaisers has quit IRC22:21
openstackgerritMatt Riedemann proposed openstack/nova master: Find instance in another cell during floating IP re-association  https://review.opendev.org/65659422:24
*** kaisers has joined #openstack-nova22:26
*** mriedem has quit IRC22:27
*** TxGirlGeek has joined #openstack-nova22:39
openstackgerritMerged openstack/python-novaclient master: Add --migration-type and --source-compute to migration-list  https://review.opendev.org/67511722:43
*** tonyb[m] has joined #openstack-nova22:55
*** macz has quit IRC22:57
*** igordc has quit IRC23:00
*** rcernin has quit IRC23:11
*** rcernin has joined #openstack-nova23:12
*** beekneemech is now known as keanu23:26
*** keanu is now known as beekneemech23:27
melwittmriedem: yes, sorry, been caught up in downstream stuff of late. I will update those soon, probably this weekend23:39

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!