Wednesday, 2020-02-05

sean-k-mooney"Feb 04 23:58:11 cyborg devstack@cyborg-api.service[1026]: 2020-02-04 23:58:11.561 1162 DEBUG wsme.api [req-e73cfa00-93fe-48ec-ac53-f7cde5e7ba68 587231efdc20474ba378d4dff3c7bfd3 aaa44c9e9ced4e328a3503c86899f01f - default default] Client-side error: ExtArq not found with uuid=cb27c864-8a56-4440-85f1-09023019e1ab format_exception /usr/local/lib/python3.7/dist-packages/wsme/api.py:222" that is the error00:00
sean-k-mooneyin the cyborg log00:00
sean-k-mooneyoh there is a traceback in the cyborg conductor too00:01
sean-k-mooneyoh that is unrealated i tried to create duplicate profiles previously00:03
sean-k-mooneyoh i found the issue its the same one i had before i think http://paste.openstack.org/show/789142/00:05
sean-k-mooneyso it looks like the cyborg api is not able to call back the nova api to notify it of the binding completion00:05
sean-k-mooneyim sligly confused why the cyborg api woudl be trying to do that and not the cyborg conductor but that is the issue00:06
sean-k-mooneyefried: dansmith sundar ^ any ideas. i might look at the tempest jobs to see what they are setting00:07
dansmithyeah, I dunno, but agreed it seems weird that it would be cyborg api00:08
dansmithis the "conductor" really the compute agent?00:08
sean-k-mooneyno that is seperate00:08
sean-k-mooneythey have all 300:08
sean-k-mooneycopying novas architecure00:08
dansmithokay, I guess I'd have thought it'd be the compute agent sending that then, but I guess if the conductor is blocking on the compute to finish then maybe that makes sense00:09
dansmithI gotta run in a sec to meet people, but maybe we can get an answer from sundar tomorrow00:09
sean-k-mooneywell it seams like the api is blocking on the agent or conductor00:09
sean-k-mooneyya00:10
dansmiththat sounds like a recipe for disaster :/00:10
sean-k-mooneyim going to call it a day anyway00:10
dansmithcool, thanks for looking at this00:10
sean-k-mooneyno worries00:11
sean-k-mooneyhuh so in the ci i see " Ignoring Nova notification error that the instance 79390576-dd09-4f1c-a925-5f8d30c83bdc is not yet associated with a host." at the point i see the trace back in my setup. so im guessing if we win the race so that nova has assocaited the instance with the host cyborg expolodes00:18
dansmithokay that's an expected situation for the fake driver00:19
dansmithshould not be tracing on the cyborg side, but..00:19
sean-k-mooneywell both are the fake driver00:19
sean-k-mooneyanyway tomorrows problem o/00:20
*** mlavalle has quit IRC00:22
*** rcernin has joined #openstack-nova01:06
*** bbowen has quit IRC01:11
*** slaweq_ has joined #openstack-nova01:11
*** bbowen has joined #openstack-nova01:11
*** Liang__ has joined #openstack-nova01:12
*** slaweq_ has quit IRC01:16
*** mdbooth has quit IRC01:20
*** mdbooth has joined #openstack-nova01:22
*** ociuhandu has joined #openstack-nova01:31
*** ociuhandu has quit IRC01:35
*** TxGirlGeek has quit IRC01:36
*** vesper has joined #openstack-nova01:38
huaqiangstephenfin: got your message, thanks.01:38
*** vesper11 has quit IRC01:39
*** igordc has joined #openstack-nova01:52
*** gyee has quit IRC02:00
*** Liang__ has quit IRC02:04
*** vishalmanchanda has joined #openstack-nova02:10
*** damien_r has joined #openstack-nova02:11
*** TxGirlGeek has joined #openstack-nova02:45
*** mkrai has joined #openstack-nova03:08
*** slaweq_ has joined #openstack-nova03:11
*** TxGirlGeek has quit IRC03:14
*** slaweq_ has quit IRC03:16
*** rcernin has quit IRC03:31
*** nweinber_ has joined #openstack-nova03:34
*** TxGirlGeek has joined #openstack-nova03:38
*** kiseok7 has quit IRC03:39
*** nweinber_ has quit IRC04:10
openstackgerritSundar Nadathur proposed openstack/nova master: Bump compute rpcapi version and reduce Cyborg calls.  https://review.opendev.org/70422704:31
openstackgerritSundar Nadathur proposed openstack/nova master: Add cyborg tempest job.  https://review.opendev.org/67099904:31
*** links has joined #openstack-nova04:43
*** adriant has quit IRC04:45
*** adriant has joined #openstack-nova04:49
*** udesale has joined #openstack-nova04:53
*** bnemec has joined #openstack-nova04:53
*** hongbin has joined #openstack-nova04:55
*** rcernin has joined #openstack-nova04:57
*** rcernin has quit IRC04:58
*** rcernin has joined #openstack-nova04:58
*** igordc has quit IRC04:59
*** slaweq_ has joined #openstack-nova05:11
*** slaweq_ has quit IRC05:15
*** TxGirlGeek has quit IRC05:20
*** hongbin has quit IRC05:28
*** evrardjp has quit IRC05:33
*** evrardjp has joined #openstack-nova05:34
*** ratailor has joined #openstack-nova05:40
*** mkrai has quit IRC05:43
*** mkrai has joined #openstack-nova05:57
*** rcernin has quit IRC06:03
*** Liang__ has joined #openstack-nova06:05
*** tetsuro has joined #openstack-nova06:06
*** tetsuro has quit IRC06:10
*** tetsuro has joined #openstack-nova06:15
*** Liang__ has quit IRC06:16
*** tetsuro has quit IRC06:27
openstackgerritQiu Fossen proposed openstack/nova-specs master: Support fuzzy querying instance by tag  https://review.opendev.org/69165106:37
*** tetsuro has joined #openstack-nova06:58
*** tetsuro has quit IRC07:01
*** tetsuro has joined #openstack-nova07:01
*** tetsuro has quit IRC07:02
*** tetsuro has joined #openstack-nova07:03
*** slaweq_ has joined #openstack-nova07:11
*** slaweq_ has quit IRC07:16
*** tetsuro has quit IRC07:17
huaqiangls07:18
*** tetsuro has joined #openstack-nova07:19
*** tetsuro has quit IRC07:19
*** tetsuro has joined #openstack-nova07:20
*** MUUSEE_ has joined #openstack-nova07:21
MUUSEE_Hi can anyone help me with a query regarding " libvirt: virtio-net multiqueue "07:23
*** tetsuro has quit IRC07:31
*** tetsuro has joined #openstack-nova07:32
*** yedongcan has joined #openstack-nova07:34
*** ociuhandu has joined #openstack-nova07:38
*** jaosorior has joined #openstack-nova07:38
*** dpawlik has joined #openstack-nova07:49
*** ociuhandu has quit IRC07:50
*** dpawlik has quit IRC07:53
*** jaosorior has quit IRC07:53
*** iurygregory has quit IRC07:53
*** dpawlik has joined #openstack-nova07:56
*** tetsuro has quit IRC07:57
*** tetsuro has joined #openstack-nova08:00
*** slaweq_ has joined #openstack-nova08:00
*** maciejjozefczyk has joined #openstack-nova08:03
*** jaosorior has joined #openstack-nova08:06
*** slaweq__ has joined #openstack-nova08:09
*** slaweq_ has quit IRC08:10
*** iurygregory has joined #openstack-nova08:11
*** tkajinam has quit IRC08:12
*** tetsuro has quit IRC08:14
*** slaweq has joined #openstack-nova08:14
*** tesseract has joined #openstack-nova08:16
*** slaweq__ has quit IRC08:16
*** ociuhandu has joined #openstack-nova08:30
*** rcernin has joined #openstack-nova08:31
*** rpittau|afk is now known as rpittau08:34
*** ociuhandu has quit IRC08:34
*** ociuhandu has joined #openstack-nova08:36
*** ociuhandu has quit IRC08:40
gibicores: easy +A potential https://review.opendev.org/#/c/69805308:42
*** ralonsoh has joined #openstack-nova08:42
*** ociuhandu has joined #openstack-nova08:44
bauzasgibi: looking08:45
bauzaseasy indeed08:45
*** rcernin has quit IRC08:48
*** slaweq has quit IRC08:48
*** slaweq_ has joined #openstack-nova08:49
*** ociuhandu has quit IRC08:49
*** rcernin has joined #openstack-nova08:54
*** xek has joined #openstack-nova08:54
*** ociuhandu has joined #openstack-nova08:59
*** slaweq__ has joined #openstack-nova09:06
*** slaweq_ has quit IRC09:07
*** rcernin has quit IRC09:07
*** ociuhandu has quit IRC09:13
*** rcernin has joined #openstack-nova09:16
*** rcernin has quit IRC09:20
*** slaweq__ is now known as slaweq09:25
*** rcernin has joined #openstack-nova09:26
*** ociuhandu has joined #openstack-nova09:32
*** rcernin has quit IRC09:33
*** martinkennelly has joined #openstack-nova09:35
*** ociuhandu has quit IRC09:38
*** derekh has joined #openstack-nova09:40
openstackgerritBrin Zhang proposed openstack/nova master: Store instance action event exc_val fault details  https://review.opendev.org/69442809:55
*** ociuhandu has joined #openstack-nova10:14
*** ociuhandu has quit IRC10:20
openstackgerritQiu Fossen proposed openstack/nova-specs master: specify mac for creating instance  https://review.opendev.org/70042910:36
openstackgerritMerged openstack/nova master: Minor improvements to cell commands  https://review.opendev.org/69805310:37
*** abhishekk is now known as abhishekk|away10:39
*** slaweq_ has joined #openstack-nova10:40
*** slaweq has quit IRC10:42
*** yedongcan has left #openstack-nova10:53
*** slaweq__ has joined #openstack-nova11:01
*** slaweq_ has quit IRC11:03
*** mkrai has quit IRC11:05
stephenfingibi: Could you revisit https://review.opendev.org/#/c/705760/ ? ralonsoh spotted some paths I'd missed11:07
stephenfinand dansmith had comments but I've addressed them, afaict11:07
*** rpittau is now known as rpittau|bbl11:10
*** sandonov has joined #openstack-nova11:12
sandonovopenstack-nova11:12
*** udesale_ has joined #openstack-nova11:13
*** sandonov has left #openstack-nova11:14
*** udesale has quit IRC11:16
*** gentoorax has quit IRC11:17
*** gentoorax has joined #openstack-nova11:18
*** psachin has joined #openstack-nova11:30
openstackgerritBrin Zhang proposed openstack/nova-specs master: FUP: Fixed the invalid index in References  https://review.opendev.org/70593311:35
*** pcaruana has quit IRC11:37
*** MUUSEE_ has quit IRC11:41
*** tbachman has quit IRC11:44
*** pcaruana has joined #openstack-nova11:50
*** ociuhandu has joined #openstack-nova11:52
openstackgerritStephen Finucane proposed openstack/nova master: trivial: Remove unused 'cache_utils' APIs  https://review.opendev.org/70565211:52
openstackgerritStephen Finucane proposed openstack/nova master: trivial: Address TODO  https://review.opendev.org/70565311:52
openstackgerritStephen Finucane proposed openstack/nova master: trivial: Bump minimum version of websockify  https://review.opendev.org/70565411:52
openstackgerritStephen Finucane proposed openstack/nova master: trivial: Merge unnecessary 'NovaProxyRequestHandlerBase' separation  https://review.opendev.org/70565511:52
openstackgerritStephen Finucane proposed openstack/nova master: trivial: Remove 'run_once' helper  https://review.opendev.org/70565611:52
openstackgerritStephen Finucane proposed openstack/nova master: tox: Integrate mypy  https://review.opendev.org/67620811:52
openstackgerritStephen Finucane proposed openstack/nova master: mypy: Add type annotations to 'nova.pci'  https://review.opendev.org/67620911:52
openstackgerritStephen Finucane proposed openstack/nova master: mypy: Add nova.cmd, nova.conf, nova.console  https://review.opendev.org/70565711:52
openstackgerritStephen Finucane proposed openstack/nova master: WIP: mypy: Add type annotations to top-level modules  https://review.opendev.org/70565811:52
*** mkrai has joined #openstack-nova11:52
*** ociuhandu has quit IRC11:58
kashyapstephenfin: Hiya; I've addressed your questions, and added additional useful bits where needed: https://review.opendev.org/#/c/693844/12:03
*** ociuhandu has joined #openstack-nova12:05
*** derekh has quit IRC12:07
*** derekh has joined #openstack-nova12:08
*** ociuhandu has quit IRC12:11
*** tosky has joined #openstack-nova12:12
*** ociuhandu has joined #openstack-nova12:17
*** Liang__ has joined #openstack-nova12:19
*** slaweq__ has quit IRC12:19
*** ociuhandu has quit IRC12:22
*** slaweq__ has joined #openstack-nova12:23
openstackgerritMerged openstack/nova master: nova-net: Remove use of legacy 'SecurityGroup' object  https://review.opendev.org/69715512:25
*** mkrai has quit IRC12:26
*** mriosfer has joined #openstack-nova12:28
*** brinzhang has joined #openstack-nova12:29
*** brinzhang has quit IRC12:30
*** brinzhang has joined #openstack-nova12:30
*** brinzhang has quit IRC12:31
*** brinzhang has joined #openstack-nova12:32
*** amoralej has joined #openstack-nova12:34
mriosferHi, We got Openstack Queens RDO. Some customers are telling us that their Windows VM CPU performs bad as a hell. We deployed the lasted virtio already to check improvements. I got a video an its true work its frustating, I/O is less than <1ms12:36
sean-k-mooneywhat are you deploying on libvirt/kvm12:40
sean-k-mooneythe first thing i woudl do is alter the cpu topology to be sane for windows12:41
sean-k-mooneyby defualt we allcoate 1 cpu socket per favor.vcpu12:41
mriosferwe're ussing quemu-kvm for instances .12:42
sean-k-mooneyif the guest dont have a numa toplogy set hw:cpu_sockets=1 and hw:cpu:threads=212:42
mriosferHow do you do that?12:42
sean-k-mooneyassuming the host has Hyper thread enabled.12:42
mriosferyes of course12:43
mriosferdo you setup that extra features in the flavors?12:43
sean-k-mooneyyes you can do it in the flavor or image12:43
sean-k-mooneyin the image its hw_ instead of hw:12:43
mriosferi prefer image12:44
sean-k-mooneythat should help the windows shcduler if you have more then 4 vcpu in the vm12:44
mriosferyes.. some cpu got 8cpu or 12vcpu12:44
mriosferand with terminal server the performance its terrible12:44
sean-k-mooneyya so windows will see that as 8 to 12 cpu sockets with out default setting12:45
*** brinzhang has quit IRC12:45
sean-k-mooneythe next thing to look at would be the graphics device model12:45
sean-k-mooneyyou said you were on queens12:45
mriosferyes12:45
mriosferour nodes dont have pcie addtional graphic card just the integrated one by Dell12:45
sean-k-mooneyya that is ok12:46
sean-k-mooneyon train+ would suggest using the virtio gpu but on queens you should set hw_video_model=qxl12:46
sean-k-mooneyi belive qxl is the best performing graphic device that qemu can emulate prior to support for virtio12:47
sean-k-mooneyhttps://github.com/openstack/glance/blob/master/etc/metadefs/compute-libvirt-image.json#L71-L8412:47
sean-k-mooneyyou could also use the hw_video_ram element to increase that12:47
sean-k-mooneyi belive it default to 64mb12:47
mriosferthath ram uses the instance ram no?12:48
sean-k-mooneyno12:48
sean-k-mooneyits addtional host ram12:48
mriosferoh , ok12:48
sean-k-mooneyit can only be set on the image if the flavor allows it by setting  hw_video:ram_max_mb12:48
sean-k-mooneyi would start with the qxl video model and then if that is not enough after the cpu changes then increae the ram to say 128 or 25612:49
mriosfersean do you know why error? https://gyazo.com/10156259f833a20decbdce821ef8e31b12:49
sean-k-mooneyyes12:50
sean-k-mooneythis is one of the model that is in the metadefs12:50
sean-k-mooneyso there is a drop down for it12:50
sean-k-mooneyuse the filter box above12:50
sean-k-mooneyinstead of a custom filed12:50
sean-k-mooneyour you can scoll down there should be a libvirt dropdown in the availabel metadata12:51
mriosferoh12:51
*** damien_r has quit IRC12:51
mriosferim fuck*** noob12:51
sean-k-mooneythe gui is "trying" to be help full12:52
sean-k-mooneyit actully really annoyed me when they added that "feature"]12:52
mriosferrebooting instance will apply the new changes?12:52
sean-k-mooneyno12:52
sean-k-mooneyunfortunetly not12:52
sean-k-mooneyyou can only update the instnace with a rebuild12:53
sean-k-mooneywhich would loose data on the root disk12:53
mriosferoh fuck12:53
sean-k-mooneywe make a copy of the data in the vm when we boot it so that it intentionall will not be affected by change in glance12:54
sean-k-mooneyif we did not you could break all your deployed vms with a typo12:54
sean-k-mooneywhich would be bad12:54
sean-k-mooneyin your case once you are happy with the final image and its performace12:54
mriosfermmm maybe i can do a workarround at ceph level :)12:55
sean-k-mooneyyou could do a db update to add the missing keys to the copy12:55
sean-k-mooneythen a hard reboot of the vm would fix it12:55
sean-k-mooneyi belive we store the instance image keys in the system_metadata table prefixed with img_12:56
mriosfernow looks much better the image : https://gyazo.com/cacf129fc79e2f1b23b607704b645d7112:56
sean-k-mooneyyes but you will need to update the flaovr to have the hw_video_ram work12:56
*** ociuhandu has joined #openstack-nova12:57
sean-k-mooneythe reason being we did not want tenant to be abel to upload there own image and request 256GB of video ram12:58
mriosferdo you also setup the Max vCPU threats and vCpu Cores at flavor level?12:59
sean-k-mooneyyou dont need too but can12:59
sean-k-mooneyfor the video ram it will hit this code https://github.com/openstack/nova/blob/a948a803b561606a892133940915caae610c080b/nova/virt/vmwareapi/vmops.py#L334-L33812:59
mriosferthat just for vmware no?13:00
sean-k-mooneyoh wrong file let me check the libvirt driver13:00
sean-k-mooneyvmware would reject it if the flavor is not set13:00
sean-k-mooneyya its the same https://github.com/openstack/nova/blob/master/nova/virt/libvirt/driver.py#L5057-L506313:01
sean-k-mooneyi would just lave that out for now13:01
sean-k-mooneyas you would also have to update the embeded copy of the flavor or do a resize to pick up the change13:02
mriosferyes i will do for sure13:02
sean-k-mooneyif you find qxl and the toplogy chagne are not enough the try the ram change13:02
mriosferif it works i will send you a pack a beers!13:03
sean-k-mooneyha :) well on other question is the slowness you are finding with the horizon console or do you still see that with rdp in the guest13:03
mriosferi see in the rdp guest13:04
stephenfinbauzas: Assuming efried and dansmith aren't online yet, I think you ought to look at this and send it through so we can unblock neutron13:04
mriosferi dont recomend users work through horizon13:04
sean-k-mooneyright ok then ya this shoudl help13:04
stephenfinWe can use a follow-up if dansmith isn't happy with any of my answers13:04
* bauzas has to disappear for 30 mins13:04
sean-k-mooneyi have seen issue with novnc adding lag in the past13:05
bauzasstephenfin: which change do you want me ?13:05
stephenfinbauzas: oh, whoops https://review.opendev.org/#/c/70576013:05
sean-k-mooneynot in recent times but i had to swap to spice at one point to work around it13:05
bauzasah13:05
bauzaswill look at it in 30 mins13:05
stephenfinperfect, thanks :)13:05
sean-k-mooneymriosfer: anyway hopefully that will help13:05
sean-k-mooneyim going to go get lunch13:06
mriosferyes be sure that yes at least much more than google13:07
* mriosfer lunch13:07
*** tbachman has joined #openstack-nova13:10
*** slaweq has joined #openstack-nova13:11
*** ratailor has quit IRC13:12
*** slaweq__ has quit IRC13:13
*** ociuhandu has quit IRC13:13
*** mkrai has joined #openstack-nova13:15
*** rpittau|bbl is now known as rpittau13:15
*** smcginnis|FOSDEM is now known as smcginnis13:19
*** brinzhang has joined #openstack-nova13:20
*** brinzhang_ has joined #openstack-nova13:20
*** damien_r has joined #openstack-nova13:20
*** ociuhandu has joined #openstack-nova13:22
*** lpetrut has joined #openstack-nova13:24
*** ociuhandu has quit IRC13:27
*** ociuhandu has joined #openstack-nova13:28
*** ociuhandu has quit IRC13:29
*** ociuhandu has joined #openstack-nova13:29
*** amoralej is now known as amoralej|lunch13:31
*** mkrai has quit IRC13:35
*** liuyulong has joined #openstack-nova13:58
*** Liang__ is now known as LiangFang14:00
*** eharney has joined #openstack-nova14:01
*** amoralej|lunch is now known as amoralej14:01
*** rpittau is now known as rpittau|mtg14:03
*** HagunKim has quit IRC14:04
*** dtantsur is now known as dtantsur|brb14:05
*** nweinber has joined #openstack-nova14:10
gibistephenfin: I have a question in https://review.opendev.org/#/c/705784/1/nova/network/neutron.py@266214:15
*** pcaruana has quit IRC14:17
bauzassean-k-mooney: efried: I'm currently writing a new revision for https://review.opendev.org/#/c/552924 given https://etherpad.openstack.org/p/mem_page_size_and_placement14:24
bauzassean-k-mooney: efried: and I'm a bit afraid about a possible issue14:24
bauzasshould we always ask for MEMORY_PAGE_SIZE_SMALL ?14:25
*** rpittau|mtg is now known as rpittau14:25
bauzasI mean, if so, we won't get hosts not having NUMA topogilies14:25
*** psachin has quit IRC14:25
bauzastopologies14:25
bauzaseg. a flavor with VCPU=2 and MEMORY_MB=409614:26
bauzas(see https://etherpad.openstack.org/p/mem_page_size_and_placement L62)14:27
*** ociuhandu has quit IRC14:29
stephenfingibi: replied14:31
stephenfingibi: tl;dr: in theory yes, but I don't think we need to worry about it14:31
stephenfincos quotas14:31
dansmithstephenfin: bauzas: replied in the neutron fix thing.. I didn't -1 and there was no need to hold it for me, I was just unsure (still am a little) but if it's right and works...14:32
gibistephenfin: OK, thanks14:32
*** ociuhandu has joined #openstack-nova14:33
efriedbauzas: Remember, these queries are only for flavors that *do* request a NUMA topology.14:34
bauzasefried: ok, if so I was confused by L6214:34
bauzasand I agree with you14:34
efriedin the spec or the etherpad?14:34
*** jamesdenton has joined #openstack-nova14:34
bauzasin the etherpad like I said14:35
bauzasdansmith: ack, like I said, I'm not really a network expert so I was afraid you should have concerns14:35
efriedbauzas: Yeah, so best to confirm with sean-k-mooney, but I think if no page size is requested, you want to add MEMORY_PAGE_SIZE_SMALL, because that's the default today.14:35
bauzasyou at least know it better than me :)14:35
bauzasefried: that's my point14:36
efriedbauzas: And no, you won't land on a host that doesn't have the NUMA split. But you weren't going to land there anyway, because segregation.14:36
bauzasefried: if you don't ask for page sizes, we should call placement with this trait14:36
efriedyes14:36
bauzaswe shouldn't* grah14:36
efriedwe *should*14:36
efriedbecause otherwise you might land on huge pages14:36
efriedwhich we don't want.14:36
bauzasefried: then, if so, you won't get other hosts14:36
efriedIf by "other hosts" you mean flat non-NUMA hosts, that's correct, and that's what we want.14:37
bauzasthe ones that don't have this trait, ie. the ones that weren't modified14:37
bauzasright14:37
bauzasbut,14:37
bauzassee the problem14:37
efriedI don't14:37
efriednot yet14:37
bauzasexample14:37
bauzas:14:37
*** brinzhang_ has quit IRC14:37
*** ociuhandu has quit IRC14:37
bauzasI'm an operator and I don't configure any host for asking a NUMA topology14:38
bauzasso, none of my hosts have this trait14:38
bauzasnow, I have existing flavors14:38
bauzasbut I upgrade to Ussuri14:38
bauzasand then the scheduler now transforms the Placement call to ask for this trait14:39
bauzasthen I will get NoValidHosts14:39
stephenfindansmith: replied (you read it correctly, yeah)14:39
bauzasefried: I don't see a way to say : ask for this trait for all NUMA hosts, but don't ask for it for the others14:40
efriedbauzas: We talked about this yesterday a bit. You're correct: if you upgrade but don't switch your hosts to NUMA-aware, you won't be able to land NUMA flavors. That's as designed.14:41
dansmithstephenfin: can you read my reply just now in that case?14:41
sean-k-mooneyefried: i need to sumerise the options for the config option in the spec form the irc conversation and other i had. ill try and do that in about an hour14:42
sean-k-mooneyi need to run to the bank now14:42
stephenfindansmith: I think what's there is correct14:42
stephenfinline 2678 is going to retrieve the list of ports, which will look like this https://docs.openstack.org/api-ref/network/v2/?expanded=list-ports-detail#list-ports14:43
bauzasefried: I'm unclear sorry14:44
sean-k-mooneybauzas: efried if we go with the bool config option one of the out standing question is the default for that value. long term it should default to true. not sure if we should default to true in u or v however14:44
efriedbauzas: one way or another, you're going to have to decide which hosts are going to NUMA and which are going to be flat.14:44
sean-k-mooneyany brb14:44
dansmithstephenfin: I was going on the blob you put in the comment, but maybe that's wrong? I don't see a nested port_details in he api ref like you have in your blob14:44
stephenfindansmith: Oh, there isn't one14:44
stephenfinThis is something I made up14:45
stephenfinwait14:45
bauzasefried: if we say that this trait is needed *anyway* (eg. even with a standard flavor asking for memory and vcpus), then I'm afraid we would get NoValidHosts for "flat" hosts14:45
stephenfindansmith: Sorry, confusing myself. I made up the 'network_details' field14:46
stephenfinbut the port_details field _is_ returned by the '/v2.0/floatingips' API https://docs.openstack.org/api-ref/network/v2/?expanded=list-ports-detail,list-floating-ips-detail#list-floating-ips14:46
bauzasefried: unless "flat" hosts also have this trait14:46
efriedbauzas: we were going to get that anyway. One way or another, with the proposed segregation, NUMA flavors will all & only land on hosts with the NUMA flag on; and flat flavors will all & only land on hosts with the NUMA flag off.14:46
dansmithstephenfin: right, isn't that what you're trying to fake/fill-in here?14:46
efriedThat's what we *want*14:47
stephenfinport_details, yeah14:47
bauzasefried: the latter is not what we agreed14:47
stephenfinbut it's not always there14:47
efriedum14:47
stephenfinit's dependent on the extension being present/enabled14:47
dansmithstephenfin: right, so I get that part14:47
bauzasefried: flat flavors will get NoValidHosts14:47
bauzasbecause we're asking for a trait anyway14:47
efriedThey will only get NVH if you don't have any flat hosts in your cloud14:48
dansmithstephenfin: the thing I'm asking is whether or not ports[port_id] is equivalent to the port_details field on that api, because according to the example blob in your comment from just now, it should have port_details *inside* it14:48
bauzasefried: no, again14:48
bauzassee L6214:48
dansmithstephenfin: but I don't see that on the list_ports api-ref, so I think that example blob is wrong about that nesting.. right?14:48
bauzasif we ask for MEMORY_PAGE_SIZE_SMALL, then all flat hosts won't get it14:48
sean-k-mooneybauzas: we wont alway add the trait:numa_node=require trait only if you request numa14:48
efriedbauzas: ohh, I see the confusion.14:48
efriedYou only add MEMORY_PAGE_SIZE_SMALL if you're translating a *NUMA* flavor14:48
efriedyeah, what sean-k-mooney said.14:49
bauzasagain, L62 is very confusing14:49
efriedThe etherpad is only talking about what we do for NUMA-aware hosts14:49
efriedand yeah, the etherpad is confusing, so we should really write it more cleanly, say, in a spec :P14:49
bauzasthere are gaps then that I need to provide14:49
bauzasok, then we need to trigger exactly when asking for this trait14:50
*** links has quit IRC14:50
stephenfindansmith: Not sure I get you, so apologies if I repeat stuff you already know14:50
efriedbauzas: All of the stuff in the etherpad is triggered under the same condition: if the flavor asks for a numa topo.14:50
stephenfinif the extension is present, we'd expect a response like this http://paste.openstack.org/show/789164/14:50
stephenfinIf that's not enabled, the port_details field isn't present so we add it manually14:51
dansmithstephenfin: yes, understand, it's the "adding it manually" part I'm talking about14:51
dansmithstephenfin: but let me just stop you for a sec14:51
efriedbauzas: there are things other than PAGE_SIZE_* that make sure you always & only land on a NUMA host. For instance, the NUMA_ROOT trait.14:51
dansmithstephenfin: I think the problem is in a comment by you on the gerrit review, not the code as it is now, if i'm reading the api-ref correctly14:51
bauzasefried: correct, but I need to think about this14:52
dansmithstephenfin: are we able to test this with an old or duly configured neutron (i.e. without that extension) to prove it works? other than you manually with a devstack?14:52
efriedokay. Happy to help. (I have a call at the top of the hour FYI.)14:52
bauzascool14:52
stephenfindansmith: Yeah, let me whip up a functional test real quick14:53
stephenfinthis is what I expect our "fake" response to look like with this change, btw http://paste.openstack.org/show/789165/14:53
stephenfinwe only return the list, stripping the outer container, of course14:53
*** ociuhandu has joined #openstack-nova14:54
dansmithstephenfin: yeah understand that's the desired output of your function14:54
dansmithstephenfin: a functional test isn't going to tell us anything if you're faking the response from neutron I think14:54
stephenfinIt should, assuming our mock of the 'list_ports' API is correct14:55
stephenfinwhich I really hope it is since we use it everywhere14:55
stephenfinCould we do a depends-on from the affected neutron project(s)?14:55
dansmithoh, maybe I misunderstood you.. stephenfin when you said "Oh, yes, that exactly. This should return something like:" did you mean "this whole function should return..." or did you mean "list_ports should return"14:55
stephenfinDoh /o\ "This" = this function, not "list_ports"14:56
dansmithokay, I had quoted and was asking about list_ports originally, so I thought you were giving me the output of list_ports14:57
stephenfinYup, my mistake. Sorry14:57
dansmiththis is why I was saying I think the comment is the problem, because api-ref makes it look like your code is correct14:57
stephenfinYeah, per api-ref 'list_ports' will return something like https://docs.openstack.org/api-ref/network/v2/?expanded=list-ports-detail#list-ports of course14:58
dansmithyeah when I first asked, I was expecting a detail=True to get the detailed listing, which I think is how ours works14:58
stephenfinYup, I don't think we need that since we don't need the detailed view. 'device_id' is the only thing we do care about and that's always there15:00
dansmithyou mean for our current usage I guess15:00
stephenfinyeah15:00
stephenfinif the extension is enabled and 'port_details' _is_ present, we'd only be getting a summary view anyway15:00
dansmithif we only put device_id in there, then someday someone could expect port_details to be full and not understand why it's not15:00
stephenfinwe could, but given these are only used by the deprecated network API proxies, I figured the risk was low15:01
dansmithI'm assuming this is only a neutron-goes-first upgrade scenario and we'll never hit this in the future because someone turned off this extension15:01
dansmithright15:01
stephenfinI'm under the impression that the networking backends choose their extensions or something15:02
stephenfinbecause ralonsoh referred to OVN specifically15:02
dansmithreally?15:02
stephenfinand we didn't see this blow up in our gate, which I guess is using ml2-ovs or ml2-lb15:02
ralonsohstephenfin, yes, this is failing in OVN15:02
ralonsohbut this could happen in other situations, luckily we found it in the CI15:03
stephenfinralonsoh: am I correct in thinking that's because networking-ovn needs to add support for this extension?15:03
ralonsohstephenfin, it can... but this is not a requirement15:03
stephenfinwell, what was networking-ovn and is now in the core15:03
stephenfinright, and ml2-ovs _does_ have it?15:04
ralonsohyes15:04
stephenfindansmith: ^15:04
dansmithyikes :/15:04
stephenfinso I should probably bug lucasagomes or someone to go add that for us at some point15:05
stephenfincan I just say neutron's extension model is weird :)15:05
ralonsohstephenfin, I can do it15:05
stephenfinnot bad. just weird15:05
ralonsohbut as commented, this is not mandatory15:05
ralonsohhttps://github.com/openstack/openstacksdk/blob/master/openstack/network/v2/floating_ip.py#L60-L6415:05
stephenfinralonsoh: No huge panic. Again, this is only used by deprecated networking proxy APIs in nova, which we'd be hoping no one would be using any more15:06
stephenfinbut then again, OVN is the future so...15:06
* stephenfin thinks if we hit something like this again, we might want to consider a gating OVN tempest job15:07
*** tbachman has quit IRC15:08
*** mriedem has joined #openstack-nova15:10
*** eharney has quit IRC15:13
*** jmlowe has joined #openstack-nova15:13
*** jmlowe has quit IRC15:17
*** jmlowe has joined #openstack-nova15:17
*** awestin1 has quit IRC15:19
*** awestin1 has joined #openstack-nova15:20
*** ociuhandu_ has joined #openstack-nova15:22
*** CeeMac has joined #openstack-nova15:25
sean-k-mooneywe could run a ovn job on nova and have it trigger on any cnages to nova/network subtree15:25
sean-k-mooneywe can just grab the neutorn one and set a filter on ther files15:25
*** ociuhandu has quit IRC15:25
sean-k-mooneywant me to submit a patch for that?15:26
*** ociuhandu_ has quit IRC15:27
sean-k-mooneyby the way im not sure we need the port detail form the floating ip. we need the port detail from the port that has the floating ip but that is different15:27
sean-k-mooneystephenfin: ralonsoh ^ i have been looking at other things so im not fully in sync with what the patch/code is doing and your conversation15:28
*** lpetrut has quit IRC15:28
sean-k-mooneybut the only thing nova uses is the port detail form the neutron port itself15:28
ralonsohsean-k-mooney, I've submitted a patch to add this extension in OVN15:29
sean-k-mooneythat will always be populated if the port is bound15:29
ralonsohhttps://review.opendev.org/#/c/705982/15:29
sean-k-mooneyralonsoh: sure but nova never need that info and enduser should not be relying on it15:29
ralonsohyes, that's why I insisted saying that this extension is not mandatory15:30
sean-k-mooneyim pretty sure that optional extention was added after we deprecated the proxy apis in nova15:30
sean-k-mooneyralonsoh: i assume there is more to supporting the exteion then just adding that 1 line15:31
ralonsohthis is the list of supported OVN extensions15:32
sean-k-mooneyunless this is entrily implement in the ml2 core plugin above the drivers?15:32
ralonsohthis dict is used to create the config in the CI too15:32
sean-k-mooneyright but if networking-ovn does not have code support for it and its not implemented at teh plugin level then that is incorrect to add15:32
ralonsohsean-k-mooney, http://codesearch.openstack.org/?q=ML2_SUPPORTED_API_EXTENSIONS&i=nope&files=&repos=15:32
sean-k-mooneyso im asking does the networking-ovn ml2 driver need to be extended to supprot it15:32
ralonsohhttps://opendev.org/openstack/networking-ovn/src/branch/master/networking_ovn/l3/l3_ovn.py15:33
mriosferis recomended enable watchdog in openstack instances?15:33
sean-k-mooneyok so this still seams wrong to me you should not need to enable it in the neutorn tree the driver networking-ovn repo should be provideing the support exteion list15:34
sean-k-mooneymriosfer: am i dont know of any guidence either way. if yo need it then you can use it but its just an optional feature some operators wanted15:35
ralonsohsean-k-mooney, this is something still under discussion15:35
sean-k-mooneyralonsoh: is the  neutron/common/ovn/extensions directory added as part of try ing to merge networking-ovn back in tree15:35
mriosfersean: im going to test your notes in instances right now :)15:36
ralonsohsean-k-mooney, yes, thats in the neutron repo now15:36
* sean-k-mooney hides in case it goes wrong hehe15:36
sean-k-mooneyralonsoh: so networking-ovn is nolonger required at all15:37
ralonsohnope15:37
sean-k-mooneyralonsoh: actully thats off topic we can talk about it someother time15:37
ralonsohsean-k-mooney, sure!15:37
stephenfinsean-k-mooney: I changed the behaviour to rely on that 'port_details' field in a recent patch because I didn't know it was an optional extension15:42
stephenfinsean-k-mooney: we need that info purely so we can get the 'device_id' field,  which is the instance UUID, for the deprecated floating IP proxy APIs15:43
stephenfindeprecated by not removed15:43
sean-k-mooneywe should not need that however.15:46
sean-k-mooneywe can list the ports assocaiated with an insnatce and then we should eb able to list the floating ips assinged ot each port15:47
mriosfersean : :The requested amount of video memory 128 is higher than the maximum allowed by flavor 0 :( something i changed wrong  https://gyazo.com/302f96f1f0e2363da2b9dd10ad741e3e?token=b6e41e022fa6260b90802950e02137bf15:47
stephenfinsean-k-mooney: That sounds like a lot more rework though :)15:48
stephenfinPossible, yes. Worth it?15:48
* stephenfin shrugs15:48
*** brinzhang has quit IRC15:48
*** pcaruana has joined #openstack-nova15:49
*** Sundar has joined #openstack-nova15:50
mriosfersean: found the parameter is : hw_video:ram_max_mb15:53
*** dtantsur|brb is now known as dtantsur15:53
openstackgerritStephen Finucane proposed openstack/nova master: Rework how we check for extensions  https://review.opendev.org/70579215:54
sean-k-mooneymriosfer: that is the flavor one15:54
sean-k-mooneyyes15:54
Sundarsean-k-mooney: I am here if you have any comments or questions about your evaluation of Cyborg patches.15:56
sean-k-mooneySundar: when i tested them yesterday it failed in the cyborg api to send the arq binding notification to nova15:56
Sundarsean-k-mooney: Could you point me to the cyborg logs?15:59
*** ociuhandu has joined #openstack-nova16:01
*** TxGirlGeek has joined #openstack-nova16:01
*** tbachman has joined #openstack-nova16:01
sean-k-mooneySundar: http://paste.openstack.org/show/789142/16:02
*** eharney has joined #openstack-nova16:03
*** tbachman has quit IRC16:05
Sundarsean-k-mooney: Looks like you are pulling in an old version of Cyborg patches. NovaAPIConnectFailure exception has been replaced with InvalidAPIResponse exception: https://review.opendev.org/#/c/698846/6/cyborg/common/nova_client.py16:05
*** openstackstatus has joined #openstack-nova16:05
*** ChanServ sets mode: +v openstackstatus16:05
*** ociuhandu has quit IRC16:05
mriosfersean: dxdiag should show the param of 128MB for vRAM?16:06
mriosferhttps://gyazo.com/1ad894cd16911a7d6f3a9083fbd65fad16:07
*** jmlowe has quit IRC16:08
sean-k-mooneyi think so yes16:09
*** brinzhang has joined #openstack-nova16:09
sean-k-mooneybut i have not tested that16:10
*** mlavalle has joined #openstack-nova16:18
*** slaweq_ has joined #openstack-nova16:23
*** udesale_ has quit IRC16:24
*** slaweq has quit IRC16:25
mriosferhumm im not sure if machine is getting the 128MB vram16:32
mriosfersockets and threats now looks better16:32
efrieddansmith: Left a review on https://review.opendev.org/#/c/631243/16:32
efriedTL;DR: the structural comments from PS43 still need to be addressed.16:32
efriedBut dansmith (and gibi) it would be nice if you could scan through my analysis and see if you agree, or if I'm making a big deal out of nothing.16:32
efriedBasically I'm saying the steps of processing the device profiles should follow the steps of processing bandwidth requests.16:33
dansmithefried: I don't have context on the bandwidth stuff to make that comparison, but will read16:34
efrieddansmith: I seeded the code with comments in the appropriate places, hopefully it's easy enough to follow.16:34
dansmithack16:34
dansmithefried: I think we've told him specifically to follow the network_info and block_device_info patterns everywhere, which I think his code does16:37
dansmithefried: i.e. make these look like our other attachable things, which are ports and volumes16:37
efriedI don't think what I'm suggesting deviates from that, does it?16:38
*** psachin has joined #openstack-nova16:38
dansmithseems like it, but I'm still reading16:38
openstackgerritLee Yarwood proposed openstack/nova master: compute: Report COMPUTE_RESCUE_BFV and check during rescue  https://review.opendev.org/70142916:40
openstackgerritLee Yarwood proposed openstack/nova master: api: Introduce microverion 2.82 allowing boot from volume rescue  https://review.opendev.org/70143016:40
openstackgerritLee Yarwood proposed openstack/nova master: compute: Extract _get_bdm_image_metadata into nova.utils  https://review.opendev.org/70521216:40
openstackgerritLee Yarwood proposed openstack/nova master: WIP libvirt: Support boot from volume instance rescue  https://review.opendev.org/70143116:40
dansmithI'm also pretty sure we specifically told him to not use the legacy reqspec.from_components stuff16:41
dansmithlet me see if I can find that16:41
*** Sundar has quit IRC16:42
efriedWhat he's got will work fine afaict and does seem simpler at first glance.16:44
efriedMy concern is that it's logically very similar to how we're processing port bandwidth requests (pull stuff from flavor and $api, create granular request groups, put them in a special place in the request spec),16:44
efriedso it would be nice if the reader could follow that logic similarly for both kinds of resource.16:44
dansmithhttps://review.opendev.org/#/c/631243/30/nova/objects/request_spec.py16:44
dansmithgranted what he was doing was a lot more than what I _think_ you want him using from_components() for16:45
openstackgerritStephen Finucane proposed openstack/nova master: objects: Add MigrationTypeField  https://review.opendev.org/70601316:46
efrieddansmith: okay, yeah, I agree we shouldn't be doing the api callout from there. In what I'm suggesting, the device_profile_request_groups come into from_components ready-made, just like the port_resource_requests in the preceding chunk.16:46
dansmithefried: shouldn't port_resource_requests just be resource_requests though?16:46
efriedyeah, that would be another great way to do it.16:46
dansmithI'd much prefer that than just adding a new parameter of the same type of thing for each high level thing we add16:47
*** jmlowe has joined #openstack-nova16:47
efriedAgree: fold port & device resource requests (and any others in the future) together prior to from_components. ++16:47
efriedso yeah -- how early can we fold those together? The earlier the better.16:48
efriedMaybe as early as the construction of base_options. Need to see how the supports_port_bandwith_requests business plays in.16:49
dansmiththat's not my preference, I'm just saying if we're still going to call this legacy method to set a single attribute, then they should be combined16:49
dansmithwhat I'd *rather* is he leave what he has here and I follow up and move that port_resource_requests outside of from_components where it belongs in the first place, IMHO16:49
efriedI just don't love that from_components is initializing resource_requests, and then we're extending it after, outside of that method.16:50
dansmithack, that's legit16:50
dansmithso can I follow up after his set and make PRR set after from_components() like he is doing here?16:50
efriedBut you're right, we could easily accept what's here and refactor later.16:50
dansmithif so I shall commit to it in writing16:51
efriedso, remove that as a param from from_components() entirely?16:51
*** sean-k-mooney has quit IRC16:51
dansmithyeah16:51
efriedI don't have the big picture on from_components; you hinted we should be able to get rid of it entirely?16:51
dansmithit was supposed to be bridge code to get us to objects and removed in mitaka or something16:52
efriedoh, didn't mriedem propose a WIP that started doing that?16:52
dansmithhe complained about it a lot, so probably16:52
efriedhttps://review.opendev.org/#/c/697686/ ?16:53
efriedno, not quite16:53
dansmithwell, that's what you're thinking of probably16:53
*** artom has quit IRC16:54
efriedyeah16:54
dansmithregardless, as you can see, from_components is just "set a bunch of things and no other logic"16:54
*** maciejjozefczyk has quit IRC16:54
efriedyeah. but it's common to both build and cold migrate16:54
dansmithto what end?16:55
efriedjust DRY16:55
dansmithit's not really though16:55
efriedlike, I'm not sure I see a better split logically. Whether we want to get rid of the filter_properties in some way, that's fine16:55
dansmithbecause the actual constructor can take field values16:55
efriedThere's a *little* bit of logic in there.16:56
efriedAll these _from_* methods16:56
dansmithokay, I guess so16:57
dansmithbut that would also kinda mean that the setting of a field that doesn't need any handling shouldn't be in here16:57
*** tbachman has joined #openstack-nova16:58
*** psachin has quit IRC16:58
efriedSo your refactor would be like the caller doing16:58
efriedreqspec = RequestSpec(foo=bar, ...)16:58
efriedfor all foo/bar that are static, and then16:58
efriedreqspec.fold_in_other_shit(filter_properties, maybe_others, ...)16:58
efriedwhere the latter is from_components with all the static setters removed?16:59
dansmithwell, that's a potential bigger refactor, because from_components is a classmethod17:00
dansmithand, I'm not sure that would really work with the way the _from things work.. they may expect to "go first"17:00
dansmithso I would just take the req_spec that comes from from_components, and follow up with the direct things17:01
*** brinzhang has quit IRC17:01
dansmithreq_spec = from_components(..); req_spec.requested_resources = port_requested_resources + accel_requested_resources; etc17:01
*** mriosfer has quit IRC17:03
*** efried has quit IRC17:03
*** efried has joined #openstack-nova17:04
efriedI guess I don't see the benefit of assigning some fields in the classmethod and some afterwards. Similar reasoning as mentioned previously: then you always have to go make sure you're assigning the right pieces the right way on the right side of the method boundary.17:06
efriedbut I don't feel strongly enough to -1 that.17:06
mriedembeware the request spec quagmire17:06
efriedI think we're already quagged.17:06
mriedemif you go one up that stack https://review.opendev.org/#/c/697697/ and read my comments, unwinding a bunch of that stuff is going to be dependent on doing a major conductor rpc api version17:07
mriedembecause there is backward compat code to handle the older version17:08
mriedemwhich relies on a lot of this bridge code17:08
*** iurygregory has quit IRC17:08
dansmithefried: I commented and I'm late for another meeting17:08
stephenfinmelwitt: Does my reply at https://review.opendev.org/#/c/705654/ make sense?17:11
stephenfinbefore I rewrite the thing17:11
larsksasd17:12
melwittstephenfin: oh, wow. so ... we are ok to remove the helper method so long as the minimum python version is 3.6 because the native method is doing the same thing? are we (deployers) guaranteed to not be running <= 3.6 at this point?17:15
stephenfinmelwitt: yup https://github.com/openstack/nova/blob/master/setup.cfg#L917:16
stephenfinyou can't install nova on anything less17:16
melwittstephenfin: cool, thanks17:16
melwittyup, that all makes sense then17:16
stephenfinappears to have been the behavior since Python 3.317:17
stephenfinhttps://github.com/python/cpython/blob/v3.3.0/Lib/http/server.py#L562-L56517:17
stephenfincool, I'll rework that so. Thanks for the review17:17
*** tbachman_ has joined #openstack-nova17:25
*** tbachman has quit IRC17:26
melwittstephenfin: I just looked this up, apparently the original commit you referenced _would_ fix the issue because the reverse dns lookup was happening as a result of simply logging a message (I did not expect this) https://github.com/openstack/nova/commit/c0f773a616fb48bf699539c5ac18bd9c55a540c917:28
*** gyee has joined #openstack-nova17:29
*** tbachman_ has quit IRC17:29
stephenfinmelwitt:Oh, fun. Fancy suggesting a commit message I should use? :)17:30
*** tbachman has joined #openstack-nova17:30
melwittstephenfin: technically your commit message is right, so I think we can leave it. let me just add a comment and change my vote17:32
*** evrardjp has quit IRC17:33
*** evrardjp has joined #openstack-nova17:34
*** igordc has joined #openstack-nova17:34
stephenfinmelwitt: Okay. I do need to repush that series to address gibi's comments on the base patch though so if I do need to edit the commit message, let me know in the next 2 mins :)17:35
*** sean-k-mooney has joined #openstack-nova17:36
melwittstephenfin: k, I think it's fine as-is17:39
*** sean-k-mooney has quit IRC17:41
openstackgerritStephen Finucane proposed openstack/nova master: trivial: Remove unused 'cache_utils' APIs  https://review.opendev.org/70565217:43
openstackgerritStephen Finucane proposed openstack/nova master: trivial: Fetch 'Service' objects once when building AZs  https://review.opendev.org/70565317:43
openstackgerritStephen Finucane proposed openstack/nova master: trivial: Bump minimum version of websockify  https://review.opendev.org/70565417:43
openstackgerritStephen Finucane proposed openstack/nova master: trivial: Merge unnecessary 'NovaProxyRequestHandlerBase' separation  https://review.opendev.org/70565517:43
openstackgerritStephen Finucane proposed openstack/nova master: trivial: Remove 'run_once' helper  https://review.opendev.org/70565617:43
openstackgerritStephen Finucane proposed openstack/nova master: tox: Integrate mypy  https://review.opendev.org/67620817:43
openstackgerritStephen Finucane proposed openstack/nova master: mypy: Add type annotations to 'nova.pci'  https://review.opendev.org/67620917:43
openstackgerritStephen Finucane proposed openstack/nova master: mypy: Add nova.cmd, nova.conf, nova.console  https://review.opendev.org/70565717:43
openstackgerritStephen Finucane proposed openstack/nova master: WIP: mypy: Add type annotations to top-level modules  https://review.opendev.org/70565817:43
*** sean-k-mooney has joined #openstack-nova17:43
*** vdrok has quit IRC17:45
*** vdrok has joined #openstack-nova17:45
stephenfinmelwitt: There's a super trivial follow-up to that too, btw, if you'd like to take a look https://review.opendev.org/#/c/705655/17:47
stephenfinI, however, am outta here17:47
* stephenfin ->🏃17:47
*** dtantsur is now known as dtantsur|afk17:48
*** sean-k-mooney1 has joined #openstack-nova17:48
efriedsean-k-mooney (sean-k-mooney1): https://blueprints.launchpad.net/nova/+spec/config-tsc-freq looks like something you would be able to grok. I talked to umbSublime and he's interested in working the implementation. At a glance, it seems like it should be fairly straightforward, but I'd like your opinion if you've got a moment to look.17:48
melwittstephenfin: will do17:48
*** sean-k-mooney has quit IRC17:48
*** sean-k-mooney1 is now known as sean-k-mooney17:49
sean-k-mooneyya i was ment to write up a spec/blueprint17:49
sean-k-mooneyor help them17:49
*** jaosorior has quit IRC17:49
*** NobodyCam has quit IRC17:50
sean-k-mooneyefried: the propoasal was to basically to add 2 extra specs one for the tsc frequency and the other for enableing invtsc17:50
*** NobodyCam has joined #openstack-nova17:51
sean-k-mooneyalso add a trait to land on host that support invtsc17:51
efriedsean-k-mooney: cool. This would be umbSublime's first contribution, so he'll need some mentoring...17:52
sean-k-mooneylooking at my browser history we created 1 an i think i was ment to help draft the update to the bluepirnt.17:53
sean-k-mooneyhttps://etherpad.openstack.org/p/invtsc17:53
sean-k-mooneyis umbSublime about17:53
efriedThat etherpad looks like the same content as what's in the bp description.17:54
sean-k-mooneymy isp decied to force recet my router again and broke my internet17:54
sean-k-mooneyyes17:54
sean-k-mooneyso i obvioulsy never wrote the updated draft17:54
efriedfrom what I could glean, "we" decided at some point that the blueprint could be specless.17:54
*** jmlowe has quit IRC17:54
sean-k-mooneyyes i brought it up in the nova meeting17:54
sean-k-mooneyand we said if we spell out the minor changes need then it could be specless17:55
efriedgreat17:55
*** martinkennelly has quit IRC17:56
sean-k-mooneyumbSublime: this sliped off my radar sorry.17:56
umbSublimereporting in o/17:57
sean-k-mooneyumbSublime: if you are still up for working on the code cahgne ill try and draft some worth in the ether path to describe what is needed. if you are happy with tem we can update the blueprint and i can help you with reviewing your patch17:58
umbSublimeSounds good to me! I don't think I'll have much time to work on this this week, but next week should be fine!17:58
umbSublimesean-k-mooney, in which etherpad would that be ?17:59
efriedsean-k-mooney, umbSublime: we'll want something in approvable form by next week (spec freeze).17:59
sean-k-mooneyhttps://etherpad.openstack.org/p/invtsc18:00
umbSublimeefried, I see18:00
sean-k-mooneyumbSublime: so looking at my private irc logs from talking to you in novemeber18:00
*** TheJulia has quit IRC18:01
sean-k-mooneyi think the proposal was to add a hw:tsc_freq_mhz flavor extra spec18:01
*** TheJulia has joined #openstack-nova18:01
sean-k-mooneyand a hw:invtsc=on|off extra spec18:01
*** rpittau is now known as rpittau|afk18:01
sean-k-mooneythat would be how you requested the feature i the flavor18:02
umbSublimeyes iirc that's we had discussed last time. I'll have to read back my notes to18:02
sean-k-mooneyform a shcduler point of view we would need a new trait for invtsc support18:02
*** johnsom has quit IRC18:03
sean-k-mooneywhich would be auto added if you enabled the flavor extra spec18:03
*** johnsom has joined #openstack-nova18:03
*** mriosfer has joined #openstack-nova18:03
sean-k-mooneyand then the last bit was just the libvirt driver chagne to add the xml elements and report support18:03
mriosferHi , where i can find a suported list of gpu supported by queens?18:04
mriosfersean: looks like hw:video_ram dont report vram to windows for some reason :S18:05
sean-k-mooneyumbSublime: do you want to take go at writing that up? or shall i?18:05
sean-k-mooneyumbSublime: i also linked https://review.opendev.org/#/c/671338/ to you as an example of how to add the feature just to refresh your memory18:05
sean-k-mooneymriosfer: so in the image its hw_video_ram and you need to set hw_video:max_ram in the flavor i think18:06
mriosferyes i got it18:06
mriosferim going to double check18:06
umbSublimeConsidering I'm not really familiar with the codebase yet, I think it would help me if you'd point me in the right direction (right I had forgotten that link). I'll read on this and take a closer look this afternoon. I'll report back if have any questions/concerns18:06
*** artom has joined #openstack-nova18:06
sean-k-mooneyumbSublime: ok i wont get to it today but ill write a rough draft of the chagne in the etherpad tomorrow. then we can update the blue print and review it with the nova team18:08
sean-k-mooneyumbSublime: would you be able to attend the nova team meeting tommorow in case people have questions about your usecase18:09
umbSublimeThank you so much, I hope this isn't too much of a burden :/18:09
umbSublimeYes I should be available tomorrow18:09
sean-k-mooneywell normally if we had more time its something that we would help to do. its almost the feature proposal deadline so it will need to be approve this week or next so i can help a little more then normal18:10
*** tesseract has quit IRC18:11
umbSublimeThere a few few "newcoming contributer" docs out there for OS, but is there one more specific to nova I should up on ?18:12
sean-k-mooneywe do have a nova one im not sure if it differe much form the generic one let me see if i can find it18:12
sean-k-mooneyumbSublime: they are here https://docs.openstack.org/nova/latest/contributor/index.html18:14
umbSublime\o/18:14
sean-k-mooneyumbSublime: you might find this one useful https://docs.openstack.org/nova/latest/contributor/how-to-get-involved.html18:15
umbSublimeI'll try and get up to speed on this tonight18:16
*** amoralej is now known as amoralej|off18:16
*** masayukig has quit IRC18:34
*** masayukig has joined #openstack-nova18:35
*** tbachman has quit IRC18:35
openstackgerritMerged openstack/nova master: nova-net: Remove now unnecessary nova-net workaround  https://review.opendev.org/70244018:38
*** tbachman has joined #openstack-nova18:41
*** tbachman has quit IRC18:45
*** eharney has quit IRC18:49
openstackgerritEric Fried proposed openstack/nova master: WIP: refactor: Do network & accel discovery near volumes  https://review.opendev.org/70608318:49
efrieddansmith: Does that ^ refactor look like what you had in mind?18:49
dansmithefried: from a skim, yeah18:51
efrieddansmith: If so, my main objections to the preceding patch are addressed. The other bits I noted are still worth doing, but could conceivably be FUP'd if there's motivation not to respin the series.18:52
dansmithack18:53
*** ralonsoh has quit IRC18:54
*** dpawlik has quit IRC19:04
*** dpawlik has joined #openstack-nova19:05
*** artom has quit IRC19:15
*** larsks has left #openstack-nova19:28
*** sean-k-mooney has quit IRC19:29
*** sean-k-mooney has joined #openstack-nova19:32
*** LiangFang has quit IRC19:36
*** masayukig has quit IRC19:38
*** NobodyCam has quit IRC19:39
*** vdrok has quit IRC19:39
*** awestin1 has quit IRC19:39
*** TheJulia has quit IRC19:39
*** johnsom has quit IRC19:40
*** tbachman has joined #openstack-nova19:46
*** xek_ has joined #openstack-nova19:48
*** xek has quit IRC19:51
*** TxGirlGe_ has joined #openstack-nova19:59
*** jmlowe has joined #openstack-nova20:01
*** TxGirlGeek has quit IRC20:01
*** jmlowe has quit IRC20:01
*** eharney has joined #openstack-nova20:04
openstackgerritMykola Yakovliev proposed openstack/nova master: Fix boot_roles in InstanceSystemMetadata  https://review.opendev.org/69804020:07
*** jmlowe has joined #openstack-nova20:15
*** artom has joined #openstack-nova20:17
*** dklyle has quit IRC20:19
*** david-lyle has joined #openstack-nova20:19
*** TxGirlGe_ has quit IRC20:28
*** TxGirlGeek has joined #openstack-nova20:29
*** mriosfer has quit IRC20:31
sean-k-mooneyumbSublime: i have updated https://etherpad.openstack.org/p/invtsc with what i think woudl be sufficent for the blueprint.20:33
sean-k-mooneyumbSublime: i have not spellchecked it and as efried will attest that is likely required.20:34
sean-k-mooneyill add this to the end of the nova meeting agendra for tomorrow but if you are happy with the content can you update the blueprint.20:35
*** slaweq_ has quit IRC20:44
*** slaweq_ has joined #openstack-nova20:47
umbSublimesean-k-mooney, thanks! I did a first pass on spellcheck. I think I'll wait until tomorrow to post it in case I think of something to add in the meantime20:49
*** mgariepy has quit IRC20:54
sean-k-mooneyumbSublime: ya if you want to reframe the problem statement etc feel free20:55
*** jmlowe has quit IRC21:08
*** jmlowe has joined #openstack-nova21:10
*** TxGirlGe_ has joined #openstack-nova21:14
*** slaweq_ has quit IRC21:15
*** TxGirlGeek has quit IRC21:15
*** tosky has quit IRC21:18
*** spatel has joined #openstack-nova21:19
spatelsean-k-mooney: hey21:19
spatelI am seeing strange performance issue, I am running erlang application on openstack instance which is giving me very poor performance but if i run same application on bare metal performance is really good21:20
spatelI have created big VM and allocated all vCPU to instance so only single VM running on compute node..21:21
spateltrying to understand where is the bottleneck21:21
*** jmlowe has quit IRC21:22
*** xek_ has quit IRC21:27
openstackgerritStephen Finucane proposed openstack/nova master: trivial: Fetch 'Service' objects once when building AZs  https://review.opendev.org/70565321:31
openstackgerritStephen Finucane proposed openstack/nova master: trivial: Bump minimum version of websockify  https://review.opendev.org/70565421:31
openstackgerritStephen Finucane proposed openstack/nova master: trivial: Merge unnecessary 'NovaProxyRequestHandlerBase' separation  https://review.opendev.org/70565521:31
openstackgerritStephen Finucane proposed openstack/nova master: trivial: Remove 'run_once' helper  https://review.opendev.org/70565621:31
openstackgerritStephen Finucane proposed openstack/nova master: tox: Integrate mypy  https://review.opendev.org/67620821:31
openstackgerritStephen Finucane proposed openstack/nova master: mypy: Add type annotations to 'nova.pci'  https://review.opendev.org/67620921:31
openstackgerritStephen Finucane proposed openstack/nova master: mypy: Add nova.cmd, nova.conf, nova.console  https://review.opendev.org/70565721:31
openstackgerritStephen Finucane proposed openstack/nova master: WIP: mypy: Add type annotations to top-level modules  https://review.opendev.org/70565821:31
sean-k-mooneyspatel: erlang is pretty memory sensitive so using hugepages in the gust and likely on the host for the vm should help21:32
sean-k-mooneyspatel: you shoudl also look into http://erlang.org/doc/man/HiPE_app.html21:33
openstackgerritMerged openstack/nova master: Handle neutron without the fip-port-details extension  https://review.opendev.org/70576021:35
spatelsean-k-mooney: i am using hugepage21:38
spatelI have tried all possible best practices21:38
spatelearlier i gave all CPU to vm (32 core) so thinking i should just create VM with 16 vCPU so it stay one side of NUMA zone to see how its performing..21:39
*** liuyulong has quit IRC21:41
sean-k-mooneyya numa affinitising the vm might help or create a dual numa guest21:42
sean-k-mooneythat would report the numa affintiy to erlang but if you are using hugepags it has a numa toplogy of 1 already21:42
spatelI am only curious if i run same benchmark on bare metal then i get good result but if i run on VM then it gives poor result21:43
sean-k-mooneywhat it could be is the cpu model21:43
sean-k-mooneyif you are not useing host-passthouhg you might not have all the cpu feature in the vm that you have in the host21:43
spatelmy CPU mode is passthrough21:43
sean-k-mooneywell if you have it numa affined and cpu passthohg21:43
sean-k-mooneyare you using cpu pinning21:44
spatelYes21:44
spatelCPU pinning21:44
sean-k-mooneythen the only bottelnecs you have left are disk io or netwrok21:44
sean-k-mooneywhat disk cache mode are you using21:44
spatelI am using SR-IOV and not seeing any single packetloss or performance issue on network layer21:44
spatelWe have SSD21:45
spatelnot sure about disk cache mode?21:45
sean-k-mooneyhttps://docs.openstack.org/nova/latest/configuration/config.html#libvirt.disk_cachemodes21:45
spatelI have noticed in SS command lost of packet getting in queue and increasing latency21:46
sean-k-mooneyfile=writeback,block=writeback,network=writeback would be the highest performace setting in most cases21:46
spatelhmm21:48
spatelI will try that too21:48
spatelis that kvm setting ?21:48
sean-k-mooneyya its a setting for the libvirt dirver21:49
spatelI have flavor setting numa_node=221:49
sean-k-mooneyso in the [libvirt] section21:49
spatelok.. i will give it a try and see if that make any difference21:49
spateltomorrow i am going to run more erlang test and see where is the bottleneck.. i will keep you posted..21:50
*** spatel has quit IRC21:51
*** nicolasbock has joined #openstack-nova21:56
*** damien_r has quit IRC21:57
*** maciejjozefczyk has joined #openstack-nova22:02
*** nweinber has quit IRC22:07
*** maciejjozefczyk has quit IRC22:18
*** mriedem has left #openstack-nova22:19
*** nicolasbock has quit IRC22:23
openstackgerritGhanshyam Mann proposed openstack/nova master: Deprecate base rules in favor of new rules  https://review.opendev.org/70162422:26
openstackgerritGhanshyam Mann proposed openstack/nova master: Fix os-attach-interfaces policy to be admin_or_owner  https://review.opendev.org/70513522:26
openstackgerritGhanshyam Mann proposed openstack/nova master: Remove old policy enforcement in attach_interfaces  https://review.opendev.org/70512722:26
openstackgerritGhanshyam Mann proposed openstack/nova master: Add test coverage of existing attach_interfaces policies  https://review.opendev.org/70512622:26
openstackgerritGhanshyam Mann proposed openstack/nova master: Introduce scope_types in os-attach-interfaces  https://review.opendev.org/70579922:27
*** N3l1x has quit IRC22:38
efriedgibi: stephenfin, other: how many people were in the room at the Shanghai PTG?22:44
*** tkajinam has joined #openstack-nova22:48
*** vdrok has joined #openstack-nova23:00
*** NobodyCam has joined #openstack-nova23:00
*** TheJulia has joined #openstack-nova23:00
*** masayukig has joined #openstack-nova23:00
*** awestin1 has joined #openstack-nova23:02
*** johnsom has joined #openstack-nova23:02
*** sean-k-mooney has quit IRC23:28
*** sean-k-mooney has joined #openstack-nova23:29
*** ociuhandu has joined #openstack-nova23:30
*** mlavalle has quit IRC23:32
*** ociuhandu has quit IRC23:35
alex_xuefried: 20 maybe23:36

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!