Thursday, 2019-08-08

« Wednesday, 2019-08-07 Index Friday, 2019-08-09 »
*** markvoelker has joined #openstack-nova00:09
*** slaweq_ has joined #openstack-nova00:11
*** slaweq_ has quit IRC00:15
*** ociuhandu has joined #openstack-nova00:30
*** ociuhandu has quit IRC00:35
*** betherly has joined #openstack-nova00:35
*** betherly has quit IRC00:39
*** markvoelker has quit IRC00:42
*** markvoelker has joined #openstack-nova00:43
*** threestrands has joined #openstack-nova00:46
*** gyee has quit IRC00:49
*** _mmethot_ has joined #openstack-nova01:02
*** ricolin has joined #openstack-nova01:03
*** mmethot_ has quit IRC01:07
*** abaindur has quit IRC01:12
openstackgerritBrin Zhang proposed openstack/python-novaclient master: Microversion 2.76: Add user_id/project_id to migration-list API  https://review.opendev.org/67502301:16
openstackgerritBrin Zhang proposed openstack/python-novaclient master: Microversion 2.76: Add delete_on_termination to volume-attach API  https://review.opendev.org/67348501:24
*** betherly has joined #openstack-nova01:30
*** tbachman has quit IRC01:32
*** betherly has quit IRC01:35
*** markvoelker has quit IRC01:37
*** tetsuro has joined #openstack-nova01:40
openstackgerritMerged openstack/python-novaclient master: docs: clarify nova migration-list --host option  https://review.opendev.org/67495001:41
*** markvoelker has joined #openstack-nova01:44
openstackgerritBrin Zhang proposed openstack/python-novaclient master: Microversion 2.76: Add user_id/project_id to migration-list API  https://review.opendev.org/67502301:45
*** jhesketh has quit IRC01:53
*** tbachman has joined #openstack-nova01:53
openstackgerritMerged openstack/nova stable/stein: Don't generate service UUID for deleted services  https://review.opendev.org/67381401:53
*** bhagyashris has joined #openstack-nova01:57
*** betherly has joined #openstack-nova02:01
*** betherly has quit IRC02:05
*** slaweq_ has joined #openstack-nova02:11
*** tbachman has quit IRC02:12
*** slaweq_ has quit IRC02:15
*** jhesketh has joined #openstack-nova02:22
openstackgerritya.wang proposed openstack/nova master: Scheduler translate properties to traits  https://review.opendev.org/67452402:24
openstackgerritya.wang proposed openstack/nova master: Change method: _live_migration_flags  https://review.opendev.org/67452502:24
openstackgerritya.wang proposed openstack/nova master: Add releasenote and update docs.  https://review.opendev.org/67452602:24
*** tbachman has joined #openstack-nova02:35
*** betherly has joined #openstack-nova02:40
openstackgerritKuo-tung Kao proposed openstack/nova master: qemu-img run slowly without proper permission  https://review.opendev.org/67524102:40
*** tbachman has quit IRC02:53
*** betherly has quit IRC02:53
*** adriant has quit IRC02:54
*** betherly has joined #openstack-nova03:08
*** whoami-rajat has joined #openstack-nova03:12
*** igordc has joined #openstack-nova03:16
*** BjoernT has joined #openstack-nova03:17
*** tbachman has joined #openstack-nova03:18
*** betherly has quit IRC03:21
*** BjoernT has quit IRC03:21
*** BjoernT has joined #openstack-nova03:21
*** tetsuro has quit IRC03:25
*** jhesketh has quit IRC03:29
*** psachin has joined #openstack-nova03:32
*** jhesketh has joined #openstack-nova03:36
*** betherly has joined #openstack-nova03:39
*** betherly has quit IRC03:44
openstackgerritEric Fried proposed openstack/nova master: Provider config file schema and loader  https://review.opendev.org/67334103:48
openstackgerritGhanshyam Mann proposed openstack/nova master: Pass RequestContext to oslo_policy  https://review.opendev.org/67403803:50
eanderssonAnyone ever seen errors similar to04:02
eandersson> Failed to schedule instances: OrphanedObjectError_Remote04:02
eandersson> OrphanedObjectError: Cannot call obj_load_attr on orphaned Instance object04:04
*** tetsuro has joined #openstack-nova04:07
*** eharney has quit IRC04:08
*** udesale has joined #openstack-nova04:09
*** slaweq_ has joined #openstack-nova04:11
*** slaweq_ has quit IRC04:15
*** BjoernT has quit IRC04:17
*** mkrai has joined #openstack-nova04:19
*** eharney has joined #openstack-nova04:21
*** BjoernT has joined #openstack-nova04:26
*** betherly has joined #openstack-nova04:38
*** betherly has quit IRC04:42
*** adriant has joined #openstack-nova04:45
*** bbowen__ has quit IRC04:48
*** bbowen__ has joined #openstack-nova04:49
*** Luzi has joined #openstack-nova05:00
*** spsurya has joined #openstack-nova05:02
*** dave-mccowan has quit IRC05:03
*** BjoernT has quit IRC05:15
openstackgerritGhanshyam Mann proposed openstack/nova master: Pass RequestContext to oslo_policy  https://review.opendev.org/67403805:21
*** fnordahl has quit IRC05:23
*** lennyb has quit IRC05:26
*** betherly has joined #openstack-nova05:29
*** betherly has quit IRC05:37
*** fnordahl has joined #openstack-nova05:38
*** yaawang has quit IRC05:43
*** yaawang has joined #openstack-nova05:44
*** ratailor has joined #openstack-nova05:44
*** jaosorior has quit IRC05:56
*** shilpasd has joined #openstack-nova05:56
*** igordc has quit IRC05:59
*** betherly has joined #openstack-nova06:01
*** ccamacho has quit IRC06:03
*** betherly has quit IRC06:06
*** jaosorior has joined #openstack-nova06:10
*** slaweq_ has joined #openstack-nova06:11
*** mkrai has quit IRC06:13
*** mkrai has joined #openstack-nova06:15
*** slaweq_ has quit IRC06:16
*** rcernin has quit IRC06:18
openstackgerritGhanshyam Mann proposed openstack/nova master: Add test coverage of existing os-services policies  https://review.opendev.org/66918106:18
openstackgerritGhanshyam Mann proposed openstack/nova master: Introduce scope_types in os-services  https://review.opendev.org/64542706:19
openstackgerritGhanshyam Mann proposed openstack/nova master: Add new default roles and mapping in policy base class  https://review.opendev.org/64545206:19
openstackgerritGhanshyam Mann proposed openstack/nova master: Add new default roles in os-services API policies  https://review.opendev.org/64848006:19
*** dpawlik has joined #openstack-nova06:22
*** ralonsoh has joined #openstack-nova06:25
*** lennyb has joined #openstack-nova06:27
*** jaosorior has quit IRC06:29
*** tetsuro has quit IRC06:32
*** udesale has quit IRC06:41
*** udesale has joined #openstack-nova06:42
*** maciejjozefczyk has joined #openstack-nova06:49
*** slaweq_ has joined #openstack-nova06:59
*** betherly has joined #openstack-nova07:00
*** udesale has quit IRC07:03
*** ccamacho has joined #openstack-nova07:03
*** udesale has joined #openstack-nova07:03
*** aojea has joined #openstack-nova07:05
openstackgerritMerged openstack/nova stable/rocky: Replace non-nova server fault message  https://review.opendev.org/67484807:05
*** betherly has quit IRC07:05
*** tetsuro has joined #openstack-nova07:14
*** jaosorior has joined #openstack-nova07:16
*** xek has joined #openstack-nova07:17
*** tetsuro has quit IRC07:18
*** betherly has joined #openstack-nova07:20
*** tssurya has joined #openstack-nova07:23
*** betherly has quit IRC07:25
*** tesseract has joined #openstack-nova07:31
*** panda has quit IRC07:35
*** panda has joined #openstack-nova07:38
*** ociuhandu has joined #openstack-nova07:56
*** udesale has quit IRC07:57
*** udesale has joined #openstack-nova07:58
*** brault has joined #openstack-nova07:59
*** takashin has left #openstack-nova08:01
*** rpittau|afk is now known as rpittau08:04
*** takamatsu has quit IRC08:06
*** tetsuro has joined #openstack-nova08:17
*** ivve has joined #openstack-nova08:19
*** threestrands has quit IRC08:19
*** betherly has joined #openstack-nova08:22
*** helenafm has joined #openstack-nova08:24
*** derekh has joined #openstack-nova08:30
*** brinzhang has quit IRC08:32
*** brinzhang has joined #openstack-nova08:32
*** cdent has joined #openstack-nova09:07
openstackgerritStephen Finucane proposed openstack/nova-specs master: Correct nit with cpu-resources  https://review.opendev.org/67527609:13
openstackgerritLee Yarwood proposed openstack/nova master: libvirt: Remove native LUKS compat code  https://review.opendev.org/66912109:17
openstackgerritBalazs Gibizer proposed openstack/nova master: DNM: Serialize datetime in NovaObjectSerializer  https://review.opendev.org/67528109:41
*** bhagyashris has quit IRC09:44
*** udesale has quit IRC10:01
*** udesale has joined #openstack-nova10:01
*** udesale has quit IRC10:02
*** udesale has joined #openstack-nova10:03
*** huaqiang has quit IRC10:06
*** huaqiang has joined #openstack-nova10:07
openstackgerritAlex Xu proposed openstack/nova master: Add functional tests for resize with VGPU  https://review.opendev.org/67510610:14
*** kdean has quit IRC10:21
*** jaosorior has quit IRC10:26
*** ociuhandu has quit IRC10:26
*** udesale has quit IRC10:38
*** ociuhandu has joined #openstack-nova10:47
*** mkrai has quit IRC10:49
openstackgerritMerged openstack/os-vif master: only disable mac ageing for ovs hybrid plug  https://review.opendev.org/67283410:50
*** tetsuro has quit IRC11:02
*** sahid has joined #openstack-nova11:07
*** tbachman has quit IRC11:08
*** yaawang has quit IRC11:08
*** yaawang has joined #openstack-nova11:09
*** jaosorior has joined #openstack-nova11:10
*** aojea has quit IRC11:24
*** lpetrut has joined #openstack-nova11:34
*** tbachman has joined #openstack-nova11:35
*** lpetrut has quit IRC11:49
*** tssurya has quit IRC11:50
*** tssurya has joined #openstack-nova11:50
openstackgerritBalazs Gibizer proposed openstack/nova master: Test heal port allocations in nova-next  https://review.opendev.org/66987911:52
*** ratailor has quit IRC12:02
*** cdent has quit IRC12:04
*** ricolin_ has joined #openstack-nova12:05
*** ricolin has quit IRC12:07
*** takashin has joined #openstack-nova12:19
*** jaosorior has quit IRC12:22
*** ccamacho has quit IRC12:24
*** udesale has joined #openstack-nova12:28
openstackgerritShilpa Devharakar proposed openstack/nova master: Nova object changes for forbidden aggregates request filter  https://review.opendev.org/67107212:30
openstackgerritShilpa Devharakar proposed openstack/nova master: DB API changes to get non-matching aggregates from metadata  https://review.opendev.org/67107412:31
*** Luzi has quit IRC12:32
openstackgerritShilpa Devharakar proposed openstack/nova master: Add a new request filter to isolate aggregates  https://review.opendev.org/67107512:35
openstackgerritShilpa Devharakar proposed openstack/nova master: Enable request filter isolate_aggregates  https://review.opendev.org/66795212:37
*** jmlowe has quit IRC12:43
openstackgerritBalazs Gibizer proposed openstack/nova master: Error out interrupted builds  https://review.opendev.org/66685712:49
*** eharney has quit IRC12:51
*** cdent has joined #openstack-nova13:05
*** mriedem has joined #openstack-nova13:16
openstackgerritsean mooney proposed openstack/nova master: Libvirt: report storage bus traits  https://review.opendev.org/66691413:19
openstackgerritsean mooney proposed openstack/nova master: libvirt: use domain capablites to get supported device models  https://review.opendev.org/66691513:20
openstackgerritsean mooney proposed openstack/nova master: Add transform_image_metadata request filter  https://review.opendev.org/66577513:20
*** ccamacho has joined #openstack-nova13:23
*** jcosmao has joined #openstack-nova13:27
*** BjoernT has joined #openstack-nova13:28
*** slaweq_ is now known as slaweq13:29
*** eharney has joined #openstack-nova13:32
openstackgerritStephen Finucane proposed openstack/nova master: scheduler: Flatten 'ResourceRequest.from_extra_specs', 'from_image_props'  https://review.opendev.org/67489413:34
openstackgerritStephen Finucane proposed openstack/nova master: Add support for translating CPU policy extra specs, image meta  https://review.opendev.org/67180113:34
openstackgerritStephen Finucane proposed openstack/nova master: Add reshaper for PCPU  https://review.opendev.org/67489513:34
openstackgerritStephen Finucane proposed openstack/nova master: Remove 'hw:cpu_policy', 'hw:mem_page_size' extra specs from API samples  https://review.opendev.org/67533813:34
*** stephenfin has quit IRC13:35
*** stephenfin has joined #openstack-nova13:36
*** mchlumsky has joined #openstack-nova13:41
openstackgerritGhanshyam Mann proposed openstack/nova master: Add new default roles in os-services API policies  https://review.opendev.org/64848013:42
*** KeithMnemonic has quit IRC13:45
*** KeithMnemonic has joined #openstack-nova13:45
*** hemna has joined #openstack-nova13:47
*** maciejjozefczyk is now known as mjozefcz|lunch13:48
mriedemlyarwood: dansmith: we're going to be doing a rocky release so can i drop some rocky reviews on you? starting with https://review.opendev.org/#/q/topic:bug/1837955+branch:stable/rocky13:51
dansmithyeah13:51
lyarwoodyup post away in here and I'll get to them after this call13:54
mriedemhttps://review.opendev.org/#/c/674770/13:54
*** dave-mccowan has joined #openstack-nova13:54
mriedemdansmith: https://review.opendev.org/#/c/672747/13:55
*** ricolin_ is now known as ricolin13:55
mriedemwell or lee can upgrade to +W that one13:55
mriedemhttps://review.opendev.org/#/q/topic:bug/1764556+branch:stable/rocky13:56
dansmithmriedem: I'm not sure what https://review.opendev.org/#/c/674770/1 is actually fixing14:00
mriedemhttps://review.opendev.org/#/c/669393/14:00
mriedemdansmith: redundant db calls14:01
mriedemefried: meeting?14:01
efriedthanks mriedem, got distracted.14:02
dansmithmriedem: is it actually doing less work?14:02
*** yaawang has quit IRC14:02
mriedemthe bug report has the details14:02
*** yaawang has joined #openstack-nova14:03
mriedemoh you mean since it still calls group.get_hosts() right?14:03
dansmithI guess I need to dig further into what is going on under the covers, but it seems like this is just a tiny optimization, not really a fix no?14:04
dansmiththe bug seems to be "clean up dead code" or so14:05
*** brault has quit IRC14:05
*** liuyulong has quit IRC14:05
mriedemyes i think it is,14:05
mriedemotherwise we'd change to:14:05
mriedemgroup_hosts = group.hosts yeah?14:05
mriedemguess i never saw the original https://review.opendev.org/#/c/665954/14:07
dansmithI'm just not sure what the point is14:08
dansmithbut it seems like kinda pointless for a backport14:08
*** francoisp has joined #openstack-nova14:09
mriedemdropped my +214:09
mriedemthe point was to remove the redundant db query14:09
mriedemwhich it's not14:09
dansmithwhich redundant query? the hit to list cells first or something?14:10
mriedemgroup.get_hosts (and the scatter/gather thing)14:10
openstackgerritFrançois Palin proposed openstack/nova master: Add retry when attachment_delete or terminate_connection fails  https://review.opendev.org/66967414:17
mriedemstephenfin: https://review.opendev.org/#/c/583247/2 leads to a change in merge conflict, i'd like to abandon the entire series unless there is a valid reason we should move forward with adding code that's not used14:20
stephenfinmriedem: Kill it14:20
mriedemgladly14:21
stephenfin(Nice idea but I don't have time to finish it and guess he won't now)14:22
*** mjozefcz|lunch is now known as mjozefcz14:26
*** dave-mccowan has quit IRC14:30
*** ociuhandu has quit IRC14:30
*** ociuhandu has joined #openstack-nova14:31
*** cdent has quit IRC14:31
*** ociuhandu has quit IRC14:35
openstackgerritMatt Riedemann proposed openstack/nova stable/queens: Add functional recreate test for regression bug 1825537  https://review.opendev.org/67535514:37
openstackbug 1825537 in OpenStack Compute (nova) rocky "finish_resize failures incorrectly revert allocations" [Medium,In progress] https://launchpad.net/bugs/1825537 - Assigned to Matt Riedemann (mriedem)14:37
dansmithmriedem: okay I think I got them all, yeah?14:39
mriedemdansmith: yup i think so, thanks!14:40
*** mlavalle has joined #openstack-nova14:43
dansmithcool14:44
gibimriedem: I'm reading your grenade + resize comment in https://review.opendev.org/#/c/655721 Would it be enough to tag one of the resize tempest test with "smoke" to get the coverage you miss or we need to change some grenade logic as well?14:45
mriedemtagging a resize tempest test with smoke would make it run during grenade yes14:49
artomLol, we ain't very smart14:50
artomWhen we detach an interface, we catch a bunch of exceptions: https://github.com/openstack/nova/blob/master/nova/api/openstack/compute/attach_interfaces.py#L20914:50
artomWe then do some RPC14:51
artomhttps://github.com/openstack/nova/blob/master/nova/compute/api.py#L430514:51
artom... which is a cast: https://github.com/openstack/nova/blob/master/nova/compute/rpcapi.py#L57214:51
mriedemartom: the only thing that wouldn't be in that list is PortNotFound14:52
*** tbachman has quit IRC14:52
mriedemthe others are all failures from the api14:52
mriedemfor all i know some years ago it might have been a call, or the api might have looked up the port first14:52
artommriedem, ah, good point14:52
mriedemwho's not smart now14:53
artomYeah, I'm sure there's a valid historical reason for it14:53
artommriedem, when did I ever claim to be smart?14:53
artomIt was a just a cute "end state"14:53
mriedemhttps://review.opendev.org/#/c/130028/2/nova/api/openstack/compute/plugins/v3/attach_interfaces.py@15814:54
mriedemeli pointed this out in 201414:54
artom(Got reported as a bug downstream, that we can pass anything as a port ID and it still accepts it)14:54
artomHow in the hell did you pull that review from 5 years ago so fast14:54
openstackgerritShilpa Devharakar proposed openstack/nova master: Enable request filter isolate_aggregates  https://review.opendev.org/66795214:55
mriedempycharm annotation14:56
efriedmriedem: been seeing this weirdness of late https://logs.opendev.org/41/673341/10/check/nova-tox-functional-py36/b2b2fa5/testr_results.html.gz14:57
efriedis this the race you were talking about yesterday?14:57
mriedemno, but i don't remember talking about weird races yesterday either14:57
*** takashin has left #openstack-nova15:00
gibiefried, mriedem: this was seen in https://review.opendev.org/#/c/666857/ which was reviewed by both of you15:01
mriedemartom: it was all dansmith's fault https://review.opendev.org/#/c/21819/15:02
mriedemin...grizzly15:02
artomI do love it when that happens15:02
mriedemwell, note the size of the change15:02
mriedemDan Smith <danms@us.ibm.com>15:03
*** ociuhandu has joined #openstack-nova15:04
mriedemartom: push a patch to remove the PortNotFound handling if you want15:04
dansmithwhat's up now? I tried to catch exceptions around a rpc cast?15:04
mriedemdansmith: yes, and 6 years later much hilarity ensued15:04
artommriedem, I'd rather look up the port ID at the REST API level, and throw the 40415:04
artomWould that work?15:04
mriedemyou mean your customer would15:05
mriedemlet's front-load validate all incoming IDs because of one red hat customer!15:05
mriedemyay!15:05
artomI actually don't think there's a customer15:06
artomIt's more a matter of aligning with our api-ref docs: https://docs.openstack.org/api-ref/compute/?expanded=detach-interface-detail#detach-interface15:06
artomWhich state a 404 could happen15:06
artom... or we should change the doc15:06
mriedemtechnically the 404 is for the instance not being found15:06
mriedemthe port not being found is a 40015:06
mriedemaccording to what i've learned over the years from cdent and edleafe15:06
mriedemand the api sig15:06
mriedemmissing things that aren't the main resource are 400s15:07
edleafemriedem is a good student :)15:07
mriedemi only had to ask about 10 times15:07
artomAh, so since the api-ref doens't state a 400 is a possible response, we're technically good?15:08
artomStill feels weird15:08
artomAlthough I suppose there's precedent15:09
artomWith live-migration, for example15:09
stephenfinefried: You should look at this too, I'd say https://review.opendev.org/#/c/666032/15:12
* efried clicks15:12
efrieddansmith: mriedem: FYI I'm +2 all the way up on aggregate isolation https://review.opendev.org/#/c/671075/15:13
efriedstephenfin: I've been avoiding that one15:13
efriedI never understood the original spec15:13
efriedbut I'll have a look at some point if you get desperate.15:14
stephenfinefried: TBH, it's what I'm doing anyway so it really just needs a sanity check15:14
*** psachin has quit IRC15:15
efriedpoint is, if I do manage to review the code, I'll be doing it against whatever the spec says. So if I rubber-stamp the spec based on the fact that that's what you're coding... https://previews.123rf.com/images/sudowoodo/sudowoodo1708/sudowoodo170800024/84484656-ouroboros-symbol-snake-eating-its-own-tail-modern-alchemy-logo-vector-illustration-.jpg15:16
*** tbachman has joined #openstack-nova15:30
openstackgerritShilpa Devharakar proposed openstack/nova-specs master: Update spec: filtering of alloc candidates by forbidden aggregates  https://review.opendev.org/67538415:30
*** dave-mccowan has joined #openstack-nova15:32
*** spsurya has quit IRC15:34
aspiershow can I debug placement returning no allocation candidates on devstack, when debug is already set to True and showing nothing useful?15:37
aspiersGET /placement/allocation_candidates?limit=1000&required=%21COMPUTE_STATUS_DISABLED&resources=DISK_GB%3A20%2CMEMORY_MB%3A2048%2CVCPU%3A115:37
aspiersI've already gone through the three resource classes via openstack resource provider inventory show $rp_uuid15:38
aspierseach one is showing a total and max_unit greater than what is being asked for15:38
efriedaspiers: You'll want to show usages15:38
aspiersisn't that what I just did? ^^15:39
efriedYou showed inventories15:39
efriedinventories: "how big is it"15:39
efriedusages: "how much of it is consumed"15:39
aspiersok15:39
aspierssurely placement should offer to DEBUG-log details of why it's rejecting candidates?15:40
efriedthat's not nearly as easy as you would think15:40
aspiers"why am I getting NoValidHost" is the #1 FAQ I've ever heard from operators in 7 years of OpenStack15:40
efriedditto15:41
aspiersof course it wouldn't be ;-/15:41
aspiersotherwise someone would have done it, I guess15:41
efriedAnd this would be an absolutely overwhelming amount of data15:41
aspierswell sure15:41
efriedbut mainly15:41
aspiersthat's why it would be at DEBUG level15:41
efriedwe reject the vast majority of things based on a database query15:41
aspierswell even logging what's left after that would be a start15:42
efriedwe do have some debug logs for that15:42
efriedand the fact that you're not seeing any means... we've done all the filtering at the db level :)15:42
aspiersok15:42
aspierslooks like I've used 20 out of 35 DISK_GB and am asking for another 2015:43
aspiersnext question is why is it so damn low15:43
efriedthat'd do it15:43
*** tssurya has quit IRC15:43
*** gyee has joined #openstack-nova15:43
efriedyou can show allocations15:43
*** ricolin_ has joined #openstack-nova15:43
aspiers"openstack resource provider allocation show $rp_uuid" shows nothing15:43
efriedallocation show takes an instance UUID15:44
efriedI think you should be able to do allocation list15:44
openstackgerritMatt Riedemann proposed openstack/nova master: Fix rebuild of baremetal instance when vm_state is ERROR  https://review.opendev.org/52355915:44
mriedemmelwitt: addressed your comments i think ^15:44
mriedemefried: or a migration uuid :)15:44
edleafeartom: mriedem: I just looked at the API you were discussing earlier for missing ports, and I think it *should* be a 404, since the resource you are requesting is the port interface. 400 would be correct if you were searching for a server that had that port.15:44
mriedemjroll: TheJulia: could use ironic subteam re-review on this https://review.opendev.org/52355915:44
*** helenafm has quit IRC15:45
mriedemedleafe: both are on the path15:45
mriedemserver_id and port_id15:45
mriedem404 cool for all things on the path then?15:45
melwittmriedem: that's a blast from the past. will have to reload a lot of context when I go back to review15:45
*** ricolin_ has quit IRC15:45
*** ricolin has quit IRC15:46
*** ricolin_ has joined #openstack-nova15:46
edleafemriedem: is this the GET or DELETE call?15:46
mriedemedleafe: anyway, it's essentially dead code anyway15:46
mriedemDELETE15:46
*** ricolin_ is now known as ricolin15:47
mriedemhttps://docs.openstack.org/api-ref/compute/#detach-interface15:47
edleafeThe call doesn't delete the server. The resource it is concerned with is the port. If that resource doesn't exist, then 404 is correct15:47
aspiersefried: I mean, why is the *total* inventory so damn low, not the usage15:47
aspiersI guess that's in n-cpu15:47
efriedyes15:48
mriedemedleafe: ack15:48
edleafeOf course, if you subscribe to the idempotent DELETE school of thought, 204 would be correct15:48
efriedaspiers: I assume libvirt driver?15:48
mriedemlike i said, it's dead code at this point unless we do a proxy neutronv2.api.show_port call on that port_id15:48
openstackgerritShilpa Devharakar proposed openstack/nova-specs master: Update spec: filtering of alloc candidates by forbidden aggregates  https://review.opendev.org/67538415:48
openstackgerritBalazs Gibizer proposed openstack/nova master: allow getting resource request of every bound ports of an instance  https://review.opendev.org/65511015:49
aspiersefried: virsh shows no pools15:49
efriedI don't know what that means15:49
aspiersvirsh pool-list15:49
efriedlook at _get_local_gb_info15:49
aspiersok thanks15:49
efriedwhat's your [libvirt]images_type?15:49
mriedemartom: you know, could just move this validation to the api https://github.com/openstack/nova/blob/master/nova/compute/manager.py#L637315:50
sean-k-mooneyaspiers: i dont think openstack uses libvirt storage pools15:50
aspierssean-k-mooney: yeah that was a guess15:50
mriedemcompute is already checking the info cache before doing anything with the port at the neutron level15:50
sean-k-mooneyaspiers: mybe we might for lvm image backend? but im pretty sure tis not used for qcow/raw/flat iamge type15:51
aspierslibvirt.images_type            = default15:51
efriedaspiers: assuming your libvirt.images_type is unset or anything other than lvm/rbd, it'll be looking at the free space on your instances_path15:52
aspiersthat makes more sense15:52
efriedaspiers: ...which looks like it'll be by default whatever file system your nova installation is15:54
efriedif your nova is in /foo/bar/nova, your instances_path will by default be /foo/bar/instances, I think.15:54
efriedso15:55
efrieddf -h /foo/bar ought to tell you how much space it thinks you have.15:55
openstackgerritBalazs Gibizer proposed openstack/nova master: Pass network API to the conducor's MigrationTask  https://review.opendev.org/65511115:55
aspiersyeah it's basic sysadmin / lvm stuff now15:55
aspiersI'll take it from here. thanks for pointers :)15:56
efriedgood luck15:56
openstackgerritMatt Riedemann proposed openstack/nova master: Fix rebuild of baremetal instance when vm_state is ERROR  https://review.opendev.org/52355915:57
openstackgerritMatt Riedemann proposed openstack/nova master: WIP: run test_rebuild_server_in_error_state in ironic CI job  https://review.opendev.org/67539115:57
openstackgerritMatt Riedemann proposed openstack/nova master: WIP: run test_rebuild_server_in_error_state in ironic CI job  https://review.opendev.org/67539115:58
*** mjozefcz has quit IRC16:00
*** sahid has quit IRC16:13
*** brault has joined #openstack-nova16:15
mriedemgibi: efried: seeing another notification sample race here https://logs.opendev.org/36/614036/46/check/nova-tox-functional/b505ffb/testr_results.html.gz16:15
mriedemidk if the db stuff in there is just noise16:16
*** beekneemech has joined #openstack-nova16:16
*** beekneemech has quit IRC16:16
mriedembtw, unrelated to that, i noticed we have a novnc test that fails a lot in multinode jobs http://status.openstack.org/elastic-recheck/#166946816:17
*** brault has quit IRC16:19
openstackgerritMerged openstack/nova stable/rocky: Avoid logging traceback when detach device not found  https://review.opendev.org/67406816:19
openstackgerritMerged openstack/nova stable/rocky: Add functional regression test for bug 1837955  https://review.opendev.org/67353616:19
openstackbug 1837955 in OpenStack Compute (nova) rocky "MaxRetriesExceeded sometime fails with messaging exception" [Medium,In progress] https://launchpad.net/bugs/1837955 - Assigned to Matt Riedemann (mriedem)16:19
openstackgerritMerged openstack/nova stable/rocky: Cleanup when hitting MaxRetriesExceeded from no host_available  https://review.opendev.org/67355316:22
artommriedem, ack16:25
artommriedem, although. Since we should be raising a 400 in the port not found case, technically it's a new microversion16:26
artomSo, being the lazy schmuck that I am, do I want to bother?16:26
mriedemartom: you missed edleafe saying it should be a 40416:26
artomAh, so I did.16:27
openstackgerritMatt Riedemann proposed openstack/nova master: Dump versioned notifications when len assertions fail  https://review.opendev.org/67540416:29
mriedem^ can help with debugging racy versioned notification functional test failures ^16:29
*** cdent has joined #openstack-nova16:29
mriedemthis is very weird https://logs.opendev.org/32/634832/43/check/nova-tox-functional-py36/d4f3be5/testr_results.html.gz16:33
openstackgerritMatt Riedemann proposed openstack/nova master: Dump versioned notifications when len assertions fail  https://review.opendev.org/67540416:33
mriedemthe test is disabling a service and asserting the notifications,16:33
mriedembut in the err output there is a failing resize happening b/c of not using the neuron fixture16:34
mriedembut the test isn't doing anything with resize16:34
*** rpittau is now known as rpittau|afk16:36
mriedemhttp://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22ERROR%20%5Bnova.network.neutronv2.api%5D%20The%20%5Bneutron%5D%20section%20of%20your%20nova%20configuration%20file%20must%20be%20configured%20for%20authentication%20with%20the%20networking%20service%20endpoint.%5C%22%20AND%20tags%3A%5C%22console%5C%22&from=7d16:39
mriedemlooks like it started around 8/516:40
mriedemso i wonder if it's this https://github.com/openstack/nova/commit/8db712fe040b15f2b8bc5538338658d3aac246e316:40
*** jangutter has quit IRC16:44
openstackgerritAdam Spiers proposed openstack/nova master: Provide HW_CPU_X86_AMD_SEV trait when SEV is supported  https://review.opendev.org/63868016:47
openstackgerritAdam Spiers proposed openstack/nova master: Add extra spec parameter and image property for memory encryption  https://review.opendev.org/66442016:47
openstackgerritAdam Spiers proposed openstack/nova master: Extract SEV-specific bits on host detection  https://review.opendev.org/63633416:47
openstackgerritAdam Spiers proposed openstack/nova master: Add <launchSecurity> and <driver iommu='on' /> to config.py  https://review.opendev.org/63631816:47
openstackgerritAdam Spiers proposed openstack/nova master: Apply SEV-specific guest config when SEV is required  https://review.opendev.org/64456516:47
openstackgerritAdam Spiers proposed openstack/nova master: Enable booting of libvirt guests with AMD SEV memory encryption  https://review.opendev.org/66661616:47
mriedemhttps://bugs.launchpad.net/nova/+bug/183951516:48
openstackLaunchpad bug 1839515 in OpenStack Compute (nova) "Weird functional test failures hitting neutron API in unrelated resize flows since 8/5" [Undecided,New]16:48
mriedemgibi: ^ your recent functional compute service restart test might have a similar issue16:48
mriedemhttps://review.opendev.org/#/c/667913/ i mean16:48
*** Sundar has joined #openstack-nova16:50
openstackgerritAdam Spiers proposed openstack/nova master: Allow assertXmlEqual() to pass options to matchers.XMLMatches  https://review.opendev.org/67462816:52
openstackgerritAdam Spiers proposed openstack/nova master: Fix libvirt driver tests to use LibvirtConfigCapsGuest instances  https://review.opendev.org/67462916:52
openstackgerritAdam Spiers proposed openstack/nova master: Track libvirt host/domain capabilities for multiple machine types  https://review.opendev.org/67315116:52
openstackgerritAdam Spiers proposed openstack/nova master: Provide HW_CPU_X86_AMD_SEV trait when SEV is supported  https://review.opendev.org/63868016:52
openstackgerritAdam Spiers proposed openstack/nova master: Add extra spec parameter and image property for memory encryption  https://review.opendev.org/66442016:52
openstackgerritAdam Spiers proposed openstack/nova master: Extract SEV-specific bits on host detection  https://review.opendev.org/63633416:52
openstackgerritAdam Spiers proposed openstack/nova master: Add <launchSecurity> and <driver iommu='on' /> to config.py  https://review.opendev.org/63631816:52
openstackgerritAdam Spiers proposed openstack/nova master: Apply SEV-specific guest config when SEV is required  https://review.opendev.org/64456516:52
openstackgerritAdam Spiers proposed openstack/nova master: Enable booting of libvirt guests with AMD SEV memory encryption  https://review.opendev.org/66661616:52
*** Sundar has quit IRC16:54
*** derekh has quit IRC17:00
openstackgerritMerged openstack/nova stable/rocky: Revert "[libvirt] Filter hypervisor_type by virt_type"  https://review.opendev.org/67274717:02
openstackgerritMerged openstack/nova stable/rocky: Drop source node allocations if finish_resize fails  https://review.opendev.org/66939317:02
openstackgerritMerged openstack/nova stable/rocky: Add functional recreate test for bug 1764556  https://review.opendev.org/67381617:02
openstackbug 1764556 in OpenStack Compute (nova) rocky ""nova list" fails with exception.ServiceNotFound if service is deleted and has no UUID" [Medium,In progress] https://launchpad.net/bugs/1764556 - Assigned to Matt Riedemann (mriedem)17:02
openstackgerritMerged openstack/nova stable/rocky: Add functional regression test for bug 1778305  https://review.opendev.org/67381817:02
openstackbug 1778305 in OpenStack Compute (nova) rocky "Nova may erronously look up service version of a deleted service, when hostname have been reused" [Medium,In progress] https://launchpad.net/bugs/1778305 - Assigned to Matt Riedemann (mriedem)17:02
*** BjoernT has quit IRC17:02
*** BjoernT has joined #openstack-nova17:03
openstackgerritMatt Riedemann proposed openstack/nova stable/rocky: Don't generate service UUID for deleted services  https://review.opendev.org/67382117:03
*** BjoernT has quit IRC17:03
*** BjoernT has joined #openstack-nova17:05
*** BjoernT has quit IRC17:05
*** BjoernT has joined #openstack-nova17:06
*** BjoernT has quit IRC17:06
*** udesale has quit IRC17:08
*** BjoernT has joined #openstack-nova17:08
*** BjoernT has quit IRC17:09
*** BjoernT has joined #openstack-nova17:09
*** BjoernT has quit IRC17:10
*** BjoernT_ has joined #openstack-nova17:10
*** BjoernT_ has quit IRC17:10
*** BjoernT has joined #openstack-nova17:11
*** BjoernT has quit IRC17:11
sean-k-mooneyefried: mriedem is tehre a target milestone for the U release yet?17:12
*** BjoernT_ has joined #openstack-nova17:12
*** markvoelker has quit IRC17:12
*** BjoernT_ has quit IRC17:12
efriedsean-k-mooney: what do you mean?17:12
openstackgerritMatt Riedemann proposed openstack/nova master: Skip test_migrate_disk_and_power_off_crash_finish_revert_migration  https://review.opendev.org/67541717:13
mriedemgibi: efried: ^17:13
*** igordc has joined #openstack-nova17:13
mriedemskip that functional test for now since it's leaking into other tests on the same worker and failing17:13
mriedemsean-k-mooney: https://releases.openstack.org/17:13
mriedemsean-k-mooney: you mean release date?17:13
mriedemthat's a question for ttx probably17:13
mriedemor smcginnis17:13
sean-k-mooneywell im guessing this should be targeted at U since i doubt it will be appoved for Train with a feature exception https://blueprints.launchpad.net/nova/+spec/sriov-numa-affinity-policy-via-flavor-and-image17:14
sean-k-mooneymriedem: no i mentioned i the whiteboard that this should proably be for U release17:14
*** ociuhandu has quit IRC17:14
mriedemthere is no series goal in launchpad b/c we don't know what the name of the release is going to be17:14
sean-k-mooneybut i was wondering if the openstack Next milstone means next milsteon or next release17:14
mriedemsee the shenanigans in the ml17:14
*** BjoernT has joined #openstack-nova17:15
*** BjoernT has quit IRC17:15
*** BjoernT has joined #openstack-nova17:15
sean-k-mooneymriedem: ya i know i was wondering what "milestone target" was. anyway its in the whiteboard but that is the bluepint for the sriov numa stuff we talked about yesterday17:15
openstackgerritArtom Lifshitz proposed openstack/nova master: [WIP] Move port ID verification to REST API  https://review.opendev.org/67541817:16
mriedemack17:16
efriedmriedem: would the right thing be to kill the sleeping thread?17:16
*** BjoernT has quit IRC17:16
mriedemefried: yeah something like that17:16
artomBleah, took forever because I forgot I was in the middle of a rebase, but there ^^17:16
mriedemrather than time.sleep, do a wait/notify or something17:16
*** ricolin_ has joined #openstack-nova17:16
artomI'll let CI tell me which unit tests I broke17:16
*** BjoernT has joined #openstack-nova17:16
*** BjoernT has quit IRC17:17
*** BjoernT_ has joined #openstack-nova17:17
*** BjoernT_ has quit IRC17:17
*** BjoernT has joined #openstack-nova17:18
mriedemeasy -117:18
*** BjoernT has quit IRC17:18
mriedem:)17:18
artomDammi, I said [WIP] and everything :P17:19
*** ricolin has quit IRC17:19
mriedemi need the stackalytics points17:19
*** BjoernT has joined #openstack-nova17:19
artomWhat's their exchange rate for real currencies?17:20
*** BjoernT has quit IRC17:20
*** BjoernT_ has joined #openstack-nova17:20
*** BjoernT_ has quit IRC17:20
sean-k-mooneyif review were a crypto currancy mriedem would never need to work again17:21
mriedemhalf a looney17:21
*** BjoernT has joined #openstack-nova17:21
*** BjoernT has quit IRC17:21
mriedemi don't review much anymore b/c all anyone pushes up for review these days is low level hw optimizations17:21
sean-k-mooneyuntill the gpu powered review bot take over and +1 everaything and make the crypto currancy collapes17:21
artomWe don't have those, that'd be two caribous17:21
*** BjoernT has joined #openstack-nova17:22
*** BjoernT has quit IRC17:22
artommriedem, that's where the money is :/17:22
*** BjoernT has joined #openstack-nova17:22
*** BjoernT has quit IRC17:23
*** BjoernT_ has joined #openstack-nova17:23
sean-k-mooneymriedem: given you still the numer 2 reivewer in nova this cycle what would you consider reviewing a lot lol17:24
artom...17:24
*** tssurya has joined #openstack-nova17:24
artomWho's number 1 o_O17:24
sean-k-mooneyartom: efried currenlty17:24
artomDamn mriedem dude, you're slackin'17:25
artom(You're not, don't worry)17:25
efriedyeah, now look at how much code17:25
* efried <== ι17:26
*** markvoelker has joined #openstack-nova17:26
*** tesseract has quit IRC17:27
*** BjoernT_ has quit IRC17:28
*** ricolin_ is now known as ricolin17:32
sean-k-mooneyhow would peopel feel about using a [U Release] tag in commit title for stuff that is for next cycle17:34
sean-k-mooneye.g. i want to push up some patches but dont really need/want them to be review until we get to U17:35
sean-k-mooneyor maybe [next-release]17:35
dansmithhow about [WIP] ? :D17:36
sean-k-mooneykindo of like th [WIP] and [DNM] tags we use17:36
dansmithif it's not to be merged right away, then WIP or DNM fit pretty well17:36
sean-k-mooneywell [WIP] does not signel its not for train17:36
dansmithno?17:36
*** ociuhandu has joined #openstack-nova17:36
dansmithWIP signals to me "you can review this right now, but it's not ready"17:36
dansmithwhich seems about the same AFAICT17:36
sean-k-mooneyok17:36
sean-k-mooneyi can use [WIP] and [DNM]17:37
sean-k-mooneybut ya that was the intent. "you can review if you want but this is not planned to be merged soon"17:38
dansmithyup17:39
*** shilpasd has quit IRC17:39
*** ociuhandu has quit IRC17:40
efriedIt's not like we have any automation on the tags17:44
efried[WIP:U] would be fine as well17:44
mriedemsean-k-mooney: or just push it up tied to the blueprint and i'll procedurally -2 it17:44
mriedemlike we do sometimes for unapproved blueprints17:44
sean-k-mooneysure its just nice to have a convention17:44
sean-k-mooney:) yes that is always an option17:44
openstackgerritGhanshyam Mann proposed openstack/nova master: Add test coverage of existing admin_actions policies  https://review.opendev.org/65769817:45
*** panda has quit IRC17:53
*** cdent has quit IRC17:53
*** ralonsoh has quit IRC17:54
*** panda has joined #openstack-nova17:54
*** BjoernT has joined #openstack-nova17:57
*** psachin has joined #openstack-nova18:08
*** bbowen__ has quit IRC18:09
*** psachin has quit IRC18:11
*** nweinber has joined #openstack-nova18:12
sean-k-mooneyby the way anyone recognise what package these deprecation warning come from18:25
sean-k-mooneyhttp://paste.openstack.org/show/755668/18:25
sean-k-mooneyim guessing oslo somthing18:25
sean-k-mooneyhttps://github.com/openstack/oslo.service maybe18:26
*** igordc has quit IRC18:28
sean-k-mooneyoh its form the openstack sdk18:28
sean-k-mooneyand mriedem fixed it 6 days ago https://github.com/openstack/openstacksdk/commit/799bd0a773fb082c1a45262922d87630abc9bd3018:29
*** mchlumsky has quit IRC18:39
mriedemyeah, would be nice to get that released, mordred ^18:48
mriedemotherwise we should add a warnings filter to squash that noise from our test logs18:48
mordredmriedem: I have a release request in the releaseteam queue18:49
mriedemcool18:49
mordredmriedem: https://review.opendev.org/#/c/674660/18:49
efriedmriedem: gate failures are killinme, any problem if I fast approve those two patches to add logs and skip the test?18:56
mriedemefried: i'm assuming dansmith or melwitt could hit these? https://review.opendev.org/#/q/topic:bug/1839515+status:open18:59
dansmithit hath been done19:00
mriedemdanke19:00
efriedthanks dansmith19:00
mriedemshould see this drop off once that skip merges http://status.openstack.org/elastic-recheck/#183951519:03
mriedemmordred: +1 on that release request - could have probably been 0.32.1 given the contents but not a big deal when you're <1.0.019:06
mordredmriedem: yeah - but numbers are cheap too :)19:08
* mriedem looks for an accountant joke19:09
openstackgerritsean mooney proposed openstack/nova master: [WIP:U] support pci numa affinity policies in flavor and image  https://review.opendev.org/67407219:15
*** BjoernT has quit IRC19:15
*** BjoernT has joined #openstack-nova19:16
tbachmanmriedem: Ben Affleck in “The Accountant”19:17
dansmithhah19:17
mriedemtbachman: i thought of that movie too19:19
tbachmanheh19:19
mriedemand my neighbor who won't make eye contact19:19
mriedemand is a CPA19:19
tbachmansteer clear of them19:19
*** brinzhang has quit IRC19:20
*** BjoernT has quit IRC19:20
*** brinzhang has joined #openstack-nova19:20
*** ivve has quit IRC19:29
*** noonedeadpunk has quit IRC19:42
*** noonedeadpunk has joined #openstack-nova19:43
*** bbowen__ has joined #openstack-nova19:44
mriedemlordy test_verify_created_server_ephemeral_disk sometimes takes 5 minutes19:56
*** panda has quit IRC19:57
sean-k-mooneyis that a functional test?19:57
sean-k-mooneyor i guess proably a tempest senario test given the lenght19:58
mriedemtempest19:58
*** eharney has quit IRC19:58
mriedemit's not a scenario test, api test19:58
mriedemshould probably be a scenario test19:58
sean-k-mooneyor at least in the slow set19:58
mriedemgmann: what do you think about dropping the --serial for running scenario tests serially in tempest-integrated-compute ?19:58
mriedemlooking at https://logs.opendev.org/78/658478/16/check/tempest-integrated-compute/33847ea/controller/logs/stackviz/#/stdin/timeline?test=tempest.scenario.test_network_basic_ops.TestNetworkBasicOps.test_hotplug_nic there are only 2 scenario tests that get run19:58
*** panda has joined #openstack-nova19:59
mriedemhmm, why aren't things like test_stamp_pattern and test_volume_boot_pattern running in there?19:59
mriedemoh b/c it's marked slow20:00
mriedemnvm20:00
sean-k-mooneyany tests that ssh into a vm shoudl be grouped under senario but ya i guess it could be filtered out20:01
mriedemsean-k-mooney: that's the way things used to be long ago,20:01
mriedemqa team enabled run_validation in api tests quite awhile ago,20:01
mriedemso a lot of the api tests are also ssh'ing into guests20:02
sean-k-mooneywell run validation is different20:02
mriedemrun_validation == enable ssh20:02
sean-k-mooneyin that its optional20:02
mriedemsure, but it's basically on by default in tempest runs in the gate20:02
sean-k-mooneye.g the test shoudl still be runale without it20:02
sean-k-mooneyya20:02
*** noonedeadpunk has quit IRC20:03
mriedemwe should probably disable ssh validation in the nova-next job20:04
mriedemwe already get that in tempest-full and tempest-slow20:04
sean-k-mooneywell its realy jsut intened to prevent race conditions20:05
sean-k-mooneye.g. in senario tests it can help with cases were you need to wait for the guest os to start before you proceed with the test20:05
mriedemi don't think we're talking about the same thing20:05
sean-k-mooneythat is what the validation was ortinglly for as far as i knew20:06
mriedemare you thinking of the nova vif_plugging_* options?20:06
sean-k-mooneyno20:06
mriedemtempest uses it for all kinds of shit https://github.com/openstack/tempest/search?q=CONF.run_validation&unscoped_q=CONF.run_validation20:06
sean-k-mooneyyes but if i rememebr correctly it was originally added to make test pause on server create until you could ssh in20:08
sean-k-mooneythis was need for running tempst against ironic clouds20:09
donnydis the lxc driver for nova still functional?20:09
sean-k-mooneyam kind of20:09
sean-k-mooneyit was broke for 2 years20:09
sean-k-mooneyand got fixed recnetly20:09
sean-k-mooneyas in like a month ago20:10
sean-k-mooneybut we have not ci so maybe?20:10
donnydI see someone asking about lxd on the ml and was going to ask  if we could point them at lxc instead... but figured i should ask the state of it first20:10
sean-k-mooneycannoical are stoping developing nova-lxd and i dont think tere was much of a comunity outside of them maintaining it20:11
sean-k-mooneythe nova libvirt/lxc backend has little use or testing as far as i know20:12
mriedemi'd be surprised if the lxc driver didn't shit the bed under normal usage20:13
*** tbachman has quit IRC20:13
donnydLOL mriedem20:13
*** tbachman has joined #openstack-nova20:13
sean-k-mooneydonnyd: the reason it was broken for ~2years was we change the args to mount20:13
mriedemhonestly, i tried getting a ci job going for that thing years ago and something would fail during server delete and it would just snowball and fail everything20:13
donnydseriously, that made me laugh harder than I should have20:13
mriedemnbd mount issues from what i remember20:13
mriedemdonnyd: we could easily get a simple ci job setup for it which just runs a simple create/delete server scenario20:14
mriedemand see what explodes20:14
donnydI have seen a couple lxd requests come across the ml recently and was curious to the state of it.20:14
mriedemsince nova-lxd isn't in tree it's not really our concern20:15
mriedemour == nova20:15
donnydyea, makes sense20:15
mriedembut yeah, as soon as someone kills support for something everyone comes out of the weeds20:15
donnydrather point them to something in tree anyways20:15
mriedem"i was just about to go production!"20:15
sean-k-mooneyfor what its worth i do like lxc. and it would actully be nice for developement and testing20:15
donnydI use them all the time for statefully things20:16
mriedemosc is still all lxc right?20:16
mriedemit's kolla that uses docker?20:16
donnydI think OSA is20:16
sean-k-mooneyim a bad conter person and jsut use docker for statefull things because i now how it works and can bend it to my will20:16
mriedemyeah sorry, osa20:16
donnydyea, kolla, 3O20:16
sean-k-mooneymriedem: kolla use docker yes20:17
sean-k-mooneyand osa used to support lxc for contol plane20:17
donnydI am pretty sure it still uses lxc, but i am not an osa expert so don't beat me too badly if they switched to docker20:17
sean-k-mooneyso osa used to install the compute nodes to bare metal but allow you to install the controler services in lxc20:17
donnydIt would be great if we could get our CI to use LXC... at least for my shabby little cloud. I could probably do more20:19
sean-k-mooneythe first intel nfv ci was interesting... it used docker(pre 1.0) with the lxc backend to spin up super privaldges contaienr with which we then ran devstack in20:21
sean-k-mooneywe got really good density but it was entirly custom hacking with ducktape and sting20:21
donnydwell duct tape does work... just sayin20:22
*** nweinber has quit IRC20:23
sean-k-mooneyit allowed use to handel every patch to neuton and respond back in less then 2 hours on a singel server with 40 cores and 128GB of ram20:23
*** boxiang has quit IRC20:23
*** boxiang has joined #openstack-nova20:24
sean-k-mooneyour nova ci that used vms to 3-4 servers to do the same with kvm20:24
donnydyea osa still uses lxc20:24
donnydi have 520 core and 1.1tb of memory here and I can only do 120 instances for nodepool20:25
sean-k-mooneyyour ram limited20:26
*** betherly has quit IRC20:27
donnydwell I did have 2Tb of memory, but the other 1tb was unused20:27
sean-k-mooneyare you pinning the vms or disableing cpu oversuciption20:28
donnydseems like the right balance for my setup is 2:1 in cpu oversubscription20:28
donnydno20:28
sean-k-mooneywe used to run 4:1 for the old intel ci20:29
sean-k-mooneyand used hugepages for the memroy20:29
*** tbachman has quit IRC20:29
donnydI would oversubscribe more, but when i do nodepool spins up every tempest job it has in the pool and hands them all to the same hypervisor20:29
donnydliterally burned up a cpu a few weeks back20:30
sean-k-mooneythat sound like you dont have either enough cooling or node pool is not configured correctly20:30
donnydservers run at 79 degrees20:31
donnydmaybe a little on the warm side, but not too bad20:31
donnydi can't control what the CI nodepool does20:31
sean-k-mooneywell you can limit the quota it has on your provider20:32
*** ivve has joined #openstack-nova20:32
sean-k-mooneyor you can limite the instance count on each server20:32
donnydYes and yes - but if I take the ratio to 4:1 it will load all the tempest jobs it can find onto a single compute node.. (there is a great need for a sarcasm font)20:34
sean-k-mooneyyou can tweek the weigher to spread by default20:34
donnydI do have the quota limited to 70 right now because I am adding in local nvmes20:34
sean-k-mooneyinstead of pack20:34
sean-k-mooneydonnyd: set https://docs.openstack.org/nova/latest/configuration/config.html#filter_scheduler.cpu_weight_multiplier to ~8 and https://docs.openstack.org/nova/latest/configuration/config.html#filter_scheduler.ram_weight_multiplier to ~1020:38
donnydI don't have someone elses money to buy more compute nodes, so I try to lean more toward the conservative side20:38
sean-k-mooneyoh you only have a single host20:38
donnydno, I have 820:38
donnydi just don't want to replace anymore of them20:39
donnydLOL20:39
sean-k-mooneyright. again it should be safe but its totally up to you.20:39
donnydAnd we were seeing high counts of timeouts with the ratios up higher20:39
sean-k-mooneythe point is when you change the allocation ratio you also want to adjust the multipliers20:40
donnydthat also could have been from storage20:40
donnydoh ic20:40
*** xek has quit IRC20:41
donnydthanks for the help on the lxc thing sean-k-mooney20:42
*** betherly has joined #openstack-nova20:48
*** abaindur has joined #openstack-nova20:50
donnydand i learned something new about nova today :)20:50
*** betherly has quit IRC20:52
mriedemcouple of queens backports that could use a +W here https://review.opendev.org/#/q/status:open+project:openstack/nova+branch:stable/queens+label:Code-Review=220:57
*** tjgresha has joined #openstack-nova20:58
*** noonedeadpunk has joined #openstack-nova20:59
*** whoami-rajat has quit IRC21:01
*** rm_work has quit IRC21:05
*** rm_work has joined #openstack-nova21:05
*** betherly has joined #openstack-nova21:08
*** betherly has quit IRC21:12
*** igordc has joined #openstack-nova21:13
*** abaindur has quit IRC21:25
*** abaindur has joined #openstack-nova21:25
*** betherly has joined #openstack-nova21:28
*** vishwanathj has joined #openstack-nova21:31
*** betherly has quit IRC21:33
mriedembnemec: is there anything like this in some oslo lib? https://github.com/openstack/nova/blob/master/nova/utils.py#L53021:34
bnemecmriedem: Not that I'm aware of.21:35
bnemecWhich is not necessarily the same thing as "no". :-)21:35
*** markvoelker has quit IRC21:36
*** betherly has joined #openstack-nova21:36
mriedemyeah it's ok21:36
efriedmriedem: fwiw, that's effectively the same as21:38
efriedwith mock.patch.object(...):21:38
efriedit would be a little weird to use mock in prod code... but it's doing the same thing.21:39
mriedemyeah21:39
mriedemthis isn't prod code, novaclient functional tests, but not a big deal21:40
*** betherly has quit IRC21:40
*** takashin has joined #openstack-nova21:47
sean-k-mooneydo release notes need to use external link to like to the docs since they are built seperatly? i assume yes21:52
*** markvoelker has joined #openstack-nova21:53
*** betherly has joined #openstack-nova21:57
*** markvoelker has quit IRC21:57
efriedsean-k-mooney: yes22:01
*** betherly has quit IRC22:01
efriedbuilt separately and also tied to their own release22:01
efriedAt some point I looked into what would be involved in supporting :ref:/:doc:/:oslo.config:*:22:02
efriedand it was a nonstarter.22:02
sean-k-mooneycool22:02
*** panda has quit IRC22:03
efriedhasta friday cats o/22:03
*** panda has joined #openstack-nova22:03
sean-k-mooneyo/22:03
openstackgerritMatt Riedemann proposed openstack/python-novaclient master: Add --migration-type and --source-compute to migration-list  https://review.opendev.org/67511722:07
openstackgerritsean mooney proposed openstack/nova master: Deprecate Aggregate[Core|Ram|Disk]Filters  https://review.opendev.org/67349622:08
sean-k-mooneymriedem: i think i have fixed the issues you noted.22:08
sean-k-mooneyim off tomorow so if you see anything else ill take a look on monday22:09
sean-k-mooneywith that i think ill call it a night o/22:09
mriedem-1!!!22:09
mriedemyour sleep depravation has gotten the better of you22:10
* mriedem cashes in on the sweet stats22:10
sean-k-mooneyhaha i mess that up22:11
sean-k-mooneyi was working on image meta data suff before22:11
openstackgerritsean mooney proposed openstack/nova master: Deprecate Aggregate[Core|Ram|Disk]Filters  https://review.opendev.org/67349622:13
sean-k-mooneyno still wrong22:13
openstackgerritMatt Riedemann proposed openstack/python-novaclient master: Add --migration-type and --source-compute to migration-list  https://review.opendev.org/67511722:13
openstackgerritMerged openstack/nova stable/pike: Implement power_off/power_on for the FakeDriver  https://review.opendev.org/67487622:14
openstackgerritsean mooney proposed openstack/nova master: Deprecate Aggregate[Core|Ram|Disk]Filters  https://review.opendev.org/67349622:14
*** slaweq has quit IRC22:15
sean-k-mooneyok fixed good catch o/22:15
mriedemsean-k-mooney: ok bug me about it tomorrow22:15
*** panda has quit IRC22:16
*** betherly has joined #openstack-nova22:18
*** abaindur has quit IRC22:18
*** panda has joined #openstack-nova22:20
mnaseris the compute_nodes table exposed anywhere?22:22
mnasers/table/data/22:23
*** betherly has quit IRC22:23
sean-k-mooneyits in the hyperviors api i think22:23
mnaserwith ironic and for some reason its complaining that there is a duplicate compute_node listed22:23
*** whoami-rajat has joined #openstack-nova22:24
mnaserRemote error: DBDuplicateEntry (pymysql.err.IntegrityError) (1062, u"Duplicate entry '77788ad5-f1a4-46ac-8132-2d88dbd4e594' for key 'compute_nodes_uuid_idx'")22:25
mnaser2019-08-08 22:21:25.284 82770 WARNING nova.compute.resource_tracker [req-a58eb5e2-9be0-4503-bf68-dff32ff87a3a - - - - -] No compute node record for ctl1-xxxx:77788ad5-f1a4-46ac-8132-2d88dbd4e594: ComputeHostNotFound_Remote: Compute host ctl1-xxxx could not be found.22:27
mnaserhmm22:27
*** ociuhandu has joined #openstack-nova22:30
mriedemis one (soft) deleted?22:32
*** ociuhandu has quit IRC22:35
mnaseryes22:41
mnaserbut i think this is tricking me22:41
mnaser2019-08-08 22:21:23.467 82770 ERROR ironicclient.common.http [req-a58eb5e2-9be0-4503-bf68-dff32ff87a3a - - - - -] Error contacting Ironic server: Unable to attach VIF because VIF 6a785d92-8459-42a8-ac06-4f2d1d222cd6 is already attached to Ironic Port 439f4090-9824-4456-85ca-6417503a89cc (HTTP 409). Attempt 2 of 2: Conflict: Unable to attach VIF because VIF 6a785d92-8459-42a8-ac06-4f2d1d222cd6 is already attached to22:41
mnaserIronic Port 439f4090-9824-4456-85ca-6417503a89cc (HTTP 409)22:41
mnaseri see that happen right before it, so i wonder if those warnings/errors make nova-compute blow up and bubble up the wrong exception22:41
mnaserthere is a soft deleted one though, i wonder if thats the issue22:43
mriedemyeah the unique index is on uuid not including the deleted column22:44
mriedemunlike uniq_compute_nodes0host0hypervisor_hostname0deleted22:44
mriedemi can't remember what release it was, but semi-recently we started using the ironic node uuid as the compute node uuid for easier correlation of those records22:44
mnaserwell deleting it fixing it but that sounds like a bug then i guess22:44
mnaseryeah that might explain it22:44
mriedembut, since the compute node uuid is not unique now when it gets created, it'll be a problem if the ironic node / compute node is soft-deleted and not archived...22:45
mnasermriedem: https://github.com/openstack/nova/commit/9f28727eb75e05e07bad51b6eecce667d09dfb6522:45
mriedemwhich...does that mean a recreate is as simple as creating a server from a specific bm node, deleting the server, and then creating another server from the same node?22:45
mriedemyes that's how i knew, i wrote it :)22:46
mriedemwent into rocky so it must not be that common of an issue22:46
mnaseri think this might have been after moving a node to maintenance22:48
mnaserand then out of it22:48
mnaseri think?22:48
mnaserdoes that result in the compute node being deleted? ironic is still new to me22:48
openstackgerritMatt Riedemann proposed openstack/nova stable/queens: Add functional recreate test for regression bug 1825537  https://review.opendev.org/67535522:48
openstackbug 1825537 in OpenStack Compute (nova) queens "finish_resize failures incorrectly revert allocations" [Medium,Confirmed] https://launchpad.net/bugs/182553722:48
mriedemdo you see this for that node? https://github.com/openstack/nova/blob/master/nova/compute/manager.py#L830622:48
mnasermriedem: 2019-08-08 17:20:13.921 6379 INFO nova.compute.manager [req-c71e5c81-eb34-4f72-a260-6aa7e802f490 - - - - -] Deleting orphan compute node 31 hypervisor host is 77788ad5-f1a4-46ac-8132-2d88dbd4e594, nodes are set([u'6d556617-2bdc-42b3-a3fe-b9218a1ebf0e', u'a634fab2-ecea-4cfa-be09-032dce6eaf51', u'2dee290d-ef73-46bc-8fc2-af248841ca12'])22:50
mriedemheh, bingo22:51
mnaseryep, and i think this is when we set it to maintenance mode i think22:51
mriedemwhich we likely don't test in ironic+nova ci22:51
mnaserhttps://github.com/openstack/nova/blob/master/nova/compute/manager.py#L829822:51
mriedemright...22:52
mriedemwell, report a bug first,22:52
mriedemwe could revert https://review.opendev.org/#/c/571535/ which would be unfortunate,22:52
mriedemor change https://github.com/openstack/nova/blob/master/nova/compute/manager.py#L8311 to be a hard delete for ironic,22:52
mriedembut i'm not sure how people would feel about that22:52
mriedemwe like to pack rat22:52
mriedemi sorta feel that ironic compute nodes are special though unlike hypervisor compute ndoes (libvirt) and are more ephemeral22:53
mriedemmeaning hard delete isn't so bad22:53
mnaseri agree22:53
mriedemyou're just going to archive those dorks anyway22:53
*** rcernin has joined #openstack-nova22:53
mriedemehh...the shitty thing is ComptueNode.destroy is remotable so adding a hard_delete kwarg to that or something would be an rpc version change, which means not backportable22:54
mriedemnot even sure about changing the behavior within ComputeNode.destroy...22:55
mnaserunless the conductor does the hard delete if it knows that the destroy is for an ironic h v22:55
mriedemright that's what i mean by within the method22:55
mriedemwould need to run it past dansmith in the morning probably22:55
mriedemfor now you can archive the db to workaround it22:55
mriedemreport a bug and i can talk with dan in the early morn22:55
mnaseror maybe you can pull *all* baremetal nodes and only delete those that are missing (and ignore disabled/maintenance oness)22:57
* mnaser almost done draft bug22:57
mriedemumm22:58
mriedemi don't know that disabled/maintenance mode is a thing we track in the compute node object22:58
mriedemunless it's in one of those json-ified fields22:58
mnaseri meant when the ironic driver pulls the list of nodes22:58
mriedemoh side-step the cn.destroy22:59
mriedemyeah maybe, not sure the implications of that though22:59
mriedemif you're host is managing 1000 nodes, that could be an expensive pull every 60 seconds :)22:59
mriedem*your22:59
mnaseryou were gonna pull 1000 anyways, the only thing where it would be an issue is if you have 1000 with 990 in maint mode23:00
mriedem999 problems but the node ain't one?23:00
mnaserhad to go with 990 to not let you get away with that one ha23:00
mnaserhttps://bugs.launchpad.net/nova/+bug/183956023:00
openstackLaunchpad bug 1839560 in OpenStack Compute (nova) "ironic: moving node to maintenance makes it unusable afterwards" [Undecided,New]23:00
mnaseroh wait23:00
mnaseri just realized the update_available_resource thing runs inside manager.py so its not ironic specific23:01
*** betherly has joined #openstack-nova23:01
mriedemyou'd still have an issue if you deleted the compute service record which will (soft) delete all of the compute nodes associated with it23:01
mriedemright23:01
mriedemthought that node destroy stuff is really for ironic23:01
mriedemin update_available_resource23:01
mriedem*though23:01
mriedemhttps://github.com/openstack/nova/blob/master/nova/db/sqlalchemy/api.py#L40123:01
mnaserah yeah but i figured thats you burning yourself if you delete a compute service record (but i can imagine you would have the assumption that it automatically appears if you deleted it again so yeah)23:02
mriedem^ is a problem when deleting the managing nova-compute service record23:02
mriedemit will when you restart the nova-compute service23:02
*** ricolin_ has joined #openstack-nova23:02
mriedemor if you leave nova-compute running when you destroy the service23:02
mriedemsee note on https://docs.openstack.org/api-ref/compute/?expanded=delete-compute-service-detail#delete-compute-service23:02
mriedemso, i guess the revert is probably easiest backportable thing23:03
mnaseryeah i cant think of anything better, anything else makes the api layer code too ironic-y23:04
*** ricolin has quit IRC23:05
*** betherly has quit IRC23:06
*** tbachman has joined #openstack-nova23:09
mriedemi'll post a revert for discussion anyway23:10
*** slaweq has joined #openstack-nova23:11
mriedemwarren zevon is going to help me get through this23:12
*** tbachman has quit IRC23:14
*** tbachman has joined #openstack-nova23:14
*** slaweq has quit IRC23:15
openstackgerritMatt Riedemann proposed openstack/nova master: Revert "Match ComputeNode.uuid to ironic node uuid in RT"  https://review.opendev.org/67549623:23
openstackgerritMatt Riedemann proposed openstack/nova master: Skip test_migrate_disk_and_power_off_crash_finish_revert_migration  https://review.opendev.org/67541723:28
openstackgerritMatt Riedemann proposed openstack/nova master: Dump versioned notifications when len assertions fail  https://review.opendev.org/67540423:28
mriedemgod i must break more things than i fix23:28
*** mlavalle has quit IRC23:31
*** tbachman has quit IRC23:40
*** betherly has joined #openstack-nova23:54
*** betherly has quit IRC23:59
« Wednesday, 2019-08-07 Index Friday, 2019-08-09 »

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!