Friday, 2019-06-28

*** mriedem_afk is now known as mriedem00:03
openstackgerritMatt Riedemann proposed openstack/nova master: Add InstanceAction/Event create() method  https://review.opendev.org/61403600:14
openstackgerritMatt Riedemann proposed openstack/nova master: Add Instance.hidden field  https://review.opendev.org/63112300:14
openstackgerritMatt Riedemann proposed openstack/nova master: Add TargetDBSetupTask  https://review.opendev.org/62789200:14
openstackgerritMatt Riedemann proposed openstack/nova master: Add CrossCellMigrationTask  https://review.opendev.org/63158100:14
openstackgerritMatt Riedemann proposed openstack/nova master: Execute TargetDBSetupTask  https://review.opendev.org/63385300:14
openstackgerritMatt Riedemann proposed openstack/nova master: Add prep_snapshot_based_resize_at_dest compute method  https://review.opendev.org/63329300:14
openstackgerritMatt Riedemann proposed openstack/nova master: Add PrepResizeAtDestTask  https://review.opendev.org/62789000:14
openstackgerritMatt Riedemann proposed openstack/nova master: Add prep_snapshot_based_resize_at_source compute method  https://review.opendev.org/63483200:14
openstackgerritMatt Riedemann proposed openstack/nova master: Add nova.compute.utils.delete_image  https://review.opendev.org/63760500:14
openstackgerritMatt Riedemann proposed openstack/nova master: Add PrepResizeAtSourceTask  https://review.opendev.org/62789100:14
openstackgerritMatt Riedemann proposed openstack/nova master: Refactor ComputeManager.remove_volume_connection  https://review.opendev.org/64218300:14
openstackgerritMatt Riedemann proposed openstack/nova master: Add power_on kwarg to ComputeDriver.spawn() method  https://review.opendev.org/64259000:14
openstackgerritMatt Riedemann proposed openstack/nova master: Add finish_snapshot_based_resize_at_dest compute method  https://review.opendev.org/63508000:14
openstackgerritMatt Riedemann proposed openstack/nova master: Add FinishResizeAtDestTask  https://review.opendev.org/63564600:14
openstackgerritMatt Riedemann proposed openstack/nova master: Add Destination.allow_cross_cell_move field  https://review.opendev.org/61403500:14
openstackgerritMatt Riedemann proposed openstack/nova master: Execute CrossCellMigrationTask from MigrationTask  https://review.opendev.org/63566800:14
openstackgerritMatt Riedemann proposed openstack/nova master: Plumb allow_cross_cell_resize into compute API resize()  https://review.opendev.org/63568400:14
openstackgerritMatt Riedemann proposed openstack/nova master: Filter duplicates from compute API get_migrations_sorted()  https://review.opendev.org/63622400:14
openstackgerritMatt Riedemann proposed openstack/nova master: Support cross-cell moves in external_instance_event  https://review.opendev.org/65847800:14
openstackgerritMatt Riedemann proposed openstack/nova master: Add cross-cell resize policy rule and enable in API  https://review.opendev.org/63826900:14
openstackgerritMatt Riedemann proposed openstack/nova master: WIP: Enable cross-cell resize in the nova-multi-cell job  https://review.opendev.org/65665600:14
*** brault has quit IRC00:27
openstackgerritMatt Riedemann proposed openstack/nova master: WIP: Add placement request filter for disabled computes  https://review.opendev.org/65459600:35
*** hamzy has joined #openstack-nova00:38
*** tbachman has quit IRC00:48
openstackgerritMatt Riedemann proposed openstack/nova master: Drop pre-cinder 3.44 version compatibility  https://review.opendev.org/62106100:51
openstackgerritMatt Riedemann proposed openstack/nova master: Follow-up for I6a777b4b7a5729488f939df8c40e49bd40aec3dd  https://review.opendev.org/66496700:51
openstackgerritMatt Riedemann proposed openstack/nova master: Remove 'MultiattachSupportNotYetAvailable' exception  https://review.opendev.org/65131500:51
openstackgerritMatt Riedemann proposed openstack/nova-specs master: Enable rebuild for instances in cell0  https://review.opendev.org/64868601:06
*** imacdonn has quit IRC01:09
*** imacdonn has joined #openstack-nova01:10
*** lbragstad has quit IRC01:11
*** bhagyashris has joined #openstack-nova01:15
*** tbachman has joined #openstack-nova01:16
bhagyashrismriedem, Sean-k-mooney, alex_xu: Hi, I would like your insights on https://review.opendev.org/#/c/612626/01:17
openstackgerritMerged openstack/nova-specs master: Enable rebuild for instances in cell0  https://review.opendev.org/64868601:26
*** adriant has joined #openstack-nova01:40
adriantHello! Out of curiosity, was there ever a bug in Nova that stopped delete-on-terminate working specifically for volumes created from an instance snapshot?01:41
adriant^ we have a case where a volume made from a volume snapshot, delete-on-terminate works. The volume just goes away with the instance01:42
adriantbut a volume made from an instance snapshot, delete-on-terminate doesn't work. And the volume must be manually delete afterwards.01:42
adriantwe are running an old version of nova, so just curious if this was a known bug, and if it ever got fixed, or if this is just some weird combination of failures somewhere on our deployment01:43
openstackgerritMerged openstack/nova stable/stein: Noop CantStartEngineError in targets_cell if API DB not configured  https://review.opendev.org/66303001:44
sean-k-mooneynot that i see in launchpad01:44
openstackgerritMerged openstack/nova stable/stein: Fix python3 compatibility of rbd get_fsid  https://review.opendev.org/66451201:44
adriantyeah, I didn't find anything either01:45
*** yaawang has quit IRC01:45
sean-k-mooneyto be clear your workflow is you boot a vm. create a snapshot then later boot another vm from the instnace shapshot with delete-on-terminate01:46
adriantyes01:46
adriantand if you instance make a volume snapshot of that instances root disk, and boot from that, it works as expected01:46
adriantyour*01:46
sean-k-mooneythat should be the same as a boot form a standard image really01:46
adriantI know :P01:47
adriantbut somehow it isn't01:47
*** gouthamr has quit IRC01:49
adriantI'm just confirming that I can replicate it with a new instance01:50
adriantbut I've seen a client do it01:50
*** gouthamr has joined #openstack-nova01:52
*** mriedem has quit IRC01:54
*** rajinir has quit IRC01:55
yonglihemriedem,  sean-k-mooney:  what i got from discuss, 1) use UUID instead of name for querying orphans instances 2) checking metadata (which is already done  per Sean's suggest),  3) clean up if possible ,  if db record gone, i don't how to do that.  for reap/poweroff/delete, we already had configuration options.   4) re-use/extending current period task instead of create new one.01:58
yonglihemriedem,  sean-k-mooney:  these changes seems quite a lot for this 2 patches. i want to make sure i'm on right track before i start it.01:59
*** gouthamr has quit IRC02:02
*** gouthamr has joined #openstack-nova02:06
*** brinzhang has joined #openstack-nova02:07
*** gouthamr has quit IRC02:07
sean-k-mooneyyonglihe: you likely do not need to have 2 seperate patches.02:09
sean-k-mooneymost of the chagnes in the first patch are not required.02:09
sean-k-mooneyyonglihe: melwitt: was suggeting we might be able to jsut create a fake instance object and reuse the existing destroy method in the driver02:10
sean-k-mooneythat occured to me als but i was not sure it that would be valid in all cases02:10
sean-k-mooneybut yes 1.) we would prefer to use uuids instead of names, 2.) for the libvirt dirver check the metadata(which you have already done) 3.) determin if we can clean up local resoces like disk or ports on a vswitch. 4.) combine you chagnes with the existing periodic task as a new confg option reap_unknown02:12
*** gouthamr has joined #openstack-nova02:14
*** gouthamr has quit IRC02:17
*** gouthamr has joined #openstack-nova02:19
*** ricolin has joined #openstack-nova02:20
*** threestrands has joined #openstack-nova02:23
yonglihesean-k-mooney:  sure, seems we cloud move on, it's gonna drop a lot and come up with new one, though.  about clean up local resources,  i might need reach for help, might trouble you -:)02:27
*** gouthamr has quit IRC02:29
sean-k-mooneythe concern is if we jsut delete the domain we would leave the root disk and vswitch ports behind. and in that case the operator would not know what instance they belonged or if it wa safe to remvoe them02:31
sean-k-mooneywe might decide that is ok but if we can sanely clean up that would be nice to do02:32
*** tbachman has quit IRC02:33
*** gouthamr has joined #openstack-nova02:35
*** tbachman has joined #openstack-nova02:36
yonglihewe might need to add power_off_unknow in case admin want to keep them for investigating.02:38
sean-k-mooneyyes i think that would be fine and in that case no cleanup is needed02:40
*** gouthamr has quit IRC02:42
yongliheI gonna add them both.02:47
*** gouthamr has joined #openstack-nova02:49
*** gouthamr has quit IRC02:54
*** tbachman has quit IRC02:55
*** gouthamr has joined #openstack-nova02:58
*** whoami-rajat has joined #openstack-nova03:05
*** gouthamr has quit IRC03:05
*** tonyb has quit IRC03:05
*** gouthamr has joined #openstack-nova03:10
*** cfriesen has quit IRC03:17
*** gouthamr has quit IRC03:27
*** gouthamr has joined #openstack-nova03:31
*** gouthamr has quit IRC03:33
*** tonyb has joined #openstack-nova03:36
*** gouthamr has joined #openstack-nova03:37
*** psachin has joined #openstack-nova03:40
openstackgerritsean mooney proposed openstack/nova master: [WIP] Libvirt: report storage bus traits  https://review.opendev.org/66691403:45
openstackgerritsean mooney proposed openstack/nova master: [WIP] use domain capablites to get supported device models  https://review.opendev.org/66691503:45
openstackgerritsean mooney proposed openstack/nova master: Add transform_image_metadata request filter  https://review.opendev.org/66577503:45
*** gouthamr has quit IRC03:49
*** tbachman has joined #openstack-nova03:54
*** gouthamr has joined #openstack-nova03:58
*** gouthamr has quit IRC04:00
*** guozijn has joined #openstack-nova04:04
*** gouthamr has joined #openstack-nova04:06
openstackgerritGhanshyam Mann proposed openstack/nova master: Fix test_flavors to run with correct microversion  https://review.opendev.org/66653604:09
openstackgerritGhanshyam Mann proposed openstack/nova master: Add missing tests for flavor extra_specs mv 2.61  https://review.opendev.org/66760004:10
openstackgerritGhanshyam Mann proposed openstack/nova master: Multiple API cleanup changes  https://review.opendev.org/66688904:10
*** gouthamr has quit IRC04:10
*** gouthamr has joined #openstack-nova04:16
*** gouthamr has quit IRC04:17
*** gouthamr has joined #openstack-nova04:22
*** mkrai_ has joined #openstack-nova04:23
*** udesale has joined #openstack-nova04:24
*** _alastor_ has quit IRC04:25
*** gouthamr has quit IRC04:25
*** brault has joined #openstack-nova04:26
*** gouthamr has joined #openstack-nova04:30
*** mkrai_ has quit IRC04:33
*** gouthamr has quit IRC04:35
*** pcaruana has joined #openstack-nova04:36
*** mkrai_ has joined #openstack-nova04:39
*** guozijn has quit IRC04:41
*** gouthamr has joined #openstack-nova04:41
*** gouthamr has quit IRC04:50
*** gouthamr has joined #openstack-nova04:57
*** gouthamr has quit IRC05:06
*** gouthamr has joined #openstack-nova05:12
*** gouthamr has quit IRC05:19
*** gouthamr has joined #openstack-nova05:24
*** Luzi has joined #openstack-nova05:26
*** gouthamr has quit IRC05:31
*** gouthamr has joined #openstack-nova05:37
*** guozijn has joined #openstack-nova05:38
openstackgerritMerged openstack/nova master: Fix test_flavors to run with correct microversion  https://review.opendev.org/66653605:39
*** bbowen__ has quit IRC05:42
*** ivve has quit IRC05:43
*** gouthamr has quit IRC05:44
*** luksky has joined #openstack-nova05:45
*** ratailor has joined #openstack-nova05:51
*** gouthamr has joined #openstack-nova05:51
*** gouthamr has quit IRC05:52
*** gouthamr has joined #openstack-nova05:58
*** slaweq has joined #openstack-nova06:00
*** guozijn has quit IRC06:01
*** guozijn has joined #openstack-nova06:02
*** gouthamr has quit IRC06:03
*** gouthamr has joined #openstack-nova06:08
*** lpetrut has joined #openstack-nova06:09
*** lpetrut has quit IRC06:09
*** lpetrut has joined #openstack-nova06:10
*** gouthamr has quit IRC06:11
*** gouthamr has joined #openstack-nova06:16
*** gouthamr has quit IRC06:16
*** dpawlik has joined #openstack-nova06:18
*** gouthamr has joined #openstack-nova06:22
*** ivve has joined #openstack-nova06:22
*** gouthamr has quit IRC06:29
*** threestrands has quit IRC06:34
*** gouthamr has joined #openstack-nova06:36
*** maciejjozefczyk has joined #openstack-nova06:41
*** gouthamr has quit IRC06:46
gmannefried: can you re-approve this. it was rebased - https://review.opendev.org/#/c/667600/406:51
*** mkrai has quit IRC06:52
*** ccamacho has joined #openstack-nova06:53
*** artom has joined #openstack-nova06:53
*** artom is now known as artom|gmtplus306:54
*** gouthamr has joined #openstack-nova06:58
openstackgerritGhanshyam Mann proposed openstack/nova master: Multiple API cleanup changes  https://review.opendev.org/66688907:03
*** gouthamr has quit IRC07:04
*** damien_r has joined #openstack-nova07:08
*** guozijn has quit IRC07:09
*** gouthamr has joined #openstack-nova07:12
*** guozijn has joined #openstack-nova07:13
*** gouthamr has quit IRC07:15
*** rcernin has quit IRC07:16
*** rdopiera has joined #openstack-nova07:17
*** tesseract has joined #openstack-nova07:17
*** guozijn has quit IRC07:18
*** gouthamr has joined #openstack-nova07:23
*** ttsiouts has joined #openstack-nova07:26
*** tesseract has quit IRC07:30
*** gouthamr has quit IRC07:30
*** tesseract has joined #openstack-nova07:32
*** itssurya has joined #openstack-nova07:34
*** gouthamr has joined #openstack-nova07:39
*** brault has quit IRC07:49
*** guozijn has joined #openstack-nova07:49
kashyaplyarwood: Morning; stable/stein has merged, if you want to ACK its stable/rocky counterpart: https://review.opendev.org/#/c/667436/107:49
*** gouthamr has quit IRC07:51
* kashyap goes to fiddle with Queens07:54
*** lpetrut has quit IRC07:56
*** gouthamr has joined #openstack-nova07:57
*** gouthamr has quit IRC07:59
*** brault has joined #openstack-nova07:59
lyarwoodkashyap: the same rules apply to stable as master fwiw so I can't just +2+W this, I'll recheck and leave it to others to +W.08:00
*** ralonsoh has joined #openstack-nova08:01
kashyaplyarwood: Okay, thought I was under the illusion that johnthetubaguy had also ACKed the stable/rocky one08:01
lyarwoodkashyap: I think that was a different one no?08:03
kashyaplyarwood: Which one was it?  Strangely I recall there were 2 ACKs from John on this same change08:03
kashyapMaybe he ACKed the Git/master one and I confused with it; /me goes to look08:03
kashyap(No, never mind; I'll chalk it up to poor memory)08:04
*** brault has quit IRC08:04
*** gouthamr has joined #openstack-nova08:04
*** yonglihe has quit IRC08:05
kashyapI find it dispiriting that even for multiple backports there's more burden of getting *two* "ACKs" from separate companies.08:06
openstackgerritKashyap Chamarthy proposed openstack/nova stable/queens: libvirt: Rework 'EBUSY' (SIGKILL) error handling code path  https://review.opendev.org/66811108:07
*** psachin has quit IRC08:08
*** gouthamr has quit IRC08:09
*** gouthamr has joined #openstack-nova08:14
*** xek has joined #openstack-nova08:14
*** gouthamr has quit IRC08:17
*** gouthamr has joined #openstack-nova08:23
*** psachin has joined #openstack-nova08:25
*** gouthamr has quit IRC08:30
*** whoami-rajat has quit IRC08:34
*** gouthamr has joined #openstack-nova08:40
*** guozijn has quit IRC08:42
*** gouthamr has quit IRC08:47
*** gouthamr has joined #openstack-nova08:53
*** gouthamr has quit IRC08:56
*** jaosorior has quit IRC08:57
*** jaosorior has joined #openstack-nova09:00
*** gouthamr has joined #openstack-nova09:02
*** davidsha has joined #openstack-nova09:04
*** gouthamr has quit IRC09:05
openstackgerritLee Yarwood proposed openstack/nova stable/stein: libvirt: flatten rbd images when unshelving an instance  https://review.opendev.org/66811809:07
openstackgerritLee Yarwood proposed openstack/nova stable/rocky: libvirt: flatten rbd images when unshelving an instance  https://review.opendev.org/66811909:08
*** gouthamr has joined #openstack-nova09:14
*** psachin has quit IRC09:16
*** gouthamr has quit IRC09:19
*** ricolin has quit IRC09:21
openstackgerritLee Yarwood proposed openstack/nova stable/queens: libvirt: flatten rbd images when unshelving an instance  https://review.opendev.org/66812309:22
*** gouthamr has joined #openstack-nova09:23
*** gouthamr has quit IRC09:25
*** trident has quit IRC09:27
*** trident has joined #openstack-nova09:29
openstackgerritLee Yarwood proposed openstack/nova stable/stein: DNM - Test rbd unshelve fix on stable stein  https://review.opendev.org/66812609:30
*** gouthamr has joined #openstack-nova09:32
*** mkrai_ has quit IRC09:36
*** mkrai__ has joined #openstack-nova09:36
*** gouthamr has quit IRC09:37
*** gouthamr has joined #openstack-nova09:42
*** gouthamr has quit IRC09:43
openstackgerritzhufl proposed openstack/nova master: Fix invalid assertIsNone states  https://review.opendev.org/66813209:43
*** mkrai__ has quit IRC09:44
*** jaosorior has quit IRC09:49
*** itssurya has quit IRC09:54
*** gouthamr has joined #openstack-nova09:54
*** ivve has quit IRC09:56
*** xek has quit IRC09:59
*** gouthamr has quit IRC10:00
*** gouthamr has joined #openstack-nova10:06
*** ociuhandu has joined #openstack-nova10:07
*** ociuhandu has quit IRC10:14
*** ociuhandu has joined #openstack-nova10:17
*** gouthamr has quit IRC10:18
*** gouthamr has joined #openstack-nova10:26
*** ccamacho has quit IRC10:30
*** ccamacho has joined #openstack-nova10:31
*** gouthamr has quit IRC10:35
openstackgerritBrin Zhang proposed openstack/nova-specs master: Add flavor group  https://review.opendev.org/66356310:36
openstackgerritMerged openstack/python-novaclient stable/rocky: Use SHA256 instead of MD5 in completion cache  https://review.opendev.org/66402510:38
*** brinzhang has quit IRC10:38
*** shilpasd has joined #openstack-nova10:39
*** ivve has joined #openstack-nova10:40
*** gouthamr has joined #openstack-nova10:41
*** bbowen has joined #openstack-nova10:43
*** gouthamr has quit IRC10:43
openstackgerritDakshina Ilangovan proposed openstack/nova-specs master: Spec: Provider config YAML file  https://review.opendev.org/61249710:45
*** bhagyashris has quit IRC10:51
*** whoami-rajat has joined #openstack-nova10:51
*** gouthamr has joined #openstack-nova10:52
*** aloga has quit IRC10:53
*** mrch_ has joined #openstack-nova10:54
*** ttsiouts has quit IRC10:56
*** ttsiouts has joined #openstack-nova10:57
*** gouthamr has quit IRC10:57
openstackgerritLee Yarwood proposed openstack/nova stable/rocky: DNM - Test rbd unshelve fix on stable rocky  https://review.opendev.org/66814111:03
*** gouthamr has joined #openstack-nova11:06
openstackgerritLee Yarwood proposed openstack/nova stable/queens: DNM - Test rbd unshelve fix on stable queens  https://review.opendev.org/66814211:06
*** ccamacho has quit IRC11:11
openstackgerritLee Yarwood proposed openstack/nova stable/queens: libvirt: Do not reraise DiskNotFound exceptions during resize  https://review.opendev.org/66036311:21
openstackgerritLee Yarwood proposed openstack/nova stable/queens: libvirt: flatten rbd images when unshelving an instance  https://review.opendev.org/66812311:21
openstackgerritLee Yarwood proposed openstack/nova stable/queens: DNM - Test rbd unshelve fix on stable queens  https://review.opendev.org/66814211:21
*** tbachman has quit IRC11:29
*** yikun has quit IRC11:33
*** ratailor has quit IRC11:35
*** tbachman has joined #openstack-nova11:47
*** dpawlik has quit IRC12:15
efriedgmann: done12:17
*** dpawlik has joined #openstack-nova12:20
*** udesale has quit IRC12:22
*** udesale has joined #openstack-nova12:23
*** mvkr has quit IRC12:29
*** lbragstad has joined #openstack-nova12:31
*** pcaruana has quit IRC12:40
*** pcaruana has joined #openstack-nova12:41
*** eharney has joined #openstack-nova12:53
openstackgerritMerged openstack/nova master: Fix invalid assertIsNone states  https://review.opendev.org/66813213:04
*** ivve has quit IRC13:04
*** lbragstad has quit IRC13:08
*** lbragstad has joined #openstack-nova13:08
*** jdillaman has quit IRC13:13
*** tbachman has quit IRC13:15
*** KeithMnemonic has joined #openstack-nova13:21
*** lpetrut has joined #openstack-nova13:23
*** mriedem has joined #openstack-nova13:24
kashyapjohnthetubaguy: Can you please put this stable/rocky backport through?  https://review.opendev.org/#/c/667436/13:26
kashyapjohnthetubaguy: You've ACKed the stable/stein one13:26
kashyap(And this, stable/rocky version, already has one +2)13:27
kashyapOr anyone else, if John is not around13:27
*** jdillaman has joined #openstack-nova13:28
*** davidsha has quit IRC13:30
KeithMnemonicis there anyone here that is maintaining or familiar with https://github.com/openstack/nova/blob/master/nova/virt/vmwareapi/ds_util.py13:32
kashyapKeithMnemonic: Your best bet is GKotton (who is not here on IRC)13:33
efriedmriedem, dansmith: placement agg sync is automatic now, right? No need to run nova-manage placement sync-aggregates? https://review.opendev.org/#/c/667952/1/doc/source/reference/forbidden-aggregates.rst@4413:34
kashyapKeithMnemonic: You might want to try his e-mail (gkotton@vmware.com).13:34
dansmithefried: yeah, the manage sync is for fixup and upgrades at this point I think13:34
kashyap[And of course, Cc the list, so others could learn, too.]13:34
efriedthanks dansmith13:34
KeithMnemonicyes i was looking for him, we know each other so hopefully i can find him next week13:35
KeithMnemonicthanks kashyap13:35
kashyapKeithMnemonic: By "the list" I meant: openstack-discuss@lists.openstack.org13:36
mriedemefried: right what dan said,13:36
mriedemexcept if the api fails to remove a provider from an aggregate, sync_aggregates won't fix that13:36
mriedemsync_aggregates is only additive13:36
KeithMnemonicok, i thought you meant wait for him to show up back here. i will send an email today13:36
efriedalso, mriedem, /me Remind / ping / harass re https://review.opendev.org/662881 (sdk spec)13:38
mriedemd'oh!13:39
mriedemcan i hit snooze on that until after i take my kid to camp?13:39
efriedof course. I can hit you up hourly13:39
dansmithcron13:40
dansmithit's they only way he'll learn.13:40
*** BjoernT has joined #openstack-nova13:42
openstackgerritMerged openstack/nova master: Add missing tests for flavor extra_specs mv 2.61  https://review.opendev.org/66760013:46
*** mlavalle has joined #openstack-nova13:54
*** trident has quit IRC13:56
*** ratailor has joined #openstack-nova13:56
*** shilpasd has quit IRC13:57
*** trident has joined #openstack-nova13:58
*** Luzi has quit IRC14:02
*** davidsha has joined #openstack-nova14:08
efriedugh, do we not have `openstack resource provider trait add` ?14:08
*** liuyulong has joined #openstack-nova14:10
*** dpawlik has quit IRC14:10
*** lpetrut has quit IRC14:11
mriedemefried: https://docs.openstack.org/osc-placement/latest/cli/index.html#trait-create14:24
mriedemosc verbs are create/set/unset/delete/list14:24
mriedem and /show14:24
efriedRight, I mean "add this trait to resource provider X without fing with any of its existing traits"14:25
efriedso I don't have to do trait list + add to that + trait set14:25
mriedemoh, yeah we have a few gaps in ux like that in osc-placement14:25
mriedemit's annoying, especially for things like adding inventory with a new resource class to a provider or allocations14:25
efriedmeanwhile, how tf do I get a compute node UUID?14:26
mriedemopenstack --os-compute-api-version 2.53 hypervisor list14:26
efriedby name?14:26
efriedgot it14:26
efriedphew14:26
mriedem--matching <hostname>14:26
mriedemhttps://docs.openstack.org/python-openstackclient/latest/cli/command-objects/hypervisor.html#hypervisor-list14:26
* mriedem starts to charge a fee14:27
efriedopenstack --os-compute-api-version 2.53 hypervisor show my-compute-name -f value -c id14:27
efried?14:27
mriedemto get the id, probably -c ID14:28
mriedemnode_uuid=$(openstack --os-compute-api-version 2.53 hypervisor show <hostname> -f value -c ID)14:28
efried`ID` wasn't working for me, but `id` does.14:29
mriedemi'm not totally sure osc is working with 2.53 everywhere yet14:29
mriedemah ok14:29
mriedemi've noticed some inconsistencies with ID vs id in osc14:29
efriedyeah, totally14:29
mriedemfor image nad server it's ID i think14:29
openstackgerritBalazs Gibizer proposed openstack/nova master: WIP: Add rollback to heal port allocation  https://review.opendev.org/66818414:29
mriedemopen a story bug14:29
gibimriedem, efried: I hacked up the rollback code for heal port allocation. https://review.opendev.org/668184 Based on the code I feel I'm just pushing the human-interaction-needed problem one level deeper, when the rollback fails14:30
mriedemefried: there is a story for the trait append thing https://storyboard.openstack.org/#!/story/200525814:30
mriedemi knew it sounded familiar....14:30
efriedthanks14:30
*** _alastor_ has joined #openstack-nova14:32
mriedemgibi: i left some comments,14:33
*** mrch_ has quit IRC14:34
mriedembut i haven't fully thought through which is worse - the port with the binding:profile.allocation set to something when the allocation doesn't exist in neutron vs the allocation existing in neutron but the port binding profile not mapped to that provider14:34
mriedem*doesn't exist in placement14:34
gibimriedem: if the rollback retry fails the is it OK to ask for the human to help?14:35
gibiI feel at the end we need the human anyhow14:35
*** lbragstad has quit IRC14:36
*** artom|gmtplus3 has quit IRC14:37
gibiif we set the allocation key in neutron without having the allocation placemen then we tell neutron to use a resource that is not really allocated. But the physical bandwidth anyhow was used even before we started to heal14:37
mriedemso the risk there is over-committing the resource right?14:38
mriedemb/c placement isn't tracking the allocation14:38
gibiyes, but the overcommit situation can already exists (hence the need of healing)14:38
mriedemthen isn't that better than potentially having the allocations in placement w/o the neutron port binding profile tracking the allocation and if the admin screws up the manual steps, doubling the allocation by re-running the command? iow, it's no different than the situation they could already be in14:40
mriedemif you tried to run the command again we wouldn't heal that instance / port combo b/c the port would already say it's allocated when really it might not be14:40
mriedemi agree there is some amount of "we failed our main objective, and we failed to rollback, you need to step in now" if we get there14:40
mriedembut i would rather we at least *try* to rollback if possible14:41
sean-k-mooneyi have not been following two closely but how do you determin currently a port needs healing?14:41
mriedemand it sounds like rolling back the allocation changes is harder since we merged the resources14:41
mriedemsean-k-mooney: it's a port with a resource_request and doesn't have an allocation set in the binding profile14:41
sean-k-mooneythat could be a problem14:42
mriedemthat makes me think,14:42
mriedemwe should also be making sure the port is actually bound to a host right?14:42
gibimriedem: rolling back the allocations can be done by saving what was the original allocation to restore14:42
sean-k-mooneywhat about cases where we set the qos policy on a network14:42
mriedemgibi: ...yeah but that could also get messy right b/c we could lose a race and our generation is off14:43
sean-k-mooneywe only create the allocation if you pass in the port right14:43
mriedemthen what do we do?14:43
mriedemrollling back the port binding profile allocation field seems easier to me14:43
mriedemsean-k-mooney: yes14:43
gibimriedem: correct, if something else updates the allocation in between then we are rolling back to a wrong allocation14:43
mriedemwe do'nt support creating ports on networks with a qos policy14:43
gibimriedem: rolling back the neutron updated seems easy to me too14:44
sean-k-mooneyat all?14:44
gibieasyier14:44
sean-k-mooneyor we create the ports but dont request the allcotion14:44
*** cfriesen has joined #openstack-nova14:44
sean-k-mooneybecause we created the port in the compute node14:44
mriedemsean-k-mooney: this is the code that determines if we need to heal allocations for the port https://review.opendev.org/#/c/637955/28/nova/cmd/manage.py@178314:45
mriedemsean-k-mooney: we fail14:45
mriedemsean-k-mooney: https://github.com/openstack/nova/blob/master/nova/network/neutronv2/api.py#L46814:45
*** lpetrut has joined #openstack-nova14:46
mriedemgibi: so i think we agree that rolling back the port binding update is simpler than the allocation14:46
gibimriedem: good point about port bound to a host. But can it be a port with device_id=instance_uuid that is not bound?14:46
mriedemand i'd prefer we include a rollback14:46
* gibi is slow14:46
mriedemgibi: "But can it be a port with device_id=instance_uuid that is not bound?" that i'm not sure about14:47
mriedemsean-k-mooney: ^14:47
sean-k-mooney... ok was an api breakage on upgrade but i understand why it was done14:47
mriedemsean-k-mooney: oh i think we can,14:47
mriedembecause of shelve offload14:47
*** ratailor has quit IRC14:47
mriedema shelved instance still has its ports and volumes14:47
mriedembut those ports and volumes aren't "bound" to a host14:47
sean-k-mooneyyes shelve offloaded would still have the device id set14:47
gibiack14:47
gibithen I have to check for boundness as well14:47
mriedemdoes heal_allocations filter out instances that aren't on a host....14:48
mriedemit should implicitly,14:48
mriedembecause it's using instance.node14:48
*** xek has joined #openstack-nova14:48
*** panda has quit IRC14:48
sean-k-mooneygibi: you can filter py ports with vif_type!=vif-unbound14:48
gibiyeah it checks instance.node then14:48
gibiso we don't need an extra  vif_type!=vif-unbound check14:48
*** panda has joined #openstack-nova14:48
sean-k-mooneyok i guess that makes sense14:49
gibiif we don't know the where the instance runs then we don't know which RP tree need to be targeted with the healing14:49
gibimriedem: I don't get your comment at https://review.opendev.org/#/c/668184/1/nova/cmd/manage.py@189114:50
sean-k-mooneyyep. so we whould never need to heal offloaded instances14:50
mriedemnow i'm not so sure where we unbind the port on shelve offload,14:50
gibimriedem: is it about trying the rollback for each port even if one of it fails?14:50
mriedembecause https://github.com/openstack/nova/blob/324da0532f3b59aa16233a93a260d289e55860fb/nova/compute/manager.py#L5168 is a noop for neutron14:50
mriedemgibi: yes14:51
mriedemright now you'd stop on the first port update that fails14:51
mriedemi'm saying we should try to clean all and fail at the end14:51
gibimriedem: OK got it, that could be done14:51
mriedemhttps://github.com/openstack/nova/blob/324da0532f3b59aa16233a93a260d289e55860fb/nova/network/neutronv2/api.py#L318314:52
mriedemso are ports unbound when the instance is shelved offloaded....14:52
sean-k-mooneymaybe in driver.destroy14:53
sean-k-mooneyill go check14:53
*** ttsiouts has quit IRC14:53
sean-k-mooneybut they might not be14:53
sean-k-mooneydestroy should at least unplug them14:53
*** ttsiouts has joined #openstack-nova14:54
mriedemright14:54
sean-k-mooneyacttuly it cant unbind14:54
mriedemdriver.destroy just does the unplug, the virt drivers never mess with port bindings14:54
sean-k-mooneybecause we use destoy in reboot14:54
*** ttsiouts has quit IRC14:54
sean-k-mooneyand we dont ubind there14:54
mriedemanyway for gibi's heal_allocation changes it doesn't really matter b/c we wouldn't get that far anyway for a shelved offloaded instance b/c we won't try to heal allocations on an instance that doesn't have a host est14:54
mriedem*set14:55
gibiack14:55
sean-k-mooneyright its just something to look into to see if we have a latent bug14:56
sean-k-mooneyi would expect it to happen as part of moving it ot offloaded14:56
*** Conqueror has left #openstack-nova14:57
*** mkrai_ has joined #openstack-nova14:57
sean-k-mooneybut i dont think it would break anything either14:57
gibiso in summary. I will add the rollback of neutron updates. But I only add a todo to possible rollback-retry14:57
gibimriedem: is that works for you ?14:57
gibimriedem: or you are a bit more on the paranoid side :)14:57
mriedemthe backoff retry could be a follow up14:58
gibimriedem: OK14:58
mriedemtry to stagger these changes as much as possible14:59
mriedemsince this is a large change14:59
sean-k-mooneyoh other edgecase. do we currently allow attaching ports with resouce requests14:59
mriedemsean-k-mooney: no14:59
sean-k-mooneycool14:59
gibi:)14:59
*** jangutter has quit IRC14:59
gibimriedem: I can split out the helper functions from the current commit if that helps14:59
mriedemgibi: sure15:00
mriedemsean-k-mooney: this is where that would fail https://github.com/openstack/nova/blob/324da0532f3b59aa16233a93a260d289e55860fb/nova/compute/api.py#L440815:00
sean-k-mooneycool. AttachInterfaceWithQoSPolicyNotSupported is slightly missleading but it makes sense15:01
sean-k-mooneywe allow attaching ports with QoS polices jut not with min bandwith qos policies15:01
sean-k-mooneye.g. DSCP is fine as is max bandwith15:01
sean-k-mooneyits really we dont allow attaching with resource requests.15:02
gibisean-k-mooney: file a bug, I'm happy to amend the message of AttachInterfaceWithQoSPolicyNotSupported exception with a more specific message15:02
mriedemefried: the reason i didn't jump on https://storyboard.openstack.org/#!/story/2005258 was because of the osc semantic patterns to follow for other osc commands as mentioned in the comments - i gets a bit hairy15:04
mriedemrather than just like a simple --append or --remove15:04
sean-k-mooneygibi: sure i could but we will also be changing that at some point too right15:04
sean-k-mooneye.g. we will eventrually allow it if the host can fulfil the request but like not until U+15:04
gibisean-k-mooney: based on my current speed of progress with the server move operation patches it will be far in the future15:04
gibisean-k-mooney: but yes, in theory I would like to get the support for attach interface if the request fitts on the current host15:05
*** lpetrut has quit IRC15:05
sean-k-mooneywell i know that  neutorn want to start modeing ip pools in placment and use resouce resquest in the future15:05
sean-k-mooneyso i expect it to become more common15:06
gibiyeah15:06
mriedemare you talking about for routed networks?15:06
mriedemmodeling routed networks15:06
sean-k-mooneynot just routed netwrok but that is one usecase15:06
mriedemi.e. https://review.opendev.org/#/c/656885/15:06
mriedemthat's been kicked around for many years and is going to be complicated i imagine15:07
mriedemwe don't even have integration testing for the routed networks support that is documented in the neutron docs15:07
* gibi feels sometimes that bandwidth is already too complicated 15:07
mriedemhttps://docs.openstack.org/neutron/latest/admin/config-routed-networks.html15:08
sean-k-mooneyi think they also wanted to supprot using sharing aggrates to model things like floating ips and what host can recive an ip form which subnet pool15:08
mriedemwell the good news is pop up teams officially exist now so all of this work can get done!15:08
sean-k-mooneyya without a cross project effort this is hard to make progress on15:09
mriedemi was being a sarcastic jerk but yeah15:09
sean-k-mooneyhehe well i have tried to do enough stuff that crosses the nova neutron boundry that i know without getting buyin form both core teams up front and finding people to care about it it will fail15:11
*** whoami-rajat has quit IRC15:11
sean-k-mooneyanyway thats a tangent15:11
mriedemlyarwood: i guess you got busy https://review.opendev.org/#/q/topic:bug/1653953+(status:open+OR+status:merged)15:11
*** rajinir has joined #openstack-nova15:12
lyarwoodmriedem: busy spamming yes, hopefully that's what you had in mind.15:13
mriedemit is15:13
lyarwoodwonderful, there are also DNM changes actually testing things on each branch btw15:14
lyarwoodI'll add them to the same topic now to make it clear15:14
mriedemoh i see15:14
mriedemnova DNM changes that depend on the related devstack-plugin-ceph change per branch that depeds on the backport per nova branch....15:15
lyarwoodand around and around we go15:15
lyarwoodbut that should allow the fix to land then the change to enable the test without everything blowing up like it did with my cinder migrate series.15:16
mriedemthis is going to be a pain for cross-cell resize, i'm not sure how i'm going to handle that15:16
*** mkrai_ has quit IRC15:16
*** igordc has joined #openstack-nova15:17
lyarwoodI really need to go over the spec and code for that, are you using shelve/unshelve there?15:17
mriedemno,15:17
mriedembut for non-volume-backed servers, like shelve i'm creating a snapshot to get the disk from the source cell to the target cell15:18
mriedemand then spawning from that snapshot in the target cell and then delete the temporary snapshot, like unshelve15:18
mriedemso the fix here about checking the vm_state wont' work for cross-cell resize15:18
*** whoami-rajat has joined #openstack-nova15:19
lyarwoodyeah true15:19
mriedemwe could check something like the task_state used during that part of the cross-cell resize spawn, but that's going to be used during a normal resize as well, and i'm not sure if flatten should be used there15:19
mriedemwe could temporarily hack something into the image meta or the instance system_metadata to be read by the driver, but again that's super hacky15:19
lyarwoodYou could make it optional for resize and spawn tbh15:19
mriedemoptional how?15:20
*** jistr is now known as jistr|off15:20
lyarwoodwith a configurable, but it would slightly slow down spawns and resizes etc in any env that used it15:20
lyarwoodso maybe not15:20
mriedemi think i know a bit better solution,15:20
mriedemthe instance.migration_context should be set during a cross-cell resize, and from that we can get the Migration record which will have a cross_cell_move boolean on it so i'd just check that15:21
mriedemfrom the driver15:21
mriedemit's not fun but it's better than driver interface changes or hacking metadata temporarily or checking vm/task state15:21
lyarwoodack that works15:22
lyarwoodwe don't have a migration record for shelve/unshelve right?15:22
lyarwoodIOW I haven't missed an easier way of fixing my issue?15:22
lyarwoodignore me, of course we don't.15:23
mriedemnope15:23
dansmiththat's interesting,15:24
dansmithI've never thought about that15:24
dansmitha migration record would let us provide some unshelve (whilst offloaded) host affinity15:25
dansmithnot sure how important that is really, but...15:25
efrieddansmith, mriedem: How do y'all feel about reorganizing how request filters are enabled via conf?15:25
efriedAs it stands, it looks like we're headed for random options scattered around various groups.15:25
efriedWondering if it would make sense e.g. to collect them into [request_filter] $name = {enabled|disabled}15:25
efriedor [scheduler] request_filters_enabled = $name[,...]15:25
efriedor ...15:25
efriedMy hesitation is that that's really more of a dev-oriented structure; I don't know that the filters are really similar enough in spirit that that would make sense to the operator.15:25
efriedThoughts?15:25
mriedemdansmith: what do you mean? unshelve back to the original host?15:26
dansmithefried: I think we should aim for enabling behaviors not filters15:26
*** xek has quit IRC15:26
dansmithefried: and we should aim for most of those to be always on by default15:26
mriedemi'd like to avoid a list like enabled_filters15:27
dansmithyeah15:27
dansmithpeople enable or disable filters based on name, without really knowing that their behavior can be composite of multiple things15:28
efriedokay, that's kind of where my head was going, thanks for the validation.15:28
dansmithmriedem: yes, for unshelve affinity back to the original host, after offload15:30
mriedemdansmith: we save off the original host in system_metadata15:30
dansmithoh, do we?15:30
mriedemhttps://github.com/openstack/nova/blob/master/nova/compute/manager.py#L511215:30
dansmithokay15:31
dansmitha migration record would make that a little less special-to-shelve-y and more a record of the move15:31
dansmithbut anyway, just hadn't thought about it before15:31
mriedemyeah i'm not sure how much it matters either,15:32
mriedemthe original host could be gone by the time you unshelve15:32
dansmithsure, it'd just be a weight thing15:32
mriedemright, was just going to say that,15:32
mriedempass a weight hint to the scheduler15:32
mriedemif you can, do else no biggy15:32
dansmithright15:33
dansmithin "edge" cases,15:33
dansmithwhere you have two boxes at a site that are largely for hosting specific types of things,15:33
dansmithyou'd have better image affinity15:33
*** _alastor_ has quit IRC15:33
dansmithif you're enforcing it then it doesn't matter15:34
dansmithanyway,15:34
dansmithjust a thought that shelve could be more of a move for harmony with the other ops15:34
*** jistr|off is now known as jistr15:42
openstackgerritEric Fried proposed openstack/nova master: Support filtering of hosts by forbidden aggregates  https://review.opendev.org/66795215:44
efrieddonnyd: Yo Mr. Docs, would you mind giving this a scan and seeing if it makes sense to you with your operator hat on?  https://review.opendev.org/#/c/667952/2/doc/source/reference/forbidden-aggregates.rst15:46
efriedmriedem: btw, that's what all those handy dandy CLIs were for ^15:47
mriedemjesus that is a large patch,15:50
mriedemcan that be split up?15:50
donnydIt reads fine to me. I would just use the same terminology everywhere. we call the function host-aggregate, so when referring to aggregate i would sub for host-aggregate15:50
donnydor whatever the proper term is15:50
mriedemmaybe it's mostly testing and docs...15:52
efriedmriedem: It could maybe be split up, but it would be kind of contrived. It's pretty easy to read imo15:52
efriedyes.15:52
donnydYea, i had no issue at all understanding what is trying to be conveyed15:53
efrieddonnyd: Thanks for the look. I actually don't know what the accepted term is, I probbaly flubbed several of those.15:53
efrieddonnyd: (mriedem was talking about the patch as a whole, not specifically the doc)15:54
efriedI'll just go on record as saying I think this is a pretty darn cool feature.15:54
* mriedem makes a note for when all of the regressions and weird side issues show up15:55
efriedConsidering how weird and complex the implementation is, the external-facing usage model is pretty slick.15:55
efriedthe most effed up part is how you have to collect the existing traits before you can add the one you want to the host :P15:56
efried...which of course has nothing to do with the forbidden aggregates feature.15:56
dansmithis this already merged?15:57
mriedemok ok you win, we can add the PUT /resource_providers/{uuid}/traits/{trait_name} you've always wanted15:57
dansmithI'm confused about how you get the exclusion just because one flavor is tagged15:58
mriedemjust like this fine mess https://developer.openstack.org/api-ref/compute/#server-metadata-servers-metadata15:58
dansmithoh jeez,15:58
efrieddansmith: not merged15:58
dansmiththis is implementation and docs too15:58
dansmiththought this was just a doc patch15:58
efriedmriedem: I'm not pulling for an *API* to add a trait, though that wouldn't be terrible. I don't mind if osc-placement does a get-add-replace. What I mind is that the CLI user has to do that. That sucks.16:00
efriedbut we've already talked about this, hence the story16:00
*** damien_r has quit IRC16:01
mriedemefried: i was joking about the api16:05
mriedemthe cli sugar should happen, it just needs a warm body,16:06
efriedyuh16:06
mriedemspeaking of which, i remember someone telling me recently they wanted to work on some code...16:06
efriedI think I have zero client patches to my name16:06
efriedactually, I might have an ironicclient patch or two :P16:06
mriedemconsider this an opportunity for growth16:07
*** davidsha has quit IRC16:07
mriedemefried: dustinc: lots of comments and such in https://review.opendev.org/#/c/662881/ but +2; i'm going to post to the ML that i think other cores should be on board with this or nack it before EOD next tuesday for the spec review sprint16:11
efriedI count16:12
efried3 python-ironicclient16:12
efried1 python-glanceclient16:12
efried1 python-cinderclient16:12
efriedAnd pretty sure none of those were actually CLI.16:12
efriedthanks mriedem16:12
*** tbachman has joined #openstack-nova16:14
*** efried is now known as fried_rolls16:15
*** ociuhandu has quit IRC16:16
*** mriedem is now known as mriedem_away16:18
*** wwriverrat has joined #openstack-nova16:21
*** udesale has quit IRC16:26
*** rdopiera has quit IRC16:30
sean-k-mooneyi havent reviewed https://review.opendev.org/#/c/662881/8/specs/train/approved/openstacksdk-in-nova.rst16:36
sean-k-mooneybut im in favor of using the sdk in general16:36
sean-k-mooneyill try to read it while i wait for dinner but ill proably be +1 on it16:37
sean-k-mooneythanks for mailing the ist as it was not on my radar16:37
*** BjoernT has quit IRC16:46
*** kmalloc is now known as needscoffee16:47
*** ralonsoh has quit IRC17:18
*** ricolin has joined #openstack-nova17:19
*** mriedem_away is now known as mriedem17:27
*** bbowen has quit IRC17:27
*** ivve has joined #openstack-nova17:38
*** ricolin has quit IRC17:45
*** damien_r has joined #openstack-nova17:48
dustincthanks for reviewing everyone17:49
*** mrch_ has joined #openstack-nova17:50
*** whoami-rajat has quit IRC17:51
*** damien_r has quit IRC17:52
*** mvkr has joined #openstack-nova17:52
*** tesseract has quit IRC18:13
openstackgerritMerged openstack/nova stable/stein: Fix type error on call to mount device  https://review.opendev.org/66792119:00
*** Sundar has joined #openstack-nova19:01
openstackgerritGhanshyam Mann proposed openstack/nova master: Remove needs:* todo from deprecated APIs api-ref  https://review.opendev.org/66823419:07
openstackgerritMatt Riedemann proposed openstack/nova master: Add StopWatch profiling to SchedulerManager.select_destinations  https://review.opendev.org/66823519:08
openstackgerritGhanshyam Mann proposed openstack/nova master: Remove needs:* todo from deprecated APIs api-ref  https://review.opendev.org/66823419:09
*** bbowen has joined #openstack-nova19:14
*** whoami-rajat has joined #openstack-nova19:14
openstackgerritMatt Riedemann proposed openstack/nova master: Add StopWatch profiling to SchedulerManager.select_destinations  https://review.opendev.org/66823519:15
openstackgerritMerged openstack/nova stable/stein: Change InstanceFaultRollback handling in _error_out_instance_on_exception  https://review.opendev.org/66663819:16
openstackgerritMerged openstack/nova stable/stein: Raise InstanceFaultRollback for UnableToMigrateToSelf from _prep_resize  https://review.opendev.org/66663919:17
openstackgerritMerged openstack/nova stable/stein: Avoid unnecessary joins in InstanceGroup.get_hosts  https://review.opendev.org/66427119:17
openstackgerritMerged openstack/nova stable/stein: Handle PortLimitExceeded in POST /servers/{server_id}/os-interface  https://review.opendev.org/66551319:17
openstackgerritMerged openstack/nova stable/rocky: Share snapshot image membership with instance owner  https://review.opendev.org/64385319:17
*** tbachman has quit IRC19:20
mriedemfried_rolls: gah -f value -c ID fails me again http://logs.openstack.org/94/667994/2/check/nova-next/1519821/logs/devstack-gate-post_test_hook.txt.gz#_2019-06-28_01_50_02_14519:20
*** ivve has quit IRC19:28
openstackgerritMatt Riedemann proposed openstack/nova master: Add integration testing for heal_allocations  https://review.opendev.org/66799419:29
*** Sundar has quit IRC19:29
*** dklyle has quit IRC19:36
*** maciejjozefczyk has quit IRC19:39
*** Sundar has joined #openstack-nova19:42
mriedemfried_rolls: what are your thoughts on doing an os-traits release next week? trying to workaround not having https://review.opendev.org/#/c/667209/ released is becoming a pain.19:45
*** fried_rolls is now known as efried19:49
efriedmriedem: There's a release proposed right now, which includes ^. smcginnis is waiting for cdent (PTL) ack, but we could probably convince him to push it now if it's going to help you.19:49
efriedmriedem: https://review.opendev.org/66711119:50
smcginnisIf it's holding up something, we can push it through. Even though it's a Friday and we generally try to avoid that kind of thing right before the weekend.19:50
smcginnisJust depends how time critical it is.19:51
efriedthis one's pretty benign, release-risk-wise19:51
efriedlike, really really benign19:51
efriednearly betegn19:51
smcginnisYeah, I figured something like that would be low risk.19:51
mriedemcool thanks19:52
efriedmriedem: you're still a placement core. Add your +1 and let's rip it. cdent will forgive us.19:53
mriedemdone19:54
efriednice19:54
efriedsmcginnis: ^19:54
smcginnisOK, I'll send 'er through. And watch out for the wrath of cdent. :D19:54
*** tbachman has joined #openstack-nova19:55
efriedthank you sir19:55
efriedmriedem: You'll need requirements bump before you can use it anyway.19:56
mriedemoh i know19:58
mriedembut that's automatic19:58
mriedemthe proposal anyway19:58
*** BjoernT has joined #openstack-nova19:59
*** BjoernT has quit IRC20:04
*** BjoernT has joined #openstack-nova20:08
mriedemAHA https://bugs.launchpad.net/nova/+bug/183469120:09
openstackLaunchpad bug 1834691 in OpenStack Compute (nova) "AttributeError: 'HostState' object has no attribute 'failed_builds'" [Undecided,New]20:09
*** dklyle has joined #openstack-nova20:18
efriedmriedem: takes a while though.20:19
efried(requirements bot)20:20
openstackgerritMatt Riedemann proposed openstack/nova master: Init HostState.failed_builds  https://review.opendev.org/66824320:21
mriedemwhat doesn't take awhile is +2ing this trivial bug fix ^ melwitt dansmith efried20:21
*** slaweq has quit IRC20:23
efriedmriedem: done. tbc the other thing is a separate issue?20:38
mriedemyup20:38
mriedempatch forthcoming20:38
efriedmriedem: requirements bump https://review.opendev.org/#/c/668245/20:39
efriedsmcginnis: https://review.opendev.org/#/c/668245/ do you have the power?20:39
*** mvkr has quit IRC20:40
smcginnisefried: Just waiting for zuul to report results on that.20:40
efriedslave to process you are, sheesh20:40
smcginnisI can +2, but don't want to ninja that one through.20:40
smcginnis:D20:40
Sundarefried, cdent: Is there a master topic/blueprint for all nested magic stuff? I am looking for better query than https://review.opendev.org/#/q/project:openstack/placement+nested20:42
efriedSundar: It's all in a single spec20:43
efriedSundar: https://docs.openstack.org/placement/latest/specs/train/approved/2005575-nested-magic-1.html20:43
efriedunless you mean a doc for all currently implemented features. That would be https://docs.openstack.org/placement/latest/usage/provider-tree.html20:43
efriedplus the api-reference https://developer.openstack.org/api-ref/placement/20:44
SundarWe've also had: https://review.opendev.org/657419. Some of your specs, since abandoned (like https://review.opendev.org/658510) were also useful to read20:44
SundarIt would be useful to have a single topic or blueprint for all of these. What do you think?20:45
efriedSundar: That patch was for the arbitrary suffix feature, which is included in the above spec.20:46
efriedIf you want to track all the patches, you can look at the story20:46
efriedhttps://storyboard.openstack.org/#!/story/200557520:46
efriedagree that topic branch story/$story-$task isn't as useful as just story/$story for searching purposes.20:47
efriedbut too late to modify topic for merged patches, sorry.20:47
SundarNP, got it. Good to know that the spec and storyboard is all there is. Is there a spec/patch on Nova side for how it all comes together -- from flavor to Placement query?20:49
Sundari.e. 'all there is to read for somebody to understand the whole shebang20:49
*** mvkr has joined #openstack-nova20:53
Sundarefried: ^20:55
efriedSundar: There's not likely to be something that high level. And there's as yet no blueprint/spec I'm aware of on the nova side for exploiting these features.21:02
melwittmriedem: the failed builds thing, my fave21:02
*** mvkr has quit IRC21:07
*** slaweq has joined #openstack-nova21:11
*** Sundar has quit IRC21:12
openstackgerritMatt Riedemann proposed openstack/nova master: Update HostState.*_allocation_ratio earlier  https://review.opendev.org/66825221:14
*** slaweq has quit IRC21:15
*** pcaruana has quit IRC21:16
*** dpawlik has joined #openstack-nova21:20
*** mvkr has joined #openstack-nova21:21
*** dpawlik has quit IRC21:22
*** tbachman has quit IRC21:32
*** tbachman has joined #openstack-nova21:33
mriedemwow some fun debugging here21:53
mriedemon initial start of a comptue service, we'll create the compute node, then later try to update it here https://github.com/openstack/nova/blob/324da0532f3b59aa16233a93a260d289e55860fb/nova/compute/resource_tracker.py#L101021:53
mriedemif that fails, we've changed old_resources and then a subsequent run will return false from _resource_change and we'll go on to create the resource provider in placement and report inventory21:53
mriedembut the compute node is botched21:53
mriedemand then things in the scheduler blow up if we try to use that node21:54
*** tbachman_ has joined #openstack-nova21:59
*** tbachman has quit IRC21:59
*** tbachman_ is now known as tbachman21:59
openstackgerritMerged openstack/nova stable/queens: libvirt: Do not reraise DiskNotFound exceptions during resize  https://review.opendev.org/66036322:00
*** slaweq has joined #openstack-nova22:11
*** slaweq has quit IRC22:16
mriedemefried: melwitt: would you think that on start of the compute service, if we hit a DBConnectionError here https://github.com/openstack/nova/blob/324da0532f3b59aa16233a93a260d289e55860fb/nova/compute/manager.py#L8129 and startup=true, we should bail and not let the service start?22:20
mriedemi guess it might correct itself...22:21
mriedemmaybe that's why we don't bail out22:21
efriedmriedem: it might. But in the meantime, everything is effed. Should at least disable the service at that point.22:21
mriedemheh, if you can....need a db connection to do that :)22:21
melwittI know that in tripleo-land they don't want it to bail out because of puppet starting up services in different orders etc22:22
melwittthey want it to keep retrying the db (for at least a limited time) before bailing out22:22
mriedemyeah, i wonder if https://docs.openstack.org/nova/latest/configuration/config.html#database.use_db_reconnect would help since it's disabled by default i've never tried it22:23
melwittI've dealt multiple times with tickets getting opened for not services not retrying [temporarily] upon startup22:23
melwittmriedem: are you saying it is or isn't bailing out today? sorry I don't remember how it behaves at this point. some of our services retry db connection underneath because oslo.service does it. but for some reason that I don't remember, it didn't work on compute22:24
mriedemuse_db_reconnect wouldn't help nova since it only tells a decorator in oslo.db it's ok to retry (safe_for_db_retry) which we don't use22:25
mriedemhttps://docs.openstack.org/nova/latest/configuration/config.html#database.max_retries indicates that we'd have retries by default on connection failures22:26
mriedemwith 10 seconds between retries by default (according to retry_interval)22:26
melwittyeah. last I knew, compute does not retry and I can't remember if I figured out why it doesn't. because from all evidence, it should, the way all our other services do22:26
melwittlet me dig up that old moldy BZ22:27
mriedemit'd be whatever conductor the compute is talking to22:27
mriedemanyway, the connectoin failure doesn't kill the compute on startup so we can get into a weird state with what nova has in the compute nodes table and what is in placement for the provider22:28
mriedemi've debugged one issue as a result https://bugs.launchpad.net/nova/+bug/183471222:28
openstackLaunchpad bug 1834712 in OpenStack Compute (nova) "ResourceTracker._update should restore previous old_resources value if ComputeNode.save fails" [Medium,Triaged]22:28
efriedwe need some kind of decorator to... like... @safely_connect22:28
mriedemheh22:28
mriedemi mean, i have a very unrealistic overloaded host situation here22:29
melwittthat bug report makes my head hurt22:29
efriedand with that lightning jolt of sarcasm, I'm outta here. See y'all Monday.22:29
melwittI think that situation would be realistic for our RDO cloud. I've seen so much weird crap happen with it22:29
mriedemyeah this is a critical section it turns out https://github.com/openstack/nova/blob/324da0532f3b59aa16233a93a260d289e55860fb/nova/compute/resource_tracker.py#L1005-L101222:30
melwittlike orphaned build request records etc because of spurious db errors22:30
melwittinstance records that live in both cell0 and cell1 (I still don't get how on earth that is possible to happen)22:30
mriedemand _resource_change mutates a thing in the RT that is checked on a later periodic run22:30
*** factor has joined #openstack-nova22:31
melwittyeah, I see22:31
melwittI'd be in favor of your rollback idea, or any other way to keep it in sync without killing the service22:31
openstackgerritMatt Riedemann proposed openstack/nova master: Update HostState.*_allocation_ratio earlier  https://review.opendev.org/66825222:33
mriedemyeah killing the service was an idea but not a good one b/c of unknown side effects22:33
melwittyeah22:34
*** brinzhang1 has joined #openstack-nova22:38
*** rajinir has quit IRC22:42
*** luksky has quit IRC22:52
openstackgerritMatt Riedemann proposed openstack/nova master: Restore RT.old_resources if ComputeNode.save() fails  https://review.opendev.org/66826323:05
*** whoami-rajat has quit IRC23:13
openstackgerritMerged openstack/nova stable/stein: libvirt: flatten rbd images when unshelving an instance  https://review.opendev.org/66811823:22
*** spatel has joined #openstack-nova23:25
*** spatel has quit IRC23:29
*** _alastor_ has joined #openstack-nova23:58

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!