Thursday, 2018-10-25

« Wednesday, 2018-10-24 Index Friday, 2018-10-26 »
*** gyee has quit IRC00:14
*** openstackgerrit has joined #openstack-nova00:18
openstackgerritmelanie witt proposed openstack/nova stable/rocky: Add rolling upgrade info to enable_consoleauth workaround reno  https://review.openstack.org/61317400:18
*** takashin has joined #openstack-nova00:23
*** trungnv has quit IRC00:35
openstackgerritTao Li proposed openstack/python-novaclient master: Deprecate the unused instance-name  https://review.openstack.org/60252000:54
*** bhagyashris has joined #openstack-nova01:08
openstackgerritmelanie witt proposed openstack/nova master: Add nova-status upgrade check for consoles  https://review.openstack.org/61121401:11
*** brinzhang has joined #openstack-nova01:13
*** mrsoul has quit IRC01:19
*** hongbin has joined #openstack-nova01:21
*** tiendc has joined #openstack-nova01:33
*** mhen has quit IRC01:40
*** Dinesh_Bhor has joined #openstack-nova01:44
*** mhen has joined #openstack-nova01:45
*** efried has quit IRC01:49
*** efried has joined #openstack-nova02:01
*** tinwood has quit IRC02:10
*** tinwood has joined #openstack-nova02:11
*** READ10 has quit IRC02:12
*** READ10 has joined #openstack-nova02:21
*** tetsuro has joined #openstack-nova02:23
*** munimeha1 has quit IRC02:30
openstackgerritTakashi NATSUME proposed openstack/nova master: Add a hacking rule for deprecated assertion methods  https://review.openstack.org/61318402:51
*** psachin has joined #openstack-nova02:54
*** READ10 has quit IRC02:54
*** alex_xu has joined #openstack-nova02:55
*** tbachman_ has joined #openstack-nova02:57
*** tbachman has quit IRC02:58
*** tbachman_ is now known as tbachman02:58
openstackgerritFan Zhang proposed openstack/nova master: Retry after hitting libvirt error VIR_ERR_OPERATION_INVALID in live migration.  https://review.openstack.org/61227203:03
*** Dinesh_Bhor has quit IRC03:07
*** Dinesh_Bhor has joined #openstack-nova03:13
*** itlinux has quit IRC03:17
*** lpetrut has joined #openstack-nova03:30
*** cfriesen has quit IRC03:50
*** lpetrut has quit IRC03:56
*** whoami-rajat has joined #openstack-nova03:56
*** lbragstad has quit IRC04:00
*** hongbin has quit IRC04:08
*** erlon has joined #openstack-nova04:10
*** janki has joined #openstack-nova04:12
*** ivve has joined #openstack-nova04:26
*** udesale has joined #openstack-nova04:27
openstackgerritMerged openstack/python-novaclient master: Deprecate the unused instance-name  https://review.openstack.org/60252004:29
*** tetsuro has quit IRC04:33
*** brinzh has joined #openstack-nova04:33
*** tetsuro has joined #openstack-nova04:33
*** brinzhang has quit IRC04:36
*** spsurya has joined #openstack-nova04:50
*** ratailor has joined #openstack-nova04:58
*** k_mouza has joined #openstack-nova05:03
*** k_mouza has quit IRC05:07
*** Dinesh_Bhor has quit IRC05:20
*** Dinesh_Bhor has joined #openstack-nova05:23
*** erlon has quit IRC05:26
openstackgerrityenai proposed openstack/nova master: Delete redundant spaces  https://review.openstack.org/61319905:38
*** brinzhang has joined #openstack-nova05:43
*** brinzh has quit IRC05:45
*** adrianc has joined #openstack-nova05:53
*** Luzi has joined #openstack-nova06:00
*** lpetrut has joined #openstack-nova06:03
*** Dinesh_Bhor has quit IRC06:08
*** Dinesh_Bhor has joined #openstack-nova06:10
*** moshele has joined #openstack-nova06:22
*** slaweq has joined #openstack-nova06:35
*** whoami-rajat has quit IRC06:54
*** rcernin has quit IRC07:00
*** pcaruana has joined #openstack-nova07:04
*** cfriesen has joined #openstack-nova07:08
*** yingjun has joined #openstack-nova07:08
*** ccamacho has joined #openstack-nova07:09
*** SpamapS has quit IRC07:11
*** jpena|off is now known as jpena07:15
*** gibi has joined #openstack-nova07:23
*** SpamapS has joined #openstack-nova07:24
*** aperevalov has quit IRC07:35
*** aperevalov has joined #openstack-nova07:35
*** helenafm has joined #openstack-nova07:38
*** sridharg has joined #openstack-nova07:39
openstackgerritTakashi NATSUME proposed openstack/nova master: api-ref: Remove unnecessary minimum microversion  https://review.openstack.org/61322607:40
*** alexchadin has joined #openstack-nova07:42
*** Dinesh_Bhor has quit IRC07:42
openstackgerritTakashi NATSUME proposed openstack/nova master: api-ref: Add descriptions of error cases  https://review.openstack.org/61042907:46
*** Dinesh_Bhor has joined #openstack-nova07:46
openstackgerritTakashi NATSUME proposed openstack/nova master: Add API ref guideline for examples  https://review.openstack.org/60406007:46
openstackgerritTakashi NATSUME proposed openstack/nova master: Add API ref guideline for body text  https://review.openstack.org/60562807:47
*** cfriesen has quit IRC07:48
*** ralonsoh has joined #openstack-nova07:50
*** ratailor_ has joined #openstack-nova07:56
*** ratailor has quit IRC07:57
*** sapd1 has quit IRC07:57
*** sapd1 has joined #openstack-nova07:58
*** ccamacho has quit IRC08:00
*** ccamacho has joined #openstack-nova08:01
*** helenafm has quit IRC08:08
*** bjolo has joined #openstack-nova08:10
*** moshele has quit IRC08:14
*** k_mouza has joined #openstack-nova08:15
*** ttsiouts has joined #openstack-nova08:26
*** takashin has left #openstack-nova08:31
*** k_mouza has quit IRC08:35
openstackgerritTetsuro Nakamura proposed openstack/nova-specs master: Spec: Support filtering by forbidden aggregate  https://review.openstack.org/60335208:36
openstackgerritSeyeong Kim proposed openstack/nova master: Enable connection_info refresh for new-style attachments  https://review.openstack.org/57900408:38
*** helenafm has joined #openstack-nova08:47
*** ccamacho has quit IRC08:54
*** ccamacho has joined #openstack-nova08:55
*** ttsiouts has quit IRC09:02
*** ttsiouts has joined #openstack-nova09:02
*** dtantsur|afk is now known as dtantsur09:09
*** ttsiouts has quit IRC09:21
*** ttsiouts has joined #openstack-nova09:22
*** ratailor has joined #openstack-nova09:22
*** ratailor_ has quit IRC09:23
*** ccamacho has quit IRC09:27
*** tbachman_ has joined #openstack-nova09:31
*** tbachman has quit IRC09:32
*** tbachman_ is now known as tbachman09:32
*** psachin has quit IRC09:35
*** panda|off has quit IRC09:35
pooja_jadhavhi team, I facing issue on nova-compute : (1054, u"Unknown column 'block_device_mapping.volume_type' in 'field list'"). For that I have did nova-manage db sync and api_db sync as well. still facing same issue. Can anyone help me to resolve this issue?09:37
*** panda has joined #openstack-nova09:38
*** yingjun has quit IRC09:38
*** derekh has joined #openstack-nova09:39
*** psachin has joined #openstack-nova09:40
*** ratailor_ has joined #openstack-nova09:43
*** ratailor has quit IRC09:46
*** adrianc has quit IRC09:46
*** adrianc has joined #openstack-nova09:51
*** psachin has quit IRC10:03
*** bhagyashris has quit IRC10:04
*** psachin has joined #openstack-nova10:05
openstackgerritJan Gutter proposed openstack/os-vif master: Extend port profiles with datapath offload type  https://review.openstack.org/57208110:09
*** cdent has joined #openstack-nova10:10
*** ttsiouts has quit IRC10:11
*** Dinesh_Bhor has quit IRC10:11
*** spatel has joined #openstack-nova10:22
*** psachin has quit IRC10:25
*** spatel has quit IRC10:27
*** psachin has joined #openstack-nova10:27
*** Dinesh_Bhor has joined #openstack-nova10:29
*** sahid has joined #openstack-nova10:32
*** udesale has quit IRC10:36
*** sahid has quit IRC10:38
*** sahid has joined #openstack-nova10:41
openstackgerritRadoslav Gerganov proposed openstack/nova stable/pike: Don't persist zero allocation ratios in ResourceTracker  https://review.openstack.org/61326310:48
*** aperevalov has quit IRC10:48
openstackgerritRadoslav Gerganov proposed openstack/nova stable/pike: Update resources once in update_available_resource  https://review.openstack.org/61229510:50
*** Dinesh_Bhor has quit IRC10:56
openstackgerritRadoslav Gerganov proposed openstack/nova stable/queens: Don't persist zero allocation ratios in ResourceTracker  https://review.openstack.org/61327111:02
*** psachin has quit IRC11:02
*** moshele has joined #openstack-nova11:04
openstackgerritRadoslav Gerganov proposed openstack/nova stable/queens: Update resources once in update_available_resource  https://review.openstack.org/61229411:04
*** ttsiouts has joined #openstack-nova11:04
*** ccamacho has joined #openstack-nova11:07
*** dave-mccowan has joined #openstack-nova11:16
*** tbachman_ has joined #openstack-nova11:21
*** tbachman has quit IRC11:22
*** tbachman_ is now known as tbachman11:22
*** erlon has joined #openstack-nova11:29
*** jpena is now known as jpena|lunch11:34
*** lpetrut has quit IRC11:44
*** ratailor_ has quit IRC12:01
*** gibi has quit IRC12:03
*** erlon has quit IRC12:07
*** yingjun has joined #openstack-nova12:08
*** tbachman_ has joined #openstack-nova12:12
*** tbachman_ has quit IRC12:14
*** jistr_ is now known as jistr12:14
*** gibi has joined #openstack-nova12:16
*** tbachman has quit IRC12:16
openstackgerritTetsuro Nakamura proposed openstack/nova master: Add recreate test for bug 1799892  https://review.openstack.org/61330412:21
openstackbug 1799892 in OpenStack Compute (nova) "Placement API crashes with 500s in Rocky upgrade with downed compute nodes" [Undecided,New] https://launchpad.net/bugs/179989212:21
openstackgerritTetsuro Nakamura proposed openstack/nova master: Consider root id is None in the database case  https://review.openstack.org/61330512:21
*** tetsuro has quit IRC12:23
*** pcaruana has quit IRC12:26
*** beagles is now known as beagles_mtg12:29
*** janki has quit IRC12:31
*** janki has joined #openstack-nova12:31
*** liuyulong has quit IRC12:31
*** jpena|lunch is now known as jpena12:33
* cdent sighs at ^12:34
cdentmnaser gets the gold star award for finding good bugs12:34
*** pcaruana has joined #openstack-nova12:39
*** artom has quit IRC12:41
*** ttsiouts has quit IRC12:47
openstackgerritJan Gutter proposed openstack/os-vif master: Extend port profiles with datapath offload type  https://review.openstack.org/57208112:48
*** ttsiouts has joined #openstack-nova12:48
*** erlon has joined #openstack-nova12:51
*** brinzhang has quit IRC12:53
*** janki has quit IRC12:56
*** bnemec has joined #openstack-nova12:56
*** awaugama has joined #openstack-nova13:00
*** lpetrut has joined #openstack-nova13:03
*** alexchadin has quit IRC13:05
*** liuyulong has joined #openstack-nova13:08
*** tbachman has joined #openstack-nova13:11
openstackgerritAdam Spiers proposed openstack/nova-specs master: Add spec for libvirt driver launching AMD SEV-encrypted instances  https://review.openstack.org/60977913:15
*** tbachman has quit IRC13:15
*** artom has joined #openstack-nova13:16
*** artom has quit IRC13:16
*** artom has joined #openstack-nova13:17
*** eharney has joined #openstack-nova13:19
*** tbachman has joined #openstack-nova13:20
openstackgerritMerged openstack/nova master: Remove the CachingScheduler  https://review.openstack.org/61172313:22
*** artom has quit IRC13:23
*** artom has joined #openstack-nova13:23
*** spatel has joined #openstack-nova13:24
spatelsean-k-mooney: morning13:24
spatelI did debug and here is the source machine log http://paste.openstack.org/show/733042/13:25
sean-k-mooneyspatel: hi o/13:25
spatelThere are some ERROR13:26
spatelsomething like this - MessagingTimeout: Timed out waiting for a reply to message ID 80fcdfa3d90d4c12828482202498270613:27
sean-k-mooneythe messaging timeout is strange but may just meean the compute node was not back up after your config change13:27
spatelLet me tell you what happened...13:27
spatel1. build foo instance13:27
spatel2. did block migration - and it failed with error13:28
spatel3. tried again and succeeded but vm failed to boot on destination compute node13:28
*** jistr is now known as jistr|call13:29
spatelif you see my log i have break them in two section first one is for first attempt and second block is retried13:29
sean-k-mooneyright13:29
sean-k-mooneythe second migration looks ok bu i take it you had the same truncated disk on the dest13:30
spatelyes it had only 1MB disk on destination, it didn't copy full disk13:30
spatelare you seeing anything interesting in logs?13:31
*** rnoriega has quit IRC13:31
sean-k-mooneyno it looks relitivly normal13:32
spatelhmm!13:32
spatelDo you need this line in - live_migration_flag=VIR_MIGRATE_UNDEFINE_SOURCE,VIR_MIGRATE_PEER2PEER,VIR_MIGRATE_LIVE13:32
spatelI have meeting i will be back in few mi n13:33
spatelmin13:33
*** lbragstad has joined #openstack-nova13:33
sean-k-mooneyspatel:  i think we removed those flags a few releases go from the config13:34
sean-k-mooneymdbooth: are you aware of any bugs related to block_migration with raw images13:35
mdboothsean-k-mooney: Can you be more specific?13:35
sean-k-mooneymdbooth: the migration is completing but the disk image is truncated on the destination13:36
mdboothOh...13:36
sean-k-mooneymdbooth: then on reboot the vm is borked because the disk is curropt13:36
* mdbooth reads backscroll13:36
*** pcaruana has quit IRC13:37
*** rnoriega has joined #openstack-nova13:38
mdboothblock migration == cold migration, right?13:39
sean-k-mooneymdbooth: no live migration with block migrations of disk13:39
mdbooth... and the guest is fine until reboot?13:39
sean-k-mooneye.g. no shared sotrage and not boot form volume13:39
sean-k-mooneymdbooth: yes the guest is alive after live migration and then if you reboot it it erros out13:40
mdboothThat's improbable.13:40
mdboothThere's something going on there.13:40
mdboothsomething else...13:40
sean-k-mooneymdbooth: fine is proably an over statement the kernel is running from ram at that point13:40
sean-k-mooneyif you try to use the disk its proably going to break13:40
mdboothENOTENOUGHINFO13:41
*** alexchadin has joined #openstack-nova13:44
*** erlon has quit IRC13:46
*** spatel has quit IRC13:48
openstackgerritBalazs Gibizer proposed openstack/nova master: Reject forced move with nested source allocation  https://review.openstack.org/60578513:48
openstackgerritBalazs Gibizer proposed openstack/nova master: Run negative server moving tests with nested RPs  https://review.openstack.org/60412513:50
sean-k-mooneymdbooth: by the way looking at http://paste.openstack.org/show/733042/ is it valid to use imagecahing with raw images?13:50
*** beagles_mtg is now known as beagles13:51
*** rpittau has quit IRC13:52
mdboothsean-k-mooney: I don't know what I'm looking at there, seems truncated. However yes, imagecache is always used. I rewrote all that stuff to make it a whole lot more obvious a couple of years ago, btw, but it didn't land.13:53
openstackgerritBalazs Gibizer proposed openstack/nova master: Handle allocations consuming only from the child RPs  https://review.openstack.org/60829813:53
*** mchlumsky_ has quit IRC13:54
sean-k-mooneymdbooth: ok you looking at part of a systemd log for the n-cpus i think. its an openstack ansible deployment so not really sure how they do logging13:55
openstackgerritMerged openstack/nova master: Drop legacy cold migrate allocation compat code  https://review.openstack.org/61197013:55
sean-k-mooneymdbooth: anywya what it feels like is only the delta disk was copied insteda of the full disk13:55
mdboothsean-k-mooney: There is no delta disk for raw.13:56
sean-k-mooneywhen spatel requires i might suggest moving to qcow2 for not raw13:56
mdboothsean-k-mooney: Is it possible that they've got qcow2 on the source and flat on the dest?13:56
*** efried has quit IRC13:56
mdboothBecause if so then I'm pretty sure that would break.13:56
openstackgerritBalazs Gibizer proposed openstack/nova master: Add request_spec.RequestGroup versioned object  https://review.openstack.org/56884013:57
sean-k-mooneyam maybe i would be very surpried but that could explain things13:57
*** efried has joined #openstack-nova13:57
openstackgerritBalazs Gibizer proposed openstack/nova master: Add requested_resources field to RequestSpec  https://review.openstack.org/56726713:57
mdboothThe reason being we explicitly only copy the delta, as we copy the backing file explicitly oob.13:57
mdboothBut if the dest was raw/flat there would be no backing file.13:57
openstackgerritBalazs Gibizer proposed openstack/nova master: Add bandwidth related standard resource classes  https://review.openstack.org/57084713:57
*** Luzi has quit IRC13:57
sean-k-mooneymdbooth: ill ask for more info when spatel is back on line. they were going to open a bug for this anyway13:57
mdboothThat would be NOTABUG13:58
mdboothWe don't support that. Not that it wouldn't be nice, but we don't.13:58
sean-k-mooneywell its a bug that nova is not chekcing for this and refusing to migrate13:58
sean-k-mooneyfrom an api persepcitive the migration completed sucessfully13:58
sean-k-mooneyit also deleted the original disk on the souce13:58
mdboothDid we ever get sticky host aggregates?13:59
mdboothi.e. we don't care which host aggregate is selected, but whichever one it is we never go anywhere else.13:59
sean-k-mooneye.g. can migrate out of them by defualt without force13:59
*** mlavalle has joined #openstack-nova13:59
sean-k-mooneymdbooth: no that only happens for AZs today14:00
sean-k-mooneyat least as far as i am aware14:00
*** spatel has joined #openstack-nova14:06
spatelback14:06
*** mchlumsky has joined #openstack-nova14:06
spatelsean-k-mooney: Do you want me to file a bug on this issue14:07
spatelso we have more eyeballs14:08
openstackgerritMatt Riedemann proposed openstack/nova master: Update reserved/allocation_ratio in placement inventory if config changes  https://review.openstack.org/61312614:09
sean-k-mooneyyes please. one thing that mdbooth suggested was to check that the raw images are used on both compute hosts14:10
sean-k-mooneyspatel: if you are using qcow on the souce node and raw on the dest we think it could cause the issue you are seeing14:12
spatelI am using raw image14:12
mdboothspatel: On both source and destination computes?14:13
openstackgerritMerged openstack/nova master: Consider allocations invovling child providers during allocation cleanup  https://review.openstack.org/60605014:13
spatelI think yes.. i think if you move machine from A to B disk alway stay same right?14:14
mdboothspatel: look in nova.conf14:14
spatelwhat option you want to check14:14
mdboothOn both source and destination computes14:14
mdboothimages_type14:15
spatelI don't have that option in nova.conf so must be using default one14:15
spatelis default Qcow ?14:16
mdboothYes14:16
sean-k-mooneymdbooth: if you have a raw image in glance it gets converted to qcow by defualt correct14:16
mdboothActually, default is modified by use_cow config option14:16
mdboothspatel: Do you have use_cow set on either source or dest?14:16
sean-k-mooneywhich defualt to True14:16
spateluse_cow where? on nova.conf  ( no)14:18
mdboothspatel: Yes.14:18
mdboothuse_cow_images on both source and dest14:18
spatelI don't have that option in  nova.conf14:18
sean-k-mooneyok so in that chase your vms should be using qcow images when spawned14:19
spatelcat /etc/nova/nova.conf | grep use_cow   (nothing found)14:19
mdboothsean-k-mooney: IIRC the flat backend can not convert qcow2 glances images if so configured, but they're always flat (no backend)14:19
openstackgerritMatt Riedemann proposed openstack/nova master: Fix min config value for shutdown_timeout option  https://review.openstack.org/61302814:19
mdboothspatel: Are you using containers?14:19
mdboothspatel: i.e. are you sure you're looking in the correct nova.conf14:19
spatelNo containers14:20
sean-k-mooneymdbooth: i think the glance iamges are raw not qcow spatel can you do an image show?14:20
sean-k-mooneyspatel: is this not an openstack stack ansible deployment. it used to default to lxc contaier no?14:20
spateldisk_format               | raw14:21
spatelYes i am using openstack-ansible ( it only use LXC for infra services like glance, neutron etc..) compute node run without container14:21
spatelI am 100% sure i am looking at correct /etc/nova/nova.conf14:22
openstackgerritMaciej Jozefczyk proposed openstack/nova master: Force refresh instance info_cache during heal  https://review.openstack.org/59160714:23
-spatel- [root@ostack-compute-82 ~]# locate nova.conf14:23
-spatel- /etc/nova/nova.conf14:23
spatelI have only one file called nova.conf14:23
-spatel- [root@ostack-compute-82 6a441aa4-448c-4ee1-be2b-515371baa0a1]# file disk14:24
-spatel- disk: QEMU QCOW Image (v3), has backing file (path /var/lib/nova/instances/_base/68e4d13dacff5cffeaacecf533afab659), 42949672960 bytes14:24
*** ttsiouts has quit IRC14:24
sean-k-mooneyspatel: ok can you wrap this all up in a bug. the more infor you can provide the petter so if you can link an sanitised copy of your nova.conf that would help alos14:24
sean-k-mooneye.g. remove any sensive info such as password or ips that you dont want to share14:25
spatelI will :)14:25
*** erlon has joined #openstack-nova14:25
spatelLast question my glance image showing raw image but when i deploy VM it convert image to QCOW why?14:25
spatelis that normal ?14:26
*** ttsiouts has joined #openstack-nova14:26
openstackgerritBalazs Gibizer proposed openstack/nova master: Transfer port.resource_request to the scheduler  https://review.openstack.org/56726814:28
openstackgerritBalazs Gibizer proposed openstack/nova master: Send resource allocations in the port binding  https://review.openstack.org/56945914:28
openstackgerritBalazs Gibizer proposed openstack/nova master: Test boot with more ports with bandwidth request  https://review.openstack.org/57331714:28
*** moshele has quit IRC14:29
*** sridharg has quit IRC14:29
-openstackstatus- NOTICE: Zuul and Nodepool services are being restarted to migrate them to a new Zookeeper cluster. THis brings us an HA database running on newer servers.14:40
*** maciejjozefczyk has quit IRC14:42
*** munimeha1 has joined #openstack-nova14:42
*** tiendc has quit IRC14:45
*** yingjun has quit IRC14:49
*** Swami has joined #openstack-nova14:49
*** jistr|call is now known as jistr14:50
*** Swami has quit IRC14:50
openstackgerritMatt Riedemann proposed openstack/nova master: Drop legacy live migrate allocation compat code  https://review.openstack.org/61197514:59
openstackgerritMatt Riedemann proposed openstack/nova master: Remove restart_scheduler_service() method  https://review.openstack.org/61334814:59
*** cfriesen has joined #openstack-nova15:01
*** Luzi has joined #openstack-nova15:01
*** jpena is now known as jpena|brb15:04
*** alexchadin has quit IRC15:05
*** ccamacho has quit IRC15:05
*** erlon has quit IRC15:13
*** jpena|brb is now known as jpena15:15
*** helenafm has quit IRC15:18
*** Luzi has quit IRC15:26
-openstackstatus- NOTICE: The Zuul and Nodepool database transition is complete. Changes updated during the Zuul outage may need to be rechecked.15:31
melwitt15:33
openstackgerritMatt Riedemann proposed openstack/nova master: Migrate old style volume attachments on nova-compute startup  https://review.openstack.org/54913015:36
*** ccamacho has joined #openstack-nova15:38
*** liuyulong is now known as liuyulong|away15:54
openstackgerritMerged openstack/nova-specs master: Support initial allocation ratios  https://review.openstack.org/55210515:56
*** swamireddy has quit IRC15:57
*** derekh has quit IRC15:58
*** spatel has quit IRC16:00
*** itlinux has joined #openstack-nova16:06
*** sahid has quit IRC16:08
*** spatel has joined #openstack-nova16:09
*** ccamacho has quit IRC16:15
*** gyee has joined #openstack-nova16:16
*** ivve has quit IRC16:16
*** ivve has joined #openstack-nova16:17
stephenfincfriesen, bauzas, artom, sean-k-mooney: So, what way are we going on https://review.openstack.org/611088 ?16:18
*** jiaopengju has quit IRC16:18
sean-k-mooneystephenfin: please have a config option and i would prefer it to not be enabled by defualt but i can live with it so long as i can turn it off16:19
stephenfinPersonally, I would rather do nothing because I consider the whole thing totally broken. However, that's likely not a realistic approach. As such, it's between a config option and not using the check for forced instances16:19
stephenfinsean-k-mooney: So default to allowing live migrations w/ NUMA or not allowing them?16:19
sean-k-mooneydefault to allowing it16:19
stephenfinWhy?16:19
sean-k-mooneymainly for ovs-dpdk16:19
cfriesenstephenfin: default to safe, allow operators to opt in to being unsafe16:19
sean-k-mooneyall ovs-dpdk deployment have numa toplogyes because of hugepages16:20
cfriesenopposite of sean-k-mooney  :)16:20
stephenfincfriesen: Yeah, that's what I was thinking16:20
stephenfinsean-k-mooney: Relatively speaking though, I imagine users with NUMA topologies >>> users with OVS-DPDK16:20
sean-k-mooneyi know yahoo japan use livemigration in there production ovs-dpdk based deployment16:21
cfriesensean-k-mooney: wouldn't anyone using ovs-dpdk also use pinned CPUs?16:21
*** jiaopengju has joined #openstack-nova16:21
sean-k-mooneycfriesen: no16:21
cfriesenfreaky16:21
sean-k-mooneythey may but that should not be assumed16:21
sean-k-mooneycfriesen: if you entire cloud uses ovs-dpdk you still have instance that you want to over subscive so no pinning16:22
*** dtantsur is now known as dtantsur|afk16:22
stephenfinsean-k-mooney: I imagine it's still fairly common though16:22
sean-k-mooneythere are 2 main usecases for ovsdpdk hosting providres that want to run allot of instnace on a singel host where the aggreate throuput exceed what kernel ovs can do16:23
sean-k-mooneyor a small number of vnf that each neeed a lot of bandwith16:23
sean-k-mooneythe frist group dont use pinning the second do16:23
stephenfincfriesen: So, to be clear, workaround option is good enough, as opposed to allowing it for migrations with a requested_destination?16:24
stephenfinFor you, at least16:24
*** jhesketh has joined #openstack-nova16:25
stephenfinsean-k-mooney: The way I'm thinking is that this has a potentially devastating impact on workloads and their performance, so if operators are knowingly doing it, they will switch the toggle16:25
sean-k-mooneystephenfin: in the pinning case16:25
stephenfinbut in the meanwhile, we'll catch all the other folks who didn't know this wasn't working the way they thought16:25
*** mnaser has quit IRC16:26
sean-k-mooneyin the hugepages case the livemigation will fail if it can allocat the right hugepages16:26
sean-k-mooneyand there is no performacne impact16:26
*** mnaser has joined #openstack-nova16:26
sean-k-mooneymodulo your numa aware vswithc stuff16:26
stephenfincan or can't?16:26
*** jhesketh_ has quit IRC16:27
sean-k-mooneythe migration will fail if it can not alloate hugepages16:27
cfriesenstephenfin: yes, workaround option would be good enough for me.  Let's keep it simple for now.16:27
sean-k-mooneythe dest node will emit a qemu error and we will abort the migration16:27
cfriesenI'd rather see live migrations get fixed than get fancy with the workaround.16:28
sean-k-mooneycfriesen: same16:28
stephenfincfriesen: Agreed. With any luck, this won't even survive the release16:28
stephenfinBut it'll be a useful backport, IMO16:28
stephenfinsean-k-mooney: There's also the non-homogeneous hosts case to consider too16:28
sean-k-mooneystephenfin: if you backport it it needs to be off by defualt16:28
sean-k-mooneystephenfin: yes i know16:29
stephenfinsean-k-mooney: Yup, in that case, agreed16:29
stephenfincfriesen: Cool. I'll wait to see if artom or bauzas want to pipe up. If not, I'll rework accordingly16:29
sean-k-mooneybasically want yahoo were doing is if the migration aported (that will happen in the non-homogeneous case too) then they did a manual migration specifying the host16:30
sean-k-mooneye.g. if it did not jsut work the found a node that could fit it and forced the migration by setting it16:30
stephenfinIt wouldn't fail if the guest ended up split over multiple host NUMA nodes though16:31
sean-k-mooneystephenfin: you can proably simply this a lot but doing the check at the api level and if the vm has a numa toplogy just reject the migration based on the config value16:31
stephenfinI know that brings us back into performance issue territory but still, just shows how this is all sorts of broken rn16:32
sean-k-mooneystephenfin: you are thinkg of pinning agin yes. and ture16:32
stephenfinNo, pinning isn't needed. We restrict an instance NUMA node to a range of host cores corresponding to host NUMA node16:33
cfriesensean-k-mooney: do you have any thoughts on whether we should support TIS vs CRB in the emulated TPM spec?  and if so, should that be a separate thing from the TPM version?16:33
sean-k-mooneystephenfin: yes it is but we know that you are technically in undefined behavior land if you migrate with a numa toplogy16:33
sean-k-mooneycfriesen: i know thate they are a thing but not enough to comment on that16:33
stephenfinAlso, mriedem noted about doing it higher but it seemed no simpler. This doesn't affect anything but libvirt so that still seems like the place to do this.16:34
sean-k-mooneycfriesen: prably use two traits one for the version and a second for the type16:34
cfriesensean-k-mooney: basically, with TPM 2.0 there are two different ways of talking to the TPM.16:34
cfriesenand qemu supports both16:34
sean-k-mooneyok16:34
cfriesenso I was thinking it's not really a trait16:34
cfriesensince it's not schedulable, at least for qemu16:34
sean-k-mooneywell then it s a flavor extra spec16:35
sean-k-mooneyand or image meta16:35
sean-k-mooneyso retuired trait for tpm_2.0 and hw:tpm_connection_mode=CRB16:36
cfriesenif they specify CRB with TPM 1.2, should I ignore the CRB request or error out?16:36
*** ttsiouts has quit IRC16:36
sean-k-mooneycfriesen: proabley error but you could leave that up to qemu16:36
sean-k-mooneylibvirt/qemu should be validteing there input and should reject it if invalide16:36
sean-k-mooneyyou could optimes by checking in the driver but not sure that needs to be in the spec16:37
cfriesenmakes sense16:37
*** panda is now known as panda|off16:38
*** ivve has quit IRC16:39
sean-k-mooneystephenfin: if you have a config or provide a way to bypass the check im ok with it16:39
sean-k-mooneystephenfin: if i cant turn it off you will break people and im not really ok with saying vpp ovs-dpdk, vrouter and snabb swtich deployments can nolonger live migrate16:40
*** imacdonn has quit IRC16:42
*** imacdonn has joined #openstack-nova16:43
*** adrianc_ has joined #openstack-nova16:52
*** lpetrut has quit IRC16:54
*** lpetrut has joined #openstack-nova16:55
*** moshele has joined #openstack-nova16:55
*** adrianc has quit IRC16:55
*** adrianc_ has quit IRC16:57
*** erlon has joined #openstack-nova16:57
*** jpena is now known as jpena|off16:58
kmalloccdent: out of curiosity, does Nova really support passing config data via paste-ini or is it mostly assumed you're passing through config via .conf files [even to middleware]?16:58
kmalloccdent: looking at how deep i need to support config passing into the new loader when allowing folks to drop paste.16:59
kmallocif you know.16:59
cdentit's not so much the passing of config data (although that may be in there) but the idea that paste.ini can be edited to add other middlewares16:59
kmallocright. that is fine.17:00
kmalloci'm planning on supporting that17:00
cdentand so it falls under the domain of stuff that we need to carry on dealing with incase people have their own changes17:00
kmallocthe other question i have is: if i only allow middleware to be injected into a specific place in the pipeline, where would that be?17:00
kmallocor do we need it to really be very dynamic17:00
kmalloc?17:00
cdentsean dague had more to say on the topic way back in the day, and probably new more about what was happening in the wild17:00
kmallocright. i can flip the coin either way, obviously i wont allow injecting things *after* nova's application17:01
cdentI'm unable to answer that question with  about "where" with regard to nova17:01
* kmalloc nods.17:01
cdentsome of the nova-api oriented folks might be able to (gmann, alex_xu )17:01
kmallocthnx :)17:02
*** ccamacho has joined #openstack-nova17:02
kmalloci figure nova is the ideal target to aim for here, since much follows in it's lead17:02
cdentyup17:03
*** pcaruana has joined #openstack-nova17:03
kmallockeystone was the outlier but we drew the line and said "nope, no unwhitelisted middleware in our app space" but mostly for security/maintenance/behavior reasons17:04
kmallocit's super easy to add outer middleware to anything, but i don't want to block people who legitimately need things somewhere in the middle17:04
kmallocsay post keystonemiddleware.17:04
* cdent nods17:04
* kmalloc will bug gmann and alex_xu as they appear on irc :)17:05
melwittthey're APAC, so you might consider the dev ML with [nova][api]17:07
*** Sundar has joined #openstack-nova17:09
*** eharney has quit IRC17:11
kmallocmelwitt: ah thnx. yeah Pacific has poor overlap with <insert most currently relevant timezones for OpenStack here>17:15
kmallocmelwitt: yeah i'll get an email spun up as well.17:15
*** moshele has quit IRC17:15
melwittsounds cool17:16
*** eharney has joined #openstack-nova17:26
*** lpetrut has quit IRC17:28
openstackgerritChris Dent proposed openstack/nova master: WIP: Integration tests using gabbi-tempest  https://review.openstack.org/61338617:32
*** eharney has quit IRC17:35
cfriesenis there a current recommendation on whether to enable the "send_service_user_token" config option?  It still defaults to "False", just wondering what the downsides would be...is there an additional keystone round-trip?17:46
*** eharney has joined #openstack-nova17:49
*** mvkr has quit IRC17:50
artomstephenfin, sorry, missed your ping, I think you settled on config option? I'm cool with that17:51
melwittcfriesen: found this in the spec, maybe that is the main downside https://specs.openstack.org/openstack/nova-specs/specs/ocata/implemented/use-service-tokens.html#performance-impact17:51
*** betherly has joined #openstack-nova17:55
*** cdent has quit IRC17:56
*** zzzeek_ has joined #openstack-nova17:59
*** betherly has quit IRC17:59
*** whoami-rajat has joined #openstack-nova18:10
openstackgerritMerged openstack/nova master: api-ref: Add descriptions of error cases  https://review.openstack.org/61042918:26
*** bnemec has quit IRC18:29
Sundarmelwitt: Do specs need to merged this week (for Stein milestone 1) to be considered for implementation in Stein?18:32
melwittSundar: no. our spec freeze is milestone 2, Jan 1018:33
Sundarmelwitt: Thanks. Is there any way to accelerate the coverage for the spec https://review.openstack.org/#/c/603955/ ?18:35
SundarI understand there are lots of other stuff in the Nova plate18:35
*** tbachman has quit IRC18:36
melwittSundar: looks like there was some review last week, and you've made some updates18:37
*** munimeha1 has quit IRC18:37
*** moshele has joined #openstack-nova18:38
melwittI can mention it in the nova meeting today that it's been updated and ready for review again18:39
SundarGreat. Thanks!18:39
melwittsean-k-mooney: could you please reply on this os-vif release review about whether we need a major version bump? https://review.openstack.org/61284418:47
*** itlinux has quit IRC18:52
*** moshele has quit IRC18:57
*** tbachman has joined #openstack-nova18:57
*** tbachman has quit IRC19:02
*** itlinux has joined #openstack-nova19:03
*** Sundar has quit IRC19:04
*** tbachman has joined #openstack-nova19:06
openstackgerritmelanie witt proposed openstack/nova master: Add functional recreate test for bug 1764556  https://review.openstack.org/56204119:07
openstackbug 1764556 in OpenStack Compute (nova) ""nova list" fails with exception.ServiceNotFound if service is deleted and has no UUID" [Medium,In progress] https://launchpad.net/bugs/1764556 - Assigned to melanie witt (melwitt)19:07
openstackgerritmelanie witt proposed openstack/nova master: Add functional regression test for bug 1778305  https://review.openstack.org/58240719:07
openstackgerritmelanie witt proposed openstack/nova master: Don't generate service UUID for deleted services  https://review.openstack.org/58240819:07
openstackbug 1778305 in OpenStack Compute (nova) "Nova may erronously look up service version of a deleted service, when hostname have been reused" [Undecided,In progress] https://launchpad.net/bugs/1778305 - Assigned to melanie witt (melwitt)19:07
melwittdangit, added a test change to the wrong commit19:10
*** spatel has quit IRC19:20
openstackgerritMatt Riedemann proposed openstack/nova master: Default AZ for instance if cross_az_attach=False and checking from API  https://review.openstack.org/46967519:21
openstackgerritMatt Riedemann proposed openstack/osc-placement stable/queens: Do not depend on jenkins user in devstack gate  https://review.openstack.org/55666619:23
openstackgerritMatt Riedemann proposed openstack/osc-placement stable/queens: Migrate legacy-osc-placement-dsvm-functional job in-tree  https://review.openstack.org/55663519:23
openstackgerritmelanie witt proposed openstack/nova master: Add functional recreate test for bug 1764556  https://review.openstack.org/56204119:30
openstackbug 1764556 in OpenStack Compute (nova) ""nova list" fails with exception.ServiceNotFound if service is deleted and has no UUID" [Medium,In progress] https://launchpad.net/bugs/1764556 - Assigned to melanie witt (melwitt)19:30
openstackgerritmelanie witt proposed openstack/nova master: Add functional regression test for bug 1778305  https://review.openstack.org/58240719:30
*** tbachman has quit IRC19:30
openstackgerritmelanie witt proposed openstack/nova master: Don't generate service UUID for deleted services  https://review.openstack.org/58240819:30
openstackbug 1778305 in OpenStack Compute (nova) "Nova may erronously look up service version of a deleted service, when hostname have been reused" [Undecided,In progress] https://launchpad.net/bugs/1778305 - Assigned to melanie witt (melwitt)19:30
openstackgerritMatt Riedemann proposed openstack/nova master: Use RequestSpec.user_id in scheduler.utils.claim_resources  https://review.openstack.org/56891719:42
openstackgerritMatt Riedemann proposed openstack/nova master: Cleanups for the scheduler code  https://review.openstack.org/52934319:42
*** lbragstad has quit IRC19:43
*** lbragstad has joined #openstack-nova19:43
openstackgerritMatt Riedemann proposed openstack/nova master: RT: replace _instance_in_resize_state with _is_trackable_migration  https://review.openstack.org/56046719:45
*** NostawRm has quit IRC19:45
*** irclogbot_2 has joined #openstack-nova20:00
*** moshele has joined #openstack-nova20:01
*** mriedem has joined #openstack-nova20:03
*** moshele has quit IRC20:04
openstackgerritMatt Riedemann proposed openstack/nova master: Block swap volume on volumes with >1 rw attachment  https://review.openstack.org/57279020:08
openstackgerritMatt Riedemann proposed openstack/nova master: Keep attach_mode as top-level field in _translate_attachment_ref  https://review.openstack.org/57441320:08
*** spatel has joined #openstack-nova20:16
*** irclogbot_2 has quit IRC20:22
openstackgerritMatt Riedemann proposed openstack/nova master: Add post-test hook for testing evacuate  https://review.openstack.org/60217420:26
openstackgerritMatt Riedemann proposed openstack/nova master: Add volume-backed evacuate test  https://review.openstack.org/60439720:26
openstackgerritMatt Riedemann proposed openstack/nova master: Add functional regression test for bug 1794996  https://review.openstack.org/60610620:26
openstackbug 1794996 in OpenStack Compute (nova) "_destroy_evacuated_instances fails and kills n-cpu startup if lazy-loading flavor on a deleted instance" [High,In progress] https://launchpad.net/bugs/1794996 - Assigned to Matt Riedemann (mriedem)20:26
openstackgerritMatt Riedemann proposed openstack/nova master: Fix InstanceNotFound during _destroy_evacuated_instances  https://review.openstack.org/60612220:26
openstackgerritMatt Riedemann proposed openstack/nova master: Refactor TestEvacuateDeleteServerRestartOriginalCompute  https://review.openstack.org/60870520:26
openstackgerritMatt Riedemann proposed openstack/nova master: Run evacuate tests with local/lvm and shared/rbd storage  https://review.openstack.org/60440020:27
*** imacdonn has quit IRC20:31
*** imacdonn has joined #openstack-nova20:32
*** jcosmao has quit IRC20:35
*** awaugama has quit IRC20:48
melwittnova meeting in 10 min20:50
*** takashin has joined #openstack-nova20:50
*** betherly has joined #openstack-nova20:56
*** munimeha1 has joined #openstack-nova20:58
*** betherly has quit IRC21:01
*** erlon has quit IRC21:03
*** irclogbot_2 has joined #openstack-nova21:14
*** betherly has joined #openstack-nova21:16
openstackgerritEric Fried proposed openstack/nova master: PowerVM: update_provider_tree() (compatible)  https://review.openstack.org/46856021:20
efriedjaypipes: trivial update on rebase -^21:21
*** betherly has quit IRC21:21
aspiersjaypipes, sean-k-mooney, mriedem, artom: I'm (perhaps naively) hoping that this can be the final patchset of https://review.openstack.org/#/c/609779/ ;-)21:22
artomaspiers, heh, I don't have +2 powers21:23
aspiersartom: that's OK, a +1 is welcome too ;-)21:23
artomI've been seeing all the activity and been meaning to look it over21:23
artomI'll try *real hard this time* to actually get to it ;)21:23
aspiersThanks :) No rush. Just wanted to point out my hope that I've addressed enough of the feedback that it should be stabilising now and the flurry of activity should slow right down21:24
*** betherly has joined #openstack-nova21:37
*** tbachman has joined #openstack-nova21:40
*** betherly has quit IRC21:42
*** efried is now known as pot21:43
*** pot is now known as efried21:43
efriedmriedem: https://blueprints.launchpad.net/nova/+spec/run-meta-api-per-cell lgtm, except I would like to see the actual config option section/name in that text.21:46
mriedemi don't have it yet...21:46
efriedsince it's taking the place of a spec, which would surely have that.21:46
mriedemCONF.api.local_meta_per_cell?21:47
*** spatel has quit IRC21:48
openstackgerritMerged openstack/nova master: Make CellDatabases fixture reentrant  https://review.openstack.org/61166521:51
efriedmetadata_host and metadata_port are in the [DEFAULT] section. (Not arguing anything yet, just collating data)21:51
efriedother metadata_* options too ^21:52
*** itlinux has quit IRC21:52
efriedbut metadata_cache_expiration is in [api]21:53
efriedmriedem: [api] section works; [cells] is another possibility.21:55
efriedmriedem: I would spell out 'metadata' (local_metadata_per_cell).21:56
efriedand that's about all the opinion I've got on the matter. I defer to someone who actually understands the feature.21:56
melwittfwiw, I agree to use 'metadata' instead of just meta21:57
*** betherly has joined #openstack-nova21:58
melwittI notice we have other cells related options in [api] so I think that's fine21:58
mriedemefried: we don't use [cells] anymore since that's cells v121:58
efriedokey21:58
mriedemhttps://docs.openstack.org/nova/latest/configuration/config.html#cells21:58
efriedThe [api] section says it's for the nova API21:58
mriedemDEPRECATED: Cells options allow you to use cells v1 functionality in an OpenStack deployment.21:58
mriedemmetadata api can be run from the nova api service21:58
efriedbut that may implicitly incorporate the metadata API.21:58
efriedYeah, that.21:58
melwittyeah, the all-in-one metadata API does from nova api service21:59
efriedso an aside, should we be moving [DEFAULT]metadata_* into [api]?22:00
efriedor... creating a [metadata_api] section?22:00
mriedemprobably the former, but moving 100 options every release gets tiresome22:00
mriedemboth for us and i'm sure for deployers22:00
efriedk, shrug, I've got no skin in the game.22:01
*** betherly has quit IRC22:02
*** betherly has joined #openstack-nova22:18
*** mriedem has quit IRC22:21
*** betherly has quit IRC22:23
*** mvkr has joined #openstack-nova22:35
*** eharney has quit IRC22:48
*** betherly has joined #openstack-nova22:49
*** betherly has quit IRC22:53
*** spatel has joined #openstack-nova22:54
*** spatel has quit IRC22:59
*** munimeha1 has quit IRC23:00
*** betherly has joined #openstack-nova23:09
*** betherly has quit IRC23:14
*** mlavalle has quit IRC23:35
*** brinzhang has joined #openstack-nova23:52
*** gyee has quit IRC23:57
« Wednesday, 2018-10-24 Index Friday, 2018-10-26 »

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!