Thursday, 2018-01-04

*** liverpooler has quit IRC00:18
*** sdague has quit IRC00:18
mriedemnailed it
mriedemjaypipes: could you have guessed the stats / num_instances / host_state.instances stuff could have all gotten screwy?00:23
jaypipesmriedem: not surprised.00:26
mriedemheh "Reported number of instances (0) does not match the tracked number of instances (3)."00:31
mriedemnot even close00:31
*** awestin1 has joined #openstack-nova00:31
*** jobewan has quit IRC00:32
openstackgerritMatt Riedemann proposed openstack/nova stable/pike: doc: Add configuration index page
*** claudiub has quit IRC00:41
*** tetsuro_ has joined #openstack-nova00:48
openstackgerritTakashi NATSUME proposed openstack/python-novaclient master: Microversion 2.59 - List/Show all server migration types
*** ihrachys has joined #openstack-nova00:54
mriedemwill need someone more familiar with ironic to triage this
openstackLaunchpad bug 1730834 in OpenStack Compute (nova) "Ironic compute node doesn't take over nodes with instance when the owner compute node is down" [Undecided,New]01:02
mriedemi don't know what "take over" means here01:02
jrollmriedem: that's expected behavior01:03
jroll'take over' meaning have another compute service manage the instance, because the hash ring thing01:03
jrollwe should make that better so it isn't expected behavior, but low priority I guess01:04
mriedemwell then01:04
* jroll triages01:05
jrolloh, I can't set importance ¯\_(ツ)_/¯01:07
mriedemjoin the bug team01:15
mriedemshould be able to then01:15
*** Swami has quit IRC01:17
jrollidk, you might make me do stuff01:17
*** purplerbot has quit IRC01:18
*** bjhuangr has joined #openstack-nova01:25
bjhuangrmriedem, hi, do you have a chance to review ? Thanks in advance .01:26
openstackgerritTetsuro Nakamura proposed openstack/nova master: [libvirt] Add _get_XXXpin_cpuset()
openstackgerritTetsuro Nakamura proposed openstack/nova master: Add NumaTopology support for libvirt/qemu driver
openstackgerritTetsuro Nakamura proposed openstack/nova master: disable cpu pinning with libvirt/qemu driver
*** Dinesh_Bhor has joined #openstack-nova01:45
*** smatzek has joined #openstack-nova01:53
*** threestrands has joined #openstack-nova01:55
*** smatzek has quit IRC01:57
*** kaisers has quit IRC01:58
*** liuyulong has joined #openstack-nova02:02
*** tetsuro_ has quit IRC02:11
*** tetsuro_ has joined #openstack-nova02:12
*** smcginnis has quit IRC02:13
*** zhurong has joined #openstack-nova02:13
mriedemkashyap: could use some help with this if you get a chance - trying to get multiattach working which used to be ok before qemu 2.10 but now we hit issues with a write lock when attaching the volume to the 2nd guest, and i thought we could pass the force flag to the attach device call to libvirt but i got this error:02:20
mriedemlibvirtError: unsupported flags (0x4) in function qemuDomainAttachDeviceLiveAndConfig02:21
Kevin_Zhengseems Nova is broken after add uuid to BDM02:25
Kevin_Zhengdue to
openstackgerritLance Bragstad proposed openstack/nova master: Simplify logic in get_enforcer
Kevin_Zhengthe field could not be added to cell1 db02:25
mriedemKevin_Zheng: that nova-manage code is really old02:27
mriedemif the bdm uuid change broke that, we should have seen it in CI02:27
mriedemsince devstack runs this02:27
mriedemthe cell1 sync happens here
mriedemafter running the schema migrations for cell002:27
Kevin_ZhengHmm, but my db didn't got updated02:28
mriedemis your nova.conf correct?02:28
Kevin_ZhengI will check, but I have been using this for sometime, it should be correct.02:30
*** ljjjustin has joined #openstack-nova02:31
*** Dinesh_Bhor has quit IRC02:32
Kevin_ZhengHmm... in config file for api service, the database connection should be cell0 db, correct?02:32
*** Dinesh_Bhor has joined #openstack-nova02:34
mriedemthat's what we have in
mriedembut devstack will sync using the cell1 conf too02:36
mriedemwhich has the cell1 database in the [database] section02:36
mriedemthis runs it for cell0
Kevin_Zhengah, I see02:37
mriedemthis was never implemented to hit all cells
mriedemi had a patch for it but it must be abandoned02:37
openstackgerritMatt Riedemann proposed openstack/nova master: Fix up formatting for deprecate-api-extensions-policies release note
Kevin_ZhengThanks, problem solved02:39
Kevin_Zhengdid we mentioned this in any docs?02:39
mriedemit should be in the install guide02:41
mriedemoh, well,
melwittmriedem: re: that multiattach thing, did you see this bug? based on that it looks like there needs to be share-rw=on property set under <shareable/> in order for it to allow the concurrent access02:42 bug 1378242 in libvirt "QEMU image file locking (libvirt)" [Unspecified,On_qa] - Assigned to pkrempa02:42
mriedemKevin_Zheng: that install guide works because it's configuring nova.conf to set the [database] to the cell1 db02:42
mriedemKevin_Zheng: that install guide was written before the superconductor mode stuff that dansmith did in devstack02:42
mriedemwhere the controllers are pointed at cell002:43
mriedemKevin_Zheng: we also have
mriedemit's not very clear, but there is a note in there too02:44
mriedem"At this point, the API database can now find the cell database, and further commands will attempt to look inside. If this is a completely fresh database (such as if you’re adding a cell, or if this is a new deployment), then you will need to run nova-manage db sync on it to initialize the schema."02:44
mriedemKevin_Zheng: might be a good FAQs entry
mriedemmelwitt: nope never seen that02:44
Kevin_ZhengI will add it later02:45
mriedemmelwitt: not mentioned in the domain xml docs at all
mriedemand by the looks of when this was 'fixed' i'm guessing we'd need super modern versions of qemu to use this02:47
*** namnh has joined #openstack-nova02:47
melwittyeah, I was just thinking the same02:47
mriedemwell, so much for multiattach in queens02:48
melwittthis all looks fairly recent, which would explain the absence in the docs02:48
mriedemwah wah02:48
mriedemcommit 860a3c4bea1d24773d8a495f213d5de3ac48a462 Author: Peter Krempa <> Date:   Wed Nov 15 15:02:58 2017 +010002:48
mriedemwe could have conditional logic around the version of qemu being used, but that sucks,02:50
mriedemand i don't know if like, qemu < 2.10 works, and then is broken until qemu >= x02:50
mriedemso you'd have a middle ground where things just don't work02:50
melwittwhat do you mean, like if this was a regression there could be a window in the middle where things don't work?02:53
mriedemwe didn't have this problem around ~newton when i wrote the original tempest test for multiattach02:53
mriedemyou could do the 2 attachments fine, it's just that nova didn't orchestrate the detach properly02:53
melwittoh :(02:53
mriedemnow we're in a case with newer qemu where the 2nd attach fails02:53
openstackgerritChen Hanxiao proposed openstack/nova master: log test: use fixtures.StandardLogging in setUp
mriedemso apparently fixed in libvirt-3.9.0-3.el702:54
mriedemwe are testing against 3.6.002:54
mriedem bug 1378242 in libvirt "QEMU image file locking (libvirt)" [Unspecified,On_qa] - Assigned to pkrempa02:55
melwittand this is the qemu bug that the libvirt bug was cloned from bug 1378241 in qemu-kvm-rhev "QEMU image file locking" [Unspecified,Verified] - Assigned to famz02:55
mriedemsame thing we're hitting02:55
melwittthat says qemu-kvm-rhev though.02:56
mriedemyeah so that caused the bug we had with 2.10 where we were hitting the same issue with qemu-img info02:57
mriedemand had to start using the force flag for that02:57
openstackLaunchpad bug 1718295 in OpenStack Compute (nova) "Live migration fails with qemu-img >= 2.10: "Failed to get shared "write" lock\nIs another process using the image?"" [High,Fix released] - Assigned to Sean Dague (sdague)02:57
mriedemso now our support matrix is likely something like,02:57
mriedem1. do it the old way if qemu<2.10, else02:57
mriedem2. do it the new way if qemu>=2.10 AND libvirt>=3.9.002:58
mriedem3. shart your pants02:58
mriedemand only 3 is testable in our current CI env :(02:58
melwittsigh, so this has some history02:58
clarkbdidnt nova implement its own lock wrapper for that?03:01
mriedemfor what?03:01
openstackgerritEli Qiao proposed openstack/nova master: Api-guide: Add Block Device Mapping
*** yikun has joined #openstack-nova03:03
clarkbmriedem: for the lack of working locks03:03
clarkbthere wa sa while wrapper script thing iirc03:03
bjhuangrmriedem: hi Matt, do you have a chance to review ? Thanks in advance03:04
mriedembjhuangr: not right now sorry03:05
mriedemclarkb: sounds like something different03:05
mriedemmelwitt: looking at it actually looks like that went into libvirt 3.1003:06
mriedemand it's unclear to me if we'd actually have to put something in the disk config xml or if libvirt just handles that for us03:06
*** Dinesh_Bhor has quit IRC03:07
melwittoh, hm. I wasn't sure about that either03:09
melwittlooking at this, it looks like libvirt handles it. but not sure03:10
mriedemyeah that's what i'm wondering03:10
mriedembased on bug 1378242 in libvirt "QEMU image file locking (libvirt)" [Unspecified,On_qa] - Assigned to pkrempa03:10
mriedemi think peter is saying, you need qemu 2.10 with the new lock stuff and the fixes mentioned in comment 10 to libvirt03:11
mriedemso if qemu>=2.10, libvirt must be >=3.10 (i think)03:11
mriedemsays it was fixed in libvirt-3.9.0-3.el7 but maybe that's an rpm package version with the patch backported?03:11
mriedembecause says 3.1003:12
melwittthat's how I interpret it too03:12
mriedemi wonder if i could swindle jamespage and the gang to backport that to libvirt 3.6.0 :)03:12
melwittyeah, I'm honestly surprised this is all so recent03:13
melwittwith all of the multiattach talk that's been happening for years, I never thought there could be a blocker at the qemu and libvirt level03:13
mriedemsweet revenge03:14
melwittand the "fixed-in-version 3.9" with the commit in 3.10 doesn't make sense to me either03:15
mriedemdid you notice the guy complaining in comment 14 is using oracle RAC?03:15
mriedemwhich is like the #1 use case for multiattach and why oracle wants it in nova so badly03:15
mriedemcomment 13 i should say03:15
mriedemstvnoyes: fyi as this is all going to impact multiattach support in nova in queens ^03:15
melwittwhat a mess03:16
mriedemstvnoyes: if you're looking for something to do, we need some testing for the multiattach patches using qemu 2.10 and libvirt 3.1.003:16
mriedemoh ha03:19
mriedemlyarwood: bug 1415250 in openstack-nova "QEMU image file locking (RHOS)" [High,Post] - Assigned to lyarwood03:19
mriedemmelwitt: ^03:19
melwittoh hello friends03:20
mriedemleft a comment because i like to comment in bz's03:21
mriedemfeels like i'm part of the team03:21
mriedemooo cc'ed to markmc@redhat.com03:21
*** ejat has quit IRC03:40
*** stvnoyes has quit IRC03:41
openstackgerritMatt Riedemann proposed openstack/nova master: [libvirt] Allow multiple volume attachments
openstackgerritMatt Riedemann proposed openstack/nova master: [api] Allow multi-attach in compute api
*** thingee has joined #openstack-nova03:43
*** thingee has left #openstack-nova03:43
openstackgerritMatt Riedemann proposed openstack/nova master: [libvirt] Allow multiple volume attachments
openstackgerritMatt Riedemann proposed openstack/nova master: [api] Allow multi-attach in compute api
*** abhishekk has joined #openstack-nova03:47
*** mriedem has quit IRC03:48
*** lbragstad has quit IRC04:11
*** s1061123 has quit IRC04:28
*** gyee has quit IRC04:29
*** armax has quit IRC04:32
*** armax has joined #openstack-nova04:33
*** armax has quit IRC04:33
*** armax has joined #openstack-nova04:33
*** armax has quit IRC04:34
*** armax has joined #openstack-nova04:34
*** armax has quit IRC04:34
*** armax has joined #openstack-nova04:35
*** armax has quit IRC04:35
*** udesale has joined #openstack-nova04:37
*** takashin has quit IRC04:38
*** takashin has joined #openstack-nova04:45
*** nicolasbock has quit IRC04:53
*** takashin has quit IRC04:53
*** s1061123 has joined #openstack-nova05:00
*** takashin has joined #openstack-nova05:00
*** takashin has quit IRC05:12
*** takashin has joined #openstack-nova05:17
openstackgerritTakashi NATSUME proposed openstack/nova master: Adds view builders for keypairs controller
openstackgerritTakashi NATSUME proposed openstack/nova master: Adds view builders for keypairs controller
*** janki has joined #openstack-nova05:43
*** markmcclain has quit IRC05:51
*** markmcclain has joined #openstack-nova05:54
*** takashin has quit IRC06:21
*** hongbin has joined #openstack-nova06:22
*** hongbin has quit IRC06:22
*** 07EAAP3JW has joined #openstack-nova06:42
*** 07EAAP3JW has quit IRC06:43
*** hongbin has joined #openstack-nova06:43
*** zhurong has quit IRC06:47
*** zhurong has joined #openstack-nova06:52
*** ratailor has joined #openstack-nova06:54
*** abhishekk has quit IRC06:57
*** jaosorior has quit IRC07:01
*** takashin has joined #openstack-nova07:01
openstackgerritZhenyu Zheng proposed openstack/nova master: Use neutron port_list when filtering instance by ip
*** Brin has joined #openstack-nova07:05
*** jaosorior has joined #openstack-nova07:11
*** jaosorior has quit IRC07:13
*** jaosorior has joined #openstack-nova07:14
*** jaosorior has quit IRC07:14
*** jaosorior has joined #openstack-nova07:14
*** abhishekk has joined #openstack-nova07:15
*** takashin has quit IRC07:18
*** takashin has joined #openstack-nova07:19
openstackgerritOpenStack Proposal Bot proposed openstack/nova master: Imported Translations from Zanata
*** sbezverk has quit IRC07:21
*** threestrands has quit IRC07:25
*** takashin has quit IRC07:25
*** takashin has joined #openstack-nova07:25
*** annp has joined #openstack-nova07:27
*** pcaruana has joined #openstack-nova07:32
*** elod has joined #openstack-nova07:32
*** bjhuangr has quit IRC07:34
takashingit log -107:42
*** jaypipes has quit IRC07:53
*** markvoelker has quit IRC07:55
*** takashin has left #openstack-nova08:00
*** ljjjustin has quit IRC08:02
*** rcernin has quit IRC08:04
openstackgerritYikun Jiang (Kero) proposed openstack/nova master: Add cross cell sort support for get_migrations
openstackgerritYikun Jiang (Kero) proposed openstack/nova master: Add pagination and Changes-since filter support for os-migrations.
*** liusheng has quit IRC08:30
*** MikeG451 has quit IRC08:47
*** dtantsur|afk is now known as dtantsur08:49
*** lucas-afk is now known as lucasagomes08:49
*** cdent has joined #openstack-nova08:50
*** hongbin has quit IRC09:03
*** tetsuro_ has quit IRC09:09
*** tetsuro_ has joined #openstack-nova09:09
*** tetsuro_ has left #openstack-nova09:09
*** derekh has joined #openstack-nova09:24
*** fragatina has quit IRC09:49
openstackgerritYikun Jiang (Kero) proposed openstack/nova master: [WIP] Fix 500 in test_resize_server_negative_invalid_state
openstackgerritIldiko Vancsa proposed openstack/nova master: Use volume shared_targets to lock during attach/detach
openstackgerritIldiko Vancsa proposed openstack/nova master: [libvirt] Allow multiple volume attachments
openstackgerritIldiko Vancsa proposed openstack/nova master: [api] Allow multi-attach in compute api
*** markvoelker has joined #openstack-nova09:56
openstackgerritStephen Finucane proposed openstack/nova master: zuul: Move legacy jobs to project
openstackgerritYikun Jiang (Kero) proposed openstack/nova master: [WIP] Fix 500 in test_resize_server_negative_invalid_state
*** Brin has quit IRC10:01
openstackgerritYikun Jiang (Kero) proposed openstack/nova master: Add index(instance_uuid, updated_at) on instance_actions table
openstackgerritIldiko Vancsa proposed openstack/nova master: [api] Allow multi-attach in compute api
*** annp has quit IRC10:12
*** toabctl has quit IRC10:16
mdboothIs today finally going to be the day when doesn't fail spuriously in the gate?10:21
mdboothThis is the fourth attempt, though, each attempt taking a full working day.10:21
mdboothIt's not looking good for the plucky patch.10:21
*** toabctl has joined #openstack-nova10:23
*** namnh has quit IRC10:27
*** markvoelker has quit IRC10:30
kashyapildikov: Hi, do you have some CI Gate log of this -- bug 1415250 in openstack-nova "QEMU image file locking (RHOS)" [High,Post] - Assigned to lyarwood10:32
kashyapildikov: So I could fish through the debug logs to find relevant bits10:32
kashyapildikov: Actually, disregard me, it's here:
ildikovkashyap: I got the logs from Matt with the following comment yeterday: "<mriedem> i can't link to these n-cpu logs because of how infra has changed where the logs live now, but that's the issue"10:34
*** abhishekk has quit IRC10:34
kashyapAh, I didn't even realize openstack-infra has changed the log location10:34
ildikovkashyap: no worries, Matt added a few extra bits to that patch yesterday10:35
*** ArchiFleKs has joined #openstack-nova10:35
kashyapYeah, just reading the review through10:35
ildikovwe haven't tested multi-attach for a while and this got us as a surprise10:35
ildikovI only fixed other easy bits in the chain hoping to get a few small dependencies out of the way10:36
openstackgerritMarcin Juszkiewicz proposed openstack/nova master: libvirt: use 'host-passthrough' as default on AArch64
hrwstephenfin: reported bug, added Closes-bug into commit message.10:36
hrwstephenfin: have to check at tests. Will take a while as I do not know nova code10:37
ildikovkashyap: thanks for looking into it10:37
kashyapildikov: No worries; Matt already did the sluething last night (I was away early yesterday).   W.r.t versioning of the 'shareable' lock10:38
ArchiFleKsHi, this may sound like a trivial question but do you have more information about how nova and neutron dns worked together, I'm using neutron as a dns server, which work fine, I can have instances with .openstacklocal domain or a custom domain, but on ths instances itself, the hostname always get sets to .novalocal anyway, is that a behavior that will change ultimatly ? If I set dhcp_domain to an10:39
ArchiFleKsempty string in nova, will it solve this ?10:39
ildikovkashyap: yeah, I gave up around midnight my time too... :/ :)10:39
kashyapMidnight...Sounds a bit too late :-)10:39
* lyarwood reads up10:40
kashyaplyarwood: No action needed; you were prompted because you were the current bug asignee10:40
lyarwoodhmmm I don't see what the qemu-img change has to do with this10:40
ildikovkashyap: it's just such a bummer as I remember chatting with Daniel Berrange back at the time about the 'shareable' flag and the support for that is in for ages and kinda thought that the virt layer is a done deal and we "only" need to figure out Cinder and Nova itself...10:41
kashyaplyarwood: It's due to the the versioning in Gate; just have to have some conditional logic in Nova10:41
kashyapildikov: BTW, some of the image locking stuff was a bit more recent (as recent as last month -- bug 1378242 in libvirt "QEMU image file locking (libvirt)" [Unspecified,On_qa] - Assigned to pkrempa10:42
kashyapBut yeah, the 'shareable' flag has been around for ages10:42
ildikovkashyap: yeah, I saw that one this morning10:42
lyarwoodkashyap: I'm not sure I agree with Matt's point in re the versions10:43
lyarwoodkashyap: doesn't suggest we need to do anything different here, libvirt and QEMU should work out when to provide the share-rw=on flag10:44
* kashyap clicks10:44
kashyaplyarwood: I think it's too late for that10:46
lyarwoodkashyap: for what?10:47
openstackgerritMarcin Juszkiewicz proposed openstack/nova master: libvirt: use 'host-passthrough' as default on AArch64
kashyapFor libvirt / QEMU to do the "right thing"10:47
kashyaplyarwood: At the moment, Nova simply needs the conditional logic, going by the errors Matt and ildikov are reporting10:47
lyarwoodI'm obviously missing something here because what you're saying makes no sense10:48
lyarwoodthe unsupported flag error is a docs bug10:48
hrwnow the worst part - reading nova/libvirt tests to find where it needs to be changed for patch10:48
kashyapDid you also look at this change:
kashyaphrw: Heya, just saw your change, note that 'host-passthrough' means, migration will completely break *unless* all the CPUs involved are all *identical*10:49
lyarwoodkashyap: that's totally unrelated to this isn't it?10:49
kashyaphrw: I'll add some comments10:49
kashyaplyarwood: No, it is related.10:49
hrwkashyap: thanks10:49
lyarwoodkashyap: that just allows us to run qemu-img info against a live instance10:50
hrwkashyap: I am aware of that issue.10:50
hrwkashyap: getting that fixed layer after layer.10:51
*** markmcclain has quit IRC10:51
kashyaplyarwood: That's true; the two aspects here are - file locking from the standalone tool, 'qemu-img', and the locking semantics when running qemu-system-x86_6410:51
kashyapGive me a few minutes before I coment more10:51
lyarwoodildikov: are we adding <shareable/> to the disk XML of multi-attached disks?10:52
hrwkashyap: Kolla deploys Nova now with cpu_mode set. But not everyone uses Kolla-ansible to deploy. Then libvirt needs to be changed probably. I hope to meet some virt-arm people face-to-face later this month and want to discuss that10:52
ildikovlyarwood: theoretically yes, proposed code change is here:
lyarwoodildikov: thanks10:53
lyarwoodildikov: hmmm shouldn't that be data.get('multiattach', False) ?10:54
lyarwoodgood old connection_info, confusing everyone yet again10:54
kashyaphrw: What does the 'Kolla' tool set it to?10:55
hrwkashyap: host-passthrough10:55
kashyaphrw: Hmm, that's wrong...10:55
* kashyap bbiab10:55
ildikovlyarwood: nope, it's not coming from Cinder, I add it here:
ildikovlyarwood: I mean it's in the volume details and I chose connection_info to pass it with the other stuff10:56
lyarwoodildikov: ah, I assumed the backend was writing this in, sorry10:56
ildikovcan add it to the data part of it, we actually wanted to remove the nested dict structure of lovely connection_info, but seemed like a too huge headache10:57
ildikovdon't get me started on it... :)10:57
ildikovlyarwood: no worries, would be nice, but it's a less direct interaction in the sense, that you can only create a multi-attach volume if the back end supports it, but then the info is stored in the volume object10:58
*** markmcclain has joined #openstack-nova10:59
lyarwoodildikov: yeah true11:06
*** Guest71543 has quit IRC11:07
*** yasemin_ has joined #openstack-nova11:09
yasemin_hi, i want to add to hyper-v compute node to ocata openstack system, i installed it, but it gives an error "Filter ImagePropertiesFilter returned 0 hosts" ? Do you have any idea? Could you help me ?11:10
openstackgerritStephen Finucane proposed openstack/nova master: console: introduce framework for RFB authentication
openstackgerritStephen Finucane proposed openstack/nova master: console: introduce the VeNCrypt RFB authentication scheme
openstackgerritStephen Finucane proposed openstack/nova master: console: Provide an RFB security proxy implementation
openstackgerritStephen Finucane proposed openstack/nova master: doc: Document TLS security setup for noVNC proxy
openstackgerritYikun Jiang (Kero) proposed openstack/nova master: test
openstackgerritYikun Jiang (Kero) proposed openstack/nova master: Add index(updated_at) on migrations table.
openstackLaunchpad bug 1673467 in OpenStack nova-compute charm "[ocata] unsupported configuration: CPU mode 'host-model' for aarch64 kvm domain on aarch64 host is not supported by hypervisor" [High,Fix released] - Assigned to James Page (james-page)11:14
hrw11:14 (228s) linaro@cb-r1-m1-c1n1:~$ uname -m;virsh domcapabilities --emulatorbin /usr/bin/qemu-system-aarch64 | grep host-model11:14
hrw    <mode name='host-model' supported='no'/>11:14
openstackgerritStephen Finucane proposed openstack/nova master: console: introduce framework for RFB authentication
openstackgerritStephen Finucane proposed openstack/nova master: console: introduce the VeNCrypt RFB authentication scheme
openstackgerritStephen Finucane proposed openstack/nova master: console: Provide an RFB security proxy implementation
openstackgerritStephen Finucane proposed openstack/nova master: doc: Document TLS security setup for noVNC proxy
openstackgerritYikun Jiang (Kero) proposed openstack/nova master: Add index(updated_at) on migrations table.
*** nicolasbock has joined #openstack-nova11:17
kashyaphrw: Thanks for the link11:22
*** markvoelker has joined #openstack-nova11:27
-openstackstatus- NOTICE: zuul seems to have gotten stuck and will probably need a restart, please be patient11:27
*** openstackstatus has quit IRC11:28
*** openstack has quit IRC11:28
*** openstack has joined #openstack-nova13:08
*** ChanServ sets mode: +o openstack13:08
*** openstackstatus has joined #openstack-nova13:10
*** ChanServ sets mode: +v openstackstatus13:10
yasemin_hi, i want to add to hyper-v compute node to ocata openstack system, i installed it, but it gives an error "Filter ImagePropertiesFilter returned 0 hosts" ? Do you have any idea? Could you help me ?13:18
*** liuyulong has quit IRC13:23
*** lucas-hungry is now known as lucasagomes13:32
alex_xucdent: the patch LGTM, just a small thing, but I won't block on it13:35
jaypipesyasemin_: please see /topic. better to ask your question on the openstack@ mailing list with the [nova][hyper-v] subject markers.13:38
hrwkashyap: custom/cortex-a53 fails :(13:39
yasemin_<jaypipes> okey thank you13:39
kashyaphrw: Oops13:40
kashyap(Okay; let's move this to OFTC, Virt channel.)13:42
cdentalex_xu: I borrowed that get schema from nova's positive_integer in parameter_types, which has the same redundancies (presumably to protect against the query string parsing library changing from underneath it)13:42
*** tbachman has joined #openstack-nova13:43
*** tbachman_ has joined #openstack-nova13:46
*** tbachman has quit IRC13:48
*** tbachman_ is now known as tbachman13:48
*** dtantsur is now known as dtantsur|brb13:49
*** efried_ has joined #openstack-nova13:54
*** efried_ is now known as efried13:56
alex_xucdent: the positive_integer is used for the request body, not just the query string14:07
efriedjaypipes: I was planning to answer mriedem's call for more details on nrp (  Unless you were already doing that?14:07
jaypipesefried: no. please go ahead.14:08
jaypipesefried: in the middle of trying to get the aggregate affinity series straightened up. should take me another few hours.14:08
efriedjaypipes: Roger.  FYI, the main thing I'd be throwing you under the bus for is accomodating nrps in GET /allocation_candidates14:09
efried(throwing under bus - was that the right analogy?  mebbe not)14:09
jaypipesefried: that's cool.14:10
*** amodi has quit IRC14:10
*** dtantsur|brb is now known as dtantsur14:22
*** dansmith has quit IRC14:25
*** smatzek has quit IRC14:25
*** sapcc-bot1 has quit IRC14:26
*** dgonzalez_5 has quit IRC14:27
*** mkoderer_ has quit IRC14:27
openstackgerritClaudiu Belu proposed openstack/nova master: hyperv: Cleans up live migration Planned VM
*** dgonzalez_3 has quit IRC14:28
*** hongbin_ has joined #openstack-nova14:29
*** mriedem has joined #openstack-nova14:29
*** hongbin_ has quit IRC14:29
*** hongbin has joined #openstack-nova14:30
openstackgerritLajos Katona proposed openstack/nova master: Deduplicate aggregate notification samples
*** jobewan has joined #openstack-nova14:33
*** esberglu has joined #openstack-nova14:35
openstackgerritMarcin Juszkiewicz proposed openstack/nova master: libvirt: use 'host-passthrough' as default on AArch64
*** efried is now known as efried_tmp14:38
*** efried has joined #openstack-nova14:38
*** efried_tmp has quit IRC14:39
*** burt has joined #openstack-nova14:39
*** lbragstad has joined #openstack-nova14:43
*** yamamoto has quit IRC14:43
-openstackstatus- NOTICE: zuul has been restarted, all queues have been reset. please recheck your patches when appropriate14:47
*** eharney has joined #openstack-nova14:54
openstackgerritMatthew Booth proposed openstack/nova master: Fix fake libvirt XML generation for disks
*** esberglu has quit IRC14:57
mriedemkashyap: lyarwood: thanks for looking at - i think we're on the same page,14:58
mriedemlibvirt 3.10 has a fix for a regression introduced in qemu 2.1014:58
kashyapmriedem: I noticed your ping too last evening14:58
kashyapI saw your sleuthing on the review14:58
mriedemso i added the version check on startup such that multiattach isn't supported unless qemu<2.10 or libvirt>=3.1014:58
mriedemwhich sucks because we don't have those versions in our CI env14:58
*** esberglu has joined #openstack-nova14:59
kashyap(Maybe we should finally get that utopic gate job that will run on newer versions to allow us to test at least)14:59
mriedemwe are using the pike UCA which is about as new as it gets for ubuntu14:59
*** gouthamr has joined #openstack-nova15:00
mriedemjamespage: coreycb: what do you think are the chances of getting these libvirt patches backported from 3.10 to 3.6 in the pike UCA? bug 1378242 in libvirt "QEMU image file locking (libvirt)" [Unspecified,On_qa] - Assigned to pkrempa15:00
kashyapYep, I recall; mriedem -- I wonder if you've ever seen this (for DevStack, though) --
mriedemalternatively, i'm going to have to probably look at using a special job which doesn't use the pike UCA so we just use the xenial package versions which would have qemu<2.1015:01
*** yamamoto has joined #openstack-nova15:01
mriedemkashyap: so i thought markus_z and tonyb already had a job that does something like this15:01
*** smatzek has joined #openstack-nova15:01
kashyapmriedem: I vaguely recall talking to them too, at Barcelona, but it didn't go anywhere15:02
* kashyap looks at the list archives15:02
*** smatzek has quit IRC15:02
mriedemi have the job name handy, sec15:02
openstackgerritMatt Riedemann proposed openstack/nova stable/newton: Raise MarkerNotFound if BuildRequestList.get_by_filters doesn't find marker
*** smatzek has joined #openstack-nova15:02
mriedemkashyap: legacy-tempest-dsvm-nova-libvirt-kvm-apr15:03
kashyapThanks; let me look up the job definition15:04
kashyapHeh, you're faster than Google15:04
jamespagemriedem: looking15:04
kashyapOh, didn't know of that thing15:05
kashyapWhy is it called 'legacy'?15:05
mriedembecause it existed before zuulv315:06
coreycbjamespage: mriedem: do those patches enable sharable disks with qemu 2.10? 2.10 is what we have for pike/queens atm.15:06
mriedemcoreycb: yes15:06
mriedemcoreycb: and yeah, qemu 2.10 is precisely the problem15:06
kashyapOkido, makes sense.15:06
mriedemkashyap: i don't know if that apr job works, but i can propose that we run it on devstack experimental queue so i can see15:07
kashyapmriedem: Yeah, I'm first looking at what's in there - git://
kashyapLast updated 01-Apr-2016.15:08
kashyapSo at least it needs updates to the file: devstack/lib/libvirt15:09
kashyap(To reflect correct versions.)15:09
hrwuf. one patch, one bug. and then 3 other bugs for the same. ouch15:10
kashyaphrw: What are the three other bugs?  It's the same one we were talking about earlier, right15:10
mriedemkashyap: oh hmm
kashyaphrw: I'm even surprised that OpenStack even works on AArch6415:10
mriedempointing at liberty still yeah...15:10
kashyapYeah, the last Git commit should gave it away15:11
mriedemkashyap: and i guess these versions for libvirt and qemu
hrwkashyap: was opened by me for my patch. Then found with some extra info. Then reported bug at upstream libvirt: and got pointed to discussion in one where again good info is provided.15:11
openstackLaunchpad bug 1741230 in OpenStack Compute (nova) "libvirt: use 'host-passthrough' as default on AArch64" [Undecided,In progress] - Assigned to Marcin Juszkiewicz (hrw)15:11
openstackLaunchpad bug 1673467 in OpenStack nova-compute charm "[ocata] unsupported configuration: CPU mode 'host-model' for aarch64 kvm domain on aarch64 host is not supported by hypervisor" [High,Fix released] - Assigned to James Page (james-page)15:11 bug 1531076 in libvirt "Support 'host-model' on aarch64" [Unspecified,New] - Assigned to libvirt-maint15:11 bug 1430987 in libvirt "No cpu model and feature in capabilities" [High,Assigned] - Assigned to abologna15:11
hrwkashyap: we use OpenStack on AArch64 since Liberty ;D15:11
kashyaphrw: Yeah, at this point, your getting traction on the upstream Bugzilla filed is the best bet.15:12
mriedemlooking at
hrwkashyap: first edition of Linaro Developer Cloud was Liberty based. and then ~200 users/projects/companies used it.15:12
mriedemubuntu doesn't have a libvirt 3.10 yet15:12
kashyapEven the UCA?15:13
hrwkashyap: when Mitaka got released we migrated as it gave us UEFI support which simplified booting A LOT.15:13
mriedemi assume Pike UCA is the latest, and that has libvirt 3.615:13
mriedemsame as bionic15:13
hrwkashyap: then we migrated to Newton.15:13
hrwkashyap: there are some testing deployments of Pike but we target Queens15:13
mriedemsame issue for qemu
mriedem2.10 is the latest in bionic15:13
hrwmriedem: there is Queens UCA15:13
mriedemand pike UCA15:13
mriedemhrw: orly15:14
hrwmriedem: but still libvirt 3.6.0 ;(15:14
kashyaphrw: Okay.  But meanwhile, my Aarch64 Mustang is remote, and I haven't accessed it in a while.  If you get sometime, can you power your machine on (the one you talked about earlier in the day) & try?15:15
hrwkashyap: I use xgene1 based HPe Moonshots15:15
hrwkashyap: remotely as we have a bunch of those at Linaro lab15:15
mriedemyup still 3.6
mriedemso that doesn't help15:15
hrwmriedem: make a PPA and grab libvirt from buster?15:16
kashyaphrw: Are there any real users using OpenStack on AArch64, besides test / devel at Linaro?15:16
hrwmriedem: that's what I did for Debian stable at Linaro15:16
hrwkashyap: yes, they are15:16
hrwkashyap: can not list names15:16
openstackgerritEric Berglund proposed openstack/nova master: WIP: PowerVM Driver: vSCSI
kashyapNo prob15:17
hrwkashyap: we provide VM instances as build slaves for several CI systems for example15:17
hrwkashyap: also provided set of VMs as porter boxes for several projects15:17
mriedemalternatively, i can push a devstack DNM patch which doesn't use the Pike UCA and we'll get qemu 2.515:17
kashyapmriedem: You mean, just to exercise the relevant code for tests?15:18
mriedemeven if jamespage / coreycb were able to backport those patches from 3.10 to 3.6 in the pike UCA, we'd have to put a workaround option in nova to bypass the version check and just run the multiattach code15:18
hrwkashyap: the plan is to migrate all our cloud setups from our Newton to clean Queens. from set of venvs to containers15:18
mriedemi.e. 'i've patched my packages, so don't care about the versions you think are required'15:19
hrwmriedem: can not libvirt report that multiattach feature in domcapabilities or sth?15:19
kashyapYeah, I saw your version check code in that patch series15:20
hrwmriedem: so instead "if libvirt >= x.y" you can use "if libvirt.capabilities.contains('multiattach')"15:20
mriedemhrw: maybe?
hrwversion check suxx when features are not enabled on all archs15:20
mriedemis that how this would work with ^ and "disk-share-rw"?15:20
hrwkashyap: so, can I get +2 from you on patch? ;D15:21
kashyaphrw: I can't +215:21
kashyaphrw: But I'd like someone else's opinion, too, like mriedem15:22
kashyapOr mdrabe15:22
hrwkashyap: or w8... I was supposed to do something with tests... no idea how to do that part15:22
kashyapErr, mdbooth I mean15:22
mriedemkashyap: can you tell me if is used to expose a hypervisor capability?15:22
mdboothkashyap: ?15:23
kashyaphrw: I have one more comment on the change15:23
kashyapAlmost about to hit send15:23
mdboothWhich change?15:23
kashyapmriedem: 1 sec, let me look15:23
mriedemlike in
kashyapmdbooth: It's arch-specific, but migration-related, too.  I noted the concern there15:23
mdboothkashyap: I don't have a useful opinion on that unfortunately without doing my own doc diving.15:24
*** smatzek has quit IRC15:24
kashyapmdbooth: Okido, I have enough context there.  Disregard15:24
hrwmdbooth: imho is best part of info on subject15:25 bug 1430987 in libvirt "No cpu model and feature in capabilities" [High,Assigned] - Assigned to abologna15:25
kashyapmriedem: Still checking15:25
hrwkashyap: thx15:25
*** smatzek has joined #openstack-nova15:26
*** smatzek_ has joined #openstack-nova15:27
*** smatzek_ has quit IRC15:28
*** smatzek_ has joined #openstack-nova15:28
kashyapmriedem: Hmm, I just built the newest libvirt-python bindings, and don't see it any of the capabilities; let me ask one of the libvirt folks15:29
kashyapmriedem: So no -- it isn't exposed via any capabilities currently (like `virsh (dom)capablities`)15:30
kashyapmriedem: So Peter (who wrote that commit says):15:30
kashyap16:29 < pkrempa> kashyap: it is not exposed currently, since it's supposed to be transparent for the users15:31
kashyap16:30 < pkrempa> and if it's not transparent I'd suggest to complain to qemu15:31
*** smatzek has quit IRC15:31
mriedemkashyap: ok so we're stuck with version checks15:31
mriedemthanks for asking15:31
mriedemwe could, as noted, add a workaround config option to bypass the version checks if you've patched your packages15:31
kashyapmriedem: But that'd be asking the user to be too awake, and alert and aware :P15:32
kashyapMore seriously, yeah - workaround config sounds good15:33
mriedemit wouldn't be the end user, it'd be the deployer15:33
mriedembut yes15:33
mriedemdepends on how much people want their multiattach15:33
kashyapYou mean in the [workarounds] section, right.  Like that live snapshots thing (that now we removed, IIRC)15:33
*** smatzek_ has quit IRC15:33
openstackgerritMarcin Juszkiewicz proposed openstack/nova master: libvirt: use 'host-passthrough' as default on AArch64
mriedema "this isn't tested upstream so it's not supported thing, but it's a backdoor if you need it"15:34
kashyapHeh, yeah15:34
mdboothmriedem: lyarwood I also commented on the multi attach patch. There must be a better place to stash multiattach than connection_info. That's essentially adding pain to something we're effectively deprecating.15:35
kashyaphrw: So, on the above patch, thanks for addressing that.  Besides that, I don't have anything else15:35
hrwkashyap: thanks for help15:35
coreycbmriedem: will get back to you hopefully shortly. i'm checking with our libvirt maintainer.15:35
mdboothIncidentally, I think a 'can multiattach' flag on the BDM object would be appropriate.15:36
ildikovmdbooth: what are we deprecating?15:36
*** smatzek has joined #openstack-nova15:36
mdboothildikov: We're trying to get rid of the requirement for Nova to know anything about connection_info15:36
kashyaphrw: Oh, hang on -- same update needs to be done in the rel note.15:37
mdboothIdeally we can just fetch it from cinder whenever we need it.15:37
kashyaphrw: As that's more public facing; sorry, should've caught it earlier15:37
mriedemmdbooth: with 2 weeks to feature freeze, i think that's a future improvement15:37
hrwkashyap: will do15:37
mriedemmdbooth: because i don't want to drag this out for queens with a schema and object migration and all that15:37
mdboothildikov: We definitely don't want to be stashing more stuff in a foreign opaque dict, and then relying on it.15:37
mriedemmdbooth: hell, we could just add a "multiattach" boolean flag to driver.attach_volume15:37
mdboothmriedem: Yeah, I wondered about that, but I think it needs to be persistent.15:38
mriedemwe get the multiattach value from the volume15:38
mriedemi don't really want to persist cinder state in nova's db15:38
mdboothmriedem: Admittedly I didn't stare at it for a long time, but it's used in volume_driver.get_config()15:38
*** smatzek has quit IRC15:38
mriedemmdbooth: yeah i think that's called from driver.attach_volume15:38
mdboothI'm pretty sure we can call that outside of the context of attach()15:38
mriedemso we can pass a boolean arg down15:38
hrwkashyap: I just copy comment to release notes basically in next patch15:39
mriedemmdbooth: hmm, for live migrate maybe15:39
kashyaphrw: Yep15:39
mdboothmriedem: However, if that's not true (I didn't check), that would be great15:39
ildikovmdbooth: mriedem: I'm open to suggestions it's just the first idea I had two years ago...15:39
mdboothAnyway, just my 2c, and I totally get that pragmatism might be required here.15:40
mdboothBut it raised a flag for me. We'll need to unwind it eventually.15:40
hrwalso commit message got rewritten15:41
openstackgerritMarcin Juszkiewicz proposed openstack/nova master: libvirt: use 'host-passthrough' as default on AArch64
mriedemyeah i think passing a boolean through attach_volume to get_config is easy, it's live migrate that i'm worried about15:41
ildikovmdbooth: not the first time it came up, but we didn't manage to have this as a top priority problem as of yet to find a better way :(15:41
mdboothildikov: At the very least we'll need to be able to find it in order to unwind it.15:42
* mdbooth wonders where the best place to put a warning about it would be.15:42
kashyapmriedem: Oh the previous point about capablities, libvirt upstream says, if I file a bug they could add it - as it shouldn't be too difficult15:43
kashyapmriedem: That'd be cleaner for us (Nova), isn't it?15:43
mdboothildikov: Incidentally, does service_uuid indicate a shared pool of volumes?15:43
mdboothildikov: e.g. multiple volumes on the same NFS mount?15:44
ildikovmdbooth: no, that's independent15:44
mdboothildikov: Where can I read what it means?15:44
mriedemkashyap: long-term that would be cleaner yes,15:44
mriedembut not something that is going to help me in queens15:45
kashyapRight.  /me imagines: If you add the version check, and then the capability comes along later, no one will remember to swap that, until prompted by something15:45
mriedemkashyap: feel free to file a bug if you want :)15:46
kashyapildikov: Can you point to the latest URL of the multi-attach specification, please?15:46
openstackgerritMatthew Booth proposed openstack/nova master: Pass DriverBlockDevice to driver.attach_volume
openstackgerritMatthew Booth proposed openstack/nova master: Use real block_device_info data in libvirt tests
openstackgerritMatthew Booth proposed openstack/nova master: Fix libvirt volume tests passing invalid disk_info
openstackgerritMatthew Booth proposed openstack/nova master: Pass disk_info dict to libvirt_info
openstackgerritMatthew Booth proposed openstack/nova master: Expose volume host type and path independent of libvirt config
openstackgerritMatthew Booth proposed openstack/nova master: Don't generate fake disk_info in swap_volume
openstackgerritMatthew Booth proposed openstack/nova master: Local disk serial numbers for the libvirt driver
openstackgerritMatthew Booth proposed openstack/nova master: Remove redundant swap_volume tests
*** eharney has quit IRC15:46
kashyapmriedem: Will do15:46
openstackgerritAndreas Karis proposed openstack/nova master: Add debug output for selected page size
coreycbmriedem: this is being tracked in
openstackLaunchpad bug 1716028 in libvirt (Ubuntu) "qemu 2.10 locks images with no feature flag" [Medium,Triaged]15:48
coreycbmriedem: cpaelzer says that after bionic is done he'll take a look at a potential SRU, although he currently has the SRU to artful(pike) as a low priority.15:48
mriedemcoreycb: ok thanks15:49
kashyapIt sucks hard that one cannot access the URLs ( without Gerrit account15:51
*** smatzek has joined #openstack-nova15:51
kashyapCan't pass in-progress patch URLs to people aren't Gerrit users.  Surely there must be a way15:51
kashyapWithout bothering people to ask to make an account (much like mailing lists).15:51
mriedemkashyap: i'm not signed in and i can view
mriedemyou just can't comment or vote15:52
kashyapmriedem: Err, sorry.  The person was complaining about typing in a comment.15:52
kashyapmriedem: Anyway, the quick point that Peter wanted to add was: " sharing disk image is possible even with current qemu/libvirt if the image is 'raw' and <shareable/> is used"15:54
mriedemwe must be using qcow2 images15:55
*** smatzek has quit IRC15:55
*** armax has joined #openstack-nova15:56
mriedemanother thing i can try15:56
ildikovmdbooth: the service_uuid field was added here:
*** smatzek has joined #openstack-nova15:57
mdboothildikov: Just found it. I think it's orthogonal to multi-attach, tbh.15:57
mriedemkashyap: although it's a bit confusing,15:57
mriedemwe dump the disk config before trying to attach the device15:57
mriedemand it says type="raw"15:58
mdboothildikov: Still a good idea. I could also remove my NFS locking stuff in Nova if we had that, although my NFS locking is finer grained but significantly more complex.15:58
*** imacdonn has quit IRC15:58
ildikovmdbooth: it was added along with a shared_targets field so we can use a lock in case the target exported by the back end is shared among volumes/attachments15:59
mdboothildikov: I get it. It's a good idea, I just don't see the relationship to multi-attach.15:59
kashyapmriedem: (Aside - we both wrote almost same comment 4 mins apart)15:59
ildikovmdbooth: it is supposed to help to solve our detach problems15:59
kashyapmriedem: Looking at your paste15:59
mdboothWe hit this with or without multi-attach.16:00
ildikovmdbooth: as if the target is shared and gets removed with the first attachment then the remaining attachments are screwed16:00
mdboothildikov: Right, but you can can do that without multi-attach.16:00
*** smatzek_ has joined #openstack-nova16:00
ildikovmdbooth: and the Nova patches using it are dependencies to multi-attach as it might be a problem otherwise as well16:00
mdboothMulti-attach doesn't even make the problem particularly worse.16:00
*** hemna_ has joined #openstack-nova16:01
*** udesale has joined #openstack-nova16:01
*** jafeha has quit IRC16:01
ildikovmdbooth: well, I got it in referring to multi-attach, but I guess the path doesn't matter once you got where you wanted... :)16:01
*** smatzek has quit IRC16:01
efriedjaypipes mriedem cdent alex_xu Draft: -- anything missing/incorrect/silly?16:01
* kashyap bbiab; need a few to bike home.16:02
cdentefried: will look in a mo, thanks for doing that16:02
mdboothildikov: Hehe, I hear you :)16:02
mriedemmdbooth: is it just me or is libvirt.images_type, use_cow_images and force_raw_images set of options totally confusing?16:04
stephenfinalex_xu: Done (
*** smatzek_ has quit IRC16:05
*** smatzek has joined #openstack-nova16:06
*** smatzek has quit IRC16:07
*** smatzek has joined #openstack-nova16:08
*** tbachman has quit IRC16:09
*** udesale has quit IRC16:14
*** edmondsw has joined #openstack-nova16:14
*** smatzek has quit IRC16:15
*** smatzek_ has joined #openstack-nova16:15
*** smatzek_ has quit IRC16:16
efriedmriedem Log processing & coloring is handled in the openstack-infra/os-loganalyze project (but I think you knew that).  Is there something in particular you're having trouble finding in there?16:18
openstackgerritStephen Finucane proposed openstack/nova master: conf: Deprecate 'network_manager'
mriedemefried: what in infra actually calls os-loganalyze to format the logs16:19
openstackgerritStephen Finucane proposed openstack/nova master: conf: Use new-style choice values
efriedmriedem Ah - it's an apache plugin thingy.  The files aren't actually modified - they get twiddled on the fly when you do your http request.16:20
efriedmriedem So you just need to muck with the filters to make sure they're being run on the files you're interested in.16:21
efriedmriedem If you have access to a log server, you can play by fiddling with the os-loganalyze source in place, restarting the apache server, and then reloading your browser.16:22
*** smatzek has joined #openstack-nova16:22
*** eharney has joined #openstack-nova16:24
*** nicolasbock has quit IRC16:24
*** nicolasbock has joined #openstack-nova16:25
*** smatzek has quit IRC16:27
mriedemi don't16:27
*** smatzek has joined #openstack-nova16:30
*** smatzek_ has joined #openstack-nova16:34
*** smatzek has quit IRC16:35
*** dtantsur is now known as dtantsur|afk16:41
cdentefried: information seems accurate at the detail level, but feels like it needs some kind of executive summary or something, a kind of "here's what we're trying to accomplish" which is then followed by the "and this is how it is being done" (which is what you've already got)16:44
efriedcdent Okay.  I felt like it was already getting kinda long, but... tough :)16:45
*** esberglu has quit IRC16:46
cdentI think length is useful in this case because there's been not enough in the way of spec, so this kind of stands in for that16:46
*** esberglu has joined #openstack-nova16:48
*** cdent has quit IRC16:49
*** gcb has quit IRC16:49
*** ChanServ sets mode: -r 16:49
*** syjulian has joined #openstack-nova16:50
*** MikeG451 has joined #openstack-nova16:50
*** gcb has joined #openstack-nova16:50
clarkbmriedem: the test framework for it runs without apache16:50
*** tbachman has joined #openstack-nova16:52
*** pcaruana has quit IRC16:52
openstackgerritEric Berglund proposed openstack/nova master: WIP: PowerVM Driver: vSCSI
*** cdent has joined #openstack-nova16:52
*** smatzek_ has quit IRC16:53
hrwkashyap: ^^16:53
kashyaphrw: On a call, will read :-)16:54
hrwkashyap: thx ;)16:54
kashyapDamn, I already clicked16:54
*** sdague has joined #openstack-nova16:56
*** smatzek_ has joined #openstack-nova16:56
*** danpawlik has quit IRC16:56
*** danpawlik has joined #openstack-nova16:57
*** sheel has joined #openstack-nova16:57
*** jafeha has joined #openstack-nova16:57
hrwkashyap: :D16:58
kashyaphrw: It looks fine; there's an extra tab in there16:58
kashyap"It" == the review16:59
rybridgesHey guys, can anyone confirm for sure whether or not ephemeral GB is part of a VMs snapshot?16:59
hrwkashyap: can you mark it in comment? I am unable to find it locally17:00
kashyaphrw: Done.17:00
kashyapIt's extra tab17:00
kashyapIn the rel note file17:00
*** smatzek_ has quit IRC17:00
hrw4 spaces to be exact ;d17:01
hrwthat's why I did not ofund17:01
openstackgerritMarcin Juszkiewicz proposed openstack/nova master: libvirt: use 'host-passthrough' as default on AArch64
* hrw off17:01
efriedcdent (jaypipes) Howzat:
*** esberglu has quit IRC17:03
*** esberglu has joined #openstack-nova17:03
*** gyee has joined #openstack-nova17:17
*** smatzek has joined #openstack-nova17:18
*** eharney has quit IRC17:20
mnaseris stable/pike ci broken?17:21
*** smatzek has quit IRC17:21
*** smatzek has joined #openstack-nova17:21
*** eharney has joined #openstack-nova17:22
mnaserok, i guess i'll apply the patch locally till it lands in stable/pike (
edleafemriedem: so for the migration bug: would the fix be to just log that there were no orig_alloc found, and not raise the exception?17:24
mriedemedleafe: well, there are really 2 fixes,17:24
edleafeOr should I also check if the migration is the orig_alloc (as in retries)17:24
mriedem1. If using the CachingScheduler, there won't be allocations and we need to just log something and ignore it17:25
mnaseralso if any stable cores for nova are around, this is pretty useful -
mriedem2. If we're rescheduling (we should know this via filter_properties 'retry' entry), then we need to modify how we swap allocations to only change the allocation for the instance record, and leave the migration allocation on the source node untouched17:25
mriedemedleafe: so i'm thinking 2 separate patches17:26
mriedemedleafe: i could probably wip up a simple regression test for the caching scheduler one and fix in the same patch, which could go below yours17:27
mriedemsince resize is just busted with caching scheduler regardless of reschedule17:27
*** moshele has joined #openstack-nova17:27
edleafemriedem: on #2, not following. The source allocs will be the migration, and the dest allocs won't be done because we haven't picked a target host yet17:27
mriedemedleafe: on #2 the problem is when we reschedule right?17:28
edleafeit's checking that the instance is allocated against the source17:28
mriedemat the point of the first reschedule, the source node allocs are on the migration record and the failed first chosen host allocs are on the instance17:28
*** yikun has quit IRC17:28
mriedemright, i'm saying,17:28
edleafeon a retry, the allocs on the source will be the migration17:28
mriedemwe need to modify the logic that assumes the source node allocs are on the instance,17:28
mriedemif we know we're doing a reschedule,17:28
*** yikun has joined #openstack-nova17:29
mriedemand we can determine that based on the filter_properties 'retry' entry17:29
mriedemso on a retry, we don't do anything with the source,17:29
mriedemwe just update the dest17:29
mriedemupdate the instance allocation from failed dest 1 to next dest 217:29
edleafemriedem: what did you mean then by "only change the allocation for the instance record"?17:29
mriedemon a reschedule, we move the instance allocation from the failed dest host to the next alternative host17:30
mriedemand don't touch the migration allocation on the source host17:30
edleafethe instance won't be allocated after a fail. Those should be rolled back, no?17:30
edleafeno, the next host isn't selected until after this part of the code is run17:30
edleafeThis is happening in _preallocate_migration17:31
mriedemwhere do we rollback allocations for the instance on a fail?17:32
edleafeWhen we eventually select a host (either through select_destinations (now) or alternate host (when that last patch merges), that's when we allocate the instance to the target17:32
edleafemriedem: in compute. Let me look17:32
mriedemdelete_allocation_for_failed_resize in prep_resize?17:33
mriedemoh i see, the call to _revert_allocation17:33
edleafedamn, you're faster than I am :)17:34
mriedemso if the compute reverts the allocations before casting back to conductor to reschedule,17:35
mriedemwhy is conductor failing on the reschedule17:35
edleafeBecause it is confirming that the instance has allocations on the source17:35
edleafeBut at that point on a retry, it's the migration that has allocations on the source17:35
mriedemyeah, which should be the case if we got here
mriedemnot if we did ^17:36
*** mchiappero has quit IRC17:36
*** moshele has quit IRC17:36
mriedemis that failing?17:36
edleafedunno - all I know is that the instance didn't have allocations on the source. Let me check that out.17:37
*** mchiappero has joined #openstack-nova17:39
mriedemi've created to deal with the caching scheduler + resize stuff17:43
openstackLaunchpad bug 1741307 in OpenStack Compute (nova) "Resize always fails when using the CachingScheduler" [High,Triaged] - Assigned to Matt Riedemann (mriedem)17:43
mriedemkashyap: fyi that i updated the devstack patch to test multiattach to set CONF.libvirt.images_type=raw and CONF.use_cow_images=False to see if that makes a difference17:45
*** lucasagomes is now known as lucas-afk17:46
openstackgerritMerged openstack/nova master: Revert "Modify _poll_shelved_instances periodic task call _shelve_offload_instance()"
*** armax has quit IRC17:50
*** armax has joined #openstack-nova17:50
*** damien_r has joined #openstack-nova17:50
*** Apoorva has joined #openstack-nova17:59
*** david-lyle has quit IRC18:00
*** david-lyle has joined #openstack-nova18:01
*** derekh has quit IRC18:03
mriedemwow, if i'm reading his results correctly, Kevin_Zheng got a 90% improvement in listing instances with details using an ip filter after we proxy to neutron first18:19
mriedemsdague: ^18:19
mriedem2000 instances in nova and 2K ports in neutron18:19
mriedemtold him to send the details to the dev list18:21
openstackgerritMarcin Juszkiewicz proposed openstack/nova master: libvirt: use 'host-passthrough' as default on AArch64
openstackgerritMatt Riedemann proposed openstack/nova master: Add regression test for resizing failing when using CachingScheduler
mriedemedleafe: ^ functional regression test for resize + caching scheduler failure18:27
edleafemriedem: thx18:29
*** yamamoto has quit IRC18:29
cdentefried: sorry for delay, better ✔18:29
efriedThanks cdent.  jaypipes, you want to look before I send it?
*** yamamoto has joined #openstack-nova18:33
efriedgoing, going, gone.18:36
*** yamamoto has quit IRC18:37
cdentshut your filthy mouth efried18:47
cdent(re notifications)18:47
efriedHehehehe, thought you'd like that.18:47
*** efried is now known as efried_nomnom18:48
*** stvnoyes has joined #openstack-nova18:53
openstackgerritMerged openstack/nova master: Add support for getting volume details with a specified microversion
*** sheel has quit IRC19:06
*** tbachman has quit IRC19:14
openstackgerritMatt Riedemann proposed openstack/nova master: Handle no allocations during migrate
*** yamamoto has joined #openstack-nova19:35
openstackgerritMerged openstack/nova master: DriverBlockDevice: make subclasses inherit _proxy_as_attr
openstackgerritMerged openstack/nova master: Expose BDM uuid to drivers
*** smatzek has quit IRC19:39
*** armax has quit IRC19:40
mriedemmdbooth: i was looking at where _get_volume_config is used and it's used in quite a few places in the libvirt driver, like live migration, swap volume, and generally anywhere we build a guest xml, so also for resize/cold migrate,19:40
mriedemand in a lot of those places, we don't have a volume to know if it's multiattach or not to set the conf.shareable flag,19:40
mriedemso despite it not being great, stashing the multiattach value in the connection_info which gets saved in the bdm is about the best we can do right now19:41
mriedembecause all of these places in the driver that call _get_volume_config are using the connection_info off the bdm19:41
*** yamamoto has quit IRC19:42
*** tbachman has joined #openstack-nova19:49
sdaguemriedem: nice19:53
sdaguemriedem: I'd totally believe it, we do extra dumb stuff in there19:53
mriedemsubstring match is a bit slower, as expected, but still much better than doing it in nova19:54
sdaguemriedem: yep19:54
mriedem4000ms w/o the patch, 400ms with the patch and no substring, about 900ms with the patch and substring match19:54
*** eharney has quit IRC20:01
*** efried_nomnom is now known as efried20:03
mriedemcdent: edleafe: it's your favorite game show, "IS IT 400 OR 409?!"20:03
* cdent puts hand over button20:03
mriedemso, say i'm trying to attach a multiattach volume to an instance and the compute is too old to support multiattach,20:04
mriedemis that a 400, or a 409? knowing that at some point the compute might be upgraded to support multiattach.20:04
cdentthe state of the compute is out of sync with the desired state, but you made a valid request (had it been in the right state), so I'd go with 40920:06
mriedemartom: remember how we punted for tagged attached on a shelved offloaded instance because we couldn't tell if the host supported it?20:06
mriedemcdent: ok that's what i was thinking too20:06
cdent \o/20:06
mriedemartom: but we don't make that same distinction in the API for tagged bdms with boot from volume20:07
mriedemcan you remember why?20:07
mriedemartom: my guess was "you could reschedule and attempt to hit a host that does support device tagging" but we raise BuildAbortException so we don't even reschedule in that case20:08
*** markmcclain has quit IRC20:10
*** markmcclain has joined #openstack-nova20:11
openstackgerritmelanie witt proposed openstack/nova master: Add access_url_base to console_auth_tokens table
openstackgerritmelanie witt proposed openstack/nova master: Optionalize instance_uuid in console_auth_token_get_valid()
*** nicolasbock has quit IRC20:13
openstackgerritmelanie witt proposed openstack/nova master: Add ConsoleAuthToken object
openstackgerritmelanie witt proposed openstack/nova master: Add periodic task to clean expired console tokens
openstackgerritmelanie witt proposed openstack/nova master: Use ConsoleAuthToken object to generate authorizations
openstackgerritmelanie witt proposed openstack/nova master: Convert websocketproxy to use db for token validation
* edleafe reads back20:15
edleafeyeah, the request isn't malformed; the system can't support it20:15
artommriedem, yeah, I think your guess is correct20:16
cdentsupport it _right now_20:16
mriedemartom: ok, so that's not really any different from punting when trying to attach a volume to a shelved offloaded instance with tags20:16
mriedemartom: i'm only asking b/c we're going to have a similar situation with multiattach volumes20:16
mriedemtrying to decide if we should support attaching a multiattach volume to a shelved offloaded instance - same with bfv, either way it might fail once we get to the compute and you're dead20:17
mriedemat some point in the future, which no one will probably ever work on, we could do a scheduler filter for this stuff to make it pick a host which supports the thing you need20:18
artomWith the full 100% knowledge of every bit of context about multiattach, I'd vote to be consistent and refuse it for shelved offloaded20:18
mgagneif running mitaka, is legacy v2 API still used or is it dead code?20:18
artomBecause we have no idea what host it'll end up on20:18
artomWhen it's unshelved20:18
mriedemartom: the same is true for boot from volume20:18
artommriedem, eh, how so?20:19
mriedembut we allow tagged bdms with bfv20:19
mriedemwhen you create a server, or unshelve a server, they both go through the scheduler to pick a host,20:19
mriedemyou have an equal chance in either scenario of picking a host that doesn't support the capability20:19
mriedemso the fact we allow tagged bdms with bfv but not shelved offloaded instances is inconsistent20:20
artommriedem, how is bfv different from normal volume tagging? IIRC the api just checked service level, to see if the cloud was fully upgraded to support it20:20
openstackgerritLee Yarwood proposed openstack/nova master: libvirt: Attach and detach encryptors during swap_volume
mriedemartom: ok yeah that's the one difference then, is that we check in the api when creating a server if the computes are new enough to handle tagged bdms20:21
mriedemwe could easily have done the same for shelved offloaded attach20:21
artommriedem, so there's this 2 x 2 matrix of stuff20:22
artomwe have tagged boot / tagged attach20:22
mriedemthat's basically what i was saying we'd do in those two cases for multiattach too - check the min compute service version and fail if computes aren't upgraded yet20:22
artomAnd compute manager supports it / virt driver supports it20:22
artomFor tagged boot, we used the service level check to see if compute manager supports it20:22
artomFor tagged boot/virt driver... I forget how we handled it20:23
mriedemwe just fail in the compute20:23
mriedemand raise BuildAbortException20:23
artomAh, right, hopefully rescheduling to a virt driver that supports it20:23
mriedemno, we don't reschedule20:23
mriedemBuildAbortException means abort, don't reschedule20:24
artomAh, ok20:24
artomYeah, that's not awesome, but no other way of doing it I guess20:24
mriedemBuildRescheduledException means kick edleafe in the head a few times20:24
artomAnd then for tagged attach...20:24
artomI think they were all RPC calls down to the compute20:25
artomSo virt driver support was straightforward20:25
artomExcept for shelved offloaded, which was a cast20:25
*** chyka has joined #openstack-nova20:25
mriedemyeah that one is easy20:25
mriedemunless the server is shelved offloaded20:25
artomSo that one we decided to just fail in the API20:25
mriedemin the case of shelved offloaded, we don't do anything with the compute b/c there is no compute20:25
artomInstead of having the unshelve fail like, 3 years later20:25
mriedemyeah, i'm saying, i think at a minimum we could do the same for shelved offloaded as we do for tagged boot which is check the service version20:26
mriedemwe don't reschedule on an unshelve failure, but we don't reschedule on a tagged boot failure either20:26
mriedemso it's basically the same20:26
*** cdent has quit IRC20:26
artomOne is definitely more immediate than the other20:26
artomThough I guess we if we relay the exception properly, it's not a massive deal20:27
edleafemriedem: so it seems that the bug I found was the result of a race20:27
edleafeThe call to retry on runs the retry before the 'finally:' clause reight below it gets hit.20:27
*** jackie-truong has joined #openstack-nova20:27
edleafeIf I add the call the self._revert_allocation() before L4087, the test I wrote passes.20:27
artomLike, if nova show unshelved-instance has 'you unshelved on a virt that doesn't support tagging' somewhere in there (but more cloudy) it can work20:27
mriedemedleafe: ah just like the thing we hit in the other patch20:28
mriedemneed to cleanup the allocation before casting to conductor20:28
mriedemartom: we'd record a fault20:28
mriedemthat's about it20:28
edleafemriedem: do you have a fix for the caching scheduler? Or should I do that?20:29
artommriedem, is that obvious to the user?20:29
mdboothmriedem: My patch series removes it from swap_volume, btw, but not surprised by the others.20:29
mriedemartom: not really, the fault's traceback is only available to admins20:29
mriedemmdbooth: swap_volume is the one case where i don't think this matters because we don't rely on the conf.shareable attribute20:29
artommriedem, so give this lack of feedback to the user, it's probably better to fail fast in the API, no?20:29
mriedemartom: maybe20:30
mriedemi can see the argument for being consistent20:30
mdboothmriedem: Ok, so how can we flag this as a second connection_info wart?20:30
mriedemmdbooth: 2nd as in device_path is #1?20:30
artommriedem, what's the issue we see with multiattach though?20:30
mdboothI mean there's this and also device_path20:31
edleafemriedem: ah, I just saw the test patch20:31
mdboothmriedem: Right, yeah. What you said :)20:31
artomI haven't followed closely, so it may or may not be worth it to dump all the context on me for whatever my opinion is worth ;)20:31
* mdbooth just got in from a bike ride...20:31
mriedemartom: nothing - i'm just weighing options since we don't have the api plumbed in for all of the multiattach stuff20:31
mriedemso i'm thinking through how to handle this,20:31
mriedembecause the backend capabilty checking is the same as with tagged attach20:31
mriedemand i need to get out of my head sometimes20:32
mriedemmdbooth: i'm not sure how to flag this20:32
artomAre we talking about attaching to a shelved offloaded instance? And we don't know whether the eventual compute would support it?20:32
mriedemartom: correct20:32
mdboothmriedem: I don't think there's a good central focal point for connection_info cruft. Which is another problem, tbh.20:33
jackie-truongsdague: The nova-queens-blueprint-status etherpad mentioned that you needed Johns Hopkins to sync up with you on the certificate validation feature20:33
artommriedem, so exactly the same problem? 1. check that compute manager is new enough 2. check that virt driver supports it20:34
artomOr is multi-attach virt-agnostic?20:34
jackie-truongsdague: We created an etherpad ( to walk through usage and testing. Let me know if you need more information or have any questions.20:35
mriedemartom: it's the same problem20:35
mriedemin queens, assuming we ship this code, only the libvirt driver will support multiattach20:36
artommriedem, one thing we talked about was scheduling with compute driver capabilities taken into account, probably through placement/resource providers20:37
artomI don't think that work is ready yet, though20:37
artomThen again, by the time multiattach lands, there might be talks of replacing placement with a new quantum-powered scheduler20:37
mriedemsure, the ambiguity goes away if we had the CapabilitiesFilter aware of this and handling it20:39
mriedem"request says it wants multiattach, find a host that supports multiattach"20:39
mriedemanywho, i'll work on what i know we need to support for now, and get testing going, and then bikeshed on the rest20:40
openstackgerritJay Pipes proposed openstack/nova master: allow compute nodes to be associated with host agg
openstackgerritJay Pipes proposed openstack/nova master: Remove server group sched filter support caching
openstackgerritJay Pipes proposed openstack/nova master: WIP Support aggregate affinity filters
openstackgerritJay Pipes proposed openstack/nova master: get instance group's aggregate associations
*** READ10 has joined #openstack-nova20:43
mgagneanswering myself, legacy v2 code is still used in mitaka:
*** takashin has joined #openstack-nova20:48
*** takashin has left #openstack-nova20:49
*** smatzek has joined #openstack-nova20:50
openstackgerritMatt Riedemann proposed openstack/nova master: Use volume shared_targets to lock during attach/detach
openstackgerritMatt Riedemann proposed openstack/nova master: [libvirt] Allow multiple volume attachments
openstackgerritMatt Riedemann proposed openstack/nova master: WIP: [api] Allow multi-attach in compute api
openstackgerritMatt Riedemann proposed openstack/nova master: WIP: Pass multiattach flag to reserve_block_device_name
mriedemildikov: here is the reserve_block_device_name change ^20:52
ildikovmriedem: cool, thanks!20:52
*** takashin has joined #openstack-nova20:52
*** edmondsw has quit IRC20:54
*** eharney has joined #openstack-nova20:56
mriedemnova meeting in 1 minute20:58
*** lyan has joined #openstack-nova21:00
*** lyan has quit IRC21:02
*** smatzek has quit IRC21:05
*** smatzek has joined #openstack-nova21:06
*** lyan has joined #openstack-nova21:06
*** smatzek has quit IRC21:10
*** fragatina has joined #openstack-nova21:11
*** edmondsw has joined #openstack-nova21:19
openstackgerritMerged openstack/os-traits master: Add NIC Switchdev feature
*** smcginnis has joined #openstack-nova21:27
*** fragatina has quit IRC21:27
*** moshele has joined #openstack-nova21:33
*** threestrands has joined #openstack-nova21:35
*** ab2434_ has joined #openstack-nova21:36
jaypipesab2434_: ok, so a couple questions for you...21:39
jaypipesab2434_: question #1: if a VM consumes a VF, why should the VF's PF information be made available to it?21:39
ab2434_its being used for Active & Available Inventory21:40
jaypipesab2434_: and?21:40
ab2434_mainly thats it21:41
jaypipesab2434_: what purpose does having the PF's PCI address serve?21:41
ab2434_mainly for inventory21:42
ab2434_in this case21:42
*** sbezverk has joined #openstack-nova21:43
*** chyka has quit IRC21:43
*** chyka has joined #openstack-nova21:44
jaypipesab2434_: I still don't see what purpose having the PF's PCI address serves. I mean, I can kind of see an inventory management system taking inventory of all hardware on compute hosts, but that's not what the Neutron port binding profile is for. Why doesn't AAI just, you know... inventory the systems itself?21:45
ab2434_one other thing is SDN-F application can configure switch port interfaces based on the mapping21:45
jaypipesab2434_: that particular VNF should have the *PF* assigned to the VM, then, so that it can inventory the VFs on the PF itself (and control them accordingly). no?21:46
ab2434_true. but there is a corresponding swith port that needs to be configured21:47
jaypipesab2434_: and you need the PF's PCI address in order to figure out which switch port the PF is associated with?21:47
jaypipesab2434_: why is the switch port or tag decorating the Neutron port binding?21:48
jaypipeswhy *isn't*...21:48
ab2434_i dont think thats what's happening today21:48
ab2434_as a side note there could be multiple switches21:49
jaypipesab2434_: but why should Nova twist and turn to satisfy the needs of one particular VNF? :)21:49
jaypipesab2434_: especially when said VNF isn't *really* a VM but instead just hardware masquerading as software ;)21:50
jaypipesbut I digress...21:50
ab2434_well. it boils down to the mapping, having nova provide a way to map the ports21:50
ab2434_for the sdn-f to configure the correct switch /port for the vm21:50
mriedemdo vif tags not work here?21:52
jaypipesab2434_: my point is this: if the Neutron port binding can be decorated with the switch group or port tag, then that information can be passed down to Nova (in the instance PCI request) and used to identify the physical function that should be selected for the VM. what is being proposed here is the opposite of that design. the proposal here is to essentially inventory the topology and hardware for the entire deployment ahead of time and21:52
jaypipespre-schedule/place VMs that consume specific PCI devices on specific hardware all at once.21:52
jaypipesit's the opposite of cloud... the opposite of on-demand.21:52
*** esberglu has quit IRC21:54
jaypipesab2434_: the world that we (Nova/OpenStack, whatever) is trying to get to is a world where the VNFs/applications *describe to Nova the resources and traits that it needs* and Nova goes and finds an appropriate place for that workload to land and devices to consume. What the world that NFV is trying to force on Nova is the opposite of that: a world where a VNF says to Nova "hey, put me on node X and PF Y and use VF foo. Oh, and also pin me to21:55
jaypipesCPUs 1-12, 18-24 and NUMA cell 0."21:55
ab2434_yes i see your point21:56
jaypipesab2434_: it's this incongruence of worldviews that is at the root of the issue I think.21:56
jaypipesab2434_: and yes, I understand I work for Verizon (used to be AT&T) and that certain groups at Verizon think that OpenStack should just get on with the business of being an NFVI and nothing more ;)21:56
*** smatzek has joined #openstack-nova21:57
tonybkashyap, mriedem: we had the beginings of one but it bitrotted.  We could look at reviving it after the PTG.21:58
mriedemtonyb: i don't think that job is useful anymore now that we're using the pike UCA by default in devstack21:59
tonybmriedem: Yeah the UCA stuff was always s'posed to be a POC the real plan was to have a tandem repo where we built tagged snapshots and use that22:00
*** moshele has quit IRC22:02
tonybmriedem: At times there is a reasonable Fedora$current image, if ianw_pto has that working we can use the std. virt repo for Fedora also (again that was part of the plan)22:03
tonybmriedem, kashyap: I'm really happy to help revive that work QA, Neuton and infra all want somethign like that I just can't really be the driver22:03
*** jose-phillips has quit IRC22:05
jaypipesefried: still around?22:05
*** esberglu has joined #openstack-nova22:06
jaypipesefried: so... this will fail a functional test: and I'm not entirely sure why. perhaps if you're around later you could pull that patch and have a looksie? it looks like the instance creation ain't actually working.22:06
mriedemmdbooth: so, long-term we should probably store the multiattach value on the bdm record...22:07
mriedemthe more i think about it22:07
mriedemjust like a tag22:07
jaypipesefried: in any case, meh, will hit it later and tomorrow but if you have any time, could use your eyeballs.22:07
jaypipesefried: it will fail the assertion here: on line 35122:07
mriedemmdbooth: and that always tells us, the volume representing this bdm was attached and supported multiattach at that time, so treat it like that until it's detached and the bdm is deleted22:07
*** jackie-truong has quit IRC22:08
*** jose-phillips has joined #openstack-nova22:08
*** markmcclain has quit IRC22:10
*** markmcclain has joined #openstack-nova22:11
*** rcernin has joined #openstack-nova22:12
*** smatzek has quit IRC22:15
*** markmcclain has quit IRC22:19
*** markmcclain has joined #openstack-nova22:20
*** smatzek has joined #openstack-nova22:21
efriedjaypipes Sorry, I'm back now.  Catching up...22:22
*** jose-phillips has quit IRC22:22
*** smatzek has quit IRC22:22
*** smatzek has joined #openstack-nova22:23
efriedjaypipes wtf is an instance group?22:23
efriedjaypipes And are these host aggregates (as opposed to placement aggregates)?22:25
hemna_mriedem, hey man, I added an update to bug
openstackLaunchpad bug 1452641 in OpenStack Compute (nova) "Static Ceph mon IP addresses in connection_info can prevent VM startup" [Medium,Confirmed]22:26
hemna_mriedem, thought you might want to take a look and see what you thought.22:26
*** smatzek has quit IRC22:27
*** jistr has quit IRC22:29
*** jose-phillips has joined #openstack-nova22:30
mriedemyikes, super rbd specific code in there22:30
mriedemhemna_: at the ptg in denver we said we'd just always refresh the connection_info when we needed it
hemna_yah, that was just a customer's hacked patch to get it to work22:32
lyarwoodthat's imagebackend rbd btw22:32
lyarwoodnot connection_info volume rbd22:32
mriedemsure. the forced refresh_connection_info=True thing would arguably be much simpler, if it actually works22:32
hemna_my customer tried to live migrate and it didn't get the updated info, hence his hack22:32
lyarwoodhemna_: vms/3b97914e-3f9b-410a-b3d9-6c1a83244136_disk isn't a volume however right?22:33
jaypipesefried: and instance group is an abomination otherwise called a server group. never mind, though... I'm gonna sleep on it and tackle manana22:33
hemna_I'm not sure, this is a dump direct from the customer's env.22:34
mriedemhemna_: there is a refresh_conn_info kwarg to _get_instance_block_device_info22:34
efriedjaypipes Okay.  I'll play with it a bit if I get some time here.22:34
mriedemso the idea from the ptg was just always pass refresh_conn_info=True there when we were going to do something like this22:34
hemna_mriedem, is that an option at nova cmdln time ?22:34
*** ihrachys has quit IRC22:34
mriedemit was a way to do this w/o any api changes22:35
*** ihrachys has joined #openstack-nova22:35
lyarwoodmriedem: the issue isn't with the rbd volume, but the imagebackend rbd images22:35
mriedemi don't know what that means22:35
mriedemi thought the issue was stale rbd information in the connection_info, which nova gets from cinder22:35
lyarwoodmriedem: volumes/volume-6d04520d-0029-499c-af81-516a7ba37a54 is the volume22:35
mriedemand uses to populate the disk config22:35
lyarwoodmriedem: not for ephemerial rbd images, it's all hard coded from the local nova.conf iirc22:36
mriedemlyarwood: yeah i'm not talking about ephemeral,22:36
mriedemjust volumes22:36
lyarwoodmriedem: yeah that appears to be updated in the LM flow22:37
lyarwoodmriedem:  <source protocol='rbd' name='volumes/volume-6d04520d-0029-499c-af81-516a7ba37a54'> <-- this one is changed, new ips22:37
mriedemright, because the live migration flow calls _get_volume_config22:37
mriedemusing the bdm.connection_info22:37
mriedemwhich is stale22:37
mriedemwhat we said in denver,22:38
mriedemwas when we start live migration, and get the bdms, we call cinder to refresh the connection info22:38
mriedemby pass refresh_conn_info=True to
*** Apoorva has quit IRC22:38
mriedemthat will make a new os-initialize_connection call to cinder and return the latest connection_info22:38
* lyarwood nods22:39
mriedemthen we save that on the bdm.connection_info22:39
*** Apoorva has joined #openstack-nova22:39
mriedemfor reasons, 'ive just never written that patch,22:39
mriedembut i also have concerns about the new attach flow making that no longer work22:39
mriedemwith the new flow, nova calls cinder to get the attachment record which has the connection_info stored in the cinder db,22:39
mriedemso i don't think it actually creates a new connection (export?) so we wouldn't refresh22:39
hemna_which could be stale...22:40
hemna_in this case we just moved the stale info from nova to cinder, and cinder is stale?22:40
mriedemhemna_: right, if cinder doesn't update that, and just returns whta's in the db, it would be stale - and we'd have the same problem as using the stale nova db info22:40
*** burt has quit IRC22:41
mriedemyeah idk, nova could still call os-initialize_connection if we wanted to,22:42
mriedemi don't know if that would update the attachment record in cinder or not22:42
hemna_so I thought the purpose of storing the conn info in cinder's db is for force delete time when nova doesn't know anything22:43
mriedemotherwise we'd likely need like a refresh=True query parameter to GET /attachments/{id}22:43
mriedemhemna_: that's part of it yeah22:43
hemna_in which case, can't cinder always refresh that if nova asks for it?22:43
hemna_I dunno22:43
mriedembut how does cinder know that we're asking for a refresh?22:43
mriedemor just a simple read-only get22:43
mriedemthis is where we got talking about API changes to force a refresh22:44
*** nicolasbock has joined #openstack-nova22:44
hemna_I guess this could affect other cinder backends too22:44
hemna_not just ceph22:44
hemna_multipath IPs can change in between attaching the same volume, if the cinder backend gets changed22:46
hemna_ok so for now, I'll just tell my customer to bounce their VMs.  :(22:46
mriedemlike a retype?22:46
*** armax has joined #openstack-nova22:47
hemna_well, more like a storage array loses one of it's interfaces22:47
mriedemor you mean the backend backend22:47
hemna_and the IP changes22:47
hemna_or new interfaces are added22:47
hemna_it's kinda the same thing as a new ceph monitor IP22:47
lyarwoodhemna_: FWIW restarting the instance is the only way to update the mon IPs for the ephemerial rbd images.22:48
openstackgerritTakashi NATSUME proposed openstack/nova master: [placement] Add sending global request ID in put (1)
hemna_lyarwood, yah.  they were trying to avoid that, as there could be tons of VMs to bounce22:48
mriedemhmm, actually it looks like for live migration we do refresh the connection_info
mriedemin pre-live migration on the dest host22:49
lyarwoodright, the issue hemna_ reported in the bug isn't with rbd volumes :)22:49
mriedemnow i get it22:50
*** jose-phillips has quit IRC22:51
mriedemwell, i'm reminded once again that i'd like to just give up and buy a farm in austria and just live out my days there22:51
mriedembecause this whole software business isn't worth it22:51
hemna_you aren't the only one.....I've been in a youtube black whole, watching dudes build log cabins...22:52
mriedemthat's the manliest thing i've heard all day22:52
*** jose-phillips has joined #openstack-nova22:53
*** damien_r has quit IRC22:54
*** lyan has quit IRC22:56
*** edmondsw has quit IRC22:56
*** edmondsw has joined #openstack-nova22:57
hemna_lyarwood, should I file a separate bug then to cover the rbd image update during LM ?22:58
lyarwoodhemna_: yeah I think so, it's a different codepath etc23:00
hemna_ok will do23:00
lyarwoodhemna_: cool thanks, I'll take a look tomorrow23:00
*** edmondsw has quit IRC23:01
*** smatzek has joined #openstack-nova23:04
*** mvk has joined #openstack-nova23:05
openstackgerritEric Fried proposed openstack/nova master: Track associated sharing RPs in report client
openstackgerritEric Fried proposed openstack/nova master: Raise on API errors getting aggregates/traits
openstackgerritEric Fried proposed openstack/nova master: ProviderTree.populate_from_iterable
openstackgerritEric Fried proposed openstack/nova master: Track tree-associated providers in report client
openstackgerritEric Fried proposed openstack/nova master: WIP: Scheduler[Report]Client.get_provider_tree
openstackgerritEric Fried proposed openstack/nova master: WIP: ComputeDriver.update_provider_tree()
openstackgerritEric Fried proposed openstack/nova master: WIP: Use update_provider_tree from resource tracker
openstackgerritEric Fried proposed openstack/nova master: Fix nits in update_provider_tree series
efriedjaypipes ^ -- and response to review comments on the way...23:07
openstackLaunchpad bug 1741364 in OpenStack Compute (nova) "ceph ephemeral info not updated during live migrate" [Undecided,New]23:08
*** dave-mccowan has joined #openstack-nova23:08
efriedjaypipes See response on -- not sure if I've answered the right question there.23:11
*** ab2434_ has quit IRC23:14
*** flwang has quit IRC23:14
*** stvnoyes has quit IRC23:18
*** hongbin has quit IRC23:18
*** smatzek has quit IRC23:19
*** smatzek has joined #openstack-nova23:19
*** jistr has joined #openstack-nova23:19
*** smatzek has quit IRC23:21
*** smatzek has joined #openstack-nova23:21
*** flwang has joined #openstack-nova23:24
mnaserdoes nova not set the instance status to ERROR if cinder volume detach fails?23:26
*** smatzek has quit IRC23:26
mnaserI have some cinder volumes which show attached servers, but the server is deleted23:26
mriedemmnaser: correct23:31
mriedemnormal volume detach? or volume detach during instance delete?23:31
mnasermriedem: volume detach during instance delete23:31
*** dave-mccowan has quit IRC23:31
openstackgerritMatt Riedemann proposed openstack/nova master: [libvirt] Allow multiple volume attachments
openstackgerritMatt Riedemann proposed openstack/nova master: Pass multiattach flag to reserve_block_device_name
openstackgerritMatt Riedemann proposed openstack/nova master: WIP: [api] Allow multi-attach in compute api
mriedemmnaser: yeah that happens here
mnaser(boot from volume instance)23:32
mriedemif it fails we log something but keep going23:32
mriedembecause we've already destroyed the guest on the hypervisor23:32
mriedemthis is where we try to delete the volume if delete_on_termination=True
mnaserbecause now i end up with volumes with attachments (for whatever reason) and there doesn't seem to be a straight forward way of clearing that out23:34
mriedemforce detach in cinder?23:34
mnaseri see force delete but not force detach23:35
mnaser(force delete is ok for me though)23:35
mriedemthat might be what i was thinking of23:36
mriedemsmcginnis: knows23:36
openstackgerritAndreas Karis proposed openstack/nova master: Add debug output for selected page size
mnaserDelete for volume 973d5a3d-07f3-4d05-b62c-8dfe746d298c failed: Invalid volume: Volume  must not be migrating, attached, belong to a group, have snapshots or be disassociated from snapshots after volume transfer. (HTTP 400) (Request-ID: req-ecfad96c-8125-47e4-8e25-9a9e892139aa)23:36
mnaseri guess this is in cinder-land23:37
mnaser(cant force delete an attach volume)23:37
mordredwow. y'all are having all the fun23:38
*** Eran_Kuris has quit IRC23:38
mriedemmordred: this channel is a barrel of laughs all day every day23:38
mordredmriedem: that's what I tell people23:38
mordred"looking for a barrel of laughs? go check out #openstack-nova!", I tell them23:39
*** mtreinish has quit IRC23:39
mnasermriedem: interesting, i could probably even normal detach, but i wonder if the cinder cli client has it23:40
mnaserhopefully scaling up our cinder api endpoints should mean that we don't see this again23:41
mriedemi don't see a force detach command in cinderclient23:41
*** mtreinish has joined #openstack-nova23:42
SamYaplei dont think there is one...23:46
SamYapleinconsitencies between nova and cinder ive always had to go to the db to solve23:46
mriedemthere is an api23:47
mriedemso curl should work23:47
mriedemshould also be easy enough to add a cli for this in cinderclient23:47
*** Eran_Kuris has joined #openstack-nova23:51
*** jistr has quit IRC23:54
*** lyan has joined #openstack-nova23:54
*** tetsuro has joined #openstack-nova23:54
*** jistr has joined #openstack-nova23:55
*** liangy has joined #openstack-nova23:56
*** lyan has quit IRC23:58
*** nicolasbock has quit IRC23:58

Generated by 2.15.3 by Marius Gedminas - find it at!