15:00:16 <danpb> #startmeeting libvirt
15:00:19 <openstack> Meeting started Tue May 20 15:00:16 2014 UTC and is due to finish in 60 minutes.  The chair is danpb. Information about MeetBot at http://wiki.debian.org/MeetBot.
15:00:20 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
15:00:23 <openstack> The meeting name has been set to 'libvirt'
15:00:42 * kashyap waves hi as a  bug traiger
15:00:48 <apmelton> o/
15:00:49 <danpb> #topic Intro
15:01:00 <vladikr> o/
15:01:12 <danpb> Hi folks, welcome to the first libvirt sub-team meeting !
15:01:20 <thomasem> hey hey!
15:01:21 <dirk> hurray!
15:01:25 <s1rp> howdy!
15:01:25 <sahid> :)
15:01:33 * bauzas lurking...
15:01:34 <danpb> agenda / place for taking live notes is https://etherpad.openstack.org/p/nova-libvirt-meeting-agenda
15:02:36 <thomasem> Okay cool
15:02:42 <danpb> just so that we know each other, would folks just introduce their real name and area of interest
15:03:07 <s1rp> Rick Harris, improving LXC support in nova-libvirt
15:03:08 <danpb> I'm Daniel Berrange, @ Red Hat, upstream libvirt maintainer & interested in anything libvirt/kvm/lxc related in nova
15:03:20 <apmelton> Andrew Melton, LXC in nova-libvirt
15:03:26 <BobBall> Bob Ball @ Citrix, interested in xen
15:03:40 <dirk> Dirk Mueller @ SUSE, working with BobBall on Xen/libvirt now
15:03:42 <thomasem> I'm Thomas Maddox, @ Rackspace, interested in LibvirtLXC and nova-libvirt primarily
15:03:58 <bauzas> Sylvain Bauza @Red Hat, working on Scheduling aspects for HA, including libvirt capabilities
15:04:08 <vladikr> I'm Vladik Romanovsky @ eNovance, interested in libvirt/kvm/lxc in Nova
15:04:08 <hallyn> Serge Hallyn @ ubuntu, packaging libvirt and qemu
15:04:18 <berendt> Christian Berendt @ B1 Systems, primary interested in libvirt/xen
15:04:36 <nelsnelson> Nels Nelson @ Rackspace, security testing for libvirt-lxc applications
15:04:37 <thimble> Andre Naehring @ B1 Systems, interested in libvirt/xen
15:04:54 <ndipanov> are we doing names now?
15:05:08 <danpb> ndipanov: yep, just quickly introducing ourselves
15:05:12 <ndipanov> k\
15:05:47 <danpb> ok, seems like we have critical mass of attendees for the topics in the agenda at least
15:05:48 <ndipanov> Nikola Dipanov - Red Hat engineer working on libvirt (among other things) - also Nova core reviewer
15:05:56 <kashyap> I'm Kashyap Chamarthy, working for Red Hat, upstream tester for KVM/QEMU based virt-stack and OpenStack RDO community engineer.
15:06:10 <gcb> I'm ChangBo Guo @easystack  interested in libvirt/kvm
15:06:21 <ndipanov> s/libvirt/libvirt driver in my previous intro
15:06:26 <sahid> Sahid Orentino Ferdjaoui - Cloudwatt interested in libvirt/kvm
15:06:32 <danpb> #topic Suitability of meeting time
15:07:02 <thomasem> This time actually works great for me.
15:07:07 <dirk> +1
15:07:09 <s1rp> ++, love this time
15:07:09 <danpb> I picked this time slot to suit myself primarily :-)  is this slot practical for people ongoing
15:07:20 <BobBall> +1
15:07:21 <vladikr> +1
15:07:23 <sahid> +1
15:07:23 <berendt> +1
15:07:24 <apmelton> +1
15:07:27 <ndipanov> +1
15:07:31 <gcb> +1
15:07:37 <danpb> hehe, well that's nice to see
15:07:41 <thomasem> haha
15:07:44 <bauzas> there is a conflict with Nova scheduler meeting, so we need to discuss about a possible another stop for Gantt meetings
15:07:46 <BobBall> +2 +A?
15:07:47 <thimble> +1
15:08:08 <kashyap> Since I'm in a slightly weird tz than CET/EST/PST, /me is fine w/ 20:30 meeting once a week :-)
15:08:24 <danpb> do people think there is value in alternating timeslot every two weeks ?  eg have a Tuesday @0900  UTC timeslot
15:08:38 <danpb> this would exclude most of the  US, but make it easier for China/India/Japan
15:09:11 * danpb notes that the people who would care about the alternate time probably aren't here to answer .... :-(
15:09:14 <BobBall> I'd agree with alternate time slots - I think it works well for nova - although they go the other way with an evening slot that I can't typically attend
15:09:16 <kashyap> Maybe start w/ once a month, if there's more folks from that region possibly could alternate. But that's just me
15:09:18 <s1rp> should we put that to a vote?
15:10:20 <danpb> perhaps we should just send another message to  openstack-dev to ask if there is demand for an alternate slot
15:10:20 <ndipanov> I'm fine with both - but danpb mentioned we'd be willing to do it on the email - let\s see if someone wants it first maybe
15:10:37 <kashyap> Yep
15:10:57 <berendt> danpb: +1
15:11:33 <bauzas> danpb: I'm fine with the alternate timeslot, great to see it
15:11:48 <danpb> ok, so lets stick with this time slot for now, and check demand from mailing list before adding a 2nd option
15:12:06 <thomasem> sounds good to me
15:12:25 <danpb> moving swiftly onto the more interesting items....
15:12:34 <danpb> #topic HV driver support requirements
15:13:00 <danpb> looking at the mess we made of Xen and LXC support in Icehouse, it occurred to me that we don't have a clear idea of what nova requires from a libvirt driver
15:13:34 <danpb> so I've started trying to clarify this on this page so we can identify gaps https://wiki.openstack.org/wiki/LibvirtDriverSupportMatrix
15:13:49 <vishy> o/
15:14:18 <danpb> besides APIs, i'm looking to add list of constants used with the APIs, and list of guest configuration options
15:14:39 <danpb> if anyone wants to help out with fleshing out this page, it'd be appreciated
15:14:49 <s1rp> awesome, can we do this programmatically?
15:15:07 <danpb> s1rp: if by programmatically you mean 'grep' then yes :-)
15:15:14 <s1rp> danpb: yes exactly :)
15:15:19 <apmelton> danpb: how about flags as well. I know lxc doesn't support starting paused
15:15:30 <danpb> apmelton: yeah that's what i mean by constants actually
15:15:35 <apmelton> gotcha
15:15:41 <directxman12> apmelton: yeah, that's a important bit
15:16:00 <danpb> at least that's the main bit which hurt us in Icehouse
15:16:42 <danpb> so lets move onto Xen...
15:16:45 <BobBall> I wonder if some of https://wiki.openstack.org/wiki/HypervisorSupportMatrix should be moved into it's own libvirt page
15:17:07 <directxman12> BobBall: I was thinking that, and wondering if libvirt actually had anything similar
15:17:18 <directxman12> if not, it really should
15:17:25 <BobBall> well - 4 of the columns in that table are libvirt
15:17:25 <danpb> BobBall: hmm, i can see it both values
15:17:38 <danpb> it is valuable to have a complete picture of all drivers in one page
15:18:14 <danpb> if you just had a single Libvirt column, then you'd have to then look at a separate libvirt page to find out what your specific libvirt HV supported
15:18:17 <s1rp> the libvirt columns should be next to each other to make parity easier to check
15:18:18 <BobBall> I agree - just raising it as an option
15:18:38 <berendt> s1rp: +1
15:18:39 <danpb> s1rp: yeah, we could move the Xen column over
15:18:47 <thomasem> +1
15:19:15 <BobBall> Or have a different layout of the table - for example a libvirt column with the default being KVM then Q/X/L in the column as well if supported by the other drivers
15:20:29 <danpb> BobBall: i think we could keep separate columns for now - unless the number of libvirt drivers really grows alot
15:20:48 <danpb> lets talk Xen now though
15:20:53 <danpb> #topic Xen Driver Support
15:20:55 <BobBall> OK
15:21:09 <danpb> berendt: this was your agenda item...
15:21:28 <berendt> just noted this entry. i think there are some more interested parties
15:22:14 <berendt> we need libvirt/xen working and we would like to help building a CI environment
15:22:15 <BobBall> Well I'm definitely an interested party - but don't have an update to share this week really
15:22:34 <dirk> I'm also interested.. we're looking at setting up a CI for libvirt/xen atm
15:22:44 <dirk> probably expect some update within ~ 2 weeks
15:22:54 <danpb> from my POV, I think it is important to keep Xen working well with Libvirt driver in Nova, but I don't have resource to spend on dev/CI myself -
15:23:06 <dirk> for now it would be great to have this not removed from the code so that we can start looking at the issues
15:23:15 <danpb> i'd encourage those interested in the CI side of things to try to work together
15:23:16 <BobBall> Then we should talk berendt :) I've got a 3rd party CI for XenServer set up running in RAX and it should be quite straight forward to update that (possibly to run in a different OS cloud) to run libvirt+xen
15:23:53 <berendt> dirk: to you plan to involve other parties in the planned CI or is this SUSE cloud specific?
15:24:10 <berendt> BobBall: yes
15:24:40 <dirk> berendt: this is unrelated to SUSE Cloud
15:24:57 <dirk> berendt: it is just about setting up testing with libvirt / xen backend for upstream reviews
15:25:05 <BobBall> Using SUSE as the OS running devstack + tempest would be interesting though
15:25:14 <danpb> dirk: unless I've misjudged opinions of other core devs, we're not at imminent risk of being forced to remove Xen support from libvirt driver, but I think they'd like to see CI up & running in the near future
15:25:18 <dirk> BobBall and I met at the summit and talked a bit about it, and we're now looking into the options
15:25:28 <BobBall> particularly becasue it'd be another env that devstack doesn't currently get tested in
15:25:31 <dirk> danpb: great, we're aligned then.
15:25:44 <berendt> I would suggest to share efforts and to discuss about details in a separate meeting (dirk, BobBall)
15:25:47 <dirk> danpb: we'd also like to see the CI running so that we don't have to catch regressions downstream
15:26:00 <BobBall> Agreed berendt - I'm happy to take this offline.
15:26:18 <BobBall> well, not offline, but out of here so we can get progress / questions / etc to report here ;)
15:26:22 <danpb> berendt: agreed, sounds like RAX & SUSE folks can co-ordinate and just report interesting progress as & when needed
15:26:32 <berendt> and B1 Systems ;)
15:26:39 <danpb> opp, yes, sorry :-)
15:26:39 <BobBall> +1
15:26:46 <dirk> wfm
15:27:18 <danpb> so, besides CI testing, does anyone have other Xen related issues to raise today
15:28:42 <berendt> not at the moment, I think a working CI environment is the primary task
15:29:42 <danpb> yep, should give us a much better idea of what state we're in
15:30:23 <danpb> so, lets move onto LXC
15:30:30 <danpb> #topic  LXC Driver Support
15:30:56 <danpb> s1rp: do you want to start it off...
15:31:00 <s1rp> we should clean up the image/disk handling code :)
15:31:12 <apmelton> +1
15:31:17 <s1rp> that was mentioned quite a few times at the summit
15:31:17 <thomasem> +1
15:31:42 <danpb> assume you're specifically referring to the hacks we do with loopback and/or qemu-nbd for mounting the root filesystem
15:32:00 <s1rp> the disk-mount code is particularly hard to follow, so i was looking into using libvirts 'block' type to clean it up
15:32:08 <s1rp> danpb: yeah that's part of it
15:32:38 <thomasem> s1rp, There was some testing done by sew yesterday specifically regarding that, I think. I don't know if you caught that.
15:32:51 <thomasem> (it was a bit after hours :)
15:32:55 <danpb> yep, i added support to libvirt lxc for  type=file & type=block filesystems, specifically to let us remove this in nova
15:33:03 <thomasem> oh neat
15:33:08 <s1rp> thomasem: yeah, he mentioned that the 'block' type didn't work off the bat w/ uid-mapping enabled
15:33:22 <s1rp> thomasem: so definitely need to look into why
15:33:40 <danpb> the problem with the usernamespace uid/gid mapping stuff
15:33:42 <thomasem> Yeah, and then he mentioned that using the nova-compute method he was able to get it... so let's get more detail on that.
15:33:47 <thomasem> hmm?
15:33:50 <danpb> is that we'd have todo a massive chown across the entire filesystem
15:34:01 <vishy> danpb, s1rp is there any way we could avoid using nbd / lo at all?
15:34:03 <danpb> since presumably the user uploaded images will actually use  uid=0
15:34:20 <danpb> and we'll need to remap that to uid=NNN instead, to match whatever uid nova configures for the root user
15:34:41 <danpb> vishy: not if our filesystems are file based
15:34:46 <apmelton> danpb: unless users are aware they need to create images owned by NNN
15:35:05 <danpb> apmelton: feels kind of dirty to require the end user to know that though
15:35:10 <apmelton> yes it does
15:35:11 <thomasem> How would we communicate that? Doesn't it pick from a range?
15:35:26 * danpb wishes the kernel let you specify a uid/gid mapping for filesystem mounts
15:35:28 <apmelton> thomasem: the patch I have right now uses a static mapping
15:35:32 <thomasem> oh
15:35:35 <thomasem> okay
15:35:59 <danpb> a static mapping is sufficient to protect the kernel from the container processes
15:36:00 <hallyn> the idea of a stackable fs to do that mapping has been floated for a few years :)  noone's done it yet though
15:36:15 <apmelton> danpb: so the way image import works at rackspace now is a async task in glance, perhaps the import process for lxc images would include chowning that image for the user
15:37:06 <danpb> apmelton: yep, if we have a static mapping for all containers, ideally we'd only apply the chown once for image uploaded to glance, or at least to the base image cached by nova
15:37:17 <danpb> certainly want to try to avoid doing it for every instance start
15:37:38 <apmelton> danpb: so for the cached image by nova, that would mean each compute would need to chown at least once per image
15:38:00 <danpb> apmelton: yeah
15:38:39 <apmelton> danpb: so, I'm going to be pushing up a spec for this today, if we wanna push this discussion onto the spec, that might save time here
15:38:43 <danpb> hallyn: would it have to be a stacked FS - i can see that would be nice for dealing with bind mount approach
15:38:59 <apmelton> this discussion being user namespaces in nova
15:39:06 <danpb> hallyn: but if you're doing the first mount of a block device, feels like it'd be nice to be able to just pass a mapping straight to the kernel at first mount
15:39:29 <danpb> apmelton: yep, doing it in the spec sounds like a good idea
15:39:50 <hallyn> danpb: ouch, i'd have to think through it more.  it dpeneds on whether we could do it without separate inodes per file
15:40:00 <hallyn> (per uid-shifted mount, i mean)
15:40:09 <hallyn> but agreed, a property of a bind mount would be far nicer
15:41:03 <danpb> hallyn: probably out of scope for us to solve here :-)
15:41:27 <hallyn> but in scope to motivate here, so cool
15:42:56 <danpb> apmelton: s1rp: so  env variables / init path
15:43:17 <danpb> i agree it'd be good to be able to set those
15:43:29 <danpb> env variables i think is actually missing in libvirt itself too
15:43:44 <s1rp> yeah, think you can hack it by setting them on the init_path
15:43:47 <apmelton> env variables needs support in libvirt before we can expose it, but init path already works, it's just hard coded in nova
15:43:58 <danpb> s1rp: hmm, if that works, it is by luck
15:44:00 <s1rp> but would be nice to support via an 'env' XML tag
15:44:21 <danpb> but i'm fairly sure libvirt is directly calling  execve() so the initpath must be a real path and nothing else
15:44:25 <thomasem> You're not kidding. Hard coded.
15:44:39 <s1rp> danpb: haven't confirmed it does, just looked like it'd be possible
15:44:55 <s1rp> danpb: but either way, we should support env variables, passing args, etc
15:45:03 <danpb> the way we expose these concepts in nova is probably something we should defer to the general containers sub-team, whose meeting is in 15 mins time
15:45:10 <s1rp> so we also need a way to collect the return code
15:45:14 <danpb> since I know this is a wishlist item for Docker guys too
15:45:19 <s1rp> yeah
15:45:22 <s1rp> lots of overlap
15:45:46 <danpb> we should at least maintain a list of what we want specificially for libvirt LXC though
15:46:34 <danpb> so those 3 things can we our starting point
15:46:53 <danpb> so lets talk CI again
15:47:01 <danpb> i recall from the summit several people wanted to help with CI
15:47:18 <s1rp> RAX is interested in helping w/ LXC testing
15:47:29 <danpb> and (IIRC) jog0 said it could be part of the official CI, not a 3rd party CI
15:47:50 <apmelton> mikal: was also asking for lxc to be in the upstream gate I believe
15:48:50 <danpb> presumably though we'll need some interested people to make sure the CI stuff is actually working well enough for LXC before infra will be willing to enable it
15:48:59 <thomasem> yeah
15:49:23 <thomasem> We have one person working on those tests at the moment
15:49:35 <danpb> excellent
15:50:00 <thomasem> I'll catch up with him and see if we can get him coming to these meetings too.
15:50:07 <danpb> so i suggest we just do same as with Xen CI -  RAX and other interested parties just go away and work on it, and report back progress and/or problems as needed
15:50:09 <thomasem> Might be nice to have a regular update on that sort of progress
15:51:32 <danpb> yep, again it'd be excellent if we can get something active during Juno
15:51:40 <thomasem> I agree
15:51:41 <thomasem> =]
15:51:56 <danpb> just time for one last topic....
15:52:01 <danpb> #topic Bug Triage
15:52:31 <kashyap> Hi, currently there are 118 bugs tagged as 'libvirt'
15:52:45 <apmelton> is there a query I can use to see untriaged bugs?
15:53:02 <danpb> anyone know what other virt driver sub-teams do for bug triage ?
15:53:07 <kashyap> apmelton, All "New" Nova bugs are here -- https://bugs.launchpad.net/nova/+bugs?search=Search&field.status=New
15:53:16 <apmelton> thanks kashyap
15:53:18 <danpb> ie have they got a good practice we can/should follow
15:53:19 <kashyap> danpb, VMWare folks did it recenty
15:53:34 <kashyap> The process is outlined here -- https://wiki.openstack.org/wiki/BugTriage
15:53:55 <danpb> ah, i see
15:53:57 <directxman12> kashyap: I've been trying to help out with triaging new bugs by following the above posted wiki page
15:54:24 <kashyap> Only thing is consistently applying it - (a) Go through list of Nova bugs periodically, tag relevant bugs as 'libvirt' appropriately (b) Root-cause analysis/poke reporters for bugs w/ less info, etc.
15:54:29 <danpb> so I guess the real need here is to have a few people who actually have time to actively do bug triage ...
15:55:05 <danpb> and to draw attention to high priority issues
15:55:06 <vladikr> I'll try to help on that
15:55:07 <kashyap> I can volunteer
15:55:15 <kashyap> directxman12, Likewise here too
15:55:47 <kashyap> Also a lot of bugs I see are woefully short on info, so I wrote this page and post it as a gentle reminder where appropriate -- https://wiki.openstack.org/wiki/BugFilingRecommendations
15:56:32 <danpb> so i'd say anyone interested just go off and do it, and if you see high priority bugs that need to be addressed, add them to the agenda for the weekly meeting and/or ping  relevant people on irc
15:56:44 <directxman12> kashyap: for ones already tagging libvirt, I've been marking them as incomplete and asking for more info if they don't provide version information and logs of some sort
15:57:20 <kashyap> directxman12, Yep, I just did that discreetly for a few bugs while this meeting is going on
15:57:38 <danpb> it might be an idea to setup a wiki page describing the things we commonly need for troubleshooting libvirt related problems
15:57:44 <danpb> eg on fedora we've got this  http://fedoraproject.org/wiki/How_to_debug_Virtualization_problems
15:58:01 <danpb> there's probably something we could do that's a bit more tailored to openstack's needs
15:58:08 <kashyap> Ah, true, I was about to just post that.
15:58:24 <kashyap> Yeah, nova debug/verbose logs enabling
15:58:36 <danpb> well looks like we're out of time here
15:58:49 <danpb> thanks to everyone for coming along, and  see you all next week at same time !
15:58:55 <thomasem> cheers!
15:58:55 <directxman12> yeah, occaisonally people post pretty useless logs
15:59:01 <vladikr> thanks
15:59:03 <directxman12> au revoir
15:59:06 <s1rp> thanks all!
15:59:14 <dirk> thanks!
15:59:24 * s1rp off to the containers meeting
15:59:30 <kashyap> See ya.
15:59:42 <danpb> #endmeeting