15:00:16 #startmeeting libvirt 15:00:19 Meeting started Tue May 20 15:00:16 2014 UTC and is due to finish in 60 minutes. The chair is danpb. Information about MeetBot at http://wiki.debian.org/MeetBot. 15:00:20 Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 15:00:23 The meeting name has been set to 'libvirt' 15:00:42 * kashyap waves hi as a bug traiger 15:00:48 o/ 15:00:49 #topic Intro 15:01:00 o/ 15:01:12 Hi folks, welcome to the first libvirt sub-team meeting ! 15:01:20 hey hey! 15:01:21 hurray! 15:01:25 howdy! 15:01:25 :) 15:01:33 * bauzas lurking... 15:01:34 agenda / place for taking live notes is https://etherpad.openstack.org/p/nova-libvirt-meeting-agenda 15:02:36 Okay cool 15:02:42 just so that we know each other, would folks just introduce their real name and area of interest 15:03:07 Rick Harris, improving LXC support in nova-libvirt 15:03:08 I'm Daniel Berrange, @ Red Hat, upstream libvirt maintainer & interested in anything libvirt/kvm/lxc related in nova 15:03:20 Andrew Melton, LXC in nova-libvirt 15:03:26 Bob Ball @ Citrix, interested in xen 15:03:40 Dirk Mueller @ SUSE, working with BobBall on Xen/libvirt now 15:03:42 I'm Thomas Maddox, @ Rackspace, interested in LibvirtLXC and nova-libvirt primarily 15:03:58 Sylvain Bauza @Red Hat, working on Scheduling aspects for HA, including libvirt capabilities 15:04:08 I'm Vladik Romanovsky @ eNovance, interested in libvirt/kvm/lxc in Nova 15:04:08 Serge Hallyn @ ubuntu, packaging libvirt and qemu 15:04:18 Christian Berendt @ B1 Systems, primary interested in libvirt/xen 15:04:36 Nels Nelson @ Rackspace, security testing for libvirt-lxc applications 15:04:37 Andre Naehring @ B1 Systems, interested in libvirt/xen 15:04:54 are we doing names now? 15:05:08 ndipanov: yep, just quickly introducing ourselves 15:05:12 k\ 15:05:47 ok, seems like we have critical mass of attendees for the topics in the agenda at least 15:05:48 Nikola Dipanov - Red Hat engineer working on libvirt (among other things) - also Nova core reviewer 15:05:56 I'm Kashyap Chamarthy, working for Red Hat, upstream tester for KVM/QEMU based virt-stack and OpenStack RDO community engineer. 15:06:10 I'm ChangBo Guo @easystack interested in libvirt/kvm 15:06:21 s/libvirt/libvirt driver in my previous intro 15:06:26 Sahid Orentino Ferdjaoui - Cloudwatt interested in libvirt/kvm 15:06:32 #topic Suitability of meeting time 15:07:02 This time actually works great for me. 15:07:07 +1 15:07:09 ++, love this time 15:07:09 I picked this time slot to suit myself primarily :-) is this slot practical for people ongoing 15:07:20 +1 15:07:21 +1 15:07:23 +1 15:07:23 +1 15:07:24 +1 15:07:27 +1 15:07:31 +1 15:07:37 hehe, well that's nice to see 15:07:41 haha 15:07:44 there is a conflict with Nova scheduler meeting, so we need to discuss about a possible another stop for Gantt meetings 15:07:46 +2 +A? 15:07:47 +1 15:08:08 Since I'm in a slightly weird tz than CET/EST/PST, /me is fine w/ 20:30 meeting once a week :-) 15:08:24 do people think there is value in alternating timeslot every two weeks ? eg have a Tuesday @0900 UTC timeslot 15:08:38 this would exclude most of the US, but make it easier for China/India/Japan 15:09:11 * danpb notes that the people who would care about the alternate time probably aren't here to answer .... :-( 15:09:14 I'd agree with alternate time slots - I think it works well for nova - although they go the other way with an evening slot that I can't typically attend 15:09:16 Maybe start w/ once a month, if there's more folks from that region possibly could alternate. But that's just me 15:09:18 should we put that to a vote? 15:10:20 perhaps we should just send another message to openstack-dev to ask if there is demand for an alternate slot 15:10:20 I'm fine with both - but danpb mentioned we'd be willing to do it on the email - let\s see if someone wants it first maybe 15:10:37 Yep 15:10:57 danpb: +1 15:11:33 danpb: I'm fine with the alternate timeslot, great to see it 15:11:48 ok, so lets stick with this time slot for now, and check demand from mailing list before adding a 2nd option 15:12:06 sounds good to me 15:12:25 moving swiftly onto the more interesting items.... 15:12:34 #topic HV driver support requirements 15:13:00 looking at the mess we made of Xen and LXC support in Icehouse, it occurred to me that we don't have a clear idea of what nova requires from a libvirt driver 15:13:34 so I've started trying to clarify this on this page so we can identify gaps https://wiki.openstack.org/wiki/LibvirtDriverSupportMatrix 15:13:49 o/ 15:14:18 besides APIs, i'm looking to add list of constants used with the APIs, and list of guest configuration options 15:14:39 if anyone wants to help out with fleshing out this page, it'd be appreciated 15:14:49 awesome, can we do this programmatically? 15:15:07 s1rp: if by programmatically you mean 'grep' then yes :-) 15:15:14 danpb: yes exactly :) 15:15:19 danpb: how about flags as well. I know lxc doesn't support starting paused 15:15:30 apmelton: yeah that's what i mean by constants actually 15:15:35 gotcha 15:15:41 apmelton: yeah, that's a important bit 15:16:00 at least that's the main bit which hurt us in Icehouse 15:16:42 so lets move onto Xen... 15:16:45 I wonder if some of https://wiki.openstack.org/wiki/HypervisorSupportMatrix should be moved into it's own libvirt page 15:17:07 BobBall: I was thinking that, and wondering if libvirt actually had anything similar 15:17:18 if not, it really should 15:17:25 well - 4 of the columns in that table are libvirt 15:17:25 BobBall: hmm, i can see it both values 15:17:38 it is valuable to have a complete picture of all drivers in one page 15:18:14 if you just had a single Libvirt column, then you'd have to then look at a separate libvirt page to find out what your specific libvirt HV supported 15:18:17 the libvirt columns should be next to each other to make parity easier to check 15:18:18 I agree - just raising it as an option 15:18:38 s1rp: +1 15:18:39 s1rp: yeah, we could move the Xen column over 15:18:47 +1 15:19:15 Or have a different layout of the table - for example a libvirt column with the default being KVM then Q/X/L in the column as well if supported by the other drivers 15:20:29 BobBall: i think we could keep separate columns for now - unless the number of libvirt drivers really grows alot 15:20:48 lets talk Xen now though 15:20:53 #topic Xen Driver Support 15:20:55 OK 15:21:09 berendt: this was your agenda item... 15:21:28 just noted this entry. i think there are some more interested parties 15:22:14 we need libvirt/xen working and we would like to help building a CI environment 15:22:15 Well I'm definitely an interested party - but don't have an update to share this week really 15:22:34 I'm also interested.. we're looking at setting up a CI for libvirt/xen atm 15:22:44 probably expect some update within ~ 2 weeks 15:22:54 from my POV, I think it is important to keep Xen working well with Libvirt driver in Nova, but I don't have resource to spend on dev/CI myself - 15:23:06 for now it would be great to have this not removed from the code so that we can start looking at the issues 15:23:15 i'd encourage those interested in the CI side of things to try to work together 15:23:16 Then we should talk berendt :) I've got a 3rd party CI for XenServer set up running in RAX and it should be quite straight forward to update that (possibly to run in a different OS cloud) to run libvirt+xen 15:23:53 dirk: to you plan to involve other parties in the planned CI or is this SUSE cloud specific? 15:24:10 BobBall: yes 15:24:40 berendt: this is unrelated to SUSE Cloud 15:24:57 berendt: it is just about setting up testing with libvirt / xen backend for upstream reviews 15:25:05 Using SUSE as the OS running devstack + tempest would be interesting though 15:25:14 dirk: unless I've misjudged opinions of other core devs, we're not at imminent risk of being forced to remove Xen support from libvirt driver, but I think they'd like to see CI up & running in the near future 15:25:18 BobBall and I met at the summit and talked a bit about it, and we're now looking into the options 15:25:28 particularly becasue it'd be another env that devstack doesn't currently get tested in 15:25:31 danpb: great, we're aligned then. 15:25:44 I would suggest to share efforts and to discuss about details in a separate meeting (dirk, BobBall) 15:25:47 danpb: we'd also like to see the CI running so that we don't have to catch regressions downstream 15:26:00 Agreed berendt - I'm happy to take this offline. 15:26:18 well, not offline, but out of here so we can get progress / questions / etc to report here ;) 15:26:22 berendt: agreed, sounds like RAX & SUSE folks can co-ordinate and just report interesting progress as & when needed 15:26:32 and B1 Systems ;) 15:26:39 opp, yes, sorry :-) 15:26:39 +1 15:26:46 wfm 15:27:18 so, besides CI testing, does anyone have other Xen related issues to raise today 15:28:42 not at the moment, I think a working CI environment is the primary task 15:29:42 yep, should give us a much better idea of what state we're in 15:30:23 so, lets move onto LXC 15:30:30 #topic LXC Driver Support 15:30:56 s1rp: do you want to start it off... 15:31:00 we should clean up the image/disk handling code :) 15:31:12 +1 15:31:17 that was mentioned quite a few times at the summit 15:31:17 +1 15:31:42 assume you're specifically referring to the hacks we do with loopback and/or qemu-nbd for mounting the root filesystem 15:32:00 the disk-mount code is particularly hard to follow, so i was looking into using libvirts 'block' type to clean it up 15:32:08 danpb: yeah that's part of it 15:32:38 s1rp, There was some testing done by sew yesterday specifically regarding that, I think. I don't know if you caught that. 15:32:51 (it was a bit after hours :) 15:32:55 yep, i added support to libvirt lxc for type=file & type=block filesystems, specifically to let us remove this in nova 15:33:03 oh neat 15:33:08 thomasem: yeah, he mentioned that the 'block' type didn't work off the bat w/ uid-mapping enabled 15:33:22 thomasem: so definitely need to look into why 15:33:40 the problem with the usernamespace uid/gid mapping stuff 15:33:42 Yeah, and then he mentioned that using the nova-compute method he was able to get it... so let's get more detail on that. 15:33:47 hmm? 15:33:50 is that we'd have todo a massive chown across the entire filesystem 15:34:01 danpb, s1rp is there any way we could avoid using nbd / lo at all? 15:34:03 since presumably the user uploaded images will actually use uid=0 15:34:20 and we'll need to remap that to uid=NNN instead, to match whatever uid nova configures for the root user 15:34:41 vishy: not if our filesystems are file based 15:34:46 danpb: unless users are aware they need to create images owned by NNN 15:35:05 apmelton: feels kind of dirty to require the end user to know that though 15:35:10 yes it does 15:35:11 How would we communicate that? Doesn't it pick from a range? 15:35:26 * danpb wishes the kernel let you specify a uid/gid mapping for filesystem mounts 15:35:28 thomasem: the patch I have right now uses a static mapping 15:35:32 oh 15:35:35 okay 15:35:59 a static mapping is sufficient to protect the kernel from the container processes 15:36:00 the idea of a stackable fs to do that mapping has been floated for a few years :) noone's done it yet though 15:36:15 danpb: so the way image import works at rackspace now is a async task in glance, perhaps the import process for lxc images would include chowning that image for the user 15:37:06 apmelton: yep, if we have a static mapping for all containers, ideally we'd only apply the chown once for image uploaded to glance, or at least to the base image cached by nova 15:37:17 certainly want to try to avoid doing it for every instance start 15:37:38 danpb: so for the cached image by nova, that would mean each compute would need to chown at least once per image 15:38:00 apmelton: yeah 15:38:39 danpb: so, I'm going to be pushing up a spec for this today, if we wanna push this discussion onto the spec, that might save time here 15:38:43 hallyn: would it have to be a stacked FS - i can see that would be nice for dealing with bind mount approach 15:38:59 this discussion being user namespaces in nova 15:39:06 hallyn: but if you're doing the first mount of a block device, feels like it'd be nice to be able to just pass a mapping straight to the kernel at first mount 15:39:29 apmelton: yep, doing it in the spec sounds like a good idea 15:39:50 danpb: ouch, i'd have to think through it more. it dpeneds on whether we could do it without separate inodes per file 15:40:00 (per uid-shifted mount, i mean) 15:40:09 but agreed, a property of a bind mount would be far nicer 15:41:03 hallyn: probably out of scope for us to solve here :-) 15:41:27 but in scope to motivate here, so cool 15:42:56 apmelton: s1rp: so env variables / init path 15:43:17 i agree it'd be good to be able to set those 15:43:29 env variables i think is actually missing in libvirt itself too 15:43:44 yeah, think you can hack it by setting them on the init_path 15:43:47 env variables needs support in libvirt before we can expose it, but init path already works, it's just hard coded in nova 15:43:58 s1rp: hmm, if that works, it is by luck 15:44:00 but would be nice to support via an 'env' XML tag 15:44:21 but i'm fairly sure libvirt is directly calling execve() so the initpath must be a real path and nothing else 15:44:25 You're not kidding. Hard coded. 15:44:39 danpb: haven't confirmed it does, just looked like it'd be possible 15:44:55 danpb: but either way, we should support env variables, passing args, etc 15:45:03 the way we expose these concepts in nova is probably something we should defer to the general containers sub-team, whose meeting is in 15 mins time 15:45:10 so we also need a way to collect the return code 15:45:14 since I know this is a wishlist item for Docker guys too 15:45:19 yeah 15:45:22 lots of overlap 15:45:46 we should at least maintain a list of what we want specificially for libvirt LXC though 15:46:34 so those 3 things can we our starting point 15:46:53 so lets talk CI again 15:47:01 i recall from the summit several people wanted to help with CI 15:47:18 RAX is interested in helping w/ LXC testing 15:47:29 and (IIRC) jog0 said it could be part of the official CI, not a 3rd party CI 15:47:50 mikal: was also asking for lxc to be in the upstream gate I believe 15:48:50 presumably though we'll need some interested people to make sure the CI stuff is actually working well enough for LXC before infra will be willing to enable it 15:48:59 yeah 15:49:23 We have one person working on those tests at the moment 15:49:35 excellent 15:50:00 I'll catch up with him and see if we can get him coming to these meetings too. 15:50:07 so i suggest we just do same as with Xen CI - RAX and other interested parties just go away and work on it, and report back progress and/or problems as needed 15:50:09 Might be nice to have a regular update on that sort of progress 15:51:32 yep, again it'd be excellent if we can get something active during Juno 15:51:40 I agree 15:51:41 =] 15:51:56 just time for one last topic.... 15:52:01 #topic Bug Triage 15:52:31 Hi, currently there are 118 bugs tagged as 'libvirt' 15:52:45 is there a query I can use to see untriaged bugs? 15:53:02 anyone know what other virt driver sub-teams do for bug triage ? 15:53:07 apmelton, All "New" Nova bugs are here -- https://bugs.launchpad.net/nova/+bugs?search=Search&field.status=New 15:53:16 thanks kashyap 15:53:18 ie have they got a good practice we can/should follow 15:53:19 danpb, VMWare folks did it recenty 15:53:34 The process is outlined here -- https://wiki.openstack.org/wiki/BugTriage 15:53:55 ah, i see 15:53:57 kashyap: I've been trying to help out with triaging new bugs by following the above posted wiki page 15:54:24 Only thing is consistently applying it - (a) Go through list of Nova bugs periodically, tag relevant bugs as 'libvirt' appropriately (b) Root-cause analysis/poke reporters for bugs w/ less info, etc. 15:54:29 so I guess the real need here is to have a few people who actually have time to actively do bug triage ... 15:55:05 and to draw attention to high priority issues 15:55:06 I'll try to help on that 15:55:07 I can volunteer 15:55:15 directxman12, Likewise here too 15:55:47 Also a lot of bugs I see are woefully short on info, so I wrote this page and post it as a gentle reminder where appropriate -- https://wiki.openstack.org/wiki/BugFilingRecommendations 15:56:32 so i'd say anyone interested just go off and do it, and if you see high priority bugs that need to be addressed, add them to the agenda for the weekly meeting and/or ping relevant people on irc 15:56:44 kashyap: for ones already tagging libvirt, I've been marking them as incomplete and asking for more info if they don't provide version information and logs of some sort 15:57:20 directxman12, Yep, I just did that discreetly for a few bugs while this meeting is going on 15:57:38 it might be an idea to setup a wiki page describing the things we commonly need for troubleshooting libvirt related problems 15:57:44 eg on fedora we've got this http://fedoraproject.org/wiki/How_to_debug_Virtualization_problems 15:58:01 there's probably something we could do that's a bit more tailored to openstack's needs 15:58:08 Ah, true, I was about to just post that. 15:58:24 Yeah, nova debug/verbose logs enabling 15:58:36 well looks like we're out of time here 15:58:49 thanks to everyone for coming along, and see you all next week at same time ! 15:58:55 cheers! 15:58:55 yeah, occaisonally people post pretty useless logs 15:59:01 thanks 15:59:03 au revoir 15:59:06 thanks all! 15:59:14 thanks! 15:59:24 * s1rp off to the containers meeting 15:59:30 See ya. 15:59:42 #endmeeting