19:05:58 <SpamapS> #startmeeting tripleo
19:05:58 <openstack> Meeting started Tue Feb 11 19:05:58 2014 UTC and is due to finish in 60 minutes.  The chair is SpamapS. Information about MeetBot at http://wiki.debian.org/MeetBot.
19:06:00 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
19:06:02 <openstack> The meeting name has been set to 'tripleo'
19:06:06 <SpamapS> Ahoy there!
19:06:06 <lsmola> SpamapS: but works with slagle 's undercloud-live as rlandy tried
19:06:27 <lsmola> _o/
19:06:31 <SpamapS> lsmola: wait one moment while we get to the bugs topic :)
19:06:42 <SpamapS> #topic bugs
19:06:46 <lsmola> SpamapS: I am a skipper :-)
19:07:01 <lsmola> hehe that was fast :-D
19:07:17 <SpamapS> oh I forgot
19:07:19 <SpamapS> #link https://wiki.openstack.org/wiki/Meetings/TripleO
19:07:36 <SpamapS> #link https://bugs.launchpad.net/tripleo/
19:07:36 <SpamapS> #link https://bugs.launchpad.net/diskimage-builder/
19:07:36 <SpamapS> #link https://bugs.launchpad.net/os-refresh-config
19:07:36 <SpamapS> #link https://bugs.launchpad.net/os-apply-config
19:07:36 <SpamapS> #link https://bugs.launchpad.net/os-collect-config
19:07:39 <SpamapS> #link https://bugs.launchpad.net/tuskar
19:07:41 <SpamapS> #link https://bugs.launchpad.net/tuskar-ui
19:07:43 <SpamapS> #link https://bugs.launchpad.net/python-tuskarclient
19:08:40 <slagle> https://bugs.launchpad.net/tripleo/+bug/1279011 needs triage
19:08:55 <slagle> lsmola: i assume the heat syack-update in the description is just a typo in the bug?
19:09:00 <SpamapS> slagle: it is assigned a High importance, it just isn't known to be reproducible.. :)
19:09:10 <lsmola> slagle: yes sorry
19:09:30 <slagle> SpamapS: it should still be set to Triaged though? or no?
19:10:13 <SpamapS> slagle: Well I see New as a signal to developers to try and clarify the bug... Triaged would just put it on the stack of things to fix.
19:10:14 <lsmola> slagle: set
19:10:33 <lsmola> SpamapS: unset? :-)
19:11:00 <SpamapS> lsmola: if you think devs can work it then Triaged is appropriate. If you'd like to see if more users experience the problem, New.
19:11:10 <SpamapS> I'm fine either way.
19:11:17 <SpamapS> Triaged is really such a weird status name.
19:11:38 <lsmola> SpamapS: ok, I vote for more users
19:11:49 <SpamapS> :) ok
19:11:50 <lsmola> so anybody else experienced it?
19:11:55 <SpamapS> so https://bugs.launchpad.net/tripleo/+bug/1271344 has been hitting us on the cd-undercloud lately
19:12:10 <SpamapS> but I think we _also_ have some machines in our rack that don't work well
19:12:29 <SpamapS> I've been working on isolating those so we can open tickets and get them fixed
19:13:09 <SpamapS> Also I totally forgot releases last week so the fix committed bugs can all go to fix released soon
19:13:42 <jistr> all check-tripleo-undercloud-precise Jenkins jobs fail on this: https://bugs.launchpad.net/tripleo/+bug/1278861
19:13:55 <jistr> there is workaround available if running locally, so i set it to high
19:14:05 <SpamapS> I think this bug needs some interaction https://bugs.launchpad.net/tuskar/+bug/1278976 ... the poster gives no justification for why /etc is a bad place for t-h-t
19:14:15 <jistr> but i wonder if it should be critical because it breaks Jenkins
19:14:57 <SpamapS> jistr: I think so.. the bar for me is the amount of affect it has on users.. and jenkins not working is indeed a large effect!
19:15:16 <SpamapS> Any other bugs?
19:15:17 <jistr> SpamapS: ok, done
19:16:26 <SpamapS> #topic reviews
19:16:40 <SpamapS> #link http://russellbryant.net/openstack-stats/tripleo-openreviews.html
19:17:15 <SpamapS> seems we've been leaving things sitting a bit more this past week
19:17:41 <SpamapS> Consider this a reminder that we need to keep up with the in-flow.
19:17:51 <SpamapS> I don't think we have a team shortage.. it seems like we've done less reviews is all.
19:19:12 <SpamapS> #topic Projects needing releases
19:19:26 <SpamapS> shadower: I think you wanted to do the release process?
19:19:36 <shadower> SpamapS: yup
19:19:38 <SpamapS> rpodolyaka1: perhaps you can assist shadower ?
19:19:47 <rpodolyaka1> SpamapS: shadower: sure
19:19:48 <SpamapS> rpodolyaka1: (and welcome back from vacation :)
19:19:50 <shadower> any docs you can point me at?
19:19:57 <rpodolyaka1> SpamapS:  thanks :)
19:20:03 <ccrouch> tripleo-heat-templates, tripleo-image-elements and diskimage-builder have a bunch of changes and don't look like they have had a release in ~3wks
19:20:14 <rpodolyaka1> shadower: just ping me, if you need any help with releases
19:20:24 <shadower> rpodolyaka1: will do, thanks
19:20:34 <rpodolyaka1> shadower: np
19:21:15 <SpamapS> The process is basically "make a signed tag. Push that tag to gerrit. Close bugs"
19:21:27 <SpamapS> Not sure if we have an automated thing for "Close bugs"
19:21:46 <rpodolyaka1> yeah, we have a script
19:21:52 <SpamapS> rpodolyaka1: sweet
19:21:56 <SpamapS> #topic CD Cloud status
19:22:04 <shadower> sweet
19:22:23 <SpamapS> tripleo-cd is failing always now because of a combination of hardware issues and the neutron bug I cited earlier
19:22:45 <SpamapS> I believe I'm close to isolating _12_ "bad" machines.
19:23:28 <SpamapS> but it could be some of them are just victims of the neutron bug so I am going to be disabling them all and then enabling each one to see if it can deploy.
19:23:29 <tchaypo> What does bad mean in this case?
19:23:47 <SpamapS> tchaypo: won't pxe boot or fail somewhere between pxe boot and running.
19:24:26 <SpamapS> remotely debugging them via ilo can be tedious because of the 8 minute POST windows where the console shows nothing. :-P
19:24:35 <tchaypo> So possibly a variety of causes, but all we care about is that they aren't useable right now
19:24:59 <SpamapS> tchaypo: right, we want to just make sure we stop trying to deploy to the bad ones.
19:25:02 <lsmola> have to leave early today, see you guys tomorrow
19:25:16 <SpamapS> and that we get our NOC to fix them
19:25:46 <tchaypo> At least you can enjoy a coffee while waiting for the ILO. Tends to be frowned on if you're physically in front of the machine
19:26:48 <SpamapS> hah true
19:27:03 <SpamapS> ok so that is where it is at. Hopefully will recover and regain our 50% uptime rating this week. :-P
19:27:23 <SpamapS> #topic CI virtualized testing status
19:27:45 <SpamapS> It seems like they're deep in discussion about tripleo things in the infra meeting..
19:27:48 <SpamapS> so lets skip and come back
19:28:02 <SpamapS> #topic open discussion
19:28:12 * pino|work has a topic
19:28:17 * matty_dubs as well
19:28:21 <SpamapS> lifeless: when you're done in infra meeting.. ping back here for the CI virtualized testing stuff
19:28:24 <SpamapS> pleia2: you too
19:28:25 <SpamapS> dprince: you three
19:28:36 <SpamapS> pino|work: fire away
19:28:57 <pino|work> it is something that has been proposed few times in the past
19:29:07 <SpamapS> Puppies for everyone?
19:29:10 <SpamapS> I told you, the shipping costs...
19:29:13 <SpamapS> oh ;)
19:29:29 <pino|work> ie basically making use of the guestfs tools (virt-builder, virt-sysprep, guestfish, etc) to handle the generation and edit of images
19:30:06 <SpamapS> pino|work: I think we'd all be open to other tools. We have a _lot_ invested in tripleo elements. But diskimage-builder's guts are pretty light weight.
19:30:11 <pino|work> ... instead of disk-image-builder (which, at least to my looks earlier today, seemed a bit... fragile and possibly insecure)
19:30:39 <lifeless> SpamapS: aiieee
19:31:17 <pino|work> since rwmjones and me are starting to head toward a new 1.26 release in a month or two, it would be a great time for asking feedback from you guys and have this reflected in our tools
19:31:25 <SpamapS> pino|work: spin up a VM.. run dib in it.. tear down VM. Not sure where the insecurity is.
19:32:00 <pino|work> that's what our tools do as well
19:32:03 <jistr> pino|work: is it clear how to make the guestfs tools use tripleo-image-elements? or is it something that needs to be worked out?
19:32:14 <SpamapS> pino|work: I would be willing to bet that you could change dib to work via other tools than the current chroot based solution.
19:32:48 <pino|work> SpamapS: i remember about the chroot, it was not totally clear it was spawned within a vm
19:33:00 <lifeless> SpamapS: so am here
19:33:05 <lifeless> question was?
19:33:09 <jprovazn> I agreee that DIB elements should might work with virt-builder
19:33:12 <SpamapS> pino|work: it is not, but it could be with relative ease
19:33:24 <SpamapS> lifeless: CI virtualized testing progress
19:33:31 <jprovazn> and AFAIK virt-builder supports building of windows images
19:33:32 <SpamapS> lifeless: but let's address pino's issue first
19:33:44 <SpamapS> Ok this is TripleO...
19:33:50 <SpamapS> we're not running pieces of OpenStack on Windows. :)
19:33:51 <lifeless> pino|work: whats insecure about dib? Note that if you say 'the image you unpack might be hostile' - note that if the image is hostile, the code in it certainly is, and that code is going to be your /cloud/.
19:34:02 <rwmjones> guestfs does the "spinning up in a VM" already (and uses things like sVirt and containers too)
19:34:20 <pino|work> lifeless: handling tools like kpartx, losetup, etc as root
19:34:34 <jprovazn> so it would be great to consider virt-builder before guys working on DIB for windows go too far
19:34:36 <slagle> fyi, i wrote a thing a while back to apply elements not using a chroot. virt-builder could use that with relative ease i suspect, since it can run arbitrary scripts
19:34:43 <SpamapS> jprovazn: definitely
19:34:45 <lifeless> pino|work: right, this debate is all based around the argument that you're unpacking random images that might be hostile.
19:34:52 <SpamapS> slagle: indeed!
19:35:07 <SpamapS> slagle: the dib element interface is well documented and _relatively_ stable.
19:35:12 <lifeless> pino|work: Which is a scenario Nova has, but we don't.
19:35:15 <pino|work> jprovazn: exactly, i was told about the windows support, which at least we should have at some degree (of course, we're open to improvements)
19:35:35 <SpamapS> so if there is a concern over dib, I'd like to see a clear write up of the concerns as a blueprint, and an open discussion on the ML where the concerns can be articulated.
19:35:52 <SpamapS> blueprint or bugs or both actually :)
19:35:55 <lifeless> well
19:35:58 <lifeless> lets start with a bug
19:36:07 <lifeless> blueprints are timeline coordination tools
19:36:11 <pino|work> regarding the elements: i looked at the current elements, and most of the features done with them are currently doable with virt-builder/virt-sysprep/guestfish
19:36:13 <lifeless> a bug/etherpad/mailing list discussion.
19:36:55 <jistr> pino|work: is it possible to build images with guestfs tools without having sudo rights?
19:37:26 <lifeless> pino|work: so, I'm still unclear: whats the problem statement.
19:37:50 <lifeless> pino|work: and what are the benefits/tradeoffs. Technical ability to do the thing comes second IMO.
19:37:53 <pino|work> jistr: noting that we don't do installations from scratch/kickstart/d-i/etc, the rest is done with no need for sudo/root
19:38:22 <rwmjones> jistr: yes, no sudo or root needed at all for anything libguestfs related
19:38:30 <pino|work> lifeless: benefits: image manipulation doable without root rights at all, in an isolated vm
19:38:50 <rwmjones> jistr: except if you have an image which is owned by root and -rw------ but I guess that goes without saying ..
19:39:02 <jistr> rwmjones, pino|work: sounds good
19:39:17 <SpamapS> pino|work: we can do that, just by running dib.. in an isolated vm.. :p
19:39:24 <lifeless> pino|work: as SpamapS says, we have that.
19:39:34 <lifeless> pino|work: sdake put a heat template together for it.
19:39:34 <pino|work> lifeless: userbase very-well tested, also in production, and with easier user handling than shell scripts
19:39:35 <SpamapS> pino|work: if you could automate just doing that.. that might be attractive. :)
19:39:58 <lifeless> pino|work: what do you mean by 'easier user handling than shell scripts'?
19:41:19 <slagle> dib could still destroy the vm it's in due to it's chroot bind mounting
19:41:39 <pino|work> lifeless: easier way to do operations than using chunks of shell code
19:41:40 <SpamapS> slagle: I doubt we'd shed a tear for said VM. ;)
19:41:54 <slagle> SpamapS: sure, it's just a very unfriendly thing
19:42:02 <slagle> and hard(ish) to debug
19:42:18 <lifeless> pino|work: uhm
19:42:29 <lifeless> pino|work: I still don't understand 'easier than shell code'
19:42:30 <pino|work> lifeless: for example, virt-builder can handle users creation/passwords, package installation, file editing and more during the image building phase
19:42:32 <SpamapS> slagle: we destroyed /dev on everybody's laptop and, while mildly inconvenient, we didn't stop using it or start using it in vms the next day ;)
19:42:36 <lifeless> pino|work: since the whole point of dib is to run shell code.
19:42:56 <SpamapS> pino|work: that is complecting
19:43:20 <slagle> SpamapS: indeed, but i'm not thinking about "us"
19:43:22 <pino|work> "complecting"?
19:43:31 <SpamapS> pino|work: dib runs elements. It is simple and does one thing well. I don't want dib to know how to make users.
19:43:34 <matty_dubs> In any case, is this the place to sort it out? Or should this be taken to a bug/mailing list discussion?
19:43:35 <ccrouch> http://www.thefreedictionary.com/complecting
19:43:37 <rwmjones> I think "complicating"
19:43:38 <slagle> SpamapS: more so a cloud admin/operator
19:43:43 <matty_dubs> I feel like this conversation could go on for several hours
19:44:01 <bnemec> matty_dubs: +1
19:44:01 <rwmjones> crumbs, that is a real word :-)
19:44:04 <jprovazn> SpamapS, I remember that vaporized /dev on my laptop was beautiful welcome message when I was starting :)
19:44:17 <SpamapS> Right I'm suggesting that this is something that needs an articulated document and some thought that a 59 minute meeting will not allow.
19:44:25 <lifeless> agreed
19:44:42 <lifeless> so look, in principle I'm not against dib dying a beautiful death as patches to something else.
19:44:45 <lifeless> But
19:44:48 <SpamapS> jprovazn: welcome to Seattle, you didn't need /dev anyway
19:44:50 <pino|work> sure, i was generally trying to restart a discussion about it
19:45:00 <jprovazn> hehe
19:45:07 <lifeless> We've a -very- tight timeline for delivery of key things for multiple vendors
19:45:11 <lifeless> and dib is finished.
19:45:17 <lifeless> It works, it meets all our current use cases.
19:45:38 <lifeless> So I have approximately 0 interest in fixing it for fixing sake at the moment.
19:45:39 <pino|work> i was told you need windows support... which means kind of rewriting it
19:45:47 <SpamapS> we do not
19:45:52 <rwmjones> well we're going to work on some stuff to make virt-builder do everything that dib can do, so suggestions are welcome
19:46:01 <lifeless> pino|work: There's a vendor that wants a window image building thing analagous to dib.
19:46:11 <SpamapS> OpenStack on OpenStack has no need to deploy workloads on Windows AFAIK.
19:46:23 <lifeless> pino|work: We'll point them at virt-builder and see if it meets their use case.
19:46:23 <jprovazn> pino|work, not rewriting but creating another "dib windows" project
19:46:35 <SpamapS> we may _deploy_ windows images.. but that is way different than needing windows for TripleO.
19:46:38 <lifeless> pino|work: if it doesn't, they might offer patches, or do something new.
19:46:48 <lifeless> pino|work: dunno yet. How mature is virt-builders windows support ?
19:47:02 <lifeless> rwmjones: ok, cool!
19:47:05 <rwmjones> it can create a windows image, but not install software or firstboot scripts yet
19:47:17 <rwmjones> resize supports works at the moment
19:47:23 <lifeless> rwmjones: ok, so the interesting bit is installing stuff :)
19:47:35 <rwmjones> certainly, but we'll get that soon
19:47:50 <lifeless> rwmjones: on making virt-builder do what dib can do: - key features for me:
19:47:53 <lifeless> - run simple shell scripts
19:47:59 <rwmjones> (done!)
19:48:04 <lifeless> - access resources outside the image being built
19:48:08 <pino|work> lifeless: i was not trying to said "kill dib now", just that we could use your feedback in making our tools have the functionalities you need right now with dib (as to eveventually switch to them in some future)
19:48:10 <rwmjones> done!
19:48:26 <dprince> rwmjones: the only hangup for me is installing RPMs (without using a firstboot script)
19:48:29 <rwmjones> (you can attach disks, or grab stuff off the network)
19:48:29 <lifeless> - let users interactively poke around within the thing, to debug things that failed.
19:48:36 <rwmjones> you can install RPMs
19:48:45 <lifeless> rwmjones: ah, so neither of attach disks or grabb off network are what I mean
19:48:46 <rwmjones> lifeless: that's a good point, our debug support isn't great
19:49:02 <lifeless> rwmjones: I mean 'I have a local file <here>' and I want it <there> in the image.
19:49:13 <rwmjones> lifeless: oh right, yes you can do that now, using --upload option
19:49:29 <lifeless> rwmjones: so that e.g. RPM can share its cache with the host os.
19:49:30 <pino|work> (or the upload command in guestfish, for example)
19:49:32 <rwmjones> but only files, we could/should let you upload more substantial stuff
19:49:35 <lifeless> rwmjones: which --upload doesn't deliver.
19:49:46 <rwmjones> like directories
19:49:51 <lifeless> rwmjones: and uploading the whole cache to pull out what files are needed sounds expensive.
19:50:06 <rwmjones> well we could do it with 9pfs, which libguestfs supports but virt-builder doesn't use right now
19:50:15 <lifeless> rwmjones: ah, ok.
19:50:29 <rwmjones> ok, all good ideas anyway, thanks
19:50:32 <lifeless> rwmjones: so anyhow - teh debug support we have is also limited today but it is super useful
19:50:39 <lifeless> rwmjones: the bind mounting stuff is super useful
19:50:51 <SpamapS> the use case is that we will run dib on a machine that already has git trees checked out and ready to use..
19:51:00 <lifeless> rwmjones: dib itself has a few minor os specific abstractions, I'm sure virt-builder has them too
19:51:02 <SpamapS> among other things
19:51:05 <rwmjones> ok I'm off now, but thanks -- will read the IRC log when it is posted
19:51:05 <pino|work> we have guestmount to mount images on your host using fuse
19:51:11 <slagle> like well populated yum caches :)
19:51:13 <lifeless> pino|work: we don't do that
19:51:31 <lifeless> pino|work: we start with tarballs - when we download a qcow2 or something, the first thing we do is unpack it into a tarball
19:51:42 <matty_dubs> May I humbly submit that this feels quite off-topic and has taken about a good chunk of the meeting?
19:51:48 <SpamapS> agreed
19:51:50 <lifeless> agreed
19:51:52 <rwmjones> agreed
19:51:54 <SpamapS> and matty_dubs has another topic
19:52:11 <matty_dubs> Oh, right! Mine is probably far less-interesting, though.
19:52:23 <SpamapS> YOU want puppies too? ;)
19:52:27 <matty_dubs> Was just curious about the meetup. Is there a venue set?
19:52:36 <matty_dubs> Hotel recommendation?
19:52:43 <matty_dubs> And, will there be puppies handed out at the meeting?
19:52:44 * jcoufal shares the same question
19:52:47 <tchaypo> What's all this about puppies? I was promised ponies!
19:52:57 <shadower> matty_dubs: what a bore!
19:53:00 <SpamapS> matty_dubs: City: yes, Venue: may shift.
19:53:21 <SpamapS> Hotel is something we should have "shortly" with a group rate.
19:53:29 <matty_dubs> Oh, and are we meeting Friday, or is it a travel day? I had conflicting information. (Either's A-OK with me.)
19:53:44 <SpamapS> I believe the venue will be open and some will stay through Friday
19:53:50 <SpamapS> I am leaving Friday morning.
19:54:27 <matty_dubs> Ah, okay, so either will work then
19:55:06 <matty_dubs> So we should book flights to be there Monday-Friday, but not everyone will be there Friday, and we shouldn't book a hotel until we hear about a group rate.
19:55:10 <matty_dubs> ^ apt summary?
19:56:05 <SpamapS> matty_dubs: I believe so yes
19:56:11 <matty_dubs> Awesome, thanks.
19:56:12 <SpamapS> Not sure why the hotel is taking so long.
19:56:16 <SpamapS> Thought we'd have that by now.
19:56:28 <SpamapS> with that I believe we will come to a close.
19:56:37 <SpamapS> watch the ML for more details about the meetup.
19:57:06 <SpamapS> thanks everyone!
19:57:13 <jomara> thanks SpamapS
19:57:15 <SpamapS> #endmeeting