14:00:46 <dprince> #startmeeting tripleo
14:00:47 <openstack> Meeting started Tue Feb 16 14:00:46 2016 UTC and is due to finish in 60 minutes.  The chair is dprince. Information about MeetBot at http://wiki.debian.org/MeetBot.
14:00:48 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
14:00:50 <openstack> The meeting name has been set to 'tripleo'
14:01:08 <d0ugal> o/
14:01:18 <dprince> whos here for a meeting?
14:01:22 <marios> hello
14:01:28 <akrivoka> hello
14:01:32 <rbrady> hello
14:02:53 <trown> o.
14:03:12 <shardy> o/
14:03:17 <jdob> o/
14:03:46 <dprince> #topic agenda
14:03:46 <dprince> * bugs
14:03:47 <dprince> * Projects releases or stable backports
14:03:47 <dprince> * CI
14:03:48 <dprince> * Specs
14:03:51 <dprince> * one off agenda items
14:03:54 <dprince> * open discussion
14:04:06 <dprince> normal agenda... anything new to add this week speficically?
14:05:08 <dprince> okay, lets get started then
14:05:12 <dprince> #topic bugs
14:06:15 <dprince> Several bugs earlier this week effected or related to CI outages
14:06:22 <dprince> most have been closed I think
14:07:29 <dprince> any other bugs to highlight this week?
14:07:55 <trown> https://review.openstack.org/#/q/topic:bug/1542486 is still keeping us from moving trunk
14:08:11 <trown> although there are now other issues blocking that, one of which is not even triaged yet
14:08:17 <bnemec> o/
14:09:14 <dprince> trown: will that undercloud patch pass CI then? should we recheck it?
14:09:36 <trown> dprince: it cant pass without removing the puppet-nova pin
14:09:57 <trown> dprince: I was using https://review.openstack.org/#/c/278553/ to test it, but I think now we have new issues
14:10:10 <trown> s/think/know/
14:10:14 <dprince> trown: okay, so could we do this. Update tripleo-ci to cherry pick this solution in, and remove the pin at the same time
14:10:23 <dprince> trown: then we go back and land your instack undercloud solution
14:11:16 <trown> dprince: it is all more tangled than that... we need newer nova too, hence testing with a review that removes delorean pin and puppet pin at the same time
14:11:32 <trown> with depends-on on the fixes
14:11:49 <dprince> trown: sure, lets just do it all in a tripleo-ci patch. I'm fine cherry picking whatever...
14:12:40 <dprince> trown: also, rather than go direct for 'current' it might be nice to use another stable link to some hash/uuid'd repo
14:12:43 <trown> ok, I will put something up once I get the current issue triaged, not much point until we have a patch for that
14:13:00 <dprince> trown: cool, thanks for the updates
14:13:03 <trown> dprince: ya, but which? :)
14:13:13 <trown> that is the hard part when it has been so long
14:13:22 <dprince> trown: how about today, any of them
14:13:30 <dprince> trown: don't care, just needs to pass
14:13:40 <trown> oh, I see, so we dont keep chasing our tail with new breakages
14:13:48 <trown> just keep testing against the same thing
14:13:55 <dprince> trown: yes
14:14:01 <trown> dprince: cool will do
14:14:12 <dprince> cool, lets move on
14:14:14 <dprince> #topic CI
14:14:27 <dprince> CI seems to be running fine mostly
14:14:40 <shardy> master CI looks much better, but there are still a lot of failures on stable/liberty
14:14:48 <dprince> other than not being able to bump our pin on master
14:15:25 <bnemec> Erm, CI looks stuck right now to me.
14:15:36 <dprince> shardy: any bugs that need attention for stable CI?
14:15:39 <bnemec> The top of the zuul queue has been waiting for almost 12 hours.
14:15:49 <dprince> bnemec: oh?
14:16:11 <shardy> dprince: not sure, I've been out for a couple of days so just catching up, but I assume we're missing some backports for stable that fix the issues
14:16:11 <bnemec> dprince: http://status.openstack.org/zuul/
14:16:17 <bnemec> Nothing is running AFAICT.
14:16:31 <shardy> the patches I looked at were failing pingtest for various reasons, e.g nova failing to spawn the VM
14:17:07 <dprince> bnemec: okay, thanks
14:17:43 <dprince> okay, it shoulds like we have a CI outage. Would be nice to indicate the number of queued jobs on our tripleo.org CI status page too :/
14:17:48 <trown> I have this in my logs from last night
14:17:49 <trown> [20:05:03] <greghaynes> slagle: ohai
14:17:49 <trown> [20:05:09] <greghaynes> slagle: Your CI is broken (see -infra)
14:17:49 <trown> [20:05:15] <greghaynes> slagle: https://ci-overcloud.rh1.tripleo.org:13000/v2.0
14:17:49 <trown> [20:05:28] <greghaynes> that refers to localhost which causes nodepool to try and auth against localhost
14:18:15 <marios> shardy: slagle fixup up the "overcloud heat isn't available on pingtest" yesterday with a ci review https://review.openstack.org/#/c/279894/ which may have helped somehow (at least it got a green run for the mido stable/liberty heat templates)
14:18:35 <marios> s/fixup/fixed
14:19:06 <shardy> marios: ack, thanks - the patches I saw weren't that issue but nova errors, will check the latest results after that fix tho
14:19:07 <slagle> yea i was on last night and infra was pinging us about our cloud
14:19:35 <slagle> i dont have any more details, i tried to login and look, but i can't seem to login to the ci cloud any longer
14:19:52 <dprince> I will look into this after the meeting
14:20:22 <marios> regardless of any issues ci is having today, there will likely be a big backlog of stable/liberty (which couldn't land for a while cos ci) so the load may/not be helping here
14:20:24 <dprince> slagle: in the future please email derek and myself if infra pokes you about an outage
14:20:41 <slagle> dprince: well i just happened to be on, and didn't have time to send an email as well
14:20:47 <dprince> slagle: as for access, perhaps poke will foster
14:20:48 <slagle> no one else was on
14:21:06 <EmilienM> do we have monitoring?
14:21:11 <dprince> EmilienM: no
14:21:15 <dprince> EmilienM: we need it
14:21:21 <slagle> dprince: i'd sync with infra about who to contact about the ci cloud and issues they are having
14:21:38 <slagle> dprince: and how you'd like for them to do that
14:21:52 <dprince> slagle: they probably just reach out to those online at the time
14:22:01 <dprince> i.e. come into #tripleo and mention it
14:22:07 <slagle> yep
14:22:30 <EmilienM> dprince: we can talk about monitoring in a separated topic or offline or ML but I'm interested to help here
14:23:01 <dprince> EmilienM: sure, keep in mind we want to re-spin our CI cloud entirely. And if that effort includes monitoring then it would be great too
14:23:20 <dprince> EmilienM: we are using a quite an old release ATM for our CI cloud
14:23:41 <dprince> anyways, lets move on. We can diagnose the CI outage later I think
14:24:00 <dprince> #topic Projects releases or stable backports
14:24:12 <apetrich> I have a question
14:24:14 <dprince> any stable branch updates this week
14:24:24 <dprince> apetrich: ?
14:24:46 <apetrich> sorry still in CI this bug was closed and reopened https://bugzilla.redhat.com/show_bug.cgi?id=1295914
14:24:46 <openstack> bugzilla.redhat.com bug 1295914 in rhel-osp-director "rhel-osp-director: Try to deploy oc 7.2 with netiso, from uc 8.0, fails right away with: Resource CREATE failed: resources.Networks: Property error: resources.StorageMgmtNetwork.properties.StorageMgmtNetValueSpecs: Value must be a string" [High,On_qa] - Assigned to hbrock
14:25:05 <apetrich> it is about backwards compatibility for 8 and 7.3
14:25:52 <marios> apetrich: looks like you didn't set the storage environment?
14:26:04 <shardy> dprince: I started a thread on stable branch policy for Mitaka on openstack-dev
14:26:14 <dprince> shardy: ++, I will reply
14:26:22 <shardy> would be good to get input there so we can agree the plan and announce our FF deadline etc
14:27:01 <trown> +1 thanks for getting that rolling shardy
14:27:06 <apetrich> marios, I can look at it cheers
14:27:47 <marios> apetrich: am not sure, but the original description says "--ceph-storage-scale 1" but no storage env specified afaics, and then steps to reproduce doesn't . i should take a closer look
14:29:01 <apetrich> marios, I have a poc job for uploading of 7 overcloud image into a 8 undercloud I can give you access to the machine
14:29:09 <apetrich> marios, we can talk later
14:29:27 <dprince> okay, well lets save any thoughts for stable barnches for the email thread
14:29:34 <marios> ack apetrich
14:29:42 <dprince> #topic specs
14:30:13 <rbrady> I have a spec for the mistral integration at https://review.openstack.org/#/c/280407
14:31:12 <dprince> rbrady: nice, thanks for posting that
14:32:36 <dprince> I'm starting to get some feedback on pluggable services again
14:32:53 <dprince> https://review.openstack.org/#/c/245804/
14:33:18 <dprince> slagle: could you boil down your concerns into a couple sentances that we might discuss here?
14:34:19 <dprince> slagle: is it primarily the fact that we concatonate the puppet manifests taht you don't like?
14:34:29 <slagle> sure
14:34:53 <slagle> i'm not stricitly opposed to the concatenation in and of itself
14:35:01 <slagle> more about what the implementation implies
14:35:35 <slagle> so, if i wanted to use our puppet api services, and a 3rd party api service
14:35:48 <dprince> slagle: the implementation strives to make it so that a "service architect" (i.e. a team focussed on trove, or sahara) can cleanly integrate their service without looking at the rest of the manifest
14:36:01 <slagle> i'd have to do the 3rd party one in puppet, within the limitation that the puppet manifest would be concatenated with the others
14:36:40 <dprince> slagle: this spec doesn't change the fact that we use Puppet for our configuration with t-h-t at this point
14:36:46 <slagle> and i think that's too restrictive of a framework
14:37:09 <dprince> slagle: I think that is a high bar to set for this pass. And there are consequences too
14:37:33 <slagle> i dont think we have to solve this in the first pass
14:37:48 <EmilienM> wait, I thought the spec was just about decomposing THT manfiests into smaller manifests in puppet-tripleo or in hieradata directly, like glance
14:37:53 <slagle> but i would like to define the framework/interface that we would eventually like to get to
14:38:20 <slagle> i dont feel like i know what direction this first step is even going
14:38:46 <slagle> if it's clear to everyone but me, i'll dig into it a bit more and figure it out
14:38:58 <slagle> it just seemed non-obvious what the end goal was
14:39:40 <dprince> slagle: sure. I thought we had more concensus on this from a few weeks ago before devconf (in Brno). Seems like that discussion only confused the issues?
14:39:52 <slagle> there's soem mention of developing frameworks and interfaces for composability, but no definition of what those are
14:40:16 <dprince> slagle: the spec oulines 7 steps, plus a pre and post step
14:40:19 <marios> slagle: given the goal is 3rd party service integration, ideally would be nice to see an example of what it looks like this way, i.e. what dprince is suggesting with some existing reviews and the spec itself
14:40:39 <dprince> slagle: that is the interface which would fit into our existing architecture quite nicely
14:41:00 <dprince> FWIW I demo'ed this in tokyo: https://review.openstack.org/#/c/237370/
14:41:14 <dprince> That is glance-api and glance-registry split out into this architecture
14:41:30 <slagle> dprince: if we focus on just the 7 steps, then i think the interface can be defined as puppet snippets for each step that are all concatenated together and run as one at each step
14:42:02 <slagle> and personally i think that's too restrictive of a framework
14:42:05 <dprince> slagle: yes, that is the interface
14:42:41 <dprince> slagle: the architecture of t-h-t puppet is someone influenced by previous installers like Spinal Stack, etc.
14:43:02 <EmilienM> that's wrong
14:43:06 <dprince> slagle: With DIB tripleo previously we had a single pass per role. os-collect-config runs once
14:43:12 <EmilienM> in spinalstack we had a composable composition layer
14:43:15 <dprince> slagle: and does everything
14:43:19 <EmilienM> something that is not in tripleo
14:43:30 <dprince> EmilienM: ++ I said influenced. please wait just a second
14:43:37 <EmilienM> https://github.com/redhat-cip/puppet-openstack-cloud/tree/master/manifests
14:43:37 <shardy> It's somewhat difficult to fully define a "plugin" API until we decompose the current monolithic model IMO - I see the work proposed by dprince as just the first step down that path
14:43:54 <EmilienM> the link I posted is something composable
14:44:00 <slagle> i guess the use case i'm thinking of is if i had a service written in bash, or soem locally applied ansible playbooks and wanted those executed alongside the puppet ones
14:44:07 <dprince> With t-h-t we are trying to have a stepwise architecture (not like DIB which was single pass). So step1 executes on all the roles.
14:44:31 <slagle> and i'd envinsion where i wrote my templates, and just wrote out the softwraedeployments for 7 steps for my custom service
14:44:40 <slagle> and that integrated cleanly alongside all the puppet stuff
14:44:52 <slagle> so if i still wanted to run the puppet api services alongside my own, i could
14:45:00 <slagle> that's the use case i'm thinking of anyway
14:45:41 <slagle> instead of having to bolt my bash or ansible thing on at the end via ExtraConfig
14:45:46 <slagle> i'd like to use the service interface
14:46:04 <dprince> The initial vision was using stepwise execution. Step 1 on all the roles, then a cluster wide validation to make sure it works. Something like this (which we never landed) https://review.openstack.org/#/c/174150/
14:46:15 <dprince> Step 2 -> Then a step 2 cluster wide validation
14:46:26 <slagle> and have it be more or less opaque to the the templates making up the framework what the actual implementation is
14:46:56 <dprince> slagle: we do have ExtraConfig templates for some things in t-h-t that allow for some pluggablity in what you are asking
14:46:58 <slagle> as is, the restriction is that it must be puppet, it must play nice when concatenated with all the other steps at the same sequence
14:47:09 <dprince> slagle: and I think it would be totally reasonable to add to our interfaces in the future
14:47:46 <dprince> slagle: I'm not restricting any future additions to the heat templates, rather I'm solving a problem with have with the puppet implementation today. Namely that it is monolithic
14:47:55 <EmilienM> that won't help much, but I thought it would be useful to share. I wrote a small tool with ansible to compose puppet hieradata and deploy openstack in a composable way https://github.com/EmilienM/ansible-marionette
14:48:00 <dprince> slagle: if we don't take this first step (of splitting things up) we can't move forwards
14:49:17 <dprince> slagle: what I'm saying is, lets iterate towards these things. We can refine our interfaces in the years to come. But what I'm trying to solve here is service pluggability in puppet
14:50:30 <dprince> slagle: So if you look at my validations patch you'll notice it allows you to plug in a shell script between each deployment run. I think something like that is what you are talking about. If not perhaps you could throw up a patch which gives a demonstration of how you'd like to accomidate these extra interfaces in Heat?
14:51:12 <dprince> Or perhaps we need to take this to the mailing list to discuss further.
14:51:18 <slagle> i think that would be premature
14:51:30 <shardy> I wonder if we can address the requirement expressed by slagle via encoding the config group in the per-service mapping:
14:51:32 <slagle> i'm not trying to come up with patches, merely define what interfaces we are aiming for
14:51:36 <shardy> https://review.openstack.org/#/c/237370/7/puppet/roles/glance-api.yaml
14:51:49 <slagle> so that there's agreement on that
14:51:53 <shardy> e.g rather than just "config_step4: manifest"
14:51:56 <marios> dprince: would it make sense to make the steps themselves pluggable, for example have a step4/5 (whichever corresponds to where neutron services are declared) for each neutron service provoider/intergration... we may have some duplication there though
14:52:07 <shardy> you'd have config_step4 contain e.g a "puppet" key or something
14:52:27 <shardy> then you have an interface we could potentially extend to also chain e.g scripts or whatever together
14:52:31 <slagle> dprince: i think i am thinking too much about the problem description in the spec
14:52:44 <dprince> slagle: sometimes having a patch helps people see your idea, and/or formulate it. That is all I'm saying
14:52:57 <slagle> where as what (i think) you're saying is that the implementation isn't going to address a lot of that problem description
14:53:37 <slagle> it sounds like the problem description is "puppet manifests are monolithic"
14:53:44 <slagle> or should be, rather
14:53:59 <dprince> slagle: I had a section called "taking it a step further" (to truely composable roles). Would you like me to remove that language?
14:54:27 <marios> dprince: i mean change the values of the output in https://review.openstack.org/#/c/236243/18/puppet/controller-config.yaml so we use the corresponding Step5 for example, for given integration
14:55:04 <slagle> dprince: it's more the previous 2 paragraphs tbh
14:55:30 <dprince> marios, slagle, shardy: would you be interested in a video conference this week to speak on these issues then?
14:56:01 <slagle> the 2nd paragraph talks about defining the interface
14:56:09 <dprince> At this point I'm almost inclined to delete the spec and say we can just use patches (which may better explain my motivations than the spec does.
14:56:16 <shardy> dprince: sure
14:56:19 <slagle> and i think the implementation defines an interface i don't feel is flexible/pluggable enough
14:56:37 <dprince> slagle: it works, and is an iterative step forwards
14:56:39 <slagle> sure, a call would be fine for me as well
14:56:54 <marios_> dprince: sorry, client dropped... yeah would be happy to join a call
14:57:23 <shardy> It's kinda interesting looking at the fuel plugin interface btw - they have covered a lot of these same issues (deployment steps, dependencies between steps, running scripts vs puppet etc etc)
14:57:31 <dprince> okay, we are almost out of meeting time
14:58:44 <dprince> thanks everyone
14:59:06 <dprince> #endmeeting tripleo