14:03:13 <fifieldt> #startmeeting ops_tags
14:03:14 <openstack> Meeting started Thu Sep 24 14:03:13 2015 UTC and is due to finish in 60 minutes.  The chair is fifieldt. Information about MeetBot at http://wiki.debian.org/MeetBot.
14:03:16 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
14:03:18 <openstack> The meeting name has been set to 'ops_tags'
14:03:34 <fifieldt> ok, with luck, the minutes will be sent to the right place, unlike last time
14:03:36 <fifieldt> roll call
14:03:40 <fifieldt> anyone here for the ops tags meeting?
14:03:44 <shamail> Present.
14:03:47 <fifieldt> wohoo
14:03:56 <jproulx> distracted but nearby...
14:03:57 <shamail> The one who said I can't make it, lol.
14:04:03 <fifieldt> thanks jproulx
14:04:15 <fifieldt> looks like we don't have a maish today, oh well
14:04:34 <fifieldt> On the agenda is:
14:04:34 <fifieldt> 1. Planning our summit session
14:04:34 <fifieldt> 2. Review of open reviews (https://review.openstack.org/#/q/status:open+project:stackforge/ops-tags-team,n,z )
14:04:34 <fifieldt> 3. Tag Brainstorming
14:04:48 <fifieldt> does anyone want to manipulate the agenda before we begin?
14:05:20 <fifieldt> going once ...
14:05:27 <fifieldt> going twice ...
14:05:32 <fifieldt> going three times ...
14:05:41 <fifieldt> and, it looks like we're sold on the agenda!
14:05:43 <fifieldt> great
14:05:44 <fifieldt> so
14:05:50 <fifieldt> #topic Planning out summit session
14:05:57 <fifieldt> If we look at https://docs.google.com/spreadsheets/d/1EUSYMs3GfglnD8yfFaAXWhLe0F5y9hCUKqCYe0Vp1oA/edit#gid=1480678842
14:06:12 <fifieldt> we have a 40 minute slot on Tuesday at 3:40 - 4:20
14:06:18 <fifieldt> what, do you think we should be doing there?
14:06:44 <shamail> Is the intention to be a working session or update session?
14:07:09 <fifieldt> working session, however, given the relative low profile of our group, it might be good to start it with an update
14:07:10 <shamail> I assume working but wanted to confirm
14:07:18 <shamail> +1
14:07:48 <fifieldt> I guess one of the aims I hope for is that we drag some passers by into doing things with the group on an ongoing basis
14:08:01 <shamail> Agreed
14:08:43 <shamail> Starting with an update and how this group is building tags that are different from TC would be good
14:08:51 <fifieldt> yup
14:08:54 <shamail> Followed by tags proposed so far
14:08:58 <fifieldt> +1
14:09:41 <shamail> Which tags are left open?  I think rally tag is moving to TC right?
14:09:57 <fifieldt> oh eys, the rally tag is progressing well thanks to the rally team
14:10:14 <shamail> I'd like to discuss the scale tag (or an alternative form) at the summit
14:10:15 <fifieldt> here's our list: https://review.openstack.org/#/q/ops-tags-team,n,z
14:10:28 <fifieldt> sounds nice to me
14:10:36 <fifieldt> we could kick start that here, too
14:11:08 <shamail> Ok, I'll throw out an idea when we get to agenda item #3
14:11:09 <fifieldt> I would like to talk in general about 'maturity', and the various ways we can proxy it
14:11:12 <fifieldt> ok, thanks
14:11:20 <fifieldt> so yes, it seems at the summit
14:11:22 <shamail> fifieldt: +1
14:11:31 <fifieldt> 1. intro 2. tags proposed 3. tag work/brainstorming
14:11:39 <fifieldt> that good for our summit session?
14:11:46 <fifieldt> with an aim on increasing participation?
14:11:49 <jproulx> soounds like that would well fill 40min
14:12:43 <shamail> I think that's good too
14:12:44 <fifieldt> shamail, you're moderating, is that OK with you?
14:12:46 <fifieldt> great
14:12:53 <fifieldt> then I think we can move on to our second agenda topic
14:13:04 <fifieldt> #topic Review of open reviews (https://review.openstack.org/#/q/status:open+project:stackforge/ops-tags-team,n,z )
14:13:25 <fifieldt> so, an easy one first
14:13:33 <fifieldt> I went back and did the production-use data for juno
14:13:34 <fifieldt> https://review.openstack.org/#/c/223461/
14:13:42 <fifieldt> that one's probably good to go, since it's just data
14:13:43 <fifieldt> no new tag
14:14:26 <fifieldt> ya?
14:14:42 <shamail> I'll go through and review this weekend too
14:14:53 <fifieldt> cool
14:15:02 <fifieldt> well, how about a new tag then:
14:15:02 <fifieldt> https://review.openstack.org/#/c/225539/
14:15:05 <fifieldt> sdk-support
14:15:28 <fifieldt> it's pretty basic, just go through the list of SDKs that aren't dead, work out which projects they support, and count it up
14:15:40 * shamail is reading
14:16:17 <shamail> So just a count of SDKs versus languages etc?
14:16:43 <fifieldt> SDKs vs projects, ya
14:17:29 <fifieldt> potential uses - approximating maturity - more SDKs supporting a project means it gets more use developing apps, or something
14:17:44 <shamail> Got it.  What would be the benefit to the user?
14:17:50 <shamail> Hah, you read my mind
14:18:28 <shamail> Makes sense, do you think we could expand to show API version covered too?
14:18:35 <shamail> Or would that be troublesome?
14:18:54 <fifieldt> certainly possible to extend it in a number of ways :) I was just starting simple
14:19:11 <fifieldt> actually working out the API support is a bit more complex than the project support, in my experience :)
14:19:29 <fifieldt> I guess this is only useful in aggregate for now
14:20:02 <shamail> It would be a lot more complex.
14:20:08 <fifieldt> aye
14:20:14 <shamail> Good starting point though, as you said, can be expanded later.
14:20:14 <jproulx> since we're doing structured data would a list of which SDK these were be usefull?
14:20:47 <jproulx> for example so you could filter on projects that were supported by your SDK or choice, or is that too much fo rnow?
14:21:01 <fifieldt> that might be a good extension
14:21:09 <jproulx> maybe for round 2
14:21:11 <fifieldt> do you think we should merge the simple version first, then extend?
14:21:19 <fifieldt> in followup patches, that is
14:21:20 <shamail> jproulx: +1
14:21:24 <shamail> Yep!!
14:21:58 <jproulx> I guess question to fifieldt would it be trivial to add now or would it be significant work? if we have the list why not expose it
14:22:21 <fifieldt> not so trivial - some typing and checking for me :)
14:22:34 <jproulx> lets leave it for future work tehn.
14:22:40 <fifieldt> okay
14:22:52 <fifieldt> ok, well, swiftly looking at another one
14:22:56 <fifieldt> medberry did containerizable
14:23:04 <fifieldt> https://review.openstack.org/#/c/214801/
14:23:16 <fifieldt> description is done and ready for review
14:23:22 <fifieldt> the data doesn't exist yet
14:23:45 <fifieldt> but we don't necessarily need it to to get the tag in ...
14:24:42 <fifieldt> any thoughts>?
14:25:24 <shamail> I like this one
14:25:39 <jproulx> is there any reason a service wouldn't be containerizable though?
14:25:43 <shamail> The Kolla team has built 90 images for 5 distributions
14:25:50 <fifieldt> actually, yes jproulx - some folks found some
14:25:54 <shamail> Being able to capture which services has images available would be good
14:26:13 <fifieldt> can you say more shamail? I'm not familiar with that :D
14:26:14 <jproulx> OK then.
14:26:34 <shamail> Sure.  I don't have much more but I can certainly get more info.
14:26:57 <fifieldt> that might be interesting :)
14:27:01 <coolsvap> Kolla has containerized most core services for centos and ububtu
14:27:02 <fifieldt> well, don't want to put you on the spot
14:27:04 <shamail> When asking the PTLs for a liberty update, the Kolla team said they have built about 400 containerized services for OpenStack
14:27:10 <fifieldt> ooh
14:27:18 <coolsvap> with source and binary both
14:27:19 <shamail> I assume this is meaning a service is decomposed into multiple containers
14:27:31 <coolsvap> yes
14:27:35 <fifieldt> did they find any issues do you know coolsvap?
14:27:43 <shamail> Furthermore they have done wit with Ubuntu, CentOS and 3 more distributions
14:27:55 <shamail> So 90 * 5 images
14:28:00 <jproulx> so does this basicly mean Kolla-ised? else how do we know if some randome site is using it ocntainerized in production
14:28:03 <coolsvap> shamail, yes
14:28:13 <fifieldt> I found this bug: https://bugs.launchpad.net/ubuntu/+source/lxc/+bug/1226855
14:28:14 <openstack> Launchpad bug 1226855 in lxc (Ubuntu) "Cannot use open-iscsi inside LXC container" [Wishlist,Confirmed]
14:28:14 <coolsvap> currently most services can be build
14:28:17 <fifieldt> no iscsi inside containers
14:28:33 <coolsvap> we are working on getting all services functional by liberty
14:28:46 <coolsvap> most services are already functional
14:28:54 <fifieldt> if everything works flawlessly, then maybe this tag isn't needed
14:29:01 <coolsvap> and we can actually create openstack
14:29:11 <fifieldt> were there any major issues coolsvap?
14:29:11 <coolsvap> deployment using containers
14:29:13 <coolsvap> using kolla
14:29:14 <fifieldt> like that iscsi one?
14:29:34 <coolsvap> neutron, cinder occassionally fail
14:29:43 <fifieldt> but it seems fixable?
14:29:49 <coolsvap> sometimes due to dependency updates and some issue are still sorted out
14:29:50 <coolsvap> yes
14:29:55 <fifieldt> ok, cool
14:29:56 <shamail> coolsvap: Is it implied that the conatiners are running in privileged mode?  Are we going to test in unpriv too (that might nake the # of services that can run containerized a bit less.
14:30:10 <fifieldt> ah, that might be it, yeah
14:30:33 <fifieldt> medberry's patch has that as a flag
14:30:43 <coolsvap> shamail, some containers need priv mode
14:30:57 <fifieldt> approx 50%, 75% 25%?
14:31:17 <jproulx> 'privileged' is an optional attribute in the proposed tag BTW
14:31:22 <fifieldt> aye
14:31:23 <coolsvap> fifieldt, i can give you exact update in a while
14:31:23 <shamail> Is that identified or would that be good data to add to tag? (Requires priv)
14:31:31 <shamail> Awesome
14:31:32 <fifieldt> that would be great coolsvap
14:31:51 <jproulx> privileged   - optional flag indicating that the service only works within a privileged container
14:31:57 <fifieldt> coolsvap,  would you be able to just dump it either on https://review.openstack.org/#/c/214801 or the ops mailing list - whatever works best for you?
14:32:04 <fifieldt> that would be great information to have if you;'ve got it
14:32:11 <fifieldt> since we can tell ops of the good work you'
14:32:11 <shamail> I got some reading to do
14:32:12 <fifieldt> ve
14:32:12 <coolsvap> sure will do
14:32:13 <fifieldt> being doing
14:32:15 <fifieldt> thanks!
14:32:22 <fifieldt> ok, we;'ve had a good discussion on this one
14:32:25 <fifieldt> what else is on the list?
14:32:36 <fifieldt> ops:ha https://review.openstack.org/#/c/200128/
14:32:48 <fifieldt> It looks like maish just needs to update that one based on the feedback
14:32:57 <fifieldt> but please do add your own feedback if you have not seen it yet
14:33:17 <shamail> Fifeldt: so i was thinking starting with Kolla images makes sense.  Can discuss further at summit.
14:33:23 <fifieldt> excellent
14:33:55 <fifieldt> ops:packaged (https://review.openstack.org/#/c/186633/ ) still has one dissenting view, but no achievable proposal to fix it
14:34:08 <fifieldt> I'm just about ready to add the basics for liberty for ops:packaged
14:34:16 <fifieldt> would be nice to get the tag description signed off before that :)
14:35:23 <shamail> How does this fit with the packaging discussion at the last ops summit?  Is this alluding to packages created by community or ones that are available already?
14:35:35 <fifieldt> basically just for available ones
14:35:36 <shamail> I apologize again, I got reading to do.  :)
14:35:45 <fifieldt> just giving a general state for each project
14:35:58 <shamail> That's a good starting point.  I'll review it soon.
14:36:02 <fifieldt> ok, cheers
14:36:13 <fifieldt> that rounds out our list
14:36:23 <fifieldt> would anyone like to add anything on any of the open reviews before we move on?
14:37:36 <fifieldt> going once ...
14:37:39 <fifieldt> going twice ...
14:37:44 <fifieldt> going three times ...
14:37:49 <fifieldt> and ..
14:37:59 <fifieldt> #topic Tag brainstorming
14:38:07 <fifieldt> shamail, you mentioned ideas!
14:38:11 <fifieldt> ideas are excellent :)
14:38:12 <shamail> Sure
14:38:21 <shamail> So I was thinking about the discussion on scale
14:38:35 <shamail> I think on the ML (or review maybe) it was deemed hard to capture
14:38:58 <shamail> So I wanted to propose an alternate: top50deployed, top20deloyrd
14:39:20 <shamail> Using data from the user survey, tag services that over 50% of the respondents are using
14:39:23 <shamail> Or 89%
14:39:29 <shamail> 80%*
14:39:49 <shamail> This would give three tiers of adoption data and adds a dimension to giving hints about maturity
14:40:05 <fifieldt> are we talking about scale here?
14:40:09 <shamail> That's was the idea
14:40:15 <j_king> definition of scale?
14:40:19 <shamail> Not really scale, more about deployment adoption
14:40:25 <j_king> better, ty.
14:40:26 <fifieldt> ok, so it's the user survey adoption data, unioned with like # of nodes or something
14:40:35 <fifieldt> ?
14:40:43 <shamail> Starting point being adoption data
14:40:51 <fifieldt> we have the adoption data as a tag already
14:40:54 <shamail> We could then discuss enhancements to add scale dimension with nodes etx
14:40:56 <fifieldt> let me link
14:41:07 <fifieldt> https://review.openstack.org/#/c/189168/
14:41:17 <fifieldt> if you look at https://review.openstack.org/#/c/189168/3/kilo/ops-production-use.json
14:41:22 <jproulx> perhaps is we define 'big site' we could get % of big sites using a project?
14:41:33 <fifieldt> aye
14:41:35 <shamail> Makes sense.
14:41:40 <fifieldt> but what is big site :)
14:41:52 <jproulx> exactly, talk to large doploymnets wg?
14:41:53 <shamail> We could use large deployment teams definitions
14:41:54 <fifieldt> is it # of nodes, # of TB, # of datacentres, # of users?
14:42:04 <shamail> They use # nodes
14:42:08 <j_king> amount of network traffic, latency, etc
14:42:29 <j_king> storage volume, throughput
14:42:40 <fifieldt> indeed
14:42:51 <fifieldt> so, two ways we can do this, I guess
14:43:10 <fifieldt> we can either come up with some thresholds based on the metrics we just listed
14:43:23 <fifieldt> or we can choose some deployments by name that "everyone" agrees are large scale
14:43:36 <jproulx> I think nodes is a reasonable proxy for most of this in the interest of simplification
14:43:53 <fifieldt> jproulx, but I have 256 core nodes :(
14:44:12 <shamail> I think Large Deploy starts at 1000 nodes today
14:44:12 <j_king> "nodes" is too abstract. :)
14:44:15 <fifieldt> (joking - just pointing out the challenge)
14:44:21 <shamail> j_king: agree
14:44:41 <fifieldt> ok, so what I'm seeing is we ask LDT for help with this ?
14:44:51 <shamail> fifieldt: +1
14:45:13 <fifieldt> cool
14:45:18 <fifieldt> seems like a way forward on that one?
14:45:20 <jproulx> well I do think we should refer to 'large deployment team' for consistency if at all possible
14:45:22 <shamail> On the adoption topic, I skimmed the existing tag... I guess what I was proposing would be an enhancement
14:45:54 <shamail> Instead of just capturing adoption %, we should build tiers 50, 80% so it is simpler to see a collection of services that meet a threshold.
14:45:55 <fifieldt> enhancements also good :)
14:46:12 <fifieldt> I think that we can do that in the presentation layer
14:46:17 <shamail> It's possible to build grouping with actual values but I don't know if exact values provide any additional value
14:46:18 <fifieldt> but keep the data as a continuum
14:46:27 <shamail> fifieldt: +1
14:46:42 <shamail> Wanted to keep it simple at presentation late yet
14:46:42 <fifieldt> it kinda leads to my pet topic for today
14:46:45 <shamail> Layer*
14:47:02 <fifieldt> which is if we want to make something at a higher layer, collating a bunch of things, to invent a term "maturity"
14:47:05 <fifieldt> what would we collate?
14:47:18 <fifieldt> we have *production usage, * # SDKs, * packaged, * install guide
14:47:35 <fifieldt> what else goes in that bucket of bits we can sum to some magic maturity level?
14:47:59 <shamail> Has diverse affiliation tag
14:48:12 <shamail> I assume maturity could use both ops and TC tags
14:48:19 <fifieldt> yup
14:48:53 <fifieldt> open floor here to include anything :) but we're the guys making the ops tags, so that's where I put more of my focus :)
14:49:08 <shamail> Are there metrics on bug triage or close average times?
14:49:26 <fifieldt> they exist
14:49:28 <shamail> How long issues are outstanding
14:49:34 <fifieldt> I am not sure how useful they are to be honest :(
14:49:42 <fifieldt> some projects leave bugs open for a looonnnggg time :)
14:49:45 <jproulx> to me 'mature' also implies a lower change velocity and something about live upgradability...not sure we capture either (upgrades especially important not so sure about change rate)
14:49:47 <fifieldt> worth investigating though
14:50:52 <fifieldt> upgradabilkity is an interesting one
14:51:02 <fifieldt> I put a call on the ML to see about that recently
14:51:05 <shamail> jproulx: rolling upgrade support could be tracked by whether projects have adopted things like versioned objects and RPC messaging
14:51:21 <fifieldt> medberry signed up, but I think he might need prodding
14:51:26 <jproulx> shamail +1
14:51:39 <fifieldt> this is true
14:51:48 <fifieldt> but what about "my database took 30 hours to run the migrations"
14:52:02 <j_king> 'mature' also means, to me, that if you put "scalable" on the tin then there are numbers to go along with it.
14:52:19 <j_king> do any operators share their telemetry or metrics?
14:52:47 <fifieldt> so like rax has 100-600 hosts per cell in nova
14:52:48 <jproulx> fifieldt is my database usable for the 30hr it's migrating?  if it's because it's doing lazy updates I don't care if it's off line I care very much
14:53:02 <fifieldt> exactly
14:53:05 <j_king> we've been finding interesting behaviours of certain services we can only find "at scale"
14:53:05 <shamail> j_king: none that I am aware of.. It would be interesting to have a data set to build a predictive model.
14:53:26 <j_king> race conditions and such
14:53:38 <shamail> jproulx: +1 and this is where being able to leverage an older scheme comes handy for services.
14:53:48 <shamail> Schema*
14:54:15 <fifieldt> I agree it would be nice to have but, we spend about an hour talking about a scalability tag last ops meetup and came to the conclusion to have a tag that sums it up
14:54:24 <fifieldt> "The end conclusion was that whether a service scales or not is extremely complex information and has an associated decision making process that means it isn't possible to condense into a tag or even a few tags. The only real way to do this is to provide information about example deployments (eg rax, cern) "
14:54:36 <fifieldt> so we can't talk about scalability directly as a single tag
14:54:42 <fifieldt> we can only infer it from some other things
14:54:52 <fifieldt> upgradability on the other hand is simpler to do
14:55:12 <shamail> yep and factors into the maturity umbrella
14:55:15 <j_king> fifieldt: agreed. it'd be nice to have some ballpark figures for various services.
14:55:40 <j_king> given their wide range of configurations it's impossible of course. :)
14:55:45 <fifieldt> aye :)
14:55:50 <shamail> I think building as much information into maturity is a good starting point.  If/when we solve the scalability tag challenge, it would be a nice addition to maturity.
14:55:57 <jproulx> I think large deployment utilization is best proxy we can get for scales at this time
14:56:18 <fifieldt> anyway, aside from scale and upgradability
14:56:20 <shamail> Sounds like an action item before the summit. :)
14:56:24 <fifieldt> anything else you can think of that is maturity?
14:56:31 <shamail> Someone needs to talk to LDT
14:57:00 <fifieldt> aye
14:57:06 <fifieldt> anyone want to sign up for that?
14:57:07 <j_king> anyone notice that the maintenance ML almost always reports build failures for stable packages?
14:57:17 * fifieldt does :|
14:57:43 <fifieldt> we have 3 minutes left in this meeting
14:57:52 <fifieldt> closing ideas? takeaway actions?
14:57:57 <shamail> diverse-affiliation, branching
14:57:58 <j_king> No build failures in X days!
14:58:08 <shamail> I'll think about additional ops tags for maturity
14:58:14 <fifieldt> cheers
14:58:16 <shamail> I'll take the action item
14:58:19 <fifieldt> #action shamail to think
14:58:22 <shamail> Talk to LDT
14:58:30 <fifieldt> #action shamail to talk to LDT
14:59:04 <fifieldt> ok
14:59:08 <fifieldt> well, thank you all very much
14:59:13 <fifieldt> I guess we'll see you on the mailing list
14:59:16 <j_king> thank you. :)
14:59:16 <fifieldt> post lots before the summit
14:59:20 <fifieldt> tell your friends about us :)
14:59:21 <shamail> 😬 Fifeldt. Thanks for the reminder, sometimes I forget to think and go into streaming mode.
14:59:25 <fifieldt> and keep cool!
14:59:39 <fifieldt> #endmeeting