17:09:37 <mrmartin> #startmeeting community
17:09:38 <openstack> Meeting started Mon Mar  2 17:09:37 2015 UTC and is due to finish in 60 minutes.  The chair is mrmartin. Information about MeetBot at http://wiki.debian.org/MeetBot.
17:09:39 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
17:09:41 <openstack> The meeting name has been set to 'community'
17:10:05 <mrmartin> I hope it will create the logs.
17:10:10 <reed> thanks mrmartin
17:10:24 <mrmartin> do we have an agenda?
17:10:48 <reed> yes, we need to start with a status report to understand where we're at with migration activity board to infra
17:11:10 <reed> the objective of the meeting is to set a roadmap (draft a spec)
17:11:12 <mrmartin> #topic status report of activity.o.o migration
17:11:22 <dizquierdo> oh welcome dpose, reed, mrmartin let me introduce dpose to you
17:11:30 <dizquierdo> he's puppetizing cvsanaly :)
17:11:30 <mrmartin> hi dpose
17:11:38 <mrmartin> sound so strange
17:11:53 <dpose> hi mrmartin
17:12:19 <reed> do we have a story for this task?
17:12:24 <mrmartin> how is activity.o.o had been deployed actually?
17:12:35 <dizquierdo> ok, let's start from the beginning
17:12:47 <dizquierdo> we do not have a story (if you mean in storyboard)
17:12:54 <dizquierdo> so we can open one there I guess
17:13:12 <dizquierdo> then, the current deployment
17:13:14 <mrmartin> I think if you don't have those assets, we need to setup everything, infra projects, storyboard, etc.
17:13:33 <dizquierdo> ok
17:13:47 <dizquierdo> well, the project is now under openstack-infra/activity-board
17:13:51 <dizquierdo> and this is under gerrit and git
17:13:57 <dizquierdo> but we're not using storyboard
17:14:04 <reed> # link https://storyboard.openstack.org/#!/project/709
17:14:06 <dizquierdo> this could be a good opportunity to start using this
17:14:19 <dizquierdo> great, thanks :)
17:14:22 <reed> #info we have the project Activity Board on infra
17:14:48 <dizquierdo> (oh sorry for not using meetbot tags, I still have to learn u_u)
17:14:51 <reed> no we don't
17:15:23 <reed> yes we do... I'm so confused, been pinged from too many places already and it's only 9am here :)
17:15:24 <mrmartin> #info we have the project Activity Board on Storyboard instead
17:16:00 <reed> so I think we should use this meeting to lay down a roadmap, draft a spec
17:16:14 <dizquierdo> that sounds good
17:16:16 <mrmartin> ok, so we have a storyboard account, a git repo and some gerrit integration
17:16:29 <dizquierdo> I can detail if you want how we currently deploy activity board
17:16:37 <mrmartin> please.
17:16:38 <dizquierdo> and then we may prepare that plan
17:16:46 <dizquierdo> ok, briefly
17:17:05 <dizquierdo> there are 3 main legs: retrieval process, metrics and analytics side, and visualization
17:17:28 <dizquierdo> the first step is python-based and goes through all of the data sources (irc, git, mailing lists, storyboard, etc) and retrieve all of the info
17:17:36 <dizquierdo> that stores all of the information in MySQL databases
17:17:46 <mrmartin> is it some scheduled process?
17:17:54 <dizquierdo> each 24 hours this is run
17:17:58 <dizquierdo> using cron
17:18:00 <mrmartin> ok.
17:18:08 <dizquierdo> (indeed the whole process)
17:18:20 <dizquierdo> the second part basically goes through all of the databases and start producing JSON files
17:18:45 <dizquierdo> then, those JSON files are visualized by the platform, and this is what you see in activvity.o.o
17:18:59 <dizquierdo> this is a basic a simple approach to the whole platform
17:19:20 <dizquierdo> as we were discussing during FOSDEM, a good starting point would be to start migrating the first part: the retrieval tools
17:19:39 <dizquierdo> so far, there's nothing created under the openstack infrastructure, but the visualization
17:20:08 <dizquierdo> mrmartin, shall I go a bit deeper into the first part: the retrieval tools?
17:20:17 <mrmartin> yes
17:20:20 <dizquierdo> ok
17:20:30 <mrmartin> https://github.com/openstack-infra/system-config/blob/master/manifests/site.pp
17:20:32 <reed> #info draft spec for this process https://etherpad.openstack.org/p/activity-infratization-spec
17:20:46 <mrmartin> I'm checking this meanwhile, but I cannot found any servers related to activity.o.o
17:20:47 <dizquierdo> this is based on Metrics Grimoire https://github.com/MetricsGrimoire
17:20:48 <reed> geez, it took me 5 mins to find the template spec
17:21:04 <dizquierdo> and there's a tool for each data source
17:21:13 <dizquierdo> for instance, cvsanaly is focused on git activity
17:21:19 <reed> mrmartin, the current server was created manually
17:21:25 <mrmartin> ok.
17:21:26 <dizquierdo> Mailing List Stats is focused on mailing lists
17:21:32 <reed> it's not a puppet server
17:21:44 <dizquierdo> Bicho is focused on Gerrit and Launchpad
17:21:49 <dizquierdo> (and lately storyboard)
17:21:51 <dizquierdo> etc
17:22:07 <dizquierdo> and those tools are installed by cloning the repo plus some standard libraries
17:22:21 <mrmartin> do you have some packaging for python apps, like pip or deb ?
17:22:25 <dizquierdo> then, there's a list of options for each tool, like the URL of the service
17:22:33 <dizquierdo> mrmartin, I'm afraid we do not have that
17:22:55 <dizquierdo> but we may start producing those if that's easier for you
17:23:22 <mrmartin> so for example, you are using an app for Mailing list stats, do you deploy multiple apps with different configurations
17:23:27 <dizquierdo> in the initial puppet recipe that was prepared by dpose, we cloned the repo and installed from sources (just for information)
17:23:40 <mrmartin> or you have single app, and use a monolitic config?
17:24:16 <dizquierdo> mrmartin, afair this is all monolitic config
17:24:22 <mrmartin> ok.
17:25:08 <mrmartin> do you have a git repo for the puppet dpose is working on?
17:25:12 <dizquierdo> and well, as a first approach, it may be interesting to move the retrieval process under openstack
17:25:31 <dizquierdo> mrmartin, not yet, we can publish that for sure
17:25:45 <dizquierdo> some place under infra?
17:25:51 <dizquierdo> or maybe personal github account
17:26:12 <mrmartin> ok, because it is a question, whether you like to maintain the puppet module, or move it under infra?
17:26:42 <mrmartin> the infra's advantage that we can use the same toolset, as we are using for all other projects, including code review etc.
17:26:44 <dizquierdo> probably to move that under infra
17:26:59 <dizquierdo> I think that's the idea, reed ?
17:27:09 <dizquierdo> so anyone is more than welcome to participate :)
17:27:09 <mrmartin> and then here, we need to do our fight with infra with let in this puppet
17:27:41 <dizquierdo> mrmartin, my main concern is that sometimes we need to reorganize part of the MySQL databases
17:27:48 <dizquierdo> based on features in the retrieval tools
17:27:51 <mrmartin> what it means?
17:27:58 <mrmartin> schema changes?
17:28:01 <dizquierdo> some example: we need either to add a new column to the database for some reason
17:28:04 <dizquierdo> yes, things like that
17:28:20 <mrmartin> can we automatize that?
17:28:21 <dizquierdo> and I'm not sure if that's an issue
17:28:29 <dizquierdo> I'd say so
17:28:45 <reed> I think we can maintain the puppet module, it scales better than github pull requests
17:28:47 <mrmartin> will this come as a part of the update?
17:28:50 <reed> the problem is the CLA
17:29:07 <reed> if you don't want to use the CLA then we can put it on stackforge
17:29:28 <dizquierdo> reed, what do you foresee as an issue wrt the CLA?
17:29:30 <reed> if you don't want to use stackforge, put it on github in your grimoire set
17:29:32 <mrmartin> what is the license of those tools?
17:29:40 <dizquierdo> GPLv3
17:29:41 <reed> dizquierdo, it's annoying as hell
17:29:43 <mrmartin> oh
17:29:47 <dizquierdo> ah ok reed
17:29:50 <reed> mrmartin, that's not an issue
17:30:24 <reed> probably github on your repos is the easiest option, since you already use it
17:30:31 <mrmartin> as I know the puppet-modules under infra will be automagically published as puppet modules on forge in the near future
17:30:40 <mrmartin> that's the plan
17:30:41 <dizquierdo> sure, from our side that's not an issue
17:31:15 <mrmartin> but if you like to maintain your own puppet repo / handle module, it can work too
17:31:37 <mrmartin> the openstack way have the advantage that we don't need to reinvent the wheel regarding some processes and toolsets
17:31:37 <dizquierdo> so reed  your opinion is that we should maintain the puppet repo under our github account
17:31:48 <dizquierdo> I see mrmartin
17:32:02 <mrmartin> but it is a decision, that must be made
17:32:02 <reed> mrmartin, I leave that decision up to you, I don't have a strong opinion
17:32:18 <reed> dizquierdo, are you confortable with the CLA?
17:32:26 <reed> you've already signed it, right?
17:32:32 <dizquierdo> for sure, as first step, we still need to play a bit, so we may have first versions under github, and later migrate to infra
17:32:37 <mrmartin> we can start with a separate github repo, and move-in to openstack-infra if it is mature enough
17:32:37 <dizquierdo> reed, yes I signed that
17:32:46 <dizquierdo> yes, that sounds like a plan mrmartin
17:32:53 <reed> the other thing is that creating a new repo on infra will require weeks, while a new github repo is immediate
17:33:06 <reed> yeah, let's do that
17:33:10 <dizquierdo> yep
17:33:14 <reed> #info  we can start with a separate github repo, and move-in to openstack-infra if it is mature enough
17:33:37 <mrmartin> reed: event if it not takes for weeks, patch acceptance have some delivery time depending on the load of infra
17:33:40 <mrmartin> event / even
17:34:13 <reed> #the github repos will contain the puppet modules to deploy and configure all grimoire toolsets (cvsanaly, mailing list stats, bicho, etc)
17:34:16 <dizquierdo> then, as a first action, we'll create that github repo
17:34:30 <dizquierdo> and send to you an email
17:34:40 <dizquierdo> mrmartin, we may need some help from your side
17:35:09 <dizquierdo> I hope we wont' be too annoying
17:35:37 <mrmartin> reed, is the part of the OpenStack Foundation project?
17:36:01 <mrmartin> so this migration project
17:36:38 <mrmartin> dizquierdo, how much puppet experience your team have?
17:37:01 <dizquierdo> mrmartin, low I'm afraid
17:37:04 <dizquierdo> beginners
17:37:22 <mrmartin> ok, so I'm used the puppetize using the following process:
17:37:37 <mrmartin> 1, write some spec about architecture, components, config required
17:38:05 <mrmartin> 2, do some detailed deployment guide, that explains everything step by step, so if I give it to a linux sysadmin he can do that
17:38:13 <mrmartin> 3, if all above works, puppetize the result
17:38:20 <reed> mrmartin, what's your question again? not sure I understand it
17:38:22 <dizquierdo> ok
17:38:58 <dizquierdo> we can start in that way
17:39:16 <mrmartin> so if point 2 works well, we can identify some pain points the can cause problems during automatization
17:39:28 <mrmartin> for example this schema change
17:39:51 <mrmartin> or the deployment / packaging questions
17:39:55 <dizquierdo> ok, mrmartin we'll also add a list of potential issues we typically face when updating the toolset
17:39:56 <mrmartin> or how-to upgrade things
17:40:05 <dizquierdo> yes, so we can deal with those in an automated way
17:40:13 <mrmartin> update used to be a major problem usually
17:40:24 <dizquierdo> they're quite stable, but well...
17:40:25 <dizquierdo> there are updates
17:41:23 <mrmartin> ok, so first of all I suggest to write the initial specs
17:41:39 <mrmartin> setup the required github repos and accesses
17:41:59 <mrmartin> and when we have a spec, we can estimate some delivery time
17:42:11 <dizquierdo> mrmartin, ok
17:42:17 <mrmartin> and do regular meetings about the progress
17:42:32 <mrmartin> and start to use storyboard
17:42:38 <dizquierdo> sure
17:43:01 <mrmartin> reed, do you know when will storyboard email notifications start to work?
17:43:21 <reed> nope
17:43:55 <mrmartin> #action write activity board migration spec
17:44:15 <reed> # info draft available on https://etherpad.openstack.org/p/activity-infratization-spec
17:44:29 <reed> #info draft available on https://etherpad.openstack.org/p/activity-infratization-spec
17:44:39 <mrmartin> reed, do we need to create this spec under infra specs?
17:44:52 <reed> I think so, rigth?
17:44:57 <mrmartin> ok, I agree
17:45:04 <reed> it's a similar process to askbot infratization
17:45:28 <mrmartin> than the same story here, let's use the etherpad for collaborative work, to create a basic spec
17:45:40 <dizquierdo> mrmartin, http://paste.openstack.org/show/185288/
17:45:42 <mrmartin> and when it is in a good shape, push it through infra as an official spec
17:45:42 <reed> the thing is that we need to highlight in the spec that it's a long road and that we start small
17:45:49 <dizquierdo> these are our first steps with Puppet
17:45:52 <dizquierdo> (as an example)
17:46:39 <mrmartin> I see, they should kill you asap due missing permission in file resources ;)
17:46:47 <mrmartin> but looks promising
17:46:48 <reed> no requirements.txt?
17:47:09 <mrmartin> ok, but first, let's focus on the spec
17:47:16 <reed> dizquierdo, is repositoryHandler in pip?
17:47:19 <dizquierdo> mmm we files that were created were init.pp, nodes.pp and site.pp
17:47:21 <dizquierdo> reed, nop
17:47:23 <reed> I agree, let's focus on the spec
17:47:25 <dizquierdo> ok
17:47:39 <mrmartin> I suggest to create some minimal architecture diagram, that shows the basics
17:48:02 <mrmartin> and explain all of the modules required, and highlight if some special configuration neeeded.
17:48:15 <reed> mrmartin, good idea...
17:48:15 <dizquierdo> ok
17:48:32 <reed> dizquierdo, do you have already a diagram for the grimoire tools?
17:48:32 <mrmartin> #info create basic architecture diagram
17:48:57 <dizquierdo> reed, emm if you mean in terms of a picture, I think we have it around
17:49:11 <dizquierdo> if you mean in terms of specifications in a text file, I'm afraid not
17:49:18 <reed> yes, that's what I mean, I'm sure you have a picture somewhere
17:49:27 <dizquierdo> I'll look for that
17:49:55 <mrmartin> a simple text diagram is enough
17:49:57 <reed> #action dizquierdo to look for a diagram (picture) of grimoire architecture to illustrate all the pieces
17:50:59 <mrmartin> I'll try to find an example I made for one project
17:51:20 <mrmartin> this one: https://review.openstack.org/#/c/140043/11/doc/source/askbot.rst
17:51:22 <dizquierdo> oh that's perfect, we can modify that according to the grimoire toolset
17:51:52 <mrmartin> ok, and I suggest to do regular meetings
17:52:01 <mrmartin> are you guys all located in EU?
17:52:03 <dizquierdo> mrmartin, sure
17:52:06 <dizquierdo> mrmartin, we are
17:52:17 <dizquierdo> not reed
17:52:36 <reed> I'm ok with taking this slot for community infra meeting
17:52:39 <mrmartin> yes, I feel his problem in SFO ;)
17:52:45 <mrmartin> this time is ok for me too.
17:52:53 <reed> i have no problem, I can also do one hour earlier
17:53:05 <mrmartin> this 18.00 o'clock was better for me
17:53:16 <dizquierdo> this time is ok, and even later around 23 pm CET is ok for me
17:53:21 <reed> shall we set the meeting in the #openstack-meeting calendar?
17:53:26 <reed> and take this channel?
17:53:38 <dizquierdo> ok
17:53:39 <mrmartin> yes, we can add this on wikipage
17:53:42 <reed> mrmartin, we end up syncing on other issues as well on MOndays anyway :)
17:54:21 <reed> #action reed to edit Meetings page and take #openstack-meeting-alt at 1700 UTC for #community meeting
17:54:22 <mrmartin> so this channel is available until 19.00GMT (20.00CET)
17:54:39 <mrmartin> 17.00 UTC is perfect
17:54:58 <mrmartin> ok, dizquierdo You can reach me during working hours also
17:54:59 <dizquierdo> it is
17:55:11 <mrmartin> and we can share the results with reed on weekly meetings
17:55:12 <dizquierdo> mrmartin, ok, I'm also around during working hours
17:55:13 <dizquierdo> thanks
17:55:21 <dizquierdo> thanks in advance for your time mrmartin
17:55:54 <mrmartin> ok, I guess the initial tasks are clear
17:56:21 <dizquierdo> yes
17:56:23 <reed> cool
17:56:39 <reed> thanks folks, I think we have a path forward. Let's meet again next week
17:56:48 <dizquierdo> thanks mrmartin  and reed
17:56:59 <mrmartin> ok, so the next meeting will be on March 09, 17.00GMT
17:57:00 <reed> we'll try to keep it shorter and keep on working offline, on community list
17:57:19 <reed> I'll be at the Operators Summit, I may be 'distracted' :)
17:57:24 <reed> but let's do it
17:57:32 <dizquierdo> :)
17:57:45 <mrmartin> yeah, we are travelling sometimes
17:57:59 <mrmartin> #endmeeting