17:01:21 <mrmartin> #startmeeting community
17:01:22 <openstack> Meeting started Mon Mar  9 17:01:21 2015 UTC and is due to finish in 60 minutes.  The chair is mrmartin. Information about MeetBot at http://wiki.debian.org/MeetBot.
17:01:22 <reed> dizquierdo, o/
17:01:23 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
17:01:25 <openstack> The meeting name has been set to 'community'
17:01:37 <mrmartin> #topic actions from last meeting
17:01:39 <reed> #link agenda https://wiki.openstack.org/wiki/Community/MeetingAgenda
17:01:43 <dizquierdo> hey
17:02:00 <dizquierdo> dpose is also around
17:02:03 <reed> write activity board migration spec
17:02:22 <dpose> hi
17:02:23 <reed> #info What's the status of  the activity board migration spec?
17:02:28 <reed> #undo
17:02:44 <sarob> hey there
17:02:54 <mrmartin> hey Sean
17:03:04 <dizquierdo> ok, what we have right now is the following repository
17:03:07 <dizquierdo> https://github.com/MetricsGrimoire/puppet-metricsgrimoire
17:03:14 <reed> #info the draft of the spec is on https://etherpad.openstack.org/p/activity-infratization-spec
17:03:33 <mrmartin> #info puppet repository https://github.com/MetricsGrimoire/puppet-metricsgrimoire
17:03:46 <dizquierdo> before writting the step, we went for the architecture file as suggested by mrmartin
17:04:07 <mrmartin> dizquierdo, are you working on the architecture diagram, right?
17:04:12 <dizquierdo> #info architecture definition at https://raw.githubusercontent.com/MetricsGrimoire/puppet-metricsgrimoire/master/architecture.rst
17:04:23 <dizquierdo> that's right mrmartin, please have a look at https://raw.githubusercontent.com/MetricsGrimoire/puppet-metricsgrimoire/master/architecture.rst
17:04:45 <dizquierdo> I tried to follow the example you provided to us based on askbot
17:04:58 <mrmartin> ok
17:05:41 <dizquierdo> and then we uploaded the first steps with Puppet and one of the tools mentioned in the architecture
17:05:43 <mrmartin> #info architecture info: https://raw.githubusercontent.com/MetricsGrimoire/puppet-metricsgrimoire/master/architecture.rst
17:06:05 <mrmartin> reed to edit Meetings page and take #openstack-meeting-alt
17:06:05 <mrmartin> at 1700 UTC for #community meeting
17:06:09 <mrmartin> it was also done.
17:06:17 <reed> yep
17:06:18 <dizquierdo> https://github.com/MetricsGrimoire/puppet-metricsgrimoire/tree/master/CVSAnalY
17:06:44 <reed> what are next steps then?
17:06:46 <mrmartin> ok, so we have a good progress with that
17:06:57 <dizquierdo> good to know we're in the good path
17:06:58 <mrmartin> a question, can I get an ssh access to activity.o.o?
17:07:19 <mrmartin> helps for me to understand the pieces of actual deployment.
17:07:20 <dizquierdo> mrmartin, we're only publishing JSON, HTML, CSS and JS there
17:07:29 <dizquierdo> so that's the really final step of the whole platform
17:07:41 <mrmartin> ok, but where the metrics apps running?
17:07:43 <dizquierdo> all of the retrieval process + analytics is in our internal servers
17:07:49 <mrmartin> oh, ok right
17:08:09 <dizquierdo> we simply rsync to activity.o.o
17:08:23 <fungi> oh, sorry, missed the meeting start time but here now
17:08:24 <mrmartin> so the goal is to move those metrics and analytics apps related to openstack to a separate instance
17:08:33 <mrmartin> hi fungi
17:08:37 <dizquierdo> hi :)
17:08:44 <dizquierdo> the final goal is that one
17:08:51 <dizquierdo> the first step is to migrate the retrieval platform
17:08:53 <mrmartin> fungi: we are talking about moving the metrics part of activity.o.o to infra
17:09:09 <fungi> thanks, just caught up on scrollback
17:09:11 <mrmartin> becuase it is actually lives in bitergia servers.
17:09:18 <dizquierdo> yep
17:09:33 <mrmartin> #topic migrating activity.o.o metrics part to infra
17:10:02 <mrmartin> #info metrics apps are living at bitergia servers actually
17:10:15 <dizquierdo> so, after moving this first step, I have some doubts mrmartin, I may try to condense some of them here and try to look for answers
17:10:57 <mrmartin> ok, I'm listening
17:11:01 <dizquierdo> some examples: what do we do if we need to update databases schemas?
17:11:14 <dizquierdo> how should we proceed if we need to update tools versions?
17:11:41 <dizquierdo> How can we have an access to such databases? -> we query them later to retrieve the final JSON files
17:11:52 <dizquierdo> (an option would be to dump them at some place)
17:12:01 <dizquierdo> we need to run some of the tools in some order:
17:12:07 <dizquierdo> First: all of the retrieval tools
17:12:33 <dizquierdo> Second: the unique identities heuristics script, that allows to merge identities across all of the data sources (git, mailing lists, storyboard, etc)
17:12:45 <dizquierdo> we need to run the tools each 24 hours
17:13:31 <dizquierdo> and at some point, we need to update affiliations information, so let's say that we need to access the database, update info and close the connection
17:13:44 <fungi> so something like continuously deploying the tools, or at least updating them before each metric run?
17:13:48 <dizquierdo> (this is going to be done through a tool in the near future, but this is stilli n progress)
17:13:55 <mrmartin> for the schema upgrades we are doing something similar with the groups portal, and using a green / blue deployment model with a slot0 / slot1 switched between each upgrades
17:14:00 <dizquierdo> and that's all
17:14:08 <dizquierdo> emm that's it fungi
17:14:16 <mrmartin> currently it is implemented on the same instance with some scripting magic
17:14:22 <reed> dizquierdo, I think we should postopone the conversation until after you have produced a visualization of the whole machinery
17:14:37 <dizquierdo> they are quite stable in any case, I do not foresee more than 1 change per month? at most!
17:14:45 <fungi> we definitely have existing infra systems with tools continuously deployed to them from various git repositories (tracking a particular branch or whatever)
17:15:00 <reed> the grimoire toolset is fairly simple in the end but it's made of a lot of small pieces and it's hard to communicate
17:15:14 <dizquierdo> I agree with you reed
17:15:25 <fungi> generally they check for new git commits every ~15 minutes and update if found, which can trigger actions like rerunning installation scripts
17:15:58 <dizquierdo> fungi, that sounds like more than useful for this case, so we have an answer for that question :)
17:16:05 <fungi> so definitely doable if you need something like that
17:16:09 <dizquierdo> reed, do you think I should keep working on the architecture side?
17:16:35 <reed> dizquierdo, I think so, it's useful for you too IMHO to have one picture that shows how the whole grimoire set works
17:16:53 <reed> in general, not just the implementation of openstack
17:17:01 <dizquierdo> ok, then reed I should update this file https://raw.githubusercontent.com/MetricsGrimoire/puppet-metricsgrimoire/master/architecture.rst
17:17:19 <mrmartin> dizquierdo: do you plan some packaging (pip, etc) for those apps?
17:17:54 <dizquierdo> mrmartin, we typically install them from sources, but this is in our roadmap
17:18:02 <dizquierdo> they are all setuptools based
17:18:21 <mrmartin> yes, because using live git repos is the best way to broke production system.
17:18:30 <dizquierdo> I know u_u
17:18:35 <reed> fungi, cvsanaly is packaced in pip
17:19:24 <dizquierdo> reed,  that's maybe an old version and not maintained package :S
17:19:29 <mrmartin> ok, so I think we need to focus on finishing the spec in some form, and get a deployment guide
17:19:34 <dizquierdo> not sure, I would need to double check that
17:19:37 <fungi> in that case, we deal with that fine too (just make the puppet manifest specify pip as the package provider with a version of "latest")
17:19:39 <mrmartin> so we can start to write the puppet scripts based on that
17:20:03 <fungi> oh, but yeah i'm assuming of course that it's being properly released to pypi ;)
17:20:09 <dizquierdo> :)
17:20:16 <reed> mrmartin, I agree on the priority: finish the spec first, get a clear idea of what needs to happen, maybe split it into different phases
17:20:32 <dizquierdo> ok, I can work then in the https://etherpad.openstack.org/p/activity-infratization-spec file
17:20:40 <dizquierdo> and keep working on the architecture file
17:20:47 <reed> cool
17:21:05 <mrmartin> fungi: for continous deployment, don't we like to use a PaaS solution?
17:21:12 <dizquierdo> just as a brief introduction, the first step is the one that we wrote in the architecture.rst file
17:21:20 <reed> #action dizquierdo and dpose to keep working on the https://etherpad.openstack.org/p/activity-infratization-spec file and the architecture diagram
17:21:34 <mrmartin> we could use it for other projects too, and can separate the different versions better
17:22:06 <reed> mrmartin, we can chat about that some other time :)
17:22:25 <reed> anything else on this topic?
17:22:27 <dizquierdo> ok, so two clear tasks from our side :)
17:22:28 <mrmartin> the problem here, that it is not enough to deploy a newer pip, some db schema change / or triggering other task is required
17:22:43 <dizquierdo> mrmartin, not sure if you want us to keep adding some other puppet recipes to the repository
17:22:48 <mrmartin> ok, so I suggest to close this, and we can join in with puppet scripting
17:22:50 <dizquierdo> this may help you, just in case
17:23:07 <mrmartin> I'll review that.
17:23:18 <reed> next topic?
17:23:21 <dizquierdo> ok mrmartin
17:23:35 <mrmartin> it could be easier to check what we have with an ssh access, but I understand that it is deployed to your internal servers.
17:23:53 <dizquierdo> #action dpose and dizquierdo to keep working on the puppet recipes at https://github.com/MetricsGrimoire/puppet-metricsgrimoire
17:23:58 <dizquierdo> mrmartin, let me discuss that internally
17:24:04 <dizquierdo> I think it's hard ;)
17:24:05 <dizquierdo> but well
17:24:23 <reed> mrmartin, I don't udnerstand why you need ssh access to run a sql script
17:24:58 <mrmartin> I don't need that to run an sql script, it could be helpful to overview the metrics components.
17:25:10 <fungi> mrmartin: no idea how you're defining a paas-based continuous deployment vs something else
17:25:42 <dizquierdo> mrmartin, I'm considering the option of having a virtual machine around, that you can have access
17:25:50 <dizquierdo> so you (or others) can have a look at it
17:26:12 <mrmartin> fungi: exactly the same model as we are using to deploy into a slot0 / slot1 directory, but deploy into separate instances.
17:26:30 <dizquierdo> indeed, mrmartin I think we have a vagrant machine that you can download and play with that to check how that works
17:26:33 <mrmartin> the puppet recipes are not so nice due this slot change.
17:26:42 <mrmartin> dizquierdo: awesome, this is perfect
17:26:51 <dizquierdo> ok, that sounds like an option
17:26:52 <dizquierdo> let me check
17:27:12 <reed> let's leave these details to the spec
17:27:17 <dizquierdo> #action dizquierdo will check the vagrant machine to have a look at the internal machinery of grimoire toolset
17:27:36 <reed> and I think you guys have enough work for a week already :)
17:27:36 <fungi> mrmartin: ahh, no idea if that's preferred or not honestly. we have a lot of different deployment models we've been supporting so far
17:27:42 <dizquierdo> :)
17:28:30 <reed> mrmartin, shall we switch to next topic?
17:28:38 <mrmartin> yes if we have
17:29:01 <reed> the agenda has: Update about Askbot migration to infra (mrmartin)
17:29:06 <mrmartin> oh right
17:29:31 <reed> #topic Update about Askbot migration to infra (mrmartin)
17:29:44 <reed> you need to do it
17:30:03 <mrmartin> ok, I need to talk Jeblair, he promised an approval on the askbot patch.
17:30:09 <mrmartin> fungi: are you back from holiday?
17:30:23 <mrmartin> could you help us to launch this askbot instance?
17:30:26 <fungi> mrmartin: yep, just catching up now
17:30:53 <mrmartin> it could be great to move forward with this during that week, because on next week I'll be on travel.
17:31:10 <reed> how about adding an action item to the infra agenda meeting tomorrow?
17:31:18 <mrmartin> reed: it is there
17:31:22 <mrmartin> as a priority project
17:31:26 <fungi> mrmartin: yeah, i just need to look back over https://review.openstack.org/140043 and see if there's anything else missing besides what needs to be added to hiera
17:31:46 <mrmartin> fungi: I've added some items to the storyboard of askbot.
17:32:00 <mrmartin> so we can track the progress there
17:32:21 <mrmartin> https://storyboard.openstack.org/#!/story/2000158
17:33:12 <mrmartin> fungi: and the db migration description available here: https://review.openstack.org/#/c/160693/
17:34:08 <mrmartin> #info db migration spec extension: https://review.openstack.org/#/c/160693/
17:34:20 <mrmartin> #info askbot migration storyboard tasks: https://storyboard.openstack.org/#!/story/2000158
17:34:28 <mrmartin> anything else on askbot?
17:34:34 <reed> #link https://review.openstack.org/#/c/160693
17:34:41 <reed> #link https://storyboard.openstack.org/#!/story/2000158
17:34:53 <reed> so the action item is to nudge Infra?
17:35:03 <mrmartin> politely
17:35:11 <fungi> mrmartin: reed: consider me nudged
17:35:20 <fungi> i'll try to work it in this afternoon
17:35:26 <reed> fungi, I was about to ask if this was enough :)
17:35:26 <mrmartin> thanks
17:35:28 <reed> thanks
17:35:41 <reed> so #action check back with fungi in a couple of days?
17:36:05 <mrmartin> #topic Catchup on Groups portal work items
17:36:07 <fungi> yep
17:36:09 <reed> #action check back with fungi in a couple of days
17:36:21 <reed> #action reed to check back with fungi in a couple of days
17:36:33 <reed> alright, groups portal items
17:36:33 <mrmartin> ok, groups portal, 'cause we'll run out of time
17:36:51 <mrmartin> ok, I did a correction of the group membership data
17:37:07 <reed> mrmartin, we can keep it short because I have the Ops summit to attend to
17:37:14 <mrmartin> we had some problem with Israel, meetup.com reported a wrong number
17:37:23 <reed> I just wanted to bring to your attention  https://storyboard.openstack.org/#!/story/2000193
17:37:27 <mrmartin> I wrote an email to Nati to provide me the valid numbers
17:37:56 <mrmartin> yes, I found some small issues with event import
17:38:15 <reed> the start and end date seem to go into the html
17:38:29 <reed> which could be fine
17:38:34 <mrmartin> for example, if the event don't have an assigned venue, it is defaults to US, which is a bit confusing
17:39:06 <mrmartin> and I suggest to disable of the notifications coming from meetup.com imported events
17:39:20 <reed> so I have asked tipit to import the events from groups into openstack.org/community/events
17:39:35 <reed> and they noticed that there are no xml nodes for start/end date
17:39:37 <mrmartin> because I got a double notification, one from meetup.com and one from groups.portal
17:39:50 <mrmartin> reed, ok I'll check that, it is a Commons thing
17:40:00 <reed> later they noticed that the html has start-date and end-date
17:40:18 <mrmartin> need to investigate that
17:40:28 <reed> please have a look at the data in that xml and let me know if I should tell tipit to use the html in there
17:40:47 <mrmartin> if the html have the data, I can put it somehow into the xml
17:40:50 <reed> or if it's better/easier/cleaner to modify groups
17:41:09 <reed> mrmartin, whatever is cleaner, I wouldn't want to spend too much time on this task
17:41:23 <mrmartin> #action add start-date end-date to event xml, so tipit can consume it
17:41:43 <reed> I think that if the html is clean and consistent then we can keep the groups side as it is
17:41:51 <reed> #link https://storyboard.openstack.org/#!/story/2000193
17:41:59 <mrmartin> I think it is easier to parse the xml
17:43:03 <mrmartin> ok, I'm doing a report about the group completeness status
17:43:17 <mrmartin> so we can see where we have a missing organizer registration, meetup.com link, etc.
17:43:33 <mrmartin> not a big thing, but helps us to clean up the data sources
17:43:42 <reed> ok
17:43:46 <mrmartin> and we have a new group request for Argentina
17:43:56 <reed> let's move this to the list
17:43:59 <reed> I have to go now
17:44:12 <mrmartin> ok
17:44:28 <mrmartin> any other questions?
17:44:37 <reed> no
17:44:56 <mrmartin> thanks to everyone
17:45:03 <mrmartin> #endmeeting