17:01:43 <reed> #startmeeting community
17:01:44 <openstack> Meeting started Mon Mar 23 17:01:43 2015 UTC and is due to finish in 60 minutes.  The chair is reed. Information about MeetBot at http://wiki.debian.org/MeetBot.
17:01:45 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
17:01:48 <openstack> The meeting name has been set to 'community'
17:02:04 <dizquierdo> here
17:02:06 <mrmartin> hi
17:02:11 <reed> #link https://wiki.openstack.org/wiki/Community/MeetingAgenda#Agenda_for_March_23.2C_2015
17:02:18 <mrmartin> sorry for the last week's meeting, I was on a plane, and forget to tell
17:02:28 <reed> #topic Infratization of Activity Board
17:02:47 <reed> no problem mrmartin, we moved forward via mailing list and in person anyway :)
17:03:07 <mrmartin> yeah, was awesome
17:03:21 <reed> dizquierdo, what's the status of https://etherpad.openstack.org/p/activity-infratization-spec?
17:03:25 <mrmartin> let's see the activity spec
17:03:46 <dizquierdo> mrmartin, reed I finished a version of the architecture file at https://raw.githubusercontent.com/MetricsGrimoire/puppet-metricsgrimoire/master/architecture.rst
17:04:02 <mrmartin> ok, nice, can you move it to the spec file?
17:04:17 <mrmartin> or just add a reference link there.
17:04:18 <dizquierdo> ok, if that's good enough, I'll do it
17:04:35 <reed> dizquierdo, does ircanalysis store data in mysql?
17:04:45 <reed> and mlstats, too
17:04:46 <dizquierdo> reed, yes it does
17:04:49 <reed> right?
17:04:52 <dizquierdo> all of the tools indeed
17:05:24 <reed> ok, in the architecture rst file some tools explicitly say "This is later stored in a MySQL database."
17:05:34 <reed> and mlstats, ircanalysis don't state that
17:05:39 <dizquierdo> ok, I'll update that
17:05:41 <reed> it's a minor bug :)
17:05:54 <dizquierdo> all of the tools have as output a mysql database
17:05:56 <reed> because the schema later is explicit
17:06:16 <reed> ok
17:06:32 <reed> that file is a bit hard to read but it's good enough
17:06:44 <reed> i'll try to make an image
17:06:49 <dizquierdo> sorry about that, I can try to improve the reading
17:06:53 <reed> now, the specification
17:07:05 <reed> dizquierdo, don't worry, the image is the only thing that will make it clearer
17:07:07 <reed> :)
17:07:12 <dizquierdo> ah ok
17:07:19 <dizquierdo> I started to produce that, but well u_u
17:07:43 <reed> so, is the draft spec ready to be proposed?
17:07:49 <dizquierdo> I'd say so
17:08:02 <mrmartin> so we can move a short one-two lines of description to the components
17:08:34 <mrmartin> I guess this spec is around 10% ready, the infra would kill us if try to commit that in this form
17:08:40 <reed> :)
17:08:45 <dizquierdo> oops, ok :)
17:09:18 <mrmartin> so if the architecture part is ready, need to move forward on, what puppet components we need, what is the proper way of migration
17:09:21 <mrmartin> like we did with askbot
17:09:46 <mrmartin> how to backup / restore databases, etc.
17:09:57 <reed> we don't need any of that, I thin
17:10:05 <reed> we can start from scratch
17:10:13 <reed> actually, we have to start from scratch
17:10:29 <dizquierdo> we should take care with all of the work regarding affiliation matching
17:10:32 <mrmartin> but we don't need to move-in the existing data?
17:10:37 <reed> so, problem description: is that enough?
17:10:44 <reed> mrmartin, no, we can recreate all of that
17:10:44 <mrmartin> or the plugin collects that froms scratch?
17:10:48 <mrmartin> ok
17:10:51 <mrmartin> I got it
17:11:48 <mrmartin> $
17:11:54 <mrmartin> sry
17:12:14 <mrmartin> #info move architecture reference to spec
17:12:38 <reed> #action reed to prepare a visual diagram based on dizquierdo's rst file
17:12:54 <reed> is Problem description clear enough?
17:13:00 <dizquierdo> oh, with this respect
17:13:01 <reed> https://etherpad.openstack.org/p/activity-infratization-spec ?
17:13:09 <dizquierdo> mrmartin,  did you have the cance to have a look at the vagrant machine?
17:13:14 <dizquierdo> https://github.com/VizGrimoire/VizGrimoireUtils/tree/master/vagrant
17:13:17 <reed> #link https://raw.githubusercontent.com/MetricsGrimoire/puppet-metricsgrimoire/master/architecture.rst
17:13:32 <dizquierdo> just to make sure that you understand how that works
17:13:33 <mrmartin> dizquierdo, not yet, but I'll
17:13:37 <dizquierdo> ah ok ok
17:13:44 <dizquierdo> let me know if you have any issue with this
17:13:47 <reed> dizquierdo, wait, let's make sure we make progress on the spec first
17:13:53 <dizquierdo> ok
17:14:00 <dizquierdo> sorry
17:14:04 <mrmartin> ok, so this vagrant sets up everything using shell scripts?
17:14:17 <mrmartin> nice work!
17:14:26 <dizquierdo> yes, that should work in that way :)
17:15:20 <mrmartin> looks promising, and definitely can help in puppetization
17:15:28 <dizquierdo> perfect
17:15:38 <dizquierdo> you need to update some config files, but that should be all
17:15:57 <mrmartin> great, I'll allocate some time tomorrow to test it
17:16:08 <dizquierdo> I'll be around just in case you need some help
17:16:14 <reed> ok, I assume that Problem description is good enough. Let's move on the spec draft :)
17:16:30 <reed> check the Proposed change:
17:16:44 <reed> dizquierdo, is anything missing from the list of tools?
17:17:13 <dizquierdo> reed, I don't think so, we're nowadays migrating OpenStack to SortingHat, a new tool to manage identities, but that's all
17:17:27 <reed> so let's add SortingHat
17:17:31 <dizquierdo> this is the unique identitites database
17:17:45 <dizquierdo> ok
17:17:50 <dizquierdo> I'll add that to the rst file
17:17:57 <reed> oh, wait...
17:18:08 <reed> right, it'll be needed there too
17:18:46 <reed> so this part of the draft spec requires more work
17:18:58 <reed> the template says: Here is where you cover the change you propose to make in detail. How do you propose to solve this problem?
17:19:30 <reed> for example, we need to write here that we need mysql databases
17:19:43 <dizquierdo> mmm having a look at the spec file
17:20:09 <reed> the puppet scripts for the various tools is one step, then we need databases (are they on the same machine? would a trove instance be enough?)
17:20:33 <reed> are they all different databases? etc
17:21:02 <mrmartin> yeap, I suggest to enlist all databases we required, if need more
17:21:18 <dizquierdo> in the case of grimoire, they are all different databases, but it's enough to have all of them in the same machine
17:21:41 <mrmartin> ok, if we are going to use trove, we need to pass of required db-s to infra because they need to create it
17:22:00 <reed> and as for scope, I would limit this spec to the collecction of data only
17:22:13 <reed> I'd avoid going into the data analysis yet
17:22:22 <dizquierdo> I agree with you
17:23:45 <mrmartin> ok let's move on to Mailing Lists Stats puppet recipes
17:23:45 <reed> #action reed to describe the phased approach, first the data collection and then the data analysis
17:24:41 <reed> mrmartin, I would suggest to do that offline, unless you already have questions
17:25:32 <mrmartin> ok, I just like to notice here, that I did a quick review, and it requires refactoring to match the patterns and modules structure of other openstack puppet modules.
17:26:01 <dizquierdo> ok, we should probably work on creating them more similar to openstack's
17:26:26 <dizquierdo> mrmartin,  a link to some documentation would be good, or some examples at least :)
17:26:48 <mrmartin> dizquierdo: all of the puppet- repositories at github.com/openstack-infra/
17:27:02 <dizquierdo> great, thanks!
17:27:06 <mrmartin> dizquierdo: I can help with that refactoring, because some basic fundamentals is missing here
17:27:23 <dizquierdo> that would be appreciated
17:27:23 <mrmartin> like file resource handling, dependencies, etc.
17:27:28 <reed> #action dizquierdo to look at other openstack-infra/puppet-* modules and refactor mlstats puppet
17:27:47 <reed> ok,we can postpone this task until we have the spec done
17:28:02 <mrmartin> it's not a problem, that's a normal learning path required for puppet. the basic thing that puppet is handling states instead of processes
17:28:06 <reed> #info postpone working on the puppet scripts until the spec is done
17:28:39 <reed> we have a long way to go before... so let's move to the next topic
17:28:59 <reed> dizquierdo, what are the issues that you see in database upgrades and stuff?
17:29:19 <dizquierdo> reed, basically issues related to updates of the database schema
17:29:21 <reed> those concerns are useful to clarify the spec
17:29:32 <dizquierdo> and addition or removals of repositories
17:29:42 <reed> how do you do them now? sql scripts and configuration changes, right?
17:29:44 <dizquierdo> updates of projects to get integrated, incubated, etc
17:29:53 <dizquierdo> that's it reed
17:29:57 <dizquierdo> sql basically
17:30:08 <dizquierdo> if it's possible to apply sql scripts
17:30:12 <dizquierdo> that would be a solution for us
17:30:13 <mrmartin> you could check how Drupal or Laravel solving those problems
17:30:14 <reed> those can be managed by the puppet recipes
17:30:27 <mrmartin> usually there is a tool that can apply the change
17:30:42 <reed> like bash is such a tool, right? :)
17:30:48 <dizquierdo> :)
17:31:02 <mrmartin> or this tool for python's sqlalchemy: https://alembic.readthedocs.org/en/latest/
17:31:37 <mrmartin> the trick here, that those tools storing the schema version somewhere, so it is easy to do schema updates / downgrades
17:32:02 <mrmartin> so if you like to revert a schema due a failed upgrade, than you can do it easily
17:32:18 <dizquierdo> that sounds good for sure
17:32:41 <mrmartin> requires a bit more preparation from application side, but you win a lot at operation side.
17:32:50 <reed> so dizquierdo this goes back into how cvsanaly is designed and treats upgrades in general
17:33:32 <reed> IIRC it has sql scripts to create the db and upgrades, and python setup takes care of running thos
17:33:35 <reed> e
17:33:44 <dizquierdo> let's say that cvsanaly does not care about updates
17:33:56 <dizquierdo> if there's a new database schema, cvsanaly will start populating the database with such new schema
17:34:11 <dizquierdo> internal upgrades are typically done through sql scripts
17:34:39 <reed> using mysql client?
17:34:46 <dizquierdo> yes
17:35:43 <reed> ok, so we'll need to add to the spec how those need to be managed
17:35:57 <mrmartin> with other projects we used to package those updates next to the app, so the update tool automatically can pick up the new schemas and apply
17:36:00 <reed> either by developing a tool, a bash script or something
17:36:15 <mrmartin> exactly
17:36:30 <dizquierdo> sounds good, yep
17:36:51 <reed> #action dizquierdo to add to the spec the description of database management
17:37:00 <reed> I added a NOTE to the spec draft
17:37:23 <reed> next topic?
17:37:40 <dizquierdo> seems to be the activity board issues in github
17:37:47 <dizquierdo> btw, no advances with this respect
17:37:53 <dizquierdo> we have planned them for this week
17:37:54 <reed> #topic review pending issues
17:38:03 <reed> dizquierdo, ok
17:38:41 <reed> #info no progress on github issues about grimoire set, reconvene next week
17:39:02 <mrmartin> ok, then let's go on with askbot
17:39:28 <reed> #topic askbot infratization
17:39:37 <mrmartin> I've added the missing cron jobs: https://review.openstack.org/166882
17:39:58 <mrmartin> they are not doing too much, and seems to be that the Chinese issue is a separated one
17:40:00 <reed> #info reed tested the new system, found it lacking Chinese search indexes
17:40:13 <reed> #info mrmartin added the missing cron jobs: https://review.openstack.org/166882
17:40:25 <mrmartin> so I'm looking it, and will work with fungi to find the root of the issue
17:40:31 <reed> #info the issue with Chinese language search is still being investigated
17:40:50 <reed> #action mrmartin to keep investigating the root cause of missing Chinese language search
17:40:52 <mrmartin> actually I'm testing that in Vagrant, as I remember it was working before, but needs a little time to review solr, askbot integration
17:41:10 <reed> ok, so we're blocked until this is solved
17:41:12 <mrmartin> anyway, if it is done, we can move forward with testing
17:41:54 <reed> #info the migration is on hold until the integration solr/askbot is cleared and Chinese search works
17:42:12 <reed> so we can skip the following next steps in the migration, we have no date to announce downtime
17:42:40 <reed> mrmartin, please make askbot your priority this week
17:42:49 <mrmartin> ok, I'm sure we can solve it in this week, there was some magic around the chinese solr analyzers
17:42:57 <reed> cool
17:43:11 <reed> I know we're close :)
17:43:23 <reed> so next topic?
17:43:26 <mrmartin> groups
17:43:35 <reed> #topic User Groups portal: review of pending issues
17:43:44 <reed> #link https://storyboard.openstack.org/#!/project_group/58
17:44:02 <mrmartin> ok, I'm was working with the meetup.com event data exchange issue
17:44:52 <mrmartin> it is working on dev, except I found an issue with event location format, and there is a new patch for that: https://review.openstack.org/166917
17:45:32 <mrmartin> the root of the problem, that we don't know the exact format how to meetup.com is passing the location, and I already meet with at least 4-5 different internal format
17:45:43 <reed> #info mrmartin working to add location details to events imported from meetup.com
17:45:54 <reed> ouch
17:46:07 <mrmartin> this is the test for different formats: https://github.com/openstack-infra/groups/blob/master/modules/groups/groups_feeds/groups_feeds.test
17:46:35 <reed> is there a way to simply get the city and the state?
17:46:45 <mrmartin> so the event list looks better now on dev: https://groups-dev.openstack.org/
17:47:20 <mrmartin> I need to talk with Jimmy what they are using finally, the xml or the html for parsign
17:47:23 <mrmartin> parsing
17:47:53 <mrmartin> because the RSS feed not exports the location by default, but I checked what we have there, and Commons provides an iCal exporter too
17:48:23 <reed> we may want to export the ical for individual groups
17:48:36 <reed> but we can discuss this on the mailing list
17:49:13 <mrmartin> that's a good question, because we have ical's actually for groups who are using meetup.com
17:49:21 <reed> mrmartin, I find it surprising that meetup.com API don't give you simply the city and the state/nation of the event
17:49:38 <mrmartin> I think it was agile
17:50:09 <reed> what is agile?
17:50:15 <mrmartin> meetup.com development :D
17:50:18 <mrmartin> check this: http://paste.openstack.org/show/195485/
17:50:18 <reed> ah
17:50:35 <mrmartin> so they are providing a different format for US events
17:50:42 <mrmartin> and a different one for other countries
17:50:55 <mrmartin> and event for US events they have at least 3 different representation
17:51:01 <reed> crap
17:51:17 <mrmartin> including ADDRESS, CITY, STATE
17:51:27 <reed> let's have a look, probably it's not worth pursuing this
17:51:32 <mrmartin> or ADDRESS, CITY, STATE POCODE
17:51:52 <reed> wait a second... maybe we can solve this issue in a much simpler way
17:52:19 <mrmartin> ok, I guess the parser is quite nice I wrote, so it is handling that well, just need to watch for missing addresses it cannot parse well, I found only one.
17:52:19 <reed> the problem we're trying to solve is that on http://openstack.org/community/events there is no information about the 'place' of the event
17:52:54 <reed> if the place is the name of the user group, that may be enough
17:52:55 <mrmartin> exactly, and we had the field that for the Commons by default, but the meetup.com importer was not imported that value well
17:53:32 <mrmartin> ok, this location parser is ready now, I need to accept the patch waiting in the queue, and do a release
17:53:36 <reed> well... if the parser is almost done then let's try it
17:53:44 <reed> ok
17:54:10 <mrmartin> don't throw that out, I spent last week with polishing that :)
17:54:14 <reed> #info this location parser is ready and only waiting for a release
17:54:28 <reed> if I knew how hard this was, I would have stopped you :)
17:54:51 <mrmartin> it was exciting, not hard :D
17:54:56 <mrmartin> anyway
17:55:05 <reed> exactly, I would have taken the toy off your plate :)
17:55:17 <mrmartin> it is ready, so I can roll it out, and move forward with other open things
17:55:24 <reed> ok
17:55:28 <mrmartin> we have a priority here anyway
17:55:47 <reed> #action mrmartin to merge the new parser for location and release new groups portal
17:55:51 <mrmartin> https://www.drupal.org/node/2455135
17:56:02 <mrmartin> this one, is a major security upgrade for Commons modules
17:56:16 <reed> #info let's watch the exported events feed for broken locations
17:56:23 <mrmartin> so I need to backport this or upgrade all commons modules to this new version, which is a major jump
17:57:09 <reed> #info major security upgrade required for Commons modules, mrmartin to investigate whether backport it or upgrade all modules
17:57:26 <reed> what's the cost of upgrading?
17:58:03 <mrmartin> we had some patches that fixed bugs of Commons, I need to check whether those were applied in the newer release
17:58:22 <mrmartin> or need to refactor them to match the new modules
17:58:35 <mrmartin> we have 4-5 patches for commons
17:58:46 <reed> no more patches :)
17:59:02 <mrmartin> it is a good opportunity to contrib back if we have some leftovers
17:59:09 <reed> indeed
17:59:15 <mrmartin> yeap, commons is patch the core drupal too
17:59:24 <mrmartin> but it is trackable well with the drush make file
17:59:35 <mrmartin> so not a huge deal
17:59:36 <reed> can you work on this security fix *and* on askbot  too?
17:59:41 <mrmartin> yes
18:00:03 <reed> can they go in parallel or do you need to set one priority higher?
18:00:19 <mrmartin> so if the security fix is a not huge deal, I can release it together with the event location patch
18:00:32 <mrmartin> so I can save some release time here
18:00:39 * reed writes in English sometimes in uber-convoluted ways :)
18:01:12 <reed> what I mean is to understand whether the security issues need to go higher in priority than the askbot
18:01:13 <mrmartin> we need a more prior priority
18:01:28 <mrmartin> I hope both of them can roll-out tomorrow
18:01:39 <reed> both which ones?
18:01:47 <mrmartin> the askbot and the sec fix
18:01:58 <mrmartin> but I guess is sec fix is very important
18:02:00 <reed> do the sec fix first
18:02:34 <reed> #info reshuffle priorities: apply drupal security fix first and then investigate the askbot solr integration issue
18:02:40 <reed> ok, we need to close the meeting
18:02:47 <mrmartin> we are ready anyway
18:02:57 <reed> yep, indeed
18:03:09 <reed> #endmeeting