15:01:02 <SlickNik> #startmeeting trove
15:01:03 <openstack> Meeting started Wed Jun 28 15:01:02 2017 UTC and is due to finish in 60 minutes.  The chair is SlickNik. Information about MeetBot at http://wiki.debian.org/MeetBot.
15:01:04 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
15:01:07 <openstack> The meeting name has been set to 'trove'
15:01:50 <SlickNik> Courtesy ping for johnma slicknik songjian trevormc amrith
15:02:37 <SlickNik> The agenda for today's meeting is at https://wiki.openstack.org/wiki/Meetings/TroveMeeting#Trove_Meeting.2C_Agenda
15:03:04 <SlickNik> Pretty light today.
15:03:51 <SlickNik> #topic Re-architect Trove Proposal
15:04:59 <SlickNik> amrith has sent an email to the openstack development mailing list to begin the conversation re-architecting trove.
15:05:24 <SlickNik> Link to the discussion is at:
15:05:26 <SlickNik> #link http://openstack.markmail.org/thread/36lkyacftu5geqcf
15:05:35 <johnma> Hi slicknik, sorry a bit late
15:05:56 <SlickNik> johnma: no worries — pretty light agenda today.
15:07:09 <SlickNik> I suspect there will be a little more discussion as well — amrith mentions that once the discussion stops, he will summarize and reply to the thread.
15:07:28 <smatzek> here for Trove mtg, sorry I'm late
15:07:42 <johnma> that would be nice.
15:07:51 <SlickNik> smatzek: welcome — no problem
15:08:58 <SlickNik> johnma: +1 It would be good to have some concrete next steps after the discussion. Looking forward to it.
15:09:20 <johnma> yes
15:10:32 <SlickNik> There was some mention of discussing it at the next video meeting that trevormc is organizing as well. So stay tuned for that.
15:10:58 <SlickNik> Any questions while we're on that topic?
15:12:10 <SlickNik> …
15:12:15 <SlickNik> Moving on
15:12:21 <SlickNik> #topic Recent gate failures
15:13:01 <SlickNik> There has been an incidence of certain gate failures recently. Failures where the glance service fails to start for some reason.
15:13:22 <SlickNik> Logs in this case look like:
15:13:24 <SlickNik> #link http://logs.openstack.org/57/472957/9/check/gate-trove-scenario-dsvm-mysql-single-ubuntu-xenial/1b395ae/console.html#_2017-06-27_13_22_48_752039
15:13:46 <SlickNik> Notice the "SKIP: Failure in <function start_glance_api at 0x7f4a134092a8>"
15:14:31 <SlickNik> and "RuntimeError: Issued the command successfully but the service (['python', u'/usr/local/bin/glance-api', '/vagrant/conf/glance-reg.conf']) never seemed to start."
15:15:12 <SlickNik> Not sure what the issue with the glance api service is
15:15:39 <SlickNik> Anyone have any ideas why this might be happening, or what needs to be looked at in this case?
15:16:34 <johnma> not sure SlickNik,
15:16:55 <smatzek> are other gates (Nova?) failing on the glance issue? ie. is it specific to Trove?
15:17:32 <SlickNik> smatzek: I haven't seen other failures in the gate for other projects — so it seems specific to how trove is running in devstack-gate, I think.
15:17:51 <SlickNik> I haven't really had a chance to look into this — will have some time later today during which I can take a look.
15:18:59 <SlickNik> If there are any volunteers who can and want to look into this earlier — would help as well.
15:20:26 <SlickNik> #topic Open Discussion
15:20:27 <smatzek> I took a look at the glance screen log from the job you mention and it looks like Glance api is starting and logging
15:21:33 <smatzek> on the open discussion front, could you give a bit of history lesson on the periodic tasks in trove-taskmanager that handle notifications and the intended consumers of them?
15:21:55 <SlickNik> smatzek: Yes, I think amrith mentioned that that might be the case as well — so it's probably the detection code in trove-integration that's failing to recognize that it is running.
15:21:57 <smatzek> DBaaS quotas and instance exists - both appear to be broken at master for different reasons
15:23:06 <SlickNik> Not sure if something changed in glance that requires an update to the detection piece in trove/integration/tests
15:23:20 <smatzek> do you have a quick link to that code?
15:24:02 <SlickNik> one sec
15:24:59 <smatzek> on the notification front, I'm interested because I'm using OpenStack Ansible to setup my dev env as I'm not on x86, and it enables the instance exists periodic task by default. It appears it's been broken since Ocata.
15:26:05 <smatzek> Depending on the intended purpose for these notifications (ceilometer billing, etc,) it may gauge the priority of fixing them up.
15:26:20 <SlickNik> smatzek: Essentially this => https://github.com/openstack/trove/blob/4bedc8b6e0fd462a7811baa7efcc9f9a0cb5859b/integration/tests/integration/tests/util/services.py#L178-L199
15:27:27 <smatzek> time frame on when these failures started appearing so we can compare past / successful runs?
15:27:56 <SlickNik> which gets called from here https://github.com/openstack/trove/blob/4bedc8b6e0fd462a7811baa7efcc9f9a0cb5859b/integration/tests/integration/tests/util/services.py#L229-L233
15:29:05 <smatzek> the thing that immediately stands out to me is it's searching for a PID and in the head of the glance screen log I see Glance forking off child pids for uWSGI workers.  If the process tree for glance recently changed, the code may be looking for the wrong pid, or looking for a pid that no longer exists because it just forked daemons
15:32:09 <SlickNik> Yes, that's possible - especially if glance just forked to daemonize and the initial process exited or something.
15:32:27 <SlickNik> I'm not sure, would have to take a closer look to find out what's going on.
15:33:34 <smatzek> comparing to a successful run of the same job, the glance log looks a lot different and importantly doesn't have the wgsi pid spawning at the head of the log http://logs.openstack.org/05/454205/7/check/gate-trove-scenario-dsvm-mysql-single-ubuntu-xenial/56a3e72/logs/screen-g-api.txt.gz
15:34:05 <SlickNik> aha, so maybe they did change something here recently?
15:34:23 <smatzek> yeah, looks like it, the head of the logs are very different.
15:36:45 <SlickNik> taking a look at https://github.com/openstack/glance/commits/master I don't see anything obvious
15:38:16 <SlickNik> Maybe https://github.com/openstack/glance/commit/5e260055faada3849340e4cd31227e6244c3af06?
15:38:34 <SlickNik> corresponding review https://review.openstack.org/#/c/459450/
15:39:41 <smatzek> yeah, I was thinking the same thing
15:41:40 <SlickNik> Anyway, it looks like we probably need to update the logic for service detection.
15:42:48 <SlickNik> Moving on back to Open Discussion.
15:43:04 <SlickNik> Did anyone have anything else they'd like to cover?
15:43:52 <SlickNik> ...
15:44:20 <SlickNik> #endmeeting