14:00:41 <rosmaita> #startmeeting glance
14:00:42 <openstack> Meeting started Thu Jan 25 14:00:41 2018 UTC and is due to finish in 60 minutes.  The chair is rosmaita. Information about MeetBot at http://wiki.debian.org/MeetBot.
14:00:43 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
14:00:45 <openstack> The meeting name has been set to 'glance'
14:00:52 <abhishekk> o/
14:00:57 <kairat> o/
14:01:03 <McClymontS> o/
14:01:52 <jokke_> o/
14:02:33 <smcginnis> o/
14:02:44 <rosmaita> looks like a good turnout!
14:03:06 <rosmaita> i have a million windows open, give me a sec to find the agenda
14:03:16 <rosmaita> #link https://etherpad.openstack.org/p/glance-team-meeting-agenda
14:03:24 <rosmaita> #topic updates
14:03:35 <rosmaita> some good news and some bad news
14:03:39 <rosmaita> first, the good news
14:03:49 <rosmaita> python-glanceclient 2.9.1 was released yesterday and the stable/queens branch was cut
14:04:04 <rosmaita> glance_store 0.23.0 was released last week and the stable/queens branch was cut
14:04:21 <rosmaita> and the bad news is that Q-3 must be released today
14:04:28 <rosmaita> which isn't actually the bad news
14:04:41 <rosmaita> the bad news is that the functional tests are a mess
14:04:44 <rosmaita> but i'll talk about that later
14:04:51 <rosmaita> here are some more updates
14:05:20 <rosmaita> i was working from the release timeline about when the PTL nomination period starts
14:05:27 <rosmaita> and the date is incorrect there
14:05:35 <rosmaita> it's actually next week
14:05:45 <rosmaita> #link https://governance.openstack.org/election/
14:06:21 <rosmaita> so there's still time to throw your hat into the ring
14:06:54 <rosmaita> next, in case you haven't seen this on the ML, there is a new Depends-on syntax to use in commit messages
14:07:04 <rosmaita> #link http://lists.openstack.org/pipermail/openstack-dev/2018-January/126535.html
14:07:24 <rosmaita> the old way (using change id) will work for a while, not sure when it will be phased out
14:07:31 <rosmaita> may say in that email if you read it carefully
14:07:46 <rosmaita> anyway, the new way is to use the review URL
14:08:26 <rosmaita> next thing, the google summer of code competition is coming up and openstack wants to cash in
14:08:41 <rosmaita> and they could use some glance projects
14:08:49 <rosmaita> #link https://wiki.openstack.org/wiki/GSoC2018
14:08:55 <McClymon_> I have done this before rosmaita
14:09:03 <rosmaita> it's not just a project, though, it requires a mentoring commitment
14:09:12 <rosmaita> McClymon_ do you want to pay back?
14:09:19 <McClymon_> appears my handle has been changed lol
14:09:23 <rosmaita> i mean, do you want to be a glance mentor for a GSoC student?
14:09:31 <McClymon_> let me get back to you
14:09:39 <McClymon_> but I am familiar and usually participate in it yearly
14:09:39 <rosmaita> sure
14:09:47 <rosmaita> anyway, something for everyone to think about
14:10:05 <rosmaita> it's kind of related to the next item, actually ... the PTG proposals
14:10:32 <rosmaita> what i mean is, if you think of something that could be student-doable, don't hesitate to add it as a proposal for the ptg to be discusse
14:10:41 <rosmaita> *discussed
14:10:54 <abhishekk> ack
14:11:19 <rosmaita> anyway, i don't know what the odds are of us actually getting someone from GSoC
14:11:34 <rosmaita> at this point, i think the foundation is pushing so that we don't look bad as a community
14:11:54 <rosmaita> but, it could be a good way to get some people interested in glance
14:12:16 <rosmaita> i mean, having do-able projects defined for someone to pick up and get involved, even if they're not part of GSoC
14:12:41 <rosmaita> anyway, i have committed to putting a topic or two on that wiki page
14:13:00 <rosmaita> but feel free to contribute yourself if you have something in minde
14:13:04 <rosmaita> *mind
14:13:10 <rosmaita> ok, last update
14:13:24 <rosmaita> the long-awaited PTG planning etherpad is now available
14:13:32 <rosmaita> #link https://etherpad.openstack.org/p/glance-rocky-ptg-planning
14:13:49 <rosmaita> #action rosmaita send email about PTG planning to ML
14:14:14 <rosmaita> i've seeded it with a few topics, please add more
14:14:41 <rosmaita> ok, that's all the updates
14:14:43 <smcginnis> rosmaita: I've added it to the list of Rocky etherpads.
14:14:49 <rosmaita> smcginnis ty!
14:14:55 <smcginnis> #link https://wiki.openstack.org/wiki/PTG/Rocky/Etherpads Full PTG Etherpad list
14:15:18 <rosmaita> smcginnis you are a handy person to have around
14:15:25 <smcginnis> :)
14:15:28 <rosmaita> and now for the bad news
14:15:38 <rosmaita> #topic functional gate situation
14:15:54 <rosmaita> has been bad since over the weekend, and has actually not improved
14:16:07 <rosmaita> here is some light reading summarizing the situation
14:16:14 <rosmaita> we actually have 2 issues
14:16:26 <rosmaita> first one concerns the py27 functional tests
14:16:43 <rosmaita> which are actually the same as the py35 tests, just run under py27
14:16:59 <rosmaita> so i'm really talking about the gate jobs
14:17:18 <rosmaita> anyway, under py27, our functional tests no longer work
14:17:29 <rosmaita> i learned this morning that searchlight is in the same boat
14:17:41 <rosmaita> but i'm not aware of anyone else
14:18:02 <rosmaita> here's a summary if you read through this short thread:
14:18:11 <rosmaita> #link http://lists.openstack.org/pipermail/openstack-dev/2018-January/126460.html
14:18:38 <smcginnis> There apparently are networking issues in the gate right now too, so that's going to slow things down and probably cause a few more rechecks too.
14:18:41 <rosmaita> so the crisis this caused is that over the weekend, the cross-glance-py27 job was removed from the requirements gate
14:18:54 <rosmaita> because it was failing
14:19:27 <rosmaita> smcginnis had a patch from june or so that we should've paid attention to sooner
14:19:36 <rosmaita> separating out the unit tests from the functional tests
14:20:05 <rosmaita> so, we spent a few days fighting the gates, which have been remarkably slow, to get that merged
14:20:29 <rosmaita> because i do not like the idea of having absolutely no glance participation in the py27 for requirements changes
14:20:46 <rosmaita> that way, the requirements team could at least run the unit tests
14:20:55 <rosmaita> which is actually what they do for most projects, by the way
14:21:09 <rosmaita> i think nova has a separate functional test job, but they're the only ones
14:21:12 <jokke_> well clearly something failed already when requirements/baseline change broke that gate without warning
14:21:49 <rosmaita> yeah, and i didn't appreciate the quick dump-glance-becasue-it's-milestone-week action
14:21:55 <rosmaita> but, one can see their point
14:22:29 <rosmaita> anyway, i put up a patch to revert our removal, because there's nothing wroing with our py27 unit tests
14:22:50 <rosmaita> but, that has been a nightmare too just in terms of getting through the gate
14:23:00 <rosmaita> https://review.openstack.org/#/c/536946/
14:23:01 <jokke_> have you received anyinsight from infra what might have changed around the time when the gate broke?
14:23:20 <rosmaita> no
14:23:38 <rosmaita> all we can figure out is that there was some kind of distro-package change
14:23:47 <abhishekk> clueless at the moment
14:24:23 <rosmaita> btw, i couldn't get the failure locally until rebuilding a spanking new ubuntu and doing a dist-upgrade on it
14:24:25 <jokke_> ok, and this is ubuntu only issue I assume?
14:24:34 <rosmaita> do not know
14:24:42 <abhishekk> jokke_, yes
14:24:58 <jokke_> interesting ... rosmaita do you have the old env still?
14:25:09 <jokke_> comparing the packages could give some insight
14:25:17 <smcginnis> You've been able to repro this failure on a newer Ubuntu install, right?
14:25:28 <rosmaita> funny you should ask that.  Somehow my old VM has gotten hosed
14:25:36 <rosmaita> smcginnis yes
14:25:45 <smcginnis> Can we get the output from "sudo apt list --isntalled" from the new and the old and see if we can diff out anything between the two?
14:25:51 <smcginnis> *installed
14:25:53 <rosmaita> jokke_ maybe the old infra images around
14:26:22 <rosmaita> i was talking to steve mcclellan, searchlight ptl this morning
14:26:24 <jokke_> rosmaita: yeah, that's where I would start
14:26:42 <rosmaita> he said there's been an update
14:27:04 <rosmaita> to the xenial package manager
14:27:06 <rosmaita> https://launchpad.net/ubuntu/+source/python2.7/2.7.12-1ubuntu0~16.04.3
14:27:12 <rosmaita> on jan 18
14:27:16 <rosmaita> which fits the time frame
14:27:29 <rosmaita> but i haven't had time to look at that
14:27:44 <rosmaita> because i learned about it 5 minutes ago
14:28:07 <rosmaita> also, it doesn't help that the infra team has their hands full with all the other stuff happening
14:28:41 <rosmaita> ok, so that's pretty bad
14:28:44 <rosmaita> and then suddenly, it got worse
14:28:57 <jokke_> It's very interesting that it's just us (glance and SL) breaking if it's something that fundamental that was changed
14:29:16 <rosmaita> well, we have the same basic code base
14:29:22 <jokke_> unfortunately I do not know ubuntu well enough to even start looking into that level changes
14:29:46 <rosmaita> yeah
14:30:03 <rosmaita> so, now the really bad news
14:30:13 <rosmaita> the py35 functional tests suddenly started breaking
14:30:23 <rosmaita> and this i haven't been able to reproduce locally
14:30:32 <abhishekk> neither do i
14:30:41 <rosmaita> #link http://lists.openstack.org/pipermail/openstack-dev/2018-January/126519.html
14:30:46 <rosmaita> so it was just one test
14:31:07 <rosmaita> testing the alembic migration for ocata (i think, just one release anyway)
14:31:20 <rosmaita> so, i put up a skiptest patch for that
14:31:27 <rosmaita> which finally merged late last night
14:31:57 <rosmaita> but, that doesn't seem to have stopped that test from being executed
14:32:06 <rosmaita> i do not know wtf is going on with that
14:32:22 <rosmaita> the alembic test code uses a bunch of mixins
14:32:41 <jokke_> is zuul honouring the other skips still?
14:32:41 <rosmaita> which i personally find difficult to read and troubleshoot
14:32:42 <rosmaita> (and understand, to be frank)
14:32:59 <rosmaita> i guess so, but i don't know
14:33:10 <rosmaita> yeah, it must be
14:33:22 <rosmaita> because we have some skips for py35
14:33:24 <rosmaita> (still)
14:33:52 <rosmaita> so my point is, i don;'t know if i put the skip in the wrong place or what
14:34:16 <rosmaita> when i ran locally, i was worried that skipping one class might cause a cascade of failures
14:34:19 <rosmaita> but it did not
14:34:41 <rosmaita> and i'm pretty sure that test was skipped, but i really don;'t know
14:34:42 <rosmaita> need to re-check
14:34:53 <rosmaita> anyway, here is the patch with the skip:
14:35:13 <rosmaita> https://review.openstack.org/#/c/536939/
14:35:40 <rosmaita> and here is the patch showing a failure in the py35 functional check job:
14:36:22 <rosmaita> https://review.openstack.org/#/c/536630/
14:36:41 <rosmaita> just noticed that abhishek and andreas have some comments on there while i was asleep
14:37:19 <rosmaita> Ok, so pretty bad news that (a) this is happening, and (b) we don't know why
14:37:23 <rosmaita> but the really bad news is
14:37:35 <rosmaita> we have no functional tests running in the gate right now
14:37:48 <rosmaita> i want to stress that
14:38:10 <abhishekk> nova is using "@test.testtools.skip("bug 1189462")"
14:38:11 <openstack> bug 1189462 in OpenStack Compute (nova) "some foreign key constraints are missing in sqlite" [Low,Opinion] https://launchpad.net/bugs/1189462
14:38:51 <rosmaita> wow that's an old bug
14:39:13 <rosmaita> but i think our failure was in the mysql mixin ?
14:39:50 <jokke_> rosmaita: so by quick look we actually don't use unittest.skip() decorator anywhere else in our tests
14:40:05 <rosmaita> there is actually one place else
14:40:11 <rosmaita> (pretty sure)
14:40:26 <rosmaita> it may be skiptest though
14:40:51 <rosmaita> this one, i couldn't figure out how to skip one test so i did the entire class
14:40:55 <jokke_> there is two invocations of skipIf checking if it's running under PY3 and skipping then
14:41:32 <rosmaita> i guess i could try something like that
14:41:41 <rosmaita> problem is, i am pretty damned discouraged
14:41:42 <jokke_> and then there is few instances untilizing self.SkipTest that is inherited somewhere from the test class
14:41:51 <rosmaita> took me like 48 hours to get the skip patch merged
14:42:21 <rosmaita> i did get a message locally about the skip happening
14:42:28 <jokke_> yeah ... zuulv3.openstack.org is not responding at all atm
14:42:29 <rosmaita> pretty sure, anyway, that was like 48 hours ago
14:42:33 <rosmaita> arrrrrgh
14:42:52 <rosmaita> yeah, and you saw my comment in the email about us getting caught in the gerrit restart
14:43:04 <rosmaita> apparently at the same time zuul was saying OK, gerrit was unavailable
14:43:08 <smcginnis> zuulv3.openstack.org is gone. Just use zuul.openstack.org.
14:43:16 <rosmaita> and the OK disappeared into the ether
14:43:27 <smcginnis> Apparently the v3 version was only meant to be temporary and they removed it over the weekend.
14:43:36 <jokke_> graah
14:43:45 <rosmaita> lots of nefarious stuff happend over the weekend!
14:43:49 <smcginnis> jokke_: I had the same reaction.
14:44:13 <rosmaita> ok, so here's the first question
14:44:18 <rosmaita> we need to cut Q-3
14:44:41 <rosmaita> the thing sitting at head right now is my skip-test patch that apparently doesn't work
14:44:49 <rosmaita> i guess we could cut Q-3 one below that?
14:45:36 <smcginnis> That should be fine. We'll just have to pick up any blocked things in the RC.
14:45:36 <rosmaita> http://git.openstack.org/cgit/openstack/glance/log/
14:46:05 <jokke_> we have 535166 gating still, which is something I'd really like to have merged before cutting
14:46:22 <jokke_> that is the oslo.serialization fix
14:46:42 <jokke_> which might contribute to lots of this
14:47:13 <rosmaita> wtf, that got workflowed on jan 20
14:47:47 <jokke_> and rechecked about 7 times since
14:48:41 <rosmaita> well, it may not have been rechecked since sean's unit/func test split patch was merged
14:49:12 <jokke_> so regardless how bad anyone might feel of not having the milestone out in time, I do not feel comfortable cutting the release on current state
14:49:19 <jokke_> rosmaita: it has and it's gating
14:49:34 <jokke_> all the tests were started moments ago
14:49:42 <jokke_> so please don't do new recheck on it
14:50:22 <rosmaita> starting gate jobs at 3:44 am?
14:50:25 <rosmaita> good grief
14:50:34 <jokke_> it's been on gate queue for over 15hrs
14:51:05 <rosmaita> yes indeed
14:51:15 <rosmaita> ok, we need a specific plan of action here
14:51:30 <rosmaita> other than rewriting glance in go
14:51:58 <jokke_> lets get the meeting through and make war plan on #os-g after
14:52:02 <jokke_> we have like 10min left
14:52:14 <rosmaita> ok, works for me ... abhishekk can you stick around?
14:52:28 <abhishekk> yes
14:52:41 <rosmaita> ok, next item i will table for next week as i don't know who put it on the agenda
14:52:43 <rosmaita> #topic open discussion
14:52:51 <rosmaita> any open discussion?
14:53:14 <rosmaita> otherwise, let's adjourn early and go to glance channel
14:53:41 <jokke_> i don't think I have anything for now
14:53:59 * jokke_ is still trying to catch up after PTO
14:54:25 <rosmaita> ok i will end at 14:55 if i don't hear anything
14:54:41 <abhishekk> nothing from me
14:55:01 <rosmaita> ok, thanks everyone!
14:55:07 <rosmaita> #endmeeting glance