21:01:52 <devkulkarni> #startmeeting Solum Team Meeting
21:01:53 <openstack> Meeting started Tue Jan 20 21:01:52 2015 UTC and is due to finish in 60 minutes.  The chair is devkulkarni. Information about MeetBot at http://wiki.debian.org/MeetBot.
21:01:54 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
21:01:57 <openstack> The meeting name has been set to 'solum_team_meeting'
21:02:20 <devkulkarni> #link https://wiki.openstack.org/wiki/Meetings/Solum#Agenda_for_2015-01-13_2100_UTC Our Agenda
21:02:24 <james_li> james li
21:02:28 <devkulkarni> I haven't updated the agenda
21:02:33 <devkulkarni> #topic Roll Call
21:02:37 <roshanagr> Roshan Agrawal
21:02:39 <devkulkarni> devdatta
21:02:40 <muralia> murali allada
21:02:40 <mkam> Melissa Kam
21:02:52 <phiche> philip cheong
21:03:00 <devkulkarni> hey everyone
21:03:14 <datsun180b> ed cranford
21:03:24 <devkulkarni> #topic Announcements
21:03:42 <devkulkarni> does anyone have any announcements, news to share, etc.?
21:04:20 <dimtruck> dimitry ushakov
21:04:28 <gpilz> gilbert pilz
21:04:35 <devkulkarni> #topic review action items
21:04:51 <devkulkarni> we had one item which was assigned to adrian
21:05:06 <devkulkarni> but I haven't heard from him on that. we can carry it forward I guess
21:05:11 <datsun180b> guess so
21:05:23 <devkulkarni> #action adrian_otto to follow up with mistral devs to arrange a tagged release that we can use as a stable dependency
21:05:41 <devkulkarni> was there any other action item that we remember?
21:06:03 <devkulkarni> if not, we can jump to next topic where I am sure there are several points to discuss
21:06:17 <datsun180b> i say move ahead
21:06:36 <devkulkarni> #topic reviews etc.
21:06:50 <devkulkarni> so lots of gate failures in last couple of days
21:06:56 <gpilz> yes
21:06:59 <devkulkarni> datsun180b: you want to lead us on what you found?
21:07:20 <devkulkarni> I can give updates on what I found after you are done datsun180b
21:07:27 <datsun180b> well today it's just an upstream thing with devstack, that's getting addressed presently
21:07:47 <devkulkarni> do you have a link to the bug and the patch that you can share with us?
21:07:51 <datsun180b> simple fix, just a matter of playing the openstack red tape game
21:08:24 <datsun180b> see that's funny, i filed a bug, but there's three reviews attacking it. two of them mention the bug, and the other one is the only one that's approved and being verified presently
21:08:37 <devkulkarni> datsun180b: do you have links to share with us
21:09:00 <datsun180b> #link https://review.openstack.org/#/c/148651/
21:09:06 <datsun180b> #link https://bugs.launchpad.net/devstack/+bug/1412861
21:09:40 <datsun180b> i guess our bot is missing
21:09:48 <devkulkarni> looks like the fix was approved
21:10:10 <devkulkarni> what do we need to do for our gate to pick it up? my guess is probably nothing..
21:10:13 <devkulkarni> right?
21:10:31 <datsun180b> yes, it's devstack
21:10:44 <devkulkarni> datsun180b: thanks for catching this one..
21:11:02 <devkulkarni> although I don't think the failures from yesterday have anything to do with this change
21:11:15 <devkulkarni> am I right?
21:11:27 <datsun180b> right, this only showed up this morning
21:11:42 <datsun180b> before that, i can't remember. something about cinder and six?
21:11:47 <devkulkarni> right. good that you picked it up and pursued it
21:11:50 <devkulkarni> yep
21:11:53 <devkulkarni> cinder and six
21:12:04 <devkulkarni> so here is the latest on it.
21:12:29 <devkulkarni> since yesterday we are seeing devstack gate failing on cinder-manage
21:12:59 <devkulkarni> the error shows up as something related to oslo.utils
21:13:08 <devkulkarni> which in turn depends on the package six.
21:13:26 <devkulkarni> I googled for the failure and the suggested resolution was to bump up the version of six
21:13:42 <devkulkarni> for us, the version is 1.7.0
21:13:50 <devkulkarni> global requirements is also at that version
21:14:00 <devkulkarni> so we cannot bump it any further for now
21:14:18 <devkulkarni> I chatted with folks on the cinder irc channel
21:14:36 <devkulkarni> i was informed that this error has been showing up only in solum
21:14:49 <devkulkarni> so we decided to recreate it with a test patch
21:15:08 <devkulkarni> and the test patch is now failing for a different reason on the gate (this has to do with tempest issue)
21:15:17 <devkulkarni> mkam has submitted a patch to fix the tempest issue
21:15:38 <devkulkarni> when that merges, we will be able to know whether the test-patch would retrigger the cinder six issue
21:15:56 <devkulkarni> until then our devstack gate will be in the current state (which is broken)
21:16:06 <gpilz> :((
21:16:23 <devkulkarni> yeah.. sorry about that gpilz.. can't do much at this point
21:16:29 <gpilz> it is what it is
21:16:33 <datsun180b> it'll take just a little longer for the fix to merge
21:16:43 <datsun180b> i'm watching it intently
21:16:51 <devkulkarni> if we can get mkam's patch move forward quickly
21:16:54 <devkulkarni> it will be awesome
21:17:08 <devkulkarni> looks like it failed again
21:17:44 <devkulkarni> #link http://logs.openstack.org/99/148699/2/check/gate-solum-devstack-dsvm/6eeca51/console.html
21:17:45 <datsun180b> want to see my johnny carson impression? "devstack failure about local messages"
21:18:16 <devkulkarni> looks like there is more to tempest change than what mkam has submitted
21:18:24 <devkulkarni> mkam: you around?
21:18:48 <datsun180b> well how about that, those failures are on us after all
21:19:11 <devkulkarni> which ones?
21:20:05 <devkulkarni> they seem to be from tempest_lib/common/rest_client
21:20:54 <devkulkarni> lets circle back on this topic in open discussion
21:20:59 <mkam> I can look into it more
21:21:27 <devkulkarni> mkam: thanks
21:21:51 <devkulkarni> any other reviews that folks want to discuss?
21:22:53 <devkulkarni> if not lets move to next topic..
21:23:04 <devkulkarni> #topic open discussion
21:23:54 <datsun180b> well the devstack thing merged
21:24:33 <gpilz> I have a good idea what those errors in test_assembly are about
21:24:41 <devkulkarni> datsun180b: now, until we get tempest issues fixed we won't know whether cinder+oslo+six issues are real
21:24:52 <gpilz> they stem from a bug I submitted against the rest_client
21:25:03 <devkulkarni> gpilz: do you have a link handy?
21:25:54 <gpilz> not handy
21:25:57 <gpilz> one sec …
21:27:05 <gpilz> https://bugs.launchpad.net/tempest/+bug/1407140
21:28:30 <datsun180b> #link https://bugs.launchpad.net/tempest/+bug/1407140
21:28:36 <datsun180b> oh right, the bot
21:28:40 <gpilz> hmmmm
21:28:46 <gpilz> that doesn't make sense though
21:28:51 <devkulkarni> gpilz: so what is your suggestion that we do on our side?
21:29:08 <gpilz> the test expects BadRequest and rest_client is throwing BadRequest
21:29:16 <gpilz> I don't get it
21:29:23 <devkulkarni> specifically, I am wondering how to address that 'BadRequest'
21:29:27 <devkulkarni> yeah
21:29:55 <devkulkarni> why would 'Details: The request is malformed. Reason: Plan is invalid. An error occurred during YAML parsing. Error position: (1:1)" this show up?
21:30:09 <devkulkarni> nothing changed on the API or the tests or the plan format
21:30:29 <devkulkarni> is this related to stannie42's patches which were merged recently?
21:30:31 <gpilz> I've seen similar code which worked around the bug in rest_client by asserting that UnexpectedResponseCode would be thrown when, in fact, BadRequest should have been thrown
21:30:52 <datsun180b> without seeing the content of the planfile i can't be sure
21:31:01 <datsun180b> whatever it is, it doesn't like the first character
21:31:19 <gpilz> the planfile is {}
21:31:48 <gpilz> functionaltests.api.v1.test_assembly.TestAssemblyController.test_assemblies_create_none  -  right?
21:32:41 <devkulkarni> yeah, that is the first one in the series
21:33:32 <datsun180b> is '{}' valid yaml?
21:34:15 <datsun180b> looks like it is
21:34:44 <gpilz> it is, but the YAML parser we're using raises an exception when it tries to parse "{}"
21:35:16 <gpilz> there are tests against the "plans" resource that depend on the same behavior
21:35:28 <devkulkarni> actually for test_assemblies_create_none, the failure we are getting is "Details: {u'debuginfo': None, u'faultcode': u'Client', u'faultstring': u"Invalid input for field/attribute name. Value: 'None'. Mandatory field missing."}"
21:35:45 <datsun180b> what's type safety
21:35:47 <devkulkarni> this is not yaml error
21:36:39 <gpilz> https://github.com/stackforge/solum/blob/master/functionaltests/api/v1/test_plan.py#L92
21:36:54 <gpilz> isn't that ^^^ exactly the same test?
21:37:01 <datsun180b> seems reasonable
21:37:47 <devkulkarni> yes the test input is same. but the error is not yaml parsing error in case of test_asemblies_create_none
21:38:12 <devkulkarni> anyways. point is,
21:38:18 <devkulkarni> all these tests have been passing
21:38:24 <devkulkarni> all along.
21:38:49 <devkulkarni> so most likely something that changed in tempest_lib is now causing these failures..
21:39:12 <devkulkarni> what's our plan to resolve this?
21:39:24 <datsun180b> we dig
21:39:41 <devkulkarni> +1
21:39:43 <gpilz> nothing else we can do
21:39:46 <devkulkarni> mkam
21:39:56 <devkulkarni> lets work on this together..
21:40:17 <devkulkarni> if you hear about any other folks seeing such errors from tempest that would be great to know
21:40:38 <gpilz> I'm at an ISO meeting for the rest of today
21:40:54 <gpilz> i'll be able to pitch in tomorrow (if you haven't figured it out by then)
21:41:04 <devkulkarni> gpilz: sounds like a plan
21:41:13 <adrian_otto> o/
21:41:20 <devkulkarni> hi adrian_otto
21:41:35 <devkulkarni> quick update for you
21:41:36 <adrian_otto> hi devkulkarni
21:41:59 <devkulkarni> our gates are currently broken. we are working on resolving the issues (we don't yet what the issue is)
21:42:14 <devkulkarni> there are guesses, but no clear answer
21:42:15 <adrian_otto> ok
21:42:30 <adrian_otto> have we touched base with the Infra team for guidance?
21:42:43 <devkulkarni> no, not yet
21:42:58 <devkulkarni> we reached out to cinder folks, devstack folks, and tempest folks
21:43:09 <devkulkarni> we found out issues arising in each of those
21:43:10 <adrian_otto> are they also impacted?
21:43:33 <devkulkarni> don't know.. our issues are showing up in those three projects
21:43:41 <devkulkarni> i mean on our devstack gates
21:43:54 <devkulkarni> those are the projects that are failing
21:43:54 <adrian_otto> sounds like devstack is the common thread there
21:44:11 <devkulkarni> datsun180b found a bug this morning in devstack
21:44:18 <devkulkarni> which has now been fixed in the upstream
21:44:28 <devkulkarni> now we are wading through bunch of tempest changes
21:44:29 <adrian_otto> does that require us to rebase anything?
21:44:36 <datsun180b> no, it was in devstack
21:44:38 <devkulkarni> no, nothing needs to be done on our side
21:44:48 <adrian_otto> ok, because we don't pull any of that in
21:44:53 <devkulkarni> right
21:44:59 <datsun180b> i found it because of my habit of burning my vm down and starting from scratch in the morning
21:45:22 <adrian_otto> something we should probably continue!
21:45:31 <adrian_otto> possibly as an automated process
21:45:35 <devkulkarni> right now, fixing tempest issues is our priority
21:45:35 <datsun180b> that alias is of course 'nero'
21:45:44 <adrian_otto> ok
21:46:03 <devkulkarni> once that is fixed, we will get to the issues that we were facing in cinder+six (package)
21:46:27 <devkulkarni> so the plan is for bunch of us to concentrate on fixing tempest issues right now.
21:46:44 <devkulkarni> .. and that is where we were in the discussion just before you joined us
21:46:52 <adrian_otto> thanks for the ramp up.
21:47:04 <datsun180b> yeah, i'm about ready to put down one set of spinning plates to dive into this
21:47:22 <devkulkarni> cool.. that will be really helpful datsun180b
21:47:49 <devkulkarni> lets coordinate with mkam to quickly resolve this
21:48:06 <adrian_otto> tempest broke our gates last week too, right?
21:48:15 <adrian_otto> is this a continuation of those problems?
21:48:17 <datsun180b> probably! where's my pitchfork
21:48:41 <devkulkarni> yeah.. today's problem is because they changed the rest library that they are using
21:48:49 <devkulkarni> they moved it to a separate library
21:49:00 <devkulkarni> and looks like they might have added stricter input checking
21:49:15 <devkulkarni> which is leading to some of our tests that have empty inputs (such as "{
21:49:18 <devkulkarni> "{}
21:49:23 <devkulkarni> to fail
21:49:43 <gpilz> but the test that is failing is expecting BadRequest and BadRequest is being thrown
21:49:48 <devkulkarni> so we might have to modify some of our test cases
21:49:50 <adrian_otto> any way for us to pin to an older version of tempest?
21:50:05 <devkulkarni> good question.
21:50:17 <datsun180b> i don't think we get to pick which tempest we use via test-reqs
21:50:18 <devkulkarni> we could do that
21:50:22 <gpilz> unless, for some reason, exception.BadRequest in the rest_client is not lining up with tempest_exceptions.BadRequest in the test code?
21:50:36 <devkulkarni> gpilz: very possible
21:50:43 <devkulkarni> the types won't match
21:50:55 <gpilz> the patch that i have in the queue depends upon a later version of the rest_client
21:50:57 <devkulkarni> datsun180b: you are probably right
21:51:30 <gpilz> if you use the version prior to the fix for https://bugs.launchpad.net/tempest/+bug/1407140
21:51:43 <devkulkarni> datsun180b: we might have to check with openstack-qa or openstack-infr folks on how to use specific version of tempest
21:51:47 <gpilz> a lot of my plan tests (and some Solum plant tests) will fail
21:52:20 <devkulkarni> solum folks: can we hold off sending patches for now
21:52:22 <datsun180b> i bet there's an argument to be made about solidarity/consistency across projects
21:52:29 <gpilz> i can
21:52:40 <datsun180b> devkulkarni: glad i submitted two patches like 90 seconds ago
21:52:54 <devkulkarni> datsun180b:  my comment was upon seeing those patches :D
21:53:05 <akshayc> :)
21:53:09 <datsun180b> hey i told kebray "today" and i meant it
21:53:14 <devkulkarni> we don't want zuul to keep its queue full when we know that gates are broken
21:53:35 <devkulkarni> acked datsun180b
21:53:46 <devkulkarni> but for now, lets hold off on any more patch submissions
21:54:05 <devkulkarni> okay, now that we have that covered
21:54:15 <devkulkarni> lets quickly go back to the tempest version discussion
21:54:21 <devkulkarni> how/can we pin the version
21:54:24 <devkulkarni> hi akshayc
21:54:40 <akshayc> hi
21:54:49 <devkulkarni> anyone want to follow up with either openstack-infra or openstack-qa folks to find out how to do that?
21:55:21 <devkulkarni> I can check that
21:55:44 <devkulkarni> anything else we should discuss today? 5 minute warning.
21:55:59 <akshayc> FYI, I have updated solum wiki
21:56:06 <datsun180b> nice
21:56:12 <akshayc> to include links for 2015 meetings
21:56:16 <devkulkarni> akshayc: cool.. what changes did you make?
21:56:18 <akshayc> meeting archives*
21:56:19 <devkulkarni> oh cool
21:56:23 <datsun180b> that reminds me, anyone have a clue how to update our readthedocs pages? adrian_otto ?
21:56:25 <devkulkarni> thanks for doing that akshayc
21:56:43 <devkulkarni> oh thats right, thanks for bringing that up datsun180b
21:56:57 <gpilz> they moved rest_client to tempest_lib
21:57:13 <devkulkarni> right gpil
21:57:29 <devkulkarni> thats what mkam's patch was supposed to fix on our end
21:57:35 <datsun180b> i'm making some fun cosmetic changes to the cli to make it easier to manage stuff without having to understand plans and assemblies and whatnot
21:57:56 <devkulkarni> datsun180b: cool
21:58:52 <devkulkarni> okay.. ending the meeting.. see you next week
21:58:58 <devkulkarni> #endmeeting