02:02:35 <adreznec> #startmeeting PowerVM CI Meeting
02:02:36 <openstack> Meeting started Wed Nov  9 02:02:35 2016 UTC and is due to finish in 60 minutes.  The chair is adreznec. Information about MeetBot at http://wiki.debian.org/MeetBot.
02:02:37 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
02:02:39 <openstack> The meeting name has been set to 'powervm_ci_meeting'
02:03:00 <adreznec> #topic Current status
02:03:41 <adreznec> So, I haven't had a chance to look at the latest revision of your patchset that was put up wangqwsh. Do you want to run us through status?
02:04:28 <wangqwsh> 1. pypowervm adapter patch's done. 2. we can install openstack with osa
02:04:52 <thorst_> whoa, that's pretty good.
02:05:17 <wangqwsh> I update the script related to tempest variables
02:05:20 <thorst_> so what's next there?  Tempest tests?
02:05:20 <adreznec> Awesome, you've gotten things running in the staging environment then?
02:05:23 <thorst_> ahh
02:05:54 <wangqwsh> there is an issue in galera while running tempest
02:06:08 <wangqwsh> i am trying to find the root casue.
02:06:19 <wangqwsh> casue -> cause
02:06:27 <adreznec> In galera while running tempest? Are we seeing database failures?
02:06:35 <wangqwsh> yes
02:06:45 <wangqwsh> the mysql is not running
02:06:46 <adreznec> Hmm ok
02:06:57 <adreznec> How much memory are we assigning to these VMs?
02:07:05 <wangqwsh> 8g
02:07:08 <adreznec> I know we've been seeing some issues in OSA AIO testing with 8g lately
02:07:15 <thorst_> that's too little?
02:07:15 <wangqwsh> yes
02:07:17 <adreznec> lately being yesterday/today
02:07:32 <adreznec> Yeah, issues with mysql running out of memory. Still trying to debug exactly why
02:07:34 <thorst_> yikes...that's a fair amount.
02:07:48 <thorst_> Lets definitely not double that...but instead maybe step to 10...
02:08:00 <adreznec> It's the absolute minimum req for OSA AIO FWIW
02:08:18 <thorst_> and it probably doesn't help that we're SMT-8
02:08:27 <adreznec> Recommended is 16
02:08:27 <thorst_> though I doubt that has the same impact to OSA as it does devstck
02:08:37 <thorst_> yikes...and this would be all our nodes...
02:08:42 <thorst_> not just OSA nodes?
02:08:42 <adreznec> Yeah, I'm not sure how much we can tie back to that
02:08:54 <adreznec> Unless we decide to split nodes
02:09:00 <adreznec> I know, we already have memory issues
02:09:17 <adreznec> The OSA gate upstream uses 8gb
02:09:30 <thorst_> hmm...well, it generally isn't too stressed (each node runs 4 at once, each with 8 GB, but has 128 GB memory)
02:09:37 <adreznec> So I think we should be able to make it work with that amount with tweaking... just not sure what's the issue
02:09:43 <thorst_> so we have SOME space...but we need to leave more for the VMs during the tempest runs themselves.
02:09:50 <adreznec> Right
02:10:08 <thorst_> lets maybe try 12 and back track from there...
02:10:09 <adreznec> wangqwsh: how far into tempest are you making it?
02:11:06 <wangqwsh> 3h
02:11:17 <adreznec> Whoa
02:11:24 <adreznec> Tempest definitely shouldn't take that long to run
02:11:57 <wangqwsh> because I restart the db by manual :)
02:12:06 <adreznec> Ah, I see
02:12:20 <thorst_> well...
02:12:28 <wangqwsh> if not, a few seconds
02:12:31 <thorst_> tempest used to take 4 hours until we bumped up the threads
02:13:36 <adreznec> Yeah, I was just hoping we wouldn't have that same issue here
02:14:52 <thorst_> yeah....just need to make sure we have the right thread count
02:15:03 <thorst_> but it sounds like overall we're making good progress there  :-)
02:15:14 <adreznec> All right, so wangqwsh do you want to try bumping things to 10gb (or 12gb) and see if we still have galera issues?
02:15:25 <wangqwsh> sure.
02:15:48 <adreznec> All right
02:16:08 <wangqwsh> i used this image for CI, Ubuntu_16-04-01_2g.img
02:16:11 <adreznec> #action wangqwsh to try increasing OSA VM memory to 10gb/12gb and retesting
02:16:19 <wangqwsh> is that fine?
02:16:27 <adreznec> That should be right
02:16:34 <wangqwsh> ok
02:17:01 <adreznec> All right. On to other status
02:17:29 <adreznec> thorst_ has his well named fifo_pipo patch on deck
02:17:42 <thorst_> actually efried has taken over driving that
02:17:51 <thorst_> and PipesOutLightsOut has become RESTAPIPipe
02:17:57 <thorst_> I like my name better.
02:18:15 <adreznec> It'll always be fifo_pipo to me
02:18:17 <adreznec> and git
02:18:29 <thorst_> :-)
02:18:52 <thorst_> so that's under testing
02:18:58 <adreznec> All right
02:18:59 <thorst_> making progress...hopefully more known tomorrow
02:19:07 <adreznec> I believe esberglu was testing that, right?
02:20:16 <thorst_> yep
02:20:55 <adreznec> #action esberglu to continue testing RESTAPIPipe patch with assistance from efried
02:21:05 <adreznec> Who are here only in spirit
02:21:13 <adreznec> All righty then
02:21:34 <adreznec> #topic Next steps
02:21:57 <adreznec> Looking forward then, it seems like things are starting to shape up
02:22:22 <thorst_> definitely.  I think the next thing - perhaps out of CI - is the pypowervm integrated into global-requirements
02:22:32 <thorst_> asking for that to be done...but - not really tied to CI
02:22:41 <adreznec> wangqwsh: it sounds like next up after the galera you're going to be digging into tempest config?
02:22:59 <wangqwsh> yes
02:23:09 <adreznec> thorst_: yeah, I spent some time with that today. I'm having some issues with getting setuptools to build with appropriate versioning
02:23:42 <thorst_> :-/  alright.  We need to get that into the priority view next week...once CI stabilizes
02:23:46 <adreznec> I talked with esberglu about how he supposedly got setuptools/pbr to disregard PEP440 before, but he couldn't remember and also had no notes
02:24:10 <adreznec> I'll continue digging and see what I can get worked out here
02:24:27 <adreznec> Hopefully the necessary pypowervm version will be up tomorrow
02:24:56 <adreznec> #action adreznec to get pypowervm update published to pypi to drive towards getting pypowervm in g-r
02:25:16 <thorst_> yikes...I thought he had a wiki page of notes somewhere on it...
02:25:26 <adreznec> Nope...
02:25:42 <thorst_> gr8
02:25:46 <adreznec> unfortunate because this is the same issue I hit last time I tried
02:26:01 <thorst_> maybe we ask him to do it from scratch and he stumbles upon the same result?
02:26:14 <adreznec> This is where I'd normally put a random, semi-related giphy
02:26:24 <thorst_> heh
02:26:27 <adreznec> I'll give it a bit more trying tomorrow
02:26:31 <adreznec> and if I can't get it I'll walk over to his desk
02:26:36 <thorst_> sounds good
02:26:38 <adreznec> and we'll figure it out
02:26:58 <thorst_> agenda for next week...getting the logs published on nova patches...
02:27:11 <adreznec> So I know esberglu isn't here, but it sounds like mriedem wants us to change the way we're handling our skip list?
02:28:03 <thorst_> yeah, I sent that to esberglu already and he took that as a TODO
02:28:07 <thorst_> but...for good measure
02:28:23 <thorst_> #action esberglu to switch temptest skip list from name to test id
02:28:29 <adreznec> Yeah, I saw that mentioned earlier
02:28:47 <adreznec> Are we going to have to start maintaining 2 skip lists now?
02:28:52 <adreznec> One for dsvm and one for osa?
02:29:09 <thorst_> I don't see a need ... yet
02:29:15 <adreznec> Right
02:29:25 <thorst_> might need for integrated nova driver versus out of tree though
02:29:41 <adreznec> in osa since we're using ovs we'll be supporting a theoretical superset of function
02:29:56 <adreznec> sec groups, l3, etc
02:30:30 <thorst_> right...but lets start small and keep consistent
02:30:35 <thorst_> then add complexity as we go
02:30:36 <adreznec> k
02:30:52 <adreznec> integrated vs OOT we'll definitely need two lists, at least at first
02:31:20 <adreznec> All right, so wangqwsh should be fine using the existing tempest configs then
02:31:26 <adreznec> in terms of tests to run/skip
02:32:10 <adreznec> Ok, so I know we're over time here
02:32:20 <thorst_> its because we're all watching florida
02:32:37 <adreznec> It's always Florida
02:32:41 <thorst_> any other aspects we need to cover?
02:32:49 <adreznec> wangqwsh anything else from your end?
02:32:55 <wangqwsh> nope
02:33:12 <adreznec> All right
02:33:21 <adreznec> That's a wrap then
02:33:24 <adreznec> Thanks all!
02:33:27 <adreznec> #endmeeting