00:01:41 #startmeeting CongressTeamMeeting 00:01:42 Meeting started Thu Sep 8 00:01:41 2016 UTC and is due to finish in 60 minutes. The chair is thinrichs. Information about MeetBot at http://wiki.debian.org/MeetBot. 00:01:43 Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 00:01:46 The meeting name has been set to 'congressteammeeting' 00:02:03 hello! 00:02:11 hi all 00:02:56 Hi all 00:03:04 Here's agenda for this week.. 00:03:06 hi 00:03:23 1. Testing for newton release 00:03:38 2. First project teams gathering 00:03:52 3. Status 00:04:12 Anything else? 00:04:44 We'll start with number 2 00:04:50 #topic First project teams gathering 00:05:18 Here's Thierry's announcement about how the OpenStack conference and design summit are being split 00:05:21 #link http://lists.openstack.org/pipermail/openstack-dev/2016-September/102981.html 00:05:36 Or at least, that's the announcement for the first occurrence of the design summit after the split 00:06:59 One of the ramifications is that the Ocata session will be slightly shorter than normal. 00:07:18 The bigger consequence is that there will be 4 OpenStack events per year instead of 2. 00:08:03 We'll all need to figure out how travel budgets will impact our ability to congregate and coordinate which events we'll be attending. 00:08:21 Questions, comments? 00:08:23 Have you heard whether the Project Team Gatherings will be held outside of North America once a year? 00:08:46 I have no info outside the mailing list. 00:09:01 So no. I don't know where they're planning on holding events. 00:09:04 OK - there's nothing on the official FAQ either. 00:09:22 I'd guess they'd rotate it like they do today, given how many international contributors there are 00:09:39 that would be fair 00:10:04 #topic Testing for newton 00:10:45 Today I was trying to test HA/HT and realized I don't know exactly how to do it. 00:10:52 The HA guides are a start. 00:11:11 Has anyone set up the full HAHT of Congress and done manual testing? 00:11:29 Multiple PE+API nodes, 1 DSD node, a load-balancer 00:11:45 running with the other OpenStack components like Nova/Neutron 00:11:54 not yet. still working to finish automated testing. 00:12:22 not yet 00:12:28 we need to add load-balancer instructions to guide. 00:13:02 Do we know how to set it up with devstack? 00:13:05 It'd be nice if we could spin up devstack to get all the other components. 00:13:19 Then to spin up Congress on 2 other VMs 00:13:53 Maybe 3 VMs (2 for PE+API and 1 DSD) 00:13:59 And then 1 VM for a load balancer 00:14:27 should be relatively straightforward. 00:14:49 devstack problem is we cant start 2 PE+API like we discussed before 00:15:04 because they start on same node? 00:15:08 just need to start other instances of congress using the same configs, after devstack is done setting up. 00:15:25 ramineni_: Even if we could start 2 PE+APIs inside devstack, we'd want to test on separate nodes. 00:15:34 right 00:15:35 Devstack only runs on 1 node, right? 00:15:55 devstack can run on multiple nodes 00:15:57 multinode is there i suppose..but i dont know how that works 00:16:08 you can also run multiple processes on a single instance too 00:16:11 ekcs: If I remember right, a standalone Congress install isn't trivial. And we'd need to add rabbitmq 00:16:15 might need to modify devstack to make that happen though 00:16:16 http://docs.openstack.org/developer/devstack/guides/multinode-lab.html 00:17:16 aimeeu: thanks for the link 00:17:27 I haven't yet tried to do that 00:17:29 clarkb: is that the right link to help us, would you say? 00:17:45 Does anyone know how other teams test their HA deployments? 00:17:47 thinrichs: right… 00:18:07 Do they automate the whole thing with puppet/ansible/…? 00:18:12 thats basically how the gate multinode works, we run devstack twice once on each node with different settings for each 00:19:08 clarkb: naive question: do the 2 devstack nodes communicate correctly over rabbitmq? 00:19:17 thinrichs: yes 00:19:40 its a functionaing two node cloud. One node is controller + compute all in one and the second node is a compute host 00:20:52 hmm that’d be helpful to leverage devstack to set up additional nodes. cuz manual install in complex like thinrichs said. will have to slog through devstack settings I guess to do it correctly. 00:21:22 and add to the plugin piece. 00:21:31 Seems we'd want 4 devstack nodes then: 1 with everything but congress, 2 with just Congress in PE+API mode, and 1 with just Congress in DSD mode. 00:21:53 Then we'd just need a LB, and we could (manually) test. 00:22:10 thinrichs: is this helpful? Manual validation of cinder a/a patches: http://gorka.eguileor.com/manual-validation-of-cinder-aa-patches/ 00:22:14 That doesn't sound too bad, since we probably all have a devstack VM we can clone a few times 00:23:06 aimeeu: that looks really helpful—exactly the kind of testing we're trying to do 00:23:34 I'll take an action item to try to get something working and report back. 00:23:49 Anyone else want to give it a try, so we don't have 1 point of failure? 00:24:30 I can take a look too, but atm I’m prioritizing 00:24:35 same node diff process testing. 00:24:44 ekcs: you're busy getting the auto testing in place 00:24:53 ekcs: right that that's higher priority 00:25:00 I should be able to find the time to do this 00:25:25 #action thinrichs will set up multi-node manual testing and report back on how to do it 00:25:48 Moving on... 00:25:53 #topic Status 00:26:05 ramineni_: want to give your status update? 00:26:39 sure 00:27:23 added a non-voting multiple PE gate job, so we could test it properly 00:27:50 and fixed some of the issues we came across the HA testing 00:28:06 thats it from my side 00:28:19 Any of those issues during HA testing worrisome? 00:28:56 still no stable yet ..hopefully we will get it done.. eric is working on that too 00:29:16 ok 00:29:30 aimeeu: want to go next? 00:29:35 ok 00:29:45 I've been looking into configuring Congress for pydevd remote debugging. Looked at several OpenStack projects - I like the way Trove handles it. Tried implementing it in Congress - didn't work but I am learning quite a bit about how Congress starts up; also looking at historical code changes in github regarding congress installation on devstack. I have 00:29:45 questions that I'll put into an email later after I've organized my thoughts. 00:30:04 that's it for me 00:30:49 aimeeu: thanks. Let us know if you need any help digging through the startup code 00:30:58 thanks - will do 00:31:13 ekcs: want to go next? 00:31:17 sure. 00:31:24 Mostly resolving testing issues with HAHT. Getting the tempest test to stabilize. Still not sure why occassional errors occur with instances not converging. 00:31:27 Hoping that local multi-process testing will shed more light. I have a basic local multi-process test framework setup. #link https://review.openstack.org/#/c/358927/ 00:31:28 patch 358927 - congress - WIP - local HA tests 00:32:29 it’s working on my machine, but somehow failing on gerrit. I’m hoping it’s not because of a system config that disallows spawning new processes and http’ing them. 00:33:15 but even if it ultimately doesn’t work in gerrit it’s still useful tests to have in place. 00:33:34 other than that, just identifying scattered issues and bugs as I go. 00:34:23 you cant bind to privileged ports in those jobs but you can use many of the other thousands of ports 00:34:35 running tox should not require root 00:35:07 clarkb: spinning up new processes is allowed (e.g. with subprocess.Popen)? 00:35:12 clarkb: ah great. 00:35:23 thinrichs: yes 00:35:34 I’m using ports 4001 and 4002. 00:36:46 We're passing cwd=helper.root_path(). Could that be a problem on a different machine? 00:37:08 thinrichs: shouldn’t be because it’s relative to the helper.py file. 00:37:28 thinrichs: but still worth checking at some point. 00:38:17 I can do a quick status update 00:38:23 anyway the failure is that I Popen a new congress server, and then http it, but connertion fails exceeding retry limit. 00:40:06 thinrichs: didn’t mean to cut you off. 00:40:15 Nothing jumps to mind, other than a firewall on the OS. 00:40:20 ekcs: no problem 00:40:31 Did some manual testing. 00:41:05 Reasonably comprehensive within devstack. Not trying corner-cases, but just hitting all the core functionality. 00:41:21 Some testing across processes; seems good too. 00:41:39 Getting a real HAHT env set up is next 00:42:01 Did notice we're returning some stack-traces as part of the HTTP error message. 00:42:15 That's some low-hanging fruit I'd say. 00:42:18 #link https://bugs.launchpad.net/congress/+bug/1620868 00:42:18 Launchpad bug 1620868 in congress "Stack trace returned to end-user" [Undecided,New] 00:42:42 aimeeu: might be a good one to take on 00:43:00 yep I was just looking at it 00:43:00 That's it from me. 00:43:14 aimeeu: cool! 00:43:33 Just found out I need to leave the meeting early today (in 2 min). 00:43:39 Anything else for me before I take off? 00:43:42 #topic open discussion 00:44:08 I'll try to make ekcs chair before I leave… 00:44:10 #chair ekcs 00:44:10 Current chairs: ekcs thinrichs 00:44:22 im on vacation next week , wont be able to attend meting 00:44:33 clarkb: any suggestions for how to see if a port may be blocked on a gerrit vm? 00:44:39 ramineni_: okay. Thanks. Enjoy your vacation! 00:44:53 That reminds me. I can't make the meeting either. I'll be traveling. 00:44:58 ekcs: I can look at job logs in a minute 00:45:09 I'm over in #openstack-infra 00:45:20 ekcs: can you run next week's meeting? 00:45:28 clarkb: got it thanks so much. 00:45:34 thinrichs: sure thing. 00:45:50 Got to run. See you all later! 00:46:01 later. 00:47:16 so… anything else to discuss? or should we just end early? 00:47:26 anyone followed why the design summit split? 00:48:00 4 events a year is A LOT 00:48:21 I heard it was to facilitate better inter-team collaboration 00:48:57 as well as to provide developers with a better working atmosphere - something about too many parties at the regular Summit (gossip of course) 00:49:08 hahaha. 00:49:33 the Foundation said it was going to look at boring cities for the gatherings 00:50:03 ekcs: I think that your pe1 and pe2 objects are local to setUp and will be GC'd so there isn't anything listening for the cleint to connect to 00:50:51 but I agree - 4x/year is a lot. The FAQ said it's up to individual teams to decide which events to attend 00:51:55 clarkb: oh thanks a lot for the help! But Hmm I didn’t know that it mattered if the process handle got GC’ed. 00:52:59 if your process isn't backgrounding that means it gets killed I think. (and it probably shouldn't background as init won't know what to do with it) 00:53:01 clarkb: plus, the failure is occurring within setUp where I http to the new process just after starting up. 00:53:55 ah I see that now maybe you aren't waiting long enough for the listening process to actually bind? 00:54:44 also typically we will recommend you listen on port 0 then get the actual port somehow to avoid conflicts between test 00:54:46 clarkb: I could retry longer. But I tihnk it’s up to 20s now. should be plenty? 00:55:00 depends on the process you are starting. I don't know how long it will take 00:55:43 clarkb: oh that sounds like something I should learn more about. starting on port 0 then get port. 00:56:00 clarkb: is there an example of that somewhere? Thanks so much! I’m very new to this. 00:56:13 also both processes use the same sqlite db (not sure sqlite will like that) and the path is rooted to a path that may not exist? not sure how that gets translated to a disk path 00:57:41 clarkb: Right I’ll need to be careful with sqlite. Thanks! 00:58:07 clarkb: it may be that the path translation works differently between my machine and gerrit. thanks for the pointer! 00:58:26 anyway we’re coming up to time. anything last minute? 00:58:47 I don't recall the location of any good port 0 usage off the top of my head. But when you use port zero linux gives you a free port to listen on then you have to determine which port it is someway. YOu can ask the scoket object ni python. But this ensures you don't have conflicting port use between threads or processes 00:59:31 clarkb: got it thanks for the crash course! 00:59:47 Alright time’s up. 00:59:52 Thanks all! 01:00:03 #endmeeting