17:03:11 #startmeeting: tailgate 17:03:13 Meeting started Thu Aug 27 17:03:11 2015 UTC and is due to finish in 60 minutes. The chair is malini. Information about MeetBot at http://wiki.debian.org/MeetBot. 17:03:14 Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 17:03:16 The meeting name has been set to '__tailgate' 17:03:26 I hope I used the right meeting name 17:03:45 I remember screwing up once & still dont know what is the right one :) 17:03:59 anybody else here besides spyderdyne? 17:04:07 #topic Rollcall 17:04:16 here 17:04:21 me too 17:04:42 here again 17:04:50 guess we'll have ppl join along the way :) 17:05:46 #topic: Review Actions from http://eavesdrop.openstack.org/meetings/tailgate/2015/tailgate.2015-08-13-17.00.txt 17:06:22 something is wrong - we shud see the bot set the topic.. 17:06:24 hmmmm 17:06:27 o/ hi all actualy made it today, time conflicts last few mtgs 17:06:37 hey beisner! 17:06:44 hi malini ! 17:06:55 we cud find a different time if tht'll make it easier for all 17:07:11 generally i think either gema or i can be here, so no need to scoot it on my account 17:07:20 ok :) 17:07:26 going back to actions 17:07:37 spyderdyne to give more updates on HA testing 17:08:17 spyderdyne: do you have something for us? 17:08:56 i do 17:09:04 not quite complete 17:09:08 the floor is yours :) 17:09:41 #link http://spyderdyne.com/?p=192 Odin - A BASH Loader for Openstack Rally Scale Testing 17:09:58 I am adding the code this afternoon 17:10:16 that is neat 17:10:35 it is pretty basic, but it gets the job done nicely 17:10:51 neat 17:10:58 we can always build on top of it 17:11:07 its really nice to be able to set something in motion on friday afternoon and have it load tests and collect results all weekend 17:11:28 spyderdyne: can you just put the code in your personal repo for now? 17:11:31 spyderdyne for president 17:11:41 previously I was having to be up all hours of the day and night guestimating when tests would complete 17:11:44 lol 17:11:55 :D 17:12:10 i dont have a personal repo but am working on getting a corp sponsored one for it 17:12:22 for the near future it remains a blog entry 17:12:28 ok.. 17:12:39 will put it on github when i am able 17:12:56 great..moving onto next 17:12:59 jasonb to initiate the review of the questionaire on the ML 17:12:59 and handle review comments, review due to end early Sept (in case 17:12:59 people are on holidays) 17:13:32 can you push that out a week. i'm sorry 17:13:37 sure..no worries 17:13:42 #action: jasonb to initiate the review of the questionaire on the ML 17:13:43 and handle review comments, review due to end early Sept (in case 17:13:43 people are on holidays) 17:13:53 big push here to get our product done. didn't have much time left over 17:14:09 totally get it :/ 17:14:15 next one, 17:14:21 jasonb to figure out the mechanics of getting endpoints and gurumeditationreport and maybe create a simple script that does this 17:14:33 same :( 17:14:39 ok.. 17:14:41 it should be easy 17:14:49 #action: jasonb to figure out the mechanics of getting endpoints and gurumeditationreport and maybe create a simple script that does this 17:14:58 next one, 17:15:04 malini to try to write a test with tempest-lib that spins a VM and destroys is, as a POC 17:15:13 this will have to go over too :( 17:15:22 #action: malini to try to write a test with tempest-lib that spins a VM and destroys is, as a POC 17:15:45 next one, 17:15:47 jose-idar to start a spec for component tests for nova (user 17:15:47 side, no admin), so we can all contribute cases and maybe implement 17:15:47 them at some point , but at least start to get organised 17:15:58 jose-idar around? 17:16:20 moving it on 17:16:32 #action: jose-idar to start a spec for component tests for nova (user side, no admin), so we can all contribute cases and maybe implement them at some point , but at least start to get organised 17:16:39 #topic Mythos Project - Mythology themed Openstack Scale Testing Utilities 17:16:40 #topic Medusa - Capacity and Scale testing for Openstack using Rundeck, NMap, Phoronix, Netcat, and BASH 17:16:58 that's what im working on for the next 2 weeks 17:17:04 I like that :) 17:17:12 Odin would fall under Mythos 17:17:20 now I have to google odin 17:17:48 god of death, healing, and sorcery who gets his pick of half the warriors who enter Valhalla 17:17:56 :) 17:18:30 neat 17:18:32 linpack? 17:18:36 now openstack shud become Valhalla 17:18:39 (i'm hpc guy in background) 17:19:33 for now i have an image i have generated with test suites included and will use phonomic server/client for benchmarking 17:19:43 then it handles result sets 17:19:54 i am getting really good at killing large architectures 17:19:55 :) 17:20:01 what is a phonomic server? 17:20:17 the management utility for the phoronix test suite 17:20:38 an http frontend for execution, scheduling, discovery, and results aggregation 17:20:59 comes prepackaged in the leter versions of phoronix-test-suite 17:21:15 I have never seen http://www.phoronix-test-suite.com/ - but looks neat 17:21:23 i encourage everyone to test their kvm vs. vmware vs. aws performance 17:21:41 it will blow you away how much faster openstack is with disk, ram, and proc 17:21:44 i have seen some very slow kvm instances before 17:21:50 it would be nice to test for that 17:22:18 massive gains against AWS and decent gains against vmware 17:22:35 who allocates 2.03 processors? 17:22:35 lol 17:22:55 bookmarking it 17:23:19 qemulator is faster than full KVM in my results, FYI 17:24:41 I will just carry over the rest of actions for next week, since its mostly gema - we can use the time so spyderdyne can enlighten us more & we can also talk abt the tokyo talk 17:25:06 #ACTION: gema to continue with the strategy doc 17:25:19 # ACTION: gema to check what parts of keystone are accesible without admin on ubuntu's distro 17:25:30 #ACTION: gema to write a blog post about refstack-client running 17:25:30 experience 17:25:43 #ACTION: gema to send out the defcore details on actionable gaps 17:25:52 * malini thinks those tags are doing nothing 17:26:34 spyderdyne: do you want to talk more abt phoronix & ur findings? 17:27:09 i suggest everyone look at the linuxbenchmarking.org tests 17:27:12 they are very good 17:27:46 chess - processor, uses chess game AI and pitts cores against one another in simulated games 17:28:20 they have everythign from unreal tournament engine benchmarks, to apache compile time, to disk IO simulations and network stack 17:28:27 neat. i've used phoronix before to test out 3d vnc 17:28:31 quite good 17:29:05 it looks neat - I will start playing wit it 17:29:58 unfortunately it only covers local loopback network performance, so I have gotten creative 17:30:03 tell me how this sounds 17:30:49 medusa is an instance with a floating IP that runsinside the infrastructure on a large enough flavor to prevent other instances from living on the same hypervisor 17:31:02 rundeck provides a web portal to kick off 17:31:23 kick off nmap scans of all the instances on a shared neutron internal net and build hosts lists 17:32:15 (you can cross check with nuetron port-list) 17:32:28 menawhile the rundeck openstakc plugin finds all the nodes by tenant and can perform ssh cmmand execution on the in groups 17:33:16 we kick off nmap scans of the private networks that the instances on the shared network are attached to to build hosts lists, then use netcat to send requests to every one from everyone 17:33:23 massive network load 17:33:56 then phonomic server discovers all the instances as well and can initiate intance internal testing 17:34:56 i am working on setting the medusa node as a jmeter master instead and making every single test instance an active slave to do the same and actually record the results and ship them back to medusa 17:35:43 i like it 17:35:44 this allows us to perform full L2, L3, and IPv6 testing inside the overlay, still access our management host via floating IP address, and receive meaningful reports to take action on 17:36:02 future iterations will include logstash and zabbix server/agents 17:36:17 tht is neat! 17:36:19 i have been thinking what kind of testing i would do to compare ironic setup vs kvm for same application 17:36:30 this sounds like nice solution 17:36:34 once its ready i will use vargant to build these images on the fly so all you need to do is clone and exec 17:37:09 we are pretty fed up with rally leaving messes behind and not getting the same results under the same conditions each time 17:37:33 its also very difficult to troubleshoot and manage test plans, and the code is not very well documented 17:37:56 i have volunteered to help on the rally project with the docs and am learning how sphinx works now though 17:38:28 it keeps getting better, but isnt quite what we need yet 17:39:07 that's about all i have on the topic 17:39:25 i wonder something 17:39:39 i was about to turn on vxlan offload with new mellanox firmware 17:39:58 but i have been told that the silicon i have does drop ethernet frames 17:40:07 i would like a load test which would actually show that 17:40:27 same goes for dell broadcom chipsets of you arent already aware 17:40:28 so has to generate network load, but be sensitive to the network hardware falling on its face 17:40:42 bnx2x? 17:40:51 yes 17:41:03 wasn't aware but doesn't suprise 17:41:06 massive issues with OVS in an overlay mode at scale 17:41:30 i was looking at finding a way to orchestrate HTOP or something to collect things like that 17:42:07 for a local chipset test phoronix shoudl be able to cause most issues with tests to local stack i think 17:42:08 i was thinking of taking all VM's 17:42:24 take 2 at a time and run linpack 17:42:30 do for all permutations 17:42:54 then ramp up the number until all nodes are being used by one linpack 17:43:10 and see how bad the performance drops off 17:43:50 but i still need a way to get a baseline for network between 2 VM's which would show dropped frames 17:43:51 and other badness 17:44:22 what local chipset test would you recommend? 17:44:29 kloudbuster lets you pick specific hypervisors using scheduler hints and will create and delete them for each test run for you 17:44:51 best node to node generator i know of 17:45:04 then its just a matter of collecting the failures when they happen 17:45:12 hmmmm. 17:45:29 your right 17:45:31 you can do node to node, rack to rack, whatever topology you like 17:45:59 servers: hypervisor.domain.com 17:46:07 clients: hypervisor.domain.com 17:46:31 i'll do both 17:46:32 you just feed it a list so you give it nodes in whatever physical location you want to hit 17:46:35 lets see if they agree 17:46:41 good idea 17:47:02 i will take a look at linpack too 17:47:09 in my experience nothing breaks hardware like linpack 17:47:34 i managed to kill a cisco 9508 switch with kloudbuster 17:47:35 :) 17:47:45 ooooh neat 17:47:57 those things hover and configure themselves ;) 17:48:08 extra points for big number model #'s :) 17:48:49 have you tested with various gre/vxlan offload settings? 17:48:59 should that be under purvue of tests somehow? 17:49:13 (to generate complete test matrix) 17:49:46 we are only using VXLAN on overcloud with VLAN undercloud right now 17:50:40 our focus so far has been to scale out to thousands of VLANS to see how many total ports, L3 agents, networks, etc. we can support in a physical topology 17:51:04 thats really good stuff 17:51:19 and very tricky to get your head around, even for control plane only 17:51:23 working to be able to set up a builder that we can guarantee that automagically tells you how many net nodes, hpyervisors, control nodes, etc you need under an SLA 17:51:30 it is 17:51:58 my ambitions are much more humble 17:52:00 my secret goal is to show why you shouldnt run an overcloud and need to move to containerized services for multiple reasons 17:52:23 its been fun smashing things along the way but that's the loftier goal 17:52:35 i want to find out same info, but which passes criteria of not pushing data or control plane so hard that something breaks which can't be fixed with a script 17:52:42 (IE: needs a human) 17:52:43 * malini thinks should upgrade my goals 17:52:46 right 17:53:09 lol 17:53:17 i'm moving to kolla 17:53:22 so +1 17:53:54 so far i have just been building nuclear devices and setting them off in data centers but we are getting better at reporting and quantifying 17:54:07 malini, you can put spyderdyne down for first one to put a blog post 17:54:34 I am overwhelmed with all the info, I dont even know what to put him down for :D 17:54:37 do you run l2pop? 17:54:38 does it make a difference? 17:55:05 spyderdyne: do you want to actionize urself? I dont think I can do justice 17:55:36 what do i need to do? 17:55:57 'you can put spyderdyne down for first one to put a blog post' tht was from jasonb 17:56:16 which means malini, me, and gema all get action items :) 17:56:17 just drop in etherpad? first post!? 17:56:24 spyderdyne is done 17:56:43 hmm..so tht means now we get to work :D 17:56:50 sureā€¦sounds good 17:57:02 switch to tokyo talk? 17:57:06 #info: spyderdyne gets the first blop post award 17:57:15 Sure 17:57:22 #topic: Tokyo talk 17:57:49 so our talk is in alternative accepted state - I am not sure where tht puts us in the possibility spectrum 17:58:08 Anybody else knows? 17:58:17 feasible, but not as interesting as something else maybe 17:58:31 i was told typically the committee accepts 10 talks and 2 alternates 17:58:32 im sure there are several talks that have to be cancelled each summit 17:58:42 so if anybody drops there is good chance 17:58:56 but thsi may be a datapoint from older summit, so the numbers might be different now 17:59:06 i dont know how any of it works 17:59:09 :) 17:59:30 we shud just start prepping as if we were accepted ? Its going to be a good exercise for us 17:59:36 i think so 17:59:40 sure 17:59:42 if anybody drops we have 50/50 chance maybe 17:59:46 but i'm not sure 17:59:47 which talk was accepted? 17:59:53 which track? 18:00:02 we shud just lock some of those accepted ppl in a hotel room ..BWAHAHAHAA 18:00:30 chking the track now 18:00:33 do we all have to fill out the web form saying we are going to be in tokyo and give our phone number? 18:00:37 i can work up a presentation on tools, and if we dont use it we can put it on slideshare anyway 18:00:38 what if we all dont'? 18:01:56 I guess we should still do it (if we get to) if atleast some of us go 18:02:18 I am not sure if I can go with the travel budget stuff 18:02:29 you have free pass 18:02:50 i booked hotel last night 18:02:51 i have paid my own way the two summits i attended, and i plan to go as long as nothing crazy comes up preventing me from it 18:02:53 still need flight 18:03:51 did you go to paris? 18:03:59 grr..I cant find the talk submissions :/ 18:03:59 that was expensiiiiivvveeee 18:04:14 tokyo will be too :( 18:04:24 i didnt go to paris 18:04:35 our currency is too weak there :) 18:04:52 medium cappucino 18:04:54 and i would probably apply for citizenship while is was there and never come back 18:04:55 $7 18:05:35 we are out of time & I never noticed. I'll end meeting bust we can continue chat here 18:05:35 malini, we probably should do roll call for who is going and can give the talk 18:06:02 jasonsb: ok..lets do tht now before ending 18:06:09 i was thihnking on ML 18:06:48 sounds good..can does it have to be just the listed speakers or anybody in the ML? 18:07:27 technically, or just general concept? 18:07:39 technically the speakers can still be adjusted, no? 18:07:56 yes 18:08:15 i have no opinion 18:08:17 I am not sure if its technically allowed - but I know ppl have done tht 18:08:26 if the talk actually happens i can be there 18:08:46 same here - I have a better chance if it actually happens 18:09:08 i have started eating sushi so i dont starve to death when i get there :) 18:09:18 yumm..yummm... 18:09:42 you can eat like a king in asia 18:10:05 yes :) 18:10:13 er queen 18:10:23 you meant eat royally :) 18:10:33 being 2 to 3 times the size of the average resident i will need to 18:11:04 hopefully i dont get pulled into competitive sumo and miss the conference 18:11:09 :D 18:11:23 jasonsb: can u start the thread to poll who is going? 18:12:26 ending for real now 18:12:29 #endmeeting