17:03:11 <malini> #startmeeting: tailgate
17:03:13 <openstack> Meeting started Thu Aug 27 17:03:11 2015 UTC and is due to finish in 60 minutes.  The chair is malini. Information about MeetBot at http://wiki.debian.org/MeetBot.
17:03:14 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
17:03:16 <openstack> The meeting name has been set to '__tailgate'
17:03:26 <malini> I hope I used the right meeting name
17:03:45 <malini> I remember screwing up once & still dont know what is the right one :)
17:03:59 <malini> anybody else here besides spyderdyne?
17:04:07 <malini> #topic Rollcall
17:04:16 <spyderdyne> here
17:04:21 <malini> me too
17:04:42 <spyderdyne> here again
17:04:50 <malini> guess we'll have ppl join along the way :)
17:05:46 <malini> #topic: Review Actions from http://eavesdrop.openstack.org/meetings/tailgate/2015/tailgate.2015-08-13-17.00.txt
17:06:22 <malini> something is wrong - we shud see the bot set the topic..
17:06:24 <malini> hmmmm
17:06:27 <beisner> o/  hi all actualy made it today, time conflicts last few mtgs
17:06:37 <malini> hey beisner!
17:06:44 <beisner> hi malini !
17:06:55 <malini> we cud find a different time if tht'll make it easier for all
17:07:11 <beisner> generally i think either gema or i can be here, so no need to scoot it on my account
17:07:20 <malini> ok :)
17:07:26 <malini> going back to actions
17:07:37 <malini> spyderdyne to give more updates on HA testing
17:08:17 <malini> spyderdyne: do you have something for us?
17:08:56 <spyderdyne> i do
17:09:04 <spyderdyne> not quite complete
17:09:08 <malini> the floor is yours :)
17:09:41 <spyderdyne> #link http://spyderdyne.com/?p=192 Odin - A BASH Loader for Openstack Rally Scale Testing
17:09:58 <spyderdyne> I am adding the code this afternoon
17:10:16 <malini> that is neat
17:10:35 <spyderdyne> it is pretty basic, but it gets the job done nicely
17:10:51 <jasonsb> neat
17:10:58 <malini> we can always build on top of it
17:11:07 <spyderdyne> its really nice to be able to set something in motion on friday afternoon and have it load tests and collect results all weekend
17:11:28 <malini> spyderdyne: can you just put the code in your personal repo for now?
17:11:31 <jasonsb> spyderdyne for president
17:11:41 <spyderdyne> previously I was having to be up all hours of the day and night guestimating when tests would complete
17:11:44 <spyderdyne> lol
17:11:55 <malini> :D
17:12:10 <spyderdyne> i dont have a personal repo but am working on getting a corp sponsored one for it
17:12:22 <spyderdyne> for the near future it remains a blog entry
17:12:28 <malini> ok..
17:12:39 <spyderdyne> will put it on github when i am able
17:12:56 <malini> great..moving onto next
17:12:59 <malini> jasonb to initiate the review of the questionaire on the ML
17:12:59 <malini> and handle review comments, review due to end early Sept (in case
17:12:59 <malini> people are on holidays)
17:13:32 <jasonsb> can you push that out a week.  i'm sorry
17:13:37 <malini> sure..no worries
17:13:42 <malini> #action: jasonb to initiate the review of the questionaire on the ML
17:13:43 <malini> and handle review comments, review due to end early Sept (in case
17:13:43 <malini> people are on holidays)
17:13:53 <jasonsb> big push here to get our product done.  didn't have much time left over
17:14:09 <malini> totally get it :/
17:14:15 <malini> next one,
17:14:21 <malini> jasonb to figure out the mechanics of getting endpoints and gurumeditationreport and maybe create a simple script that does this
17:14:33 <jasonsb> same :(
17:14:39 <malini> ok..
17:14:41 <jasonsb> it should be easy
17:14:49 <malini> #action: jasonb to figure out the mechanics of getting endpoints and gurumeditationreport and maybe create a simple script that does this
17:14:58 <malini> next one,
17:15:04 <malini> malini to try to write a test with tempest-lib that spins a VM and destroys is, as a POC
17:15:13 <malini> this will have to go over too :(
17:15:22 <malini> #action: malini to try to write a test with tempest-lib that spins a VM and destroys is, as a POC
17:15:45 <malini> next one,
17:15:47 <malini> jose-idar to start a spec for component tests for nova (user
17:15:47 <malini> side, no admin), so we can all contribute cases and maybe implement
17:15:47 <malini> them at some point , but at least start to get organised
17:15:58 <malini> jose-idar around?
17:16:20 <malini> moving it on
17:16:32 <malini> #action: jose-idar to start a spec for component tests for nova (user side, no admin), so we can all contribute cases and maybe implement them at some point , but at least start to get organised
17:16:39 <spyderdyne> #topic Mythos Project - Mythology themed Openstack Scale Testing Utilities
17:16:40 <spyderdyne> #topic Medusa - Capacity and Scale testing for Openstack using Rundeck, NMap, Phoronix, Netcat, and BASH
17:16:58 <spyderdyne> that's what im working on for the next 2 weeks
17:17:04 <malini> I like that :)
17:17:12 <spyderdyne> Odin would fall under Mythos
17:17:20 <malini> now I have to google odin
17:17:48 <spyderdyne> god of death, healing, and sorcery who gets his pick of half the warriors who enter Valhalla
17:17:56 <spyderdyne> :)
17:18:30 <jasonsb> neat
17:18:32 <jasonsb> linpack?
17:18:36 <malini> now openstack shud become Valhalla
17:18:39 <jasonsb> (i'm hpc guy in background)
17:19:33 <spyderdyne> for now i have an image i have generated with test suites included and will use phonomic server/client for benchmarking
17:19:43 <spyderdyne> then it handles result sets
17:19:54 <spyderdyne> i am getting really good at killing large architectures
17:19:55 <spyderdyne> :)
17:20:01 <malini> what is a phonomic server?
17:20:17 <spyderdyne> the management utility for the phoronix test suite
17:20:38 <spyderdyne> an http frontend for execution, scheduling, discovery, and results aggregation
17:20:59 <spyderdyne> comes prepackaged in the leter versions of phoronix-test-suite
17:21:15 <malini> I have never seen http://www.phoronix-test-suite.com/ - but looks neat
17:21:23 <spyderdyne> i encourage everyone to test their kvm vs. vmware vs. aws performance
17:21:41 <spyderdyne> it will blow you away how much faster openstack is with disk, ram, and proc
17:21:44 <jasonsb> i have seen some very slow kvm instances before
17:21:50 <jasonsb> it would be nice to test for that
17:22:18 <spyderdyne> massive gains against AWS and decent gains against vmware
17:22:35 <spyderdyne> who allocates 2.03 processors?
17:22:35 <spyderdyne> lol
17:22:55 <malini> bookmarking it
17:23:19 <spyderdyne> qemulator is faster than full KVM in my results, FYI
17:24:41 <malini> I will just carry over the rest of actions for next week, since its mostly gema - we can use the time so spyderdyne can enlighten us more & we can also talk abt the tokyo talk
17:25:06 <malini> #ACTION: gema to continue with the strategy doc
17:25:19 <malini> # ACTION: gema to check what parts of keystone are accesible without admin on ubuntu's distro
17:25:30 <malini> #ACTION: gema to write a blog post about refstack-client running
17:25:30 <malini> experience
17:25:43 <malini> #ACTION: gema to send out the defcore details on actionable gaps
17:25:52 * malini thinks those tags are doing nothing
17:26:34 <malini> spyderdyne: do you want to talk more abt phoronix & ur findings?
17:27:09 <spyderdyne> i suggest everyone look at the linuxbenchmarking.org tests
17:27:12 <spyderdyne> they are very good
17:27:46 <spyderdyne> chess - processor, uses chess game AI and pitts cores against one another in simulated games
17:28:20 <spyderdyne> they have everythign from unreal tournament engine benchmarks, to apache compile time, to disk IO simulations and network stack
17:28:27 <jasonsb> neat.  i've used phoronix before to test out 3d vnc
17:28:31 <jasonsb> quite good
17:29:05 <malini> it looks neat - I will start playing wit it
17:29:58 <spyderdyne> unfortunately it only covers local loopback network performance, so I have gotten creative
17:30:03 <spyderdyne> tell me how this sounds
17:30:49 <spyderdyne> medusa is an instance with a floating IP that runsinside the infrastructure on a large enough flavor to prevent other instances from living on the same hypervisor
17:31:02 <spyderdyne> rundeck provides a web portal to kick off
17:31:23 <spyderdyne> kick off nmap scans of all the instances on a shared neutron internal net and build hosts lists
17:32:15 <jasonsb> (you can cross check with nuetron port-list)
17:32:28 <spyderdyne> menawhile the rundeck openstakc plugin finds all the nodes by tenant and can perform ssh cmmand execution on the in groups
17:33:16 <spyderdyne> we kick off nmap scans of the private networks that the instances on the shared network are attached to to build hosts lists, then use netcat to send requests to every one from everyone
17:33:23 <spyderdyne> massive network load
17:33:56 <spyderdyne> then phonomic server discovers all the instances as well and can initiate intance internal testing
17:34:56 <spyderdyne> i am working on setting the medusa node as a jmeter master instead and making every single test instance an active slave to do the same and actually record the results and ship them back to medusa
17:35:43 <jasonsb> i like it
17:35:44 <spyderdyne> this allows us to perform full L2, L3, and IPv6 testing inside the overlay, still access our management host via floating IP address, and receive meaningful reports to take action on
17:36:02 <spyderdyne> future iterations will include logstash and zabbix server/agents
17:36:17 <malini> tht is neat!
17:36:19 <jasonsb> i have been thinking what kind of testing i would do to compare ironic setup vs kvm for same application
17:36:30 <jasonsb> this sounds like nice solution
17:36:34 <spyderdyne> once its ready i will use vargant to build these images on the fly so all you need to do is clone and exec
17:37:09 <spyderdyne> we are pretty fed up with rally leaving messes behind and not getting the same results under the same conditions each time
17:37:33 <spyderdyne> its also very difficult to troubleshoot and manage test plans, and the code is not very well documented
17:37:56 <spyderdyne> i have volunteered to help on the rally project with the docs and am learning how sphinx works now though
17:38:28 <spyderdyne> it keeps getting better, but isnt quite what we need yet
17:39:07 <spyderdyne> that's about all i have on the topic
17:39:25 <jasonsb> i wonder something
17:39:39 <jasonsb> i was about to turn on vxlan offload with new mellanox firmware
17:39:58 <jasonsb> but i have been told that the silicon i have does drop ethernet frames
17:40:07 <jasonsb> i would like a load test which would actually show that
17:40:27 <spyderdyne> same goes for dell broadcom chipsets of you arent already aware
17:40:28 <jasonsb> so has to generate network load, but be sensitive to the network hardware falling on its face
17:40:42 <jasonsb> bnx2x?
17:40:51 <spyderdyne> yes
17:41:03 <jasonsb> wasn't aware but doesn't suprise
17:41:06 <spyderdyne> massive issues with OVS in an overlay mode at scale
17:41:30 <spyderdyne> i was looking at finding a way to orchestrate HTOP or something to collect things like that
17:42:07 <spyderdyne> for a local chipset test phoronix shoudl be able to cause most issues with tests to local stack i think
17:42:08 <jasonsb> i was thinking of taking all VM's
17:42:24 <jasonsb> take 2 at a time and run linpack
17:42:30 <jasonsb> do for all permutations
17:42:54 <jasonsb> then ramp up the number until all nodes are being used by one linpack
17:43:10 <jasonsb> and see how bad the performance drops off
17:43:50 <jasonsb> but i still need a way to get a baseline for network between 2 VM's which would show dropped frames
17:43:51 <jasonsb> and other badness
17:44:22 <jasonsb> what local chipset test would you recommend?
17:44:29 <spyderdyne> kloudbuster lets you pick specific hypervisors using scheduler hints and will create and delete them for each test run for you
17:44:51 <spyderdyne> best node to node generator i know of
17:45:04 <spyderdyne> then its just a matter of collecting the failures when they happen
17:45:12 <jasonsb> hmmmm.
17:45:29 <jasonsb> your right
17:45:31 <spyderdyne> you can do node to node, rack to rack, whatever topology you like
17:45:59 <spyderdyne> servers: hypervisor.domain.com
17:46:07 <spyderdyne> clients: hypervisor.domain.com
17:46:31 <jasonsb> i'll do both
17:46:32 <spyderdyne> you just feed it a list so you give it nodes in whatever physical location you want to hit
17:46:35 <jasonsb> lets see if they agree
17:46:41 <spyderdyne> good idea
17:47:02 <spyderdyne> i will take a look at linpack too
17:47:09 <jasonsb> in my experience nothing breaks hardware like linpack
17:47:34 <spyderdyne> i managed to kill a cisco 9508 switch with kloudbuster
17:47:35 <spyderdyne> :)
17:47:45 <jasonsb> ooooh neat
17:47:57 <spyderdyne> those things hover and configure themselves ;)
17:48:08 <jasonsb> extra points for big number model #'s :)
17:48:49 <jasonsb> have you tested with various gre/vxlan offload settings?
17:48:59 <jasonsb> should that be under purvue of tests somehow?
17:49:13 <jasonsb> (to generate complete test matrix)
17:49:46 <spyderdyne> we are only using VXLAN on overcloud with VLAN undercloud right now
17:50:40 <spyderdyne> our focus so far has been to scale out to thousands of VLANS to see how many total ports, L3 agents, networks, etc. we can support in a physical topology
17:51:04 <jasonsb> thats really good stuff
17:51:19 <jasonsb> and very tricky to get your head around, even for control plane only
17:51:23 <spyderdyne> working to be able to set up a builder that we can guarantee that automagically tells you how many net nodes, hpyervisors, control nodes, etc you need under an SLA
17:51:30 <spyderdyne> it is
17:51:58 <jasonsb> my ambitions are much more humble
17:52:00 <spyderdyne> my secret goal is to show why you shouldnt run an overcloud and need to move to containerized services for multiple reasons
17:52:23 <spyderdyne> its been fun smashing things along the way but that's the loftier goal
17:52:35 <jasonsb> i want to find out same info, but which passes criteria of not pushing data or control plane so hard that something breaks which can't be fixed with a script
17:52:42 <jasonsb> (IE: needs a human)
17:52:43 * malini thinks should upgrade my goals
17:52:46 <spyderdyne> right
17:53:09 <spyderdyne> lol
17:53:17 <jasonsb> i'm moving to kolla
17:53:22 <jasonsb> so +1
17:53:54 <spyderdyne> so far i have just been building nuclear devices and setting them off in data centers but we are getting better at reporting and quantifying
17:54:07 <jasonsb> malini, you can put spyderdyne down for first one to put a blog post
17:54:34 <malini> I am overwhelmed with all the info, I dont even know what to put him down for :D
17:54:37 <jasonsb> do you run l2pop?
17:54:38 <jasonsb> does it make a difference?
17:55:05 <malini> spyderdyne: do you want to actionize urself? I dont think I can do justice
17:55:36 <spyderdyne> what do i need to do?
17:55:57 <malini> 'you can put spyderdyne down for first one to put a blog post' tht was from jasonb
17:56:16 <jasonsb> which means malini, me, and gema all get action items :)
17:56:17 <spyderdyne> just drop in etherpad? first post!?
17:56:24 <jasonsb> spyderdyne is done
17:56:43 <malini> hmm..so tht means now we get to work :D
17:56:50 <malini> sure…sounds good
17:57:02 <jasonsb> switch to tokyo talk?
17:57:06 <malini> #info: spyderdyne gets the first blop post award
17:57:15 <malini> Sure
17:57:22 <malini> #topic: Tokyo talk
17:57:49 <malini> so our talk is in alternative accepted state - I am not sure where tht puts us in the possibility spectrum
17:58:08 <malini> Anybody else knows?
17:58:17 <spyderdyne> feasible, but not as interesting as something else maybe
17:58:31 <jasonsb> i was told typically the committee accepts 10 talks and 2 alternates
17:58:32 <spyderdyne> im sure there are several talks that have to be cancelled each summit
17:58:42 <jasonsb> so if anybody drops there is good chance
17:58:56 <jasonsb> but thsi may be a datapoint from older summit, so the numbers might be different now
17:59:06 <spyderdyne> i dont know how any of it works
17:59:09 <spyderdyne> :)
17:59:30 <malini> we shud just start prepping as if we were accepted ? Its going to be a good exercise for us
17:59:36 <jasonsb> i think so
17:59:40 <spyderdyne> sure
17:59:42 <jasonsb> if anybody drops we have 50/50 chance maybe
17:59:46 <jasonsb> but i'm not sure
17:59:47 <spyderdyne> which talk was accepted?
17:59:53 <spyderdyne> which track?
18:00:02 <malini> we shud just lock some of those accepted ppl in a hotel room ..BWAHAHAHAA
18:00:30 <malini> chking the track now
18:00:33 <jasonsb> do we all have to fill out the web form saying we are going to be in tokyo and give our phone number?
18:00:37 <spyderdyne> i can work up a presentation on tools, and if we dont use it we can put it on slideshare anyway
18:00:38 <jasonsb> what if we all dont'?
18:01:56 <malini> I guess we should still do it (if we get to) if atleast some of us go
18:02:18 <malini> I am not sure if I can go with the travel budget stuff
18:02:29 <jasonsb> you have free pass
18:02:50 <jasonsb> i booked hotel last night
18:02:51 <spyderdyne> i have paid my own way the two summits i attended, and i plan to go as long as nothing crazy comes up preventing me from it
18:02:53 <jasonsb> still need flight
18:03:51 <jasonsb> did you go to paris?
18:03:59 <malini> grr..I cant find the talk submissions :/
18:03:59 <jasonsb> that was expensiiiiivvveeee
18:04:14 <malini> tokyo will be too :(
18:04:24 <spyderdyne> i didnt go to paris
18:04:35 <spyderdyne> our currency is too weak there :)
18:04:52 <jasonsb> medium cappucino
18:04:54 <spyderdyne> and i would probably apply for citizenship while is was there and never come back
18:04:55 <jasonsb> $7
18:05:35 <malini> we are out of time & I never noticed. I'll end meeting bust we can continue chat here
18:05:35 <jasonsb> malini, we probably should do roll call for who is going and can give the talk
18:06:02 <malini> jasonsb: ok..lets do tht now before ending
18:06:09 <jasonsb> i was thihnking on ML
18:06:48 <malini> sounds good..can does it have to be just the listed speakers or anybody in the ML?
18:07:27 <jasonsb> technically, or just general concept?
18:07:39 <jasonsb> technically the speakers can still be adjusted, no?
18:07:56 <malini> yes
18:08:15 <jasonsb> i have no opinion
18:08:17 <malini> I am not sure if its technically allowed - but I know ppl have done tht
18:08:26 <spyderdyne> if the talk actually happens i can be there
18:08:46 <malini> same here - I have a better chance if it actually happens
18:09:08 <spyderdyne> i have started eating sushi so i dont starve to death when i get there :)
18:09:18 <malini> yumm..yummm...
18:09:42 <jasonsb> you can eat like a king in asia
18:10:05 <malini> yes :)
18:10:13 <jasonsb> er queen
18:10:23 <malini> you meant eat royally :)
18:10:33 <spyderdyne> being 2 to 3 times the size of the average resident i will need to
18:11:04 <spyderdyne> hopefully i dont get pulled into competitive sumo and miss the conference
18:11:09 <malini> :D
18:11:23 <malini> jasonsb: can u start the thread to poll who is going?
18:12:26 <malini> ending for real now
18:12:29 <malini> #endmeeting