14:00:27 #startmeeting networking 14:00:28 Meeting started Tue Aug 25 14:00:27 2015 UTC and is due to finish in 60 minutes. The chair is mestery. Information about MeetBot at http://wiki.debian.org/MeetBot. 14:00:30 Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 14:00:31 hi 14:00:33 The meeting name has been set to 'networking' 14:00:36 hi 14:00:38 hi 14:00:39 o/ 14:00:50 #link https://wiki.openstack.org/wiki/Network/Meetings Agenda 14:00:53 Aloha 14:00:54 o/ 14:00:59 o/ 14:01:00 salv-orlando-mob: -mob? 14:01:06 #topic Announcements 14:01:15 #link https://wiki.openstack.org/wiki/Liberty_Release_Schedule 14:01:22 #info Liberty-3 is next week! The end is near! 14:01:32 #info mestery is on vacation next week and won't be on IRC Sun-Thur 14:01:44 #info armax carl_baldwin and dougwig will handle release duties for Neutron for Liberty-3 14:02:04 Please, be gentle with armax carl_baldwin and dougwig folks ;) 14:02:24 We have a massive backlog of stuff in Liberty-3, we'll get to that later in the meeting. 14:02:52 #link https://wiki.openstack.org/wiki/ReleaseNotes/Liberty#OpenStack_Liberty_Release_Notes 14:03:09 #info Continue adding release notes for Liberty as features merge or critically customer visible bugs are found 14:03:15 Any other announcements or shall we continue along? 14:03:32 late o/ :) 14:03:36 o/ 14:03:50 Lets move along to bugs! 14:03:52 #topic Bugs 14:04:02 Up first 14:04:12 #link https://bugs.launchpad.net/neutron/+bug/1484148 14:04:12 Launchpad bug 1484148 in neutron "neutronclient gate broken following VPNaaS infra changes" [Critical,Confirmed] - Assigned to Paul Michali (pcm) 14:04:18 pc_m: What is the curretn status of this one? 14:04:27 It's our only Critical bug at the moment 14:05:26 pc_m: It looks like https://review.openstack.org/#/c/209887/ needs to merge yet? But it's workflow -1 at the moment. 14:05:36 mestery: There is currently a workaround that disables VPN test cases. 14:05:51 pc_m: So, is this bug no longer critical then? 14:05:54 mestery: I have a commit in, but there is an open question posted to ML. 14:06:09 pc_m: Can you dig that lnk out so we can add it in the logs here? 14:06:12 I'll wait for am inute ;) 14:06:15 mestery: For a final solution, namely to add the plugin 14:06:29 looking... 14:07:39 http://lists.openstack.org/pipermail/openstack-dev/2015-August/072763.html ? 14:07:54 #link http://lists.openstack.org/pipermail/openstack-dev/2015-August/072772.html 14:07:54 no, that's something different 14:08:03 Thanks pc_m and neiljerram 14:08:12 OK, lets move to the next bug 14:08:24 #link https://bugs.launchpad.net/neutron/+bug/1488282 14:08:24 Launchpad bug 1488282 in neutron "Gate failures with 'the resource could not be found'" [High,New] 14:08:25 mestery: Need decision on how to handle jobs. 14:08:41 #undo 14:08:41 Removing item from minutes: 14:08:55 pc_m: Reading that email again 14:09:30 #action mestery to followup with pc_m on a way forward for the client with regards to VPN plugin 14:09:35 So, bug is not critial, but to do final solution we need to know if want one job for all tests, one for core and adv svcs, or one for each (core, VPN, LB, FW). 14:09:40 pc_m: Lets do that post-meeting, I'll need to noodle on that email a bit 14:09:46 mestery: ok 14:09:49 pc_m: I hear you 14:10:06 mestery: cool, I'll have some noodles too ;) 14:10:10 pc_m: I know dougwig is still stuck in Utah (true story) as well, so we may want to wait until tomorrow to circle with him 14:10:35 OK, lets move on to armax's bug even though he's chosen not to join us this morning 14:10:42 #link https://bugs.launchpad.net/neutron/+bug/1488282 14:10:42 Launchpad bug 1488282 in neutron "Gate failures with 'the resource could not be found'" [High,New] 14:10:43 Or evening 14:10:46 DEpending on where you are 14:10:51 :) 14:10:52 So, armax reported this one last night 14:11:11 It's a gate failure .... 14:11:21 * mestery runs the logstash query 14:11:31 there is a similar signature appearing in the check pipeline as well 14:11:51 regXboi: Oh really? 14:12:16 yes, I just had a tempest patch fail on show_tenant_networks returning 404 14:12:17 Looks like 27 hits according to the query armax posted in the bug 14:12:29 regXboi: Was that the FWaaS thing you were telling me about? 14:12:34 mestery: nak 14:12:48 38 hits 14:12:52 mestery: the FWaaS thing is something different 14:13:02 regXboi: ack 14:13:22 what's tempest.api.compute.admin.test_servers.ServersAdminTestJSON.test_list_servers_by_admin_with_all_tenants failing to look for? 14:13:22 * regXboi is wondering if there isn't some crosstalk in nodepool again 14:13:32 regXboi: Feel free to mark that bug as confirmed since you've seen it as well 14:13:47 mestery: done 14:13:59 mestery: wait a sec 14:14:05 * mestery waits a sec 14:14:17 I found a concurrency issue in tempest: https://review.openstack.org/#/c/195345/ 14:14:40 I never went around to complete the fix as it was not showing up again, I wonder if this trace has the same origin 14:15:24 salv-orlando: *that* is the signature that I saw in the check pipeline 14:15:31 salv-orlando: It's worth looking into, can you add that into the bug by chance? 14:15:45 I can take over the bug for triaging if you like, my master 14:16:05 salv-orlando: Thank you ;) 14:16:32 but isn't that test function targeting nova? 14:16:54 * ajo misses something :) 14:17:33 ajo: I know but every tempest test, when run with neutron creates some resources 14:17:41 ajo: it targets nova APIs, but the underlying resources are neutrons 14:17:50 in this case I expect that if it creates a server a network and a post must be created somewhere 14:18:27 ajo: I mean... not "somewhere" - I know *exactly* where they are created 14:18:30 or at least I should 14:18:36 lol 14:18:48 Thanks for traiging further salv-orlando 14:19:10 Those were the new bugs I wanted to highlight this week, does anyone have any other bugs they'd like to share with the broader team this week? 14:19:15 mestery: FYI, https://review.openstack.org/#/c/214131/ has broken VPN (changed module path and name). I had put a fix in with https://review.openstack.org/#/c/213253/, but do it as a separate commit. 14:19:24 but will 14:19:26 ack :) 14:19:31 mestery: i've also seen jenkins trip over this bug a lot recently, https://bugs.launchpad.net/neutron/+bug/1488284 - review sent out this morning 14:19:31 Launchpad bug 1488284 in neutron "_clean_updated_sg_member_conntrack_entries() is throwing an AttributeError occasionally" [Undecided,In progress] - Assigned to Brian Haley (brian-haley) 14:19:41 pc_m: armax and I merged a fix for that last night 14:19:43 logstash found 23K entries :( 14:19:49 pc_m: I don't have the commit handy, search for merges from armax 14:19:49 hmm 14:19:54 yikes 14:19:54 mestery: cool 14:20:17 haleyb: Yikes, thanks for jumping on that 14:20:26 haleyb: looking at the patch, thanks :) 14:20:41 haleyb: I made that high 14:20:52 mestery: it was causing another patch of mine to fail jenkins 14:21:03 haleyb: I also added it to our bug-tracker in the meeting 14:21:19 OK, lets move along now! 14:21:41 I see we have emagana this week, so I wanted to see if he would like to give us an update from the Ops Meetup last week? regXboi was tehre too and could chime in as well 14:21:46 #topic Ops Meetup Summary 14:21:53 emagana regXboi: Sound ok? 14:21:58 I am around! 14:22:02 * regXboi defers to emagana to start 14:22:38 * regXboi also notes dougwig was there, but appears to still be stuck in Utah (and I want to hear that story) 14:22:52 There were two important sessions for this team. The first one was called "burning issues" and the second "networking model" 14:22:59 regXboi: It involves a broken axle on his trailer and a blown tire as well. It's not pretty ;) 14:23:07 #link https://etherpad.openstack.org/p/PAO-ops-burning-issues 14:23:14 emagana: Also, the large deployer session that carl_baldwin and I dialed in for. 14:23:21 #link https://etherpad.openstack.org/p/PAO-ops-network-model 14:23:22 #link: https://etherpad.openstack.org/p/PAO-ops-burning-issues 14:23:35 ok... I am slow today.. on the links.. 14:23:40 * mestery notes that session makes it sound like burning man for networking 14:23:41 * salv-orlando wonders if AAA does not operate in Utah 14:23:45 * mestery hands emagana some coffee 14:23:46 #link https://etherpad.openstack.org/p/PAO-ops-large-deployments 14:23:52 that should be the links 14:23:55 salv-orlando: Not in the middle of the desert 14:23:59 on the networking model you will find that only 1 operators is still using nova-network 14:24:11 emagana: YAAAAAAAAAAAYYYYYYYYYY!!!! 14:24:15 MOAR NEUTRON! 14:24:28 * mestery quiets down now 14:24:31 everybody else uses neutron between ovs and lb will say is 50% - 50% 14:24:39 um 14:24:39 so, it makes sense to provide the same support to both 14:24:40 Cool 14:24:43 I would correct that 14:24:50 I would have said 33/33/33 14:24:53 regXboi: Yes, please, I did not review the latest changes 14:24:56 emagana: Ack, that's what sc68cal is leading for us 14:25:04 33/33/33 regXboi? 14:25:09 ? 14:25:09 equal between ovs/lb/something else 14:25:10 OVS - LB - nova-network? 14:25:14 Ah 14:25:16 something else 14:25:16 from what emagana just said, I'd stop even thinking about migration then 14:25:22 something else = use Neutron as API only 14:25:29 something else being "human defined networking" /cc salv-orlando 14:25:40 mestery: rotfl - Nice 14:25:44 regXboi: that is right but the other 33 could be for SDN vendor specific 14:25:44 salv-orlando: I stopped thinking about it a while ago ;) 14:25:54 Those pesky vendors ;) 14:25:56 which means this team should keep supporting that model somehow 14:26:00 YEs 14:26:02 YES 14:26:02 YES! 14:26:10 ha ha ha 14:26:20 And this time, no one wanted to burn us all at the stake either, right? 14:26:23 * ajo looks for the LB/QoS volunteers that vanished... will do himself if on time after bugkilling 14:26:29 anyway.. the topic nova-network - neutron migration was not even mention 14:26:30 O:) 14:26:55 I'd agree - the LB "vs" OVS discussion was much more interesting 14:27:07 There was a request for a patch to be merge regarding greceful restart of OVS and it is not merged and reported back to the Ops 14:27:09 * mestery loves OVS AND Linuxbridge 14:27:19 emagana: it merged 14:27:20 emagana: We should be able to land that before Liberty ships I expect 14:27:23 wow 14:27:25 nice ajo! 14:27:26 emagana: actually, I think that patch merged 14:27:26 cool! 14:27:34 * regXboi was watching it 14:27:45 :D 14:27:59 Yeap! All the details on the etherpads. I dont want to take all the time... 14:28:02 emagana regXboi: Thanks for the update here! 14:28:10 Any other questions from anyone before we move on? 14:28:11 but regXboi mention anything that I am missing 14:28:21 I actually have one that isn't neutron but dear to my heart 14:28:27 mestery: yeah 14:28:31 pretty much none of the operators are using ceilometer 14:28:34 no complain about scale this time? 14:28:40 regXboi: That is correct! 14:28:43 salv-orlando: Not a single one 14:28:48 salv-orlando: well.... 14:28:54 salv-orlando: They've run out of things to complain about on neutron, they moved on to somethiong else :) 14:29:00 I wouldn't say there were no complaints about scale 14:29:04 no complain about complexity of managing components of the reference impl for a conf perspective? 14:29:05 but it wasn't *neutron* scale 14:29:09 it was *openstack* scale 14:29:10 regXboi: Maybe less complaints? 14:29:22 salv-orlando: the scalability issues were discussed previously and everybody pointed to rabbitmq :-) 14:29:36 salv-orlando: that was part of the gist of the LB "vs" OVS discussion 14:30:05 but I think that covers it 14:30:05 regXboi: so in 21st century IT lingo "it does not scale" means I really have no idea whether it works or not but some folks I know had an issue with it so it does not scale" 14:30:29 lol 14:30:37 rabbitmq is not scaling? :) 14:30:39 emagana: ok, what about managing configuration complexity? 14:30:40 OK, lets move along, we've got more excitement to come in this meeting! 14:30:49 salv-orlando: yes, most of the issues reported were anecdotal at best 14:30:51 may be we need a rabbit per openstack component in some deployments ;) 14:30:51 that was the other think that concerned operators 14:31:20 * regXboi watches mestery herd the cats 14:31:24 operators meet-up was weird. The level of participation was not as good as in Philadelphia 14:31:41 emagana: Maybe the ops were too busy using OpenStack happily to attend? :) 14:31:48 that's because philadelphia is better than every other city 14:31:49 * sc68cal ducs 14:31:52 mestery: +1 :D 14:31:55 * sc68cal ducks 14:31:57 sc68cal: +1 14:32:06 :) 14:32:09 ;) 14:32:14 * regXboi remembers batteries in the snowballs in Philadelphia 14:32:18 maybe there are less distractions in Philly 14:32:18 mestery: yeah.. too much noise before the session 14:32:31 OK, now lets move along folks 14:32:33 #topic Docs 14:32:40 ok.. I keep going! 14:32:40 emagana: Back to you sir for a quick update! 14:32:43 lol 14:32:55 To make it short I have updated the wiki yesterday night withpatches needed 14:33:15 reminder - Please help documenting but no verbatim copies from internet! 14:33:28 Link? 14:33:29 emagana: Good idea 14:33:29 emagana, link? 14:33:35 neiljerram :) 14:33:38 dup 14:33:53 jinx! 14:33:55 #link: https://review.openstack.org/#/c/196541/ 14:34:20 One review has a dependency in neytron code: 14:34:23 #link: https://review.openstack.org/#/c/185977/ 14:34:38 I meant the link to the wiki that you mentioned. 14:35:05 neiljerram: https://wiki.openstack.org/wiki/Network/Meetings 14:35:06 Can we just deprecate the wiki for documentation? 14:35:10 emagana: That neutron patch was supposed to merge last night but it got caught in the gate by a merge conflict 14:35:24 emagana: Ready to go now, just needs another +2 and a workflow 14:35:25 john-davidge: Thanks, will keep an eye on it.. 14:35:40 When I said wiki, I mean the Neutron Wiki page 14:35:41 HenryG: +1000.5 14:35:46 HenryG: We can deprecate but in order for people to know we should write it on the wiki ;) 14:35:48 * mestery wants to deprecate all wikis 14:36:15 unless you find a way to tell google to give wikis a lower page rank 14:36:21 salv-orlando: lol 14:36:26 HenryG: We can move the remaining tasks for docs into a etherpad (already did) if the team prefers that 14:36:41 emagana HenryG: Ack on that 14:36:54 Every wiki page could just be a single link to the real docs? 14:36:57 I'm happy to spend some time helping with docs, but it's not clear to me, from https://wiki.openstack.org/wiki/Network/Meetings, what still needs doing. 14:37:26 neiljerram: I can point you to the right direction after the meeting ;-) 14:37:32 just ping me 14:37:35 Cool, we'll do that. 14:37:54 Thanks for the update emagana! 14:37:59 OK 14:38:03 neiljerram: or you can take this one: https://review.openstack.org/#/c/196541 needs to be re-edited 14:38:05 Lets keep the train moving, we're almost out of gas 14:38:11 Or coal 14:38:17 #topic Pecan Branch 14:38:31 salv-orlando kevinbenton blogan: Go or no go for merging this back during Liberty yet? 14:38:43 I still see some patches outstanding when I last looked yesterday 14:38:53 I think we need to rename the folder 14:38:57 And merge it in 14:39:04 kevinbenton: Ack 14:39:17 kevinbenton: Want to work with me on that so we can plan to merge it back later this week? 14:39:26 Sure 14:39:58 kevinbenton: so you've got the remaining todos ready to push? like bulk support? 14:39:59 Awesome. 14:40:26 emagana: OK, I'll do that. 14:40:35 mestery, kevinbenton: it does not have to be perfect before merging back, but it must be imho 1) working, and 2) "quasi functionally equivalente" 14:40:49 salv-orlando: Agreed 14:40:58 anyway feel free to keep me involved in the decision for merging it back 14:41:02 It will obviously be disabled by default for Liberty 14:41:10 salv-orlando: Absolutely, and thank you! 14:41:14 now we can move on, my gracious majesty 14:41:19 makes sense 14:41:24 lol 14:41:30 Right, bulk support will be questionable 14:41:55 #topic Liberty-3 BPs hanging over our heads 14:41:59 We have a lot 14:42:01 A LOT 14:42:02 #link https://launchpad.net/neutron/+milestone/liberty-3 14:42:05 31 14:42:08 * ajo looks 14:42:12 * mestery weeps 14:42:36 I'm going to spend time cleaning this up, but we've reached the part of the cycle where everyone will dislike me as I move things out of Liberty that realistically won't make it 14:42:43 mestery: there's nothing to weep - it's a LOT LESS than the previous cycles 14:42:48 QoS API extension could almost be cleared out, cleaning out some bugs, and done 14:43:03 salv-orlando: I think armax said that since the vendor decomposition, work has moved to other repos. He may be right. 14:43:10 mystery the Octavia one is misclassified: We are almost done 14:43:15 ajo: I'd be good with that, especially if you file the bugs 14:43:15 mestery 14:43:25 xgerman: You mean it will make Liberty yet? Or no? 14:43:26 mestery: bugs are filed, /me looks for link 14:43:29 dougwig owed me an answero n that 14:43:32 mestery: I think service decomp was even more effective. But we can bikeshed on this another day 14:43:33 will make Liberty - 14:43:33 mestery: https://review.openstack.org/#/c/196541 Is good to go for a seconf merge attempt once it gets another +2. That will close out the IPv6 PD blueprint 14:43:35 ajo: Cool! Please mark the BP complete then 14:43:43 salv-orlando: ack ack ack 14:43:55 mestery: wrong link! 14:44:03 john-davidge: Cool! I bet haleyb or carl_baldwin will take you up on that once you get the correct link 14:44:10 mestery: correct link https://review.openstack.org/#/c/185977/ :) 14:44:26 xgerman: So, I'll mark Octavia as "NEeds Review" then 14:44:31 yep 14:45:14 haleyb: carl_baldwin: I am beholden to your reviews : ) https://review.openstack.org/#/c/185977/ 14:45:17 haleyb carl_baldwin: Could use your eyes and gracious approval on https://review.openstack.org/#/c/185977/ 14:45:21 lol 14:45:22 jinx! 14:45:26 mestery: ack 14:45:50 kevinbenton: Have you had much of a chance to look at this one: https://blueprints.launchpad.net/neutron/+spec/vlan-aware-vms 14:45:52 Or anyone else? 14:45:54 I see a bunch of patches 14:46:01 john-davidge: ack, i see it was just rebased 14:46:03 But I'm worried they showed up late in the cycle 14:46:08 john-davidge: I had it up this morning. 14:46:09 mestery: only briefly 14:46:16 Not looking like it will make it 14:46:19 kevinbenton: Looks like you're the only one to briefly review it so far 14:46:21 yeah 14:46:53 ajo: Waht is the status of this one? https://blueprints.launchpad.net/neutron/+spec/ml2-ovs-qos-with-dscp 14:47:06 I think they need a Nova change anyway 14:47:07 mestery: that one is proposed as RFE now, 14:47:10 carl_baldwin: haleyb: Thanks guys! 14:47:24 mestery: I told them "show me the code", and we will propose, otherwise: M 14:47:40 ajo: It's out 14:47:41 mestery: I guess the realistic thing is M, 14:47:45 yep 14:48:01 ajo: In agreement with you 14:48:17 * mestery thinks carl_baldwin is busy working on unit tests for address scopes right now 14:48:39 carl_baldwin: haleyb: Would appreciate some eyes on https://review.openstack.org/#/c/212688/ as well. There’s a bug fix that depends on it 14:48:40 mestery: Not looking good unless I can find a chunk of time this week. 14:48:51 salv-orlando: This one looks like Mitaka, ack if you agree: https://blueprints.launchpad.net/neutron/+spec/consolidate-extensions 14:48:58 mestery: also LB/QoS, if volunteers don't reappear I can try for L soon, and we can decide based on how invasive does it look, and otherwise postpone to M 14:48:59 carl_baldwin: We could do an FFE for that one too 14:49:18 mestery: yes, because I am stashing the work on top of the pecan switch 14:49:24 we cannot really make both for L 14:49:29 salv-orlando: ack ack ack 14:49:33 not a slight grim minimal chance 14:49:36 ajo: LB/QoS I will be taking it up in M 14:50:00 mestery: It is so close. 14:50:07 vikram__, ideally we should have had it by L :) 14:50:14 carl_baldwin: I know, we'll figure it out 14:50:27 mestery: I've got some doubts about finishing https://blueprints.launchpad.net/neutron/+spec/bgp-dynamic-routing, code and reviews just coming in too late. Not sure if FFE is feasible 14:50:33 ajo: I think initial QoS got it there so late :) 14:50:40 yep 14:50:49 mestery: I tried to mark https://blueprints.launchpad.net/neutron/+spec/quantum-qos-api as complete, no permissions 14:50:53 tidwellr: I'd be up for an FFE for that, but let me work with carl_baldwin and haleyb to understand what that would look like 14:51:28 https://blueprints.launchpad.net/neutron/+spec/restructure-l2-agent needs reviews. I'd appreciate anybody having a look. 14:51:36 tidwellr: mestery: Let’s sync up on Thursday. See what you can do with it. 14:51:44 ack 14:51:53 carl_baldwin: thanks 14:51:58 rossella_s: ack 14:52:02 carl_baldwin: ack 14:52:03 rossella_s: Yes! That will get an FFE for sure, you only need reviews there 14:52:10 kevinbenton: Get your team on those! ;) 14:52:20 Team, look at those 14:52:21 rossella_s: ack 14:52:21 mestery: I assume the reference implementation split is going to M? 14:52:25 kevinbenton: Nice work 14:52:40 HenryG: Yes, I'm not keen to be burned this late in the cycle 14:52:41 mestery, kevinbenton ajo thanks! 14:52:41 I like my skin in an unburned state 14:52:41 * kevinbenton has upper management skills :) 14:52:43 kevinbenton: ack 14:52:49 * mestery promotes kevinbenton 14:52:54 lol 14:53:01 You're a Major now kevinbenton! 14:53:17 god save the Major! 14:53:28 lol 14:53:34 #topic Open Discussion 14:53:36 talking of this, does it make sense that I merge QoS stuff now, like this now: https://review.openstack.org/#/c/216717/ ? 14:53:50 for resource timestamp https://blueprints.launchpad.net/neutron/+spec/add-port-timestamp, code is reviewing, Not sure if FFE is feasible 14:54:21 ajo: isn't that merged? 14:54:25 I'm supposed to do that for reference control plane, but I guess I have good knowledge of the QoS-api / qos-DB / objects / rpc callbacks to do such thing now 14:54:35 kevinbenton... 14:54:36 ehhmmm 14:54:37 wrong link 14:54:38 sorry 14:54:41 #info Happy birthday kevinbenton! 14:54:42 #link https://youtu.be/1fnhgzV1HhA 14:54:46 #link https://review.openstack.org/#/c/214776/ 14:54:47 :) 14:54:48 https://review.openstack.org/#/c/214212/ 14:54:49 :-) 14:54:53 more flavor fun 14:55:00 I was meaning stuff like that ^ :) 14:55:05 kevinbenton, happy birthday!!! ;) 14:55:22 Happy B'DAY Kekvinbenton! 14:55:23 * mestery hopes armax and marun take kevinbenton out for a birthday celebration 14:55:25 happy birthday kevinbenton 14:55:27 kevinbenton, happy birthday!!! 14:55:31 Happy Birthday kevinbenton 14:55:40 kevinbenton: You're 17 today, right? ;) 14:55:42 Happy Birthday!! 14:55:43 kevinbenton Happy Birthday 14:55:46 mestery: lol 14:55:48 kevinbenton, have a beer today 14:55:50 :D 14:55:54 boy genius? 14:55:55 Thanks! 14:56:00 kevinbenton, happy birthday:) 14:56:08 happy birthday Kevin!! 14:56:19 Happy one! Felicidades! 14:56:21 yongshenggong_: we had enough beer last week :) 14:56:22 kevinbenton: Happy birthday :) 14:56:24 happy birthday kevin! 14:56:24 kevinbenton: And thanks for making OpenStack China hackathon succesful! Next week I'll leave time for you to update the team on what went on there :) 14:56:50 kevinbenton: we want to know everything that happened in China! 14:56:56 kevinbenton: feliz cumpleanios! 14:56:56 mestery: I'm on my phone, not enough time to type lots of details 14:56:57 ++ 14:57:00 It went well 14:57:01 ugh, had the meeting time wrong on my calendar :( 14:57:03 pictures? 14:57:05 kevinbenton: Next week 14:57:05 kevinbenton: everything! 14:57:08 Lots of good scalability bugs 14:57:13 * john-davidge pours kevinbenton a birthday beer 14:57:14 ++!! :) 14:57:14 russellb: It's ok, next week you can lead the meeting! :D 14:57:23 beers for everyone! 14:57:24