| *** saneax has quit IRC | 00:04 | |
| *** zhangjn has quit IRC | 00:05 | |
| *** mpopow has quit IRC | 00:24 | |
| *** esker has quit IRC | 00:27 | |
| *** zhangjn has joined #openstack-operators | 00:49 | |
| *** zhangjn has quit IRC | 00:51 | |
| *** Marga_ has joined #openstack-operators | 00:51 | |
| *** zhangjn has joined #openstack-operators | 00:53 | |
| *** zhangjn has quit IRC | 00:54 | |
| *** Marga_ has quit IRC | 00:54 | |
| *** Marga_ has joined #openstack-operators | 00:55 | |
| *** zhangjn has joined #openstack-operators | 00:55 | |
| *** Marga_ has quit IRC | 01:00 | |
| *** Marga_ has joined #openstack-operators | 01:00 | |
| *** saneax has joined #openstack-operators | 01:16 | |
| *** gtt116 has joined #openstack-operators | 01:44 | |
| *** Marga_ has quit IRC | 02:28 | |
| *** hakimo_ has joined #openstack-operators | 02:52 | |
| *** hakimo has quit IRC | 02:55 | |
| *** SimonChung has joined #openstack-operators | 03:30 | |
| *** SimonChung1 has joined #openstack-operators | 03:31 | |
| *** SimonChung has quit IRC | 03:34 | |
| *** jproulx1 has quit IRC | 03:47 | |
| *** jproulx1 has joined #openstack-operators | 03:48 | |
| *** swat30 has quit IRC | 03:58 | |
| *** harshs has quit IRC | 04:01 | |
| *** swat30 has joined #openstack-operators | 04:03 | |
| *** jproulx1 has quit IRC | 04:09 | |
| *** jproulx1 has joined #openstack-operators | 04:10 | |
| *** jproulx1 has left #openstack-operators | 04:13 | |
| *** saneax has quit IRC | 04:54 | |
| *** Guest53419 has quit IRC | 05:01 | |
| *** zeus has joined #openstack-operators | 05:03 | |
| *** zeus is now known as Guest22530 | 05:04 | |
| *** bvandenh has joined #openstack-operators | 05:11 | |
| *** zerda has joined #openstack-operators | 05:28 | |
| *** albertom has quit IRC | 05:29 | |
| *** albertom has joined #openstack-operators | 05:31 | |
| *** Marga_ has joined #openstack-operators | 05:51 | |
| *** Marga_ has quit IRC | 05:51 | |
| *** Marga_ has joined #openstack-operators | 05:51 | |
| *** fawadkhaliq has joined #openstack-operators | 06:21 | |
| *** albertom has quit IRC | 06:34 | |
| *** stanchan has joined #openstack-operators | 06:38 | |
| *** albertom has joined #openstack-operators | 06:40 | |
| *** stanchan has quit IRC | 06:48 | |
| *** bvandenh has quit IRC | 07:00 | |
| *** matrohon has joined #openstack-operators | 07:44 | |
| *** saneax has joined #openstack-operators | 07:47 | |
| *** derekh has joined #openstack-operators | 08:40 | |
| *** saneax has quit IRC | 09:18 | |
| *** zhangjn has quit IRC | 10:15 | |
| *** Marga_ has quit IRC | 10:24 | |
| *** zerda has quit IRC | 12:21 | |
| *** Marga_ has joined #openstack-operators | 12:35 | |
| *** Marga_ has quit IRC | 12:56 | |
| *** gfa is now known as gfa_ | 12:58 | |
| *** gfa_ is now known as gfa | 12:59 | |
| *** Marga_ has joined #openstack-operators | 13:07 | |
| *** regXboi has joined #openstack-operators | 13:28 | |
| *** kencjohnston has joined #openstack-operators | 13:44 | |
| *** ferest has joined #openstack-operators | 13:50 | |
| *** ferest has quit IRC | 13:54 | |
| *** Piet has quit IRC | 13:56 | |
| *** fawadkhaliq has quit IRC | 14:04 | |
| *** fawadkhaliq has joined #openstack-operators | 14:05 | |
| *** fawadkhaliq has quit IRC | 14:08 | |
| *** csoukup has joined #openstack-operators | 14:40 | |
| *** csoukup has quit IRC | 14:42 | |
| *** fawadkhaliq has joined #openstack-operators | 15:06 | |
| *** SimonChung1 has quit IRC | 15:09 | |
| *** kencjohnston has quit IRC | 15:10 | |
| *** fawadkhaliq has quit IRC | 15:10 | |
| *** klindgren has joined #openstack-operators | 15:13 | |
| *** alop has joined #openstack-operators | 15:15 | |
| *** gfa is now known as gfa_ | 15:19 | |
| *** gfa_ is now known as gfa | 15:19 | |
| gfa | 15:25 | |
|---|---|---|
| gfa | ouch | 15:26 |
| gfa | sorry | 15:26 |
| *** alop has quit IRC | 15:43 | |
| *** Marga_ has quit IRC | 15:43 | |
| *** alop has joined #openstack-operators | 15:46 | |
| *** Marga_ has joined #openstack-operators | 15:49 | |
| *** saneax has joined #openstack-operators | 15:53 | |
| *** VW_ has joined #openstack-operators | 16:00 | |
| VW_ | morning folks - anyone able to make it in for the Large Deployments meeting? | 16:01 |
| klindgren | I am here | 16:02 |
| VW_ | I think it just may just be the two of us | 16:05 |
| VW_ | but oh well... | 16:05 |
| VW_ | #startmeeting Large Deployments Team August 2015 Meeting | 16:06 |
| openstack | Meeting started Thu Aug 27 16:06:26 2015 UTC and is due to finish in 60 minutes. The chair is VW_. Information about MeetBot at http://wiki.debian.org/MeetBot. | 16:06 |
| openstack | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 16:06 |
| openstack | The meeting name has been set to 'large_deployments_team_august_2015_meeting' | 16:06 |
| VW_ | #topic Neutron and Segments | 16:07 |
| VW_ | so, klindgren, I moved from one OpenStack event to another | 16:07 |
| VW_ | anything happen after the LDT session where the Neutron folks dialed in? | 16:08 |
| VW_ | guessing not :) | 16:12 |
| klindgren | no - sorry didn't think this started until like another 20 min | 16:12 |
| VW_ | tis OK | 16:12 |
| VW_ | I'm actually doing this, and sitting in keynotes and OSSV15, so .... | 16:13 |
| klindgren | trying to rmeember | 16:13 |
| klindgren | I believe there was some discussion around scale. Where scale in this case was relatively small | 16:13 |
| *** fawadkhaliq has joined #openstack-operators | 16:13 | |
| klindgren | like 100 networks with neutron-client blowing up, and something about clients not handling multi-region well at all | 16:14 |
| klindgren | when a common keystone was used | 16:14 |
| klindgren | I believe all of those things were fixed upstream | 16:15 |
| klindgren | but points to a lack of functional testing in the gate | 16:15 |
| *** SimonChung has joined #openstack-operators | 16:15 | |
| klindgren | I think that was it | 16:16 |
| klindgren | Oh - some stuff about trying to get a taxonomy group formed | 16:17 |
| klindgren | so that people can communicat using the same languauge that has the same definitions - since some words are overloaded in openstack/cloud | 16:18 |
| VW_ | yeah - there was that too | 16:18 |
| VW_ | not sure if anyone else from that conversation showed up | 16:19 |
| VW_ | today that is | 16:19 |
| klindgren | Some discussion around lawfull intercept as well | 16:19 |
| VW_ | yeah | 16:19 |
| VW_ | so, to me, that's part of the larger conversation of whehter or not there needs to be a Public Cloud Team seperate from the LDT | 16:20 |
| klindgren | yea - I think a number of large deployers *ARE* public clouds | 16:20 |
| serverascode | I do think there should be a public cloud working group myself | 16:21 |
| klindgren | and you can run an internal cloud like a public cloud as well. | 16:21 |
| serverascode | we are public but not necessarily large | 16:21 |
| klindgren | But I would be fine with scale issues being LDT focus, and better multi-tenancy and other stuff being the focus of public cloud | 16:22 |
| VW_ | that's fair | 16:23 |
| klindgren | both are currently far from ideal :-) | 16:24 |
| VW_ | indeed | 16:24 |
| VW_ | Ok | 16:25 |
| klindgren | I jsut ask that we try not to schedule LDT and public-cloud stuff together | 16:25 |
| VW_ | #action VW_ talk to Tom F. about a Public Cloud working group of some sort | 16:25 |
| gfa | i think will be a lot of people in common on both groups | 16:25 |
| VW_ | there are a lot of common folks gfa | 16:25 |
| VW_ | ant that is my struggle | 16:25 |
| klindgren | I am also ok expanding LDT to include both | 16:26 |
| VW_ | serverascode: what blocks you from being comfortable from joining the LDT and helping drive the public cloud issues too? | 16:26 |
| serverascode | I find the LDT is mostly about large scale networks | 16:27 |
| gfa | as klindgren says, as long as schedule don't overlap it will be ok | 16:27 |
| gfa | i think latter both groups can merge back if they are too small/similar | 16:28 |
| serverascode | there aren't very many public clouds afaik, so I would imagine it would be small group | 16:28 |
| gfa | maybe experiment 2 groups and if does not work. go back to one group | 16:28 |
| *** matrohon has quit IRC | 16:29 | |
| VW_ | those are fair comments | 16:30 |
| gfa | if we keep the same IRC channel and mailing list, split/join the groups should be easy IMHO | 16:30 |
| VW_ | serverascode: we are focused on networks a lot right now becuase our interaction with Neutron, but we really try to think about scaling problems in general | 16:30 |
| VW_ | yes gfa - LDT has pretty much settled on the "operators" - both list and channel - for pretty much everything | 16:31 |
| serverascode | yeah, I might just be floating out here on my own with no real perspective :) I dunno. I'd just like a group of ppl with similar problems to talk about, and I don't think that is the LDT for me right now, but again I might be a one-off kinda situation | 16:33 |
| *** carl_baldwin has joined #openstack-operators | 16:35 | |
| VW_ | no, those are valid points serverascode | 16:37 |
| VW_ | like I said, I can argue it either way | 16:37 |
| VW_ | serverascode, gfa - are you coming to Tokyo | 16:38 |
| gfa | i'm :( | 16:38 |
| gfa | :) | 16:38 |
| serverascode | yeah should be | 16:38 |
| VW_ | ok - for now, let's do this. I'll see if we are going to get a good block of time for LDT (like we did in YVR) for LDT | 16:39 |
| VW_ | if so, then why don't you join us and we'll talk this out for a bit in person | 16:39 |
| serverascode | sounds good to me, thanks :) | 16:40 |
| VW_ | sure! | 16:40 |
| VW_ | I'll work on it and try to get an update out vie ML and the September LDT meeting | 16:40 |
| VW_ | so, on that note | 16:40 |
| VW_ | #topic LDT bug list for cells V1 | 16:41 |
| VW_ | ^ that is probably our next big collective push with devs | 16:41 |
| VW_ | agree/disagree? | 16:41 |
| klindgren | agree | 16:42 |
| klindgren | Just want to try to consolidate the patchsets that we all carry | 16:43 |
| *** derekh has quit IRC | 16:44 | |
| klindgren | and try to get those upstreamed | 16:44 |
| klindgren | since it looks like cells v1 is going to be around for atleast 1+ years | 16:44 |
| VW_ | yep | 16:44 |
| klindgren | (I also want to try to do the same thing for general opperators as well) | 16:44 |
| VW_ | did we ever etherpad that? | 16:44 |
| klindgren | we did - but Only a little bit | 16:45 |
| VW_ | yes, I think this goes beyond LDT | 16:45 |
| klindgren | https://etherpad.openstack.org/p/PAO-LDT-cells-patches | 16:45 |
| VW_ | sweet! | 16:45 |
| klindgren | I htink sam and I are the only ones that added anything | 16:46 |
| klindgren | ;-) | 16:46 |
| klindgren | *cough* VW *cough* | 16:46 |
| * VW_ duly notes the public shaming | 16:46 | |
| VW_ | This should be a big topic of ours in Tokyo | 16:47 |
| VW_ | but we can do a lot of prework in the next two months | 16:47 |
| klindgren | yea | 16:48 |
| VW_ | ok, so I'll my act together and get some updates on there | 16:48 |
| VW_ | then we'll get it out to the list | 16:48 |
| VW_ | Cern should probably have some they can put there too | 16:49 |
| klindgren | I plan on starting another etherpad of patches that we have for just business logic | 16:49 |
| VW_ | That's a good idea | 16:49 |
| VW_ | Those may be harder because all of us will probably end up with a few that are so specific, they won't go upstream | 16:50 |
| klindgren | right - but until we talk about it - we never know | 16:50 |
| klindgren | 4 people might be doing the same thing | 16:50 |
| klindgren | like 3 people having patches to fix vif plugging under cells v1 | 16:51 |
| klindgren | but as someone else mentioned it possible too that instead of a patch - we could add in a hook | 16:52 |
| klindgren | if 4 people want to do something on x event but want to do different things | 16:52 |
| VW_ | agreed | 16:52 |
| VW_ | OK, let's put that at the top of the list for next month then | 16:53 |
| VW_ | get some traction on the cells v1 in the interim and potentially tee both of them up for Tokyo working session | 16:54 |
| VW_ | with the Public Cloud bit, of course | 16:54 |
| VW_ | :) | 16:54 |
| VW_ | wow - we may have just planned an agenda two months out | 16:55 |
| klindgren | So the only other thing that I have is that we did some work to get a better handle on doing QoS/policing on vm traffic to prevent outbound ddos from vm's | 16:55 |
| VW_ | ooh yeah | 16:55 |
| VW_ | #topic open discussion | 16:55 |
| *** harlowja has joined #openstack-operators | 16:55 | |
| VW_ | you should share some more details about that here for us to capture for all | 16:55 |
| klindgren | but it was done via libvirt/qemu hooks to apply custom tc rules to a vm on startup/shutdown | 16:56 |
| VW_ | interesting | 16:56 |
| *** harshs has joined #openstack-operators | 16:57 | |
| klindgren | but the requirement was to filter specific sized traffic to a lower rate, and attempt to be able to police traffic to a set threshold depnding on flavor | 16:57 |
| VW_ | what have the results been? | 16:57 |
| klindgren | no dedtactable outbound ddos's so far | 16:58 |
| klindgren | since we had a free beta going on - we had a fair amount of cruft that got in | 16:58 |
| gfa | what you did is cool | 16:58 |
| klindgren | but against is 100% outside of openstack | 16:59 |
| gfa | we have talked about elasticsearch, kibana, etc before. I don't think is OT | 17:00 |
| klindgren | but figure other public cloud people probably have this issue and probably want a soltuoion - and the default tuneables in nova/neutron don't give you that ability yet | 17:00 |
| klindgren | VW_ also no customer complaints - yet | 17:00 |
| VW_ | Yeah, I'm really interested in where it goes. Like I told you when we chatted, our upstream Backbone folks usually detect and then our AUP suspends | 17:01 |
| klindgren | we have used something similar on our VPS product for a few years now | 17:01 |
| VW_ | always a good litmus test, klindgren | 17:01 |
| VW_ | alright, well I'm happy to chat more, but I'll end the recording since we are at time | 17:01 |
| VW_ | thank you all for joining. | 17:01 |
| VW_ | sorry for being a bit distracted and conferencing while meeting at the same time | 17:02 |
| VW_ | #end meeting | 17:02 |
| VW_ | #endmeeting | 17:02 |
| openstack | Meeting ended Thu Aug 27 17:02:20 2015 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 17:02 |
| openstack | Minutes: http://eavesdrop.openstack.org/meetings/large_deployments_team_august_2015_meeting/2015/large_deployments_team_august_2015_meeting.2015-08-27-16.06.html | 17:02 |
| openstack | Minutes (text): http://eavesdrop.openstack.org/meetings/large_deployments_team_august_2015_meeting/2015/large_deployments_team_august_2015_meeting.2015-08-27-16.06.txt | 17:02 |
| openstack | Log: http://eavesdrop.openstack.org/meetings/large_deployments_team_august_2015_meeting/2015/large_deployments_team_august_2015_meeting.2015-08-27-16.06.log.html | 17:02 |
| VW_ | hey, sidenote, klindgren. I know you guys are very interested in the Auditing and Resolver services of ours. What about the inventory bits? | 17:03 |
| *** Guest22530 has quit IRC | 17:04 | |
| klindgren | I am interested. We have our own homegrown inventory thing | 17:04 |
| VW_ | ok. Got a VP pushing me to see what we could actually put out there by Tokyo :O | 17:05 |
| VW_ | my money is on the inventory and provisioning bits | 17:05 |
| VW_ | but I have a truckload of stuff to figure out - quickly | 17:05 |
| klindgren | but I think we are also looking at moving towards servicenow for some of that stuff | 17:05 |
| klindgren | but from the cloud side - I am ok having 2 inventory systems, on for the company another for the cloud | 17:06 |
| VW_ | well, remember, for us it's kind of that way too, but in our world view the "company's" system is the asset management piece | 17:07 |
| VW_ | and we are just aggregating | 17:07 |
| VW_ | actually, I have some cool slides from my Fleet Management talk yesterday that covers that and more | 17:08 |
| klindgren | feel free to send them my way :-) | 17:08 |
| *** Marga_ has quit IRC | 17:08 | |
| *** chyka has joined #openstack-operators | 17:17 | |
| *** chyka is now known as chykz | 17:26 | |
| *** Marga_ has joined #openstack-operators | 17:34 | |
| *** setmason has joined #openstack-operators | 17:37 | |
| *** SimonChung1 has joined #openstack-operators | 17:41 | |
| *** SimonChung has quit IRC | 17:41 | |
| *** gfa is now known as gfa_ | 17:41 | |
| *** VW_ has quit IRC | 17:45 | |
| *** VW_ has joined #openstack-operators | 17:46 | |
| *** VW_ has quit IRC | 17:50 | |
| *** Piet has joined #openstack-operators | 18:03 | |
| *** alop has quit IRC | 18:05 | |
| *** alop has joined #openstack-operators | 18:08 | |
| *** verdurin has quit IRC | 18:35 | |
| *** verdurin has joined #openstack-operators | 18:39 | |
| *** Marga_ has quit IRC | 18:50 | |
| *** csoukup has joined #openstack-operators | 18:50 | |
| mnaser | anyone see this come up from time to time in their compute logs: "NeutronClientException: Authentication required" ? | 18:54 |
| *** alop has quit IRC | 19:03 | |
| *** alop has joined #openstack-operators | 19:03 | |
| raginbajin | klindgren: We are using service now as our asset management stuff. | 19:08 |
| raginbajin | We've done a lot of home grown scripts to pull data out and organize it in some sort of sane manner. | 19:09 |
| klindgren | raginbajin, are those scripts somewhere? | 19:10 |
| raginbajin | We don't do a lot of things publicly yet, but I may be able to share privately. | 19:11 |
| klindgren | I am ok having the data in two places, with one being the asset management view, and the other being the real view of the system and its state. | 19:12 |
| raginbajin | Yeah. I'd like to reduce the views if I can. We have service now, we have a "showback" dashboard in Kibana, and then what you find in OpenStack itself | 19:13 |
| *** chykz_ has joined #openstack-operators | 19:31 | |
| *** esker has joined #openstack-operators | 19:31 | |
| *** chykz has quit IRC | 19:34 | |
| *** fawadkhaliq has quit IRC | 19:35 | |
| *** chykz has joined #openstack-operators | 19:37 | |
| *** chykz_ has quit IRC | 19:41 | |
| *** matrohon has joined #openstack-operators | 19:45 | |
| *** esker has quit IRC | 20:04 | |
| *** fawadkhaliq has joined #openstack-operators | 20:06 | |
| *** esker has joined #openstack-operators | 20:08 | |
| *** chykz_ has joined #openstack-operators | 20:09 | |
| *** matrohon has quit IRC | 20:09 | |
| *** chykz has quit IRC | 20:11 | |
| *** esker has quit IRC | 20:13 | |
| *** matrohon has joined #openstack-operators | 20:16 | |
| *** reed has quit IRC | 20:16 | |
| *** chykz has joined #openstack-operators | 20:17 | |
| *** chykz_ has quit IRC | 20:20 | |
| *** reed has joined #openstack-operators | 20:21 | |
| *** matrohon has quit IRC | 20:23 | |
| *** matrohon has joined #openstack-operators | 20:25 | |
| *** saneax has quit IRC | 20:28 | |
| *** markvoelker has quit IRC | 20:33 | |
| *** signed8bit has joined #openstack-operators | 20:48 | |
| *** Marga_ has joined #openstack-operators | 20:57 | |
| *** fawadkhaliq has quit IRC | 21:00 | |
| *** Piet has quit IRC | 21:13 | |
| *** SimonChung1 has quit IRC | 21:24 | |
| *** SimonChung has joined #openstack-operators | 21:24 | |
| mgagne | klindgren: (I'm very late to the conversation) you said you had stuff for QoS? Was the QoS support for libvirt insuffisant? | 21:26 |
| klindgren | mgagne, we do - and it was | 21:27 |
| klindgren | for us atleast | 21:27 |
| mgagne | klindgren: was [insuffisant] ? | 21:28 |
| mgagne | klindgren: we found out that udp traffic is not properly handled by the libvirt qos filters | 21:28 |
| klindgren | We wanted to do things like limit all output of all vm's to 90% of the available server bandwidth | 21:29 |
| klindgren | we also have expereince with a number of outbound ddos that use small packet sizes | 21:29 |
| *** SimonChung has quit IRC | 21:29 | |
| *** SimonChung1 has joined #openstack-operators | 21:29 | |
| mgagne | oh, so it's not a hard limit | 21:29 |
| klindgren | so we wanted to rate limit vm's on small packet sizes to something like say 500kbits/s | 21:30 |
| mgagne | is this something tc can do? | 21:30 |
| *** Piet has joined #openstack-operators | 21:30 | |
| klindgren | then some product managers decided that they wanted to offer flavors that are limited to say 100mbit/s | 21:30 |
| mgagne | if so, we might just move to tc instead of whatever libvirt provides | 21:30 |
| klindgren | so we use policing in tc to shape the traffic to that as well | 21:31 |
| klindgren | mgagne, it is indeed | 21:31 |
| klindgren | I should say we use shaping vs's policing | 21:32 |
| mgagne | right | 21:32 |
| klindgren | where the libvirt stuff was like anything higher than 100bmit/s drop | 21:32 |
| mgagne | yea, it's doesn't work that well to prevent outgoing DoS :-/ | 21:32 |
| klindgren | shapping will doe queuing and the like to allow tcp windowing to come int to play and keep the transfer rate down | 21:32 |
| mgagne | works well on legit trafic over TCP | 21:32 |
| *** regXboi has quit IRC | 21:32 | |
| klindgren | yea | 21:32 |
| klindgren | let me see if I can get our libvirt hook out onto our github | 21:33 |
| mgagne | we are also thinking about integrating our smtp throttling into neutron | 21:34 |
| mgagne | we currently have custom scripts hooking into iptables to throttle smtp. | 21:34 |
| klindgren | we do that differently | 21:34 |
| mgagne | while we don't want to block 100% of it, we still wish to limit damage | 21:35 |
| klindgren | we have a hard block on outbound trafic | 21:35 |
| mgagne | and not maintain a smarthost/relay | 21:35 |
| klindgren | and relay servers that you need ot use to relay traffic | 21:35 |
| klindgren | that are tied to a quota system | 21:35 |
| mgagne | yes, that's a great idea. we just don't have the man power to maintain it =) | 21:35 |
| klindgren | we've used that for many many years so - it doesn;'t make sense to replace the tooling thats already there | 21:36 |
| mgagne | if we had such systems, we sure would have hooked it up to our openstack product too =) | 21:37 |
| klindgren | hehe | 21:38 |
| klindgren | anyway the basic premis is that we hook into the vm.start and vm.stop events that libvirt emits | 21:38 |
| klindgren | and we modify the TC rules for the vm oursleves | 21:38 |
| klindgren | that way you can do qdisks with subqdisks and the like | 21:39 |
| mgagne | that's something I should read about then | 21:39 |
| mgagne | oh btw, we also did our first nova cells migration yesterday. ~15m max and we were done. :D | 21:40 |
| klindgren | cool | 21:41 |
| mgagne | went something like this (very simplified): stop services ; mysqldump nova | mysql nova_c1 ; update configs ; start services | 21:42 |
| klindgren | yea - iirc that was pretty close to what we did too | 21:46 |
| mgagne | yea, I every much based mine on mike dorman's blog post | 21:47 |
| klindgren | ha - k | 21:47 |
| klindgren | hopefully completed without the large train wreck photo | 21:48 |
| mgagne | so far, no train wreck :D | 21:48 |
| mgagne | we didn't go down the rabbit split brain path. we are using rabbit virtual host for now | 21:49 |
| klindgren | ah | 21:49 |
| klindgren | yea | 21:49 |
| klindgren | honestly on that | 21:49 |
| klindgren | I would have just spun up a new rabbitmq | 21:49 |
| klindgren | with everything stopped | 21:50 |
| mgagne | we already have a region running cells with vhost, no problem so far. | 21:50 |
| klindgren | changed the service configs to use the new rabbitmq and started everything up | 21:50 |
| mgagne | with rabbitmq clustering, this means 3 new servers to maintain. =) | 21:50 |
| klindgren | the thing we are seeing is that or cell app servers are pretty busy | 21:50 |
| klindgren | with nova-conductor and rabbitmq | 21:51 |
| *** chykz_ has joined #openstack-operators | 21:52 | |
| *** chykz has quit IRC | 21:54 | |
| *** chykz has joined #openstack-operators | 21:57 | |
| *** chykz_ has quit IRC | 22:00 | |
| *** chykz has quit IRC | 22:04 | |
| klindgren | mgagne, https://github.com/krislindgren/libvirt-tc-hook | 22:06 |
| klindgren | >_> | 22:06 |
| klindgren | <_< | 22:06 |
| mgagne | awesome, tyvm! | 22:07 |
| logan2 | wow that was on my todo list for next week... | 22:08 |
| logan2 | thanks | 22:12 |
| klindgren | now you guys can enter the lovely world of TC | 22:16 |
| klindgren | full of dark magic and headaches | 22:16 |
| *** csoukup has quit IRC | 22:19 | |
| mgagne | ^^' | 22:19 |
| *** matrohon has quit IRC | 22:22 | |
| klindgren | I want it to do more, but for now - it solved the asks from one of the business people here | 22:22 |
| klindgren | I should note that we only use this on some HV's | 22:22 |
| *** SimonChung1 has quit IRC | 22:22 | |
| *** SimonChung has joined #openstack-operators | 22:22 | |
| *** SimonChung has quit IRC | 22:23 | |
| *** SimonChung1 has joined #openstack-operators | 22:23 | |
| *** signed8bit has quit IRC | 22:26 | |
| *** signed8bit has joined #openstack-operators | 22:31 | |
| klindgren | I updated to include some install stuff | 22:31 |
| *** esker has joined #openstack-operators | 22:42 | |
| *** esker has quit IRC | 22:48 | |
| *** esker has joined #openstack-operators | 22:51 | |
| *** alop has quit IRC | 22:52 | |
| *** saneax has joined #openstack-operators | 22:52 | |
| *** esker has quit IRC | 22:56 | |
| *** setmason has quit IRC | 22:59 | |
| *** setmason has joined #openstack-operators | 23:00 | |
| *** SimonChung has joined #openstack-operators | 23:05 | |
| *** SimonChung1 has quit IRC | 23:05 | |
| *** SimonChung1 has joined #openstack-operators | 23:06 | |
| *** SimonChung has quit IRC | 23:06 | |
| *** SimonChung has joined #openstack-operators | 23:06 | |
| *** SimonChung1 has quit IRC | 23:06 | |
| *** VW_ has joined #openstack-operators | 23:16 | |
| *** VW_ has quit IRC | 23:18 | |
| *** Marga_ has quit IRC | 23:27 | |
| *** SimonChung1 has joined #openstack-operators | 23:34 | |
| *** SimonChung has quit IRC | 23:34 | |
| * mgagne trying to develop/patch code in openstack: http://imgur.com/gallery/t0XHtgJ | 23:40 | |
| *** setmason has quit IRC | 23:44 | |
| *** setmason has joined #openstack-operators | 23:52 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!