| *** markvoelker has quit IRC | 00:06 | |
| *** geoffarnold is now known as geoffarnoldX | 00:15 | |
| *** VW_ has quit IRC | 00:30 | |
| *** VW_ has joined #openstack-operators | 00:31 | |
| *** VW_ has quit IRC | 00:36 | |
| *** SimonChung1 has quit IRC | 00:37 | |
| *** priteau has joined #openstack-operators | 00:43 | |
| *** priteau has quit IRC | 00:48 | |
| *** geoffarnoldX is now known as geoffarnold | 00:50 | |
| *** geoffarnold is now known as geoffarnoldX | 00:59 | |
| *** harshs has joined #openstack-operators | 01:00 | |
| *** markvoelker has joined #openstack-operators | 01:07 | |
| *** markvoelker_ has joined #openstack-operators | 01:08 | |
| *** derekh has quit IRC | 01:11 | |
| *** markvoelker has quit IRC | 01:12 | |
| *** markvoelker_ has quit IRC | 01:16 | |
| *** sdake has joined #openstack-operators | 01:33 | |
| *** SimonChung has joined #openstack-operators | 01:38 | |
| *** SimonChung1 has joined #openstack-operators | 01:42 | |
| *** SimonChung has quit IRC | 01:45 | |
| *** gtt116_ has quit IRC | 01:45 | |
| *** alop has quit IRC | 01:53 | |
| *** geoffarnoldX is now known as geoffarnold | 02:21 | |
| *** sdake_ has joined #openstack-operators | 02:24 | |
| *** sdake has quit IRC | 02:27 | |
| *** sdake__ has joined #openstack-operators | 02:27 | |
| *** sdake_ has quit IRC | 02:30 | |
| *** sdake__ is now known as sdake | 02:33 | |
| *** markvoelker has joined #openstack-operators | 02:35 | |
| *** markvoelker has quit IRC | 02:49 | |
| *** markvoelker has joined #openstack-operators | 02:50 | |
| *** hakimo_ has joined #openstack-operators | 02:53 | |
| *** hakimo has quit IRC | 02:55 | |
| *** markvoelker has quit IRC | 03:06 | |
| *** sdake_ has joined #openstack-operators | 03:20 | |
| *** geoffarnold is now known as geoffarnoldX | 03:21 | |
| *** sdake has quit IRC | 03:25 | |
| *** flwang1 has quit IRC | 03:28 | |
| *** flwang1 has joined #openstack-operators | 03:34 | |
| *** sdake_ is now known as sdake | 03:40 | |
| *** sdake_ has joined #openstack-operators | 03:44 | |
| *** sdake_ has quit IRC | 03:45 | |
| *** sdake_ has joined #openstack-operators | 03:45 | |
| *** sdake has quit IRC | 03:48 | |
| *** gtt116 has joined #openstack-operators | 03:52 | |
| *** markvoelker has joined #openstack-operators | 03:53 | |
| *** markvoelker_ has joined #openstack-operators | 03:57 | |
| *** markvoelker has quit IRC | 03:58 | |
| *** markvoelker_ has quit IRC | 03:58 | |
| *** maishsk has quit IRC | 04:47 | |
| klindgren_ | sorrison, have you been able to reliably reproduce the ipset issue? | 04:52 |
|---|---|---|
| sorrison | klindgren_ no been playing with that now | 04:52 |
| sorrison | seems random to me | 04:52 |
| klindgren_ | We only set it in production and even then not very frequently (3 times since our kilo upgrade). Pre-prod and dev/test is fine | 04:52 |
| klindgren_ | see* | 04:52 |
| sorrison | I just hit it on 2/3 compute nodes in our dev env | 04:53 |
| klindgren_ | Once it churned on iptables - the other two times have been removing ipsets | 04:53 |
| sorrison | I have been spinning up 80 instances then deleting 80 instances | 04:53 |
| sorrison | I think that triggers it | 04:53 |
| klindgren_ | the thing I find weird is that - I am seeing a ton of repeating req-id's for security group removals | 04:54 |
| sorrison | yeah that's weird | 04:54 |
| klindgren_ | and logging of: "port_unbound(): net_uuid None not in local_vlan_map" | 04:55 |
| klindgren_ | Our logstash data rightn ow only goes back to 8/12 -which is why that stops - so I don't have pre-kilo to compare to | 04:55 |
| klindgren_ | TWC - confirmed in neutron they are seeing something simlar - but they aren't running into issues for it yet | 04:56 |
| klindgren_ | I don't think they use ipsets though | 04:56 |
| sorrison | If you need me to run any commands or things to search in logs let me know. Kilo neutron is our first neutron so haven't tried juno | 04:57 |
| *** maishsk has joined #openstack-operators | 04:57 | |
| sorrison | btw, you going to be in Tokyo? | 04:57 |
| klindgren_ | I will be | 04:58 |
| * klindgren_ needs to book a hotel :-/ | 04:58 | |
| klindgren_ | are you? | 04:58 |
| *** harshs has quit IRC | 05:02 | |
| sorrison | yip, hotel booked :-) | 05:06 |
| klindgren_ | Cool - save a little bit of room in your suit case :-) | 05:07 |
| *** sdake_ is now known as sdake | 05:09 | |
| *** derekh has joined #openstack-operators | 05:30 | |
| *** maishsk has quit IRC | 05:32 | |
| *** maishsk has joined #openstack-operators | 05:37 | |
| *** sdake has quit IRC | 05:39 | |
| *** derekh has quit IRC | 05:41 | |
| klindgren_ | sorrison, so it looks like whenever a security group member is updated, it also triggers a firewall refresh | 05:48 |
| klindgren_ | which also triggers the port unbound stff | 05:49 |
| *** sdake has joined #openstack-operators | 06:10 | |
| *** vsedelnik has joined #openstack-operators | 07:03 | |
| *** Piet has quit IRC | 07:06 | |
| *** baffle_ has joined #openstack-operators | 07:07 | |
| *** baffle_ has quit IRC | 07:07 | |
| *** sdake_ has joined #openstack-operators | 07:11 | |
| *** sdake has quit IRC | 07:13 | |
| *** sdake__ has joined #openstack-operators | 07:14 | |
| *** matrohon has joined #openstack-operators | 07:17 | |
| *** sdake_ has quit IRC | 07:17 | |
| *** priteau has joined #openstack-operators | 07:31 | |
| *** Marga_ has joined #openstack-operators | 07:49 | |
| *** Marga_ has quit IRC | 07:49 | |
| *** Marga_ has joined #openstack-operators | 07:49 | |
| *** derekh has joined #openstack-operators | 08:07 | |
| *** derekh has quit IRC | 08:08 | |
| *** derekh has joined #openstack-operators | 08:08 | |
| *** Marga_ has quit IRC | 08:11 | |
| *** Marga_ has joined #openstack-operators | 08:11 | |
| *** sdake has joined #openstack-operators | 08:12 | |
| *** sdake__ has quit IRC | 08:12 | |
| *** sdake_ has joined #openstack-operators | 08:12 | |
| *** sdake has quit IRC | 08:16 | |
| *** jhyang_ has joined #openstack-operators | 08:18 | |
| *** Marga_ has quit IRC | 08:25 | |
| *** bvandenh has joined #openstack-operators | 08:37 | |
| *** Marga_ has joined #openstack-operators | 08:41 | |
| *** bradjones has joined #openstack-operators | 08:47 | |
| *** bradjones has quit IRC | 08:47 | |
| *** bradjones has joined #openstack-operators | 08:47 | |
| *** maishsk has quit IRC | 08:54 | |
| *** Marga_ has quit IRC | 08:58 | |
| *** Marga_ has joined #openstack-operators | 08:59 | |
| *** Marga_ has quit IRC | 09:14 | |
| *** miyagishi_t has quit IRC | 10:12 | |
| *** vsedelnik has quit IRC | 11:01 | |
| *** bvandenh has quit IRC | 11:13 | |
| *** harshs has joined #openstack-operators | 11:14 | |
| *** harshs has quit IRC | 11:19 | |
| *** vsedelnik has joined #openstack-operators | 11:19 | |
| *** harshs has joined #openstack-operators | 11:21 | |
| *** harshs has quit IRC | 11:22 | |
| *** vsedelnik has quit IRC | 11:23 | |
| *** harshs has joined #openstack-operators | 11:28 | |
| *** harshs has quit IRC | 11:45 | |
| *** vsedelnik has joined #openstack-operators | 11:52 | |
| *** delattec has quit IRC | 12:00 | |
| *** cdelatte has quit IRC | 12:00 | |
| *** jhyang_ has quit IRC | 12:13 | |
| *** cdelatte has joined #openstack-operators | 12:15 | |
| *** delattec has joined #openstack-operators | 12:15 | |
| *** bvandenh has joined #openstack-operators | 12:17 | |
| *** geoffarnoldX is now known as geoffarnold | 12:20 | |
| *** vsedelnik has quit IRC | 12:34 | |
| *** vsedelnik has joined #openstack-operators | 12:35 | |
| *** delattec has quit IRC | 12:41 | |
| *** cdelatte has quit IRC | 12:41 | |
| *** cdelatte has joined #openstack-operators | 12:43 | |
| *** maishsk has joined #openstack-operators | 12:54 | |
| *** Marga_ has joined #openstack-operators | 12:54 | |
| *** vsedelnik has quit IRC | 13:07 | |
| *** vsedelnik has joined #openstack-operators | 13:09 | |
| *** esker has quit IRC | 13:15 | |
| *** esker has joined #openstack-operators | 13:15 | |
| *** Marga_ has quit IRC | 13:17 | |
| *** Marga_ has joined #openstack-operators | 13:18 | |
| *** esker has quit IRC | 13:20 | |
| *** vinsh_ is now known as Vinsh | 13:24 | |
| *** Marga_ has quit IRC | 13:27 | |
| *** laron has joined #openstack-operators | 13:32 | |
| *** dabernie_ has joined #openstack-operators | 13:42 | |
| *** dabernie_ has quit IRC | 13:42 | |
| *** sdake has joined #openstack-operators | 13:49 | |
| *** sdake_ has quit IRC | 13:51 | |
| *** alaski is now known as lascii | 13:55 | |
| *** esker has joined #openstack-operators | 13:56 | |
| *** esker has quit IRC | 13:56 | |
| *** esker has joined #openstack-operators | 13:56 | |
| *** vsedelnik has quit IRC | 14:04 | |
| *** sdake_ has joined #openstack-operators | 14:07 | |
| *** sdake has quit IRC | 14:10 | |
| *** vsedelnik has joined #openstack-operators | 14:11 | |
| *** csoukup has joined #openstack-operators | 14:13 | |
| *** VW_ has joined #openstack-operators | 14:14 | |
| *** VW_ has quit IRC | 14:14 | |
| *** VW_ has joined #openstack-operators | 14:15 | |
| *** matrohon has quit IRC | 14:15 | |
| *** maishsk_ has joined #openstack-operators | 14:28 | |
| *** maishsk has quit IRC | 14:29 | |
| *** maishsk_ is now known as maishsk | 14:29 | |
| *** maishsk has quit IRC | 14:34 | |
| klindgren_ | sorrison, on your 80 vm spin up/spin down are you running those all under the same project? IE is it having to update ipsets across a large number of hosts? | 14:42 |
| klindgren_ | One of the default rules is a rule that allows communication from vm's in the same project. Wondering if its a race condition due to that where the ipset info gets out of sync, I haven't checked the code. But thinking it's possible that an ipset update made it into a previous ipset removal, then a second thread for ipset updates comes along an they dont correctly handle removing an laready remove ipset | 14:45 |
| *** maishsk has joined #openstack-operators | 14:47 | |
| *** maishsk has quit IRC | 14:53 | |
| *** geoffarnold is now known as geoffarnoldX | 14:54 | |
| *** laron has quit IRC | 15:06 | |
| *** emagana has joined #openstack-operators | 15:10 | |
| egon | we've definitely had performance problems provisioning VMs in large pools with that rule in place. | 15:10 |
| egon | the port takes too long to configure, and the dhcp packets don't make it. | 15:11 |
| *** VW_ has quit IRC | 15:12 | |
| *** SimonChung1 has quit IRC | 15:12 | |
| *** VW_ has joined #openstack-operators | 15:12 | |
| *** VW_ has quit IRC | 15:14 | |
| mgagne | is there a doc for humans somewhere about how to build coreos with a custom oem overlay? | 15:15 |
| *** priteau has quit IRC | 15:15 | |
| *** VW_ has joined #openstack-operators | 15:16 | |
| *** geoffarnoldX is now known as geoffarnold | 15:17 | |
| *** sdake has joined #openstack-operators | 15:21 | |
| *** alop has joined #openstack-operators | 15:21 | |
| *** sdake_ has quit IRC | 15:24 | |
| *** sdake has quit IRC | 15:25 | |
| *** VW__ has joined #openstack-operators | 15:28 | |
| *** VW_ has quit IRC | 15:29 | |
| *** vsedelnik has quit IRC | 15:30 | |
| *** vsedelnik has joined #openstack-operators | 15:33 | |
| *** VW__ has quit IRC | 15:36 | |
| *** VW_ has joined #openstack-operators | 15:36 | |
| *** VW_ has quit IRC | 15:41 | |
| klindgren_ | lol @ doc for humans - but I felt the same way last time I worked with CoreOS | 15:41 |
| mgagne | but really, how are people doing it? :-/ | 15:45 |
| mgagne | I found https://coreos.com/os/docs/latest/notes-for-distributors.html but after I injected /usr/share/oem/cloud-config.yml, it got override by something else :-/ | 15:46 |
| *** wasmum has joined #openstack-operators | 15:48 | |
| mgagne | I'm just wondering why I can't take a base image and install my overlay without having to rebuild from sources :-/ | 15:53 |
| *** geoffarnold has quit IRC | 16:02 | |
| *** VW_ has joined #openstack-operators | 16:08 | |
| wasmum | anyone using elasticsearch and kibana 4? Anyone create the dashboard yet or ported it over from kibana 3? | 16:24 |
| *** bvandenh has quit IRC | 16:26 | |
| *** SimonChung has joined #openstack-operators | 16:27 | |
| *** esker has quit IRC | 16:29 | |
| *** vsedelnik has quit IRC | 16:37 | |
| *** geoffarnold has joined #openstack-operators | 16:44 | |
| *** derekh has quit IRC | 16:59 | |
| *** VW_ has quit IRC | 17:18 | |
| *** VW_ has joined #openstack-operators | 17:19 | |
| *** sdake has joined #openstack-operators | 17:19 | |
| *** j^2 has quit IRC | 17:20 | |
| *** j^2 has joined #openstack-operators | 17:20 | |
| *** carl_baldwin has joined #openstack-operators | 17:21 | |
| *** VW_ has quit IRC | 17:23 | |
| klindgren_ | wasmum, we (Godaddy) haven't made the change over to kabana4 - not really sure I love the new look/feel to be honest. Also, not in love with the new requirements. So we for the time being are still rocking older elasticsearch + older kibana | 17:25 |
| *** carl_baldwin has quit IRC | 17:33 | |
| *** jaypipes has quit IRC | 17:33 | |
| *** esker has joined #openstack-operators | 17:34 | |
| *** esker has quit IRC | 17:34 | |
| *** esker has joined #openstack-operators | 17:35 | |
| *** Piet has joined #openstack-operators | 17:35 | |
| *** dminer has joined #openstack-operators | 17:39 | |
| *** vsedelnik has joined #openstack-operators | 17:43 | |
| *** VW_ has joined #openstack-operators | 17:45 | |
| *** devlaps has joined #openstack-operators | 17:56 | |
| *** vsedelnik has quit IRC | 17:56 | |
| wasmum | klindgren: agree with everything you said. obstacles we're facing is esec pounding v3 | 18:01 |
| *** SimonChung has quit IRC | 18:02 | |
| *** SimonChung1 has joined #openstack-operators | 18:02 | |
| *** gyee has joined #openstack-operators | 18:08 | |
| *** VW_ has quit IRC | 18:16 | |
| *** VW_ has joined #openstack-operators | 18:17 | |
| *** bvandenh has joined #openstack-operators | 18:19 | |
| *** VW_ has quit IRC | 18:21 | |
| *** bvandenh has quit IRC | 18:30 | |
| *** VW_ has joined #openstack-operators | 18:35 | |
| *** sdake has quit IRC | 18:45 | |
| *** vsedelnik has joined #openstack-operators | 18:53 | |
| *** mpopow has joined #openstack-operators | 19:02 | |
| *** SimonChung1 has quit IRC | 19:03 | |
| *** SimonChung has joined #openstack-operators | 19:03 | |
| *** bradjones has quit IRC | 19:03 | |
| *** WormMan has quit IRC | 19:05 | |
| *** resker has joined #openstack-operators | 19:06 | |
| *** sdake has joined #openstack-operators | 19:07 | |
| *** vsedelnik has quit IRC | 19:08 | |
| *** vsedelnik has joined #openstack-operators | 19:09 | |
| *** vsedelnik has quit IRC | 19:09 | |
| *** vsedelnik has joined #openstack-operators | 19:09 | |
| *** esker has quit IRC | 19:10 | |
| *** bradjones has joined #openstack-operators | 19:10 | |
| *** bradjones has quit IRC | 19:10 | |
| *** bradjones has joined #openstack-operators | 19:10 | |
| *** Marga_ has joined #openstack-operators | 19:29 | |
| *** Marga_ has quit IRC | 19:29 | |
| *** Marga_ has joined #openstack-operators | 19:30 | |
| *** SimonChung has quit IRC | 19:37 | |
| *** cdelatte has quit IRC | 19:49 | |
| *** Marga_ has quit IRC | 20:05 | |
| *** vsedelnik has quit IRC | 20:15 | |
| *** __nick has joined #openstack-operators | 20:15 | |
| *** markd_ has joined #openstack-operators | 20:15 | |
| *** _nick has quit IRC | 20:16 | |
| *** mdavidson has quit IRC | 20:16 | |
| *** lascii has quit IRC | 20:16 | |
| *** alaski has joined #openstack-operators | 20:17 | |
| *** matrohon has joined #openstack-operators | 20:23 | |
| *** alaski is now known as lascii | 20:24 | |
| *** SimonChung has joined #openstack-operators | 20:33 | |
| *** Piet has quit IRC | 20:38 | |
| *** gyee has quit IRC | 20:50 | |
| *** gyee has joined #openstack-operators | 20:50 | |
| mgagne | jlk: I think this could fix the redirect problem: https://gist.github.com/mgagne/b4c817cd2aa86b37daee | 20:57 |
| *** HenryG has quit IRC | 21:07 | |
| *** dminer has quit IRC | 21:07 | |
| *** gyee has quit IRC | 21:08 | |
| *** sgordon has quit IRC | 21:09 | |
| *** HenryG has joined #openstack-operators | 21:11 | |
| *** geoffarnold has quit IRC | 21:11 | |
| *** whydidyoustealmy has quit IRC | 21:11 | |
| *** sgordon has joined #openstack-operators | 21:13 | |
| *** geoffarnold has joined #openstack-operators | 21:14 | |
| *** whydidyoustealmy has joined #openstack-operators | 21:18 | |
| *** gyee has joined #openstack-operators | 21:21 | |
| *** VW_ has quit IRC | 21:24 | |
| *** lascii is now known as alaski | 21:25 | |
| *** VW_ has joined #openstack-operators | 21:25 | |
| *** matrohon has quit IRC | 21:28 | |
| *** SimonChung1 has joined #openstack-operators | 21:28 | |
| *** carl_baldwin has joined #openstack-operators | 21:29 | |
| *** SimonChung has quit IRC | 21:29 | |
| *** VW_ has quit IRC | 21:30 | |
| *** kurt_ny has joined #openstack-operators | 21:30 | |
| *** carl_baldwin has quit IRC | 21:36 | |
| *** dminer has joined #openstack-operators | 21:50 | |
| jlk | ah, yeah. Most likely yes. That was our conclusion too, that Nova just isn't hanlding the X_FORWARDED correctly | 21:55 |
| *** kurt_ny has quit IRC | 22:06 | |
| *** geoffarnold has quit IRC | 22:07 | |
| *** sdake_ has joined #openstack-operators | 22:15 | |
| *** Piet has joined #openstack-operators | 22:15 | |
| *** sdake has quit IRC | 22:18 | |
| *** ed_b has joined #openstack-operators | 22:28 | |
| *** VW_ has joined #openstack-operators | 22:29 | |
| *** csoukup has quit IRC | 22:30 | |
| *** VW__ has joined #openstack-operators | 22:32 | |
| *** VW_ has quit IRC | 22:36 | |
| *** VW__ has quit IRC | 22:37 | |
| *** emagana has quit IRC | 22:38 | |
| *** dminer has quit IRC | 23:12 | |
| *** alop has quit IRC | 23:41 | |
| *** gyee has quit IRC | 23:41 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!