| *** pasquier-s has quit IRC | 00:02 | |
| *** VW_ has quit IRC | 00:03 | |
| *** VW__ has quit IRC | 00:14 | |
| *** pasquier-s has joined #openstack-operators | 00:15 | |
| *** ozialien_ has quit IRC | 00:18 | |
| *** blairo has joined #openstack-operators | 00:22 | |
| *** pasquier-s has quit IRC | 00:27 | |
| *** signed8bit has quit IRC | 00:28 | |
| *** david-lyle is now known as david-lyle_afk | 00:33 | |
| *** Kerralissa has joined #openstack-operators | 00:43 | |
| *** yapeng has joined #openstack-operators | 00:46 | |
| *** Kerralissa has quit IRC | 00:49 | |
| *** pasquier-s has joined #openstack-operators | 00:50 | |
| *** mdorman has quit IRC | 00:51 | |
| *** yapeng has quit IRC | 00:52 | |
| *** dmsimard_away is now known as dmsimard | 00:56 | |
| *** VW_ has joined #openstack-operators | 01:14 | |
| *** blairo has quit IRC | 01:15 | |
| *** furlongm has quit IRC | 01:15 | |
| *** VW_ has quit IRC | 01:15 | |
| *** blair has joined #openstack-operators | 01:17 | |
| *** hj-hp has joined #openstack-operators | 01:22 | |
| mfisch | we sync across DCs | 01:22 |
|---|---|---|
| mfisch | but we own the pipes so its doable | 01:22 |
| mfisch | we do use one node as primary in the LB even though galera is multi-master | 01:23 |
| jlk | yeah I don't really want to know what it would do to bandwidth to try and galera sync across DCs | 01:25 |
| *** Marga_ has joined #openstack-operators | 01:33 | |
| *** hj-hp has quit IRC | 01:33 | |
| *** SimonChung has quit IRC | 01:35 | |
| *** Marga_ has quit IRC | 01:37 | |
| *** Marga_ has joined #openstack-operators | 01:38 | |
| xavpaice | you're doing galera across DCs? | 01:42 |
| *** alop has quit IRC | 01:43 | |
| mfisch | yes | 01:43 |
| mfisch | the keystone traffic really isnt that much | 01:43 |
| xavpaice | we have galera within the DC, LB has one primary, but haven't attempted to replicate that to another DC yet | 01:43 |
| xavpaice | tokens is quite a bit | 01:43 |
| xavpaice | (particularly as we're using PKI tokens and they're huge) | 01:44 |
| mfisch | [ | 01:44 |
| mfisch | we;re not using PKI tokens currently | 01:44 |
| mfisch | but will have to consider that when we do | 01:44 |
| * xavpaice is considering switching to UUID | 01:44 | |
| mfisch | we don't really support the use case of getting a token and immediately using it in the other DC | 01:44 |
| mfisch | but we do have cross-site monitoring that uses replicated tokens | 01:45 |
| mfisch | they run every few minutes and I'd say we get 4-5 failures per day where they've not replicated | 01:45 |
| mfisch | at most it takes a few seconds | 01:45 |
| xavpaice | that's pretty reasonable | 01:45 |
| mfisch | we have an arbitrator in a 3rd DC to prevent split brain | 01:45 |
| xavpaice | so you're using Galera for the replication across DC? | 01:46 |
| mfisch | we still get random network glitches which can be annoying | 01:46 |
| mfisch | yeah | 01:46 |
| mfisch | 7 node cluster | 01:46 |
| mfisch | 3 per DC + 1 arb | 01:46 |
| xavpaice | cool - might have to go back and re-read the docs | 01:46 |
| xavpaice | although we don't have a vast amount of bandwidth between the DC | 01:46 |
| mfisch | we have really low latency which helps too | 01:47 |
| xavpaice | :) comms is expensive and slow in NZ | 01:47 |
| mfisch | 37ms latency now | 01:47 |
| mfisch | about 2000 miles | 01:47 |
| xavpaice | our DC's are miles closer than that, so I better have a look again at repl - once we've switched to uuid tokens | 01:48 |
| xavpaice | also considering the effect of PKIZ tokens in Juno | 01:49 |
| mfisch | I'd like to know about those if you do them | 01:50 |
| mfisch | the main reason we do one keystone is so that people can switch seamlessly between regions | 01:50 |
| mfisch | without recreating roles and projects | 01:50 |
| xavpaice | yeah, that's the same reason we have one keystone too - but using PKI tokens with >1 region makes the tokens massive | 01:52 |
| mfisch | you must have "north" and "south" DCs down there | 01:52 |
| mfisch | "Island 1" and "Island 2" ;) | 01:52 |
| xavpaice | kinda - currently they're pretty close but on opposite sides of a fault line | 01:52 |
| mfisch | thats no fun | 01:52 |
| mfisch | also hobbits and orcs in between too | 01:53 |
| xavpaice | lots of people do opposite ends of the north island, some do use the South Is as well but that increases costs quite a bit | 01:53 |
| xavpaice | it's the damn goblins, that's where the trouble is | 01:53 |
| *** yapeng has joined #openstack-operators | 02:00 | |
| jlk | Well for us, we have all our services in the same database cluster. so it wouldn't just be keystone going back and forth, it'd be nova and the others too | 02:00 |
| *** Marga_ has quit IRC | 02:14 | |
| *** Marga_ has joined #openstack-operators | 02:15 | |
| *** furlongm has joined #openstack-operators | 02:16 | |
| *** dmsimard is now known as dmsimard_away | 02:17 | |
| *** verdurin has quit IRC | 02:35 | |
| *** verdurin has joined #openstack-operators | 02:39 | |
| *** VW_ has joined #openstack-operators | 02:56 | |
| *** harlowja is now known as harlowja_away | 02:57 | |
| *** blair has quit IRC | 03:25 | |
| *** blair has joined #openstack-operators | 03:25 | |
| *** VW_ has quit IRC | 03:32 | |
| *** Marga__ has joined #openstack-operators | 03:34 | |
| *** Marga_ has quit IRC | 03:34 | |
| *** blair has quit IRC | 03:36 | |
| *** markvoelker has quit IRC | 03:57 | |
| *** markvoelker has joined #openstack-operators | 03:57 | |
| *** markvoelker has quit IRC | 04:02 | |
| *** VW_ has joined #openstack-operators | 04:09 | |
| *** SimonChung has joined #openstack-operators | 04:15 | |
| *** SimonChung1 has joined #openstack-operators | 04:21 | |
| *** SimonChung has quit IRC | 04:23 | |
| *** sanjayu has joined #openstack-operators | 04:27 | |
| *** ruagair has quit IRC | 04:49 | |
| *** ruagair has joined #openstack-operators | 05:03 | |
| *** VW_ has quit IRC | 05:10 | |
| *** VW_ has joined #openstack-operators | 05:12 | |
| *** markvoelker has joined #openstack-operators | 05:13 | |
| *** VW_ has quit IRC | 05:16 | |
| *** Marga__ has quit IRC | 05:17 | |
| *** pasquier-s has quit IRC | 05:24 | |
| *** markvoelker has quit IRC | 05:34 | |
| *** markvoelker has joined #openstack-operators | 05:34 | |
| *** yapeng has quit IRC | 05:37 | |
| *** VW_ has joined #openstack-operators | 05:38 | |
| *** markvoelker has quit IRC | 05:39 | |
| *** pasquier-s has joined #openstack-operators | 05:41 | |
| *** markvoelker has joined #openstack-operators | 06:35 | |
| *** markvoelker has quit IRC | 06:41 | |
| *** Kerralissa has joined #openstack-operators | 07:02 | |
| *** avozza is now known as zz_avozza | 07:20 | |
| *** yapeng has joined #openstack-operators | 07:25 | |
| *** VW_ has quit IRC | 07:28 | |
| *** yapeng has quit IRC | 07:30 | |
| *** SimonChung1 has quit IRC | 07:37 | |
| *** markvoelker has joined #openstack-operators | 07:38 | |
| *** SimonChung has joined #openstack-operators | 07:38 | |
| *** Miouge has joined #openstack-operators | 07:39 | |
| *** zz_avozza is now known as avozza | 07:40 | |
| *** markvoelker has quit IRC | 07:43 | |
| *** avozza is now known as zz_avozza | 07:50 | |
| *** belmoreira has joined #openstack-operators | 07:54 | |
| *** zz_avozza is now known as avozza | 07:54 | |
| *** SimonChung has quit IRC | 07:54 | |
| *** SimonChung1 has joined #openstack-operators | 07:54 | |
| *** avozza is now known as zz_avozza | 08:05 | |
| *** markvoelker has joined #openstack-operators | 08:13 | |
| *** markvoelker has quit IRC | 08:17 | |
| *** zz_avozza is now known as avozza | 08:35 | |
| *** Miouge has quit IRC | 08:38 | |
| *** bvandenh has joined #openstack-operators | 08:55 | |
| *** derekh has joined #openstack-operators | 09:02 | |
| *** markvoelker has joined #openstack-operators | 09:14 | |
| *** yapeng has joined #openstack-operators | 09:14 | |
| *** yapeng has quit IRC | 09:19 | |
| *** markvoelker has quit IRC | 09:19 | |
| *** Piet has quit IRC | 09:29 | |
| *** matrohon has joined #openstack-operators | 09:39 | |
| *** SimonChung1 has quit IRC | 09:41 | |
| *** Miouge has joined #openstack-operators | 09:45 | |
| *** markvoelker has joined #openstack-operators | 10:15 | |
| *** markvoelker has quit IRC | 10:22 | |
| *** yapeng has joined #openstack-operators | 11:03 | |
| *** avozza is now known as zz_avozza | 11:07 | |
| *** zz_avozza is now known as avozza | 11:08 | |
| *** yapeng has quit IRC | 11:08 | |
| *** pasquier-s has quit IRC | 11:24 | |
| *** pasquier-s has joined #openstack-operators | 11:37 | |
| *** SimonChung has joined #openstack-operators | 11:37 | |
| *** Kerralissa has quit IRC | 11:40 | |
| *** Miouge has quit IRC | 11:42 | |
| *** pboros has joined #openstack-operators | 12:08 | |
| *** markvoelker has joined #openstack-operators | 12:19 | |
| *** markvoelker has quit IRC | 12:24 | |
| *** subscope has quit IRC | 12:33 | |
| *** subscope has joined #openstack-operators | 12:33 | |
| *** Marga_ has joined #openstack-operators | 12:51 | |
| *** yapeng has joined #openstack-operators | 12:52 | |
| *** yapeng has quit IRC | 12:57 | |
| *** Marga_ has quit IRC | 13:04 | |
| *** Marga_ has joined #openstack-operators | 13:05 | |
| *** Marga_ has quit IRC | 13:10 | |
| *** Marga_ has joined #openstack-operators | 13:14 | |
| *** markvoelker has joined #openstack-operators | 13:20 | |
| *** markvoelker has quit IRC | 13:25 | |
| *** Marga_ has quit IRC | 13:25 | |
| *** sanjayu has quit IRC | 13:27 | |
| *** markvoelker has joined #openstack-operators | 13:36 | |
| *** Ctina_ has joined #openstack-operators | 14:04 | |
| *** Marga_ has joined #openstack-operators | 14:09 | |
| *** markvoelker has quit IRC | 14:11 | |
| *** VW_ has joined #openstack-operators | 14:33 | |
| *** yapeng has joined #openstack-operators | 14:41 | |
| *** yapeng has quit IRC | 14:46 | |
| *** matrohon has quit IRC | 14:51 | |
| *** dmsimard_away is now known as dmsimard | 15:10 | |
| *** signed8bit has joined #openstack-operators | 15:26 | |
| *** Miouge has joined #openstack-operators | 15:33 | |
| *** markvoelker has joined #openstack-operators | 15:47 | |
| *** Miouge_ has joined #openstack-operators | 15:47 | |
| *** Miouge has quit IRC | 15:48 | |
| *** Miouge_ is now known as Miouge | 15:48 | |
| *** harlowja_away has quit IRC | 15:49 | |
| *** mdorman has joined #openstack-operators | 15:50 | |
| *** hj-hp has joined #openstack-operators | 15:50 | |
| *** radez_g0n3 is now known as radez | 15:51 | |
| *** david-lyle_afk is now known as david-lyle | 16:01 | |
| *** reed has joined #openstack-operators | 16:06 | |
| *** Miouge has quit IRC | 16:16 | |
| *** avozza is now known as zz_avozza | 16:28 | |
| *** yapeng has joined #openstack-operators | 16:30 | |
| *** yapeng has quit IRC | 16:35 | |
| *** SimonChung1 has joined #openstack-operators | 16:46 | |
| *** SimonChung has quit IRC | 16:48 | |
| *** Marga_ has quit IRC | 16:51 | |
| *** Marga_ has joined #openstack-operators | 16:51 | |
| *** Marga_ has quit IRC | 16:56 | |
| *** zz_avozza is now known as avozza | 17:01 | |
| *** avozza is now known as zz_avozza | 17:08 | |
| *** hj-hp has quit IRC | 17:15 | |
| *** Miouge has joined #openstack-operators | 17:19 | |
| *** hj-hp has joined #openstack-operators | 17:21 | |
| *** Marga_ has joined #openstack-operators | 17:22 | |
| *** SimonChung1 has quit IRC | 17:23 | |
| *** alop has joined #openstack-operators | 17:24 | |
| *** Marga_ has quit IRC | 17:26 | |
| *** belmoreira has quit IRC | 17:43 | |
| *** bvandenh has quit IRC | 17:45 | |
| *** Miouge has quit IRC | 17:55 | |
| *** Marga_ has joined #openstack-operators | 17:58 | |
| *** Ctina_ has quit IRC | 17:59 | |
| *** Ctina__ has joined #openstack-operators | 17:59 | |
| *** derekh has quit IRC | 18:03 | |
| *** hj-hp has quit IRC | 18:03 | |
| *** SimonChung has joined #openstack-operators | 18:04 | |
| *** hj-hp has joined #openstack-operators | 18:11 | |
| *** yapeng has joined #openstack-operators | 18:19 | |
| *** yapeng has quit IRC | 18:24 | |
| *** Piet has joined #openstack-operators | 18:27 | |
| *** harlowja has joined #openstack-operators | 18:28 | |
| *** Miouge has joined #openstack-operators | 18:35 | |
| *** Miouge has quit IRC | 18:38 | |
| *** hj-hp has quit IRC | 18:42 | |
| *** hj-hp has joined #openstack-operators | 18:52 | |
| *** markvoelker has quit IRC | 18:56 | |
| *** markvoelker has joined #openstack-operators | 18:56 | |
| *** markvoelker has quit IRC | 19:01 | |
| *** VW_ has quit IRC | 19:02 | |
| *** VW_ has joined #openstack-operators | 19:07 | |
| *** hj-hp has quit IRC | 19:12 | |
| *** Miouge has joined #openstack-operators | 19:31 | |
| klindgren_ | Is anyone else running compute/neutron networking hosts on cent7 (rhel7)? Are you having issues with reboot commands powering off the server? | 19:37 |
| beddari | klindgren_: reboot commands? | 19:51 |
| klindgren_ | beddari, /sbin/reboot | 19:51 |
| beddari | inside the instances? not seen it | 19:52 |
| klindgren_ | noticing a pretty common issue that all of my servers running neutron-openvswtich-agent and openvswitch when I reboot the server via reboot | 19:52 |
| klindgren_ | the server actually powers off | 19:52 |
| beddari | hah :) that's ... off | 19:53 |
| klindgren_ | yea | 19:53 |
| beddari | sorry, have no idea | 19:53 |
| klindgren_ | epecially when you are patching for the glibc bug - good times | 19:53 |
| beddari | "can I leave the room yet?" "no, still two racks left" | 19:54 |
| beddari | hoping you got a bmc interface hehe | 19:54 |
| klindgren_ | well catchign this issue in our dev/test environments | 19:55 |
| klindgren_ | so.... its not hugely impactful yet | 19:56 |
| *** yapeng has joined #openstack-operators | 20:08 | |
| *** yapeng has quit IRC | 20:13 | |
| *** Piet has quit IRC | 20:49 | |
| mgagne | anyone using boot from volume with cells? | 20:56 |
| mgagne | Looks like 2 entries are created in the nova.block_device_mapping table in the API cell, confusing Nova when it comes to perform a snapshot/create an image. | 20:56 |
| mdorman | not us | 21:01 |
| mdorman | sorry :( | 21:01 |
| mgagne | mdorman: I just thought about a hypothesis. I think API cell creates the block_device_mapping first (with mostly empty fields), compute cell grabs the call, performs all the schmilblick and create an entry into its own compute cell database. Updated volume entry is then "replicated" to the api cell through nova-cells. What I suspect is that the match between the entries is done using the auto_increment id which doesn't match between both da | 21:08 |
| mdorman | that got cut off at “between both da..” | 21:09 |
| mdorman | but yeah i see what you’re getting at | 21:09 |
| mgagne | both databases and therefore a new entry (and chasos) is created in the api cell. | 21:09 |
| mgagne | 4:09 | 21:09 |
| mdorman | and that does seem quite likely | 21:09 |
| mgagne | mdorman: like with nova flavors =( | 21:09 |
| mdorman | yeah | 21:09 |
| mgagne | mdorman: it's really a show stopper tbh | 21:09 |
| mgagne | mdorman: tested against icehouse, not sure if it got fixed in later version, can't find any git log related to this issue | 21:10 |
| mdorman | that’s a real bummer | 21:14 |
| mdorman | seems like a pretty big bug. surprised others using cells haven’t run into this | 21:15 |
| mdorman | though i guess maybe a lot of people aren’t using boot from volume | 21:15 |
| *** zz_avozza is now known as avozza | 21:20 | |
| *** Piet has joined #openstack-operators | 21:31 | |
| *** radez is now known as radez_g0n3 | 21:31 | |
| *** Miouge has quit IRC | 21:36 | |
| *** markvoelker has joined #openstack-operators | 21:38 | |
| *** hj-hp has joined #openstack-operators | 21:53 | |
| mgagne | mdorman: we should ask rackspace, AFAIK, they have cells and recently had support for BFV | 21:56 |
| mgagne | mdorman: http://www.rackspace.com/knowledge_center/article/boot-a-server-from-a-cloud-block-storage-volume "Boot a server from a volume (Cloud Control Panel)" | 21:57 |
| *** yapeng has joined #openstack-operators | 21:57 | |
| *** yapeng has quit IRC | 22:02 | |
| *** hj-hp has quit IRC | 22:04 | |
| mdorman | yeah. VW_ any thoughts? | 22:04 |
| VW_ | hey golks | 22:04 |
| VW_ | folks even | 22:04 |
| VW_ | what's up | 22:04 |
| *** hj-hp has joined #openstack-operators | 22:05 | |
| *** hj-hp has quit IRC | 22:06 | |
| VW_ | mgagne: mdorman - I'd have to chat with Andrew Laski on how we made it all work | 22:06 |
| VW_ | but yes, we are using BFV with cells | 22:06 |
| VW_ | in fact we have two flavor classes running on diskless hypervisors | 22:07 |
| VW_ | but that's not to say we didn't have to put some "magic" in to make it all jive | 22:07 |
| VW_ | :) | 22:07 |
| VW_ | all our devs are the mid-cycle at the moment, though | 22:09 |
| *** Ctina__ has quit IRC | 22:11 | |
| *** SimonChung has quit IRC | 22:11 | |
| *** SimonChung has joined #openstack-operators | 22:11 | |
| *** SimonChung1 has joined #openstack-operators | 22:12 | |
| *** SimonChung has quit IRC | 22:12 | |
| mgagne | VW_: I found that IF I provide the device_name in the BDMv2, volume entry doesn't get duplicated. | 22:12 |
| mgagne | VW_: could be that if the device name is not passed, the compute node itself decides of the device name. Since it's done within the compute cell (not the api cell), api cell might have difficulties finding the volume. | 22:14 |
| mgagne | VW_: I found block_device_mapping_update_or_create in nova.db and the following comment: "Only update a BDM when device_name was provided." | 22:14 |
| mgagne | VW_: what a fail =) | 22:15 |
| *** hj-hp has joined #openstack-operators | 22:16 | |
| *** Marga_ has quit IRC | 22:17 | |
| mgagne | mdorman: ^ if you are interested | 22:18 |
| mdorman | ah yeah sorry, dealing with a network loop issue caused by ovs over here :) | 22:29 |
| mdorman | good info VW_ , thanks | 22:30 |
| *** SimonChung1 has quit IRC | 22:32 | |
| jlk | mdorman: "business as usual" | 22:34 |
| mgagne | mdorman: only thing I can think of to "fix" this is injecting the device_name in the BDMv2 if the user does not provide it and assume it would always be vda | 22:35 |
| * mgagne invokes the power of wafflehaus | 22:35 | |
| VW_ | mdorman, mgagne - captured some of the above - let me get with some of our devs and confirm/see if I can get you a better answer than "magic" :) | 22:48 |
| mdorman | htanks | 22:52 |
| *** hj-hp has quit IRC | 22:52 | |
| *** SimonChung has joined #openstack-operators | 22:54 | |
| mgagne | hahaha, instance lock doesn't work (in Icehouse) with cells. | 22:56 |
| mgagne | once locked, you are stucked | 22:56 |
| *** pboros has quit IRC | 22:57 | |
| mdorman | what’s instance locK/ | 23:02 |
| mdorman | ? | 23:02 |
| *** hj-hp has joined #openstack-operators | 23:03 | |
| *** SimonChung has quit IRC | 23:03 | |
| *** SimonChung1 has joined #openstack-operators | 23:03 | |
| VW_ | try a hard reboot | 23:03 |
| *** SimonChung has joined #openstack-operators | 23:04 | |
| *** SimonChung1 has quit IRC | 23:04 | |
| VW_ | I feel like we were trying to accomplish some things recently with lock and it turned out the hard reboot was a way around it, but I could be mistaken | 23:04 |
| mgagne | nova lock | 23:04 |
| *** SimonChung1 has joined #openstack-operators | 23:05 | |
| *** SimonChung has quit IRC | 23:05 | |
| mgagne | VW_: there is a typo in cells_api.py which got "fixed" but removing the override in cells_api for lock/unlock methods | 23:05 |
| mgagne | VM_: https://git.openstack.org/cgit/openstack/nova/commit/?id=fb9b60fb5d3084d4cab29936fd1f534e5599f231 | 23:05 |
| *** SimonChung has joined #openstack-operators | 23:05 | |
| *** SimonChung1 has quit IRC | 23:05 | |
| *** hj-hp has quit IRC | 23:17 | |
| *** dmsimard is now known as dmsimard_away | 23:21 | |
| *** blair has joined #openstack-operators | 23:27 | |
| *** VW_ has quit IRC | 23:36 | |
| *** signed8bit has quit IRC | 23:36 | |
| *** Piet has quit IRC | 23:38 | |
| *** hj-hp has joined #openstack-operators | 23:44 | |
| *** yapeng has joined #openstack-operators | 23:46 | |
| *** yapeng has quit IRC | 23:50 | |
| *** VW_ has joined #openstack-operators | 23:54 | |
| VW_ | nice - thanks mgagne | 23:55 |
| *** hj-hp has quit IRC | 23:56 | |
| *** Kerralissa has joined #openstack-operators | 23:56 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!