| *** Acinonyx has quit IRC | 00:14 | |
| *** slaweq has quit IRC | 00:32 | |
| *** aspiers has quit IRC | 00:37 | |
| *** kjackal has quit IRC | 00:45 | |
| *** fsimonce has quit IRC | 01:30 | |
| *** gyee has quit IRC | 01:39 | |
| *** cloudrancher has joined #openstack | 01:41 | |
| *** slaweq has joined #openstack | 01:42 | |
| *** slaweq has quit IRC | 01:47 | |
| *** dskull has quit IRC | 02:13 | |
| *** dskull has joined #openstack | 02:15 | |
| *** dskull has quit IRC | 02:17 | |
| *** dskull has joined #openstack | 02:19 | |
| *** Son_Goku has joined #openstack | 02:23 | |
| *** mrsoul has quit IRC | 02:36 | |
| *** harlowja has quit IRC | 02:58 | |
| *** sunil_bansal has joined #openstack | 03:02 | |
| *** sunil_bansal has quit IRC | 03:07 | |
| *** lborda has quit IRC | 03:41 | |
| *** yamahata has quit IRC | 03:48 | |
| *** sunil_bansal has joined #openstack | 04:02 | |
| *** sunil_bansal has quit IRC | 04:06 | |
| *** sunil_bansal has joined #openstack | 04:22 | |
| *** sunil_bansal has quit IRC | 04:27 | |
| *** faizy_ has joined #openstack | 04:42 | |
| *** faizy98 has quit IRC | 04:45 | |
| *** tserong_ has joined #openstack | 05:06 | |
| *** tserong has quit IRC | 05:08 | |
| *** Son_Goku has quit IRC | 05:24 | |
| *** radeks_ has joined #openstack | 05:38 | |
| *** radeks__ has joined #openstack | 05:40 | |
| *** bdperkin has quit IRC | 05:41 | |
| *** cloudrancher has quit IRC | 05:43 | |
| *** radeks_ has quit IRC | 05:43 | |
| *** cloudrancher has joined #openstack | 05:43 | |
| *** bdperkin has joined #openstack | 06:00 | |
| *** tserong_ is now known as tserong | 06:02 | |
| *** yamahata has joined #openstack | 06:25 | |
| *** aspiers has joined #openstack | 06:25 | |
| *** radeks__ has quit IRC | 06:32 | |
| *** sauvin has joined #openstack | 06:41 | |
| *** skazi has quit IRC | 06:53 | |
| *** skazi has joined #openstack | 06:59 | |
| *** kjackal has joined #openstack | 06:59 | |
| *** Warped has quit IRC | 07:12 | |
| *** belmoreira has joined #openstack | 07:24 | |
| *** belmoreira has quit IRC | 07:33 | |
| *** Warped has joined #openstack | 07:38 | |
| *** kjackal has quit IRC | 07:42 | |
| *** slaweq has joined #openstack | 08:16 | |
| *** slaweq has quit IRC | 08:42 | |
| *** yamahata has quit IRC | 08:45 | |
| *** adrianc has joined #openstack | 09:02 | |
| *** adrianc has quit IRC | 09:21 | |
| *** radeks__ has joined #openstack | 09:29 | |
| *** radeks__ has quit IRC | 09:40 | |
| *** sshnaidm|off has quit IRC | 10:04 | |
| *** sshnaidm|off has joined #openstack | 10:10 | |
| *** Son_Goku has joined #openstack | 10:29 | |
| *** jistr has quit IRC | 10:30 | |
| *** ivve has joined #openstack | 10:31 | |
| *** jistr has joined #openstack | 10:31 | |
| *** radeks__ has joined #openstack | 10:31 | |
| *** bobh has joined #openstack | 10:33 | |
| *** Acinonyx has joined #openstack | 10:42 | |
| *** belmoreira has joined #openstack | 10:48 | |
| *** belmoreira has quit IRC | 10:49 | |
| *** faizy98 has joined #openstack | 10:54 | |
| *** faizy_ has quit IRC | 10:57 | |
| *** suuuper has joined #openstack | 10:58 | |
| *** sunil_bansal has joined #openstack | 11:02 | |
| *** bobh has quit IRC | 11:02 | |
| *** radeks__ has quit IRC | 11:09 | |
| *** luizbag has joined #openstack | 11:42 | |
| *** kjackal has joined #openstack | 11:59 | |
| *** dasTor has joined #openstack | 12:23 | |
| *** radeks__ has joined #openstack | 12:33 | |
| *** LinuxKnight has quit IRC | 12:34 | |
| *** LinuxKnight has joined #openstack | 12:35 | |
| *** radeks__ has quit IRC | 12:39 | |
| *** luizbag has quit IRC | 12:40 | |
| *** dasTor has quit IRC | 12:52 | |
| *** Liang__ has joined #openstack | 13:05 | |
| *** zimboboyd has joined #openstack | 13:11 | |
| *** gmoro has joined #openstack | 13:12 | |
| *** Acinonyx has quit IRC | 13:14 | |
| *** gmoro has quit IRC | 13:20 | |
| *** ianychoi has quit IRC | 14:09 | |
| *** kjackal has quit IRC | 14:16 | |
| *** kjackal_v2 has joined #openstack | 14:16 | |
| *** Geronimo has joined #openstack | 14:18 | |
| *** tolisbar has joined #openstack | 14:22 | |
| *** slaweq has joined #openstack | 14:23 | |
| *** dhill_ has joined #openstack | 14:36 | |
| *** tolisbar1 has joined #openstack | 14:44 | |
| *** tolisbar1 has quit IRC | 14:44 | |
| *** dhill_ has quit IRC | 14:45 | |
| *** tolisbar has quit IRC | 14:46 | |
| *** belmoreira has joined #openstack | 14:56 | |
| *** belmorei_ has joined #openstack | 14:58 | |
| *** belmorei_ has quit IRC | 14:59 | |
| *** belmore__ has joined #openstack | 14:59 | |
| *** belmoreira has quit IRC | 15:01 | |
| *** radeks__ has joined #openstack | 15:01 | |
| *** belmoreira has joined #openstack | 15:03 | |
| *** belmore__ has quit IRC | 15:04 | |
| *** belmoreira has quit IRC | 15:04 | |
| *** belmoreira has joined #openstack | 15:04 | |
| *** slaweq has quit IRC | 15:07 | |
| *** belmoreira has quit IRC | 15:09 | |
| *** belmoreira has joined #openstack | 15:11 | |
| *** greatgatsby has quit IRC | 15:15 | |
| *** belmoreira has quit IRC | 15:24 | |
| *** radeks__ has quit IRC | 15:28 | |
| *** gmoro has joined #openstack | 15:29 | |
| *** slaweq has joined #openstack | 15:32 | |
| *** gmoro has quit IRC | 15:33 | |
| *** belmoreira has joined #openstack | 15:39 | |
| *** slaweq has quit IRC | 15:40 | |
| *** [diablo] has joined #openstack | 16:03 | |
| *** jtomasek has joined #openstack | 16:08 | |
| *** mgutehall has joined #openstack | 16:19 | |
| *** hoonetorg has quit IRC | 16:30 | |
| *** belmoreira has quit IRC | 16:34 | |
| *** belmoreira has joined #openstack | 16:40 | |
| *** hoonetorg has joined #openstack | 16:42 | |
| *** belmoreira has quit IRC | 16:56 | |
| *** vicious has quit IRC | 17:07 | |
| *** radeks__ has joined #openstack | 17:09 | |
| *** vicious has joined #openstack | 17:14 | |
| *** imacdonn has quit IRC | 17:16 | |
| *** imacdonn has joined #openstack | 17:16 | |
| *** dasTor has joined #openstack | 17:27 | |
| *** bringha has joined #openstack | 17:39 | |
| *** tolisbar has joined #openstack | 17:48 | |
| *** faizy_ has joined #openstack | 17:52 | |
| *** tolisbar1 has joined #openstack | 17:53 | |
| *** tolisbar has quit IRC | 17:54 | |
| *** tolisbar1 is now known as tolisbar | 17:54 | |
| *** faizy98 has quit IRC | 17:56 | |
| *** dasTor has quit IRC | 18:06 | |
| netserf | Hi, I reported an issue with an openstack private cloud I've been using a few days ago (coming from the tenant perspective). | 18:07 |
|---|---|---|
| *** slaweq has joined #openstack | 18:08 | |
| *** mgutehall has quit IRC | 18:08 | |
| netserf | I went back and built 4 VMs using HEAT - varying complexity. All 4 instances came online without issue. | 18:08 |
| netserf | I setup a script to poll the hosts to see if SSH was available. The next day ~12 hours later all 4 VMs failed within a minute of each other. | 18:09 |
| larsks | Were the vms still running? | 18:10 |
| larsks | Ie, is this a "vms are crashing" issue, or a network issue? | 18:10 |
| netserf | Using the openstack OSC client, I saw the hosts were still up. I tried to stop/start them all. None of them came back online. | 18:11 |
| netserf | I also tried the Horizon console. None were available there either. | 18:11 |
| *** crisloma has joined #openstack | 18:12 | |
| larsks | So they show was shutoff or something else other than active? | 18:12 |
| netserf | I have other VMs online, so its not a network issue. | 18:12 |
| larsks | Oh, interesting. So it's just some *specific* vms but not a general problem? | 18:12 |
| netserf | Yes, they showed as SHUTOFF. | 18:12 |
| *** slin has joined #openstack | 18:12 | |
| netserf | That's right. | 18:12 |
| larsks | Is there anything in particular that differentiates these from the ones that are working? Distribution, application, etc? | 18:13 |
| netserf | At the recommendation of the admin, I built these latest VMs with varying levels of complexity, but the most | 18:13 |
| netserf | complex one was the same as one of the ones that is still working. | 18:14 |
| netserf | The other odd behaviour is that I tried deploying software to one of my earlier VMs and it took over 24 hours to install | 18:15 |
| *** dtrainor has joined #openstack | 18:15 | |
| netserf | I tried the exact same build in a public openstack. It took 10 minutes ... same resource allocations. | 18:15 |
| larsks | The only thing I can think of on the openstack side would be compute hosts running out of memory: if they have swap, then as memory pressure increases i/o performance would suffer. When they run out of memory, the kernel will start killing things. | 18:16 |
| larsks | That matches the symptoms you are reporting, but there are probably other causes I'm not thinking of that could cause the same behavior. | 18:16 |
| netserf | Yeah, and that makes sense to me. I'm trying to convince the admin that this may be the case, but why is openstack showing that we have plenty of resources. | 18:17 |
| netserf | I'm thinking maybe the openstack platform is not aware of the resource constraints at the hypervisor level, maybe? | 18:18 |
| larsks | That is a good question but really requires access to the compute hosts to diagnose. | 18:18 |
| netserf | yeah, totally. Unfortunately, my openstack support has been quite opaque. | 18:18 |
| netserf | Its frustrating because the admins are telling me others work in the same environment, so it must be something I'm doing. | 18:19 |
| larsks | The only other thing I can think to check is watching the console log for the failing instances and see if there are any clues there. | 18:20 |
| netserf | I'm using HEAT and a vanilla CentOS 7 image and it works in a 3rd party openstack. What else can I demonstrate to make them see the light? | 18:20 |
| *** suuuper has quit IRC | 18:21 | |
| netserf | The console logs wouldn't report anything after the VMs failed. | 18:21 |
| larsks | Ignoring Heat, do you see the same behavior if you just boot a generic CentOS image using Nova directly? | 18:21 |
| *** gmoro has joined #openstack | 18:21 | |
| netserf | So just bring up a VM through Horizon or the cli? | 18:22 |
| larsks | Yeah. I mean, if you boot a generic CentOS VM with Nova and it fails, there is really nothing that you could be doing that could cause a failure. | 18:22 |
| larsks | Assuming that it's consistently reproducible, that seems like a pretty concrete case for further administrative investigation on the backend. | 18:23 |
| netserf | I haven't tried to remove HEAT from the equation. I'll give that a try. Thanks! | 18:23 |
| larsks | fwiw, you don't need to write Heat in all caps :) | 18:23 |
| netserf | Yeah, I think so too and they are seeing the light. I have a meeting scheduled on Monday to review. :-) | 18:24 |
| netserf | Oh thanks ... I'm still learning the openstack lingo. :-) | 18:24 |
| netserf | Awesome larsks! Thank you for the ideas. Cheers! | 18:25 |
| larsks | Sure! | 18:25 |
| *** radeks__ has quit IRC | 18:26 | |
| *** jtomasek has quit IRC | 18:31 | |
| *** gmoro has quit IRC | 18:33 | |
| *** sunil_bansal has quit IRC | 18:44 | |
| *** greatgatsby has joined #openstack | 18:48 | |
| *** K0HAX_ has joined #openstack | 18:48 | |
| *** K0HAX has quit IRC | 18:49 | |
| *** slaweq has quit IRC | 18:57 | |
| *** dtrainor has quit IRC | 18:57 | |
| *** slaweq has joined #openstack | 19:04 | |
| *** slaweq has quit IRC | 19:11 | |
| *** slaweq has joined #openstack | 19:26 | |
| *** Son_Goku has quit IRC | 20:15 | |
| *** trident has quit IRC | 20:19 | |
| *** trident has joined #openstack | 20:20 | |
| *** bobh has joined #openstack | 20:20 | |
| *** slaweq has quit IRC | 20:24 | |
| *** bobh has quit IRC | 20:30 | |
| *** crisloma has quit IRC | 20:31 | |
| *** belmoreira has joined #openstack | 20:47 | |
| *** Geronimo has left #openstack | 20:54 | |
| *** shrasool has joined #openstack | 21:04 | |
| *** shrasool has quit IRC | 21:11 | |
| *** Son_Goku has joined #openstack | 21:17 | |
| *** bobh has joined #openstack | 21:18 | |
| *** bobh has quit IRC | 21:28 | |
| *** kjackal_v2 has quit IRC | 21:33 | |
| *** belmoreira has quit IRC | 21:34 | |
| *** shrasool has joined #openstack | 21:52 | |
| *** radeks has joined #openstack | 21:52 | |
| *** radeks has quit IRC | 22:02 | |
| *** bobh has joined #openstack | 22:28 | |
| *** bobh has quit IRC | 22:35 | |
| *** rossella_s has quit IRC | 22:44 | |
| *** faizy98 has joined #openstack | 22:45 | |
| *** faizy_ has quit IRC | 22:47 | |
| *** phausman has joined #openstack | 22:56 | |
| *** ivve has quit IRC | 23:01 | |
| *** StuartMI has quit IRC | 23:13 | |
| *** boazel has joined #openstack | 23:16 | |
| *** boazel_ has quit IRC | 23:19 | |
| *** sm806 has quit IRC | 23:21 | |
| *** sm806 has joined #openstack | 23:22 | |
| *** shrasool has quit IRC | 23:28 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!