Wednesday, 2016-11-16

*** zul has quit IRC00:13
*** zul has joined #openstack-operators00:43
*** arif-ali has quit IRC00:51
*** klindgren__ has joined #openstack-operators00:54
*** bdeschenes has quit IRC00:55
*** arif-ali has joined #openstack-operators00:56
*** klindgren_ has quit IRC00:56
*** mriedem has joined #openstack-operators01:45
mriedemis anyone relying on file injection in their cloud rather than config drive?01:50
mriedemhttp://lists.openstack.org/pipermail/openstack-operators/2016-November/012057.html01:50
mriedemnova would like to deprecate file injection01:50
mriedemotherwise known as the personality files API01:50
*** ckonstanski has quit IRC02:38
*** Apoorva has quit IRC02:39
*** armax has quit IRC02:44
*** fifieldt has joined #openstack-operators02:51
*** armax has joined #openstack-operators03:07
*** ckonstanski has joined #openstack-operators03:08
*** mriedem has quit IRC03:18
*** udesale has joined #openstack-operators03:25
*** armax has quit IRC03:39
*** fragatin_ has joined #openstack-operators03:47
*** fragatina has quit IRC03:51
*** fragatin_ has quit IRC03:52
*** fifieldt has quit IRC04:03
*** racedo has joined #openstack-operators04:05
*** armax has joined #openstack-operators04:16
*** racedo has quit IRC04:17
*** sudipto has joined #openstack-operators04:24
*** khamtamtun has joined #openstack-operators04:40
*** khamtamtun has quit IRC04:44
*** sudipto has quit IRC04:53
*** darrenc is now known as darrenc_afk05:19
*** dims has quit IRC05:20
*** twiggy has joined #openstack-operators05:40
*** harlowja has quit IRC05:55
*** darrenc_afk is now known as darrenc05:57
*** dims has joined #openstack-operators05:59
*** markvoelker has quit IRC06:28
*** Miouge has joined #openstack-operators06:41
*** Miouge has quit IRC06:43
*** Miouge has joined #openstack-operators06:51
*** armax has quit IRC06:52
*** slaweq has joined #openstack-operators07:38
*** simon-AS559 has quit IRC07:38
*** pcaruana has joined #openstack-operators07:41
*** simon-AS559 has joined #openstack-operators07:56
*** dbecker has joined #openstack-operators08:05
*** markvoelker has joined #openstack-operators08:29
*** MrDan has joined #openstack-operators08:34
*** markvoelker has quit IRC08:34
*** saneax-_-|AFK is now known as saneax08:36
*** saneax is now known as saneax-_-|AFK08:55
*** rarcea has joined #openstack-operators08:57
*** saneax-_-|AFK is now known as saneax09:00
*** zioproto has joined #openstack-operators09:12
*** saneax is now known as saneax-_-|AFK09:13
zioprotoHello09:14
zioprotoanyone knows if existing neutron routers can be set to ha=True if they have been initially created with ha=False ?09:15
*** makowals has quit IRC09:22
*** makowals has joined #openstack-operators09:22
*** bdeschenes has joined #openstack-operators09:24
*** twiggy has quit IRC09:30
*** paramite has joined #openstack-operators09:33
*** matrohon has joined #openstack-operators09:36
*** makowals has quit IRC09:36
_nickzioproto: yeah they can09:36
*** makowals has joined #openstack-operators09:36
_nickif you're on liberty then you can just migrate legacy routers to l3-ha in place09:37
*** twiggy has joined #openstack-operators09:37
_nickif you're on kilo there's a patch that was backported which you can apply09:37
_nickzioproto: https://bugs.launchpad.net/neutron/+bug/136542609:37
openstackLaunchpad bug 1365426 in neutron "Support migration of non-HA router to HA and reverse" [Medium,Fix released] - Assigned to John Schwarz (jschwarz)09:37
*** saneax-_-|AFK is now known as saneax09:38
_nickworked for us quite nicely on kilo09:38
*** _nick is now known as yankcrime09:38
zioprotogreat09:40
zioprotoI just found the way09:40
zioprotothe does not come with the openstack client09:40
yankcrime👍09:40
zioprotoI had to use the neutron cli client09:40
yankcrimeyeah09:41
zioprotoit is so annoying the openstack client still misses so many features09:41
zioproto:)09:41
zioprotothanks !09:41
yankcrimethere's still quite a few esoteric options missing from the openstack client09:41
*** derekh has joined #openstack-operators09:41
yankcrimedoesn't surprise me that's one of them09:41
*** electrofelix has joined #openstack-operators09:54
*** saneax is now known as saneax-_-|AFK09:59
*** dbecker has quit IRC10:04
*** udesale has quit IRC10:26
*** markvoelker has joined #openstack-operators10:30
*** markvoelker has quit IRC10:35
*** bdeschenes has quit IRC10:37
zioprotoI upgraded my openstack client and now when trying 'openstack network list' I get a nice 'ascii' codec can't encode characters in position 5441-5446: ordinal not in range(128)10:40
zioprotoand same for neutron net-list10:40
zioprotoI guess is some low level library10:40
zioprotoany hint ?10:41
*** dbecker has joined #openstack-operators10:50
*** zioproto has quit IRC10:53
*** zioproto has joined #openstack-operators11:56
*** stupidnic has quit IRC12:16
*** stupidnic has joined #openstack-operators12:16
*** catintheroof has joined #openstack-operators12:23
*** portante has quit IRC12:38
*** portante has joined #openstack-operators12:40
*** ducttape_ has joined #openstack-operators13:08
*** ducttape_ has quit IRC13:09
*** ducttape_ has joined #openstack-operators13:10
*** bdeschenes has joined #openstack-operators13:20
*** bdeschenes has quit IRC13:27
*** ducttape_ has quit IRC13:38
*** ducttape_ has joined #openstack-operators13:38
*** ducttape_ has quit IRC13:43
*** markvoelker has joined #openstack-operators13:44
*** udesale has joined #openstack-operators13:56
*** rarcea has quit IRC14:04
*** mriedem has joined #openstack-operators14:07
*** racedo has joined #openstack-operators14:08
*** ducttape_ has joined #openstack-operators14:20
*** uxdanielle has joined #openstack-operators14:26
*** ducttape_ has quit IRC14:31
*** ducttape_ has joined #openstack-operators14:32
*** ducttape_ has quit IRC14:36
*** ducttape_ has joined #openstack-operators14:54
*** rmart04 has joined #openstack-operators14:58
*** twiggy has quit IRC15:00
*** ckonstanski has quit IRC15:33
*** ducttape_ has quit IRC15:35
*** udesale has quit IRC15:35
openstackgerritMerged openstack/osops-tools-contrib: Add script to balance load of Neutron DHCP agents  https://review.openstack.org/39796015:39
*** racedo has quit IRC15:42
*** racedo has joined #openstack-operators15:42
*** armax has joined #openstack-operators15:49
*** uxdanielle has quit IRC15:52
*** MrDan has quit IRC15:52
*** uxdanielle has joined #openstack-operators16:01
*** ducttape_ has joined #openstack-operators16:02
*** kstev has joined #openstack-operators16:14
*** armax has quit IRC16:16
*** kstev has quit IRC16:18
*** kstev has joined #openstack-operators16:19
*** MrDan has joined #openstack-operators16:21
*** ducttape_ has quit IRC16:22
*** ducttape_ has joined #openstack-operators16:23
*** armax has joined #openstack-operators16:23
*** erhudy has joined #openstack-operators16:24
erhudyanyone ever seen nova-compute run itself out of FDs by opening a huge number of connections to the Ceph monitors? this has started cropping up recently on one particular cluster and is quite perplexing16:25
erhudyit seems to go off the rails when terminating instances occasionally16:26
*** ckonstanski has joined #openstack-operators16:30
*** kstev has quit IRC16:36
*** kstev has joined #openstack-operators16:39
*** rmart04 has quit IRC16:39
*** MrDan has quit IRC16:40
*** slaweq_ has joined #openstack-operators16:48
*** twiggy has joined #openstack-operators16:50
*** paramite has quit IRC16:51
logan-erhudy: I recall having some FD issues on hypervisors a long time ago when the rbd client admin socket was enabled.16:53
logan-struggling to find the details... i'll do some digging16:54
erhudywhen i lsof'd the nova-compute master process it had many hundreds of connections open over the network to the mons16:55
erhudystracing nova-compute and all its forks revealed that it had run itself out of open files16:55
erhudynova-compute doesn't log anything of interest, though we don't run with debug logging on by default because it would probably choke ELK16:55
*** dbecker has quit IRC17:04
*** Rockyg has joined #openstack-operators17:04
zioprotoerhudy: but is the ceph cluster working normally ? from the host where nova-compute runs can you do ceph -s normally ?17:05
*** Rockyg has quit IRC17:10
*** zioproto has quit IRC17:16
*** matrohon has quit IRC17:23
*** rmart04 has joined #openstack-operators17:24
erhudyyes, ceph is fine17:29
erhudyrestarting the failed nova-compute reliably unsticks it, and VMs running on the node are not affected17:29
*** fragatina has joined #openstack-operators17:41
*** Apoorva has joined #openstack-operators17:41
*** simon-AS5591 has joined #openstack-operators17:43
*** simon-AS559 has quit IRC17:46
*** derekh has quit IRC17:47
*** david-lyle has quit IRC17:48
*** david-lyle has joined #openstack-operators17:48
*** simon-AS5591 has quit IRC17:49
*** uxdanielle has quit IRC17:52
*** cdelatte has joined #openstack-operators18:00
*** fragatin_ has joined #openstack-operators18:04
*** makowals_ has joined #openstack-operators18:05
*** makowals_ has quit IRC18:07
*** makowals_ has joined #openstack-operators18:08
*** fragatina has quit IRC18:08
*** catinthe_ has joined #openstack-operators18:13
*** harlowja has joined #openstack-operators18:14
*** catintheroof has quit IRC18:15
*** catintheroof has joined #openstack-operators18:15
*** catinthe_ has quit IRC18:18
*** dbecker has joined #openstack-operators18:20
*** uxdanielle has joined #openstack-operators18:27
*** electrofelix has quit IRC18:31
*** dbecker has quit IRC18:32
*** simon-AS559 has joined #openstack-operators18:41
*** rmart04_ has joined #openstack-operators18:48
*** rmart04 has quit IRC18:49
*** rmart04_ is now known as rmart0418:49
*** chlong has joined #openstack-operators18:53
*** MrDan has joined #openstack-operators19:10
*** jamesdenton has joined #openstack-operators19:19
*** haplo37_ has quit IRC19:20
*** matrohon has joined #openstack-operators19:26
*** haplo37_ has joined #openstack-operators19:29
*** fragatin_ has quit IRC19:41
*** fragatina has joined #openstack-operators19:41
*** pcaruana has quit IRC19:51
*** uxdanielle has quit IRC19:52
*** rmart04 has left #openstack-operators20:02
stupidnicI am wondering if somebody can assist me here. I have a CentOS 7 instance that the customer performed a hard reboot on. The instance has come up but is stuck in maintenance mode (asking for the root password). Problem is, I can't access it via the novnc console. Is there some other way of interacting with the instance?20:08
mrhillsmanstupidnic in such cases what i have done personally may be a bit far-fetched to you20:20
mrhillsmando you know why the instance is in maintenance mode?20:20
stupidnicI suspect the customer has butchered something in the init process20:21
stupidnicsince the error isn't fsck related in nature20:21
stupidnicbut rather referencing something in the systemd journal20:21
mrhillsmani personally use libguest tools20:21
mrhillsmanlike guestfish20:21
stupidnicman... I was hoping to avoid that.20:21
stupidnicI know where you are going with that.20:22
mrhillsmanonly other thing i have done20:22
mrhillsmanis use qemu-nbd to mount it20:22
mrhillsmanand work on it from there20:22
mrhillsmanessentially that is doing the same thing in my opinion20:23
mrhillsmanguestfish/libguest at least allows you to isolate things20:23
stupidnicYeah. From looking at things... It looks like serial0 is only setup as a file device in libvirt20:23
mrhillsmanyou can definitely work through a few steps and probably get it working20:24
mrhillsmani am not overly familiar with centos' getting serial console20:24
stupidnicmrhillsman: customer is antsy... they are the ones that broke it... but at this point they are just going to trash the instance I guess20:25
mrhillsmanah ok20:25
*** slaweq_ has quit IRC20:25
stupidnicI appreciate the bouncing of ideas though20:25
mrhillsmanwell heck, if it is cloudy20:25
mrhillsmantrash it20:25
stupidnicYeah thankfully it is. Cattle versus pets.20:25
mrhillsmanif it ain't a pet, don't treat it like one :)20:25
stupidnicI did admonish them with "This is why we don't test in production, kids."20:26
mrhillsmannice pat on the hand to not do anymore and go on with ya day20:26
*** cheetah has quit IRC20:35
*** matrohon has quit IRC20:47
*** rcernin has joined #openstack-operators20:58
*** hughsaunders has quit IRC21:03
*** makowals_ has quit IRC21:05
*** hughsaunders has joined #openstack-operators21:06
*** uxdanielle has joined #openstack-operators21:12
*** volenbovsky has joined #openstack-operators21:13
*** twiggy has quit IRC21:23
*** cheetah has joined #openstack-operators21:28
*** simon-AS559 has quit IRC21:30
*** catinthe_ has joined #openstack-operators21:32
*** cdelatte has quit IRC21:34
*** catintheroof has quit IRC21:36
*** catintheroof has joined #openstack-operators21:51
*** mihalis68 has quit IRC21:51
*** catinthe_ has quit IRC21:51
*** uxdanielle has quit IRC21:52
*** slaweq_ has joined #openstack-operators22:25
*** jamesdenton has quit IRC22:26
*** volenbovsky has quit IRC22:40
*** mriedem has quit IRC22:41
*** MrDan has quit IRC22:42
*** catintheroof has quit IRC22:54
*** chlong has quit IRC22:59
*** ducttape_ has quit IRC23:16
*** chlong has joined #openstack-operators23:17
*** racedo has quit IRC23:23
*** dfflanders has joined #openstack-operators23:25
*** chlong has quit IRC23:27
*** khamtamtun has joined #openstack-operators23:39
*** khamtamtun has quit IRC23:42
*** cdelatte has joined #openstack-operators23:49
*** david-lyle_ has joined #openstack-operators23:50
*** erhudy has quit IRC23:51
*** david-lyle has quit IRC23:51
*** racedo has joined #openstack-operators23:51
*** rcernin has quit IRC23:52
mnaserstupidnic / mrhillsman: why not just nova rescue <uuid>23:56
mnaseryou'll be able to ssh into it, mount, make changes, unrescue23:57
mnaserwithout the access to vnc console23:57

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!