*** jamesdenton has quit IRC | 00:00 | |
*** jamesdenton has joined #openstack-operators | 00:17 | |
*** manheim has joined #openstack-operators | 00:25 | |
*** jamesdenton has quit IRC | 00:28 | |
*** VW has joined #openstack-operators | 00:28 | |
*** manheim has quit IRC | 00:29 | |
*** gyee has quit IRC | 00:29 | |
*** rebase_ has quit IRC | 00:53 | |
*** rebase_ has joined #openstack-operators | 00:56 | |
*** bsilverman_13 has quit IRC | 01:03 | |
*** bsilverman_ has joined #openstack-operators | 01:03 | |
*** rebase_ has quit IRC | 01:16 | |
*** rebase_ has joined #openstack-operators | 01:16 | |
*** Apoorva__ has quit IRC | 01:21 | |
*** masber has quit IRC | 01:56 | |
*** marst has joined #openstack-operators | 02:01 | |
*** rebase_ has quit IRC | 02:01 | |
*** mriedem has quit IRC | 02:03 | |
*** rebase_ has joined #openstack-operators | 02:04 | |
*** VW has quit IRC | 02:04 | |
*** VW has joined #openstack-operators | 02:05 | |
*** VW has quit IRC | 02:09 | |
*** rebase_ has quit IRC | 02:19 | |
*** yankcrime has quit IRC | 02:40 | |
*** markvoelker has quit IRC | 02:40 | |
*** fragatina has joined #openstack-operators | 02:51 | |
*** masber has joined #openstack-operators | 02:52 | |
*** fragatina has quit IRC | 02:53 | |
*** fragatina has joined #openstack-operators | 02:53 | |
*** fragatin_ has quit IRC | 02:54 | |
*** fragatin_ has joined #openstack-operators | 03:01 | |
*** shash____ has joined #openstack-operators | 03:02 | |
*** Long_yanG has joined #openstack-operators | 03:03 | |
*** fragatina has quit IRC | 03:04 | |
*** fragatin_ has quit IRC | 03:05 | |
*** cemason has joined #openstack-operators | 03:06 | |
*** shasha_t- has joined #openstack-operators | 03:06 | |
*** cemason has quit IRC | 03:10 | |
*** bauruine_ has joined #openstack-operators | 03:10 | |
*** tdasilva- has quit IRC | 03:11 | |
*** blairo has quit IRC | 03:11 | |
*** shasha___ has quit IRC | 03:11 | |
*** LongyanG has quit IRC | 03:11 | |
*** dalees has quit IRC | 03:11 | |
*** bauruine has quit IRC | 03:11 | |
*** mdavidson has quit IRC | 03:11 | |
*** cemason has joined #openstack-operators | 03:11 | |
*** trungnv has quit IRC | 03:18 | |
*** dalees has joined #openstack-operators | 03:18 | |
*** mdavidson has joined #openstack-operators | 03:18 | |
*** trungnv has joined #openstack-operators | 03:18 | |
*** tdasilva has joined #openstack-operators | 03:23 | |
*** masber has quit IRC | 03:24 | |
*** blair has joined #openstack-operators | 03:28 | |
*** markvoelker has joined #openstack-operators | 03:44 | |
*** shash____ has quit IRC | 04:08 | |
*** shasha_t_ has joined #openstack-operators | 04:09 | |
*** arnewiebalck_ has joined #openstack-operators | 04:09 | |
*** masber has joined #openstack-operators | 04:11 | |
*** fragatina has joined #openstack-operators | 04:39 | |
*** fragatina has quit IRC | 04:40 | |
*** fragatina has joined #openstack-operators | 04:40 | |
*** simon-AS559 has joined #openstack-operators | 04:44 | |
*** slaweq has quit IRC | 04:46 | |
*** zenirc369 has joined #openstack-operators | 04:48 | |
*** simon-AS5591 has joined #openstack-operators | 05:02 | |
*** simon-AS559 has quit IRC | 05:02 | |
*** marst has quit IRC | 05:11 | |
*** simon-AS5591 has quit IRC | 05:12 | |
*** simon-AS559 has joined #openstack-operators | 05:18 | |
*** trungnv has quit IRC | 05:38 | |
*** Dinesh_Bhor has quit IRC | 05:41 | |
*** Dinesh_Bhor has joined #openstack-operators | 05:41 | |
*** Dinesh_Bhor has quit IRC | 05:43 | |
*** shasha_t_ has quit IRC | 05:44 | |
*** yprokule has joined #openstack-operators | 05:47 | |
*** Dinesh_Bhor has joined #openstack-operators | 05:48 | |
*** trungnv has joined #openstack-operators | 05:48 | |
*** trungnv has quit IRC | 05:51 | |
*** zenirc369 has quit IRC | 05:56 | |
*** trungnv has joined #openstack-operators | 06:01 | |
*** Oku_OS-away is now known as Oku_OS | 06:28 | |
*** simon-AS5591 has joined #openstack-operators | 06:35 | |
*** simon-AS559 has quit IRC | 06:35 | |
*** simon-AS5591 has quit IRC | 06:43 | |
*** rmart04_ has joined #openstack-operators | 06:44 | |
*** pcaruana has joined #openstack-operators | 06:45 | |
*** slaweq has joined #openstack-operators | 06:47 | |
*** slaweq has quit IRC | 06:52 | |
*** simon-AS559 has joined #openstack-operators | 06:57 | |
*** zenirc369 has joined #openstack-operators | 07:00 | |
*** simon-AS5591 has joined #openstack-operators | 07:05 | |
*** Caterpillar has joined #openstack-operators | 07:07 | |
*** simon-AS559 has quit IRC | 07:08 | |
*** manheim has joined #openstack-operators | 07:19 | |
*** tesseract has joined #openstack-operators | 07:29 | |
*** shasha_t_ has joined #openstack-operators | 07:44 | |
*** aojea has joined #openstack-operators | 07:48 | |
*** shasha_t_ has quit IRC | 07:49 | |
*** rmart04_ has quit IRC | 07:49 | |
*** racedo has joined #openstack-operators | 07:52 | |
*** yprokule_ has joined #openstack-operators | 07:55 | |
*** simon-AS5591 has quit IRC | 07:57 | |
*** yprokule has quit IRC | 07:57 | |
*** yprokule_ is now known as yprokule | 07:57 | |
*** simon-AS559 has joined #openstack-operators | 08:01 | |
*** simon-AS559 has quit IRC | 08:04 | |
*** arnewiebalck_ has quit IRC | 08:05 | |
*** simon-AS559 has joined #openstack-operators | 08:07 | |
*** yprokule_ has joined #openstack-operators | 08:10 | |
*** yprokule has quit IRC | 08:12 | |
*** yprokule_ is now known as yprokule | 08:12 | |
*** rarcea has joined #openstack-operators | 08:14 | |
*** rmart04 has joined #openstack-operators | 08:15 | |
*** rarcea has quit IRC | 08:17 | |
*** paramite has joined #openstack-operators | 08:21 | |
*** derekh has joined #openstack-operators | 08:36 | |
*** simon-AS5591 has joined #openstack-operators | 08:44 | |
*** simon-AS559 has quit IRC | 08:44 | |
*** _nick has joined #openstack-operators | 08:44 | |
*** Tyrantelf has quit IRC | 08:45 | |
*** racedo has quit IRC | 08:45 | |
*** Tyrantelf has joined #openstack-operators | 08:45 | |
*** _nick is now known as yankcrime | 08:45 | |
*** slaweq has joined #openstack-operators | 08:48 | |
*** slaweq has quit IRC | 08:53 | |
*** arnewiebalck_ has joined #openstack-operators | 09:13 | |
*** markvoelker has quit IRC | 09:25 | |
*** rmart04 has quit IRC | 09:29 | |
*** j^2 has quit IRC | 09:40 | |
*** j^2 has joined #openstack-operators | 09:41 | |
*** markvoelker has joined #openstack-operators | 10:26 | |
*** markvoelker has quit IRC | 10:33 | |
*** arnewiebalck_ has quit IRC | 10:42 | |
*** mrhillsman has quit IRC | 10:48 | |
*** mrhillsman has joined #openstack-operators | 10:49 | |
*** slaweq has joined #openstack-operators | 10:49 | |
*** slaweq has quit IRC | 10:53 | |
*** paramite has quit IRC | 11:21 | |
*** catintheroof has joined #openstack-operators | 11:47 | |
*** catintheroof has quit IRC | 11:51 | |
*** simon-AS559 has joined #openstack-operators | 12:17 | |
*** simon-AS5592 has joined #openstack-operators | 12:19 | |
*** simon-AS5591 has quit IRC | 12:19 | |
*** simon-AS559 has quit IRC | 12:21 | |
*** liverpooler has joined #openstack-operators | 12:22 | |
*** slaweq has joined #openstack-operators | 12:50 | |
*** slaweq has quit IRC | 12:54 | |
*** paramite has joined #openstack-operators | 12:59 | |
*** zenirc369 has quit IRC | 13:02 | |
*** dminer has joined #openstack-operators | 13:05 | |
*** jamesdenton has joined #openstack-operators | 13:13 | |
*** bvandenh has joined #openstack-operators | 13:20 | |
*** BoogPowell_ has joined #openstack-operators | 13:22 | |
*** markvoelker has joined #openstack-operators | 13:23 | |
*** mriedem has joined #openstack-operators | 13:24 | |
*** mriedem has left #openstack-operators | 13:25 | |
*** mriedem has joined #openstack-operators | 13:25 | |
*** chlong has joined #openstack-operators | 13:32 | |
BoogPowell_ | Seeking any helpl possible! I have Openstack Newton on Ubuntu 16.04 with 3 controllers, 16 compute, and 16 storage nodes. I'm running linuxbridge in Neutron and everything has been working since last August. Monday I started getting VirtualInterfaceCreateException: Virtual Interface creation failed | 13:34 |
---|---|---|
BoogPowell_ | I tried the vif timeout params and it only masks the issue. | 13:35 |
BoogPowell_ | Has anyone seen this before and know what the true fix is? | 13:35 |
BoogPowell_ | nc shows all ports responding from all servers. All services show Enabled/UP from the command line and via Horizon. | 13:36 |
BoogPowell_ | and curl to the endpoints is successful for all services | 13:36 |
BoogPowell_ | I know this is a major deal, but can someone at least tell me the proper startup sequence for Openstack services? I can't find it anywhere in the doc and I know neutron needs to come up in a certain order, although the systemd scripts don't specify anything outside of keystone and MQ. | 13:44 |
*** arnewiebalck_ has joined #openstack-operators | 13:51 | |
*** chlong has quit IRC | 14:01 | |
*** VW_ has joined #openstack-operators | 14:01 | |
*** chlong has joined #openstack-operators | 14:14 | |
*** arnewiebalck_ has quit IRC | 14:14 | |
BoogPowell_ | Maybe this will make it easier: I ran rabbitmqctl list_queues and the dhcp_agent.neutron* queues are climbing. What needs to be restarted? | 14:19 |
yankcrime | BoogPowell_: sounds like some of your agents haven't started | 14:19 |
BoogPowell_ | They're all running | 14:19 |
BoogPowell_ | and have been restarted | 14:19 |
yankcrime | BoogPowell_: do 'openstack network agent list' and see if anything's marked as dead | 14:19 |
*** VW_ has quit IRC | 14:19 | |
yankcrime | and if there are any down, check out the logs from the agents themselves | 14:19 |
BoogPowell_ | Everything is marked Enabled/Up | 14:19 |
*** arnewiebalck_ has joined #openstack-operators | 14:19 | |
*** VW has joined #openstack-operators | 14:20 | |
*** catintheroof has joined #openstack-operators | 14:20 | |
yankcrime | there must be a reason as to why your dhcp agents aren't handling queued messages - i'd start looking at their logs first of all | 14:20 |
yankcrime | and also neutron-server's logs | 14:20 |
BoogPowell_ | DHCP is probably the effect of another issue, namely this: VirtualInterfaceCreateException: Virtual Interface creation failed | 14:22 |
*** catinthe_ has joined #openstack-operators | 14:23 | |
BoogPowell_ | I tried the vif_plugging_is_fatal and vif_pluggin_timeout settings, but they seem only to mask the real issue - which is what has me baffled. | 14:23 |
yankcrime | that's a pretty common error and can come about for all kinds of reasons | 14:24 |
yankcrime | does the instance actually get scheduled to a compute node? | 14:24 |
*** catintheroof has quit IRC | 14:24 | |
BoogPowell_ | I think so, but will have to check on that. | 14:27 |
BoogPowell_ | It should show in nova-scheduler.log, right? | 14:28 |
BoogPowell_ | Sorry, I'm going on 50+ hours and I'm brain dead. :-) | 14:28 |
*** markvoelker has quit IRC | 14:31 | |
*** manheim has quit IRC | 14:33 | |
*** slaweq has joined #openstack-operators | 14:51 | |
*** tongli has joined #openstack-operators | 14:52 | |
*** markvoelker has joined #openstack-operators | 14:53 | |
yankcrime | if it ends up on a compute node and you're seeing that error then look in the nova compute agent logs on your hypervisor | 14:54 |
yankcrime | there'll almost certainly be an exception thrown | 14:54 |
yankcrime | if it's not making it to a compute node then it's something a bit more fundamental - look in neutron-server's log as you launch an instance as well as nova-api and nova-scheduler's logs | 14:55 |
*** slaweq has quit IRC | 14:55 | |
yankcrime | and yeah, openstack server show - if you've the admin role assigned - should show the hypervisor that instance has been scheduled to | 14:55 |
BoogPowell_ | VirtualInterfaceCreateException: Virtual Interface creation failed | 14:56 |
yankcrime | i assume you don't have anything like elasticsearch / logstash / kibana installed? | 14:57 |
yankcrime | you might need to bump up logging levels to get more information but i'd expect to see an exception being thrown somewhere | 14:57 |
BoogPowell_ | Failed to allocate the network(s), not rescheduling. | 14:57 |
yankcrime | tbh elk or something like it is a pre-requisite for operating openstack, there's too many moving parts and too many log files to hunt through | 14:57 |
yankcrime | really you need to be able to just search of instance id or request id and have a way of viewing all associated log entries and failure levels | 14:58 |
BoogPowell_ | Thanks. | 14:58 |
yankcrime | bump up nova's logging and neutron's logging and then try again | 14:59 |
BoogPowell_ | It'll take an act of congress but I will do that as soon as they let me. Thanks for your assistance! I really appreciate your efforts. | 14:59 |
yankcrime | good luck | 15:00 |
BoogPowell_ | It is NOT getting scheduled to a compute node! OS-EXT-SRV-ATTR:hypervisor_hostname | None | 15:03 |
*** cemason has quit IRC | 15:07 | |
*** shasha_t_ has joined #openstack-operators | 15:07 | |
*** cemason has joined #openstack-operators | 15:07 | |
*** Oku_OS is now known as Oku_OS-away | 15:11 | |
*** shasha_t_ has quit IRC | 15:11 | |
*** zenirc369 has joined #openstack-operators | 15:16 | |
*** arnewiebalck_ has quit IRC | 15:20 | |
*** arnewiebalck_ has joined #openstack-operators | 15:27 | |
*** yprokule has quit IRC | 15:29 | |
*** marst has joined #openstack-operators | 15:31 | |
*** simon-AS559 has joined #openstack-operators | 15:34 | |
*** simon-AS559 has quit IRC | 15:35 | |
*** chyka has joined #openstack-operators | 15:35 | |
*** chlong has quit IRC | 15:36 | |
*** simon-AS5592 has quit IRC | 15:38 | |
mriedem | does anyone know if paypal or ebay are using cellsv1? | 15:39 |
*** VW has quit IRC | 15:43 | |
mriedem | or if paypal or ebay ops actually hang out here? | 15:46 |
*** gyee has joined #openstack-operators | 15:50 | |
klindgren | We hired a manager from paypal. They used cells v1 | 15:53 |
klindgren | mainly for ironic/non-ironic split. But they had a few clouds that were nova-cells for compute due to size. | 15:54 |
*** Apoorva has joined #openstack-operators | 15:54 | |
*** dtrainor has quit IRC | 15:55 | |
*** slaweq has joined #openstack-operators | 16:00 | |
*** simon-AS559 has joined #openstack-operators | 16:02 | |
*** simon-AS5591 has joined #openstack-operators | 16:03 | |
*** vinsh has quit IRC | 16:05 | |
*** simon-AS559 has quit IRC | 16:07 | |
*** mrhillsman has quit IRC | 16:35 | |
*** j^2 has quit IRC | 16:35 | |
*** pcaruana has quit IRC | 16:40 | |
*** j^2 has joined #openstack-operators | 16:44 | |
*** shasha_t_ has joined #openstack-operators | 16:44 | |
*** mrhillsman has joined #openstack-operators | 16:46 | |
klindgren | Can someone explain to me on how to use the openstack client with an auth token. Because the documentation is pure shit and their are no examples. | 16:48 |
klindgren | $ openstack --os-token <token> --os-auth-type token --os-endpoint <nova-endpoint> --os-auth-url <auth url> server list | 16:52 |
klindgren | Missing value auth-url required for auth plugin token | 16:52 |
*** simon-AS5591 has quit IRC | 16:53 | |
*** fragatina has quit IRC | 16:54 | |
*** derekh has quit IRC | 17:01 | |
*** hieulq_ has joined #openstack-operators | 17:05 | |
*** paramite has quit IRC | 17:06 | |
*** aojea has quit IRC | 17:10 | |
*** aojea has joined #openstack-operators | 17:11 | |
*** fragatina has joined #openstack-operators | 17:13 | |
*** aojea has quit IRC | 17:15 | |
*** cemason has quit IRC | 17:19 | |
BoogPowell_ | @klindgren - do you have an adminrc file...a file containging your OS* variables? | 17:22 |
klindgren | I am not attempting to do that. I got an auth token, and with just using CLI flags trying to use that auto token in openstack client. | 17:23 |
klindgren | bascially I am scripting a bunch of stuff. So instead of doing auth a bunch of times. I jsut want to auth once and then re-use the token in the later commands | 17:26 |
*** tesseract has quit IRC | 17:30 | |
*** hieulq_ has quit IRC | 17:37 | |
*** hieulq_ has joined #openstack-operators | 17:39 | |
*** basilAB has quit IRC | 17:45 | |
*** gyee has quit IRC | 17:45 | |
*** vaishali has quit IRC | 17:46 | |
*** fragatina has quit IRC | 17:54 | |
*** fragatina has joined #openstack-operators | 17:54 | |
*** Apoorva has quit IRC | 17:56 | |
klindgren | so I partially figured it out. --os-token --os-url are needed. But openstack client blows up on a server show. Because it attempts to use the os-url for nova in the glance client to get the information about the image thats used | 17:57 |
klindgren | and blows up on parsing the v2/tenant_id on the nova api endpoint | 17:57 |
*** rebase has joined #openstack-operators | 18:02 | |
*** VW has joined #openstack-operators | 18:04 | |
*** slaweq has quit IRC | 18:17 | |
*** slaweq has joined #openstack-operators | 18:18 | |
*** harlowja has quit IRC | 18:19 | |
*** efried has joined #openstack-operators | 18:20 | |
mriedem | klindgren: mdorman: re: the service catalog thread in nova, are you guys using effectively 'internal' or 'admin' endpoint interfaces for inter-service communication, i.e. nova to cinder/glance/neutron? | 18:21 |
mriedem | the spec is proposing that if an interface is not configured specifically, nova would iterate over the 3 looking for one that works | 18:22 |
mriedem | wondering if we start with admin or internal | 18:22 |
mriedem | https://review.openstack.org/#/c/461481/ | 18:22 |
klindgren | currently for us - they are all the same. Planning on redoing it so public is ots own setup of api's | 18:22 |
mriedem | ksa defaults to 'public' if you don't specify an interface | 18:22 |
mriedem | ok | 18:23 |
klindgren | I would prefer to default to internal -> admin -> public | 18:23 |
mriedem | ok that's what mordred had proposed too | 18:23 |
mriedem | efried: ^ there is your answer | 18:23 |
efried | beautiful, thanks y'all. | 18:23 |
*** slaweq has quit IRC | 18:23 | |
*** vaishali has joined #openstack-operators | 18:32 | |
*** basilAB has joined #openstack-operators | 18:33 | |
*** simon-AS559 has joined #openstack-operators | 18:39 | |
*** zenirc369 has quit IRC | 18:46 | |
*** Apoorva has joined #openstack-operators | 18:50 | |
*** manheim_ has joined #openstack-operators | 18:54 | |
*** simon-AS559 has quit IRC | 18:55 | |
*** manheim_ has quit IRC | 19:10 | |
*** rebase has quit IRC | 19:25 | |
*** rebase has joined #openstack-operators | 19:26 | |
*** aojea has joined #openstack-operators | 19:27 | |
*** hieulq_ has quit IRC | 19:28 | |
*** VW has quit IRC | 19:36 | |
*** shasha_t_ has quit IRC | 19:39 | |
*** hieulq_ has joined #openstack-operators | 19:41 | |
*** harlowja has joined #openstack-operators | 19:48 | |
*** aojea has quit IRC | 19:50 | |
*** aojea has joined #openstack-operators | 19:50 | |
*** aojea_ has joined #openstack-operators | 19:52 | |
*** aojea has quit IRC | 19:53 | |
*** hieulq_ has quit IRC | 19:55 | |
*** Apoorva_ has joined #openstack-operators | 20:05 | |
*** simon-AS559 has joined #openstack-operators | 20:08 | |
*** Apoorva has quit IRC | 20:09 | |
*** liverpooler has quit IRC | 20:09 | |
*** aojea_ has quit IRC | 20:19 | |
*** slaweq has joined #openstack-operators | 20:22 | |
*** simon-AS559 has quit IRC | 20:31 | |
*** VW has joined #openstack-operators | 20:34 | |
*** shasha_t_ has joined #openstack-operators | 20:35 | |
*** shasha___ has joined #openstack-operators | 20:41 | |
*** cnf has quit IRC | 20:41 | |
*** shasha_t_ has quit IRC | 20:41 | |
*** verdurin_ is now known as verdurin | 20:42 | |
*** cnf has joined #openstack-operators | 20:42 | |
*** simon-AS559 has joined #openstack-operators | 21:04 | |
*** Apoorva_ has quit IRC | 21:04 | |
*** Apoorva has joined #openstack-operators | 21:05 | |
klindgren | anybody know what channel supports openstack-client? | 21:08 |
klindgren | its bascially worthless as hell with a token | 21:08 |
*** BoogPowell_ has quit IRC | 21:09 | |
efried | klindgren I think #openstack-sdks | 21:19 |
efried | oh, looks like you found it ;-) | 21:19 |
klindgren | yea asking my question - because this appears to be full of fail | 21:19 |
klindgren | but thanks | 21:20 |
*** marst has quit IRC | 21:23 | |
*** rebase has quit IRC | 21:25 | |
*** rebase has joined #openstack-operators | 21:27 | |
*** marst has joined #openstack-operators | 21:30 | |
*** dminer has quit IRC | 21:36 | |
*** Caterpillar has quit IRC | 21:39 | |
*** manheim has joined #openstack-operators | 21:43 | |
*** simon-AS559 has quit IRC | 21:48 | |
*** jbadiapa has quit IRC | 22:01 | |
*** Apoorva_ has joined #openstack-operators | 22:05 | |
*** jamesdenton has quit IRC | 22:08 | |
*** Apoorva has quit IRC | 22:09 | |
*** jbadiapa has joined #openstack-operators | 22:15 | |
*** slaweq has quit IRC | 22:22 | |
*** manheim has quit IRC | 22:51 | |
*** craigs_ has quit IRC | 22:52 | |
*** Apoorva has joined #openstack-operators | 22:56 | |
*** Apoorva_ has quit IRC | 22:56 | |
*** catinthe_ has quit IRC | 23:08 | |
*** rebase has quit IRC | 23:15 | |
*** chyka has quit IRC | 23:37 | |
*** chyka has joined #openstack-operators | 23:38 | |
*** rebase has joined #openstack-operators | 23:41 | |
*** chyka has quit IRC | 23:43 | |
*** Apoorva has quit IRC | 23:52 | |
*** rebase has quit IRC | 23:54 | |
*** Apoorva has joined #openstack-operators | 23:54 | |
*** rebase has joined #openstack-operators | 23:55 |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!