| *** luksky has quit IRC | 00:03 | |
| *** tosky has quit IRC | 00:10 | |
| *** waxfire7 has joined #openstack-ansible | 01:13 | |
| *** waxfire has quit IRC | 01:13 | |
| *** waxfire7 is now known as waxfire | 01:13 | |
| *** spatel has joined #openstack-ansible | 01:16 | |
| *** LowKey has quit IRC | 01:25 | |
| *** LowKey has joined #openstack-ansible | 02:20 | |
| *** LowKey has quit IRC | 02:25 | |
| *** spatel has quit IRC | 03:15 | |
| *** spatel has joined #openstack-ansible | 03:17 | |
| *** waxfire4 has joined #openstack-ansible | 03:41 | |
| *** waxfire has quit IRC | 03:41 | |
| *** waxfire4 is now known as waxfire | 03:41 | |
| *** evrardjp has quit IRC | 05:38 | |
| *** evrardjp has joined #openstack-ansible | 05:39 | |
| *** spatel has quit IRC | 06:18 | |
| *** gyee has quit IRC | 06:31 | |
| *** lemko has quit IRC | 07:30 | |
| *** lemko7 has joined #openstack-ansible | 07:30 | |
| *** lemko7 is now known as lemko | 07:30 | |
| MrClayPole | admin0: A 169.254.x.x address is assigned to a DHCP client when it can't get a DHCP address. Could it be the script is running to early in the boot process? | 07:32 |
|---|---|---|
| *** waxfire has quit IRC | 07:36 | |
| *** waxfire has joined #openstack-ansible | 07:36 | |
| *** dirtwash has joined #openstack-ansible | 09:00 | |
| dirtwash | hi, im running openstack victoria with ceph rbd backend. snapshots are failing with rbd incomplete write errors, heres glance log, maybe someone has any idea or hint? http://paste.openstack.org/show/802855/ | 09:00 |
| *** luksky has joined #openstack-ansible | 09:22 | |
| *** tosky has joined #openstack-ansible | 10:13 | |
| openstackgerrit | Merged openstack/openstack-ansible master: Stop installing client packages for ansible venv https://review.opendev.org/c/openstack/openstack-ansible/+/774472 | 10:20 |
| *** pcaruana has quit IRC | 10:27 | |
| *** waxfire3 has joined #openstack-ansible | 10:33 | |
| *** waxfire has quit IRC | 10:35 | |
| *** waxfire3 is now known as waxfire | 10:35 | |
| *** jbadiapa has quit IRC | 10:45 | |
| admin0 | MrClayPole, the script is passed via cloud init | 10:59 |
| MrClayPole | Ah, so its more Likely the script is being proulated with the wrong value? | 11:01 |
| jrosser | 169.254.169.254 is the correct address for the metadata service | 11:02 |
| jrosser | it's a link-local address used before the real ip is known | 11:03 |
| CeeMac | admin0: any issues or delays with the vif port binding? | 11:08 |
| *** noonedeadpunk has quit IRC | 11:16 | |
| *** noonedeadpunk has joined #openstack-ansible | 11:17 | |
| admin0 | this is what I get in the cloud-init logs .. https://gist.github.com/a1git/c7e8b5de5b4cfb28e203d01a3d164b67 .. but as soon as it boots and i login and do the curl, its fine .. i can curl the metadata service | 11:20 |
| CeeMac | Anything in nova logs? If there was a delay plugging the vif it could explain this maybe? | 11:23 |
| admin0 | actually , just booted up a new instance and i cannot curl to that metadata service even after it booted up | 11:24 |
| admin0 | so there were almost 20 instances like this where i have the logs saying cannot connect and then it was able to connect | 11:24 |
| admin0 | tried a new one today, and this one cannot connect to the metadata service even after its booted up | 11:25 |
| CeeMac | All on same host or? | 11:25 |
| admin0 | when i do ip route, i see 2 entries for metadata .. one for .1 and one for .22( could be the metadata ip) | 11:25 |
| CeeMac | I presume the metadata service is up and alive? | 11:26 |
| admin0 | via admin , i can see that all neutron agents and services are alive and working | 11:26 |
| CeeMac | If you can check the events for one of the new instances to get the req id check nova logs for that req id and see if there are any vif port bind errors | 11:27 |
| admin0 | why would there be 2 entries for the metadata service | 11:27 |
| admin0 | one to 192.168.0.1 and 192.168.0.22 | 11:28 |
| admin0 | i am checking what .22 is | 11:28 |
| *** LowKey has joined #openstack-ansible | 11:28 | |
| admin0 | so .22 is network dhcp | 11:29 |
| CeeMac | 1 would be the router I'd guess? | 11:30 |
| admin0 | yeah | 11:30 |
| admin0 | do you guys limit the number of dhcp per network ? | 11:31 |
| CeeMac | Ip range? | 11:31 |
| admin0 | neutron_conf_overrides: dhcp_agents_per_network | 11:33 |
| CeeMac | Oh, no I dont do that | 11:34 |
| admin0 | is metadata service provided by one of the dhcp agents ? | 11:34 |
| CeeMac | I thought there were specific metadata agents | 11:35 |
| CeeMac | Still wortg checking nova logs to see what's occurring during the deployment phase | 11:35 |
| admin0 | deploying a new vm to check this out | 11:39 |
| admin0 | using anfi-affnity so that i can spawn one in every host | 11:39 |
| admin0 | which i did before and all worked fine | 11:39 |
| admin0 | so it seems to be not affecting every node | 11:40 |
| admin0 | maybe its a specific compute => network node combo | 11:40 |
| CeeMac | Could be one host having issues with its agents? | 11:40 |
| admin0 | is there a way to validate if a metadata service is working correctly ? | 11:44 |
| jrosser | you go in the active router network namespace and you will see an instance of haproxy running bound to 169.254.169.254 | 11:45 |
| admin0 | ok .. so what could cause a new route in the routing table pointing 169.254.169.254 to a dhcp instance -- and also 1 exist for router | 11:51 |
| admin0 | i meant 2 routing entries | 11:52 |
| *** LowKey has quit IRC | 12:00 | |
| *** LowKey has joined #openstack-ansible | 12:00 | |
| CeeMac | What do you see on other tenant networks? | 12:17 |
| admin0 | i am first going to limit the number of network nodes | 12:23 |
| admin0 | so that i can isolate this | 12:23 |
| admin0 | then delete all workload and then redo it again | 12:23 |
| *** waxfire9 has joined #openstack-ansible | 12:52 | |
| *** waxfire has quit IRC | 12:53 | |
| *** waxfire9 is now known as waxfire | 12:53 | |
| *** waxfire has quit IRC | 13:35 | |
| *** waxfire has joined #openstack-ansible | 13:35 | |
| openstackgerrit | Merged openstack/openstack-ansible master: Use zuul provided requirements repo during bootstrap-ansible.sh https://review.opendev.org/c/openstack/openstack-ansible/+/774444 | 13:45 |
| *** SiavashSardari has joined #openstack-ansible | 13:58 | |
| *** pto has joined #openstack-ansible | 14:25 | |
| admin0 | i had some hosts configured as network hosts .. now I don't want them to be network hosts . how shoud I remove them ? i delete the agents .. btu they come back up | 14:26 |
| CeeMac | I think the agents have to be stopped / disabled before you delete them | 14:27 |
| *** LowKey has quit IRC | 14:28 | |
| CeeMac | You'll have to update your config file and prune the inventory I guess too | 14:29 |
| *** luksky has quit IRC | 14:29 | |
| *** pto has quit IRC | 14:29 | |
| CeeMac | Not sure if you'd need to manually remove the netns | 14:29 |
| CeeMac | Might be better purging the host from inventory and redepolying it? | 14:30 |
| *** waxfire has quit IRC | 14:31 | |
| *** waxfire has joined #openstack-ansible | 14:31 | |
| admin0 | CeeMac, how do I purge those hosts ? | 14:40 |
| admin0 | just delete them ? | 14:40 |
| admin0 | they are not controllers | 14:40 |
| admin0 | so no chance of ips being mismatch | 14:40 |
| admin0 | yeah .. seems to have workd for the infra | 14:44 |
| admin0 | so i had to rm -rf /openstack/venv/neutron and rerun the playbooks | 14:44 |
| CeeMac | If you were changing/removing roles on a host I would probably -r them from the inventory, clear out the ansible_facts, update the group mappings in o_u_c.yml then RE deploy / pxe the o/s and run the required playbooks to add it back as a fresh node | 14:47 |
| CeeMac | Removing the venv doesn't clean up systemctl units etc so you'd be left with bits and bobs lurking around | 14:48 |
| admin0 | doing a pxe was not an option .. now what i have to do is manually remove the systectl for l3 agent, metadata, and dhcp | 15:00 |
| admin0 | i did the -r from inventory, cleaned up the uc.yml and rm -rf /etc/neutron and /venv before re-running the playbooks | 15:00 |
| SiavashSardari | hey everyone. I have and old lab setup, today I rebooted the controllers and now most of lxc containers won't start. this is the out put of lxc-start --logfile | 15:02 |
| SiavashSardari | http://paste.openstack.org/show/802858/ | 15:02 |
| SiavashSardari | any Idea about this issue? | 15:02 |
| *** tosky has quit IRC | 15:35 | |
| *** luksky has joined #openstack-ansible | 15:54 | |
| admin0 | SiavashSardari, is it an aio ? | 15:57 |
| admin0 | normally if controllers are restarted, lxc containers start up just fine | 15:57 |
| SiavashSardari | not exactly, but controllers are computes too | 15:58 |
| SiavashSardari | kinda like an ha aio | 15:58 |
| admin0 | for aio .. when used with swift/ceph .. i know it does some loopback device mapping that does not go well during reboots and re-running of the playbooks | 16:03 |
| admin0 | so i am not sure | 16:03 |
| admin0 | what i normally do is create virtual machines, virtual ceph and then run my lab ( and never use an aio ) | 16:03 |
| SiavashSardari | didn't know that. Thank you. | 16:16 |
| *** ioni has quit IRC | 16:34 | |
| *** waxfire0 has joined #openstack-ansible | 16:45 | |
| *** waxfire has quit IRC | 16:46 | |
| *** waxfire0 is now known as waxfire | 16:46 | |
| *** d34dh0r53 has quit IRC | 17:09 | |
| *** SiavashSardari has quit IRC | 18:16 | |
| *** waxfire7 has joined #openstack-ansible | 18:18 | |
| *** waxfire has quit IRC | 18:20 | |
| *** waxfire7 is now known as waxfire | 18:20 | |
| *** lemko has quit IRC | 18:48 | |
| *** lemko has joined #openstack-ansible | 18:49 | |
| *** lemko5 has joined #openstack-ansible | 19:42 | |
| *** lemko has quit IRC | 19:42 | |
| *** lemko5 is now known as lemko | 19:42 | |
| *** tosky has joined #openstack-ansible | 20:21 | |
| *** pto has joined #openstack-ansible | 20:26 | |
| *** pto has quit IRC | 20:30 | |
| *** luksky has quit IRC | 20:42 | |
| *** luksky has joined #openstack-ansible | 21:01 | |
| *** ioni has joined #openstack-ansible | 21:45 | |
| *** luksky has quit IRC | 22:54 | |
| *** dave-mccowan has quit IRC | 23:41 | |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!