| noonedeadpunk | spatel: there's no cloud-init in coreos35, so it will never work. User-data must be provided manually there in the format coreos understands | 08:33 |
|---|---|---|
| admin1 | i had success with this image: fedora-coreos-33.20210426.3.0-openstack.x86_64.qcow2 | 09:56 |
| noonedeadpunk | coreos replaced cloud-init with ignition | 10:06 |
| noonedeadpunk | so you would need to write ignition file https://docs.fedoraproject.org/en-US/fedora-coreos/producing-ign/#_writing_the_butane_config | 10:07 |
| noonedeadpunk | and then pass it with --user-data flag during instance creation | 10:07 |
| noonedeadpunk | magnum does support that atm just in case | 10:07 |
| opendevreview | Marios Andreou proposed openstack/openstack-ansible-os_tempest stable/wallaby: Add centos-9 tripleo standalone job for wallaby zuul layout https://review.opendev.org/c/openstack/openstack-ansible-os_tempest/+/831196 | 10:49 |
| opendevreview | Marios Andreou proposed openstack/ansible-role-python_venv_build stable/wallaby: Add centos-9 tripleo standalone job for wallaby zuul layout https://review.opendev.org/c/openstack/ansible-role-python_venv_build/+/831198 | 11:00 |
| *** sshnaidm|off is now known as sshnaidm | 11:38 | |
| mgariepy | good morning eveyone | 13:35 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible stable/xena: Bump SHAs for Xena https://review.opendev.org/c/openstack/openstack-ansible/+/830398 | 13:39 |
| noonedeadpunk | o/ | 13:39 |
| jrosser | hello | 13:59 |
| opendevreview | James Denton proposed openstack/openstack-ansible-os_neutron stable/xena: Change os_region to region_name https://review.opendev.org/c/openstack/openstack-ansible-os_neutron/+/831181 | 14:26 |
| mgariepy | anyone seen a volume that get to state: reserved once the vms to which it's attached is deleted? | 14:58 |
| mgariepy | i'm on latest U (almost, only missing the commit related to : https://bugs.launchpad.net/nova/+bug/1945646 | 15:01 |
| mgariepy | about the same thing as this: https://github.com/zonca/jupyterhub-deploy-kubernetes-jetstream/issues/40 | 15:06 |
| mgariepy | hmm dangling entry in cinder DB. | 15:33 |
| opendevreview | Jonathan Rosser proposed openstack/ansible-role-pki master: Add molecule testing https://review.opendev.org/c/openstack/ansible-role-pki/+/831236 | 17:38 |
| jrosser | noonedeadpunk: ^ theres a start on some role testing | 17:39 |
| jrosser | i have done nothing yet to make that work as a zuul job but (with properly configured) vm it passes some validation tests | 17:39 |
| opendevreview | Jonathan Rosser proposed openstack/ansible-role-pki master: Add molecule testing https://review.opendev.org/c/openstack/ansible-role-pki/+/831236 | 17:42 |
| jamesdenton | was the openstack_ prefix dropped for vars in the pki-related playbooks? | 18:16 |
| jrosser | jamesdenton: the role is hopefully used outside of openstack, so all the vars in the pki role are pki_<foobar> | 18:18 |
| jrosser | then in openstack-ansible we connect those up to a whole bunch of openstack_pki_<foobar> | 18:18 |
| jrosser | like this https://github.com/openstack/openstack-ansible/blob/master/playbooks/certificate-authority.yml#L25-L28 | 18:19 |
| jamesdenton | i see, thank you | 18:20 |
| jamesdenton | this page references a 'openstack_pki_regen_ca' var, but i don't see it being references anywhere. there are others like it. maybe i'm dense. | 18:22 |
| jamesdenton | https://docs.openstack.org/openstack-ansible/latest/user/security/index.html | 18:22 |
| jrosser | huh thats a bug, nice spot | 18:24 |
| jrosser | https://codesearch.opendev.org/?q=openstack_pki_regen_ca&i=nope&literal=nope&files=&excludeFiles=&repos= | 18:24 |
| opendevreview | Jonathan Rosser proposed openstack/openstack-ansible master: Connect openstack_pki_regen_ca variable to pki role https://review.opendev.org/c/openstack/openstack-ansible/+/831242 | 18:26 |
| jamesdenton | i'll pass it along, thank you | 18:27 |
| jamesdenton | will that actually create the pki dir in openstack_deploy? | 18:28 |
| jamesdenton | rather, does the role create the pki dir | 18:30 |
| jrosser | yes, it makes /etc/openstack_deploy/pki/<stuff> by default | 18:32 |
| jrosser | it's primarily about internal SSL rather than the public endpoint | 18:32 |
| jamesdenton | ok, sure | 18:35 |
| jamesdenton | thanks for the quick turnaround | 18:35 |
| spatel | I have question related Magnum deployment. | 20:08 |
| spatel | My openstack running on rfc1918 address space, including horizon all endpoint. Now i am trying to deploy k8s cluster and its getting fail | 20:09 |
| spatel | after debug found kube-master container trying to talk to Keystone and its failing | 20:10 |
| spatel | k8s create own private network and that network not able to talk to keystone because they are totally isolated | 20:10 |
| spatel | How does people running k8s in this kind of scenario ? | 20:11 |
| spatel | jrosser Do you have any idea about that? | 21:37 |
| jrosser | spatel: you need to have access from inside the magnum vm to the keystone external endpont | 21:54 |
| spatel | Hmm! in short to fix my issue i have to move my keystone to public IP. right? | 21:54 |
| jrosser | it doesnt matter rfc1918 or not, you need and IP route from whatever network the VM are on to get to the APIs | 21:54 |
| jrosser | if it's an OSA deploy then you have internal and external VIP? | 21:55 |
| spatel | k8s creates private network itself so not sure how do i make that routable | 21:55 |
| jrosser | right, but theres a neutron router or something? | 21:56 |
| spatel | neutron router need to hook up with public network or routable network | 21:56 |
| spatel | let me try more debug and see.. | 21:58 |
| jrosser | you have external_network_id in the cluster template?\ | 21:58 |
| spatel | Yes, that where i use public subnet | 21:59 |
| spatel | I have private1 and public1 two network | 21:59 |
| jrosser | what happens is a bunch of software gets deployed into the magnum vm | 21:59 |
| spatel | when i create k8s i use external_network_id=public1 | 21:59 |
| jrosser | and the only way that heat knows that the deployment is complete is if the heat agent in the vm contacts the API endpoint and makes a callback to say it is done | 22:00 |
| spatel | Yes you are correct, heat-container-agent process talks to keystone | 22:00 |
| jrosser | so several things have to line up | 22:00 |
| jrosser | the details of the API endpoint have to be correctly passed into the vm from magum through cloud-init | 22:01 |
| jrosser | then the callback has succeed contacting the endpoint / validating the certificate blah blah | 22:01 |
| spatel | I think easy solution is to move endpoint to public IP otherwise i have to make some routing adjustment | 22:02 |
| jrosser | that is probably the simplest, then it will be just via the default route of the neutron router | 22:02 |
| jrosser | i'm not sure to what extent the service catalog is involved, this may be quite messy inside heat/magnum | 22:03 |
| spatel | Yes, if VIP is setup on public then anyone can talk | 22:03 |
| jrosser | so i would check that the catalog is updated for the new public vip | 22:03 |
| spatel | Yes i can do that | 22:04 |
| spatel | One more question, in centos7 i am getting error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:618)\nYou are using pip version 9.0.1 | 22:05 |
| spatel | Its pip SSL related issue | 22:05 |
| spatel | You told me how to fix it but i forgot :( | 22:05 |
| spatel | i vaguely remember that you told me edit /etc/pip.conf and change cert path or something | 22:06 |
| jrosser | errm | 22:06 |
| spatel | https://paste.opendev.org/show/bTOcYTkSVXX12E2QiK6u/ | 22:06 |
| jrosser | is this becasue of letsencrypt new root CA? | 22:06 |
| spatel | is this correct? | 22:06 |
| spatel | This is stein release of openstack very older version | 22:07 |
| jrosser | where do you get that error? | 22:07 |
| spatel | Its running on CentOS7 | 22:07 |
| spatel | I am adding more compute hw and at this point i got that error - TASK [python_venv_build : Upgrade pip/setuptools/wheel to the versions we want] | 22:08 |
| spatel | https://paste.opendev.org/show/bR2iDaLzAyXSbCThscDI/ | 22:09 |
| spatel | I did yum install ca-certificates to update CA | 22:10 |
| jrosser | ok, in all hosts/containers? | 22:10 |
| spatel | pip still doesn't like it | 22:10 |
| jrosser | ooooh right yes | 22:10 |
| jrosser | pip/python use their own CA store independant of the system one | 22:10 |
| spatel | how do i update that one? | 22:11 |
| jrosser | you pretty much cant | 22:12 |
| spatel | i didn't run ca-certificates on all LXC container but just run on new compute nodes | 22:12 |
| jrosser | just a moment | 22:12 |
| spatel | some folks saying you should do this create /etc/pip.conf file with this - https://paste.opendev.org/show/bTOcYTkSVXX12E2QiK6u/ | 22:12 |
| spatel | https://stackoverflow.com/questions/25981703/pip-install-fails-with-connection-error-ssl-certificate-verify-failed-certi?page=1&tab=scoredesc#tab-top | 22:13 |
| jrosser | it's not clear which url you are failing with though | 22:13 |
| jrosser | i know why it is | 22:13 |
| jrosser | your paste doesnt show which url it failes with https://paste.opendev.org/show/bR2iDaLzAyXSbCThscDI/ | 22:14 |
| jrosser | but this is not too difficult | 22:15 |
| jrosser | you need updated ca-certificates everwhere | 22:15 |
| jrosser | then you can tell python that it needs to *use* those certificates by setting REQUESTS_CA_BUNDLE=/etc/pki/tls/certs/ca-bundle.crt | 22:16 |
| jrosser | that should go in /etc/environment or somewhere like that | 22:16 |
| spatel | hmm | 22:16 |
| jrosser | python requests library does not use the system ca store | 22:17 |
| spatel | currently in pip.conf i have only this line - cert = /etc/pki/tls/certs/ca-bundle.crt | 22:17 |
| spatel | running playbook with -vvvv and see if i can find URL | 22:19 |
| jrosser | it will be in the contstraints file | 22:20 |
| spatel | let me check /var/log/python_venv_build.log | 22:20 |
| jrosser | look in /openstack/venvs/nova-19.0.0.0rc3.dev6/constraints.txt | 22:20 |
| spatel | just movement | 22:21 |
| jrosser | the you can activate the nova venv and try to use the requests library manually | 22:21 |
| jrosser | really trivial to try, see this https://docs.python-requests.org/en/master/user/quickstart/ | 22:21 |
| spatel | https://paste.opendev.org/show/br457WKEozTx1T7LDi3h/ | 22:23 |
| spatel | I did activate venv and try those test which work | 22:25 |
| spatel | also i did install some random package using pip install foo and that works too | 22:26 |
| jrosser | if that comes from pypi using a CA which is still trusted then that will work | 22:30 |
| jrosser | but opendev.org uses an LE cert which will no longer be trusted | 22:30 |
| jrosser | i have to go now but REQUESTS_CA_BUNDLE is what you need | 22:30 |
| spatel | I did this which pass that task - https://paste.opendev.org/show/bvOx5TiwuggyE1QYZ2Ac/ | 22:32 |
| spatel | now getting different error related nova parsing issue or something which i am trying to debug | 22:33 |
| spatel | That was it.. now everything works after adding https://paste.opendev.org/show/bvOx5TiwuggyE1QYZ2Ac/ | 22:46 |
| spatel | jrosser damn it issue is, we have to run this command after ca-certificates update - update-ca-trust | 23:01 |
| jrosser | really? | 23:02 |
| jrosser | sounds like a failure of the rpm package to do that automatically imho | 23:02 |
| jrosser | unless thats special RH behaviour | 23:03 |
| *** mac189_ is now known as mac189 | 23:39 | |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!