| *** cshen has joined #openstack-ansible | 00:18 | |
| *** cshen has quit IRC | 00:23 | |
| *** itandops has quit IRC | 00:23 | |
| *** ianychoi__ is now known as ianychoi | 00:44 | |
| *** renich has quit IRC | 00:56 | |
| *** gyee has quit IRC | 01:25 | |
| *** MickyMan77 has joined #openstack-ansible | 01:53 | |
| *** MickyMan77 has quit IRC | 01:58 | |
| *** cshen has joined #openstack-ansible | 02:18 | |
| *** cshen has quit IRC | 02:24 | |
| *** partlycloudy has quit IRC | 02:33 | |
| *** spatel has joined #openstack-ansible | 02:33 | |
| *** partlycloudy has joined #openstack-ansible | 02:36 | |
| *** MickyMan77 has joined #openstack-ansible | 02:56 | |
| *** MickyMan77 has quit IRC | 03:01 | |
| *** dave-mccowan has joined #openstack-ansible | 03:49 | |
| *** spatel has quit IRC | 04:17 | |
| *** evrardjp has quit IRC | 04:33 | |
| *** evrardjp has joined #openstack-ansible | 04:33 | |
| *** MickyMan77 has joined #openstack-ansible | 04:54 | |
| *** MickyMan77 has quit IRC | 05:00 | |
| *** dave-mccowan has quit IRC | 05:00 | |
| *** renich has joined #openstack-ansible | 05:37 | |
| *** gokhani has joined #openstack-ansible | 05:45 | |
| *** idlemind_ has quit IRC | 05:50 | |
| *** shyamb has joined #openstack-ansible | 05:50 | |
| *** idlemind has joined #openstack-ansible | 05:54 | |
| *** MickyMan77 has joined #openstack-ansible | 05:58 | |
| *** renich has quit IRC | 05:59 | |
| *** MickyMan77 has quit IRC | 05:59 | |
| *** MickyMan77 has joined #openstack-ansible | 05:59 | |
| *** idlemind_ has joined #openstack-ansible | 06:08 | |
| *** idlemind has quit IRC | 06:09 | |
| *** renich has joined #openstack-ansible | 06:13 | |
| *** miloa has joined #openstack-ansible | 06:17 | |
| *** idlemind_ has quit IRC | 06:20 | |
| *** idlemind has joined #openstack-ansible | 06:21 | |
| BlackFX | Is there a way to set the containers up to log to the log host using systemd-journal-remote ? | 06:47 |
|---|---|---|
| *** cshen has joined #openstack-ansible | 06:49 | |
| *** kukacz has joined #openstack-ansible | 06:49 | |
| *** shyamb has quit IRC | 06:50 | |
| *** shyamb has joined #openstack-ansible | 07:01 | |
| *** MickyMan77 has quit IRC | 07:03 | |
| *** shyamb has quit IRC | 07:25 | |
| jrosser | BlackFX: everything is there to allow systemd remote journals but you need to be aware of this https://opendev.org/openstack/openstack-ansible/src/branch/master/releasenotes/notes/disable-journald-remote-33d3b43e538786ed.yaml | 07:26 |
| BlackFX | Awesome. thank you! | 07:28 |
| *** andrewbonney has joined #openstack-ansible | 07:35 | |
| *** shyamb has joined #openstack-ansible | 07:39 | |
| *** tosky has joined #openstack-ansible | 07:43 | |
| *** itandops has joined #openstack-ansible | 07:50 | |
| jrosser | BlackFX: something really handy to know about is codesearch.openstack.org | 08:03 |
| jrosser | if you search for the variable that enables remote journals, like this http://codesearch.openstack.org/?q=journald_remote_enabled | 08:04 |
| jrosser | you can click straight through to the git repo and look at the code | 08:04 |
| BlackFX | Oh, I like that - thank you :) | 08:04 |
| *** yolanda has quit IRC | 08:22 | |
| *** yolanda has joined #openstack-ansible | 08:22 | |
| *** mensis has joined #openstack-ansible | 08:32 | |
| mensis | jrosser: hi, thanks for the reply, i'm trying to install constraints like in glance as you mentioned. While there is one repo in glance, there are many repos in monasca. How can i combine those repos of monasca into one constraint and use it with python_venv role? | 08:36 |
| mensis | This is what i've tried, in defaults/main.yml http://paste.openstack.org/show/797915/, and in tasks/main.yml http://paste.openstack.org/show/797916/ | 08:38 |
| mensis | Should i write python_venv_build for every service of monasca? (monasca-api, monasca-thresh etc.) | 08:39 |
| mensis | Thanks in advance for your help | 08:39 |
| *** shyam89 has joined #openstack-ansible | 08:41 | |
| *** shyamb has quit IRC | 08:43 | |
| *** renich has quit IRC | 08:47 | |
| jrosser | mensis: for something with multiple repos, neutron is a good example https://github.com/openstack/openstack-ansible-os_neutron/blob/master/defaults/main.yml#L53-L90 | 08:50 |
| mensis | thanks | 08:51 |
| *** shyam89 has quit IRC | 09:10 | |
| *** kukacz has quit IRC | 09:38 | |
| *** kukacz has joined #openstack-ansible | 09:40 | |
| *** SecOpsNinja has joined #openstack-ansible | 09:55 | |
| *** jawad_axd has joined #openstack-ansible | 10:07 | |
| SecOpsNinja | hello everyone. how is the best way to find the corresponding lxc container to tap... interface in a brq... bridge? | 10:11 |
| *** cshen has quit IRC | 10:12 | |
| SecOpsNinja | im asking because in my external bridge i cant find the corresponding interfaces corresponding to ${CONTAINER_UNIQUE_ID}_${NETWORK_DEVICE_NAME} | 10:14 |
| openstackgerrit | Dmitriy Rabotyagov (noonedeadpunk) proposed openstack/openstack-ansible-os_magnum master: Define condition for the first play host one time https://review.opendev.org/751780 | 10:22 |
| *** shyamb has joined #openstack-ansible | 10:32 | |
| *** cshen has joined #openstack-ansible | 10:40 | |
| CeeMac | jrosser: reading that thread for journal vacuum looks like it was resolved in systemd v244, is it the case we're waiting on systemd v244 getting incorporated somewhere upstream for osa? | 11:25 |
| jrosser | CeeMac: yeah, i guess it needs distros to be shipping that, or someone adventurous to upgrade theirs! | 11:26 |
| CeeMac | jrosser: fair. I'm sure ive looked into that thread a while back too | 11:26 |
| CeeMac | looks like focal is shipping with 245 | 11:28 |
| CeeMac | centos 8 might be stuck with 239 though | 11:31 |
| *** shyamb has quit IRC | 11:33 | |
| openstackgerrit | Dmitriy Rabotyagov (noonedeadpunk) proposed openstack/openstack-ansible-os_barbican master: Define condition for the first play host one time https://review.opendev.org/751248 | 11:51 |
| openstackgerrit | Dmitriy Rabotyagov (noonedeadpunk) proposed openstack/openstack-ansible-os_neutron master: Define condition for the first play host one time https://review.opendev.org/751790 | 11:51 |
| openstackgerrit | Dmitriy Rabotyagov (noonedeadpunk) proposed openstack/openstack-ansible-os_heat master: Define condition for the first play host one time https://review.opendev.org/751274 | 11:54 |
| openstackgerrit | Dmitriy Rabotyagov (noonedeadpunk) proposed openstack/openstack-ansible-os_glance master: Define condition for the first play host one time https://review.opendev.org/751270 | 11:54 |
| openstackgerrit | Dmitriy Rabotyagov (noonedeadpunk) proposed openstack/openstack-ansible-os_cinder master: Define condition for the first play host one time https://review.opendev.org/751260 | 11:54 |
| openstackgerrit | Dmitriy Rabotyagov (noonedeadpunk) proposed openstack/openstack-ansible-os_designate master: Define condition for the first play host one time https://review.opendev.org/751268 | 11:55 |
| openstackgerrit | Dmitriy Rabotyagov (noonedeadpunk) proposed openstack/openstack-ansible-os_gnocchi master: Define condition for the first play host one time https://review.opendev.org/751272 | 11:55 |
| *** cshen has quit IRC | 12:39 | |
| *** dave-mccowan has joined #openstack-ansible | 12:41 | |
| *** cshen has joined #openstack-ansible | 13:13 | |
| *** cshen has quit IRC | 13:18 | |
| *** cshen has joined #openstack-ansible | 13:26 | |
| SecOpsNinja | hello everyone. how can i find the cause of a k8s failure in a specif stack? i was able to fidn that the stack that failed was of type OS::Heat::ResourceGroup but i can't find any information in lxc heat container logs | 13:27 |
| SecOpsNinja | second is there any way to know what coreos password did heat put in fedora-coreos configuration when instructed by magnum to create a k8s cluster? | 13:28 |
| *** sshnaidm is now known as sshnaidm|afk | 13:32 | |
| openstackgerrit | Dmitriy Rabotyagov (noonedeadpunk) proposed openstack/openstack-ansible-os_ceilometer master: Define condition for the first play host one time https://review.opendev.org/751257 | 13:49 |
| openstackgerrit | Dmitriy Rabotyagov (noonedeadpunk) proposed openstack/openstack-ansible-os_mistral master: Define condition for the first play host one time https://review.opendev.org/751784 | 13:50 |
| openstackgerrit | Dmitriy Rabotyagov (noonedeadpunk) proposed openstack/openstack-ansible-os_masakari master: Define condition for the first play host one time https://review.opendev.org/751783 | 13:50 |
| *** d34dh0r53 has joined #openstack-ansible | 13:54 | |
| *** owalsh has quit IRC | 14:12 | |
| *** spatel has joined #openstack-ansible | 14:15 | |
| *** krah has quit IRC | 14:22 | |
| *** krah has joined #openstack-ansible | 14:45 | |
| *** gyee has joined #openstack-ansible | 15:05 | |
| *** zerozephyrum has joined #openstack-ansible | 15:07 | |
| *** kukacz has quit IRC | 15:12 | |
| *** kukacz has joined #openstack-ansible | 15:14 | |
| *** shyamb has joined #openstack-ansible | 15:18 | |
| *** shyamb has quit IRC | 15:21 | |
| *** owalsh has joined #openstack-ansible | 15:24 | |
| *** miloa has quit IRC | 15:39 | |
| *** mgariepy has quit IRC | 15:51 | |
| *** mgariepy has joined #openstack-ansible | 15:51 | |
| openstackgerrit | Merged openstack/openstack-ansible master: Added Openstack Senlin role deployment https://review.opendev.org/749540 | 15:52 |
| *** ChiTo has joined #openstack-ansible | 15:53 | |
| *** cshen has quit IRC | 15:55 | |
| ChiTo | Hi guys, I am deploying an environment but I am getting stuck at the venv wheel builds for keystone node/service because there is a file that the system is expecting but it is not on the repo server, the file is the keystone-10.1.5-constraints.txt, it simply says the file is not found and the playbook just finishes with the Failed status on that node, do you have any clue why that specific constraints file is not | 15:56 |
| ChiTo | loaded/downloaded? I noticed the other constraints files like globals, etc. are already on the /var/www/repo/os-releases but the only one missing is the keystone-20.1.5-constraints.txt. Any clue by chance what is going on? | 15:56 |
| spatel | ChiTo: are you doing AIO or 3 node controller? | 15:57 |
| ChiTo | spatel: I am using 3 node controller | 16:00 |
| ChiTo | spatel: with HAProxy as well | 16:00 |
| spatel | check all 3 repo server and see if they are in sync (lsyncd daemon is responsible for that) | 16:01 |
| spatel | i would say verify that file across all 3 nodes | 16:01 |
| ChiTo | spatel: yes, I did it actually, but that file is the only one missing | 16:02 |
| *** zerozephyrum has quit IRC | 16:02 | |
| ChiTo | spatel: Is there any tearing down playbook or role, to reprovision the repo servers? | 16:02 |
| ChiTo | or in general all my environment | 16:02 |
| ChiTo | because usually I just leverage the nature of ansible and I reexecute the playbooks | 16:03 |
| ChiTo | but I would prefer to tear everything down and to redeploy | 16:03 |
| ChiTo | to see if that file is downloaded properly and synced across the controllers/repos | 16:03 |
| spatel | I don't think its repo server issue. but if you want to destroy everything and re-build then you need to delete all container and remove data directory then you can re-build everything from scratch | 16:03 |
| ChiTo | I am using just bare metal/VMs | 16:04 |
| ChiTo | with no_containers on true | 16:04 |
| spatel | oh | 16:04 |
| spatel | you can just try to re-run repo playbook and see | 16:04 |
| noonedeadpunk | ChiTo: that doesn't cancel the fact of existance of repo stuff | 16:04 |
| noonedeadpunk | it's just placed on nodes but not in containers | 16:05 |
| spatel | what is the reason you pick bare metal and not container? | 16:05 |
| noonedeadpunk | as well as lsyncd is expected to exist | 16:05 |
| ChiTo | so the keystone-**-constraints is triggered through the taskfor the venv on openstack yamls or the setup-hosts (repo playbook?) | 16:05 |
| noonedeadpunk | can you past error? | 16:06 |
| ChiTo | noonedeadpunk: understood, so at the end I need to redeploy the repo and to figure out why the keystone-20.1.5-constraints.txt is not loaded? | 16:06 |
| ChiTo | sure thing | 16:06 |
| jrosser | that file should be created by the python_venv_build role | 16:06 |
| jrosser | and i think that there is some cleanup that may delete files if the venv build fails | 16:07 |
| ChiTo | I am reexecuting now to catch the error / raise right now | 16:07 |
| spatel | jrosser: does keystone role deploy that file? | 16:07 |
| jrosser | each role which calls python_venv_build will result in some of those files being created, so yes it will be done via the keystone role | 16:09 |
| noonedeadpunk | but still I think it's delegated to the "repo hosts" | 16:09 |
| *** cshen has joined #openstack-ansible | 16:10 | |
| jrosser | yes i think so | 16:10 |
| noonedeadpunk | *venv_build_host | 16:10 |
| noonedeadpunk | so probably in case of lsyncd issue, it can be missing during futher retrieving | 16:10 |
| noonedeadpunk | (not sure though) | 16:10 |
| spatel | jrosser: just curious why do we need keystone-20.1.5-constraints.txt file on repo, if i am not wrong this file is part of keystone source package and setup can read from source directory itself. (just curious to find out) | 16:11 |
| *** tosky has quit IRC | 16:11 | |
| jrosser | afaik is a file generated by the venv build role to specify the exact constraints which are used for building the keystone venv | 16:11 |
| ChiTo | fatal: [brlp-ope-key1 -> 172.28.118.11]: FAILED! => {"changed": false, "msg": "file not found: /var/www/repo/os-releases/20.1.5/keystone-20.1.5-constraints.txt"} | 16:11 |
| noonedeadpunk | I gues it's to use pre-build wheels | 16:11 |
| spatel | noonedeadpunk: in past i had issue with lsyncd (where it failed to start) and i had seen this kind of errors.. | 16:11 |
| ChiTo | that is the fatal/FAILED error | 16:12 |
| noonedeadpunk | https://opendev.org/openstack/ansible-role-python_venv_build/src/branch/master/tasks/python_venv_wheel_build.yml#L162-L169 | 16:12 |
| noonedeadpunk | ChiTo: please check lsyncd status on repo_all[0] host | 16:12 |
| ChiTo | sure thing | 16:13 |
| spatel | noonedeadpunk: but if that file missing on all 3 repo nodes then its not going to help with lsyncd right? | 16:13 |
| noonedeadpunk | yeah. right | 16:13 |
| noonedeadpunk | but it's being removed in case of failure I guess | 16:13 |
| *** cshen has quit IRC | 16:14 | |
| noonedeadpunk | (I can mix things up ofc) | 16:14 |
| spatel | check logs on like /var/log/python_venv_build* (something like that) | 16:14 |
| noonedeadpunk | so you should fast ctrl+c to see it existing :p | 16:14 |
| ChiTo | the only lsycnd.log information is coming from yesterday night | 16:15 |
| ChiTo | it looks like it did not log on this latest triggering | 16:15 |
| ChiTo | I just see: | 16:15 |
| ChiTo | Tue Sep 15 22:34:45 2020 Normal: Executing precmd: /usr/bin/ssh -l nginx -i /var/www/.ssh/id_rsa -o StrictHostKeyChecking=no -o ServerAliveInterval=30 -o ServerAliveCountMax=5 172.28.118.13 "sudo /var/www/repo/repo_prepost_cmd.sh pre-www" | 16:15 |
| ChiTo | Tue Sep 15 22:34:46 2020 Normal: Collecting event, exitcode: 0 | 16:15 |
| ChiTo | Tue Sep 15 22:34:46 2020 Normal: Finished a list after exitcode: 0 | 16:15 |
| ChiTo | Tue Sep 15 22:34:46 2020 Normal: Executing postcmd: /usr/bin/ssh -l nginx -i /var/www/.ssh/id_rsa -o StrictHostKeyChecking=no -o ServerAliveInterval=30 -o ServerAliveCountMax=5 172.28.118.12 "sudo /var/www/repo/repo_prepost_cmd.sh post-www" | 16:15 |
| ChiTo | Tue Sep 15 22:34:47 2020 Normal: Collecting event, exitcode: 0 | 16:15 |
| ChiTo | Tue Sep 15 22:34:47 2020 Normal: Finished a list after exitcode: 0 | 16:16 |
| ChiTo | Tue Sep 15 22:34:47 2020 Normal: Executing postcmd: /usr/bin/ssh -l nginx -i /var/www/.ssh/id_rsa -o StrictHostKeyChecking=no -o ServerAliveInterval=30 -o ServerAliveCountMax=5 172.28.118.13 "sudo /var/www/repo/repo_prepost_cmd.sh post-www" | 16:16 |
| noonedeadpunk | looks valid... | 16:17 |
| ChiTo | this log is on the ctrl1 | 16:17 |
| ChiTo | but there is no lsyncd.log on ctrl2 nor ctrl3 | 16:17 |
| noonedeadpunk | (ps please use paste.openstack.org for pasting logs) | 16:17 |
| ChiTo | sure thing thx | 16:17 |
| noonedeadpunk | there should be only one lsyncd | 16:18 |
| noonedeadpunk | and can you check if https://opendev.org/openstack/ansible-role-python_venv_build/src/branch/master/tasks/python_venv_wheel_build.yml#L162-L169 is delegated to the same host? | 16:18 |
| noonedeadpunk | 172.28.118.11 I guess | 16:18 |
| ChiTo | that is the correct the .11 | 16:19 |
| ChiTo | it is delegated there | 16:19 |
| ChiTo | I am gonna increase the verbosity on the openstack-ansible and re execute because I remember I catched an ssh failed connection | 16:20 |
| ChiTo | let me re execute it | 16:20 |
| spatel | ChiTo: try "locate keystone-20.1.5-constraints.txt" and see if you can find it anywhere in system | 16:21 |
| spatel | it should be in /openstack/vens/keystone..... | 16:21 |
| ChiTo | spatel: Yep, I have search for it but unfortunately is not on the system | 16:21 |
| noonedeadpunk | But that task passes for you which means that it was created.... | 16:22 |
| ChiTo | noonedeadpunk: It should be on the repo right? | 16:22 |
| noonedeadpunk | I believe it is. in case you run in verbose, you can check where it puts it | 16:23 |
| ChiTo | yep it supposes it is created on the /var/www/repo/os-releases | 16:23 |
| ChiTo | let me reexecute it brb guys | 16:24 |
| spatel | also check logs /var/log/python_venv_build* it will give you lots of info | 16:24 |
| *** jawad_axd has quit IRC | 16:24 | |
| ChiTo | spatel: on the controller right? | 16:24 |
| spatel | yes | 16:24 |
| spatel | in my case i have keystone container where i found them | 16:25 |
| spatel | in -vvvvv it will show you where its dumping logs | 16:25 |
| spatel | its /var/log/python_venv_build.log | 16:27 |
| *** d34dh0r53 has quit IRC | 16:32 | |
| *** d34dh0r53 has joined #openstack-ansible | 16:35 | |
| spatel | may be something here - https://opendev.org/openstack/ansible-role-python_venv_build/src/branch/master/tasks/python_venv_wheel_build.yml#L43 | 16:36 |
| spatel | ansible-role-python_venv_build playbook is responsible to dump file on repo server and somehow its not. | 16:36 |
| spatel | as jrosser mentioned if it failed somewhere then it won't copy that file.. or clean up | 16:37 |
| *** jawad_axd has joined #openstack-ansible | 16:37 | |
| *** tobberydberg has quit IRC | 16:51 | |
| *** tobberydberg_ has joined #openstack-ansible | 16:51 | |
| *** johanssone_ has quit IRC | 16:59 | |
| *** johanssone has joined #openstack-ansible | 17:03 | |
| *** jawad_axd has quit IRC | 17:15 | |
| *** mathlin has quit IRC | 17:27 | |
| *** mathlin has joined #openstack-ansible | 17:29 | |
| mgariepy | ebbex, ceph-volume stuff: https://tracker.ceph.com/issues/47502 | 17:32 |
| openstackgerrit | Merged openstack/openstack-ansible stable/rocky: How to upgrade from xenial to bionic on rocky https://review.opendev.org/710803 | 17:34 |
| ChiTo | Hi guys, I reexecuted the playbook after removing all the files in the repo server, I noticed the file is missing yet but now I got a different error related to a Failed pip building "scrypt" I got this msg: | 17:51 |
| ChiTo | http://paste.openstack.org/show/797943/ | 17:51 |
| ChiTo | and based on your spatel recommendation I got the error in the python_wheel_build.log: http://paste.openstack.org/show/797944/ | 17:53 |
| spatel | ChiTo: that explain everything | 17:54 |
| spatel | what OS ? version are you running? | 17:54 |
| ChiTo | spatel: I am on CentOS 7.8 | 17:54 |
| spatel | openstack version? | 17:54 |
| ChiTo | spatel: Train with OSA 20.1.5 | 17:54 |
| ChiTo | spatel | 17:55 |
| ChiTo | but I did not get the | 17:55 |
| ChiTo | missing file | 17:55 |
| ChiTo | even if it was not generated, are those files generated after the pip success? | 17:55 |
| spatel | jrosser: and noonedeadpunk can explain more about that | 17:56 |
| spatel | based on error mesg - ERROR: Failed building wheel for scrypt | 17:56 |
| spatel | did you check /var/log/python_venv_build.log | 17:57 |
| ChiTo | spatel: yes, but I just got these latest lines: http://paste.openstack.org/show/797945/ | 17:58 |
| *** jawad_axd has joined #openstack-ansible | 18:00 | |
| noonedeadpunk | lgtm | 18:00 |
| noonedeadpunk | can you paste like whole output of the run? | 18:01 |
| ChiTo | noonedeadpunk: sure thing, the ansible.log or the venv logs? | 18:01 |
| ChiTo | I ran it with 5 v's | 18:01 |
| noonedeadpunk | ansible | 18:02 |
| ChiTo | sure | 18:02 |
| noonedeadpunk | maybe even without so much v's :) | 18:02 |
| ChiTo | sure let me reexecute it with just 2 v's :) | 18:02 |
| *** SecOpsNinja has left #openstack-ansible | 18:07 | |
| *** jawad_axd has quit IRC | 18:15 | |
| *** spatel has quit IRC | 18:25 | |
| *** spatel has joined #openstack-ansible | 18:47 | |
| *** tosky has joined #openstack-ansible | 18:51 | |
| ChiTo | noonedeadpunk: not sure why but the Lodge It server from paste.openstack.org does limit about 600 lines, so I share you this link: https://paste.centos.org/view/cfd4c388 | 18:55 |
| ChiTo | this was the last ansible.log try and now it shows up the file missing for the constraints.txt | 18:55 |
| noonedeadpunk | oh. hm | 18:58 |
| noonedeadpunk | `Task "Build constraints file for installation purposes" has been omitted from the job` | 18:59 |
| noonedeadpunk | ^ that shouldn't happen | 18:59 |
| noonedeadpunk | actually it should execute but not be skipped | 18:59 |
| ChiTo | should I change the keystone_install_method variable perhaps? | 19:00 |
| noonedeadpunk | I think it's source by default | 19:00 |
| noonedeadpunk | I'm wondering where this condition comes from | 19:01 |
| noonedeadpunk | I see only `venv_wheel_build_enable` in that file | 19:02 |
| noonedeadpunk | ah, I see | 19:03 |
| noonedeadpunk | block inside block | 19:03 |
| noonedeadpunk | you know what.... | 19:03 |
| ChiTo | noonedeadpunk: got it, shall we tear it down and re execute setup-hosts? | 19:04 |
| noonedeadpunk | ChiTo: try just dropping /var/www/repo/os-releases/20.1.5/keystone-20.1.5-requirements.txt on .11 host | 19:05 |
| noonedeadpunk | then it will trigger _requirement_file as changed | 19:05 |
| noonedeadpunk | and should finally build constraints | 19:06 |
| ChiTo | sure, let me try it, the lsyncd automatically will replicate to the other controllers? | 19:06 |
| noonedeadpunk | but probably worth adding some force variable.... | 19:06 |
| noonedeadpunk | it's supposed to) | 19:07 |
| ChiTo | got it, re-deploying playbook... | 19:09 |
| noonedeadpunk | also ofc you can try disabling venv_wheel_build_enable, but not sure it's really good idea (in terms of deployment speed) | 19:09 |
| ChiTo | without the requirements on the repo | 19:09 |
| noonedeadpunk | yeah, shoot it | 19:09 |
| noonedeadpunk | ah, you just needed to use `-e venv_rebuild=true` as argument - my bad sorry | 19:11 |
| noonedeadpunk | that would drop these files for you:) | 19:12 |
| ChiTo | noonedeadpunk: oh great hint, you mean along the openstack-ansible cli right? | 19:13 |
| ChiTo | just as an additional flag | 19:13 |
| ChiTo | right now the wheels are in the process of building | 19:15 |
| ChiTo | and so far we are getting a: FAILED - RETRYING: Build wheels for the packages to be installed into the venv (3 retries left). | 19:15 |
| spatel | 19:16 | |
| spatel | 19:16 | |
| spatel | trying to catchup.. so what was the issue? | 19:17 |
| ChiTo | spatel: Hi, still on diagnosing with noonedeadpunk but so far we have removed the *requirements.txt to allow to rebuild the files, right now the wheels are rebuilding | 19:18 |
| *** jawad_axd has joined #openstack-ansible | 19:19 | |
| spatel | just curious why did you did that bug? | 19:19 |
| *** MickyMan77 has joined #openstack-ansible | 19:20 | |
| spatel | s/did/hit/ | 19:20 |
| ChiTo | noonedeadpunk: This is the output after removing the keystone-20.1.5-requirements.txt file, I noticed it was regenerated but still missing the keystone-20.1.5-constraints.txt: https://paste.centos.org/view/f7b63041 | 19:24 |
| *** MickyMan77 has quit IRC | 19:25 | |
| ChiTo | spatel: yeah me too, it is the first time I hit it, I did not see it in previous versions but not sure if the venv_build change some code from the 20.1.1 to the 20.1.5 | 19:25 |
| ChiTo | the only change I have done in the last 5 days was to upgrade from .1 to .5 but I never ran the setup-openstack with the .1 so it is the 1st I hit the bug | 19:26 |
| *** andrewbonney has quit IRC | 19:28 | |
| spatel | I can still see error - ERROR: Failed building wheel for scrypt | 19:28 |
| ChiTo | noonedeadpunk: I am reexecuting with the venv_rebuild extra var to make sure everythiing is rebuilt | 19:28 |
| ChiTo | noonedeadpunk: This is the log with venv_rebuild=true https://paste.centos.org/view/d2202d8b | 19:40 |
| noonedeadpunk | ChiTo: sorry, was out of pc | 19:41 |
| ChiTo | noonedeadpunk: no prob, on the contrary thanks for your time | 19:42 |
| noonedeadpunk | ok, this is closer to the issue, which is actually build failure | 19:42 |
| ChiTo | this is the https://paste.centos.org/view/e4367315 list of files generated | 19:42 |
| ChiTo | since Sep 11th I noticing the utility-* as well | 19:42 |
| noonedeadpunk | `ERROR: Failed building wheel for scrypt` looks like the isue | 19:43 |
| noonedeadpunk | ooook, I think the issue might be in py2..... | 19:43 |
| noonedeadpunk | and py3 for centos is supported only since ussuri (21.0.0) | 19:44 |
| noonedeadpunk | I think you can reproduce it by running on .11 | 19:45 |
| noonedeadpunk | `/openstack/venvs/wheel-builder-python2/bin/pip wheel --requirement /var/www/repo/os-releases/20.1.5/keystone-20.1.5-requirements.txt --constraint /var/www/repo/os-releases/20.1.5/keystone-20.1.5-global-constraints.txt --constraint /var/www/repo/os-releases/20.1.5/keystone-20.1.5-source-constraints.txt --wheel-dir /tmp/keystone-20.1.5/ --find-links /var/www/repo/os-releases/20.1.5/centos-7.8-x86_64/ --log /var/log/python_wheel_build.log` | 19:45 |
| ChiTo | understood, let me execute it | 19:45 |
| noonedeadpunk | oh wait | 19:46 |
| ChiTo | sure | 19:46 |
| noonedeadpunk | can you post output of /var/log/python_wheel_build.log ? | 19:46 |
| noonedeadpunk | scrypt 0.8.13 seems to be supported by py2 | 19:47 |
| spatel | ChiTo: why are you not using CentOS-8 ? believe starting something on centos-7 today would be little dangerous because developer move forward.. | 19:48 |
| jrosser | didn’t we ha e the venvs as py3 for train? | 19:49 |
| ChiTo | spatel: unfortunately my end user needs train for that deployment and not ussuri/centos 8.x :( | 19:49 |
| jrosser | even if the host was py2-ish | 19:49 |
| noonedeadpunk | jrosser: only for ubuntu? | 19:50 |
| noonedeadpunk | we had hook that prevented centos from py3 venvs | 19:51 |
| jrosser | oh hmm | 19:51 |
| noonedeadpunk | and backporting py3 not sure how good idea is... | 19:51 |
| *** cshen has joined #openstack-ansible | 19:51 | |
| *** kukacz has quit IRC | 19:52 | |
| noonedeadpunk | and tbh CI looked fine recently as well.... | 19:53 |
| spatel | noonedeadpunk: i would like to give it a shot with centos-7.8 + train (next week i am planning to upgrade my ocata to train ) | 19:54 |
| noonedeadpunk | can wish only good luck with that.... | 19:54 |
| noonedeadpunk | I mean ocata.... | 19:55 |
| spatel | don't say that :) | 19:55 |
| *** cshen has quit IRC | 19:56 | |
| noonedeadpunk | yeah, sorry, I should have been more optimistic :) | 19:56 |
| spatel | This is very big problem sitting between two major OS release :( | 19:56 |
| noonedeadpunk | imo it's big problem when release you're sitting goes to eol.... | 19:57 |
| *** MickyMan77 has joined #openstack-ansible | 19:57 | |
| noonedeadpunk | since then upgrade path might become pretty tricky... | 19:57 |
| spatel | its not easy to upgrade OS on 400 compute nodes and expect no issue :) | 19:57 |
| noonedeadpunk | esp, when most of the releases are on the upgrade path way are eol as well | 19:58 |
| spatel | currently i am testing everything on lab with ussuri but with in few week we will have victoria :( | 19:58 |
| noonedeadpunk | ofc you can try skipping some.... | 19:58 |
| spatel | I am thinking to go with victoria direct | 19:59 |
| noonedeadpunk | I guss generally we provide upgrades once a year or so | 19:59 |
| noonedeadpunk | yeah, opensatck is missing some LTS releases... | 20:00 |
| spatel | everyone wants new feature every 6 months.. | 20:00 |
| noonedeadpunk | I mean openstack i on the level when it's more about maintenance then really features | 20:01 |
| spatel | +1 | 20:02 |
| spatel | noonedeadpunk: when are we going to merge senlin... (sorry for asking same question :D) | 20:03 |
| spatel | i know its holding back on other patches but just curious when it will get clear | 20:03 |
| noonedeadpunk | as soon as it will pass gates:) | 20:04 |
| noonedeadpunk | if it won't fail then I guess about 3 hours | 20:04 |
| spatel | which gates holding it back? I am seeing all patches passed | 20:04 |
| noonedeadpunk | yeah, needed to push it what I just did | 20:05 |
| *** jawad_axd has quit IRC | 20:06 | |
| spatel | can you point me to patch which is failing or blocking gates | 20:09 |
| noonedeadpunk | ok, let me rephrase it - https://review.opendev.org/749540 was a dependant patch. It has merged, but not from the first try (but after recheck). In this case your patch got a bit stuck, as in such situations it won't go to gates automatically. Instead it should be pushed there with adding (or removing and adding) +W | 20:11 |
| *** owalsh has quit IRC | 20:11 | |
| noonedeadpunk | so now it in gates as I've just pushed it there | 20:11 |
| noonedeadpunk | and nothing blocks it at this point | 20:12 |
| spatel | thank you for explanation | 20:13 |
| ChiTo | noonedeadpunk: The wheels log is huge but I am noticing there is no aes.h from openssl library, let me try to install openssl-devel on the centos 7 to see if that works | 20:18 |
| noonedeadpunk | I'm wondering if you have aes cpu instruction as well | 20:18 |
| ChiTo | yes I do, I listed in proc/cpuinfo and the aes flag is shown | 20:20 |
| noonedeadpunk | huh, yeah. try openssl-devel then, but I thought we install it... | 20:20 |
| noonedeadpunk | https://opendev.org/openstack/openstack-ansible-os_keystone/src/branch/master/vars/redhat.yml#L30 | 20:21 |
| ChiTo | noonedeadpunk: http://paste.openstack.org/show/797959/ this is the error I caught let me install openssl-devel on 3 ctrollers because actually I did not have it | 20:21 |
| *** MickyMan77 has quit IRC | 20:21 | |
| noonedeadpunk | hm keystone_devel_distro_packages should be installed on the host where wheels are build (.11) | 20:22 |
| ChiTo | are those vars set by default? because I did not set it on user_variables nor any additional yaml file | 20:23 |
| noonedeadpunk | oh | 20:23 |
| noonedeadpunk | we don't have it for train o_O | 20:23 |
| noonedeadpunk | ok, I added it lol https://opendev.org/openstack/openstack-ansible-os_keystone/commit/e0197db984f8e772b9a1c235e4f347d596255603 | 20:24 |
| ChiTo | Awesome!! :) | 20:25 |
| noonedeadpunk | but didn't backport | 20:25 |
| ChiTo | I have installed openssl-devel, re executing now | 20:25 |
| openstackgerrit | Dmitriy Rabotyagov (noonedeadpunk) proposed openstack/openstack-ansible-os_keystone stable/train: Add libssl devel package for CentOS and Suse https://review.opendev.org/752310 | 20:25 |
| spatel | fingers cross ** | 20:29 |
| ChiTo | it worked now! | 20:33 |
| ChiTo | 2020-09-16T17:30:56,299 Successfully built argparse keystone python-ldap munch Mako pyngus Tempita simplejson pyinotify prettytable SQLAlchemy unicodecsv pycparser PyYAML pyperclip warlock python-qpid-proton dogpile.cache scrypt systemd-python yappi wrapt pyrsistent repoze.lru functools32 alembic scandir uWSGI | 20:33 |
| ChiTo | 20:33 | |
| noonedeadpunk | cool | 20:33 |
| ChiTo | thank you Dmitriy! | 20:34 |
| noonedeadpunk | not sure how we had it working in CI.... maybe because we have single host and we don't build wheels because of that.... | 20:34 |
| ChiTo | now it is installing everything related to the keystone role so far so good | 20:34 |
| ChiTo | noonedeadpunk: unless you already had the openssl-devel installed on the CI environment | 20:35 |
| noonedeadpunk | or that, yes | 20:35 |
| noonedeadpunk | jrosser: how do you think - is it worth explicitly enabling venv_wheel_build_enable for CI builds? | 20:35 |
| noonedeadpunk | as I guess it's more typical path? | 20:36 |
| noonedeadpunk | but it will be even more time in CI | 20:36 |
| noonedeadpunk | or maybe same... | 20:36 |
| *** spatel has quit IRC | 20:37 | |
| openstackgerrit | Dmitriy Rabotyagov (noonedeadpunk) proposed openstack/openstack-ansible master: Enable venv_wheel_build_enable for CI https://review.opendev.org/752311 | 20:40 |
| noonedeadpunk | just placed it to see current status... we can have more input to decide... | 20:40 |
| *** owalsh has joined #openstack-ansible | 20:43 | |
| openstackgerrit | Dmitriy Rabotyagov (noonedeadpunk) proposed openstack/openstack-ansible-tests master: Remove centos 7 jobs https://review.opendev.org/752313 | 20:47 |
| openstackgerrit | Dmitriy Rabotyagov (noonedeadpunk) proposed openstack/openstack-ansible-tests master: Remove centos 7 jobs from templates https://review.opendev.org/752313 | 20:48 |
| *** owalsh has quit IRC | 20:56 | |
| *** owalsh has joined #openstack-ansible | 21:18 | |
| openstackgerrit | Merged openstack/openstack-ansible-os_senlin master: Initial commit to os_senlin https://review.opendev.org/749365 | 21:35 |
| *** cshen has joined #openstack-ansible | 21:52 | |
| *** cshen has quit IRC | 21:56 | |
| *** tosky_ has joined #openstack-ansible | 21:56 | |
| *** tosky has quit IRC | 21:59 | |
| *** tosky_ is now known as tosky | 22:00 | |
| *** MickyMan77 has joined #openstack-ansible | 22:00 | |
| *** MickyMan77 has quit IRC | 22:11 | |
| *** MickyMan77 has joined #openstack-ansible | 22:41 | |
| *** MickyMan77 has quit IRC | 22:50 | |
| *** tosky has quit IRC | 22:59 | |
| ChiTo | unfortunately I am hitting a new issue | 23:10 |
| ChiTo | it looks like the keystone servics are installed on the key1 server but not on the key2 nor key3 because it says that is waiting until the services are up | 23:10 |
| ChiTo | and it goes to the 5000 port on the vip from my haproxy but I am assuming it does not respond with a valid response code from keystone | 23:11 |
| ChiTo | It is stuck at the Wait for services to be up TASK forboth key2 node and key3 node: http://paste.openstack.org/show/797963/ | 23:17 |
| *** ChiTo has left #openstack-ansible | 23:17 | |
| *** ChiTo has joined #openstack-ansible | 23:18 | |
| *** ChiTo has left #openstack-ansible | 23:18 | |
| *** ChiTo has joined #openstack-ansible | 23:18 | |
| *** ChiTo has quit IRC | 23:45 | |
| *** cshen has joined #openstack-ansible | 23:52 | |
| *** cshen has quit IRC | 23:56 | |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!