| *** luksky has quit IRC | 00:08 | |
| *** cshen has quit IRC | 00:26 | |
| *** pcaruana has quit IRC | 01:12 | |
| *** pcaruana has joined #openstack-ansible | 01:12 | |
| dotnetted | Hey all - I can ping the br-vlan gateway from the host through the br-vlan interface without issue, but pinging from the namespaced external network results in an unanswered ARP request for the gateway IP - I'm a networking noob, anything obvious I might be missing here? Thanks! http://paste.openstack.org/show/Ci6iMM3MjlSBtbfitFkg/ | 01:40 |
|---|---|---|
| *** macz_ has quit IRC | 01:43 | |
| *** tinwood has quit IRC | 02:08 | |
| *** tinwood has joined #openstack-ansible | 02:11 | |
| *** macz_ has joined #openstack-ansible | 02:21 | |
| *** cshen has joined #openstack-ansible | 02:22 | |
| *** cshen has quit IRC | 02:26 | |
| *** spatel has joined #openstack-ansible | 02:27 | |
| *** spatel has quit IRC | 02:29 | |
| *** spatel has joined #openstack-ansible | 02:30 | |
| *** macz_ has quit IRC | 02:30 | |
| *** spatel has quit IRC | 02:31 | |
| *** spatel has joined #openstack-ansible | 02:33 | |
| *** cshen has joined #openstack-ansible | 02:45 | |
| *** spatel has quit IRC | 02:49 | |
| *** cshen has quit IRC | 02:49 | |
| *** spatel has joined #openstack-ansible | 02:55 | |
| *** johanssone has quit IRC | 03:00 | |
| *** johanssone has joined #openstack-ansible | 03:01 | |
| *** cshen has joined #openstack-ansible | 04:45 | |
| *** macz_ has joined #openstack-ansible | 04:47 | |
| *** cshen has quit IRC | 04:50 | |
| *** macz_ has quit IRC | 04:52 | |
| *** evrardjp has quit IRC | 05:33 | |
| *** evrardjp has joined #openstack-ansible | 05:33 | |
| *** spatel has quit IRC | 05:47 | |
| *** gyee has quit IRC | 06:07 | |
| *** cshen has joined #openstack-ansible | 06:30 | |
| *** cshen has quit IRC | 06:35 | |
| *** miloa has joined #openstack-ansible | 06:43 | |
| *** SiavashSardari has joined #openstack-ansible | 06:49 | |
| *** spotz has quit IRC | 06:54 | |
| *** antonym has quit IRC | 06:56 | |
| *** antonym has joined #openstack-ansible | 06:57 | |
| *** cshen has joined #openstack-ansible | 07:01 | |
| frickler | dotnetted: seems like you might be stacking two vlans? you have br-vlan on ethxxx.80 and then use eth1.80, likely one of them is too much. | 07:35 |
| *** SiavashSardari has quit IRC | 07:42 | |
| *** SiavashSardari has joined #openstack-ansible | 07:46 | |
| *** luksky has joined #openstack-ansible | 07:50 | |
| *** luksky has quit IRC | 07:53 | |
| *** rpittau|afk_ is now known as rpittau | 07:57 | |
| SiavashSardari | morning, does anyone have volume.provider.capacity.allocated metric in the OpenStack Ceilometer/Gnocchi? I'm having trouble with volume provider metrics in my setup | 08:00 |
| openstackgerrit | Jonathan Rosser proposed openstack/openstack-ansible-os_zun master: Move zun pip packages from constraints to requirements https://review.opendev.org/c/openstack/openstack-ansible-os_zun/+/772300 | 08:03 |
| *** andrewbonney has joined #openstack-ansible | 08:19 | |
| *** maharg101 has joined #openstack-ansible | 08:28 | |
| noonedeadpunk | SiavashSardari: I don't collect it :( But I think I had disk.device.allocation and it was working (except windows machines) | 08:45 |
| *** tosky has joined #openstack-ansible | 08:45 | |
| jrosser | morning | 08:47 |
| CeeMac | o/ | 08:48 |
| SiavashSardari | noonedeadpunk I was thinking about disk.device.allocation but we have to sum all values for all volumes to have an overview about ceph allocations. | 08:48 |
| SiavashSardari | or maybe there is another solution for that. | 08:49 |
| noonedeadpunk | I guess it would be as incorrect as inside ceph because of not using discard properly unless you set volumes to use iscsi mode instead of ide | 08:50 |
| MickyMan77 | good morning, | 08:50 |
| MickyMan77 | I do get an error when I deploy version 21.2.2.. | 08:51 |
| MickyMan77 | http://paste.openstack.org/show/801975/ | 08:51 |
| openstackgerrit | Jonathan Rosser proposed openstack/openstack-ansible-os_ironic master: Move ironic pip packages from constraints to requirements https://review.opendev.org/c/openstack/openstack-ansible-os_ironic/+/772284 | 08:51 |
| noonedeadpunk | but to get overwiew of ceph alklocation I guess I'd rather use ceph df or smth like that... | 08:51 |
| SiavashSardari | noonedeadpunk yeah you are right about that. but the problem with ceph df is that ceph images are thin provisioned and while we use e.g. 100G but the allocation might be 1T | 08:53 |
| noonedeadpunk | Ok, wait. So you want so see how much you've sold and not how much is currently occupied? | 09:00 |
| noonedeadpunk | Because allocation I think should be currently occupied space and iirc that's what ceph df shows? | 09:01 |
| openstackgerrit | Jonathan Rosser proposed openstack/openstack-ansible-os_murano master: Use the utility host for db setup tasks https://review.opendev.org/c/openstack/openstack-ansible-os_murano/+/747236 | 09:05 |
| openstackgerrit | Jonathan Rosser proposed openstack/openstack-ansible-os_murano master: Move murano pip packages from constraints to requirements https://review.opendev.org/c/openstack/openstack-ansible-os_murano/+/772292 | 09:08 |
| *** jbadiapa has joined #openstack-ansible | 09:10 | |
| SiavashSardari | noonedeadpunk yeah exactly. | 09:11 |
| frickler | MickyMan77: seems you somehow managed to create multiple placement service records. try "openstack service list" and check for duplicates and remove them | 09:12 |
| openstackgerrit | Jonathan Rosser proposed openstack/openstack-ansible-os_gnocchi master: Move gnocchi pip packages from constraints to requirements https://review.opendev.org/c/openstack/openstack-ansible-os_gnocchi/+/772281 | 09:12 |
| noonedeadpunk | since otherwise you can use volume.size which will be totall size of cinder volume. and it's working perfectly for me | 09:13 |
| admin0 | morning \o | 09:13 |
| admin0 | noonedeadpunk, we do a query in the db directly to get allocated size vs ceph df to get current size to get an idea on it | 09:14 |
| SiavashSardari | BTW while we are on the subject of metrics, gnocchi role on branch stable/ussuri didn't work for me. I managed to get it running manually with updating tooz and ceilometer pip packages and I forgot to ask here if something should get fixed. | 09:14 |
| admin0 | mysql read only queries are fast and direct .. ( less demanding ) then using ceilometer/gnocchi and callign the apis just to get the same data | 09:15 |
| admin0 | in my opinion .. so i create a read-only user and run the data directy into a custom dashboard | 09:15 |
| noonedeadpunk | but you need to follow all db migrations which is kind of burden imo | 09:16 |
| noonedeadpunk | maintenance of direct db request results in more work that needs to be done if you want it to work reliably | 09:16 |
| SiavashSardari | admin0 that sounds like a good solution. did you implement that with ansible and have a role/playbook? | 09:16 |
| noonedeadpunk | and all api changed are pretty well documented, with release notes and some term of deprecation | 09:16 |
| admin0 | SiavashSardari, its a small php dashboard with direct mysql connection to the openstack db .. and i generate pie charts like most used images, most used flavor, vm created/deleted line graphs in the last 24 hours, last week .. etc | 09:18 |
| noonedeadpunk | btw, I guess to get data from cinder volumes you should have appropriate timerd https://opendev.org/openstack/openstack-ansible-os_cinder/src/branch/master/tasks/main.yml#L118-L127 | 09:18 |
| admin0 | at some point, i was also thinking of making it open source and maintaining it | 09:18 |
| admin0 | it gives stats that openstack does not provide at once .. like projects that are using most ram, cpu, ip addresses, volumes etc | 09:19 |
| admin0 | those kind of stats and graphs | 09:19 |
| noonedeadpunk | regarding gnocchi - not sure tbh. maybe it needs fixing, but have no idea atm | 09:19 |
| noonedeadpunk | SiavashSardari: if you have written steps that you used to make it working that would be helpful to check things out | 09:19 |
| admin0 | mysql ENTER ( in the util container) ; select sum(size) from cinder.volumes where status = 'in-use' and attach_status='attached'; | 09:24 |
| admin0 | will give you the total size of volumes vs df from the pool = you have the data | 09:24 |
| SiavashSardari | noonedeadpunk actually the cinder-volume-usage-audit caught my eyes too, but I didn't know what should I expect from it and how to get data from that service. I checked it's log and sounds like it's getting enough data to solve this issue. but I didn't know how to take advantage of that and didn't find anything on docs | 09:24 |
| noonedeadpunk | that service push useage report into notifications queue that is listened by ceilometer in this case. | 09:26 |
| *** lkoranda has joined #openstack-ansible | 09:26 | |
| *** SecOpsNinja has joined #openstack-ansible | 09:34 | |
| SiavashSardari | noonedeadpunk about gnocchi, my first problem was with tooz error on get_lock https://github.com/openstack/tooz/blob/stable/ussuri/tooz/drivers/zookeeper.py#L516 which I found out it was resolved on victoria release. then I had another error regarding redis which in kolla they fixed it in | 09:35 |
| SiavashSardari | https://review.opendev.org/c/openstack/kolla-ansible/+/668592 . and at last I had an error like https://bugzilla.redhat.com/show_bug.cgi?id=1883733 which was happening in gnocchi 4.3.3.dev (I'm not sure about the version, but it was something like that) which is the version installed by osa. then I upgrade gnocchi to 4.4.0 and then it worked | 09:35 |
| openstack | bugzilla.redhat.com bug 1883733 in gnocchi "Gnocchi API 500 Error with 'TypeError: keys must be a string'" [High,Verified] - Assigned to mmagr | 09:35 |
| SiavashSardari | properly. | 09:35 |
| noonedeadpunk | SiavashSardari: but we don't have redis at all? | 09:36 |
| SiavashSardari | the redis part is not osa issue, but we should update the docs about gnocchi with redis in osa docs | 09:36 |
| noonedeadpunk | um? | 09:36 |
| noonedeadpunk | Honestly I'd say we should use zookeeper instead | 09:37 |
| SiavashSardari | https://docs.openstack.org/openstack-ansible/latest/user/prod/gnocchi_redis.html | 09:37 |
| noonedeadpunk | oh | 09:37 |
| SiavashSardari | no we use zookeeper but I was testing to use redis as measure store and ceph as aggregate store | 09:38 |
| noonedeadpunk | aha, yes, I got now why redis raised here | 09:38 |
| noonedeadpunk | yeah, we should update docs for sure | 09:38 |
| SiavashSardari | there was some minor changes which I don't remember right now. I will check them out and upload a batch for that | 09:40 |
| openstackgerrit | Merged openstack/openstack-ansible-os_cloudkitty master: Move cloudkitty pip packages from constraints to requirements https://review.opendev.org/c/openstack/openstack-ansible-os_cloudkitty/+/772279 | 09:42 |
| openstackgerrit | Dmitriy Rabotyagov proposed openstack/openstack-ansible master: Switch gnocchi release to 4.4 https://review.opendev.org/c/openstack/openstack-ansible/+/772485 | 09:44 |
| SiavashSardari | noonedeadpunk I didn't test it but updating gnocchi will update tooz too? | 09:50 |
| openstackgerrit | Merged openstack/ansible-role-python_venv_build master: Optimize the link address in docs https://review.opendev.org/c/openstack/ansible-role-python_venv_build/+/579756 | 09:50 |
| noonedeadpunk | I think tooz version will go from u-c https://releases.openstack.org/constraints/upper/ussuri | 09:51 |
| openstackgerrit | Merged openstack/openstack-ansible-os_aodh master: Fix documentation formatting https://review.opendev.org/c/openstack/openstack-ansible-os_aodh/+/587019 | 09:51 |
| openstackgerrit | Merged openstack/openstack-ansible-tests master: Use setuptools constraint from global-requirements-pins rather than u-c https://review.opendev.org/c/openstack/openstack-ansible-tests/+/771770 | 09:51 |
| openstackgerrit | Merged openstack/ansible-role-python_venv_build master: Add source code and bugs url link to README https://review.opendev.org/c/openstack/ansible-role-python_venv_build/+/598580 | 09:52 |
| SiavashSardari | I'm not sure which version is equivalent of victoria release but I have 2.8.0 and it works fine. | 09:53 |
| admin0 | how to allow horizon to upload images ? i forgot that variable .. and 2nd qustion is .. if there is already ceph in the cluster, how to tell swift to use ceph as a backend ? | 10:10 |
| jrosser | if you want to use ceph radosgw for object storage then you don't have swift | 10:12 |
| openstackgerrit | Jonathan Rosser proposed openstack/openstack-ansible-os_zun master: Move zun pip packages from constraints to requirements https://review.opendev.org/c/openstack/openstack-ansible-os_zun/+/772300 | 10:17 |
| *** dtantsur|afk is now known as dtantsur | 10:18 | |
| MickyMan77 | frickler: Can I remove the placement service and then rerun the deployment ? | 10:22 |
| MickyMan77 | http://paste.openstack.org/show/801978/ | 10:22 |
| frickler | MickyMan77: hmm, wait, looks like I misread your log (and ansible could be more verbose), can you also show the output of "openstack project list"? | 10:24 |
| frickler | admin0: jrosser: I also stumbled about horizon failing to upload images, they always end up in state "queued" and horizon pops up an error. is that a known issue? | 10:26 |
| admin0 | frickler, there needs to be a variable set in user_variables .. which is not set by default .. some legacy mode .. but i forgot it | 10:26 |
| MickyMan77 | frickler: http://paste.openstack.org/show/801980/ | 10:28 |
| MickyMan77 | the first "openstack service list" output was from wrong farm... :) | 10:29 |
| jrosser | MickyMan77: you have somehow got two 'service' projects, thats not right | 10:30 |
| frickler | admin0: HORIZON_IMAGES_UPLOAD_MODE can be legacy or direct, but I'm wondering why the default isn't a working setup | 10:30 |
| MickyMan77 | Yes, I did see that.. | 10:30 |
| admin0 | frickler, the default is that it does not work and need to use the cli | 10:33 |
| admin0 | why exactly, not sure | 10:33 |
| frickler | bad user experience for sure | 10:33 |
| MickyMan77 | I think that is old settings for the LDAP connection... I will remove one of them. | 10:33 |
| openstackgerrit | Siavash Sardari proposed openstack/openstack-ansible master: Update Gnocchi with Ceph and Redis documentation. https://review.opendev.org/c/openstack/openstack-ansible/+/772494 | 10:34 |
| frickler | MickyMan77: be careful to remove the correct one, otherwise your whole setup might break | 10:34 |
| MickyMan77 | yes. I will remove that one that was use for ldap... | 10:35 |
| openstackgerrit | Dmitriy Rabotyagov proposed openstack/openstack-ansible master: Install rsync for CentOS bootstrap https://review.opendev.org/c/openstack/openstack-ansible/+/772495 | 10:40 |
| openstackgerrit | Merged openstack/openstack-ansible-os_masakari master: Move masakari pip packages from constraints to requirements https://review.opendev.org/c/openstack/openstack-ansible-os_masakari/+/772288 | 10:43 |
| openstackgerrit | Merged openstack/openstack-ansible-os_blazar master: Move blazar pip packages from constraints to requirements https://review.opendev.org/c/openstack/openstack-ansible-os_blazar/+/772261 | 10:44 |
| openstackgerrit | Merged openstack/openstack-ansible-os_designate master: Move designate pip packages from constraints to requirements https://review.opendev.org/c/openstack/openstack-ansible-os_designate/+/772280 | 10:44 |
| openstackgerrit | Merged openstack/openstack-ansible-os_senlin master: Move senlin pip packages from constraints to requirements https://review.opendev.org/c/openstack/openstack-ansible-os_senlin/+/772296 | 10:47 |
| openstackgerrit | Merged openstack/openstack-ansible-os_swift master: Move swift packages from constraints to requirements https://review.opendev.org/c/openstack/openstack-ansible-os_swift/+/772297 | 10:50 |
| jrosser | frickler: if you are able to help improve any of these user experience things it would be awesome | 10:50 |
| openstackgerrit | Merged openstack/openstack-ansible-os_sahara master: Move sahara pip packages from constraints to requirements https://review.opendev.org/c/openstack/openstack-ansible-os_sahara/+/772295 | 10:52 |
| *** SiavashSardari has quit IRC | 11:07 | |
| *** rpittau is now known as rpittau|bbl | 11:09 | |
| *** SiavashSardari has joined #openstack-ansible | 11:18 | |
| *** masterpe has quit IRC | 11:19 | |
| *** ioni has quit IRC | 11:19 | |
| *** csmart has quit IRC | 11:19 | |
| *** manti has quit IRC | 11:19 | |
| *** fridtjof[m] has quit IRC | 11:20 | |
| noonedeadpunk | jrosser: shouldn't we also merge https://review.opendev.org/c/openstack/ansible-role-python_venv_build/+/771195 ? | 11:24 |
| jrosser | noonedeadpunk: yes we should | 11:26 |
| noonedeadpunk | it seems it was working.... | 11:26 |
| *** masterpe has joined #openstack-ansible | 11:31 | |
| jrosser | feels like some effort needed on a few roles though, unrelated to the pip changes | 11:32 |
| openstackgerrit | Merged openstack/openstack-ansible-os_neutron master: Prevent neutron-l3-agent killing keepalived on restart https://review.opendev.org/c/openstack/openstack-ansible-os_neutron/+/771791 | 11:33 |
| jrosser | should we backport that ^^ | 11:37 |
| *** fridtjof[m] has joined #openstack-ansible | 11:58 | |
| *** ioni has joined #openstack-ansible | 11:58 | |
| openstackgerrit | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_neutron master: L3 agent cleanup_on_shutdown https://review.opendev.org/c/openstack/openstack-ansible-os_neutron/+/771826 | 11:58 |
| *** csmart has joined #openstack-ansible | 11:58 | |
| *** manti has joined #openstack-ansible | 11:58 | |
| noonedeadpunk | I'd say yes | 11:58 |
| noonedeadpunk | at least to V | 11:59 |
| noonedeadpunk | murano needs a lot I think and not sure if ipv6 solved octavia issue? | 12:00 |
| jrosser | i think the ipv6 thing just revealed another problem | 12:00 |
| jrosser | i was wondering if this existing setting is wrong https://github.com/openstack/openstack-ansible/blob/master/tests/roles/bootstrap-host/templates/user_variables_octavia.yml.j2#L30 | 12:01 |
| jrosser | ironic something is broken too | 12:01 |
| noonedeadpunk | oh, yes, but it's broken on tempestconf | 12:02 |
| noonedeadpunk | well, at least setting is present https://github.com/openstack/octavia-tempest-plugin/blob/master/octavia_tempest_plugin/config.py#L143 | 12:03 |
| noonedeadpunk | oh, you mean format... | 12:04 |
| noonedeadpunk | it's wrong :( | 12:04 |
| noonedeadpunk | it should be just `test_with_ipv6: false` indeed | 12:04 |
| jrosser | yeah, so my patch may be not needed | 12:06 |
| *** jbadiapa has quit IRC | 12:06 | |
| jrosser | well depends, if we want optional ipv6 tests or no ipv6 tests | 12:06 |
| noonedeadpunk | it should adjust just different thing :) | 12:06 |
| noonedeadpunk | I'd say we can live without ipv6 | 12:07 |
| jrosser | i was surprised tbh that it worked at all | 12:07 |
| noonedeadpunk | we should ensure that lb is functional and that's it I guess | 12:07 |
| noonedeadpunk | maybe it was valid at some point... unlikely though | 12:07 |
| openstackgerrit | Jonathan Rosser proposed openstack/openstack-ansible master: Disable octavia ipv6 tempest test when the node does not have an ipv6 address https://review.opendev.org/c/openstack/openstack-ansible/+/772379 | 12:09 |
| openstackgerrit | Jonathan Rosser proposed openstack/openstack-ansible master: Disable octavia ipv6 tempest test by fixing config error https://review.opendev.org/c/openstack/openstack-ansible/+/772379 | 12:09 |
| openstackgerrit | Dmitriy Rabotyagov proposed openstack/openstack-ansible master: Install rsync for CentOS bootstrap https://review.opendev.org/c/openstack/openstack-ansible/+/772495 | 12:31 |
| openstackgerrit | Dmitriy Rabotyagov proposed openstack/openstack-ansible master: Remove note about metal/horizon compatability https://review.opendev.org/c/openstack/openstack-ansible/+/771573 | 12:37 |
| jrosser | ironic is still broken with tempestconf | 12:45 |
| jrosser | it now does use the internal endpoint to get the service catalog, but then it uses the public endpoints from the catalog | 12:45 |
| jrosser | and fails again with SSL error in CI | 12:46 |
| jrosser | there seems to be a patch https://opendev.org/osf/python-tempestconf/commit/f146f810695e83d2a8ce0fcdb94ff32e75ebdb20 | 12:46 |
| jrosser | but i do really fail to understand, --insecure passed where...... | 12:47 |
| openstackgerrit | Jonathan Rosser proposed openstack/openstack-ansible-os_tempest master: Use internal endpoint for tempestconf and respect tempest_keystone_interface_insecure https://review.opendev.org/c/openstack/openstack-ansible-os_tempest/+/769966 | 12:58 |
| *** rpittau|bbl is now known as rpittau | 12:59 | |
| *** jbadiapa has joined #openstack-ansible | 13:03 | |
| *** rh-jelabarre has joined #openstack-ansible | 13:04 | |
| *** macz_ has joined #openstack-ansible | 13:45 | |
| *** macz_ has quit IRC | 13:50 | |
| openstackgerrit | Merged openstack/openstack-ansible-galera_server master: Bring db setup vars in line with other roles https://review.opendev.org/c/openstack/openstack-ansible-galera_server/+/771652 | 13:57 |
| *** spatel has joined #openstack-ansible | 14:02 | |
| *** MickyMan77 has quit IRC | 14:02 | |
| *** macz_ has joined #openstack-ansible | 14:06 | |
| *** MickyMan77 has joined #openstack-ansible | 14:08 | |
| *** macz_ has quit IRC | 14:11 | |
| *** SiavashSardari has quit IRC | 14:11 | |
| spatel | noonedeadpunk: or jrosser I am dealing with very bizarre issue with RC1 release, whenever i add new compute node and run playbook os-nova-install.yml or os-neutron-install.yml i hit this error http://paste.openstack.org/show/801986/ | 14:11 |
| spatel | Do you guys have any history around this issue? | 14:12 |
| spatel | workaround is to re-run -e venv_rebuild=yes | 14:12 |
| noonedeadpunk | my guess would be that something drops your contraints from repo container | 14:12 |
| noonedeadpunk | maybe 2 running lsyncd on different containers? | 14:12 |
| jrosser | i would say some sort of partial upgrade odd behaviour, look at the versions | 14:13 |
| jrosser | nova==22.0.2.dev5 and your venv is /openstack/venvs/nova-22.0.0.0rc1 | 14:13 |
| spatel | hmm, i thought about that so i remove 2 repo and kept just single repo in LB | 14:13 |
| jrosser | release candidate vs. stable/victoria as a guess | 14:14 |
| spatel | jrosser: i am also confused about that so not sure from where this is coming | 14:14 |
| jrosser | did you forget to bootstrap-ansible.sh after moving the branch forward, something like that? | 14:14 |
| spatel | i am 100% sure i checkout tag when i was deploying OSA and never touch it after that | 14:15 |
| jrosser | what SHA do you have in openstack-ansible repo? | 14:15 |
| spatel | let me check | 14:16 |
| *** dotnetted_alt has joined #openstack-ansible | 14:16 | |
| *** dotnetted has quit IRC | 14:17 | |
| spatel | git log showing commit ac43c0ce9442184c7041146c6d4fddd1fcbf9d36 (HEAD, tag: 22.0.0.0rc1) | 14:17 |
| spatel | is that what you asking? | 14:18 |
| admin0 | from which variable can i update/overrride the default lxcbr0 ip range used in openstack to something else ( that is not used in the network ) that is conflicting | 14:19 |
| jrosser | spatel: try /opt/ansible-runtime/bin/python setup.py --version | 14:20 |
| spatel | 22.0.0.0rc1 | 14:20 |
| jrosser | hmm | 14:20 |
| *** sshnaidm|ruck is now known as sshnaidm|afk | 14:20 | |
| spatel | it always works with -e venv_rebuild=yes | 14:21 |
| spatel | let me check lsyncd stuff | 14:24 |
| jrosser | also check the wheel build log on the repo server to see what actual version of the nova wheel it built, and check the constraints file that is shown in your paste too that calls it up | 14:26 |
| spatel | noonedeadpunk: hey, does lsyncd run on all 3 repo container right? | 14:26 |
| jrosser | only on the first iirc | 14:26 |
| spatel | that is what i am seeing its only running on first container | 14:26 |
| jrosser | the wheel build should happen on the first, and the results be then rsync out to the rest | 14:26 |
| noonedeadpunk | maybe build host !=lsyncd host? | 14:27 |
| jrosser | that would certainly make things go odd | 14:27 |
| jrosser | spatel: check the ansible log for which host is delegated to during python_venv_build role wheel builds | 14:28 |
| jrosser | make sure that is the same on as you see lsyncd running on | 14:28 |
| spatel | in my LB repo-1 has high priority and that one is running lsyncd service | 14:28 |
| spatel | checking logs on repo-1 | 14:29 |
| jrosser | it's not really to do with the loadbalancer | 14:29 |
| spatel | if you want i can disable other repo-2 and 3 | 14:29 |
| spatel | Ok i got what you saying its on playbook to pick repo | 14:29 |
| jrosser | there are two sides to it, wheels are built in one place and should be rsynced to the others | 14:29 |
| jrosser | then all repo servers have the content to serve via the LB | 14:30 |
| spatel | should i run playbook with -vvvv to find out which repo its picking also tailing logs on repo to see what is going on /var/log/python_wheel_build.log | 14:31 |
| spatel | jrosser: on repo-2/3 file python_wheel_build.log doesn't existing so its clear repo-1 building wheels | 14:32 |
| jrosser | ok, then you should check the lsyncd log on repo-1 | 14:33 |
| spatel | so far no error but let me tail logs and run playbook to see | 14:35 |
| spatel | jrosser: tell me one thing, why lsyncd is important here if i shutdown repo-2 and repo-3 ? | 14:35 |
| spatel | then lsyncd has no role right? | 14:35 |
| jrosser | true | 14:35 |
| spatel | if i disable repo-2/3 in my F5 then no request will land there also. | 14:36 |
| spatel | jrosser: my os-releases/22.0.0.0rc1/nova-22.0.0.0rc1-constraints.txt has nova==22.0.2.dev5 | 14:38 |
| spatel | does that normal to you? | 14:38 |
| jrosser | spatel: the latest tag on nova is https://github.com/openstack/nova/tags 22.0.1 | 14:39 |
| jrosser | so it is reasonable that a commit ahead of that is 22.0.2.dev<N-commits-ahead> | 14:40 |
| spatel | in my lab i am running 22.0.0.0b2 and i am not seeing any issue there, its working fine. | 14:40 |
| jrosser | now you should check the pythin_venv_build log on the host that failed, there will be lots more detail than the ansible log | 14:41 |
| spatel | Let me find fresh file of pythin_venv_build.log from compute host | 14:41 |
| *** dave-mccowan has joined #openstack-ansible | 14:44 | |
| *** dave-mccowan has quit IRC | 14:46 | |
| spatel | jrosser: whole logs files look good so far and in end it barf with this error - http://paste.openstack.org/show/801989/ | 14:55 |
| spatel | ERROR: Could not find a version that satisfies the requirement nova==22.0.2.dev5 | 14:56 |
| spatel | very odd that its saying not able to find so where actually its trying to find, does it searching in some kind of array to match? | 15:01 |
| spatel | hmm! if you check logs Found link https: is searching for each nova version one by one | 15:04 |
| spatel | and matching hash | 15:05 |
| jrosser | yes you have to read through the log carefully and see if it finds the one from the repo server | 15:05 |
| jrosser | then try to figure out why pip decided that was not a suitable candidate | 15:05 |
| jrosser | you should have requirements/constraints files dropped on the compute host which are inputs to pip | 15:06 |
| spatel | that file isn't there - /openstack/venvs/nova-22.0.0.0rc1/ | 15:07 |
| spatel | playbook deleting itself when build fails | 15:07 |
| spatel | it was there while build was running | 15:07 |
| *** sshnaidm|afk is now known as sshnaidm|ruck | 15:07 | |
| spatel | its interesting venv_rebuild=yes not complaining about anything and it works | 15:08 |
| MickyMan77 | Hi all, I have a panic issue. the galera-container is not able to start up mysql... | 15:09 |
| spatel | pip is trying to find 22.0.2 which is not existing anywhere because latest version is 22.0.1 | 15:10 |
| spatel | how that version endup in nova-22.0.0.0rc1-constraints.txt | 15:11 |
| openstackgerrit | Andrew Bonney proposed openstack/openstack-ansible-os_neutron stable/victoria: Prevent neutron-l3-agent killing keepalived on restart https://review.opendev.org/c/openstack/openstack-ansible-os_neutron/+/772538 | 15:20 |
| openstackgerrit | Merged openstack/openstack-ansible-os_trove master: Move trove pip packages from constraints to requirements https://review.opendev.org/c/openstack/openstack-ansible-os_trove/+/772299 | 15:21 |
| openstackgerrit | Andrew Bonney proposed openstack/openstack-ansible-os_octavia master: Add variable to override keypair setup python interpreter https://review.opendev.org/c/openstack/openstack-ansible-os_octavia/+/772559 | 15:23 |
| *** gary_perkins has quit IRC | 15:28 | |
| jrosser | spatel: the python_venv_build role creates those constraints files, and the contents is based on the actual wheels that got built | 15:29 |
| jrosser | so if a wheel for 22.0.2.dev5 was built, thats what gets written to the constraints | 15:29 |
| openstackgerrit | Andrew Bonney proposed openstack/openstack-ansible master: Remove localhost references from galera docs following bind change https://review.opendev.org/c/openstack/openstack-ansible/+/772560 | 15:29 |
| jrosser | comment out the tasks that delete the constraints files on venv build failure, then take a look at them | 15:30 |
| spatel | sounds good | 15:30 |
| spatel | let me try that | 15:30 |
| spatel | i thought playbook download constraints file directly from repo server so it should be same what we have on repo | 15:31 |
| openstackgerrit | Dmitriy Rabotyagov proposed openstack/ansible-hardening master: Make possible to avoid aide installation https://review.opendev.org/c/openstack/ansible-hardening/+/772561 | 15:32 |
| openstackgerrit | Merged openstack/openstack-ansible-os_rally master: Ensure that the rally venv build is self contained https://review.opendev.org/c/openstack/openstack-ansible-os_rally/+/771421 | 15:33 |
| spatel | jrosser: i can see all 3 file on compute machine constraints.txt, global-constraints.txt & requirements.txt and they are similar to what we have on repo server | 15:41 |
| openstackgerrit | Merged openstack/openstack-ansible-os_magnum master: Move magnum pip packages from constraints to requirements https://review.opendev.org/c/openstack/openstack-ansible-os_magnum/+/772285 | 15:58 |
| noonedeadpunk | #startmeeting openstack_ansible_meeting | 16:00 |
| openstack | Meeting started Tue Jan 26 16:00:43 2021 UTC and is due to finish in 60 minutes. The chair is noonedeadpunk. Information about MeetBot at http://wiki.debian.org/MeetBot. | 16:00 |
| openstack | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 16:00 |
| *** openstack changes topic to " (Meeting topic: openstack_ansible_meeting)" | 16:00 | |
| openstack | The meeting name has been set to 'openstack_ansible_meeting' | 16:00 |
| noonedeadpunk | #topic office hours | 16:00 |
| *** openstack changes topic to "office hours (Meeting topic: openstack_ansible_meeting)" | 16:01 | |
| noonedeadpunk | o/ | 16:01 |
| jrosser | hello | 16:01 |
| *** macz_ has joined #openstack-ansible | 16:02 | |
| noonedeadpunk | So, regarding renos. First of all we need to stop publishing new ones. I placed bunch of PRs to cover that. Once that will be done, I will go to infra team regarding removing already published ones | 16:03 |
| *** macz_ has joined #openstack-ansible | 16:03 | |
| noonedeadpunk | which means that we can just abandon changes like https://review.opendev.org/c/openstack/openstack-ansible-os_mistral/+/768663 | 16:03 |
| jrosser | yeah, we should do that.... the only one (if its there?) which should stay is on openstack-ansible i guess | 16:04 |
| *** gshippey has joined #openstack-ansible | 16:04 | |
| noonedeadpunk | yeah. Also I'm not sure if we should leave for ansible-hardening... | 16:04 |
| noonedeadpunk | I've pushed PR but now I'm not so sure | 16:05 |
| noonedeadpunk | considering repo is also tagless, then probably yes... | 16:05 |
| jrosser | ok well -W any you are not sure about | 16:05 |
| noonedeadpunk | that was the only one I think. | 16:07 |
| jrosser | i think i may have figured out this tempestconf stuff | 16:07 |
| noonedeadpunk | they are kind of broken anyway there | 16:07 |
| noonedeadpunk | oh, rly? I tried to push some patch but it was not the cause https://review.opendev.org/c/openstack/openstack-ansible-os_tempest/+/769966 | 16:08 |
| noonedeadpunk | or it ws not the only one | 16:08 |
| jrosser | yes i fiddled with that today | 16:08 |
| noonedeadpunk | aha, yes, just noticed that | 16:09 |
| jrosser | just seems i make silly error with the command | 16:09 |
| jrosser | [Errno 2] No such file or directory: '/root/workspace/etc/profile.yaml --insecure False' | 16:09 |
| jrosser | like it takes the whole thing as the --profile parameter, which is strange | 16:10 |
| *** jamesdenton has quit IRC | 16:11 | |
| *** jamesden_ has joined #openstack-ansible | 16:11 | |
| noonedeadpunk | maybe it's because of "" | 16:11 |
| jrosser | also https://opendev.org/openstack/openstack-ansible-os_tempest/src/branch/master/defaults/main.yml#L228 kind of not what i expected either | 16:11 |
| noonedeadpunk | since they start before profile and end at the end | 16:11 |
| jrosser | there must be a better variable for that | 16:12 |
| noonedeadpunk | we use keystone_service_internaluri_insecure everywhere | 16:13 |
| noonedeadpunk | or what do you mean& | 16:13 |
| jrosser | i think i need a variable that talks about the external IP | 16:13 |
| noonedeadpunk | I think we should just make tempestconf use internalurl for interaction? | 16:14 |
| jrosser | well, your patch made it do that | 16:14 |
| jrosser | it gets the service catalog from the internal endpoint | 16:15 |
| jrosser | then uses the public entries in the catalog itself by the look of it | 16:15 |
| noonedeadpunk | uh.... | 16:15 |
| jrosser | i think that may be what is happening here http://paste.openstack.org/show/801994/ | 16:16 |
| noonedeadpunk | there sould be some extra I guess to select internal instead of public | 16:16 |
| noonedeadpunk | actually the same issue spatel reported about senlin | 16:16 |
| noonedeadpunk | that it connects to keystone through internal but takes public endpoints from catalog | 16:16 |
| jrosser | i can't otherwise see how it has discovered the IP of eth0 | 16:17 |
| spatel | Yes, its known issue and may required senlin code change | 16:17 |
| noonedeadpunk | I'm pretty sure you're right about tempestconf picking up public endpoint | 16:18 |
| jrosser | anyway, my hope was that adding --insecure would make it not worry about the certificate | 16:18 |
| jrosser | i think it may be legitimate behaviour as tempest kind of pretends to be an end user | 16:18 |
| noonedeadpunk | yeah | 16:18 |
| noonedeadpunk | will see... | 16:18 |
| spatel | senlin acting like end-user and using all public endpoint by default | 16:18 |
| jrosser | theres is two things, making the tempestconf discovery not worry about the cert | 16:18 |
| jrosser | then it has to propery write out a tempest.conf that *also* doesnt worry about the cert for the actual tests | 16:19 |
| jrosser | seems only recently this is possible https://opendev.org/osf/python-tempestconf/commit/f146f810695e83d2a8ce0fcdb94ff32e75ebdb20 | 16:19 |
| noonedeadpunk | ok, we can set verify: false | 16:20 |
| *** lkoranda has quit IRC | 16:20 | |
| jrosser | do you know where we would do that? | 16:21 |
| jrosser | the documentation for this is kind of sparse | 16:21 |
| openstackgerrit | Jonathan Rosser proposed openstack/openstack-ansible-os_tempest master: Use internal endpoint for tempestconf and respect tempest_keystone_interface_insecure https://review.opendev.org/c/openstack/openstack-ansible-os_tempest/+/769966 | 16:22 |
| noonedeadpunk | maybe add to tempest_tempestconf_profile ? | 16:22 |
| jrosser | as far as i can see you can't specify this in the profile | 16:23 |
| jrosser | i think that this must be picking up a clouds.yaml from somewhere | 16:24 |
| jrosser | anyway, we maybe should move on a bit? | 16:25 |
| noonedeadpunk | where does it takes auth creds from? | 16:25 |
| noonedeadpunk | clouds.yaml? | 16:25 |
| noonedeadpunk | yeah, let's move | 16:25 |
| jrosser | andrewbonney is doing a U->V upgrade in our lab today | 16:25 |
| noonedeadpunk | We've done T->V in lab and it went beautiful | 16:26 |
| noonedeadpunk | Planning do it in prod next week | 16:26 |
| jrosser | we have a bunch of issues and will make some patches | 16:26 |
| jrosser | actually some already done | 16:26 |
| jrosser | maybe most surprisng was something during setup-hosts restarting all the api containers at the same time | 16:27 |
| MickyMan77 | jrosser: can you help out, I'm not able to start, stop or restart the mysql in the galera_container. It just hanging.. I can't found any log about the issue... | 16:27 |
| jrosser | some releasenotes may be missing too, like bind-to-mgmt means rabbitmq containers need /etc/hosts fixing up a bit | 16:29 |
| openstackgerrit | Gaudenz Steinlin proposed openstack/openstack-ansible-os_cinder stable/ussuri: Define credentials for nova interaction https://review.opendev.org/c/openstack/openstack-ansible-os_cinder/+/772539 | 16:29 |
| openstackgerrit | Merged openstack/openstack-ansible master: Use TCP mode for console if SSL is configured https://review.opendev.org/c/openstack/openstack-ansible/+/574153 | 16:30 |
| jrosser | did we have a bug / explanation for the designate pool UUID issue? | 16:30 |
| andrewbonney | jrosser: fwiw our designate issue is this one: https://bugs.launchpad.net/designate/+bug/1897936 | 16:31 |
| openstack | Launchpad bug 1897936 in Designate "Pool update fails when zones exist" [Undecided,Fix released] - Assigned to Mark Goddard (mgoddard) | 16:31 |
| noonedeadpunk | No, I can't remeber if we have one | 16:31 |
| noonedeadpunk | but it was quite silly that we used config param that was not preset in designate anymore | 16:32 |
| noonedeadpunk | what caused containers to restart? | 16:32 |
| noonedeadpunk | have you upgraded to focal? | 16:33 |
| admin0 | MickyMan77, is the container broken or only mysql ? | 16:33 |
| spatel | I thought designate issue has been resolved, i have deployed couple of time with UUID patch and no issue so far | 16:33 |
| admin0 | if container is broken, is it only 1 container, or 3 containers ( if you are in HA setup) | 16:33 |
| jrosser | andrewbonney: looks like that patch was reverted https://review.opendev.org/c/openstack/designate/+/755429 | 16:33 |
| noonedeadpunk | btw is it smth you mentioned about rabbitmq? https://bugs.launchpad.net/openstack-ansible/+bug/1824857 | 16:35 |
| openstack | Launchpad bug 1824857 in openstack-ansible "Rabbitmq join cluster fail" [Undecided,New] | 16:35 |
| openstackgerrit | Merged openstack/openstack-ansible-apt_package_pinning master: [reno] Stop publishing release notes https://review.opendev.org/c/openstack/openstack-ansible-apt_package_pinning/+/772007 | 16:36 |
| jrosser | yes, as part of bind-to-mgmt we had 3 patches about that | 16:36 |
| *** gary_perkins has joined #openstack-ansible | 16:36 | |
| noonedeadpunk | have we missed smth from V? | 16:36 |
| prometheanfire | has anyone upgraded from ussuri to victoria? | 16:37 |
| jrosser | noonedeadpunk: currently this has to be dealt with manually https://review.opendev.org/c/openstack/openstack-ansible-openstack_hosts/+/670706 https://review.opendev.org/c/openstack/openstack-ansible-lxc_container_create/+/670705 https://review.opendev.org/c/openstack/openstack-ansible/+/670392 | 16:37 |
| jrosser | prometheanfire: we discuss this right now for the last 10 mins :) | 16:38 |
| *** gyee has joined #openstack-ansible | 16:40 | |
| noonedeadpunk | I think https://review.opendev.org/c/openstack/openstack-ansible-openstack_hosts/+/670706 should not be relevant now? | 16:40 |
| noonedeadpunk | or dunno... | 16:40 |
| noonedeadpunk | not quite understand why issue is raised now... | 16:40 |
| jrosser | i think the activation of bind-to-mgmt across everything comes in for V | 16:41 |
| noonedeadpunk | we now generate hosts file quite differently with blockinfile | 16:42 |
| jrosser | that is true | 16:42 |
| jrosser | do you have a link to that patch? | 16:42 |
| noonedeadpunk | so we just need to drop everything except this block? | 16:42 |
| noonedeadpunk | https://opendev.org/openstack/openstack-ansible-openstack_hosts/commit/c64e1caf72c20a2ffcce7b1d92e8b8cc8093a808 | 16:42 |
| jrosser | right so even blockinfile won't remove stuff thats wrong | 16:42 |
| jrosser | i think our issue was the hosts files are long lived and had accumulated stuff from many releases | 16:43 |
| noonedeadpunk | that';s interesting. I think I might have exactly the same issue | 16:44 |
| prometheanfire | jrosser: good timing :D | 16:45 |
| noonedeadpunk | so eventually dropping all except generated block should help it? | 16:45 |
| jrosser | maybe a bit too much to assume the deployer has not put things in there | 16:46 |
| jrosser | not sure OSA 'owns' then whole file | 16:46 |
| openstackgerrit | Merged openstack/openstack-ansible-plugins master: [reno] Stop publishing release notes https://review.opendev.org/c/openstack/openstack-ansible-plugins/+/772054 | 16:48 |
| noonedeadpunk | nah, ofc it's not what I'm thinking to script, but actually how it might be solved _when_ we will face it here:) | 16:49 |
| jrosser | i think it was a trivial fix manually once we figured what was going on | 16:50 |
| noonedeadpunk | yeah... | 16:52 |
| noonedeadpunk | I'm still not sure what caused bug https://bugs.launchpad.net/openstack-ansible/+bug/1824857 since what you said should not be the issue for clen deployments | 16:53 |
| openstack | Launchpad bug 1824857 in openstack-ansible "Rabbitmq join cluster fail" [Undecided,New] | 16:53 |
| noonedeadpunk | Will try to set up 3 nodes sandbox tomorrow | 16:53 |
| openstackgerrit | Merged openstack/openstack-ansible-lxc_container_create master: [reno] Stop publishing release notes https://review.opendev.org/c/openstack/openstack-ansible-lxc_container_create/+/772013 | 16:54 |
| * jrosser just looking at bugs | 16:54 | |
| noonedeadpunk | https://bugs.launchpad.net/openstack-ansible/+bug/1911482 has come here in IRC and said that it was issue in inventory or smth | 16:58 |
| openstack | Launchpad bug 1911482 in openstack-ansible "neutron-l3-agent broken after train upgrade" [Undecided,New] | 16:58 |
| noonedeadpunk | I'm wondering if we can move it to incomplete or invalid... | 16:58 |
| jrosser | comment "resolved via IRC" and incomplete sounds good | 17:00 |
| noonedeadpunk | Should we do anything regarding https://bugs.launchpad.net/openstack-ansible/+bug/1877421 ? | 17:01 |
| openstack | Launchpad bug 1877421 in openstack-ansible "Cinder-volume is not able to recognize a ceph cluster on OpenStack Train." [Undecided,Confirmed] | 17:01 |
| jrosser | oh, | 17:05 |
| *** rh-jelabarre has quit IRC | 17:06 | |
| jrosser | wasnt there something magical about RBD, in that you couldnt ever have ceph@RBD becasue it's some kind of keyword? | 17:06 |
| jrosser | there was a huge long irc thread about this some time ago | 17:06 |
| noonedeadpunk | yeah, I can recall smth like that | 17:06 |
| jrosser | as soon as you use ceph@some-other-pool-name it's all ok | 17:06 |
| noonedeadpunk | I'm wondering if we have defined smth like this in docs? | 17:07 |
| noonedeadpunk | oh.... | 17:08 |
| noonedeadpunk | #endmeeting | 17:08 |
| *** openstack changes topic to "Launchpad: https://launchpad.net/openstack-ansible || Weekly Meetings: https://wiki.openstack.org/wiki/Meetings/openstack-ansible || Review Dashboard: http://bit.ly/osa-review-board-v3" | 17:08 | |
| openstack | Meeting ended Tue Jan 26 17:08:16 2021 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 17:08 |
| openstack | Minutes: http://eavesdrop.openstack.org/meetings/openstack_ansible_meeting/2021/openstack_ansible_meeting.2021-01-26-16.00.html | 17:08 |
| openstack | Minutes (text): http://eavesdrop.openstack.org/meetings/openstack_ansible_meeting/2021/openstack_ansible_meeting.2021-01-26-16.00.txt | 17:08 |
| openstack | Log: http://eavesdrop.openstack.org/meetings/openstack_ansible_meeting/2021/openstack_ansible_meeting.2021-01-26-16.00.log.html | 17:08 |
| noonedeadpunk | yeah, we have that in docs | 17:09 |
| noonedeadpunk | but we have it defined in aio as well... | 17:09 |
| openstackgerrit | Merged openstack/openstack-ansible-os_cloudkitty master: [reno] Stop publishing release notes https://review.opendev.org/c/openstack/openstack-ansible-os_cloudkitty/+/772026 | 17:09 |
| noonedeadpunk | https://opendev.org/openstack/openstack-ansible/src/branch/master/tests/roles/bootstrap-host/tasks/prepare_ceph.yml#L93-L102 | 17:10 |
| jrosser | i wonder how that works in CI | 17:11 |
| noonedeadpunk | or you can't name it as "ceph"... | 17:11 |
| noonedeadpunk | uh | 17:12 |
| jrosser | mine are rbd_<storagetype> with hdd/ssd/nvme.... | 17:13 |
| jrosser | in cinder_backends | 17:13 |
| noonedeadpunk | I also have SSD/HDD/STOR | 17:14 |
| noonedeadpunk | ebbex: maybe you have some newer context? | 17:15 |
| noonedeadpunk | for https://bugs.launchpad.net/openstack-ansible/+bug/1877421 | 17:15 |
| openstack | Launchpad bug 1877421 in openstack-ansible "Cinder-volume is not able to recognize a ceph cluster on OpenStack Train." [Undecided,Confirmed] | 17:15 |
| openstackgerrit | Merged openstack/openstack-ansible-tests master: [reno] Stop publishing release notes https://review.opendev.org/c/openstack/openstack-ansible-tests/+/772060 | 17:19 |
| openstackgerrit | Jonathan Rosser proposed openstack/openstack-ansible-os_horizon master: Move horizon pip packages from constraints to requirements https://review.opendev.org/c/openstack/openstack-ansible-os_horizon/+/770283 | 17:20 |
| *** rh-jelabarre has joined #openstack-ansible | 17:22 | |
| *** rh-jelabarre has quit IRC | 17:23 | |
| *** rh-jelabarre has joined #openstack-ansible | 17:23 | |
| *** rpittau is now known as rpittau|afk | 17:27 | |
| spatel | noonedeadpunk: what -e venv_rebuild=yes does behind the scenes ? | 17:27 |
| spatel | I think i am done with debug, not able to find why its not able to find version but when i use venv_rebuild it does find version | 17:28 |
| spatel | or better i should upgrade RC1 to RC2 ? | 17:30 |
| jrosser | can you paste the relevant part of the python_venv_build log from the compute host? | 17:31 |
| noonedeadpunk | venv_rebuild force re-creates contriaints files on repo containers and rebuilds wheels and recreate venvs | 17:31 |
| spatel | Yes let me do that, log file is very very log so let me find out how i can share | 17:35 |
| jrosser | only the last part where it tries and fails to do nova | 17:36 |
| jrosser | the sections should be clear where it deals with each package | 17:36 |
| spatel | ok | 17:36 |
| spatel | jrosser: here is the sample - http://paste.openstack.org/show/802008/ | 17:40 |
| spatel | I am having same issue with nova/neutron both (both saying not able to find version but if i run venv_rebuild=yes works) | 17:41 |
| noonedeadpunk | do you have wheel built up in /var/www/repo/os-releases/22.0.0.0rc1/ubuntu-20.04-x86_64/ inside repo containers? | 17:44 |
| spatel | let me show you content of that folder | 17:45 |
| jrosser | also do wget on http://10.65.0.16:8181/os-releases/22.0.0.0rc1/ubuntu-20.04-x86_64 | 17:45 |
| spatel | http://paste.openstack.org/show/802009/ | 17:45 |
| jrosser | and see if the thing that it says is not available (like neutron==17.1.0.dev119) is listed in what you get back | 17:45 |
| spatel | yes i can wget all file from that repo | 17:46 |
| noonedeadpunk | hm, I think it should containe neutron, but it's not | 17:46 |
| jrosser | in your last paste there is no... | 17:46 |
| jrosser | ^that | 17:46 |
| spatel | i don't have anything start with neutron* in ubuntu-20.04-x86_64 foler | 17:46 |
| jrosser | so something is wrong with the wheel build | 17:46 |
| openstackgerrit | Merged openstack/ansible-role-python_venv_build master: Replace pip requirement source specs with the package name for built wheels https://review.opendev.org/c/openstack/ansible-role-python_venv_build/+/771195 | 17:47 |
| jrosser | the error from the venv build is absolutely correct | 17:47 |
| spatel | hmm | 17:47 |
| jrosser | from the POV of pip on the compute node, the wheel of that version is not availble at the places its been given to look | 17:47 |
| noonedeadpunk | actually these dev can't be found anywhere if there are no wheels built | 17:48 |
| jrosser | spatel: on the repo server, do you have /var/log/python_wheel_build.log? | 17:48 |
| spatel | yes | 17:49 |
| spatel | its very long and big | 17:49 |
| jrosser | ok, so just rename that file to something new | 17:49 |
| spatel | do you want me to remove it and re-run playbook with fresh error? | 17:49 |
| spatel | ok | 17:49 |
| jrosser | then re-run just one of your failing things | 17:49 |
| spatel | on it | 17:49 |
| spatel | running playbook | 17:50 |
| jrosser | you should see the python_venv_build role do the wheel build | 17:51 |
| jrosser | see if those tasks do anything, or are all skipped | 17:51 |
| jrosser | if they do something, the task header will show if they are delegated to another host (hopefully the repo server) | 17:51 |
| *** maharg101 has quit IRC | 17:55 | |
| spatel | that log file is massive | 17:55 |
| spatel | i trying to find keyword skipped and found nothing | 17:56 |
| spatel | hold on damn it i was looking into python_venv_build.log file not wheel :) | 17:57 |
| jrosser | lets say you re-ran nova playbook | 17:57 |
| spatel | jrosser: it didn't created any python_wheel_build.log file | 17:57 |
| spatel | I am on repo-1 | 17:57 |
| spatel | very odd | 17:58 |
| jrosser | right, so check the ansible log if the wheel build steps were skipped, or where executed | 17:58 |
| openstackgerrit | Merged openstack/openstack-ansible-memcached_server master: [reno] Stop publishing release notes https://review.opendev.org/c/openstack/openstack-ansible-memcached_server/+/772015 | 17:58 |
| spatel | jrosser: here its full output of playbook - http://paste.openstack.org/show/802013/ | 17:59 |
| jrosser | TASK [python_venv_build : Build wheels for the packages to be installed into the venv] *********************************************************************** | 17:59 |
| jrosser | skipping: [ostack-phx-comp-sriov-1-7] | 17:59 |
| jrosser | ^ there | 17:59 |
| jrosser | did you do -e venv_rebuild=true? | 18:00 |
| jrosser | which if i recall, is when it works? | 18:01 |
| jrosser | spatel: i have to go, but you see what we are doing | 18:02 |
| jrosser | start with the error message and work back | 18:02 |
| spatel | thanks let me see what i can do | 18:02 |
| spatel | latest connect later | 18:03 |
| openstackgerrit | Merged openstack/openstack-ansible-openstack_openrc master: [reno] Stop publishing release notes https://review.opendev.org/c/openstack/openstack-ansible-openstack_openrc/+/772019 | 18:04 |
| openstackgerrit | Merged openstack/openstack-ansible-os_heat master: [reno] Stop publishing release notes https://review.opendev.org/c/openstack/openstack-ansible-os_heat/+/772031 | 18:04 |
| openstackgerrit | Merged openstack/openstack-ansible-os_keystone master: [reno] Stop publishing release notes https://review.opendev.org/c/openstack/openstack-ansible-os_keystone/+/772034 | 18:11 |
| openstackgerrit | Merged openstack/openstack-ansible-rabbitmq_server master: [reno] Stop publishing release notes https://review.opendev.org/c/openstack/openstack-ansible-rabbitmq_server/+/772055 | 18:11 |
| openstackgerrit | Merged openstack/openstack-ansible-rsyslog_client stable/victoria: Remove examples for variables which no longer exist https://review.opendev.org/c/openstack/openstack-ansible-rsyslog_client/+/772223 | 18:12 |
| openstackgerrit | Merged openstack/openstack-ansible-os_placement master: [reno] Stop publishing release notes https://review.opendev.org/c/openstack/openstack-ansible-os_placement/+/772044 | 18:14 |
| openstackgerrit | Merged openstack/openstack-ansible-galera_server master: [reno] Stop publishing release notes https://review.opendev.org/c/openstack/openstack-ansible-galera_server/+/772012 | 18:19 |
| openstackgerrit | Merged openstack/openstack-ansible-os_masakari master: [reno] Stop publishing release notes https://review.opendev.org/c/openstack/openstack-ansible-os_masakari/+/772037 | 18:19 |
| openstackgerrit | Merged openstack/openstack-ansible-os_rally master: [reno] Stop publishing release notes https://review.opendev.org/c/openstack/openstack-ansible-os_rally/+/772045 | 18:20 |
| openstackgerrit | Merged openstack/openstack-ansible-repo_server master: [reno] Stop publishing release notes https://review.opendev.org/c/openstack/openstack-ansible-repo_server/+/772056 | 18:20 |
| openstackgerrit | Merged openstack/openstack-ansible-rsyslog_client master: [reno] Stop publishing release notes https://review.opendev.org/c/openstack/openstack-ansible-rsyslog_client/+/772057 | 18:20 |
| spatel | jrosser: i didn't use -e venv_rebuild=true? so question is why Build wheels for the packages to be installed into the venv getting skipped | 18:23 |
| openstackgerrit | Merged openstack/ansible-role-systemd_networkd master: [reno] Stop publishing release notes https://review.opendev.org/c/openstack/ansible-role-systemd_networkd/+/772005 | 18:26 |
| openstackgerrit | Merged openstack/openstack-ansible-os_blazar master: [reno] Stop publishing release notes https://review.opendev.org/c/openstack/openstack-ansible-os_blazar/+/772021 | 18:26 |
| openstackgerrit | Merged openstack/openstack-ansible-os_glance master: [reno] Stop publishing release notes https://review.opendev.org/c/openstack/openstack-ansible-os_glance/+/772028 | 18:26 |
| openstackgerrit | Merged openstack/openstack-ansible-os_mistral master: [reno] Stop publishing release notes https://review.opendev.org/c/openstack/openstack-ansible-os_mistral/+/772038 | 18:26 |
| openstackgerrit | Merged openstack/openstack-ansible-os_swift master: [reno] Stop publishing release notes https://review.opendev.org/c/openstack/openstack-ansible-os_swift/+/772049 | 18:26 |
| openstackgerrit | Merged openstack/openstack-ansible-os_magnum master: [reno] Stop publishing release notes https://review.opendev.org/c/openstack/openstack-ansible-os_magnum/+/772035 | 18:32 |
| openstackgerrit | Merged openstack/openstack-ansible-os_manila master: Move manila pip packages from constraints to requirements https://review.opendev.org/c/openstack/openstack-ansible-os_manila/+/772287 | 18:32 |
| openstackgerrit | Merged openstack/openstack-ansible-os_sahara master: [reno] Stop publishing release notes https://review.opendev.org/c/openstack/openstack-ansible-os_sahara/+/772048 | 18:32 |
| openstackgerrit | Merged openstack/openstack-ansible-os_barbican master: [reno] Stop publishing release notes https://review.opendev.org/c/openstack/openstack-ansible-os_barbican/+/772022 | 18:39 |
| openstackgerrit | Merged openstack/openstack-ansible-os_ceilometer master: [reno] Stop publishing release notes https://review.opendev.org/c/openstack/openstack-ansible-os_ceilometer/+/772024 | 18:39 |
| openstackgerrit | Merged openstack/openstack-ansible-os_gnocchi master: [reno] Stop publishing release notes https://review.opendev.org/c/openstack/openstack-ansible-os_gnocchi/+/772030 | 18:39 |
| openstackgerrit | Merged openstack/openstack-ansible-os_tacker master: [reno] Stop publishing release notes https://review.opendev.org/c/openstack/openstack-ansible-os_tacker/+/772050 | 18:39 |
| openstackgerrit | Merged openstack/openstack-ansible-os_horizon master: [reno] Stop publishing release notes https://review.opendev.org/c/openstack/openstack-ansible-os_horizon/+/772032 | 18:46 |
| openstackgerrit | Merged openstack/openstack-ansible-os_mistral master: Move mistral pip packages from constraints to requirements https://review.opendev.org/c/openstack/openstack-ansible-os_mistral/+/772291 | 18:51 |
| openstackgerrit | Merged openstack/openstack-ansible-os_nova master: [reno] Stop publishing release notes https://review.opendev.org/c/openstack/openstack-ansible-os_nova/+/772041 | 18:52 |
| *** miloa has quit IRC | 18:53 | |
| prometheanfire | why are release notes stopping? | 18:55 |
| prometheanfire | ah, integrated reno | 18:56 |
| prometheanfire | I looked at backlog and didn't see U->V upgrade conversations, was it the rabbit cluster join issue? | 18:56 |
| openstackgerrit | Merged openstack/openstack-ansible-os_neutron master: L3 agent cleanup_on_shutdown https://review.opendev.org/c/openstack/openstack-ansible-os_neutron/+/771826 | 19:13 |
| *** andrewbonney has quit IRC | 19:16 | |
| openstackgerrit | Merged openstack/openstack-ansible-os_barbican master: Move barbican pip packages from constraints to requirements https://review.opendev.org/c/openstack/openstack-ansible-os_barbican/+/772260 | 19:22 |
| spatel | noonedeadpunk: quick question i have this "neutron-17.1.0.dev119-py3-none-any.whl" in /var/www/repo/os-releases/22.0.0.0rc1/ubuntu-20.04-x86_64 | 19:25 |
| *** cloudnull has quit IRC | 19:26 | |
| spatel | does os-neutron-install.yml remove this file before running? | 19:26 |
| *** cloudnull has joined #openstack-ansible | 19:26 | |
| noonedeadpunk | no, unless you run with venv_rebuild | 19:37 |
| spatel | hmm! | 19:41 |
| spatel | let me run some experiment to understand why that file disappearing | 19:41 |
| spatel | is it possible lsyncd is removing ? | 19:42 |
| noonedeadpunk | yes, totally | 19:48 |
| noonedeadpunk | that file should be on the container where lsyncd is running. if it on some other host, the lsyncd will drop it | 19:49 |
| *** maharg101 has joined #openstack-ansible | 19:51 | |
| spatel | hmm | 19:53 |
| spatel | let me check | 19:54 |
| spatel | noonedeadpunk: i do have neutron-17.1.0.dev119-py3-none-any.whl in all repo | 19:56 |
| spatel | that is how lsyncd works right | 19:56 |
| *** maharg101 has quit IRC | 19:56 | |
| spatel | it copy files from repo-1 to 2 and 3 | 19:56 |
| noonedeadpunk | yep | 19:56 |
| noonedeadpunk | but it should not be deleted | 19:56 |
| spatel | hmm! let me keep digging... i think i am slowly understanding how these nuts and bolts working here :) | 19:57 |
| noonedeadpunk | haha | 19:58 |
| spatel | noonedeadpunk: are we going to release rc3 or stable ? | 20:02 |
| spatel | how much work is pending for victoria? | 20:02 |
| spatel | I was thinking to do minor upgrade from rc1 to rc2 or wait for stable one | 20:02 |
| *** simondodsley has quit IRC | 20:14 | |
| *** hindret has quit IRC | 20:14 | |
| *** hindret has joined #openstack-ansible | 20:16 | |
| jrosser | spatel: 22.0.0 is released, i mentioned this the other day | 20:16 |
| *** simondodsley has joined #openstack-ansible | 20:16 | |
| jrosser | rc2 is the same SHA sa 22.0.0 | 20:16 |
| jrosser | *as | 20:16 |
| spatel | so its pretty much stable release right? | 20:17 |
| jrosser | ? | 20:17 |
| jrosser | 22.0.0 is the first stable release tag on the stable/victoria branch, after the previous rc1 | 20:18 |
| spatel | your answered my question | 20:18 |
| jrosser | the release process requires the final non-rc release to be at the same point as the last rc | 20:18 |
| jrosser | and as we had merged fixes on top of rc1, we had to make rc2, then put 22.0.0 tag at the same place as rc2 | 20:18 |
| spatel | got it so 22.0.0 = rc2 | 20:20 |
| jrosser | correct | 20:20 |
| jrosser | it's just a git tag, no magic | 20:20 |
| spatel | :) | 20:21 |
| spatel | I am seeing very strange thing with my issue.. let me explain what is going on | 20:21 |
| spatel | when i use -e venv_rebuild=yes its destroying whole folder in repo /var/www/repo/os-releases/22.0.0.0rc1/ubuntu-20.04-x86_64 | 20:22 |
| spatel | and re-creating again | 20:22 |
| spatel | is that normal behavior? | 20:22 |
| spatel | I think that is what my problem is | 20:23 |
| spatel | So lets if i run os-nova-install.yml -e venv_rebuild=yes it destroying repo folder and recreating but neutron-* whl package is missing, so next time when i run os-neutron-install.yml -e venv_rebuild=yes it destroying repo folder and re-creating without nova-* whl package | 20:25 |
| spatel | that is why i am seeing this issue everytime i add new compute node | 20:26 |
| spatel | jrosser: ^^ | 20:26 |
| spatel | This is like chicken egg issue | 20:26 |
| jrosser | https://github.com/openstack/ansible-role-python_venv_build/blob/master/tasks/python_venv_wheel_build.yml#L43-L54 | 20:32 |
| spatel | it does delete folder but is creating issue for me. | 20:34 |
| spatel | nova deleting neutron and neutron deleting nova whl files | 20:34 |
| jrosser | right, and when you run the role next time it should rebuild the wheels anyway | 20:34 |
| jrosser | because of this https://github.com/openstack/ansible-role-python_venv_build/blob/master/tasks/python_venv_wheel_build.yml#L128 | 20:35 |
| jrosser | if for some reason it does not (like in a paste from earlier) then that condition is not being made true | 20:35 |
| jrosser | so the first link i gave you is where all the files get deleted | 20:36 |
| jrosser | when you re-run it, the files are re-created and the condition for the second link (do the wheel build) will be true | 20:36 |
| jrosser | that should build the wheels and put them in the var | 20:37 |
| jrosser | www/repo directory | 20:37 |
| spatel | yes | 20:37 |
| jrosser | but you see on later runs that the wheels are not present | 20:38 |
| jrosser | so either they are not being built on the repo server that you think they are | 20:38 |
| jrosser | or there is some lsyncd/rsync thing where they are getting deleted because the ordering of the repo servers is somehow messed up | 20:39 |
| jrosser | you will be able to see that perhaps in the lsync log | 20:39 |
| spatel | see - http://paste.openstack.org/show/802020/ | 20:39 |
| spatel | I can see wheels getting builds on repo-1 and then lsyncd copying them to repo-2 and repo-3 | 20:40 |
| spatel | I can see all contents are synced | 20:40 |
| spatel | can i stop lsyncd on repo-1 and try ? | 20:41 |
| spatel | question when i run -e venv_rebuild=yes in that case does it going to rebuild all whl including nova and neutron also right? | 20:44 |
| spatel | I am seeing with os-nova-install.yml its just build nova whl not neutron and that is why i am in that race condition | 20:45 |
| jrosser | so | 20:45 |
| spatel | seems like bug | 20:46 |
| jrosser | look at the condition here https://github.com/openstack/ansible-role-python_venv_build/blob/stable/victoria/tasks/python_venv_wheel_build.yml#L128 | 20:46 |
| jrosser | when is that true? | 20:46 |
| spatel | one of this file - /var/www/repo/os-releases/22.0.0.0rc1/nova-22.0.0.0rc1-requirements.txt | 20:47 |
| spatel | right | 20:48 |
| jrosser | as i read the code now, running venv_rebuild=true will delete the wheels in ubuntu-18.04-x86_64 or whatever the subdirectory is | 20:49 |
| spatel | yes i can see its destroying whole folder | 20:49 |
| jrosser | but it will leave behind everything else except the service you are deploying, because of https://github.com/openstack/ansible-role-python_venv_build/blob/stable/victoria/tasks/python_venv_wheel_build.yml#L49-L52 | 20:50 |
| jrosser | so it seems that if you venv_rebuild=true for nova, it will leave the neutron constraints/requirements/global-constraints files present | 20:50 |
| spatel | they are present there | 20:51 |
| jrosser | which means that next time you come to run neutron install, the wheels are gone but the condition that says to build them is false | 20:51 |
| spatel | when you saying file is changed does that means timestamp or content of files? | 20:52 |
| jrosser | for example _requirement_file is changed | 20:52 |
| jrosser | that is the registered result from this _requirement_file is changed | 20:53 |
| jrosser | oops | 20:53 |
| jrosser | https://github.com/openstack/ansible-role-python_venv_build/blob/stable/victoria/tasks/python_venv_wheel_build.yml#L79-L86 | 20:53 |
| jrosser | if the ansible task is 'changed', i.e the content in the file is different | 20:53 |
| jrosser | now i am really surprised that we don't see this before, because i do venv_rebuild=true all the time in AIO for developing stuff | 20:54 |
| jrosser | but that is generally for just one service | 20:54 |
| spatel | i haven't seen this issue also but suddenly seeing with new deployment | 20:55 |
| jrosser | i would be really interested to know what noonedeadpunk thinks of this | 20:55 |
| spatel | something definitely changed | 20:55 |
| *** dtantsur is now known as dtantsur|afk | 20:55 | |
| spatel | jrosser: i have noticed whenever i run -e venv_rebuild its changing timestamp of nova-22.0.0.0rc1-requirements.txt file | 21:00 |
| spatel | feels like new file | 21:00 |
| jrosser | for which playbook? | 21:00 |
| spatel | nova or neutron any playbook | 21:00 |
| spatel | currently i can see nova-22.0.0.0rc1-requirements.txt file has current timestamp | 21:01 |
| jrosser | thats what i'm trying to explain i think | 21:01 |
| jrosser | for the playbook you run (lets say nova), it will delete those files | 21:01 |
| jrosser | https://github.com/openstack/ansible-role-python_venv_build/blob/stable/victoria/tasks/python_venv_wheel_build.yml#L49-L52 | 21:01 |
| jrosser | but not the ones for any other service | 21:01 |
| jrosser | this is why it seems to work for you when you try nova with venv_rebuild=true | 21:02 |
| jrosser | but then the next service will fail | 21:02 |
| spatel | yes | 21:02 |
| jrosser | the ones that get deleted trigger the wheel build for that service (only) when they are re-created | 21:02 |
| spatel | do you think this is still related to lsync issue? | 21:03 |
| jrosser | nothing to do with lsync | 21:03 |
| spatel | ok | 21:03 |
| jrosser | as far as i can see, it looks really like a logic error in python_venv_build | 21:04 |
| spatel | good if we know what is going on so i stop poking around :) | 21:04 |
| jrosser | so - delete all those .txt files in /var/www/repo/.... and you should be good | 21:04 |
| spatel | lets see if noonedeadpunk has anything to offer us tomorrow | 21:04 |
| jrosser | that will force the wheel build | 21:04 |
| jrosser | without deleting any existing wheels | 21:05 |
| spatel | you want me to delete nova/neutron *.txt file from /var/www/repo ? | 21:05 |
| spatel | and re-run build? | 21:05 |
| jrosser | are you still stuck? | 21:06 |
| spatel | i am not stuck (workaround is -e venv_rebuild=yes) | 21:06 |
| jrosser | ok then leave it | 21:06 |
| spatel | i can add computes but trying to fix this issue so my other team member don't ask why its failing :) | 21:06 |
| spatel | I need to educate them for this issue. | 21:07 |
| jrosser | well i think i would advise not using venv_rebuild=true, or telling people to use it to 'fix' things | 21:07 |
| jrosser | as that will create mythology :) | 21:08 |
| spatel | Yes that is what i m not telling them | 21:08 |
| spatel | i want to fix this issue not following bad practice | 21:08 |
| jrosser | if you want to clean your current deployment remove all those txt files and run without venv_rebuild=true | 21:08 |
| jrosser | i think it will then re-create everything you need | 21:08 |
| spatel | all *txt or just for nova/neutron? | 21:08 |
| jrosser | well, if all the wheels have been deleted then all those files are unhelpful now | 21:09 |
| spatel | let me try with just nova/neutron first and if won't help then delete all | 21:10 |
| spatel | in worst case i will upgrade to 22.0.0 | 21:10 |
| jrosser | yeah, then your compute add should just work | 21:10 |
| spatel | Lets talk to noonedeadpunk tomorrow with fresh mind :) | 21:10 |
| spatel | do you want me to open bug to keep trace of it or we can poke him here ? | 21:10 |
| jrosser | here is good for now | 21:11 |
| spatel | cool! | 21:11 |
| jrosser | thanks for taking the time to look into it | 21:11 |
| *** gshippey has quit IRC | 21:12 | |
| spatel | no worry! i want to make sure we are good before anyone else see this issue :) | 21:12 |
| spatel | thank to you for sticking around with this issue | 21:12 |
| jrosser | no problem :) | 21:12 |
| spatel | jrosser: here is the result of experiment, i deleted nova*.txt and neutron*.txt from repo and re-run without venv_rebuild | 21:20 |
| spatel | successfully able to run both playbook and i can see both nova*.whl and neutron*.whl files | 21:20 |
| spatel | this is good state | 21:21 |
| jrosser | fabulous | 21:21 |
| jrosser | that kind of confirms what is happening | 21:21 |
| spatel | now if i add next compute it won't error out because they can see both whl files | 21:21 |
| spatel | let me prepare next compute node | 21:21 |
| spatel | now if i run venv_rebuild=yes then i will go back to same chicken egg issue | 21:22 |
| jrosser | i think so yes | 21:22 |
| spatel | just confirming | 21:22 |
| spatel | so tomorrow we need to find out how to fix that | 21:23 |
| *** SecOpsNinja has left #openstack-ansible | 21:44 | |
| *** jbadiapa has quit IRC | 22:06 | |
| *** spatel has quit IRC | 22:50 | |
| *** tosky has quit IRC | 23:18 | |
| *** admin0 has quit IRC | 23:39 | |
| *** cshen has quit IRC | 23:48 | |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!