| *** rpittau|afk is now known as rpittau | 07:29 | |
| tosky | hi! In a review which fixes sahara after the removal of cinderclient.v2, it seems the sahara-openstack-ansible-functional doesn't load the change: https://review.opendev.org/c/openstack/sahara/+/802415/ | 08:04 |
|---|---|---|
| tosky | (also, that job deploys on bionic, but I guess that's a different issue) | 10:30 |
| tosky | uhm, maybe we only miss an explicit openstack/sahara in required-projects | 10:40 |
| jrosser | tosky: just looking at this | 10:46 |
| jrosser | tosky: i'm not sure a depends-on is going to work here https://zuul.opendev.org/t/openstack/build/47fcf20e03a9442788ad4340265d83e3/log/logs/openstack/sahara1/python_venv_build.log.txt#1847 | 10:53 |
| jrosser | oh sorry it's not a depends-on, but the thing is the OSA job is testing the os_sahara ansible role and will respect changes in that repo | 10:54 |
| jrosser | but in turn it clones sahara itself from opendev.org rather than use the code prepared by zuul on the test node | 10:54 |
| tosky | jrosser: so basically that job can't be used to test changes | 10:55 |
| tosky | I would argue that it should be possible to tell os_* roles to not clone if the code is available already | 10:56 |
| jrosser | this is a legacy OSA job and we have newer ones which do respect the zuul repos | 10:57 |
| opendevreview | Jonathan Rosser proposed openstack/openstack-ansible master: Add integrated build job to use in sahara repo https://review.opendev.org/c/openstack/openstack-ansible/+/802457 | 11:06 |
| jrosser | tosky: lets see how we get on with https://review.opendev.org/c/openstack/sahara/+/802478 | 11:08 |
| opendevreview | Jonathan Rosser proposed openstack/openstack-ansible master: Add integrated build job to use in sahara repo https://review.opendev.org/c/openstack/openstack-ansible/+/802457 | 11:12 |
| tosky | jrosser: I think you mentioned in the past some plan to have a replacement job that can be used by projects to gate on changes | 11:12 |
| tosky | apart from sahara, which is not exactly the most active one, this could still be important for all the other projects | 11:13 |
| jrosser | yeah - we had the opposite problem where we wanted to test unmerged changes to things like keystone in the context of OSA | 11:13 |
| jrosser | also lots of job failures when cloning everything from opendev.org rather than use the local copy | 11:14 |
| tosky | isn't it the same problem? Test unmerged changes | 11:14 |
| jrosser | it's a fine balance though because our end-users all expect totally deterministic deployments driven by a manifest of git SHA | 11:14 |
| jrosser | so making this all sensible in CI but also representative of real deployments is tricky | 11:14 |
| tosky | right, the default behavior is fine | 11:15 |
| tosky | it's just the possibility of overriding that git SHA with a specific local place | 11:15 |
| jrosser | yes, hopefully in these newer jobs thats all automatic | 11:15 |
| opendevreview | Satish Patel proposed openstack/openstack-ansible-os_neutron master: Set ovn hostname using nodename facts https://review.opendev.org/c/openstack/openstack-ansible-os_neutron/+/802134 | 12:58 |
| opendevreview | Jonathan Rosser proposed openstack/openstack-ansible master: Add integrated build job to use in sahara repo https://review.opendev.org/c/openstack/openstack-ansible/+/802457 | 13:12 |
| jrosser | tosky: looks like good progress on https://review.opendev.org/c/openstack/sahara/+/802478 | 15:02 |
| jrosser | we see it use the local repo for sahara now https://paste.opendev.org/show/807730/ | 15:02 |
| jrosser | however it's not running any actual sahara tests as you see here https://1f13c8c9ef933c99f2a6-66a8f55a1185bbba9ec0e2f41773aeb5.ssl.cf5.rackcdn.com/802478/2/check/openstack-ansible-deploy-aio_sahara_metal-ubuntu-focal/1d358ee/logs/openstack/aio1-utility/stestr_results.html | 15:03 |
| tosky | oh | 15:04 |
| jrosser | it would be really helpful to get some help with a suitable/minimal set of tempest config that we can use | 15:04 |
| jrosser | this is kind of expected though, we need to add another patch OSA side to allow-list some of the tempest tests | 15:04 |
| tosky | all the ones from sahara-tests | 15:04 |
| tosky | I guess one just need to override a variable with the regexp of the tests? Does it work if the tests are defined in a tempest plugin? | 15:05 |
| jrosser | it's quite possible we don't load the plugin yet - i need to take more of a look | 15:05 |
| jrosser | is there any assumption about available services, other than basic like nova/keystone/glance etc... ? | 15:06 |
| tosky | heat is needed | 15:06 |
| opendevreview | Jonathan Rosser proposed openstack/openstack-ansible master: Add heat service when scenario includes sahara https://review.opendev.org/c/openstack/openstack-ansible/+/802535 | 15:11 |
| opendevreview | Jonathan Rosser proposed openstack/openstack-ansible master: Enable tempest tests for sahara https://review.opendev.org/c/openstack/openstack-ansible/+/802551 | 16:06 |
| opendevreview | Jonathan Rosser proposed openstack/openstack-ansible master: Add heat service when scenario includes sahara https://review.opendev.org/c/openstack/openstack-ansible/+/802535 | 16:07 |
| opendevreview | Jonathan Rosser proposed openstack/openstack-ansible master: Enable tempest tests for sahara https://review.opendev.org/c/openstack/openstack-ansible/+/802551 | 16:07 |
| opendevreview | Satish Patel proposed openstack/openstack-ansible-os_neutron master: Set ovn hostname using nodename facts https://review.opendev.org/c/openstack/openstack-ansible-os_neutron/+/802134 | 16:17 |
| spatel | jrosser do you know what noonedeadpunk was trying to say here - https://review.opendev.org/c/openstack/openstack-ansible-os_neutron/+/802135 | 16:17 |
| spatel | because of this patch OVN is failing pretty much on all neutron builds | 16:18 |
| jrosser | he is saying that the metadata service is https (or not) as defined here https://opendev.org/openstack/openstack-ansible/src/branch/master/inventory/group_vars/haproxy/haproxy.yml#L318 | 16:18 |
| jrosser | so it's a variable | 16:18 |
| * jrosser just looking at more detail | 16:20 | |
| spatel | yes, haproxy only enabled https vip but ovn-metadata using http and its failing | 16:20 |
| spatel | so i told in patch use https for ovn | 16:21 |
| jrosser | haproxy can be either, thats his point | 16:21 |
| jrosser | it's a deployer variable to set haproxy_ssl_all_vips true/false as needed | 16:21 |
| jrosser | so you need to make the template for the neutron role respect this setting | 16:21 |
| spatel | but how to tell that in AIO and CI-CD jobs ? shouldn't that be default | 16:22 |
| spatel | what is neutron-metadata services use ? http or https? | 16:22 |
| jrosser | thats the point :) | 16:22 |
| jrosser | it can be either | 16:22 |
| spatel | yes agreed | 16:22 |
| spatel | but if i build AIO right now does it use default http or https? | 16:23 |
| jrosser | you need to make the setting of haproxy_ssl_all_vips set this | 16:23 |
| jrosser | https://opendev.org/openstack/openstack-ansible/src/branch/master/tests/roles/bootstrap-host/templates/user_variables.aio.yml.j2#L269 | 16:23 |
| spatel | This is default true haproxy_ssl_all_vips right? | 16:25 |
| jrosser | the default in the haproxy role is false https://opendev.org/openstack/openstack-ansible-haproxy_server/src/branch/master/defaults/main.yml#L85 | 16:25 |
| jrosser | the AIO overrides that to be true | 16:25 |
| jrosser | this is not just simple answer i can give you straight away | 16:26 |
| spatel | Yes, i just build AIO and all haproxy vips are SSL so assuming default is all SSL | 16:26 |
| spatel | I believe old neutron-metadata-agent also using https API to make a call | 16:27 |
| spatel | But ovn-metdata default using http and that is why its failing so now question is, should we make haproxy_ssl_all_vips: false ? or make change in ovn-metadata to tell use https ? | 16:28 |
| jrosser | this was only changed one month ago https://opendev.org/openstack/openstack-ansible/commit/6e5b0094d52bb5972e3b5d805afc5302f8696d2f | 16:29 |
| spatel | That is why i wonder i haven't seen this issue month ago but suddenly its failing on metadata service :) | 16:29 |
| jrosser | i don't have a straight-away answer because this is one service needing to know about the https/http setup of a different one | 16:32 |
| jrosser | i.e you need to put the right thing in the neutron ovn config based on something to do with nova | 16:32 |
| *** sshnaidm is now known as sshnaidm|afk | 16:34 | |
| jrosser | one way to do this is to make a new variable in os_neutron/defaults/main.yml neutron_nova_metadata_protocol: "{{ nova_metadata_protocol | default(http) }}" | 16:38 |
| jrosser | then use "{{ neutron_nova_metadata_protocol }}" in the template | 16:38 |
| jrosser | and it will pick up the necessary value from here https://opendev.org/openstack/openstack-ansible/src/branch/master/inventory/group_vars/all/nova.yml#L18 | 16:39 |
| jrosser | see that you need to be using something in the neutron role from group_vars/all/... becasue anything specific to nova is not in scope when deploying neutron | 16:40 |
| *** rpittau is now known as rpittau|afk | 16:40 | |
| jrosser | spatel: ^ does this help? | 16:41 |
| spatel | let me understand.. | 16:42 |
| spatel | I got it what you saying.. | 16:44 |
| spatel | first find out if nova-metadata-api is http or https and according tell ovn-metadata to use protocol | 16:44 |
| jrosser | correct | 16:46 |
| jrosser | generally each role should have all the 'knobs and dials' defined in defaults/main.yml, so thats why i think we should add another one | 16:47 |
| jrosser | then in the main openstack-ansible we do 'wiring' to connect global settings like openstack_service_internaluri_proto to all the places in all the roles that need it | 16:48 |
| jrosser | doing things this way you can then override this in a deployment for nova, or neuton, or both, or neither etc etc | 16:48 |
| spatel | +1 | 16:49 |
| jrosser | it's how a lot of the OSA flexibility is implemented in practice by having careful attention to how all these variables connect together | 16:49 |
| spatel | i can understand otherwise hard to manage | 16:53 |
| spatel | jrosser while you here, what is your thought about this? https://review.opendev.org/c/openstack/openstack-ansible-os_neutron/+/802402 | 16:59 |
| spatel | This is centos-8 beast and in 5 month it will be dead anyway. | 16:59 |
| jrosser | well, having to be careful about centos-8 vs centos-8-stream here in terms of ansible version | 17:00 |
| jrosser | i think stream comes up as 8 | 17:00 |
| spatel | centos folks moved all packages to centos-8-stream so openvswitch installation was failing. centos-8.4 has only openvswitch2.13 version left. | 17:01 |
| jrosser | and old style centos is 8.x | 17:01 |
| jrosser | so this is just terrible for version logic in ansible | 17:01 |
| jrosser | so < 8.4 will also be true for stream | 17:01 |
| spatel | agreed but centos-8.4 will be dead by end of 2021 | 17:01 |
| jrosser | i don't know if thats what you intend | 17:01 |
| spatel | my patch pass both centos-8.4 and centos-8-stream | 17:02 |
| jrosser | right sure, in this cycle we need to remove all support for old centos | 17:02 |
| spatel | Yes i would say lets remove centos-8.x and keep focus on stream | 17:02 |
| jrosser | well i don't know - we don't run your patch at all on stream with OVS though? | 17:03 |
| spatel | there is no point to fix 8.4 at this stage | 17:03 |
| spatel | openstack-ansible-deploy-aio_metal-centos-8-stream | 17:03 |
| jrosser | not with OVS though? | 17:03 |
| jrosser | default is linuxbridge | 17:03 |
| spatel | ah! | 17:03 |
| spatel | i know what you saying | 17:04 |
| jrosser | yeah, it makes these version logic operators really nasty | 17:04 |
| spatel | i can test that patch in my lab :) | 17:04 |
| jrosser | will be *much* cleaner if we switch everything to stream and drop the old jobs | 17:04 |
| jrosser | and then go clear out all the legacy if version < / > 8.x that we have in a few places | 17:05 |
| spatel | That is what i am saying, lets create centos-8-stream job and delete centos-8.x | 17:05 |
| spatel | its hard to maintain two version which is overlap a lot | 17:06 |
| jrosser | so neutron is a bit of a special case, the centos/OVS jobs are defined in the os_neutron repo, so patch this https://github.com/openstack/openstack-ansible-os_neutron/blob/master/zuul.d/jobs.yaml | 17:06 |
| jrosser | change the job names to end in -stream | 17:07 |
| jrosser | and use nodeset: centos-8-stream | 17:07 |
| spatel | metal and lxc both? | 17:07 |
| jrosser | yep | 17:08 |
| jrosser | then in the openstack-ansible repo, pretty much all of the rest of the jobs are defined like this for old centos-8 https://github.com/openstack/openstack-ansible/blob/master/zuul.d/jobs.yaml#L485-L548 | 17:08 |
| jrosser | and they are used mostly in here to define a template thats used for all the os_<blah> roles https://github.com/openstack/openstack-ansible/blob/master/zuul.d/project-templates.yaml | 17:09 |
| jrosser | but for these its a pretty good idea to use codesearch.opendev.org to check everywhere else too, just like with neutron there might be a couple of other places that roles define special jobs in their own repo | 17:10 |
| jrosser | spatel: also think about upgrades - do we need to fix this on wallaby? | 17:10 |
| spatel | yes i think so | 17:11 |
| jrosser | right, so unfortunatley thats harder | 17:11 |
| jrosser | you need a patch which addresses both centos-8 and centos8-stream, backport it | 17:11 |
| jrosser | then clean up all the old centos stuff afterward | 17:11 |
| jrosser | sorry :( | 17:11 |
| spatel | then lets leave that alone, it will be too much work | 17:11 |
| jrosser | wallaby is the transition release from centos -> stream for OSA users | 17:12 |
| jrosser | but theres kind of no testing for OVS there at all...... | 17:12 |
| jrosser | OVS+centos stream i mean | 17:12 |
| spatel | BRB | 17:12 |
| jrosser | yeah me too - have to go | 17:13 |
| spatel | Thank you! i will add new jobs for stream and will see | 17:16 |
| opendevreview | David Moreau Simard proposed openstack/openstack-ansible master: DNM: Test ara 1.5.7rc1 with --diff https://review.opendev.org/c/openstack/openstack-ansible/+/696634 | 18:17 |
| opendevreview | Jonathan Rosser proposed openstack/openstack-ansible master: Enable tempest tests for sahara https://review.opendev.org/c/openstack/openstack-ansible/+/802551 | 19:40 |
| dmsimard | found a bug in 1.5.7rc1 with the new delegated host tracking, turns out ansible sometimes returns an unresolved jinja expression (i.e: delegate_to: "{{ item }}") will return "{{ item }}" instead of the actual value | 19:43 |
| dmsimard | I'll work out a fix and run another test | 19:43 |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!