| *** mhen_ is now known as mhen | 02:10 | |
| opendevreview | Rodolfo Alonso proposed openstack/tempest master: Wait for the router port activation before deletion https://review.opendev.org/c/openstack/tempest/+/936337 | 09:44 |
|---|---|---|
| *** ykarel__ is now known as ykarel | 13:26 | |
| opendevreview | Merged openstack/devstack master: Add start time (in seconds) to the WSGI configuration file https://review.opendev.org/c/openstack/devstack/+/936669 | 15:38 |
| opendevreview | Stephen Finucane proposed openstack/devstack stable/2024.2: Install setuptools 'core' extra https://review.opendev.org/c/openstack/devstack/+/937203 | 18:54 |
| frickler | kopecmartin: clarkb: fungi: https://bugs.launchpad.net/devstack/+bug/2082617 looks weird to me, but the new failure seems to match what was seen earlier. I have no idea however what might be happening there | 21:07 |
| clarkb | roles/write-devstack-local-conf writes that out from the ansible variables I think | 21:13 |
| clarkb | so it would be in there somewhere most likely | 21:13 |
| clarkb | frickler: https://zuul.opendev.org/t/openstack/build/d6a4f75860334f1cb352955801564779/console#4/0/59/compute1 compare to https://zuul.opendev.org/t/openstack/build/d6a4f75860334f1cb352955801564779/console#4/0/59/controller | 21:22 |
| clarkb | note the localrc is null for controller | 21:22 |
| clarkb | if you look at https://zuul.opendev.org/t/openstack/build/d6a4f75860334f1cb352955801564779/log/zuul-info/inventory.yaml it appears devstack_localrc is set for controller compute1 and globally so at some point that value disappears? | 21:25 |
| clarkb | localrc: "{{ devstack_localrc|default(omit) }}" is how it gets passed to the modile | 21:25 |
| clarkb | *module | 21:25 |
| clarkb | I don't see anything in the previous playbooks like would be candidates for unloading that variable somehow | 21:29 |
| clarkb | frickler: might want to add some additional debugging to the devstack pre run playbook to see what variables are present in the playbook at that point then work backward from there? | 21:30 |
| clarkb | corvus: ^ this might be an interesting one in case it is related to ansible | 21:30 |
| fungi | could it be the same implicit localhost in the hostvars array? | 21:32 |
| clarkb | fungi: well we haven't updated our zuul yet so that shouldn't be it but also it should find the correct var for each host in the inventory | 21:34 |
| fungi | yeah, good point | 21:35 |
| clarkb | in this case the controller node runs the write-devstack-local-conf role with a null devstack_localrc value despite it being in the inventory file for it twice (node specific and global) | 21:35 |
| clarkb | ya I'm a bit stumped on this one | 21:48 |
| clarkb | at first I thought it was likel ya bug in the role itself but that console log entry seems to say we just don't pass the data to the role in the first place | 21:48 |
| clarkb | maybe its a bug in the inventory itself /me grabs the raw file | 21:53 |
| clarkb | I can yaml.safe_load() the file and i['all']['hosts']['controller']['devstack_localrc']['DATABASE_PASSWORD'] is in there | 21:59 |
| clarkb | i['all']['hosts']['controller']['devstack_localrc'] == i['all']['vars']['devstack_localrc'] evaluates to true too s owe aren't hitting some weird variable precedence thing | 22:00 |
| clarkb | I mean precdence may be in play but if the dicts are equivalent then it shouldn't matter | 22:00 |
| clarkb | I think that I've ruled out a bug in the python that implements the role and a bug in the inventory itself | 22:01 |
| clarkb | which leaves zuul, the zuul job, or ansible? | 22:02 |
| clarkb | I think the timing of the first occurence in that bug is too eraly to be ansible 9 related | 22:03 |
| clarkb | yes that happened october 8 | 22:03 |
| clarkb | and the bug is from late september | 22:03 |
| clarkb | corvus: iirc the inventory file we write out for logging purposes is not actually what th ejobs use? is that correct. Maybe we're masking some problem with var loading in the inventory for a specific host? | 22:05 |
| clarkb | and if the bug isn't super consistent (which I don't think it is) maybe it has to do with variable inputs like ip addresses or hostnames for mirrors or something? | 22:05 |
| clarkb | we have a winner | 22:09 |
| clarkb | https://zuul.opendev.org/t/openstack/build/d6a4f75860334f1cb352955801564779/log/zuul-info/host-info.controller.yaml#63 does not contain an address key: https://zuul.opendev.org/t/openstack/build/d6a4f75860334f1cb352955801564779/log/zuul-info/inventory.yaml#221 | 22:10 |
| clarkb | frickler: ^ the job is broken not a bug in devstack or zuul | 22:11 |
| clarkb | that said I wonder if we can expose that better | 22:11 |
| clarkb | corvus: ^ do you know why that isn't a higher level bug if the vars fail to load properly in the inventory? | 22:11 |
| clarkb | s/higher level bug/higher level error or failure/ | 22:12 |
| clarkb | I wonder if ansible just does that automatically for inventory as some sort of "do the best we can in the face of problems" behavior | 22:13 |
| clarkb | https://opendev.org/openstack/devstack-plugin-ceph/src/branch/master/.zuul.yaml#L92 https://opendev.org/openstack/manila-tempest-plugin/src/branch/master/zuul.d/manila-tempest-jobs.yaml#L531 and https://opendev.org/openstack/manila-tempest-plugin/src/branch/master/zuul.d/manila-tempest-jobs.yaml#L611 are the three occurences of this issue that are immediately discoverable via | 22:18 |
| clarkb | codesearch | 22:18 |
| corvus | clarkb: i do not know the answer to your question off the top of my head | 22:21 |
| clarkb | new theory: ansible lazy evaluates that stuff and we don't refer to devstack_localrc until we try to use it in that role which does a |default(omit) and maybe |default(omit) masks the error | 22:24 |
| clarkb | I wonder if we could do something like set a default value of {} and then drop the |default(omit) and see if that exposes it | 22:28 |
| clarkb | could probably artificially test this too. Though I find that whenever I do that I end up fighitng ansible over the localhost inventory magic which might make results less clear | 22:28 |
| clarkb | frickler: Isummarized all of my debugging rambling above on the issue | 22:52 |
| clarkb | and added manila to the issue too as manila's templest plugin jobs are also broken | 22:52 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!