| opendevreview | Merged openstack/devstack-plugin-ceph master: [manila] Enforce cephfsnative as the default driver https://review.opendev.org/c/openstack/devstack-plugin-ceph/+/922678 | 02:42 |
|---|---|---|
| opendevreview | Amit Uniyal proposed openstack/tempest master: Adds placement trait api calls https://review.opendev.org/c/openstack/tempest/+/922379 | 07:08 |
| opendevreview | Amit Uniyal proposed openstack/tempest master: Adds placement resource provider traits API calls https://review.opendev.org/c/openstack/tempest/+/922567 | 07:08 |
| *** tosky_ is now known as tosky | 12:37 | |
| dtantsur | hey, anyone seen this? | 15:55 |
| dtantsur | [ERROR] /opt/stack/devstack/inc/meta-config:191 bogus config file specification $NOVA_CPU_CONF ($NOVA_CPU_CONF=/etc/nova/nova-cpu.conf, /etc/nova is not a directory) | 15:55 |
| dtantsur | This is on ironic jobs that do not have Nova | 15:55 |
| dtantsur | example https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_9d0/922024/3/gate/ironic-tempest-functional-python3/9d0423d/job-output.txt | 15:55 |
| dtantsur | Could be due to this added to local.conf by something: | 16:00 |
| clarkb | dtantsur: if nova isnt' configured then /etc/nova is probably not being created. You probably need to work backward and see what is trying t oconfigure nova config | 16:00 |
| dtantsur | [[post-config|$NOVA_CPU_CONF]] | 16:00 |
| dtantsur | [libvirt] | 16:00 |
| dtantsur | tb_cache_size = 128 | 16:00 |
| dtantsur | clarkb: yep, just pasted, now need to trace where this local.conf part is coming from | 16:00 |
| dtantsur | which is not immediately obvious to me.. | 16:02 |
| frickler | that was a recent addition by sean-k-mooney | 16:03 |
| dtantsur | sean-k-mooney: the evidence is pointing at you https://opendev.org/openstack/devstack/commit/db305d2a4bb36c3d3a4ef4a108069cd77bca540e | 16:03 |
| dtantsur | right :) | 16:03 |
| dtantsur | does it make sense to check if the file exists first? | 16:03 |
| dtantsur | or is the expectation to also populate non-existing files? | 16:03 |
| clarkb | probably need to check if n-cpu is enabled first | 16:05 |
| clarkb | since that is trying to update the ncpu config and in your case n cpu is not enabled | 16:05 |
| dtantsur | I'm not sure how to do that: the code is completely agnostic | 16:05 |
| dtantsur | * service-agnostic | 16:05 |
| dtantsur | can I just create the parent directory here? https://opendev.org/openstack/devstack/src/branch/master/inc/meta-config#L92-L94 | 16:05 |
| dtantsur | that would be the easiest workaround IMO | 16:05 |
| sean-k-mooney | so that option should not matter if your not using the libvirt driver | 16:05 |
| clarkb | oh it is happening in the base jobs | 16:05 |
| clarkb | sean-k-mooney: right the problem is that /etc/nova is only created if nova is enabled | 16:06 |
| dtantsur | sean-k-mooney: definitely, but it tries to access /etc/nova anyway | 16:06 |
| sean-k-mooney | what tries to access it? | 16:06 |
| sean-k-mooney | devstack? | 16:06 |
| dtantsur | sean-k-mooney: yep, inside of merge_config_file | 16:06 |
| dtantsur | see the link in the beginning | 16:06 |
| sean-k-mooney | ah i alwasy tought that did a mkdir -p before tryign to create the file | 16:07 |
| sean-k-mooney | and merge it | 16:07 |
| dtantsur | yeah, so I think we should just add the mkdir -p | 16:07 |
| clarkb | looking at the zuul jobs nova is a required project and n-cpu is enabled so I think that is fine | 16:07 |
| clarkb | I suspect that dtantsur's job should parent to devstack-minimal instead | 16:07 |
| sean-k-mooney | thats an option i guess | 16:08 |
| dtantsur | I *think* there are reasons why it's not inheriting devstack-minimal | 16:08 |
| clarkb | dtantsur: well then don't disable nova... since the job you're based on assumes nova is enabled and I think that is fair (iirc thati s why devstack-minimal was created to enable these different configurations that are more optimized for specific needs) | 16:08 |
| sean-k-mooney | i mean yes and no | 16:09 |
| dtantsur | clarkb: then I need to copy-paste the initialization of swift, neutron, etc.. and we're broken again once somebody changes devstack | 16:09 |
| sean-k-mooney | we have several jobs that inherit form devstack | 16:09 |
| sean-k-mooney | were later jobs disbale services | 16:09 |
| sean-k-mooney | like we turn of swift in many nova jobs | 16:09 |
| clarkb | sean-k-mooney: that really seems backwards to me. It should be a tree that builds up instead | 16:09 |
| sean-k-mooney | that not how its worked historically | 16:10 |
| JayF | I think in practice we treat it more like a menu of configs to pick from | 16:10 |
| sean-k-mooney | yep | 16:10 |
| clarkb | in that case you should remove all of the post config stuff from that job | 16:10 |
| sean-k-mooney | i would consider turning off nova valid | 16:10 |
| clarkb | because you can break any other job | 16:10 |
| clarkb | and move the configuration stuff into devstack proper probably based on whether or not things like services are enabled/disabled | 16:11 |
| sean-k-mooney | we can do that yes | 16:11 |
| sean-k-mooney | i can drop the cache stuff and just do it in devstack | 16:11 |
| opendevreview | Dmitry Tantsur proposed openstack/devstack master: Create parent directory in merge_config_file https://review.opendev.org/c/openstack/devstack/+/922846 | 16:11 |
| sean-k-mooney | i think there was a flag for that at one point | 16:11 |
| dtantsur | meanwhile, ^^^ | 16:11 |
| clarkb | the problem is those vars are merged in child jobs so you can't claer out what is set in parents | 16:11 |
| clarkb | which means if you want to mix and match later then you need to avoid oversetting things like that in the parents | 16:12 |
| sean-k-mooney | only if you are usign the merge_config_file funcitonality | 16:12 |
| sean-k-mooney | most of the other section supprot mergeing in a way that wont break | 16:13 |
| clarkb | right, its limited to any of the local conf directives | 16:13 |
| clarkb | because once you set them you can't unset them | 16:13 |
| dtantsur | Testing the fix in https://review.opendev.org/c/openstack/ironic/+/922847 | 16:13 |
| sean-k-mooney | this is kind of speical in that devstack is not checking if the parent of the file exist and there is no way to make it conditional because of how we source the local.conf | 16:13 |
| sean-k-mooney | clarkb:right you can only overried them. | 16:14 |
| sean-k-mooney | dtantsur: ill work on a patch to fix this shortly | 16:14 |
| dtantsur | sean-k-mooney: https://review.opendev.org/c/openstack/devstack/+/922846 | 16:15 |
| dtantsur | if you're satisfied with this approach, that is | 16:15 |
| sean-k-mooney | i mean that shoudl be fine also | 16:15 |
| sean-k-mooney | it will just ensure the target directory is alwasy created | 16:16 |
| dtantsur | this is just a quick idea, if you want something better - feel free to take over the patch | 16:16 |
| sean-k-mooney | i can move it to configure_nova_hypervisor here https://github.com/openstack/devstack/blob/master/lib/nova_plugins/hypervisor-libvirt#L39 | 16:16 |
| sean-k-mooney | there was an old devstack review to do this in the past https://review.opendev.org/c/openstack/devstack/+/881580 | 16:17 |
| dtantsur | Dunno, I feel like then we risk it blowing up in a difference place? | 16:17 |
| dtantsur | Generally, if we expect to be able to create the file, it's not unreasonable to also create the directory | 16:17 |
| dtantsur | But up to you, as long as our CI is green again :) | 16:17 |
| sean-k-mooney | ya i think checkign the parent dir tree exist is more robost | 16:18 |
| sean-k-mooney | i generally dont like adding oneoff devstack flag for each possibel config option | 16:18 |
| frickler | we don't have any devstack deployments that do not use libvirt anymore, or do we? like in 3rd party testing? | 16:19 |
| sean-k-mooney | ironic | 16:19 |
| sean-k-mooney | its entirly valid to stand up openstack without nova using just ironic api directly | 16:20 |
| frickler | I meant nova with something other than libvirt | 16:20 |
| sean-k-mooney | oh am wel you can have nova with ironic but other then that no | 16:20 |
| sean-k-mooney | if you were deploying nova at all on the node then it would not fail | 16:20 |
| sean-k-mooney | its /etc/nova that is missing not a libvirt specicfic dir | 16:21 |
| JayF | 3rd party CI for nova vmware driver does exist now, too, right? | 16:21 |
| frickler | if would not fail, but setting a libvirt parameter if in fact like vmware is being used also doesn't seem quite right, even if nova will just ignore that config | 16:21 |
| sean-k-mooney | ya i think we should take both approches | 16:22 |
| frickler | so I'd prefer to move it into lib/nova_plugins/hypervisor-libvirt | 16:22 |
| sean-k-mooney | make it so the post-config stuff wont fail if the parent does not exist | 16:22 |
| sean-k-mooney | and sepreatly ill move this libvirt speciric turning into hypervior-libvirt | 16:22 |
| sean-k-mooney | dtantsur: one comment on "mkdir -p $(dirname $real_configfile) || die " | 16:23 |
| sean-k-mooney | that wil fail if the direcory extis | 16:23 |
| dtantsur | sean-k-mooney: mkdir -p does not fail when the directory exist, does it? | 16:23 |
| JayF | no it won't | 16:23 |
| JayF | test on my local linux machine https://usercontent.irccloud-cdn.com/file/V9G7Cc4x/image.png | 16:24 |
| sean-k-mooney | maybe that is only when not using -p | 16:24 |
| dtantsur | correct | 16:24 |
| sean-k-mooney | mkdir normally only returns 0 if it creatd the dir | 16:24 |
| sean-k-mooney | ok | 16:24 |
| JayF | yeah, -p is explicitly for scripts where you just want to ensure the dir exists | 16:24 |
| JayF | and don't care if you created it or not | 16:24 |
| sean-k-mooney | i rememebr using mkdir as a poor mans file system lock in the past and checking the return code | 16:24 |
| JayF | life before `fuser` was rough :P | 16:25 |
| sean-k-mooney | trying to run multiple devstack in diffent docker contaienr on the same host to test ovs-dpdk is rougher | 16:27 |
| sean-k-mooney | one of the intel third party cis spwaned a super privldage docker conttainer and hten ran devstack in that. this was when docker still used lxc as a backend. we need the mkdir lock to make claiming hugepages work for ovs-dpdk | 16:28 |
| opendevreview | sean mooney proposed openstack/devstack master: [nova] Add flag to set libvirt_tb_cache_size https://review.opendev.org/c/openstack/devstack/+/881580 | 16:37 |
| sean-k-mooney | so i just rebased ^ and moved the devstack job to use the new NOVA_LIBVIRT_TB_CACHE_SIZE variable | 16:38 |
| sean-k-mooney | oh actully thats wrong | 16:39 |
| sean-k-mooney | we moved it before we merged it | 16:39 |
| sean-k-mooney | from workarounds to libvirt | 16:39 |
| opendevreview | sean mooney proposed openstack/devstack master: [nova] Add flag to set libvirt tb_cache_size https://review.opendev.org/c/openstack/devstack/+/881580 | 16:40 |
| sean-k-mooney | ok ^ should work | 16:41 |
| sean-k-mooney | im just talking to melwitt about https://bugs.launchpad.net/nova/+bug/2070501 | 16:44 |
| sean-k-mooney | and its likely that also cause by setting the post_config data in the devstack job | 16:44 |
| sean-k-mooney | melwitt: oh actully i know why i think | 16:46 |
| sean-k-mooney | nova-next does not define devstack_local_conf in group vars | 16:47 |
| sean-k-mooney | only vars | 16:47 |
| sean-k-mooney | so prior to defienign the tb cache it was inheritign the content form the vars definition | 16:48 |
| sean-k-mooney | but when i defiend devstack_local_conf in the group vars of the parent job that fallback nolonger happened | 16:48 |
| sean-k-mooney | so we lost setting q35 | 16:48 |
| clarkb | +2 from me on the fix | 16:57 |
| sean-k-mooney | sorry about that i guess i should have been a little more concervitive with the virt driver specific option | 17:01 |
| melwitt | gmann: ^ heads up on devstack change https://review.opendev.org/c/openstack/devstack/+/881580 which should fix the 100% fail on nova-next. I just "check experimental" it to verify it fixes nova-next | 17:02 |
| sean-k-mooney | https://zuul.opendev.org/t/openstack/build/a59b9eafcc4b48258a8a64e418bafdae nova-next passed | 18:46 |
| sean-k-mooney | although the other jobs in the build stet are still pending for the experimetal run | 18:46 |
| sean-k-mooney | https://zuul.opendev.org/t/openstack/buildset/15c1b0eff1d14b03870193806586ce67 | 18:46 |
| sean-k-mooney | https://zuul.openstack.org/status/change/881580,3 | 18:46 |
| sean-k-mooney | that shoudl finish soon | 18:47 |
| gmann | melwitt: \o/ | 19:04 |
| opendevreview | Rajat Dhasmana proposed openstack/tempest master: WIP: Test image <-> volume <-> server dependency https://review.opendev.org/c/openstack/tempest/+/922873 | 19:22 |
| melwitt | openstacksdk-functional-devstack failed on the patch in the gate queue :( gonna have to recheck it | 20:07 |
| *** dmellado0755 is now known as dmellado075 | 21:06 | |
| opendevreview | Merged openstack/tempest master: Modify mounting paths https://review.opendev.org/c/openstack/tempest/+/922738 | 21:51 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!
