| tonyb | Okay I'm confused and missing $something. For the DIB+Nodepool functional testing the devstack we create has public and private networks and assigns a floating IP to the node. What I can't see is how the demo user/project gets access to private network and router. | 01:55 |
|---|---|---|
| tonyb | I see the network/subnet/router being added via the 'devstack-demo-admin' cloud but it isn't visible to the demo user in the demo project | 01:56 |
| tonyb | (when I say isn't I mean in my devstack setup, it is visible in the nodepool testing) | 02:02 |
| ianw | i don't know if it helps but i feel like the job only ever looked for the floating ip and ssh'd to that? | 02:14 |
| tonyb | ianw: sort of In the nodepool case the devstack cloud which is the demo project and demo user. An instance spawned by nodepool gets an IP in the private network (which has a running DHCP server) and then neutron maps that (private) Fixed-IP to an assigned Floating IP | 02:28 |
| tonyb | So yeah we only ever really talk to the floating IP but the provate network is essential to get that going | 02:28 |
| tonyb | My devstack setup looks to have the same config but the devstack cloud (again demo project and user) can't see the private network | 02:29 |
| tonyb | but the admin user on the demo project *can* see the private network | 02:30 |
| tonyb | `openstack --os-cloud devstack-admin-demo server create --flavor dib1024 --image test-image --key-name root --network private --use-config-drive test-admin; | 03:00 |
| tonyb | openstack --os-cloud devstack-admin-demo floating ip create --fixed-ip-address 10.1.0.49 --port 098b5e8b-d5c0-48df-9b98-7e72308d5402 public` works but trying the same thing as a regular user says no such network private (or words to that effect) | 03:00 |
| tonyb | I could just use the devstack-admin-demo cloud but that seems silly | 03:01 |
| JayF | to be explicit: what is the cloud that doesn't work? | 03:17 |
| JayF | I would expect --os-cloud devstack to work, but not devstack-admin or devstack-system-admin | 03:19 |
| tonyb | JayF: `--os-cloud devstack` AFACIT cloud=devstack maps to project demo, user demo and cloud=devstack-admin-demo maps to project demo, user admin. | 03:20 |
| JayF | yep that's what I was looking at, and you are right afaik | 03:20 |
| JayF | I am not an expert in devstack generally, but that's the behavior I'd expect too. | 03:20 |
| JayF | Sorry if that's not helpful just making sure scope mixups weren't happening; we get that a lot since Ironic rbac went default (baremetal node list returns nothing unless you are admin scoped) | 03:21 |
| tonyb | JayF: `openstack --os-cloud devstack network list` only has the 'public' but `openstack --os-cloud devstack-admin-demo network list` has both public and private. | 03:22 |
| JayF | interesting, maybe perms are different by default and that's expected for testing scenario purposes? | 03:22 |
| tonyb | I don't know. The part I'm struggling with is when devstack is installed as part of the nodepool testing (which admitedly is a different role) the 'devstack' cloud shows both networks. | 03:25 |
| tonyb | so clearly there is something I need to do, or not do, with the devstack role but I can't find it | 03:25 |
| JayF | I pointed claude at the devstack codebase to see if it could figure how to make private network visible | 03:26 |
| tonyb | it doesn't look to be RBAC as the 2 comfigs are comprable there ? | 03:26 |
| tonyb | claude? | 03:26 |
| JayF | as in claude-code | 03:27 |
| JayF | just seeing how well it does with devstack; so far, not very | 03:27 |
| tonyb | LOL | 03:27 |
| JayF | giant files are extra hard because it doesn't want to load the whole thing into context | 03:28 |
| JayF | it claims "openstack network set --share private" as demo-admin should do it | 03:29 |
| JayF | I am skeptical | 03:29 |
| tonyb | Yeah that's not it | 03:29 |
| tonyb | the (working) nodepool setup doesn't share the provate network | 03:30 |
| JayF | well I can tell you in the ironic devstack configuration, it's sharing the private network | 03:31 |
| JayF | at least in the one I have on my local VM, devstack user can see private network, and private network shows as shared | 03:31 |
| JayF | s/user/os-cloud/ | 03:31 |
| JayF | but we change enough about networking who knows :( it's late here I'm gonna step away, good luck, sorry I couldn't help | 03:32 |
| tonyb | `openstack --os-cloud devstack-admin-demo network show -f value -c shared private` says False (everywhere) | 03:34 |
| tonyb | JayF: I appreciate the help. It never hurts to double/triple check things | 03:35 |
| ianw | hrm, it really does seem to just use "devstack" cloud https://opendev.org/zuul/nodepool/src/branch/master/playbooks/nodepool-functional-container-openstack/templates/nodepool.yaml.j2#L20 | 03:53 |
| ianw | but then yeah, private -> https://opendev.org/zuul/nodepool/src/branch/master/playbooks/nodepool-functional-container-openstack/templates/nodepool.yaml.j2#L32 | 03:53 |
| tonyb | ianw: Yup. And I can't find the difference important difference in devstack setup between https://fac63a6a94019ab8151d-353a8055100be238a18e62fdcc374ef1.ssl.cf2.rackcdn.com/openstack/3ff15981db0847079b8948f7b066eafa/job-output.txt (nodepool/working) and https://c77aa8f87e095b99b6e6-3a828353f8ae4adcb13d9c2988d28730.ssl.cf2.rackcdn.com/openstack/5385c2051a904ad48f8b3bbda187ec57/job-output.txt (devstack/not working) | 05:47 |
| ianw | is it something about the way nodepool launches it with it's --net command? | 06:02 |
| tonyb | I do need to be able to do that, but right now if add --net private all I get is "can't locate network private" (or similar) | 06:04 |
| ianw | like nodepool does 2025-06-18 11:27:59,764 DEBUG nodepool.OpenStackAdapter.devstack: API call get_network in 0.09363511099991229 and then iirc launches it maybe with an id? | 06:05 |
| tonyb | Hmmm I can try that | 06:05 |
| ianw | https://opendev.org/zuul/nodepool/src/branch/master/nodepool/driver/openstack/adapter.py#L636 -- so i think it's setting that there | 06:06 |
| frickler | to me this looks like devstack is working fine, the job is ubuntu noble is passing, just almalinux is acting weird? | 06:06 |
| frickler | weird as in "networkmanager fails to configure the interface in the instance" | 06:07 |
| tonyb | ianw: https://paste.opendev.org/show/biAsRqRLzdcfng29PQiw/ | 06:07 |
| ianw | i feel like i've never seen devstack-admin-demo used at all | 06:08 |
| tonyb | frickler: Yes I think there are two issues here 1) Almalinux is weird and not writing out a network config at all (where as Ubuntu does) ; and 2) the devstack testing setup is different to the nodepool setup and those differences hide "1". | 06:09 |
| ianw | i feel like nodepool is only ever given the "openstack" cloud to work from in the clouds.yaml? | 06:10 |
| tonyb | ianw: devstack uses it "a bunch" eg https://opendev.org/openstack/devstack/src/branch/master/lib/neutron_plugins/services/l3#L266 | 06:10 |
| frickler | well it does boot the almalinux instance just fine in both logs you posted earlier, doesn't it? so I don't see this "network private not found" error there | 06:10 |
| ianw | (sorry - "devstack" - https://fac63a6a94019ab8151d-353a8055100be238a18e62fdcc374ef1.ssl.cf2.rackcdn.com/openstack/3ff15981db0847079b8948f7b066eafa/nodepool/nodepool.yaml). so it can't have been acting as anyone else, like the admin? | 06:11 |
| tonyb | ianw: Yes that's true but in this case "testing", we also build the devstack so I'm looking for $something | 06:11 |
| tonyb | ianw: not AFAICT, unless nodepool is using a different clouds.yaml | 06:11 |
| tonyb | frickler: Yes in the gate it boots but that's because I'm directly attaching the VM to the public network which doesn't work because of item 1 a few lines back. Ubuntu does work. | 06:12 |
| tonyb | frickler: nodepool doesn't directly attach to the public network, instead is attaches to the private network and then adds a floating IP to the port on the private network. | 06:13 |
| tonyb | frickler: which I can make work but not for the cloud "devstack" which /etc/openstack/clouds.yaml maps to demo/demo | 06:14 |
| ianw | https://c77aa8f87e095b99b6e6-3a828353f8ae4adcb13d9c2988d28730.ssl.cf2.rackcdn.com/openstack/5385c2051a904ad48f8b3bbda187ec57/controller/logs/etc/openstack/clouds.yaml i guess you mean there | 06:17 |
| tonyb | Correct | 06:17 |
| ianw | 2025-06-18 10:46:58.356 | ++ lib/neutron_plugins/services/l3:create_neutron_initial_network:202 : oscwrap --os-cloud devstack --os-region RegionOne network create private -f value -c id | 06:18 |
| ianw | https://c77aa8f87e095b99b6e6-3a828353f8ae4adcb13d9c2988d28730.ssl.cf2.rackcdn.com/openstack/5385c2051a904ad48f8b3bbda187ec57/controller/logs/devstacklog.txt | 06:18 |
| ianw | that really seemed like it created "private" :/ | 06:19 |
| tonyb | 2025-06-18 11:13:44.150433 | ubuntu-jammy | ++ lib/neutron_plugins/services/l3:create_neutron_initial_network:202 : oscwrap --os-cloud devstack --os-region RegionOne network create private -f value -c id | 06:20 |
| tonyb | https://fac63a6a94019ab8151d-353a8055100be238a18e62fdcc374ef1.ssl.cf2.rackcdn.com/openstack/3ff15981db0847079b8948f7b066eafa/job-output.txt (nodepool devstack install) | 06:20 |
| tonyb | wait | 06:20 |
| tonyb | it was created in the 'devstack' cloud and now it's not found in the 'devstack' cloud ? | 06:21 |
| frickler | ah, yes, in the public network dhcp would be failing, so that part is to be expected. do you have a devstack run where the private network failure can be seen? maybe hold a node for that to look at things in place, I can check in my afternoon then | 06:24 |
| tonyb | 158.69.71.4 | 06:25 |
| ianw | one other thing to try might be to add in https://opendev.org/zuul/nodepool/src/commit/47193a46b91f8b4f220b455c93c876d389ce0802/playbooks/nodepool-functional-container-openstack/run.yaml#L29 to create the unmanaged vm first. that dumps out the networks etc. | 06:25 |
| tonyb | it's the node I'm currently working on/debugging with | 06:25 |
| tonyb | ianw: I can add that but I felt like that was mostly there to test that nodepool didn't impact any non-nodepool nodes | 06:32 |
| tonyb | that unmanaged-vm ends up in error state anyway | 06:32 |
| frickler | so the private network belongs to the admin tenant, not to the demo tenant. so it is expected that the demo tenant cannot see it. the question is why this happens | 06:33 |
| tonyb | frickler: where did you see that? | 06:33 |
| tonyb | frickler: I swear I looked for that type of mistake | 06:34 |
| frickler | "openstack --os-cloud devstack-admin network show private -c project_id" and compare with "... project list" | 06:34 |
| frickler | on 158.69.71.4 | 06:34 |
| frickler | oh, wait, that could be the osc wrapper daemon | 06:37 |
| frickler | it essentially ignores the --os-cloud option | 06:37 |
| frickler | probably this isn't seen in normal CI because nothing actually uses the private network, tempest creates its own | 06:38 |
| ianw | another weird thing -> https://opendev.org/zuul/nodepool/src/commit/47193a46b91f8b4f220b455c93c876d389ce0802/playbooks/nodepool-functional-openstack/write_clouds_yaml.yaml ... this re-writes clouds.yaml to force ipv4 | 06:39 |
| ianw | but that is not in 158.69.71.4 ... suggesting the file isn't updated? | 06:39 |
| tonyb | I don't call that playbook on the devstack node | 06:40 |
| ianw | ahh, ok | 06:42 |
| ianw | can you just do this all as "devstack-admin"? | 06:45 |
| frickler | tonyb: try to disable the openstack-cli-server service in local.conf | 06:46 |
| frickler | oh, it doesn't get used when "if use_library_from_git "python-openstackclient"" is true, that also explains lack of CI coverage I guess https://opendev.org/openstack/devstack/src/branch/master/stack.sh#L1014-L1023 | 06:49 |
| tonyb | ianw: I can, but I was hoping to be as close to "correct" as possible | 06:49 |
| tonyb | frickler: Noted. | 06:49 |
| ianw | hrm, i'm starting to think that may be correct :) | 06:50 |
| frickler | essentially when you call "openstack --os-cloud xyz" in devstack with cli-server active, the command gets executed with "devstack-admin" creds | 06:50 |
| tonyb | ianw: LOL | 06:51 |
| frickler | and I seem to see a lot of code in neutron plugins that doesn't test the right thing as a result | 06:51 |
| frickler | though likely this should be treated as bug in the cli server. maybe we can work around it in the oscwrap function | 06:56 |
| tonyb | Yeah it does seem like a bug in the cli-server. Once we've proven that's the problem I can chase that up | 06:56 |
| frickler | well I think essentially I did prove this? the log says "openstack --os-cloud devstack network create private", but the network ends up in the admin tenant | 06:58 |
| tonyb | That's very true. | 06:58 |
| frickler | I've also known about the wrong clouds issue for years, I just need to find where I added a workaround | 06:58 |
| tonyb | Ah | 06:59 |
| tonyb | Well I'll get a new revisions of the devstack stuff going, which includes floating IPs and disabling the cli-server | 06:59 |
| frickler | ah, it is in a wip patch of mine https://review.opendev.org/c/openstack/devstack/+/923944/4/lib/keystone#445 | 07:00 |
| tonyb | Ah | 07:01 |
| frickler | I just wasn't aware that it also affects existing parts of devstack, because ... well everything did seem to work? | 07:01 |
| tonyb | Yeah. yay us finding hidden bugs | 07:01 |
| frickler | well currently I'm a bit afraid of how many bugs might be uncovered when we fix this. like a whole lot of srbac stuff might be not testing what it expects to test | 07:03 |
| tonyb | ergh | 07:04 |
| *** tosky is now known as Guest18397 | 07:37 | |
| *** tosky_ is now known as tosky | 07:37 | |
| opendevreview | Tony Breeds proposed openstack/diskimage-builder master: Add new openstack/devstack based functional testing https://review.opendev.org/c/openstack/diskimage-builder/+/949942 | 07:38 |
| frickler | tonyb: ^^ doesn't work that way, not sure about the right option though | 07:52 |
| frickler | ah, "devstack_services:\nopenstack-cli-server: false" with proper indentation | 07:52 |
| tonyb | I'll figure it out, just got to grab my son from a class | 07:53 |
| frickler | stumbled upon this independently, but it somehow fits. sorry if this is well known already. "Two of the most famous products of Berkeley are LSD and Unix. I don’t think that this is a coincidence." https://web.mit.edu/~simsong/www/ugh.pdf | 08:09 |
| opendevreview | Tony Breeds proposed openstack/diskimage-builder master: Add new openstack/devstack based functional testing https://review.opendev.org/c/openstack/diskimage-builder/+/949942 | 08:25 |
| *** amoralej_ is now known as amoralej | 12:06 | |
| fungi | technically bsd was a product of bell labs (at&t), uc berkeley licensed it from them and then expanded on it, diverging from at&t's own continued development for system v | 12:30 |
| opendevreview | Merged opendev/base-jobs master: Enable ovh log uploads in test https://review.opendev.org/c/opendev/base-jobs/+/952873 | 12:34 |
| opendevreview | James E. Blair proposed zuul/zuul-jobs master: DNM: use base-test to validate log uploads https://review.opendev.org/c/zuul/zuul-jobs/+/952905 | 15:44 |
| opendevreview | James E. Blair proposed opendev/system-config master: DNM: Test "Lit" version of zuul-results-plugin https://review.opendev.org/c/opendev/system-config/+/952907 | 16:03 |
| opendevreview | James E. Blair proposed openstack/project-config master: Add gerrit project stanzas https://review.opendev.org/c/openstack/project-config/+/952934 | 16:31 |
| corvus | infra-root, config-core: ^ if anyone happens to be around to approve that, paladox is currently updating that plugin and it'd be cool to get a depends-on build with it. | 16:33 |
| fungi | done | 16:41 |
| opendevreview | Merged openstack/project-config master: Add gerrit project stanzas https://review.opendev.org/c/openstack/project-config/+/952934 | 16:47 |
| opendevreview | James E. Blair proposed opendev/zone-opendev.org master: Replace zl01 and zl02 https://review.opendev.org/c/opendev/zone-opendev.org/+/952941 | 17:32 |
| opendevreview | James E. Blair proposed opendev/system-config master: Replace zl01 and zl02 https://review.opendev.org/c/opendev/system-config/+/952942 | 17:32 |
| corvus | fungi: ^ i launched 2 new zuul-launcher nodes; they are 2GB instead of 8GB, since it looks like that should be enough. this also replaces our existing zl01 jammy node. | 17:33 |
| fungi | lgtm, approved both, thanks! | 17:36 |
| opendevreview | Merged opendev/zone-opendev.org master: Replace zl01 and zl02 https://review.opendev.org/c/opendev/zone-opendev.org/+/952941 | 17:48 |
| corvus | fungi: according to https://review.opendev.org/952905 ovh looks good, i think we can merge https://review.opendev.org/952874 to re-enable them | 17:51 |
| fungi | agreed, will do | 17:54 |
| fungi | and approved as well, thanks again! | 17:55 |
| opendevreview | Merged opendev/system-config master: Replace zl01 and zl02 https://review.opendev.org/c/opendev/system-config/+/952942 | 18:19 |
| fungi | gonna go grab a late lunch/early dinner, back in an hour-ish | 18:33 |
| corvus | #status log replaced zl01-02 with new 2GB noble vms; deleted old servers | 20:16 |
| opendevstatus | corvus: finished logging | 20:18 |
| corvus | that appeared here: https://fosstodon.org/@opendevinfra/114711876425306841 | 20:19 |
| opendevreview | Merged opendev/base-jobs master: Enable ovh log uploads https://review.opendev.org/c/opendev/base-jobs/+/952874 | 20:20 |
| corvus | that seems to confirm the access token fix | 20:20 |
| corvus | i've gone ahead and launched 8 new zm servers; we can probably knock those out pretty easily too. | 20:21 |
| fungi | excelent | 20:21 |
| fungi | l | 20:21 |
| fungi | infra-prod-remote-puppet-else failed in the deploy for 952942: https://zuul.opendev.org/t/openstack/build/0f1ef685cc6247b285bdc3850fa4cd24 | 20:27 |
| fungi | TASK [puppet-setup-ansible : Run puppet module install on bridge] ... fatal: could not read Username for 'https://github.com': No such device or address | 20:29 |
| corvus | https://paste.opendev.org/show/b48WHFIfZ9AxpJGJEVOS/ | 20:29 |
| corvus | yeah, that series of errors seems strange... | 20:29 |
| fungi | got me scratching my head too | 20:29 |
| corvus | install_modules.sh: line 19: puppet: command not found | 20:30 |
| corvus | that line is apparently fine | 20:30 |
| corvus | it shows up in previous successful runs | 20:30 |
| corvus | maybe there was a momentary dns snafu | 20:31 |
| fungi | i think that "could not read Username" error can also be a misleading indicator for when a repository has been renamed or made private | 20:31 |
| corvus | mm yeah. also, there were 4 repos that updated in that pass; that may be an unusually high amount of activity | 20:32 |
| corvus | https://github.com/puppetlabs/puppetlabs-rabbitmq is a 404 | 20:33 |
| corvus | maybe they made that private | 20:33 |
| corvus | that's for storyboard | 20:34 |
| fungi | we could just stick it in the emergency disable list if the problem persists, and see if that solves it | 20:35 |
| corvus | maybe we can/should migrate to https://github.com/voxpupuli/puppet-rabbitmq | 20:35 |
| fungi | oh probably | 20:35 |
| fungi | puppetlabs is shedding everything and anything | 20:35 |
| fungi | that makes sense, it's a community module now | 20:35 |
| corvus | i'll write some changes | 20:39 |
| opendevreview | James E. Blair proposed opendev/system-config master: Update rabbitmq puppet module https://review.opendev.org/c/opendev/system-config/+/952946 | 20:40 |
| fungi | i think that'll be self-testing | 20:40 |
| corvus | puppet-storyboard needs a change too... not sure if they can be sequenced.... | 20:40 |
| opendevreview | James E. Blair proposed opendev/puppet-storyboard master: Update rabbitmq puppet module https://review.opendev.org/c/opendev/puppet-storyboard/+/952947 | 20:43 |
| corvus | now that i wrote that... i don't actually know if that's necessary for us to run it. i guess we'll see. | 20:43 |
| opendevreview | James E. Blair proposed opendev/system-config master: DNM: Test "Lit" version of zuul-results-plugin https://review.opendev.org/c/opendev/system-config/+/952907 | 20:45 |
| opendevreview | James E. Blair proposed opendev/zone-opendev.org master: Replace zuul mergers https://review.opendev.org/c/opendev/zone-opendev.org/+/952948 | 20:49 |
| opendevreview | James E. Blair proposed opendev/system-config master: Replace zuul mergers https://review.opendev.org/c/opendev/system-config/+/952949 | 20:50 |
| corvus | those two changes for the mergers are ready to go, along with the testing update: https://review.opendev.org/952696 | 20:50 |
| fungi | one thing that should have dawned on me with the zl replacements... in the past we've avoided recycling server names because ansible facts get cached on bridge. maybe that's no longer an issue? | 20:52 |
| fungi | or maybe the (now long ago) switch to using ip addresses for `ansible_host` solved it | 20:54 |
| corvus | i can clear them out... i'd rather do the extra work and reuse them. i spend way too much time ssh-ing into n01... nl02... nl03... nl04... nl05 yes that's the one... | 20:54 |
| fungi | yeah | 20:54 |
| opendevreview | Merged opendev/zone-opendev.org master: Replace zuul mergers https://review.opendev.org/c/opendev/zone-opendev.org/+/952948 | 20:55 |
| fungi | and at least we don't preserve root's ssh known_hosts entries on bridge | 20:55 |
| corvus | i haven't cleaned out any facts yet, and ansible hasn't seemed to have a problem so far | 20:55 |
| fungi | it's worth taking the easier path here to see if anything still breaks | 20:56 |
| corvus | yeah, and we write out a global known_hosts file on bridge that ansible uses | 20:56 |
| corvus | ++ these are good test hosts | 20:56 |
| opendevreview | Merged opendev/system-config master: Replace zuul mergers https://review.opendev.org/c/opendev/system-config/+/952949 | 21:51 |
| opendevreview | Tony Breeds proposed openstack/diskimage-builder master: Add new openstack/devstack based functional testing https://review.opendev.org/c/openstack/diskimage-builder/+/949942 | 22:02 |
| corvus | the update to the zuul-results-summary plugin that paladox made checks out in our test jobs. https://gerrit-review.googlesource.com/c/plugins/zuul-results-summary/+/485281 | 23:00 |
| paladox | Nice! | 23:01 |
| paladox | merged! | 23:02 |
| corvus | cool! we may want to keep than in mind for our next gerrit update (might be worth updating it soon afterwords, just to make sure everything's okay). it should be a noop -- it's a build and javascript modernization change | 23:02 |
| corvus | paladox: thanks again! :) | 23:03 |
| paladox | yw! | 23:03 |
| corvus | #status log replaced all zuul mergers with new noble vms; deleted old servers | 23:03 |
| opendevstatus | corvus: finished logging | 23:04 |
| fungi | sounds gere | 23:10 |
| fungi | great | 23:10 |
| opendevreview | Tony Breeds proposed openstack/diskimage-builder master: Add new openstack/devstack based functional testing https://review.opendev.org/c/openstack/diskimage-builder/+/949942 | 23:26 |
| corvus | zm01 graphs on cacti look as expected; except that apparently we have a /boot partition now, and we did not before | 23:41 |
| corvus | hopefully we don't start running out of space due to kernels | 23:42 |
| opendevreview | Merged opendev/system-config master: Switch zuul-mergers to noble https://review.opendev.org/c/opendev/system-config/+/952696 | 23:48 |
Generated by irclog2html.py 4.0.0 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!