| *** mhen_ is now known as mhen | 02:55 | |
| opendevreview | Merged openstack/nova master: Do not fork compute workers in native threading mode https://review.opendev.org/c/openstack/nova/+/965466 | 05:23 |
|---|---|---|
| *** ralonsoh_ is now known as ralonsoh | 07:08 | |
| ralonsoh | gibi, hello! I've tested pyroute2 0.9.5 in a /requirements patch | 07:17 |
| ralonsoh | https://review.opendev.org/c/openstack/requirements/+/973210 | 07:17 |
| ralonsoh | do you mind if we bump to this version (latest) right now? If we detect any other issue with os-vif, we can revert it again | 07:18 |
| ralonsoh | just for reference: https://zuul.opendev.org/t/openstack/buildset/1b150188643d435e8813c9d8be2ea184. I added nova-ceph-multistore and tempest-integrated-compute jobs | 07:39 |
| opendevreview | Merged openstack/os-vif master: reno: Update master for unmaintained/2024.1 https://review.opendev.org/c/openstack/os-vif/+/965493 | 08:08 |
| gokhan | Hi everyone, | 10:21 |
| gokhan | We are currently running OpenStack Caracal and our security audit team has raised several critical concerns regarding VNC console access via nova-novncproxy. I would like to get your insights on the best practices or if there are any missing configurations on our side. | 10:21 |
| gokhan | Lack of Session Timeout: Once a VNC connection is established, it remains open indefinitely (or for very long periods), which is seen as a major risk if a workstation is left unattended. | 10:21 |
| gokhan | Missing Rate Limiting: There is no built-in mechanism to limit the number of console URLs a single user can request in a short period, potentially leading to resource exhaustion or brute-force attempts | 10:21 |
| gokhan | URL Hijacking Risk: If a console URL is intercepted, it allows immediate access to the VM without further authentication. | 10:22 |
| gokhan | we are planning to implement: Setting [console] enforce_session_timeout = True and adjusting [consoleauth] token_ttl to enforce a hard disconnect (e.g., after 30-60 minutes), Implementing rate limiting at the HAProxy level using stick-tables based on source IP for the /remote-consoles endpoint, Removing none from auth_schemes to enforce vencrypt only. | 10:23 |
| gokhan | Does enforce_session_timeout in Caracal fully address the "infinite session" issue by terminating the WebSocket proxy connection on the server side once the TTL expires? | 10:24 |
| gokhan | Are there any known side effects of setting a very low token_ttl (e.g., 30s) while enforce_session_timeout is active? | 10:24 |
| gokhan | Regarding the password authentication spec, what is the current stability status for KVM/Libvirt in Caracal? https://specs.openstack.org/openstack/nova-specs/specs/wallaby/approved/nova-support-webvnc-with-password-authentication.html | 10:25 |
| gokhan | I would appreciate any guidance or best practices from the you, thanks :) | 10:25 |
| gibi | gokhan: about vnc timeout we have https://specs.openstack.org/openstack/nova-specs/specs/2024.1/implemented/enforce-remote-console-session-timeout.html | 10:30 |
| gibi | > Does enforce_session_timeout in Caracal fully address the "infinite session" issue by terminating the WebSocket proxy connection on the server side once the TTL expires? | 10:35 |
| gibi | I believe so | 10:35 |
| gibi | > Are there any known side effects of setting a very low token_ttl (e.g., 30s) while enforce_session_timeout is active? | 10:35 |
| gibi | as far as I understand that would mean that the VNC session would disconnect after 30secs, if that is OK for you as a user experience then it is fine from nova perspective to configure such short ttl | 10:37 |
| gibi | regarding password auth, it was never implemented. I think the last attempt was https://review.opendev.org/c/openstack/nova/+/622336 | 10:40 |
| gibi | it had an approved spec so I think the nova core team was OK with that feature proposal. So if you are interested it can be revived. | 10:42 |
| gibi | if you are new to openstack then here are our contributor guide https://docs.openstack.org/nova/latest/contributor/index.html | 10:43 |
| gibi | JayF: +2+A thanks for the explanation | 10:48 |
| gokhan | Thanks a lot for the clarification, Gibi! | 10:56 |
| gokhan | It is very helpful to know that enforce_session_timeout is the server-side solution we were looking for. We now understand that token_ttl serves as both the link validity period and the maximum session duration when this feature is enabled. We will set a reasonable TTL (like 3600s) to balance security and user experience. | 10:56 |
| gokhan | Regarding the password authentication; it’s a bit disappointing that it wasn’t fully implemented, but knowing its current status helps us explain the situation to our security audit team. We will evaluate the possibility of developing and reviving this feature with our team. | 10:56 |
| gibi | ralonsoh: re pyroute2 0.9.5 the job you run actaully shows the same error about the event loop https://zuul.opendev.org/t/openstack/build/9e54327cbaa94a4b8ecf9c5472934c97/log/controller/logs/screen-n-cpu.txt#48564-48581 | 10:58 |
| gibi | gokhan: cool. Happy to help | 10:59 |
| ralonsoh | gibi, ok I thought that was fixed | 11:00 |
| gokhan | gibi, I have also created a bug : Live Migration Rollback Corrupts Block Device Mapping of Unrelated VM : https://bugs.launchpad.net/nova/+bug/2133501 do you have any insgihts about it ? we will send a patch for this. this worked for us but I want to be sure I don't ignore critical things > https://bugs.launchpad.net/nova/+bug/2133501/comments/10 | 11:04 |
| sean-k-mooney | gokhan: passward auth was not impletned because it very insecure | 12:11 |
| sean-k-mooney | gokhan: we felt adding ti was mroe harmful as it woudl lead operator to think that it was a sufficnet security mechanium to use in production | 12:12 |
| sean-k-mooney | gokhan: vnc's password auth can have a maxium of i belive 7 ascii pritnabale charters as the password so the password space is very small | 12:12 |
| sean-k-mooney | gokhan: we implented vencypt to do cert based auth between the proxy and the qemu instance instead | 12:13 |
| sean-k-mooney | and then expect the guest passwoard auth to be used on the guest | 12:13 |
| sean-k-mooney | gokhan:... any corss instece behiver is generaly treated as a possible secuirty bug so it shoudl not be filed publicly but its too late to change that now | 12:15 |
| sean-k-mooney | hum i see that proably should have been set as public security rahter hten just public | 12:18 |
| sean-k-mooney | as there are still security and multi tenant aspect to this | 12:18 |
| opendevreview | Balazs Gibizer proposed openstack/nova master: Upgrade note for concurrency mode default change https://review.opendev.org/c/openstack/nova/+/969888 | 12:29 |
| opendevreview | Balazs Gibizer proposed openstack/nova master: Remove eventlet from CacheConcurrencyTestCase https://review.opendev.org/c/openstack/nova/+/970069 | 12:39 |
| opendevreview | Balazs Gibizer proposed openstack/nova master: Remove eventlet from libvirt/test_driver https://review.opendev.org/c/openstack/nova/+/970070 | 12:39 |
| opendevreview | Balazs Gibizer proposed openstack/nova master: Remove eventlet from libvirt/volume/test_mount https://review.opendev.org/c/openstack/nova/+/970071 | 12:39 |
| opendevreview | Balazs Gibizer proposed openstack/nova master: Remove eventlet from libvirt/volume/test_mount https://review.opendev.org/c/openstack/nova/+/970071 | 12:42 |
| opendevreview | Takashi Natsume proposed openstack/nova-specs master: Create specs directory for 2026.2 Hibiscus https://review.opendev.org/c/openstack/nova-specs/+/973373 | 12:55 |
| opendevreview | Merged openstack/nova master: [ironic] Use constants from Ironic, test w/ddt https://review.opendev.org/c/openstack/nova/+/969321 | 12:57 |
| gokhan | Thanks for the warning and the detailed explanation, Sean. | 12:58 |
| gokhan | I shared this publicly because I saw the "approved" status on the Wallaby spec and assumed it was a standard feature discussion rather than a zero-day vulnerability. Interestingly, even that spec mentions: "the VNC password is not a very strong security mechanism" which aligns perfectly with what you said. | 12:58 |
| gokhan | Following your advice, we will rely on vEncrypt for proxy-to-instance security and utilize enforce_session_timeout (implemented in Caracal) to mitigate the risks raised by our security team. | 12:58 |
| gokhan | I now understand the rationale behind not implementing the 7-8 character weak password auth. We will focus on hardening the session management side instead. Thanks again for your time and the heads-up! | 12:58 |
| gokhan | sean-k-mooney, ^^ | 12:59 |
| sean-k-mooney | no worries. when we were eveualting the password auth we had some converstation with the qemu and libvirt maintainer and it was there advice to not implement it in nova and pivot to the vencypt approch orginally | 13:01 |
| sean-k-mooney | the enforce_session_timeout was also part of the dicsussion on general hardening of the capablities | 13:02 |
| opendevreview | Takashi Kajinami proposed openstack/nova master: libvirt: Use firmware auto-selection by libvirt https://review.opendev.org/c/openstack/nova/+/969132 | 13:04 |
| gokhan | Thanks for the insights on vEncrypt and the 8-character limit of VNC passwords. sean-k-mooney. we have a specific "Session Hijacking" concern. When a legitimate user requests a console URL, logs into the Guest OS, and then someone else copies that exact URL (with the token) to another browser/machine, they find themselves already logged into the session without needing any credentials. | 13:12 |
| gokhan | Is there a way to enforce "One-Time Use" for console tokens, so that once a WebSocket connection is established, the token becomes invalid for any subsequent connection attempts? | 13:12 |
| sean-k-mooney | gokhan: so the mitigation for that is intened to be using ssl | 13:13 |
| sean-k-mooney | i.e. so that the console token cannot eb intersepted over the wire | 13:13 |
| sean-k-mooney | if your worred about client side hijacts in the browser then that diffent | 13:13 |
| sean-k-mooney | and now we do not have the consept of one time use tokens | 13:13 |
| sean-k-mooney | one time use tokes woudl break browser reresh or when you pop the console out into its own tab in horizong but it could eb something we consider supportign eventually | 13:14 |
| sean-k-mooney | it would not be soemthign we could enabel by default IMO based on the end user ux regression it woudl bring | 13:15 |
| opendevreview | sean mooney proposed openstack/os-vif master: Add TAP device pre-creation support for OVS/OVN https://review.opendev.org/c/openstack/os-vif/+/971231 | 13:40 |
| tkajinam | stephenfin, hi do you mind voting +2+A to https://review.opendev.org/c/openstack/osc-placement/+/970058 again ? | 14:48 |
| tkajinam | previous review scores were wiped when I fixed that depends-on link ... | 14:49 |
| tkajinam | stephenfin, oh, gibi++ already approved it :-) | 14:59 |
| opendevreview | Merged openstack/osc-placement master: Add Python 3.13 functional tests https://review.opendev.org/c/openstack/osc-placement/+/970058 | 15:03 |
| gokhan | thanks sean-k-mooney I am worried about client side hijacks in the browser. Now I am considering a workaround at the Load Balancer (HAProxy) level to mitigate the risk of URL sharing/theft. I am trying to implement IP-to-Token Pinning" using HAProxy stick-tables. Catch the token parameter from the VNC URL.Store the Source IP of the first requester for that specific token in a stick-table. Reject (403 Forbidden) any subsequent requests for the same | 15:39 |
| gokhan | token if they originate from a different Source IP. | 15:39 |
| sean-k-mooney | gibi so for the os-vif init its implemnted here https://github.com/openstack/os-vif/blob/master/os_vif/__init__.py#L24-L49 which calls load on the plugin to regester the plugin config options but i dont see anythign that would cause the ovs conenction to be established | 16:07 |
| sean-k-mooney | gibi: so that backgorund heatbeat to vos shodul not be established until the first time we call plug on a vif object that uses ovs | 16:08 |
| sean-k-mooney | i.e. after the service is fully created so it shoudl not be a factor in the service creation workflow | 16:09 |
| sean-k-mooney | i lazy load teh ovs db connection on first use here https://github.com/openstack/os-vif/blob/master/vif_plug_ovs/ovsdb/ovsdb_lib.py#L35-L44 | 16:10 |
| gibi | sean-k-mooney: cool. thanks for checking it | 16:41 |
| *** root is now known as Guest35866 | 17:09 | |
| opendevreview | sean mooney proposed openstack/nova master: Support os-vif TAP pre-creation for OVS/OVN ports https://review.opendev.org/c/openstack/nova/+/973414 | 17:10 |
| opendevreview | sean mooney proposed openstack/nova master: Support os-vif TAP pre-creation for OVS/OVN ports https://review.opendev.org/c/openstack/nova/+/973149 | 17:11 |
| sean-k-mooney | ralonsoh: ^ should work with your chagnes now | 17:14 |
| sean-k-mooney | ignore 973414 i abandoed that the change id got messed up | 17:15 |
| ralonsoh | sean-k-mooney, I'll recheck the testing patch again | 18:27 |
| opendevreview | sean mooney proposed openstack/nova master: FairLockGuard: Support cross-thread sharing and nesting https://review.opendev.org/c/openstack/nova/+/973438 | 20:47 |
| gmaan | dansmith: melwitt: this is one cleanup for RBAC alias, please check whenever you have time. I just realized that it is still not merged https://review.opendev.org/c/openstack/nova/+/968410 | 20:54 |
| opendevreview | Merged openstack/nova master: Upgrade note for concurrency mode default change https://review.opendev.org/c/openstack/nova/+/969888 | 21:23 |
Generated by irclog2html.py 4.0.0 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!