| ildikov | lol, wow | 00:01 |
|---|---|---|
| ildikov | there's technically no usable information in that one | 00:02 |
| sean-k-mooney | i was hopign it was a duplicate of https://bugs.launchpad.net/nova/+bug/2088341 since it has the same title but that is fileed by someone else | 00:03 |
| ildikov | I mean, with the amount of information in the first one, saying it's a duplicate is just as accurate as saying it's not | 00:05 |
| ildikov | maybe the two people know each other and worked smh together :) | 00:06 |
| sean-k-mooney | i think they just took oru generic api error header too litrally | 00:07 |
| sean-k-mooney | "Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible." | 00:07 |
| sean-k-mooney | some other software project put an identifyer that allows them to track wehre the issue came form in messages like that | 00:08 |
| s3rj1k | reporter fact-proved that "Unexpected API Error" exists | 00:08 |
| sean-k-mooney | hopefully there issue either went away or they cna come back and tell us a littel more | 00:09 |
| sean-k-mooney | anywya im going to drop again for the night o/ | 00:09 |
| ildikov | ah, I see | 00:12 |
| ildikov | I have my fingers crossed they'll come back if the issue persists! | 00:13 |
| ildikov | thanks again for checking my bug report! | 00:13 |
| ildikov | good night! | 00:13 |
| opendevreview | melanie witt proposed openstack/nova master: Add [quota]unified_limits_resource_(strategy|list) https://review.opendev.org/c/openstack/nova/+/924025 | 02:19 |
| tkajinam | bauzas, hi could you check https://review.opendev.org/c/openstack/releases/+/935892 when you have time ? | 09:07 |
| bauzas | tkajinam: just did | 09:09 |
| tkajinam | bauzas, thanks 1 | 09:21 |
| tkajinam | ! | 09:21 |
| opendevreview | renliang proposed openstack/nova master: Update documentation link for openSUSE index. https://review.opendev.org/c/openstack/nova/+/927705 | 09:48 |
| stephenfin | gmann: Thanks for the very thorough review on https://review.opendev.org/c/openstack/nova/+/915743 That must have taken you a while. I appreciate it and will respin quickly 🙏 | 12:37 |
| ratailor | bauzas, sean-k-mooney could you please check https://review.opendev.org/c/openstack/nova/+/934294 https://review.opendev.org/c/openstack/nova/+/873901 | 12:38 |
| ratailor | gibi, could you please check this https://review.opendev.org/c/openstack/nova/+/927732 its clean cherry-pick and required 1 more +2. | 12:42 |
| gibi | ratailor: done | 12:47 |
| ratailor | gibi, Thanks! | 12:47 |
| ratailor | gibi, also is this approach looks good. WDYT https://review.opendev.org/c/openstack/nova/+/933107 | 12:50 |
| s3rj1k | sean-k-mooney: can you please take a look at https://bugs.launchpad.net/nova/+bug/2089386 when you have time, thanks | 14:06 |
| sean-k-mooney | s3rj1k: so we dont really recomemnt usign the periodic in production | 14:08 |
| sean-k-mooney | you can but it does not scale | 14:08 |
| sean-k-mooney | the database constraits should be sufficent to have correct behaivor even without a distributed lock | 14:09 |
| s3rj1k | sean-k-mooney: even without periodics there is an issue with concurrency for running multiple CLI jobs | 14:09 |
| sean-k-mooney | how so. the unique constraits will fail to allow multipel cli jobs to add the same host | 14:10 |
| sean-k-mooney | so one will just fail and the other will succeed | 14:10 |
| sean-k-mooney | s3rj1k: nova would like to remove our use of tooz in general as an fyi | 14:11 |
| sean-k-mooney | its currently only used for the ironic driver and the functionaltiy that uses it is drepcated for removal | 14:11 |
| s3rj1k | hmm, so maybe periodics should be marked as deprecated? | 14:12 |
| sean-k-mooney | s3rj1k: we could impove this behaivor but our current docs say that you may only enable the perodic in one schduler instnace | 14:12 |
| sean-k-mooney | so this is a known limitation in the curent design | 14:12 |
| sean-k-mooney | s3rj1k: the enhancement is not invlaid but its somethin we have decied not to adress in the past | 14:13 |
| s3rj1k | this is for HA setups, where there are multiple schedulers, so there is a need to have at least 3 periodics enabled | 14:13 |
| sean-k-mooney | right so thats not supported today | 14:14 |
| s3rj1k | think Nova in k8s basically | 14:14 |
| sean-k-mooney | yep in our new k8s operator we expiclty dont use this because of this limitation | 14:14 |
| sean-k-mooney | currntly we requre the human ot manualy run it if they scale out comptue nodes | 14:14 |
| sean-k-mooney | given that is an infrequet operation | 14:15 |
| sean-k-mooney | gibi: ^ you might be interested in thsi conversation | 14:15 |
| s3rj1k | it would be nicer to remove a need of manual tasking here | 14:15 |
| sean-k-mooney | s3rj1k: a better appoch might be to make the mappign code more robost | 14:16 |
| sean-k-mooney | i.e. to catch the db conflict/duplicate key error and proceed with the rest of the mappings | 14:16 |
| sean-k-mooney | s3rj1k: for what its worth we did consier automating it in a k8s chron job | 14:18 |
| s3rj1k | what about periodics? I mean we should decide what we should do with it, having this around and saying that it should not be used looks a bit off | 14:18 |
| sean-k-mooney | that not what we say | 14:18 |
| sean-k-mooney | we say it can be used but must only be enabled in one schduler instance | 14:18 |
| sean-k-mooney | and htat for large deployment its avised to not run it frequently | 14:18 |
| s3rj1k | same a basically setting up plain Linux cron with CLI | 14:19 |
| sean-k-mooney | but yes we coudl deprecate it for removal, but i dont think we want to break existing users | 14:19 |
| sean-k-mooney | s3rj1k: yes and no. the delta is that its ha | 14:19 |
| sean-k-mooney | i.e. as long asa you k8s cluster is functional it can schduler to any approate worker | 14:20 |
| s3rj1k | I mean in case of single node setups, cron vs periodic is same thin, no? | 14:20 |
| sean-k-mooney | the reason we didnt do that of our installer is we did not think this was a good default for production | 14:20 |
| sean-k-mooney | sure but then you cant have the conflcit | 14:20 |
| sean-k-mooney | unless your running multipel schduler on a single node cluster? doable but proably not super useful | 14:21 |
| s3rj1k | yea, point is case env setups what are ok to have periodics enabled are not HA at all and there is no point in supporting periodics in case of not also supporting HA with it, since plain cron can replace it on non HA | 14:22 |
| s3rj1k | so effectively dead code in terms of production, fine for small setups only or dev clusters | 14:23 |
| sean-k-mooney | so where this is used in production today its not enabeld on multiple hosts | 14:23 |
| sean-k-mooney | but i agree its not as useful as it could be | 14:23 |
| sean-k-mooney | s3rj1k: i dont think the request for enhancement is invlaid but pulling in tooz for jsut this is a hard cell | 14:24 |
| sean-k-mooney | *sell | 14:25 |
| sean-k-mooney | we generally want the scheuler ot be stateless and nova does not use any distibuted lockign today partly for horizontal scaleableity | 14:25 |
| sean-k-mooney | we may be able to do https://en.wikipedia.org/wiki/Rendezvous_hashing | 14:26 |
| sean-k-mooney | so that only one schdluler will try to map a given host on any given invocation but that woudl make the time to mapping less deterministic so im conflicted on that | 14:27 |
| sean-k-mooney | s3rj1k: for right now im going to set the bug to Opinion but i think it would be good to either bring this up in the next team metting, ask others for input on irc or start a mailing list thread on this topic | 14:39 |
| sean-k-mooney | s3rj1k: at a minium we could imporve the docuemation aroudn this | 14:39 |
| sean-k-mooney | we may be able to do more but adding a distribute lock manager to nova is definally a feature not a bug | 14:40 |
| s3rj1k | yea, feature, tagged as rfe | 14:42 |
| s3rj1k | for sure nit a bug | 14:42 |
| sean-k-mooney | another approch might be to catch the "host is not mapped to a cell" excption w eget today and use that to trigger a mapping reactively | 14:42 |
| s3rj1k | also would like to note that this is no k8s specific and oldschool phys servers HA setup is probably even more impacted by this | 14:44 |
| sean-k-mooney | sure as i said this is an old topic | 14:45 |
| sean-k-mooney | the tension is that this really only make sense in an env wehre the number of compute nodes changes frequently | 14:45 |
| sean-k-mooney | in such a env setting along interval is not palatable | 14:45 |
| sean-k-mooney | so you have to set the period to be run often | 14:45 |
| sean-k-mooney | which mean it adds a lot of load espcitlly if you run multipel isntances of the scduelr each doing it on the db | 14:46 |
| sean-k-mooney | so the more frequently you add comptues, the more often you have to run it, and the more load it creates, which also gets more expensive the more host you have as there are more hosts to process | 14:47 |
| sean-k-mooney | s3rj1k: in our operator we orgingally hoped to have the golang reconsile loop run the host mapping when compute were added | 14:48 |
| sean-k-mooney | but architecturla design reasons prevent that form happing today | 14:49 |
| sean-k-mooney | s3rj1k: im not sure if you have the same desgin limistiaon | 14:49 |
| s3rj1k | I guess that the baremininum thing is to ensure that CLI Jobs are running sequentially | 14:50 |
| sean-k-mooney | tl;dr our nova-operator is not allowed to monitor the addtion/removal of compute nodes, and our compute node deployment code is not allowed to acess the db (run nova-manage) or talk to k8s | 14:50 |
| s3rj1k | but for that it would be nice to have some more visible descision around internal periodics, as they always come up | 14:51 |
| sean-k-mooney | s3rj1k: melwitt had a propsoal for a seperate nova-audit deamon a long time ago to have a seperate process dedicated to running them | 14:52 |
| sean-k-mooney | we did not move forward with that but the idea wwas to move this perodic and the db archive/peruge operatiosn to that seperate optional demaon | 14:52 |
| sean-k-mooney | the idea was to centralise all of the perodic mantance and self healign task into a seperatly deployable but optional compoent | 14:54 |
| s3rj1k | and how to handle HA in this case? | 14:54 |
| sean-k-mooney | intially active passive so in k8s you would only run 1 as deplyment/statefulset | 14:55 |
| sean-k-mooney | for non k8s deploymetsn you would use pacemaker ectra to manage which one was running | 14:55 |
| sean-k-mooney | you could use a DLM as well | 14:55 |
| sean-k-mooney | but given this is not a performace critical part that is less concerning | 14:56 |
| s3rj1k | so a lot of extra things to hack this up in case of plain servers | 14:59 |
| s3rj1k | for k8s case, active/passive could probably work | 15:00 |
| sean-k-mooney | yep as i siad this is old ground but there is a renewd interst in makign nova be more "ks8 native" | 15:00 |
| sean-k-mooney | s3rj1k: This was the last time nova-audit was serisoly dicussed https://specs.openstack.org/openstack/nova-specs/specs/xena/approved/nova-audit.html | 15:01 |
| sean-k-mooney | s3rj1k: we discussed soem other k8s painpoint like graceful shutdown too at the last ptg | 15:02 |
| sean-k-mooney | and healthchecks | 15:02 |
| s3rj1k | all of those would be very nice to have, yes | 15:02 |
| sean-k-mooney | so i intentilaly didnt close https://bugs.launchpad.net/nova/+bug/2089386 as invialid, but hostircally we have confired issues like this as out of scope | 15:02 |
| sean-k-mooney | s/confired/considered/ | 15:04 |
| sean-k-mooney | s3rj1k: i need to go finsih something but do bring this up in the next irc meeting or start a mailing list thrad about how to move this forward to get input form others | 15:04 |
| s3rj1k | sure, let's let it brew over a weekend, I'll ping back on this in next meetup | 15:06 |
| s3rj1k | have a nice weekend! | 15:06 |
| sean-k-mooney | o/ | 15:06 |
| opendevreview | ribaudr proposed openstack/nova master: Attach Manila shares via virtiofs (manila abstraction) https://review.opendev.org/c/openstack/nova/+/831194 | 15:11 |
| opendevreview | ribaudr proposed openstack/nova master: Use client token when talking to manila https://review.opendev.org/c/openstack/nova/+/925277 | 15:11 |
| opendevreview | ribaudr proposed openstack/nova master: Attach Manila shares via virtiofs (drivers and compute manager part) https://review.opendev.org/c/openstack/nova/+/833090 | 15:11 |
| opendevreview | ribaudr proposed openstack/nova master: Mounting the shares as part of the initialization process https://review.opendev.org/c/openstack/nova/+/880075 | 15:11 |
| opendevreview | ribaudr proposed openstack/nova master: Deletion of associated share mappings on instance deletion https://review.opendev.org/c/openstack/nova/+/881472 | 15:12 |
| opendevreview | ribaudr proposed openstack/nova master: Add metadata for shares https://review.opendev.org/c/openstack/nova/+/850500 | 15:12 |
| opendevreview | ribaudr proposed openstack/nova master: Add share_info parameter to reboot method for each driver (driver part) https://review.opendev.org/c/openstack/nova/+/854823 | 15:12 |
| opendevreview | ribaudr proposed openstack/nova master: Support rebooting an instance with shares (compute manager part) https://review.opendev.org/c/openstack/nova/+/854824 | 15:12 |
| opendevreview | ribaudr proposed openstack/nova master: Add share_info parameter to resume method for each driver (driver part) https://review.opendev.org/c/openstack/nova/+/860284 | 15:12 |
| opendevreview | ribaudr proposed openstack/nova master: Support resuming an instance with shares (compute manager part) https://review.opendev.org/c/openstack/nova/+/860285 | 15:12 |
| opendevreview | ribaudr proposed openstack/nova master: Add helper methods to rescue/unrescue shares https://review.opendev.org/c/openstack/nova/+/860286 | 15:12 |
| opendevreview | ribaudr proposed openstack/nova master: Support rescuing an instance with shares https://review.opendev.org/c/openstack/nova/+/860287 | 15:12 |
| opendevreview | ribaudr proposed openstack/nova master: Allow to mount manila share using Cephfs protocol https://review.opendev.org/c/openstack/nova/+/883862 | 15:12 |
| opendevreview | ribaudr proposed openstack/nova master: Check shares support (compute manager) https://review.opendev.org/c/openstack/nova/+/885751 | 15:12 |
| opendevreview | ribaudr proposed openstack/nova master: Attach Manila shares via virtiofs (API) https://review.opendev.org/c/openstack/nova/+/836830 | 15:12 |
| opendevreview | ribaudr proposed openstack/nova master: Add helper methods to attach/detach shares https://review.opendev.org/c/openstack/nova/+/885753 | 15:12 |
| opendevreview | ribaudr proposed openstack/nova master: Add instance.share_attach notification https://review.opendev.org/c/openstack/nova/+/850501 | 15:12 |
| opendevreview | ribaudr proposed openstack/nova master: Add instance.share_detach notification https://review.opendev.org/c/openstack/nova/+/851028 | 15:12 |
| opendevreview | ribaudr proposed openstack/nova master: Add shares to InstancePayload https://review.opendev.org/c/openstack/nova/+/851029 | 15:12 |
| opendevreview | ribaudr proposed openstack/nova master: Add instance.share_attach_error notification https://review.opendev.org/c/openstack/nova/+/860282 | 15:12 |
| opendevreview | ribaudr proposed openstack/nova master: Add instance.share_detach_error notification https://review.opendev.org/c/openstack/nova/+/860283 | 15:12 |
| opendevreview | ribaudr proposed openstack/nova master: Reports instance events to the DB regarding attaching and detaching a share https://review.opendev.org/c/openstack/nova/+/927088 | 15:12 |
| opendevreview | ribaudr proposed openstack/nova master: Add libvirt test to ensure metadata are working. https://review.opendev.org/c/openstack/nova/+/852086 | 15:12 |
| opendevreview | ribaudr proposed openstack/nova master: Add virt/libvirt error test cases https://review.opendev.org/c/openstack/nova/+/852087 | 15:12 |
| opendevreview | ribaudr proposed openstack/nova master: Manila shares admin guide documentation https://review.opendev.org/c/openstack/nova/+/871642 | 15:12 |
| opendevreview | ribaudr proposed openstack/nova master: Refactor test_server_shares: Mock in Base Class and trait verification https://review.opendev.org/c/openstack/nova/+/935861 | 15:12 |
| opendevreview | Elod Illes proposed openstack/nova stable/2024.1: [stable-only] Set grenade-skip-level to non-voting https://review.opendev.org/c/openstack/nova/+/936033 | 15:55 |
| opendevreview | Stephen Finucane proposed openstack/nova master: api: Add response body schemas for server action APIs https://review.opendev.org/c/openstack/nova/+/915742 | 16:53 |
| opendevreview | Stephen Finucane proposed openstack/nova master: api: Add response body schemas for remaining server action APIs https://review.opendev.org/c/openstack/nova/+/915743 | 16:53 |
| opendevreview | Stephen Finucane proposed openstack/nova master: api: Add response body schemas for host aggregate actions API https://review.opendev.org/c/openstack/nova/+/924584 | 16:53 |
| opendevreview | Stephen Finucane proposed openstack/nova master: api: Add response body schemas for host aggregate APIs https://review.opendev.org/c/openstack/nova/+/924585 | 16:53 |
| opendevreview | Stephen Finucane proposed openstack/nova master: api: Add response body schemas for extensions API https://review.opendev.org/c/openstack/nova/+/924586 | 16:53 |
| opendevreview | Stephen Finucane proposed openstack/nova master: api: Add response body schemas for versions APIs https://review.opendev.org/c/openstack/nova/+/924587 | 16:53 |
| opendevreview | Stephen Finucane proposed openstack/nova master: api: Add response body schemas for assisted volume snapshots APIs https://review.opendev.org/c/openstack/nova/+/924588 | 16:53 |
| opendevreview | Stephen Finucane proposed openstack/nova master: api: Add response body schemas for port interface APIs https://review.opendev.org/c/openstack/nova/+/924589 | 16:53 |
| opendevreview | Stephen Finucane proposed openstack/nova master: api: Add response body schemas for availability zone APIs https://review.opendev.org/c/openstack/nova/+/924590 | 16:53 |
| opendevreview | Stephen Finucane proposed openstack/nova master: api: Add response body schemas for bare metal node APIs https://review.opendev.org/c/openstack/nova/+/924591 | 16:53 |
| opendevreview | Stephen Finucane proposed openstack/nova master: api: Add response body schemas for console auth token APIs https://review.opendev.org/c/openstack/nova/+/924592 | 16:53 |
| opendevreview | Stephen Finucane proposed openstack/nova master: api: Add response body schemas for flavor access API https://review.opendev.org/c/openstack/nova/+/924593 | 16:53 |
| opendevreview | Stephen Finucane proposed openstack/nova master: api: Add response body schemas for flavor extra specs APIs https://review.opendev.org/c/openstack/nova/+/924594 | 16:53 |
| opendevreview | Stephen Finucane proposed openstack/nova master: api: Add response body schemas for flavors APIs https://review.opendev.org/c/openstack/nova/+/924595 | 16:53 |
| opendevreview | Stephen Finucane proposed openstack/nova master: api: Add response body schemas for floating IP pool APIs https://review.opendev.org/c/openstack/nova/+/924596 | 16:53 |
| opendevreview | melanie witt proposed openstack/nova master: Add [quota]unified_limits_resource_(strategy|list) https://review.opendev.org/c/openstack/nova/+/924025 | 19:43 |
| gmann | stephenfin: yeah, rebuild schema is with all microvesion changes is complex one. As this looks good, other schema should be quick one | 20:36 |
| opendevreview | Merged openstack/nova master: Attach Manila shares via virtiofs (manila abstraction) https://review.opendev.org/c/openstack/nova/+/831194 | 20:37 |
| gmann | waiting for gate to finish the tests but I am targeting to review this series today or before Monday otherwise I keep distracting from other things | 20:38 |
| opendevreview | Takashi Kajinami proposed openstack/nova master: Replace distutils https://review.opendev.org/c/openstack/nova/+/936067 | 20:41 |
| opendevreview | melanie witt proposed openstack/nova master: Add [quota]unified_limits_resource_(strategy|list) https://review.opendev.org/c/openstack/nova/+/924025 | 20:51 |
| opendevreview | Merged openstack/nova master: Use client token when talking to manila https://review.opendev.org/c/openstack/nova/+/925277 | 21:23 |
| opendevreview | Merged openstack/nova master: Attach Manila shares via virtiofs (drivers and compute manager part) https://review.opendev.org/c/openstack/nova/+/833090 | 21:28 |
| opendevreview | Merged openstack/nova master: api: Add response body schemas for server action APIs https://review.opendev.org/c/openstack/nova/+/915742 | 22:06 |
| opendevreview | melanie witt proposed openstack/nova master: Add [quota]unified_limits_resource_(strategy|list) https://review.opendev.org/c/openstack/nova/+/924025 | 22:09 |
| opendevreview | Ghanshyam proposed openstack/nova master: api: Add response body schemas for remaining server action APIs https://review.opendev.org/c/openstack/nova/+/915743 | 22:24 |
| opendevreview | Stephen Finucane proposed openstack/nova master: api: Add response body schemas for host aggregate actions API https://review.opendev.org/c/openstack/nova/+/924584 | 23:01 |
| opendevreview | Stephen Finucane proposed openstack/nova master: api: Add response body schemas for host aggregate APIs https://review.opendev.org/c/openstack/nova/+/924585 | 23:01 |
| opendevreview | Stephen Finucane proposed openstack/nova master: api: Add response body schemas for extensions API https://review.opendev.org/c/openstack/nova/+/924586 | 23:01 |
| opendevreview | Stephen Finucane proposed openstack/nova master: api: Add response body schemas for versions APIs https://review.opendev.org/c/openstack/nova/+/924587 | 23:01 |
| opendevreview | Stephen Finucane proposed openstack/nova master: api: Add response body schemas for assisted volume snapshots APIs https://review.opendev.org/c/openstack/nova/+/924588 | 23:01 |
| gmann | stephenfin: done with the review, ^^ i rebased some of them but some are left. will check those once updated. | 23:48 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!