Wednesday, 2021-06-09

*** martinkennelly_ has quit IRC00:02
*** martinkennelly has quit IRC00:02
*** derekh has quit IRC00:04
opendevreviewMerged openstack/nova master: Remove unnecessary mocks from unit test  https://review.opendev.org/c/openstack/nova/+/79463700:23
*** rloo has quit IRC00:23
*** rloo has joined #openstack-nova00:24
opendevreviewMerged openstack/nova master: docs: Address TODO  https://review.opendev.org/c/openstack/nova/+/79236300:24
*** suzhengwei__ has joined #openstack-nova00:47
opendevreviewMerged openstack/nova master: Create a fixture around fake_notifier  https://review.opendev.org/c/openstack/nova/+/75844600:54
opendevreviewMerged openstack/nova master: Use NotificationFixture for legacy notifications too  https://review.opendev.org/c/openstack/nova/+/75844800:55
*** brinzhang has joined #openstack-nova00:55
*** brinzhang_ has quit IRC01:00
*** suzhengwei__ has quit IRC01:02
*** suzhengwei__ has joined #openstack-nova01:02
opendevreviewmelanie witt proposed openstack/nova stable/ussuri: zuul: Replace grenade and nova-grenade-multinode with grenade-multinode  https://review.opendev.org/c/openstack/nova/+/79467501:20
opendevreviewmelanie witt proposed openstack/nova stable/ussuri: zuul: Start to migrate nova-live-migration to zuulv3  https://review.opendev.org/c/openstack/nova/+/79543201:20
opendevreviewmelanie witt proposed openstack/nova stable/ussuri: zuul: Replace nova-live-migration with zuulv3 jobs  https://review.opendev.org/c/openstack/nova/+/79543301:20
opendevreviewmelanie witt proposed openstack/nova stable/ussuri: Remove broken legacy zuul jobs  https://review.opendev.org/c/openstack/nova/+/79537401:22
*** brinzhang has quit IRC01:22
*** brinzhang has joined #openstack-nova01:23
*** suzhengwei__ has quit IRC01:32
opendevreviewmelanie witt proposed openstack/nova stable/train: zuul: Replace grenade and nova-grenade-multinode with grenade-multinode  https://review.opendev.org/c/openstack/nova/+/79468601:35
opendevreviewmelanie witt proposed openstack/nova stable/train: zuul: Start to migrate nova-live-migration to zuulv3  https://review.opendev.org/c/openstack/nova/+/79543501:35
opendevreviewmelanie witt proposed openstack/nova stable/train: zuul: Replace nova-live-migration with zuulv3 jobs  https://review.opendev.org/c/openstack/nova/+/79543601:35
*** suzhengwei__ has joined #openstack-nova02:03
*** suzhengwei__ has quit IRC04:17
*** vishalmanchanda has joined #openstack-nova04:37
*** abhishekk has joined #openstack-nova04:47
*** brinzhang_ has joined #openstack-nova05:16
*** brinzhang0 has joined #openstack-nova05:17
*** brinzhang has quit IRC05:22
*** brinzhang_ has quit IRC05:24
*** suzhengwei__ has joined #openstack-nova05:31
*** luksky has joined #openstack-nova05:56
opendevreviewYongli He proposed openstack/nova master: Smartnic support - cyborg drive  https://review.opendev.org/c/openstack/nova/+/77136206:06
opendevreviewYongli He proposed openstack/nova master: smartnic support - new vnic type  https://review.opendev.org/c/openstack/nova/+/77136306:06
opendevreviewYongli He proposed openstack/nova master: smartnic support  https://review.opendev.org/c/openstack/nova/+/75894406:06
opendevreviewYongli He proposed openstack/nova master: smartnic support - reject server move and suspend  https://review.opendev.org/c/openstack/nova/+/77991306:06
opendevreviewYongli He proposed openstack/nova master: smartnic support - functional tests  https://review.opendev.org/c/openstack/nova/+/78014706:06
*** whoami-rajat has joined #openstack-nova06:12
*** ralonsoh has joined #openstack-nova06:29
*** osmanlic- has joined #openstack-nova06:45
*** osmanlicilegi has quit IRC06:45
*** dklyle has quit IRC06:58
*** mdbooth has quit IRC06:58
*** osmanlic- has quit IRC07:01
gibilyarwood: hi! Please file a bp for https://review.opendev.org/c/openstack/nova-specs/+/794799 then I will +207:02
*** osmanlicilegi has joined #openstack-nova07:05
*** andrewbonney has joined #openstack-nova07:16
opendevreviewMerged openstack/nova-specs master: Re-propose Unified Limits in Nova  https://review.opendev.org/c/openstack/nova-specs/+/78958507:17
opendevreviewWenping Song proposed openstack/nova master: Replaces tenant_id with project_id from List/Update Servers APIs  https://review.opendev.org/c/openstack/nova/+/76429207:19
*** swp20 has joined #openstack-nova07:22
lyarwoodgibi:  yup will do shortly07:35
gibithanks07:40
lyarwoodI shouldn't say shortly, I'm currently stuck in standing traffic after the school run. \o/07:45
gibilyarwood: ohh. no worries.07:49
gibibauzas: I left a suggestion in the mdev spec https://review.opendev.org/c/openstack/nova-specs/+/792796 about trait support.07:49
gibi(I know bauzas is on PTO)07:49
*** tosky has joined #openstack-nova07:53
opendevreviewBalazs Gibizer proposed openstack/nova-specs master: Fix the bp link in the cyborg admin token spec  https://review.opendev.org/c/openstack/nova-specs/+/79549308:00
*** martinkennelly has joined #openstack-nova08:05
*** martinkennelly_ has joined #openstack-nova08:05
*** derekh has joined #openstack-nova08:08
*** lucasagomes has joined #openstack-nova08:10
opendevreviewWenping Song proposed openstack/nova stable/train: Change get console auth token policy to RULE_ADMIN_OR_OWNER  https://review.opendev.org/c/openstack/nova/+/79549608:21
*** abhishekk has quit IRC08:23
*** suzhengwei__ has quit IRC09:11
*** suzhengwei__ has joined #openstack-nova09:11
*** rpittau|afk is now known as rpittau09:22
opendevreviewBalazs Gibizer proposed openstack/nova master: [DNM]Test with sqla 1.4 and try to tease out slow tests  https://review.opendev.org/c/openstack/nova/+/79550609:24
*** belmoreira has joined #openstack-nova09:28
*** mdbooth has joined #openstack-nova09:42
stephenfingibi++ Thanks for the review on the RPC 6.0 cleanup series. Delighted to not have to rebase that again :-D09:45
opendevreviewLee Yarwood proposed openstack/nova-specs master: Add attachmentId to responses of the os-volume_attachments API  https://review.opendev.org/c/openstack/nova-specs/+/79479909:49
lyarwoodgibi: would you mind hitting https://review.opendev.org/c/openstack/nova/+/794639 and https://review.opendev.org/c/openstack/nova/+/793463 if you have time today please?09:54
lyarwoodhappy to help with any reviews in return09:54
gibilyarwood: queued them up09:55
lyarwoodmany thanks09:55
opendevreviewLuigi Toscano proposed openstack/nova stable/ussuri: Remove broken legacy zuul jobs  https://review.opendev.org/c/openstack/nova/+/79537410:02
sean-k-mooneythis is not something we can do right https://review.opendev.org/c/openstack/nova/+/79549610:08
sean-k-mooneyam is gerrit broken?10:10
gibisean-k-mooney: I agree that patch seems strange10:12
gibiat least is needs a lot of arguing in the commit message why the author wants this10:13
opendevreviewBalazs Gibizer proposed openstack/nova master: [DNM]Test with sqla 1.4 and try to tease out slow tests  https://review.opendev.org/c/openstack/nova/+/79550610:13
sean-k-mooneygibi: well ideally if they did want this it would be done on master first not on stable/train10:14
gibisean-k-mooney: yes.10:14
sean-k-mooneywith either a bug or blueprint10:14
gibisean-k-mooney: and zuul will also vote -1 as it is not a backport10:14
gibiso we have automatic rejection for such cahnges10:14
gibichanges10:14
sean-k-mooneywhat this is doing is allowing you to look up the connection details if you happen to fined a token10:14
sean-k-mooneygibi: beacuase of the pep8 cherry-pick check or something else10:15
sean-k-mooneyi was not aware of sometrhing that did that automatically10:15
gibisean-k-mooney: yepp10:16
gibisean-k-mooney: we have our own script10:16
gibisean-k-mooney: as part of pep810:16
gibihttps://github.com/openstack/nova/blob/master/tools/check-cherry-picks.sh10:16
sean-k-mooneyi tought the cherry pick script would only detect this if you had cherry-pick lines10:16
sean-k-mooneyi did not think it would run on this patch10:16
gibisean-k-mooney: it reject things that does not have cherry-pick lines and does not have [stable only] heading10:17
sean-k-mooneyah  echo 'Stable branch requires either cherry-pick -x headers or [stable-only] tag!'10:17
sean-k-mooneyya10:17
sean-k-mooneyok i did not think we had that last bit. neet10:18
swp20sean-k-mooney: recently, our client meet the problem that the instance's owner has no permission to get console auth token with T version. From U we have used new policy rule, so i propose this patch base on Train branch: https://review.opendev.org/c/openstack/nova/+/79549610:30
*** kashyap has joined #openstack-nova10:31
* kashyap was accidentally hanging out on the Libera -nova network10:31
sean-k-mooneyswp20: that cahnge does not allow them to get a console auth token10:31
sean-k-mooneyswp20: it allows them to use a token to get the connection info10:32
swp20we have tested this change works.10:32
sean-k-mooneywell you can do it in you policy.json without the code change but that is not what the rule is for10:33
sean-k-mooneythat is for https://docs.openstack.org/api-ref/compute/?expanded=show-console-connection-information-detail#show-console-connection-information10:34
lyarwoodsean-k-mooney: the host there is the compute host right?10:36
sean-k-mooneyi think its the proxy10:37
sean-k-mooneybut it could be10:37
swp20sean-k-mooney: how can i do in policy.json? does the instance's own donnot has the permission?10:45
sean-k-mooneythis is what i would have used in the past https://docs.openstack.org/api-ref/compute/?expanded=get-vnc-console-os-getvncconsole-action-deprecated-detail#get-vnc-console-os-getvncconsole-action-deprecated10:49
sean-k-mooneythat will give you the console url and the auth token10:49
sean-k-mooneythe newer way woudl be to use https://docs.openstack.org/api-ref/compute/?expanded=create-console-detail#create-console10:51
*** abhishekk has joined #openstack-nova10:52
swp20this is no what we want.10:53
sean-k-mooneyswp20: to override the policy in policy.json you just do "rule name":["rules"]10:53
sean-k-mooneyswp20: it is what you said your customer wanted the creaqte api will return a new auth token/url for the server console10:54
swp20we have two platform A and B, A platform want use the console token to visit the instance on B platform, we need the host and port with the api https://docs.openstack.org/api-ref/compute/?expanded=show-console-connection-information-detail,create-console-detail#show-console-connection-information10:56
sean-k-mooneyswp20: why does A not just use the create api on b to get the url and token10:57
sean-k-mooneyswp20: is A unwaware of the server uuid10:57
swp20yes, A is unware of the server uuid.10:57
sean-k-mooneyi see well given the console_token is not intended to be used by normal users as a handel to identify a server im not conviced we should make that change in code10:58
swp20or A use no admin account.10:58
sean-k-mooneyyou can do it locally on your cloud though10:59
sean-k-mooneyA presumable owns the server10:59
sean-k-mooneycorrect10:59
swp20with the new policy rule, dose the system_reader has the permission?11:00
sean-k-mooneyin the patch you added11:01
swp20not in the patch. i find the rule change to SYSTEM_READER after U.11:02
sean-k-mooneyno it does no tsupport the new project scopes and roles11:02
sean-k-mooneycheck_str='is_admin:True or project_id:%(project_id)s',11:02
sean-k-mooney system_reader will not allow the owner of the token to use that endpoint no11:03
swp20ack, thanks.11:03
sean-k-mooneyswp20: this api endpoint was only intended to help admins debug console issues11:04
swp20but maybe the user has the requirement.11:04
sean-k-mooneyhow does A get the console token for the server on B11:05
sean-k-mooneythe console tokens expire and are only valid for a short period of time so they are not ment to be used to identify the server11:06
sean-k-mooneythe default token time to live is 10 minutes https://docs.openstack.org/nova/latest/configuration/config.html#consoleauth.token_ttl11:07
swp20and the doc has no desciption.11:07
swp20i will ask the client what's his scenes.11:07
swp20i rememered, they just want to visit the instance through host and port.11:08
sean-k-mooneywell a normal end user shoudl not know the host on which the instnace is running11:09
sean-k-mooneyunless its a private cloud that is a pretty big security risk11:09
swp20yeah, it's a really a private cloud.:')11:10
sean-k-mooneythe sample policy.json is rendered here https://docs.openstack.org/nova/latest/configuration/sample-policy.html11:11
sean-k-mooneythey just need to put that in /etc/nova/policy.json and uncomment #"os_compute_api:os-console-auth-tokens": "rule:system_reader_api"11:11
sean-k-mooneythen update the rule11:12
swp20got it, thanks.11:12
sean-k-mooneyjust make sure you customer is aware of the console auth token ttl and the fact that in theory the tokens could be resused11:13
sean-k-mooneyits very unlikely but it can happen11:14
swp20yeah, right.11:14
sean-k-mooneyif they are storing the token in a db somewhere and expecting to be able to use it days later its going to break there usecase11:14
swp20ack11:16
swp20i'll abandon this patch and talk to my customer.11:17
*** suzhengwei__ has quit IRC11:37
*** suzhengwei__ has joined #openstack-nova11:37
*** suzhengwei__ has quit IRC11:43
*** suzhengwei__ has joined #openstack-nova11:43
*** abhishekk has quit IRC11:52
*** mdbooth has quit IRC12:02
*** raildo has joined #openstack-nova12:18
*** raildo has quit IRC12:18
*** raildo has joined #openstack-nova12:18
*** artom has quit IRC12:28
*** suzhengwei__ has quit IRC12:38
*** Corwin has joined #openstack-nova12:41
opendevreviewStephen Finucane proposed openstack/nova master: requirements: Add typing-paramiko  https://review.opendev.org/c/openstack/nova/+/79553312:45
stephenfingate is down - we need that (and the related requirements fix) to unblock it ^12:45
stephenfins/down/broken/12:46
opendevreviewStephen Finucane proposed openstack/nova master: requirements: Add types-paramiko  https://review.opendev.org/c/openstack/nova/+/79553312:49
* lyarwood looks12:56
*** rloo has joined #openstack-nova12:57
lyarwoodstephenfin:  Why the Depends-On?13:03
stephenfinbecause we need to get it in u-c before the requirements job will pass, no?13:03
lyarwoodstephenfin: ah does that actually fail if we don't have a uc for a given test-requirement?13:06
lyarwoodTIL13:06
stephenfinI'm pretty sure it does, yeah13:06
*** raildo has quit IRC13:06
amodilyarwood: hi, can u help out with the issue in https://review.opendev.org/c/openstack/whitebox-tempest-plugin/+/786979, when u can. i don't have a devstack env to debug13:08
lyarwoodstephenfin:  requirements-check looks like it's failing in the nova change13:08
lyarwoodERROR: Package 'types-paramiko' is used in test-requirements.txt but not in lower-constraints.txt13:09
stephenfindrat, will fix13:09
gibistephenfin: what is the way to reproduce the issue?  I run tox -e mypy -r locally but it passes on nova master13:10
lyarwoodamodi: looking13:10
stephenfingibi: that did the trick for me. Only odd thing for me is the use of Python 3.913:11
stephenfinsince it's F3413:11
opendevreviewStephen Finucane proposed openstack/nova master: requirements: Add types-paramiko  https://review.opendev.org/c/openstack/nova/+/79553313:11
stephenfinlyarwood: ^13:11
gibigibizer@riverbed:~/upstream/git/nova [master]                                                                                                                                   [000] [15:09:30]13:11
gibi$ source .tox/shared/bin/activate13:12
gibigibizer@riverbed:~/upstream/git/nova (shared)[master]                                                                                                                           [000] [15:11:46]13:12
gibi$ python --version13:12
gibiI use python 3.9 too13:12
gibiPython 3.9.213:12
gibigibizer@riverbed:~/upstream/git/nova (shared)[master]                                                                                                                           [000] [15:11:48]13:12
stephenfinodd13:12
stephenfinhere's the first failing job I saw https://review.opendev.org/c/openstack/nova/+/77854713:12
stephenfinlook at the pep8 env13:12
sean-k-mooneystephenfin: what is types-paramiko used for13:13
stephenfintype hints for paramiko13:13
sean-k-mooneywhich is what the ssh lib we use13:13
stephenfincorrect13:13
sean-k-mooneyhum its not really a test-requireemnt is it13:14
stephenfinit is. mypy isn't a runtime dependency13:14
sean-k-mooneyoh for the mypy job?13:14
sean-k-mooneyok13:14
sean-k-mooneystephenfin: did you ever add mypy to pre-commit?13:15
stephenfinnot yet, no. Just haven't set aside the time to figure it out13:15
sean-k-mooneyno worries13:15
gibiI cannot reproduce it with pep8 env either. but it seem my machine installs a bit older mypy than the gate job13:16
* gibi cleans cache13:16
toskymelwitt: I think there is a chicken-and-egg problem with the backports, you may either need to squash them (but maybe that would be too much) or set the failing legacy job in the first patch to non-vpting, to have it replaced with a native one with one of the newer patches13:19
gibistephenfin: OK it needs a new mypy version to13:19
gibio13:19
lyarwoodamodi: updated, can you build a devstack env and get this passing before you push another revision?13:19
gibistephenfin: it does not fail with mypy==0.812 but it fails with mypy==0.90113:20
stephenfinhmm, good spot. I saw the release date on types-paramiko was pretty recent so I figured that was it13:20
stephenfinI'll see if I can figure out what changed13:20
lyarwoodso we could cap mypy to unblock the gate instead right?13:21
amodilyarwood: um, i havent deployed one in a long time and how would i deploy with these parameters, ill hv to figure out13:21
sean-k-mooneylyarwood: ya that would work if we use UC for it13:21
stephenfinlyarwood: we'd have to cap it13:22
stephenfinno uncap13:22
stephenfinthe higher version is the issue13:22
sean-k-mooneyits in test-requrimets not installed in the tox.ini13:22
sean-k-mooneyso if we cap it in upper-constratints in the requirements repo it shoudl work13:23
gibistephenfin: "The upcoming switch to modular typeshed in mypy 0.900"13:23
gibithat was hit us13:23
stephenfinahhh13:23
kashyapstephenfin: Saw your review on that display change.  I'll respond in a bit13:23
kashyapThank you13:23
stephenfinwonderful \o/13:23
gibihttps://mypy-lang.blogspot.com/2021/05/the-upcoming-switch-to-modular-typeshed.html13:23
gibiwe are lucky that only one thing was missing so far :)13:24
sean-k-mooney:( non nova type definition we can steal13:24
sean-k-mooneyi wonder is there a way we can tell it to ignore third party libs if the hints are not present13:26
sean-k-mooneyi.e. can we supress the error and maybe make ti a warning13:26
sean-k-mooney"You can also now run mypy --install-types to install all missing stub packages in the mypy run (or the previous run, if you don’t pass any files to check): "13:29
sean-k-mooneywe could look at that or perhaps using  --ignore-missing-imports13:29
stephenfinWe already have that configured via '[mypy] ignore_missing_imports = true' in setup.cfg13:30
stephenfinapparently it doesn't work for things in typeshed13:30
sean-k-mooneyya it is ignored for thing that were previously bundeled13:30
sean-k-mooneyhttps://mypy-lang.blogspot.com/2021/06/13:30
sean-k-mooneyhttps://mypy-lang.blogspot.com/2021/06/mypy-0900-released.html13:30
sean-k-mooneyoh TypeGuard we can now do user defined narrowing13:31
amodilyarwood: https://review.opendev.org/c/openstack/whitebox-tempest-plugin/+/786979/20/whitebox_tempest_plugin/config.py#13813:31
sean-k-mooneystephenfin: do we want to consider adding mypy --install-types to the command13:34
sean-k-mooneybefore we run mypy13:34
stephenfinhmm, that would be a good short-term fix13:34
stephenfinlong-term though, I suspect these things might want to be tracked?13:35
sean-k-mooneyim not sure13:35
sean-k-mooneythey might want to be yes13:35
sean-k-mooneybut since its just for type checking im not sure13:35
stephenfinfair point13:35
stephenfinlemme try that13:35
*** abhishekk has joined #openstack-nova13:39
sean-k-mooneystephenfin: by the way we could use https://github.com/pre-commit/mirrors-mypy13:40
*** Gues_____ is now known as jamesdenton13:40
opendevreviewRodrigo Barbieri proposed openstack/nova stable/victoria: Error anti-affinity violation on migrations  https://review.opendev.org/c/openstack/nova/+/79554213:41
stephenfin'mypy --install-types' is interactive13:42
stephenfin:(13:42
stephenfinso I have to pipe 'yes' into it13:43
stephenfinlovely13:43
stephenfinsean-k-mooney: https://github.com/python/mypy/issues/10600#issuecomment-85735115213:43
stephenfinsee next comment13:43
stephenfinI wouldn't recommend running --install-types as it currently works in CI, since in the worst case it can almost double the mypy runtime. It also produces noisy output, as mentioned above.13:43
stephenfin</quote>13:43
stephenfinso we can do it to unblock the gate, but it definitely shouldn't stay13:44
sean-k-mooneystephenfin: ok i guess we can just run it once do a freeze before and after and see what it adds13:44
sean-k-mooneyif this happens again13:44
stephenfinSuccessfully installed types-cryptography-0.1.1 types-enum34-0.1.5 types-ipaddress-0.1.2 types-paramiko-0.1.313:44
stephenfinI guess not all of those are actually needed since we don't typecheck everything13:44
sean-k-mooneynot right now at least13:45
sean-k-mooneybut if we expand the checking they proably will be13:45
opendevreviewStephen Finucane proposed openstack/nova master: requirements: Add types-paramiko  https://review.opendev.org/c/openstack/nova/+/79553313:46
opendevreviewStephen Finucane proposed openstack/nova master: tox: Install mypy type stubs  https://review.opendev.org/c/openstack/nova/+/79554413:46
stephenfinlyarwood, gibi: Alternate approach ^13:46
*** spatel has joined #openstack-nova14:11
spatelsean-k-mooney morning! i am trying to delete one of vm which doesn't existing on compute machine (i can't see it on compute host).  i am not able to delete it, any good way to clean up ?14:12
gibistephenfin: https://review.opendev.org/c/openstack/nova/+/795544 doesnt seem to work14:13
*** tosky has quit IRC14:16
* stephenfin looks14:17
stephenfinoh, well then14:17
stephenfinso I guess you need to run it, let it build a cache, then run it again to install the missing deps14:17
stephenfinso that won't work14:18
stephenfinpity :-(14:18
*** tosky has joined #openstack-nova14:19
opendevreviewStephen Finucane proposed openstack/nova master: requirements: Add types-paramiko  https://review.opendev.org/c/openstack/nova/+/79553314:19
stephenfinback to v2 we go14:19
sean-k-mooneyspatel: i think we have  force delete option14:24
spatelhmm --force ?14:24
sean-k-mooneyspatel: if the vm no longere exits on the compute node but is still in the nova db that is what i woudl try next14:25
spatelperfect! let me try14:25
sean-k-mooneywell i was thinking of "nova force-delete <server uuid>"14:25
sean-k-mooneylet me check if its in osc14:25
sean-k-mooneyyes there is a --force14:26
sean-k-mooneyos openstack server delete --force <uuid>14:26
spatelpenstack server delete: error: unrecognized arguments: --force14:27
spatelI am running victoria14:27
spatelnova force-delete works!14:28
spatelsean-k-mooney openstack server delete doesn't have --force option14:29
sean-k-mooneyit does in the latest release14:30
sean-k-mooneyspatel: stephenfin  added it along with closing most of the other gaps recently14:30
*** jparker has quit IRC14:30
spatelmay be available in current release but not in victoria14:31
sean-k-mooneywhen i said recent i ment in the last month or so14:31
sean-k-mooneynova force-delete does exactly the same thing14:32
*** jparker has joined #openstack-nova14:32
opendevreviewsean mooney proposed openstack/osc-placement master: default to max version when no session  https://review.opendev.org/c/openstack/osc-placement/+/79427614:33
sean-k-mooneygibi: filed https://storyboard.openstack.org/#!/story/2008958 and updated ^14:33
sean-k-mooneygibi: let me know if there is anything else i need to do14:33
sean-k-mooneystoryboard is still kind of a pain to naveigate but its got much faster then i rememeber14:34
opendevreviewBalazs Gibizer proposed openstack/osc-placement master: default to max version when no session  https://review.opendev.org/c/openstack/osc-placement/+/79427614:39
gibisean-k-mooney: thanks. just added the task number to the commit message too ^^14:40
gibithis will connect the review to the task in storyboard14:40
sean-k-mooneyoh ok i have seen both but i tought task was optional14:40
gibiand update its state14:40
sean-k-mooneyoh ok14:40
sean-k-mooneycool14:40
sean-k-mooneyam general question14:40
sean-k-mooneyif placment is going to remain a deliverable of compute14:41
sean-k-mooneydo we want to continue to maintain both storyboard and lauchpad14:41
sean-k-mooneyi also just realised the placement repariting spec should have been proposed in the placement repo14:41
sean-k-mooneysince that is where placment specs are ment to be tracked14:42
sean-k-mooneyhttps://docs.openstack.org/placement/latest/specs/index.html14:42
gibisean-k-mooney: I checked the placement repo but it does not have specs folder14:42
sean-k-mooneyits in the docs folder14:42
sean-k-mooneyso that it renders there14:42
gibi /o\14:42
sean-k-mooneyhttps://github.com/openstack/placement/tree/master/doc/source/specs14:43
sean-k-mooneyi kind of forgot this was there until i rememebred a few days ago14:43
gibiOK, I will move the re-prenting spec to the placement repo :/14:44
sean-k-mooneywell i was not going to ask you to do that14:45
sean-k-mooneyjust going forward do we want to keep the split14:45
sean-k-mooneye.g. nova has nova-specs and launchpad placement has in repo specs and sotry borad14:45
sean-k-mooneythis is more a wider team question or mailing list topic14:46
gibiI don't feel the power to do a migration from storyboard14:46
sean-k-mooneybut that bug just reminded me of it14:46
gibiwe have very few incoming things in placemnet14:46
sean-k-mooneytrue14:47
sean-k-mooneydo we want to add it to the meeting adgenda14:47
sean-k-mooneywe have the bug review for nova in lanuch pad14:48
sean-k-mooneydo we have somethign that will do the same for story borad14:48
sean-k-mooneye.g. a linke to the open stories14:48
sean-k-mooneyits pretty stable and well tested so placment bugs are rare14:48
gibisean-k-mooney: please put it up to the agenda. Honestly I don't think it worth the effort to do extra things with placement but it make sense to discuss it14:50
sean-k-mooneyoh i was suggesting adding a "check for open placment bugs" item14:52
sean-k-mooneylike when we do the nova bug review14:52
sean-k-mooneymore just to prompt use to look for placment bugs if tehy areever reported14:52
sean-k-mooneymost get brought up here anyway since nova typicly is what finds them if they exist14:53
gibihonestly I don't belive that people will start looking at open bug reports in placement by that prompt. but I might be negative14:57
sean-k-mooneyok then lets leave it be so15:01
melwitttosky: guh, ok, I was wondering if there might be something like that since the fails. I'll look into a potential squash, thanks for the hint15:03
gibiI just don't have the bandwidth to do a proper care for placement as well, but it seem that placement does not require that level of care most of the time15:04
*** dklyle has joined #openstack-nova15:04
masterpe[m]Hi, we have Train running, yesterday we got the error "No valid host was found. There are not enough hosts available" on our platform. If I count the free RAM and vcpu of the nodes, I should have enough resources left to get the instances started. Is there something that I miss?15:05
melwittsean-k-mooney, gibi: I knew that placement had its own specs repo but I thought it was intentional that proposed spec to the nova-specs repo since the projects are back together again /shrug15:05
melwittgibi: there is a review I have hanging around if you might be able to spare some time, it is not urgent https://review.opendev.org/c/openstack/nova/+/78039515:09
sean-k-mooneymelwitt: ya so i didnt know i we had decied to not use the in repos specs or not15:15
sean-k-mooneye.g. if we had jsut defaulted to nova-spec or if it was intentional15:15
melwittsean-k-mooney: no, I'm not saying there was a decision, I had assumed something like that had occurred because of where the re-parent spec was proposed, but I was wrong15:16
gibimelwitt: queued that up for review15:19
melwittthank you gibi15:20
*** LinPeiWen has quit IRC15:25
opendevreviewPierre Riteau proposed openstack/nova master: Fix typos in minimum version policy docs  https://review.opendev.org/c/openstack/nova/+/79557515:26
melwittmasterpe[m]: you need to look at the debug level logs in the placement api logs and the nova-scheduler logs to see details about why no valid host was found15:26
masterpe[m]To solve it for now I enabled a node that was disabled/marked as spare. So I need to enable debug mode to troubleshoot?15:28
*** suzhengwei__ has joined #openstack-nova15:31
melwittmasterpe[m]: to get full detail, yes. what you can do without debug mode is check whether there is a message like "no allocation candidates" in the nova-scheduler log, if you see that, it's likely cpu/ram/disk (as those resources are filtered by placement service)15:32
melwittif you don't see that, it means one or more of the nova side scheduler filters eliminated all hosts15:33
opendevreviewBalazs Gibizer proposed openstack/placement master: [DNM]: test consumer_type rollback due to consumer_gen conflict  https://review.opendev.org/c/openstack/placement/+/79557715:34
melwittgibi: oh heh, I had wondered if you saw my latest confusing comment. I was thinking the same thing this morning that it would be best to check what happens with a test. thanks for doing that15:36
gibimelwitt: yeah, you are right the whole PUT /allocations/{uuid} call is not a single transaction15:36
gibiso with the current implementation we can leak a consumer_type update from a rejected call :/15:37
gibiunfortunatly create a test case for a real race is too hard15:39
melwittgibi: ah, I see your new comment on the review. yeah ok, agree we need something higher up the stack then in order to not leak (or some other better idea yet to be thought of)  :( I will work on it15:39
gibiit is hard :/15:40
melwittyeah, would require faking things with mocks at best. I didn't explore that yet so not sure how messy or possible15:42
*** swp20 has quit IRC15:44
spatelsean-k-mooney periodically i am seeing this issue on my nova logs not sure if this is bug or not - http://paste.openstack.org/show/806496/15:46
melwittspatel: that error means connection to the database failed while trying to write a service record update (heartbeat), not a bug. error being emitted from pymysql client15:51
spatelmelwitt today i said terraform to build 20 vm and 4 vm failed and stick in BUILD process, when i check their compute nodes i found that error15:52
melwittspatel: that paste wouldn't be directly related to a build failure (that paste is a service record update for a service heartbeat) so you'd need to look at which errors are related to the build fail, if any15:55
*** suzhengwei__ has quit IRC16:01
spatellet me see, may be issue of rabbitmq in that case16:03
melwittdansmith: someone linked this review in the freenode channel and I thought you might have opinion about it. it's a proposal to allow config of rbd direct snapshot to enable an operator to disallow the fast clone, with the use case being the operator is migrating the backend and the fast clone wouldn't honor the glance backend config (I don't fully understand this part) https://review.opendev.org/c/openstack/nova/+/79483716:05
*** lucasagomes has quit IRC16:14
*** rpittau is now known as rpittau|afk16:16
spatelmelwitt also noticed periodically rabbitMQ connection reset error from nova-api - http://paste.openstack.org/show/806498/16:30
spatelis this a normal behavior ?16:31
melwittspatel: it's not uncommon but it's not desirable. I have seen it many times, related to environmental issues like network disruptions. it's also similar to the DBConnectionError you pasted earlier, that one was about the pymysql client failing to be able to connect to the database, these rabbit errors are showing connection reset or closed trying to connect to rabbitmq16:35
spatelThese controller and compute node on same rack16:37
spatellet me restart my whole rabbitMQ cluster16:37
spatelwhen i delete vm they getting stuck in deleting state16:38
*** artom has joined #openstack-nova17:04
*** andrewbonney has quit IRC17:05
dansmithmelwitt: ack, thanks17:14
*** abhishekk has quit IRC17:25
*** ralonsoh has quit IRC17:34
opendevreviewArtom Lifshitz proposed openstack/nova master: docs: Explicitly state lack of support for node renaming  https://review.opendev.org/c/openstack/nova/+/79560717:39
*** derekh has quit IRC17:45
artomHuh, looks like the docs gate is broken on nova/crypto.py17:58
artom^^ somehow failed it17:58
artomActually, just pep8 in general is broken: https://zuul.opendev.org/t/openstack/build/0b3b9315c9cb4d489f2ffb1355145a9918:03
artomAh, it's mypy...18:04
*** whoami-rajat has quit IRC18:13
lyarwoodartom: the fix for that is in the gate18:16
artomlyarwood, oh. And here I was filing a bug and writing a fix.18:16
lyarwoodhttps://review.opendev.org/c/openstack/nova/+/79553318:16
lyarwoodAlways enjoyable when that happens ^_^18:17
artomI mean, no mailing list message, no bug filed, nothing :P18:18
artomHrmm, I wonder if my fix could stiff have value18:18
artomI guess not, since we'll need to manually update requirements anyways?18:21
artomI did it by just adding `python -m mypy --install-types` to tools/mypywrap.sh18:21
lyarwoodYeah I think stephenfin tried that earlier18:23
lyarwoodbut it wasn't enough in the gate for some reason18:23
artomI guess because lower-constraints.txt?18:24
*** hemna has quit IRC18:35
*** belmoreira has quit IRC18:36
*** hemna has joined #openstack-nova18:37
cz3This is probably a trivial question... I wanted to import Nova modules into interactive Python shell, but I cannot figure out how to load the config from default nova.conf. Is `CONF = nova.conf.CONF` supposed to handle this?18:42
cz3I wanted to play around with objects module through interactive shell but the lack of loaded config is blocking me18:43
artomcz3, never tried it myself, but looking at the code, yeah, import nova.conf; CONF = nova.conf.CONF should do it18:45
cz3yeah, that's what I tried... doesn't seem to work. looks like it is still trying to use the defaults. I assume that because it tries to look up things in sqlite instead of mysql that I have in nova.conf18:49
artomMaybe it's not finding the config file, and using the defaults...18:51
artomIn fact, if you're importing it via Python (and not via running a Python executable on the command line), I'm not even sure how it finds the config file to use...18:52
artomI *think* you want to do CONF = nova.conf.CONF(project='nova') in such a case...18:54
artomAnd then it'll look for nova.conf in the "normal" places18:54
cz3yes, I will poke around the nova.conf.CONF, maybe it needs some additional parameters when it is imported into interactive shell18:56
sean-k-mooneycz3: you need to initalise the conf form the file too18:56
sean-k-mooneyif you just import it it will get the default form code18:56
sean-k-mooneywe do this https://github.com/openstack/nova/blob/master/nova/config.py#L94-L9818:57
sean-k-mooneywhere conf is CONF = nova.conf.CONF18:57
sean-k-mooneyall the command lets initalise it here https://github.com/openstack/nova/blob/d64edd3da2336a5c7c8f69cced45272cbaf638a9/nova/cmd/status.py#L37419:02
sean-k-mooneywell liek ^19:02
sean-k-mooneyso "import nova.conf; from nova import config; CONF = nova.conf.CONF; config.parse_args(sys.argv)"19:03
sean-k-mooneycz3: if you do that i think it should work19:03
sean-k-mooneyalthough im not sure you need to initalise rpc or configre the db19:04
cz3in this case I just need the db access19:05
sean-k-mooneywhich looks like its jsut   sqlalchemy_api.configure(CONF)19:06
cz3sean-k-mooney: config.parse_args did the trick19:07
cz3thank you!19:07
sean-k-mooneyin any case i would use https://github.com/openstack/nova/blob/master/nova/cmd/manage.py as a reference19:07
sean-k-mooneyno worries19:07
sean-k-mooney   CONF(argv[1:],19:08
sean-k-mooney         project='nova',19:08
sean-k-mooney         version=version.version_string(),19:08
sean-k-mooney         default_config_files=default_config_files)19:08
sean-k-mooneythat is likely the bit you really needed19:08
sean-k-mooneythat is invokeing the gloabl constuctor for the singelton19:08
cz3I actually aim to add something to nova-manage and send it over for review, just wanted to test few things beforehand19:09
cz3I emailed the openstack-discuss list last week about this, it was re: request_specs regeneration19:09
cz3where I wanted to find a way to sync extra_specs from flavor data into existing instances19:09
sean-k-mooneyoh yes i remember19:10
cz3long time ago I did something like this by directly appending stuff into request_spec json in database, but this time I want to do it the proper way19:10
cz3I know that flavors are supposed to be immutable etc. but in this case I just want the existing request_specs to reflect the actual reality and not make the scheduler migrate instances where it was not supposed to after underlying flavor was updated with extra_specs19:13
cz3I will try to add that to nova-manage and send it over to review via gerrit, maybe it will be good enough to get upstreamed19:15
sean-k-mooneythe main consurn we woudl have it that such an update can lead to the vm being broken or on an invalid host also19:15
sean-k-mooneyon one hand nova manage is admin only and require direct acess to the server hosting the cloud to get the configs19:16
sean-k-mooneybut from a downstream product point of view im not sure we would support customer executing this command19:16
sean-k-mooneywe may be with the caveat tha that if you break things you fix them19:17
sean-k-mooneyits something we would have to think about19:17
cz3yeah, I do share the concern, I don't want to send something that would end up breaking deployments for other people if used improperly19:18
sean-k-mooneyyou proably would need to run som of the flavor/image compatiabity checks before doing the update to make it safe19:18
sean-k-mooneyour add a --yes-i-really-really-mean-it flag19:18
sean-k-mooneyanyway o/ time to go have dinner19:19
cz3sure19:19
cz3thanks again!19:20
*** kashyap has quit IRC19:50
*** vishalmanchanda has quit IRC19:55
*** spatel has quit IRC20:42
*** efried has quit IRC21:16
*** efried has joined #openstack-nova21:21
*** brinzhang_ has joined #openstack-nova21:43
*** brinzhang has joined #openstack-nova21:45
*** brinzhang0 has quit IRC21:50
*** brinzhang0 has joined #openstack-nova21:50
*** brinzhang_ has quit IRC21:51
*** brinzhang has quit IRC21:53
opendevreviewmelanie witt proposed openstack/nova stable/ussuri: zuul: Start to migrate nova-live-migration to zuulv3  https://review.opendev.org/c/openstack/nova/+/79543221:57
opendevreviewmelanie witt proposed openstack/nova stable/ussuri: Remove broken legacy zuul jobs  https://review.opendev.org/c/openstack/nova/+/79537421:57
*** luksky has quit IRC21:58
*** luksky has joined #openstack-nova22:00
toskymelwitt: an supersquash! Would it make sense to change the first line of the commit message? It is more than just that single commit it was taken from22:07
opendevreviewmelanie witt proposed openstack/nova master: api: Log correct client IP if load balancer in use  https://review.opendev.org/c/openstack/nova/+/78676622:21
gmannstephenfin: can you check if anything else you want to add in this spec, leaving +w to you https://review.opendev.org/c/openstack/nova-specs/+/79162022:25
*** tosky has quit IRC22:47
*** whoami-rajat has joined #openstack-nova22:58
*** luksky has quit IRC23:43

Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!