Saturday, 2020-05-02

*** Goneri has quit IRC01:05
*** spiral has quit IRC01:14
*** jmasud has quit IRC01:15
*** escarluka has joined #openstack01:22
*** escarluka has left #openstack01:37
*** jmasud has joined #openstack01:41
*** gyee has quit IRC01:46
*** arif-ali has quit IRC01:57
*** gyee has joined #openstack01:59
*** gyee has quit IRC02:06
*** jmasud has quit IRC02:16
*** jmasud has joined #openstack02:18
*** drznpy has quit IRC02:24
*** drznpy has joined #openstack02:27
*** haleyb has joined #openstack02:28
*** jmasud has quit IRC02:30
*** weaselkeeper has quit IRC02:54
*** brokencycle has quit IRC03:00
*** nethans has joined #openstack03:08
*** arnoldoree has joined #openstack03:28
*** kenperkins has quit IRC04:32
*** slaweq has joined #openstack04:54
*** slaweq has quit IRC04:59
*** bobey has quit IRC05:27
*** bobey has joined #openstack05:28
*** jmasud has joined #openstack05:49
*** jangutter has joined #openstack05:54
*** jmasud has quit IRC06:16
*** slaweq has joined #openstack06:17
*** jmasud has joined #openstack06:18
*** arnoldoree has quit IRC06:54
*** sauvin has joined #openstack06:55
*** arnoldoree has joined #openstack06:57
*** jmasud has quit IRC07:03
*** jmasud has joined #openstack07:06
*** slaweq has quit IRC07:12
*** pcaruana has quit IRC07:17
*** albert23 has joined #openstack07:27
*** pcaruana has joined #openstack07:29
*** arnoldoree has quit IRC07:33
*** arnoldoree has joined #openstack07:46
*** jmasud has quit IRC07:52
*** slaweq has joined #openstack07:58
*** skyraven has quit IRC07:59
*** hoonetorg has quit IRC08:24
*** eb0t has quit IRC08:31
*** hoonetorg has joined #openstack08:39
*** zadock has joined #openstack09:12
*** zadock has quit IRC09:13
*** eb0t has joined #openstack09:14
*** skyraven has joined #openstack09:15
*** hoonetorg has quit IRC09:22
*** bbowen has joined #openstack09:23
*** bbowen_ has quit IRC09:24
*** skyraven has quit IRC09:25
*** jangutter has quit IRC09:31
*** hoonetorg has joined #openstack09:34
*** jangutter has joined #openstack09:46
*** eb0t has quit IRC10:02
*** jmasud has joined #openstack10:03
*** jmasud has quit IRC10:16
*** jmasud has joined #openstack10:17
*** LarsErikP has joined #openstack10:21
*** jmasud has quit IRC10:21
*** born2bake has joined #openstack10:26
*** laerling has quit IRC10:29
*** mchlumsky has quit IRC11:48
*** mchlumsky has joined #openstack11:48
*** astrospec has quit IRC11:49
*** marbindrakon has quit IRC11:49
*** marbindrakon has joined #openstack11:49
*** partlycloudy has quit IRC11:49
*** BlessJah has quit IRC11:51
*** sajoupa has quit IRC11:51
*** darvon has quit IRC11:51
*** BlessJah has joined #openstack11:52
*** darvon has joined #openstack11:52
*** mattiasj has joined #openstack11:52
*** partlycloudy has joined #openstack11:53
*** bbowen has quit IRC11:55
*** born2bake has quit IRC11:58
*** born2bake has joined #openstack11:59
*** nksegos has joined #openstack12:03
*** peeejayz has quit IRC12:13
*** arnoldoree has quit IRC12:14
*** arnoldoree has joined #openstack12:14
*** kenperkins has joined #openstack12:34
*** drznpy has quit IRC13:37
*** drznpy has joined #openstack13:37
*** slaweq has quit IRC14:21
Eryn_1983_FLok i got some errors14:22
Eryn_1983_FLcan you guys take a peark14:23
*** slaweq has joined #openstack14:30
*** belmoreira has joined #openstack14:31
*** belmoreira has quit IRC14:31
Eryn_1983_FLi keep getting dpdk errors14:35
*** LaurentDumont has joined #openstack14:59
*** DK21 has joined #openstack15:24
*** skyraven has joined #openstack15:37
*** Sajesajama has joined #openstack15:38
*** slaweq has quit IRC15:39
*** Sajesajama_ has joined #openstack15:42
*** Sajesajama has quit IRC15:44
*** Sajesajama__ has joined #openstack15:45
*** Sajesajama_ has quit IRC15:47
kenperkinsok so theory question: running (non-prod) environment, found a bunch of orphaned server groups (perhaps terraform is not cleaning up server group resources), so I ran a script that purged all groups with no members, and of course I fat fingered the condition and deleted the groups with members16:04
kenperkinsgood news is it's my environment and it's just for day-to-day dev work16:04
kenperkinsbut what will happen? :D16:05
LaurentDumontHum, not 100% sure, but if the group doesn't exist anymore, it might just mean that the policy won't apply anymore.16:08
kenperkinswhich would be completely fine, the resources rarely last more than a day anyway16:08
kenperkinsI also noticed none of the server groups, regardless if they were in use or not, had a project id16:09
kenperkinsalthough they do appear to be attributed to project quotes16:09
LaurentDumontAh, found this :
openstackLaunchpad bug 1298509 in OpenStack Compute (nova) ""nova server-group-delete" allows deleting server group with members" [Wishlist,Opinion]16:09
LaurentDumontYeah looks like it just means that the policy won't apply anymore.16:09
LaurentDumont(if you migrate/resize the existing servers)16:10
LaurentDumontHum, not 100% sure what it looks like for ServerGroups with servers inside. Let me check in my dev env.16:10
kenperkinshm.  it does appear terraform has an implementation for deleting server group; can't figure out why they appear to be orphaned16:11
*** LaurentDumont has quit IRC16:14
kenperkinsLaurentDumont: more specifically, when doing `openstack server group list --all-projects --long` the project id column was empty for every group16:14
kenperkinsI need to query in a non-admin context to see16:14
*** LaurentDumont has joined #openstack16:14
LaurentDumontHum, it doesn't see to attach server groups to a project.16:16
kenperkins"it" being openstack?16:17
kenperkinsnova rather16:17
LaurentDumontWell, the show command doesn't output a project. But that might just be the openstack cli.16:18
*** takamatsu has quit IRC16:18
kenperkinswell I know it has some relationship with project as one of my ci jobs (which re-uses a projecT) started failing this morning on a project quota for server groups16:19
LaurentDumontThe CLI is going to get your the server-group in the project you are sourced into. you can pass it --all-projects if you want all server-groups. It's just strange that the ProjectId and UserID are blank.16:22
kenperkinsyea, that's exactly what I'm seeing16:23
kenperkinsI was speculating if i used the project context (sourced) would I see them then16:23
LaurentDumontDoesn't look like it.16:26
LaurentDumontEven sourced into a dummy project, it doesn't show userid or projectid.16:27
*** shibboleth has joined #openstack16:29
*** jmasud has joined #openstack16:29
kenperkinssometimes I shake my fists at openstack16:30
*** brokencycle has joined #openstack16:34
LaurentDumontHahaha - it works with nova server-group-list16:38
kenperkinsopenstack tooling is so bad16:39
setuidIn what way?16:39
*** jangutter has quit IRC16:39
kenperkins@LaurentDumont if I read that correctly, are you saying that the nova tool uses that  api microversion?16:40
kenperkinsand openstack cli does not (at least at the version I'm using)16:40
*** jangutter has joined #openstack16:41
LaurentDumontNot 100% sure, the api call from the "openstack" cli doesn't return any metadata for Project and User for server-groups16:42
LaurentDumontMy client is pretty old though - 3.1816:43
kenperkinslet me see what mine is16:45
kenperkinsI don't even bother running local tools because of how much i dislike python2, python 3, all the venv stuff16:46
kenperkinssetuid, to a degree, that's part of my personal frustration16:46
kenperkinshaving worked on go-based tooling for openstack, the python based stuff is so frustrating; in addition to the contention/incompatibility at times between nova/etc/tool and openstack cli16:46
kenperkinsI'm even older, 3.16.116:47
kenperkinsI guess I need to update my docker image for running cli16:47
*** ldumont has joined #openstack16:57
*** LaurentDumont has quit IRC17:00
*** spiral has joined #openstack17:00
*** jmasud has quit IRC17:05
*** shibboleth has quit IRC17:13
*** Sajesajama__ has quit IRC17:24
*** yiipi has joined #openstack17:35
*** waxfire has joined #openstack17:36
yiipianyone had issue with live migration and nova-compute service down on source host?17:37
*** slaweq has joined #openstack17:40
ldumontI don't think you can do that though?17:41
ldumontlive migration requires both source and target compute to be up.17:41
yiipiwhen I restart nova-compute on source host and try to live-migrate instance on any other host I get ERROR oslo_messaging.rpc.server nova.exception.InvalidCPUInfo: Unacceptable CPU info: CPU doesn't have compatibility.17:41
yiipiboth r up when I start migration17:41
yiipiwhen the migration finish17:42
yiipisource compute service went down17:42
yiipiand no error in logs17:42
yiipiwhen I try to live migrate to another running node with same cpu I get this error ERROR oslo_messaging.rpc.server nova.exception.InvalidCPUInfo: Unacceptable CPU info: CPU doesn't have compatibility.17:42
yiipiand when I restart nova-copute on node I've firstly start live migration I get same error17:43
yiipiWhen I soft restart, resize or migrate instance I can live migrate it again and then I get same issue.17:44
yiipiLike the live migration rewrite something in xml of quemu or something but quemu process has same parameters like in source compute node17:45
yiipifrinding my brain for few days about it without any luck.17:45
yiipi-cpu host is parameter in qemu process17:46
*** slaweq has quit IRC17:48
*** slaweq has joined #openstack17:53
*** jmasud has joined #openstack17:54
ldumontNever saw that before. Are all the computes using the same CPU model?17:58
yiipiall are the same17:59
ldumontWhat release of Openstack?17:59
yiipiit's a rocky on debian buster17:59
yiipitry to solve this and upgrade to stain after that17:59
ldumontWhat does cpu_mode and cpu_model look like in nova.conf?18:00
yiipilet me check18:00
*** bbowen has joined #openstack18:00
yiipicpu-mode= host-passthrough18:00
yiipion all the compute nodes18:02
ldumontMaybe you can try more generic settings on those two nova hosts18:02
yiipiok will try that18:02
ldumontBut I have never seen that before. Surprised that you are seeing this with the same CPU model with a recent release like Rocky.18:02
ldumontHow was it deployed?18:02
yiipiafter the change I can restart only nova-compute18:03
ldumontI think so, yes.18:03
*** ldumont has quit IRC18:03
*** LaurentDumont has joined #openstack18:04
yiipiIt;s deployed manual with mariadb glaera cluster haproxy with keepalived and 3 hosts for controllers18:04
yiipieverything else works ok18:04
LaurentDumontOkay, so you made that nova.conf file yourself?18:04
LaurentDumontOr whatever the nova-compute package installed?18:04
yiipiyes I've edited form distibution (Debian Buster) file18:05
yiipidpkg made it and I've edited it for my setup.18:05
LaurentDumontSo yeah, as far as I know, these are the settings you need to look at.18:07
LaurentDumontMaybe comparing the libvirt VM XML also (for the cpu portion)18:07
*** slaweq has quit IRC18:11
*** timburke has quit IRC18:14
*** hugokuo has quit IRC18:14
*** jmasud has quit IRC18:17
*** jmasud has joined #openstack18:18
*** nksegos has quit IRC18:20
*** nsegkos has joined #openstack18:20
*** cah_link has joined #openstack18:21
*** nsegkos is now known as nksegos18:23
yiipiafter I've changed cpu_mode and cpu_model to custom,kvm64 got same thing18:29
yiipibut now I do not get  ERROR oslo_messaging.rpc.server nova.exception.InvalidCPUInfo: Unacceptable CPU info: CPU doesn't have compatibility.18:30
yiipionly nova-compute hangs18:30
yiipi| 14 | nova-compute   | OpenStack-controler-1 | nova     | enabled | down  | 2020-05-02T18:30:02.000000 |18:31
*** Sajesajama has joined #openstack18:32
yiipiafter restart this service everything work good. Service get back up and running and live migration is doing great again18:32
yiipione issue down one to go ;)18:32
yiipino error in log file. Just state go from up to down18:33
LaurentDumontHum, is it stable after the service restart?18:48
yiipionly live migration hit it18:49
LaurentDumontSo, you changed the nova.conf, restart the service and everything appeared stable. You started one live migration and nova-compute went down on the source compute?18:50
LaurentDumontRestarted the service on the source compute and the live-migration worked?18:50
LaurentDumontThat is strange, never saw that before. Does the actual compute agent die on compute? No logs at all? Maybe debug level for logs can help here.18:52
yiipibefore I've changed model and mode It gave me an error cpu does not have capability and I needed to reboot instance to live migrate again. Also restarting nova-compute on source node was must.18:52
yiipino agent does not die18:52
yiipibut does not work either.18:53
yiipinothing in logs18:53
yiipiprocess seams like working regular18:53
yiipino zombie or hang in other sys way18:53
LaurentDumontAny logs on the controllers regarding that compute?18:54
LaurentDumontSpecifically from anything nova related?18:54
yiipidont think so but let me check18:56
LaurentDumontReally weird though, never saw something like this. Especially if the service is still active on the compute but the controller reports it as down18:58
*** slaweq has joined #openstack19:00
*** aedc has quit IRC19:09
*** aedc has joined #openstack19:15
yiipino nothing in log's about hanging compute service19:35
yiipicould be something with libvirtd?19:35
yiipialso when I give nova list I only get list of instances on one compute node. Other gives me an empty table19:39
yiipinova show gives me all the info about instance on all compute nodes19:41
yiipiand when I fire up nova list -all I get list on all nodes.19:42
yiipiLike the nova-compute on two nodes does not see instance is in project that should be19:43
*** slaweq has quit IRC19:44
yiipimy mistake OS_PROJECT_NAME was on two servers admin and on one was name of the other project, so nova sends back everything like it should19:46
LaurentDumontopenstack server list --all will give you all projects instances if you are admin.19:48
LaurentDumontIf you are sourced into a non-admin project, youll only get your instances :)19:48
yiipiI'm admin :)19:51
yiipior trying to be ;)19:52
yiipishould I elevate log to debug and see will it threw an error about dying service?19:55
*** slaweq has joined #openstack19:55
*** slaweq has quit IRC20:00
*** slaweq has joined #openstack20:05
*** sauvin has quit IRC20:10
*** Goneri has joined #openstack20:13
LaurentDumontThat would be something to try.20:16
LaurentDumontI'd say both on the compute and controller20:16
*** sauvin has joined #openstack20:18
*** hqdruxn08__ has joined #openstack20:19
*** hqdruxn08_ has quit IRC20:23
*** slaweq has quit IRC20:23
yiipiI've tried and nothing new except debug info of processes . I've noticed that privsep-helper after migration has 3 process that never ends on compute node that went down.20:25
yiipiprivsep_context are os_brick.privileged.default nova.privsep.sys_admin_pctxt and vif_plug_linux_bridge.privsep.vif_plug20:29
yiipiis that normal?20:29
*** jmasud has quit IRC20:33
*** slaweq has joined #openstack20:35
*** slaweq has quit IRC20:40
LaurentDumontNo idea sorry :(20:44
*** LaurentDumont has quit IRC20:46
*** jmasud has joined #openstack20:50
*** jangutter_ has joined #openstack20:53
Eryn_1983_FLhey guys20:55
Eryn_1983_FLi keep getting these errors we cant get our networking up20:55
Eryn_1983_FLopenvswitch is down and dpdk20:55
*** jangutter has quit IRC20:56
*** cah_link has quit IRC20:57
*** matt_kosut has quit IRC21:02
*** hamalq has joined #openstack21:12
*** shibboleth has joined #openstack21:23
*** satanist has left #openstack21:37
*** jmasud has quit IRC21:40
*** slaweq has joined #openstack21:41
Eryn_1983_FLok i made another paste21:42
*** slaweq has quit IRC21:47
*** slaweq has joined #openstack21:53
*** slaweq has quit IRC21:58
*** JamesBenson has quit IRC21:58
*** jmasud has joined #openstack22:05
*** yiipi has quit IRC22:06
*** jmasud has quit IRC22:18
*** jmasud has joined #openstack22:19
*** jmasud has quit IRC22:40
*** matt_kosut has joined #openstack23:02
*** matt_kosut has quit IRC23:07
*** nksegos has quit IRC23:09
*** nsegkos has joined #openstack23:09
*** nsegkos is now known as nksegos23:12
*** spiral has quit IRC23:20
*** nksegos has quit IRC23:22
*** nsegkos has joined #openstack23:22
*** nsegkos is now known as nksegos23:25
*** Dantalio- has quit IRC23:28
*** bbowen has quit IRC23:33
*** jmasud has joined #openstack23:47

Generated by 2.17.2 by Marius Gedminas - find it at!