| *** Goneri has quit IRC | 01:05 | |
| *** spiral has quit IRC | 01:14 | |
| *** jmasud has quit IRC | 01:15 | |
| *** escarluka has joined #openstack | 01:22 | |
| *** escarluka has left #openstack | 01:37 | |
| *** jmasud has joined #openstack | 01:41 | |
| *** gyee has quit IRC | 01:46 | |
| *** arif-ali has quit IRC | 01:57 | |
| *** gyee has joined #openstack | 01:59 | |
| *** gyee has quit IRC | 02:06 | |
| *** jmasud has quit IRC | 02:16 | |
| *** jmasud has joined #openstack | 02:18 | |
| *** drznpy has quit IRC | 02:24 | |
| *** drznpy has joined #openstack | 02:27 | |
| *** haleyb has joined #openstack | 02:28 | |
| *** jmasud has quit IRC | 02:30 | |
| *** weaselkeeper has quit IRC | 02:54 | |
| *** brokencycle has quit IRC | 03:00 | |
| *** nethans has joined #openstack | 03:08 | |
| *** arnoldoree has joined #openstack | 03:28 | |
| *** kenperkins has quit IRC | 04:32 | |
| *** slaweq has joined #openstack | 04:54 | |
| *** slaweq has quit IRC | 04:59 | |
| *** bobey has quit IRC | 05:27 | |
| *** bobey has joined #openstack | 05:28 | |
| *** jmasud has joined #openstack | 05:49 | |
| *** jangutter has joined #openstack | 05:54 | |
| *** jmasud has quit IRC | 06:16 | |
| *** slaweq has joined #openstack | 06:17 | |
| *** jmasud has joined #openstack | 06:18 | |
| *** arnoldoree has quit IRC | 06:54 | |
| *** sauvin has joined #openstack | 06:55 | |
| *** arnoldoree has joined #openstack | 06:57 | |
| *** jmasud has quit IRC | 07:03 | |
| *** jmasud has joined #openstack | 07:06 | |
| *** slaweq has quit IRC | 07:12 | |
| *** pcaruana has quit IRC | 07:17 | |
| *** albert23 has joined #openstack | 07:27 | |
| *** pcaruana has joined #openstack | 07:29 | |
| *** arnoldoree has quit IRC | 07:33 | |
| *** arnoldoree has joined #openstack | 07:46 | |
| *** jmasud has quit IRC | 07:52 | |
| *** slaweq has joined #openstack | 07:58 | |
| *** skyraven has quit IRC | 07:59 | |
| *** hoonetorg has quit IRC | 08:24 | |
| *** eb0t has quit IRC | 08:31 | |
| *** hoonetorg has joined #openstack | 08:39 | |
| *** zadock has joined #openstack | 09:12 | |
| *** zadock has quit IRC | 09:13 | |
| *** eb0t has joined #openstack | 09:14 | |
| *** skyraven has joined #openstack | 09:15 | |
| *** hoonetorg has quit IRC | 09:22 | |
| *** bbowen has joined #openstack | 09:23 | |
| *** bbowen_ has quit IRC | 09:24 | |
| *** skyraven has quit IRC | 09:25 | |
| *** jangutter has quit IRC | 09:31 | |
| *** hoonetorg has joined #openstack | 09:34 | |
| *** jangutter has joined #openstack | 09:46 | |
| *** eb0t has quit IRC | 10:02 | |
| *** jmasud has joined #openstack | 10:03 | |
| *** jmasud has quit IRC | 10:16 | |
| *** jmasud has joined #openstack | 10:17 | |
| *** LarsErikP has joined #openstack | 10:21 | |
| *** jmasud has quit IRC | 10:21 | |
| *** born2bake has joined #openstack | 10:26 | |
| *** laerling has quit IRC | 10:29 | |
| *** mchlumsky has quit IRC | 11:48 | |
| *** mchlumsky has joined #openstack | 11:48 | |
| *** astrospec has quit IRC | 11:49 | |
| *** marbindrakon has quit IRC | 11:49 | |
| *** marbindrakon has joined #openstack | 11:49 | |
| *** partlycloudy has quit IRC | 11:49 | |
| *** BlessJah has quit IRC | 11:51 | |
| *** sajoupa has quit IRC | 11:51 | |
| *** darvon has quit IRC | 11:51 | |
| *** BlessJah has joined #openstack | 11:52 | |
| *** darvon has joined #openstack | 11:52 | |
| *** mattiasj has joined #openstack | 11:52 | |
| *** partlycloudy has joined #openstack | 11:53 | |
| *** bbowen has quit IRC | 11:55 | |
| *** born2bake has quit IRC | 11:58 | |
| *** born2bake has joined #openstack | 11:59 | |
| *** nksegos has joined #openstack | 12:03 | |
| *** peeejayz has quit IRC | 12:13 | |
| *** arnoldoree has quit IRC | 12:14 | |
| *** arnoldoree has joined #openstack | 12:14 | |
| *** kenperkins has joined #openstack | 12:34 | |
| *** drznpy has quit IRC | 13:37 | |
| *** drznpy has joined #openstack | 13:37 | |
| *** slaweq has quit IRC | 14:21 | |
| Eryn_1983_FL | ok i got some errors | 14:22 |
|---|---|---|
| Eryn_1983_FL | can you guys take a peark | 14:23 |
| Eryn_1983_FL | peak | 14:23 |
| Eryn_1983_FL | https://pastebin.com/UFizJiii | 14:24 |
| *** slaweq has joined #openstack | 14:30 | |
| *** belmoreira has joined #openstack | 14:31 | |
| *** belmoreira has quit IRC | 14:31 | |
| Eryn_1983_FL | i keep getting dpdk errors | 14:35 |
| *** LaurentDumont has joined #openstack | 14:59 | |
| *** DK21 has joined #openstack | 15:24 | |
| *** skyraven has joined #openstack | 15:37 | |
| *** Sajesajama has joined #openstack | 15:38 | |
| *** slaweq has quit IRC | 15:39 | |
| *** Sajesajama_ has joined #openstack | 15:42 | |
| *** Sajesajama has quit IRC | 15:44 | |
| *** Sajesajama__ has joined #openstack | 15:45 | |
| *** Sajesajama_ has quit IRC | 15:47 | |
| kenperkins | ok so theory question: running (non-prod) environment, found a bunch of orphaned server groups (perhaps terraform is not cleaning up server group resources), so I ran a script that purged all groups with no members, and of course I fat fingered the condition and deleted the groups with members | 16:04 |
| kenperkins | good news is it's my environment and it's just for day-to-day dev work | 16:04 |
| kenperkins | but what will happen? :D | 16:05 |
| LaurentDumont | Hum, not 100% sure, but if the group doesn't exist anymore, it might just mean that the policy won't apply anymore. | 16:08 |
| kenperkins | which would be completely fine, the resources rarely last more than a day anyway | 16:08 |
| kenperkins | I also noticed none of the server groups, regardless if they were in use or not, had a project id | 16:09 |
| kenperkins | although they do appear to be attributed to project quotes | 16:09 |
| kenperkins | quotas* | 16:09 |
| LaurentDumont | Ah, found this : https://bugs.launchpad.net/nova/+bug/1298509 | 16:09 |
| openstack | Launchpad bug 1298509 in OpenStack Compute (nova) ""nova server-group-delete" allows deleting server group with members" [Wishlist,Opinion] | 16:09 |
| LaurentDumont | Yeah looks like it just means that the policy won't apply anymore. | 16:09 |
| LaurentDumont | (if you migrate/resize the existing servers) | 16:10 |
| LaurentDumont | Hum, not 100% sure what it looks like for ServerGroups with servers inside. Let me check in my dev env. | 16:10 |
| kenperkins | hm. it does appear terraform has an implementation for deleting server group; can't figure out why they appear to be orphaned | 16:11 |
| kenperkins | https://github.com/terraform-providers/terraform-provider-openstack/blob/e9e493cc5bf6a6f421c44392fc344285381ab1bf/openstack/resource_openstack_compute_servergroup_v2.go#L111-L123 | 16:11 |
| *** LaurentDumont has quit IRC | 16:14 | |
| kenperkins | LaurentDumont: more specifically, when doing `openstack server group list --all-projects --long` the project id column was empty for every group | 16:14 |
| kenperkins | I need to query in a non-admin context to see | 16:14 |
| *** LaurentDumont has joined #openstack | 16:14 | |
| LaurentDumont | Hum, it doesn't see to attach server groups to a project. | 16:16 |
| kenperkins | "it" being openstack? | 16:17 |
| kenperkins | nova rather | 16:17 |
| LaurentDumont | Well, the show command doesn't output a project. But that might just be the openstack cli. | 16:18 |
| *** takamatsu has quit IRC | 16:18 | |
| kenperkins | well I know it has some relationship with project as one of my ci jobs (which re-uses a projecT) started failing this morning on a project quota for server groups | 16:19 |
| LaurentDumont | The CLI is going to get your the server-group in the project you are sourced into. you can pass it --all-projects if you want all server-groups. It's just strange that the ProjectId and UserID are blank. | 16:22 |
| LaurentDumont | http://paste.openstack.org/show/793001/ | 16:22 |
| kenperkins | yea, that's exactly what I'm seeing | 16:23 |
| kenperkins | I was speculating if i used the project context (sourced) would I see them then | 16:23 |
| LaurentDumont | Doesn't look like it. | 16:26 |
| LaurentDumont | Even sourced into a dummy project, it doesn't show userid or projectid. | 16:27 |
| *** shibboleth has joined #openstack | 16:29 | |
| *** jmasud has joined #openstack | 16:29 | |
| kenperkins | sometimes I shake my fists at openstack | 16:30 |
| *** brokencycle has joined #openstack | 16:34 | |
| LaurentDumont | https://specs.openstack.org/openstack/nova-specs/specs/mitaka/implemented/add-project-id-in-os-server-groups-api.html | 16:37 |
| LaurentDumont | Hahaha - it works with nova server-group-list | 16:38 |
| kenperkins | openstack tooling is so bad | 16:39 |
| setuid | In what way? | 16:39 |
| *** jangutter has quit IRC | 16:39 | |
| kenperkins | @LaurentDumont if I read that correctly, are you saying that the nova tool uses that api microversion? | 16:40 |
| kenperkins | and openstack cli does not (at least at the version I'm using) | 16:40 |
| *** jangutter has joined #openstack | 16:41 | |
| LaurentDumont | Not 100% sure, the api call from the "openstack" cli doesn't return any metadata for Project and User for server-groups | 16:42 |
| LaurentDumont | My client is pretty old though - 3.18 | 16:43 |
| kenperkins | let me see what mine is | 16:45 |
| kenperkins | I don't even bother running local tools because of how much i dislike python2, python 3, all the venv stuff | 16:46 |
| kenperkins | setuid, to a degree, that's part of my personal frustration | 16:46 |
| kenperkins | having worked on go-based tooling for openstack, the python based stuff is so frustrating; in addition to the contention/incompatibility at times between nova/etc/tool and openstack cli | 16:46 |
| kenperkins | I'm even older, 3.16.1 | 16:47 |
| kenperkins | I guess I need to update my docker image for running cli | 16:47 |
| *** ldumont has joined #openstack | 16:57 | |
| *** LaurentDumont has quit IRC | 17:00 | |
| *** spiral has joined #openstack | 17:00 | |
| *** jmasud has quit IRC | 17:05 | |
| *** shibboleth has quit IRC | 17:13 | |
| *** Sajesajama__ has quit IRC | 17:24 | |
| *** yiipi has joined #openstack | 17:35 | |
| *** waxfire has joined #openstack | 17:36 | |
| yiipi | anyone had issue with live migration and nova-compute service down on source host? | 17:37 |
| *** slaweq has joined #openstack | 17:40 | |
| ldumont | I don't think you can do that though? | 17:41 |
| ldumont | live migration requires both source and target compute to be up. | 17:41 |
| yiipi | when I restart nova-compute on source host and try to live-migrate instance on any other host I get ERROR oslo_messaging.rpc.server nova.exception.InvalidCPUInfo: Unacceptable CPU info: CPU doesn't have compatibility. | 17:41 |
| yiipi | yes | 17:41 |
| yiipi | both r up when I start migration | 17:41 |
| yiipi | when the migration finish | 17:42 |
| yiipi | source compute service went down | 17:42 |
| yiipi | and no error in logs | 17:42 |
| yiipi | when I try to live migrate to another running node with same cpu I get this error ERROR oslo_messaging.rpc.server nova.exception.InvalidCPUInfo: Unacceptable CPU info: CPU doesn't have compatibility. | 17:42 |
| yiipi | and when I restart nova-copute on node I've firstly start live migration I get same error | 17:43 |
| yiipi | When I soft restart, resize or migrate instance I can live migrate it again and then I get same issue. | 17:44 |
| yiipi | Like the live migration rewrite something in xml of quemu or something but quemu process has same parameters like in source compute node | 17:45 |
| yiipi | frinding my brain for few days about it without any luck. | 17:45 |
| yiipi | grinding | 17:45 |
| yiipi | -cpu host is parameter in qemu process | 17:46 |
| *** slaweq has quit IRC | 17:48 | |
| *** slaweq has joined #openstack | 17:53 | |
| *** jmasud has joined #openstack | 17:54 | |
| ldumont | Never saw that before. Are all the computes using the same CPU model? | 17:58 |
| yiipi | yup | 17:59 |
| yiipi | all are the same | 17:59 |
| ldumont | What release of Openstack? | 17:59 |
| yiipi | it's a rocky on debian buster | 17:59 |
| yiipi | try to solve this and upgrade to stain after that | 17:59 |
| ldumont | What does cpu_mode and cpu_model look like in nova.conf? | 18:00 |
| yiipi | let me check | 18:00 |
| *** bbowen has joined #openstack | 18:00 | |
| yiipi | cpu-mode= host-passthrough | 18:00 |
| yiipi | cpu_model=<None> | 18:01 |
| yiipi | on all the compute nodes | 18:02 |
| ldumont | Maybe you can try more generic settings on those two nova hosts | 18:02 |
| ldumont | [libvirt] | 18:02 |
| ldumont | cpu_mode=custom | 18:02 |
| ldumont | cpu_model=kvm64 | 18:02 |
| yiipi | ok will try that | 18:02 |
| ldumont | But I have never seen that before. Surprised that you are seeing this with the same CPU model with a recent release like Rocky. | 18:02 |
| ldumont | How was it deployed? | 18:02 |
| yiipi | after the change I can restart only nova-compute | 18:03 |
| ldumont | I think so, yes. | 18:03 |
| *** ldumont has quit IRC | 18:03 | |
| *** LaurentDumont has joined #openstack | 18:04 | |
| yiipi | It;s deployed manual with mariadb glaera cluster haproxy with keepalived and 3 hosts for controllers | 18:04 |
| yiipi | everything else works ok | 18:04 |
| LaurentDumont | Okay, so you made that nova.conf file yourself? | 18:04 |
| LaurentDumont | Or whatever the nova-compute package installed? | 18:04 |
| yiipi | yes I've edited form distibution (Debian Buster) file | 18:05 |
| yiipi | dpkg made it and I've edited it for my setup. | 18:05 |
| LaurentDumont | So yeah, as far as I know, these are the settings you need to look at. | 18:07 |
| LaurentDumont | Maybe comparing the libvirt VM XML also (for the cpu portion) | 18:07 |
| *** slaweq has quit IRC | 18:11 | |
| *** timburke has quit IRC | 18:14 | |
| *** hugokuo has quit IRC | 18:14 | |
| *** jmasud has quit IRC | 18:17 | |
| *** jmasud has joined #openstack | 18:18 | |
| *** nksegos has quit IRC | 18:20 | |
| *** nsegkos has joined #openstack | 18:20 | |
| *** cah_link has joined #openstack | 18:21 | |
| *** nsegkos is now known as nksegos | 18:23 | |
| yiipi | after I've changed cpu_mode and cpu_model to custom,kvm64 got same thing | 18:29 |
| yiipi | but now I do not get ERROR oslo_messaging.rpc.server nova.exception.InvalidCPUInfo: Unacceptable CPU info: CPU doesn't have compatibility. | 18:30 |
| yiipi | only nova-compute hangs | 18:30 |
| yiipi | | 14 | nova-compute | OpenStack-controler-1 | nova | enabled | down | 2020-05-02T18:30:02.000000 | | 18:31 |
| *** Sajesajama has joined #openstack | 18:32 | |
| yiipi | after restart this service everything work good. Service get back up and running and live migration is doing great again | 18:32 |
| yiipi | one issue down one to go ;) | 18:32 |
| yiipi | no error in log file. Just state go from up to down | 18:33 |
| LaurentDumont | Hum, is it stable after the service restart? | 18:48 |
| yiipi | yup | 18:49 |
| yiipi | only live migration hit it | 18:49 |
| LaurentDumont | So, you changed the nova.conf, restart the service and everything appeared stable. You started one live migration and nova-compute went down on the source compute? | 18:50 |
| LaurentDumont | Restarted the service on the source compute and the live-migration worked? | 18:50 |
| yiipi | yes | 18:50 |
| LaurentDumont | That is strange, never saw that before. Does the actual compute agent die on compute? No logs at all? Maybe debug level for logs can help here. | 18:52 |
| yiipi | before I've changed model and mode It gave me an error cpu does not have capability and I needed to reboot instance to live migrate again. Also restarting nova-compute on source node was must. | 18:52 |
| yiipi | no agent does not die | 18:52 |
| yiipi | but does not work either. | 18:53 |
| yiipi | nothing in logs | 18:53 |
| yiipi | process seams like working regular | 18:53 |
| yiipi | no zombie or hang in other sys way | 18:53 |
| LaurentDumont | Any logs on the controllers regarding that compute? | 18:54 |
| LaurentDumont | Specifically from anything nova related? | 18:54 |
| yiipi | dont think so but let me check | 18:56 |
| LaurentDumont | Really weird though, never saw something like this. Especially if the service is still active on the compute but the controller reports it as down | 18:58 |
| *** slaweq has joined #openstack | 19:00 | |
| *** aedc has quit IRC | 19:09 | |
| *** aedc has joined #openstack | 19:15 | |
| yiipi | no nothing in log's about hanging compute service | 19:35 |
| yiipi | could be something with libvirtd? | 19:35 |
| yiipi | also when I give nova list I only get list of instances on one compute node. Other gives me an empty table | 19:39 |
| yiipi | nova show gives me all the info about instance on all compute nodes | 19:41 |
| yiipi | and when I fire up nova list -all I get list on all nodes. | 19:42 |
| yiipi | Like the nova-compute on two nodes does not see instance is in project that should be | 19:43 |
| yiipi | och | 19:44 |
| *** slaweq has quit IRC | 19:44 | |
| yiipi | my mistake OS_PROJECT_NAME was on two servers admin and on one was name of the other project, so nova sends back everything like it should | 19:46 |
| LaurentDumont | openstack server list --all will give you all projects instances if you are admin. | 19:48 |
| LaurentDumont | If you are sourced into a non-admin project, youll only get your instances :) | 19:48 |
| yiipi | I'm admin :) | 19:51 |
| yiipi | or trying to be ;) | 19:52 |
| yiipi | should I elevate log to debug and see will it threw an error about dying service? | 19:55 |
| *** slaweq has joined #openstack | 19:55 | |
| *** slaweq has quit IRC | 20:00 | |
| *** slaweq has joined #openstack | 20:05 | |
| *** sauvin has quit IRC | 20:10 | |
| *** Goneri has joined #openstack | 20:13 | |
| LaurentDumont | That would be something to try. | 20:16 |
| LaurentDumont | I'd say both on the compute and controller | 20:16 |
| *** sauvin has joined #openstack | 20:18 | |
| *** hqdruxn08__ has joined #openstack | 20:19 | |
| *** hqdruxn08_ has quit IRC | 20:23 | |
| *** slaweq has quit IRC | 20:23 | |
| yiipi | I've tried and nothing new except debug info of processes . I've noticed that privsep-helper after migration has 3 process that never ends on compute node that went down. | 20:25 |
| yiipi | privsep_context are os_brick.privileged.default nova.privsep.sys_admin_pctxt and vif_plug_linux_bridge.privsep.vif_plug | 20:29 |
| yiipi | is that normal? | 20:29 |
| *** jmasud has quit IRC | 20:33 | |
| *** slaweq has joined #openstack | 20:35 | |
| *** slaweq has quit IRC | 20:40 | |
| LaurentDumont | No idea sorry :( | 20:44 |
| *** LaurentDumont has quit IRC | 20:46 | |
| *** jmasud has joined #openstack | 20:50 | |
| *** jangutter_ has joined #openstack | 20:53 | |
| Eryn_1983_FL | hey guys | 20:55 |
| Eryn_1983_FL | i keep getting these errors we cant get our networking up | 20:55 |
| Eryn_1983_FL | https://pastebin.com/3SLJxQrX | 20:55 |
| Eryn_1983_FL | openvswitch is down and dpdk | 20:55 |
| *** jangutter has quit IRC | 20:56 | |
| *** cah_link has quit IRC | 20:57 | |
| *** matt_kosut has quit IRC | 21:02 | |
| *** hamalq has joined #openstack | 21:12 | |
| *** shibboleth has joined #openstack | 21:23 | |
| *** satanist has left #openstack | 21:37 | |
| *** jmasud has quit IRC | 21:40 | |
| *** slaweq has joined #openstack | 21:41 | |
| Eryn_1983_FL | ok i made another paste | 21:42 |
| Eryn_1983_FL | https://pastebin.com/LcAMu4Uy | 21:42 |
| *** slaweq has quit IRC | 21:47 | |
| *** slaweq has joined #openstack | 21:53 | |
| *** slaweq has quit IRC | 21:58 | |
| *** JamesBenson has quit IRC | 21:58 | |
| *** jmasud has joined #openstack | 22:05 | |
| *** yiipi has quit IRC | 22:06 | |
| *** jmasud has quit IRC | 22:18 | |
| *** jmasud has joined #openstack | 22:19 | |
| *** jmasud has quit IRC | 22:40 | |
| *** matt_kosut has joined #openstack | 23:02 | |
| *** matt_kosut has quit IRC | 23:07 | |
| *** nksegos has quit IRC | 23:09 | |
| *** nsegkos has joined #openstack | 23:09 | |
| *** nsegkos is now known as nksegos | 23:12 | |
| *** spiral has quit IRC | 23:20 | |
| *** nksegos has quit IRC | 23:22 | |
| *** nsegkos has joined #openstack | 23:22 | |
| *** nsegkos is now known as nksegos | 23:25 | |
| *** Dantalio- has quit IRC | 23:28 | |
| *** bbowen has quit IRC | 23:33 | |
| *** jmasud has joined #openstack | 23:47 | |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!