| openstackgerrit | James Kirsch proposed openstack/kolla-ansible master: Use kolla_toolbox to execute REST methods https://review.opendev.org/700788 | 01:14 |
|---|---|---|
| openstackgerrit | James Kirsch proposed openstack/kolla-ansible master: Generate self signed TLS certificates https://review.opendev.org/701323 | 01:14 |
| openstackgerrit | James Kirsch proposed openstack/kolla-ansible master: CI: Add TLS tests https://review.opendev.org/701414 | 01:14 |
| *** stackedsax has quit IRC | 02:21 | |
| *** stackedsax has joined #openstack-kolla | 02:21 | |
| *** r3ap3r has quit IRC | 02:21 | |
| *** r3ap3r has joined #openstack-kolla | 02:21 | |
| *** mgoddard has quit IRC | 03:39 | |
| *** mgoddard has joined #openstack-kolla | 03:45 | |
| *** k_mouza has joined #openstack-kolla | 05:34 | |
| *** evrardjp has quit IRC | 05:34 | |
| *** evrardjp has joined #openstack-kolla | 05:34 | |
| *** k_mouza has quit IRC | 05:38 | |
| *** vmixor has joined #openstack-kolla | 07:12 | |
| *** kozhukalov has joined #openstack-kolla | 07:19 | |
| *** vmixor has quit IRC | 07:48 | |
| *** k_mouza has joined #openstack-kolla | 08:15 | |
| *** k_mouza has quit IRC | 08:20 | |
| openstackgerrit | Merged openstack/kolla-ansible master: Ansible lint: disable some checks https://review.opendev.org/702898 | 08:30 |
| *** cah_link has joined #openstack-kolla | 09:14 | |
| *** cah_link1 has joined #openstack-kolla | 09:17 | |
| *** cah_link has quit IRC | 09:17 | |
| *** cah_link1 is now known as cah_link | 09:17 | |
| *** kozhukalov has quit IRC | 09:31 | |
| *** generalfuzz has quit IRC | 09:48 | |
| *** crindi has quit IRC | 09:48 | |
| *** crindi has joined #openstack-kolla | 09:49 | |
| *** cz3 has quit IRC | 09:56 | |
| *** cz3 has joined #openstack-kolla | 09:57 | |
| *** kozhukalov has joined #openstack-kolla | 09:57 | |
| cosmicsound | Release for CentOS Linux 8 (1911) | 10:23 |
| *** kozhukalov has quit IRC | 10:25 | |
| yoctozepto | cosmicsound: yeah :-) | 10:26 |
| *** xaban has joined #openstack-kolla | 11:09 | |
| osmanlicilegi | xaban: welcome :) | 11:19 |
| xaban | osmanlicilegi Thanks! | 11:35 |
| xaban | We did have problems with our RabbitMQ, with some reason it couldn't form a cluster. Now it is even worse. It starts rabbitmq-bundle-0 on am6-controller-0, rabbitmq-bundle-1 on am6-controller-2 (where it should be am6-controller-1) and rabbitmq-bundle-2 is not being started. Any ideas? | 11:49 |
| openstackgerrit | Radosław Piliszek proposed openstack/kolla-ansible master: CI: Refactor base jobs https://review.opendev.org/703231 | 12:04 |
| *** kozhukalov has joined #openstack-kolla | 12:05 | |
| yoctozepto | xaban: kolla channel not really the best place to discuss tripleo ;D mind you we are not using pacemaker in kolla | 12:05 |
| *** born2bake has joined #openstack-kolla | 12:07 | |
| openstackgerrit | Radosław Piliszek proposed openstack/kolla-ansible master: CI: Refactor base jobs https://review.opendev.org/703231 | 12:09 |
| *** born2bake has quit IRC | 12:19 | |
| *** born2bake has joined #openstack-kolla | 12:33 | |
| *** born2bake has quit IRC | 12:38 | |
| *** kozhukalov has quit IRC | 12:43 | |
| *** kozhukalov has joined #openstack-kolla | 12:44 | |
| *** kozhukalov has quit IRC | 13:05 | |
| *** dciabrin_ has joined #openstack-kolla | 13:13 | |
| *** dciabrin has quit IRC | 13:18 | |
| *** zhanglong has joined #openstack-kolla | 14:30 | |
| *** kozhukalov has joined #openstack-kolla | 15:06 | |
| openstackgerrit | Radosław Piliszek proposed openstack/kolla-ansible master: DNM: test nfv https://review.opendev.org/703246 | 15:12 |
| openstackgerrit | Radosław Piliszek proposed openstack/kolla-ansible master: DNM: test nfv https://review.opendev.org/703246 | 15:12 |
| *** zhanglong has quit IRC | 15:14 | |
| *** kozhukalov has quit IRC | 15:22 | |
| *** dave-mccowan has joined #openstack-kolla | 16:22 | |
| *** dave-mccowan has quit IRC | 16:39 | |
| *** born2bake has joined #openstack-kolla | 16:43 | |
| r3ap3r | yoctozepto: I don't mean to be nosey but how in the world did you know xaban was talking about tripleO?? | 17:33 |
| *** evrardjp has quit IRC | 17:34 | |
| *** evrardjp has joined #openstack-kolla | 17:34 | |
| *** k_mouza has joined #openstack-kolla | 17:48 | |
| cosmicsound | r3ap3r , guess from the file structure -controlle-2 bundle-1 :) | 17:51 |
| yoctozepto | r3ap3r: what cosmicsound said ;-) | 17:53 |
| yoctozepto | also, we (me and osmanlicilegi) had already spoken with xaban on #openstack | 17:53 |
| yoctozepto | that's why we knew it exactly :-) | 17:54 |
| r3ap3r | Oh, lol. | 17:57 |
| r3ap3r | I have ALOT more to learn about Openstack deployments before I can do something like that without having a previous conversation with someone. lol | 17:58 |
| yoctozepto | so make it your new year's goal :-) | 17:59 |
| yoctozepto | year still young :-) | 17:59 |
| r3ap3r | Oh trust me, it is definitly on the list. Working on learning everything about Kolla that I can first and then move on to others but Kolla supports pretty much all of the things I want to tinker with to start out at deployment vs the others. ;-) | 18:01 |
| yoctozepto | r3ap3r: glad to hear that :-) | 18:02 |
| r3ap3r | Only thing I cannot figure out, and I may have missed it in a previous conversation yall have had, is why dropping support for deploying Ceph? I know I can connect to an existing one that I build before deployment but I was just curious why it was dropped? | 18:04 |
| yoctozepto | r3ap3r: lack of human resources to keep it up-to-date - with kolla-ansible we promise a path of upgrade and upgrading storage cluster is no easy deal, sum it up with openstack and you get serious workload (at least from time to time) - we kinda tried to inform ppl about it and get to know their opinion via the ml and originally got nothing (I guess | 18:08 |
| yoctozepto | ppl are shy) but sporadically folks pop up here asking the very same question so I'm really puzzled how to best approach this :-) - anyways, we are (well, mostly mnasiadka is) working on conversion path to use ceph-ansible (for now as it is going to be deprecated too lol, what a mess) | 18:08 |
| r3ap3r | Ah, understood. That makes sense. Ceph-Ansible being deprecated must be frustrating. I hear Ceph-Deploy is supposed to be pretty good, I was planning on using that to try my hand at deploying Ceph for the first time, maybe that could be integrated? | 18:14 |
| r3ap3r | Disclaimer: I am by no stretch of the imagination a Developer so feel free to tell me that wouldn't work at all. ;-) | 18:15 |
| yoctozepto | r3ap3r: well, we would still need to ditch upgrade with either path; ceph-deploy is worse in this context in that it happens only after you have deployed mon (and mgr afair, I think it runs off mgr) and it really helps with adding osds and other daemons (so scaling) | 18:17 |
| r3ap3r | Ok, makes sense from my current understanding of things. Thanks for explaining things the way you do. | 18:19 |
| yoctozepto | yw | 18:21 |
| *** dave-mccowan has joined #openstack-kolla | 18:33 | |
| *** xaban has quit IRC | 18:49 | |
| openstackgerrit | Radosław Piliszek proposed openstack/kolla master: Revert "Debian/source: do not force tag in build jobs" https://review.opendev.org/703252 | 19:09 |
| openstackgerrit | Radosław Piliszek proposed openstack/kolla master: Revert "Debian/source: do not force tag in build jobs" https://review.opendev.org/703252 | 19:15 |
| openstackgerrit | Radosław Piliszek proposed openstack/kolla master: Revert "Debian/source: do not force tag in build jobs" https://review.opendev.org/703252 | 19:15 |
| *** oncall-pokemon has joined #openstack-kolla | 19:38 | |
| oncall-pokemon | Hi, we are setting up kolla and are running into privsep errors where it says things like "privsep helper command exited non-zero (1)". But then deeper in the logs we see things like the log:" sudo: unknown uid 42435: who are you?" we just have the kolla user but it seems to want to run things with rootwrap as somebody else | 19:46 |
| oncall-pokemon | any ideas? | 19:46 |
| yoctozepto | oncall-pokemon: well, all users should be installed in base | 20:01 |
| oncall-pokemon | what does that mean specifically yoctozepto ? Like in the containers or on the base system? I guess you mean the base system? Is there a guide on who to create? | 20:02 |
| yoctozepto | oncall-pokemon: nah, the base image aka the image called 'base' :-) | 20:02 |
| yoctozepto | all users are created upfront | 20:03 |
| yoctozepto | with those high ids | 20:03 |
| yoctozepto | I'm wondering why it knew that high id, yet said "unknown, who are you" | 20:04 |
| yoctozepto | hmm | 20:04 |
| oncall-pokemon | so it should be a container running with the word base in it? Because I did all the steps and I don't think I saw it | 20:04 |
| oncall-pokemon | yeah it knew the id | 20:04 |
| yoctozepto | oncall-pokemon: nah, it's a parent image of all the images | 20:04 |
| yoctozepto | every image kolla builds is a child of base | 20:04 |
| yoctozepto | (direct or indirect) | 20:05 |
| oncall-pokemon | ahh ok. yeah we keep getting these privsep errors and stuff. kinda all new to us. | 20:09 |
| yoctozepto | privsep requires elevation | 20:09 |
| yoctozepto | so it uses sudo | 20:09 |
| yoctozepto | and sudoers are also installed in proper image | 20:10 |
| oncall-pokemon | ahh I see and i guess because the users aren't there it's getting confused | 20:10 |
| yoctozepto | yeah, though I don't know how "they are not there" ;D | 20:10 |
| oncall-pokemon | maybe if I do kolla-ansible pull and then restart them all? | 20:12 |
| yoctozepto | you could but they were like never missing this part, it must know the names because they are setup via names, not ids and it knows the id to fail on | 20:13 |
| yoctozepto | odd | 20:13 |
| oncall-pokemon | ahh ok if you think of anything. where are those uid's defined? I saw them once someplace but can't remember where now | 20:19 |
| *** k_mouza has quit IRC | 20:19 | |
| oncall-pokemon | if I go into the nova-api container for example in the passwd file this exists `neutron❌42435:42435::/home/neutron:/usr/sbin/nologin` | 20:22 |
| oncall-pokemon | so it looks like it is there | 20:22 |
| oncall-pokemon | id 42435 | 20:23 |
| oncall-pokemon | uid=42435(neutron) gid=42435(neutron) groups=42435(neutron) | 20:23 |
| oncall-pokemon | This error was from "id 42435 | 20:23 |
| oncall-pokemon | uid=42435(neutron) gid=42435(neutron) groups=42435(neutron)" | 20:23 |
| oncall-pokemon | sorry | 20:23 |
| oncall-pokemon | This error was from site-packages/nova/compute/manager.py", line 2517, in _build_and_run_instance instance_uuid=instance.uuid, reason=six.text_type(e)) | 20:24 |
| yoctozepto | oncall-pokemon: they are in kolla config.py | 20:28 |
| yoctozepto | they go to /etc/passwd and shadow | 20:28 |
| yoctozepto | yeah, 'tis neutron | 20:28 |
| oncall-pokemon | ahh yes that log is on the compute node where the vm was destined | 20:30 |
| oncall-pokemon | There's also this above the error https://pastebin.com/2Xn1YTq8 | 20:33 |
| oncall-pokemon | not sure if that's the base you're talking about | 20:33 |
| yoctozepto | nah, this is some usage of word "base" in nova | 20:35 |
| yoctozepto | if you exec into the container that is having issues | 20:36 |
| yoctozepto | can you verify both passwd and shadow contain proper users? | 20:36 |
| oncall-pokemon | says I can't read the shadow file as the nova user which is what I become when I do docker exec | 20:38 |
| oncall-pokemon | Permission denied rather than can't read | 20:38 |
| oncall-pokemon | if I do --user root then I can see that yes it's in the shadow file | 20:40 |
| oncall-pokemon | this is the nova-compute container | 20:40 |
| oncall-pokemon | the password field though shows !! which says the account is locked | 20:45 |
| oncall-pokemon | says that for all users that aren't centos standard | 20:45 |
| *** factor has joined #openstack-kolla | 20:50 | |
| yoctozepto | that's no problem | 20:55 |
| oncall-pokemon | the uid thing is just an error. as to why `sudo nova-rootwrap /etc/nova/rootwrap.conf privsep-helper --config-file /usr/share/nova/nova-dist.conf --config-file /etc/nova/nova.conf --privsep_context nova.privsep.sys_admin_pctxt --privsep_sock_path /tmp/tmpeNTxkt/privsep.sock` exists with a 1 is the issue it seems | 20:55 |
| yoctozepto | it really should consult only /etc/passwd | 20:55 |
| oncall-pokemon | i mean to say it says the uid thing is a warning | 20:55 |
| oncall-pokemon | The nova-compute log says "Instance failed to spawn: FailedToDropPrivileges: privsep helper command exited non-zero (1)" | 20:56 |
| oncall-pokemon | But it doesn't say why | 20:56 |
| yoctozepto | well, sudo error is the reason | 20:57 |
| oncall-pokemon | ahh ok | 21:00 |
| oncall-pokemon | if I just run `sudo nova-rootwrap` it asks for password | 21:01 |
| oncall-pokemon | oh wait there's a special nopasswd | 21:01 |
| yoctozepto | indeed, that's how it works | 21:02 |
| *** k_mouza has joined #openstack-kolla | 21:02 | |
| oncall-pokemon | If I run the command at the top I get the following https://pastebin.com/JA5b9585 | 21:03 |
| oncall-pokemon | now it's not the full command but i'm just desperate I guess :) | 21:03 |
| oncall-pokemon | oh no i guess it's waiting for the path later in the command. bummer | 21:04 |
| oncall-pokemon | well you're right I guess | 21:04 |
| yoctozepto | oncall-pokemon: well, sudo worked there | 21:05 |
| yoctozepto | the rest is irrelevant for now | 21:05 |
| oncall-pokemon | yeah it did but it still complains about no socket at /tmp/tmpeNTxkt/privsep.sock | 21:06 |
| yoctozepto | yeah, because it's not there when you are not nova the daemon :-) | 21:06 |
| yoctozepto | the temporary path is indeed temporary | 21:06 |
| yoctozepto | anyways, this proves sudo works | 21:07 |
| yoctozepto | it should also work for the daemon itself | 21:08 |
| * yoctozepto going to sleep, waving good night | 21:08 | |
| oncall-pokemon | Thanks. well let me try restarting. if you think of anything please let me know. good night | 21:08 |
| *** k_mouza has quit IRC | 21:20 | |
| oncall-pokemon | restarting the container worked actually. | 21:31 |
| oncall-pokemon | is libvirtd supposed to be installed on the compute hypervisor or is that containerized too? | 21:31 |
| *** k_mouza has joined #openstack-kolla | 21:35 | |
| openstackgerrit | Marcin Juszkiewicz proposed openstack/kolla-ansible master: CI: Add ansible-lint to tox https://review.opendev.org/694779 | 21:36 |
| r3ap3r | oncall-pokemon: From my understanding, libvirtd is installed on the Compute Node and the Nova Container utilizes API calls to interact with libvirtd on the physical host. | 21:40 |
| *** k_mouza has quit IRC | 21:40 | |
| *** k_mouza has joined #openstack-kolla | 21:42 | |
| hrw | oncall-pokemon: grab clean machines with freshly installed minimal OS. run kolla-ansible bootstrap-servers precheck to have docker installed on each. do deploy. | 21:47 |
| hrw | oncall-pokemon: libvirtd will run in container | 21:47 |
| hrw | iirc precheck checks for libvirtd running on host | 21:47 |
| * hrw out | 21:47 | |
| r3ap3r | hrw: thanks for the clarification. :) | 21:48 |
| *** cah_link1 has joined #openstack-kolla | 23:18 | |
| *** cah_link has quit IRC | 23:21 | |
| *** cah_link1 is now known as cah_link | 23:21 | |
| openstackgerrit | James Kirsch proposed openstack/kolla-ansible master: Copy CA into containers. https://review.opendev.org/699888 | 23:22 |
| openstackgerrit | James Kirsch proposed openstack/kolla-ansible master: Use kolla_toolbox to execute REST methods https://review.opendev.org/700788 | 23:22 |
| openstackgerrit | James Kirsch proposed openstack/kolla-ansible master: Generate self signed TLS certificates https://review.opendev.org/701323 | 23:22 |
| openstackgerrit | James Kirsch proposed openstack/kolla-ansible master: CI: Add TLS tests https://review.opendev.org/701414 | 23:22 |
| *** k_mouza has quit IRC | 23:23 | |
| *** born2bake has quit IRC | 23:33 | |
| oncall-pokemon | Thanks | 23:43 |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!