| *** macz_ has joined #openstack-ansible | 00:00 | |
| *** macz_ has quit IRC | 00:04 | |
| *** maharg101 has joined #openstack-ansible | 00:35 | |
| *** maharg101 has quit IRC | 00:39 | |
| *** poopcat has quit IRC | 01:09 | |
| *** poopcat has joined #openstack-ansible | 01:11 | |
| *** spatel has joined #openstack-ansible | 01:28 | |
| *** spatel has quit IRC | 01:28 | |
| *** macz_ has joined #openstack-ansible | 02:01 | |
| *** macz_ has quit IRC | 02:06 | |
| *** maharg101 has joined #openstack-ansible | 02:36 | |
| *** maharg101 has quit IRC | 02:41 | |
| *** partlycloudy has quit IRC | 02:46 | |
| *** partlycloudy has joined #openstack-ansible | 03:22 | |
| *** partlycloudy has quit IRC | 03:32 | |
| *** jfan has joined #openstack-ansible | 04:14 | |
| *** maharg101 has joined #openstack-ansible | 04:36 | |
| *** maharg101 has quit IRC | 04:41 | |
| *** evrardjp has quit IRC | 05:33 | |
| *** evrardjp has joined #openstack-ansible | 05:33 | |
| *** yasemind has joined #openstack-ansible | 05:50 | |
| *** yasemind has quit IRC | 06:36 | |
| *** maharg101 has joined #openstack-ansible | 06:38 | |
| *** maharg101 has quit IRC | 06:42 | |
| *** alanmeadows has quit IRC | 06:44 | |
| *** simondodsley has quit IRC | 06:44 | |
| *** alanmeadows has joined #openstack-ansible | 06:44 | |
| *** jungleboyj has quit IRC | 06:45 | |
| *** jungleboyj has joined #openstack-ansible | 06:46 | |
| *** simondodsley has joined #openstack-ansible | 06:47 | |
| *** yasemind has joined #openstack-ansible | 06:52 | |
| *** SiavashSardari has joined #openstack-ansible | 07:00 | |
| *** yasemind has quit IRC | 07:24 | |
| *** miloa has joined #openstack-ansible | 07:35 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/openstack-ansible master: Imported Translations from Zanata https://review.opendev.org/c/openstack/openstack-ansible/+/773229 | 07:40 |
|---|---|---|
| *** cshen has joined #openstack-ansible | 07:45 | |
| *** pcaruana has quit IRC | 07:56 | |
| *** pcaruana has joined #openstack-ansible | 08:08 | |
| *** rpittau|afk is now known as rpittau | 08:11 | |
| *** andrewbonney has joined #openstack-ansible | 08:16 | |
| openstackgerrit | Andrew Bonney proposed openstack/openstack-ansible-os_zun master: defaults: set up docker overrides using systemd role https://review.opendev.org/c/openstack/openstack-ansible-os_zun/+/771217 | 08:27 |
| jrosser | morning | 08:36 |
| noonedeadpunk | o/ | 08:36 |
| CeeMac | \o | 08:36 |
| *** maharg101 has joined #openstack-ansible | 08:39 | |
| openstackgerrit | Andrew Bonney proposed openstack/openstack-ansible-lxc_container_create master: Fix formatting of LXC container config files https://review.opendev.org/c/openstack/openstack-ansible-lxc_container_create/+/772706 | 08:42 |
| *** tosky has joined #openstack-ansible | 08:43 | |
| openstackgerrit | Jonathan Rosser proposed openstack/openstack-ansible master: Set centos8 jobs as non voting https://review.opendev.org/c/openstack/openstack-ansible/+/773309 | 09:01 |
| openstackgerrit | Jonathan Rosser proposed openstack/openstack-ansible master: Revert "Set centos8 jobs as non voting" https://review.opendev.org/c/openstack/openstack-ansible/+/773310 | 09:01 |
| admin0 | \o | 09:01 |
| noonedeadpunk | andrewbonney: oh, you was so right, that we can't use regexp here...... | 09:06 |
| noonedeadpunk | (I mean 772706) | 09:07 |
| noonedeadpunk | as lxc config really have same keys so regexp will ruin everything... | 09:08 |
| noonedeadpunk | I think we should just return to the previous version of the patch and merge it... | 09:09 |
| * noonedeadpunk feels like should have a vacation as start to fail with everything... | 09:19 | |
| *** MickyMan77 has quit IRC | 09:38 | |
| andrewbonney | noonedeadpunk: ah no worries. I hadn't clocked that the keys could be identical | 09:44 |
| openstackgerrit | Andrew Bonney proposed openstack/openstack-ansible-lxc_container_create master: Fix formatting of LXC container config files https://review.opendev.org/c/openstack/openstack-ansible-lxc_container_create/+/772706 | 09:47 |
| *** yasemind has joined #openstack-ansible | 10:04 | |
| *** SiavashSardari has quit IRC | 10:10 | |
| *** yasemind has quit IRC | 10:10 | |
| *** yasemind has joined #openstack-ansible | 10:18 | |
| openstackgerrit | Andrew Bonney proposed openstack/openstack-ansible-galera_server stable/victoria: Bring db setup vars in line with other roles https://review.opendev.org/c/openstack/openstack-ansible-galera_server/+/772550 | 10:21 |
| noonedeadpunk | jrosser: seems related https://github.com/celery/py-amqp/blob/master/Changelog#L42 | 10:32 |
| noonedeadpunk | and U-C just changed as well https://opendev.org/openstack/requirements/commit/0284f5ec9837c92cc92699076ccba3231ab8b1f5 | 10:33 |
| noonedeadpunk | so amqp 5.0.3 breaks source install the same way as does rdo | 10:34 |
| *** sshnaidm|off is now known as sshnaidm|ruck | 10:35 | |
| noonedeadpunk | I guess that exmplains why our master bump is completely broken now | 10:37 |
| jrosser | oh i see | 10:38 |
| jrosser | so unless there is a way to pass 'no verify' through oslo to that then we are in trouble | 10:38 |
| noonedeadpunk | yep... or unless we generate ca and define ssl_ca_file it as rdo does in their tests I guess https://logserver.rdoproject.org/ci.centos.org/weirdo-generic-packstack-scenario001/12670/weirdo-project/logs/etc/nova/nova.conf.txt.gz | 10:39 |
| *** jbadiapa has joined #openstack-ansible | 10:40 | |
| openstackgerrit | Jonathan Rosser proposed openstack/openstack-ansible master: Increase git clone depth from 10 to 20 https://review.opendev.org/c/openstack/openstack-ansible/+/773352 | 10:43 |
| jrosser | andrewbonney: ^ | 10:43 |
| andrewbonney | Ta | 10:43 |
| noonedeadpunk | jrosser: I just faced that | 10:44 |
| noonedeadpunk | and was pretty frustrated with what is happening... | 10:44 |
| jrosser | would be good to know if that fixes it, i've not tried it | 10:44 |
| jrosser | just discussed the same here with andrewbonney | 10:44 |
| jrosser | this time though there is certainly >10 commits on top of our SHA for ceph-ansible | 10:45 |
| noonedeadpunk | I'm not sure I understand right now why this is happening... I mean shouldn't I be able to checkout to any commit at any given time? | 10:46 |
| noonedeadpunk | As I was failing to checkout even with bare cli command | 10:46 |
| *** gokhani has joined #openstack-ansible | 10:47 | |
| * noonedeadpunk goes to read through man git-clone | 10:47 | |
| jrosser | --depth <depth> Create a shallow clone with a history truncated to the specified number of commits. Implies --single-branch unless --no-single-branch is given to fetch the histories near the tips of all branches. | 10:48 |
| noonedeadpunk | and we use it to save some diskspace and speedup. ok... | 10:49 |
| jrosser | i think also it puts less load on the git server | 10:50 |
| jrosser | which when we were not using the zuul cached repos was probably important | 10:50 |
| gokhani | Hi folks, I am tring to add 2 gpu compute nodes (centos8.3) (Because nvidia vgpu driver doesn't support ubuntu) on my OSA Ussuri deployment, but I am getting errors at gpg tooling.It tries to install python3-apt but my distrubition is centos8. How can ı fix this ? Logs: http://paste.openstack.org/show/802166/ | 10:54 |
| noonedeadpunk | ugh https://github.com/celery/py-amqp/commit/343a00e828d9d2d33998ccaf96dca0b9417f04af | 10:54 |
| jrosser | gokhani: we have nvidia vgpu running here on ubuntu | 10:56 |
| jrosser | doesnt support != doesnt work | 10:57 |
| jrosser | depends if you want support from nvidia, and in fact the latest 11.x GRID driver installs without any trouble on ubuntu which the earlier ones didnt, so they have done work on compatibility wider than RHEL recently | 10:58 |
| gokhani | jrosser ohh really, good news. in fact ı tried it on ubuntu 18.04 but ı can't run "ls /sys/class/mdev_bus/*/mdev_supported_types" and ı don't know how to discover gpu types. so ı gave it up. so it is possible install NVIDIA-Linux-x86_64-460.32.04-vgpu-kvm.run on ubuntu 18.04. Can you share with me deployment steps ? | 11:07 |
| *** SiavashSardari has joined #openstack-ansible | 11:14 | |
| openstackgerrit | Dmitriy Rabotyagov proposed openstack/openstack-ansible-openstack_hosts stable/victoria: Set RDO repo to Victoria https://review.opendev.org/c/openstack/openstack-ansible-openstack_hosts/+/773369 | 11:19 |
| noonedeadpunk | jrosser: so, providing ssl_ca_file to root ca works like a charm and fixes isuse | 11:21 |
| noonedeadpunk | at least we have some way of proceeding with this considering that we wanted to do that way anyway it might be acceptable. if only things were not completely broken atm | 11:25 |
| noonedeadpunk | jrosser: maybe instead of setting centos jobs to nv we might temporary disable ssl for rabbit? | 12:06 |
| noonedeadpunk | until we get root ca done and placed on all hosts | 12:07 |
| jrosser | noonedeadpunk: so you made a CA and a cert and it was happy? | 12:10 |
| noonedeadpunk | yep | 12:10 |
| noonedeadpunk | and amqp to version 5.0.5 | 12:10 |
| noonedeadpunk | well, placed ca to /etc/pki/ca-trust/source/anchors/osa_root.pem and ran /usr/bin/update-ca-trust force-enable && /usr/bin/update-ca-trust extract | 12:11 |
| noonedeadpunk | but yeah | 12:11 |
| noonedeadpunk | I think it's mainly this commit that changed all logic https://github.com/celery/py-amqp/commit/343a00e828d9d2d33998ccaf96dca0b9417f04af#diff-8bf65e6879a19d14c2d02835dc148a1b045e2d1c56fa13c6fd89de2426e54d03 | 12:13 |
| noonedeadpunk | but even if we do change to oslo.messaging, that would mean we need to patch all services configs to disable verification I guess | 12:14 |
| jrosser | it's easier just to make a CA i guess | 12:15 |
| noonedeadpunk | yeah | 12:15 |
| noonedeadpunk | we anyway were going to do this | 12:15 |
| jrosser | i had a ugly patch which was the start of this a long time ago https://review.opendev.org/c/openstack/openstack-ansible/+/644555 | 12:16 |
| noonedeadpunk | I think we need a repo for that? | 12:17 |
| noonedeadpunk | with a role, that can be included wherever it's needed | 12:18 |
| noonedeadpunk | ie during setup_hosts to distribute root ca and etc | 12:18 |
| jrosser | yes we do | 12:19 |
| *** ilush has joined #openstack-ansible | 12:22 | |
| openstackgerrit | Dmitriy Rabotyagov proposed openstack/openstack-ansible master: Temporary disable SSL for rabbit connections https://review.opendev.org/c/openstack/openstack-ansible/+/773376 | 12:25 |
| openstackgerrit | Jonathan Rosser proposed openstack/openstack-ansible master: Disable ssl for rabbitmq https://review.opendev.org/c/openstack/openstack-ansible/+/773377 | 12:27 |
| openstackgerrit | Jonathan Rosser proposed openstack/openstack-ansible master: Revert "Disable ssl for rabbitmq" https://review.opendev.org/c/openstack/openstack-ansible/+/773378 | 12:27 |
| jrosser | lol | 12:27 |
| *** gaudenz has quit IRC | 12:27 | |
| noonedeadpunk | whatever, let me abandon mine | 12:27 |
| noonedeadpunk | any good ideas how to call the repo? | 12:28 |
| jrosser | openstack-ansible-pki ? | 12:29 |
| jrosser | it's kind of multifunction too becasue it needs to create/maintain the root CA | 12:29 |
| jrosser | and also be able to issue server certs on demand | 12:29 |
| noonedeadpunk | yep | 12:29 |
| jrosser | feels like another case to use tasks_from: | 12:30 |
| jrosser | or two roles..... | 12:30 |
| noonedeadpunk | or jsut condition like we do with galera?:) | 12:30 |
| *** pcaruana has quit IRC | 12:30 | |
| jrosser | i think we will use it very much like python_venv_build, with a set of vars: to pass in | 12:31 |
| noonedeadpunk | yeah, might make sense to use tasks_from to generate root ca | 12:31 |
| jrosser | yeah, as thats the 'unusual' one off use of it | 12:32 |
| noonedeadpunk | or just provide pki_root_generate_only: true | 12:32 |
| noonedeadpunk | but tasks from sounds like more appropriate | 12:33 |
| * jrosser asks my team to review the CA spec some more | 12:38 | |
| *** pcaruana has joined #openstack-ansible | 12:42 | |
| gokhani | jrosser, are you using focal or bionic for nvidia vgpu ? | 12:53 |
| noonedeadpunk | https://review.opendev.org/q/topic:%22osa%252Fpki%22+(status:open%20OR%20status:merged) | 13:01 |
| * noonedeadpunk googles grammarly integration into vs code | 13:05 | |
| jrosser | gokhani: we have a lab environment which runs bionic + victoria release | 13:05 |
| jrosser | with T4 vgpu | 13:05 |
| noonedeadpunk | or should give my wife to read through all texts I'm writing... | 13:07 |
| openstackgerrit | Andrew Bonney proposed openstack/openstack-ansible stable/ussuri: Ensure kuryr repo is available within CI images https://review.opendev.org/c/openstack/openstack-ansible/+/771608 | 13:08 |
| *** hamzaachi has joined #openstack-ansible | 13:08 | |
| gokhani | jrosser, ok thanks. ı have v100 and rtx 8000 gpus. I think I need to enable VFIO in kernel. ı hope it also works in focal. ı will try it for both focal and bionic. | 13:13 |
| jrosser | gokhani: as far as i can see the nvidia driver is handling that http://paste.openstack.org/show/802171/ | 13:18 |
| jrosser | then on the kernel boot parameters we have modprobe.blacklist=nouveau intel_iommu=on | 13:19 |
| *** rh-jelabarre has joined #openstack-ansible | 13:24 | |
| *** rh-jelabarre has quit IRC | 13:25 | |
| *** rh-jelabarre has joined #openstack-ansible | 13:25 | |
| jrosser | oddly, we may need to backport and merge this to stable/victoria before it can merge to master https://review.opendev.org/c/openstack/openstack-ansible/+/773352 | 13:26 |
| jrosser | the ceph-ansible sha will be the same on both branches :( | 13:26 |
| jrosser | well, or make upgrade jobs nv actually | 13:27 |
| noonedeadpunk | to fix upgrade... yeah... | 13:27 |
| noonedeadpunk | and we have circular one with centos distro jobs... | 13:28 |
| jrosser | any preference? i just saw the job is failing on the master patch | 13:28 |
| jrosser | so we should do something for victora | 13:28 |
| jrosser | maybe patch directly with the same topic | 13:29 |
| gokhani | jrosser, thanks a lot, I am trying now. I will inform you. | 13:30 |
| jrosser | gokhani: we have an ansible role for this but not public currently, i will look if i can get that changed | 13:31 |
| noonedeadpunk | jrosser: we will need to do smth nasty with V anyway actually | 13:31 |
| noonedeadpunk | because centos distro is failing there as well because of https://review.opendev.org/c/openstack/openstack-ansible-openstack_hosts/+/773369 | 13:32 |
| noonedeadpunk | which won't merge without bump or that patch... | 13:32 |
| noonedeadpunk | probably in bump we should just set centos distro to nv | 13:32 |
| jrosser | i have a victoria patch for the clone depth | 13:33 |
| jrosser | i can make all the centos stuff nv in the same one | 13:33 |
| noonedeadpunk | then let's do this then | 13:33 |
| noonedeadpunk | and then partially revert | 13:34 |
| jrosser | then we should be able to rebase the others on top and hopefully see it succeed | 13:34 |
| jrosser | +/- upgrade jobs maybe | 13:34 |
| *** SiavashSardari has quit IRC | 13:34 | |
| *** gokhani has quit IRC | 13:35 | |
| *** yasemind has quit IRC | 13:35 | |
| *** gokhani has joined #openstack-ansible | 13:35 | |
| openstackgerrit | Daniel Meloy proposed openstack/openstack-ansible-os_nova master: Add Virtual GPU Config to nova.conf template https://review.opendev.org/c/openstack/openstack-ansible-os_nova/+/768117 | 13:35 |
| *** yasemind has joined #openstack-ansible | 13:36 | |
| openstackgerrit | Jonathan Rosser proposed openstack/openstack-ansible stable/victoria: Increase git clone depth from 10 to 20 https://review.opendev.org/c/openstack/openstack-ansible/+/773391 | 13:37 |
| openstackgerrit | Jonathan Rosser proposed openstack/openstack-ansible stable/victoria: Return centos-8 jobs to voting https://review.opendev.org/c/openstack/openstack-ansible/+/773393 | 13:38 |
| *** zul has joined #openstack-ansible | 13:40 | |
| openstackgerrit | Merged openstack/openstack-ansible master: Imported Translations from Zanata https://review.opendev.org/c/openstack/openstack-ansible/+/773229 | 13:42 |
| *** SiavashSardari has joined #openstack-ansible | 13:44 | |
| openstackgerrit | Daniel Meloy proposed openstack/openstack-ansible-os_nova master: Add Virtual GPU Config to nova.conf template https://review.opendev.org/c/openstack/openstack-ansible-os_nova/+/768117 | 13:46 |
| openstackgerrit | Merged openstack/openstack-ansible-lxc_container_create master: Fix formatting of LXC container config files https://review.opendev.org/c/openstack/openstack-ansible-lxc_container_create/+/772706 | 13:50 |
| *** d34dh0r53 has joined #openstack-ansible | 13:51 | |
| openstackgerrit | Andrew Bonney proposed openstack/openstack-ansible-lxc_container_create stable/victoria: Fix formatting of LXC container config files https://review.opendev.org/c/openstack/openstack-ansible-lxc_container_create/+/773416 | 13:56 |
| *** spatel has joined #openstack-ansible | 14:19 | |
| spatel | jrosser: could you please share your journalbeat config file? i am trying to play but somehow its not working so i am sure i missed something. | 14:21 |
| openstackgerrit | Dmitriy Rabotyagov proposed openstack/openstack-ansible-specs master: Add Root CA spec https://review.opendev.org/c/openstack/openstack-ansible-specs/+/758805 | 14:22 |
| *** gaudenz has joined #openstack-ansible | 14:24 | |
| gaudenz | Hi | 14:24 |
| gaudenz | I already asked about the status of the centralized logging infrastructure in OSA last week. I had a closer look now, but it's still not clear to me how this is supposed to work in Rocky and later. | 14:25 |
| gaudenz | I now that (container) services are now configured to log to the systemd journal. And I know about the infra-journal-remote.yml playbook and that this is currently disabled because of a systemd bug. | 14:26 |
| spatel | gaudenz: only solution is to use third-party log shipper to ship log to centralized logging server | 14:27 |
| gaudenz | But AFAICS this playbook only configures remote journal shipping for physical hosts. | 14:27 |
| spatel | gaudenz: that bug has been fixed in Ubuntu 20.04 i believe so worth trying to enable to test out. | 14:27 |
| gaudenz | spatel: So there is no longer a built in solution to ship all logs to a central container? This was very useful up until now. | 14:27 |
| gaudenz | I know the bug has been fixed and AFAIK it can also be worked around by manually deleting journal files. But what I'm missing is how I can configure journal log shipping from containers to a central logging container like it was setup with rsyslog. | 14:28 |
| spatel | gaudenz: because of that bug no. (in my world i used dedicated graylog server for logging. i don't use container based syslog. | 14:28 |
| openstackgerrit | Dmitriy Rabotyagov proposed openstack/openstack-ansible-specs master: Add Root CA spec https://review.opendev.org/c/openstack/openstack-ansible-specs/+/758805 | 14:28 |
| spatel | for new deployment i am trying to setup journalbeat to ship logs to centralized graylog. | 14:29 |
| gaudenz | spatel: But even without the bug, as far as I understand the playbook it does not setup remote log shipping from containers. Or do I miss something? I'm more looking for a short term solution to at least have all logs in one place again. Longer term I'm looking into a log aggregation outside of OSA. | 14:31 |
| spatel | gaudenz: playbook is here but may need to test if it works or not after enabling journactl loggin https://github.com/openstack/openstack-ansible/blob/master/playbooks/infra-journal-remote.yml | 14:35 |
| spatel | I didn't test all those stuff because i already have my own logging infrastucture | 14:36 |
| gaudenz | I know this playbook and already tried it out. But it only configures logging on physical hosts. It does not even target the containers... My question is if this is intentional or if this is just a bug in this playbook. | 14:37 |
| *** partlycloudy has joined #openstack-ansible | 14:40 | |
| SiavashSardari | gaudenz the mentioned playbook is quite nice, I had the same requirement and after testing some tools I decided to go with vector and ship logs directly to elasticsearch. https://vector.dev/ | 14:53 |
| spatel | SiavashSardari: what client are you using to ship logs? | 14:56 |
| gaudenz | SiavashSardari: Did you modify the playbook to also install systemd-journal-upload into containers or how do ship logs from containers? | 14:58 |
| spatel | i am playing with journalbeat but somehow logs not ending up on centralized server, I can see on tcpdump but not in elk index | 14:58 |
| SiavashSardari | @spa | 14:59 |
| SiavashSardari | spatel I used vector as the client. https://vector.dev/docs/reference/sources/journald/ | 15:00 |
| SiavashSardari | journalbeat has some issues. I don't remember what they were, but finally we chose vector over journalbeat. | 15:01 |
| spatel | SiavashSardari: interesting, do you have sample config file? If possible | 15:02 |
| SiavashSardari | gaudenz it was a long time ago, but I have some vague memory of using systemd-journal-upload. maybe I can find more info on our git logs | 15:03 |
| SiavashSardari | spatel it is very simple. let me find one for you | 15:03 |
| SiavashSardari | spatel http://paste.openstack.org/show/802179/ | 15:05 |
| SiavashSardari | you can find more sophisticated scenarios on their website. | 15:06 |
| spatel | thanks you | 15:08 |
| SiavashSardari | gaudenz the infra-journal-remote playbook already installs journal upload and journal remote. I guess I didn't understand what is your problem with that | 15:10 |
| *** ilush has quit IRC | 15:16 | |
| *** yasemind has quit IRC | 15:16 | |
| *** gokhani has quit IRC | 15:16 | |
| *** SiavashSardari has quit IRC | 15:18 | |
| openstackgerrit | Dmitriy Rabotyagov proposed openstack/openstack-ansible-specs master: Add Root CA spec https://review.opendev.org/c/openstack/openstack-ansible-specs/+/758805 | 15:39 |
| *** bverschueren has joined #openstack-ansible | 15:39 | |
| *** gokhani has joined #openstack-ansible | 15:43 | |
| *** gokhani has quit IRC | 15:48 | |
| *** jbadiapa has quit IRC | 15:56 | |
| *** miloa has quit IRC | 15:59 | |
| *** macz_ has joined #openstack-ansible | 16:11 | |
| *** jbadiapa has joined #openstack-ansible | 16:28 | |
| *** gaudenz has quit IRC | 16:30 | |
| jrosser | oh too late gaudenz has left, but the container journals are bind mounted onto the hosts which is why you don't need to put journalbeat on all the containers | 16:40 |
| jrosser | so you can put journalbeat on each metal host | 16:40 |
| jrosser | or play with systemd log forwarding and get everything to one host (spof?) and run journalbeat there | 16:40 |
| jrosser | to make this work whatever journal collector you use needs to be able to accept a set of extra journal locations to consume | 16:43 |
| openstackgerrit | Merged openstack/openstack-ansible stable/ussuri: Bump SHAs for stable/ussuri https://review.opendev.org/c/openstack/openstack-ansible/+/773263 | 16:46 |
| openstackgerrit | Dmitriy Rabotyagov proposed openstack/openstack-ansible-specs master: Add Root CA spec https://review.opendev.org/c/openstack/openstack-ansible-specs/+/758805 | 16:46 |
| jrosser | seems we unwedge victoria a bit now https://review.opendev.org/c/openstack/openstack-ansible/+/773391 | 16:50 |
| openstackgerrit | Dmitriy Rabotyagov proposed openstack/openstack-ansible-openstack_hosts stable/victoria: Set RDO repo to Victoria https://review.opendev.org/c/openstack/openstack-ansible-openstack_hosts/+/773369 | 16:53 |
| *** cshen has quit IRC | 16:56 | |
| *** cshen has joined #openstack-ansible | 17:06 | |
| *** cshen has quit IRC | 17:11 | |
| *** rpittau is now known as rpittau|afk | 17:21 | |
| spatel | jrosser: hey! did you put journalbeat on Host/metal only or each container? | 17:22 |
| jrosser | only on the host | 17:26 |
| *** ChiTo has joined #openstack-ansible | 17:27 | |
| spatel | can you share your journalbeat.yml file. i would be interested to see | 17:27 |
| jrosser | https://github.com/openstack/openstack-ansible-ops/blob/master/elk_metrics_7x/roles/elastic_journalbeat/templates/journalbeat.yml.j2#L16-L20 | 17:27 |
| ChiTo | Hi OSA team | 17:27 |
| jrosser | it's all in the elk roles in the ops repo | 17:27 |
| *** hamzaachi has quit IRC | 17:27 | |
| jrosser | spatel: from the template the file is huge, which bit do you want to see | 17:27 |
| ChiTo | I am getting a lot of RabbitMQ errors from the oslo.messaging driver when I enable the notifications, by chance have you experienced this issue? | 17:28 |
| spatel | journalbeat.inputs: section | 17:28 |
| spatel | you have put each mounted directory in path right? | 17:28 |
| ChiTo | Feb 1 14:24:41 comp1 nova-compute: 2021-02-01 14:24:41.726 146452 INFO oslo.messaging._drivers.impl_rabbit [req-5b67bc33-dd95-4f92-ab3a-5abc0105e64c - - - - -] [2c1a5e2b-44a2-42a7-a638-6394c3bb7673] Reconnected to AMQP server on 172.28.118.74:5671 via [amqp] client with port 35180. | 17:28 |
| ChiTo | Feb 1 14:24:41 comp1 nova-compute: 2021-02-01 14:24:41.730 146452 ERROR oslo.messaging._drivers.impl_rabbit [req-5b67bc33-dd95-4f92-ab3a-5abc0105e64c - - - - -] The broker has blocked the connection: connection blocked, see broker logs | 17:28 |
| jrosser | spatel: http://paste.openstack.org/show/802190/ | 17:29 |
| jrosser | the template takes care of generating all of that | 17:29 |
| ChiTo | I have read a lot about this, but there are a l ot of theories, some of them that there is a bug on oslo.messaging and that there is a class called oslo.rootwrap that it looks it is not part of my cluster. I am on Train, but just wonder if you have expxeriencecd this kind of outages, it only happens when I enable notifications for Oslo, due I need panko eventually | 17:30 |
| jrosser | spatel: look at the docs https://www.elastic.co/guide/en/beats/journalbeat/current/journalbeat-installation-configuration.html | 17:30 |
| jrosser | you can now give a directory instead of listing all the journals | 17:31 |
| jrosser | i've not tried that though | 17:31 |
| *** cshen has joined #openstack-ansible | 17:31 | |
| spatel | jrosser: let me understand what are these directories? /var/log/journal/ | 17:32 |
| spatel | are they part of each containers? | 17:32 |
| spatel | trying to understand how container journactl endup on host /var/log/journal/ directories | 17:34 |
| *** jbadiapa has quit IRC | 17:34 | |
| jrosser | https://github.com/openstack/openstack-ansible-lxc_container_create/blob/master/defaults/main.yml#L171-L175 | 17:35 |
| jrosser | https://opendev.org/openstack/openstack-ansible-lxc_container_create/src/branch/master/tasks/lxc_container_config.yml#L261-L291 | 17:36 |
| spatel | oh! so these journal directories are mapped with container | 17:37 |
| jrosser | well actually this is confusing | 17:38 |
| jrosser | becasue https://review.opendev.org/c/openstack/openstack-ansible/+/771205 | 17:38 |
| noonedeadpunk | btw should we backport this? | 17:40 |
| noonedeadpunk | I guess not? | 17:40 |
| * jrosser wondering why this is in two places | 17:40 | |
| spatel | jrosser: in this solution don't you think it would be little complicated to search logs for specific container ? | 17:42 |
| noonedeadpunk | you still can get into container and search logs for it | 17:43 |
| noonedeadpunk | otherwsie you can see journal for needed services only | 17:43 |
| spatel | in my older cloud i have each container sending logs to graylog and i can see source of that logs | 17:43 |
| jrosser | all it does is bring all the journal files to be visible on the host | 17:43 |
| jrosser | then you can use one collector rather than one collector per container | 17:43 |
| jrosser | the journal files contain enough data to describe the source of each entry | 17:44 |
| spatel | let me give it a shot and see how it looks on centralize server | 17:44 |
| jrosser | we found that 15 copies of journalbeat per host was chewing a lot of resource and it was much better to have just the one | 17:44 |
| jrosser | but do check that the journals look correct on the host in light of the patch i just posted | 17:45 |
| *** maharg101 has quit IRC | 17:48 | |
| spatel | jrosser: let me see | 17:48 |
| jrosser | noonedeadpunk: discussion just now in #openstack-keystone about memcached implementation in keystone and workarounds being made in debian packages | 17:50 |
| jrosser | this has a very similar feel to memcached related CI failures I see inside keystone reguarly on debian jobs | 17:51 |
| openstackgerrit | Dmitriy Rabotyagov proposed openstack/ansible-hardening master: Fix linter errors https://review.opendev.org/c/openstack/ansible-hardening/+/771481 | 17:55 |
| noonedeadpunk | just joined (wasn't in the cahnnel :() | 17:55 |
| jrosser | https://review.opendev.org/c/openstack/kolla-ansible/+/746966 | 17:55 |
| jrosser | and the comments on here are enlightening https://review.opendev.org/c/openstack/oslo.cache/+/742193 | 17:56 |
| noonedeadpunk | memcache_use_advanced_pool whaaaat | 17:57 |
| noonedeadpunk | well, it seems it's under control kind of.... | 17:58 |
| jrosser | unfortunately looks like opt-in for all the services | 18:01 |
| noonedeadpunk | I'm wondering if they're going to backport | 18:05 |
| *** andrewbonney has quit IRC | 18:05 | |
| jrosser | i'm failing to find a job failed with keystone blowing up with memcached connection trouble | 18:06 |
| noonedeadpunk | I can recall smth like this yes | 18:06 |
| noonedeadpunk | never catched in prod though | 18:08 |
| jrosser | looks like the patch to oslo.cache is kind of additional issue to using internal vs. oslo cache pool in keystonemiddleware | 18:10 |
| *** fridtjof[m] has quit IRC | 18:10 | |
| *** fridtjof[m] has joined #openstack-ansible | 18:24 | |
| noonedeadpunk | wtf is going on with hardening... | 18:27 |
| noonedeadpunk | apparmor seems to be installed https://6d5945a71e766b4afadf-276cdc89d597e728af064ca9b4be4e44.ssl.cf1.rackcdn.com/771481/5/check/openstack-ansible-functional-ubuntu-focal/dcff537/logs/ara-report/results/14.html | 18:27 |
| noonedeadpunk | but it's not in apt history log https://6d5945a71e766b4afadf-276cdc89d597e728af064ca9b4be4e44.ssl.cf1.rackcdn.com/771481/5/check/openstack-ansible-functional-ubuntu-focal/dcff537/logs/host/apt/history.log.txt | 18:27 |
| noonedeadpunk | and neither present https://6d5945a71e766b4afadf-276cdc89d597e728af064ca9b4be4e44.ssl.cf1.rackcdn.com/771481/5/check/openstack-ansible-functional-ubuntu-focal/dcff537/logs/ara-report/results/71.html | 18:27 |
| noonedeadpunk | hm and I can reproduce that... | 18:30 |
| mgariepy | isn't apparmor installed by default in the imagE? | 18:38 |
| noonedeadpunk | in infra one it's not nowadays | 18:39 |
| noonedeadpunk | the question is why it's not installed with asnible... | 18:39 |
| openstackgerrit | Dmitriy Rabotyagov proposed openstack/openstack-ansible-tests master: Bump ansible-base to 2.10.5 https://review.opendev.org/c/openstack/openstack-ansible-tests/+/773467 | 18:48 |
| openstackgerrit | Dmitriy Rabotyagov proposed openstack/openstack-ansible-tests master: Bump ansible-base to 2.10.5 https://review.opendev.org/c/openstack/openstack-ansible-tests/+/773467 | 18:49 |
| *** poopcat has quit IRC | 18:50 | |
| *** hamzaachi has joined #openstack-ansible | 18:50 | |
| *** poopcat has joined #openstack-ansible | 18:51 | |
| mgariepy | noonedeadpunk, it seems like the logs are cutting at a certain time.. | 18:52 |
| noonedeadpunk | yeah and feels like at point where apparmor should be setting up | 18:53 |
| mgariepy | dpkg.log ends a 18:07 | 18:53 |
| mgariepy | the install is only done a 18:11 | 18:54 |
| *** hamzaachi has quit IRC | 18:56 | |
| *** hamzaachi has joined #openstack-ansible | 18:56 | |
| noonedeadpunk | the weird thing is that tox -e functional fail, while running role in shell not (ie .tox/functional/bin/ansible-playbook tests/test.yml -i tests/inventory) and correctly installs apparmor | 18:57 |
| mgariepy | is tox using the flaw in sudo that was patched last week ? lol | 18:58 |
| noonedeadpunk | lol | 19:03 |
| mgariepy | when you run via tox do you see if the pkg not geting installed via ansible ? | 19:07 |
| mgariepy | or it's installed and wiped after? | 19:08 |
| noonedeadpunk | it's not installed I'd say | 19:09 |
| mgariepy | bionic have the same issue. | 19:09 |
| noonedeadpunk | yeah, I'm playing on bionic actually right now | 19:10 |
| mgariepy | wonder what changed.. on the jan 17 it was passing.. | 19:23 |
| noonedeadpunk | sounds like sudo patch... | 19:23 |
| mgariepy | lol. | 19:24 |
| mgariepy | not really. | 19:24 |
| mgariepy | opps. on jan 13 it was failing. | 19:25 |
| noonedeadpunk | maybe just infra changed images actually | 19:26 |
| mgariepy | lol. | 19:26 |
| mgariepy | i'm all foo bar in the dates.. | 19:26 |
| *** hamzaachi_ has joined #openstack-ansible | 19:33 | |
| *** hamzaachi has quit IRC | 19:33 | |
| openstackgerrit | Merged openstack/openstack-ansible stable/victoria: Increase git clone depth from 10 to 20 https://review.opendev.org/c/openstack/openstack-ansible/+/773391 | 19:35 |
| *** hamzaachi_ has quit IRC | 19:36 | |
| *** hamzaachi has joined #openstack-ansible | 19:40 | |
| *** hamzaachi has quit IRC | 19:41 | |
| *** maharg101 has joined #openstack-ansible | 19:45 | |
| *** maharg101 has quit IRC | 19:50 | |
| spatel | I am getting this error on jouenalbeat - Failed to connect to backoff(elasticsearch(http://10.30.0.92:5044)): Get "http://10.30.0.92:5044": EOF | 19:59 |
| spatel | i can see 5044 port is opened and i can telnet/netcat on that port | 20:00 |
| spatel | working now! it was stupid typo error in config file :) | 20:05 |
| spatel | jrosser: in journal logs i am not seeing any container hostname, all its showing logs coming from infra* nodes | 20:17 |
| spatel | in your case are you seeing source of logs file? | 20:18 |
| *** guilhermesp__ has joined #openstack-ansible | 20:18 | |
| *** jrosser_ has joined #openstack-ansible | 20:18 | |
| *** fyx_ has joined #openstack-ansible | 20:18 | |
| *** gundalow_ has joined #openstack-ansible | 20:18 | |
| *** janno_ has joined #openstack-ansible | 20:20 | |
| *** cshen has quit IRC | 20:22 | |
| *** cshen has joined #openstack-ansible | 20:22 | |
| *** gundalow has quit IRC | 20:26 | |
| *** jrosser has quit IRC | 20:26 | |
| *** guilhermesp has quit IRC | 20:26 | |
| *** fyx has quit IRC | 20:26 | |
| *** janno has quit IRC | 20:26 | |
| *** zigo has quit IRC | 20:26 | |
| *** jrosser_ is now known as jrosser | 20:26 | |
| *** guilhermesp__ is now known as guilhermesp | 20:26 | |
| *** fyx_ is now known as fyx | 20:26 | |
| *** gundalow_ is now known as gundalow | 20:26 | |
| *** zigo has joined #openstack-ansible | 20:31 | |
| *** gyee has joined #openstack-ansible | 21:16 | |
| *** ChiTo has quit IRC | 21:35 | |
| *** cshen has quit IRC | 21:41 | |
| *** maharg101 has joined #openstack-ansible | 21:46 | |
| *** cshen has joined #openstack-ansible | 21:50 | |
| *** maharg101 has quit IRC | 21:50 | |
| *** jbadiapa has joined #openstack-ansible | 22:03 | |
| *** jbadiapa has quit IRC | 22:07 | |
| *** Underknowledge has quit IRC | 22:07 | |
| *** Underknowledge has joined #openstack-ansible | 22:08 | |
| *** Underknowledge has quit IRC | 22:16 | |
| *** Underknowledge has joined #openstack-ansible | 22:17 | |
| *** poopcat has quit IRC | 22:39 | |
| *** poopcat has joined #openstack-ansible | 22:41 | |
| *** Underknowledge has quit IRC | 22:47 | |
| *** Underknowledge has joined #openstack-ansible | 22:47 | |
| *** spatel has quit IRC | 22:53 | |
| *** poopcat has quit IRC | 23:07 | |
| *** poopcat has joined #openstack-ansible | 23:09 | |
| *** poopcat has quit IRC | 23:45 | |
| *** maharg101 has joined #openstack-ansible | 23:47 | |
| *** poopcat has joined #openstack-ansible | 23:47 | |
| *** tosky has quit IRC | 23:52 | |
| *** maharg101 has quit IRC | 23:52 | |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!