| moha7 | For OpenStack monitoring, is there a specially developed tool, something like vRops or OneView for VMWare products? | 05:48 |
|---|---|---|
| moha7 | In general, do you know of a reference where I can read about OpenStack monitoring practices? Or, I would be grateful if you could share the experiences and tools you use here. | 05:49 |
| opendevreview | Jorge San Emeterio proposed openstack/openstack-ansible-os_tempest master: [DNM] Restarting glance before running tempest https://review.opendev.org/c/openstack/openstack-ansible-os_tempest/+/862304 | 07:59 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible master: Add zookeeper deployment https://review.opendev.org/c/openstack/openstack-ansible/+/864750 | 09:01 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible master: Try to adjust Apache directives for AIO https://review.opendev.org/c/openstack/openstack-ansible/+/862922 | 09:01 |
| opendevreview | Jorge San Emeterio proposed openstack/openstack-ansible-os_tempest master: [DNM] Restarting glance before running tempest https://review.opendev.org/c/openstack/openstack-ansible-os_tempest/+/862304 | 09:48 |
| noonedeadpunk | well looks like client TLS works for zookeeper. Quorum TLS still not :( | 09:59 |
| noonedeadpunk | nah, it's not( | 10:12 |
| noonedeadpunk | ok, I found mistake at least.... | 10:32 |
| noonedeadpunk | jrosser: have you combined smth like that format for any service? https://paste.openstack.org/show/b5zMRAbdsZoSDxiGBCDS/ | 10:33 |
| noonedeadpunk | like.... libvirt? | 10:36 |
| jrosser | noonedeadpunk: i'm not sure - is that the cert and it's own private key in the same file? | 10:47 |
| jrosser | libvirt was also strange too | 10:47 |
| noonedeadpunk | and info file on top I believe | 10:48 |
| jrosser | huh no | 10:49 |
| jrosser | the info file really was only ever for my own benefit for debugging | 10:49 |
| jrosser | surprising if zk wants that, is it a standardised format? | 10:50 |
| noonedeadpunk | or well | 10:50 |
| noonedeadpunk | I have close to no idea how stadartized format it is - it's Java after all | 10:51 |
| noonedeadpunk | so it supposed to be even jks or smth, but pem somehow is supported as well | 10:51 |
| jrosser | the only way that libvirt was strange was how it wanted a chain constructed | 10:52 |
| jrosser | which was different from anything else | 10:52 |
| noonedeadpunk | ah, ok... | 10:52 |
| jrosser | two different things we already had combined in the same file | 10:53 |
| noonedeadpunk | I wonder how info file is required... | 10:53 |
| noonedeadpunk | Basically I failed to generate with pki role till now, but this https://opendev.org/zuul/zuul/src/branch/master/tools/zk-ca.sh does work | 10:54 |
| noonedeadpunk | So I'd suppose it should be jsut cert/key... | 10:54 |
| jrosser | what java version do you have | 10:55 |
| noonedeadpunk | (not sure about ca) but then it fails to read private key | 10:55 |
| noonedeadpunk | 11 | 10:55 |
| noonedeadpunk | hm. maybe it doesn't like SAN | 10:57 |
| jrosser | well maybe we need to extend the pki role to make use of keytool? | 10:58 |
| noonedeadpunk | nah, openssl works just nice for infra | 10:58 |
| noonedeadpunk | and certs generated by zk-ca.sh works nicely | 10:58 |
| jrosser | theres a couple of different choices of keystore format i think | 10:58 |
| noonedeadpunk | I'm indeed about to claim SAN cert I've tried out.... | 11:00 |
| jrosser | feels like we have a ton to merge / fix and not a whole lot of time | 11:03 |
| noonedeadpunk | yeah, that's true | 11:03 |
| noonedeadpunk | as in fact info at beginning seems not required | 11:06 |
| *** dviroel|afk is now known as dviroel | 11:15 | |
| noonedeadpunk | so the only differences I can spot are subject and rsa length.... | 11:26 |
| noonedeadpunk | https://paste.openstack.org/show/bRsXk3jT6vdR5QJbonnU/ test is generated with pki role and refused, test1 is generated with infra script and works | 11:27 |
| noonedeadpunk | If it simply doesn't like 4096 length of rsa.... | 11:29 |
| noonedeadpunk | I will be surprised tbh | 11:29 |
| noonedeadpunk | well, 2048 didn't hlped either... | 11:31 |
| noonedeadpunk | But fail with ` Inappropriate key specification: IOException : algid parse error, not a sequence` https://paste.openstack.org/show/beg6kQfUXXQzeDlA9OFO/ | 11:32 |
| noonedeadpunk | evnetually.... header for infra rsa starts with -----BEGIN PRIVATE KEY----- and our with -----BEGIN RSA PRIVATE KEY----- | 11:34 |
| noonedeadpunk | huh, that's interesting | 11:35 |
| noonedeadpunk | that's way closer to the issue I guess :) | 11:36 |
| noonedeadpunk | damn yes | 11:39 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/ansible-role-pki master: Allow to specify format for private keys https://review.opendev.org/c/openstack/ansible-role-pki/+/865420 | 11:49 |
| opendevreview | Marcus Klein proposed openstack/openstack-ansible-os_neutron master: Allow to set dnsmasq configuration options https://review.opendev.org/c/openstack/openstack-ansible-os_neutron/+/864872 | 12:21 |
| kleini | noonedeadpunk: ^^^ I tried to write release notes. Hope, this is understandable. | 12:23 |
| opendevreview | Merged openstack/openstack-ansible stable/wallaby: Bump services for EM release https://review.opendev.org/c/openstack/openstack-ansible/+/864263 | 13:51 |
| noonedeadpunk | kleini: well, you need only features section out of all of that | 14:00 |
| noonedeadpunk | Well, you can also leave prelude maybe, but keep in mind that this release note will be renderred for whole openstack-ansible release | 14:01 |
| kleini | So I should drop prelude and upgrade? | 14:02 |
| noonedeadpunk | well, that would made sense to me when it end up here https://docs.openstack.org/releasenotes/openstack-ansible/unreleased.html | 14:13 |
| opendevreview | Marcus Klein proposed openstack/openstack-ansible-os_neutron master: Allow to set dnsmasq configuration options https://review.opendev.org/c/openstack/openstack-ansible-os_neutron/+/864872 | 14:20 |
| kleini | DONE | 14:20 |
| spatel | My company asking to deploy Openstack billing to find out COGS and expense to run VMs :) | 15:04 |
| spatel | Does anyone running any kind of billing with openstack? I am very interested to hear stories | 15:04 |
| *** dviroel is now known as dviroel|lunch | 15:07 | |
| noonedeadpunk | Well, there're multiple ways to do that depending on what/how you want to bill | 15:08 |
| noonedeadpunk | On one of previous companis we were running ceilometer+gnocchi and then self-written plugin for billing system that was fetching data from gnocchi and actually invoice customers | 15:09 |
| noonedeadpunk | But eventually you can get also cloudkitty for that | 15:09 |
| spatel | noonedeadpunk what are you guys using right now? | 15:20 |
| spatel | ceilometer+gnocchi+ cloudkitty ? | 15:20 |
| spatel | or in-house billing module | 15:21 |
| spatel | I think i have to start playing.. then i will know what is good and bad.. | 15:23 |
| noonedeadpunk | right now it's fully in-house and we bill just by flavors | 15:23 |
| spatel | assuming you have dedicated gnocchi/ceilometer box for billing and not part of control plane | 15:23 |
| noonedeadpunk | so no fancy stuff like iops, cpu cycles, etc | 15:23 |
| spatel | Man!! that is what i want.. i don't care about iops/cpu etc.. | 15:24 |
| spatel | I want flat billing.. if flavor-1 vm running last 6 days then bill will be $$ | 15:24 |
| noonedeadpunk | well, you stil lcan do that with ceilometer | 15:24 |
| noonedeadpunk | and cloudkitty | 15:24 |
| noonedeadpunk | should be super straighforward tbh | 15:25 |
| spatel | Even for flavor base billing do i need ceilometer/gnocchi ? | 15:25 |
| spatel | sorry if i am asking stupid question because i am new for billing stuff :) | 15:26 |
| noonedeadpunk | So for that you need to ask nova/cinder for list of vms, sort it by projects, have some database and do calculations on your billing side. | 16:02 |
| noonedeadpunk | What does ceilometer make - every time when instance is created nova can post message into rabbitmq notifications queue | 16:02 |
| noonedeadpunk | then ceilometer consumes it and stores data in publisher (like gnocchi) | 16:03 |
| noonedeadpunk | So when vm is created or deleted or resized - ceilometer will know that from nova through rabbit queue | 16:03 |
| noonedeadpunk | cloudkitty is already a billing, but it counts in units. So you can tell cloudkitty that 1gb of disk costs 0.05 units/hour, 1 cpu costs 0.02 units, etc. And cloudkitty based on gnocchi will make a report on consumtions | 16:04 |
| noonedeadpunk | so if you want to have an in-house solution - that is doable but there're quite a lot of things you need to think through | 16:05 |
| *** dviroel_ is now known as dviroel | 16:15 | |
| mgariepy | https://zuul.opendev.org/t/openstack/build/de1cde92ab6a4e8fba2c80f9cdc85c14/log/logs/host/neutron-server.service.journal-15-45-35.log.txt#4043 | 16:29 |
| mgariepy | which leads to : https://bugs.launchpad.net/ubuntu/+source/neutron/+bug/1995738 | 16:30 |
| mgariepy | i was looking at the progress of https://review.opendev.org/c/openstack/openstack-ansible/+/865312 | 16:42 |
| mgariepy | this time around rocky9 had failed. | 16:43 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/ansible-role-zookeeper master: Add SSL support for zookeeper https://review.opendev.org/c/openstack/ansible-role-zookeeper/+/865449 | 16:43 |
| noonedeadpunk | ouch | 16:48 |
| mgariepy | tempest failed on c9s because keystone went down :/ | 16:48 |
| mgariepy | not sure if it's because it was lacking some threads or something else. | 16:49 |
| noonedeadpunk | Yeah. And to merge keystone patch we need this one anyway | 16:49 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/ansible-role-zookeeper master: Add SSL support for zookeeper https://review.opendev.org/c/openstack/ansible-role-zookeeper/+/865449 | 16:49 |
| mgariepy | not 100% sure why it's so instable. | 16:49 |
| mgariepy | when we get on slower compute node.. we fall into a race condition ? | 16:50 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible master: Add zookeeper deployment https://review.opendev.org/c/openstack/openstack-ansible/+/864750 | 16:51 |
| noonedeadpunk | Not sure. But I also catched same on localhost | 16:51 |
| mgariepy | so we need to bump the threads for the services ? | 16:52 |
| noonedeadpunk | I had troubles with even listing things, becuase apache connections were ran out and haproxy due to that was marking keystone as down | 16:52 |
| noonedeadpunk | I wanted to try this out https://review.opendev.org/c/openstack/openstack-ansible/+/862922/2 | 16:52 |
| noonedeadpunk | as what I saw locally was specifically apache mpm stuff | 16:57 |
| mgariepy | for rocky 9 it was galera and neutron (i think neutron has restarted. | 16:57 |
| mgariepy | not sure why galera went down either. haven't found anything. | 16:58 |
| noonedeadpunk | well... maybe indeed we have forgotten about some service... | 17:14 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/ansible-role-zookeeper master: Add SSL support for zookeeper https://review.opendev.org/c/openstack/ansible-role-zookeeper/+/865449 | 17:35 |
| noonedeadpunk | ok, now zookeeper tls seems working nicely | 17:47 |
| jrosser | nice work :) | 17:47 |
| mgariepy | woohoo | 17:47 |
| jrosser | good find on the rsa format nonsense too | 17:48 |
| noonedeadpunk | I will push patch for cinder/designate tomorrow | 17:48 |
| noonedeadpunk | it was tricky I must admit.... | 17:48 |
| noonedeadpunk | rsa nonsense | 17:48 |
| noonedeadpunk | as it's smth I wasn't expecting can be a thing at all | 17:48 |
| noonedeadpunk | unfortunate part I guess is that we don't store path to ca-certificates somewhere globally, do we? | 17:49 |
| noonedeadpunk | I mean this one https://review.opendev.org/c/openstack/ansible-role-zookeeper/+/865449/3/vars/debian.yml#21 | 17:50 |
| jrosser | No I think we have that already in openstack_hosts and also PKI | 17:55 |
| *** dviroel is now known as dviroel|afk | 21:25 | |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!