| *** fyx has quit IRC | 00:03 | |
| *** fyx has joined #openstack-ansible | 00:03 | |
| *** cshen has joined #openstack-ansible | 01:31 | |
| *** cshen has quit IRC | 01:36 | |
| *** cloudnull8 has joined #openstack-ansible | 02:04 | |
| *** cloudnull has quit IRC | 02:04 | |
| *** bverschueren has quit IRC | 02:04 | |
| *** cloudnull8 is now known as cloudnull | 02:04 | |
| *** bverschueren has joined #openstack-ansible | 02:04 | |
| *** timburke has quit IRC | 02:05 | |
| *** prometheanfire has quit IRC | 02:06 | |
| *** prometheanfire has joined #openstack-ansible | 02:07 | |
| *** timburke has joined #openstack-ansible | 02:11 | |
| *** dave-mccowan has quit IRC | 02:44 | |
| *** pcaruana has quit IRC | 04:04 | |
| *** andrea10 has quit IRC | 04:17 | |
| *** nurdie has quit IRC | 04:20 | |
| *** raukadah is now known as chkumar|rover | 04:25 | |
| *** rh-jelabarre has quit IRC | 04:29 | |
| *** evrardjp has quit IRC | 04:33 | |
| *** evrardjp has joined #openstack-ansible | 04:33 | |
| *** yolanda has quit IRC | 04:41 | |
| *** cloudnull is now known as kecarter | 05:19 | |
| *** kecarter is now known as cloudnull | 05:19 | |
| *** gyee has quit IRC | 05:20 | |
| *** andrea10 has joined #openstack-ansible | 05:23 | |
| *** ianychoi_ has joined #openstack-ansible | 05:25 | |
| *** ianychoi has quit IRC | 05:28 | |
| *** miloa has joined #openstack-ansible | 06:20 | |
| *** yolanda has joined #openstack-ansible | 06:50 | |
| *** jbadiapa has joined #openstack-ansible | 06:57 | |
| *** cshen has joined #openstack-ansible | 06:58 | |
| *** mensis has left #openstack-ansible | 07:04 | |
| *** rohit02 has joined #openstack-ansible | 07:05 | |
| openstackgerrit | Dmitriy Rabotyagov (noonedeadpunk) proposed openstack/openstack-ansible-os_congress stable/ussuri: Replace py2 packages with py3 https://review.opendev.org/746659 | 07:07 |
|---|---|---|
| jrosser | morning | 07:11 |
| masterpe | morning | 07:35 |
| masterpe | noonedeadpunk: about https://review.opendev.org/#/c/739146/9 and/or https://github.com/noonedeadpunk/ansible-pacemaker-corosync only installes corosync & pacemaker on the compute nodes. But if the pool of compute nodes is bigger than 16 nodes you will hit into trouble. | 07:40 |
| openstackgerrit | Chandan Kumar (raukadah) proposed openstack/openstack-ansible-os_tempest master: Set mode for copy operation https://review.opendev.org/746848 | 07:49 |
| noonedeadpunk | masterpe: I think pacemaker has bigger max size (ie 56 or 64 - smth like that) | 07:54 |
| noonedeadpunk | but can't find any proofs of that at the moment, not sure now | 07:55 |
| noonedeadpunk | Like you can set different clusters when it's bigger than allowed | 07:55 |
| noonedeadpunk | ie group computes in stack of 50 nodes or smth | 07:56 |
| noonedeadpunk | so you just need to define different pacemaker_corosync_group | 07:57 |
| noonedeadpunk | I think somebody here has such setup and it worked | 07:57 |
| *** mensis has joined #openstack-ansible | 08:18 | |
| *** tosky has joined #openstack-ansible | 08:28 | |
| snadge | evening ;) | 09:05 |
| masterpe | noonedeadpunk: If i read https://blog.gnuoy.eu/2019/04/openstack-automated-instance-recovery.html they stated 16 nodes. | 09:10 |
| masterpe | But indeed that I can cluster the nodes in smaller groups. | 09:10 |
| masterpe | that blog post they advised to use pacemaker remote. | 09:11 |
| openstackgerrit | Dmitriy Rabotyagov (noonedeadpunk) proposed openstack/openstack-ansible-os_ceilometer master: Updated from OpenStack Ansible Tests https://review.opendev.org/745842 | 09:23 |
| noonedeadpunk | masterpe: if you try it and it works - let me know :) | 09:28 |
| noonedeadpunk | I'd love to adjust role to get this working there as well | 09:29 |
| noonedeadpunk | or maybe you even can PR :p | 09:29 |
| noonedeadpunk | we can even create a separate containers on controlers to place pacemaker there | 09:31 |
| masterpe | we are going to use pacemaker remote, then we will place it in separate containers. | 09:32 |
| masterpe | On the controllers or other place (in our setup we also have separate ceph monitor servers) | 09:33 |
| masterpe | If I going to change it a PR is not a bad idea. | 09:38 |
| noonedeadpunk | eventually sharing the right way to do that will also help) as it seems more correct way, than splitting to groups of 16... | 10:04 |
| openstackgerrit | Chandan Kumar (raukadah) proposed openstack/openstack-ansible-os_tempest master: Set mode for copy operation https://review.opendev.org/746848 | 10:19 |
| openstackgerrit | Dmitriy Rabotyagov (noonedeadpunk) proposed openstack/openstack-ansible master: Decrease amount of jobs and update distros https://review.opendev.org/746881 | 10:35 |
| rohit02 | Hey there! does anyone have steps to configure external ceph in openstack-ansible ussuri | 11:01 |
| *** miloa has quit IRC | 11:03 | |
| noonedeadpunk | I think you just need either to provide directory with config files or list of monitors (at least one) where role can get everything needed | 11:18 |
| noonedeadpunk | ceph_mons variable for monitors, ceph_keyrings_dir for config dir with prepared stuff | 11:19 |
| noonedeadpunk | https://docs.openstack.org/openstack-ansible-ceph_client/latest/ | 11:19 |
| watersj | rohit02, here is an older example i used, http://paste.openstack.org/show/795976/ | 11:27 |
| watersj | you may need to specify version of ceph you are using too: "ceph_stable_release: nautilus" | 11:28 |
| kleini | rohit02, that example from watersj only works, if Ansible has access to ceph MONs. If that access is not possible, you can use the Ceph client config from file feature. http://paste.openstack.org/show/796289/ <- user_variables.yml and http://paste.openstack.org/show/796290/ <- an example for a keyring file | 11:54 |
| *** rh-jelabarre has joined #openstack-ansible | 11:58 | |
| *** cshen has quit IRC | 12:09 | |
| *** sshnaidm is now known as sshnaidm|afk | 12:26 | |
| masterpe | I'm using osa 19.1.0 and I want to install masakari, Like I said I have the issue that vine is to new, but if I limit the playbook os-masakari-install.yml to masakari_api and so it will left alone the monitors the correct vine version gets installed | 12:36 |
| masterpe | Does the settings for the masakari monitor mess something up? | 12:37 |
| jrosser | masterpe: i gave the place where the constraints file for the venv was in a link yesterday | 12:37 |
| jrosser | it would be worth checking if that is being applied | 12:37 |
| *** dave-mccowan has joined #openstack-ansible | 12:37 | |
| jrosser | (well its clearly not, but the question is why....) | 12:38 |
| masterpe | you mean https://github.com/openstack/requirements/blob/master/upper-constraints.txt#L133 ? | 12:38 |
| jrosser | that should be used here https://github.com/openstack/openstack-ansible-os_masakari/blob/master/defaults/main.yml#L33-L36 | 12:39 |
| jrosser | then used here https://github.com/openstack/openstack-ansible-os_masakari/blob/master/tasks/main.yml#L71 | 12:39 |
| masterpe | I have also https://review.opendev.org/#/c/744673/7/defaults/main.yml active | 12:40 |
| masterpe | so it also installs the monitors on the computes. | 12:40 |
| *** rh-jelabarre has quit IRC | 12:41 | |
| jrosser | masterpe> on the masakari container I get vine==5.0.0a1 | 12:41 |
| jrosser | ^ this? | 12:41 |
| masterpe | yes | 12:43 |
| masterpe | So when I use os-masakari-install.yml without --limit masakari_api it installs vine 5.0.0a1 and with os-masakari-install.yml --limit masakari_api it installs vine 1.2.0 | 12:44 |
| masterpe | my feeling is that it has todo with https://review.opendev.org/#/c/744673/7/ | 12:45 |
| jrosser | there is a very detailed log for the python_venv_build which will show exactly where everything has come from | 12:49 |
| openstackgerrit | Dmitriy Rabotyagov (noonedeadpunk) proposed openstack/openstack-ansible master: Decrease amount of jobs and update distros https://review.opendev.org/746881 | 12:56 |
| *** cshen has joined #openstack-ansible | 12:59 | |
| masterpe | jrosser: probably python_venv_build.log? | 13:01 |
| *** rh-jelabarre has joined #openstack-ansible | 13:02 | |
| jrosser | masterpe: yes, and that'll be in your repo server for a multinode deploy | 13:05 |
| jrosser | noonedeadpunk: i was thinking about the memory pressure thing again, we maybe not reducing the number of uwsgi threads for some of the newer roles...... | 13:06 |
| jrosser | theres a whole load of vars for that in the template for the AIO config | 13:06 |
| noonedeadpunk | I checked configs for uwsgi and all were like 2 processes | 13:07 |
| jrosser | ah ok | 13:07 |
| noonedeadpunk | maybe I missed smth... | 13:07 |
| jrosser | you are right though lots and lots of things are failing | 13:08 |
| jrosser | all of a sudden lots of unreliability in the CI when we had a good run of it being pretty spot on | 13:08 |
| *** rohit02 has quit IRC | 13:29 | |
| watersj | masakari error, list index out of range? http://paste.openstack.org/show/796948/ | 13:30 |
| openstackgerrit | Charles Farquhar proposed openstack/openstack-ansible-os_tempest stable/rocky: Use pip_install_upper_constraints for plugin deps https://review.opendev.org/746928 | 13:52 |
| openstackgerrit | Charles Farquhar proposed openstack/openstack-ansible-os_tempest stable/rocky: Use pip_install_upper_constraints for plugin deps https://review.opendev.org/746928 | 13:53 |
| guilhermesp | guys, i think we have a bug in the way we bump ironic-inspector in osa | 13:56 |
| guilhermesp | for example, currently on ussuri, the venv gets ironic-inspector==10.3.0 | 13:56 |
| guilhermesp | which is inspecetor for victoria | 13:57 |
| guilhermesp | that breaks ironic-inspector | 13:57 |
| guilhermesp | i was thinking we could pin inspector as well in our openstack_services.yml | 13:57 |
| guilhermesp | i.e | 13:57 |
| guilhermesp | https://www.irccloud.com/pastebin/pulbRh5e/ | 13:57 |
| openstackgerrit | Charles Farquhar proposed openstack/openstack-ansible-os_neutron stable/rocky: Remove dragonflow ML2 driver https://review.opendev.org/745377 | 13:57 |
| noonedeadpunk | oh, indeed we don't have it bumped.... | 13:58 |
| openstackgerrit | Dmitriy Rabotyagov (noonedeadpunk) proposed openstack/openstack-ansible master: Add ironic-inspector to the openstack_services https://review.opendev.org/746931 | 14:01 |
| guilhermesp | ^ ++ | 14:01 |
| guilhermesp | lets wait and hopefully we get it backported to ussuri, but btw, i have added locally those services above and re-ran ironic playbooks | 14:02 |
| guilhermesp | it seems we still get 10.3.0 | 14:02 |
| noonedeadpunk | guilhermesp: no, it's wrong | 14:04 |
| noonedeadpunk | actually not wrong, but won't help at the moment | 14:04 |
| noonedeadpunk | doing another bit to get it handled | 14:05 |
| guilhermesp | ++ noonedeadpunk | 14:05 |
| noonedeadpunk | we just had the same thing with masakari-monitors... | 14:06 |
| *** KeithMnemonic has joined #openstack-ansible | 14:09 | |
| openstackgerrit | Dmitriy Rabotyagov (noonedeadpunk) proposed openstack/openstack-ansible master: Add ironic-inspector to the openstack_services https://review.opendev.org/746931 | 14:10 |
| masterpe | jrosser: https://gist.github.com/8277eeddd10b1bfe240db134ee431fcc is the python_venv_build.log where I fist installed makari on a clean container, with the command os-masakari-install.yml without --limit (2020-08-19T13:30:03) and after that I run os-masakari-install.yml with limit(2020-08-19T13:36:43). At first it installs vine 5.0.0a1 and at the second run it installes 1.2.0. | 14:10 |
| openstackgerrit | Dmitriy Rabotyagov (noonedeadpunk) proposed openstack/openstack-ansible-os_ironic master: Add ironic-inspector to constraints https://review.opendev.org/746933 | 14:12 |
| noonedeadpunk | guilhermesp: I think this is also needed ^ | 14:12 |
| * noonedeadpunk needs to adjust git config.... | 14:13 | |
| jrosser | masterpe: so that you just pasted is only the log from the first run which gave 1.2.0? | 14:13 |
| masterpe | no both | 14:14 |
| masterpe | that file contains 40329 lines | 14:15 |
| masterpe | I see 2020-08-19T13:30:03 Collecting vine==5.0.0a1 (from amqp>=2.5.2->oslo.messaging>=5.29.0->masakari) | 14:17 |
| masterpe | does that means that it installs vine there it is required by amqp? | 14:17 |
| guilhermesp | yep noonedeadpunk that makes sense | 14:18 |
| guilhermesp | https://www.irccloud.com/pastebin/NCd0QqBk/ | 14:28 |
| guilhermesp | noonedeadpunk: sounds good | 14:28 |
| openstackgerrit | Dmitriy Rabotyagov (noonedeadpunk) proposed openstack/openstack-ansible stable/rocky: Simplify pre-gate tasks https://review.opendev.org/746938 | 14:34 |
| noonedeadpunk | looks really nice) | 14:34 |
| jrosser | masterpe: the important point is that the whole installation should be subject to the constraints file | 14:36 |
| jrosser | so if you are getting something different when running with --limit then somehow the constraints are not being applied | 14:37 |
| *** sshnaidm|afk is now known as sshnaidm | 14:42 | |
| kleini | Does OSA support the setup of a shared storage on compute nodes for ephemeral VM storage? I currently don't have a shared storage and used block migration between VMs in the past. With my OSA based deployment this does not work any more as live_migration_tunnelled = True is hardcoded in nova.conf. The commit message adding this says nothing, just restoring live_migration_uri, which got lost during remove of Nova | 14:48 |
| kleini | deprecated options. | 14:48 |
| *** cshen has quit IRC | 14:49 | |
| noonedeadpunk | kleini: yu can override any value in nova config | 14:50 |
| noonedeadpunk | https://opendev.org/openstack/openstack-ansible-os_nova/src/branch/master/defaults/main.yml#L490-L497 | 14:50 |
| noonedeadpunk | for this case I think it's nova_nova_conf_overrides | 14:50 |
| kleini | I know and used that for tenant isolation filter. I was just asking | 14:52 |
| kleini | Does OSA support setup of some kind of shared storage for compute nodes? | 14:52 |
| noonedeadpunk | tbh I'm not sure, we just providing overall default, which obviously won't fit everyone | 14:52 |
| noonedeadpunk | like nfs? | 14:53 |
| noonedeadpunk | but tbh I'd say that ceph is recommended option for the shared storage... | 14:53 |
| noonedeadpunk | and it's supported yes | 14:53 |
| kleini | like glusterfs, lustre, sheepdog | 14:55 |
| jrosser | nova docs describe how you could use nfs https://docs.openstack.org/nova/latest/admin/configuring-migrations.html#shared-storage | 14:56 |
| noonedeadpunk | Like we don't have playbooks for deploying that | 14:56 |
| kleini | ceph scales very good but IOPS go down compared to local storages due to network round trip | 14:56 |
| noonedeadpunk | but you obviously may use that and use overrides to configure nova | 14:57 |
| kleini | I am fine now with overriding the nova.conf. Was just curious, why tunnelled is added to the nova.conf.j2 template while documentation says it is incompatible with block migration. | 14:58 |
| noonedeadpunk | you mean that one? https://opendev.org/openstack/openstack-ansible-os_nova/commit/12e09a3402cb810c53188a94ad1c820086d8e302 | 15:00 |
| noonedeadpunk | I'd say it's smth that we need to look into as live_migration_uri at least is deprecated for a long time | 15:03 |
| jrosser | sort of - i think that still the patch for replacing it is not merged | 15:04 |
| noonedeadpunk | oh, we have one?:) | 15:04 |
| jrosser | oh no, on the nova side | 15:04 |
| noonedeadpunk | ah | 15:04 |
| noonedeadpunk | tbh I wasn't deep diving in the topic - current worked for me, so had no reasons to be bothered... | 15:05 |
| *** kberger_ has joined #openstack-ansible | 15:08 | |
| noonedeadpunk | I guess it was enabled to speedup live migrations as like enabling tls for live migrated instances adds really ton's of encryption overhead on busy instances? | 15:08 |
| noonedeadpunk | dunno, we can change that, but it needs testing... | 15:09 |
| noonedeadpunk | kleini: if you're ready with better solution with current state of nova, you can propose patch | 15:09 |
| *** KeithMnemonic has quit IRC | 15:10 | |
| noonedeadpunk | but I think we will need to configure CA and stuff... | 15:10 |
| *** spatel has joined #openstack-ansible | 15:13 | |
| *** chkumar|rover is now known as raukadah | 15:14 | |
| kleini | I plan to use nova_nova_conf_overrides and set live_migration_tunnelled = False. This worked very well for me in the past and max_concurrent_live_migrations is 1 per default, so there is not that much overhead as one CPU core is required. | 15:15 |
| jrosser | in the long term we need to switch it to TLS | 15:18 |
| noonedeadpunk | yeah | 15:18 |
| kleini | I don't really have an idea, what a good patch could be except removing live_migration_tunnelled option. | 15:18 |
| noonedeadpunk | I think that makes possible to use block migration?:) | 15:19 |
| noonedeadpunk | ie replaces live_migration_tunnelled with live_migration_with_native_tls ? | 15:19 |
| noonedeadpunk | and drops deprecated live_migration_uri | 15:19 |
| kleini | Yes, but I don't really understand what "native transport" is compared to tunnelled? | 15:20 |
| noonedeadpunk | iirc native is qemu one, and tunneled with libvirt? | 15:21 |
| kleini | in libvirtd.conf I already only see listen_tls = 1 and listen_tcp = 0. So transport should already be encrypted, no? | 15:21 |
| noonedeadpunk | yes, but for tunnelled only I guess? | 15:21 |
| noonedeadpunk | as for block migration there're other requirements ie https://docs.openstack.org/nova/ussuri/configuration/config.html#libvirt.live_migration_with_native_tls | 15:22 |
| *** cshen has joined #openstack-ansible | 15:23 | |
| jrosser | isnt it a bit contradictory | 15:25 |
| jrosser | well it's confusing about when the block stream works at all | 15:27 |
| jrosser | vs when it is/isnt encrypted | 15:27 |
| noonedeadpunk | oh, eventually yes, I think it can be unencrypted when both tunneled and live_migration_with_native_tls are et to 0 I guess | 15:28 |
| *** cshen has quit IRC | 15:28 | |
| openstackgerrit | zhoulinhui proposed openstack/openstack-ansible-os_nova master: libvirt: Drop support for Xen https://review.opendev.org/746960 | 15:29 |
| kleini | Now I am really confused. Live migration worked for me now with tunnelled = false. I looked at network traffic on mgmt network and I only see SSH traffic on the mgmt network between compute nodes. libvirtd does not listen on any port. How is the memory state transferred? | 15:34 |
| openstackgerrit | zhoulinhui proposed openstack/openstack-ansible-os_nova master: libvirt: Drop support for Xen https://review.opendev.org/746960 | 15:35 |
| openstackgerrit | Merged openstack/openstack-ansible stable/ussuri: Make upgrade jobs get role relevant versions https://review.opendev.org/746366 | 15:37 |
| openstackgerrit | Georgina Shippey proposed openstack/openstack-ansible-os_keystone stable/train: Add Paramaters to httpd.conf template https://review.opendev.org/746964 | 15:37 |
| kleini | nevermind I see now the native transfer with an unencrypted connection. | 15:38 |
| kleini | So, live_migration_with_native_tls would be the way to go with proper setup of CA, server and client certificates, keys and file permissions. | 15:39 |
| noonedeadpunk | yep | 15:40 |
| noonedeadpunk | And I think that' probably good way of changing defaults | 15:40 |
| noonedeadpunk | and at the same time I think it's wh this hasn't been changed yet) | 15:41 |
| noonedeadpunk | (needs research, time, etc) | 15:43 |
| kleini | one more on my contribution TODO list. a hard one as this requires to add the feature to setup CA, certificates, keys and so on | 15:43 |
| noonedeadpunk | I think jrosser did that for some other roles | 15:43 |
| noonedeadpunk | ie haproxy | 15:43 |
| *** gyee has joined #openstack-ansible | 15:44 | |
| kleini | will try to find that code and see, what I can learn from it | 15:44 |
| noonedeadpunk | probaby... we should create a separate role to configure SSLs as I think we have that in many placed duplicated | 15:44 |
| jrosser | i have a WIP patch which does a bunch of it, but never merged | 15:44 |
| noonedeadpunk | ;( | 15:44 |
| jrosser | yes exactly that | 15:44 |
| jrosser | we do it in haproxy, and i was trying to factor that out | 15:44 |
| jrosser | https://review.opendev.org/#/c/644555/ | 15:45 |
| jrosser | i think that the ansible modules may be significantly improved since i did that - seem to remember using a lot of shell | 15:45 |
| noonedeadpunk | last time I looked at them - they are not( | 15:45 |
| noonedeadpunk | maybe we really should create a separate repo for ssl configuration? | 15:46 |
| noonedeadpunk | in case we will have that not only haproxy | 15:46 |
| noonedeadpunk | we can use same set of certififcates ofc, but not sure how good that is | 15:47 |
| noonedeadpunk | esp when we can generate them.... | 15:47 |
| jrosser | yes, so it's either a role, or an include_tasks with a bunch of vars a bit like we do for service_setup | 15:49 |
| jrosser | not really sure what the best way is | 15:49 |
| noonedeadpunk | or that, yes. | 15:49 |
| jrosser | or even a python ansible module | 15:49 |
| noonedeadpunk | considering amount of shell.... | 15:50 |
| noonedeadpunk | we should also get used to collections and make our stuff as collections somehow... | 15:51 |
| noonedeadpunk | but that's completely another topic | 15:51 |
| noonedeadpunk | from what I see in https://review.opendev.org/#/c/644555/11 it's not just include_tasks, as it needs packages to be present and contains template | 15:57 |
| jrosser | most of that is to do with creating the CA | 16:04 |
| jrosser | so then theres some more work to do so that the "make a cert for service A on host B" is as simple/clean as possible | 16:04 |
| noonedeadpunk | oh, so you mean to create CA with ... openstack_hosts? and then just generate certs with that single play | 16:05 |
| noonedeadpunk | or module | 16:06 |
| jrosser | the CA is on the deploy host, so something needs to bootstrap that 'early' | 16:06 |
| noonedeadpunk | yeah, then module I think might be even better | 16:06 |
| jrosser | then as you go through the various plays $thing then needs to cut certs as they're needed, store them on the deploy host and copy into place | 16:06 |
| noonedeadpunk | yeah, ok, you're right, it may really stay as you've patched | 16:07 |
| jrosser | i think it's really that last thing about how to make them nicely in the rest of the plays that needs looking at | 16:07 |
| noonedeadpunk | we can totally make a module then | 16:08 |
| noonedeadpunk | as anyway we're doing everyting in the integrated repo | 16:08 |
| jrosser | thats why i wasnt sure if it was an include_tasks | 16:10 |
| noonedeadpunk | considering amount of bash there... | 16:11 |
| jrosser | it's possibly just two tasks, openssl_csr followed by either anther ansible module or command to create the cert, then copy | 16:12 |
| noonedeadpunk | maybe. anyway this topic needs more love) | 16:13 |
| jrosser | yeah :) | 16:13 |
| openstackgerrit | Charles Farquhar proposed openstack/openstack-ansible-os_neutron stable/rocky: Remove dragonflow ML2 driver https://review.opendev.org/745377 | 16:33 |
| openstackgerrit | Dmitriy Rabotyagov (noonedeadpunk) proposed openstack/openstack-ansible master: Add ironic-inspector to the openstack_services https://review.opendev.org/746931 | 16:45 |
| openstackgerrit | Dmitriy Rabotyagov (noonedeadpunk) proposed openstack/openstack-ansible master: Bump ceilometer SHA to ublock telemetry https://review.opendev.org/746679 | 16:55 |
| openstackgerrit | Dmitriy Rabotyagov (noonedeadpunk) proposed openstack/openstack-ansible-os_ceilometer master: Updated from OpenStack Ansible Tests https://review.opendev.org/745842 | 16:56 |
| *** watersj has quit IRC | 17:12 | |
| *** watersj has joined #openstack-ansible | 17:15 | |
| *** carlosmss has joined #openstack-ansible | 17:19 | |
| carlosmss | Hi guys, someone knows how to fix this follow issue: I have 3 hosts in group repo-infra-host, but the packages required by openstack services is installed only in 1 host. repo1/os-releases/19.1.2.dev2/ (OK), repo1/os-releases/19.1.2.dev2/ (empty), repo1/os-releases/19.1.2.dev2/ (empty) | 17:24 |
| *** cshen has joined #openstack-ansible | 17:24 | |
| jrosser | carlosmss: it should be running lysncd/rsync from the first repo server to the others | 17:25 |
| jrosser | on the one which is the source ([0] in the ansible group i think) its worth having a look at whats happening with lsyncd, possibly restart it | 17:25 |
| jrosser | if that all looks OK then maybe something fails with rsync, i think it has ssh keys setup | 17:26 |
| *** cshen has quit IRC | 17:29 | |
| carlosmss | Thanks jrosser, I will check if lysncd/rsync is running | 17:33 |
| carlosmss | jrosser I checked and the first repo-host is really the unique repo host with some data. But the lsyncd is not running on the repo host of index 0. When this tool is called by repo_container? There is some trick to run this lsync forcely? | 17:45 |
| *** mathlin has quit IRC | 17:50 | |
| *** cshen has joined #openstack-ansible | 17:50 | |
| *** spatel has quit IRC | 17:58 | |
| *** cshen has quit IRC | 18:00 | |
| carlosmss | jrosser I've found some sintax error calling the configuration of lsyncd in lua format | 18:01 |
| carlosmss | there is some quotes that isn't closed | 18:02 |
| openstackgerrit | Guilherme Steinmuller Pimentel proposed openstack/openstack-ansible-os_nova master: Use the utility host for db setup tasks https://review.opendev.org/747006 | 18:15 |
| openstackgerrit | Dmitriy Rabotyagov (noonedeadpunk) proposed openstack/openstack-ansible master: Decrease amount of jobs and update distros https://review.opendev.org/746881 | 18:27 |
| noonedeadpunk | guilhermesp: I was just going to make these patches for bind-to-mgmt :p So I can pick-up the rest of them tomorrow (so you don't have to patch _all_ roles) | 18:28 |
| noonedeadpunk | btw revote on https://review.opendev.org/#/c/746931/ needed) | 18:29 |
| guilhermesp | hahaha yeah, i just ended up with that issue deploying nova now... to be honest, for other services i didnt patch, as I was re-running osa so just did --skip common-db :P | 18:30 |
| guilhermesp | but yeah, i will try to push some other today | 18:30 |
| guilhermesp | to help | 18:30 |
| guilhermesp | others* | 18:30 |
| *** spatel has joined #openstack-ansible | 18:32 | |
| noonedeadpunk | you can omit it if you don't need that at the moment | 18:33 |
| noonedeadpunk | whatever works for you I mean:) | 18:33 |
| guilhermesp | noonedeadpunk: ++ | 18:33 |
| noonedeadpunk | we never reject help :p | 18:33 |
| carlosmss | Hi guys, about my reported errors on repo server sync, this issue has been fixed by commit c801daedaf on branch Stable/Stein | 18:41 |
| carlosmss | This commit fixes that missing quotes in template and was commited one month ago. Thanks for help jrosser. Have a nice day. | 18:42 |
| openstackgerrit | Merged openstack/openstack-ansible-os_designate stable/ussuri: Add CentOS-8 support https://review.opendev.org/746006 | 18:56 |
| masterpe | I have the issue that vine version 5.0.0a1 gets installed in the masakari container, on https://gist.github.com/mpiscaer/6231885d798129b5764d74b6d48f64bd I see that "Build wheels for the packages to be installed into the venv" is done on compute02 and not on a masakari container. Can this be the reason of the failure? | 19:09 |
| jrosser | carlosmss: exellent! good to hear it is fixed | 19:13 |
| jrosser | masterpe: the task you mention is delegated, thats shown with changed: [compute02 -> 172.19.0.88] | 19:16 |
| masterpe | yes | 19:16 |
| jrosser | so what might be happening is that variables are required that are defined for the masakari containers, but not defined correctly on the compute nodes | 19:18 |
| masterpe | but on compute you only want to install masakari-monitor and on the api you install the masakari-api, can this the reason that the right upper is set? | 19:18 |
| masterpe | but on compute you only want to install masakari-monitor and on the api you install the masakari-api, can this the reason that the right upper is not set? | 19:18 |
| masterpe | that the reason is that vine is to new? | 19:19 |
| masterpe | Sorry that I bother you all about this. But I want to unterstand it and find the reason. For me pip and wheel are new concepts. I',m more familiar with apt. | 19:21 |
| jrosser | masterpe: sadly i have never deployed masakari so i'm just having to find my way around it | 19:22 |
| jrosser | where is the playbook that deploys to the computes? | 19:22 |
| masterpe | The role https://opendev.org/openstack/openstack-ansible-os_masakari installes the masakari and masakari-monitor | 19:25 |
| masterpe | I also needed to use https://review.opendev.org/#/c/679487 and https://review.opendev.org/#/c/744673 to get this working. | 19:27 |
| masterpe | https://opendev.org/openstack/openstack-ansible/src/branch/master/playbooks/os-masakari-install.yml#L23 uses hosts masakari_all but that group contains two groups api and monitors. | 19:29 |
| masterpe | Monitors needs to get installed on the compute nodes and api on the controllers. | 19:29 |
| masterpe | The monitors is https://opendev.org/openstack/masakari-monitors and masakari-api is https://opendev.org/openstack/masakari | 19:30 |
| jrosser | masterpe: ok, and masakari_monitors contains the compute nodes? | 19:32 |
| masterpe | yes | 19:32 |
| masterpe | so two different applications and as I understand it it should be two wheels? | 19:32 |
| jrosser | its not really about the wheel building, more the dependancies that get pulled in | 19:44 |
| jrosser | so from your python_venv_build log we see that masakari needs oslo.messaging | 19:44 |
| jrosser | 2020-08-19T13:29:55 Collecting amqp>=2.5.2 (from oslo.messaging>=5.29.0->masakari) | 19:44 |
| jrosser | and then kind of surprisingly this then happens | 19:45 |
| jrosser | 2020-08-19T13:29:55 Using version 5.0.0a1 (newest of versions: 2.5.2, 2.6.0, 2.6.1, 5.0.0a1) | 19:45 |
| masterpe | yes so it looks like the upper for this is missing, is what I got told. | 19:46 |
| jrosser | so that actual issue is not vine, its the wrong version of amqp which then pulls in the wrong version of vine | 19:47 |
| jrosser | which we see here https://github.com/celery/py-amqp/blob/v5.0.0a1/requirements/default.txt | 19:47 |
| masterpe | but it is working fine if I use os-masakari-install.yml --limit masakari_api_container | 19:48 |
| jrosser | thats the symptom yes but i dont know why, we need to find out | 19:51 |
| jrosser | the thing to look at next is probably this https://github.com/openstack/ansible-role-python_venv_build/blob/master/tasks/python_venv_install.yml#L85-L98 | 19:51 |
| jrosser | when the python packages are installed into the venv a constraints file is generated which should set the upper bounds on all the things | 19:51 |
| jrosser | and i am guessing that the contents of that is somehow different for the two different behaviours you see | 19:52 |
| masterpe | ydx | 19:52 |
| masterpe | yes excectly | 19:53 |
| masterpe | and this is done on one of the compute nodes in my env. | 19:54 |
| jrosser | i dont see oslo.messaging in here https://github.com/openstack/masakari-monitors/blob/master/requirements.txt | 19:55 |
| jrosser | thats suspicious | 19:55 |
| masterpe | And both are different for the monitors and api | 19:55 |
| *** cshen has joined #openstack-ansible | 19:55 | |
| *** cshen has quit IRC | 20:00 | |
| masterpe | That oslo.messaging is missing isn't that a different issue? | 20:14 |
| jrosser | masterpe: sorry it’s late here | 20:25 |
| *** spatel has quit IRC | 20:25 | |
| jrosser | but if the contents of the requirements file is used to built the constraints for the venv then missing a requirement could be an issue | 20:25 |
| jrosser | because it works for when you limit to the api, and that has a requirement for oslo.messaging | 20:26 |
| jrosser | but on the compute when we see the venv build done, we see no requirement for oslo.messaging, so in turn its requirements may not be accounted for | 20:27 |
| jrosser | and that would be a reasonable explanation for why amqp and vine become unconstrained | 20:27 |
| masterpe | Ah | 20:30 |
| masterpe | It is also late here in the Netherlands. But tomorrow is another day. | 20:31 |
| openstackgerrit | Dmitriy Rabotyagov (noonedeadpunk) proposed openstack/openstack-ansible-os_ceilometer master: Disable ceilometer distro jobs https://review.opendev.org/747032 | 20:41 |
| openstackgerrit | Dmitriy Rabotyagov (noonedeadpunk) proposed openstack/openstack-ansible-os_ceilometer master: Updated from OpenStack Ansible Tests https://review.opendev.org/745842 | 20:50 |
| *** dmsimard7 has joined #openstack-ansible | 21:14 | |
| *** dmsimard has quit IRC | 21:15 | |
| *** dmsimard7 is now known as dmsimard | 21:15 | |
| *** tosky has quit IRC | 21:47 | |
| *** cshen has joined #openstack-ansible | 21:55 | |
| *** cshen has quit IRC | 22:00 | |
| *** spatel has joined #openstack-ansible | 23:24 | |
| openstackgerrit | Merged openstack/openstack-ansible-os_nova master: Use the utility host for db setup tasks https://review.opendev.org/747006 | 23:25 |
| *** spatel has quit IRC | 23:30 | |
| *** spatel has joined #openstack-ansible | 23:40 | |
| *** cshen has joined #openstack-ansible | 23:56 | |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!