| csmart | re hardening, are the containers actually running SSH though? | 00:05 |
|---|---|---|
| csmart | I'm showing my ignorance, but it seems a little strange to me to SSH directly into a container, i.e. they might have configs but they aren't used | 00:06 |
| *** macz_ has quit IRC | 00:09 | |
| *** gyee has quit IRC | 00:21 | |
| *** spatel has joined #openstack-ansible | 00:36 | |
| *** MickyMan77 has joined #openstack-ansible | 00:39 | |
| *** NewJorg has quit IRC | 00:41 | |
| *** rfolco has quit IRC | 00:42 | |
| *** NewJorg has joined #openstack-ansible | 00:42 | |
| *** MickyMan77 has quit IRC | 00:47 | |
| *** NobodyCam_ has joined #openstack-ansible | 00:52 | |
| *** NobodyCam has quit IRC | 00:52 | |
| *** NobodyCam_ is now known as NobodyCam | 00:52 | |
| *** mugsie has quit IRC | 01:00 | |
| *** mugsie has joined #openstack-ansible | 01:04 | |
| *** MickyMan77 has joined #openstack-ansible | 01:18 | |
| *** dave-mccowan has quit IRC | 01:20 | |
| *** cshen has joined #openstack-ansible | 01:28 | |
| *** cshen has quit IRC | 01:32 | |
| *** MickyMan77 has quit IRC | 01:53 | |
| *** MickyMan77 has joined #openstack-ansible | 02:49 | |
| *** spatel has quit IRC | 03:15 | |
| *** MickyMan77 has quit IRC | 03:23 | |
| *** cshen has joined #openstack-ansible | 03:28 | |
| *** cshen has quit IRC | 03:33 | |
| *** MickyMan77 has joined #openstack-ansible | 04:20 | |
| *** MickyMan77 has quit IRC | 04:55 | |
| *** cshen has joined #openstack-ansible | 05:28 | |
| *** evrardjp has quit IRC | 05:33 | |
| *** evrardjp has joined #openstack-ansible | 05:33 | |
| *** cshen has quit IRC | 05:33 | |
| *** MickyMan77 has joined #openstack-ansible | 05:50 | |
| *** MickyMan77 has quit IRC | 06:09 | |
| *** MickyMan77 has joined #openstack-ansible | 06:09 | |
| *** cshen has joined #openstack-ansible | 06:30 | |
| *** cshen has quit IRC | 06:35 | |
| *** recyclehero has joined #openstack-ansible | 06:40 | |
| *** miloa has joined #openstack-ansible | 06:43 | |
| openstackgerrit | amolkahat proposed openstack/openstack-ansible-os_tempest master: Migrate to content provider jobs/templates https://review.opendev.org/759287 | 07:20 |
| *** rpittau|afk is now known as rpittau | 07:42 | |
| noonedeadpunk | csmart: we don't ssh into containers. We ssh only to lxc_hosts and use lxc tooling to enter container rather than ssh | 07:44 |
| csmart | noonedeadpunk: yep that's what I thought đź‘Ť makes sense to me, ta | 07:45 |
| *** nurdie has quit IRC | 07:47 | |
| openstackgerrit | Dmitriy Rabotyagov (noonedeadpunk) proposed openstack/openstack-ansible master: Actually destroy container even if keeping data https://review.opendev.org/729533 | 07:48 |
| *** cshen has joined #openstack-ansible | 07:51 | |
| *** andrewbonney has joined #openstack-ansible | 08:10 | |
| *** jbadiapa has quit IRC | 08:28 | |
| pto | Why have you not chosen an pre-building approach, where the images are build and staged, then deployed to the hosts? | 08:30 |
| *** macz_ has joined #openstack-ansible | 08:33 | |
| *** tosky has joined #openstack-ansible | 08:36 | |
| *** macz_ has quit IRC | 08:37 | |
| *** jbadiapa has joined #openstack-ansible | 08:38 | |
| *** cshen has quit IRC | 08:50 | |
| *** cshen has joined #openstack-ansible | 08:50 | |
| cshen | noonedeadpunk: Hi, I found an etherpad link from you. https://etherpad.opendev.org/p/osa-rocky-bionic-upgrade Just want to know if this doc is still up2date? | 08:53 |
| noonedeadpunk | cshen: yes and no:) it can be used for some basic understanding. but thanks to ebbex we also have this https://docs.openstack.org/openstack-ansible/rocky/admin/upgrades/distribution-upgrades.html | 09:02 |
| cshen | noonedeadpunk: thank you very much! | 09:03 |
| noonedeadpunk | but I think that etherpad might be still helpful | 09:07 |
| kleini | would be the upgrade from bionic to focal similar? | 09:08 |
| ebbex | kleini: Most of the steps should be similar, yes. Problems may appear in handling packages to repo_server, and might need some modifications to https://review.opendev.org/#/c/714483/ | 09:12 |
| cshen | noonedeadpunk: yes, I read the page. It seems to mainly focus on control plane. Your etherpad link is also very useful. | 09:13 |
| kleini | ebbex, thanks. Unfortunately I have absolutely not idea about the repo server. Need to understand that first and then try to do the upgrade. | 09:16 |
| MickyMan77 | I have added a certificate for the setting haproxy_user_ssl_cert and it's wokring perfect when I access the Horizon gui via a url. But when I try to access the console on the instances it point me the the ip addess instead of the url that point to the external vip addess. | 09:19 |
| MickyMan77 | how can I change from ip addess to url ? | 09:20 |
| ebbex | MickyMan77: perhaps look at how nova_spice_html5proxy_base_uri resolves | 09:27 |
| MickyMan77 | https://computersweden.idg.se/2.2683/1.741771/lacka-gunnebo-it-attack | 09:29 |
| ebbex | looks like it uses external_lb_vip_address | 09:29 |
| *** pto has quit IRC | 09:35 | |
| *** pto has joined #openstack-ansible | 09:40 | |
| *** pto_ has joined #openstack-ansible | 09:41 | |
| *** pto__ has joined #openstack-ansible | 09:41 | |
| *** pto_ has quit IRC | 09:45 | |
| *** pto__ has quit IRC | 09:53 | |
| *** pto has joined #openstack-ansible | 09:55 | |
| openstackgerrit | Gaudenz Steinlin proposed openstack/openstack-ansible master: Use TCP mode for console if SSL is configured https://review.opendev.org/574153 | 10:03 |
| *** spatel has joined #openstack-ansible | 10:05 | |
| *** spatel has quit IRC | 10:10 | |
| *** rfolco has joined #openstack-ansible | 10:31 | |
| *** pto has quit IRC | 10:39 | |
| *** pto_ has joined #openstack-ansible | 10:40 | |
| *** pto_ has quit IRC | 10:51 | |
| *** gshippey has joined #openstack-ansible | 10:55 | |
| *** macz_ has joined #openstack-ansible | 11:25 | |
| *** macz_ has quit IRC | 11:30 | |
| *** pto has joined #openstack-ansible | 11:31 | |
| *** pto has quit IRC | 11:36 | |
| *** pto_ has joined #openstack-ansible | 11:36 | |
| *** pto has joined #openstack-ansible | 11:38 | |
| openstackgerrit | Dmitriy Rabotyagov (noonedeadpunk) proposed openstack/openstack-ansible-repo_server master: Allow remote detection of repo sync status https://review.opendev.org/660788 | 11:40 |
| *** pto_ has quit IRC | 11:42 | |
| *** rh-jelabarre has joined #openstack-ansible | 11:53 | |
| *** cshen_ has joined #openstack-ansible | 11:57 | |
| *** cshen has quit IRC | 12:00 | |
| pto | I dont quite understand the purpose of nova_libvirt_images_rbd_pool? Is this a flag to enable the Nova host to support ceph? Shoudl it be the same as the volume_backend_name defined in cinder_backends? | 12:09 |
| noonedeadpunk | pto: it's the storage for nova ephemerl drives, config drives or swap ones | 12:10 |
| pto | Aah... that makes sense. So it should be a sepperate pool from where the volumes are stored? | 12:11 |
| pto | Is there a supported way to use an external ceph cluster (not provisioned from the openstack ansible)? | 12:13 |
| guilhermesp | hey there, has anyone experienced octavia in a env with lxc + neutron-linuxbridge? | 12:18 |
| admin0 | pto, external ceph is supported | 12:25 |
| pto | admin0: Cool. Do you have a hit how? Currently I have not defined any ceph nodes in openstack_user_config.yml, which makes the play skip the ceph client part | 12:28 |
| admin0 | you have to give the fsid, pools and mons in user_variables | 12:28 |
| pto | I have already done that and the play completes, but it does not create auth and pools. Is that working as intended? | 12:30 |
| admin0 | it will not | 12:30 |
| admin0 | you have to create the auths and pools yourself | 12:30 |
| admin0 | that is the whole idea isn't it | 12:30 |
| admin0 | you create the pools and auths | 12:30 |
| admin0 | and then it will just copy the files for it to work | 12:30 |
| admin0 | but skip touching ceph altogether | 12:31 |
| admin0 | https://gist.github.com/a1git/1274f8593bf8257644d48579f1519099 | 12:31 |
| admin0 | you have to create the pools and auths beforehand | 12:31 |
| pto | admin0: thanks. I see the ceph_client role is skipped when: ceph_mons | list | length == 0 | 12:33 |
| pto | admin0: Just wondering, if the mons should be defined in the group ceph_mons? | 12:34 |
| pto | admin0: Or will that make the play configure the existing mons? | 12:34 |
| admin0 | the mons must be defined, so that it can ssh to the mons and copy the ceph configs | 12:34 |
| admin0 | it will not create the mons for you .. just ssh and download the configs from it | 12:35 |
| pto | admin0: So, should i define ceph-mon_hosts to the existing mons in the openstack_user_variables.yml? | 12:36 |
| admin0 | i gave you a paste ^^ | 12:36 |
| pto | admin0: sure, the user_variables.yml parts. Not the openstack_user_config.yml | 12:38 |
| admin0 | user config will have nothing that says ceph | 12:39 |
| admin0 | no mention of it | 12:39 |
| jrosser | guilhermesp: we use octavia like that here, lxc + linuxbridge | 12:40 |
| admin0 | cinder will use this driver: cinder.volume.drivers.rbd.RBDDriver | 12:40 |
| admin0 | and then you will use the username and pool name | 12:40 |
| pto | The official docs says: https://docs.openstack.org/openstack-ansible-ceph_client/latest/config-from-file.html very similar to your gist, but if you define ceph_keyrings_dir: it should suck up the keys from there, but the play is skipped until you define ceph-mon_hosts. Im confused? | 12:40 |
| jrosser | pto: theres a subtely between the ceph mons ansible group and the list you can provide ceph_mons | 12:41 |
| jrosser | one is an ansible group that will create mons, the other is a list you can make that says "my externally provisioned mons are here" | 12:41 |
| pto | I see... Thanks allot! I will give it a test drive right away | 12:42 |
| jrosser | they are named very similar so in the code, maybe _ vs - in the name so needs careful reading | 12:42 |
| *** spatel has joined #openstack-ansible | 12:57 | |
| pto | Thanks for the clarification. With the attach gist, will openstack obtain the keys from the mons directly? | 13:02 |
| admin0 | it will .. if the deploy can ssh to it | 13:03 |
| jrosser | pto: is that OK that the deploy host can ssh to the mons? you can give all the keys manually if that is not possible | 13:05 |
| pto | jrosser: That was also my understanding, but for some reason, the pools were not created on the ceph cluster. It might be due to the unsupported --cluster parameter. I have tried to fix it with symlinks so far | 13:08 |
| jrosser | pto: from memory it is ceph-ansible (for example) that would create the pools | 13:10 |
| jrosser | so if you have an external cluster then i think it would be an action for the cluster admin to make the pools and keys | 13:10 |
| pto | Thank you all for helping. You are awsome! | 13:12 |
| pto | Is the manila project a mandatory part of Ussuri? It seems impossible to disable it | 13:15 |
| pto | and the installation is broken with default config (only config is manila-infra_hosts & manila-data_hosts set to infra hosts). The installation breaks at Copy manila configs with "msg": "'null' is undefined" | 13:18 |
| admin0 | projects will not be added unless you defined it in the configs | 13:19 |
| admin0 | maybe you had older artifacts ? | 13:19 |
| *** hindret has joined #openstack-ansible | 13:22 | |
| noonedeadpunk | needed another vote for https://review.opendev.org/#/c/759308/3 | 13:23 |
| *** mmethot has joined #openstack-ansible | 13:23 | |
| pto | admin0: Nope. Its a fresh install and fresh config | 13:24 |
| admin0 | then you can redo again :D | 13:24 |
| admin0 | backup the /etc/openstack_deploy folder and try again | 13:25 |
| jrosser | pto: when you do your own deploy (i.e not with an AIO) you will get what you define in /etc/openstack_deploy/openstack_user_config.yml | 13:28 |
| jrosser | there is no default for that and you can opt in/out of different services exactly as you wish | 13:28 |
| jrosser | there are a bunch of examples here https://github.com/openstack/openstack-ansible/tree/master/etc/openstack_deploy | 13:29 |
| jrosser | but really they are kind of just inspiration / hints rather than something you must take completely | 13:29 |
| pto | jrosser: I think there is a bug in the os_manila role then. It were partly deployed without the openstac_user_variables.yml containing the defintions. Im gonna redeploy later this week, and then I will verify if its a bug and report it, if so, | 13:32 |
| jrosser | there may be additional config required for manila | 13:33 |
| jrosser | quite a good way to find out about that is to look at the CI jobs, for example OSA manila is here https://review.opendev.org/#/q/project:openstack/openstack-ansible-os_manila | 13:33 |
| jrosser | you would be able to run up a local AIO with manila as an exact copy of those CI runs | 13:34 |
| pto | Cool! I will have a look at it | 13:34 |
| *** cshen has joined #openstack-ansible | 13:35 | |
| noonedeadpunk | well yeah, we even see issues with manila with lxc deployment, so some work probably needs to be done. In CI we run with ceph-nfs backend (with ganesha) | 13:35 |
| *** cshen_ has quit IRC | 13:35 | |
| noonedeadpunk | but generic drivers should also work | 13:35 |
| jrosser | you can look in the stuff that sets up the CI job to see the settings we use https://github.com/openstack/openstack-ansible/blob/master/tests/roles/bootstrap-host/templates/user_variables_manila.yml.j2 | 13:35 |
| pto | jrosser: I finally got the federated identity working on ussuri. Its far from hitless, but it seems to work. The major "but" is that the setup of federated domain cannot be done before the keystone-manage db_sync --contracts has been run | 13:35 |
| jrosser | pto: that is really interesting | 13:36 |
| pto | jrosser: When keystone comes up initially, there are multiple SQL triggers which protects the db until it has been patched with the contracts | 13:36 |
| jrosser | i do not really know enough about how the db migrations are done to say if it is a bug in keystone or not | 13:36 |
| jrosser | but what you describe does make sense, in deployments here we have always done a simple first pass without federation | 13:37 |
| jrosser | then enabled federation later once we were happy the deployment was good | 13:37 |
| *** nurdie has joined #openstack-ansible | 13:38 | |
| pto | jrosser: I dont know why the actual db_migrates is executed last? https://github.com/openstack/openstack-ansible/blob/47e5a90a7fcc78adc44bbd0803e0faabb56197b6/playbooks/os-keystone-install.yml#L135 I would think it should be run as one of the first things in the actual os_keystone role? | 13:38 |
| jrosser | noonedeadpunk: ^ i'm not sure what we should do about that, part of the db migrations is in post_tasks in the os_keystone playbook | 13:39 |
| jrosser | feels kind of wierd | 13:39 |
| jrosser | oh actually it's a second play entirely, there must be a really specific reason for that | 13:39 |
| noonedeadpunk | well it was added during upgrades | 13:40 |
| pto | jrosser: I have no clue why is has been put so late, but the play https://github.com/openstack/openstack-ansible-os_keystone/blob/bc3f922b903efeeb99164010bfdea744f3d3de6b/tasks/main.yml#L230 will fail if the db has not been migrated before | 13:41 |
| noonedeadpunk | And I think it's run after first host is installed or smth like that | 13:41 |
| jrosser | this is why https://github.com/openstack/openstack-ansible/commit/ecf32e20784d733fa89e1b0a392fd4b565ca4f41 | 13:41 |
| pto | I actually discovred the problem in the queens release and the same workaround worked (commenting the part out, and finalize the keystone install, then comment it in and run it again) | 13:42 |
| jrosser | i think that the safest change would be to remove the federation setup completely out of main.yml in os_keystone | 13:43 |
| jrosser | then we could call the role again right at the end of os_keystone with tasks_from: federation_setup.yml | 13:43 |
| pto | Alternative, the flag need_db_contract = False in /etc/ansible/facts.d/openstack_ansible.fact could be checked and the task skipped | 13:45 |
| pto | But anyway it should be moved to later in the play | 13:45 |
| jrosser | well it's in a role, thats the trouble | 13:48 |
| jrosser | not the play | 13:48 |
| *** dave-mccowan has joined #openstack-ansible | 13:48 | |
| openstackgerrit | James Denton proposed openstack/openstack-ansible-os_neutron master: Implement uWSGI for neutron-api https://review.opendev.org/486156 | 14:01 |
| openstackgerrit | Merged openstack/openstack-ansible-lxc_hosts master: Determine latest base image available https://review.opendev.org/759229 | 14:05 |
| *** sshnaidm|rover has quit IRC | 14:05 | |
| *** sshnaidm has joined #openstack-ansible | 14:05 | |
| *** sshnaidm is now known as sshnaidm|rover | 14:06 | |
| openstackgerrit | James Denton proposed openstack/ansible-role-systemd_networkd master: Add GPG Key for EPEL8 Repo https://review.opendev.org/759145 | 14:08 |
| openstackgerrit | Dmitriy Rabotyagov (noonedeadpunk) proposed openstack/openstack-ansible master: Fix upgrade jobs for bind-to-mgmt https://review.opendev.org/758461 | 14:09 |
| openstackgerrit | Merged openstack/openstack-ansible-tests stable/stein: Pin virtualenv<20 for python2 functional tests https://review.opendev.org/759308 | 14:17 |
| openstackgerrit | Dmitriy Rabotyagov (noonedeadpunk) proposed openstack/openstack-ansible-os_adjutant master: Make role fit to the OSA standards https://review.opendev.org/756313 | 14:20 |
| *** cshen has quit IRC | 14:20 | |
| noonedeadpunk | jrosser: did we want to bump amount of threads in this patch? https://review.opendev.org/#/c/705680/2/defaults/main.yml | 14:37 |
| noonedeadpunk | I think just number of processes? | 14:38 |
| noonedeadpunk | glance_api_threads just confusing dublicate name of glance_api_workers | 14:38 |
| *** macz_ has joined #openstack-ansible | 14:49 | |
| openstackgerrit | James Denton proposed openstack/openstack-ansible-os_neutron master: Clean up Neutron metering configuration https://review.opendev.org/759882 | 14:53 |
| openstackgerrit | James Denton proposed openstack/openstack-ansible-os_neutron master: Clean up Neutron metering configuration https://review.opendev.org/759882 | 14:54 |
| *** macz_ has quit IRC | 14:54 | |
| openstackgerrit | Dmitriy Rabotyagov (noonedeadpunk) proposed openstack/openstack-ansible-os_aodh master: Reduce number of processes on small systems https://review.opendev.org/759883 | 14:55 |
| jrosser | noonedeadpunk: oh yes that is really confusing | 15:01 |
| jrosser | extra variable for really no purpose | 15:02 |
| *** macz_ has joined #openstack-ansible | 15:10 | |
| *** macz_ has joined #openstack-ansible | 15:10 | |
| *** yann-kaelig has joined #openstack-ansible | 15:31 | |
| *** gyee has joined #openstack-ansible | 15:34 | |
| *** miloa has quit IRC | 15:41 | |
| admin0 | TASK [lxc_hosts : Ensure image has been pre-staged] -- this tries to access http://cdimage.ubuntu.com/ubuntu-base/releases/18.04/release/ubuntu-base-18.04.3-base-amd64.tar.gz which does not exist anymore | 15:55 |
| kleini | admin0, https://review.opendev.org/#/c/759229 | 15:57 |
| noonedeadpunk | admin0: http://lists.openstack.org/pipermail/openstack-discuss/2020-October/018280.html | 15:58 |
| noonedeadpunk | but yeah, this patch works:) | 15:59 |
| openstackgerrit | Dmitriy Rabotyagov (noonedeadpunk) proposed openstack/openstack-ansible-os_barbican master: Reduce number of processes on small systems https://review.opendev.org/759895 | 16:03 |
| *** tosky_ has joined #openstack-ansible | 16:07 | |
| gshippey | @noonedeadpunk are we doing bug triage in the meeting now, or at following the PTG schedule later in the day? | 16:07 |
| *** tosky has quit IRC | 16:08 | |
| noonedeadpunk | gshippey: I think we agreed to skip PTG | 16:08 |
| noonedeadpunk | but yes, I think we need to discuss some topics regarding bugs as well | 16:10 |
| openstackgerrit | James Denton proposed openstack/openstack-ansible-os_neutron master: Implement uWSGI for neutron-api https://review.opendev.org/486156 | 16:10 |
| openstackgerrit | Jonathan Rosser proposed openstack/openstack-ansible-lxc_hosts stable/ussuri: Determine latest base image available https://review.opendev.org/759900 | 16:12 |
| jrosser | oh, hmm | 16:12 |
| noonedeadpunk | https://review.opendev.org/#/c/759298/ ? | 16:13 |
| jrosser | yeah, just realised | 16:13 |
| noonedeadpunk | I've placed for all branches at once | 16:13 |
| noonedeadpunk | just for stein there's extra dependency | 16:13 |
| jrosser | yeah makes sense | 16:13 |
| noonedeadpunk | as we can't backport renos :) and different set of os anyway | 16:14 |
| jrosser | noonedeadpunk: is it meeting time? | 16:22 |
| noonedeadpunk | mmmm | 16:25 |
| noonedeadpunk | summer saving time damn it | 16:26 |
| noonedeadpunk | #startmeeting openstack_ansible_meeting | 16:26 |
| openstack | Meeting started Tue Oct 27 16:26:22 2020 UTC and is due to finish in 60 minutes. The chair is noonedeadpunk. Information about MeetBot at http://wiki.debian.org/MeetBot. | 16:26 |
| openstack | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 16:26 |
| *** openstack changes topic to " (Meeting topic: openstack_ansible_meeting)" | 16:26 | |
| openstack | The meeting name has been set to 'openstack_ansible_meeting' | 16:26 |
| noonedeadpunk | #topic office hours | 16:27 |
| *** openstack changes topic to "office hours (Meeting topic: openstack_ansible_meeting)" | 16:27 | |
| noonedeadpunk | Well I realized we have some topics to discuss after PTG | 16:27 |
| noonedeadpunk | first of all, I clean forgot to raise topic with our bugs | 16:28 |
| noonedeadpunk | I think we need to somehow cleanup current bug reports, as there are tons of obsolete ones or issued for non supported releases | 16:29 |
| jrosser | were do we say non supported is? | 16:30 |
| noonedeadpunk | I was thinking about ones in EM | 16:30 |
| noonedeadpunk | But we don't say this directly at the moment I think | 16:31 |
| noonedeadpunk | but we can close them conditionally when we obviously see this is should not be relevant anymore | 16:31 |
| noonedeadpunk | like https://bugs.launchpad.net/openstack-ansible/+bug/1736726 | 16:33 |
| openstack | Launchpad bug 1736726 in openstack-ansible "Pike update causes parallel restart of Galera containers" [High,Confirmed] - Assigned to Jesse Pretorius (jesse-pretorius) | 16:33 |
| noonedeadpunk | I never faced this issue tbh | 16:33 |
| noonedeadpunk | But it seemed relevant these days | 16:33 |
| noonedeadpunk | or should we try to solve https://bugs.launchpad.net/openstack-ansible/+bug/1778663 ? | 16:33 |
| openstack | Launchpad bug 1778663 in openstack-ansible "Pike→Queens upgrade fails on final step in run_upgrade.sh (running haproxy-install.yml)" [High,In progress] - Assigned to Antony Messerli (antonym) | 16:33 |
| jrosser | i guess relatedly we should move one/some more releases to EM | 16:33 |
| noonedeadpunk | well yes, it's stein... | 16:34 |
| noonedeadpunk | I want to do last point release with fixed images and move to stable/stein branch afterwards | 16:34 |
| noonedeadpunk | like we did for rocky | 16:35 |
| noonedeadpunk | So ideas what should we do with pretty old bugs? Should we try to resolve them? Or close with saying that in case it happens for currently supported release please re-submit? | 16:38 |
| noonedeadpunk | As I really want to start keeping track on our bugtracker | 16:38 |
| jrosser | given we can only really actively support the later branches i would be closing the bugs where possible | 16:40 |
| gshippey | https://bugs.launchpad.net/openstack-ansible/+bug/1744014 - have to still fix up my patch but hopefully we can get some movement on this one soon. I think we should cull the older bugs, especially if they're minor and ask for resubmission's where necessary. https://bugs.launchpad.net/openstack-ansible/+bugs?orderby=-id&memo=&start=300&direction=backwards <- bugs listed oldest to newest, might be useful | 16:40 |
| openstack | Launchpad bug 1744014 in openstack-ansible "[Docs] Expected Backups in OpenStack-Ansible" [Medium,Confirmed] | 16:40 |
| *** SecOpsNinja has joined #openstack-ansible | 16:41 | |
| noonedeadpunk | yeah, it is:) | 16:42 |
| jrosser | some of this is really easy, its for deprecated things | 16:43 |
| Adri2000 | I wouldn't be shocked if older bugs, especially if no one can reproduce them on recent releases, were closed | 16:43 |
| Adri2000 | (even my bugs :)) | 16:43 |
| noonedeadpunk | #agreed to close old bugs for non supported / EM releases with asking to re-create in case still relevant | 16:44 |
| noonedeadpunk | I will try to go through them | 16:44 |
| *** cshen has joined #openstack-ansible | 16:45 | |
| noonedeadpunk | Regarding bug https://bugs.launchpad.net/openstack-ansible/+bug/1901619 | 16:46 |
| openstack | Launchpad bug 1901619 in openstack-ansible "Ansble-hardening role is not applied to containers" [Undecided,New] | 16:46 |
| noonedeadpunk | I'd say it's invalid? | 16:46 |
| noonedeadpunk | oh, wait | 16:46 |
| SecOpsNinja | hi everyone. im having a strange error iin nova api regargind oslo.messaging._drivers.impl_rabbit ... Connection failed: [Errno 113] EHOSTUNREACH (retrying in 2.0 seconds): OSError: [Errno 113] EHOSTUNREACH but i have confirmed that rabbitmq cluster is runing ok with 3 runing nodes and all the porst configured in /etc/nova/nova.conf are accesable using telnet test. what could be the cause o | 16:46 |
| SecOpsNinja | f this failure? | 16:46 |
| noonedeadpunk | let's first decide if we want to return recent bug triage?:) | 16:46 |
| noonedeadpunk | I think that might be pretty useful to get other team opinion on bugs | 16:48 |
| noonedeadpunk | and maybe will bring more involvment into meetings (hopefully but unlikely) | 16:48 |
| noonedeadpunk | As I feel that doing triage was a good thing back then | 16:49 |
| jrosser | i think getting more folk involved is good, particularly anyone using older releases | 16:49 |
| *** cshen has quit IRC | 16:49 | |
| gshippey | đź‘Ť | 16:49 |
| openstackgerrit | Merged openstack/openstack-ansible-lxc_hosts stable/ussuri: Determine latest base image available https://review.opendev.org/759298 | 16:50 |
| jrosser | mine tend to be N-1 age, others may have a different perspective | 16:50 |
| jrosser | ebbex: is this something you can help with? | 16:50 |
| noonedeadpunk | I think we here do N-1.5 or smth like this:) So not using every release but jumping through one, so doing upgrades on yearly basis | 16:51 |
| noonedeadpunk | #agreed doing bug triage during meetings | 16:54 |
| noonedeadpunk | so, https://bugs.launchpad.net/openstack-ansible/+bug/1901619 :) | 16:54 |
| openstack | Launchpad bug 1901619 in openstack-ansible "Ansble-hardening role is not applied to containers" [Undecided,New] | 16:54 |
| noonedeadpunk | anyone thinks we should run hardening against containers as well? | 16:55 |
| gshippey | perhaps from a technical perspective, but doesnt the hardening role take ages to run | 16:56 |
| noonedeadpunk | I don't think it's pretty useful there as we're using minimal images, time is taken from host and containers shouldn't be directly accesible | 16:56 |
| noonedeadpunk | gshippey: well it's pretty fast | 16:56 |
| *** tosky_ is now known as tosky | 16:56 | |
| gshippey | don't know why i feel it always takes an age - ignore me! | 16:56 |
| noonedeadpunk | takes 2 mins according to ara:) | 16:57 |
| noonedeadpunk | https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_da2/759229/4/check/openstack-ansible-deploy-infra_lxc-centos-8/da27ca7/logs/ara-report/playbooks/2.html | 16:57 |
| noonedeadpunk | I think I know why - it has so many tasks.... | 16:58 |
| noonedeadpunk | so easy thing we can do if we want hardening to run against containers - is just change the order of execution in setup-hosts.yml | 16:58 |
| noonedeadpunk | actually, hosts set is defined with a variable | 16:59 |
| openstackgerrit | Merged openstack/openstack-ansible-lxc_hosts stable/train: Determine latest base image available https://review.opendev.org/759299 | 16:59 |
| noonedeadpunk | so see no reason in not changing the order and allowing ppl to decide if they want it or not | 16:59 |
| jrosser | they containers still do need some ssh | 17:01 |
| jrosser | theres some rsync and things isnt there? and keystone key distribution | 17:02 |
| jrosser | and rotation | 17:02 |
| openstackgerrit | Dmitriy Rabotyagov (noonedeadpunk) proposed openstack/openstack-ansible master: Run hardening after container deployment https://review.opendev.org/759907 | 17:03 |
| noonedeadpunk | well yes, I saw ssh running | 17:03 |
| noonedeadpunk | But I thought that it's for repo server only | 17:04 |
| *** odyssey4me is now known as odyssey4me|PTO | 17:04 | |
| noonedeadpunk | well we've returned it to the list of the packages but I forgot what removal of openssh breaks except repo | 17:05 |
| noonedeadpunk | so should we try to run hardening against all hosts by default | 17:06 |
| noonedeadpunk | ? | 17:06 |
| *** aedc_ has joined #openstack-ansible | 17:10 | |
| *** aedc has quit IRC | 17:11 | |
| jrosser | sorry just dealing with other things here | 17:11 |
| noonedeadpunk | let me end meeting then and we can continue later:) | 17:15 |
| noonedeadpunk | #endmeeting | 17:16 |
| *** openstack changes topic to "Launchpad: https://launchpad.net/openstack-ansible || Weekly Meetings: https://wiki.openstack.org/wiki/Meetings/openstack-ansible || Review Dashboard: http://bit.ly/osa-review-board-v2" | 17:16 | |
| openstack | Meeting ended Tue Oct 27 17:16:03 2020 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 17:16 |
| openstack | Minutes: http://eavesdrop.openstack.org/meetings/openstack_ansible_meeting/2020/openstack_ansible_meeting.2020-10-27-16.26.html | 17:16 |
| openstack | Minutes (text): http://eavesdrop.openstack.org/meetings/openstack_ansible_meeting/2020/openstack_ansible_meeting.2020-10-27-16.26.txt | 17:16 |
| openstack | Log: http://eavesdrop.openstack.org/meetings/openstack_ansible_meeting/2020/openstack_ansible_meeting.2020-10-27-16.26.log.html | 17:16 |
| *** MickyMan77 has quit IRC | 17:16 | |
| openstackgerrit | James Denton proposed openstack/openstack-ansible-os_neutron master: Clean up Neutron metering configuration https://review.opendev.org/759882 | 17:16 |
| *** rpittau is now known as rpittau|afk | 17:34 | |
| openstackgerrit | Dmitriy Rabotyagov (noonedeadpunk) proposed openstack/openstack-ansible-os_neutron master: Switch functional tests to focal https://review.opendev.org/759914 | 17:43 |
| openstackgerrit | Dmitriy Rabotyagov (noonedeadpunk) proposed openstack/openstack-ansible-os_cinder master: Reduce number of processes on small systems https://review.opendev.org/759916 | 17:50 |
| *** MickyMan77 has joined #openstack-ansible | 18:01 | |
| *** MickyMan77 has quit IRC | 18:10 | |
| *** ThiagoCMC has quit IRC | 18:17 | |
| *** andrewbonney has quit IRC | 18:21 | |
| *** cshen has joined #openstack-ansible | 18:25 | |
| openstackgerrit | Dmitriy Rabotyagov (noonedeadpunk) proposed openstack/openstack-ansible master: Bump SHAs for master https://review.opendev.org/755973 | 19:04 |
| openstackgerrit | Dmitriy Rabotyagov (noonedeadpunk) proposed openstack/openstack-ansible master: Added Openstack Adjutant role deployment https://review.opendev.org/756310 | 19:04 |
| *** MickyMan77 has joined #openstack-ansible | 19:06 | |
| *** itsjg has joined #openstack-ansible | 19:07 | |
| SecOpsNinja | is there any way to check what ERROR oslo.messaging._drivers.impl_rabbit [req-6f0baa57-aa50-4652-9539-a6ed93f25bad - - - - -] is causing this? | 19:09 |
| itsjg | SecOpsNinja It's strange, I see this message on both my controllers and compute nodes every so often as well. However I always confirm my rabbitmq cluster is healthy and I don't have network connectivity issues. I've never seen it directly impact services if the message is occasional. If you're seeing it non-stop and your noticing services crashing that's probably an indication of greater rabbitMQ issue. Does it appear to be impacting services? | 19:14 |
| *** MickyMan77 has quit IRC | 19:15 | |
| SecOpsNinja | yep im trying to start a vm and doesn t work it gets stuck. but all services say they are up and runing | 19:15 |
| SecOpsNinja | even when going to rabbitmq cluster it says is has all 3 nodes | 19:15 |
| SecOpsNinja | and from nova containers where im seeing does logs i can ping each rabbitmq cluster and cluseck that is ports are open | 19:16 |
| SecOpsNinja | nova placement is workign correctly | 19:16 |
| SecOpsNinja | but i can put any vm up and runing.... | 19:16 |
| SecOpsNinja | *can't | 19:17 |
| itsjg | Ahh... I believe I ran into this once before and had to restart all my RabbitMQ containers. Couldn't figure out the root cause, though. If you go into the rabbitmq container and check in /var/log/rabbitmq, do you see any hints? | 19:17 |
| admin0 | is there a known issue in creating galera users ? | 19:18 |
| openstackgerrit | Dmitriy Rabotyagov (noonedeadpunk) proposed openstack/openstack-ansible-os_tempest stable/train: Bump magnum tempest plugin https://review.opendev.org/759932 | 19:18 |
| noonedeadpunk | admin0: not I'm aware about | 19:18 |
| openstackgerrit | Dmitriy Rabotyagov (noonedeadpunk) proposed openstack/openstack-ansible-os_magnum stable/train: Add deployment of keystone_auth_default_policy https://review.opendev.org/759472 | 19:19 |
| SecOpsNinja | itsjg, i have restart all containers and changed keepalive to confirm that all control panel where restarted | 19:19 |
| SecOpsNinja | in /var/log/rabbitmq/rabbit\@*.log i have information that various machines connected to rabbitmq | 19:20 |
| noonedeadpunk | SecOpsNinja: what release is it? | 19:20 |
| SecOpsNinja | ussuri i believe | 19:20 |
| admin0 | The PyMySQL (Python 2.7 and Python 3.X) or MySQL-python (Python 2.X) module is required. | 19:21 |
| admin0 | this is 20.x branch on 18.04 | 19:21 |
| noonedeadpunk | SecOpsNinja: and how instance creation fails? | 19:22 |
| noonedeadpunk | I mean what error does it gives? | 19:22 |
| noonedeadpunk | As porbably rabbit may be not directy related here | 19:22 |
| noonedeadpunk | *probably | 19:23 |
| noonedeadpunk | admin0: hm, and you don't have any overrides of *_db_setup_host or maybe you've pulled some master role? | 19:26 |
| admin0 | its on 20.0.4 branch | 19:26 |
| SecOpsNinja | noonedeadpunk, the problem was strating vms is being stuck in starting phase. now i was able to start one but console doesn't work i was able to confirm its runign in one compute host with virsh list. but if i try to create a new instance from scratch it stays in building forever. i try to create a new volume (based on a image at gives erro very quickly | 19:26 |
| noonedeadpunk | well it feels like rabbit indeed | 19:27 |
| noonedeadpunk | what I always recommend with rabbit - run rabbitmq-install.yml -e rabbitmq_upgrade=true | 19:28 |
| noonedeadpunk | this re-creates cluster and queues | 19:29 |
| noonedeadpunk | at least doesn't makes things worse | 19:29 |
| SecOpsNinja | ok i will trty that | 19:29 |
| SecOpsNinja | i think its a rabbitmq problem but it doesn seem to be a connection but probably queues | 19:29 |
| noonedeadpunk | admin0: for 20.0.4 we usually delegate to galera_all[0] node. and use system pip there | 19:31 |
| noonedeadpunk | *system python | 19:31 |
| noonedeadpunk | but I'd expect it to have PyMySQL module installed | 19:31 |
| noonedeadpunk | the only thing that may go wrong, is if ansible used py2 but now is using py3. so it has installed dependency for py2 but now tries to use another binary which does not have it | 19:32 |
| *** mmethot has quit IRC | 19:33 | |
| noonedeadpunk | the most safe thing would be just install PyMySQL on that container manually, or you can try re-running galera_client role | 19:33 |
| noonedeadpunk | hm, looking through role I can't find where we did install PyMySQL there... | 19:35 |
| jrosser | admin0: it would be useful to see a paste of the error and task output, there could be something we fixed on master and did not backport that far | 19:35 |
| SecOpsNinja | noonedeadpunk, is there any list i can check regarding what queues i have available in cluster ? because when i do rabbitmqctl list_queues it only shows Timeout: 60.0 seconds ... and Listing queues for vhost / ... | 19:36 |
| jrosser | noonedeadpunk: it's kind of here on master https://github.com/openstack/openstack-ansible/blob/master/inventory/group_vars/utility_all.yml#L58 | 19:37 |
| noonedeadpunk | jrosser: I think we could add https://opendev.org/openstack/openstack-ansible-galera_server/commit/021b1621e4a9d4ac5de606884eecd8ffb9efcc7c after 20.0.4 | 19:37 |
| noonedeadpunk | not sure | 19:37 |
| *** mmethot has joined #openstack-ansible | 19:37 | |
| jrosser | these need to be in the utility venv with pip i think | 19:38 |
| noonedeadpunk | SecOpsNinja: we have a separate vhost for each service | 19:38 |
| jrosser | becasue it's ansible tasks which do the db user creation | 19:38 |
| noonedeadpunk | jrosser: we were delegating to different host in train | 19:38 |
| jrosser | ooooohhhh | 19:38 |
| noonedeadpunk | which was galera_all[0] | 19:38 |
| noonedeadpunk | we've change that just in master I think | 19:39 |
| noonedeadpunk | SecOpsNinja: so you need to rabbitmqctl list_queues -p nova (or whatever) | 19:39 |
| noonedeadpunk | but rabbit was able to stuck very weirdly, when it was operating from the first sight, but queues were broken somehow | 19:40 |
| noonedeadpunk | not sure how to explain it as never understood, but also didn't faced with that since train... | 19:40 |
| SecOpsNinja | noonedeadpunk, i idn't understand the -p part. i should put the name of the contaiener or the host? | 19:42 |
| noonedeadpunk | of the rabbitmq vhost.... | 19:42 |
| noonedeadpunk | rabbitmqctl list_vhosts | 19:42 |
| * noonedeadpunk needs to go off for today | 19:43 | |
| SecOpsNinja | noonedeadpunk, thanks gain for all the help | 19:43 |
| SecOpsNinja | regarding nova-scheduler is this normal? nova.scheduler.host_manager [req-90773d87-a4ed-45f7-ba7e-5429daf0a0fa - - - - -] Received a sync request from an unknown host 'osa-compute-02'. Re-created its InstanceList. | 19:55 |
| dmsimard | that's a somewhat normal message, iirc it means it's the first time that node is checking in to nova conductor | 20:00 |
| SecOpsNinja | ok i enable debug mode in nova api to see if i can understand that error and try to understand what is causing that | 20:03 |
| SecOpsNinja | i can't understand this error... | 20:04 |
| *** ChiTo has joined #openstack-ansible | 20:04 | |
| SecOpsNinja | is there a way to check that req id to see who started it and to where is trying to connect ? | 20:05 |
| itsjg | What kind of operation was it, resize, migration, etc? You can get more info if you do a grep for the req ID in journalctl (helps to know what the operation was to know which container to look in, assuming you don't have centralized logging) | 20:06 |
| *** melwitt has joined #openstack-ansible | 20:06 | |
| SecOpsNinja | because if i start a instance by dashbord, it rerunt 504 timeout and when i reload it it says in p+owering on and doesn't leave that state | 20:07 |
| itsjg | Oh I see. It would be helpful to run a journalctl on the compute node hosting that and grep for the reqID to get more detailed info, you may also have to enable debug logging on the compute node's nova.conf if there isn't much info | 20:09 |
| *** ChiTo has quit IRC | 20:10 | |
| *** MickyMan77 has joined #openstack-ansible | 20:11 | |
| SecOpsNinja | yep that is the very strange part . i dont see any error in nova-compute or neutron service in compute hosts.... | 20:12 |
| SecOpsNinja | regarding tthe req that gives that Connection failed: [Errno 113] EHOSTUNREACH the previous logs messages regarding that is http://paste.openstack.org/show/799444/ where is talks about Releasing lock "compute-rpcapi-router" ... any ideia what maybe causing this? | 20:15 |
| *** rpittau|afk is now known as rpittau | 20:18 | |
| SecOpsNinja | and the again another starnge fact... the vms started a very long wait lol | 20:18 |
| SecOpsNinja | *after a very long wait in powering up.... | 20:19 |
| *** MickyMan77 has quit IRC | 20:20 | |
| SecOpsNinja | i will try to check tomorow with fresh eyes. thanks everyone o7 | 20:25 |
| *** SecOpsNinja has left #openstack-ansible | 20:31 | |
| *** cshen has quit IRC | 20:31 | |
| admin0 | i removed all /etc/ansible/roles and did a fresh start ( to ensure nothing from any other/master branch has an influence) | 20:50 |
| admin0 | if i see the error, i will paste it | 20:50 |
| *** gouthamr has quit IRC | 20:58 | |
| *** logan- has quit IRC | 20:58 | |
| *** gouthamr has joined #openstack-ansible | 20:59 | |
| *** gouthamr has quit IRC | 20:59 | |
| *** gouthamr has joined #openstack-ansible | 20:59 | |
| *** logan- has joined #openstack-ansible | 21:01 | |
| *** rpittau is now known as rpittau|afk | 21:03 | |
| *** MickyMan77 has joined #openstack-ansible | 21:05 | |
| *** MickyMan77 has quit IRC | 21:13 | |
| *** yann-kaelig has quit IRC | 21:29 | |
| *** spatel has quit IRC | 21:38 | |
| *** aedc_ has quit IRC | 21:42 | |
| *** sshnaidm|rover is now known as sshnaidm|afk | 21:50 | |
| *** rfolco has quit IRC | 22:12 | |
| *** cshen has joined #openstack-ansible | 22:24 | |
| *** cshen has quit IRC | 22:29 | |
| *** MickyMan77 has joined #openstack-ansible | 22:40 | |
| *** pcaruana has quit IRC | 22:46 | |
| *** MickyMan77 has quit IRC | 22:48 | |
| fridtjof[m] | An issue I've consistently been having over multiple releases on initial deployments: | 23:00 |
| fridtjof[m] | When setting up everything (currently Train, ubuntu 18.04, simple storage node with lvm), creating instances fails with a stacktrace containing the message "Connector doesn't have required information: initiator". | 23:02 |
| fridtjof[m] | plugging this into google, I stumble upon this old post: https://ask.openstack.org/en/question/118921/unable-to-attach-volume-due-to-error-invalidinput-invalid-input-received-connector-doesnt-have-required-information-initiator/ | 23:03 |
| fridtjof[m] | which tells me to start iscsid on the compute hosts (it's enabled, so a reboot would work too, I guess). After doing that, everything works fine. | 23:03 |
| fridtjof[m] | Seems like a fairly simple to fix oversight in some playbook? | 23:03 |
| *** cshen has joined #openstack-ansible | 23:27 | |
| *** gshippey has quit IRC | 23:32 | |
| *** cshen has quit IRC | 23:32 | |
| *** macz_ has quit IRC | 23:37 | |
| *** MickyMan77 has joined #openstack-ansible | 23:45 | |
| *** tosky has quit IRC | 23:52 | |
| *** MickyMan77 has quit IRC | 23:53 | |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!