| *** dviroel|rover is now known as dviroel|out | 00:20 | |
| *** ysandeep|out is now known as ysandeep | 04:11 | |
| *** ysandeep is now known as ysandeep|afk | 05:39 | |
| *** ysandeep|afk is now known as ysandeep | 06:00 | |
| *** ysandeep is now known as ysandeep|lunch | 07:32 | |
| *** ysandeep|lunch is now known as ysandeep | 09:16 | |
| chandankumar | jrosser: noonedeadpunk hello, please have a look at this patch https://review.opendev.org/c/openstack/openstack-ansible-os_tempest/+/839790 when free, thanks! | 09:47 |
|---|---|---|
| jrosser | chandankumar: i would have a question, as to why https://opendev.org/openstack/openstack-ansible-os_tempest/src/commit/1baf863bf6dc2630e826fee66541c8203a12609c/defaults/main.yml#L316 is the only variable that seems to be "statically" defined in the profile there? | 09:53 |
| chandankumar | jrosser: https://review.opendev.org/c/openstack/openstack-ansible-os_tempest/+/839790/1#message-97ce8edbfdf91dd194728dad18f94af6cda8e28d | 10:20 |
| chandankumar | jrosser: earlier this var got set fact to empty string as there was no when clause defined and I cannot remember why I added the code like that | 10:28 |
| opendevreview | Pooja Jadhav proposed openstack/openstack-ansible-os_tempest stable/victoria: Ussuri tear down as EOL https://review.opendev.org/c/openstack/openstack-ansible-os_tempest/+/836276 | 10:44 |
| *** ysandeep is now known as ysandeep|afk | 11:03 | |
| *** dviroel|out is now known as dviroel | 11:21 | |
| opendevreview | Marc Gariépy proposed openstack/openstack-ansible-os_mistral master: Refactor use of include_vars https://review.opendev.org/c/openstack/openstack-ansible-os_mistral/+/824325 | 11:38 |
| opendevreview | Marc Gariépy proposed openstack/openstack-ansible-os_cloudkitty master: Refactor use of include_vars https://review.opendev.org/c/openstack/openstack-ansible-os_cloudkitty/+/824336 | 11:38 |
| opendevreview | Marc Gariépy proposed openstack/openstack-ansible-lxc_container_create master: Updated from OpenStack Ansible Tests https://review.opendev.org/c/openstack/openstack-ansible-lxc_container_create/+/835691 | 12:00 |
| spatel | noonedeadpunk or jrosser i have question related deployment node. I am deploying 65 nodes cloud and thinking not use dedicated deployment node because it will take one of my server :) is that ok to have deployment on same infra node? | 13:23 |
| spatel | My all production cloud running on separate deployment node so i don't know if any issue to have deployment node on same infra node. | 13:24 |
| opendevreview | Merged openstack/openstack-ansible-os_octavia master: Change octavia private key ciphers to type 'auto' https://review.opendev.org/c/openstack/openstack-ansible-os_octavia/+/836325 | 13:27 |
| *** ysandeep|afk is now known as ysandeep | 13:36 | |
| mgariepy | arf seems like the gpg server for lxc-create has disapeared | 13:49 |
| mgariepy | arf : openstack-ansible-deploy-hosts_lxc-ubuntu-focal https://zuul.opendev.org/t/openstack/build/3f8c45b06a1f4a8ab7c10a5787710be4 : FAILURE in 18m 38s (non-voting) | 13:55 |
| mgariepy | https://review.opendev.org/c/openstack/openstack-ansible-lxc_container_create/+/839167 | 13:55 |
| mgariepy | https://zuul.opendev.org/t/openstack/build/3f8c45b06a1f4a8ab7c10a5787710be4/log/job-output.txt#5132 | 13:56 |
| opendevreview | Marc Gariépy proposed openstack/openstack-ansible-lxc_container_create master: fix focal relase for lxc https://review.opendev.org/c/openstack/openstack-ansible-lxc_container_create/+/839951 | 13:59 |
| mgariepy | spatel, i don't think it's a really bad idea, just make sure you backup your files correctly in case you loose that server. | 14:10 |
| spatel | mgariepy hope there is not conflict of anything in future :) | 14:11 |
| mgariepy | create a container then to run it ? | 14:11 |
| spatel | container for deployment node? | 14:12 |
| mgariepy | how many cloud do you manage? | 14:12 |
| spatel | 5 | 14:13 |
| mgariepy | i usually have an lxd host for my deployment node. | 14:13 |
| mgariepy | then i create a bunch of containers for different services that i need | 14:13 |
| spatel | interesting.. do you run that lxc on same infra node? | 14:13 |
| mgariepy | usually not. | 14:13 |
| spatel | so you have some kind of dedicated node for all shared tooling ? | 14:14 |
| mgariepy | i either buy extra server for it or i do use an older machine. | 14:14 |
| spatel | Just not sure what is the point to run deployment node on dedicated hardware? | 14:14 |
| spatel | You can loose deployment server and you can loose any infra node also.. | 14:15 |
| mgariepy | yep i run ceph-ansible in separate conainer also central log collections and so on. | 14:15 |
| spatel | why OSA document suggesting to have dedicated deployment node for production ( I am 100% agreed but not sure what is the point) | 14:15 |
| mgariepy | if you loose a controller , it's a bad day. but if it's your deployment node it gets a bit worst ? | 14:17 |
| mgariepy | from where do you install the base-os on those node ? | 14:17 |
| spatel | This is rental datacenter and they have kickstart etc from their portal. | 14:22 |
| mgariepy | ha ok | 14:23 |
| spatel | I can just go to portal and request whatever server and OS i want | 14:23 |
| spatel | then i can create VLANs etc.. and just start deploying OSA | 14:23 |
| spatel | This is rental so i don't want to waste money for deployment node :) | 14:24 |
| mgariepy | well you can do whatever you want, where will you collocate osa deployment tool then ? | 14:24 |
| spatel | I am thinking i can make 3 deployment node running on 3 infra.. In short bootstrap all 3 infra nodes and shared all file on all 3 nodes | 14:24 |
| spatel | in one of my other production deployment i have 2 OSA deployment node (Like HA for high availability) | 14:27 |
| opendevreview | Merged openstack/openstack-ansible-lxc_container_create stable/xena: Wait for 2s after restarting systemd-networkd https://review.opendev.org/c/openstack/openstack-ansible-lxc_container_create/+/839730 | 14:33 |
| opendevreview | Merged openstack/openstack-ansible-lxc_container_create stable/victoria: Wait for 2s after restarting systemd-networkd https://review.opendev.org/c/openstack/openstack-ansible-lxc_container_create/+/839732 | 14:34 |
| spatel | mgariepy i like your idea about running in LXC container.. just trying to test that out | 14:45 |
| mgariepy | well it will probably be an issue of chicken and egg.. | 14:45 |
| spatel | why? | 14:46 |
| mgariepy | depends where you run it. but you might want ot have lxc-host role applied before creating the containers | 14:46 |
| spatel | I was thinking to deploy LXC container manually. using apt-get install lxc on one of my compute node :) | 14:47 |
| mgariepy | okie. but why lxc and not lxd then ? | 14:47 |
| spatel | hmm what is the difference? | 14:48 |
| spatel | if lxd is future then i can do that | 14:48 |
| mgariepy | lxd is pretty nice. does comes installed by default on ubuntu-servers | 14:48 |
| mgariepy | not sure we will switch to lxd soon tho. | 14:49 |
| spatel | oh! let me try lxd then :) | 14:49 |
| mgariepy | didn't you did a lxd / fake metal deployment at some point? | 14:50 |
| spatel | i was thinking but never did it | 14:51 |
| jrosser | remember only to have one inventory | 14:52 |
| jrosser | spatel> I am thinking i can make 3 deployment node running on 3 infra.. In short bootstrap all 3 infra nodes and shared all file on all 3 nodes | 14:52 |
| jrosser | ^ careful here | 14:52 |
| spatel | jrosser hmmm ? | 14:52 |
| jrosser | the files in /etc/openstack_deploy store state | 14:53 |
| jrosser | you can only have one authoritative copy | 14:53 |
| spatel | what if i keep rsync or lsyncd to keep them replicate to other 2 nodes? | 14:54 |
| spatel | or put them on shared glusterfs filesystem to share across all 3 nodes? | 14:54 |
| jrosser | however you like :) | 14:55 |
| jrosser | i just wanted to point out that "bootstrap all 3 infra nodes" could have unexpected outcome unless you do something to ensure there is only one copy of the config | 14:56 |
| spatel | just trying to explore option to see what would be best fit here.. ofc i want you guys tell me anything missing or wrong | 14:56 |
| spatel | 100% i would keep same copy on all 3 nodes otherwise it won't work | 14:56 |
| spatel | i am going to test this out in LAB first | 14:56 |
| spatel | it would be good to have all 3 nodes acting like deployment node so we can sleep well. | 14:57 |
| mgariepy | jrosser, seems like lxc-create had small difference between ubuntu and the other distros: distribution_major_version vs distribution_release :/ | 14:57 |
| jrosser | yes | 14:58 |
| mgariepy | this breaks all the other distro but it did fix focal. https://review.opendev.org/c/openstack/openstack-ansible-lxc_container_create/+/839951 | 14:58 |
| jrosser | wait | 14:59 |
| mgariepy | is hte lxc_hosts change suppose to be fixing the checks ?:/ | 14:59 |
| *** ysandeep is now known as ysandeep|out | 14:59 | |
| mgariepy | my patch doesn't work now.. | 14:59 |
| jrosser | there are two patches | 15:00 |
| mgariepy | https://review.opendev.org/c/openstack/openstack-ansible-lxc_hosts/+/838836 | 15:00 |
| jrosser | this to lxc hosts https://review.opendev.org/c/openstack/openstack-ansible-lxc_hosts/+/838836 | 15:00 |
| jrosser | yes | 15:00 |
| jrosser | and the one to lxc_container_create https://review.opendev.org/c/openstack/openstack-ansible-lxc_container_create/+/839167 | 15:01 |
| jrosser | they can't both merge simultaneously | 15:02 |
| jrosser | so i thought i was making 839167 focal jobs NV | 15:02 |
| jrosser | so that we could then merge 838836 and the two roles would be in alignment again regarding distribution_major_version vs distribution_release | 15:03 |
| mgariepy | well we need to merge lxc_hosts | 15:03 |
| mgariepy | since the merge of lxc_container_create did break everything :D | 15:03 |
| jrosser | and that should be possible now we have merged 839167, it's depends-on? | 15:03 |
| jrosser | oh yes we totally do | 15:03 |
| jrosser | becasue it has the depends-on we should know that it all works out ok in the end :) | 15:04 |
| jrosser | just intermediate brokenness which i don't think can be avoided | 15:04 |
| mgariepy | well it's ok | 15:05 |
| jrosser | andrewbonney: ^ can you look at https://review.opendev.org/c/openstack/openstack-ansible-lxc_hosts/+/838836 | 15:05 |
| mgariepy | just that this morning i was testing in my vm. then key server were downn .. then lxc-create didn't work.. lol | 15:05 |
| jrosser | this is a bit like the old galera_server / galera_client role where they are so tightly bound together, some changes are impossible to land | 15:06 |
| mgariepy | yep, that's ok | 15:06 |
| mgariepy | now it's a bit more clearer. lol | 15:06 |
| andrewbonney | jrosser: is there any risk around existing containers and triggering restarts, or does that only impact initial creation? | 15:08 |
| jrosser | good question - i expect the config file for the container might get re-written to point to the changed name of the base image | 15:09 |
| jrosser | but i'm not really sure about that though | 15:09 |
| jrosser | unless the name of the image is actually stored in the template, rather than the config file | 15:10 |
| jrosser | in which case it's one step decoupled | 15:10 |
| mgariepy | i guess that if we would have been approved lxc_hosts first then lxc_container_create, both patches would had merged in the correct order because of the depends on ? | 15:11 |
| jrosser | i think they both break each other | 15:13 |
| mgariepy | ha ok lol | 15:14 |
| jrosser | this would be a great example of something to have in a collection rather than seperate role repos, then we could change things like this all at the same time | 15:14 |
| *** dviroel is now known as dviroel|lunch | 15:45 | |
| opendevreview | Marc Gariépy proposed openstack/openstack-ansible-lxc_container_create master: Restore focal lxc tests https://review.opendev.org/c/openstack/openstack-ansible-lxc_container_create/+/839964 | 15:56 |
| opendevreview | Marc Gariépy proposed openstack/openstack-ansible-lxc_container_create master: Restore focal lxc tests https://review.opendev.org/c/openstack/openstack-ansible-lxc_container_create/+/839964 | 15:58 |
| *** dviroel|lunch is now known as dviroel | 16:47 | |
| opendevreview | Merged openstack/openstack-ansible-lxc_hosts master: Unify debian and ubuntu cache prep scripts https://review.opendev.org/c/openstack/openstack-ansible-lxc_hosts/+/838836 | 17:05 |
| opendevreview | Marc Gariépy proposed openstack/openstack-ansible-lxc_container_create master: Restore focal lxc tests https://review.opendev.org/c/openstack/openstack-ansible-lxc_container_create/+/839964 | 17:05 |
| mgariepy | does a patch need to have the verified +1 in order to pass gate ? | 17:08 |
| mgariepy | like this one: https://review.opendev.org/c/openstack/openstack-ansible-lxc_hosts/+/834183 | 17:09 |
| opendevreview | Merged openstack/openstack-ansible-os_tempest stable/victoria: Ussuri tear down as EOL https://review.opendev.org/c/openstack/openstack-ansible-os_tempest/+/836276 | 17:28 |
| mgariepy | qemu-kvm: -accel tcg: mprotect of jit buffer: Cannot allocate memory | 17:51 |
| jrosser | is that centos? | 17:52 |
| mgariepy | how do you know | 17:52 |
| mgariepy | https://review.opendev.org/c/openstack/openstack-ansible-os_mistral/+/824325 | 17:53 |
| jrosser | becasue i guess you are looking at a telemetry related job and get OOM | 17:53 |
| jrosser | or that :) | 17:53 |
| jrosser | memory usage on centos seems high and OOM'ey | 17:53 |
| mgariepy | or fragmentation ? | 17:54 |
| jrosser | particularly hard to merge antthing for ceilometer / etc for a while now | 17:54 |
| mgariepy | but the vm is only 256mb.. | 17:54 |
| mgariepy | come onlol | 17:54 |
| mgariepy | do we have the free memory somewhere ? | 17:55 |
| mgariepy | in the logs | 17:55 |
| jrosser | oh we should fix this https://zuul.opendev.org/t/openstack/build/e5a2d384b53c4a009405221dd6cf3de2/log/logs/openstack/instance-info/dstat.log.txt | 17:58 |
| jrosser | long long ago these jobs used to make a dstat report | 17:58 |
| jrosser | im sure there used to be an html version made as well | 18:00 |
| mgariepy | https://zuul.opendev.org/t/openstack/build/e5a2d384b53c4a009405221dd6cf3de2/log/logs/openstack/instance-info/free_12-53-43.log.txt | 18:00 |
| mgariepy | the data is there :D | 18:01 |
| jrosser | it's odd, becasue octaiva is kind of OK and thats also booting a larger VM than that | 18:02 |
| jrosser | looks like we don't limit the mistral threads/processes properly though https://zuul.opendev.org/t/openstack/build/e5a2d384b53c4a009405221dd6cf3de2/log/logs/openstack/instance-info/ps_12-53-43.log.txt | 18:03 |
| jrosser | thats the biggest thing by 2x | 18:04 |
| jrosser | it needs something like this https://github.com/openstack/openstack-ansible/blob/master/tests/roles/bootstrap-host/templates/user_variables.aio.yml.j2#L64-L70 | 18:06 |
| jrosser | here we go https://opendev.org/openstack/openstack-ansible-os_mistral/src/branch/master/defaults/main.yml#L83-L86 | 18:07 |
| mgariepy | https://github.com/openstack/openstack-ansible-tests/blob/master/test-vars.yml#L395-L420 | 18:08 |
| jrosser | thats only for functional tests which we pretty much got rid of all now | 18:09 |
| opendevreview | Marc Gariépy proposed openstack/openstack-ansible master: Limit mistral worker https://review.opendev.org/c/openstack/openstack-ansible/+/839988 | 18:14 |
| mgariepy | like that ? | 18:14 |
| jrosser | feels like these threads/api/processes variables need a thorough review | 18:14 |
| jrosser | yeah | 18:15 |
| mgariepy | yep i guess it would be needed. | 18:15 |
| jrosser | like placement_wsgi_processes_max setting is pointless if we already override placement_wsgi_processes | 18:15 |
| jrosser | and quite a few others like that | 18:15 |
| mgariepy | or only set the max one ;) | 18:15 |
| jrosser | right :) | 18:16 |
| mgariepy | wo we validate the min filter is still working ... lol | 18:16 |
| jrosser | i guess that part of the downside to having a venv per service is that the whole python environment is unique pages for each service | 18:17 |
| opendevreview | Marc Gariépy proposed openstack/openstack-ansible-os_mistral master: Refactor use of include_vars https://review.opendev.org/c/openstack/openstack-ansible-os_mistral/+/824325 | 18:17 |
| jrosser | so there is no sharing of memory there for all the common stuff as there might otherwise be with a single system python | 18:18 |
| opendevreview | Marc Gariépy proposed openstack/openstack-ansible master: Limit cloudkitty workers https://review.opendev.org/c/openstack/openstack-ansible/+/839989 | 18:24 |
| opendevreview | Marc Gariépy proposed openstack/openstack-ansible-os_cloudkitty master: Refactor use of include_vars https://review.opendev.org/c/openstack/openstack-ansible-os_cloudkitty/+/824336 | 18:25 |
| mgariepy | yay 2 more fixes :D | 18:26 |
| opendevreview | Marc Gariépy proposed openstack/openstack-ansible-os_cloudkitty master: Updated from OpenStack Ansible Tests https://review.opendev.org/c/openstack/openstack-ansible-os_cloudkitty/+/835703 | 18:35 |
| opendevreview | Marc Gariépy proposed openstack/openstack-ansible-os_mistral master: Updated from OpenStack Ansible Tests https://review.opendev.org/c/openstack/openstack-ansible-os_mistral/+/835714 | 18:45 |
| opendevreview | Marc Gariépy proposed openstack/openstack-ansible-os_mistral master: Use common service setup tasks from a collection rather than in-role https://review.opendev.org/c/openstack/openstack-ansible-os_mistral/+/824383 | 18:46 |
| opendevreview | Marc Gariépy proposed openstack/openstack-ansible-os_mistral master: Remove legacy policy.json cleanup handler https://review.opendev.org/c/openstack/openstack-ansible-os_mistral/+/827448 | 18:47 |
| opendevreview | Marc Gariépy proposed openstack/openstack-ansible-os_cloudkitty master: Remove legacy policy.json cleanup handler https://review.opendev.org/c/openstack/openstack-ansible-os_cloudkitty/+/827436 | 18:48 |
| opendevreview | Marc Gariépy proposed openstack/openstack-ansible-os_cloudkitty master: Use common service setup tasks from a collection rather than in-role https://review.opendev.org/c/openstack/openstack-ansible-os_cloudkitty/+/824399 | 18:49 |
| opendevreview | Jonathan Rosser proposed openstack/openstack-ansible-repo_server master: Add upgrade path from lsyncd to shared filesystem. https://review.opendev.org/c/openstack/openstack-ansible-repo_server/+/839411 | 18:52 |
| opendevreview | Jonathan Rosser proposed openstack/openstack-ansible-repo_server master: Add upgrade path from lsyncd to shared filesystem. https://review.opendev.org/c/openstack/openstack-ansible-repo_server/+/839411 | 18:55 |
| opendevreview | Jonathan Rosser proposed openstack/openstack-ansible-repo_server master: Remove all code for lsync, rsync and ssh https://review.opendev.org/c/openstack/openstack-ansible-repo_server/+/837588 | 18:55 |
| opendevreview | Jonathan Rosser proposed openstack/openstack-ansible-repo_server master: Clean up legacy lsycnd, rsync and ssh key config https://review.opendev.org/c/openstack/openstack-ansible-repo_server/+/837859 | 18:55 |
| mgariepy | hmm. rocky zull image doesn't have passwd installed.. :/ | 19:45 |
| *** spotz_ is now known as spotz | 19:53 | |
| mgariepy | on this, have a nice weekend | 19:55 |
| opendevreview | Merged openstack/openstack-ansible-lxc_container_create master: Updated from OpenStack Ansible Tests https://review.opendev.org/c/openstack/openstack-ansible-lxc_container_create/+/835691 | 20:35 |
| opendevreview | Merged openstack/openstack-ansible-lxc_container_create stable/wallaby: Wait for 2s after restarting systemd-networkd https://review.opendev.org/c/openstack/openstack-ansible-lxc_container_create/+/839731 | 20:35 |
| opendevreview | Merged openstack/openstack-ansible-lxc_hosts master: Replace systemd-mount template with role https://review.opendev.org/c/openstack/openstack-ansible-lxc_hosts/+/834183 | 20:56 |
| opendevreview | Jonathan Rosser proposed openstack/openstack-ansible stable/stein: Mark OSA repository as safe in git.config in CI https://review.opendev.org/c/openstack/openstack-ansible/+/838023 | 21:31 |
| opendevreview | Merged openstack/openstack-ansible master: Do not encrypt SSL for CentOS distro path https://review.opendev.org/c/openstack/openstack-ansible/+/837571 | 22:50 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!