| *** tosky has quit IRC | 00:04 | |
| *** cshen has joined #openstack-ansible | 00:29 | |
| *** cshen has quit IRC | 00:33 | |
| *** yann-kaelig has quit IRC | 00:41 | |
| *** maharg101 has joined #openstack-ansible | 00:51 | |
| *** maharg101 has quit IRC | 00:56 | |
| *** macz_ has quit IRC | 00:58 | |
| *** spatel has joined #openstack-ansible | 01:59 | |
| *** dasp has quit IRC | 02:20 | |
| *** dasp has joined #openstack-ansible | 02:21 | |
| *** cshen has joined #openstack-ansible | 02:29 | |
| *** cshen has quit IRC | 02:33 | |
| *** macz_ has joined #openstack-ansible | 02:55 | |
| *** macz_ has quit IRC | 03:00 | |
| *** rfolco has joined #openstack-ansible | 03:11 | |
| snadge | https://freedomben.medium.com/centos-is-not-dead-please-stop-saying-it-is-at-least-until-you-read-this-4b26b5c44877 .. this is a bit offtopic but it's an interesting read if you're at all concerned about this recent redhat/centos debacle | 03:27 |
|---|---|---|
| *** jamesden_ has quit IRC | 03:29 | |
| snadge | its from the perspective of someone who works at redhat, but is not directly involved.. moreso a community / user perspective since its not an official redhat statement/opinion | 03:29 |
| *** rfolco has quit IRC | 03:32 | |
| *** gyee has quit IRC | 03:43 | |
| *** macz_ has joined #openstack-ansible | 03:46 | |
| *** macz_ has quit IRC | 03:50 | |
| *** cshen has joined #openstack-ansible | 04:29 | |
| *** cshen has quit IRC | 04:33 | |
| *** shyamb has joined #openstack-ansible | 04:37 | |
| *** shyamb has quit IRC | 05:05 | |
| *** shyamb has joined #openstack-ansible | 05:05 | |
| *** evrardjp has quit IRC | 05:33 | |
| *** evrardjp has joined #openstack-ansible | 05:33 | |
| *** spatel has quit IRC | 05:36 | |
| *** shyamb has quit IRC | 05:41 | |
| *** rohit02 has joined #openstack-ansible | 05:43 | |
| rohit02 | @Team Openstack ansible ussuri failed on centos 8 with Error " "msg": "file not found: /var/www/repo/os-releases/21.2.1.dev6/keystone-21.2.1.dev6-constraints.txt" | 05:46 |
| rohit02 | in keystone container | 05:46 |
| *** shyamb has joined #openstack-ansible | 05:58 | |
| *** cshen has joined #openstack-ansible | 06:00 | |
| *** cshen has quit IRC | 06:04 | |
| *** shyamb has quit IRC | 06:11 | |
| jrosser | rohit02: yes, the release of centos 8.3 this week had a bunch of breaking changes unfortunately, and it's taken a while to work out how to deal with those | 06:21 |
| jrosser | please can you check that you have this patch to openstack_hosts ansible role https://review.opendev.org/c/openstack/openstack-ansible-openstack_hosts/+/766707 | 06:21 |
| *** shyamb has joined #openstack-ansible | 06:21 | |
| *** SiavashSardari has joined #openstack-ansible | 06:44 | |
| *** shyamb has quit IRC | 07:03 | |
| *** shyamb has joined #openstack-ansible | 07:08 | |
| *** cshen has joined #openstack-ansible | 07:19 | |
| *** lkoranda has joined #openstack-ansible | 07:24 | |
| noonedeadpunk | morning | 07:31 |
| noonedeadpunk | uh, zun timedout :( | 07:32 |
| *** pto has joined #openstack-ansible | 07:34 | |
| openstackgerrit | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_glance stable/train: Trigger uwsgi restart https://review.opendev.org/c/openstack/openstack-ansible-os_glance/+/767145 | 07:34 |
| *** lkoranda has quit IRC | 07:35 | |
| *** shyam89 has joined #openstack-ansible | 07:56 | |
| *** shyamb has quit IRC | 07:59 | |
| *** shyam89 has quit IRC | 08:00 | |
| *** andrewbonney has joined #openstack-ansible | 08:07 | |
| *** rpittau|afk is now known as rpittau | 08:18 | |
| *** rohit02 has quit IRC | 08:26 | |
| *** rohit02 has joined #openstack-ansible | 08:26 | |
| *** tosky has joined #openstack-ansible | 08:33 | |
| *** maharg101 has joined #openstack-ansible | 08:41 | |
| *** jawad_axd has joined #openstack-ansible | 08:43 | |
| jrosser | morning | 08:45 |
| SiavashSardari | morning | 08:47 |
| SiavashSardari | I have a question, how can I use another disk for galera containers? is it possible in OSA? | 08:48 |
| SiavashSardari | for example I'm using sda for root and all containers and I wanna use sdb for mysql data path (or even the whole galera container) | 08:50 |
| jrosser | that is down to how you configure the lxc_hosts role | 09:01 |
| frickler | so IIUC nspawn is being phased out, seems that should be reflected in https://docs.openstack.org/project-deploy-guide/openstack-ansible/latest/app-aboutosa.html#when-not-to-choose-openstack-ansible | 09:02 |
| jrosser | if you look at the CI results for this job, you'll see that there are a number of storage backends that are tested for LXC | 09:02 |
| jrosser | https://review.opendev.org/c/openstack/openstack-ansible-lxc_container_create/+/755996 | 09:02 |
| jrosser | SiavashSardari: currently you can choose to have lxc backed by either dir, overlayfs, btrfs or zfs | 09:03 |
| SiavashSardari | jrosser Thank you. I figured that I should look into lxc_hosts role, I just didn't find a way of how to make a special case for galera containers. I will check the CI results(y) | 09:06 |
| jrosser | you can't make a special case for galera really | 09:06 |
| jrosser | it's the backing store for the whole of lxc on a particular host that you can decide | 09:06 |
| jrosser | frickler: i'm somewhat struggling to find that text in the source code..... | 09:07 |
| frickler | jrosser: https://opendev.org/openstack/openstack-ansible/src/branch/master/deploy-guide/source/app-aboutosa.rst | 09:08 |
| jrosser | oh! well thats becasue i'm looking in doc :) | 09:08 |
| SiavashSardari | jrosser hmmm, but isn't it a nice feature? what is your opinion in this matter? I wanna work on that and I'd like to contribute that to upstream. | 09:09 |
| jrosser | SiavashSardari: well, if lxc has the capability for different containers to be on different backing store, then sure | 09:10 |
| jrosser | i have them separated, operating system on one disk(s) and containers on others | 09:12 |
| *** shyamb has joined #openstack-ansible | 09:17 | |
| openstackgerrit | Jonathan Rosser proposed openstack/openstack-ansible master: Update deploy guide to reflect the current status of OSA https://review.opendev.org/c/openstack/openstack-ansible/+/767285 | 09:22 |
| SiavashSardari | jrosser I was thinking abount mounting galera container dir berfor initializing the container, this way doesn't complicate things on lxc storage backends | 09:25 |
| SiavashSardari | or even maybe mount sdb to galera data path before installing galera | 09:27 |
| jrosser | there are native means in openstack-ansible to do all of this | 09:28 |
| jrosser | even now just looking at the lxc_container_create role there is this https://github.com/openstack/openstack-ansible-lxc_container_create/blob/master/defaults/main.yml#L56 | 09:29 |
| *** jawad_axd has quit IRC | 09:30 | |
| SiavashSardari | yeah but that applies to all containers | 09:30 |
| jrosser | you'd have to test it but certainly for the directory backend (i don't know which you are using?) it may just be that you need to override that variable in the group_vars for galera | 09:30 |
| *** jawad_axd has joined #openstack-ansible | 09:30 | |
| jrosser | no, it's ansible :) | 09:30 |
| jrosser | you take advantage of host / group vars to set all the role defaults as you want them | 09:30 |
| jrosser | remember that user_variables is global | 09:31 |
| jrosser | host/group vars can be specific | 09:31 |
| jrosser | the same goes for bind mounts into containers etc etc | 09:31 |
| SiavashSardari | oh you're right, nice hint (y) | 09:32 |
| jrosser | you'll find that there is a huge amount of flexibility already in all of the role defaults/main.yml | 09:32 |
| jrosser | most things that appear 'hard wired' are actually only really set as defaults that can be further overridden | 09:32 |
| SiavashSardari | yeah and I love this level of flexibility in OSA. funny thing is that I'm already using group vars for other stuff but never thought about them in this matter. | 09:35 |
| *** shyam89 has joined #openstack-ansible | 09:36 | |
| SiavashSardari | Thanks jrosser | 09:36 |
| *** KelvinLai has joined #openstack-ansible | 09:37 | |
| *** shyamb has quit IRC | 09:38 | |
| SiavashSardari | BTW, I just found out that maas 2.9 has added OVS support. if anyone is interested to try... | 09:38 |
| jrosser | KelvinLai: can you paste the failing part from python_venv_build.log? | 09:39 |
| KelvinLai | that's the thing I couldn't find it. | 09:40 |
| KelvinLai | Is this the one? 2020-12-16T01:00:18,417 *** Error compiling '/openstack/venvs/cinder-22.0.0.0b2.dev54/lib/python3.8/site-packages/testtools/_compat2x.py'...2020-12-16T01:00:18,417 File "/openstack/venvs/cinder-22.0.0.0b2.dev54/lib/python3.8/site-packages/testtools/_compat2x.py", line 162020-12-16T01:00:18,417 raise exc_class, exc_obj, | 09:40 |
| KelvinLai | exc_tb2020-12-16T01:00:18,417 ^2020-12-16T01:00:18,417 SyntaxError: invalid syntax | 09:40 |
| KelvinLai | I'm not too sure but don't think that's the error as the playbook error message was specific on netifaces | 09:42 |
| KelvinLai | There are only 2 paragraph in the whole file that has the Error word and it's the above and this one | 09:44 |
| KelvinLai | 2020-12-16T01:00:18,417 *** Error compiling '/openstack/venvs/cinder-22.0.0.0b2.dev54/lib/python3.8/site-packages/testtools/_compat2x.py'...2020-12-16T01:00:18,417 File "/openstack/venvs/cinder-22.0.0.0b2.dev54/lib/python3.8/site-packages/testtools/_compat2x.py", line 162020-12-16T01:00:18,417 raise exc_class, exc_obj, | 09:44 |
| KelvinLai | exc_tb2020-12-16T01:00:18,417 ^2020-12-16T01:00:18,417 SyntaxError: invalid syntax | 09:44 |
| jrosser | thats from during the build of cinder | 09:45 |
| KelvinLai | Eh... Sorry. | 09:45 |
| KelvinLai | 2020-12-16T01:16:19,220 *** Error compiling '/openstack/venvs/nova-22.0.0.0b2.dev54/lib/python3.8/site-packages/testtools/_compat2x.py'...2020-12-16T01:16:19,220 File "/openstack/venvs/nova-22.0.0.0b2.dev54/lib/python3.8/site-packages/testtools/_compat2x.py", line 162020-12-16T01:16:19,220 raise exc_class, exc_obj, | 09:46 |
| KelvinLai | exc_tb2020-12-16T01:16:19,220 ^2020-12-16T01:16:19,220 SyntaxError: invalid syntax2020-12-16T01:16:19,220 | 09:46 |
| KelvinLai | cinder and nova build had this error in the whole of the log file. | 09:46 |
| *** tosky_ has joined #openstack-ansible | 09:47 | |
| KelvinLai | grep -i error /var/log/python_venv_build.log didn't give any other errors other than the 2 listed above | 09:47 |
| KelvinLai | http://paste.openstack.org/show/801079/ | 09:48 |
| KelvinLai | this is the playbook error message | 09:48 |
| jrosser | could you look at the python_venv_build log with 'less' or an editor and skip right to the end | 09:49 |
| *** tosky is now known as Guest24372 | 09:49 | |
| *** tosky_ is now known as tosky | 09:49 | |
| jrosser | see if there is anything bad looking, as it will be pretty much the last stuff in the file which has gone wrong | 09:49 |
| *** Guest24372 has quit IRC | 09:50 | |
| KelvinLai | the playbook error message in paste.openstack.org above is "Building wheel for netifaces (setup.py): started\n Building wheel for netifaces (setup.py): finished with status 'error'" | 09:50 |
| KelvinLai | hang on I'll check the log file | 09:51 |
| jrosser | noonedeadpunk: do you have any ideas on this? ^ | 09:53 |
| KelvinLai | the last few lines is this. | 09:56 |
| KelvinLai | 2020-12-16T13:17:34,862 Installed version (0.35.1) is most up-to-date (past versions: 0.35.1)2020-12-16T13:17:34,862 Requirement already up-to-date: wheel==0.35.1 in /openstack/venvs/neutron-22.0.0.0b2.dev54/lib/python3.8/site-packages (from -c /openstack/venvs/neutron-22.0.0.0b2.dev54/global-constraints.txt (line 3)) | 09:56 |
| KelvinLai | (0.35.1)2020-12-16T13:17:35,444 WARNING: You are using pip version 20.2.3; however, version 20.3.3 is available.2020-12-16T13:17:35,444 You should consider upgrading via the '/openstack/venvs/neutron-22.0.0.0b2.dev54/bin/python -m pip install --upgrade pip' command.2020-12-16T13:17:35,444 Removed build tracker: '/tmp/pip-req-tracker-vaprelfd' | 09:56 |
| KelvinLai | This is a recurring error. I did fresh setup of Ubuntu 20.04. And followed the instructions for AIO. | 09:57 |
| jrosser | whats a bit confusing is that the error message is about stackviz here http://paste.openstack.org/show/801079/ | 09:57 |
| jrosser | but the things you are finding in the wheel build log are to do with cinder and neutron | 09:58 |
| KelvinLai | I did a bit of google search and found some doc that mentioned the raise error was something that can be ignored. | 10:00 |
| KelvinLai | i mean article. | 10:01 |
| jrosser | has this been reproducible? | 10:02 |
| KelvinLai | yup. i reformatted and reinstalled everything from fresh. | 10:03 |
| KelvinLai | twice | 10:03 |
| jrosser | ok, i will make a focal AIO here and take a look | 10:03 |
| jrosser | and which branch / tag did you use? | 10:04 |
| *** shyamb has joined #openstack-ansible | 10:06 | |
| jrosser | KelvinLai: it looks like checking out master branch? | 10:06 |
| KelvinLai | master | 10:06 |
| KelvinLai | yup the master branch | 10:06 |
| jrosser | i'm doing this in a VM from the ubuntu focal cloud image | 10:07 |
| jrosser | hopefully similar enough | 10:07 |
| KelvinLai | ok. Thanks :) | 10:08 |
| *** shyam89 has quit IRC | 10:09 | |
| *** KelvinLai has quit IRC | 10:09 | |
| *** KelvinLai has joined #openstack-ansible | 10:12 | |
| KelvinLai | jrosser dont know why my session got disconnected. will be waiting for your focal Ubuntu testing | 10:15 |
| openstackgerrit | Jonathan Rosser proposed openstack/openstack-ansible master: Update deploy guide to reflect the current status of OSA https://review.opendev.org/c/openstack/openstack-ansible/+/767285 | 10:25 |
| *** SiavashSardari has quit IRC | 10:41 | |
| *** kukacz has quit IRC | 11:04 | |
| noonedeadpunk | hm, interesting but have no idea what went wrog here. as I believe we do install stackviz with all ci jobs | 11:07 |
| KelvinLai | not too sure if it's stackviz but the message seemed to be pointing to netifaces "Successfully built stackviz\nFailed to build netifaces" | 11:16 |
| noonedeadpunk | it doesn't seem that stackviz required netifaces at all? | 11:18 |
| *** macz_ has joined #openstack-ansible | 11:18 | |
| KelvinLai | I'm not sure. It's all new to me. | 11:22 |
| *** macz_ has quit IRC | 11:23 | |
| *** kukacz has joined #openstack-ansible | 11:26 | |
| noonedeadpunk | 99% you don't need stackwiz then :) | 11:28 |
| noonedeadpunk | you can easily define `tempest_run_stackviz: false` in user_variables.yml | 11:29 |
| KelvinLai | but is the problem with stackviz? why is the error saying that stackviz is success but netifaces fail? | 11:30 |
| noonedeadpunk | anyway according to paste issue is raised during stackviz installation into it's dedicated venv. Once you won't need to install it, the issue won't happen I guess. But I'm not sure what's wrong with netifaces module and if we need them somewhere except stackviz. Considering stackviz installation should be the last step of openstack deployment, everything else should have passed | 11:32 |
| *** cshen has quit IRC | 11:33 | |
| noonedeadpunk | According to https://pypi.org/project/netifaces/ module just do not have py3.8 support... | 11:33 |
| KelvinLai | ok will try again after this 3rd install try. in the middle of 3 time trying. | 11:37 |
| *** mensis has joined #openstack-ansible | 11:48 | |
| *** shyamb has quit IRC | 11:51 | |
| *** KelvinLai has quit IRC | 12:00 | |
| *** rfolco has joined #openstack-ansible | 12:01 | |
| *** mgariepy has quit IRC | 12:02 | |
| jrosser | noonedeadpunk: you know i'm not sure that constraints are applied here https://github.com/openstack/openstack-ansible-os_nova/blob/master/tasks/nova_db_post_setup.yml#L23 | 12:04 |
| jrosser | argh wrong ling | 12:05 |
| jrosser | here https://github.com/openstack/openstack-ansible-os_tempest/blob/master/tasks/tempest_install.yml#L36-L44 | 12:05 |
| jrosser | so this is totally possible to be the new pip resolver | 12:05 |
| noonedeadpunk | welll....... it could be..... | 12:07 |
| *** KelvinLai has joined #openstack-ansible | 12:10 | |
| KelvinLai | THANKS for all the help :) | 12:11 |
| KelvinLai | tempest_run_stackviz: false did the trick | 12:12 |
| noonedeadpunk | it feel like really new resolver since it's the only place we're not sure about :) | 12:12 |
| jrosser | first think i'll check is if this has effect for stackviz https://opendev.org/openstack/openstack-ansible/src/branch/master/inventory/group_vars/all/all.yml#L149 | 12:13 |
| KelvinLai | finally managed to get the setup-openstack.yml to run successfully | 12:13 |
| jrosser | awesome! sorry for the trouble, it shouldnt break like that | 12:13 |
| jrosser | particularly as for some reason it all passes in CI ...... | 12:13 |
| KelvinLai | btw i guess why the python logs aren't showing might be because the built is running inside the containers? | 12:14 |
| jrosser | oh well yes, the python_venv_build log will be inside the repo container | 12:14 |
| jrosser | when it's an lxc deployment | 12:14 |
| KelvinLai | jrosser no worries man. You all did a heck of a great job and made it easier for lots of people to try openstack out. | 12:16 |
| KelvinLai | yeah I thought so. Cause the log wasn't updating and I looked closely and noticed the continer built. | 12:17 |
| jrosser | when you've got a host being the repo server, that one does the wheel builds once on behalf of all the other hosts | 12:17 |
| noonedeadpunk | we also have some interminent issue during galera deployment which results in timeouts.... | 12:19 |
| jrosser | which step did that happen at? | 12:19 |
| jrosser | becasue i just had something wierd in this AIO when "adding users to db" | 12:20 |
| noonedeadpunk | I was noticing on root user removal | 12:20 |
| jrosser | i didnt debug it, just delete/recreate galera lxc and it worked straight away, so yes intermittent | 12:20 |
| noonedeadpunk | yeah :( and I'm not sure at all why this is hapenning... not removing root user was causing some other issues iirc | 12:23 |
| jrosser | well we could 10.5.6 -> 10.5.8 but ....... | 12:26 |
| jrosser | no seeing anything really related in the changelog though | 12:28 |
| jrosser | 10.5.7 does seem to have a large number of fixes mind you | 12:28 |
| noonedeadpunk | I was given some recomendations from the mariadb folks (to be specific from monty) but can't find them | 12:29 |
| noonedeadpunk | and it was specificly about the way we operate with users... | 12:29 |
| noonedeadpunk | well, we anyway need to bump mariadb version | 12:30 |
| openstackgerrit | Jonathan Rosser proposed openstack/openstack-ansible-galera_server master: Bump mariadb version to 10.5.8 https://review.opendev.org/c/openstack/openstack-ansible-galera_server/+/767341 | 12:32 |
| noonedeadpunk | btw, new ansiblebase just released as well:) | 12:33 |
| noonedeadpunk | iirc the idea was not to touch system root user | 12:36 |
| openstackgerrit | Jonathan Rosser proposed openstack/openstack-ansible master: Bump ansible version to 2.10.4 https://review.opendev.org/c/openstack/openstack-ansible/+/767343 | 12:39 |
| openstackgerrit | Jonathan Rosser proposed openstack/openstack-ansible-tests master: Bump ansible version to 2.10.4 https://review.opendev.org/c/openstack/openstack-ansible-tests/+/767344 | 12:41 |
| *** KelvinLai has quit IRC | 12:45 | |
| openstackgerrit | Jonathan Rosser proposed openstack/openstack-ansible master: Bump ansible collection versions https://review.opendev.org/c/openstack/openstack-ansible/+/767345 | 12:45 |
| openstackgerrit | Jonathan Rosser proposed openstack/openstack-ansible-tests master: Bump ansible collection versions https://review.opendev.org/c/openstack/openstack-ansible-tests/+/767346 | 12:47 |
| *** KelvinLai has joined #openstack-ansible | 13:02 | |
| *** mgariepy has joined #openstack-ansible | 13:09 | |
| jrosser | KelvinLai: i now have a easier to read version of the stackviz error http://paste.openstack.org/show/801088/ | 13:20 |
| *** spatel has joined #openstack-ansible | 13:22 | |
| KelvinLai | it's so missleading the building message where it says Successfully built stackviz | 13:23 |
| jrosser | on line 66 the wheel for stackviz is correctly built | 13:25 |
| KelvinLai | btw, how can I install zun now that I've managed to finish the setup-openstack.yml? I read the docs, it says I can just go run openstack-ansible os-install-zun.yml but that doesn't work because the inventory wasn't set. Do I need to run the bootstrap-aio.sh script again? | 13:25 |
| jrosser | it can perhaps be done like that | 13:28 |
| jrosser | bootstrap_aio.sh understands the SCENARIO environment variable, and depending on the setting you can get extra components added | 13:28 |
| jrosser | export SCENARIO=aio_lxc_zun would be an example of that | 13:29 |
| KelvinLai | ok will try. Praying that I don't need to resetup everything. 😅 | 13:29 |
| jrosser | but there is a big caveat for zun, you will totally need this patch to the os_zun ansible role https://review.opendev.org/c/openstack/openstack-ansible-os_zun/+/763141/ | 13:30 |
| *** sep has left #openstack-ansible | 13:30 | |
| jrosser | alternative to adjusting SCENARIO is adding hosts to the inventory yourself and adding the conf.d files that the bootstrap script would have done | 13:31 |
| jrosser | the bootstrap script is really a helper for AIO testing and CI jobs, no more than that | 13:31 |
| jrosser | noonedeadpunk: heres the failure for building netifaces http://paste.openstack.org/show/801089/ | 13:35 |
| jrosser | and the reason it is pulled in at all is this https://github.com/openstack/oslo.utils/blame/master/requirements.txt#L16 | 13:40 |
| KelvinLai | jrosser how would I go about patchine the os_zun role? I cant seem to find the role under the /opt/openstack-ansible structure | 13:45 |
| KelvinLai | patching | 13:45 |
| jrosser | the ansible roles are in /etc/ansible/roles | 13:45 |
| KelvinLai | oh... stupid me. | 13:45 |
| jrosser | if you click the link for the patch first | 13:45 |
| jrosser | then sort of top-right there is a menu with 3 little blobs, click that | 13:46 |
| jrosser | choose "download patch" | 13:46 |
| jrosser | and press the button at the right hand end of the "cherry pick" line in the pop up dialog | 13:46 |
| jrosser | thats copied a command to your clipboard | 13:47 |
| jrosser | then cd /etc/ansible/roles/os_zun | 13:47 |
| jrosser | paste the command and run it, it should download and apply that patch to your local copy of the zun role | 13:47 |
| jrosser | you can confirm that with git log / git show | 13:48 |
| admin0 | KelvinLai, what tag are yuo using ? | 13:54 |
| admin0 | i sometimes see this on master | 13:54 |
| admin0 | but on a tagged one, should be good | 13:54 |
| admin0 | i see you are trying to use dev 22.0 | 13:54 |
| admin0 | i have not(yet) tried aio on the 22.0 b branch | 13:55 |
| KelvinLai | no worries. I will try it. I just followed the doc and git checkout master. So i guess it's the latest tag | 13:57 |
| KelvinLai | no worries. I'll just try | 13:57 |
| KelvinLai | I just patched my os_zun will try to install later. | 13:58 |
| admin0 | KelvinLai, if possible, stick to tagged branches ? | 13:59 |
| KelvinLai | ok. Just checked it's 22.0.0.0b1 | 14:01 |
| tbarron | I'd like to check on the status of manila in OSA. How integrated is it now? What manila backends are supported? Are there obstacles from OSA viewpoint? | 14:02 |
| noonedeadpunk | tbarron: hi! a the moment we test in CI ganesha+nfs backends | 14:03 |
| tbarron | ^^ This is for a doc I'm writin in manila proper, where I'm collecting status w.r.t. various deployment tools. | 14:03 |
| tbarron | noonedeadpunk: working out for you? | 14:03 |
| noonedeadpunk | but we don't test ha mode of it yet - had no time to add backend in front and test containers deployment | 14:03 |
| noonedeadpunk | well, I think it does and ppl were reporting that they managed to use it as well with small adjustments | 14:04 |
| noonedeadpunk | I think we're also running basic backend (can't recall how properly it's called) here internally | 14:05 |
| noonedeadpunk | under test I mean we're running tempest | 14:05 |
| tbarron | noonedeadpunk: do you know if the "basic" back end is the "generic" backend? or possibly the lvm back end? | 14:06 |
| noonedeadpunk | it's generic, yes :) | 14:06 |
| tbarron | noonedeadpunk: ty | 14:06 |
| tbarron | noonedeadpunk: I will mention that we are starting to pursue an alternative to ganesha as an gateway for native CephFS | 14:07 |
| tbarron | and actually for other remote fiilesystems as well | 14:07 |
| tbarron | noonedeadpunk: https://etherpad.opendev.org/p/nova-manila-virtio-fs-support-vptg-october-2020 | 14:07 |
| tbarron | noonedeadpunk: one motivation is the HA challenges with ganesha that you mentioned | 14:08 |
| tbarron | noonedeadpunk: we have pacemaker machinery in tripleo to assure fault tolerance for ganesha but it"s pretty heavy-weight and limits scale out | 14:09 |
| noonedeadpunk | eventually out of tempest we just run test_mount_share_one_vm and test_write_with_ro_access from manila_tempest_tests.tests.scenario.test_share_basic_ops.TestShareBasicOpsNFS | 14:09 |
| tbarron | noonedeadpunk: thanks, I get why you'd use generic for those tempest tests | 14:09 |
| noonedeadpunk | no, for these test we use ganesha and cephfs :) | 14:10 |
| tbarron | noonedeadpunk: ah, better, didn't look hard aat the tests you mentioned :) | 14:10 |
| *** macz_ has joined #openstack-ansible | 14:11 | |
| tbarron | cephfs with ganesha will work for the tests that don't require so-called DHSS=true mode | 14:11 |
| *** kukacz has quit IRC | 14:11 | |
| noonedeadpunk | well, in our case we have keepalived anyway along with haproxy for balancing, so giving ha for ganesha is totally ok for us - just needs a bit of love and time | 14:11 |
| tbarron | noonedeadpunk: will be interested to follow when you get to that. NFS HA is tricky ... | 14:12 |
| noonedeadpunk | well yes, agree.... | 14:13 |
| noonedeadpunk | well looked blueprint briefly and it sounds really good | 14:13 |
| tbarron | early stages, but wanted to give a heads up to folks who may be interested | 14:14 |
| noonedeadpunk | Not sure at the moment what challenges will rise from the deployer prespective though | 14:14 |
| noonedeadpunk | But I'df love to get this implemented in osa once it will release | 14:15 |
| *** macz_ has quit IRC | 14:15 | |
| noonedeadpunk | it seems that libvirt support is even in progress at the moment :( | 14:16 |
| tbarron | noonedeadpunk: yes, this relies on bleeding edge libvirt/qemu. Looks like it's happening though. | 14:17 |
| tbarron | noonedeadpunk: the idea is to be ready for it by developing in parallel. | 14:18 |
| noonedeadpunk | It would require some time for libvirt to land on OS as well.. | 14:18 |
| noonedeadpunk | yeah, agree and really looks very nice but also pretty challenging from development prespective :) | 14:18 |
| tbarron | noonedeadpunk: Then when it's ready I think deployment projects should be able to use it more simply than today's manila integration. | 14:18 |
| tbarron | noonedeadpunk: agree. And most of the work will be in nova. | 14:19 |
| noonedeadpunk | and what will be manila future? as eventually it seems like manila will become kind of redundant once it's implemented? | 14:20 |
| *** KelvinLai has quit IRC | 14:21 | |
| noonedeadpunk | (I really did very brief look though) | 14:21 |
| tbarron | noonedeadpunk: manila won't be redundant, it will still be the self-service shared filesystem service. It's just | 14:21 |
| tbarron | that if you mount the shares from Nova VMs (vs bare metal compute, say) then | 14:21 |
| tbarron | you'd be able to "attach" them to Nova VMs more or less like you can attach Cinder volumes. | 14:22 |
| tbarron | But you'd do all the CRUD operations, etc. on the shares using Manila. | 14:22 |
| tbarron | Similar to Nova/Cinder. | 14:23 |
| tbarron | And simpler for deployment projects because you don't have to stand up ganesha with HA in order to get highly available "safe" CephFS shares. | 14:24 |
| tbarron | If a compute node crashes and the guest VM goes to another node then it will find a virtiofs gateway there and the same share export location for mounts | 14:25 |
| noonedeadpunk | oh, ok, got it! | 14:25 |
| noonedeadpunk | thanks much for clarifying | 14:25 |
| tbarron | noonedeadpunk: well thanks for your time catching me up on manila status in OSA! | 14:26 |
| noonedeadpunk | tbarron: sure, anytime! | 14:26 |
| *** jbadiapa has joined #openstack-ansible | 14:48 | |
| jrosser | tbarron: that sounds very interesting and a really nice architectural improvement | 14:50 |
| *** rohit02 has quit IRC | 14:51 | |
| jrosser | noonedeadpunk: i know why the stackviz fails, it's 2020-12-16T13:27:43,619 netifaces.c:1:10: fatal error: Python.h: No such file or directory | 14:52 |
| jrosser | becasue it builds on the utility rather than repo containers, so the header is missing | 14:52 |
| jrosser | but as to why this does not happen in CI, i'm not sure | 14:53 |
| jrosser | perhaps the wheel is in the cache in zuul | 14:53 |
| noonedeadpunk | uh.... | 14:54 |
| noonedeadpunk | I just thought that in case of wheel build failure it should fallback just to package build | 14:55 |
| noonedeadpunk | was didn't really pay attention to that part | 14:55 |
| noonedeadpunk | but we probably should delegate it | 14:56 |
| *** spatel has quit IRC | 15:09 | |
| *** kukacz has joined #openstack-ansible | 15:11 | |
| noonedeadpunk | why zun can't just merge.... | 15:24 |
| *** spatel has joined #openstack-ansible | 15:28 | |
| *** rohit02 has joined #openstack-ansible | 15:35 | |
| *** rohit02 has quit IRC | 15:49 | |
| spatel | jrosser: How did you generated hash password for autoinstall Ubuntu 20.04 ? | 15:55 |
| spatel | i had command earlier but i think i lost my notes | 15:56 |
| spatel | I think i can copy paste one from /etc/shadow same cipher | 15:57 |
| *** macz_ has joined #openstack-ansible | 16:04 | |
| *** mgariepy has quit IRC | 16:04 | |
| *** kukacz has quit IRC | 16:20 | |
| *** jawad_axd has quit IRC | 16:23 | |
| jrosser | spatel: yes i'm sure thats where the one in my example would have come from | 16:39 |
| *** mgariepy has joined #openstack-ansible | 16:41 | |
| spatel | jrosser: if i don't specify storage section then does ubuntu create automatically disk partition? | 16:41 |
| spatel | I am trying to create universal autoinstall file so it can fit on any kind of disk size (i don't want to specify partition size) | 16:43 |
| *** kukacz has joined #openstack-ansible | 16:43 | |
| mgariepy | spatel i thing the last partition will take all the remaining space | 16:46 |
| spatel | something strange going on i am getting this error - https://ibb.co/pPtpDHM | 16:48 |
| spatel | No disk space left | 16:48 |
| mgariepy | how do you boot ? | 16:49 |
| spatel | PXE boot | 16:49 |
| spatel | mgariepy: here is my pxe file - http://paste.openstack.org/show/801093/ | 16:50 |
| spatel | I have 50G disk and trying to create two partition 1G - /boot 15G - / | 16:50 |
| spatel | what is this filesystem for - disk-sda, size: 1M, flag: bios_grub | 16:51 |
| noonedeadpunk | folks, ansible question :) If I already running play with become (ie connect to VM as user "ubuntu") should I be able for specific task set become_user to something different? It feels like ansible does not expect doing double become... | 16:51 |
| mgariepy | spatel, that's the user-data what is the pxe options ? | 16:53 |
| spatel | noonedeadpunk: i think yes task specific you can be different become | 16:54 |
| mgariepy | i have something like: imgargs vmlinuz root=/dev/ram0 ramdisk_size=1500000 i | 16:55 |
| mgariepy | via ipxe | 16:55 |
| noonedeadpunk | well, I get ` Operation not permitted\nchown: changing ownership of '/tmp/ansible-tmp-1608137696.4008777-598087-244490161738678/AnsiballZ_command.py': Operation not permitted\n})` :( | 16:55 |
| spatel | mgariepy: here its TFTP config - http://paste.openstack.org/show/801095/ | 16:55 |
| noonedeadpunk | I belive the reson might be that I login as user ubuntu, then become root for all tasks, and for specific task I want to become user 'mysql'... | 16:56 |
| mgariepy | how much ram ? | 16:56 |
| spatel | 2GB | 16:56 |
| mgariepy | i guess it's not enough.. | 16:57 |
| mgariepy | can you bump it to 4G to see ? | 16:57 |
| spatel | hmm.. this is vmware box to test kickstart | 16:57 |
| noonedeadpunk | but https://docs.ansible.com/ansible/latest/user_guide/become.html#risks-of-becoming-an-unprivileged-user says it's possible.... uh... | 16:58 |
| spatel | I built many centOS 8.x VM using 2GB memory | 16:58 |
| mgariepy | yep. | 16:58 |
| mgariepy | sure but. | 16:58 |
| mgariepy | this is not centos.. | 16:58 |
| spatel | look like ubuntu asking more.. | 16:58 |
| mgariepy | ;p | 16:58 |
| spatel | let me try 4G | 16:58 |
| mgariepy | the installer do download the entire iso.. | 16:58 |
| mgariepy | twice if i'm not mistaken .. ;p | 16:58 |
| spatel | can i handover folder instead of ISO ? | 16:59 |
| spatel | mgariepy: look like 4G passed that stage | 17:01 |
| spatel | noonedeadpunk: does that user has permission to become_user? | 17:02 |
| spatel | believe me you are more expert in ansible than me :) | 17:03 |
| noonedeadpunk | yep, I eventually create another user with it, and then I want to run command as newly created user... | 17:03 |
| mgariepy | anything needs review ? | 17:04 |
| mgariepy | i can give a push on some patches today and tomorrow before end of year vacation. | 17:05 |
| spatel | didn't you see anything in logs? try to set temporary dir and give world r/w permission (just gessing) | 17:05 |
| spatel | mgariepy: Thanks for the help | 17:05 |
| mgariepy | i don't know if it's possible to boot it from nfs or something else. | 17:06 |
| spatel | mgariepy: so best option is ISO right? anyway i will try something and see if possible because its double work for installer to take ISO and then mount etc.. | 17:07 |
| *** gyee has joined #openstack-ansible | 17:07 | |
| spatel | it would be great if http:// directly serve files. | 17:08 |
| *** rpittau is now known as rpittau|afk | 17:11 | |
| noonedeadpunk | spatel: I had to define `allow_world_readable_tmpfiles` | 17:30 |
| spatel | cool | 17:30 |
| *** kukacz has quit IRC | 17:40 | |
| noonedeadpunk | uh, or just install `acl` on dest host... | 17:40 |
| *** kukacz has joined #openstack-ansible | 18:07 | |
| jrosser | spatel: you were asking about what the first bios_grub partition was for | 18:18 |
| jrosser | it's because the disk layout in my example is two drives mirror | 18:18 |
| spatel | yes 1M ? | 18:18 |
| jrosser | small partition on the front not mirrored for booting and bios to understand | 18:18 |
| jrosser | the rest of the disk mirrored | 18:18 |
| jrosser | mirrored with md, then lvm volume on top of that | 18:19 |
| spatel | let me understand 1M is for bios_grub and then 1G is for /boot | 18:19 |
| jrosser | which the bios/booting will not understand | 18:19 |
| spatel | may be UEFI using 1M size | 18:19 |
| jrosser | /boot is a partition on the md device | 18:20 |
| jrosser | the rest of the md device is lvm | 18:20 |
| jrosser | i think you probably want a diagram if trying to visualise this | 18:21 |
| jrosser | chunk on the front of both disks for grub -> raid0 with md -> /boot on 1G of the md -> rest of the md as lvm | 18:22 |
| jrosser | lvm then divided into root / var / swap / split | 18:22 |
| jrosser | split is "the rest of the disk" which we have an ansible role which comes along and makes anything specific out of that spare space if needed | 18:23 |
| openstackgerrit | Merged openstack/openstack-ansible stable/train: Make upgrade jobs get role relevant versions https://review.opendev.org/c/openstack/openstack-ansible/+/767048 | 18:23 |
| jrosser | spatel: this is quite complicated tbh, but mirrored boot/root disks seem to be awkward with software raid | 18:24 |
| jrosser | and i just don't like bios raid at all, so would rather never use that | 18:24 |
| spatel | I have hardware raid controller so i never used software raid | 18:24 |
| *** maharg101 has quit IRC | 18:24 | |
| jrosser | oh right so you can get rid of most of my example then | 18:24 |
| spatel | I did that but just had confusion about that 1M partition but anyway i am keeping it, look like ubuntu need that | 18:25 |
| jrosser | but the syntax in the lvm bit will be similar for your 1G /boot 15G /root then a '-1' for 'whatever is left' | 18:25 |
| spatel | jrosser: first time its very confusing but now everything make sense how this storage section layout | 18:27 |
| spatel | jrosser: question when you running POST script are you using /target filesystem or plain bash script? | 18:28 |
| spatel | ubuntu doc saying you should do curtin in-target --target /target | 18:28 |
| *** dpaclt has joined #openstack-ansible | 18:47 | |
| admin0 | this is the right channel for this dpaclt | 18:47 |
| dpaclt | Thanks for pointing out | 18:47 |
| dpaclt | Hi All | 18:48 |
| kleini | grub became to large to fit into the master boot record of the msdos partition table. therefore bios_grub partition is necessary where grub stores its majority of its code. only a small loader for grub in bios_grub partition is placed into master boot record | 18:48 |
| dpaclt | please | 18:48 |
| spatel | kleini: Thanks got it, so it use that 1M partition to load first stage of grub and then pull rest of the world from /boot | 18:55 |
| spatel | I heard that BIOS can only read first sector of disk to load bootloader and that is where this design coming from. | 18:56 |
| kleini | not exactly. first stage of grub is in master boot record, second stage in bios_grub partition. configuration, modules, themes and fonts then in /boot/grub | 19:03 |
| spatel | master boot record located on that 1M partition? | 19:08 |
| spatel | should i keep Snapd on ubuntu server for OSA or remove it? trying to clean up all unwanted packages and services | 19:08 |
| kleini | https://wiki.archlinux.org/index.php/GRUB#BIOS_systems <- this one clearly describes that bios_grub is only necessary for a BIOS booting a GPT partitioned disc. it is not required for msdos partitioned discs | 19:20 |
| *** nsmeds has quit IRC | 19:20 | |
| kleini | So even I did not exactly know, what was going on. | 19:20 |
| *** kukacz has quit IRC | 19:21 | |
| *** andrewbonney has quit IRC | 19:26 | |
| jrosser | spatel: here is a late_command script like used in my user_data example http://paste.openstack.org/show/801098/ | 19:38 |
| spatel | Thank you, so its plain shell script | 19:40 |
| *** pto has quit IRC | 19:41 | |
| jrosser | well, from the original it is wget then executed as a shell script | 19:41 |
| jrosser | so it's been our choice to do it that way | 19:42 |
| spatel | jrosser: i like that | 19:43 |
| jrosser | late_commands is literally commands, iirc in the old preseed stuff on bionic you could only do one command there | 19:43 |
| *** kukacz has joined #openstack-ansible | 19:44 | |
| spatel | I was looking at these example - https://gist.github.com/s3rj1k/55b10cd20f31542046018fcce32f103e | 19:44 |
| jrosser | so you needed some funky one-liner to wget then execute all in one go, when you had a ton of stuff to do in late_command | 19:44 |
| spatel | - curtin in-target --target=/target -- apt-get clean | 19:44 |
| spatel | I will sure go with wget/script way | 19:44 |
| *** kukacz has quit IRC | 19:45 | |
| * jrosser not even sure really what curtin is | 19:46 | |
| spatel | My ubuntu is ready and seeing bunch of stuff running on top output. trying to understand what i can remove like snapd etc.. | 19:46 |
| spatel | i want very small footprint | 19:46 |
| jrosser | but really what i showed you is the smallest change we made to get our old preseed stuff working with autoinstall | 19:46 |
| spatel | I am doing it script way now. | 19:46 |
| *** pto has joined #openstack-ansible | 19:49 | |
| spatel | now trying to configure bonding in ubuntu | 19:49 |
| spatel | jrosser: do you prefer systemd-networkd supported resolver.conf or configure manually? | 19:53 |
| spatel | i meant systemd-resolved.service | 19:54 |
| admin0 | in one deployment, new one, rabbit always fails with this: https://gist.github.com/a1git/643c5a18e7c427c8c7bb57ae94423fe7 | 19:54 |
| admin0 | someone knows what exact this is ? | 19:54 |
| spatel | assuming no issue with node connectivity, they can talk to each other | 19:56 |
| admin0 | there is no issue with connectivity | 19:56 |
| admin0 | iperf gives a 9gbps between them | 19:56 |
| spatel | pick one node and run command manually and check logs | 19:56 |
| admin0 | i have destroyed the containers, deleted the inventory.json so that they get diff ips .. | 19:57 |
| admin0 | but everytime, it gets stuck on c2 on this exact issue | 19:57 |
| spatel | hmm that is very odd, can you ping using name ? | 19:57 |
| spatel | ping c1-rabbit-mq-container-a30243a0 | 19:57 |
| admin0 | spatel, https://gist.github.com/a1git/18467f943f2a000bcb2a1e09d2735e39 | 20:00 |
| admin0 | from one container to another ? | 20:00 |
| admin0 | one container can ping the other just fine | 20:01 |
| admin0 | using name/ip | 20:01 |
| spatel | yes each container maintain /etc/hosts file for name resolution | 20:02 |
| spatel | maintained* | 20:02 |
| spatel | so they can ping each other using name not IP (i had issue about name resolution in past so i always make sure /etc/hosts is correct) | 20:03 |
| *** kukacz has joined #openstack-ansible | 20:17 | |
| *** maharg101 has joined #openstack-ansible | 20:21 | |
| *** maharg101 has quit IRC | 20:25 | |
| *** Martouf has joined #openstack-ansible | 20:29 | |
| admin0 | i am going to re-do the whole bootstrap ..delete stuff and re-clone again | 20:43 |
| *** rfolco has quit IRC | 20:56 | |
| spatel | did you find something or just go to re-build it | 21:16 |
| admin0 | i am doing a rebuild now | 21:17 |
| admin0 | not the OS but everything else, including bootstrapping it | 21:17 |
| admin0 | will try 21.1.0 insead of 21.2.0 -- though i already setup like 10x 21.2.0 since it came out | 21:17 |
| admin0 | all on prod and never got this issue | 21:18 |
| spatel | 21.x is ussuri right? | 21:19 |
| admin0 | yep | 21:20 |
| spatel | or may be bad version of rabbitmq but then CI-CD job also fail | 21:21 |
| jrosser | admin0: really all this swapping branches will not just magically fix rabbitmq | 21:21 |
| admin0 | :D | 21:21 |
| jrosser | there is nothing changed with rabbit for the whole ussuri release | 21:21 |
| *** dpaclt has quit IRC | 21:30 | |
| spatel | admin0: what is wrong here - http://paste.openstack.org/show/801102/ | 21:36 |
| admin0 | space | 21:39 |
| spatel | hmm | 21:39 |
| admin0 | the vlans and bonds and bridges and ethernet should all be in the same spacing | 21:39 |
| spatel | let me check | 21:40 |
| spatel | is this looks good - http://paste.openstack.org/show/801103/ | 21:43 |
| spatel | still not working | 21:43 |
| admin0 | yu give a name to vlan | 21:52 |
| admin0 | not bond0.64 | 21:52 |
| admin0 | that conflicts with it | 21:52 |
| admin0 | just call it vlan64 | 21:52 |
| admin0 | or v64 | 21:52 |
| admin0 | after netplan apply, do cat /proc/net/vlan/config and see if its up and activated .. or you might have something else or multiple 64 | 21:53 |
| admin0 | if its done then it will show vlan64 bond0 64 | 21:53 |
| spatel | https://bugs.launchpad.net/netplan/+bug/1781203 | 21:54 |
| openstack | Launchpad bug 1746419 in nplan (Ubuntu) "duplicate for #1781203 bond parameters are not changed by 'netplan apply'" [Undecided,Confirmed] | 21:54 |
| spatel | its not working i tried all spacing etc.. | 21:54 |
| spatel | i did change bond0.64 to vlan.64 | 21:55 |
| spatel | admin0: ^ | 21:55 |
| admin0 | why do you need a . ? | 21:55 |
| admin0 | what is wrong with just vlan64 :) | 21:55 |
| spatel | removed (.) and still same error - bond0: reverting custom parameters for bridges and bonds is not supported | 21:56 |
| spatel | look like issue is something else.. how to debug netplan config to see what is doing behind the screen | 21:57 |
| spatel | i used 2 space instead of 4 space (hope that is not problem | 21:57 |
| admin0 | what does netplan generate give ? | 21:58 |
| admin0 | same error ? | 21:58 |
| spatel | it was my bad, my interface name was wrong :) | 22:10 |
| spatel | thanks admin0 | 22:10 |
| spatel | everything looks good | 22:10 |
| admin0 | \o/ | 22:10 |
| spatel | netplan rocks :) | 22:12 |
| admin0 | lol | 22:12 |
| spatel | Thank you and now i have to go.. making dinner | 22:12 |
| spatel | see you tomorrow! | 22:12 |
| *** mensis has quit IRC | 22:16 | |
| *** spatel has quit IRC | 22:28 | |
| *** spatel has joined #openstack-ansible | 22:30 | |
| *** spatel has quit IRC | 22:34 | |
| *** spatel has joined #openstack-ansible | 22:40 | |
| *** spatel has quit IRC | 22:43 | |
| openstackgerrit | Dmitriy Rabotyagov proposed openstack/openstack-ansible master: Remove duplicated defenition of tempest_run_concurrency https://review.opendev.org/c/openstack/openstack-ansible/+/767430 | 22:59 |
| *** kukacz has quit IRC | 23:02 | |
| *** jbadiapa has quit IRC | 23:14 | |
| *** kukacz has joined #openstack-ansible | 23:30 | |
| *** stee has joined #openstack-ansible | 23:48 | |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!