*** Sravan has joined #openstack-kolla | 00:11 | |
*** Sravan has quit IRC | 00:15 | |
*** igordc has quit IRC | 00:49 | |
*** cgrosjean has quit IRC | 00:49 | |
*** schwicht has joined #openstack-kolla | 00:57 | |
*** zhanglong has joined #openstack-kolla | 01:13 | |
*** happyhemant has quit IRC | 01:46 | |
*** unicell has joined #openstack-kolla | 01:49 | |
*** schwicht has quit IRC | 01:49 | |
*** lei-zh has joined #openstack-kolla | 01:54 | |
*** BjoernT_ has joined #openstack-kolla | 01:59 | |
*** BjoernT has quit IRC | 02:02 | |
*** lei-zh has quit IRC | 02:05 | |
*** lei-zh has joined #openstack-kolla | 02:06 | |
openstackgerrit | gugug proposed openstack/kolla-ansible master: Fix the genconfig subcommand failed https://review.opendev.org/668773 | 02:08 |
---|---|---|
openstackgerrit | gugug proposed openstack/kolla-ansible master: Fix the genconfig subcommand failed when enable_ceph https://review.opendev.org/668773 | 02:15 |
*** iclon_ has joined #openstack-kolla | 02:47 | |
*** iclon has quit IRC | 02:50 | |
*** BjoernT_ has quit IRC | 03:04 | |
openstackgerrit | Merged openstack/kolla-ansible stable/stein: Use become for all docker tasks https://review.opendev.org/666140 | 03:30 |
openstackgerrit | Merged openstack/kolla-ansible master: Cloudkitty InfluxDB Storage backend via Kolla-ansible https://review.opendev.org/658418 | 03:45 |
*** lei-zh1 has joined #openstack-kolla | 03:58 | |
*** lei-zh has quit IRC | 04:01 | |
*** skramaja has joined #openstack-kolla | 04:07 | |
*** Sravan has joined #openstack-kolla | 04:29 | |
*** shyamb has joined #openstack-kolla | 04:33 | |
*** whoami-rajat has joined #openstack-kolla | 04:55 | |
*** altlogbot_0 has quit IRC | 04:57 | |
*** altlogbot_3 has joined #openstack-kolla | 04:58 | |
*** shyamb has quit IRC | 05:05 | |
*** shyamb has joined #openstack-kolla | 05:05 | |
openstackgerrit | gugug proposed openstack/kolla-ansible master: Fix the genconfig subcommand failed when enable_ceph https://review.opendev.org/668773 | 05:16 |
*** Sravan has quit IRC | 05:21 | |
*** Luzi has joined #openstack-kolla | 05:29 | |
*** lei-zh1 has quit IRC | 05:32 | |
*** iclon__ has joined #openstack-kolla | 05:46 | |
*** iclon_ has quit IRC | 05:48 | |
*** luksky11 has joined #openstack-kolla | 05:50 | |
*** cah_link has joined #openstack-kolla | 06:13 | |
*** factor has joined #openstack-kolla | 06:16 | |
*** shyamb has quit IRC | 06:24 | |
*** shyamb has joined #openstack-kolla | 06:35 | |
yoctozepto | morning | 06:43 |
*** Luzi has quit IRC | 06:55 | |
*** dciabrin has joined #openstack-kolla | 06:55 | |
*** ivve has joined #openstack-kolla | 06:57 | |
*** Luzi has joined #openstack-kolla | 06:57 | |
openstackgerrit | wangwei proposed openstack/kolla-ansible master: Support mon and osd to be named with hostname https://review.opendev.org/654417 | 07:01 |
*** luksky11 has quit IRC | 07:02 | |
*** jaicaa has quit IRC | 07:08 | |
*** jaicaa has joined #openstack-kolla | 07:11 | |
*** shyamb has quit IRC | 07:17 | |
*** hamzaachi has joined #openstack-kolla | 07:22 | |
*** rpittau|afk is now known as rpittau | 07:23 | |
*** unicell has quit IRC | 07:25 | |
*** zhanglong has quit IRC | 07:26 | |
*** shyamb has joined #openstack-kolla | 07:27 | |
*** zhanglong has joined #openstack-kolla | 07:27 | |
*** pcaruana has joined #openstack-kolla | 07:28 | |
*** zhanglong has quit IRC | 07:32 | |
*** shyamb has quit IRC | 07:33 | |
*** zhanglong has joined #openstack-kolla | 07:33 | |
openstackgerrit | Merged openstack/kolla master: CI: set the same gate queue for kolla and kolla-ansible https://review.opendev.org/668971 | 07:39 |
openstackgerrit | Merged openstack/kolla-ansible master: CI: Keep stderr in ansible logs https://review.opendev.org/668739 | 07:45 |
*** heikkine has joined #openstack-kolla | 07:46 | |
openstackgerrit | zhulingjie proposed openstack/kolla-cli master: Fixing test failures https://review.opendev.org/657086 | 07:47 |
openstackgerrit | Radosław Piliszek proposed openstack/kolla-ansible stable/stein: CI: Keep stderr in ansible logs https://review.opendev.org/669086 | 07:48 |
mnasiadka | morning | 07:50 |
ohwhyosa[m] | Morning! | 07:53 |
openstackgerrit | zhulingjie proposed openstack/kolla-cli master: Sync Sphinx requirement https://review.opendev.org/577636 | 07:53 |
*** luksky11 has joined #openstack-kolla | 07:54 | |
*** happyhemant has joined #openstack-kolla | 08:16 | |
mgoddard | morning | 08:18 |
*** shyamb has joined #openstack-kolla | 08:26 | |
*** pcaruana has quit IRC | 08:28 | |
*** zhanglong has quit IRC | 08:28 | |
*** zhanglong has joined #openstack-kolla | 08:31 | |
*** k_mouza has joined #openstack-kolla | 08:31 | |
openstackgerrit | Mark Goddard proposed openstack/kolla-ansible master: Wait for all compute services before cell discovery https://review.opendev.org/668623 | 08:32 |
*** Wasaac has joined #openstack-kolla | 08:34 | |
*** cgrosjean has joined #openstack-kolla | 08:34 | |
openstackgerrit | Mark Goddard proposed openstack/kolla stable/stein: Fixes ceph-osd deploys with loop devices. https://review.opendev.org/668291 | 08:35 |
openstackgerrit | Mark Goddard proposed openstack/kolla stable/rocky: Fixes ceph-osd deploys with loop devices. https://review.opendev.org/668286 | 08:35 |
yoctozepto | http://zuul.openstack.org/status | 08:39 |
yoctozepto | ^ k, gating works - one queue | 08:39 |
yoctozepto | I will propose backports to the supported branches | 08:39 |
openstackgerrit | Radosław Piliszek proposed openstack/kolla stable/stein: CI: set the same gate queue for kolla and kolla-ansible https://review.opendev.org/669103 | 08:40 |
openstackgerrit | Radosław Piliszek proposed openstack/kolla stable/rocky: CI: set the same gate queue for kolla and kolla-ansible https://review.opendev.org/669104 | 08:41 |
openstackgerrit | Radosław Piliszek proposed openstack/kolla stable/queens: CI: set the same gate queue for kolla and kolla-ansible https://review.opendev.org/669105 | 08:41 |
openstackgerrit | Radosław Piliszek proposed openstack/kolla-ansible stable/stein: CI: set the same gate queue for kolla and kolla-ansible https://review.opendev.org/669106 | 08:41 |
openstackgerrit | Radosław Piliszek proposed openstack/kolla-ansible stable/rocky: CI: set the same gate queue for kolla and kolla-ansible https://review.opendev.org/669108 | 08:41 |
yoctozepto | hmm, queens has no gating | 08:45 |
*** k_mouza has quit IRC | 08:45 | |
yoctozepto | ;D | 08:45 |
mgoddard | yoctozepto: nope | 08:47 |
*** k_mouza has joined #openstack-kolla | 08:48 | |
*** Wasaac has quit IRC | 08:53 | |
openstackgerrit | Merged openstack/kolla-cli master: Fixing test failures https://review.opendev.org/657086 | 08:56 |
*** shyamb has quit IRC | 08:57 | |
*** shyamb has joined #openstack-kolla | 08:57 | |
*** Wasaac has joined #openstack-kolla | 08:59 | |
yoctozepto | wow, kolla-cli activity | 08:59 |
*** Wasaac has quit IRC | 09:00 | |
*** Wasaac has joined #openstack-kolla | 09:02 | |
*** pcaruana has joined #openstack-kolla | 09:04 | |
mnasiadka | yeah, saw some movement in there | 09:21 |
mnasiadka | mgoddard: so did anyone volunteer to pick it up? | 09:21 |
mgoddard | mnasiadka: nope | 09:21 |
mgoddard | I asked oracle folks and they agreed to drop it | 09:21 |
mgoddard | we haven't made that change yet though, need to read how to do it | 09:21 |
*** shyamb has quit IRC | 09:24 | |
mgoddard | mnasiadka: weird ceph error during stein upgrade job: http://logs.openstack.org/14/668514/3/check/kolla-ansible-centos-source-upgrade-ceph-3/c527c65/primary/logs/ansible/upgrade | 09:33 |
mgoddard | ceph_argparse.JsonFormat: unknown type CephBool | 09:33 |
*** owalsh_ is now known as owalsh | 09:34 | |
mnasiadka | yeah, see that, hmm | 09:35 |
mnasiadka | https://tracker.ceph.com/issues/39355 | 09:36 |
yoctozepto | so we need to wait for all daemons to reregister with their new release | 09:37 |
yoctozepto | great | 09:38 |
yoctozepto | ;D | 09:38 |
yoctozepto | also "rc": 0, | 09:38 |
yoctozepto | aah, because of cut | 09:38 |
yoctozepto | though I thought bash pipeline should break in this case | 09:39 |
mnasiadka | we can force running the command on the first upgraded mon, with -m | 09:40 |
mnasiadka | and we can use some jinja filter instead of cut :) | 09:41 |
yoctozepto | $ false | true | 09:42 |
yoctozepto | $ echo $? | 09:42 |
yoctozepto | 0 | 09:42 |
yoctozepto | all those years in ignorance | 09:42 |
mnasiadka | ignorance is a bliss | 09:42 |
yoctozepto | indeed, indeed | 09:43 |
yoctozepto | > <mnasiadka> we can force running the command on the first upgraded mon, with -m | 09:45 |
yoctozepto | yup, we redirect to the first container but we don't force ceph command to use it | 09:45 |
yoctozepto | good catch | 09:46 |
mgoddard | set -o pipefail | 09:47 |
yoctozepto | mgoddard: felt like it was the default | 09:49 |
yoctozepto | anyway, you were right about stein being the most tested release | 09:49 |
yoctozepto | so many ceph upgrade jobs run ;D | 09:49 |
mgoddard | that would be far too sensible | 09:49 |
*** JangwonLee has quit IRC | 09:55 | |
*** Wasaac has quit IRC | 09:56 | |
*** Wasaac has joined #openstack-kolla | 09:56 | |
*** shyamb has joined #openstack-kolla | 10:01 | |
*** shyamb has quit IRC | 10:09 | |
*** shyamb has joined #openstack-kolla | 10:09 | |
*** Wasaac has quit IRC | 10:10 | |
*** pcaruana has quit IRC | 10:12 | |
openstackgerrit | Mark Goddard proposed openstack/kolla-ansible stable/stein: Wait for mariadb to stop after shutdown https://review.opendev.org/667363 | 10:12 |
*** zhanglong has quit IRC | 10:16 | |
*** skramaja has quit IRC | 10:16 | |
openstackgerrit | Mark Goddard proposed openstack/kolla-ansible stable/stein: DNM: Try importing stop.yml in handlers https://review.opendev.org/669131 | 10:17 |
openstackgerrit | Merged openstack/kolla-ansible stable/stein: Check for 'become' in tasks that use Docker in pep8 https://review.opendev.org/666141 | 10:18 |
openstackgerrit | Merged openstack/kolla stable/pike: Move pike jobs to use validated RDO Trunk https://review.opendev.org/668522 | 10:18 |
openstackgerrit | Merged openstack/kolla-ansible master: Don't rotate keystone fernet keys during deploy https://review.opendev.org/666882 | 10:18 |
openstackgerrit | Radosław Piliszek proposed openstack/kolla-ansible stable/stein: Don't rotate keystone fernet keys during deploy https://review.opendev.org/669132 | 10:20 |
yoctozepto | mgoddard: maybe we should test ironic whenever nova code is changed in kolla-ansible, hmm? | 10:22 |
mgoddard | yoctozepto: makes sense | 10:22 |
*** Dubla has quit IRC | 10:22 | |
yoctozepto | mgoddard: ok, proposing | 10:22 |
openstackgerrit | Radosław Piliszek proposed openstack/kolla-ansible master: WIP: Test ironic also when nova role is modified https://review.opendev.org/669136 | 10:26 |
openstackgerrit | Radosław Piliszek proposed openstack/kolla-ansible master: WIP: Test ironic also when nova role is modified https://review.opendev.org/669136 | 10:26 |
openstackgerrit | Mark Goddard proposed openstack/kolla-ansible master: CI: Pull images before upgrade https://review.opendev.org/666881 | 10:34 |
mgoddard | mnasiadka: if you have a minute, https://review.opendev.org/#/c/647699/ | 10:35 |
mnasiadka | mgoddard: yeah, was just going through the latest changes in that | 10:35 |
mgoddard | thanks | 10:36 |
*** Dubla has joined #openstack-kolla | 10:37 | |
openstackgerrit | caoyuan proposed openstack/kolla-cli master: Sync Sphinx requirement https://review.opendev.org/577636 | 10:37 |
mnasiadka | finished ;) | 10:37 |
openstackgerrit | caoyuan proposed openstack/kolla-cli master: Update hacking version https://review.opendev.org/627351 | 10:37 |
openstackgerrit | Mark Goddard proposed openstack/kolla-ansible stable/queens: Restart all nova services after upgrade https://review.opendev.org/667937 | 10:37 |
openstackgerrit | Mark Goddard proposed openstack/kolla-ansible master: Fixes for MariaDB recovery https://review.opendev.org/667904 | 10:46 |
openstackgerrit | Mark Goddard proposed openstack/kolla-ansible master: Test MariaDB operations in CI https://review.opendev.org/655663 | 10:46 |
*** Wasaac has joined #openstack-kolla | 10:48 | |
*** Wasaac has quit IRC | 10:53 | |
*** shyamb has quit IRC | 10:55 | |
*** shyamb has joined #openstack-kolla | 10:56 | |
yoctozepto | mgoddard: seen that? something strange in rocky | 11:01 |
yoctozepto | lots of issues unrelated to patch | 11:02 |
yoctozepto | and nova is failing on waiting for up | 11:02 |
openstackgerrit | Merged openstack/kolla-cli master: Update hacking version https://review.opendev.org/627351 | 11:10 |
*** Wasaac has joined #openstack-kolla | 11:21 | |
openstackgerrit | Mark Goddard proposed openstack/kolla-ansible stable/stein: Wait for mariadb to stop after shutdown https://review.opendev.org/667363 | 11:22 |
*** Wasaac has quit IRC | 11:26 | |
*** Wasaac has joined #openstack-kolla | 11:31 | |
*** shyamb has quit IRC | 11:31 | |
mnasiadka | so guys - what's the approach with checking ceph versions and setting require-osd-ver? It would be best to run it after all MONs and OSDs are upgraded - now it's not the case | 11:34 |
mnasiadka | maybe we should just document steps that should be done after Ceph upgrade? if somebody chooses to upgrade half of Ceph MONs and OSDs - it's not the best idea to enforce higher Ceph version functionalities... | 11:36 |
openstackgerrit | Merged openstack/kolla-ansible stable/stein: CI: set the same gate queue for kolla and kolla-ansible https://review.opendev.org/669106 | 11:40 |
*** shyamb has joined #openstack-kolla | 11:42 | |
openstackgerrit | Merged openstack/kolla-ansible stable/rocky: CI: set the same gate queue for kolla and kolla-ansible https://review.opendev.org/669108 | 11:44 |
*** cah_link has quit IRC | 11:46 | |
yoctozepto | mgoddard: replace still broken "\\1-ironic" | 11:48 |
yoctozepto | could it be that there is still too much escaping | 11:49 |
*** skramaja has joined #openstack-kolla | 11:49 | |
yoctozepto | {{ hosts | map('regex_replace', '^(.*)$', 'https://\\1') | list }} | 11:51 |
yoctozepto | it's an example from the ansible docs | 11:51 |
openstackgerrit | Rafael Weingärtner proposed openstack/kolla-ansible master: Add ability to kolla-ansible to switch between fetchers backend https://review.opendev.org/659607 | 11:51 |
yoctozepto | (groups['nova-compute-ironic'] | | 11:51 |
yoctozepto | intersect(ansible_play_batch) | | 11:51 |
yoctozepto | map('extract', hostvars, 'ansible_hostname') | | 11:51 |
yoctozepto | map('regex_replace', '^(.*)$', '\\1-ironic') | | 11:51 |
yoctozepto | list) | 11:51 |
yoctozepto | ^ this is what you're doing | 11:51 |
yoctozepto | looks legit | 11:52 |
yoctozepto | but nonetheless does not work | 11:52 |
*** cah_link has joined #openstack-kolla | 11:54 | |
*** Wasaac has quit IRC | 11:56 | |
openstackgerrit | Merged openstack/kolla stable/stein: Extract CI vars for use in kolla-ansible jobs https://review.opendev.org/668815 | 11:56 |
*** cah_link has quit IRC | 11:57 | |
*** cah_link has joined #openstack-kolla | 11:57 | |
openstackgerrit | caoyuan proposed openstack/kolla-cli master: Sync Sphinx requirement https://review.opendev.org/577636 | 11:58 |
yoctozepto | in another place we have supported_policy_files: "{{ supported_policy_format_list | map('regex_replace', '(.*)', '{{ project_name }}_\\1') | list }}" | 12:00 |
yoctozepto | which works | 12:00 |
yoctozepto | so it seems as if doing '-' after number is problematic for him | 12:01 |
openstackgerrit | caoyuan proposed openstack/kolla-cli master: change the description of the class https://review.opendev.org/621883 | 12:02 |
openstackgerrit | jacky06 proposed openstack/kolla-cli master: Replace git.openstack.org URLs with opendev.org URLs https://review.opendev.org/654678 | 12:07 |
*** Wasaac has joined #openstack-kolla | 12:08 | |
openstackgerrit | caoyuan proposed openstack/kolla-cli master: Change openstack-dev to openstack-discuss https://review.opendev.org/621950 | 12:09 |
*** henriqueof has joined #openstack-kolla | 12:09 | |
*** shyamb has quit IRC | 12:16 | |
*** shyamb has joined #openstack-kolla | 12:22 | |
openstackgerrit | jacky06 proposed openstack/kolla-cli master: Replace git.openstack.org URLs with opendev.org URLs https://review.opendev.org/654678 | 12:32 |
*** schwicht has joined #openstack-kolla | 12:32 | |
openstackgerrit | jacky06 proposed openstack/kolla-cli master: update the tox minversion to 2.0 https://review.opendev.org/626225 | 12:34 |
*** shyamb has quit IRC | 12:37 | |
yoctozepto | > <yoctozepto> so it seems as if doing '-' after number is problematic for him | 12:40 |
yoctozepto | just tested - wrong | 12:40 |
yoctozepto | but I proved that in this particular context you have to do just \1 | 12:40 |
yoctozepto | seems like ansible bug | 12:40 |
openstackgerrit | Merged openstack/kolla-cli master: Change openstack-dev to openstack-discuss https://review.opendev.org/621950 | 12:40 |
yoctozepto | they have fixed the \\\\ case before 2.0 | 12:40 |
yoctozepto | and docs state it should always be \\1 | 12:41 |
yoctozepto | yet it is not | 12:41 |
yoctozepto | odd | 12:41 |
openstackgerrit | Merged openstack/kolla-cli master: change the description of the class https://review.opendev.org/621883 | 12:43 |
openstackgerrit | Radosław Piliszek proposed openstack/kolla-ansible master: CI: Test ironic also when nova role is modified https://review.opendev.org/669136 | 12:46 |
mgoddard | yoctozepto: just tested - you are right about \1. It's because we're using a yaml literal block scalar (>-) | 13:00 |
mgoddard | just did some testing | 13:00 |
*** Luzi has quit IRC | 13:01 | |
openstackgerrit | Mark Goddard proposed openstack/kolla-ansible master: Wait for all compute services before cell discovery https://review.opendev.org/668623 | 13:03 |
openstackgerrit | Merged openstack/kolla-cli master: Replace git.openstack.org URLs with opendev.org URLs https://review.opendev.org/654678 | 13:06 |
mgoddard | yoctozepto: https://github.com/markgoddard/ansible-experiments/tree/master/03-regex-replace | 13:09 |
*** skramaja has quit IRC | 13:10 | |
*** iclon__ has quit IRC | 13:13 | |
*** iclon has joined #openstack-kolla | 13:14 | |
*** fxpester has joined #openstack-kolla | 13:16 | |
fxpester | hi all | 13:16 |
fxpester | got a strange problem, deploy kolla queens, everything ok and running | 13:17 |
*** iclon has quit IRC | 13:20 | |
*** iclon has joined #openstack-kolla | 13:20 | |
*** whoami-rajat has quit IRC | 13:23 | |
mgoddard | fxpester: that doesn't sound like a problem :) | 13:25 |
mnasiadka | mgoddard: seen my comment on ceph earlier? Need some guidance :) | 13:29 |
*** henriqueof has quit IRC | 13:33 | |
mgoddard | mnasiadka: missed that | 13:35 |
mgoddard | we run require-osd-release at the end of the upgrade, not sure what's wrong? | 13:35 |
mgoddard | is the problem that we don't wait for them to initialise? | 13:36 |
fxpester | do second kolla reconfigure or kolla deploy , it is doing containers restart - and some files are disappearing - for sure it is "paste" configs like keystone-paste.conf | 13:38 |
fxpester | no problems with kolla configs that deployed by dump_start | 13:39 |
fxpester | reproduced on redhat 7 with docker 17.09.1-ce and 18.09.6 | 13:40 |
mgoddard | fxpester: do you have some error logs to share? | 13:42 |
fxpester | 2019-07-04 16:22:15.958 7 CRITICAL nova [-] Unhandled error: ConfigNotFound: Could not find config at /etc/nova/api-paste.ini | 13:45 |
mgoddard | fxpester: does the container image contain that file? | 13:48 |
*** fxpester has quit IRC | 13:48 | |
mgoddard | which container is that? | 13:48 |
*** fxpester has joined #openstack-kolla | 13:48 | |
fxpester | yes files present in images if I run `docker run -ti $image_ID bash` I can see them | 13:49 |
*** iclon has quit IRC | 13:50 | |
fxpester | I guess it is docker problem, but surprised I can find any information about this in google | 13:50 |
*** iclon has joined #openstack-kolla | 13:50 | |
mgoddard | strange | 13:53 |
mgoddard | is it a permissions issue? | 13:53 |
mgoddard | centos queens nova-compute image has this: | 13:53 |
mgoddard | -rw-r----- 1 root nova 2923 Mar 24 23:12 /etc/nova/api-paste.ini | 13:53 |
mgoddard | so nova group should be able to read | 13:54 |
fxpester | trying to reproduce now, just did a fresh install so - | 13:56 |
fxpester | ad9a14526614 kolla/centos-binary-nova-api:queens "dumb-init --singl..." 4 minutes ago Up 4 minutes nova_api | 13:56 |
*** fxpester has quit IRC | 14:00 | |
*** iclon has quit IRC | 14:01 | |
*** fxpester has joined #openstack-kolla | 14:01 | |
openstackgerrit | Mark Goddard proposed openstack/kolla-ansible stable/stein: Wait for mariadb to stop after shutdown https://review.opendev.org/667363 | 14:02 |
mgoddard | ^ ready to go | 14:03 |
fxpester | so ok, I just did docker stop nova-api and docker start, now nova - 2019-07-04 17:01:15.407 7 ERROR nova ConfigNotFound: Could not find config at /etc/nova/api-paste.ini | 14:03 |
*** iclon has joined #openstack-kolla | 14:03 | |
fxpester | ad9a14526614 kolla/centos-binary-nova-api:queens "dumb-init --singl..." 11 minutes ago Restarting (1) 55 seconds ago nova_api | 14:03 |
openstackgerrit | Mark Goddard proposed openstack/kolla-ansible master: Test MariaDB operations in CI https://review.opendev.org/655663 | 14:06 |
*** hamzaachi has quit IRC | 14:07 | |
mgoddard | fxpester: can you docker export nova_api and check the file? | 14:08 |
*** hamzaachi has joined #openstack-kolla | 14:09 | |
*** schwicht has quit IRC | 14:10 | |
fxpester | mgoddard: nova-paste.ini is not present in tar archive | 14:11 |
mgoddard | fxpester: :/ so where has it gone? | 14:11 |
fxpester | trying to debug this right now, it disapper with docker restart | 14:11 |
fxpester | *docker container restart* | 14:12 |
mgoddard | fxpester: try running the commands in a new container manually using 'docker run' | 14:13 |
fxpester | inside keystone container right now: -rw-r----- 1 root keystone 2493 Nov 1 2018 keystone-paste.ini | 14:13 |
fxpester | right now doing keystone restart | 14:13 |
fxpester | did docker start and file just disappear, no old keystone config files present only new generated by kolla | 14:14 |
fxpester | drwx------ 2 keystone keystone 6 Jul 4 17:13 domains -rw------- 1 keystone keystone 645 Jul 4 17:13 keystone.conf | 14:15 |
*** luksky11 has quit IRC | 14:17 | |
yoctozepto | > <mgoddard> yoctozepto: https://github.com/markgoddard/ansible-experiments/tree/master/03-regex-replace | 14:18 |
yoctozepto | yeah, testing the same | 14:18 |
yoctozepto | each day wiser ;-) | 14:18 |
*** fxpester has quit IRC | 14:19 | |
openstackgerrit | Merged openstack/kolla-ansible stable/stein: Don't rotate keystone fernet keys during deploy https://review.opendev.org/669132 | 14:19 |
yoctozepto | mgoddard, mnasiadka: if -m would work, then we should just use it, we already delegate to one container but do not point the ceph client to it exclusively | 14:19 |
yoctozepto | another issue is the incomplete upgrade | 14:20 |
mgoddard | yoctozepto: what can we do about the incomplete upgrade? | 14:22 |
yoctozepto | mgoddard: we can check all versions as previously suggested, wait some retries to increase version, and bail out if they do not ;-) | 14:23 |
*** BjoernT has joined #openstack-kolla | 14:23 | |
mgoddard | sounds reasonable | 14:23 |
mgoddard | is it a release blocker? | 14:23 |
yoctozepto | I doubt, but let mnasiadka confirm it | 14:25 |
yoctozepto | most ceph upgrades pass now happily | 14:25 |
mgoddard | we need to get this ceph support deprecated | 14:25 |
*** fxpester has joined #openstack-kolla | 14:26 | |
yoctozepto | we do, it's against the kolla spirit | 14:26 |
mgoddard | too late to depreate for stein? :) | 14:26 |
yoctozepto | I think we can declare it deprecated as long as we support it | 14:26 |
yoctozepto | in stein still | 14:26 |
yoctozepto | but not in train | 14:27 |
yoctozepto | and provide migration path | 14:27 |
yoctozepto | to external ceph + ceph-ansible | 14:27 |
*** BjoernT_ has joined #openstack-kolla | 14:27 | |
yoctozepto | or similar | 14:27 |
mgoddard | that's the dream | 14:27 |
*** BjoernT has quit IRC | 14:27 | |
openstackgerrit | Christian Berendt proposed openstack/kolla-ansible master: Add parameters to configure number of processes and threads of WSGI in the Horizon container https://review.opendev.org/669193 | 14:30 |
*** henriqueof has joined #openstack-kolla | 14:30 | |
*** hamzaachi_ has joined #openstack-kolla | 14:32 | |
*** hamzaachi has quit IRC | 14:32 | |
*** fxpester has quit IRC | 14:33 | |
*** goldyfruit has quit IRC | 14:42 | |
*** goldyfruit has joined #openstack-kolla | 14:42 | |
*** ivve has quit IRC | 14:51 | |
*** luksky11 has joined #openstack-kolla | 14:59 | |
openstackgerrit | Christian Berendt proposed openstack/kolla-ansible master: Prevent access to server-statu from Horizon https://review.opendev.org/669204 | 15:19 |
*** cah_link has quit IRC | 15:21 | |
openstackgerrit | Christian Berendt proposed openstack/kolla-ansible master: Add parameters to configure number of processes and threads of horizon https://review.opendev.org/669193 | 15:24 |
openstackgerrit | Christian Berendt proposed openstack/kolla-ansible master: Prevent access to server-status from horizon https://review.opendev.org/669204 | 15:24 |
*** frawg12 has joined #openstack-kolla | 15:26 | |
*** Wasaac has quit IRC | 15:29 | |
*** Wasaac has joined #openstack-kolla | 15:30 | |
*** priteau has joined #openstack-kolla | 15:30 | |
*** frawg12 has quit IRC | 15:32 | |
*** frawg12 has joined #openstack-kolla | 15:36 | |
*** Dubla has quit IRC | 15:38 | |
*** hamzaachi_ has quit IRC | 15:43 | |
*** hamzaachi_ has joined #openstack-kolla | 15:45 | |
*** whoami-rajat has joined #openstack-kolla | 15:47 | |
*** BjoernT_ has quit IRC | 15:51 | |
frawg12 | Hi, I'm looking for some support to fix my kolla-ansible testbed deployment | 15:55 |
frawg12 | I'm deploying a multinode setup over 3 VMs running on an OVH server | 15:55 |
frawg12 | so with bridges with fixed IP/Mac for public access | 15:55 |
frawg12 | I can't even access the dashboard/APIs from the public IP (but I can SSH to it to access the controler) | 15:56 |
frawg12 | the dashboard is accessible from the private network but can't get it to work from public | 15:56 |
frawg12 | and I'm trying without haproxy | 15:57 |
mgoddard | frawg12: perhaps you need to open up a port? | 15:58 |
frawg12 | it seems that on the control nodes iptables is not blocking anything | 15:59 |
mloza | in stable/stein, credentials-keys directory of keystone container of my 3 controller nodes has different set of keys with throws an error 'Credential could not be decrypted. Please contact the administrator: InvalidToken' | 16:00 |
mloza | Should it be same across 3 controller nodes? | 16:00 |
mgoddard | frawg12: but what about in OVH? security groups? | 16:06 |
frawg12 | It's on a dedicate server, so nothing is blocked | 16:06 |
frawg12 | and I've deployed KVM/libvirt on top of it | 16:07 |
frawg12 | and can access all ports from services running inside VMs | 16:07 |
yoctozepto | frawg12: inaccessible in what way? | 16:08 |
yoctozepto | horizon is a django app and it probably dislikes being addressed using unknown name (foreign ip address) | 16:08 |
frawg12 | wget or web browser access to port 80 on the public IP get no page | 16:08 |
yoctozepto | if it connects, and tells you it doesn't want to work, then it might be this | 16:08 |
yoctozepto | but is this no page or connection timeout/reject? | 16:09 |
frawg12 | nope | 16:09 |
frawg12 | wget: | 16:09 |
frawg12 | Connecting to 54.37.216.22:80... connected. | 16:09 |
frawg12 | HTTP request sent, awaiting response... No data received. | 16:09 |
frawg12 | Retrying. | 16:09 |
frawg12 | oops, put the real IP | 16:09 |
frawg12 | and that public IP leads me appropriately (with ssh) to the control node | 16:11 |
yoctozepto | try configuring ALLOWED_HOSTS in horizon local settings | 16:11 |
yoctozepto | though I get connection refused with this ip address | 16:12 |
yoctozepto | could be some ovh firewall | 16:12 |
yoctozepto | ssh answers | 16:12 |
yoctozepto | http not | 16:12 |
frawg12 | yes no TLS for the time being | 16:12 |
frawg12 | (one problem at a time) | 16:12 |
frawg12 | https://docs.openstack.org/kolla-ansible/rocky/reference/horizon-guide.html | 16:12 |
yoctozepto | I did not say a thing about tls :D | 16:12 |
frawg12 | for ALLOWED_HOSTS? | 16:13 |
yoctozepto | yup but it seems the port is not listening | 16:13 |
yoctozepto | so it's not this | 16:13 |
frawg12 | the VM have multiple netowrk interfaces, one private/internal, one public on the OVH bridge, and another one on the OVH bridge for neutron | 16:14 |
*** rpittau is now known as rpittau|afk | 16:14 | |
frawg12 | and looking at the domain names/IP in openstack endpoint list the public URL are correct | 16:14 |
yoctozepto | check ip address horizon is listening at | 16:15 |
yoctozepto | maybe it's listening on the private one only? | 16:15 |
*** Sravan has joined #openstack-kolla | 16:17 | |
frawg12 | not sure where to get that info | 16:17 |
frawg12 | for sure I can access it on the private, I'm logged on it now | 16:17 |
frawg12 | OK, I'm loooking inside the running horizon container to see what I can find | 16:21 |
*** Sravan has quit IRC | 16:21 | |
*** henriqueof has quit IRC | 16:21 | |
frawg12 | so /etc/httpd/conf.d/horizon.conf | 16:22 |
frawg12 | contains only a Listen on the private IP | 16:22 |
ohwhyosa[m] | and on local_settings / local_settings.py? | 16:28 |
*** Wasaac has quit IRC | 16:28 | |
ohwhyosa[m] | If you are using kolla I believe it's exposed on /etc/kolla/horizon/local_settings | 16:29 |
*** Wasaac has joined #openstack-kolla | 16:30 | |
ohwhyosa[m] | Which reminds me I have to check again if kolla-ansible reconfigure on rocky with centos images responds to changes on settings.py | 16:30 |
ohwhyosa[m] | local_settings, sorry | 16:30 |
ohwhyosa[m] | Hey people, how would you go about adding a service to an already deployed openstack kolla-ansible? | 16:31 |
frawg12 | OPENSTACK_HOST and OPENSTACK_KEYSTONE_URL point to the private IP | 16:32 |
ohwhyosa[m] | Just add the service on globals and deploy again or something else? | 16:32 |
ohwhyosa[m] | And is there any ALLOWED_HOSTS? | 16:35 |
frawg12 | yes: ALLOWED_HOSTS = ['*'] | 16:36 |
ohwhyosa[m] | lemme check for a sec | 16:36 |
ohwhyosa[m] | Yep, look, this is an example file | 16:36 |
ohwhyosa[m] | https://github.com/openstack/horizon/blob/master/openstack_dashboard/local/local_settings.py.example | 16:36 |
ohwhyosa[m] | lines 36-40 | 16:36 |
ohwhyosa[m] | hmmm | 16:36 |
frawg12 | yes, my conf seems OK (at least for this line) | 16:37 |
ohwhyosa[m] | okay, try | 16:38 |
ohwhyosa[m] | ss -uptan | grep 80 | 16:38 |
ohwhyosa[m] | no 443, right? | 16:38 |
ohwhyosa[m] | I mean, you have it listening with http, no tls enabled, frawg12 ? | 16:39 |
frawg12 | tcp LISTEN 0 128 192.168.100.22:80 *:* users:(("httpd",pid=7,fd=3)) | 16:39 |
frawg12 | yes no TLS / 443 | 16:39 |
ohwhyosa[m] | Hmmm there you have it, the apache httpd process is binding only on the private ip | 16:40 |
ohwhyosa[m] | could you pastebin your httpd.conf? | 16:40 |
frawg12 | sure | 16:41 |
frawg12 | https://pastebin.com/b6Ee5gW5 | 16:43 |
frawg12 | /etc/httpd/conf.d/horizon.conf | 16:43 |
frawg12 | again only the private IP | 16:43 |
*** unicell has joined #openstack-kolla | 16:44 | |
ohwhyosa[m] | Do you want to open it to the world? | 16:44 |
ohwhyosa[m] | You could Listen on 0.0.0.0:80 (though it may be too much) | 16:44 |
ohwhyosa[m] | and restart httpd | 16:44 |
ohwhyosa[m] | ? | 16:44 |
ohwhyosa[m] | could you paste also the globals.yml? | 16:44 |
frawg12 | it looks a bit hackish to do this kind of thing, would think that ansible-kolla should provide required things | 16:45 |
ohwhyosa[m] | pastebin, I mean, be careful or you can get banned on IRC | 16:45 |
frawg12 | I think I'm just doing something wrong with the network conf | 16:45 |
frawg12 | sure, will post/clean it | 16:45 |
ohwhyosa[m] | I do have a fun story regarding ASCII Art, Unicorns and an office wide ban | 16:45 |
ohwhyosa[m] | Cool | 16:45 |
frawg12 | https://pastebin.com/eSdAaBAt | 16:49 |
frawg12 | nothing very fancy, I tried to use priv + pub networks, and haproxy disabled (as I was having troubles with the VIP address stuff) | 16:51 |
*** k_mouza_ has joined #openstack-kolla | 16:55 | |
ohwhyosa[m] | is it an AIO? | 16:55 |
ohwhyosa[m] | Yup, right? | 16:56 |
frawg12 | no, I'm having 3 nodes: control, compute and storage | 16:56 |
frawg12 | (3 different VMs) | 16:56 |
frawg12 | (and a foruth deploy host) | 16:57 |
*** k_mouza has quit IRC | 16:59 | |
frawg12 | https://github.com/openstack/kolla-ansible/blob/stable/rocky/ansible/roles/horizon/templates/horizon.conf.j2#L2 | 16:59 |
*** k_mouza_ has quit IRC | 17:00 | |
frawg12 | si it seems that this would always use the api_interface_address, that is configured to be the private one | 17:00 |
ohwhyosa[m] | yup, probably it is the haproxy the one that balances that | 17:02 |
frawg12 | as said I'm not able to get haproxy conf to pass | 17:02 |
ohwhyosa[m] | I had understood that disabling haproxy and using the host network was only for AIO deployments? | 17:02 |
ohwhyosa[m] | don't remember where I read that | 17:02 |
frawg12 | not clear in fact | 17:03 |
frawg12 | but my problem is that with haproxy I'm not sure how it should operate | 17:03 |
ohwhyosa[m] | gimme a sec | 17:03 |
ohwhyosa[m] | Could you meanwhile ss -uptan | grep your_pub_ip | 17:03 |
ohwhyosa[m] | ? | 17:03 |
frawg12 | nothing from inside the horizon container | 17:04 |
ohwhyosa[m] | from the external host | 17:05 |
ohwhyosa[m] | meaning | 17:05 |
ohwhyosa[m] | the host itself | 17:05 |
frawg12 | for haproxy I need to have kolla_external_vip_interface be a public IP, and for this to work I need to have a predefined mac | 17:05 |
ohwhyosa[m] | The comment I mean, on the pastebin you shared, is right above kolla_internal_vip address | 17:05 |
yoctozepto | api_interface should be public one because it is assumed external clients will be using it | 17:07 |
yoctozepto | otherwise it is for private use | 17:07 |
frawg12 | nothing from the host | 17:07 |
yoctozepto | and can be accessed via vpn | 17:07 |
yoctozepto | if you set api_interface to the internal one, then only on the internal network will the services work | 17:07 |
frawg12 | so with this it may work without haproxy | 17:08 |
frawg12 | so either haproxy + private and pub IPs, or no haproxy but only pub IP | 17:08 |
yoctozepto | haproxy can be omitted when there is only one controller | 17:08 |
yoctozepto | no, it's orthogonal to public/private | 17:08 |
*** cgrosjean has quit IRC | 17:09 | |
ohwhyosa[m] | <yoctozepto "haproxy can be omitted when ther"> Makes sense | 17:10 |
ohwhyosa[m] | Anycase, the horizon dashboard should be accessible from the outside even if the rest of the apis aren't, right yoctozepto ? | 17:11 |
frawg12 | ok so only setting api_interface = eth2 # (my public interface) | 17:11 |
frawg12 | may be enough? | 17:11 |
ohwhyosa[m] | well, apis | 17:11 |
ohwhyosa[m] | Exposing all the api endpoints seems a bit much, depending on the use case (like, I may want my "customers" to be able to reach their horizon dashboard and stuff like console, but never to be able to reach my api endpoints and control/operate the cloud itself) | 17:14 |
ohwhyosa[m] | I'm quite propably 80% wrong at least, though | 17:14 |
*** icarusfactor has joined #openstack-kolla | 17:16 | |
*** unicell has quit IRC | 17:16 | |
*** henriqueof has joined #openstack-kolla | 17:18 | |
*** factor has quit IRC | 17:19 | |
*** k_mouza has joined #openstack-kolla | 17:19 | |
*** icarusfactor has quit IRC | 17:21 | |
*** frawg12 has quit IRC | 17:24 | |
*** frawg12 has joined #openstack-kolla | 17:24 | |
*** k_mouza has quit IRC | 17:24 | |
frawg12 | was disconnected sorry, not sur if I missed something | 17:24 |
mnasiadka | mgoddard: Currently we do ceph release check on each host in serial mode, so after host1 ceph upgrade, we run it. It would make sense to run it only after all hosts are upgraded, or do something like „kolla-ansible ceph-post-upgrade”, run some checks, do what needs to be done and output some report. | 17:24 |
yoctozepto | ohwhyosa[m]: control/operate is already controlled by authorization mechanisms, no need to add complexity :-) | 17:28 |
frawg12 | ohwhyosa[m]: I will make some more test with what you said and also try to get haproxy work with the network conf I try to use | 17:29 |
frawg12 | ohwhyosa[m]: thanks! | 17:29 |
*** luksky11 has quit IRC | 17:33 | |
*** ivve has joined #openstack-kolla | 17:36 | |
mgoddard | mnasiadka: could you point me to that serial release check? | 17:39 |
ohwhyosa[m] | frawg12: Nah, thank the rest of the people around here, just trying to pay forward! | 17:40 |
mnasiadka | mgoddard: bottom of upgrade.yml, the one that yoctozepto fixed, it’s going serial (which makes sense in ceph upgrade) - but not the require-osd version stuff :) | 17:40 |
ohwhyosa[m] | yoctozepto: Makes sense, though there's something about having them out in the open that makes the lil sysadmin in my shake in terror | 17:41 |
ohwhyosa[m] | in me* | 17:41 |
mgoddard | mnasiadka: I don't see how it's serial. It has run_once and delegate_to | 17:41 |
ohwhyosa[m] | Could always firewall it and restrict ips, though | 17:41 |
yoctozepto | mnasiadka, mgoddard: me neither, where should I look? | 17:42 |
yoctozepto | ohwhyosa[m]: sure, layers of security usually help | 17:42 |
mnasiadka | mgoddard: still it somehow runs after only first host upgrade, not all of them - or I misread ara logs :) | 17:42 |
yoctozepto | or at least make us feel more secure | 17:42 |
mgoddard | mnasiadka: hmm, perhaps there was an earlier failure that prevented all hosts from executing? | 17:43 |
mnasiadka | mgoddard: might be, will look into that tomorrow | 17:44 |
mgoddard | mnasiadka: looked at the logs again, that is what happened. Mariadb failed earlier | 17:46 |
mgoddard | we need any_errors_fatal (if only it worked) | 17:46 |
henriqueof | Is kolla-ansibe based ironic deployment fully funciontal? | 17:47 |
henriqueof | I've been trying for more than amont to boot a single server but it won't work. | 17:47 |
mgoddard | henriqueof: we use ironic with kolla-ansible a lot | 17:50 |
mgoddard | henriqueof: we use centos/binary, and centos/source should be fine. Haven't tried with ubuntu | 17:51 |
henriqueof | mgoddard: I've been trying for more than a month ot boot a remote server using IPMI but it doesn't work. | 17:51 |
mgoddard | yoctozepto: mnasiadka: what would you think about a last minute deprecation of ceph deployment in Stein? | 17:51 |
mgoddard | henriqueof: yeah it's not trivial | 17:52 |
henriqueof | novcompute-ironic says "because one or more parameters are missing from its instance_info and insufficent information is present to boot" | 17:52 |
mgoddard | henriqueof: which OS/type? | 17:52 |
mgoddard | henriqueof: does it say which parameters? | 17:52 |
henriqueof | I am using centos images. | 17:53 |
mgoddard | henriqueof: ironic conductor logs are often more helpful than nova logs | 17:53 |
henriqueof | mgoddard: Yep but the params are set like the docs except image_source. | 17:53 |
henriqueof | Ironic conductor logs are way less helpful, there's only one line when it starts and one when it fails, no aditiuonal info. | 17:54 |
henriqueof | I think I am going crazy with this. hahah | 17:55 |
mgoddard | henriqueof: try enabling debug? | 17:55 |
mgoddard | henriqueof: if you want an example of a working setup, try following https://kayobe.readthedocs.io/en/latest/development/automated.html#overcloud | 17:57 |
mgoddard | henriqueof: you can use the tenks project to create some 'fake bare metal VMs' | 17:57 |
mgoddard | henriqueof: kayobe uses kolla-ansible underneath, so the config can be easily transferred | 17:58 |
yoctozepto | mgoddard: already expressed my posture - deprecate now, remove right in train but support in stein at fullest so we need to fix it anyway :P | 17:58 |
henriqueof | mgoddard: Will take a look, thanks! | 17:58 |
yoctozepto | (well, unless nobody cares) | 17:58 |
mgoddard | yoctozepto: sorry, forgot. I figure if we know we're going to remove one day we might as well make it known as early as possible, even if we don't do it any sooner | 17:59 |
*** priteau has quit IRC | 17:59 | |
yoctozepto | mgoddard: you read my mind | 18:00 |
*** Wasaac has quit IRC | 18:06 | |
openstackgerrit | Mark Goddard proposed openstack/kolla-ansible master: Deprecate Ceph deployment https://review.opendev.org/669214 | 18:06 |
mgoddard | yoctozepto: ^ | 18:07 |
openstackgerrit | Mark Goddard proposed openstack/kolla-ansible master: CI: Pull images before upgrade https://review.opendev.org/666881 | 18:11 |
yoctozepto | mgoddard: seen | 18:16 |
*** hamzaachi_ has quit IRC | 18:16 | |
yoctozepto | I would add that ceph, or any storage cluster, deployment is orthogonal to openstack | 18:18 |
mnasiadka | mgoddard: deprecation ... I don’t know, let me test some ceph scenarios and let’s see :) | 18:19 |
mgoddard | ok, we don't need to rush into it | 18:19 |
*** KeithMnemonic has quit IRC | 18:30 | |
kplant | mgoddard: not that my opinion carries weight here but i don't think it's irresponsible throwing out the fyi notices asap | 18:32 |
kplant | a new-to-kolla-ansible user may see the notice and decide to choose a different deployment method for ceph and save themselves from being forced off of kolla deployed ceph | 18:33 |
yoctozepto | kplant: that's the point | 18:34 |
kplant | :-) | 18:35 |
mgoddard | kplant: of course your opinion carries weight! | 18:40 |
*** Wasaac has joined #openstack-kolla | 18:43 | |
*** whoami-rajat has quit IRC | 18:47 | |
*** Wasaac has quit IRC | 18:48 | |
*** whoami-rajat has joined #openstack-kolla | 19:07 | |
*** hamzaachi has joined #openstack-kolla | 19:36 | |
*** frawg12 has quit IRC | 19:40 | |
*** EmilienM is now known as EvilienM | 19:54 | |
*** luksky11 has joined #openstack-kolla | 19:55 | |
*** EvilienM is now known as EmilienM | 19:56 | |
*** dciabrin has quit IRC | 20:09 | |
*** Wasaac has joined #openstack-kolla | 20:43 | |
*** ivve has quit IRC | 20:47 | |
*** Wasaac has quit IRC | 20:48 | |
henriqueof | mgoddard: still no luck with Ironic. | 20:52 |
henriqueof | I have an IPMI interface connected to the openstack managemente network and a pxe interface connected to the providser netowrk, cleaning works but provisioning don't. | 20:52 |
henriqueof | What can be the problem? | 20:52 |
*** Wasaac has joined #openstack-kolla | 21:04 | |
*** Wasaac has quit IRC | 21:10 | |
*** whoami-rajat has quit IRC | 21:27 | |
openstackgerrit | Merged openstack/kolla-ansible master: Simplify handler conditionals https://review.opendev.org/647699 | 21:34 |
*** cgrosjean has joined #openstack-kolla | 21:35 | |
*** dciabrin has joined #openstack-kolla | 21:39 | |
*** henriqueof has quit IRC | 21:45 | |
*** hamzaachi has quit IRC | 21:48 | |
*** cgrosjean has quit IRC | 21:56 | |
*** Dubla has joined #openstack-kolla | 22:03 | |
*** Wasaac has joined #openstack-kolla | 22:10 | |
*** hamzaachi has joined #openstack-kolla | 22:10 | |
*** Wasaac has quit IRC | 22:14 | |
*** luksky11 has quit IRC | 22:43 | |
*** k_mouza has joined #openstack-kolla | 22:58 | |
*** k_mouza has quit IRC | 23:02 | |
*** k_mouza has joined #openstack-kolla | 23:09 | |
*** hamzaachi has quit IRC | 23:11 | |
*** jbadiapa has quit IRC | 23:31 |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!