*** k_mouza has joined #openstack-kolla | 00:21 | |
*** pbing19 has quit IRC | 00:37 | |
*** zhanglong has joined #openstack-kolla | 01:14 | |
*** cah_link has quit IRC | 01:34 | |
*** k_mouza has quit IRC | 01:36 | |
*** noxoid has quit IRC | 02:11 | |
*** k_mouza has joined #openstack-kolla | 03:37 | |
*** k_mouza has quit IRC | 03:41 | |
*** skramaja has joined #openstack-kolla | 03:48 | |
*** ykarel|away is now known as ykarel | 03:54 | |
*** skramaja has quit IRC | 03:58 | |
*** skramaja_ has joined #openstack-kolla | 03:58 | |
*** kcnomad has quit IRC | 04:01 | |
*** Tony31 has quit IRC | 04:22 | |
*** evrardjp has quit IRC | 04:36 | |
*** evrardjp has joined #openstack-kolla | 04:37 | |
openstackgerrit | James Kirsch proposed openstack/kolla-ansible master: Add support for encrypting backend Keystone HAProxy traffic https://review.opendev.org/713986 | 05:01 |
---|---|---|
*** JangwonLee_ has joined #openstack-kolla | 05:51 | |
*** JangwonLee__ has quit IRC | 05:54 | |
*** JangwonLee__ has joined #openstack-kolla | 05:55 | |
*** JangwonLee_ has quit IRC | 05:59 | |
*** Tony31 has joined #openstack-kolla | 06:08 | |
Tony31 | morning | 06:09 |
*** bjolo has joined #openstack-kolla | 06:17 | |
*** srini_ has joined #openstack-kolla | 06:18 | |
srini_ | morning Tony31 | 06:18 |
Tony31 | Morning srini_ how are you doing? | 06:19 |
Tony31 | Hope you and your family are keeping well | 06:20 |
srini_ | I am doing fine, Yes everyone is good. thanks for asking | 06:21 |
*** igordc has quit IRC | 06:21 | |
Tony31 | good to hear srini_ | 06:25 |
srini_ | Tony31, quick update, we are really considering running control-planes in Ovirt https://access.redhat.com/documentation/en-us/red_hat_openstack_platform/13/html/director_installation_and_usage/creating-virtualized-control-planes | 06:31 |
srini_ | Tony31, Thanks for the suggestion :) | 06:32 |
Tony31 | Just checking this link | 06:35 |
Tony31 | a few months ago I Was looking for ovirt / openstack integrations | 06:35 |
osmanlicilegi | morning | 06:35 |
Tony31 | morning osmanlicilegi | 06:35 |
osmanlicilegi | how do you do folks? | 06:36 |
mnasiadka | morning | 06:38 |
Tony31 | morning mnasiadka | 06:38 |
Tony31 | All good osmanlicilegi | 06:38 |
Tony31 | @srini_ - thanks for the link, I am doing this at the moment. I have a task in my list to try and run a performance optimised VM. For the new openstack I am working on, I've not done that though but I do have 3 x controllers | 06:40 |
Tony31 | if you run haproxy on ovirt, you need to set the network to have no filter. By default it does not allow mac spoof and because of the floating virtual ip and virtual mac address for kolla / kayobe haproxy, it does not work unless you configure RHV | 06:41 |
Tony31 | otherwise, all good :) | 06:41 |
srini_ | Tony31, thanks for the tip, :) | 06:42 |
Tony31 | :) | 06:46 |
Tony31 | I have an issue with `kolla_internal_tls_cert:` the error during the deploy is cert validation failed. I am trying to use the same cert for the internal API as well as the external endpoint - is it possible? | 06:57 |
Tony31 | the cert is signed by godaddy and I can access the external / horizon using https without any issue or error. But when I try to apply the same to the internal api i get a validation error. | 06:58 |
Tony31 | thought of asking here in case you guys already ran into the same issue ? | 06:58 |
yoctozepto | Tony31: does the internal fqdn agree? I would suspect name validation | 07:01 |
yoctozepto | morning folks | 07:01 |
yoctozepto | bline: any and all help is appreciated | 07:02 |
Tony31 | yoctozepto - it's a wildcard cert *.domain.com internal api is stack-int.domain.com and external is stack.domain.com | 07:03 |
Tony31 | so it should be working as I understand it. I previously used *.domain.com for jira-staging.domain.com and the "-staging" didnt cause any validation issue | 07:04 |
*** hjensas|afk is now known as hjensas | 07:05 | |
*** mrunge_ is now known as mrunge | 07:05 | |
*** bengates has joined #openstack-kolla | 07:11 | |
*** cah_link has joined #openstack-kolla | 07:16 | |
*** bengates has quit IRC | 07:16 | |
*** bengates has joined #openstack-kolla | 07:17 | |
*** ykarel is now known as ykarel|afk | 07:21 | |
yoctozepto | Tony31: ack, makes sense; is there nothing more about that validation failure? no hints? | 07:30 |
Tony31 | snippet: | 07:37 |
Tony31 | 'auth_url is correct. %s' % e)\nkeystoneauth1.exceptions.discovery.DiscoveryFailure: Could not find versioned identity endpoints when attempting to authenticate. Please check that your auth_url is correct. SSL exception connecting to https://kayostack-int.domain.com:35357: HTTPSConnectionPool(host='kayostack-int.domain.com', port=35357): Max retries exceeded with url: / (Caused by SSLError(SSLError(\"bad handshake: Error([('SSL routines', | 07:37 |
Tony31 | 'tls_process_server_certificate', 'certificate verify failed')],)\",),))\n", "module_stdout": "", "msg": "MODULE FAILURE\nSee stdout/stderr for the exact error", "rc": 1} | 07:37 |
*** ykarel|afk is now known as ykarel | 07:37 | |
Tony31 | im going to re-deploy without a `-` in the dns and see if it makes a difference, since there is no `-` in the external dns name and it is working. it's the only difference I can see | 07:38 |
openstackgerrit | Pierre Riteau proposed openstack/kayobe stable/queens: Install python-openstackclient using upper constraints https://review.opendev.org/717208 | 07:44 |
*** jbadiapa has joined #openstack-kolla | 07:45 | |
*** ktibi has joined #openstack-kolla | 07:47 | |
*** cah_link has quit IRC | 07:51 | |
*** ykarel is now known as ykarel|afk | 07:55 | |
*** rpittau|afk is now known as rpittau | 07:56 | |
*** JangwonLee__ has quit IRC | 08:06 | |
*** JangwonLee__ has joined #openstack-kolla | 08:06 | |
*** ykarel|afk is now known as ykarel|lunch | 08:09 | |
mgoddard | Tony31: hi. Internal TLS is a much bigger change than external TLS, which is why it was only recently added | 08:16 |
mgoddard | Tony31: with external TLS, only external API consumers need to use TLS | 08:16 |
mgoddard | Tony31: for internal TLS, all the openstack services communicate with each other using TLS | 08:16 |
mgoddard | Tony31: the python requests library can be quite selective in where it will accept CA certs | 08:17 |
mgoddard | Tony31: there have been quite a few changes in master for internal TLS. Have you set openstack_cacert? | 08:18 |
*** cah_link has joined #openstack-kolla | 08:21 | |
mgoddard | yoctozepto: this is the link that fails in the monasca-grafana build: http://mirror.ord.rax.opendev.org:8080/registry.npmjs/@types%2fcolor-name | 08:21 |
mgoddard | looks like it is still 404 | 08:21 |
Tony31 | hi mgoddard thanks for this mate. I've not set "openstack_cacert" | 08:22 |
Tony31 | . I just done a search for that and it's found in venv/ and src. Is this something I need to manually add somewhere in the kayobe config? | 08:22 |
mgoddard | Tony31: it should be set in the kolla globals.yml file - for kayobe this is etc/kayobe/kolla/globals.yml in kayobe-config | 08:23 |
mgoddard | Tony31: we thought it would not be required for publicly signed certs, but probably it was not tested | 08:23 |
*** scottsol has joined #openstack-kolla | 08:27 | |
cosmicsound | hi | 08:28 |
cosmicsound | is someone using gocchi source in grafana, what auth method is best or working? | 08:29 |
Tony31 | mgoddard the godaddy ca cert should be trusted by the system by the local CA cert store as it's a trusted cert authority. The "openstack_cacert" is not in kolla/globals.yml but i could add it and point it to the same as `kolla_internal_fqdn_cacert:` | 08:29 |
mgoddard | Tony31: I'd suggest it's worth a try | 08:33 |
mgoddard | where is that error coming from btw? | 08:33 |
Tony31 | mgoddard it's during the reconfigure deployment | 08:45 |
mgoddard | Tony31: ok | 08:45 |
Tony31 | mgoddard I've just completed a fresh deployment once again. I'll try adding the cert config back again now and re-test, because I've also changed the dns name. If it still fails then i'll add the openstack_cacert and post back :) Thanks for this | 08:45 |
*** rmart04 has joined #openstack-kolla | 08:49 | |
openstackgerrit | Mark Goddard proposed openstack/kolla-ansible stable/stein: CI: Use upper constraints to install clients https://review.opendev.org/717259 | 08:52 |
yoctozepto | mgoddard: yeah, I'm wondering why servers still haven't reconfigured | 08:54 |
mgoddard | yoctozepto: are you sure they have not? | 08:54 |
yoctozepto | mgoddard: sorry, they did, we both looked at old failure | 08:55 |
*** pbing19 has joined #openstack-kolla | 08:55 | |
yoctozepto | mgoddard: see https://a28720641abd1053931d-69746c322fe5e3cf8e2976f654b96d92.ssl.cf5.rackcdn.com/717075/1/check/kolla-build-centos8-binary/9e67bb6/ | 08:55 |
yoctozepto | mgoddard: no scaring me! | 08:55 |
yoctozepto | mgoddard: ruling this issue out | 08:56 |
mgoddard | yoctozepto: cool so monasca-grafana issue solved? | 08:56 |
*** hrw has quit IRC | 08:57 | |
*** hrw has joined #openstack-kolla | 08:57 | |
*** k_mouza has joined #openstack-kolla | 08:59 | |
yoctozepto | mgoddard: yeah, master is green | 09:00 |
yoctozepto | mgoddard: overall status: amber | 09:00 |
hrw | morgen | 09:00 |
yoctozepto | mgoddard: merge me those unittest.mock before ci decides to go red again | 09:00 |
mgoddard | hi hrw | 09:00 |
yoctozepto | hrw: buongiorno, signore | 09:01 |
hrw | yoctozepto: приве́т! | 09:01 |
yoctozepto | hrw: хахаха | 09:01 |
*** sorin-mihai_ has joined #openstack-kolla | 09:02 | |
openstackgerrit | Mark Goddard proposed openstack/kolla stable/stein: Fix multiple CI issues https://review.opendev.org/717008 | 09:02 |
yoctozepto | mgoddard: ganesha needs some care | 09:02 |
mgoddard | yoctozepto: I think I just patched the wrong line | 09:03 |
mgoddard | yoctozepto: there are two centos mirror seds in stein & earlier, cherry pick got the wrong one | 09:03 |
*** sorin-mihai has quit IRC | 09:03 | |
*** sorin-mihai__ has joined #openstack-kolla | 09:04 | |
mrunge | hi, at a quick glance, I couldn't find where the replacement of repo urls for centos is done | 09:05 |
mrunge | any pointers? | 09:05 |
hrw | argh. | 09:05 |
mrunge | there is still the issue, that additional repos are using a mirrorlist which is disliked by ci | 09:05 |
*** sorin-mihai_ has quit IRC | 09:07 | |
mnasiadka | mrunge: tests/templates/template-override.j2 in kolla | 09:07 |
mnasiadka | mrunge: well, one repo - the collectd one :) | 09:07 |
openstackgerrit | Mark Goddard proposed openstack/kolla-ansible stable/rocky: CI: Use upper constraints to install clients https://review.opendev.org/717262 | 09:07 |
mrunge | mnasiadka, are the tests being executed before build? | 09:07 |
yoctozepto | mgoddard: ok | 09:07 |
mnasiadka | mrunge: no, it's just a file location, not really a test :) | 09:08 |
mnasiadka | yoctozepto, mgoddard: ganesha had a problem because it was https://mirror.centos.org, and I thought I fixed it? | 09:08 |
mgoddard | mnasiadka: the patch wasn't merged, and didn't propoerly apply to stein & earlier | 09:09 |
mnasiadka | well, that it's not easily backportable was obvious | 09:09 |
mnasiadka | I wish we had a better mechanism for setting mirrors :) | 09:09 |
hrw | one week without any changes in master... | 09:10 |
openstackgerrit | Mark Goddard proposed openstack/kolla stable/rocky: Fix multiple CI issues https://review.opendev.org/717024 | 09:12 |
hrw | mgoddard: can we get RP+2 on important patches? | 09:12 |
mgoddard | hrw: ok | 09:13 |
mrunge | sigh, the replacement in tests/templates/template-override.j2 works with the repo, do we have a stale mirror again? | 09:13 |
hrw | I do not follow kolla development daily and have a feeling that last time whenever I read channel it is 'CI is fucked up again' | 09:13 |
hrw | which makes it hard to follow important patches in review | 09:13 |
mnasiadka | mrunge: it runs once a day I think | 09:16 |
mnasiadka | mrunge: I mean the mirror-update playbook | 09:16 |
mrunge | okay, that might explain this situation | 09:16 |
mrunge | thank you mnasiadka | 09:16 |
mrunge | I'll recheck later today then | 09:17 |
yoctozepto | hrw: yeah, we can officially announce the kolla fubar days | 09:18 |
yoctozepto | there is a lot of fire fighting, both proactive and reactive | 09:18 |
mrunge | mnasiadka, hmmm, the updated repo config is already on the mirror node being used :-/ | 09:19 |
yoctozepto | INFO:kolla.common.utils.fluentd:curl (https://codeload.github.com/monasca/fluentd-monasca/tar.gz/0.1.1): response: 500, time: 6.129151, size: 26 | 09:19 |
yoctozepto | INFO:kolla.common.utils.fluentd:[91mtar: This does not look like a tar archive | 09:19 |
hrw | mgoddard, yoctozepto: I think that after Ussuri release we should run some analytics on it and decide which things we remove as giving most of issues. | 09:19 |
yoctozepto | always monasca to blame | 09:19 |
hrw | as I have a feeling that ~50% of time we spend on fixing CI rather on development | 09:20 |
openstackgerrit | Mark Goddard proposed openstack/kayobe master: Use unittest.mock instead of PyPI mock https://review.opendev.org/717266 | 09:23 |
openstackgerrit | Marcin Juszkiewicz proposed openstack/kolla master: CI: Add https match for mirror.centos.org to template-overrides https://review.opendev.org/716548 | 09:23 |
hrw | opstools use baseurl=http:// | 09:23 |
mnasiadka | mrunge: seems so, just checked in a fresh container and it worked - so maybe just run a recheck ;) | 09:24 |
mrunge | mnasiadka, apparently Piotr already hit recheck :) The last ci run used the older package | 09:25 |
mnasiadka | ok, so problem solved (I hope) :) | 09:26 |
*** ykarel|lunch is now known as ykarel | 09:26 | |
openstackgerrit | Pierre Riteau proposed openstack/kayobe stable/queens: Fix multiple CI failures https://review.opendev.org/717208 | 09:38 |
mgoddard | hrw: we can do a review of breakages if you like | 09:38 |
hrw | Am I right? We have to fix rocky to fix stein to fix train to fix master? | 09:38 |
mgoddard | hrw: possibly to get upgrade jobs going | 09:39 |
hrw | mgoddard: I see that patches flow so will look at RP+2 ones and at own ones | 09:39 |
mgoddard | hrw: master isn't broken | 09:39 |
*** pbing19 has quit IRC | 09:40 | |
*** pbing19 has joined #openstack-kolla | 09:40 | |
mnasiadka | mgoddard: actually - do we really need nfs-ganesha repo now in master - since ceph is gone? | 09:41 |
mnasiadka | or maybe it's sucked in as dep in centos-release-ceph-nautilus, ok | 09:41 |
mgoddard | mnasiadka: correct | 09:42 |
yoctozepto | mnasiadka: just need more reliable url replace mechanism, that's all | 09:42 |
mnasiadka | so we just disable it and never use again - that's why master works | 09:42 |
yoctozepto | mnasiadka: it works because you fixed it | 09:43 |
mnasiadka | yoctozepto: well, we could fetch baseurl using yum-config-manager and set it to mirror - if sed is not reliable :D | 09:43 |
mgoddard | although we don't use centos-release-ceph-nautilus on master | 09:43 |
mgoddard | master was never broken | 09:43 |
hrw | mgoddard: we use it for ceph client libs iirc | 09:43 |
mnasiadka | mgoddard: because there is no centos-release-ceph-nautilus in c8 yet ;) | 09:44 |
mnasiadka | hrw: mgoddard meant we don't use the ,,repo'' package, just define the repo manually | 09:44 |
hrw | k | 09:44 |
yoctozepto | mnasiadka: it's in a ducking comment | 09:48 |
mnasiadka | yoctozepto: so yum-config-manager won't print it, geez | 09:49 |
*** e0ne has joined #openstack-kolla | 09:54 | |
mnasiadka | yoctozepto: so we're left with sed, it hasn't failed us in the past, we just didn't think http:// will change into https:// ;) | 09:54 |
openstackgerrit | Michal Nasiadka proposed openstack/kolla-ansible master: WIP: Custom haproxy script for monitoring galera https://review.opendev.org/710213 | 09:56 |
hrw | mnasiadka: https://review.opendev.org/#/c/715164/ +2? | 09:57 |
patchbot | patch 715164 - kolla-ansible - CI: Always use upper-constraints - 4 patch sets | 09:57 |
mnasiadka | hrw: +8 | 09:59 |
hrw | muchas gracias, señor | 10:00 |
*** gfidente|afk is now known as gfidente | 10:02 | |
openstackgerrit | Pierre Riteau proposed openstack/kolla-ansible stable/queens: DNM: Try using a mirror of the legacy YUM Docker repository https://review.opendev.org/717276 | 10:02 |
openstackgerrit | Marcin Juszkiewicz proposed openstack/kolla-ansible stable/stein: CI: Use upper constraints to install clients https://review.opendev.org/717259 | 10:06 |
hrw | added depends-on on kolla patch | 10:06 |
openstackgerrit | Marcin Juszkiewicz proposed openstack/kolla-ansible stable/stein: CI: Use upper constraints to install clients https://review.opendev.org/717259 | 10:07 |
hrw | ops. unlooped | 10:07 |
hrw | fsck | 10:07 |
yoctozepto | mnasiadka: and many other things... | 10:09 |
mnasiadka | yoctozepto: well, as always :D | 10:18 |
*** rpittau is now known as rpittau|bbl | 10:29 | |
*** born2bake has joined #openstack-kolla | 10:44 | |
*** zhanglong has quit IRC | 10:44 | |
openstackgerrit | Michal Nasiadka proposed openstack/kolla-ansible master: WIP: Custom haproxy script for monitoring galera https://review.opendev.org/710213 | 10:48 |
*** cah_link has quit IRC | 10:54 | |
openstackgerrit | Radosław Piliszek proposed openstack/kolla master: Fix glance-api privsep errors https://review.opendev.org/715051 | 10:57 |
openstackgerrit | Bharat Kunwar proposed openstack/kolla stable/train: CI: Add https match for mirror.centos.org to template-overrides https://review.opendev.org/716616 | 11:11 |
openstackgerrit | Bharat Kunwar proposed openstack/kolla stable/train: Bump up magnum and manila versions https://review.opendev.org/716339 | 11:11 |
openstackgerrit | Bharat Kunwar proposed openstack/kolla stable/train: CI: Add https match for mirror.centos.org to template-overrides https://review.opendev.org/716616 | 11:11 |
*** cah_link has joined #openstack-kolla | 11:14 | |
Tony31 | mgoddard - same cert verify issue after changing dns and adding the variable | 11:18 |
Tony31 | mgoddard I dont think it's a config error because I copied the yaml for the cert that was working for the external API. | 11:19 |
Tony31 | mgoddard also initially I had the same certificate CA bundle for both variables kolla_internal_fqdn_cacert: and kolla_external_fqdn_cacert: | 11:19 |
hrw | 5 patches at the gate. good | 11:20 |
Tony31 | mgoddard im thinking to leave out internal TLS to move forward, is there anything else I could try to rule out being a config issue or a bug? | 11:20 |
mgoddard | Tony31: I would suggest using the kolla_toolbox container to iterate and find the problem | 11:22 |
mgoddard | docker exec -it kolla_toolbox openstack <args> | 11:22 |
mgoddard | if you can get that to work, then that would be a good start | 11:22 |
mgoddard | you may need the --os-cacert argment | 11:22 |
Tony31 | mgoddard sorry mate I'm not on the same page. This maybe too far advanced for me. I tried to run curl to check on this. I think this sounds like it's a bit more advanced than checking curl or logs for user-readable error logs :) | 11:23 |
mgoddard | Tony31: we have the openstack CLI installed in kolla-toolbox. You should be able to use it to reach the APIs via the internal network | 11:24 |
Tony31 | mgoddard - ok thanks Mark I'll look it up :) | 11:25 |
mgoddard | np | 11:25 |
mgoddard | yoctozepto: Looks like we have a CI dependency loop between kolla and k-a. | 11:25 |
mgoddard | kolla-toolbox image broken so we can't fix k-a | 11:26 |
mgoddard | k-a clients broken for init-runonce so we can't fix kolla | 11:26 |
mgoddard | non-voting deploy jobs in kolla :( | 11:26 |
mgoddard | We could really use a proper promotion process for our images, so we don't publish those that fail deploy jobs | 11:27 |
hrw | mgoddard: or force k-a jobs to build images until fixed? | 11:28 |
mgoddard | we should publish as <release>-stage, then retag | 11:28 |
mgoddard | hrw: that is a better solution, thanks | 11:28 |
mgoddard | although upgrade jobs wouldn't pass | 11:29 |
hrw | mgoddard: do a patch which adds 'build by default' and makes upgrades not voting. rebase fixes on top. we merge, fix situiation, revert | 11:29 |
*** scottsol has quit IRC | 11:30 | |
mgoddard | hrw: no upgrade jobs in rocky, so we could fix rocky, then fix stein tomorrow using fresh rocky images | 11:30 |
hrw | https://review.opendev.org/#/c/717024/ - mnasiadka yoctozepto +2 presto! | 11:31 |
patchbot | patch 717024 - kolla (stable/rocky) - Fix multiple CI issues - 4 patch sets | 11:31 |
mgoddard | hrw: it failed | 11:32 |
hrw | joking a bit ;D | 11:32 |
yoctozepto | mgoddard: indeed it seems so | 11:33 |
mgoddard | hrw: flaw in your plan - images won't build without kolla patch | 11:34 |
hrw | I love deploops | 11:36 |
*** pescobar has quit IRC | 11:37 | |
hrw | there are moments when I do not give a shit and just move to other project/thing | 11:37 |
hrw | maybe it is time to just go under desk and improve cabling in my desktop ;D | 11:37 |
yoctozepto | we should make another pair of stickers: "don't publish broken images" | 11:38 |
yoctozepto | with dead koala logo | 11:39 |
openstackgerrit | Mark Goddard proposed openstack/kolla stable/rocky: Fix multiple CI issues https://review.opendev.org/717024 | 11:41 |
openstackgerrit | Mark Goddard proposed openstack/kolla stable/stein: Fix multiple CI issues https://review.opendev.org/717008 | 11:44 |
hrw | yoctozepto: so you suggest that publish jobs should build, deploy all-in-one, test and then publish? | 11:44 |
yoctozepto | mgoddard: commented, I think we are missing # variants in some places | 11:44 |
hrw | +2 to both | 11:45 |
yoctozepto | hrw: yeah, but it's a too-lengthy process | 11:45 |
openstackgerrit | Merged openstack/kolla master: Bump prometheus_openstack_exporter to 1.0.0 https://review.opendev.org/715521 | 11:45 |
openstackgerrit | Merged openstack/kolla-ansible master: CI: Always use upper-constraints https://review.opendev.org/715164 | 11:45 |
openstackgerrit | Merged openstack/kolla master: Use unittest.mock instead of PyPI mock https://review.opendev.org/717075 | 11:45 |
openstackgerrit | Merged openstack/kolla-ansible master: Be less confusing about custom Docker registry https://review.opendev.org/715601 | 11:45 |
mgoddard | hrw, yoctozepto: not a single job, but dependent jobs | 11:45 |
mgoddard | we should publish to a master-stage tag, then a deploy job runs against those images | 11:46 |
mgoddard | if it passes, a third job promotes the images | 11:46 |
mgoddard | one issue is that it's not very fine grained, so a bad apple spoils the bunch | 11:46 |
mgoddard | but that's probably true anyway and old images are better than a broken image | 11:47 |
yoctozepto | mgoddard: and lengthy | 11:47 |
yoctozepto | large surface of breakage | 11:48 |
mgoddard | what do you mean? | 11:48 |
openstackgerrit | Mark Goddard proposed openstack/kolla stable/stein: Fix multiple CI issues https://review.opendev.org/717008 | 11:50 |
*** pescobar has joined #openstack-kolla | 11:50 | |
*** sorin-mihai_ has joined #openstack-kolla | 11:51 | |
openstackgerrit | Mark Goddard proposed openstack/kolla stable/rocky: Fix multiple CI issues https://review.opendev.org/717024 | 11:51 |
openstackgerrit | Mark Goddard proposed openstack/kolla-ansible stable/rocky: CI: Use upper constraints to install clients https://review.opendev.org/717262 | 11:52 |
*** sorin-mihai__ has quit IRC | 11:53 | |
openstackgerrit | Lin PeiWen proposed openstack/kolla-ansible master: manila share container name variable https://review.opendev.org/717293 | 11:56 |
yoctozepto | mgoddard: network connectivity likes to fail from time to time - if we create large chains of network-based actions then we have more occassions to fail to publish | 11:57 |
openstackgerrit | Radosław Piliszek proposed openstack/kolla stable/rocky: Fix multiple CI issues https://review.opendev.org/717024 | 12:00 |
openstackgerrit | Radosław Piliszek proposed openstack/kolla stable/rocky: Fix multiple CI issues https://review.opendev.org/717024 | 12:00 |
openstackgerrit | Radosław Piliszek proposed openstack/kolla stable/stein: Fix multiple CI issues https://review.opendev.org/717008 | 12:01 |
yoctozepto | go, go, go, go | 12:02 |
*** bengates_ has joined #openstack-kolla | 12:02 | |
*** bengates has quit IRC | 12:03 | |
*** rpittau|bbl is now known as rpitau | 12:08 | |
*** rpitau is now known as rpittau | 12:09 | |
*** bengates has joined #openstack-kolla | 12:11 | |
*** bengates_ has quit IRC | 12:11 | |
openstackgerrit | Michal Nasiadka proposed openstack/kolla master: neutron: Add ovn-octavia-driver https://review.opendev.org/717296 | 12:14 |
openstackgerrit | Michal Nasiadka proposed openstack/kolla master: neutron: Add ovn-octavia-driver https://review.opendev.org/717296 | 12:16 |
*** pescobar has quit IRC | 12:22 | |
openstackgerrit | Michal Nasiadka proposed openstack/kolla master: neutron: Add ovn-octavia-driver https://review.opendev.org/717296 | 12:25 |
ykarel | mgoddard, can u check comment https://review.opendev.org/#/c/710822/ | 12:25 |
patchbot | patch 710822 - kolla (stable/train) - Build collectd image for CentOS8 (MERGED) - 3 patch sets | 12:25 |
*** heikkine has quit IRC | 12:29 | |
*** bengates has quit IRC | 12:39 | |
*** bengates has joined #openstack-kolla | 12:39 | |
mgoddard | ykarel: makes sense. Can you propose a patch? | 12:40 |
ykarel | mgoddard, okk doing | 12:40 |
openstackgerrit | yatin proposed openstack/kolla stable/train: Ignore epel disable https://review.opendev.org/717300 | 12:44 |
ykarel | mgoddard, check please ^ | 12:44 |
*** Fl1nt has joined #openstack-kolla | 12:45 | |
Fl1nt | Good afternoon everyone! | 12:45 |
Fl1nt | quick question, was just reviewing https://opendev.org/openstack/kolla-ansible/src/branch/master/tools/kolla-ansible bash script, and I was wondering: | 12:46 |
Fl1nt | I know about the ${VAR} notation in bash | 12:47 |
Fl1nt | but what that @${VAR} often use mean? | 12:47 |
*** scottsol has joined #openstack-kolla | 12:48 | |
hrw | ykarel: gentle -1 given | 12:49 |
ykarel | hrw, ack checking | 12:49 |
ykarel | ok fixing | 12:50 |
openstackgerrit | Merged openstack/kolla master: CI: Add https match for mirror.centos.org to template-overrides https://review.opendev.org/716548 | 12:53 |
hrw | ykarel: commit message needs to be clear message. "where: what\n\nwhy" | 12:53 |
ykarel | hrw, got it | 12:54 |
openstackgerrit | yatin proposed openstack/kolla stable/train: Collectd: Ignore failure of --disable epel repos https://review.opendev.org/717300 | 12:54 |
hrw | ykarel: I have simple rule: if I need to read code to find out what change does then commit message is wrong | 12:54 |
hrw | ykarel: s/--//? | 12:54 |
hrw | 'collect: ignore failure of disabling epel repos' | 12:55 |
openstackgerrit | Bharat Kunwar proposed openstack/kolla stable/train: Bump up magnum and manila versions https://review.opendev.org/716339 | 12:56 |
ykarel | hrw, okk fixing | 12:56 |
mgoddard | Fl1nt: it's for ansible. ansible-playbook -e @<file> | 12:56 |
*** pbing19 has quit IRC | 12:58 | |
openstackgerrit | yatin proposed openstack/kolla stable/train: Collectd: ignore failure of disabling epel repos https://review.opendev.org/717300 | 12:59 |
*** srini_ has quit IRC | 13:00 | |
openstackgerrit | Marcin Juszkiewicz proposed openstack/kolla master: kolla-toolbox: use requirements for pip constraints https://review.opendev.org/717080 | 13:00 |
hrw | pep8 fixes | 13:00 |
hrw | ykarel: +2 | 13:00 |
ykarel | hrw, Thanks | 13:00 |
openstackgerrit | Piotr Rabiega proposed openstack/kolla master: add dpdk_telemetry and logparser, use centos-release-opstools for C8 https://review.opendev.org/715923 | 13:01 |
openstackgerrit | Pierre Riteau proposed openstack/kolla-ansible stable/queens: CI: Use upper constraints to install clients https://review.opendev.org/717302 | 13:10 |
Fl1nt | oh ok, thx @mgoddard so that's a specific ansible char in order for him to load bunch of vars from file, noticed. | 13:14 |
*** sorin-mihai_ has quit IRC | 13:18 | |
*** sorin-mihai_ has joined #openstack-kolla | 13:21 | |
hrw | https://review.opendev.org/#/c/713135 anyone? | 13:23 |
patchbot | patch 713135 - kolla - CI: Avoid tox - 33 patch sets | 13:23 |
*** k_mouza has quit IRC | 13:26 | |
openstackgerrit | Merged openstack/kolla-ansible master: Use unittest.mock instead of PyPI mock https://review.opendev.org/717073 | 13:29 |
openstackgerrit | Pierre Riteau proposed openstack/kolla-ansible stable/queens: CI: Use upper constraints to install clients https://review.opendev.org/717302 | 13:40 |
openstackgerrit | Marcin Juszkiewicz proposed openstack/kolla master: CI: Avoid tox https://review.opendev.org/713135 | 13:47 |
hrw | rebased | 13:47 |
openstackgerrit | Mark Goddard proposed openstack/kolla-ansible master: Remove default(omit) from openstack_cacert in templates https://review.opendev.org/717308 | 13:49 |
openstackgerrit | Merged openstack/kolla master: Revert "openstack-base: pin setuptools < 46 to get horizon working" https://review.opendev.org/715166 | 13:53 |
openstackgerrit | Marcin Juszkiewicz proposed openstack/kolla master: CI: Ansiblify setup of builders https://review.opendev.org/715202 | 14:00 |
hrw | rebased | 14:00 |
hrw | bbl | 14:01 |
*** noxoid has joined #openstack-kolla | 14:14 | |
*** Chaserjim has quit IRC | 14:16 | |
*** TrevorV has joined #openstack-kolla | 14:18 | |
*** cdearborn has joined #openstack-kolla | 14:25 | |
*** EmilienM is now known as EvilienM | 14:40 | |
openstackgerrit | Merged openstack/kolla stable/stein: Fix multiple CI issues https://review.opendev.org/717008 | 14:42 |
hrw | so which branches we have working now? | 14:45 |
*** pbing19 has joined #openstack-kolla | 14:48 | |
*** Chaserjim has joined #openstack-kolla | 14:49 | |
openstackgerrit | Pierre Riteau proposed openstack/kayobe master: Prevent openrc files from using wrong OS_CACERT value https://review.opendev.org/717314 | 14:49 |
*** ykarel is now known as ykarel|away | 14:52 | |
*** Chaserjim has quit IRC | 14:57 | |
openstackgerrit | Pierre Riteau proposed openstack/kolla-ansible stable/queens: DNM: Try using mirrors of the legacy YUM Docker repository https://review.opendev.org/717276 | 15:06 |
openstackgerrit | Pierre Riteau proposed openstack/kolla-ansible stable/queens: CI: Use upper constraints to install clients https://review.opendev.org/717302 | 15:06 |
*** klindgren_ has quit IRC | 15:06 | |
*** klindgren has joined #openstack-kolla | 15:07 | |
*** e0ne has quit IRC | 15:12 | |
*** yoctozepto has quit IRC | 15:17 | |
*** cah_link has quit IRC | 15:17 | |
openstackgerrit | Pierre Riteau proposed openstack/kayobe master: Prevent openrc files from using wrong OS_CACERT value https://review.opendev.org/717314 | 15:18 |
*** yoctozepto8 has joined #openstack-kolla | 15:18 | |
*** Wasaac has quit IRC | 15:18 | |
*** Wasaac has joined #openstack-kolla | 15:19 | |
*** hjensas is now known as hjensas|afk | 15:21 | |
*** happyhemant has joined #openstack-kolla | 15:26 | |
*** k_mouza has joined #openstack-kolla | 15:27 | |
*** k_mouza has quit IRC | 15:31 | |
*** igordc has joined #openstack-kolla | 15:32 | |
*** k_mouza has joined #openstack-kolla | 15:34 | |
openstackgerrit | Merged openstack/kolla stable/rocky: Fix multiple CI issues https://review.opendev.org/717024 | 15:38 |
*** JamesBenson has joined #openstack-kolla | 15:38 | |
*** cah_link has joined #openstack-kolla | 15:40 | |
*** rpittau is now known as rpittau|afk | 15:43 | |
openstackgerrit | Marcin Juszkiewicz proposed openstack/kolla stable/rocky: Disable Rocky periodic publishing jobs https://review.opendev.org/715136 | 15:46 |
hrw | rebased | 15:46 |
hrw | https://review.opendev.org/#/c/713135/ anyone? so https://review.opendev.org/715202 will merge too | 15:49 |
patchbot | patch 713135 - kolla - CI: Avoid tox - 34 patch sets | 15:49 |
patchbot | patch 715202 - kolla - CI: Ansiblify setup of builders - 9 patch sets | 15:49 |
openstackgerrit | Marcin Juszkiewicz proposed openstack/kolla stable/stein: Fix invalid test in cinder data migration script https://review.opendev.org/715163 | 15:53 |
hrw | rebased | 15:53 |
hrw | so we have fixed rocky. and fixed stein | 15:54 |
hrw | train needs stein images so need some +2+W on monday. then we need to publish working images to get master upgrade jobs working. Right? | 15:55 |
*** bengates has quit IRC | 15:55 | |
born2bake | Hi, I am wondering why cinder-volume is failing for me. Kolla-ansible 9.0, train, ubuntu, sources: enable_cinder: "yes", enable_cinder_backup: "yes", using external ceph - cinder_backend_ceph: "yes" | 15:57 |
born2bake | nova, glance are working fine | 15:57 |
born2bake | cinder-volume.log ERROR cinder.cmd.volume [-] Configuration for cinder-volume does not specify "enabled_backends". Using DEFAULT section to configure drivers is not supported since Ocata. | 15:57 |
hrw | no idea | 15:59 |
hrw | but I do not deploy | 15:59 |
born2bake | how can I debug it if container does not show any logs? | 16:01 |
*** e0ne has joined #openstack-kolla | 16:02 | |
born2bake | 2020-04-03 17:54:58.589 7 WARNING cinder.scheduler.filter_scheduler [req-53f559bd-225f-4820-ae33-2f26aca86bcd e10497c216a94c23a881d4ab14618da3 7d971ea300594b0d933389e8dcc19276 - default default] No weighed backend found for volume with properties: {'id': '5072a931-765d-49e3-8519-573c9d0c23da', 'name': '__DEFAULT__', 'description': 'Default Volume Type', 'is_public': True, 'projects': [], 'extra_specs': {}, 'qos_specs_id': None, | 16:03 |
born2bake | 'created_at': '2020-04-03T12:59:13.000000', 'updated_at': '2020-04-03T12:59:13.000000', 'deleted_at': None, 'deleted': False} | 16:03 |
*** sorin-mihai_ has quit IRC | 16:03 | |
*** sorin-mihai_ has joined #openstack-kolla | 16:03 | |
born2bake | seems like similar issue https://bugs.launchpad.net/openstack-ansible/+bug/1768231 | 16:04 |
openstack | Launchpad bug 1768231 in openstack-ansible "Cinder filter scheduler not enabled with multiple storage back ends" [Medium,In progress] | 16:04 |
born2bake | but I do not have multiple storage back end... | 16:04 |
*** jimcrowleyibm has joined #openstack-kolla | 16:09 | |
*** dking_desktop has joined #openstack-kolla | 16:12 | |
openstackgerrit | Mark Goddard proposed openstack/kayobe master: Avoid unconditional fact gathering https://review.opendev.org/716315 | 16:14 |
mgoddard | hrw: train images are fine because we use upper-constraints | 16:15 |
mgoddard | waiting on rocky & stein images to fix rocky -> train | 16:15 |
born2bake | okay it seems like kolla's automatic external ceph support with cinder is broken. I followed stein guide and created cinder.conf files manually and its working now. | 16:16 |
openstackgerrit | Radosław Piliszek proposed openstack/kolla stable/train: Collectd: ignore failure of disabling epel repos https://review.opendev.org/717300 | 16:22 |
openstackgerrit | Mark Goddard proposed openstack/kayobe master: Use Ansible connection reset support in docker role https://review.opendev.org/717332 | 16:31 |
*** evrardjp has quit IRC | 16:36 | |
*** evrardjp has joined #openstack-kolla | 16:37 | |
*** skramaja_ has quit IRC | 16:40 | |
openstackgerrit | Pierre Riteau proposed openstack/kolla-ansible stable/queens: CI: Use upper constraints to install clients https://review.opendev.org/717302 | 16:41 |
yoctozepto8 | mgoddard: https://review.opendev.org/717310 oslo running away from mock drop ;/ | 16:42 |
patchbot | patch 717310 - oslotest - Revert "Switch to unittest.mock from mock" - 1 patch set | 16:42 |
mgoddard | yoctozepto8: not too surprising :) | 16:43 |
yoctozepto8 | mgoddard: well, yeah | 16:43 |
*** yoctozepto8 is now known as yoctozepto | 16:43 | |
*** gfidente is now known as gfidente|afk | 16:43 | |
yoctozepto | I wonder why bouncer failed to change nick | 16:43 |
openstackgerrit | Radosław Piliszek proposed openstack/kolla-ansible master: CI: Avoid tox (and clean up gate setup) https://review.opendev.org/713134 | 16:46 |
*** sorin-mihai_ has quit IRC | 16:52 | |
*** sorin-mihai has joined #openstack-kolla | 16:55 | |
*** Tony31 has quit IRC | 16:56 | |
*** sorin-mihai has quit IRC | 16:57 | |
*** scottsol has quit IRC | 17:01 | |
*** sorin-mihai has joined #openstack-kolla | 17:01 | |
openstackgerrit | Mark Goddard proposed openstack/kolla-ansible master: CI: Avoid tox (and clean up gate setup) https://review.opendev.org/713134 | 17:05 |
*** ktibi has quit IRC | 17:11 | |
openstackgerrit | Radosław Piliszek proposed openstack/kolla master: neutron: Add ovn-octavia-driver https://review.opendev.org/717296 | 17:16 |
cosmicsound | is this https://docs.openstack.org/kolla-ansible/latest/reference/storage/external-ceph-guide.html supposed to work on train also with the variables in globals.yml, does this mean the .conf files in /etk/kolla/config are not needed ? | 17:21 |
*** Fl1nt has quit IRC | 17:22 | |
*** jbadiapa has quit IRC | 17:29 | |
*** JangwonLee__ has quit IRC | 17:33 | |
*** JangwonLee__ has joined #openstack-kolla | 17:34 | |
*** k_mouza has quit IRC | 17:40 | |
yoctozepto | cosmicsound: no, for train it is different | 17:41 |
yoctozepto | switch 'latest' to 'train' | 17:41 |
*** dking_desktop has left #openstack-kolla | 17:45 | |
*** dking_desktop has joined #openstack-kolla | 17:45 | |
cdearborn | hey folks, I work on the Dell iDRAC driver in the Ironic project. We're working on transitioning to the Redfish protocol and have started using the Sushy project to do so. We have a new plugin to Sushy called sushy-oem-idrac (https://opendev.org/x/sushy-oem-idrac) that we need to include in the container that runs ironic-conductor. Any idea what needs to be done to make that happen? I really know | 17:48 |
cdearborn | virtually nothing about Kolla. | 17:48 |
hrw | cdearborn: is it present in requirements.txt? | 17:52 |
cdearborn | hrw: ironic has a driver-requirements.txt, and it is present there. not sure if that is sufficient... | 17:53 |
cdearborn | https://github.com/openstack/ironic/blob/master/driver-requirements.txt#L23 | 17:54 |
hrw | let me check | 17:55 |
hrw | building image | 17:55 |
cdearborn | hrw: thx! | 17:55 |
*** sean-k-mooney has quit IRC | 17:56 | |
cosmicsound | ceph_pool = gnocchi is not needed anymore in gnocchi/ | 18:01 |
*** cah_link1 has joined #openstack-kolla | 18:07 | |
*** cah_link has quit IRC | 18:09 | |
*** cah_link1 is now known as cah_link | 18:09 | |
yoctozepto | cosmicsound: it gets simpler in ussuri, yeah; less conditionals all around | 18:11 |
cosmicsound | +1 | 18:14 |
*** rmart04 has quit IRC | 18:38 | |
openstackgerrit | Merged openstack/kolla stable/stein: Fix invalid test in cinder data migration script https://review.opendev.org/715163 | 18:45 |
*** k_mouza has joined #openstack-kolla | 19:06 | |
cosmicsound | so qcow2 is for images is not recommended in ceph | 19:13 |
cosmicsound | rather raw | 19:13 |
kplant | correct | 19:13 |
cosmicsound | noted that downn | 19:16 |
kplant | iirc you can still upload qcow2 images to glance | 19:17 |
cosmicsound | ive oticed | 19:17 |
kplant | however when you create an instance, it will copy the qcow2 into a raw container | 19:17 |
kplant | so you lose the benefit of ceph zero byte copies | 19:17 |
kplant | and it's much slower | 19:17 |
cosmicsound | thanks for the explain | 19:18 |
cosmicsound | makes sense | 19:18 |
cosmicsound | il make sure all are raw by default | 19:18 |
kplant | the other thing to be aware of is, if you do upload raw images into glance | 19:18 |
cosmicsound | its weird enough i cant delete images now | 19:18 |
kplant | if you haev images deployed with that image, you can't delete the image from glance | 19:18 |
cosmicsound | :D | 19:18 |
kplant | since they'll depend on it | 19:18 |
cosmicsound | lol | 19:18 |
kplant | what you can do, is inflate all of the instance volumes | 19:18 |
cosmicsound | just as i wrote i cant delete images | 19:19 |
kplant | then you can delete the glance image | 19:19 |
cosmicsound | funny you mentioned this | 19:19 |
kplant | yeah it kind of sucks | 19:19 |
cosmicsound | inflate the volume? | 19:19 |
cosmicsound | hmm not sure i got to that chapter | 19:19 |
cosmicsound | is this ceph action or openstack | 19:19 |
kplant | that's a ceph specific thing | 19:19 |
cosmicsound | ok | 19:19 |
cosmicsound | looks like a pool stuff | 19:19 |
kplant | so if you don't want to deal with that, you can upload qcow2s | 19:19 |
kplant | but it'll be less efficient | 19:19 |
*** k_mouza has quit IRC | 19:21 | |
cosmicsound | i dont have any instaces deployed with an image | 19:22 |
cosmicsound | and i still cant delete it | 19:22 |
cosmicsound | will look intot hat deflate actions | 19:22 |
openstackgerrit | Dincer Celik proposed openstack/kolla master: Upgrade RabbitMQ to 3.8 https://review.opendev.org/694799 | 19:30 |
kplant | that's odd | 19:40 |
kplant | do you have any volumes hanging around | 19:40 |
kplant | you can delete a raw image with a ceph backend if there are no deps | 19:40 |
openstackgerrit | Merged openstack/kolla master: add dpdk_telemetry and logparser, use centos-release-opstools for C8 https://review.opendev.org/715923 | 19:48 |
*** TrevorV has quit IRC | 19:54 | |
cosmicsound | i think there are volumes hanging | 19:59 |
cosmicsound | not sure | 19:59 |
cosmicsound | if they are they arent visible in horizon | 19:59 |
openstackgerrit | Merged openstack/kolla master: Update hacking for Python3 https://review.opendev.org/716691 | 20:03 |
*** sorin-mihai has quit IRC | 20:03 | |
*** sorin-mihai has joined #openstack-kolla | 20:04 | |
openstackgerrit | Andreas Jaeger proposed openstack/kolla-ansible master: Update hacking for Python3 https://review.opendev.org/717382 | 20:13 |
*** jimcrowleyibm has quit IRC | 20:13 | |
*** sorin-mihai_ has joined #openstack-kolla | 20:36 | |
*** sorin-mihai has quit IRC | 20:38 | |
*** e0ne has quit IRC | 20:38 | |
*** sorin-mihai__ has joined #openstack-kolla | 20:43 | |
*** sorin-mihai_ has quit IRC | 20:46 | |
*** sorin-mihai has joined #openstack-kolla | 20:49 | |
*** sorin-mihai has quit IRC | 20:50 | |
*** sorin-mihai has joined #openstack-kolla | 20:51 | |
*** sorin-mihai__ has quit IRC | 20:51 | |
*** sorin-mihai_ has joined #openstack-kolla | 20:53 | |
*** sorin-mihai has quit IRC | 20:55 | |
*** EvilienM is now known as EmilienM | 21:07 | |
*** ktibi has joined #openstack-kolla | 21:07 | |
born2bake | mnasiadka Hi, I still feel like you migh help me with that one :) - https://bugs.launchpad.net/kolla-ansible/+bug/1846476 ; I ve added heat resource-list/show output here - https://bugs.launchpad.net/kolla-ansible/+bug/1846476/comments/8 | 21:08 |
openstack | Launchpad bug 1846476 in kolla-ansible "K8s cluster deployment is stuck at kube_cluster_deploy OS::Heat::SoftwareDeployment" [Undecided,New] | 21:08 |
*** ktibi has quit IRC | 21:11 | |
*** KeithMnemonic has quit IRC | 21:34 | |
*** k_mouza has joined #openstack-kolla | 21:43 | |
*** k_mouza has quit IRC | 21:47 | |
*** k_mouza has joined #openstack-kolla | 21:48 | |
*** JamesBenson has quit IRC | 21:50 | |
*** gfidente|afk has quit IRC | 21:53 | |
*** born2bake has quit IRC | 22:04 | |
*** vedup19 has joined #openstack-kolla | 22:16 | |
*** pbing19 has quit IRC | 22:18 | |
*** vedup19 has quit IRC | 22:31 | |
*** chrizl has quit IRC | 22:32 | |
*** igordc has quit IRC | 22:36 | |
*** igordc has joined #openstack-kolla | 22:36 | |
*** chrizl has joined #openstack-kolla | 22:43 | |
*** negronjl has quit IRC | 23:17 | |
*** negronjl has joined #openstack-kolla | 23:19 | |
hrw | cdearborn: INFO:kolla.common.utils.ironic-conductor:Successfully installed ansible-2.9.6 ironic-staging-drivers-0.12.0 proliantutils-2.9.4 pyasn1-0.4.8 pycryptodomex-3.9.7 pyghmi-1.5.13 pysmi-0.3.4 pysnmp-4.4.12 python-dracclient-3.2.0 python-ibmcclient-0.1.0 python-scciclient-0.8.1 python-xclarityclient-0.1.7 sushy-3.2.0 sushy-oem-idrac-0.0.2 | 23:59 |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!