| openstackgerrit | RedHat RDO CI proposed openstack/tripleo-quickstart-extras master: GATE CHECK for quickstart-extras https://review.openstack.org/560445 | 00:00 |
|---|---|---|
| openstackgerrit | RedHat RDO CI proposed openstack/tripleo-heat-templates master: GATE CHECK for TripleO https://review.openstack.org/604298 | 00:00 |
| openstackgerrit | RedHat RDO CI proposed openstack/tripleo-heat-templates stable/rocky: GATE CHECK for TripleO https://review.openstack.org/604293 | 00:00 |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: Update test coverage for openSUSE-minial to 15.0 https://review.openstack.org/610436 | 00:01 |
| *** ooolpbot has joined #tripleo | 00:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 00:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1798195 | 00:10 |
| openstack | Launchpad bug 1798195 in tripleo "rdo-cloud yum repos unavailable during container updates and failing the undercloud install " [Critical,Triaged] | 00:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1799895 | 00:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1800737 | 00:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1800742 | 00:10 |
| *** ooolpbot has quit IRC | 00:10 | |
| openstack | Launchpad bug 1799895 in tripleo "CI: undercloud takes long time which causes job fail with timeout" [Critical,Triaged] | 00:10 |
| openstack | Launchpad bug 1800737 in tripleo "podman, "panic: runtime error: invalid memory address or nil pointer dereference"," [Critical,Triaged] | 00:10 |
| openstack | Launchpad bug 1800742 in tripleo "tempest.lib.exceptions.IdentityError: Got identity error, undercloud-containers" [Critical,Triaged] | 00:10 |
| *** salmankhan has quit IRC | 00:11 | |
| openstackgerrit | Merged openstack/diskimage-builder master: Turn on quiet mode when logfile specified https://review.openstack.org/612865 | 00:15 |
| *** evrardjp has quit IRC | 00:50 | |
| *** ssbarnea has quit IRC | 00:52 | |
| *** phuongnh has joined #tripleo | 01:01 | |
| *** ooolpbot has joined #tripleo | 01:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 01:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1798195 | 01:10 |
| openstack | Launchpad bug 1798195 in tripleo "rdo-cloud yum repos unavailable during container updates and failing the undercloud install " [Critical,Triaged] | 01:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1799895 | 01:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1800737 | 01:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1800742 | 01:10 |
| openstack | Launchpad bug 1799895 in tripleo "CI: undercloud takes long time which causes job fail with timeout" [Critical,Triaged] | 01:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1800748 | 01:10 |
| *** ooolpbot has quit IRC | 01:10 | |
| openstack | Launchpad bug 1800737 in tripleo "podman, "panic: runtime error: invalid memory address or nil pointer dereference"," [Critical,Triaged] | 01:10 |
| openstack | Launchpad bug 1800742 in tripleo "tempest.lib.exceptions.IdentityError: Got identity error, undercloud-containers" [Critical,Triaged] | 01:10 |
| openstack | Launchpad bug 1800748 in tripleo "undercloud-containers job does not configure a docker mirror" [Critical,Triaged] | 01:10 |
| openstackgerrit | Merged openstack/diskimage-builder master: Remove python3 legacy jobs https://review.openstack.org/614047 | 01:18 |
| *** ccamacho has quit IRC | 01:24 | |
| *** ccamacho has joined #tripleo | 01:25 | |
| jrist | chem: like mwhahaha says, we've always had this, it was just previously broken since Ocata | 01:30 |
| mwhahaha | which one | 01:30 |
| jrist | https://review.openstack.org/#/c/611305/ | 01:30 |
| jrist | mwhahaha: ^ | 01:30 |
| mwhahaha | ah yes | 01:30 |
| jrist | and we really need this merge | 01:31 |
| jrist | like | 01:31 |
| jrist | bad | 01:31 |
| jrist | it's really hurting testing and a blocker | 01:31 |
| mwhahaha | that and everything else :o | 01:31 |
| jrist | downstream | 01:31 |
| jrist | true | 01:31 |
| jrist | I thought it was podman breaking things | 01:31 |
| jrist | :) | 01:31 |
| mwhahaha | it's sure not helping | 01:32 |
| mwhahaha | EmilienM: we need to chat about podman + proxy for container pulls tomorrow | 01:32 |
| mwhahaha | stevebaker: i guess you would also be the person to chat about, how do we make sure the skopeo (is this the default now?) image uploader uses the proxy | 01:35 |
| stevebaker | mwhahaha: skopeo is not the default for copy, and we use python for inspect, so getting that to respect a mirror variable should be simple | 01:37 |
| stevebaker | mwhahaha: I'm working on a concurrent copy and duplicate layer detection in skopeo so it is fast enough for us | 01:37 |
| mwhahaha | stevebaker: ok so that's likely an issue we're running into in the CI because previously i think the docker client would respect the docker mirror config | 01:38 |
| mwhahaha | stevebaker: but when we switch to podman, the docker .json no longer has this configuration | 01:38 |
| mwhahaha | stevebaker: so it is still using the DockerImageUploader today? | 01:38 |
| stevebaker | mwhahaha: correct, with a python equivalent of skopeo inspect that is not yet mirror aware | 01:39 |
| mwhahaha | ok so that's likely a problem too | 01:39 |
| mwhahaha | will resurect the bug i just created and will deal with that later | 01:39 |
| mwhahaha | ugh this stuff is so complicated | 01:40 |
| mwhahaha | https://bugs.launchpad.net/tripleo/+bug/1800748 | 01:41 |
| openstack | Launchpad bug 1800748 in tripleo "undercloud-containers job does not configure a docker mirror when podman is the container cli" [High,Triaged] | 01:41 |
| * mwhahaha wanders off and will ponder this problem later | 01:41 | |
| jrist | ps guys | 01:45 |
| jrist | vote for train | 01:45 |
| *** ooolpbot has joined #tripleo | 02:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 02:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1798195 | 02:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1799895 | 02:10 |
| openstack | Launchpad bug 1798195 in tripleo "rdo-cloud yum repos unavailable during container updates and failing the undercloud install " [Critical,Triaged] | 02:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1800737 | 02:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1800742 | 02:10 |
| *** ooolpbot has quit IRC | 02:10 | |
| openstack | Launchpad bug 1799895 in tripleo "CI: undercloud takes long time which causes job fail with timeout" [Critical,Triaged] | 02:10 |
| openstack | Launchpad bug 1800737 in tripleo "podman, "panic: runtime error: invalid memory address or nil pointer dereference"," [Critical,Triaged] | 02:10 |
| openstack | Launchpad bug 1800742 in tripleo "tempest.lib.exceptions.IdentityError: Got identity error, undercloud-containers" [Critical,Triaged] | 02:10 |
| *** chem has quit IRC | 02:11 | |
| *** tzumainn has quit IRC | 02:17 | |
| *** tzumainn has joined #tripleo | 02:19 | |
| *** thrash is now known as thrash|g0ne | 02:26 | |
| *** ykarel has joined #tripleo | 02:34 | |
| *** rh-jelabarre has quit IRC | 02:39 | |
| *** ramishra has quit IRC | 02:43 | |
| *** itlinux has joined #tripleo | 02:45 | |
| *** ramishra has joined #tripleo | 02:49 | |
| *** mrsoul has joined #tripleo | 02:53 | |
| *** psachin has joined #tripleo | 02:53 | |
| *** mschuppert has quit IRC | 02:56 | |
| *** itlinux has quit IRC | 02:56 | |
| *** ooolpbot has joined #tripleo | 03:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 03:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1798195 | 03:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1799895 | 03:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1800737 | 03:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1800742 | 03:10 |
| openstack | Launchpad bug 1798195 in tripleo "rdo-cloud yum repos unavailable during container updates and failing the undercloud install " [Critical,Triaged] | 03:10 |
| *** ooolpbot has quit IRC | 03:10 | |
| openstack | Launchpad bug 1799895 in tripleo "CI: undercloud takes long time which causes job fail with timeout" [Critical,Triaged] | 03:10 |
| openstack | Launchpad bug 1800737 in tripleo "podman, "panic: runtime error: invalid memory address or nil pointer dereference"," [Critical,Triaged] | 03:10 |
| openstack | Launchpad bug 1800742 in tripleo "tempest.lib.exceptions.IdentityError: Got identity error, undercloud-containers" [Critical,Triaged] | 03:10 |
| openstackgerrit | Alex Schultz proposed openstack-infra/tripleo-ci master: WIP: Container build CI job https://review.openstack.org/614392 | 03:18 |
| openstackgerrit | Alex Schultz proposed openstack-infra/tripleo-ci master: WIP: Container build CI job https://review.openstack.org/614392 | 03:26 |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: Add ubuntu-systemd-container operating-system element https://review.openstack.org/563748 | 03:29 |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: Add systemd-containers functional tests https://review.openstack.org/614051 | 03:29 |
| *** huynq has joined #tripleo | 03:30 | |
| *** udesale has joined #tripleo | 03:47 | |
| *** janki has joined #tripleo | 03:47 | |
| *** tzumainn has quit IRC | 04:09 | |
| *** ooolpbot has joined #tripleo | 04:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 04:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1798195 | 04:10 |
| openstack | Launchpad bug 1798195 in tripleo "rdo-cloud yum repos unavailable during container updates and failing the undercloud install " [Critical,Triaged] | 04:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1799895 | 04:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1800737 | 04:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1800742 | 04:10 |
| openstack | Launchpad bug 1799895 in tripleo "CI: undercloud takes long time which causes job fail with timeout" [Critical,Triaged] | 04:10 |
| *** ooolpbot has quit IRC | 04:10 | |
| openstack | Launchpad bug 1800737 in tripleo "podman, "panic: runtime error: invalid memory address or nil pointer dereference"," [Critical,Triaged] | 04:10 |
| openstack | Launchpad bug 1800742 in tripleo "tempest.lib.exceptions.IdentityError: Got identity error, undercloud-containers" [Critical,Triaged] | 04:10 |
| *** abishop has quit IRC | 04:13 | |
| *** abishop has joined #tripleo | 04:15 | |
| *** abishop has quit IRC | 04:27 | |
| huynq | Goodmorning! Did you get SSLError when import overcloud nodes? | 04:28 |
| huynq | http://paste.openstack.org/show/733674/ | 04:28 |
| flaper87 | mwhahaha: ahhaa, as you should. That was the main reason, yeah. Not sure if it's used now that the apb's experiment was put on hold | 04:29 |
| mwhahaha | It's all good. I proposed moving them around for now. We shall see how that works | 04:30 |
| *** abishop has joined #tripleo | 04:32 | |
| *** itlinux has joined #tripleo | 05:05 | |
| *** saneax has joined #tripleo | 05:09 | |
| *** ooolpbot has joined #tripleo | 05:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 05:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1798195 | 05:10 |
| openstack | Launchpad bug 1798195 in tripleo "rdo-cloud yum repos unavailable during container updates and failing the undercloud install " [Critical,Triaged] | 05:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1799895 | 05:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1800737 | 05:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1800742 | 05:10 |
| openstack | Launchpad bug 1799895 in tripleo "CI: undercloud takes long time which causes job fail with timeout" [Critical,Triaged] | 05:10 |
| *** ooolpbot has quit IRC | 05:10 | |
| openstack | Launchpad bug 1800737 in tripleo "podman, "panic: runtime error: invalid memory address or nil pointer dereference"," [Critical,Triaged] | 05:10 |
| openstack | Launchpad bug 1800742 in tripleo "tempest.lib.exceptions.IdentityError: Got identity error, undercloud-containers" [Critical,Triaged] | 05:10 |
| *** ykarel has quit IRC | 05:21 | |
| *** ykarel has joined #tripleo | 05:21 | |
| *** noama has joined #tripleo | 05:22 | |
| noama | hi, im trying to build overcloud images with fedora OS, it seems that package "osops-tools-monitoring-oschecks" is not built for fedora delorean repo https://trunk.rdoproject.org/fedora/current/ | 05:27 |
| *** pcaruana|elisa| has joined #tripleo | 05:29 | |
| *** pcaruana|elisa| has quit IRC | 05:37 | |
| *** janki has quit IRC | 05:46 | |
| *** jaganathan has joined #tripleo | 05:48 | |
| *** janki has joined #tripleo | 05:49 | |
| *** ykarel has quit IRC | 05:51 | |
| Tengu | EmilienM, weshay : well, the "real" error seems to be the absence of config-data directory. I'd say "race condition", but need some more time. | 05:52 |
| *** ratailor has joined #tripleo | 05:58 | |
| Tengu | yeah. race condition, I'm pretty sure of that. | 06:01 |
| Tengu | the config-data directory is created directly here: https://github.com/openstack/tripleo-heat-templates/blob/master/common/deploy-steps-tasks.yaml#L246 - in step 1. | 06:02 |
| Tengu | so that might be best in some host_prep_tasks. | 06:02 |
| *** itlinux has quit IRC | 06:04 | |
| *** saneax has quit IRC | 06:07 | |
| openstackgerrit | inspurericzhang proposed openstack/ansible-role-tripleo-memcached master: fix the module name in setup.cfg https://review.openstack.org/614407 | 06:09 |
| *** ooolpbot has joined #tripleo | 06:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 06:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1798195 | 06:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1799895 | 06:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1800737 | 06:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1800742 | 06:10 |
| openstack | Launchpad bug 1798195 in tripleo "rdo-cloud yum repos unavailable during container updates and failing the undercloud install " [Critical,Triaged] | 06:10 |
| *** ooolpbot has quit IRC | 06:10 | |
| openstack | Launchpad bug 1799895 in tripleo "CI: undercloud takes long time which causes job fail with timeout" [Critical,Triaged] | 06:10 |
| openstack | Launchpad bug 1800737 in tripleo "podman, "panic: runtime error: invalid memory address or nil pointer dereference"," [Critical,Triaged] | 06:10 |
| openstack | Launchpad bug 1800742 in tripleo "tempest.lib.exceptions.IdentityError: Got identity error, undercloud-containers" [Critical,Triaged] | 06:10 |
| *** janki has quit IRC | 06:18 | |
| *** agurenko has joined #tripleo | 06:33 | |
| *** tongl has quit IRC | 06:33 | |
| *** jfrancoa has joined #tripleo | 06:35 | |
| *** quiquell|off is now known as quiquell | 06:37 | |
| *** janki has joined #tripleo | 06:37 | |
| openstackgerrit | Yael Azulay proposed openstack/puppet-tripleo master: Changes are needed for performance and stability at large intervals https://review.openstack.org/603183 | 06:41 |
| *** ksambor has joined #tripleo | 06:41 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/tripleo-ui stable/queens: Imported Translations from Zanata https://review.openstack.org/614410 | 06:46 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/tripleo-ui master: Imported Translations from Zanata https://review.openstack.org/614411 | 06:48 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/tripleo-ui stable/rocky: Imported Translations from Zanata https://review.openstack.org/614412 | 06:49 |
| *** yprokule has joined #tripleo | 06:54 | |
| *** ooolpbot has joined #tripleo | 07:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 07:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1798195 | 07:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1799895 | 07:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1800737 | 07:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1800742 | 07:10 |
| openstack | Launchpad bug 1798195 in tripleo "rdo-cloud yum repos unavailable during container updates and failing the undercloud install " [Critical,Triaged] | 07:10 |
| *** ooolpbot has quit IRC | 07:10 | |
| openstack | Launchpad bug 1799895 in tripleo "CI: undercloud takes long time which causes job fail with timeout" [Critical,Triaged] | 07:10 |
| openstack | Launchpad bug 1800737 in tripleo "podman, "panic: runtime error: invalid memory address or nil pointer dereference"," [Critical,Triaged] | 07:10 |
| openstack | Launchpad bug 1800742 in tripleo "tempest.lib.exceptions.IdentityError: Got identity error, undercloud-containers" [Critical,Triaged] | 07:10 |
| quiquell | Good morning | 07:11 |
| Tengu | «o/ quiquell :) | 07:11 |
| quiquell | marios, jfrancoa: Can you +w this for the f28 stuff https://review.openstack.org/#/c/614168/ ? | 07:12 |
| quiquell | Tengu: ^ | 07:12 |
| quiquell | Tengu: good morning btw | 07:12 |
| Tengu | can't +w :( | 07:12 |
| quiquell | It's just adding missing ansible nodes at zuul job for fedora28 | 07:13 |
| openstackgerrit | inspurericzhang proposed openstack/ansible-role-tripleo-octavia master: Update min tox version to 2.0 https://review.openstack.org/614417 | 07:13 |
| quiquell | A sorry it's the python writting stuff | 07:13 |
| *** chkumar|off is now known as chandankumar | 07:13 | |
| jfrancoa | quiquell: done | 07:14 |
| jfrancoa | quiquell: good morning btw | 07:14 |
| quiquell | jfrancoa: thanks mate | 07:14 |
| noama | quiquell: hi, i see you active on fedora, did you built overcloud imanges? | 07:15 |
| *** ccamacho has quit IRC | 07:15 | |
| *** ccamacho has joined #tripleo | 07:15 | |
| quiquell | noama: Good morning, just running standalone | 07:15 |
| quiquell | noama: what's your issue ? | 07:16 |
| noama | https://bugs.launchpad.net/tripleo/+bug/1800768 | 07:16 |
| openstack | Launchpad bug 1800768 in tripleo "package "osops-tools-monitoring-oschecks" not exist for fedora" [Undecided,New] | 07:16 |
| noama | good morning | 07:16 |
| noama | quiquell: this package exist for centos but not for fedora | 07:17 |
| quiquell | noama: Bet place is #rdo | 07:18 |
| *** saneax has joined #tripleo | 07:18 | |
| quiquell | noama: RDO is packaging those things, some packages could be miss or not totally correct for fedora28 yet | 07:18 |
| noama | quiquell: ok ill ask there | 07:19 |
| openstackgerrit | Quique Llorente proposed openstack/tripleo-quickstart-extras master: DNM: squash to test standalone-f28 https://review.openstack.org/613316 | 07:21 |
| *** ykarel has joined #tripleo | 07:42 | |
| *** pcaruana|elisa| has joined #tripleo | 07:45 | |
| *** shardy has joined #tripleo | 07:45 | |
| *** hjensas has quit IRC | 07:46 | |
| *** shardy_ has joined #tripleo | 07:46 | |
| *** ykarel has quit IRC | 07:49 | |
| openstackgerrit | Merged openstack/tripleo-quickstart-extras master: Add new standalone_container_cli variable https://review.openstack.org/613972 | 07:50 |
| openstackgerrit | Merged openstack/puppet-tripleo master: mysql: use clustercheck credentials to poll galera state https://review.openstack.org/603612 | 07:50 |
| openstackgerrit | Merged openstack/tripleo-upgrade master: Fix variables for workaround variables. https://review.openstack.org/612607 | 07:50 |
| openstackgerrit | Merged openstack/tripleo-ui stable/rocky: Fix network lines rendering https://review.openstack.org/614169 | 07:50 |
| *** ykarel has joined #tripleo | 07:50 | |
| *** gkadam has joined #tripleo | 07:54 | |
| *** ykarel has quit IRC | 07:56 | |
| openstackgerrit | RedHat RDO CI proposed openstack/tripleo-quickstart-extras master: GATE CHECK for quickstart-extras https://review.openstack.org/560445 | 08:00 |
| *** ykarel has joined #tripleo | 08:02 | |
| *** sshnaidm|off is now known as sshnaidm|ruck | 08:03 | |
| *** florianf|afk is now known as florianf | 08:05 | |
| *** jaganathan has quit IRC | 08:08 | |
| *** holser_ has joined #tripleo | 08:09 | |
| *** ooolpbot has joined #tripleo | 08:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 08:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1798195 | 08:10 |
| openstack | Launchpad bug 1798195 in tripleo "rdo-cloud yum repos unavailable during container updates and failing the undercloud install " [Critical,Triaged] | 08:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1799895 | 08:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1800737 | 08:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1800742 | 08:10 |
| *** ooolpbot has quit IRC | 08:10 | |
| openstack | Launchpad bug 1799895 in tripleo "CI: undercloud takes long time which causes job fail with timeout" [Critical,Triaged] | 08:10 |
| openstack | Launchpad bug 1800737 in tripleo "podman, "panic: runtime error: invalid memory address or nil pointer dereference"," [Critical,Triaged] | 08:10 |
| openstack | Launchpad bug 1800742 in tripleo "tempest.lib.exceptions.IdentityError: Got identity error, undercloud-containers" [Critical,Triaged] | 08:10 |
| *** ykarel has quit IRC | 08:10 | |
| *** cylopez has joined #tripleo | 08:10 | |
| *** jtomasek has joined #tripleo | 08:18 | |
| *** hberaud has quit IRC | 08:18 | |
| *** hberaud has joined #tripleo | 08:19 | |
| openstackgerrit | Yurii Prokulevych proposed openstack/tripleo-upgrade stable/rocky: Fix variables for workaround variables. https://review.openstack.org/614220 | 08:19 |
| *** cylopez has left #tripleo | 08:21 | |
| openstackgerrit | Jiri Tomasek proposed openstack/tripleo-ui master: [WIP] Convert SelectRolesDialog to ModalPanel https://review.openstack.org/614151 | 08:22 |
| *** amoralej|off is now known as amoralej | 08:23 | |
| *** ykarel has joined #tripleo | 08:27 | |
| *** jpich has joined #tripleo | 08:28 | |
| *** ykarel has quit IRC | 08:28 | |
| *** ykarel has joined #tripleo | 08:29 | |
| *** ykarel_ has joined #tripleo | 08:31 | |
| *** akrivoka has joined #tripleo | 08:34 | |
| *** ykarel has quit IRC | 08:34 | |
| *** ykarel has joined #tripleo | 08:34 | |
| *** ykarel_ has quit IRC | 08:38 | |
| *** ssbarnea has joined #tripleo | 08:40 | |
| *** ykarel has quit IRC | 08:40 | |
| *** derekh has joined #tripleo | 08:51 | |
| *** jpena|off is now known as jpena | 08:52 | |
| openstackgerrit | Sorin Sbarnea proposed openstack/tripleo-quickstart master: Make quickstack.sh avoid needing sudo on missing virtualenv https://review.openstack.org/613797 | 08:56 |
| quiquell | amoralej: rdopkg not found, has being able to reproduce the issue :-), thanks for the help | 08:57 |
| amoralej | :) | 08:57 |
| quiquell | I mean command not found | 08:57 |
| amoralej | glad to help | 08:57 |
| *** aufi has joined #tripleo | 08:58 | |
| *** panda|off is now known as panda | 09:01 | |
| *** d0ugal has joined #tripleo | 09:02 | |
| *** chem has joined #tripleo | 09:05 | |
| *** ooolpbot has joined #tripleo | 09:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 09:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1798195 | 09:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1799895 | 09:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1800737 | 09:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1800742 | 09:10 |
| openstack | Launchpad bug 1798195 in tripleo "rdo-cloud yum repos unavailable during container updates and failing the undercloud install " [Critical,Triaged] | 09:10 |
| *** ooolpbot has quit IRC | 09:10 | |
| openstack | Launchpad bug 1799895 in tripleo "CI: undercloud takes long time which causes job fail with timeout" [Critical,Triaged] | 09:10 |
| openstack | Launchpad bug 1800737 in tripleo "podman, "panic: runtime error: invalid memory address or nil pointer dereference"," [Critical,Triaged] | 09:10 |
| openstack | Launchpad bug 1800742 in tripleo "tempest.lib.exceptions.IdentityError: Got identity error, undercloud-containers" [Critical,Triaged] | 09:10 |
| *** pcaruana|elisa| has quit IRC | 09:13 | |
| *** pcaruana|elisa| has joined #tripleo | 09:15 | |
| openstackgerrit | Jose Luis Franco proposed openstack/tripleo-heat-templates stable/queens: [Queens only] Use rhel-registration tool in Rhsm to update repos before upgrade https://review.openstack.org/611879 | 09:19 |
| *** bogdando has joined #tripleo | 09:21 | |
| *** hjensas has joined #tripleo | 09:29 | |
| openstackgerrit | Bogdan Dobrelya proposed openstack/tripleo-heat-templates stable/rocky: Ensure unique containers names in docker-puppet https://review.openstack.org/614450 | 09:35 |
| derekh | Anybody have time to take a look at two patches for a CI job to test ironic in the overcloud https://review.openstack.org/#/c/579603/ https://review.openstack.org/#/c/509728/ | 09:36 |
| *** gfidente has joined #tripleo | 09:37 | |
| openstackgerrit | Marios Andreou proposed openstack-infra/tripleo-ci master: WIP: expose zuul variables for reproducer https://review.openstack.org/613571 | 09:41 |
| openstackgerrit | Bogdan Dobrelya proposed openstack/tripleo-common master: Improve exp backoff for images pull/push/rpm-upd https://review.openstack.org/575742 | 09:44 |
| *** lennyb has quit IRC | 09:48 | |
| openstackgerrit | yolanda.robla proposed openstack/tripleo-common stable/queens: Add UEFI support for hardened images https://review.openstack.org/614453 | 09:57 |
| *** aufi has quit IRC | 10:02 | |
| openstackgerrit | yolanda.robla proposed openstack/tripleo-image-elements stable/queens: Add overcloud-secure-uefi element https://review.openstack.org/614455 | 10:04 |
| *** kopecmartin|off is now known as kopecmartin | 10:05 | |
| *** salmankhan has joined #tripleo | 10:05 | |
| *** ooolpbot has joined #tripleo | 10:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 10:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1798195 | 10:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1799895 | 10:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1800737 | 10:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1800742 | 10:10 |
| openstack | Launchpad bug 1798195 in tripleo "rdo-cloud yum repos unavailable during container updates and failing the undercloud install " [Critical,Triaged] | 10:10 |
| *** ooolpbot has quit IRC | 10:10 | |
| openstack | Launchpad bug 1799895 in tripleo "CI: undercloud takes long time which causes job fail with timeout" [Critical,Triaged] | 10:10 |
| openstack | Launchpad bug 1800737 in tripleo "podman, "panic: runtime error: invalid memory address or nil pointer dereference"," [Critical,Triaged] | 10:10 |
| openstack | Launchpad bug 1800742 in tripleo "tempest.lib.exceptions.IdentityError: Got identity error, undercloud-containers" [Critical,Triaged] | 10:10 |
| openstackgerrit | Cédric Jeanneret proposed openstack/tripleo-quickstart master: Move standalone job to podman instead of docker https://review.openstack.org/613975 | 10:12 |
| *** apetrich has quit IRC | 10:12 | |
| *** suuuper has joined #tripleo | 10:16 | |
| openstackgerrit | Rabi Mishra proposed openstack/tripleo-heat-templates master: Check for available networks for a role https://review.openstack.org/614457 | 10:16 |
| openstackgerrit | Merged openstack/tripleo-heat-templates stable/queens: Turning off extra UIs from Horizon fixes also SQLite issue https://review.openstack.org/613943 | 10:17 |
| openstackgerrit | Sorin Sbarnea proposed openstack/tripleo-quickstart master: wip: enable testing using ansible molecule https://review.openstack.org/613672 | 10:25 |
| *** apetrich has joined #tripleo | 10:27 | |
| *** huynq has quit IRC | 10:37 | |
| *** phuongnh has quit IRC | 10:38 | |
| openstackgerrit | Giulio Fidente proposed openstack/tripleo-heat-templates stable/queens: DO-NOT-MERGE Test new ceph-container builds (OSP queens) https://review.openstack.org/562215 | 10:46 |
| openstackgerrit | Giulio Fidente proposed openstack/tripleo-heat-templates stable/queens: DO-NOT-MERGE Test new ceph-container builds (OSP queens) https://review.openstack.org/562215 | 10:47 |
| openstackgerrit | Giulio Fidente proposed openstack/tripleo-heat-templates stable/rocky: DO-NOT-MERGE Test new ceph-container builds (OSP rocky) https://review.openstack.org/601598 | 10:47 |
| openstackgerrit | Giulio Fidente proposed openstack/tripleo-heat-templates stable/rocky: DO-NOT-MERGE Test new ceph-container builds (OSP rocky) https://review.openstack.org/601598 | 10:47 |
| openstackgerrit | Giulio Fidente proposed openstack/tripleo-heat-templates master: DO-NOT-MERGE Test new ceph-container builds https://review.openstack.org/562213 | 10:47 |
| openstackgerrit | Giulio Fidente proposed openstack/tripleo-heat-templates master: DO-NOT-MERGE Test new ceph-container builds https://review.openstack.org/562213 | 10:48 |
| *** dtantsur|afk is now known as dtantsur | 10:49 | |
| chem | therve: hey, is it possible to get the $HOME variable back into a tht template somehow? | 10:49 |
| therve | chem: Pass a parameter? | 10:50 |
| chem | therve: oki, no other way then, just wondering out loud. Thanks. | 10:51 |
| therve | chem: I mean... Not sure what you're expecting. I don't even know which $HOME you're talking about | 10:51 |
| therve | heat? root? stack? | 10:51 |
| *** thrash|g0ne is now known as thrash | 10:51 | |
| chem | therve: the one from the user triggering the command | 10:52 |
| chem | therve: in the undercloud | 10:52 |
| therve | chem: Right, so there is no way server side to know that | 10:52 |
| therve | We have a keystone user, not a unix one | 10:52 |
| openstackgerrit | Merged openstack/tripleo-heat-templates stable/rocky: Turning off extra UIs from Horizon fixes also SQLite issue https://review.openstack.org/613942 | 10:53 |
| openstackgerrit | Merged openstack/ansible-role-container-registry master: Add xfs ftype check during install https://review.openstack.org/613429 | 10:53 |
| openstackgerrit | Steven Hardy proposed openstack/tripleo-heat-templates master: Revert "Revert "Convert *tasks from bootstrap_nodeid to short_bootstrap_node_name"" https://review.openstack.org/611800 | 10:55 |
| openstackgerrit | Slawek Kaplonski proposed openstack/tripleo-heat-templates master: Remove external_network_bridge Neutron option https://review.openstack.org/604764 | 10:57 |
| openstackgerrit | Cédric Jeanneret proposed openstack/tripleo-quickstart master: Move standalone job to podman instead of docker https://review.openstack.org/613975 | 10:59 |
| openstackgerrit | Merged openstack/python-tripleoclient stable/rocky: Increase the deploy_plan timeout in tripleoclient https://review.openstack.org/613623 | 10:59 |
| *** iurygregory is now known as iury|lunch | 11:00 | |
| *** paramite has joined #tripleo | 11:10 | |
| *** ooolpbot has joined #tripleo | 11:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 11:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1798195 | 11:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1799895 | 11:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1800737 | 11:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1800742 | 11:10 |
| openstack | Launchpad bug 1798195 in tripleo "rdo-cloud yum repos unavailable during container updates and failing the undercloud install " [Critical,Triaged] | 11:10 |
| *** ooolpbot has quit IRC | 11:10 | |
| openstack | Launchpad bug 1799895 in tripleo "CI: undercloud takes long time which causes job fail with timeout" [Critical,Triaged] | 11:10 |
| openstack | Launchpad bug 1800737 in tripleo "podman, "panic: runtime error: invalid memory address or nil pointer dereference"," [Critical,Triaged] | 11:10 |
| openstack | Launchpad bug 1800742 in tripleo "tempest.lib.exceptions.IdentityError: Got identity error, undercloud-containers" [Critical,Triaged] | 11:10 |
| openstackgerrit | Gael Chamoulaud proposed openstack/tripleo-validations master: Avoid Ansible Deprecation Warning https://review.openstack.org/614466 | 11:10 |
| openstackgerrit | wanghui proposed openstack/tripleo-heat-templates master: Fix a typo of "rabbitmq" https://review.openstack.org/614467 | 11:12 |
| openstackgerrit | Sorin Sbarnea proposed openstack/tripleo-quickstart master: Make quickstack.sh avoid needing sudo on missing virtualenv https://review.openstack.org/613797 | 11:13 |
| *** jtomasek has quit IRC | 11:16 | |
| openstackgerrit | Merged openstack/tripleo-common master: Per server deployments instead of per role https://review.openstack.org/612687 | 11:16 |
| *** aufi has joined #tripleo | 11:20 | |
| *** bfournie has quit IRC | 11:22 | |
| *** ratailor has quit IRC | 11:26 | |
| openstackgerrit | Athlan-Guyot sofer proposed openstack/tripleo-heat-templates master: WIP: Restore tripleo_undercloud_conf_file hiera variable. https://review.openstack.org/614470 | 11:28 |
| *** ramishra has quit IRC | 11:28 | |
| openstackgerrit | Sorin Sbarnea proposed openstack/tripleo-quickstart master: Make quickstack.sh avoid needing sudo on missing virtualenv https://review.openstack.org/613797 | 11:29 |
| *** udesale has quit IRC | 11:32 | |
| openstackgerrit | Sorin Sbarnea proposed openstack/tripleo-quickstart master: Make quickstack.sh avoid needing sudo on missing virtualenv https://review.openstack.org/613797 | 11:34 |
| *** abishop has quit IRC | 11:38 | |
| *** jtomasek has joined #tripleo | 11:38 | |
| *** ramishra has joined #tripleo | 11:40 | |
| openstackgerrit | Quique Llorente proposed openstack/tripleo-quickstart-extras master: Use dnf and python3 on platforms where these are default https://review.openstack.org/610491 | 11:46 |
| *** rh-jelabarre has joined #tripleo | 11:46 | |
| quiquell | marios: ^ addressed the yum thing, thanks ! | 11:47 |
| quiquell | mandre: can you review again ? | 11:47 |
| quiquell | sshnaidm|ruck, panda: the py3 for tqe for f28 ^ | 11:48 |
| *** ramishra has quit IRC | 11:49 | |
| *** raildo has joined #tripleo | 11:51 | |
| *** morazi has joined #tripleo | 11:53 | |
| *** udesale has joined #tripleo | 11:54 | |
| *** ramishra has joined #tripleo | 11:56 | |
| openstackgerrit | Marios Andreou proposed openstack-infra/tripleo-ci master: WIP: expose zuul variables for reproducer https://review.openstack.org/613571 | 11:56 |
| jaosorior | Tengu: around? | 11:57 |
| Tengu | jaosorior: yep ? | 11:57 |
| *** bfournie has joined #tripleo | 11:57 | |
| *** jfrancoa has quit IRC | 11:58 | |
| jaosorior | #startmeeting TripleO Security Squad | 12:00 |
| openstack | Meeting started Wed Oct 31 12:00:32 2018 UTC and is due to finish in 60 minutes. The chair is jaosorior. Information about MeetBot at http://wiki.debian.org/MeetBot. | 12:00 |
| openstack | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 12:00 |
| *** openstack changes topic to " (Meeting topic: TripleO Security Squad)" | 12:00 | |
| openstack | The meeting name has been set to 'tripleo_security_squad' | 12:00 |
| jaosorior | hello folks! | 12:01 |
| jaosorior | who's around? | 12:01 |
| *** jfrancoa has joined #tripleo | 12:01 | |
| Tengu | «o/ | 12:01 |
| EmilienM | mwhahaha, weshay : please see my comment: https://review.openstack.org/#/c/614364/ | 12:01 |
| *** iury|lunch is now known as iurygregory | 12:01 | |
| jaosorior | raildo, lhinds, owalsh, redrobot, | 12:01 |
| raildo | o/ | 12:01 |
| owalsh | oh, stupid time change o/ | 12:02 |
| Tengu | owalsh: I'm not alone then :D | 12:02 |
| jaosorior | haha yeah, I was confused about it yesterday | 12:02 |
| * owalsh changes the event in my calendar to UTC | 12:02 | |
| jaosorior | IIRC, this daylight savings time business will stop in europe | 12:03 |
| openstackgerrit | Brent Eagles proposed openstack/tripleo-heat-templates master: Handle LP openvswitch meta-package on upgrade https://review.openstack.org/605200 | 12:03 |
| openstackgerrit | Sagi Shnaidman proposed openstack/tripleo-quickstart master: Make quickstack.sh avoid needing sudo on missing virtualenv https://review.openstack.org/613797 | 12:03 |
| Tengu | jaosorior: yeah, maybe, should be decided next year I think. We might stick to Summer time (which is stupid as hell, but, at least, we won't change it twice a year) | 12:04 |
| EmilienM | (sorry folks) | 12:04 |
| jaosorior | Tengu: Finland did decide to stick with winter time. | 12:04 |
| jaosorior | Anyway: | 12:04 |
| owalsh | jaosorior: maybe not here... it's only a recommendation AFAIK | 12:04 |
| EmilienM | mwhahaha, weshay : https://bugs.launchpad.net/tripleo/+bug/1800737 I renamed the bug. this is the actual problem | 12:04 |
| openstack | Launchpad bug 1800737 in tripleo "relabel failed /var/lib/config-data: no such file or directory" [Critical,Triaged] | 12:04 |
| jaosorior | #topic Secret Management Update | 12:04 |
| *** openstack changes topic to "Secret Management Update (Meeting topic: TripleO Security Squad)" | 12:04 | |
| jaosorior | raildo: ^^ | 12:04 |
| *** trown|outtypewww is now known as trown | 12:05 | |
| *** abishop has joined #tripleo | 12:05 | |
| weshay | EmilienM, sounds good | 12:05 |
| raildo | Hey folks, just a quick update on where we are about the secrets management stuff, more specifically regarding the Castellan driver for oslo.config | 12:05 |
| weshay | EmilienM, the gate is foooked http://dashboard-ci.tripleo.org/d/cEEjGFFmz/cockpit?orgId=1 | 12:06 |
| Tengu | weshay: will comment that LP. have an idea, need some info/feedbacks for solution. | 12:06 |
| raildo | so, at this point we wrote the castellan driver https://review.openstack.org/#/c/599589/ but we're pending to write some unit tests for it | 12:06 |
| weshay | just fyi.. we're down to 72.5% that is really bad | 12:06 |
| *** ramishra has quit IRC | 12:07 | |
| raildo | unfortunately, Moises who wrote the driver, is on PTO for a while, and we didn't have so much progress on those tests, until he come back, so I'll sync with him to have this done in the next couple weeks | 12:07 |
| jaosorior | raildo: so, the driver is scheduled to land in Stein, right? Are we also aiming to get some tripleo usage of this in Stein? or is that for the next release? | 12:07 |
| Tengu | weshay: https://bugs.launchpad.net/tripleo/+bug/1800737/comments/3 | 12:08 |
| openstack | Launchpad bug 1800737 in tripleo "relabel failed /var/lib/config-data: no such file or directory" [Critical,Triaged] | 12:08 |
| raildo | so, the driver will land on Stein, the idea is to implement some gate jobs over Castellan testing those scenarios, but I'm not expecting to have time to have some of this work on TripleO, it'll have to wait for the next release | 12:09 |
| jaosorior | got it | 12:09 |
| raildo | but would be great for TripleO start understanding/reviewing what we are doing in the Castellan/oslo side, since we'll probably be discussing about it more deeply in the next PTG | 12:09 |
| openstackgerrit | Sagi Shnaidman proposed openstack/tripleo-quickstart master: Make quickstack.sh avoid needing sudo on missing virtualenv https://review.openstack.org/613797 | 12:09 |
| *** ooolpbot has joined #tripleo | 12:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 12:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1798195 | 12:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1799895 | 12:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1800737 | 12:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1800742 | 12:10 |
| openstack | Launchpad bug 1798195 in tripleo "rdo-cloud yum repos unavailable during container updates and failing the undercloud install " [Critical,Triaged] | 12:10 |
| *** ooolpbot has quit IRC | 12:10 | |
| openstack | Launchpad bug 1799895 in tripleo "CI: undercloud takes long time which causes job fail with timeout" [Critical,Triaged] | 12:10 |
| openstack | Launchpad bug 1800737 in tripleo "relabel failed /var/lib/config-data: no such file or directory" [Critical,Triaged] | 12:10 |
| openstack | Launchpad bug 1800742 in tripleo "tempest.lib.exceptions.IdentityError: Got identity error, undercloud-containers" [Critical,Triaged] | 12:10 |
| raildo | that's all that I have for now | 12:10 |
| *** ramishra has joined #tripleo | 12:10 | |
| jaosorior | raildo: thanks for the update! | 12:10 |
| raildo | sure, no problem! | 12:10 |
| jaosorior | #topic SELinux for containers update | 12:11 |
| *** openstack changes topic to "SELinux for containers update (Meeting topic: TripleO Security Squad)" | 12:11 | |
| jaosorior | Tengu: ^^ | 12:11 |
| Tengu | hey, that's me :) | 12:11 |
| Tengu | so, basically, it's "mostly working" at least for: undercloud deploy with podman, overcloud deploy with podman and NON-HA (1 controller + 1 compute tested so far) | 12:11 |
| jaosorior | cool!! | 12:12 |
| Tengu | we have some non-selinux issues with the HA, so for now I can't say how will keepalived/pcmk/friends work with podman + selinux | 12:12 |
| Tengu | ah, and when I say "with selinux"; it's the full thing, meaning: enforcing + separation | 12:12 |
| jaosorior | what are the non-selinux issues? | 12:12 |
| Tengu | ah well, pcmk not supporting podman, that kind of things. | 12:12 |
| jaosorior | oh | 12:12 |
| jaosorior | :( | 12:12 |
| jaosorior | got it | 12:12 |
| Tengu | we need a specific package, already in the pipe. | 12:12 |
| Tengu | so yeah. for none-HA I get something working, still hitting some random issues with neutron though | 12:13 |
| Tengu | this bunch of containers do create issues, as they have high privileges and the like. | 12:13 |
| Tengu | speaking of privileges: we're currently unable to drop the "--privileged". | 12:13 |
| Tengu | this is mainly due to the fact we must support both docker and podman for a while, and apparently docker doesn't work well with the cap-add. | 12:14 |
| Tengu | so we stick with a bunch of "privileged" containers for now. | 12:14 |
| jaosorior | Tengu: do you have any more details related to that? | 12:14 |
| jaosorior | as far as I had understood, adding custom capabilities has for long been the recommended approach to locking down your docker deployment. | 12:15 |
| Tengu | also, some containers deactivate the labelling, dropping the selinux separation. This isn't great, but I don't have alternative for now. | 12:15 |
| jaosorior | and even docker were the ones that originally came up with the set of minimum "secure" defaults | 12:15 |
| Tengu | jaosorior: I didn't work on the cap-add part - bogdando did some tests with that and apparently it wasn't that great. | 12:15 |
| jaosorior | Tengu: that was to be expected, openshift does the same for their openvswitch container | 12:15 |
| jaosorior | bogdando: are you around? | 12:15 |
| *** panda is now known as panda|lunch | 12:15 | |
| Tengu | imho we should stick with the --privileged *for now*, as a first iteration. | 12:15 |
| jaosorior | Tengu: sure, it's not a blocker; and just having SELinux labeling enabled for most containers already reduces a lot the attack surface | 12:16 |
| jaosorior | I guess bogdando is not online right now, I'll ping him off this meeting just to understand things better. | 12:17 |
| jaosorior | Tengu: thanks! | 12:17 |
| Tengu | np :) | 12:17 |
| jaosorior | #topic TLS everywhere in CI | 12:17 |
| *** openstack changes topic to "TLS everywhere in CI (Meeting topic: TripleO Security Squad)" | 12:17 | |
| jaosorior | So, we did some work on this on the PTG | 12:18 |
| jaosorior | and agreed with the CI team that is was appropriate to keep this in OVB | 12:18 |
| jaosorior | sshnaidm|ruck: was mainly working on this, but the last I knew about it was that there were some issues with DNS | 12:18 |
| jaosorior | sshnaidm|ruck: is this still the case? or was there another issue that prevented this work from continuing? | 12:18 |
| openstackgerrit | Sagi Shnaidman proposed openstack/tripleo-quickstart master: Make quickstack.sh avoid needing sudo on missing virtualenv https://review.openstack.org/613797 | 12:19 |
| *** cylopez has joined #tripleo | 12:19 | |
| *** dprince has joined #tripleo | 12:20 | |
| sshnaidm|ruck | jaosorior, it's still the issue and rdo cloud breakages prevented from me to proceed with this unfortunately.. | 12:20 |
| jaosorior | sshnaidm|ruck: got it | 12:21 |
| *** cylopez has quit IRC | 12:21 | |
| jaosorior | sshnaidm|ruck: do you think (assuming RDO cloud would work in the near future), that we could get around these DNS issues? | 12:21 |
| openstackgerrit | Sorin Sbarnea proposed openstack/tripleo-quickstart master: Fix quickstart.sh --install-deps on fedora26 https://review.openstack.org/613797 | 12:21 |
| sshnaidm|ruck | jaosorior, yeah, the main problem is that you can't run it to test, not dns issue itself | 12:22 |
| jaosorior | sshnaidm|ruck: what do you mean? | 12:22 |
| sshnaidm|ruck | jaosorior, I mean that it can run only on private tenant of rdo cloud currently, and if it's broken - you can't run it to test | 12:23 |
| jaosorior | sshnaidm|ruck: right, so that's still related to the RDO cloud issues, right? | 12:24 |
| *** cylopez has joined #tripleo | 12:24 | |
| *** cylopez has left #tripleo | 12:24 | |
| weshay | jaosorior, morning.. we leaving the gate as is w/ mirror and podman issues? | 12:26 |
| jaosorior | We were discussing with my team the possiblity of adding the necessary things in the workflow to support the TLS everywhere setup without relying on FreeIPA discovery through DNS (so no DNS needed from FreeIPA). | 12:26 |
| jaosorior | sshnaidm|ruck: would that be helpful? or all the issue falls down due to the RDO cloud issues? | 12:26 |
| sshnaidm|ruck | jaosorior, I think it might be helpful | 12:27 |
| sshnaidm|ruck | jaosorior, as it's currently a blocker | 12:27 |
| jaosorior | sshnaidm|ruck: got it. I'll report back to my team to try to prioritize that. | 12:27 |
| sshnaidm|ruck | jaosorior, cool, thanks | 12:27 |
| jaosorior | sshnaidm|ruck: thanks for all the help man | 12:27 |
| weshay | jaosorior, can you comment on this epic with your thoughts or a looks good if it's ok atm https://tree.taiga.io/project/tripleo-ci-board/epic/298 | 12:28 |
| *** jchhatbar has joined #tripleo | 12:28 | |
| jaosorior | weshay: will do after this meeting | 12:28 |
| weshay | sorry | 12:28 |
| jaosorior | #topic Read Only policy files | 12:28 |
| *** openstack changes topic to "Read Only policy files (Meeting topic: TripleO Security Squad)" | 12:28 | |
| *** jchhatbar has quit IRC | 12:28 | |
| *** janki has quit IRC | 12:29 | |
| *** janki has joined #tripleo | 12:29 | |
| jaosorior | So... it's not uncommon that folks modify the policy files for their deployments. However, there are some policy modifications that are done time and time again | 12:29 |
| jaosorior | fixing these issues in the default policy files of OpenStack is work in progress (but it'll take several releases) | 12:30 |
| jaosorior | so some folks have suggested maintaining policy references (best practices) in a repo | 12:30 |
| jaosorior | these policies could be directly taken into use in deployments | 12:30 |
| jaosorior | so I was given the suggestion of including them under the TripleO umbrella | 12:31 |
| jaosorior | #link https://pagure.io/openstack-access-policy | 12:31 |
| jaosorior | That would be the work that's been done so far ^^ | 12:31 |
| jaosorior | the main change in those policy files is the inclusion of a read-only role | 12:31 |
| jaosorior | which allows folks not to require admin privileges to do operations like listing users and projects | 12:32 |
| jaosorior | which is nice :D | 12:32 |
| jaosorior | now, including it as part of TripleO would require us to test these files somehow (although this all becomes a bit easier with the standalone jobs) | 12:32 |
| jaosorior | Do people have thoughts about this? | 12:32 |
| jaosorior | For reference, these are the supported services for this kind of custom policy: | 12:33 |
| jaosorior | #link https://pagure.io/openstack-access-policy/blob/master/f/etc | 12:33 |
| *** ansmith has joined #tripleo | 12:34 | |
| openstackgerrit | Sagi Shnaidman proposed openstack/tripleo-quickstart master: Make quickstack.sh avoid needing sudo on missing virtualenv https://review.openstack.org/613797 | 12:35 |
| jaosorior | If there are no opinions about this on the security squad, I'll bring it up anyway to the weekly meeting next week. | 12:36 |
| jaosorior | #topic Any Other business | 12:37 |
| *** openstack changes topic to "Any Other business (Meeting topic: TripleO Security Squad)" | 12:37 | |
| jaosorior | Does anybody have something else to bring up to the security squad? | 12:37 |
| *** rlandy has joined #tripleo | 12:38 | |
| openstackgerrit | wes hayutin proposed openstack-infra/tripleo-ci master: ovb: reduce the number of workers on the undercloud https://review.openstack.org/613640 | 12:38 |
| jaosorior | Alright! thanks for joining everyone! | 12:39 |
| jaosorior | #endmeeting | 12:39 |
| *** openstack changes topic to "Welcome to Stein | CI Status: GREENish - don't mess this up | https://docs.openstack.org/tripleo-docs/latest/" | 12:39 | |
| openstack | Meeting ended Wed Oct 31 12:39:16 2018 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 12:39 |
| openstack | Minutes: http://eavesdrop.openstack.org/meetings/tripleo_security_squad/2018/tripleo_security_squad.2018-10-31-12.00.html | 12:39 |
| openstack | Minutes (text): http://eavesdrop.openstack.org/meetings/tripleo_security_squad/2018/tripleo_security_squad.2018-10-31-12.00.txt | 12:39 |
| openstack | Log: http://eavesdrop.openstack.org/meetings/tripleo_security_squad/2018/tripleo_security_squad.2018-10-31-12.00.log.html | 12:39 |
| openstackgerrit | Sagi Shnaidman proposed openstack/tripleo-quickstart master: Make quickstack.sh avoid needing sudo on missing virtualenv https://review.openstack.org/613797 | 12:40 |
| jaosorior | weshay: alright, checking out the epic you sent me out | 12:41 |
| weshay | chandankumar, arxcruz https://review.openstack.org/#/c/614321/ see nhicher's comment | 12:41 |
| chandankumar | weshay: on it | 12:42 |
| jaosorior | Getting acquainted with taiga too.. | 12:42 |
| weshay | jaosorior, ya.. thank you! | 12:42 |
| jaosorior | weshay: it looks good to me. I added myself to the "watching" list. | 12:42 |
| jaosorior | weshay: is there a field to comment there? | 12:42 |
| weshay | jaosorior, can you add a comment | 12:42 |
| weshay | ya.. at the bottom | 12:42 |
| jaosorior | I don't see a field | 12:42 |
| jaosorior | Do I need to be added to that group in taiga or something of the sort? | 12:43 |
| weshay | hrm.. /me logs out | 12:43 |
| weshay | ya.. authentication required :( | 12:43 |
| jaosorior | weshay: I'm logged in | 12:43 |
| jaosorior | so I guess it's a permissions issue | 12:43 |
| *** jpena is now known as jpena|lunch | 12:46 | |
| EmilienM | mwhahaha: see my comment on https://bugs.launchpad.net/tripleo/+bug/1800748$ | 12:49 |
| openstack | Launchpad bug 1800748 in tripleo "undercloud-containers job does not configure a docker mirror when podman is the container cli" [High,Triaged] | 12:49 |
| *** sshnaidm|ruck is now known as sshnaidm|bbl | 12:50 | |
| jaosorior | weshay: I think I need to be added here https://tree.taiga.io/project/tripleo-ci-board/team | 12:50 |
| openstackgerrit | Chandan Kumar proposed openstack/tripleo-quickstart-extras master: Add podman support to validate-tempest role https://review.openstack.org/605356 | 12:50 |
| chandankumar | EmilienM: ^^ | 12:50 |
| *** quiquell is now known as quiquell|lunch | 12:51 | |
| *** akrivoka has quit IRC | 12:51 | |
| *** akrivoka has joined #tripleo | 12:51 | |
| chandankumar | weshay: mwhahaha EmilienM https://review.openstack.org/#/c/606046/ what about just running smoke test for tempest here | 12:51 |
| chandankumar | ? | 12:51 |
| chandankumar | nhicher: weshay I am updating the same fs01 reivew with fs035 and fs053 since it is not merged if it is ok? | 12:53 |
| EmilienM | chandankumar: will review after my meetings | 12:54 |
| chandankumar | EmilienM: ack | 12:54 |
| bogdando | EmilienM: updated bz with fixed in version. what else can I do? | 12:56 |
| EmilienM | bogdando: all good. | 12:56 |
| *** gkadam has quit IRC | 12:57 | |
| *** panda|lunch is now known as panda | 13:02 | |
| *** gkadam has joined #tripleo | 13:03 | |
| openstackgerrit | Ronelle Landy proposed openstack/tripleo-quickstart-extras master: Resolve get_build_command output before container_build_id https://review.openstack.org/602734 | 13:07 |
| openstackgerrit | James Slagle proposed openstack/tripleo-heat-templates master: Make $SSH_OPTIONS overrideable https://review.openstack.org/614493 | 13:07 |
| *** agurenko has quit IRC | 13:08 | |
| *** raildo has quit IRC | 13:09 | |
| *** raildo_ has joined #tripleo | 13:09 | |
| *** agurenko has joined #tripleo | 13:10 | |
| *** ooolpbot has joined #tripleo | 13:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 13:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1798195 | 13:10 |
| openstack | Launchpad bug 1798195 in tripleo "rdo-cloud yum repos unavailable during container updates and failing the undercloud install " [Critical,Triaged] | 13:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1799895 | 13:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1800737 | 13:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1800742 | 13:10 |
| openstack | Launchpad bug 1799895 in tripleo "CI: undercloud takes long time which causes job fail with timeout" [Critical,Triaged] | 13:10 |
| *** ooolpbot has quit IRC | 13:10 | |
| openstack | Launchpad bug 1800737 in tripleo "relabel failed /var/lib/config-data: no such file or directory" [Critical,Triaged] | 13:10 |
| openstack | Launchpad bug 1800742 in tripleo "tempest.lib.exceptions.IdentityError: Got identity error, undercloud-containers" [Critical,Triaged] | 13:10 |
| openstackgerrit | Emilien Macchi proposed openstack/python-tripleoclient master: standalone/undercloud: run post_upgrade_tasks when upgrading https://review.openstack.org/614349 | 13:10 |
| openstackgerrit | Chris Jones proposed openstack/tripleo-common master: GenerateFencingParametersAction now works with uppercase MACs. https://review.openstack.org/614498 | 13:14 |
| *** mcornea has joined #tripleo | 13:18 | |
| *** agopi is now known as agopi|brb | 13:18 | |
| *** tzumainn has joined #tripleo | 13:19 | |
| rfolco|rover | EmilienM, any specific data or test to help you with runtime errors ? | 13:21 |
| EmilienM | rfolco|rover: Tengu and I are looking at it | 13:21 |
| EmilienM | rfolco|rover: https://bugs.launchpad.net/tripleo/+bug/1800737/ | 13:21 |
| openstack | Launchpad bug 1800737 in tripleo "relabel failed /var/lib/config-data: no such file or directory" [Critical,Triaged] | 13:21 |
| EmilienM | we're still looking for a root cause | 13:22 |
| EmilienM | Tengu: I wonder if it's because the container pull failed | 13:22 |
| *** janki has quit IRC | 13:22 | |
| EmilienM | so the relabel would fail | 13:22 |
| *** janki has joined #tripleo | 13:23 | |
| Tengu | would be great to find out WHERE that relabel happens actually. | 13:23 |
| EmilienM | Tengu: during the container_cli run? | 13:23 |
| *** agopi|brb has quit IRC | 13:24 | |
| Tengu | nah, if podman tries to run a container with a volume that doesn't exist on the host, it fails with a clean error | 13:24 |
| openstackgerrit | Dan Prince proposed openstack/tripleo-heat-templates master: Disable MistralDockerGroup creation with podman https://review.openstack.org/614500 | 13:25 |
| Tengu | hmm. I see two times the zaqar container as "failing" with that. | 13:25 |
| dprince | Tengu: ^^ see the MistralDockerGroup above | 13:25 |
| Tengu | dprince: you say "disable" while still setting it to true? | 13:26 |
| Tengu | I think you want "false" | 13:26 |
| dprince | Tengu: sigh, yes. I do | 13:26 |
| Tengu | :) | 13:26 |
| openstackgerrit | Dan Prince proposed openstack/tripleo-heat-templates master: Disable MistralDockerGroup creation with podman https://review.openstack.org/614500 | 13:26 |
| Tengu | dprince: let's see how zuul eats it. | 13:27 |
| *** sambetts|afk is now known as sambetts | 13:28 | |
| *** ade_lee has joined #tripleo | 13:28 | |
| *** aufi has quit IRC | 13:29 | |
| *** slaweq has quit IRC | 13:29 | |
| EmilienM | Tengu: and we have label=disable | 13:30 |
| EmilienM | Tengu: for podman only. | 13:31 |
| EmilienM | which should be good | 13:31 |
| EmilienM | Tengu: the zaqar image was pulled: http://logs.openstack.org/58/609858/6/gate/tripleo-ci-centos-7-containers-multinode/27bedd3/logs/undercloud/var/log/extra/podman/podman_allinfo.log.txt.gz | 13:34 |
| Tengu | I'm wondering if the issue is really the relabelling. | 13:35 |
| Tengu | the error just before that is the docker-puppet.py thingy.... | 13:36 |
| EmilienM | Tengu: the error happens during the log.debug('Running %s command: %s' % (container_cli, ' '.join(dcmd))) | 13:36 |
| EmilienM | in docker/docker-puppet.py | 13:36 |
| EmilienM | for sure | 13:36 |
| openstackgerrit | Yurii Prokulevych proposed openstack/tripleo-upgrade master: Add container image prepare script to upgrade workflow. https://review.openstack.org/614503 | 13:38 |
| *** karthiks has joined #tripleo | 13:42 | |
| karthiks | dsneddon, ping | 13:44 |
| *** amoralej is now known as amoralej|lunch | 13:47 | |
| *** jpena|lunch is now known as jpena | 13:48 | |
| *** agopi has joined #tripleo | 13:48 | |
| openstackgerrit | Sorin Sbarnea proposed openstack/tripleo-quickstart master: wip: enable testing using ansible molecule https://review.openstack.org/613672 | 13:49 |
| openstackgerrit | Marios Andreou proposed openstack-infra/tripleo-ci master: WIP: expose zuul variables for reproducer https://review.openstack.org/613571 | 13:51 |
| *** vinaykns has joined #tripleo | 13:54 | |
| openstackgerrit | Ronelle Landy proposed openstack/tripleo-quickstart-extras master: Add v3 project domain settings required for stacks/keypairs https://review.openstack.org/611674 | 13:56 |
| jaosorior | rlandy: commented ^^ | 13:57 |
| rlandy | jaosorior: thanks - correcting | 13:57 |
| openstackgerrit | Alex Schultz proposed openstack-infra/tripleo-ci master: WIP: Container build CI job https://review.openstack.org/614392 | 13:58 |
| openstackgerrit | Ronelle Landy proposed openstack/tripleo-quickstart master: DNM - Testing no undercloud cleaning https://review.openstack.org/611602 | 13:59 |
| *** quiquell|lunch is now known as quiquell | 14:00 | |
| jrist | chem: are you ok with the discussion here? https://review.openstack.org/#/c/611305/ | 14:00 |
| jrist | chem: can we get this merged now? | 14:00 |
| jrist | (yes I realize gate might still be hosed) | 14:00 |
| jrist | EmilienM: is gate still hosed? | 14:00 |
| *** raildo_ has quit IRC | 14:00 | |
| openstackgerrit | Ronelle Landy proposed openstack/tripleo-ha-utils master: DNM - Work with zuul config copies https://review.openstack.org/610570 | 14:01 |
| openstackgerrit | Ronelle Landy proposed openstack-infra/tripleo-ci master: Add tripleo-ha-utils to extra-requirements https://review.openstack.org/610508 | 14:01 |
| EmilienM | jrist: I don't haven numbers | 14:02 |
| EmilienM | but i've seen a bunch of errors | 14:02 |
| jrist | :( | 14:02 |
| chem | jrist: hi, removed the -1, but not sure why we need to maintain 2 plans, but I guess there is something here I'm missing. | 14:04 |
| chem | jrist: good luck with the gate | 14:04 |
| chem | "you shall not pass!" | 14:05 |
| EmilienM | rfolco|rover: do we have nubmers on gate? | 14:05 |
| jrist | chem: is there another plan? I guess I misunderstood your -1 | 14:05 |
| jrist | chem: I believe there is only one "default plan" and it's "overcloud" | 14:06 |
| rfolco|rover | EmilienM, stats on which jobs affected, % rate? | 14:06 |
| openstackgerrit | Merged openstack/tripleo-heat-templates stable/rocky: Delete empty karaf directory on host https://review.openstack.org/613881 | 14:06 |
| openstackgerrit | Merged openstack/os-collect-config stable/queens: Don't ignore SIGPIPE https://review.openstack.org/609470 | 14:06 |
| EmilienM | rfolco|rover: stats on % of failure in gate, not sure we have that number | 14:06 |
| rfolco|rover | let me see if i can get something, I am looking at elastic search now | 14:07 |
| mwhahaha | rlandy, sshnaidm|bbl, weshay: can we please get https://review.openstack.org/#/c/610102/ (it was holding up some python3 patches) | 14:09 |
| *** ooolpbot has joined #tripleo | 14:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 14:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1798195 | 14:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1799895 | 14:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1800737 | 14:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1800742 | 14:10 |
| openstack | Launchpad bug 1798195 in tripleo "rdo-cloud yum repos unavailable during container updates and failing the undercloud install " [Critical,Triaged] | 14:10 |
| *** ooolpbot has quit IRC | 14:10 | |
| openstack | Launchpad bug 1799895 in tripleo "CI: undercloud takes long time which causes job fail with timeout" [Critical,Triaged] | 14:10 |
| openstack | Launchpad bug 1800737 in tripleo "relabel failed /var/lib/config-data: no such file or directory" [Critical,Triaged] | 14:10 |
| openstack | Launchpad bug 1800742 in tripleo "tempest.lib.exceptions.IdentityError: Got identity error, undercloud-containers" [Critical,Triaged] | 14:10 |
| openstackgerrit | Quique Llorente proposed openstack/tripleo-quickstart-extras master: Upgrade rdopkg to 0.47.3 https://review.openstack.org/614516 | 14:12 |
| openstackgerrit | Quique Llorente proposed openstack/tripleo-quickstart-extras master: Upgrade rdopkg to 0.47.3 https://review.openstack.org/614516 | 14:12 |
| bnemec | Are we still claiming ci is green-ish with a 24 hour gate queue? | 14:13 |
| bnemec | Also, weshay, I don't think I can comment on that taiga task. Does someone need to give me access? | 14:13 |
| openstackgerrit | Marios Andreou proposed openstack/tripleo-quickstart-extras master: WIP: add zuul variables for reproducer https://review.openstack.org/613572 | 14:13 |
| *** lblanchard has joined #tripleo | 14:14 | |
| *** d0ugal has quit IRC | 14:14 | |
| quiquell | chem, jistr: forgot yesterday about git review the postarg of emit releases unit test review | 14:15 |
| quiquell | chem, jistr: https://review.openstack.org/#/c/611842/9 | 14:16 |
| weshay | bnemec, ya.. sorry I didn't realize that when I asked :( | 14:17 |
| weshay | will try to fix and let you know | 14:17 |
| bnemec | weshay: No worries, but I do have some thoughts to add so it would be good if I could do that there instead of through email. | 14:18 |
| EmilienM | jaosorior: I think we should stop approving things | 14:18 |
| EmilienM | the gate is 24h | 14:18 |
| EmilienM | mwhahaha: ^ | 14:18 |
| mwhahaha | i'd like to switch all scenarios to non-voting | 14:18 |
| mwhahaha | and pull them out of the gate | 14:19 |
| EmilienM | cedric and I are looking at this /var/lib/config-data | 14:19 |
| mwhahaha | EmilienM: are you sure that was there with docker? | 14:19 |
| jaosorior | EmilienM: OK, I'll purge the gate. Lets update the status on this channel | 14:19 |
| jaosorior | so folks are aware | 14:19 |
| chem | quiquell: nice :) | 14:20 |
| openstackgerrit | Derek Higgins proposed openstack-infra/tripleo-ci master: Add job for testing Ironic in the Overcloud https://review.openstack.org/582294 | 14:22 |
| chem | morazi: so I hack together something but this needs testing. There is no real rational to push it to another release IMHO, so at least it's there for the taking | 14:23 |
| morazi | chem, ack | 14:24 |
| jaosorior | gate is clear. | 14:26 |
| *** Vorrtex has joined #tripleo | 14:26 | |
| mwhahaha | ok so now we need to figure out how we fix this | 14:26 |
| *** d0ugal has joined #tripleo | 14:27 | |
| EmilienM | mwhahaha: can you +o jaosorior plz | 14:28 |
| *** mwhahaha sets mode: +o jaosorior | 14:28 | |
| *** mwhahaha sets mode: +o EmilienM | 14:28 | |
| *** jaosorior changes topic to "DO NOT APPROVE PATCHES | CI Status: RED | https://docs.openstack.org/tripleo-docs/latest/" | 14:29 | |
| *** raildo has joined #tripleo | 14:30 | |
| derekh | marios: shardy thanks for looking at the scenario012 patch, I've updated the patch that adds the new job https://review.openstack.org/#/c/582294/ (there is aslo another depended patch if ye have the time to look at it) | 14:30 |
| *** d0ugal has quit IRC | 14:32 | |
| rfolco|rover | EmilienM, while panic runtime errors show 243 hits since 10/21, panic runtime error + relabel error started at 10/26 and has 102 hits. Continue to gather more data. | 14:32 |
| EmilienM | rfolco|rover: panic runtime error is red hearing | 14:33 |
| EmilienM | it is not the root cause of failing jobs | 14:33 |
| EmilienM | if it is please show me and let me know | 14:33 |
| EmilienM | the problem is relabel I think | 14:33 |
| rfolco|rover | yeah since 10/26 then | 14:33 |
| mwhahaha | so what merged on 10/26 to cause the relabel problem | 14:34 |
| EmilienM | logstash shows me relabel error from October 22th | 14:34 |
| EmilienM | but i'm still digging | 14:34 |
| openstackgerrit | Marios Andreou proposed openstack-infra/tripleo-ci master: WIP: expose zuul variables for reproducer https://review.openstack.org/613571 | 14:34 |
| rfolco|rover | ah ok, the two errors are disconnect, refreshing my query | 14:35 |
| EmilienM | http://logstash.openstack.org/#dashboard/file/logstash.json?query=build_status%3A%20FAILURE%20AND%20message%3A%20%5C%22relabel%20failed%5C%22 | 14:35 |
| EmilienM | build_status: FAILURE AND message: "relabel failed" | 14:35 |
| marios | added to my reviews list derekh ack | 14:35 |
| openstackgerrit | Merged openstack/tripleo-ui stable/rocky: Imported Translations from Zanata https://review.openstack.org/614412 | 14:36 |
| openstackgerrit | Merged openstack/tripleo-ui stable/queens: Imported Translations from Zanata https://review.openstack.org/614410 | 14:36 |
| rfolco|rover | hmm are you sure you see relabel from 10/22 ? I see relabel failed since 10/26 w/ 142 hits | 14:37 |
| rfolco|rover | EmilienM, ^ | 14:37 |
| rfolco|rover | unless the job did not fail | 14:37 |
| mwhahaha | yea same | 14:38 |
| mwhahaha | i don't see any hits from 10/22 in logstash (i looked 30 days back) | 14:38 |
| EmilienM | sorry | 14:38 |
| EmilienM | you are right | 14:38 |
| EmilienM | so: | 14:39 |
| EmilienM | label issue since Oct 26th | 14:39 |
| EmilienM | podman panic since forever but red hearing and in progress by podman team | 14:39 |
| EmilienM | https://review.openstack.org/#/q/status:merged+age:4days+tripleo | 14:40 |
| mwhahaha | first one was 6am | 14:40 |
| mwhahaha | so wonder what build that was | 14:40 |
| *** slaweq has joined #tripleo | 14:41 | |
| *** saneax has quit IRC | 14:42 | |
| larsks | What could cause config download (osp13/queens) get stuck? It's not clear if it's actually hung or just really, really slow... | 14:42 |
| openstackgerrit | Pierre Riteau proposed openstack/diskimage-builder master: Increase size of EFI system partition https://review.openstack.org/614526 | 14:45 |
| *** aufi has joined #tripleo | 14:45 | |
| openstackgerrit | Quique Llorente proposed openstack-infra/tripleo-ci master: DNM: Try to dump vars from inventory https://review.openstack.org/614528 | 14:46 |
| *** quiquell is now known as quiquell|brb | 14:48 | |
| *** mburned is now known as mburned_out | 14:50 | |
| *** mburned_out is now known as mburned | 14:50 | |
| EmilienM | mwhahaha: so logtash says it first failed on this one: https://review.openstack.org/#/c/604757/ | 14:52 |
| EmilienM | http://logs.openstack.org/57/604757/7/gate/tripleo-ci-centos-7-undercloud-containers/17acf7b/logs/undercloud/home/zuul/undercloud_install.log.txt.gz#_2018-10-26_23_32_48 | 14:52 |
| EmilienM | maybe I'm blind but in the patch I don't see the failing job | 14:52 |
| mwhahaha | EmilienM: cause it failed but the gate reset | 14:53 |
| mwhahaha | https://www.diffchecker.com/ez7vmmvu | 14:53 |
| mwhahaha | so from that change | 14:53 |
| EmilienM | ok | 14:53 |
| mwhahaha | i took a job from 10/23 vs the failed logs | 14:53 |
| EmilienM | ok nice you diffed alraedy | 14:53 |
| mwhahaha | those are the package diffs | 14:53 |
| EmilienM | excellent | 14:53 |
| mwhahaha | http://logs.openstack.org/57/604757/7/gate/tripleo-ci-centos-7-undercloud-containers/17acf7b/ was the failed run | 14:53 |
| EmilienM | we need to bisect tripleo-common, THT and paunch | 14:54 |
| EmilienM | also it could be the podman version which broke the security-opts | 14:54 |
| mwhahaha | i checked the patches that were infront of the change and i don't see anything specific | 14:54 |
| EmilienM | so podman? | 14:54 |
| mwhahaha | i'd guess it was podman from that diff but we weren't running podman by default were we? | 14:55 |
| *** amoralej|lunch is now known as amoralej | 14:55 | |
| EmilienM | for undercloud-containers yes | 14:55 |
| mwhahaha | ok yea that's kinda why i wanted to revert it last night | 14:55 |
| *** d0ugal has joined #tripleo | 14:55 | |
| mwhahaha | a relabel thing in containers points to the container engine | 14:55 |
| mwhahaha | so yea all these failures are in containers-multindoe and udnercloud-containers | 14:56 |
| *** quiquell|brb is now known as quiquell | 14:56 | |
| mwhahaha | (or upgrade job) | 14:56 |
| mwhahaha | so we need to revert podman | 14:56 |
| mwhahaha | for everything | 14:56 |
| mwhahaha | EmilienM: please un -2 https://review.openstack.org/#/c/614364/ | 14:57 |
| EmilienM | mwhahaha: I'm fine with a revert *now* | 14:57 |
| EmilienM | since the issue is really identified | 14:57 |
| EmilienM | because the podman pull thing was red hearing and I didn't want to revert for that reason | 14:57 |
| EmilienM | mwhahaha: there is 3 patches to revert | 14:57 |
| EmilienM | we ahve 3 jobs | 14:57 |
| EmilienM | also we need to purge the gate | 14:58 |
| mwhahaha | the gate is purged | 14:58 |
| EmilienM | IMHO we should not revert, but disable podman in 1 patch | 14:58 |
| EmilienM | 3 reverts increases the chance to hit the error in gate | 14:58 |
| EmilienM | no? | 14:58 |
| mwhahaha | sure propose a patch | 14:58 |
| EmilienM | ok | 14:58 |
| EmilienM | I'm stashing them | 14:59 |
| *** dsneddon has quit IRC | 14:59 | |
| *** slaweq has quit IRC | 15:01 | |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-quickstart master: (squash) disabling podman everywhere in gate https://review.openstack.org/614537 | 15:03 |
| EmilienM | mwhahaha: ^ done | 15:03 |
| mwhahaha | k | 15:03 |
| weshay | ah.. thanks guys | 15:04 |
| weshay | rfolco|rover, sshnaidm|bbl ^ | 15:05 |
| *** jistr is now known as jistr|call | 15:05 | |
| openstackgerrit | Quique Llorente proposed openstack-infra/tripleo-ci master: DNM: Try to dump vars from inventory https://review.openstack.org/614528 | 15:05 |
| *** dsneddon has joined #tripleo | 15:08 | |
| *** ooolpbot has joined #tripleo | 15:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 15:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1798195 | 15:10 |
| openstack | Launchpad bug 1798195 in tripleo "rdo-cloud yum repos unavailable during container updates and failing the undercloud install " [Critical,Triaged] | 15:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1799895 | 15:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1800737 | 15:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1800742 | 15:10 |
| *** ooolpbot has quit IRC | 15:10 | |
| openstack | Launchpad bug 1799895 in tripleo "CI: undercloud takes long time which causes job fail with timeout" [Critical,Triaged] | 15:10 |
| openstack | Launchpad bug 1800737 in tripleo "relabel failed /var/lib/config-data: no such file or directory" [Critical,Triaged] | 15:10 |
| openstack | Launchpad bug 1800742 in tripleo "tempest.lib.exceptions.IdentityError: Got identity error, undercloud-containers" [Critical,Triaged] | 15:10 |
| *** rpioso|afk is now known as rpioso | 15:11 | |
| rpioso | Good morning | 15:11 |
| openstackgerrit | Yurii Prokulevych proposed openstack/tripleo-upgrade master: Add container image prepare script to upgrade workflow. https://review.openstack.org/614503 | 15:11 |
| *** dsneddon has quit IRC | 15:12 | |
| *** ccamacho has quit IRC | 15:13 | |
| openstackgerrit | Quique Llorente proposed openstack-infra/tripleo-ci master: DNM: Try to dump vars from inventory https://review.openstack.org/614528 | 15:13 |
| openstackgerrit | Harald Jensås proposed openstack/tripleo-heat-templates master: Move UndercloudPostDeployment to python https://review.openstack.org/614540 | 15:16 |
| openstackgerrit | Quique Llorente proposed openstack/tripleo-quickstart-extras master: Use dnf and python3 on platforms where these are default https://review.openstack.org/610491 | 15:17 |
| openstackgerrit | John Fulton proposed openstack/tripleo-heat-templates master: Add additional whitespace around "ansible_connection: local" https://review.openstack.org/613793 | 15:19 |
| openstackgerrit | Harald Jensås proposed openstack/tripleo-heat-templates master: Move UndercloudPostDeployment to python https://review.openstack.org/614540 | 15:22 |
| openstackgerrit | Quique Llorente proposed openstack/tripleo-quickstart-extras master: DNM: squash to test f28 py3 job https://review.openstack.org/614551 | 15:27 |
| *** ksambor has quit IRC | 15:29 | |
| *** slaweq has joined #tripleo | 15:30 | |
| *** yprokule has quit IRC | 15:31 | |
| *** jistr|call is now known as jistr | 15:35 | |
| openstackgerrit | Quique Llorente proposed openstack/tripleo-quickstart-extras master: DNM: squash to test standalone-f28 https://review.openstack.org/613316 | 15:37 |
| *** quiquell is now known as quiquell|off | 15:38 | |
| cgoncalves | is there a parameter one can use in THT/docker/services to get the distro the undercloud runs on? | 15:41 |
| mwhahaha | cgoncalves: no | 15:41 |
| cgoncalves | I want to append to the volumes list a directory conditional on the distro | 15:41 |
| cgoncalves | ;( | 15:42 |
| cgoncalves | mwhahaha, ok, asked for the octavia image thing | 15:42 |
| mwhahaha | why would you need to know the distro? | 15:42 |
| openstackgerrit | Alan Bishop proposed openstack/puppet-tripleo master: Handle deprecated cinder iSCSI parameters https://review.openstack.org/614557 | 15:42 |
| cgoncalves | mwhahaha, to add /usr/share/rhos-director-images to the volumes if RHEL | 15:43 |
| mwhahaha | cgoncalves: that's a downstream patch | 15:43 |
| cgoncalves | mwhahaha, hmm ok. I was trying to find an acceptable way upstream first | 15:44 |
| openstackgerrit | Sorin Sbarnea proposed openstack/tripleo-quickstart master: Make quickstack.sh avoid needing sudo on missing virtualenv https://review.openstack.org/613797 | 15:44 |
| mwhahaha | cgoncalves: alternatively do a host_prep_task to create a generic place to copy the assumed image to | 15:44 |
| mwhahaha | cgoncalves: it'd probably be best to create a new undercloud.conf setting for that location and just always include the value of that | 15:45 |
| mwhahaha | cgoncalves: i think we already customize the mistral volumes for the undercloud anyway (if that is where you were going to mount it) | 15:47 |
| *** itlinux has joined #tripleo | 15:47 | |
| *** kopecmartin is now known as kopecmartin|off | 15:49 | |
| cgoncalves | mwhahaha, oh, ok, I have to check how the undercloud.conf looks currently | 15:56 |
| openstackgerrit | Alex Schultz proposed openstack-infra/tripleo-ci master: Switch scenarios to non-voting https://review.openstack.org/614566 | 15:59 |
| openstackgerrit | Alex Schultz proposed openstack/tripleo-heat-templates master: Remove non-voting scenarios from gate https://review.openstack.org/614567 | 15:59 |
| openstackgerrit | RedHat RDO CI proposed openstack/tripleo-heat-templates stable/queens: GATE CHECK for TripleO https://review.openstack.org/567224 | 16:00 |
| openstackgerrit | RedHat RDO CI proposed openstack/tripleo-heat-templates stable/rocky: GATE CHECK for TripleO https://review.openstack.org/604293 | 16:00 |
| openstackgerrit | Alex Schultz proposed openstack/tripleo-common master: Remove non-voting scenarios from gate https://review.openstack.org/614570 | 16:01 |
| openstackgerrit | Alex Schultz proposed openstack/tripleo-quickstart master: Remove non-voting scenarios from gate https://review.openstack.org/614571 | 16:02 |
| mwhahaha | EmilienM, weshay, jaosorior: https://review.openstack.org/#/q/topic:reduce-tripleo-usage+(status:open+OR+status:merged) a proposal | 16:03 |
| hjensas | I am seeing this error in every other overcloud stack create|stack update today in my local test env: | 16:03 |
| hjensas | fatal: [overcloud-compute-leaf1-1]: FAILED! => {"msg": "Error in jmespath.search in json_query filter plugin:\ninvalid literal for int() with base 10: '-'"} | 16:03 |
| hjensas | Any pointers troubleshooting? It's task: TASK [Lookup deployment UUID] | 16:04 |
| weshay | mwhahaha, rock on | 16:04 |
| weshay | rfolco|rover, ^ | 16:04 |
| weshay | thanks | 16:04 |
| weshay | sshnaidm|bbl, ^ | 16:04 |
| *** agopi is now known as agopi|food | 16:04 | |
| jaosorior | mwhahaha: what was the background on that? | 16:05 |
| jaosorior | ah | 16:06 |
| jaosorior | saw the commit message now | 16:06 |
| openstackgerrit | Jose Luis Franco proposed openstack/tripleo-common master: Add general LOG in kolla_builder.py. https://review.openstack.org/599346 | 16:06 |
| mwhahaha | jaosorior: see https://review.openstack.org/#/c/614566/ | 16:06 |
| *** ooolpbot has joined #tripleo | 16:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 16:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1798195 | 16:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1799895 | 16:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1800737 | 16:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1800742 | 16:10 |
| openstack | Launchpad bug 1798195 in tripleo "rdo-cloud yum repos unavailable during container updates and failing the undercloud install " [Critical,Triaged] | 16:10 |
| *** ooolpbot has quit IRC | 16:10 | |
| openstack | Launchpad bug 1799895 in tripleo "CI: undercloud takes long time which causes job fail with timeout" [Critical,Triaged] | 16:10 |
| openstack | Launchpad bug 1800737 in tripleo "relabel failed /var/lib/config-data: no such file or directory" [Critical,Triaged] | 16:10 |
| openstack | Launchpad bug 1800742 in tripleo "tempest.lib.exceptions.IdentityError: Got identity error, undercloud-containers" [Critical,Triaged] | 16:10 |
| openstackgerrit | Janki Chhatbar proposed openstack/tripleo-heat-templates stable/queens: Delete empty karaf directory on host https://review.openstack.org/614578 | 16:11 |
| openstackgerrit | Sorin Sbarnea proposed openstack/tripleo-quickstart master: wip: enable testing using ansible molecule https://review.openstack.org/613672 | 16:12 |
| *** janki has quit IRC | 16:13 | |
| *** dsneddon has joined #tripleo | 16:15 | |
| *** jfrancoa has quit IRC | 16:17 | |
| *** gkadam has quit IRC | 16:20 | |
| *** agurenko has quit IRC | 16:24 | |
| weshay | mwhahaha, jaosorior do you think we should do this right now? https://review.openstack.org/#/c/614566/1/zuul.d/multinode-jobs.yaml w/o w/o announcing at a tripleo mtg? | 16:26 |
| mwhahaha | so we can send a ML note | 16:28 |
| mwhahaha | i'll leave it up to you people, my take is that we need those converted to standalone like yesterday and we are consuming far to many resources with them | 16:30 |
| mwhahaha | ovb needs to get fixed and we should be using that for complex testing | 16:30 |
| weshay | mwhahaha, ok.. so help me decide.. everything takes longer than we want it to.. atm most of the team is working on f28 standalone | 16:31 |
| * jaosorior in a meeting right now | 16:31 | |
| weshay | then we'll work on converting the multinode jobs to standalone | 16:31 |
| jaosorior | what's the current status on converting to standalone? | 16:31 |
| weshay | https://tree.taiga.io/project/tripleo-ci-board/wiki/notes-from-pm | 16:32 |
| weshay | jaosorior, working on f28, not started on scenario01 | 16:32 |
| weshay | panda, ^ | 16:32 |
| weshay | is that accurate | 16:32 |
| mwhahaha | weshay: so they'll still run in promotion, so it's not like we won't have a blocking check | 16:32 |
| jaosorior | is f28 a prerequisite for standalone? | 16:32 |
| panda | weshay: yes, no work on scenario conversion yet | 16:33 |
| panda | jaosorior: no but we're locking them together | 16:33 |
| mwhahaha | it's not exactly a prereq but we do need to get the f28 versions of the scenarios also going at some point for promotion pipelines | 16:34 |
| weshay | gernally taking longer because it's been very difficult to merge patches afaict.. panda may know more | 16:34 |
| mwhahaha | right which this should actually help with | 16:34 |
| *** phuongnh has joined #tripleo | 16:34 | |
| weshay | mwhahaha, yes, was just talking about that... https://tree.taiga.io/project/tripleo-ci-board/us/192?milestone=203564 | 16:34 |
| *** pcaruana|elisa| has quit IRC | 16:37 | |
| jaosorior | I don't know if this makes sense. But wouldn't it be better to port the jobs to standalone first, and after that start switching to f28? Given that it standalone would reduce the resources we're using, and help merge stuff faster | 16:38 |
| *** phuongnh has quit IRC | 16:39 | |
| *** trown is now known as trown|lunch | 16:40 | |
| *** gfidente is now known as gfidente|afk | 16:40 | |
| dtantsur | is it fair to say that we're paying the price for a small number of really tightly coupled projects with a lot of patches? | 16:41 |
| dtantsur | and that decoupling things from tripleo core may be the only ultimate answer.. | 16:41 |
| mwhahaha | we're paying the price for complexity and instability | 16:42 |
| mwhahaha | even if it's all tht, we have a bunch of patches | 16:42 |
| mwhahaha | the probem with splitting them out is being able to rely on the patches to tripleo-common/tht/tripleoclient would require constant releasing | 16:42 |
| dtantsur | I mean, if we had more projects that are less coupled, we would not have to co-gate them | 16:42 |
| dtantsur | well, oslo has the same problem, and they release like every week or every other week | 16:42 |
| dtantsur | release numbers are cheap, release automation is great :) | 16:43 |
| mwhahaha | but then we have to land the patches to bump everything | 16:43 |
| dtantsur | mwhahaha: only when you need it | 16:44 |
| mwhahaha | which is more often than i like :D | 16:44 |
| dtantsur | sure :D | 16:44 |
| mwhahaha | even if they were seperate we still have gate issues | 16:44 |
| mwhahaha | primarily because of long complex jobs | 16:44 |
| dtantsur | right, I'm mostly talking about the problem that the gate gets backed up immediately on any issue | 16:45 |
| mwhahaha | which won't help anything | 16:45 |
| mwhahaha | the valume of patches + instability is the issue | 16:45 |
| mwhahaha | not the projects | 16:45 |
| mwhahaha | and the fact that we take out capacity for all the other projects and check queue as well doesn't help | 16:46 |
| roger2 | I'm trying to deploy an overcloud. The controller failed to deploy. I'm following the troubleshooting docs, but need some help. "openstack stack resource list overcloud" says "Controller" got UPDATE_FAILED. "openstack stack resource show overcloud Controller" shows "resource_status_reason | UPDATE aborted (Task update from ResourceGroup "Controller" [4f50d4...] Stack "overcloud" [d31e3e...] Timed out)". | 16:48 |
| roger2 | "openstack server list" shows overcloud-controller-0 with status BUILD and no networks. My env is queens attempting to deploy to physical nodes with the baremetal flavor. | 16:48 |
| roger2 | I'm searching logs and not finding any relevant errors yet. | 16:48 |
| mwhahaha | roger2: if it's in build check nova/ironic logs as to why it's not provisioning. perhaps ipmi issues? | 16:49 |
| mwhahaha | weshay: so my take on it is that the scenario jobs will still live in the pipeline so issues will be raised appropriately but by pulling them from the gate we increase the likely hood that merges will continue with less issues. | 16:51 |
| mwhahaha | weshay: we reduce our upstream usage for the gate and reduce the risk of timeouts causing gate resets, etc. | 16:51 |
| *** rh-jelabarre has quit IRC | 16:52 | |
| cgoncalves | mwhahaha, can't find where mistral volumes for the UC are being customized. care to point me please? | 16:52 |
| *** chandankumar is now known as chkumar|off | 16:53 | |
| mwhahaha | cgoncalves: https://github.com/openstack/tripleo-heat-templates/blob/master/environments/undercloud.yaml#L109 | 16:55 |
| *** tongl_ has joined #tripleo | 16:56 | |
| mwhahaha | cgoncalves: you likely could just hard code that, i think docker/podman will just create an empty folder if it doesn't exist. check with EmilienM | 16:56 |
| cgoncalves | mwhahaha, thanks! I looked everywhere, instack-undercloud included | 16:56 |
| mwhahaha | also https://github.com/openstack/tripleo-heat-templates/blob/master/environments/undercloud.yaml#L146 | 16:56 |
| *** shardy has quit IRC | 16:56 | |
| mwhahaha | you probably want to change L146 | 16:56 |
| *** shardy_ has quit IRC | 16:56 | |
| jaosorior | sshnaidm|bbl: still around? | 16:57 |
| cgoncalves | mwhahaha, so you're suggesting appending /usr/share/rhos-director-images to L146 regardless of distro? hmm | 16:58 |
| mwhahaha | cgoncalves: yea | 16:58 |
| *** rh-jelabarre has joined #tripleo | 16:59 | |
| *** agopi|food is now known as agopi | 16:59 | |
| cgoncalves | well, that would be a quick and easy fix :) | 16:59 |
| *** jpich has quit IRC | 16:59 | |
| weshay | mwhahaha, that is reasonable | 17:02 |
| weshay | mwhahaha, however I still think we should expose that to more folks before doing it | 17:03 |
| *** udesale has quit IRC | 17:03 | |
| *** dsneddon has quit IRC | 17:10 | |
| *** ooolpbot has joined #tripleo | 17:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 17:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1798195 | 17:10 |
| openstack | Launchpad bug 1798195 in tripleo "rdo-cloud yum repos unavailable during container updates and failing the undercloud install " [Critical,Triaged] | 17:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1799895 | 17:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1800737 | 17:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1800742 | 17:10 |
| openstack | Launchpad bug 1799895 in tripleo "CI: undercloud takes long time which causes job fail with timeout" [Critical,Triaged] | 17:10 |
| *** ooolpbot has quit IRC | 17:10 | |
| openstack | Launchpad bug 1800737 in tripleo "relabel failed /var/lib/config-data: no such file or directory" [Critical,Triaged] | 17:10 |
| openstack | Launchpad bug 1800742 in tripleo "tempest.lib.exceptions.IdentityError: Got identity error, undercloud-containers" [Critical,Triaged] | 17:10 |
| chem | weshay: EmilienM hey, I've got SIGSEGV for podman pulling an image, is that something already seen before ? | 17:12 |
| weshay | chem, ya | 17:12 |
| weshay | it's getting pulled out for now | 17:12 |
| chem | sigsegv, that's refreshing :) | 17:12 |
| chem | weshay: oki, so I just recheck my job then ? | 17:13 |
| weshay | chem, not yet | 17:13 |
| chem | weshay: oki, holding on then. | 17:13 |
| chem | weshay: what should I track to know when ? | 17:13 |
| weshay | chem, another day another dolla | 17:13 |
| chem | hehe | 17:13 |
| weshay | chem, it's on the list openstack-dev[tripleo], the patch to watch is https://review.openstack.org/#/c/614537/ | 17:14 |
| chem | weshay:thanks | 17:14 |
| openstackgerrit | Alex Schultz proposed openstack/puppet-tripleo master: Remove non-voting scenarios from gate https://review.openstack.org/614593 | 17:17 |
| openstackgerrit | Marios Andreou proposed openstack/tripleo-quickstart-extras master: WIP: add zuul variables for reproducer https://review.openstack.org/613572 | 17:18 |
| *** dsneddon has joined #tripleo | 17:21 | |
| dsneddon | karthiks, You were looking for me? | 17:22 |
| *** hjensas has quit IRC | 17:22 | |
| mwhahaha | weshay: done, http://lists.openstack.org/pipermail/openstack-dev/2018-October/136192.html | 17:24 |
| openstackgerrit | Carol Bouchard proposed openstack/tripleo-heat-templates master: CiscoNexus: Update and Remove deprecated vars https://review.openstack.org/606958 | 17:24 |
| openstackgerrit | Merged openstack/tripleo-upgrade stable/rocky: Fix variables for workaround variables. https://review.openstack.org/614220 | 17:32 |
| openstackgerrit | Merged openstack/tripleo-ui master: Imported Translations from Zanata https://review.openstack.org/614411 | 17:32 |
| *** bogdando has quit IRC | 17:33 | |
| *** holser_ has quit IRC | 17:33 | |
| *** florianf is now known as florianf|afk | 17:38 | |
| *** sambetts is now known as sambetts|afk | 17:46 | |
| *** aufi has quit IRC | 17:50 | |
| *** fuzzball81 has quit IRC | 17:52 | |
| *** jjoyce has joined #tripleo | 17:53 | |
| *** panda is now known as panda|off | 17:53 | |
| *** mburned is now known as mburned_out | 17:57 | |
| *** mburned_out is now known as mburned | 17:57 | |
| roger2 | How should I go about resetting the instances before I try to deploy again? Asking because right now I have one active compute node, but one controller stuck in BUILD/scheduling. | 18:01 |
| *** derekh has quit IRC | 18:01 | |
| roger2 | I couldn't find a good reason my last deploy failed, other than maybe that I did it when it was already in BUILD/scheduling before I started this deploy. | 18:04 |
| *** jpena is now known as jpena|off | 18:04 | |
| *** trown|lunch is now known as trown | 18:04 | |
| *** holser_ has joined #tripleo | 18:07 | |
| *** hjensas has joined #tripleo | 18:09 | |
| *** ooolpbot has joined #tripleo | 18:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 18:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1798195 | 18:10 |
| openstack | Launchpad bug 1798195 in tripleo "rdo-cloud yum repos unavailable during container updates and failing the undercloud install " [Critical,Triaged] | 18:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1799895 | 18:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1800737 | 18:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1800742 | 18:10 |
| *** ooolpbot has quit IRC | 18:10 | |
| openstack | Launchpad bug 1799895 in tripleo "CI: undercloud takes long time which causes job fail with timeout" [Critical,Triaged] | 18:10 |
| openstack | Launchpad bug 1800737 in tripleo "relabel failed /var/lib/config-data: no such file or directory" [Critical,Triaged] | 18:10 |
| openstack | Launchpad bug 1800742 in tripleo "tempest.lib.exceptions.IdentityError: Got identity error, undercloud-containers" [Critical,Triaged] | 18:10 |
| gouthamr | hi folks, looking for some reviews on this one: https://review.openstack.org/#/c/600556/ | 18:16 |
| *** dtantsur is now known as dtantsur|afk | 18:17 | |
| openstackgerrit | Carlos Goncalves proposed openstack/tripleo-heat-templates master: Mount /usr/share/rhosp-director-images into mistral-executor https://review.openstack.org/614609 | 18:17 |
| *** noama has quit IRC | 18:25 | |
| *** jjoyce has quit IRC | 18:28 | |
| *** holser_ has quit IRC | 18:30 | |
| *** jjoyce has joined #tripleo | 18:33 | |
| *** jjoyce is now known as fuzzball81 | 18:34 | |
| *** psachin has quit IRC | 18:36 | |
| *** tzumainn has quit IRC | 18:39 | |
| openstackgerrit | Alex Schultz proposed openstack-infra/tripleo-ci master: WIP: Container build CI job https://review.openstack.org/614392 | 18:43 |
| weshay | dam yo | 18:47 |
| openstackgerrit | Alex Schultz proposed openstack-infra/tripleo-ci master: WIP: Container build CI job https://review.openstack.org/614392 | 18:48 |
| openstackgerrit | Alex Schultz proposed openstack/tripleo-heat-templates master: WIP: Flatten Aodh service configuration https://review.openstack.org/611188 | 18:54 |
| openstackgerrit | Alex Schultz proposed openstack/tripleo-heat-templates master: Add chrony time service https://review.openstack.org/588111 | 18:55 |
| openstackgerrit | Alex Schultz proposed openstack/tripleo-heat-templates master: Switch to Chrony by default https://review.openstack.org/611969 | 18:55 |
| *** sshnaidm|bbl is now known as sshnaidm|ruck | 18:57 | |
| *** suuuper has quit IRC | 18:57 | |
| openstackgerrit | Alex Schultz proposed openstack/puppet-tripleo master: Remove non-voting scenarios from gate https://review.openstack.org/614593 | 18:58 |
| openstackgerrit | Alex Schultz proposed openstack/tripleo-common master: Remove non-voting scenarios from gate https://review.openstack.org/614570 | 18:59 |
| openstackgerrit | Sorin Sbarnea proposed openstack/tripleo-quickstart master: wip: enable testing using ansible molecule https://review.openstack.org/613672 | 18:59 |
| openstackgerrit | Sagi Shnaidman proposed openstack-infra/tripleo-ci master: DNM: test tripleo-ci patch in CI https://review.openstack.org/608649 | 19:02 |
| mwhahaha | weshay: https://review.openstack.org/#/c/614392/ config file for what? | 19:03 |
| *** zaneb has quit IRC | 19:03 | |
| weshay | mwhahaha, I saw a couple of ansible vars there.. but I can give it a go and see what's up.. | 19:05 |
| mwhahaha | weshay: this is the one i'm using for my f28 testing, http://paste.openstack.org/show/733720/ | 19:06 |
| mwhahaha | it's basically straight of the docs | 19:06 |
| mwhahaha | weshay: https://docs.openstack.org/tripleo-docs/latest/install/containers_deployment/3rd_party.html#building-new-containers-with-kolla-build | 19:06 |
| weshay | rock..thanks mwhahaha | 19:07 |
| *** salmankhan has quit IRC | 19:07 | |
| mwhahaha | EmilienM: so podman, where is the proxy config for that? Or does it not even support a mirror like docker | 19:08 |
| EmilienM | mwhahaha: it should be in /etc/container, give me one sec plz I'm redeploying my env | 19:08 |
| mwhahaha | yea sure | 19:08 |
| *** ooolpbot has joined #tripleo | 19:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 19:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1798195 | 19:10 |
| openstack | Launchpad bug 1798195 in tripleo "rdo-cloud yum repos unavailable during container updates and failing the undercloud install " [Critical,Triaged] | 19:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1799895 | 19:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1800737 | 19:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1800742 | 19:10 |
| openstack | Launchpad bug 1799895 in tripleo "CI: undercloud takes long time which causes job fail with timeout" [Critical,Triaged] | 19:10 |
| *** ooolpbot has quit IRC | 19:10 | |
| openstack | Launchpad bug 1800737 in tripleo "relabel failed /var/lib/config-data: no such file or directory" [Critical,Triaged] | 19:10 |
| openstack | Launchpad bug 1800742 in tripleo "tempest.lib.exceptions.IdentityError: Got identity error, undercloud-containers" [Critical,Triaged] | 19:10 |
| EmilienM | mwhahaha: still searching | 19:16 |
| mwhahaha | EmilienM: yea cause i looked in /etc/containers in the CI logs and i didn't spot anything | 19:17 |
| mwhahaha | EmilienM: example, http://logs.openstack.org/92/614392/3/check/tripleo-ci-centos-7-containers-multinode/92e74fe/logs/undercloud/etc/containers/registries.conf.txt.gz | 19:17 |
| EmilienM | yea that's for insecure stuff | 19:17 |
| EmilienM | i'm digging in code | 19:17 |
| EmilienM | their doc is terrible | 19:17 |
| mwhahaha | EmilienM: my money is that they don't support a mirror like docker (but we need them to) | 19:17 |
| Vorrtex | What kind of authentication do I need to include for this command: curl -H 'Accept: application/vnd.docker.distribution.manifest.v2+json' https://registry-1.docker.io/v2/tripleomaster/centos-binary-iscsid/manifests/current-tripleo | 19:17 |
| EmilienM | they don't I'm sure | 19:17 |
| EmilienM | maybe in buildah or skopeo, lemme check | 19:18 |
| EmilienM | there is no proxy/mirror support I think | 19:19 |
| mwhahaha | Vorrtex: no idea, that's more a question for docker themselves | 19:21 |
| *** jcoufal has joined #tripleo | 19:21 | |
| mwhahaha | Vorrtex: we do have some basic auth code for the label fetching, https://github.com/openstack/tripleo-common/blob/master/tripleo_common/image/image_uploader.py#L270 maybe that could help you. though i'm not sure how all that stuff works specifically | 19:21 |
| Vorrtex | mwhahaha that's practically where I've been looking (not that line specifically), I'll look there now. thanks | 19:22 |
| Vorrtex | Nevermind, I *was* looking there... My line numbers are quite different than what's here | 19:23 |
| mwhahaha | Vorrtex: stevebaker would be the one who could provide some details on that auth bits | 19:23 |
| *** morazi has quit IRC | 19:24 | |
| *** amoralej is now known as amoralej|off | 19:24 | |
| Vorrtex | mwhahaha is stevebaker also the person you pointed me to last week for a different issue? I can't quite recall. Either way, thanks for the help. | 19:24 |
| EmilienM | mwhahaha: maybe I found something | 19:25 |
| EmilienM | https://github.com/containers/image/blob/bd10b1b53b2976f215b3f2f848fb8e7cad779aeb/pkg/sysregistriesv2/system_registries_v2.go#L27-L33 | 19:25 |
| EmilienM | I'm still digging | 19:25 |
| mwhahaha | EmilienM: https://github.com/containers/image/blob/bd10b1b53b2976f215b3f2f848fb8e7cad779aeb/pkg/sysregistriesv2/system_registries_v2_test.go#L58 | 19:26 |
| EmilienM | yea | 19:26 |
| mwhahaha | EmilienM: so we need to be able to specify a registries.mirror | 19:26 |
| EmilienM | mwhahaha: wait | 19:26 |
| EmilienM | I'm testing it now | 19:26 |
| EmilienM | i don't trust these tests :D | 19:26 |
| mwhahaha | EmilienM: verify then trust :D | 19:27 |
| stevebaker | Vorrtex: this is the logic required to auth against a registry, it would be a bit tedious to do with bash https://github.com/openstack/tripleo-common/blob/master/tripleo_common/image/image_uploader.py#L270-L308 | 19:27 |
| Vorrtex | stevebaker how familiar are you with this whole image download/upload portion of the undercloud setup? | 19:28 |
| stevebaker | Vorrtex: waaay to familiar ;) | 19:29 |
| Vorrtex | Then you're my guy to brain-pick. Would PM's be better or you okay talking here? | 19:29 |
| stevebaker | Vorrtex: either way is good for me | 19:30 |
| Vorrtex | Alright then. So the problem I'm currently digging into is looking for the right place to include multiarch support in the image download/upload portion of the undercloud install. Its been quite cumbersome because I'm personally not all that familiar with PDB and that seems to be the best way to test minor changes and inspect values throughout the code. | 19:31 |
| Vorrtex | I'm on a macbook, you see. | 19:31 |
| EmilienM | stevebaker: we disabled podman in our gate FYI | 19:32 |
| EmilienM | stevebaker: things went south | 19:32 |
| EmilienM | stevebaker: https://github.com/containers/libpod/issues/1739 | 19:32 |
| EmilienM | oh and good morning sir | 19:32 |
| Vorrtex | In any case, I've been digging through, and I've been trying to identify exactly how the code gathers the specific image shas used for uploading the images into the undercloud's private registry, so I can extend that to grab for "any and all architectures | 19:32 |
| *** abishop has quit IRC | 19:33 | |
| stevebaker | EmilienM: ok, ok | 19:34 |
| Vorrtex | stevebaker that seems to come from the "_inspect()" function, but I was going to manually walk through and see what changes *might* be necessary to extend that return object to "multiple architectures". Am I in the right place, or are you about to tell me I'm a doofus and should start elsewhere? ha ha, I'm okay with that.... :D | 19:35 |
| stevebaker | Vorrtex: we need to sort this for tonyb | 19:36 |
| stevebaker | 's power work anyway | 19:36 |
| Vorrtex | stevebaker oddly enough that's who I work with ;) | 19:36 |
| Vorrtex | He's the one that asked me to look into this :D | 19:37 |
| stevebaker | Vorrtex: oh that makes it easier. So right now the only way to get an image from a remote registry to the undercloud registry is by doing a docker pull/push, which may be a problem for multiarch right there | 19:38 |
| Vorrtex | stevebaker handled bit by bit its not that big of a deal, being that I'm relatively familiar with how to do it all manually. I just have to translate what I know into python and generalize it enough to make it openstack-worthy ha ha. | 19:39 |
| stevebaker | Vorrtex: but when the skopeo uploader becomes the default, as long as you're not modifying the image during prepare then there will be a registry->registry skopeo copy https://review.openstack.org/#/c/590087/ | 19:40 |
| Vorrtex | stevebaker I'm altogether unfamiliar with skopeo. Does it support manifest lists? | 19:40 |
| Vorrtex | stevebaker sorry, I guess my question is "if registry1 is populated with manifest lists, and registry2 is the undercloud private registry, would skopeo copy only images, or will it maintain manifest lists with that registry->registry copy?" | 19:42 |
| Vorrtex | registry1->registry2 copy?*** | 19:42 |
| stevebaker | Vorrtex: I'm not actually sure, but I'm starting to make contributions to skopeo too, so I can help with anything which is missing. I'd suggest for a start having a play with "skopeo copy" yourself then coming up with a concrete use case for what is needed | 19:44 |
| *** betherly has joined #tripleo | 19:45 | |
| stevebaker | Vorrtex: I'm familiar with manifest lists but have not needed to do anything with them yet | 19:45 |
| Vorrtex | stevebaker the problem I'm imagining (which could be solved already, since we both don't know if it would work or not) is since a manifest list image is really just a "smart pointer" to architecture-specific images, that it might not be included in a skopeo copy, since it has no tangible image associated with it. I'll take a look at it, though. | 19:46 |
| Vorrtex | stevebaker also how do you mean "concrete use case for what is needed"? Isn't "multiarch support" a concrete use-case? ha ha | 19:47 |
| stevebaker | Vorrtex: like a skopeo copy command involving a manifest list which should work but doesn't | 19:47 |
| *** jcoufal has quit IRC | 19:48 | |
| *** betherly has quit IRC | 19:49 | |
| *** apetrich has quit IRC | 19:50 | |
| *** tzumainn has joined #tripleo | 19:51 | |
| stevebaker | Vorrtex: this is the code which skopeo/podman etc use https://github.com/containers/image/blob/master/image/sourced.go#L62-L66 | 19:52 |
| *** raildo has quit IRC | 19:53 | |
| *** two_oes has joined #tripleo | 19:59 | |
| *** akrivoka has quit IRC | 20:00 | |
| *** apetrich has joined #tripleo | 20:04 | |
| Vorrtex | stevebaker ah, the infamous "go" language, which I've hardly touched. Cool, gotta learn so many things. ha ha | 20:09 |
| *** ooolpbot has joined #tripleo | 20:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 20:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1798195 | 20:10 |
| openstack | Launchpad bug 1798195 in tripleo "rdo-cloud yum repos unavailable during container updates and failing the undercloud install " [Critical,Triaged] | 20:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1799895 | 20:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1800737 | 20:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1800742 | 20:10 |
| openstack | Launchpad bug 1799895 in tripleo "CI: undercloud takes long time which causes job fail with timeout" [Critical,Triaged] | 20:10 |
| *** ooolpbot has quit IRC | 20:10 | |
| openstack | Launchpad bug 1800737 in tripleo "relabel failed /var/lib/config-data: no such file or directory" [Critical,Triaged] | 20:10 |
| openstack | Launchpad bug 1800742 in tripleo "tempest.lib.exceptions.IdentityError: Got identity error, undercloud-containers" [Critical,Triaged] | 20:10 |
| stevebaker | Vorrtex: yeah, this is my first exposure and I'm actually enjoying it. Look, I did concurrency! https://github.com/containers/image/pull/528/files | 20:10 |
| *** ramishra has quit IRC | 20:13 | |
| *** apetrich has quit IRC | 20:14 | |
| *** ansmith has quit IRC | 20:18 | |
| *** raildo has joined #tripleo | 20:22 | |
| EmilienM | mwhahaha: https://trello.com/c/TujUSC2A/46-mirror-support-in-containers-image FYI | 20:22 |
| EmilienM | I filed a bug (more a question for now) | 20:22 |
| tonyb | stevebaker, Vorrtex: 'sup? | 20:22 |
| *** morazi has joined #tripleo | 20:22 | |
| stevebaker | tonyb: just talking about multiarch images and manifest lists | 20:22 |
| *** raildo has quit IRC | 20:22 | |
| tonyb | stevebaker: Ahh goodo carry on :) | 20:23 |
| tonyb | anything I can do let me know. | 20:23 |
| *** paramite has quit IRC | 20:27 | |
| openstackgerrit | Sorin Sbarnea proposed openstack/tripleo-quickstart master: Make quickstack.sh avoid needing sudo on missing virtualenv https://review.openstack.org/613797 | 20:28 |
| stevebaker | tonyb: can you point me at a registry path which is storing a manifest list? | 20:28 |
| *** trown is now known as trown|outtypewww | 20:29 | |
| tonyb | stevebaker: ummmm IIRC the centos-base image is a manifest list but perhaps not with power | 20:29 |
| stevebaker | tonyb: ah, ok | 20:30 |
| *** bfournie has quit IRC | 20:37 | |
| bnemec | hjensas: I pushed a change to split the routed-networks environments in OVB so there aren't any path problems: https://github.com/cybertron/openstack-virtual-baremetal/commit/e4edf35b6423430454b6fb68e02f0f6e264137ce | 20:40 |
| bnemec | If you get a chance can you take a look and make sure everything looks sane? | 20:40 |
| *** sai_p has joined #tripleo | 20:40 | |
| openstackgerrit | Sam Doran proposed openstack/tripleo-ansible master: Add podman_image module https://review.openstack.org/611698 | 20:43 |
| openstackgerrit | Carlos Goncalves proposed openstack/tripleo-common master: Stop installing octavia-amphora-image package https://review.openstack.org/614631 | 20:45 |
| *** salmankhan has joined #tripleo | 20:45 | |
| hjensas | bnemec: ack, will take a look tomorrow. | 20:45 |
| bnemec | hjensas: Sounds good. I just realized I forgot to add one of the files, so at least one followup is coming. :-) | 20:46 |
| *** tongl_ has quit IRC | 20:49 | |
| *** salmankhan has quit IRC | 20:50 | |
| openstackgerrit | Sagi Shnaidman proposed openstack/tripleo-heat-templates stable/rocky: DNM: test patch on vexxhost rocky https://review.openstack.org/614632 | 20:51 |
| openstackgerrit | Sagi Shnaidman proposed openstack/tripleo-heat-templates master: DNM: test patch on vexxhost https://review.openstack.org/614633 | 20:51 |
| EmilienM | mwhahaha, weshay : I removed alert & promotion blocker on https://bugs.launchpad.net/tripleo/+bug/1800737/ | 20:52 |
| openstack | Launchpad bug 1800737 in tripleo "relabel failed /var/lib/config-data: no such file or directory" [Critical,Triaged] - Assigned to Emilien Macchi (emilienm) | 20:52 |
| EmilienM | and added technical-debt | 20:52 |
| EmilienM | since we reverted everything | 20:52 |
| openstackgerrit | Carlos Goncalves proposed openstack/tripleo-heat-templates master: Mount /usr/share/rhosp-director-images into mistral-executor https://review.openstack.org/614609 | 20:55 |
| openstackgerrit | Carlos Goncalves proposed openstack/tripleo-common master: Stop installing octavia-amphora-image package https://review.openstack.org/614631 | 20:56 |
| *** abishop has joined #tripleo | 20:58 | |
| *** zaneb has joined #tripleo | 21:00 | |
| openstackgerrit | John Fulton proposed openstack/tripleo-docs master: DNM/WIP: Add Deploy the remote compute node example https://review.openstack.org/614636 | 21:00 |
| *** lblanchard has quit IRC | 21:03 | |
| openstackgerrit | John Fulton proposed openstack/tripleo-docs master: DNM/WIP: Add deploy remote compute node example https://review.openstack.org/614636 | 21:04 |
| *** two_oes has quit IRC | 21:06 | |
| weshay | thanks | 21:09 |
| *** ooolpbot has joined #tripleo | 21:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 21:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1798195 | 21:10 |
| openstack | Launchpad bug 1798195 in tripleo "rdo-cloud yum repos unavailable during container updates and failing the undercloud install " [Critical,Triaged] | 21:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1799895 | 21:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1800742 | 21:10 |
| *** ooolpbot has quit IRC | 21:10 | |
| openstack | Launchpad bug 1799895 in tripleo "CI: undercloud takes long time which causes job fail with timeout" [Critical,Triaged] | 21:10 |
| openstack | Launchpad bug 1800742 in tripleo "tempest.lib.exceptions.IdentityError: Got identity error, undercloud-containers" [Critical,Triaged] | 21:10 |
| *** boazel has joined #tripleo | 21:11 | |
| *** boazel_ has quit IRC | 21:13 | |
| *** Vorrtex has quit IRC | 21:14 | |
| *** bfournie has joined #tripleo | 21:15 | |
| *** dprince has quit IRC | 21:20 | |
| openstackgerrit | Ronelle Landy proposed openstack-infra/tripleo-ci master: Collect files in tripleo_root https://review.openstack.org/613678 | 21:20 |
| openstackgerrit | Ronelle Landy proposed openstack-infra/tripleo-ci master: Use ready rendered templates for reproducer https://review.openstack.org/614638 | 21:20 |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-heat-templates master: docker-puppet: retry container run command https://review.openstack.org/614639 | 21:21 |
| EmilienM | weshay, mwhahaha ^ meh | 21:21 |
| * weshay looks | 21:21 | |
| mwhahaha | moar retries | 21:21 |
| weshay | mwhahaha, think that would be fine w/ standalone | 21:22 |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-heat-templates master: docker-puppet: retry container run command https://review.openstack.org/614639 | 21:22 |
| weshay | not so much w/ multinode | 21:22 |
| mwhahaha | i don't think we should be working around that in docker-puppet | 21:22 |
| weshay | there were a couple patches on configuring the mirror in jobs.. /me looks for those | 21:22 |
| *** srini_ has joined #tripleo | 21:23 | |
| *** sri_ has quit IRC | 21:24 | |
| EmilienM | mwhahaha: me neither | 21:24 |
| EmilienM | I'm just exploring options here | 21:24 |
| mwhahaha | ya we'll see | 21:24 |
| EmilienM | brb | 21:24 |
| *** betherly has joined #tripleo | 21:27 | |
| *** chem has quit IRC | 21:28 | |
| openstackgerrit | Alex Schultz proposed openstack-infra/tripleo-ci master: WIP: Container build CI job https://review.openstack.org/614392 | 21:32 |
| *** betherly has quit IRC | 21:32 | |
| openstackgerrit | Alex Schultz proposed openstack-infra/tripleo-ci master: WIP: Container build CI job https://review.openstack.org/614392 | 21:33 |
| openstackgerrit | Alex Schultz proposed openstack-infra/tripleo-ci master: WIP: Container build CI job https://review.openstack.org/614392 | 21:39 |
| *** vinaykns has quit IRC | 21:45 | |
| * tonyb Wonders about zigo's 'OpenStack Cluster Installer (OCI)' and how that isn't basically tripleo just s/\(CentOS|RHEL|RDO\)/Debian/g ? | 21:48 | |
| *** mcornea has quit IRC | 21:52 | |
| *** itlinux has quit IRC | 21:57 | |
| openstackgerrit | Alex Schultz proposed openstack-infra/tripleo-ci master: WIP: Container build CI job https://review.openstack.org/614392 | 21:58 |
| *** apetrich has joined #tripleo | 21:58 | |
| *** agopi is now known as agopi|brb | 22:04 | |
| EmilienM | stupid 6pm idea but | 22:04 |
| EmilienM | I was wondering | 22:04 |
| EmilienM | in CI, we update containers (with yum update), so we can test our changes in projects like THT, tripleo-common etc | 22:04 |
| EmilienM | instead of doing that, why don't we bind mount them from the host | 22:04 |
| mwhahaha | that only works for tht | 22:05 |
| EmilienM | maybe we could implement an interface (I'm sure we have one alraedy) | 22:05 |
| EmilienM | and puppet-tripleo too no? | 22:05 |
| mwhahaha | i think we'd have too many bind mounts in random places | 22:05 |
| mwhahaha | the yum update is really because of lack of promotions | 22:05 |
| mwhahaha | or when we get screwed by the up coming 7.6 transition | 22:05 |
| mwhahaha | if we get consistent promotions, it's less of a problem | 22:06 |
| mwhahaha | the longer things go without a promotion the worse it gets | 22:06 |
| *** agopi|brb has quit IRC | 22:08 | |
| EmilienM | like i said it was a bad idea | 22:09 |
| mwhahaha | we really need to address the time it takes to run stuff | 22:09 |
| *** ooolpbot has joined #tripleo | 22:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 22:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1798195 | 22:10 |
| openstack | Launchpad bug 1798195 in tripleo "rdo-cloud yum repos unavailable during container updates and failing the undercloud install " [Critical,Triaged] | 22:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1799895 | 22:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1800742 | 22:10 |
| *** ooolpbot has quit IRC | 22:10 | |
| mwhahaha | like hjensas pointed out on the ML, we've got places where we can shave minutes if we look | 22:10 |
| openstack | Launchpad bug 1799895 in tripleo "CI: undercloud takes long time which causes job fail with timeout" [Critical,Triaged] | 22:10 |
| openstack | Launchpad bug 1800742 in tripleo "tempest.lib.exceptions.IdentityError: Got identity error, undercloud-containers" [Critical,Triaged] | 22:10 |
| openstackgerrit | Ronelle Landy proposed openstack/tripleo-quickstart-extras master: DNM - Testing getting playbooks from console log https://review.openstack.org/614645 | 22:16 |
| EmilienM | hjensas: wow nice work | 22:16 |
| EmilienM | on https://review.openstack.org/#/c/614540/ | 22:16 |
| EmilienM | hjensas: it failed here: http://logs.openstack.org/40/614540/2/check/tripleo-ci-centos-7-containers-multinode/cf06848/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz#_2018-10-31_18_08_01 | 22:17 |
| openstackgerrit | Merged openstack/tripleo-quickstart master: (squash) disabling podman everywhere in gate https://review.openstack.org/614537 | 22:17 |
| *** salmankhan has joined #tripleo | 22:18 | |
| EmilienM | voilà | 22:19 |
| EmilienM | the gate should be better now | 22:19 |
| EmilienM | recheck? | 22:19 |
| mwhahaha | not until we nuke scenarios | 22:20 |
| EmilienM | ah yeah | 22:20 |
| EmilienM | it's in gate? | 22:20 |
| mwhahaha | should be | 22:21 |
| mwhahaha | heh no | 22:22 |
| mwhahaha | it failed | 22:22 |
| mwhahaha | container upgrades | 22:22 |
| mwhahaha | er updates | 22:22 |
| mwhahaha | ugh puppet-tripleo is broke | 22:23 |
| EmilienM | we didn't have enough trouble | 22:23 |
| hjensas | EmilienM: I see, something not quite right with setting up the snmp password in the env. | 22:24 |
| EmilienM | hjensas: yeah I found it | 22:24 |
| EmilienM | echo "{\"name\": \"tripleo.undercloud-config\", \"variables\": {\"undercloud_ceilometer_snmpd_password\": \"$snmp_readonly_user_password\"}}" > $TMP_MISTRAL_ENV | 22:24 |
| EmilienM | you need to convert that in your script | 22:24 |
| EmilienM | you did snmp_readonly_user_password | 22:25 |
| EmilienM | but not ceilo | 22:25 |
| *** salmankhan has quit IRC | 22:25 | |
| mwhahaha | or the snmp module is too new | 22:25 |
| hjensas | EmilienM: ah, ok. I copied some stuff from instack-undercloud ... will push another version. | 22:25 |
| EmilienM | good | 22:26 |
| *** rlandy is now known as rlandy|bbl | 22:26 | |
| EmilienM | hjensas: bnemec also has a valid comment | 22:26 |
| EmilienM | mwhahaha: pin? | 22:26 |
| mwhahaha | yea probably | 22:26 |
| mwhahaha | let me dig up the last good commit | 22:26 |
| mwhahaha | https://github.com/voxpupuli/puppet-snmp/commit/8432216cd87fd893409fd43600c194e4950e9a1b broke it | 22:26 |
| EmilienM | our version of facter? | 22:28 |
| mwhahaha | rspec-puppet-facts are missing them | 22:29 |
| EmilienM | mwhahaha: we probably need to align stars | 22:30 |
| EmilienM | and squash in one commit your zuul + snmp module stuff | 22:30 |
| openstackgerrit | Alex Schultz proposed openstack/puppet-tripleo master: Pin puppet-snmp https://review.openstack.org/614647 | 22:30 |
| mwhahaha | we should be able to land that one | 22:30 |
| mwhahaha | the rest are ok | 22:30 |
| mwhahaha | greate gerrit is giving me a 500 trying to rebase on that | 22:32 |
| mwhahaha | heh | 22:32 |
| openstackgerrit | Alex Schultz proposed openstack/puppet-tripleo master: Remove non-voting scenarios from gate https://review.openstack.org/614593 | 22:33 |
| EmilienM | we'll probably to backport that | 22:33 |
| mwhahaha | yea we'll need to | 22:33 |
| EmilienM | +2 +A | 22:33 |
| *** gfidente|afk has quit IRC | 22:36 | |
| mwhahaha | the update process fail in haproxy on the tripleo-ci scenarios non-voting patch, http://logs.openstack.org/66/614566/1/gate/tripleo-ci-centos-7-scenario000-multinode-oooq-container-updates/226f017/logs/undercloud/var/log/containers/mistral/package_update.log.txt.gz#_2018-10-31_21_31_08_543 | 22:38 |
| mwhahaha | i hope that's not a sign that the updates job is broken | 22:38 |
| *** agopi|brb has joined #tripleo | 22:44 | |
| openstackgerrit | Alex Schultz proposed openstack-infra/tripleo-ci master: WIP: Container build CI job https://review.openstack.org/614392 | 22:47 |
| EmilienM | mwhahaha: http://zuul.openstack.org/builds?job_name=tripleo-ci-centos-7-scenario000-multinode-oooq-container-updates | 22:47 |
| EmilienM | there is some red there | 22:48 |
| mwhahaha | it's more of a light yellow | 22:48 |
| mwhahaha | but yes | 22:48 |
| mwhahaha | :D | 22:48 |
| *** itlinux has joined #tripleo | 22:50 | |
| *** sai_p has quit IRC | 23:09 | |
| *** ooolpbot has joined #tripleo | 23:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 23:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1798195 | 23:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1799895 | 23:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1800742 | 23:10 |
| *** ooolpbot has quit IRC | 23:10 | |
| openstack | Launchpad bug 1798195 in tripleo "rdo-cloud yum repos unavailable during container updates and failing the undercloud install " [Critical,Triaged] | 23:10 |
| openstack | Launchpad bug 1799895 in tripleo "CI: undercloud takes long time which causes job fail with timeout" [Critical,Triaged] | 23:10 |
| openstack | Launchpad bug 1800742 in tripleo "tempest.lib.exceptions.IdentityError: Got identity error, undercloud-containers" [Critical,Triaged] | 23:10 |
| *** jonaspaulo has joined #tripleo | 23:20 | |
| jonaspaulo | Hi all | 23:21 |
| jonaspaulo | getting some errors using OSP13 with a ceph deployment and don't understand why | 23:21 |
| jonaspaulo | (undercloud) [stack@undercloud ~]$ sudo grep "end: 0" /var/log/nova/nova-scheduler.log | 23:22 |
| jonaspaulo | 2018-10-30 17:14:17.805 24046 INFO nova.filters [req-c06d8ad6-4a63-43b8-9b56-aff77e231d19 0424b354949845b9bb8735b29556495d e0e383dee8ca4b9a8c8e861ee31667d8 - default default] Filtering removed all hosts for the request with instance ID '9c5cd087-2657-4e20-83f1-8cf135a5d513'. Filter results: ['RetryFilter: (start: 6, end: 6)', 'TripleOCapabilitiesFilter: (start: 6, end: 6)', 'ComputeCapabilitiesFilter: (start: 6, end: 3)', 'AvailabilityZoneFilter: (st | 23:22 |
| jonaspaulo | art: 3, end: 3)', 'RamFilter: (start: 3, end: 3)', 'DiskFilter: (start: 3, end: 0)'] | 23:22 |
| jonaspaulo | seems to be the disk filter | 23:22 |
| jonaspaulo | which indeed presents an error | 23:22 |
| jonaspaulo | 2018-10-30 17:14:17.804 24046 DEBUG nova.scheduler.filters.disk_filter [req-c06d8ad6-4a63-43b8-9b56-aff77e231d19 0424b354949845b9bb8735b29556495d e0e383dee8ca4b9a8c8e861ee31667d8 - default default] (undercloud.casa.pt, 173a1655-7536-4f9c-84cd-9568a387678d) ram: 8192MB disk: 39936MB io_ops: 0 instances: 0 does not have 40960 MB usable disk space before overcommit, it only has 39936 MB. host_passes /usr/lib/python2.7/site-packages/nova/scheduler/filter | 23:22 |
| jonaspaulo | s/disk_filter.py:55 | 23:22 |
| jonaspaulo | but the size is right | 23:22 |
| jonaspaulo | (undercloud) [stack@undercloud ~]$ openstack baremetal node show ceph0 -c properties | 23:23 |
| jonaspaulo | +------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | 23:23 |
| jonaspaulo | | Field | Value | | 23:23 |
| jonaspaulo | +------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | 23:23 |
| jonaspaulo | | properties | {u'cpu_arch': u'x86_64', u'root_device': {u'by_path': u'/dev/disk/by-path/pci-0000:00:07.0'}, u'cpus': u'6', u'capabilities': u'profile:ceph-storage,boot_option:local', u'memory_mb': u'8192', u'local_gb': 39} | | 23:23 |
| jonaspaulo | +------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | 23:23 |
| jonaspaulo | 23:23 | |
| jonaspaulo | any ideas? | 23:23 |
| jonaspaulo | i have tagged the root disk of course since there are 3 on that server | 23:23 |
| jonaspaulo | run the introspect afterwards | 23:23 |
| jonaspaulo | but still no go | 23:23 |
| openstackgerrit | Harald Jensås proposed openstack/tripleo-heat-templates master: Move UndercloudPostDeployment to python https://review.openstack.org/614540 | 23:31 |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!