| *** mjturek has quit IRC | 00:32 | |
| rfolco | weshay, whats up with the promoter service ? | 00:48 |
|---|---|---|
| *** dsneddon has quit IRC | 01:10 | |
| *** rlandy|ruck has quit IRC | 01:10 | |
| *** hamzy has joined #oooq | 01:22 | |
| *** dsneddon has joined #oooq | 01:36 | |
| *** dsneddon has quit IRC | 01:40 | |
| *** apetrich has quit IRC | 01:59 | |
| *** dsneddon has joined #oooq | 02:29 | |
| *** dsneddon has quit IRC | 02:36 | |
| *** dsneddon has joined #oooq | 02:49 | |
| *** dsneddon has quit IRC | 02:56 | |
| *** dsneddon has joined #oooq | 02:57 | |
| *** dsneddon has quit IRC | 03:05 | |
| *** dsneddon has joined #oooq | 03:11 | |
| *** dsneddon has quit IRC | 03:15 | |
| *** ykarel|away has joined #oooq | 03:19 | |
| *** dsneddon has joined #oooq | 03:49 | |
| *** dsneddon has quit IRC | 03:54 | |
| *** ykarel|away has quit IRC | 04:27 | |
| *** dsneddon has joined #oooq | 04:29 | |
| *** udesale has joined #oooq | 04:37 | |
| *** ykarel|away has joined #oooq | 04:43 | |
| *** dsneddon has quit IRC | 04:43 | |
| *** ratailor has joined #oooq | 04:51 | |
| *** ratailor has quit IRC | 04:51 | |
| *** ratailor has joined #oooq | 04:53 | |
| *** ykarel|away is now known as ykarel | 05:10 | |
| *** dsneddon has joined #oooq | 05:37 | |
| *** marios has joined #oooq | 05:39 | |
| *** dsneddon has quit IRC | 05:44 | |
| *** quiquell|off is now known as quiquell|rover | 05:47 | |
| *** dsneddon has joined #oooq | 06:20 | |
| *** dsneddon has quit IRC | 06:25 | |
| quiquell|rover | sshnaidm|afk: ping | 06:33 |
| *** udesale has quit IRC | 06:37 | |
| *** udesale has joined #oooq | 06:38 | |
| *** udesale has quit IRC | 06:39 | |
| *** udesale has joined #oooq | 06:44 | |
| *** holser_ has joined #oooq | 06:44 | |
| *** dsneddon has joined #oooq | 06:45 | |
| *** udesale has quit IRC | 06:46 | |
| sshnaidm|afk | quiquell|rover, hi | 06:56 |
| *** udesale has joined #oooq | 06:56 | |
| quiquell|rover | sshnaidm|afk: I am investigating the tripleo infra DNS server | 06:56 |
| quiquell|rover | sshnaidm|afk: how can I access it ? | 06:56 |
| quiquell|rover | sshnaidm|afk: Sometimes I get very slow responses like having cache invalidated | 06:57 |
| quiquell|rover | sshnaidm|afk: but I have to wait to get slow responses (maybe we have to dimension cache there) | 06:57 |
| quiquell|rover | sshnaidm|afk: Query time: 1389 msec for pypi.org | 06:58 |
| quiquell|rover | sshnaidm|afk: but this is with freeipa in the middle | 06:58 |
| sshnaidm|afk | quiquell|rover, added your key: ssh centos@38.145.33.91 | 06:59 |
| sshnaidm|afk | quiquell|rover, from yesterdays investigation it turns out that ipa doesn't use it at all | 06:59 |
| quiquell|rover | you kidding ? | 06:59 |
| sshnaidm|afk | quiquell|rover, forwarders in ipa are 192.168.100.2,192.168.100.3, 192.168.100.4 - it's routers IPs | 07:00 |
| sshnaidm|afk | quiquell|rover, https://review.openstack.org/#/c/653174/ | 07:00 |
| quiquell|rover | sshnaidm|afk: damn let's merge this | 07:01 |
| sshnaidm|afk | quiquell|rover, tripleo-ci-centos-7-ovb-3ctlr_1comp_1supp-featureset039 even passed there, but let's check again | 07:01 |
| sshnaidm|afk | quiquell|rover, lemme fix it better, we need to consider parameters and libvirt case, this role is used with quickstart.sh too | 07:01 |
| sshnaidm|afk | quiquell|rover, I'll complete it today later | 07:01 |
| quiquell|rover | cool! | 07:01 |
| quiquell|rover | thanks man | 07:01 |
| sshnaidm|afk | quiquell|rover, sure, np | 07:01 |
| quiquell|rover | can you reference the bug in the review ? | 07:02 |
| quiquell|rover | https://bugs.launchpad.net/tripleo/+bug/1824772 | 07:02 |
| openstack | Launchpad bug 1824772 in tripleo "freeipa not resolving mirror.regionone.rdo-cloud-tripleo.rdoproject.org broken trust chain resolving" [Critical,In progress] - Assigned to Quique Llorente (quiquell) | 07:02 |
| sshnaidm|afk | quiquell|rover, yeah, will add it | 07:05 |
| quiquell|rover | ok will destroy my server | 07:05 |
| Tengu | quiquell|rover: regarding freeipa, any way to disable dnssec for a while? | 07:06 |
| quiquell|rover | Tengu: looks like we were not using correct DNS servers | 07:07 |
| quiquell|rover | Tengu: just router's ones | 07:07 |
| Tengu | oh. that won't help :D | 07:07 |
| quiquell|rover | nope :-/ | 07:07 |
| quiquell|rover | Tengu: sshnaidm|afk is into it https://review.openstack.org/#/c/653174 | 07:07 |
| Tengu | just push 1.1.1.1 and 8.8.8.8 :D | 07:07 |
| *** amoralej|off is now known as amoralej | 07:07 | |
| *** yolanda_ has joined #oooq | 07:07 | |
| quiquell|rover | Tengu: I am going to run at the reproducer with your service logging thingy | 07:08 |
| Tengu | ah, wasn't far away. | 07:08 |
| quiquell|rover | Tengu: for the other issue | 07:08 |
| Tengu | although I hate opendns. | 07:08 |
| Tengu | they are lying. | 07:08 |
| quiquell|rover | Yep they depends on how good the project handle them | 07:08 |
| quiquell|rover | :-/ | 07:08 |
| Tengu | they shouldn't be used for an infra. the "open" in their name is a lie. | 07:08 |
| quiquell|rover | how so ? | 07:09 |
| Tengu | and they never return NXDOMAIN - they redirect to some web-ad-page. | 07:09 |
| quiquell|rover | are we using opendns ? | 07:09 |
| quiquell|rover | a you mean 8.8.8.8 ? | 07:09 |
| Tengu | "RDO cloud DNS server, google and OpenDNS" | 07:09 |
| Tengu | according to the commit message. | 07:09 |
| quiquell|rover | a ok sorry | 07:09 |
| Tengu | better replace opendns by cloudflare. | 07:09 |
| quiquell|rover | yep well is the third option | 07:09 |
| *** kopecmartin|off is now known as kopecmartin | 07:09 | |
| quiquell|rover | Tengu: add a comment there | 07:10 |
| Tengu | yeah, comment was on its way :). it's now done. | 07:10 |
| Tengu | there's also the opennicproject - they are better suited, although it might happen an NS crashes. | 07:11 |
| Tengu | so not suited for CI - might be OK on a private infra. | 07:11 |
| quiquell|rover | Tengu: do you know why weshay deactivated validation ? | 07:12 |
| quiquell|rover | Tengu: issue is still there | 07:12 |
| Tengu | quiquell|rover: for the other fs? no idea. probably because it was a promotion blocker? | 07:13 |
| quiquell|rover | Tengu: well tempest is failing too | 07:13 |
| Tengu | at least I do hope so, else the deactivation is useless. | 07:13 |
| Tengu | lol | 07:13 |
| quiquell|rover | humm wait no | 07:14 |
| quiquell|rover | tempest is ok upstreawm | 07:14 |
| quiquell|rover | weird | 07:14 |
| quiquell|rover | tempest fails only at periodics :-/ | 07:14 |
| Tengu | "yay" :) | 07:14 |
| quiquell|rover | Well I am going to investigate it though | 07:15 |
| *** tosky has joined #oooq | 07:29 | |
| Tengu | quiquell|rover: that said.... the failing job was non-voting apparently. No idea why weshay wanted to drop the validation check on that one... | 07:32 |
| *** dtantsur|afk is now known as dtantsur | 07:33 | |
| *** quiquell|rover is now known as quique|rover|brb | 07:33 | |
| quique|rover|brb | Tengu: well we want to make it voting at one moment | 07:34 |
| quique|rover|brb | Tengu: and ensure that past the validation everything else works fine | 07:34 |
| Tengu | yeah, well, as commented in the change disabling the validation, it's just hiding our head in the sand. | 07:34 |
| Tengu | it is NOT working since there is a failed service. | 07:34 |
| *** apetrich has joined #oooq | 07:34 | |
| * Tengu doesn't like playing the ostrich | 07:35 | |
| Tengu | so it's not a fix. a lame workaround that has no real value. | 07:35 |
| *** ykarel is now known as ykarel|lunch | 07:43 | |
| *** dsneddon has quit IRC | 07:51 | |
| *** quique|rover|brb is now known as quiquell|rover | 08:05 | |
| quiquell|rover | Tengu: well could be related to the other problematic service we have | 08:06 |
| Tengu | quiquell|rover: it actually is. since the validation detects it's down for some reason ;). | 08:06 |
| *** dsneddon has joined #oooq | 08:07 | |
| *** dtrainor has quit IRC | 08:09 | |
| *** dtrainor has joined #oooq | 08:09 | |
| *** ykarel|lunch is now known as ykarel | 08:31 | |
| quiquell|rover | Tengu: have the 137 exit status here | 08:31 |
| quiquell|rover | Tengu: we the log review applied | 08:32 |
| Tengu | quiquell|rover: any output ? | 08:32 |
| quiquell|rover | logs are empty :-( | 08:33 |
| quiquell|rover | -rw-------. 1 root root 0 Apr 17 08:32 neutron-haproxy-ovnmeta-73965872-dbf5-4fab-a688-16bb8dbea5f5.log | 08:33 |
| quiquell|rover | -rw-------. 1 root root 0 Apr 17 08:29 neutron-haproxy-ovnmeta-7a7a37d3-f53c-4033-8919-b62642f3fc3f.log | 08:33 |
| quiquell|rover | -rw-------. 1 root root 0 Apr 17 08:30 neutron-haproxy-ovnmeta-f2cec374-790f-4ab2-9e42-d8a48e10c962.log | 08:33 |
| Tengu | humpf. | 08:33 |
| Tengu | you want beagles on that one. | 08:33 |
| quiquell|rover | he is in the other time zone I think | 08:34 |
| Tengu | yeah, he'll be up in about 4 hours. | 08:34 |
| quiquell|rover | can I look elsewhere why it exist with 137 | 08:35 |
| quiquell|rover | like manually run it or the like | 08:35 |
| Tengu | I'm checking what's launched. | 08:35 |
| Tengu | so it's from haproxy.epp in puppet-tripleo/templates/neutron | 08:35 |
| quiquell|rover | yep | 08:35 |
| quiquell|rover | [zuul@standalone stdouts]$ sudo podman ps -a |grep haproxy | 08:36 |
| quiquell|rover | 2711dbeac3a7 192.168.24.1:8787/tripleomaster/fedora-binary-neutron-metadata-agent-ovn:bd07eada87635320be8d1b42cc22ab5b4af6ba27_ea85b618-updated-20190417074230 dumb-init --singl... About a minute ago Exited (137) 47 seconds ago neutron-haproxy-ovnmeta-26f8cfce-a01a-476d-8d5e-7ee66296b21e | 08:36 |
| Tengu | the command is apparetly '$(if [ -f /usr/sbin/haproxy-systemd-wrapper ]; then echo "/usr/sbin/haproxy -Ds"; else echo "/usr/sbin/haproxy -Ws"; fi)' | 08:36 |
| quiquell|rover | but those logs have different names | 08:36 |
| Tengu | name is built | 08:36 |
| quiquell|rover | there is no log file for neutron-haproxy-ovnmeta-26f8cfce-a01a-476d-8d5e-7ee66296b21e | 08:37 |
| Tengu | do you have a tmate? | 08:37 |
| quiquell|rover | yep | 08:38 |
| Tengu | care to share? would be easier imho :) | 08:38 |
| *** udesale has quit IRC | 08:51 | |
| *** ccamacho has joined #oooq | 08:53 | |
| zbr | i se few few discussions on list about replacing py35 jobs with py36 and this makes me realise: the only redhat distro having it right now is fedora28. One or another this will hit us. | 08:59 |
| marios | anyone know if we need something special for depends-on in code.engineering ... its for SF... in https://code.engineering.redhat.com/gerrit/#/c/167540/8 the depends-on aren't getting included (comment here https://code.engineering.redhat.com/gerrit/#/c/167540/8//COMMIT_MSG ) | 09:07 |
| marios | sshnaidm|afk: ? ^ | 09:07 |
| marios | quiquell|rover: ^? | 09:07 |
| quiquell|rover | marios: zuul should clone it correctly | 09:10 |
| quiquell|rover | marios: and we apply those changes with build-test-packges for DLRN projects | 09:10 |
| marios | quiquell|rover: trying again with the URL instead of the chid | 09:10 |
| quiquell|rover | marios: and with normal setuptools for non DLRN tripleo projects | 09:10 |
| quiquell|rover | marios: let me look | 09:10 |
| *** bogdando has joined #oooq | 09:11 | |
| quiquell|rover | marios: those depends-on are not quite right | 09:14 |
| quiquell|rover | marios: you have to point to URLs | 09:14 |
| quiquell|rover | also sf-config is config project | 09:14 |
| marios | quiquell|rover: ack trying that theory with the URL (i thought using url is for new gerrit/zuul but thought we have old style there in downstream | 09:15 |
| marios | quiquell|rover: ack on the config project thought about that but the one i am trying to test right now is the jobs one | 09:15 |
| quiquell|rover | marios: all new and shiny there | 09:15 |
| marios | quiquell|rover: thanks for looking | 09:15 |
| marios | quiquell|rover: lets see if this version does what i need and wil change the others too | 09:16 |
| * marios food biab | 09:16 | |
| quiquell|rover | marios: do the change with the URLs so I can inspect them too | 09:16 |
| quiquell|rover | marios: this is config project https://code.engineering.redhat.com/gerrit/#/c/168028/ | 09:17 |
| quiquell|rover | marios: you have to merge it first | 09:18 |
| *** udesale has joined #oooq | 09:20 | |
| quiquell|rover | ykarel: what was the dashboard for promotions ? | 09:21 |
| quiquell|rover | ykarel: the URL | 09:21 |
| ykarel | quiquell|rover, rhosp one? | 09:23 |
| quiquell|rover | yep | 09:24 |
| ykarel | http://rhos-release.virt.bos.redhat.com:3030/rhosp | 09:24 |
| ykarel | quiquell|rover, is promoter server down? | 09:24 |
| ykarel | i can't see master promotion | 09:24 |
| ykarel | after removing fs039 from criteria | 09:24 |
| quiquell|rover | ykarel: yep | 09:24 |
| quiquell|rover | ykarel: we have to wait for weshay | 09:24 |
| ykarel | quiquell|rover, ohhk, there is some issue with promoter? | 09:25 |
| quiquell|rover | ykarel: https://bugs.launchpad.net/tripleo/+bug/1825059 | 09:25 |
| openstack | Launchpad bug 1825059 in tripleo "dlrn promoter is promoting multiple releases concurrently and stalling" [Critical,Triaged] - Assigned to wes hayutin (weshayutin) | 09:25 |
| quiquell|rover | ykarel: you were right there | 09:25 |
| ykarel | okk | 09:26 |
| quiquell|rover | ykarel: btw... i see centosci for queens passed | 09:26 |
| quiquell|rover | ykarel: but it's not promoted | 09:26 |
| quiquell|rover | ykarel: ahh the promoter :-) | 09:26 |
| ykarel | hmm | 09:27 |
| *** rascasoft has quit IRC | 09:27 | |
| *** rascasoft has joined #oooq | 09:28 | |
| *** holser_ is now known as holser|lunch | 09:49 | |
| *** sshnaidm|afk is now known as sshnaidm | 09:56 | |
| *** dsneddon has quit IRC | 10:02 | |
| *** dsneddon has joined #oooq | 10:08 | |
| zbr | can people from outside rh join bj sessions? | 10:11 |
| *** dsneddon has quit IRC | 10:14 | |
| *** gkadam has joined #oooq | 10:16 | |
| quiquell|rover | arxcruz: do you know if running validate-services after tempest is on purpose ? | 10:26 |
| *** dsneddon has joined #oooq | 10:27 | |
| quiquell|rover | marios: workflowed https://code.engineering.redhat.com/gerrit/#/c/168028 | 10:28 |
| *** sanjayu_ has quit IRC | 10:31 | |
| *** sanjayu_ has joined #oooq | 10:31 | |
| *** dsneddon has quit IRC | 10:37 | |
| zbr | who was interested about the missing selinux inside virtualenvs? now there is a solution: "pip install selinux" which resolves the issue, as long the OS has the package installed. | 10:45 |
| zbr | the pip one is a shim that detects location of real package and loads it. worked on centos7 and fedora28. --- do not install it outside virtualenv, i am not sure what it does (may mess real package). | 10:46 |
| arxcruz | quiquell|rover: i don't know, but imho should be before, because if services are failing, there's a big chance tempest fails as well | 10:46 |
| arxcruz | actually, whatever is faster to fail | 10:46 |
| quiquell|rover | Tengu: P | 10:47 |
| quiquell|rover | Tengu: ^ | 10:47 |
| Tengu | arxcruz: hmm makes sense. | 10:47 |
| quiquell|rover | arxcruz, Tengu: https://review.openstack.org/653383 | 10:47 |
| marios | thanks :D quiquell|rover | 10:47 |
| Tengu | quiquell|rover: if you want, you can provide a patch moving the validation before tempest. And we can therefore see how it goes. | 10:47 |
| quiquell|rover | Tengu: I supose there are some time after the deploy | 10:47 |
| quiquell|rover | Tengu: will also add arevert to f28 disabling and activation on centos-7 | 10:48 |
| Tengu | quiquell|rover: works for me! Lemme try to make neutron cleaner :) | 10:48 |
| quiquell|rover | marios: any more issues with Depends-On ? | 10:48 |
| quiquell|rover | Tengu: yep thanks | 10:49 |
| marios | quiquell|rover: not yet lets see now that this one merged i can recheck lets see what happens with the job | 10:49 |
| quiquell|rover | ack | 10:49 |
| ykarel | panda, hi, panda i am planning to propose https://review.openstack.org/#/c/652508/7/roles/build-containers/templates/kolla-build.conf.j2 change, setting rpm_setup_config kolla param via a role variable, i guess you will also need something for your rhel7 work? | 10:49 |
| ykarel | in rdo we need ^^ to pass extra repo for dependencies | 10:50 |
| quiquell|rover | arxcruz: added neutron guys https://bugs.launchpad.net/neutron/+bug/1824315 | 10:51 |
| openstack | Launchpad bug 1824315 in tripleo "periodic fedora28 standalone job failing at test_volume_boot_pattern" [Critical,In progress] - Assigned to Quique Llorente (quiquell) | 10:51 |
| ykarel | panda like: https://review.rdoproject.org/r/#/c/20144/21/playbooks/run-rdoinfo.yaml@305 | 10:51 |
| arxcruz | quiquell|rover: neutron ? | 10:53 |
| quiquell|rover | arxcruz: argg big brain fart | 10:55 |
| quiquell|rover | arxcruz: ok nova it's | 10:56 |
| quiquell|rover | arxcruz: thanks for the help | 10:56 |
| *** gkadam is now known as gkadam-afk | 11:06 | |
| panda | ykarel: probably, but I'm still stuck at setting the proper repos | 11:15 |
| *** panda is now known as panda|lunch | 11:18 | |
| ykarel | panda|lunch, okk, /me will propose that and also remove duplicated call for tripleo-repos that i missed in original patch | 11:19 |
| apetrich | hey weshay can I expense one of this: https://www.rugvista.com/carpet/mistral?artno=RVD20332 ? | 11:20 |
| *** holser|lunch is now known as holser_ | 11:26 | |
| *** dsneddon has joined #oooq | 11:32 | |
| *** dsneddon has quit IRC | 11:38 | |
| quiquell|rover | panda|lunch: ping | 11:38 |
| *** ratailor has quit IRC | 11:48 | |
| *** gkadam-afk is now known as gkadam | 11:51 | |
| weshay | apetrich heh.. if you can find one in red :) | 11:59 |
| quiquell|rover | weshay: o/ what's up with promoter ? | 12:00 |
| weshay | quiquell|rover rfolco when rlandy comes on.. let's get on a call re: that | 12:01 |
| quiquell|rover | ack | 12:01 |
| weshay | quiquell|rover it was running the container promote at the same time across three releases | 12:01 |
| quiquell|rover | weshay: can produce this ? https://bugs.launchpad.net/tripleo/+bug/1825158 | 12:01 |
| openstack | Launchpad bug 1825158 in tripleo "periodic f28 standalone Not found image:" [Critical,Triaged] | 12:01 |
| weshay | quiquell|rover ya.. the promoter is fucked | 12:03 |
| quiquell|rover | ack | 12:03 |
| quiquell|rover | weshay: also we have found the issue with https://bugs.launchpad.net/tripleo/+bug/1824977 | 12:04 |
| openstack | Launchpad bug 1824977 in tripleo "fedora-28 standalone failing at neutron-haproxy-ovnmeta service" [Critical,In progress] - Assigned to Quique Llorente (quiquell) | 12:04 |
| quiquell|rover | weshay: it's all there let's wait for rlandy too | 12:04 |
| weshay | heh | 12:06 |
| *** dsneddon has joined #oooq | 12:06 | |
| rfolco | promoter service ? | 12:07 |
| *** dtantsur is now known as dtantsur|brb | 12:07 | |
| *** dsneddon has quit IRC | 12:13 | |
| *** rlandy has joined #oooq | 12:20 | |
| rlandy | quiquell|rover: hey | 12:23 |
| *** rlandy is now known as rlandy|ruck | 12:27 | |
| rlandy|ruck | weshay: quiquell|rover: rfolco: did we come to a decision re: dlrn promoter | 12:27 |
| quiquell|rover | rlandy|ruck: o/ | 12:28 |
| weshay | rlandy|ruck no.. we need to chat.. | 12:29 |
| weshay | I'll be avail in a few | 12:29 |
| rlandy|ruck | ok | 12:31 |
| weshay | rlandy|ruck quiquell|rover rfolco 3min | 12:32 |
| *** panda|lunch is now known as panda | 12:32 | |
| panda | quiquell|rover: pong | 12:32 |
| quiquell|rover | panda: false ping | 12:35 |
| marios | the boy who cried ping | 12:35 |
| zbr | weshay: ping me when you have ~5min bj. | 12:36 |
| zbr | marios: errata: the boy who cried ping6 ! | 12:36 |
| marios | :D | 12:37 |
| weshay | ok.. folks... quiquell|rover rlandy|ruck rfolco please join my blue | 12:38 |
| quiquell|rover | weshay, rlandy|ruck: fyi rest of the week is bank holidays here at spain | 12:38 |
| quiquell|rover | will not be rovering | 12:38 |
| weshay | quiquell|rover k.. I'll cover | 12:38 |
| weshay | quiquell|rover you guys have banks? | 12:38 |
| * rlandy|ruck needs to move to spain | 12:38 | |
| marios | anyone know if we shouldn't have playbooks in the rdo-jobs (and instead go the inheritance route) comment in https://review.rdoproject.org/r/#/c/20241/1/zuul.d/standalone-jobs.yaml | 12:44 |
| marios | rfolco: panda any idea? ^ | 12:44 |
| zbr | weshay: wow, I just realized that I will be off Friday/Monday dues to the same reason too, and Tue/Wed next week due the exams. | 12:46 |
| zbr | btw, what is with the promoter and the annoying broadcast message? | 12:46 |
| *** dsneddon has joined #oooq | 12:51 | |
| *** dsneddon has quit IRC | 12:57 | |
| panda | marios: it's because ansible is run as root | 13:00 |
| panda | marios: well at least part of it | 13:01 |
| *** amoralej is now known as amoralej|lunch | 13:01 | |
| panda | marios: maybe with the latest buildah changes | 13:01 |
| panda | ? | 13:01 |
| marios | panda: ack i mean, i got that bit cos the default is like build_repo_dir: "{{ ansible_user_dir }}" | 13:02 |
| marios | panda: so you don't think it is something to do with the inheritance? i was getting ready to post patches to do that but maybe it isn't necessary. perhaps i can just set build_repo_dir to /home/zuul ? :/ | 13:03 |
| marios | panda: perhaps its better if we use 'correct' inheritance anyway, i mean regardless of this. i'll post the patches and we can decide which way to keep. but still got to sort out that issue though re root | 13:04 |
| marios | panda: thanks for looking | 13:05 |
| *** Goneri has joined #oooq | 13:05 | |
| panda | marios: even if you fix teh intheritance, if ansible_user is root, you may hit some other errors later. | 13:07 |
| *** dtantsur|brb is now known as dtantsur | 13:07 | |
| panda | marios: how do you want to change the inheritance. What makes you think the inheritance is causing this ? | 13:08 |
| marios | panda: no i don't think (now) its related, i wasn't sure if that was the cause i.e. something special about rdo jobs i didn't know | 13:10 |
| marios | panda: right now... the inheritance is like in here "Background, links on inheritance path" https://tree.taiga.io/project/tripleo-ci-board/task/990 i.e. rdo standalone-job --> tripleo-ci-base-standalone-periodic --> tripleo-ci-base-standalone-rdo --> (upstream) tripleo-ci-base-standalone | 13:11 |
| marios | panda: and my new job followed the same, i.e. parenting onto upstream standalone, instead of upstream standalone upgrade. which is why i had to carry those playbooks there https://review.rdoproject.org/r/#/c/20241/1/zuul.d/standalone-jobs.yaml | 13:12 |
| marios | panda: hope it makes sense | 13:12 |
| weshay | panda can we jump on your blue? | 13:13 |
| ykarel | marios, can you try without os_tempest, i think that's related someho | 13:13 |
| marios | ykarel: ah ok you think? | 13:13 |
| ykarel | marios, yes | 13:13 |
| ykarel | marios, because of http://git.openstack.org/cgit/openstack/tripleo-quickstart-extras/tree/playbooks/tempest.yml#n53 | 13:13 |
| marios | ykarel: but that happens muuuch later (looking) | 13:14 |
| marios | ykarel: ah | 13:14 |
| ykarel | and http://git.openstack.org/cgit/openstack/tripleo-quickstart-extras/tree/playbooks/multinode-standalone.yml#n52 | 13:14 |
| marios | ykarel: that may be it ! | 13:14 |
| ykarel | gather_facts is true by default i think | 13:14 |
| panda | weshay: ok | 13:14 |
| ykarel | so changing ansible_user_dir | 13:14 |
| marios | ykarel: thank you! | 13:14 |
| ykarel | marios, please try and confirm the theory :) | 13:15 |
| marios | ykarel: ack will do | 13:15 |
| marios | panda: back on the parenting... its probably better to do it 'right' but we will need more jobs for the inheritance like i'll need to define tripleo-ci-base-standalone-upgrade-periodic & tripleo-ci-base-standalone-upgrade-rdo as well. i'll try ykarel suggestion first and see about the inheritance ... not sure which way to go yet | 13:16 |
| weshay | panda ur muted | 13:16 |
| rlandy|ruck | rfolco: https://bugs.launchpad.net/tripleo/+bug/1825059 | 13:18 |
| openstack | Launchpad bug 1825059 in tripleo "dlrn promoter is promoting multiple releases concurrently and stalling" [Critical,Triaged] - Assigned to wes hayutin (weshayutin) | 13:18 |
| *** mjturek has joined #oooq | 13:22 | |
| weshay | panda https://code.engineering.redhat.com/gerrit/gitweb?p=tripleo-environments.git;a=blob;f=config/release/master-rhel.yml;h=31388a9e13ad7d61b77cadbad804d9954e26ef59;hb=HEAD | 13:24 |
| *** dsneddon has joined #oooq | 13:26 | |
| *** Vorrtex has joined #oooq | 13:27 | |
| *** dsneddon has quit IRC | 13:32 | |
| *** ykarel is now known as ykarel|afk | 13:36 | |
| *** ykarel|afk has quit IRC | 13:41 | |
| *** vinaykns has joined #oooq | 13:45 | |
| rlandy|ruck | quiquell|rover: rfolco: https://review.rdoproject.org/r/20276 Update dlrn promoter exec call | 13:45 |
| *** quiquell|rover has quit IRC | 14:00 | |
| *** quiquell has joined #oooq | 14:00 | |
| *** quiquell is now known as quiquell|off | 14:00 | |
| *** amoralej|lunch is now known as amoralej | 14:06 | |
| rlandy|ruck | marios: hello | 14:08 |
| sshnaidm | rfolco, the image is started to build, but it hits problem with package dependencies: https://sf.hosted.upshift.rdu2.redhat.com/logs/29/167929/7/check/tripleo-ci-rhel-7-standalone-master-buildimage/f1e43cf/logs/undercloud/home/zuul/overcloud_image_build.log.txt.gz#_2019-04-17_13_50_01 | 14:09 |
| sshnaidm | rfolco, do you want to handle it from this point? | 14:09 |
| rlandy|ruck | marios: re; https://tree.taiga.io/project/tripleo-ci-board/task/1024?kanban-status=1447274 | 14:09 |
| marios | rlandy|ruck: gimme sec posting some patches | 14:09 |
| marios | panda: alternative 2 (re parenting the standalone upgrade) here fyi https://review.rdoproject.org/r/20278 (and follow the 2 depends-on https://review.rdoproject.org/r/20277 https://review.openstack.org/653440 ... i'll add to the card. | 14:11 |
| marios | rlandy|ruck: o/ rocky standalone? | 14:12 |
| marios | rlandy|ruck: ah internal sf | 14:12 |
| rfolco | sshnaidm, ok, will take a look at the conflicts/package deps | 14:12 |
| *** aakarsh has quit IRC | 14:12 | |
| rlandy|ruck | marios; sec - now I'm reviewing :) | 14:13 |
| marios | rlandy|ruck: ack :) | 14:13 |
| rlandy|ruck | marios: ok | 14:15 |
| rlandy|ruck | marios: so I left some notes there - should be pretty simple to add that job | 14:16 |
| rlandy|ruck | can put up a review quickly unless you want to take it | 14:16 |
| sshnaidm | rlandy|ruck, quiquell|off weshay I'm gonna merge this: https://code.engineering.redhat.com/gerrit/#/c/168169/ please be aware | 14:16 |
| marios | rlandy|ruck: ack, not gonna get to it today - so if there is nothing on the card tomorrow morning i'll jump onit? | 14:16 |
| rlandy|ruck | sshnaidm: waht you will probably see of a bunch of errors show up in the tenant linters | 14:16 |
| rlandy|ruck | marios:ack - let's see how rucking goes today | 14:17 |
| marios | rlandy|ruck: k thanks | 14:17 |
| rlandy|ruck | sshnaidm: probably no jobs failures | 14:17 |
| sshnaidm | rlandy|ruck, will fix if will be such | 14:17 |
| rlandy|ruck | iirc quiquell|off excluded those jobs to get rid of the linter errors | 14:17 |
| rlandy|ruck | sshnaidm: it won't stop anything running afaict - so go ahead | 14:18 |
| sshnaidm | rlandy|ruck, if everything explodes, we'll revert | 14:18 |
| rlandy|ruck | sshnaidm:ack - nobody will die from that merge either way | 14:18 |
| *** dsneddon has joined #oooq | 14:22 | |
| *** ykarel|afk has joined #oooq | 14:24 | |
| *** mjturek has quit IRC | 14:25 | |
| *** ykarel|afk is now known as ykarel | 14:25 | |
| *** dsneddon has quit IRC | 14:28 | |
| ykarel | rlandy|ruck, are they some uncleaned stacks? | 14:28 |
| ykarel | i am seeing address already in use in one of my patch | 14:28 |
| *** mjturek has joined #oooq | 14:29 | |
| *** dsneddon has joined #oooq | 14:30 | |
| rlandy|ruck | ykarel: checking | 14:30 |
| *** aakarsh has joined #oooq | 14:32 | |
| marios | ykarel: re https://review.rdoproject.org/r/#/c/20241/1/zuul.d/standalone-jobs.yaml@36 - looks like it worked ... at least its past that part now https://review.rdoproject.org/zuul/stream/fc039ba6d5164167be0ecd8dff81a28d?logfile=console.log | 14:34 |
| ykarel | marios, hmm i saw | 14:34 |
| ykarel | it's running upgrade | 14:35 |
| marios | panda: added notes in https://tree.taiga.io/project/tripleo-ci-board/task/990 - we need to choose betwee 1/2 (in reviews) | 14:35 |
| *** dsneddon has quit IRC | 14:36 | |
| rlandy|ruck | ykarel: openstack stack list | grep FAILED retruns nothing | 14:37 |
| ykarel | rlandy|ruck, ack so this means address already in use is not related to uncleaned state | 14:37 |
| ykarel | so somewhere address assignment is going wrong | 14:37 |
| ykarel | s/state/stacks | 14:38 |
| rlandy|ruck | ykarel: can you paste? | 14:38 |
| ykarel | rlandy|ruck, same is for server list also? | 14:38 |
| rlandy|ruck | let me check servers | 14:38 |
| ykarel | rlandy|ruck, https://logs.rdoproject.org/08/653408/1/openstack-check/tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001/5c9b021/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz#_2019-04-17_13_07_08 | 14:38 |
| rfolco | sshnaidm, what if I try rocky instead ? | 14:38 |
| ykarel | other one: https://logs.rdoproject.org/08/653408/1/openstack-check/tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset053/93a4ea1/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz#_2019-04-17_13_15_39 | 14:38 |
| rlandy|ruck | openstack server list | grep ERROR also returned nothing | 14:39 |
| ykarel | ack | 14:39 |
| ykarel | rlandy|ruck, sorry it didn't need to be in error, it's just ip is used by some port | 14:40 |
| ykarel | can you check what instance is using that port | 14:40 |
| ykarel | rlandy|ruck, so check 172.17.0.77 and 172.17.0.93 | 14:40 |
| rlandy|ruck | ykarel: you are correct, we usually see that error when there are failed stacks | 14:40 |
| rlandy|ruck | let's check loading | 14:41 |
| ykarel | rlandy|ruck, can you check port list for ^^ ips | 14:41 |
| ykarel | to be sure if they are used by some running servers | 14:41 |
| rlandy|ruck | yes | 14:41 |
| rlandy|ruck | usually overcloud_internal network | 14:45 |
| weshay | panda ok.. can pick it back up if ur avail | 14:46 |
| weshay | panda my blue | 14:46 |
| rlandy|ruck | ykarel: openstack port list doesn't show either of those ports | 14:48 |
| ykarel | rlandy|ruck, ack possibly those are deleted by now | 14:48 |
| rlandy|ruck | ykarel: but the cleanup script runs from cron | 14:48 |
| rlandy|ruck | and they could have been in the process of being deleted | 14:48 |
| rlandy|ruck | ykarel: let's open a LP bug to track this | 14:49 |
| ykarel | rlandy|ruck, i think bug is already there for it | 14:49 |
| ykarel | we see this from time to time | 14:49 |
| * rlandy|ruck looks | 14:49 | |
| rlandy|ruck | otherwise we need to halt new stack creation while we are deleting stacks | 14:49 |
| rlandy|ruck | idk even if that is possible | 14:50 |
| weshay | rlandy|ruck so w/ regards to the promoter, where did we leave it.. just monitor and create a story to prevent/test issues? | 14:51 |
| rlandy|ruck | weshay: we merged the gerrit change to fix the script that runs on the service exec | 14:52 |
| weshay | saw that | 14:52 |
| rlandy|ruck | to make the change permanent | 14:52 |
| rlandy|ruck | reloaded the daemon | 14:52 |
| rlandy|ruck | and we'll see if that hold out | 14:52 |
| rlandy|ruck | quiquell|off seemed to think the multiple processes are ok | 14:53 |
| rlandy|ruck | due to parent child scripts now | 14:53 |
| weshay | aye | 14:53 |
| rlandy|ruck | really idk | 14:53 |
| rlandy|ruck | we will have to watch it | 14:53 |
| rlandy|ruck | and we will only know after a full promotion cycle has run | 14:53 |
| weshay | arxcruz we should be getting you help from someone in nova soon | 14:53 |
| rlandy|ruck | weshay: quiquell|off and rfolco seemed pretty confident that the daemon fix will work | 14:54 |
| weshay | arxcruz re:f28 volume_boot_pattern | 14:54 |
| rlandy|ruck | ykarel: k - I'll investigate the multiple port issue | 14:54 |
| arxcruz | weshay: yes, that's what I point in the trello board | 14:54 |
| ykarel | rlandy|ruck, ack | 14:54 |
| weshay | arxcruz aye :) | 14:54 |
| weshay | arxcruz anything else you need atm? | 14:55 |
| arxcruz | weshay: beer? but i can only in one hour :D | 14:55 |
| weshay | heh.. sure man.. /me sends arxcruz beer | 14:55 |
| arxcruz | weshay: btw, friday is holiday | 14:55 |
| arxcruz | probably in US as well ... | 14:56 |
| rfolco | rlandy|ruck, you rock!! | 14:56 |
| weshay | aye | 14:56 |
| rfolco | rlandy|ruck++ | 14:56 |
| rfolco | rlandy++ | 14:56 |
| arxcruz | weshay: because we have 1-1 and i see you update it | 14:56 |
| rfolco | bot, where u when I need you | 14:56 |
| arxcruz | quiquell|off: where's the hubot ? | 14:56 |
| weshay | arxcruz in the infra tenant | 14:56 |
| weshay | code in ci-scripts | 14:56 |
| arxcruz | i mean, he's not around :) | 14:57 |
| *** weshay is now known as weshay|rover | 14:57 | |
| ykarel | rlandy|ruck, bug in case u didn't found already:- https://bugs.launchpad.net/tripleo/+bug/1818060 | 14:57 |
| openstack | Launchpad bug 1818060 in tripleo "Nodes periodic-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-master fail to get usable IPs though os-net-config with Error, some other host (BE:E5:4F:B9:21:B0) already uses address" [Critical,Fix released] - Assigned to Gabriele Cerami (gcerami) | 14:57 |
| *** udesale has quit IRC | 14:57 | |
| rfolco | weshay, I apologize, my change broke promoter, rlandy|ruck spot the root cause and fixed. | 14:57 |
| weshay|rover | rfolco was it just the dlrn-promoter.service file? | 14:58 |
| *** udesale has joined #oooq | 14:59 | |
| weshay|rover | rfolco imho I think the "fix" is to setup testing in the next sprint | 14:59 |
| *** udesale has quit IRC | 14:59 | |
| rfolco | weshay|rover, execstart=... was pointing to the previous dlrn script | 14:59 |
| *** udesale has joined #oooq | 14:59 | |
| weshay|rover | rfolco panda another user story for next sprint :) | 14:59 |
| weshay|rover | rfolco ya | 14:59 |
| rfolco | weshay|rover, yeah, need a more robust ci around promoter | 15:00 |
| *** udesale has quit IRC | 15:06 | |
| rlandy|ruck | ykarel: there were no stack spikes show here for a while: http://dashboard-ci.tripleo.org/d/cEEjGFFmz/cockpit?orgId=1 | 15:06 |
| rlandy|ruck | so my guess is you caught a deleting stack - but that is not unusual | 15:07 |
| rlandy|ruck | we may need to do a port check | 15:07 |
| *** dsneddon has joined #oooq | 15:08 | |
| weshay|rover | rlandy|ruck I'm going to go create some grafana panes to help the tempest folks.. | 15:12 |
| weshay|rover | rlandy|ruck ping me w/ what ever | 15:12 |
| weshay|rover | panda ping me when ur back | 15:12 |
| panda | weshay|rover: coming | 15:13 |
| ykarel | rlandy|ruck, ack, hmm issue needs some debugging | 15:13 |
| ykarel | what actually causing the issue | 15:13 |
| *** dsneddon has quit IRC | 15:14 | |
| panda | weshay|rover: your bj ? | 15:15 |
| weshay|rover | aye | 15:15 |
| rlandy|ruck | weshay|rover: k - thanks | 15:16 |
| rlandy|ruck | ykarel: ack - I don't think it's the same bug | 15:16 |
| ykarel | rlandy|ruck, you mean https://bugs.launchpad.net/tripleo/+bug/1818060 is different to the logs i shared above? | 15:21 |
| openstack | Launchpad bug 1818060 in tripleo "Nodes periodic-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-master fail to get usable IPs though os-net-config with Error, some other host (BE:E5:4F:B9:21:B0) already uses address" [Critical,Fix released] - Assigned to Gabriele Cerami (gcerami) | 15:21 |
| rlandy|ruck | ykarel:log is the same but cause is different | 15:21 |
| rlandy|ruck | we can hit this with clean stacks | 15:22 |
| chandankumar | so now https://github.com/openstack/openstack-ansible-os_tempest/commit/25e299d1f62eaa3eed742ead82b2bd6e5ee8b88b we have novajoin support in os tempest | 15:22 |
| chandankumar | sshnaidm: rlandy|ruck https://review.openstack.org/#/c/652983/ please have a look at this when free | 15:22 |
| chandankumar | sshnaidm: consumed here https://review.openstack.org/#/c/639324/ | 15:23 |
| ykarel | rlandy|ruck, ack, looks similar to me as bug title and description specifies nothing related to cleaned/uncleaned states | 15:24 |
| *** dsneddon has joined #oooq | 15:25 | |
| rlandy|ruck | ykarel: reopened https://bugs.launchpad.net/tripleo/+bug/1818060 with new comments | 15:26 |
| openstack | Launchpad bug 1818060 in tripleo "Nodes periodic-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-master fail to get usable IPs though os-net-config with Error, some other host (BE:E5:4F:B9:21:B0) already uses address" [Critical,In progress] - Assigned to Gabriele Cerami (gcerami) | 15:26 |
| ykarel | rlandy|ruck, ack | 15:27 |
| rlandy|ruck | chandankumar: looks good - is there a log of the tempest run on 039? | 15:28 |
| chandankumar | rlandy|ruck: 039 broken | 15:30 |
| chandankumar | at undercloud install | 15:30 |
| * chandankumar is not sure how to move on that | 15:30 | |
| chandankumar | kopecmartin: http://logs.openstack.org/40/645240/27/check/python-tempestconf-tempest-devstack-admin-plugins/da01b63/controller/logs/ I think we lost temest results html file | 15:31 |
| rlandy|ruck | chandankumar: good point. | 15:31 |
| rlandy|ruck | hopefully we merged a change to fix that | 15:31 |
| rlandy|ruck | chandankumar: ok - I'll +2 your change - if tempest fails on 039 we will fix it then | 15:31 |
| chandankumar | kopecmartin: sure | 15:32 |
| *** dsneddon has quit IRC | 15:32 | |
| chandankumar | sorry | 15:32 |
| chandankumar | rlandy|ruck: sure, thanks! | 15:32 |
| chandankumar | kopecmartin: related review https://review.openstack.org/#/c/645240/ | 15:33 |
| kopecmartin | chandankumar, i don't we had any results.html file in our gates logs | 15:33 |
| kopecmartin | * i don't think | 15:33 |
| chandankumar | ack, relying on job-output then | 15:34 |
| chandankumar | kopecmartin: by the way nice work on getting heat tempest plugin support | 15:34 |
| chandankumar | kopecmartin++ | 15:34 |
| kopecmartin | chandankumar, thanks | 15:35 |
| chandankumar | kopecmartin: please enable the same in https://github.com/rdo-infra/rdo-jobs/blob/master/zuul.d/standalone-jobs.yaml#L60 | 15:37 |
| chandankumar | scenario standalone 1 | 15:37 |
| chandankumar | with proper heat tests | 15:37 |
| chandankumar | arxcruz: https://review.openstack.org/#/c/645500/ https://review.openstack.org/#/c/648121/ https://review.openstack.org/#/c/651866/ are good to go, please have alook when free! | 15:38 |
| kopecmartin | chandankumar, no clue how .. create a card for that and we'll take a look | 15:38 |
| chandankumar | kopecmartin: sure will do that. | 15:38 |
| rlandy|ruck | weshay|rover: promoter looks ok so far - only master is going atm. also merging first pidone jobs | 15:43 |
| *** sanjayu_ has quit IRC | 15:45 | |
| ykarel | rlandy|ruck, master promotion failed | 15:59 |
| ykarel | promoter Couldn't create directory: Failure | 15:59 |
| ykarel | okk looks like false alarm | 15:59 |
| ykarel | those failures are ignored due to - | 15:59 |
| ykarel | RDO Phase 1 started | 16:00 |
| rlandy|ruck | promoter FINISHED promotion process | 16:00 |
| rlandy|ruck | ykarel: looks ok - checking containers/images | 16:01 |
| ykarel | yes it was ago just there were errors, but ignored | 16:01 |
| ykarel | s/ago/good | 16:01 |
| *** bogdando has quit IRC | 16:01 | |
| rlandy|ruck | https://images.rdoproject.org/master/rdo_trunk/current-tripleo-rdo/ 04/11 | 16:03 |
| rlandy|ruck | https://ci.centos.org/view/rdo/view/promotion-pipeline/job/rdo_trunk-promote-master-current-tripleo/ just kicked ok | 16:03 |
| ykarel | yup | 16:03 |
| ykarel | we are good | 16:03 |
| ykarel | sorry for the false alarm | 16:03 |
| rlandy|ruck | ykarel: np | 16:04 |
| *** holser_ has quit IRC | 16:05 | |
| *** ykarel is now known as ykarel|away | 16:05 | |
| marios | ttyl folks | 16:11 |
| *** ccamacho has quit IRC | 16:16 | |
| *** marios has quit IRC | 16:18 | |
| *** dtantsur is now known as dtantsur|afk | 16:24 | |
| *** dsneddon has joined #oooq | 16:32 | |
| *** dsneddon has quit IRC | 16:40 | |
| *** gkadam has quit IRC | 16:57 | |
| rlandy|ruck | 2019-04-17 13:50:55 | TASK [Run tripleo-container-image-prepare logged to /var/log/tripleo-container-image-prepare.log] *** | 17:05 |
| rlandy|ruck | 2019-04-17 13:50:56 | fatal: [undercloud]: FAILED! => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": true} | 17:05 |
| rlandy|ruck | new | 17:05 |
| *** kopecmartin is now known as kopecmartin|off | 17:05 | |
| *** dsneddon has joined #oooq | 17:08 | |
| *** dsneddon has quit IRC | 17:14 | |
| *** aakarsh has quit IRC | 17:27 | |
| *** aakarsh has joined #oooq | 17:33 | |
| *** dsneddon has joined #oooq | 17:34 | |
| *** arxcruz is now known as arxcruz|off|23 | 17:53 | |
| rlandy|ruck | weshay|rover: this look familiar to you? f28 master promotion failure ... -ci-fedora-28-standalone-master/cdbb808/logs/undercloud/var/log/tripleo-container-image-prepare.log.txt.gz | 17:58 |
| rlandy|ruck | otherwise logging bug | 17:58 |
| * weshay|rover looks | 17:58 | |
| weshay|rover | rlandy|ruck link again please? | 17:58 |
| rlandy|ruck | https://logs.rdoproject.org/openstack-periodic-master/git.openstack.org/openstack-infra/tripleo-ci/master/periodic-tripleo-ci-fedora-28-standalone-master/cdbb808/logs/undercloud/var/log/tripleo-container-image-prepare.log.txt.gz | 17:59 |
| weshay|rover | ooh.. | 17:59 |
| weshay|rover | that is new | 17:59 |
| rlandy|ruck | bug it is | 18:00 |
| rlandy|ruck | weshay|rover: thanks - logging | 18:00 |
| *** amoralej is now known as amoralej|off | 18:03 | |
| *** Goneri has quit IRC | 18:06 | |
| *** saneax has joined #oooq | 18:19 | |
| *** rfolco has quit IRC | 18:20 | |
| *** Goneri has joined #oooq | 18:21 | |
| *** rfolco has joined #oooq | 18:21 | |
| rfolco | weshay|rover, want to chat about buildah and rhel build image, can you? | 18:30 |
| weshay|rover | ya.. jump on | 18:35 |
| rfolco | k | 18:36 |
| rlandy|ruck | sshnaidm: ping re: sova and new pipeline names https://github.com/sshnaidm/sova/blob/promotion/tripleoci/config.py#L65 | 18:40 |
| rlandy|ruck | we need to update the pipeline names | 18:40 |
| rlandy|ruck | since weshay|rover and I always do this on the wrong branch, pls advise | 18:40 |
| weshay|rover | nice | 18:43 |
| weshay|rover | sshnaidm get on my blue when you have a sec | 18:44 |
| weshay|rover | sshnaidm nevermind | 18:48 |
| *** dtrainor has quit IRC | 18:51 | |
| *** saneax has quit IRC | 18:58 | |
| *** ykarel|away has quit IRC | 19:02 | |
| rlandy|ruck | weshay|rover: marios: https://code.engineering.redhat.com/gerrit/168210 Add rocky standalone job to run on internal sf | 19:08 |
| *** holser_ has joined #oooq | 19:12 | |
| *** dtrainor has joined #oooq | 19:16 | |
| rlandy|ruck | weshay|rover: what was the outcome of the pidone ovb discussion? | 19:16 |
| weshay|rover | rlandy|ruck /me looks.. rlandy|ruck he understood what I was talking about w/ ovb. He was curious about baas and whether or not we should set that up.. I told him that we've requested it as part of upshift for RDO and that it was better to wait on it | 19:21 |
| rlandy|ruck | ok | 19:21 |
| *** Goneri has quit IRC | 19:21 | |
| weshay|rover | rlandy|ruck +2 | 19:22 |
| rlandy|ruck | weshay|rover: we have some sova work to do ... | 19:23 |
| rlandy|ruck | and rocky standalone | 19:24 |
| rlandy|ruck | the internal jobs | 19:24 |
| rlandy|ruck | but I want sshnaidm to be in attendance - will raise it at tomorrow's meeting | 19:24 |
| rlandy|ruck | new pipelines | 19:24 |
| rlandy|ruck | weshay|rover: lastly - should we ping #tripleo on https://bugs.launchpad.net/tripleo/+bug/1825220 or wait for it to happen again? | 19:25 |
| openstack | Launchpad bug 1825220 in tripleo "f28 master standalone job fails container-image-prepare - No module named 'pkg_resources'" [Critical,Triaged] | 19:25 |
| rfolco | weshay|rover, can you change to -1 so I continue same patch https://review.openstack.org/#/c/652126/ | 19:29 |
| weshay|rover | rfolco you can add patches w/ a -2 | 19:29 |
| rfolco | weshay|rover, I know, my COD doesn't let me | 19:30 |
| weshay|rover | cash on delivery? | 19:31 |
| rfolco | compulsive obsessive disorder | 19:31 |
| weshay|rover | rfolco it's good to overcome your fears :) | 19:31 |
| weshay|rover | rlandy|ruck kick of the reproducer on it | 19:32 |
| weshay|rover | rlandy|ruck then maybe you can fix it too :) | 19:32 |
| weshay|rover | rlandy|ruck probably just a spec change | 19:32 |
| rlandy|ruck | weshay|rover: ack | 19:32 |
| rfolco | weshay|rover, you're my boss, not my psychologist | 19:32 |
| weshay|rover | lolz... | 19:32 |
| weshay|rover | usually w/ most people I'm a little bit of both | 19:32 |
| rlandy|ruck | weshay|rover has multiple talents :) | 19:32 |
| rfolco | :) | 19:33 |
| *** Goneri has joined #oooq | 19:34 | |
| rlandy|ruck | and ci-centos | 19:36 |
| rlandy|ruck | get to that next | 19:37 |
| rlandy|ruck | :( | 19:37 |
| sshnaidm | rlandy|ruck, seems like you need to change pipeline names to new ones | 19:37 |
| rlandy|ruck | sshnaidm: ack | 19:37 |
| rlandy|ruck | which branch? | 19:37 |
| sshnaidm | rlandy|ruck, but it'll miss all jobs from previous pipeline accordingly, so you'll see only new ones | 19:37 |
| rlandy|ruck | promtest? | 19:37 |
| rlandy|ruck | can we just add new ones and leave the old one? | 19:37 |
| sshnaidm | rlandy|ruck, promotion | 19:37 |
| sshnaidm | rlandy|ruck, mm.. I'll look at it tomorrow | 19:38 |
| sshnaidm | rlandy|ruck, maybe possible | 19:38 |
| rlandy|ruck | sshnaidm: and the branch to dd new check jobs? | 19:38 |
| sshnaidm | rlandy|ruck, master | 19:38 |
| rlandy|ruck | need to add rocky standalone | 19:38 |
| rlandy|ruck | ok | 19:38 |
| rlandy|ruck | sshnaidm: let's chat tomorrow | 19:38 |
| sshnaidm | rlandy|ruck, ack | 19:39 |
| * sshnaidm is desperate to inherit rdo jobs in internal SF | 19:41 | |
| sshnaidm | seems like quiquell|off was right, while using secrets nothing works.. | 19:41 |
| rlandy|ruck | weshay|rover: need to go afk ... I think I know what is wrong with https://ci.centos.org/artifacts/rdo/jenkins-tripleo-quickstart-promote-stein-current-tripleo-delorean-minimal-19/undercloud/home/stack/overcloud_deploy.log.gz- will add commit when I get back | 19:46 |
| *** rlandy|ruck is now known as rlandy|afk | 19:46 | |
| sshnaidm | rlandy|ruck, https://code.engineering.redhat.com/gerrit/168217 | 19:46 |
| zbr | weshay|rover: rlandy|afk : please have a look at hubbot gate recheck removal https://review.rdoproject.org/r/#/c/20240/ --- a bit of OCD, but this is how I think a change should look, to contain code that tests it. | 19:58 |
| zbr | obviously that when I tried to run it.... it started to rain with bugs. | 19:58 |
| *** mjturek has quit IRC | 20:20 | |
| *** Goneri has quit IRC | 20:40 | |
| *** holser_ has quit IRC | 20:54 | |
| weshay|rover | zbr heh.. thank you!! | 21:11 |
| weshay|rover | zbr++ | 21:11 |
| *** Goneri has joined #oooq | 21:29 | |
| *** Goneri has quit IRC | 21:36 | |
| *** Vorrtex has quit IRC | 21:38 | |
| *** Goneri has joined #oooq | 21:55 | |
| *** Goneri has quit IRC | 22:40 | |
| *** aakarsh has quit IRC | 22:49 | |
| *** vinaykns has quit IRC | 23:03 | |
| *** rlandy|afk is now known as rlandy|ruck | 23:03 | |
| rlandy|ruck | weshay|rover: hi - back | 23:04 |
| weshay|rover | rlandy|ruck howdy | 23:04 |
| rlandy|ruck | weshay|rover: how goes it here? | 23:05 |
| weshay|rover | I think ok... | 23:05 |
| * weshay|rover updating the dashboard finally | 23:05 | |
| weshay|rover | only took all week | 23:05 |
| rlandy|ruck | | 2019-04-17 19:27:44Z [standalone.StandaloneServiceChain.ServiceChain]: CREATE_FAILED StackValidationFailed: resources.ServiceChain: Property error: resources[2].properties: Property DockerCinderCoStack create failed; | 23:05 |
| rlandy|ruck | broken standalone :( | 23:05 |
| weshay|rover | rlandy|ruck is that the repro that you were trying to reproduce the import error? | 23:06 |
| rlandy|ruck | not even - downstream failing | 23:06 |
| rlandy|ruck | tripleo-ci-rhel-7-standalone-master https://sf.hosted.upshift.rdu2.redhat.com/logs/10/168210/1/check/tripleo-ci-rhel-7-standalone-master/a9d331b/ : FAILURE in 18m 00s (non-voting) | 23:06 |
| rlandy|ruck | tripleo-ci-rhel-7-standalone-rhos-14 https://sf.hosted.upshift.rdu2.redhat.com/logs/10/168210/1/check/tripleo-ci-rhel-7-standalone-rhos-14/8e4a790/ : FAILURE in 21m 25s | 23:06 |
| rlandy|ruck | tripleo-ci-centos-7-standalone-internal-rocky https://sf.hosted.upshift.rdu2.redhat.com/logs/10/168210/1/check/tripleo-ci-centos-7-standalone-internal-rocky/4aad173/ : SUCCESS in 1h 18m 02s (non-voting) | 23:06 |
| rlandy|ruck | master fails, rocky passes, rhos-14 fails | 23:07 |
| rlandy|ruck | will deal with that tomorrow | 23:07 |
| weshay|rover | hrm | 23:08 |
| weshay|rover | k | 23:08 |
| rlandy|ruck | missing successful jobs: [u'tripleo-quickstart-promote-queens-rdo_trunk-minimal'] | 23:08 |
| rlandy|ruck | queens didn't promote :( | 23:08 |
| rlandy|ruck | we have an issue with ci.centso | 23:08 |
| weshay|rover | rlandy|ruck it did in tripleo-ci | 23:08 |
| weshay|rover | rlandy|ruck meh.. probably just the hardware | 23:08 |
| weshay|rover | I rekicked master ci.centos | 23:09 |
| rlandy|ruck | they all failed | 23:09 |
| weshay|rover | wait for that.. I'll kick queens later | 23:09 |
| weshay|rover | rlandy|ruck ya.. remember it depends on getting a good node from ci.centos | 23:09 |
| weshay|rover | so I need to see it fail at a slow time to be concerned | 23:09 |
| rlandy|ruck | weshay|rover; this is not a node failure ... https://ci.centos.org/artifacts/rdo/jenkins-tripleo-quickstart-promote-stein-current-tripleo-delorean-minimal-19/undercloud/home/stack/overcloud_deploy.log.gz | 23:10 |
| rlandy|ruck | openstack overcloud deploy: error: unrecognized arguments: 90 | 23:11 |
| weshay|rover | hrm | 23:11 |
| rlandy|ruck | I think that is the falvor args | 23:11 |
| * rlandy|ruck gets | 23:11 | |
| rlandy|ruck | they are not recognized in master | 23:12 |
| rlandy|ruck | ha - ok this impacts stein as well ... | 23:14 |
| rlandy|ruck | https://logs.rdoproject.org/openstack-periodic-latest-released/git.openstack.org/openstack-infra/tripleo-ci/master/periodic-tripleo-ci-fedora-28-standalone-stein/37309be/logs/undercloud/var/log/tripleo-container-image-prepare.log.txt.gz | 23:14 |
| weshay|rover | rlandy|ruck want to divide and concur? | 23:20 |
| rlandy|ruck | weshay|rover: k ... | 23:21 |
| rlandy|ruck | putting in a review to remove flavors args from stein | 23:21 |
| rlandy|ruck | then we need to reproduce the standalone issue | 23:21 |
| rlandy|ruck | just updated https://bugs.launchpad.net/tripleo/+bug/1825220 | 23:22 |
| openstack | Launchpad bug 1825220 in tripleo "f28 master standalone job fails container-image-prepare - No module named 'pkg_resources'" [Critical,Triaged] | 23:22 |
| rlandy|ruck | weshay|rover: also not sure why queens didn't kick in ci-centos | 23:25 |
| weshay|rover | rlandy|ruck that module may be provided by python_setuptools | 23:32 |
| weshay|rover | python-setuptools.rpm | 23:32 |
| rlandy|ruck | ok - whose responsibility is it to add that rpm? | 23:34 |
| weshay|rover | rlandy|ruck ya.. probably python3-setuptools | 23:34 |
| weshay|rover | rlandy|ruck there is an exception list | 23:34 |
| * weshay|rover looks for it | 23:34 | |
| *** tosky has quit IRC | 23:38 | |
| *** aakarsh has joined #oooq | 23:41 | |
| weshay|rover | rlandy|ruck https://github.com/rdo-packages/tripleo-common-distgit/blob/rpm-master/openstack-tripleo-common.spec#L63 | 23:41 |
| rlandy|ruck | looking | 23:48 |
| rlandy|ruck | weshay|rover: ^^ ok - so I am not on the up and up here ... | 23:48 |
| rlandy|ruck | should that not include what we need? | 23:48 |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!