| *** vinaykns has quit IRC | 00:09 | |
| *** rlandy has quit IRC | 00:16 | |
| *** rascasoft has joined #oooq | 00:17 | |
| *** rascasoft has quit IRC | 00:24 | |
| *** hamzy has joined #oooq | 02:13 | |
| *** apetrich has quit IRC | 03:14 | |
| *** agopi has joined #oooq | 03:16 | |
| *** hamzy has quit IRC | 03:20 | |
| *** skramaja has joined #oooq | 03:43 | |
| *** skramaja_ has joined #oooq | 03:48 | |
| *** skramaja has quit IRC | 03:48 | |
| *** udesale has joined #oooq | 03:56 | |
| *** ykarel|away has joined #oooq | 04:08 | |
| *** ykarel|away is now known as ykarel | 04:09 | |
| *** hamzy has joined #oooq | 04:16 | |
| *** ykarel has quit IRC | 04:48 | |
| *** ykarel has joined #oooq | 05:05 | |
| *** ratailor has joined #oooq | 06:05 | |
| *** jtomasek has joined #oooq | 06:28 | |
| *** saneax has joined #oooq | 06:31 | |
| *** jfrancoa has joined #oooq | 06:46 | |
| *** quiquell|off is now known as quiquell|rover | 06:49 | |
| *** ccamacho has quit IRC | 07:29 | |
| *** ccamacho has joined #oooq | 07:29 | |
| *** udesale has quit IRC | 07:34 | |
| *** udesale has joined #oooq | 07:35 | |
| *** quiquell|rover is now known as quique|rover|brb | 07:36 | |
| *** apetrich has joined #oooq | 07:37 | |
| *** rascasoft has joined #oooq | 07:38 | |
| *** ratailor_ has joined #oooq | 07:40 | |
| *** ratailor has quit IRC | 07:43 | |
| *** ykarel is now known as ykarel|lunch | 07:49 | |
| *** chandankumar is now known as chkumar|ruck | 07:59 | |
| chkumar|ruck | quique|rover|brb: \o/ | 08:02 |
|---|---|---|
| chkumar|ruck | quique|rover|brb: rdo cloud showing major outage | 08:02 |
| quique|rover|brb | chkumar|ruck: yep, bye did recheck the Geneva stuff | 08:03 |
| quique|rover|brb | Transient issue at gate job | 08:03 |
| chkumar|ruck | gouthamr: for os_tempest is working fine | 08:03 |
| chkumar|ruck | gouthamr: for validate-tempest, we need to check the jobs | 08:03 |
| chkumar|ruck | sorry | 08:04 |
| chkumar|ruck | quique|rover|brb: ^^ | 08:04 |
| *** quique|rover|brb is now known as quiquell|rover | 08:04 | |
| quiquell|rover | chkumar|ruck: At least the review will fix the ones using os_tempest | 08:04 |
| chkumar|ruck | quique|rover|brb: let me first fix the container check patch related to service failures | 08:05 |
| chkumar|ruck | quiquell|rover: for ovn they have disabled a bunch of tests, I need to check why | 08:05 |
| quiquell|rover | chkumar|ruck: ack | 08:05 |
| *** kopecmartin|off is now known as kopecmartin | 08:09 | |
| chkumar|ruck | quiquell|rover: sshnaidm I am going a cleanup of down ports there are still 694 | 08:17 |
| chkumar|ruck | quiquell|rover: https://snapshot.raintank.io/dashboard/snapshot/1ALYKFAvcKq81Ak2CxGim9dsBNkDwUy2?orgId=2 | 08:17 |
| quiquell|rover | chkumar|ruck: ack, thanks | 08:17 |
| *** skramaja_ is now known as skramaja | 08:28 | |
| *** amoralej|off is now known as amoralej | 08:30 | |
| *** dtantsur|afk is now known as dtantsur | 08:34 | |
| *** tosky has joined #oooq | 08:43 | |
| *** holser_ has joined #oooq | 08:50 | |
| jfrancoa | quiquell|rover: hey, this issue during the undercloud installation and upgrade that shows an error "Configured hostname is not fully qualified. ", is there any lp to track it? I think I already saw it yesterday when checking some CI logs | 08:50 |
| *** ykarel|lunch is now known as ykarel | 08:51 | |
| quiquell|rover | jfrancoa: give me a log | 08:52 |
| jfrancoa | quiquell|rover: http://logs.openstack.org/25/637925/1/check/tripleo-ci-centos-7-containerized-undercloud-upgrades/154b304/logs/undercloud/home/zuul/undercloud_install.log.txt.gz | 08:53 |
| *** holser_ has quit IRC | 08:53 | |
| chkumar|ruck | ykarel: hello | 08:58 |
| chkumar|ruck | ykarel: do we have fs16 passing now? | 08:58 |
| ykarel | chkumar|ruck, i just checked your patch, and there test was failing | 08:58 |
| quiquell|rover | jfrancoa: not qualified here http://logs.openstack.org/25/637925/1/check/tripleo-ci-centos-7-containerized-undercloud-upgrades/154b304/logs/undercloud/etc/hostname.txt.gz | 08:58 |
| ykarel | chkumar|ruck, you mean telemetry or something else? | 08:59 |
| jfrancoa | quiquell|rover: so, I understand the issue is not being tracked, right? | 08:59 |
| quiquell|rover | jfrancoa: nope | 08:59 |
| jfrancoa | quiquell|rover: I'll create a lp | 08:59 |
| chkumar|ruck | ykarel: telemetry one | 08:59 |
| quiquell|rover | ack | 08:59 |
| *** holser_ has joined #oooq | 09:00 | |
| ykarel | chkumar|ruck, ack | 09:00 |
| ykarel | that failed when i last checked | 09:00 |
| *** jpena|off is now known as jpena | 09:00 | |
| quiquell|rover | jfrancoa: this is related to validations ? | 09:00 |
| quiquell|rover | jfrancoa: I see something regarding preflight | 09:01 |
| jfrancoa | quiquell|rover: mmm...I'm not sure...maybe some validation was being add, which is making the job fail. I need to dig into it | 09:01 |
| chkumar|ruck | ykarel: quiquell|rover so we are hitting new error now | 09:02 |
| chkumar|ruck | ykarel: quiquell|rover https://logs.rdoproject.org/49/18749/11/check/periodic-tripleo-ci-centos-7-multinode-1ctlr-featureset016-master/a103a28/logs/undercloud/home/zuul/tempest.log.txt.gz#_2019-02-19_14_31_34 | 09:02 |
| quiquell|rover | jfrancoa: tell me so, I see at standalone different use of the function | 09:02 |
| quiquell|rover | chkumar|ruck: this can be nova issue ? | 09:04 |
| quiquell|rover | heat in the middle | 09:04 |
| *** saneax has quit IRC | 09:04 | |
| *** saneax has joined #oooq | 09:05 | |
| quiquell|rover | chkumar|ruck: I see neutron issues at errors.txt | 09:06 |
| chkumar|ruck | quiquell|rover: we need to compare from previous success logs | 09:07 |
| quiquell|rover | into it | 09:07 |
| quiquell|rover | chkumar|ruck: neutron error is not here http://logs.rdoproject.org/openstack-periodic/git.openstack.org/openstack-infra/tripleo-ci/master/periodic-tripleo-ci-centos-7-multinode-1ctlr-featureset016-master/297d1c2/logs/undercloud/var/log/extra/errors.txt.gz | 09:08 |
| *** bogdando has joined #oooq | 09:08 | |
| quiquell|rover | nah it is | 09:08 |
| chkumar|ruck | quiquell|rover: we really need a third party CI there on gnocchi side | 09:09 |
| chkumar|ruck | quiquell|rover: I am going to drop an email there | 09:10 |
| quiquell|rover | chkumar|ruck: this issue is gnocchi ? | 09:10 |
| chkumar|ruck | quiquell|rover: not sure it is related to gnocchi, passed the error to silheat on #rhos-dev | 09:10 |
| chkumar|ruck | quiquell|rover: he is going to look into that | 09:10 |
| quiquell|rover | chkumar|ruck: ack cool | 09:11 |
| quiquell|rover | chkumar|ruck: Let's see what he said before open LP | 09:11 |
| chkumar|ruck | quiquell|rover: we donot need new lp | 09:11 |
| chkumar|ruck | quiquell|rover: https://bugs.launchpad.net/tripleo/+bug/1816414 -> queens has the same issue | 09:11 |
| openstack | Launchpad bug 1816414 in tripleo "[queens][fs017] telemetry tempest plugin tests failed in multinode promotion pipeline" [Critical,Triaged] | 09:11 |
| chkumar|ruck | quiquell|rover: either we can close queens bug and reuse the same? | 09:12 |
| quiquell|rover | chkumar|ruck: well the error was different was related to gnocci | 09:12 |
| quiquell|rover | chkumar|ruck: we should close it I think | 09:12 |
| quiquell|rover | And open new one for autoscaling | 09:12 |
| quiquell|rover | chkumar|ruck: At least we are going beyond gnocci issue now ? | 09:13 |
| *** ykarel is now known as ykarel|afk | 09:13 | |
| *** ykarel|afk is now known as ykarel | 09:17 | |
| chkumar|ruck | quiquell|rover: as per silheat it might be related to this https://logs.rdoproject.org/49/18749/11/check/periodic-tripleo-ci-centos-7-multinode-1ctlr-featureset016-master/a103a28/logs/subnode-2/var/log/containers/heat/heat-engine.log.txt.gz#_2019-02-19_14_25_56_850 | 09:17 |
| quiquell|rover | chkumar|ruck: I think there is a LP for that | 09:18 |
| quiquell|rover | I have se it somewhere | 09:18 |
| chkumar|ruck | quiquell|rover: no lp, I have fixed proactively as seen in os_tempest python-temepstconf side first | 09:19 |
| quiquell|rover | chkumar|ruck: Ahh yep | 09:19 |
| quiquell|rover | chkumar|ruck: so maybe backport is needed ? | 09:19 |
| chkumar|ruck | quiquell|rover: no revert 7.noarch | 09:21 |
| chkumar|ruck | 2019-02-19 14:23:47 | python2-tempestconf-2.0.1-0.20190115181339.f9b3c05.el7.noarch | 09:21 |
| chkumar|ruck | is coming from container | 09:21 |
| chkumar|ruck | quiquell|rover: for that we need to promote temepst container | 09:22 |
| chkumar|ruck | https://github.com/openstack/python-tempestconf/commit/72f0edffb0ecb899e278137102915e58b7ddbe23 -> is missing | 09:22 |
| jfrancoa | quiquell|rover: https://bugs.launchpad.net/tripleo/+bug/1816720 | 09:23 |
| openstack | Launchpad bug 1816720 in tripleo "Undercloud CI jobs fail with "Configured hostname is not fully qualified."" [Medium,New] | 09:23 |
| chkumar|ruck | quiquell|rover: let me do a hack, to make sure everything works | 09:23 |
| quiquell|rover | jfrancoa: It's upgrade jobs only ? | 09:25 |
| jfrancoa | quiquell|rover: it looks like..it's the only one failing here https://review.openstack.org/#/c/637925/ | 09:27 |
| jfrancoa | quiquell|rover: might be something in the featureset configuration... | 09:27 |
| chkumar|ruck | ykarel: quiquell|rover testing here https://review.rdoproject.org/r/#/c/18749/ by running tempest from packages | 09:28 |
| chkumar|ruck | for telemetry | 09:29 |
| quiquell|rover | jfrancoa: maybe we are not disabling some validations there let me check | 09:29 |
| quiquell|rover | Let me check what triggers this | 09:30 |
| jfrancoa | quiquell|rover: no..the validation is right. the problem is the undercloud hostname in that job, it usually is (in the jobs that pass) undercloud.localdomain | 09:30 |
| quiquell|rover | chkumar|ruck: but this is going to use the same package | 09:31 |
| jfrancoa | quiquell|rover: but this job has the nodepool node name as hotname | 09:31 |
| quiquell|rover | chkumar|ruck: container packges are updated from outter repos I think | 09:31 |
| jfrancoa | quiquell|rover: to me ,it's something in the zuul job config | 09:31 |
| chkumar|ruck | quiquell|rover: nope, it is going to use the latest packages from the periodic promotion job | 09:31 |
| chkumar|ruck | quiquell|rover: there will be no temepst container there | 09:31 |
| chkumar|ruck | but temepst will run from packages | 09:31 |
| quiquell|rover | chkumar|ruck: So why the container is not taking the packages ? | 09:32 |
| quiquell|rover | chkumar|ruck: that we are trying to promote ? | 09:32 |
| quiquell|rover | chkumar|ruck: Ahh well this is the tempest container | 09:32 |
| quiquell|rover | chkumar|ruck: Different mechanism than tripleo containers | 09:32 |
| chkumar|ruck | quiquell|rover: it needs a promotion then it will take the latest packages as it depends on build container job i think | 09:32 |
| quiquell|rover | chkumar|ruck: Is not like, install .repo file at containers and update them all and build container ? | 09:34 |
| quiquell|rover | chkumar|ruck: at least at tripleo containers, maybe tempest container is missing this ? | 09:34 |
| chkumar|ruck | quiquell|rover: I think it is already a part of https://github.com/openstack/tripleo-common/blob/3482a7758eaf17eb58df6f0893cfa2f9dc1ab5fc/container-images/overcloud_containers.yaml#L210 | 09:35 |
| quiquell|rover | jfrancoa: yep http://logs.openstack.org/38/637938/2/check/tripleo-ci-centos-7-containerized-undercloud-upgrades/3b85473/logs/undercloud/etc/hostname.txt.gz | 09:35 |
| quiquell|rover | jfrancoa: you are right | 09:35 |
| quiquell|rover | jfrancoa: I see the ir a "fix_etc_host" at the check_hostname function :-/ | 09:36 |
| chkumar|ruck | quiquell|rover: fix for queen https://github.com/gnocchixyz/gnocchi/pull/1019 | 09:36 |
| *** kopecmartin is now known as kopecmartin|afk | 09:37 | |
| panda|off | marios: replied https://review.openstack.org/634725 quiquell|rover sshnaidm can you please revote ? had to solve merge conflict yesterday thanks. | 09:42 |
| quiquell|rover | panda|off: done | 09:42 |
| chkumar|ruck | ykarel: seen this issue https://logs.rdoproject.org/openstack-periodic-24hr/git.openstack.org/openstack-infra//tripleo-ci/master/periodic-tripleo-ci-centos-7-multinode-1ctlr-featureset017-queens/b273262/logs/subnode-2/var/log/containers/heat/heat-engine.log.txt.gz?level=ERROR | 09:43 |
| marios | panda|off: thanks replied don't think we need it to run for ceph-loop unless you have ceph | 09:43 |
| quiquell|rover | jfrancoa: undercloud_hostname is missing | 09:44 |
| jfrancoa | quiquell|rover: ahh...that must be it | 09:44 |
| chkumar|ruck | ykarel: can we skip the telemetry tests in poi ? | 09:44 |
| quiquell|rover | jfrancoa: http://logs.openstack.org/25/637925/1/check/tripleo-ci-centos-7-containerized-undercloud-upgrades/154b304/logs/undercloud/home/zuul/undercloud.conf.txt.gz | 09:44 |
| quiquell|rover | jfrancoa: let me track it | 09:44 |
| chkumar|ruck | ykarel: I think it will work there | 09:44 |
| *** derekh has joined #oooq | 09:45 | |
| panda|off | marios: I may agre, but currently the role is run by default in pre-run in the base | 09:45 |
| *** panda|off is now known as panda | 09:46 | |
| jfrancoa | quiquell|rover: exactly...the undercloud_undercloud_hostname parameter is missing in the featureset | 09:46 |
| jfrancoa | quiquell|rover: I'm going to add it | 09:46 |
| quiquell|rover | jfrancoa: but for howlong ? | 09:47 |
| marios | panda: you mean ceph-loop don't see it in pre | 09:47 |
| quiquell|rover | jfrancoa: ack, make sure you don't miss the others | 09:48 |
| panda | marios: https://github.com/openstack-infra/tripleo-ci/blob/04c09e9ba03b61305a60eae852e0de2d8b64abc1/zuul.d/base.yaml#L41 | 09:49 |
| jfrancoa | quiquell|rover: sure, I'll check the others too. thanks for the help man | 09:49 |
| panda | marios: maybe we're running it twice then | 09:49 |
| quiquell|rover | jfrancoa: no problem | 09:49 |
| ykarel | chkumar|ruck, nope that heat error is very generic, | 09:49 |
| ykarel | must have seen, you can find actual error in service which is returning that | 09:49 |
| chkumar|ruck | ykarel: as confirmed by sileht it will be fixd by https://github.com/openstack/telemetry-tempest-plugin/commit/7f0e315a78df17d981ed86ebddf87759cd97eedf | 09:49 |
| marios | panda: yeah. i don't think we want that in the base | 09:50 |
| marios | panda: ok then it is run everywhere... | 09:50 |
| ykarel | chkumar|ruck, ack | 09:50 |
| chkumar|ruck | ykarel: preparing a new release | 09:50 |
| panda | marios: we should fix it, but not in this review, ok for you if I follow up ? | 09:51 |
| marios | panda: yah commented and revoted thanks | 09:51 |
| panda | marios: thanks, let's see if this can finally merge | 09:53 |
| zbr | marios: rfolco: can we sync on f28 work? | 09:53 |
| marios | zbr: i'll join if you folks need me otherwise I'll pass today | 09:53 |
| marios | zbr: its a bit early for rfolco no? | 09:54 |
| zbr | mainly I added a simple change that would enable us to add rdo f28 job before merging the big one: https://review.openstack.org/#/c/638108/ | 09:54 |
| marios | zbr: yeah i saw it fly by the inbox and added a brief comment there so i guess you're going to decouple that bit from folco review | 09:55 |
| ykarel | sshnaidm, you remember https://bugs.launchpad.net/tripleo/+bug/1815048? | 09:55 |
| openstack | Launchpad bug 1815048 in tripleo "[ovb][collect_logs] ovb collect logs job runs twice " [High,Fix released] - Assigned to Sagi (Sergey) Shnaidman (sshnaidm) | 09:55 |
| zbr | once we do this, we can merge the rdo job (linked from that one) and see results. | 09:55 |
| zbr | marios: the idea was to decouple to allow us to test/merge folco one in the end. | 09:55 |
| zbr | mainly we want to see the f28 on rdo replying to folco change. | 09:55 |
| zbr | but this cannot be done without making upstream job def reusable downstream and adding the job on rdo. quite simple stepts. | 09:56 |
| panda | zbr: marios anything I can review for you ? | 09:56 |
| zbr | panda: *yes*: https://review.openstack.org/#/c/638108 thanks | 09:57 |
| marios | zbr: a bit confused by the commit message though and it will be confusing in git history at https://review.openstack.org/#/c/638108/7 | 09:57 |
| marios | zbr: just added another comment | 09:57 |
| marios | zbr: that is adding a new base with files: definition. the base without files for the periodic is already there | 09:57 |
| zbr | marios: yeah, i will rephrase it to make it clear. | 09:59 |
| panda | mmmhhh, we are adding so many layers now | 09:59 |
| panda | what's the next step ? base-wf-wn-wt ? | 10:00 |
| marios | panda: any idea why this no merge? https://review.rdoproject.org/r/#/c/18896/ | 10:00 |
| quiquell|rover | marios: there were RDO issues | 10:00 |
| marios | quiquell|rover: ah thanks | 10:00 |
| quiquell|rover | marios: check the review at zuul status | 10:01 |
| marios | quiquell|rover: ack going | 10:01 |
| panda | marios: because in the last meeting we decided to block all your patches. Forever. | 10:01 |
| marios | paack thanks | 10:01 |
| marios | panda: ^ | 10:01 |
| zbr | marios: it can happen for zuul to miss some events, solution is easy "recheck" - is documented that recheck works for gates too. already did it. | 10:05 |
| marios | zbr: thanks yeah don't see it in https://review.rdoproject.org/zuul/status quiquell|rover gonna do le recheck | 10:05 |
| quiquell|rover | marios: ack do it | 10:06 |
| zbr | i am not sure what happens on rdo as i see "tox-linters" being queued.... if a linter job needs to a wait in the queue... we are in a deep cith | 10:11 |
| sshnaidm | ykarel, yeah, will look at it today | 10:13 |
| ykarel | sshnaidm, ack Thanks | 10:13 |
| zbr | panda: that zuul "bug" will. not be fixed. I asked, is a feature and expected behavior. | 10:14 |
| zbr | some could argue that is a feature. i asked on #zuul channel (not my own oppinion) | 10:15 |
| panda | zbr: that periodic jobs matches changes on files ? | 10:15 |
| quiquell|rover | zbr, panda: They are just want to know what to do there | 10:15 |
| *** matbu has quit IRC | 10:15 | |
| quiquell|rover | So yep, if implemented it will take time | 10:15 |
| quiquell|rover | I am trying to put a review though | 10:17 |
| zbr | i do think that using "-wf" prefix is a good enough approach as is kinda self explanatory. | 10:17 |
| zbr | quiquell|rover: on the other hand, if you can fix it to allow us to override it with empty list, it could be even better. but in any way it will take considerable amount of time to get the fix shipped to both production zuuls. | 10:18 |
| panda | zbr: it may be a good approach, it's not self explanatory, and "From now on we all should use this" is a bald statement | 10:18 |
| quiquell|rover | zbr: to software factory at least, upstream zuul they update often | 10:19 |
| quiquell|rover | zbr: but we will be able to reduce complexity in the future | 10:19 |
| *** matbu has joined #oooq | 10:19 | |
| panda | zbr: -with-files would be more self explanatory for example | 10:22 |
| zbr | panda: sure, make a comment. once i get a 2nd one to support your proposed name i will update the review. I am not in a mood for long terminology discussions. I used -nf as we already seen "-nv" wildly used, i never seen "-non-voting" used as a suffix. | 10:24 |
| zbr | at least this is an abstract job, so we do not endup with longer names on final jobs. | 10:25 |
| zbr | (worried only due to PTSD from downstream job names) | 10:26 |
| panda | rfolco: can you give me access to taigacli host | 10:28 |
| panda | ? | 10:28 |
| zbr | rfolco: what was the link to zuul bug? I want to include it in the source | 10:32 |
| marios | quiquell|rover: do we still need to pass the ssh-keygen -m PEM -t rsa for the repro | 10:32 |
| quiquell|rover | marios: keys have to be generated with -m PEM yep | 10:37 |
| quiquell|rover | marios: zuul is still affected by paramiko stuff | 10:37 |
| marios | quiquell|rover: k thanks | 10:38 |
| *** ccamacho has quit IRC | 10:54 | |
| *** ccamacho has joined #oooq | 11:12 | |
| *** udesale has quit IRC | 11:14 | |
| panda | marios: not that patch's lucky day | 11:14 |
| *** holser_ is now known as holser|lunch | 11:19 | |
| *** ratailor_ has quit IRC | 11:23 | |
| marios | panda: maybe zuul needs more blood sacrifice? | 11:33 |
| panda | marios: I have a better idea. Did you eve watch a movie called "Wicker man" ? | 11:35 |
| marios | haha | 11:35 |
| marios | panda: for the greater good right | 11:35 |
| zbr | sshnaidm: i seen you asked on #rdo regarding not triggered jobs. Do we need to change something to unblock these jobs? | 12:03 |
| *** kopecmartin|afk is now known as kopecmartin | 12:07 | |
| sshnaidm | zbr, apevec replied in #rdo, it's infrastructure problem | 12:11 |
| *** skramaja_ has joined #oooq | 12:12 | |
| *** jpena is now known as jpena|lunch | 12:12 | |
| *** skramaja has quit IRC | 12:12 | |
| zbr | sshnaidm: i seen but i didn't fully understand if now we are using containers or not for these jobs and if this could change the outcome or not. | 12:18 |
| sshnaidm | zbr, I think the problem is with rdo-cloud, but better to ask in #rdo | 12:19 |
| zbr | sshnaidm: sure. | 12:20 |
| sshnaidm | zbr, well, I see jobs run, so should be fine | 12:20 |
| sshnaidm | panda, any objections to add a new tls-all job to periodics? | 12:21 |
| sshnaidm | panda, we can add just to run it and add to criteria later | 12:22 |
| sshnaidm | weshay, ^^ | 12:22 |
| panda | sshnaidm: fine by me | 12:25 |
| rfolco | zbr, zuul bug ? | 12:27 |
| *** skramaja has joined #oooq | 12:27 | |
| *** skramaja_ has quit IRC | 12:28 | |
| quiquell|rover | zbr: o/ is this logstash query correct ? http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%20%5C%22no%20response%20to%20inactivity%20probe%20after%205%20seconds%2C%20disconnecting%5C%22 | 12:31 |
| zbr | quiquell|rover: nope, no space after colon. | 12:32 |
| quiquell|rover | zbr: do I need wildcards ? | 12:32 |
| *** holser|lunch is now known as holser_ | 12:55 | |
| *** rlandy has joined #oooq | 13:00 | |
| rfolco | sprint end / retrospective meeting ping -- marios, quiquell, sshnaidm, weshay, panda, rlandy, arxcruz, mwhahaha, rfolco, chkumar, ssbarnea, kopecmartin | 13:00 |
| panda | pass | 13:00 |
| panda | marios: I realized it's only lately I started to see you from a different perspective | 13:01 |
| rfolco | bluejeans is reconnecting me, my internet is ok.... | 13:02 |
| panda | marios: anyway I can see your NSFW from your monitors. | 13:02 |
| *** saneax has quit IRC | 13:10 | |
| *** saneax has joined #oooq | 13:11 | |
| *** quiquell|rover is now known as quique|rover|mtg | 13:20 | |
| *** trown is now known as trown|outtypewww | 13:31 | |
| bogdando | hi devops folks. How do we usually cast some idempotency check involved CI jobs upon patches? | 13:34 |
| bogdando | is it full-ci-check perhaps? | 13:34 |
| *** amoralej is now known as amoralej|lunch | 13:34 | |
| bogdando | (tried abrakadabra as well) | 13:34 |
| *** jpena|lunch is now known as jpena | 13:34 | |
| *** agopi has quit IRC | 13:35 | |
| weshay | quique|rover|mtg fyi https://review.openstack.org/638154 | 13:44 |
| * weshay adds related bug | 13:44 | |
| bogdando | no one?.. I know the periodic job runs idempotency checks, but I can't easily invoke that for a patch... | 13:46 |
| bogdando | just wanted to make sure mysqld gets a gracefull stop signal here https://review.openstack.org/#/c/635161/ ... | 13:47 |
| bogdando | nvm, I think after https://launchpad.net/bugs/1810690 fixed the idempotency job won't restart mysqld w/o changes found, so that needs local testing... | 13:48 |
| openstack | Launchpad bug 1810690 in tripleo "periodic-tripleo-ci-centos-7-ovb-1ctlr_1comp-featureset020-master - which includes idempotency check - is timing out during introspection" [Critical,Fix released] - Assigned to Bogdan Dobrelya (bogdando) | 13:48 |
| quique|rover|mtg | weshay: done | 13:48 |
| apetrich | so weshay quique|rover|mtg is totally right. mistral doesn't even run on standalone | 13:52 |
| quique|rover|mtg | apetrich: maybe config-downlaod part ? | 13:53 |
| quique|rover|mtg | marios: ^ ? | 13:53 |
| apetrich | quique|rover|mtg, not even. I think it is just the ansible part of it | 13:53 |
| weshay | apetrich quique|rover|mtg fyi https://review.openstack.org/#/c/638154/ updated | 13:53 |
| quique|rover|mtg | ack then no mistral there | 13:53 |
| apetrich | weshay, cheers | 13:55 |
| weshay | zbr++ | 13:58 |
| *** ykarel is now known as ykarel|pto | 14:02 | |
| *** jtomasek has quit IRC | 14:02 | |
| *** jtomasek has joined #oooq | 14:04 | |
| *** vinaykns has joined #oooq | 14:08 | |
| *** agopi has joined #oooq | 14:10 | |
| sshnaidm | rlandy, ptal https://review.rdoproject.org/r/#/c/18898/ | 14:12 |
| chkumar|ruck | weshay: please have a look at this epic https://tree.taiga.io/project/tripleo-ci-board/us/702?milestone=217491 | 14:27 |
| chkumar|ruck | *user story | 14:27 |
| *** ykarel|pto has quit IRC | 14:29 | |
| chkumar|ruck | weshay: quique|rover|mtg can you take care of production chain call, I need to leave now! | 14:32 |
| chkumar|ruck | *production chain escalations | 14:33 |
| quique|rover|mtg | chkumar|ruck: yep | 14:33 |
| quique|rover|mtg | no problem | 14:33 |
| chkumar|ruck | see ya | 14:34 |
| quique|rover|mtg | no more rr tomorrow \o/ !!! | 14:35 |
| *** chkumar|ruck is now known as kmrchdn | 14:35 | |
| kmrchdn | quique|rover|mtg: hehe | 14:35 |
| kmrchdn | quique|rover|mtg: we need to find a time for tomorrow handoff :-) | 14:35 |
| quique|rover|mtg | ack no problem | 14:36 |
| vinaykns | sshnaidm: hi, I'm trying to install tls everywhere using quickstart but I'm encountering http://pastebin.test.redhat.com/717896 | 14:36 |
| vinaykns | sshnaidm: any workaround for that would be helpful.!! | 14:37 |
| sshnaidm | vinaykns, how exactly do you try? | 14:38 |
| vinaykns | sshnaidm: I used an external freeipa server | 14:38 |
| vinaykns | and given its details to the qs | 14:39 |
| sshnaidm | vinaykns, what kind of error is that? which stage? | 14:40 |
| sshnaidm | vinaykns, I don't think I can help with IPA specific errors, better ask jaosorior | 14:41 |
| vinaykns | sshnaidm: I'm gettting this while deploying overcloud | 14:41 |
| vinaykns | sshnaidm: nojoin is not running on port 9090 | 14:41 |
| jaosorior | vinaykns: what's up? | 14:42 |
| sshnaidm | vinaykns, you can check if you have similar deploy arguments in passing all-tls job: http://logs.rdoproject.org/00/637800/3/openstack-check/tripleo-ci-centos-7-ovb-3ctlr_1comp_1supp-featureset039/687087a/logs/undercloud/home/zuul/overcloud-deploy.sh | 14:42 |
| vinaykns | jaosorior: hey, could you help in debugging this issue http://pastebin.test.redhat.com/717896 | 14:42 |
| vinaykns | jaosorior: I'm unable to start the novajoin service in the UC | 14:43 |
| jaosorior | vinaykns: I'll help you out in a bit. Just debuggnig another issue at the moment | 14:44 |
| vinaykns | jaosorior: Okay..no worries | 14:44 |
| sshnaidm | quique|rover|mtg, gonna merge this patch: https://review.rdoproject.org/r/#/c/18898/ is may affect ovb jobs | 14:45 |
| sshnaidm | quique|rover|mtg, but in reproducer it worked :) | 14:46 |
| quique|rover|mtg | sshnaidm: is my last hours of rr so be careful | 14:47 |
| sshnaidm | quique|rover|mtg, well, then you don't need to worry ;) | 14:47 |
| *** ykarel|pto has joined #oooq | 14:50 | |
| weshay | quique|rover|mtg was wondering if these results were rdo cloud related https://review.rdoproject.org/r/#/c/18914/ | 14:52 |
| weshay | oh ya.. | 14:52 |
| weshay | node failure | 14:52 |
| quique|rover|mtg | Yep NODE_FAILURE at f28 | 15:00 |
| quique|rover|mtg | weshay: are still hit by libvirt with force_tcg | 15:01 |
| quique|rover|mtg | weshay: http://logs.rdoproject.org/14/18914/1/check/tripleo-ci-reproducer-centos-7-libvirt/63cfa29/tripleo-ci-reproducer/libguestf-env.sh | 15:01 |
| *** amoralej|lunch is now known as amoralej | 15:01 | |
| *** quique|rover|mtg is now known as quiquell | 15:06 | |
| sshnaidm | rlandy, panda please take a look https://review.rdoproject.org/r/#/q/topic:periodic-tls | 15:19 |
| rlandy | sshnaidm: left comment on https://review.rdoproject.org/r/#/c/18922/ - let me know what you think | 15:28 |
| weshay | rfolco so who is the new ruck/rover? | 15:32 |
| weshay | and are you guys still in the mtg? | 15:32 |
| panda | weshay: no to both questions | 15:32 |
| panda | weshay: My last time as ruck/rover was on Unified sprint 1, I wanted to inject my candidacy too. | 15:33 |
| *** saneax has quit IRC | 15:33 | |
| panda | weshay: so volunteers are Arx, Marios and me. | 15:33 |
| weshay | rfolco we closed the mtg w/o a ruck/rover? | 15:33 |
| panda | sshnaidm: +1 on one, a question on another | 15:41 |
| *** skramaja has quit IRC | 15:43 | |
| zbr | marios: panda: fixed the -wf prefix. https://review.openstack.org/#/c/638108/ ok now? | 15:44 |
| kmrchdn | does these jobs tripleo-quickstart-extras-gate-master-delorean-quick-basic adds value in our upstream ci testing? | 15:45 |
| kmrchdn | I see it most of the time faild | 15:45 |
| sshnaidm | panda, replied | 15:46 |
| marios | lgtm zbr | 15:46 |
| weshay | quiquell let's remove multinode scen01-04 from master promotion | 15:46 |
| vinaykns | jaosorior: how do we get the value ansible_nodename in the novajoin-container-puppet.yaml | 15:47 |
| vinaykns | is that fqdn..? | 15:47 |
| quiquell | weshay: ack, will do tomorrow | 15:47 |
| quiquell | weshay: also I don't know if we need standalone job at mistral looks like is not using mistral not even at config-download | 15:47 |
| vinaykns | jaosorior: I guess in my setup the principal name is incorrectly being identified. | 15:48 |
| jaosorior | vinaykns: that's provided by ansible | 15:48 |
| *** kmrchdn is now known as chandankumar | 15:48 | |
| weshay | quiquell probably true | 15:48 |
| weshay | let's do one at a time though | 15:48 |
| quiquell | ack | 15:50 |
| rfolco | weshay, we have volunteers, its your call now, on you | 15:51 |
| sshnaidm | rlandy, tbh I don't know why not to inherit all non-master jobs from masters one, it will save all these dups | 15:52 |
| sshnaidm | rlandy, and allow things to be changed in one place | 15:53 |
| *** quiquell is now known as quiquell|off | 15:56 | |
| weshay | rfolco panda please join my blue | 15:58 |
| rfolco | ok | 15:59 |
| sshnaidm | panda, wrt https://review.rdoproject.org/r/#/c/18922/2/zuul.d/ovb-jobs.yaml - not sure what is difference between this and any other branch periodic job | 16:00 |
| sshnaidm | panda, in 001 rocky job we don't use override checkout too | 16:00 |
| panda | sshnaidm: we recently discovered that just using branch_override wasn't enough for some jobs | 16:01 |
| panda | sshnaidm: maybe there we are making the same mistake | 16:01 |
| sshnaidm | panda, yeah, I think we talked about it with quiquell|off.. | 16:01 |
| sshnaidm | panda, but then other do the same currently | 16:01 |
| sshnaidm | panda, yeah, it should be fixed for all jobs at once | 16:04 |
| panda | sshnaidm: ok, do you prefer to merge this for consistency and then try to understand if you need to chenge everything ? | 16:04 |
| panda | sshnaidm: it can be two separated patches. | 16:05 |
| panda | sshnaidm: so at least tls master will run in the meantime | 16:05 |
| sshnaidm | panda, we do need change everything, but it's completely different task | 16:05 |
| panda | sshnaidm: yep agree | 16:05 |
| sshnaidm | I wonder if zuul jobs could support multiple inheritance.. | 16:06 |
| *** jfrancoa has quit IRC | 16:15 | |
| ykarel|pto | sshnaidm, you mean support two parent? | 16:27 |
| sshnaidm | ykarel|pto, yep | 16:27 |
| ykarel|pto | sshnaidm, see https://review.openstack.org/#/c/629983/3 | 16:27 |
| ykarel|pto | possibly you are looking for similar | 16:28 |
| sshnaidm | ykarel|pto, hmm.. not sure I understand this patch.. I see 2 definitions of the same job "system-config-build-image-gitea" - does it mean it merges vars from first time and then other vars from second definition? | 16:33 |
| sshnaidm | need to try this in reproducer.. | 16:34 |
| ykarel|pto | sshnaidm, i too haven't looked much on it, just saw multiple parent discussion few days back | 16:36 |
| ykarel|pto | and when you asked about it, i shared it to you from irc logs | 16:36 |
| sshnaidm | ykarel|pto, cool, seems like requested feature.. | 16:37 |
| ykarel|pto | sshnaidm, you can find more info here http://eavesdrop.openstack.org/irclogs/%23zuul/%23zuul.2019-01-29.log.html#t2019-01-29T00:18:59 | 16:38 |
| * ykarel|pto can only check once i am back on Friday | 16:38 | |
| sshnaidm | ykarel|pto, great, thanks for the info | 16:38 |
| jaosorior | vinaykns: hey, did you sort out the TLS everywhere issue you were seeing? | 17:01 |
| jaosorior | sorry for the late replies, it's been hectic with meetings | 17:01 |
| chandankumar | weshay: quiquell|off telemetry tests passing now https://logs.rdoproject.org/49/18749/13/check/periodic-tripleo-ci-centos-7-multinode-1ctlr-featureset016-master/c3084fb/logs/tempest.html | 17:10 |
| *** dtantsur is now known as dtantsur|afk | 17:14 | |
| chandankumar | sshnaidm: http://logs.openstack.org/59/638159/1/check/tripleo-ci-centos-7-standalone/404e982/logs/undercloud/var/log/extra/dstat.html.gz dstat ouput from standalone | 17:15 |
| sshnaidm | chandankumar++ | 17:16 |
| chandankumar | sshnaidm: do we need atop and dstat both in standalone? | 17:17 |
| sshnaidm | chandankumar, atop gives a little bit different info and it's browseable, so I think it's worth to have them both | 17:17 |
| chandankumar | sshnaidm: cool then ;-) | 17:17 |
| *** bogdando has quit IRC | 17:18 | |
| *** chandankumar is now known as kmrchdn | 17:19 | |
| vinaykns | jaosorior: I just ran the install script, still could see the error | 17:21 |
| vinaykns | jaosorior: thought it will work now | 17:21 |
| jaosorior | vinaykns: how did you run the script? | 17:24 |
| *** dsneddon has quit IRC | 17:25 | |
| vinaykns | http://pastebin.test.redhat.com/718728 | 17:25 |
| jaosorior | vinaykns: ah! you're trying with an external FreeIPA | 17:26 |
| vinaykns | yes | 17:26 |
| jaosorior | vinaykns: do you need that external FreeIPA? quickstart can also install that for you if you need | 17:26 |
| vinaykns | jaosorior: I tried that..i didn't get successful results | 17:26 |
| jaosorior | vinaykns: did you try: ./quickstart.sh --no-clone --teardown all --clean -p quickstart-extras.yml \ | 17:26 |
| vinaykns | so i went through this option | 17:26 |
| jaosorior | -N config/nodes/1ctlr_1comp_1supp.yml \ | 17:26 |
| jaosorior | -c config/general_config/ipa.yml \ | 17:26 |
| jaosorior | -R master-tripleo-ci \ | 17:26 |
| jaosorior | --tags all \ | 17:26 |
| jaosorior | $VIRTHOST | 17:27 |
| vinaykns | yes | 17:27 |
| jaosorior | vinaykns: there were some errors upstream cause of the flattening work | 17:27 |
| jaosorior | but most of those issues should have been fixed by now | 17:27 |
| jaosorior | vinaykns: sshnaidm even had a successful run for TLS everywhere in CI today | 17:27 |
| jaosorior | vinaykns: so, you might wanna try again | 17:27 |
| *** dsneddon has joined #oooq | 17:27 | |
| weshay | marios I didn't see another review btw https://review.rdoproject.org/r/#/c/18926/ | 17:28 |
| sshnaidm | jaosorior, vinaykns not sure quickstart-libvirt version still works | 17:28 |
| sshnaidm | jaosorior, vinaykns need to test this.. | 17:28 |
| jaosorior | sshnaidm: aha | 17:28 |
| jaosorior | let me try that out | 17:28 |
| jaosorior | I'll let you know how it goes | 17:28 |
| vinaykns | sshnaidm: ohh | 17:29 |
| vinaykns | jaosorior: okay..I'll wait for your response | 17:29 |
| jaosorior | sshnaidm, vinaykns: running a new deployment right now | 17:30 |
| jaosorior | lets see how that goes! | 17:30 |
| vinaykns | using libvirt version right..? | 17:30 |
| jaosorior | vinaykns: yeah; running on the host under my desk | 17:30 |
| vinaykns | jaosorior: okay..cool.! | 17:31 |
| weshay | rlandy where did the periodic pipeline config move to? https://review.rdoproject.org/r/#/c/18926/ see that error | 17:33 |
| weshay | rdo-jobs? | 17:33 |
| * weshay doesn't see it | 17:33 | |
| rlandy | move? | 17:37 |
| rlandy | weshay: config | 17:38 |
| rlandy | weshay: https://github.com/rdo-infra/review.rdoproject.org-config/blob/master/zuul.d/tripleo.yaml#L38 | 17:39 |
| weshay | aye thanks | 17:41 |
| weshay | I had a old copy of this repo | 17:41 |
| weshay | thanks https://review.rdoproject.org/r/#/c/18928/ | 17:46 |
| *** jaosorior has quit IRC | 17:49 | |
| *** jaosorior has joined #oooq | 17:49 | |
| *** derekh has quit IRC | 17:50 | |
| *** panda is now known as panda|off | 17:59 | |
| rlandy | weshay: 1-1? | 18:02 |
| weshay | rlandy aye | 18:02 |
| weshay | marios you still around? | 18:02 |
| weshay | do I just respond first to you to your email w/ approval for $x amount? | 18:02 |
| *** jpena is now known as jpena|off | 18:11 | |
| *** kopecmartin is now known as kopecmartin|off | 18:15 | |
| *** amoralej is now known as amoralej|off | 18:21 | |
| *** ykarel|pto has quit IRC | 18:22 | |
| *** holser_ has quit IRC | 18:28 | |
| zbr | please help with https://review.openstack.org/#/c/638108/ -- needed for f28 (thanks marios) | 18:30 |
| weshay | rlandy https://review.openstack.org/#/c/638016/ | 18:43 |
| *** jaosorior has quit IRC | 19:18 | |
| rfolco | zbr, around ? | 19:19 |
| zbr | rfolco: yes, but kinda busy with weekly community meeting on molecule. | 19:20 |
| rfolco | ah ok np zbr | 19:20 |
| zbr | rfolco: what did you want to ask? | 19:23 |
| rfolco | you remember that tripleo-common depends-on | 19:23 |
| rfolco | on ps 20 it had a depends-on for kolla patch --> https://review.openstack.org/#/c/629679/20 | 19:24 |
| rfolco | which was Depends-On: https://review.openstack.org/#/c/632156/ | 19:24 |
| rfolco | why we just dropped it ? | 19:24 |
| rfolco | zbr, ^ | 19:24 |
| rfolco | I am trying to find what is missing coz our job is failing -- http://logs.openstack.org/60/636160/35/check/tripleo-build-containers-fedora-28/2c31281/logs/build-err.log.txt.gz | 19:25 |
| zbr | rfolco: because it would have never merged with it. the kolla patch is not supposed to merge. | 19:25 |
| rfolco | zbr, but we did not applied that patch anywhere | 19:25 |
| rfolco | did not apply* | 19:25 |
| rfolco | kolla build is failing on | 19:26 |
| rfolco | ERROR:kolla.common.utils.base:The command '/bin/sh -c CURRENT_DISTRO_RELEASE=$(awk '{match($0, /[0-9]+/,version)}END{print version[0]}' /etc/system-release); if [ $CURRENT_DISTRO_RELEASE != "28" ]; then echo "Only release '28' is supported on fedora"; false; fi && cat /tmp/kolla_bashrc >> /etc/bashrc && sed -i 's|^\(override_install_langs=.*\)|# \1|' /etc/yum.conf' returned a non-zero code: 2 | 19:26 |
| rfolco | so that patch might be the reason ... will take a look more closely | 19:26 |
| *** jaosorior has joined #oooq | 19:28 | |
| *** saneax has joined #oooq | 19:28 | |
| *** rfolco has quit IRC | 19:32 | |
| zbr | rfolco: if I remember well alex had two ways to fix kolla, one is 624838 and the other one is 632156 - we cannot use both of them and based on the feedback the one we have in cherry pick seems to be the winner. | 19:35 |
| zbr | or to put it i a different way: our tripleo-ci change must avoid any depends-on on changes that will not be merged very soon. we need the job running, not this does mean that the f28 job must be passing. | 19:37 |
| *** rfolco has joined #oooq | 19:39 | |
| *** saneax has quit IRC | 19:53 | |
| panda|off | zbr: rfolco disconnected exactly the time you took to write the two lines. | 20:13 |
| rfolco | zbr, panda|off true! | 20:13 |
| rfolco | I missed your response probably zbr | 20:13 |
| rfolco | zbr, I am trying to understand if https://review.openstack.org/#/c/632156/ is still required, probably you or alex can tell | 20:14 |
| panda|off | zbr | rfolco: if I remember well alex had two ways to fix kolla, one is 624838 and the other one is 632156 - we cannot use both of them and based on the feedback the one we have in cherry pick seems to be the winner. | 20:15 |
| panda|off | zbr | or to put it i a different way: our tripleo-ci change must avoid any depends-on on changes that will not be merged very soon. we need the job running, not this does mean that the f28 job must be passing. | 20:15 |
| rfolco | 632156 was depends-on for the tripleo-common patch that merged. This is not the same as 624838 which was being cherry-picked already. | 20:16 |
| rfolco | on patchset 20, zbr just dropped it from depends-on on tripleo-common patch | 20:17 |
| panda|off | rfolco: just pasted you the two lines you missed. | 20:18 |
| rfolco | thanks panda|off | 20:18 |
| rfolco | panda|not-so-off :) | 20:18 |
| *** jtomasek has quit IRC | 20:37 | |
| weshay | rfolco are we looking for https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-centos-7-master-containers-build-push in the periodic jobs, do I have the name correct? | 20:52 |
| weshay | helloooooooooooo | 21:01 |
| weshay | is it me your looking for | 21:01 |
| weshay | papa can you hear me | 21:01 |
| rlandy | it's a whole concert going on here | 21:06 |
| weshay | :) | 21:07 |
| weshay | zbr so while poking at tripleo-repos, shouldn't the delorean-deps repo get installed automatically w/ current-tripleo or other tag? | 21:08 |
| *** dsneddon has quit IRC | 21:40 | |
| *** dsneddon has joined #oooq | 21:42 | |
| rlandy | weshay: pls take a look at https://code.engineering.redhat.com/gerrit/#/c/163443/ and comment on the parent/inheritance | 22:54 |
| *** rascasoft has quit IRC | 23:12 | |
| *** rlandy is now known as rlandy|bbl | 23:14 | |
| *** agopi has quit IRC | 23:17 | |
| *** vinaykns has quit IRC | 23:27 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!