| *** saneax has quit IRC | 00:30 | |
| *** ykarel|away has joined #oooq | 01:45 | |
| *** apetrich has quit IRC | 02:10 | |
| *** udesale has joined #oooq | 03:55 | |
| *** raukadah is now known as chandankumar | 03:58 | |
| *** ykarel|away has quit IRC | 04:01 | |
| *** ykarel|away has joined #oooq | 04:22 | |
| *** ykarel|away is now known as ykarel | 04:22 | |
| *** skramaja has joined #oooq | 04:31 | |
| *** saneax has joined #oooq | 04:42 | |
| *** marios has joined #oooq | 05:14 | |
| *** soniya29 has joined #oooq | 05:23 | |
| *** dsneddon has quit IRC | 05:27 | |
| *** dsneddon has joined #oooq | 05:29 | |
| *** ksambor has joined #oooq | 05:32 | |
| *** ratailor has joined #oooq | 05:40 | |
| *** matbu has quit IRC | 06:00 | |
| chandankumar | kopecmartin|off: https://review.rdoproject.org/r/#/c/23033/ | 06:01 |
|---|---|---|
| *** kopecmartin|off is now known as kopecmartin | 06:07 | |
| *** slaweq has joined #oooq | 06:08 | |
| *** jfrancoa has joined #oooq | 06:09 | |
| *** jbadiapa has joined #oooq | 06:29 | |
| *** d0ugal has quit IRC | 06:32 | |
| *** jaosorior has quit IRC | 06:33 | |
| ykarel | chandankumar, fyi https://review.opendev.org/#/c/687249/ was need for Train phase 1 | 06:36 |
| ykarel | can u add it planning doc, so it's not missed during next release | 06:36 |
| chandankumar | ykarel: sure | 06:41 |
| marios | sshnaidm|pto: remove -1 if you have a sec please thanks :) https://review.opendev.org/#/c/683826/1 | 06:41 |
| ykarel | Thanks | 06:43 |
| *** jtomasek has joined #oooq | 06:50 | |
| *** jtomasek has quit IRC | 06:52 | |
| *** jtomasek has joined #oooq | 07:03 | |
| *** jaosorior has joined #oooq | 07:04 | |
| *** tesseract has joined #oooq | 07:13 | |
| *** tosky has joined #oooq | 07:14 | |
| *** saneax has quit IRC | 07:20 | |
| *** bogdando has joined #oooq | 07:21 | |
| *** apetrich has joined #oooq | 07:24 | |
| *** ccamacho has joined #oooq | 07:24 | |
| *** holser has joined #oooq | 07:34 | |
| *** surpatil has joined #oooq | 07:41 | |
| *** slaweq has quit IRC | 07:42 | |
| *** ykarel is now known as ykarel|lunch | 07:44 | |
| *** jpena|off is now known as jpena | 07:46 | |
| *** holser has quit IRC | 07:49 | |
| *** dtantsur|afk is now known as dtantsur | 07:49 | |
| *** holser has joined #oooq | 07:49 | |
| *** slaweq has joined #oooq | 07:49 | |
| *** tosky has quit IRC | 07:58 | |
| *** tosky has joined #oooq | 07:59 | |
| *** soniya29 has quit IRC | 08:02 | |
| *** chem|eod is now known as chem | 08:02 | |
| *** akahat has joined #oooq | 08:05 | |
| *** saneax has joined #oooq | 08:32 | |
| *** derekh has joined #oooq | 08:33 | |
| *** ykarel|lunch is now known as ykarel | 08:34 | |
| *** soniya29 has joined #oooq | 08:38 | |
| *** epoojad1 has joined #oooq | 08:52 | |
| *** jaosorior has quit IRC | 08:53 | |
| *** d0ugal has joined #oooq | 08:56 | |
| chandankumar | arxcruz|rover: Hello | 09:08 |
| arxcruz|rover | chandankumar: hello | 09:08 |
| chandankumar | arxcruz|rover: I was going through fs020 tempest results and found that, we donot run ironic tests there | 09:08 |
| arxcruz|rover | chandankumar: how ironic it is we don't run ironic ? | 09:08 |
| arxcruz|rover | ba dum tssss | 09:09 |
| chandankumar | arxcruz|rover: http://logs.rdoproject.org/10/23010/1/check/periodic-tripleo-ci-centos-7-ovb-1ctlr_2comp-featureset020-master/baa4d80/logs/tempest.html.gz | 09:09 |
| chandankumar | arxcruz|rover: ironic disabled | 09:09 |
| arxcruz|rover | chandankumar: is ironic enabled on fs020 ? | 09:09 |
| arxcruz|rover | i mean, the service | 09:09 |
| chandankumar | arxcruz|rover: from tempest.conf check the service list http://logs.rdoproject.org/10/23010/1/check/periodic-tripleo-ci-centos-7-ovb-1ctlr_2comp-featureset020-master/baa4d80/logs/undercloud/home/zuul/tempest/etc/tempest.conf.txt.gz | 09:09 |
| arxcruz|rover | chandankumar: sorry, first coffee yet, is python-tempestconf detects ironic ? | 09:10 |
| arxcruz|rover | if so, that means ironic service is not enabled right ? | 09:10 |
| chandankumar | arxcruz|rover: let me check that | 09:11 |
| arxcruz|rover | dmellado: hallo :) | 09:11 |
| arxcruz|rover | dmellado: wie geht's dein Sohn? | 09:12 |
| chandankumar | arxcruz|rover: in fs60 and 54 ironic is enabled | 09:19 |
| chandankumar | as standalone-scenario12 and multinode | 09:19 |
| arxcruz|rover | chandankumar: so, that means tempestconf can detect right ? | 09:19 |
| chandankumar | arxcruz|rover: yes | 09:21 |
| chandankumar | arxcruz|rover: https://f14d9b704a9d947abb07-39d40a512ecd567406090b9a01c1d98a.ssl.cf1.rackcdn.com/679558/7/check/tripleo-ci-centos-7-scenario012-standalone/5899d31/logs/undercloud/home/zuul/tempest.log.txt.gz | 09:21 |
| chandankumar | but currently the job is broken | 09:21 |
| chandankumar | arxcruz|rover: in fs020 ironic is not there | 09:22 |
| arxcruz|rover | so it's right | 09:22 |
| arxcruz|rover | brb | 09:22 |
| chandankumar | and temepstconf can detect ironic | 09:22 |
| *** jaosorior has joined #oooq | 09:31 | |
| *** surpatil has quit IRC | 09:59 | |
| *** d0ugal has quit IRC | 10:04 | |
| *** dsneddon has quit IRC | 10:35 | |
| *** soniya29 has quit IRC | 10:36 | |
| *** tesseract has quit IRC | 10:44 | |
| *** tesseract has joined #oooq | 10:46 | |
| *** d0ugal has joined #oooq | 10:52 | |
| *** dsneddon has joined #oooq | 11:04 | |
| *** akahat has quit IRC | 11:05 | |
| *** ratailor has quit IRC | 11:18 | |
| *** dtantsur is now known as dtantsur|brb | 11:22 | |
| *** udesale has quit IRC | 11:22 | |
| *** jpena is now known as jpena|lunch | 11:29 | |
| *** d0ugal has quit IRC | 11:37 | |
| *** dsneddon has quit IRC | 11:41 | |
| *** apetrich has quit IRC | 11:50 | |
| weshay|ruck | epoojad1, hey.. can you chat in like 20 min? | 11:56 |
| weshay|ruck | sorry I missed the mtg, but that's 5:30 am for me.. I start at 6am | 11:57 |
| panda | weshay|ruck: slacker. | 11:57 |
| *** jaosorior has quit IRC | 11:58 | |
| weshay|ruck | panda, :) you work crazy hours too!! | 11:58 |
| *** epoojad1 has quit IRC | 11:59 | |
| chandankumar | weshay|ruck: Pooja will be joining the call soon | 11:59 |
| *** dsneddon has joined #oooq | 12:00 | |
| *** apetrich has joined #oooq | 12:03 | |
| weshay|ruck | arxcruz|rover, morning.. | 12:06 |
| arxcruz|rover | weshay|ruck: hey boss | 12:06 |
| weshay|ruck | arxcruz|rover, can you investigate stein rdo phase 1 | 12:06 |
| weshay|ruck | we're almost entirely green :) | 12:07 |
| weshay|ruck | http://rhos-release.virt.bos.redhat.com:3030/rhosp | 12:07 |
| weshay|ruck | arxcruz|rover, it could be ci.centos | 12:07 |
| weshay|ruck | but the fact the other releases are passing, makes me wonder. | 12:07 |
| weshay|ruck | last few fails have been ci.centos I think | 12:08 |
| weshay|ruck | arxcruz|rover, you can also run ./quickstart.sh locally on stein | 12:08 |
| weshay|ruck | if that passes we'll waive it | 12:08 |
| arxcruz|rover | weshay|ruck: tripleo-quickstart-promote-stein-current-tripleo-delorean-minimal | 12:08 |
| weshay|ruck | arxcruz|rover, https://ci.centos.org/view/rdo/view/promotion-pipeline/job/rdo_trunk-promote-stein-current-tripleo/202/ login, click "resume build" | 12:09 |
| weshay|ruck | arxcruz|rover, 2019-10-09 05:15:22 | Ansible timed out at 3414 seconds. | 12:09 |
| weshay|ruck | https://ci.centos.org/artifacts/rdo/jenkins-tripleo-quickstart-promote-stein-current-tripleo-delorean-minimal-207/undercloud/home/stack/overcloud_deploy.log.gz | 12:09 |
| arxcruz|rover | weshay|ruck: yes, that's the one i'm looking right now | 12:10 |
| weshay|ruck | arxcruz|rover, --timeout 90 | 12:10 |
| weshay|ruck | arxcruz|rover, that should be at 120 | 12:10 |
| weshay|ruck | so.. something is wrong w/ ci.centos as it's not picking up the change | 12:10 |
| weshay|ruck | arxcruz|rover, suggestion is to kill jenkins, then run locally | 12:11 |
| ykarel | weshay|ruck, which change is not being picked up in ci.centos? timeout was updated for only ovb jobs afaik | 12:13 |
| weshay|ruck | ykarel, ah.. that could be | 12:13 |
| ykarel | i mentioned ^^ in meeting with u few days back | 12:13 |
| weshay|ruck | :) | 12:13 |
| weshay|ruck | ykarel, I thought we adjusted the default | 12:14 |
| * weshay|ruck looks | 12:14 | |
| chandankumar | arxcruz|rover: weshay|ruck did we have a patch up for updating the http://rhos-release.virt.bos.redhat.com:3030/rhosp ? | 12:14 |
| rfolco | panda, you fixed it :) | 12:15 |
| weshay|ruck | https://opendev.org/openstack/tripleo-quickstart-extras/src/branch/master/roles/overcloud-deploy/defaults/main.yml#L30 | 12:15 |
| weshay|ruck | arxcruz|rover, so change that to 120 | 12:15 |
| *** apetrich has quit IRC | 12:18 | |
| weshay|ruck | chandankumar, panda https://review.opendev.org/#/c/687498 when you have a sec | 12:20 |
| weshay|ruck | arxcruz|rover, no risks in the next month | 12:22 |
| weshay|ruck | other than upstream gate timeouts | 12:22 |
| weshay|ruck | arxcruz|rover, ^ | 12:23 |
| *** jpena|lunch is now known as jpena | 12:23 | |
| weshay|ruck | arxcruz|rover, thanks | 12:24 |
| arxcruz|rover | chandankumar: yes, wes have a patch | 12:25 |
| rfolco | panda, can I remove wip from commit msg ? can we merge it ? | 12:26 |
| *** jfrancoa has quit IRC | 12:26 | |
| *** jaosorior has joined #oooq | 12:28 | |
| chandankumar | arxcruz|rover: do we have a bug for that https://review.opendev.org/#/c/687546/ ? | 12:32 |
| *** jfrancoa has joined #oooq | 12:32 | |
| chandankumar | I think on monday we have increased the deploy timeout globally | 12:32 |
| arxcruz|rover | chandankumar: not exactly for this one | 12:32 |
| arxcruz|rover | chandankumar: this is for rdo phase 1 | 12:32 |
| chandankumar | arxcruz|rover: ok | 12:34 |
| chandankumar | arxcruz|rover: do we have a bug for scenario 12 http://zuul.openstack.org/builds?job_name=tripleo-ci-centos-7-scenario012-standalone it is broken from very long time | 12:35 |
| rfolco | panda, do not merge yet, I am fixing image prepare log (empty) | 12:42 |
| panda | rfolco: ok | 12:43 |
| panda | I was having lunch anyway | 12:44 |
| rfolco | panda, ok thanks for fixing the job :) | 12:44 |
| panda | rfolco: it's my job, to fix the job | 12:45 |
| panda | rfolco: you have 10 minutes to fix the logs | 12:47 |
| rfolco | pressure pressure pressure | 12:47 |
| *** aakarsh has quit IRC | 12:49 | |
| arxcruz|rover | chandankumar: no | 12:49 |
| rfolco | panda, why @dev tools in setup_staging AND prepare ? | 12:50 |
| rfolco | panda, for delegated, gcc is installed in https://review.rdoproject.org/r/#/c/22988/44/playbooks/molecule-delegated-pre.yml, L33 | 12:53 |
| panda | rfolco: it's in prepare because the gcc must be present in the container, not in the host machine. | 12:53 |
| panda | rfolco: If we don't need gcc at the point you wanted to install it, then you can remove that task | 12:53 |
| rfolco | panda, I'll keep it anyway, we can remove dups later | 12:54 |
| panda | yes, let's do it like this now. we can change it later (TM) | 12:54 |
| weshay|ruck | panda, you want to chat or skip? | 12:57 |
| panda | weshay|ruck: you mean postpone the 1-1 ? I may have something to chat later yes. | 12:58 |
| weshay|ruck | panda, ok.. feel free to reschedule | 12:59 |
| weshay|ruck | chandankumar, is the new reproducer still working for you? | 13:00 |
| weshay|ruck | chandankumar, if it is.. can you try w/ https://review.rdoproject.org/r/#/c/21672/ | 13:01 |
| chandankumar | weshay|ruck: ok, trying now | 13:02 |
| weshay|ruck | chandankumar, thank you arxcruz|rover ^ | 13:02 |
| *** chem has quit IRC | 13:03 | |
| weshay|ruck | chandankumar, mine keeps dying in pre.. when it tries to pull all the repos | 13:03 |
| weshay|ruck | and I can't see what is failing | 13:03 |
| *** chem has joined #oooq | 13:03 | |
| panda | oh the restro is in 30 minutes | 13:03 |
| panda | rfolco: so you have another 30 minutes to fix the logs | 13:03 |
| weshay|ruck | rfolco, fyi.. I have several conflicts today, I'll be in and out | 13:03 |
| rfolco | panda, I just tried moving 2>&1 to the end of the shell cmd | 13:04 |
| rfolco | submitted a new ps, look there | 13:04 |
| rfolco | weshay|ruck, ack | 13:04 |
| *** dsneddon has quit IRC | 13:04 | |
| panda | rfolco: was it empty also in the molecule job \ | 13:05 |
| panda | ? | 13:05 |
| weshay|ruck | arxcruz|rover, jump on w/ me for a sec | 13:05 |
| panda | rfolco: fi yes, I suggest to add a test to check if the logfile size is > 0 | 13:05 |
| *** apetrich has joined #oooq | 13:05 | |
| weshay|ruck | arxcruz|rover, https://meet.google.com/bkq-cvfe-ymp | 13:05 |
| rfolco | panda, you becoming paranoiac | 13:06 |
| *** slaweq has quit IRC | 13:07 | |
| panda | rfolco: that's what tests are for. you find a bug and you put a check to not let the bug happen ever again. | 13:08 |
| rfolco | panda, check if its really molecule executing the test and not something else ? | 13:08 |
| rfolco | panda, I'm joking, see my smile here >>>> :) | 13:08 |
| panda | rfolco: yes! metatest, check if the author of the patch is really who we think it is. We can use machine learning techniques to match each team member style | 13:09 |
| panda | rfolco: test if the test is really a test. | 13:09 |
| rfolco | hahah | 13:09 |
| rfolco | good one | 13:09 |
| arxcruz|rover | weshay|ruck: https://review.opendev.org/#/c/687497/ | 13:10 |
| weshay|ruck | arxcruz|rover, http://zuul.openstack.org/builds?job_name=tripleo-ci-centos-7-scenario010-standalone | 13:12 |
| *** d0ugal has joined #oooq | 13:13 | |
| weshay|ruck | arxcruz|rover, beagles on irc | 13:17 |
| panda | rfolco: http://logs.rdoproject.org/88/22988/45/check/tripleo-ci-promotion-staging/df533aa/logs/tripleo-container-image.prepare.log | 13:18 |
| panda | rfolco: but you broke rdo-tox again | 13:18 |
| rfolco | panda, I did not touch package install | 13:19 |
| panda | rfolco: you're lying | 13:19 |
| rfolco | did I ? | 13:19 |
| panda | rfolco: dunno, I'm rechecking, seems a temporary reachability problem | 13:20 |
| *** aakarsh has joined #oooq | 13:21 | |
| rfolco | panda, add a test to check yum cache | 13:21 |
| rfolco | panda, another one to check if committer is lying | 13:22 |
| panda | rfolco: seriously, we can and should add test for everything that is under OUR control. | 13:22 |
| rfolco | panda, relax, I agree, I'm just annoying you | 13:22 |
| rfolco | retro mtg starts in 5 min, please add your cards so we save some time | 13:25 |
| rfolco | https://trello.com/b/0VFswmht/rdo-infra-retrospective?menu=filter&filter=label:UniSprint16 | 13:26 |
| panda | rfolco: I claim the right of vengeance, I challenge you to a Gik'tal | 13:28 |
| panda | rfolco: I'd also like to merge 22890 before tomorrow | 13:29 |
| rfolco | panda, no idea what you talking about | 13:29 |
| rfolco | panda, ok lets do it, we're close | 13:29 |
| panda | rfolco: all right, it's on! defend yourself. | 13:29 |
| *** dsneddon has joined #oooq | 13:30 | |
| *** jaosorior has quit IRC | 13:33 | |
| ykarel | arxcruz|rover, weshay|ruck are u aware of the issue with full tempest jobs? | 13:34 |
| ykarel | seems real issue, it's timing out always from some time | 13:34 |
| ykarel | nothing found in etherpad, so seems not known | 13:36 |
| ykarel | https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-centos-7-standalone-full-tempest-master&job_name=periodic-tripleo-ci-centos-7-ovb-1ctlr_2comp-featureset020-master&job_name=periodic-tripleo-ci-centos-7-ovb-1ctlr_2comp-featureset020-train&job_name=periodic-tripleo-ci-centos-7-ovb-1ctlr_2comp-featureset020-train&job_name=periodic-tripleo-ci-centos-7-ovb-1ctlr_2comp-featureset020-stein&job_name=periodic-tripleo-ci-centos-7-st | 13:36 |
| ykarel | andalone-full-tempest-train | 13:36 |
| *** dsneddon has quit IRC | 13:36 | |
| panda | rfolco: jobs passed , voted +2 | 13:42 |
| *** saneax has quit IRC | 13:43 | |
| arxcruz|rover | weshay|ruck: https://review.opendev.org/#/q/topic:scen10-nv+(status:open+OR+status:merged) | 13:44 |
| *** udesale has joined #oooq | 13:45 | |
| marios | rfolco: chandankumar fyi created UniSprint17 label on https://trello.com/b/0VFswmht/rdo-infra-retrospective?menu=filter&filter=label:UniSprint17 | 13:46 |
| *** rf0lc0 has joined #oooq | 13:51 | |
| *** chem_ has joined #oooq | 13:54 | |
| *** chem has quit IRC | 13:56 | |
| *** rfolco has quit IRC | 13:56 | |
| *** apetrich has quit IRC | 13:56 | |
| *** migi has quit IRC | 13:56 | |
| *** mjturek has quit IRC | 13:56 | |
| *** mjturek has joined #oooq | 13:56 | |
| *** openstackstatus has quit IRC | 13:58 | |
| *** Vorrtex has joined #oooq | 13:58 | |
| *** apetrich has joined #oooq | 14:00 | |
| panda | marios: do you have the powers of +2 in ci-config ? | 14:02 |
| weshay|ruck | panda, what review? | 14:02 |
| *** rf0lc0 is now known as rfolco | 14:02 | |
| panda | weshay|ruck: final promoter patch, https://review.rdoproject.org/r/22988, but it needs some background, I think marios should at least rereview it | 14:03 |
| *** jaosorior has joined #oooq | 14:03 | |
| marios | panda: ack gimme couple mins will check | 14:03 |
| marios | panda: i saw it earlier but it was red you updated looks like | 14:03 |
| panda | marios: yep, also rfolco adjusted some bits. | 14:04 |
| rfolco | panda, marios lets merge it ! \o/ | 14:04 |
| *** dtantsur|brb is now known as dtantsur | 14:05 | |
| *** dsneddon has joined #oooq | 14:05 | |
| *** Vorrtex has quit IRC | 14:07 | |
| *** Vorrtex has joined #oooq | 14:07 | |
| *** rascasoft has quit IRC | 14:08 | |
| weshay|ruck | FYI... skopeo is deprecated https://review.opendev.org/#/c/686831/ | 14:09 |
| *** rascasoft has joined #oooq | 14:11 | |
| rfolco | skopeo is out of scope | 14:14 |
| rfolco | panda, ^ | 14:14 |
| *** ChanServ sets mode: +o panda | 14:15 | |
| *** rfolco was kicked by panda (rfolco) | 14:15 | |
| *** panda sets mode: -o panda | 14:15 | |
| *** rfolco has joined #oooq | 14:15 | |
| rfolco | panda, rrrrrrrhhhh | 14:15 |
| *** Goneri has joined #oooq | 14:16 | |
| rfolco | marios, I am verifying as qe your manifest task, delegated is the only job that really exercises manifests - http://logs.rdoproject.org/02/22002/72/check/molecule-delegated/35f2282/job-output.txt.gz | 14:16 |
| marios | rfolco: thanks | 14:17 |
| marios | rfolco: no also the ci-promotion-staging | 14:17 |
| marios | rfolco: it tests manifests too.. basically anywhere that does container push | 14:17 |
| rfolco | marios, show me where | 14:18 |
| rfolco | please | 14:18 |
| rfolco | http://logs.rdoproject.org/02/22002/72/check/tripleo-ci-promotion-staging/04863ec/job-output.txt.gz | 14:18 |
| rfolco | cannot find here | 14:18 |
| *** apetrich has quit IRC | 14:18 | |
| marios | rfolco: here http://logs.rdoproject.org/88/22988/45/check/tripleo-ci-promotion-staging/8e8e6fa/logs/promoter_logs/centos7_master.log | 14:18 |
| panda | rfolco: that review is ancient, probably even before the amanifest push was plugged into the job | 14:18 |
| rfolco | aaah that exaplains | 14:19 |
| marios | rfolco: 2019-10-09 13:30:19,337 18309 INFO promoter included: /home/zuul/ci-config/ci-scripts/container-push/roles/containers-promote/tasks/manifest-push.yml for localhost | 14:19 |
| rfolco | explains | 14:19 |
| rfolco | ok cool, thx marios panda | 14:19 |
| marios | panda: rfolco unrelated.. had a go at that today https://tree.taiga.io/project/tripleo-ci-board/task/1325 check hashes digging/starging dlrnapi_promoter.py ;) posted skeleton https://review.rdoproject.org/r/23037 it will obviously be bumped to sprint 17 but wondering what you think ... if that's what we had in mind also cc weshay|ruck | 14:19 |
| rfolco | marios, this can be handled as an additional test by the person who will take over promoter improvements | 14:21 |
| rfolco | IMHO | 14:21 |
| panda | marios: very related, git diff --stat sasy we added approximately 3500 lines of code for the promoter tests. | 14:21 |
| rfolco | wow | 14:21 |
| panda | marios: and removed ~800 | 14:21 |
| marios | panda: what do you mean just your one change was +800 -800 so surely more than that ;) | 14:22 |
| rfolco | we'll remove more when we cut dups off | 14:22 |
| rfolco | marios, lol | 14:22 |
| panda | marios: 98 files changed, 4187 insertions(+), 988 deletions(-) but there are other patches in the middle of my diff --stat | 14:22 |
| chandankumar | weshay|ruck: arxcruz|rover seen this issue with the reproducer http://paste.openstack.org/show/782451/ | 14:26 |
| chandankumar | open /proc/sys/net/ipv4/tcp_keepalive_intvl: no such file or directory | 14:26 |
| *** apetrich has joined #oooq | 14:27 | |
| chandankumar | the file exists on the system | 14:27 |
| panda | marios: as a first step I think we can do this directly in the container-push playbook, before and after we run the container-promote role. | 14:31 |
| marios | panda: ack... i thought it belonged in the promoter itself, like per hash we are checking | 14:32 |
| marios | panda: anyway it is just a skeleton and worst case i spent more time digging in that code | 14:32 |
| marios | panda: we should add some developer docs as a task for this too rfolco | 14:33 |
| marios | panda: rfolco 'this' i mean all promoter staging and molecule $stuffs | 14:33 |
| panda | marios: It's not necessary for all the commits, only during the promotion, so for the commit that we know we want to promote. I'll add some comments, inlin and unit tests to verify taht this is really doing what is suppoed to. | 14:34 |
| panda | marios: I will not add unit tests by my self, only suggest to add. | 14:34 |
| marios | panda: yes yes it depends if whoever picks it up will follow that or do it in the playbook which if it is you (most likely, though i think it should be someone else or at least have someone else followingyou to learn about that ) | 14:35 |
| arxcruz|rover | weshay|ruck: CIX meeting ? | 14:36 |
| weshay|ruck | arxcruz|rover, all you baby :) | 14:36 |
| arxcruz|rover | lol | 14:36 |
| weshay|ruck | in a prod-chain mtg atm | 14:36 |
| arxcruz|rover | ok | 14:37 |
| panda | marios: nice the idea to do this in python, frankly I really have the secret purpose of moving all the promoter components to python, instead of having to orchestrate python ansible and bash in a single run | 14:37 |
| panda | did I say that out loud ? | 14:38 |
| panda | dammit. | 14:38 |
| arxcruz|rover | lol | 14:38 |
| panda | rfolco: going to merge 22988 then | 14:39 |
| *** udesale has quit IRC | 14:41 | |
| rfolco | panda you are the promoter king | 14:42 |
| panda | rfolco: waitamminute, WAITAMMINUTE | 14:42 |
| panda | rfolco: this is all duplicated | 14:42 |
| chandankumar | rebooted the repr system and started working | 14:43 |
| chandankumar | what the heck is docker compose | 14:43 |
| marios | panda: rfolco i added a comment about the duplication if that is what you mean | 14:43 |
| rfolco | panda, yes it is. Duplicated and adapted to zuul workflow... | 14:43 |
| marios | like sure lets merge and tidy later if we can't just include_tasks | 14:43 |
| panda | rfolco: you are cheating | 14:44 |
| rfolco | panda, I told you about massive dup | 14:44 |
| marios | panda: are we "prepped" do we have a server etc tmux and all that /me checks invite | 14:45 |
| panda | rfolco: I claim the right of deduplication, I challenge you to a Gik'tal | 14:45 |
| panda | rfolco: prepare to die. | 14:45 |
| rfolco | panda, lol | 14:46 |
| panda | marios: I think weshay|ruck wants to do it himself. | 14:46 |
| panda | marios: from scratch | 14:46 |
| rfolco | panda, are you confident? | 14:46 |
| panda | marios: but I had a simulation of this yesteday, everything work up to the point were we need to really promote, I could not test that part. | 14:47 |
| panda | rfolco: I claim the right of confidence. I challenge you to a Gik'tal. defend yourself! | 14:48 |
| rfolco | panda, I don't know how this gik'tal works, I am googling ... | 14:48 |
| *** udesale has joined #oooq | 14:54 | |
| arxcruz|rover | i'm leaving for german class :/ | 14:58 |
| rfolco | ping weshay|ruck promoter stand up | 15:01 |
| panda | rfolco: is it ok if I sit ? | 15:02 |
| rfolco | panda, no | 15:02 |
| rfolco | you must stand up | 15:02 |
| panda | rfolco: 'sit not ok ? | 15:02 |
| *** jaosorior has quit IRC | 15:11 | |
| *** udesale has quit IRC | 15:14 | |
| *** ykarel is now known as ykarel|afk | 15:26 | |
| *** jfrancoa has quit IRC | 15:28 | |
| *** skramaja has quit IRC | 15:35 | |
| chandankumar | weshay|ruck: registry_login_enabled: false | 15:37 |
| *** jfrancoa has joined #oooq | 15:44 | |
| weshay|ruck | https://review.rdoproject.org/r/#/c/22892/ | 15:46 |
| weshay|ruck | marios, panda ^ | 15:46 |
| weshay|ruck | marios, http://logs.rdoproject.org/92/22892/17/check/tripleo-ci-promotion-staging/ae53273/logs/promoter_logs/centos7_master.log | 15:47 |
| weshay|ruck | marios, panda https://review.rdoproject.org/r/#/c/22765/2/ci-scripts/dlrnapi_promoter/dlrnapi_promoter.py | 15:49 |
| weshay|ruck | panda, /usr/bin/timeout --preserve-status -k 120m 115m python ~/ci-config/ci-scripts/dlrnapi_promoter/dlrnapi_promoter.py ~/ci-config/ci-scripts/dlrnapi_promoter/config/CentOS-7/train.ini | 15:50 |
| weshay|ruck | panda, https://github.com/weshayutin.keys | 15:59 |
| panda | rfolco: 38.145.33.174 | 16:03 |
| *** ccamacho has quit IRC | 16:03 | |
| marios | panda can you please add http://paste.openstack.org/raw/782454/ | 16:08 |
| panda | marios: you said you're in ? | 16:09 |
| marios | panda: yes | 16:10 |
| panda | marios: ook | 16:10 |
| marios | centos@promoter-new | 16:10 |
| marios | panda: ^^ | 16:10 |
| marios | to be clear :D | 16:10 |
| panda | weshay|ruck: it just failed | 16:11 |
| panda | weshay|ruck: it's removing the containers locally | 16:12 |
| panda | weshay|ruck: but it failed | 16:12 |
| panda | weshay|ruck: containers-promote : attempt a push to the logged in registry | 16:13 |
| panda | weshay|ruck: the login check is failing | 16:13 |
| weshay|ruck | :) | 16:13 |
| weshay|ruck | ya | 16:13 |
| weshay|ruck | panda, hrm.. wonder why this is not in the code https://review.rdoproject.org/r/#/c/22806/ | 16:15 |
| weshay|ruck | was it readded? | 16:15 |
| panda | weshay|ruck: it was never removed, I had to port the check to the role, but the removal only touched the old code, not the code in the role | 16:16 |
| weshay|ruck | ya.. it failed | 16:17 |
| weshay|ruck | panda, ok.. we'll need to port it | 16:17 |
| weshay|ruck | panda, marios rfolco ok.. shall I turn on the old promoter? | 16:17 |
| *** holser has quit IRC | 16:17 | |
| weshay|ruck | or let's leave it.. so we can fix and see a rhel8 promote? | 16:18 |
| panda | weshay|ruck: the only way to be sure the new promoter is working is to see a promotion pass | 16:18 |
| panda | weshay|ruck: we have one in our hands now. If we skip thsis opportunity, we'll need to wait for womthing to be promotable | 16:19 |
| panda | promotible ? | 16:19 |
| panda | promotionabol ? | 16:19 |
| panda | whatev | 16:19 |
| panda | I would prever to fix as much as possible on this occasion | 16:20 |
| panda | remembering we don't have any ansible-pull running at the moment | 16:21 |
| weshay|ruck | ok.. I'll leave the old promoter off atm | 16:22 |
| panda | weshay|ruck: https://review.rdoproject.org/r/23040 | 16:24 |
| weshay|ruck | panda, so.. I guess we have a 1-1 | 16:28 |
| *** derekh has quit IRC | 16:28 | |
| weshay|ruck | might as well jump on meet | 16:28 |
| panda | weshay|ruck: which meet id ? | 16:28 |
| weshay|ruck | https://meet.google.com/sgp-krvp-ixa | 16:29 |
| *** ykarel|afk is now known as ykarel | 16:35 | |
| *** marios has quit IRC | 16:36 | |
| weshay|ruck | panda, oh.. forgot to ask.. we can pull the change and retry rhel8 | 16:50 |
| weshay|ruck | panda, I'll try that | 16:50 |
| panda | weshay|ruck: yes | 16:51 |
| *** jpena is now known as jpena|off | 16:53 | |
| *** panda is now known as panda|off | 16:53 | |
| panda|off | rfolco: we all count on you now. | 16:53 |
| panda|off | :) | 16:53 |
| panda|off | (I'll still pop up now and then) | 16:54 |
| weshay|ruck | rfolco, change pulled.. rhel8 master restarted | 16:59 |
| rfolco | weshay|ruck, panda|off which change ? | 17:05 |
| rfolco | what has changed? | 17:05 |
| panda|off | rfolco: https://review.rdoproject.org/r/23040 | 17:11 |
| chandankumar | weshay|ruck: there is something wrong with full tempest and fs01 os_tempest http://logs.rdoproject.org/34/23034/2/check/periodic-tripleo-ci-centos-7-ovb-1ctlr_2comp-featureset020-master/00a2dd4/ | 17:13 |
| chandankumar | current one got timedout | 17:13 |
| weshay|ruck | chandankumar, locally or in ci? | 17:21 |
| weshay|ruck | chandankumar, in my tests it got through the deploy and into tempest and failed in basic_ops | 17:21 |
| chandankumar | weshay|ruck: nope, in ci | 17:22 |
| chandankumar | weshay|ruck: fs039 is running currently | 17:22 |
| weshay|ruck | chandankumar, ok.. but do we have it in a reproducer that we can log into? | 17:23 |
| weshay|ruck | that's what we need | 17:23 |
| ykarel | weshay|ruck, you saw my message regarding full tempest timeout earlier? | 17:26 |
| ykarel | just to be sure u are already on it | 17:26 |
| weshay|ruck | ya.. and the standalone right? | 17:27 |
| ykarel | weshay|ruck, yes both fs020 and standalone-full-tempest | 17:27 |
| chandankumar | weshay|ruck: pub key? | 17:27 |
| ykarel | in train/stein/master | 17:27 |
| weshay|ruck | chandankumar, https://github.com/weshayutin.keys | 17:27 |
| chandankumar | weshay|ruck: zuul@38.145.35.92 | 17:28 |
| chandankumar | for fs039 undercloud | 17:28 |
| chandankumar | it is still running | 17:28 |
| weshay|ruck | chandankumar, perfect.. thanks | 17:28 |
| * weshay|ruck fires up a tmux | 17:29 | |
| chandankumar | once it finishes, please dig in | 17:29 |
| weshay|ruck | chandankumar, aye.. thanks | 17:29 |
| chandankumar | weshay|ruck: I have shared the zuul console log in pm | 17:29 |
| chandankumar | weshay|ruck: panda|off rfolco I will be out on 11th Sept | 17:30 |
| chandankumar | *Oct | 17:30 |
| chandankumar | for pycon india | 17:30 |
| weshay|ruck | chandankumar, k | 17:30 |
| *** ykarel_ has joined #oooq | 17:32 | |
| *** ykarel_ has quit IRC | 17:33 | |
| chandankumar | weshay|ruck: did you get a chance to look at my last comment https://review.opendev.org/#/c/686430/ on proxy patch | 17:34 |
| chandankumar | if you agree, I can update the patch | 17:34 |
| weshay|ruck | chandankumar, ugh.. another area where tripleo and infra can't agree.. tbh we need to redo that code completely | 17:35 |
| baha | rfolco: About to upload the patch to adjust timings to line up with the tripleo promotion job - do you know what timezone the tripleo jobs run in? | 17:35 |
| baha | RDO is just UTC | 17:35 |
| *** ykarel has quit IRC | 17:35 | |
| chandankumar | weshay|ruck: if we get to the point where proxy and trunk does not match, then we can blame AFS mirror setting | 17:37 |
| weshay|ruck | chandankumar, /me kind of sick of it tbh | 17:37 |
| weshay|ruck | chandankumar, leave it for the ruck / rovers please | 17:38 |
| chandankumar | weshay|ruck: ack, rfolco it is coming to your court https://review.opendev.org/#/c/686430/ | 17:38 |
| weshay|ruck | chandankumar, https://review.rdoproject.org/r/23042 | 17:50 |
| chandankumar | weshay|ruck: done | 17:52 |
| weshay|ruck | rock | 17:52 |
| weshay|ruck | chandankumar, u work too much :) | 17:52 |
| chandankumar | weshay|ruck: one more interesting thing about to find out the set of tests we run in full tempest is kind of similar with fs020 | 17:52 |
| * chandankumar is waiting for the results only to prove that | 17:53 | |
| weshay|ruck | k | 17:54 |
| rfolco | baha, oops sorry for the delay | 17:55 |
| baha | rfolco: All good, you've been busy =) | 17:55 |
| rfolco | baha, should be what zuul rdo timezone... let me see | 17:56 |
| rfolco | baha, utc too | 17:57 |
| baha | rfolco: Thank you! | 17:57 |
| rfolco | just opened a console in zuul rdo and checked live timestamps :) | 17:58 |
| rfolco | https://review.rdoproject.org/zuul/stream/bf0cce2d3afb4082bd2daef0ef0d8499?logfile=console.log | 17:58 |
| rfolco | baha, ^ | 17:58 |
| rfolco | 17:57 == UTC :) | 17:58 |
| baha | Awesome! I'll go push up that patch then | 18:00 |
| rfolco | weshay|ruck, promoter failed ... I have no idea why | 18:03 |
| weshay|ruck | 2019-10-09 17:56:36,700 p=22643 u=centos | failed: [localhost] (item=tripleoclient) => {"ansible_loop_var": "item", "changed": false, "item": "tripleoclient", "msg": "Error pulling image tru | 18:03 |
| weshay|ruck | nk.registry.rdoproject.org/tripleomaster/rhel-binary-tripleoclient:ab34c90c15f5cb5efb7236fdf4b22020e6f3cd04_67a09fe9_x86_64 - 404 Client Error: Not Found (\"{\"message\":\"manifest for trunk. | 18:03 |
| weshay|ruck | registry.rdoproject.org/tripleomaster/rhel-binary-tripleoclient:ab34c90c15f5cb5efb7236fdf4b22020e6f3cd04_67a09fe9_x86_64 not found: manifest unknown: manifest unknown\"}\")"} | 18:03 |
| weshay|ruck | failed to pull a few containers | 18:03 |
| *** dtantsur is now known as dtantsur|afk | 18:12 | |
| panda|off | this is the registry-push already, are the _x86_64 images there for rhel8 ? | 18:48 |
| rfolco | why don't we try centos ? | 18:52 |
| weshay|ruck | rfolco, it's not rhel8 | 18:57 |
| weshay|ruck | rhel8 promotes fine on the old promoter | 18:57 |
| weshay|ruck | rfolco, http://dashboard-ci.tripleo.org/d/YRJtmtNWk/cockpit?orgId=1&fullscreen&panelId=284 | 18:57 |
| weshay|ruck | rfolco, can we tell if it's quay causing the failure? | 18:59 |
| weshay|ruck | that is the more likely scenario | 18:59 |
| rfolco | 404 on rdotrunk registry... | 18:59 |
| rfolco | how this is quay related ? | 19:00 |
| weshay|ruck | oh.. we're getting 404's from rdo? | 19:00 |
| weshay|ruck | ah see that | 19:00 |
| weshay|ruck | 2019-10-09 18:52:06,610 p=24141 u=centos | failed: [localhost] (item=horizon) => {"ansible_loop_var": "item", "changed": false, "item": "horizon", "msg": "Error pulling image trunk.registry.rdoproject.org/tripleomaster/rhel-binary-horizon:ab34c90c15f5cb5efb7236fdf4b22020e6f3cd04_67a09fe9_x86_64 - 404 Client Error: Not Found (\"{\"message\":\"manifest for trunk.registry.rdoproject.org/tripleomaster/rhel-binary-horizon:ab34c90c15f5cb5efb723 | 19:00 |
| weshay|ruck | 6fdf4b22020e6f3cd04_67a09fe9_x86_64 not found: manifest unknown: manifest unknown\"}\")"} | 19:00 |
| rfolco | checking hashes | 19:00 |
| weshay|ruck | so let's see if that's there | 19:00 |
| rfolco | _manifest isn't there yet... I'm confused | 19:01 |
| weshay|ruck | rfolco, this is really slow https://trunk.registry.rdoproject.org:8443/oapi/v1/namespaces/tripleomaster/imagestreamtags/ | 19:04 |
| weshay|ruck | rfolco, the containers are there | 19:08 |
| weshay|ruck | this is killing my browser | 19:08 |
| weshay|ruck | rfolco, check ur email | 19:09 |
| rfolco | k | 19:10 |
| rfolco | weshay|ruck, can we try pulling a container from rdo registry manually ... from the new promoter server I mean | 19:11 |
| weshay|ruck | rfolco, ya.. go 4 it.. | 19:11 |
| rfolco | k | 19:12 |
| weshay|ruck | rfolco, it's not unusual for 1-2 containers to fail to pull or push | 19:12 |
| weshay|ruck | we just never see it .. as we're not watching that closely | 19:12 |
| weshay|ruck | but. this is a little unusual atm | 19:12 |
| rfolco | but its all them failing from I've seen | 19:12 |
| rfolco | or just a few? | 19:12 |
| *** tesseract has quit IRC | 19:20 | |
| *** jfrancoa has quit IRC | 19:21 | |
| *** holser has joined #oooq | 19:28 | |
| *** holser has quit IRC | 19:33 | |
| *** kopecmartin is now known as kopecmartin|off | 19:35 | |
| weshay|ruck | rfolco, try current-tripleo | 19:39 |
| weshay|ruck | or tripleo-ci-testing | 19:39 |
| rfolco | I know whats going on | 19:39 |
| weshay|ruck | rfolco, wazz up | 19:39 |
| rfolco | rhel8 images are not tagged w/ _x86_64 | 19:39 |
| rfolco | centos are | 19:40 |
| weshay|ruck | ah.. booog | 19:40 |
| weshay|ruck | this is good :) | 19:40 |
| rfolco | weshay|ruck, compare https://console.registry.rdoproject.org/registry#/images/tripleomaster/rhel-binary-octavia-api | 19:41 |
| rfolco | with https://console.registry.rdoproject.org/registry#/images/tripleomaster/centos-binary-octavia-api | 19:41 |
| rfolco | see the additional tag there | 19:41 |
| weshay|ruck | rfolco, ok.. so we'll have to put in a fix to tag rhel8 | 19:41 |
| rfolco | weshay|ruck, looking at the retag code to see why | 19:43 |
| rfolco | weshay|ruck, hmmmm this rings a bell... https://github.com/openstack/tripleo-ci/blob/master/roles/build-containers/tasks/main.yaml#L193 | 19:46 |
| rfolco | oops sorry | 19:47 |
| rfolco | nm | 19:47 |
| rfolco | not this one, let me confirm my theory first | 19:47 |
| rfolco | weshay|ruck, need more eyes on this.... I don't understand why... but its skipping a task here so containers are not retagged in rhel8 | 19:56 |
| rfolco | 2019-10-09 12:54:01.286329 | TASK [build-containers : Retrieve built images] | 19:56 |
| rfolco | 2019-10-09 12:54:01.612900 | primary | skipping: Conditional result was False | 19:56 |
| *** holser has joined #oooq | 19:56 | |
| rfolco | http://logs.rdoproject.org/openstack-periodic-master/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-rhel-8-master-containers-build-push/4ac6f50/job-output.txt | 19:56 |
| rfolco | compared to centos | 19:56 |
| * weshay|ruck looks | 19:56 | |
| rfolco | centos: http://logs.rdoproject.org/openstack-periodic-master/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-centos-7-master-containers-build-push/a0798fe/job-output.txt | 19:56 |
| rfolco | the code is this one | 19:57 |
| rfolco | https://github.com/openstack/tripleo-ci/blob/master/roles/build-containers/tasks/main.yaml#L187 | 19:57 |
| rfolco | oh jeez | 19:58 |
| rfolco | push_containers | 19:58 |
| rfolco | the new variable | 19:58 |
| rfolco | weshay|ruck, found it! https://github.com/rdo-infra/review.rdoproject.org-config/blob/master/zuul.d/tripleoci.yaml#L202 | 20:01 |
| weshay|ruck | rfolco, ah.. need another line in that when | 20:01 |
| weshay|ruck | - push_containers_podman | bool | 20:02 |
| rfolco | weshay|ruck, yep, coming | 20:02 |
| rfolco | weshay|ruck, https://review.opendev.org/687665 | 20:13 |
| weshay|ruck | rfolco, see comment | 20:22 |
| rfolco | weshay|ruck, sure? I think its AND by default | 20:23 |
| rfolco | double checking | 20:23 |
| weshay|ruck | ah ur right | 20:23 |
| weshay|ruck | playbook tests that | 20:23 |
| weshay|ruck | rfolco, ya.. this works - foo_true | bool or buzz_true | bool | 20:25 |
| weshay|ruck | ya.. and works on multiple lines | 20:25 |
| weshay|ruck | k.. merging | 20:25 |
| weshay|ruck | I have to go | 20:26 |
| weshay|ruck | will have to wait for another promotion | 20:26 |
| rfolco | k | 20:26 |
| rfolco | thx | 20:26 |
| *** aakarsh|2 has joined #oooq | 20:37 | |
| *** aakarsh has quit IRC | 20:39 | |
| *** aakarsh|2 has quit IRC | 20:40 | |
| *** aakarsh has joined #oooq | 20:41 | |
| *** Goneri has quit IRC | 20:42 | |
| *** aakarsh has quit IRC | 20:46 | |
| *** rfolco has quit IRC | 21:28 | |
| *** Vorrtex has quit IRC | 21:33 | |
| *** holser has quit IRC | 21:45 | |
| *** holser has joined #oooq | 21:50 | |
| *** holser has quit IRC | 22:34 | |
| *** saneax has joined #oooq | 23:14 | |
| *** openstackstatus has joined #oooq | 23:18 | |
| *** ChanServ sets mode: +v openstackstatus | 23:18 | |
| *** tosky has quit IRC | 23:30 | |
| *** aakarsh has joined #oooq | 23:54 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!