| *** dsneddon has joined #oooq | 00:03 | |
| *** dsneddon has quit IRC | 00:11 | |
| *** chem has quit IRC | 00:12 | |
| *** dsneddon has joined #oooq | 00:14 | |
| *** panda has quit IRC | 00:17 | |
| *** panda has joined #oooq | 00:19 | |
| *** dsneddon has quit IRC | 00:21 | |
| *** dsneddon has joined #oooq | 00:39 | |
| *** dsneddon has quit IRC | 00:44 | |
| *** jschlueter has quit IRC | 00:55 | |
| *** dtantsur|afk has quit IRC | 00:55 | |
| *** ChanServ has quit IRC | 00:55 | |
| *** rlandy|ruck|bbl has quit IRC | 00:55 | |
| *** fmount has quit IRC | 00:55 | |
| *** jbadiapa has quit IRC | 00:56 | |
| *** hamzy_ has quit IRC | 00:56 | |
| *** radez has quit IRC | 00:56 | |
| *** weshay has quit IRC | 00:56 | |
| *** rfolco has quit IRC | 00:56 | |
| *** apetrich has quit IRC | 00:56 | |
| *** yolanda has quit IRC | 00:56 | |
| *** matbu has quit IRC | 00:56 | |
| *** rnoriega- has quit IRC | 00:56 | |
| *** panda has quit IRC | 00:56 | |
| *** mjturek has quit IRC | 00:56 | |
| *** sshnaidm|afk has quit IRC | 00:56 | |
| *** jjoyce has quit IRC | 00:56 | |
| *** amoralej|off has quit IRC | 00:56 | |
| *** jpena|off has quit IRC | 00:56 | |
| *** rodrigods has quit IRC | 00:56 | |
| *** aakarsh|2 has quit IRC | 00:56 | |
| *** Tengu has quit IRC | 00:56 | |
| *** irclogbot_0 has quit IRC | 00:56 | |
| *** jdennis has quit IRC | 00:56 | |
| *** jrist has quit IRC | 00:56 | |
| *** gchamoul has quit IRC | 00:56 | |
| *** kopecmartin|off has quit IRC | 00:56 | |
| *** zbr has quit IRC | 00:56 | |
| *** d0ugal has quit IRC | 00:56 | |
| *** dtrainor has quit IRC | 00:56 | |
| *** redrobot has quit IRC | 00:56 | |
| *** dmellado has quit IRC | 00:56 | |
| *** gouthamr has quit IRC | 00:56 | |
| *** arxcruz has quit IRC | 00:56 | |
| *** arxcruz has joined #oooq | 01:14 | |
| *** gouthamr has joined #oooq | 01:14 | |
| *** dmellado has joined #oooq | 01:14 | |
| *** redrobot has joined #oooq | 01:14 | |
| *** dtrainor has joined #oooq | 01:14 | |
| *** weshay has joined #oooq | 01:14 | |
| *** radez has joined #oooq | 01:14 | |
| *** hamzy_ has joined #oooq | 01:14 | |
| *** jbadiapa has joined #oooq | 01:14 | |
| *** mjturek has joined #oooq | 01:14 | |
| *** panda has joined #oooq | 01:14 | |
| *** jschlueter has joined #oooq | 01:14 | |
| *** rnoriega- has joined #oooq | 01:14 | |
| *** matbu has joined #oooq | 01:14 | |
| *** yolanda has joined #oooq | 01:14 | |
| *** apetrich has joined #oooq | 01:14 | |
| *** rfolco has joined #oooq | 01:14 | |
| *** dtantsur|afk has joined #oooq | 01:14 | |
| *** fmount has joined #oooq | 01:14 | |
| *** rlandy|ruck|bbl has joined #oooq | 01:14 | |
| *** d0ugal has joined #oooq | 01:14 | |
| *** zbr has joined #oooq | 01:14 | |
| *** ChanServ has joined #oooq | 01:14 | |
| *** card.freenode.net sets mode: +oo arxcruz ChanServ | 01:14 | |
| *** sshnaidm|afk has joined #oooq | 01:14 | |
| *** jjoyce has joined #oooq | 01:14 | |
| *** amoralej|off has joined #oooq | 01:14 | |
| *** jpena|off has joined #oooq | 01:14 | |
| *** rodrigods has joined #oooq | 01:14 | |
| *** jdennis has joined #oooq | 01:14 | |
| *** gchamoul has joined #oooq | 01:14 | |
| *** jrist has joined #oooq | 01:14 | |
| *** kopecmartin|off has joined #oooq | 01:14 | |
| *** aakarsh has joined #oooq | 01:21 | |
| *** irclogbot_0 has joined #oooq | 01:23 | |
| *** dsneddon has joined #oooq | 01:41 | |
| *** dsneddon has quit IRC | 01:51 | |
| *** apetrich has quit IRC | 02:10 | |
| *** rlandy|ruck|bbl is now known as rlandy|ruck | 02:17 | |
| *** dsneddon has joined #oooq | 02:21 | |
| *** dsneddon has quit IRC | 02:32 | |
| *** hamzy has joined #oooq | 02:36 | |
| *** hamzy_ has quit IRC | 02:36 | |
| *** dsneddon has joined #oooq | 03:04 | |
| *** dsneddon has quit IRC | 03:27 | |
| *** gkadam has joined #oooq | 03:37 | |
| *** gkadam is now known as gkadam-afk | 03:38 | |
| *** rlandy|ruck has quit IRC | 03:46 | |
| *** dsneddon has joined #oooq | 03:47 | |
| *** dsneddon has quit IRC | 03:51 | |
| *** surpatil has joined #oooq | 03:54 | |
| *** dsneddon has joined #oooq | 04:10 | |
| *** dsneddon has quit IRC | 04:14 | |
| *** udesale has joined #oooq | 04:16 | |
| *** gkadam has joined #oooq | 04:17 | |
| *** surpatil has quit IRC | 04:18 | |
| *** surpatil has joined #oooq | 04:18 | |
| *** gkadam-afk has quit IRC | 04:19 | |
| *** SurajPatil has joined #oooq | 04:32 | |
| *** surpatil has quit IRC | 04:32 | |
| *** Tengu has joined #oooq | 04:38 | |
| *** dsneddon has joined #oooq | 04:42 | |
| *** ksambor has joined #oooq | 04:48 | |
| *** bhagyashris has joined #oooq | 05:00 | |
| *** dsneddon has quit IRC | 05:06 | |
| *** jbadiapa has quit IRC | 05:15 | |
| *** ratailor has joined #oooq | 05:18 | |
| *** fmount has quit IRC | 05:20 | |
| *** jaosorior has joined #oooq | 05:29 | |
| *** skramaja has joined #oooq | 05:33 | |
| *** dsneddon has joined #oooq | 05:41 | |
| *** udesale has quit IRC | 05:44 | |
| *** udesale has joined #oooq | 05:45 | |
| *** rfolco has quit IRC | 05:49 | |
| *** rfolco has joined #oooq | 05:49 | |
| *** dsneddon has quit IRC | 05:51 | |
| *** SurajPatil has quit IRC | 05:57 | |
| *** jbadiapa has joined #oooq | 06:19 | |
| *** dsneddon has joined #oooq | 06:19 | |
| *** kopecmartin|off is now known as kopecmartin | 06:26 | |
| *** SurajPatil has joined #oooq | 06:29 | |
| *** dsneddon has quit IRC | 06:32 | |
| *** fmount has joined #oooq | 06:36 | |
| *** fmount has quit IRC | 06:37 | |
| *** fmount has joined #oooq | 06:40 | |
| *** saneax has joined #oooq | 06:42 | |
| *** SurajPatil has quit IRC | 07:01 | |
| *** udesale has quit IRC | 07:02 | |
| *** udesale has joined #oooq | 07:02 | |
| *** udesale has quit IRC | 07:02 | |
| *** udesale has joined #oooq | 07:03 | |
| *** dsneddon has joined #oooq | 07:08 | |
| *** udesale has quit IRC | 07:11 | |
| *** udesale has joined #oooq | 07:11 | |
| *** dsneddon has quit IRC | 07:13 | |
| *** SurajPatil has joined #oooq | 07:15 | |
| *** apetrich has joined #oooq | 07:31 | |
| *** jpena|off is now known as jpena | 07:33 | |
| *** amoralej|off is now known as amoralej | 07:38 | |
| *** dsneddon has joined #oooq | 07:39 | |
| *** dsneddon has quit IRC | 08:02 | |
| *** SurajPatil has quit IRC | 08:25 | |
| *** dtantsur|afk is now known as dtantsur | 08:28 | |
| *** dsneddon has joined #oooq | 08:29 | |
| *** dsneddon has quit IRC | 08:43 | |
| *** chem has joined #oooq | 09:00 | |
| *** dsneddon has joined #oooq | 09:17 | |
| *** dsneddon has quit IRC | 09:33 | |
| *** SurajPatil has joined #oooq | 09:49 | |
| *** SurajPatil has quit IRC | 10:00 | |
| *** dsneddon has joined #oooq | 10:00 | |
| *** dsneddon has quit IRC | 10:05 | |
| zbr | does anyone knows when we will have first set of rhel containers pushed? | 10:11 |
|---|---|---|
| *** bhagyashris has quit IRC | 10:12 | |
| *** dsneddon has joined #oooq | 10:41 | |
| panda | zbr: we had a promotion in rhel8 already | 10:43 |
| zbr | ok, that means that probably the errors i seen was from a job that run before the promotion. | 10:45 |
| *** sshnaidm|afk is now known as sshnaidm | 10:53 | |
| *** dsneddon has quit IRC | 11:00 | |
| *** jpena is now known as jpena|lunch | 11:06 | |
| *** tesseract has joined #oooq | 11:07 | |
| *** amoralej is now known as amoralej|lunch | 11:09 | |
| panda | rfolco: don't touch anythin until we talk | 11:12 |
| panda | rfolco: I'm juggling commits while standing on a rope | 11:12 |
| *** udesale has quit IRC | 11:14 | |
| *** dsneddon has joined #oooq | 11:36 | |
| *** dsneddon has quit IRC | 11:52 | |
| rfolco | panda, ack, let me know when you are ready to talk | 12:04 |
| *** dsneddon has joined #oooq | 12:05 | |
| *** dsneddon has quit IRC | 12:11 | |
| *** jpena|lunch is now known as jpena | 12:14 | |
| *** amoralej|lunch is now known as amoralej | 12:16 | |
| *** rlandy has joined #oooq | 12:17 | |
| *** rlandy is now known as rlandy|ruck | 12:17 | |
| *** ratailor has quit IRC | 12:21 | |
| rlandy|ruck | oh gosh - no raukadah and no yatin :( | 12:23 |
| rlandy|ruck | amoralej: hi - I keep rerunning https://ci.centos.org/job/weirdo-stein-promote-puppet-openstack-scenario004/ in the hope that it will turn green. Can you tell from the limited log output if the failure is legit? | 12:24 |
| amoralej | let me check rlandy|ruck | 12:24 |
| amoralej | rlandy|ruck, mmm is timing out | 12:28 |
| amoralej | let me lock the instance for current run | 12:28 |
| panda | rfolco: ready | 12:30 |
| rfolco | panda, join my blue | 12:31 |
| panda | rfolco: joining your bj so we are ready for scrum | 12:31 |
| rlandy|ruck | amoralej: k - thanks - let me know if you need something from our side | 12:31 |
| *** gkadam has quit IRC | 12:32 | |
| amoralej | sure | 12:33 |
| rlandy|ruck | arxcruz; hi - need to borrow your tempest brain | 12:41 |
| arxcruz | rlandy|ruck: hi | 12:42 |
| rlandy|ruck | arxcruz: hey - so queens tempest tests were failing ... getting lp | 12:42 |
| rlandy|ruck | https://trello.com/c/ZvV5ul81/1029-cixlp1836046tripleociproa-tempestscenariotestnetworkbasicopstestnetworkbasicops-failing-on-queens | 12:42 |
| rlandy|ruck | arxcruz: so chandan made a fix in fs001 | 12:42 |
| arxcruz | yup | 12:43 |
| arxcruz | i saw | 12:43 |
| rlandy|ruck | which works but then the failure showed up elsewhere | 12:43 |
| rlandy|ruck | so we had a conversation about adding the fix to the validate_tempest role | 12:43 |
| rlandy|ruck | which I think is correct | 12:43 |
| rlandy|ruck | so I went to the role to change that var | 12:43 |
| rlandy|ruck | but it's overridden in tempest.conf | 12:43 |
| *** dsneddon has joined #oooq | 12:43 | |
| rlandy|ruck | what's the right thing to do here? | 12:43 |
| arxcruz | what you mean it's overridden in tempest.conf ? | 12:44 |
| rlandy|ruck | keeping queens promoting by adding a review with the same change to fs035 atm | 12:44 |
| rlandy|ruck | arxcruz, https://github.com/openstack/tripleo-quickstart-extras/blob/master/roles/validate-tempest/defaults/main.yml#L71 | 12:44 |
| arxcruz | rlandy|ruck: should be added here to change everywhere https://github.com/openstack/tripleo-quickstart-extras/blob/master/roles/validate-tempest/templates/configure-tempest.sh.j2#L199 | 12:46 |
| rlandy|ruck | arxcruz: chandan set that var in fs001 | 12:46 |
| rlandy|ruck | arxcruz: k - putting up a patch for your revuew | 12:46 |
| rlandy|ruck | review - thanks | 12:46 |
| rlandy|ruck | {% for key, value in tempest_extra_config.items() %} | 12:46 |
| rlandy|ruck | arxcruz: so iiuc, the setting does not have to remain in tempest_extra_config - I can just add it directly | 12:48 |
| panda | zbr: you available ? | 12:50 |
| zbr | panda: f u always | 12:50 |
| arxcruz | rlandy|ruck: yes, if you put there, it will never be overridden | 12:50 |
| zbr | panda: https://bluejeans.com/2655417928 ? | 12:50 |
| arxcruz | rlandy|ruck: add a if queens, i'm still not convinced that this will fix the issue... | 12:50 |
| panda | zbr: join rfolco's | 12:52 |
| rlandy|ruck | arxcruz, https://review.opendev.org/678008 Add compute_feature_enabled.config_drive to configure-tempest.sh | 12:52 |
| arxcruz | rlandy|ruck: so, this will add the config also to other releases, is that fine? i don't see a problem | 12:54 |
| rlandy|ruck | arxcruz: well chandan's fix is not release-specific | 12:54 |
| arxcruz | ok | 12:54 |
| rlandy|ruck | arxcruz: we can spin master tempest fs020 on the change | 12:54 |
| rlandy|ruck | and see what we kill | 12:55 |
| arxcruz | ok | 12:55 |
| rfolco | ping weshay scrum | 13:00 |
| chem | rlandy|ruck: hey, just wondering if the broken package dependency in stein for ceph-ansible-4.0.0-0.rc13.1.el7.noarch is a known issue ? | 13:03 |
| *** dsneddon has quit IRC | 13:04 | |
| rlandy|ruck | chem: not that we came across - stein promoted yesterday | 13:04 |
| rlandy|ruck | let's see what version we pick up | 13:04 |
| rlandy|ruck | yep - stein promotion is less than a day out - checking logs | 13:05 |
| rlandy|ruck | chem: Aug 21 22:27:21 Installed: ceph-ansible-4.0.0-0.rc13.1.el7.noarch - we did pick that up | 13:08 |
| chem | rlandy|ruck: I'm looking harder at the exact error message | 13:08 |
| rlandy|ruck | chem: can you link to where you are seeing the error | 13:08 |
| rlandy|ruck | maybe we are not using it - even if we pick it up | 13:08 |
| chem | rlandy|ruck: it seems to have trouble for getting ansible >=2.8 | 13:08 |
| chem | rlandy|ruck: https://storage.bhs1.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/logs_86/677786/1/check/tripleo-ci-centos-7-containerized-undercloud-upgrades/316fa10/logs/undercloud/home/zuul/undercloud_upgrade.log.txt.gz | 13:08 |
| chem | rlandy|ruck: look for the "Update all packages" task | 13:09 |
| chem | rlandy|ruck: well it looks like it cannot find ansible 2.8 (but those dnf error are hard to read really :)) | 13:10 |
| rlandy|ruck | chem: http://zuul.openstack.org/builds?job_name=tripleo-ci-centos-7-containerized-undercloud-upgrades that job has been failing forever - I actually chatted with weshay about it | 13:10 |
| rlandy|ruck | but it should have been picked up elsewhere | 13:10 |
| rlandy|ruck | we upgraded to ansible 2.8 about a week ago | 13:11 |
| rlandy|ruck | chem: https://github.com/openstack/tripleo-quickstart/commit/801c660196f9cca147b36c9c9991608345ef2a1d | 13:11 |
| rlandy|ruck | committed 13 days ago | 13:12 |
| chem | rlandy|ruck: something may be wrong with the repos ... oki I give it a deep look | 13:13 |
| rlandy|ruck | chem: ok - we have some similar jobs running if you want me to compare those | 13:14 |
| chem | rlandy|ruck: I'm using the job in the zuul history currently, let's see if I find something. | 13:15 |
| rlandy|ruck | chem: http://cistatus.tripleo.org/ (look for check jobs link) may give you more history | 13:16 |
| chem | rlandy|ruck: thanks | 13:17 |
| rlandy|ruck | weird we don't pick that up anywhere | 13:17 |
| panda | zbr: it's not just the button | 13:17 |
| panda | zbr: abandoning is not allowed even via API | 13:17 |
| zbr | panda: i know. | 13:17 |
| rlandy|ruck | I know if osp we set the version of ceph-ansible | 13:17 |
| *** Goneri has joined #oooq | 13:18 | |
| rlandy|ruck | chem: (to stand on my soapbox) that is why I hate non-voting jobs. we ruck/rover/job-watchers miss stuff because we expect the job to fail | 13:21 |
| chem | rlandy|ruck: so it's only non-voting on master (for which I have filled a lp for later review https://bugs.launchpad.net/tripleo/+bug/1841059) but it's voting in stein :) | 13:23 |
| openstack | Launchpad bug 1841059 in tripleo "undercloud-upgrade jobs on master fails on validation." [High,Triaged] | 13:23 |
| chem | rlandy|ruck: so now I have a placeholder for master, and focus on stein issue: why we cannot pull in ansible-2.8 | 13:24 |
| rlandy|ruck | ok | 13:25 |
| rlandy|ruck | https://github.com/openstack/tripleo-quickstart/commit/7d149d750053498002024af2a5602a4e2df0cd2b maybe this has something to do with it | 13:27 |
| rlandy|ruck | no exclude on rocky? | 13:27 |
| chem | rlandy|ruck: I have to catchup on those files, cannot tell | 13:30 |
| *** skramaja has quit IRC | 13:34 | |
| rlandy|ruck | arxcruz: spinning master tempest with the conf change https://review.rdoproject.org/r/21683 Test f20 master and tempest | 13:35 |
| rlandy|ruck | will see what shakes out | 13:35 |
| arxcruz | rlandy|ruck: ack | 13:36 |
| weshay | zbr, sshnaidm can we figure out why fluent is still used here 2019-08-22 09:23:17 | tripleo_common.image.exception.ImageNotFoundException: Not found image: docker://trunk.registry.rdoproject.org/tripleomaster/rhel-binary-fluentd:c02f4c8633d95b9190fadfdb5b6656c5d451f160_c19a0cf9 | 13:36 |
| *** dsneddon has joined #oooq | 13:37 | |
| rfolco | panda, are you submitting a new ps to provisioning ? | 13:39 |
| rfolco | panda, will wait your signal to resume the work there | 13:39 |
| weshay | zbr can you please rebase all those patches and try it one more time | 13:41 |
| rfolco | panda, send me a smoke signal, anything | 13:41 |
| rlandy|ruck | sshnaidm++ sova is back in action :) | 13:41 |
| sshnaidm | weshay, where? | 13:41 |
| zbr | or sova strikes again? ;) | 13:41 |
| sshnaidm | rlandy|ruck, np | 13:41 |
| weshay | sshnaidm zbr https://review.rdoproject.org/zuul/status w/ 21787 | 13:42 |
| weshay | zbr go rebase please | 13:42 |
| panda | rfolco: I don't smoke | 13:44 |
| sshnaidm | weshay, well, rhel_containers: should be "true" here: http://logs.rdoproject.org/60/21860/6/check/periodic-tripleo-ci-rhel-8-scenario002-standalone-master/5f03e02/logs/undercloud/home/zuul/containers-prepare-parameters.yaml.txt.gz | 13:44 |
| sshnaidm | zbr, ^^ | 13:44 |
| sshnaidm | that's the whole point.. | 13:44 |
| weshay | yup | 13:45 |
| weshay | thanks | 13:45 |
| *** altlogbot_0 has joined #oooq | 13:46 | |
| *** altlogbot_0 has quit IRC | 13:48 | |
| *** altlogbot_2 has joined #oooq | 13:51 | |
| panda | rfolco: if you're waiting for me, do you want to do some pair programming ? I can't move on without this part being done. Maybe you can handle the tags for now. | 13:57 |
| *** dsneddon has quit IRC | 14:00 | |
| weshay | sshnaidm zbr shouldn't rhel_containers be automatically flipped to true via https://opendev.org/openstack/tripleo-quickstart-extras/src/branch/master/roles/extras-common/defaults/main.yml#L40 | 14:03 |
| sshnaidm | weshay, yeah, but it's not handled in standalone role | 14:04 |
| weshay | sshnaidm ah k | 14:04 |
| sshnaidm | weshay, need to add setting this parameter in sa role | 14:04 |
| weshay | zbr so do you know where to fix that? | 14:04 |
| rfolco | panda, ok will work on tags and manifests | 14:06 |
| rfolco | panda, let me know when you finish this part, so I can resume | 14:07 |
| sshnaidm | zbr, probably here https://github.com/openstack/tripleo-quickstart-extras/blob/master/roles/standalone/templates/standalone_config.yaml.j2 | 14:07 |
| *** dsneddon has joined #oooq | 14:08 | |
| amoralej | rlandy|ruck, i see the deployment part went fine and on time, the problem is that tempest is taking very long, i'm checking if it's running more jobs that expected or so | 14:09 |
| rlandy|ruck | amoralej: we can increase the job time | 14:10 |
| amoralej | 2hr should be more that enough | 14:11 |
| rlandy|ruck | lol - ok | 14:11 |
| amoralej | those jobs are used to gate puppet changes | 14:11 |
| amoralej | so we want to keep it short | 14:11 |
| rlandy|ruck | ack | 14:11 |
| rlandy|ruck | if we rerun them enough they sometimes pass, but that's not a great way to do ci | 14:12 |
| *** dsneddon has quit IRC | 14:13 | |
| *** dsneddon has joined #oooq | 14:15 | |
| sshnaidm | rlandy|ruck, for "private" network in ovb port_security_enabled should be true, right? | 14:17 |
| rlandy|ruck | sshnaidm: hmm - haven't work on this stiff for a while - last time I did, the only edits we made to port security was in the ovb templates | 14:19 |
| rlandy|ruck | ie: if you look at multi-nic etc. | 14:19 |
| sshnaidm | rlandy|ruck, I see in current tenant port_security_enabled | True | 14:19 |
| rlandy|ruck | sshnaidm: right - if you have no ipmi etc. you will know, you have that security wrong | 14:20 |
| sshnaidm | rlandy|ruck, ack | 14:20 |
| rlandy|ruck | for the private network, I'd have to search back | 14:20 |
| rlandy|ruck | if you look at the template edits that is all I ever did | 14:20 |
| rlandy|ruck | but basically, it was trial and error on the rest | 14:21 |
| rlandy|ruck | if you get an error, ping me, I'll look | 14:21 |
| weshay | zbr I guess triple-common would not detect which distro it's called on https://review.opendev.org/#/c/676497/ | 14:40 |
| weshay | and it's safer to set a working global default | 14:41 |
| *** jaosorior has quit IRC | 14:44 | |
| zbr | ... and less work for us to patch broken defaults | 14:45 |
| *** Vorrtex has joined #oooq | 14:55 | |
| *** kopecmartin is now known as kopecmartin|off | 14:59 | |
| weshay | rlandy|ruck can you please take 10min to review that patches in https://tree.taiga.io/project/tripleo-ci-board/task/1182?kanban-status=1447276 | 15:00 |
| rlandy|ruck | ack | 15:01 |
| *** jbadiapa has quit IRC | 15:02 | |
| *** jbadiapa has joined #oooq | 15:03 | |
| rlandy|ruck | are they all merge ready? | 15:03 |
| rlandy|ruck | zbr: ^^? | 15:03 |
| rlandy|ruck | weshay: ? | 15:03 |
| rlandy|ruck | Left to merge (in order) - nvm - I guess yes | 15:03 |
| zbr | weshay: rlandy|ruck: that should enable rhel_containers for standalone, at least locally it did the replacement well. https://review.opendev.org/#/c/678038/1/roles/standalone/tasks/main.yml | 15:13 |
| zbr | we should find out in ~20mins | 15:13 |
| weshay | zbr k.. /me watching in zuul.. nothing running there yet | 15:13 |
| rlandy|ruck | zbr: k - on it now | 15:14 |
| zbr | i am watching the zuulflix live streaming... | 15:14 |
| weshay | see it now.. was looking at the wrong change | 15:16 |
| rlandy|ruck | rdo-tox-moleculeFAILURE in 4m 17s (non-voting) | 15:18 |
| rlandy|ruck | registry-login-integrationRETRY_LIMIT in 3m 32s (non-voting) | 15:18 |
| weshay | zbr hrm.. failed | 15:18 |
| rlandy|ruck | zbr: ^^ https://review.opendev.org/#/c/673481/ fails those two tests | 15:18 |
| rlandy|ruck | first one on the merge list | 15:19 |
| weshay | scen01 | 15:19 |
| rlandy|ruck | egistry-login-integration I understand - might be missing secret | 15:19 |
| rlandy|ruck | no log on the tox failure | 15:19 |
| rlandy|ruck | hard to know if it's legit | 15:19 |
| rlandy|ruck | panda: ^^ | 15:20 |
| *** ksambor is now known as ksambor|away | 15:23 | |
| zbr | weshay: that is because we enabled the job before we had the configuration on, and we ended up with some debates: https://review.opendev.org/#/c/673554/ | 15:24 |
| zbr | mainly we need to engaje with cloudnull and find some common ground or disable the rdo job. | 15:24 |
| zbr | because the rdo jobis trying to execute an environment that does not exist on tox.ini file | 15:25 |
| zbr | to to rephrase: the rdo-tox-molecule was broken before panda made any changes, safe to ignore for the moment. | 15:26 |
| weshay | hrm.. zbr scen01 failed on http://logs.rdoproject.org/60/21860/8/check/tripleo-ci-rhel-8-scenario001-standalone-rdo/53bc843/logs/undercloud/home/zuul/standalone_deploy.log.txt.gz | 15:28 |
| weshay | have not see that before | 15:28 |
| zbr | mainly openstack/ansible went into using molecule with delegated (no docker) on several roles and even without tox. it is still usable, but a total different way of calling it, one that does not allow you to test locally (not unless you take care of creating a testing inventory yourself) | 15:28 |
| zbr | this being one of the reasons I joined the new SIG, as I did not like the directions. | 15:29 |
| zbr | for me it was key to be able to test roles on BOTH: zuul and personal machines, not only one of them. | 15:29 |
| weshay | rlandy|ruck have you seen | 15:31 |
| weshay | 2019-08-22 11:17:57 | /usr/lib64/python3.6/copy.py:289: ResourceWarning: unclosed <ssl.SSLSocket fd=18, family=AddressFamily.AF_INET, type=2049, proto=6, laddr=('192.168.100.154', 57780), raddr=('38.145.32.81', 443)> | 15:31 |
| weshay | in 3rd party? | 15:31 |
| weshay | sshnaidm ^ | 15:31 |
| rlandy|ruck | not really | 15:31 |
| weshay | context: http://logs.rdoproject.org/60/21860/8/check/tripleo-ci-rhel-8-scenario002-standalone-rdo/8a5dd0b/logs/undercloud/home/zuul/standalone_deploy.log.txt.gz | 15:31 |
| sshnaidm | weshay, I saw in ovb something similar, but it was just noise | 15:32 |
| panda | rlandy|ruck: sorry was talking with mjturek | 15:32 |
| *** dsneddon has quit IRC | 15:32 | |
| zbr | weshay: sounds very similar with a problem alex tried to fix. there is a LP. | 15:32 |
| rlandy|ruck | pand: nvm - found tests run in https://review.rdoproject.org/r/#/c/21593/ | 15:32 |
| panda | rlandy|ruck: so rdo-tox-molecule is broken, but it's not critical | 15:33 |
| panda | rlandy|ruck: and yes the integration test that is failing is actually in queue to be removed, because it doesn't fetch the change | 15:33 |
| rlandy|ruck | panda: will you be around to help debug if we merge this today? | 15:34 |
| rlandy|ruck | I would prefer to do the merge when we have people to debug fallout | 15:34 |
| panda | rlandy|ruck: no, I need to run in 30 minutes :( | 15:34 |
| panda | rlandy|ruck: but we can merge everything except for the actuall switch | 15:35 |
| rlandy|ruck | panda - where is the actual switch? | 15:35 |
| panda | rlandy|ruck: until https://review.rdoproject.org/r/21547 it's all fun and games | 15:35 |
| rlandy|ruck | famous last words | 15:35 |
| rlandy|ruck | panda: do you have +2 rights on these repos? | 15:36 |
| rlandy|ruck | panda: if so, I can +2 the changes and you can w+ them in your morning hours tomorrow | 15:36 |
| panda | rlandy|ruck: I have, the only thing that I can't merge alone is https://review.opendev.org/673481 | 15:37 |
| sshnaidm | weshay, it's the same error: http://logs.rdoproject.org/60/21860/8/check/tripleo-ci-rhel-8-scenario001-standalone-rdo/53bc843/logs/undercloud/var/log/heat-launcher/undercloud_deploy-lg7dk5fo/heat.log.txt.gz | 15:37 |
| sshnaidm | Property ContainerFluentdImage not assigned | 15:37 |
| rlandy|ruck | panda: k - let's merge that one now | 15:38 |
| weshay | sshnaidm hrm.. /me looks in a sec | 15:38 |
| rlandy|ruck | no expected impact, right? | 15:38 |
| rlandy|ruck | panda: ^^ | 15:38 |
| weshay | arxcruz .. need ur eyes on fs21 master when you have a sec... the tempest reporting html looks busted and we have a bunch of working tests that need to be unskipped | 15:38 |
| weshay | 2019-08-21 16:16:23 | Ran: 348 tests in 1274.0000 sec. | 15:38 |
| weshay | 2019-08-21 16:16:23 | - Passed: 313 | 15:38 |
| weshay | 2019-08-21 16:16:23 | - Skipped: 21 | 15:38 |
| weshay | 2019-08-21 16:16:23 | - Expected Fail: 0 | 15:38 |
| weshay | 2019-08-21 16:16:23 | - Unexpected Success: 0 | 15:38 |
| weshay | 2019-08-21 16:16:23 | - Failed: 14 | 15:38 |
| weshay | http://logs.rdoproject.org/openstack-periodic-master/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-7-ovb-1ctlr_2comp-featureset021-master/103afe1/logs/undercloud/home/zuul/tempest.log.txt.gz | 15:39 |
| weshay | rlandy|ruck fyi ^ | 15:39 |
| arxcruz | weshay: ack, i have a preliminary script already that detects the passing ones, i'll work on the skip list | 15:39 |
| weshay | arxcruz rock on dude | 15:40 |
| weshay | sshnaidm so we have to go update service def's for rhel? | 15:40 |
| weshay | 2019-08-22 11:18:18.979 5545 ERROR heat.engine.resource ValueError: Property ContainerFluentdImage not assigned | 15:41 |
| sshnaidm | weshay, probably | 15:41 |
| weshay | ugh | 15:41 |
| weshay | sshnaidm would we have the same issue on centos8 in your opinion? | 15:41 |
| weshay | I think we would | 15:41 |
| sshnaidm | weshay, why? it's related to lack of needed packages downstream | 15:42 |
| sshnaidm | weshay, centos shouldn't be affected if we have same set of packages as for 7 | 15:42 |
| weshay | sshnaidm not sure we will right.. we'll have the same packages we have for rhel8 in theory | 15:42 |
| sshnaidm | weshay, for rhel we have rhui only | 15:43 |
| weshay | aye | 15:43 |
| weshay | w/ osp deps | 15:43 |
| sshnaidm | weshay, we have fluentd for centos7, why not to have it for centos8? | 15:43 |
| weshay | sshnaidm ya.. the answers are not always so straight forward to me | 15:44 |
| sshnaidm | weshay, I'm pretty sure it's rhel thing than 7/8 | 15:44 |
| rlandy|ruck | panda; ok - here you go https://review.opendev.org/#/c/673481/ - please don't break anything | 15:44 |
| rlandy|ruck | actually - ... | 15:45 |
| rlandy|ruck | weshay: will be get yelled at for w+ in openstack/ansible-role-container-registry? | 15:45 |
| panda | rlandy|ruck: actually ? | 15:45 |
| sshnaidm | weshay, actually #rdo folks should answer this better, what they plan to have in 8 | 15:45 |
| weshay | rlandy|ruck it's a good question... it's not so clear.. I could make the case it's a ci role | 15:46 |
| weshay | sshnaidm agree | 15:46 |
| rlandy|ruck | weshay: well, I just did | 15:46 |
| weshay | sshnaidm its ceilometer req right? | 15:46 |
| rlandy|ruck | that may cause a disaster - but panda says not | 15:46 |
| sshnaidm | weshay, fluentd? | 15:47 |
| weshay | oh.. sorry opstools | 15:47 |
| panda | rlandy|ruck: consider these: 1) we have a lot of checks there to prevent disaster. 2) I did not touch the cotnainer role itself, only the login part 3) the login part maintains backwards compatibility, as tripleo roles currently use the docker-login.yaml file that I left under indication by Alex. | 15:49 |
| rlandy|ruck | panda: I w'ed it anyways | 15:49 |
| rlandy|ruck | will see how it goes | 15:49 |
| weshay | sshnaidm do we know which packages were missing on the fluent containers? | 15:50 |
| sshnaidm | weshay, I've heard we'll use rsyslog instead of fluentd | 15:50 |
| weshay | hrm.. ok | 15:50 |
| sshnaidm | weshay, no idea tbh | 15:50 |
| weshay | sshnaidm ok.. I need to poke your brain a little more | 15:50 |
| panda | rlandy|ruck: if we make a disaster, we can actually expose the bug and update the tests there to consistently not have the same disaster in the future | 15:51 |
| weshay | sshnaidm can you jump on my blue for a sec.. zbr you should jump in and listen | 15:51 |
| rlandy|ruck | gates job started | 15:51 |
| zbr | sure | 15:53 |
| *** Vorrtex has quit IRC | 15:53 | |
| *** Vorrtex has joined #oooq | 15:54 | |
| rfolco | panda, I'll propose a dnm follow-up patch just to test the multi-arch workflow for x86, including post-task retag/push | 15:56 |
| *** jpena is now known as jpena|off | 15:59 | |
| *** sshnaidm is now known as sshnaidm|afk | 16:01 | |
| *** gkadam has joined #oooq | 16:02 | |
| *** dsneddon has joined #oooq | 16:02 | |
| weshay | sshnaidm|afk zbr so I need to go back and find the container builds that we're failing fluent | 16:03 |
| weshay | fluentd, sensu, skydive | 16:03 |
| rlandy|ruck | panda: sorry - I w+ed that where I should not have | 16:06 |
| rlandy|ruck | alex has an objection | 16:06 |
| rlandy|ruck | and it's in the gate queue | 16:06 |
| *** gkadam has quit IRC | 16:06 | |
| panda | .... | 16:07 |
| panda | .................... | 16:07 |
| panda | ........................................ | 16:07 |
| rlandy|ruck | panda; if we update the patch, it will stop it | 16:08 |
| rlandy|ruck | up to you | 16:08 |
| *** dsneddon has quit IRC | 16:09 | |
| panda | why did emilien ask to expand the role then ? | 16:09 |
| rlandy|ruck | no idea | 16:10 |
| rlandy|ruck | anyways - it's going to merge :( | 16:10 |
| panda | and then I asked "this doesn't have podman support" and Emilien "let's add it" | 16:10 |
| *** altlogbot_2 has quit IRC | 16:14 | |
| *** altlogbot_3 has joined #oooq | 16:14 | |
| raukadah | \o/ | 16:21 |
| raukadah | rlandy|ruck: hello | 16:22 |
| raukadah | rlandy|ruck: How is CI? | 16:22 |
| rlandy|ruck | raukadah: not bad opened a new bug for the tempest.api.volume.* failures | 16:23 |
| raukadah | rlandy|ruck: ah cool, will look into that tomorrow | 16:23 |
| rlandy|ruck | raukadah: abishop is on it | 16:23 |
| rlandy|ruck | raukadah: all notes are on etherpad | 16:24 |
| rlandy|ruck | also upgrades issue | 16:24 |
| rlandy|ruck | but that test is non-voting | 16:24 |
| weshay | raukadah go away | 16:25 |
| weshay | :) | 16:25 |
| raukadah | weshay: aye sir | 16:25 |
| *** jbadiapa has quit IRC | 16:26 | |
| weshay | rfolco does the rhel8 container build workflow fail on the first failed container build? or Does it try to build all then report what failed / passed | 16:26 |
| rlandy|ruck | raukadah missed ruck/rover so much - he had to check in :) | 16:26 |
| panda | rlandy|ruck: change blocked | 16:27 |
| panda | rlandy|ruck: thanks anyway. | 16:27 |
| rlandy|ruck | panda: k - | 16:27 |
| panda | rlandy|ruck: I'll need to run now. | 16:27 |
| * panda is deeply disappointed | 16:27 | |
| rlandy|ruck | panda: when you ready, I'll look at it again | 16:27 |
| rlandy|ruck | better now than later | 16:27 |
| rfolco | weshay, I think it keeps running if a failed container, but need to revisit this | 16:27 |
| panda | rlandy|ruck: well, if alex is not happy, the only alternative is to move everything to rdo-job, and I mean duplicating the entire role and making the changes there | 16:28 |
| rfolco | weshay, I remember there was a patch to fix return code from openstack container build cmd | 16:28 |
| panda | rfolco: I uploaded the last version of the staging setup, but it has problems | 16:28 |
| panda | rfolco: and I have no time now anymmore | 16:28 |
| rfolco | panda, ok will see there | 16:28 |
| rlandy|ruck | panda: open source is very open to objections sometimes | 16:28 |
| rfolco | panda, ok will try to see what is wrong and fix | 16:29 |
| panda | rlandy|ruck: yes, but objection at this late stage and in contradiction with was was asked before, are quite distrupting. | 16:30 |
| *** dsneddon has joined #oooq | 16:30 | |
| panda | if they didn't ask, this role would have lived quetly in rdo-jobs from the start. | 16:31 |
| panda | I'm out | 16:31 |
| panda | solicitor here I come ... | 16:31 |
| *** panda is now known as panda|out | 16:31 | |
| rlandy|ruck | panda|out: sorry | 16:32 |
| *** dsneddon has quit IRC | 16:39 | |
| *** dtantsur is now known as dtantsur|afk | 16:39 | |
| *** dsneddon has joined #oooq | 16:48 | |
| sshnaidm|afk | weshay, we skip these containers, look at build script in build containers role | 16:49 |
| weshay | sshnaidm|afk ya .. I know | 16:52 |
| *** dsneddon has quit IRC | 16:52 | |
| weshay | sshnaidm|afk testing each one to see what the issues are https://review.opendev.org/#/c/678056/ | 16:53 |
| weshay | for example | 16:53 |
| rlandy|ruck | wow fs039 | 16:55 |
| rlandy|ruck | https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-centos-7-ovb-3ctlr_1comp_1supp-featureset039-master | 16:55 |
| rlandy|ruck | never seen anything like it | 16:55 |
| rlandy|ruck | passes 50% | 16:55 |
| rlandy|ruck | every second time | 16:55 |
| rlandy|ruck | same with stein | 16:55 |
| rlandy|ruck | every second time | 16:56 |
| weshay | rlandy|ruck you can poke harry r | 16:57 |
| rlandy|ruck | I am not sure it's on his side | 16:57 |
| *** dsneddon has joined #oooq | 17:00 | |
| *** jtomasek has joined #oooq | 17:06 | |
| *** tesseract has quit IRC | 17:13 | |
| *** panda|out has quit IRC | 17:16 | |
| *** panda has joined #oooq | 17:20 | |
| amoralej | rlandy|ruck, i've done some progress | 17:23 |
| rlandy|ruck | great | 17:23 |
| amoralej | tempest executions get stuck | 17:23 |
| amoralej | forever | 17:24 |
| amoralej | i'd say it's related to watcher or l2gw tests | 17:24 |
| raukadah | amoralej: why we are running those which we donot support? | 17:24 |
| amoralej | they are supported in rdo | 17:24 |
| rlandy|ruck | raukadah: you're on vacation!! | 17:25 |
| amoralej | are supported in p-o-i | 17:25 |
| raukadah | amoralej: ack! | 17:25 |
| amoralej | rlandy|ruck, i'll continue tomorrow | 17:25 |
| rlandy|ruck | amoralej: ack - k | 17:26 |
| raukadah | rlandy|ruck: :-) | 17:26 |
| rlandy|ruck | raukadah: log off - I promise to leave you notes | 17:26 |
| rfolco | weshay, in https://review.rdoproject.org/r/#/c/21873/ your comment is to run the promoter script as standalone (not as a service) and ignore telegraf service ? | 17:30 |
| *** amoralej is now known as amoralej|off | 17:31 | |
| rlandy|ruck | k - back to limestone | 17:31 |
| weshay | rfolco nope.. my comment is to not automatically start it.. until all the config is laid down properly.. e..g. the diagram | 17:37 |
| *** jbadiapa has joined #oooq | 17:37 | |
| rfolco | weshay, ack, disabling for now to fix rest of stuff first | 17:38 |
| rfolco | weshay, thx | 17:38 |
| weshay | zbr sshnaidm|afk fyi https://bugs.launchpad.net/tripleo/+bug/1841096 | 17:43 |
| openstack | Launchpad bug 1841096 in tripleo "fluentd missing deps in building containers for RHEL 8" [Critical,Triaged] | 17:43 |
| *** Vorrtex has quit IRC | 17:47 | |
| arxcruz | zbr: save some time tomorrow so we can discuss molecule ok? :) | 17:49 |
| *** Vorrtex has joined #oooq | 17:50 | |
| zbr | sure! just ping me. early is usually better, after US wakes it starts to become busy ;) | 17:51 |
| zbr | arxcruz: or I can book a flight to berlin, I kinda miss the german beer. | 17:54 |
| arxcruz | zbr: works better to me | 17:54 |
| arxcruz | zbr: do it while you can, before brexit :P | 17:55 |
| zbr | well, i have dual citizenship, whatever happens I can jump in another boat. | 17:56 |
| *** jbadiapa has quit IRC | 17:56 | |
| arxcruz | zbr: they started to call people from uk living here to take the visa in advance to brexit | 17:57 |
| arxcruz | raukadah: I found the issue with os_tempest and fs001 | 17:57 |
| arxcruz | the issues* in plural | 17:57 |
| *** Vorrtex has quit IRC | 18:00 | |
| *** jbadiapa has joined #oooq | 18:01 | |
| rlandy|ruck | weshay: I don;t have much traction on the limestone failures | 18:03 |
| rlandy|ruck | three times it was the same review | 18:03 |
| rlandy|ruck | https://9c17408a0cf9894af819-5f89b8b4a95cc06ff3feb4ddf9cead5d.ssl.cf5.rackcdn.com/676373/8/check/tripleo-ci-centos-7-containers-multinode/01dd3d8/job-output.txt | 18:03 |
| rlandy|ruck | ^^ is it still dlrn access though | 18:04 |
| weshay | rlandy|ruck we need to figure out to catch / retry that | 18:04 |
| rlandy|ruck | leaving on ruck/rover page to track | 18:04 |
| * rlandy|ruck looks | 18:04 | |
| weshay | rlandy|ruck perhaps we can pull it out of config and make it a direct ansible tasks.. and set the var for the config to pick up | 18:05 |
| rlandy|ruck | problem is the errors aren;t consistent | 18:09 |
| rlandy|ruck | sometimes they are in deploy | 18:09 |
| rlandy|ruck | looking for a single cause here | 18:09 |
| rlandy|ruck | on the upside - general gate failures are down | 18:10 |
| *** rlandy|ruck is now known as rlandy|ruck|brb | 18:13 | |
| *** irclogbot_0 has quit IRC | 18:14 | |
| *** irclogbot_0 has joined #oooq | 18:14 | |
| *** jtomasek has quit IRC | 18:21 | |
| *** jtomasek has joined #oooq | 18:25 | |
| *** altlogbot_3 has quit IRC | 18:29 | |
| *** ksambor|away is now known as ksambor | 18:30 | |
| *** rlandy|ruck|brb is now known as rlandy|ruck | 18:31 | |
| *** dsneddon has quit IRC | 18:32 | |
| *** jtomasek has quit IRC | 18:36 | |
| *** jtomasek has joined #oooq | 18:39 | |
| rlandy|ruck | arxcruz: k - I don;t think my tempest change broke anything https://review.rdoproject.org/r/#/c/21683/ | 18:45 |
| arxcruz | rlandy|ruck: and I believe it fix the other that was failing right ? | 18:46 |
| rlandy|ruck | arxcruz; the queens jobs | 18:46 |
| arxcruz | yup | 18:46 |
| rlandy|ruck | master never failed with this | 18:46 |
| rlandy|ruck | but I just wanted to check I didn't hose anything else | 18:47 |
| arxcruz | yup | 18:47 |
| rlandy|ruck | raukadah: tomorrow, pls vote on https://review.opendev.org/#/c/678008 | 18:47 |
| arxcruz | i +1 that's all i can do :/ | 18:47 |
| rlandy|ruck | arxcruz: well your +1 in tempest stuff is worth a lot | 18:47 |
| *** jtomasek has quit IRC | 18:49 | |
| weshay | zbr I've updated your user story https://tree.taiga.io/project/tripleo-ci-board/us/1228?milestone=236933 | 18:55 |
| *** dsneddon has joined #oooq | 19:04 | |
| *** saneax has quit IRC | 19:17 | |
| *** saneax has joined #oooq | 19:17 | |
| *** panda has quit IRC | 19:23 | |
| *** panda has joined #oooq | 19:29 | |
| *** saneax has quit IRC | 19:55 | |
| *** Goneri has quit IRC | 21:14 | |
| *** Goneri has joined #oooq | 22:14 | |
| *** rlandy|ruck has quit IRC | 23:50 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!