| *** d0ugal has quit IRC | 00:06 | |
| *** d0ugal has joined #tripleo | 00:22 | |
| openstackgerrit | Steve Baker proposed openstack/tripleo-heat-templates master: Create an endpoint entry for novajoin https://review.opendev.org/683241 | 00:30 |
|---|---|---|
| *** dsneddon has joined #tripleo | 00:48 | |
| *** dsneddon has quit IRC | 00:52 | |
| *** gbarros has quit IRC | 00:59 | |
| openstackgerrit | Steve Baker proposed openstack/tripleo-heat-templates master: Create an endpoint entry for novajoin https://review.opendev.org/683241 | 01:04 |
| *** d0ugal has quit IRC | 01:07 | |
| *** d0ugal has joined #tripleo | 01:23 | |
| *** weshay has quit IRC | 01:24 | |
| *** rfolco has joined #tripleo | 01:40 | |
| *** ykarel|away has joined #tripleo | 01:57 | |
| openstackgerrit | Merged openstack/diskimage-builder master: Install Python 3 libselinux packages for Fedora https://review.opendev.org/683295 | 02:05 |
| *** rfolco has quit IRC | 02:08 | |
| *** aakarsh has quit IRC | 02:20 | |
| EmilienM | sshnaidm: hey, please push over https://review.opendev.org/#/c/683802/2/tripleo_ansible/ansible_plugins/modules/paunch.py if you want -check my comments first) | 02:32 |
| *** dtrainor has joined #tripleo | 02:37 | |
| openstackgerrit | Emilien Macchi proposed openstack/paunch stable/stein: Improve volume validation https://review.opendev.org/683386 | 02:41 |
| *** dsneddon has joined #tripleo | 02:48 | |
| ykarel|away | EmilienM, can u review https://review.opendev.org/#/c/683771/ to clear queens promotion | 02:50 |
| *** dsneddon has quit IRC | 02:53 | |
| EmilienM | ykarel|away: looking | 02:56 |
| ykarel|away | EmilienM, Thnks | 03:04 |
| ykarel|away | Thanks | 03:04 |
| openstackgerrit | Merged openstack/python-tripleoclient stable/stein: Rename 'undercloud-' to '<stack_name>-' https://review.opendev.org/682504 | 03:17 |
| *** ricolin has joined #tripleo | 03:20 | |
| *** psachin has joined #tripleo | 03:28 | |
| openstackgerrit | Merged openstack/puppet-tripleo master: Use memcached for token caching in authtoken for telemetry services https://review.opendev.org/681245 | 03:30 |
| *** ykarel|away has quit IRC | 03:37 | |
| *** dmacpher has quit IRC | 03:50 | |
| *** dmacpher has joined #tripleo | 03:51 | |
| *** ykarel|away has joined #tripleo | 03:57 | |
| sshnaidm | EmilienM, then it doesn't support check mode :) Need to know before the possible change if it will be changed | 04:10 |
| *** ykarel|away is now known as ykarel | 04:11 | |
| *** surpatil has joined #tripleo | 04:19 | |
| *** ramishra has joined #tripleo | 04:24 | |
| *** brault has quit IRC | 04:43 | |
| *** holser has joined #tripleo | 04:49 | |
| *** dsneddon has joined #tripleo | 04:49 | |
| *** holser has quit IRC | 04:51 | |
| *** dsneddon has quit IRC | 04:54 | |
| *** slaweq has joined #tripleo | 04:57 | |
| *** slaweq has quit IRC | 05:05 | |
| *** soniya29 has joined #tripleo | 05:06 | |
| openstackgerrit | Merged openstack/tripleo-common stable/queens: Raise an error if deployment_name cannot be determined https://review.opendev.org/682067 | 05:08 |
| openstackgerrit | Merged openstack/paunch stable/stein: Fixing upper stable url https://review.opendev.org/683413 | 05:08 |
| *** ratailor has joined #tripleo | 05:10 | |
| *** slaweq has joined #tripleo | 05:11 | |
| *** iurygregory has joined #tripleo | 05:12 | |
| *** lmiccini has joined #tripleo | 05:12 | |
| *** skramaja has joined #tripleo | 05:15 | |
| *** waleedm has joined #tripleo | 05:16 | |
| *** marios has joined #tripleo | 05:19 | |
| *** slaweq has quit IRC | 05:28 | |
| *** slaweq has joined #tripleo | 05:30 | |
| openstackgerrit | Sagi Shnaidman proposed openstack/tripleo-ansible master: paunch: fix check mode https://review.opendev.org/683802 | 05:33 |
| openstackgerrit | Sagi Shnaidman proposed openstack/tripleo-ansible master: paunch: disable check mode https://review.opendev.org/683802 | 05:35 |
| openstackgerrit | Sagi Shnaidman proposed openstack/tripleo-heat-templates master: deploy-steps-tasks: replace paunch CLI by ansible-paunch module https://review.opendev.org/682094 | 05:35 |
| *** yolanda has quit IRC | 05:42 | |
| *** jaosorior has joined #tripleo | 05:43 | |
| *** yolanda has joined #tripleo | 05:49 | |
| *** brault has joined #tripleo | 05:49 | |
| openstackgerrit | Sagi Shnaidman proposed openstack/tripleo-heat-templates master: DNM: test sc012 job https://review.opendev.org/683848 | 05:57 |
| *** raukadah is now known as chandankumar | 06:01 | |
| *** rpioso is now known as rpioso|afk | 06:05 | |
| *** yprokule has joined #tripleo | 06:10 | |
| Tengu | hello there | 06:16 |
| *** brault has quit IRC | 06:17 | |
| *** yolanda has quit IRC | 06:22 | |
| *** florianf has joined #tripleo | 06:25 | |
| openstackgerrit | Slawek Kaplonski proposed openstack/tripleo-heat-templates master: Remove leftovers after NeutronExternalNetworkBridge option https://review.opendev.org/683380 | 06:25 |
| openstackgerrit | Kamil Sambor proposed openstack/tripleo-heat-templates master: Add posibilities to set ovn_openflow_probe_interval for controller https://review.opendev.org/683294 | 06:26 |
| *** slaweq has quit IRC | 06:27 | |
| *** saneax has joined #tripleo | 06:30 | |
| *** hamdyk has joined #tripleo | 06:33 | |
| *** slaweq has joined #tripleo | 06:34 | |
| *** rpittau|afk is now known as rpittau | 06:40 | |
| *** brault has joined #tripleo | 06:41 | |
| *** brault has quit IRC | 06:41 | |
| *** brault has joined #tripleo | 06:41 | |
| *** avivgt has quit IRC | 06:44 | |
| *** fmount- is now known as fmount | 06:48 | |
| *** dsneddon has joined #tripleo | 06:50 | |
| *** bogdando has joined #tripleo | 06:52 | |
| *** dsneddon has quit IRC | 06:55 | |
| *** abishop has joined #tripleo | 06:56 | |
| *** cylopez has joined #tripleo | 06:57 | |
| *** florianf has quit IRC | 06:58 | |
| *** florianf has joined #tripleo | 06:59 | |
| *** tesseract has joined #tripleo | 07:00 | |
| *** rcernin has quit IRC | 07:04 | |
| *** slaweq has quit IRC | 07:04 | |
| *** slaweq has joined #tripleo | 07:07 | |
| *** alexmcleod has joined #tripleo | 07:13 | |
| *** amoralej|off is now known as amoralej | 07:15 | |
| *** ccamacho has joined #tripleo | 07:16 | |
| *** xek has joined #tripleo | 07:17 | |
| *** tosky has joined #tripleo | 07:20 | |
| *** ramishra has quit IRC | 07:21 | |
| *** ramishra has joined #tripleo | 07:36 | |
| *** mschuppert has joined #tripleo | 07:38 | |
| *** apetrich has joined #tripleo | 07:39 | |
| *** jaganathan has joined #tripleo | 07:41 | |
| *** pcaruana has joined #tripleo | 07:41 | |
| *** jpena|off is now known as jpena | 07:42 | |
| *** hberaud has joined #tripleo | 07:42 | |
| *** slaweq has quit IRC | 07:47 | |
| *** jtomasek has quit IRC | 07:47 | |
| *** jtomasek has joined #tripleo | 07:48 | |
| *** avivgts has joined #tripleo | 07:51 | |
| *** jpich has joined #tripleo | 07:56 | |
| *** ykarel is now known as ykarel|lunch | 08:00 | |
| openstackgerrit | Martin Schuppert proposed openstack/tripleo-heat-templates master: Depreace EC2 API https://review.opendev.org/683319 | 08:04 |
| *** yolanda has joined #tripleo | 08:09 | |
| *** waleedm has quit IRC | 08:14 | |
| openstackgerrit | Jose Luis Franco proposed openstack/tripleo-upgrade master: Handle when ceph_osds is a dictionary or a list. https://review.opendev.org/683868 | 08:17 |
| *** iurygregory has quit IRC | 08:19 | |
| *** brault has quit IRC | 08:21 | |
| *** cylopez has quit IRC | 08:22 | |
| *** cylopez has joined #tripleo | 08:24 | |
| *** waleedm has joined #tripleo | 08:28 | |
| *** brault has joined #tripleo | 08:32 | |
| openstackgerrit | Michele Baldessari proposed openstack/puppet-pacemaker master: Silence useless facter errors when calling crm_node https://review.opendev.org/683878 | 08:34 |
| openstackgerrit | Martin Schuppert proposed openstack/tripleo-common stable/queens: [Queens] Add workflow to do cellv2 host discovery https://review.opendev.org/683067 | 08:36 |
| *** holser has joined #tripleo | 08:36 | |
| *** iurygregory has joined #tripleo | 08:45 | |
| *** rcernin has joined #tripleo | 08:46 | |
| *** derekh has joined #tripleo | 08:49 | |
| *** yolanda has quit IRC | 08:50 | |
| *** dsneddon has joined #tripleo | 08:51 | |
| *** dsneddon has quit IRC | 08:55 | |
| openstackgerrit | Martin Schuppert proposed openstack/tripleo-heat-templates master: Deprecate EC2 API https://review.opendev.org/683319 | 08:57 |
| openstackgerrit | Bogdan Dobrelya (bogdando) proposed openstack/ansible-role-tripleo-modify-image master: Add support for yum caching when buildah updates https://review.opendev.org/683100 | 08:57 |
| *** gfidente has joined #tripleo | 08:58 | |
| *** ykarel|lunch is now known as ykarel | 08:58 | |
| *** holser has quit IRC | 08:59 | |
| *** chem has joined #tripleo | 09:01 | |
| openstackgerrit | Jose Luis Franco proposed openstack/tripleo-heat-templates master: Get rid of docker removing in post_upgrade tasks. https://review.opendev.org/681669 | 09:01 |
| *** dtantsur|afk is now known as dtantsur | 09:07 | |
| *** holser has joined #tripleo | 09:10 | |
| *** paramite has joined #tripleo | 09:19 | |
| openstackgerrit | Jose Luis Franco proposed openstack/tripleo-upgrade stable/stein: Ensure pacemaker bootstrap controller is upgraded first. https://review.opendev.org/683903 | 09:23 |
| openstackgerrit | Natal Ngétal proposed openstack/python-tripleoclient master: Fix message of unused_params and invalid_role_specific_params. https://review.opendev.org/674570 | 09:35 |
| *** pierreprinetti has joined #tripleo | 09:37 | |
| openstackgerrit | Bogdan Dobrelya (bogdando) proposed openstack/ansible-role-tripleo-modify-image master: DNM Test CI jobs with yum caching ON https://review.opendev.org/683904 | 09:38 |
| openstackgerrit | Bogdan Dobrelya (bogdando) proposed openstack/ansible-role-tripleo-modify-image master: DNM Test CI jobs with yum caching ON https://review.opendev.org/683904 | 09:39 |
| *** rohit02 has joined #tripleo | 09:40 | |
| *** dsneddon has joined #tripleo | 09:40 | |
| rohit02 | Hi | 09:40 |
| *** ricolin has quit IRC | 09:48 | |
| *** chem has quit IRC | 09:52 | |
| *** amoralej is now known as amoralej|brb | 09:56 | |
| *** hberaud is now known as hberaud|school-r | 09:57 | |
| openstackgerrit | Merged openstack/tripleo-ansible master: increase coverage of tripleo-ansible https://review.opendev.org/683587 | 10:01 |
| *** amoralej|brb is now known as amoralej | 10:03 | |
| *** morazi has quit IRC | 10:04 | |
| *** hberaud|school-r is now known as hberaud|lunch | 10:05 | |
| openstackgerrit | Gael Chamoulaud proposed openstack/tripleo-validations master: Add new healthcheck-service-status validation https://review.opendev.org/682377 | 10:06 |
| sshnaidm | arxcruz|rover, zbr|ruck fyi: https://bugs.launchpad.net/tripleo/+bug/1844989 | 10:12 |
| openstack | Launchpad bug 1844989 in tripleo "Tempest tests fail on standalone ironic job scenario012" [High,Triaged] | 10:12 |
| sshnaidm | arxcruz|rover, zbr|ruck it's not promotion blocker, just fyi | 10:12 |
| sshnaidm | dtantsur, derekh hi, do you know who can look at ironic standalone job failure? https://bugs.launchpad.net/tripleo/+bug/1844989 | 10:13 |
| arxcruz|rover | sshnaidm: i blame dtantsur | 10:13 |
| sshnaidm | arxcruz|rover, because he's closer? :) | 10:14 |
| arxcruz|rover | sshnaidm: nah, he left berlim | 10:14 |
| *** waleedm has quit IRC | 10:14 | |
| sshnaidm | Tengu, hi, is validate-services role coming from your squad? | 10:20 |
| *** waleedm has joined #tripleo | 10:21 | |
| openstackgerrit | Merged openstack/tripleo-heat-templates master: Support deploying multiple Cinder Pure Storage backends https://review.opendev.org/681137 | 10:21 |
| sshnaidm | Tengu, I have failure here for podman only, wonder if you're familiar with that: https://606e7073949a555d6ce7-a5a94d16cd1e63fdf610099df3afaf88.ssl.cf5.rackcdn.com/682341/6/check/tripleo-ci-centos-7-scenario012-standalone/e20ae64/job-output.txt | 10:22 |
| *** paramite has quit IRC | 10:25 | |
| derekh | sshnaidm: looks like it might be related to the latest change to create-node.sh https://review.opendev.org/#/c/681446/ | 10:31 |
| derekh | dtantsur: any idea if we need to update something to use the new create-node.sh ? ^ | 10:31 |
| *** brault has quit IRC | 10:36 | |
| *** brault has joined #tripleo | 10:39 | |
| gchamoul | sshnaidm: that's not a role coming from tripleo-validations if it's what you meant | 10:41 |
| gchamoul | sshnaidm: that's a oooq-extras role | 10:41 |
| openstackgerrit | Gael Chamoulaud proposed openstack/tripleo-validations master: Add new healthcheck-service-status validation https://review.opendev.org/682377 | 10:42 |
| *** brault has quit IRC | 10:44 | |
| openstackgerrit | Kamil Sambor proposed openstack/puppet-tripleo master: Use correct paths to configure ovn dbs certs https://review.opendev.org/683916 | 10:47 |
| *** brault has joined #tripleo | 10:49 | |
| openstackgerrit | Kamil Sambor proposed openstack/tripleo-heat-templates master: WIP: Support connecting OVN DB over SSL https://review.opendev.org/680345 | 10:50 |
| *** brault has quit IRC | 10:53 | |
| *** soniya29 has quit IRC | 10:53 | |
| *** waleedm has quit IRC | 10:56 | |
| ykarel | bogdando, commented on abandon patch:- https://review.opendev.org/#/c/683101/5 | 10:57 |
| ykarel | let me know if i am unclear | 10:57 |
| *** jaosorior has quit IRC | 10:58 | |
| dtantsur | derekh: yep. a storage pool is no longer created and started in this script | 10:58 |
| dtantsur | (it shouldn't have been there to begin with) | 10:58 |
| dtantsur | so yeah, arxcruz|rover is right to blame me :) | 10:59 |
| arxcruz|rover | ;) | 11:00 |
| EmilienM | sshnaidm: thank you | 11:01 |
| *** marios has quit IRC | 11:02 | |
| sshnaidm | EmilienM, np | 11:05 |
| sshnaidm | gchamoul, ack | 11:05 |
| Tengu | sshnaidm: so I did add it something like 6 months ago to oooq-extras. | 11:06 |
| sshnaidm | I see tripleo_swift_account_server_healthcheck.service healthcheck works for docker, but fails for podman.. | 11:09 |
| openstackgerrit | Merged openstack/puppet-pacemaker master: Silence useless facter errors when calling crm_node https://review.opendev.org/683878 | 11:13 |
| sshnaidm | EmilienM, do you know who can help with tripleo_swift_account_server_healthcheck.service ? https://606e7073949a555d6ce7-a5a94d16cd1e63fdf610099df3afaf88.ssl.cf5.rackcdn.com/682341/6/check/tripleo-ci-centos-7-scenario012-standalone/e20ae64/logs/undercloud/var/log/extra/failed_services.txt.gz | 11:13 |
| EmilienM | sshnaidm: cschwede ^ who can help? | 11:14 |
| *** jaganathan has quit IRC | 11:15 | |
| *** soniya29 has joined #tripleo | 11:16 | |
| cschwede | EmilienM, sshnaidm: will have a look | 11:16 |
| EmilienM | cschwede: thanks! | 11:17 |
| *** dsneddon has quit IRC | 11:17 | |
| sshnaidm | cschwede, thanks, docker job: https://storage.bhs1.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_bbe/683848/1/check/tripleo-ci-centos-7-scenario012-standalone/bbee5fe/job-output.txt podman job: https://606e7073949a555d6ce7-a5a94d16cd1e63fdf610099df3afaf88.ssl.cf5.rackcdn.com/682341/6/check/tripleo-ci-centos-7-scenario012-standalone/e20ae64/job-output.txt | 11:17 |
| *** raildo has joined #tripleo | 11:18 | |
| *** cschwede is now known as cschwede|afk | 11:20 | |
| *** ccamacho has quit IRC | 11:24 | |
| *** dtrainor has quit IRC | 11:27 | |
| *** jpena is now known as jpena|lunch | 11:28 | |
| *** jbadiapa has quit IRC | 11:30 | |
| *** lucasagomes has joined #tripleo | 11:31 | |
| *** rh-jelabarre has joined #tripleo | 11:31 | |
| *** amoralej is now known as amoralej|lunch | 11:32 | |
| *** cylopez has quit IRC | 11:33 | |
| *** dtrainor has joined #tripleo | 11:33 | |
| *** hberaud|lunch is now known as hberaud | 11:35 | |
| bogdando | ykarel: thanks | 11:36 |
| ykarel | bogdando, just curious how much is the improvement in runtime with the cache thing | 11:37 |
| bogdando | it's easy to compare once we have the CI results | 11:37 |
| ykarel | ack | 11:38 |
| *** paramite has joined #tripleo | 11:43 | |
| *** waleedm has joined #tripleo | 11:43 | |
| *** rh-jelabarre has quit IRC | 11:45 | |
| *** rh-jelabarre has joined #tripleo | 11:45 | |
| *** morazi has joined #tripleo | 11:46 | |
| *** zul has joined #tripleo | 11:47 | |
| *** waleedm_ has joined #tripleo | 11:47 | |
| *** dsneddon has joined #tripleo | 11:47 | |
| *** rcernin has quit IRC | 11:49 | |
| openstackgerrit | Gael Chamoulaud proposed openstack/tripleo-docs master: Fix tripleo-validations documentation page URL https://review.opendev.org/683931 | 11:51 |
| *** waleedm has quit IRC | 11:51 | |
| openstackgerrit | Chandan Kumar (raukadah) proposed openstack/tripleo-quickstart master: [CentOS-7]Added release file for train https://review.opendev.org/683932 | 11:52 |
| *** cylopez has joined #tripleo | 11:54 | |
| *** bfournie has quit IRC | 11:57 | |
| *** ekultails has joined #tripleo | 11:59 | |
| *** marios has joined #tripleo | 12:00 | |
| *** iurygregory has quit IRC | 12:01 | |
| *** suuuper has joined #tripleo | 12:02 | |
| openstackgerrit | Bogdan Dobrelya (bogdando) proposed openstack/tripleo-quickstart-extras master: Use cache for yum update in containers prepare https://review.opendev.org/683936 | 12:02 |
| openstackgerrit | Cédric Jeanneret (Tengu) proposed openstack/puppet-tripleo master: Update log-driver value for podman https://review.opendev.org/683937 | 12:04 |
| openstackgerrit | Gael Chamoulaud proposed openstack/tripleo-docs master: Fix tripleo-validations documentation page URL https://review.opendev.org/683931 | 12:06 |
| *** rfolco has joined #tripleo | 12:12 | |
| *** skramaja has quit IRC | 12:15 | |
| *** weshay has joined #tripleo | 12:15 | |
| *** weshay is now known as weshay|rover | 12:16 | |
| *** brault has joined #tripleo | 12:20 | |
| *** brault has quit IRC | 12:22 | |
| *** brault has joined #tripleo | 12:22 | |
| openstackgerrit | Damien Ciabrini proposed openstack/tripleo-heat-templates master: WIP HA: reorder init_bundle and restart_bundle for improved updates https://review.opendev.org/675993 | 12:22 |
| *** dsneddon has quit IRC | 12:22 | |
| openstackgerrit | Merged openstack/tripleo-docs master: Replace StandaloneLocalMtu with InterfaceLocalMtu https://review.opendev.org/683351 | 12:25 |
| *** rpioso|afk is now known as rpioso | 12:28 | |
| *** rlandy has joined #tripleo | 12:29 | |
| *** rascasoft has joined #tripleo | 12:31 | |
| *** dsneddon has joined #tripleo | 12:33 | |
| *** rcernin has joined #tripleo | 12:34 | |
| *** amoralej|lunch is now known as amoralej | 12:34 | |
| *** ratailor has quit IRC | 12:35 | |
| *** dsneddon has quit IRC | 12:38 | |
| cloudnull | mornings | 12:39 |
| *** lbragstad_ is now known as lbragstad | 12:39 | |
| *** psachin has quit IRC | 12:39 | |
| EmilienM | weshay|rover, rlandy : can we sync up on https://review.rdoproject.org/r/#/c/22445/ today? | 12:41 |
| weshay|rover | cool w/ me | 12:41 |
| EmilienM | weshay|rover, rlandy : I can only do post 4pm est | 12:43 |
| rlandy | fine by me | 12:44 |
| *** brault has quit IRC | 12:45 | |
| *** bfournie has joined #tripleo | 12:47 | |
| *** jpena|lunch is now known as jpena | 12:47 | |
| *** rohit02 has quit IRC | 12:48 | |
| openstackgerrit | Kevin Carter (cloudnull) proposed openstack/tripleo-heat-templates master: Move basic global functions into ansible tasks https://review.opendev.org/683286 | 12:49 |
| *** jcoufal has joined #tripleo | 12:50 | |
| openstackgerrit | Kevin Carter (cloudnull) proposed openstack/tripleo-heat-templates master: Convert subprocess to a reusable method https://review.opendev.org/683429 | 12:51 |
| openstackgerrit | Kevin Carter (cloudnull) proposed openstack/tripleo-heat-templates master: Move globals into main level 0 case https://review.opendev.org/683590 | 12:51 |
| openstackgerrit | Kevin Carter (cloudnull) proposed openstack/tripleo-heat-templates master: Move config processing into consolodated methods https://review.opendev.org/683670 | 12:51 |
| *** iurygregory has joined #tripleo | 12:51 | |
| *** aakarsh has joined #tripleo | 12:55 | |
| *** haleyb has joined #tripleo | 12:55 | |
| *** ansmith has joined #tripleo | 12:55 | |
| *** matbu is now known as matbu|hfpto | 12:58 | |
| *** ricolin has joined #tripleo | 12:59 | |
| *** jfrancoa has joined #tripleo | 13:00 | |
| EmilienM | rlandy, weshay|rover I need to push one more thing in my patch for a curl then you can review it maybe. Could we manually execute it and test it? | 13:00 |
| *** fultonj has joined #tripleo | 13:00 | |
| *** gbarros has joined #tripleo | 13:02 | |
| chandankumar | weshay|rover EmilienM I build the podman rpm locally | 13:03 |
| *** rlandy is now known as rlandy|mtg | 13:03 | |
| chandankumar | it takes 10-15 minutes downloading go binaries | 13:04 |
| rlandy|mtg | yes | 13:04 |
| cloudnull | weshay|rover EmilienM how are things on the perf front, any new developments ? | 13:04 |
| cloudnull | anything you all want / need help digging into ? | 13:04 |
| weshay|rover | cloudnull, EmilienM it's gotten worse :( | 13:04 |
| EmilienM | chandankumar: what? wow for me it takes 2 or 3 min | 13:04 |
| weshay|rover | http://dashboard-ci.tripleo.org/d/si1tipHZk/jobs-exploration?orgId=1&from=now-90d&to=now&var-influxdb_filter=job_name%7C%3D%7Ctripleo-ci-centos-7-containers-multinode&var-influxdb_filter=branch%7C%3D%7Cmaster&fullscreen&panelId=16 | 13:04 |
| weshay|rover | 9/18 - 9/19 | 13:05 |
| chandankumar | EmilienM: may be it depends on my side network connectivity | 13:05 |
| openstackgerrit | Michele Baldessari proposed openstack/tripleo-heat-templates master: Give the OVN DBS service a separate Vip https://review.opendev.org/669847 | 13:05 |
| cloudnull | weshay|rover yeah I saw - https://review.opendev.org/#/c/682945/ @ 4+ hours :'( | 13:05 |
| chandankumar | EmilienM: I generated using running this command https://github.com/containers/libpod/blob/master/contrib/build_rpm.sh | 13:05 |
| cloudnull | are we running cent 7.7? | 13:06 |
| *** waleedm has joined #tripleo | 13:06 | |
| cloudnull | maybe related ? | 13:06 |
| weshay|rover | cloudnull, yes | 13:06 |
| *** aakarsh has quit IRC | 13:06 | |
| cloudnull | package mirrors not quite sync'd so things are slow ? | 13:06 |
| *** aakarsh has joined #tripleo | 13:06 | |
| weshay|rover | would assume they are synced by now | 13:06 |
| EmilienM | if we suspect rpm download time, we need to compare with how long it took a few week/days ago | 13:07 |
| weshay|rover | arxcruz|rover, ^ | 13:07 |
| weshay|rover | can you hook that up? | 13:07 |
| * arxcruz|rover reading the context | 13:08 | |
| *** waleedm_ has quit IRC | 13:08 | |
| cloudnull | for the time to increase by an hour +, when we're disabling things to make them faster I would suspect an external factor | 13:10 |
| *** mcornea has joined #tripleo | 13:10 | |
| *** jaosorior has joined #tripleo | 13:11 | |
| cloudnull | it looks like the other jobs are running a little longer now too, 2~3 hours | 13:11 |
| cloudnull | http://dashboard-ci.tripleo.org/d/si1tipHZk/jobs-exploration?orgId=1&from=now-90d&to=now&var-influxdb_filter=job_name%7C%3D%7Ctripleo-ci-centos-7-standalone&var-influxdb_filter=branch%7C%3D%7Cmaster | 13:12 |
| weshay|rover | yup.. standalone as well.. over 3 hours | 13:12 |
| *** aakarsh has quit IRC | 13:13 | |
| *** dsneddon has joined #tripleo | 13:14 | |
| *** artom has joined #tripleo | 13:19 | |
| *** beekneemech is now known as bnemec | 13:24 | |
| *** brault has joined #tripleo | 13:26 | |
| *** brault has quit IRC | 13:26 | |
| *** waleedm has quit IRC | 13:26 | |
| *** waleedm has joined #tripleo | 13:27 | |
| *** gbarros has quit IRC | 13:35 | |
| *** surpatil has quit IRC | 13:40 | |
| openstackgerrit | Martin Schuppert proposed openstack/tripleo-heat-templates master: Deprecate EC2 API https://review.opendev.org/683319 | 13:44 |
| *** yolanda has joined #tripleo | 13:44 | |
| *** rcernin has quit IRC | 13:45 | |
| openstackgerrit | Alan Bishop proposed openstack/tripleo-heat-templates stable/stein: Fix selinux context for glance-api https://review.opendev.org/683960 | 13:46 |
| *** dsneddon has quit IRC | 13:46 | |
| *** bdodd has joined #tripleo | 13:50 | |
| *** yolanda has quit IRC | 13:52 | |
| *** redrobot has quit IRC | 13:53 | |
| *** dsneddon has joined #tripleo | 13:54 | |
| ykarel | EmilienM, hi | 13:55 |
| EmilienM | ykarel: what's up | 13:56 |
| ykarel | EmilienM, re. the performance issue which is seen in CI, are the things collected at some common place? | 13:56 |
| ykarel | i want to see what part is affecting the most | 13:56 |
| weshay|rover | ykarel, if you want to join our call in 5 | 13:56 |
| weshay|rover | I'll ping you | 13:56 |
| ykarel | weshay|rover, sure | 13:56 |
| ykarel | will join | 13:56 |
| *** yolanda has joined #tripleo | 13:56 | |
| weshay|rover | sshnaidm, arxcruz|rover ykarel https://meet.google.com/fvy-jskc-qmn?authuser=1 | 13:59 |
| *** dsneddon has quit IRC | 14:00 | |
| *** rlandy|mtg is now known as rlandy | 14:00 | |
| *** jcoufal_ has joined #tripleo | 14:00 | |
| ykarel | ack | 14:00 |
| *** Vorrtex has joined #tripleo | 14:01 | |
| *** slaweq has joined #tripleo | 14:01 | |
| *** yolanda has quit IRC | 14:03 | |
| *** jcoufal has quit IRC | 14:04 | |
| *** Guest30550 has joined #tripleo | 14:05 | |
| *** saneax has quit IRC | 14:06 | |
| *** brault has joined #tripleo | 14:08 | |
| *** brault has quit IRC | 14:08 | |
| *** Guest30550 is now known as redrobot | 14:08 | |
| *** brault has joined #tripleo | 14:09 | |
| *** brault has quit IRC | 14:09 | |
| *** jpena is now known as jpena|brb | 14:11 | |
| openstackgerrit | Matthew Booth proposed openstack/openstack-virtual-baremetal master: Don't use diskimage-builder to build IPXE image https://review.opendev.org/683972 | 14:13 |
| openstackgerrit | Matthew Booth proposed openstack/openstack-virtual-baremetal master: Explicitly set MTU on interfaces in IPXE image https://review.opendev.org/683973 | 14:13 |
| openstackgerrit | Dmitry Tantsur proposed openstack/tripleo-heat-templates master: Ironic: disallow deployment and cleaning in maintenance mode https://review.opendev.org/683975 | 14:15 |
| *** dsneddon has joined #tripleo | 14:16 | |
| *** mcornea has quit IRC | 14:17 | |
| *** yolanda has joined #tripleo | 14:19 | |
| *** dsneddon has quit IRC | 14:21 | |
| *** iurygregory has quit IRC | 14:23 | |
| *** hberaud is now known as hberaud|school-r | 14:24 | |
| *** gbarros has joined #tripleo | 14:27 | |
| *** mcornea has joined #tripleo | 14:32 | |
| *** hberaud|school-r is now known as hberaud | 14:34 | |
| openstackgerrit | Merged openstack/tripleo-docs master: Fix tripleo-validations documentation page URL https://review.opendev.org/683931 | 14:35 |
| *** soniya29 has quit IRC | 14:36 | |
| *** hamdyk has quit IRC | 14:38 | |
| openstackgerrit | yatin proposed openstack/tripleo-common master: [DNM] Test container prepare without gathering facts https://review.opendev.org/683981 | 14:39 |
| ykarel | weshay|rover, sshnaidm arxcruz|rover ^^ | 14:39 |
| *** tosky has quit IRC | 14:40 | |
| *** kopecmartin is now known as kopecmartin|off | 14:44 | |
| openstackgerrit | Kamil Sambor proposed openstack/puppet-tripleo master: Use correct paths to configure ovn dbs certs https://review.opendev.org/683916 | 14:54 |
| *** yolanda has quit IRC | 14:54 | |
| *** dsneddon has joined #tripleo | 14:56 | |
| openstackgerrit | Kamil Sambor proposed openstack/tripleo-heat-templates master: Support connecting OVN DB over SSL https://review.opendev.org/680345 | 14:58 |
| cloudnull | ykarel weshay|rover regarding fact gathering odyssey4me has a review up to use cached facts - https://review.opendev.org/#/c/682855/ - those two efforts may compliment one another. | 14:58 |
| * weshay|rover looks | 14:59 | |
| weshay|rover | cloudnull, cool. .thanks for the pointer | 15:00 |
| * weshay|rover wonders what ansible facts are needed per container for tripleo-image-modify | 15:00 | |
| cloudnull | less fact gathering and more stregigic gathering in general will definitely help deployment times. | 15:00 |
| weshay|rover | aye | 15:00 |
| weshay|rover | cloudnull, we're seeing 3min per container.. not all the containers but a lot for fact gathering | 15:01 |
| ykarel | cloudnull, cool, did you noticed some improvement with that patch | 15:01 |
| odyssey4me | cloudnull++ being able to set gather_subset to something that does not include facter/ohai would make a big difference... especially in large deployments | 15:01 |
| * weshay|rover wants to see the fall out from just turning it off | 15:01 | |
| *** cdearborn has joined #tripleo | 15:01 | |
| *** gfidente has quit IRC | 15:02 | |
| weshay|rover | arxcruz|rover, ping me when ur ready to look at this stuff again | 15:02 |
| arxcruz|rover | weshay|rover: ok | 15:02 |
| *** jpena|brb is now known as jpena | 15:05 | |
| *** pcaruana has quit IRC | 15:05 | |
| *** dsneddon has quit IRC | 15:06 | |
| *** ykarel is now known as ykarel|afk | 15:08 | |
| *** udesale has joined #tripleo | 15:09 | |
| lbragstad | o/ hey folks - quick question on operating an overcloud, is there a docs page floating around somewhere that describes service management (e.g., restarting keystone for example)? | 15:15 |
| *** abishop has quit IRC | 15:16 | |
| lbragstad | actually - https://fatmin.com/2018/11/10/red-hat-openstack-13-containerized-services-operations-guide/ probably has my answer | 15:20 |
| *** avivgts has quit IRC | 15:25 | |
| *** dtantsur is now known as dtantsur|afk | 15:26 | |
| *** jschlueter has quit IRC | 15:26 | |
| *** ykarel|afk is now known as ykarel | 15:27 | |
| arxcruz|rover | weshay|rover: done | 15:37 |
| weshay|rover | arxcruz|rover, k.. let's chat for a few | 15:38 |
| *** yolanda has joined #tripleo | 15:40 | |
| *** marios is now known as marios|out | 15:40 | |
| *** jfrancoa has quit IRC | 15:44 | |
| *** lmiccini has quit IRC | 15:44 | |
| *** jschlueter has joined #tripleo | 15:45 | |
| *** dsneddon has joined #tripleo | 15:45 | |
| *** marios|out has quit IRC | 15:47 | |
| *** ykarel is now known as ykarel|afk | 15:47 | |
| openstackgerrit | Merged openstack/tripleo-upgrade stable/stein: Change docker references to podman. https://review.opendev.org/683418 | 15:47 |
| *** cylopez has quit IRC | 15:50 | |
| *** rpittau is now known as rpittau|afk | 15:52 | |
| openstackgerrit | Merged openstack/tripleo-heat-templates stable/queens: Stop bindmounting /usr/bin https://review.opendev.org/683771 | 15:52 |
| *** suuuper has quit IRC | 15:57 | |
| *** yprokule has quit IRC | 16:03 | |
| *** jfrancoa has joined #tripleo | 16:04 | |
| *** pcaruana has joined #tripleo | 16:06 | |
| *** alexmcleod has quit IRC | 16:07 | |
| openstackgerrit | Luke Short proposed openstack/paunch master: Handle defined containers that are stopped (version 2). https://review.opendev.org/678873 | 16:10 |
| *** slaweq has quit IRC | 16:10 | |
| *** ykarel|afk is now known as ykarel|away | 16:11 | |
| *** jpich has quit IRC | 16:13 | |
| *** bogdando has quit IRC | 16:20 | |
| *** ykarel|away has quit IRC | 16:36 | |
| *** jschlueter has quit IRC | 16:42 | |
| *** amoralej is now known as amoralej|off | 16:45 | |
| *** jschlueter has joined #tripleo | 16:45 | |
| *** pcaruana has quit IRC | 16:46 | |
| openstackgerrit | Merged openstack/tripleo-upgrade stable/stein: Fix passing tags through infrared to the tripleo-upgrade role. https://review.opendev.org/683379 | 16:49 |
| *** Goneri has joined #tripleo | 16:54 | |
| *** saneax has joined #tripleo | 16:57 | |
| *** ykarel|away has joined #tripleo | 16:57 | |
| *** udesale has quit IRC | 17:00 | |
| *** b3nt_pin is now known as beagles | 17:00 | |
| *** hberaud is now known as hberaud|gone | 17:03 | |
| *** ykarel|away has quit IRC | 17:12 | |
| weshay|rover | cloudnull, your proxy patch is going to sit for a week | 17:12 |
| cloudnull | ++ | 17:13 |
| sshnaidm | If somebody is interested, there is AnsibleFest Community Contributors Summit is broadcasted in bluejeans now: https://bluejeans.com/3008457278/ | 17:14 |
| *** jpena is now known as jpena|off | 17:16 | |
| *** waleedm has quit IRC | 17:16 | |
| openstackgerrit | Sergii Golovatiuk proposed openstack/tripleo-upgrade master: Fix "unbound" variable https://review.opendev.org/684010 | 17:21 |
| *** derekh has quit IRC | 17:24 | |
| *** saneax has quit IRC | 17:27 | |
| *** dsneddon has quit IRC | 17:38 | |
| *** jtomasek has quit IRC | 17:42 | |
| *** ykarel has joined #tripleo | 17:43 | |
| cloudnull | sshnaidm cool! | 17:44 |
| sshnaidm | cloudnull, also discussion in #ansible-meeting | 17:44 |
| openstackgerrit | Sergii Golovatiuk proposed openstack/tripleo-upgrade master: Fix "unbound" variable https://review.opendev.org/684010 | 17:53 |
| *** boazel has joined #tripleo | 17:57 | |
| openstackgerrit | Sergii Golovatiuk proposed openstack/tripleo-upgrade master: Fix "unbound" variable https://review.opendev.org/684010 | 18:00 |
| *** ricolin has quit IRC | 18:02 | |
| openstackgerrit | Nicolas Hicher proposed openstack/tripleo-quickstart-extras master: DNM: fail before introspection for debugging https://review.opendev.org/684043 | 18:03 |
| weshay|rover | tbarron, you need anything from the ci team? | 18:10 |
| weshay|rover | pointer to repos etc? | 18:10 |
| *** dsneddon has joined #tripleo | 18:12 | |
| *** Goneri has quit IRC | 18:18 | |
| *** dsneddon has quit IRC | 18:18 | |
| *** holser has quit IRC | 18:19 | |
| *** dsneddon has joined #tripleo | 18:24 | |
| *** chem has joined #tripleo | 18:27 | |
| openstackgerrit | Brent Eagles proposed openstack/tripleo-ci master: Change featurset for octavia standalone to support full tempest https://review.opendev.org/669740 | 18:35 |
| *** morazi has quit IRC | 18:35 | |
| *** jtomasek has joined #tripleo | 18:36 | |
| *** slaweq has joined #tripleo | 18:44 | |
| *** brault has joined #tripleo | 18:54 | |
| *** ykarel has quit IRC | 19:01 | |
| *** lucasagomes has quit IRC | 19:06 | |
| *** lucasagomes has joined #tripleo | 19:06 | |
| openstackgerrit | Athlan-Guyot sofer proposed openstack/tripleo-upgrade master: Remove optimization during check of AP services. https://review.opendev.org/684122 | 19:08 |
| *** slaweq has quit IRC | 19:11 | |
| *** lucasagomes has quit IRC | 19:14 | |
| openstackgerrit | Merged openstack/tripleo-heat-templates master: Use _uri which is wrapped if IPv6 for ironic tftp https://review.opendev.org/683224 | 19:22 |
| openstackgerrit | Merged openstack/tripleo-upgrade master: Fix overcloud update workarounds creation. https://review.opendev.org/680612 | 19:22 |
| openstackgerrit | Merged openstack/puppet-tripleo stable/queens: Collectd QDR connection https://review.opendev.org/643596 | 19:22 |
| *** morazi has joined #tripleo | 19:27 | |
| weshay|rover | sshnaidm, https://9c08bb97d91554639aeb-1e695c46ce38568fe2b9076122bc0b06.ssl.cf1.rackcdn.com/682341/6/check/tripleo-ci-centos-7-containers-multinode-stein/65be156/logs/undercloud/var/log/tripleo-container-image-prepare.log.txt.gz | 19:28 |
| weshay|rover | arxcruz|rover, ^ | 19:29 |
| weshay|rover | over an hour | 19:29 |
| weshay|rover | cloudnull, ^ | 19:29 |
| *** jcoufal_ has quit IRC | 19:30 | |
| *** jcoufal has joined #tripleo | 19:30 | |
| weshay|rover | https://snapshot.raintank.io/dashboard/snapshot/SjZ16RtMXZCFbEIOjgZ7N7KKJvOpKBhB | 19:31 |
| weshay|rover | arxcruz|rover, sshnaidm cloudnull mwhahaha ^ | 19:31 |
| mwhahaha | wtf | 19:31 |
| cloudnull | yikes | 19:31 |
| *** Vorrtex has quit IRC | 19:33 | |
| sshnaidm | a lot of timeouts to repos.. | 19:36 |
| cloudnull | so its failing in the container prep process, not in the tool itself ? | 19:36 |
| mwhahaha | centos7.7? | 19:39 |
| cloudnull | ^ I would assume so. | 19:43 |
| cloudnull | with a mess of out of sync repos | 19:43 |
| *** pcaruana has joined #tripleo | 19:43 | |
| cloudnull | or just out-dated mirror list | 19:43 |
| *** Vorrtex has joined #tripleo | 19:44 | |
| *** florianf has quit IRC | 19:46 | |
| openstackgerrit | Kamil Sambor proposed openstack/tripleo-heat-templates master: Support connecting OVN DB over SSL https://review.opendev.org/680345 | 19:46 |
| *** holser has joined #tripleo | 19:48 | |
| cloudnull | mwhahaha weshay|rover sshnaidm is it possible for us to restrict the repos to a single, known good one, and try building from that? | 19:53 |
| weshay|rover | cloudnull, in the container update? | 19:54 |
| weshay|rover | the base repos should not be in play at except for a new dep | 19:55 |
| weshay|rover | arxcruz|rover, collision :( | 19:56 |
| weshay|rover | I'll fix | 19:56 |
| tbarron | weshay|rover: I raised https://bugzilla.redhat.com/show_bug.cgi?id=1754641 for the ceph-ansible issue that is impacting the RDO scenario004 run in https://review.opendev.org/#/c/682853 | 19:57 |
| openstack | bugzilla.redhat.com bug 1754641 in Ceph-Ansible "Failure to get cephfs_data_pool and cephfs_metadata_pool variable settings" [Unspecified,New] - Assigned to gabrioux | 19:57 |
| weshay|rover | tbarron++ | 19:58 |
| tbarron | weshay|rover: they may rightly ask me to raise an upstream ceph-ansible "issue" for same and I'll do that but this should get attention | 19:58 |
| weshay|rover | tbarron, /me wonders if there is something in internal gerrit that fixes that | 19:58 |
| tbarron | maybe | 19:58 |
| *** derekh has joined #tripleo | 20:01 | |
| *** derekh has quit IRC | 20:01 | |
| *** brault has quit IRC | 20:02 | |
| *** boazel has quit IRC | 20:10 | |
| mwhahaha | meh i just noticed that we're dumping ansible_facts in the container-image-prepare log | 20:11 |
| * mwhahaha goes to track down where to add no_log | 20:11 | |
| *** pcaruana has quit IRC | 20:12 | |
| weshay|rover | mwhahaha, https://review.opendev.org/#/c/683981/ fyi | 20:12 |
| *** jfrancoa has quit IRC | 20:12 | |
| mwhahaha | i was going to just silence it | 20:12 |
| mwhahaha | though it's likely that it's not needed at all | 20:13 |
| mwhahaha | because it's only taking 2seconds | 20:13 |
| *** brault has joined #tripleo | 20:18 | |
| openstackgerrit | Alex Schultz proposed openstack/tripleo-common master: Run container prepare without gathering facts https://review.opendev.org/683981 | 20:26 |
| mwhahaha | weshay|rover, -^ he's right we don't need it | 20:27 |
| weshay|rover | :) | 20:27 |
| tbarron | weshay|rover: i tried running the reproducer script with libvirt for https://review.opendev.org/#/c/682853 but it doesn't like me, if your team can do it on | 20:28 |
| tbarron | RDOCloud or whatever | 20:28 |
| tbarron | sopped before the overcloud deploy and let me use that then I could test a proposed fix for the ceph-ansible bug | 20:29 |
| weshay|rover | tbarron, http://tripleo.org/install/containers_deployment/standalone.html | 20:29 |
| cloudnull | weshay|rover mwhahaha - regarding what we write to tripleo-container-image-prepare.log, can we separate out our ansible vs python executable logs? in digging into the log file it looks like write everything to that one file? | 20:30 |
| mwhahaha | not really | 20:31 |
| mwhahaha | it's techincally all output from python | 20:31 |
| EmilienM | lbragstad: which version? stein and onward you want to use systemd commands to restart containers. | 20:31 |
| lbragstad | EmilienM ack - i was able to get it by restarting the overcloud containers | 20:32 |
| lbragstad | fwiw - i'm using stable/queens | 20:32 |
| lbragstad | i followed the basic installation instructions and i noticed the overcloud doesn't have caching enabled (despite memcached being available) | 20:33 |
| EmilienM | lbragstad: queens: docker restart | 20:33 |
| *** saneax has joined #tripleo | 20:33 | |
| *** gbarros has quit IRC | 20:35 | |
| *** gbarros has joined #tripleo | 20:36 | |
| lbragstad | EmilienM after updating keystone's memcached server and switching the cache_driver to use dogpile.cache.memcached i noticed about a 37% increase in performance using the overcloud | 20:38 |
| weshay|rover | EmilienM, rlandy let's chat for 5 if you have time | 20:41 |
| rlandy | sure | 20:41 |
| EmilienM | ok just fyi I have really low bandwidth, on sat connection | 20:42 |
| weshay|rover | ah.. on the bat phone | 20:42 |
| EmilienM | lbragstad: will look later | 20:42 |
| weshay|rover | rlandy, EmilienM is interested on how he can test quay change | 20:42 |
| weshay|rover | rlandy, can you please show him the molecule job | 20:43 |
| *** aakarsh has joined #tripleo | 20:43 | |
| *** jtomasek has quit IRC | 20:43 | |
| rlandy | weshay|rover: hmmm ... that one uses a local registry | 20:43 |
| rlandy | probably not the closest comparison now | 20:43 |
| lbragstad | EmilienM sounds good - i'm trying to figure if there are changes i need to make to tht to set those configuration options at deploy time | 20:44 |
| rlandy | weshay|rover: I would think that we need a zuul job due to secrets | 20:44 |
| openstackgerrit | Kevin Carter (cloudnull) proposed openstack/tripleo-common master: Write ansible log content to a different log file https://review.opendev.org/684131 | 20:49 |
| cloudnull | mwhahaha that looks like the only place we ingest ansible log content into LOG.info | 20:50 |
| mwhahaha | yes | 20:50 |
| *** ansmith_ has joined #tripleo | 20:51 | |
| cloudnull | my interest in separating out the logs is because we see things like this https://pasted.tech/pastes/025288e93b0ef7237a244e64fb33bd7275166f93 | 20:52 |
| cloudnull | 14:26:24 next line 14:15:51 | 20:52 |
| mwhahaha | just grep INFO/DEBUG? | 20:52 |
| cloudnull | thats what I ended up doing | 20:53 |
| cloudnull | grep timestamp, then sort | 20:53 |
| *** ansmith has quit IRC | 20:53 | |
| cloudnull | its cosmetic, but easier (for me) to grok. maybe not desirable? | 20:54 |
| mwhahaha | we want all for troubleshooting | 20:55 |
| mwhahaha | i get what you're getting at but we don't specify either/or | 20:55 |
| *** ansmith_ has quit IRC | 20:56 | |
| cloudnull | k | 20:56 |
| openstackgerrit | Alex Schultz proposed openstack/ansible-role-tripleo-modify-image master: Fix exists check https://review.opendev.org/684134 | 20:56 |
| mwhahaha | weshay|rover: sooo that container image prepare log that took ove ran ahour | 21:01 |
| mwhahaha | weshay|rover: http://mirror.bhs1.ovh.openstack.org/centos/7/updates/x86_64/repodata/fc5ccb8c18d581a699bcb5fb20b80b9885355894eb8540129ec6fb33c540f818-primary.sqlite.bz2: [Errno 12] Timeout on http://mirror.bhs1.ovh.openstack.org/centos/7/updates/x86_64/repodata/fc5ccb8c18d581a699bcb5fb20b80b9885355894eb8540129ec6fb33c540f818-primary.sqlite.bz2: (28, 'Operation too slow. Less than 1000 bytes/sec transferred the last | 21:01 |
| mwhahaha | 30 seconds') | 21:01 |
| mwhahaha | it retries 10x times | 21:02 |
| mwhahaha | at 30 seconds each | 21:02 |
| *** jcoufal has quit IRC | 21:04 | |
| *** Vorrtex has quit IRC | 21:04 | |
| mwhahaha | also | 21:05 |
| mwhahaha | 2019-09-23 14:22:23.507 16031 WARNING urllib3.connectionpool [ ] Retrying (Retry(total=7, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ProtocolError('Connection aborted.', BadStatusLine("''",))': /cloudflare/registry-v2/docker/registry/v2/blobs/sha256/25/2567b1d338c484424e836580f563a5b6e50978fa05ae69d727745fd1e58a38b8/data?verify=1569251543-1Cr7AUjSrCAOZUKz%2BotiDBD5YWk%3D | 21:05 |
| *** xek has quit IRC | 21:05 | |
| *** dmacpher has quit IRC | 21:07 | |
| *** dmacpher has joined #tripleo | 21:07 | |
| *** saneax has quit IRC | 21:09 | |
| openstackgerrit | Alex Schultz proposed openstack/ansible-role-tripleo-modify-image master: Fix exists check https://review.opendev.org/684134 | 21:11 |
| EmilienM | weshay|rover: hey so on quay.io, a few things | 21:19 |
| EmilienM | weshay|rover: it seems like I can't re-use the same token for all namespaces | 21:19 |
| EmilienM | e.g. tripleomaster | 21:19 |
| EmilienM | weshay|rover: I'm going to check if we can use the API to get temp token just for the curl command | 21:20 |
| EmilienM | or just use the credentials (login/pass) | 21:20 |
| weshay|rover | k | 21:21 |
| EmilienM | https://docs.quay.io/api/ | 21:21 |
| EmilienM | it seems like I can | 21:21 |
| weshay|rover | mwhahaha, aye.. crazy | 21:22 |
| *** tesseract has quit IRC | 21:23 | |
| weshay|rover | mwhahaha, /me working to get that container-prep-time in the dashboard for reelz.. can chat anytime | 21:23 |
| *** paramite has quit IRC | 21:25 | |
| mwhahaha | weshay|rover: we need to chat about launchpad bugs and stuff for ptl at some point | 21:25 |
| weshay|rover | rlandy, see EmilienM's text ^ | 21:25 |
| weshay|rover | the token you salted last week may not be valid for all the namespaces | 21:25 |
| weshay|rover | so . I think EmilienM is going to add to the patch to generate a token from the user/pass on the fly in the push playbook | 21:26 |
| weshay|rover | mwhahaha, aye we do | 21:26 |
| EmilienM | I'm trying to make it simple, one set of creds for everything | 21:26 |
| EmilienM | let me try | 21:26 |
| weshay|rover | rlandy, up here | 21:26 |
| rlandy | reading back | 21:26 |
| rlandy | either way we need the password salted | 21:27 |
| *** bdodd has quit IRC | 21:28 | |
| cloudnull | weshay|rover - https://review.opendev.org/#/c/683240/ - is being merged | 21:30 |
| weshay|rover | cloudnull, ah nice | 21:30 |
| weshay|rover | rlandy, EmilienM ok.. so now we'll have proxy servers | 21:31 |
| EmilienM | w00t | 21:31 |
| *** bfournie has quit IRC | 21:33 | |
| mwhahaha | weshay|rover: where's the magical dashboard that shows the number of jobs we run per day over time? | 21:36 |
| rlandy | weshay|rover: pls check https://review.rdoproject.org/r/#/c/22521/1/zuul.d/secrets.yaml - note now we have quay_secret and quay_secret_password - the first one should probably change names | 21:37 |
| weshay|rover | http://dashboard-ci.tripleo.org/d/YRJtmtNWk/cockpit?orgId=1 | 21:38 |
| weshay|rover | scroll down to "ci stats weekly check jobs" | 21:39 |
| weshay|rover | it's pretty light atm | 21:39 |
| *** rcernin has joined #tripleo | 21:39 | |
| *** rcernin has quit IRC | 21:40 | |
| *** rcernin has joined #tripleo | 21:40 | |
| *** cdearborn has quit IRC | 21:41 | |
| rlandy | we need a second change in config to pick up those as env vars | 21:44 |
| *** raildo has quit IRC | 21:45 | |
| *** mcornea has quit IRC | 21:45 | |
| *** chem has quit IRC | 21:49 | |
| *** aakarsh has quit IRC | 21:57 | |
| *** holser has quit IRC | 21:58 | |
| *** ansmith has joined #tripleo | 22:10 | |
| mwhahaha | weshay|rover: why are we running container image prepare twice? | 22:15 |
| mwhahaha | we're running it for the overcloud & undercloud | 22:15 |
| mwhahaha | thought that was the point of running it ont he undercloud | 22:15 |
| * mwhahaha forgets | 22:15 | |
| *** gbarros has quit IRC | 22:18 | |
| weshay|rover | mwhahaha, heh | 22:18 |
| weshay|rover | ya.. it runs twice.. we shouldn't but do | 22:19 |
| weshay|rover | because the cotainers for the overcloud may be different than the just the undercloud | 22:19 |
| * mwhahaha enjoys having the same conversations again | 22:19 | |
| weshay|rover | I've mentioned that in the past | 22:19 |
| mwhahaha | i thought we solved for that but i guess not | 22:19 |
| mwhahaha | i remember it not actually improving things | 22:19 |
| weshay|rover | mwhahaha, that would be a huge savings if we could just pull what we need once | 22:20 |
| mwhahaha | well at least if we started at the updated containers | 22:20 |
| mwhahaha | maybe it would help | 22:20 |
| mwhahaha | but since it's not the full list, it doesn't really help i guess | 22:21 |
| weshay|rover | man.. I hate grafana right now | 22:22 |
| weshay|rover | mwhahaha, couldn't we just take the hit and load all our containers | 22:22 |
| weshay|rover | ? | 22:22 |
| weshay|rover | for multinode jobs | 22:22 |
| weshay|rover | upfront | 22:23 |
| weshay|rover | ? | 22:23 |
| openstackgerrit | Merged openstack/tripleo-validations stable/stein: Add different logic when running against the Undercloud or the Controller https://review.opendev.org/682597 | 22:23 |
| mwhahaha | i don't think we have logic to handle that | 22:23 |
| mwhahaha | maybe we could look into an all option | 22:23 |
| mwhahaha | for mirroring purposes | 22:23 |
| *** rfolco has quit IRC | 22:29 | |
| openstackgerrit | Steve Baker proposed openstack/tripleo-heat-templates master: Override policy for undercloud novajoin https://review.opendev.org/684142 | 22:30 |
| weshay|rover | stevebaker, can we revisit what is pulled re: containers.. so we don't have to run it twice/ | 22:30 |
| weshay|rover | > | 22:30 |
| weshay|rover | ? | 22:31 |
| weshay|rover | there we go | 22:31 |
| mwhahaha | let me see what it would take to remove teh filtering | 22:31 |
| stevebaker | weshay|rover: existing images are skipped the second time, so most of the time for the second run is transferring images that only the overcloud needs. What is the issue, it is a time thing? | 22:32 |
| *** bfournie has joined #tripleo | 22:33 | |
| weshay|rover | stevebaker, aye | 22:33 |
| weshay|rover | time | 22:33 |
| mwhahaha | stevebaker: not sure that's true given the logs | 22:33 |
| *** ekultails has quit IRC | 22:33 | |
| mwhahaha | https://storage.gra1.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_4ef/683425/1/check/tripleo-ci-centos-7-containers-multinode/4ef875a/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz | 22:34 |
| mwhahaha | overcloud run took 20mins for container image prepare | 22:34 |
| stevebaker | mwhahaha: hmm, if it is re-transferring existing images then that is a bug. I put quite some effort into avoiding unnecessary transfers | 22:34 |
| mwhahaha | 21 mins for the undercloud | 22:34 |
| mwhahaha | https://storage.gra1.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_4ef/683425/1/check/tripleo-ci-centos-7-containers-multinode/4ef875a/logs/undercloud/home/zuul/undercloud_install.log.txt.gz | 22:35 |
| mwhahaha | it'll run the update 2x | 22:35 |
| mwhahaha | which may be the 20 mins | 22:35 |
| mwhahaha | the original containers might be there but we rerun the update process | 22:35 |
| mwhahaha | or maybe we're doing it wrong where we need to include the modifier param | 22:35 |
| mwhahaha | so it checks to see if the updated containers are there | 22:36 |
| mwhahaha | cause we currently don't provide update tag | 22:36 |
| stevebaker | mwhahaha: what is actually going on will be logged here https://storage.gra1.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_4ef/683425/1/check/tripleo-ci-centos-7-containers-multinode/4ef875a/logs/undercloud/var/log/tripleo-container-image-prepare.log.txt.gz | 22:36 |
| mwhahaha | yea we've all been staring at that for weeks now :D | 22:36 |
| stevebaker | lol | 22:36 |
| *** rfolco has joined #tripleo | 22:37 | |
| *** holser has joined #tripleo | 22:40 | |
| stevebaker | mwhahaha: the already-exists check is a little different for modify. Instead of asserting that the exact same image is in the local registry, it just checks that there is an image in the local registry with the expected tag (in this case that would be 41408527ae88ac42de11ca2818ff405320795a8b_9e598812-updated-20190923030645 | 22:40 |
| mwhahaha | yea looks like we're updating the haproxy container again and some stuff | 22:40 |
| mwhahaha | need to look closer at those logs | 22:40 |
| mwhahaha | i was poking at the undercloud job instead of the multinode job | 22:40 |
| mwhahaha | meh we don't have timestamps for that log | 22:41 |
| mwhahaha | not helpful | 22:41 |
| mwhahaha | well this is generally confusing because it's updating haproxy/keepalived for the overcloud | 22:43 |
| mwhahaha | anyway yes, it does seem that they are properly filters so it only updates the ones that don't exist | 22:44 |
| mwhahaha | so that's good | 22:44 |
| mwhahaha | this seems single threaded tho | 22:44 |
| stevebaker | yes some timestamps on /var/log/tripleo-container-image-prepare.log.txt.gz would be very helpful | 22:44 |
| mwhahaha | they exist for the undercloud jobs so i'm guessing we've miseed a flag somewhere | 22:45 |
| stevebaker | maybe quickstart does that piping through date thing | 22:46 |
| mwhahaha | you might be right | 22:46 |
| EmilienM | rlandy, weshay|rover: I sent tripleoci credentials for quay.io to Wes, can we land https://review.rdoproject.org/r/#/c/22445/ and add the credentials sent by email to the secrets, and see how the CI job works? | 22:46 |
| stevebaker | mwhahaha: the modify mechanism it meant to run concurrently, but that may have been turned down because of the podman storage locking bugs | 22:47 |
| mwhahaha | it looks like it's supposed to be in parallel | 22:47 |
| rlandy | if we only need those creds on the promoter server, we can add them right away | 22:47 |
| stevebaker | mwhahaha: or podman storage locking is causing only one thing to happen at a time | 22:47 |
| weshay|rover | EmilienM, we need to wait for rlandy | 22:47 |
| rlandy | weshay|rover: I am not sure we do | 22:48 |
| mwhahaha | stevebaker: i wouldn't be shocked if it's buildah/podman | 22:48 |
| rlandy | see my conversation on #oooq | 22:48 |
| EmilienM | rlandy: where else would they be needed? | 22:48 |
| stevebaker | mwhahaha: here we go, concurrency is related to CPU count https://opendev.org/openstack/tripleo-common/src/branch/master/tripleo_common/image/image_uploader.py#L1073 | 22:50 |
| stevebaker | wait no | 22:50 |
| mwhahaha | i see in the logs 3 "Running ansible-playbook command" items | 22:50 |
| mwhahaha | so it ntheory it should be multiprocess | 22:51 |
| stevebaker | the python uploader https://opendev.org/openstack/tripleo-common/src/branch/master/tripleo_common/image/image_uploader.py#L1914 | 22:51 |
| mwhahaha | might just be a side effect of the logging | 22:52 |
| mwhahaha | and it is actually doing multiples but we don't print the logs untilt he end | 22:52 |
| mwhahaha | weird the timestamp formatting is in the tripleo-container-image-prepare script but it didn't seem to take for that run | 22:55 |
| mwhahaha | oh we don't have debug tied to Debug | 22:56 |
| weshay|rover | dear lord.. that took way too much time to fix that stuff | 22:58 |
| * mwhahaha flips tables | 22:58 | |
| cloudnull | EmilienM and I worked on https://review.opendev.org/#/c/682731 which tried to address some of the worker issues, https://review.opendev.org/#/c/682731/10/tripleo_common/image/image_uploader.py@98 - though we were still seeing timeouts on that too | 23:01 |
| *** tkajinam has joined #tripleo | 23:01 | |
| EmilienM | mwhahaha: haven't we? I vaguely remember ContainerImagePrepareDebug | 23:02 |
| mwhahaha | EmilienM: we do, but it's not tied to Debug and we don't set it anywhere | 23:03 |
| EmilienM | cloudnull: lol I like your wording. I already told you it's your patch now ahah | 23:03 |
| EmilienM | i thought we did on the undercloud | 23:03 |
| cloudnull | EmilienM your still the author :) | 23:03 |
| openstackgerrit | Alex Schultz proposed openstack/tripleo-quickstart-extras master: Set ContainerImagePrepareDebug https://review.opendev.org/684144 | 23:03 |
| mwhahaha | EmilienM: we only do it on the undercloud, overcloud doesn't | 23:03 |
| EmilienM | https://opendev.org/openstack/python-tripleoclient/src/branch/master/tripleoclient/v1/undercloud_config.py#L405 | 23:03 |
| EmilienM | ah right | 23:03 |
| EmilienM | I guess we could have a condition in THT | 23:04 |
| mwhahaha | it's a bool, so no | 23:04 |
| EmilienM | where it first tries to get Debug, then ContainerImagePrepareDebug | 23:04 |
| mwhahaha | it doesn't fit the normal pattern | 23:04 |
| * mwhahaha blames you for that pattern | 23:04 | |
| EmilienM | if Debug False, then try to get ContainerImagePrepareDebug | 23:04 |
| cloudnull | pushups | 23:04 |
| mwhahaha | where <service>Debug is a string | 23:04 |
| mwhahaha | not a bool | 23:04 |
| mwhahaha | we generally don't want debug logs | 23:05 |
| mwhahaha | so it's really a ci setting | 23:05 |
| *** rh-jelabarre has quit IRC | 23:06 | |
| EmilienM | Debug is a boolean as well | 23:06 |
| EmilienM | what did you mean by "doesn't fit the normal pattern" | 23:07 |
| * EmilienM tries to understand what I did wrong | 23:07 | |
| mwhahaha | https://github.com/openstack/tripleo-heat-templates/blob/master/deployment/heat/heat-base-puppet.yaml#L7-L16 | 23:08 |
| mwhahaha | https://github.com/openstack/tripleo-heat-templates/blob/master/deployment/heat/heat-base-puppet.yaml#L133 | 23:08 |
| mwhahaha | https://github.com/openstack/tripleo-heat-templates/blob/master/deployment/heat/heat-base-puppet.yaml#L149-L153 | 23:08 |
| mwhahaha | vs | 23:08 |
| mwhahaha | https://github.com/openstack/tripleo-heat-templates/blob/master/deployment/container-image-prepare/container-image-prepare-baremetal-ansible.j2.yaml#L60-L63 | 23:08 |
| mwhahaha | so while we do have a ContainerImagePrepareDebug, it is not a string | 23:09 |
| mwhahaha | when you implemented the <service>Debug stuff, you made it a string so we could fall back to Debug | 23:09 |
| EmilienM | I got it now | 23:09 |
| mwhahaha | either way i think we should jsut fix it it in the container-image-prepare yaml anyway | 23:09 |
| mwhahaha | since we don't have timestamps for the udnercloud bits, that might also mean we're not running debug in containers-multinode for the undercloud | 23:10 |
| mwhahaha | hrm it's true, weird | 23:11 |
| EmilienM | http://codesearch.openstack.org/?q=undercloud_undercloud_debug&i=nope&files=&repos= | 23:11 |
| EmilienM | yeah it's true | 23:11 |
| mwhahaha | eh figure it out tomorrow | 23:12 |
| * mwhahaha wanders off | 23:12 | |
| *** holser has quit IRC | 23:13 | |
| *** d0ugal has quit IRC | 23:16 | |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-heat-templates master: Adapt ContainerImagePrepareDebug to the string pattern https://review.opendev.org/684145 | 23:18 |
| EmilienM | mwhahaha: ^ | 23:19 |
| EmilienM | have a good night | 23:19 |
| *** morazi has quit IRC | 23:20 | |
| *** morazi has joined #tripleo | 23:23 | |
| *** d0ugal has joined #tripleo | 23:29 | |
| openstackgerrit | Takashi Kajinami proposed openstack/tripleo-heat-templates master: DNM: Test libvirt logging https://review.opendev.org/683811 | 23:35 |
| openstackgerrit | Kevin Carter (cloudnull) proposed openstack/tripleo-common master: Relax retries, use default pool settings, and tune workers https://review.opendev.org/682731 | 23:47 |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!