| *** gbarros has quit IRC | 00:03 | |
| *** ansmith has quit IRC | 00:07 | |
| openstackgerrit | Takashi Kajinami proposed openstack/puppet-tripleo master: Disable keystone token_flush in default https://review.opendev.org/682512 | 00:39 |
|---|---|---|
| openstackgerrit | Takashi Kajinami proposed openstack/puppet-tripleo master: Disable keystone token_flush by default https://review.opendev.org/682512 | 00:40 |
| openstackgerrit | Takashi Kajinami proposed openstack/tripleo-heat-templates master: Don't run keystone_cron container if fernet token is used https://review.opendev.org/682265 | 00:41 |
| openstackgerrit | Sagi Shnaidman proposed openstack/tripleo-heat-templates master: Don't use pacemaker in ironic sa job https://review.opendev.org/682410 | 01:05 |
| openstackgerrit | Sagi Shnaidman proposed openstack/tripleo-ci master: Use podman for scenario012 https://review.opendev.org/682341 | 01:08 |
| openstackgerrit | Takashi Kajinami proposed openstack/puppet-tripleo master: Add basic unit tests for tripleo::profile::base::keystone https://review.opendev.org/682519 | 01:15 |
| *** mschuppert has quit IRC | 01:23 | |
| openstackgerrit | Sagi Shnaidman proposed openstack/tripleo-ansible master: WIP: POC of podman_container https://review.opendev.org/682458 | 01:29 |
| *** Goneri has quit IRC | 01:31 | |
| openstackgerrit | Sagi Shnaidman proposed openstack/tripleo-ansible master: Introduce new module for Paunch https://review.opendev.org/682093 | 01:38 |
| openstackgerrit | Takashi Kajinami proposed openstack/puppet-tripleo master: Add basic unit tests for tripleo::profile::base::keystone https://review.opendev.org/682519 | 01:48 |
| openstackgerrit | Emilien Macchi proposed openstack/paunch stable/rocky: Add --cpuset-cpus support https://review.opendev.org/682491 | 01:52 |
| *** weshay has quit IRC | 01:56 | |
| openstackgerrit | Emilien Macchi proposed openstack/python-tripleoclient master: Deprecate overcloud image prepare https://review.opendev.org/682404 | 01:56 |
| openstackgerrit | Emilien Macchi proposed openstack/python-tripleoclient master: Deprecate overcloud image prepare https://review.opendev.org/682404 | 01:57 |
| *** apetrich has quit IRC | 02:11 | |
| *** mmethot_ has quit IRC | 02:11 | |
| *** mmethot_ has joined #tripleo | 02:12 | |
| *** csmart has quit IRC | 02:19 | |
| *** gbarros has joined #tripleo | 02:19 | |
| *** csmart has joined #tripleo | 02:26 | |
| *** rfolco has quit IRC | 02:28 | |
| *** ykarel|away has joined #tripleo | 02:28 | |
| openstackgerrit | Takashi Kajinami proposed openstack/puppet-tripleo master: Add basic unit tests for tripleo::profile::base::keystone https://review.opendev.org/682519 | 02:35 |
| *** jschlueter has quit IRC | 02:50 | |
| *** rajinir has quit IRC | 02:50 | |
| *** jschlueter has joined #tripleo | 02:56 | |
| *** rajinir has joined #tripleo | 02:56 | |
| *** gbarros has quit IRC | 02:58 | |
| *** radeks has quit IRC | 03:00 | |
| *** jschlueter has quit IRC | 03:18 | |
| *** rajinir has quit IRC | 03:18 | |
| *** ricolin has joined #tripleo | 03:22 | |
| *** psachin has joined #tripleo | 03:22 | |
| openstackgerrit | Steve Baker proposed openstack/tripleo-common master: mistral-executor image install novajoin package https://review.opendev.org/682527 | 03:23 |
| *** jschlueter has joined #tripleo | 03:24 | |
| *** rajinir has joined #tripleo | 03:24 | |
| openstackgerrit | Takashi Kajinami proposed openstack/puppet-tripleo master: Add basic unit tests for tripleo::profile::base::keystone https://review.opendev.org/682519 | 03:32 |
| *** ramishra has joined #tripleo | 03:33 | |
| *** aakarsh|2 has quit IRC | 03:40 | |
| openstackgerrit | Steve Baker proposed openstack/tripleo-common master: mistral-executor image install novajoin package https://review.opendev.org/682527 | 03:41 |
| *** rh-jelabarre has quit IRC | 04:04 | |
| *** surpatil has joined #tripleo | 04:13 | |
| openstackgerrit | Takashi Kajinami proposed openstack/puppet-tripleo master: Add basic unit tests for tripleo::profile::base::keystone https://review.opendev.org/682519 | 04:15 |
| *** ratailor has joined #tripleo | 04:29 | |
| openstackgerrit | Rajesh Tailor proposed openstack/tripleo-puppet-elements stable/stein: Add proper permissions on script https://review.opendev.org/682534 | 04:32 |
| openstackgerrit | Rajesh Tailor proposed openstack/tripleo-puppet-elements stable/rocky: Add proper permissions on script https://review.opendev.org/682535 | 04:34 |
| openstackgerrit | Rajesh Tailor proposed openstack/tripleo-puppet-elements stable/rocky: Add proper permissions on script https://review.opendev.org/682535 | 04:34 |
| openstackgerrit | Rajesh Tailor proposed openstack/tripleo-puppet-elements stable/queens: Add proper permissions on script https://review.opendev.org/682537 | 04:35 |
| *** ykarel|away is now known as ykarel | 04:49 | |
| *** pdeore has joined #tripleo | 04:50 | |
| openstackgerrit | Jaganathan Palanisamy proposed openstack/tripleo-common stable/rocky: OvS DPDK derive parameters timeout issue https://review.opendev.org/682540 | 04:53 |
| openstackgerrit | Jaganathan Palanisamy proposed openstack/tripleo-common stable/queens: Derives NUMA aware vSwitches parameters https://review.opendev.org/682541 | 04:56 |
| *** udesale has joined #tripleo | 05:01 | |
| *** radeks has joined #tripleo | 05:01 | |
| *** radeks has quit IRC | 05:03 | |
| *** skramaja has joined #tripleo | 05:03 | |
| *** morazi has quit IRC | 05:12 | |
| *** soniya29 has joined #tripleo | 05:14 | |
| *** pcaruana has joined #tripleo | 05:16 | |
| *** marios has joined #tripleo | 05:16 | |
| *** ykarel has quit IRC | 05:18 | |
| *** dsneddon has quit IRC | 05:21 | |
| *** dsneddon has joined #tripleo | 05:28 | |
| *** pcaruana has quit IRC | 05:29 | |
| *** waleedm has joined #tripleo | 05:30 | |
| *** dsneddon has quit IRC | 05:34 | |
| *** sshnaidm|afk is now known as sshnaidm|pto | 05:35 | |
| *** holser has joined #tripleo | 05:37 | |
| *** dciabrin_ has joined #tripleo | 05:38 | |
| *** ykarel has joined #tripleo | 05:38 | |
| *** dciabrin has quit IRC | 05:41 | |
| *** pdeore has quit IRC | 05:45 | |
| *** Tengu has joined #tripleo | 05:53 | |
| *** sanjayu_ has joined #tripleo | 05:54 | |
| *** spsurya has joined #tripleo | 05:55 | |
| *** paramite has joined #tripleo | 05:58 | |
| *** sanjayu_ has quit IRC | 06:00 | |
| *** saneax has joined #tripleo | 06:01 | |
| *** lmiccini has joined #tripleo | 06:03 | |
| *** yprokule has joined #tripleo | 06:15 | |
| *** jtomasek has joined #tripleo | 06:23 | |
| *** egonzalez has joined #tripleo | 06:25 | |
| *** holser has quit IRC | 06:25 | |
| *** slaweq has joined #tripleo | 06:26 | |
| *** kopecmartin|off is now known as kopecmartin | 06:31 | |
| *** morazi has joined #tripleo | 06:45 | |
| *** artom has quit IRC | 06:56 | |
| *** xek has joined #tripleo | 06:57 | |
| *** artom has joined #tripleo | 07:03 | |
| *** xek has quit IRC | 07:08 | |
| *** jbadiapa has quit IRC | 07:08 | |
| *** ratailor has quit IRC | 07:21 | |
| *** ratailor has joined #tripleo | 07:22 | |
| openstackgerrit | Rabi Mishra proposed openstack/tripleo-common master: WIP Handle non-config-download deployment types https://review.opendev.org/680957 | 07:23 |
| *** jbadiapa has joined #tripleo | 07:24 | |
| *** jpich has joined #tripleo | 07:25 | |
| *** tosky has joined #tripleo | 07:27 | |
| openstackgerrit | Merged openstack/tripleo-common master: Activate health check script for qdrouterd image https://review.opendev.org/672969 | 07:27 |
| *** mschuppert has joined #tripleo | 07:27 | |
| *** rpittau|afk is now known as rpittau | 07:28 | |
| *** saneax has quit IRC | 07:28 | |
| *** cylopez has joined #tripleo | 07:33 | |
| *** apetrich has joined #tripleo | 07:40 | |
| *** slaweq has quit IRC | 07:43 | |
| *** pkopec has joined #tripleo | 07:45 | |
| *** jpena|off is now known as jpena | 07:46 | |
| *** rcernin has quit IRC | 07:47 | |
| *** ykarel is now known as ykarel|lunch | 07:48 | |
| *** iurygregory has joined #tripleo | 07:48 | |
| *** slaweq has joined #tripleo | 07:52 | |
| *** amoralej|off is now known as amoralej | 08:02 | |
| *** tkajinam has quit IRC | 08:03 | |
| *** yprokule has quit IRC | 08:04 | |
| *** iurygregory has quit IRC | 08:04 | |
| *** yprokule has joined #tripleo | 08:04 | |
| *** alexmcleod has joined #tripleo | 08:06 | |
| *** bogdando has joined #tripleo | 08:10 | |
| *** holser has joined #tripleo | 08:14 | |
| *** iurygregory has joined #tripleo | 08:16 | |
| *** pbandark has joined #tripleo | 08:20 | |
| *** derekh has joined #tripleo | 08:33 | |
| openstackgerrit | Bogdan Dobrelya (bogdando) proposed openstack/tripleo-quickstart master: Fix secondary (overcloud) subnode indexes https://review.opendev.org/682346 | 08:34 |
| openstackgerrit | Juan Badia Payno proposed openstack/tripleo-ansible master: Let ReaR uses different backup strategy https://review.opendev.org/682583 | 08:39 |
| *** gfidente has joined #tripleo | 08:44 | |
| openstackgerrit | Kamil Sambor proposed openstack/puppet-tripleo master: Correct wrong name in certmonger_certificate for ovn_controller https://review.opendev.org/682586 | 08:48 |
| *** slaweq has quit IRC | 08:52 | |
| *** florianf has joined #tripleo | 08:55 | |
| *** pkopec has quit IRC | 09:04 | |
| *** ykarel|lunch is now known as ykarel | 09:08 | |
| *** slaweq has joined #tripleo | 09:11 | |
| *** pcaruana has joined #tripleo | 09:19 | |
| *** ratailor has quit IRC | 09:23 | |
| *** ratailor has joined #tripleo | 09:24 | |
| *** saneax has joined #tripleo | 09:24 | |
| *** sanjayu_ has joined #tripleo | 09:25 | |
| *** saneax has quit IRC | 09:29 | |
| openstackgerrit | Saravanan KR proposed openstack/tripleo-ansible master: Add kernelargs and tuned core configuration https://review.opendev.org/679809 | 09:39 |
| *** cfontain has joined #tripleo | 09:43 | |
| *** cfontain has quit IRC | 09:43 | |
| openstackgerrit | Kamil Sambor proposed openstack/tripleo-heat-templates master: WIP: Support connecting OVN DB over SSL https://review.opendev.org/680345 | 09:49 |
| openstackgerrit | Gael Chamoulaud proposed openstack/tripleo-validations stable/stein: Add different logic when running against the Undercloud or the Controller https://review.opendev.org/682597 | 10:03 |
| *** ramishra_ has joined #tripleo | 10:07 | |
| *** pcaruana has quit IRC | 10:07 | |
| *** dtantsur|afk is now known as dtantsur | 10:07 | |
| *** ramishra has quit IRC | 10:08 | |
| *** ansmith has joined #tripleo | 10:12 | |
| *** surpatil has quit IRC | 10:13 | |
| *** waleedm_ has joined #tripleo | 10:20 | |
| *** waleedm has quit IRC | 10:23 | |
| *** ratailor has quit IRC | 10:25 | |
| *** ratailor has joined #tripleo | 10:26 | |
| *** pbandark has quit IRC | 10:26 | |
| *** udesale has quit IRC | 10:34 | |
| *** udesale has joined #tripleo | 10:34 | |
| *** egonzalez has quit IRC | 10:34 | |
| *** pbandark has joined #tripleo | 10:34 | |
| *** ansmith has quit IRC | 10:43 | |
| *** waleedm_ has quit IRC | 10:43 | |
| *** paramite has quit IRC | 10:53 | |
| *** paramite has joined #tripleo | 10:55 | |
| *** pcaruana has joined #tripleo | 10:55 | |
| *** lucasagomes has joined #tripleo | 11:08 | |
| openstackgerrit | Juan Badia Payno proposed openstack/tripleo-ansible master: Let ReaR uses different backup strategy https://review.opendev.org/682583 | 11:16 |
| *** jpena is now known as jpena|lunch | 11:17 | |
| *** amoralej is now known as amoralej|lunch | 11:17 | |
| openstackgerrit | Merged openstack/tripleo-common master: Ignore failure calling cell_v2_discover_hosts from deployment https://review.opendev.org/681944 | 11:19 |
| *** ratailor has quit IRC | 11:27 | |
| *** ratailor has joined #tripleo | 11:27 | |
| *** brault has joined #tripleo | 11:31 | |
| *** pierreprinetti has joined #tripleo | 11:33 | |
| *** weshay has joined #tripleo | 11:33 | |
| *** brault has quit IRC | 11:36 | |
| openstackgerrit | Merged openstack/tripleo-ansible master: Introduce new module for Paunch https://review.opendev.org/682093 | 11:36 |
| *** dsneddon has joined #tripleo | 11:37 | |
| *** panda is now known as panda|ruck | 11:41 | |
| *** dsneddon has quit IRC | 11:42 | |
| *** sanjayu_ has quit IRC | 11:44 | |
| *** raildo has joined #tripleo | 11:59 | |
| *** bfournie has quit IRC | 12:03 | |
| *** rfolco has joined #tripleo | 12:04 | |
| *** gfidente has quit IRC | 12:05 | |
| *** matbu has quit IRC | 12:12 | |
| *** rlandy has joined #tripleo | 12:18 | |
| *** matbu has joined #tripleo | 12:18 | |
| weshay | nkinder, latest error looks like Certmonger param | 12:20 |
| weshay | "puppet-user: Error: Could not find resource 'Certmonger_certificate[ovn_controller]' in parameter 'require' (file: /etc/puppet/modules/tripleo/manifests/certmonger/ovn_controller.pp, line: 64 | 12:20 |
| ksambor | weshay: fix is here https://review.opendev.org/#/c/682586/ | 12:21 |
| *** rh-jelabarre has joined #tripleo | 12:22 | |
| ksambor | and it is working | 12:22 |
| cloudnull | mornings | 12:25 |
| cloudnull | if folks have a moment, can I get a couple reviews on https://review.opendev.org/#/c/682057/ | 12:25 |
| *** soniya29 has quit IRC | 12:26 | |
| *** mcornea has joined #tripleo | 12:26 | |
| *** pcaruana has quit IRC | 12:27 | |
| *** openstackstatus has quit IRC | 12:28 | |
| *** openstack has joined #tripleo | 12:29 | |
| *** ChanServ sets mode: +o openstack | 12:29 | |
| jaosorior | weshay: this looks like the fix https://review.opendev.org/#/c/682586/1/manifests/certmonger/ovn_controller.pp | 12:29 |
| *** openstackstatus has joined #tripleo | 12:30 | |
| *** ChanServ sets mode: +v openstackstatus | 12:30 | |
| weshay | jaosorior, thanks! including that patch in the test :) | 12:31 |
| *** gfidente has joined #tripleo | 12:32 | |
| *** sanjayu_ has joined #tripleo | 12:33 | |
| openstackgerrit | Raildo Mascena proposed openstack/tripleo-quickstart-extras master: WIP: allow external ansible role for 3rd party job https://review.opendev.org/673926 | 12:34 |
| openstackgerrit | Raildo Mascena proposed openstack/tripleo-quickstart-extras master: Add keycloak-setup role https://review.opendev.org/671262 | 12:34 |
| openstackgerrit | Saravanan KR proposed openstack/tripleo-heat-templates master: Remove OpenDaylight templates and environments https://review.opendev.org/682630 | 12:35 |
| *** jpena|lunch is now known as jpena | 12:37 | |
| *** Goneri has joined #tripleo | 12:39 | |
| openstackgerrit | Raildo Mascena proposed openstack/tripleo-quickstart-extras master: WIP: allow external ansible role for 3rd party job https://review.opendev.org/673926 | 12:39 |
| openstackgerrit | Gael Chamoulaud proposed openstack/tripleo-validations master: [WIP] Add new container-healthcheck-status validation https://review.opendev.org/682377 | 12:39 |
| openstackgerrit | wes hayutin proposed openstack/tripleo-heat-templates master: WIP: Support connecting OVN DB over SSL https://review.opendev.org/680345 | 12:39 |
| *** ansmith has joined #tripleo | 12:41 | |
| openstackgerrit | Jens Harbott (frickler) proposed openstack/diskimage-builder master: Update gentoo test to current system profile https://review.opendev.org/682635 | 12:43 |
| *** ratailor has joined #tripleo | 12:43 | |
| *** amoralej|lunch is now known as amoralej | 12:44 | |
| openstackgerrit | Bogdan Dobrelya (bogdando) proposed openstack/tripleo-docs master: PDF documentation build https://review.opendev.org/682362 | 12:45 |
| openstackgerrit | Gael Chamoulaud proposed openstack/tripleo-validations master: [WIP] Add new healthcheck-service-status validation https://review.opendev.org/682377 | 12:53 |
| *** cfontain has joined #tripleo | 12:53 | |
| *** bfournie has joined #tripleo | 12:53 | |
| *** jcoufal has joined #tripleo | 12:55 | |
| *** redrobot has joined #tripleo | 12:55 | |
| openstackgerrit | Bogdan Dobrelya (bogdando) proposed openstack/tripleo-docs master: PDF documentation build https://review.opendev.org/682362 | 12:56 |
| panda|ruck | mmmhhh 20 minutes in containers-multinode to perform a single step | 12:58 |
| panda|ruck | 2019-09-17 10:00:53 | command: overcloud deploy -> tripleoclient.v1.overcloud_deploy.DeployOvercloud (auth=True) | 12:58 |
| panda|ruck | 2019-09-17 10:00:53 | Using auth plugin: password | 12:58 |
| panda|ruck | 2019-09-17 10:00:56 | No stack found, will be doing a stack create | 12:58 |
| panda|ruck | 2019-09-17 10:19:22 | Performing Heat stack create | 12:58 |
| panda|ruck | 2019-09-17 10:19:45 | Removing the current plan files | 12:58 |
| *** suuuper has joined #tripleo | 12:58 | |
| *** pcaruana has joined #tripleo | 12:59 | |
| EmilienM | panda|ruck: hey | 13:07 |
| EmilienM | panda|ruck: is tripleo-ci-centos-7-scenario009-multinode-oooq-container broken on stable/rocky? | 13:07 |
| EmilienM | https://zuul.opendev.org/t/openstack/build/8352d58fec904a0b9312839a2482d2d9 | 13:07 |
| *** ratailor has quit IRC | 13:08 | |
| EmilienM | mandre: ^ ping | 13:08 |
| EmilienM | TASK [openshift_node : Install node, clients, and conntrack packages] | 13:08 |
| EmilienM | it's failing on that | 13:08 |
| *** mcornea has quit IRC | 13:09 | |
| *** paramite|clone has joined #tripleo | 13:09 | |
| mandre | EmilienM: it looks like mirror issue, https://de339b72b688803b5a43-80f76e47f83bff14c764cd8b2b7f1a08.ssl.cf2.rackcdn.com/682506/1/check/tripleo-ci-centos-7-scenario009-multinode-oooq-container/8352d58/logs/undercloud/var/lib/mistral/config-download-latest/openshift/playbook.log.txt.gz | 13:11 |
| openstackgerrit | Emilien Macchi proposed openstack/paunch stable/rocky: Add --cpuset-cpus support https://review.opendev.org/682491 | 13:12 |
| *** paramite has quit IRC | 13:12 | |
| openstackgerrit | Carlos Goncalves proposed openstack/tripleo-heat-templates master: Add new parameter options to Octavia service https://review.opendev.org/682641 | 13:13 |
| *** lmiccini has quit IRC | 13:13 | |
| EmilienM | mandre: right but still failing after 2 recheck :/ | 13:13 |
| EmilienM | trying again | 13:13 |
| *** dsneddon has joined #tripleo | 13:13 | |
| EmilienM | mandre: thx for looking. Just to confirm the job is supposed to be stable | 13:13 |
| panda|ruck | it has failed quite a bit on the last few days, but every time looks like temporary failure | 13:14 |
| EmilienM | panda|ruck: do we know why the gate is so backed up? | 13:14 |
| panda|ruck | EmilienM: conteiners-multinode jobs are timing out during deployy and resetting the queue | 13:15 |
| panda|ruck | EmilienM: there are a few places that seems to take more time than usual during deploy | 13:16 |
| EmilienM | that's not good | 13:16 |
| EmilienM | where? | 13:16 |
| panda|ruck | EmilienM: one I pasted above | 13:16 |
| panda|ruck | EmilienM: the other I've seen is 20 minutes aourn containers prep, but I'm checking if it's consistent | 13:16 |
| mandre | EmilienM: it was green on https://review.opendev.org/#/c/681782/2 a few days ago | 13:17 |
| openstackgerrit | Martin Schuppert proposed openstack/tripleo-heat-templates stable/queens: Revert "Move nova cell v2 discovery to deploy_steps_tasks" https://review.opendev.org/682644 | 13:18 |
| EmilienM | mandre: thanks for confirming | 13:19 |
| weshay | we need to fix the ara reports upstream... broken since the log server change | 13:20 |
| * weshay interested in finding out what is taking longer than usual | 13:20 | |
| panda|ruck | not consistent, I see another 10 minutes on that step, and a gap of 15 minutes around TASK [check if libvirt is installed] | 13:21 |
| *** aakarsh|2 has joined #tripleo | 13:21 | |
| *** ade_lee_ has joined #tripleo | 13:21 | |
| *** gbarros has joined #tripleo | 13:21 | |
| EmilienM | gchamoul: great start on https://review.opendev.org/#/c/682377/ - I commented | 13:21 |
| gchamoul | EmilienM: thanks but yes it's still not perfect | 13:22 |
| *** mcornea has joined #tripleo | 13:23 | |
| *** ade_lee has quit IRC | 13:24 | |
| *** gbarros has quit IRC | 13:27 | |
| *** pcaruana has quit IRC | 13:28 | |
| weshay | tripleo-ci community call is happening now | 13:31 |
| *** cdearborn has joined #tripleo | 13:38 | |
| *** psachin has quit IRC | 13:43 | |
| openstackgerrit | Emilien Macchi proposed openstack/paunch master: Cleanup useless warnings for podman https://review.opendev.org/682651 | 13:46 |
| *** sanjayu_ has quit IRC | 13:48 | |
| *** hberaud has joined #tripleo | 13:54 | |
| *** xek has joined #tripleo | 13:56 | |
| *** ykarel is now known as ykarel|afk | 13:56 | |
| *** pcaruana has joined #tripleo | 13:59 | |
| mwhahaha | #startmeeting tripleo | 14:00 |
| mwhahaha | #topic agenda | 14:00 |
| mwhahaha | * Review past action items | 14:00 |
| mwhahaha | * One off agenda items | 14:00 |
| mwhahaha | * Squad status | 14:00 |
| mwhahaha | * Bugs & Blueprints | 14:00 |
| mwhahaha | * Projects releases or stable backports | 14:00 |
| mwhahaha | * Specs | 14:00 |
| mwhahaha | * open discussion | 14:00 |
| mwhahaha | Anyone can use the #link, #action and #info commands, not just the moderatorǃ | 14:00 |
| mwhahaha | Hi everyone! who is around today? | 14:00 |
| openstack | Meeting started Tue Sep 17 14:00:13 2019 UTC and is due to finish in 60 minutes. The chair is mwhahaha. Information about MeetBot at http://wiki.debian.org/MeetBot. | 14:00 |
| openstack | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 14:00 |
| *** openstack changes topic to " (Meeting topic: tripleo)" | 14:00 | |
| weshay | o/ | 14:00 |
| openstack | The meeting name has been set to 'tripleo' | 14:00 |
| *** openstack changes topic to "agenda (Meeting topic: tripleo)" | 14:00 | |
| dpeacock | o/ | 14:00 |
| marios | hello | 14:01 |
| ksambor | o/ | 14:01 |
| *** cfontain has quit IRC | 14:02 | |
| beagles | o/ | 14:02 |
| fultonj | o/ | 14:02 |
| *** migi has joined #tripleo | 14:03 | |
| openstackgerrit | Kevin Carter (cloudnull) proposed openstack/tripleo-ansible master: Add test to find broken symlinks https://review.opendev.org/682419 | 14:04 |
| cloudnull | o/ | 14:04 |
| mwhahaha | alright let's get going | 14:04 |
| mwhahaha | #topic review past action items | 14:04 |
| mwhahaha | mwhahaha to provide weshay with launchpad overview & scripts - NOT DONE | 14:04 |
| mwhahaha | #mwhahaha sync with weshay on ptl things | 14:04 |
| migi | rfolco: hey, so let's go back to the nested virt question. Is it enabled in tripleO jobs in upstream. I know it's relaying on what's provided, but is there kvm enabled one or it's not really executed | 14:04 |
| *** openstack changes topic to "review past action items (Meeting topic: tripleo)" | 14:04 | |
| *** suuuper has quit IRC | 14:04 | |
| rfolco | migi, tripleo mtg here, lets jump to #oooq if not in agenda for this mtg | 14:05 |
| mwhahaha | so i'll try and steal some time from wehsay this week | 14:05 |
| mwhahaha | that's it on the past action items | 14:05 |
| migi | ack | 14:05 |
| weshay | mwhahaha, I can buy some beers near you if needed :) | 14:05 |
| mwhahaha | :D | 14:06 |
| chandankumar | \o/ | 14:06 |
| mwhahaha | #topic one off agenda items | 14:06 |
| mwhahaha | #link https://etherpad.openstack.org/p/tripleo-meeting-items | 14:06 |
| *** openstack changes topic to "one off agenda items (Meeting topic: tripleo)" | 14:06 | |
| mwhahaha | (weshay) centos 8 scheduled to released on sept 24 https://twitter.com/CentOSProject/status/1173652996305170432 | 14:06 |
| mwhahaha | \o/ | 14:06 |
| weshay | aye.. zbr will be working w/ infra folks on the nodepool / dib updates | 14:07 |
| fultonj | ++ | 14:07 |
| mwhahaha | so there's a question about centos8 support in the upstream | 14:08 |
| mwhahaha | does the tripleo project need to be concerned about centos-8 support upstream for ussuri? | 14:08 |
| mwhahaha | https://review.opendev.org/#/c/681242 | 14:08 |
| cloudnull | woot! | 14:08 |
| mwhahaha | it's likely that we'll just have to carry any support burden | 14:08 |
| weshay | aye.. zbr ^ | 14:08 |
| mwhahaha | we'll definately support it, but the question is the impact on the wider services | 14:08 |
| weshay | k. .thanks.. just double checking etc | 14:09 |
| *** rpittau is now known as rpittau|afk | 14:10 | |
| weshay | one unrelated thing.. train ml3 release has a +1 finally https://review.opendev.org/#/c/681897/ | 14:10 |
| mwhahaha | any other comments on this item? | 14:10 |
| weshay | nothing else on centos-8 from me | 14:11 |
| mwhahaha | k thanks | 14:11 |
| mwhahaha | #topic Squad status | 14:11 |
| mwhahaha | ci | 14:11 |
| mwhahaha | #link https://etherpad.openstack.org/p/tripleo-ci-squad-meeting | 14:11 |
| mwhahaha | upgrade | 14:11 |
| mwhahaha | #link https://etherpad.openstack.org/p/tripleo-upgrade-squad-status | 14:11 |
| mwhahaha | edge | 14:11 |
| mwhahaha | #link https://etherpad.openstack.org/p/tripleo-edge-squad-status | 14:11 |
| mwhahaha | integration | 14:11 |
| *** openstack changes topic to "Squad status (Meeting topic: tripleo)" | 14:11 | |
| mwhahaha | #link https://etherpad.openstack.org/p/tripleo-integration-squad-status | 14:11 |
| mwhahaha | validations | 14:11 |
| mwhahaha | #link https://etherpad.openstack.org/p/tripleo-validations-squad-status | 14:11 |
| mwhahaha | networking | 14:11 |
| mwhahaha | #link https://etherpad.openstack.org/p/tripleo-networking-squad-status | 14:11 |
| mwhahaha | transformation | 14:11 |
| mwhahaha | #link https://etherpad.openstack.org/p/tripleo-ansible-agenda | 14:11 |
| mwhahaha | any status related highlights? | 14:11 |
| weshay | 3rd party rhel 8 jobs are done for now.. ended up w/ ovb fs001, standalone, standalone001/002; 003/004 are blocked and escalated | 14:13 |
| weshay | that will complete the rhel8 work for now | 14:13 |
| openstackgerrit | Tom Barron proposed openstack/tripleo-heat-templates master: DNM - exploring running scenario004 w/o pacemaker for manila https://review.opendev.org/679365 | 14:14 |
| openstackgerrit | Brent Eagles proposed openstack/tripleo-common stable/queens: Allow distribution of non-autogenerated certs https://review.opendev.org/682659 | 14:16 |
| mwhahaha | anything else? | 14:17 |
| mwhahaha | sounds like nope, moving on | 14:18 |
| mwhahaha | #topic bugs & blueprints | 14:18 |
| mwhahaha | #link https://launchpad.net/tripleo/+milestone/train-rc1 | 14:18 |
| mwhahaha | For Train we currently have 27 blueprints and 506 (-3) open Launchpad Bugs. 14 train-3, 5 train-rc1, 487 ussuri-1. 166 (+0) open Storyboard bugs. | 14:18 |
| mwhahaha | #link https://storyboard.openstack.org/#!/project_group/76 | 14:18 |
| mwhahaha | #action mwhahaha to close out train-m3 | 14:18 |
| *** openstack changes topic to "bugs & blueprints (Meeting topic: tripleo)" | 14:18 | |
| openstackgerrit | Natal Ngétal proposed openstack/tripleo-upgrade master: Remove the free-form. https://review.opendev.org/663539 | 14:18 |
| mwhahaha | once we merge the release change, i'll close out the milestone and move everything forward | 14:19 |
| mwhahaha | any comments/concerns/etc? | 14:19 |
| openstackgerrit | Brent Eagles proposed openstack/tripleo-heat-templates stable/queens: Only generate Octavia certs on stack create https://review.opendev.org/682660 | 14:20 |
| mwhahaha | #topic projects releases or stable backports | 14:21 |
| *** openstack changes topic to "projects releases or stable backports (Meeting topic: tripleo)" | 14:21 | |
| mwhahaha | we'll cut m3 soonish, so we'll be pushing for rc1 | 14:21 |
| openstackgerrit | Bogdan Dobrelya (bogdando) proposed openstack/tripleo-docs master: PDF documentation build https://review.opendev.org/682362 | 14:21 |
| mwhahaha | any comments on backports/releases? | 14:22 |
| *** lbragstad_ has joined #tripleo | 14:26 | |
| mwhahaha | #topic specs | 14:26 |
| mwhahaha | #link https://review.openstack.org/#/q/project:openstack/tripleo-specs+status:open | 14:26 |
| *** openstack changes topic to "specs (Meeting topic: tripleo)" | 14:26 | |
| mwhahaha | please update any openspecs to target to ussuri | 14:27 |
| mwhahaha | any one have any specific specs they want to talk about? | 14:27 |
| *** Vorrtex has joined #tripleo | 14:27 | |
| *** ykarel|afk is now known as ykarel | 14:28 | |
| mwhahaha | #topic open discussion | 14:28 |
| *** lbragstad has quit IRC | 14:28 | |
| *** openstack changes topic to "open discussion (Meeting topic: tripleo)" | 14:28 | |
| mwhahaha | anything else? | 14:28 |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-ansible master: paunch: add support for 'cleanup' https://review.opendev.org/682662 | 14:34 |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-ansible master: paunch: add support for 'cleanup' https://review.opendev.org/682662 | 14:37 |
| openstackgerrit | Bogdan Dobrelya (bogdando) proposed openstack/tripleo-docs master: PDF documentation build https://review.opendev.org/682362 | 14:38 |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-heat-templates master: nova-libvirt: set 'cpuset_cpus' to 'all' https://review.opendev.org/682665 | 14:41 |
| mwhahaha | thanks everyone | 14:42 |
| mwhahaha | #endmeeting | 14:42 |
| *** openstack changes topic to "CI Status: GREENish RDOCloud Status: MEHish | community irc meeting Tues@1400 UTC - tripleo-ci-community meeting Tues@1330 UTC | https://docs.openstack.org/tripleo-docs/latest/" | 14:42 | |
| openstack | Meeting ended Tue Sep 17 14:42:02 2019 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 14:42 |
| openstack | Minutes: http://eavesdrop.openstack.org/meetings/tripleo/2019/tripleo.2019-09-17-14.00.html | 14:42 |
| openstack | Minutes (text): http://eavesdrop.openstack.org/meetings/tripleo/2019/tripleo.2019-09-17-14.00.txt | 14:42 |
| openstack | Log: http://eavesdrop.openstack.org/meetings/tripleo/2019/tripleo.2019-09-17-14.00.log.html | 14:42 |
| *** gbarros has joined #tripleo | 14:42 | |
| *** dtantsur is now known as dtantsur|afk | 14:52 | |
| *** pcaruana has quit IRC | 14:55 | |
| *** hjensas is now known as hjensas|afk | 14:59 | |
| *** yprokule has quit IRC | 15:05 | |
| *** ykarel is now known as ykarel|away | 15:08 | |
| *** udesale has quit IRC | 15:08 | |
| *** Vorrtex has quit IRC | 15:09 | |
| openstackgerrit | Sergii Golovatiuk proposed openstack/tripleo-heat-templates master: Replace include_role with import_role https://review.opendev.org/682672 | 15:10 |
| *** iurygregory has quit IRC | 15:11 | |
| openstackgerrit | Sergii Golovatiuk proposed openstack/tripleo-upgrade stable/stein: Explicitly make a list https://review.opendev.org/682674 | 15:15 |
| openstackgerrit | Bogdan Dobrelya (bogdando) proposed openstack/tripleo-docs master: PDF documentation build https://review.opendev.org/682362 | 15:21 |
| *** gbarros has quit IRC | 15:21 | |
| *** ade_lee_ has quit IRC | 15:22 | |
| *** ade_lee_ has joined #tripleo | 15:22 | |
| *** cylopez has quit IRC | 15:23 | |
| *** lbragstad_ is now known as lbragstad | 15:23 | |
| weshay | mwhahaha, we may want to flip gate status to orange/red | 15:26 |
| *** mwhahaha changes topic to "CI Status: REDish RDOCloud Status: MEHish | community irc meeting Tues@1400 UTC - tripleo-ci-community meeting Tues@1330 UTC | https://docs.openstack.org/tripleo-docs/latest/" | 15:27 | |
| mwhahaha | what did you break? | 15:27 |
| cloudnull | rutroh | 15:29 |
| cloudnull | pushups ! | 15:29 |
| weshay | mwhahaha, as far as we can tell.. we're hitting timeouts | 15:29 |
| weshay | trying now to get our ara reports back | 15:30 |
| mwhahaha | did we get 7.7'd | 15:30 |
| weshay | hrm.. /me checks | 15:31 |
| weshay | it did release | 15:33 |
| weshay | https://blog.centos.org/2019/09/release-for-centos-linux-7-1908-on-the-x86_64architecture/ | 15:33 |
| *** hberaud is now known as hberaud|school-r | 15:33 | |
| weshay | panda|ruck, fyi ^ | 15:33 |
| mwhahaha | so if it's the updating, we need 7.7 containers | 15:34 |
| *** cfontain has joined #tripleo | 15:34 | |
| weshay | ya... ok.. we'll focus on promotion of master then.. | 15:34 |
| weshay | containers are 3 days old | 15:34 |
| * weshay looks at update log | 15:34 | |
| *** aakarsh|2 is now known as aakarsh | 15:38 | |
| weshay | http://zuul.openstack.org/builds?job_name=tripleo-ci-centos-7-containers-multinode&result=TIMED_OUT | 15:41 |
| weshay | not much worse today.. yet | 15:41 |
| *** jcoufal has quit IRC | 15:42 | |
| *** hberaud|school-r is now known as hberaud | 15:42 | |
| openstackgerrit | James Slagle proposed openstack/tripleo-common stable/queens: Raise an error if deployment_name cannot be determined https://review.opendev.org/682067 | 15:42 |
| *** jcoufal has joined #tripleo | 15:43 | |
| weshay | hrm.. I don't think the centos base updates for 7.7 have hit the openstack mirrors yet | 15:45 |
| weshay | http://mirror.regionone.fortnebula.opendev.org/centos/7/os/x86_64/Packages/NetworkManager-1.12.0-6.el7.x86_64.rpm | 15:45 |
| weshay | http://mirror.regionone.fortnebula.opendev.org/centos/7/cr/x86_64/Packages/NetworkManager-1.18.0-5.el7.x86_64.rpm | 15:45 |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-ansible master: paunch: add support for 'cleanup' https://review.opendev.org/682662 | 15:45 |
| openstackgerrit | wes hayutin proposed openstack/tripleo-quickstart master: [DNM] Test Centos 7.7 with CR repos https://review.opendev.org/618832 | 15:45 |
| *** ykarel_ has joined #tripleo | 15:48 | |
| *** ykarel|away has quit IRC | 15:50 | |
| *** ykarel_ has quit IRC | 15:54 | |
| weshay | centos-release is still at 7.6 in upstream https://openstack.fortnebula.com:13808/v1/AUTH_e8fd161dc34c421a979a9e6421f823e9/zuul_opendev_logs_42c/682276/1/gate/tripleo-ci-centos-7-containers-multinode-stein/42c709b/logs/undercloud/etc/centos-release.txt.gz | 16:00 |
| openstackgerrit | Gabriele Cerami proposed openstack/tripleo-ci master: Reenable ARA html generation from collect logs https://review.opendev.org/682679 | 16:02 |
| openstackgerrit | Carlos Goncalves proposed openstack/tripleo-heat-templates master: Add new parameter options to Octavia service https://review.opendev.org/682641 | 16:03 |
| *** marios is now known as marios|out | 16:03 | |
| *** morazi has quit IRC | 16:07 | |
| *** cfontain has quit IRC | 16:08 | |
| cgoncalves | bogdando, thanks for the quick reviews! | 16:10 |
| *** bogdando has quit IRC | 16:13 | |
| *** marios|out has quit IRC | 16:15 | |
| EmilienM | weshay: AFIK cloud images aren't ready yet | 16:19 |
| EmilienM | not sure if we pull them | 16:19 |
| EmilienM | or build them from source | 16:19 |
| weshay | probably tonight | 16:19 |
| *** rlandy has quit IRC | 16:30 | |
| *** gfidente has quit IRC | 16:33 | |
| *** jpich has quit IRC | 16:33 | |
| *** rlandy has joined #tripleo | 16:35 | |
| *** derekh has quit IRC | 16:41 | |
| *** iurygregory has joined #tripleo | 16:42 | |
| openstackgerrit | James Slagle proposed openstack/tripleo-common master: Warn when looking up grandparent resource name https://review.opendev.org/682694 | 16:43 |
| openstackgerrit | James Slagle proposed openstack/tripleo-common master: Warn when looking up grandparent resource name https://review.opendev.org/682694 | 16:44 |
| *** hberaud is now known as hberaud|gone | 16:49 | |
| *** rfolco is now known as rfolco|dentist | 16:51 | |
| *** brault has joined #tripleo | 16:52 | |
| *** pcaruana has joined #tripleo | 16:55 | |
| *** jpena is now known as jpena|off | 16:59 | |
| openstackgerrit | Brent Eagles proposed openstack/tripleo-ci master: Change featurset for octavia standalone to support full tempest https://review.opendev.org/669740 | 17:00 |
| *** amoralej is now known as amoralej|off | 17:00 | |
| openstackgerrit | Saravanan KR proposed openstack/tripleo-heat-templates master: Remove OpenDaylight templates and environments https://review.opendev.org/682630 | 17:04 |
| openstackgerrit | Nathan Kinder proposed openstack/tripleo-heat-templates master: WIP: Support connecting OVN DB over SSL https://review.opendev.org/680345 | 17:05 |
| *** skramaja has quit IRC | 17:05 | |
| *** kopecmartin is now known as kopecmartin|off | 17:06 | |
| *** ramishra_ has quit IRC | 17:08 | |
| *** brault has quit IRC | 17:10 | |
| *** iurygregory has quit IRC | 17:12 | |
| openstackgerrit | Emilien Macchi proposed openstack/python-tripleoclient stable/stein: Replace warn with warning method https://review.opendev.org/682702 | 17:24 |
| EmilienM | holser: ^ doing backports for you | 17:24 |
| EmilienM | saw the warning on osp15 | 17:24 |
| holser | EmilienM thanks a lot | 17:25 |
| EmilienM | np | 17:29 |
| openstackgerrit | Kevin Carter (cloudnull) proposed openstack/tripleo-ansible master: Add test to find broken symlinks https://review.opendev.org/682419 | 17:29 |
| *** gfidente has joined #tripleo | 17:29 | |
| cloudnull | anyone around to do a couple reviews on https://review.opendev.org/#/c/682057/ | 17:30 |
| dpaterson | cloudnull, will take a look | 17:32 |
| cloudnull | thanks dpaterson! | 17:33 |
| *** gfidente is now known as gfidente|afk | 17:36 | |
| zbr | weshay: i was able to get 7.7 after doing a clean all | 17:39 |
| weshay | re: libselinux? | 17:40 |
| zbr | weshay: i bet not, no replies on https://bugs.centos.org/view.php?id=16389 | 17:41 |
| zbr | if it will happen, we will first see some replies on that ticket. | 17:41 |
| zbr | weshay: it may be even more problematic. if ansible decides to prefer python3 for some reason (internal or from us), it will fail. | 17:42 |
| weshay | zbr, if that's the case.. I don't think upstream infra would move to 7.7 | 17:43 |
| weshay | bah.. but they would have to for updates | 17:43 |
| zbr | afaik ansible picks version based on os version, so we should be safe. | 17:43 |
| weshay | kind of hate how centos works | 17:43 |
| mwhahaha | actually it uses /usr/lib/system-python or something now | 17:43 |
| mwhahaha | which exists as of 7.6 | 17:43 |
| zbr | you are not alone, memo is quite ho these days | 17:44 |
| zbr | hot | 17:44 |
| openstackgerrit | Alex Schultz proposed openstack/tripleo-common master: Split template override files https://review.opendev.org/682455 | 17:46 |
| *** mmethot_ has quit IRC | 17:53 | |
| *** mmethot_ has joined #tripleo | 17:53 | |
| openstackgerrit | Brent Eagles proposed openstack/tripleo-heat-templates master: Add Octavia driver agent service https://review.opendev.org/658118 | 17:53 |
| openstackgerrit | Kevin Carter (cloudnull) proposed openstack/tripleo-heat-templates master: [WIP] Convert the container-puppet to ansible https://review.opendev.org/661883 | 17:56 |
| *** pbandark has quit IRC | 17:56 | |
| *** pbandark has joined #tripleo | 17:56 | |
| *** alexmcleod has quit IRC | 17:57 | |
| *** pbandark has quit IRC | 18:01 | |
| zbr | weshay: we are "safe" with 7.7, ansible still uses py27 regardless if python3 is installed. | 18:08 |
| zbr | other than this upgrade went nice and quick, but I was already using "cr" repos. | 18:09 |
| weshay | ok.. good, thanks for checking that out | 18:09 |
| weshay | hrm.. jobs timing out in tempest | 18:09 |
| weshay | the queque is going to get a little crazy | 18:10 |
| weshay | ha. .the queue that is | 18:11 |
| mwhahaha | queueueueueue | 18:11 |
| *** florianf has quit IRC | 18:15 | |
| EmilienM | 21h32min | 18:15 |
| EmilienM | great | 18:15 |
| EmilienM | is there any patch which needs to land to stabilize the situation? | 18:16 |
| EmilienM | if yes, then we can reset the gate | 18:16 |
| weshay | EmilienM, I don't have anything for you re: a common pattern.. seems to be just slow.. panda|ruck is getting our ara reports for the overcloud now | 18:17 |
| weshay | it's not 7.7 | 18:17 |
| weshay | it's not 1 cloud | 18:17 |
| weshay | and tomorrow is probably going to be worse | 18:18 |
| weshay | ovb jobs are completing more quickly than containers-multinode | 18:19 |
| weshay | tripleo-ci-centos-7-ovb-1ctlr_1comp-featureset001SUCCESS in 2h 38m 05s | 18:19 |
| EmilienM | weshay: can you give me a link of a job that timeouted? | 18:20 |
| weshay | tripleo-ci-centos-7-containers-multinodeSUCCESS in 3h 16m 57s | 18:20 |
| EmilienM | i'll look at it | 18:20 |
| weshay | EmilienM, your patch just timed out in tempest.. but others are here http://dashboard-ci.tripleo.org/d/YRJtmtNWk/cockpit?orgId=1&fullscreen&panelId=61 | 18:20 |
| EmilienM | weshay: "my" patch? which one lol | 18:21 |
| weshay | nevermind.. urs is still going | 18:21 |
| weshay | top patch in the queue | 18:21 |
| weshay | that's going to merge | 18:21 |
| weshay | http://zuul.openstack.org/builds?job_name=tripleo-ci-centos-7-containers-multinode&result=TIMED_OUT | 18:23 |
| EmilienM | tripleo-ci-centos-7-containers-multinode took 2h38 2 w ago | 18:24 |
| EmilienM | now it takes more than 3h | 18:24 |
| EmilienM | let's take a look | 18:24 |
| openstackgerrit | Merged openstack/tripleo-specs master: fix the spelling mistakes https://review.opendev.org/681717 | 18:26 |
| EmilienM | tripleo-container-image-prepare seems slower (at first look) | 18:27 |
| EmilienM | which can be caused by: more containers, more rpms, mirror issue | 18:27 |
| * EmilienM digs deeper | 18:27 | |
| weshay | ya.. compared to stein.. major time diff http://zuul.openstack.org/builds?job_name=tripleo-ci-centos-7-containers-multinode-stein | 18:27 |
| EmilienM | i'm looking at stein jobs btw | 18:27 |
| EmilienM | not even master | 18:27 |
| weshay | EmilienM, there are only about 10 rpms getting patched in a few jobs I looked at.. | 18:28 |
| EmilienM | btw, do we see rocky/jobs timeouting? if yes: likely due to some infra issue (networking, mirror, etc) | 18:28 |
| EmilienM | rocky/queens sorry ^ | 18:28 |
| *** ricolin has quit IRC | 18:28 | |
| weshay | there are a few stein jobs timing out http://zuul.openstack.org/builds?job_name=tripleo-ci-centos-7-containers-multinode-stein&result=TIMED_OUT | 18:29 |
| EmilienM | i can't find it in cockpit | 18:29 |
| EmilienM | can you figure out if rocky/queens has timeouts too? | 18:29 |
| weshay | the branch is listed on the left | 18:29 |
| weshay | sure | 18:29 |
| * weshay looks | 18:29 | |
| EmilienM | if no, then our issue is maybe in tripleo itself. Still digging | 18:30 |
| *** holser has quit IRC | 18:30 | |
| *** Vorrtex has joined #tripleo | 18:30 | |
| weshay | nothing since june on rocky http://zuul.openstack.org/builds?job_name=tripleo-ci-centos-7-containers-multinode-rocky&result=TIMED_OUT | 18:31 |
| EmilienM | ok thanks | 18:31 |
| EmilienM | give me a few, i'm digging now | 18:31 |
| weshay | nothing in sept. from queens http://zuul.openstack.org/builds?job_name=tripleo-ci-centos-7-containers-multinode-queens&result=TIMED_OUT | 18:31 |
| EmilienM | btw i already see it being podman/buildah thing but I won't spoil | 18:32 |
| weshay | this is good info http://dashboard-ci.tripleo.org/d/si1tipHZk/jobs-exploration?orgId=1&fullscreen&panelId=7 | 18:33 |
| weshay | lolz | 18:33 |
| weshay | EmilienM, how can you tell? | 18:33 |
| EmilienM | can you add the branch on that link? | 18:34 |
| EmilienM | weshay: how can I tell? containers are the root cause of many of our problems. If you need examples, let me find some ;-) | 18:34 |
| weshay | lol | 18:34 |
| weshay | 2019-09-10tripleo-ci-centos-7-containers-multinode-stein931 week87% | 18:35 |
| weshay | 2019-09-10tripleo-ci-centos-7-containers-multinode-rocky931 week92% | 18:35 |
| weshay | 2019-09-10tripleo-ci-centos-7-containers-multinode-queens | 18:35 |
| weshay | so.. I'll try get these split by branch too.. but there are branchful jobs there to compare to | 18:35 |
| weshay | would be nice to add the average run time | 18:35 |
| *** brault has joined #tripleo | 18:42 | |
| weshay | man.. is the bot on here? | 18:42 |
| EmilienM | I think I found it | 18:50 |
| EmilienM | and it's clearly during container image prepare | 18:50 |
| *** xek_ has joined #tripleo | 18:54 | |
| EmilienM | the job used to take 2h10 1 month ago :D | 18:55 |
| *** xek has quit IRC | 18:57 | |
| EmilienM | look on this one: https://review.opendev.org/#/c/676218/ | 19:01 |
| EmilienM | 2h12 :D | 19:01 |
| * EmilienM continues to dig logs | 19:01 | |
| *** lucasagomes has quit IRC | 19:01 | |
| EmilienM | I'm not sure why we spit all the facts in https://openstack.fortnebula.com:13808/v1/AUTH_e8fd161dc34c421a979a9e6421f823e9/zuul_opendev_logs_42c/682276/1/gate/tripleo-ci-centos-7-containers-multinode-stein/42c709b/logs/undercloud/var/log/tripleo-container-image-prepare.log.txt.gz | 19:08 |
| *** pierreprinetti has quit IRC | 19:09 | |
| *** pierreprinetti has joined #tripleo | 19:09 | |
| EmilienM | i suspect https://review.opendev.org/#/c/676387/ | 19:13 |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-common master: Revert "Close the http sessions of registry on image prepare" https://review.opendev.org/682717 | 19:13 |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-common master: Revert "Close the http sessions of registry on image prepare" https://review.opendev.org/682717 | 19:14 |
| EmilienM | just testing now | 19:14 |
| EmilienM | I looked at Zuul and all tripleo-ci-centos-7-containers-multinode jobs increased by 30 min | 19:15 |
| EmilienM | (easily) | 19:15 |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-common stable/stein: Revert "Close the http sessions of registry on image prepare" https://review.opendev.org/682719 | 19:16 |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-common stable/stein: Revert "Close the http sessions of registry on image prepare" https://review.opendev.org/682719 | 19:16 |
| *** brault has quit IRC | 19:16 | |
| EmilienM | so the patch that I'm reverting landed on Aug 21th | 19:19 |
| EmilienM | go on https://review.opendev.org/#/q/status:merged+tripleo+age:3week | 19:19 |
| EmilienM | and look at a patch which landed before 21th | 19:19 |
| EmilienM | https://review.opendev.org/#/c/664168/ | 19:19 |
| EmilienM | tripleo-ci-centos-7-containers-multinode SUCCESS in 2h 15m 05s | 19:19 |
| EmilienM | now let's look at let's say 24th | 19:20 |
| EmilienM | https://review.opendev.org/#/c/677923/ | 19:20 |
| EmilienM | tripleo-ci-centos-7-containers-multinode SUCCESS in 2h 48m 09s | 19:20 |
| EmilienM | mwhahaha: ^ | 19:20 |
| EmilienM | in the middle, we have the patch i'm reverting | 19:20 |
| EmilienM | (i confirmed with a bunch of other patches) | 19:20 |
| mwhahaha | i'm uncertain but we can revert in the mean time | 19:21 |
| EmilienM | why? | 19:21 |
| EmilienM | because we'll re-open the bug that he closed? | 19:21 |
| EmilienM | probably | 19:21 |
| *** brault has joined #tripleo | 19:26 | |
| EmilienM | I think we need to reset the gate | 19:28 |
| EmilienM | we have identified the problem | 19:28 |
| EmilienM | we should not send any patch into gate from now | 19:28 |
| EmilienM | until we have a solution, revert or not revert (fix) | 19:29 |
| EmilienM | that is merged | 19:29 |
| EmilienM | (and backported to stein) | 19:29 |
| *** brault has quit IRC | 19:32 | |
| *** mmethot_ has quit IRC | 19:37 | |
| *** mmethot_ has joined #tripleo | 19:37 | |
| *** mmethot_ has quit IRC | 19:40 | |
| *** mmethot_ has joined #tripleo | 19:41 | |
| weshay | cool w/ me | 19:41 |
| openstackgerrit | Martin Mágr proposed openstack/puppet-tripleo stable/queens: Collectd QDR connection https://review.opendev.org/643596 | 19:41 |
| weshay | guess I need to know how to do that soon | 19:41 |
| EmilienM | actually | 19:42 |
| EmilienM | https://review.opendev.org/#/c/674919/ is suspect as well | 19:42 |
| EmilienM | weshay: I just read logs and zuul results for that job | 19:43 |
| EmilienM | and bisected | 19:43 |
| EmilienM | first i figured that it was fast early august and slow end of august | 19:43 |
| EmilienM | then i looked at undercloud and figured out it was during container image prepare | 19:44 |
| EmilienM | then i looked at code changes in tripleo-common, regarding container images around mid August | 19:44 |
| EmilienM | and found these 2 patches | 19:44 |
| *** jcoufal has quit IRC | 19:45 | |
| * weshay almost done w/ this call | 19:45 | |
| weshay | back | 19:46 |
| weshay | ok | 19:46 |
| EmilienM | Saravanan's patch landed before cloudnull's one | 19:46 |
| EmilienM | but it took 2h30 when it landed, which isn't that bad | 19:47 |
| EmilienM | while kevin's patch took 2h50 | 19:47 |
| EmilienM | https://review.opendev.org/#/c/674919/ | 19:47 |
| EmilienM | I think that's kevin's patch in fact | 19:47 |
| weshay | same patch is on stein right? re.. the close | 19:47 |
| EmilienM | yeah he backported it | 19:47 |
| *** spsurya has quit IRC | 19:48 | |
| EmilienM | cloudnull: you around? | 19:48 |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-common master: Revert "Log exceptions when checking status" https://review.opendev.org/682729 | 19:48 |
| EmilienM | of course it needs to be rebased :/ | 19:48 |
| *** holser has joined #tripleo | 19:49 | |
| EmilienM | I'm proposing a revert until we patch it, just to confirm numbers | 19:49 |
| weshay | we landed this.. https://review.opendev.org/#/c/674919/ when we were getting all those ovh errors I think | 19:49 |
| EmilienM | and we didn't realize the perf issue | 19:49 |
| EmilienM | I have to hard stop in 15min or so, will be back later | 19:49 |
| weshay | k | 19:50 |
| weshay | EmilienM, you going to kill the gate? | 19:51 |
| EmilienM | we have to | 19:51 |
| EmilienM | like I suggested | 19:51 |
| weshay | agree | 19:51 |
| EmilienM | ok let me do it... | 19:51 |
| EmilienM | I was rebasing the revert on master | 19:51 |
| EmilienM | then I'll reset the gate | 19:51 |
| EmilienM | weshay: please send email | 19:51 |
| EmilienM | core: no +A plz | 19:51 |
| weshay | seen a handful of rpm download failures today too | 19:52 |
| EmilienM | not that harmful vs this timeout imho | 19:52 |
| weshay | https://3b726b2d725bb6994141-b19ca10c1c9573b87287c7426b5b2015.ssl.cf5.rackcdn.com/681171/1/gate/tripleo-ci-centos-7-containers-multinode/f2b7e5b/logs/undercloud/home/zuul/install-undercloud.log.txt.gz | 19:52 |
| weshay | EmilienM, totally.. timeouts are the worst | 19:52 |
| weshay | EmilienM, things had been merging on a fast clip .. pretty much the whole cycle until this hit | 19:54 |
| EmilienM | my hope is cloudnull to be ready for the pull ups | 19:55 |
| weshay | ha ha.. he mentioned pushups earlier | 19:55 |
| * weshay posts on his twitter ;) | 19:55 | |
| EmilienM | not sure where he is now lol | 19:55 |
| weshay | EmilienM, but.. to be fair.. that ovh was killer, turned out not to be us/tripleo but some good debug came out of that | 19:56 |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-common master: Revert "Log exceptions when checking status" https://review.opendev.org/682729 | 19:56 |
| weshay | EmilienM, I think tomorrow is going to be shit too.. if all the proxies are taking on new 7.7 rpms | 19:56 |
| weshay | in that patch did we add more retries? | 19:56 |
| * weshay looks | 19:56 | |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-common stable/stein: Revert "Log exceptions when checking status" https://review.opendev.org/682730 | 19:57 |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-common stable/stein: Revert "Log exceptions when checking status" https://review.opendev.org/682730 | 19:57 |
| EmilienM | ok reverts proposed & rebased | 19:57 |
| weshay | EmilienM, thank you! | 19:58 |
| weshay | I don't see retries in that patch | 19:58 |
| weshay | just logging | 19:58 |
| EmilienM | but: | 19:59 |
| EmilienM | adapter = HTTPAdapter( | 19:59 |
| EmilienM | max_retries=8, | 19:59 |
| EmilienM | pool_connections=24, | 19:59 |
| EmilienM | pool_maxsize=24, | 19:59 |
| EmilienM | pool_block=False | 19:59 |
| EmilienM | ) | 19:59 |
| EmilienM | pool_connections=24 might be too much | 19:59 |
| EmilienM | not sure how he picked these numbers | 19:59 |
| weshay | kill the gate :) | 20:00 |
| EmilienM | these params are useful for multi-thread but i'm unsure we use it correctly later | 20:00 |
| EmilienM | let's try 5 | 20:02 |
| EmilienM | weshay: do we have LP? | 20:04 |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-common master: image_uploader: relax HTTPAdapter pool_connections and pool_maxsize https://review.opendev.org/682731 | 20:04 |
| * weshay looks | 20:04 | |
| EmilienM | weshay: potential fix ^ | 20:04 |
| weshay | no | 20:05 |
| * weshay opens | 20:05 | |
| weshay | https://bugs.launchpad.net/tripleo/+bug/1844446 | 20:08 |
| openstack | Launchpad bug 1844446 in tripleo "multiple tripleo jobs timing out upstream causing gate resets in train" [Critical,Triaged] | 20:08 |
| cloudnull | back | 20:08 |
| weshay | EmilienM, ^ | 20:08 |
| * cloudnull wonders what he missed | 20:08 | |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-common master: image_uploader: relax HTTPAdapter pool_connections and pool_maxsize https://review.opendev.org/682731 | 20:08 |
| EmilienM | ok patch updated | 20:08 |
| weshay | cloudnull, no good deed goes unpunished | 20:09 |
| EmilienM | but again unsure it's the exact root cause. The only thing I can tell is cloudnull's patch causing the timeouts | 20:09 |
| weshay | cloudnull, the patches you put up for ovh container debug may have degraded perf | 20:09 |
| * cloudnull reading | 20:09 | |
| cloudnull | the connection pool ? | 20:09 |
| weshay | EmilienM, go kill the gate | 20:10 |
| cloudnull | is it only in one region again ? | 20:10 |
| weshay | before you have to go | 20:10 |
| weshay | cloudnull, no.. perf regression is all regions | 20:10 |
| EmilienM | yes I'm doing it now! | 20:10 |
| weshay | first gate kill of train | 20:10 |
| EmilienM | weshay: send email | 20:10 |
| weshay | ok | 20:10 |
| *** jtomasek has quit IRC | 20:11 | |
| cloudnull | EmilienM did https://review.opendev.org/#/c/682731/ fix the performance ? | 20:12 |
| weshay | he just put it up.. we should see that job complete in roughly 2.5 horus | 20:12 |
| weshay | hours | 20:12 |
| weshay | cloudnull, we were about to send you a tweet.. PUSH UPS / PULL UPS | 20:12 |
| weshay | :) | 20:12 |
| cloudnull | hahaha | 20:12 |
| EmilienM | cloudnull: I don't know if it does | 20:14 |
| EmilienM | I literally spent 10 min investigating that patch so not sure | 20:14 |
| EmilienM | now I need to take off for 1h | 20:14 |
| EmilienM | weshay: change topic please | 20:14 |
| EmilienM | no more +A | 20:14 |
| EmilienM | gate RED | 20:14 |
| * weshay needs ops | 20:15 | |
| weshay | ops me cause I'm lame | 20:15 |
| EmilienM | I can't | 20:15 |
| EmilienM | mwhahaha please ops me and weshay when you come back | 20:15 |
| EmilienM | gate is clear | 20:15 |
| * EmilienM afk 1h | 20:15 | |
| weshay | BOOM | 20:15 |
| weshay | EmilienM, thank you sir!! | 20:16 |
| *** mwhahaha sets mode: +o EmilienM | 20:16 | |
| *** mwhahaha sets mode: +o weshay | 20:17 | |
| weshay | heh.. cloudnull check it out http://dashboard-ci.tripleo.org/d/YRJtmtNWk/cockpit?orgId=1&fullscreen&panelId=398 | 20:17 |
| weshay | thanks mwhahaha | 20:18 |
| *** weshay changes topic to "CI Status: RED ( no rechecks, wait for https://review.opendev.org/#/c/682729/ ) | community irc meeting Tues@1400 UTC - tripleo-ci-community meeting Tues@1330 UTC | https://docs.openstack.org/tripleo-docs/latest/" | 20:19 | |
| cloudnull | weshay looking at some of the timeout logs, there are LOTS of retries, however, it seems like those bits are working, albeit super slow: | 20:26 |
| cloudnull | https://storage.bhs1.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_a32/682265/3/check/tripleo-ci-centos-7-containers-multinode/a321f1d/logs/undercloud/var/log/tripleo-container-image-prepare.log.txt.gz | 20:26 |
| cloudnull | https://openstack.fortnebula.com:13808/v1/AUTH_e8fd161dc34c421a979a9e6421f823e9/zuul_opendev_logs_d19/679558/4/check/tripleo-ci-centos-7-containers-multinode/d19d73a/logs/undercloud/var/log/tripleo-container-image-prepare.log.txt.gz | 20:26 |
| cloudnull | interestingly enough the job that keeps timing out is the "multinode" job | 20:26 |
| cloudnull | the rest seem to complete in ~2.5 hours | 20:27 |
| *** holser has quit IRC | 20:29 | |
| *** panda|ruck is now known as panda|ruck|off | 20:29 | |
| weshay | well.. that's one of the few multinode jobs that deploy most of the services | 20:30 |
| *** ansmith has quit IRC | 20:30 | |
| weshay | cloudnull, ^ | 20:30 |
| weshay | tripleo-ci-centos-7-scenario000-multinode-oooq-container-updatesSUCCESS in 2h 49m 47s | 20:31 |
| weshay | tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgradesSUCCESS in 2h 35m 44s (non-voting) | 20:31 |
| weshay | are keystone only | 20:31 |
| weshay | and still takes that long .. lolz | 20:31 |
| weshay | everything else is one node | 20:31 |
| *** weshay is now known as weshay|ruck | 20:32 | |
| weshay|ruck | rhel 8 standalone running in 1 hour 11min on that patch | 20:35 |
| cloudnull | weshay|ruck on https://review.opendev.org/#/c/682731/ ? | 20:40 |
| weshay|ruck | cloudnull, https://review.opendev.org/#/c/682265/ | 20:40 |
| *** bdodd_ has joined #tripleo | 20:41 | |
| cloudnull | yeah , everything but that job seems to complete in time | 20:41 |
| weshay|ruck | cloudnull, but it makes sense | 20:41 |
| weshay|ruck | the others are either single node or only deploying a few services | 20:41 |
| weshay|ruck | tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001SUCCESS in 2h 56m 11s | 20:41 |
| weshay|ruck | is the same number of nodes, and running introspection | 20:42 |
| weshay|ruck | but I think rdo has more horse power than some of the providers now | 20:42 |
| *** bdodd has quit IRC | 20:42 | |
| weshay|ruck | at least in our benchmarking | 20:42 |
| weshay|ruck | cloudnull, make more sense now? | 20:43 |
| cloudnull | ++ | 20:44 |
| cloudnull | what was out average run-time of those jobs before the timeouts started ? | 20:45 |
| cloudnull | were they always on the edge ? | 20:45 |
| weshay|ruck | we can look at stein | 20:46 |
| *** Goneri has quit IRC | 20:46 | |
| weshay|ruck | cloudnull, http://zuul.openstack.org/builds?job_name=tripleo-ci-centos-7-containers-multinode-stein&result=TIMED_OUT | 20:46 |
| weshay|ruck | cloudnull, was trying to get the same info | 20:47 |
| weshay|ruck | going to try and add it to http://dashboard-ci.tripleo.org/d/si1tipHZk/jobs-exploration?orgId=1&fullscreen&panelId=7 | 20:47 |
| *** brault has joined #tripleo | 20:48 | |
| *** mmethot_ has quit IRC | 20:48 | |
| *** mmethot has joined #tripleo | 20:49 | |
| cloudnull | looks like that job in stein job, when it completed finished in ~9-11k seconds (2.5-3) hours. | 20:50 |
| cloudnull | in master it looks like the average is 10-12k seconds (2.7-3.5) hours | 20:51 |
| cloudnull | when it succeeds | 20:51 |
| weshay|ruck | aye.. agree | 20:51 |
| *** cdearborn has quit IRC | 20:52 | |
| cloudnull | do we have profiling for the various components ? | 20:52 |
| weshay|ruck | cloudnull, we didn't have that many timeout.. just enough to kill the queue http://dashboard-ci.tripleo.org/d/YRJtmtNWk/cockpit?orgId=1&fullscreen&panelId=61 | 20:52 |
| cloudnull | can we compare where slowdowns are from stein to master? | 20:52 |
| *** brault has quit IRC | 20:52 | |
| weshay|ruck | cloudnull, we had ara for the overcloud.. but infra changed the log servers and blew it up | 20:52 |
| weshay|ruck | cloudnull, I tried to get the perf team's jobs upstream.. but they dissappeared | 20:53 |
| *** gfidente|afk has quit IRC | 20:53 | |
| *** mmethot has quit IRC | 20:53 | |
| weshay|ruck | cloudnull, https://browbeatproject.org/ | 20:53 |
| *** mmethot has joined #tripleo | 20:54 | |
| *** mmethot has quit IRC | 20:55 | |
| *** mmethot has joined #tripleo | 20:55 | |
| *** brault has joined #tripleo | 20:57 | |
| cloudnull | http://dashboard-ci.tripleo.org/d/YRJtmtNWk/cockpit?orgId=1&from=now-7d&to=now&fullscreen&panelId=164 | 20:58 |
| cloudnull | from looking at that, the timout ratio is low | 20:58 |
| *** pcaruana has quit IRC | 20:58 | |
| nkinder | weshay|ruck, I got a bit further on the fs039 failure for https://review.opendev.org/#/c/680345 | 20:59 |
| *** gbarros has joined #tripleo | 20:59 | |
| nkinder | weshay|ruck, it is getting certs now via certmonger for all of the OVN stuff, but the deploy is breaking elsewhere now | 20:59 |
| nkinder | weshay|ruck, I'm having trouble finding any actionable errors in the log, but my theory is that there is potentially a cert trust issue or something with the OVN components | 21:00 |
| weshay|ruck | nkinder, k | 21:00 |
| weshay|ruck | had a chat w/ your guys about a standalone w/ an ipa container.. sounds like they thought it was a good idea | 21:01 |
| nkinder | weshay|ruck, do we grab the logs from the containers on overcloud nodes? I don't see a /var/log/containers in zuul for any of the overcloud nodes | 21:01 |
| weshay|ruck | we should have them.. although I saw a patch about that a few days ago | 21:01 |
| * weshay|ruck looks | 21:01 | |
| *** brault has quit IRC | 21:01 | |
| nkinder | weshay|ruck, here is the run I'm looking at - https://logs.rdoproject.org/45/680345/21/openstack-check/tripleo-ci-centos-7-ovb-3ctlr_1comp_1supp-featureset039/2b3c233/logs/overcloud-controller-0/var/log/ | 21:02 |
| weshay|ruck | ya.. they are not there :( | 21:03 |
| * weshay|ruck opens a bug | 21:03 | |
| weshay|ruck | fun day | 21:03 |
| nkinder | weshay|ruck, I've been pitching the standalone idea for TLS with my team as well | 21:04 |
| weshay|ruck | :) | 21:05 |
| weshay|ruck | nkinder, I see the logs upstream, just not third party | 21:05 |
| weshay|ruck | https://b4fb9c87d3423cbe5a0c-8e40407516a8c23d7cf04400cc86fdf3.ssl.cf2.rackcdn.com/682586/1/check/tripleo-ci-centos-7-containers-multinode/ba2b3b1/logs/subnode-2/var/log/containers/ | 21:05 |
| weshay|ruck | nkinder, https://bugs.launchpad.net/tripleo/+bug/1844454 | 21:06 |
| openstack | Launchpad bug 1844454 in tripleo "overcloud controller containers are missing from ci logs" [Critical,Triaged] | 21:06 |
| * weshay|ruck looks at the collect code | 21:07 | |
| nkinder | weshay|ruck, I do see /var/log/containers in the logs for the OC nodes here - https://logs.rdoproject.org/45/680345/21/openstack-check/tripleo-ci-centos-7-ovb-3ctlr_1comp_1supp-featureset039/2b3c233/logs/quickstart_collect_logs.log | 21:07 |
| *** slaweq has quit IRC | 21:10 | |
| *** slaweq has joined #tripleo | 21:11 | |
| weshay|ruck | nkinder, looks like it was just an error .. I see container logs on previous runs http://logs.rdoproject.org/72/21672/2/check/tripleo-ci-centos-7-ovb-1ctlr_2comp_1supp-featureset039/478d25c/logs/overcloud-controller-0/var/log/containers/ | 21:11 |
| *** raildo has quit IRC | 21:12 | |
| nkinder | weshay|ruck, I can try a recheck | 21:12 |
| *** gbarros has quit IRC | 21:12 | |
| *** mmethot has quit IRC | 21:13 | |
| *** mmethot has joined #tripleo | 21:14 | |
| *** ansmith has joined #tripleo | 21:15 | |
| *** slaweq has quit IRC | 21:16 | |
| *** Vorrtex has quit IRC | 21:21 | |
| *** rfolco|dentist is now known as rfolco | 21:23 | |
| *** gbarros has joined #tripleo | 21:23 | |
| *** ade_lee_ has quit IRC | 21:28 | |
| *** bfournie has quit IRC | 21:30 | |
| *** slaweq has joined #tripleo | 21:32 | |
| *** slaweq has quit IRC | 21:36 | |
| weshay|ruck | cloudnull, https://snapshot.raintank.io/dashboard/snapshot/iZgzUjP3T29q5bhGrYlzyLSNEY4yeXVf | 21:39 |
| *** cfontain has joined #tripleo | 21:45 | |
| cloudnull | looks like the data starts sep 7 | 21:47 |
| weshay|ruck | EmilienM, cloudnull, sshnaidm|pto https://review.rdoproject.org/r/#/c/22339/ | 21:47 |
| weshay|ruck | cloudnull, dev environment | 21:47 |
| cloudnull | ah | 21:48 |
| weshay|ruck | https://github.com/rdo-infra/ci-config/tree/master/ci-scripts/infra-setup/roles/rrcockpit | 21:48 |
| sshnaidm|pto | weshay|ruck, total time is more important to know where we waste resources, not time of one job | 21:49 |
| weshay|ruck | sshnaidm|pto, ok.. you are supposed to be gone :) but I'll update it so we have total time too | 21:49 |
| weshay|ruck | sshnaidm|pto, ok? | 21:50 |
| sshnaidm|pto | weshay|ruck, let's add one mean or median, no need to have both or max | 21:50 |
| weshay|ruck | sshnaidm|pto, throw me a bone :) | 21:50 |
| *** gbarros has quit IRC | 21:51 | |
| * weshay|ruck adds total time | 21:51 | |
| weshay|ruck | for the record, sshnaidm|pto never likes my ideas | 21:51 |
| weshay|ruck | lolz | 21:51 |
| sshnaidm|pto | why? I like them! | 21:51 |
| sshnaidm|pto | partially | 21:51 |
| weshay|ruck | :) | 21:52 |
| weshay|ruck | they like me.. they really really like me | 21:52 |
| weshay|ruck | https://images.app.goo.gl/bYrVKLR6jVdKeQGL9 | 21:52 |
| weshay|ruck | sshnaidm|pto, there is a moving average, but I couldn't get it to work | 21:53 |
| weshay|ruck | bbl | 21:54 |
| weshay|ruck | kids | 21:54 |
| *** aakarsh has quit IRC | 21:54 | |
| EmilienM | sometimes I wonder if we should build our own docker registry and host it | 22:01 |
| EmilienM | rather than relying on docker.io | 22:02 |
| EmilienM | many issues around docker.io we have seen are workarounded with retries, which of course makes deployments longer | 22:02 |
| cloudnull | EmilienM couldn't we use redhat.registery.io ? | 22:04 |
| cloudnull | s/redhat.registery.com/g | 22:04 |
| cloudnull | I did get infra to create a proxy for us, so that should be available to us ? | 22:05 |
| EmilienM | we would need push on a namespace | 22:06 |
| EmilienM | and ensure they have enough storage for us | 22:06 |
| EmilienM | we have a ton of data | 22:06 |
| cloudnull | something like https://registry.access.redhat.com/tripleo/ | 22:07 |
| * cloudnull doesn't think that exists | 22:07 | |
| cloudnull | but I guess it could ? | 22:07 |
| EmilienM | I'm clearing the check quueue as well for tripleo jobs | 22:10 |
| cloudnull | EmilienM I wonder if we could the folks that run https://registry.access.redhat.com to mirror https://hub.docker.com/u/tripleomaster/ ? | 22:11 |
| EmilienM | ¯\_(ツ)_/¯ | 22:13 |
| EmilienM | maybe? | 22:13 |
| *** mcornea has quit IRC | 22:14 | |
| *** xek_ has quit IRC | 22:15 | |
| EmilienM | mwhahaha: do you remember how the proxy thing works | 22:18 |
| EmilienM | when we look at https://openstack.fortnebula.com:13808/v1/AUTH_e8fd161dc34c421a979a9e6421f823e9/zuul_opendev_logs_42c/682276/1/gate/tripleo-ci-centos-7-containers-multinode-stein/42c709b/logs/undercloud/home/zuul/containers-prepare-parameter.yaml.txt.gz | 22:18 |
| EmilienM | it doesn't seem we're using any proxy | 22:19 |
| mwhahaha | it uses the python proxy | 22:19 |
| EmilienM | I remember with docker it was in the docker registry config | 22:19 |
| EmilienM | right, but are we doing it really? | 22:19 |
| mwhahaha | it also uses the one out of the podman settings | 22:19 |
| mwhahaha | (i think) | 22:19 |
| EmilienM | if you look at logs: | 22:19 |
| mwhahaha | gimme a few and i'll try and find it | 22:19 |
| EmilienM | https://openstack.fortnebula.com:13808/v1/AUTH_e8fd161dc34c421a979a9e6421f823e9/zuul_opendev_logs_42c/682276/1/gate/tripleo-ci-centos-7-containers-multinode-stein/42c709b/logs/undercloud/var/log/tripleo-container-image-prepare.log.txt.gz | 22:19 |
| EmilienM | you can see a bunch of docker.io | 22:19 |
| mwhahaha | http://mirror.regionone.limestone.openstack.org:8082/v2 | 22:20 |
| mwhahaha | it's using the proxy | 22:20 |
| mwhahaha | the docker.io shit is for the auth stuff | 22:20 |
| mwhahaha | to get a tocken | 22:20 |
| mwhahaha | token | 22:20 |
| EmilienM | ok | 22:20 |
| mwhahaha | that's not proxiable | 22:20 |
| EmilienM | yeah | 22:20 |
| EmilienM | i wonder if our problem is a mix between the fact we now close http sessions properly AND the retries AND the pools | 22:21 |
| mwhahaha | when we close the sessions we have to reauth | 22:28 |
| mwhahaha | so we'd go back to docker.io more | 22:28 |
| EmilienM | yeah | 22:28 |
| cloudnull | I think you may be on the right track with lowering the max pool size. | 22:33 |
| cloudnull | given the use of futures it just may all be too much | 22:33 |
| cloudnull | we have a mix of multi-processing functions and not, with a mix of requests connections and sessions. | 22:35 |
| mwhahaha | so we should probably revert the close | 22:35 |
| mwhahaha | and lower the counts | 22:35 |
| cloudnull | +1 sounds sensible to me | 22:35 |
| mwhahaha | in theory that'll reduce the docker.io call | 22:36 |
| EmilienM | mwhahaha, cloudnull : do you want me to push over https://review.opendev.org/#/c/682731/2/tripleo_common/image/image_uploader.py to use the defaults from requests or wait for the (running) CI job to end? | 22:36 |
| mwhahaha | i'd just leave it for now | 22:37 |
| cloudnull | +1 | 22:37 |
| EmilienM | ok | 22:37 |
| EmilienM | the close revert is already proposed: https://review.opendev.org/#/c/682717/ | 22:37 |
| cloudnull | The default is 10, however I think 4 might be a better option | 22:37 |
| EmilienM | i've put 5, let's see | 22:37 |
| * cloudnull likes powers of 2 | 22:38 | |
| * cloudnull does more pushups | 22:38 | |
| EmilienM | I wonder if the new centos7 changed the version of pacemaker so we could run with podman? | 22:52 |
| EmilienM | bandini: ^ | 22:52 |
| *** bfournie has joined #tripleo | 22:52 | |
| mwhahaha | no | 22:55 |
| mwhahaha | i think it's a major rev | 22:56 |
| mwhahaha | in 8 | 22:56 |
| openstackgerrit | Emilien Macchi proposed openstack/python-tripleoclient master: container_image: avoid extra space in logs https://review.opendev.org/682763 | 22:57 |
| *** tkajinam has joined #tripleo | 23:04 | |
| *** tosky has quit IRC | 23:11 | |
| *** slaweq has joined #tripleo | 23:11 | |
| *** bdodd_ has quit IRC | 23:13 | |
| *** slaweq has quit IRC | 23:15 | |
| *** dsneddon has quit IRC | 23:35 | |
| *** rcernin has joined #tripleo | 23:44 | |
| *** rlandy has quit IRC | 23:51 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!