| openstackgerrit | Merged openstack/tripleo-heat-templates master: Fix selinux context for glance-api https://review.opendev.org/682768 | 00:18 |
|---|---|---|
| *** morazi has quit IRC | 00:37 | |
| *** morazi has joined #tripleo | 00:41 | |
| *** morazi has quit IRC | 00:52 | |
| EmilienM | cloudnull: can we fix it in another ? (on top) https://review.rdoproject.org/r/#/c/22445 | 00:55 |
| *** morazi has joined #tripleo | 00:58 | |
| openstackgerrit | Merged openstack/paunch master: Fix discovering container names https://review.opendev.org/678577 | 01:00 |
| *** morazi has quit IRC | 01:05 | |
| *** morazi has joined #tripleo | 01:07 | |
| *** gbarros has joined #tripleo | 01:09 | |
| cloudnull | EmilienM +1 | 01:14 |
| *** mschuppert has quit IRC | 01:15 | |
| * cloudnull doesn't have +2 there | 01:15 | |
| openstackgerrit | Merged openstack/tripleo-heat-templates master: Ensure consistent mirroring across config-data https://review.opendev.org/674580 | 01:17 |
| openstackgerrit | Kevin Carter (cloudnull) proposed openstack/tripleo-heat-templates master: Move nested bash script into a real file https://review.opendev.org/683270 | 01:23 |
| openstackgerrit | Kevin Carter (cloudnull) proposed openstack/tripleo-heat-templates master: Move basic global functions into ansible tasks https://review.opendev.org/683286 | 01:24 |
| openstackgerrit | Kevin Carter (cloudnull) proposed openstack/tripleo-heat-templates master: Convert subprocess to a reusable method https://review.opendev.org/683429 | 01:24 |
| openstackgerrit | Kevin Carter (cloudnull) proposed openstack/tripleo-heat-templates master: Move globals into main level 0 case https://review.opendev.org/683590 | 01:25 |
| openstackgerrit | Kevin Carter (cloudnull) proposed openstack/tripleo-heat-templates master: Move config processing into consolodated methods https://review.opendev.org/683670 | 01:25 |
| *** morazi has quit IRC | 01:45 | |
| *** morazi has joined #tripleo | 01:45 | |
| *** pierrepr1netti has joined #tripleo | 01:52 | |
| *** pierreprinetti has quit IRC | 01:56 | |
| openstackgerrit | pengyuesheng proposed openstack/tripleo-common master: Update the constraints url https://review.opendev.org/684160 | 02:04 |
| openstackgerrit | pengyuesheng proposed openstack/tripleo-validations master: Update the constraints url https://review.opendev.org/684161 | 02:06 |
| *** saneax has joined #tripleo | 02:08 | |
| *** weshay|rover has quit IRC | 02:20 | |
| openstackgerrit | Merged openstack/tripleo-common master: Add profile_tasks to ansible runs https://review.opendev.org/683425 | 02:27 |
| *** rfolco has quit IRC | 02:27 | |
| *** dsneddon has quit IRC | 02:29 | |
| *** rlandy has quit IRC | 02:33 | |
| *** saneax has quit IRC | 02:40 | |
| *** ricolin has joined #tripleo | 02:48 | |
| *** rohit02 has joined #tripleo | 03:24 | |
| *** ykarel has joined #tripleo | 03:32 | |
| *** poopcat has quit IRC | 03:32 | |
| *** poopcat has joined #tripleo | 03:34 | |
| openstackgerrit | Emilien Macchi proposed openstack/paunch stable/stein: Revert "Add --cpuset-cpus support for both Docker and Podman" https://review.opendev.org/684180 | 03:39 |
| openstackgerrit | Emilien Macchi proposed openstack/paunch stable/stein: Revert "Add --cpuset-cpus support for both Docker and Podman" https://review.opendev.org/684180 | 03:39 |
| openstackgerrit | Emilien Macchi proposed openstack/paunch master: Add --cpuset-cpus support for both Docker and Podman https://review.opendev.org/681257 | 03:48 |
| openstackgerrit | Emilien Macchi proposed openstack/paunch stable/stein: Improve volume validation https://review.opendev.org/683386 | 03:49 |
| *** artom has quit IRC | 03:56 | |
| openstackgerrit | yatin proposed openstack/tripleo-quickstart master: [DNM] just test effect of container update https://review.opendev.org/684187 | 03:56 |
| openstackgerrit | Emilien Macchi proposed openstack/paunch stable/stein: Add --cpuset-cpus support for both Docker and Podman https://review.opendev.org/684189 | 04:01 |
| EmilienM | mwhahaha: ^ sorry for the paunch mess, should be fixed (i reverted the backport, backported your volume patch and on top, cherry-pick of the cpu thing) | 04:01 |
| *** skramaja has joined #tripleo | 04:05 | |
| *** jamesdenton has quit IRC | 04:07 | |
| openstackgerrit | Merged openstack/tripleo-upgrade master: Fix "unbound" variable https://review.opendev.org/684010 | 04:10 |
| *** morazi has quit IRC | 04:17 | |
| *** morazi has joined #tripleo | 04:19 | |
| openstackgerrit | Merged openstack/tripleo-ansible master: paunch: disable check mode https://review.opendev.org/683802 | 04:21 |
| *** ykarel has quit IRC | 04:24 | |
| *** gbarros has quit IRC | 04:29 | |
| *** jamesden_ has joined #tripleo | 04:43 | |
| *** ratailor has joined #tripleo | 04:43 | |
| *** ykarel has joined #tripleo | 04:55 | |
| *** abishop has joined #tripleo | 04:56 | |
| openstackgerrit | Chandan Kumar (raukadah) proposed openstack/tripleo-quickstart master: [CentOS-7]Added release file for train https://review.opendev.org/683932 | 04:56 |
| *** holser has joined #tripleo | 04:57 | |
| openstackgerrit | Sergii Golovatiuk proposed openstack/tripleo-upgrade stable/stein: Fix "unbound" variable https://review.opendev.org/684191 | 04:57 |
| *** ratailor has quit IRC | 05:00 | |
| *** udesale has joined #tripleo | 05:00 | |
| *** bnemec has quit IRC | 05:09 | |
| *** abishop has quit IRC | 05:10 | |
| *** brault has quit IRC | 05:12 | |
| *** brault has joined #tripleo | 05:15 | |
| *** lmiccini has joined #tripleo | 05:15 | |
| *** bnemec has joined #tripleo | 05:15 | |
| *** morazi has quit IRC | 05:25 | |
| *** ratailor has joined #tripleo | 05:26 | |
| *** jtomasek has joined #tripleo | 05:31 | |
| *** morazi has joined #tripleo | 05:31 | |
| *** dsneddon has joined #tripleo | 05:41 | |
| openstackgerrit | Cédric Jeanneret (Tengu) proposed openstack/tripleo-validations master: New validation: detect failed containers https://review.opendev.org/656785 | 05:41 |
| *** dsneddon has quit IRC | 05:46 | |
| *** holser has quit IRC | 05:53 | |
| *** yprokule has joined #tripleo | 05:58 | |
| *** marios has joined #tripleo | 05:59 | |
| *** jfrancoa has joined #tripleo | 06:00 | |
| *** jfrancoa has quit IRC | 06:04 | |
| *** kopecmartin|off is now known as kopecmartin | 06:05 | |
| *** surpatil has joined #tripleo | 06:05 | |
| *** slaweq has joined #tripleo | 06:09 | |
| *** saneax has joined #tripleo | 06:10 | |
| *** brault has quit IRC | 06:17 | |
| *** mschuppert has joined #tripleo | 06:23 | |
| *** morazi has quit IRC | 06:24 | |
| *** yolanda has quit IRC | 06:34 | |
| *** abishop has joined #tripleo | 06:42 | |
| *** rpittau|afk is now known as rpittau | 06:43 | |
| *** morazi has joined #tripleo | 06:44 | |
| *** tosky has joined #tripleo | 06:53 | |
| *** pcaruana has joined #tripleo | 06:56 | |
| *** gfidente has joined #tripleo | 06:59 | |
| *** hberaud|gone is now known as hberaud | 07:05 | |
| openstackgerrit | yatin proposed openstack/tripleo-quickstart master: [DNM] just test effect of container update https://review.opendev.org/684187 | 07:05 |
| *** shyam89 has joined #tripleo | 07:07 | |
| *** tesseract has joined #tripleo | 07:07 | |
| *** paramite has joined #tripleo | 07:09 | |
| *** soniya29 has joined #tripleo | 07:12 | |
| *** xek has joined #tripleo | 07:16 | |
| *** hjensas has quit IRC | 07:23 | |
| *** chem has joined #tripleo | 07:24 | |
| Tengu | hmmm... gate is terribly slow right? | 07:31 |
| *** bogdando has joined #tripleo | 07:31 | |
| *** brault has joined #tripleo | 07:38 | |
| *** holser has joined #tripleo | 07:40 | |
| *** avivgta has joined #tripleo | 07:40 | |
| *** brault has quit IRC | 07:43 | |
| ksambor | Tengu: yeah and timeouts appears sometimes | 07:43 |
| *** amoralej|off is now known as amoralej | 07:44 | |
| Tengu | ksambor: apparently, yes. apparently due to the container-prepare thingy. | 07:44 |
| Tengu | since the in-flight were deactivated last week. | 07:44 |
| *** shyam89 has quit IRC | 07:45 | |
| *** shyam89 has joined #tripleo | 07:45 | |
| *** gfidente has quit IRC | 07:46 | |
| *** iurygregory has joined #tripleo | 07:47 | |
| *** shyam89 has quit IRC | 07:51 | |
| *** jfrancoa has joined #tripleo | 07:53 | |
| openstackgerrit | pengyuesheng proposed openstack/diskimage-builder master: Update the constraints url https://review.opendev.org/684216 | 07:53 |
| openstackgerrit | pengyuesheng proposed openstack/tripleo-quickstart master: Update the constraints url https://review.opendev.org/684221 | 07:56 |
| *** psachin has joined #tripleo | 08:00 | |
| *** florianf has joined #tripleo | 08:00 | |
| *** jpich has joined #tripleo | 08:01 | |
| openstackgerrit | pengyuesheng proposed openstack/ansible-role-container-registry master: Update the constraints url https://review.opendev.org/684226 | 08:02 |
| *** alexmcleod has joined #tripleo | 08:05 | |
| *** cylopez has joined #tripleo | 08:07 | |
| *** ykarel is now known as ykarel|lunch | 08:08 | |
| *** hjensas has joined #tripleo | 08:09 | |
| *** apetrich has quit IRC | 08:12 | |
| *** tkajinam has quit IRC | 08:14 | |
| *** apetrich has joined #tripleo | 08:22 | |
| *** derekh has joined #tripleo | 08:28 | |
| *** jpena|off is now known as jpena | 08:30 | |
| *** hamdyk has joined #tripleo | 08:33 | |
| *** jbadiapa has joined #tripleo | 08:35 | |
| *** jbadiapa has quit IRC | 08:37 | |
| *** jbadiapa has joined #tripleo | 08:38 | |
| *** shyam89 has joined #tripleo | 08:41 | |
| *** jbadiapa has quit IRC | 08:42 | |
| *** holser has quit IRC | 08:43 | |
| *** holser has joined #tripleo | 08:44 | |
| *** lmiccini has quit IRC | 08:44 | |
| *** lyarwood has quit IRC | 08:45 | |
| *** mburned has quit IRC | 08:45 | |
| *** ykarel|lunch is now known as ykarel | 08:46 | |
| *** lyarwood has joined #tripleo | 08:46 | |
| openstackgerrit | Bogdan Dobrelya (bogdando) proposed openstack/tripleo-ci master: Fallback to public IPv4 when there is no priv. IP https://review.opendev.org/682092 | 08:48 |
| *** ccamacho has joined #tripleo | 09:01 | |
| *** rcernin has quit IRC | 09:04 | |
| *** brault has joined #tripleo | 09:05 | |
| *** ricolin_ has joined #tripleo | 09:05 | |
| *** ricolin has quit IRC | 09:08 | |
| *** brault has quit IRC | 09:09 | |
| openstackgerrit | Chandan Kumar (raukadah) proposed openstack/tripleo-ci master: Added support for train for emit release file https://review.opendev.org/684260 | 09:17 |
| *** florianf has quit IRC | 09:17 | |
| *** ricolin_ is now known as ricolin | 09:23 | |
| *** ratailor_ has joined #tripleo | 09:26 | |
| openstackgerrit | mathieu bultel proposed openstack/tripleo-common master: Implement Ansible fact cache for Mistral executor https://review.opendev.org/682855 | 09:26 |
| *** shyam89 has quit IRC | 09:26 | |
| Tengu | marios: heya! quick question: would it be possible to get some info about the swift storage dedicated for the logs? I suspect it might be the cause of the current timeouts we see in Zuul.... | 09:27 |
| *** ratailor has quit IRC | 09:28 | |
| Tengu | marios: at least I don't see any timeout in this log: https://zuul.opendev.org/t/openstack/build/8b79b9ca3b9d4ff58c931d0bac313289/log/job-output.txt - but it has failed :/ | 09:28 |
| Tengu | oh. sh*t... it's "timed_out", not "time out" nor "timeout" | 09:30 |
| openstackgerrit | Martin Schuppert proposed openstack/tripleo-heat-templates stable/queens: [Queens] Run cellv2 host discovery via workflow or deploy_steps_tasks https://review.opendev.org/683069 | 09:30 |
| marios | Tengu: looking in the logs for the timeout now but not sure what you mean about swift storage for logs cos not sure we have much say about that sshnaidm do we have control on log server? | 09:31 |
| Tengu | marios: thought it might be linked since it was the last line there, but apparently the timeout happens during the tempest run -.-'. | 09:31 |
| Tengu | I just didn't think to search for "timed_out" string in there. pfff. | 09:32 |
| marios | 2019-09-23 19:31:27.524944 | primary | sort: write failed: standard output: Broken pipe on collect logs but not sure that it | 09:32 |
| marios | at least the time jumps from 04 to 31 there | 09:32 |
| marios | so maybe collect logs failed or times out if you already found the timeout | 09:32 |
| Tengu | marios: yeah, so that's apparently not main failure cause. | 09:32 |
| Tengu | https://zuul.opendev.org/t/openstack/build/8b79b9ca3b9d4ff58c931d0bac313289/log/job-output.txt#7236 | 09:32 |
| *** saneax is now known as saneax|AFK | 09:34 | |
| *** saneax|AFK has quit IRC | 09:40 | |
| *** shyam89 has joined #tripleo | 09:40 | |
| *** brault has joined #tripleo | 09:42 | |
| *** brault has quit IRC | 09:42 | |
| *** brault has joined #tripleo | 09:42 | |
| *** hberaud is now known as hberaud|lunch | 09:51 | |
| openstackgerrit | Saravanan KR proposed openstack/tripleo-ansible master: Add kernelargs and tuned core configuration https://review.opendev.org/679809 | 09:51 |
| owalsh | Tengu: hey, I really don't understand the comment on https://review.opendev.org/681442... how can fixing a huge race condition be incompatible with the specs? | 09:54 |
| openstackgerrit | Hamdy Khader proposed openstack/tripleo-heat-templates master: Add heat templates to deploy mellanox neutron agent https://review.opendev.org/674486 | 09:55 |
| openstackgerrit | Hamdy Khader proposed openstack/tripleo-heat-templates master: Add support for Neutron Multi interface driver https://review.opendev.org/679558 | 09:55 |
| *** florianf has joined #tripleo | 10:01 | |
| *** lmiccini has joined #tripleo | 10:01 | |
| owalsh | Tengu: e.g https://review.opendev.org/681525 should have failed CI as the RPC healthchecks were not completely fixed. With the fix for the race we catch it https://review.opendev.org/681605 | 10:02 |
| openstackgerrit | Sagi Shnaidman proposed openstack/tripleo-ci master: DNM: run ara undercloud https://review.opendev.org/684267 | 10:08 |
| *** dsneddon has joined #tripleo | 10:09 | |
| *** brault has quit IRC | 10:11 | |
| *** cylopez has quit IRC | 10:11 | |
| *** openstackstatus has quit IRC | 10:12 | |
| *** openstack has joined #tripleo | 10:14 | |
| *** ChanServ sets mode: +o openstack | 10:14 | |
| *** openstackstatus has joined #tripleo | 10:14 | |
| *** ChanServ sets mode: +v openstackstatus | 10:14 | |
| *** morazi has quit IRC | 10:17 | |
| *** dtantsur|afk is now known as dtantsur | 10:17 | |
| bandini | is 'Exception: Not found image: docker://trunk.registry.rdoproject.org/tripleomaster/rhel-binary-cron:fd46c80632bbd7abc8411277bfc4819f6b95a485_bf6d0acd' known as an issue? (http://logs.rdoproject.org/47/669847/24/openstack-check/tripleo-ci-rhel-8-scenario001-standalone-rdo/07c8836/logs/undercloud/home/zuul/standalone_deploy.log.txt.gz) | 10:20 |
| *** shyam89 has quit IRC | 10:23 | |
| *** pcaruana has quit IRC | 10:24 | |
| owalsh | mschuppert: hey, re https://review.opendev.org/683069 ... | 10:27 |
| mschuppert | owalsh: o/ | 10:27 |
| openstackgerrit | Kamil Sambor proposed openstack/tripleo-heat-templates master: Support connecting OVN DB over SSL https://review.opendev.org/680345 | 10:28 |
| owalsh | mschuppert: IIUC the workflow would be run twice when we have nova_compute + nova_ironic | 10:29 |
| mschuppert | owalsh: yes thats a scenario I am going to test right now and to see how to deal with. thats why I had it initially delecated to the nova_api | 10:30 |
| owalsh | mschuppert: ack, I guess running twice is ok | 10:30 |
| owalsh | mschuppert: but probably need to drop the params and hardcode the service/container | 10:31 |
| amoralej | hi, https://bugs.launchpad.net/tripleo/+bug/1845181 is blocking eventlet promotion in rdo, which is required for other projects | 10:31 |
| openstack | Launchpad bug 1845181 in tripleo "Problem with tripleo_common git functionalities and eventlet 0.25.1" [Undecided,New] | 10:31 |
| amoralej | EmilienM, ^ fyi | 10:31 |
| amoralej | apetrich, i see you worked in the initial fix, ^ you may have a good solution for that | 10:31 |
| mschuppert | owalsh: yes, I added them only from the manager thing you work on. but as we said we won't support that on queens so I could hard code it | 10:32 |
| openstackgerrit | Sergii Golovatiuk proposed openstack/tripleo-heat-templates master: Replace include_role with import_role https://review.opendev.org/682672 | 10:32 |
| owalsh | mschuppert: will need to support it in queens I expect | 10:32 |
| mschuppert | oh | 10:32 |
| owalsh | hrmm, maybe default to hardcoded but use the params if they are set? | 10:33 |
| *** shyam89 has joined #tripleo | 10:33 | |
| owalsh | which could be a follow-up patch when the manager thing is being backported | 10:34 |
| mschuppert | owalsh: ack | 10:34 |
| mschuppert | makes sense | 10:34 |
| apetrich | amoralej, sorry, I was about to go to lunch. Is that all right if I look as soon as I'm back | 10:35 |
| apetrich | ? | 10:35 |
| amoralej | sure | 10:35 |
| amoralej | no problem | 10:35 |
| owalsh | mschuppert: the other thing I was wondering... when is workflow_tasks step5 run? Wondering if we'll have the same problem I've got with https://review.opendev.org/681042 (I need a step6!) | 10:36 |
| owalsh | https://review.opendev.org/#/c/681042/12/deployment/nova/nova-compute-container-puppet.yaml@946 | 10:36 |
| mschuppert | owalsh: it runs before deploy task step5, see http://pastebin.test.redhat.com/800058 | 10:39 |
| owalsh | mschuppert: damn, nova_ironic start the container in step5 | 10:41 |
| mschuppert | owalsh: where? not in queens. in mater? | 10:43 |
| mschuppert | master | 10:43 |
| owalsh | mschuppert: queens IIRC | 10:43 |
| owalsh | mschuppert: https://github.com/openstack/tripleo-heat-templates/blob/stable/queens/docker/services/nova-ironic.yaml#L163 | 10:44 |
| owalsh | mschuppert: ah, no, step4 https://github.com/openstack/tripleo-heat-templates/blob/stable/queens/docker/services/nova-ironic.yaml#L118 | 10:44 |
| mschuppert | yes, thats only discovery | 10:44 |
| owalsh | mschuppert: but I need to move it to step5 in https://review.opendev.org/681042 | 10:45 |
| mschuppert | owalsh: then we need a step 6 there as well | 10:45 |
| *** pcaruana has joined #tripleo | 10:48 | |
| *** shyam89 has quit IRC | 10:48 | |
| mschuppert | owalsh: or we add a wait_for_nova_api before we start the nova_compute in step4 | 10:50 |
| *** weshay has joined #tripleo | 10:50 | |
| owalsh | mschuppert: but wait for how long? | 10:50 |
| owalsh | pulling the image doesn't really have an upper bound (well, somthing else will timeout but....) | 10:51 |
| mschuppert | owalsh: yeah good point | 10:51 |
| owalsh | mschuppert: actually adding a step6 for workflow tasks may not be that bad - https://github.com/openstack/tripleo-heat-templates/blob/stable/queens/common/deploy-steps.j2#L87 | 10:53 |
| owalsh | and we could call the 6th step 'post' ;-) | 10:55 |
| mschuppert | owalsh: yes :) | 10:55 |
| *** ratailor_ has quit IRC | 10:58 | |
| *** iurygregory_ has joined #tripleo | 11:03 | |
| *** iurygregory has quit IRC | 11:05 | |
| openstackgerrit | Harald Jensås proposed openstack/tripleo-ansible master: Support 'ipversion' property in rules https://review.opendev.org/684277 | 11:06 |
| openstackgerrit | Harald Jensås proposed openstack/tripleo-ansible master: Open ipv6-icmp traffic by default https://review.opendev.org/684278 | 11:06 |
| apetrich | amoralej, back. Looking | 11:06 |
| openstackgerrit | wes hayutin proposed openstack/tripleo-heat-templates master: Add no_log to include_vars https://review.opendev.org/682988 | 11:08 |
| *** udesale has quit IRC | 11:08 | |
| *** udesale has joined #tripleo | 11:09 | |
| Tengu | is there a reason for tripleo-ci-centos-7-containers-multinode to often have timeouts? Apparently that's the only job hitting this issue now :( | 11:10 |
| Tengu | just trying to understand | 11:10 |
| *** hberaud|lunch is now known as hberaud | 11:16 | |
| *** artom has joined #tripleo | 11:17 | |
| *** avivgta has quit IRC | 11:17 | |
| *** lucasagomes has joined #tripleo | 11:21 | |
| openstackgerrit | Adriano Petrich proposed openstack/tripleo-common master: Limit the eventlet version to lower than 0.25 as it fails tests https://review.opendev.org/684282 | 11:22 |
| apetrich | amoralej, It is the version 0.25 of the eventlet | 11:23 |
| apetrich | amoralej, wait is version 0.25 required for other projects | 11:24 |
| amoralej | apetrich, yes, we need to bump to 0.25.1 | 11:24 |
| apetrich | amoralej, that's an issue. | 11:24 |
| apetrich | ok I will abandon my patch and rething that | 11:24 |
| amoralej | it's in upper-constraints so it's the version that the rest of projects are using to test upstream | 11:25 |
| amoralej | and in this case swift is actively requesting ti | 11:25 |
| amoralej | it | 11:25 |
| amoralej | https://review.rdoproject.org/r/#/c/22509/ | 11:25 |
| apetrich | amoralej, ok. That is going to take a bit more time. I'm looking into it | 11:26 |
| amoralej | sure, no problem, thanks | 11:26 |
| *** shyam89 has joined #tripleo | 11:27 | |
| *** ansmith has quit IRC | 11:31 | |
| *** amoralej is now known as amoralej|lunch | 11:32 | |
| weshay | zbr|ruck, arxcruz|rover ok.. if you narrow the time in the job explorer.. you get a decent snapshot of container build times | 11:32 |
| weshay | http://dashboard-ci.tripleo.org/d/si1tipHZk/jobs-exploration?orgId=1&from=now-12h&to=now&var-influxdb_filter=branch%7C%3D%7Cmaster | 11:32 |
| weshay | EmilienM, fyi ^ | 11:32 |
| weshay | sorry.. container-prep times on the undercloud/standalone | 11:32 |
| weshay | note.. rdo periodic jobs hitting the rdo registry takes 6.5 minutes | 11:32 |
| *** marios has quit IRC | 11:33 | |
| weshay | tripleo-ci-centos-7-containers-multinode has a 50% pass rate :( | 11:34 |
| *** jpena is now known as jpena|lunch | 11:34 | |
| openstackgerrit | mathieu bultel proposed openstack/tripleo-heat-templates master: Use update_serial as an ansible variable https://review.opendev.org/684287 | 11:38 |
| zbr|ruck | weshay: can we have a sync session? i could use some help. bj? | 11:38 |
| Tengu | weshay: why only this one for the container-prepare thingy? is it due to multi-node, or is there some other specifics? | 11:40 |
| weshay | https://trello.com/c/KkBkHEOD/1118-cixlp1844446tripleociproa-multiple-tripleo-jobs-timing-out-upstream-causing-gate-resets-in-train | 11:41 |
| weshay | Tengu, ^ | 11:41 |
| openstackgerrit | Dmitry Tantsur proposed openstack/tripleo-common master: Correct deprecated usage of ironicclient (broken with 3.0.0) https://review.opendev.org/684288 | 11:41 |
| dtantsur | arxcruz|rover: this ^^ may be it | 11:41 |
| dtantsur | I'm not sure how to test it with 3.0.0 though | 11:42 |
| * dtantsur gets food now | 11:42 | |
| weshay | Tengu, zbr|ruck look at https://snapshot.raintank.io/dashboard/snapshot/SjZ16RtMXZCFbEIOjgZ7N7KKJvOpKBhB?orgId=2 | 11:42 |
| weshay | that is data from yesterday | 11:42 |
| Tengu | weshay: yes, apparently only the multinode has timed_out right? why this one in particular? | 11:43 |
| *** noama has joined #tripleo | 11:46 | |
| *** morazi has joined #tripleo | 11:49 | |
| *** rh-jelabarre has joined #tripleo | 11:54 | |
| *** bfournie has quit IRC | 11:54 | |
| *** boazel has joined #tripleo | 11:54 | |
| *** suuuper has joined #tripleo | 11:57 | |
| EmilienM | amoralej: Ack | 11:58 |
| *** marios has joined #tripleo | 11:58 | |
| *** jpich has quit IRC | 11:58 | |
| *** stevebaker has quit IRC | 11:59 | |
| *** derekh has quit IRC | 12:00 | |
| *** stevebaker has joined #tripleo | 12:00 | |
| apetrich | EmilienM, amoralej|lunch that is a bit tricky. there's some bad interactions between gitpython and eventlet 0.25. I'm working on it | 12:05 |
| *** cylopez has joined #tripleo | 12:05 | |
| EmilienM | apetrich: thanks | 12:07 |
| EmilienM | weshay: looking | 12:07 |
| *** ekultails has joined #tripleo | 12:07 | |
| *** rfolco has joined #tripleo | 12:10 | |
| openstackgerrit | Chandan Kumar (raukadah) proposed openstack/tripleo-quickstart-extras master: Added train release tempest skip list https://review.opendev.org/684295 | 12:16 |
| weshay | EmilienM, on the agenda for tripleo mtg.. <weshay> thoughts on a daily periodic job to update master containers w/ update_repo: "delorean-current,quickstart-centos-ceph-nautilus" ? | 12:17 |
| *** avivgta has joined #tripleo | 12:20 | |
| EmilienM | weshay: yes | 12:25 |
| apetrich | amoralej|lunch, bbiab. have to pick up the kid at the nursery. | 12:25 |
| EmilienM | to reduce what is updated in each job? | 12:25 |
| weshay | EmilienM, aye. | 12:25 |
| weshay | EmilienM, btw.. the periodic rdo jobs.. pull containers in 5-6 min | 12:26 |
| *** aakarsh has joined #tripleo | 12:27 | |
| EmilienM | weshay: but did you apply https://review.rdoproject.org/r/#/c/22445/ ? | 12:29 |
| weshay | EmilienM, waiting to chat to rlandy about that | 12:29 |
| weshay | and the other folks working on the promoter atm.. I just pinged them all in #oooq | 12:30 |
| weshay | about it | 12:30 |
| *** jamesden_ has quit IRC | 12:34 | |
| *** jamesden_ has joined #tripleo | 12:34 | |
| *** jpena|lunch is now known as jpena | 12:35 | |
| *** shyam89 has quit IRC | 12:37 | |
| *** brault has joined #tripleo | 12:37 | |
| *** ade_lee_ has quit IRC | 12:38 | |
| *** rlandy has joined #tripleo | 12:39 | |
| *** brault has quit IRC | 12:40 | |
| *** brault has joined #tripleo | 12:40 | |
| *** ansmith has joined #tripleo | 12:42 | |
| *** jamesden_ is now known as jamesdenton | 12:44 | |
| *** bfournie has joined #tripleo | 12:46 | |
| *** amoralej|lunch is now known as amoralej | 12:46 | |
| *** gfidente has joined #tripleo | 12:47 | |
| *** psachin has quit IRC | 12:49 | |
| *** brault has quit IRC | 12:49 | |
| *** brault has joined #tripleo | 12:50 | |
| openstackgerrit | Gael Chamoulaud proposed openstack/tripleo-validations master: Add new healthcheck-service-status validation https://review.opendev.org/682377 | 12:50 |
| *** rohit02 has quit IRC | 12:52 | |
| *** aakarsh has quit IRC | 12:54 | |
| *** mcornea has joined #tripleo | 12:59 | |
| openstackgerrit | Sorin Sbarnea proposed openstack/tripleo-common stable/stein: Add profile_tasks to ansible runs https://review.opendev.org/684305 | 12:59 |
| *** jaosorior has quit IRC | 12:59 | |
| openstackgerrit | Carlos Goncalves proposed openstack/diskimage-builder master: WIP: CentOS 8 support https://review.opendev.org/684308 | 13:02 |
| openstackgerrit | Sorin Sbarnea proposed openstack/tripleo-common stable/rocky: Add profile_tasks to ansible runs https://review.opendev.org/684309 | 13:02 |
| *** brault has quit IRC | 13:03 | |
| *** raildo has joined #tripleo | 13:04 | |
| openstackgerrit | Sorin Sbarnea proposed openstack/tripleo-common stable/queens: Add profile_tasks to ansible runs https://review.opendev.org/684310 | 13:04 |
| *** brault has joined #tripleo | 13:05 | |
| zbr|ruck | weshay: EmilienM sshnaidm please help me backport the profiling ^ | 13:06 |
| *** jcoufal has joined #tripleo | 13:07 | |
| weshay | zbr|ruck, can we get stein please? | 13:07 |
| zbr|ruck | weshay: stein is among them, there are 3 in total | 13:07 |
| weshay | ah sorry | 13:08 |
| * weshay looks | 13:08 | |
| sshnaidm | zbr|ruck, ERROR Unable to find role in /var/opt/rh/rh-python35/lib/zuul/builds/bc245bb870f048ac8ee3a1b00ebd6199/ansible/pre_playbook_7/role_1/tripleo-ansible in https://review.opendev.org/#/c/684305/ | 13:09 |
| *** brault has quit IRC | 13:10 | |
| zbr|ruck | sshnaidm: what can I say.... unrelated :D | 13:10 |
| *** iurygregory_ is now known as iurygregory | 13:11 | |
| *** saneax|AFK has joined #tripleo | 13:11 | |
| openstackgerrit | Luke Short proposed openstack/tripleo-heat-templates master: Convert firewall rules to use TripleO-Ansible https://review.opendev.org/677237 | 13:15 |
| openstackgerrit | Luke Short proposed openstack/tripleo-heat-templates master: Use the Ansible role for tripleo-packages. https://review.opendev.org/678620 | 13:15 |
| ykarel | sshnaidm, zbr|ruck seems happening due to stable/stein branch in tripleo-ansible | 13:19 |
| openstackgerrit | Bogdan Dobrelya (bogdando) proposed openstack/ansible-role-tripleo-modify-image master: Add support for yum caching when buildah updates https://review.opendev.org/683100 | 13:19 |
| ykarel | was that branch created intentionally? | 13:19 |
| ykarel | isn't tripleo-ansible going to be branchless? | 13:19 |
| *** psachin has joined #tripleo | 13:20 | |
| cloudnull | mornings | 13:20 |
| EmilienM | cloudnull: hey good morning, have you seen https://bugs.launchpad.net/bugs/1845170 and https://bugs.launchpad.net/bugs/1845175 yet? | 13:20 |
| openstack | Launchpad bug 1845170 in tripleo "tripleo-ansible - uses 'proto' field to decide if rule is created in iptables or ip6tables" [High,In progress] - Assigned to Harald Jensås (harald-jensas) | 13:20 |
| openstack | Launchpad bug 1845175 in tripleo "tripleo-ansible - firewall role does not open ipv6-icmp" [High,In progress] - Assigned to Harald Jensås (harald-jensas) | 13:20 |
| sshnaidm | ykarel, good question, weshay cloudnull do we need stein in tripleo-ansible? | 13:21 |
| EmilienM | sshnaidm: no | 13:21 |
| ykarel | good to clean up then https://github.com/openstack/tripleo-ansible/branches | 13:22 |
| cloudnull | EmilienM I have not seen the bugs, though there was chatter in the the firewall review about it - https://review.opendev.org/#/c/677237/ | 13:23 |
| *** ade_lee_ has joined #tripleo | 13:23 | |
| cloudnull | https://review.opendev.org/#/c/684278/ https://review.opendev.org/#/c/684277/ - looks like reviews are up, now we just need the multinode job to not timeout :) | 13:25 |
| Tengu | weshay: heya! isn't your script for trello going nut? | 13:27 |
| openstackgerrit | Karthik S proposed openstack/tripleo-heat-templates master: Fixed the path for neutron-base.yaml https://review.opendev.org/684318 | 13:27 |
| weshay | Tengu, going nuts? | 13:27 |
| weshay | Tengu, looks fine | 13:28 |
| weshay | why? | 13:28 |
| Tengu | weshay: yeah, duplicating cards in CIX board | 13:28 |
| weshay | Tengu, lolz.. you have to update the bug first brotha | 13:28 |
| weshay | :) | 13:28 |
| weshay | Tengu, stop cheating ;) | 13:28 |
| weshay | Tengu, which bug are you referring to? | 13:28 |
| Tengu | https://trello.com/c/Ix9GT9O8/1126-cixbz1844856ospphase1osp16puppet-tripleopodmanjson-file-is-not-valid-arg-with-podman-142 | 13:30 |
| cloudnull | ykarel +1 feel free to remove stein branch. unless EmilienM or mwhahaha can think of how/where we're using it I don't think we need it. | 13:30 |
| *** gbarros has joined #tripleo | 13:30 | |
| *** surpatil has quit IRC | 13:30 | |
| EmilienM | I don't think we need it | 13:31 |
| Tengu | weshay: ah, there's an issue, LP id instead of BZ.... | 13:31 |
| Tengu | weird. | 13:31 |
| hjensas | EmilienM: I'm working on [Bug 1845153] Re: puppet-tripleo - not possible to create IPv4 or IPv6 only rules | 13:31 |
| openstack | bug 1845153 in tripleo "puppet-tripleo - not possible to create IPv4 or IPv6 only rules" [High,Triaged] https://launchpad.net/bugs/1845153 - Assigned to Emilien Macchi (emilienm) | 13:31 |
| EmilienM | hjensas: ack, ok, I saw it unassigned so I took it :) I'll update launchpad | 13:32 |
| hjensas | EmilienM: specs, and an actual test run to go. | 13:32 |
| EmilienM | thanks for working on it ! | 13:32 |
| mwhahaha | ykarel, cloudnull: no it's not going to be branchless | 13:32 |
| *** mcornea has quit IRC | 13:32 | |
| cloudnull | hjensas - did you see https://review.opendev.org/#/c/684277/1/tripleo_ansible/roles/tripleo-firewall/tasks/tripleo_firewall_add.yml ? | 13:33 |
| cloudnull | otherwise looks good. thanks for working on it | 13:33 |
| ykarel | mwhahaha, you mean stable/stein is used or will be used? | 13:33 |
| cloudnull | without the default its resulting in https://427ec8d7b2c1b1e2ac8c-16c430a2363acd21a3fbf84e4ca76ae5.ssl.cf1.rackcdn.com/684277/1/check/tripleo-ansible-centos-7-molecule-tripleo-firewall/7f86851/reports.html | 13:33 |
| weshay | Tengu, so we're good? anything to look at? | 13:33 |
| hjensas | cloudnull: ok, thanks. I'll do an update later today. | 13:34 |
| cloudnull | ++ | 13:34 |
| mwhahaha | ykarel: i don't think we referenceds tripleo-ansible in stein, that would be the question | 13:34 |
| EmilienM | right | 13:35 |
| ykarel | mwhahaha, it's not referenced, but wherever zuul tries to clone in branched projects, it will checkout stable/stein if it exists | 13:35 |
| mwhahaha | ykarel: i'm missing the original context, i'm just commenting ont he branchlessness | 13:35 |
| ykarel | mwhahaha, see comment from RDO zuul:- https://review.opendev.org/#/c/684305/ | 13:35 |
| ykarel | and i think that error is happening due to existance of stable/stein in tripleo-ansible which has missing roles | 13:36 |
| mwhahaha | ykarel: are we referencing tirpleo-ansible in zuul? | 13:36 |
| mwhahaha | that seems wrong | 13:36 |
| ykarel | mwhahaha, yes for container login part | 13:37 |
| ykarel | https://github.com/rdo-infra/review.rdoproject.org-config/blob/master/zuul.d/tripleo-rdo-base.yaml#L62 | 13:37 |
| mwhahaha | can you specify a branch there | 13:37 |
| ykarel | chandankumar, fyi ^^ | 13:37 |
| *** soniya29 has quit IRC | 13:38 | |
| weshay | iirc we were asked to reuse login.. | 13:38 |
| mwhahaha | also isn't the container login in the othe rrole? | 13:38 |
| mwhahaha | not in tripleo-ansible | 13:38 |
| weshay | panda, ^ | 13:38 |
| mwhahaha | it's fine to reuse it if you can specify a branch | 13:38 |
| ykarel | for required projects i think we can specify, but not sure for role | 13:39 |
| ykarel | jpena, any idea ^^ | 13:39 |
| panda | weshay: mwhahaha: we are using ansible-role-container-registry for docker and tripleo-ansible for podman | 13:39 |
| * mwhahaha points to why putting all roles in a single repo is a bad idea | 13:40 | |
| openstackgerrit | mathieu bultel proposed openstack/tripleo-heat-templates master: Use update_serial as an ansible variable https://review.opendev.org/684287 | 13:44 |
| jpena | ykarel: reading at https://zuul-ci.org/docs/zuul/user/config.html#attr-job.roles, it looks like we can use job.required-projects.override-checkout to select which branch we want to use for the role, see https://zuul-ci.org/docs/zuul/user/config.html#attr-job.required-projects.override-checkout | 13:46 |
| *** mcornea has joined #tripleo | 13:47 | |
| ykarel | jpena, hmm seems it should work | 13:48 |
| ykarel | zbr|ruck, sshnaidm ^^ | 13:48 |
| mwhahaha | we use that for puppet-ceph | 13:49 |
| sshnaidm | ykarel, yeah, we use it for ovb repo | 13:49 |
| sshnaidm | ykarel, but what we'll do in train then? | 13:50 |
| ykarel | sshnaidm, but i don't see ovb in roles: https://codesearch.rdoproject.org/?q=openstack-virtual-baremetal&i=nope&files=&repos= | 13:50 |
| sshnaidm | ykarel, it's in upstream, not in rdo | 13:51 |
| ykarel | seems that is used at quickstart | 13:51 |
| openstackgerrit | Francesco Pantano proposed openstack/tripleo-heat-templates master: Allow using registry authentication to pull ceph related containers https://review.opendev.org/682015 | 13:51 |
| mwhahaha | https://opendev.org/openstack/puppet-openstack-integration/src/branch/stable/queens/.zuul.yaml#L25 | 13:51 |
| mwhahaha | it might be on a branch | 13:51 |
| mwhahaha | which doesn't show up in codesearch | 13:51 |
| mwhahaha | (kinda annoying when you try to find this stuff) | 13:51 |
| sshnaidm | https://github.com/openstack/tripleo-ci/blob/master/zuul.d/base.yaml#L33-L41 | 13:52 |
| sshnaidm | ykarel, ^ | 13:52 |
| ykarel | sshnaidm, found, but it's in required-projects, not in roles: | 13:52 |
| sshnaidm | ykarel, right | 13:53 |
| openstackgerrit | Francesco Pantano proposed openstack/tripleo-heat-templates master: Allow using registry authentication to pull ceph related containers https://review.opendev.org/682015 | 13:53 |
| ykarel | hmm codesearch indexes only master branch :( | 13:53 |
| EmilienM | bandini, weshay: https://wiki.centos.org/Manuals/ReleaseNotes/CentOS8.1905#head-b7a63b88b2399f78f44d05be1ff8eb95bc679d5a | 13:54 |
| weshay | EmilienM, aye.. I think we're still missing qcow2 images | 13:54 |
| EmilienM | right | 13:54 |
| openstackgerrit | Kevin Carter (cloudnull) proposed openstack/tripleo-heat-templates master: Move nested bash script into a real file https://review.opendev.org/683270 | 13:55 |
| bandini | EmilienM: nice, can't wait :) | 13:55 |
| openstackgerrit | Kevin Carter (cloudnull) proposed openstack/tripleo-heat-templates master: Move basic global functions into ansible tasks https://review.opendev.org/683286 | 13:55 |
| openstackgerrit | Carlos Goncalves proposed openstack/diskimage-builder master: WIP: CentOS 8 support https://review.opendev.org/684308 | 13:55 |
| openstackgerrit | Kevin Carter (cloudnull) proposed openstack/tripleo-heat-templates master: Convert subprocess to a reusable method https://review.opendev.org/683429 | 13:55 |
| openstackgerrit | Kevin Carter (cloudnull) proposed openstack/tripleo-heat-templates master: Move globals into main level 0 case https://review.opendev.org/683590 | 13:55 |
| openstackgerrit | Hamdy Khader proposed openstack/tripleo-heat-templates master: Add heat templates to deploy mellanox neutron agent https://review.opendev.org/674486 | 13:56 |
| openstackgerrit | Hamdy Khader proposed openstack/tripleo-heat-templates master: Add support for Neutron Multi interface driver https://review.opendev.org/679558 | 13:56 |
| openstackgerrit | Emilien Macchi proposed openstack/paunch stable/stein: Add --cpuset-cpus support for both Docker and Podman https://review.opendev.org/684189 | 13:56 |
| Tengu | weshay: sooo I don't know what to do - BZ is ON_DEV, LP is "in progress" as well.... but apparently the script associate the LP id to the BZ.... | 13:58 |
| zbr|ruck | EmilienM: weshay i was told that official qcow2 images will arrive in few days. | 13:58 |
| Tengu | weshay: https://bugzilla.redhat.com/show_bug.cgi?id=1754416 vs https://bugs.launchpad.net/tripleo/+bug/1844856 - and card title is [CIX][BZ:1844856] <- that's the LP. | 13:59 |
| openstack | bugzilla.redhat.com bug 1754416 in puppet-tripleo "ovn-metadata-agent unable start haproxy" [High,On_dev] - Assigned to cjeanner | 13:59 |
| openstack | Launchpad bug 1844856 in tripleo "podman 1.5 - json-file logging specified but not supported. Choosing k8s-file logging instead" [High,In progress] - Assigned to Cédric Jeanneret (cjeanner) | 13:59 |
| weshay | oh... we have a bug # collision | 13:59 |
| Tengu | :D | 13:59 |
| *** chem has quit IRC | 14:00 | |
| mwhahaha | #startmeeting tripleo | 14:00 |
| openstack | Meeting started Tue Sep 24 14:00:12 2019 UTC and is due to finish in 60 minutes. The chair is mwhahaha. Information about MeetBot at http://wiki.debian.org/MeetBot. | 14:00 |
| mwhahaha | #topic agenda | 14:00 |
| mwhahaha | * Review past action items | 14:00 |
| mwhahaha | * One off agenda items | 14:00 |
| mwhahaha | * Squad status | 14:00 |
| mwhahaha | * Bugs & Blueprints | 14:00 |
| openstack | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 14:00 |
| mwhahaha | * Projects releases or stable backports | 14:00 |
| mwhahaha | * Specs | 14:00 |
| mwhahaha | * open discussion | 14:00 |
| *** openstack changes topic to " (Meeting topic: tripleo)" | 14:00 | |
| mwhahaha | Anyone can use the #link, #action and #info commands, not just the moderatorǃ | 14:00 |
| openstack | The meeting name has been set to 'tripleo' | 14:00 |
| mwhahaha | Hi everyone! who is around today? | 14:00 |
| *** openstack changes topic to "agenda (Meeting topic: tripleo)" | 14:00 | |
| dpeacock | o/ | 14:00 |
| weshay | 0/ | 14:00 |
| ksambor | o/ | 14:00 |
| Tengu | \o/ | 14:00 |
| ekultails | o/ | 14:00 |
| rlandy | o/ | 14:01 |
| owalsh | o/ | 14:01 |
| marios | hello | 14:01 |
| *** Vorrtex has joined #tripleo | 14:01 | |
| chandankumar | ykarel: which branch? master ? | 14:01 |
| chandankumar | \o/ | 14:02 |
| *** dsneddon has joined #tripleo | 14:02 | |
| rfolco | o/ | 14:02 |
| mschuppert | o/ | 14:02 |
| ykarel | chandankumar, yes | 14:03 |
| ykarel | o/ | 14:03 |
| chandankumar | patch coming | 14:03 |
| mwhahaha | alright let's get on with the meeting | 14:04 |
| mwhahaha | #topic review past action items | 14:04 |
| mwhahaha | mwhahaha to close out train-m3 - DONE | 14:04 |
| *** openstack changes topic to "review past action items (Meeting topic: tripleo)" | 14:04 | |
| mwhahaha | I went ahead and closed out the launchpad milestone and we release m3 last week | 14:04 |
| mwhahaha | any questions on that? | 14:04 |
| rh-jelabarre | o/ | 14:05 |
| mwhahaha | sounds like nope | 14:05 |
| mwhahaha | sync with weshay on ptl things - NOT DONE | 14:06 |
| mwhahaha | #action mwhahaha sync with weshay on ptl things | 14:06 |
| mwhahaha | weshay: do you have time later today so we can get just mark this one done | 14:06 |
| weshay | slackers | 14:06 |
| weshay | yes | 14:06 |
| mwhahaha | k i'll ping you a bit later | 14:06 |
| mwhahaha | that's it on the past items | 14:06 |
| mwhahaha | moving on to more exciting things | 14:06 |
| mwhahaha | #topic one off agenda items | 14:06 |
| mwhahaha | #link https://etherpad.openstack.org/p/tripleo-meeting-item | 14:06 |
| *** openstack changes topic to "one off agenda items (Meeting topic: tripleo)" | 14:06 | |
| mwhahaha | centos8/pcmk ; clues when it will be available? should ci switch back to pcmk when it becomes available? (gfidente/bandini) | 14:06 |
| mwhahaha | IMHO the answer to pcmk would be yes | 14:07 |
| mwhahaha | weshay, do we have any thoughts on the turn around time for centos8, is that a question for rdo as well? | 14:07 |
| *** dsneddon has quit IRC | 14:07 | |
| weshay | once we have a upstream nodepool image.. I'm hoping we'll see standalone running with in days.. and i'll provide an estimate then | 14:08 |
| mwhahaha | k | 14:08 |
| ykarel | RDO would need time to prepare | 14:08 |
| bandini | mwhahaha: yes, once we can use centos8 we should switch back ci to pcmk | 14:09 |
| bandini | and also work on ha by default | 14:09 |
| mwhahaha | we'll also finally be able to switch to podman on ovb as well | 14:09 |
| *** bdodd has joined #tripleo | 14:09 | |
| mwhahaha | and remove docker packaging | 14:09 |
| mwhahaha | so there's a bunch pending centos8 | 14:09 |
| mwhahaha | hopefully we can switch early in the U cycle | 14:09 |
| mjturek | rfolco: does this look good? ppc container builds are failing due to kolla issue and this is the fix I think https://review.opendev.org/#/c/683997/ | 14:09 |
| weshay | mjturek, in a mtg fyi | 14:10 |
| rfolco | mjturek, lets chat on #oooq | 14:10 |
| mjturek | whoops, I'm sorry | 14:10 |
| ykarel | yes hopefully for U cycle we will be able to go with CentOS | 14:10 |
| bandini | aye | 14:10 |
| ykarel | but more update will be there in coming days | 14:10 |
| ykarel | once it's actually out | 14:10 |
| mwhahaha | any other pacemaker/centos8 thoughts? | 14:10 |
| gfidente | weshay mwhahaha yeah and both scenario004 and scenario001 | 14:10 |
| *** lennyb has quit IRC | 14:11 | |
| cloudnull | o/ | 14:11 |
| ykarel | also we would need other SIGs to be ready apart from cloud sig | 14:11 |
| ykarel | like virt and ceph to be ready with centos8 | 14:11 |
| *** dsneddon has joined #tripleo | 14:12 | |
| mwhahaha | #link http://mirror.centos.org/centos-8/ | 14:12 |
| mwhahaha | looks like we're still pending qcow cloud images, but at least a repo is available | 14:12 |
| ykarel | regarding RDO and CentOS8, couple of info can be found at https://review.rdoproject.org/etherpad/p/moving-rdo-to-centos8 | 14:13 |
| mwhahaha | k | 14:14 |
| mwhahaha | I think that's about it for now. centos8 is still a WIP | 14:14 |
| mwhahaha | moving on | 14:14 |
| mwhahaha | <weshay> quay registry | 14:14 |
| weshay | ya... | 14:14 |
| weshay | just a quick FYI.. we're exploring using quay ( key ) lolz.. in upstream. patches are being tested now | 14:14 |
| weshay | expect containers to live in both docker.io and quay for some time | 14:15 |
| weshay | https://review.rdoproject.org/r/#/c/22445/ <- being tested | 14:15 |
| weshay | that's it | 14:15 |
| weshay | mwhahaha, ^ | 14:16 |
| *** skramaja has quit IRC | 14:16 | |
| mwhahaha | k | 14:16 |
| mwhahaha | moving on | 14:16 |
| weshay | next item is mine as well | 14:16 |
| mwhahaha | <weshay> thoughts on a daily periodic job to update master containers w/ update_repo: "delorean-current,quickstart-centos-ceph-nautilus" ? | 14:16 |
| mwhahaha | w/ standalone, or other job.. if it passes update containers in docker.io or quay | 14:16 |
| *** chem has joined #tripleo | 14:16 | |
| *** dsneddon has quit IRC | 14:17 | |
| weshay | ya.. I have arxcruz|rover and zbr|ruck looking at building a periodic (daily) job to update containers w/ | 14:17 |
| weshay | "delorean-current,quickstart-centos-ceph-nautilus" | 14:17 |
| sshnaidm | it shouldn't even pass, just build and push | 14:17 |
| mwhahaha | that doesnt' seem correct | 14:17 |
| *** dsneddon has joined #tripleo | 14:17 | |
| weshay | and push to docker.io / quay.. to shorten the loop w/ containers prep | 14:17 |
| weshay | sshnaidm, k.. we'll consider that | 14:17 |
| mwhahaha | shouldn't it match our includepkgs list (tripleo-repos current-tripleo-dev) | 14:17 |
| sshnaidm | anyway we update to these repos containers in every job, and it's merged only if it's passed | 14:18 |
| weshay | again.. we're seeing very long container prep times in CI | 14:18 |
| weshay | causing failures | 14:18 |
| mwhahaha | which is the equivalent of CI | 14:18 |
| mwhahaha | when was the last promotion? | 14:18 |
| weshay | some examples can be found here: https://snapshot.raintank.io/dashboard/snapshot/SjZ16RtMXZCFbEIOjgZ7N7KKJvOpKBhB?orgId=2 | 14:18 |
| weshay | master is at 4 days | 14:18 |
| bogdando | weshay: yes, it's about ~1h++ for each job | 14:18 |
| *** gbarros has quit IRC | 14:18 | |
| mwhahaha | hmm k | 14:18 |
| weshay | you can see.. we have infra issues w/ ovh and fortnebula | 14:18 |
| bogdando | only to update the containers with modify image role | 14:18 |
| weshay | most of the infra guys are out atm.. at ansible-fest | 14:19 |
| weshay | in rdo periodic jobs w/o yum updates.. containers prep is taking 5-6 minutes | 14:19 |
| ykarel | really? | 14:19 |
| *** michchap has quit IRC | 14:19 | |
| weshay | so we're still exploring ways to get our job times back down to a reasonable time... updates will come | 14:19 |
| weshay | ykarel, aye | 14:20 |
| mwhahaha | k | 14:20 |
| ykarel | okk nice, any link to logs | 14:20 |
| weshay | http://dashboard-ci.tripleo.org/d/si1tipHZk/jobs-exploration?orgId=1&from=now-12h&to=now&var-influxdb_filter=branch%7C%3D%7Cmaster | 14:20 |
| weshay | ykarel, look for the rdo provider | 14:20 |
| weshay | hrm.. let me get you a better link | 14:21 |
| sshnaidm | container build takes about 15 minutes? maybe worth just to build them and not to update? :) | 14:21 |
| ykarel | i see 25,21, 15 minutes | 14:21 |
| *** hberaud is now known as hberaud|school-r | 14:21 | |
| mwhahaha | you mean building them in CI? | 14:21 |
| sshnaidm | yeah | 14:21 |
| mwhahaha | i think it'll probably take longer than that on some of these clouds | 14:22 |
| mwhahaha | though maybe we would get better cache results for packages than containers | 14:22 |
| *** dsneddon has quit IRC | 14:22 | |
| mwhahaha | we could try it | 14:22 |
| weshay | ykarel, use https://snapshot.raintank.io/dashboard/snapshot/S9m9pvyn4iPSELe0bd566ZlMv4KvVWwO?orgId=2 | 14:22 |
| mwhahaha | but it requires an undercloud to build | 14:22 |
| mwhahaha | because $reasons | 14:22 |
| mwhahaha | i'm not sure it'd be a simple switch | 14:22 |
| *** lennyb has joined #tripleo | 14:23 | |
| weshay | http://logs.rdoproject.org/openstack-periodic-master/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-rhel-8-scenario001-standalone-master/7b77387/logs/undercloud/home/zuul/standalone_deploy.log.txt.gz | 14:23 |
| bogdando | ykarel: just checked, http://logs.rdoproject.org/openstack-periodic-master/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-rhel-8-scenario001-standalone-master/7b77387/logs/undercloud/var/log/tripleo-container-image-prepare.log.txt.gz is exactly 5 minutes | 14:24 |
| ykarel | rhel8 | 14:24 |
| bogdando | ykarel: 2019-09-24 01:32:26 - 2019-09-24 01:37:21 | 14:24 |
| weshay | doesn't matter | 14:24 |
| mwhahaha | anyway sounds like we need to continue investigating our options around containers | 14:25 |
| mwhahaha | anythign else? | 14:25 |
| weshay | nope | 14:25 |
| weshay | aye | 14:25 |
| mwhahaha | k moving on | 14:25 |
| mwhahaha | #topic Squad status | 14:25 |
| mwhahaha | ci | 14:25 |
| mwhahaha | #link https://etherpad.openstack.org/p/tripleo-ci-squad-meeting | 14:25 |
| mwhahaha | upgrade | 14:25 |
| mwhahaha | #link https://etherpad.openstack.org/p/tripleo-upgrade-squad-status | 14:25 |
| *** openstack changes topic to "Squad status (Meeting topic: tripleo)" | 14:25 | |
| mwhahaha | edge | 14:25 |
| mwhahaha | #link https://etherpad.openstack.org/p/tripleo-edge-squad-status | 14:25 |
| mwhahaha | integration | 14:25 |
| mwhahaha | #link https://etherpad.openstack.org/p/tripleo-integration-squad-status | 14:25 |
| mwhahaha | validations | 14:25 |
| mwhahaha | #link https://etherpad.openstack.org/p/tripleo-validations-squad-status | 14:25 |
| mwhahaha | networking | 14:25 |
| mwhahaha | #link https://etherpad.openstack.org/p/tripleo-networking-squad-status | 14:25 |
| mwhahaha | transformation | 14:25 |
| mwhahaha | #link https://etherpad.openstack.org/p/tripleo-ansible-agenda | 14:25 |
| mwhahaha | any status highlights? | 14:25 |
| ykarel | weshay, i see http://dashboard-ci.tripleo.org/d/si1tipHZk/jobs-exploration?orgId=1&from=now-12h&to=now&var-influxdb_filter=branch%7C%3D%7Cmaster&var-influxdb_filter=provider%7C%3D%7Crdo-cloud and in this i see 25,21,15 | 14:25 |
| ykarel | the other link seems not working | 14:25 |
| weshay | ykarel, k.. let's go out of channel | 14:26 |
| ykarel | ack | 14:26 |
| *** brault has joined #tripleo | 14:26 | |
| bogdando | ykarel: but that was prolly a bad example, there was nothing to update at all :) | 14:27 |
| * ykarel checked what's with rhel8, there is dnf issue, so it didn't run completely | 14:27 | |
| weshay | there are never container updates in periodic.. | 14:27 |
| openstackgerrit | Karthik S proposed openstack/tripleo-heat-templates master: Fixed the path for neutron-base.yaml https://review.opendev.org/684318 | 14:28 |
| mwhahaha | ok moving on | 14:30 |
| mwhahaha | #topic bugs & blueprints | 14:30 |
| mwhahaha | #link https://launchpad.net/tripleo/+milestone/train-rc1 | 14:30 |
| mwhahaha | For Train we currently have 27 blueprints and 521 (+15) open Launchpad Bugs. 28 train-rc1, 493 ussuri-1. 91 (-75) open Storyboard bugs. | 14:30 |
| mwhahaha | #link https://storyboard.openstack.org/#!/project_group/76 | 14:30 |
| *** openstack changes topic to "bugs & blueprints (Meeting topic: tripleo)" | 14:30 | |
| mwhahaha | I cleaned up the tripleo-ui story board items last week. Additionally I closed out milestone 3. I will review the open rc1 bugs to move them to ussuri-1 if they aren't really blocking the rc1. | 14:30 |
| mwhahaha | #action mwhahaha review train-rc1 open bugs | 14:30 |
| mwhahaha | any questions/concerns around bugs/blueprints? | 14:31 |
| apetrich | amoralej, so I've narrowed down the issue to this https://i.imgflip.com/3bex5p.jpg it is a bit weird. I'm trying to understand what causes that | 14:32 |
| mwhahaha | sounds like that's in reference to https://bugs.launchpad.net/tripleo/+bug/1845181 | 14:33 |
| openstack | Launchpad bug 1845181 in tripleo "Problem with tripleo_common git functionalities and eventlet 0.25.1" [Medium,In progress] - Assigned to Adriano Petrich (apetrich) | 14:33 |
| mwhahaha | also an appropriate use of memes | 14:33 |
| mwhahaha | anything else? | 14:33 |
| apetrich | mwhahaha, it is. | 14:33 |
| apetrich | mwhahaha, so tempted to add that to the bug | 14:34 |
| *** hberaud|school-r is now known as hberaud | 14:34 | |
| openstackgerrit | Merged openstack/tripleo-heat-templates stable/stein: Enable deep_compare by default for stonith resources https://review.opendev.org/683409 | 14:34 |
| openstackgerrit | Merged openstack/tripleo-heat-templates stable/rocky: Enable deep_compare by default for stonith resources https://review.opendev.org/683404 | 14:34 |
| openstackgerrit | Merged openstack/tripleo-heat-templates stable/queens: Enable deep_compare by default for stonith resources https://review.opendev.org/683405 | 14:34 |
| openstackgerrit | Merged openstack/tripleo-validations master: Update the constraints url https://review.opendev.org/684161 | 14:34 |
| openstackgerrit | Merged openstack/tripleo-upgrade stable/stein: Fix "unbound" variable https://review.opendev.org/684191 | 14:34 |
| weshay | \0/ | 14:35 |
| mwhahaha | alright moving on | 14:35 |
| mwhahaha | #topic projects releases or stable backports | 14:35 |
| *** openstack changes topic to "projects releases or stable backports (Meeting topic: tripleo)" | 14:35 | |
| mwhahaha | any release related topics? | 14:35 |
| *** jpich has joined #tripleo | 14:37 | |
| mwhahaha | moving on | 14:37 |
| mwhahaha | #topic specs | 14:37 |
| mwhahaha | #link https://review.openstack.org/#/q/project:openstack/tripleo-specs+status:open | 14:37 |
| *** openstack changes topic to "specs (Meeting topic: tripleo)" | 14:37 | |
| mwhahaha | the U cycle will be starting soon, don't forget to update your specs | 14:37 |
| mwhahaha | any spec related items? | 14:38 |
| mwhahaha | #topic open discussion | 14:40 |
| *** openstack changes topic to "open discussion (Meeting topic: tripleo)" | 14:40 | |
| mwhahaha | anything else? | 14:40 |
| *** kopecmartin is now known as kopecmartin|off | 14:40 | |
| mwhahaha | alright thanks everyone | 14:43 |
| mwhahaha | #endmeeting | 14:43 |
| *** openstack changes topic to "CI Status: Orange (be kind with recheck and +A, watch for zuul queue) | community irc meeting Tues@1400 UTC - tripleo-ci-community meeting Tues@1330 UTC | https://docs.openstack.org/tripleo-docs" | 14:43 | |
| openstack | Meeting ended Tue Sep 24 14:43:03 2019 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 14:43 |
| openstack | Minutes: http://eavesdrop.openstack.org/meetings/tripleo/2019/tripleo.2019-09-24-14.00.html | 14:43 |
| openstack | Minutes (text): http://eavesdrop.openstack.org/meetings/tripleo/2019/tripleo.2019-09-24-14.00.txt | 14:43 |
| openstack | Log: http://eavesdrop.openstack.org/meetings/tripleo/2019/tripleo.2019-09-24-14.00.log.html | 14:43 |
| *** hamdyk has quit IRC | 14:46 | |
| openstackgerrit | Sagi Shnaidman proposed openstack/openstack-virtual-baremetal master: DNM: test master https://review.opendev.org/684364 | 14:46 |
| *** leanderthal has joined #tripleo | 14:48 | |
| *** mdbooth has joined #tripleo | 14:53 | |
| mdbooth | odyssey4me: Hey there. I'm trying to patch some code which you touched most recently ;) https://review.opendev.org/#/c/683972/ | 14:54 |
| mdbooth | odyssey4me: I'm getting a CI failure which afaict must be bogus, because I can't see where the failing job loads the ipxe-boot image. | 14:57 |
| openstackgerrit | Carlos Goncalves proposed openstack/diskimage-builder master: WIP: CentOS 8 support https://review.opendev.org/684308 | 14:58 |
| mdbooth | IOW I don't think the job is testing my change, which is also an interesting observation. | 14:58 |
| mwhahaha | weshay, cloudnull, EmilienM: fun fact, we duplicate the layers for all the containers when we write them out | 14:58 |
| openstackgerrit | mathieu bultel proposed openstack/tripleo-validations master: Fix get plan_name default value for inventory script https://review.opendev.org/684369 | 14:59 |
| sshnaidm | mdbooth, the job doesn't do anything with this ipxe image, it uses the hosted one on CI tenant | 14:59 |
| weshay | mwhahaha, when we write them out.. at build time? | 14:59 |
| mwhahaha | weshay, cloudnull, EmilienM: when we upload them to the container image registry, we dont' de-dupe already fetched laywers | 14:59 |
| *** ratailor has joined #tripleo | 15:00 | |
| mwhahaha | weshay, cloudnull, EmilienM: http://paste.openstack.org/show/779191/ | 15:00 |
| weshay | marios, panda rfolco rlandy fyi ^ | 15:00 |
| mwhahaha | so there's an optimization to be had if we cache the sha's and symlink them | 15:00 |
| *** tosky has quit IRC | 15:00 | |
| mwhahaha | so we don't fetch all the layers over and over again | 15:00 |
| * mwhahaha flips tables | 15:00 | |
| mwhahaha | well we may not be refetching them (we shouldn't be) but we are writing them out over and over | 15:01 |
| mwhahaha | and each time we write out the base layer it's taking like 24 sec on my test vm | 15:01 |
| cloudnull | :\ | 15:01 |
| *** pcaruana has quit IRC | 15:01 | |
| weshay | mwhahaha, before you go all steve ballmer, is that something we can change in the container push? | 15:02 |
| mwhahaha | no | 15:02 |
| mwhahaha | it's code we've written | 15:02 |
| mwhahaha | if only we didn't have to implement out own container registry | 15:02 |
| mdbooth | sshnaidm: That would make sense based on the failure of my grep-fu ;) | 15:03 |
| odyssey4me | mdbooth: looking | 15:03 |
| mwhahaha | it looks liek there might be soem de-dupe code but i need to dig deeper | 15:03 |
| *** paramite has quit IRC | 15:03 | |
| *** ratailor has quit IRC | 15:03 | |
| mwhahaha | oh i take that back, we do de-dupe it to an extent | 15:03 |
| *** abishop has quit IRC | 15:04 | |
| mwhahaha | but it's still taking way to long to write out these files | 15:04 |
| marios | weshay: whats the paste ... that quay? | 15:04 |
| Tengu | mwhahaha: rewrite it in go | 15:04 |
| * Tengu already out | 15:04 | |
| marios | weshay: (also on the phone) | 15:04 |
| *** ccamacho has quit IRC | 15:05 | |
| mwhahaha | Tengu: oh you | 15:05 |
| *** gfidente has quit IRC | 15:05 | |
| Tengu | :D | 15:05 |
| mwhahaha | looks like we might be hardlinking some stuff | 15:06 |
| mwhahaha | but i think it's the streaming of the content in python that might be causing issues | 15:06 |
| *** gbarros has joined #tripleo | 15:08 | |
| mwhahaha | https://github.com/openstack/tripleo-common/blob/master/tripleo_common/image/image_export.py#L91-L99 | 15:08 |
| mwhahaha | i think that's where some of the perf problems are | 15:09 |
| openstackgerrit | Francesco Pantano proposed openstack/tripleo-heat-templates master: Allow using registry authentication to pull ceph related containers https://review.opendev.org/682015 | 15:09 |
| mdbooth | odyssey4me: That particular change was motivated by cutting down the development cycle whilst trying to produce the follow-up patch, btw. | 15:10 |
| Tengu | mwhahaha: well, the hashlib.sha256() calls might as well create some slowdowns.. | 15:10 |
| *** Vorrtex has quit IRC | 15:10 | |
| mdbooth | odyssey4me: 3 second rebuild vs 15 minutes or something like that. | 15:10 |
| odyssey4me | mdbooth: yeah, it looks like it's got all the right bits as far as I can see - when I did the last modification I honestly thought that using dib was silly, but went with it anyway | 15:11 |
| Tengu | mwhahaha: maybe we can add buffering=0 to the "open" ? that should allow a direct write instead of buffering/waiting | 15:11 |
| mwhahaha | maybe, going to poke at that | 15:11 |
| odyssey4me | mdbooth: as sshnaidm the failing job is in RDO and defined outside of the openstack repositories... so you'll have to hunt down the third party CI owner | 15:12 |
| odyssey4me | mdbooth: although, it's possible that the issue is the deletion of the resulting qcow2 file from the repo | 15:13 |
| *** saneax|AFK has quit IRC | 15:14 | |
| mdbooth | odyssey4me: That was my primary suspect if it was an issue with my patch. | 15:15 |
| mdbooth | odyssey4me: Ok, thanks. Appreciate your review on the patch and the follow-up addition to your ipxe boot script, btw. | 15:15 |
| odyssey4me | mdbooth: while I'm not a fan of including the binary - it seems that there may be things relying on it... so we're kinda stuck for now with keeping it there | 15:15 |
| sshnaidm | mdbooth, seems like master is broken http://logs.rdoproject.org/72/683972/1/openstack-check/tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-ovb-master/aa37cca/job-output.txt.gz | 15:16 |
| sshnaidm | bnemec, ^^ | 15:16 |
| mdbooth | odyssey4me: I can certainly put it back in. However, if we can find what is relying on it, I'm also happy to fix that instead. | 15:16 |
| sshnaidm | ovb stack is not created http://logs.rdoproject.org/72/683972/1/openstack-check/tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-ovb-master/aa37cca/logs/failed_ovb_stack.log | 15:18 |
| sshnaidm | Resource CREATE failed: WaitConditionFailure: resources.baremetal_env.resources.bmc.resources.bmc_wait_condition: Signal 1 received | 15:18 |
| *** brault has quit IRC | 15:18 | |
| sshnaidm | bnemec, any ideas why could it happen maybe? ^ | 15:18 |
| *** brault has joined #tripleo | 15:19 | |
| *** brault has quit IRC | 15:19 | |
| bnemec | sshnaidm: Sounds like bmc deployment failed. You'll need to look at the console log to see why. | 15:19 |
| *** lmiccini has quit IRC | 15:23 | |
| *** leanderthal has quit IRC | 15:23 | |
| openstackgerrit | Kevin Carter (cloudnull) proposed openstack/tripleo-ansible master: Pivot ovs-dpdk molecule to ansible tasks https://review.opendev.org/684379 | 15:25 |
| openstackgerrit | Kevin Carter (cloudnull) proposed openstack/tripleo-ansible master: Add kernelargs and tuned core configuration https://review.opendev.org/679809 | 15:25 |
| *** lennyb has quit IRC | 15:25 | |
| sshnaidm | bnemec, if node is not created, so it doesn't have console - isn't it? | 15:28 |
| bnemec | sshnaidm: If the node wasn't created I would have expected it to fail on that resource, not the wait condition. | 15:30 |
| bnemec | But yeah, if the node is missing then that's a problem. | 15:30 |
| *** lennyb has joined #tripleo | 15:30 | |
| *** iurygregory has quit IRC | 15:32 | |
| sshnaidm | bnemec, seems like it's missing.. | 15:32 |
| sshnaidm | ok, will create a bug, seems like need to check with reproducer | 15:32 |
| sshnaidm | fyi https://bugs.launchpad.net/tripleo/+bug/1845225 | 15:34 |
| openstack | Launchpad bug 1845225 in tripleo "OVB master branch job is broken, can't create bmc" [High,Triaged] | 15:34 |
| sshnaidm | weshay, ^ | 15:35 |
| openstackgerrit | Adriano Petrich proposed openstack/tripleo-common master: Make gitpython and eventlet work with eventlet 0.25.1 https://review.opendev.org/684380 | 15:35 |
| sshnaidm | mdbooth, so for now you can ignore 3party ci on master, see https://bugs.launchpad.net/tripleo/+bug/1845225 | 15:37 |
| openstack | Launchpad bug 1845225 in tripleo "OVB master branch job is broken, can't create bmc" [High,Triaged] | 15:37 |
| weshay | sshnaidm, thanks | 15:37 |
| cloudnull | mwhahaha I think you may be right about the performance issue with writing content, I think we're in some cases processing the same data stream 2, sometime 3, times per container. | 15:38 |
| mwhahaha | cloudnull: i'm seeing delays in teh writing of chunks | 15:39 |
| mwhahaha | i added some debugging in and it's hanging every one in a while | 15:39 |
| cloudnull | we have things like so https://github.com/openstack/tripleo-common/blob/master/tripleo_common/image/image_export.py#L91-L99, | 15:39 |
| mwhahaha | it might be a buffering thign on output | 15:39 |
| mwhahaha | cloudnull: we're reading in chunks of 1meg | 15:39 |
| mdbooth | weshay: Did I break it, btw, or was it already broken? | 15:39 |
| cloudnull | so its coming from the requests stream generator, to our generator, to the chunk writing loop | 15:40 |
| mwhahaha | cloudnull: yea, i'm going to try turning off the write buffer | 15:40 |
| cloudnull | ++ | 15:40 |
| weshay | mdbooth, looks like it was broken before you got there... https://review.opendev.org/#/c/684364/ | 15:40 |
| weshay | sshnaidm, has a no-op change.. and it's boosted | 15:41 |
| * mwhahaha waits for ctl+c to actualy process | 15:41 | |
| openstackgerrit | Harald Jensås proposed openstack/puppet-tripleo master: Add 'ipversion' to firewall/rule.pp https://review.opendev.org/684384 | 15:42 |
| cloudnull | and it looks like we have two different stream processors, the same, but different thing seems to be happening here too - https://github.com/openstack/tripleo-common/blob/master/tripleo_common/image/image_uploader.py#L1615-L1644 | 15:42 |
| sshnaidm | weshay, just a month before it was ok: https://review.opendev.org/#/c/675601/ | 15:42 |
| mdbooth | weshay: \o/ ;) | 15:42 |
| openstackgerrit | Harald Jensås proposed openstack/tripleo-heat-templates master: Add DHCPv6 rules + set ipversion for dhcp rules https://review.opendev.org/684385 | 15:42 |
| openstackgerrit | Harald Jensås proposed openstack/tripleo-ansible master: Support 'ipversion' property in rules https://review.opendev.org/684277 | 15:44 |
| openstackgerrit | Harald Jensås proposed openstack/tripleo-ansible master: Open ipv6-icmp traffic by default https://review.opendev.org/684278 | 15:44 |
| openstackgerrit | Harald Jensås proposed openstack/openstack-virtual-baremetal master: Add press Ctrl-B for iPXE command line https://review.opendev.org/684387 | 15:47 |
| openstackgerrit | Harald Jensås proposed openstack/openstack-virtual-baremetal master: Add possiblity to build IPv6 IPXE iamge https://review.opendev.org/684388 | 15:47 |
| mwhahaha | cloudnull, EmilienM: so i think the sluggish writing is actualy a function of the speed we pull from docker.io (and mirrors) | 15:51 |
| mwhahaha | cloudnull, EmilienM: since we're using streams and fixed chunk sizes, we hang until the chunk is full, then we write it out. | 15:51 |
| mwhahaha | cloudnull, EmilienM: What i'm seeing is that writting occurs, then hangs in the middle, then continues | 15:51 |
| cloudnull | do you think we're being rate-limited from docker? | 15:51 |
| mwhahaha | cloudnull, EmilienM: I think it's because we're using the fixed 1meg chunksize for the stream | 15:52 |
| mwhahaha | cloudnull: i don't think it's a ratelimit, i think it's just a transit issue | 15:52 |
| mwhahaha | the mirroring should help but it doesn't seem to be, i wonder if cache hits are bad or something | 15:52 |
| EmilienM | should we try to reduce the chunk size? | 15:52 |
| mwhahaha | https://opendev.org/openstack/tripleo-common/src/branch/master/tripleo_common/image/image_uploader.py#L1350-L1360 | 15:52 |
| mwhahaha | we might want to drop it for the url requests | 15:53 |
| mwhahaha | because it'll return what it has available if not specified | 15:53 |
| mwhahaha | https://2.python-requests.org/en/master/api/#requests.Response.iter_content | 15:53 |
| *** yprokule has quit IRC | 15:53 | |
| mwhahaha | let me try setting to None in my env to see if I see the studdering downloads/writing | 15:53 |
| *** ykarel is now known as ykarel|afk | 15:54 | |
| cloudnull | I wonder if setting it to something like 8192 instead of 1048576 would be a better overall chuck size? | 15:54 |
| mwhahaha | on my test vm, i'm not iowating on disk so I don't think it's pending disk wait | 15:55 |
| mwhahaha | which is why i think it's network | 15:55 |
| mwhahaha | i think we might get better perf w/o it specified | 15:56 |
| mwhahaha | i'm seeing a wider range of chunks | 15:56 |
| mwhahaha | 8416890 | 15:56 |
| mwhahaha | 37687476 | 15:56 |
| openstackgerrit | Takashi Kajinami proposed openstack/tripleo-heat-templates master: Use OS::TripleO::Services::Logging::NovaLibvirt to configure libvirt logging https://review.opendev.org/683752 | 15:56 |
| openstackgerrit | Takashi Kajinami proposed openstack/tripleo-heat-templates master: DNM: Test libvirt logging https://review.opendev.org/683811 | 15:56 |
| mwhahaha | 35854360 | 15:56 |
| cloudnull | ++ the default of 512 should perform fine, and we can raise it if needed | 15:57 |
| mwhahaha | i specified None actually | 15:57 |
| mwhahaha | and let it figure it out | 15:57 |
| mwhahaha | i think it returns what is available | 15:57 |
| cloudnull | +1 | 15:57 |
| *** marios has quit IRC | 15:57 | |
| * mwhahaha proposes patch | 15:57 | |
| openstackgerrit | Takashi Kajinami proposed openstack/tripleo-heat-templates master: Use OS::TripleO::Services::Logging::NovaLibvirt to configure libvirt logging https://review.opendev.org/683752 | 15:58 |
| openstackgerrit | Takashi Kajinami proposed openstack/tripleo-heat-templates master: DNM: Test libvirt logging https://review.opendev.org/683811 | 15:58 |
| cloudnull | "chunk_size must be of type int or None. A value of None will function differently depending on the value of stream. stream=True will read data as it arrives in whatever size the chunks are received. If stream=False, data is returned as a single chunk." | 15:58 |
| cloudnull | None seems like the right option for our use | 15:58 |
| cloudnull | nice find/catch | 15:58 |
| openstackgerrit | Alex Schultz proposed openstack/tripleo-common master: Remove chunk size for url stream https://review.opendev.org/684391 | 15:59 |
| mwhahaha | lets see how that works, i'll fix tests ands tuff in a bit | 15:59 |
| *** suuuper has quit IRC | 16:01 | |
| *** panda is now known as panda|bbl | 16:02 | |
| openstackgerrit | Brent Eagles proposed openstack/tripleo-quickstart master: Add support for running tempest against Octavia featuresets https://review.opendev.org/669666 | 16:03 |
| openstackgerrit | Brent Eagles proposed openstack/tripleo-quickstart master: Set the amphora image to stein branch https://review.opendev.org/678899 | 16:03 |
| bogdando | mwhahaha, cloudnull: yea, that was indeed the great finding | 16:04 |
| * bogdando rants around for now solving images streaming issues for containers... | 16:05 | |
| bogdando | how long to implement a registry? :) | 16:05 |
| mwhahaha | yeaaaa | 16:05 |
| *** alexmcleod has quit IRC | 16:06 | |
| mwhahaha | if only there was a quality single host registry solution that didn't require the a super computer to serve files | 16:06 |
| *** rpittau is now known as rpittau|afk | 16:08 | |
| *** cylopez has quit IRC | 16:10 | |
| *** udesale has quit IRC | 16:10 | |
| apetrich | amoralej, EmilienM I have a patch up for https://bugs.launchpad.net/tripleo/+bug/1845181 from my local testing it fixes the issue. | 16:12 |
| openstack | Launchpad bug 1845181 in tripleo "Problem with tripleo_common git functionalities and eventlet 0.25.1" [Medium,In progress] - Assigned to Adriano Petrich (apetrich) | 16:12 |
| apetrich | I'm going to clock out now. if you need anything just ping here that I will read before tomorrow | 16:13 |
| amoralej | apetrich, nice, let me do a quick test | 16:13 |
| apetrich | amoralej, I'm here waiting for the test then :) | 16:13 |
| amoralej | you can go, i'll leave the results here | 16:14 |
| apetrich | amoralej, cheers | 16:14 |
| *** dtantsur is now known as dtantsur|afk | 16:15 | |
| mwhahaha | meh still seeing stalls in processing, not sure where it's hanging now. I guess need more debug logs | 16:19 |
| *** brault has joined #tripleo | 16:20 | |
| amoralej | apetrich, i still see the issue with your patch :( | 16:21 |
| *** jpich has quit IRC | 16:21 | |
| apetrich | amoralej, let me double check. I probably messed something copying | 16:22 |
| amoralej | apetrich, how are you reproducing? | 16:22 |
| openstackgerrit | Harald Jensås proposed openstack/tripleo-heat-templates master: Add DHCPv6 rules + set ipversion for dhcp rules https://review.opendev.org/684385 | 16:22 |
| apetrich | amoralej, two scripts based on the bug report. but I was coding on the one of the scripts and when it worked I copied to the tripleo-common. let me do it proper | 16:23 |
| amoralej | ok | 16:23 |
| amoralej | note it only fails when trying to commit the first time | 16:23 |
| amoralej | if there is a commit, it works | 16:24 |
| apetrich | amoralej, http://paste.openstack.org/show/779198/ | 16:24 |
| *** holser has quit IRC | 16:24 | |
| amoralej | dunno why | 16:24 |
| apetrich | amoralej, that is due to the gitpython | 16:24 |
| apetrich | if it finds a head it uses it, but for the first commit there's no head and it lazy loads git.refs and creates a head | 16:24 |
| amoralej | ok | 16:25 |
| apetrich | it is failing on the get path for the head creation | 16:25 |
| *** brault has quit IRC | 16:27 | |
| amoralej | apetrich, that script fails in my env with eventlet-0.25.1 | 16:27 |
| apetrich | amoralej, wha | 16:27 |
| apetrich | amoralej, not for me. with eventlet 0.24 and 0.25 | 16:27 |
| *** brault has joined #tripleo | 16:28 | |
| *** brault has quit IRC | 16:28 | |
| apetrich | amoralej, ok. let me double check everything. | 16:28 |
| *** brault has joined #tripleo | 16:28 | |
| *** bogdando has quit IRC | 16:28 | |
| amoralej | http://paste.openstack.org/show/779200/ | 16:28 |
| amoralej | mmm | 16:28 |
| amoralej | with 0.24.1 passes | 16:28 |
| amoralej | let me think what i may have done wrong | 16:28 |
| *** jaosorior has joined #tripleo | 16:29 | |
| *** brault has quit IRC | 16:29 | |
| amoralej | i'm running in fedora with python2, you are using on centos7? | 16:29 |
| apetrich | amoralej, no you are right. I did something weird. it is late | 16:29 |
| amoralej | ok, no problem | 16:30 |
| *** hberaud is now known as hberaud|gone | 16:30 | |
| apetrich | amoralej, I see what I did. I didn't save a file from eventlet that I was debugging. my patch doesn't work. I will plow through it some more tomorrow | 16:31 |
| apetrich | amoralej, EmilienM sorry about that. | 16:31 |
| *** brault has joined #tripleo | 16:32 | |
| amoralej | no problem, can wait | 16:32 |
| *** yoctozepto has quit IRC | 16:35 | |
| *** yoctozepto has joined #tripleo | 16:40 | |
| *** tesseract has quit IRC | 16:46 | |
| mwhahaha | cloudnull, EmilienM: i found our usage of futures is less than optimal. I shall propose a fix | 16:46 |
| *** jaosorior has quit IRC | 16:51 | |
| openstackgerrit | Merged openstack/paunch stable/stein: Revert "Add --cpuset-cpus support for both Docker and Podman" https://review.opendev.org/684180 | 16:54 |
| *** panda|bbl is now known as panda | 16:56 | |
| *** Vorrtex has joined #tripleo | 16:56 | |
| *** waleedm has joined #tripleo | 16:56 | |
| *** ramishra has quit IRC | 16:57 | |
| openstackgerrit | Takashi Kajinami proposed openstack/tripleo-heat-templates master: Use OS::TripleO::Services::Logging::NovaLibvirt to configure libvirt logging https://review.opendev.org/683752 | 16:59 |
| openstackgerrit | Takashi Kajinami proposed openstack/tripleo-heat-templates master: DNM: Test libvirt logging https://review.opendev.org/683811 | 16:59 |
| *** ricolin has quit IRC | 17:03 | |
| openstackgerrit | Adriano Petrich proposed openstack/tripleo-common master: Make gitpython and eventlet work with eventlet 0.25.1 https://review.opendev.org/684380 | 17:16 |
| apetrich | amoralej, was able to squeeze a change before bedtime for the kids. See you tomorrow | 17:17 |
| amoralej | ack, see you | 17:17 |
| amoralej | apetrich, worked fine in my env | 17:21 |
| openstackgerrit | Merged openstack/paunch stable/stein: Improve volume validation https://review.opendev.org/683386 | 17:21 |
| apetrich | amoralej, nice to know :) | 17:27 |
| EmilienM | weshay: I'm going to add tripleorocky & tripleoqueens namespaces to quay.io, you'll get invit on the triploe ci account again | 17:31 |
| *** brault has quit IRC | 17:31 | |
| EmilienM | weshay: if you can accept so I finish the process and we won't have perm errors | 17:32 |
| weshay | k | 17:32 |
| *** jpena is now known as jpena|off | 17:32 | |
| * weshay chatting w/ the folks now.. they will take the patch over | 17:32 | |
| *** hjensas has quit IRC | 17:36 | |
| *** waleedm has quit IRC | 17:36 | |
| EmilienM | weshay: thanks | 17:43 |
| EmilienM | weshay: master/stein/rocky/queens is allset | 17:44 |
| EmilienM | weshay: I'm manually pushing current-tripleo in tripleomaster now | 17:45 |
| weshay | rlandy, panda rfolco ^ | 17:57 |
| weshay | EmilienM, ya.. let's rekick the test patch that consumes containers.. now that we have the proxy | 17:58 |
| * weshay logs into quay | 17:58 | |
| *** psachin has quit IRC | 17:59 | |
| *** ykarel|afk is now known as ykarel|away | 18:00 | |
| *** avivgta has quit IRC | 18:01 | |
| *** amoralej is now known as amoralej|off | 18:02 | |
| *** dsneddon has joined #tripleo | 18:05 | |
| *** Vorrtex has quit IRC | 18:07 | |
| *** Goneri has joined #tripleo | 18:10 | |
| openstackgerrit | Alex Schultz proposed openstack/tripleo-common master: Improve ThreadPoolExecutor usage https://review.opendev.org/684411 | 18:11 |
| mwhahaha | cloudnull, EmilienM -^ | 18:11 |
| *** ykarel|away has quit IRC | 18:15 | |
| EmilienM | mwhahaha: looking | 18:18 |
| mwhahaha | it may not address everything but should give us a mor accurate picture of when things finish | 18:19 |
| EmilienM | nicely don | 18:20 |
| *** jtomasek has quit IRC | 18:21 | |
| *** Vorrtex has joined #tripleo | 18:22 | |
| *** bdodd has quit IRC | 18:22 | |
| *** bdodd has joined #tripleo | 18:24 | |
| *** bdodd has quit IRC | 18:24 | |
| *** morazi has quit IRC | 18:25 | |
| mwhahaha | wonder if we should adjust the upload workers as well | 18:26 |
| EmilienM | should we stop tripleo jobs in check to give a chance to your patches? | 18:28 |
| weshay | can you do that? | 18:28 |
| EmilienM | weshay: if I abandon them yes | 18:29 |
| EmilienM | i suspect alex's patches which fix the world will not be tested before tomorrow if we do nothing | 18:30 |
| weshay | we're not bleeding in the gate, check queue is at 7hrs.. just fyi | 18:30 |
| EmilienM | yes I know | 18:30 |
| EmilienM | which is why i'm saying that | 18:30 |
| EmilienM | thoughts? | 18:31 |
| *** ykarel|away has joined #tripleo | 18:35 | |
| EmilienM | weshay: tripleomaster -- pushed. now pushing stein rocky and queens | 18:35 |
| *** d0ugal has quit IRC | 18:37 | |
| EmilienM | weshay: ah I'm blocked, please accept invit to join https://quay.io/organization/tripleostein/teams/tripleoci?tab=teams | 18:37 |
| EmilienM | mwhahaha: ^ thought son check? | 18:37 |
| *** Goneri has quit IRC | 18:37 | |
| mwhahaha | no let's not do that for now | 18:38 |
| mwhahaha | i'm running tests locally and will likely ahve more improvements | 18:38 |
| EmilienM | k | 18:38 |
| mwhahaha | at the moment i think well only be able to make certain smaller incremental fixes | 18:38 |
| mwhahaha | maybe the cumulative fixes will help but i don't think we have a single one to fix it all yet | 18:39 |
| cloudnull | mwhahaha nice | 18:39 |
| EmilienM | mwhahaha: you still believe in quay.io switch? | 18:40 |
| cloudnull | we ran into issues with futures not raising an exception when there's a fault in a thread, we had to check change that in buildha.py to inspect the jobs - https://github.com/openstack/tripleo-common/blob/master/tripleo_common/image/builder/buildah.py#L170-L193 - but if this works then its a much cleaner solution | 18:40 |
| mwhahaha | cloudnull: not sure if we've ever had an exception in that bit of code, but we'll see | 18:40 |
| mwhahaha | EmilienM: it won't hurt | 18:40 |
| cloudnull | ++ fair enough | 18:40 |
| cloudnull | the implementation of futures throught the tripleo-common code base is fairly copy/paste, I wonder if we shouldn't move this into some common method ? | 18:41 |
| mwhahaha | not necessarily | 18:42 |
| mwhahaha | but we'll see | 18:42 |
| *** ykarel has joined #tripleo | 18:42 | |
| mwhahaha | the copy image loop is duped, but the others were p.map invocations | 18:42 |
| *** ykarel|away has quit IRC | 18:43 | |
| cloudnull | do we even need to use map ? | 18:45 |
| cloudnull | I suspect a simple thread should work just fine ? | 18:45 |
| mwhahaha | i think map is correct | 18:49 |
| mwhahaha | i need to review what's available in futures but i think that's what we're wanting | 18:49 |
| cloudnull | I guess map / submit one in the same given we're not using the ProcessPoolExecutor | 18:49 |
| cloudnull | map will just auto-iterate over the the set, so +1 map seems like the better all around solution | 18:50 |
| mwhahaha | ultimately i think map is what we're using everywhere | 18:52 |
| mwhahaha | where we put the work together than then collect the results | 18:52 |
| mwhahaha | ah map is in order | 18:55 |
| mwhahaha | and we don't care about the order for the layers i guess | 18:55 |
| *** lucasagomes has quit IRC | 19:00 | |
| *** ansmith_ has joined #tripleo | 19:02 | |
| *** d0ugal has joined #tripleo | 19:02 | |
| *** ansmith has quit IRC | 19:04 | |
| *** morazi has joined #tripleo | 19:07 | |
| openstackgerrit | Alex Schultz proposed openstack/tripleo-common master: Improve ThreadPoolExecutor usage https://review.opendev.org/684411 | 19:07 |
| openstackgerrit | Francesco Pantano proposed openstack/tripleo-heat-templates master: Add registry authentication parameters to pull ceph related containers https://review.opendev.org/677699 | 19:16 |
| tbarron | weshay: so I'm still trying to locally repro the ceph-ansible issue from the rhel8 job in https://review.opendev.org/#/c/682853/ but I'm missing how to get the ceph-ansible pkg that has the issue installed on my standalone | 19:22 |
| weshay | tbarron, ah | 19:22 |
| * weshay helps | 19:23 | |
| tbarron | the job pulls it from a pulp mirror, no baseurl | 19:23 |
| weshay | tbarron, seriously appologize we'd be more involved if the world were not on fire | 19:23 |
| tbarron | my standalone is centos7 not rhel8 too .. | 19:23 |
| openstackgerrit | Kevin Carter (cloudnull) proposed openstack/tripleo-ansible master: Pivot ovs-dpdk molecule to ansible tasks https://review.opendev.org/684379 | 19:23 |
| tbarron | weshay: yah, feel free to tell me to go away and come back later | 19:23 |
| openstackgerrit | Kevin Carter (cloudnull) proposed openstack/tripleo-ansible master: Pivot ovs-dpdk molecule to ansible tasks https://review.opendev.org/684379 | 19:24 |
| openstackgerrit | Kevin Carter (cloudnull) proposed openstack/tripleo-ansible master: Add kernelargs and tuned core configuration https://review.opendev.org/679809 | 19:24 |
| *** morazi has quit IRC | 19:25 | |
| *** morazi has joined #tripleo | 19:29 | |
| openstackgerrit | Francesco Pantano proposed openstack/tripleo-heat-templates master: Allow using registry authentication to pull ceph related containers https://review.opendev.org/682015 | 19:34 |
| *** morazi has quit IRC | 19:38 | |
| openstackgerrit | Francesco Pantano proposed openstack/tripleo-heat-templates master: Allow using registry authentication to pull ceph related containers https://review.opendev.org/682015 | 19:42 |
| openstackgerrit | Alex Schultz proposed openstack/tripleo-common stable/rocky: Add profile_tasks to ansible runs https://review.opendev.org/684309 | 19:43 |
| openstackgerrit | Alex Schultz proposed openstack/tripleo-common stable/queens: Add profile_tasks to ansible runs https://review.opendev.org/684310 | 19:44 |
| *** morazi has joined #tripleo | 19:46 | |
| *** ykarel has quit IRC | 19:47 | |
| EmilienM | weshay: queens / rocky done, stein almost done | 19:47 |
| *** rlandy has quit IRC | 19:50 | |
| *** rlandy has joined #tripleo | 19:54 | |
| weshay | EmilienM, k.. cool | 20:02 |
| *** noama has quit IRC | 20:07 | |
| cloudnull | anyone around available to give https://review.opendev.org/#/c/684379 a nudge - fixes the ovs-dpdk jobs unblocking 679809 | 20:08 |
| openstackgerrit | Jose Luis Franco proposed openstack/tripleo-upgrade master: Sync up upgrades and updates workarounds mechanism. https://review.opendev.org/684419 | 20:09 |
| *** aakarsh has joined #tripleo | 20:13 | |
| *** jfrancoa has quit IRC | 20:17 | |
| *** ansmith has joined #tripleo | 20:19 | |
| *** ansmith_ has quit IRC | 20:20 | |
| *** ade_lee_ has quit IRC | 20:22 | |
| *** florianf has quit IRC | 20:25 | |
| *** morazi has quit IRC | 20:25 | |
| *** holser has joined #tripleo | 20:32 | |
| *** ansmith_ has joined #tripleo | 20:32 | |
| *** ansmith has quit IRC | 20:34 | |
| *** jcoufal has quit IRC | 20:34 | |
| *** ansmith_ has quit IRC | 20:36 | |
| *** Vorrtex has quit IRC | 20:38 | |
| *** morazi has joined #tripleo | 20:43 | |
| *** ade_lee_ has joined #tripleo | 20:59 | |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-repos master: Add support for CentOS8 https://review.opendev.org/684426 | 21:00 |
| *** jschlueter has quit IRC | 21:01 | |
| *** jschlueter has joined #tripleo | 21:03 | |
| *** ade_lee_ has quit IRC | 21:05 | |
| *** ade_lee_ has joined #tripleo | 21:06 | |
| *** mcornea has quit IRC | 21:09 | |
| *** rfolco has quit IRC | 21:11 | |
| *** rfolco has joined #tripleo | 21:11 | |
| *** xek has quit IRC | 21:11 | |
| openstackgerrit | Merged openstack/puppet-tripleo stable/rocky: Enable deep_compare for fencing resource https://review.opendev.org/683390 | 21:13 |
| openstackgerrit | Merged openstack/puppet-tripleo stable/queens: Enable deep_compare for fencing resource https://review.opendev.org/683391 | 21:13 |
| *** aakarsh has quit IRC | 21:16 | |
| *** artom has quit IRC | 21:17 | |
| *** artom has joined #tripleo | 21:18 | |
| openstackgerrit | Merged openstack/tripleo-ansible master: Pivot ovs-dpdk molecule to ansible tasks https://review.opendev.org/684379 | 21:26 |
| openstackgerrit | Lance Bragstad proposed openstack/tripleo-heat-templates master: Enable caching for keystone overcloud https://review.opendev.org/684432 | 21:33 |
| lbragstad | ^ gave me a nice performance bump on basic overcloud installations | 21:34 |
| lbragstad | (i only timed keystone performance though) | 21:34 |
| stevebaker | mwhahaha: hey, I need some puppet-nova advice | 21:42 |
| mwhahaha | orly | 21:42 |
| stevebaker | mwhahaha: puppet-nova also configures novajoin, a completely seperate side-project. I need to do the novajoin equivalent of nova::policies, but it is not obvious to me what to call it or where to put it https://opendev.org/openstack/puppet-nova/src/branch/master/manifests/policy.pp | 21:44 |
| * mwhahaha hates novajoin | 21:44 | |
| stevebaker | mwhahaha: it would be identical to nova::policies but it writes to /etc/novajoin/policy.json | 21:44 |
| mwhahaha | put it in metadata/novajoin | 21:45 |
| stevebaker | so metadata/novajoin/policy.pp? ok | 21:46 |
| EmilienM | écopy/pasta | 21:46 |
| *** bfournie has quit IRC | 21:47 | |
| mwhahaha | stevebaker: yes | 21:51 |
| stevebaker | mwhahaha: ok thanks | 21:51 |
| *** raildo has quit IRC | 21:53 | |
| EmilienM | stevebaker: I thought policy.json was somehow deprecated and replaced by policy in code | 21:57 |
| *** holser has quit IRC | 21:57 | |
| mwhahaha | i thought the default is now in code | 21:57 |
| mwhahaha | but you could still have one | 21:57 |
| *** weshay has quit IRC | 21:58 | |
| *** slaweq has quit IRC | 22:00 | |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-repos master: Add support for CentOS8 https://review.opendev.org/684426 | 22:02 |
| *** gchamoul has quit IRC | 22:04 | |
| *** rcernin has joined #tripleo | 22:07 | |
| *** gchamoul has joined #tripleo | 22:14 | |
| *** weshay has joined #tripleo | 22:19 | |
| EmilienM | cloudnull: have you seen that before ? http://paste.openstack.org/show/779217/ | 22:21 |
| EmilienM | seems like centos8 ships ansible 2.7.6 | 22:22 |
| cloudnull | no i have not | 22:23 |
| EmilienM | probably the kernel ansible module isn't compatible with 2. | 22:23 |
| EmilienM | 2.7 | 22:23 |
| cloudnull | we've been testing with 2.8 | 22:23 |
| cloudnull | I think that's what upstream has been running ? | 22:23 |
| EmilienM | rigt, i'm now testing centos8 | 22:23 |
| cloudnull | oh. | 22:23 |
| EmilienM | https://trunk.rdoproject.org/rhel8-master/deps/latest/Packages/ansible-2.7.6-1.el8.noarch.rpm | 22:23 |
| EmilienM | that's what is shipped in rdo for rhel8 | 22:24 |
| cloudnull | should we be testing with 2.7 ? | 22:24 |
| * cloudnull can change that if needed | 22:24 | |
| stevebaker | EmilienM: I want to override the code policy because it limits calling novajoin to the nova user | 22:24 |
| EmilienM | cloudnull: my rhel8 env is down now | 22:25 |
| EmilienM | I don't know what version we run there | 22:25 |
| EmilienM | stevebaker: ack, thx | 22:25 |
| cloudnull | rhel8 ==ansible 2.8.4 | 22:26 |
| cloudnull | w/ osp16 | 22:26 |
| weshay | EmilienM, the rhel8 rhui repos have 2.8 | 22:26 |
| * weshay double checks my math | 22:26 | |
| EmilienM | ah so the rdo repo is wrong | 22:27 |
| cloudnull | EmilienM I would suspect that filter exists in 2.7 but the keyword arg is just not present . | 22:27 |
| EmilienM | right | 22:27 |
| weshay | ansible-2.8.3-1.el8ae.noarch | 22:27 |
| weshay | EmilienM, you might want to hold back for a bit there cowboy | 22:27 |
| EmilienM | no, I'm trying to deploy w/ workarounds until it's finished | 22:28 |
| EmilienM | so we can find potential issues quickly | 22:28 |
| weshay | that's why we have rhel | 22:28 |
| weshay | the package set should be very close... /me hopes | 22:28 |
| *** rfolco has quit IRC | 22:29 | |
| EmilienM | how do we handle these deps? is there a repo somewhere or it's manual? | 22:29 |
| *** rfolco has joined #tripleo | 22:29 | |
| weshay | EmilienM, it's in the RHUI | 22:31 |
| weshay | EmilienM, so.. the quay job returned | 22:31 |
| weshay | 2019-09-24 01:23:48 | HTTPError: 401 Client Error: UNAUTHORIZED for url: https://quay.io/v2/tripleomaster/centos-binary-cron/manifests/current-tripleo | 22:31 |
| weshay | first job didn't hit that.. second one did.. | 22:31 |
| EmilienM | ok let me try | 22:32 |
| EmilienM | mhh | 22:33 |
| weshay | slow connection here | 22:33 |
| weshay | trying it manually | 22:33 |
| EmilienM | me too | 22:34 |
| weshay | cloudnull, we have to keep EmilienM focused some times | 22:34 |
| cloudnull | lol | 22:34 |
| EmilienM | error pulling image "quay.io/tripleomaster/centos-binary-aodh-api": unable to pull quay.io/tripleomaster/centos-binary-aodh-api: unable to pull image: Error determining manifest MIME type for docker://quay.io/tripleomaster/centos-binary-aodh-api:latest: Error reading manifest latest in quay.io/tripleomaster/centos-binary-aodh-api: manifest unknown: manifest unknown | 22:35 |
| weshay | working for me w/ the tag current-tripleo | 22:39 |
| EmilienM | ah I missed :current-trpleo | 22:39 |
| EmilienM | yea | 22:39 |
| * weshay checks the test change | 22:39 | |
| EmilienM | let me check the cron | 22:39 |
| weshay | hrm... looks ok | 22:39 |
| EmilienM | it's public :/ | 22:39 |
| weshay | which cron? | 22:39 |
| EmilienM | the container | 22:40 |
| EmilienM | centos-binary-cron | 22:40 |
| EmilienM | where do you see that error? | 22:40 |
| * weshay gets | 22:40 | |
| weshay | https://55cb9f8472ab875bc5fc-29df7c45e84d4e54520155da59ed8678.ssl.cf1.rackcdn.com/683212/3/check/tripleo-ci-centos-7-containers-multinode/e8709ed/logs/undercloud/home/zuul/undercloud_install.log.txt.gz | 22:41 |
| weshay | we probably need to set the token in a pre.yaml ? | 22:42 |
| weshay | or should that just work like it does via cli | 22:42 |
| EmilienM | no | 22:42 |
| EmilienM | it should work fine | 22:42 |
| EmilienM | when did the job run? | 22:42 |
| EmilienM | I could pull the image on a node where I'm not logged | 22:43 |
| *** bfournie has joined #tripleo | 22:43 | |
| *** ekultails has quit IRC | 22:43 | |
| EmilienM | maybe a transient error? | 22:44 |
| EmilienM | recheck | 22:44 |
| weshay | EmilienM, maybe they got my email.. and said f.. off | 22:44 |
| *** holser has joined #tripleo | 22:46 | |
| EmilienM | lol | 22:47 |
| EmilienM | very lickely | 22:47 |
| EmilienM | likely damn I can't type with that lag | 22:48 |
| EmilienM | "libfacter was not found. Please make sure it was installed to the expected location." | 22:50 |
| EmilienM | another packaging issue | 22:51 |
| *** dtantsur|afk has quit IRC | 22:51 | |
| mwhahaha | did you install ruby-facter | 22:51 |
| mwhahaha | or something | 22:51 |
| *** stevebaker has quit IRC | 22:51 | |
| *** tkajinam has joined #tripleo | 22:51 | |
| EmilienM | yes, ruby-facter-1:3.9.3-9.el8ost.x86_64 | 22:52 |
| EmilienM | cloudnull: FYI I requested a new ansible in RDO but also created https://bugs.launchpad.net/tripleo/+bug/1845272 for visibility | 22:52 |
| openstack | Launchpad bug 1845272 in tripleo "tripleo-kernel fails on ansible 2.7" [Medium,Triaged] | 22:52 |
| cloudnull | I suspect we have several roles that will fail on 2.7 | 22:53 |
| EmilienM | I have facter-1:3.9.3-9.el8ost.x86_64 | 22:53 |
| cloudnull | If we think it best I can change our role test strategy to run w/ 2.7 | 22:53 |
| EmilienM | cloudnull: yeah, that's what I thought | 22:53 |
| *** aakarsh has joined #tripleo | 22:55 | |
| *** stevebaker has joined #tripleo | 23:06 | |
| EmilienM | mwhahaha: yeah I have /usr/lib64/libfacter.so.3.9.3 | 23:07 |
| openstackgerrit | Steve Baker proposed openstack/tripleo-heat-templates master: Override policy for undercloud novajoin https://review.opendev.org/684142 | 23:08 |
| *** rfolco has quit IRC | 23:13 | |
| *** rfolco has joined #tripleo | 23:14 | |
| *** stevebaker has quit IRC | 23:19 | |
| *** stevebaker has joined #tripleo | 23:20 | |
| *** ansmith has joined #tripleo | 23:21 | |
| *** chem has quit IRC | 23:26 | |
| *** holser has quit IRC | 23:29 | |
| *** weshay has quit IRC | 23:31 | |
| mwhahaha | cloudnull, EmilienM soooo https://opendev.org/openstack/tripleo-common/commit/e06cf4482da855d63ab21f8183cea66499e6f5f9 | 23:31 |
| *** gbarros has quit IRC | 23:31 | |
| mwhahaha | cloudnull, EmilienM: that changed up our workers, which made it CPU-1 * CPU-1 | 23:31 |
| EmilienM | pushups | 23:31 |
| mwhahaha | instead of CPU-1 / 2 * CPU-1 / 2 | 23:32 |
| cloudnull | ? | 23:32 |
| mwhahaha | er CPU/2 * CPU/2 | 23:32 |
| EmilienM | max(2, (processutils.get_worker_count() - 1)) | 23:33 |
| mwhahaha | the entry point to our processing is https://opendev.org/openstack/tripleo-common/src/branch/master/tripleo_common/image/image_uploader.py#L1913-L1918 | 23:33 |
| cloudnull | that should be cpu-1 or 2 | 23:33 |
| mwhahaha | hrm wait | 23:33 |
| mwhahaha | mayeb i'm looking at a bad copy | 23:33 |
| mwhahaha | either way, we threadpool the uploaders | 23:33 |
| mwhahaha | and then threadpool the layers in each one | 23:33 |
| mwhahaha | so it's likely that we need to tone down the workers | 23:34 |
| mwhahaha | because for each uploader, we end up doing various threading operations in it | 23:34 |
| mwhahaha | from tag fetching to layer transfers | 23:34 |
| cloudnull | we've been trying something similar here https://review.opendev.org/#/c/682731 | 23:36 |
| mwhahaha | so we don't want them to be dynamic | 23:36 |
| mwhahaha | because it's likely the source is the same place | 23:37 |
| openstackgerrit | Alex Schultz proposed openstack/tripleo-common master: Improve ThreadPoolExecutor usage https://review.opendev.org/684411 | 23:37 |
| cloudnull | wouldn't we want to use more workers on a stronger system? like in the case of prod? | 23:37 |
| mwhahaha | no | 23:37 |
| mwhahaha | because http requests limitations | 23:37 |
| mwhahaha | you can't spawn 100 requests to docker.io | 23:38 |
| mwhahaha | i mean you can | 23:38 |
| mwhahaha | but it's going to run terrible | 23:38 |
| mwhahaha | kinda like what we've got now :D | 23:39 |
| cloudnull | before it would have been cpu/2 or 2, so on a 48 core system it'd have 24 workers, so maybe we need to change that worker setting, and set an upper limit? | 23:39 |
| mwhahaha | it's possible | 23:40 |
| mwhahaha | but let's see if we can get it down for 8 cpu | 23:40 |
| mwhahaha | it's not like we have more layers/containers | 23:40 |
| cloudnull | and if run on a power system we'd see something like 64+ workers :) | 23:40 |
| cloudnull | but it makes sense. if we cant spawn that many connections to docker then I can see how itd become crazy slow | 23:41 |
| mwhahaha | i might commit my logging improvements | 23:42 |
| mwhahaha | makes this multithreading stuff easier to follow | 23:42 |
| cloudnull | +2 | 23:46 |
| mwhahaha | oh god damn it | 23:49 |
| mwhahaha | who wants to guess what's wrong with this http://paste.openstack.org/show/779220/ | 23:50 |
| cloudnull | processes not being cleaned up ? | 23:50 |
| cloudnull | sessions not closing ? | 23:51 |
| mwhahaha | ding | 23:51 |
| mwhahaha | want to guess how much we use this code | 23:51 |
| cloudnull | 80% | 23:51 |
| mwhahaha | https://github.com/openstack/tripleo-common/blob/master/tripleo_common/image/image_uploader.py#L1588-L1612 | 23:51 |
| mwhahaha | and we're doing it to ourselves | 23:52 |
| mwhahaha | because apache | 23:52 |
| *** gbarros has joined #tripleo | 23:52 | |
| mwhahaha | thats why i was seeing hangs | 23:52 |
| mwhahaha | it hangs trying to close-wait to the registry | 23:52 |
| mwhahaha | and can't open more requests | 23:52 |
| cloudnull | time to upgrade to NGINX? | 23:53 |
| * cloudnull hides | 23:53 | |
| mwhahaha | pfffft | 23:53 |
| mwhahaha | eh i'll fix it tomorrow | 23:58 |
| *** gbarros_ has joined #tripleo | 23:59 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!
