| *** slaweq has quit IRC | 00:02 | |
| openstackgerrit | Takashi Kajinami proposed openstack/puppet-tripleo master: WIP: Allow to configure token caching in keystone https://review.opendev.org/683010 | 00:04 |
|---|---|---|
| weshay | 2019-09-18 23:30:57 | TASK [Fail if nova-metadata healthcheck report failed status] ****************** | 00:10 |
| weshay | 2019-09-18 23:30:57 | fatal: [standalone]: FAILED! => {"changed": false, "msg": "nova-metadata isn't working (healthcheck failed)"} | 00:10 |
| weshay | 2019-09-18 23:30:57 | | 00:10 |
| weshay | https://openstack.fortnebula.com:13808/v1/AUTH_e8fd161dc34c421a979a9e6421f823e9/zuul_opendev_logs_f26/636323/3/gate/tripleo-ci-centos-7-scenario001-standalone/f261970/logs/undercloud/home/zuul/standalone_deploy.log.txt.gz | 00:10 |
| openstackgerrit | Takashi Kajinami proposed openstack/puppet-tripleo master: Add support to configure token caching in keystone https://review.opendev.org/683010 | 00:16 |
| *** mmethot has quit IRC | 00:18 | |
| *** mmethot has joined #tripleo | 00:18 | |
| *** slaweq has joined #tripleo | 00:18 | |
| *** mmethot has quit IRC | 00:19 | |
| *** mmethot has joined #tripleo | 00:19 | |
| *** slaweq has quit IRC | 00:23 | |
| *** mpjetta has quit IRC | 00:27 | |
| openstackgerrit | wes hayutin proposed openstack/tripleo-ci master: Run standalone scenario jobs at tht podman changes https://review.opendev.org/636323 | 00:30 |
| weshay | that patch may not have been rebased | 00:30 |
| *** aakarsh has joined #tripleo | 00:40 | |
| *** Goneri has quit IRC | 00:45 | |
| *** pierrepr1netti has joined #tripleo | 00:51 | |
| openstackgerrit | Takashi Kajinami proposed openstack/tripleo-heat-templates master: WIP: Enable token caching https://review.opendev.org/683012 | 00:51 |
| *** pierreprinetti has quit IRC | 00:54 | |
| openstackgerrit | Takashi Kajinami proposed openstack/puppet-tripleo master: Add support to configure token caching in keystone https://review.opendev.org/683010 | 01:06 |
| *** mmethot has quit IRC | 01:10 | |
| *** mmethot has joined #tripleo | 01:10 | |
| *** dsneddon has joined #tripleo | 01:12 | |
| *** dsneddon has quit IRC | 01:17 | |
| *** mschuppert has quit IRC | 01:21 | |
| openstackgerrit | Merged openstack/tripleo-ansible master: Ignore the Sphinx documentation build https://review.opendev.org/682848 | 01:49 |
| *** gbarros has joined #tripleo | 01:57 | |
| openstackgerrit | Merged openstack/tripleo-ansible master: Add test to find broken symlinks https://review.opendev.org/682419 | 02:01 |
| *** rlandy|bbl is now known as rlandy | 02:01 | |
| openstackgerrit | Takashi Kajinami proposed openstack/puppet-tripleo master: Add support to configure token caching in keystone https://review.opendev.org/683010 | 02:01 |
| *** rlandy has quit IRC | 02:01 | |
| *** apetrich has quit IRC | 02:10 | |
| EmilienM | weshay: back | 02:15 |
| EmilienM | weshay: https://review.opendev.org/#/c/682731/ tripleo-ci-centos-7-containers-multinode SUCCESS in 2h 55m 27s | 02:16 |
| EmilienM | 2019-09-18 23:07:02 | TASK [tripleo-container-image-prepare : Run tripleo-container-image-prepare logged to: /var/log/tripleo-container-image-prepare.log] *** | 02:17 |
| EmilienM | 2019-09-18 23:26:16 | changed: [undercloud] | 02:17 |
| EmilienM | 19min... didn't help | 02:17 |
| EmilienM | cloudnull: ^ | 02:17 |
| *** mmethot has quit IRC | 02:18 | |
| *** mmethot has joined #tripleo | 02:18 | |
| *** dsneddon has joined #tripleo | 02:18 | |
| weshay | EmilienM, /me looks | 02:22 |
| EmilienM | weshay: should we slowly restore patches? | 02:22 |
| EmilienM | a few at least | 02:22 |
| *** ricolin has joined #tripleo | 02:22 | |
| EmilienM | or merge the reverts? | 02:22 |
| weshay | ya.. I did a few I killed | 02:22 |
| weshay | but was wondering about that nova validation I posted.. I think it just wasn't rebased | 02:23 |
| weshay | but wasn't 100% sure | 02:23 |
| *** dsneddon has quit IRC | 02:23 | |
| weshay | at 18:10 | 02:23 |
| EmilienM | yeah it was a rebase issue | 02:23 |
| weshay | that should be off right now afaik.. the patch was in a weird state.. ya.. ok | 02:23 |
| EmilienM | I'm restoring a few | 02:24 |
| weshay | ok.. /me doesn't see a lot of new results | 02:24 |
| weshay | 682679,1 will be interesting | 02:24 |
| EmilienM | weshay: yeah i'm afraid of making it slower again | 02:25 |
| EmilienM | remember last year | 02:25 |
| EmilienM | we did that | 02:25 |
| EmilienM | we broke our gate | 02:25 |
| EmilienM | ansible was slower | 02:25 |
| weshay | ya.. used to be a lot worse | 02:25 |
| *** mmethot has quit IRC | 02:25 | |
| weshay | first one in train | 02:25 |
| *** mmethot has joined #tripleo | 02:25 | |
| *** ade_lee__ has quit IRC | 02:28 | |
| weshay | EmilienM, we should see under 3hrs | 02:28 |
| EmilienM | yea | 02:28 |
| EmilienM | i'm restoring | 02:28 |
| EmilienM | we need to merge stuff | 02:28 |
| weshay | ya | 02:28 |
| weshay | we may need to pull this trigger.. https://review.opendev.org/#/c/682945/ | 02:28 |
| weshay | but.. | 02:29 |
| weshay | we'll know more tomorrow | 02:29 |
| EmilienM | we'll still have timeouts on stein | 02:29 |
| EmilienM | i guess | 02:29 |
| EmilienM | but less | 02:29 |
| weshay | really? | 02:29 |
| EmilienM | the inflight validations don't run on stein | 02:29 |
| * weshay looks | 02:29 | |
| *** mmethot has quit IRC | 02:30 | |
| weshay | http://zuul.openstack.org/builds?job_name=tripleo-ci-centos-7-containers-multinode | 02:30 |
| weshay | oh.. 1 job | 02:30 |
| weshay | no.. none | 02:30 |
| EmilienM | weshay: I did recheck on https://review.opendev.org/#/c/682945/ | 02:30 |
| EmilienM | to see how it works with the tripleoclient things | 02:30 |
| weshay | k | 02:30 |
| EmilienM | gchamoul: FYI https://review.opendev.org/#/c/683001/ | 02:31 |
| EmilienM | at least they know | 02:31 |
| *** mmethot has joined #tripleo | 02:31 | |
| EmilienM | weshay: i'll send email w/ status | 02:32 |
| EmilienM | shortly | 02:33 |
| *** rainleander has joined #tripleo | 02:33 | |
| EmilienM | I need to debug CIX on kolla | 02:33 |
| weshay | EmilienM, let's merge the revert look at https://review.opendev.org/#/c/682341/ | 02:34 |
| weshay | it's not better | 02:34 |
| * weshay looks for validations.. maybe it's still old | 02:34 | |
| EmilienM | let me check | 02:35 |
| *** leanderthal has quit IRC | 02:35 | |
| EmilienM | it doesn't have validations | 02:35 |
| weshay | nope | 02:36 |
| EmilienM | meh | 02:36 |
| weshay | still too long | 02:36 |
| weshay | 22 rpms updated | 02:37 |
| weshay | er.. | 02:37 |
| weshay | 22 transactions | 02:37 |
| weshay | 12 rpms | 02:37 |
| weshay | I'll push on my guys for a promotion | 02:37 |
| * weshay looking for under 3 hours | 02:38 | |
| weshay | tripleo-ci-centos-7-containers-multinodeSUCCESS in 2h 48m 35s | 02:38 |
| weshay | just came in | 02:38 |
| weshay | https://review.opendev.org/#/c/682853/ | 02:39 |
| weshay | fak | 02:39 |
| weshay | too many clouds | 02:39 |
| weshay | http://dashboard-ci.tripleo.org/d/si1tipHZk/jobs-exploration?orgId=1&from=now-90d&to=now&var-influxdb_filter=job_name%7C%3D%7Ctripleo-ci-centos-7-containers-multinode&var-influxdb_filter=branch%7C%3D%7Cmaster&var-influxdb_filter=result%7C%3D%7CSUCCESS | 02:40 |
| EmilienM | i've restored all patches | 02:40 |
| EmilienM | and recheck | 02:40 |
| weshay | meh k | 02:40 |
| EmilienM | now let's sacrify an animal | 02:40 |
| weshay | at least 7.7 didn't jack us | 02:40 |
| EmilienM | *yet* | 02:41 |
| EmilienM | wait for centos8 in 1 week | 02:41 |
| EmilienM | what's your agenda next week? lol | 02:41 |
| weshay | heh.. well.. that's not bad because nothing is defined as centos8 | 02:41 |
| weshay | we can control that | 02:41 |
| weshay | and we should be in ok shape re: rhel8 jobs | 02:41 |
| EmilienM | at least something we can control | 02:41 |
| weshay | http://dashboard-ci.tripleo.org/d/si1tipHZk/jobs-exploration?orgId=1&from=now-6h&to=now&var-influxdb_filter=job_name%7C%3D%7Ctripleo-ci-centos-7-containers-multinode&var-influxdb_filter=branch%7C%3D%7Cmaster | 02:44 |
| weshay | last six hours | 02:45 |
| weshay | average is 2.93 hours :) | 02:45 |
| tkajinam | I see that some patches were restored. Is the gate fixed now ? | 02:46 |
| *** EmilienM changes topic to "CI Status: Orange (be kind with recheck and +A, watch for zuul queue) | community irc meeting Tues@1400 UTC - tripleo-ci-community meeting Tues@1330 UTC | https://docs.openstack.org/tripleo-docs" | 02:46 | |
| EmilienM | tkajinam: ^ | 02:46 |
| EmilienM | and i'm sending an email right now with status | 02:47 |
| weshay | k.. | 02:47 |
| EmilienM | sent | 02:48 |
| *** gbarros has quit IRC | 02:48 | |
| tkajinam | EmilienM, understood, thanks! I'll watch how restored patches go trough CI . | 02:49 |
| *** skramaja has joined #tripleo | 02:49 | |
| EmilienM | weshay: I'm off friday btw | 02:50 |
| cloudnull | weshay EmilienM https://storage.bhs1.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_da0/682731/6/check/tripleo-ci-centos-7-containers-multinode/da0ed31/logs/undercloud/var/log/tripleo-container-image-prepare.log.txt.gz 19 min in rax w/ 4 re-auth events, that's not super terrible | 02:50 |
| EmilienM | but 19 min :/ | 02:51 |
| EmilienM | 2019-09-18 23:07:02 | TASK [tripleo-container-image-prepare : Run tripleo-container-image-prepare logged to: /var/log/tripleo-container-image-prepare.log] *** | 02:51 |
| EmilienM | 2019-09-18 23:26:16 | changed: [undercloud] | 02:51 |
| cloudnull | without the re-auth it would have just been a fail | 02:51 |
| * cloudnull playing devils advocate | 02:52 | |
| EmilienM | i agree | 02:52 |
| cloudnull | search for "Error detected in auth headers: error invalid_token" | 02:52 |
| EmilienM | which is why i'm not in favour of merging the reverts :/ | 02:52 |
| weshay | ya.. it's a good feature | 02:53 |
| cloudnull | so the thread workers are set to 16 in most places. | 02:53 |
| cloudnull | maybe we set that to the 2x the number of cores ? | 02:53 |
| cloudnull | or something similar ? | 02:53 |
| cloudnull | in that way it should auto-adjust the worker threads when multi-processing based on the underyling system . | 02:54 |
| * cloudnull thinking that maybe 16 is just too much based on how we're doing things | 02:54 | |
| weshay | http://dashboard-ci.tripleo.org/d/si1tipHZk/jobs-exploration?orgId=1&from=now-6h&to=now&var-influxdb_filter=job_name%7C%3D%7Ctripleo-ci-centos-7-containers-multinode&var-influxdb_filter=branch%7C%3D%7Cmaster | 02:54 |
| openstackgerrit | Saravanan KR proposed openstack/tripleo-heat-templates master: Remove OpenDaylight templates and environments https://review.opendev.org/682630 | 02:54 |
| weshay | has 11% timeouts | 02:54 |
| weshay | wait.. that's not right.. there is just one | 02:55 |
| cloudnull | which looks like just 1 | 02:55 |
| weshay | ya.. | 02:55 |
| weshay | 11% over 24 hours.. I need to fix that.. so it changes w/ the filter | 02:55 |
| weshay | and that could have the old patches | 02:56 |
| weshay | meh.. restore and monitor :) | 02:57 |
| EmilienM | so it's not a myth, people don't have a life in open source | 02:58 |
| weshay | life .. what is life | 02:59 |
| weshay | I know lifeless | 02:59 |
| * lbragstad lingers at invalid tokens | 03:00 | |
| EmilienM | omg | 03:00 |
| EmilienM | you call token, you got lbragstad :D | 03:00 |
| lbragstad | ;) | 03:00 |
| EmilienM | I'm sure lbragstad has a token everywhere | 03:00 |
| lbragstad | i hope not, for security reasons | 03:01 |
| lbragstad | are tokens expiring on ci? | 03:04 |
| EmilienM | lbragstad: no, we have issues with docker.io auth | 03:05 |
| EmilienM | lbragstad: keystone is fine | 03:05 |
| lbragstad | oh | 03:05 |
| EmilienM | I would have called you WAY before | 03:05 |
| lbragstad | lol | 03:05 |
| lbragstad | i was just passively following along and thought it may have been a keystone issue (it's usually a keystone issue) | 03:05 |
| openstackgerrit | Kevin Carter (cloudnull) proposed openstack/tripleo-common master: image_uploader: relax HTTPAdapter pool_connections and pool_maxsize https://review.opendev.org/682731 | 03:06 |
| cloudnull | ^ updated to try auto-tuning the workers | 03:06 |
| * cloudnull doesn't know if it will help ... | 03:07 | |
| EmilienM | cloudnull: smart | 03:07 |
| weshay | dang it.. https://review.opendev.org/#/c/682988/ | 03:08 |
| weshay | 4 min ago | 03:08 |
| EmilienM | it doesn't disable validations or? | 03:09 |
| cloudnull | I suspect the different clouds provide different resource types, when I was working with infra most instances were 8 cores 8GiB of ram though some were 4 cores 8 threads, others were just different. IDK if they're all the same these days. | 03:09 |
| weshay | nope.. no validations | 03:10 |
| weshay | DFW | 03:10 |
| weshay | provider | 03:10 |
| *** ykarel|away has joined #tripleo | 03:11 | |
| *** ade_lee__ has joined #tripleo | 03:17 | |
| openstackgerrit | Kevin Carter (cloudnull) proposed openstack/tripleo-common master: image_uploader: relax HTTPAdapter pool_connections and pool_maxsize https://review.opendev.org/682731 | 03:17 |
| cloudnull | messed up one... | 03:17 |
| * cloudnull goes back to being a zombie | 03:18 | |
| EmilienM | exit 0 | 03:18 |
| openstackgerrit | Merged openstack/tripleo-ci master: Reenable ARA html generation from collect logs https://review.opendev.org/682679 | 03:20 |
| EmilienM | weshay: I don't feel it good ^ | 03:21 |
| openstackgerrit | Kevin Carter (cloudnull) proposed openstack/tripleo-heat-templates master: [WIP] Convert the container-puppet to ansible https://review.opendev.org/661883 | 03:21 |
| EmilienM | but we'll see | 03:21 |
| EmilienM | i don't think it's the right time to add ara | 03:22 |
| weshay | that rendering is done in post | 03:22 |
| EmilienM | ok | 03:22 |
| EmilienM | but the plugin is executing during the playbook run | 03:22 |
| EmilienM | which makes the results biased I think | 03:22 |
| EmilienM | maybe dmsimard can comment on that | 03:22 |
| EmilienM | but I remember these days where we enabled it and it was a nightmare | 03:23 |
| *** ade_lee__ has quit IRC | 03:23 | |
| *** ade_lee__ has joined #tripleo | 03:23 | |
| weshay | EmilienM, ok.. I'll turn it back off.. and we'll just pull or send the sqllite db | 03:23 |
| openstackgerrit | wes hayutin proposed openstack/tripleo-ci master: Revert "Reenable ARA html generation from collect logs" https://review.opendev.org/683025 | 03:24 |
| EmilienM | i don't say to turn it back off now | 03:24 |
| EmilienM | but to be careful | 03:24 |
| EmilienM | and monitor results | 03:24 |
| weshay | meh | 03:25 |
| EmilienM | good night all | 03:25 |
| weshay | we can do better than ara reports in each job | 03:25 |
| weshay | it's kind of worthless in each job | 03:25 |
| weshay | better to have them harvested | 03:25 |
| weshay | later | 03:27 |
| *** weshay has quit IRC | 03:27 | |
| * cloudnull going to crash myself, see you all in the AM | 03:29 | |
| *** psachin has joined #tripleo | 03:35 | |
| *** udesale has joined #tripleo | 04:04 | |
| openstackgerrit | Emilien Macchi proposed openstack/python-tripleoclient stable/stein: (squash) Clean backport of inflight-validations for UC/OC/standalone https://review.opendev.org/683027 | 04:10 |
| *** rh-jelabarre has quit IRC | 04:14 | |
| *** surpatil has joined #tripleo | 04:17 | |
| *** dsneddon has joined #tripleo | 04:19 | |
| *** dsneddon has quit IRC | 04:24 | |
| *** ykarel|away has quit IRC | 04:30 | |
| *** pcaruana has joined #tripleo | 04:42 | |
| *** ramishra has joined #tripleo | 04:43 | |
| openstackgerrit | Merged openstack/tripleo-heat-templates master: Add redis password for ml2 ansible coordination https://review.opendev.org/681298 | 04:44 |
| openstackgerrit | Merged openstack/tripleo-validations stable/stein: Switch to promote docs job https://review.opendev.org/682982 | 04:44 |
| openstackgerrit | Rabi Mishra proposed openstack/tripleo-common master: WIP Handle non-config-download deployment types https://review.opendev.org/680957 | 04:46 |
| *** ratailor has joined #tripleo | 04:49 | |
| *** ykarel|away has joined #tripleo | 04:54 | |
| *** rcernin_ has joined #tripleo | 04:56 | |
| *** ykarel|away is now known as ykarel | 04:57 | |
| *** rcernin has quit IRC | 04:59 | |
| *** jtomasek has joined #tripleo | 05:02 | |
| openstackgerrit | Merged openstack/tripleo-ci master: Reduce jobs by optimization of irrelevant files https://review.opendev.org/682278 | 05:15 |
| openstackgerrit | Merged openstack/tripleo-heat-templates master: Misspelling of words https://review.opendev.org/681708 | 05:15 |
| *** janki has joined #tripleo | 05:21 | |
| *** radeks has joined #tripleo | 05:28 | |
| *** lmiccini has joined #tripleo | 05:32 | |
| *** yprokule has joined #tripleo | 05:33 | |
| *** soniya29 has joined #tripleo | 05:53 | |
| *** avivgt has joined #tripleo | 05:55 | |
| *** marios has joined #tripleo | 05:56 | |
| openstackgerrit | Merged openstack/tripleo-upgrade master: Fix image validation when AP services get several images. https://review.opendev.org/682854 | 06:07 |
| *** kopecmartin|off is now known as kopecmartin | 06:18 | |
| *** cylopez has joined #tripleo | 06:18 | |
| *** dsneddon has joined #tripleo | 06:20 | |
| *** holser has joined #tripleo | 06:21 | |
| *** dsneddon has quit IRC | 06:24 | |
| *** mschuppert has joined #tripleo | 06:25 | |
| *** hamdyk has joined #tripleo | 06:28 | |
| openstackgerrit | Michele Baldessari proposed openstack/python-tripleoclient master: Add ovn_dbs_vip to export_data https://review.opendev.org/682294 | 06:30 |
| openstackgerrit | Michele Baldessari proposed openstack/tripleo-ansible master: Fix the non-HA case where OVN has its own VIP https://review.opendev.org/682968 | 06:36 |
| openstackgerrit | Michele Baldessari proposed openstack/tripleo-ansible master: Fix the non-HA case where OVN has its own VIP https://review.opendev.org/682968 | 06:41 |
| *** jaosorior has joined #tripleo | 06:43 | |
| *** dsneddon has joined #tripleo | 06:45 | |
| *** rcernin_ has quit IRC | 06:48 | |
| *** cylopez has quit IRC | 06:52 | |
| *** psachin has quit IRC | 06:53 | |
| *** gfidente has joined #tripleo | 06:54 | |
| *** udesale has quit IRC | 06:57 | |
| *** udesale has joined #tripleo | 06:58 | |
| *** rpittau|afk is now known as rpittau | 07:05 | |
| *** xek has joined #tripleo | 07:05 | |
| *** slaweq has joined #tripleo | 07:08 | |
| *** matbu has joined #tripleo | 07:14 | |
| *** apetrich has joined #tripleo | 07:15 | |
| openstackgerrit | Michele Baldessari proposed openstack/puppet-tripleo master: Add support for separate VIP in ovn_dbs https://review.opendev.org/672673 | 07:16 |
| *** udesale has quit IRC | 07:18 | |
| *** udesale has joined #tripleo | 07:19 | |
| *** slaweq has quit IRC | 07:20 | |
| *** tosky has joined #tripleo | 07:23 | |
| *** tesseract has joined #tripleo | 07:26 | |
| *** bogdando has joined #tripleo | 07:27 | |
| openstackgerrit | Sagi Shnaidman proposed openstack/ansible-role-collect-logs master: Don't gzip ARA when generating HTML https://review.opendev.org/683039 | 07:27 |
| openstackgerrit | Martin Schuppert proposed openstack/tripleo-heat-templates master: Delete deprecated nova-consoleauth-container-puppet.yaml https://review.opendev.org/683040 | 07:31 |
| *** jpena|off is now known as jpena | 07:33 | |
| *** slaweq has joined #tripleo | 07:33 | |
| *** amoralej|off is now known as amoralej | 07:37 | |
| *** cylopez has joined #tripleo | 07:38 | |
| *** xek has quit IRC | 07:38 | |
| *** ykarel is now known as ykarel|lunch | 07:38 | |
| *** xek has joined #tripleo | 07:40 | |
| *** jpich has joined #tripleo | 07:47 | |
| openstackgerrit | Natal Ngétal proposed openstack/python-tripleoclient master: Fix message of unused_params and invalid_role_specific_params. https://review.opendev.org/674570 | 07:48 |
| *** dsneddon has quit IRC | 07:49 | |
| *** chem has joined #tripleo | 07:49 | |
| *** jaosorior has quit IRC | 07:51 | |
| *** zbr is now known as zbr|ruck | 07:57 | |
| *** dsneddon has joined #tripleo | 08:00 | |
| *** iurygregory has joined #tripleo | 08:00 | |
| openstackgerrit | Takashi Kajinami proposed openstack/puppet-tripleo master: Add support to configure token caching in keystone https://review.opendev.org/683010 | 08:02 |
| *** soniya29 has quit IRC | 08:04 | |
| *** hjensas|afk is now known as hjensas | 08:06 | |
| *** paramite has joined #tripleo | 08:08 | |
| *** chem has quit IRC | 08:13 | |
| *** chem has joined #tripleo | 08:13 | |
| *** chem has quit IRC | 08:14 | |
| *** chem has joined #tripleo | 08:15 | |
| openstackgerrit | Kamil Sambor proposed openstack/tripleo-heat-templates master: WIP: Support connecting OVN DB over SSL https://review.opendev.org/680345 | 08:16 |
| *** tkajinam has quit IRC | 08:19 | |
| *** arxcruz|ruck is now known as arxcruz|rover | 08:20 | |
| *** ykarel|lunch is now known as ykarel | 08:21 | |
| *** lennyb has quit IRC | 08:21 | |
| *** chem has quit IRC | 08:22 | |
| *** chem has joined #tripleo | 08:23 | |
| *** cfontain has joined #tripleo | 08:25 | |
| *** chem has quit IRC | 08:26 | |
| *** chem has joined #tripleo | 08:27 | |
| openstackgerrit | Merged openstack/paunch stable/queens: Add --cpuset-cpus support https://review.opendev.org/682495 | 08:27 |
| openstackgerrit | Arx Cruz proposed openstack/tripleo-quickstart master: Adapt os_tempest in FS001 https://review.opendev.org/673400 | 08:28 |
| *** derekh has joined #tripleo | 08:30 | |
| *** zzzeek has quit IRC | 08:30 | |
| *** cfontain has quit IRC | 08:30 | |
| *** zzzeek has joined #tripleo | 08:31 | |
| *** alexmcleod has joined #tripleo | 08:32 | |
| *** cylopez has quit IRC | 08:32 | |
| *** dtantsur|afk is now known as dtantsur | 08:32 | |
| *** lennyb has joined #tripleo | 08:34 | |
| *** cylopez has joined #tripleo | 08:35 | |
| *** chem has quit IRC | 08:35 | |
| *** chem has joined #tripleo | 08:35 | |
| openstackgerrit | Arx Cruz proposed openstack/tripleo-quickstart master: Adapt os_tempest in FS001 https://review.opendev.org/673400 | 08:37 |
| *** suuuper has joined #tripleo | 08:39 | |
| openstackgerrit | Michele Baldessari proposed openstack/tripleo-heat-templates master: Give the OVN DBS service a separate Vip https://review.opendev.org/669847 | 08:41 |
| openstackgerrit | Merged openstack/tripleo-upgrade stable/stein: Fix container validation after overcloud update. https://review.opendev.org/681511 | 08:56 |
| openstackgerrit | Athlan-Guyot sofer proposed openstack/tripleo-upgrade stable/stein: Fix image validation when AP services get several images. https://review.opendev.org/683066 | 08:57 |
| openstackgerrit | Martin Schuppert proposed openstack/tripleo-common stable/queens: [WIP] Add workflow to do cellv2 host discovery https://review.opendev.org/683067 | 09:02 |
| *** dsneddon has quit IRC | 09:04 | |
| openstackgerrit | Martin Schuppert proposed openstack/tripleo-heat-templates stable/queens: [WIP] Queens-only: Run cellv2 host discovery via workflow https://review.opendev.org/683069 | 09:07 |
| *** gfidente has quit IRC | 09:07 | |
| *** gfidente has joined #tripleo | 09:10 | |
| *** iurygregory has quit IRC | 09:13 | |
| *** iurygregory has joined #tripleo | 09:14 | |
| *** slaweq has quit IRC | 09:25 | |
| *** slaweq has joined #tripleo | 09:27 | |
| *** soniya29 has joined #tripleo | 09:34 | |
| *** hberaud|gone is now known as hberaud | 09:36 | |
| openstackgerrit | Jose Luis Franco proposed openstack/tripleo-upgrade master: Change docker references to podman. https://review.opendev.org/683081 | 09:42 |
| openstackgerrit | Sagi Shnaidman proposed openstack/tripleo-ci master: DNM: test ara with tripleo-ci https://review.opendev.org/683083 | 09:44 |
| openstackgerrit | Sagi Shnaidman proposed openstack/ansible-role-collect-logs master: DNM: Adjust paths and commands for ara collection for >1.0 https://review.opendev.org/682958 | 09:46 |
| *** cylopez has quit IRC | 09:46 | |
| openstackgerrit | Rabi Mishra proposed openstack/tripleo-quickstart-extras master: Don't install centos-release-openstack-ocata repo https://review.opendev.org/683088 | 09:50 |
| openstackgerrit | Sagi Shnaidman proposed openstack/tripleo-quickstart master: DNM: Test ara master branch and send results to a remote server https://review.opendev.org/682956 | 09:51 |
| *** janki has quit IRC | 09:57 | |
| *** udesale has quit IRC | 10:01 | |
| *** gbarros has joined #tripleo | 10:03 | |
| openstackgerrit | Bogdan Dobrelya (bogdando) proposed openstack/ansible-role-tripleo-modify-image master: Add support for yum caching when buildah updates https://review.opendev.org/683100 | 10:07 |
| openstackgerrit | Sagi Shnaidman proposed openstack/tripleo-quickstart master: DNM: Test ara master branch and send results to a remote server https://review.opendev.org/682956 | 10:09 |
| openstackgerrit | Sagi Shnaidman proposed openstack/tripleo-quickstart master: DNM: Test ara master branch and send results to a remote server https://review.opendev.org/682956 | 10:11 |
| *** cylopez has joined #tripleo | 10:12 | |
| openstackgerrit | Bogdan Dobrelya (bogdando) proposed openstack/tripleo-ci master: DNM test CI containers update with yum caching https://review.opendev.org/683101 | 10:13 |
| *** rcernin_ has joined #tripleo | 10:16 | |
| openstackgerrit | Bogdan Dobrelya (bogdando) proposed openstack/ansible-role-tripleo-modify-image master: Add support for yum caching when buildah updates https://review.opendev.org/683100 | 10:16 |
| *** cylopez has quit IRC | 10:20 | |
| openstackgerrit | Bogdan Dobrelya (bogdando) proposed openstack/tripleo-ci master: DNM test CI containers update with yum caching https://review.opendev.org/683101 | 10:21 |
| *** gbarros has quit IRC | 10:21 | |
| *** ykarel is now known as ykarel|afk | 10:27 | |
| openstackgerrit | Martin Mágr proposed openstack/tripleo-heat-templates master: Collectd-sensubility https://review.opendev.org/682039 | 10:28 |
| *** udesale has joined #tripleo | 10:32 | |
| *** rcernin_ has quit IRC | 10:36 | |
| *** udesale has quit IRC | 10:37 | |
| *** udesale has joined #tripleo | 10:37 | |
| *** jaganathan has joined #tripleo | 10:46 | |
| *** pcaruana has quit IRC | 10:54 | |
| *** rpittau is now known as rpittau|afk | 10:56 | |
| *** dsneddon has joined #tripleo | 11:00 | |
| *** udesale has quit IRC | 11:01 | |
| openstackgerrit | Merged openstack/tripleo-docs master: Adding upgrade with os upgrade graphics. https://review.opendev.org/631537 | 11:03 |
| *** jpena is now known as jpena|lunch | 11:04 | |
| gchamoul | EmilienM: hi, about https://review.opendev.org/#/c/683001/, the undercloud_enable_validations var is just enabling or not the tripleo-validations installation in the Undercloud and not controlling the execution of the validations during the install! The only validation which could be executed during the undercloud install process is the undercloud-disk-space val, but it's not controlled by this ansible var! | 11:04 |
| *** dsneddon has quit IRC | 11:05 | |
| gchamoul | EmilienM, I don't think that patch was really necessary IMO | 11:06 |
| *** amoralej is now known as amoralej|lunch | 11:07 | |
| openstackgerrit | Sagi Shnaidman proposed openstack/tripleo-quickstart master: DNM: Test ara master branch and send results to a remote server https://review.opendev.org/682956 | 11:10 |
| *** pcaruana has joined #tripleo | 11:17 | |
| *** ansmith has quit IRC | 11:21 | |
| *** gfidente has quit IRC | 11:28 | |
| openstackgerrit | Merged openstack/puppet-tripleo stable/queens: Safeguard against 'Undef' length() checks. https://review.opendev.org/682387 | 11:30 |
| openstackgerrit | Hamdy Khader proposed openstack/tripleo-heat-templates master: Add support for Neutron Multi interface driver https://review.opendev.org/679558 | 11:33 |
| *** lucasagomes has joined #tripleo | 11:33 | |
| *** gfidente has joined #tripleo | 11:43 | |
| *** rh-jelabarre has joined #tripleo | 11:54 | |
| EmilienM | gchamoul: that we wanted to do is to set enable_validations to false in undercloud.conf | 11:55 |
| EmilienM | gchamoul: check out https://review.opendev.org/#/c/682943/ | 11:56 |
| EmilienM | https://review.opendev.org/#/c/682943/2/tripleoclient/v1/undercloud_config.py | 11:56 |
| openstackgerrit | Hamdy Khader proposed openstack/tripleo-heat-templates master: Add heat templates to deploy mellanox neutron agent https://review.opendev.org/674486 | 11:59 |
| openstackgerrit | Hamdy Khader proposed openstack/tripleo-heat-templates master: Add support for Neutron Multi interface driver https://review.opendev.org/679558 | 11:59 |
| *** weshay_ has joined #tripleo | 11:59 | |
| EmilienM | weshay_: morning. Sounds like stuff merged, much less timeouts, still on the edge though | 12:02 |
| *** ykarel|afk is now known as ykarel | 12:02 | |
| *** rfolco has joined #tripleo | 12:04 | |
| *** ricolin has quit IRC | 12:06 | |
| *** Goneri has joined #tripleo | 12:07 | |
| *** amoralej|lunch is now known as amoralej | 12:08 | |
| *** bfournie has quit IRC | 12:10 | |
| *** jpena|lunch is now known as jpena | 12:23 | |
| *** mcornea has joined #tripleo | 12:24 | |
| *** jfrancoa has joined #tripleo | 12:28 | |
| *** ade_lee__ has quit IRC | 12:28 | |
| *** lbragstad has quit IRC | 12:29 | |
| *** lbragstad has joined #tripleo | 12:30 | |
| *** rlandy has joined #tripleo | 12:31 | |
| *** brault has joined #tripleo | 12:37 | |
| openstackgerrit | Hamdy Khader proposed openstack/tripleo-heat-templates master: Add heat templates to deploy mellanox neutron agent https://review.opendev.org/674486 | 12:41 |
| openstackgerrit | Hamdy Khader proposed openstack/tripleo-heat-templates master: Add support for Neutron Multi interface driver https://review.opendev.org/679558 | 12:41 |
| *** ansmith has joined #tripleo | 12:44 | |
| *** aakarsh has quit IRC | 12:46 | |
| weshay_ | EmilienM, ya.. going through it w/ my ruck / rovers | 12:48 |
| weshay_ | EmilienM, pushing them for a master promotion because containers are on 7.6 | 12:49 |
| weshay_ | maybe 7.7 will help.. | 12:49 |
| weshay_ | still on the edge | 12:49 |
| *** brault has quit IRC | 12:50 | |
| *** brault has joined #tripleo | 12:51 | |
| *** brault has quit IRC | 12:55 | |
| *** zzzeek has quit IRC | 12:56 | |
| *** zzzeek has joined #tripleo | 12:58 | |
| *** dsneddon has joined #tripleo | 13:01 | |
| openstackgerrit | Merged openstack/tripleo-validations master: Deprecation of the ceilometerdb-size validation https://review.opendev.org/680694 | 13:01 |
| *** bfournie has joined #tripleo | 13:03 | |
| *** dsneddon has quit IRC | 13:05 | |
| *** slaweq has quit IRC | 13:07 | |
| owalsh | EmilienM, gchamoul: hey, have we figured out what to do with the healthcheck deployment tasks going forward? | 13:10 |
| owalsh | if we continue with the checks after each step then I've a problem moving nova-compute to step 5 in https://review.opendev.org/681042 | 13:10 |
| EmilienM | owalsh: gchamoul has a patch in tripleo-validation | 13:10 |
| EmilienM | not sure status and if we can use it | 13:10 |
| owalsh | EmilienM: ack, wondering how we'll use it... after each step or at the end | 13:11 |
| *** hjensas has quit IRC | 13:13 | |
| *** dtantsur is now known as dtantsur|bbl | 13:14 | |
| owalsh | at the end would minimise the impact on CI I expect | 13:16 |
| *** ade_lee__ has joined #tripleo | 13:17 | |
| *** jaosorior has joined #tripleo | 13:18 | |
| *** brault has joined #tripleo | 13:18 | |
| *** brault has quit IRC | 13:20 | |
| *** brault has joined #tripleo | 13:20 | |
| *** aakarsh has joined #tripleo | 13:20 | |
| EmilienM | owalsh: i guess in this particular case, an exception would be made and the healthchecks would be done at the end? | 13:21 |
| EmilienM | gchamoul and other validation folks can help you | 13:21 |
| owalsh | EmilienM: TBH I think they shouldn't be in deploy_steps_tasks (runs at the beginning of the step), we really need post_deploy_steps_tasks for this | 13:23 |
| EmilienM | I agree | 13:23 |
| *** udesale has joined #tripleo | 13:23 | |
| EmilienM | Tengu is on PTO, he has the lead on that, tbh I don't know much about $topic | 13:23 |
| *** udesale has quit IRC | 13:24 | |
| EmilienM | he's back on monday | 13:24 |
| owalsh | EmilienM: ack, thanks | 13:25 |
| *** udesale has joined #tripleo | 13:25 | |
| *** pcaruana has quit IRC | 13:28 | |
| *** derekh has quit IRC | 13:29 | |
| *** devep has joined #tripleo | 13:38 | |
| gchamoul | owalsh: was afk, sorry! look at how the ceph team runs the ceph-health validation from THT as inflight validation https://github.com/openstack/tripleo-heat-templates/blob/master/deployment/ceph-ansible/ceph-base.yaml#L540-L550 | 13:39 |
| owalsh | gchamoul: not asking how, asking when/where | 13:39 |
| gchamoul | owalsh: oh ok! | 13:40 |
| owalsh | gchamoul: at the moment they are run in deploy_steps_tasks in the step that follows (so e.g step5 validates the step4 services)... I need to move nova_compute to step5 so where to run the validation tasks? | 13:42 |
| owalsh | also since we have disabled the tasks by default to keep CI happen what's the point? We're just going to miss failure again. Better to run them once at the end (if that's faster) and enable in CI | 13:44 |
| owalsh | s/happen/happy | 13:44 |
| *** gfidente has quit IRC | 13:45 | |
| weshay_ | 2 gate jobs failing on infra :( | 13:47 |
| weshay_ | 2019-09-19 13:03:57.045137 | primary | fatal: [subnode-2]: UNREACHABLE! => { | 13:48 |
| weshay_ | 2019-09-19 13:44:07.414398 | primary | ERROR: Could not find a version that satisfies the requirement distro (from bindep) (from versions: none) | 13:48 |
| gchamoul | owalsh: about where to run the validation tasks in THT, I am afraid to not be helpful. If Tengu was here, he could likely know... | 13:48 |
| cloudnull | mornings | 13:50 |
| owalsh | gchamoul: ack | 13:50 |
| owalsh | gchamoul: will follow up on Monday, mostly need to know so I can land https://review.opendev.org/681042 and backport it | 13:50 |
| gchamoul | owalsh: did you read my answer to your comment on the patch? | 13:51 |
| gchamoul | the tripleo-validations one | 13:51 |
| *** ratailor has quit IRC | 13:51 | |
| gchamoul | owalsh: about the variables? | 13:51 |
| owalsh | gchamoul: looking.... | 13:52 |
| owalsh | gchamoul: if the healtcheck has run it doesn't loop, so why not? | 13:52 |
| gchamoul | owalsh: about the CI, I understand your concerns and I have the same because I would really like to get some CI around tripleo-validations since a long time | 13:53 |
| gchamoul | owalsh: the validations should be run manually from the CLI, and don't to get many retries and delay | 13:54 |
| gchamoul | owalsh: in post-deployment, this validation should be faster | 13:54 |
| *** pcaruana has joined #tripleo | 13:54 | |
| owalsh | gchamoul: when run manually from the CLI there would be no retries (regardless of the role defaults), assuming the services have been running for >120s... so why not set the safe default for retry/delay | 13:56 |
| owalsh | with no retry we get the old behaviour of false positives | 13:56 |
| *** derekh has joined #tripleo | 13:58 | |
| owalsh | if you look at the task, it's just waiting for an ExecPID which means the healthcheck has been started at least once, so should not retry unless it's still starting | 13:58 |
| EmilienM | weshay_, mwhahaha : can we talk about docker.io | 13:58 |
| weshay_ | ya | 13:58 |
| mwhahaha | maybe | 13:58 |
| EmilienM | should we evaluate the option of pushing our containers to another registry | 13:58 |
| EmilienM | maybe quay or something | 13:59 |
| * mwhahaha doesn't care | 13:59 | |
| mwhahaha | though i thought getting an account on quay was the problem previously | 13:59 |
| EmilienM | i have no idea about this technical detail | 13:59 |
| EmilienM | let me check | 13:59 |
| gchamoul | owalsh: yes right, I missed that! so you suggest to keep the retry/delay as in the Rajesh patch | 14:00 |
| mwhahaha | also i would guess we do a whole bunch of traffic so i'm uncertain of limiations by switching | 14:00 |
| EmilienM | I guess it's worth a try | 14:00 |
| *** tesseract has quit IRC | 14:00 | |
| *** saneax has joined #tripleo | 14:01 | |
| owalsh | gchamoul: the original patch didn't have any retry/delay, I added it in https://review.opendev.org/681442 when we realised the healthchecks were broken but the checks were passing | 14:01 |
| openstackgerrit | Merged openstack/tripleo-docs master: PDF documentation build https://review.opendev.org/682362 | 14:02 |
| *** Vorrtex has joined #tripleo | 14:04 | |
| *** soniya29 has quit IRC | 14:04 | |
| *** saneax has quit IRC | 14:04 | |
| *** saneax has joined #tripleo | 14:05 | |
| *** surpatil has quit IRC | 14:05 | |
| * owalsh wonders why am I getting +1s from retired footballers e.g Inzaghi and Shevchenko | 14:05 | |
| paramite | Hey guys, can we merge this one please: https://review.opendev.org/#/c/678863/ ? | 14:05 |
| *** hjensas has joined #tripleo | 14:07 | |
| weshay_ | EmilienM, re perf? | 14:09 |
| *** ayoung has quit IRC | 14:09 | |
| EmilienM | weshay_: yes | 14:09 |
| EmilienM | I'm going to push tripleo containers to my quay account | 14:09 |
| EmilienM | and try out deployments | 14:09 |
| EmilienM | see how it works | 14:09 |
| openstackgerrit | Kevin Carter (cloudnull) proposed openstack/tripleo-common master: Relax retries, use default pool settings, and tune workers https://review.opendev.org/682731 | 14:09 |
| *** saneax has quit IRC | 14:12 | |
| *** ykarel is now known as ykarel|afk | 14:13 | |
| *** raildo has joined #tripleo | 14:13 | |
| cloudnull | weshay_ EmilienM - so funny enough "tripleo-ci-centos-7-containers-multinodeTIMED_OUT in 3h 48m 58s" - https://review.opendev.org/#/c/682945/ | 14:16 |
| weshay_ | ya.. saw | 14:16 |
| cloudnull | not that that's helpful in any way... | 14:16 |
| EmilienM | yeah | 14:16 |
| EmilienM | wtf | 14:16 |
| EmilienM | was it in post or what? | 14:16 |
| cloudnull | yup | 14:16 |
| weshay_ | tempest | 14:16 |
| cloudnull | 2019-09-19 08:34:13.054226 | TASK [upload-logs-swift : Upload logs to swift] | 14:17 |
| cloudnull | oh nevermind | 14:17 |
| cloudnull | This was the last from the overcloud deploy log | 14:21 |
| cloudnull | 2019-09-19 08:27:11 | TASK [Write kolla config json files] ******************************************* | 14:21 |
| *** ykarel|afk is now known as ykarel|away | 14:22 | |
| *** gfidente has joined #tripleo | 14:23 | |
| mwhahaha | EmilienM: container image build is still using docker by default | 14:24 |
| EmilienM | mwhahaha: you mean the CI job? we have another one for buildah right? | 14:26 |
| mwhahaha | i mean the command | 14:26 |
| EmilienM | ah, probably | 14:26 |
| mwhahaha | https://docs.openstack.org/project-deploy-guide/tripleo-docs/latest/deployment/3rd_party.html#building-new-containers-with-kolla-build | 14:26 |
| mwhahaha | openstakc overcloud container image build is docker by default still | 14:26 |
| EmilienM | mwhahaha: should we change it I guess? | 14:26 |
| mwhahaha | i'd just document it until we get centos8 | 14:27 |
| openstackgerrit | Takashi Kajinami proposed openstack/puppet-tripleo master: Add support to configure token caching in keystone https://review.opendev.org/683010 | 14:32 |
| owalsh | cloudnull: FYI https://github.com/olliewalsh/openstack-ansible-plugins/commit/ffd129d9b04462f4f66095d9d06ab3158c198710 | 14:33 |
| *** pcaruana has quit IRC | 14:33 | |
| *** mmedvede has quit IRC | 14:36 | |
| cloudnull | awesome ! | 14:36 |
| *** mmedvede has joined #tripleo | 14:37 | |
| openstackgerrit | Takashi Kajinami proposed openstack/tripleo-heat-templates master: Add support to configure token caching https://review.opendev.org/683012 | 14:38 |
| openstackgerrit | Takashi Kajinami proposed openstack/tripleo-heat-templates master: DNM: Test token caching in keystone https://review.opendev.org/683172 | 14:40 |
| *** jfrancoa has quit IRC | 14:40 | |
| EmilienM | mwhahaha, cloudnull, weshay_: I'm using skopeo to copy docker.io/tripleomaster into https://quay.io/repository/tripleomaster/ | 14:41 |
| *** ykarel|away has quit IRC | 14:41 | |
| weshay_ | k | 14:41 |
| EmilienM | maybe we can run a few jobs on it | 14:41 |
| chandankumar | stevebaker: Hello | 14:45 |
| cloudnull | EmilienM ++ | 14:45 |
| chandankumar | stevebaker: Please have a look at this patch https://review.opendev.org/#/c/681352/, thansk! | 14:45 |
| *** iurygregory has quit IRC | 14:46 | |
| *** brault has quit IRC | 14:46 | |
| *** iurygregory has joined #tripleo | 14:47 | |
| *** hamdyk has quit IRC | 14:47 | |
| *** rainleander is now known as leanderthal | 14:48 | |
| openstackgerrit | Bogdan Dobrelya (bogdando) proposed openstack/ansible-role-tripleo-modify-image master: Add support for yum caching when buildah updates https://review.opendev.org/683100 | 14:49 |
| openstackgerrit | Bogdan Dobrelya (bogdando) proposed openstack/tripleo-ci master: DNM test CI containers update with yum caching https://review.opendev.org/683101 | 14:50 |
| *** mcornea has quit IRC | 14:51 | |
| *** mcornea has joined #tripleo | 14:52 | |
| *** dtantsur|bbl is now known as dtantsur | 14:54 | |
| openstackgerrit | Alex Schultz proposed openstack/tripleo-common master: Cleanup override file https://review.opendev.org/682455 | 14:55 |
| openstackgerrit | Alex Schultz proposed openstack/tripleo-common master: Remove tacker container code https://review.opendev.org/682457 | 14:56 |
| weshay_ | arxcruz|rover, zbr|ruck doing ok? | 14:57 |
| *** tosky has quit IRC | 14:57 | |
| openstackgerrit | Alex Schultz proposed openstack/tripleo-common master: Remove Trove password constant https://review.opendev.org/682479 | 14:57 |
| arxcruz|rover | weshay_: jobs are running, the containers push pass | 14:58 |
| zbr|ruck | weshay_: not sure yet. | 14:58 |
| weshay_ | k | 14:58 |
| *** tosky has joined #tripleo | 14:58 | |
| *** bogdando has quit IRC | 14:59 | |
| *** dsneddon has joined #tripleo | 15:00 | |
| zbr|ruck | weshay_: there are few things you could help with: a) get kola containter building logs to a size that does not crash my browser (and is humanly browsable) | 15:02 |
| weshay_ | zbr|ruck, ha.. you use a mac :) | 15:02 |
| zbr|ruck | b) check why rdo kibana is down, as we cannot search the logs (already reported on rdo), still i prefer not to be the one doing the uptime checks | 15:03 |
| *** skramaja has quit IRC | 15:03 | |
| zbr|ruck | weshay_: i kinda lied, not really crashed the browser (40GB RAM) but made the machine crumble.... even scrolling was not working. | 15:03 |
| *** slaweq has joined #tripleo | 15:04 | |
| zbr|ruck | and you already know that a side effect of big logs is also extra load on log servers. | 15:04 |
| weshay_ | zbr|ruck, would be interesting to have ansible-role-collect-logs.. automatically role big logs | 15:04 |
| weshay_ | sorry.. roll | 15:04 |
| weshay_ | you know | 15:04 |
| zbr|ruck | my vision gets blurry while scrolling/searching.... by the time i read the right spot I forget what I was looking for ;) | 15:05 |
| weshay_ | zbr|ruck, give me the link to the job you are looking at | 15:05 |
| zbr|ruck | weshay_: the failure from last night http://logs.rdoproject.org/openstack-periodic-master/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-centos-7-master-containers-build-push/51d361e/logs/build.log.txt.gz | 15:05 |
| zbr|ruck | cannot even sent a link to the effetive push timeout | 15:05 |
| weshay_ | http://logs.rdoproject.org/openstack-periodic-master/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-centos-7-master-containers-build-push/51d361e/logs/containers-build-errors.log.txt.gz | 15:05 |
| weshay_ | should be populated | 15:05 |
| zbr|ruck | search for "CalledProcessError" | 15:07 |
| zbr|ruck | better "Unknown error when pushing" | 15:07 |
| zbr|ruck | weshay_: 1st: i am not a kolla expert but my impression is that "'Trying to push the image" code does not have any retry mechanism around it. | 15:11 |
| *** dsneddon has quit IRC | 15:11 | |
| *** dsneddon has joined #tripleo | 15:11 | |
| zbr|ruck | if true, that is pretty bad because networking can fail, is not a question, is guaranteed to fail. | 15:11 |
| zbr|ruck | that is why we use TCP more than UDP :D | 15:12 |
| openstackgerrit | Brent Eagles proposed openstack/tripleo-ci master: Change featurset for octavia standalone to support full tempest https://review.opendev.org/669740 | 15:12 |
| *** jfrancoa has joined #tripleo | 15:13 | |
| weshay_ | 2019-09-19 01:21:35 | ============= SUMMARY =============== | 15:13 |
| weshay_ | 2019-09-19 01:21:35 | Expected................... 135 | 15:13 |
| weshay_ | 2019-09-19 01:21:35 | Built...................... 135 | 15:13 |
| weshay_ | 2019-09-19 01:21:35 | ===================================== | 15:13 |
| weshay_ | zbr|ruck, ^ | 15:13 |
| zbr|ruck | weshay_: *push* failed, everythign faled appart, i want to see it re-trying at least once or twice before failing. | 15:15 |
| weshay_ | zbr|ruck, /me seeing 2019-09-19 00:37:02 | INFO:kolla.common.utils.tempest:[91mhttp://mirror.centos.org/centos/7/virt/x86_64/kvm-common/repodata/repomd.xml: [Errno 14] HTTP Error 403 - Forbidden | 15:15 |
| weshay_ | a lot | 15:15 |
| *** derekh has quit IRC | 15:16 | |
| zbr|ruck | weshay_: the link i sent you about last night failure is almost identical with https://bugs.launchpad.net/tripleo/+bug/1771634 | 15:17 |
| openstack | Launchpad bug 1771634 in tripleo "periodic: container build jobs are failing when pushing to rdo registry (500, 504, read timeout)" [Critical,Fix released] - Assigned to Quique Llorente (quiquell) | 15:17 |
| zbr|ruck | look at the last comment on that ticket | 15:17 |
| zbr|ruck | that is exactly what we encountered | 15:17 |
| weshay_ | zbr|ruck, k.. #sf-ops talk to david simard | 15:18 |
| weshay_ | about the push errors | 15:18 |
| weshay_ | zbr|ruck, or EOD | 15:18 |
| *** derekh has joined #tripleo | 15:18 | |
| *** derekh has quit IRC | 15:18 | |
| openstackgerrit | Alex Schultz proposed openstack/tripleo-heat-templates master: Remove Tacker service https://review.opendev.org/682463 | 15:21 |
| *** ricolin has joined #tripleo | 15:21 | |
| weshay_ | the gate is very red again | 15:22 |
| openstackgerrit | wes hayutin proposed openstack/tripleo-upgrade master: Fix passing tags through infrared to the tripleo-upgrade role. https://review.opendev.org/681268 | 15:22 |
| weshay_ | pip issues in a docs job :( | 15:23 |
| *** hjensas has quit IRC | 15:24 | |
| openstackgerrit | Alex Schultz proposed openstack/tripleo-common master: Cleanup override file https://review.opendev.org/682455 | 15:24 |
| openstackgerrit | Alex Schultz proposed openstack/tripleo-common master: Remove tacker container code https://review.opendev.org/682457 | 15:24 |
| weshay_ | limestone issues as well | 15:24 |
| weshay_ | 2019-09-19 14:28:04.502830 | primary | WARNING: Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError("HTTPConnectionPool(host='mirror.regionone.limestone.openstack.org', port=80): Read timed out. (read timeout=60.0)",)': /pypi/simple/bindep/ | 15:24 |
| openstackgerrit | Alex Schultz proposed openstack/tripleo-common master: Remove Trove password constant https://review.opendev.org/682479 | 15:24 |
| openstackgerrit | wes hayutin proposed openstack/tripleo-quickstart-extras master: Statically import CI reproducer roles https://review.opendev.org/681171 | 15:25 |
| weshay_ | zbr|ruck, over here for a sec | 15:28 |
| *** hberaud is now known as hberaud|school-r | 15:28 | |
| weshay_ | limestone has something wrong in their cdn/proxy | 15:28 |
| weshay_ | zbr|ruck, limestone has multiple failures pip installing bindep | 15:28 |
| weshay_ | zbr|ruck, https://d3f814c2cb98e7325b60-dc7b8fac10688d132d58a9050206c3ae.ssl.cf2.rackcdn.com/682385/1/gate/tripleo-ci-centos-7-undercloud-containers/3f5c9c4/job-output.txt | 15:29 |
| weshay_ | zbr|ruck, is there anything you think we can do about that? | 15:29 |
| zbr|ruck | yeah, this for infra | 15:29 |
| *** yprokule has quit IRC | 15:32 | |
| weshay_ | zbr|ruck, fyi.. http://paste.openstack.org/show/777705/ | 15:34 |
| weshay_ | zbr|ruck, /me opens a bug | 15:35 |
| *** gfidente has quit IRC | 15:35 | |
| *** kopecmartin is now known as kopecmartin|off | 15:35 | |
| weshay_ | zbr|ruck, arxcruz|rover fyi https://bugs.launchpad.net/tripleo/+bug/1844700 | 15:37 |
| openstack | Launchpad bug 1844700 in tripleo "Read timed out. (read timeout=60.0)",)': /pypi/simple/bindep/, ERROR: No matching distribution found for bindep" [Critical,Triaged] | 15:37 |
| weshay_ | zbr|ruck, arxcruz|rover https://review.opendev.org/683188 | 15:40 |
| *** hberaud|school-r is now known as hberaud | 15:43 | |
| weshay_ | mwhahaha, EmilienM fyi.. lots of failures in the gate due to bindep.. https://bugs.launchpad.net/tripleo/+bug/1844700 | 15:46 |
| openstack | Launchpad bug 1844700 in tripleo "Read timed out. (read timeout=60.0)",)': /pypi/simple/bindep/, ERROR: No matching distribution found for bindep" [Critical,Triaged] | 15:46 |
| *** ykarel|away has joined #tripleo | 15:46 | |
| *** Goneri has quit IRC | 15:47 | |
| EmilienM | weshay_: what is root cause? | 15:47 |
| weshay_ | EmilienM, infra | 15:48 |
| weshay_ | node_provider | 15:48 |
| weshay_ | 78% limestone-regionone | 15:48 |
| weshay_ | 21% fortnebula-regionone | 15:48 |
| openstackgerrit | Martin Schuppert proposed openstack/tripleo-common stable/queens: [WIP] Add workflow to do cellv2 host discovery https://review.opendev.org/683067 | 15:51 |
| openstackgerrit | Merged openstack/puppet-tripleo stable/rocky: Safeguard against 'Undef' length() checks. https://review.opendev.org/682386 | 15:52 |
| openstackgerrit | wes hayutin proposed openstack/tripleo-ci master: Run build containers on rocky branch https://review.opendev.org/682335 | 15:53 |
| *** fultonj has quit IRC | 15:54 | |
| *** ykarel|away has quit IRC | 15:54 | |
| zbr|ruck | weshay_: limestone discussion should go in #openstack-infra right? | 15:57 |
| weshay_ | zbr|ruck, ya.. but I'm waiting to see it in one more job | 15:57 |
| weshay_ | <weshay_> node_provider | 15:57 |
| weshay_ | <weshay_> 78% limestone-regionone | 15:57 |
| weshay_ | <weshay_> 21% fortnebula-regionone | 15:57 |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-docs master: Document --use-buildah usage https://review.opendev.org/683194 | 15:59 |
| EmilienM | mwhahaha: ^ done | 15:59 |
| weshay_ | zbr|ruck, thanks | 15:59 |
| zbr|ruck | they confirmed a problem, aparently the pypi CDN issue, the IP given timeout. | 15:59 |
| *** fultonj has joined #tripleo | 16:00 | |
| weshay_ | 2019-09-19 09:55:02.707 17304 WARNING tripleoclient.v1.tripleo_deploy.Deploy [ ] TASK [tripleo-container-image-prepare : Run tripleo-container-image-prepare logged to: /var/log/tripleo-container-image-prepare.log] *** | 16:01 |
| weshay_ | 2019-09-19 10:38:25.200 17304 WARNING tripleoclient.v1.tripleo_deploy.Deploy [ ] changed: [undercloud] | 16:01 |
| weshay_ | 42min | 16:02 |
| weshay_ | in ovh | 16:02 |
| *** marios has quit IRC | 16:02 | |
| *** lmiccini has quit IRC | 16:03 | |
| *** openstackgerrit has quit IRC | 16:06 | |
| *** iurygregory has quit IRC | 16:08 | |
| zbr|ruck | weshay_: good news on kolla, found the lack of retry bug, a fix is in the works. | 16:11 |
| weshay_ | cool | 16:12 |
| *** holser has quit IRC | 16:13 | |
| weshay_ | zbr|ruck, we may also want to look why that wasn't being caputured in the error log | 16:13 |
| zbr|ruck | weshay_: i already know why. the fix will address that. what happened is that it did retried and succeeded but once the status was set to failure it was never reverted by the retry | 16:14 |
| weshay_ | zbr|ruck++ | 16:14 |
| zbr|ruck | weshay_: just lucky, kudos should go to mgoddard on kolla | 16:15 |
| *** alexmcleod has quit IRC | 16:17 | |
| *** ykarel|away has joined #tripleo | 16:18 | |
| *** chem has quit IRC | 16:18 | |
| *** dtantsur is now known as dtantsur|afk | 16:22 | |
| *** weshay_ is now known as weshay | 16:23 | |
| weshay | zbr|ruck, so infra is taking limestone out of the rotation for now? | 16:27 |
| *** udesale has quit IRC | 16:27 | |
| *** udesale has joined #tripleo | 16:28 | |
| jrist | thank you for all of the email, mwhahaha | 16:28 |
| mwhahaha | enjoy | 16:28 |
| mwhahaha | also, you should have done this :D | 16:29 |
| *** Goneri has joined #tripleo | 16:29 | |
| jrist | oh | 16:29 |
| jrist | TIL | 16:29 |
| jrist | still need me to? or was that a script? | 16:29 |
| mwhahaha | i'm almost done i think | 16:29 |
| jrist | I mean, you can just poke me, you know | 16:29 |
| mwhahaha | pfft i need to look like i'm actually doing work | 16:29 |
| mwhahaha | though you may wish to disable the email notifications for a while :D | 16:30 |
| *** suuuper has quit IRC | 16:34 | |
| jrist | haha | 16:35 |
| *** itlinux has joined #tripleo | 16:35 | |
| *** dciabrin has quit IRC | 16:41 | |
| *** Goneri has quit IRC | 16:41 | |
| *** jpich has quit IRC | 16:46 | |
| *** holser has joined #tripleo | 16:47 | |
| *** udesale has quit IRC | 16:52 | |
| cloudnull | wow. looks like we have a 13+ hour queue ? | 16:57 |
| *** devep has quit IRC | 16:58 | |
| weshay | cloudnull, happens fast | 16:58 |
| weshay | it's going to get worse | 16:59 |
| *** pcaruana has joined #tripleo | 17:01 | |
| *** Goneri has joined #tripleo | 17:01 | |
| *** hjensas has joined #tripleo | 17:04 | |
| sshnaidm | mwhahaha, rebased on your patch with volumes validation, but sc012 still fails, maybe another difference between docker and podman? because same task pass in dockeR: https://bbe9f5c1c0e7811201f1-c17654115fca547aa4e7e3bb20a7910e.ssl.cf1.rackcdn.com/682341/5/check/tripleo-ci-centos-7-scenario012-standalone/99eda0f/logs/undercloud/home/zuul/standalone_deploy.log.txt.gz | 17:04 |
| sshnaidm | mwhahaha, although volumes patch worked | 17:05 |
| *** zbr|ruck is now known as zbr | 17:07 | |
| *** Goneri has quit IRC | 17:14 | |
| *** pcaruana has quit IRC | 17:14 | |
| *** jcoufal has joined #tripleo | 17:21 | |
| *** jpena is now known as jpena|off | 17:22 | |
| *** ramishra has quit IRC | 17:23 | |
| *** pcaruana has joined #tripleo | 17:25 | |
| *** amoralej is now known as amoralej|off | 17:26 | |
| *** ricolin has quit IRC | 17:31 | |
| *** openstackgerrit has joined #tripleo | 17:34 | |
| openstackgerrit | Merged openstack/tripleo-quickstart-extras master: Support direct provisioning of bare metal https://review.opendev.org/664456 | 17:34 |
| *** ykarel|away has quit IRC | 17:34 | |
| *** holser has quit IRC | 17:36 | |
| *** pcaruana has quit IRC | 17:55 | |
| *** ansmith_ has joined #tripleo | 17:58 | |
| *** ansmith has quit IRC | 18:00 | |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-quickstart master: DNM - test CI against quay.io https://review.opendev.org/683212 | 18:01 |
| *** brault has joined #tripleo | 18:09 | |
| *** brault has quit IRC | 18:10 | |
| *** Goneri has joined #tripleo | 18:12 | |
| *** HardCase has joined #tripleo | 18:14 | |
| *** HardCase has quit IRC | 18:24 | |
| EmilienM | mwhahaha: we never used that tripleoclient container right? | 18:27 |
| mwhahaha | Correct | 18:27 |
| mwhahaha | Do we still build it? | 18:27 |
| EmilienM | yeah :/ | 18:28 |
| mwhahaha | If there are no plans to use it, kill it | 18:29 |
| EmilienM | will take a look | 18:33 |
| *** gbarros has joined #tripleo | 18:49 | |
| *** radeks has quit IRC | 18:51 | |
| *** lucasagomes has quit IRC | 18:53 | |
| *** brault has joined #tripleo | 18:55 | |
| *** brault has quit IRC | 18:56 | |
| *** brault has joined #tripleo | 18:56 | |
| *** gbarros has quit IRC | 19:01 | |
| *** leanderthal has quit IRC | 19:02 | |
| openstackgerrit | Merged openstack/puppet-tripleo master: Use memcached for token caching in octavia authtoken https://review.opendev.org/680898 | 19:04 |
| openstackgerrit | Merged openstack/tripleo-ci master: Add no_log to skip long list of files https://review.opendev.org/682276 | 19:04 |
| openstackgerrit | Merged openstack/python-tripleoclient master: Remove old releasenote script https://review.opendev.org/682493 | 19:09 |
| openstackgerrit | Merged openstack/tripleo-heat-templates master: Don't use docker override in scenario012 standalone ironic https://review.opendev.org/682206 | 19:09 |
| openstackgerrit | Merged openstack/tripleo-quickstart master: Fail when $VIRTHOST is localhost https://review.opendev.org/672075 | 19:09 |
| openstackgerrit | Merged openstack/python-tripleoclient master: Deprecate overcloud image prepare https://review.opendev.org/682404 | 19:09 |
| openstackgerrit | Merged openstack/tripleo-heat-templates master: Add IPv6 condition to set to the local_address https://review.opendev.org/682057 | 19:09 |
| *** mcornea has quit IRC | 19:10 | |
| *** hberaud is now known as hberaud|gone | 19:12 | |
| weshay | \0/ small victory | 19:12 |
| *** holser has joined #tripleo | 19:14 | |
| *** mcornea has joined #tripleo | 19:15 | |
| *** mcornea has quit IRC | 19:15 | |
| *** mcornea has joined #tripleo | 19:15 | |
| *** brault has quit IRC | 19:16 | |
| *** ansmith_ has quit IRC | 19:25 | |
| *** jcoufal has quit IRC | 19:26 | |
| mwhahaha | sshnaidm: you can't use podman when pacemaker is involved on centos7 | 19:27 |
| sshnaidm | mwhahaha, I thought I removed it: https://review.opendev.org/#/c/682410/ | 19:28 |
| mwhahaha | sshnaidm: that error was for cinder-volume runnin under pacemaker | 19:28 |
| mwhahaha | 2019-09-19 00:38:07 | "Error: Evaluation Error: Error while evaluating a Function Call, The $pcs_password param is undefined, did you forget to include ::tripleo::profile::base::pacemaker in your role? (file: /etc/puppet/modules/tripleo/manifests/profile/base/pacemaker.pp, line: 112, column: 5) on node standalone.localdomain", | 19:28 |
| sshnaidm | mwhahaha, aaa.. ../../deployment/cinder/cinder-volume-pacemaker-puppet.yaml | 19:28 |
| sshnaidm | mwhahaha, I'll change it to something like ../../deployment/cinder/cinder-volume-container-puppet.yaml | 19:30 |
| openstackgerrit | Sagi Shnaidman proposed openstack/tripleo-heat-templates master: Don't use pacemaker in ironic sa job https://review.opendev.org/682410 | 19:30 |
| *** ansmith_ has joined #tripleo | 19:31 | |
| mwhahaha | yes | 19:31 |
| *** jfrancoa has quit IRC | 19:37 | |
| openstackgerrit | Matthew Booth proposed openstack/diskimage-builder master: Fix invalid element-dep in vm https://review.opendev.org/683221 | 19:51 |
| *** pcaruana has joined #tripleo | 19:51 | |
| *** HardCase has joined #tripleo | 20:00 | |
| openstackgerrit | Harald Jensås proposed openstack/tripleo-heat-templates master: Use _uri which is wrapped if IPv6 for ironic tftp https://review.opendev.org/683224 | 20:01 |
| *** mmethot_ has joined #tripleo | 20:10 | |
| *** mmethot has quit IRC | 20:11 | |
| *** holser has quit IRC | 20:20 | |
| *** pcaruana has quit IRC | 20:31 | |
| *** ansmith_ has quit IRC | 20:34 | |
| *** holser has joined #tripleo | 20:34 | |
| *** sshnaidm is now known as sshnaidm|off | 20:42 | |
| *** brault has joined #tripleo | 20:51 | |
| *** bfournie has quit IRC | 20:53 | |
| *** brault has quit IRC | 20:55 | |
| *** ade_lee__ has quit IRC | 20:56 | |
| *** Vorrtex has quit IRC | 20:58 | |
| *** slaweq has quit IRC | 21:01 | |
| *** brault has joined #tripleo | 21:02 | |
| openstackgerrit | Merged openstack/tripleo-puppet-elements stable/queens: Add proper permissions on script https://review.opendev.org/682537 | 21:08 |
| *** brault has quit IRC | 21:08 | |
| *** pierreprinetti has joined #tripleo | 21:11 | |
| *** mcornea has quit IRC | 21:12 | |
| *** pierrepr1netti has quit IRC | 21:12 | |
| *** aakarsh has quit IRC | 21:13 | |
| *** brault has joined #tripleo | 21:17 | |
| HardCase | Hey all, looking for help. I have a deployment that hangs on a step that says DELETE_IN_PROGRESS but never actually completes. Looking at the details of the stack it references a computeiha node that is 'unknown'. Anyone know how I can get this deployment to finish? | 21:29 |
| *** ansmith_ has joined #tripleo | 21:34 | |
| *** lbragstad has quit IRC | 21:37 | |
| *** lbragstad has joined #tripleo | 21:37 | |
| *** ade_lee__ has joined #tripleo | 21:39 | |
| *** ade_lee__ has quit IRC | 21:45 | |
| *** ade_lee__ has joined #tripleo | 21:46 | |
| openstackgerrit | Steve Baker proposed openstack/tripleo-heat-templates master: Create an endpoint entry for novajoin https://review.opendev.org/683241 | 21:47 |
| *** holser has quit IRC | 21:48 | |
| *** xek has quit IRC | 21:51 | |
| *** bfournie has joined #tripleo | 21:54 | |
| *** ade_lee__ has quit IRC | 21:55 | |
| *** ade_lee__ has joined #tripleo | 21:56 | |
| *** raildo has quit IRC | 22:01 | |
| *** mmethot_ has quit IRC | 22:02 | |
| *** mmethot_ has joined #tripleo | 22:02 | |
| *** brault has quit IRC | 22:22 | |
| *** rlandy is now known as rlandy|biab | 22:30 | |
| *** ksambor has quit IRC | 22:37 | |
| *** holser has joined #tripleo | 22:38 | |
| *** Tengu has quit IRC | 22:51 | |
| *** holser has quit IRC | 22:59 | |
| *** tkajinam has joined #tripleo | 23:02 | |
| *** Tengu has joined #tripleo | 23:03 | |
| *** tosky has quit IRC | 23:12 | |
| *** rlandy|biab is now known as rlandy|bbl | 23:18 | |
| *** stevebaker has quit IRC | 23:23 | |
| *** rcernin has joined #tripleo | 23:25 | |
| *** artom has joined #tripleo | 23:48 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!