| *** irclogbot_0 has quit IRC | 02:09 | |
| *** irclogbot_2 has joined #oooq | 02:11 | |
| *** saneax has joined #oooq | 03:18 | |
| *** saneax has quit IRC | 03:37 | |
| *** recheck_ has joined #oooq | 04:35 | |
| *** recheck has quit IRC | 04:38 | |
| *** d0ugal has quit IRC | 04:38 | |
| *** aakarsh has quit IRC | 04:48 | |
| *** d0ugal has joined #oooq | 04:52 | |
| *** ykarel|away has joined #oooq | 05:09 | |
| *** marios has joined #oooq | 05:19 | |
| *** kopecmartin|off is now known as kopecmartin | 05:49 | |
| *** jfrancoa has joined #oooq | 05:54 | |
| *** tesseract has joined #oooq | 06:57 | |
| *** jbadiapa has joined #oooq | 07:00 | |
| zbr|ruck | morning! | 07:03 |
|---|---|---|
| *** jaosorior has joined #oooq | 07:03 | |
| zbr|ruck | arxcruz|ruck: see why https://review.opendev.org/#/c/683350/ is still failing on rocky | 07:03 |
| *** jbadiapa has quit IRC | 07:13 | |
| zbr|ruck | arxcruz|ruck: aparently r8 jobs are failing because tripleo_common.image.exception.ImageNotFoundException: Not found image: docker://trunk.registry.rdoproject.org/tripleomaster/rhel-binary-aodh-api:fd46c80632bbd7abc8411277bfc4819f6b95a485_bf6d0acd | 07:16 |
| zbr|ruck | the image is not there indeeed/ | 07:16 |
| *** tosky has joined #oooq | 07:17 | |
| *** jbadiapa has joined #oooq | 07:19 | |
| zbr|ruck | arxcruz|ruck: or panda : give https://review.rdoproject.org/r/#/c/22754/ a kick | 07:25 |
| *** pierreprinetti has joined #oooq | 07:28 | |
| *** amoralej|off is now known as amoralej | 07:30 | |
| *** bogdando has joined #oooq | 07:33 | |
| *** holser has joined #oooq | 07:34 | |
| *** jbadiapa has quit IRC | 07:34 | |
| zbr|ruck | panda: are you here? i wanted to ask something regarding promotion | 07:42 |
| *** jpena|off is now known as jpena | 07:48 | |
| *** jpena is now known as jpena|brb | 08:10 | |
| *** weshay has quit IRC | 08:15 | |
| *** weshay has joined #oooq | 08:16 | |
| *** ykarel|away has quit IRC | 08:28 | |
| *** ykarel has joined #oooq | 08:30 | |
| *** ykarel is now known as ykarel|away | 08:30 | |
| *** jpena|brb is now known as jpena | 08:32 | |
| *** jaosorior has quit IRC | 08:37 | |
| *** derekh has joined #oooq | 08:38 | |
| zbr|ruck | does anyone know why we have centos on ssh deny on promoter? | 08:40 |
| *** panda has quit IRC | 08:48 | |
| *** panda has joined #oooq | 08:49 | |
| *** holser has quit IRC | 09:02 | |
| *** holser has joined #oooq | 09:02 | |
| *** holser has quit IRC | 09:03 | |
| *** holser has joined #oooq | 09:03 | |
| *** jfrancoa has quit IRC | 09:03 | |
| *** jfrancoa has joined #oooq | 09:18 | |
| *** jbadiapa has joined #oooq | 09:30 | |
| *** holser has quit IRC | 09:34 | |
| *** holser has joined #oooq | 09:36 | |
| *** jaosorior has joined #oooq | 09:52 | |
| zbr|ruck | marios: hey! | 09:54 |
| marios | zbr|ruck: food brb | 10:25 |
| *** ykarel|away has quit IRC | 10:31 | |
| zbr|ruck | marios: i will also go for food+swim, brb in in 1.5h, as I supposed it will be a long day after. | 10:31 |
| marios | ack zbr|ruck i have a call at 4 with folco maybe that will be a good time for whatever/more folks will be around too | 10:33 |
| marios | zbr|ruck: err... in T+2.5 hours | 10:33 |
| *** zbr|ruck is now known as zbr|lunch | 10:39 | |
| *** dsneddon has quit IRC | 10:44 | |
| *** tosky_ has joined #oooq | 10:49 | |
| *** tosky has quit IRC | 10:52 | |
| *** jbadiapa has quit IRC | 10:57 | |
| *** tosky_ is now known as tosky | 10:58 | |
| *** tosky_ has joined #oooq | 11:02 | |
| *** tosky is now known as Guest18639 | 11:02 | |
| *** tosky_ is now known as tosky | 11:02 | |
| *** jaosorior has quit IRC | 11:02 | |
| *** Guest18639 has quit IRC | 11:04 | |
| *** dsneddon has joined #oooq | 11:09 | |
| *** openstackstatus has quit IRC | 11:09 | |
| *** rnoriega- has quit IRC | 11:09 | |
| *** rnoriega_ has joined #oooq | 11:13 | |
| marios | logs.rdoproject very slow right now? | 11:19 |
| *** jpena is now known as jpena|lunch | 11:24 | |
| *** jaosorior has joined #oooq | 11:28 | |
| *** dsneddon has quit IRC | 11:55 | |
| *** dsneddon has joined #oooq | 12:03 | |
| *** jbadiapa has joined #oooq | 12:06 | |
| *** amoralej is now known as amoralej|lunch | 12:07 | |
| *** dsneddon has quit IRC | 12:13 | |
| *** dsneddon has joined #oooq | 12:19 | |
| weshay | arxcruz|ruck, feel free to mention the issue.. highlighting the links, but probably no need to go in depth | 12:21 |
| *** dsneddon has quit IRC | 12:25 | |
| *** derekh has quit IRC | 12:27 | |
| arxcruz|ruck | weshay: i believe you sumarize everything pretty well :) | 12:27 |
| arxcruz|ruck | and I was too late to see your message here :P | 12:27 |
| *** jpena|lunch is now known as jpena | 12:29 | |
| weshay | arxcruz|ruck, can you check http://dashboard-ci.tripleo.org/d/si1tipHZk/jobs-exploration?orgId=1&from=now-7d&to=now and see if the logging changes have impacted the regular expressoin | 12:29 |
| weshay | arxcruz|ruck++ | 12:29 |
| weshay | panda, rfolco can we chat in a few re: queens | 12:33 |
| * weshay doesn't want to do anything on the promoter w/o your ack | 12:33 | |
| rfolco | weshay, k | 12:33 |
| weshay | arxcruz|ruck, fyi.. I assigned this one to you btw.. https://trello.com/c/CQljcztK/1161-cixlp1846270tripleociproa-unhandled-ioerror-tempest-queens-ovb | 12:35 |
| *** jbadiapa has quit IRC | 12:35 | |
| weshay | arxcruz|ruck, seems like kforde fix the oom's so you should be able to get that far now | 12:35 |
| arxcruz|ruck | weshay: doesn't seems to be broken, but the hour is rounding to low, one of the logs tooke 72 minutes, and it's showing as 1 hour, less than 1 hour is showing correctly the minutes | 12:36 |
| weshay | hrm.. k.. I see a lot of zeros.. thought something may have changed.. meh.. we'll continue to poke at that as time permits | 12:37 |
| *** rlandy has joined #oooq | 12:40 | |
| weshay | zbr|lunch, if you want to kick queens off.. that's fine. just waiting for panda to get back from lunch or what ever | 12:41 |
| arxcruz|ruck | weshay: the 0 comes from upgrade jobs, molecule, and tripleo-ci-centos-7-undercloud-oooq who doesn't have either the standalone_deploy.log.txt.gz or install-undercloud.log.txt.gz files that the script looks for | 12:41 |
| arxcruz|ruck | weshay: what about stein? I see today only fails the fs035 due timeout and the upgrade job | 12:42 |
| weshay | arxcruz|ruck, I see plenty of jobs that should have it... that do not | 12:42 |
| weshay | 2019-10-02 12:21:30 | 12:42 |
| weshay | master | 12:42 |
| weshay | 684295 | 12:42 |
| weshay | tripleo-ci-centos-7-containers-multinode | 12:42 |
| weshay | 3 hours, 16 minutes | 12:42 |
| weshay | YES | 12:42 |
| weshay | SUCCESS | 12:42 |
| weshay | 0 ns | 12:42 |
| weshay | fortnebula | 12:42 |
| arxcruz|ruck | weshay: let me check | 12:42 |
| weshay | 2019-10-02 10:20:24.720 12542 WARNING tripleoclient.v1.tripleo_deploy.Deploy [ ] tripleo-container-image-prepare : Run tripleo-container-image-prepare logged to: /var/log/tripleo-container-image-prepare.log 1114.59s | 12:43 |
| weshay | here's another one | 12:43 |
| weshay | 2019-10-02 12:03:10master683350tripleo-ci-centos-7-containers-multinode3 hours, 30 minutesYESSUCCESS0 nsovhGRA1 | 12:43 |
| arxcruz|ruck | weshay: you are right, this one the regex didn't get | 12:48 |
| arxcruz|ruck | weshay: in my defense, i'm not seeing these in my dashboard :P | 12:48 |
| weshay | arxcruz|ruck, no worries mate | 12:49 |
| rlandy | chandankumar: hi - where are we with the train stuff? | 12:50 |
| *** amoralej|lunch is now known as amoralej | 12:51 | |
| arxcruz|ruck | weshay: i'm submiting the new regex that will handle both cases | 12:52 |
| *** aakarsh has joined #oooq | 12:52 | |
| weshay | arxcruz|ruck++ | 12:53 |
| arxcruz|ruck | weshay: https://regex101.com/r/rUZeYv/1 | 12:53 |
| *** Goneri has joined #oooq | 12:53 | |
| weshay | arxcruz|ruck, zbr|lunch can we summarize and write any rhel8 master/train issues so we can get a good idea where we are at | 12:56 |
| * weshay also is going to add something to the cockpit for rhel8 today unless either of you are interested in doing htat | 12:56 | |
| weshay | that | 12:56 |
| *** dsneddon has joined #oooq | 12:58 | |
| *** derekh has joined #oooq | 12:59 | |
| arxcruz|ruck | weshay: Run.*tripleo-container-image-prepare.log\s*\-?\s*([1-9]*\.?[1-9]*)s | 12:59 |
| rfolco | marios, sorry I was distracted debugging something | 13:02 |
| rfolco | marios, we meet now ? | 13:02 |
| marios | rfolco: yeah | 13:03 |
| marios | rfolco: join us | 13:04 |
| chandankumar | rlandy: welcome back | 13:05 |
| chandankumar | rlandy: on train, we are waiting for reviews to merge, we cannot do anything | 13:05 |
| rlandy | chandankumar: ok - thanks | 13:05 |
| *** jaosorior has quit IRC | 13:08 | |
| chandankumar | arxcruz|ruck: does the rocky job failure got fixed? | 13:09 |
| arxcruz|ruck | chandankumar: yes | 13:09 |
| arxcruz|ruck | maybe you need to rebase, i got rocky fs020 passing in one of my patches already | 13:09 |
| weshay | zbr|lunch, ping me when you are done lunching :) need your docker sk$ilz | 13:14 |
| weshay | arxcruz|ruck, perhaps a testproject patch to kick rocky periodic would be ++ | 13:15 |
| weshay | when you think it's time | 13:15 |
| weshay | although it may kick today | 13:15 |
| weshay | in openstack-periodic-wednesday-weekend | 13:16 |
| chandankumar | arxcruz|ruck: marios rlandy https://review.opendev.org/#/c/686149/ | 13:18 |
| arxcruz|ruck | weshay: it will fail because the skip list patch isn't merged yet | 13:20 |
| weshay | k | 13:20 |
| weshay | baseurl=https://trunk.rdoproject.org/centos7-queens/e2/20/e2208117a31eea79c50330333501b6581f9faafe_15a9b671 | 13:28 |
| weshay | marios, rfolco rlandy please review this today https://review.rdoproject.org/r/#/c/22743/ | 13:42 |
| marios | weshay: ack will check in bit | 13:43 |
| *** ykarel has joined #oooq | 13:45 | |
| *** zbr|lunch is now known as zbr|ruck | 13:48 | |
| zbr|ruck | weshay: got back | 13:48 |
| weshay | zbr|ruck, hello sir :) | 13:48 |
| weshay | zbr|ruck, join me on a meet | 13:48 |
| weshay | zbr|ruck, https://meet.google.com/zwe-wdex-waw?authuser=1 | 13:49 |
| weshay | zbr|ruck, http://logs.rdoproject.org/openstack-periodic-wednesday-weekend/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-queens/b1cb01f/job-output.txt.gz | 13:50 |
| weshay | 2019-09-30 07:08:07.113399 | TASK [ansible-role-container-registry : Install Docker] | 13:50 |
| weshay | zbr|ruck, https://review.rdoproject.org/r/22776 | 13:51 |
| weshay | zbr|ruck, https://bugs.launchpad.net/tripleo/+bug/1845166/comments/22 | 13:51 |
| openstack | Launchpad bug 1845166 in tripleo "[queens] [Periodic][check] OVB jobs failing in OC deploy when trying to ssh to nodes" [Critical,Triaged] | 13:51 |
| weshay | zbr|ruck, https://docs.docker.com/engine/reference/commandline/dockerd/ | 13:51 |
| weshay | zbr|ruck, --iptables Enable addition of iptables rules (default true) | 13:52 |
| weshay | zbr|ruck, https://github.com/openstack/ansible-role-container-registry/blob/master/tasks/install-engine.yml#L8 | 13:54 |
| weshay | zbr|ruck, https://github.com/openstack/ansible-role-container-registry/blob/master/tasks/install-engine.yml#L14 | 13:54 |
| *** jaosorior has joined #oooq | 13:54 | |
| zbr|ruck | rfolco: hi | 13:59 |
| zbr|ruck | i need your help re promoter a little bit | 14:00 |
| zbr|ruck | arxcruz|ruck: if you can attend the esca meeting it would be good | 14:00 |
| arxcruz|ruck | zbr|ruck: ok | 14:00 |
| zbr|ruck | arxcruz|ruck: if anyone asks: i am working on promoter and on queens docker error with iptables. i did not had any time to look at scenario 004. | 14:01 |
| zbr|ruck | rfolco: did you re-enable sync on promoter and if so, where is the place to turn it on/off. | 14:01 |
| zbr|ruck | my guess, you did not, i still see my cherry pick which is fine. | 14:02 |
| weshay | zbr|ruck, Alex is saying shutting off iptables will be fine.. the undercloud doesn't use docker in a way that we'll be impacted | 14:02 |
| zbr|ruck | weshay: please post me link to role you showed me and I can make a change there. | 14:02 |
| zbr|ruck | ahh, is already ^ | 14:03 |
| weshay | https://github.com/openstack/ansible-role-container-registry/blob/master/tasks/install-engine.yml#L14 | 14:03 |
| *** panda has quit IRC | 14:06 | |
| *** panda has joined #oooq | 14:08 | |
| weshay | zbr|ruck, merging https://review.rdoproject.org/r/#/c/22743/ | 14:10 |
| weshay | zbr|ruck, I'm back.. call canceled | 14:10 |
| weshay | zbr|ruck, another place to potentially do your trick re: login verification https://github.com/openstack/ansible-role-container-registry/blob/master/tasks/registry-login.yml | 14:11 |
| weshay | ok.. so that's enough work for you :) | 14:11 |
| weshay | arxcruz|ruck, ping me when you have a sec | 14:11 |
| zbr|ruck | weshay: i have some problems understanding where to add the iptables obption because on my c7 system I do not have any /etc/sysconfig/docker | 14:11 |
| weshay | zbr|ruck, I pointed you at the doc :) | 14:12 |
| *** gkadam has joined #oooq | 14:12 | |
| *** gkadam has quit IRC | 14:12 | |
| weshay | zbr|ruck, have that mac.. fire up a vm doood | 14:12 |
| zbr|ruck | installing docker-ce does not create it, but looking at service definition there is a way to control the service | 14:12 |
| zbr|ruck | i said "C7" centos7 | 14:12 |
| weshay | lolz | 14:12 |
| zbr|ruck | take a look http://paste.openstack.org/show/780756/ | 14:13 |
| zbr|ruck | so ideal place is to define DOCKER_NETWORK_OPTIONS variable | 14:13 |
| rfolco | zbr|ruck, re-enabled the service only. The service script is not doing continuous delivery and not downloading latest code from ci-config | 14:13 |
| arxcruz|ruck | weshay: ping | 14:13 |
| zbr|ruck | but i am not sure if older versions of docker can load it, need to check with centos own one. | 14:13 |
| weshay | arxcruz|ruck, howdy | 14:13 |
| weshay | arxcruz|ruck, pita here.. let's chat for asec | 14:13 |
| arxcruz|ruck | weshay: google meet ? | 14:14 |
| weshay | arxcruz|ruck, ah sorry https://meet.google.com/zcq-xwqx-rco?authuser=1 | 14:15 |
| *** jaosorior has quit IRC | 14:18 | |
| arxcruz|ruck | weshay: https://review.opendev.org/#/c/683350/ | 14:22 |
| weshay | arxcruz|ruck, 2019-10-02 08:34:40 | tripleo_common.image.exception.ImageNotFoundException: Not found image: docker://trunk.registry.rdoproject.org/tripleomaster/rhel-binary-aodh-api:fd46c80632bbd7abc8411277bfc4819f6b95a485_bf6d0acd | 14:26 |
| *** openstackstatus has joined #oooq | 14:27 | |
| *** ChanServ sets mode: +v openstackstatus | 14:27 | |
| Tengu | weshay: heya! would you be so kind to handle the cix call? would be really nice :) | 14:34 |
| weshay | arxcruz|ruck, http://logs.rdoproject.org/openstack-periodic-master/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-rhel-8-buildimage-ironic-python-agent-master/6430aa9/job-output.txt.gz | 14:35 |
| weshay | Tengu, ya | 14:35 |
| Tengu | weshay: thanks :) | 14:36 |
| arxcruz|ruck | weshay: https://logs.rdoproject.org/openstack-periodic-master/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-rhel-8-buildimage-ironic-python-agent-master/6430aa9/job-output.txt.gz#_2019-10-01_14_52_03_748232 | 14:38 |
| arxcruz|ruck | (╯°□°）╯︵ ┻━┻ | 14:38 |
| *** Vorrtex has joined #oooq | 14:47 | |
| arxcruz|ruck | weshay: rejoining | 14:48 |
| panda | back | 14:49 |
| arxcruz|ruck | weshay: we need to change the script, because it's failing due the disk space, but the job is marking as success | 14:50 |
| arxcruz|ruck | i'm not finding the curl script though | 14:50 |
| arxcruz|ruck | going to german class, back in a few hours, but i'll be with my laptop | 14:53 |
| zbr|ruck | weshay: we need to talk... apparently this is caused by use of "tasks_from" on role import, something I strongly advocated against. | 14:57 |
| panda | did I miss anything ? | 14:59 |
| panda | ouch | 15:01 |
| panda | I think I overwrote some change | 15:01 |
| panda | no, just rebases | 15:02 |
| weshay | arxcruz|ruck, https://meet.google.com/ucy-kvxo-skn?authuser=1 | 15:02 |
| weshay | panda, ur off.. go away | 15:03 |
| panda | only half day off ... | 15:03 |
| panda | I was able to work a bit on the airplane, thenks to functional tests | 15:03 |
| weshay | panda, ah.. welcome back.. get to work | 15:04 |
| weshay | :) | 15:05 |
| weshay | arxcruz|ruck, jump on the meet | 15:05 |
| weshay | jpena++ | 15:05 |
| weshay | thanks | 15:05 |
| rlandy | panda: hello | 15:06 |
| rlandy | panda: see you are busy refactoring test | 15:06 |
| weshay | zbr|ruck, arxcruz|ruck /me in https://meet.google.com/ucy-kvxo-skn?authuser=1 | 15:07 |
| weshay | arxcruz|ruck, jpena freed up the space | 15:07 |
| weshay | panda, did the promoter server ever promote rhel-8.. I think it did a couple times ya? | 15:08 |
| panda | weshay: yes, we had at least one | 15:08 |
| panda | rlandy: yep, wanna sync to take the work back ? | 15:08 |
| rlandy | panda: ok | 15:09 |
| *** Vorrtex has quit IRC | 15:10 | |
| panda | rlandy: do you know how to create a google meet ? I don't ... :( | 15:10 |
| rlandy | panda: no - but I'll try | 15:11 |
| rlandy | panda: https://meet.google.com/qnc-jkyp-hvp | 15:13 |
| *** dtrainor has quit IRC | 15:21 | |
| *** aakarsh has quit IRC | 15:29 | |
| *** marios has quit IRC | 15:35 | |
| weshay | zbr|ruck, when you have a patch, do a depends on https://review.rdoproject.org/r/#/c/22618/ and recheck | 15:39 |
| *** tosky_ has joined #oooq | 15:41 | |
| *** tosky is now known as Guest83058 | 15:41 | |
| *** tosky_ is now known as tosky | 15:41 | |
| *** Guest83058 has quit IRC | 15:43 | |
| weshay | arxcruz|ruck, when ur back from class FYI https://review.rdoproject.org/r/22785 | 15:46 |
| *** aakarsh has joined #oooq | 15:50 | |
| *** dtrainor has joined #oooq | 15:51 | |
| *** ykarel is now known as ykarel|away | 16:01 | |
| *** jfrancoa has quit IRC | 16:04 | |
| *** aakarsh has quit IRC | 16:17 | |
| *** jfrancoa has joined #oooq | 16:20 | |
| *** rlandy is now known as rlandy|brb | 16:22 | |
| *** tesseract has quit IRC | 16:29 | |
| *** bogdando has quit IRC | 16:30 | |
| zbr|ruck | anyone https://review.rdoproject.org/r/#/c/22754/ | 16:36 |
| *** rlandy|brb is now known as rlandy | 16:37 | |
| weshay | arxcruz|ruck, when ur back.. fyi 22793 | 16:39 |
| weshay | oops | 16:39 |
| weshay | https://review.rdoproject.org/r/22793 images built successfully and are on the server :) | 16:39 |
| *** tosky_ has joined #oooq | 16:40 | |
| *** aakarsh has joined #oooq | 16:40 | |
| weshay | rfolco, https://review.rdoproject.org/r/#/c/22754/ | 16:41 |
| *** tosky has quit IRC | 16:43 | |
| *** derekh has quit IRC | 16:43 | |
| rfolco | weshay, hmm sorry, dentist | 16:44 |
| *** rfolco is now known as rfolco|dentist | 16:44 | |
| *** tosky_ is now known as tosky | 16:45 | |
| weshay | np | 16:46 |
| zbr|ruck | weshay: https://review.rdoproject.org/r/#/c/22794/ (simple) | 16:49 |
| chandankumar | rlandy: please +w https://review.opendev.org/#/q/topic:train_release remaining train patches | 16:52 |
| rlandy | https://review.opendev.org/#/c/684295/ is workflowed - failed testing | 16:53 |
| rlandy | chandankumar: https://review.opendev.org/#/c/684260/ upgrades tests -which would use this code fail on not finding the release file for train. These tests should go further once the release file is merged. | 16:56 |
| chandankumar | rlandy: scenario 10 standalone octavia failure | 16:56 |
| rlandy | for this patch, I would doa depends on | 16:57 |
| rlandy | or retest when merged | 16:57 |
| chandankumar | ok sure | 16:57 |
| rlandy | bunch of rdo failures | 16:58 |
| rlandy | which I would guess are unrelated | 16:58 |
| rlandy | the rest is fine and w+ | 16:58 |
| *** dsneddon has quit IRC | 16:58 | |
| zbr|ruck | weshay: i wonder when I will see the next promotion as I have some code on promoter server that I want to see if it works as intended. | 17:01 |
| weshay | zbr|ruck, you get the change into tqe re: queens and iptables? that could be our next | 17:01 |
| weshay | zbr|ruck, we need to get the build images job passing in queens.. and I'll promote it | 17:01 |
| zbr|ruck | weshay: like you asked me, on https://review.rdoproject.org/r/#/c/22618/ | 17:01 |
| *** holser has quit IRC | 17:02 | |
| zbr|ruck | one more hour | 17:02 |
| weshay | zbr|ruck, we need the fs002 image upload job | 17:02 |
| weshay | zbr|ruck, or else we can't promote | 17:03 |
| weshay | image upload is what takes the overcloud images and initially pushes them to the image server | 17:03 |
| weshay | can be done in another review | 17:03 |
| * weshay gets | 17:03 | |
| weshay | zbr|ruck, periodic-tripleo-ci-centos-7-ovb-1ctlr_1comp-featureset002-queens-upload | 17:04 |
| weshay | zbr|ruck, please fire that one off too | 17:04 |
| zbr|ruck | ok | 17:04 |
| *** dtrainor has quit IRC | 17:07 | |
| *** dtrainor has joined #oooq | 17:11 | |
| zbr|ruck | panda: still here? i wonder if we should not make molecule-delegated voting, before we brake it. | 17:22 |
| *** ykarel|away has quit IRC | 17:23 | |
| *** jpena is now known as jpena|off | 17:27 | |
| *** dsneddon has joined #oooq | 17:29 | |
| rlandy | weshay: 1-on-1? | 17:33 |
| *** dsneddon has quit IRC | 17:37 | |
| *** amoralej is now known as amoralej|off | 17:40 | |
| weshay | rlandy, aye.. sorry had to deal | 17:40 |
| weshay | w/ the stuff | 17:40 |
| rlandy | weshay: k - can skip if you want to | 17:41 |
| rlandy | in google meet if you want to join | 17:41 |
| *** dsneddon has joined #oooq | 17:42 | |
| *** dsneddon has quit IRC | 17:53 | |
| *** ykarel|away has joined #oooq | 17:57 | |
| *** aakarsh has quit IRC | 18:13 | |
| *** dsneddon has joined #oooq | 18:15 | |
| *** ykarel|away has quit IRC | 18:16 | |
| rlandy | panda: just checking - we're still waiting on your other patches to merge? or I should just go ahead and split the test patch now? | 18:54 |
| panda | rlandy: split, the merge will take some time still | 18:59 |
| rlandy | panda: ack ok | 19:00 |
| *** holser has joined #oooq | 19:02 | |
| arxcruz|ruck | weshay: hi boss, i'm back, anything you need from me? | 19:04 |
| panda | rlandy: we need to be sure that whatever we merge we is not going to break production server first, as happened on monday. | 19:04 |
| *** kopecmartin is now known as kopecmartin|off | 19:05 | |
| rlandy | panda: ack - ok - will do the split later - busy with ilo atm | 19:05 |
| weshay | arxcruz|ruck, ya | 19:10 |
| weshay | arxcruz|ruck, you getting sick of me yet? | 19:10 |
| arxcruz|ruck | weshay: come on, i love you :* | 19:10 |
| weshay | arxcruz|ruck, /me looking at queens and rhel8 | 19:11 |
| weshay | arxcruz|ruck, pick one.. I can give you context | 19:11 |
| weshay | arxcruz|ruck, btw.. nice view https://trunk-primary.rdoproject.org/api-centos-queens/api/civotes_detail.html?commit_hash=e2208117a31eea79c50330333501b6581f9faafe&distro_hash=15a9b671b405f92b4fc5f2f8b12ae92dfe136fd8 | 19:11 |
| weshay | https://meet.google.com/age-qvrx-nqe | 19:11 |
| weshay | arxcruz|ruck, ^ | 19:11 |
| *** ykarel has joined #oooq | 19:14 | |
| weshay | arxcruz|ruck, https://trunk-primary.rdoproject.org/api-centos-queens/api/civotes_detail.html?commit_hash=e2208117a31eea79c50330333501b6581f9faafe&distro_hash=15a9b671b405f92b4fc5f2f8b12ae92dfe136fd8 | 19:14 |
| weshay | arxcruz|ruck, https://review.rdoproject.org/r/#/c/22793/ | 19:15 |
| weshay | arxcruz|ruck, https://review.rdoproject.org/r/#/c/22793/2/.zuul.yaml | 19:20 |
| weshay | arxcruz|ruck, https://review.opendev.org/#/c/685449/ | 19:22 |
| *** ccamacho has joined #oooq | 19:24 | |
| weshay | https://review.opendev.org/#/c/682853/ | 19:24 |
| weshay | arxcruz|ruck, [whayutin@localhost testproject]$ cat .gitreview | 19:26 |
| weshay | [gerrit] | 19:26 |
| weshay | host=review.rdoproject.org | 19:26 |
| weshay | port=29418 | 19:26 |
| weshay | project=testproject | 19:26 |
| weshay | defaultbranch=master | 19:26 |
| *** ccamacho has quit IRC | 19:28 | |
| weshay | 2019-10-02 07:21:22 | TASK [Configure octavia on overcloud] ****************************************** | 19:29 |
| *** ykarel is now known as ykarel|away | 19:30 | |
| *** holser has quit IRC | 19:40 | |
| weshay | tripleo-ci-centos-7-scenario010-standaloneopenstack/tripleo-commonmasterperiodicmaster56212019-10-02T06:16:54FAILURE | 19:44 |
| weshay | arxcruz|ruck, ^ | 19:44 |
| *** ykarel|away has quit IRC | 19:46 | |
| arxcruz|ruck | weshay: https://review.rdoproject.org/r/#/c/22800/ | 19:52 |
| *** holser has joined #oooq | 20:00 | |
| *** aakarsh has joined #oooq | 20:09 | |
| rlandy | weshay: I am reducing the frequency of checking for changes when triggering the bm periodic jobs. The check is happening every hour now. I'd like to move it o very 4 hours | 20:17 |
| weshay | k | 20:17 |
| rlandy | http://git.app.eng.bos.redhat.com/git/openstack/tripleo-ci-internal-config.git/tree/zuul.d/pipelines.yaml | 20:17 |
| weshay | arxcruz|ruck, https://review.rdoproject.org/r/22802 | 20:20 |
| arxcruz|ruck | weshay: https://review.rdoproject.org/r/#/c/22793/ | 20:24 |
| rlandy | weshay: fyi ... https://code.engineering.redhat.com/gerrit/182596 Reduce bm jobs runs to the same frequency as periodic | 20:31 |
| weshay | arxcruz|ruck, don't see logs https://review.rdoproject.org/r/#/c/22793/ | 20:34 |
| weshay | rlandy, is there a trick to get logging to run from testproject jobs? | 20:35 |
| weshay | rlandy, https://review.rdoproject.org/r/#/c/22793/ | 20:35 |
| rlandy | logging should run - checking | 20:35 |
| weshay | 2019-10-02 20:14:54.746610 | TASK [Collect logs from OVB before deleting nodes] | 20:35 |
| weshay | ya.. ok | 20:36 |
| rlandy | http://logs.rdoproject.org/93/22793/1/check/periodic-tripleo-ci-rhel-8-ovb-3ctlr_1comp-featureset001-master/6b8af1e/ | 20:36 |
| arxcruz|ruck | weshay: http://logs.rdoproject.org/93/22793/3/check/periodic-tripleo-ci-rhel-8-ovb-3ctlr_1comp-featureset001-master/0ce4bae/logs/ | 20:36 |
| weshay | 2019-10-02 20:20:55.972446 | Timeout exception waiting for the logger. Please check connectivity to [38.145.35.84:19885] | 20:36 |
| weshay | arxcruz|ruck, ah I'm blind I guess??? | 20:36 |
| * weshay looks | 20:36 | |
| rlandy | let's see what happens with envE now | 20:36 |
| arxcruz|ruck | weshay: https://logs.rdoproject.org/93/22793/3/check/periodic-tripleo-ci-rhel-8-ovb-3ctlr_1comp-featureset001-master/0ce4bae/logs/undercloud/home/zuul/undercloud_install.log.txt.gz#_2019-10-02_16_14_35 | 20:38 |
| weshay | Writing manifest to image destination\\n Storing signatures\\n stderr_lines: <omitted>\\n stdout: e14444094323f41b6599c6ca17867266f18c642f5a498a7f1def96f37f731517\\n stdout_lines: <omitted>\\nWednesday 02 October 2019 16:13:40 -0400 (0:00:00.693) 0:00:20.474 ***** \\n\\nTASK [tripleo-modify-image : Cleanup working container] ************************\\nfatal: [localhost]: FAILED! => changed=true \\n cmd:\\n - buildah\\n - rm\\n | 20:38 |
| weshay | - rhel-binary-swift-proxy-server-working-container\\n delta: \\'0:00:07.917929\\'\\n end: \\'2019-10-02 16:13:48.658808\\'\\n msg: non-zero return code\\n rc: 1\\n start: \\'2019-10-02 16:13:40.740879\\'\\n stderr: \\'error removing container \"rhel-binary-swift-proxy-server-working-container\": error deleting build container \"d5a5b6b1a49f4c0df13d251d98d5332242c796f14521860355c2c9cb9c5daf9c\": remove /var/lib/containers/storage/overlay/ | 20:38 |
| weshay | 9cdd774a37289bdfa7e58dfba15dd357cfbf17f4e630ba5e45fe34bdbbe53a2a/merged: device or resource busy\\'\\n stderr_lines: <omitted>\\n stdout: \\'\\'\\n stdout_lines: <omitted>\\n\\nPLAY RECAP *********************************************************************\\nlocalhost : ok=15 changed=9 unreachable=0 failed=1 skipped=12 rescued=1 ignored=0 \\n\\nWednesday 02 October 2019 16:13:48 -0400 (0:00:08.121) | 20:38 |
| weshay | 0:00:28.596 ***** \\n=============================================================================== \\n'", | 20:39 |
| weshay | "Stderr: \" [WARNING]: provided hosts list is empty, only localhost is available. Note\\nthat the implicit localhost does not match 'all'\\n [WARNING]: Consider using the replace, lineinfile or template module rather\\nthan running 'sed'. If you need to use command because replace, lineinfile or\\ntemplate is insufficient you can add 'warn: false' to this command task or set\\n'command_warnings=False' in ansible.cfg to get rid of this | 20:39 |
| weshay | message.\\n\"", | 20:39 |
| weshay | "", | 20:39 |
| weshay | "tripleo_common.image.exception.ImageUploaderException: Modifying image 192.168.24.1:8787/tripleomaster/rhel-binary-swift-proxy-server:968c9526ff2eeafaa13207a791f61173db604fff_9e598812 failed" | 20:39 |
| weshay | hrm.. | 20:39 |
| rlandy | panda: ok if I rebase https://review.rdoproject.org/r/#/c/22654/? | 20:45 |
| rlandy | patches above that can't merge | 20:45 |
| arxcruz|ruck | weshay: periodic-tripleo-ci-rhel-8-scenario001-standalone-master http://logs.rdoproject.org/93/22793/3/check/periodic-tripleo-ci-rhel-8-scenario001-standalone-master/3c00791/ : SUCCESS in 1h 12m 12s | 20:47 |
| arxcruz|ruck | the standalone pass | 20:48 |
| weshay | ah nice | 20:51 |
| weshay | arxcruz|ruck, put up another patch to disable fs001 for rhel8 | 20:51 |
| weshay | promotion | 20:51 |
| arxcruz|ruck | ok | 20:51 |
| weshay | arxcruz|ruck, hrm.. is it even in there | 20:53 |
| weshay | ? | 20:53 |
| weshay | I see scen01-03 and the build jobs | 20:53 |
| weshay | ah ya.. it is | 20:53 |
| weshay | then meet me on the tmux | 20:53 |
| weshay | [u'periodic-tripleo-ci-rhel-8-ovb-3ctlr_1comp-featureset001-master'] | 20:53 |
| weshay | missing successful jobs: [u'periodic-tripleo-ci-rhel-8-ovb-3ctlr_1comp-featureset001-master'] | 20:54 |
| weshay | arxcruz|ruck, we should mark fs001 non-voting in 3rd party as well | 20:54 |
| arxcruz|ruck | weshay: https://review.rdoproject.org/r/22803 | 20:55 |
| *** Goneri has quit IRC | 21:02 | |
| *** aakarsh has quit IRC | 21:03 | |
| *** tosky_ has joined #oooq | 21:05 | |
| *** tosky has quit IRC | 21:06 | |
| *** tosky_ is now known as tosky | 21:08 | |
| weshay | arxcruz|ruck, python failures | 21:11 |
| weshay | rfolco|dentist, panda, rlandy promoter is broken | 21:12 |
| weshay | failed: [localhost] (item=[u'openstack-base', u'current-tripleo']) => {"ansible_loop_var": "item", "changed": false, "item": ["openstack-base", "current-tripleo"], "msg": "Failed to import the required Python library (Docker SDK for Python: docker (Python >= 2.7) or docker-py (Python 2.6)) on promoter's Python /home/centos/promoter_venv/bin/python. Please read module documentation and install in the appropriate location, for example via `pip | 21:12 |
| weshay | install docker` or `pip install docker-py` (Python 2.6). The error was: No module named docker"} | 21:12 |
| rlandy | weshay: ugh - since which merge? new error? | 21:12 |
| * rlandy looks at last merge | 21:13 | |
| rlandy | weshay: manual run or can you post full log link? | 21:17 |
| weshay | rlandy, http://promoter.rdoproject.org/redhat8_master.log | 21:19 |
| *** aakarsh has joined #oooq | 21:22 | |
| rlandy | https://github.com/rdo-infra/ci-config/commit/2ff9791109de5c1b7cf61e0dd6c32be70d1d6e7f | 21:24 |
| *** jfrancoa has quit IRC | 21:27 | |
| rlandy | https://github.com/rdo-infra/ci-config/blob/master/ci-scripts/infra-setup/roles/promoter/tasks/main.yml#L37 | 21:27 |
| weshay | rlandy, that's not the issue you think? | 21:29 |
| rlandy | weshay: I think the issue is that we remove docker | 21:30 |
| rlandy | Removed old/distro version of docker | 21:30 |
| weshay | rlandy, from the promoter? | 21:30 |
| weshay | rlandy, hrm.. talk to me | 21:30 |
| rlandy | weshay: let's google meet | 21:30 |
| rlandy | and I'll point to it | 21:30 |
| weshay | https://meet.google.com/xgb-ebtf-xdz | 21:30 |
| weshay | rlandy, https://review.rdoproject.org/r/22805 | 21:42 |
| *** holser has quit IRC | 21:56 | |
| rlandy | tox failure | 21:59 |
| rlandy | weshay: ^^ checking that | 22:00 |
| rlandy | http://logs.rdoproject.org/05/22805/1/check/rdo-tox-molecule/45a1b08/tox/reports.html | 22:00 |
| rlandy | trying an edit to that patch | 22:02 |
| *** rfolco|dentist is now known as rfolco | 22:17 | |
| weshay | rlandy, rfolco https://review.rdoproject.org/r/22806 | 22:21 |
| rfolco | looking | 22:21 |
| weshay | rfolco, you have any teeth left? https://review.rdoproject.org/r/#/c/22805/ | 22:22 |
| weshay | :) | 22:22 |
| rfolco | weshay, it really sucks, a family trip to visit a dentist and discover dental plan does not cover procedures and you have to leave an organ to pay for it | 22:23 |
| panda | I'd say we disable the jobs until we branch | 22:23 |
| panda | we are trying to fix the old code with tests that are meant for the new code | 22:24 |
| panda | tomorrow we branch and we limit the tests to master | 22:24 |
| rlandy | weshay: ansible may need to be in the env for tests | 22:26 |
| rlandy | I removed the check | 22:26 |
| rlandy | will see if it really breaks otherwise | 22:26 |
| weshay | rlandy, for test... there should be test-requirments | 22:27 |
| panda | rdo-tox-molecule is failing on things that are supposed to work with the new code | 22:28 |
| panda | the venv is created installing requirements.txt | 22:28 |
| panda | that's why molecule test is not working | 22:29 |
| weshay | rlandy, do you mind making a comment in the requirements file that ansible and docker are rpm installed | 22:30 |
| weshay | so we don't get mixed up in the futre :) | 22:31 |
| weshay | future | 22:31 |
| weshay | panda, hey.. since you are here... the docker.io registry login.. is that a real login w/ creds? | 22:31 |
| panda | weshay: wes, our repos are password protected | 22:32 |
| weshay | the test sorin added worked w/ rdo, failed w/ docker | 22:32 |
| weshay | in docker.io? | 22:32 |
| panda | weshay: yes, for the push | 22:33 |
| weshay | panda, ok.. so I'll abandon my patch.. | 22:33 |
| weshay | maybe it's a valid login failure | 22:33 |
| panda | weshay: any logs on how it failed ? on the promoter ? | 22:33 |
| weshay | https://review.rdoproject.org/r/#/c/22806/ | 22:33 |
| weshay | panda, sent an email | 22:33 |
| weshay | panda, http://promoter.rdoproject.org/redhat8_master.log | 22:34 |
| weshay | TASK [Validate docker login] *************************************************** | 22:34 |
| weshay | errors:", "denied: requested access to the resource is denied", "error parsing HTTP 401 response body: unexpected end of JSON input | 22:35 |
| weshay | worked w/ rdo though | 22:35 |
| weshay | "Successfully built 4e670c48589d", "Successfully tagged tripleomaster/nop:latest", "The push refers to repository [docker.io/tripleomaster/nop]" | 22:35 |
| weshay | maybe it's an error but the build / tag still worked | 22:35 |
| panda | "errors:\ndenied: requested access to the resource is denied\nerror parsing HTTP 401 response body: unexpected end of JSON input | 22:36 |
| weshay | saw this today https://www.reddit.com/r/funny/comments/dcee02/the_5_stages_of_software_development/?utm_medium=android_app&utm_source=share | 22:38 |
| rlandy | https://review.rdoproject.org/r/#/c/22805/ passes tox now | 22:39 |
| rlandy | removed the check for ansible | 22:39 |
| panda | I'm at step 3 | 22:39 |
| rlandy | ok - failing introspection - at least it's consistent | 22:40 |
| panda | rlandy: yep, and without ansible in venv the integration test is back to square 1 | 22:41 |
| weshay | the promoter is updating itself | 22:43 |
| weshay | I've been cherry-picking, but ansible was back in the python virt-env | 22:43 |
| panda | yes. We need the branch, otherwise the new code will never be merged, or the production server will continue to break | 22:44 |
| weshay | panda, fine by me | 22:44 |
| weshay | maybe I need to restart the service | 22:44 |
| panda | the ansible module needs to be removed manually from the venv | 22:54 |
| weshay | panda, I did | 22:54 |
| weshay | fyi.. sorin got us a real name http://promoter.rdoproject.org/ | 22:54 |
| panda | no facilitations, we suffer like real developers. | 22:55 |
| weshay | heh | 22:55 |
| weshay | stein is running atm http://promoter.rdoproject.org/centos7_stein.log | 22:55 |
| panda | the alternative to the branch is that before merging each patch, we run it manually on a freshly provisioned server, and see if we are breaking anything, if we are, fix the code and spin another round of manual tests. | 22:57 |
| weshay | heh | 23:03 |
| * weshay doesn't say a word | 23:03 | |
| *** tosky has quit IRC | 23:16 | |
| panda | we'll need to merge this before resuming the work on the promoter https://review.rdoproject.org/r/22807 either just before the branch or right after. At this point I's say after the branch | 23:30 |
| *** panda is now known as panda|off | 23:39 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!