| *** d0ugal has quit IRC | 00:07 | |
| *** d0ugal has joined #oooq | 00:23 | |
| *** ykarel|away has joined #oooq | 01:55 | |
| *** udesale has joined #oooq | 04:04 | |
| *** surpatil has joined #oooq | 04:18 | |
| *** surpatil has quit IRC | 04:20 | |
| *** surpatil has joined #oooq | 04:29 | |
| *** surpatil has joined #oooq | 04:29 | |
| *** ykarel|away has quit IRC | 04:36 | |
| *** jtomasek has quit IRC | 04:36 | |
| *** ykarel|away has joined #oooq | 04:53 | |
| *** jaosorior has joined #oooq | 05:03 | |
| *** ratailor has joined #oooq | 05:25 | |
| *** soniya29 has joined #oooq | 05:30 | |
| *** raukadah is now known as chandankumar | 05:33 | |
| *** marios has joined #oooq | 05:37 | |
| *** jfrancoa has joined #oooq | 05:51 | |
| *** soniya29 has quit IRC | 06:04 | |
| *** soniya29 has joined #oooq | 06:05 | |
| *** soniya29 has quit IRC | 06:12 | |
| *** soniya29 has joined #oooq | 06:14 | |
| *** jtomasek has joined #oooq | 06:16 | |
| *** soniya29 has quit IRC | 06:28 | |
| *** soniya29 has joined #oooq | 06:29 | |
| *** ykarel|away is now known as ykarel | 06:31 | |
| *** kopecmartin has joined #oooq | 07:05 | |
| *** jtomasek has quit IRC | 07:06 | |
| *** jtomasek has joined #oooq | 07:07 | |
| *** amoralej|off is now known as amoralej | 07:10 | |
| *** tesseract has joined #oooq | 07:17 | |
| marios | arxcruz|zzz: zbr o/ is the containers-multinode-rocky one known i didn't see something in https://etherpad.openstack.org/p/ruckroversprint16 but it looks to be real lots of fails at http://zuul.openstack.org/builds?job_name=tripleo-ci-centos-7-containers-multinode-rocky & blocking that https://review.opendev.org/#/c/684260/ (cc chandankumar ) | 07:25 |
|---|---|---|
| *** arxcruz|zzz is now known as arxcruz|ruck | 07:25 | |
| marios | https://2738dc1e3903897cb454-9e9b3039052bbd5ed4d3c1ac40f1e9bf.ssl.cf2.rackcdn.com/684260/4/check/tripleo-ci-centos-7-containers-multinode-rocky/80e6764/logs/undercloud/home/zuul/undercloud_install.log.txt.gz issue during image prepare? | 07:25 |
| arxcruz|ruck | marios: hey, no, i wasn't aware | 07:27 |
| arxcruz|ruck | let me check | 07:27 |
| *** jpena|off is now known as jpena | 07:29 | |
| marios | arxcruz|ruck: ack just came there from code review (its chandankumar patch) but the job looks like its been red for a while (and it votes) | 07:30 |
| *** bogdando has joined #oooq | 07:30 | |
| ykarel | marios, arxcruz|ruck need https://review.opendev.org/#/c/685466/ to fix rocky ^^ | 07:31 |
| marios | ykarel: thx :) | 07:32 |
| ykarel | arxcruz|ruck, are you aware of overcloud image build issue in ovb jobs in all releases? | 07:32 |
| ykarel | i see all failing while contacting cloud.centos.org | 07:32 |
| arxcruz|ruck | hey guys, I just started the monday, give me a sec :) | 07:33 |
| marios | arxcruz|ruck: happy monday ! | 07:33 |
| arxcruz|ruck | marios: thanks :) | 07:33 |
| arxcruz|ruck | marios: can you +w https://review.opendev.org/#/c/685466/ ? | 07:34 |
| arxcruz|ruck | or you don't have permission for this repo ? | 07:34 |
| marios | arxcruz|ruck: checking | 07:36 |
| ykarel | arxcruz|ruck, ack take ur time, sorry | 07:37 |
| ykarel | marios, can u also review https://review.opendev.org/#/c/685343/ backport | 07:39 |
| ykarel | also the depends on https://review.opendev.org/#/c/685368/ | 07:39 |
| *** tesseract has quit IRC | 07:43 | |
| *** tesseract has joined #oooq | 07:43 | |
| marios | ykarel: ack but https://review.opendev.org/#/c/685368/ already has votes | 07:44 |
| *** bogdando has quit IRC | 07:44 | |
| marios | ykarel: (but red job and you already recheck) | 07:44 |
| ykarel | marios, hmm i rechecked, those failures are not related to the patch though | 07:45 |
| marios | ykarel: ok but unles you make the job non voting not much i can do more then | 07:46 |
| marios | ykarel: even if i +A it wont merge | 07:46 |
| ykarel | marios, those job should pass this time, | 07:47 |
| ykarel | ok i will ping again one jobs are green for _W | 07:47 |
| ykarel | +W | 07:47 |
| marios | ykarel: ack ping me | 07:47 |
| arxcruz|ruck | ykarel: seems cloud.centos.org is down | 07:51 |
| arxcruz|ruck | at least the service at port 80 | 07:51 |
| arxcruz|ruck | i can ping though | 07:51 |
| arxcruz|ruck | who is responsible for centos? | 07:51 |
| arxcruz|ruck | nah, nevermind | 07:51 |
| ykarel | arxcruz|ruck, see #rdo, discussion going there | 07:51 |
| ykarel | arxcruz|ruck, but good to create bug | 07:52 |
| ykarel | and track there | 07:52 |
| *** soniya29 has quit IRC | 07:53 | |
| *** ykarel is now known as ykarel|lunch | 07:53 | |
| arxcruz|ruck | ykarel|lunch: creating | 07:53 |
| arxcruz|ruck | ykarel|lunch: https://bugs.launchpad.net/tripleo/+bug/1845918 | 07:54 |
| openstack | Launchpad bug 1845918 in tripleo "OVB jobs failing to connect to cloud.centos.org" [Critical,Triaged] | 07:54 |
| *** bogdando has joined #oooq | 08:00 | |
| *** dtantsur|afk is now known as dtantsur | 08:24 | |
| panda | I'm home | 08:32 |
| *** chem has joined #oooq | 08:32 | |
| *** derekh has joined #oooq | 08:38 | |
| *** ykarel|lunch is now known as ykarel | 08:43 | |
| panda | marios: anyting you want to tell me ? | 08:46 |
| marios | panda: yes. hello how are you | 08:46 |
| marios | panda: lets sync in a bit? i saw your mail. would you like to schedule a call ... you just got it... like 1 hour? | 08:46 |
| marios | s/got it/got in/ | 08:46 |
| panda | marios: ack | 08:47 |
| *** soniya29 has joined #oooq | 08:53 | |
| *** tosky has joined #oooq | 09:10 | |
| *** holser has joined #oooq | 09:21 | |
| *** zbr is now known as zbr|ruck | 09:24 | |
| zbr|ruck | morning! looking now... | 09:24 |
| zbr|ruck | marios: in fact i asked on friday about missing yum_update.sh but nobody answered | 09:25 |
| marios | zbr|ruck: 10:31 < ykarel> marios, arxcruz|ruck need https://review.opendev.org/#/c/685466/ to fix rocky ^^ | 09:31 |
| zbr|ruck | i seen it, if we are lucky in 3h is fixed. | 09:32 |
| arxcruz|ruck | yup | 09:32 |
| panda | we are never lucky | 09:56 |
| *** tosky has quit IRC | 10:17 | |
| *** tosky has joined #oooq | 10:18 | |
| *** soniya29 has quit IRC | 10:26 | |
| *** soniya29 has joined #oooq | 10:26 | |
| zbr|ruck | sadly panda was right on https://review.opendev.org/#/c/685466/ | 11:05 |
| zbr|ruck | i keep seeing There was an issue creating /root/DLRN as requested: [Errno 13] Permission denied: '/root/DLRN' | 11:06 |
| zbr|ruck | seen it last week at least twice, not sure where as logstash does not help. | 11:07 |
| zbr|ruck | arxcruz|ruck: ^ any idea about what could caused this? | 11:11 |
| chandankumar | marios: panda arxcruz|ruck kopecmartin I am taking sick leave, will see ya tomorrow | 11:11 |
| *** chandankumar is now known as raukadah | 11:11 | |
| panda | I lost count, what was I right about ? | 11:11 |
| panda | raukadah: get well, and if I see you chatting here today, I'll ban you :) | 11:12 |
| marios | raukadah: ack | 11:12 |
| marios | raukadah: hope you're better soon | 11:13 |
| *** udesale has quit IRC | 11:13 | |
| zbr|ruck | arxcruz|ruck: http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22There%20was%20an%20issue%20creating%20%2Froot%2FDLRN%20as%20requested%5C%22 | 11:19 |
| zbr|ruck | known or we should raise new one? | 11:19 |
| zbr|ruck | or is just a side effect from another failure? | 11:20 |
| zbr|ruck | panda: marios what user is supposed to be used by build-test-packages role? i am looking at tasks and lots of them are missing become | 11:25 |
| zbr|ruck | seems almost random | 11:25 |
| marios | zbr|ruck: not sure haven't looked there lately | 11:27 |
| zbr|ruck | marios: tx, trying to make a patch now. | 11:29 |
| zbr|ruck | what I do not understand is why they randomly fail | 11:30 |
| *** jpena is now known as jpena|lunch | 11:31 | |
| ykarel | zbr|ruck, how many times this is seen, atleast https://review.opendev.org/#/c/685466/ it was fallback of rerunning the deployment | 11:34 |
| ykarel | seems some network error in between, then zuul reran the script | 11:34 |
| ykarel | it reran after reaching tempest poing | 11:35 |
| ykarel | s/poing/point | 11:35 |
| ykarel | good to check if same symptom at other places where u noticed the same failure | 11:35 |
| zbr|ruck | ykarel: for the moment I raised https://bugs.launchpad.net/tripleo/+bug/1845942 | 11:36 |
| openstack | Launchpad bug 1845942 in tripleo "tqe: There was an issue creating /root/DLRN as requested" [High,Triaged] | 11:36 |
| ykarel | zbr|ruck, ack i see 5 failures in last 7 days, good to check if all have network issue + reran, or something else | 11:37 |
| ykarel | iirc that role run with non root | 11:38 |
| ykarel | and there was some mess related to gather_facts, which changes the user facts in between | 11:39 |
| ksambor | hey question, what I should do to use different ovn version in our ci jobs? Because this patch https://review.rdoproject.org/r/#/c/22609/ was merged 4 days ago and when I run reproducer containers with ovn still contains previous ovn | 11:39 |
| ykarel | ksambor, since this is a dependencies, it should be already available in repos | 11:40 |
| ykarel | may be some mirror were not updated, any link where u seeing issue | 11:41 |
| ykarel | the package is available in trunk deps repo:- https://trunk.rdoproject.org/centos7-master/deps/latest/x86_64/ovn-2.11.1-3.el7.x86_64.rpm | 11:41 |
| ykarel | ohh i think u are looking at containers, if yes then it needs promoted containers | 11:42 |
| ksambor | unfortunatly on ci this job is timeouting on images prepere so I have only local env | 11:42 |
| *** weshay has joined #oooq | 11:43 | |
| ykarel | ksambor, so on local env you are looking ovn version in containers, right? | 11:43 |
| ykarel | and you are using containers from docker.io, right? | 11:43 |
| ykarel | if both are yes, it needs promotion of containers with new packages | 11:44 |
| weshay | ykarel, which branch? | 11:44 |
| ykarel | weshay, master | 11:44 |
| weshay | ykarel, lp#? | 11:44 |
| ykarel | weshay, ksambor is looking why https://review.rdoproject.org/r/#/c/22609/ ovn update is not in containers yet | 11:45 |
| ykarel | so /me just answering, not sure if there is bug, or is really needed for this case | 11:46 |
| ykarel | and yes master promotion is blocked with other issue | 11:46 |
| ksambor | ykarel yeah I look inside containers and yes they are docker containers | 11:46 |
| weshay | we need a promotion on master because we bumped ovn? | 11:46 |
| ksambor | ovn is bumped because if we are want to use ovn with ssl previous version have issues with that (so this is needed only for this patch https://review.opendev.org/#/c/680345/) | 11:48 |
| ksambor | weshay: ^^ | 11:48 |
| ykarel | ksambor, ack, so containers being build for promotion already have ovn version u looking for:- https://logs.rdoproject.org/openstack-periodic-master/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-centos-7-master-containers-build-push/5be927d/logs/build.log.txt.gz#_2019-09-30_00_30_05 | 11:48 |
| ykarel | so once master have a promotion, you will have new containers(with new ovn) at docker.io | 11:49 |
| ksambor | cool thanks! | 11:49 |
| ykarel | and if you are just looking for testing it, you can adjust update_repo in container-prepare-parameter to include master-testing repo | 11:49 |
| ksambor | ykarel, weshay: thank you ! | 11:50 |
| *** amoralej is now known as amoralej|lunch | 11:50 | |
| weshay | zbr|ruck, arxcruz|ruck morning guys | 11:54 |
| panda | marios: fixed the staging script to have the correct hierarchy, now focusing on the split | 11:55 |
| panda | marios: also addressing your comments | 11:55 |
| tosky | why oh why is rocky consistently failing in https://review.opendev.org/#/c/683932/ ? | 11:57 |
| arxcruz|ruck | tosky: we are working on it | 11:59 |
| arxcruz|ruck | weshay: hi boss, need 10 min | 11:59 |
| marios | panda: ack nothing major there was just reading through and didn't even get to the docker/socket stuff | 12:01 |
| arxcruz|ruck | weshay: hi boss | 12:05 |
| weshay | zbr|ruck, arxcruz|ruck https://meet.google.com/rzp-osav-fgz | 12:06 |
| arxcruz|ruck | weshay: ok | 12:06 |
| weshay | ykarel, turning off dns, no luck https://review.opendev.org/#/c/685639/ we see it in current-tripleo as well | 12:07 |
| ykarel | weshay, didn't get, see it in current-tripleo as well | 12:09 |
| ykarel | any log/reference? | 12:09 |
| ykarel | since it's network related, it should be affected at all places, all releases check/promotion | 12:10 |
| ykarel | so, wherever image get's build and trying access to cloud.centos.org | 12:11 |
| weshay | ykarel, ya.. saw that all weekend.. re: cloud.centos.org | 12:13 |
| ykarel | weshay, yes and also it's randomly hitting | 12:13 |
| ykarel | there are some jobs which passed | 12:13 |
| ykarel | so what i noticed whenever it hit server 38.110.33.4, it fails, while with other server it passes | 12:14 |
| ykarel | but currently it's working locally atleast | 12:14 |
| ykarel | so running more jobs with https://review.rdoproject.org/r/#/c/20171/ to see if it's still hitting, if issue is still there we need to check with rdo operators | 12:15 |
| ykarel | if there is some network issue in RDO Cloud | 12:16 |
| ykarel | also fyi i discussed the issue in #centos-devel, and as per them there is no issue on cloud.centos.org | 12:16 |
| * weshay gets on my test system.. chatting w/ arx atm | 12:17 | |
| weshay | ykarel, we're going to move the cloud image to the rdo image server | 12:19 |
| ykarel | weshay, ack, may be it's already there good to check that too | 12:19 |
| ykarel | but possible latest one is not there | 12:20 |
| weshay | ykarel, arxcruz|ruck images will be moved here http://images.rdoproject.org/base/centos7/ | 12:29 |
| ykarel | weshay, ack | 12:29 |
| *** jpena|lunch is now known as jpena | 12:30 | |
| weshay | zbr|ruck, need you responding to my pings this week | 12:31 |
| *** rfolco has joined #oooq | 12:33 | |
| zbr|ruck | weshay: i back | 12:39 |
| zbr|ruck | (lunch) | 12:39 |
| weshay | zbr|ruck, k.. arxcruz|ruck is getting totally slammed atm.. we need your man power! | 12:39 |
| weshay | zbr|ruck, we're on an escalation call atm, out in 25min and lets sync up and see how we can help him | 12:40 |
| *** Goneri has joined #oooq | 12:44 | |
| *** amoralej|lunch is now known as amoralej | 12:44 | |
| ykarel | arxcruz|ruck, weshay, fyi the three ovb jobs i ran with https://review.rdoproject.org/r/#/c/20171/ to check if issue improved currently, it passed the build-images role | 12:50 |
| weshay | ykarel, k.. | 12:56 |
| *** aakarsh has joined #oooq | 12:56 | |
| *** dtantsur is now known as dtantsur|afk | 12:58 | |
| jfrancoa | zbr|ruck: hey, a heads up for a new issue with the undercloud upgrades job https://bugs.launchpad.net/tripleo/+bug/1845935 | 13:02 |
| openstack | Launchpad bug 1845935 in tripleo "Dying containers during undercloud-upgrade" [High,New] | 13:02 |
| jfrancoa | zbr|ruck: I've been checking it the whole morning but I can't find out what's happening, neither narrow down when did it start happening..it was on Friday 27 for sure..but I can't find the patch that caused it | 13:03 |
| *** ratailor has quit IRC | 13:10 | |
| weshay | arxcruz|ruck, zbr|ruck https://meet.google.com/cwy-duhx-xum?authuser=1 | 13:10 |
| arxcruz|ruck | weshay: so, back to hangouts ? | 13:10 |
| weshay | arxcruz|ruck, zbr|ruck https://review.rdoproject.org/r/#/c/22609/ | 13:16 |
| weshay | zbr|ruck, http://logs.rdoproject.org/openstack-periodic-master/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-7-standalone-full-tempest-master/6742c15/logs/stestr_results.html | 13:19 |
| zbr|ruck | weshay: managed to create bug https://bugs.launchpad.net/tripleo/+bug/1845956 | 13:28 |
| openstack | Launchpad bug 1845956 in tripleo "neutron_tempest_plugin.scenario.test_trunk.TrunkTest failing on master" [Undecided,Confirmed] | 13:28 |
| weshay | zbr|ruck, k.. cool.. skip list it for master :) | 13:28 |
| zbr|ruck | weshay: do I need to add it to trello or this is automatic? | 13:29 |
| *** soniya29 has quit IRC | 13:31 | |
| arxcruz|ruck | weshay: https://review.opendev.org/#/c/685709/ | 13:34 |
| weshay | arxcruz|ruck, keep going . .need it through queens | 13:35 |
| zbr|ruck | weshay: what reason do i need to put? | 13:35 |
| weshay | zbr|ruck, check the failure :) | 13:35 |
| zbr|ruck | not sure, but I would paste Failed to establish authenticated ssh connection to cirros | 13:36 |
| arxcruz|ruck | weshay: queens doesn't have the settings, shall I add it? | 13:37 |
| *** chem has quit IRC | 13:37 | |
| *** chem has joined #oooq | 13:37 | |
| weshay | arxcruz|ruck, https://review.opendev.org/#/c/685346/ | 13:38 |
| arxcruz|ruck | weshay: it's not merged yet, i update my patch | 13:39 |
| weshay | ya please.. thanks | 13:41 |
| weshay | marios, let me change the invite to a google meet 2 seconds | 13:42 |
| weshay | marios, https://meet.google.com/fag-ppbp-beo?authuser=1 | 13:43 |
| marios | weshay: k | 13:43 |
| rfolco | panda, the reviews on your taiga tasks are all merged, what is the top patch that I need to rebase ? and the other 2 splits ? | 13:46 |
| panda | rfolco: https://review.rdoproject.org/r/#/c/22656/ | 13:51 |
| panda | rfolco: the other two are https://review.rdoproject.org/r/#/c/22655 and https://review.rdoproject.org/r/#/c/22654 | 13:51 |
| rfolco | panda, thanks | 13:52 |
| *** redrobot has joined #oooq | 14:00 | |
| arxcruz|ruck | ykarel: didn't get your comments on https://review.opendev.org/#/c/685709 | 14:05 |
| * ykarel looks | 14:06 | |
| rfolco | panda, # The call in production is currently done by a root crontab installed a | 14:06 |
| rfolco | # intance creation time by cloud-init. | 14:06 |
| arxcruz|ruck | ykarel: we just want to move the promotion so far, according weshay | 14:07 |
| rfolco | panda, crontab? cloud-init? dlrn-promoter is a service script. What you talking about there?? | 14:07 |
| ykarel | arxcruz|ruck, ok with just promotion part, my main point was baseos_undercloud_image_url var is not used at the place we are seeing issue | 14:08 |
| ykarel | that cloud.centos.org image url is set in diskimage-builder element | 14:09 |
| ykarel | for which i shared link in the comment | 14:09 |
| panda | rfolco: the provisioning part | 14:10 |
| panda | rfolco: there's a whole part that we are not seeing here | 14:10 |
| panda | rfolco: in infra-setup itself | 14:10 |
| rfolco | panda, ok still needs to be root | 14:11 |
| panda | rfolco: wanna sync on this stuff ? | 14:15 |
| rfolco | panda, your job will be tougher at this moment to explain me changes... but if you don't mind, I accept | 14:16 |
| rfolco | panda, I have one thing to say about base and openstack-base... these 2 are not pushed to registry, these are intermediate containers. So we might need to change these for ppc pushed containers like you did with nova-compute and neutron-server for x86 | 14:18 |
| panda | rfolco: doesn't matter, those two are for testing, but they are included by default in the current container-push playbook, so it looks for them, even if they are not pushed. I've changed already too much on container-push, I don't wwant to touch the core logic right now. | 14:20 |
| panda | rfolco: in general it's one thing that we are avoiding as much as possible, changing the core logic | 14:21 |
| *** Vorrtex has joined #oooq | 14:22 | |
| rfolco | panda, ok | 14:22 |
| arxcruz|ruck | weshay: check ykarel comments | 14:22 |
| * weshay looking at it | 14:23 | |
| weshay | arxcruz|ruck, we can't do anything w/ https://opendev.org/openstack/diskimage-builder/src/branch/master/diskimage_builder/elements/centos7/root.d/10-centos7-cloud-image#L14 | 14:24 |
| rfolco | panda, 54 fails delegated, 55 passes, expected to merge 54 w/ failing state ? | 14:24 |
| weshay | we could override it in our jobs I guess | 14:24 |
| arxcruz|ruck | weshay: yup, but in which level ? | 14:25 |
| weshay | arxcruz|ruck, /me gets | 14:25 |
| panda | rfolco: 54 fails rdo-tox-molecule, I'm gong to fix it, because it's a voting job, but i completely change the code that is failing in 55 | 14:27 |
| panda | so it's just to please the gate jobs ... | 14:27 |
| panda | rfolco: if the explanation of the infra-setup credentials part is useful for your task then I'll explain now, if not we can chat later | 14:28 |
| arxcruz|ruck | weshay: https://opendev.org/openstack/tripleo-quickstart-extras/src/branch/master/roles/build-images/templates/overcloud-image-build.sh.j2 | 14:28 |
| panda | maybe marios will have the same quekstion, and I'll explain once how all that part is working | 14:28 |
| weshay | arxcruz|ruck, yup.. there you go | 14:29 |
| rfolco | panda, I think it's easier to start a new patch and just use my old patch as reference... instead of fixing rebases... my patch isn't functional anyway | 14:30 |
| rfolco | panda, give me a green light to review all your 3 when they are ready | 14:30 |
| panda | rfolco: green light. fixing rdo-tox-molecule will not really change anything there | 14:32 |
| rfolco | panda, I burned start | 14:33 |
| arxcruz|ruck | ykarel: so, we can set base_image_url and base_image_path that will be set in https://opendev.org/openstack/tripleo-quickstart-extras/src/branch/master/roles/build-images/templates/overcloud-image-build.sh.j2#L28 | 14:34 |
| rfolco | panda, actually this expression makes no sense... should be jumped the gun I think | 14:34 |
| ykarel | arxcruz|ruck, looking | 14:38 |
| ykarel | arxcruz|ruck, yes looks like that's the right place | 14:40 |
| ykarel | good to attempt | 14:40 |
| zbr|ruck | weshay: question re strategy to ease merges, the yum_update.sh just went (again) into gate few minutes ago but we have other important changes that failed to merge due to it.should I update them to add depends-on? -- this would lose their wf, example https://review.opendev.org/#/c/685466/ | 14:47 |
| *** surpatil has quit IRC | 14:48 | |
| arxcruz|ruck | ykarel: https://review.opendev.org/#/c/685709 take a look now | 14:58 |
| arxcruz|ruck | have a feeling we will promote master :) | 15:01 |
| arxcruz|ruck | the pipeline looks good so far | 15:01 |
| arxcruz|ruck | brb | 15:01 |
| weshay | zbr|ruck, arxcruz|ruck back.. sorry.. mtg.. | 15:04 |
| weshay | zbr|ruck, which patches are you referring to? /me not in context | 15:05 |
| zbr|ruck | https://review.opendev.org/#/c/685466/ -- this is critical to merge | 15:05 |
| weshay | zbr|ruck, do you have an example of a job where yum_update.sh is causing a failure? | 15:07 |
| * weshay looking | 15:07 | |
| weshay | agree it should merge, but also I never found where exactly that is killing us | 15:07 |
| weshay | not that I looked much | 15:07 |
| zbr|ruck | https://review.opendev.org/#/c/683350/ -- | 15:08 |
| weshay | thanks | 15:08 |
| zbr|ruck | tripleo-ci-centos-7-containers-multinode-rocky failed due to this | 15:08 |
| ykarel | arxcruz|ruck, ack looking | 15:08 |
| weshay | zbr|ruck, ya.. that is rocky only issue.. but yes needs to merge asap | 15:12 |
| weshay | nice find by ykarel | 15:12 |
| zbr|ruck | weshay: rocky fails but this bug prevents most of other fixes from merging. we already lost 6h+ hours due to another random failure. i am keeping a eye on it to recheck the others as soon it merges. | 15:14 |
| ykarel | arxcruz|ruck, looks good, and commented on https://review.rdoproject.org/r/#/c/22663/ as it's not testing the upstream patch | 15:15 |
| weshay | zbr|ruck, k.. the other option is to ask in openstack-infra to send it to the top of our gate queue | 15:15 |
| weshay | not to force merge, just prioritize | 15:16 |
| zbr|ruck | weshay: maybe we need to activate prio label on our repos, few projects already have them. | 15:16 |
| ykarel | arxcruz|ruck, also not sure if qcow2 will work there, as earlier compressed file .xz was used | 15:19 |
| ykarel | but yes CI will tell that if it will not work, or else need to check dib code for it | 15:19 |
| ykarel | mainly https://opendev.org/openstack/diskimage-builder/src/branch/master/diskimage_builder/elements/redhat-common/bin/extract-image script is used there | 15:19 |
| *** ykarel is now known as ykarel|afk | 15:20 | |
| weshay | thanks ykarel|afk | 15:20 |
| arxcruz|ruck | ykarel|afk: done | 15:21 |
| arxcruz|ruck | brb, going to german class, be back in a few hours | 15:21 |
| *** dsneddon has joined #oooq | 15:21 | |
| marios | panda: rfolco i am out tomorrow is a public holiday here | 15:24 |
| marios | weshay: (as mentioned earlier ^^^) | 15:24 |
| marios | panda: rfolco sorry should have mentioned in our call earlier but i think the work split means i am not blocking folks | 15:24 |
| panda | marios: you cant be on PTO | 15:25 |
| panda | marios: I protest. | 15:25 |
| marios | panda: i will not be :D | 15:25 |
| marios | panda: i am paying for it | 15:26 |
| panda | marios: I'll make you pay when you come back | 15:26 |
| * panda shakes fists | 15:26 | |
| marios | panda: thanks i will spend my whole day tomorrow thinking about that and looking forward to wednesday morning | 15:27 |
| panda | marios: not wednesday morning, I'll be on PTO | 15:27 |
| marios | :D | 15:27 |
| marios | its all good panda rfolco will finish it all today anyway | 15:28 |
| panda | rfolco: thanks folco! | 15:28 |
| marios | what a guy right panda ? | 15:28 |
| marios | go folco | 15:28 |
| marios | woo | 15:28 |
| panda | folcgo | 15:29 |
| weshay | marios, can you meet on tuesday at 3pm utc.. you have a meeting there | 15:29 |
| weshay | re: scenarios for rhel | 15:29 |
| marios | weshay: yes sure | 15:29 |
| marios | weshay: like half hour ago but tomorrow :D | 15:30 |
| marios | weshay: did you send an invite /me doesn't see it | 15:30 |
| weshay | marios, tues oct 1 3pm utc | 15:32 |
| marios | weshay: ack i see the invite just came | 15:32 |
| weshay | thanks | 15:35 |
| *** ykarel|afk is now known as ykarel|away | 15:44 | |
| *** chem has quit IRC | 15:45 | |
| *** chem has joined #oooq | 15:45 | |
| weshay | marios, panda zbr|ruck http://38.145.34.55/centos7_master.log have an error here | 15:55 |
| weshay | ImportError: No module named markupsafe | 15:55 |
| *** tesseract has quit IRC | 15:55 | |
| zbr|ruck | weshay: i am on it... | 15:55 |
| marios | zbr|ruck: weshay me leaving in few need something? | 15:59 |
| weshay | marios, nope thanks | 15:59 |
| zbr|ruck | added https://review.rdoproject.org/r/#/c/22665/ for promoter | 16:07 |
| *** bogdando has quit IRC | 16:08 | |
| *** marios has quit IRC | 16:14 | |
| zbr|ruck | arxcruz|ruck: marios: should I attempt to manually promote from promoter or just wait for it to run again? | 16:19 |
| weshay | zbr|ruck, just hold.. don't manually promote | 16:27 |
| zbr|ruck | panda: is tripleo-ci-promotion-staging still supposed to fail? | 16:29 |
| panda | zbr|ruck yes, it's not ready yet | 16:30 |
| zbr|ruck | ok. just wanted to known if is safe to ignore. | 16:30 |
| panda | zbr|ruck: we'll remove the non=voting when is. | 16:30 |
| panda | I'll never ever split anything ever again in my life | 16:43 |
| panda | half a day spent on code errors that i know I already solved in other patches | 16:43 |
| *** jpena is now known as jpena|off | 16:45 | |
| *** jaosorior has quit IRC | 16:46 | |
| *** chem has quit IRC | 16:49 | |
| *** zbr|ruck is now known as zbr | 16:52 | |
| *** derekh has quit IRC | 17:00 | |
| *** ykarel|away has quit IRC | 17:06 | |
| weshay | rfolco, the promoter appears to be broken | 17:10 |
| rfolco | :( | 17:11 |
| weshay | http://38.145.34.55/centos7_master.log | 17:11 |
| *** amoralej is now known as amoralej|off | 17:11 | |
| weshay | this just merged.. but was fixing a diff issue https://review.rdoproject.org/r/#/c/22665/ | 17:11 |
| weshay | zbr, ^ | 17:11 |
| weshay | rfolco, I think you are the only on at this point that can fix it | 17:12 |
| rfolco | weshay, do the patch did not fix it ? | 17:12 |
| rfolco | so.. | 17:12 |
| weshay | rfolco, looks like it's missing docker now | 17:13 |
| weshay | "ansible_loop_var": "item", "changed": false, "item": ["openstack-base", "current-tripleo"], "msg": "Failed to import docker or docker-py (Docker SDK for Python) - No module named requests.exceptions. Try `pip install docker` or `pip install docker-py` (Python 2.6)."} | 17:14 |
| rfolco | weshay, will log into promoter to check venv | 17:15 |
| weshay | rfolco, adding ansible to the python virtenv probably changes where the other requirements are pulled from | 17:16 |
| weshay | https://github.com/rdo-infra/ci-config/commit/56d86e1ed1076e2565d0dad0f96f9d529cc7b05f#diff-db3078c05dda6f0e6fbc3a633f2f379c | 17:17 |
| rfolco | isn't panda around ? | 17:20 |
| weshay | panda, ? | 17:21 |
| weshay | you still around? | 17:22 |
| weshay | rfolco, docker is on the box | 17:23 |
| rfolco | yes docker-ce | 17:23 |
| rfolco | but then docker-py is required for delegated molecules tests | 17:23 |
| rfolco | and probably that f***ed the production workflow | 17:23 |
| weshay | only avail to root | 17:24 |
| *** ykarel|away has joined #oooq | 17:24 | |
| weshay | rfolco, let's chat | 17:25 |
| rfolco | k | 17:26 |
| weshay | rfolco, 3min | 17:26 |
| panda | ... | 17:26 |
| rfolco | panda, please look at promoter logs | 17:26 |
| rfolco | TASK [Pull the images from rdoproject registry] ******************************** | 17:27 |
| rfolco | failed: [localhost] (item=aodh-api) => {"ansible_loop_var": "item", "changed": false, "item": "aodh-api", "msg": "Failed to import docker or docker-py (Docker SDK for Python) - No module named requests.exceptions. Try `pip install docker` or `pip install docker-py` (Python 2.6)."} | 17:27 |
| weshay | panda, https://meet.google.com/nmy-dvix-edn?authuser=1 rfolco | 17:28 |
| weshay | panda, join the call please | 17:30 |
| *** holser has quit IRC | 18:17 | |
| *** holser has joined #oooq | 18:23 | |
| panda | weshay: fails with authetication error (?) | 18:31 |
| weshay | panda, let's roll back | 18:35 |
| panda | weshay: wait | 18:35 |
| weshay | rfolco, panda I have a holiday lunch I need to go to shortly | 18:35 |
| weshay | panda, k.. waiting | 18:35 |
| weshay | panda, can we open a proper tmux session on the promoter | 18:36 |
| weshay | I'll have to close my laptop soonish | 18:36 |
| panda | weshay: ok | 18:39 |
| panda | weshay: tmux atta if you want | 18:39 |
| panda | weshay: but I think we got a bogus token before | 18:39 |
| weshay | k.. /me gets on | 18:40 |
| panda | weshay: maybe it was not our fault | 18:40 |
| weshay | panda++ | 18:40 |
| weshay | panda, /me on centos user tmux | 18:40 |
| weshay | panda, which user? | 18:41 |
| panda | weshay: I was with root | 18:42 |
| panda | weshay: another attempt running ... | 18:43 |
| panda | weshay: not much to see now, but I was looking at the .docker/config.json to look at the token, and it was a small 4 character token | 18:43 |
| *** jtomasek has quit IRC | 18:43 | |
| weshay | k | 18:43 |
| panda | weshay: now we have a proper token | 18:44 |
| weshay | ah cool | 18:44 |
| weshay | is it pull from rdo or pushing to docker.io ? | 18:44 |
| * weshay looks | 18:44 | |
| weshay | pulling | 18:44 |
| panda | weshay: it was failing to push to rdo-registry | 18:45 |
| panda | weshay: we push the common-tripleo tag there too (I forgot why) | 18:45 |
| rfolco | current | 18:45 |
| panda | rfolco: yeah that one | 18:47 |
| panda | now it's pulling | 18:47 |
| *** jtomasek has joined #oooq | 18:48 | |
| panda | weshay: you can go, I'll keep an eye on things, then reactivate the loop over releases | 18:50 |
| *** holser has quit IRC | 18:50 | |
| weshay | panda++ | 18:50 |
| weshay | thanks | 18:50 |
| *** jtomasek has quit IRC | 18:50 | |
| *** jtomasek has joined #oooq | 18:51 | |
| *** jtomasek has quit IRC | 18:52 | |
| *** holser has joined #oooq | 18:53 | |
| *** ykarel|away has quit IRC | 18:54 | |
| *** weshay has quit IRC | 18:59 | |
| *** holser has quit IRC | 19:07 | |
| panda | rfolco: I think rdoregistry is having issues | 19:10 |
| panda | [WARNING]: The value of the "source" option was determined to be "pull". Please set the "source" option explicitly. Autodetection will be removed in Ansible 2.12. | 19:10 |
| panda | failed: [localhost] (item=glance-api) => {"ansible_loop_var": "item", "changed": false, "item": "glance-api", "msg": "Error pulling image trunk.registry.rdoproject.org/tripleomaster/centos-binary-glance-api:91964b910a29b9793d9298a6d1b236341ba90e9e_638b9ca3 - 500 Server Error: Internal Server Error (\"{\"message\":\"error parsing HTTP 400 response body: unexpected end of JSON input | 19:10 |
| panda | : \\\"\\\"\"}\")"} | 19:10 |
| *** holser has joined #oooq | 19:11 | |
| *** holser has quit IRC | 19:12 | |
| *** jbadiapa has joined #oooq | 19:17 | |
| *** ccamacho has joined #oooq | 19:22 | |
| *** kopecmartin is now known as kopecmartin|off | 19:47 | |
| *** holser has joined #oooq | 19:49 | |
| *** jbadiapa has quit IRC | 20:27 | |
| *** aakarsh has quit IRC | 20:35 | |
| *** jfrancoa has quit IRC | 20:46 | |
| *** jbadiapa has joined #oooq | 20:46 | |
| *** irclogbot_0 has quit IRC | 20:55 | |
| *** irclogbot_2 has joined #oooq | 20:55 | |
| *** ccamacho has quit IRC | 20:57 | |
| *** irclogbot_2 has quit IRC | 21:01 | |
| *** irclogbot_2 has joined #oooq | 21:01 | |
| *** Goneri has quit IRC | 21:01 | |
| *** irclogbot_2 has quit IRC | 21:07 | |
| *** irclogbot_3 has joined #oooq | 21:07 | |
| *** irclogbot_3 has quit IRC | 21:13 | |
| *** irclogbot_1 has joined #oooq | 21:13 | |
| *** irclogbot_1 has quit IRC | 21:19 | |
| *** irclogbot_1 has joined #oooq | 21:19 | |
| *** irclogbot_1 has quit IRC | 21:25 | |
| *** irclogbot_0 has joined #oooq | 21:25 | |
| *** irclogbot_0 has quit IRC | 21:33 | |
| *** irclogbot_0 has joined #oooq | 21:35 | |
| *** irclogbot_0 has quit IRC | 21:41 | |
| *** irclogbot_3 has joined #oooq | 21:41 | |
| *** irclogbot_3 has quit IRC | 21:46 | |
| *** irclogbot_1 has joined #oooq | 21:47 | |
| *** irclogbot_1 has quit IRC | 21:53 | |
| *** irclogbot_0 has joined #oooq | 21:53 | |
| *** holser has quit IRC | 21:58 | |
| *** irclogbot_0 has quit IRC | 21:59 | |
| *** irclogbot_1 has joined #oooq | 21:59 | |
| *** irclogbot_1 has quit IRC | 22:05 | |
| *** irclogbot_2 has joined #oooq | 22:05 | |
| *** irclogbot_2 has quit IRC | 22:15 | |
| *** irclogbot_0 has joined #oooq | 22:15 | |
| *** tosky has quit IRC | 22:24 | |
| *** jbadiapa has quit IRC | 22:53 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!