| *** sanjayu_ has joined #oooq | 01:55 | |
| *** udesale has joined #oooq | 03:11 | |
| *** sanjayu_ has quit IRC | 03:17 | |
| *** ykarel has joined #oooq | 04:32 | |
| *** udesale_ has joined #oooq | 04:37 | |
| *** udesale has quit IRC | 04:39 | |
| *** marios has joined #oooq | 05:08 | |
| *** ratailor has joined #oooq | 05:13 | |
| *** udesale__ has joined #oooq | 05:45 | |
| *** udesale_ has quit IRC | 05:48 | |
| *** jfrancoa has joined #oooq | 05:53 | |
| *** ysandeep|away is now known as ysandeep | 06:10 | |
| marios | zbr: sorry about the calendar invites | 06:18 |
|---|---|---|
| marios | zbr: google calendar was weird is all i can say | 06:18 |
| marios | zbr: man it was worse than i thought | 06:19 |
| marios | sorry folks | 06:19 |
| marios | :/ | 06:19 |
| marios | zbr: there are so many of them ! | 06:19 |
| marios | zbr: maybe i should delete the event? | 06:19 |
| marios | zbr: DOS by google calendar | 06:21 |
| marios | :( | 06:21 |
| marios | zbr: akahat: bhagyashri|rover: pojadhav: sorry about that folks :/ ^^ | 06:22 |
| bhagyashri|rover | marios, it's ok np | 06:24 |
| *** sanjayu_ has joined #oooq | 06:27 | |
| pojadhav | marios, np :) | 06:29 |
| *** jtomasek has joined #oooq | 06:58 | |
| *** jpena|off is now known as jpena | 07:22 | |
| *** bogdando has joined #oooq | 07:33 | |
| *** jtomasek has quit IRC | 07:39 | |
| *** jtomasek has joined #oooq | 07:41 | |
| *** ysandeep is now known as ysandeep|lunch | 07:46 | |
| *** apetrich has joined #oooq | 08:09 | |
| *** apetrich has quit IRC | 08:11 | |
| *** apetrich has joined #oooq | 08:12 | |
| *** chem has quit IRC | 08:19 | |
| *** apetrich has quit IRC | 08:20 | |
| *** chem has joined #oooq | 08:21 | |
| *** apetrich has joined #oooq | 08:24 | |
| *** apetrich has quit IRC | 08:25 | |
| *** tosky has joined #oooq | 08:27 | |
| *** apetrich has joined #oooq | 08:27 | |
| *** derekh has joined #oooq | 08:42 | |
| *** saneax has joined #oooq | 08:45 | |
| *** sanjayu_ has quit IRC | 08:46 | |
| *** arxcruz has joined #oooq | 09:07 | |
| *** ChanServ sets mode: +o arxcruz | 09:07 | |
| *** ysandeep|lunch is now known as ysandeep | 09:10 | |
| *** sshnaidm|pto is now known as sshnaidm | 09:10 | |
| *** ratailor has quit IRC | 09:19 | |
| *** ratailor has joined #oooq | 09:20 | |
| *** ratailor has quit IRC | 09:21 | |
| *** ratailor has joined #oooq | 09:25 | |
| *** ratailor has quit IRC | 09:34 | |
| *** ratailor has joined #oooq | 09:41 | |
| *** ratailor has quit IRC | 09:42 | |
| *** udesale_ has joined #oooq | 09:45 | |
| *** holser has joined #oooq | 09:47 | |
| *** udesale__ has quit IRC | 09:47 | |
| *** ratailor has joined #oooq | 09:48 | |
| *** ratailor has quit IRC | 09:49 | |
| *** ratailor has joined #oooq | 09:51 | |
| chandankumar | sshnaidm: Hello & welcome back :-) | 09:52 |
| chandankumar | sshnaidm: please have a look at this review https://review.opendev.org/#/c/749645/ | 09:53 |
| *** ratailor has quit IRC | 09:54 | |
| *** ratailor has joined #oooq | 09:54 | |
| *** ratailor has quit IRC | 09:55 | |
| sshnaidm | chandankumar, hi, thanks :) can we chat for a min? | 09:59 |
| chandankumar | sshnaidm: yes sure | 09:59 |
| sshnaidm | https://meet.google.com/_meet/rew-rnqe-qgm | 10:00 |
| *** ratailor has joined #oooq | 10:01 | |
| chandankumar | sshnaidm: https://review.opendev.org/#/c/749761/ | 10:01 |
| chandankumar | sshnaidm: https://30eb5777dd42ff5a076a-a1e6c8ac102e28467a2eeedfac6c37ed.ssl.cf1.rackcdn.com/749761/5/check/tripleo-build-containers-centos-7-buildah-train/0e347b5/job-output.txt | 10:02 |
| chandankumar | sshnaidm: build-containers : Tag for provider registry and push image: 127.0.0.1:8787/tripleotrain/centos-binary-placement-api] | 10:02 |
| *** ratailor has quit IRC | 10:02 | |
| *** ratailor has joined #oooq | 10:04 | |
| *** ratailor has quit IRC | 10:07 | |
| *** ratailor has joined #oooq | 10:12 | |
| *** ratailor has quit IRC | 10:12 | |
| *** udesale__ has joined #oooq | 10:13 | |
| *** udesale_ has quit IRC | 10:15 | |
| ysandeep | chandankumar, arxcruz when you have time, Could you please me some pointers about these tempest failures - https://sf.hosted.upshift.rdu2.redhat.com/logs/openstack-component-cinder/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-rhel-8-scenario001-standalone-cinder-rhos-16.2/c4185f8/logs/undercloud/var/log/tempest/stestr_results.html | 10:18 |
| ysandeep | https://sf.hosted.upshift.rdu2.redhat.com/logs/openstack-component-cloudops/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-rhel-8-scenario002-standalone-cloudops-rhos-16.2/794d10a/logs/undercloud/var/log/tempest/stestr_results.html | 10:18 |
| ysandeep | give* me some pointers | 10:18 |
| arxcruz | ysandeep: at first glance I would say something is wrong with keystone or the credentials you're using to run tempest | 10:19 |
| arxcruz | for the first log | 10:19 |
| arxcruz | the second | 10:19 |
| arxcruz | urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='192.168.24.1', port=9696): Max retries exceeded with url: /v2.0/routers/b9d37c82-20c8-4154-bbb4-9ee64b78da8e/remove_router_interface (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f2886652ac8>: Failed to establish a new connection: [Errno 111] Connection refused',)) | 10:19 |
| arxcruz | seems to be firewall issue if the url is correct | 10:19 |
| arxcruz | or the service isn't running at all | 10:19 |
| ysandeep | arxcruz, thanks! | 10:23 |
| *** ratailor has joined #oooq | 10:25 | |
| *** ratailor has quit IRC | 10:26 | |
| *** dtantsur|afk is now known as dtantsur | 10:27 | |
| *** ratailor has joined #oooq | 10:27 | |
| *** ratailor has quit IRC | 10:28 | |
| *** saneax has quit IRC | 10:34 | |
| *** saneax has joined #oooq | 10:36 | |
| *** matbu has quit IRC | 10:38 | |
| *** matbu has joined #oooq | 10:39 | |
| *** jbadiapa has joined #oooq | 11:16 | |
| chandankumar | kopecmartin: https://review.opendev.org/751685 and https://review.opendev.org/751760 | 11:20 |
| *** ratailor has joined #oooq | 11:22 | |
| *** ratailor has quit IRC | 11:23 | |
| *** ratailor has joined #oooq | 11:23 | |
| *** ratailor has quit IRC | 11:24 | |
| *** ratailor has joined #oooq | 11:25 | |
| *** ratailor has quit IRC | 11:26 | |
| *** jmasud has quit IRC | 11:26 | |
| *** jpena is now known as jpena|lunch | 11:31 | |
| chandankumar | arxcruz: https://review.opendev.org/#/c/751760/ please +w it | 11:35 |
| weshay|ruck | bhagyashri|rover, 0/ fyi.. my morning schedule is a little whacky today. Please prep to attend cix | 11:45 |
| bhagyashri|rover | weshay|ruck, ack | 11:45 |
| *** holser has quit IRC | 11:47 | |
| *** ratailor has joined #oooq | 11:51 | |
| *** rfolco has joined #oooq | 11:55 | |
| *** rfolco is now known as rfolco|ruck | 11:55 | |
| *** holser has joined #oooq | 12:04 | |
| arxcruz | ysandeep: what's your gerrit id? | 12:05 |
| ysandeep | arxcruz, sandeepyadav93 | 12:06 |
| arxcruz | ysandeep: not finding, add yourself as reviewer on https://review.opendev.org/#/c/751771/ if you want | 12:07 |
| arxcruz | :) | 12:07 |
| ysandeep | arxcruz, ack o/ | 12:08 |
| *** jmasud has joined #oooq | 12:11 | |
| *** Goneri has joined #oooq | 12:11 | |
| *** amoralej is now known as amoralej|lunch | 12:12 | |
| *** rlandy has joined #oooq | 12:14 | |
| rlandy | bhagyashri|rover: hey ... is there a fix for the failing train containers multinode check jobs? | 12:16 |
| rlandy | ysandeep: hi .. | 12:16 |
| ysandeep | rlandy, o/ hello | 12:16 |
| rlandy | ysandeep: lots of tempest failures downstream | 12:17 |
| rlandy | would like to promote 16.2 and 17 with you | 12:17 |
| rlandy | but we have a lot of failures there | 12:17 |
| ysandeep | rlandy, looks like i.e rabbit/mysql - slo_db.exception.DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '192.168.24.1' ([Errno 111] ECONNREFUSED)") | 12:18 |
| ysandeep | https://sf.hosted.upshift.rdu2.redhat.com/logs/openstack-component-cinder/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-rhel-8-scenario001-standalone-cinder-rhos-16.2/2c77b03/logs/undercloud/var/log/containers/nova/nova-conductor.log | 12:18 |
| rlandy | tempest.lib.exceptions.UnexpectedResponseCode: Unexpected response code received | 12:19 |
| rlandy | Details: 503 | 12:19 |
| rlandy | Details: {'message': 'This server could not verify that you are authorized to access the document you requested. Either you supplied the wrong credentials (e.g., bad password), or your browser does not understand how to supply the credentials required.<br /><br />\n\n\n', 'code': '401 Unauthorized', 'title': 'Unauthorized'} | 12:19 |
| rlandy | }}} | 12:19 |
| rlandy | ysandeep: are you seeing on 16.2 the same issue as 17 or something different? | 12:20 |
| rlandy | https://sf.hosted.upshift.rdu2.redhat.com/logs/openstack-periodic-rhos-17/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-rhel-8-scenario002-standalone-rhos-17/8d626c2/logs/undercloud/var/log/tempest/stestr_results.html | 12:20 |
| rlandy | ^^ looking at this | 12:20 |
| *** saneax has quit IRC | 12:20 | |
| *** saneax has joined #oooq | 12:20 | |
| bhagyashri|rover | rlandy, not yet | 12:21 |
| rlandy | bhagyashri|rover: known problem, though, right? | 12:21 |
| *** matbu has quit IRC | 12:21 | |
| weshay|ruck | chandankumar, sshnaidm 0/ keep driving the content-provider work .. :) we have a planning meeting tomorrow about it.. | 12:21 |
| ysandeep | rlandy, i think above unauthorized is because keystone is unable to communicate with db. | 12:21 |
| ysandeep | https://sf.hosted.upshift.rdu2.redhat.com/logs/openstack-component-cinder/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-rhel-8-scenario001-standalone-cinder-rhos-16.2/2c77b03/logs/undercloud/var/log/containers/keystone/keystone.log | 12:21 |
| ysandeep | ~~~ | 12:21 |
| ysandeep | 2020-09-11 04:33:59.369 181 ERROR keystone.server.flask.request_processing.middleware.auth_context oslo_db.exception.DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '192.168.24.1' ([Errno 111] Connection refused)") | 12:21 |
| ysandeep | ~~~ | 12:21 |
| sshnaidm | weshay|ruck, ack | 12:21 |
| rlandy | ysandeep: on 17 it looks like keystone - crrect | 12:21 |
| rlandy | ysandeep: on friday, you worked with shreshtha to up the version of python-ansible-runner, right? | 12:22 |
| rlandy | to a version above what upstream has | 12:22 |
| rlandy | perhaps we have an incompatible version? | 12:22 |
| *** matbu has joined #oooq | 12:23 | |
| ysandeep | rlandy, actually rabi requested to bump python-ansible-runner version.. they bumped to python-ansible-runner-1.4.6-3.el8ost | 12:25 |
| *** jpena|lunch is now known as jpena | 12:28 | |
| rlandy | ysandeep: hmmm ... in the BZ, it says, "We don't see it upstream as we use 1.4.5." | 12:28 |
| rlandy | ysandeep: guessing here - but maybe it's worth taking 1.4.5 | 12:29 |
| * rlandy looks where we can install 1.4.5. from | 12:29 | |
| rlandy | we really need to promote both | 12:29 |
| rlandy | ysandeep: ha ... rabi responded in the BZ | 12:32 |
| * rlandy should read more carefully | 12:32 | |
| rlandy | Looks like it's the OOM killer. May be we reduce the number of workers as undercloud mostly runs on VMs everywhere. | 12:33 |
| rlandy | A process in the process pool was terminated abruptly while the future was running or pending.", "params": {}, "success": false} | 12:33 |
| rlandy | 2020-09-10 23:15:25 | | 12:33 |
| rlandy | maybe tempest is hitting the same? | 12:33 |
| ysandeep | rlandy, checking 17 standalone journal logs.. to see if we have oom there | 12:35 |
| rlandy | ysandeep: idk if tempest shows the same error or if it is showing this one instead | 12:37 |
| rlandy | sshnaidm: should ansible-collections-openstack have a stable/train branch? responding to https://bugzilla.redhat.com/show_bug.cgi?id=1867133 | 12:40 |
| openstack | bugzilla.redhat.com bug 1867133 in ansible-collections-openstack "Downstream repo needed for ansible-collections-openstack" [Medium,Modified] - Assigned to shrjoshi | 12:40 |
| ysandeep | meh.. looks like we don't capture journal for standalone | 12:40 |
| rlandy | I think it should only be 17.0 as upstream it's only master | 12:40 |
| bhagyashri|rover | rlandy, sorry i am in meeting will get back to you .. | 12:41 |
| rlandy | bhagyashri|rover: np | 12:41 |
| rlandy | ysandeep: ok - let me try drop the tempest_workers and the undercloud_workers like in https://github.com/openstack/tripleo-ci/blob/master/toci-quickstart/config/testenv/ovb.yml#L74 | 12:43 |
| ysandeep | rlandy, okay | 12:44 |
| sshnaidm | rlandy, lemme check | 12:45 |
| sshnaidm | rlandy, not yet in train, but I have a patch to add it | 12:47 |
| *** ratailor has quit IRC | 12:49 | |
| *** ratailor has joined #oooq | 12:50 | |
| *** ratailor has quit IRC | 12:51 | |
| weshay|ruck | marios, may need your help describing what you are seeing from zuul... re: the root vs zuul user | 12:59 |
| ykarel | rlandy, stable/train or train-rdo? | 12:59 |
| ykarel | isn't that branchless upstream? | 12:59 |
| weshay|ruck | marios, here.. https://trello.com/c/Os3f6nVQ/1660-cixlp1895138tripleociproa-centos-8-standalone-upgrade-ussuri-fails-build-test-packages-issue-creating-root-dlrn | 12:59 |
| weshay|ruck | marios, so we can get some folks from zuul to engage | 13:00 |
| rlandy | ykarel: if it's branchless upstream then downstream we have to pick a branch | 13:01 |
| rlandy | there is no master equivalent | 13:01 |
| rlandy | so atm it's 17 | 13:01 |
| rlandy | if there will be a stable/train, then we need 16.2 - if it's branchless then not | 13:02 |
| ykarel | rlandy, actually question is why it was built for downstream without a train counterpart in RDO | 13:02 |
| rlandy | ykarel: not sur e- upstream should always be first | 13:03 |
| marios | weshay|ruck: ack but lets talk about it on scrum now? | 13:03 |
| ykarel | rlandy, general answer is yes, until and unless there is exception to it | 13:03 |
| weshay|ruck | arxcruz, rfolco, zbr, sshnaidm, rlandy, marios, ysandeep, bhagyashris, svyas, pojadhav, akahat, weshay, chandankumar mtg | 13:03 |
| ykarel | BZ doesn't have any thing mentioned related to exeception | 13:03 |
| rlandy | ykarel: correct - which os why I think 17 should be the only branch - and is the only branch I see atm | 13:04 |
| ykarel | rlandy, ok something wrong then | 13:04 |
| ykarel | me talking about builds, and u branches | 13:04 |
| ykarel | u seen my comment? | 13:05 |
| *** amoralej|lunch is now known as amoralej | 13:06 | |
| rlandy | ykarel: ack - branches are what concerns us atm | 13:07 |
| rlandy | so we pull the right version in jobs | 13:08 |
| ykarel | rlandy, so build is not needed? and you not requested for it, right? | 13:09 |
| rlandy | ykarel: correct ... "This BZ is marked 16.2 - but https://opendev.org/openstack/ansible-collections-openstack currently only has a master branch (which would be rhos-17.0-trunk-patches)." | 13:10 |
| rlandy | which is what we have for the moment | 13:10 |
| rlandy | the BZ is 16.2 because of component pipeline epic tracking | 13:11 |
| ykarel | ok will talk with shreshtha what led to confusion, afaiu https://opendev.org/openstack/ansible-collections-openstack is branchless | 13:12 |
| ykarel | iirc sshnaidm shared ^ | 13:12 |
| sshnaidm | ykarel, yeah, it's branchless | 13:15 |
| ykarel | sshnaidm, so u planning to move with https://review.rdoproject.org/r/#/c/29000/ ? | 13:18 |
| sshnaidm | ykarel, yeah | 13:18 |
| sshnaidm | ykarel, can you remind please what should it depend on? | 13:19 |
| *** irclogbot_3 has quit IRC | 13:19 | |
| ykarel | sshnaidm, send rdoinfo patch to update tags/train.yml which depends-on ^ | 13:20 |
| chandankumar | rfolco|ruck: bhagyashri|rover https://review.opendev.org/#/c/751760/ to skip tests based on job name | 13:20 |
| bhagyashri|rover | chandankumar, ack, thanks | 13:20 |
| *** irclogbot_1 has joined #oooq | 13:24 | |
| weshay|ruck | https://review.rdoproject.org/r/#/c/29108/ | 13:24 |
| rfolco|ruck | arxcruz, I just did a revert here and its failing pep8, any clues? https://review.opendev.org/751077 | 13:27 |
| arxcruz | rfolco|ruck: yes, the flake8 on test-requirements is < 3.0.0 and it's not playing well with python 3.8 | 13:28 |
| arxcruz | rfolco|ruck: I fix it already, can you rebase your patch ? | 13:28 |
| rfolco|ruck | arxcruz, k thx | 13:29 |
| *** TrevorV has joined #oooq | 13:33 | |
| marios | https://bugs.launchpad.net/tripleo/+bug/1895138 | 13:33 |
| openstack | Launchpad bug 1895138 in tripleo "centos-8 standalone-upgrade-ussuri fails build-test-packages issue creating /root/DLRN" [Critical,Triaged] - Assigned to Marios Andreou (marios-b) | 13:33 |
| marios | weshay|ruck: ^ | 13:33 |
| ykarel | ohh it's back again /root/DLRN thing | 13:35 |
| ykarel | is it consistent this time, | 13:35 |
| akahat | marios, i think we need to revert it back https://review.rdoproject.org/r/#/c/28081/.. and let's have experiments with parent patch https://review.rdoproject.org/r/#/c/28019/ | 13:38 |
| *** jtomasek has quit IRC | 13:39 | |
| chandankumar | rlandy: arxcruz marios rfolco|ruck weshay|ruck zbr sshnaidm movie link https://www.imdb.com/title/tt3449292/ | 13:39 |
| bhagyashri|rover | rlandy, regarding failing train containers multinode check jobs i just checked those job before CIX meeting so dont have more updates .. will look into it | 13:39 |
| marios | akahat: ack yeah i saw it | 13:39 |
| marios | akahat: ok go ahead and i will work on the parent patch | 13:40 |
| weshay|ruck | chandankumar, you mentioned a multinode bug.. do you have a link? | 13:45 |
| rlandy | bhagyashri|rover: no worries - know are looking into it | 13:46 |
| rlandy | chandankumar: sorry you missed the planning meeting - from your and weshay|ruck's comment on dependency work ... moved to use a file with repos defined explicitly ... https://review.opendev.org/#/c/750769/13/config/release/dependency_ci/centos8stream/repo_config.yaml | 13:47 |
| rlandy | ^^ think this is what you wanted? | 13:48 |
| chandankumar | weshay|ruck: not a bug, but it is failing on my patches also | 13:48 |
| bhagyashri|rover | rlandy, ack | 13:48 |
| chandankumar | weshay|ruck: https://review.opendev.org/#/c/749645/ | 13:49 |
| weshay|ruck | chandankumar, thanks.. looking | 13:49 |
| chandankumar | let me open a bug | 13:49 |
| weshay|ruck | chandankumar, k.. that's the fix? | 13:49 |
| chandankumar | weshay|ruck: nope, that is a different patch all together | 13:50 |
| chandankumar | on my patch the multinode train c7 and c8 are failing | 13:50 |
| weshay|ruck | k | 13:51 |
| weshay|ruck | chandankumar, sshnaidm can you folks hackmd up a list of patches that need to merge to resolve in a hackmd | 13:51 |
| weshay|ruck | please | 13:51 |
| rfolco|ruck | ykarel, can you please point me to the role you suggest in https://review.opendev.org/#/c/733659/79/roles/tripleo-repos/tasks/main.yml ? | 13:53 |
| weshay|ruck | rlandy, there is quite a train of patches that need to land before multinode is fixed | 13:55 |
| weshay|ruck | chandankumar, sshnaidm perhaps we need to have others stand down for a bit as we get your patches landed.. and arx's? | 13:55 |
| weshay|ruck | perhaps move some jobs to non-voting I suppose is an option too | 13:56 |
| chandankumar | weshay|ruck: I am thinking to get rid of c7 train jobs, not sure we are adding value there since cern have their own forks | 13:57 |
| chandankumar | and they donot depend on our containers and promoted contents | 13:57 |
| chandankumar | it will help us one less thing to worry about | 13:57 |
| weshay|ruck | chandankumar, so.. make it non-voting for now if it's causing headaches | 13:58 |
| weshay|ruck | anything else we can move to nv to help get this work through? | 13:58 |
| sshnaidm | weshay|ruck, I don't think I have any patches to merge, seems like all was merged during the week | 14:00 |
| rlandy | weshay|ruck: k, np - will recheck when those patches are in | 14:01 |
| chandankumar | weshay|ruck: rlandy c8 train bug https://bugs.launchpad.net/tripleo/+bug/1895290 already filed | 14:01 |
| openstack | Launchpad bug 1895290 in tripleo "[train] ERROR root [ ] Image prepare failed: A process in the process pool was terminated abruptly while the future was running or pending failing on few jobs" [Critical,Triaged] | 14:01 |
| chandankumar | rlandy: weshay|ruck on master, sc12 is also broken currently fix in puupet-nova for sc12 needs to be merged https://review.opendev.org/750967 and it depends on https://review.opendev.org/#/c/751771/ | 14:06 |
| *** sshnaidm is now known as sshnaidm|afk | 14:07 | |
| weshay|ruck | chandankumar, ya.. if that needs to go nv.. fine by me.. we added scenario012 to the compnent pipeline on friday w/ rfolco|ruck | 14:07 |
| weshay|ruck | to prevent that crap in the future | 14:08 |
| rlandy | ok | 14:08 |
| ykarel | rfolco|ruck, looking | 14:08 |
| weshay|ruck | bhagyashri|rover, this was the patch you were talking about in cix https://bugs.launchpad.net/tripleo/+bug/1895290 can you add ur review | 14:09 |
| openstack | Launchpad bug 1895290 in tripleo "[train] ERROR root [ ] Image prepare failed: A process in the process pool was terminated abruptly while the future was running or pending failing on few jobs" [Critical,Triaged] | 14:09 |
| ykarel | rfolco|ruck, i meant https://review.opendev.org/#/c/733659/79/roles/tripleo-repos/defaults/main.yml@9 | 14:09 |
| ykarel | virtualenv_python: python3 role var | 14:09 |
| bhagyashri|rover | weshay|ruck, yes | 14:10 |
| bhagyashri|rover | weshay|ruck, rlandy chandankumar here is patch https://review.rdoproject.org/r/#/c/29313/ and tested here https://review.rdoproject.org/r/#/c/29252/, https://review.rdoproject.org/r/#/c/29254/ | 14:10 |
| chandankumar | rfolco|ruck: bhagyashri|rover weshay|ruck rlandy https://review.opendev.org/#/c/751815/ to c7 train multinode to nv | 14:10 |
| rfolco|ruck | ykarel, the recommended way is to use python3 -m venv to avoid problems like | 14:11 |
| rfolco|ruck | # NOTE(rfolco): we need to set both virtualenv_command and | 14:11 |
| rfolco|ruck | # ansible_python_interpreter, otherwise we hit this error: | 14:11 |
| rfolco|ruck | # Failed to import the required Python library (setuptools) | 14:11 |
| rfolco|ruck | # This happens in a dual python env like centos7.8 | 14:11 |
| rlandy | chandankumar: thanks | 14:11 |
| rlandy | bhagyashri|rover: chandankumar: ok if I merge https://review.rdoproject.org/r/#/c/29313/? | 14:12 |
| ykarel | rfolco|ruck, i don't have objection on python3 -m venv thing | 14:12 |
| ykarel | seems /me not explaining clearly | 14:12 |
| chandankumar | rlandy: yes more context here https://bugs.launchpad.net/tripleo/+bug/1895290/comments/2 and 3 also | 14:13 |
| openstack | Launchpad bug 1895290 in tripleo "[train] ERROR root [ ] Image prepare failed: A process in the process pool was terminated abruptly while the future was running or pending failing on few jobs" [Critical,Triaged] | 14:13 |
| rlandy | done | 14:13 |
| chandankumar | rlandy: bhagyashri|rover thanks! | 14:13 |
| rfolco|ruck | ykarel, This parameter should not be used when virtualenv_command is using pyvenv or the -m venv module. | 14:14 |
| rfolco|ruck | ykarel, either you use command or this | 14:14 |
| rfolco|ruck | https://docs.ansible.com/ansible/latest/modules/pip_module.html#parameter-virtualenv_python | 14:14 |
| ykarel | rfolco|ruck, so again, i meant /usr/bin/python3 -m venv --> /usr/bin/{{ virtualenv_python }} -m venv | 14:16 |
| ykarel | may be ^ more clear | 14:16 |
| ykarel | as there is role var named virtualenv_python in tripleo-repos role | 14:16 |
| ykarel | but that's not being used after this patch | 14:16 |
| ykarel | or u can try cleaning up that var if u don't want to use that var in that role | 14:16 |
| marios | weshay|ruck: https://bugs.launchpad.net/tripleo/+bug/1895138/comments/2 | 14:21 |
| openstack | Launchpad bug 1895138 in tripleo "centos-8 standalone-upgrade-ussuri fails build-test-packages issue creating /root/DLRN" [Critical,Triaged] - Assigned to Marios Andreou (marios-b) | 14:21 |
| *** ykarel_ has joined #oooq | 14:25 | |
| *** ykarel has quit IRC | 14:28 | |
| *** ykarel__ has joined #oooq | 14:32 | |
| rfolco|ruck | ykarel_, thanks for the clarification | 14:34 |
| *** ykarel_ has quit IRC | 14:35 | |
| *** ykarel__ is now known as ykarel | 14:35 | |
| rfolco|ruck | ykarel, the idea is to always use python3, so no python2 anymore. That's why we enforce python3 | 14:35 |
| ykarel | rfolco|ruck, may be my messages stuck | 14:36 |
| ykarel | i got disconnected | 14:36 |
| ykarel | checks history, if not sent will resend | 14:36 |
| ykarel | it's in history, may be u not received? | 14:37 |
| ykarel | or seen | 14:37 |
| ykarel | http://eavesdrop.openstack.org/irclogs/%23oooq/%23oooq.2020-09-14.log.html#t2020-09-14T14:16:48 | 14:37 |
| rfolco|ruck | ykarel, I got it. I just think python3 is the the only option :) | 14:37 |
| ykarel | ack so unused var can be get rid off | 14:39 |
| rfolco|ruck | ykarel, will check that again, and will ping with the new patchset, thanks for your review | 14:40 |
| rfolco|ruck | ykarel, yep, can get rid of var in tripleo-repos defaults, updated https://review.opendev.org/733659 | 14:42 |
| rfolco|ruck | thx | 14:42 |
| ykarel | ack | 14:42 |
| chandankumar | rlandy: weshay|ruck bhagyashri|rover merged c7 multinode nv and c8 train fix together https://review.opendev.org/751828 | 14:43 |
| chandankumar | otherwise it will never merge | 14:43 |
| weshay|ruck | chandankumar, thanks | 14:44 |
| weshay|ruck | marios, I think if we change the ci logging to -vvv we'll see the user info in the job.txt | 14:44 |
| rfolco|ruck | chandankumar, arxcruz please review https://review.opendev.org/#/c/751077 | 14:45 |
| chandankumar | arxcruz: putting a depends on your tempest tqe patch by editing it | 14:45 |
| chandankumar | if ok? | 14:45 |
| weshay|ruck | marios, have you tried that? | 14:45 |
| weshay|ruck | marios, also do you want to sync | 14:45 |
| arxcruz | chandankumar: in this case it's not required | 14:45 |
| chandankumar | arxcruz: https://review.opendev.org/751828 is needed on your patch to merge | 14:46 |
| marios | weshay|ruck: sure | 14:47 |
| chandankumar | weshay|ruck: rfolco|ruck rlandy reviews in order to fix master check job https://review.opendev.org/#/c/751828/ -> https://review.opendev.org/#/c/751771/ -> https://review.opendev.org/750967 | 14:49 |
| *** udesale_ has joined #oooq | 14:51 | |
| *** udesale__ has quit IRC | 14:54 | |
| rlandy | reviewed | 14:54 |
| *** ykarel is now known as ykarel|away | 15:00 | |
| ysandeep | rlandy, weshay|ruck fyi.. pulling latest podman from osp-trunk-deps worked for baremetal bm job - https://code.engineering.redhat.com/gerrit/#/c/211641/ | 15:04 |
| ysandeep | rlandy, weshay|ruck which component we need to select to open a bz for RelDel? | 15:04 |
| weshay|ruck | ysandeep, let them | 15:05 |
| ysandeep | rlandy, weshay|ruck we opened a bz last week to add lower constraint in 16.2 for podman - https://bugzilla.redhat.com/show_bug.cgi?id=1878189 , but I guess we need a different bz to bump the version.. | 15:05 |
| openstack | bugzilla.redhat.com bug 1878189 in rhosp-release "Add lower contraint for podman version" [High,New] - Assigned to jjoyce | 15:05 |
| rlandy | ysandeep: that's fine ^^ component-wise | 15:06 |
| rlandy | they will change it if need be | 15:06 |
| ysandeep | weshay|ruck, will select rhos-release for now | 15:06 |
| ysandeep | rlandy, thanks.. i am opening a separate bz to bump the version | 15:07 |
| rlandy | distribution - I think we've used that | 15:07 |
| ysandeep | rlandy, yes we used distribution for https://bugzilla.redhat.com/show_bug.cgi?id=1873465 | 15:08 |
| openstack | bugzilla.redhat.com bug 1873465 in distribution "RHEL 8.3 nodepool node required for testing in zuul - hitting libgcrypt issue described in BZ1840485" [Medium,Verified] - Assigned to rlandy | 15:08 |
| rlandy | ysandeep: you can always ping lon and ask for the relevant component - he triages the bugs | 15:12 |
| weshay|ruck | chandankumar, anything else I can assist w/? | 15:15 |
| ysandeep | rlandy, ack thanks!, checking with him and then i will cix it.. if it will take time i think we can probably merge this for the meantime - https://code.engineering.redhat.com/gerrit/#/c/211641/ | 15:17 |
| weshay|ruck | rlandy, I'll brief you on cedric's request later | 15:18 |
| rlandy | weshay|ruck: ack | 15:19 |
| *** ykarel|away has quit IRC | 15:20 | |
| weshay|ruck | rlandy, https://code.engineering.redhat.com/gerrit/#/c/211641/ | 15:20 |
| rlandy | weshay|ruck: ^^ I'm not sure about pulling podman 1.9.3 | 15:23 |
| chandankumar | weshay|ruck: nope, we just need to merge those patches that's it | 15:24 |
| rlandy | mass enabling hotfixes could also cuase issues | 15:24 |
| rlandy | weshay|ruck: ysandeep: ^^ | 15:24 |
| rlandy | we disabled that because the new update were killing us | 15:24 |
| rlandy | since it's only podman and dicker it may be ok | 15:25 |
| rlandy | ysandeep: ^^ did you test this on all tests | 15:25 |
| weshay|ruck | bhagyashri|rover++ | 15:26 |
| ysandeep | rlandy, tested with 16.2 standalone and bm - https://code.engineering.redhat.com/gerrit/#/c/211643/ | 15:26 |
| weshay|ruck | thanks for the upstream/rdo patches for swap | 15:26 |
| ysandeep | rlandy, i can run against more jobs | 15:26 |
| rlandy | ysandeep: I think it's ok because we are now building on 8.3 nodes | 15:27 |
| rlandy | which we weren't before | 15:27 |
| rlandy | http://git.app.eng.bos.redhat.com/git/tripleo-environments.git/commit/config/release/rhos-16.2.yml?id=8d82cb3f9c8c2440057fc3b6a549be379f48b10a | 15:28 |
| rlandy | ysandeep: weshay|ruck: what about container builds> | 15:28 |
| bhagyashri|rover | weshay|ruck, :) | 15:29 |
| rlandy | http://git.app.eng.bos.redhat.com/git/openstack/tripleo-ci-internal-jobs.git/tree/playbooks/tripleo-buildcontainers/pre_minimal.yaml#n33 | 15:30 |
| rlandy | and when we build containers in a job | 15:30 |
| rlandy | what version of podman do we use there? | 15:30 |
| rlandy | and should they be the same? | 15:31 |
| ysandeep | rlandy, i think version should be same everywhere unless we hit issues and have reasons to not have them same.. but i didn't turn module_hotfixes in my above patch... | 15:32 |
| rlandy | ysandeep: checking what version of podman we pick up there ... | 15:32 |
| rlandy | I think it's locked ... | 15:32 |
| ysandeep | 1.9.3-2 | 15:33 |
| ysandeep | i just removed beta, enabled osp-trunk-dep which include_pkgs on podman, podman-docker .. it worked for the job | 15:34 |
| ysandeep | s/which/with/ | 15:34 |
| rlandy | podman.x86_64 1.6.4-14.module+el8.3.0+7660+b7198318 @rhosp-rhel-8.3-appstream | 15:35 |
| rlandy | https://sf.hosted.upshift.rdu2.redhat.com/logs/openstack-periodic-integration-rhos-16.2/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-build-containers-ubi-8-internal-rhel-8-build-push-upload-rhos-16.2/de4cb56/logs/system/installed_pkgs.txt | 15:35 |
| rlandy | ysandeep: imho we should check if we can build containers with that version | 15:36 |
| rlandy | otherwise we'll be out of sync | 15:36 |
| *** marios has quit IRC | 15:36 | |
| rlandy | not sure if this impacts overcloud image builds | 15:36 |
| rlandy | but if image build in BM, that should be ok | 15:37 |
| ysandeep | rlandy, i agree with you.. yes image build works in bm.. I will test more jobs/ and container build - first thing tomorrow morning | 15:37 |
| rlandy | ysandeep: if we're good with image builds, I'll merge tomorrow | 15:37 |
| rlandy | just don't want you to have more headaches :) | 15:38 |
| rlandy | I mean container builds | 15:38 |
| ysandeep | rlandy, preparing patch to test container build | 15:39 |
| rlandy | cool - thanks | 15:39 |
| rlandy | ysandeep: have a testproject running now to test reducing undercloud_workers and tempest_workers | 15:41 |
| *** jfrancoa has quit IRC | 15:42 | |
| *** jfrancoa has joined #oooq | 15:42 | |
| weshay|ruck | relocating.. back in 1.5 hours | 15:54 |
| rlandy | chandankumar: hey - https://sf.hosted.upshift.rdu2.redhat.com/logs/openstack-periodic-integration-rhos-16.2/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-rhel-8-scenario001-standalone-rhos-16.2/cc156ec/logs/undercloud/var/log/tempest/stestr_results.html | 16:16 |
| rlandy | we've got consistent failures in tempest scenario tests | 16:16 |
| rlandy | failing login | 16:16 |
| rlandy | https://sf.hosted.upshift.rdu2.redhat.com/logs/95/200295/19/check/periodic-tripleo-ci-rhel-8-scenario010-standalone-rhos-17/841632a/logs/undercloud/var/log/tempest/stestr_results.html | 16:17 |
| rlandy | ^^ does that look at all familiar to you? | 16:17 |
| *** derekh has quit IRC | 16:22 | |
| *** derekh has joined #oooq | 16:23 | |
| *** bogdando has quit IRC | 16:24 | |
| *** amoralej is now known as amoralej|off | 16:25 | |
| *** jpena is now known as jpena|off | 16:31 | |
| ysandeep | see you tomorrow guys o/ | 16:31 |
| *** ysandeep is now known as ysandeep|away | 16:31 | |
| *** udesale_ has quit IRC | 16:38 | |
| *** ykarel|away has joined #oooq | 16:56 | |
| *** derekh has quit IRC | 17:00 | |
| *** ykarel|away has quit IRC | 17:05 | |
| *** holser has quit IRC | 17:18 | |
| weshay|ruck | back | 17:28 |
| *** dtantsur is now known as dtantsur|afk | 17:35 | |
| *** panda has joined #oooq | 17:40 | |
| rlandy | weshay|ruck: hey - want to talk about the dependency work? | 17:57 |
| *** holser has joined #oooq | 17:57 | |
| rlandy | debugging the first failure ... | 17:57 |
| rlandy | Error: error configuring network namespace for container dd2dd92296b7cdc8c62e3a5cd8b1343001dbd002ae6ddf091f080a4f6eac6c95: Missing CNI default network | 17:57 |
| weshay|ruck | rlandy, sure | 18:00 |
| weshay|ruck | rlandy, in https://meet.google.com/duh-wwnu-pcq?authuser=1 | 18:00 |
| weshay|ruck | rlandy, https://opendev.org/openstack/ansible-role-container-registry | 18:06 |
| weshay|ruck | rlandy https://sf.hosted.upshift.rdu2.redhat.com/logs/openstack-periodic-rhos-17/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-build-containers-ubi-8-internal-rhel-8-build-push-upload-rhos-17/a831e94/logs/system/installed_pkgs.txt | 18:10 |
| *** panda has quit IRC | 18:14 | |
| *** dsneddon has joined #oooq | 18:23 | |
| rfolco|ruck | chandankumar, arxcruz anyone aware of this bug ? AttributeError: 'TagFloatingIpTestJSON' object has no attribute 'assertItemsEqual' | 18:46 |
| rfolco|ruck | happening in periodic master now https://logserver.rdoproject.org/openstack-periodic-integration-main/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-standalone-master/b7b427b/logs/undercloud/var/log/tempest/stestr_results.html.gz | 18:46 |
| rfolco|ruck | weshay|ruck, have u seen this ? | 18:47 |
| weshay|ruck | rfolco|ruck, nope.. AttributeError: 'UpdateTagsTest' object has no attribute 'assertItemsEqual' | 18:50 |
| weshay|ruck | rfolco|ruck, looks like all the same bug | 18:51 |
| rfolco|ruck | weshay|ruck, yeah its seems to be a problem in the neutron tempest plugin | 18:51 |
| weshay|ruck | aye | 18:51 |
| rfolco|ruck | weshay|ruck, ok will file a bug | 18:51 |
| rfolco|ruck | thx | 18:51 |
| *** jtomasek has joined #oooq | 18:56 | |
| rlandy | weshay|ruck: so I don;t think regular container builds run that local registry process | 19:07 |
| rlandy | will have to build an internal standalone job that builds containers | 19:07 |
| *** jtomasek has quit IRC | 19:08 | |
| weshay|ruck | rfolco|ruck, can you please run a couple of the failed master jobs w/ validation errors w/ a depends on https://review.opendev.org/751883 | 19:31 |
| weshay|ruck | rfolco|ruck, seeing stuff like https://logserver.rdoproject.org/openstack-periodic-integration-main/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001-master/49caebf/logs/undercloud/home/zuul/undercloud_install.log.txt.gz | 19:32 |
| rfolco|ruck | weshay|ruck, yep | 19:32 |
| rfolco|ruck | weshay|ruck, https://review.rdoproject.org/r/29332 Test 751883 | 19:44 |
| weshay|ruck | rfolco|ruck, meh... still pulled from https://trunk.rdoproject.org/centos8-master/component/tripleo/current/ | 20:17 |
| weshay|ruck | rfolco|ruck, ya.. excludepkgs is not implemented https://logserver.rdoproject.org/32/29332/1/check/periodic-tripleo-ci-centos-8-scenario001-standalone-master/da96162/logs/undercloud/etc/yum.repos.d/delorean-current.repo.txt.gz | 20:17 |
| weshay|ruck | hrm.. sorry.. it's there | 20:18 |
| weshay|ruck | {% if repo.exclude is defined %}exclude={{ repo.exclude|join(",") }} | 20:18 |
| weshay|ruck | {% endif %} | 20:18 |
| weshay|ruck | rfolco|ruck, updated depends-on and rechecked | 20:23 |
| rfolco|ruck | weshay|ruck, ok, thx will watch, I am behind you on that | 20:24 |
| weshay|ruck | rfolco|ruck, ok.. now it's recreating | 20:52 |
| weshay|ruck | using current here I think is confusing things | 20:52 |
| weshay|ruck | rfolco|ruck, if they can tell us which rpms to use.. we can get through this.. | 20:53 |
| weshay|ruck | move all the validation rpms to the validation component.. and this kind of shit will be caught in the future | 20:53 |
| rfolco|ruck | weshay|ruck, ok I saw your email | 20:53 |
| rfolco|ruck | nice | 20:53 |
| weshay|ruck | rfolco|ruck, and upstream is fooked until this merges https://review.opendev.org/#/c/751828/ | 20:54 |
| rfolco|ruck | weshay|ruck, wow still on gate | 20:55 |
| weshay|ruck | ya.. that's what resets do :) | 20:56 |
| rfolco|ruck | weshay|ruck, I am still confused if tripleo-ci-centos-8-containers-multinode and tripleo-ci-centos-8-containers-multinode-train are failing on specific patches or not, can you please have a quick look ? | 20:59 |
| rfolco|ruck | https://zuul.opendev.org/t/openstack/builds?job_name=tripleo-ci-centos-8-containers-multinode-train&project=openstack/tripleo-quickstart | 20:59 |
| rfolco|ruck | fs037 shouldn't break it | 21:01 |
| rfolco|ruck | https://review.opendev.org/#/c/751340/ | 21:01 |
| rfolco|ruck | will check later or tomorrow | 21:08 |
| *** rfolco|ruck has quit IRC | 21:12 | |
| *** TrevorV has quit IRC | 21:33 | |
| *** tosky has quit IRC | 22:33 | |
| *** Goneri has quit IRC | 23:53 | |
| *** holser has quit IRC | 23:55 | |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!