| *** rf0lc0 has quit IRC | 00:38 | |
| *** rf0lc0 has joined #oooq | 00:56 | |
| *** ysandeep is now known as ysandeep|afk | 01:28 | |
| *** rf0lc0 has quit IRC | 01:29 | |
| *** rlandy|rover|bbl is now known as rlandy|rover | 03:15 | |
| *** jmasud has joined #oooq | 03:34 | |
| bhagyashris | soniya29|ruck, hey | 03:56 |
|---|---|---|
| bhagyashris | tripleo-tox-molecule and openstack-tox-py36 is failing on upstream check is this a know issue | 03:57 |
| bhagyashris | log url is showing Not Found issue https://95c0997776a39d59f4de-b1c6ced64813770b10e3ee34efa4a7c5.ssl.cf2.rackcdn.com/757861/5/check/openstack-tox-py36/51feec3/tox/reports.html | 03:58 |
| *** udesale has joined #oooq | 03:59 | |
| *** jbadiapa has joined #oooq | 05:13 | |
| *** marios has joined #oooq | 05:13 | |
| *** udesale_ has joined #oooq | 05:31 | |
| *** udesale has quit IRC | 05:32 | |
| *** udesale__ has joined #oooq | 05:54 | |
| *** udesale_ has quit IRC | 05:58 | |
| *** ysandeep|afk is now known as ysandeep | 06:12 | |
| chandankumar | zbr: Hello | 06:16 |
| chandankumar | zbr: https://cd5ea8073cdb9af0c36c-e2761754b9e1d0aad3806c0457d8cf4d.ssl.cf1.rackcdn.com/752421/85/check/tripleo-tox-molecule/bd2d22a/job-output.txt | 06:16 |
| chandankumar | zbr: ImportError: cannot import name 'ansible_version need help on fixing it | 06:16 |
| chandankumar | zbr: i got the fix from rlandy https://review.opendev.org/#/c/758502/1 | 06:18 |
| *** ykarel|away has joined #oooq | 06:26 | |
| *** ykarel|away has quit IRC | 06:27 | |
| *** saneax has joined #oooq | 06:28 | |
| soniya29|ruck | bhagyashris hello | 06:41 |
| bhagyashris | soniya29|ruck, tripleo-tox-molecule and openstack-tox-py36 is failing on upstream check is this a know issue | 06:45 |
| bhagyashris | log url is showing Not Found issue https://95c0997776a39d59f4de-b1c6ced64813770b10e3ee34efa4a7c5.ssl.cf2.rackcdn.com/757861/5/check/openstack-tox-py36/51feec3/tox/reports.html | 06:45 |
| soniya29|ruck | bhagyashris: yes it is a known issue | 06:50 |
| soniya29|ruck | we have bug filed for it:- https://bugs.launchpad.net/tripleo/+bug/1900033 | 06:52 |
| openstack | Launchpad bug 1900033 in tripleo "openstack-tox-molecule jobs are failing with "ImportError: cannot import name 'ansible_version'"" [Critical,In progress] - Assigned to Ronelle Landy (rlandy) | 06:52 |
| bhagyashris | soniya29|ruck, ack | 06:55 |
| zbr | chandankumar: well... yesterday i send message on tripleo with instruction on how to deal, today I see fix not following the guidelines. | 07:01 |
| zbr | i wonder why we do not use ansible 2.10 for molecule testing | 07:03 |
| chandankumar | zbr: well sshnaidm has putted a test patch for 2.10 https://review.opendev.org/#/c/758369/ | 07:05 |
| chandankumar | zbr: I see http://eavesdrop.openstack.org/irclogs/%23tripleo/%23tripleo.2020-10-15.log.html#t2020-10-15T12:39:06 | 07:07 |
| *** pmannidi has quit IRC | 07:15 | |
| zbr | i am working on a proper fix | 07:17 |
| chandankumar | zbr: cool,thanks :-) | 07:18 |
| chandankumar | zbr ++ | 07:18 |
| chandankumar | zbr: https://review.opendev.org/#/c/758502/ can we add the recommendation link in commit msg? | 07:19 |
| *** jtomasek has joined #oooq | 07:24 | |
| zbr | sure, i am still working on the best approach | 07:31 |
| marios | chandankumar: so will you abandon? https://review.opendev.org/#/c/758529/2 | 07:32 |
| zbr | chandankumar: current breakage is the achievement of my last week work on molecule.... | 07:32 |
| marios | chandankumar: or perhaps we want it until the other one merges? | 07:32 |
| * zbr thinking one needs to break few eggs to make a good omelette | 07:33 | |
| chandankumar | zbr: oh | 07:33 |
| chandankumar | zbr: https://review.opendev.org/#/c/758529/2 can we merge this? till it unblocks other patches | 07:35 |
| *** ysandeep is now known as ysandeep|lunch | 07:35 | |
| zbr | chandankumar: if it passes CI yes, it is a workaround that gives time for better fix, with included links. | 07:36 |
| chandankumar | zbr: ok, sounds good to me | 07:37 |
| zbr | i am not against workarounds when they have links to bugs, so we can safely undo them when they are no longer needed. | 07:37 |
| chandankumar | zbr: https://f9c28e74a4e26f2c90e8-fa663b3b43bb6eacc0d3184a52007f13.ssl.cf5.rackcdn.com/758502/2/check/openstack-tox-molecule/e34fcd7/ | 07:38 |
| chandankumar | current update on tq patch is failing | 07:38 |
| zbr | wait, i am releasing two new upstream wheel that should fix it, luckly the tests upstream run in 5-10mins. | 07:39 |
| chandankumar | ok | 07:39 |
| * zbr happy to share success in getting 10% EPP discount on Apple accessories (LG monitor) | 07:48 | |
| zbr | chandankumar: do you know than ansible 2.11 will require py38? | 07:52 |
| zbr | ... on controller. | 07:52 |
| chandankumar | nope | 07:52 |
| zbr | i found it yesterday as AnsibleFest, is not yet implemented but they were quite confident. | 07:53 |
| zbr | i argued against, proposing py36 as minimal as being more common sense. | 07:54 |
| chandankumar | zbr: let me know new wheels are pushed to pypi | 07:58 |
| *** Tengu has quit IRC | 08:00 | |
| zbr | release notifications go directly to #ansible-molecule -- in fact tags creation goes there, few more minutes after the wheel appears. | 08:01 |
| *** ysandeep|lunch is now known as ysandeep | 08:09 | |
| *** sanjayu_ has joined #oooq | 08:12 | |
| zbr | chandankumar: we really need to add PIP_USE_FEATURE=2020-resolver to out tox.ini files. | 08:12 |
| zbr | that makes pip use the newer resolved, much better. they will switch to it by default soon, but better to fix issues before it hits us. | 08:13 |
| zbr | newer one refuses to create conflicts | 08:14 |
| *** saneax has quit IRC | 08:14 | |
| *** Tengu has joined #oooq | 08:22 | |
| *** Tengu has quit IRC | 08:36 | |
| *** Tengu has joined #oooq | 08:43 | |
| *** ysandeep is now known as ysandeep|afk | 08:45 | |
| zbr | i am facing some local pip issues, pip seems to get stuck trying to install various packages, cpu remains at 100%. | 08:47 |
| zbr | looks like an endless loop | 08:47 |
| chandankumar | zbr: i think wheels got updated | 08:48 |
| zbr | haha... i found bug with new resolver, endless loop. | 08:49 |
| chandankumar | cool | 08:49 |
| zbr | there is a reason why they did not just release it.... | 08:50 |
| marios | can i get some votes there please https://review.opendev.org/#/c/755472/ | 08:50 |
| marios | chandankumar: so we will merge that for now ? https://review.opendev.org/#/c/758529/ | 08:50 |
| marios | chandankumar: i saw you restored | 08:50 |
| chandankumar | marios: it is failing still | 08:51 |
| marios | chandankumar: you mean even with the pin/? | 08:51 |
| chandankumar | marios: yes | 08:52 |
| chandankumar | marios: trying different route now https://review.opendev.org/#/c/758536/ | 08:52 |
| chandankumar | if this one work, we will merge it | 08:53 |
| marios | chandankumar: ack | 08:53 |
| *** tosky has joined #oooq | 08:59 | |
| zbr | marios: chandankumar: we no longer need to install ansible inside venv for molecule, we could use system ansible. | 09:01 |
| zbr | this means we can add ansible to bindep, and also speedup install speed. | 09:02 |
| zbr | what do you think? | 09:02 |
| marios | zbr: i mean it shouldn't cost us something... we already have ansible ... so why not? | 09:04 |
| marios | zbr: but if it is part of the fix for +bug/1900033 i would rather we fix something quick first to unblock gate | 09:04 |
| marios | zbr: and then do this | 09:04 |
| zbr | we can still install it inside venv if we want a different version than the system one. | 09:04 |
| zbr | sure, that was unrelated to the fix. | 09:04 |
| *** jbadiapa has quit IRC | 09:05 | |
| *** jbadiapa has joined #oooq | 09:06 | |
| chandankumar | https://48f2879a4f28ca42458a-5e9e014bbb914c03e8d5277ea7a5cf3c.ssl.cf5.rackcdn.com/758502/3/check/openstack-tox-molecule/58c9f73/job-output.txt still failing | 09:12 |
| chandankumar | if we just install ansible http://paste.openstack.org/show/799108/ we get this | 09:12 |
| *** jtomasek has quit IRC | 09:13 | |
| *** dsneddon has quit IRC | 09:50 | |
| *** udesale_ has joined #oooq | 09:55 | |
| *** udesale__ has quit IRC | 09:58 | |
| *** ysandeep|afk is now known as ysandeep | 10:03 | |
| chandankumar | zbr: marios https://review.opendev.org/#/c/758529/ it works | 10:06 |
| chandankumar | same need on tq side | 10:06 |
| chandankumar | but we need to wait for 3 hrs to get it merged | 10:07 |
| marios | chandankumar: ship it | 10:09 |
| chandankumar | let me do the same for tq patch | 10:09 |
| marios | chandankumar: ++ | 10:09 |
| arxcruz | chandankumar: https://review.opendev.org/#/c/758547/ | 10:18 |
| arxcruz | the --deployment | 10:18 |
| chandankumar | arxcruz: will take a look | 10:23 |
| *** chandankumar has quit IRC | 10:25 | |
| *** chandan_kumar has joined #oooq | 10:27 | |
| chandan_kumar | arxcruz: Hello | 10:28 |
| chandan_kumar | arxcruz: during quay pull for which namespace you are seeing rate limit? | 10:28 |
| chandan_kumar | marios: https://review.opendev.org/#/c/758502/4 also good to go | 10:36 |
| zbr | chandan_kumar: i should not have touched ronelle fix, it seems there is more work to fix it (other bugs). | 10:39 |
| zbr | i raised https://review.opendev.org/758502 as temporary until i fix the rest | 10:39 |
| chandan_kumar | zbr: great thanks :-) | 10:40 |
| chandan_kumar | marios: few comments https://review.opendev.org/#/c/755472/13 rest looks good | 10:40 |
| *** Tengu has quit IRC | 10:41 | |
| arxcruz | chandan_kumar: the limit is related to the username | 10:42 |
| arxcruz | tripleoci not the namespace | 10:42 |
| arxcruz | if you do a podman login it will reach the limit as well | 10:42 |
| chandan_kumar | arxcruz: https://quay.io/repository/tripleoci/registry | 10:42 |
| chandan_kumar | it is an username as well as namespace | 10:42 |
| chandan_kumar | let me try logging and see | 10:43 |
| marios | chandan_kumar: thanks | 10:43 |
| marios | chandan_kumar: will update in a bit | 10:43 |
| chandan_kumar | marios: put this as a depends-on https://review.opendev.org/758502 on tq patch | 10:44 |
| chandan_kumar | to speed up the ci | 10:44 |
| marios | chandan_kumar: k | 10:46 |
| arxcruz | soniya29|ruck: https://review.opendev.org/#/c/758430/ to fix https://bugs.launchpad.net/tripleo/+bug/1899980 | 10:49 |
| openstack | Launchpad bug 1899980 in tripleo "tempest.scenario.test_network_basic_ops.TestNetworkBasicOps.test_port_security_macspoofing_port fails in stein inspite of test moved in skiplist" [Critical,In progress] - Assigned to Arx Cruz (arxcruz) | 10:49 |
| chandan_kumar | arxcruz: http://paste.openstack.org/show/799112/ | 10:51 |
| chandan_kumar | arxcruz: any more steps I am missing? | 10:52 |
| *** Tengu has joined #oooq | 10:53 | |
| chandan_kumar | marios: bhagyashris soniya29|ruck zbr arxcruz I am stepping out for an 1 hr or 2 | 11:05 |
| marios | chandan_kumar: ack | 11:06 |
| soniya29|ruck | chandan_kumar, ack | 11:06 |
| arxcruz | chandan_kumar: yup, that's it, i was able to login right now too | 11:10 |
| arxcruz | let me try to run the push-quay | 11:11 |
| arxcruz | chandan_kumar: try to login again | 11:11 |
| arxcruz | ack | 11:16 |
| soniya29|ruck | arxcruz, thanks | 11:19 |
| soniya29|ruck | marios, weshay, rlandy|rover, I am going out for half and hour | 11:38 |
| marios | soniya29|ruck: ack | 11:38 |
| bhagyashris | akahat, arxcruz marios ysandeep weshay sshnaidm rlandy|rover pojadhav soniya29|ruck please review this patches https://review.rdoproject.org/r/#/c/30328/ , https://review.rdoproject.org/r/#/c/30353/ https://review.rdoproject.org/r/#/q/topic:pipeline-renaming i have send an email for the same. Thank you :) | 12:01 |
| marios | bhagyashris: will do | 12:02 |
| bhagyashris | marios, thanks :) | 12:02 |
| ysandeep | bhagyashris: will do o/ | 12:03 |
| bhagyashris | we need every ones vote on above patches please review ^ thanks :) | 12:03 |
| *** rlandy has joined #oooq | 12:03 | |
| *** rlandy is now known as rlandy|rover | 12:03 | |
| rlandy|rover | soniya29|ruck: hey | 12:05 |
| rlandy|rover | what's with all the gate failures? | 12:06 |
| rlandy|rover | weshay: hey - morning | 12:06 |
| *** rf0lc0 has joined #oooq | 12:06 | |
| weshay | morning | 12:08 |
| rlandy|rover | weshay: hey - just going through the gate failures | 12:15 |
| rlandy|rover | 38 of them :( | 12:15 |
| rlandy|rover | not sure if zuul was drunk this morning or if we have a real issue yet | 12:15 |
| rlandy|rover | on the upside, the gate should be quiet today | 12:16 |
| weshay | rlandy|rover, did your changes merge for molecule? | 12:16 |
| weshay | it's just us today :)) | 12:16 |
| rlandy|rover | weshay: ack - that is passing now | 12:16 |
| weshay | you can walk around the irc chat room naked :) | 12:16 |
| rlandy|rover | lol | 12:16 |
| rlandy|rover | no this one | 12:16 |
| rlandy|rover | we're rucking and rovering here | 12:17 |
| rlandy|rover | zbr: ping re: openstack-tox-molecule tests | 12:17 |
| rlandy|rover | https://bugs.launchpad.net/tripleo/+bug/1900033 | 12:17 |
| openstack | Launchpad bug 1900033 in tripleo "openstack-tox-molecule jobs are failing with "ImportError: cannot import name 'ansible_version'"" [Critical,In progress] - Assigned to Sorin Sbarnea (ssbarnea) | 12:17 |
| weshay | bhagyashris, I need to see the trigger times for https://review.rdoproject.org/r/#/c/30328/ | 12:18 |
| weshay | oh I see I think | 12:19 |
| zbr | rlandy|rover: yes. | 12:19 |
| rlandy|rover | zbr: hey - I guess that was a temp change - what's the real action there? | 12:19 |
| zbr | rlandy|rover: i am working on it since this early morning, fixed one bug, found another one. is up to me to fix. | 12:20 |
| weshay | thanks zbr | 12:20 |
| rlandy|rover | zbr: it's tq and tqe - thanks | 12:21 |
| weshay | bhagyashris, is there a patch that creates integration-stable6 | 12:21 |
| bhagyashris | weshay, yes here https://review.rdoproject.org/r/#/c/30322/7/zuul.d/upstream.yaml@213 | 12:23 |
| bhagyashris | here ^ we are triggering | 12:23 |
| weshay | bhagyashris, thanks | 12:23 |
| weshay | zbr, let me know if you see anything stupid here https://review.opendev.org/#/c/758453/4/install-deps.sh | 12:23 |
| weshay | my worst patch ever | 12:23 |
| bhagyashris | weshay, and here i changes the integration pipeline name to stable6 https://review.rdoproject.org/r/#/c/30353/2/zuul.d/integration-pipeline-queens.yaml | 12:24 |
| weshay | bhagyashris++ | 12:26 |
| weshay | zbr, nice job advocating for change | 12:26 |
| marios | rf0lc0: * https://review.opendev.org/#/c/758502/2 * https://review.opendev.org/#/c/758136/1 | 12:32 |
| marios | rf0lc0: please add vote (will update in bit as discussed) https://review.opendev.org/#/c/755472/ | 12:33 |
| rlandy|rover | weshay: can we rekick promoter? master should promote now | 12:33 |
| weshay | rlandy|rover, k | 12:33 |
| rlandy|rover | https://review.rdoproject.org/zuul/builds?pipeline=openstack-periodic-integration-main | 12:33 |
| weshay | arxcruz, zbr https://review.opendev.org/#/c/758441/ | 12:34 |
| arxcruz | ack | 12:35 |
| rlandy|rover | weshay: thanks | 12:36 |
| rlandy|rover | weshay: and ussuri should be ready to promote after that | 12:36 |
| rf0lc0 | marios, https://review.opendev.org/758214 | 12:37 |
| pojadhav | bhagyashris, ack | 12:37 |
| marios | rf0lc0: https://review.opendev.org/#/c/753708/26/roles/extras-common/defaults/main.yml | 12:38 |
| soniya29|ruck | rlandy|rover, many of gate failures are due to failure of non-voting jobs | 12:39 |
| weshay | soniya29|ruck, they don't fail the job.. when they are non-voting | 12:39 |
| weshay | query on the count is.. | 12:40 |
| weshay | SELECT count("result_num") FROM "build" WHERE ("pipeline" = 'gate' AND "passed" = 'False' AND "voting" = 'True' AND "result" != 'NODE_FAILURE' AND "result" != 'RETRY') AND $timeFilter | 12:40 |
| rlandy|rover | weshay: soniya29|ruck: I rechecked the failed jobs | 12:40 |
| weshay | thanks | 12:40 |
| rlandy|rover | the content provider changes the numbers a bit | 12:40 |
| rlandy|rover | weshay: soniya29|ruck: ^^ | 12:40 |
| weshay | master is promoting | 12:40 |
| weshay | ya | 12:40 |
| rlandy|rover | if the content provider job fails | 12:40 |
| rlandy|rover | which a few did, | 12:41 |
| soniya29|ruck | weshay, for example: https://review.opendev.org/#/c/757121/ | 12:41 |
| rlandy|rover | the whole line registers as failed | 12:41 |
| rlandy|rover | I just learned that | 12:41 |
| weshay | soniya29|ruck, that merged | 12:41 |
| rlandy|rover | so skipped gets counted as a failure | 12:41 |
| rlandy|rover | something to kepe in mind | 12:41 |
| weshay | rlandy|rover, aye | 12:41 |
| weshay | skipped = failed | 12:41 |
| ysandeep | bhagyashris, small comment on https://review.rdoproject.org/r/#/c/30353/ for clarity | 12:41 |
| zbr | weshay: rlandy|rover: i see a huge number of jobs running when is not needed on https://review.opendev.org/#/c/758502/ | 12:42 |
| weshay | soniya29|ruck, rlandy|rover also .. note the voting column http://dashboard-ci.tripleo.org/d/Z4vLSmOGk/cockpit?orgId=1&fullscreen&panelId=61 | 12:42 |
| weshay | should help see the signal vs. noise | 12:42 |
| rlandy|rover | yep | 12:42 |
| zbr | touching molecule-requirements.txt should only trigger linters and molecule jobs, not 10x heavy jobs. | 12:42 |
| rf0lc0 | marios, I forgot the first condition in my patch | 12:42 |
| rf0lc0 | {% if job is defined and | 12:42 |
| rf0lc0 | job.build_container_images is defined and | 12:42 |
| rf0lc0 | job.build_container_images|default(false)|bool or | 12:42 |
| rf0lc0 | build_container_images|default(false)|bool -%} | 12:42 |
| rf0lc0 | 127.0.0.1:5001 | 12:42 |
| rlandy|rover | zbr: we can change the templates triggering | 12:43 |
| rf0lc0 | marios, the local one | 12:43 |
| rf0lc0 | marios, but if consumer_job then its not local build | 12:43 |
| weshay | rlandy|rover, let's chat re: https://code.engineering.redhat.com/gerrit/#/c/215284/ later today | 12:43 |
| zbr | yep, but i am not sure where to put the filters, for example tripleo-ci-centos-8-content-provider has no filters at all. | 12:44 |
| rlandy|rover | weshay: ack - I just wanted to take a shot at deciphering the introspection failure on ci.centos this morning | 12:44 |
| rlandy|rover | let's chat re: downstream in a bit | 12:44 |
| weshay | chandan_kumar, please xfer rdoci-hp-01.v100.rdoci.lab.eng.rdu2.redhat.com | 12:47 |
| weshay | to ysandeep in beaker | 12:47 |
| rlandy|rover | weshay: left comment on https://code.engineering.redhat.com/gerrit/#/c/215284/1 - for later discussion | 12:47 |
| weshay | k | 12:47 |
| ysandeep | chandan_kumar: hi, you might have also received an auto request from beaker to loan it to me.. appreciate if you can please loan it to me :) | 12:49 |
| weshay | soniya29|ruck, rlandy|rover the other thing re: gate failures.. is to add to sova in ansible-role-collect-logs | 12:50 |
| weshay | to automatically triage for us | 12:51 |
| soniya29|ruck | weshay okay | 12:51 |
| * rlandy|rover needs to look into that - not sure how sova triage works | 12:52 | |
| marios | rf0lc0: yes the if consumer_job conditional is fine we have used that in most places now | 12:53 |
| marios | rf0lc0: i mean where we are setting the registry_ip_address_branch[release] | 12:53 |
| weshay | https://github.com/openstack/ansible-role-collect-logs/blob/master/vars/sova-patterns.yml | 12:54 |
| weshay | soniya29|ruck, ^ | 12:54 |
| rf0lc0 | marios, registry_ip_address_branch[release] will always exist ? | 12:54 |
| rf0lc0 | marios, or in master registry_ip_address_branch[release] is not defined ? | 12:54 |
| weshay | 42 tripleo patches merged in the last 24hrs.. so about average | 12:54 |
| marios | rf0lc0: if it is a consumer job then yes and if it doesn't then it should fail cos something is wrong | 12:54 |
| marios | rf0lc0: always defined | 12:55 |
| marios | rf0lc0: in master jobs there will be a registry_ip_address_branch['master'] | 12:55 |
| soniya29|ruck | weshay: yeah | 12:55 |
| zbr | i have the impression that irrelevant-files are not inherited | 12:55 |
| marios | rf0lc0: https://opendev.org/openstack/tripleo-ci/src/commit/03cedc4fc0a1f4393916177e274d5a1242b525a8/playbooks/tripleo-ci/run-provider.yml#L23-L64 | 12:56 |
| marios | rf0lc0: it is set without condition for all branches ^ | 12:56 |
| rf0lc0 | marios, ok so I am wondering if the local build condition is missing in my patch: https://review.opendev.org/#/c/758214/4/roles/extras-common/defaults/main.yml | 12:56 |
| marios | rf0lc0: lgtm ? it wasn't there before so... and docker_registry_host is being set in the release files already... honestly this one is sneaky i don't like that we are setting it there... but anyway | 12:58 |
| rf0lc0 | marios, ok with the releases change, the default becomes useless | 12:58 |
| rf0lc0 | marios, thanks | 12:59 |
| marios | rf0lc0: synergy | 13:00 |
| zbr | who can help me with https://review.opendev.org/#/c/758561/ ? | 13:01 |
| zbr | that upgrade job runs regardless the fact that this exact filename is part of irrelevant. | 13:01 |
| bhagyashris | ysandeep, replied | 13:02 |
| rf0lc0 | marios, :) | 13:04 |
| zbr | not sure who implemented sa_upgrade_ignore but it had a really bad effect on irrelevant-files | 13:15 |
| zbr | forcing us to run upgrade jobs on irrelevant changes | 13:16 |
| rlandy|rover | weshay: somewhere did we increase the requirements for a node in image build or something like that? | 13:19 |
| * rlandy|rover thinks the ci.cemtos nodes are not meeting criteria | 13:19 | |
| rlandy|rover | from ussuri up | 13:19 |
| rlandy|rover | ie: failing introspection | 13:19 |
| rlandy|rover | changed the swap maybe | 13:20 |
| weshay | rlandy|rover, ya.. /me cleans my test box | 13:20 |
| weshay | arxcruz, did you want to sync today or skip | 13:20 |
| rlandy|rover | disk image building? | 13:20 |
| arxcruz | weshay: up to you | 13:20 |
| rlandy|rover | https://opendev.org/openstack/tripleo-ci/commit/ec5be10bed1773bc9abc6b3cc6d5d1170ecb843c | 13:23 |
| rlandy|rover | something like that | 13:23 |
| rlandy|rover | bigger requirements | 13:23 |
| weshay | k.. running locally | 13:27 |
| *** jtomasek has joined #oooq | 13:29 | |
| rlandy|rover | weshay: so maybe this ... https://github.com/openstack/tripleo-ci/blame/master/roles/prepare-node/tasks/main.yaml | 13:31 |
| rlandy|rover | although that should hit all releases | 13:31 |
| rlandy|rover | ah | 13:31 |
| rlandy|rover | only those using this role | 13:31 |
| weshay | hrm.. http://pastebin.test.redhat.com/911027 | 13:33 |
| arxcruz | weshay: skipped then ? | 13:34 |
| rlandy|rover | weird | 13:35 |
| rlandy|rover | you should get images | 13:35 |
| rlandy|rover | chandan_kumar: hey - you around? | 13:35 |
| rlandy|rover | need to consult your image build expertise | 13:36 |
| weshay | arxcruz, let's have a quicky | 13:36 |
| *** jtomasek has quit IRC | 13:40 | |
| weshay | rlandy|rover, images are fine.. removing cache | 13:42 |
| *** udesale__ has joined #oooq | 13:44 | |
| *** udesale_ has quit IRC | 13:46 | |
| rlandy|rover | memory"8192" | 13:49 |
| rlandy|rover | disk"50" | 13:49 |
| rlandy|rover | weshay: ^^ may be too small now | 13:49 |
| weshay | https://images.rdoproject.org/centos8/ussuri/rdo_trunk/ | 14:05 |
| weshay | "cmd": [ | 14:06 |
| weshay | "curl", | 14:06 |
| weshay | "-skfL", | 14:06 |
| weshay | "https://images.rdoproject.org/centos8/ussuri/rdo_trunk/1b15401d052f2d9df13f56108e854d13dbb0f8ff_8d16be20", | 14:06 |
| weshay | image is not there | 14:06 |
| zbr | rlandy|rover: weshay: chicken-egg conundrum with https://review.opendev.org/#/c/758568/ and https://review.opendev.org/#/c/758561/ | 14:09 |
| rlandy|rover | weshay: the images are named with agg hash https://images.rdoproject.org/centos8/ussuri/rdo_trunk/ | 14:09 |
| zbr | molecule fix runsin few minutes,... if we first fix the incorrect irrelevant-files. | 14:09 |
| zbr | but that fix also triggers molecule job which is broken now. | 14:09 |
| rlandy|rover | get the fix in | 14:10 |
| rlandy|rover | then fix job triggers imo | 14:10 |
| rlandy|rover | imho :) | 14:10 |
| zbr | i have an idea,... | 14:10 |
| rlandy|rover | weshay: no worries about trying to reproduce | 14:11 |
| rlandy|rover | I think it's a matter of the node size | 14:11 |
| rlandy|rover | so a local machine may not reproduce | 14:11 |
| weshay | rlandy|rover, if that's the case.. we wavie | 14:11 |
| weshay | wave | 14:11 |
| weshay | waive | 14:11 |
| rlandy|rover | weshay: I am trying to find where there was a change in the image requirement | 14:13 |
| rlandy|rover | there must have been | 14:13 |
| weshay | rlandy|rover, last known good https://artifacts.ci.centos.org/rdo/jenkins-tripleo-quickstart-promote-ussuri-current-tripleo-delorean-minimal-89/undercloud/var/log/containers/ironic-inspector/ironic-inspector.log | 14:16 |
| weshay | Reading S.M.A.R.T information on /dev/vda\\nread_smart: Reading S.M.A.R.T information on /dev/vda with -d ata\\nread_smart: no device /dev/vda\\nTraceback (most recent call last):\\n File "/usr/bin/hardware-detect", line 10, in <module>\\n sys.exit(main())\\n File "/usr/lib/python3.6/site-packages/hardware/detect.py", line 731, in main\\n if not detect_system(hrdw):\\n File "/usr/lib/python3.6/site-packages/hardware/detect.py", line | 14:17 |
| weshay | 250, in detect_system\\n mobo_id = _get_value(hw_lst, \\\'system\\\', \\\'motherboard\\\', \\\'serial\\\')\\n File "/usr/lib/python3.6/site-packages/hardware/detect.py", line 194, in _get_value\\n if i[0:3] == vect:\\nTypeError: \\\'bool\\\' object is not subscriptable\\n\''} api_continue /usr/lib/python3.6/site-packages/ironic_inspector/main.py:297 | 14:17 |
| weshay | progress running ./quickstart.sh --bootstrap --tags all -w /var/tmp/working_dir/ --config ~/tripleo-quickstart/config/general_config/minimal.yml --release centosci/ussuri-current-tripleo -e dlrn_hash=bf903d956d77a52a01c92bdf14ba47d5 127.0.0.2 | 14:19 |
| weshay | it's downloading the image now | 14:19 |
| rlandy|rover | https://artifacts.ci.centos.org/rdo/jenkins-tripleo-quickstart-promote-ussuri-current-tripleo-delorean-minimal-104/undercloud/var/log/containers/ironic/ironic-conductor.log | 14:20 |
| rlandy|rover | it can boot the box | 14:20 |
| *** ysandeep is now known as ysandeep|away | 14:21 | |
| ysandeep|away | rlandy|rover: hey o/ if 16.2 line passes please promote , and wish you a happy weekend :) | 14:23 |
| rlandy|rover | sure | 14:23 |
| ysandeep|away | thank you :) | 14:23 |
| weshay | rlandy|rover, does seem like a real boog | 14:25 |
| rlandy|rover | weshay: I'd like to try pass a var to that job ... | 14:25 |
| weshay | what var? | 14:26 |
| rlandy|rover | configure_swap_size | 14:26 |
| * rlandy|rover checks what it is now | 14:26 | |
| * zbr thinks that the job-mess in tripleo-ci will hit us harder and harder | 14:28 | |
| weshay | ya.. makes sens | 14:28 |
| weshay | zbr, what mess? | 14:28 |
| weshay | the molecule issue? | 14:28 |
| weshay | ah.. from your patches.. seems so | 14:29 |
| marios | proposed Tripleo meetup @PTG schedule http://lists.openstack.org/pipermail/openstack-discuss/2020-October/018039.html | 14:31 |
| chandan_kumar | rlandy|rover: how can i help on image | 14:36 |
| chandan_kumar | weshay: rlandy|rover please +w it https://review.opendev.org/#/c/758529/ | 14:37 |
| rlandy|rover | weshay: look familiar: https://bugs.launchpad.net/tripleo/+bug/1818554? | 14:38 |
| openstack | Launchpad bug 1818554 in tripleo "Phase 1 tripleo-quickstart-promote-rocky-rdo_trunk-minimal job fail on master and rocky. Introspection failed to run hardware-detect utility" [Critical,Fix released] - Assigned to Gabriele Cerami (gcerami) | 14:38 |
| rlandy|rover | chandan_kumar: zbr is busy with that work | 14:38 |
| rlandy|rover | https://review.opendev.org/#/c/758529/ | 14:38 |
| chandan_kumar | rlandy|rover: we are merging initial workaround to unblock the gates for Tq toci and tqe | 14:39 |
| rlandy|rover | k he undid my one patch | 14:39 |
| rlandy|rover | chandan_kumar: wrt image, did we increase the requirements for master/ussuri | 14:40 |
| rlandy|rover | on memory or disk or something? | 14:40 |
| chandan_kumar | rlandy|rover: nope | 14:40 |
| rlandy|rover | k - was a thought | 14:40 |
| rlandy|rover | weshay: maybe we reopen https://bugs.launchpad.net/tripleo/+bug/1818554? | 14:41 |
| openstack | Launchpad bug 1818554 in tripleo "Phase 1 tripleo-quickstart-promote-rocky-rdo_trunk-minimal job fail on master and rocky. Introspection failed to run hardware-detect utility" [Critical,Fix released] - Assigned to Gabriele Cerami (gcerami) | 14:41 |
| rlandy|rover | https://artifacts.ci.centos.org/rdo/jenkins-tripleo-quickstart-promote-ussuri-current-tripleo-delorean-minimal-90/undercloud/var/log/containers/ironic-inspector/ironic-inspector.log | 14:41 |
| rlandy|rover | same error | 14:41 |
| rlandy|rover | https://github.com/redhat-cip/hardware/commits/master | 14:44 |
| rlandy|rover | real bug here | 14:44 |
| weshay | rlandy|rover, not sure if that trace matches | 15:04 |
| weshay | unless I looked too quickly | 15:04 |
| rlandy|rover | weshay: logging a new bug | 15:04 |
| rlandy|rover | referencing the older one | 15:05 |
| rlandy|rover | let's get someone from baremetal to confirm | 15:05 |
| rlandy|rover | possible change: | 15:05 |
| rlandy|rover | https://github.com/redhat-cip/hardware/commit/d14a46fd8d03b079a350ca47d43d97c928ef80a5 | 15:05 |
| rlandy|rover | weshay: https://bugs.launchpad.net/tripleo/+bug/1900158 | 15:11 |
| openstack | Launchpad bug 1900158 in tripleo "ci.centos jobs in master and ussuri are failing introspection - failed to run hardware-detect utility" [Critical,Triaged] | 15:11 |
| rlandy|rover | we're done here | 15:11 |
| rlandy|rover | will see what traction we can get on monday's CIX | 15:11 |
| rlandy|rover | weshay: check this out ... 17 ... | 15:19 |
| rlandy|rover | 2020-10-15 17:57:31.975708 | fa163e14-add8-9def-cd93-000000000229 | FATAL | Deploy release version package | standalone | error={"msg": "The conditional check '(tripleo_bootstrao_release_version_package |list | length) > 0' failed. The error was: error while evaluating conditional ((tripleo_bootstrao_release_version_package |list | length) > 0): 'tripleo_bootstrao_release_version_package' is undefined\n\nThe error | 15:19 |
| rlandy|rover | appears to be in '/usr/share/ansible/roles/tripleo_bootstrap/tasks/main.yml': line 36, column 3, but may\nbe elsewhere in the file depending on the exact syntax problem.\n\nThe offending line appears to be:\n\n# Currently only supported on RHEL as tripleo does not have a version package\n- name: Deploy release version package\n ^ here\n"} | 15:19 |
| rlandy|rover | 'tripleo_bootstrao_release_version_package | 15:19 |
| rlandy|rover | bootstrap | 15:19 |
| rlandy|rover | how was that caught for the first time in the 17 integration line??? | 15:19 |
| rlandy|rover | we promoted components :) | 15:19 |
| weshay | rlandy|rover, 17 integration? | 15:20 |
| weshay | rlandy|rover, would have to look at packages :)) | 15:20 |
| weshay | rlandy|rover, which job? | 15:21 |
| rlandy|rover | weshay: ack ... https://sf.hosted.upshift.rdu2.redhat.com/logs/openstack-periodic-rhos-17/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-rhel-8-standalone-rhos-17/ba72d8c/logs/undercloud/home/zuul/standalone_deploy.log | 15:22 |
| rlandy|rover | all of them actually | 15:22 |
| rlandy|rover | https://sf.hosted.upshift.rdu2.redhat.com/zuul/t/tripleo-ci-internal/builds?pipeline=openstack-periodic-rhos-17 | 15:22 |
| rlandy|rover | https://sf.hosted.upshift.rdu2.redhat.com/logs/openstack-periodic-rhos-17/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-rhel-8-multinode-1ctlr-featureset010-rhos-17/c8ff138/logs/undercloud/home/zuul/undercloud_install.log | 15:23 |
| rlandy|rover | same there | 15:23 |
| zbr | rlandy|rover: weshay : +W https://review.opendev.org/#/c/758561/ | 15:23 |
| zbr | and https://review.opendev.org/#/c/758502/ | 15:24 |
| weshay | https://opendev.org/openstack/tripleo-ansible/src/branch/master/tripleo_ansible/roles/tripleo_bootstrap/vars/redhat-8.yml | 15:24 |
| rlandy|rover | weshay: https://opendev.org/openstack/tripleo-ansible/src/branch/master/tripleo_ansible/roles/tripleo_bootstrap/tasks/main.yml#L43 | 15:24 |
| rlandy|rover | yep | 15:24 |
| rlandy|rover | how did that get into the code-base?? | 15:25 |
| * rlandy|rover fuxes | 15:25 | |
| rlandy|rover | fixes | 15:25 |
| rlandy|rover | a month ago | 15:25 |
| rlandy|rover | ah - only triggers on rhe; | 15:26 |
| rlandy|rover | rhel | 15:26 |
| weshay | naw | 15:28 |
| *** holser has joined #oooq | 15:28 | |
| weshay | rlandy|rover, ah maybe.. don't see https://1befaf2f8cf3f2b924d7-2e7261ce7c38fece73c783229912a436.ssl.cf1.rackcdn.com/757649/5/check/tripleo-ci-centos-8-standalone/0643284/logs/undercloud/home/zuul/standalone_deploy.log | 15:30 |
| weshay | rlandy|rover, 2020-10-16 11:40:18.878147 | fa163e0a-c70c-bb2e-1c99-000000000228 | OK | Gather variables for each operating system | standalone | item=/usr/share/ansible/roles/tripleo_bootstrap/vars/redhat-8.yml | 15:30 |
| weshay | https://1befaf2f8cf3f2b924d7-2e7261ce7c38fece73c783229912a436.ssl.cf1.rackcdn.com/757649/5/check/tripleo-ci-centos-8-standalone/0643284/logs/undercloud/home/zuul/ansible.log | 15:31 |
| rlandy|rover | (ansible_distribution | lower) == 'redhat' | 15:31 |
| weshay | rlandy|rover, centos resolvs to that | 15:34 |
| weshay | doesn't it? | 15:34 |
| rlandy|rover | - ansible_distribution in ['RedHat', 'CentOS'] | 15:35 |
| weshay | aye | 15:35 |
| weshay | ur right | 15:35 |
| rlandy|rover | we usually use both if we want both | 15:35 |
| weshay | ansible_distribution: CentOS | 15:35 |
| weshay | rlandy|rover, ya.. /me wonders why that is just for RH | 15:35 |
| rlandy|rover | idk | 15:35 |
| rlandy|rover | says only supported on rhel | 15:35 |
| rlandy|rover | idk why | 15:35 |
| rlandy|rover | Currently only supported on RHEL as tripleo does not have a version package | 15:36 |
| weshay | k | 15:36 |
| *** marios is now known as marios|out | 16:16 | |
| *** marios|out has quit IRC | 16:18 | |
| weshay | rlandy|rover, failing in the same way | 16:28 |
| weshay | https://termbin.com/75hp | 16:28 |
| weshay | and termbin rocks | 16:29 |
| rlandy|rover | k - we have a legit bug | 16:29 |
| weshay | rlandy|rover, k.. and a live env | 16:29 |
| weshay | mention that in the bug please | 16:29 |
| rlandy|rover | will do | 16:36 |
| rf0lc0 | weshay, rlandy|rover looks like this molecule job in tripleo-ipa is broken, https://review.opendev.org/#/c/757104/... I don't know how to fix this | 16:43 |
| rf0lc0 | I fixed the gpg error here https://review.opendev.org/#/c/758222 | 16:44 |
| rf0lc0 | there is a legit error on molecule converge though | 16:45 |
| rf0lc0 | TASK [tripleo_ipa_setup : add Nova Host Manager role] ************************** | 16:45 |
| rf0lc0 | fatal: [centos7]: FAILED! => changed=false | 16:45 |
| rf0lc0 | msg: 'login: Request failed: <urlopen error [Errno -2] Name or service not known>' | 16:45 |
| weshay | rf0lc0, https://review.opendev.org/#/c/758529/7/test-requirements.txt | 16:46 |
| weshay | try something like that | 16:46 |
| rlandy|rover | sorry - lunch - reading back | 16:46 |
| rlandy|rover | rf0lc0: ask lance | 16:47 |
| rlandy|rover | weshay: don't want to jinx it (poo poo poo) but downloading images on 16.2 is not exactly the ultimate savior | 16:48 |
| weshay | ya.. I think the build was just 45min | 16:49 |
| weshay | where the diff was like 2hours | 16:49 |
| rlandy|rover | weshay: so you can feel unstupid now | 16:49 |
| weshay | so.. we just need to make sure the jobs are equiv though | 16:49 |
| rlandy|rover | correct | 16:49 |
| rlandy|rover | added live env comment to introspection bug | 16:55 |
| rf0lc0 | rlandy|rover, thx | 17:03 |
| *** udesale__ has quit IRC | 17:09 | |
| weshay | rf0lc0, need reviews? | 17:35 |
| rf0lc0 | weshay, mostly | 17:35 |
| rf0lc0 | yes | 17:35 |
| weshay | new-ci-job? | 17:35 |
| rf0lc0 | https://review.opendev.org/#/q/status:open+topic:new-ci-job | 17:36 |
| rlandy|rover | weshay: k - looking at the rhos-13 work now | 17:40 |
| rlandy|rover | weshay: ysandeep|away: promoting 16.2 | 17:45 |
| weshay | rlandy|rover, 10-16 looks good | 17:46 |
| rlandy|rover | yep - promotion in progress | 17:46 |
| rlandy|rover | weshay: you wanted to chat about rhos-13?? | 17:51 |
| weshay | sure | 17:52 |
| weshay | https://meet.google.com/qtb-eosu-iep?authuser=1 | 17:52 |
| *** sanjayu__ has joined #oooq | 17:53 | |
| weshay | rf0lc0, k.. only saw 3 ready for +2 | 17:53 |
| *** jmasud has quit IRC | 17:53 | |
| weshay | rlandy|rover, ^ | 17:54 |
| weshay | ugh this is dying https://review.opendev.org/#/c/755472/ | 17:54 |
| weshay | rlandy|rover, https://code.engineering.redhat.com/gerrit/#/c/215284/1/playbooks/base/pre.yaml | 17:55 |
| *** sanjayu_ has quit IRC | 17:55 | |
| rf0lc0 | weshay, your forgot this one https://review.opendev.org/#/c/757917/ | 17:56 |
| rf0lc0 | weshay, the others are rechecking | 17:57 |
| rf0lc0 | or waiting molecule fix to recheck | 17:57 |
| *** sanjayu__ has quit IRC | 18:08 | |
| weshay | rlandy|rover, http://pastebin.test.redhat.com/911069 | 18:37 |
| *** jbadiapa has quit IRC | 19:03 | |
| *** holser has quit IRC | 19:21 | |
| *** holser has joined #oooq | 19:22 | |
| *** tosky has quit IRC | 19:37 | |
| *** tosky has joined #oooq | 19:39 | |
| *** yolanda has quit IRC | 20:24 | |
| *** yolanda has joined #oooq | 20:24 | |
| *** holser has quit IRC | 20:48 | |
| *** holser has joined #oooq | 20:49 | |
| *** rlandy|rover has quit IRC | 21:02 | |
| *** rf0lc0 has quit IRC | 21:20 | |
| *** tosky has quit IRC | 22:29 | |
| *** jmasud has joined #oooq | 23:57 | |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!