14:02:49 <amoralej> #startmeeting RDO meeting - 2020-02-19
14:02:50 <openstack> Meeting started Wed Feb 19 14:02:49 2020 UTC and is due to finish in 60 minutes.  The chair is amoralej. Information about MeetBot at http://wiki.debian.org/MeetBot.
14:02:51 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
14:02:53 <openstack> The meeting name has been set to 'rdo_meeting___2020_02_19'
14:03:04 <amoralej> #topic roll call
14:03:11 <ykarel> o/
14:03:17 <mjturek> o/
14:03:19 <jcapitao> o/
14:03:23 <baha> o/
14:03:24 <leanderthal> o/
14:03:28 <amoralej> #chair ykarel mjturek jcapitao baha leanderthal
14:03:29 <rh-jelabarre> o/
14:03:30 <openstack> Current chairs: amoralej baha jcapitao leanderthal mjturek ykarel
14:03:37 <amoralej> #chair rh-jelabarre
14:03:39 <openstack> Current chairs: amoralej baha jcapitao leanderthal mjturek rh-jelabarre ykarel
14:03:50 <jpena> o/
14:04:08 <amoralej> #chair jpena
14:04:08 <openstack> Current chairs: amoralej baha jcapitao jpena leanderthal mjturek rh-jelabarre ykarel
14:04:28 <amoralej> let's start with first topic
14:04:31 <amoralej> #topic Cannot login as root from RDO jenkins slaves to cico nodes
14:04:46 <amoralej> #info Bug is open https://bugs.centos.org/view.php?id=17066
14:04:52 <amoralej> mjturek, baha ^
14:05:04 <mjturek> this seems to be moving along
14:05:19 <amoralej> mjturek, it's fixed already?
14:05:21 <mjturek> seems that the ssh key made it to jenkins users but not root
14:05:40 <mjturek> I don't think it's fixed yet, but ykarel has been telling centos how to handle
14:06:00 <mjturek> and we have a contact in centos helping
14:06:04 <amoralej> ykarel, so you know if they will add the key to root user also?
14:06:18 <ykarel> amoralej, they add keys only for root user
14:06:29 <ykarel> ^^ is they shared
14:06:51 <mjturek> ykarel I thought it was only for jenkins... am I misunderstanding?
14:06:52 <amoralej> i understand the opposite, it was in jenkins but not root
14:06:52 <ykarel> they just added keys in there db, i triggered https://ci.centos.org/job/tripleo-upstream-containers-build-master-ppc64le/2033/console to see if it's good now
14:07:21 <jpena> ykarel: when did they do it?
14:07:22 <ykarel> amoralej, mjturek i asked them if they add for jenkins, they said we add for root
14:07:27 <amoralej> ok
14:07:30 <mjturek> ahhh
14:07:34 <ykarel> jpena, when cico node get run
14:07:39 <amoralej> i see jome jobs passed already so ssh seems to be fine now
14:07:41 <mjturek> ykarel it looks like it's working
14:08:00 <ykarel> jpena, they add public key for .ssh/id_rsa to root user of the node
14:08:02 <jpena> ykarel: I mean, when were the keys added to the db?
14:08:23 <ykarel> jpena, ohh, they add to db after it's requested, like https://bugs.centos.org/view.php?id=17066
14:08:52 <amoralej> #info required ssh keys have been added to cico nodes deployment and ssh from jenkins slave looks fine now
14:09:03 <jpena> ykarel: ack. I was testing some minutes ago and couldn't make it work, so apparently I was unlucky
14:09:17 <mjturek> thanks ykarel :)
14:09:27 <rdogerrit> Merged rdo-jobs master: Add EPEL repo for CentOS 8 rpm-packaging jobs  https://review.rdoproject.org/r/25044
14:09:28 <ykarel> jpena, so in past when were the keys changed?
14:09:37 <ykarel> on slaves
14:09:46 <ykarel> apart from the recent ones
14:09:49 <amoralej> i don't remember last keys rotation
14:09:53 <amoralej> before this one
14:10:14 <jpena> same here, I think we never changed those
14:10:21 <ykarel> ohhk then it's fine, i was not sure how in past this used to be handled
14:10:38 <ykarel> so good to add this to knowledge base so we don't miss next time
14:10:56 <amoralej> yeap +1
14:11:04 <amoralej> is it documented somewhere in centos ci docs?
14:11:10 <amoralej> should be there, i think
14:11:36 <jpena> I have been unable to find any pointer in the centos ci documentation
14:12:15 <amoralej> ok
14:12:28 <amoralej> so i think we can give this issue as closed
14:12:29 <amoralej> good
14:12:45 <ykarel> so now job failed at other step
14:13:11 <mjturek> ykarel yeaaaah not sure what that is, but it's not ssh related at least
14:13:29 <ykarel> it's ssh related but for logserver Failed to connect to the host via ssh: Host key verification failed
14:13:44 <amoralej> first it complains about playbook syntax
14:13:51 <mjturek> ykarel yep I need to add the StrictHostKeyChecking=no for those
14:13:59 <amoralej> The error appears to be in '/home/jenkins/workspace/tripleo-upstream-containers-build-master-ppc64le/roles/build-containers/tasks/pre.yaml': line 21, column 17, but may
14:14:11 <amoralej> then it also fails to collect logs
14:14:25 <amoralej> but there seems to be some issue with role syntax
14:14:30 <mjturek> the ssh problem I know how to fix
14:14:51 <ykarel> ohhk i know this issue, i mean syntax issue
14:14:51 <mjturek> the syntax problem confuses me as we don't touch the pre playbook
14:15:00 <ykarel> let's take that offline
14:15:02 <amoralej> ERROR! the role 'ansible-role-container-registry' was not found
14:15:07 <amoralej> missing role
14:15:12 <mjturek> oh nice!
14:15:14 <mjturek> sounds good ykarel
14:15:22 <amoralej> ok, let's move on
14:15:39 <amoralej> #topic logserver does not appear to have folder for ppc64le containers-build job
14:16:06 <amoralej> my understanding is that it will be created first time collect-logs works fine
14:16:24 <baha> We've actually had collect-logs fire once into that logserver already, and the folder didn't pop up
14:16:32 <amoralej> mmmm
14:16:45 <rdogerrit> Chandan Kumar created config master: [container-login] Install podman for c8 and RHEl-8  https://review.rdoproject.org/r/25046
14:17:02 <amoralej> baha, you have logs of some success job?
14:17:10 <amoralej> i mean, success = uploading logs
14:17:11 <mjturek> baha but it's been failing with ssh since around the 12th are we sure about that?
14:17:26 <baha> The job wasn't successful, but it was trying to upload console text. I'd need a bit to track down the exact run number
14:17:42 <baha> But it got far enough into the upload to put out the expected URL, which only happens after it confirms connection
14:17:59 <mjturek> last time ssh worked https://ci.centos.org/job/tripleo-upstream-containers-build-master-ppc64le/2009/console
14:19:09 <baha> Ahh, you're right. I was looking at 2011, we just ended up in a weird state due to the other failures
14:19:12 <mjturek> actually it's 2010
14:19:24 <mjturek> baha gotcha
14:19:45 <baha> So yeah, entirely possible that folder will just appear once the job's more stable then.
14:19:54 <mjturek> we'll fix the ssh issue, and see after the next run
14:20:12 <amoralej> that job pushed to the old server
14:20:12 <baha> I remembered last time around we needed to manually create the folder, so I figured I'd bring it up. If that's not the case this time then it's a non-issue =)
14:20:43 <mjturek> amoralej: I think we're good here!
14:20:49 <ykarel> mjturek, need to add ansible-role-container-registry like https://github.com/rdo-infra/ci-config/blob/master/jenkins/jobs/tripleo-upstream-builders.yml#L20-L21
14:20:51 <amoralej> i'd say it's automatically created
14:20:57 <ykarel> needed because of https://review.opendev.org/#/c/707330/5/roles/build-containers/tasks/pre.yaml@20
14:20:59 <amoralej> jpena, you know if it needs to be pre-created?
14:21:06 <ykarel> can u post a patch for this
14:21:26 <mjturek> ykarel: thank you! will do!
14:21:26 <jpena> it shouldn't be pre-created, the log collecting playbook should take care of that
14:21:39 <rdogerrit> Chandan Kumar proposed rdo-jobs master: [WIP] Added CentOS-8 build images  https://review.rdoproject.org/r/24997
14:21:53 <amoralej> ack
14:21:59 <mjturek> jpena amoralej: sounds good, we'll fix the ssh issue and confirm with you guys once it shows up
14:22:08 <amoralej> mjturek, baha let us know if you still see issues
14:22:12 <baha> Will do! Thanks guys
14:22:15 <mjturek> will do
14:22:30 <rdogerrit> Merged config master: [container-login] Install podman for c8 and RHEl-8  https://review.rdoproject.org/r/25046
14:23:02 <amoralej> #info folder for ppc64le containers build job in logserver will be created on first successful collect-logs execution
14:23:14 <amoralej> ok, i think we can move to next one
14:23:29 <amoralej> #topic CentOS8 updates
14:23:59 <amoralej> #info Working on getting train ready in CBS - https://bugs.centos.org/view.php?id=17066
14:24:19 <amoralej> ykarel, you want to comment on this one?
14:24:41 <ykarel> mmm
14:24:46 <ykarel> i think i added trello link
14:24:51 <ykarel> let me check
14:25:08 <ykarel> sorry my mistake
14:25:12 <amoralej> #undo
14:25:12 <ykarel> correct link is https://trello.com/c/y4IfMNvu
14:25:13 <openstack> Removing item from minutes: #info Working on getting train ready in CBS - https://bugs.centos.org/view.php?id=17066
14:25:25 <amoralej> #info Working on getting train ready in CBS - https://trello.com/c/y4IfMNvu
14:25:44 <ykarel> now looks good
14:26:08 <ykarel> so update is we have started preparing for CentOS8 CBS builds for Train
14:26:32 <ykarel> Currently we are fixing jobs, once done we will start with CBS builds
14:27:01 <ykarel> till then hopefully we may also have some news from centos wrt to mirror setup
14:27:18 <ykarel> so the builds available on buildlogs and centos mirrors
14:27:24 <amoralej> ykarel, when you have a chance, review https://review.rdoproject.org/r/#/c/25041/ which is needed to rebuild in c8 without bumping nvr
14:27:36 <ykarel> amoralej, sure let me check
14:27:44 <ykarel> wrt Train that's it from me
14:28:01 <amoralej> yes, in fact, push for sig c8 builds seems to be wip in centos side
14:28:11 <amoralej> but at least we will start building all packages
14:28:24 <amoralej> in cloud8-openstack-train-candidate
14:28:36 <ykarel> yeap
14:28:38 <amoralej> and at some point we will promote
14:28:39 <amoralej> ok
14:28:41 <amoralej> thx ykarel
14:29:14 <amoralej> #info Dependencies repo reorganization in CentOS 8
14:29:23 <rdogerrit> Merged gating_scripts master: Continue execution if NVR has not been changed  https://review.rdoproject.org/r/25041
14:29:50 <amoralej> we are working also in some reorganization of deps repo for some reasons
14:30:24 <amoralej> #info rabbitmq-server and deps will be moved to messaging sig and until then will be in a separated repo in trunk.r.o
14:31:05 <amoralej> #info storage sig ceph repos will be synced to trunk.rdoproject.org to be used by upstream projects until sync to mirrors is fixed
14:31:51 <amoralej> #info some changes may be needed in repos configuration for centos8, we will work with consumer projects to fix it
14:32:12 <rdogerrit> User mjturek created rdo-infra/ci-config master: ppc64le: Fix ssh for collecting logs and delorean reporting  https://review.rdoproject.org/r/25047
14:32:21 <amoralej> i think this is what i wanted to share
14:32:42 <amoralej> anything else that i'm missing about c8?
14:33:24 <ykarel> i think that's it
14:33:49 <amoralej> ok
14:34:12 <amoralej> #topic DocsDays is this week
14:34:26 <leanderthal> #link http://rdoproject.org/events/docdays/
14:34:27 <amoralej> #link http://rdoproject.org/events/docdays/
14:34:31 <amoralej> #undo
14:34:32 <leanderthal> oh.
14:34:32 <openstack> Removing item from minutes: #link http://rdoproject.org/events/docdays/
14:34:42 <amoralej> go ahead leanderthal :)
14:34:45 <leanderthal> :-D
14:34:54 <leanderthal> #info to improve the documentation and the website by spending an hour or two looking for errors, outdated information, or anything else that could be better
14:35:11 <leanderthal> just that we're doing #DocDays this week which we haven't done in quite a few years.
14:36:08 <leanderthal> It basically means we're getting metrics on how active PRs are on the website / documentation this week and then we'll move forward from there - to see if we want to make this a regular focus, et cetera
14:36:31 <leanderthal> any questions?
14:36:34 <amoralej> #action everyone to report and/or fix issues in documentation
14:36:40 <leanderthal> thx amoralej
14:36:46 <amoralej> :)
14:36:51 <rdogerrit> User mjturek created rdo-infra/ci-config master: ppc64le: Include ansible-role-container-registry in roles  https://review.rdoproject.org/r/25048
14:38:32 <amoralej> i think we can move to next one
14:38:40 <amoralej> #topic Chair for next meeting
14:38:45 <amoralej> first volunteer wins :)
14:38:50 <ykarel> i can take it :)
14:39:03 <amoralej> #action ykarel to chair next meeting
14:39:05 <amoralej> thanks ykarel
14:39:17 <amoralej> #topic open floor
14:39:40 <amoralej> something else you want to share?
14:39:49 <rdogerrit> User mjturek proposed rdo-infra/ci-config master: ppc64le: Include ansible-role-container-registry in roles  https://review.rdoproject.org/r/25048
14:43:18 <mjturek> amoralej: curious how the migration to the new registry is going
14:43:48 <amoralej> jpena, ^
14:44:28 <jpena> we're working to get everything migrated to Vexxhost by mid-March. That includes the registry
14:44:39 <mjturek> jpena cool!
14:44:50 <jpena> finger crossed :)
14:45:13 <mjturek> please let us knoww if there's any way we can help, it's needed to finally get ppc published to dockerhub
14:48:27 <rdogerrit> Merged rdo-infra/ci-config master: ppc64le: Fix ssh for collecting logs and delorean reporting  https://review.rdoproject.org/r/25047
14:50:15 <amoralej> seems there is nothing else
14:50:27 <amoralej> so i'll close the meeting and give you some minutes back
14:50:33 <amoralej> thanks all for joining
14:50:54 <amoralej> #endmeeting