| mnasiadka | Morning, can anybody add me to newly created Gerrit group ‘kolla-reviewers’ so I can populate it? | 06:17 |
|---|---|---|
| *** ralonsoh_ is now known as ralonsoh | 07:08 | |
| frickler | I can do that later, just need to tackle some local stuff first | 08:18 |
| mnasiadka | Thanks :) | 08:44 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/project-config master: Revive os_freezer role for OSA https://review.opendev.org/c/openstack/project-config/+/973363 | 10:37 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/project-config master: Revive os_freezer role for OSA https://review.opendev.org/c/openstack/project-config/+/973363 | 10:43 |
| frickler | mnasiadka: added you to the group, please check | 11:10 |
| mhu | mhu: In case you're interested, the issue was caused by ... cybersquatting and using "microshift.dev" as our test FQDN. Long story short that test domain lead to resolv.conf in pods to be set to try and add .dev to any resolution attempt, meaning there was an attempt to resolve opendev.org.dev ... which is cyber-squatted by bodis.com | 11:11 |
| mhu | IDK if you want to do anything about that rogue domain, but I thought I'd let you know | 11:12 |
| frickler | looks like they have a wildcard match for *.org.dev, I don't think one can do much about it. thanks for the update anyway, confirms my aversion against using weird TLDs ;) | 11:19 |
| mhu | this was a head-scratcher, I am grateful for the help of the OpenShift team as they figured it out | 11:20 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/project-config master: Revive os_freezer role for OSA https://review.opendev.org/c/openstack/project-config/+/973363 | 14:09 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/project-config master: Add template to the os_freezer repo https://review.opendev.org/c/openstack/project-config/+/973383 | 14:10 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/project-config master: Revive os_watcher role for OSA https://review.opendev.org/c/openstack/project-config/+/973387 | 14:17 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/project-config master: Add os_watcher zuul templates https://review.opendev.org/c/openstack/project-config/+/973388 | 14:18 |
| fungi | mhu: too bad there's not, like, an ietf rfc about iana reserved domain names for use in testing and documentation | 14:31 |
| fungi | oh, wait, there is! ;) https://www.rfc-editor.org/rfc/rfc2606 | 14:31 |
| fungi | huh, i just happened across https://nitter.net/about | 14:51 |
| fungi | apparently you can s/x.com/nitter.net/ in twitter urls and then read without needing a login | 14:52 |
| fungi | handy for people like me who don't have a twitter account but get referred to things like vulnerability discussions there | 14:52 |
| mnasiadka | So - yesterday’s revert - any idea what I can improve to test the patch better? :-) | 14:56 |
| fungi | i think clarkb and corvus will probably have better suggestions once they're up. i picked the wrong moment to run errands while it was landing and got back after the revert merged | 14:58 |
| clarkb | mnasiadka: I think the main issue is that opendev/base-jobs/roles/mirror-info depends on the mirror_fqdn var being set. Due to the way ansible handles variables I expect we have to treat all of those variables as a public api for these central roles an can't simply remove them | 15:45 |
| clarkb | mnasiadka: so for this change in particular we need to continue to set mirror_fqdn to something (and probably ideally something that makes sense) then just not use it internally within the role? | 15:46 |
| clarkb | mnasiadka: then corvus was talking about testing this better. We may need to use base-test in opendev/base-jobs to test a copy of mirror configuration and the mirror info roles to ensure they work as expected before we land the change everyone will see | 15:47 |
| clarkb | as a heads up jitsi meet made a release yesterday just prior to our daily runs. I expect this means meetpad upgraded during the daily runs. I'll check on that in a bit | 15:49 |
| mnasiadka | clarkb: testing it better might be the best option, I’d be happier to see breakage before the change is merged :) | 15:57 |
| corvus | ping | 15:57 |
| clarkb | corvus: pong? | 15:57 |
| clarkb | mnasiadka: so ya I think the first step is a new version of the change that doesn't remove any variables, but changes how the variabels are used internally. Then we can sort out how to configure base-test to test it (possibly with an entire copy of the role) | 15:58 |
| corvus | i think my matrix federation lagged a bit, it's catching up now. i'm going to resend something from earlier -- it may show up again later, sorry | 15:58 |
| clarkb | ah that explains the ping | 15:58 |
| corvus | mnasiadka: i think it needs a test plan -- which might look like making sure all the related roles are covered by jobs in zuul-jobs that exercise them all when any of them change (this is ideal), and/or using test versions of the roles and the base-test job. i think that those roles should all be able to be exercised in normal zuul-test jobs, so that's why i think that should be preferred. we shouldn't rely on base-test unless it's impossible to | 15:59 |
| corvus | test otherwise. but it is an option if the other approach has obstacles. | 15:59 |
| corvus | ^ i wrote that before clarkb showed up; so understand it doesn't have the context of what he said | 15:59 |
| clarkb | the main reason I mention base-test is that the mirror-info role that failed doesn't live in zuul-jobs. It appears to be opendev specific. That said I think we'd be happy to move that into zuul-jobs and test it there if we think there is a generic need for a role like that | 16:00 |
| clarkb | but yes testing that without the base-test dance would be great if we can make that happen | 16:00 |
| corvus | good point; i agree. if we want to keep it where it is, then we'll need to use base-test. but maybe now that we have such good coverage of the base roles otherwise, it makes sense to try to move it to zuul-jobs, and then rely on zuul-jobs testing. | 16:02 |
| mnasiadka | Ok, I’ll propose moving that role (and find anything else that uses mirror_fqdn) and make it less prone to missing mirror_fqdn | 16:03 |
| clarkb | mnasiadka: that sounds great thanks | 16:03 |
| clarkb | meetpad did update ~13 hours ago | 16:03 |
| clarkb | I haven't tested the service itself just looked at docker things. I expect it is running happily, but keep that in mind | 16:04 |
| mnasiadka | clarkb: looking at mirror_info.sh template in mirror-info role - there’s probably a lot of history behind NODEPOOL_* env vars (https://opendev.org/opendev/base-jobs/src/branch/master/roles/mirror-info/templates/mirror_info.sh.j2). Are we sure that should land in zuul/zuul-jobs? | 17:04 |
| clarkb | mnasiadka: no, I wasn't sure about that. Maybe what we can do is have a second role in zuul-jobs (possibly just for testing) that simulates what opendev's mirror-info is doing with vars like mirror-info | 17:06 |
| clarkb | mnasiadka: so don't port over the specific functionality, but just test that particular use case of reconsuming mirror_fqdn in another role? | 17:06 |
| clarkb | looks like a number of roles do use a mirror_fqdn value in zuul-jobs but they all default to zuul_site_mirror_fqdn which is what we thought would cover us before we reverted | 17:09 |
| clarkb | but maybe we can unset zuul_site_mirror_fqdn and test it via one of those existing roles? | 17:09 |
| clarkb | but otherwise a test specific role that mimics mirror-info is probably fine | 17:09 |
| mnasiadka | Yes, I think that’s the only role that doesn’t default mirror_fqdn to zuul_site_mirror_fqdn | 17:14 |
| mnasiadka | (At least based on codesearch.opendev.org output) | 17:15 |
| mnasiadka | But I’m wondering how many users of mirror_fqdn are there in their own Zuul (outside of OpenDev) | 17:15 |
| opendevreview | Jeremy Stanley proposed openstack/project-config master: Clean up unused DockerHub credentials https://review.opendev.org/c/openstack/project-config/+/973415 | 17:16 |
| opendevreview | Michal Nasiadka proposed zuul/zuul-jobs master: Revert^2 "Use mirror_info in configure-mirrors role" https://review.opendev.org/c/zuul/zuul-jobs/+/973416 | 17:17 |
| clarkb | mnasiadka: yup exactly why I suspect having a stand in role is a good idea | 17:18 |
| opendevreview | Michal Nasiadka proposed zuul/zuul-jobs master: Revert^2 "Use mirror_info in configure-mirrors role" https://review.opendev.org/c/zuul/zuul-jobs/+/973416 | 17:18 |
| clarkb | since that is something of a public api based on our own experience | 17:18 |
| mnasiadka | Ok then, but ideally that role should not be under roles/ directory, so nobody uses that | 17:19 |
| clarkb | correct, I think it can be test only | 17:19 |
| clarkb | and live in the test-playbooks dir or something like that | 17:19 |
| opendevreview | Merged openstack/project-config master: Clean up unused DockerHub credentials https://review.opendev.org/c/openstack/project-config/+/973415 | 17:27 |
| opendevreview | Michal Nasiadka proposed zuul/zuul-jobs master: Revert^2 "Use mirror_info in configure-mirrors role" https://review.opendev.org/c/zuul/zuul-jobs/+/973416 | 17:29 |
| opendevreview | Michal Nasiadka proposed zuul/zuul-jobs master: Revert^2 "Use mirror_info in configure-mirrors role" https://review.opendev.org/c/zuul/zuul-jobs/+/973416 | 17:30 |
| opendevreview | Michal Nasiadka proposed zuul/zuul-jobs master: Revert^2 "Use mirror_info in configure-mirrors role" https://review.opendev.org/c/zuul/zuul-jobs/+/973416 | 17:41 |
| opendevreview | Michal Nasiadka proposed zuul/zuul-jobs master: Revert^2 "Use mirror_info in configure-mirrors role" https://review.opendev.org/c/zuul/zuul-jobs/+/973416 | 18:08 |
| opendevreview | Michal Nasiadka proposed zuul/zuul-jobs master: Revert^2 "Use mirror_info in configure-mirrors role" https://review.opendev.org/c/zuul/zuul-jobs/+/973416 | 18:15 |
| opendevreview | Michal Nasiadka proposed zuul/zuul-jobs master: Use mirror_info in configure-mirrors role for pypi/wheel - take 2 https://review.opendev.org/c/zuul/zuul-jobs/+/973423 | 18:18 |
| mnasiadka | clarkb: ^^ that should be a bit better, also to test that properly - I’ll follow later this week with the same for package repos | 18:24 |
| clarkb | mnasiadka: thanks I'll take a look shortly | 18:26 |
| clarkb | infra-root Looking at a calendar I'm going to propose the service coordinator election nomination period occur February 3 - 17. THen if we need to have an election that will run February 18 - 25 | 18:27 |
| clarkb | I'll throw that on the meeting agenda for next Tuesday and if that set of dates seems reaonsbale (it should be basically 6 months after the last election) I'll make it official next week | 18:28 |
| clarkb | mnasiadka: looks like the main difference is a new flag indicating to use zuul_site_mirror_fqdn as a fallback value (whcih default to true) rather than relying on mirror_fqdn directly? Then also in the test case you're invoking the opendev mirror-info role to see if it is happy? | 18:34 |
| clarkb | I guess mirror-info is available there because it is in the base-jobs role which the zuul tenant includes roles from so that should work. corvus may have thoughts on whether or not that is a valid cross over but I think it should work for our purposes | 18:35 |
| mnasiadka | clarkb: the fallback flag was there before, I just restored back setting mirror_fqdn and improved testing (and noticed there’s mirror_info.wheel.url in Zuul example docs so changed wheel_mirror value to be similar to pypi_mirror one) | 18:36 |
| mnasiadka | In the example (https://zuul-ci.org/docs/zuul-jobs/latest/mirror.html) Ubuntu and Debian are quite good documented, so I’ll rework the package mirror setting for them - but I think RHEL clones might need something more complicated than what is in the example | 18:39 |
| clarkb | got it | 18:39 |
| opendevreview | Jeremy Stanley proposed openstack/project-config master: Rotate Launchpad token for release jobs https://review.opendev.org/c/openstack/project-config/+/973429 | 19:11 |
| clarkb | fungi: for the gerrit side of ^ you generated a different token for the same account I guess? | 19:12 |
| clarkb | (just clarifying the commit message indicatng they are separate now and wondering what the mechanism was for that) | 19:13 |
| fungi | yes, there are now two new tokens authorized for "change non-private data" | 19:13 |
| fungi | one is called "Release Jobs" and the other is "Gerrit Hooks" | 19:14 |
| fungi | the latter won't go into effect until the next gerrit server deploy from ansible updates the launchpad config, so i'm leaving the old token (from 2014-05-07) authorized until we see the new one working | 19:15 |
| fungi | but they're both application (oauth) tokens for our hudson-openstack account in lp | 19:16 |
| clarkb | sounds good thnaks | 19:17 |
| fungi | i followed the steps from https://documentation.ubuntu.com/launchpad/user/how-to/launchpadlib/using-launchpadlib/index.html#authenticated-access-for-website-integration to generate them, for future reference | 19:17 |
| opendevreview | Merged openstack/project-config master: Rotate Launchpad token for release jobs https://review.opendev.org/c/openstack/project-config/+/973429 | 19:45 |
| opendevreview | Jeremy Stanley proposed openstack/project-config master: Drop openstack-fips and openstack_ubuntu_fips https://review.opendev.org/c/openstack/project-config/+/973434 | 20:10 |
| fungi | https://launchpad.net/~hudson-openstack/+karma should indicate if things are still working once the launchpad token updates | 21:07 |
| fungi | unfortunately it doesn't link to the actual activity, so hard to know whether it's from gerrit or zuul | 21:07 |
| clarkb | the gerrit homedir perms change is merging | 21:20 |
| opendevreview | Merged opendev/system-config master: Set perms and ownership on Gerrit's homedir https://review.opendev.org/c/opendev/system-config/+/970919 | 21:20 |
| clarkb | and the deplyoment is running now | 21:22 |
| clarkb | homedir is now gerrit2:gerrit2. Still waiting on launchpad lib creds update | 21:23 |
| clarkb | both are updated now | 21:23 |
| clarkb | the job reports success. At a high level this all looks good to me | 21:24 |
| fungi | perfect | 21:26 |
| fungi | that should have updated the lp creds the hooks are using too, checking | 21:26 |
| clarkb | fungi: re your comments about knowing when this is working above I guess we expect someone pushing or merging a change with a bug link in the commit message to update that bug right? | 21:26 |
| clarkb | fungi: yup the timestamp on that creds file udpated at least. I didn't look in the file | 21:26 |
| fungi | right. though comments because of release jobs could also show up in the activity list and lp doesn't give us a way to distinguish between them, but we can at least gauge whether it was around the time any release requests merged | 21:27 |
| fungi | though i suppose gerrit also logs running the hooks | 21:32 |
| clarkb | yes, it complains a lot when they fail too which may be a good indicator | 21:33 |
| fungi | yeah, ~gerrit2/.launchpadlib/creds has the new values | 21:33 |
| opendevreview | Merged openstack/project-config master: Drop openstack-fips and openstack_ubuntu_fips https://review.opendev.org/c/openstack/project-config/+/973434 | 21:40 |
| fungi | the last reference to update_bug in gerrit's error_log is from 18:39:59 so nothing has triggered it since the creds were updated at 21:23 | 21:42 |
| fungi | still no newer hits to update_bug.py in the log, i'll check it again in the morning | 23:15 |
Generated by irclog2html.py 4.0.0 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!