| *** Guest2352 is now known as prometheanfire | 00:34 | |
| *** prometheanfire is now known as Guest2640 | 00:35 | |
| opendevreview | Merged opendev/system-config master: Test the rename_repos playbook https://review.opendev.org/c/opendev/system-config/+/802112 | 00:35 |
|---|---|---|
| *** Guest2640 is now known as prometheanfire | 00:36 | |
| opendevreview | Ian Wienand proposed opendev/base-jobs master: Remove debian-stable nodeset https://review.opendev.org/c/opendev/base-jobs/+/802639 | 00:45 |
| opendevreview | Ian Wienand proposed opendev/base-jobs master: Remove debian-stable nodeset https://review.opendev.org/c/opendev/base-jobs/+/802639 | 00:51 |
| ianw | it's interesting that ^ both fails config and gets jobs run against it. i'm going to assume it's some sort of multi-tenant/parenting magic | 00:53 |
| ianw | although now it has a verified +1 but is also a configuration error. i wonder if it would merge? | 01:04 |
| Clark[m] | I want to say it will. Base jobs is in all the tenants so we get those reports | 01:09 |
| opendevreview | Merged opendev/system-config master: Remove review-test https://review.opendev.org/c/opendev/system-config/+/801556 | 01:48 |
| *** ykarel_ is now known as ykarel | 05:15 | |
| *** marios is now known as marios|ruck | 05:41 | |
| *** amoralej|off is now known as amoralej | 06:26 | |
| *** rpittau|afk is now known as rpittau | 07:10 | |
| *** ykarel is now known as ykarel|lunch | 08:24 | |
| sshnaidm | maybe someone knows, in which pipeline I can make a commit and push it to gerrit? Would appreciate any examples with commits during the jobs | 08:27 |
| *** ykarel|lunch is now known as ykarel | 09:51 | |
| opendevreview | Ananya proposed opendev/elastic-recheck master: Run elastic-recheck in container https://review.opendev.org/c/opendev/elastic-recheck/+/802866 | 11:52 |
| *** amoralej is now known as amoralej|lunch | 12:31 | |
| opendevreview | Dmitry Tantsur proposed ttygroup/gertty master: examples: match 'commit <hash>' https://review.opendev.org/c/ttygroup/gertty/+/799642 | 12:40 |
| *** amoralej|lunch is now known as amoralej | 13:24 | |
| fungi | sshnaidm: theoretically any pipeline, though you'd generally need to have that push authenticated with a gerrit account which will require a secret, and you can't really use secrets in the run phase of a job unless it's run in a post-review pipeline (gate, promote, post, tag, et cetera) | 13:29 |
| fungi | sshnaidm: examples i'm aware of are openstack release jobs which propose .gitreview file adjustments to new branches when they're created, and periodic jobs which propose dependency bumps to openstack requirements | 13:31 |
| sshnaidm | fungi, thanks, will look into release jobs | 13:31 |
| sshnaidm | I'd like to do it in post after it was merged by gate job | 13:32 |
| fungi | sshnaidm: there's also the promote pipeline as an alternative, depending on whether you want guaranteed runs, and whether you want them to be relative to the change which merged or to the commit state of the branch | 13:34 |
| fungi | the openstack tenant's post pipeline uses a supercedent pipeline manager, which skips some intermediate refs for efficiency but guarantees the latest branch state will always get jobs run | 13:35 |
| fungi | in contrast, the openstack tenant's promote pipeline runs jobs for each change which has merged | 13:35 |
| fungi | and doesn't skip any even if there's a queue of them | 13:36 |
| sshnaidm | fungi, is it possible to do in release/tag pipeline? I'd like to change file in release and commit it | 13:36 |
| sshnaidm | in the same job | 13:37 |
| fungi | yes, you could do something similar from a tag-based pipeline like tag or release, but keep in mind that git tags don't have a direct relationship with branches, and if it's a multi-branch repository you're going to have to rely on mechanisms to guess which branch is most relevant for a particular tag | 13:38 |
| sshnaidm | no, fortunately it's master only repo | 13:39 |
| fungi | (not as much a zuul or gerrit limitation as a git data model limitation) | 13:39 |
| fungi | yeah then that's probably fairly easy | 13:39 |
| sshnaidm | fungi, thanks for explanations | 13:39 |
| fungi | you might also pay attention to the logic in the requirements proposal jobs since they're smart enough to push revisions for an open change rather than creating new (and conflicting) changes on every run, if that's a problem you may need to avoid | 13:40 |
| fungi | though that's probably only an option if the patch is being created idempotently | 13:41 |
| sshnaidm | fungi, what is its name for example? | 13:43 |
| sshnaidm | req proposal job | 13:43 |
| fungi | sshnaidm: sorry, had to step away for a few, will try to go find it shortly | 14:00 |
| *** Guest1365 is now known as hashar | 14:04 | |
| *** hashar is now known as Guest2704 | 14:05 | |
| *** Guest2704 is now known as hashar | 14:06 | |
| clarkb | fungi: thinking about the renaming now that the testing change has merged I think one thing we should check is that ssh keys for ssh'ing to localhost port 29418 are set up | 14:51 |
| fungi | yeah, and the known-hosts entry is present | 14:52 |
| clarkb | yup exactly | 14:52 |
| fungi | also i wanted to double-check that the ssh task is set to the review host not the bridge host where we run the playbook | 14:52 |
| clarkb | fungi: I think it is and that should be tested in our job as we have a multinode job with bridge and review split out | 14:53 |
| fungi | since the bridge ssh'ing to localhost is clearly not what we would want | 14:53 |
| fungi | yeah, agreed, if the job is running the playbook on a separate bridge node then it proved that's right already | 14:53 |
| fungi | which i guess it would need to since it also has to run tasks on gitea servers | 14:53 |
| clarkb | I think the last major item is to get the TC to ack this before we proceed | 14:53 |
| clarkb | I agree we seem to have slaweq's go ahead and the project itself requested it from what I can tell. So it would just be the openstack tc giving an ok to move it under openstack/ that is left? | 14:54 |
| clarkb | fungi: there are also storyboard tasks which might be worth glancing over to ensure you think they'll do the right thing. We don't have the same level of functional testing for storyboard so hard to confirm in testing | 14:54 |
| clarkb | fungi: https://review.opendev.org/c/openstack/governance/+/802833 we do have a governance change now which I'll mention in the TC meeting | 15:00 |
| fungi | yeah, the sb tasks shouldn't have changed but i'll take a closer look | 15:02 |
| fungi | once diablo_rojo's container deployment stuff for sb is squared away, we could probably easily include a test for renames there too | 15:03 |
| clarkb | yup | 15:04 |
| opendevreview | Jing Li proposed openstack/diskimage-builder master: Add new element rocky https://review.opendev.org/c/openstack/diskimage-builder/+/802902 | 15:04 |
| clarkb | fungi: also if review's LE cert hasn't sorted itself by the time we do the renames we should restart apache as part of the downtime. I'm fairly certain the new cert is ready on disk based on timestamps and we just need to convince apache to read it | 15:08 |
| fungi | yeah | 15:13 |
| *** ykarel is now known as ykarel|away | 15:50 | |
| *** rpittau is now known as rpittau|afk | 16:03 | |
| *** amoralej is now known as amoralej|off | 16:10 | |
| *** marios|ruck is now known as marios|out | 16:18 | |
| fungi | clarkb: i updated the plan for tomorrow to also include disabling ansible for sb, since we don't want a deploy job recreating the old projects on it while we're waiting on the changes to replicate | 17:00 |
| clarkb | ++ | 17:01 |
| clarkb | fungi: if I run our ssh command from the rename playbook with the command ls-projects I get asked to accept the ssh key fingerprint | 17:01 |
| clarkb | I did not do that. Will see if config managment can be convinced to do that update for us. We can always accept it first thing tomorrow if that doesn't ahpepn | 17:02 |
| fungi | yeah, i had a feeling that was missing since we needed to explicitly do it in the test | 17:02 |
| clarkb | ya it seems to be completely missing from config management. I thought it might be there for review02.opendev.org and review.opndev.org just not localhost but I can't find evidence of ither | 17:03 |
| fungi | i think because manage-projects connects by public hostname | 17:04 |
| fungi | like the rename playbook used to | 17:04 |
| clarkb | ya I think I found it for manage-projects | 17:05 |
| clarkb | fungi: I'm thinking this may not get sorted via config management before tomorrow. Its kind of a mess :/ I'll have ac hange up shortly that does things for testing though. | 17:25 |
| clarkb | infra-root ^ we probably need to dobule check if manage-projects is even working on the new server? I'm not sure how keys are getting in there | 17:26 |
| fungi | no worries, we can manually accept the ssh host key in the meantime | 17:26 |
| fungi | oh, yeah that's a good idea too | 17:26 |
| opendevreview | Clark Boylan proposed opendev/system-config master: Improve gerrit known_hosts management https://review.opendev.org/c/opendev/system-config/+/802922 | 17:28 |
| clarkb | I think ^ is a good step 0. That should at least check if what is minimally there is working. I left a TODO in the commit message describing what I think needs to be done in addition to that. If people reviewing that can actualyl double check the prod server too that would be good. There are host keys for something in roots known_hosts key file which is what manage projects bind | 17:30 |
| clarkb | mounts in. However I have no idea how to confirm that those hostkeys belong to the current server without testing ssh directly? | 17:30 |
| clarkb | is there a better way for hashed names? | 17:30 |
| clarkb | ok if I try to ssh using the root known_hosts key I get authentication errors and not the host key verification warning | 17:32 |
| clarkb | I think that means we have accepted the host key there somehow, but not via config management. My change should add that in if we set the proper vars. | 17:33 |
| clarkb | For tomorrow we only need to update gerrit2's known_hosts I think (someone other than me should probably confirm ) | 17:33 |
| clarkb | "Host parameter does not match hashed host field in supplied key" now to figure that out | 18:16 |
| fungi | that's an ansible error? | 18:18 |
| clarkb | ya, its specific to the known_hosts module | 18:18 |
| clarkb | the issue is its supplied as a separate parameter to what is in the gerrit_self_hostkey file so I need to figure out how to reconcile that between testing and prod | 18:18 |
| clarkb | I think I can do a simple hack around it | 18:20 |
| opendevreview | Clark Boylan proposed opendev/system-config master: Improve gerrit known_hosts management https://review.opendev.org/c/opendev/system-config/+/802922 | 18:20 |
| clarkb | that might work | 18:20 |
| clarkb | oh actually I think I could do an even more hacky but reliable thing and put [{{ gerrit_vhost_name }}]:29418 in that var? | 18:24 |
| clarkb | then the gerrit_vhost_name will always show up in the host key value and hash properly | 18:24 |
| clarkb | Let's see if the simpler patch above works and if it does I can push ^ up | 18:24 |
| clarkb | if it doesn't work then its back to debugging further | 18:24 |
| clarkb | https://gerrit-review.googlesource.com/c/gerrit/+/312302 has merged to gerrit stable-3.2 | 18:30 |
| clarkb | apparently you get to clikc the submit button yourself on that server as the code contributor. I did not expect that at all. | 18:30 |
| *** sshnaidm is now known as sshnaidm|afk | 18:30 | |
| fungi | that does seem bizarre to me | 18:32 |
| clarkb | that and the submit button is now in the top right so I just wasn't lokoing for it | 18:33 |
| * clarkb is looking at gerrit forward merges. Got the 3.2 into 3.3 pushed. 3.3 into 3.4 is a lot more complicated I'm quite confused | 19:39 | |
| opendevreview | Clark Boylan proposed opendev/system-config master: Improve gerrit known_hosts management https://review.opendev.org/c/opendev/system-config/+/802922 | 19:43 |
| clarkb | the previous ps worked so now lets try the more automagic version | 19:43 |
| clarkb | ok the use of the variable there does not work | 20:41 |
| clarkb | gerrit_vhost_name is defined in a host vars file and then we try to use it in a group vars file | 20:41 |
| clarkb | I guess ansible doesn't evaluate those in the order necessary to make this work. That makes sense since host vars override group vars | 20:42 |
| clarkb | I can add a review02 test host vars file but at that point it seems like I may as well just set the name direclty | 20:42 |
| clarkb | I'll revert to the previous patchset and reviewers can tell me if they don't like that | 20:42 |
| mordred | clarkb: that's unexpected - I thought variable expansion from host/group vars was late-bound | 20:43 |
| opendevreview | Clark Boylan proposed opendev/system-config master: Improve gerrit known_hosts management https://review.opendev.org/c/opendev/system-config/+/802922 | 20:44 |
| clarkb | mordred: seems not to be | 20:44 |
| clarkb | mordred: https://zuul.opendev.org/t/openstack/build/7134da8ccb824717bb737a441807ce9c/log/job-output.txt#1520-1525 is where ansible broke | 20:44 |
| mordred | oh - wait ... that's the host_vars templating | 20:46 |
| mordred | clarkb: that's not expansion not working due to precedence - that's our test framework jinja rendering thinking _it_ needs to be the one to expand that jinja | 20:47 |
| clarkb | oh | 20:47 |
| clarkb | I guess I can use a raw quote and then that would be removed when it gets written out ? | 20:48 |
| clarkb | let me try that | 20:48 |
| mordred | yah | 20:49 |
| opendevreview | Clark Boylan proposed opendev/system-config master: Improve gerrit known_hosts management https://review.opendev.org/c/opendev/system-config/+/802922 | 20:50 |
| clarkb | like that maybe | 20:50 |
| mordred | yeah | 20:51 |
| mordred | that's totally going to work | 20:51 |
| mordred | unless it doesn't | 20:51 |
| clarkb | its a bit weird to me that ansible insists on checking the name passed against the key passed | 20:52 |
| clarkb | but I guess that is so they can safely delete if you esnure absent | 20:52 |
| clarkb | fungi: I added a step about adding the host key for loclahost as gerrit2 to known hosts on the etherpad | 21:30 |
| fungi | perfect, thanks | 21:30 |
| clarkb | fungi: we should probably add the playbook command to the etehrpad too? | 21:36 |
| fungi | i can though i think we already have it in the linked doc. i'll add it if not | 21:36 |
| clarkb | ok | 21:37 |
| fungi | clarkb: i've cut and pasted the example command from step 4 into the etherpad, if you want to double-check that still looks correct | 21:51 |
| clarkb | fungi: I updated the path to the playbook but I think that looks good | 21:51 |
| fungi | thanks! | 21:52 |
| ianw | clarkb: lgtm. i guess i only ssh'd as my admin user during the cutover | 22:13 |
| clarkb | ianw: note that change won't actually fix prod alone since we need to add the var in in prod hostvars | 22:13 |
| clarkb | and then we need to followup with gitea host keys though those are already in there | 22:14 |
| ianw | i'm almost certain i didn't manually add the gitea host keys? | 22:15 |
| ianw | oh, no, that's right, we added a step maybe to do that didn't we | 22:16 |
| clarkb | ya we did it during the move when I realized it would break replication without it | 22:17 |
| ianw | yea step 11.2 @ https://etherpad.opendev.org/p/gerrit-upgrade-2021 | 22:17 |
| clarkb | I think what we'll end up doing is having a { name: keyvalue } dict that we iterate over for gerrit2 and root known_hosts and add them in | 22:17 |
| clarkb | as a followon to what I've done aboev as a first step | 22:17 |
| clarkb | since ansible wants the name to match what is in keyvalue we have to provide both I think | 22:18 |
| ianw | yep; there's probably a bit of prior work in the borg-backup roles which sets up ssh permissions between server -> backup host | 22:18 |
| ianw | fungi: have you ever had any special dealings with libvirt-python wheels? | 23:16 |
| ianw | this is in relation to https://bugs.launchpad.net/devstack/+bug/1933096 | 23:17 |
| clarkb | ianw: libvirt-python is updated/released in sync with libvirt itself, but a new libvirt-python can be built against old libvirt and that is expected to work fine | 23:21 |
| clarkb | in the case above I think the issue is we are treating centos-8 and centos-8-stream as equivalent so the libvirt-python built for centos-8 is installed on centos-8-stream | 23:21 |
| ianw | 2021-06-27 04:37:51.647585 | controller | Downloading https://mirror.mtl01.inap.opendev.org/wheel/centos-8-x86_64/libvirt-python/libvirt_python-7.4.0-cp36-cp36m-linux_x86_64.whl (554 kB) | 23:21 |
| clarkb | this is a problme because the wheels needs to be built against the actual libvirt that you have | 23:21 |
| ianw | Jun 27 05:03:22.685506 centos-8-stream | 23:22 |
| clarkb | basically we need a centos-8-stream wheel mirror | 23:22 |
| ianw | haha yes, i think we just reached the same conclusion :) | 23:22 |
| ianw | this is a usual yak shaving exercise. that change installed the package libvirt in devstack. i'm trying to update devstack to use the latest pip. the latest pip refuses to uninstall packaged libvirt with the "this is a distutils package" stuff | 23:23 |
| clarkb | and then devstack fixed this by not installing libvirt-python with pip which is proably fine as long as nova doesn't need newer features in the library. But doesn't fix the issue that this same problem could pop up for other wheels | 23:23 |
| fungi | ianw: long ago i recall we had to install python-libvirt distro packages and they did not even publish the wrapper to pypi. they finally worked out a build process where they could generate python packages of it, but yes still tied to fairly specific libvirt versions i believe | 23:23 |
| ianw | well, this becomes a problem when something installs using upper-constraints | 23:24 |
| clarkb | ianw: wow centos-8-stream libvirt-python is still distutils packaged? | 23:24 |
| ianw | no, this is actually on ubuntu | 23:24 |
| clarkb | ah | 23:24 |
| opendevreview | Merged opendev/gear master: Add libffi header dependency https://review.opendev.org/c/opendev/gear/+/800325 | 23:24 |
| clarkb | but the package install was done for all distros got it | 23:24 |
| opendevreview | Merged opendev/gear master: Overhaul package metadata and contributor info https://review.opendev.org/c/opendev/gear/+/796704 | 23:24 |
| ianw | anyway, i should be able to prune fedora 32 today | 23:25 |
| ianw | which is where all this started! | 23:25 |
| fungi | congrats! | 23:26 |
| corvus | i think codesearch may be out of date | 23:40 |
| corvus | https://codesearch.opendev.org/?q=report-build-page&i=nope&files=&excludeFiles=&repos= is showing more results in zuul/zuul than are actually there | 23:41 |
| opendevreview | James E. Blair proposed openstack/project-config master: Remove report-build-page from zuul tenant config https://review.opendev.org/c/openstack/project-config/+/802973 | 23:42 |
| opendevreview | James E. Blair proposed zuul/zuul-jobs master: Remove success-url https://review.opendev.org/c/zuul/zuul-jobs/+/802974 | 23:43 |
| opendevreview | James E. Blair proposed opendev/base-jobs master: Remove success-url https://review.opendev.org/c/opendev/base-jobs/+/802975 | 23:44 |
| corvus | clarkb, ianw: i think if you could follow up on https://review.opendev.org/800506 (matrix-gerritbot) that would be nice | 23:49 |
| clarkb | corvus: I believe that codesearch operates on a pull system. Not sure how frequently it does that. Are we talking 15 minutes out of date or days? | 23:52 |
| corvus | clarkb: many days | 23:53 |
| ianw | hrm, it has "Unable to create '/run/data/vcs-a833d4e625f7834f10cb701f8d40d2235258d8e0/.git/index.lock': File exists." | 23:54 |
| ianw | although it does also say "Continuing..." | 23:55 |
| corvus | ...to fail :) | 23:55 |
| clarkb | corvus: re the gerritbot change just looking for review on latest ps? I'm not sure what more followup is needed other than for me to dig in enough to +2 ( I didn't prioritze that as others had already done so) | 23:56 |
| corvus | clarkb: yep, just looking for closure from you and ianw one way or another :) | 23:57 |
| corvus | ianw: maybe it's safe to just rm that index.lock? | 23:57 |
| ianw | yeah, i've done that (it's some sort of "compass-adapters" repo). | 23:58 |
| corvus | ianw: i'm guessing you just did that? :) | 23:58 |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!