Thursday, 2021-07-29

*** Guest2352 is now known as prometheanfire00:34
*** prometheanfire is now known as Guest264000:35
opendevreviewMerged opendev/system-config master: Test the rename_repos playbook
*** Guest2640 is now known as prometheanfire00:36
opendevreviewIan Wienand proposed opendev/base-jobs master: Remove debian-stable nodeset
opendevreviewIan Wienand proposed opendev/base-jobs master: Remove debian-stable nodeset
ianwit's interesting that ^ both fails config and gets jobs run against it.  i'm going to assume it's some sort of multi-tenant/parenting magic00:53
ianwalthough now it has a verified +1 but is also a configuration error.  i wonder if it would merge?01:04
Clark[m]I want to say it will. Base jobs is in all the tenants so we get those reports01:09
opendevreviewMerged opendev/system-config master: Remove review-test
*** ykarel_ is now known as ykarel05:15
*** marios is now known as marios|ruck05:41
*** amoralej|off is now known as amoralej06:26
*** rpittau|afk is now known as rpittau07:10
*** ykarel is now known as ykarel|lunch08:24
sshnaidmmaybe someone knows, in which pipeline I can make a commit and push it to gerrit? Would appreciate any examples with commits during the jobs08:27
*** ykarel|lunch is now known as ykarel09:51
opendevreviewAnanya proposed opendev/elastic-recheck master: Run elastic-recheck in container
*** amoralej is now known as amoralej|lunch12:31
opendevreviewDmitry Tantsur proposed ttygroup/gertty master: examples: match 'commit <hash>'
*** amoralej|lunch is now known as amoralej13:24
fungisshnaidm: theoretically any pipeline, though you'd generally need to have that push authenticated with a gerrit account which will require a secret, and you can't really use secrets in the run phase of a job unless it's run in a post-review pipeline (gate, promote, post, tag, et cetera)13:29
fungisshnaidm: examples i'm aware of are openstack release jobs which propose .gitreview file adjustments to new branches when they're created, and periodic jobs which propose dependency bumps to openstack requirements13:31
sshnaidmfungi, thanks, will look into release jobs13:31
sshnaidmI'd like to do it in post after it was merged by gate job13:32
fungisshnaidm: there's also the promote pipeline as an alternative, depending on whether you want guaranteed runs, and whether you want them to be relative to the change which merged or to the commit state of the branch13:34
fungithe openstack tenant's post pipeline uses a supercedent pipeline manager, which skips some intermediate refs for efficiency but guarantees the latest branch state will always get jobs run13:35
fungiin contrast, the openstack tenant's promote pipeline runs jobs for each change which has merged13:35
fungiand doesn't skip any even if there's a queue of them13:36
sshnaidmfungi, is it possible to do in release/tag pipeline? I'd like to change file in release and commit it13:36
sshnaidmin the same job13:37
fungiyes, you could do something similar from a tag-based pipeline like tag or release, but keep in mind that git tags don't have a direct relationship with branches, and if it's a multi-branch repository you're going to have to rely on mechanisms to guess which branch is most relevant for a particular tag13:38
sshnaidmno, fortunately it's master only repo13:39
fungi(not as much a zuul or gerrit limitation as a git data model limitation)13:39
fungiyeah then that's probably fairly easy13:39
sshnaidmfungi, thanks for explanations13:39
fungiyou might also pay attention to the logic in the requirements proposal jobs since they're smart enough to push revisions for an open change rather than creating new (and conflicting) changes on every run, if that's a problem you may need to avoid13:40
fungithough that's probably only an option if the patch is being created idempotently13:41
sshnaidmfungi, what is its name for example?13:43
sshnaidmreq proposal job13:43
fungisshnaidm: sorry, had to step away for a few, will try to go find it shortly14:00
*** Guest1365 is now known as hashar14:04
*** hashar is now known as Guest270414:05
*** Guest2704 is now known as hashar14:06
clarkbfungi: thinking about the renaming now that the testing change has merged I think one thing we should check is that ssh keys for ssh'ing to localhost port 29418 are set up14:51
fungiyeah, and the known-hosts entry is present14:52
clarkbyup exactly14:52
fungialso i wanted to double-check that the ssh task is set to the review host not the bridge host where we run the playbook14:52
clarkbfungi: I think it is and that should be tested in our job as we have a multinode job with bridge and review split out14:53
fungisince the bridge ssh'ing to localhost is clearly not what we would want14:53
fungiyeah, agreed, if the job is running the playbook on a separate bridge node then it proved that's right already14:53
fungiwhich i guess it would need to since it also has to run tasks on gitea servers14:53
clarkbI think the last major item is to get the TC to ack this before we proceed14:53
clarkbI agree we seem to have slaweq's go ahead and the project itself requested it from what I can tell. So it would just be the openstack tc giving an ok to move it under openstack/ that is left?14:54
clarkbfungi: there are also storyboard tasks which might be worth glancing over to ensure you think they'll do the right thing. We don't have the same level of functional testing for storyboard so hard to confirm in testing14:54
clarkbfungi: we do have a governance change now which I'll mention in the TC meeting15:00
fungiyeah, the sb tasks shouldn't have changed but i'll take a closer look15:02
fungionce diablo_rojo's container deployment stuff for sb is squared away, we could probably easily include a test for renames there too15:03
opendevreviewJing Li proposed openstack/diskimage-builder master: Add new element rocky
clarkbfungi: also if review's LE cert hasn't sorted itself by the time we do the renames we should restart apache as part of the downtime. I'm fairly certain the new cert is ready on disk based on timestamps and we just need to convince apache to read it15:08
*** ykarel is now known as ykarel|away15:50
*** rpittau is now known as rpittau|afk16:03
*** amoralej is now known as amoralej|off16:10
*** marios|ruck is now known as marios|out16:18
fungiclarkb: i updated the plan for tomorrow to also include disabling ansible for sb, since we don't want a deploy job recreating the old projects on it while we're waiting on the changes to replicate17:00
clarkbfungi: if I run our ssh command from the rename playbook with the command ls-projects I get asked to accept the ssh key fingerprint17:01
clarkbI did not do that. Will see if config managment can be convinced to do that update for us. We can always accept it first thing tomorrow if that doesn't ahpepn17:02
fungiyeah, i had a feeling that was missing since we needed to explicitly do it in the test17:02
clarkbya it seems to be completely missing from config management. I thought it might be there for and just not localhost but I can't find evidence of ither17:03
fungii think because manage-projects connects by public hostname17:04
fungilike the rename playbook used to17:04
clarkbya I think I found it for manage-projects17:05
clarkbfungi: I'm thinking this may not get sorted via config management before tomorrow. Its kind of a mess :/ I'll have ac hange up shortly that does things for testing though.17:25
clarkbinfra-root ^ we probably need to dobule check if manage-projects is even working on the new server? I'm not sure how keys are getting in there17:26
fungino worries, we can manually accept the ssh host key in the meantime17:26
fungioh, yeah that's a good idea too17:26
opendevreviewClark Boylan proposed opendev/system-config master: Improve gerrit known_hosts management
clarkbI think ^ is a good step 0. That should at least check if what is minimally there is working. I left a TODO in the commit message describing what I think needs to be done in addition to that. If people reviewing that can actualyl double check the prod server too that would be good. There are host keys for something in roots known_hosts key file which is what manage projects bind17:30
clarkbmounts in. However I have no idea how to confirm that those hostkeys belong to the current server without testing ssh directly?17:30
clarkbis there a better way for hashed names?17:30
clarkbok if I try to ssh using the root known_hosts key I get authentication errors and not the host key verification warning17:32
clarkbI think that means we have accepted the host key there somehow, but not via config management. My change should add that in if we set the proper vars.17:33
clarkbFor tomorrow we only need to update gerrit2's known_hosts I think (someone other than me should probably confirm )17:33
clarkb"Host parameter does not match hashed host field in supplied key" now to figure that out18:16
fungithat's an ansible error?18:18
clarkbya, its specific to the known_hosts module18:18
clarkbthe issue is its supplied as a separate parameter to what is in the gerrit_self_hostkey file so I need to figure out how to reconcile that between testing and prod18:18
clarkbI think I can do a simple hack around it18:20
opendevreviewClark Boylan proposed opendev/system-config master: Improve gerrit known_hosts management
clarkbthat might work18:20
clarkboh actually I think I could do an even more hacky but reliable thing and put [{{ gerrit_vhost_name }}]:29418 in that var?18:24
clarkbthen the gerrit_vhost_name will always show up in the host key value and hash properly18:24
clarkbLet's see if the simpler patch above works and if it does I can push ^ up18:24
clarkbif it doesn't work then its back to debugging further18:24
clarkb has merged to gerrit stable-3.218:30
clarkbapparently you get to clikc the submit button yourself on that server as the code contributor. I did not expect that at all.18:30
*** sshnaidm is now known as sshnaidm|afk18:30
fungithat does seem bizarre to me18:32
clarkbthat and the submit button is now in the top right so I just wasn't lokoing for it18:33
* clarkb is looking at gerrit forward merges. Got the 3.2 into 3.3 pushed. 3.3 into 3.4 is a lot more complicated I'm quite confused19:39
opendevreviewClark Boylan proposed opendev/system-config master: Improve gerrit known_hosts management
clarkbthe previous ps worked so now lets try the more automagic version19:43
clarkbok the use of the variable there does not work20:41
clarkbgerrit_vhost_name is defined in a host vars file and then we try to use it in a group vars file20:41
clarkbI guess ansible doesn't evaluate those in the order necessary to make this work. That makes sense since host vars override group vars20:42
clarkbI can add a review02 test host vars file but at that point it seems like I may as well just set the name direclty20:42
clarkbI'll revert to the previous patchset and reviewers can tell me if they don't like that20:42
mordredclarkb: that's unexpected - I thought variable expansion from host/group vars was late-bound20:43
opendevreviewClark Boylan proposed opendev/system-config master: Improve gerrit known_hosts management
clarkbmordred: seems not to be20:44
clarkbmordred: is where ansible broke20:44
mordredoh - wait ... that's the host_vars templating20:46
mordredclarkb: that's not expansion not working due to precedence - that's our test framework jinja rendering thinking _it_ needs to be the one to expand that jinja20:47
clarkbI guess I can use a raw quote and then that would be removed when it gets written out ?20:48
clarkblet me try that20:48
opendevreviewClark Boylan proposed opendev/system-config master: Improve gerrit known_hosts management
clarkblike that maybe20:50
mordredthat's totally going to work20:51
mordredunless it doesn't20:51
clarkbits a bit weird to me that ansible insists on checking the name passed against the key passed20:52
clarkbbut I guess that is so they can safely delete if you esnure absent20:52
clarkbfungi: I added a step about adding the host key for loclahost as gerrit2 to known hosts on the etherpad21:30
fungiperfect, thanks21:30
clarkbfungi: we should probably add the playbook command to the etehrpad too?21:36
fungii can though i think we already have it in the linked doc. i'll add it if not21:36
fungiclarkb: i've cut and pasted the example command from step 4 into the etherpad, if you want to double-check that still looks correct21:51
clarkbfungi: I updated the path to the playbook but I think that looks good21:51
ianwclarkb: lgtm.  i guess i only ssh'd as my admin user during the cutover22:13
clarkbianw: note that change won't actually fix prod alone since we need to add the var in in prod hostvars22:13
clarkband then we need to followup with gitea host keys though those are already in there22:14
ianwi'm almost certain i didn't manually add the gitea host keys?22:15
ianwoh, no, that's right, we added a step maybe to do that didn't we22:16
clarkbya we did it during the move when I realized it would break replication without it22:17
ianwyea step 11.2 @
clarkbI think what we'll end up doing is having a { name: keyvalue } dict that we iterate over for gerrit2 and root known_hosts and add them in22:17
clarkbas a followon to what I've done aboev as a first step22:17
clarkbsince ansible wants the name to match what is in keyvalue we have to provide both I think22:18
ianwyep; there's probably a bit of prior work in the borg-backup roles which sets up ssh permissions between server -> backup host22:18
ianwfungi: have you ever had any special dealings with libvirt-python wheels?23:16
ianwthis is in relation to
clarkbianw: libvirt-python is updated/released in sync with libvirt itself, but a new libvirt-python can be built against old libvirt and that is expected to work fine23:21
clarkbin the case above I think the issue is we are treating centos-8 and centos-8-stream as equivalent so the libvirt-python built for centos-8 is installed on centos-8-stream23:21
ianw2021-06-27 04:37:51.647585 | controller | Downloading (554 kB)23:21
clarkbthis is a problme because the wheels needs to be built against the actual libvirt that you have23:21
ianwJun 27 05:03:22.685506 centos-8-stream23:22
clarkbbasically we need a centos-8-stream wheel mirror23:22
ianwhaha yes, i think we just reached the same conclusion :)23:22
ianwthis is a usual yak shaving exercise.  that change installed the package libvirt in devstack.  i'm trying to update devstack to use the latest pip.  the latest pip refuses to uninstall packaged libvirt with the "this is a distutils package" stuff23:23
clarkband then devstack fixed this by not installing libvirt-python with pip which is proably fine as long as nova doesn't need newer features in the library. But doesn't fix the issue that this same problem could pop up for other wheels23:23
fungiianw: long ago i recall we had to install python-libvirt distro packages and they did not even publish the wrapper to pypi. they finally worked out a build process where they could generate python packages of it, but yes still tied to fairly specific libvirt versions i believe23:23
ianwwell, this becomes a problem when something installs using upper-constraints23:24
clarkbianw: wow centos-8-stream libvirt-python is still distutils packaged?23:24
ianwno, this is actually on ubuntu23:24
opendevreviewMerged opendev/gear master: Add libffi header dependency
clarkbbut the package install was done for all distros got it23:24
opendevreviewMerged opendev/gear master: Overhaul package metadata and contributor info
ianwanyway, i should be able to prune fedora 32 today23:25
ianwwhich is where all this started!23:25
corvusi think codesearch may be out of date23:40
corvus is showing more results in zuul/zuul than are actually there23:41
opendevreviewJames E. Blair proposed openstack/project-config master: Remove report-build-page from zuul tenant config
opendevreviewJames E. Blair proposed zuul/zuul-jobs master: Remove success-url
opendevreviewJames E. Blair proposed opendev/base-jobs master: Remove success-url
corvusclarkb, ianw: i think if you could follow up on (matrix-gerritbot) that would be nice23:49
clarkbcorvus: I believe that codesearch operates on a pull system. Not sure how frequently it does that. Are we talking 15 minutes out of date or days?23:52
corvusclarkb: many days23:53
ianwhrm, it has "Unable to create '/run/data/vcs-a833d4e625f7834f10cb701f8d40d2235258d8e0/.git/index.lock': File exists."23:54
ianwalthough it does also say "Continuing..."23:55 fail :)23:55
clarkbcorvus: re the gerritbot change just looking for review on latest ps? I'm not sure what more followup is needed other than for me to dig in enough to +2 ( I didn't prioritze that as others had already done so)23:56
corvusclarkb: yep, just looking for closure from you and ianw one way or another :)23:57
corvusianw: maybe it's safe to just rm that index.lock?23:57
ianwyeah, i've done that (it's some sort of "compass-adapters" repo).  23:58
corvusianw: i'm guessing you just did that? :)23:58

Generated by 2.17.2 by Marius Gedminas - find it at!