Tuesday, 2020-11-17

*** tosky has quit IRC00:06
ianwit looks like codesearch testing is failing repeatedly01:10
*** mlavalle has quit IRC01:10
ianwError: /Stage[main]/Hound/Exec[build_hound]: Failed to call refresh: 'go install github.com/hound-search/hound/cmds/...' returned 1 instead of one of [0]",01:12
ianwi can't actually seem to see what it's saying01:12
*** iurygregory has quit IRC01:32
openstackgerritPaul Belanger proposed zuul/zuul-jobs master: Use --password-stdin for upload-container-image  https://review.opendev.org/76293901:46
openstackgerritPaul Belanger proposed zuul/zuul-jobs master: Use --password-stdin for upload-container-image  https://review.opendev.org/76293901:47
ianwi can probably get this containerised quicker than sorting it out in puppet ...02:00
openstackgerritPaul Belanger proposed zuul/zuul-jobs master: Use --password-stdin for upload-container-image  https://review.opendev.org/76293902:00
openstackgerritPaul Belanger proposed zuul/zuul-jobs master: Use --password-stdin for upload-container-image  https://review.opendev.org/76293902:04
openstackgerritPaul Belanger proposed zuul/zuul-jobs master: Use --password-stdin for upload-container-image  https://review.opendev.org/76293902:09
*** stevebaker has quit IRC02:39
*** stevebaker has joined #opendev03:47
*** auristor has quit IRC03:49
*** donnyd has quit IRC04:07
*** donnyd has joined #opendev04:08
*** stevebaker has quit IRC04:11
*** cmurphy has quit IRC04:21
*** cmurphy has joined #opendev04:22
*** johnsom has quit IRC04:46
*** johnsom has joined #opendev04:49
*** raukadah is now known as chandankumar05:01
*** ykarel has joined #opendev05:13
*** johnsom has quit IRC05:18
*** johnsom has joined #opendev05:19
*** aprice has quit IRC05:27
*** aprice has joined #opendev05:27
*** clayg has quit IRC05:30
*** clayg has joined #opendev05:32
*** ykarel_ has joined #opendev06:03
*** ykarel_ has quit IRC06:05
*** ykarel_ has joined #opendev06:05
*** ykarel has quit IRC06:06
*** marios has joined #opendev06:09
openstackgerritIan Wienand proposed opendev/system-config master: [WIP] codesearch container  https://review.opendev.org/76296006:21
ianwclarkb/fungi: ^ i know we've talked about deprecating this for gitea, but i think the agreement is it's not quite there yet.  i'll work tomorrow to plug the container into system-config, but that works locally for me06:22
*** johnsom has quit IRC06:27
*** johnsom has joined #opendev06:27
*** marios is now known as marios|ruck06:38
*** ykarel_ has quit IRC06:43
*** auristor has joined #opendev06:44
*** ykarel has joined #opendev07:11
*** sboyron has joined #opendev07:22
*** rm_work has quit IRC07:26
*** rm_work has joined #opendev07:28
openstackgerritMerged openstack/project-config master: Build newer gerrit images when jeepyb updates  https://review.opendev.org/76219607:36
*** eolivare has joined #opendev07:42
*** ralonsoh has joined #opendev07:43
*** DSpider has joined #opendev07:52
*** slaweq has joined #opendev07:58
*** rpittau|afk is now known as rpittau08:05
*** ykarel_ has joined #opendev08:09
*** hashar has joined #opendev08:11
*** ykarel has quit IRC08:11
*** guillaumec has quit IRC08:14
*** guillaumec has joined #opendev08:15
*** tobiash has quit IRC08:15
*** tobiash has joined #opendev08:16
*** andrewbonney has joined #opendev08:17
*** guillaumec has quit IRC08:23
*** guillaumec has joined #opendev08:24
*** ykarel_ is now known as ykarel08:51
*** mgoddard has joined #opendev08:52
*** iurygregory has joined #opendev08:52
*** tosky has joined #opendev08:56
*** dtantsur|afk is now known as dtantsur09:04
openstackgerritMerged opendev/irc-meetings master: Update TC to meet weekly  https://review.opendev.org/76284509:07
*** ykarel has quit IRC09:12
*** ykarel has joined #opendev09:12
openstackgerritMerged opendev/irc-meetings master: Update Airship meeting time and week  https://review.opendev.org/76292209:13
openstackgerritMerged opendev/irc-meetings master: move heat meeting time to  Wednesday 1400 UTC  https://review.opendev.org/76028609:13
openstackgerritDmitriy Rabotyagov (noonedeadpunk) proposed openstack/diskimage-builder master: Fix python-stow-versions  https://review.opendev.org/75161009:23
openstackgerritDmitriy Rabotyagov (noonedeadpunk) proposed openstack/diskimage-builder master: Run autoremove on post-install step  https://review.opendev.org/75161409:23
*** sean-k-mooney has quit IRC09:27
*** sean-k-mooney has joined #opendev09:27
*** mgoddard has quit IRC09:29
*** odyssey4me has quit IRC09:50
*** odyssey4me has joined #opendev09:50
*** whoami-rajat__ has joined #opendev10:12
*** mgoddard has joined #opendev10:19
*** kwazar is now known as quasar_10:55
*** quasar_ is now known as quasar10:55
*** quasar is now known as quasar`10:56
*** brinzhang0 has joined #opendev11:50
*** brinzhang_ has quit IRC11:53
*** brinzhang_ has joined #opendev11:55
*** slaweq has quit IRC11:56
*** slaweq has joined #opendev11:57
*** brinzhang0 has quit IRC11:58
*** mgoddard has quit IRC12:01
*** mgoddard has joined #opendev12:15
*** eolivare_ has joined #opendev12:23
*** lpetrut has joined #opendev12:24
*** eolivare has quit IRC12:26
*** roman_g has joined #opendev12:47
yoctozeptoinfra-core http://lists.openstack.org/pipermail/openstack-discuss/2020-November/018817.html - I want your thoughts on this13:30
yoctozeptoand morning!13:31
frickleryoctozepto: use either "infra-root" or "config-core" to trigger some attention ;)13:36
fricklerwe discussed that topic two weeks ago http://eavesdrop.openstack.org/meetings/infra/2020/infra.2020-11-03-19.01.log.html#l-2213:36
fricklerbut no feedback since then, I was about to answer docker that we are not interested in their proposal13:37
fricklerif you want to take over on behalf of the kolla namespace, I think you could either fill out that form for yourself or I can answer docker and then hand over to you13:38
fricklerthis is the reply I received from docker https://etherpad.opendev.org/p/EAfLWowNY8N96APS1XXM13:38
yoctozeptofrickler: hah, thanks; I think someone has already told me that but I keep forgetting :D13:51
yoctozeptofrickler: thanks, interesting; though the mail looks very similar to https://www.docker.com/blog/expanded-support-for-open-source-software-projects/13:52
yoctozeptofrickler: so I am assuming they gave up on the weird parts about 'requiring docker' to be used13:52
yoctozeptofrickler: we can generally agree to all the terms except the last one (as we are looking towards using podman heh)13:52
*** mlavalle has joined #opendev13:56
fricklerwell not sure whether that is considered negotiable, I'd tend to assume that it is rather not13:58
*** roman_g has quit IRC13:59
openstackgerritSlawek Kaplonski proposed zuul/zuul-jobs master: [multi-node-bridge] Add script to configure connectivity  https://review.opendev.org/76265014:00
frickleryoctozepto: would you have time to join the opendev meeting tonight? not sure we'd have much time to discuss docker with all the late gerrit upgrade prep going on, but might be a chance to get feedback from most people involved. though likely I won't attend myself, either14:00
yoctozeptofrickler: well, that's a very encouraging invitation, would not you say? :D14:02
yoctozeptoanyhow, just tried filling out the form template for docker exemption and, oh dear, they are asking many too many questions; seems they are focused on company-owned open source projects14:04
yoctozeptohence all the bs about marketing things14:05
yoctozeptooh well14:05
frickleryoctozepto: I half guessed most of the answers, not sure how relevant those are anyway14:05
yoctozeptonot sure either14:06
yoctozeptoI wonder whether foundation should step in14:06
yoctozeptobut then again14:06
yoctozeptomost projects are not docker-focused14:07
yoctozeptoand we are looking at the podman alternative too14:07
frickleryoctozepto: IIRC clarkb wanted to ask them about their view on the docker offer, but I didn't see any response to that, either14:07
yoctozeptoyeah, saw14:07
frickleryoctozepto: does podman have any image store of their own?14:08
yoctozeptofrickler: none that I know of, just sucking docker's blood by default14:08
yoctozeptolet's say I am not looking for a final solution at the moment, just trying to enumerate our options14:09
yoctozeptoany idea what tripleo decided on doing finally?14:09
fricklerno idea. setting up our own persistent image store might be an option, but that would likely only work with enough contributions from the consuming projects14:11
mgoddardyoctozepto: tripleo switched to a content provider job model14:13
mgoddardyoctozepto: and publishes quay.io14:14
mgoddardyoctozepto: quay.io being RH owned14:14
mgoddardthey do seem to have free public image repos, with no tiering14:14
mgoddardsorry, no rate limiting14:14
mgoddardwhether that will stay forever I don't know14:15
yoctozeptomgoddard: but 19:21:24 <clarkb> yes I seem to recall tripleo ruled out quay as a quic fix because they have rate limits too14:15
yoctozeptowell, if it does not14:16
yoctozeptothen we can migrate there too14:16
yoctozeptono regrets, mr dockerhub14:16
mgoddarddifferent rate limit: https://bugs.launchpad.net/tripleo/+bug/188912214:19
openstackLaunchpad bug 1889122 in tripleo "mirror timeouts in upstream causing undercloud and standalone failures" [Critical,In progress] - Assigned to Ronelle Landy (rlandy)14:19
*** d34dh0r53 has quit IRC14:22
openstackgerritMerged openstack/project-config master: Clean up pti-python-tarball python-branch-tarball  https://review.opendev.org/76269914:25
*** brinzhang_ has quit IRC14:26
mgoddardyoctozepto: are you making notes somewhere?14:27
fricklerhttps://docs.quay.io/issues/429.html still doesn't mention any concrete limit. it might get mitigated by not using our mirrors for that, increasing total load on them in turn. someone inside redhat might want to talk to the proper people to get some reasonable solution there14:27
fricklerlike not limiting our mirrors14:27
yoctozeptofrickler: yeah, that sounds feasible14:28
yoctozeptomgoddard: I can start14:28
mgoddardthis sort of links up with switching to more of a build pipeline, in which case we would use the opendev intermediate registry14:29
*** Vivek has joined #opendev14:30
*** d34dh0r53 has joined #opendev14:30
yoctozeptomgoddard: yes, changing the approach is fine for other usage but it will not help with kolla-ansible jobs that do not wish to build14:33
yoctozeptoand do note these are the only ones affected14:33
yoctozeptotripleo seems to have it a bit more complicated though14:33
mgoddardyoctozepto: build jobs still need to pull base images, I have seen them hit14:33
yoctozeptomgoddard: but base distro images should be already exempted14:34
mgoddardseems not14:34
yoctozeptowell then we can do nada about it14:34
mgoddardI saw it with ubuntu:20.0414:34
yoctozeptoeh, life14:36
fungiyeah, canonical would presumably have to ask dockerhub to grant their namespace an exemption14:36
yoctozeptothough it could be related to the fact that we pull all the others14:36
yoctozeptofrickler, fungi: I am not really getting that mirror/proxy thing that has been mentioned14:37
yoctozeptoI thought dockerhub/quay.io gets contacted directly no matter what14:37
yoctozeptoor is it related to how tripleo did it (i.e. they used infra mirroring for quay)14:37
yoctozeptomgoddard: we are not using a proxy/mirror either, are we?14:38
mgoddardyoctozepto: we use the registry mirror14:38
mgoddardwhich hurts in some ways and helps in others14:39
yoctozeptoah, right14:39
yoctozeptoI must get my mindset fixed that we mean pulls and not pushes14:40
yoctozeptowell, it depends on how cdn-like dockerhub is14:40
yoctozeptothe closer to a real cdn, the better to skip the mirror14:41
yoctozeptothough the internet pipe is unhappy14:41
yoctozeptodecisions, decisions...14:41
mgoddardwell I think there are different mirrors for different clouds14:42
yoctozeptoyeah, but then it's a reason why the failures were random (different clouds)14:43
yoctozeptorather than any consolation14:43
yoctozeptoso the quay.io issue might also be amended by making the mirrors artificially rate-limit themselves14:48
yoctozeptosince they know how often they can contact upstream14:49
yoctozeptoI think nginx can do that14:49
*** lpetrut has quit IRC15:16
fricklerinfra-root: since we don't seem to be able to get any fix or feedback regarding limestone, I'm going to propose disabling the cloud now, despite increasing our resource issue, but maybe the reduction in retries needed can make up for it15:44
fricklersee also https://52409a2182703e644968-eb284aa94e3a202614740e20d7b20329.ssl.cf5.rackcdn.com/762650/5/check/zuul-jobs-test-multinode-roles-opensuse-15/ffc91be/job-output.txt for yet another failure mode15:44
openstackgerritJens Harbott (frickler) proposed openstack/project-config master: Disable limestone provider due to IPv4-less nodes  https://review.opendev.org/76303815:47
clarkbthe way our docker mirrors work is they always talk to docker hub for the manifests because manifest fetches are authenticated even when pulling anonymously. This makes apache not cache those responses even though headers are set allowing that15:48
clarkbthen from the manifest it pulls the layer blobs from object storage without auth and those are cached15:49
clarkbpreviously docker rate limited layer pulls becausethey are the actual data transfer15:49
clarkbwe did the right thing in that setup15:49
clarkbnoe they rate limit manifests15:49
clarkbpeople found the old limits too confusing or something so they changed it15:50
*** ykarel has quit IRC15:56
clarkbmgoddard: yoctozepto is there a tldr on the quay issue? fwiw I reached out to some people at red hat/ibm who checked and there aren't supposed to be any existing quay limits aiui. But understanding what tripleo hit would be good for understanding this general problem space better16:00
fungiearly in the discussion about dockerhub rate limits, there was speculation that quay *might* impose similar limits in the future if they get overrun16:08
fungiwhich prompted the idea that relying on one global registry is a risk on its own16:08
*** iurygregory has quit IRC16:29
*** hamalq has quit IRC16:56
*** hamalq has joined #opendev16:57
*** marios|ruck is now known as marios|out17:01
*** Vivek has quit IRC17:01
yoctozeptoclarkb: only https://docs.quay.io/issues/429.html17:09
fungii guess we should be on the lookout for http 429 errors there17:11
fricklerfungi: clarkb: wdyt regarding disabling limestone? https://review.opendev.org/76303817:18
*** mlavalle has quit IRC17:18
clarkbI think wecan do that uf we dont want ti try the ipv4 connectivity test in the basejob when we check dns resolution and similar17:19
*** iurygregory has joined #opendev17:19
*** pabelanger has joined #opendev17:21
pabelangerclarkb: fungi: do you happen to know the spec on insecure-registry?17:22
pabelangerI cannot see it in cacti17:22
*** marios|out has quit IRC17:25
*** rpittau is now known as rpittau|afk17:27
clarkbpabelanger: 4vcpu and 4GB of ram17:27
clarkbmost of the memory is used by caches17:27
*** eolivare_ has quit IRC17:27
clarkbstorage is all remote in swift17:27
pabelangerclarkb: is it zuul-registry that needs the RAM or something else?17:29
pabelangerI am curious if 1vcpu / 1gb would fligh17:29
clarkbits actively using about 400MB of memory but then also using the rest for caches17:29
clarkbI think having a bit more memroy for caches is worthwhile since it should speed up data transfers I expect17:30
fungivirt:1575340, res:95808, shared:391617:30
fungiper top17:30
fungifor the zuul-registry process17:30
pabelangerk, so it will cache layer in RAM when possible, then hit swift?17:30
*** mlavalle has joined #opendev17:31
fungilooks that way17:31
pabelangerTIL, thanks17:31
fungidockred is using more vmem than the zuul-registry process17:32
fungier, dockerd17:32
fungibut i guess it's the parent17:32
pabelangerah, right. it runs as a container17:33
clarkbfrickler: validate-host is the role I'm thinking of: https://opendev.org/zuul/zuul-jobs/src/branch/master/roles/validate-host17:33
fungimmm, actually containerd is the parent (indirectly, via containerd-shim) according to the process list17:34
pabelangerstill, that is helpful. thanks17:34
fungidockerd may just be an inefficient memory hog17:34
clarkbfrickler: maybe we can add a flag to make https://opendev.org/zuul/zuul-jobs/src/branch/master/roles/validate-host/library/zuul_debug_info.py#L66-L90 require ipv4?17:34
fungicould insert a "passed = False" around line 8517:36
clarkbfungi: ya but I don't think we want to require working ipv4 in all cases since this is in zuul-jobs17:37
clarkbbut if we can make that an option it should force those jobs to retry up to 3 times if they don't have working ipv4 which should work for our purposes17:37
fungiright, a conditional there i suppose17:38
fungithough best would be to refactor that routine to set two results and then use conditionals on rolevars to decide what's required17:39
fricklernot sure which jobs all would need that check17:39
clarkbfrickler: for us that runs on every job17:39
clarkbwhich is possibly overkill but maybe that is ok17:40
clarkb(that == the validate-host role)17:40
frickleralso not sure how big the chances are for hitting three limestone nodes in a row17:40
openstackgerritTristan Cacqueray proposed zuul/zuul-jobs master: DNM: generate ansi log for benchmark purpose  https://review.opendev.org/76305417:40
clarkbfrickler: well also most limestone nodes seem to work17:40
fricklerbut feel free to try that if you want, /me doesn't have time for it currently17:41
clarkbI think we can disable the cloud now since it is a quick and effective solution17:41
clarkbthen if we find time we can update the validate-host role and reenable limestone17:41
fricklermnaser: did you see the vexxhost ci failures due to a missing master branch?17:41
clarkb(I hvae +2'd the limestone disable change)17:41
mnaserfrickler: yes, spent a whole bunch of time trying to fix them to no avail, override-branch is set to `main` and i have dug through the zuul code a bunch, still trying to figure out the root cause17:42
clarkbpabelanger: ^ may have insight since they use non master branches too17:42
pabelangeryah, we've migrated some projects in ansible to main17:43
pabelangerand deleted master17:43
pabelangerseems to work17:43
clarkbpabelanger: are any of them config projects?17:43
clarkbI seem to recall that we thought that may be related to mnaser's situation17:44
pabelangerclarkb: no, just untrusted right now17:44
pabelangerclarkb: do you mind adding me to https://review.opendev.org/#/admin/groups/2067,members and https://review.opendev.org/#/admin/groups/2068,members https://review.opendev.org/685791/ was the change to create the groups17:45
fungii can do it if clarkb is busy17:45
clarkbfungi: pleaes do, I don't have that ssh key loaded right now17:45
*** roman_g has joined #opendev17:45
pabelangerfungi: wfm, thank you17:46
fungi#status log added pabelanger an initial member of new ansible-role-zuul-registry-core and ansible-role-zuul-registry-release groups in gerrit17:47
openstackstatusfungi: finished logging17:47
pabelangerI have likely asked before, for non-openstack project is the plan to move them out of that tenant into say the opendev tenant?17:54
clarkbwe've already been putting new things in their own tenants or opendev/ so yes17:55
clarkbbut I have no idea when that will get prioritized just looking at the list of todos17:55
pabelangerhow is it decided to get a new tenant or go into opendev?17:55
clarkbI don't think there has been any hard rule on it. But more and more we're prefering a separate tenant17:57
openstackgerritJeremy Stanley proposed zuul/zuul-jobs master: validate-host: Options to require v4 and v6 routes  https://review.opendev.org/76306518:01
*** mgoddard has quit IRC18:02
*** andrewbonney has quit IRC18:05
openstackgerritJeremy Stanley proposed zuul/zuul-jobs master: validate-host: Options to require v4 and v6 routes  https://review.opendev.org/76306518:20
*** cloudnull is now known as kecarter18:36
*** kecarter is now known as cloudnull18:36
openstackgerritPaul Belanger proposed openstack/project-config master: Add zuul/zuul-regsitry to openstack tenant  https://review.opendev.org/76307118:40
pabelangerclarkb: fungi: corvus: do you mind commenting on ^, zuul-registry isn't in the openstack tenant, which is something I'd like to be able to reference from ansible-role-zuul-registry role18:41
clarkbthe key is not loading configs from it which you've got so +318:43
*** d34dh0r53 has quit IRC18:43
*** d34dh0r53 has joined #opendev18:46
fungiyep, seems fine to me18:49
fungipabelanger: wrt the openstack tenant, the main reason unofficial projects might still get added there is if they plan to make use of job configuration from openstack projects, that quickly gets complicated to do cross-tenant with things like devstack/tempest18:51
fungiit's doable, but you're talking probably 20+ required projects to pick up services, plugins, et cetera18:52
fungiand as soon as the official jobs add another project into the mix you're broken until you add that to your tenant18:53
pabelangerfungi: makes sense18:53
fungiunfortunately there's no magic "i want to run this job, figure out what repos i need and include them in my tenant" functionality18:54
fungiso it can be a fair about of trial and error to get working, and then a bit of a treadmill to keep it working over time18:55
openstackgerritMerged openstack/project-config master: Add zuul/zuul-regsitry to openstack tenant  https://review.opendev.org/76307119:02
*** mgoddard has joined #opendev19:21
*** stevebaker has joined #opendev19:22
clarkbfungi: I just approved https://review.opendev.org/#/c/762712/. Maybe you can take a look at https://review.opendev.org/#/c/757153/ and https://review.opendev.org/#/c/762191/ and approve those if they look good to you?19:39
clarkbthats all pre upgrade prep (tehre are other changes under that topic too if you have time)19:39
fungiyup, was on my slate for after the meeting19:39
clarkbfungi: I'll be finding lunch next but will monitor irc and be around to help with those changes as they land if necessary (so feel free to approve if they look good)19:49
fungisounds good, reviewing them in a few19:50
*** roman_g has quit IRC20:16
*** dtantsur is now known as dtantsur|afk20:17
openstackgerritIan Wienand proposed opendev/system-config master: [WIP] codesearch container  https://review.opendev.org/76296020:17
*** roman_g has joined #opendev20:18
*** smcginnis has quit IRC20:24
*** mgoddard has quit IRC20:31
openstackgerritMerged opendev/system-config master: Add example maintenance message for Gerrit server  https://review.opendev.org/76271220:36
clarkbtheres one20:41
fungimost have been approved, i'm waiting for the jeepyb change to merge before approving its dependency though20:44
fungier, reverse-dependency i guess20:46
openstackgerritMerged zuul/zuul-jobs master: Use --password-stdin for upload-container-image  https://review.opendev.org/76293920:53
*** roman_g has quit IRC20:56
*** roman_g has joined #opendev20:57
*** roman_g has quit IRC20:57
*** roman_g has joined #opendev20:58
*** roman_g has quit IRC20:58
*** roman_g has joined #opendev20:58
*** roman_g has quit IRC20:59
clarkbfungi: for the maintenance notice, should we just go ahead and hand edit that file on the server now? then we can do the comment removals in apache config day of?20:59
*** roman_g has joined #opendev20:59
*** roman_g has quit IRC20:59
clarkbetherpad is coming together I think21:00
*** roman_g has joined #opendev21:00
*** roman_g has quit IRC21:00
*** roman_g has joined #opendev21:01
*** roman_g has quit IRC21:01
*** roman_g has joined #opendev21:01
*** roman_g has quit IRC21:02
*** roman_g has joined #opendev21:02
*** roman_g has quit IRC21:02
*** roman_g has joined #opendev21:03
*** roman_g has quit IRC21:03
*** roman_g has joined #opendev21:04
openstackgerritIan Wienand proposed opendev/system-config master: [WIP] codesearch container  https://review.opendev.org/76296021:04
*** roman_g has quit IRC21:04
*** roman_g has joined #opendev21:05
*** roman_g has quit IRC21:05
*** hashar has quit IRC21:06
*** hashar_ has joined #opendev21:06
clarkbfungi: looks like the apache config applied as expected on prod21:09
clarkbI'm going to put review-dev in the emergency file once these other changes have applied as they should be the last set of safe things for it'21:09
fungiyeah, we can draft the maintenance message ahead of time. happy to do that in a bit21:11
openstackgerritIan Wienand proposed opendev/system-config master: [WIP] codesearch container  https://review.opendev.org/76296021:13
openstackgerritMerged opendev/jeepyb master: Make local git dir creation optional  https://review.opendev.org/75859721:15
*** hashar_ is now known as hashar21:20
openstackgerritIan Wienand proposed opendev/system-config master: [WIP] codesearch container  https://review.opendev.org/76296021:20
openstackgerritMerged opendev/system-config master: Disable change.move and enableSignedPush in gerrit  https://review.opendev.org/75715321:22
openstackgerritMerged opendev/system-config master: Set gerrit.serverId in gerrit.config  https://review.opendev.org/76219121:22
pabelangergreetings, is build-python-release jobs broken? https://zuul.opendev.org/t/zuul/build/225b52058c8e454ea4152f9008a0b5a321:26
pabelangerpython setuptools is missing is seems21:26
clarkbhrm no that job is in zuul-jobs and hasn't changed in a long time21:27
clarkbI thought it may be related to changes fungi had made to release jobs but those must've been openstack psecific21:27
pabelangerokay. Trying to see if setuptools has been removed from images at all21:28
pabelangerupdating here, for people following. I missed setting release_python: python321:30
fungisounds like it was that we install the package toolchain for python3 in our jobs by default, you have to tell it if you want python2 libs added21:30
clarkbfungi: I think those next changes have applied too21:33
clarkbfungi: also fyi you responded to frickler but not in the way gerrit expected at https://review.opendev.org/#/c/758598/1/playbooks/roles/gerrit/tasks/main.yaml21:35
clarkbtook me a minute to figure that out :)21:36
fungioh, yep, i keep forgetting gertty picks the old side by default if you comment on an unchanged line in the unified diff view21:37
clarkbfungi: also I'm thinking that since we're doing these config changes and also docker volume mount changes we should maybe restart gerrit in the near future just to be sure everything is happy before we go to the upgrade?21:37
clarkbmaybe that is a tomorrow morning task21:38
fungiyeah, just so there's no nasty surprises if we need to roll back21:38
fungii agree21:38
clarkbfungi: ianw https://github.com/go-gitea/gitea/milestone/67 looks like there are still two open issues on the gitea 1.13 milestone21:51
hasharclarkb: hi ! For your Gerrit upgrade,  we had a lot of fix up for the zuul plugin.  The series end at https://gerrit-review.googlesource.com/c/plugins/zuul/+/27505521:54
clarkbhashar: thanks. We aren't going to add that plugin to start. Idea is to upgrade, get settled in then start investigating plugins and features21:55
hasharclarkb: I am not sure whether you are using it at opendev, but if so, you will definitely require those. That makes it compatible ;)21:55
hasharah ok :)21:55
clarkbhashar: I think we'd like to, we've just decided that we need to do one thing at a time to make progress21:55
hasharsounds wise21:55
hasharI have updated our doc tonight, and we have some basic procedure to fetch the proper plugins from upstream https://wikitech.wikimedia.org/wiki/Gerrit#Fetch_our_additional_plugins21:56
hasharmaybe that will be of some help when you reach that point :]21:56
clarkbdocs are always helpful, thank you for the link21:57
hasharelse you can probably borrow them from upstream CI : https://gerrit-ci.gerritforge.com/view/Plugins-stable-3.2/21:57
ianwclarkb: huh, they look different to the blockers i saw before, so, progress? :)21:58
clarkbianw: I think that list may be what I had last seen a couple weeks ago? it does seem very close21:58
*** sboyron has quit IRC21:58
*** smcginnis has joined #opendev21:59
ianwsigh, so the devel job uses tox and installs ansible, and then ansible-base can't install over it22:02
*** ralonsoh has quit IRC22:03
pabelangerianw: yup, it isn't fun22:03
ianwdoes anyone know why we run "run tox without tests" in the tox role first, then install siblings?22:03
clarkbianw: its to install the sibling via pypi then overwrite it iirc22:03
clarkbianw: otherwise there isn't really a way to split out the package ebcause pip doesn't let you exclude things iirc22:03
clarkb(so a dep of a dep can pull it in at a different version and now you don't test the sibling)22:04
pabelangerianw: one thing we are going to do, is create a fake 'ansible' package when installing from ansible/ansible devel to handle requirments files22:05
pabelangerbut haven't done that work yet22:05
ianwsigh yeah.  it breaks in this case though because the initial testinfra install pulls in ansible.  i wonder if it could uninstalling siblings first, then reinstall22:05
pabelangerbut also, ansible/ansible devel is not ACD22:05
ianwpabelanger: you know, i feel like we already did that somewhere ...22:05
ianwyep @ https://opendev.org/opendev/system-config/src/branch/master/playbooks/roles/install-ansible/tasks/install_ansible_stub.yaml22:06
ianwproblem is the testinfra tox install22:07
ianwthis did work, briefly.  the "will not install" thing must be newish22:07
ianwpart of the point was to test testinfra against ansible-devel ...22:08
pabelangerianw: testinfra on devel may just no work any more22:10
pabelangergiven there might be modules now missing22:10
pabelangerwhat you should test against is ansible (ACD) devel22:10
pabelangerwhich is different then ansible/ansible devel22:10
pabelangeransible/ansible is becoming ansible-core22:10
ianwpabelanger: so ... is there something I can clone? :)  i'm around the whole ansible/ansible-base split but i'm not sure how acd is built22:12
pabelangergit clone, I don't believe so22:12
pabelangerthe acd package is build via scripts of collections from galaxy22:13
pabelangerlet me see if I can find where the snapshot is stored22:13
openstackgerritMerged opendev/system-config master: Stop managing gerrit's local git mirror dir  https://review.opendev.org/75859822:13
ianwpabelanger: hrm, that's still going to be hard to integrate into a tox environment using zuul-jobs tox runner anyway.  it will install the siblings from required-projects:, but not a tarball/egg22:14
pabelangerianw: yah, there is no single git source now for 'released' ansible22:15
pabelangerit is pull in from all over the place22:15
ianwi'm thinking maybe the zuul-jobs tox role should get something like "pre_unisntall_packages" or something; then it could remove "ansible" and then it would correctly install ansible-base22:15
ianwalthough, i'm not sure if the actual dependency resolver would then get unhappy and try removing testinfra and everything from the environment :/22:15
pabelangerianw: I am being pointed to https://github.com/ansible-community/antsibull/pull/19522:16
pabelangerwhich should create nightly builds22:16
pabelangerwhat would be neat, is if we turned that into a zuul-jobs22:17
pabelangerthen we both could use it22:17
ianwdmsimard is @redhat again?22:19
pabelangerre-joined last week I think22:19
pabelangerI have to run, good luck on ansible-base issue22:20
ianwpabelanger: hrm, yes nightly would be ok, but i'm still not sure how to convince tox to install it22:21
ianwwe could hack maybe.  have a requirements.txt that usually just specifies "ansible" but for the devel tests is modified to install from local checkout22:25
*** hashar has quit IRC22:26
fungiyeah, any hacks you're considering around selectively removing dependencies, even temporarily, are likely break horribly when the next pip release happens and it switches to statefully considering the installed dependency set22:28
fungiright now you can say "uninstall this package (which is a dependency of something i have installed)" and pip will be like, okay whatevs22:31
fungisoon pip will likely refuse, "the package you're trying to remove is needed by another package" ...or maybe remove the packages which depend on that too22:32
fungianyway, i wouldn't assume pip's behavior will remain consistent there. if you're going to try a clever trick like that, be sure to also test it with the experimental dep solver flipped on22:33
clarkbI wonder if that will break install siblings22:34
openstackgerritIan Wienand proposed opendev/system-config master: [WIP] codesearch container  https://review.opendev.org/76296022:34
clarkbfungi: looks like the jeeypb update may have happened?22:34
clarkbbasing that on the contents of the docker compose file22:34
ianwfungi: yeah, i'm not sure it can be done.  if testinfra says it depends on "ansible" and "ansible-base" will not install when "ansible" exists ... it's just broken :/22:35
ianwthere's no way to say "ansible | ansible-base"22:35
* clarkb gonna take a break for a bit now for some exercise since those changes all seem to be landing without major issue22:35
*** stevebaker has quit IRC22:37
ianwwait, it's "ansible-core" ans "ansible-base" ... but same general issue22:37
*** stevebaker has joined #opendev22:38
fungiit used to be possible before the expectation of static dep metadata for python packages22:42
fungiin short, you could create a package which dynamically adjusts its install_requires and then depend on that22:43
fungigranted, the install_requires become chiseled into stone at install time, so you'd have to reinstall to change them, which probably doesn't solve much22:45
ianwof course there's "set the ANSIBLE_SKIP_CONFLICT_CHECK environment variable", the change that adds that https://github.com/ansible/ansible/commit/54b002e1acad1e8d88e81965323d47ddb8c234fb tells me about how it fixed newlines, but doesn't describe something like *why* this necessary so it could be worked around22:51
ianwohhh, it's because it's renaming itself ansible-core; this is why it used to work, when it was ansible base i think22:55
fungiso can we just directly use ansible-core there?23:00
openstackgerritIan Wienand proposed opendev/system-config master: test if ansible-core install works  https://review.opendev.org/76309923:03
*** quasar` is now known as parallax23:03
*** parallax is now known as kwazar23:07
*** kwazar is now known as parallac23:13
*** parallac is now known as parallax23:13
*** parallax has joined #opendev23:14
*** slaweq has quit IRC23:21
dmsimardianw: o/ I'm indeed back at red hat in the ansible community team23:22
ianwdmsimard: cool, they're lucky to have you :)23:24
dmsimardstill learning about the different bits for ansible >=2.10, collections and I see you've already found out about ansible-base/core rename23:24
dmsimardTransition from 2.9 to 2.10 is a bit... hard. There's a lot of work to do to improve it due to all the underlying changes from splitting out what isn't "core"23:25
ianwyes; https://review.opendev.org/#/c/763099/ ... i can either have the job working against -base or -core, but not both :/23:27
dmsimardI wasn't even aware that core was a thing yet, only heard about plans for the rename23:27
dmsimarddoesn't seem like it's a thing yet, at least according to https://pypi.org/project/ansible-core/#history23:29
ianwyeah, that's the problem; the base job tries to install that blank package23:31
ianwnot ansible-base, but the system-config base23:31
ianwheh, what a mess23:31
openstackgerritIan Wienand proposed opendev/system-config master: [WIP] codesearch container  https://review.opendev.org/76296023:32
dmsimardyeah no comments on the rename :/23:33
ianwis there actually any problem installing ansible-core when ansible-base is installed?  i feel like the check might actually be wrong23:33
ianwdmsimard: yeah, if you could get ansible changelog comments to be more openstacky, that would be great :)  i think it's something we generally do well at23:34
dmsimardthey actually adopted something reno-ish for changelog fragments23:35
dmsimardand I say reno-ish because it's the same general concept but not reno23:36
dmsimardI don't know about ansible-core vs ansible-base (yet), sorry :(23:37
ianwthat's ok, testing against bleeding edge is probably prety niche23:38
dmsimardah, so reading my backscroll23:39
dmsimardgit source devel is now "ansible-core"23:39
dmsimardbut it's not released yet23:39
ianwdmsimard: right, to sumarise our issue:23:41
ianwwe install ansible-base in the tox environment for testinfra @ https://opendev.org/opendev/system-config/src/branch/master/tox.ini#L4123:41
ianwfor the system-config-base-devel-job, we have ansible as a required-project and so the tox role tries to install it as a sibling, this is to test our stuff against bleeding edge23:41
ianwhowever, that now fails, as ansible-core refuses to install over/alongside ansible-base23:42
dmsimardmakes sense23:42
dmsimardI don't have a solution for you but let me take that feedback to the team tomorrow23:42
ianwi think we need to install ansible-base in the "release" job, and ansible-core in the -devel job.  not quite sure how to do that from our side23:44
dmsimardthe error message is misleading too23:44
*** DSpider has quit IRC23:52
clarkbianw: in the release job we should just install "ansible" iirc23:53
clarkbsince "ansible" is the compiled package?23:54
ianwclarkb: yeah, i think it's working ATM because what we need from testinfra can survive with "ansible-base"23:54
ianwbasically testinfra is just using ansible to ssh in, and it's convenient because we just pass it the already written inventory file23:55
*** zbr has quit IRC23:57
*** zbr4 has joined #opendev23:57
*** tosky has quit IRC23:58

Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!